Last week I posted a question on AI testing. I am excited to receive so many responses and viewpoints. As I continue my journey in the AI world, I feel inspired to contribute my thoughts to the chatter.
Looking back, I remember, the same day last week, I was a blank canvas in the AI world, trying to find even the smallest information I could about AI testing. I am thankful to the ML and Testing community who shared their thoughts and helped in forming the vision for AI testing. Today I feel I am climbing the stairs of the AI skyscraper. I have no idea how tall is the building as the field of AI is so vast and still growing every day. But I know I am climbing up. So here are my baby steps thoughts about AI Testing.
Firstly, to set the context, this conversation is about testing AI applications built to deliver business solutions. It is not about using AI in testing tools.
I come from the testing world, mind-born and grown up in the testing world. In my mind, I was trying to find out whether the guiding principles I value in the testing world still adds value to the AI world. Test early, Test often and Test based on business value. As I just climbed to the first floor in the AI building, my viewpoint is, it does.
Test Early: My understanding so far is we usually have at least three sections in the AI-based applications – Data engineering, Data science, and Front-end application(maybe API based ). Yes, it makes sense to start testing early and start testing the data that the AI models use. As all of us working in AI knows – Garbage in, garbage out. If the quality of data that is fed into the AI model is not good, we can’t have the right predictions. Testing the data and any data transformation that is happening on the data is critical. Testing data allows you to integrate testing early into the AI pipeline.
Test Often: We can think of testing often in two ways. Testing frequently and testing each and every component of the AI pipeline. Building automated tests and setting them up on the continuous integration pipeline will allow running the tests frequently and on-demand. If the testing pipeline is hooked up to the code deployment process, it will allow adopting quality-built in approach for AI development process. As the developer is building the new code or refactoring they can always rely on the built-in test harness to give them feedback and allow them to know if they have achieved the definition of done for the requirement/user story they are working on.
Each and every component in AI: Data Engineering, Data Science and Front end application need to have their own tests and finally need to be tested as end to end. This will allow you to identify the root cause of issues faster and also allows to ensure all the integration points are working well together. My viewpoint to the question I posted last week – “Does the concepts of unit testing and integration testing still work in AI world or the borders of types of testing become fade in AI ?” – Yes, it works. AI pipeline components should have unit tests built into the code and also have integration test to ensure the components of Data Engineering, Data Science and Front ends are giving you the right predictions.
Test based on business value: In the context of testing an AI application which is built to deliver business value, it is critical to understand the business problem statement and expectation. Every step we should be having tests which focus on alignment of each and every component to the business goal.
Data engineering side it is extremely important to ensure that the data quality is tested keeping in mind the business meaning of the data. From a technical perspective the data may be fine, e.g. if the data is numeric – 0, 100, 200 all may be valid input. But business may say if the data is over 100, it may indicate an error condition for the sensor which is sending the data. When building the test for data engineering we always have to keep in mind the business accuracy of the data.
Data Science is no exception, as we build the model we always have to keep in mind the expectation that business has on the prediction trends. As I learned from an article from one of the best minds in testing, we have to keep checking if the data pattern changes, is the model prediction changing toward the expected direction. Is the model picking up trigger points in the data patterns to start shifting the prediction? We have to keep real-time monitoring of the performance of the model to ensure the model is processing the training data, test data, hold data and producing predictions in a way that business expects.
Front end application as we already know in the testing world, need to be aligned to end-user expectation. It is no exception in the AI world.
I am climbing the tallest building in my testing world journey, so I am sure my learning at each step will help to shape my understanding even deeper -they may change, they may remain same, I don’t know. But I guess that is what Agile is( which I am a big follower of), learn as you go. I am glad to make the journey with the AI, ML and Testing community and learn from them. Together we climb!