AI Automation Testing

About the Role

Seeking an AI Automation Testing Lead/Manager to design, validate, and automate testing for AI-powered platforms, including computer vision solutions, machine learning models, and GenAI/RAG applications. The role requires a strong understanding of AI system behavior and robust quality engineering practices to ensure accuracy, reliability, performance, security, and production readiness across the AI solution lifecycle.

Key Responsibilities

• Own end-to-end quality assurance for Vision AI platforms and applications, covering UI, APIs, workflows, integrations, AI models, and production releases.
• Define and implement test strategies, test plans, regression suites, quality metrics, and release gates for Vision AI solutions.
• Execute functional, integration, regression, exploratory, system, and user acceptance testing across the application stack.
• Develop and maintain automated test frameworks for UI, API, and backend services using Python, Selenium, and AI testing frameworks.
• Integrate automated testing into CI/CD pipelines to enable smoke, regression, nightly, and release validation.
• Validate computer vision models and pipelines, including image/video ingestion, object detection, classification, tracking, and event/alert generation.
• Design, curate, and maintain test datasets and benchmark scenarios to evaluate model accuracy, performance, robustness, and edge cases.
• Conduct AI model validation and regression testing using performance metrics, baselines, and real-world datasets.
• Verify the reliability and consistency of GenAI-powered capabilities, including prompt workflows, reasoning, and AI-assisted automation features.
• Perform performance, scalability, and reliability testing for real-time Vision AI services, ensuring required throughput, latency, and system stability.
• Lead security testing activities, including vulnerability assessment, threat validation, adversarial testing, and secure deployment verification for AI applications.
• Drive defect triage, root-cause analysis, quality reporting, and continuous improvement initiatives in collaboration with engineering, product, and DevOps teams.
• Ensure Vision AI solutions meet quality, security, compliance, and production-readiness standards before release.

Requirements & Tech Stack

• Strong foundation in Quality Assurance principles, including test strategy, test planning, test design, defect management, and root-cause analysis.
• Hands-on experience in test automation using Python and industry-standard frameworks such as Selenium, Cypress, or equivalent.
• Proven experience testing web applications, APIs, and microservices, including REST API validation, authentication workflows, and payload verification using tools such as Postman.
• Good understanding of databases and SQL, with the ability to query and validate data.
• Experience integrating automated test suites into CI/CD pipelines and supporting continuous quality validation.
• Strong understanding of AI/ML systems and methodologies for validating model accuracy, reliability, and performance.
• Experience testing GenAI and LLM-based applications, including prompt engineering workflows, multi-turn conversations, tool/function calling, and response evaluation.
• Knowledge of hallucination detection, grounding validation, response quality assessment, and safety testing for AI applications.
• Understanding of computer vision solutions and video analytics, including validation of object detection, classification, tracking, and common AI performance metrics such as Precision, Recall, F1-Score, and Accuracy.
• Experience designing and executing security testing using tools and methodologies such as OWASP ZAP, application security assessments, vulnerability validation, and secure SDLC practices.
• Hands-on experience with performance, scalability, and load testing using tools such as JMeter, LoadRunner, or equivalent.
• Strong analytical, troubleshooting, communication, and stakeholder collaboration skills.

Familiarity with cloud-native applications, containerized environments, and modern software development and DevOps practices is preferred.