GenAI QA

About the Role

We are looking for an experienced GenAI QA Engineer responsible for testing, validating, and ensuring the quality, reliability, accuracy, and performance of Generative AI applications and Large Language Models (LLMs).

Key Responsibilities

• GenAI Application Testing
• Test Generative AI applications, chatbots, virtual assistants, and AI-powered solutions.
• Validate LLM-generated responses for accuracy, relevance, consistency, and completeness.
• Test prompt engineering scenarios and prompt-response behavior.
• Validate multi-turn conversations and contextual understanding.
• LLM Testing & Evaluation
• Evaluate LLM responses for hallucinations, bias, toxicity, and factual correctness.
• Perform prompt injection and jailbreak testing.
• Validate response quality, relevance, groundedness, and context retention.
• Define test datasets and evaluation benchmarks.
• RAG Testing
• Test Retrieval-Augmented Generation (RAG) applications.
• Validate document ingestion, chunking, embedding, and retrieval accuracy.
• Test semantic search and vector database integration.
• Validate whether generated responses are grounded in retrieved documents.
• Test citation accuracy and source relevance.
• API & Automation Testing
• Perform API testing using Postman or REST Assured.
• Develop automation scripts using Python or Java.
• Automate GenAI test scenarios and regression test suites.
• Integrate automated tests into CI/CD pipelines.
• Performance & Security Testing
• Test response latency, throughput, scalability, and token consumption.
• Validate application behavior under concurrent user loads.
• Test security vulnerabilities specific to LLM applications.
• Validate data privacy and sensitive information leakage.
• Reporting & Collaboration
• Prepare test strategies, test plans, test cases, and test reports.
• Collaborate with developers, data scientists, and AI engineers.
• Identify defects, analyze root causes, and track resolutions.

Requirements & Tech Stack

• 1+ years of software testing experience
• 1+ years of exposure to Generative AI / LLM testing
• Strong understanding of software testing methodologies
• Hands-on experience with Python-based testing and API automation
• Experience testing chatbot or AI-powered applications
• Familiarity with RAG architectures, vector databases, AI agents, and LLM evaluation frameworks is highly desirable.

Qualifications

• Bachelor's degree in Computer Science, IT, or a related field.