Machine Learning QA Engineer

About the Role

Matters.AI builds AI native data security that detects sensitive information across enterprise documents and data. You'll work with our ML team to test whether those models actually get it right. You'll build ground truth datasets, measure model performance, dig into every false positive and false negative, and validate each new model version before it ships. This role suits freshers and early career folks who enjoy finding exactly where and why a system breaks.

Key Responsibilities

• Annotate, validate, and review datasets against defined guidelines, and maintain ground truth datasets for evaluation.
• Evaluate model outputs and classify every result as a true positive, false positive, or false negative.
• Calculate and track precision, recall, F1 score, confusion matrices, and per class performance.
• Run error analysis to find recurring failure patterns and classification gaps.
• Write and run test cases for new model versions, prompts, rules, thresholds, and classification logic.
• Run regression tests across model versions to catch improvements and regressions before release.
• Test each stage of the ML pipeline, from OCR and Named Entity Recognition (NER) to rule based processing, LLM analysis, and final classification.
• Document results, reproducible failures, and edge cases in clear, traceable reports for ML engineers.

Requirements & Tech Stack

• 0 to 2 years of experience in ML QA, data annotation, data quality, software testing, or NLP. Freshers with relevant academic projects are welcome.
• A Bachelor's or Master's degree in Computer Science, AI, Data Science, or a related field.
• A basic understanding of machine learning and classification.
• Comfort with evaluation metrics: precision, recall, F1 score, TP, FP, FN, and confusion matrices.
• Working knowledge of Python and SQL for inspecting and analyzing data.
• Strong attention to detail, analytical thinking, and the ability to explain findings clearly.

Nice to Have

• Exposure to NLP, NER, LLMs, or document classification.
• Experience with OCR, PII detection, or sensitive data classification.
• Experience building evaluation datasets or using annotation tools.
• Exposure to regression testing, model comparison, confidence scores, or threshold tuning.
• Familiarity with pandas, Git, and basic testing practices.