Senior QA & Evaluation Engineer

About the role

Coforge is hiring a Senior QA & Evaluation Engineer to design and maintain automated test suites, integrate testing into CI/CD pipelines, and build evaluation frameworks for AI capabilities.

Key responsibilities

• Design and maintain automated test suites covering unit, integration, API, and end‑to‑end testing.
• Integrate automated testing into CI/CD pipelines and establish automated suites as release quality gates.
• Build evaluation frameworks for AI capabilities, including scored rubrics, LLM‑as‑a‑judge approaches, and normalized pass thresholds.
• Build and maintain held‑out evaluation datasets, measure AI behavior against ground truth, and report defect escape and evaluation results as part of delivery metrics.

Required profile

• 4+ years of experience in Quality Engineering with a strong focus on test automation.
• Hands‑on experience with automated testing frameworks, preferably within the .NET ecosystem.
• Strong experience with API testing and automated test development.
• Experience integrating automated test suites into CI/CD pipelines.
• Understanding of testing strategies for non‑deterministic and probabilistic AI systems.
• Ability to design evaluation approaches based on scored thresholds rather than exact‑match assertions.
• Experience working with test data, ground truth datasets, and automated validation.
• Strong analytical and problem‑solving skills.
• Strong working proficiency in English.
• Bachelor's degree in Computer Science, Software Engineering, Information Technology, or related field, or equivalent professional experience.

Required skills

• .NET
• API testing
• CI/CD
• Test automation frameworks
• LLM evaluation
• Azure AI Foundry Evaluators
• promptfoo
• Ragas
• DeepEval