Senior AI QA Automation Engineer

About the role

Coforge is hiring a Senior AI QA Automation Engineer to build automated evaluation frameworks for AI agents, RAG solutions, prompts, tools, workflows and model outputs. The position is remote and open to candidates in Costa Rica, Peru, Colombia and Bolivia.

Key responsibilities

• Build automated evaluation frameworks for AI agents, RAG solutions, prompts, tools, workflows and model outputs, including benchmark datasets, golden responses, regression suites, evaluation metrics and release thresholds.
• Test AI-specific behaviors such as hallucinations, grounding, citation accuracy, prompt injection, tool selection and multi-agent coordination.
• Develop agent-powered QA solutions for test-case generation, API and workflow validation and intelligent UAT, while building UI, API, integration, performance and end-to-end automation.
• Implement CI/CD quality gates and continuously monitor production AI quality, drift, reliability and recurring failure patterns.

Required profile

• 5+ years of experience in software quality engineering, test automation or reliability engineering.
• Bachelor's degree in Computer Science, Software Engineering, Information Systems or a related field.
• Ability to turn business requirements, workflows and AI behaviors into measurable quality criteria.

Required skills

• Strong Python skills with reusable automated testing frameworks.
• Playwright and/or Selenium, and strong API, UI, integration, regression, end-to-end and CI/CD test automation.
• Azure AI Foundry Evaluations, RAG evaluation and prompt testing.
• CI/CD quality gates and production monitoring.
• Preferred: testing generative AI, RAG, copilots and AI agents, LLM evaluation frameworks, adversarial and responsible AI testing.