Freelance Agent Evaluation Engineer

About the role

Mindrift is looking for a freelance Engineer to design and evaluate datasets that test AI coding agents on realistic developer tasks. You will build simulated development environments, craft challenging prompts, and create robust evaluation criteria that distinguish strong from weak AI solutions.

Key responsibilities

• Construct virtual company environments with codebases, infrastructure, tickets, documentation, and conversation histories.
• Design tasks from intermediate states of these environments, define clear success criteria, and ensure they are solvable by an AI agent.
• Write functional and integration tests that accept all correct solutions while rejecting incorrect ones.
• Iterate on tasks and tests based on QA feedback, reviewing agent outputs and refining evaluations for fairness and robustness.

Required profile

• Minimum 5 years of professional software development experience.
• Strong expertise in Python (FastAPI) and JavaScript/TypeScript (React).
• Hands‑on experience with Docker, PostgreSQL, Kafka, and Redis.
• Proven ability to write comprehensive functional and integration tests.
• English proficiency at B2 level or higher.

Required skills

• Python
• FastAPI
• JavaScript
• TypeScript
• React
• Docker
• PostgreSQL
• Kafka
• Redis
• Test writing (functional, integration)

What we offer

• Freelance, project‑based contract with flexible scheduling.
• Compensation up to $40 USD per hour, based on experience and pace.
• Opportunity to shape cutting‑edge AI evaluation datasets for leading tech companies.