Freelance Agent Evaluation Engineer

About the role

Mindrift is looking for a freelance Engineer to design and evaluate AI coding agents. You will build realistic development environments, craft challenging tasks, and create robust evaluation criteria to assess how well AI models handle real‑world developer work.

Key responsibilities

• Construct virtual companies with codebases, infrastructure, and development history.
• Design and calibrate tasks, write clear prompts, and define fair evaluation metrics.
• Set up isolated workstation simulations (Linux, CLI tools, repositories, task trackers, documentation).
• Write comprehensive tests that accept correct solutions and reject incorrect ones.
• Iterate with AI agents to verify test effectiveness and avoid false positives/negatives.
• Review agent‑generated code, analyse failures, and create edge‑case or adversarial scenarios.
• Incorporate feedback from expert QA reviewers to improve task quality.

Required profile

• Degree in Computer Science, Software Engineering or related field.
• 5+ years of software development experience.
• Strong proficiency in Python (FastAPI, pytest, async/await, subprocess, file operations).
• Full‑stack experience with React, JavaScript/TypeScript.
• Experience writing functional and integration tests.
• Familiarity with Docker, PostgreSQL, Kafka, Redis.
• Understanding of CI/CD pipelines, especially GitHub Actions.
• English proficiency at B2 level or higher.

Required skills

• Python
• FastAPI
• pytest
• async/await
• subprocess
• file operations
• React
• JavaScript
• TypeScript
• Docker
• PostgreSQL
• Kafka
• Redis
• GitHub Actions
• CI/CD
• Linux command line

What we offer

• Project‑based, flexible freelance engagements.
• Opportunity to work on cutting‑edge AI evaluation challenges.
• Collaboration with a team of AI and software experts.