Freelance Agent Evaluation Engineer

We are looking for a Freelance Agent Evaluation Engineer to build and evaluate datasets for AI coding agents.

• Create realistic developer environments with codebase, infrastructure, tickets, documents and conversations.
• Design tasks from intermediate states, define prompts and success criteria, ensuring they are solvable by an AI agent.
• Write functional and integration tests that accept all valid solutions and reject incorrect ones.
• Iterate on tasks and tests based on QA feedback, reviewing agent solutions and refining evaluations.
• Collaborate remotely on a project‑based basis, estimating effort (≈20 hours per task) and delivering on schedule.
• Required experience: minimum 5 years in software development (Python, FastAPI, JavaScript/TypeScript, React, Docker, PostgreSQL, Kafka, Redis).
• Must be able to write tests, understand AI model failures and have English proficiency at least B2.

This role is not data labeling, not prompt engineering, and does not involve writing code from scratch; the AI agent writes most of the code under your guidance.

Compensation is up to 40 hours equivalent per task, with flexible scheduling.