Back to all jobs
Mercor

AI Evaluation Rubric Auditor

Pay

$60–85/hr

Location

Remote

Engagement

Contract · Remote

Posted

5 days ago

About the work

**The work** We are testing AI agents on realistic shopping and delivery tasks. Your role as a Project Auditor is to audit the work done by our Project Reviewers. You will audit complete tasks (prompt and grading rubric) and determine if their rubric failures are fair: \- Was the information available? \- Does the rubric accurately grade what the prompt asked for? \- Would a valid answer have been marked wrong? \- Is the rubric structure correct per the guidelines? This role includes providing feedback to reviewers regarding prompt and rubric issues. **Must-haves** \- Substantial experience writing or reviewing rubrics for LLM evals. Please list the projects in your application. \- Experience providing feedback to writers/reviewers \- Consistent availability \- Can read a long agent transcript and find where it went wrong. \- Precise writing, and you check the data instead of trusting the reference answer.

Good fit if you have

  • • Any-level experience in Writing & Editing, Evaluation & Rating, Safety & Moderation
  • • Fluency in English
  • • A reliable laptop and internet connection
  • • Comfort writing clear, structured explanations
Writing & EditingEvaluation & RatingSafety & ModerationEnglish

Applying is free. Shared links open this job page, and Apply now uses our referral link when available.

Help us measure outcomes

No name or application details are collected.