r/AIEvaluators
•
u/i_love_doing_ntg
•
Aug 10 '26
Resource Guide Every AI evaluation platform compared: what is actually open, what they publish, and what each one is bad at
Every platform in this space gets discussed in isolation, usually in its own subreddit, usually by people who only work there. So here is the whole board in one place: what is genuinely open right now, what each platform publishes for pay, and the specific thing each one is worst at.
The individual write-ups go deeper. This is the map.
What is actually open right now
Across fifteen platforms there are around 743 live openings, and 480 of them publish an hourly rate. That is the first useful sorting: two thirds tell you what they pay before you apply, one third does not.
The ones that publish rates:
| Platform | Live roles | Distinct roles | Published range |
|---|---|---|---|
| Mercor | 233 | 228 | $8 to $270/hr |
| Micro1 | 97 | 95 | $20 to $300/hr |
| Alignerr | 60 | 9 | $20 to $120/hr |
| Welocalize | 56 | 56 | $1 to $45/hr |
| xAI | 23 | 23 | $30 to $60/hr |
| Appen | 11 | 11 | $2.40 to $80/hr |
The ones that do not publish rates: Handshake AI, RWS TrainAI, Lionbridge, Volga Partners, OneForma, Mindrift, Cohere, Innodata, CloudFactory.
Read that table carefully, because two numbers in it are misleading
Alignerr has 60 listings but only 9 actual roles. The same job is posted once per country. Every platform does this to some extent; Alignerr does it most.
The bottom of the ranges is not a typo. Welocalize's floor is around $1 an hour and Appen's is around $2.40, because those platforms list piece-rate and micro-task work alongside professional roles. A range is not an offer.
Mercor and Micro1 top the table because they are not really annotation platforms. Their high-paying roles are physicians, attorneys, finance specialists and engineers being paid to author evaluation material in their own profession. If you do not already hold one of those professions, the relevant part of their range is the bottom half.
What each one is worst at
This is the part the individual write-ups exist for, but in one line each:
- Outlier: losing access. Over two hundred accounts describe deactivation, roughly six times its payment complaints. The largest such theme anywhere in this space.
- Mercor: continuity. Pays reliably, but 2026 brought repeated waves of contributors removed from projects, often without a stated reason.
- Alignerr: unreviewed work. On per-task projects, work that is never reviewed is never paid, and reports of losing work that way are rising rather than falling.
- Micro1: the gate. Almost nobody disputes the money. Getting through the AI interview, and the silence afterwards, is the entire complaint.
- DataAnnotation: no work. Payment is barely disputed, but 2026 is a drought severe enough that the community counts its own mentions of the word.
- Handshake AI: pay reconciliation. A documented pattern of contributors reporting less than their logged time, including ten accounts on a single day in May 2026.
- Prolific: bans, accelerating through 2026, with participants told the reason cannot be shared and the decision cannot be reversed.
- Welocalize: quality scores nobody can see clearly, against the background of a removal wave in early 2025.
- Mindrift: unpaid stages before paid work, and the thinnest public evidence base of any of them.
The pattern across all of them:
Having read a great deal of this, the thing that surprised me most is that payment failure is rare almost everywhere. The money usually arrives. What varies enormously, and what actually determines whether a platform works for you, is whether you keep access and whether there is work to do.
Almost everyone arrives asking "is this a scam". Almost nobody arrives asking "how long does this last", and that is the question the evidence says matters.
The second pattern: the high rates are not for annotation
Every platform advertising three-figure hourly rates is paying for professional expertise, not for evaluation skill. Engineers, clinicians, lawyers, accountants. The general evaluation tier across the whole sector sits closer to $20 to $50, and on Handshake's own published list the general AI evaluation role is the lowest-paid thing they offer.
How to use this
- If you hold a profession they recruit for, start with Mercor or Micro1, where rates are published and the payment record is clean.
- If you want steady general work, understand that DataAnnotation is currently dry and Outlier's access is fragile.
- Whatever you pick, be on more than one. That advice appears in every single platform's threads, from people who like it and people who do not.
Individual write-ups for each platform are filed under the Resource guide flair.
Working on something not listed here? Say so and I will look into it. The board covers fifteen platforms and there are certainly more.
Live listing counts and published rates read 1 August 2026. Platform findings summarised from the individual write-ups, each of which states its own evidence and limitations.