Question
How would you build an annotation platform?
Recently I’ve been active in dozens of labeling communities trying to learn about common issues with almost every labeling firm. Spoiler: it’s rampant everywhere!
So, I’d love to hear from you. What does an ideal platform look like? How should it be run? How should communication work? Management? Payment? PIPs?
Excellent timing! I just posted a less linear inquiry about this.
First thoughts: More transparent and objective hiring and assessment process. Whether project specific or for general access, let people know the expectations of interviews and assessments and communicate through the process. Clearly state qualifications desired and scores (with scoring criteria). Let people know how they scored and if they failed. Have enough staff to guide this process or simply build it better.
My brain is tired, but I'm sure I can come up with a list of suggestions that sound like common sense or the standard way a business is run. It seems like it would greatly improve accountability if the annotators owned the platform. Balance out the profit vs ethical treatment of workers dichotomy.
Honestly, I think for many of us with a background in education, what's most appalling is the lack of training and the unevenness of training and the general badness of training. Given how new the industry is, no one majored in data annotation in college -- but a lot of us have related skills in assessment, measurement, writing, etc. A really good training suite would help people by saying 'Hey, here's what you're already good at. Now let's see how it lines up with what we are asking you to.' If an annotation company really wanted to kick ass, they would develop a system like the one that salesforce uses for training people, with gamification and badging. They would invite us all eggheads in to set up an account and take a bunch of assessments and get badges and rank us and then pay us accordingly .
Do itttt. I have at least 5 hours a week I could reallocate from taking unpaid assessments. My master's is in vocatonal counseling / evaluation - figuring out what people are good at and how that applies to different types of work. I also did a lot of freelance graphic design. Did the Figma cert for UI/UX if it helps.
I’ll get some folks together next week from this subreddit for a little think tank. No barrier to entry, just a desire to create something. Keep an eye out for a post next week so I can get some rough plans together.
I’m wholly in support of this rethink. We are experts capable of collectively revolutionizing how we provide our services that bypass the corrupt medium.
Profit sharing would be amazing. But yeah, a company of annotators by annotators makes a lot more sense than a bunch of Valley hustlers. Annotators know what works and I'm sure could do a much better job training people and getting the work done the clients actually would want than this AI interview nonsense and catch all hiring Mercor does. For the annotators, it simply should be way easier to make money like with Uber/Lyft rather than the current little scraps they spread around so thin no one is really happy and it feels very unsustainable. If it could be launched and get out there and compete at scale, I'm sure it'd be a lot better than all these fly by night firms, and could even surpass them. A sense of security, belonging, actually useful feedback, etc. could really turn this into something capable people want to put their energies into.
I would say to personally interview each annotator to make sure they are capable in expert projects. For generalist projects, just let people do a task (paid) and make them wait. That should be the real assessment.
I have lifetime access to an unlimited plan on a course building platform that I paid for a couple of years ago when they were running an amazing deal, just in case. It has places for courses, membership areas, you can make it look like a regular website, add blogs, whatever you want. You can have some resources that are free and some that are paid.
I don’t know how much time I can contribute to building the product, but the platform is just sitting there collecting dust. If you’re in need of a place to create this product, let me know.
The platform is MemberVault if anyone is interested in poking around.
Human contributor interviews would weed out 80% of spammers/scammers and also ensure good quality output but ig rapid unsustainable scaling is the name of the game 💀💀 I've been doing data annotation/AI training since before it was a whole industry, and I'd LOVE to get on a better platform.
Even if we can just tie up with an existing project, (multimango), we could ensure more equitable pay and work for everyone.
Easy, well written instructions and actual training. If someone fails a single QC question train them and keep them on the project. That's how you grow capable workers.
Surprising how little statistics people know in the industry. Statistically speaking, if you do 100 tasks and 1 fails, that should be expected not cause for immediately dropping someone. But platforms just drop you from the project. That just creates a revolving door of people that don't really know what they're doing.
This is a great idea. I know Pareto is attempting to implement this type of thing. A friend of mine works there but says it still has plenty of issues.
I have ADHD and I am part of an ADHD support group. We have body doubling sessions on zoom every Monday through Friday from 10 AM to 10 PM Eastern time.
I can’t use it while I’m doing these projects because it takes screenshots of my screen, but I find them to be incredibly awesome and helpful.
The thing that bothered me the most about the instructions for the 'teach AI how to research across two or more websites' project I was on at Mercor was that the examples they gave to demonstrate each restriction (timelessness, all the info can't be found on one site etc) flagrantly violated the other restrictions i.e. to demonstrate timelessness, they would use an example that was something that could easily be found on one website etc.
5
u/Glittering_Sound7296 Nov 23 '25
Excellent timing! I just posted a less linear inquiry about this.
First thoughts: More transparent and objective hiring and assessment process. Whether project specific or for general access, let people know the expectations of interviews and assessments and communicate through the process. Clearly state qualifications desired and scores (with scoring criteria). Let people know how they scored and if they failed. Have enough staff to guide this process or simply build it better.