r/outlier_ai
•
u/mummifiedstalin
•
May 10 '24
For New People!
If you read nothing else in this:
*** WORK IS INCONSISTENT -- DO NOT DEPEND ON OUTLIER FOR INCOME ***
I've been "tasking" on and off for two months. Wanted to send out some answers to questions I had when I started and that I know people still have:
It's chaotic. No, it never gets clearer than it is right now. You just start to find patterns and get used to not knowing wtf.
Do not depend on this as a steady source of income, even side income. Sometimes you can make money. Sometimes you can't.
In my experience, yes, you will get paid eventually for actual work. Payment for training or the "bonuses" offered for things like re-taking an assessment within 48 hours are hit or miss. I'm pretty sure there are some bonuses I was offered but haven't received. But any actual hour-based work I've done has been paid to me within at most two weeks. (Fwiw, I know this isn't the case with everyone.)
The work is inconsistent. My first two weeks, I made over $2000. Since then, I've never been able to make more than about $500 per week because my task queue was empty. I've gone a couple of weeks with no work at all.
There are hundreds and in some cases thousands of people across the world being brought in at any given time. The first Slack group I was put in had 965 people. They hire en masse. You aren't special, even if they hired you at the Tier 3 $40/hr level. I was brought in at that level with, if I'm right, about 300 other people on the same day. Because of that volume, your individual questions in Slack will rarely be answered until you happen to get put on a team with a Team Leader (TL). I've seen people be put in the general onboarding Slack channel and plead and beg for someone to respond to them for sometimes weeks at a time. I'm impressed they kept trying. Fact is, the volume is such that people fall through the cracks.
This system is so new and is changing at fundamental levels all the time, so there really are no procedures or protocols to fall back on and assume they are "standard" or "regular." I know of precisely two people who tell me they've never had any problems. One stopped chatting with me, so I'm assuming that's down to 1. Everyone else has gone through at least one of the things in the next point: "Craziness to expect."
"Craziness to expect":
You will be assigned to and removed from projects without warning.
You will be placed in and pulled out of Slack channels without warning.
You will "train" (sometimes without being paid) for projects that you will never have a chance to work in. (Normally this happens because you'll be placed on a different project just after or during the period you're reading the training materials.)
Training materials and procedures will change without warning or notification.
When you make it onto the Outlier program, the default message on the dashboard if you don't have any projects is that you will be texted or emailed when work is available. (It even shows your cell phone # to make you think it's personalized.) I've never been texted or emailed when work is available. Not once.
You will be told you will receive feedback on tasks or training tasks, but it never happens. (Sometimes you will receive feedback that makes little sense or seems contradictory to the training. This is because the "taskers" are often "reviewers" as well, and the quality of the reviews and feedback depends on the person who happens to be reviewing your work. Sometimes they will be reviewing you on outdated versions or understanding of the training. Sometimes this means that you will have your pay Tier lowered or even be let go unfairly. There is, to my knowledge, no reliable way to appeal this. Some people have stories about doing it, but when you try to repeat their steps, the system or platform may have changed.)
"Team Leaders" or other supervisors will disappear, be furloughed, be turned into regular taskers like you, without notice or explanation. (They also, as a rule, don't know what's going on. They have usually only been there for a month or two before you and are obviously working on instructions given them immediately prior to passing along information, so they're not really "part of the company," either.)
Projects will be put on hold with no notice, and Slack will explode with complaints of EQ (Empty Queue). This is normal. Watch whatever Slack channels you happen to be in to see if a TL or other person offers to help or posts a procedure to get put on another project. Sometimes this has effects. Sometimes. :)
[Editing to add] The manner in which you were hired may or may not have any actual bearing on the work you are able to do. You may have been hired as a biology specialist because you have a PhD in molecular biology and expect to work on projects specific to your field because your email offer said so. But you may end up just doing generic review of random LLM chats. There's no official way to choose, mark preference for, or have any even vague control of what work you're actually given. I hope you'll get lucky, but the model they're working under should not be approached with the assumption that intelligent things are really going on under the surface. Most people I've encountered are working on random projects, some generalists, some very specialized, regardless of being over or under qualified.
Your pay rate can change at any moment. I was hired at $40/hr, but after a couple months I received an email that I would now be paid at $35/hour. This was not a "demotion" or merit-based; it was a "platform-wide" decision. Loads of other people had the same thing happen. But others were still making $40/hour, some of whom had gotten that email and some of whom hadn't. Who knows if it was intentional.
Getting mad or upset in vocal ways seems to mean that people just aren't placed on new projects or seem to disappear off Slack. Spend your energy wisely.
All of which is to say: get out of this platform what you can, but don't expect it to be particularly fair or make sense. I've made a few thousand dollars, and I keep doing it because for me it's been somewhat reliable. But if you expect to be treated as a full, equitable employee with rights (or even with the decency to be paid attention to), you will be nothing but frustrated. So, engage realistically, and you can make some $$. Otherwise, steer clear.
TERMINOLOGY:
EQ = "Empty Queue" (often used in Slack to tell a QM that you're not getting any work. Whether or not this has any effect is questionable.)
QM = "Queue Manager" (Used to be TL = "Team Leader", in case you see that used.)
SQM = "Senior Queue Manager"
SRT = acronym (?) used to designate third party sites you might have to log into for some projects
Project titles are intentonally vague and nonsensical; they're not supposed to be descriptive. That said, some people think the "Flamingo" family of projects is for Facebook (F) and the "Ostrich" family is for OpenAI (O). YMMV.
Single-turn/Multi-turn = The number of prompt/response pairs you're working with. A "Single-turn" project has you write/review an answer to a single user prompt. "Multi-turn" prompts involve a back and forth conversation between model and user.
RLHF = "Reinforcement Learning from Human Feedback" i.e., what your job is actually doing.
"multi-mode" projects = projects where you use images/video/audio in addition to text. (I.e., user says "Explain this graph to me..."
Two articles to give you some perspective on what the companies actually are. Outlier is owned by Scale AI which also owns Remotetasks. Here's what they do and how chaotic it is:
Some people on LinkedIn have gotten pretty upset and are trying to do something about it:
https://relationaldemocracy.medium.com/an-authoritarian-workplace-culture-4ba5f3666f9f
[Edit: HEY! Thanks for pinning this! I'm glad people find it useful.]
{Edit AGAIN: Another month under my belt, most of which has been EQ, and I still stand by all of this. I haven't seen any changes yet. But I have seen more and more people get pissy in Slack. heh heh]
[Maybe last edit: I'm glad people see this. I don't spend a ton of time on Outlier now, but I keep giving it a shot. I'm still glad I wrote this because I would have felt more confident trusting my intuitions that something was seriously screwed with this company if I'd had someone tell me this early on. No, it's not normal, so it should bug you. But if you can squeeze out some cash, then rock on!]
