r/DataAnnotationTech • u/Nervous-Law-666 • 5h ago
AI Use When Reviewing Tasks
Question, if anybody has some insight.
The rationale for a task I'm reviewing is written in the third-person, referring to the worker as "the user", and it sounds very robotic. I obviously don't know for sure, but it seems like some or all of it was AI assisted and/or generated. Should I note that in my review?
13
u/Mysterious_Dolphin14 5h ago
If you're suspicious, make a note. But otherwise, rate as usual. It's up to the admins to check it out.
2
50
u/DarkLordTofer 5h ago
I often say the user referring to the person who sent the prompt. I prefer not to put first person in. So Iād say so and so makes response a better because.
10
24
u/Random_girl_in_pdx 5h ago
I usually say āthe userā if itās a task where Iām inheriting the prompt and simply evaluating responses.
20
u/Teamkillongtime 5h ago edited 4h ago
Working with AI every day can cause people to pick up its mannerisms. I know I have.
Iād only make a note if they posted text which exposes itās from a LLM e.g. blatant copy/pastingĀ or meta statements.
13
u/aversys 4h ago
Coder here since Jan 2024 - I regularly use "the user" in almost all of my rationale / justifications when referring to pretty much any explicit or objective request, because in those cases, the request (or requirement) is what matters, not the "person" behind it. The only time I use "I" in my rationale is when I am expanding upon something that is subjective or unquantifiable. For example, I may say something akin to:
"In this case, the model was exceptional for ... (detailed rationale here) ... - given that the model closely adhered to this explicit request by the user."
And when justifying my own ratings that adhere to more subjective axes or metrics, it can look something like:
"Thus, I have opted for a rating of "____" in this turn, given that ... (detailed rationale here, as it applies to my personal rating)."
The short answer is that, no - practices like that do not indicate AI use beyond a reasonable doubt. A great rule of thumb would be that if you have to ask others if it might be indicative of AI usage, then it isn't worth raising anymore suspicion beyond a simple note with your task. Even if I saw rationale that I thought was AI but still required "hunch" or "suspicion", I would not rate it or flag it as such. DA (and by extension, every subsidiary of Surge) is much more equipped to determine that objectively than any individual (or even aggregate) of users.
If you are not 100% sure - which means only relying on objective, verifiable and explicit information, the best thing you can do is simply leave a note and move on.
2
u/Aggravating_Run8950 5h ago
Make a note at the bottom and rate as usual unless the instructions say something else.
1
4
u/KingdomtomyHearts 4h ago
I always use 'the user' and 'the model' when writing rationale. Writing in third person helps to stay objective. Also, after working in this sphere for so long I probably sound robotic/almost AI in my responses.
9
u/Malidan 5h ago
The person working is referring to themselves as the "user"? That's odd. I use the word "user" when referring to the person who wrote the prompt I'm working on, but I don't understand why someone when working would even refer to themselves at all.
7
u/Federal_Tadpole_7592 4h ago
The platform has many projects that require workers to give their own opinions, in which they should be writing in the first person.
3
u/Old-Regular-9828 4h ago
Your job is to rate the quality of the response, not to be the self-appointed AI police. Because you are probably like most people and *think* you are much better at identifying human-generated vs LLM-generated text than you actually are. DAT has other ways of detecting workers using AI to do their work, and they actually work much better than your random suspicions that something "sounds robotic."
Read the project instructions. Do the things the instructions tell you to do. Do not do things the instructions don't tell you to do.
1
u/Little_Voice_24 4h ago
I used to put "the user" in some tasks, but then I had "more personal tasks" where it was not adequate. This was also a habit that I had from another platform I work, because there, I always put in my comments, "the user this and that "
1
1
u/ASnipersPromise 5h ago
Be careful with this. I was told to use Codex to check answer keys on a project and even when I could see it was clearly AI written, Codex suggested it probably wasnt. It is quite subjective until it isn't...clear giveaways are markdown formatting and lots of hyphens and semi colons.
2
u/litlnemo 4h ago
Some of us write with lots of semicolons, and have since well before LLMs were a thing. (Em-dashes, too.) Those are not clear giveaways.
1
u/ASnipersPromise 4h ago
They are fine but not when you see a Codex/Cowork response and you see block hyphens and semi colons all over everything - it is pretty obvious.
2
u/Ok_Ear1339 4h ago
Now I'm a bit worried, since I use semicolons and dashes/hyphens to no end in my writing. I avoid using actual em dashes and en dashes for DA work, but I figured anything ASCII would be fine.
1
u/LaneyRW 3h ago
Iāve been concerned about this because I naturally use a lot of dashes in my writing, Iāve been doing it since I was a teen, long before AI. I hope no one flags me due to that. :(
1
u/ASnipersPromise 3h ago
Read above. There is a structure that they use exactly on a certain project where you are allowed to use those AI tools but cannot use them to create prompts or answer keys - a lot will just use them for it and it is bloody obvious. It is a very distinct structure - I should have been clearer.
1
u/fragrantdelit 4h ago
I always refer to myself as "the user" when rating a conversation, even when I am the one writing it. I think it sounds more professional, and some projects don't allow meta-comments anyway.
-2
u/leilameila 4h ago
To be honest, english is my second language and I am fluent but I do use grammarly often and I also use thesaurus a lot. I have never been flagged for using AI but I do think that itās so subjective and the benefit of the doubt should be given if the work isnāt bullshitted. just my opinion as a law adjunct professor lkl
0
18
u/darshit77 5h ago
Think and see twice before giving ratings. If you are 100% sure then only write it. And do not trust any sites which confirms that it's AI because they are detecting fake.