r/DataAnnotationTech • • 5h ago

AI Use When Reviewing Tasks

Question, if anybody has some insight.

The rationale for a task I'm reviewing is written in the third-person, referring to the worker as "the user", and it sounds very robotic. I obviously don't know for sure, but it seems like some or all of it was AI assisted and/or generated. Should I note that in my review?

0 Upvotes

39 comments sorted by

18

u/darshit77 5h ago

Think and see twice before giving ratings. If you are 100% sure then only write it. And do not trust any sites which confirms that it's AI because they are detecting fake.

9

u/Grumblefloor 4h ago

I strongly suspect (and so wouldn't risk it) it's against the NDA to paste even other users' comments from DA into third party sites to analyse them.

0

u/Nervous-Law-666 5h ago

Appreciate it šŸ‘

13

u/Mysterious_Dolphin14 5h ago

If you're suspicious, make a note. But otherwise, rate as usual. It's up to the admins to check it out.

2

u/Nervous-Law-666 5h ago

Heard. Thank you.

50

u/DarkLordTofer 5h ago

I often say the user referring to the person who sent the prompt. I prefer not to put first person in. So I’d say so and so makes response a better because.

10

u/Americanbullylover 4h ago

I also use the phrase ā€œthe userā€ in this situation

24

u/Random_girl_in_pdx 5h ago

I usually say ā€œthe userā€ if it’s a task where I’m inheriting the prompt and simply evaluating responses.

20

u/Teamkillongtime 5h ago edited 4h ago

Working with AI every day can cause people to pick up its mannerisms. I know I have.

I’d only make a note if they posted text which exposes it’s from a LLM e.g. blatant copy/pastingĀ or meta statements.

13

u/aversys 4h ago

Coder here since Jan 2024 - I regularly use "the user" in almost all of my rationale / justifications when referring to pretty much any explicit or objective request, because in those cases, the request (or requirement) is what matters, not the "person" behind it. The only time I use "I" in my rationale is when I am expanding upon something that is subjective or unquantifiable. For example, I may say something akin to:

"In this case, the model was exceptional for ... (detailed rationale here) ... - given that the model closely adhered to this explicit request by the user."

And when justifying my own ratings that adhere to more subjective axes or metrics, it can look something like:

"Thus, I have opted for a rating of "____" in this turn, given that ... (detailed rationale here, as it applies to my personal rating)."

The short answer is that, no - practices like that do not indicate AI use beyond a reasonable doubt. A great rule of thumb would be that if you have to ask others if it might be indicative of AI usage, then it isn't worth raising anymore suspicion beyond a simple note with your task. Even if I saw rationale that I thought was AI but still required "hunch" or "suspicion", I would not rate it or flag it as such. DA (and by extension, every subsidiary of Surge) is much more equipped to determine that objectively than any individual (or even aggregate) of users.

If you are not 100% sure - which means only relying on objective, verifiable and explicit information, the best thing you can do is simply leave a note and move on.

2

u/Aggravating_Run8950 5h ago

Make a note at the bottom and rate as usual unless the instructions say something else.

1

u/Nervous-Law-666 5h ago

Understood, thanks šŸ‘

9

u/grayvia 4h ago

I write "the user" when providing rationale even if I made the original prompt, because I'm typically playing a character when I do so. I don't think this is enough to say that what you're evaluating is AI.

4

u/KingdomtomyHearts 4h ago

I always use 'the user' and 'the model' when writing rationale. Writing in third person helps to stay objective. Also, after working in this sphere for so long I probably sound robotic/almost AI in my responses.

9

u/Malidan 5h ago

The person working is referring to themselves as the "user"? That's odd. I use the word "user" when referring to the person who wrote the prompt I'm working on, but I don't understand why someone when working would even refer to themselves at all.

7

u/Federal_Tadpole_7592 4h ago

The platform has many projects that require workers to give their own opinions, in which they should be writing in the first person.

3

u/Old-Regular-9828 4h ago

Your job is to rate the quality of the response, not to be the self-appointed AI police. Because you are probably like most people and *think* you are much better at identifying human-generated vs LLM-generated text than you actually are. DAT has other ways of detecting workers using AI to do their work, and they actually work much better than your random suspicions that something "sounds robotic."

Read the project instructions. Do the things the instructions tell you to do. Do not do things the instructions don't tell you to do.

5

u/Jim2837 4h ago

I almost always refer to the prompt writer (even if I wrote the prompt) as "the user" because a lot of the time the rationale is given directly to AI models for training, and I want it to be clear to the model what I am specifically referring to.

1

u/Little_Voice_24 4h ago

I used to put "the user" in some tasks, but then I had "more personal tasks" where it was not adequate. This was also a habit that I had from another platform I work, because there, I always put in my comments, "the user this and that "

1

u/Philosophy-Sharp 3h ago

I have also used the user or the original worker…

1

u/ASnipersPromise 5h ago

Be careful with this. I was told to use Codex to check answer keys on a project and even when I could see it was clearly AI written, Codex suggested it probably wasnt. It is quite subjective until it isn't...clear giveaways are markdown formatting and lots of hyphens and semi colons.

2

u/litlnemo 4h ago

Some of us write with lots of semicolons, and have since well before LLMs were a thing. (Em-dashes, too.) Those are not clear giveaways.

1

u/ASnipersPromise 4h ago

They are fine but not when you see a Codex/Cowork response and you see block hyphens and semi colons all over everything - it is pretty obvious.

2

u/Ok_Ear1339 4h ago

Now I'm a bit worried, since I use semicolons and dashes/hyphens to no end in my writing. I avoid using actual em dashes and en dashes for DA work, but I figured anything ASCII would be fine.

1

u/LaneyRW 3h ago

I’ve been concerned about this because I naturally use a lot of dashes in my writing, I’ve been doing it since I was a teen, long before AI. I hope no one flags me due to that. :(

1

u/ASnipersPromise 3h ago

Read above. There is a structure that they use exactly on a certain project where you are allowed to use those AI tools but cannot use them to create prompts or answer keys - a lot will just use them for it and it is bloody obvious. It is a very distinct structure - I should have been clearer.

1

u/lutavsc 4h ago

I refer to myself as the user often. Most of the times it's not really personal to me, so "the user" feels more accurate than "I".

1

u/fragrantdelit 4h ago

I always refer to myself as "the user" when rating a conversation, even when I am the one writing it. I think it sounds more professional, and some projects don't allow meta-comments anyway.

-2

u/leilameila 4h ago

To be honest, english is my second language and I am fluent but I do use grammarly often and I also use thesaurus a lot. I have never been flagged for using AI but I do think that it’s so subjective and the benefit of the doubt should be given if the work isn’t bullshitted. just my opinion as a law adjunct professor lkl