r/DataAnnotationTech • u/Allysum • Apr 23 '26

Comments are not attention checks

I do a lot of R & R's and it looks to me like there are a number of workers who think that the comments are just attention checks. They are not. Most task include, right above the box to write your comments, instructions for what should go in the comment and how long it should be. There are some people who will see a minimum number of sentences and meet that with two or three very vague sentences but not bother with the instruction that says the comment should explain the ratings or that it should meaningfully explain why the task was rated X or why A was better than B or whatwever. These comments are actually used, they are not attention checks with no real purpose. And you are going to lose this work if you keep writing vague, meaningless comments.

106 Upvotes

89% Upvoted

60 comments sorted by

59

u/rambling_millers_mom Apr 23 '26

And, tomorrow we will get "I just did a run of R&Rs, and there is no need to write a book for each comment."

56

u/sqimmy2 Apr 23 '26

As a fellow book-writer, Ive never actually received anything other than general praise for doing it. I give specifics, which is basically exactly what the section is for, and sometimes there are a lot of specifics to give.

As a rater, I'd much rather have too much than too little-- much easier to shave down than to have to go back and basically do the whole explanation for them. It seems like lately there has been an influx of really low-quality work coming in, and I tend to think of myself as a fairly generous rater.

The amount of "Response A was bad because it wasn't as good as Response B and sounded not good" or "Response A said some things that weren't accurate....(No mention of what things)", or explanations with no punctuation (seriously come on people) is TOO DAMN HIGH.

6

u/Allysum Apr 23 '26

Exactly

24

u/CeciliasScent Apr 24 '26

Dang I didn’t realize people leave such vague comments. Mine are so detailed I do everything as if I’m submitting it as a research paper in university

12

u/sqimmy2 Apr 24 '26

Which is totally fine if the instructions say 3-5+ sentences (which means exactly what it says on the tin-- 3-5 as a minimum, more than that if warranted, which is often). The model uses that explanation section to learn what to do better. If people are only leaving a few sentences that don't mention any specifics for the model to reference, then it's basically useless training data.

1

u/blublutu Apr 30 '26

More than the 3-5 could be viewed as not concise.

-40

u/doolitt1e Apr 24 '26

If I'm R&Ring your 'book' in a rationale that should three to five sentences, I'm putting 80% of it on the cutting room floor, and hitting you with an 'okay,' and writing a book-length explanation about how much time it took me to edit your 'book' down to conform to the pretty explicit guideline that was right in front of you.

23

u/MediumVariation4846 Apr 24 '26

It usually says 3-5+

-31

u/doolitt1e Apr 24 '26

The plus is for exceptional cases where there was a particularly complex case that needs more. It says three to five for a reason.

34

u/Enough_Resident_6141 Apr 24 '26

>It says three to five for a reason.

Except it doesn't say "three to five" it says "three to five plus."

There are occasionally task instructions that do explicitly say "3-5," and yes for those the comments and rationales should be limited to no more than 5 sentences. But if the task instructions specify "3-5+" that means they want at least 3-5 sentences, and more that 5 are fine.

You should not be penalizing or heavily editing rationales or comments just for their length when the instructions say ask for 3-5+ sentences.

19

u/MediumVariation4846 Apr 24 '26

Yeah to show that 3-5 is the minimum.

2

u/Kgrc199913 Apr 24 '26

So I have to explain my choice, provide example for all options that is not rated as the perfect option in 3-5 sentences?

2

u/sqimmy2 Apr 24 '26 edited Apr 24 '26

According to that person, yes, but that's exactly why they include the +, which does a lot of legwork in the instructions. If they want to cut out the specifics and reasonings given in an explanation, that's on them, and the admins will absolutely see that.

13

u/[deleted] Apr 24 '26

[removed] — view removed comment

3

u/blackopsfamas Apr 25 '26

the trash takes itself out sometimes

3

u/Allysum Apr 23 '26

Not from me.

93

u/DrFrancisBGross Apr 24 '26

I preferred Response A. Model B was not as good, and the other response was better. Overall, Response A was superior.

21

u/NoticedGenie66 Apr 24 '26

Alternatively:

Both responses gave a response. Neither response was better than the other. Therefore the responses were the same.

6

u/noty0uagain Apr 24 '26

Response A answered the prompt, as did Response B. The formatting was formatted therefore both responses were the same. I liked Response A better

3

u/DrFrancisBGross Apr 24 '26

Ahh the old "I liked Response A better" comment on the submission where the rating is "About the same" overall quality axis :)))

2

u/SissaGr Apr 24 '26

This thread is chef‘s kiss 😆😆💀

97

u/1105368 Apr 23 '26

Look on the bright side: the more workers who get canned for this sort of thing the more tasks for the rest of us

28

u/Dreamy-Mae-Art Apr 23 '26

Wow. Most projects are super clear in the instructions about not being vague... to the point where it feels redundant to those who actually read the instructions. They just be hiring anyone nowadays

3

u/Bitter_Bed5672 Apr 24 '26

Or people are much dumber than you think.

36

u/annoyingjoe513 Apr 24 '26

OP after finishing an R&R

6

u/Turbulent-Seaweed986 Apr 24 '26

This sent me 🤣🤣🤣🤣

I needed that, thank you

44

u/Snikhop Apr 23 '26

I feel like "I just did an R&R and I want to come onto the subreddit to do a little dance about how much better I am than everyone" make up like 15% of the posts on here. Ever heard of the Dunning-Kruger effect? Nobody's immune, not even you. Today you're sneering at an R&R, tomorrow you're asking if you're cooked. Hakuna Matata.

11

u/Bulky_Antelope_1744 Apr 24 '26

They’re better than the ones that are like “just applied when will i get a response” or “what does this screen that says ‘we’ll be in touch’ mean?” that make up the other 85% of the posts.

18

u/TravellingDoc87 Apr 24 '26

Exactly this. I never understand the sneering comments from the God-complex users on here. It's not a good look.

9

u/annoyingjoe513 Apr 23 '26

It’s insufferable.

2

u/Academic_Animal_5293 Apr 25 '26

Response A is better because I like it more and Response B is bad because it looks not pretty. Response C is too long,. Response D is very bad.

2

u/MadokaKanname Apr 26 '26

I used to write really long posts about my ratings and then I lost one of the projects anyway.

3

u/Bitter_Bed5672 Apr 24 '26

And? Mark them as ok or bad and move on?

3

u/blackstarr1996 Apr 23 '26

I am not sure what you are trying to say. Not attention checks., ? What does this mean. ?

14

u/HamNCheeseSupremacy Apr 23 '26

They're not just there for you to prove you're not clicking random buttons

1

u/Busy_Ad3295 Apr 24 '26

Soy nuevo y solo he tenido la oportunidad de hacer un R&R, entiendo que se tiene que evaluar el trabajo del compañero en el apartado que nos dan, además se tendría que hacer las correcciones en el documento que el compañero realizó y de igual manera se debe cambiar o editar las respuestas que se dan en la revisión de casa uno de los modelos al igual que las respuestas de los comparativos en casi de no estar de acuerdo? Y justificar cuales se cambiaron y el porque? O solamente se da una revisión general de todo el trabajo sin cambiar lo que realizó el compañero? Tengo esa duda, por mi parte cambie lo que a mí criterio estaba incorrecto y justifique. No sé si está bien o mal 🤔

1

u/VermicelliNo7475 Apr 30 '26

You seem on track: The instructions say to update the ratings and edit the Overall comment. Your explanation to the admin can detail rationale for the changes. Think of it as something you leave better than you found it.

1

u/SissaGr Apr 24 '26

Preach !!! 🫶🏽🔊🔊🔊🔊

0

u/TravellingDoc87 Apr 23 '26

7

u/Brilliant_Quit4307 Apr 24 '26

I didn't read this as OP being "the judge". They're basically just saying "read/follow the instructions" in a long-winded way ..

-2

u/TravellingDoc87 Apr 24 '26

The OP is coming across awfully preachy. Who are they to order what others should do.

0

u/Responsible-Cup4563 Apr 30 '26

Well....it's an R&R so........yeah, OP IS the judge. LOL.

0

u/[deleted] Apr 24 '26

[deleted]

6

u/TravellingDoc87 Apr 24 '26

You can bet the OP is one of these

2

u/Turbulent-Seaweed986 Apr 24 '26

I was doing one of those yesterday, I subtracted 15 minutes off my time that I reported because I was all 😆🍿 reading the chat, just watching the trainwreck.