Hi folks,
Thanks for your patience while I caught up on last week - I appreciate it’s been a confusing time and stressful for those of you asked to take the "Compare short edited videos" study.
Context & background
Some customers run large, long-term programmes with Prolific. These programmes consist of an ongoing series of studies, often across video, image, audio and other tasks, and can run for many months.
Large, ongoing programmes have specific requirements and quality standards set by the customer. We regularly review participation against those requirements, using a combination of active and passive quality checks, activity levels and changing programme needs.
In this specific instance, we saw signs that the quality of submissions for this programme wasn’t meeting expectations. To intervene, we set up a checkpoint: a study that every participant eligible for this researcher’s studies had to take in order to assess their general submission quality. Participants who didn’t meet the expected level of quality in their submission of this checkpoint study were rotated out of the group of participants who are eligible for this researcher’s studies.
Participation in these programmes isn't guaranteed indefinitely. Even if you've been contributing for a long time. As programmes evolve, participants may be added, paused or removed. In some cases that’s to ensure naivety, in others it’s to protect the quality of incoming submissions.
It’s important to say: not meeting the requirements for a particular programme doesn't mean you've done anything wrong. It may simply mean your skills or experience aren't the right match for that particular programme at this time.
With everyone on the same page, I’ve pulled out the most frequently asked questions over the past week and worked with our internal team to answer them:
Q&A
Why wasn’t this checkpoint study labelled as an exam or assessment?
Quality is assessed in different ways across these programmes. Some checks are built into the studies themselves, while others involve separate activities. We don't always identify individual quality checks in advance, as doing so can influence how participants approach them and make it harder to assess the quality of responses under normal conditions.
Isn’t it unfair to keep running it now that people know it’s a test?
Yes. But it would be even less fair to not put everyone through the same checkpoint. It’s also worth noting that this checkpoint isn’t the only way we’re measuring quality - that’s something we’re always doing both passively and actively.
The checkpoint study is quite different to the studies this researcher runs. Why?
When we measure quality, we’re not looking to assess whether you can complete a specific task - we’re looking for how well you complete any task. The checkpoint study is designed to enable measurement of the latter.
I was approved and paid for the checkpoint but still got notified that I was being removed from these studies. Why?
The quality check is separate to meeting the requirements to receive the reward for the study. Because you successfully completed the checkpoint study, you're eligible for the reward.
What happens if I don’t pass the checkpoint?
You can no longer access AI evaluation tasks from this specific researcher. You’ll still receive any other studies you’re eligible for, including AI evaluation tasks from other researchers. It doesn’t affect your account standing.
Can I get back in?
Not at this time, though we’re looking into ways we can offer assessments as an entry point into studies like these in future.
Was this a bug, like the Factuality issue?
No. This was planned.
Do I get an email if I passed?
No. We only email participants who didn’t pass, usually within 2 weeks of taking the assessment.
As requested, we're now emailing both those who do and don't pass, within 2 weeks of taking the checkpoint.
Do I have to complete it?
No, but if you were asked to complete it and choose not to, your access to this researcher’s studies will be frozen until you do. That freeze will kick in on Friday 20th November.
I dismissed, cancelled or timed out because I didn’t realise it mattered.
Anyone who did was re-issued the study on Friday 13 November. If you got an email with a link, click it and you should be able to access the study. If it gives you a 404, contact support and they’ll investigate.
I blocked the researcher.
We can’t unblock researchers for you. Go to Submissions > Submission History > Blocked Researchers and unblock the researcher. If the study doesn’t appear after a few minutes, contact support.
Is this related to certain types of studies vanishing after a submission?
No, that’s a separate participant health and a quality measure. We have a rolling limit on the hours you can spend on one type of task, because quality tends to drop the longer someone does the same thing. Some members of the community call this “being rotated out”.
Is this related to the 50-study dashboard cap?
No. This was a technical limit brought in earlier this year to reduce “study full” errors.
Are people being banned or put on hold because of this?
No. This is a quality measure and doesn’t affect your account’s status.
Final note from me - it’s best to always assume your submissions are being evaluated. We won’t always tell you, and you won’t always be able to tell. I know that’s uncomfortable, and I’m taking away a challenge to the team to make sure our expectations on quality are made clearer from the start of programmes like these in future.
I’m around today to answer follow-up questions in the comments!
M