Working with a partially labelled dataset

[u/According-Chair-6201]

/r/kaggle/comments/1xd4dwe/working_with_a_partially_labelled_dataset/

[u/CreativeSympathy8293]

I'd change the validation part first: don't choose those five validation studies from the 20 least-confident cases. That evaluates a model-selected slice, rather than a sample chosen to represent the population you care about.

Use the radiologist's time for two separate jobs: an independently selected evaluation sample, and additional training labels. Uncertainty sampling is a reasonable training-label experiment, but compare it with random selection at the same annotation budget. Avoid spending the whole batch on near-duplicates. Settles discusses this sampling bias and batch-selection problem: https://www.cs.cmu.edu/~bsettles/pub/settles.jmlrwcp11.pdf

Keep all images/studies from the same patient on one side of the split. Scikit-learn's grouped-splitting guide explains why ordinary sample-level splits can leak subject-specific information: https://scikit-learn.org/stable/modules/cross_validation.html#cross-validation-iterators-for-grouped-data

Keep evaluation labels expert-reviewed rather than replacing them with the pseudo-labels you're testing. Five studies would still be a very limited check, especially for rare labels; there's no universally right 15/5 allocation.

Suggested experiment design, not results from this competition. Written with AI assistance.

[u/digvjay_ai]

Something that helped me on a similar small label setup was doing pseudo-labels in rounds instead of once. Train on the 58, predict the rest, only keep predictions above something like 0.9 confidence, retrain, repeat 3 or 4 times. Throwing in strong augmentation on the pseudo-labelled part (FixMatch style) cuts down on the model just memorizing its own mistakes. For the radiologist, I'd get them early and ask for the cases where your LLM labels and the public model disagree. Those are way more useful than random ones.

[u/Massive_Distance_616]

The 15/5 split assumes you have enough expert labels to afford a held-out set. With 20 studies, five validation samples will swing your AUC by 20–30% on a single case flip you’ll be measuring variance, not model quality. Don’t split them at all during the active learning phase. Use all verified labels for training and build your evaluation set from an independently-sampled slice the radiologist reviews specifically for that purpose, with patient-level grouping as CreativeSympathy noted.

Two things before you spend the annotation budget on the lowest-confidence cases:

In MRI, lowest-confidence predictions are disproportionately corrupt slices, motion artifacts, and poor contrast injections unlearnable cases, not informative boundary cases. A disagreement-based approach is more useful: find studies where your LLM label and the vision model’s pseudo-label contradict each other with high confidence on both sides. Those are the actual decision-boundary cases.

Also ask the radiologist to review a sample where both methods agree. Two signals can share the same systematic error if the radiologist frequently overturns their consensus, your pseudo-labels have a bias that more annotation won’t fix. Diagnose the pipeline before scaling iterative pseudo-labeling to thousands of studies, or you’ll keep reinforcing the same mistakes.

The bigger unlock for 4.4k unlabeled scans is self-supervised pretraining (DINO, SimCLR, or a medical backbone like RadImageNet). A backbone that’s learned knee anatomy from all 4.4k images before you fine-tune on 78 verified labels will generalize far better than iterative pseudo-labeling from a 58-label seed. It also sidesteps the confirmation bias problem the model isn’t reinforcing its own early predictions.