I'd guess uncertain predictions are the best place to start, but I'm curious whether rare cases give you more value in practice.

Uncertainty sampling is a solid default, but I mix it: most of the batch from uncertain predictions, plus a slice of rare classes and a few random samples so you don't drift into only labeling odd borderline cases. Clustering the uncertain ones and picking from each cluster also helps you avoid labeling 50 near-duplicates.

Mixing in a few random samples is smart, I hadn't thought about drifting into only labeling borderline cases. Clustering the uncertain ones to avoid near-duplicates makes a lot of sense too.

I like balancing uncertainty with rare and random examples so the batch doesn't get skewed toward borderline cases. Clustering first also seems like a sensible way to avoid spending labels on near-duplicates.

One thing that helped me: keep a small fixed test set that you never sample from, and check it after every labeling round. It tells you quickly whether the new batch actually moved the model or just added more of the same.

A small untouched test set gives you a useful check that new labels improve generalization, not just the sampled slice. I’d keep it representative and revisit it only when the data distribution changes.