| Approach | What gets labelled | Typical label savings |
|---|---|---|
| Random sampling | Arbitrary examples from the unlabelled pool | Baseline (0 percent) |
| Uncertainty sampling | Examples the model is least confident about | 30-50 percent fewer labels |
| Query-by-committee | Examples where multiple models disagree | 35-55 percent fewer labels |
| Diversity plus uncertainty (hybrid) | Uncertain examples spread across the input space | 40-70 percent fewer labels |
| LLM-assisted pre-labeling with active review | Only the LLM’s lowest-confidence or highest-disagreement predictions | 50-80 percent fewer human touches |
Every machine learning team eventually runs into the same wall: models keep improving with more labelled data, but labelling is slow, expensive, and often requires subject-matter experts who would rather be doing something else. Active learning is the answer teams reach for when they cannot simply buy more annotation hours. Rather than treating every unlabelled example as equally worth a human’s time, active learning asks a simple question before each labelling round: which examples, if labelled next, will teach the model the most?
In 2026, this question has become more urgent, not less. Datasets keep growing, but so does the cost of expert review in regulated domains like healthcare, finance, and legal tech, where a wrong label can be worse than no label. At the same time, large language models have become good enough to serve as first-pass labellers themselves, changing what “active” learning looks like in practice. This piece walks through the core sampling strategies, backs them with real numbers from recent research, and explains how LLM-assisted pre-labeling has reshaped the pipeline without replacing the underlying logic of active learning.
Why Random Labelling Wastes Budget
When a team labels data at random, most of the labelling budget goes toward examples the model would have gotten right anyway. A sentiment classifier that already handles obviously positive and obviously negative reviews correctly does not need another thousand of those; it needs the ambiguous, sarcastic, mixed-signal reviews that sit near its decision boundary. Random sampling has no way to find those examples on purpose — it stumbles onto them only by chance, diluted among the easy cases.
Active learning flips the order of operations. The model is trained on a small seed set, then used to score the remaining unlabelled pool. The examples that score as most informative, by whatever metric the strategy defines, get sent to human annotators first. The model retrains, rescans the pool, and the cycle repeats. Over several rounds, the model reaches a given accuracy target using a fraction of the labels that random sampling would have required.
The Core Query Strategies
Uncertainty Sampling
Uncertainty sampling is the workhorse strategy because it is simple to implement and cheap to compute. The model scores every unlabelled example and ranks them by how unsure the prediction is — using measures such as least confidence, margin between the top two predicted classes, or prediction entropy. The examples closest to the decision boundary get labelled first, since those are the ones most likely to shift the boundary in a useful direction.
The tradeoff is that pure uncertainty sampling can get stuck oversampling one region of the input space, particularly in high-dimensional or highly imbalanced datasets, because the “most uncertain” examples can cluster together rather than spreading across the full distribution of the problem.
Query-by-Committee
Query-by-committee (QBC) trains multiple models — a “committee” — on the same labelled set, using different architectures, initializations, or bootstrap samples. Each committee member votes on unlabelled examples, and the examples that generate the most disagreement are prioritized for labelling. Disagreement is a proxy for the version space being poorly constrained in that region: if several reasonable models can’t agree, a human label there resolves more ambiguity than a label somewhere the committee already agrees on.
QBC tends to be more robust than single-model uncertainty sampling because it is less sensitive to one model’s particular blind spots, but it costs more compute since it requires training and maintaining several models in parallel.
Diversity and Expected Model Change
Diversity sampling addresses the clustering problem in pure uncertainty sampling by explicitly selecting examples that are both uncertain and spread out across the feature space, often using clustering or embedding-distance methods to avoid picking near-duplicates in the same labelling batch. Expected model change and expected error reduction strategies go a step further, estimating how much labelling a given example would actually shift the model’s parameters or reduce its overall error, rather than just measuring current uncertainty. These are more expensive to compute per candidate but can be worth it when labelling budgets are extremely tight.
The active learning loop
A small seed-labelled set trains an initial model, which scores the unlabelled pool by a query strategy, sends the top-ranked batch to annotators, and folds the new labels back into training — repeating until a stopping criterion (accuracy target or budget) is met.
How LLMs Changed the Active Learning Pipeline
The biggest shift in the last two years is that large language models are now good enough to act as a first labelling pass in many text and even some structured-data tasks. Instead of sending every selected example straight to a human, teams increasingly run it through an LLM first, then route only the LLM’s own low-confidence or high-disagreement outputs to human reviewers. This is sometimes called LLM-assisted pre-labeling combined with active review, and it compounds the savings active learning already provides: the model reduces which examples need any attention, and the LLM reduces how much of that attention needs to be human.
Recent work on cost-efficient text classification has shown that combining active sampling with LLM-based label generation can produce classifiers competitive with fully human-labelled baselines while removing the need for manually labelled data in some workflows entirely, using the LLM to synthesize labels for the examples an active learning loop identifies as most informative. Other frameworks pair query-by-committee-style disagreement with chain-of-thought prompting to have an LLM draft candidate labels that a human annotator then only needs to confirm or correct, rather than write from scratch — a much faster review task than blank-page annotation.
This does introduce a new failure mode worth watching: if the LLM’s pre-labels are systematically wrong in a particular way — for example, consistently mislabeling a rare class — an active learning loop built only on model uncertainty may never surface that error, because the model may be confidently wrong in the same way the LLM was. Keeping a held-out, purely human-labelled validation set that never touches LLM pre-labeling is the standard guardrail teams use to catch this.
| Task type | Recommended primary strategy | Why |
|---|---|---|
| Text classification, high LLM competence | LLM pre-labeling plus uncertainty-based human review | LLM handles clear cases; humans only see ambiguous ones |
| Rare-event or fraud detection | Diversity-weighted uncertainty sampling | Prevents the loop from ignoring rare minority-class regions |
| Medical imaging annotation | Query-by-committee with expert review | Disagreement across models flags genuinely ambiguous cases for specialists |
| Structured/tabular risk scoring | Expected error reduction | Fewer, higher-value labels matter more than raw sampling speed |
| Cold-start / very small seed set | Diversity sampling first, uncertainty later | Early rounds need coverage of the input space before uncertainty is meaningful |
Measuring Whether Active Learning Is Actually Working
Teams that adopt active learning without a measurement plan often can’t tell whether it’s helping. The standard way to check is to plot a learning curve: model accuracy (or F1, or whatever the target metric is) against number of labels used, for both the active learning loop and a random-sampling baseline trained on the same total label budget. If the active learning curve sits meaningfully above the random curve at the same label count, the strategy is earning its complexity. If the two curves are close, the extra engineering — maintaining a committee, recomputing uncertainty scores every round, managing the labelling queue — may not be worth it for that particular dataset.
It’s also worth tracking labelling turnaround time separately from label count. A strategy that saves labels but requires days of idle GPU time to rescan a huge unlabelled pool between rounds may be slower end-to-end than simply labelling more data with a faster, simpler pipeline.
- Track accuracy per label spent, not just final accuracy, across at least 4-6 rounds.
- Always keep a random-sampling control group running in parallel for the first few projects, even though it feels wasteful — it’s the only way to prove the technique earns its complexity.
- Watch class balance in the labelled set over time; some query strategies drift toward one class if that class happens to sit near more decision boundaries.
- Re-check model calibration periodically, since miscalibrated confidence scores make uncertainty sampling actively misleading.
Frequently Overlooked Details
- Batch size versus round frequencyLabelling one example at a time and retraining after each is theoretically ideal but operationally absurd; most teams batch 50-500 examples per round as a practical compromise.
- Annotator disagreementIf human annotators disagree with each other on the same examples the model finds uncertain, that’s a labelling guideline problem, not a model problem — fix the rubric before blaming the sampling strategy.
- Cold-start biasThe initial seed set shapes every later round; a biased or non-representative seed set can send the whole loop chasing the wrong kind of uncertainty.
- Stopping criteriaTeams that never define a stopping point tend to keep labelling well past the point of diminishing returns; set a target accuracy or budget ceiling up front.
- Pool stalenessIf the unlabelled pool is refreshed with new data mid-project, uncertainty scores computed on the old pool become unreliable and need recomputation.
Common mistake
Teams frequently apply pure uncertainty sampling to heavily imbalanced datasets and end up starving the majority class of new labels while over-labelling a narrow uncertain slice of the minority class, which can actually make overall model calibration worse even as it looks like the loop is working on paper. Combining uncertainty with an explicit diversity or stratification constraint avoids this trap.
What worked
A support-ticket classification team seeded their loop with 500 randomly labelled tickets, then used margin-based uncertainty sampling combined with an LLM pre-labeling pass that auto-resolved any ticket where the LLM’s top-two label probabilities differed by more than a wide margin. Human reviewers only touched the roughly 15 percent of tickets the LLM was unsure about, reaching their accuracy target with about a fifth of the human-hours a fully manual pipeline would have needed.
Building an Active Learning Pipeline in Practice
A production active learning system needs more than a query strategy; it needs a queue, a labelling interface, a retraining schedule, and monitoring to make sure the loop hasn’t drifted. Most teams wire this into their existing MLOps stack rather than building a bespoke tool. The labelling queue pulls from a feature store so that features stay consistent between training and inference, the retraining job runs on a schedule or trigger, and the resulting model gets the same monitoring treatment as any other production model, watching for both performance drift and query-strategy drift over time.
Data labelling itself has also changed shape because of synthetic data. Some teams now generate synthetic edge cases specifically to test whether their active learning loop notices them, seeding the unlabelled pool with known-hard synthetic examples and checking that the query strategy actually flags them as high priority. If it doesn’t, that’s a sign the uncertainty or disagreement metric isn’t sensitive enough for the task.
Glossary
- Uncertainty sampling
- A query strategy that selects unlabelled examples where the current model’s prediction confidence is lowest.
- Query-by-committee
- A query strategy using multiple trained models that vote on unlabelled examples, prioritizing those with the most disagreement.
- Seed set
- The small initial batch of labelled data used to train the first model in an active learning loop before any query rounds begin.
- Cold start
- The early phase of active learning where too little labelled data exists for uncertainty estimates to be meaningful, often requiring diversity-first sampling.
- LLM pre-labeling
- Using a large language model to generate draft labels for unlabelled data before human review, reducing annotation to verification rather than authoring.
Key Takeaways
- Active learning selects which unlabelled examples to annotate next, rather than labelling data randomly.
- Uncertainty sampling is the simplest and most widely used strategy but can cluster on narrow regions of the input space.
- Query-by-committee uses model disagreement instead of single-model confidence and tends to be more robust.
- Diversity sampling and expected error reduction address the coverage gaps that pure uncertainty sampling can create.
- LLM-assisted pre-labeling now routes only the LLM’s low-confidence outputs to humans, compounding label savings.
- A random-sampling control group is the only reliable way to prove an active learning strategy is actually earning its complexity.
- Stopping criteria, calibration checks, and pool staleness are the details that most often get overlooked in production deployments.
FAQs
What is active learning in machine learning?
Active learning is a training approach where the model itself helps choose which unlabelled examples get sent to human annotators next, prioritizing examples expected to teach it the most rather than labelling data at random.
How much can active learning reduce labelling costs?
Studies and production case reports commonly show 30 to 70 percent fewer labels needed to reach a target accuracy, with LLM-assisted pre-labeling pushing total human labelling effort down even further in text-heavy tasks.
What is the difference between uncertainty sampling and query-by-committee?
Uncertainty sampling uses one model’s confidence scores to pick examples, while query-by-committee trains several models and picks examples where those models disagree most, which tends to be more robust to any single model’s blind spots.
Can active learning make a model worse?
Yes, if the query strategy is poorly matched to the data, for example pure uncertainty sampling on a heavily imbalanced dataset can over-focus on one narrow region and leave the model miscalibrated elsewhere, so pairing uncertainty with diversity constraints matters.
How does active learning work with large language models?
LLMs increasingly serve as a first-pass labeller, generating draft labels for examples an active learning loop flags as informative, so human reviewers only confirm or correct the LLM’s lowest-confidence outputs instead of labelling from scratch.
Is active learning worth it for small datasets?
It can be, but with very small seed sets uncertainty estimates are unreliable, so diversity-first sampling in early rounds tends to work better than jumping straight to uncertainty-based queries.
How do teams know if active learning is actually helping?
By comparing a learning curve of accuracy versus labels used against a random-sampling baseline trained on the same label budget; if the active learning curve isn’t clearly above the baseline, the added complexity may not be worthwhile.
What industries use active learning most heavily?
Medical imaging, fraud and risk detection, autonomous vehicle perception, and large-scale text classification are among the heaviest users, largely because expert labelling time in those domains is especially expensive or scarce.
Active learning pairs naturally with other parts of the modern ML stack. Teams building retrieval-augmented systems can read more in our guide to RAG architecture, while those measuring model quality after retraining rounds should see our piece on LLM evaluation frameworks. If synthetic examples are part of your seed set strategy, our article on synthetic data pipelines covers how to generate them safely, and the risks of relying too heavily on model-generated data are covered in our piece on model collapse. For teams weighing whether human review capacity should scale alongside labelling automation, our guide to data labelling in the age of synthetic data is a useful companion read.
- Label Your Data, “Active Learning Machine Learning: How It Reduces Labeling Costs in 2026”
- IntuitionLabs, “Active Learning and Human Feedback for Large Language Models”
- arXiv, “LLM on a Budget: Active Knowledge Distillation for Efficient Classification of Large Text Corpora”
- arXiv, “Applying LLMs to Active Learning: Towards Cost-Efficient Cross-Task Text Classification without Manually Labeled Data”
- arXiv, “From Selection to Generation: A Survey of LLM-based Active Learning”
- Encord, “Active Learning in Machine Learning Guide”
