Recommender Systems Rebuilt With Embeddings and LLMs

The Recommendation Stack Is Being Rebuilt From the Retrieval Layer Up

For most of the last decade, recommender systems followed a predictable shape: a candidate generation stage built on collaborative filtering or matrix factorization, followed by a lightweight ranking model that scored a few hundred candidates, followed by business rules that filtered the output. That architecture is now being pulled apart and reassembled around dense embeddings and large language models, and the change is not cosmetic. It touches how candidates are generated, how they are scored, how cold-start items and users are handled, and what the system can tell a user about why an item showed up at all.

The shift is visible in public engineering writing from the largest consumer platforms. Netflix’s AI Platform team described treating a member’s entire interaction history, every play, search, rating, and browse event, as a long sequence and framing the recommendation problem as next-token prediction, the same objective that trains a chat model. Spotify has talked publicly about semantic-ID generative retrieval for podcast discovery, and about an agentic router that decides per-query whether to invoke an expensive LLM reasoning layer or fall back to a cheaper collaborative-filtering path. Neither company replaced its existing infrastructure wholesale. Both layered LLM-driven components on top of embedding infrastructure that already existed, which is the pattern most teams rebuilding a recommender system in 2026 are actually following.

Why Classic Collaborative Filtering Hits a Wall

Collaborative filtering works by finding statistical co-occurrence: users who liked item A also liked item B. It is cheap, well understood, and still forms the backbone of many production systems. But it has three structural weaknesses that dense embeddings and LLMs directly address.

• Cold start. A brand-new item or user has no interaction history, so a pure collaborative signal has nothing to work with. Content-based embeddings, and especially LLM-generated embeddings from item descriptions, give a new item a position in vector space before a single click has happened.
• Semantic blindness. Two items can be behaviorally similar without being semantically related, and two items can be semantically identical (a product description rewritten by a different seller) without collaborative signal ever connecting them. Text and multimodal embeddings close that gap.
• No explanation surface. A matrix factorization score is a number. It cannot tell a user, in their own language, why a show or product was suggested. An LLM sitting downstream of the same retrieval signal can generate that explanation as a byproduct of scoring.

Dense Embeddings as the New Substrate

The architectural move that makes everything else possible is representing users and items as dense vectors in a shared embedding space, rather than as rows and columns in a sparse interaction matrix. Two-tower models remain the workhorse here: one tower encodes the user (recent history, profile signals, session context), the other encodes the item (title, description, image, metadata), and the two towers are trained so that vectors for a user and the items they engaged with end up close together in the space. Retrieval then becomes an approximate nearest-neighbor search, which scales to catalogs of hundreds of millions of items with sub-30-millisecond latency using libraries built for that exact problem.

What changed recently is who encodes the item tower. Instead of a bag-of-words or a small trained encoder, teams are using LLM-derived embeddings that capture nuance a shallow model misses, tone, sentiment, implied use case, brand voice, from raw product or content descriptions. An embedding alignment step is often needed here: item embeddings from a pretrained collaborative filtering model are aligned with text embeddings from an LLM encoder so the two vector spaces are compatible before they are fused or handed to a downstream ranker.

Two-Tower Models and Semantic IDs

A newer variant compresses each item’s dense embedding into a short, discrete token sequence, commonly called a Semantic ID. Instead of an opaque integer item ID, an item becomes something like a three-to-eight-token code where nearby codes correspond to semantically similar items. This representation is what makes generative retrieval possible, because it turns “find similar items” into “generate the next tokens,” a task language models are already extremely good at.

Generative Retrieval: Asking the Model to Produce the Item, Not Search For It

Generative retrieval, sometimes called GenIR in the research literature, flips the retrieval step on its head. Instead of embedding a query and searching an index for nearby item vectors, an LLM is trained or prompted to directly generate the identifier, title, or Semantic ID tokens of the relevant item. The model has effectively memorized a compressed map of the catalog during training or fine-tuning, and retrieval becomes a decoding problem rather than a search problem.

This matters for three practical reasons. First, it removes the disconnect between the objective the retrieval model was trained on and the objective the ranking model actually cares about, because the same generative model can, in principle, do both. Second, it makes personalization context, a user’s stated preference, a recent complaint, a follow-up question, easy to fold in through the prompt rather than through a separate feature pipeline. Third, it opens the door to conversational recommendation, where a user can say “something like the last show but shorter episodes” and the model can reason over that instruction directly instead of mapping it to a fixed filter.

The tradeoff is amplification bias and homogeneity: research on decoding strategies for LLM-based recommendation has flagged that naive generative retrieval can over-recommend already-popular items and reduce catalog diversity, because the token sequences for popular items are simply easier for the model to generate with high confidence. Teams shipping generative retrieval in production report needing explicit diversity-promoting decoding strategies, not just top-k sampling, to counter this.

LLMs as Re-Rankers: The Pragmatic Middle Ground

Full generative retrieval is still expensive and, for most catalogs, unnecessary. The far more common production pattern in 2026 is a hybrid: keep an existing embedding-based or collaborative-filtering candidate generator, and add an LLM re-ranking stage on top of a shortlist of 20 to 200 candidates. The re-ranker sees the user’s context, the candidate list, and often auxiliary signals like recency or inventory constraints, and produces both an ordering and, optionally, a short rationale for the top items.

This pattern is attractive because it isolates risk. If the re-ranker misbehaves, the system falls back to the underlying candidate order, which is usually still reasonable. It also lets teams apply an LLM’s reasoning ability, understanding that a user browsing running shoes late at night after searching “marathon training” probably wants something different than someone browsing the same shoes after searching “gift for dad”, without retraining the entire retrieval stack.

Solving Cold Start With Language

Cold start remains the single most cited justification for adopting LLMs in recommendation, because it is the place where collaborative signal is, by definition, absent. Two applications dominate here. For new items, an LLM reads the raw description, images, and metadata and produces both a semantic embedding and a set of inferred tags, effectively giving the item a synthetic interaction profile before any real interactions exist. For new users, an LLM can synthesize a starter profile from onboarding responses, browsing referral context, or even a short natural-language prompt (“I’m looking for something for a beginner home cook”), producing a usable embedding on the first session instead of waiting for enough clicks to train a collaborative signal.

Explanations as a First-Class Output

One of the most requested features from product teams, and one collaborative filtering structurally cannot deliver, is a plain-language reason for a recommendation. Because an LLM re-ranker already reasons over the candidate and the user context to produce an ordering, generating a one-sentence explanation is close to free computationally. This has become a differentiator in category pages and in support flows, where “recommended because it pairs with items in your recent order” or “similar to shows you finished this month” measurably increases click-through and reduces “why am I seeing this” support tickets.

Latency, Cost and the Hybrid Reality

The honest constraint on all of this is money and milliseconds. A full LLM forward pass over a candidate list costs orders of magnitude more than a nearest-neighbor lookup, and homepage rendering budgets are typically measured in tens of milliseconds, not hundreds. This is why the “agentic router” pattern described by Spotify engineers, and echoed across other teams building similar systems, is becoming the default: cheap embedding retrieval handles the bulk of traffic, and the LLM layer is invoked selectively, for ambiguous queries, for cold-start users, for surfaces where an explanation adds clear value, rather than for every request. Building this router well, deciding when the expensive path is worth it, has become its own optimization problem, sitting alongside the retrieval and ranking models themselves.

Migrating an Existing Recommender Without a Rewrite

Almost nobody rebuilding a recommender in 2026 is starting from a blank slate, and the migration path that keeps showing up in practitioner write-ups follows a consistent sequence rather than a single big-bang cutover. Teams start by swapping the item-side features feeding an existing two-tower model for LLM-derived embeddings, keeping the retrieval architecture untouched so the blast radius of a bad rollout is limited to relevance quality, not system uptime. Only after that swap is validated do teams introduce a re-ranking stage, usually behind a feature flag limited to a small percentage of sessions, and only after the re-ranker has proven itself on relevance and latency does anyone consider generative retrieval for a specific, bounded surface such as a “for you” row or a conversational search box, rather than the entire catalog.

This staged approach exists because each layer introduces its own failure mode, and stacking all of them at once makes root-causing a regression close to impossible. An embedding swap can silently shift what “similar” means for an entire catalog segment. A re-ranker can introduce latency spikes under load that never show up in offline testing. Generative retrieval can quietly narrow the catalog’s effective diversity in a way that takes weeks to notice in aggregate metrics. Teams that skip the staging and adopt all three simultaneously consistently report longer, more painful debugging cycles when something in the pipeline underperforms, because there is no way to isolate which layer is responsible without re-running experiments layer by layer after the fact.

Organizationally, this also changes who owns what. A pure collaborative filtering system typically sits with a single ranking team. Once LLM-derived embeddings, a re-ranker, and possibly generative retrieval are all in the stack, the system spans applied ML, platform, and increasingly the same LLM infrastructure team responsible for other generative features in the product, which makes clear ownership boundaries and shared evaluation dashboards a practical necessity rather than a nice-to-have.

Evaluation Is Harder Than It Looks

Offline metrics like recall@k and NDCG were built for a world where the retrieval model’s only job was ranking a fixed catalog by similarity. They do not capture whether a generated explanation is accurate, whether generative retrieval has quietly narrowed the diversity of what gets served, or whether a cold-start recommendation that looks reasonable offline actually converts online. Teams shipping these systems increasingly pair offline ranking metrics with online experiments that specifically track catalog coverage and long-tail exposure, not just top-line click-through, because it is easy for an LLM-driven system to look like it is improving relevance while quietly consolidating attention on a smaller set of popular items.

• Embedding alignmentCollaborative-filtering vectors and LLM text embeddings live in different spaces and need an explicit alignment step before they can be fused or compared.
• Decoding diversity controlsGenerative retrieval needs deliberate diversity-promoting decoding, not just higher temperature, to avoid collapsing onto popular items.
• Router qualityThe logic deciding when to invoke the expensive LLM path matters as much as the LLM itself for cost control.
• Explanation faithfulnessA generated rationale can sound plausible while not actually reflecting the signal that drove the ranking, which erodes user trust once noticed.
• Catalog freshness lagGenerative models trained or fine-tuned on a catalog snapshot can lag behind embedding indexes when it comes to reflecting newly added items.
• Incremental learning gapMost generative recommenders still need periodic retraining rather than true online incremental updates, an active research gap as of 2026.
• Offline-online mismatchRecall and NDCG improvements offline do not reliably predict online diversity or long-term engagement, so both must be measured.
• Cost attributionWithout per-query cost tracking for the LLM path, it is easy to lose sight of how much the reasoning layer actually costs relative to the lift it delivers.

Semantic ID

A short sequence of discrete tokens representing an item’s position in embedding space, used as the generation target in generative retrieval instead of an opaque numeric item ID.

Two-tower model

An architecture with separate encoders for users and items, trained so that vectors for engaged user-item pairs end up close together in a shared embedding space.

Generative retrieval

A retrieval approach where an LLM directly generates the identifier or descriptive tokens of a relevant item, rather than searching an index of precomputed vectors.

Cold start

The problem of recommending for new items or new users who have no interaction history for a collaborative signal to draw on.

Agentic router

A lightweight decision layer that determines, per query, whether to invoke an expensive LLM reasoning path or fall back to a cheaper embedding-based path.

Re-ranker

A second-stage model, often an LLM, that reorders a shortlist of candidates produced by an upstream retrieval system, frequently adding an explanation as a byproduct.

FAQs

What is the difference between a two-tower model and generative retrieval?

A two-tower model encodes users and items into a shared vector space and finds matches through nearest-neighbor search. Generative retrieval skips the search step entirely, having an LLM directly generate the token sequence that identifies the relevant item.

Do LLM recommender systems replace collaborative filtering entirely?

Rarely. Most production systems keep collaborative filtering or two-tower embedding retrieval as the default candidate generator and add an LLM as a selective re-ranking or explanation layer, reserving full generative retrieval for specific high-value surfaces.

How do LLMs help with cold-start recommendations?

They generate a semantic embedding directly from an item’s description or a user’s onboarding responses, giving a usable representation before any real interaction data exists, which collaborative filtering cannot do on its own.

Why do generated recommendation explanations sometimes feel wrong?

An LLM can generate a plausible-sounding rationale that does not actually reflect the signal that drove the ranking decision, since explanation generation and ranking are not always tightly coupled inside the model.

What is a Semantic ID in recommendation systems?

It is a compact, discrete token representation of an item’s embedding, designed so that semantically similar items get similar token sequences, which makes the item generatable by a language model rather than only searchable.

Why does generative retrieval risk reducing catalog diversity?

Popular items have token sequences the model can generate with high confidence, so naive decoding tends to favor them repeatedly, requiring explicit diversity-promoting decoding strategies to keep long-tail items visible.

What latency budget should teams plan for an LLM re-ranking layer?

Expect to add roughly 100 to 400 milliseconds on top of an existing sub-30-millisecond retrieval stage, which is why most teams invoke the LLM path selectively rather than on every request.

How should teams evaluate an LLM-based recommender beyond click-through rate?

Pair standard ranking metrics like recall and NDCG with explicit tracking of catalog coverage and long-tail exposure, since a system can show strong click-through gains while quietly narrowing what users ever see.

For teams building the retrieval half of this stack, the patterns described in retrieval-augmented architecture design apply directly, since generative retrieval and RAG share the same nearest-neighbor and indexing foundations. Measuring whether a re-ranking layer is actually working requires the kind of rigor covered in LLM evaluation frameworks, and rolling out a new ranking model safely depends on the experimental design discussed in A/B testing for machine learning models. Once a recommender is live, the operational discipline in model monitoring in production is what catches the popularity-amplification drift described above before it shows up in revenue. Teams debating build-versus-buy tradeoffs for the embedding and retrieval infrastructure itself should also read feature stores explained, and teams running the staged migration described above will want the organizational playbook in building an ML platform team.

• Netflix AI Platform team, Scaling LLM Post-Training at Netflix, February 2026
• Decoding Matters: Addressing Amplification Bias and Homogeneity Issue for LLM-based Recommendation, arXiv:2406.14900
• Continual Low-Rank Adapters for LLM-based Generative Recommender Systems, arXiv:2510.25093
• Leveraging Memory Retrieval to Enhance LLM-based Generative Recommendation, arXiv:2412.17593
• Inside Spotify’s Recommendation System: Complete Guide, music-tomorrow.com
• Bandits vs LLM Costs: Hybrid Recommender Systems 2026, philippdubach.com
• alopatenko/LLMSearchRecommender compendium, GitHub