Stanford's AI Lab recently published findings from a paper called "The Embedder's Dilemma," attributed to researcher Muennighoff, showing that LLMs now outperform embedding models on retrieval tasks, but at substantially higher cost. (source) That single finding crystallises a design decision that engineering teams are quietly agonising over right now: when do you call the expensive model, and when does the cheap one already do the job?
This isn't a philosophical question. It shows up in your cloud bill, your p95 latency, and your users' patience.
An embedding model takes a piece of text, turns it into a vector of numbers, and finds the nearest neighbours in your index. It's fast, cheap, and stateless. You run it once per document at index time, then again at query time, and the arithmetic does the rest.
An LLM call is something different in kind. The model reads the query, reasons about it, and can synthesise, interpret, or re-rank results using actual language understanding. It's also slower and costs two or three orders of magnitude more per query, depending on the model and token count. (Treat those figures as illustrative, not gospel; check current provider pricing before you commit to anything.)
The Stanford finding says LLMs win on retrieval quality. What it doesn't say is that you should use them for every search query in your product. That would be like hiring a Michelin-trained chef to make toast every morning. Technically possible, obviously wasteful.
Pull up your query logs and you'll likely find something close to a bimodal distribution. A large proportion of queries are short, specific, and well-formed: "reset password," "invoice #4471," "cancel subscription," "Python string split." For these, an embedding model is not a compromise, it's the correct tool.
The characteristics that make embeddings the right call:
In these conditions, a capable embedding model handles retrieval cleanly. The "cheaper" option isn't a sacrifice; it's a sensible match between the tool and the task.
The other cluster in those logs looks quite different. Queries like "which of our pricing plans makes sense for a ten-person team that needs SSO but doesn't need analytics," or "summarise what changed in our product between Q1 and Q3," or "what's the difference between feature A and feature B for someone coming from Competitor X." These are not lookup tasks. They require synthesis, contextual judgment, or multi-step reasoning that an embedding model simply isn't built for.
An LLM call is worth the cost when:
I find this genuinely clarifying: the question is not "which model is better" in the abstract. It's "what is the query actually asking for, and what does a good answer look like?" The shape of the right answer determines the tool.
The pattern that's gaining traction, and that teams building cost-efficient AI products are starting to treat as standard practice, is a routing layer that classifies queries before dispatching them. Simple, high-confidence queries go directly to the embedding index. Complex, ambiguous, or synthesis-heavy queries go to the LLM. Some queries hit a hybrid path: embeddings retrieve candidates, then the LLM re-ranks or reformulates the final answer.
The router itself can be lightweight. A small classifier trained on your own query logs, a few heuristics around query length and vocabulary, or even a fast compact model like Qwen2.5-7B sitting in front of your main pipeline can do this job without adding meaningful latency. You're not asking the router to answer the question; you're asking it to recognise the shape of the question.
One tactic I think is underrated here: semantic caching. If you cache LLM responses against semantically similar queries, some teams report cache hit rates high enough to meaningfully cut their inference bill. Common questions in a SaaS product, pricing queries, onboarding help, feature explanations, recur constantly. You pay for the LLM call once and serve cached results to the next hundred users asking roughly the same thing.
If you're a founder or a product team deciding how to build search into your application, the practical takeaway is this: don't default to the most capable model because it feels safer. That instinct is expensive.
Start by categorising what your users are actually searching for. If most of it is lookup, finding records, articles, products, past conversations, an embedding model gets you most of the way there at a fraction of the cost. Add LLM capability at the edges: complex queries, synthesis tasks, cases where a wrong answer has real consequences.
I'd argue the teams that get this right aren't the ones who pick the most powerful model. They're the ones who know exactly which queries need that power and route everything else to something faster and cheaper. Twenty years of building software has taught me that the cost discipline you build into an architecture at the start is much easier to maintain than cost discipline you try to retrofit later, when users are already complaining about slow search and your cloud bill has doubled.
The Stanford finding is worth taking seriously. LLMs do outperform embedding models on retrieval, full stop. The design question is whether that performance delta matters for your specific query, at your specific scale, at your specific cost tolerance. Often it doesn't. When it does, spend the money without hesitation.
Does the Stanford finding mean I should replace my embedding-based search with an LLM? Not necessarily. The finding says LLMs outperform embedding models on retrieval quality, but at much higher cost. For most products, the right approach is routing, sending simple lookup queries to embeddings and reserving LLM calls for queries that genuinely need reasoning or synthesis.
How do I know which of my queries are "complex enough" to warrant an LLM call? Look at your query logs. Short, specific queries ("cancel subscription," "invoice #4471") are almost always handled well by embeddings. Longer, conversational, or comparison-style queries are where LLM calls earn their price. A lightweight classifier trained on your own logs can do this routing automatically.
What's the real cost difference between embedding models and LLM calls? The gap is large, often two to three orders of magnitude per query depending on the models and token counts involved. The exact figures change as providers update pricing, so verify against current pricing pages before designing your architecture around a specific number.
What is semantic caching and does it actually help? Semantic caching stores LLM responses against semantically similar queries so you serve a cached answer instead of making a fresh LLM call. For SaaS products where users repeatedly ask the same types of questions, some teams report hit rates high enough to meaningfully reduce inference costs.
Can I build this kind of hybrid routing without a large engineering team? Yes. The router itself can be a lightweight classifier or a small fast model. The embedding index handles the majority of queries. LLM calls are reserved for the edges. The architecture is not complex; the discipline is in classifying your query types correctly at the outset.
