BM25 vs. Embedding Models for Agent Tool Retrieval

Have you ever opened a drawer looking for one screwdriver and found a few hundred other things crammed in there? With ten things in the drawer you grab the right one without thinking. With a few hundred, you start digging, and sooner or later you grab the wrong one.

Agents are heading into the same drawer. MCP has made tools so easy to install that a few servers can hand your agent hundreds of them, and on every request it has to answer the same question first: which one do I use? The usual advice is to put all the tools in the prompt while they fit, and switch to retrieval once they don’t. What nobody tells you is what happens after that switch. Does retrieval hold up as you keep adding tools, or does it slowly fall apart? And does paying for a bigger model help?

That’s what I wanted to find out. So I took 200 questions from ToolRet [1], a public benchmark for tool retrieval, and kept them exactly the same while I kept making the pile bigger, from 1,000 tools all the way up to 3,794. The right answers never moved. Only the amount of junk around them grew. That way, if a method gets worse, we know the bigger pile did it.

Here’s what came out of it. Every method got worse as the pile grew, and by roughly the same amount, about a fifth of its score. A plain keyword search from the 1990s kept up with an embedding model 24 times its size while running hundreds of times faster. Putting every tool in the prompt ran out of room long before the pile stopped growing. And the one change that helped more than a bigger model was simply giving the retriever the full tool documentation instead of a one-line summary.

The code, the eval set and every per-query result are on GitHub if you want to rerun any of it.

Experiment setup

ToolRet pulls together more than thirty tool-use datasets into about 7,600 tasks over 43,000 tools, and each task is labelled with the tools that solve it [1]. It came out before MCP and isn’t about MCP at all, but it tests the exact same problem. You have a request, a big pile of tools, and you need to find the right ones.

I used its code category, which has 3,794 tools. I picked it for a slightly sneaky reason. Its tool descriptions are wordy enough that Gemma 3’s 131,072-token context window fills up at about 2,500 tools. That puts the point where the prompt approach breaks right in the middle of the experiment, so we get to watch it happen.

The 200 questions need 186 different tools between them. Every pile contains all 186, and the rest is filler:

Each pile is the previous one with more tools added on top, so nothing else changes from row to row.

To score each method we look at its top ten picks and ask three questions. Did the right tool make the top ten at all? That’s Recall@10. Was it near the top, where the agent is most likely to use it? That’s nDCG@10, the main score. And if the task needed three tools, did all three make it? That’s Completeness@10, and it matters because an agent with two out of three tools still can’t finish the job.

Each tool is shown as just its name and a short description, trimmed to 180 characters. We’ll come back to that choice near the end, because it turns out to matter more than any model.

Why not just put every tool in the prompt?

This is what most people do first, and it makes sense. Paste the whole catalogue into the prompt, describe the task, and let the model pick. Nothing gets filtered out by mistake, because the right tool is always right there in front of it.

I tried this with Gemma 3 12B running locally through Ollama on an M5 Pro with 48 GB of memory. At 1,000 tools the prompt is 53,160 tokens, and the model took 215 seconds just to read it before spending 2.3 seconds on the answer. Doing that for 200 questions across the four piles that fit would take about 82 hours. So I timed a few queries and stopped there. That means I don’t have accuracy numbers for this approach, and I can’t tell you whether it would have beaten retrieval on the smaller piles.

The bigger problem is space. The full 3,794-tool pile comes to about 198,000 tokens even with the trimmed descriptions, which is well past Gemma 3’s window. With the full documentation it doesn’t even fit at 1,000 tools. Whatever the LLM does, it always works from a cut-down description of its tools. Keep that in mind for later.

If you do go this route, two things will save you some pain. First, Ollama’s default context window is small, and when your prompt is longer it quietly chops tokens off the front and answers anyway, with no error. Set num_ctx big enough and check prompt_eval_count in the response to make sure the whole catalogue was actually read. Second, put the catalogue first and the question last. The catalogue is the same every time, so the model can cache it and only read the new question on each request. Put the question first and you pay the full 215 seconds every single time.

def build_prompt(catalogue: str, query: str, instruction: str) -> str:    # catalogue first so the KV cache is reused across queries    task = f"TASK: {query}nWHAT TO LOOK FOR: {instruction}"    return PREAMBLE + catalogue + INSTRUCTIONS + task + "nnJSON:"

A hosted model with a 1M-token window can fit the whole pile, but you pay for it on every request. At the list prices when I ran this, one uncached query over all 3,794 tools cost about $0.99 on Claude Opus 5 and $0.40 on Sonnet 5. Prompt caching brings that down a lot, but you’re still sending the whole catalogue each time.

The retrievers

BM25, with one fix for code

BM25 [2] is keyword search. It gives a tool points for every word it shares with the question, more points for rare words, and a small bonus for short descriptions. There’s no model and no GPU.

There’s one trap here, though. Tool names are code, like get_factors or QueryHistoryToday. If you split text on spaces, get_factors stays one long word, and no question in plain English will ever contain it. The tool just quietly scores zero, which looks like a bad result rather than a bug. The fix is to split names on underscores and capital letters, and keep the original too:

_SPLIT = re.compile(r"[^A-Za-z0-9]+")_CAMEL = re.compile(r"(?<=[a-z0-9])(?=[A-Z])")def tokenize(text: str) -> list[str]:    out = []    for chunk in _SPLIT.split(text):        if not chunk:            continue        parts = [p.lower() for p in _CAMEL.sub(" ", chunk).split()]        if len(parts) > 1:            out.append(chunk.lower())   # keep the compound too        out.extend(parts)    return [t for t in out if t not in STOPWORDS and len(t) > 1]

Now get_factors gets indexed as get_factors, get and factors, and a question about factors can find it. The rest of BM25 is about thirty lines of standard code, which you’ll find in the repo.

Embedding models

An embedding model turns every tool and every question into a list of numbers, and tools whose numbers sit close to the question’s numbers win. The nice part is that the words don’t have to match. A question about “finding the factors of a number” can still land on a tool that “returns all divisors”.

I tried three of them: BGE-large (335M parameters) [3], e5-large-v2 (335M) [4] and Qwen3-Embedding-8B [5], which is 24 times bigger and needs 16 GB of memory. We encode every tool once up front, and after that each question is a single encode and a quick multiply.

model = SentenceTransformer("BAAI/bge-large-en-v1.5", device="mps")tool_vectors = model.encode(texts, normalize_embeddings=True)  # once per corpusdef search(query, k=10):    q = model.encode(QUERY_PREFIX + query, normalize_embeddings=True)    scores = tool_vectors @ q   # cosine, since both sides are normalised    top = np.argpartition(-scores, k)[:k]    return [tool_ids[i] for i in top[np.argsort(-scores[top])]]

One thing that’s easy to miss: each model family expects its own little prefix in front of the text, and using the wrong one costs you points without any warning.

If you pick the prefix by looking at the model’s name, check the most specific names first. e5-mistral-7b-instruct has “e5” in its name, but it wants the instruct format, not E5’s.

Retrieve, then rerank

An embedding model never looks at the question and the tool together. It turns each one into numbers separately and hopes they end up close. A cross-encoder reads the two side by side and gives the pair a score, which should be sharper. The catch is that it has to run once for every tool it scores, so we can’t point it at the whole pile. Instead, BGE-large grabs its top 50, and the cross-encoder (bge-reranker-v2-m3) reorders just those.

reranker = CrossEncoder("BAAI/bge-reranker-v2-m3", device="mps", max_length=512)def two_stage(query, k=10, depth=50):    candidates = search(query, k=depth)    scores = reranker.predict([(query, text_by_id[c]) for c in candidates])    return [c for _, c in sorted(zip(scores, candidates), reverse=True)][:k]

The reranker can only shuffle what it’s given. If the right tool isn’t in those 50, it’s gone for good.

You might wonder why I didn’t fine-tune a model, since ToolRet comes with 200,000 training examples. The problem is that those examples come from the same datasets as the test questions, so a fine-tuned model would partly be learning the test. There’s also a practical reason. Every time you install a new server, a fine-tuned model has to be retrained to know about its tools, while BM25 and an off-the-shelf embedder can search a new tool the moment you add it.

Result 1: everything gets worse, and at about the same speed

nDCG@10 with the trimmed tool text, 200 questions.

Remember, the questions and answers are identical in every row. The only thing that changed is how much junk we added, and every single method slid down. Going from 1,000 to 3,794 tools cost BGE-large 0.118, BM25 0.128, the reranker 0.139 and Qwen3-8B 0.151. Those drops are solid, not noise. I resampled the questions 10,000 times, and every drop stayed above 0.09.

None of the lines level off, either. Retrieval wears down a little with every tool you add. Compare that with the prompt approach, which works fine until the catalogue doesn’t fit, and then doesn’t work at all. One fades slowly and the other falls off a cliff.

The surprise was Qwen3-8B. It started on top and fell the furthest, so by the full pile its lead was gone. Being 24 times bigger didn’t make it any better at ignoring the extra junk.

Result 2: a bigger model cost more but didn’t score higher

On the full pile, BM25, BGE-large, Qwen3-8B and the rerank pipeline all land between 0.481 and 0.494. Honestly, with 200 questions those are ties. When I resampled, the gap between BM25 and BGE-large could just as easily have gone either way, and the same was true for Qwen3-8B. Only e5-large-v2 was clearly behind. This lines up with what the ToolRet authors found across their whole benchmark, where BM25 and BGE-large also ended up close together [1].

Where they really differ is cost:

So for the same accuracy, BM25 answers about 40 times faster than BGE-large and about 2,000 times faster than the rerank pipeline, and it doesn’t need a GPU at all.

Result 3: the reranker found more tools but ranked them worse

This one confused me at first. The reranker got the right tool into the top ten more often than BGE-large on its own, on every pile. At 1,000 tools, Recall@10 went from 0.742 to 0.798, and at 3,794 from 0.604 to 0.641. But its nDCG was lower on four of the five piles. How can both be true?

It pulled the right tools up from somewhere in positions 11 to 50 into the top ten, but then it put them in a worse order than BGE-large had. And it wasn’t short of material. The right tools were in its top 50 between 79% and 90% of the time, so most of the answers were sitting right there. It only recovered 8% to 36% of that room. Faced with fifty tools that all sound alike, a general-purpose reranker couldn’t reliably tell which one was right.

Result 4: better descriptions helped more than a bigger model

Remember the trimmed descriptions? We only trimmed them because the LLM needed them short. Retrieval has no such limit, so I ran BM25 and BGE-large again on the full tool documentation, including the parameter details. Tang and colleagues argue that how you describe a tool matters as much as which retriever you use [6], and this is the simplest way to test that.

The full documentation wins in every single row, by about 0.02 to 0.04. That’s roughly the same boost Qwen3-8B gave over BGE-large on the smallest pile, except here you don’t need a 16 GB model to get it. I want to be careful, though. Because the piles overlap, those ten wins aren’t ten separate experiments, and only a couple of them clearly hold up on their own when I resample, BM25’s gain on the full pile being one. The direction is consistent, but the exact size is fuzzy.

BM25 on the full documentation was the best setup I tested, with 0.533 nDCG on the full pile at about a millisecond per question. I didn’t run Qwen3-8B or the reranker on the full documentation, though, so I can’t claim BM25 beats them there. My guess is they’d improve too.

This also makes the prompt approach look a bit worse. The text that helped retrieval the most is exactly the text the LLM can’t fit.

What does it mean for your agent?

An nDCG of 0.53 sounds okay. So here’s the same result put another way. With 3,794 tools, even the best setup got every tool a task needed into the top ten only 65% of the time. On the other 35% of tasks, the agent was missing at least one tool before it even started. Qwen3-8B managed 60% and BGE-large 56%. That’s a lot of tasks that fail before the agent does anything at all.

What this experiment can’t tell you

The code category is friendly to keyword search. A lot of tool names here literally say what they do, like get_factors, so a question and the right tool tend to share words. On a catalogue full of vague or branded names, embeddings might pull ahead. ToolRet’s web and customized categories would be the place to check.

Each question also comes with an instruction that ToolRet wrote using GPT-4o. That adds extra words to every question, which probably helps BM25 more than the embedding models. I haven’t run it without the instruction yet.

Two hundred questions isn’t a lot, which is why I keep calling most of the gaps ties. And the absolute numbers don’t line up with the ToolRet paper’s own leaderboard, which uses all 43,000 tools across every category. What carries over is the shape of the curves, not the exact scores.

Finally, there’s plenty I didn’t try: mixing BM25 and embeddings into a hybrid search, a reranker trained on tool data, or measuring how accurate the LLM actually is on the piles where it fits. Hybrid search is the first thing I’d try next.

So what should you do?

If your agent has a few dozen tools, don’t bother with any of this. Put them all in the prompt and let the model choose. Once you’re into the hundreds, retrieve first. And expect it to get a bit worse with every tool you add. In this experiment it was about 0.04 nDCG for every extra thousand.

For the retriever itself, I’d start with BM25 on the full tool documentation, with a tokenizer that splits code-style names. It kept up with models up to 24 times its size, runs in under a millisecond on a CPU, and can find a new tool the moment you add it. If you have time to spend on making tool selection better, spend it on the tool names and descriptions before you spend it on a bigger model. Better text helped about as much as the bigger model did, and it’s much cheaper.

Even the best setup missed a tool it needed on about a third of the tasks, so this problem is far from solved. It’s worth knowing that before you trust any tool picker blindly.

All the code, the eval set and the per-question results are at github.com/ankur287/toolret-tool-retrieval.

References

[1] Z. Shi, Y. Wang, L. Yan, P. Ren, S. Wang, D. Yin and Z. Ren, Retrieval Models Aren’t Tool-Savvy: Benchmarking Tool Retrieval for Large Language Models (2025), Findings of ACL 2025, arXiv:2503.01763

[2] S. Robertson and H. Zaragoza, The Probabilistic Relevance Framework: BM25 and Beyond (2009), Foundations and Trends in Information Retrieval

[3] S. Xiao, Z. Liu, P. Zhang, N. Muennighoff, D. Lian et al., C-Pack: Packed Resources For General Chinese Embeddings (2023), arXiv:2309.07597

[4] L. Wang, N. Yang, X. Huang, B. Jiao, L. Yang et al., Text Embeddings by Weakly-Supervised Contrastive Pre-training (2022), arXiv:2212.03533

[5] Y. Zhang, M. Li, D. Long, X. Zhang, H. Lin et al., Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models (2025), arXiv:2506.05176

[6] Y. Tang, W. Su, Y. Liu and Q. Ai, Multi-Field Tool Retrieval (2026), arXiv:2602.05366