# How do I choose the right language model for each task?

> Write the pass marks first, test every candidate on a fixed set of real cases, keep the model a config value behind one layer, and take the cheapest model that passes

- Author: Ash Gaikwad (https://ashgaikwad.com)
- Published: 2025-12-12
- Topics: llm, evaluation, architecture, ai engineering
- Canonical: https://ashgaikwad.com/blogs/how-do-i-choose-the-right-language-model-for-each-task

Choose a language model for each task by measuring it. Write down what passing means for that one job, collect a fixed set of real cases with known answers, run every candidate through the same cases, and take the cheapest model that passes. Keep every model behind one layer so the choice is a config value, and run the check again whenever the model, the prompt or the rules change. A bigger model is the right call when nothing cheaper passes, and the check is what tells you so.

I am Ash, a Staff engineer in Toronto. I call every model through one provider-neutral layer, so the model for each job is a config value, and each job gets the cheapest model from any provider that passes a measured check on real data. On one platform I ran, I built an evaluation suite that replays a fixed set of questions against every prompt or model change before release. This article is for engineers and leads whose product started with one default model for everything.

![How I pick the model for each job in five steps. I collect a fixed set of real cases with known answers. I write pass marks for quality, speed and data rules before the first run. I run the same cases through models from any provider. I set the cheapest one that passes as a config value. I run the check again on every change. If no cheap model passes, the bigger model is the answer.](/images/blogs/model-per-job-measured-check.svg)

## Pick per job, not per product

A task here means one job with one kind of input and output. Sorting a support message is a job, and so is deciding whether a listing may go live. Each gets its own check and its own model, because they fail in different ways.

Vendor guidance agrees. Anthropic's [guide to choosing a model](https://platform.claude.com/docs/en/about-claude/models/choosing-a-model) says that choosing "means balancing capabilities, speed, and cost", and that "Tuning effort is often a better lever than switching models". So you pick a model together with its settings. OpenAI's [model selection guide](https://developers.openai.com/api/docs/guides/model-selection) calls its own advice a starting point and says to "experiment using the same inputs to compare results and keep the lightest setting that meets your quality bar".

## Build the check from real cases with known answers

The Anthropic page says it plainly. "Having a good evaluation set is the most important step in the process." A good set has three properties. The cases are real, each has an answer you already trust, and the set stays fixed while you compare models.

Real means the cases come from the job as it runs. OpenAI's [evaluation best practices](https://developers.openai.com/api/docs/guides/evaluation-best-practices) say to "Log everything" so you can "mine your logs for good eval cases", and Anthropic's [guide to building evaluations](https://platform.claude.com/docs/en/test-and-evaluate/develop-tests) says to "mirror your real-world task distribution". Add the awkward cases on purpose, such as empty input and cases where two careful people would disagree.

Known answers are the part people skip, and the cheapest source is a decision a person already made. In my own products, the model that reviews new listings ran in shadow mode against 761 listings that a person had already decided (466 approved and 295 hidden).

Without past decisions, grade in this order. Use checks that code can run first, such as valid structure or a match against a label. Anthropic's guide says to "Automate when possible" and lists string match, code-graded and model-graded checks. Use a second model as a judge only for what code cannot measure, and calibrate it. OpenAI's guide says to "Use human feedback to calibrate automated scoring", and [Eugene Yan's survey of task-specific evals](https://eugeneyan.com/writing/evals/) calls human evaluation "the gold standard (for now)" for complex tasks.

How many cases? Anthropic says to "Prioritize volume over quality", meaning more cases with an imperfect automatic grader beat a few hand-graded ones. There is no magic number, but I would worry below a few hundred cases for a job where a wrong answer costs something. Hamel Husain writes that [unsuccessful AI products](https://hamel.dev/blog/posts/evals/) "almost always share a common root cause: a failure to create robust evaluation systems". Print the misses where you can read them.

## Write the pass marks before you run anything

A pass mark is a number you commit to before you see a result. Anthropic's guide says most use cases need "multidimensional evaluation along several success criteria", with latency and price among them. When a model decides something, I would add an outcome for "send it to a person", because a model that may abstain is safer.

My listing review had three marks, set before the first run. It had to agree with at least 95% of the listings a person approved, send fewer than 20% of all listings to a person and approve none of the unsafe ones. The first run missed. The misses showed that my past reviewers had approved any AI-powered product while my prompt asked for an AI agent. The labels and the prompt measured two different rules, and no threshold could fix that, so I wrote down the rule we followed in practice, widened the prompt and ran again.

The second run still missed, at 83.8% against 95%. Some misses were sites the fetcher could reach but not read, so those now go to a person. I then moved two marks to the measured figures, at least 92% agreement and under 30% sent to a person, and left the safety mark at zero. At those marks the model approves 92.0% of the person-approved listings it decides alone, sends 27.2% to a person and approves none of the 27 unsafe listings. A mark may move only with a written reason, and the safety mark does not move. Eugene Yan puts the general point well. "different use cases come with different levels of risk", so "our evaluation standards should be calibrated accordingly".

## Filter by latency and data rules before you compare quality

Some candidates should never reach the quality run. Latency is one filter. A background job can be slow and thorough, while a model inside a form that a person is waiting on needs a budget in milliseconds. Write the budget as a pass mark and measure it on your own traffic, because a vendor's speed figure does not include your prompt length or your queue.

Data rules are the stronger filter. On one platform I ran, the models behind the AI co-pilot were served in-region with zero data retention, and customer document processing ran on a different managed model platform, so each workload used the platform that fit its data rules. A model that wins on quality but cannot be used for that data has won nothing.

Do not assume one data setting covers every model. OpenAI's page on [your data](https://developers.openai.com/api/docs/guides/your-data) says its zero data retention controls "are subject to prior approval" and that endpoints it lists as ineligible "may retain application state when used, even if you have Zero Data Retention enabled". Read the eligibility for each model and endpoint you test. Availability is a filter too. The judging endpoint my listing review uses refused the two larger models I tried, so the small model was the only one that could run.

The [NIST AI Risk Management Framework](https://www.nist.gov/itl/ai-risk-management-framework) is voluntary and exists "to incorporate trustworthiness considerations into the design, development, use, and evaluation" of AI systems, which makes it a good prompt for what to record about each choice. On that platform I built an audit trail that records each prompt, retrieval and tool call with its user and tenant, so "which model saw this data" has an answer.

## Run the candidates and take the cheapest that passes

Run the same cases, with the same prompt and settings, through each candidate. Change one thing at a time, and write down any prompt change a model needed. Then apply the marks and nothing else. Of the models that pass, take the cheapest. Read the misses of the failures first, because a model's mistake and a bad label look the same in a score.

My page-rewrite job shows what comes out of this. A small model and a larger one each rewrote 20 approved listings. A nine-rule check scored each page, and I needed 18 of 20 to pass. The small model passed 16, so the larger model writes the pages, a second model takes over when a page fails the check, and a better prompt for the small model is the next job. Anthropic's guide describes this shape under "Combine models".

## One layer, so the model is a config value

A result is only useful if it can change production in one line. So every model call in my products goes through one provider-neutral layer. The shape I would aim for is that a feature asks for a job and config says which model and settings serve it. Run the check through the same layer, so what you measured is what ships.

This has a cost. The layer has to hide differences between providers, and features that exist on one provider are hard to reach through a common interface. You maintain an integration for each provider you test, and the evaluation set needs an owner. I accept that, because the alternative is provider code spread through feature code, where changing a model is a project and nobody does it.

## What I rejected

- **One large model for everything.** It is the simplest design and I would start there. In my experience it stays because moving off it is hard.
- **Choosing from a public benchmark.** OpenAI's best practices list "Overly generic metrics" as an anti-pattern. A leaderboard tests a model in general, and your job is specific.
- **Choosing by feel.** The same page lists "Vibe-based evals" as an anti-pattern.
- **A router that picks a model on every request.** It can pay off when one job has mixed traffic, but it needs its own check. I would pick a model per job first and add escalation on a failed check before adding anything smarter.

## When to run the check again

Treat the check as a test suite with triggers. A new model from a provider, the retirement of a model you depend on, any prompt change and any change to the rule the job follows are all triggers. OpenAI's model selection guide says "Whenever a new model comes out, try revisiting your previous settings." Its best practices ask for "continuous evaluation" that runs "on every change". The suite I built on that platform replayed its fixed questions before release on every prompt or model change.

An upgrade inside one model family is not safe either. Eugene Yan describes a team whose eval setup for intent classification caught a 10% performance drop when they moved to a newer version of the same model family. Add each real failure from production to the set.

## When the bigger model is the right call

Anthropic's guide describes two ways to start. You can begin with an efficient model and move up for a specific gap, or begin with the strongest model for complex work and move down. Both end at the cheapest model that passes.

Use the bigger model when nothing cheaper reaches the marks after a fair prompt change, or when a wrong answer is costly enough that the marks are tight. Use it also for open-ended work where you cannot write down one known answer, because there the check is weak and a pass proves little. My plan for the listing review said that if no threshold met the marks, every listing would still go to a person and the model would only sort the queue.

## A plan for the first week

1. List the model calls in your product and name each job.
2. For the riskiest job, collect every past decision or log you can, and remove personal data.
3. Write the pass marks, including latency, data rules and "send to a person", before you run anything.
4. Put that job's calls behind one function that reads the model from config.
5. Run two or three candidates through the set, read the misses and take the cheapest that passes.
6. Add a release step that runs the set on every prompt or model change.

For the setup I run this inside, including the spend cap, read [how I run my products with AI agents writing the code](/blogs/how-i-run-my-products-with-ai-agents). For the failure cases a model check does not cover, I wrote about [testing AI agent guardrails](/blogs/how-to-test-guardrails-for-ai-agents).

The best result is a boring one. The model for each job is written in a file, and a number says why.

## Frequently asked questions

### How many test cases do I need to compare language models?

As many real cases as you can collect. Anthropic's guide says to prioritize volume over quality. I measured the listing review on 761 past decisions. With a few dozen cases, treat the result as a warning and not as proof.

### Should I use a model to grade the answers?

Use code checks first, because they are exact and cheap. Use a model as a judge for what code cannot measure, calibrate it against human grades on a sample, hide the model names and randomize the order of the pairs.

### Should I use one large model or a different model for each task?

Start with one capable model, because it gives you a baseline. Move a job to a cheaper model only when the check shows it passes that job's marks. Each extra model adds an integration and tests to maintain.
