Ninety seconds to read a number off a screen.
That was the first live test of the camera checks in Irris, our product for factories and labs. A phone pointed at a controller display, a small vision model on our own server, and a simple question: what does the display say?
The answer never came. After ninety seconds the request timed out.
The same model on another machine answered in well under a minute. Same model, same question, same picture. Something about this particular server was making it crawl.
This post is about what that something was, which turned out to be two bugs, not one, and about the bigger question it pushed us to answer properly: when should you run a small model on your own hardware, and when should you pay for something bigger?
Some vocab first
This one gets a little technical, so here are the words in plain terms.
A CPU core is one worker inside the processor. A server might have dozens.
A thread is a line of work that a program asks a core to do. A program can start as many threads as it likes, but only one thread runs on a core at a time.
A container is a sealed box a program runs in. The box can be given a share of the machine, say 24 cores out of 48. That share is its CPU quota.
Oversubscription is when programs start more threads than there are cores to run them. The threads take turns, and a lot of time goes on swapping between them instead of doing the work.
A token is a chunk of text or image a model reads or writes. Speed is measured in tokens per second.
A GPU is a graphics processor. It has thousands of small workers that suit the maths models do, so the same model can run many times faster on one.
The slow look
So we did the thing we always do when something is slow and we don't know why. We stopped guessing and read the logs.
The model was working through its input at about ten tokens a second. A single image becomes hundreds of tokens before the model even gets to the question, so the arithmetic was brutal. The answer was never going to arrive before the timeout.
Ten tokens a second is far below what that model should manage on a server that size. When a model is that much slower than it should be, the cause is usually something around it.
Forty-eight chairs
The first thing we found was a counting mistake.
Our container had been given 24 cores. The model server, when it started, asked the system how many cores there were. The system told it about the whole host machine: 48.
So the model started 48 threads, one for each core it thought it had.
Picture a room with 24 chairs and 48 people who each need to sit down to do any work. At any moment, half of them are standing, waiting. Worse, each time someone stands up and someone else sits down, a little time is lost in the shuffle. Multiply that by thousands of shuffles a second and the room spends more time shuffling than working.
That's oversubscription. With twice as many threads as cores, every thread spends a lot of its time waiting for the others.
More threads than cores is slower than fewer.
Everyone else wanted the cores too
That alone would have hurt. It wasn't alone.
The same server was running other engines alongside the vision model: speech-to-text for voice commands, OCR for reading printed text, and a search engine for looking things up in documents. Each of those also defaulted to using every core it could see.
And several of them didn't just use the cores when they had work. They spun.
Spinning is a trick some maths libraries use to react quickly. Instead of going to sleep when there's nothing to do, a thread stays awake in a tight loop, checking over and over whether new work has arrived. It saves a few microseconds when work does arrive. It also keeps a core completely busy doing nothing.
So the picture inside our container was this. The model had started twice as many threads as it had cores. Three other engines had each started threads for every core as well. And several of those threads were awake and looping even when idle.
The model was standing in a very crowded room.
The fix
The fix had three parts, all of them small.
First, read the real quota. On Linux, a container's CPU quota is written in a file, and you can read it at startup instead of trusting the core count the system reports:
CPUS=$(awk '$1 != "max" { print int(($1 + $2 - 1) / $2) }' /sys/fs/cgroup/cpu.max)
On our server that gives 24, the true number, not 48.
Second, pin the model to a sensible share of that. We gave it 16 threads, leaving room for everything else on the box.
Third, cap the other engines at two threads each, and tell them to sleep when idle instead of spinning. (For libraries built on OpenMP, the spinning behaviour is governed by a setting called OMP_WAIT_POLICY. Other libraries have their own switches. It is worth checking each one.)
After those three changes, the same look on the same server answered "Display shows 182.4" in 2.6 to 4.2 seconds.
From over ninety seconds to under five, without changing the model, the server or the question.
Most "slow AI" is slow plumbing.
The model that thought too hard
Fast, though, still came back empty some of the time. That was the second bug, and it was a funny one.
The small model we use can "think" before it answers: it writes out some reasoning first, then the answer. That helps on hard questions. But every call also has an answer budget, a maximum number of tokens it is allowed to write.
For short questions we'd set that budget low, because "is the clamp closed?" needs one word back. The model, given a simple question and a small budget, spent the entire budget thinking. It ran out before it got to the answer. So it returned nothing.
It's a bit like asking someone for the time and watching them use the whole conversation to explain how clocks work.
Short, structured questions don't need thinking. So we turned thinking off for them. Longer, open questions still get it. The empty answers went away.
Speech, freed
There was a pleasant side effect.
Speech-to-text, the engine that turns a spoken command into text, had been taking about twelve seconds per request. We had assumed that was just how long it took on a CPU.
Once it stopped fighting the vision model and the others for cores, it dropped to about two seconds. Nothing about speech had been fixed directly. It had simply been waiting in the same crowded room.
When everything is slow, look at what they share.
When a GPU earns its keep
All of that was on a CPU. It's fair to ask why we didn't just use a GPU.
Sometimes we do. A GPU makes a real difference for longer looks: a fuller description of a scene that took around thirty seconds on the CPU came back in about two on a GPU. If people are standing waiting on the answer, that gap matters.
But a GPU costs money all the time it's switched on, whether or not anyone is using it. So it's worth doing the sums.
At a rental price of about $0.20 an hour, a GPU that is always on costs roughly $146 a month (0.20 times the 730 hours in an average month). That's reasonable if it's busy. It's a waste if it sits idle most of the day.
If you rent by the second and switch it on only when there's work, the bill follows the work instead. Say the GPU is genuinely busy for two hours a day: that's about 60 hours a month, or around $12. The catch is start-up time. A machine that has to wake up first adds seconds, sometimes more, to the first request.
So the systems we build try the fast option first and fall back automatically. GPU if it's available, CPU if not, and the CPU path is tuned well enough to be useful on its own. Switching the GPU off slows things down. It doesn't break them.
Where the energy goes
There is another number we care about: how much energy each answer took.
A small model on a few CPU cores uses a fraction of the power of a large model on a rack of GPUs. That's obvious once said. It is also easy to lose sight of, because hosted models hide their energy use behind a price per token.
We now report two numbers for every deployment: the outcome we agreed with the client at the start, and the energy and cost per task. When the second number is visible, it becomes much easier to argue for the smaller model, and to notice when a bigger one has crept into a job that doesn't need it.
We'd be overclaiming if we said we could measure the energy of a hosted model call precisely. We can't. What we can do is measure what runs on hardware we control, and choose the smallest thing that does the job.
Where small is enough
We're not alone in thinking small models are underused.
A position paper by researchers at NVIDIA, Small Language Models are the Future of Agentic AI, argues that much of what AI agents do is narrow and repetitive: pull fields out of a document, decide which tool to call, format a reply. For that kind of work, they argue, small models are capable enough, and they estimate that serving a 7 billion parameter model costs 10 to 30 times less than serving one with 70 to 175 billion.
That matches what we see. Our rough rule:
- Small model on a CPU for well-defined jobs: reading a display, routing a request, filling in a form, yes or no checks. Cheap, private, a few seconds.
- Small or medium model on a GPU when people are waiting and seconds matter.
- Large hosted model for open-ended reasoning, long documents and writing that has to be good, with the small model kept as a fallback for when the big one is unavailable.
Use the smallest model that does the job, and measure whether it does.
Arguing against ourselves
There's a fair case against all this.
Engineering time costs money too. The days we spent finding thread counts and spinning libraries could have paid for a lot of hosted model calls. For a small team with a small workload, "just use the big API" is often the right call, and we'd tell you so.
Hosted models also keep getting cheaper and faster. Some of the gap we're describing will close on its own.
And small models fail in ways that are harder to spot. A big model that doesn't know something will often say so. A small model is more likely to answer confidently and wrongly, which is why our camera checks surround the model with rules and votes, and why we wrote a whole post about that.
So where does that leave us? Somewhere unexciting. Small models on your own hardware are the right answer more often than most people assume, and the wrong answer more often than their fans admit. The only honest way to choose is to try both on your real task and measure.
A short checklist
If you're running models on your own hardware, these are the things we now check first:
- Read the container's real CPU quota at startup. Don't trust the core count.
- Give the model a fixed number of threads, below the quota.
- Cap every other engine on the box, and stop idle threads spinning.
- Turn off "thinking" for short, structured answers.
- Time things on the hardware you'll deploy on, not your laptop.
- Do the GPU sums: always on, or billed by the second?
- Report energy and cost per task next to the outcome.
Back to ninety seconds
Ninety seconds looked like a verdict on small models. It was a verdict on our setup. A good model had been put in a crowded room and asked to work fast.
Once it had room, it read the display in about four seconds. Speech went from twelve to two. Nobody bought anything.
Sometimes the right-sized model is the one you already had.
If you want help working out what size your AI should be, we're happy to talk.
Sources
- Peter Belcak et al. Small Language Models are the Future of Agentic AI. NVIDIA Research, arXiv 2506.02153.
- The Linux kernel documentation. Control Group v2, the
cpu.maxinterface. - OpenMP Architecture Review Board. The OpenMP API specification, the
OMP_WAIT_POLICYenvironment variable. - Emtif. Checking a procedure with a phone camera.