All writing

11 min readBy EmtifIrrisVision

Checking a procedure with a phone camera

A vision model that can describe a picture is easy to find. One you can trust to sign off a step needs small questions, a bit of geometry and the nerve to say "I can't see".

Ninety seconds.

That is how long our first live camera check took to read one number off one display. Then it timed out and gave us nothing at all.

The setup was simple. A phone pointed at a controller. A step in a procedure that said, more or less, "check the temperature". A small vision model running on our own server, asked to look at the frame and tell us what the display said.

It thought about it for a minute and a half. Then it gave up.

This is the story of how that same check came to answer "Display shows 182.4" in a few seconds, and why, once it did, we decided we still couldn't trust it with the harder half of the job.

Some vocab first

A few words will come up a lot, so here they are in plain terms.

Irris is our product for factories and labs. It watches someone work through a written procedure on a phone or tablet camera and says, step by step, whether each one was done.

A vision model (people also say vision-language model, or VLM) takes an image and some text and answers in text. You show it a picture, ask "what's on the display?", and it writes back.

A token is a chunk of text or image that the model reads or writes. Speed is usually measured in tokens per second. More is better.

An analogue dial has a needle. A digital display has numbers. That difference turns out to matter far more than we expected.

OCR is optical character recognition: software that reads printed or displayed characters out of an image. It is old, boring and very good at its job.

The first look

The idea behind Irris is easy to say. A procedure is a list of steps. Many of those steps leave something visible behind: a tag on an isolator, a clamp closed, a reading on a display. If a camera can see that evidence, it can tell you whether the step was done, and write it into the batch record for you.

So the first thing we built was a "look". Take the current frame, ask the model a question about it, get an answer back.

On a fast machine, it worked. On the server it would actually run on, a single look took more than ninety seconds and then hit the timeout.

Nobody waits ninety seconds for a step check. You'd have moved on to the next step, and the one after that, while the camera was still thinking about the first.

Ten tokens a second

So we did the thing we always do when something is slow and we don't know why. We stopped guessing and read the logs.

The model was getting through roughly ten tokens a second. An image becomes hundreds of tokens before the model even starts on the question, so at that rate the answer was never going to arrive in time.

The model was fine. The server was fine. The trouble was how they were set up together: the model was fighting the rest of our own software for the processor, and losing. That half of the story has its own post, Right-sizing AI, because it deserves one and because it is the kind of bug that will happen to you too.

The short version: once every engine on the box stopped grabbing every core, the same look on the same server answered "Display shows 182.4" in 2.6 to 4.2 seconds.

Fast was the easy part.

For about a day that felt like a victory lap. Then we started testing dials.

Dials are hard

Here is something we didn't know when we started, and which you might not either.

Vision models are surprisingly bad at reading instruments with needles.

The clearest evidence we've found is MeasureBench, a benchmark published at CVPR 2026 that tests vision-language models on reading measuring instruments. On real photographs of analogue dials, the best model they tested got about 30% right. On digital displays, models did much better, around 80%. Small models, in the 2 to 3 billion parameter range that we can afford to run on a CPU, managed about 10% on dials.

Ten percent.

The failure is a sneaky one, too. The authors note that models usually get the unit right and the value wrong. So the answer looks plausible. It says "°C" in the right place. The number is just made up.

If you are writing a batch record, a confidently wrong temperature is worse than no temperature at all. A blank gets noticed. A plausible number gets signed.

A model that guesses well is still guessing.

A ruler for the needle

So we stopped asking the model to read dials.

Reading a dial is geometry. There is a centre, a needle, and a scale that runs from one value to another around an arc. If you know where the centre is and where the two ends of the scale are, the needle's angle tells you the value. No intelligence required, just a protractor.

In Irris, each analogue instrument gets calibrated once. Someone taps three points on a still image: the centre of the dial, the start of the scale and the end of the scale. They type in the values at the two ends. That's it.

From then on, finding the needle and turning its angle into a number is ordinary image processing. In our test, the geometry read 181.2 where the true value was 182. That is well under half a percent of the scale, from a method that has no opinions about anything.

Digital displays get a different treatment, because there a model or an OCR engine really can read the characters. The trick is to not believe any single reading:

  • the text has to match the format the step expects, for example three digits and one decimal place
  • two out of three recent frames have to agree
  • any value close to a limit asks a person to confirm before it goes into the record

Each of those is a small, dull rule. Together they turn "the model said 182.4" into "we saw 182.4 on three frames, in the right format, comfortably inside the range".

Small questions

The other early mistake was the question itself.

Our first version asked something like "is step 4 done correctly?". It is a natural thing to ask. It is also a terrible question for a small model, because "correctly" hides a dozen things the model has to work out on its own, and when it gets one wrong you can't tell which.

Now each step carries two to four small questions about things you can actually see in the frame:

  • Is a red tag hanging on the isolator?
  • Are all four clamps closed?
  • Is the controller display readable?

Each question is marked as required or optional, and the decision is plain code. Every required answer is yes: the step passes. Any required answer is no: it fails. Anything else means look again, or ask the person.

Each question is asked over the last few frames, more than once, and the majority wins. A single flickery answer doesn't decide anything.

We were relieved to find research pointing the same way. In one study of procedural mistake detection, a 7B open model that asked itself well-chosen yes or no questions beat GPT-4o on the same task, 67.8% against 55.4% (Transparent and Coherent Procedural Mistake Detection).

Ask a small question and you can check a small answer.

Refusing to look

Some frames can't answer any question.

The camera is pointing at the ceiling. Someone's hand is over the valve. The light is behind the instrument. The phone moved and everything is a smear.

A model will still answer. That is the uncomfortable thing about them: they rarely say "I don't know" on their own. One study of video question answering found that a model's self-reported confidence barely moved even when it was shown far fewer frames (Explicit abstention knobs for video QA). Less evidence, same certainty.

So before any model is called, Irris checks the frame itself. Is it sharp enough? Bright enough? Is there glare? Are hands covering the part that matters? Is the thing the step expects actually in view?

If not, it doesn't guess. It tells you what to change, something like "I can't see the valve. Tilt the camera so it's in view."

This check is cheap. It runs in a fraction of the time of a model call, and it removes a large share of wrong answers before they happen. In a lot of those wrong answers, the model had been shown nothing and asked for something.

Seen twice

One last rule, and it is the one we'd keep if we could only keep one.

A step only counts as done when it has been seen twice: two passing checks a few seconds apart. Or when a person passes it by hand, which is always allowed.

The model can propose that a step is done. Code decides whether that is allowed. A step can't complete before the steps it depends on. A finished step is never reopened by the model, only by a person.

That split, where the model suggests and code decides, turned out to have support in the research, with numbers behind it.

What the research says

Two recent papers made us feel less alone.

The first, RegulAR, looks at exactly this kind of procedure checking. When the authors let code validate what the model proposed, instead of taking the model's word for it, false "missed step" alarms dropped from 45 to 5 (RegulAR). That is the difference between a system people mute and a system people listen to.

The second, TRAFA, asks a different question: what if the system warned you before the mistake, instead of after? Warning people ahead of the step raised task success from 15% to 65% in their study (TRAFA).

We find that second number the most interesting of everything here. Catching a mistake is useful. Not making it is better. Irris already shows the next step and what it expects to see. Getting the warning in early, at the right moment, is where we think the next real gains are.

Catching mistakes is good. Preventing them is better.

Arguing against ourselves

It would be dishonest to stop there, so here is the case against what we've built.

All this machinery is a lot of work per instrument. Calibrating dials, writing questions for each step, setting formats and limits. A big general model that "just looks" needs far less setup. If your procedures change every week, that setup cost is real, and it lands on the people who know the process best and have the least spare time.

Models are also getting better, quickly. The 30% figure on dials will not stay at 30%. Some of our rules may look fussy in a year or two.

And there are things our approach is bad at. Anything that can't be reduced to a visible yes or no is a poor fit. "Was the operator careful?" is not a question a camera should be answering, and we don't try.

Our answer, for now, goes like this. Setup happens once, and the person doing it is the one who already knows what "done" looks like. Rules can be loosened as models improve, while trust is very hard to win back after a system has signed off a wrong number. And where a step really needs judgement, the right answer is a person, with the camera as a second pair of eyes.

We might have the balance wrong. We'd rather be wrong in the cautious direction.

If you're building one

If you are putting a camera on a procedure, here is what we'd tell you over a coffee.

  1. Measure speed on the hardware you will actually run on, early. Ours was fine everywhere except where it mattered.
  2. Don't ask a language model to read a needle. Calibrate once and use geometry.
  3. For digital displays, check the format and wait for agreement across frames.
  4. Break every check into two to four questions you could answer by looking.
  5. Check the frame before you check the step, and say what to fix.
  6. Let the model propose and let code decide. Require a step to be seen twice.
  7. Put a person in charge of anything close to a limit.

For the curious, here is what a step looks like in Irris:

{
  "instruction": "Check the platen temperature",
  "kind": "measurement",
  "instrument_id": "platen-thermometer",
  "reading": { "min": 175, "max": 185, "unit": "C" },
  "questions": [
    { "q": "Is the controller display readable?", "must": true }
  ],
  "severity": "major"
}

None of it is clever.

Back to ninety seconds

The first look took ninety seconds and returned nothing. At the time it felt like the model had failed us.

Looking back, the model was the most capable part of the system and the least trustworthy one. Everything we added afterwards (small questions, a vote, a check on the frame, a ruler for the needle, the rule about seeing things twice) was there to give it less to guess about.

The model still does the part only a model can do: looking at a messy, real picture and answering a plain question about it.

We just stopped asking it to do the rest.

If you're working on something similar, we'd like to hear about it.

Sources