One photo of a vending machine instead of checking shelves by eye
For a vending operator: a technician photographs the machine and the system checks, slot by slot, whether the right product is where the plan says it should be.
See how the system checks one machine
The demo plays by itself; click any step to go back to it.
A technician takes one photo of the machine through the glass and gives the machine number. The shelf plan and the product photos come from the operator’s systems.
- Photo and machine number
- Plan: 4 shelves, 24 slots
- Reference photos of 12 products
AI finds every product on the shelves and cuts it out along its outline. A glare on the glass on its own is not counted as a product.
- 23 products found
- Glare on the glass dropped
Each cut-out product is compared with the reference photos of this machine’s products. When the front of a product cannot be seen, because a glare covers it or the product faces backwards or sideways, the system says “don’t know” instead of guessing.
- 20 recognised for sure
- 3 times “don’t know”
Each product goes to its slot and is checked against the plan. Every slot gets a verdict: correct, wrong product, empty or don’t know.
- 19 slots correct
- B4: wrong product
- C5: empty
The operator sees at once that the machine needs a visit and why. Nobody compares shelves by eye.
photo_M-214.jpg
- MachineM-214from the technician
- Shelf plan4 shelves × 6 slotsfrom the operator’s system
Reference photos12 productsfrom the operator’s system
A new machine only needs its plan and product photos, no model training.
Products found23
Each product is cut out along its outline, so the background and the spiral do not disturb the comparison.
- Recognised for sure20
- “Don’t know”3
An answer counts as sure only when the best reference clearly beats the next one.
- correct 19
- wrong product 1
- empty 1
- don’t know 3
Recognition looks only at the photo, so the wrong stock cannot hide behind the plan.
Done
- slots checked
- 24
- need a visit
- 2
- “don’t know”
- 3
- needs a visitB4: swap for sparkling water
- needs a visitC5: restock wafers
- don’t knowB6, C3, D4: a glance at the photo is enough
No false alarms: nobody drives out to a machine that is fine.
A walkthrough on a fictional machine and products. Numbers in the demo are illustrative.
What it does
One photo is enough to check a machine against its plan, without comparing shelves by eye. The system names a product only when it is sure, and otherwise says “not sure”, because a false alarm costs a technician’s trip.
In numbers
- wrong product recognitions; the simple approach got 11 of 78 wrong
- 0
- On the test photos the system was not wrong once. A simple method that always picks the most similar product was wrong 11 times out of 78 on the same photos.
- products recognised automatically, the rest marked “not sure”
- 74%
- Share of products on the test photos that got a confident answer. The system marks the rest as unsure instead of guessing.
- photo per machine is all the technician provides
- 1
- The technician sends one photo of the shelves and the machine number. The plan and product photos come from the operator's systems.
The problem
- Every vending machine has a plan of which product belongs in which slot. Checking that a machine really looks like that meant comparing the shelves with the plan by eye, machine by machine. That does not scale to a whole network.
- Each machine carries a different mix of products, the mix changes, and flavours of the same drink come in nearly identical packaging. The photo is taken through glass, in a dark cabinet, with products tilted on their spirals. And there were no labelled photos to train a model on.
Decisions that shaped it
Look up, do not train
There were no labelled photos, and every machine stocks something different. Instead of training a model, the system compares each product on the photo with images of the products that should be in this machine. For the operator, adding a machine or a product means adding a picture and a plan, not starting a machine learning project.
A false alarm costs more than a missing answer
A wrong verdict sends a technician out for nothing, and after a few of those people stop trusting the alerts. So the rule from the first week was: no errors first, coverage second. Every variant that added even one wrong answer was rejected, and “not sure” counts as a legitimate result.
The plan is not allowed to help
It is tempting to settle a close call in favour of the product that is supposed to be in the slot. Then the system would always confirm the plan and never catch the wrong product, which is the one thing it is for. Recognition looks only at the photo; the plan comes in afterwards.
A second opinion, and colour only as a tie-breaker
Adding colour to the similarity score looked like an easy win and was tried several times. Each version either made mistakes on the second machine or recognised fewer products. Now colour decides only between candidates the image model already considers equally likely, and a second model is asked only when the first one holds back.
These rules were worked out together with the client, to fit how they work. In your company they may look different, and I will fit the system to the way you work.
Built with
- Amazon SageMaker
- AWS Step Functions
- AWS Lambda
- S3
- DynamoDB
- SQS
- SAM 3
- DINOv3
- SigLIP
- PyTorch
- Terraform
- Python