Vision models scored on real food-photo calorie estimation. Each model runs the production prompt over labelled images; we measure accuracy vs the ground truth. Higher score is better.
FuelBench measures vision LLMs on the one task that matters for Fuel: estimating calories and macros from a single food photo. Every model runs the exact production prompt over the same labelled images, and we score how close its estimate lands to the truth — not how "smart" it sounds on paper. Each model is run several times per image, so we also track how much the same photo wobbles run-to-run.
FuelBench keeps historical suites separate from the current benchmark:
For every call we log calorie error, the share within 20% / 30% of truth, macro error, internal coherence (do the macros add up to the calories?), run-to-run consistency, latency, token usage including reasoning tokens, and the real cost billed by the gateway. The headline score is the share of scans within 20% of the true calories (higher is better), plotted against cost on a log axis. The dashed quadrant lines run through our production model, Gemini 3.5 Flash Lite — so the shaded upper-left zone means "more accurate and cheaper than what we ship today".
Why build our own? Public leaderboards such as Artificial Analysis score text reasoning, which does not predict food-vision quality. Across 13 models we found the "smarter", more expensive reasoning models were consistently slower and no more accurate — often worse — at reading a plate, while burning far more tokens. FuelBench measures the real thing: our images, our prompt, real cost and latency. Results are appended to an immutable log and every score is timestamped, so the index grows over time.