Food Detection — On-Device Nutrition Estimation
We worked with a client in the food and nutrition space to let users scan roughly a million foods, assigning each item an accurate nutritional value. A basic image classifier can name the items on a plate and pull approximate nutrition figures from a database — but "approximate" is the weak point, since the same dish can carry very different calories depending on how much is actually there. We closed that gap by estimating the volume of each individual food item from photos, so the nutritional value reflects the real quantity on the plate rather than a database average. The work ran from 2022 to 2024 and shipped across both web and mobile platforms.
Volume-aware nutrition estimation
I oversaw the project end to end and was an individual contributor on the volume estimation model itself. The system first identifies the distinct food items in a photo and detects the exact boundary of each one using an object-segmentation model fine-tuned on a private dataset, then combines multiple pictures taken from different angles to reconstruct and estimate the volume of each item. That volume, mapped back to the nutrition database, is what makes the final assessment meaningfully more accurate than a plate-level guess.
Making it fast and cheap
Getting this to run reliably — and cheaply — took most of the engineering effort. We built a robust data pipeline with custom augmentations to make the segmentation models hold up across the messiness of real-world food photos, and implemented custom prediction pipelines tuned for high inference speed. Rather than leave the expensive post-processing that object-detection models require — non-max suppression, mask scaling, and similar steps — outside the model, we folded those operations into the model graph itself, trimming inference time further.
Optimizing the runtime with ONNX and TensorRT cut model latency by 2.5×. Finally, we invested in model compression so the whole thing could run directly on a phone: deploying on-device kept latency low, worked offline, and largely eliminated the need for costly cloud hosting on mobile. For the web platform, the same models were served through a Kubernetes-based deployment that scaled with demand.