UpDish

Overview

UpDish is a native iOS app for reviewing how a meal is divided across visible food components and comparing that balance with Indonesia's Isi Piringku guideline.

A user can take or select a photo, review the components and proportions detected from it, correct anything that looks wrong, and continue to an evaluation. Completed evaluations are stored locally so they can be revisited later.

Why we started looking at meal preparation

At Apple Developer Academy, our team started from a broad challenge: Help Others.

We chose to look at mothers who regularly prepare food for their families. We did not start with a meal-scanning app or even with the idea of using AI. We started by looking at how they thought about the meals they were already preparing.

Finding the problem

Early on, we explored ideas around calories, protein, and more precise nutritional measurements.

Those felt useful because they gave us concrete numbers to work with. The problem was that getting those numbers would often require more from the user: weighing food, entering quantities, or trusting estimates that were difficult to verify. From the people we spoke with, that direction started to feel like we were adding another task to something they already did every day.

We also noticed that a meal could feel balanced simply because the expected food groups were present. Rice, chicken, and vegetables might all be on the plate, for example, even when rice occupied most of it and the vegetables only made up a small part. That made proportion more interesting to us than simply checking whether a food group existed.

While looking into that, we learned about Isi Piringku, which gave us a way to talk about the composition of a meal without trying to calculate exact nutrition from a photo.

The question became much narrower:

Could we help someone review the visible proportions of a meal without asking them to measure everything themselves?

We tested that direction before committing to the full product flow. The concept gradually moved away from detailed nutritional tracking and toward something simpler: take a photo, review what is visibly on the plate, and use those proportions as the basis for an evaluation.

That narrowing ended up shaping most of the engineering decisions that followed.

Choosing the detection approach

I explored several ways the app could turn a meal photo into components and proportions before settling on the current approach.

Traditional image classifiers were not a great fit because they usually answer a different question: what dish is this? UpDish needed to identify several visible components from the same image.

Object detection and segmentation were also interesting. Segmentation in particular could give a more direct estimate of visible portion size by measuring the area of each region in the image.

Recognition was the harder part. Most of the pretrained food models I found used a fixed set of classes, and many of those classes did not map well to the variety of Indonesian meals we wanted the app to handle. We also did not have a labeled dataset of our own to train around that gap.

A vision-language model had almost the opposite trade-off. It was much more flexible in what it could recognize and name, including mixed dishes and foods outside a predefined class list, but its portion estimates would still be estimates rather than measured areas.

For this version of UpDish, that trade-off made sense because the detection only needed to produce a useful first draft. The user would still review it before anything was evaluated. I chose a cloud vision-language model, using Gemini through Firebase AI Logic, instead of introducing a separate inference backend or training a model specifically for the app.

The more important question then became how much responsibility to give it.

Constraining the model output

I did not want the rest of the application to depend on interpreting arbitrary generated text, so the model is given a narrow job: identify food that is actually visible and estimate the relative proportion of each visible component.

It is explicitly told not to infer things such as:

• weight,
• calories,
• nutritional values,
• hidden ingredients,
• or food that cannot be distinguished from the image.

The model is instructed to return integer percentages totaling 100%, and the response is constrained to a predictable structure containing a meal name and its components.

mealName

Nasi Padang

components[]

name	portionPercentage
Nasi putih	50%
Rendang sapi	30%
Daun singkong	20%

Constraining the response to known fields gives the rest of the app something predictable to work with. The UI can display and edit those values directly, while malformed responses can be rejected instead of interpreted as free-form text.

It also keeps the model’s role narrow: produce a draft of what is visible, then let the rest of the application decide what happens with it.

Treating the result as a draft

Structured output does not make the output correct. A component can still be misidentified, something can be missed, or a proportion can look wrong to someone who knows what was actually on the plate. Because of that, we treated detection as a draft rather than the final state.

The review screen lets the user rename components, adjust their proportions, remove incorrect components, or add something that was missed. Before continuing, the edited proportions must still total 100%, so correction is part of the normal flow rather than an exception.

The model makes the first pass. The user decides what is actually evaluated.

Keeping the evaluation deterministic

Once the components have been reviewed, the app maps them into the Isi Piringku categories and evaluates their proportions using application logic. The meal-detection model does not decide whether the meal is balanced.

That creates a useful split between two different kinds of work.

The model handles the uncertain question:

What food appears to be visible, and roughly how much of the meal does each component occupy?

The application handles the rule-based question:

Given the reviewed components and their Isi Piringku categories, how does this meal compare with the guideline?

1. Meal photo
2. Model draft
3. User-reviewed components
4. Isi Piringku rules
5. Evaluation

Once the components have been assigned to Isi Piringku categories, the evaluation itself is deterministic. The app sums the proportions within each category and applies the same rules to derive the result.

I prefer that boundary because the uncertain part of the problem benefits from a model, while the part that can be expressed clearly with application logic does not need the same uncertainty.

Building the analysis flow

My main engineering contribution was the path from choosing a meal photo to getting a reviewed result through the rest of the application.

The user can either capture a photo with the camera or choose one from the photo library. Both eventually enter the same detection flow. Getting that happy path working was fairly straightforward; most of the complexity appeared once the path could be interrupted.

A user can leave while detection is still running. The device can have no connection before analysis starts, or lose the connection while the request is already in progress. The model can return no recognizable food, or it can return something that does not match the expected structure. Those cases should not all become the same generic error.

The app handles them separately so it can distinguish between things such as:

• no internet before starting,
• connection loss during detection,
• task cancellation,
• no food detected,
• and an invalid model response.

Detection also sits behind a MealDetectionService abstraction rather than being tied directly to the Firebase implementation. That keeps the photo and navigation flow independent of the particular model provider and also makes it possible to use a mock implementation during development.

The AI call itself is only one part of the feature. Most of the application work is around deciding when it should run, what state the interface should be in while it runs, and what should happen when it does not finish normally.

Preserving reviewed results

Completed evaluations are stored locally with SwiftData. The saved record keeps the reviewed meal information and the evaluation needed to reconstruct the result later.

When an older entry is opened from history, the app uses that stored result instead of sending the original image through detection again. History represents what the user reviewed and accepted at that point in time; running the same image through the model again could produce a slightly different interpretation and turn an old evaluation into a new one. That is why the accepted result is what gets preserved.

What testing changed

Once the complete flow existed, testing shifted our attention away from whether the core idea worked and toward how understandable the product was when something did not go perfectly.

Across heuristic evaluation and usability testing, we surfaced issues around proportion validation, canceling an analysis while it was still running, feedback when there was no internet connection, navigation, wording, and accessibility. Some of those sounded small compared with meal detection, but they changed how complete the feature actually felt.

For example, preventing the user from continuing when the proportions do not add up to 100% only works if the interface also makes it clear what needs to be corrected. The same applies to detection: a long-running request needs a way to be canceled, and a failed request should tell the user something more useful than that an unknown error occurred.

Several of those issues were in the flow I was working on, and they changed the way I thought about the feature. The analysis was not finished just because the model returned the right data. Loading, cancellation, validation, failure states, navigation, and accessibility were part of the experience too.

Reflection

UpDish changed how I think about both product research and AI features.

On the product side, our first ideas assumed that more detailed information would be more useful. Calories, nutrients, and precise measurements sounded valuable, but the investigation pushed us toward a much smaller question that fit more naturally into how the people we spoke with were already preparing food. I would keep that narrower framing.

On the engineering side, I would also keep the separation between model interpretation, user review, and deterministic evaluation. The model is useful where the input is ambiguous, but I do not think it needs to control parts of the system that can be expressed more clearly with normal application logic.

What I would change is how I evaluate the detection itself. I spent time comparing different approaches before choosing a vision-language model, including classification, detection, and segmentation. For the version we were building, the VLM was the practical choice because it could recognize a wider range of food without requiring a dataset of our own, and its output was already designed to be corrected by the user.

I would validate that trade-off more systematically now. If I continued the project, I would build a small evaluation set covering different Indonesian meals, mixed dishes, lighting conditions, camera angles, and presentation styles. I would measure component detection and portion error instead of judging the model mostly from the examples we happened to encounter while developing the app.

I would also benchmark at least one segmentation-assisted approach. Segmentation may be weaker at naming unfamiliar food, but it could give a better signal for how much visible space each component actually occupies.

The main thing I took away from UpDish is that model output is more useful to me when I treat it as one input to a larger system, rather than the system itself.

The project also made me less likely to think of investigation, implementation, and testing as separate phases. Each of them changed what we thought the product needed.