I learned during my Meta hackathon build that building a product involves repeatedly realizing I am solving the wrong problem.
Initially, I told the model exactly what to extract from an image.
This reflection came from my Meta hackathon build, which also involved hackathon app frustration. The post does not specify what the model extracted or what I changed afterward.
I found that the model looked for the exact words I supplied, not necessarily other items with the same meaning; adding synonyms helped in some cases but raised the question of how many synonyms I would keep adding.
I described the goal as teaching the model what to eat, instead of teaching it how to use a spoon and eat on its own, because I’m not going to sit next to every user and explain their input to the app.
I recognized that the approach needed to change: People will upload normal, weird images. They won't know which label gets the best result or which part confused the model.
My proposed self-directed extraction method was to identify the extractable elements in an image, name or tag them, locate them, and then extract them; those extracted elements could then be fed into the image generation model.
I am still testing this approach; no completed evaluation or outcome is given.
I am concerned that a demo that works only when I keep feeding it the answers may not survive someone else trying it.
I see this as a task-framing problem, not simply a need for a better prompt.