I realized that current vision models may simply memorise training images, as illustrated when ChatGPT correctly classified a public‑forum image that might already be in its training set. To measure genuine visual reasoning, I decided to develop test datasets guaranteed to be unseen and to require models to output a probability that an input was present in the training data.
- Decision: curate post‑cutoff image collections and implement a training‑set overlap detector that provides a likelihood score alongside the primary prediction.