I ran one of my own agents 8 times with the same model, same task, and same code; pass@8: 28%; pass^8: 4%. I described the result as “One demo, one in twenty-five real runs.”
I said the gap between pass@8: 28% and pass^8: 4% is why I do this. I argued that a single clean run does not show whether an agent keeps working, and that continued reliability is what users experience.