My brother-in-law told me to stop trusting the model's confidence score and he was dead right

Sean Anderson · 2026-10-20T21:00:04.432161+00:00

Back in March I told my brother-in-law, who does ML work at a bank in Charlotte, that I was picking between two image classifiers by looking at which one came back with higher confidence numbers, and he said that number is mostly just the model being loud, not being correct. I ignored him for two weeks until my setup with the 0.94 scores started failing on cloudy outdoor shots while the lower scoring one held up fine. So now I want to know, how do you all actually check if a model is calibrated without spending a week on it?

🐿️Pinecone Attack

Loading…

My brother-in-law told me to stop trusting the model's confidence score and he was dead right | AI Innovations - Pinecone Attack