I spent a while building an evaluation suite for Credential Sentinel: fifty hand-written test cases scored on routing, safety, faithfulness, and a few other things. The whole point was to stop trusting my gut about whether the agent was actually getting better.
The lesson landed when I made a small change to a prompt to fix one weakness. The change worked. It also quietly broke something else. A test case with a malicious instruction buried in it, the kind the agent had been correctly ignoring, suddenly started obeying it. My fix had introduced a security regression I would never have spotted by eye.
The only reason I caught it was that I ran the exact same fifty cases before and after the change and laid the two runs side by side. The broken metric had nowhere to hide.
The takeaway: a single ‘after’ number tells you almost nothing. Improvement only means something measured against a baseline you scored the same way. An eval is not really about proving your thing works. It is about catching the thing you broke without noticing.