While building Credential Sentinel, an agent that rotates the credentials nobody is tracking, the part I kept circling back to was the order of operations during a cutover.
The tempting version is simple. Swap in the new credential, revoke the old one, done. It feels clean. It is also how you take down production at 2am, because the second the old credential is gone you have no way back if the new one turns out to be broken.
So the agent does it in a deliberately boring order instead. Promote the new credential, repoint everything to it, verify it actually works, and only then revoke the old one. The old credential stays valid as a safety net the whole time. If verification fails, the agent rolls back to it and pulls in a human rather than leaving anything stranded halfway.
The takeaway: never decommission the old thing until the new thing has proven itself in production. It costs you a little overlap and one extra step in the sequence. It buys you a way back, which is the only thing that matters on the day something goes wrong.