The scariest failure mode in a distributed system is not a request that fails loudly. It is a request that fails silently and gets retried. Two shipments booked with one merchant intent. Two payouts against one order. Two webhooks that spawn two records that will confuse a support agent forever.

The habit

Before it is a header, idempotency is a habit. Every write endpoint asks itself: if I get called twice with the same intent, do I still produce one outcome? That question drives table design (natural keys, unique constraints on intent, not on internal id), retry policies, and the very shape of the API you expose.

Only after you have built that habit does adding an Idempotency-Key header on the wire start to earn its keep. The header is a way to make the promise explicit to your caller; the habit is what makes the promise true.

Where teams typically slip

The most common miss is treating idempotency as a queue-worker concern only. It has to hold at the ingress — the moment a webhook lands, before it fans out to jobs, before it touches state. Otherwise a slow ack turns into a duplicate record two workers deep.