How to rotate API secrets without downtime

Rotating an API secret shouldn't need both sides to deploy at the same minute. If it does, it won't happen, and the secret from three years ago stays in production.

The fix is an overlap: for a while, both the old and the new secret work. Callers switch on their own schedule, and the old secret stops working on yours.

How do you rotate a secret without downtime?

Let the old and the new secret both work for a set overlap, then expire the old one once no request uses it. Step by step:

1. Issue a new secret, and give the old one an expiry. Don't revoke anything yet. Pick an expiry that fits the caller's release cycle: a day for a team that deploys hourly, two weeks for one that ships monthly.
2. Deliver the new secret through a secret manager. Not email, not chat. Put it where the caller's service already reads secrets from.
3. The caller deploys or reloads. Requests start arriving with the new secret. The old one still works, so nothing breaks mid-rollout.
4. Watch which secret each request uses. When the old one stops appearing, the caller has finished.
5. Expire the old secret. On the date, or earlier once it's unused.

How long should the old and new secret overlap?

Long enough for the slowest caller to ship one ordinary release, and no longer. An overlap is a window in which two secrets open the same door, so every extra day is a day an old, possibly copied secret still works.

If one caller needs much longer than the rest, give that caller its own credential. Then a slow rotation for one service doesn't hold an old secret open for all of them.

How does the server accept two secrets at once?

It keeps a hash of both and accepts a request that matches either, as long as the old one hasn't expired. Store a hash of each secret, never the plaintext:

import hashlib
import hmac
from datetime import datetime, timezone

def hash_secret(secret: str, pepper: bytes) -> str:
    return hmac.new(pepper, secret.encode(), hashlib.sha256).hexdigest()

def matching_secret(credential, presented: str, pepper: bytes) -> str | None:
    digest = hash_secret(presented, pepper)
    if hmac.compare_digest(digest, credential.secret_hash):
        return "current"
    if (
        credential.previous_secret_hash
        and credential.previous_expires_at > datetime.now(timezone.utc)
        and hmac.compare_digest(digest, credential.previous_secret_hash)
    ):
        return "previous"
    return None

Log the return value with the client ID. That log line is step 4: a count of "previous" by client tells you who hasn't switched yet.

Two details matter in that code. hmac.compare_digest compares in constant time, so the response time doesn't leak how much of a guess was right. And the expiry check runs on every request, not in a cleanup job, so the old secret stops working at the minute it expires even if the job is late. A job can still clear the expired hash afterwards, as housekeeping.

What does the caller have to change?

Nothing but the secret's value, if their service reads it from configuration. The client ID, the token URL and the request format stay the same, so the change is a config update and a reload, not a code change. Tell them three things in the rotation notice:

• Where the new secret is, and that the client ID has not changed.
• The date and time, with time zone, when the old secret stops working.
• How they can check they switched: a test call, or a line in your usage view.

What goes wrong most often?

• Tokens outlive secrets. If the secret is exchanged for an access token, tokens minted with the old secret keep working until they expire. Add the token lifetime to your overlap, or revoke tokens when the old secret expires.
• Environment variables need a restart. A service that reads its secret at boot won't pick up the new one until it restarts. Know which of your callers work this way before you set the expiry.
• Keep the client ID. Rotate the secret only. A new client ID changes the caller's configuration, their logs and your usage history.
• One secret, several services. If three services share a secret, all three have to switch before the old one can expire, and you can't tell from the logs which one is late. Split them into one credential each.
• Rotating twice in a row. Most schemes keep one previous secret. A second rotation during the overlap pushes the first old secret out early, and any caller still on it breaks.
• Rotate on a schedule. Every 90 days, say. A process you run four times a year works when you need it in a hurry.

How often should you rotate API secrets?

On a fixed schedule, such as every 90 days, and at once whenever someone who had access leaves or a secret may have been exposed. The schedule matters less for the number of days than for the habit: a rotation you run every quarter is a script and a checklist, while one you run every three years is an incident. Put the next date in the credential's record when you issue it, so the reminder doesn't depend on anyone's memory.

What if the secret leaked?

Expire the old secret now, with no overlap. A short outage for a caller is better than an attacker's overlap window. Then issue the new secret, check your logs for calls made with the old one from places you don't recognize, and tell the caller what happened and when the old secret stopped working.

Sources

• OWASP Secrets Management Cheat Sheet, on rotation, storage and what to do after a leak.
• Python hmac documentation, including compare_digest, used in the server code above.
• RFC 6749, section 4.4, the OAuth client credentials grant, where a secret is exchanged for an access token.

Where EndPointBlank fits

EndPointBlank gives each service its own client ID and secret, scoped to one environment. Rotating a secret issues a new one while the previous secret keeps working for 7 days, and the client ID stays the same. Secrets are stored as digests and shown once, at creation.

See how machine-to-machine auth works or start free.