During a routine employee offboarding, we revoked that employee’s access to Fastly.

Revoking their access to Fastly also revoked all access tokens that engineer created. Unfortunately, this included the central API token all our systems use to communicate with the Fastly API.

This had two immediate impacts:

1. Purging started failing silently: Stellate’s purging API kept returning successful responses even though data would not be evicted from the cache.
2. Service configuration updates failing silently: Service configuration updates appeared to persist even though they were not updated in the CDN.

As part of the incident response, we switched the central Fastly API token to a new token owned by a shared engineering account.

Further, we will work on gaining better visibility and alerting on failure conditions with the purging API, as well as audit all tokens in use by our services to ensure they are not owned by individual engineers.