• Stellate relies on Fastly infrastructure for our offerings
• Fastly experienced a partial outage of their KV Store offering on June 17th and June 18th, which affected Stellate. They provide a summary of this incident on their status page at https://www.fastlystatus.com/incident/376022

Timeline

• August 17th 10:46 UTC - A customer reported their stellate endpoint failing in the FRA (Frankfurt) point of presence (POP), as well as in several other edge locations. This was due to them pushing an update to their configuration, specifically the originUrl .
• 10:50 - We identified the issue as being a stale KV value in the FRA POP, as well as several others.
• 10:55 - We created an incident on our status page for degraded KV in the FRA POP and several others.
• 13:08 - We realized that Rate Limiting and Developer Portals were affected by this outage as well.
• 13:30 - We reported this incident to Fastly.
• August 18th 4:00 UTC - Fastly was not yet able to provide us with a satisfactory response on what was causing this and didn’t acknowledge the ongoing outage.
• 6:23 - A large e-commerce customer reported their website was unavailable. This was due to a KV key disappearing in the FRA POP, as well as several others.
• 7:09 - Additional reports started to come in via Intercom about services not responding properly.
• 7:15 - We escalated the incident with Fastly as from our view more regions seemed to be affected and becoming unavailable.
• 7:16 - We deployed a partial fix that disabled our new infrastructure. This fixed edge caching for users who didn’t recently push configuration changes (the majority of services). Rate Limiting, JWT-based scopes, and the Developer Portal were still affected by the KV outage.
• 8:01 - Fastly was able to reproduce the bug based on a reproduction that we provided earlier and started working on a fix.
• 9:02 - Fastly opened an official incident on their status page.
• 10:04 - Fastly marked the incident as resolved
• 10:19 - Fastly communicated to us that the cause was an issue with surrogate keys in their C@E caching layer.
• August 22nd - Fastly shared their confidential Fastly Service Advisory with us providing additional information about this incident and how they want to prevent this from happening again.

Next Steps

• We have had several calls with Fastly over the last couple of days, working with them to analyze what went wrong, why it took them so long to escalate this internally, and how we can improve communication and collaboration going forward.

  • As a direct outcome of this, we have re-connected with our European contacts at Fastly and designated a direct contact to involve in conversations and escalations going forward.

• We are going to investigate a fallback option for Fastly KV.

• Additionally, we will review all possible failure points that could make Stellate core services inaccessible (in the event of a third-party outage) and investigate options for additional redundancies for those services.