Domain resolution

Incident Report for Stellate

Postmortem

Summary

On August 20, 2025, our services experienced elevated errors on our KV stores due to a third-party vendor issue. The incident was caused by a faulty enhancement deployment on the vendor's platform, which was subsequently rolled back to restore service.

Root Cause

Third-party vendor deployed a faulty enhancement that caused backend errors for a subset of customers, primarily in Singapore and California regions.

Resolution

Vendor engineering team rolled back the problematic deployment and confirmed service restoration.

Lessons Learned

  • Vendor responded quickly with clear communication
  • Need better early detection of vendor-side issues
  • Consider additional monitoring for vendor dependencies

Related: https://www.fastlystatus.com/incident/377843.

Posted Aug 21, 2025 - 16:35 UTC

Resolved

This incident has been resolved.
Posted Aug 20, 2025 - 22:50 UTC

Update

Posted Aug 20, 2025 - 22:43 UTC

Monitoring

A fix has been implemented and we are monitoring the results.
Posted Aug 20, 2025 - 22:42 UTC

Update

Noticing elevated error rates across our KV stores.
Posted Aug 20, 2025 - 21:57 UTC

Update

We are continuing to investigate this issue.
Posted Aug 20, 2025 - 21:36 UTC

Investigating

We are currently investigating this issue.
Posted Aug 20, 2025 - 21:35 UTC
This incident affected: GraphQL Edge Caching, GraphQL Metrics, GraphQL Rate Limiting, GraphQL Developer Portals, User API, and Admin API.