Incident Description: On the 19th and 20th of September, we experienced a bug that impacted our service and caused disruption for our customers, preventing them from replying to reviews.
Incident Timeline:
- Replies Endpoint Modification: Complexities emerged during endpoint changes.
- Unidentified Processing Bug: A bug went unnoticed due to testing limitations.
- Simultaneous Disruptions: Two disruptions affecting replies functionality emerged at the same time:
- App Store side, which required App Store Connect users to accept an updated agreement on their end.
- Internal Bug: Endpoint changes caused the issue.
Key Takeaways and Plan:
* Adjustment of deployment processes
* Enhance monitoring and metrics.
* Implement real-time alerts.
* Enforce vigilant monitoring.