Impact:
We had 40 minutes where API calls relying on context objects were failing.
Detection:
Our internal monitoring detected this issue.
Root Cause:
We introduced a new security model to our context API, and as a result, caused a callback loop between that API and another service.
Resolution:
We reverted the code within 40 minutes of our internal monitoring raising an alert and have reimplemented the new security model in a way that doesn’t cause the callback loop.