Impact:

We had 40 minutes where API calls relying on context objects were failing.

Detection:

Our internal monitoring detected this issue.

Root Cause:

We introduced a new security model to our context API, and as a result, caused a callback loop between that API and another service.

Resolution:

We reverted the code within 40 minutes of our internal monitoring raising an alert and have reimplemented the new security model in a way that doesn’t cause the callback loop.