We started receiving reports around 7:00 PM PDT of Web Services and embedded pages failing to load. Affected requests returned 404 Not Found or 500 Internal Server Error responses. Some customers also saw “The server was unable to process the request due to an internal error.” Beacon API responses could contain an XML exception stating that the serviceDebug start tag did not match the error end tag.

The issue was intermittent, including during testing and service restoration. One server was failing requests despite its services reporting as running. Service was restored by approximately 9:00 PM PDT.

We will continue to identify better monitoring techniques and expand our coverage so we can detect similar failures sooner.