Date of Incident: February 4, 2025
Duration: 12 hours
Impact: Customers experienced authentication errors in application I-frames.
Incident Summary
On February 4th, we performed scheduled maintenance to update security certificates used for authentication. Following this update, we identified an issue that prevented our application from properly authenticating with our identity provider, causing errors in application I-frames.
Impact
- Customers experienced authentication failures when accessing certain embedded application features.
- The issue persisted until corrective actions were taken.
Root Cause
The issue stemmed from a change in the security certificate’s identifier, which was not immediately recognized by the application. This caused authentication failures until the configuration was updated and services were restarted.
Detection & Response
- Our monitoring systems detected authentication failures shortly after the update.
- Our engineering team immediately began investigating and applied multiple corrective actions.
- The issue was resolved after multiple adjustments to the authentication configuration and a full service restart.
Preventive Measures & Next Steps
To prevent similar issues in the future, we are implementing the following improvements:
- Extending Certificate Lifespan – Reducing the frequency of certificate changes by extending expiration from 1 year to 5 years.
- Enhanced Change Management – Adjusting maintenance windows to allow more time for validation and rollback if needed.
- Safer Certificate Rollovers – Ensuring old certificates remain active until new ones are fully validated.
- Improving Authentication Resilience – Exploring ways to reduce dependency on manual configuration updates for authentication.
We sincerely apologize for any inconvenience this may have caused and appreciate your patience as we work to enhance the reliability of our authentication processes. If you have any questions, please reach out to our support team.