OrbusInfinity - All Regions: Periodic Application Latency
Incident Report for Orbus Software
Postmortem

Impact

OrbusInfinity users may have experienced periodic and short spells of application latency and slow response times when logging in to OrbusInfinity for the first time. Some users may have experienced rare and occasional timeouts and received errors through the course of using the OrbusInfinity application.

Root Cause Analysis

Sporadic but large increases in application usage and traffic volumes resulted in an OrbusInfinity application service slowing down and eventually being auto restarted.

The service is configured to handle deviations from normal traffic volumes and system load with auto-scaling in place to account for the fluctuations.  In this instance, the profile of the traffic did not trigger the auto-scale operations.  The traffic volumes were extremely high but short in duration.  This meant auto-scale trigger thresholds were not met over the course of the evaluation periods.

As part of remediation activities, Orbus engineers have updated the configuration for auto-scaling to better identify and react to the traffic volumes and profiles of this nature.  Thresholds and evaluation periods have been adjusted accordingly meaning auto-scale operations are working as expected.  This has been validated by engineers through the period of monitoring post incident.

Posted Jan 16, 2025 - 10:18 UTC

Resolved
Following the restoration of normal service levels and a stable period of monitoring, the incident has been marked as resolved.
Posted Jan 15, 2025 - 17:31 UTC
Update
The OrbusInfinity platform remains stable across all regions. Engineers will continue to monitor for an extended period.
Posted Jan 14, 2025 - 17:20 UTC
Monitoring
A fix has been implemented with normal service restored across all OrbusInfinity regions. Engineers will continue to actively monitor the platform over the next 12-24 hours, after which the Incident will be closed.
Posted Jan 13, 2025 - 18:24 UTC
Investigating
We are investigating an incident impacting all OrbusInfinity regions which is causing periodic and short spells of application latency. The impact to end-users is minimal but you may notice occasional delayed responses to application requests.

Engineers are actively engaged and investigating the issue.
Posted Jan 13, 2025 - 18:02 UTC
This incident affected: Australia East (OrbusInfinity), Canada Central (OrbusInfinity), Qatar Central (OrbusInfinity), South Africa North (OrbusInfinity), United Arab Emirates North (OrbusInfinity), United Kingdom South (OrbusInfinity), Europe West (OrbusInfinity), United States East (OrbusInfinity), United States West (OrbusInfinity), and IRAP (Australia) (OrbusInfinity).