The issue has been resolved. MS Azure has confirmed that they have repaired the impacted links, restoring networking capacity between datacenters. Many internal and external services saw signs of recovery from 15:30 UTC, but we continued to work on mitigating some latency issues. We apologize for any inconvenience. Kindly find the below summary.
Incident summary: Number of links between two datacenters have become unavailable due to a fiber cut.
Issue start time: 07:22 UTC on 05 Jul 2023
Issue resolution time: 15:30 UTC on 05 Jul 2023
Severity: 2
Impacted Services: Custody API
Root cause from Service Provider(MS Azure): Several network links between two datacenters became unavailable, due to a fiber cut caused by severe weather conditions in the Netherlands. This primarily impacted network connectivity between datacenters within the region, not traffic to and from the region itself.
Recovery from Service Provider(MS Azure): We have repaired the impacted links, restoring networking capacity between datacenters. Many internal and external services saw signs of recovery from 15:30 UTC, but we continued to work on mitigating some latency issues.
Posted Jul 06, 2023 - 12:11 CEST
Update
We are continuing to monitor for any further issues.
Posted Jul 06, 2023 - 10:40 CEST
Monitoring
Azure Status: We are currently applying mitigation actions and monitoring the repair of the affected fiber. The next update will be provided in 60 minutes, or as events warrant. For more updates, kindly visit https://azure.status.microsoft/de-de/status/
Posted Jul 05, 2023 - 15:44 CEST
Identified
Azure-Status: We determined a number of links between two datacenters have become unavailable due to a fiber cut, caused by severe weather conditions in the Netherlands, predominantly impacting communications between these datacenters in the West Europe region. As a result of this interruption, Azure services that rely on internal communications with other services may experience degraded performance, manifesting in the issues. For more updates, kindly visit https://azure.status.microsoft/de-de/status/
Posted Jul 05, 2023 - 15:43 CEST
Investigating
We’re currently experiencing degraded performance issues with Custody-API due to Azure experiencing packet drops, timeouts and increased latencies in responses.
Activity start time : 07:22 UTC on 05 Jul 2023
Activity End time : TBD
Posted Jul 05, 2023 - 15:40 CEST
This incident affected: Tangany Custody API (API backend).