SY4 PVE Host Service Outage

Incident Report for Fusion Broadband Pty Ltd

Postmortem

Dear Fusion Customers,

At 18:04 AEST on 24/05/2026, some hosted firewall users experienced a service disruption impacting firewall and Internet connectivity. The total impact window was approximately 3 hours 44 minutes.

Timeline (AEST)

  • 18:04 — Physical host lost connectivity. Most VMs successfully invoked HA and migrated to alternate hosts, maintaining service.
  • 18:30 — Investigations determined that the NIC had failed.
  • 19:00 — Technicians were dispatched to the DC to replace the failed NIC.
  • 19:21 — Service notification issued advising customers that Fusion staff were investigating.
  • 21:22 — Onsite engineers commenced replacement of the failed NIC.
  • 21:48 — Services restored. VMs that had invoked HA were migrated back to their assigned primary host.

Root cause

A Network Interface Card (NIC) failure on the physical host caused the loss of connectivity. While HA functioned as designed for most VMs, 5 VMs failed to migrate successfully. Investigation found that disk replication for these VMs was in progress at the time of failure, resulting in corruption on the replica disks. The endpoint disks were therefore unable to start due to missing data.

Resolution

The failed NIC was replaced and all affected VMs returned to service. HA configurations for the 5 impacted VMs have been rebuilt. We will coordinate with affected customers to conduct further validation testing to confirm correct operation.

Detection and response

We acknowledge there was a delay between the incident occurring and the customer notification being issued. The incident began on a Sunday evening, which contributed to the extended detection-to-notification time.

Preventive actions

To reduce the likelihood and impact of similar incidents, we are:

  • Enhancing failure detection monitoring to reduce escalation lead time, particularly outside business hours.
  • Conducting spot checks on HA configurations across the fleet to identify any replication or configuration anomalies before they cause impact.

We apologise for the inconvenience this caused. Thank you for your patience and understanding as we worked to restore service.

Kind regards, The Fusion Team

Posted May 25, 2026 - 10:50 AEST

Resolved

This incident has been resolved.
Posted May 24, 2026 - 22:26 AEST

Monitoring

Engineers have replaced the failed NIC, services have been restored, we will be monitoring and running routine checks.
Posted May 24, 2026 - 21:48 AEST

Update

Engineers are onsite, beginning remediation works.
Posted May 24, 2026 - 21:22 AEST

Update

Engineers will be onsite at the DC in 35 Minutes, suspected failed NIC Cards. Will update once replacement is installed.
Posted May 24, 2026 - 20:25 AEST

Identified

Dear Fusion Customers,

We have identified an outage affecting some firewalls within our SY4 data centre, we are working to resolve this asap.

Unfortunately the configured HA has not invoked and we will also be determining why.

More updates will be posted as they come to hand.
Posted May 24, 2026 - 19:21 AEST
This incident affected: SD-WAN routing Groups (SYDNEY Hosted Firewall Stack).