We are experiencing an outage on the Swisscom BIS platform. All Business Internet Service customers are affected. Investigation is in progress.
Update 13:20 CEST
Swisscom is still investigating and we are waiting for an ETR. As soon as new information becomes available we will update.
Upate 13:43 CEST
Latest update from Swisscom: "The manager on duty (MoD) has been mobilized, due to risk of high customer impact."
Upate 16:15 CEST
Swisscom has involved their network vendor supplier in resolving the disturbance. We are still waiting for an ETR.
Update 17:39 CEST
We see some, but not all, of the affected routers back online. Still no information from supplier.
Update 21:13 CEST
Supplier informed us that since 06:30 p.m. the situation is stable in all regions. The specialists of our supplier will continue to work hard on the root cause analysis.
Update 26.05.2016 08:45 CEST
While our suppliers ticket is still open, they sent us a interim report what happened:
Impact on customers
- In Switzerland, all customers that are connected via EAP are affected by the incident and do not dispose of any network connection.
- Customers with a second network connection outside EAP are not affected by the incident.
- Since 03:30 p.m., the situation has soothed. By 6 p.m., are most services fully available again. Currently is the situation stabile.
Root Cause
- The trigger is in the software environment, the detailed cause analysis continues. The experts could reproduce the disturbance behavior in the laboratory. The specialists are working now on preventive protection mechanisms, to further stabilize the network.
Measures planned
- Root cause being analyzed with highest priority
- Continuity of task force
- Thorough monitoring of live situation
- Field Services on stand-by
- Planned maintenance at night on May 26, 2016: which contribute preventively the network stabilization. Between 2:00 and 5:30, it may therefore be a short service disruptions of 2 to 5 seconds.
Timeline / Details
Milestones / Date, Time
Start of incident / 24 May 2016, 12:00 p.m.
Start of major incident / 24 May 2016, 13:40 p.m.
Most services fully available again / 24 May 2016, 18:00 p.m.
Measures taken
Building of task force / 24 May 2016, 12:30 p.m.
Involvement of platform supplier / 24 May 2016, 12:30 p.m.
VPLS interface: Cut-off and controlled reboot / 24 May 2016, since 01:45 p.m.
Planned maintenance for network stabilization / 26 May 2016, 2:00-5:30 a.m.m.
| 
