AS13030

Status Info

Ticket:#10348
Title:Outage: Routing issues / random linecard reboots / emergency downgrade
Duration:2016-09-16 22:45 - 2016-09-21 18:00
Severity (1-5):3
Status: Done

We are experiencing unspecific routing issues in our network. Investigation is in work.

Update 20160917 01:23 CEST

Routing seems to be stable again. The reason has still to be investigated. Ticket is held open.

Update 20160918 11:00 CEST

Tonight there occured the same routing problems again. Investigation is still in work.

Update 20160918 16:40 CEST

The problems are still occuring. Our vendor is looking at the issue.

Update 20160918 20:00 CEST

After consulting our vendor we decided to downgrade the affected routers. These are 

r1ams2.core
r1win4.core
r1dus1.core
r1bsl2.core
r1fra2.core
r1glb2.core
r1ams1.core
r1ber1.core
r1stg1.core
r1zrh2.core
r1fra3.core
r1vie1.core
r1zug2.core

We will prepare the devices now and start with downgrading at around 23:00 CEST.

Update 20160919 05:00 CEST

All routers except r1ams1.core, r1ams2.core, r1dus1.core and r1vie1.core have successfully been updated. The device r1dus1.core has not came back online after reboot and needs advanced troubleshooting.

On all other routers the ports and sessions should work stable again. In any case of failure please let us know.

Update 20160920 09:20 CEST

We managed to downgrade our Router in Amsterdam. We have an open case with our vendor for our router in Dusseldorf. Downgrade AMS-1 and VIE-1 are still pending due to dependency issues. We haven't experienced another lincard reboot since yesterday morning. We are still monitoring the situation.

Update 20160921 18:00 CEST

Today we were able to downgrade the remaining devices located at AMS-1 and VIE-1. Additionally we were able to recover the device in Dusseldorf  (DUS-1). All services should be stable again now.

Back