Ticket: | #10348 |
---|---|
Title: | Outage: Routing issues / random linecard reboots / emergency downgrade |
Duration: | 2016-09-16 22:45 - 2016-09-21 18:00 |
Severity (1-5): | 3 |
Status: | Done |
We are experiencing unspecific routing issues in our network. Investigation is in work. Update 20160917 01:23 CEST Routing seems to be stable again. The reason has still to be investigated. Ticket is held open. Update 20160918 11:00 CEST Tonight there occured the same routing problems again. Investigation is still in work. Update 20160918 16:40 CEST The problems are still occuring. Our vendor is looking at the issue. Update 20160918 20:00 CEST After consulting our vendor we decided to downgrade the affected routers. These are
r1ams2.core We will prepare the devices now and start with downgrading at around 23:00 CEST. Update 20160919 05:00 CEST All routers except r1ams1.core, r1ams2.core, r1dus1.core and r1vie1.core have successfully been updated. The device r1dus1.core has not came back online after reboot and needs advanced troubleshooting. On all other routers the ports and sessions should work stable again. In any case of failure please let us know. Update 20160920 09:20 CEST We managed to downgrade our Router in Amsterdam. We have an open case with our vendor for our router in Dusseldorf. Downgrade AMS-1 and VIE-1 are still pending due to dependency issues. We haven't experienced another lincard reboot since yesterday morning. We are still monitoring the situation. Update 20160921 18:00 CEST Today we were able to downgrade the remaining devices located at AMS-1 and VIE-1. Additionally we were able to recover the device in Dusseldorf (DUS-1). All services should be stable again now. |