Update (31/05/2018 20:47): Our NOC has identified a network incident affecting several data centres on our UK network.
Update (31/05/2018 21:19): It is suspected that the convergence of two diverse fibre paths, from an underlying provider Zayo, may have been severed resulting in network isolation for DC 8 (St. Asaph). Engineers are working on three resolution paths (transit re-route, fibre repair and new fibre provisioning) with an ETA of 2-12 hours for full service restoration.
Update (31/05/2018 21:30): No new information available. Updates to follow within 15 minutes.
Update (31/05/2018 21:45): Zayo are investigating the fibre failures in North West England
Update (31/05/2018 22:00): No new information available. Updates to follow within 15 minutes.
Update (31/05/2018 22:15): No new information available. Updates to follow within 15 minutes.
Update (31/05/2018 22:30): No new information available. Updates to follow within 15 minutes.
Update (31/05/2018 22:45): We are awaiting confirmation, but current investigation would indicate the break of two, diverse fibre paths.
Update (31/05/2018 23:00): There appears to be a fibre event 27km from Manchester. Once we have an ETA from our fibre repair group it will be provided. Further updates to follow as more information is available.
Update (31/05/2018 23:15): It is confirmed that there was no planned works occurring at the time of the break. Updates to follow.
Update (31/05/2018 23:30): No new information available. Updates to follow within 15 minutes.
Update (31/05/2018 23:45): No new information available. Updates to follow within 15 minutes.
Update (01/06/2018 00:00): A technician is en route to the break location with an ETA of 20 minutes. We will begin testing as soon as the technician arrives.
Update (01/06/2018 00:35): No new information available. Updates to follow within 15 minutes.
Update (01/06/2018 00:45): The on-site tech has determined this is due to a large cable cut and narrowed the potential damage location. Our underlying fibre provider, Zayo, have OSP construction and splice crews en-route to a common Splice Enclosure just North of Stretford. Further updates to follow as more information is available.
Update (01/06/2018 01:00): No new information available. Updates to follow within 15 minutes.
Update (01/06/2018 01:15): No new information available. Updates to follow within 15 minutes.
Update (01/06/2018 01:45): Our fibre provider, Zayo, are still attempting to locate the exact location of the fibre break. It has however been narrowed down significantly.
Update (01/06/2018 02:00: No new information available. Updates to follow within 15 minutes.
Update (01/06/2018 02:15: The location of the fibre break has been narrowed down further. We are awaiting a further update from Zayo.
Update (01/06/2018 02:30: No new information available. Updates to follow within 15 minutes.
Update (01/06/2018 02:45: The fibre cut has been located and the damage is being assessed now. Zayo note that from the looks of things, there are two 96 count under ground cables that appear to be damaged.
Update (01/06/2018 03:00: No new information available. Updates to follow within 15 minutes.
Update (01/06/2018 03:15: No new information available. Updates to follow within 15 minutes.
Update (01/06/2018 03:30: No new information available. Updates to follow within 15 minutes.
Update (01/06/2018 03:35: Zayo have updated: Engineers are still on the ground trying to locate the exact damage location. The Cable crew is en route with an ETA of 4:00 AM BST. The construction crew is en route with an ETA of 5:00 AM BST. Further updates to follow as more information is available.
Update (01/06/2018 03:56: No new information available. Updates to follow within 34 minutes.
Update (01/06/2018 04:16: After walking the span where the damage should be the OSP crew were unable to locate any sign of construction or damage that would cause the outage. The OSP are now going to open the SEP's on each side of the suspected damage and trace the exact distance in order to give them the exact location of the physical damage. Further updates to follow as more information is available.
Update (01/06/2018 04:30: No new information available. Updates to follow within 30 minutes.
Update (01/06/2018 05:03: OSP have identified fibers between the Computer Centre and one of their Splice Enclosures. The location matches the 1.2 km from another one of their SEPs. OSP is currently looking for signs any visible damage. As it is now starting to get light outside, it should assist in finding the damage. Updates to follow within 30 minutes.
Update (01/06/2018 05:30: No new information available. Updates to follow within 30 minutes.
Update (01/06/2018 05:54: The Zayo OSP has located the damage in a field where it appears a trencher has hit the cable and duct severing the full cable. The crew are currently working to devise a plan to get into the joints on each side of the damage as well as what they will do to repair the conduit. Unfortunately, this is in the middle of a field so there does not look to be spare duct run in the area. Once a plan has been devised we will share the plan as well as push our OSP for a potential estimated time to repair. Updates to follow within 36 minutes.
Update (01/06/2018 06:08: The Zayo OSP crew has identified the location of the cut, there is a 5" (five inch) micro trench that is approximately 2 meters in depth, that has been dug across a field our fiber traverses. This trench has severed multiple conduits and completely severed their cables.f 2m. . Zayo is currently assessing the damage to find the nearest chambers with the cabler team in an effort to identify the quickest possible restoration, whether that be a temporary (section throw/splice) or permanent cable pull/splice. The location is in the middle of a farmers field which has added time to our ability to locate. Updates to follow within 30 minutes.
Update (01/06/2018 06:40: No new information available. Updates to follow within 30 minutes.
Update (01/06/2018 07:02: No new information available. Updates to follow within 30 minutes.
Update (01/06/2018 07:20: Zayo have found the 2 chambers either side of the fibre fault and found some slack cable but the chambers are 900 and 1200m away. The preferred option is to dig and pull the slack to create 1 extra joint, if they are unable to do this they may have to add extra joints. They are currently trying to gain access to the fields due to multiple ownership to access areas to where the fault is located but are in conversation with the landowners to get to the fault. Updates to follow within 30 minutes.
Update (01/06/2018 07:44: No new information available. Updates to follow within 30 minutes.
Update (01/06/2018 08:02: OSP is currently excavating in the field to try get to the cables, and sub-duct. They are doing this to try, and pull the slack cable through. Updates to follow within 30 minutes.
Update (01/06/2018 08:30: No new information available. Updates to follow within 30 minutes.
Update (01/06/2018 08:50: No new information available. Updates to follow within 30 minutes.
Update (01/06/2018 09:20): The excavation continues. We will update you as soon as we any new information.
Update (01/06/2018 09:50): Civil crews are still working to complete excavation at this time. Crews expect to be completed soon, and should be pulling slack on the cable to prep for splicing. More updates to follow.
Update (01/06/2018 10:00): No new information available. Updates to follow within 30 minutes.
Update (01/06/2018 10:30): Excavation has been completed, but the cable did not have enough slack to pull through. Crews are now prepping a new 50m strap cable between the 2 exposed ends of the cable. More updates to follow.
Update (01/06/2018 10:55): Fibre engineers estimate a repair schedule of 1-7 hours, with a possible ETA of 18:00
Update (01/06/2018 11:20): Fibre engineers intend to run a temporary fix over ground splicing to commence around 12:00 for services to start coming back up.
Update (01/06/2018 11:40): Crews have completed prepping the cable into the joints and are now taking the cable over to the excavation location to be prepped and spliced. We estimate we will begin splicing at 12 GMT.
Update (01/06/2018 12:00): No new information available. Updates to follow within 30 minutes.
Update (01/06/2018 12:30): No new information available. Updates to follow within 30 minutes.
Update (01/06/2018 12:40): Crews are nearing completion of all the cable prep and should begin splicing in 30 minutes. Will provide an additional update when splicing begins.
Update (01/06/2018 12:45): We have partially restored DNS services for customers. Please note that DNS responses may be intermittent and slower than usual.
Update (01/06/2018 13:00): No new information available. Updates to follow within 30 minutes.
Update (01/06/2018 13:20): Crews are now starting to splice the cable. You will begin to see services restore as splicing continues. No ETR at this time. Will provide additional updates as splicing continues.
Update (01/06/2018 13:45): Connectivity has been restored to our DC 8 (St. Asaph) facility. A temporary overground fix has been put in place. Please note that service is still at risk and there are likely to be interruptions.
Update (01/06/2018 14:15): Repair work continues on a proper fix for the fibre cuts. At the moment, both of our paths are back online albeit with a temporary fix. We will provide further updates within the next 60 minutes.
Update (01/06/2018 15:15): Both paths continue to be up and running while full repair work continues. We will provide further updates within the next 60 minutes.
Update (01/06/2018 16:15): No new information available. Updates to follow within 60 minutes.
Update (01/06/2018 17:15): Repair work continues. Updates to follow within 60 minutes.
Update (01/06/2018 16:16): Zayo splicing crews have reported an ETR of 15 minutes from now. We appreciate your continued patience.
Update (01/06/2018 17:49): Zayo OSP crews have completed splicing and are hands off. We appreciate your patience throughout this repair process. We'll monitor closely over the weekend.
