Today at 3:12 am (EDT/Washington DC time), OVHcloud carried out engineering interventions on a router in their Vint Hill data center in the United States, which caused disruptions on their entire global network backbone. OVHcloud’s global network was completely down for an hour. Through social media it became clear that not all customers were understanding of the situation.
These engineering interventions were aimed at reinforcing OVHcloud’s anti-DDoS protections. DDoS attacks have been particularly intense in recent weeks, stated OVHcloud.
Following a human error during the reconfiguration of the network in their Vint Hill data center in the United States, OVHcloud experienced a severe outage on their whole global backbone. The OVHcloud engineering teams intervened to isolate the equipment at 4:15 am, after which the configuration could be fixed. The cloud hosting provider’s services have been restored since this intervention.
OVHcloud is currently conducting a verification campaign with its customers to confirm that all their services have been restored.
Some of the reactions on the severe outage by OVHcloud customers and others on Twitter:
“Any possibility for coordinated better and _faster_ communication through official OVH accounts in global outages? (…) Now we’re in the dark for half an hour and get info from OVH CTO in french.”
“A human error? In 21st century? All of our production systems are inaccessible due to that person. But worst, you have no processes in place prevent such human error from happening. What is the ETA (I dare to ask as a long-time customer)?”
“Octave, I think people live in fear of another building burning down… it’s a problem that we are not reassured. Your personal communication shows us that you are dedicated to your passion: music. However, since then, the OVH service is losing quality.”
“For a company that presents itself as the future of the cloud in France, it’s a real mess. What a lack of professionalism, this is really DIY. It’s all very well to criticize the US competition, but if you want SLAs and security, there’s no contest.”
“A human failure? Did you plan maintenance work on a Wednesday from 9:00 to 10:30, knowing that the bulk of your customers are in Europe and that this time is CRITICAL for most companies? Was there really no better time?”