CFOtech New Zealand logo
Technology news for Kiwi CFOs and financial decision-makers
Story image

Cloudflare outage in 19 data centers worldwide due to own error

By Jessie Chiang
Wed 22 Jun 2022

Content delivery network Cloudflare says its outage for 19 of its data centers yesterday was because of a change in a long-running project to increase resilience in its busiest locations.

Many sites went down as a result, including Discord and Shopify. The company says even though these locations, which include Mumbai, Osaka, Singapore, Sydney, and Tokyo, are only 4% of its total network, the outage impacted 50% of total requests. 

Cloudflare says the first outage started at 06:27 UTC on June 21. At 06:58 UTC, the company brought the first data center back online, and by 07:42 UTC, all data centers were online and working correctly.

In a statement on its blog, Cloudflare says, "We are very sorry for this outage. This was our error and not the result of an attack or malicious activity."

The company says over the last 18 months, it has been working on converting all of its busiest locations to a more flexible and resilient architecture. During this time, Cloudflare converted 19 of its data centers to this architecture, internally called Multi-Colo PoP (MCP).

Cloudflare says this new architecture provides significant reliability improvements, allowing it to run maintenance in those locations without disrupting customer traffic. But those locations also carry a substantial proportion of the Cloudflare traffic, so any problem there can have a wide impact.

In order to be reachable on the internet, networks like Cloudflare make use of a protocol called BGP. As part of this protocol, operators define policies that decide which prefixes (a collection of adjacent IP addresses) are advertised to peers (the other networks they connect to) or accepted from peers.

Cloudflare says these policies have individual components, which are evaluated sequentially. The result is that any given prefixes will either be advertised or not advertised. A policy change can mean a previously advertised prefix is no longer advertised, known as being "withdrawn", and those IP addresses will no longer be reachable on the internet.

The company says while deploying a change to its prefix advertisement policies, a re-ordering of terms caused it to withdraw a critical subset of prefixes.

Due to this withdrawal, Cloudflare engineers experienced added difficulty in reaching the affected locations to revert the problematic change. However, the company says it has backup procedures for handling such an event and used them to take control of the affected locations.

But F5 Labs threat research lead David Warburton says this should be a reminder of the dangers of the centralisation of large cloud solution providers.

"In a traditional internet app deployment model, an outage of a server or misconfigured application might take out a single website," he says.

"But similar problems with a cloud solution provider can end up taking out all of their customers, resulting in not one website being taken offline, but hundreds or thousands. The impact can potentially affect organisations' digital experiences, revenues and reputations." 

Warburton says cloud solution providers provide immeasurable benefits to their users, but the re-centralisation of the internet through these cloud solutions is now causing the very problems the original design of the internet was intended to avoid through redundancy. 

ServerChoice commercial director Adam Bradshaw says outages can cause businesses serious reputational and financial damage. 

"For a smaller or medium-sized business, the risk associated with such failures in service can be critical. Even short outages in the digital economy could cause damage," he says.

"Diversifying an IT environment reduces the likelihood of third-party outages negatively impacting a business. Owning the hardware gives an organisation more control, with IT components such as colocation services providing an organisation with a backup when cloud services fail."

Related stories
Top stories
Story image
Commerce Commission
ComCom puts electronics sector on notice over resale price maintenance
The Commerce Commission has concluded an investigation into allegations that television manufacturers were engaging in illegal resale price maintenance.
Story image
Market growth
Salesforce unveils new offerings for consumer goods companies
Salesforce has announced new products for consumer goods companies to help brands navigate increasing market complexity more easily.
Story image
Government
Cyclone selected as NZ MOE software licensing partner
Following a recent Request for Proposal (RFP), Christchurch-based company Cyclone Computer Company Ltd (Cyclone) has been selected as The Ministry of Education’s software licensing partner.
Story image
Healthcare
Workday winning on culture and family focus
This family-first approach sees all employees receive access to family-wide private healthcare cover, as well as income protection and life insurance policies.
Story image
Digital Transformation
Govt IT spending on the rise, driven by XaaS - Gartner
Worldwide government IT spending is forecast to total $565.7 billion in 2022, an increase of 5% from 2021, according to Gartner.
Story image
Innovation
Huawei unveils cutting-edge inventions at innovation event
Huawei has unveiled a range of new inventions as part of its biennial Top Ten Inventions Awards at the Broadening the Innovation Landscape 2022 forum.
Story image
Appointments
Tech job moves - EOS, Rubrik, SAP, Talent, Verizon & Zoom
We round up all job appointments from June 2-8, 2022, in one place to keep you updated with the latest from across the tech industries.
Story image
Artificial Intelligence
Salesforce harnesses automated solutions with new developments
Salesforce has launched Sales Cloud Unlimited, a new feature to help accelerate productivity with AI and automation.
Story image
API
Industry-first comprehensive risk-based API security enhances protection
Application Programming Interfaces (APIs) have become a crucial part of operating web and mobile application businesses and are causing significant economic growth in the digital sector.
Story image
Data
How a single mandate changed software development forever
There’s conjecture about exactly when it was issued and by whom, but a mandate made twenty years ago is continuing to shape the software development process today.
Story image
Open source
DataStax secures US$115 million to fund database expansion
DataStax has secured US$115 million in funding, which it will use to develop and expand its Astra DB multi-cloud database and Astra Streaming service globally.
Story image
Unified Communications
Video: 10 Minute IT Jams - An update from Access4
Access4 are specialists in the unified communications sector, and Peter Eldon joins us today to discuss how UCaaS can play a vital role in business development.
Story image
Microsoft
Volpara, Microsoft project to detect cardiovascular issues
Volpara Health Technologies is working with Microsoft on a research and development project to speed up creating a product that detects and quantifies breast arterial calcifications (BACs).
Story image
Digital Transformation
Apptio adds portfolio enhancements to promote digital strategy
"While digitalisation creates opportunities, it also makes budgeting far more complex, leading many companies to waste substantial funds."
Story image
Broadband
Most NZ broadband can deliver four HD Netflix streams at the same time
Measuring Broadband New Zealand's latest report shows 99% of Fibre 300, Fibre Max, and HFC Max plans can handle four simultaneous high-definition Netflix streams.
Story image
Cloud
Cloudflare outage in 19 data centers worldwide due to own error
Cloudflare says its outage for 19 of its data centers yesterday was because of a change in a long-running project to increase resilience in its busiest locations.
Story image
Remote Working
Globalization Partners to improve global talent hire
Globalization Partners says the global availability of its services will make it fast and simple for companies to hire and pay anyone, anywhere in the world.
Story image
PagerDuty
Ready for anything with the PagerDuty Operations Cloud
In a world of digital everything, teams face increasing complexity. Ever-growing dependencies across systems and processes put customer and employee experience, not to mention revenue, at risk.
Story image
Robotic Process Automation / RPA
rapidMATION helps Coates achieve success with landmark RPA solution
A strong Robotic Process Automation solution (RPA) can help solve many complex issues that businesses face daily. 
Story image
Entelar
How TruSens air purifiers can create healthier workspaces
The pandemic has heightened our awareness of our own and others’ health, and made us all much more conscious of the environments we work in.
Story image
Citrix
The best ways to attract young talent during labour shortages
New research from Citrix reveals hybrid working and ventures into the metaverse are top of mind for Gen Z workers.
Story image
Dark web
Cybercrime in Aotearoa: How does New Zealand law define it?
‘Cybercrime’ is a term we hear all the time, but what exactly is it, and how does New Zealand define it in legal terms?
Story image
Cloud
Automation a point of difference for APJ enterprises - report
There has been a massive shift in which departments are using automation tools and creating those automations, according to Workato.
Story image
Hybrid workforce
Why hybrid working is here to stay and how to ace it
Citrix's new report reveals hybrid workers are more productive and engaged at work than their office and completely remote counterparts.
Story image
Axway
New financial accounting hub can manage 40 million events daily
The new Axway Financial Accounting Hub can manage 40 million events daily and halve costs and integration time for ERP Finance migration projects.
The Access Group
Increasing headcount isn't always the best way to grow. A good financial strategy can help solve many issues, and The Access Group shares the secret to success.
Link image
Story image
Cloud
Chatbots gaining momentum in customer service space
Chatbots are gaining momentum in the customer service space, but a human touch still unbeatable, according to a new study.
Story image
Orbital Insight
Orbital Insight solution set to drive better data-driven decisions
The company says the new Site Intelligence solution will provide granular visibility, behaviour analytics and deep insights about customers and competitors at any location. 
Story image
Tech job moves
Tech job moves - Boomi, Limepay, Thales, VMware & Zoom
We round up all job appointments from June 6-16, 2022, in one place to keep you updated with the latest from across the tech industries.
Story image
N4L
N4L, Spark, Chorus partner for Hyperfibre school upgrade
Networks for Learning (N4L) has partnered with Spark and Chorus to upgrade Wellington College to Hyperfibre, fostering stronger outcomes for students and teachers.
Story image
Data analytics
Qlik releases new capabilities, underpinned by Snowflake integration
“We're seeing continued customer success and growth in demand to leverage Qlik and Snowflake together to advance cloud data analytics strategies."
Story image
Contact Centre
Customer service agents don't want to return to contact centres
A new report has revealed that 85% of customer service agents want to work full-time at home and not return to contact centre offices.
Story image
Logitech
How technology can level the playing field in the hybrid era
As businesses across the globe continue to grapple with the aftershocks of the COVID-19 crisis, it’s become crystal clear that the hybrid work model is here to stay.
Story image
eInvoicing
Airwallex, Xero extend partnership with easier invoice payments
Airwallex has extended its long-term partnership with Xero by releasing a new payment link integration for Xero invoices that will make receiving them easier and faster for Australian businesses.
Story image
Open XR
Juniper Networks, Sumitomo Electric, Arrcus join Open XR forum
Building a robust multi-vendor ecosystem is essential to enable network operators to achieve increased network capabilities provided by XR optics technology.
Story image
Cloud
Cisco Live showcases new offerings in its first hybrid event
Cisco Live 2022 has seen Cisco executives and customers take the stage to present a range of discussions in the company’s first-ever hybrid event.
The Access Group
Struggling to understand which transformative technologies will help your business? The Access Group provides a look into key opportunities and impacts for finance.
Link image
Story image
Robotic Process Automation / RPA
Study shows prioritising IA can deliver better operational outcomes
A new Everest Group Pinnacle Model study, supported by SS&C Blue Prism, has found that businesses that use and develop solid automated practices tend to see better operational outcomes.
Story image
Cloud
Dynatrace named Leader in Gartner’s 2022 Magic Quadrant
Gartner has named Dynatrace a Leader in the 2022 Magic Quadrant for Application Performance Monitoring (APM) and Observability.
Story image
Cloud
Boomi surpasses 20,000 customers. Sets record for the iPaaS space
Boomi has announced it has surpassed the 20,000 customer mark, setting the record for the largest customer base among iPaaS vendors.
Story image
Cybersecurity
Why is NZ lagging behind the world in cybersecurity?
A recent report by TUANZ has revealed that we are ranked 56th in the world when it comes to cybersecurity - a look into why we're so behind and what needs to be done.
The Access Group
Health and social care organisations are currently under significant financial pressure. Find out how financial transformation can help provide an effective route forward.
Link image
Story image
Testing
Video: 10 Minute IT Jams - An update from Tricentis
Tricentis provides software testing automation, and software quality assurance products for enterprise software.
Story image
Cloud
TradeWindow using Nintex platform amid global expansion
TradeWindow is using Nintex Promapp to support its recent expansion and to prepare for ambitious international growth.