COVID-19 and the Tech World: Hyper Growth and Hyper Slowdowns

Last week I wrote about How IT Can Scale and Manage Costs in an Economic Downturn. In that blog, I talked about ways to cut costs and prepare your systems for what could be next. With any massive world-changing event, there are going to be companies that have to scale down and weather the proverbial storm and try to wait out the current situation. Industries like airlines, travel, brick and mortar retail and entertainment (i.e. movie theaters, theme parks) are greatly impacted as business dwindles, but it is not the end for these industries — just a bump in the road.

This week I want to talk about the companies that are actually seeing an overwhelming increase in systems usage and traffic. During a crisis like the one we are experiencing, the shift in how people interact and behave has accelerated the usage of certain services and introduced people into new ways of working, living, and interacting. In many ways, it’s forcing the modernization of many jobs and people.

For example, my wife is a middle school science teacher. Over the years she has gone into school teaching six classes a day, going through each class one by one and reteaching the next class the same concepts the previous class heard. Sure, over the years new technologies were introduced to help with grading, teaching, etc. but largely, the foundations of standing in front of a class and teaching in-person have remained. Because of COVID-19, the school was forced to convert to all online and digital (at least for a portion of the year). This caused my wife and her fellow teachers to adapt and accelerate their usage and knowledge of technology. It also opened up a brand new set of problems, like how do you do science experiments online?

Some Industries are Experiencing Massive Growth

A shift is happening across the globe in every industry but some industries are seeing massive growth and an increase in usage, such as:

• Online chat, video, and collaboration tools: Think about how many people are now using Zoom, Bluejeans, Slack, MS Team, Mattermost, etc., to communicate. Video and chat technologies are becoming a critical component to support the world.
• Grocery and pharmacies: Are there busier or more vital physical spaces in the world right now?
• Online retailers and shipping companies: As the shift in shopping is forced online, these are taking off even more than ever.
• E-learning and online education: As more and more colleges and K-12 schools are forced to send kids home, e-learning has taken off.
• Streaming services: What would we all do without access to Netflix, Hulu, Disney, or Amazon Video?
• Gaming companies: We need to pass those weekends somehow, without leaving the house.
• Medical manufacturers, telemedicine, and pharmaceutical companies: Without a doubt, there are now critical needs here.

As these industries see exponential growth, we are starting to see new problems and challenges popping up that need to be solved. For example:

• Streaming services have had to slow down their services to prevent overloading the internet.
• Xbox live had multiple outages recently, impacting service.
• Epic Game’s Fortnite had outages as well.
• Robinhood had outages during a market crash.
• Microsoft teams had outages, affecting users in Europe.
• IBM Cloud had a partial mystery outage.
• Even the video app Zoom has had outages!

These problems are not new — last year, two out of five people experienced slowdowns that impacted production while one out of five reported unplanned outages — but the massive shift in routines and workloads is making these issues more frequent and with more impact than they were in the past.

With the Shifting Workload, Is Your Company Prepared for a Surge?

News is evolving and changing daily, affecting people’s attention and priorities. Masses of people are being forced to shift their routines in a matter of minutes or hours. That means many websites, e-tailers, applications, or services can see their traffic and applications go from 0 to 100 in milliseconds. As we have seen, many of these companies’ infrastructure will not able to handle the surge. It is often hard to quantify the impact of an outage, but in today’s world, a slowdown or outage may have life-altering impacts beyond what it was just three months ago.

Some important things to consider:

• Realize a slowdown feels like an outage to many people. Over the years, people have done a variety of studies on this, showing people now have less than the attention of a goldfish or that you will lose 50% of your traffic if your app is unresponsive for more then a second or two.
• Depending on what your application is, some people may consider your service vital. An outage can have cascading impacts on your business and reputation.
• Workloads are changing so rapidly you may not be prepared for the shift. Slack, for instance, grew 40% since the start of February and Microsoft teams more than doubled their active users since last year! Most companies don’t plan for doubling or tripling traffic in days or even in a few months. They may have dreamt of it, but in reality, a reasonable progression would have been more plausible — not today!

What can you do now?

If you are already seeing growth and opportunities in the current market, it’s important to ensure your systems are set up to be elastic with the ability to grow and shrink as the demand occurs. This is harder (but not impossible) if you have not set things up ahead of time. From the database perspective, ask yourself these questions:

• • Can you add or remove slaves or nodes from your cluster easily?
  • Are you spread across availability zones or providers to prevent one area or provider from causing you an outage?
  • Do you have tools set up to quickly find and fix issues as they occur?
  • Have you tested your systems under load?
  • Do you have a plan in place to test the performance of the new code before it makes it into production?
  • Do you have regular ongoing performance and scalability audits on your systems?
  • How often are you reviewing your workload for changes in the patterns?
  • When was the last time you tested your failover?
  • What happens if your hosting provider has an outage?
  • What happens if the database traffic doubles or triples overnight?

Hopefully, you have prepared as best you can, but even if you haven’t, it’s never too late to start testing, tuning, and optimizing your systems. There are plenty of great tools and companies that can help accelerate these efforts, including Percona. Our teams are here for you, and we’ll continue to keep you updated and informed. Contact us if you’d like to talk more about ways we can help scale and manage your databases in this new tech world.

How Has COVID-19 Impacted Your Company?

We want to know how your company has been impacted by COVID-19.

Note: There is a poll embedded within this post, please visit the site to participate in this post's poll.
by Matt Yonkovit via Percona Database Performance Blog