How Cloud Apps Go Wrong

Data Can Bring You Down

Welcome to the Cloud

It’s exciting! You’ve got a huge new toolbox with a cornucopia of choices. And forums with a bonanza of advice.

Whether you are creating, migrating, or something in-between, everybody has an opinion about building cloud apps:

  • Development Model: Serverless? Micro-services? Containerized? Traditional?
  • Platform: Cloud-Provider? PaaS? IaaS packages?
  • Database Stores: MongoDB? PostgreSQL? DynamoDB? S3?

While you dig through the avalanche of advice, there is one topic conspicuous by its absence… what is the right way to handle persistent data?

Data storage in the cloud is clean and easy? Courtesy:

Day 1: How Should I Manage My Application's Data?

There are even more ways to store data in cloud than there were on-premises!

You can use:

  • Database as a Service: AWS RDS, MongoDB Atlas, etc.
  • Standalone Databases: PostgreSQL, MongoDB, MySQL, etc.
  • Files: local files or cloud NAS
  • Objects: more tiers than an Instagram wedding cake

Most applications use a combination of tools to get the best results.

Once you decide how to store the data, you need to provision for capacity and performance. You might even consider how you’ll scale, deal with failures, secure the data, and meet compliance regulations (Hey, I’m an optimist).

It’s a stressful decision. Data is “the new oil”, “the lifeblood of business”, and “the new bacon”. You want to pick the right way to manage your data; your business depends on that information. Unfortunately, with so many variables, making the right decision seems impossible. Still, you’ll pick something reasonable, and then you’ll never have to worry about it again. Phew.

Except… data doesn’t take care of itself. In the cloud, there is no storage, backup, disaster recovery, or compliance team watching over your data. If you care about your application, you’ll be managing its data every day of the application’s life.

Even the Gingerbread man can’t escape data management. Courtesy:

Day 2: A Day in the Life of a Cloud App Developer

Congratulations! You’ve released an application into your environment. You’ve created something that helps people. Take a moment to celebrate.

Now, take a deep breath. Your life will never be the same. Here’s a day in your life.

06:30: Make sure backups are happening and that they’re being replicated off site. Consider testing a restore to make sure the backups are good. Resolve to do it tomorrow… for the 100th straight day. Push down the guilt.

10:30: You get an escalation about application performance. After looking at processing and network, check out the data path. Figure out which storage resources are serving the application. Look at historical data to find changes in workload, resource performance, etc. Hope that application performance goes back to normal on its own.

12:10: Discuss with others how to scale the application. If you add more compute resources, how can they share the same data? Will that become a bottleneck?

14:07: Somebody hits a bug in the application. Create a clone environment, so you can reproduce the bug and test and fix. Wait hours to clone or create a data set.

16:20: Finance has asked you to reduce the cost of running your application. Explore ways to either use less expensive storage or use it more efficiently/dynamically.

18:30: As you walk out of the building, wonder when you will have time to ever build a new application.

23:30: Security has identified a flaw in the configuration of some AWS S3 buckets. They want to make sure your data is secure and encrypted. You need to send them details of what you’re doing and how. They also demand full audit logs of who has access to the data and what they’ve done.

06:30: Do it all over again…

The cloud offers infinite tools for building applications: UI, queuing, analytics, etc. Without data management, however, you won’t have time to create new applications. Instead, you’ll become a de-facto platform administrator: managing storage, backup, and compliance for your existing apps.

There is another way. Credit: Unknown

Day 2, Take 2: A Better Day

It may feel like you only have two options:

  • Bad: Spend every day in data management hell.
  • Worse: Use only one type of storage — e.g. object, DBaaS, etc.

There is a third answer: a data management solution that does the job of managing storage, data protection, and compliance for you.

In this world, today is not the worst day of your life…

06:30: Breakfast. [Backups have been happening automatically every 4 hours.]

10:30: Develop your new application. [Storage resources automatically scaled up to meet your existing application’s performance needs.]

12:10: Eat lunch and talk with co-workers about your new application.

14:07: Somebody hits a bug in the application. Instantly clone the environment to reproduce and test a fix; send the fix through the CI/CD pipeline.

16:20: [Storage resources scale down as the load reduces.]

18:30: Kick off a set of tests with real data for your new application; head home.

23:30: Sleep. [Your data is secure and the records are available to security, auditors, etc.]

Aim for the head and take control of data in the cloud. Courtesy:

The First Day of the Rest of Your Life

Cloud offers a tantalizing array of options to help build applications. Without data management, those tools will torment you because you won’t have time to use them. Plan ahead, though, and you can find a better way. Day 2 can be the best day of your life, not a nightmare.

Cloud should give you choice. Cloud should give you automation. Cloud should make you more productive. If that’s not happening, it’s time to look for a cloud-first data management solution.

Cloud Data Protection is Business Protection

In the Cloud, Data Protection Matters More Than Ever

Cleaning up after data loss. Photo Credit:

“As I waded through a lake of rancid yogurt, each vile step fueled my rage over failed backups.” The server that ran a yogurt manufacturer’s automated packaging facility crashed. The IT team could recover some of the data, but not all. They hoped everything would “be OK”. When they restarted the production line, they learned that hope is not a plan. Machines sprayed yogurt like a 3 year old with a hose in a crowded church. By the time they shut down the line, they’d created a yogurt lake. It took two months to clean and re-certify the factory. They missed their quarterly earnings. People lost their jobs.

Data protection matters because data recovery matters. Even in the cloud. Especially in the cloud.

Businesses Run on Data

Digital transformation has turned every company into an application business.

Have you ever thought about the lightbulb business? Osram manufactured lightbulbs for almost a century. Then, LED bulbs decimated the lightbulb replacement business. Osram evolved into a lighting solution company. Osram applications optimize customers’ lighting for their houses, businesses, and stadiums. Now a high-tech company, they sold the traditional lightbulb manufacturing business in 2017.

How about the fruit business? Driscoll’s has grown berries for almost 150 years. In 2016, berries were the largest and fastest growing retail produce. Driscoll’s leads the market. They credit their Driscoll’s Delight Platform. It tracks and manages the berries from the first mile (growing) through the middle miles (shipping) to the last mile (retail consumer). Driscoll’s analyzes data at every stage to optimize the production and consumption of berries. Driscoll’s is a technology company that sells berries.

Every company is in the application business. Applications need data. To design lighting, Osram uses data about your house. To deliver the best berries, Driscoll’s analyzes data about the farms (e.g. soil, climate), shipping (e.g. temperature and route), and customer preferences.

Modern businesses depend on applications. Applications depend on data. Therefore, modern businesses depend on data.

Driscoll's: A tech company that sells berries. Credit: Jacks Sachs, The New Yorker
Flooded data centers cause data loss. Photo Credit: Unknown

Data Protection: Because of Bad Things and Bad People

Every company protects their data center because there are so many ways to lose data.

CIOs have seen their companies suffer through catastrophes. Hurricane Harvey flooded Houston data centers. Hardware fails and sometimes catches fire. Software bugs corrupt data. People delete the presentation before the biggest meeting of their lives, so they throw a stone at a wasps’ nest to incite a swarm, get rushed to the hospital with dozens of vicious stings to have an excuse to re-schedule (or so I’ve heard).

IT organizations have also survived deliberate attacks. External hackers strike for fun and profit. Ransomware has become mainstream; cyber criminals can now subscribe to Ransomware as a Service! Now, anybody can become a hacker. Some attacks happen from inside, too. A terminated contractor at an Arizona bank destroyed racks of systems with a pickaxe. (I’ll never forget the dumbfounded CIO muttering, “We think he brought the pickaxe from home.” Because that’s what mattered.)

After decades of enduring data loss, IT knows to protect the data center. Do we also need to protect data in the cloud?


A data center destroyed by a fire. Photo Credit:

Data Protection: The Cloud Has Bad Things and Bad People

Every company needs to protect their data in the cloud because there are even more ways to lose it.

Bad things happen in the cloud. First, users still make mistakes. The cloud provider is not responsible for recovering from user error. Second, the cloud is still built of hardware and software that can fail. Vendors explain, “Amazon EBS volumes are designed for an annual failure rate (AFR) of between 0.1% — 0.2%, where failure refers to a complete or partial loss of the volume.” The applications you lose may be unimportant… or they may decimate your business. Third, since you are sharing resources, performance issues can affect data access. Amazon Prime Day is the most recent example. Finally, storms trigger data loss in a public cloud data center, just like they do in a corporate data center.

Public clouds are a bigger target for bad actors. Aggressive nations (with names that rhyme with Russia, Iran, North Korea, and China), bitcoin miners, and traditional criminals hack companies running in the cloud. Those hacks obliterate companies. Hackers deleted Code Space’s data in AWS. Two days later, the business shut down. Meanwhile, the scope of the public cloud makes internal threats more serious. The pickaxe (or virus)-wielding employee can now damage hundreds of companies instead of one!

Data is not any safer in the cloud than it is on-premises. Cloud providers try to protect your data, but it’s not enough. Even in the cloud, it’s your data. It’s your business. It’s your responsibility.

Protect the Cloud Data, Protect the Business

Modern businesses run on applications. Applications run on data. Most companies that lose data go out of business in 6 months or less.

Unfortunately, bad things and bad people destroy, steal, or disable access to the data. Whether you run on-premises or in the cloud, one day you will lose data. If you have a good backup and disaster recovery solution, you can recover the data. Your business can survive.

Amazon CTO Werner Vogels declared, “Everything fails all the time.” Companies need to protect their data in the cloud, so they can recover from those failures. Now, more than ever.

Data Protection - Business Protection in the Cloud. Photo Credit: