How IT Outages Affect Business Operations

We lost millions on New Year’s Eve. Our ad servers crashed. Everything went down. No impressions. No revenue. On the single biggest ad day of the year. We were a mobile ad tech company moving hundreds of millions. But that night? • Our best engineers were partying • No one was on call • AWS alerts were misconfigured • PagerDuty didn’t go off • And no one noticed until I checked the dashboard the next morning: $0 revenue The team thought it was a frontend bug… Until angry customers started calling! That one outage triggered a domino effect: • We lost our biggest advertiser • Publishers jumped to Applovin • Our reputation cratered • The board demanded blood • We spent a fortune on consultants • We fired our CTO • Paused our roadmap for 6 months That night changed how I think about infrastructure forever. If you’re building anything that runs at scale, learn from our scars. 5 lessons I wish we knew earlier: 1. If you make money while you sleep, someone needs to be awake. Holidays don’t apply to production. Build a real on-call system with teeth. 2. If your customer notices the outage before you do, you’ve already failed. Monitoring is a product. Treat it like one. 3. Latency is a product issue. Uptime is a company issue. Founders should obsess over “5-9s” (99.999% uptime) the way they obsess over MRR. 4. Run game days. Simulate disasters. Practice escalation. Know what failure looks like. 5. DevOps isn’t a role. It’s a culture. If only one person knows how your system stays online, you’re already offline. You just don’t know it yet. We rebuilt from the ground up and also changed our culture: AWS with Azure failover. System status dashboard accessible to our BOD. Expectations clearly set that work/life balance doesn’t apply during an emergency. We never blinked during another holiday. If you’re building something important, DevOps is not optional. Don’t wait for a disaster to take it seriously. I did - and it cost me millions. Most expensive lesson ever learned!

Slack went down, and the internet panicked. But let’s be honest—this isn’t about Slack. It’s about how fragile our business operations have become when a single tool suddenly disappears. I’ve seen this play out before. A company I worked with relied heavily on a single SaaS vendor for all internal and external communication. When that platform went down—just for a few hours—it disrupted customer service, stalled sales deals, and even delayed compliance reporting. The aftermath? Scrambling to recover, frustrated clients, and a whole lot of “Why didn’t we have a backup plan?” First, redundancy is non-negotiable. Every business should have an alternative communication channel ready—whether it’s email, a second chat tool, or even (gasp) the phone. Second, product counsel should be in the room when these tools are selected. The legal team isn’t just there to review contracts—we should help assess risk, negotiate protections, and push for backup plans before disaster strikes. Third, train your teams. A business continuity plan only works if people know how to execute it. If your team’s response to an outage is “Now what?”—that’s a problem. If you’re rethinking your reliance on SaaS after this outage, good. It’s time for legal, IT, and product teams to work together to build a more resilient operation. For my take on the legal implications of SaaS downtime, check out the video—because trust me, contracts matter when things go south. -------- 💥 I’m Olga V. Mack 🔺 Expert in AI & transformative tech for product counseling 🔺 Upskilling human capital for digital transformation 🔺 Leading change management in legal innovation & operations 🔺 Keynote speaker on the intersection of business, law, & tech 🔝 Let’s connect 🔝 Subscribe to Notes to My (Legal) Self newsletter

Instead of starting with threats or systems, I start with the value stream. Why? Because business continuity isn’t really about hurricanes, power outages, or servers going down. It’s about something much simpler: preserving the flow of value through the business. Executives don’t care which database is offline. They care that customers can’t buy, contracts can’t close, or invoices can’t be sent. That’s the flow you’re protecting. Here’s how I break it down: 1️⃣ Identify the process that directly supports revenue or mission-critical outcomes. - What activity actually creates value? - For a SaaS platform, it might be the software deployment pipeline. - For a manufacturer, it might be raw materials through production to distribution. - For a hospital, it might be patient intake → treatment → billing. 2️⃣ Map each step in that process — people, systems, vendors, tools. - Who touches this? - What tech or suppliers does it rely on? - Where are the single points of failure? 3️⃣ Estimate what percentage of the company’s total revenue depends on this process. - If it fails, how much of your annual revenue would actually pause or disappear? - Is it a core process that drives 80% of revenue or a supporting function tied to 10%? 4️⃣ Estimate how much of that revenue is at risk in a realistic disruption. - Will you lose all revenue immediately? - Or just delay it? - Be conservative and credible — executives hate inflated numbers. 5️⃣ Spread that loss over operating hours to create an hourly cost of disruption. - Take the annual revenue at risk, divide it by 8,760 hours (for 24/7 ops) or by working hours for narrower processes. - Then add recovery costs (staff overtime, consultants) and reputational or compliance penalties. What you end up with isn’t perfect — but it’s credible. It turns abstract “criticality” into a number: This process costs $X per hour when it’s disrupted. Why this works: ✅ It sidesteps technical jargon — you’re talking value, not servers. ✅ It reframes continuity as a business problem, not an IT problem. ✅ It gives executives a simple, repeatable model to prioritize investments. ✅ And yes, it’s executive-friendly — because it speaks in dollars, not downtime. I’ll walk through a concrete example in my next post. But first, let me ask you — what would you add or improve in this approach? Have you seen a better way to make the financial case for continuity?

I'm proud of the series of highly informative reports that Cockroach Labs puts out annually. We started with the first truly objective and quantitatively benchmarked Cloud Report, that compared the price versus performance of the hyperscalers. We followed that up last year with a report that surveyed the true state of the industry's adoption of multi-cloud. This year, we're at it again, with a groundbreaking report on the state of resilience, and the results are astonishing. We surveyed 1,000 IT executives across North America, EMEA, and APAC and discovered that 100% of companies surveyed reported financial losses due to downtime. Companies faced an *average* of 86 incidents per years, at an average cost of $495,000 per incident. 14% experienced daily interruptions, with average downtime of over three hours! These results are a wake-up call that every company should be paying attention to. This report comes on the heels of Crowdstrike's inadvertent demonstration of the risks of over-reliance on technical monocultures. The only solution that can truly see around corners and protect your business from unknown risks is diversification – businesses should build with cloud-native technologies which allow use cases to span nodes, datacenters, regions, and even cloud providers. The report is packed with eye-watering statistics, but it also demonstrates how investing in your operational resilience strategy can reduce not only financial risks, but operational and reputational risks as well. I recommend anyone involved in technology and cloud infrastructure read this report. The conditions which cause significant outages continue to rapidly evolve, but businesses that modernize their data architectures can redefine disaster scenarios as IT resilience. https://cockroa.ch/4hnbOby 

Your cloud provider just went dark. What's your next move? If you're scrambling for answers, you need to read this: Reflecting on the AWS outage in the winter of 2021, it’s clear that no cloud provider is immune to downtime. A single power loss took down a data center, leading to widespread disruption and delayed recovery due to network issues. If your business wasn’t impacted, consider yourself fortunate. But luck isn’t a strategy. The question is—do you have a robust contingency plan for when your cloud services fail? Here's my proven strategy to safeguard your business against cloud disruptions: ⬇️ 1. Architect for resilience  - Conduct a comprehensive infrastructure assessment - Identify cloud-ready applications - Design a multi-regional, high-availability architecture This approach minimizes single points of failure, ensuring business continuity even during regional outages. 2. Implement robust disaster recovery - Develop a detailed crisis response plan - Establish clear communication protocols - Conduct regular disaster recovery drills As the saying goes, "Hope for the best, prepare for the worst." Your disaster recovery plan is your business's lifeline during cloud crises. 3. Prioritize data redundancy - Implement systematic, frequent backups - Utilize multi-region data replication - Regularly test data restoration processes Remember: Your data is your most valuable asset. Protect it vigilantly. As Melissa Palmer, Independent Technology Analyst & Ransomware Resiliency Architect, emphasizes, “Proper setup, including having backups in the cloud and testing recovery processes, is crucial to ensure quick and successful recovery during a disaster.” 4. Leverage multi-cloud strategies - Distribute workloads across multiple cloud providers - Implement cloud-agnostic architectures - Utilize containerization for portability This approach not only mitigates provider-specific risks but also optimizes performance and cost-efficiency. 5. Continuous monitoring and optimization - Implement real-time performance monitoring - Utilize predictive analytics for proactive issue resolution - Regularly review and optimize your cloud infrastructure Remember, in the world of cloud computing, complacency is the enemy of resilience. Stay vigilant, stay prepared. P.S. How are you preparing your organization to handle cloud outages? I would love to read your responses. #cloud #cloudmigration #cloudstrategy #simform PS. Visit my profile, Hiren, & subscribe to my weekly newsletter: - Get product engineering insights. - Catch up on the latest software trends. - Discover successful development strategies.