Preparing for the Unpredictable: Outage Lessons for Healthcare Tech Leaders

Table of Contents

In the wake of global technology disruptions, it’s imperative that CIOs reassess and strengthen their preparedness strategies. Now’s the time for technology leaders in healthcare to stop and ask: What if another incident like this strikes and has a significant impact on my organization? And if that happens, will we be prepared to mitigate its impact?

In healthcare, major outages have the potential to shut down entire health and hospital systems, risking patient lives. We’re already on high alert for cyber attacks on healthcare organizations. These outages remind us that now is a prime time to reboot your preparedness tactics and stay vigilant.

Update Your Preparedness

Business Continuity and Disaster Recovery Plans
Your Business Continuity and Disaster Recovery (BC/DR) plan is your lifeline during a crisis. It should encompass technical procedures, customer communication strategies, and internal messaging protocols. Clearly defining roles, responsibilities, and rationales ensures seamless execution during an emergency.

Continuous Training and Skill Development
The most recent IT outage underscored the necessity of manually deployed fixes, which extended restoration times. Maintaining current engineering training and skills across all aspects of your IT infrastructure—whether legacy or cloud—is crucial. Continuous training closes skill gaps and ensures your team can address and resolve issues efficiently.


Develop a Better Early Warning System

Comprehensive Observability
Implement comprehensive infrastructure and application observability with defined thresholds for key customer-facing workloads. Moving beyond basic CPU and memory usage monitoring, focus on critical user journeys. Establish and measure appropriate thresholds to enable early alerting and proactive issue resolution.

Performance Monitoring Websites
Utilize performance monitoring websites like DownDetector. Integrate content from these sites as RSS feeds and email alerts to identify patterns during significant outages across different reporting groups. This proactive approach helps in anticipating and mitigating potential issues before they escalate.

Cloud Vendor Service Health Monitoring
Monitor the health of your Cloud vendor services through service health dashboards (AWS: AWS Health Dashboard, Azure: Azure Status, GCP: GCP Status). Subscribe to email updates or integrate RSS feeds into collaboration tools like MS Teams and Slack for real-time updates.

Synthetic Traffic Monitoring
For critical workloads, employ synthetic traffic monitoring. Simulate traffic to replicate critical user journeys, establish thresholds, and configure alerts for deviations. This proactive approach ensures that issues are promptly addressed by your operations team.

Be Ready for Second-Order Actors

Opportunistic Resolvers
In the aftermath of visible incidents, numerous companies and individuals may offer services and support. While their intentions might not be inherently bad, introducing new vendors without thorough vetting can expose your infrastructure and applications to increased vulnerabilities. Rely on trusted advisors or well-referenced sources to secure necessary resources.

Threat Actors Exploiting Vulnerabilities
Publicized incidents attract threat actors eager to exploit both technical and human vulnerabilities. Reinforce operational messaging with your team and ensure your security program integrates this new risk landscape into your Security Operations Center (SOC) protocols.

Together, We Can Empower Healthcare Resilience in an Uncertain World

When widespread technology outages hit healthcare, the stakes couldn’t be higher. It’s clear that right now, we cannot predict the disruptions that may be headed our way, whether we’re talking cyber attackers or technology vendors pushing faulty software updates. But we can stand at the ready for whatever comes our way with proactive measures to ensure continuity and resilience. In such uncertain times, staying prepared is how we ensure healthcare resiliency.

Thank you for subscribing!