The front lines of a dark new online war
Ransomware attacks against healthcare facilities are rising at triple-digit rates (PDF). We spoke with experts, IT professionals, and incident commanders about what happens during and after a ransomware attack on healthcare facilities. Since hospital systems frequently face class action lawsuits and legal challenges, and they’re involved in investigations that border on national security, none of the details presented here are specific to any single institution. However, the experience detailed here is real and based on firsthand experiences.
Detection and initial response
Amid the long struggle against COVID-19, an IT worker on the night shift at a large hospital noticed something unusual.
A series of password reset attempts on staff laptops and accounts were being logged almost in real time. They were all in close succession across multiple users, and long after the day shift staff had left work.
Suspicious, IT staff started doing what most IT people instinctively do:
- Look for patterns
- Check IP addresses against geolocations
- Identify staff they could contact to see if they were having issues
- Monitor network traffic from far-flung regions or suspicious locations
What ensued was a series of back-and-forth skirmishes between a small night shift IT staff and a likely equally small team of pirate hackers somewhere else in the world. This hospital was under cyberattack.
If the Internet is a vast digital ocean, these opposing teams were akin to two hostile naval vessels on a chance encounter, circling each other in an attempt for supremacy. Salvos were fired and blocked, traps were laid, and systems and lives risked critical damage.
The attackers were setting up packages — small applications, usually designed to work in the background of servers, computers, or other hardware — to help get access to various systems. Like burglars rummaging through a house, there was no single target. Just anything that looked valuable, from staff email, HR records, shared network drives, phone lines, payroll, patient medical records, to billing systems and even door lock systems. Anything that could be exploited is identified as fair game for leverage. The goal is to gain enough control of something critical to ransom the hospital for money in an expensive, high-stakes hostage situation.
IT staff at the hospital in these situations can only operate on the defensive. Like a battleship without any ammunition, there isn’t much IT staff can do to “attack back” to shut an opponent down. Nor are they in the business of attacking. They’re in the business of securing systems that provide healthcare. But in order to evade and defend, they first have to know what they’re dealing with. Tonight it’s almost as if someone snuck up on the stern in total darkness — and no one noticed until it was too late. The order is to lock systems down.
To “lock systems down” is a broad term that borders on being meaningless. On servers, files can sometimes be set to “read-only”, meaning only people can see or “read” the files, but not “write” or change anything. For most hardware, like phones and laptops, “locking a system down” more or less means “disconnecting it from the Internet” or physically shutting it off. This is what IT staff chose to do, physically turning off, unplugging, or disconnecting computers, servers, and hardware from the outside world in descending order of perceived importance, from patient and employee records to email and internal messaging.
So long as the problem wasn’t inside the building, they reasoned, at least it would make the world seem smaller for a little while. A total system shutdown, however, posed life-threatening challenges of its own.
Incident Command is activated
The phone rings around 2 or 3 a.m. “We have a situation.”
Bleary-eyed, the hospital’s chief incident commander rolls over in bed and realizes the gravity of the situation. They’ve trained for scenarios like this. Tabletop exercises designed and led by Vantage Point staff have helped them think about these problems before. The hospital IT staff has also been there in those and other exercises, as every incident so frequently involves an IT component. An Incident Command Center is spun up, operating under the same procedure as rehearsed.
For most hospitals, Incident Command Centers were already operating 24/7 during COVID-19. This hospital is barely out of this situation. It’s still facing supply constraints, staff are tired, morale is low as the nation turns against masking, and demand for services is back at an all-time high.
Coming into the office, the Incident Commander is briefed on a cascade of severe realities:
- All of the hospital’s IT systems are in “full downtime.”
- The night staff are facing severe disruptions.
- Clinics and scheduled surgeries are set to start in a mere 3-4 hours.
- No one’s sure who was involved in the attack.
- No one’s sure what systems are compromised.
- Almost none of the staff know what is happening, and,
- There is almost no way to communicate what is happening to the staff.
The Incident Commander knows most clinicians, the medical staff, and hospital leadership don’t understand a lot of the technology any more than the IT staff understands how to do a Whipple procedure or conduct a colonoscopy. “Full downtime” is a phrase that is both soft and obvious, and offers only a hint to the gravity of what’s ahead.
Operational challenges become apparent to staff and patients
As hospital staff learned of the suspected attack, mostly through in-person meetings, face-to-face hearsay, and sporadic text messages to staff’s personal phone numbers through a third-party service that wasn’t on-site or part of the hospital’s core systems, the reality of “full downtime” sinks in.
- There is no WiFi or internet access anywhere.
- There is no way to send emails.
- Patient charts and Electronic Health Records are totally inaccessible.
- Surgeries must be canceled, but it’s impossible to look up most patients’ phone numbers to call them.
- Clinics must be closed because there’s no way to send prescriptions to pharmacies.
- Some of the building’s security systems, such as locks, keypads, and cameras, must be manually overridden and staffed.
- X-ray machines, MRIs, CAT scans, and lab tests are piling up because they either won’t work, can’t work, or can’t send or share the results.
And no one knows how long this will last. Nurses and physicians who yesterday walked into a modern hospital are now walking into a facility with the same technological prowess as a hospital from 1985. To everyone’s surprise, the only system that was still working was the phone, despite being a digital Voice Over IP (VoIP) line.
Risks to patient care mount
Technology generally saves lives. Digital pharmacy records ensure patients can’t change a prescription for 10 morphine pills to 100, or that pharmacists don’t misread a doctor’s note. What used to require people to know a lethal dose has been supplanted by a never-tired, never-confused computer that knows the precise minimum and maximum doses of every medication on the planet.
Even for veteran medical staff, being flung back in time has severe consequences for patient care. As one example, no pharmacy willingly accepts paper prescriptions anymore. And even if they did, the hospital can’t print the scripts because the need for printers has reduced so much that there aren’t as many of them in the building. Plus, the network is disconnected, so printing requires plugging a computer directly into scarce printers.
Now patients are being turned away and ambulances are on diversion to other hospitals. The financial loss to the hospital is mounting. For facilities in large, urban environments the diversions can be manageable. But already under strain from active COVID protocols, the loss of any healthcare facility seems unimaginable. And state and federal laws surrounding controlled substances, patient records, and privacy don’t diminish just because the hospital is under attack.
It’s in this environment that German officials declared the world’s first official casualty of a ransomware attack in 2020. An ambulance in Düsseldorf, Germany responded to the deteriorating condition of a 78-year-old woman suffering from an aortic aneurysm. The ambulance technicians radioed ahead to the local university hospital to inform staff of their impending arrival. They were told that the emergency department was closed, so they couldn’t accept the patient.
Instead, the ambulance was diverted to Helios University Hospital in Wuppertal, 20 miles away, which delayed the patient’s treatment by an hour. She died shortly after.
As one hospital representative told us, “Anytime there is an interruption in technology, there’s an increased risk [of death]. And we’ve built so many safeguards into the technology that without it you lose some safeguards.”
Law enforcement reinforcements arrive
Back in the Incident Command Center, a series of phone calls were being made to bring in reinforcements. Rather than call local police, IT staff already had a working relationship with special agents from the FBI and the U.S. Cybersecurity and Infrastructure Security Agency (CISA).
The hospital’s legal team was brought in. Some staff didn’t recognize it at the time, but having the legal team in the room began protecting the hospital through attorney-client privilege and the benefit of their perspective. The cybersecurity insurance company was called and summoned vendors who specialize in investigating and restoring ransomed systems.
Hospital leadership, communications, and department managers arrived to help coordinate plans and next steps. TV news vans began parking out front of the ER.
The FBI and CISA consulted their profile books. Not unlike TV crime dramas, they start asking the IT staff how it started, what patterns were recognized, what kinds of hardware they went after, and other details like spelling and language quirks.
Eventually, a picture emerges about how the suspected group works, what they’re likely to ask for, how they’ll ask, and when. As the insurance company’s cybersecurity vendors lend a hand in sifting through potentially millions of emails, messages, logs, IP addresses, and more digital detritus, the clues pile up.
Left unsaid to all but the top officials in the Command Center are who the people are and where they are.
Cybercriminals rarely operate in the United States and attack US infrastructure since US law enforcement can easily capture and prosecute them. Most domestic cybercriminals are usually just bored kids who treat this as “play”. Rather, cybercriminals are usually operating from parts of Eastern Europe, Russia, China, North Korea, and a sliding scale of other countries most Americans will never visit. The rogue hackers and quasi-agents sitting at their keyboards may not officially be with their nation’s government, but they are frequently supported by them financially or by being ignored by that country’s law enforcement. And they enjoy the protection that comes from being almost invisible.
“We just don’t know”
Despite the vast wealth of background information available over what became a week-long shutdown of most of the hospital’s systems, most conversations still started with “We think…” and “We’re not sure, but…”
To help rank-and-file staff understand what worked and what didn’t as some systems were manually reviewed and re-connected, a series of meetings, text messages, or cards were printed and hung up to indicate red, yellow, and green levels of confidence and operational efficiency.
The IT staff who spotted the suspicious password resets in the middle of the night had managed to defend and evade their cyber foes long enough that the attackers could never strike the final blow and share what they wanted and how. The final “Pay us this amount in Bitcoin” message never appeared. Agents believe when the attackers realized they weren’t going to be able to finish their ransomware attack they “just tried to break stuff” instead and, luckily, no sensitive or personally identifying data was lost.
But it wasn’t without severe disruption to the staff and patients during the week-long investigation. Payroll is frequently delayed in these scenarios, which causes severe financial strain on everyone from the custodial and food service workers to nurses and medical students with significant medical school loan payments. And then there are patients who just simply slip through the cracks because a follow-up call was never made or a lab result got backlogged.
On the frontlines of a new cyber war
Ransomware attacks on U.S. hospitals and healthcare systems might appear to reasonable people as terrorist attacks. They are no less severe or detrimental to American interests than an attack on the electric grid or air travel. And in many ways, may be more impactful to everyday Americans than bombings on physical military targets, since more people are involved and the scope is much larger.
American officials are not keen to call these attacks “terrorism”, however. As one source told us, “I think if they call it terrorism people will rightly demand a response. And I don’t think anyone really knows how to respond.”
That dire assessment means the world’s digital ocean is secured foremost by alert IT staff that bubbles up and down to CISA, the FBI, the CIA, NSA, and other U.S. agencies. Still, a proportional response to a ransom threat from independent cyber criminals working silently in a Shanghai or Moscow apartment leased by their government and otherwise largely ignored is almost impossible to counter and fraught with political peril.
Lessons learned and future preparedness
Hospitals will continue to be soft targets for cybercriminals. Hospitals:
- Process a lot of valuable, personal information worth selling on the black market
- Handle millions of dollars in billing
- Hold vast stores of controlled substances and drugs
- Are critical pieces of infrastructure
- Usually have limited resources, healthcare IT staff, and overall staff time
- Hospitals have shown a willingness to pay ransoms in the past, encouraging more attacks
For staff inside hospitals, their best defenses are akin to the air raid drills of the 1950s: practice, preparation, and vigilance.
- Practicing tabletop exercises and live-action drills can help identify many of the failure points most people don’t think about, like how to print paper without WiFi.
- Another point of preparation is ensuring IT staff are trained, capable, and prepared to work with the Incident Command System to ensure timely communication.
- Further preventative measures include robust, up-to-date security, hardware, software, and employee training in hospital cybersecurity.
As one representative told us, “Recognize these scenarios will always take longer than you think. And force yourself to drill when you least want to, like at 10 am on a Monday. People may say practicing a network shutdown during the day isn’t safe, but that should give you some indication of what’s at stake.”
Vantage Point Consulting conducts cybersecurity drills and tabletop exercises for healthcare facilities and industrial operations of all sizes. These drills are tailored to your needs and legal demands and frequently pose challenging “What if” scenarios designed to keep people prepared.
