How An Entire Nation Became Russia's Test Lab for Cyberwar

It was a Saturday night last December, and Oleksii Yasinsky was sitting on the couch with his wife and teenage son in the living room of their Kiev apartment. The 40-year-old Ukrainian cybersecurity researcher and his family were an hour into Oliver Stone’s film Snowden when their building abruptly lost power.

“The hackers don’t want us to finish the movie,” Yasinsky’s wife joked. She was referring to an event that had occurred a year earlier, a cyberattack that had cut electricity to nearly a quarter-million Ukrainians two days before Christmas in 2015. Yasinsky, a chief forensic analyst at a Kiev digital security firm, didn’t laugh. He looked over at a portable clock on his desk: The time was 00:00. Precisely midnight.

Yasinsky’s television was plugged into a surge protector with a battery backup, so only the flicker of images onscreen lit the room now. The power strip started beeping plaintively. Yasinsky got up and switched it off to save its charge, leaving the room suddenly silent.

Noting the precise time and the date, almost exactly a year since the December 2015 grid attack, Yasinsky felt sure that this was no normal blackout. He thought of the cold outside -- close to zero degrees Fahrenheit -- the slowly sinking temperatures in thousands of homes, and the countdown until dead water pumps led to frozen pipes.

That’s when another paranoid thought began to work its way through his mind: For the past 14 months, Yasinsky had found himself at the center of an enveloping crisis. A growing roster of Ukrainian companies and government agencies had come to him to analyze a plague of cyberattacks that were hitting them in rapid, remorseless succession. A single group of hackers seemed to be behind all of it. Now he couldn’t suppress the sense that those same phantoms, whose fingerprints he had traced for more than a year, had reached back, out through the internet’s ether, into his home.

The Cyber-Cassandras said this would happen. For decades they warned that hackers would soon make the leap beyond purely digital mayhem and start to cause real, physical damage to the world. In 2009, when the NSA’s Stuxnet malware silently accelerated a few hundred Iranian nuclear centrifuges until they destroyed themselves, it seemed to offer a preview of this new era. “This has a whiff of August 1945,” Michael Hayden, former director of the NSA and the CIA, said in a speech. “Somebody just used a new weapon, and this weapon will not be put back in the box.”

And the blackouts weren’t just isolated attacks. They were part of a digital blitzkrieg that has pummeled Ukraine for the past three years -- a sustained cyber­assault unlike any the world has ever seen. A hacker army has systematically undermined practically every sector of Ukraine: media, finance, transportation, military, politics, energy. Wave after wave of intrusions have deleted data, destroyed computers, and in some cases paralyzed organizations’ most basic functions. “You can’t really find a space in Ukraine where there hasn’t been an attack,” says Kenneth Geers, a NATO ambassador who focuses on cybersecurity.

In a public statement in December, Ukraine’s president, Petro Poroshenko, reported that there had been 6,500 cyberattacks on 36 Ukrainian targets in just the previous two months. International cybersecurity analysts have stopped just short of conclusively attributing these attacks to the Kremlin, but Poroshenko didn’t hesitate: Ukraine’s investigations, he said, point to the “direct or indirect involvement of secret services of Russia, which have unleashed a cyberwar against our country.” (The Russian foreign ministry didn’t respond to multiple requests for comment.)

To grasp the significance of these assaults -- and, for that matter, to digest much of what’s going on in today’s larger geopolitical disorder -- it helps to understand Russia’s uniquely abusive relationship with its largest neighbor to the west. Moscow has long regarded Ukraine as both a rightful part of Russia’s empire and an important territorial asset a strategic buffer between Russia and the powers of NATO, a lucrative pipeline route to Europe, and home to one of Russia’s few accessible warm-water ports. For all those reasons, Moscow has worked for generations to keep Ukraine in the position of a submissive smaller sibling.

Yushchenko, who ended up serving as Ukraine’s president from 2005 to 2010, believes that Russia’s tactics, online and off, have one single aim: “to destabilize the situation in Ukraine, to make its government look incompetent and vulnerable.” He lumps the blackouts and other cyberattacks together with the Russian disinformation flooding Ukraine’s media, the terroristic campaigns in the east of the country, and his own poisoning years ago -- all underhanded moves aimed at painting Ukraine as a broken nation. “Russia will never accept Ukraine being a sovereign and independent country,” says Yushchenko, whose face still bears traces of the scars caused by dioxin toxicity. “Twenty-­five years since the Soviet collapse, Russia is still sick with this imperialistic syndrome.”

But many global cybersecurity analysts have a much larger theory about the endgame of Ukraine’s hacking epidemic: They believe Russia is using the country as a cyberwar testing ground -- a laboratory for perfecting new forms of global online combat. And the digital explosives that Russia has repeatedly set off in Ukraine are ones it has planted at least once before in the civil infrastructure of the United States.

One Sunday morning in October 2015, more than a year before Yasinsky would look out of his kitchen window at a blacked-out skyline, he sat near that same window sipping tea and eating a bowl of cornflakes. His phone rang with a call from work. He was then serving as the director of information security at StarLightMedia, Ukraine’s largest TV broadcasting conglomerate. During the night, two of StarLight’s servers had inexplicably gone offline. The IT administrator on the phone assured him that the servers had already been restored from backups.

But Yasinsky felt uneasy. The two machines had gone dark at almost the same minute. “One server going down, it happens,” Yasinsky says. “But two servers at the same time? That’s suspicious.”

Nonetheless, Yasinsky could see he’d been lucky. Looking at StarLight’s network logs, it appeared the domain controllers had committed suicide prematurely. They’d actually been set to infect and destroy 200 more PCs at the company. Soon Yasinsky heard from a competing media firm called TRK that it had been less fortunate: That company lost more than a hundred computers to an identical attack.

Soon Yasinsky began to hear from colleagues at other companies and in the government that they too had been hacked, and in almost exactly the same way. One attack had hit Ukrzaliznytsia, Ukraine’s biggest railway company. Other targets asked Yasinsky to keep their breaches secret. Again and again, the hackers used BlackEnergy for access and reconnaissance, then KillDisk for destruction. Their motives remained an enigma, but their marks were everywhere.

“With every step forward, it became clearer that our Titanic had found its iceberg,” says Yasinsky. “The deeper we looked, the bigger it was.”

As Lee busied himself with wedding preparations, he saw news headlines claiming that hackers had just taken down a power grid in western Ukraine. A significant swath of the country had apparently gone dark for six hours. Lee blew off the story -- he had other things on his mind, and he’d heard spurious claims of hacked grids plenty of times before. The cause was usually a rodent or a bird -- the notion that squirrels represented a greater threat to the power grid than hackers had become a running joke in the industry.

Lee eventually retreated to his mother’s desktop computer in his parents’ house nearby. Working in tandem with Assante, who was at a friend’s Christmas party in rural Idaho, they pulled up maps of Ukraine and a chart of its power grid. The three power companies’ substations that had been hit were in different regions of the country, hundreds of miles from one another and unconnected. “This was not a squirrel,” Lee concluded with a dark thrill.

No one knew much about the group’s intentions. But all signs indicated that the hackers were Russian: FireEye had traced one of Sandworm’s distinctive intrusion techniques to a presentation at a Russian hacker conference. And when FireEye’s engineers managed to access one of Sandworm’s unsecured command-and-control servers, they found instructions for how to use BlackEnergy written in Russian, along with other Russian-language files.

Most disturbing of all for American analysts, Sandworm’s targets extended across the Atlantic. Earlier in 2014, the US government reported that hackers had planted BlackEnergy on the networks of American power and water utilities. Working from the government’s findings, FireEye had been able to pin those intrusions, too, on Sandworm.

For Lee, the pieces came together: It looked like the same group that had just snuffed out the lights for nearly a quarter-­million Ukrainians had not long ago infected the computers of American electric utilities with the very same malware.

The Feds had also flown Assante in from Wyoming. Lee, a hotter head than his friend, had fought with the US agencies over their penchant for secrecy, insisting that the details of the attack needed to be publicized immediately. He hadn’t been invited.

On that first day, the suits gathered in a sterile hotel conference room with the staff of Kyivoblenergo, the city’s regional power distribution company and one of the three victims of the power grid attacks. Over the next several hours, the Ukrainian company’s stoic execs and engineers laid out the blow-by-blow account of a comprehensive, almost torturous raid on their network.

Once the circuit breakers were open and the power for tens of thousands of Ukrainians had gone dead, the hackers launched another phase of the attack. They’d overwritten the firmware of the substations’ serial-to-­ethernet converters -- tiny boxes in the stations’ server closets that translated internet protocols to communicate with older equipment. By rewriting the obscure code of those chunks of hardware -- a trick that likely took weeks to devise -- the hackers had permanently bricked the devices, shutting out the legitimate operators from further digital control of the breakers. Sitting at the conference room table, Assante marveled at the thoroughness of the operation.

The hackers also left one of their usual calling cards, running KillDisk to destroy a handful of the company’s PCs. But the most vicious element of the attack struck the control stations’ battery backups. When the electricity was cut to the region, the stations themselves also lost power, throwing them into darkness in the midst of their crisis. With utmost precision, the hackers had engineered a blackout within a blackout.

“The message was, ‘I’m going to make you feel this everywhere.’ Boom boom boom boom boom boom boom,” Assante says, imagining the attack from the perspective of a bewildered grid operator. “These attackers must have seemed like they were gods.”

That night, the team boarded a flight to the western Ukrainian city of Ivano-Frankivsk, at the foot of the Carpathian Mountains, arriving at its tiny Soviet-­era airport in a snowstorm. The next morning they visited the headquarters of Prykarpattyaoblenergo, the power company that had taken the brunt of the pre-Christmas attack.

So Yasinsky took a position as the head of research and forensics for a Kiev firm called Information Systems Security Partners. The company was hardly a big name. But Yasinsky turned it into a de facto first responder for victims of Ukraine’s digital siege.

Not long after Yasinsky switched jobs, almost as if on cue, the country came under another, even broader wave of attacks. He ticks off the list of casualties: Ukraine’s pension fund, the country’s treasury, its seaport authority, its ministries of infrastructure, defense, and finance. The hackers again hit Ukraine’s railway company, this time knocking out its online booking system for days, right in the midst of the holiday travel season. As in 2015, most of the attacks culminated with a KillDisk-style detonation on the target’s hard drive. In the case of the finance ministry, the logic bomb deleted terabytes of data, just as the ministry was preparing its budget for the next year. All told, the hackers’ new winter onslaught matched and exceeded the previous year’s -- right up to its grand finale.

The technician picked up the black desk phone to his left and called an operator at Ukrenergo’s headquarters to alert him to the routine mishap. As he did, another light turned green. Then another. Zaychenko’s adrenaline began to kick in. As he hurriedly explained the situation to the remote operator, the lights kept flipping: red to green, red to green. Eight, then 10, then 12.

As the crisis escalated, the operator ordered Zaychenko to run outside and check the equipment for physical damage. At that moment, the 20th and final circuit switched off and the lights in the control room went out, along with the computer and TV. Zaychenko was already throwing a coat over his blue and yellow uniform and sprinting for the door.

But the brevity of the outage was virtually the only thing that was less menacing about the 2016 blackout. Cybersecurity firms that have since analyzed the attack say that it was far more evolved than the one in 2015: It was executed by a highly sophisticated, adaptable piece of malware now known as "CrashOverride," a program expressly coded to be an automated, grid-killing weapon.

Lee’s critical infrastructure security startup, Dragos, is one of two firms that have pored through the malware's code; Dragos obtained it from a Slovakian security outfit called ESET. The two teams found that, during the attack, CrashOverride was able to “speak” the language of the grid’s obscure control system protocols, and thus send commands directly to grid equipment. In contrast to the laborious phantom-mouse and cloned-PC techniques the hackers used in 2015, this new software could be programmed to scan a victim’s network to map out targets, then launch at a preset time, opening circuits on cue without even having an internet connection back to the hackers. In other words, it's the first malware found in the wild since Stuxnet that's designed to independently sabotage physical infrastructure.

Marina Krotofil, an industrial control systems security researcher for Honeywell who also analyzed the Ukrenergo attack, describes the hackers’ methods as simpler and far more efficient than the ones used in the previous year’s attack. “In 2015 they were like a group of brutal street fighters,” Krotofil says. “In 2016, they were ninjas.” But the hackers themselves may be one and the same; Dragos’ researchers have identified the architects of CrashOverride as part of Sandworm, based on evidence that Dragos is not yet ready to reveal.

For Lee, these are all troubling signs of Sandworm’s progress. I meet him in the bare-bones offices of his Baltimore-based critical infrastructure security firm, Dragos. Outside his office window looms a series of pylons holding up transmission lines. Lee tells me that they carry power 18 miles south, to the heart of Washington, DC.

For the first time in history, Lee points out, a group of hackers has shown that it’s willing and able to attack critical infrastructure. They’ve refined their techniques over multiple, evolving assaults. And they’ve already planted BlackEnergy malware on the US grid once before. “The people who understand the US power grid know that it can happen here,” Lee says.

In fact, in its analysis of CrashOverride, ESET found that the malware may already include one of the ingredients for that kind of destructive attack. ESET’s researchers noted that CrashOverride contains code designed to target a particular Siemens device found in power stations -- a piece of equipment that functions as a kill-switch to prevent dangerous surges on electric lines and transformers. If CrashOverride is able to cripple that protective measure, it might already be able to cause permanent damage to grid hardware.

An isolated incident of physical destruction may not even be the worst that hackers can do. The American cybersecurity community often talks about “advanced persistent threats”—sophisticated intruders who don’t simply infiltrate a system for the sake of one attack but stay there, silently keeping their hold on a target. In his nightmares, Lee says, American infrastructure is hacked with this kind of persistence: transportation networks, pipelines, or power grids taken down again and again by deep-rooted adversaries. “If they did that in multiple places, you could have up to a month of outages across an entire region,” he says. “Tell me what doesn’t change dramatically when key cities across half of the US don’t have power for a month.”

In fact, in its analysis of CrashOverride, ESET found that the malware may already include one of the ingredients for that kind of destructive attack. ESET’s researchers noted that CrashOverride contains code designed to target a particular Siemens device found in power stations -- a piece of equipment that functions as a kill-switch to prevent dangerous surges on electric lines and transformers. If CrashOverride is able to cripple that protective measure, it might already be able to cause permanent damage to grid hardware.

An isolated incident of physical destruction may not even be the worst that hackers can do. The American cybersecurity community often talks about “advanced persistent threats” -- sophisticated intruders who don’t simply infiltrate a system for the sake of one attack but stay there, silently keeping their hold on a target. In his nightmares, Lee says, American infrastructure is hacked with this kind of persistence: transportation networks, pipelines, or power grids taken down again and again by deep-rooted adversaries. “If they did that in multiple places, you could have up to a month of outages across an entire region,” he says. “Tell me what doesn’t change dramatically when key cities across half of the US don’t have power for a month.”

Yasinsky says he has tried to maintain a dispassionate perspective on the intruders who are ransacking his country. But when the blackout extended to his own home four months ago, it was “like being robbed,” he tells me. “It was a kind of violation, a moment when you realize your own private space is just an illusion.”

Yasinsky says there’s no way to know exactly how many Ukrainian institutions have been hit in the escalating campaign of cyberattacks; any count is liable to be an underestimate. For every publicly known target, there’s at least one secret victim that hasn’t admitted to being breached -- and still other targets that haven’t yet discovered the intruders in their systems.

Bracing for the next round, Yasinsky says, is like “studying for an approaching final exam.” But in the grand scheme, he thinks that what Ukraine has faced for the past three years may have been just a series of practice tests.

He sums up the attackers’ intentions until now in a single Russian word: poligon. A training ground. Even in their most damaging attacks, Yasinsky observes, the hackers could have gone further. They could have destroyed not just the Ministry of Finance’s stored data but its backups too. They probably could have knocked out Ukrenergo’s transmission station for longer or caused permanent, physical harm to the grid, he says -- a restraint that American analysts like Assante and Lee have also noted. “They’re still playing with us,” Yasinsky says. Each time, the hackers retreated before accomplishing the maximum possible damage, as if reserving their true capabilities for some future operation.

Many global cybersecurity analysts have come to the same conclusion. Where better to train an army of Kremlin hackers in digital combat than in the no-holds-barred atmosphere of a hot war inside the Kremlin’s sphere of influence? “The gloves are off. This is a place where you can do your worst without retaliation or prosecution,” says Geers, the NATO ambassador. “Ukraine is not France or Germany. A lot of Americans can’t find it on a map, so you can practice there.” (At a meeting of diplomats in April, US secretary of state Rex Tillerson went so far as to ask, “Why should US taxpayers be interested in Ukraine?”)

In that shadow of neglect, Russia isn’t only pushing the limits of its technical abilities, says Thomas Rid, a professor in the War Studies department at King’s College London. It’s also feeling out the edges of what the international community will tolerate. The Kremlin meddled in the Ukrainian election and faced no real repercussions; then it tried similar tactics in Germany, France, and the United States. Russian hackers turned off the power in Ukraine with impunity -- and, well, the syllogism isn’t hard to complete. “They’re testing out red lines, what they can get away with,” Rid says. “You push and see if you’re pushed back. If not, you try the next step.”

What will that next step look like? In the dim back room at ISSP’s lab in Kiev, Yasinsky admits he doesn’t know. Perhaps another blackout. Or maybe a targeted attack on a water facility. “Use your imagination,” he suggests drily.

This article appears in the July 2017 issue. Subscribe now.