The SolarWinds attack is shaping up to be the worst cyber incident the US has ever suffered. But what does it teach us?

In December 2020, FireEye discovered a massive ongoing hack (though other companies have also seen indications earlier) stealing sensitive documents, data, and security tools from their systems, among others via a SolarWinds product. This supply chain attack was targeting a large number of corporations and governmental entities in the United States (including FireEye itself) as well as in the other parts of the world.

The attack vector drawing the most attention was a stealthy piece of malware active in the SolarWinds Orion network management software that essentially gave the attackers full remote access to the machines running the software. The backdoored version of SolarWinds Orion was deployed as a regular update in March 2020 – and it stayed unnoticed for almost nine months!

Security experts such as Bruce Schneier and Paul R. Kolbe claim that the only way forward is recognizing the new status quo, and building appropriate cyber security defenses instead of just focusing on offense. On the other hand, in a recent Senate hearing, Microsoft president Brad Smith stated that “[the hack was] not just an attack on specific targets, but on the trust and reliability on the world’s critical infrastructure”, specifically calling out the so-called supply chain attack as an activity that should be outlawed, even as part of espionage. But either way, software like SolarWinds Orion can be considered part of a country’s critical infrastructure.

Of course, this is not the first time of a state-sponsored attack (or APT – Advanced Persistent Threat) compromising the supply chain on a larger scale. In particular, the attack is following the Petya playbook from 2017 quite closely. In that ransomware attack, attackers managed to insert a malicious backdoor into an update for the widely used M.E.Doc accounting software with devastating consequences all across Europe (but especially in Ukraine), causing over $10 billion dollars in damage.

There were several vectors used during the attack beyond SolarWinds Orion (including the exploitation of several vulnerabilities in Microsoft and VMWare products) and investigation is still ongoing. But let’s focus on the SolarWinds part of the story for now, and see how this supply chain attack took place, and what you can do to avoid something like this happening to your own product.

From sneaky sunspots to burnt wings

In a supply chain attack, the attacker implants a malicious backdoor into a component that the target organization eventually uses as a dependency. At a basic level, a supply chain attack exploits the implicit trust an organization has in third-party (software or hardware) resources.

The malware in question was SUNBURST, and its insertion into SolarWinds Orion was done by compromising SolarWinds’ automated build process. Specifically, another piece of malware called SUNSPOT was installed on the SolarWinds build servers; this malware injected the backdoor into a source code file just before compilation. We recommend CrowdStrike’s technical analysis for a detailed description of SUNSPOT’s operation.

Researchers also identified that a trial run of the malware already took place in October 2019 – at that time the attackers were very careful to cover their tracks and they avoided doing anything that would draw attention.

Of course, the question still remains on how they originally installed the malware. At the time of writing this article the original point of the breach is still unknown. There are some wild guesses though, some of them hinting at poor security practices at SolarWinds – for example, the password for the Orion software update server was allegedly ‘solarwinds123’ for over a year as reported by a security researcher. The credentials to SolarWinds machines were also making the rounds in underground forums in 2017. Of course, any of the components used by the build system could’ve been compromised, too. The New York Times brought up the TeamCity CI tool by JetBrains as a possible vector, though JetBrains has publicly denied this possibility. The APT responsible for such an elaborate attack certainly had the resources to exploit zero-day vulnerabilities. And there were also other vulnerabilities (Microsoft, VMWare, and others) that were exploited in some other vectors used in this operation.

Which then raises the question: if this attack was so widespread and if it targeted companies and governmental entities with strong security measures in place, how didn’t anyone notice?

The first part of the answer is in the nature of the target, SolarWinds. Their product (a network administration tool) was used as part of IT administration activities, implying a high level of trust – all of the malicious activity was taking place under the guise of legitimate users. Cyber security firms like FireEye are always popular targets for the same reason. And of course, using security software such as web application firewalls and IDS does not guarantee security.

But that’s only half of the explanation – the other half is the concept of the supply chain attack itself.

For Want of a Nail

A supply chain attack is difficult to deal with because of the implicit trust in the software build process. This is not a new problem. The 1984 Ken Thompson lecture Reflections on Trusting Trust described an attack where a compromised C compiler injected a backdoor during the compilation process. Since the compiler itself was written in C, it could also compile the backdoor-injecting code into its own code when its source code was updated. And – as the compiler is an inherently trusted part of the toolchain – this backdoor could potentially never be revealed unless someone took the trouble of going through its disassembled code line by line.

But even if we don’t trust the compiler, this rabbit hole can go very deep indeed. For example, it’s possible to put backdoors in cryptographic algorithms – see the story of Dual_EC-DRBG. The possibility of backdooring the computer hardware itself has been demonstrated in practice in 2008 (see Designing and implementing malicious hardware). Consider that modern manufacturing processes are also controlled by software!

The SolarWinds attack implemented a form of the attack described by Thompson: while the compiler itself was not technically compromised, the malware essentially acted as a malicious preprocessor during the build process.  And of course, after compiling the malicious code into SolarWinds Orion, it could run with the high privileges of Orion, which could allow it to quietly disable (or just hide its activities from) IDS and other types of security software as well.

The supply chain attack and dependency hell

Of course, not every software company develops software used in a critical infrastructure context. But even then, a supply chain attack can be devastating. This is becoming more and more popular with a higher reliance on third-party components – the British Airways incident in 2018 is a good example.

Most software written today makes use of a large number of third-party components: modules, libraries, or packages. And with hundreds of components, it can be much easier for a motivated attacker to compromise just one of them without being detected.

From xkcd

One relatively straightforward path for the attacker is to take over maintenance of an abandoned module, and sneakily inject a backdoor under the guise of a patch. This happened in 2018 where an attacker backdoored the event-stream npm package to steal cryptocurrency from wallets. And while theoretically the open-source nature of these packages would make such manipulation obvious, in practice there just aren’t many developers doing in-depth code reviews of every single commit to spot potential backdoors. And even if they do, a Node.js thought experiment from 2018 highlights a few of the techniques someone can use to hide their malicious JavaScript code from prying eyes. Another common type of supply chain attack (in the web world) is broken link hijacking where the attacker can take over an expired URL and exploit a dead-but-never-removed reference to a JavaScript file in order to inject arbitrary JavaScript into the page.

There are many frameworks that automatically download these components at build time, and often they trust the downloaded packages far too much – a similar concept to broken link hijacking. An attack that exploits this is a package substitution attack, also called dependency confusion. By using the right name and version, the attacker can trick the build system into downloading a package controlled by the attacker instead of the one in an internal package repository. In February 2021, a security researcher demonstrated this against several large tech companies and made over $130.000 in bug bounties. In a nutshell: after identifying the names of (non-public) packages used internally at these companies, he could upload packages under the same names to package managers (npm for Node.js, PyPI for Python, RubyGems for Ruby) with a higher version number, and the build systems would download his packages instead of the ‘real’ one.

Strengthening the supply chain

“I think there’s also some truth to the idea that if a Tier 1 adversary – a foreign nation-state – sends their A team against almost any ordinary company in the world, chances are they’re gonna get in. But that cannot be an excuse for doing nothing to build defenses and making it harder for them to be successful once they’re inside an enterprise.” (Senator Mark R. Warner, Chairman of US Senate Select Committee on Intelligence)

As a developer, the SolarWinds attack looks demoralizing at first – many aspects of it are still unclear, and it took nine months (and some luck) to even detect any evidence of the ongoing attack. But that doesn’t mean it couldn’t have been prevented. Just like supply chain attack being a very old concept, strong protections have been known for quite some time. In particular, David A. Wheeler laid out a defense in Countering Trusting Trust through Diverse Double-Compiling (2005) wherein a second compiler is used to compile the potentially compromised compiler’s source in parallel to the normal build process, and any difference in the resulting executables indicates a compromise. While this protection still necessitates a level of trust in a compiler, it is significantly harder to break, especially if the two compilers are running in separate environments with separate access control contexts (the principle of Separation of Privilege and Least Common Mechanism).

SolarWinds has started on a journey to become secure by design, and a part of that was protecting their build system. The main protection measure – as mentioned by CEO Sudhakar Ramakrishna in the Senate hearing – is to implement three parallel build systems with separate access levels. This solution is similar to Wheeler’s diverse double-compiling defense.

And of course, secure management of your dependencies is critical, and an important aspect of DevSecOps. Tools like OWASP Dependency-Check can help with identifying vulnerabilities and removing unneeded dependencies can reduce exposure to attacks as well. Finally, if you are depending on a large number of modules, libraries or packages as part of your application, consider version-locking them. Also, update them periodically only after verifying that they still work as expected.

Supply chain attacks, including untrusted functionality import are discussed in all our web application security-related courses. Check them out in our catalog.