Brian Kenny:
In the time it takes me to read this introduction, two firms will be hacked. Every 39 seconds, 24 hours a day, 352 days a year, there's a data breach. 120 million individuals in the US had their personal information exposed in data breaches in the past year, and one in three became victims of identity theft, which makes it easy to see why 42 percent of consumers have low or no trust that companies will keep their personal information safe. And 59 percent think that companies care more about profiting from their customer data than protecting it.
More than 80 percent of US firms say they have been hacked at some point and almost always as a result of human error. Data breaches, it seems, are inevitable, but how we respond to them can make or break customer relationships. Today on Cold Call, we welcome Professor Frank Nagle to discuss the case, “SolarWinds Confronts SUNBURST.” I'm your host, Brian Kenny, and you're listening to Cold Call on the HBR Podcast Network. Frank Nagle's research looks at the future of work, the economics of IT and digital transformation. He is a first-time visitor to Cold Call. Frank, thanks for joining me today.
Frank Nagle:
It's great to be here. Thanks for having me.
Brian Kenny:
I do think this is a highly relatable case as somebody who has had their personal information stolen a bunch of times. I think everybody has probably experienced that in one way or another. So, why don't we just dive right in? I'm going to ask you to start by telling us what Jason Bliss, the protagonist in this case, what does Jason learn as the case opens, and what's your cold call to start the discussion in class?
Frank Nagle:
Sure, Brian. So at the opening of the case, Jason Bliss, who is SolarWinds' general counsel at the time, has just learned from the company's outgoing CEO, who's leaving a few weeks later, that the company experienced a substantial, significant cybersecurity breach that allowed attackers direct access into the IT environments of potentially all of SolarWinds' 275,000 customers. At that point, they don't know how big it is, but that's what he's just learned. And so the way I think about opening this case in class, I usually set the stage. I talk a little bit about SolarWinds, the company, and then give the famous Mike Tyson quote, "Everyone has a plan until they get punched in the mouth." And that's essentially what happens here, is they had a pretty decent plan and they get walloped in the mouth in a way they weren't expecting.
And so I start the first cold call by asking the students, was SolarWinds really unlucky and this was just a fluke type of thing? Or were they unprepared or at least underprepared and this was more their fault? And that usually opens up a nice discussion because there's good arguments on both sides in this instance.
Brian Kenny:
Yeah, and it's a pretty dramatic case too. I love cases that start with this kind of a narrative where there's an incident that's happening and unfolding right in front of you. Why did you decide to write about this case? Why was it important to you and the kinds of things you think about as a scholar?
Frank Nagle:
Before becoming a scholar and returning to the academic world, I actually worked in cybersecurity for almost a decade at various companies, including the last one was Mandiant, which actually the CEO of that company that was later bought by FireEye ends up being a player in this case. So my old boss is a player in this case. And having that background, now I study things more related to open source software, but still a little bit about cybersecurity. There's not a whole lot of teaching cases on cybersecurity. There's a couple and they focus on different aspects. But in particular, this SolarWinds attack here was one of the most complex attacks that anybody has ever seen. And so bringing that into the classroom to show students, you talked about having your data stolen, which I think lots of people can relate to, and that's one type of attack. The customer data is stolen or financial data is stolen or something like that. But this type of supply chain attack that we see in SolarWinds is something that had very rarely been seen at that point, and this was the most extreme example of it. So we thought it would be a good teaching case to bring into the classroom to give this extreme version of what unfortunately is becoming something that's even more common today.
Brian Kenny:
That's a great segue into the next question I have, which is just, what is SolarWinds' business? Who are their customers and how do they work with them?
Frank Nagle:
What SolarWinds does is they make these small IT management tools. So if you're a network engineer or running a company's IT infrastructure, you are likely going to use their tools to really help you see into the network yourself, to understand what's going on on your network, who's doing what, and where's the traffic flowing, and things like that. And so they have lots of these different types of tools that help you do that. And so what's interesting about them is these tools are easy to use and they solve specific use cases. And so often they'd have different customers within the same company. One small IT group over here may buy some of their tools for a couple thousand dollars. A different IT group may buy a different IT tool of theirs for a few other thousand dollars. And so that ends up actually playing a role later on because they had lots of customers within the same company or within the same governmental organization.
Brian Kenny:
Yeah, it was interesting. As I was reading the case, I could see you were sprinkling all these little details throughout, and they all came to fruition as you read further on the case because it makes the whole thing much more complicated for SolarWinds as they're trying to figure out what's the scope of this situation we're dealing with, including their sales strategy. Can you talk a little bit about the way they approach sales?
Frank Nagle:
You know, their sales strategy was an outgrowth of their startup days, and it was mostly inside sales. Once they have somebody that's bought one of their tools, they try to upsell them and get them on other tools, but they didn't really have this kind of, again, centralized sales force that was going after companies as a whole. And that comes back later because when this big breach happens, they don't know who the right people at their customers are to talk to. There's 15 different people that they're connected to in a given organization and they don't know where to start. And obviously they have to try with all of them and contact all of them, and that slows down their response.
Brian Kenny:
One of the many lessons that I think comes out of this case is that you really do need to know who your contact people are at your customer sites.
Frank Nagle:
That's right.
Brian Kenny:
And then to add to the drama, they were going through a leadership transition. They were just about to leave from one CEO and welcome another. Can you talk about that dynamic?
Frank Nagle:
Yeah. I mean, this is one of the things we see in cybersecurity events. Inevitably, the attack happens at the worst possible time and that's the way it goes. So their outgoing CEO Kevin Thompson had been there for 10 years, had done a great job, really helped with this growth trajectory. He wasn't one of the founders. He came in after the founders decided it was time to bring in a more professional CEO. And so he had announced his pending retirement. The board had already found and identified a new CEO who was going to come on board on January 1st. The attack or SolarWinds discovers the attack or is told about the attack in December 12, so literally in the midst of the holiday season, which that always happens, too.
Brian Kenny:
Friday afternoon at 4:00 before a holiday.
Frank Nagle:
Exactly. Right. Exactly. And B, it's not like the new CEO has just gotten on board, but obviously the transition is in progress. The old CEO is winding things down and the new CEO has yet to start winding things up because he's not an employee yet. He's not even at the company. And so that is a pretty tricky time. And that's actually why often we have protagonists being CEOs, but Jason Bliss, who's again, the general counsel, ends up being the protagonist because of the CEO transition. And we actually have a small B case where the new CEO ends up being the protagonist.
Brian Kenny:
We'll talk a little bit about that. I think that dynamic is really, really interesting. But maybe we can pull the lens back a little bit and just talk... I mean, I teased in the intro about cyber attacks and the frequency with which they occur. Those numbers are probably a moving target, but maybe you can talk a little bit about what's the financial impact of these things? And more specifically, what was the nature of the attack at SolarWinds?
Frank Nagle:
At the high level of this kind of attacks, it really depends on the company and depends on the type of breach. But a 2023 IBM report said that the average breach cost that they saw in their work was about $5 million US dollars. Obviously there's a huge range there. But then some numbers from an IC3 report in 2022 said that in the US, cyber attacks cost companies $10 billion. These numbers aren't on the order of GDP, they're not trillions of dollars, but they're certainly growing and have been growing for many years. What's particularly concerning is that the attacks are becoming more prevalent and more sophisticated at the same time, and that's what we see in the SolarWinds attack. So this what became known as the SUNBURST attack was highly sophisticated. There were estimates that there were over 1,000 professional Russian hackers, state backed, government sponsored hackers that went into this. And that's just something that's very hard to prepare for. I mean, again, SolarWinds had a pretty good incident response plan for things like malware or ransomware or if customer data leaked out to the public or something like that, they knew what to do. They didn't have quite a plan for what happens if 1,000 government backed hackers come after you. The attack is what we call in the industry a supply chain attack. So SolarWinds itself wasn't actually the end target of we'll call them the bad guys and who was was all of SolarWinds' customers. So again, SolarWinds had a clientele list. In the case we talk about, they had I think 80 percent of the Fortune 500, most of the US governments, all the different departments of the government, many foreign governments as well. And so you can imagine how a company that's a keystone like that and has all these useful customers to an attacker might be a target of attack itself. And so a supply chain attack is when we see the attackers come in and breach, in this case, one of SolarWinds' products. Insert a backdoor into the SolarWinds product that then SolarWinds distributes to its customers. And of course, its customers trust it because it's coming from a known entity. And so when we see these supply chain attacks, they can happen at different points in the software development process. And where it happened in the SolarWinds one was in what's called the build process.
You have coders who code and they write their software, and then it all gets compiled and turned into what we end up using is the executable piece of software at the end through the build process. And so it was during that build process that the attackers got in and were able to stick the bad code in. And so if you looked at the raw source code that the developers wrote, it all looked good. There were no backdoors, no issues there. And if you looked at the end result, it was something that looked like exactly what it was supposed to coming from SolarWinds because they hit that very last step in the chain, in the software development lifecycle during this build process, and that's where they got it in. And so essentially what they did was put in a backdoor into this. And so if you installed SolarWinds' Orion product at a certain time, certain version numbers, then the Russian attackers... I think it hasn't been fully proven it was the Russians. Yes. The FBI has pointed the finger at the Russians, so we'll take them at their word. And so a very complicated attack that really ended up with all of SolarWinds' customers at least with that version of that software had a backdoor into their environment that the attackers could take advantage of.
Brian Kenny:
Super sophisticated and really alarming given the nature of what SolarWinds does, which if they have insight and a view across their customer's IT infrastructure, then you might think that the bad actors would have the same.
Frank Nagle:
Exactly the same. That's right. That's right.
Brian Kenny:
So take us inside the incident and tell us, these things unfold obviously in real time. Anybody who's ever managed a crisis of any kind knows that you get a certain amount of information in the beginning, it may or may not all be right, and then you've got to act, so you can't wait. But at the same time, you're not sure what you exactly should do because you're still waiting for more information. So it sounded like that was the dynamic here.
Frank Nagle:
Oh yes, absolutely. And they actually had even more information than you would normally have in this from the get-go, because who discovered this breach originally was, again, my old employer Mandiant, which was a division of FireEye. They found this in their own environment because they were a SolarWinds customer, and they investigated it and figured out what the malware was doing, how the backdoor worked and things like that before they brought it to SolarWinds.
When they came to SolarWinds a day or two after they had discovered it, they had a whole lot of information that they could give to SolarWinds, which made SolarWinds' job much easier because then they knew what they had to fix and they could identify what versions had that. And so they were able to move fairly rapidly, but that was partially because of how they found out about it. I will say that that may be surprising to some folks in the audience that they didn't find out about it themselves. They were told about it by a customer. That's actually I think something like 70 to 80 percent of these types of events you find out either from a customer or a law enforcement agency discovers it and lets you know that you have a problem in your network. And so often that's the first way of finding out is you get a phone call from somebody that's outside your organization that says, "You have a problem." And that's how this turned out.
And so for them at this early stage, they had a bunch of information about what it was, and then they had to go look at their software version history to understand when did this first happen. And then they could say, "All right, everybody before that version is okay. Everybody after that version potentially has a problem." And they quickly identified that of those 275,000 customers, there's about 18,000 of them at least potentially have that software, that version that has the backdoor.
So then the problem becomes that because of this distributed sales model and also the fact that the way that they sell their software is not software as a service, but instead is on-premise, so you as a customer buy this software and put it in your environment and manage it yourself, they don't have clear insight into exactly what version everybody is running. So they're kind of doing some guesswork at that point, which, of course, makes things a little bit fuzzier.
Brian Kenny:
And at the same time, they're obligated, I think, by law to inform customers about what's happening. And one of the criticisms that we often hear and these things after the fact is, why did you wait so long to tell me? Why were you sitting on this for 48 or 72 hours or whatever the number might be? How did they manage that?
Frank Nagle:
As you pointed out, this attack happened in late 2020, so only about three years ago. And even since then, back then there was really a patchwork of legal legislation and mostly at the state level in the US. And so it was a little bit unclear exactly what they had to do and who they had to report to. Today, the laws are a little bit clearer and there's more effort to nationalizing the rules around privacy breaches and cyber breaches and things like that. And so that said, at the moment they knew, because they were a public company, one of the main things they had to do was issue what's called an 8-K statement. And so this is a statement required by the SEC when a material event happens to a company. Doesn't have to be a cyber breach, could be loss of a major customer or a natural disaster wipes out a manufacturing plant or something like that. They have four business days to issue this public statement saying, "Here's what we know and here's what happened and here's the likely impact on our business."
So that was really what they organized around at first was making sure they could make that public statement as required by the SEC and use that as their anchor for talking with their customers. And so they get the phone call. I want to say it's on Friday night I believe is when they...
Brian Kenny:
Of course. Of course, it is.
Frank Nagle:
Of course. Exactly. Friday at 4:00 PM. They have four business days, but they decide to be a bit more aggressive and their goal is to release it before the markets open on Monday morning. So markets are already closed, which actually is a pretty good thing for them. They have some time over the weekend to get things together and figure out what's going on. But at the same time, they also know there's a ticking clock because they get word that The Washington Post has wind of this and is going to release a story as well. FireEye itself is going to release something because its customers are affected as well. And so there's really a ticking clock here. And as a result, they have even less information and less details.
Brian Kenny:
This was all happening during COVID, is that right?
Frank Nagle:
That's right. That's right.
Brian Kenny:
We'll throw another wrench into the thing. So how do you organize a team in essentially a war room type environment? Because this is all hands on deck, I would imagine.
Frank Nagle:
That's right.
Brian Kenny:
How do you do that when everybody's scattered?
Frank Nagle:
That's the thing, right? This was December 2020, so no vaccines yet. We're still well into COVID, work from home. Almost the whole company's been working from home. They're lucky enough that they have an auditorium in their building. So they stick most of the people in the auditorium with what they end up calling a “Tiger Team,” which is essentially the department heads that are organizing at the top level to manage the whole operation, working in a smaller boardroom in the building.
Brian Kenny:
So let's go back to the dual CEOs. We've got Thompson, who is obviously on the scene, he's the current CEO, and then we've got Ramakrishna, who has been appointed CEO. How does he plug into this?
Frank Nagle:
Ramakrishna, lucky for SolarWinds, actually had a cyber security background. His most recent job was the CEO of Pulse Secure, which is a VPN company. But he's not a part of the company yet, and so they can't tell him anything. And so they end up calling him during his birthday party, which of course, again, worst timing ever. And they say, "There's a big problem and you need to know some stuff. So we need to get you in the loop somehow." Meanwhile, the current CEO and the board chair basically say to him, "Look, this is not what you signed up for. So if you want to bail, we get it." But he says, "No, I get it. Whatever this is that I don't know the details about is going to be really bad." And so he takes it in stride and says, "Let's get going." And so the easiest way ends up actually being for them to appoint him to the board before his turn as CEO starts taking over so that he can then start accessing information. So within the first few days, that's what they do. Ramakrishna becomes a board member, even though he's not CEO yet. And he told us in our interviews that what he did was just for the first two days, he just listened to soak up information about not only the event, but also the company, to get a better sense of what the company stood for and how they prioritize things. And then on the third day, he starts offering advice. And so he brings that background from cybersecurity experience at other companies and starts helping deal with external relations and dealing with some of the customers and helping soothe some nerves by saying, "Look, we're working on it and we're going to help our customers first." And that was kind of a joint decision because SolarWinds really consider themselves a very customer oriented company. I know lots of companies would say that, but they're very customer focused. And so they make these decisions to prioritize the customers and help them get fixed. And that ends up being both Thompson and Ramakrishna's priorities. And so from the discussions with them, it really sounds like they were on the same wavelength, which is good because you can certainly imagine contentious CEOs.
Thompson was leaving on good terms, so it wasn't like he was being forced out or anything like that. So that helped as well. But again, complicated because your top level leadership is one's on the way out and one's on the way in.
Brian Kenny:
A huge trial by fire, not just in terms of his subject matter expertise, but his character and the fact that he didn't walk out or turn and run away from this. He ran into it, which is pretty amazing, and says a lot to your employees too as they see you even before you're officially on the job.
Frank Nagle:
I think that's right. He definitely had the faith of the employees before he even showed up on day one.
Brian Kenny:
I've got to believe that's true. You talk about the customers and customer first approach that they took. Not all of their customers are the same in the sense that they've got obviously private firms that they do business with, but they've got a lot of government customers. I would imagine that the stakes there are almost even higher if they've got DOD or you name whatever department. The secondary impact of this could have been really, really significant. Did they deal differently with those sets of customers?
Frank Nagle:
Yeah. That was, again, this customer-centric focus, right? They wanted to try and talk to every single customer they had, but they had to prioritize. And knowing, again, from an early stage because of the work that FireEye had done and because of some law enforcement discussions, they knew that this was likely a state attacker and that it was likely that the government customers were the real targets. And so it's possible or likely that many of their customers had infected versions of the software, but that they weren't really the targets of what was happening here. So they end up prioritizing the government clients, both the US and other countries as well, because there was some evidence that it wasn't just US customers that were the focus. And so they set up this tiered customer response approach. And so because of the technical details, they're able to at least narrow down, we think this set of 18,000 customers are the most likely ones to have this infected version.
And then of that subset, focusing on government and other marquee or high potential customers that they believe that are the true targets of this attack. They focus on them and helping them figure out which version are you running first of all. Because if they're not actually running an infected version, then they're okay. But if they are, then help them shut it down and update it and patch it and get it fixed to get the bad guys out of the network as soon as possible.
They state that 18,000 number in the 8-K when they released that on Monday morning, but knowing that that was an overstatement of how many customers were actually not only affected, but likely being breached. So you can think about this in concentric circles. There's 18,000 that might have the infected software. Of that, some subset, probably thousands, have the infected software. And then of that, some subset that ends up only being about a hundred are actively being exploited by the attackers.
Brian Kenny:
Okay, but I'm sure that didn't do much to assuage all of their customers who thought, "I may not be in the 18,000, but how do I know my information is safe?"
Frank Nagle:
And that's right. I think that that was a substantial concern. They shift their customer service organization to be basically a panic room and dealing with trying to talk to as many of the customers as possible in a way that they wanted to the extent that they could avoid having just an automated email blast, which when you have almost 300,000 customers is hard to do. And so they did a little bit of automated, but then they really set up this big operation to try and touch and talk to every single customer that they had.
Frank Nagle:
Obviously some had more priority based on whether or not they were in the affected group and who they were, but they tried to touch every customer and actually say, "Hey, look, we know you're running X version. That's not affected. We appreciate that you're concerned about this, and here are some things you can do and update to the latest version anyways and we'll help you do that." And I think one of the things that came out of this was that customer approach really ended up saving them at least from the customer side, because that year or the following year, 96 percent of their customers still re-upped their contracts and stayed there. And that's against a baseline of 98 percent. So normal year, 98 percent would renew, and that year was only 96 percent. They only lost a few handful of customers as a result of this, and I think that that customer focus throughout all of this really saved them from that perspective.
Brian Kenny:
Yeah, no doubt. Why don't we fast-forward a little bit now to the B case as you mentioned before?
Frank Nagle:
Sure.
Brian Kenny:
This was a little bit where they took a look back and said, "What worked well here? What didn't?" Can you give some of the highlights of that?
Frank Nagle:
The B case takes place I think it's roughly a month or so later. At that point, Ramakrishna is now CEO and he's having to spend a whole lot of his time doing something he wasn't really anticipating when he signed up for the job. But I think they make two main pieces of their customer response, of their industry response of how they're trying to shape this going forward. One is what they call the Orion Assistance Program where they're really helping customers that have this affected software update, patch. So they really make sure everybody's got the latest version and doesn't have affected software. So that's part one, which as you can imagine is the early stage. And then in the moderate stage, the middle stage, they introduced this concept of secure by design. And so they really revamped their whole development process such that security is baked in from the beginning. It's only a few years ago, but this really helped the industry understand that we can't just latch on security at the end, which was very much the way that software development was done for a long time. The shift of focus on secure by design is something that Ramakrishna helps introduce and really re-orients the way that the company makes its products and bakes in security from the very beginning so that this super complicated type of breach can't happen or is less likely to happen. And also more simple run-of-the-mill bugs in the software that might cause a problem down the road are also less likely to happen.
And then he goes on the road, and actually with my former boss, the CEO of FireEye, but also the head of CISA, which is the government agency in charge of managing these types of events. They are speaking at conferences and all sorts of things about this secure by design principle and to really proselytize to the industry as a whole that we're in a new world and we need to all really be much more focused on security from the ground up, not just layering it on later.
Brian Kenny:
And the bad actors, as we know, are always out there trying to think about the next vulnerability that they can exploit somewhere down the line. So it just never ends.
Frank Nagle:
Exactly. Unfortunately, it's a never-ending game of cat and mouse. It's a fascinating world to be involved in, but the bad guys are always going to think of some new way to get in. When I worked in the industry, my main job was red teaming, so it was doing the breaking in to check defenses and test defenses. And the tough thing is is that as a bad guy or as a pretend bad guy, in that case, you only need to find one way in. You need to find just one weakness that lets you in the door and then you can go from there. As the defender and the good guys, you really have to try to cover every single base. And I think this is one thing too that we've seen as a shift over time in the industry. When I started, it was all about protect the perimeter, keep the bad guys out, don't let them in. Nowadays, what most places have embraced is a concept called zero trust, where we assume that we're breached and have a lot more internal defenses and trust. There is no trust. We verify every single thing that's happening inside the organization as well. So this is why all of us that are listening to this have to deal with more two-factor authentication or entering your password more times or VPN-ing for everything or things like this.
Brian Kenny:
No matter how irritating that is.
Frank Nagle:
It's necessary even if it's slightly annoying.
Brian Kenny:
You did mention to me that there's been some late-breaking developments that aren't part of the case obviously with the SEC and charges brought against the SolarWinds chief information security officer, which seems like, I don't know, a game changer in some ways. And I'm wondering if you can talk a little bit about what the implications are for people in that role.
Frank Nagle:
And so just a few weeks ago, the details are still unfolding, but the SEC brought new charges against the company and against the chief information security officer in particular for not being fully transparent and for knowing some things that they didn't tell the public or tell their customers or tell their shareholders about. Now, I'm not here to judge on that. I don't know what the truth is. I'm sure we will find out. But indeed, I think we're seeing more of these types of charges being brought against either CISOs or VPs of security, people that are running the security operation at companies.
About a year ago, a similar type of charge was brought against folks at Uber. There I think it was a little more open and shut that there was clearly some things that they knew that they didn't tell the public and things like that. But more broadly speaking, it's concerning because we're only going to see more and more cyber attacks. And so knowing that qualified people are leaving these positions because of fear of this gray space and ending up on the wrong side of it and ending up in jail is a broader concern. So I think what can we do about that, regulatory clarity so that it's much clearer for folks what they need to do and when they need to do. Is it four business days? Is it four days? Is it 48 hours? These types of things. And consistency and clarity will go a long way in helping CISOs feel secure in that they're doing the right thing and doing what they're supposed to do.
Brian Kenny:
I think we all want the best people in those roles because really it impacts everybody. It could potentially impact everybody.
Frank Nagle:
It does. Absolutely, right? Cybersecurity matters for everyone. Even if you don't know anything about it, you're still maybe affected by it.
Brian Kenny:
Frank, this has been a great conversation. I've got one more question for you and that is simply, if you want our listeners to remember one thing about the SolarWinds case, what would it be?
Frank Nagle:
For those that are in positions of not even cybersecurity positions, but just in managerial positions, having a plan is super important because it allows you to act faster when these types of things happen. And I think in SolarWinds case, if they didn't have this pretty good incident response plan in place, everything would've been much worse.
Brian Kenny:
And for everybody listening, change your passwords.
Frank Nagle:
Regularly. That's right.
Brian Kenny:
Frank, thanks for joining me on Cold Call.
Frank Nagle:
Thanks so much for having me. It was great to be here.
Brian Kenny:
If you enjoy Cold Call, you might like our other podcasts, After Hours, Climate Rising, Deep Purpose, Idea Cast, Managing the Future of Work, Skydeck, and Women at Work. Find them on Apple, Spotify, or wherever you listen, and if you could take a minute to rate and review us, we'd be grateful. If you have any suggestions or just want to say hello, we want to hear from you. Email us at coldcall@hbs.edu. Thanks again for joining us. I'm your host, Brian Kenny, and you've been listening to Cold Call, an official podcast of Harvard Business School and part of the HBR Podcast Network.