In a recent case study, professor Alan MacCormack and research associate Jay Wynn swung for the moon—Mars, actually. After two high-profile missions to Mars failed in 2000, MacCormack wanted to understand the organizational and procedural remedies NASA and the Jet Propulsion Laboratory (JPL) would employ to successfully return to the Red Planet.
The case, "Mission to Mars," looks at changes the space agency has made not only recently but also over several decades as it followed a faster, simpler approach to program design.
MacCormack's findings may help even earth-bound managers more effectively design and measure complex projects and organizations.
Sean Silverthorne: What drew you to the NASA/JPL Mars program as potential case material?
Alan MacCormack: First, it was an ideal context in which to explore one of my current research initiatives, which looks at how organizations design "programs" or sequences of projects over time in order that learning is maximized. Most past research on innovation has focused on the management of individual projects. As a result, we know a lot about topics such as determining the optimal team structure or designing the best type of development process for a project. But my past work (studying projects in high-tech industries) had convinced me that we were missing an important level of analysis.
It seemed that decisions made in a project sometimes worked against the broader objectives of an organization, even when managers had the best of intentions. This led me to believe that some decisions are best made at the program level rather than within individual projects. So I began looking for an organization that had tackled this problem.
The second reason for studying the Mars program was opportunistic. In the early 1990s, NASA had implemented a program called "Faster, Better, Cheaper," (FBC) which involved making fundamental changes to the way the organization developed unmanned spacecraft. It was a massive organizational transformation effort that sought to deliver dramatic improvements in efficiency, robustness, and flexibility. A "natural experiment" if you like. In 1999, however, the failures of two successive missions aimed at Mars brought the initiative to a halt. In the aftermath, the finger was pointed squarely at FBC as the cause of the failures. I was intrigued by what had gone wrong, especially given that many of the early FBC missions had been tremendously successful. And I was convinced that the answers could help inform a wide variety of situations in which organizations were attempting the type of transformation NASA had sought with FBC.
Q: What was the impetus behind the "Faster, Better, Cheaper" initiative, and what did it try to achieve?
A: FBC was a response to rising development costs and the high-profile failures of several missions in the late 1980s and early 1990s. Its aim was to move toward smaller, less expensive spacecraft, increasing the number of missions that could be funded within a given budget, while reducing the negative impact of a failure. In the case of the Mars program, this meant no more funding for big, complex, billion-plus dollar missions like Viking (which succeeded in the 1970s) or Mars Observer (which failed in 1992). Instead, NASA would send smaller spacecraft to the planet every two years, using information generated in each mission to improve the effectiveness of those to follow.
While the objectives were admirable, the way NASA went about implementing FBC was somewhat surprising. When I began the study, I had expected that this would have been handled by a team of consultants, first in developing a blueprint for how an FBC project should be run, and then in rolling out the methodology across the organization. What I found, however, was quite the opposite. There was no central effort to define what an FBC project should look like. Rather, NASA forced its managers and contractors to invent new processes and procedures by imposing a set of budget, time, and weight constraints that could not be met using traditional approaches to spacecraft development. As one manager explained, "The attitude was 'the book is not working, so don't use the book—try something different, then write a new book.'"
While the objectives were admirable, the way NASA went about implementing FBC was somewhat surprising.
The approach appeared to be based upon an assumption that in such an extreme context (i.e., one in which your product had to operate 100 million miles from Earth and perform under a range of conditions which were hard to predict in advance), it was impossible to know up front exactly what form a faster, better, cheaper approach to development should take. Far better to conduct a number of organizational experiments and gradually learn over time what things worked well and what didn't. This "emergent" approach to organizational transformation therefore relied upon NASA's managers trying out different approaches to developing missions, then learning from their collective experiences as to which were the most useful to adopt moving forward. Unfortunately, it turned out that NASA had what might be called a "learning disorder."
Q: So, what went wrong with FBC?
A: The first problem lay in NASA's desire to push the envelope with FBC before it had feedback on how well the initiative was working. Developing a small spacecraft typically takes over four years, meaning the outcomes from each FBC "experiment" were not observed until years after each had begun. Yet NASA didn't wait to see how early missions performed before moving the goalposts for later ones.
For example, the two Mars missions launched in 1998 were given a budget of only half that of Mars Pathfinder, well before anyone knew how Pathfinder would fare. As one NASA executive remarked, "It was like being in the Olympic high jump...we were going to keep raising the bar, and eventually, you knew what would happen." Unfortunately, once NASA realized the bar was too high, the pipeline was full of missions that were potentially compromised. It is no surprise that later FBC missions began failing at a much higher rate than earlier ones.
To understand what happened, it is helpful to consider that the window to launch a Mars mission is essentially fixed. Approximately every two years, the Earth and Mars align in such a way to provide the lowest cost trajectory for a mission flying between them. So the launch date for a mission (and hence, its schedule) is fixed. The only project variables that can be changed are therefore the cost of the mission and its scope (or complexity). These two variables, in turn, influence the level of risk in the mission (i.e., the likelihood of errors). Cut cost significantly without reducing the scope of a project, and risk is likely to increase, all else being equal. FBC was essentially an initiative that set out to test the limits of these relationships. Eventually, it found them.
NASA's second problem came in failing to recognize that, because FBC was so learning-intensive, it would require a much more systematic approach to capturing and disseminating knowledge than had previously been the norm. For example, although NASA implemented a "lessons-learned" IT system in 1995, a 2001 survey found that only a quarter of its managers contributed to the system and a similar number were unaware that it existed. There was therefore no reliable way to transfer codified knowledge about methods and practices across the organization.
Furthermore, while "red team reviews" —progress reviews conducted by NASA's most experienced managers—proved invaluable in early FBC projects, in later missions, the frequency of these reviews was cut. As a consequence, the transfer of tacit (or experiential) knowledge across projects began to suffer. In a transformation effort like FBC, which was based primarily on trial and error learning, problems such as these were to prove costly.
By not conducting comparably detailed post-mortems on successful missions, NASA missed the opportunity to identify problems (and solutions) that might have helped avoid later failures.
A third problem lay in NASA's tendency to aggressively post mortem failed missions while paying comparatively little attention to what could be learned from those that succeeded—a form of "superstitious learning." In an environment as challenging as planetary exploration, however, the difference between a failed mission and a successful one is often extremely subtle. There is no reason to expect that the former are full of bad practices, and the latter are paragons of process virtue. For example, it is quite possible that as many mistakes were made in the celebrated 1997 Pathfinder mission as were made in the failed 1999 Polar Lander mission. But we will never know. By not conducting comparably detailed post mortems on successful missions, NASA missed the opportunity to identify problems (and solutions) that might have helped avoid later failures.
The final set of problems that conspired against the success of FBC in the Mars program was a lack of coordination between the goals of the program and the decisions that were made on a day-to-day basis within individual projects. NASA did not take steps to ensure that the need to learn at the program level was reflected in the design of each mission. Hence managers focused only on optimizing their own projects, rather than on understanding how their missions could best support program learning. For example, a $4 million transmitter that would have provided data on why the 1999 Polar Lander mission failed as it neared the surface was deemed too costly to include. Yet without this data, it was more likely that a future mission would be doomed to repeat the same mistakes. NASA failed to realize that building a successful program can require that sacrifices be made in each project to maximize the learning captured from each.
Q: You mentioned earlier the need for some decisions to be taken at the program level rather than in individual projects. What types of decisions does NASA move up to this higher level?
A: There are three areas that benefit from program management. The first is related to infrastructure investments that are common to several projects, for example, like navigation software. There is no need for each project to reinvent the wheel in these areas, given this increases a project's cost and introduces additional technical risk. It is better for a program manager to coordinate the development and validation of these core infrastructural technologies, making sure that they fulfill the requirements for multiple projects.
The second area where program management is critical is related to decisions that maximize the performance of the system as a whole, but involve trade-offs at the individual project level. For example, it is far more effective for an orbiter to relay signals from a Mars lander back to Earth than for a lander to waste valuable payload weight carrying a transmitter powerful enough to send signals to Earth independently. A program manager must therefore make calls on decisions that affect the overall system performance.
The final area that requires coordination at the program level is related to mechanisms for capturing learning that will be useful to future missions. This might include decisions on how much telemetry data to provide in flight, a useful source of fault-finding data should things go wrong, as well as the choice of when to fly a new technology that will eventually be required in future missions. Common to all these decisions is the notion that project managers, left to their own devices, do not have sufficient information to fully evaluate their potential impact on the entire system. Hence there is a need for a manager with a broader view of the program to exercise judgment.
Q: For terrestrial managers, what lessons does the Mars program teach?
A: First of all, in any organizational transformation effort based upon trial and error learning, it is critical to understand at what points you will receive feedback on how well the initiative is progressing. This in many ways will put a limit on how fast you can learn, and therefore on how fast you can improve—raising the bar before real data are available is risky, and makes learning from experience more difficult. Should the "feedback time" be long (as it is for Mars missions) consider using other techniques that might help get the feedback you need, or something close to it, at earlier stages. Examples of such techniques include simulation, prototyping, and pilot/beta testing.
Second, a trial-and-error based approach to organizational transformation requires that careful thought be given to how knowledge is captured and disseminated across an organization. This effort must encompass both a consideration of processes and systems for capturing both explicit (or codified) knowledge and tacit (or experiential) knowledge. And it must pay attention to the degree to which the outcome of a project should affect our efforts to learn from it. In particular, environments where the difference between success and failure is subtle (or a matter of "luck") require that we learn not only from failures, but also from assessing the learning that comes from success.
Institutionalizing post mortems on all projects, successful or not, is the only way of generating robust insights on how well an organizational change effort is proceeding. Treating every project as if it had failed, then trying to determine why, is the mindset that is required in order that an organization fully captures the learning from each project.
Finally, the Mars program highlights that there are some decisions that are best made outside individual projects, especially in the context of a program or change initiative where projects are linked over time, therefore creating dependencies between them. In such an environment, there is a need to define how each project should support the overall goals of the program, both in terms of performance and learning. This process should explicitly address the potential trade-offs that are likely to exist between short-term project goals and the need for longer-term learning at the program level.
Q: Your case leaves off around 2001. The current Mars Lander program has apparently been a success. Do you think the program is back on track?
A: NASA certainly did a good job recovering from the failures of 1999. They had to cancel a Mars lander due for launch in 2001, given it shared too many similarities with the 1999 craft. But they redirected their efforts into making sure that the orbiter due for launch at the same time was robust. That spacecraft, called Odyssey, has been extremely successful. Both the Rover missions on Mars now seem to be performing well, give or take a few glitches that they've been able to troubleshoot in situ. And NASA made some important changes to address the need for greater coordination at the program level.
In all likelihood, however, the pendulum has swung too far back towards the traditional style of mission management. This is understandable to some extent. After all, in 1999, Congress wasn't pleased to see $200 million of its money go up in smoke (or up in space—there is no smoke in a vacuum). Politically, the next missions had to succeed. Another failure would have been disastrous for NASA. Yet given the nature of this risky activity, we should not be that surprised when missions fail. Of the thirty-odd missions we've sent to Mars, only a third have been successful. It really is rocket science!
Personally, I think that it will be hard for NASA to make progress on reducing mission costs unless they radically re-think how to structure programs like the one to explore Mars. Presently, 70 to 80 percent of the development budget for a small spacecraft is consumed by the flight system cost—that is, the cost of getting there. That leaves precious little for the science, the main reason for going in the first place. To reduce this substantially, NASA would have to place much greater emphasis on technology re-use, developing designs that were expected to fulfill multiple missions. In this respect, borrowing some ideas from business could help. For example, NASA might separate projects to design new delivery "platforms" from "derivative" projects that use existing (and proven) designs to deliver new science experiments to the surface. Of course, the cultural change required to do this would be significant. Building spacecraft to print is cheap, but the typical project manager won't see the job as particularly prestigious.
Q: Looking ahead, President Bush recently outlined a new program for space exploration that includes a manned expedition to Mars. Do you have any thoughts on the merits of this endeavor?
A: Well, you first have to consider the issue of whether a manned mission to Mars is necessary from a science standpoint. Manned missions are at least an order of magnitude more expensive than unmanned missions. The big problem is that we weigh too much—it's enormously expensive to help a human escape Earth's gravity. So we have to evaluate whether the benefits of putting people on Mars in terms of science return are large enough to offset the far larger number of unmanned missions that could be run for the same money. Personally, I think lots more work can be done by unmanned craft before we will see the need to put live decision makers on the surface of Mars.
It is critical to understand at what points you will receive feedback on how well the initiative is progressing.
Of course, you can't judge a manned mission only in terms of cost/benefit trade-offs. There are a number of other factors that also play into the decision. For example, adopting such a lofty goal might help spur technological innovations that would ultimately feed back into the commercial world, as they did during the first "space race." The psychological impact of achieving such a goal would be huge, as would the political capital created for the nation that first achieved it. But how these factors should be factored into the decision to go or not to go is a tough question.
Ultimately, it is not so much the goal that we should be concerned about as much as the process through which we attempt to achieve it. A return trip to Mars will require that we invent many new technologies and systems, all of which will have to perform seamlessly to ensure a safe and successful mission. Given the amount of uncertainty involved in such an endeavor, it is naïve to think we can sit here today and identify the date of our first touchdown and the means by which we will get there. Instead, we need to adopt a modular, experiment-driven approach, gradually building and verifying the set of technologies that will be needed for such a mission, while adapting our plans as we learn more about what approaches have merit, and which are likely to be dead ends.
Q: It is clear that you are fascinated by the general topic of space exploration. Do you have any deeper connections to the topic?
A: Well, like many little boys, I always wanted to be an astronaut (on the days when I didn't want to be a football player). But I did take my childhood dreams a little further. Back in 1989, I applied to be the first British astronaut, along with over 10,000 other hopefuls. As I remember, the winning candidate was supposed to train for a year in the USSR before going up in a Soyuz craft. I didn't get selected. But as it happened, the break-up of the Soviet Union almost scuttled the whole project. It just goes to show, when you're dealing with space exploration, nothing is ever certain.
Note: Helen Sharman, Britain's first astronaut, spent 8 days aboard the Mir space station in May 1991.