How Toyota and Linux Keep Collaboration Simple

by Philip Evans and Bob Wolf

Tuesday, December 2, 2003
Near midnight, Andrea Barisani, system administrator in the physics department of the University of Trieste, discovered that an attacker had struck his institution's Gentoo Linux server. He traced the breach to a vulnerable spot in the Linux kernel and another in rsync, a file transfer mechanism that automatically replicates data among computers. This was a serious attack: Any penetration of rsync could compromise files in thousands of servers worldwide.

Barisani woke some colleagues, who put him in touch with Mike Warfield, a senior researcher at Internet Security Systems in Atlanta, and with Andrew "Tridge" Tridgell, a well-known Linux programmer in Australia on whose doctoral thesis rsync was based. They directed Barisani's message (made anonymous for security reasons) to another Australian, Martin Pool, who worked for Hewlett-Packard in Canberra and had been a leader in rsync's development. Although Pool was no longer responsible for rsync (nobody was), he immediately hit the phones and e-mail, first quizzing Warfield and Dave Dykstra (another early contributor to rsync's development, who was based in California) about vulnerabilities and then helping Barisani trace the failure line by line.

By morning Trieste time, Pool and Barisani had found the precise location of the breach. Pool contacted the current rsync development group, while Barisani connected with the loose affiliation of amateurs and professionals that package Gentoo Linux, and he posted an early warning advisory to the Gentoo site. Pool and Paul "Rusty" Russell (a fellow Canberran who works for IBM) then labored through the Australian night to write a patch, and within five hours Gentoo user-developers started testing the first version. Meanwhile, Tridge crafted a description of the vulnerability and its fix, being sure (at Pool's urging) to credit Barisani and Warfield for their behind-the-scenes efforts. On Thursday afternoon Canberra time, the announcement and the patch were posted to the rsync Web site and thus distributed to Linux users worldwide. [...]

Obsession, interaction, and a light touch
The rules of markets are about cash and contracts. The rules of hierarchies are about authority and accountability. But at the core of the Linux and Toyota communities are rules about three entirely different things: how individuals and small groups work together; how, and how widely, they communicate; and how leaders guide them toward a common goal.

A common work discipline. The Linux and Toyota communities are both composed of engineers, so members have the same skills as their colleagues and speak the same language. But these groups are far more disciplined and rigorous in their approach to work than are other engineering communities. Both emphasize granularity: They pay attention to small details, eliminate problems at the source, and trim anything resembling excess, whether it be work, code, or material. Linux members, for example, share an obsession with writing minimal code, compiling each day's output before proceeding to the next and extirpating programming flaws as they go along. For their part, TPS [Toyota Production System] engineers are relentless in applying short cycles of trial and error, focusing on just one thing at a time, and getting inside and observing actual processes. Both groups carry those principles to apparent extremes. Linux programmers whittle away at code in pursuit not of computational efficiency but of elegance. Toyota engineers reject stampings for the Lexus hood—while flawless and entirely within spec—because the sheen, to their eyes, lacks luster.

Widespread, granular communication. In both the Linux and Toyota communities, information about problems and solutions is shared widely, frequently, and in small increments. Most Linux hacker communication is not between individuals but by postings to open, searchable Listservs. Anyone can review the version history of the code and the Listserv debates—not executive summaries or abstracts but the raw activity itself. And every code contribution is stress tested by scores of people. As a famous open-source mixed metaphor puts it: "With a thousand eyes, all bugs are shallow." The median upload to the Linux kernel is a mere dozen lines of code. The working alpha version is recompiled every twenty-four hours, so hackers reconcile their efforts almost continuously. If someone worked in isolation for six months on even the most brilliant contribution, it would probably be rejected for lack of compatibility with the others' efforts.

The Toyota philosophy of continuous improvement likewise comprises a thousand small collaborations. Toyota engineers are famously drilled to "ask why five times" to follow a chain of causes and effects back to a problem's root. This is not a vapid cliché about thinking deeply. Quite the contrary, in fact. The precept's merit is precisely in its superficiality. Saying that B causes A is simplistic—all the complexities of multiple interactions boiled down to a single cause and effect. But the chain of thought required to discover that C causes B, and D causes C, quickly takes you into a new domain, probably someone else's. So rather than concoct complex solutions within their own domains, engineers must seek simple ones beyond them. "Doing your why-whys," as the practice is known, is not about depth at all—it's about breadth.

And as with Linux, Toyota's communication protocols enforce this discipline. Each meeting addresses just one topic and drives toward a specific outcome, even if that means the same people meet more than once in a day. Lessons are written in a standard format on a single sheet of A3 paper. And everyone learns how to craft these reports, down to the fold in the document that shows the main points and conceals the details.

Leaders as connectors. At every level, Linux and TPS leaders play three critical roles. They instruct community members—often by example—in the disciplines we've just described. They articulate clear and simple goals for each project based on their strategic vision. And they connect people, by merit of being very well connected themselves. The top Linux programmers process upwards of 300 or 400 e-mails daily. Fujio Cho, the president of Toyota, manages by similarly numerous daily interactions that transcend the normal chain of command. [...]

What they know and how they know it
At the heart of Linux and the Toyota Production System, then, is a set of work, communication, and leadership practices that contributes to a new form of collaboration. This collaboration also relies on two infrastructure components: a shared pool of knowledge and universally available tools for moving knowledge around.

Common intellectual property. The General Public License under which Linux is published requires that all distributors make their source code freely available so that others can freely emend it. This viral principle prevents code from being stowed away in proprietary products. That transparency, in turn, breaks down the distinction between producer and user. A sophisticated "customer" like Andrea Barisani is really a user-developer, who fixes flaws and adds features for his own benefit, then shares those improvements with everyone else. Such a role is impossible when proprietary code is licensed from a commercial vendor. Similarly, Toyota's supply chain is predicated on the principle that while product knowledge (such as a blueprint) is someone's intellectual property, process knowledge is shared. That breaks down some distinctions among companies. Toyota's suppliers regularly share extensive process improvement lessons both vertically and laterally, even with their competitors. In Japan, suppliers are generally exclusive to a single OEM, so the collective benefit of that shared information stays within the Toyota supply chain. But even in the United States, where Toyota is just one of several customers for most of its tier ones, the carmaker does the same thing through its Bluegrass Automotive Manufacturers Association, which disseminates best practices to all members.

Simple, pervasive technology. Although information is the lifeblood of the Linux and TPS communities, their circulation systems are surprisingly rudimentary. Linux developers produce state-of-the-art software using communication technology no more sophisticated than e-mail and Listservs—but those mundane tools are used by everyone. Indeed, so great is the value placed on universality that plain-text, rather than formatted, e-mails are the norm, ensuring that messages will appear exactly the same to all recipients. Toyota, whose products are state-of-the-art as well, also prefers simple and pervasive internal technology. An empty kanban bin signals the need for parts replenishment; a length of duct tape on the assembly-line floor allots the completion times of tasks on a moving vehicle. Quality control problems on the assembly line are announced via pagers and TV monitors. And everyone gets the alert. Even Ray Tanguay, head of Toyota Canada, is paged whenever a flaw is found in the latest Lexus consignment on the dock in Long Beach, California, or in a service bay anywhere in North America. [...]

Such extremely rich, flexible collaborations have positive psychological consequences for participants and powerful competitive ones for their organizations. Those consequences are rich common knowledge, the ability to organize teams modularly, extraordinary motivation, and high levels of trust.

Excerpted with permission from "Collaboration Rules," Harvard Business Review, Vol. 83, No. 7, July/August 2005.

[ Buy the full article ]

Philip Evans is a senior vice president at the Boston Consulting Group. He is also coauthor of the book Blown to Bits: How the New Economics of Information Transforms Strategy.

Bob Wolf is a manager at Boston Consulting Group.