The Limits of IQ Tests
Until now, the only cognitive skills measured were those initially identified to predict schoolchildren's academic performanceand traditionally such skills have been measured using IQ tests. Although IQ tests were not originally intended for use in business, studies have shown that these instruments predict work performance at least as well as competency interviews do (the most common assessment tool used today for hiring and promotion) and about ten times better than personality tests do. That's because some of the thinking skills that support academic success are also crucial to executive performance.
Yet IQ testing is not widely used as a way to identify top talent (though it plays an indirect role, as companies may choose to hire people with degrees from elite schools). The skills that IQ tests assess represent a fraction of a person's existing cognitive abilities. Some of the skills measuredsuch as vocabulary, arithmetic, and spatial reasoninghave almost no relevance to managerial work. Moreover, the topics tested would seem academic and elementaryindeed, almost insultingto people with extensive professional experience.
The format is also ill suited to business. Executives rarely if ever confront problems that have just one right answer; nor do they have the option of picking one answer from several choices listed. IQ test questions don't assess the practical, on-your-feet thinking skills needed in business. What's more, these tests have been repeatedly accused of racial and gender bias.
IQ tests don't assess the practical, on-your-feet thinking skills needed in business. |
Yet, despite these very real shortcomings, IQ tests are still a better predictor of managerial success than any other assessment tool. The business world's reluctance to use intelligence testing of any kind (other than assessments of emotional intelligence, which is really about personality and style) has robbed companies of a powerful tool for evaluating candidates for employment or promotion. It is, however, possible to create a comparable measure of intelligence for executives, one that tests for the skills managers needsuch as evaluating the quality of data or accurately identifying the core issues in a conflictand in a format that more accurately emulates the real business environment.
Interviewing for Intelligence
The most common interviewing methodology is the "past behavioral interview" (PBI). A PBI includes questions about a person's experiences performing certain activitiessuch as managing deadlines or resolving conflictsbut does not include personal questions. This form of interview has become the accepted best practice over the past thirty years, and, in fact, the PBI is a good predictor of performance. It can explain about 25 percent of the variances in performance among employees.
Still, PBIs miss a lot of what determines executives' success. That's because they don't measure what they claim to. Take two sample PBI questions. "What is the strategic direction of your company or division, and how did you go about developing it?" is designed to assess someone's competence at devising strategy. And "Describe a situation in which you had to interact with a difficult colleague and resolve a conflict" is supposed to test a person's capacity to handle conflicts. Surprisingly, you can just as accurately predict an executive's ability to devise strategy based on her answer to the second question as you can based on her answer to the first. This is not just a single example unique to these two competency questions; the same circumstance holds true for any competency question.
Research by professors Jesús F. Salgado and Silvia Moscoso of the University of Santiago de Compostela in Spain explains why. A person's performance on any behavioral interview question is dominated by the same three qualities: experience, job knowledge, and social skills. A candidate with a long work history has lots of compelling examples to draw from when asked to recount events that might illustrate a particular competency. A candidate's job knowledgespecifically, his awareness of industrial and managerial best practicescan make it easier for him to punctuate his answers with stories that will earn him high marks from interviewers. And a candidate who can relate his stories in a positive, likable manner has a distinct advantage over someone with inferior social skills. Because each question in the behavioral interview essentially assesses the same qualities, there's no need for the grueling three-to-four-hour sessions favored by hiring managers today. They need only ask enough questions to get a reliable appraisal of the candidate's work experience, job knowledge, and social skills.
Despite their advantages, behavioral interviews really only establish a candidate's minimum qualifications; they don't identify star talent. A candidate's experience, for example, is obviously an important hiring factor, but we all know seasoned executives who aren't stars. Similarly, being likable doesn't mean you have the intellectual horsepower to be a stellar leader. In short, behavioral interviews measure knowledge, not intelligence. Knowledge is information acquired through experience or formal training. Intelligence is the skill with which someone uses knowledge to solve a problem. Knowledge questions require people to recite what they have learned or experienced, while intelligence questions call for individuals to demonstrate their abilities.
So how do you measure executive intelligence? The best way is to use questions that require candidates to demonstrate their skills in an interview format. For such a measure to assess intelligence, it must raise questions and situations that the candidate has never confronted. The more novel the situation, the less rote knowledge can be applied and the more cognitive ability is required to render an answer.
Intelligence is the skill with which someone uses knowledge to solve a problem. |
The interview format is a departure for intelligence tests, which have traditionally been presented as written, multiple-choice exams because their developers believed that human judges could not make objective assessments. But that's not true. The Educational Testing Service recently changed the format of the Scholastic Assessment Test (SAT) to include essay writing. The move was controversial, because it introduced a human element into the judging, presumably making scores less objective. But, in fact, research had shown that other essay-based standardized tests, such as some Advanced Placement exams, were in many cases better predictors of academic success than the SAT. That's because most university students are graded on essay exams; almost none of their grades are derived from multiple-choice testing. It turns out that the best way to predict how well people will write essays in the future is to test them using an essay format today. In other words, to most accurately predict someone's performance, you must closely mimic the context in which the individual will have to perform.
The same holds true in the office. Executives exchange information through conversations, questions are posed, and decisions are made on the fly. The most accurate predictor of business performance would have to imitate these dynamics, and human evaluators are far and away the best judges of such interactions.
Rather than concentrating on academic subjects, executive intelligence tests should focus on the particular cognitive subjects associated with executive work: accomplishing tasks, working with and through others, and judging oneself. The questions shouldn't require specific industry expertise or experience. Any knowledge they call for must be rudimentary and common to all executives. Only then can a hiring manager be assured that the disparities among job candidates are because of differences in their processing power, not in their knowledge. And the questions should not be designed to ask whether the candidate has a particular skill; they should be configured so that the candidate will have to demonstrate the skill in the course of answering the question.
Imagine you want to determine whether someone can critically examine underlying assumptions and can anticipate likely unintended consequences. Rather than ask the candidate to recount an occasion in which she did either of these things, you must present a fact-based situation in which she would need to apply such skills. An executive intelligence evaluation designed to test these abilities might look like this:
- You are the CEO of a large software company. Your prices are being severely undercut by both domestic and foreign competitors. Your executive team recognizes a desperate need to cut costs. Your COO concludes that the answer is to outsource most of the company's programming to foreign subcontractors, thereby reducing labor costs. In fact, your COO has already received a number of bids from service firms in both India and South Korea. What questions do you have about his proposal?
A candidate displaying a high level of executive intelligence while answering this question would explain that the core assumption underlying the COO's conclusion needs to be confirmedthat is, outsourcing automatically equals cheaper production. She might point out that there may be indirect costs (up-front investment, ongoing customer service, and software development issues) involved with such a move that must be considered. Further, she would cite the probable unintended consequences of the COO's proposal, such as how using a distant workforce might affect productivity or labor relations.
The Skills that Make up Executive Intelligence
Regarding tasks, intelligent leaders: | Regarding people, intelligent leaders: | Regarding themselves, intelligent leaders: |
---|---|---|
appropriately define a problem and differentiate essential objectives from less-relevant concerns. | recognize the conclusions that can be drawn from a particular exchange. | pursue feedback that may reveal errors in their judgments and make appropriate adjustments. |
anticipate obstacles to achieving their objectives and identify sensible means to circumvent them. | recognize the underlying agendas and motivations of individuals and groups involved in a situation. | recognize their personal biases or limitations in perspective and use this understanding to improve their thinking and their action plans. |
critically examine the accuracy of underlying assumptions. | anticipate the probable reactions of individuals to actions or communications. | recognize when serious flaws in their ideas or actions require swift public acknowledgment of mistakes and a dramatic change in direction. |
articulate the strengths and weaknesses of the suggestions or arguments posed. | accurately identify the core issues and perspectives that are central to a conflict. | appropriately articulate the essential flaws in others' arguments and reiterate the strengths in their own positions. |
recognize what is known about an issue, what more needs to be known, and how best to obtain the relevant and accurate information needed. | appropriately consider the probable effects and possible unintended consequences that may result from taking a particular course of action. | recognize when it is appropriate to resist others' objections and remain committed to a sound course of action. |
use multiple perspectives to identify probable unintended consequences of various action plans. | acknowledge and balance the different needs of all relevant stakeholders. |
Excerpted with permission from "Hiring for Smarts," Harvard Business Review, Vol. 83, No. 11, November 2005.