Is good decision-making a skill?

Introducing the Assignment Game, a new way to measure your ability to make good decisions about resource allocation

TL:DR - This morning, my coauthors and I released a paper that develops a new way to measure the quality of economic decision-making. We call it the Assignment Game. We administer the Assignment Game in a field survey of full-time workers and find that it is positively related to earnings, especially in jobs where you have to make a lot of decisions.

The writeup below provides more detail and motivation. Click here if you just want to play the game!

The growing importance of decision-making on the job

The freedom to choose how I spend my work time is one of the great joys of being an academic. The flipside of this freedom, however, is needing to make an endless number of decisions. How much time should I devote to research vs. teaching and university service? Which projects should I work on? Should I cluster my standard errors?

In creative jobs, good decisions beat hard work every time. If I have a clever idea in the shower about how to analyze some data, it can save me three months or more of beating my head against the wall. The importance of good decision-making isn’t unique to academic life. The modern economy increasingly requires workers to solve complex, open-ended problems.

The previous sentence sounds suspiciously like copypasta from a McKinsey white paper. But I can show you that it really is the case!

To do that, we stitched together nearly 60 years of job vacancy data across two different sources. For 1960 to 2000, we borrowed data from a fantastic paper called The Evolution of Work in the United States.

¹

The authors collected the text of classified job ads placed in The New York Times, the Wall Street Journal, and the Boston Globe.

²

For 2007-2019, we use data from Lightcast, a labor market analytics firm that collects online job vacancy text by scraping job boards and internal company websites.

In both data sources, we calculate the share of all job ads that use words and phrases like “decision” and “problem solving”, and we weight the job ads to be nationally representative of the occupation distribution in each year. The figure below shows that the share of all job vacancies mentioning decision-making increased from 5% in 1960 to 34% in 2018.

Figure 1: Share of All Jobs Requiring Decision-Making; Calculated using weighted job vacancy data, 1960-2018

People have higher earnings in jobs where they have to make a lot of decisions. The figure below shows a scatterplot of the average income in an occupation against its “decision intensity”, which we measure using O*NET, a survey of work activities administered by the U.S. Department of Labor to a random sample of employees in each occupation.

Figure 2: Decision-Intensive Jobs Pay Higher Wages

The bottom line is that workers are increasingly valued not only for how much they can do, but also for their ability to decide what to do. 

Workers as widgets

This may seem obvious. But labor economists haven’t paid much attention to the relationship between worker decision-making and productivity. I suspect that’s because human capital theory blossomed during a time when most workers held routine, “assembly line” jobs. Mass manufacturing plugs workers into isolated parts of a production process (with hilarious depictions in popular culture like Lucy and Ethel at the Chocolate Factory or Charlie Chaplin’s “Modern Times”). In a competitive economy, workers who produce more output per hour on the assembly line will get paid higher wages.

³

As observed by Mincer (1958), Becker (1962), and many others, educated workers have higher wages, so they must be more productive (or in econ-speak, they must have a higher marginal product of labor). Education increases productive efficiency.

Yet the assembly line analogy doesn’t work very well for some jobs. We know that education improves managerial performance. But managers don’t really produce anything. Most jobs are a mix of production and allocation, but in practice we think much more about the former than the latter. How does human capital matter for decision-making?

Measuring decision-making skill – the Assignment Game

In a new paper released today, we develop a theory and measurement paradigm for assessing individual variation in the ability to make good decisions about resource allocation. We call this allocative skill.

(The link above is ungated - but here is the NBER working paper version as well.)

The first part of the paper is a formal economic model of decision-making when information about productivity is costly to observe. We consider a decision-maker who assigns factors of production to job tasks to maximize total output. Our experiment asks participants to play the role of a manager (decision-maker) assigning workers (factors) to jobs.

⁴

Workers are better at some tasks than others, and their performance fluctuates on a day-to-day basis. Managers can observe performance differences, but it takes some effort to do so. If managers have perfect information (e.g. productivity in each task is stamped on workers’ foreheads), the right answer just involves some arithmetic.

To be concrete, here is a modified screenshot of the Assignment Game:

The 3x3 table on the left-hand side shows each worker’s potential output in each task. Worker 1 could produce 9 units of output if they were assigned to Task A, 10 in Task B, and 8 in Task C. Given unlimited time, most people can solve this by staring at it long enough or perhaps just computing each of the 6 possible combinations and choosing the highest one. In the example above, we show a case where a person picks an assignment that is not the very best, but still better than random guessing.

Real-world assignment problems aren’t nearly as simple, of course. We rarely have a good measure of productivity, and the decision space is usually much larger than 3x3. However, the advantage of considering such simple cases is that we can control the parameters of the decision problem. We can vary the difficulty of the assignment problem (for example, a 4x4 is much harder than a 3x3, because there are 4!=24 possible assignments rather than just 6). We can control the amount of time people have to solve it as well as the payoff of achieving a better answer. By giving workers and tasks anodyne labels like 1, 2, 3 and A, B, C, we can ensure that all participants start from the same base of knowledge.

⁵

If we hold the key features of a decision problem fixed, and give people lots of different problems, variation in overall performance must be due to individual differences in the ability to figure out the best assignment for each worker. We call this allocative skill. Allocative skill includes pure information processing bandwidth, but also the ability to deploy attention strategically when the problem gets too complex to remember everything.

⁶

Just as productive human capital increases the marginal product of labor, allocative skill increases the marginal product of attention. The point of the model is to show this formally.

Allocative skill predicts higher earnings, especially in decision-making jobs

We test whether allocative skill matters by fielding a survey of more than one thousand full-time U.S. workers ages 25-55.

⁷

Our sample is more educated than average but otherwise representative of the working population. We paid people for their performance on 4 cognitive tests, including the Assignment Game but also a traditional nonverbal IQ test, a numeracy test, and a measure of non-instinctive, “system 2” reasoning called the Cognitive Reflection test. We also asked questions about demographics, labor income, and occupation.

People who score higher on the Assignment Game (AG) have significantly higher incomes. That by itself isn’t surprising. But it turns out that the positive relationship between the AG and income holds even after controlling for IQ, the other cognitive tests, education, demographics, and current occupation. When we put all tests on the same standardized scale, the association with income is more than twice as large for the AG than it is for IQ.

We also find that the AG score is more predictive of income for workers in decision-intensive occupations. A one standard deviation increase in the AG score is associated with an increase in income of more than 10 percent for managers, lawyers, engineers – jobs at the far right of the second figure above. On the other hand, AG score does not predict higher income for jobs like sales representative, administrative assistant, or retail service worker, where decision-making is infrequent.

We don’t think of the Assignment Game as the best or the only way to measure the skill of economic decision-making, but we think it’s a good start. The test is clearly very abstract and does not correspond very well to real life, but it measures something real.

Because the Assignment Game measures your ability to make good choices about the allocation of scarce resources, we like to think of it as an “economic IQ” test.

If you are interested in the details, please read the paper! And play the game! We have big plans for the Assignment Game and for this research agenda more broadly, including fielding the Assignment Game in labor market surveys and administrative data in the U.S. and other countries.

1. Full disclosure – I was the handling coeditor for this paper at the AEJ: Applied, which is how I know what a great paper it is! You should read it.

2. The young’uns among us may not remember the pre-Internet era, but classified ads in newspapers used to be the main way that companies advertised job openings. Now everything is online, but there was a brief interregnum in the early 2000s when classified ads were no longer used but online job ads also weren’t so common. Starting in the late 2000s, most jobs were posted online through internal company sites or job boards like Monster.com and Careerbuilder.com.

3. The classical assumptions of a perfectly competitive labor market need to apply – many equivalent firms (so none has any market power, and all have to pay market wages) and perfect information (e.g. a worker’s productivity is common knowledge to all employers).

4. In principle, the model could also apply to a person allocating their own effort over job tasks, or other similar cases.

5. Performance variation between decision-makers might arise from differences in prior knowledge, even if they are given the same information in the moment. This is why chess grandmasters can beat 20 people at the same time in simultaneous exhibitions – prior expertise allows them to process complexity much better than novices.

6. Notice in the figure that worker 3 has the greatest variability in performance across tasks (worker 1 is good at everything, and worker 2 is a complete dud). The manager in the figure would have achieved the best allocation if they had started by assigning worker 3 to the best use (task C).

7. We recruit participants on the research platform Prolific. To make sure we get “real” full-time workers, we do three things – 1) screen out at the beginning anyone who is unemployed or working part-time; 2) conducted the study on weekends so that full-time workers could participate; and 3) screened out anyone who reports earning 10 percent or more of their total income from sites like Prolific. We also employed some bot-proof attention checks and dropped anybody who failed one of them.