Summary
Many organizations seek to procure products that are not commercially available, and have to manage incentives for potential suppliers to conduct research and development (R&D). Understanding the design of competitions for R&D-intensive products is important, but economic theory gives ambiguous answers. Natural questions include the number of competitors to invite and the size of the rewards for competing firms. The precise answers will depend on the context, and investigating this may need to call on both theory and data.
One especially interesting setting is that of the United States Department of Defense (DoD). The DoD is one of the largest single R&D investors in the world, and responsible for over half of the federal R&D conducted in the United States each year. A recent paper studies the design of the DoD’s Small Business Innovation Research (SBIR) program. This program encourages firms to develop and deliver products that could be militarily useful.
The author develops a model of competition in R&D procurement contests. Using data on R&D and procurement contracts associated with the SBIR program, combined with the structure of the model, the author investigates the nature of R&D and the incentives embedded in the program. He finds that uncertainty is endemic throughout the entire research process, including its later stages. Despite this uncertainty, firms are provided with fairly strong incentives to conduct research throughout the process.
The analysis indicates that the program could be significantly improved. Although achieving full efficiency might be difficult, large social gains would come from simple design changes, such as modestly stronger incentives for potential suppliers, or more competition in the later stages of the contest. Why does the DoD not make these changes? The author finds that socially beneficial changes would typically lower the surplus captured by the DoD. In effect, the current design of incentives may reflect a tendency to seek innovation too cheaply, so that reform could bring benefits. The research casts new light on R&D procurement, and shows how even simple changes to program designs can achieve better outcomes for society as a whole.
Main article
Organizations often seek to procure innovative products that must be researched, designed, and developed. This raises an open but understudied question: how to best structure incentives for potential suppliers. A recent paper examines an important procurement program run by the United States Department of Defense (DoD). The paper finds that the current program design leaves substantial social gains on the table, but capturing these gains for society might reduce benefits to the DoD. By combining theory and data, the research shows how to understand the context and incentives at play in structuring R&D procurement, and how simple changes to program designs can achieve better outcomes.
Many buyers, such as firms and government agencies, acquire products that are not commercially available — from customized parts for automobiles or airplanes to major weapons systems with novel capabilities. Suppliers who want to compete for these procurement contracts must engage in costly R&D to design and develop the relevant products. Accordingly, the potential suppliers decide how much to invest in R&D, and procurers must decide how best to manage incentives for suppliers to encourage them to invest effectively.
Understanding how to structure incentives for innovation — both in the context of these R&D procurement contests and more generally in other product markets — is an important policy question. Worldwide R&D spending exceeded $2.2 trillion in 2019 (AAAS, 2021; CRS, 2021). The US accounts for about 30% of the world total. Governments are a major player: about a quarter of US R&D spending is due to the federal government, and the Department of Defense (DoD) accounts for more than half of this. The DoD is one of the largest single R&D investors in the world, and the focus of the research summarized in this column. Beyond the size of this particular market, its wider lessons matter, since economists generally regard innovation as a key driver of overall economic growth.
How should we organize markets to induce innovation? There is unfortunately no clear answer to this question from earlier research. Theoretical analysis shows that the relationship between market structure and innovation is ambiguous (Arrow, 1962; Gilbert and Newbery, 1982; Aghion and Howitt, 1992; Vives, 2008). Empirical analysis also suggests that it can vary with the context. Researchers have documented either increasing or “inverted-U” relationships between the extent of competition and innovation (see Blundell, Griffith, and Van Reenen, 1995, 1999; Aghion et al., 2005).
When studying this further, R&D procurement contests can be seen as a useful microcosm of more general markets for innovation. The procurer can control, at least to some degree, the extent of competition and the incentives for suppliers. But designing R&D procurement contests — whether to maximize the efficiency of procurement, or profits to the procurer — remains difficult in any given setting, since there are competing forces at play. Do suppliers typically expend more R&D effort than a social planner or benevolent government would want, or less? The structure of contests makes the answer ambiguous. On the one hand, suppliers do not capture the full benefit of their innovations: some benefits are lost to them through competition with other suppliers and bargaining with the procurer. On the other hand, suppliers have an incentive to use extra R&D effort to “steal” business from their competitors.
Would increasing the number of potential suppliers improve R&D outcomes? The answer is again unclear: while inviting more competitors seems to imply more chances to innovate successfully, all competitors would adjust how much effort they make, since they know the benefit of effort has changed (Taylor, 1995; Che and Gale, 2003). We have very little evidence in any setting, as empirical researchers have only just begun to analyze R&D contests, mostly focusing on online “ideation” competitions for sourcing ideas (Kireyev, 2020; Lemus and Marshall, 2021).
The Small Business Innovation Research Program
The US government runs the Small Business Innovation Research (SBIR) Program to help small firms bring early-stage technologies to market — either privately or to the government. The Department of Defense, and particularly the Navy, uses SBIR as a core part of its acquisition program, soliciting research on products that it wishes to acquire. The process is structured as a three-phase contest: broadly speaking, firms conduct benchtop testing in Phase I to determine technical feasibility, develop a prototype in Phase II geared towards determining commercial viability, and work with (say) the Navy to deliver the product in Phase III. The Navy whittles down the participants over three phases: about 40% of Phase I contestants are invited to participate in Phase II, and at most one firm earns a Phase III contract. Participants receive R&D contracts in Phases I and II. Where Phase III happens, it usually involves a sizable delivery contract.
The SBIR program is important in its own right, accounting for about 3% of the federal government’s large R&D budget, and has been studied both inside and outside the DoD (Lerner, 2000; Wallsten, 2000; Howell, 2017; Howell et al. 2021). It also provides a laboratory to study the design of R&D procurement contests. The three-phase structure induces competition between suppliers, with firms capturing only a portion of the surplus through the Phase III contract. Moreover, there are many levers the DoD could adjust, from changing how many competitors they allow in each stage, to making the final delivery contract more or less generous. Finally, useful data are readily available: the sizes of the contracts and the potential competitors in each phase can be obtained from the Federal Procurement Data System, and the Navy releases all solicitations publicly.
The SBIR program is especially interesting as one of many settings in which the government is the procurer. A naïve view is that the government would seek to maximize social surplus: the value of the innovation, less delivery costs and total R&D costs. This may not be the case, however. The government may be more concerned about its own spending or seeking to meet a budget. Whether it is sensible to seek results “on the cheap” is relevant in other public debates, such as spending on social programs or purchasing vaccines during a pandemic. In the SBIR setting, it can affect the incentives of suppliers to innovate.
A Model of an R&D Procurement Contest
Does the current design of the SBIR program come close to maximizing surplus, as we might expect the government to want? Would simple design changes, such as increasing competition, increase total surplus or improve outcomes for the DoD? The difficulty in answering these questions from the data is that the SBIR program takes only one form: we do not see truly random variation in how many competitors are invited, or how generous the procurement contracts are, or the structure of the program itself.
To compensate for this limitation, we analyze the rich data through the lens of an economic model of R&D contests. In the model, firms choose how much effort to make in Phases I and II. Firms which exert more effort are more likely to get favorable draws of values (in Phase I) or costs (in Phase II), leading to innovations that (i) are more likely to be technically and commercially feasible and (ii) generate more surplus, conditional on being feasible. The DoD shares a pre-specified portion of the surplus generated with the winning firm, where this reflects the DoD’s bargaining power or desired level of generosity.
As in many models of firm behavior, if we want to move beyond direct analysis of the data, a key step is to assume that agents act optimally in some sense. Firms will spend an additional dollar in research effort only if they expect another dollar of benefits — by generating a better innovation and capturing some of the surplus from Phase III. The DoD offers a Phase III contract only if this would generate positive surplus.
Using such reasoning, we can recover values and costs from the data. First, the Phase III contract provides a lot of information on values and costs: a $10 million contract, say, means that (i) the supplier can deliver the product at a cost less than $10 million and (ii) the DoD values the product at more than $10 million. Second, the model implies that firms expecting to generate a higher-value innovation will have an incentive to conduct more R&D: thus, projects with a Phase II R&D contract worth $1 million were expected to generate more surplus (have higher value to the DoD) than projects where Phase II spending was only $500,000. The research described here uses these observations to understand the variation in values and costs at the end of each phase, and the benefits of conducting R&D, given a large sample of similar contests. The research also draws on rich data on contest characteristics, extracted from the text of the Navy solicitations and used to control for differences across contests.
The Design of R&D Contests
There are two main lessons from the model estimates. First, the randomness inherent in the innovation process is not quickly resolved: there is significantly more uncertainty over the outcome of Phase II than the outcome of Phase I. Second, the DoD provides high-powered incentives to firms, which capture about half the surplus they generate through innovation.
These estimates, together with an economic model, allow us to simulate outcomes under different contracting designs. The red bar to the left in Figure 1 shows the social surplus generated by the program (per contest) in the baseline design. Moving to the socially optimal design in this setting (red bar to the right) would almost double the total surplus generated per contest.
Why is the government leaving so much surplus on the table? The efficient design would commit the government to giving all the surplus to firms while charging (perhaps hefty) participation fees along the way, but this may be difficult to implement. Would simpler design changes be enough to improve surplus, perhaps substantially? The answer turns out to be “yes”. Figure 1 shows that modestly higher rewards (say, committing to give firms 61% of the surplus) increase total surplus by about 11%. Inviting more contestants to later phases of the contest, where there is still a fair amount of uncertainty over whether innovation will succeed, increases total surplus by 25%. This casts light on a classic question in economics, the relationship between competition and innovation: in this setting, competition seems to increase aggregate innovation. Other design changes that might be considered more drastic — combining Phases I and II, or forcing firms to share Phase I breakthroughs with all their competitors — also increase total surplus (by 23% and 47%, respectively). Hence, although it may be unrealistic for the DoD to implement an efficient design, simpler design changes seem to go a long way.
Figure 1. Social surplus under various designs of the R&D procurement contest
These findings could be seen as prescriptions for the DoD, to be explored if they wish to increase total surplus. But does the DoD genuinely try to maximize total surplus? Through the model, we can ask how the DoD would alter the design to maximize its own welfare: the share of the total surplus it captures, less the costs of its R&D contracts. Here, we find starkly different results. The DoD would want to weaken incentives provided to a firm, giving them only 34% of the surplus generated by the program. The DoD would want to reduce competition to about half the current level. The DoD would prefer to keep the stages separate, and would not encourage sharing intermediate breakthroughs where that required appropriate incentives. Broadly, any socially desirable design change would be harmful to the DoD’s private returns; the observed design can be seen as trading off social welfare with that of the DoD.
Policy Implications
Naturally, the findings of this research — including the qualitative design changes that would improve overall outcomes — are specific to the Navy SBIR program over the period studied. There are some wider policy lessons, however. The analysis helps to understand how firms and governments should procure innovative products. The structure of R&D procurement in practice may be far from achieving social efficiency, and simple design changes should not be overlooked when seeking to remedy this. But perhaps justifiably, public procurers may retain incentives other than maximizing total surplus, which will affect procurement design. By drawing on the research summarized here, governments may be able to achieve better outcomes for society as a whole.
This article summarizes “An empirical model of R&D procurement contests: an analysis of the DOD SBIR program” by Vivek Bhattacharya, published in Econometrica in September 2021.
Vivek Bhattacharya is at Northwestern University.
