Measures to promote SME participation or other policy goals should be based on robust evidence of what works in public contracting – not on who shouts the loudest.

As public authorities scramble to implement thePublic Contracts Regulations 2015, many questions are being asked about the new rules aimed at improving SME access to contracts. For contracts above the EU thresholds, the process of selecting bidders has been reformed in various ways, such as limiting the financial turnover which can be requested and allowing companies to rely upon an updated self-declaration of their eligibility to bid. It is also possible to limit the number of lots which any one company may win and to make provision for the direct payment of subcontractors, with undisputed invoices to be paid within 30 days. None of these measures is particularly radical, and arguably any positive impact they may have on smaller companies bidding for contracts will be outweighed by the shorter time limits which now apply in respect of all procedures – a challenge for those companies who lack a full-time bid team.

What is more radical is the ban on pre-qualification of bidders which has been adopted for contracts below the EU thresholds but valued above £10,000 for central government and £25,000 for other public bodies. This measure, set out in Regulation 111, is based upon the recommendations which Lord Young of Graffham made in his 2013 reportGrowing Your Business. Lord Young met with a number of businesses and their associations as well as drawing upon his own experience to develop recommendations regarding how to increase the percentage of public contracts won by SMEs. Amongst his recommendations was a ban on pre-qualification questionnaires (PQQs) below threshold and the use of a single standard PQQ for above-threshold contracts in order to “avoid gold plating with a plethora of different locally-determined objectives.” These recommendations have now been implemented via the PCR 2015 andrecent guidance from the Cabinet Officeregarding the use of a standard PQQ.

So will it work to get more SMEs on to the list of government contractors? This depends very much on SMEs themselves deciding to apply for contracts. There has never been any evidence of systematic discrimination against SMEs either by central or local government – the argument was that the process of pre-qualification was overly bureaucratic and sometimes included inappropriate requirements, for example relating to financial standing or previous experience. This was supported by largely anecdotal evidence and an apparent gap between the overall role of SMEs in the economy and their share of public contracts – however such figures seldom account for the particular sectors of the economy in which SMEs are active. It is possible that removing PQQs will help to boost the number of smaller companies winning contracts, but it also comes at considerable cost to both the public and private sector. By effectively mandating the use of the open procedure for all below-threshold contracts, the new rules mean that many more tenders will be prepared and evaluated – a hugely resource-intensive task on both sides. Any policy which has the potential to incur such costs should be subject to rigorous assessment before it is adopted.

The idea of using public contracts to implement various government policies is enjoying a renaissance. The new EU procurement directives specifically endorse the idea of using public contracts to further environmental, social and innovation objectives. The potential to leverage large volumes of public spending to achieve broader goals is attractive, but the problem is we have very little idea of whether it works in most cases. I have worked with many public bodies who have seen successes in their strategic use of procurement, as well as learning from their own mistakes and the experiences of others. But there is a difference between this kind of institutional learning and the type of evidence which we need to support regulation, especially if that regulation comes with clear costs and somewhat less clear benefits. If we wish to undertake major changes in procurement policy and practice, there is a role for more scientific approaches such as randomised controlled trials to test new initiatives.

Randomised controlled trials (RCTs) are already used in many areas of public policy, and are championed by the Cabinet Office’sBehavioural Insights Teamand theWhat WorksCentres. The basic idea of an RCT is that to test any particular policy, you randomly select a sample to undergo an intervention and another from the same population to act as the control group. You then measure the results over time. The random part is important, as is the control group. RCTs can help to avoid expensive policy mistakes as well as to fine-tune a policy prior to rolling it out on a larger scale. They are not necessarily expensive to run, but they do require a reasonable sample size to produce reliable results, and effects may require some time to measure properly. They have been used in the UK to test interventions ranging from the use of text messages to increase payment of court fines to the use of incentives to boost recycling rates – as well as their widespread use in medicine and marketing. The growing body of data which will be disclosed on Contracts Finder provides an opportunity to apply this approach in procurement, as we will have a clearer picture of the overall contract population.

To test the effectiveness of policy interventions aimed at increasing SME participation in public tenders for example, it would be relatively easy to identify a set of similar contracts and randomly apply different interventions. NHS contracts or frameworks established by the Crown Commercial Service often have different lots, for example where they are divided on a regional basis or between different categories of similar products or services. Lots could be subject to different measures intended to boost SME participation: pre-procurement market engagement, setting a maximum contract size, allowing longer to respond, removing pre-qualification questions, et cetera. No such measures would be taken for the control group. The results of this kind of experiment are generally more valuable than selective assessments of what works based on anecdotal or expert evidence – which tend to be imbued with our prior biases and ignore null results. They do not replace the need for expertise or take policy entirely out of the realm of politics; someone needs to decide which questions to ask and how to interpret and act upon the findings.

If this approach to developing better procurement policy sounds expensive or labour-intensive think about the alternative: major changes which may actually have the opposite effect to that intended or come with unanticipated side-effects, adopted on little more than a finger in the wind. RCTs are not the only source of evidence on procurement, but together with the new sources of data about public contracts coming online, they point towards a better-informed future.