Andrew Leigh

Running Better Evaluations

The Daily Telegraph

As a keen runner, I’m pretty keen on going where the evidence points. In experiments, high intensity training produces remarkable gains, so I try to build it into every workout. Randomised trials suggest that stability shoes don’t much affect foot strike, so I stick with cushioning shoes instead. After a long run, I’ll use compression socks, since they’ve been shown to improve recovery. I try to eat a Mediterranean diet, randomised trials show to be linked with better heart health. I don’t bother with vitamin pills, which haven’t been shown to improve longevity among otherwise healthy people.

When it comes to improving your workout, the evidence base is remarkably good. The top athletes are always testing new training techniques and gear, and the results trickle down to also-rans like me. But when it comes to figuring out what works in policy, the evidence is a good deal patchier. Unlike the search for the best sneakers and health supplements, there’s a whole lot less impact measurement. Particularly under the last government, too many decisions were made by the principle of GOBSATT: good ol’ boys sitting around the table.

Putting the what works approach into the heart of government often produces surprises. A recent randomised trial found that offering pregnant smokers shopping vouchers to engage with stop smoking services doubles quit rates. Another randomised study found that WeightWatchers produced little or no lasting weight loss after five years.

When it comes to job training programs for low-income adults, high-quality evaluations of ten US programs found that nine didn’t work. But the one that did was a blockbuster: producing long-term earnings gains of 30 percent. I’d wager that only a tenth of us would have picked the one effective program from the nine ineffective ones. Which is the point about a good evaluation. If you’re not getting surprised, you’re not doing it right.

Finding out that a program doesn’t work is a success, not a failure. Sure, it’s frustrating to put time and effort into designing something that doesn’t have the intended impact. But isn’t it worse to keep doing the wrong thing year after year? In the case of job training programs, just imagine how many more people could be helped by shutting down the nine ineffective programs, and scaling up the effective program tenfold.

That’s why the Albanese Government is committing to improving the quality of evaluation across government. It’s about measuring what matters, and figuring out what works. This is the same practical approach to government that good athletes have to their training. When you hear that a rival has a clever way to move faster, jump higher or score more accurately, you try it out for yourself. If it works, you keep it. If it doesn’t, you keep trying. The silly idea of ‘I won’t try it because I didn’t invent it’ is unheard of on the sporting field. And it doesn’t make much sense in policymaking either.

Once you’ve been thinking about policy for a few decades, you come to recognise that many people are wasting time searching for panaceas and silver bullets. But life isn’t like that. From getting in shape to building a house, most of our achievements are made one step at a time, one nail at a time, one conversation at a time. Yet when we’re on the right path, we can eventually end up making huge gains. Good evaluation is simply about building a better feedback loop – to ensure we’re on a steady path to improvement. Try, test, adapt. And then do it again.

Originally published in the Daily Telegraph on the 7th of February 2023.