Imagine finishing a day’s work convinced that your new AI assistant has saved you an hour. Then discovering that it has actually kept you at your desk longer. That is roughly what happened to a group of experienced software developers in a randomised trial published last year.
The research organisation METR recruited 16 developers to work on software projects they knew well. They brought 246 real tasks to the experiment. For each task, random assignment determined whether they could use AI assistance or would work with conventional tools. Before they began, the developers expected AI to cut the time required by about a quarter.
The result went the other way. With access to AI, they took 19 per cent longer. And here’s the best bit of the study: even after doing the work, developers still believed AI had saved them 20 per cent of their time (Becker et al. 2025).
Remember that a lot of the time when we’re doing evaluations, our approach is to ask people how the intervention affected them. We want to prioritise local voices, recognising that those affected by a new tool, program or policy know best.
Except in this case they didn’t. If we’d assessed AI based on surveying the developers, we would’ve come away with a funky story about boosting productivity. The people closest to the work would have supplied the evidence themselves. METR only got to the truth by using random assignment, which busted the myth that the new tool raised productivity.
I’m not making a general point about AI. METR’s follow‑up, released this February, suggested newer tools were helping more, although the researchers cautioned that their results were inconclusive (METR 2026). And we’re aiming at a fast‑moving target, since November’s AI is far more capable than February’s AI.
Tech changes, but the human tendency to believe our own impressions will be familiar to anyone who has championed a policy. Evaluation gives us a way to check those impressions against what actually happens.
Nearly a century ago, the Austrian‑born philosopher Karl Popper argued that science advances when we give our favourite theories a chance to be proved wrong. If a theory can explain every possible outcome, it’s not a scientific theory. And a good test leaves room for the evidence to surprise us.
Let me give you a chance to be surprised. One of my favourite emails is the Coalition for Evidence‑Based Policy’s ‘No‑Spin Evidence Reviews’, which report on randomised trials, checking the results against preregistered analysis plans. Four recent reviews assessed personal supports for low‑income vocational education students, information about financial aid at a prestigious university, a pay‑for‑performance program designed to prevent heart attacks among older Americans, and a pay‑for‑performance approach to deliver care to adults with serious mental illness (Coalition for Evidence‑Based Policy 2026). Take a moment to guess which programs worked, and which didn’t.
If you guessed that the first 2 worked, and the last 2 didn’t, you’re right. That’s not because education programs are designed better than health programs, or because pay‑for‑performance is ineffective. It’s because problems such as improving educational outcomes for disadvantaged students and preventing heart attacks are difficult to solve. Good intentions aren’t enough – we also need to check our work.
Today’s event has demonstrated that willingness to check. The Australian Centre for Evaluation and the Behavioural Economics Team of the Australian Government have brought their talents together around a common question: how can government learn enough from its work to do it better?
My friend Dean Karlan has shown us what careful empirical work can contribute to government. The eSafety panel has shared early insights into the social media age restrictions, informing the implementation of a world‑first policy. Our final discussion with senior public service leaders examined how evidence reaches decisions, and the obstacles it encounters along the way.
ACE and BETA help agencies ask useful questions and choose methods capable of answering them. They also help public servants acquire the skills to do this work themselves. The goal is for evaluation to become part of program design, so the initial rollout also allows us to test impact.
I’m delighted to see the enthusiasm for that approach across the public service.
Evidence that changes decisions
Consider the recent Review of the Electric Car Discount, led by ACE.
The review combined stakeholder consultation with a difference‑in‑differences analysis. It compared changes in sales of battery electric vehicles priced below the discount’s eligibility threshold with changes for vehicles priced above it.
That comparison helped answer a question at the centre of any subsidy review: how much of the activity would have happened anyway?
The review estimated that the discount generated around 64,000 additional battery electric vehicle sales during its first 3 years. That is a much more credible measure of its effect than simply counting every eligible car sold.
Alongside stakeholder perspectives, the analysis informed a recommendation that the government consider changes to manage the discount’s cost, including adjusting eligibility thresholds or phasing it down. The Australian Government subsequently announced changes consistent with those findings.
The electric car market has developed rapidly since we came to government. Tax settings should keep pace. The review helped us assess what the discount had achieved and how it should adapt as the market matured (ACE 2026a; Chalmers and Bowen 2026).
For a different kind of evaluation, take something as common as a phone call.
Services Australia often sends a text message before an officer calls a customer. Earlier analysis suggested that doing so increased the chance of successful contact by around 5 percentage points. That was promising, although other differences between the customers receiving a text and those receiving a call alone could have contributed to the result.
Services Australia worked with ACE to run a randomised trial.
The effect was larger than the earlier analysis suggested. A pre‑call text lifted successful contact from 52 per cent to 64 per cent (Services Australia 2026).
For every hundred customers called, that meant 12 more successful contacts. Across a large service, those extra conversations can save officers time and help customers resolve their business sooner.
The finding supports Services Australia’s continued use of SMS alerts across existing and new service areas. It also shows how much we can learn by examining a routine part of the working day. With more of us screening our calls, a heads‑up about an important phone call can make people more likely to pick up.
BETA’s work with Treasury on banking is another example. They tested ways to encourage customers to seek a better deal on their home loan or savings account.
In the first trial, a message that customers could easily overlook had little effect. A second trial showed the value of giving the message greater prominence and tailoring it to the recipient.
When customers were told prominently that they might qualify for a lower rate on their particular loan, the share contacting the bank rose from 4.8 per cent to 7.2 per cent, an increase of 50 per cent (BETA 2026).
For someone trying to decide whether a call to the bank is worth the trouble, information about their own mortgage gives them a reason to do something. The experiment helped establish which approach could prompt that conversation.
This work builds on the Treasurer’s initiatives to help consumers secure better deals on banking products (Chalmers 2024).
These evaluations cover decisions ranging from tax settings to the wording of a customer message. Their findings give the people responsible for those decisions a way to judge whether a change is worth making.
Making evaluation part of the job
For researchers, a rigorous evaluation is an end in itself. But for government, the goal is to use evaluation to shape better policy. For that to happen routinely, people need access to the findings and confidence in how they were produced.
The eSafety Commissioner’s evaluation of Australia’s social media age restrictions illustrates the point. As you heard in the panel session, early insights were shared at the three‑month follow‑up (eSafety 2026a). The evaluation protocol was also published on the Open Science Framework, a public research platform, in February, ahead of the three‑month follow‑up survey in March (eSafety 2026b).
Preregistration helps avoid the risk of researchers going looking for significant results. A standard benchmark for statistical significance in the social sciences is a 5 per cent significance threshold. That means that even when a program has no effect, a statistical test will produce a significant result about one time in 20.
Yet if researchers have 20 possible outcomes, pure chance will make it likely that one will be statistically significant – even if the intervention truly has no effect on any of them. And without preregistration, they might convince their audience (and maybe even themselves) that there is an excellent hypothesis that explains why the one significant outcome deserves to be spotlighted in the writeup.
Those who’ve watched Christopher Nolan’s bloodthirsty epic The Odyssey will remember how Odysseus arranged to hear the Sirens while tied to the mast. He understood that his judgement might change once the singing began, so he made his decision beforehand.
Publishing a protocol serves a similar purpose. Evaluators set out their questions and intended methods while the answers are still unknown. Readers can then distinguish tests planned in advance from discoveries made while exploring the results.
Alongside this is the file drawer problem. Suppose that 20 researchers run randomised trials that analyse the same ineffective program. On average, one of them would find a statistically significant result at the 5 per cent level. If that one study gets published, while the other 19 evaluations go into the file drawer, the rest of us might get the misleading impression that the ineffective program really works.
That’s why it’s valuable for researchers to have access to all the relevant research, not just those studies with 2 asterisks next to the outcome variable.
ACE’s Evaluation Library makes scrutiny easier across government. Launched last year with the Analysis and Policy Observatory, it brings Commonwealth evaluation reports together in an open‑access collection. It now holds more than 226 evaluations, including 18 randomised trials, and continues to grow.
That gives a public servant starting a new project somewhere to look for what others have already learned. It also gives researchers and the public a better view of how government programs perform.
ACE and BETA have each committed to publishing their evaluations by default. As BETA celebrates its tenth birthday, its website has more than 56 published reports. As a younger institution, ACE is taking the same approach.
Another aspect of building evaluation capacity across government is to give people the skills to run rigorous evaluations.
Through ACE’s ‘train the trainer’ model, departments can deliver rigorous training suited to their own work. ACE also offers training through professional networks, helping people build their skills alongside colleagues facing similar questions.
Its formal departmental partnerships include Health, Disability and Aged Care, as well as Social Services. Education is also a partner. Projects are underway with Home Affairs and Services Australia, alongside work within Treasury.
According to the 2026 State of Evaluation report, 34 of the 77 agencies responding to the survey had a dedicated evaluation unit, up from around 18 in 2021. Another 5 agencies planned to establish a unit by 2028 (ACE 2026b).
A dedicated unit helps agencies access the existing evidence base, and design an evaluation that will add to that evidence base. It helps ensure that the government is a learning machine, whose policy knowledge grows year by year. That is stewardship in action.
Bringing evaluators together
Another way that evaluation is being strengthened across government is through the APS Evaluation Profession. The Evaluation Profession now has more than 8,000 members, enough to fill Margaret Court Arena in Melbourne. It holds regular events and hosts an online community where people can learn from one another.
The Impact Evaluation Practitioners Network, established by ACE and the Australian Education Research Organisation, has grown to more than 700 members over 2 years. It brings together specialists from government and academia.
ACE also chairs the Australian Public Sector Evaluation Network in collaboration with the Australian Evaluation Society. It’s a place for public servants to meet academics and consultants who share an interest in rigorous evaluation of government programs.
You can find these networks through ACE’s website. I encourage you to join the conversations and bring along a question from your own work.
Today we also mark the launch of Evidence Network Australia, funded by Minderoo.
The network will connect people who produce evidence with those who use it, including the people whose job is to help research find its way into practice. It aims to make expertise easier to find and help organisations work together. Better connections should also reduce duplication, so people can build on work already done elsewhere.
We are joined today by network lead Sharon McGowan and coordinator Susannah Schoeffel. They will be pleased to tell you more afterwards. I also acknowledge Zoe Jordan, the inaugural chair of the network, who couldn’t attend today.
Eleanor Williams sits on the advisory committee alongside Australian Statistician David Gruen. Productivity Commissioner Selwyn Button is also a member. Their involvement brings experience of both producing evidence and getting it used in public decisions.
I’m excited to see the collaborations that emerge from Evidence Network Australia.
The next stage
What’s on the horizon? The Strengthening Evaluation in the Australian Government Action Plan 2026–2030 sets out 14 actions to take this work further. It covers leadership and capability, along with planning and the use of evidence (Australian Government 2026).
The aim is close to social scientist Donald Campbell’s idea of an ‘experimenting society’, in which proposed solutions are tested and policy develops in response to the results (Campbell 1991).
For government, that requires a sense of modesty, which may not come naturally to everyone. When you’ve spent years developing a program, it can be hard to accept an evaluation that shows it to be ineffective. When an intervention feels effective, it might initially be hard to trust the data.
Yet when budgets are tight, good vibes aren’t enough. We owe it to taxpayers to evaluate programs rigorously.
Medical researchers have been dealing with disappointment for decades. In most cases, pharmaceuticals that look promising in the laboratory must pass through 3 stages of clinical trials before they make it to the market. Nine out of 10 don’t make it (Leigh 2018).
That system ensures that medicine keeps learning. When you go to a doctor today, they have better knowledge and better treatments than they did a decade ago. In a decade’s time, their knowledge and treatments will be better again.
Randomised trials are a big part of this. Recall for example the randomised trial which found that pay‑for‑performance didn’t reduce costs for adults with serious mental illness. The study also noted that a less rigorous pre‑post study would have incorrectly found large savings, illustrating why a control group is essential (Blonigen and Humphreys 2024).
In policy, randomised trials can put us on the same virtuous cycle, giving governments a chance to keep learning. A well‑designed evaluation can tell us where an approach is succeeding and where we should change tack.
That matters for productivity. Public money goes further when we understand which programs deliver worthwhile results. Evaluation can help improve the way a service operates, as the Services Australia trial showed, or inform a larger decision about where resources should go.
The Productivity Commission has advocated rigorous evaluation. CEDA has called for evaluation to be built into program design, drawing on its examination of community services. The OECD has emphasised the contribution of impact evaluation to better resource allocation and public services (Productivity Commission 2017; Winzar, Tofts‑Len and Corpuz 2023; Jacobzone et al. 2025).
CEDA’s review found cases where ineffective programs continued while effective ones ended. Better evidence, used at the right time, can help governments make those choices more wisely.
There is a democratic case here too.
Testing a policy is a way of treating the public’s confidence with care. Government puts its judgement up for examination and allows people to see the basis for the next decision. Publishing the results gives citizens a chance to assess the reasoning for themselves.
Randomised trials have a particular advantage in that conversation. Allocation by chance creates groups that are comparable on average. Differences in their outcomes can then reveal the effect of the policy being tested.
By helping establish cause and effect, randomised trials give citizens a better basis for assessing claims about what government has achieved (Tanasoca ⓡ Leigh 2024; Leigh 2025).
Australia has all the fundamentals to be a world leader in randomised trials. We have skilled public servants and terrific linked microdata. Modesty comes naturally to us. We have a strong medical research sector that has been carrying out clinical trials for many decades.
So to everyone in government, if you haven’t run your own randomised trial, give it a shot. If you’re tasked with sending out a mass email, try 2 versions, and see which produces a better response. If you’re trialling a new technology, consider randomly selecting the beta testers. If you’re progressively rolling out a new program, consider doing the rollout randomly so you can learn as you go.
If you can, preregister your randomised trial before you start. Write up your findings afterwards, regardless of whether they’re positive, negative or zero. Each study adds to our knowledge of the world, and each rigorous evaluation of a government program helps government do a better job. None of us are here forever, so the least we can do is to ensure our successors know more than we did when we began.
Thank you to ACE and BETA for bringing us together, and to everyone who has shared their work today.
Food and drinks await on the mezzanine in the gallery foyer. Please stay for a discussion with someone whose work caught your interest. If you’re not sure who to chat with, choose randomly. A conversation this afternoon might become the next useful experiment.
Note: My thanks to officials at the Australian Centre for Evaluation for valuable drafting assistance.
References
Australian Centre for Evaluation (ACE) 2026a, Statutory Review of the Electric Car Discount: Final Report, Department of the Treasury, Canberra.
Australian Centre for Evaluation (ACE) 2026b, state of Evaluation in the Australian Government 2026, Department of the Treasury, Canberra.
Australian Government 2026, Strengthening Evaluation in the Australian Government: Action Plan 2026–2030, Department of the Treasury, Canberra.
Becker, J., Rush, N., Barnes, E. and Rein, D. 2025, ‘Measuring the Impact of Early‑2025 AI on Experienced Open‑Source Developer Productivity’, arXiv, 2507.09089, https://arxiv.org/abs/2507.09089.
Behavioural Economics Team of the Australian Government (BETA) 2026, Prompting for a Better Deal: Understanding and Overcoming Barriers to Financial Product Switching, Department of the Prime Minister and Cabinet, Canberra.
Blonigen, D.M. and Humphreys, K., 2024. A randomized controlled trial of a pay‑for‑performance initiative to reduce costs of care for high‑need psychiatric patients. Psychiatric Services, 75(9), pp.863–871.
Campbell, D.T. 1991, ‘Methods for the Experimenting Society’, Evaluation Practice, vol. 12, no. 3, pp. 223–260.
Chalmers, J. 2024, Helping Australians Get a Better Deal on Banking Products, media release, Department of the Treasury, Canberra.
Chalmers, J. and Bowen, C. 2026, Joint Media Release: Fairer Tax Treatment to Encourage Affordable EVs, media release, Australian Government, Canberra.
Coalition for Evidence‑Based Policy. 2026, ‘No‑Spin Evidence Review’, 26 October, available at https://nospin.evidencebasedpolicy.org/.
eSafety Commissioner 2026a, Early Insights from eSafety’s Comprehensive Evaluation Project, media release, July, eSafety Commissioner, Canberra.
eSafety Commissioner 2026b, eSafety Begins Evaluation of Australia’s World‑First Social Media Minimum Age, media release, February, eSafety Commissioner, Canberra.
Jacobzone, S., Picalarga, S., Dynes, L., Williams, E., Greenwell, H. and Bowers, P. 2025, ‘Unleashing the Policy Potential of Rigorous Impact Evaluation and Randomised Trials’, OECD Working Papers on Public Governance, no. 87, OECD Publishing, Paris.
Leigh, A., 2018, Randomistas: How Radical Researchers Are Changing Our World. Yale University Press.
Leigh, A. 2025, ‘The Courage to Learn’, The Monthly, November, pp. 57–63.
METR 2026, ‘We are Changing our Developer Productivity Experiment Design’, 24 February, https://metr.org/blog/2026–02–24‑uplift‑update/.
Productivity Commission 2017, National Education Evidence Base, Inquiry Report, Productivity Commission, Canberra.
Winzar, C., Tofts‑Len, S. and Corpuz, E. 2023, Disrupting Disadvantage 3: Finding What Works, Committee for Economic Development of Australia, Melbourne.
Services Australia 2026, Pre‑Call SMS Randomised Trial: Evaluation Report, Australian Government, Canberra.
Tanasoca, A. ⓡ Leigh, A. 2024, ‘The Democratic Virtues of Randomized Trials’, Moral Philosophy and Politics, vol. 11, no. 1, pp. 113–140.