Andrew Leigh

E&OE TRANSCRIPT

RADIO INTERVIEW

RADIO NEW ZEALAND

SUNDAY, 8 APRIL 2018 

SUBJECT: Randomistas.

WALLACE CHAPMAN: Dr Andrew Leigh is a former economics professor at the Australian National University, a former London  lawyer and has a PhD from Harvard. These days he's an Australian politician, the Shadow Assistant Treasurer and while his political ideology flirted for a while with Tony Blair's third wave, he prefers these days to talk about a special duty to look after the most disadvantaged. He's in New Zealand this week as part of the Presbyterian Support Northern Speaker Series on improving child wellbeing. He's an author and his latest book is 'Randomistas', looking at how randomised tests are carried out every day to find out what works, what doesn't. The tests are done by supermarkets, search engines, online dating sites and also by political parties who use randomised trials to try to win elections. His book tells the stories of researchers who fight to have the findings from their research implemented. Dr Andrew Leigh has taken time from his packed schedule. Welcome.

ANDREW LEIGH, SHADOW ASSISTANT TREASURER: Thanks Wallace, great to be with you.

CHAPMAN: Tell me, how did your interest in randomised trials actually start?

LEIGH: Well, it came originally when I was doing my PhD at Harvard University and working with a range of people who were just as passionate as I was about addressing social disadvantage, but had far fewer prejudices about which programs worked and which didn't. And it made me reflect on my own philosophy on social policy. I think I’d been very attracted to solutions and much less scientific and critical than for example a medical researcher who was trying to cure cancer. I didn’t realise that actually no matter how idealistic you are, lots of things that sound good turn out not to work so well in practice. One great example is Scared Straight, a program that puts troubled youths in jail for a day in order to scare them onto the straight and narrow, but actually turned out in rigorous randomised trials to increase offending rates.

CHAPMAN: We'll talk a little bit about that in a minute, but actually your book here, which I’ve been reading 'Randomistas: How radical research has changed our world’ - you start the book with a very interesting story about scurvy. So, one Commodore George Ansen and the terrible problems he had with scurvy. Tell us a bit about this.

LEIGH: Well indeed, your odds of dying of scurvy during the 1600s and 1700s were way higher than your odds of dying in battle. Even during the Seven Years War, it was scurvy that was doing most of the killing rather than cannons. And there were many solutions that were suggested, most of them turned out to be hocus-pocus, but one young naval doctor by the name of James Lind decided that he would do a simple randomised trial. He took twelve patients as similar as he could, put them into pairs and for each of the pairs of sailors tried six different solutions - sea water, vinegar, the various remedies that had been suggested and then of course, oranges and lemons. We know citrus is the way in which you ward off scurvy. And the citrus patients were straight back on duty within a few days. Everyone else was still lying in their hammocks with their connective tissue slowly separating as scurvy does to you. So that randomised trial really underpinned the way in which the British navy managed to get on top of scurvy ahead of the French navy. If you want to know why the Battle of Waterloo turned out the way it did, part of the answer is the British navy was fighting a scurvy ravaged fleet.

CHAPMAN: It was that profound.

LEIGH: Absolutely. The accounts of what scurvy does to you are just terrifying. Some of the sailors found that old battle wounds 20 years earlier would begin to open up. You couldn’t eat as your mouth slowly swelled. Sometimes boats would go down because there weren't enough men to keep them going, and the surviving men were too weak to swim ashore. But all you have to do is just put enough oranges and lemons on the ship, and Bob's your uncle, you're able to take on the world, and that was what the Brits did.

CHAPMAN: The simplest remedy to the most horrific disease, quite an extraordinary story, and so here we had a remarkable experiment by a young surgeon who said "what if we do this?". You know, we take a randomised trial and we give half the members some citrus. Did his randomised trial catch on? How long did it take for the British navy to catch on to this trial?

LEIGH: Well, it did take longer than you'd like. Part of the problem was that rather than simply saying "oranges and lemons work, I’m not precisely sure why", Lind instead came up with a whacky set of theories as to why oranges and lemons work. I tell the tale in the book about a man by the name of William Stark who, 16 years after Lind published his results, decided that he would do an experiment on himself, to work out how different foods affected scurvy. So he gave himself scurvy after a couple of months on bread and water and then started supplementing his diet with olive oil, milk, goose and beef. He was going to come to a fresh fruit and vegetables but he had to make his way through bacon and cheese first and unfortunately perished in the attempt. Stark’s fatal self-experiment is a reminder that good evidence does not just need to be produced, it also needs to properly disseminated.

CHAPMAN: You're an advocate of randomised trials. For those of us who, you know, might not know what they are, do you want to tell us a little bit about how they work, what actually they are?

LEIGH: One easy way to think about this is to imagine we want to know whether getting an extra hour of sleep makes you happier. We might take 100 people, toss a coin - 50 people get heads, 50 people get tails. We ask the heads group to sleep an extra night and then if they reported being happier, we'd conclude that a little more snooze helps you lose the blues. That's a principle which is applied to new pharmaceuticals. You can't get a new drug approved in most advanced countries without putting it through this kind of randomised trial. And the beauty of tossing a coin is you get two groups which are otherwise comparable. So if you observe differences, then you know it's got to be the intervention rather than something else going on in the world.

CHAPMAN: And so you say that when it comes to, for example you’re talking about medicine there, but when it comes to deciding randomised trials say for social policy, the vast majority of programs designed to help the most vulnerable are grounded in beliefs that are not empirical evidence. So, that instead of rigorous evaluation, we ask the HiPPO. What is the HiPPO?

LEIGH: The HiPPO is the highest paid person's opinion. As one corporate executive said "If we have data, let's go with that. If it's opinion we're going for, we may as well use mine!" Some of the best randomistas are those who don't lose their passion for the problem, but are scientific and critical in finding out what works.

CHAPMAN: Do you really think that's what happens, the highest paid opinion counts? That the big reckon in a company or organisational institution has weight over evidence?

LEIGH: Absolutely. People who have watched the movie MoneyBall will see a great example in which data can overwhelm greybeard wisdom on the sports field. Better use of data is one of the great revolutions of our age. Leading businesses are getting on top of this - Netflix is refining its algorithm for which movie to show you next based on randomised trials. Google picked the colour of its search bar based on a randomised trial, in which they tried 40 different shades of blue in  order to find the favourite one and that added millions of dollars to their bottom line. Coles supermarket in Australia has a loyalty program called FlyBuys - one in 100 cards are a control group who don't get the marketing and that allows the Coles board to assess the efficacy of their marketing programs. So major companies are doing it and the challenge is for governments to do it just as well.

CHAPMAN: You talk a lot about in the book and the speeches you've given about randomised trials, the sort of trialling the way we approach crime, and you highlight restorative justice experiments as being an example of an evidence-based approach to crime that has worked. You mention New Zealand in your book, among other countries. What has restorative justice told us about the way we tackle crime insofar as using randomised trials?

LEIGH: Restorative justice is the idea that if we bring the offender and the perpetrator together to discuss what the perpetrator should do to repair the harm they've caused, then we not only do right by the victim, but we can also reduce the chance that the perpetrator reoffends. There has been a series of randomised trials to this approach which seem to suggest that you get a significant crime reduction. One study in London suggested that the benefits of crime reduction were worth to the community 14 times the cost of running the restorative justice process. But you also get victims who feel less inclined to take revenge. This is something that Native Americans, Indigenous Australians, Māori in New Zealand have recognised for a long time. If you want victim and offender to go back to living together, restorative justice can work. Not for every case, but for many instances.

CHAPMAN: This is a big issue in New Zealand, now Dr. We had last week on the show Serena Gluckman talking about him spearheading this new report. Really saying very similar things to what you're saying, that we need more data and less dog when it comes to our extraordinary incarceration rate and how we approach our prison rates. You agree with that?

LEIGH: Absolutely. Both New Zealand and Australia have one in 500 adults behind bars as we speak. In the United States it's bigger still. There's more African Americans under criminal supervision now than there were slaves in 1850. So there's a huge need to get more evidence into criminal justice policy. What's striking about Australia and New Zealand over the last couple of decades is that most categories of crime have fallen and yet incarceration has risen. So thinking about what are the good evidence-based policies, not just that feel good in our gut, but actually help reduce crime and incarceration - it's absolutely critical.

CHAPMAN: The issue here - that does fly in the face of our gut perhaps, Andrew – is that being tough on crime works, that they need people who committed a crime that need to be punished and punished for a decent time, put in prison. To that end, tell us more about this other experiment, the Scared Straight program.

LEIGH: Scared Straight is an attempt at reducing juvenile delinquency. The idea is that if you put troubled young men in prison for a day they'll be scared onto the straight and narrow. It emerged actually from a 1978 documentary, which proposed such an idea and then - in a kind of ‘life imitates art’ sense - many American states took it up. Low quality evaluations suggested that it cut crime. But once the randomistas got in and we got high quality evidence, it turned out Scared Straight was actually increasing crime, perhaps because young men realised that jail actually wasn't as bad as they had thought. So if you're interested in keeping the streets safe, then Scared Straight is the opposite way to do it. I suppose to people who say that criminal justice policy should be based on the gut, I'd say that I'd rather my kids were safe, I’d rather we brought down the incarceration rate and the crime rate rather than put in place policies that simply are shown by the evidence not to work.

CHAPMAN: Right, the Scared Straight program, quite a famous program - by the time the randomised trials came along, it was pretty entrenched. Very entrenched wasn't it? So, this would have been quite a surprise to some people?

LEIGH: Absolutely. You know, there was even a documentary about a decade ago called Beyond Scared Straight, which yet again tried to perpetrate the myth that Scared Straight worked and provoked quite a backlash from the experts who had known for decades that this was a program that wasn't just a failure, it was actually ineffective. It was actually increasing the odds that these kids would go to jail. It’s probably no surprise to people who spend a little time in jail that being around jail changes your friendship group. Jail can be a ‘crime university’. So you want to think quite carefully about the use of incarceration in order to reduce offending rates.

CHAPMAN: I'm speaking with Dr Andrew Leigh, all about using evidence to guide policy making in all areas, His new book is called Randomistas: How Radical Researchers Changed Our World. He's here for some speeches in New Zealand. And in the late 90s, this is quite interesting, you talk about a really radical solution to the heroin epidemic in Australia. This is mooted. New South Wales Premier Bob Carr said that he would trial a very controversial idea. Tell us about it.

LEIGH: Bob Carr had lost his brother to a heroin overdose. Australia was suffering a huge heroin epidemic in the late 1990s - about 150,000 people were shooting up regularly - and Carr was interested in whether some approach apart from traditional criminal justice sentencing might be effective. His government put in place after a drug summit the idea of a Drug Court, in which drug offenders would be put through a supervised rehab program with jail as a sanction if they didn't comply. It turns out that Drug Courts work even if you place no value on the wellbeing of drug offenders. If all you care about is the impact of crime on the community, you should support Drug Courts because they appear to reduce offending rates significantly. They pass a cost-benefit study. It was interesting, there was a retrospective on the Drug Court a while back, where the New South Wales Director of Public Prosecutions Nicholas Cowdery admitted that the evidence had turned him around. He'd been a sceptic and was now a strong supporter of Drug Courts. Again, evidence-based policy in action, reducing crime and reducing incarceration.

CHAPMAN: Oh, what an interesting idea. Drug courts. To address the critics, the state government would have had to have had some pretty damn strong evidence of their effectiveness. Otherwise it would have been politically very damaging, I could imagine.

LEIGH: Absolutely. Their evidence was the following: if you look at 100 offenders who are released, put them through a traditional criminal justice process, you get 62 drug offences for every 100 offenders. Put them through the Drug Court, you get eight offences for every 100 offenders. So you get a massive drop in offending once you deal with the fact that these are people who have an addiction problem and locking them up in the cell doesn't necessarily solve that addiction problem.

CHAPMAN: So it works?

LEIGH: Yeah. It's been replicated now in a range of other jurisdictions in Australia and it has a lot to teach other countries. We're also learning from some of the experiments in criminal justice that are being done in places like Hawaii, looking at the impact of improving ‘certainty’ around parole violations. Sanctions for breaching parole can seem ad hoc and capricious. Randomised trials suggest that if you can make the system more certain, then that has a bigger impact on crime reduction than if you just focus on the length of the sentence.

CHAPMAN: Now, most people who are listening this morning will be thinking about  randomised trials, they mind might immediately go to the medical field, won't they Andrew? In that sense, you talk about something quite amazing. You talk about placebo surgery, also known as sham surgery. This is used when researchers are actually uncertain whether or not an operation helps patients. Explain that a bit for us.

LEIGH: Sham surgery was shocking to me when I first heard about it. The idea that you would actually cut somebody open, not perform an operation and sew them back up again. But the reason that it was done was because we didn't have good evidence that many of our surgeries worked. When doctors began doing these sham surgeries, they discovered that in many cases the outcomes were no better from real surgery than from sham surgery.

CHAPMAN: Really?

LEIGH: So for example, a classic study in recent years is knee surgery performed for a torn meniscus, so called meniscectomy. It's been performed millions of times around the world every year, but the outcomes for middle-aged patients of meniscectomies don't seem any better than sham surgery. And one of the challenges here is the placebo effect in surgery is really big. So if I give you a tablet, a sugar tablet, then the chances you'll feel a little bit better. If I give you a salt water injection, then it will produce an even bigger placebo effect. But if I have an expert in a white coat slice you open, then the placebo effect of that is bigger than the injection and certainly bigger than the sugar tablet. So you need the randomised trial to actually separate out the placebo effect and to see the true effect of the surgery.

CHAPMAN: So let me get this straight in terms of methodology - you actually have surgeons bring in the patient, they'd go under, they would literally cut a person open, sew them back together again and do nothing.

LEIGH: That's right. And patients of course have to consent to this beforehand.

CHAPMAN: Controversial.

LEIGH: It is, but you've got informed consent and you've got ethics boards overseeing it. And ultimately the ethics boards regard this as being appropriate to do because being cut open and sewn back up again has far less infection risk, for example. So if you're really not sure if the surgery works, then you may well be better off be in the control group rather than the treatment group. We're learning a lot from these sham surgery randomised trials and it is suggesting in many cases we may want to look at non-surgical interventions. That's good for the taxpayer because we're paying fewer surgeries and it's also good for the patients who are then able to be moved into physical rehabilitation therapy or some other strategy which can be effective and of course much cheaper for the community.

CHAPMAN: Much of how public see the information now, Dr Leigh, has through opinion columns, op ed pieces, thought pieces, friends' opinions of Facebook and less through ‘this is what the evidence says’ or in the area of crime they call it 'penal populism'. Is this a big problem for the likes of yourself, who want to sheet home the idea that we need proof before we commit to particular policy? We need proof or evidence, before we need to talk about inequality or certain social outcomes?

LEIGH: I certainly think that there are some people who just want to shoot from the gut. There's that Stephen Colbert roast of George W. Bush, saying that Bush goes straight from the gut because there's more nerves in the gut than the brain. ‘How do I know that? Not because my brain  tells me, but because my gut tells me.’ But if you look at the successful firms, they're the ones that recognise that they need to pull together evidence. And it shouldn't be that surprising, given that nine out of ten drugs that look promising in the lab don't produce results through clinical trials, one out of five Google experiments work. In the education field, the American What Works Clearing house finds that only one out of ten education interventions works. Once you start to realise we're an uncertain world, I think you're much more inclined to get data, and certainly my host  - Presbyterian Support Northern - I've been impressed by how much they, as a charitable foundation, are interested in solutions and are scientific and critical about their approaches to get there. So I think there are a lot of good people both in government and in the communities sector in New Zealand thinking this way.

CHAPMAN: Dr Andrew Leigh, speaking in New Zealand. His new book is Randomistas: How Radical Researchers Changed Our World. Andrew, good to have you on the show.

LEIGH: Thanks so much Wallace, great to chat.

ENDS

Authorised by Noah Carroll ALP Canberra