Andrew Leigh

The Randomistas and insights of Randomised Controlled Trials

Radio Interview on ABC RN Breakfast

Tuesday, 6 March 2018

FRAN KELLY: Andrew Leigh is the Shadow Assistant Treasurer and he’s also a former professor of economics at the ANU. His new book is called ‘Randomistas: How radical researchers  changed our world’. Andrew Leigh, welcome back to RN Breakfast.

ANDREW LEIGH, SHADOW ASSISTANT TREASURER: Thanks, Fran. Great to be with you.

KELLY: So what’s a Randomista?

LEIGH: A randomista is someone who conducts randomised trials - in other words, toss a coin and randomly allocate people to the treatment group or the control group. That’s how we test our new pharmaceuticals. If you want to get a drug on the PBS, you basically have to put it through a randomised trial. But increasingly they’re being applied in a host of different areas. So if you’re using the Google home page, you’re probably participating in randomised experiments. The particular shade of blue Google uses for its search bar was chosen via a randomised trial. Amazon is carrying out tons of randomised experiments. One researcher said that ‘every pixel on the Amazon home page has had to justify its existence through a randomised trial’.

KELLY: So just before we go any further, let’s narrow down what you mean by a randomised trial. What is it?

LEIGH: Let’s take a very simply example. You want to know whether getting more sleep makes you happier. You might take 200 people, toss a coin - 100 people will get heads, 100 will get tails. Then you ask the heads people to sleep for an extra hour a night and ask everyone to rate their happiness. If you found that people whose coin came up heads were happier, then you might conclude that a little more snooze helps you lose the blues. 

KELLY: Ok. So in the past and traditionally, randomised trials were a technique used by science and medicine in particular. 

LEIGH: Absolutely. So we look at the cure for scurvy, which was found by a naval scientist called James Lind, who tested a range of remedies of the day, including vinegar, but also the one that we know works - citrus. Having assigned these across patients, he found that those who received citrus did better. Not to say it was immediately adopted, but within decades, the British Navy had taken on citrus and when the British and French navies came together at the Battle of Trafalgar, Britain won partly because they’d managed to find the cure for scurvy through a randomised trial.

KELLY: Having lemon juice on board was all it took?

LEIGH: Absolutely.

KELLY: So we’re talking some hundreds of years then?

LEIGH: I think it’s about 60 years from Lind’s trial through to the Battle of Trafalgar, but these things don’t get adopted immediately. So the randomised trial that showed that bloodletting didn’t help patients came a little too late for medicine not to name one of its best journals The Lancet. If you look at other trials, they’ve been very important in terms of showing that things don’t work. Scared Straight, a popular program in America, suggested that if you expose delinquent teens to a life behind bars, that they would be scared onto the straight and narrow. But in fact it turned out to increase crime among children that were exposed to it. There’s been randomised trials that underpin the rollout of the polio vaccine, diabetes, the contraceptive pill. But of the drugs that come out of the laboratory these days, only about one in ten of those that look promising actually make it through randomised trials.

KELLY: And randomised trials back when they were first thought of within the scientific world and for many years, I’m interested that you’ve written this book about them, were they controversial? Were there doubters of the theory that you needed to put in that work? I know there are questions we can get to about the ethics of them, but in terms of the scientific ethicacy and power of them?

LEIGH: Absolutely. Before there was evidence-based medicine, there was eminence-based medicine. Doctors were outraged when the randomistas told them that they were infecting their patients by failing to wash their hands, They felt very strongly that someone of their status couldn’t possibly be doing harm. The randomised trials that have been rolling out recently in the area of surgery have been challenging many of the core beliefs of that discipline, suggesting that operations such as a menisectomy for middle aged patients don’t produce better results than a placebo surgery, in which the patient is cut open, not operated on and then sewn back up again.

KELLY: So the point of them is that they minimise biases, as you just said, but there are ethical questions. There have always been ethical questions about them, about the notion that there’s at least one group within this randomised trial that is not getting what many suggest or suspect would be the best help. Is that an ongoing debate?

LEIGH: It’s certainly something that every randomised trial needs to confront. Let me give you one of the most confronting for me, Fran, and that’s the head injury retrieval trial - a randomised trial of the 000 system in Sydney, which for a number of years randomised patients to either the traditional approach of an ambulance and paramedics or a novel approach of the NRMA Careflight helicopter and a neurosurgeon. The second is a much more expensive approach and the randomised trial was looking to see whether the outcomes were better. Now, the patients in that study not only didn’t consent, but in most cases were unconscious. But I believe it’s entirely ethical to carry out a randomised trial in that context, because these are significant resources we’re deploying-

KELLY: What was the outcome of that?

LEIGH: Extraordinarily enough, it was inconclusive because the study didn’t get enough people through it. But the principle behind conducting the study is absolutely sound and I say this as somebody whose brother had a serious head injury during the period, who might well have been part of the randomised trial. Thankfully, my brother made a full recovery but I think it was ethical to carry out that randomised trial to work out whether the resources were best deployed in that part of the health care system.

KELLY: You’re listening to RN Breakfast. Our guest is Andrew Leigh, Shadow Assistant Treasurer, former professor of economics and his new book is called ‘Randomistas’. One of the contemporary pioneers of randomisation is the US social policy researcher Judith Gueron. In the 60s, she said researchers knew in theory the power of random assignment, they just didn’t believe it would be useful to evaluate real world social programs and address important policy questions. I guess that’s proven to be wrong, but from your research, what are some of the key social policy issues that randomised trials have dealt with and had breakthroughs with?

LEIGH: Dr Gueron is a fascinating example, because she is coming into doing randomisation at a time when the American debate over social policy is at its hottest. When you’ve got Ronald Reagan talking about ‘Cadillac driving welfare queens’ and some on the left saying just what you needed was a massive expansion of the existing programs. Judith Gueron was able to show that in many cases these programs were producing outcomes that were a little better than the total cost of the program but were far from being magic bullets that some of their advocates had suggested. She was up against some pretty tough opposition. In one study in Miami, it was accused of treating people like guinea pigs, of conducting research that was akin to pulling the legs off spiders. But by working through and being absolutely clear that the other methods relied on a lot of statistical fancy footwork - and in some cases produced conflicting results - has been able to steadily roll out randomised trials that have built an evidence base in the United States, around for example the earned income tax credit, a massive anti-poverty program that subsidises the wages of low wage workers.

KELLY: We have trials here on things like lately income management, cashless welfare. We trial them. Is that some kind of randomised trial or is that different? Should we have had randomised trials on these?

LEIGH: I believe we should have. Ad hoc is different from random. A pilot is a study without a control group. The reason that medicos have turned so much to random trials, Fran, is because they’ve recognised it’s important to know what would have happened if the patient hadn’t have gotten the treatment. Most people who go to the doctor would have gotten better next week regardless of the doctor’s visit. So simply to look at how sick you are when you go to the doctor and how sick you are next week doesn’t tell you the causal effect of going to the doctor. We get some really surprising results out of it and also some you can use in real life. When I have to take a bandaid off one of my three sons, I always say we should take it off fast rather than slow because of the James Cook University study which found that the fast method was less painful than the slow method.

KELLY: [Laughter] So you can bring it into real life. But there are some startling results. One example in your book was a randomised trial of over 400 villages in India,  Indonesia, Mali and Tanzania for sanitation education to reduce open defecation and the outcome, the result was incredible. The related villages had taller children because they weren’t suffering the effect of faeces-borne diseases. 

LEIGH: Absolutely. And in developing countries, this is really rolling out on a large scale. A recent randomised trial of biometrics smart cards in Andhra Pradesh in India had a sample size of 19 million people. There have been studies of the doubling of teacher pay in Indonesia and a host of studies that are trying to delve into the root causes of corruption. There have been studies into what happens if you provide cash bonuses to entrepreneurs and really promising results out of Nigeria.  So the Randomistas are hard at work dealing with some of the toughest social disadvantage in developing countries. The World Bank is a major funder of this and really moving again away  from ideology to building up that stock of evidence and making a little more modest approach.

KELLY: Just finally, data collection is the new holy grail here. We have great capacity for it now with our search engines, our social media sites. We know that Google for instance has more data than it could possibly know what to do with, though I’m sure it’s working that out. So, all this kind of randomised trialling is going on with the data that we’re giving out every day. Netflix, we know for instance,  the recommendations that it gives to us is based upon the data it’s collecting. I’m sure they’re doing some kind of randomised thing. It was Facebook, an experiment that Facebook did in recent times where their people were inadvertently part of a randomised trial that was really quite unnerving and quite controversial. Tell us about that.

LEIGH: Facebook decided to randomly change the emotional content of its newsfeed, asking the question if you see more happy posts,  are you more likely to post happy material? If you see your friends posting more sad material, do you go sad yourself? From a physiological standpoint, it produced the interesting result that you follow the emotions of your friends. But it didn’t ask people to consent to it and there was appropriately enough criticism of Facebook for the emotional manipulation that they engaged in. 

KELLY: Because people every day didn’t know that they were part of the experiment and having their emotions toyed with?

LEIGH: Precisely and in that case, I think Facebook should have very clearly asked people to opt into a study which potentially could have caused harm. I’d put that in a different category from a different study Facebook did trying to encourage American voters to turn out to vote in a mid-term election. I think there it’s harder to argue that people were harmed by being encouraged to vote. But if you’re doing something with potential harm, then you should ask users to opt in. Perhaps reward them in some way.

KELLY: Andrew Leigh, thank you very much.

LEIGH: Thank you, Fran.