Evidence-based politics: would such a thing ever be possible?

[el-ahrairah]

All my policies are evidence based. Otherwise how would I know they were good policies or not?

 

[teuchter]

All my policies are evidence based. Otherwise how would I know they were good policies or not?

What are your policies?

 

[paolo]

I think the analogy with science falls down on the idea that you can test something in isolation, to determine its effect. National policies such as taxation vs. public spending, immigration, etc would be impossible to trial in a limited form. Maybe it's more possible for local policies - and I think that already happens in some cases (City X studies City Y's implementation of Z).

 

[teuchter]

I think the analogy with science falls down on the idea that you can test something in isolation, to determine its effect.

Not necessarily. Drugs aren't tested in isolation - they are tested in a bunch of bodies each of which is built a bit differently and in each of which the drug will interact with other substances in different ways. A properly designed trial will be able to draw generalised conclusions given a sufficient sample size.

 

[ViolentPanda]

The question is, whether we could ever have something equivalent to, say, the laws of thermodynamics, where we can pretty much say: these things are fixed and unarguable. And whether we could have a comprehensive enough set of those rules that the majority of political decisions could be made with a high degree of certainty over the outcome.

Bypassing the whole "history of sociology" issue and the various historical perspectives inherent to it, I don't see any possibility of an equivalent to the laws of thermodynamics that would provide a basis for "evidence-based" political decision-making. Politics are contingent; the people who "do" politics are contingent, the people politics are "done to" are contingent. There are far too many variables, from a shifting economy to the assassin's gun, to enable a "law of political decision-making" to be formulated.

 

[ViolentPanda]

But do you see these failures as being something more than simply an insufficient amount of evidence to cancel out the background noise?

They're failures because (Asimov's "psycho-history" aside) you can only ever get snapshots of a part of society, "society" itself being too complex to fit into any academic model.
That's why academic sociologists tend to specialise in quite closely-defined fields; it enables them to concentrate on that part of society that interests them, without getting caught up in a futile quest to seek out general "laws" of society.

 

[articul8]

Societies - unlike phenomenona in natural science - are governed by norms.

Something like "Justice" can't be meaningfully calculated or abstracted in the way say "temperature" can. You can have a room that is 21 degrees C but not a society that is 21% just.

That doesn't meant empirical testing of truth claims isn't important when it comes to social life. But that social relationships can't be reduced to empirical data.

 

[paolo]

Not necessarily. Drugs aren't tested in isolation - they are tested in a bunch of bodies each of which is built a bit differently and in each of which the drug will interact with other substances in different ways. A properly designed trial will be able to draw generalised conclusions given a sufficient sample size.

But the bodies don't (normally) interact with each other though. With society based stuff, at the national level, it's a like a weather system.

A better analogy between national policy and medecine is disease and infection. And the future behaviour of those are informed guesses, not clinical and accurate predictions.

 

[mattie]

Not necessarily. Drugs aren't tested in isolation - they are tested in a bunch of bodies each of which is built a bit differently and in each of which the drug will interact with other substances in different ways. A properly designed trial will be able to draw generalised conclusions given a sufficient sample size.

I'm going to reply to your question addressed to me in #29 as - although not entirely addressing your question - it's easier and I'm lazy.

The point is that a patient's reaction to a drug is entirely unrelated (except psychosomatically) to another patient's reaction to a drug. You actually need the variation in person to understand relationships between factors* as you can correlate across patients. Grossly simplifying (pardon pun), with all else equal fat patients may react one way, thin another.

With society, a given person's response is a function of both any stimulus you might care to test plus other people's response to that stimulus.

That's what I mean by independence, or lack thereof.


*this is in fact the essence of Taguchi's Robust Design, an engineering method - one of the Design of Experiments stable - which allows complex systems to be tested for inter-dependencies and sensitivities by multiple observations, each at different corners of the operating envelope.

 

[ViolentPanda]

You need to explain a bit more. Not the same as what, and in what way?

I can see an argument that the level of complexity is much greater, but that in itself is not reason to say that something can not be usefully studied in a scientific way.

The human body, or the weather, are both immensely complex but a scientific approach to the study of either provides useful knowledge.

Different type of complexity. A human body is temporally finite, weather is a closed system. Society though, it just keeps evolving and hybridising anarchically, and while positivism is an approach to studying the various phenomena, it's not as well equipped as it might be to deal with a free-flowing, fast-changing subject.

 

[paolo]

I think a few of us are now saying the same thing. So that means Teuchter loses.

 

[teuchter]

Bypassing the whole "history of sociology" issue and the various historical perspectives inherent to it, I don't see any possibility of an equivalent to the laws of thermodynamics that would provide a basis for "evidence-based" political decision-making. Politics are contingent; the people who "do" politics are contingent, the people politics are "done to" are contingent. There are far too many variables, from a shifting economy to the assassin's gun, to enable a "law of political decision-making" to be formulated.

You could say something pretty similar about weather forecasting though. And the laws of thermodynamics are at least useful to weather forecasters, even if they aren't able to achieve 100% accuracy.

Would you accept that our ability to predict the effects of political decision making ought at least to be able to improve, the longer we record history?

 

[ViolentPanda]

I think a few of us are now saying the same thing. So that means Teuchter loses.

Ah, but if, as kyser claims, teuchter is a determinist, then does that mean he knew he'd "lose"?

 

[paolo]

Ah, but if, as kyser claims, teuchter is a determinist, then does that mean he knew he'd "lose"?

Maybe he's running this debate on hundreds of seperate forums, and in many others he's "won".

 

[mattie]

I can see entirely where he's coming from though - ultimately, there is a level at which we are deterministic.

 

[teuchter]

Human society on a global scale is as much a closed system as the weather (or an individual human body) is.

It's just much more complex.

 

[teuchter]

Ah, but if, as kyser claims, teuchter is a determinist, then does that mean he knew he'd "lose"?

It means that someone somewhere would know whether I'd lose, if they had access to all information about the universe.

Anyway, as the thread title and OP are a question rather than a statement, I can't lose.

 

[ymu]

Evidence-based medicine is possible because thousands, sometimes hundreds of thousands, of people can be recruited into trials. Their similarities and differences affect the overall result, but with a good experimental design it is possible to detect treatment effects above the "noise" caused by the differences between the individuals. It is never possible to predict what will happen to one individual person, only to gain evidence about which treatment is the best bet on average - based on the average gain and the average cost (in terms of side effects).

In politics, there aren't enough different societies to observe, you will almost never be able to use an experimental design, and you need to make the right decision for the particular society you are making policy for, not the best bet policy for vast numbers of different societies.

There are certainly elements of policy that can be informed by evidence, but it's never going to be that clear cut. Evidence-based medicine isn't all that clear cut a lot of the time, for reasons which would also affect evidence-based policy-making. Prophylactic anti-coagulation was useful when a large proportion of in-patients developed DVTs; now that it is much rarer due to other improvements in care, the risk of bleeding out due to treatment is greater than the benefit of clot-busting for many previously high risk groups. Treatments used in Africa in pregnancy are largely irrelevant in Europe because the population is healthier and better nourished. Japan requires additional trials in Japanese people because there are genetic differences which massively alter the effects of some treatments.

You can't solve the sample size problem by gathering data for a very long time. Time itself changes the situation. Which is why we don't use historical controls in EBM any more.

 

[teuchter]

I guess what I'm suggesting is that after zillions of years of history, if someone were to shoot at an aristocrat in a small province of a large empire, with a fairly pleasant climate, [plus a bunch of other factors typed into a big computer] there would be enough instances of similar situations in the past that the computer could think about it and give a statement about various courses of action and what their consequences would be.

Just a big game of chess really.

 

[ymu]

But one of the key breakthroughs in evidence-based medicine was recognising that historical controls are deeply inadequate.

 

[teuchter]

You can't solve the sample size problem by gathering data for a very long time. Time itself changes the situation.

Of course - but as long as you bear this in mind it needn't make things impossible to study.

Perhaps weather forecasting is a better analogy than EBM, though.

 

[ymu]

Well, no. You can't analyse out the problem of historical controls, something that was recently demonstrated by a very neat piece of research.

They took two huge international randomised controlled trials and used them to simulate concurrent non-randomised controls (by comparing control patients from one centre against test patients from another) and to simulate historical controls (by comparing control patients recruited during the early years of the study against test patients recruited in the later years).

This provided a very neat demonstration of the bias inherent in these non-randomised designs. But they went on to look at methods for adjusting the data to eliminate these biases - and found that the bias tended to get worse rather than better.

Reference: http://www.ncchta.org/execsumm/summ727.htm

 

[teuchter]

Well, no. You can't analyse out the problem of historical controls, something that was recently demonstrated by a very neat piece of research.

They took two huge international randomised controlled trials and used them to simulate concurrent non-randomised controls (by comparing control patients from one centre against test patients from another) and to simulate historical controls (by comparing control patients recruited during the early years of the study against test patients recruited in the later years).

This provided a very neat demonstration of the bias inherent in these non-randomised designs. But they went on to look at methods for adjusting the data to eliminate these biases - and found that the bias tended to get worse rather than better.

Reference: http://www.ncchta.org/execsumm/summ727.htm

That's interesting stuff. But doesn't it just demonstrate that their adjustment methods weren't very good, rather than that the adjustment is fundamentally impossible?

 

[ymu]

It demonstrates that not only can we not adjust for the factors we don't know about (which we already knew), but that not knowing about them makes any adjustment pointless and likely to make matters worse (which we didn't know).

They do look at one method called the propensity score which is more promising, but it's still based on known factors. It's the unknown factors that cause the problems - and we'll never know what they are, except in some imaginary deterministic universe where all things are known. But then we wouldn't need evidence, because we'd already know the truth.

So no, I don't think this is a problem with a solution.

 

[ViolentPanda]

You could say something pretty similar about weather forecasting though. And the laws of thermodynamics are at least useful to weather forecasters, even if they aren't able to achieve 100% accuracy.

It's not about predictive ability.
Weather, however complex, exists within a closed system of finite and reasonably quantifiable variables, the same set of variables (more or less) having existed as long as whether has.
Society isn't and never has been, even in "closed" cultures, a closed system. It is "chaotic" in the sense that actions taken can be profoundly unpredictable in their effect, in a way that just doesn't happen with weather, and it's that unpredictability, that possibility of anarchy that haunts every step, that makes sociology so interesting: There's always new knowledge to be sought out, because novel occurrences are forever springing from supposedly "everyday" interactions.

Would you accept that our ability to predict the effects of political decision making ought at least to be able to improve, the longer we record history?

I wouldn't accept that at all. I'd say that at best we could derive some possible indications of whether a particular decision might cause certain results, but no more than that.

 

[teuchter]

(@ymu)

So what you're kind of saying is that, in the instance of looking at human social history, we can record it as much as we like, but there may be certain factors that influence things significantly, but we haven't noticed them, and therefore won't know to record them in the first place?

 

[ymu]

(@ymu)

So what you're kind of saying is that, in the instance of looking at human social history, we can record it as much as we like, but there may be certain factors that influence things significantly, but we haven't noticed them, and therefore won't know to record them in the first place?

Yes.

 

[teuchter]

It's not about predictive ability.
Weather, however complex, exists within a closed system of finite and reasonably quantifiable variables, the same set of variables (more or less) having existed as long as whether has.

But I'd argue that so does human society (on a global scale). Effectively just a bunch of chemicals reacting with each other in the end.

However, I can see that those reactions are possibly getting more and more complex over time, and that there are plenty of reactions that haven't happened yet but could in the future.

 

[ViolentPanda]

Human society on a global scale is as much a closed system as the weather (or an individual human body) is.

It's just much more complex.

Nope. Closed systems are by and large static, non-evolutionary. They obey the "Laws" you're so hot on: Finite calculable outcomes from a finite number of interactions.
Society isn't/societies aren't static. Their very nature (interaction of individuals in an ever-changing pattern of infinite possibilities) means that they can't achieve stasis.

Weather's complexity is amenable to measurement. Societal complexity isn't, via any system of quantification invented in the history of man.

 

[ViolentPanda]

But I'd argue that so does human society (on a global scale). Effectively just a bunch of chemicals reacting with each other in the end.

And so we get to reductio ad absurdum, because interacting as social beings allows us to be more than the sum of our chemical constituents and their interactions.