I left off before the holidays writing about how we believe in science. People have written on this blog that anecdotal evidence is never reliable while putting scientific research on a kind of pedestal. I say both have their place. Moreover, one good thing about analyzing your own experience is that you train your mind to ask questions, and about everything, not just those things that you don’t like. I think that blind faith in scientific research is a bigger problem that most woo-woo beliefs. We aren’t actually making the observations or doing the experiments, and usually not even reading the studies, but rather trusting in what some scientists say, as reported in the media, without a critical enough eye. I find that the media likes to quote scientists saying things that aren’t very scientific and aren’t supported by the data.
For example, today there was a news item about a study in California that found counter-evidence for the theory that there is a link between autism and the vaccine preservative thimerosal. The study claims to refute this link because autism rates have continued to rise despite thimerosal being removed from some childhood vaccines. I don’t think I have a problem with this, although it is not the case that thimerosal has been removed—it’s still in flu shots and those are recommended for infants. Nevertheless, it seems like there is less of it, and if thimerosal were the major cause of autism, it is reasonable to expect that rates would go down if the exposure was reduced. (For the record, I’m not one who believes that thimerosal is the main cause of autism. But I don’t know, it’s just my belief...)
My problem is with the reporting and the conclusions of quoted experts. A large number of the articles I saw on Google News had a headline that said something about a vaccine-autism link being refuted. Well, the study didn’t look at the link between vaccination and autism, just at one particular preservative and autism. There is nothing in this study to refute other research that has found a link between vaccination and autism, such as this survey. (I’m not saying such a link has been proven, only that there are significant unanswered and unresearched questions.) I don’t find this scientific error unusual. Media articles and the scientists they quote are often guilty of overstating the conclusions of the studies that they report.
Not all media sources made this mistake. More than half correctly reported that the study evaluated a vaccine preservative and autism rates in their headlines. But most contain other shaky science. The biggest one is pretty ubiquitous: genetics. For example, US News and World Report says, “Scientists, on the other hand, think autism is largely genetic.” However, genetics cannot account for the increase in autism in the last half century. Just do the math. I’m not saying that there isn’t a genetic factor involved. There usually is. But it implies that we don’t need to question or change our environment, just wait for genetics researchers to find the answer.
More disturbing to me are the moral attacks slipped in as if they had scientific basis. For example, the article on ABC News’ website quotes a Dr. Peter Hotez as saying:
Unfortunately, this whole dialogue around vaccines and autism has been a distraction," Hotez says. "It's focused around an erroneous association between vaccines and autism, and it's sucked away resources from what we really need, which is more and better research into causes of autism and support for families.
First you have the fallacy that the association between vaccines and autism is erroneous because some scientist says so. This is false: the current data neither proves nor disproves such a link. A good scientist should know the difference between proof and evidence. But then we have the scientist claiming that scientific research that doesn’t agree with his point of view is harmful. Why is it harmful? Well, because he already knows the truth and anything else is therefore a waste of time and resources. I thought that we were supposed to use science to answer these tough questions. Instead, politically inconvenient science is discouraged and criticized on a kind of moral ground. When I read what is essentially an ad hominem fallacy telling me that some research is a waste of time instead of a good scientific argument, it makes me want to see such research even more. If a person wants to make a political or moral argument, that’s fine. But it is being presented as science in mainstream media. I think it promotes a kind of blind faith in science every bit as problematic as things that are criticized as being non-scientific beliefs.
Comments:
John - I think that most people will concur that the press is frequently inaccurate in its reporting of science and medicine, and it has a tendency to reduce disputes about evidence and methods into personal spats.
In addition, you state that:
I suggest that there has been a huge environmental change over the past century. We now have a much greater awareness of Autism and children with Autism are being systematically identified in schools and provided with assistance. One hundred years ago this was not the case. One reason for the increase in the numbers is that we are much better at identifying Autistic children.
Form my own personal experience, and that of close relatives who work with people with Autism, even as recently as the 1980s children with behaviour that resembles autism (I can’t say they were Autistic because they wern’t diagnosed) were ridiculed and bullied at school and didn’t receive any special attention from the teachers. Now they would be much more likely to be identified, diagniosed, and receive additional support.
In addition, we have a much more recent change in definitions. We now talk about ‘Autism Spectrum Disorder’. This definition now includes a much wider range of people (both Autictics and people with Asperger’s Syndrome etc). Changing definitions will clearly present an appearance of increasing numbers.
Any discussion on any rise in numbers of people diagnosed with Autism needs to take these two factors into account.
The other thing is that media reports are notoriously bad at conveying scientific information, if you want to find out what a study really said you must read it. Unfortunately most scientific literature is kept hidden behind a pay wall where the general public can’t read it even if they have a mind to. Still no scientist is asking you to believe them just because they are scientists!
Actually some of the questions you ask are good scientific criticisms. You make one big mistake though, which is a good example of the flaw in a lot of homeopaths thinking. You look at the rise in autism over the last century and you think OK perhaps some of that is due to increased reporting rates etc, but its such a big number that some of it must be due to something else. This is the same as when you look at apparent improvement in your patients and think OK some of this may be due to placebo effect but it’s just to big an improvement to be due to placebo alone. The thing is, you just do not know how big each of these effects can be, so these judgments are flawed.
A second mistake is to jump from noticing an increase in autism, if real, to identifying vaccines or more particularly thermisil as the culprit. I’m not saying it wasn’t an interesting idea, but those that thought it was the problem have been unable to produce compelling evidence. Perhaps it is something else, it could be anything that correlates to passing time in the 20th century:
Exposure to car fumes
Exposure to EMF whilst in the womb
Watching tellivision when small
Increasing wages of footballers
Global temperature increase
Flying during pregnancy
Etc etc etc
The hypothesise are endless, as they always are, being limited only by the human imagination. The trick of science to pick the best one out of the bunch.
PART 1
John
I have checked out the link you posted to “research that has found a link between vaccination and autism”. And I’m looking into it now to respond to your comments above, and in previous posts, about people ignoring inconvenient evidence (just because it doesn’t fit into their world view).
After looking at it I can say that it really doesn’t convince me of anything. Here’s why.
First, I looked at the source data .pdf and age disaggregated tables are unreadable. Not good. Because of that I’m going to have to just refer to the aggregated tables (the first three).
Second, the page uses some of the irresponsible tricks you rightly condemn in the media. It states that “Vaccinated boys were 61% more likely to have autism” That sounds bad.
But then if we look at the figures we get the following:
All Children
Unvaccinated 2%
Partially vaccinated 4%
Fully vaccinated 2%
Fully and partially vaccinated 2%
Girls
Unvaccinated 1%
Partially vaccinated 2%
Fully vaccinated 1%
Fully and partially vaccinated 1%
Boys
Unvaccinated 2%
Partially vaccinated 7%
Fully vaccinated 3%
Fully and partially vaccinated 4%
As the web page mentions, there no meaningful difference in girls. And as the web page does not mention, there is also no meaningful difference in all children.
When we look at boys, the difference between unvaccinated and fully vaccinated is 2% compared to 3%. As with girls, that’s not a meaningful difference. (More below).
In addition, we find a difference of 7% for partially vaccinated. Now that’s odd to start with because one would assume that if vaccines in some way cause autism then any increase in prevalence of autism would be in the fully vaccinated group.
PART 2
But much more important is the question - is the difference between vaccinated and unvaccinated boys meaningful? And here we have a big problem. Any paper published in a scientific journal will state the statistical significance of any result. By significance I mean the probability that any apparent result is merely due to random variations in the data. This web page and the accompanying methodology notes do not do that.
Now, that’s a big red flag. If the difference isn’t statistically significant then the survey doesn’t imply any link at all between vaccines and autism. I don’t have the wherewithal here to do any of the maths, but by the look of it any correlation at all is going to be pretty slight.
I can illustrate the problem with this analogy. We have lots of elections in the USA right now, and lots of pre-election surveys. Lets say that one survey gives Obama 33%, Edwards 28% and Clinton 30%. Lots of statistically illiterate journalists will say that Obama now has a clear lead. Actually, lots of surveys have a margin of error of +/- 5%. All three are within that margin. So all we can truthfully say is that the survey doesn’t report any significant difference in support for any of the candidates. The same may well apply to this survey on vaccines and autism.
On top of that I have a couple of other gripes. First, the 61% increase looks like it should be 66%. But perhaps they are using the base numbers rather than the percentages which have been rounded, so that may not be important. Second, the web page reports higher figures for “All vaccinated boys, removing one county with unusual results (Multnomah, OR), compared to unvaccinated boys”. There may be a reason to remove that county (perhaps the survey team messed up), but any article in an academic journal should explain why results need to be excluded. That really should be explained.
So, the reason why I accused the site of being irresponsible is that their 61% increase is actually a marginal difference that is only apparent in boys. Moreover, as far as I can tell, the ‘61% increase’ refers to the 2% of unvaccinated children with autism compared to 3% of fully vaccinated children. (Any of the other figures would produce a very different percentage). That doesn’t look like anything other than random variation to me. Moreover, its not very different from the results for girls which the survey states were not meaningfully different.
I think that they are being irresponsible because they present the above as evidence for “a strong correlation” between autism and childhood vaccination (see the press release). That’s just dishonest, I can’t see how there could possibly be a strong correlation in the above data between autism and vaccination.
Anyway, its getting on for midnight here, gotta stop now. Feel free to point out any errors in the above - I am weary.
I won’t dispute your statistics. As I said above, this study doesn’t prove anything. It was too small to produce statistically meaningful results. I wouldn’t quote the numbers here or use this to say anything more than there is possible evidence of a link.
But I don’t feel this site is irresponsible, although I do agree with you that they should have stated their margins of error on that page. I’ll let the group that did the study speak to this issue:
and
The group that commissioned this study did so to prove a point. They did a small study with a small budget using the methodology normally used by the CDC. Why has the CDC not done the larger study? They even quote the CDC director as saying it should be done, but it hasn’t been.
This is not proof, but it is false to say there is no evidence linking autism and vaccination. Just as it is false to say that there is proof of a link between the two. The science which should be done on this issue has not been done, and so I cannot make a decision about this based on the science.
I do believe that vaccines are a co-factor in autism, but my experience is far too small to make any actual claims about it. I may be wrong. All I can say is that I personally know of numerous cases where there appears to be a link but the medical establishment will not accept it. I have looked at the conditions required by the US government in order to accept a child as vaccine damaged, and they are extremely narrow. They only accept it if it is proven without any doubt. I only know that there is significant evidence that the numbers are greater than the US government will acknowledge, but I can’t say how much more so. I’d like to see honest research done on the subject.
Yes, one of the problems in medicine these days is that doctors don’t make diagnoses based on the facts, but they must make certain diagnoses in order for the patient to get insurance coverage for certain treatments.
In addition to bundling in Asperger’s, doctors also mis-diagnose pervasive developmental disorder as autism. So when you want to consider actual autism, the numbers can be misleading. But it is also fair game to consider neurological disease in general, which occurs at extremely high rates. I certainly wouldn’t say that an Asperger’s or PDD diagnosis is not a condition to take very seriously, nor AHDH for that matter.
In any case, even if you reduce the numbers to only true autism, the numbers are both alarming and cannot be accounted for by genetics alone. Autism was rare before the 1950’s. The doctor who named autism in the 1940’s (I believe) went looking for cases who might have been diagnosed as something else and found only very few.
The argument that these neurological disorders have always been around but now we are diagnosing them doesn’t hold much water. Yes, there are too many kids diagnosed with AHDH and over-medicated, and there are Asperger’s cases the weren’t diagnosed or otherwise diagnosed, but there are too many of the more severe cases of these neurological disorders to be accounted for by over-diagnosis. If only 10% are accurate there is still an alarming rate.
I am not saying any of these are caused by vaccination. I think it’s a likely co-factor, especially with Autism Spectrum Disorders. But I suspect there are probably many factors, both environmental and genetic, and not all cases have the same cause. But that’s just my opinion. There are reasons to suspect vaccines, but there have been so many environmental changes in the past 70 years.
My personal experience and that of people I know who work with special needs kids would disagree with that comment, at least when we are talking about true autism. With Asperger’s I might agree with you, at least for some of milder diagnoses.
The discussion about autism aside, the point I wanted to make here is that I’m not going to disregard my own experience and just believe the science. For one, it frequently reverses itself, at least in the medical field. And two, what I read in the press is often misleading or incorrect. Almost all of the things that I read about homeopathy are just flat out wrong. How much of what I don’t know so much about is equally wrong? I don’t have the time nor expertise to read all the actual studies, although it would be nice if I could get at more of them without paying a bunch of money. Plus the studies are sometimes not appropriate for the subject.
So I take it in with a large grain of salt, but I blend it with my own experiences and my own rational thinking. So yes, anecdotal evidence counts for me, sometimes as much as what I’ve read about the “science.”
But I try not to make claims beyond my evidence. I can tell you with extreme confidence that homeopathy is not placebo, but if you want to know about condition X, all I can say is, “In my experience, I’ve seen thus and so.”
John, that’s just wrong. The study looked at 17,674 children. Thats easily enough to produce statistically meaningful results.
My criticism is that the results they published do not show a meaningful difference between vaccinated and unvaccinated children, or even boys.
That is at complete variance with their statement that it showed a “strong collelation”. Moreover, their statement that:
is also wrong. The survey doesn’t show that there is a link at all.
Again, if scientists and doctors ignore this study, its not because they are locked in a paradigm and can’t take note of alternate views. Its because there is nothing in the study to take notice of. It merely repeats what can be found in countless other studies - that there is no correlation between vaccination and autism.
Re. your other points. Of course everyone on the planet navigates through life using a mixture of experience and learnt knowledge. The hard part is that both can be wrong and its very difficult to tell the difference.
I also rarely believe anything I read in the press. If you want to get to the bottom of any scientific or medical claim you really need to read the original article and take a hard look at the data, methodology, analysis etc.
And even then it may well turn out to be wrong. I suggest that you take a look at the work of John Ioannidis who has exhaustively found that an awful lot of initial scientific medical claims turn out to be wrong.
This isn’t at all a criticism of science. Its more a description of how research works. People frequently find sketchy evidence that, for example, eating a substance increases your risk of cancer. This finding then gets splashed across the media.
The next step is where the real research comes in. Others scruitinise the results and see if there was a flaw in the methodology etc. Still more people try to replicate the results. More often than not it is found that no one can repeat the study and/or that it contained significant flaws (the most common is that initial studies only look at vary small numbers of people). After several years, much scrutiny, and several different studies we might just come to a convincing conclusion.
In my view the media completely misrepresents this process. It massively over-hypes initial studies with poor methodology. It would be much better if it just wrote fewer and much longer articles summarising several years of research. But I doubt that thats going to happen.
Incidentally, this is another reason why people don’t give the research on homeopathy the attention you think it deserves.
I have looked at a lot of such things on the web over the years. As far as I recall, none of them have been successfully replicated. At that level, they just have the same weight as the endless media stories of vegetable X either causing or curing cancer - interesting perhaps but in no way convincing.
Actually, I was asserting that there was massive under diagnosis, and we are only now aware of the actual prevalance.
But as I’m closely reated to people who work in that field, I could be biased.
I can produce meaningful results with a study of 10, depending on the frequency of what I’m measuring and what my results are. Is only looking at 533 unvaccinated boys enough to be statistically significant in this case? I don’t know. I haven’t looked into it myself yet.
But not statistically significant is not the same as saying there is no link, espcially when the difference is a pronounced as they found in their study. It does mean, however, that randomness might account for the result. It means that a larger study is called for in order to settle the question.
I agree that no correlation has been shown. However, your statement is misleading. It implies that the issue has been well studied and no correlation has been found. In reality, no correlation has been found but the powers that be have not done due scientific diligence.
You know, this discussion illustrates my point. I didn’t sit down and work out statistical details when I read this web page. I don’t do that when I read a scientific claim. I don’t have time to do that (although I’m will now for this one). But I didn’t matter to me. Why not? Because I’m not claiming that vaccination causes autism. Because it is already the case that what I hear from my government doesn’t jibe with my personal experience, although I’m very willing to admit that I’m wrong. The conclusion of this study was that the CDC ought to do a larger study. I agree.
I do not think current research on homeopathy deserves great attention. If I didn’t have personal experience with homeopathy, I would be very skeptical of any such study reported in the media.
What I do object to is that I continually read that there is no evidence to support homeopathy. This is a false and unscientific statement. Proof and evidence are too often confounded by people with an agenda.
I don’t want to just dump on the media. Yes, they often get it wrong. But one of the main reasons for this is that “scientists” often insert their agendas into the discussion, which then appears like science to the general public. In other words, it’s not just about the facts, but instead it’s very political and even quasi-religious. On both sides. Most of the criticisms of homeopathy that I read are more about belief than fact. And unfortunately that same is true about many claims of various holistic treatments, including homeopathy. Again, what I object to is the anti-homeopathy side claiming the scientific high ground when it’s more accurate to say that there is a difference of belief. However, I believe that truth will out, science will resolve some of this in time, and 10 years from now we will have moved on from “does it work” to arguments about the details.
In any case, I’m a strong believer in health freedom, meaning that I think that people ought to be able to decide for themselves and not be at the mercy of scientists with agendas for what they can or cannot do, and have free access to everything that isn’t intrinsicly harmful.
Okay, back to the statistics from Generation Rescue’s Cal-Oregon Unvaccinated Survey. By the way, I emailed them and they sent me a nice Excel file with the data.
I disagree, and so would many scientists. There is more to statistics than significance. Sometimes the results are interesting despite lack of 95% confidence, and suggest the need for further research. Does only a 95% confidence interval count? If it’s 80%, do we just forget about it? No, although any conclusions are certainly not going to be as strong. On the other side, with a large enough sample size one can find statistically significant results that are clinically meaningless.
In the study we’re talking about, the percentages for unvaccinated vs vaccinated boys (combining partially, fully, and fully+) with autism are:
Unvaccinated boys with autism = 2.25%, 0.64% std error.
Vaccinated boys with autism = 3.63%, 0.14% std error.
These don’t overlap at the 90% confidence level. I’m not a statistician, I’m an ex-engineer. I’ve never studied the statistics associated with polling. Perhaps I’ve misinterpreted the numbers. But I certainly looks to me like there is a low likelihood of a false positive, just based on crunching the numbers.
Perhaps there is a methodological problem with this study. They claim it’s the same methodology as the CDC uses. I agree that the 7% on the partially vaccinated group is a problem. I can speculate that the partially vaccinated group is that way because of a reaction to a vaccination, and therefore would have a higher rate. Autism is much more common in boys that girls, so that aspect of the results don’t seem out of order. One concern is that unvaccinated populations may also avoid other toxins that could be a factor. But my biggest concern about this study is that the reported rates of autism are much higher that official numbers about its prevalence, which are about 1 in 150. Also the statistics depend on a measurement of a base state which uses a count of 12 for the boys and 6 for the girls. With such low numbers, I wouldn’t want to make any bold claims.
The study did not just look at autism, however. It found that the rate of neurological disorders was roughly twice as likely in vaccinated populations. There may be other issues with this study, but those numbers are clearly statistically significant.
All I know is that this looks like sufficient evidence to warrant further research, and that this is one piece of counter-evidence to the claim that there is no link between vaccination and autism. I’m not claiming anything beyond that here. I’d like to see more science on the subject so that I can make better decisions. As Carl Sagan said, “Absence of evidence is not evidence of absence.”
But if you keep lowering the threshold you will always find some pattern or other. People, medical conditions, other environmental factors aren’t uniformly distributed. There is always going to be some town where there are more autistic children per capita, or a slightly higher percentage of parents of autistic children that like a particular soft drink, etc etc. Sure, such a pattern pattern could represent a link found in reality, or it could just be a random artifact from the data. If you keep lowering the threshold you can’t tell the difference.
The problem with this survey is that even this result is pretty marginal and the other results are all over the place. Say with Girls with ASDs we have:
Unvaccinated: 3% of total
Partially vaccinated: 3% of total
Fully vaccinated: 1%
Fully and Partially combined: 1%
Does that mean that vaccination promotes autism in boys but prevents ASDs in girls? I could point out other odd results from the survey. You can’t just focus on one set of results (boys) and ignore the inconvenient ones.
As you mention, there are two startling things about the survey. The first is that partly vaccinated children have a a higher rate in several of the results, and the second is that the rate of autism is much higher than other studies suggest.
My assumption is that the two are linked. I don’t have any evidence for this. But its how I would explain the results.
In any phone poll a lot of people are going to hang up as soon as they hear who is on the end of the line. Moreover the questionnaire included numerous occasions in which the survey would end prematurely. It would be nice to know the drop-out rate but we aren’t given that info.
There has been a lot of media attention on possible links between autism, Asperger’s and vaccines. Parents have stopped their children being vaccinated because they are very concerned.
The survey questionnaire started with a statement that it was looking into “vaccinations and children’s health”. And it then asked questions exclusively about those two subjects.
My guess is that the parents of partially vaccinated children are those that are sufficiently worried about vaccines that they withdrew their children from further vaccine programmes. Moreover, that the parents of vaccinated children with autism would also be aware of the reported effects of vaccines.
Those people are more likely to stay the course and complete the survey without hanging up.
As mentioned, its just a guess. But is seems much more reasonable to me than some unknown mechanism by which being partially vaccinated is more dangerous than fully vaccinated, but only in boys, because in girls vaccination seems to prevent ASD and be neutral concerning autism etc etc etc.
I don’t disagree. You’ve expanded on the points that I mentioned in my previous comment.
I’m not claiming that this study proves anything. But neither can I claim there is no evidence, even if it looks like there are problems with that evidence. I with I could look at a larger study, but it hasn’t been done. That’s the point.
I read in the media that no link has been shown between vaccines and autism. That’s true enough, but it’s misleading when it hasn’t been properly researched. Instead what the media does is extrapolate from a study refuting a link from one particular component of vaccines and autism to the larger claim that vaccination is very safe.
Post Your Comment: