Zad Rafi • 7 years ago
Keith O'Rourke thank you for commenting. I did read that chapter by you and Sander (I'd be pretty surprised if most people who had the 3rd edition of the book didn't read it, given that meta-analysis is such a popular topic. So I don't think the order of the chapters matter as much as the popularity of given topics).
On the topic of vote counting, you say that it is not fallacious when properly done when
"all you know is the number of significant and non-significant results."
But as you may know very well (since you wrote the section), it also mentions that the test for this can be underpowered and that it would be far better to combine the exact P-values.
https://uploads.disquscdn.c...
My main contention here is to wonder how likely we are to only have information about whether a result is significant or not without any effect estimates, exact P-values, and other relevant information about study designs (sample size, etc.)? I've seen many situations like this, but they're often with old-er studies.
It seems to me that such a situation (where we only have information about significant or nonsignificant for all studies) would be incredibly rare. I may even go as far to say that it is less likely than having adequate information.
Of course, it's still likely and possible, but as you and Sander point out at the beginning of the section, it is far from ideal to engage in vote counting unless it is absolutely necessary. I think vote counting in general should be discouraged, especially for laypersons & researchers (the audience this blog post was generally directed towards) and not necessarily for highly experienced reviewers, since it is widely prone to being misused and misinterpreted
Keith O'Rourke • 7 years ago
Times change - over the years I have suggested to a few dozen people that they may have missed that chapter - you are the first to respond that you had not :-)
Perhaps limited reporting of results has become almost unheard of, but one should try to avoid giving the impression that inquiry is blocked. Inquiry can only start with what people have and if that is just the number of significant and non-significant studies you don't want them to think its hopeless.
(If interested, the more general case of what can be done given various reported summaries was covered here https://phaneron0.files.wor... )
Zad:
An aside (and to underline that the topic in the last chapter of a book is usually neglected), chapter 33 of Rothman KJ, Greenland S, Lash TL, Others. Modern epidemiology discusses vote counting and shows how comparing the number of nonsignificant findings to significant ones as evidence for no effect is NOT fallacious when done properly and all you know is the number of significant and non-significant results.