The last decade has seen a gradual replacement of hypothesis-driven research by hypothesis-free high throughput studies – at least when looking at the selection of publications that made the biggest splash. The reasons for this development are obvious: the development of new techniques, the availability of entire genome sequences, and progress in lab automation have enabled high-throughput research. Without any doubt, these large-scale studies yield more results per Euro, as it has always been much easier to find out anything in general rather than something in particular. Imagine an old fashioned biologist, throwing out his hypothesis-driven fishing line into the pond of science, hoping that a particular fish will be attracted by the highly specific bait. Then imagine a group of contemporary way cool high-throughput researchers (those guys always come in large groups), trawling the same pond with their fishing net, catching literally thousands of fishes in the same time. With a little luck, they also catch the particular fish that the first guy was after. If so, great. If not, who cares – there are lots of other fishermen out there. Some of them will be interested in the fish caught in the trawling effort.
When I come to think about this kind of situation, which I do quite often, I always feel terribly old-fashioned, if not old – much older than I really am. I always tend to sympathize with the lone fisherman (think: old man and the sea). Leaving aside thoughts about who is more deserving of the catch, there are quite a few arguments in favour of hypothesis-driven approaches. Here are the two I consider the most important ones: i) the pond of science is actually quite big, and if it is a rare fish you are after, you might not be able to find it in a non-targeted trawling approach, and ii) given the large numbers of fish that the trawlers have to deal with, plus the fact that they are often more experts in trawling techniques than in marine biology, chances are good that they don’t even recognize what they have caught – or at least do not appreciate its significance.
There is a third issue that is heard often in discussions on hypothesis-driven vs. high-througput approaches: reliability. The rest of this post will be devoted to this particular issue, and you will see that I entertain a rather unconventional opinion on this topic. For simplicity, I will restrict the discussion to a particular topic, which is the detection of protein-protein interactions. However, most of my points will also apply to most other research areas. In the protein interaction field, errors come in two flavours: reporting interactions that are not real, and missing interactions that are real. There is little doubt that systematic studies produce on average more errors than a good hypothesis-driven study. The reason is simply that because there are more results, less time and effort can be devoted to reliability testing per result. However, there are several factors counter-acting this trend, e.g. the detection of notorious trouble makers among the interaction constructs is much easier to do in a large data set than in a single experiment.
Before discussing the relative merits of the approaches, I would like to ask a question. Imagine that you are interested in a protein X. You check the literature and various databases and find a single report that the protein X interacts with the p53 protein, which is quite unexpected but could have interesting implications. Before you start to pursue this line of research, you ponder the reliability of the published evidence. At this point, please imagine 5 different scenarios concerning the evidence’s provenance :
- A researcher working on protein X had the hypothesis that it might interact with p53, tested it and found it to be true.
- A researcher working on p53 had the hypothesis that it might interact with protein X, tested it and found it to be true.
- A researcher working on protein X wanted to find out what it does, did a screen, and found that it binds to p53
- A researcher working on p53 wanted to find further interactors, did a screen, and found protein X
- The interaction p53 :: protein X was found in a systematic high-throughput study and was verified.
Let us also assume that in all five cases the same experimental methods were used (e.g. an initial yeast two-hybrid assay, followed by a confirmation through Co-IP). My question is, in which scenario would you put most trust into the reported interaction? As an unfaltering believer in published science, this should be a non-issue. The result has been verified by two independent methods, it has passed a peer review, it is published, thus you should assume that it is true. Most of us (at least those who have ever done experiments on their own) should be more skeptical. From what I wrote in the initial paragraphs of this post, you would assume that I put most trust in scenarios 1 and 2, followed by 3 and 4, and be very skeptical about scenario 5. Here is what I really think: most trust in 4 and 5, followed by 2 and3, most skeptical about 1.
Odd, huh? Let me try to explain. I am somewhat skeptical (you may say: paranoid) about scenarios 1 and 2, because here the researcher had a theory that X and p53 should interact. Without alleging any (intentional) misconduct, I would still be afraid that those researchers have tried their best to find this interaction. If it didn’t work the first time, they might have changed the conditions. If it still didn’t work, they might have used a slightly different construct, and so on. One should not over-estimate the value of controls and verifications. In a typical hypothesis-driven study, only one negative control will be provided. Strictly speaking, this only shows that protein X interacts better with p53 than does protein Y. It is still possible that half of the proteome shows an equally convincing interaction with p53 as protein X. Verifications with independent methods are certainly useful, but at that stage the researcher is even more convinced that the proteins interact: it is predicted by the hypothesis and has been shown in the Y2H experiment. As a consequence, even more parameter tweaking will take place to confirm the interaction. There are also statistical issues, which I don’t want to discuss in here any detail. While thinking about this, I can only conclude that I have lost most of my faith in published experimental confirmations of hypotheses.
Some of my scepticism about scenarios 1 and 3 is explained in my previous post about p53. The idea that people like to see interactions with exciting proteins makes me favour scenario 2 over 1, and 4 over 2; P53 researchers have not much to gain from seeing an interaction with protein X. The reasons why I would mostly trust scenario 5 are the following: The approach was unbiased and the researchers do not see the result as a proof of their superior intellect. Moreover, in a high-throughput setting, chances are good that the screen and the confirmation have only been tried under a single condition.
It is true that high-throughput studies are error prone, too, but these are different kinds of errors. High-throughput errors are often false-negatives (type II) rather than false-positives (type I), and they often are of statistical rather than systematic nature. By contrast, I would assume that hypothesis-driven research has a greater danger of type I and systematic errors, always biased towards confirming the theory.
Is the situation really all that bad, and is there anything we can do about it? Maybe I am exaggerating, maybe I am paranoid – I wouldn’t disagree on both counts. Certainly, not all scientists are equally susceptible to self-delusion. Nevertheless, I think the problem is real. If you don’t believe me, go have a look at the QED paper published 2001 in Genome Biology, a controversial but very instructive piece of work, to which I have contributed some minor examples.