Reply to comment

The only real and unique problem I see with web-based research is that there is a virtual certainty that at least some of the replies will be fakes from people who think they are being funny. In person, you could weed these out by the expression on their face, on the internet this is more difficult. I have also seen humor web sites which direct people to survey web sites with the recommendation that they give one particular answer. Statistical analysis based on how the person found the web site, time of day, bursts of similar answers, etc. could help get around this.

The presumed gold standard of in-lab tests seem to me to have far more faults than web experiments. In-lab test are almost all done at a single university on students who are all have similar ages, backgrounds, interests and motivations. Yet the results of these experiments are presumed to apply equally to all people everywhere. It is only lately that researchers are beginning to see the mistakes in this presumption. I wonder if those who are most opposed to web-based research are those who are most defensive about the limitations and error factors in their own types of research.

The advantages of web-based research seem fairly obvious: more participants at lower total cost, wider range of demographics, less chance of influencing results by the experimenter and so on.

One fault which is not necessarily exclusive to the internet is poor design. I remember one survey which stated something like "Assume you have just stolen something from one of you co-workers" and then proceeded to ask a series of questions. Since I had never done the stated action in my life, the rest of the questions were meaningless.