Autism Vox
Common Sense
Developing Intelligence
Dr. Joan Bushwell's Chimpanzee Refuge
Dr. X's Free Associations
Geek Buffet
Greg Laden's Blog
Jabal Al-Lughat
Lingformant
Literal-Minded
Living the Scientific Life (Scientist, Interrupted)
phonoloblog
Polyglot conspiracy
Providentia
Tenser, said the Tensor
The Omnibrain
The Phineas Gage Fan Club
When I first started doing Web-based experiments, a number of people in my own lab were skeptical as to whether I would get anything valuable out of them. Part of this was due to worries about method (How do you know the participants are paying attention? How do you know they are telling the truth?), but I think part of it was also a suspicion of the Internet in general, which, as we all know, is full of an awful lot of crap.
For this reason, I expected some difficulties getting my Web-based studies published. However, the first of these studies was accepted without much drama, and what concerns the reviewers did raise had nothing to do with the Web (granted that only one of the experiments in that paper was run online). Similarly, while the second study (run in collaboration with Tal Makovski) has run into some significant hurdles in getting published, none of them involved the fact that the experiments were all run online.
Until now. After major revisions and some new experiments, we submitted the paper to a new journal where we thought it would be well-received. Unfortunately, it was not, and many of the concerns involved the Web. Two of the reviewers clearly articulated that they just don't trust Web-based experiments. One went so far as to say that Web-based experiments should never be run unless there is absolutely no way to do the experiment in the lab.
(I would use direct quotes, but the reviewers certainly did not expect their comments to show up on a blog, anonymously or not. So you will have to take my word for it.)
Obviously, I trust Web-based experiments. I have written enough posts about why I think concerns are misguided, so I won't rehash that here. I am more interested in why exactly people have trouble with Web-based experiments as opposed to other methodologies.
Is it because the Web-based method is relatively new? Is it because the Internet is full of porn? Or is it simply the case that for any given method, there are a certain number of people who just don't trust it?
I have been doing street-corner surveying lately (a well-established method), and I can tell you that although it ultimately gives decent results, some very odd things happen along the way. But I suppose if, as a reviewer, I tried to reject a paper because I "just don't trust surveys," the action editor would override me.
Printer-friendly version
Comments
web-based research has advantages and disadvantages
July 26, 2008 by Anonymous, 1 year 17 weeks ago
Comment id: 31263
The only real and unique problem I see with web-based research is that there is a virtual certainty that at least some of the replies will be fakes from people who think they are being funny. In person, you could weed these out by the expression on their face, on the internet this is more difficult. I have also seen humor web sites which direct people to survey web sites with the recommendation that they give one particular answer. Statistical analysis based on how the person found the web site, time of day, bursts of similar answers, etc. could help get around this.
The presumed gold standard of in-lab tests seem to me to have far more faults than web experiments. In-lab test are almost all done at a single university on students who are all have similar ages, backgrounds, interests and motivations. Yet the results of these experiments are presumed to apply equally to all people everywhere. It is only lately that researchers are beginning to see the mistakes in this presumption. I wonder if those who are most opposed to web-based research are those who are most defensive about the limitations and error factors in their own types of research.
The advantages of web-based research seem fairly obvious: more participants at lower total cost, wider range of demographics, less chance of influencing results by the experimenter and so on.
One fault which is not necessarily exclusive to the internet is poor design. I remember one survey which stated something like "Assume you have just stolen something from one of you co-workers" and then proceeded to ask a series of questions. Since I had never done the stated action in my life, the rest of the questions were meaningless.