One of the most difficult problems in research on humans is response bias. If you want to study rabbits, it’s relatively easy to get a random sample of rabbits to test. Rabbits have very little say in the matter.
Humans, on the other hand, can choose to participate or not. Certain types of people tend not to participate in experiments (low-income, rural people, for instance), while other groups provide the bulk of test subjects (college psychology majors, for instance). If you are studying something thought to be fairly independent of geography, SES or education (low-level vision, for instance), this may not matter so much. If you are studying social attitudes, it does.
Web-based Response Bias
One potential concern about Web-based research is that it changes the dynamics of the response bias. Surveying college undergraduates may involve consider sample bias, but at least we know what the bias is. Some researchers are concerned that Web-based studies must suffer from some research bias (suburban teenagers are probably over-represented on the Web), but it’s one we don’t understand as well and have less control of.
At least one study suggests that, at least in some circumstances, we may not need to worry about this. This is according to Sabina Gesell, Maxwell Drain and Michael Sullivan, writing in the last issue of the International Journal of Internet Science. They ran an employee-satisfaction survey of hospital employees. Half were sent surveys by the Web, and half by regular mail. The response rate (the number of surveys completed) was equivalent using both methods, and the reported job satisfaction of both groups was identical.
Interestingly, the respondents were also asked whether they preferred a Web-based or paper-based survey. The most common response was that they did not care, but of those who expressed an opinion, the majority preferred the survey type they had actually received (sounds like cognitive dissonance).
——–
Sabina B. Gesell, Maxwell Drain, Michael P. Sullivan (2007). Test of a Web and paper employee satisfaction survey: comparison of respondents and non-respondents International Journal of Internet Science, 2 (1), 45-58