Q: In the abstract the authors write: ‘We argue that the differences found are attributable to sample bias rather than mode, and that the ramifications of low response rates in web surveys are more far reaching than has been recognized.’ To what extent do their data and analyses present a good argument for this? Why? What might you have suggested to them to improve this argument? What other criticisms can you offer?
A.: The article by Grandcolas, Rettie & Marusenko (2003) has for its purpose the determination of the causal relationship between the differences in surveys’ item responses (dependent variable) and sample bias and administration mode, whether paper or Internet-based (predictor variables; Easterby-Smith et al. 2012, p.283). Three major hypotheses were to be tested in the course of the research: 1. That the Web and paper-based questionnaires and surveys differed in their response distribution. 2. That the administration mode and item responses were associated/correlated with each other. 3. That the partial causality was in place between the survey’s administration mode and the item responses’ differences (Grandcolas, Rettie & Marusenko 2003, p.546).
Proceeding from the results of the study, the authors purported to achieve a conclusion that the sample bias played the greater role in the item responses’ differences than it was anticipated (Grandcolas, Rettie & Marusenko 2003, p.553). However, their answer to the third research question, in spite of their statements, may appear inconclusive. As it would understood from the subsequent discussion, the interpretation of their own research results provided by Grandcolas, Rettie & Marusenko (2003) may be lacking in some important aspects, and thus their analytical methodology is in need of several important readjustments so that the study discussion may exhibit more consistency.
To begin with, the very selection of the sample in terms of the two sub-groups of the respondents may be problematic. As noted in the discussion itself, the student population selected for the survey comprised the students that were all Internet users but in the context of this study were interviewed via mail or Internet survey, respectively (Grandcolas, Rettie & Marusenko 2003, p.546). However, this would lead to the problems with regard to the administration mode, for the study’s design would appear to incorporate the sample that was non-representative of the students who were not Internet users. While this would make the research design more complicated (Easterby-Smith et al. 2012, p.281), this decision would allow the researchers to explore additional predictor variable that might have an impact on deciding whether any causality was present between the administration mode and item responses.
The concept of multivariate analysis of causal models would be of great assistance within the context of this discussion. According to Easterby-Smith et al. (2012), in order to provide a successful multivariate analysis for causality, such aspects as the quality assessment of the fitted causal model or the observed variables analysis should be accounted for by the researcher (Easterby-Smith et al. 2012, pp.294-295). Several typical models for multivariate analysis for causality are introduced by the researchers, with the majority thereof being based on the multiple regression analysis (MRA) models (Easterby-Smith et al. 2012, p.294).
In the case of the research conducted by Grandcolas, Rettie & Marusenko (2003), ordinary least squares regression technique was utilized to assess the impact of the administration mode on the item responses’ differences between the two samples (Grandcolas, Rettie & Marusenko 2003, p.550). The authors claim that the use of such statistical analysis method was justified by the fact that no actual prediction was supposedly involved here; however, one may point out that the ordinary least squares regression is used as the measurement tool for establishing the correlation between the dependent variables and the variables that may influence the changes in the latter. As noted by Easterby-Smith et al. (2012), the simplification through statistical correlation may be useful instrument in the situations where it is necessary to control for statistical patterns in the relationship between the three variables (Easterby-Smith et al. 2012, p.282). If the authors had proceeded from the hypothesis of the plausibility of the partial correlation between administration mode and sample bias in influencing the participants’ item responses, they may have incorporated the relevant simplification tool provided by Easterby-Smith et al. If the subsequent testing had enabled Grandcolas, Rettie & Marusenko (2003) to conclude that the partial correlation between the two predictor variables may be spurious or insignificant, then the possibility of the interaction bias would have been removed from the study. For instance, the results of the study by Kader, Adams, & Mouratidis (2010) on the co-determination of underwriting and solvency risks, on the one hand, and tax expectations on the other, mutually influence the dependent variable (i.e. reinsurance level). In case of the present research, though, it is unclear how the authors controlled for the mutual interaction between the two independent variables in question.
Finally, the issue of the observed and latent variables may be raised. While Grandcolas, Rettie & Marusenko (2003) included administration mode as a dummy variable in their regression analysis (Grandcolas, Rettie & Marusenko 2003, p.550), they did not account for possible covariance between the observed and latent variables such as gender or the participants’ academic proficiency. According to Easterby-Smith et al. (2012), the assessment of the fit of the model would include the calculation of an estimated population covariance matrix and the sample covariance matrix (Easterby-Smith et al. 2012, p.309). Unfortunately, the authors failed to provide relevant calculations in the appendices to their model, so that the validity of their sample approach as such may be questioned by the independent observer.
Proceeding from the aforementioned observations, one may note that the establishment of the causality dependence between the predictor variables, on the one hand, and the dependent variable, on the other, require certain additional clarifications that in turn demand the extensive usage of various multivariate analysis models. The incompleteness of the authors’ own statistical model would not allow them to isolate the possible interaction bias in the relationship between the administration mode and sample bias variables, leading to potentially doubtful results of the survey.