After reading this chapter and the article.I think I am little confused about the difference between pilot test and pretest. On one hand, pretest is taken before services or activities begin. It is compared with the results of a posttest to show evidence of the effects of the services or activities being evaluated. On the other hand, pilot study is a mini version of a study conducted before committing to the big one. In the article, it seems to me that they did a pilot study to ensure that there is a significant difference between uncertainty levels. They used on 43 students not all students. In order to consider that a pretest, I think all participants need to take the test. Furthermore, all participants need to take the posttest too. However, the authors used only about 30 percent of the sample for what they referred as pretest. In this context, I believe that they used the term wrong.

Survey research is the collection of a small amount of data in standardized form from a large number of individuals (Robson, 2002). A survey research is different than experimental research because it measures many variables, test multiple hypotheses, and infer temporal order from questions about past behavior, experiences, or characteristics.

Robson (2002) states that surveys are used in descriptive or explanatory research. In this context, Neuman (2003) indicates that surveys with open-ended questions can be used for exploratory purposes. For instance, a researcher can use open-ended questions in pilot tests and then develop closed question responses.

 

When conducting a survey research, one has to consider both internal validity and external validity. If questions are ambiguous, the exercise will be waste of time. Therefore, a researcher uses pilot tests to ensure that the intended meaning in the questions are clear. The selection of representative samples from known population is critical in survey research because if sampling is wrong, a researcher will not be able to generalize the findings. When generalizing the findings to the population from which the sample is drawn, a researcher should also consider the homogeneity of the population.

 

Among the surveys that I have filled out for my friends, I faced the following problems defined by Neuman:

• Noattitude,
• Lack of partially open questions, and
• Length of Survey.

 

In some surveys, I had no opinion about the issue and I felt that the researcher was forcing me to make a selection. Therefore, I selected the neutral option. I am not sure whether my answers for some surveys were helpful or not. I beleive that our friends who conduct surveys should be careful about this issue because it will impact the generalizability of their findings. 

 

Another problem that I noticed (especially in the survey we filled out in our class) is the lack of partially open questions. Recently I got a high cholesterol problem and in some of close-ended question I felt that the researcher could not provide an answer that captured my true feelings.

 

Finally, the length of the surveys that our friends prepare is the scariest part for me. In my previous school, I conducted an online survey to determine students’ feelings toward online classes. In this study, my department gave me about 20 questions and I put only 5 questions to a page. The online survey had 5 pages including the “thank you” page. With this format, I got a very high return rate on the Christmas break from the students.

I believe that well designed, well performed, and well reported survey research can definitely contribute knowledge to the field.  However, it bothers me that this has become the first choice for many researchers and as such has been overused compared to other types of research methods.  Some of this may be to the bias against "Qualitative" or as Robson called "Flexible" research amongst many IS researchers.  As I was thinking of this, it might be interesting to design a research study to get at what IS researchers believe is scientific, in this way try to understand their biases for or against different research methods.  This research method should be conducted to include survey research, but not be exclusive to this method.  Perhaps this research does exist and it seems to me would be very useful to understand what will be accepted for publication.  We could then analyze the top journal articles to see if the researchers are consistent with what they say is scientific. 

The article in the link below examines the quality of survey research methodology in MIS by reviewing 122 survey based studies in our field and states that the key problem of the survey research in our field is the weakness in application of survey methodology, not inappropriate technical knowledge concerning the methodology. The results of study indicate the following problems when conducting survey research: 

1) Single method designs where multiple methods are needed

2) Unsystematic and often inadequate procedures

3) Low response rates

4) Weak linkages between units of analysis and respondents

5) Over reliance on cross-sectional surveys where longitudinal surveys are really needed 

Based on the findings, the authors say that “the quality of surveys varies significantly among studies of different purposes: exploratory and descriptive studies are of moderate to poor quality overall, and explanatory studies are of good quality. The lack of rigor in descriptive and exploratory surveys is unfortunate”. In summary, the authors in this article show disappointment after considering the extent to which survey research is used and the proportion of survey-based studies in MIS that fail to measure up. 

Pinsonneault A., Kraemer L. K: SURVEY RESEARCH METHODOLOGY IN MANAGEMENT INFORMATION SYSTEMS: AN ASSESMENT (Working Paper )

http://www.crito.uci.edu/research-archives/pdf/urb-022.pdf