Among the research methods we have seen so far, I would say that historical comparative research is the most difficult one. From my point of view, moving from field research (hanging out with some exotic group of people) to historical comparative research (“achieves located in a dusty, out of the way room of a specialized library”) seems like a big change.

 

In this chapter, I liked comparative research and secondary sources (similar to Chapter 11). However, I am not sure whether I understood Neuman when he said “most positivist research is not comparative”. When we conduct a positivist study, to a certain extent, don’t we compare our study to other current formal studies in order to show that what we are doing is in tune with other researchers’ attempts.

 

This chapter also made me think about our previous review. I think it is possible to consider the “Complementary Use of Modeling Techniques: Insights from Representation Theory and Practice” as a comparative research to some extent. There are conflicting ideas on ontological foundations of conceptual modeling between Wand\Weber and Wyssusek. In this context, the authors of that paper compare these conflicting approaches and try to justify their study based on this comparison. Furthermore, the authors also compare the results of this study with a similar, recent study to provide further confidence for the conclusion.  

Based on the papers we have read, I think conducting any study (case study, field study…) in a systematic, skeptical, and ethical study is both a form of science and art. Although there is no complicated statistics in the field study, a researcher who intends to use this method should have “a strong sense of self, an incredible ability to listen and absorb details, tremendous patience, sensitivity and empathy for others, superb social skills…” Therefore, I do not think that field study is proper for every researcher.

 

According to Neuman, “field research is valuable for micro-level or small-group face-to-face interaction and it is less effective when the concern is macro-level processes and social structures”. In this context, I wonder to what extent field study is applied in our discipline. For instance, is it possible to link ethnographic study to case study or perhaps to grounded theory when conducting IS research?

One of the points that appears in the chapter is that data contains a limited amount of information.  You can summarize it but it is difficult to make it more accurate after the fact.

The concept of significant digits or figures states that you cannot add digits after the initial values.  For example if you have a value of 3.5 you cannot arbitrarily make it 3.500.  The value that you took is actually be between 3 and 4.  It is closest to the middle but how close?  The researcher may not know if it is 3.6 or 3.4.  So looking at the raw data it would be unreasonable to assume that if you saw a value of 3.5 to assume that it is 3.500000000.  You need to need the precision of the experiment and how many significant digits there are.

 

We can tie this concept back to the chapter by looking at the amount of information in the different scales.  An ordinal scale will not have as much information in it as an interval scale.  There is a hierarchy in that you can go downward from an interval scale to an ordinal scale but not vice versa.   

The article below points out common forms of statistical errors in the biomedical literature. I wanted to talked about this discipline rather than IS because everything I read about dietary or healthy food does not sound credible. For multivariate analyses, these are:

1. Not confirming that the data met the assumptions of ANOVA.

 ANOVA assumes that the response variable is approximately normally distributed within each level of the explanatory variable and that the variability of these distributions is approximately the same. If data are not data are not normally distributed, than the data need to be mathematically transformed into distributions that are more normally distributed.In a way, this error also sound similar to a researcher using secondary data or existing statistics that are inappropriate for his or her research question, which was described in the previous chapter. 

2. Not Identifying the Procedure Used to Adjust for Multiple Comparisons in ANOVA.

 ANOVA is a group comparison that determines whether a statistically significant difference occurs somewhere among the groups studied. If a significant difference occurs, ANOVA is followed by a multiple comparison procedure that compares combinations of groups to determine which groups differ statistically. 

3. Not Testing the Explanatory Variables for Interaction or Colinearity

 Two explanatory variables are said to interact if the effect of one of the response variables depends on the level of the other. Interaction implies that the factors should be considered together, not separately. Two variables are said to be colinear if they are highly associated and therefore provide the same information in the model. 

4. Not Indicating the Goodness-of-Fit of the Model to the Data

 Goodness-of-fit indicates how well the model expresses the relationships observed in the data. In multiple regression analysis (not ANOVA), the value of R² should be reported. This value indicates how much of the variation in the response variable is explained by the factors included in the model. Thus, the higher, the better. 

Errors in interpreting differences between groups include: 

1.Not Reporting Confidence Intervals with Estimates

When interpreting any difference, whether it is statistically significant or not, the direction and magnitude of the difference should be evaluated. However, because a study is based on a sample of the population of interest, rather than on a census of the population, its results are actually estimates of the differences expected if the study were to be repeated on the entire population. Thus, another factor that should be considered when evaluating differences is the precision of the estimate.  

2. Reporting Only Relative Differences and Not Absolute Ones

The absolute difference between groups is simply the mathematical difference between their values, whereas the relative difference is the absolute difference expressed as apercentage. By themselves, relative differences can mislead because they can make differences appear to be larger or smaller than they really are.  

3. Not Differentiating Between Unit of Observation and the Number of Patients Improved

The unit of observation or the unit of analysis is what is being studied. In clinical research, the unit of observation is usually the patient. However, sometimes the unit issomething other than the patient. The problem comes when, say, differences are reported for the unit of observation but not for the number of patients in whom differences occurred. For example, if a drug markedly improves mean glomerular filtration rate in patients with renal disease, it may also be helpful to know how many patients actually improved. 

 

Reference

Lang Tom. Common statistical errors you can find. Errors in multivariable analyses and in interpreting differences between groups. AMWA Journal, Vol 18, No 3. 2003

http://www.aliquote.org/cours/2006_cogmaster_A4/ressources/Statistical_Errors_Part2.pdf 

 This chapter talks about the analysis of quantitative data, which we have already discussed in the notes Professor Ryan gave us. Although Neuman described the three elements of causality in the third chapter, he elaborates the difference between causality and correlation in this chapter. Based on the third chapter, three things are necessary to establish causality: temporal order, association, and the elimination of plausible alternatives.

In this context, I want to go back to the midterm paper we evaluated one more time. As you all remember, the midterm paper was an experimental study that looked at the relationship between “behavioral intention” to use and “positive mood” while considering uncertainty. The author established temporal order by measuring the mood of the subjects with a survey prior to the judgment task. The tables that are provided (for instance, uncertainty manipulation check with p value less than 0.001) showed that there was an association among the variables. However, I am not sure whether the author did a good job on eliminating alternative explanations for results. As Rosemary indicated, we do not know whether the students consumed the candy or perhaps some students were still a little bit drunk because they went to a club on Thursday night (I used to do this on Thursday nights often during my bachelors degree). Therefore, I believe that in the current format the author shows a correlation between positive mood and behavioral intention to use. However, the elimination of plausible alternatives needs to considered in order to show causality.

This chapter got me thinking about the problem with governmentally collected statistics, specifically the “Official inflation rate” and how it is used for a wide variety of purposes.  We are told that in the past few years, the beast of inflation has been conquered by careful planning by our central bank, with inflation now around 3% a year.  CPI (Consumer Price Index) is used to determine pay raises, cost of living increases, etc., but I believe it is often a poor measure of the increases that consumers actually are feeling.  The cost of gasoline has increased more than 3% a year, and so has the cost of housing, but somehow during the years when these two key expenses were spiking up, the CPI seemed to keep falling.  This discrepancy between the experience of the population and the “official” rate are determined to suit the purposes of government.  The CPI is an abstraction of the true rate of inflation with some spin, but it seems that research with these politically tied numbers might suffer from problematic conclusions. 

The flowchart in the link below show the steps a researcher follows when conducting a research. The flowchart is not very different than other research methods. It begins with a theory and rationale which is followed by conceptualization, operationalization. However, as Robson stated the important thing in content analysis is that instead of desiging study then collecting data, the researcher starts by finding out what data are available and then identifies a model. 

http://academic.csuohio.edu/kneuendorf/content/resources/flowc.htm

Experiments, survey research, and content analysis are different methods of conducting a research. As a researchers, we need to understand which method best applies to which situation in order to defend the findings of our studies. For instance, taking a snapshot and then talking about a process is not proper way of conducting a research.

In this chapter, Neuman elaborates the content analysis and existing statistics (secondary survey data) which he briefly described in Chapter 2. According to Neuman, both content analysis and existing statistics can be used for exploratory and explanatory purposes, but most often used for descriptive purposes. In this context, I like to approach content analysis, which is concerned with data reduction from SWOT analysis perspective.

Strengths:

• Content analysis describes what is in the text.
• The data are in permanent form and hence can be subject to re-analysis, allowing reliability checks and replication studies.
• It is based on existing documents. Hence, a researcher can observe without being observed.
• It may provide a low cost form of longitudinal analysis when a run or series of documents of a particular type is available.

Weaknesses:

• Content analysis can not reveal the intentions of those who created the text or the effects that messages in the text have on those who receive them.
•  The documents available may be limited or partial.
• The documents have been written for some purpose other than for the research.
• It is difficult to assess causal relationships. Are the documents causes of the social phenomena the researcher interested in, or reflection of them? (Robson, 358)

Opportunities:

• A researcher can measure large amounts of text with sampling and multiple coders.
• It is helpful when a topic must be studied at a distance.
• Content analysis can reveal messages in a text that are difficult to see with causal observation.

Threats:

• Intercoder reliability. The researcher must check the degree of consistency among coders.
• Visual text communicates messages or emotional content indirectly through images, symbols, and metaphors. Hence, it is difficult to measure.  

In this context, as Nicole indicated, I wonder how the PhD students in our school approach to content analysis other than literature reviews. I have not heard a student conducting a meta-analysis in our school. It seems to that most of them prefer conducting surveys. As for me, the findings from a meta-analysis paper can be as interesting as a survey research.

For instance: Dubé and Peré's Rigor in Information Systems Positivist Case Research: Current Practices, Trends, and Recommendations paper or the working paper I cited last week. These papers showed me how other researcher conduct research and the mistakes that they made.

The data is already available .Why not use it? Isn't anyone interested in examining existing data?

The article in the link below examines the quality of survey research methodology in MIS by reviewing 122 survey based studies in our field and states that the key problem of the survey research in our field is the weakness in application of survey methodology, not inappropriate technical knowledge concerning the methodology. The results of study indicate the following problems when conducting survey research: 

1) Single method designs where multiple methods are needed

2) Unsystematic and often inadequate procedures

3) Low response rates

4) Weak linkages between units of analysis and respondents

5) Over reliance on cross-sectional surveys where longitudinal surveys are really needed 

Based on the findings, the authors say that “the quality of surveys varies significantly among studies of different purposes: exploratory and descriptive studies are of moderate to poor quality overall, and explanatory studies are of good quality. The lack of rigor in descriptive and exploratory surveys is unfortunate”. In summary, the authors in this article show disappointment after considering the extent to which survey research is used and the proportion of survey-based studies in MIS that fail to measure up. 

Pinsonneault A., Kraemer L. K: SURVEY RESEARCH METHODOLOGY IN MANAGEMENT INFORMATION SYSTEMS: AN ASSESMENT (Working Paper )

http://www.crito.uci.edu/research-archives/pdf/urb-022.pdf

 

I believe that well designed, well performed, and well reported survey research can definitely contribute knowledge to the field.  However, it bothers me that this has become the first choice for many researchers and as such has been overused compared to other types of research methods.  Some of this may be to the bias against "Qualitative" or as Robson called "Flexible" research amongst many IS researchers.  As I was thinking of this, it might be interesting to design a research study to get at what IS researchers believe is scientific, in this way try to understand their biases for or against different research methods.  This research method should be conducted to include survey research, but not be exclusive to this method.  Perhaps this research does exist and it seems to me would be very useful to understand what will be accepted for publication.  We could then analyze the top journal articles to see if the researchers are consistent with what they say is scientific.