The chapter on surveys was probably the most relevant for me. At one point the author mentions how surveys can create a false sense of security because of their perceived quantitativeness. I've seen this quite a bit in organizations where people quantify survey data then drive business decisions based on those numbers. I liked that the author went into some detail about how to structure surveys well, the various ways of adminstering them and the pros and cons, and also the pitfalls of surveys altogether. I especially thought surveys were relevant because of the often qualitative nature of some IS projects. For example, what would be the impact of redesigning a very complex site using design principles? While we could measure things like time to completion of a task, often these changes can be so minor we can't really show what the benefit was. However, the qualitative perceptions of those impacted aer another story. Certainly I walked away with a greater appreciation for surveys as well as awareness not to rely on that false seen of quantitative value too much.

After introducing the main research strategies, such as fixed design research and flexible design, in part III Robson continue to talking about the methods of data collection--tactics of research. Robson not only emphasizes on “Surveys and Questionnaires”, “Interviews”, “Tests and Scales”, and Observational Methods”; but also gives us general views about additional methods of data collection, such as “Unobtrusive Measures.”

Survey research method has been used for a long time in various fields. In “real world” life, we are surrounded by surveys in various kinds of manner, such as direct-mail surveys and Internet-based surveys. Based on my prior knowledge, I think the survey is a very simple method and everybody can carry out it. But I changed my opinion about the survey after reading the chapter 8, especially “Steps in carrying out a small-scale interview-based questionnaire survey” in the Box 8.1. That box shows that a period of minimum three to four months for the whole process is required only for a small-scale survey. In the latter contents of chapter 8, Robson continues to talking about the advantages and disadvantages of the survey, which shows us how to do a right survey and how to avoid the threats. Also I like the “Carrying out a sample survey” very much since it is very practical. In that part, Robson teaches us how to carry out a survey step by step from “initial design and planning” to the final design with his valuable examples and magic “boxes”.

Interview is another familiar method in real world research, but I was still getting new thoughts during the period of reading chapter 9. As usual, Robson describes the advantages and disadvantages of interviews. But I want to talk about Robson’s advice for interviewers: “listen more than you speak”. I have similar experience in contrary to interviewer: interviewee. Some of the interviewers were talking too much of introducing the company and other irrelevant topic, and the result is I knew something but they did not get sufficient information from me.

I like Robson’s idea of using multiple methods at the end of part III. The author describes the advantages of the using of multiple methods and additionally provides “approaches to combining qualitative and quantitative methods” in Box 12.8 which is very helpful for us to conduct the research.

In the third part of his book, Robson elaborates different methods of data collection in a real world research. He explains about surveys and questionnaires (chapter 8), interviews (chapter 9), tests and scales (chapter 10), methods of observation (chapter 11), and some additional methods for collecting data (chapter 12).

I found all chapters in this part of the book, highly useful and applicable for everyone who wants to become familiar with the methods of data and information gathering in a real world research. Below, selecting the survey method section in the book, I’ll refer to some interesting and important points in this section:

 About Survey:

- Survey is an important component of a non-experimental fixed design research framework.

- It inherently includes a sense of scientific aspect, and so incorporates a level of confidentiality.

- Its role is more important than just as a tactic or method, rather, it is as important as a strategy.

- Surveys can be recognized as the best way of gathering data, if the researcher exactly knows what kind of information are required to be collected.

- It’s suited to any kind of (purposes of a) non-experimental fixed design research, except for exploratory design (in which researcher, at the beginning, doesn’t know exactly what she/he wants).

- A survey can be in the form of interview-based or posted (mail) [or internet-based] questionnaire, telephone-based or observational survey.

- The internal validity is of a great importance in a survey. It means how valid the collected information about respondents, their viewpoints, ideas, feelings, assessments, doings, etc are. For gaining an acceptable extent of validity, surveyors need to have the necessary level of proficiency in order to be able to design a set of comprehensible and clear questions, otherwise they would waste their own (and also their respondents’) time and effort.

- Generalizability or external validity would be gained when the issues like ‘appropriately sampling’ and also ‘assuming the point that necessarily what respondents say is not what they do’ and more, are taken into account.

- Reliability means whether the survey provides, in a considerable extent, all respondents with the same ‘standardized’ questions.

……………………………………………………………………………………………………………………………………………………………..

 

 - I absolutely agree with the author, that a survey to be a valid and reliable method, needs to be designed and conducted by proficient and logical-minded researchers. Otherwise, it would consist of a set of non-comprehensible and non-clear questions that would result in wasting time and effort in both surveyors and respondents. Unfortunately, nowadays, many people claim that are conducting surveys (especially over internet, which provides some facilities for running a survey, technologically), but their questions suffer from the expected characteristics for a valid survey. In my opinion, this would hurt research community, in a serious manner.

Indeed Robson continues on in a very readable fashion. However, I'm still struggling to dig for prescriptive almost checklist-type stuff that I can tick off when preparing and conducting research. There's a lot of good, common sense advice in here. For example, the author mentions things like evaluation of worth or value (p.205) and managing against tight deadlines (p.204) and provides lots of useful summaries of different Flexible Design strategies like case studies (177-185). While this info from Chapter 6-7 was interesting, it was rather light from a tactical perspective. This in contrast to Ch 4 where the author puts together a useful checklist for choosing a research strategy (p.87). The book seems to toggle between tactical and high-level rather unpredictably.

Perhaps this seems rather light on tactical info because we've already gotten a good primer from previous readings. I'm not sure where the issue lies exactly...whatever it is I continue reading hoping to get more "news I can use" but 2/3 of the way through and it's not quite happening just yet. Despite that, I do especially like how the author applied the "Law of the Hammer" to evaluation: "Get into evaluation [the proverbial hammer] and everything seems to be a [proverbial nail] candidate for an evaluation [waiting to be whacked with a hammer]" (p.204).

Robson, in the second part of his book, explains about different strategies for carrying out a real world research. He elaborates “fixed design”, “flexible design”, and also some special purpose designs such as “evaluation research”, “action research”, and “intervention and change process”. Below, are some important points and issues, extracted from this part of the book.

-A good framework for a design would be the one in which there is a high degree of compatibility among different components, i.e.,  purpose(s), theory, research questions, methods, and sampling strategy.  

- In a real world research, some of the research questions might remain unanswered. At the other extreme, in a research of this kind, investigator might come up with some unexpected and innovative findings and results.

 In a fixed design research:

- Strategies are divided into “experimental” and “non-experimental” strategies.  Experimental design might be “true experimental”, “single case experimental”, and “quasi-experimental”. In the first one, two or more groups are organized; in the second one, the focus would be on individuals not groups; and in the last one, unlike the true experimental, different conditions are not allocated randomly. In a non-experimental fixed design approach, researcher has no active manipulation. 

 - A pilot should always be used in this kind of research (even when a previous exploratory phase has been done).

  - The concept of “variable” is essential.

  - Fixed design researchers remain far from the study, in comparison to a flexible research.

  In a flexible design approach:

- Flexible design strategies are categorized into “case study”, “ethnographic study” and “grounded theory study”.

 - Unlike  fixed design approach, a great emphasis is on researcher. It relies on “researcher-as-instrument”.  It is said that, to a considerable extent, the quality of this kind of study depends on the quality of researcher. Some necessary characteristics of a flexible design investigator are: being open and enquiring minded, good listener, and sensitive enough to the opponents’ views.

- There is a considerable debate about “trustworthiness” in a flexible, qualitative design. For assuring trustworthiness, in this kind of research, there are some constraints. For instance, unlike fixed design research, a flexible research design hardly can be replicated, because initial conditions are impossible to be re-created. Pertaining to the terms of “reliability” and “validity”(though, they are denied by many of proponents of this approach), there are some alternative ways by which one can assess the trustiness of a flexible enquiry. Some of them are: checking ‘inter-observer agreement’, deploying ‘quantitative measurements’, ‘explicit controls’ for validity threats, and ‘direct replication’.

- Unlike fixed design research, in flexible design enquiry, there is no standardization of methods and instruments. So, flexible design researchers should be concern with the reliability of methods and practice are dealing with, by themselves.

- Generalization is another important issue in flexible design research. If, for instance, a researcher in  conducting  an interview or in an observation excludes some people, since finds them threatening, she subjects herself to an account bias.

- A case study is a strategy of doing research in an empirical manner about a specific phenomenon in its context. Unlike a quantitative research, a case study cannot be done disregarding its context.

………………………………………………………………………………………………………………………………………………………

- The main weakness of fixed design is that it cannot deal with the complexity of human behavior. Although, the concept of “detached” researcher for advocators of this type of research is known as an advantage (because researcher doesn’t affect on the results and findings), in my opinion, this feature would cause serious problems. The main advantage of this approach, on the other hand, is its ability to go beyond individualities and develop  generalized patterns and processes.

- In conclusion, I believe that the appropriateness of a strategy for conducting a research depends on the research question and the basic characteristics of the research. However, in social science research in general, and IS in particular, those strategies which are more flexible and more appropriate in dealing with the complexity of human behavior, seem to be more appropriate. 

Part II of Real world Research focuses on two kinds of research methods: fixed designs and flexible designs. Based on my understanding, fixed design research is trying to investigate the existing theories and phenomena; and flexible design research is trying to generate the theory.

In chapter 5, Robson gave us a general view of what fixed design is and how it works. Robson also introduced the differences between experimental fixed design and Non-experimental fixed design by the valued examples and “boxes”. This chapter provided the systemic introduction about the fixed design research, but I like the “Hawthorne effect” and “Randomized controlled trials”, which are very important for us to carry out the field experiments. We can easy to avoid the disadvantages after studying related discussing in this chapter.

In chapter 6, Robson discussed the flexible design using the same frame with last chapter. In my opinion, the concepts of this chapter are much more practical comparing to prior chapters’, such as the three boxes: “characteristics of a ‘good’ flexible design, General skill needed by flexible design investigators, and using reflexivity to identify areas of potential researcher bias”. We can learn a lot of tips for doing the flexible design research.  At the beginning, Robson compared the three influential design traditions within flexible design research which appear to be of particular relevance for real world studies: grounded theory studies, ethnographic studies and case studies. I am interesting in the case studies because I am familiar with that when I studied in the business school. At that time, I just know how to find the problem and the solution through a case study; I never considered about what the case study is and how it can be a research method. I found the answer here and gained even more about how to design case study and the concept of multiple case studies.  

Generally speaking, I like this part very much not only because of the systemic introduction about the fixed and flexible design research, but also the practical notes.

Chapter one gave us a general view of real world research; especially, the Box 1.1 showed some examples to help us to distingue the real world research and academic research. Also I learned a lot of useful preparations for doing the real world research in the future. What I like most is the Robson’s suggestion that “It is good practice to keep a full and complete record of all the various activities with which you are involved in connection with the project.”Everyone knows that the best memory is the pencil. Robson also gave us some possible content should be included in our diary. I will try to make record for some kinds of things even it is difficult to keep doing.

The chapter two introduced several main approaches to do the research, such as “positivist assumptions”, “relativistic”, “emancipator paradigm”, and “realist”. All the contents have already been introduced in prior reading, so I want to talk about the “scientific attitude”. Robson mentioned “systematically”, “sceptically”, and “ethically” involves carrying out research. In my opinion, the ethically is most important since the research result will not be widely accepted without ethical responsible.

Chapter three is most important in part I because it turns back to “real world” and provides a lot of real world approaches to develop the ideas. I like the “good research questions” in Box3.5, which clear and detail the tips for us to do a good research. It is very useful for me because I am a beginner in everything related to “research”, and I will enjoy reading the rest of this book.  

- Ah! First of all, thanks God, that, after dealing with a lot of philosophical and complicated viewpoints and sayings, for a relatively long time, at last we reached a book with many attractive statements and viewpoints about a great kind of research, which I like very much!

- Robson, in this book, talks about real world research as investigations concerning people in “real life” situations. He calls this kind of research as “flexible design” versus “fixed design”.

- In chapter 1, the author explains about real world enquiry, fixed and flexible designs, and action research, evaluation, and change.

- Fixed designs, relying on quantitative data as well as statistical generalization, usually are involved with a considerable amount of pre-specification about what and how the research is supposed to be done (Experiments and surveys are two examples of this kind of research). While, flexible design, relating to enquiry in the real world, usually doesn’t have much pre-specification, rather, is interactive and would be developed within the process of proceeding research.

- The author assesses the combination of these two methods, called as “mixed-method”, with taking advantage of using both quantitative and qualitative data, and using two or more methods, as a plausible strategy (I do agree with him about that).  

- One difference between real world research and pure research is the emphasis of the first one on “evaluation”. Based on this evaluation, researcher suggests a desirable change.

- In chapter 2, Robson explores scientific aspect of a research, and argues about its advantages. To do this, he discusses and compares positivism, post-positivism, relativists, critical approaches (especially feminism), constructivism, realism, and critical realism (as a third way, between positivism and relativism, and also as a supporter for the potential of social research to be known as an emancipator approach).

- In chapter 3, reviewing differences between fixed and flexible design, Robson, elaborates a set of steps needed in deciding on the focus and also on the questions of a research in a real world study. Furthermore, he describes the place of theories in carrying out an applied research.   

…………………………………………………………………………………………………………………………………………………………

- Although I agree with most of the ideas of the author about the advantages of real world research, I disagree with his viewpoints about science. In my opinion, by seeing that some of scientists use their science and innovations in the ways that are harmful to human being, we cannot conclude that all scientists do so. We cannot negate the humanitarian aid of thousands of scientists over the history of man, without which we might were suffering from enormously serious problems. If we accept that doing a collaborative research or evaluation in the limited size of an organization would add a great deal of values, how we can disregard the world-wide scope effects of a scientific theory or an invention.

I think even in doing a real world research, if a researcher can contribute to the construction of a theory and in this way combine her/his improvement suggestion to theory construction, she/he would be doing an excellent job (As Robson, himself, has confirmed it as a possible situation in many cases of doing flexible research- he says that although real world researchers, unlike positivist ones, don’t start their research with a theory, rather, may end up it with a theory gained from data gathered in a systematic manner-).

- In my opinion, “collaborative research” or “participatory action research” would be a convenient way for carrying out a real world enquiry and proposing a feasible change. One reason is because in this approach, the knowledge of insiders would be properly integrated with the expertise of outsiders in research and methodologies.

- I think social science theories without any help to practice, would not be of much value; not only because of having or not having effects on the lives of people, but also, because Practice has the potential of helping theory to be improved.

-  One thing, among many other new things, I learned in this course (thanks to Professor Olfman, for that). That is, necessarily each pure science research would not be of enough value. Among the papers we assigned to read and analyze in this course, I found some ones suffering from an acceptable level or degree of quality and plausibility. So, it is not reasonable to say, for example, that each quantitative or positivist research is better than an action research. As mentioned before, a qualitative research can be scientific, as well. Especially, where it takes advantage of making a link between real world research and current formal theories.

 

What a nice surprise Robson is. Clearly positioning himself as a realist in the Preface to The Second Edition, he makes no bones about his belief that the Positivist approach has failed and that the Social Constructionists are a little too loosey-goosey. Realism, at least as I understand it, appears to offer the best approach to producing research that can be used in practice, something that is hugely important to me at this stage of my development. The Realist explanation, which emphasizes "mechanism" within a context, is much more viable for explaining phenomenon that occurs in the real world than the Positivist constant conjunction approach.

Robson's "boxes" are nice little cheat sheets that should be useful for when I embark on my own real world research. In Chapter 3's "Developing Your Ideas" he gives some very good examples of how to create successful research. In Box 3.4 (page 56) he provides hints for both "successful" and "unsuccessful" research which is quite useful. "Successful" item 5 "real world value" and "unsuccessful" item 3 "motivation by publication, money or funding" resonated especially with me.

 The discussion of the "paradigm wars" on page 43 was reinforcement that some of the truculence between Positivists and Interpretivists that I have been sensing whilst reading extracurricular research is not my imagination. It is a little shocking for a freshman like myself to realize that these two camps don't like each other very much. However, if I ever do get those three letters behind my name, I'll probably take just as impassioned a position as the current incumbents ;-)

The first three chapters in Robson have helped my understanding of the vast number of philosophy of science approaches, but I'm still somewhat unsure of what hermeneutics is all about. I will say that these terms pop up in the IS literature consistently that I have read, so it is important to understand their meaning.

I found this quote rather interesting: "It would appear that no 'scientific revolution,' or Kuhn paradigm, has been developed by a scientist using the research-then-theory or Baconian strategy in his research." (p.150) One would almost wonder why we continue to do research-then-theory. In spite of that, Reynolds tries to present a balanced view by outlining which situations warrant one approach over the other to achieve better efficiency in research. The paper for this week from Dennis et. al. was a good match. I felt it was a good example of the theory-then-research model because the author's are proposing a new idea which potentially could shift how people think about media choices.

 Overall I found this to be a valuable overview of not only the various types of theory but how to test and create those theories. This would've been great to read before Godfrey as Reynolds summarizes Kuhn so well.