A research question may be stimulated by theoretical concerns. You then seek an appropriate context, a research focus, in which to ask the question. Usually the general focus comes first. However, some ways of approaching the generation of research questions are more likely to result in successful and productive enquiries than others.

 

J. T. Campbell et al. have looked at these issues and they posit an idea that the selection of innovative research questions is not a single act. Significant research is a way of thinking, an attitude.

 

In my opinion, research design has two basic purposes: (1) to provide answers to research questions and (2) to control variance. Research designs are invented to enable researchers to answer research questions as validly, objectively, accurately, and economically as possible.

Design tells us, in a sense, what observations to make, how to make them, and how to analyze the observations. Strictly speaking, design does not "tell" us precisely what to do, but rather "suggests" the direction of observation-making and analysis. An adequate design "suggests," for example, how many observations should be made. A design also tells us which type of method used for the analysis. Finally, an adequate design outlines possible conclusions to be drawn from the analysis.

A lot of people associate being scientific with coducting studies that involve collecting hard quantitative data, which are then subject to statistical analyses. Based on this erroneous conception,  some researchers who prefer a "softer" qualitative approach to research are unwilling to be laabelled as scientists. But, as Robson advises, those following qualitative approaches to research need not shun the label "scientific".  What is important is the need for imbibing the scientific attitude of carrying out research systematically, skeptically, and ethically.

Robson also distinguishes between the standard view of sciences (positivism), realitivist approaches,  and current views of social research, making a case for critical realist approaches. Positivism is dead consequent on its inadequacies in describing or explaining phenomena. Relativism considers science to give but one of equally valid accounts. There is no external reality independent of human consciousness. Reality can only be construed in terms of the meanings and classifications which people attach to the world. As such, there can be no objective reality since different cultures and societies have different conceptual systems. But Robson argues that such a position is at odds with the world. This is mainly because it is hard to prove propositions true or false in the absence of objective measures of reality, given that proponents of these approaches see reality as represented through the subjective eyes of participants. Even with their current more influential form, variously known as constructivist, naturalistic, or interpretive, you can never be sure how well you have explained the relationship of interest.

Robson makes a case for the adoption of critical realism because it is a particularly apprpriate framework for doing real world studies.

Robson in the first part of his textbook “Real World Research” differentiates between different research’s designs (flexible and fixed designs). The fixed design yields quantitative results, while the flexible design generates both quantitative data and qualitative results. The researcher in fixed design should know what he is really looking for. In contrast, the researcher in the flexible design gathers information first, go in one direction, and then may start another new direction and this can be attributed to the fact that design develops and advances as the research keep on. Robson emphasizes on the need for researchers to be innovative. Means, they can follow a design or mix of those two designs if they fit the purposes of their research. Robson also addresses the researcher’s responsibilities in the first chapter. Obviously, Researchers’ responsibilities are not only restricted to provide an understanding for what goes on, but also extend to communicate the findings of research to people who are concerned and provide possible ways to make desirable change takes place.

 

Robson explains a problematic issue which affects the public’s image to scientists. The problem starts when scientists follow softer qualitative styles. Robson emphasizes on the importance of having scientific attitude in the research for those who follow these softer styles. The scientific attitude gives the research more value since it calls for running the research systematically, sceptically, and ethically. Robson also defines the closed and open systems. He explains the nature of those two systems. In the closed systems, when researcher explains something he can also predict it. Additionally, all aspects are under the researcher’s control in close systems. On the other hand, it is hard for researcher to have control over aspects in the open system. Moreover, explanation comes after events occur and researchers can not predict them. 

Robson describes how to enquire real world, what are possible approaches to social research, what is developing way of our idea, and our attitude against research through chapter 1 to chapter 3.  He introduced mixed-method designs as well as Fixed and Flexible designs in this book. While examples of fixed designs are experiments and surveys which have reliance on quantitative data and statistical generalization flexible designs form and arise from a variety of theoretical positions typically make substantial use of methods which result in qualitative data. The comparison between fixed designs and flexible designs (p.46) is a good example to understand and distinguish two designs. Also he added mixed-method designs which can be considerable advantage and make use of two or more methods as well as quantitative and qualitative data.

He insists that for real world research it is valuable to have scientific attitude and research is carried out systematically, skeptically and ethically. He introduces relativist approaches with Feyerabend assertion and two current view of social research, Post-positivism and Constructivism. Also he added Feminist and other emancipator approaches. In chapter 3 he gives a way to develop our own idea to apply research. He enumerates each step to develop our idea such as deciding on the focus, deciding on the research questions, the place of theory, and ethical considerations.

From these chapters I could have insight how we can create topic and idea from real world and how to develop this idea through procedures. Also I think I need to have ethical attitude for vulnerable groups and people and remember they are human rather than object for our research.

I feel my research questions are gradually increasing by reading books and articles in this class.

Robson (2002) opens up the real world research by asking us to be professional in our research. Being professional or scientific is beneficial in many ways, to the clients, current researchers, or successor researchers. While doing a real world research, a researcher keeps a log or diary, containing reading reference documentations, relevant ideas, actionable to-do list, stocks, and reflective thoughts about the research process. Real world research emphasises problem solving rather than just gaining knowledge which is also the core concern of design science research. A real world researcher identifies a problem of significant effect, applies a method that fit the research question from a universe of methods, to elicit robust results where changes are feasible, within time and budget constraints.

Problem solving as real world research. In solving real world problems, five approaches are identified: (1) researcher dominated quest, (2) researcher-client bridge, (3) researcher-client equality, (4) client-researcher exploration, and (5) client dominated quest. Researcher dominated quest. The researcher conducts pure or theoretical research. The application of the research result is not a necessary outcome required by the research. Researcher-client bridge. Here, the application of the research result is a necessary outcome. Researchers attempt to communicate their findings and possible applications widely so that it can be implemented. Researcher-client equality. The researcher and the client jointly develop the research problem. The researcher carryon the quest and produce a result and instantiation of the result. Here the implementation of research’s result is an outcome. Client-researcher exploration. The clients develop the research problem.  They seek researchers who advise and recommend based on their knowledge of the field. Here, the implementation of research recommendations is an outcome. Client dominated quest. The clients develop the research problem and seek the guidance of researchers regarding ‘best current knowledge in the problem area. The interpretation of findings is heavily dependent on personal experience and ‘common sense’. Here, the implementation of the research line of action as an outcome. Whichever approach researchers take in solving a problem, they need to respond to real world research with a ‘scientific attitude’. This attitude is defined by being systematic (i.e. specifying explicitly what observations are made, the temporal context surrounding these observations, and the role of the observer), sceptical (i.e. subjecting ideas, observation, and conclusion to disconfirmation scutiny), and ethical (i.e. being guided by code of conduct of research that ensures the concerns of the respondents or target population are protected).

Real world research epistemological traditions. In pursue of solving a real world problem, six epistemology traditions are identified namely standard positivism,  relativism,  post-positivism and constructivism,  emancipator, realism,  and critical realism. Since my previous posts have explained some of these traditions, I would skip quickly to realist and critical realist traditions.

Realist Tradition. A realist holds that knowledge is a social and historical product. And that the real world is complex and stratified. Laws are characteristic pattern of activity or mechanism. The outcome of an action follows from mechanisms acting in particular contexts. Here, the mechanism or structure must first be activated, prevented from interference, then an applied action produces the conjunction relationship between the dependent and independent variable. Because knowledge is a social and historical product, realist believes it is impossible to exactly replicate studies. Thus, failure to replicate does not necessarily falsify the theory. But critics maintain that a finding is not secure if it cannot be replicate.

Critical realism. A critical realist embraces a pragmatic combination of compatible fundamentals of both qualitative (realist) and quantitative (post-positivist) traditions which are:

• Value-ladenness of enquiry
• Theory-ladennes of facts
• Reality is multiple, complex, and stratified
• Underdetermination of theory by facts

This tradition appears to be a better fit for the real world research.

Developing your ideas
Having considered epistemological traditions and their influence on research efforts and ideas, I will summarise sources of research idea and the place of theory in research. It has been found that successful research ideas comes from having good contacts both in the field and with colleagues. It also comes from convengence of activies, intuition redarding a real world value, and a concern for theoretical understanding.
Place of theory. Theories are crutial to research because they provide a base for making claims. They provide an assurance that one’s research is in tune with researchers who have attempted to understand the phenomenon in question. Critical realist embraces having foundational theories. Foundational theories  helps in formulating conceptual framework. Conceptual framework is a concept maps or diagrams showing what is going and why in a problem domain.  It helps define enabling or disabling mechanisms operating in a particular condition. It also help in defining the problem boundary or domains. 

According to Robson, after we have decided on the focus of our research and developed the research questions, then the next thing is for us to consider the research strategy, the subject of Part II. He distinguishes between the strategy and the tactics we adopt when carrying out a research.

• Strategy refers to the general broad orientation taken in addressing the research questions, that is, am I going to use a flexible or fixed design. Here, design is the approach used in turning research questions into projects.
• Tactics are specific methods of investigation, for example surveys, questionnaires, interviews, tests and observation techniques.

The general principle in choosing a research strategy or design is that the methods and techniques employed must be appropriate for the research questions we want to answer. Robson recommends that a framework for research design guide us both when are considering which research design to adopt and when we are carrying out the investigation. Essentially, the framework advises that we keep in mind the purpose of the research, its theoretical/conceptual framework, the research questions we seek to provide answers to, the methods to be used for data collection, and the sampling strategy.

Choosing an appropriate research design (flexible vs. fixed) can be difficult. And to help us decide, Robson suggests our trying to answer these questions:

• Is a fixed or flexible design strategy appropriate? While a design cannot be both fixed and flexible at the same time, a combined strategy design, when appropriate can be used. A fixed design could start out with an initial flexible design stage for the exploratory purpose of surfacing candidate mechanisms operating in the situation, contexts where they are likely to operate, and the characteristics of the participants best targeted. Thereafter, the second stage will incorporate the fixed design study.
• Is your proposed study an evaluation? Trying to establish the worthwhile value of something such as an intervention, innovation or service could be approached using a fixed or flexible strategy. According to Robson, if the focus is on outcomes, a fixed design is probably indicated; if it is on processes, then a flexible design is probably more desirable.
• Do you wish to carry out action research? If action is central to your concerns and direct participation in the research by yourself and/or others is likely, coupled with any intention to initiate change, then a flexible design is probably indicated.
• If you opt for a fixed design strategy which type is most appropriate? You still have to choose between two broad traditions of fixed designs: experimental and non-experimental designs.
• If you opt for a flexible design strategy which type is most appropriate? You still have to choose from a wide range of different flexible design traditions and disciplines, three of which are particularly relevant to real world research: case studies, ethnographic studies, and grounded theories studies.

 

How can a flexible design researcher persuade his/her audience the research findings are to be trusted? Validity and reliability have for long been common measures of quality and absence of bias in fixed quantitative research designs, but they are only now being given serious consideratios in flexible research design. The approach was for qualitative researchers to argue that the quality of research in fixed and flexible research paradigms be judged by each paradigm’s own terms. Thus, while the terms reliability and validity were considered essential criteria for quality in quantitative paradigms, proponents of flexible qualitative designs preferred the terms credibility, transferability, dependability, and confirmability.

But as Robson has argued, reliability and validity are both of which are essential ingredients to rigor in scientific studies; and to state that they are not pertinent to qualitative research is to cast qualitative research as lacking validity, reliability, and consequently scientific rigor. A way forward is for qualitative flexible design researchers to dwell not on semantic issues but on ways to operationalize these terms to suit the conditions and circumstances of flexible qualitative designs.

Threats to validity With respect to validity of a flexible design, Robson suggests that we start by considering what constitutes good quality research and the possible threats to achieving it. Threats to validity can be classified based on the kinds of understanding involved in qualitative research, i.e. description, interpretation, and theory.

Description: The main threat to a valid description of what the researcher has observed is inaccurate or incomplete data. He suggests that audio- and video-taping should be carried out whenever feasible.

Interpretation:  The main threat to a valid interpretation lies in the researcher imposing a framework or meaning on the research without allowing the meaning to emerge from what he/she learned during his/her involvement with the research setting. To ensure validity of interpretation, conclusions and findings must include a demonstration of how interpretations were reached.

Theory: The main threat to a valid theory lies in not considering alternative explanations or understandings of the phenomena being studied. This can be reduced by actively seeking data and evidence that are not in consonance with one’s theory.

Another typology divides threats to validity into reactivity, researcher biases, and respondent biases. Reactivity refers to the effects the researcher’s presence has on the research setting, particularly the way in which the presence of the researcher can influence the behavior of the people involved. Respondent bias ranges from obstructions and withholding of information to giving answers that the respondent judges to be what the researcher wants. Research bias arises from preconceptions and assumptions that influence how the researcher is conducting the research—the persons selected, the kinds of questions asked, the selection of data for analysis, and the like. These threats, though present in all research involving people, are more so in flexible research designs. Robson presents the following as commonly used strategies for dealing with them in flexible designs:

Prolonged involvement—a relatively prolonged involvement reduces reactivity and respondent bias, but may increase researcher bias. The so-called the going native threat makes it difficult for the researcher to maintain the researcher role for an extended period of time.

Triangulation—this involves the use of multiple sources to enhance the rigor of the research. It could be on the basis of data, observers, methodologies (quantitative and qualitative), and theories. Triangulation reduces all three threats (reactivity, respondent bias, and researcher bias).

Peer briefing and support—Flexible research designs can be tedious, demanding, and stressful. Peer groups help the researcher deal with these problems as well as the concomitant researcher bias. They, however, do not have any effect on reactivity and respondent bias

Member checking—this involves the researcher going back to the respondents or presenting them with material such as transcripts, accounts, and interpretations made. Done well, it should reduce reactivity, researcher bias, and respondent bias.

Negative case analysis—the researcher actively searches for instances that will disconfirm his/her theory. This can be done using data already collected or additional data collected for this express purpose. Either way, the researcher has a ethical responsibility to do it honestly and thoroughly. It reduces researcher bias but has no effect on reactivity and respondent bias.

Audit trail, as a strategy to forestall threats to validity, requires that the researcher keeps a full record of research activities, including raw data, research journal, as well as details of coding and data analysis. Audit trail reduces researcher bias.

Unlike fixed research designs where threats to validity are dealt with in advance as part of the design process, flexible designs deal with threats to validity after the research process has already begun, using data generated in the process.

Reliability in flexible designs is more or less a consequence of the validity the study. In other words, once validity of the research is assured, reliability will generally follow. But Robson points out a caveat: “reliability involves not only being thorough, careful, and honest in carrying out the research, but also being able to show others that you have been.” 

 

Deciding the appropriate research design is based on the questions that researchers are attempting to answer and their research purposes. Fixed design has special features that distinguish it from the flexible design. For the fixed design, researchers are aware of what they are exactly looking for. Robson suggested a framework for the research design that has 5 components (purposes, theory, research questions, methods, and sampling strategy). In fixed design (theory driven design), researchers make a decision about those 5 components before reaching  to the data collection stage in their research and the design can be judged of good quality when it is compatible with those 5 components. In contrast, this framework in flexible design is not decided in early research’s stages but it emerges and evolves during the research.

 

Having good design is not the only aspect in research. Yet, researchers need to give a great attention to fundamental issues (validity, generalizability, and reliability) for their researches. Also, as discussed by Robson, validity can be categorized into two categories (internal and external). Internal validity refers to the existence of casual relationship between the treatment and the outcomes of the research. While external validity can be defined as the extent to which the study’s findings can be generalized on settings other than the current study’s setting. Also, external validity is an alternative term which can be used to indicate generalizability as well. Reliability refers to the consistency of the measure. Robson also lists the threats for both internal and external validity for both kinds of research (Fixed and flexible). He didn’t only explain those threats but he also gave the strategies that can be followed to minimize the potential of having those threats. I found this part as a good source for researchers to keep in their mind what may cause such threats in order to avoid them.

 

Moreover, Robson gives a clear insight on various approaches that can be followed within the fixed and flexible designs. As for fixed design, the approaches that can be followed are true experiments, quasi experiments, and a single case experiment. Case studies, grounded theories studies, and ethnographic studies go under flexible design approaches. I found the second part in this textbook “real world research” is very interesting because it differentiated the various approaches and it also added to my knowledge other kinds of approaches which I wasn’t aware of them before reading this part.

In Robson's chapter 5 (fixed design), he discusses different types of validity. As far as I know, there are three different types of validity of our concern: content, criterion-oriented, and construct.

Content validity is the representativeness or sampling adequacy of the content-- the substance, the matter, the topic-- of a measuring instrument. Content validity is guided by the question: Is the substance or content of this measure representative of the content of the property being measured? Content validation consists essentially in judgment. Alone or with others, one judges the representativeness of the items. One may ask: Does this item measure Property X?  

Another type of validity that is very similar to content validity is face validity. Face validity is not validity in the technical sense. It refers to what the test appears to measure. Trained or untrained individuals would look at the test and decide whether or not the test measures what it was supposed to measure. There is no quantification of the judgment for face validity. However, content validity is quantifiable through the use of agreement indices of judges' evaluations. One such index is Cohen's Kappa.

 

Predictive validity and concurrent validity may be considered together as criterion-oriented validation procedures. For predictive validity and concurrent validity, the pattern of a criterion-oriented study is familiar. The investigator is primarily interested in some criterion which he wishes to predict. He administers the test, obtains an independent criterion measure on the same subjects, and computes a correlation. If the criterion is obtained some time after the test is given, he is studying predictive validity. If the test score and criterion score are determined at essentially the same time, he is studying concurrent validity. Concurrent validity is studied when one test is proposed as a substitute for another, or a test is shown to correlate with some contemporary criterion.

 

The single greatest difficulty of criterion-oriented validation is the criterion. Obtaining criteria may be difficult. Foe example, which criterion can be used to validate a measure of teacher effectiveness? Who is to judge teacher effectiveness?

 

Construct validation is involved whenever a test is to be interpreted as a measure of some attribute or quality which is not "operationally defined." The problem faced by the investigator is, "What constructs account for variance in test performance?" Usually the construct validity calls for no new scientific approach.

 

They ask, for example, what proportion of the total test variance is accounted for by each of the constructs. In short, they seek to explain individual differences in test scores.

 

Internal validity is about finding out if the treatment actually caused the outcome. If a study can demonstrate this causal relationship between treatment and outcome, it is referred to as having internal validity. However, there are threats that might happen which make us mistakenly conclude that the treatment caused the outcome. One of such threats is labeled as "history" by Campbell and Stanley-- something which happens at the same time as the treatment. Campbell and Stanley suggested eight possible threats to internal validity which might be posed by extraneous variables. The eight threats are: 1. history, 2. testing, 3. Instrumentation, 4. Regression, 5. morality, 6. maturation, 7. selection, and 8. selection by maturation interaction.

 

Campbell and Stanley say that internal validity is the sine qua non of research design, but that the ideal design should be strong in both internal validity and external validity, even though they are frequently contradictory.

The main subjects of this part are design strategies, how to organize fixed design and flexible design, and evaluation. Design concerns the various things which should be thought about and kept in mind when carrying out a research project. And the components of design are purpose, theory, research question, methods, and sampling strategy. All these aspects need to be inter-related and to be kept in balance. The figure “Framework for research design” shows the relationship between these aspects. The author recommends that when we have proposed study we need to judge design strategy appropriate for study characteristic. It can be fixed design or flexible design. The author also stated that while fixed design strategy calls for a tight pre-specification before you reach the main data collection stage and data are almost always in the form of number (hence this type is often referred to as a quantitative strategy), flexible design strategy evolves during data collection and data are typically non-numerical (hence this type is often referred to as qualitative strategy). 

Here is another suggestion by author for selecting the strategy. Flexible (qualitative) strategies are appropriate for exploratory work and non-experimental fixed strategies are appropriate for descriptive studies. Also experiments are appropriate for explanatory studies.

In Fixed design part, author stated that the relative weakness of fixed design is that they cannot capture the subtleties and complexities of individual human behavior. The variable which is a part of quantitative worldview is explained as property or characteristic of a person, thing, group or situation that can be measured in some way. Also fixed design strategy should holds validity and generalizability to establish value and trustworthiness, then you can persuade others by clarity and logically argued accounts. In addition objectivity and credibility also support value and trustworthiness of fixed design enquiry.

Fixed design has two type of enquiry such as experimental fixed design and non-experimental fixed design. Experimental fixed design includes the assignment of participants to different conditions, manipulation of variables, measurement of the effects, and the control of all other variables. Also this book explains various kinds of experimental designs.

On the contrary non-experimental fixed designs are differs from the experimental one in that the phenomena studied are not deliberately manipulated or changed by the researcher. Hence it includes relational designs, comparative designs, and longitudinal designs. In flexible designs case studies, ethnographic studies, and grounded theory studies are the three influential design traditions. The Box 6.1 explains the difference between three studies in focus, discipline origin, data collection, data analysis, and narrative form.

Especially author indicates that researcher qualities needed for flexible design research and also emphasized that the quality of a flexible design study depends to a great extent on the quality of the investigator. So this research style needs well-trained and experienced investigators. To establish trustworthiness in flexible design research, some aspects are suggested for validity such as description, interpretation, theory, prolonged involvement, triangulation, peer debriefing and support, member checking, negative case analysis, and audit trail.

In the step of evaluation, author explains that the purpose of an evaluation is to assess the effects and effectiveness of something, typically some innovation, intervention, policy, practice or service. It is often referred to as program evaluation and fixed or flexible designs can be used, with either qualitative or quantitative methods, or some combination of both types.

Just as structure follows strategy in management so is research design follows research strategy. Research strategy in turn follows the research question. A research purpose elicits the type of a research question. While designing research, it is important to pay close attention to the type of questions that are being asked. The link below shows a Research Question and Strategy Lookup Table useful for identifying candidate research strategy.

http://conversation.cgu.edu/is360f08/files/-1/2063/Research+Questio

Research strategy helps determine the kind of research design appropriate for the inquiry in question. Most variants of research design can categorised under fixed, flexible, or multiple design.

Fixed design entails pre-specifying the mechanism’s unit of analysis and data before data collection. Here there exist a substantial knowledge about the mechanisms surrounding the object or event of study. These mechanisms are relatively under the control of the researcher. Examples of fixed design are: experimental, longitudinal, cross-sectional designs, and non-experimental designs.

Experimental designs usually involve hypothesis testing. They have the following properties:

• One pre-test measure is conducted on the outcome variable
• Two groups are setup. One group, the experimental group, is exposed to an intervention while the other group, the control group, is not.
• Group members are selected randomly before pretest
• One test or intervention is administered.
• One post-test measure is conducted on the outcome variable (de Vaus 2001).

The analysis compares the outcome of the experimental group to that of the control group.
Longitudinal design shows the same properties as the experimental design but it only defines one experimental group and no control group. It performs the pre-test and post-test measurement of the outcome and compares the result.

Cross-sectional design defines groups but do not assign members randomly. It might or not administer a treatment. Data are collected at one point in time and analysed. Non-experimental design exhibits experimental properties but does not attempt to change the mechanisms ( i.e. situation or experience) of the object of study.

Flexible Design entails investing the mechanism of object of study without any pre-specification of unit of data. It evolve during data collection. This type is also referred to as qualitative strategy. Flexible design's common data collection methods include case studies, ethnographic studies, and grounded theory studies.

Other types of research strategies include evaluation research, action research, and design science research. These strategies focus on outcome. They try to establish the worth or value of some forms of intervention. They could adopt fixed, flexible, or hybrid design approach. Follow the link below to a lookup table showing where one might employ fixed, flexible, or hybrid type of research strategy in a design science research.

http://conversation.cgu.edu/is360f08/files/-1/2064/Design+Science+R

Having adopted a research strategy, conducting and presenting research findings in an open, unbiased manner is key. For fixed design, proof of trustworthiness or validity includes construct, content, internal, external, and statistical conclusion validity. While for flexible design, proof of trustworthiness can be achieved through: data triangulation, observer triangulation, methodological triangulation, theory triangulation (Denzin, 1978), peer debriefing and support, member checking, negative analysis, and audit trail.

In part 3, Robson discusses the reliability and validity of survey data. He mentions that they both depend to a considerable extent on the technical proficiency of those running the survey. Poorly designed questions may lead to the problem of internal validity, where the survey may not obtain valid information about the respondents and what they are thinking.

 

On the other hand, if the sampling is faulty, this produces a generalizability or external validity problem. Another type of external validity problem occurs if we seek to generalize from what people say in a survey to what they actually do.

 

Robson mentions that reliability is more straightforward. By presenting all respondents with the same standardized questions, carefully worded after piloting, it is possible to obtain high reliability of response.

 

Survey research has a unique advantage among social scientific methods: it is often possible to check the validity of survey data. Some of the respondents can be interviewed again, and the results of both interviews checked against each other. It has been found that the reliability of personal factual items, like age and income, is high. The reliability of attitude responses is harder to determine because a changed response can mean a changed attitude. The reliability of average responses is higher than the reliability of individual responses. Fortunately, the researcher is usually more interested in average, or group measures, than in individual responses.

 

One way of checking the validity of a measuring instrument is to use an outside criterion. One compares the results to some outside, presumably valid, criterion. Ordinarily, individual behavior is not checked because information about individuals is difficult to obtain, but group information is often available. This information can be used to test to some extent the validity of the survey sample and the responses.

                 

After reading some few pages of (Kohli & Kettinger, 2004), I delved into the observational methods in Robson part III to improve my understanding of the article.
Observational Methods. Observational methods enable researchers to observe human or animal behaviour. Most studies starts using these method either to gain an initial understanding of the phenomenon in question or as a main research method. Observational methods could be classified broadly as formal and informal observation based on the degree of pre-structure in the observation exercise. Formal observation imposes a lot of structure and direction regarding what can be observed while informal observation does not. Researches that employ observational methods could be further be classified as following:
Nonparticipant observation research – where the researcher does not participant in the situation involving the participants.
Participant observation research – where the researcher is actively participate in the situation involving the participants. Here a researcher might play the role of an observer at the same time as a participant, though this dual role might be difficult.
Disguised observation research – where the participants are unaware that their behaviour are being observed. This method might involve ethical issues if the participants are human subjects. Here are some key points to note when using observational method in action research.
Collecting data. A participant observer observes the people in the research focused group while being involved. Data are recorded on the spot during the event using methods similar to that of interview like voice recording. The observer might include the following data: running descriptions, notes of recalls of forgotten materials, notes offerings on the interpretation of the situation, personal impressions and feelings, and reminders to look for additional information.
Before the data can be analyzed in a meaningful way, it must be coded using schemes. Common coding schemes include checklist and category systems. Checklist might be static – a tally sheet used to record attributes that does not change, or action – a tally sheet used to record the presence or absence of behaviours. Category systems utilised coding scheme to record what is being observed. It is advised that researchers should use an existing coding scheme when available. But if a new coding scheme is required, it should have the following properties: (1) focus on selected aspect of the inquiry, (2) be objective, (3) non context-dependent, (4) exhaustive, (5) mutually exclusive, and (6) easy to record.

 

I found the third part of Robson textbook of a great reference to researchers once they reach to the stage of designing a questionnaire or surveys. Robson emphasizes in chapter 8 the various types of surveys and, advantages and disadvantages for each kind. Basically, survey can be defined as a questionnaire that consists of questions that need to be answered by sampling responders. Researchers need to drive Survey’s questions from the research questions or hypothesis they seeking to test. Questionnaire is like a piece of art that researchers draw. It has certain principles that need to be followed to get the best out of it. Interviews can be categorized into three categories; self-completion where responders fill the answers by themselves and the questionnaire is posted by mail, face-to-face where researchers ask questions to responders and fill the answers by themselves, and telephone interviews where researchers dial the responders, ask the questions, and record them. Robson gives a clear comparison among those different approaches in terms of costs, time, biases and other different critical criteria in page 237.

 

How can researcher carry out a survey? This question stops many beginner researchers including me. However, Robson answeres this question clearly in the third part of his textbook. There are 5 steps to carry out a survey; those steps are:

 

1. Initial design and planning: during this stage, researcher determines the purpose of his/her study, the questions he/she seeks to get answers for, the study population, and the sampling frame.
2. Designing the questionnaire: In this stage, researcher design the questions in a way that helps in achieving the goal of research and answering research questions
3. Pre-testing: In this stage, researcher takes constructive comments and thoughts from friends, colleagues, responders from the sample group after they read the questions to avoid unambiguous questions and finally implement those comments and suggestions.
4. Final design and planning data collection: This stage involves editing the final questionnaire after modifications.
5. Data analysis and reporting. Researcher gathers the returned responses and may re contact responders in case there is no response, if any, and finally the coding stage starts to help researcher organize, quantify, and analyze the collected data.

 

Robson also in chapter 9 gives explanation for the different types of interviews and he categorized them into three types, structured, semi- structured, and unstructed interviews.

Structured interviews have predetermined questions with fixed wording. As for the semi structured interviews have predetermined questions but the order of questions and its wording can be modified. For the unstructured interviews, the conversation between the researcher and responders about certain topic raises issues within the context of this particular topic. In chapter 10, Robson sheds the light on the different scales and tests (such as Likert, Guttman, and Semantic Differential approach) Moreover, Robson explains the steps in developing each kind of tests and he also provides different examples for those tests. I found this chapter of great value because it helps the reader to differentiate among those tests and to explore the advantages and disadvantages of each. For chapter 11 and 12, I would rather prefer commenting on them when I finish reading them.

 

The part III describes various methods of data collection such as observation and interviewing using questionnaires, as well as administering tests.  Surveys are more like a research strategy than a tactic or specific method and it gives reassuring scientific ring of confidence. Also reliability and validity of survey data depend to a considerable extent on the technical proficiency of those running the survey. If the questions are incomprehensible or ambiguous, the exercise is obviously a waste of time and this is a problem of internal validity. Another problem, generalizability or external validity can be produced by the sampling fault.

Survey are almost carried out as part of a non-experimental fixed design and while this can be for any of the research purposes like exploratory , descriptive, explanatory or emancipatory, surveys are not well suited to carrying out exploratory work. Most of surveys include the use of questionnaire and there are three main ways in which this questionnaire is administered. These are self-completion, face-to-face interview, and telephone interview. In addition questionnaire needs pre-testing and the larger the sample, the lover the likely error in generalizing.

Interview is a one of research methods and it includes structured, semi-structured and unstructured interviews. While fully structured interview has predetermined questions with fixed wording, in a pre-set order semi-structured interview has also predetermined questions, but the order can be modified based upon the interviewer’s perception. Unstructured interviews are used for obtain interviewer’s general idea of interest and concern and it can be completely informal. Both of semi-structured and unstructured interviews are widely used in flexible, quantitative designs.

This part also explains various tests and scales to measure attitudes such as Likert scale, Thurstone scale, Guttman scale, and semantic differential scales. Critics of both Thurstone and Likert scale have pointed out that they may contain statements which concern a variety of dimensions relating to the attitude of concern. However the Guttman approach overcomes this complexity by seeking to develop a unidimensional scale. Last chapter of this part describes observational methods. The two polar extreme types are participant observation and structured observation. While participant observation is an essentially qualitative style and originally rooted in the work of anthropologists, structured observation is a quantitative style which has been used in a variety of disciplines. Also participant observation is a widely used method in flexible designs and structured observation is almost exclusively linked to fixed designs, of both experimental and non-experimental types.

I learned there are so many research methods for each appropriate research question to collect and analyze data. This part is very helpful to consider research method which fit into my research question.    

Participant observation is perhaps the most personally demanding and analytically difficult method of data collection to undertake. It requires us as researchers to spend a great deal of our time in surroundings that we may not be familiar with; to develop and sustain relationships with people with whom we have little personal affinity; to take a lot of notes on apparently mundane activities; to expose ourselves to incidental risks of the environment of interest; and to spend months of analysis after the fieldwork, analyzing field-notes and diaries. Yet, for those of us hoping to do qualitative research, it is known to be a most rewarding method, yielding interesting insights into participants’ social lives and relationships that far outweigh any benefits derivable from fixed design methods.

Robson distinguishes four different participant observer roles:

• Complete participant: the researcher employing this role hides their identity as an observer and attempts to engage fully in the activities of the group or organization under investigation. This method could be used to collect more accurate data by a researcher investigating, say, a racist or fascist organization. It advantages, notwithstanding, its clandestine approach has strongly been objected to in terms of being ethically indefensible.
• Participant as observer: the researcher adopts an overt role, making their presence and intentions known to the group. Some have raised doubts as to whether the observer will be able to establish the necessary level of rapport with the participants after revealing their identity. As Robson observed, it is important for the observer to get the trust of key member of the group.
• The marginal participant: the researcher is uninvolved and detached, and merely, passively records behavior at a distance (e.g., a researcher sitting in a classroom, making observations of pupils and their teacher).
• Observer as participant: here the researcher makes known their status as researcher but moves away from the idea of participation. This would usually call for relatively more formal observation (e.g., ownership and structure of a firm, rather than its internal practices and norms) than either informal observation or participation. Here, there is a higher possibility of misunderstanding as as the researcher and participants are not as bonded as is the case with other observational methods. Robson argues that it is still questionable whether the researcher can be said to be a complete non-participant since he/she becomes a meaningful member of the group throughout the duration of the research.

Because data quality, reliability, and validity is a huge issue in research, a researcher must carefully plan his or her survey instrument. Survey is defined broadly as a research strategy that describes the practical and tactical matters surrounding survey instruments. These instrument are mostly questionnaires and interviews.
Questions in Questionnaire exhibit the following properties:

• Simple language avoiding jargons and ambiguities
• Question frame of reference is clear
• Closed-ended option laden question
• Partially closed-ended questions option laden question with ’others’ as option
• Questions means the same thing to all respondents
• Rating numeric scale indicating the direction and strength of the response
• Likert rating scale to judge level of agreement to a statement

When designing questions, the following should be avoided:

• Loaded questions including nonneutral or emotional laden terms
• Leading questions swaying the responded to answer in a desired manner
• Double-barreled questions asking for more than one thing
• Questions in negative making understanding difficult
• Prestige bias
• Creating opinions
• Direct questions on sensitive topics (de Vaus, 2001; Jackson, 2008)