Because data quality, reliability, and validity is a huge issue in research, a researcher must carefully plan his or her survey instrument. Survey is defined broadly as a research strategy that describes the practical and tactical matters surrounding survey instruments. These instrument are mostly questionnaires and interviews.
Questions in Questionnaire exhibit the following properties:

• Simple language avoiding jargons and ambiguities
• Question frame of reference is clear
• Closed-ended option laden question
• Partially closed-ended questions option laden question with ’others’ as option
• Questions means the same thing to all respondents
• Rating numeric scale indicating the direction and strength of the response
• Likert rating scale to judge level of agreement to a statement

When designing questions, the following should be avoided:

• Loaded questions including nonneutral or emotional laden terms
• Leading questions swaying the responded to answer in a desired manner
• Double-barreled questions asking for more than one thing
• Questions in negative making understanding difficult
• Prestige bias
• Creating opinions
• Direct questions on sensitive topics (de Vaus, 2001; Jackson, 2008)

Participant observation is perhaps the most personally demanding and analytically difficult method of data collection to undertake. It requires us as researchers to spend a great deal of our time in surroundings that we may not be familiar with; to develop and sustain relationships with people with whom we have little personal affinity; to take a lot of notes on apparently mundane activities; to expose ourselves to incidental risks of the environment of interest; and to spend months of analysis after the fieldwork, analyzing field-notes and diaries. Yet, for those of us hoping to do qualitative research, it is known to be a most rewarding method, yielding interesting insights into participants’ social lives and relationships that far outweigh any benefits derivable from fixed design methods.

Robson distinguishes four different participant observer roles:

• Complete participant: the researcher employing this role hides their identity as an observer and attempts to engage fully in the activities of the group or organization under investigation. This method could be used to collect more accurate data by a researcher investigating, say, a racist or fascist organization. It advantages, notwithstanding, its clandestine approach has strongly been objected to in terms of being ethically indefensible.
• Participant as observer: the researcher adopts an overt role, making their presence and intentions known to the group. Some have raised doubts as to whether the observer will be able to establish the necessary level of rapport with the participants after revealing their identity. As Robson observed, it is important for the observer to get the trust of key member of the group.
• The marginal participant: the researcher is uninvolved and detached, and merely, passively records behavior at a distance (e.g., a researcher sitting in a classroom, making observations of pupils and their teacher).
• Observer as participant: here the researcher makes known their status as researcher but moves away from the idea of participation. This would usually call for relatively more formal observation (e.g., ownership and structure of a firm, rather than its internal practices and norms) than either informal observation or participation. Here, there is a higher possibility of misunderstanding as as the researcher and participants are not as bonded as is the case with other observational methods. Robson argues that it is still questionable whether the researcher can be said to be a complete non-participant since he/she becomes a meaningful member of the group throughout the duration of the research.

The part III describes various methods of data collection such as observation and interviewing using questionnaires, as well as administering tests.  Surveys are more like a research strategy than a tactic or specific method and it gives reassuring scientific ring of confidence. Also reliability and validity of survey data depend to a considerable extent on the technical proficiency of those running the survey. If the questions are incomprehensible or ambiguous, the exercise is obviously a waste of time and this is a problem of internal validity. Another problem, generalizability or external validity can be produced by the sampling fault.

Survey are almost carried out as part of a non-experimental fixed design and while this can be for any of the research purposes like exploratory , descriptive, explanatory or emancipatory, surveys are not well suited to carrying out exploratory work. Most of surveys include the use of questionnaire and there are three main ways in which this questionnaire is administered. These are self-completion, face-to-face interview, and telephone interview. In addition questionnaire needs pre-testing and the larger the sample, the lover the likely error in generalizing.

Interview is a one of research methods and it includes structured, semi-structured and unstructured interviews. While fully structured interview has predetermined questions with fixed wording, in a pre-set order semi-structured interview has also predetermined questions, but the order can be modified based upon the interviewer’s perception. Unstructured interviews are used for obtain interviewer’s general idea of interest and concern and it can be completely informal. Both of semi-structured and unstructured interviews are widely used in flexible, quantitative designs.

This part also explains various tests and scales to measure attitudes such as Likert scale, Thurstone scale, Guttman scale, and semantic differential scales. Critics of both Thurstone and Likert scale have pointed out that they may contain statements which concern a variety of dimensions relating to the attitude of concern. However the Guttman approach overcomes this complexity by seeking to develop a unidimensional scale. Last chapter of this part describes observational methods. The two polar extreme types are participant observation and structured observation. While participant observation is an essentially qualitative style and originally rooted in the work of anthropologists, structured observation is a quantitative style which has been used in a variety of disciplines. Also participant observation is a widely used method in flexible designs and structured observation is almost exclusively linked to fixed designs, of both experimental and non-experimental types.

I learned there are so many research methods for each appropriate research question to collect and analyze data. This part is very helpful to consider research method which fit into my research question.    

 

I found the third part of Robson textbook of a great reference to researchers once they reach to the stage of designing a questionnaire or surveys. Robson emphasizes in chapter 8 the various types of surveys and, advantages and disadvantages for each kind. Basically, survey can be defined as a questionnaire that consists of questions that need to be answered by sampling responders. Researchers need to drive Survey’s questions from the research questions or hypothesis they seeking to test. Questionnaire is like a piece of art that researchers draw. It has certain principles that need to be followed to get the best out of it. Interviews can be categorized into three categories; self-completion where responders fill the answers by themselves and the questionnaire is posted by mail, face-to-face where researchers ask questions to responders and fill the answers by themselves, and telephone interviews where researchers dial the responders, ask the questions, and record them. Robson gives a clear comparison among those different approaches in terms of costs, time, biases and other different critical criteria in page 237.

 

How can researcher carry out a survey? This question stops many beginner researchers including me. However, Robson answeres this question clearly in the third part of his textbook. There are 5 steps to carry out a survey; those steps are:

 

1. Initial design and planning: during this stage, researcher determines the purpose of his/her study, the questions he/she seeks to get answers for, the study population, and the sampling frame.
2. Designing the questionnaire: In this stage, researcher design the questions in a way that helps in achieving the goal of research and answering research questions
3. Pre-testing: In this stage, researcher takes constructive comments and thoughts from friends, colleagues, responders from the sample group after they read the questions to avoid unambiguous questions and finally implement those comments and suggestions.
4. Final design and planning data collection: This stage involves editing the final questionnaire after modifications.
5. Data analysis and reporting. Researcher gathers the returned responses and may re contact responders in case there is no response, if any, and finally the coding stage starts to help researcher organize, quantify, and analyze the collected data.

 

Robson also in chapter 9 gives explanation for the different types of interviews and he categorized them into three types, structured, semi- structured, and unstructed interviews.

Structured interviews have predetermined questions with fixed wording. As for the semi structured interviews have predetermined questions but the order of questions and its wording can be modified. For the unstructured interviews, the conversation between the researcher and responders about certain topic raises issues within the context of this particular topic. In chapter 10, Robson sheds the light on the different scales and tests (such as Likert, Guttman, and Semantic Differential approach) Moreover, Robson explains the steps in developing each kind of tests and he also provides different examples for those tests. I found this chapter of great value because it helps the reader to differentiate among those tests and to explore the advantages and disadvantages of each. For chapter 11 and 12, I would rather prefer commenting on them when I finish reading them.

 

After reading some few pages of (Kohli & Kettinger, 2004), I delved into the observational methods in Robson part III to improve my understanding of the article.
Observational Methods. Observational methods enable researchers to observe human or animal behaviour. Most studies starts using these method either to gain an initial understanding of the phenomenon in question or as a main research method. Observational methods could be classified broadly as formal and informal observation based on the degree of pre-structure in the observation exercise. Formal observation imposes a lot of structure and direction regarding what can be observed while informal observation does not. Researches that employ observational methods could be further be classified as following:
Nonparticipant observation research – where the researcher does not participant in the situation involving the participants.
Participant observation research – where the researcher is actively participate in the situation involving the participants. Here a researcher might play the role of an observer at the same time as a participant, though this dual role might be difficult.
Disguised observation research – where the participants are unaware that their behaviour are being observed. This method might involve ethical issues if the participants are human subjects. Here are some key points to note when using observational method in action research.
Collecting data. A participant observer observes the people in the research focused group while being involved. Data are recorded on the spot during the event using methods similar to that of interview like voice recording. The observer might include the following data: running descriptions, notes of recalls of forgotten materials, notes offerings on the interpretation of the situation, personal impressions and feelings, and reminders to look for additional information.
Before the data can be analyzed in a meaningful way, it must be coded using schemes. Common coding schemes include checklist and category systems. Checklist might be static – a tally sheet used to record attributes that does not change, or action – a tally sheet used to record the presence or absence of behaviours. Category systems utilised coding scheme to record what is being observed. It is advised that researchers should use an existing coding scheme when available. But if a new coding scheme is required, it should have the following properties: (1) focus on selected aspect of the inquiry, (2) be objective, (3) non context-dependent, (4) exhaustive, (5) mutually exclusive, and (6) easy to record.

In part 3, Robson discusses the reliability and validity of survey data. He mentions that they both depend to a considerable extent on the technical proficiency of those running the survey. Poorly designed questions may lead to the problem of internal validity, where the survey may not obtain valid information about the respondents and what they are thinking.

 

On the other hand, if the sampling is faulty, this produces a generalizability or external validity problem. Another type of external validity problem occurs if we seek to generalize from what people say in a survey to what they actually do.

 

Robson mentions that reliability is more straightforward. By presenting all respondents with the same standardized questions, carefully worded after piloting, it is possible to obtain high reliability of response.

 

Survey research has a unique advantage among social scientific methods: it is often possible to check the validity of survey data. Some of the respondents can be interviewed again, and the results of both interviews checked against each other. It has been found that the reliability of personal factual items, like age and income, is high. The reliability of attitude responses is harder to determine because a changed response can mean a changed attitude. The reliability of average responses is higher than the reliability of individual responses. Fortunately, the researcher is usually more interested in average, or group measures, than in individual responses.

 

One way of checking the validity of a measuring instrument is to use an outside criterion. One compares the results to some outside, presumably valid, criterion. Ordinarily, individual behavior is not checked because information about individuals is difficult to obtain, but group information is often available. This information can be used to test to some extent the validity of the survey sample and the responses.

                 

Just as structure follows strategy in management so is research design follows research strategy. Research strategy in turn follows the research question. A research purpose elicits the type of a research question. While designing research, it is important to pay close attention to the type of questions that are being asked. The link below shows a Research Question and Strategy Lookup Table useful for identifying candidate research strategy.

http://conversation.cgu.edu/is360f08/files/-1/2063/Research+Questio

Research strategy helps determine the kind of research design appropriate for the inquiry in question. Most variants of research design can categorised under fixed, flexible, or multiple design.

Fixed design entails pre-specifying the mechanism’s unit of analysis and data before data collection. Here there exist a substantial knowledge about the mechanisms surrounding the object or event of study. These mechanisms are relatively under the control of the researcher. Examples of fixed design are: experimental, longitudinal, cross-sectional designs, and non-experimental designs.

Experimental designs usually involve hypothesis testing. They have the following properties:

• One pre-test measure is conducted on the outcome variable
• Two groups are setup. One group, the experimental group, is exposed to an intervention while the other group, the control group, is not.
• Group members are selected randomly before pretest
• One test or intervention is administered.
• One post-test measure is conducted on the outcome variable (de Vaus 2001).

The analysis compares the outcome of the experimental group to that of the control group.
Longitudinal design shows the same properties as the experimental design but it only defines one experimental group and no control group. It performs the pre-test and post-test measurement of the outcome and compares the result.

Cross-sectional design defines groups but do not assign members randomly. It might or not administer a treatment. Data are collected at one point in time and analysed. Non-experimental design exhibits experimental properties but does not attempt to change the mechanisms ( i.e. situation or experience) of the object of study.

Flexible Design entails investing the mechanism of object of study without any pre-specification of unit of data. It evolve during data collection. This type is also referred to as qualitative strategy. Flexible design's common data collection methods include case studies, ethnographic studies, and grounded theory studies.

Other types of research strategies include evaluation research, action research, and design science research. These strategies focus on outcome. They try to establish the worth or value of some forms of intervention. They could adopt fixed, flexible, or hybrid design approach. Follow the link below to a lookup table showing where one might employ fixed, flexible, or hybrid type of research strategy in a design science research.

http://conversation.cgu.edu/is360f08/files/-1/2064/Design+Science+R

Having adopted a research strategy, conducting and presenting research findings in an open, unbiased manner is key. For fixed design, proof of trustworthiness or validity includes construct, content, internal, external, and statistical conclusion validity. While for flexible design, proof of trustworthiness can be achieved through: data triangulation, observer triangulation, methodological triangulation, theory triangulation (Denzin, 1978), peer debriefing and support, member checking, negative analysis, and audit trail.

The main subjects of this part are design strategies, how to organize fixed design and flexible design, and evaluation. Design concerns the various things which should be thought about and kept in mind when carrying out a research project. And the components of design are purpose, theory, research question, methods, and sampling strategy. All these aspects need to be inter-related and to be kept in balance. The figure “Framework for research design” shows the relationship between these aspects. The author recommends that when we have proposed study we need to judge design strategy appropriate for study characteristic. It can be fixed design or flexible design. The author also stated that while fixed design strategy calls for a tight pre-specification before you reach the main data collection stage and data are almost always in the form of number (hence this type is often referred to as a quantitative strategy), flexible design strategy evolves during data collection and data are typically non-numerical (hence this type is often referred to as qualitative strategy). 

Here is another suggestion by author for selecting the strategy. Flexible (qualitative) strategies are appropriate for exploratory work and non-experimental fixed strategies are appropriate for descriptive studies. Also experiments are appropriate for explanatory studies.

In Fixed design part, author stated that the relative weakness of fixed design is that they cannot capture the subtleties and complexities of individual human behavior. The variable which is a part of quantitative worldview is explained as property or characteristic of a person, thing, group or situation that can be measured in some way. Also fixed design strategy should holds validity and generalizability to establish value and trustworthiness, then you can persuade others by clarity and logically argued accounts. In addition objectivity and credibility also support value and trustworthiness of fixed design enquiry.

Fixed design has two type of enquiry such as experimental fixed design and non-experimental fixed design. Experimental fixed design includes the assignment of participants to different conditions, manipulation of variables, measurement of the effects, and the control of all other variables. Also this book explains various kinds of experimental designs.

On the contrary non-experimental fixed designs are differs from the experimental one in that the phenomena studied are not deliberately manipulated or changed by the researcher. Hence it includes relational designs, comparative designs, and longitudinal designs. In flexible designs case studies, ethnographic studies, and grounded theory studies are the three influential design traditions. The Box 6.1 explains the difference between three studies in focus, discipline origin, data collection, data analysis, and narrative form.

Especially author indicates that researcher qualities needed for flexible design research and also emphasized that the quality of a flexible design study depends to a great extent on the quality of the investigator. So this research style needs well-trained and experienced investigators. To establish trustworthiness in flexible design research, some aspects are suggested for validity such as description, interpretation, theory, prolonged involvement, triangulation, peer debriefing and support, member checking, negative case analysis, and audit trail.

In the step of evaluation, author explains that the purpose of an evaluation is to assess the effects and effectiveness of something, typically some innovation, intervention, policy, practice or service. It is often referred to as program evaluation and fixed or flexible designs can be used, with either qualitative or quantitative methods, or some combination of both types.

In Robson's chapter 5 (fixed design), he discusses different types of validity. As far as I know, there are three different types of validity of our concern: content, criterion-oriented, and construct.

Content validity is the representativeness or sampling adequacy of the content-- the substance, the matter, the topic-- of a measuring instrument. Content validity is guided by the question: Is the substance or content of this measure representative of the content of the property being measured? Content validation consists essentially in judgment. Alone or with others, one judges the representativeness of the items. One may ask: Does this item measure Property X?  

Another type of validity that is very similar to content validity is face validity. Face validity is not validity in the technical sense. It refers to what the test appears to measure. Trained or untrained individuals would look at the test and decide whether or not the test measures what it was supposed to measure. There is no quantification of the judgment for face validity. However, content validity is quantifiable through the use of agreement indices of judges' evaluations. One such index is Cohen's Kappa.

 

Predictive validity and concurrent validity may be considered together as criterion-oriented validation procedures. For predictive validity and concurrent validity, the pattern of a criterion-oriented study is familiar. The investigator is primarily interested in some criterion which he wishes to predict. He administers the test, obtains an independent criterion measure on the same subjects, and computes a correlation. If the criterion is obtained some time after the test is given, he is studying predictive validity. If the test score and criterion score are determined at essentially the same time, he is studying concurrent validity. Concurrent validity is studied when one test is proposed as a substitute for another, or a test is shown to correlate with some contemporary criterion.

 

The single greatest difficulty of criterion-oriented validation is the criterion. Obtaining criteria may be difficult. Foe example, which criterion can be used to validate a measure of teacher effectiveness? Who is to judge teacher effectiveness?

 

Construct validation is involved whenever a test is to be interpreted as a measure of some attribute or quality which is not "operationally defined." The problem faced by the investigator is, "What constructs account for variance in test performance?" Usually the construct validity calls for no new scientific approach.

 

They ask, for example, what proportion of the total test variance is accounted for by each of the constructs. In short, they seek to explain individual differences in test scores.

 

Internal validity is about finding out if the treatment actually caused the outcome. If a study can demonstrate this causal relationship between treatment and outcome, it is referred to as having internal validity. However, there are threats that might happen which make us mistakenly conclude that the treatment caused the outcome. One of such threats is labeled as "history" by Campbell and Stanley-- something which happens at the same time as the treatment. Campbell and Stanley suggested eight possible threats to internal validity which might be posed by extraneous variables. The eight threats are: 1. history, 2. testing, 3. Instrumentation, 4. Regression, 5. morality, 6. maturation, 7. selection, and 8. selection by maturation interaction.

 

Campbell and Stanley say that internal validity is the sine qua non of research design, but that the ideal design should be strong in both internal validity and external validity, even though they are frequently contradictory.

Deciding the appropriate research design is based on the questions that researchers are attempting to answer and their research purposes. Fixed design has special features that distinguish it from the flexible design. For the fixed design, researchers are aware of what they are exactly looking for. Robson suggested a framework for the research design that has 5 components (purposes, theory, research questions, methods, and sampling strategy). In fixed design (theory driven design), researchers make a decision about those 5 components before reaching  to the data collection stage in their research and the design can be judged of good quality when it is compatible with those 5 components. In contrast, this framework in flexible design is not decided in early research’s stages but it emerges and evolves during the research.

 

Having good design is not the only aspect in research. Yet, researchers need to give a great attention to fundamental issues (validity, generalizability, and reliability) for their researches. Also, as discussed by Robson, validity can be categorized into two categories (internal and external). Internal validity refers to the existence of casual relationship between the treatment and the outcomes of the research. While external validity can be defined as the extent to which the study’s findings can be generalized on settings other than the current study’s setting. Also, external validity is an alternative term which can be used to indicate generalizability as well. Reliability refers to the consistency of the measure. Robson also lists the threats for both internal and external validity for both kinds of research (Fixed and flexible). He didn’t only explain those threats but he also gave the strategies that can be followed to minimize the potential of having those threats. I found this part as a good source for researchers to keep in their mind what may cause such threats in order to avoid them.

 

Moreover, Robson gives a clear insight on various approaches that can be followed within the fixed and flexible designs. As for fixed design, the approaches that can be followed are true experiments, quasi experiments, and a single case experiment. Case studies, grounded theories studies, and ethnographic studies go under flexible design approaches. I found the second part in this textbook “real world research” is very interesting because it differentiated the various approaches and it also added to my knowledge other kinds of approaches which I wasn’t aware of them before reading this part.