Information-Gathering Techniques

The next stage is the determination of the appropriate information-gathering techniques, including several steps:

• Select a general methodological approach
  
• Determine what sources of data would provide the information needed and assess the feasibility of the alternatives.
  
• Select data collection techniques that would gather the desired information from the identified sources.
  
• Develop a design matrix.

After the evaluation questions have been formulated, the most appropriate methods for obtaining answers must be chosen. In determining what approach to use, some initial questions need to be answered.

First, is it better to do case studies, exploring the experiences of a small number of participants in depth or is it better to use a survey approach? In the latter case, do you need to survey all participants or can you select a sample? Do you want to look only at what happens to project participants or do you want to compare the experiences of participants with those of some appropriately selected comparison group of non-participants?

How you answer some of these questions will affect the kinds of conclusions you can draw from your study. Rigorous, ‘controlled’ designs are not always needed for Formative Evaluations. But Summative Evaluations, or Impact Assessments, may gain a great deal from being based on experimental or quasi-experimental designs.

Next you need to determine the kinds of data you want to use. Some alternatives are listed in ‘Kinds of Data’. Which one or ones to use depends on a number of factors, including the questions, the timeline and the re-sources available. Another factor to take into account is the technical skill level of the evaluator or evaluation team. Some of the techniques require more skills than others to design and analyse. If you are limited in your evaluation resources, it is best to stick to the simpler approaches. For example, observational techniques can produce a rich database that can be highly informative if analysed properly. The trick here is to design instruments which are either suitable for statistical analysis, or for other analytic strategies which have been developed for case study evidence (Yin, 1989). In the absence of careful advance planning for the analysis, many an evaluator has wound up with a massive investment (both in time and in money) of data collected via observation that elude reasonable analysis.

Finally, you need to decide on the appropriate mix of data collection techniques, including both quantitative and qualitative approaches. In a broad sense, quantitative data can be defined as any data that can be represented numerically, whereas qualitative data are more frequently expressed through narrative description. Quantitative data are also useful in measuring the reactions or skills of large groups of people on a limited set of questions, whereas qualitative data provide in-depth information on a smaller number of cases (Patton, 1990). These distinctions are not, however, absolute. Rather, they can be thought of as representing two ends of a continuum rather than two discrete categories. Furthermore, in some instances qualitative data can be transformed into quantitative data using judgmental coding (for example grouping statements or themes into larger broad categories and obtaining frequencies). Conversely, well-designed quantitative studies will allow for qualitative inputs.

Both types of data can provide bases for decision-making; both should be considered in planning an evaluation. And evaluations frequently use a mix of techniques in any one study.

Once these decisions are made it is very helpful to summarize them in a "design matrix."