Project Evaluation Designs

The most elementary project evaluation design is the post-test only without control group design, in which the evaluator administers one post-test to one group of units of analysis (UsA) to assess the effectiveness of the project. In this case the evaluator could only describe the extent to which the outputs (e.g., coverage, accessibility, KAP) of the project have been achieved. A little bit more advanced is the pre- and post-test without control group design, in which the outcomes (i.e., mortality, disability, morbidity and high-risk cases or rates) of the present project intervention is compared with that of a preceding intervention or condition. The outcomes of the latter are regarded as the pre-test of the present project. A time series without control group design should be utilized if prior outcomes already showed a significant trend, as a consequence of previous interventions or conditions, or if subsequent outcomes may reveal novelty or Hawthorne effects of the present project.

Using without control group designs, or same group designs, in project evaluation has the advantage of avoiding Differential Selection – a confounding construct that becomes manifest when the evaluator uses UsA that are not matched or placed at random when forming intervention and control groups. The genetic, nutrition, immunity, reproductive, anthropometric, demographic and socio-economic make-up of the UsA of the respective groups may be different to such a degree as to effect the outcome of health project interventions (i.e., medicines, vaccines, food supplements, education, training, family planning methods, management practices, etc.). Another confounding construct that confuse the interpretation of project success in with control group designs is Central Tendency – a tendency of the mean, mode and median to move to the center when extreme UsA are placed in the intervention or control groups.

On the other hand, the administration of tests to the same group of UsA at certain time intervals (which avoid differential selection) creates confounding constructs that are related to the progression of time. Project outcomes may co-vary with: (1) internal conditions of the UsA (Maturation), environment of the UsA (History), effects of testing (Testing), testing method (Instrumentation) and drop-out of UsA, particularly in long-term projects (Mortality). These internal validity threats can be prevented if to the same UsA can be given different interventions at the same time, which is almost impossible. To overcome this dilemma a similar control group(s) must be formed in the same time frame.

Consequently, if a project evaluator is determined to control confounding variables that stem from differences in groups as well as in time frames he/she ought to use intervention and control groups, which contain UsA that are matched or randomly placed, and a pre-test or series of pre-tests (to control for trends and mortality) and a series of post-tests (to control for Hawthorne and novelty effects). With control group evaluation designs require more resources, but projects without valid or logical evaluation are a waste of resources.

Leave a comment

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s