Welcome
For my ETEC 580 independent study project I chose to explore my interest in evaluation and build upon my work in ETEC 565A Learning Analytics course. I decided to create a collection of open educational resources for educators starting evaluation projects in their workplace. The intention was to create the on-boarding materials that I wish that I had had.
I have drawn from my experience working in private industry, post-secondary institutions, non-profits, and government to make this guide accessible to users from diverse backgrounds. This guide is written in accessible language so that readers can “hit the ground runnning” and apply it to their workplace.
Introduction to Evaluation
Before diving in, it is important to zoom out and get an understanding of the purpose, structure, and circumstances that will frame any evaluation work. An infographic of evaluation first step can be found below and enlarged by clicking.
Mapping Program Theory
Mapping the program theory is an important evaluation activity that helps evaluators understand the rationale and assumptions for a program. It provides a solid foundation for a meaningful evaluation plan. In my presentation below I provide an overview on mapping program theory and walk through my own template that first time evaluators can adapt to their own workplaces.
Why is program theory important to evaluation? If we know how the program is supposed to function we can find indicators to measure its effectiveness.
For example if an outcome of a business communication course is to create professional networks we can follow up with past participants to measure how many classmates they have maintained contact with. If evaluating a leadership program intended to help high school students from marginalized communities transition to university successfully we can measure the number of program graduates that enrolled in post-secondary after completing the program.
Recommendations can also be made if the evaluation uncovers discrepancies between theory and practice. For example the program theory might reveal that a program meant to serve marginalized and vulnerable youth is actually exiting many of these youth from the program. A recommendation might be made to address these exit points so that the target audience of vulnerable youth is being served.
The data you collect and analyse are tied to the outcomes identified in the program theory. The results of the evaluation should be communicated in a way that links back to the program theory.
Planning an Evaluation
An important lesson I learned was the importance of having an evaluation plan before a program starts. The includes knowing what data you will need to collect, how it will be analysed, and how it will be presented. A plan can prevent you from having gaps in data because it wasn’t being collected. If evaluation is new to your organization you might find yourself asked to evaluate a program already in progress and your evaluation plan will have to unfortunately work around gaps due to lack of planning.
I have written pages with detailed information on the general phases of evaluation. These phases of an evaluation typically occur chronologically but not always. Some types of evaluations are iterative and these steps can be cyclical, ongoing, or even concurrent.
Essential Reading
Below is a list of readings that have been pivotal to my professional development in the field of Assessment.