Posted on

Gathering a Bundle for Indigenous Evaluation

Cultural competence is important in program evaluation. It is important that evaluation is conducted in a respectful manner that encourages buy-in and participation of the staff, participants, and community members. Cultural awareness in evaluation is critical to ensuring a fair and accurate evaluation is conducted.

The Truth and Reconciliation Commission (TRC) was formed to address the injustices of over 150 years of residential schools on Canada’s first peoples.   The Truth and Reconciliation Commission of Canada: Calls to Action include a number of steps Canadian society should take towards reconciliation. In particular, the categories of education, professional development, training for public servants, education for reconciliation, and youth programs require the participation of educators. This means that there is a lot of work to be done in Canada towards reconciliation and consequently evaluation in education will likely include questions about Indigenous topics. Canadian universities for example have made commitments towards reconciliation. The University of Manitoba where I work has included an emphasis on Indigenous content and achievement in its Strategic Plan.

Na-gah mo Waabishkizi Ojijaak Bimise Keetwaatino: Singing White Crane Flying North: Gathering a Bundle for Indigenous Evaluation is an excellent guide and highlights a unique approach. This  resource is written for working with Indigenous people but many of the broader themes apply to all audiences. The methods explored could be adapted to general audiences and still lead to rich qualitative data important to evaluation projects.

The bundle recognizes the importance of grounding evaluation within the cultural context it is occurring in. There is an emphasis on qualitative methods that are flexible and more holistic to ensure culturally significant impacts are not lost. The first half of the document provide a solid theoretical perspective that challenges traditional views of assessment and evaluation. The latter half describes what this looks like in action. There is an example of a story-telling sharing circle exercise that uses art and a tree metaphor to collect qualitative data.  

Rowe, G., & desLibris – Documents. (2018). Na-gah mo waabishkizi ojijaak bimise keetwaatino Canadian Centre for Policy Alternatives. Retrieved from https://www.policyalternatives.ca/sites/default/files/uploads/publications/Manitoba%20Office/2018/03/Gathering_A_Bundle_for_Indigenous_Evaluation.pdf

Posted on

Evaluation: A Systematic Approach

I recommend the Splash & Ripple guide as a first introduction to evaluation. This book is an excellent next step. There is a bit of overlap with Splash & Ripple but it provides a much more in depth and fulsome understanding of evaluation. It takes a step back from just looking at outcomes by asking the evaluator to explore program theory; the assumptions and underpinnings of a program.

Rossi, Freeman, & Lipsey (2004) describe program theory as the “plan of operation, the logic that connects its activities to the intended outcomes, and the rationale for why it does what it does.”  The book describes in detail the elements of program theory.

  • The program impact theory explicates the anticipated impacts of a program and how they will be achieved.  
  • The utilization plan details how the various program pathways for participants
  • Program organizational plan can take the form of a logic model. This is a standard tool in outcomes measurement and includes identifying inputs, outputs, and outcomes.

The book also provides a well-rounded and detailed overview of many other elements of evaluation including measuring outcomes, bias, experimental designs, efficiency, and social context.

Rossi, P. H., Freeman, H. E., & Lipsey, M. W. (2004). Evaluation: A systematic approach (7th ed.). Thousand Oaks, CA: SAGE Publications.

Posted on

Designing a Rubric for Selecting an LMS

For this assignment my group was tasked with creating a rubric for selecting an learning management system (LMS) for professional development for nurses. We analysed the scenario and came up with a list of 11 criteria for our rubric.

  • flexible delivery for diverse learners
  • active and current content
  • community building
  • intuitive design
  • cost-effective and scalable
  • safe and secure
  • dynamic and iterative design and development
  • integrates multimedia
  • curriculum mapping
  • training, documentation, and support
  • learning analytics for assessment and evaluation

When brainstorming the criteria we referred to the SECTIONS framework. This framework was a good starting point for general decision making but it needed to be revised to highlight the unique requirements of this scenario.

I used my experience working in the Rady Faculty of Health Sciences at the University of Manitoba which includes the College of Nursing to highlight important areas not apparent in the scenario description. One thing that was very important for me to emphasize was intuitive design. I know from experience that when introducing new technology that a difficult user interface can lead to resistance from busy learners and staff. Removing unnecessary steps, clicks, and external links that send learners outside of the LMS prevents frustrated learners from giving up (White, 2017). This means that when selecting an LMS it is important to look beyond flashy bells and whistles that can create unnecessary complexity for users. Another important aspect that was not obvious from the scenario description was the role of multimedia in health care education. For example it would be important that LMS could support incorporating video elements for standardized patient videos.

I’m pleased with the final result of the project but ultimately I think using rubrics for comparing and evaluating LMS is cumbersome. In the real world I would have tackled this project by creating a feature checklist. I would create an comprehensive list of desired features organized by categories. I think this would make it much more easier to identify exactly what is needed for a project and make better comparisons.

In a later revision of the assignment we included a recommendation for an LMS. We started by discussing what we were already familiar with professionally. We ultimately decided on recommending Elentra (also known as Entrada). Many LMS offer similar features and did well when assessed with our rubric. What really stood out to us about Elentra was the widespread adoptio of it in the health sciences field. To me this signalled opportunities for interprofessional collaboration and maybe joint funding opportunities with universities.

References
Bates, T. (2015). Teaching in a Digital Age. Chapter 8: Choosing and using media in education: the SECTIONS model. Retrieved from http://opentextbc.ca/teachinginadigitalage/part/9-pedagogical-differences-between-media/

White, M., & Shellenbarger, T. (2017). Harnessing the power of learning management systems: An E-learning approach for professional development. Journal for Nurses in Professional Development, 33(3), 138-141. Retrieved from https://oce-ovid-com.ezproxy.library.ubc.ca/article/01709760-201705000-00007/HTML