There are a number of ways data can be analysed depending on the type of data being explored. Some tools use algorithms to find relationships between variables. Other tools make complex data into visualizations that can help people make sense of data. There are also tools for qualitative data like text-analysis of written survey responses. Below is a selection of helpful and accessible tools for first time evaluators.
Cleaning Data
Tool: Spreadsheets (Excel/Google Sheets)
How does data get “dirty”? Hammond, Malec, and Buschbacher (2014) define dirty data as inaccurate, incomplete, or erroneous. They argue that no matter the precautions taken most studies will likely have to clean up some dirty data. This can include typos, duplicates, missing data coded as “0, or mix-ups when combining datasets.
Cleaning data is typically the first step evaluators must take before analysing data. Davis (2010) argues the best people to clean data are the people that collected the data because of their familiarity with protocol and the subject matter. Squire (2015) uses the metaphor of a chef that can’t properly cook in a dirty kitchen when discussing the challenges working with dirty data. She suggest using a spreadsheet in Excel or Google Sheets to convert large blocks of text and other data from online sources including PDFs. Slater, Joksimović, Kovanovic, Baker, & Gasevic (2017) recommend the use of spreadsheets as a way to present data first clearly within a fully visual interface allowing evaluators to identify structural or semantic problems in the data.
Web Analytics
Tool: Google Analytics
The ease of use, cost-free options, and insights it can generate make Google Analytics an appealing option for evaluators.
Google Analytics is a powerful tool for businesses to gain insights on how the public interacts with their online presence. It also has many applications within the field of education and evaluation. Web-based educational projects can incorporate Google Analytics into the collection, analysis, and presentation of their evaluation data. Google Analytics works by using small text files called cookies to gather and track information about website usage.
As one of the most popular web analytics tool in use today there is well-developed documentation and integrations with popular services. Some web hosting services even include options to automatically install google analytics tracking into your website.
In my own experience working in public legal education (PLE) I used Google Analytics extensively. I used Google Analytics to track what learning resources were the most popular and where the information was being accessed. I advocated for a mobile-first redevelopment of our websites using data indicating an increase in our mobile visitors. Many of the evaluation frameworks I developed used web analytics as indicators for outcomes.
Internet-Delivered Genetics Education Resource for Nurses Case Study
Kirk, Morgan, Tonkin, Mcdonald, & Skirton (2012) utilized Google Analytics to study three years worth of web usage of an online genetics education resource for nurses. Variables they tracked through Google analytics included number of visitors, unique and return visits to the site, bounce rate, location of visitors, traffic sources, referring sites, keywords used by visitors coming from search engines, language of visitors, and device used. This information was used as part of their website evaluation and optimization efforts. The information helped them to demonstrate that their resource was being accessed by their target audience; an important and common indicator in evaluations.
Mental Health Web Resource Case Study
Song et al. (2019) studied users of an educational mental health resource using information from Google Analytics. They looked at overall engagement including number of visits, where their traffic was coming form, how long visitors were staying, and how many pages they viewed. The data provided from Google Analytics when used in conjunction with qualitative data from other evaluation activities played a vital role in highlighting the preferences of users of the web-based mental health tool. The findings were used to improve the online platform by looking at what devices were being used to access the resource. Marketing strategies were informed by looking at what campaigns were generating traffic and referrals
They cautioned drawing conclusions from Google Analytics without additional sources of data. For example long visits could mean that users were engaged with the content or it could mean that they had trouble finding the information they were seeking. To remedy this they supplemented Google Analytics data with qualitative data collected from traditional tools like surveys and focus groups.
Text Analyis
Tool: Quantext, NVIVO, LIWC
There are a number of tools that have been developed to work with qualitative text data. These tools are helpful when working with large amounts of qualitative data like open-ended text response surveys or interviews.
- Quantext is a text analysis that has been designed with educators in mind. The tool is currently in development and being piloted with teacher in New Zealand.
- NVIVO is a popular choice for post-secondary settings and offers advanced functions that are probably more advanced than a first time evaluator would require.
- LIWC is another text analysis tool with a number of commercial versions. A demo where you can copy and paste text information for analysis with LIWC can be found here.
Exploring Student Surveys using Quantext
In a self-guided research project I explored how learning analytics could be applied teacher quality surveys in higher education. This was my final assignment for the Learning Analytics course in my MET.
Evaluation surveys are controversial among educators in post-secondary. Important decisions including tenure and promotions can be based on student evaluation surveys despite concerns around their validity.
I wanted to explore how text-analysis could be applied to these surveys. I used a text-analysis tool called Quantext to explore how long answer questions in a survey or even potentially student portfolios could be analysed for the purpose of instructor evaluation. I was pleased with my early experimentation and look forward to applying text analysis to my future assessment work. Below is an informal video walk through of my project that I created for my classmates.
References
Davis, M. (2010). Data cleaning. In N. J. Salkind (Ed.), Encyclopedia of research design (pp. 326-328). Thousand Oaks, CA: SAGE Publications, Inc. doi: 10.4135/9781412961288.n100
Hammond, F., Malec, J., & Buschbacher, R. (2014). Handbook for clinical research : Design, statistics, and implementation. Retrieved from https://ebookcentral.proquest.com
Kirk, M., Morgan, R., Tonkin, E., Mcdonald, K., & Skirton, H. (2012). An objective approach to evaluating an internet-delivered genetics education resource developed for nurses: using Google AnalyticsTM to monitor global visitor engagement. Journal of Research in Nursing, 17(6), 557–579. https://doi.org/10.1177/1744987112458669
Slater, S., Joksimović, S., Kovanovic, V., Baker, R., & Gasevic, D. (2017). Tools for Educational Data Mining: A Review. Journal of Educational and Behavioral Statistics, 42(1), 85–106. https://doi.org/10.3102/1076998616666808
Song, M., Ward, J., Choi, F., Nikoo, M., Frank, A., Shams, F., … Krausz, M. (2018). A Process Evaluation of a Web-Based Mental Health Portal (WalkAlong) Using Google Analytics. JMIR Mental Health, 5(3), e50. https://doi.org/10.2196/mental.8594
Squire, M. (2015). Clean data: save time by discovering effortless strategies for cleaning, organizing, and manipulating your data. UK: Packt Publishing. Retrieved from http://search.ebscohost.com.uml.idm.oclc.org/login.aspx?direct=true&db=nlebk&AN=996715&site=ehost-live