The challenge
The program ran a homegrown 24-item evaluation built in a generic survey tool. The form was long, the items mixed instruction quality with personality assessments, and the report at the end of each term was a static PDF showing means and counts only.
Three problems compounded each other. Response rates slid below 45% as students stopped seeing the value. Faculty pushed back on the report because adjacent items appeared to measure the same thing without any check that they did. The accreditation review flagged the instrument as needing psychometric documentation, which the program did not have the staff to produce.
How they designed the survey
The program switched to the ReliCheck Course Evaluation template and customized it lightly. The new instrument has 18 items mapped to four dimensions: clarity of instruction, alignment with stated learning outcomes, climate and engagement, and assessment quality. Two open-ended items at the end ask what worked and what should change.
Three design choices made the instrument shorter and more defensible. First, the program dropped two redundant items after seeing the inter-item correlation matrix from a pilot. Second, items that previously measured instructor personality were removed (they did not load on any of the four dimensions). Third, the open-ended items were placed last so respondents who run out of time still complete the closed items.
What the data showed
The first full-term run drew 161 of 240 students (67%). Reliability for each of the four dimensions came in at α between 0.81 and 0.88, with KMO above 0.78. AI theme extraction on the open-ended items surfaced 4 themes per course on average, with sample quotes the program coordinator could pass to faculty without reading every response.
Course-level reports went out to faculty within 48 hours of the survey closing. Each report opened with the four dimension scores, the alpha tile, and a 2-sentence AI plain-language summary, then drilled into item-level distributions. The same report exported to PDF and Word; the methodology appendix dropped straight into the accreditation packet.
"The accreditation reviewers asked us to document our instrument. We sent them the methodology page from ReliCheck plus the reliability statistics from one term. They wrote back the same week and said it was the cleanest documentation they had seen from any program that year."
At a methods glance
| Sample size | n = 161 across 12 courses |
| Instrument | 18 Likert items, 4 dimensions, 2 open-ended |
| Reliability | α 0.81 – 0.88 by dimension; KMO 0.78 – 0.84 |
| Item flags | 0 items below ITC 0.30 after pilot revisions |
| AI summary | 4-theme average per course, reviewed before send |
| Export | PDF for faculty, Excel for the program office, methodology appendix as Word |
What they did with the result
- Faculty received their reports with no editorial framing from the program office. The dimension scores spoke for themselves and faculty took conversations from there.
- The program office surfaced two cross-course findings to the curriculum committee: assessment quality scored consistently lower than the other three dimensions, and pacing complaints clustered in the second-year electives.
- The accreditation packet now includes the ReliCheck methodology page plus one term of reliability statistics as the instrument documentation. Reviewers accepted it without follow-up.
Read more about how education and evaluation teams use ReliCheck →