Reading your first ReliCheck report

Your survey closed last night. The first report opens to a wall of tiles, tables, and a heatmap. The instinct is to scroll until something looks familiar. The faster path is to read the report in a fixed order, top down, with a specific question in mind at each step.

Here is the order that gets you to a defensible conclusion in about ten minutes, no matter how many items the survey had.

Step 1: Sample and items tile

Always start here. The reliability numbers below this tile are only as informative as the sample they came from.

Two things to notice. First, the n that powers reliability is the "complete" count, not the total. A survey with 500 responses and 80 complete Likert blocks reports reliability on 80, not 500. Second, the dropped count is a quiet quality signal. Heavy dropping on a single scale usually means one item is confusing or sensitive enough to make people skip.

Step 2: Cronbach's alpha tile

Read the value, the band, and the sample size from Step 1, in that order. An alpha of 0.84 on n = 134 is solid. The same alpha on n = 22 would deserve a "preliminary" footnote in any report.

What the value does not say: that the scale measures the right construct, that it is unidimensional, or that it will replicate in a different sample. It says only that the items moved together in this sample.

Step 3: Split-half tile (sanity check)

Split-half is the second opinion on the alpha number. When alpha and split-half land within roughly 0.05 of each other (here, 0.84 and 0.81), the scale is behaving consistently. A larger gap suggests an order effect: the second half of the scale may include items that drift away from the construct, or the items at the end may be answered more carelessly because of fatigue.

Step 4: KMO tile (factor-analytic suitability)

KMO answers a different question than alpha: it asks whether the items have enough common variance for a factor-analytic interpretation to be appropriate. A KMO of 0.79 means yes. Below 0.60 means the data does not support that interpretation, no matter what alpha says. KMO and alpha measure related but distinct properties; reading them together gives a cleaner picture than reading either alone.

Step 5: Item-total table

This is where the per-item story lives. Each row shows the corrected item-total correlation (the item's correlation with the rest of the scale), the per-item KMO, and what alpha would be if that item were deleted. Three things to look for, in order:

1. Negative item-total correlations. Almost always a missing reverse-score flag, sometimes an off-construct item. Either way, fix before the next analysis.
2. Item-total correlations below 0.20. The item is barely participating in the scale. Review wording and content fit.
3. Alpha-if-deleted values that exceed the overall alpha by more than 0.02. The item may be hurting consistency. Read its content; if substantive fit is also weak, consider revision or removal. If substantive fit is fine, keep the item.

Step 6: Inter-item correlation heatmap

The heatmap is the report's most condensed diagnostic. Healthy single-construct scales show correlations roughly in the 0.30 to 0.70 range, with a uniform color across the off-diagonal. Two patterns are worth flagging:

• A bright block separated from the rest. Suggests a second factor is hiding in the scale. The block is items that correlate with each other more than with the rest. Sum-scoring the whole scale loses information that block is carrying.
• A row or column that is dim against everything. Suggests one item is on its own. Often the same item flagged in the item-total table.

Step 7: Per-item descriptives and distributions

Last on the reliability tour, but not least. Means, standard deviations, and the response distribution per item answer questions reliability statistics cannot, like floor and ceiling effects (every respondent picked 5; nothing left to discriminate), midpoint piling (more respondents than expected sitting on the middle), or skew (a long tail in one direction that suggests the item is too easy or too hard).

Putting it together

A ten-minute reading of the report in this order yields a short verbal summary that fits in a memo:

"The eight-item engagement scale shows good internal consistency on n = 134 (α = 0.84, split-half = 0.81), with KMO = 0.79 supporting a factor-analytic interpretation. Item E5 is the weakest contributor (item-total = 0.22) and should be revisited; removing it would raise alpha to 0.86. The heatmap shows uniform correlations between 0.35 and 0.62, with no second factor visible."

That paragraph is what every reliability check is trying to produce. The report packages the inputs; the order above turns them into the sentence.

For the formal definitions of every statistic above, see the Methodology page. For the broader concepts, the Reliability guide.