← The Signal
Survey Design

One Question, Two Jobs: The Double-Barreled Item and How to Catch It

How satisfied are you with your pay and benefits? It looks like one question. It is two, and the answer you get back means neither.

Consider a respondent who loves their salary and resents their thin benefits. Hand them a question about pay and benefits together and they have no honest answer. Whatever they pick is a compromise between two feelings, and you have no way to know which one won. Multiply that across a sample and you get a tidy average built on mush. This is the double-barreled item, and it is one of the most common ways a well-meant survey quietly ruins its own data.

Why it does more damage than it looks

The trouble is not just that the question is awkward. It is that the number it produces cannot be interpreted. When a response could have been driven by either half of the item, you can no longer say what the score measures, and an item you cannot interpret weakens every scale and every comparison it feeds. Worse, the damage is invisible in the results. The averages look fine. Only the meaning is broken, and by then the data is already collected.

How to spot one

The quickest tell is a conjunction. Scan your items for and and or joining two different ideas, as in clear and fair, training and support, safe and welcoming. Then apply the real test: could a reasonable person agree with one part and disagree with the other? If yes, the item is double-barreled, no matter how natural it sounds. These hide most often in satisfaction questions and agreement batteries, exactly the places researchers write quickly because the format feels routine.

How to fix it

The fix is almost boringly simple. Split the item in two, one for pay, one for benefits, and let each carry a single idea. You spend one extra line and you get two numbers that actually mean something, instead of one that means nothing you can name. If the two halves really do move together, you will see it in the data. That is a finding. Baking the assumption into the question is not.

Where the Survey Development System catches it for you

Reading your own survey for this is hard, because a question you wrote reads as clear to you by definition. ReliCheck's Survey Development System checks your items for writing quality before you launch, flags stems that ask more than one thing, and points you to the revision, so the double-barreled item gets caught while it is still a quick edit rather than a limitation you write up after the data is in. A survey is only as good as its worst question, and the cheapest time to find that question is before anyone answers it. That is what the system is built to give you, a clean instrument before it goes out, not a diagnosis after it comes back.

ReliCheck's Survey Development System checks item writing quality before launch, flagging questions that ask more than one thing and pointing to the fix. See it at relichecksurvey.com.