Questionnaire Design

The term 'Questionnaire' can cover a vast range of subjects and although all questionnaires are conducted for the common purpose of research the specific research subject will very much dictate the questionnaire design and layout.

This article provides a generic framework that can be used for all types of questionnaires. By following the basics of questionnaire design that is covered in other articles the information in this article can be used to provide a final check prior to a questionnaire going live.

To use this guide, for each individual question in turn, consider the following.

Is the information the question collecting available elsewhere?

If the questionnaire is being issued to a known person is it necessary to collect personal information such as name, age and gender if that information is available elsewhere and is accurate? For example if the questionnaire is aimed at employees then it is very likely that personal information was collected and verified when the employee joined the organization.

There are circumstances when collecting known information again can be justified, for example to make the analysis of the results easier or to audit information previously collected.

If the information being collected is available elsewhere, give serious consideration as to whether there are sufficient reasons that justify it being asked again, if not exclude the question.

Does the question provide information that will help achieve the questionnaire's objectives?

Each question asked should in some way be helping achieve the questionnaire's objective either by providing intelligence that directly addresses the objective, or by providing information that will allow the collated data to be analysed that in turn will contribute to achieving the questionnaire's objectives.

For example asking a respondent their age and gender may not directly have any bearing on the questionnaires objectives but may allow better analysis of the collected data.

If the question is irrelevant to the stated objectives of the questionnaire and does not provide any analysis benefits then the question should be excluded.

Is the question biased?

Not all questionnaires are intended to be unbiased but if the intention is to conduct a 'neutral' questionnaire then each question must be phrased so that there is no bias.

Bias can be in the question being asked and/or in the answer options that are available.

A question like 'Do you regularly go to Church?' may appear straight forward but the word 'church' is very specific to certain religions, those who are of different religions may go to synagogue, mosque or temple.

It is common for those who are new to questionnaire design to phrase questions from their own perspective. Depending on the nature of the questionnaire, it can be useful when reviewing to consider a list of possible and diverse imaginary personas. For example:

• married with children
• married without children
• single parent
• unemployed
• employed
• religious
• non-religious

With each persona in mind check that there is no ambiguity in the question and that the question and answer options are not bias towards any individual.

Unreliable data will be collected if the questionnaire that asks the question 'Do you regularly go to Church?' assumes that those who regularly attend 'Temple' will consistently answer 'yes' or 'no'.

There is also a condition known as order bias, where the order that the questions are asked, or the order that the answer options are listed encourages certain responses.

Order biased considerations need to be balanced with considerations over clarity. Take for example answer options that list the possible answers as:

• Very Good
• Good
• Okay
• Poor
• Very Poor

The above answer options provide clarity and in many cases clarity may win out over considering the effects of order bias. However if the question was, 'Which of the following words do you associate with the company you currently work for?', and the answer options:

• Good
• Positive
• Forward-thinking
• Caring
• Pioneering
• Inventive
• Friendly
• Bad
• Evil
• Uncaring

In this example, by listing all the positive attributes first, some may argued that it attempts to 'rig' the results. In such circumstances if the questionnaire is being conducted online or via telephone, consider randomising the order that the answer options are listed.

Is the question subjective?

Does the question contain any of the following types of subjective words:

• Often
• Regular
• Frequently
• High
• Low
• Fast
• Slow
• A lot
• A little
• Big
• Small
• Large
• Fat
• Tall

Subjective terms mean different things to different people and so two people that may answer 'often' to the same question may have completely different interpretations of what they consider 'often' to mean.

A question such as 'Do you regularly go to church?' is therefore ambiguous not only for using the word 'church', but also in determining what constitutes 'regularly'. For some, regularly may mean several times a week, for others it might mean several times a year or even once every year.

When considering each question check for any 'subjective' element and if found either use a non-subjective alternative or quantify the subjective element, so for example, 'Do you regularly (on average three of four times a month) attend a religious place of worship?'.

The use of unqualified subjective questions will result in unreliable data, as the question being asked may not match the answers being given and if there is any inconsistency in interpretation then the gathered information will be compromised.

Will the question generate data using the same metrics required for the analysis?

If the questionnaire asks respondents for their exact age, consider if the analysis requires individual ages or if the analysis will use age groups, if it is the latter then the questionnaire should collect the information in the desired age groups.

When using answer options that use scales, such as one to ten or one to hundred, consider if the analysis requires and/or will benefit from the size of the scale. In the majority of questionnaires the analysis often boils down to measuring two levels below the meridian and two levels above. If a scale of one to hundred is used, and the analysis will then simply group marks of say 75 to 100 to 'Very Good' for reporting purposes the use of a ten or hundred point scale is pointless and it is better to match the questionnaire's answer options to the analysis requirements.

Consider each question and ask if the respondent will answer truthfully?

Consider each question in terms of 'can' and 'will' the respondent answer truthfully.

Can the respondent answer truthfully?

Does the question and answer options cater for all eventualities? If the questionnaire is online and the question is a closed question and made mandatory, do all the answer options cater for all possible options?

Where only one answer response has been allowed, could multiple answer options apply?

Considered the situation where the respondent may not be able to answer, or may not want to answer the question? It is better to include a positive 'Don't know', 'No comment', 'Not applicable' response option than force an incorrect and therefore misleading one.

Will the respondent answer truthfully?

If the questionnaire is not declared as being anonymous, or it is not trusted to be anonymous, then for some types of questionnaires, and some types of questions, there may be factors that encourage or prevent the respondent from answering truthfully.

It is common for employee satisfaction questionnaires that allow the respondent to score their line managers that an intimidating line manager might unjustly score higher marks than a more open manager.

Consider the value of including any questions where there is a risk that the question may not be answered truthfully. Either take steps to improve the chances of obtaining a truthful answer, such as emphasising the anonymity aspect of the questionnaire, or allow a 'rather not say' type response.

Are the respondents qualified to answer the questions that are being asked.

Consider each question and determine if the question is written in a clear and concise way. If the question relies on specific knowledge and experience consider if it can be guarantee that the respondent will be able to intelligently answer the question.

Avoid jargon or buzz words if there is any doubt that some respondents may not fully understand them.

Summary

Questionnaires can be very effective in gathering important intelligence for a plethora of reasons. However, poorly designed questionnaires can result in a wasted opportunity and at the very worst incorrect and misleading information being gathered that if acted on could prove very damaging.

By challenging each question before publishing the questionnaire, to ensure that each question has earned its right to be asked, questionnaires can be designed that are effective in providing valuable and insightful research data.

For more information or to discuss how online questionnaires can help you please contact surveys@surveygalaxy.com or visit www.surveygalaxy.com the quick, easy and cost effective way to do online questionnaires.