Considerations for Survey Reports

Once the collating phase of a survey publication is complete the final phases of analyzing the results and publishing a report begins.

Initial Analysis

The first step should be to cast an eye over the collated data and look for anything that might be a problem such as:

• Incomplete survey responses
• Unanswered questions
• Comments that:
• include typos
• contain incorrect grammar
• are abusive and/or offensive
• that contain criticism of the survey
• foreign comments
• identifiable comments

Incomplete Survey Responses and Unanswered Questions

A major advantage that online surveys have over tradition hard-copy surveys is that with online surveys there is the option to include, or to some degree exclude, incomplete surveys.

Including all the partially completed surveys can be a problem and the decision to include or exclude is rarely cut and dry.

In a large survey there might be a number of incomplete surveys that have not responded with any meaningful information so a decision as to whether those responses are to be included or excluded from the results needs to be made.

Where the respondent might have answered just one or two preliminary questions there may be a good argument for excluding the response entirely from the results so as to avoid any analysis being unjustly skewed.

In a survey containing a substantial number of questions for all the incomplete surveys there is likely to be a spread of the number of questions that have been answered. Although there might be clear justification for excluding those respondents who failed to complete the first few questions, a decision to exclude those that completed a significant amount of questions is likely to be much harder to justify.

As to whether incomplete responses are to be included or excluded, much will depend on such considerations as:

• what was the likely reasons were for respondents abandoning the survey at such an early stage;
• the number of abandoned survey responses as a percentage of completed survey responses;
• the objective of the survey.

The Reasons for Abandoning the Survey

If it is possible to identify why the survey was abandoned it may make the decision straight forward to include or exclude the incomplete response in the analysis, for example, it wouldn't be a hard decision to exclude some incomplete responses for a survey that is targeted towards a specific age group, for respondents outside that age group it may be easy to see that instead of finishing the survey correctly, they had mostly abandoned the survey by simply closing the browser.

On the other-hand it could transpire that the survey questions were such that many respondents decided to abandon the survey and although not much could be done to rectify the problem at such a late stage, such observations should serve as experience for future surveys.

Number of Abandoned Survey Responses Versus Completed Survey Responses

If when excluding all the abandoned survey responses the remaining sample that is left is considered too small for any meaningful analysis it may be that the lessor of two evils is to include some or all of the incomplete surveys.

If the number of abandoned surveys is small compared to the numbers that have been fully completed, it may be that the effects on the results are negligible. In practice, many partially completed surveys still provide good intelligence for those questions that have been answered.

The Survey Objective

Surveys are conducted for a plethora of reasons, the survey objective will have some bearing on how incomplete survey responses are handled.

As is clear from the above there is no simple right and wrong answer as to whether incomplete surveys are included, partially or fully excluded.

In many cases it will be possible to analyse each questions response and exclude non-answers on a question by question basis.

When analysing the results if there is a requirement for second guessing why a respondent did not answer, it can lead to unreliable results.

It is important to determine why there were no answers to some questions; for example, did the question allow for a 'No Comment', 'Not Applicable' type response?

It is always recommended that a survey always obtains a positive 'No Answer', 'No Comment', 'Not Applicable' response rather than making the question optional. If the question is left optional there is no way of determining if the respondent deliberately did not answer or simply missed the question by mistake.

Typos

Where comments have been entered it is common for typos to be made by respondents. If the comments are to be used in a report it may make the report more readable if the typos are corrected.

However, if the survey was canvassing students, this may be an example where it may prove useful to see comments containing spelling mistakes.

Incorrect Grammar

For the same reason that one might want to correct typos similar reasons may apply to correcting grammar, however care should be taken.

There will be some instances where grammar is considered wrong because the way it is written could have different interpretations. By correcting the grammar and making the response clear, an assumption as to what the respondent meant could result in unreliable results.

As a general rule, if a comment is ambiguous it is probably better for it to remain ambiguous rather than assume one interpretation over another.

Abusive and/or Offensive

If comments are found to contain abusive and/or offensive remarks and the intention is to include all comments in a report for distribution, depending on the reports audience, it may be necessary to redact certain comments.

Comments that are deemed to be abusive and/or offensive could simply be removed or rewritten but in many cases redacting by leaving notification that certain comments have been removed is considered more transparent.

For example, changing an offensive comment to 'Possibly offensive comments have been removed by survey administrator' is considerably more transparent than just removing or rewriting the comment.

If the report's intended audience is sufficiently robust to accept the unedited comments then serious consideration should be made for keeping abusive and offensive comments intact.

Comments Containing Criticism of the Survey

Sometimes comments will be received that contain criticism of the survey such as pointing out that a particular question could be interpreted in a number of different ways or that the question could not be answered correctly due to the restricted number of answer options.

In some cases the criticism may be judged to be unjust or petty, in other cases it may prove to be valid and critical to the analysis. In cases where it is pointed out that the question is ambiguous it may be prudent to exclude the question and answer entirely from the final report.

Questions must have the same meaning to the publisher and the respondents and it is a common mistake for surveys to be published where the publisher poses a question and know what they mean, but the respondent when reading the question may interpret the question entirely differently; because there is no way of knowing in what way they read the question and therefore the way they answered the question, excluding the results entirely may prove to be the best option.

Foreign Comments

Translating any content runs the risk of misinterpreting the original meaning. To avoid being unfairly criticised for changing the meaning of a foreign comment an automated translation program should never be used, if practical, the translated, and the original comment should be included together.

Identifying Comments

Comments can include specific references to other people that may prove detrimental in some way, either to the person mentioned, or in an anonymous survey in a way that identifies the respondent. Much depends on what type of comment and of course whether the survey was described as anonymous and/or confidential.

As with other forms of redaction it is always better to redact in a way that ensures the redaction is conspicuous.

Confidentiality

If a survey has been promoted as being confidential and anonymous it is important that the collated information is processed in a way that honours that commitment and that may require that the source information is destroyed after use.

Before any report is distributed a check should be made that ensures that if the survey was to be confidential and anonymous the information cannot be traced back. For example if as part of an employee survey the employee's department is requested and only one person is employed in a particular department, and the results can be filtered by department, that one person's responses could be easily identified.

This can be a particular problem where the survey sample is small.

Analysis of the Results

Once the initial data normalisation has taken place it may then be possible to use the results in their entirety.

However, for many surveys there may be a need to anticipate follow-up questions, for example if a question was posed asking a respondent to consider if 'sex discrimination was considered a problem in the workplace' it may be useful to view the response data as a whole, and then also in isolation depending on gender.

Purpose

The purpose of detailed analysis of survey results may well be to bolster a weak argument though the use of statistics. The phrase 'Lies, damned lies, and statistics' is a phrase often used to doubt statistics used to prove an opponent's point.

In some cases it may be that the statistics are being used unfairly but with the purpose to bolster a specific argument, but in other cases it may be entirely justified to highlight a valid point.

For example in a group of a hundred people who were asked if 'sex discrimination was a problem in the workplace', if 90% of the respondents said 'No' and that 10% of the workforce were female the analysis would be justified in determining if the 10% that had considered it to be a problem were in fact all, or mostly, female.

Depending on which side of the argument, the statistics could be used to bolster any viewpoint.

If the intention in the report is to produce impartial findings, and provide source data so that others can interpret the results, care is needed as to when, if at all, the results are to be analyzed in detail.

Charts or Tables?

How the data is displayed should be very dependent on the report's target audience. In some cases the audience will prefer the data in its most raw form, for others there may be a preference for the report to be presented clearly and for the data to be displayed as charts.

There are many chart formats to choose from such as Pie, Bar, Column, Line, scatter, 3D or 2D.

The chart format chosen should not be arbitrary but be the format that represents the data in the clearest form however keep in mind that what determines the 'clearest' form can be subjective. Consider showing the results for the same question in a number of different formats, for example as a pie chart and as a table, this allows the benefit of both.

Executive Report

Sometimes a chart is not enough, after all charts need to be mentally interpreted and a report full of charts that at first glance appears to be easy to read can in fact prove to be very time consuming to read properly.

A method that can be deployed that will allow executives to rapidly digest a survey report is to include an Executive section.

This simply interprets each questions response in the form of a summarized sentence for example:

90% of those who responded agreed with the statement that 'the Company is a good place to work', 5% disagreed and 5% declined to answer.

When writing an executive report it is important to only include salient facts and not to include subjective statements such as:

An overwhelming majority of 90% agreed with the statement that 'the Company is a good place to work', a mere 5% disagreed and 5% declined to answer.

In this second example 'overwhelming' and 'mere' are subjective, and it may prove to be the case that the 'mere 5%' is unacceptable to the senior management.

Executive Reports are among the most expensive in terms of resources to produce. It is not a process that lends itself to automation as each question and answer must be processed by a human with the ability to summarize the results in a clear and unbiased way and in a non-monotonous style.

However, the cost of compiling an executive report section needs to be considered in how much time it may save Senior Management and it is something that could prove to be the difference between a report being read and ignored.

For more information or to discuss how online surveys can help you please contact surveys@surveygalaxy.com or visit www.surveygalaxy.com the quick, easy and cost effective way to do online surveys.