Measuring significance

Although there are many advantages to using samples rather than collecting data from the whole population under review, one of the disadvantages is that sample statistics only give an approximation of the population statistics within a given range, called confidence limits (see Section 7.5.3). Often the same measure is taken twice from two different samples. This may be from two different samples at the same time, e.g. a comparison between a sample of men and a sample of women; or between similar samples at two different points in time, e.g. recall of a brand at one point in time and again six months later after an advertising campaign. In these circumstances it is important to know whether the two figures could have arisen from the same underlying population or whether any difference in the two figures represents a real change in the state of the underlying population. Is there a real difference between men and women in their usage of or attitudes towards a brand? Has recall of the brand name changed over the test period which would indicate the need for more, less or improved advertising?

Significance tests

Significance tests measure whether the difference between two percentages is significant or not, or whether the difference between two means from different samples is significant. In each case the calculation will indicate whether the difference is significant at a particular level. A statement of significance will appear in the form, 'This figure is significant at the 5% level, or at the 1% level'. The 5% level of significance means that there is only a 1 in 20 chance that two different values could have arisen from the same underlying population. The 1% level of significance indicates that there is only 1 in 100 chance that two different values could have arisen from the same underlying population.

Significance, then, relates to sampling error. Sampling error is not the same thing as mistake. Sampling error is the variation possible between two samples drawn from the same population simply because a sample was taken and not a complete census. When only 1000 respondents are selected to represent the views of 19 million households, then however well the sample has been selected the estimate of the figure for the 19 million households will not be exact. Because of this, two different 1000-respondent samples could produce two slightly different estimates of the population parameter of the same population. Measures of significance test whether two different values could have arisen from the same population or whether their values are too far apart to have arisen from sampling error, and therefore represent an actual difference in values.

An important distinction must be drawn between the significance of a value and its importance or relevance. All that significance tests indicate is whether there is a real difference or no real difference between two values. They do not indicate that the difference or lack of difference is of any great importance or significance to the decision maker.

An insurance company carried out a piece of research to measure awareness of the company's name. They used an unaided recall question followed by an aided recall question showing a list of company names. The unaided recall question gave a figure of 3.5 per cent of respondents who spontaneously recalled the company name. Since the company achieved a much higher figure on aided recall, they decided to carry out an advertising campaign to bring their company's name more into the front of the target population's mind.

Six months later, following a £600,000 brand-awareness advertising campaign, the company repeated the piece of research using a similar research methodology. This time unaided recall for the sample was 2 per cent. Immediately, the individual who had commissioned the research became very concerned about significance testing in order to indicate whether the change in levels of recall represented a real fall in awareness following the advertising campaign, or whether both figures could have arisen from an underlying population in which the level of awareness of the company's brand name had not changed.

The significance test showed there was no significant difference between the two figures, i.e. awareness had not fallen during the period of the campaign. However, the important point here, surely, was that the best possible statistical interpretation of the figures was that there had been no change in awareness, yet this followed an expenditure of £600,000 on a campaign with increased awareness as its sole objective. Even without carrying out the significance test the best possible interpretation of the statistical data might still be considered as a measure of the failure of the advertising programme to achieve its objective. Executives in the company concerned, however, were relieved that the results of the significance test indicated no significant difference. They felt that it exonerated them from the even worse position that recall might have actually dropped following their advertising programme!_

It has already been pointed out that significance testing is only theoretically relevant for use on simple random sample data. In practice, statistical tests are applied to more complicated random sample designs and to quota samples for which they are not really appropriate. When significance tests are carried out on data from other than simple random samples a design factor of 1.5-2 is applied to the standard error used in carrying out further calculations. This has the effect of increasing possible variation in the data and therefore reflects the fact that these samples are thought to be slightly less accurate than simple random samples in producing statistical estimates.


In the same way that significance tests described in the preceding paragraphs measure the significance of the difference between two means or two percentages, there is a statistical significance test that measures whether differences in cross-tabulated data are statistically significant. This is called the chi-square test.

For example, purchase of white bread was analysed by social class to indicate any difference in purchase rates among middle-class and working-class households. Since there were different numbers of buyers and non-buyers in the sample, and different numbers of working-class and middle-class households in the sample, inevitably each cell in the table had different values. The problem for the decision maker was to interpret whether those different values represented a real difference. A chi-square test was applied to the data and indicated that working-class households were statistically significantly higher purchasers of white bread.

Since tables are commonly the output of research surveys, a general-purpose computer program used for obtaining cross-tabulations will normally also indicate significance levels for the data in the table.

Online Survey Champion

Online Survey Champion

There are people all over the world trying to find ways to make money online. From stay at home moms looking to make a few extra dollars to college students and entrepreneurs, the allure of making your own hours and working from home or from the local coffee shop is very appealing.

Get My Free Ebook

Post a comment