Quite often a data set does not cover an entire population, but is just a sample of the population. If the sample is small, there might be considerable uncertainty in applying the measured results from the sample to the population. OfficeReports provides a number of statistical tests for tables and charts. These can be used to show whether differences in the distribution of counts in tables are statistically significant or whether they are merely due to chance.
Clicking 'Statistics' will pop-up the following window:
Here a short explanation for the different settings:
Confidence levels for T-Test and Z-Test
You can select the confidence level that you require for the Z-Test (column proportions) and T-Test (column means). The default setting is 95%, but your options are from 99% down to 50%. By default, OfficeReports will display the difference with uppercase characters for the first 26 columns, lowercase characters for the next 26 columns ( in case the second confidence level is not used) and special characters up to column 96.
Optionally, a Confidence level 2 can be added in the test. The test level must be lower than Confidence level 1. OfficeReports will display the difference with lowercase characters. This means that we in one table can include both a significance test on e.g. a 95% and one a 90% level of confidence.
Read more about specifying which columns to test and how to visualize significant difference.
Minimum Base & Low Base
You can define that the significance test requires a minimum number of observations. Office Report suggest a minimum base of 30. In case you define a 'minimum base' character, this character will be added to the numbers in the table or chart in case the base is below the specified minimum. The same goes for the 'low base'.
Observe: The notification is based on the unweighted base.
Always show base notifications
By default, OfficeReports shows Minimum Base and Low Base only in case significance testing is applied. Check the ‘Always show base notifications’ box to if you always want to see the base notifications.
T-Test (Column Mean Test)
This test looks at means that are presented in a row of a table and compares pairs of columns, testing whether the mean in one column is significantly different from the mean in the other column.
By default, OfficeReports is using the Student’s T-Test, compensating for overlap. Alternatively, OfficeReports can use Welch’s T-Test. The Welch test does not assume equal variances and computes the degrees of freedom empirically using a function of the group sizes and the variances.
Z-Test (Column Proportions Test)
This test looks at the rows of a table independently and compares pairs of columns, testing whether the proportion of respondents in one column is significantly different from the proportion in the other column. The proportion is the count in the cell divided by the base of the column.
In situations where results are weighted, OfficeReports by default uses the Effective Base rather than the simple weighted base in statistical tests on weighted tables. This option has no effect on statistical tests run on unweighted tables.
The effective base is designed to reduce the likelihood of the statistical tests producing signiﬁcant results because of the adjustments made by weighting; the effective base takes these adjustments into account.
The setting of the Confidence Level defines how secure we want to be to be to express a statistical test result. When we for example claim that a certain difference of percentages or means is significant, or that a certain result 11% has a confidence interval at +/- 1,1 this does not mean that we can be 100% sure of this result, but we can be sure on a certain level.
Setting the Confidence level is set at 95% (most commonly used level in market research) this is an indication that we can be 95% sure that the result is significant or that we can be 95% sure that the true result in the population is within 9,9 -12,1%.
When adding confidence intervals to a table, OfficeReports by default assume that the sample represents a large population and the sample represents less than 10% of the population. If the sample represents a significant proportion of the population (> 10%) we can enter the Population Size in order to adjust the test formula.
Confidence Intervals can only be added to frequency tables, where an extra column will displaying the confidence level, while an extra row will show the 'mean confidence interval' in case the table contains a 'mean' row:
Chi-Square tests can be used in cross tables. The chi-square test looks at the variables on the side and top axes of a table and tests whether they are independent. For example, it can be used to show whether variations of a rating depend on gender:
|Test type||Available for:|
|Confidence Intervals||Frequency tables|
|Chi-Square test||Cross tables|
|Z-test (Column Proportions Test)||Cross tables and charts, Cross Grid tables and charts|
|T-test (Column Means Test)||Cross tables and charts, Cross Grid tables and charts|
|Correlation Test||Frequency Grid tables, Frequency Grid charts|