- False discovery using standard statistical methods is a perennial headache.
- Indeed false discovery has been blamed for the non-repliccability of many studies.
- The Benjamini-Hocberg, BH, procedure is widely recommended to help solve this problem
This blog provides an EXCEL spreadsheet to estimate the number of ‘true’, i.e. after correction for alse discovery using BH procedure. It also rpesents a behavioural scenario example of the dilemmas posed by false discovery.
Reference: Benjamini, Y., & Hochberg, Y. (1995). Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing. Journal of the Royal Statistical Society. Series B (Methodological), 57(1), 289-300. http://www.jstor.org/stable/2346101
EXCEL spreadsheet for Benjamini-Hochberg procedure
Scenario and Dilemma
There is a large organisation assessing customer satisfaction with 3200 units. Customer satisfaction is assessed annually for each unit.
NB this is REAL data. It is part of a project to evaluate efficacy of different methods for analysing ordinal data, using real [not simulated] data.
Is there any change in satisfaction between this year and last year?
- Each customer assesses just 1 unit in only 1 year.
- Satisfaction is measured on a 5-point (Likert) scale:
- strongly agree, agree, neither agree nor disagree, disagree, strongly disagree
- Binary logistic regressions are conducted separately for all 3200 units.
- Response is valence, V. V = 1 if response is agree or strongly agree, V = 0 for all other responses. Predictor is Year, confidence level p is .05, 2-tailed.
No control for false discovery
- 352 (10.9%) units had change significant at 95% confidence level
- 215 units improved, 137 units declined
Benjamini-Hochberg Control for false discovery
- The Benjamini-Hochberg procedure with false discovery rate q =.05 gave p = .0008 as the corrected decision criterion. This led to inference that
- 7 (.2%)units had changed 4 units improved, 3 units declined
At the organisation level, the change was clearly, negligible after correcting for false discovery.
What about the 347 individual units with significant change at the p =.05 level, but no change at the p = .0008 ‘protected’ level? At the single individual unit level, should assessment be affected by what did, or did not, happen in the other 3218 units?
Comments & advice welcome