- False discovery using standard statistical methods is a perennial headache.
- Indeed false discovery has been blamed for the non-repliccability of many studies.
- The Benjamini-Hochberg, BH, procedure is widely recommended to help solve this problem

This blog provides an EXCEL spreadsheet to estimate the number of ‘true’, i.e. after correction for alse discovery using BH procedure. It also rpesents a behavioural scenario example of the dilemmas posed by false discovery.

Reference: Benjamini, Y., & Hochberg, Y. (1995). Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing. Journal of the Royal Statistical Society. Series B (Methodological), 57(1), 289-300. http://www.jstor.org/stable/2346101

## EXCEL spreadsheet for Benjamini-Hochberg procedure

****NOTE Instructions modified 15/10/2017. Many thanks to Alex Agache – see comment****

BejaminiHochbergFalseDiscoveryKornbrotbejaminihochbergfalsediscoverykornbrot15Oct17

# Scenario and Dilemma

There is a large organisation assessing customer satisfaction with **3200** units. Customer satisfaction is assessed annually for each unit.

NB this is REAL data. It is part of a project to evaluate efficacy of different methods for analysing ordinal data, using real [not simulated] data.

### Research Question

Is there any change in satisfaction between this year and last year?

## Design

- Each customer assesses just 1 unit in only 1 year.
- Satisfaction is measured on a 5-point (Likert) scale:
- strongly agree, agree, neither agree nor disagree, disagree, strongly disagree

### Data analysis

- Binary logistic regressions are conducted separately for all 3200 units.
- Response is valence, V. V = 1 if response is agree or strongly agree, V = 0 for all other responses. Predictor is Year, confidence level p is .05, 2-tailed.

## Results

### No control for false discovery

**352**(10.9%) units had change significant at 95% confidence level**215**units improved,**137**units declined

### Benjamini-Hochberg Control for false discovery

- The Benjamini-Hochberg procedure with false discovery rate
**q =.05**gave**p = .0008**as the corrected decision criterion. This led to inference that **7**(.2%)units had changed**4**units improved,**3**units declined

## Conclusions?

At the organisation level, the change was clearly, negligible after correcting for false discovery.

**BUT **

What about the 347 individual units with significant change at the p =.05 level, but no change at the p = .0008 ‘protected’ level? At the single individual unit level, should assessment be affected by what did, or did not, happen in the other 3218 units?

# Comments & advice welcome

Thank you for this excel calculator, but shouldn’t the instructions be:

1. Clear p-values from column A

2. Insert your p-values in column A (not C)

3. Sort in ascending order column A

etc. and the B-H adjusted p- value will appear in cell G3

Many thanks I have now modified file