False Discovery Benjamini-Hochberg

  • False discovery using standard statistical methods is a perennial headache.
  • Indeed false discovery has been blamed for the non-repliccability of many studies.
  • The Benjamini-Hochberg, BH, procedure is widely recommended to help solve this problem

This blog provides an EXCEL spreadsheet to estimate the number of  ‘true’, i.e. after correction for alse discovery using BH procedure. It also rpesents a behavioural scenario example of the dilemmas posed by false discovery.

Reference: Benjamini, Y., & Hochberg, Y. (1995). Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing. Journal of the Royal Statistical Society. Series B (Methodological), 57(1), 289-300. http://www.jstor.org/stable/2346101

EXCEL spreadsheet for Benjamini-Hochberg procedure

**NOTE Instructions modified 15/10/2017. Many thanks to Alex Agache – see comment**

BejaminiHochbergFalseDiscoveryKornbrotbejaminihochbergfalsediscoverykornbrot15Oct17

Scenario and Dilemma

There is a large organisation assessing customer satisfaction with 3200 units. Customer satisfaction is assessed annually for each unit.

NB this is REAL data. It is part of a project to evaluate efficacy of different methods for analysing ordinal data, using real [not simulated] data.

Research Question

Is there any change in satisfaction between this year and last year?

Design

  • Each customer assesses just 1 unit in only 1 year.
  • Satisfaction is measured on a 5-point (Likert) scale:
  •  strongly agree, agree, neither agree nor disagree, disagree, strongly disagree

Data analysis

  • Binary logistic regressions are conducted separately for all 3200 units.
  • Response is valence, V. V = 1 if response is agree or strongly agree, V = 0 for all other responses. Predictor is Year, confidence level p is .05, 2-tailed.

Results

No control for false discovery

  • 352 (10.9%) units had change significant at  95% confidence level
  • 215 units improved, 137 units declined

Benjamini-Hochberg Control for false discovery

  • The Benjamini-Hochberg procedure with false discovery rate q =.05 gave p = .0008 as the corrected decision criterion. This led to inference that
  • 7 (.2%)units had changed 4 units improved, 3 units declined

Conclusions?

At the organisation level, the change was clearly, negligible after correcting for false discovery.

BUT

What about the 347 individual units with significant change at the p =.05 level, but no change at the p = .0008 ‘protected’ level? At the single individual unit level, should assessment be affected by what did, or did not, happen in the other 3218 units?

Comments & advice welcome

 

 

Advertisements

2 Responses to “False Discovery Benjamini-Hochberg”

  1. Alex Agache Wed, 20-Sep-17 at 11:54 am #

    Thank you for this excel calculator, but shouldn’t the instructions be:
    1. Clear p-values from column A
    2. Insert your p-values in column A (not C)
    3. Sort in ascending order column A
    etc. and the B-H adjusted p- value will appear in cell G3

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: