Methods Matter

Background

Statistics a Pervasive Scientific Method

Statistical analysis has long been a mainstay of scientific research.

Descriptive statistics are used for summaries, according to group or intervention e.g. arithmetic mean, median, standard deviation, Pearson correlation, regression coefficient, etc. (parametric); or median, range, Spearman’s correlation (rank based and commonly called ‘non-parametric)

Inferential statistics are used to determine whether effects are ‘significant’, i.e. unlikely to have occurred by chance if there is no difference in the population, e.g. t-tests, ANOVA, linear regression (parametric); Mann-Whitney, Kruskal-Wallis, Friedmann (rankbased).

History

Normal Distribution & Rank Methods

Historically the parametric methods used have been mostly based on the assumption that variables are metric, normally distributed and equal variance in the target population. A major reason for using these methods is that they are computationally feasible, even without a computer, and widely available in popular statistical packages for both between group and within group models (SPSS Minitab, EXCEL add-ons, SAS, etc.). The rank based have mostly been used when data is known to be ordinal, e.g. rating scales with few categories, Likert items, but have the limitation that they can only be used with a single predictor variable.

Alternatives

Generalized Linear Mixed Models, transformation

It has been known for decades that much real data violates any or all of the assumptions of metric and/or normally distributed and/or equal variance. Furthermore, generalized linear mixed models for appropriate analyses have also been known and available in more advanced or specialized software (MLwin, R).

These methods are now available and relatively easy to implement using graphic interfaces not requiring scripting in SPSS.

Does it matter?

Are normal based methods ‘robust’?

It is often claimed that traditional normal base models are ‘robust’. i.e. the same predictors are statistically significant whether using normal (traditional, for short) or generalized linear mixed model (GLM, for short) methods.

Many comparisons using simulations show that traditional & GLM models give substantially different results. However, there is relatively little work with real data.

Methods Matter Challenges

The project will compare traditional and GLM methods for target data sets for particular hypotheses. Each comparison will start with a poll asking readers to predict concordance between methods: % same in both analyses (either both significant or both non significant; significant with traditional methods only; significant with GLM methods only. After 1 week actual results will be revealed.

Challenge 1

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: