### tl;dr

Today there’s a new module available in jamovi: **esci** (effect sizes and

confidence intervals), developed by Bob Calin-Jageman and Geoff Cumming

(@TheNewStats and

TheNewStatistics.com).

As a newer module you will need a recent version of jamovi to install

esci-–probably 1.2.19 or above. You can refresh your install of jamovi

here.

esci provides an easy step into estimation statistics (aka the “new

statistics”), an approach that emphasizes effect sizes, interval estimates,

and meta-analysis. esci can provide estimates and confidence intervals for

most of the analyses you would learn in an undergraduate statistics course

**and** meta-analysis (which really should be part of a good undergraduate

statistics course). Most analyses can be run from raw data or from

summary data (enabling you to generate estimates from journal articles

that only reported hypothesis tests). All analyses generate nice

visualizations that emphasize effect sizes and uncertainty. esci is

for everyone, but was developed especially with students in mind–it

provides step-by-step instructions, clear feedback, and tries to prevent

rookie mistakes (like calculating a mean on a nominal variable).

## What is estimation statistics?

Inferential statistics has two major traditions: testing and estimation. The

testing approach is focused on decision-making. In this approach we propose a

null hypothesis, collect data, generate a test-statistic and p-value measuring

the degree to which the null hypothesis is compatible with the data, and then

make a decision about the hypothesis. For example, we might test the null

hypothesis that a drug has exactly 0 effect on depression. We collect data from

those randomly assigned to take the drug or placebo. We run a t-test comparing

these groups and find *p* = .01. We then make a decision: because *p* < .05 we

reject the null hypothesis, deciding that an effect of exactly 0 is not

compatible with the data. Huzzah.

The testing approach has its uses, but note two important issues that we have

not been addressed: 1) *How much does the drug work?* and 2) *How wrong might
we be?* That’s where estimation comes in. From the same data and assumptions

that underlie the testing approach we can generate an estimate and a confidence

interval. So, for example, we might find that the drug improved depression by

10% with a 95% CI of [1%, 19%]. This is some very useful information. It tells

us how well the drug worked in this one study (10% benefit). It also gives us

an expression of uncertainty about this estimate. Specifically, the CI gives

the entire range of benefits that are compatible with the data collected–

benefits around 1% are compatible and so are benefits around 19%.

Focusing on estimates can be really helpful:

- It helps us weigh practical significance rather than just statistical

significance. - It helps us calibrate our conclusions to the uncertainty of the study
- It fosters meta-analytic thinking, where we combine data from multiple

studies to refine our estimates (like the poll aggregators on

fivethirtyeight.com) - It calibrates expectations for replications
- It helps us think critically about optimizing procedures to maximize effect

sizes and minimize noise - And much more

Estimates and tests are linked. A null hypothesis is rejected at the alpha =

.05 level if it is outside a 95% CI and not rejected if it is inside. To put it

a different way, a 95% CI is all the null hypotheses you would not reject at

alpha .05 (and a 99% CI all those for alpha .01, etc.). This means that if you

have an estimate, you can still conduct a test–in fact you can test any null

hypothesis just by checking for it in the CI. The converse is not true, though:

knowing that a test is statistically significant does not easily let you know

the magnitude of the effect or the uncertainty around it. So when you focus on

estimation you gain some benefits, but you don’t lose anything. That makes it

rather bizarre that some fields have come to use only testing. esci is part of

an effort to change this around, and to make estimation the typical or default

approach to inference.

Want to know more about estimation? Here are some sources:

- Undergraduate textbook: Cumming, G., & Calin-Jageman, R. J. (2017).

Introduction to the new statistics: Estimation, open science, and beyond.

New York: Routledge. On Amazon. - Calin-Jageman, R. J., & Cumming, G. (2019). The New Statistics for Better

Science: Ask How Much, How Uncertain, and What Else Is Known. The American

Statistician, 73(sup1), 271–280.

https://doi.org/10.1080/00031305.2018.1518266

## An example with esci

Let’s use esci to re-analyze data from a famous paper about the “trust drug”

oxytocin. Oxytocin is a neurohormone best known for its role in human

reproduction. But in 2005, Kosfeld et al. followed up on some interesting work

in rodents to examine if oxytocin might influence trust in humans. The

researchers randomly assigned participants to receive oxytocin (squirted up the

nose) or placebo (also squirted up the nose) before playing an investment game

that depended on trusting an anonymous partner. The average amount invested by

each participant was used as a measure of trust. Kosfeld et al. found that

oxytocin produced a statistically significant increase in trust

(*t*(56) = 1.82, *p* = .037 one-tailed)*.

That sounds pretty convincing, right? It must have been, as the paper was

published in Nature and has now been cited over 4,000 times. Right from the

start, citations made the effect seem established and unequivocal. But how much

did oxytocin improve trust and how wrong might this study be?

Let’s take a look. The original data is available in .csv format

here**. Opening it in jamovi you can conduct a

standard t-test to confirm that the difference is statistically significant

(for a directional test). Now let’s generate the estimate and CI in esci using

“Estimate Independent Mean Difference”.

In the analysis options, we’ll enter Trust as the dependent variable and

Condition as the grouping variable (placebo was coded as a 0; oxytocin as a 1).

We’ll also set the confidence level to 90% to match the stringency of a

directional test.

Our output emphasizes the effect size, which in this case is the difference in

means, and reports this as both a raw difference (with a CI) and as a

standardized difference (also with a CI):

esci also generates a *difference plot*. This shows the oxytocin data (all

participants and the group mean with CI) and the placebo data (all participants

and the group mean with CI). Most importantly, the graph emphasizes the

difference between them: we draw a line from the placebo group, considering

that our benchmark, and then we measure the space between the groups, marking

the difference (delta) with a triangle on a right-side axis anchored at 0 to

the placebo group. It sounds a bit complicated to write it out, but just take a

look.

The graph shows that the difference in trust was fairly huge–a $1.41 increase

in investment in a context where a typical investment was $8-9. The change,

though, is highly uncertain, with a 95% CI that runs from $0.11 up to $2.71.

This means the data is compatible with a very large range of effect sizes–from

the vanishingly small to the dazzlingly large. In other words, this study

doesn’t really tell us much about how much oxytocin might influence trust.

Perhaps not 0, but basically almost any other positive effect size is on the

table, including ones (around $0.11) that would be very difficult to replicate.

Looked at with these eyes, it might not surprise you much to find out that the

benefit of oxytocin in human trust has not replicated well–and that the

consensus is that oxytocin probably does not have a *practically significant*

effect on trust. Unfortunately this was not obvious to researchers wedded to

the testing approach, and so much faith was put in these results that clinical

trials were launched to try to use oxytocin as a therapy for social processing

deficits (such as with autism-spectrum disorder). None of these clinical trials

have shown much benefit, but they’ve cost a ton and produced a decent handful

of (thankfully mild) adverse reactions. If you’re curious about the way the

oxytocin story imploded at great costs and hardship, check out the article

here.

This is just one example of how you can gain important insight into your data

by using estimation thinking in place of or as a supplement to testing. esci

should make it easy to get started with this approach.

* — In the original study the researchers didn’t actually use a t-test; they

compared median trust using a non-parametric test. This nuance doesn’t alter

the patterns in the data presented in this post.

** — This data was extracted by Bob and Geoff from a figure in Kosfeld et al.

(2005). The OSF page where it is posted has all the details.

## I’m used to running this test… what would I use in esci?

Glad you asked. Here’s how the most common statistical tests map on to the

estimates generated by esci:

Traditional hypothesis test |
esci in jamovi command |

One-sample t-test | Estimate Mean |

Independent samples t-test | Estimate Independent Mean Difference |

Paired samples t-test | Estimate Paired Mean Difference |

One-Way ANOVA | Estimate Ind. Group Contrasts |

2×2 ANOVA | Estimate Ind. 2×2 |

2×2 Chi Squared | Estimate Proportion Difference |

Correlation test | Estimate Correlation |

Correlation test with categorical moderator | Estimate Correlation difference |

## This module would be better if…

The esci module is still in alpha. Geoff and Bob have made this initial release

to help gather feedback as they continue to work on the module in conjunction

with a new edition of their statistics textbook. They welcome your feedback,

feature requests, and/or bug reports. Please especially consider esci through

the eyes of your students:

- What other analyses would you like to see?
- Anything in the output that is hard to understand? That should be labelled

better? That should be added or could be removed? - Would it be helpful to add the option to see all assumptions for an analysis?

Should we provide more guidance on interpreting output? - Any options missing from analyses?

The best way to provide feedback would be on the github page for this module,

which is here: https://github.com/rcalinjageman/esci.

If that’s a hassle, then by all means just email Bob directly or tweet at them

@TheNewStats.

## Haven’t I heard of this before?

Yes – Geoff Cumming has been developing versions of esci for some time. The

original versions were designed as worksheets in Excel. And in addition to

analyses, the older version of esci has some great simulations and sample-size

planning tools. You can still check these out here:

https://thenewstatistics.com/itns/esci/.

## Be the first to comment