Posts

Robust Power Assessment

An important part of planning any statistical experiment is power analysis. In this post I will focus on power analysis for linear regression models, but I am hopeful much of this can be applied to Generalized Linear Models and hence to the sorts of A/B tests I normally run.
Aug 12, 2021 6 min read

Scheffe's Method for Multiple Comparisons

I’ve written previously about using the Bonferroni correction for the multiple comparisons problem. While it is without a doubt the simplest way to correct for multiple comparisons, it is not the only way. In this post, I discuss Scheffé’s method for constructing simultaneous confidence intervals on arbitrarily many functions of the model parameters.
Aug 1, 2021 10 min read

Supervised Learning as Function Approximation

Supervised learning is perhaps the most central idea in Machine Learning. It is equally central to statistics where it is known as regression. Statistics formulates the problem in terms of identifying the distribution from which observations are drawn; Machine Learning in terms of finding a model that fits the data well.
May 15, 2021 4 min read

Naive Comparisons Under Endogeneity

Recently I have been reading Causal Inference: The Mixtape by Scott Cunningham. One thing I think Cunningham explains very well is the role of endogeneity in confounding even simple comparisons. I don’t have a background in economics, so I had never really grokked the concepts of endogenous and exogenous factors, especially as it related to causal inference.
Feb 28, 2021 9 min read

Advice for Early Career Data Scientists

Coming out of college, I had some ideas about how I was going to become successful and what my career was going to look like. Of course, I was all wrong. Here is the advice I would offer a young me.
Jan 24, 2021 6 min read

Multiple Comparisons

The simplest kind of A/B test compares two options, using a single KPI to decide which option is best. The more general theory of statistical experiment design easily handles more options and more metrics, provided we know how to incorporate the multiple comparisons involved. To see why this is important, read on!
Jan 23, 2021 6 min read

Violations of the Stable Unit Treatment Value Assumption

We have previously mentioned the Stable Unit Treatment Value Assumption, or SUTVA, a complicated-sounding term that is one of the most important assumptions underlying A/B testing (and Causal Inference in general).
Jan 10, 2021 4 min read

2020: My Year in Emacs

Other than the very fabric of society being torn apart, and other than the silver lining of getting to spend so much time with my wife and 2 year old daughter, the big theme of 2020 for me personally was Emacs.
Jan 1, 2021 7 min read

Statistics and Machine Learning: Better Together!

My masters degree focused on Machine Learning, but when I got my first job as a data scientist, I quickly realized there was a lot I still needed to learn about Statistics. Since then I have come to appreciate the nuanced differences between Statistics and Machine Learning and I’m convinced they have a lot to offer one another!
Jan 1, 2021 6 min read

Contingency Tables Part IV: The Score Test

The score test can be used to calculate p-values and confidence intervals for A/B tests. The score test considers the slope of the likelihood function at the parameter value associated with the null hypothesis.
Nov 24, 2020 11 min read