Experimental Mind: iteration, awards, autocorrelation, overtracking, CNN+ …

In this weekly newsletter I share the best experimentation related articles, jobs and events, that I have found during the week. This edition is chuck-full.

You only have time for one thing?

Make sure your read Amazon’s 2021 Letter to Shareholders.

🔎 What I’ve been reading

Amazon's 2021 Letter to Shareholders

This is the first letter by new CEO Andy Jassy and it’s all about iteration:

People often assume that the game-changing inventions they admire just pop out of somebody’s head, a light bulb goes off, a team executes to that idea, and presto—you have a new invention that’s a breakaway success for a long time. That’s rarely, if ever, how it happens.

One of the lesser known facts about innovative companies like Amazon is that they are relentlessly debating, re-defining, tinkering, iterating, and experimenting to take the seed of a big idea and make it into something that resonates with customers and meaningfully changes their customer experience over a long period of time.

Experimentation Culture Awards 2022
Experimentation Culture Awards 2022 — experimentationcultureawards.com

Submit your case now for the 2022 Experimentation Culture Awards. Deadline is May 14th.

The Dunning-Kruger effect is autocorrelation

The point of this story is to illustrate that the Dunning-Kruger effect has nothing to do with human psychology. It is a statistical artifact — an example of autocorrelation hiding in plain sight.

Overtracking and trigger analysis: reducing sample sizes while increasing the sensitivity of experiments

Tracking users we cannot treat (called overtracking) affects the variance of our metrics and dilutes the treatment effect, making its detection harder.

CNN+ shuts down after one month

Alberto Savoia explains how CNN could have tested this concept earlier.

If you build It, they will come … NOT. After just a few weeks, CNN+ shut down. This idea would have been SOOOOO EASY to pretotype before pouring in $100M (or more.)

On Causal Inference with Guido Imbens
On Causal Inference with Guido Imbens — causalinf.substack.com

Interview with 2021 Nobel Prize recipient in Economics, Guido Imbens.

There’s more to learn from tests
There’s more to learn from tests — blog.statsig.com

Testing is a tool for measuring the effect of a change in isolation. Launch tests are an important use of that tool, but only one use.

A/B testing platform health with simulated A/A tests

… we run 500 hundred simulated A/A tests every day using data accumulated from the last seven days. Each simulated A/A test randomly splits users into control and treatment groups and calculates the metrics of the two experiment groups in the same ways as the production metric calculation pipeline.

Shifting your mindset from amateur to professional analyst
Shifting your mindset from amateur to professional analyst — towardsdatascience.com

Cassie Kozyrkov shares the next 3 out of 10 differences between amateurs and professionals.

[Dutch] Je eigen A/B testing platform bouwen

Kaartje2go deed het. En Arnoud Huberts legde op Emerce Conversion uit hoe en waarom. Kaartje2go is ook op zoek naar drie nieuwe toppers: zie hieronder.

🚀 Job opportunities

All featured roles:

Also checkout the job board for more open roles in the field of experimentation, CRO and analytics.

📅 Upcoming events

This week:

Next months:

See the full overview of events to find 15+ other events and conferences for you and your team.

💬 Quote of the week

“We want people who like to experiment and tinker, and who realize launch is the starting line, not the finish line.” — Amazon’s 2021 Letter to Shareholders

😉 Fun of the week

The latest Marketoonist on product-market fit.

☕ Thanks for reading

If you’re enjoying the Experimental Mind newsletter and want to say thank you? Buy me a coffee (or a beer). Or share this newsletter with a friend or colleague.

Have a great week — and keep experimenting.

Kevin

Related Posts