Experimental Mind: what makes a metric good?

Someone asked me how I would define ‘culture of experimentation’. My answer: an environment where everyone is open to change their mind. This is not only about adjusting your opinion based on new evidence. It’s about actively seeking information that’s counter to your current thinking.

That is why I read. Only the best I will include in this newsletter.


πŸ”Ž What I’ve been reading

STEDII Properties of a good metric

From the Microsoft Experimentation team:

Good metrics enable good decisions. What makes a metric good? In this blog post we introduce the STEDII (Sensitivity, Trustworthiness, Efficiency, Debuggability, Interpretability, and Inclusivity) framework to define and evaluate the good properties of a metric and of an A/B test analysis in general.

You’re already a Data Analyst

Thinking of starting a career as Data Analyst, but think you lack the knowledge and experience? Cassie Kozyrkov makes the case that you’re already are one.

Ethical principles for influencing & changing behaviour

Bart Schutz applies the five ethical principles for influencing and changing behaviour at scale to a very concrete example from industry.

How digital design can harm competition and consumers
Two papers from the CMA’s Behavioural Hub discussing and summarising evidence on online choice architecture (OCA) and how it potentially causes harm.

'Ghost Ads' paper wins award

The ghost-ads methodology works with modern online ad platforms to reduce the cost of experimentation and improve measurement precision. … Today, several advertising platforms offer ghost ad methodologies to measure ad effectiveness.

How Farfetch approaches 'build vs buy': hybrid

In Ronny Kohavi’s class on ‘Accelerating Innovation with A/B Testing’, Luis Trindade presented Farfetch’s approach to building and/or buying experimentation tooling.

πŸ“’ In case you missed Ronny Kohavi’s class, you can join cohort #4 and he will teach you everything he knows about A/B testing in 5 two hour sessions. Subscribe via this link and you will receive a $50 Amazon gift card.

How to use correlation to make predictions

HBR article by Dean Karlan and Michael Luca:

To better leverage data, leaders need to understand the types of problems data can help solve as well as the difference between those problems that can be solved with improved prediction and those that can be solved with a better understanding of causation.

Convert.com first vendor to add SRM checks

Sample (Size) Rate Mismatch #SRM feature was added to our platform and instantly 30% of our customers had it! Looking deeper it’s 6-7% of all experiments.

Who will follow?

πŸš€ Job opportunities

Featured: CRO Specialist at Randstad
Featured: CRO Specialist at Randstadwww.randstad.com

Ready to lead a global experimentation program? As CRO Specialist at Randstad Global you will help countries validate insights, identify key areas for improvement and hypotheses for testing.

All featured roles:

Also checkout the job board for more open roles in the field of experimentation, CRO and analytics.

πŸ“… Upcoming events

Upcoming events in the next months:

See the full overview of events to find 15+ other events and conferences for you and your team.

πŸ’¬ Quote of the week

β€œFight as if you are right, and listen as if you are wrong.” β€” Karl Weick

πŸ˜‰ Fun of the week

Via Manuel Da Costa:

β˜• Thanks for reading

If you’re enjoying the Experimental Mind newsletter and want to say thank you? Buy me a coffee (or a beer). Or share this newsletter with a friend or colleague.

Have a great week β€” and keep experimenting.

Kevin