Experimental Mind: New Vistas, Sobering Truths, Delayed Reward Forecasting, Illegal Popups …

Hey friends,

Last week I started in my new role as Product Manager Experimentation at Vista. After well over decade at ING, I am really excited to start this new adventure. Almost everything is different, except for two things; my home office and the challenge ‘to make experimentation the default option by reducing friction’.

After missing last week’s newsletter (after spending a full week in the Scottish mountains), here are again some new articles, job opportunities and an updated list of upcoming events.

Everything freshly picked and curated, so you don’t have to. Enjoy!


๐Ÿ”Ž What I’ve been reading this week

1. The sobering truth about the impact of business ideas

Many companies simply execute on their business ideas without measuring if they delivered the impact that was expected. But, science-based organizations are rigorously quantifying this impact and have learned some sobering lessons.

2. Interpreting A/B test results: false negatives and power

This is the fourth post in a multi-part series on how Netflix uses A/B tests to inform decisions and continuously innovate on our products.

3. Speeding up A/B Tests with Delayed Reward Forecasting

Some experiments at Wayfair can last 60 days or more. To speed up learning in experiments while still optimizing for long term rewards, our team developed a data science platform called Demeter, that uses ML models to forecast longer term KPIs based on customer activity in the short term. In this post we provide an overview for Demeter and its theoretical foundation in causal inference.

4. Universal holdout groups at Disney Streaming

To overcome the limitations of traditional A/B testing, Disnet Streaming borrowed a tool which is common in marketing campaigns: the universal holdout group.

5. IAB Europe consent popup deemed illegal

In proceedings initiated by a group of complainants coordinated by the Irish Council for Civil Liberties, the Belgian Data Protection Authority is close to adopting a draft decision that will find IAB Europeโ€™s โ€œconsentโ€ pop-up system infringes the GDPR, vindicating our arguments over several years.

๐Ÿš€ Job opportunities

Are you looking for a new opportunity? There are now 17 open roles on the job board, including my previous role at ING.

Are you hiring? Post your open roles for free to the job board. (this is only for readers of this newsletter, make sure to use this code at the last step: SUBSCRIBER200)

๐Ÿ“… Upcoming events

This is a running list of upcoming events:

๐Ÿ’ฌ Quote of the week

“The country needs and, unless I mistake its temper, the country demands bold, persistent experimentation. It is common sense to take a method and try it: If it fails, admit it frankly and try another. But above all, try something.” โ€” Franklin Delano Roosevelt, May 22, 1932

๐Ÿ˜‰ Sketch of the week

Jono Hey from Sketchplanations.com on ‘bosses versus leaders

Bosses push information up to authority. Leaders push authority down to information.

๐Ÿ” Any feedback?

I like to hear what you think of this newsletter, Share your feedback, it takes less than a minute.

๐Ÿ™ Thanks for reading

If you’re enjoying the Experimental Mind newsletter you can buy me a coffee or beer. Thanks to the 10 people who already did it. Much appreciated.

And share this email with a friend or colleague. I would love this community of experimenters to grow. You can point them here to subscribe.

Have a great week โ€” and keep experimenting.

Kevin