Hey friends,
Last week I started in my new role as Product Manager Experimentation at Vista. After well over decade at ING, I am really excited to start this new adventure. Almost everything is different, except for two things; my home office and the challenge ‘to make experimentation the default option by reducing friction’.
After missing last week’s newsletter (after spending a full week in the Scottish mountains), here are again some new articles, job opportunities and an updated list of upcoming events.
Everything freshly picked and curated, so you don’t have to. Enjoy!
🔎 What I’ve been reading this week
1. The sobering truth about the impact of business ideas
Many companies simply execute on their business ideas without measuring if they delivered the impact that was expected. But, science-based organizations are rigorously quantifying this impact and have learned some sobering lessons.
2. Interpreting A/B test results: false negatives and power
This is the fourth post in a multi-part series on how Netflix uses A/B tests to inform decisions and continuously innovate on our products.
3. Speeding up A/B Tests with Delayed Reward Forecasting
Some experiments at Wayfair can last 60 days or more. To speed up learning in experiments while still optimizing for long term rewards, our team developed a data science platform called Demeter, that uses ML models to forecast longer term KPIs based on customer activity in the short term. In this post we provide an overview for Demeter and its theoretical foundation in causal inference.
4. Universal holdout groups at Disney Streaming
To overcome the limitations of traditional A/B testing, Disnet Streaming borrowed a tool which is common in marketing campaigns: the universal holdout group.
5. IAB Europe consent popup deemed illegal
In proceedings initiated by a group of complainants coordinated by the Irish Council for Civil Liberties, the Belgian Data Protection Authority is close to adopting a draft decision that will find IAB Europe’s “consent” pop-up system infringes the GDPR, vindicating our arguments over several years.
🚀 Job opportunities
Are you looking for a new opportunity? There are now 17 open roles on the job board, including my previous role at ING.
Are you hiring? Post your open roles for free to the job board. (this is only for readers of this newsletter, make sure to use this code at the last step: SUBSCRIBER200)
📅 Upcoming events
This is a running list of upcoming events:
- CJAM Session with Nathalie Nahai & Bart Schutz (Nov 12, online)
- CJAM Session with Dana di Tomaso & Collin Tate Crowell (Nov 18)
- The Conference formerly known as Conversion Hotel (Nov 19-21, NL)
- CJAM Session with Nir Eyal & Hazjier Pourkhalkhali (Nov 23, online)
- Design The Next Step (Nov 25, 2021)
- CJAM Session with Terry Pierelli & Erin Weigel (Dec 2, online)
💬 Quote of the week
“The country needs and, unless I mistake its temper, the country demands bold, persistent experimentation. It is common sense to take a method and try it: If it fails, admit it frankly and try another. But above all, try something.” — Franklin Delano Roosevelt, May 22, 1932
😉 Sketch of the week
Jono Hey from Sketchplanations.com on ‘bosses versus leaders‘
Bosses push information up to authority. Leaders push authority down to information.
🔁 Any feedback?
I like to hear what you think of this newsletter, Share your feedback, it takes less than a minute.
🙏 Thanks for reading
If you’re enjoying the Experimental Mind newsletter you can buy me a coffee or beer. Thanks to the 10 people who already did it. Much appreciated.
And share this email with a friend or colleague. I would love this community of experimenters to grow. You can point them here to subscribe.
Have a great week — and keep experimenting.
Kevin
