Experimental Mind: Netflix’s Culture of Learning

Hey everyone,

As always, I have handpicked the best experimentation related articles, jobs and events, so you don’t have to. Simply grab a cup of coffee and dive in.

ps. I’m looking for a couple of guest curators to help create one or more issues in 2022. If this sounds interesting, simply reply and I’ll be in touch.


πŸ”Ž What I’ve been reading last week

1. Netflix: A Culture of Learning
1. Netflix: A Culture of Learningnetflixtechblog.com

Earlier posts in this series covered the why, what and how of A/B testing, all of which are necessary to reap the benefits of experimentation for product development. But without a little magic, these basics are still not enough. The secret sauce that turns the raw ingredients of experimentation into supercharged product innovation is culture.

After reading all blog posts in this series, it’s clear that Netflix is light years ahead of many companies, with regards to experimentation and the culture around decision making. In case you missed earlier blog posts, here is the full overview. Read, share and discuss how you make this happen in your org.

  1. Decision making at Netflix
  2. What is an A/B Test?
  3. Interpreting A/B test results: false positives and statistical significance
  4. Interpreting A/B test results: false negatives and power
  5. Building confidence in a decision
  6. Experimentation is a major focus of data science across Netflix
  7. Netflix: A Culture of Learning

2. How Microsoft built their experimentation platform

Microsoft also has a world–class scalable experimentation platform. In this blog post they explain how it has been built:

  1. Develop a long-term vision and measure progress
  2. Keep a pulse on (internal) customers
  3. Create a culture of collaboration

3. [Paper] A Large-Scale Exploration of Non-Experimental Approaches to Advertising Measurement

A paper that looks at the effectiveness of alternative methods to measure the of Facebook ads:

… broadly speaking, our results suggest that state-of-the-art observational methods are unable to recover the causal effect of online advertising at Facebook. We conclude that observational methods for estimating ad effectiveness may not work until advertising platforms log auction-specific features for modeling.

4. Experiment without the wait: Speeding up the iteration cycle with Offline Replay Experimentation

Pinterest created Offline Replay Experimentation, a framework to simulate the performance of new ideas based on historical data.

While a simulation approach is less accurate than online experimentation, it complements online experimentation by eliminating waiting time for data collection, allowing for a large number of simultaneous arms, avoiding negative user impact, and saving engineering costs.

πŸš€ Job opportunities

New opportunities keep opening up. Experimentation is a fascinating field to work in. Checkout the 33 open roles.

These are the latest featured roles:

WIth so many companies investing in experimentation capabilities and building teams, it’s a challenge to find the right people. Posting your open roles to the job board is a great way to get your amazing job in front of the right people. It also helps to support the Experimental Mind newsletter.

πŸ“… Upcoming events

This is a running list of upcoming events:

I’ve started to create the full overview of events for 2022, let me know in case you are missing an event.

πŸ’¬ Quote of the week

“All life is an experiment. The more experiments you make the better. β€” Ralph Waldo Emerson

πŸ˜‰ Fun of the week

Everyone says they want to be data-driven. Not everyone wants to do the necessary work (via Timo Dechau).

πŸ™ Thanks for reading

If you’re enjoying the Experimental Mind newsletter, you can buy me a coffee or beer (in case you rather use Paypal, you can send it to ‘kevin@experimentalmind.com’. Thanks to the 21 people who have already done it. Much appreciated.

Or simply share this newsletter with a friend or colleague (just like Luka did). Point them here to subscribe.

Have a great week β€” and keep experimenting.

Kevin