What I’ve been reading this week
Part 1 in a multipart series about decision making and experimentation at Netflix.
At Netflix, we believe there’s a better way to make decisions about how to improve the experience we deliver to our members: we use A/B tests. Experimentation scales. Instead of small groups of executives or experts contributing to a decision, experimentation gives all our members the opportunity to vote, with their actions, on how to continue to evolve their joyful Netflix experience.
ps. Netflix is looking for an Experimentation & Causal Inference Intern
2. A/A’/B Testing: Evaluating Microsoft Teams across build releases
A/B testing is rarely used for comparison between whole builds. That’s because each build integrates multiple feature changes and it is hard to figure out which features cause regressions, if any. However, the Microsoft Teams Experimentation team did make an attempt to use A/B testing as an integration testing tool for builds comparison.
3. Why the term ‘North Star Metric’ is a terrible metaphor for product success — medium.com
Bhavik Patel suggests we start calling it ‘Lighthouse Metric’, not NSM.
4. The use of Dark Patterns [paper]
In this study the authors look at 50 types of dark patterns across the desktop web, mobile web, and mobile app modalities of 105 services. Most used dark patterns are privacy related, see table below.
5. Outdated belief: A/B testing is only for online systems
Jan Bosch explains why:
A/B testing is relevant in any context where a mathematical optimum can’t be established easily. As such, it’s relevant for virtually any company and system. With the increased connectivity of systems and a continuously growing expectation of improvements to systems throughout their entire economic life, it’s clear that A/B testing is a capability that every industry, company and team needs to build up.
6. The Framework You Need for Successful Marketing Experimentation
Jason Widup on how to get a systematic plan to run effective marketing experiments and learn from them.
7. Lessons from 7,000 people about ads and privacy
Google asked Ipsos to interview thousands of people about their personal privacy and data. Three tips for defining your data strategy:
- Make it meaningful
- Make it memorable
- Make it manageable
8. Adam Grant explains the 4 modes of thinking. — www.inc.com
I think too many of us spend too much time thinking like preachers, prosecutors, and politicians … We need to think more like a scientists.
Job opportunities
This week’s featured jobs:
- Technical Director Experimentation at EA Sports, Redwood/Remote US
- Sr. Product Manager Experimentation at Hula, Remote US
- Research Scientist Manager Experimentation at Facebook, London
- Head of Experimentation at Zalando, Berlin
- CRO Specialist at KPN, Amsterdam
Don’t forget to check out the other 15 opportunities on the job board. Are you hiring? Simply add the vacancy to the board.
Upcoming events
- Experimentation Works (Sep 16, online)
- MeasureSummit 2021 (Sep 28-Oct 1, online)
- Experiment Nation: The Conference (Sep 29, online)
- MIT Conference on Digital Experimentation (Nov 4-5, online)
- The Conference formerly known as Conversion Hotel (Nov 19-21, NL)
Quote of the week
If you’re not experimenting, your baseline assumption is that whatever you’re doing now is perfect — Richard Mathera
Fun of the week
As always, if you’re enjoying this email , I’d love it if you shared it with a friend or two.
You can send them here to sign up. I try to make it one of the best emails you get each week, and I hope you’re enjoying it. And should you come across anything interesting this week, send it my way! I love finding new things to read through members of this newsletter.
Have a great week — and keep experimenting.
Kevin
