This is your weekly roundup of interesting reads around experimentation with updates from Gitlab, DoorDash and Statsig. I also added three new events to the list. Enjoy!
🔎 What I’ve been reading
Experimentation at GitLab — about.gitlab.com
This documentation describes GitLab’s Growth engineering process for running experiments. Extensive read.
Top 10 Non-Controversies in A/B Testing
Great LinkedIn post by Matt Gershoff sharing his top 10 non-controversies in A/B testing. Number 10: Two Tailed vs One Tailed Tests.
Generalist vs Subject Matter Expert: Companies Grapple with How to Use Data Product
Eric Weber on the upcoming role of Data Product Manager, also for experimentation:
… the ability to have a real product experimentation platform is transformational for many companies. An experimentation platform enables companies to identify causal relationships between product changes and user behavior (or other metrics). This space is *extremely* competitive right now, and I’m a huge fan of many startups that have launched here. For data product managers, experimentation platform requires a vast array of context: from experimental design, to cohort allocation, to logging, to guardrails metrics, to experiment readouts.
How change happens — seths.blog
Working in the experimentation space can sometimes feel like an uphill battle. Reminder from Seth Godin: persist. This is how change happens.
Focused, persistent community action is how systems change. And systems concretize and enforce cultural norms.
If you care, keep talking. Keep acting. Stay focused. And don’t get bored.
The Statistics Engine of Experimentation at DoorDash
DoorDash’s centralized library for statistical analysis of experiments.
We needed a way to enforce standardization and facilitate knowledge sharing to improve the velocity of our experiment analysis.
Pulse Time Series for Deeper Insights
The Statsig Pulse results page offers a snapshot of all the metric movements driven by an experiment. Sometimes, a brief scan of the color coded score card is enough to validate that all metrics…
Experimentation Culture Awards 2022 — experimentationcultureawards.com
I’ll be one of the jury members at the 2022 Experimentation Culture Awards. Join the award session and listen to inspiration stories.
🚀 Job opportunities
All featured roles:
- Conversie Optimalisatie Specialist at Kaartje2go (Zwolle, NL)
- Conversie Optimalisatie Lead at Kaartje2go (Zwolle, NL)
- CRO Specialist at Randstad Global (Amsterdam, NL)
- Digital Analyst at RTL (Hilversum, NL)
- Webanalist at Kaartje2go (Zwolle, NL)
Also checkout the job board for more open roles in the field of experimentation, CRO and analytics.
📅 Upcoming events
Final call for joining Ronny Kohavi’s new class, where he will teach you everything he knows about A/B testing in 5 two hour sessions. First session is May 31. Subscribe via this link to receive a $50 Amazon gift card.
All upcoming events in the next months:
- ⏩ 31 May: Accelerating Innovation with A/B Testing (virtual)
- 🆕 3 June: TLC: Get Psyched for UX (virtual)
- 8-9 June: Advances with Field Experiments (Chicago)
- 🆕 9 June: DiDo#39 Ga voor (CRO) goud! (Utrecht)
- 9-10 June: NeuroPsychoEconomics (Lille)
- 15 June: Growth Marketing SUMMIT (Frankfurt am Main)
- 16-18 June: Marketing Science Conference (virtual)
- 25 June: MeasureCamp (Paris)
- 7 July: Experimentation Culture Awards (virtual)
- 🆕 4-6 Aug: Digital Elite Camp (Estonia)
Second batch of tickets will release soon:
- 🎫 18-20 Nov: Conversion Hotel (Texel, Netherlands)
See the full overview of events to find other events and conferences for you and your team.
💬 Quote of the week
Amazon’s Risky Bet, according to BusinessWeek in 2006.
[tweet https://twitter.com/JeffBezos/status/1527003895393812480]
☕ Thanks for reading
If you’re enjoying the Experimental Mind newsletter and want to say thank you? Buy me a coffee (or a beer). Or share this newsletter with a friend or colleague.
Have a great week — and keep experimenting.
Kevin
