A Bing employee's two-day test generated more than $100 million of additional revenue in one year.
1. HBR On Strategy frames the episode around Great Strategy Starts with Experimentation and Stefan Tomke's argument that businesses should test instead of relying on intuition.
2. Stefan Tomke, a Harvard Business School professor, joins Kurt Nikisch to explain why experimentation matters and why managers should care about it.
3. The episode asks what experimentation is for: finding out what works and what does not when leaders face uncertainty.
4. Tomke opens with Microsoft's Bing, where an employee ran a control test on ad display after a manager had shelved the idea.
5. That Bing test took two days of work and generated more than $100 million of additional revenue in one year.
6. Tomke says the employee's authority to launch the experiment mattered because the test, not the manager, showed the result.
7. He says many firms confuse a real scientific experiment with a simple trial or tryout.
8. Tomke names infrastructure, tools, cost, and the fact that nobody listens to the results as reasons companies do not experiment at scale.
9. He also says managers worry about traffic losses, stoppage rules, kill switches, and the humility of saying, "I don't know."
10. The discussion moves from digital products to physical retail with a company Tomke calls "Calls," which tested opening stores an hour later.
11. The controlled experiment showed that opening an hour later did not make much difference, even though a consulting analysis had predicted cost savings.
12. Tomke says companies such as Amazon, Microsoft, Netflix, and booking.com built their own experimentation infrastructure before third-party tools became available.
13. He says Microsoft's infrastructure group was about 85 to 90 people the last time he checked.
14. Tomke describes a manager who complained that a tool was wrong after an experiment showed that giving customers less choice improved performance.
15. He uses booking.com to show scale, citing more than 1.5 million room nights booked each day and over 30,000 experiments a year.
16. Tomke says statistical power changes with sample size, so smaller companies often need bigger experimental changes, and IBM consolidated websites to increase traffic on fewer pages.
17. The interview stays practical and conversational, with Kurt Nikisch asking for playbooks, examples, and the role of managers.
18. Tomke answers in a measured teaching style, using phrases like "grand challenge," "center of excellence," and "highest paid person's opinion" or "hippos."
19. Business leaders building experimentation systems and product teams testing customer behavior will find this useful.
20. Listeners looking for entertainment gossip or narrative storytelling probably will skip it.
