Netflix’s experiments
I watched a great interview on Diary Of A CEO, on YouTube, with Marc Randolph the former CEO of Netflix. He built it from a small video rental company to the giant we all know today.
He described how, in the early days, they would do experiments. But it would take them months of planning. Getting the copy just right by arguing over the wording & the messaging, and over which fonts to use. Painstakingly editing every website link and every image. Making everything 100% perfect.
Generally though, the experiment would be an obvious failure in the first few days of launching. That’s the nature of experiments. They generally fail.
So they started to try and fail quicker. They worked to be able to reduce the time to launch an experiment from three months to one. Then down to two weeks. One week. One day. Eventually doing multiple experiments EVERY, SINGLE DAY.
By the end, he explains, things were incredibly sloppy. Website links wouldn’t work. The wrong copy was used. Just a general mess
The amazing part? Some experiments were a resounding success, even though the execution sucked. Most ideas failed, even if the execution was great.
Being able to quickly try multiple things and learn almost immediately what worked is behind Netflix’s amazing success.
One such experiment was to try and do away with late fees to see what the result would be. They were sure it would fail. Before that if you didn’t return a movie on time you would be charged a penalty fee. It was a massive success. Demand for movie rentals skyrocketed as soon as they launched it. It’s one of the big reasons for Netflix’s early success.
What we can take from this
Some brilliant lessons stand out from this:
- A great idea (from the customer’s point of view) will work despite poor execution.
- A poor idea (from the customer’s point of view) will fail despite brilliant execution.
- Customer feedback will quickly let you know if an experiment worked or failed.
- Doing multiple experiments incredibly quickly and sloppy is much better than doing a handful slowly and polished.
- Hundreds of small experiments done in a short space of time will let you hone in on what works for customers and what doesn’t.
- Doing an experiment to prove a negative, or confirm a belief, is as valuable as trying to prove a positive.
- Sometimes the biggest successes are shown by experiments that you are sure WILL FAIL.
- Experiments turn theories and beliefs into actual KNOWLEDGE which you can then make informed decisions. This knowledge helps you come up with new theories and spawns more experiments.
Conclusion
How can you speed up the timeframes of rolling out experiments? Even if you aren’t client-facing, what can you experiment with and how can you do it quickly?