Return to site

The Scientific Method for Startups

When people discuss startups they tend to talk about inspiration and creativity. This leads founders to believe they can imagineer a solution to any problem they’re trying to solve. In reality, executing a startup is a balance between creativity/intuition/instinct and the scientific method: hypothesize > a/b test > conclude > repeat.

Inspiration will help you find a problem to solve. Creativity will allow you to brainstorm potential solutions to that problem. The scientific method will guide you toward which of these solutions will actually solve your customer’s problem.

Running a scientific method requires having measurement in place. It requires that you have pre-existing analytics measuring what your users are doing every day. And it requires that when you build new features, you immediately add them to your analytics system–minute one. In order for this to work you cannot have the attitude: “This is obvious, why would we measure it?”

Many companies make the mistake of only using internal analytics. They think that if they need an answer they can just write a DB query to get it. Unfortunately, the harder it is for you to ask a question, the fewer questions you will ask. By using a standard analytics product (we’ve funded a bunch of them at YC) you enable everyone in your company (even non-programmers) to ask an unlimited number of questions and get the answers quickly. This is hugely important. I actually think it would be a good idea for companies to measure employee usage of their analytics product. Measurement and talking to users improve your ability to be inspired, creative, and break the rules.

Here’s an example: Tim Robinson, one of our employees at Justin.tv, was building a payments funnel for a new product we were releasing. When trying to figure out what to put on the header and footer of the payment flow the obvious idea was to use the standard header and footer from our website. But instead of thinking, “this is obvious, why would we measure it”, he measured everything users did when going through the flow. By measuring he realized instead of moving through the flow with the “obvious” next buttons, users were clicking on the logo at the top of the page, the links on the footers, everywhere! So he started removing these items. A designer might say he was screwing up the aesthetics of the site but as an engineer he could see through the analytics that every time he removed links, payment conversion would increase. By applying the scientific method of hypothesize > a/b test > conclude > repeat he was able to build a product that produced a third of our revenue at JTV. What made his work even more amazing is that Justin.tv at that time wasn’t really data driven at all. Perhaps it was his background as a scientist led him to this method or perhaps it was common sense, but without this feature Justin.tv would have died. Thanks Tim!

The best companies (and employees) are willing to use data to back up their decision making. They don’t believe that standards or aesthetics are rules written in stone. They don’t believe that only product people or founders know the solutions to problems. They believe that their users can help them understand what to build and how to build it if they are willing to implement the analytics, listen, and test.