Sep 262011

First off, read this.

So Netflix says to the SEC that churn is not important to them. Except that they didn’t actually say that. They said “the churn metric is a less reliable measure of business performance, specifically consumer acceptance of the service.” meaning that the metric, for them, is broken and therefore should not be used to compare them to others in the marketplace. The cynic would respond “what are you hiding?” but the truth is that they are correct: in their business, churn is so different that trying to compare it across companies is a disservice to the naive public. The information would be misconstrued and therefore should not be revealed. I generally am of the mind that you let the consumer of information make the decision about the quality of information but here I am with Netflix – the consumer is likely to misuse and ignorantly, accidentally compare it to other types of “churn” (a difficult metric to define to begin with).

In the BI universe, we use KPI’s to monitor the progress or success of a product, system, business, etc. but we also use them to compare and benchmark against like products. Pageviews, time on page, click through rates, etc. are the common bellwethers of awesomeness or supreme suck in the web world. But what happens if you make a website that uses a continuous scrolling method… like say an image search results page? Suddenly your pageviews per user drops massively compared to industry standards! I would argue that the continuous scrolling image search is superior to the tired paging image search (in fact, so superior that Google ripped off Bing to some degree… a rarity to be sure) but have heard through the grapevine that one specific search-y company refused to drink the continuous scrolling Kool-ade due to the impact it would have on third party web reporting metrics. Sacrifice the user experience for sake of the KPI. So what does this mean for Netflix vs. the SEC?

When the paradigm changes, it’s often hard to jump out of the traditional KPI rut. Those KPIs are comparable, comfortable, expected (here with NFLX we are talking about quarterly churn, but we could be talking about unique users, time on page, page views, or any other metric). Remember P/E ratio arguments during the Dot Com boom? I find the same issues in my job as a BI manager – we have a product that is an Android widget and someone asks me a question about pageviews – what the hell is a pageview on a widget? Is every time a user focuses on the widget a “pageview”? Are all actions in the widget separate pageviews or part of the same initial pageview? Lastly, (and more importantly) does a pageview count matter or is it (or something similar) only useful when used as an internal metric?

The key, in my opinion, is the use of internal versus comparative metrics. Netflix is saying that giving out churn numbers as they are traditionally calculated is a great way to confuse and freak out their investors since so many customers “quit” and then rejoin a few months later. The definition of churn is too narrow (“users who quit the service” / “total users”) because a user who quits in January and rejoins in March has technically “churned” in Q1 even though they are now a customer again. As an internal metric, understanding their churn from quarter to quarter makes sense. They might want to (and surely do) offset that by calculating a metric of “sticky churn” i.e. people who, in the words of Marsellus Wallace in Pulp Fiction “and when you’re gone, you stay gone”.  Or even better would be a whole suite of metrics around churn and churn like behaviors – new never before seen people, returning after a short break people, returning after a long break people, totally gone from the system as far as we know it people. Lots of options, nothing perfect, nothing overly clear, and everything confusing to investors who only know how to compare the metrics they know and love within and across companies. I don’t blame Netflix for keeping the numbers to themselves. Of course, it would be nice for them to release a case study on all the cool and weird ways people migrate around their services – not for investors’ sakes, but for my own nerdy curiosity.