Monday, March 01, 2010

Converting Data to Recommendations

There has been a huge improvement in the area of recommendations from consumer data in the last 5 years. From Netflix to Kroger to Amazon to Google. Why are the credit card companies not as aggressive in getting into this?

Dunnhumby (partially owned by Kroger in the US and Tesco in the UK) perfected the art of sending out coupons to consumers based on their purchases. Dunnhumby combines loyalty card information (what you bought in the last few trips), sales data (basket analysis - what products were bought together)  and more traditional market research to figure out which consumers will be motivated to purchase by what offers. From all anecdotal information and news reports, the coupons have been right on the money: "Americans, even in this 'great recession,' are still redeeming only 1 percent to 3 percent of paper coupons. In contrast, Kroger, the nation’s largest traditional grocery chain says as many as half the coupons it sends regular customers do get used.".  Google has been targeting ads on the internet based on search terms, mails in Gmail etc. They have proved to be good at it based on the market share they are gaining in advertisement dollars. Amazon has very good recommendation engine not only based on what a consumer bought from them, but also what the user browsed before they bought. Netflix competition winners BellKor showed a great way to recommend movies to customers based on the movies they have watched and rated.

There is continued improvement in this area. The Netflix competition focused on increasing accuracy of recommendations.  Accuracy of recommendations is not an end-all. Suggesting one generic romantic comedy after another (say What Happens in Vegas and Just Married) just because a user rated When Harry Met Sally five stars is not helpful. In a paper that will be released by PNAS, a group of scientists, systems that base recommendations on correlations between users can miss niche items that a user might like, but would never find on his own.  Research indicates that the most interesting recommendations and information originate from "weak ties" in a system, between users that are somewhat similar but disparate enough that they can introduce novelty to each other. These researchers have come up with a hybrid algorithm that could create a body of recommendations that combined novelty items and safer, more accurate pieces.

As these recommendation engines get better, we can expect them to utilize all the trails that we leave to be utilized to bombard us with recommendations. Our internet wanderings is already being monetized (Google, Amazon, ...). Our purchases at retail stores are already being monetized (Tesco, Kroger, ...) . Now there is research that shows that "Cell phones show human movement predictable 93% of the time"  The claim is that "location awareness" using mobile phones is the next major step in recommendation targeting in 2010 (Placecast working with Northface, ...).

The credit card companies know what retailers, restaurants, laundromats etc. that we frequent. Though they don't know what we bought at each of these places, they could easily find out what we think about a particular establishment based on our repeat patronage. They know what other establishments the consumers in the same location, who patronize the same establishments, also frequent. They can send me coupons based on that and based on which retailers are willing to pay for those coupons. If I am in a new city and I am looking for restaurant choices within 5 miles of where I am, they could help me with that (of course, with some retailers highlighted with coupons when the retailer is willing to pay for that marketing through the credit card company) better than anybody else given what they know about me. May be with the reputation issues that credit card companies have today, it might not be the right time, but 2 years from now is an eternity and they need to start now for them to achieve that. Why are they slower than most in targeting us with recommendations (coupons) and get paid for it?



  1. Intriguing. I wonder if credit card companies would run into privacy concerns, or if consumers are ready for this type of personalization?

    As you note, recommendations based on correlations between users can miss niche items and aren't always diverse. This is one reason we're seeing recommendation systems that don't rely on collaborative filtering. For example, at Jinni our movie recommendations have a semantic basis.

  2. Phoebe, the shopping pattern of consumers are being used already; for example, to decide their credit risk. Significant percentage of shoppers at Kroger are happy to receive their coupons. If they see a clear value proposition, few would mind.

    Nice work with Jinni. I especially like the one click connection to a netflix account!