Premiums from Credit Data: Going Wrong

The modeling effort ran into trouble. The models were drastically underperforming from what was anticipated. The team tried every modeling approach they could think of, with little success. Eventually the whole project budget was used up in this first unsuccessful phase with little to show for it. I was brought in at the end but couldn't help much.

There's a long list of things that went wrong.

The team forgot the project they were on. They were using approaches appropriate to marketing response models and they were working in a different world. Doing 40% better than random doesn't work well for marketing response models but here it meant we could improve the insurance company rate models by 40% which is fairly impressive. Before the project started the team needed to put serious thought into what success would look like.

The team let an initial step in the project take over the project. At the least, that initial step should have been ruthlessly time-boxed. Since that initial step wasn't directly on the path towards the outcome it should not have been in the project.

The team didn't do any data exploration. When I was brought onto the project near the end, one of the first things that I did was to look closely at the data. What I found was that over 10% of the file had under $10 in six-month premiums, and many other records had extremely low six-month premiums. In other words, a large chunk of the data we were working with wasn't what we think of as insurance policies.

This goes to an earlier point, that often DBAs know the structure of their data very well but often have very little idea of the distribution and informational content of their data. Averages, minimums, maximums, most of what we can get easily through SQL don't tell the story. One has to look closely at all the values and usually this means using specialized software packages to analyze data.

We got a second chance later, fortunately.

The Next Fiasco - Premiums from Credit Data

A company I was with was building a modeling system to look at individual credit history, compare it with insurance premiums and losses, and identify customers where the insurance premium was either too high or too low. I was only peripherally involved with the project and only brought in at the end. What we were asked to predict was the overpayment or underpayment ratio so the insurance companies could adjust their premiums.

The project started by receiving large files from the client and starting the model building process. The team decided to start out with a simpler problem by predicting if there was a claim or not, and once that problem was solved using the understanding gained to move on to the larger problem.

Things didn't work out so well.

Righting the Wrong-Sizer

In order to fix this problem the company has to do some hard thinking about what kind of company they want to be and what kind of customers they want to have. Other things being equal companies want the customers to pay more for goods and services and the customers want to pay less; on the other hand companies want to attract customers and customers are willing to pay for goods and services they want. This means that in order to maximize the total return there is a real tension between maximizing the price (to get as much as possible from each customer) and minimizing the price (to attract customers and make sure they stay). How to resolve that tension is by no means trivial. One option is to assume that “our customers are stupid people and won't care that their bill just went up” but I don't think that's a good long-term strategy.

Ideally we want to find services that are cheap for the company but that customers like a lot. Standard customer surveys will just give us average tendencies when what we care about the preferences of each individual customer. Fortunately we have an excellent source of that customer's preferences: the rate plan they are on. Let's assume that the customers are in fact decently smart and are using roughly the best rate plan for them, but they might need some help fine tuning their plan.

Take the customer rate plans and divide them up into families. When a customer calls up, look at their actual usage and calculate their monthly bill in the different rate plans in their families. If a customer can save money by switching rate plans, move them but keeping them in their rate plan family. This method makes sure the customer is getting a good deal and sticking within their known preferences, and the company is still maintaining a profitable relationship with the customer.

What's Wrong with the Wrong-Sizer?

Let's start with the customer usage profile. To start out with a project that's intended to give individual recommendations to customers and start that project by assuming that all customers act the same is amazingly dense. The Rate Plan Optimizer project manager explained that they had a study done several years ago saying that most customers were fit pretty well by their profile.
First off, a study done a few years ago doesn't mean that much in a constantly changing world, not when data can be updated easily. Second, even if most customers are pretty well fit by the profile that means that some customers are badly fit by the profile and will be negatively impacted by the system's recommendations.

The reason that the IT department went with using a one-size-fits all usage pattern was that the customer data warehouse did not actually have customer usage data in it, only how the customer was billed. The IT department should have taken this project as an excuse to get the usage data into the data warehouse. The customer recommendations could have been been done at an actual customer level.

The next major problem with the Rate Plan Optimization project was choosing the rate plan that was most profitable to the company and then suggesting the customer adopt that plan. In other words, the Rate Plan Optimizer had the goal of making the customer's bills as large as possible and making sure the customer got the worst possible plan from the customer's standpoint.

How to fix it? That's tomorrow.

Real Examples: The Rate Plan Wrong-Sizer

That's a hypothetical example of information design; let's talk about some examples where I don't think that design was done so well, and how I think it could have been made better.

The Rate Plan Wrong-Sizer

I was working for a telecommunications company when my group was introduced to the Rate Plan Optimizer Project. IT had just spent one million dollars in development budget and they needed a group to take over the product.
The goal of the Rate Plan Optimizer was to help customer service reps suggest rate plan improvements to customers. The product did this by

1. Assume every customer had exactly the same usage patterns with the only difference being their minutes of use and then
2. Look at a series of rate plans and suggest to the customer the plan that would be most profitable to the company.

The product had a number of parameters that could be managed, and IT wanted our group to do the managing.

I can't tell you that much about the parameters because my group got as far away from the project as quickly as we could. The project was broken enough that no amount of parameter tweaking could fix it and we didn't want to take the blame for generating bad customer experiences.

What's wrong with the Rate Plan Optimization Project and how should it have been designed? More tomorrow.

Building an Attrition System

We're talking about setting up an attrition intervention system.

This is all about information: how to get customer care reps the exact information they need to help out our customers.

The first big step is getting commitment to build a system and do it right. A well-done simple policy is a lot better than a badly done sophisticated policy. The next step is getting commitment to test the system at every level. Customers are fickle creatures and we don't understand how they will react to our best efforts. I'll have to say something about how to measure campaigns soon, but right now let's just say that we need to do it.

Let's start with the intervention. The obvious thing is to try to throw money at customers, but buying customers can get very expense quickly. What will often work better is to talk with them and just solve their problems. But here you need a good understanding of what their problems are. We can do this by a combination of data analysis, focus groups, surveys, and talking to customer reps. There are a couple of dangers here. 1) Trying to do this by simply building an attrition model. Attrition models will typically tell us the symptoms of attrition , but not the root causes. 2) Relying on the intuitions of executive management. Executives often have some ideas about attrition but rarely have a comprehensive understanding of why customers actually leave.

The next step is trying to get an understanding of the finances involved. What are the financial implications of, say, reversing a charge the customer didn't understand? It's going to be different for one customer that has done this once and another customer that habitually tries to take advantage of the system.

Everything, everything, everything needs to be checked against hard numbers. We have experiences and form opinions on these experiences but until be check we don't know what's really going on.

The last step is what people usually start with: building an attrition model to tell when customer are likely to leave. A standard attrition model won't really give us the information we need. We don't just need the chance someone is going to leave. We need to match customer with intervention; that's a much more specific type of information.

An Example of Information Design

Let's say we're designing an attrition (attrition is when a customer leaves a company) system. When a customer call customer care, we give the representative a recommendation. We have a lot of options:

1. We can't ever know exactly who is going to leave so let's not address the problem.
2. Have an overall policy that treats all customers exactly the same.
3. Present an attrition score to the customer representative.
4. Present an attrition threat flag to the customer representative.
5. Give a graded response with reasons and some specific recommendations to the representative.

Depending on the company any one of the solutions may be appropriate.

If a company is going through a period of low attrition, ignoring attrition may be the best response. There can easily be more important problems for an organization to worry about. I have seen this happen in companies where attrition has been a critical focus of the company: the incremental effect of a new attrition-focused system is small. However, if attrition is a problem in the company (1) can be a foolish approach. It is usually impossible to tell who exactly is going to leave but good analytic design can tell you how to make bets and get a good return on your efforts.

Solution (2) is what companies usually do, and if the policy is well thought out this can be sufficient.

Solutions (3) and (4), while apparently more sophisticated that solutions (1) and (2) are asking for trouble, How are service representatives supposed to interpret the data they are given? If we give customer service representatives a raw score without guidance, then good representatives will worry about their interpretations and the attrition score will become a source of stress. If we give an “Attrition Threat: Yes / No” flag, then we've lost the ability to distinguish between a slight risk and a substantial risk. we'll be giving the representatives clear guidance but that guidance probably not be appropriate and the company will be worse off than if they had no policy.

What we want to do is solution (5): break the base down into segments with guidance and insight in each segment, making sure that our intervention is appropriate and effective in every case.

Still, there are lots of right and wrong ways of doing this -- more tomorrow.