Blog

Big Data for Small Businesses

My first exposure to ‘big data’, although we didn’t call it that, was in the insurance industry. We did however work with very large data sets, with millions of records. Insurance is an interesting product, because you don’t know the cost of the product before you sell it. It’s unlike any other product. Say you’re baking cakes, you know how much the flour cost, the eggs, the sugar etc. You know how many cakes you can make with that amount, you know your fixed costs for running the business (rent, rates, insurance etc.), and you know the price of the gas or electricity for running the ovens.

You can add this all up and divide it by how many cakes you’re making, and you know each cake cost a certain amount. You know all this before you even sell any cakes, and then you can price the cake to be higher than this amount and guarantee a profit.

An insurance policy is entirely different however. It’s an agreement to cover potential future costs for a fee – the premium. So when you sell an insurance policy the cost of it is unknown at the time of sale. You may find out later that you sold it far too cheaply, and lose a lot of money, or that you were selling it far too expensively, and putting off customers unnecessarily. This is where the actuaries and pricing teams came in. The actuaries would pore over tens of thousands of historic claims, building models to predict the average cost of the customer. It was surprisingly accurate. Sure, individual to individual, mistakes would be made. But over the long-term, things would even out quite well and we’d often be close to our predictions. Each year we could feed the new data back into the model and refine it, and get better and better over time.

We would also bring in new data sources constantly. There were some interesting things that popped up. One funny example is that we would find people who kept their cars in garages, were much more likely to have expensive claims for their cars being damaged when parked by the road-side. This at first glance seems paradoxical. Surely, a car kept in a garage, is much less likely to be damaged by the road-side. That’s the entire point of keeping it in a garage. Except people know this, and so they lie on the insurance form and select the ‘I keep my car in a locked garage’ option to try and game the system and get a cheaper premium. And so the majority of the policies with ‘cars in the garage’ were actually people in high risk areas, who when first quoted their premium were rightly being charged higher premiums, and so lied to try and save some money.

Legality aside, there is a moral issue here as well. Claims are paid not by the insurance company, but by the policy holders of people who do not claim. If the premiums are high, it’s often down to the fact that the money is needed to pay for the added risk of that type of policy. Insurance companies just calculate that risk, then add a few percentage points for profit, and then sell the insurance. So when people cheat and try and defraud insurance companies, they are really just defrauding other members of the community who are then forced into paying higher premiums to cover the additional costs.

Another interesting example of innovative data sources, was that people who buy their insurance more than one week before the previous policy expires, are much less likely to claim than people who buy the same week, or especially the same day. So let’s say your policy runs out on 15th May, if you buy before the 1st May, you are significantly less likely to claim than if you buy on the 14th or 15th of May. The reason for this may not be immediately obvious, until you realise that insurance is a people-industry, and that one of the biggest risk factors in selling an insurance policy is the person you are selling to. Someone who is purchasing last-minute insurance, is quite likely to be disorganised and irresponsible compared to someone who is purchasing well in advance. This person is more likely to leave the gas on, less likely to get electrical checks done on their property, more likely to leave repairs outstanding that could lead to water damage. These are all things that the insurance company will need to pay for in the future.

And like I said, these methods were surprisingly accurate. The European Union brought in an anti-discrimination law that prevented insurance being priced differently for men and women. Personally, I wasn’t in favour of this. Partly because it is completely fair to charge men and women differently, because the risk is actually different and pricing according to risk is the entire point of insurance. It is not sexist or discriminatory to do this, it is just cold, hard statistical fact. Insurance pricing is by nature discriminatory, and if you want to take this to it’s logical conclusion, the correct thing would be to charge everyone the same fee. Regardless of how responsible someone is and how unlikely it is they are to claim, they would be charged the same as someone who is extremely reckless and doesn’t take the same care as they do.

The second, and more practical reason I was against it, was it was a total waste of time. The law stated that we were not allowed to use the person’s gender to price the insurance, which we didn’t. The interesting thing was, we didn’t need it. Men and women’s behaviour is so consistent, that we could actually infer the person’s gender by looking at other factors. Such as, make and model of car, age of car, colour of car, occupation etc. We are able to accurately predict someone’s gender, to an accuracy of 80%, based on what they did, what car they drove, where they lived and so on. And so we transferred the price loadings out onto these different variables, and charged everyone basically the same premiums as we were doing before. It was a 2 year project that achieved nothing ultimately.

Still, it just goes to show the power of the law of large numbers, and the benefits insurance companies were reaping from this before it was given the buzz word of ‘big data’. More recently, companies like Facebook, Google and Amazon have been applying similar techniques, and in some ways more advanced techniques, to answer the age old question - “what will this person get their wallet out for and buy from me”. The question every marketer and salesperson on planet Earth wants to know the answer to. By using similar statistical analysis, and also by leveraging more modern techniques such as machine-learning, these companies harvest vast quantities of data about their potential customers, and use that to predict what they are likely to buy. It’s the exact same approach used by insurance companies for centuries – what is the probability this person will claim, and how much is that likely to be? Conversely, what is the probability this person will buy? And how much are they likely to spend?

And they know frightening amount of data about you. By tracking phone activity, Google can infer the times when you go to sleep and wake up. They can build a model of your sleep-schedule. They can also monitor what you spend on the internet, thanks to the genius that is Google Analytics (code that exists on almost every website on the internet and tracks everything you do). They might realise that when you are tired late at night, you are more likely to purchase electrical goods. They will then bombard you with these advertisements late in the evening. They might realise that people in your demographic, are more likely to book holidays when it’s raining, and so show you pictures of sunny beaches and so forth when your local weather station is reporting rain.

The amount of data being collected, and being published, is vast. There are APIs everywhere. APIs for tracking weather, shipping movements, housing data, government statistics, the stock market, customer behaviour. There are APIs for controlling remote computer programs, so that vast networks of automated bots can be built to harvest and monitor things, and execute commands in response to that. This is the ‘internet of things’, where big data and portable hardware meet to create a dystopian future where everything is tracked, logged, analysed, then automatically processed to try and make some more money. In recent years, large corporations have cottoned onto this fact, and are investing large sums of money in harnessing the power of ‘big data’. The big problem then for small companies is, how do they compete? They don’t have a multi-million dollar budget to splurge on researching new technologies, or experimenting with exciting new APIs or data-sources. What they need is plug-and-play solutions that can level the playing field and give them access to big data, and machine learning, and harness the power that is data-driven artificial intelligent systems.

There is huge potential for small businesses to start leveraging this technology as it becomes available. Wouldn’t you like to know, which of your customers are most likely to stay and pay more money? Wouldn’t it be great if you found a common denominator to the customers that never come back? Maybe they are all of a certain type of customer, or a certain age. Maybe, it turns out, that most of the people that try your product and don’t come back, are middle-aged men making small purchases. Once you know this information, you can think about what is driving it. Maybe this type of person is looking for something else, maybe they need more time or a longer trial, maybe they actually wanted something more expensive but you didn’t offer it. You can now target your actions to finding and solving the problem for this customer segment, and reap the rewards of it. Maybe you set up automatic systems in place that detect this type of customer, and automatically email them 3 days later and offer them a special discount.

(Published on 27 Dec 2019)


We need more Bankers, and quick!

I can't help thinking there is a desperate shortage of bankers worldwide.

The average banker should expect to earn over £1.5m in bonuses each year. To understand this phenomenon, think about the difference between Tom Cruise and McDonalds. In about 3 seconds I've obtained the box office results for the last few Tom Cruise films: Mission Impossible - $195,042,377 Edge of Tomorrow - $100,206,256 Oblivion - $89,107,235 Pretty decent. In fact his lowest grossing film over the last decade was a mere $15m, and that is a bit of a freak really, they are much more typically was around the $100m mark on average.

So hiring Tom Cruise to be in your film is like buying a golden goose. That's why he'll get paid tens of millions to be in the film. Now imagine this, imagine there were actually 5 Tom Cruise actors, all identical and all able to bring in that kind of cash for a film. If a film company wasn't happy with Tom Cruise #1's high fees, they could go to the Tom Cruise who is out of work and offer him less. The Toms would have to engage in a bidding war to win the business and they would all be taking lower wages. It is the simple law of supply and demand. Now imagine I walk into a McDonalds and ask £400 an hour to flip burgers. It's not nearly as much as Tom gets paid for his films, but you can be as certain I'll get turned away as you can be certain of Tom making a box-office smash hit. The reason is there are many many people that would be willing to work that job for significantly less than that, and so the McDonalds guy simply can go elsewhere and pay less. So Tom Cruise gets paid millions because he's one of the few people that can do what he does, whereas a McDonalds person gets paid minimum wage as there is no shortage of people willing and able to do that job.

So if we're worried about Bankers salaries, isn't it just a problem that we have a shortage of Bankers? The demand is so great to hire their skills, and there is such a shortage of able persons to do the job, that the banks are having to pay very large sums of money to get and keep them there. How about we just start a massive training drive to deliver the next generation of bankers to fill the demand? To be honest I don't feel that great about the idea either. I think a more basic problem of bankers receiving high wages, is how banks make those sums of money at all. There is an interesting measure of financial services industries which calculates the percentage of the economy that is down to financial services compared to other industries that like, actually do something or make something for people.

Clearly financial services are important, and can be a great help to economic prosperity. The ease of paying people online for example is one of the main ways I make money as a tutor. Easy access to car loans, mortgages, business loans, credit cards and other financial services is actually quite useful for building an economy. It is however, not the economy. It cannot actually make anything, it is the grease for the wheels and not the engine. We have in the UK around 13% of the economy roughly devoted to financial services. Compare that to Germany which is more like 3%. To me it's a bit like going to a car wash where there are about 20 guys fetching water buckets and only one guy with a sponge actually doing any work. Sure he's got ample supply of the tools and means he needs to do his job, but he's the only one actually pulling his weight and getting the show on the road.

I'm not knocking the banking sector, but I think it has gotten a bit bloated compared to what we really require to get along. Trading on the stock market is all well and good, but the trader can only make money if the actual company stock he's buying makes money. And it's not like any body else's life has gotten better when he makes a killing on the stock market. Whereas when you buy an apple, your life has actually been improved because you can now eat that apple, so the providers of the apple, the farmers and shops etc, are getting financially rewarded for helping other people, as they should. I don't even begrudge Mark Zuckerberg his wealth because to be honest Facebook has helped millions of people worldwide do what they could not do before he made it. As I said financial services have their place and they do to a certain extent contribute to the success of society. But do we really need hundreds of highly strung traders making millions sitting at a computer fiddling with bits of money back and forth? Banks make their money solely on debt, maybe the ridiculous bankers salaries are a sign we're borrowing too much?

(Published on 30 May 2016)


Why footballers get paid so much

A complaint I often hear from people is that it is somehow morally wrong for footballers to get paid vast sums of money, for example £200,000 a week, to kick a ball leather around a field of grass, when there are jobs ten times more important the get paid a fraction. For example I cannot imagine the local bin men getting paid that much, but I know who I would rather live without. The blame however does not lie with the football clubs signing the big cheques. For them it is a very sound business move to pay that sum of money, since a good player can return them more than their money back if the team can win certain competitions (they get a cash prize) and in merchandise sales. Merchandise is an important one, because the football club is a brand and they can make a lot of money just by becoming well known and selling lots of T-shirts. It's not unusual for football clubs to be earning around £500m a year, and teams like Manchester Utd for example, only make about 25% of their income through the sale of tickets, another 25% through TV rights, and the remaining 50% via other sales. So the benefits to Man Utd of buying and keeping certain players, could be worth millions in the sale of mugs and shirts, and would more than compensate for the weekly costs of keeping him in mansions and Ferrari's.

It's got nothing to do with right and wrong, it's a simple equation of spend £1 make £2. It makes complete sense and I don't blame them for it one single bit. The real question is, how on Earth can the clubs make so much money from hiring these players? Where does all that revenue come from? Well that money is all coming from ordinary people like you and me. Every person that pays a Sky Sports subscription, buys a replica shirt, goes to a pub to watch a game, buys a newspaper to read about the team, or basically in any way spends any time, attention or money on football or something related to football, he is supporting the stupendously high wages of the footballers.

Where people spend their time and attention, is where the money is made. Why is so much food sold all over the country? Because people think about food probably at least 3-4 times a day. They have their attention on it and so they spend their money on it. Youtube for example, has made an entire business of just getting people's attention and then selling that attention over to advertisers. So what we really need to look at, is why do people put so much more importance on kicking a ball of leather around compared to real-world issues like, combating corruption, organised crime, economic prosperity, health and education. It's not difficult to see why really, and that is basically those things are difficult to understand and uncomfortable to think about. It's a lot easier to read about the latest referee scandal on your daily commute and think about the game you watched last night, than it is to consider how to combat the insidious rot of corruption that is literally bringing some countries to their knees.

I'm not really against football to be honest, I quite like the game and I'll occasionally play or watch a bit of football and be entertained by it. Games are important and people should be entertained, nothing wrong with that. However I think we need to reshuffle our priorities a bit. When more people have got more opinions about the England squad selection for the next major international competition than they have about their own government's policies, something has gone amiss. So don't blame the football clubs or even the footballers for their silly wages, blame the millions of people paying their wage bill, the football fans. And before we get all high and mighty over the silly men watching their silly game, think about how much money, time and attention gets spent on make-up and that probably out-ranks thinking about or working on solving world poverty so we're all at it really.

(Published on 17 May 2016)


Crime doesn't pay, but it does make an awful lot of money

I teach Economics as well as Maths. Given my mathematical training, my background in insurance pricing, and personal interest and passion for the subject, it is something I have taught more and more over the years. Having recently done research on organised crime I was staggered to discover some of the statistics. Just in the UK alone organised crime groups make billions. If they were listed on the stock market, their share price would probably be in the FTSE 100.

And that's just the UK, globally the revenue of organised crime run into the hundreds of billions, with some estimates putting it at almost one trillion dollars. Holy Moly! That is an astonishing amount of money. Just to give some comparison, the top 100 corporations in the UK make around £514bn per year. Let's just stick with the UK though, so we can make a fair comparison. Taken from page 8 of this government Crime Report in 2012 the estimated revenue of organised crime in the UK was around £13.5 billion. To give some idea of comparison, Debenhams reported a mere £2.3 billion in revenue in 2015. So we're looking at a company roughly 5 times as large as Debenhams here.

Now these organised crime groups are not one unified force, in fact in some areas they are bitter rivals, but nevertheless we are looking at an economic force comparable to a large UK corporation. I was just quite surprised to see no mention of it in the Economics syllabus. Ok so it's not as if organised crime is taking over the economy, but I would have thought it was a large enough factor to merit if not an entire chapter, at least a paragraph. And if we're educating the next generation of potential policymakers wouldn't we want them to have an awareness of this issue? It's interesting to note that the government estimates the social costs of organised crime as quite a bit higher than their revenues, so we're looking at a significant player in the UK economy, and it's fair to assume that the problem is of a similar magnitude or even worse in other areas of the world.

What concerns me though is where is all this money going? And what are the profit margins, because I imagine they are quite high. The criminals are selling pretty inelastic goods and in most cases work hard (by murdering rivals) to get a monopoly on the market place, which creates very high profit margins. I'd also imagine they have much lower running costs, and also pay zero tax. So while their revenue may be lower than some legal organisations, their profits I would expect to be much larger relative to their size. All this wealth and money is just being piled up somewhere. Coupled with their brutal methods of influence, I can only imagine they are stockpiling large amounts of wealth and power. The main question is though, what is the trend. See if levels of organised crime have been static over the decades, then it's fair to assume that as the crime lords die or fight with each other, they will lose money and influence at roughly the same speed as they gain it, so over time no overall net gain will occur. They would under this scenario continue to have roughly the same level of power and influence in society over time.

However, if it is a growing trend, or they are somehow able to build on their gains over time and slowly accumulate more wealth and power, then what we have is a monster quietly growing in the background becoming harder and harder to deal with. As they buy more judges, intimidate more law enforcers, and tighten their grip on society they become more and more difficult to handle. I've seen a documentary about the top public lawyer in Columbia who is charged with tackling corruption in the country, and it is one of the most dangerous jobs in the world. She has had many assassination attempts and lives with a 24 hour armed guard. When you get to that level it is very hard to turn it around because anyone that tries to stand up to the criminals is instantly a target and so not enough people have the courage to stand up to them and the game is pretty much over by that point. I guess what I'm saying is, I don't know if it's all just business as usual when it comes to crime or is the world slowly sliding down the chute to a point where organised crime groups have comparable or more power than the elected governments? I don't know but if it's the latter or there is any risk of it being that then we need to do something about it now before it is too late.

(Published on 1 May 2016)