A bit of competition is healthy…

April 6, 2016

There is very little in life that isn’t a competition, and if you want to get the best results, encouraging a competitive edge is often the best way forward.

Up steps Kaggle, an online forum that runs grown-up competitions for data scientists. The aim is to solve problems ranging from how to automate the proves of recognising endangered species, such as individual right whales, through to developing an algorithm to automate measurements that are key indicators of heart disease (the cause of more than 30 per cent of deaths around the world.)

Founded in 2010, Kaggle boasts a community of several hundred thousand experts from more than 100 countries and 200 universities in such quantitative fields as computer science, econometrics, statistics, maths and physics, as well as in a variety of industries including insurance, finance, science and technology.

Kaggle relies on the fact that there are countless strategies that can be applied to any predictive modelling task and it is impossible to know at the outset which technique or analyst will be most effective.

In addition to the competitions, these braniacs use Kaggle as a soundboard to collaborate and exchange ideas. There’s no membership fee to join. The company makes its money (although not much at the moment – it recently laid off a third of its staff and closed its energy industry consulting business) by charging the sponsor of each competition to post a data problem. It also offers competitions for academic institutions free of charge, which helps broaden its community of experts. The competitions are open to anyone who can offer a solution to the problem and reap either the company-sponsored financial reward or simply the bragging rights for having found the best solution.

In effect, it is a form of crowdfunding for problem-solving. Instead of capital, companies are looking for top experts to solve a problem. Kaggle not only provides the experts without the high costs of consulting fees, it also pitches experts against each other in a bid to come up with  the best solution. The motivation isn’t just about getting paid for providing a service, it is also about proving your expertise – there is a Kaggle ranking featuring a list of those who have come up with the top solutions.

According to some reports, companies such as American Express and the New York Times have begun listing a Kaggle ranking as an essential qualification, and last year even Walmart ran its own Kaggle competition in a bid to overcome the recruitment crisis it was facing in the area of data analytics.

Potential candidates were provided with a set of historical sales data from a sample of stores, along with associated sales events, such as clearance sales and price cuts. They were asked to come up with models showing how these events would affect sales across a number of departments. As a result, several people were hired into the analytics team.

One newspaper reported cited a recruiter for Walmart’s technology division praising the Kaggle competition for opening up the retailer’s recruitment process to a much wider range of candidates. He said individuals who wouldn’t necessarily have been considered for an interview based in their resumes alone have now joined the company and are offering a great insight with a different skillset.

Similarly, US insurance giant Allstate used Kaggle to invite programmers to develop a new car accident injury algorithm. The eventual winner was reportedly 271 per cent more accurate than the company’s existing model.

Given the wealth of resource available to anyone willing to approach a problem solving exercise in this way and the vast amount of money many companies spend trying to recruit the right staff, it isn’t surprising that more  and more organisations are turning to this method. In the same way that startups use sites like Kickstarter, Crowdcube or Seedrs to target interested individuals for money to help run their business, companies and organisations an target interested individuals to take advantage of their minds and aptitude.

As one commentator stated, Kaggle provides a multitude of approaches that can be winnowed to the technique that ultimately proves ‘best of breed’ and most effective. And given the competitive world we live in today, who wouldn’t want to be the best?

Become a Brickendon Change Leader

What can we help you achieve?