In October 2015, an artificial intelligence system, developed by the University of Cincinnati, repeatedly and comprehensively beat a retired USAF colonel in aerial combat simulation.
On the face of it, this comes as no big surprise. Armed with enough computer power, it must be possible to crunch real time data quicker than a bunch of grey cells. So, like me, you probably assumed that ‘Top Gun’ was blown away by data overload and superior processing fire-power.
Well, nothing could be further from the truth. This simulation was fought using surprisingly small amounts of data and equally modest processing power – the equivalent of a Raspberry Pi.
So why did the air ace go down in flames? Well, the simple answer is that he was beaten by superior human logic. The AI system used a technique called ‘Genetic Fuzzy Tree’ – a type of fuzzy logic algorithm that reduces each control decision to a manageable number of sub-decisions. Concentrating only on the variables for each sub-division, the system deliberately imitated the way humans focus on processing small amounts of data. It just did it better and faster!
But, if the future for AI lies in mimicking human reasoning and exploiting small datasets, it does beg a fascinating and pretty fundamental question: why are we so pre-occupied with Big Data?
Big Data is one of the most touted and talked about concepts in technology. But is all the hype and hyperbole justified?
The term was first coined, about twenty years ago, when large search companies started wrestling with ways to process the huge volumes of data generated by the internet. Driven by the belief – or, to be more honest, the optimistic hope – that the magic of analytics would someday unlock valuable insights, most large companies started creating vast data lakes.
And make no mistake, the theoretical value of these data reservoirs is huge. A 2015 article in Forbes magazine predicted that, for a Fortune 1000 company, a mere 10% increase in data availability could result in $65 million worth of additional net income.
So why are there so few examples of this value actually being realised? Despite the fact that we have created more data in the last two years than in the entire history of the human race, less than 0.5% of this information is ever analysed or exploited.
We have become so focused on capturing vast amounts of historical data that we’ve lost sight of one fundamental flaw in the big data dream.
Humans are not programmed to process huge volumes of information. We are not wired to be Cray super-computers. We need our data sliced-and-diced into bite-sized portions; digestible information that we can use to directly improve our performance.
Now fortunately for us, that’s very good news. Not only do humans digest small data more efficiently but (to stretch the analogy) small data also happens to be very rich in food value. Frequently, the most fruitful insights are drawn from the thinnest, most unpromising datasets.
And I am not just talking about esoteric air-to-air combat simulations, I am referring to real-life projects where small data delivers substantial, actionable insights…
This surprising truth is beautifully illustrated by some work undertaken last year by one of our Clustre member firms…