Big Data, Big Challenge, Big Opportunity

Posted on February 21, 2012 by Gareth Spence

Big data is a hot topic. Everyone from world leaders, to CXOs, to analysts, to media and just about anyone connected to the technology industry is talking about the transformative powers of big data. It even made it onto the agenda of the 2012 annual meeting of the World Economic Forum in Davos. If the hype is to be believed, it has the power to transform businesses, governments and even society itself. It has the power to bring new insights into just about everything and drive a new era of intelligent understanding.

Yet if there’s such potential here, why aren’t we seeing more progress? There can be no question that some companies do understand the opportunities. One need only look at Google or Facebook to realise how these companies are effectively mining their data to drive new business opportunities and further monetize their value proposition. But these companies are rare exceptions. As I highlighted in my previous couple of blog posts, many companies are still very sceptical about moving from traditional business models and reluctant to embrace new opportunities.

However, resistance can only last so long, especially when we’re talking about new revenue opportunities. Last month’s World Economic Forum meeting saw people referring to data as a new class of economic asset and in many respects we’re at the start of a new gold rush. One need only look at the job market to see proof of this. Take a look at job opportunities for data scientists. The results are staggering. There’s a huge shortage of trained professionals. McKinsey & Company estimates this shortage to be almost 1.7 million big data professionals in the U.S. alone. It may be time to take your pan and start sifting.

Looking outside of staffing though, there are still two enormous obstacles. One is cultural and the second is infrastructure.

Firstly, if big data is to truly succeed we need to consider what data is being collected. Only this week headlines were made as the British government released plans to store details of every citizen’s phone calls, text messages, emails and website activity. I’m sure this act will meet with the same response as a Facebook privacy update. Yet it raises a key question, how do we feel about our data being collected and analysed? Can we overcome this barrier to see there are benefits? Can big data help us create, as Rick Smolan suggests, a human dashboard? Or will big data simply become another term for big brother?

Secondly, there can be no big data without big infrastructure. Data is useless if it sits in isolated silos. It needs to be harnessed, it needs to be collected on an enormous scale to be effectively analysed to the extent where it can offer insight and this is where we need a capable underlying network infrastructure. Whether this information is housed in the cloud or kept in your own data centres, it needs to be readily accessible, and this means we need to transport huge amounts of data quickly and securely.

What’s incredible to note is the speed at which this data is growing. IDC suggests that data is doubling every two years. This is phenomenal growth and if we’re to successfully transport this volume of data through the network, we need to continue driving forward with the optical reboot. A reboot based on the rebuilding of our networks on a core of 100G, OTNs and ROADMs and further expansion of fibre-based access and backhaul solutions. Something we’re only just at the start of doing.

What’s fascinating to note is how this data could spur a whole new industry. With Amazon’s HPC-as-a-Service platform, we’re seeing the democratization of supercomputing technology and this means big data can be accessed and analysed by a whole new subset of companies, groups and even individuals. It’s tantalising to consider what breakthroughs this may result in. This will be especially true when the Internet of Things starts to mature and we have access to a whole other subset of data.

Are you involved in big data? Have you seen the impact it can have? What’s the future here and what obstacles do we still need to overcome. I’d be interested to hear from you on this.

Tags
  • Oli

    Big data is a great term – which means whatever you want it to mean!
    My current role involves analysing data in a outsourcing telecentre. The sort of information I deal with includes personal details, and how to contact these people. I guess this could be considered big data.

    While there is a lot of info there, its all time limited. Its perfectly possible to have a few mb of data on every person on the planet. However, the more data you have, the more there is to go out of date. It becomes a constant battle to keep the info relevant. It becomes even more difficult when you have legacy or logged data that you want to cross reference.

    Every month, my uncleaned data is losing a minimum of 2% of its quality, so after 18 month, half my data is junk. It doesn’t matter if you have someone’s name and address – if they no longer live there, or their number changed, or their life style has changed due to baby, or getting married, or a change of job etc.

    Data only has a value if its good enough to move an agenda forward. As soon as it can’t do that – its useless.

    This becomes even more difficult when you are logging behaviour. We have heard about the stories of what is possible by cross referencing store discount cards. However, that can only be done if you hold ALL the information. This means that Tesco (for example) can tell you all about what you buy from them, and build nice realistic profile. However, it knows nothing about your interaction in Asda, in the corner shop and online.

    There are two big problems for the data handlers which is that of matching data from different databases, and matching behaviour correctly. For instance, it isn’t unusual to have household where two or more occupiers have the same initial. So, if J Jones has a mobile, and a land line, and J Jones has a Broadband account, and another mobile, who do map with the email address Jonesj@acme.com? Which one shops at Tesco, and which shops at Amazon?

    Finally, there is the issue of data handling because you can. A lot of today’s data gets created because its a feature, or a legal requirement. It is very rarely checked or used for anything. We end up with lots of data which gets stored, leaving companies with a legal liability but no commercial advantage. Logs of who had an interaction with who also suffer from data degradation and very convoluted field mapping!

    Big data is a great topic, but the dangers are mostly sensationalist. However, this may change!