“Big Data” continues to garner a lot of attention as we close out 2012.  What is Big Data and why is it a Big Deal?

The basic definition is “a collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools.”

According to IBM, Big Data spans four dimensions: Volume, Velocity, Variety, and Veracity.

  • Volume, of course, defines the amount of data
  • Velocity defines real-time processing and recognition of the data
  • Variety is the type – both structure and unstructured and from multiple sources
  • Veracity is the authenticity and/or accuracy of the data
Data has always been collected, but statistical analysis has traditionally been based on what has happened – Big Data lets a business understand what is happening now.

Today and going forward, data is being collected on just about everything,  and in multiple forms including text, web data, tweets and other social media feeds, sensor data, audio, video, click streams, log files and more.  Perhaps, more impactful is the sheer amount of data that is now generated and collected in a single day – let along a single minute as shown.

Gnip – a provider of firehose social media data; tracks 400 million tweets per day and over 100 billion social media activities per month; while Sprint has commented that almost half a trillion transactions goes through its networks per day.

Digital data is everywhere and offers significant transformational potential, if an organization knows how to harness it: collect it, analyze it and apply the understanding of it to solve business problems and serve customers better.

What are the key drivers for organizations to embrace Big Data?

There are many – including revenue generation, data monetization, reducing customer churn, improving the customer experience, and network optimization to name only a few.

According to a recent survey by the Economist Intelligence Unit (Big Data – A Lesson from Leaders); strategic decision making, operational efficiency and customer service were the areas in which the application of big data has improved performance.   But it is marketing and communications that are likely to see the greatest benefits as they use big data to improve the customer experience and better understand customer preferences and behavior.

Social media, transactional and location-based data used in conjunction with traditional data sources such as loyalty cards and web data, are increasing the ability of organizations to create a complete picture of customers’ preferences and demand – a key goal of marketing, sales and customer service.

Big Data Comes with Big Challenges

Unfortunately, extracting value from all of this data is not without its challenges.  According to IBM, there are four major steps to Big Data Adoption as shown.

In the Educate stage, the focus is on the development of big data awareness and knowledge.  In the Explore stage, the focus is on the development of organization’s roadmap for big data.  In the Engage stage, organizations perform assessments of their technologies and skills and begin to prove the business value of big data. Finally, in the Execute organizations begin to implement big data and analytics capabilities.

Areas of concern by organizations include a shortage of the analytical and managerial talent necessary to make the most of big data; the inability to connect data across organizational and department silos dealing with multiple & competing systems already in place; lack of understanding of how to work with unstructured data; delays caused by data validation; and trying to do too much too soon.

So what is the Big Deal about Big Data?

Big Data separates itself from traditional forms of data collection and analytics by leveraging both historic and real-time data with industry specific advanced analytics.  By setting a clear objective or target, organizations can fully utilize data as one of its most valuable resources to influence strategic decision making, improve customer services, enhance revenues, reduce expenses and improve efficiency and identify and develop new products, services and channels.

Big Data can offer transformational attributes if implemented correctly – and the time to start is now.