Taking a broader view with big data and analytics

The history of data

Big data is not new. Although the term was used for the first time in 1997, in an article by NASA researchers Michael Cox and David Ellsworth, the idea of large volumes of information to process, maintain and correlate dates back much further.

‍

This machine, the Hollerith Tabulating Machine, was invented to help process data for the 1890 U.S. Census. Since then we’ve been through information explosions (term first coined in 1941), seen the rise of concepts like Shannon’s information theory, Parkinson’s Law of Data, virtual memory and business intelligence, and the birth of the internet and more recently, the internet of things.

Data today

Experts now point to an estimated 4300% increase in annual data generation by 2020 – we live in data saturated times.

We collect personal data at a greater rate than ever before, via wearables, the internet of things and social media. Within the commercial sector almost all modern manufacturing devices collect and provide data constantly. This gives us access to measurements which were unreachable in the past, allowing us to change the way we do business - faster time to market, instant diagnosis, constant monitoring, etc.

We no longer have to store all the data either. Instead we can subscribe to services that will do the data processing for us, or allow us to use their data within our models. What a change!

Looking for the opportunity to use this data can still be a challenge though. How do we know what is the right information to collect? How can we determine where to apply this data to actually deliver faster insights?

Big data adds real value when…

• Objectives are clearly defined
• The role of the information is understood
• The boundaries are understood
• The project is fluid and pragmatic
• There is a continual validation process
• Time to insights is fast

Business intelligence and big data

Historically, business intelligence has always been about what we know and can measure. It usually involves collecting, storing, applying business rules, and building reports and visualisations for business users to consume.

Big data, on the other hand, has been less knowable, and more about trends, patterns and inferred insights.

Now, however, boundaries between the two are blurred. New tools allow end users to perform data mashups that incorporate big data feeds into traditional business intelligence. With a few clicks of the mouse BI users can connect to social media data and see how well campaigns are doing, or analyse the sentiment towards their companies.

Traditional data warehouses and reporting still have a place. They allow you to measure, refine and provide clean enriched data sets of direct business value. But big data can enhance that, and provide a broader view. By incorporating external data sources, empowering end users to explore data, and constantly refining and iterating, organisations can glean new insights and ultimately be more competitive.