The digital revolution is well and truly here and has made data easy to digitally capture and inexpensive to store. Subsequently, many organisations and businesses are increasingly collecting data about various aspects of their operation, services, customers, suppliers and competitors with the hope of finding useful information with which to gain leverage or competitive advantage. In addition, given the dependency on data, organisations have robust data resilience policies and therefore produce multiple replications of their data. As a result, many organisations are storing up to 50-100 terabytes of data finding themselves data rich yet information poor with a recent study by IBM showing that 1:3 business leaders do not trust their data. This distrust is not surprising either as 90% of this data is unstructured i.e. not just rows of text in nice well-structured tables that we can query, but also very unstructured and difficult to inspect such as Microsoft Word documents, spreadsheets, feedback comments, online news or blogs, voicemail, images or videos – how does one correlate or associate that with the statistical data stored in transaction processing systems representing key information concerning sales, stock or individual branches?’
Given today’s digital world, the volume of digital data being stored and used will continue to grow significantly. According to the leading market research company, IDC, the total amount of digital information generated globally in 2013 was in excess of 4.4 zettabytes (ZB) (where a ZB refers to 10E21 bytes). IBM estimates that 2.3 trillion gigabytes of data is being created each day and had predicted a total of 43 Trillion GB (40 ZB) being generated globally by 2020. However, IDC reported that 64.2ZB of data was generated in 2020, with Dave Reinsel, senior vice president, IDC’s Global DataSphere asserting that “The amount of digital data created over the next five years will be greater than twice the amount of data created since the advent of digital storage. The question is: How much of it should be stored?“. IDC forecasts that global data generation will achieve a compound annual growth rate of 23% over the period 2020-2025 (see The IDC report, Worldwide Global DataSphere Forecast, 2021–2025: The World Keeps Creating More Data — Now, What Do We Do with It All? )
An inherent consequence of this huge growth in data over the last decade is that there has been an exponential demand for professionals with expertise in managing and analysing big data sets in order to identify important correlations, predict future trends or classify new observations. This is particularly true for the following domains:
Evaluating buyer/seller behaviour patterns, business cycle turning points (recessions), risk assessment data, product distribution patterns, foreign exchange rates, stock indexes and prices, interest rates, credit card data, fraud;
Recognising and understanding individual and organisational behaviours relating to: policy changes, tax evasion, benefit fraud, terrorism, social problems, law and order;
Extracting useful information on the web concerning the business/organisation operation to inform future business decisions (eg social media content, competitor behaviour, customer/non-customer usage of the company website);
Analysing omics and other bioinformatics data stored by public/private healthcare management systems to inform patient care and drug development;
For cyber security purposes (eg identifying unusual data traffic patterns indicating malicious intent) or to understand operational efficiency (eg surges in data traffic for call centres or demands for products or services such as concert tickets or hotel accommodation)
Big data has been a disrupter, clearly changing the world and the way we live, and yet many organisations have struggled to look beyond the hype and noise of Big Data and harness its potential for their organisation. We therefore aim to empower you to systematically analyse large and complex data sets in a way that generates insight, informs decision making and adds true business value.