Welcome to the Data Sphere

Written by:



In the mid-1980s, at a lecture in New York, author and counterculture visionary Robert Anton Wilson introduced his theory about the growth of information, called the \”Jumping Jesus Phenomenon.\” In short, his theory was that it took until the time of Jesus (1 A.D. or thereabouts) for all the knowledge that was in the world in 1500 B.C. to double. It took 250 years for that body of knowledge to double again and then 150 years to double after that. By 1750, knowledge had increased by four times, or four Jesuses. By 1973, we had the equal to 128 Jesuses, and by 1998, 512 Jesuses. Thus, the \”Jumping Jesus Phenomenon.\”

[callout]Over 150 trillion gigabytes (150 zettabytes) of data will need analysis by 2025. That will include much more than tracking consumer purchases and NSA surveillance.[/callout]

In the 80s, computers were a nascent industry, devices mainly used by scientists, NASA and complex businesses. E-mail, internet shopping, smartphones and laptops didn\’t even exist. Wilson predicted that as computers integrated into society, it would exponentially accelerate the process of knowledge growth. And indeed, it has-the data jumps we are seeing now are rocket-fueled.

The Ever-Growing Data Sphere

Let\’s take a look at some predictions about data growth over the years:

2013: \”Data is growing incredibly fast-by one account, it is doubling every two  years.\” (Big Data: A Revolution That Will Transform How We Live, Work, and Think.)

2014: \”Humanity now produces as much data in two days as it did in all of history until the year 2003.\” (The New York Times, Pico Iyer.)

2015: \”Data is growing faster than ever before and by the year 2020, about 1.7  megabytes of new information will be created every second for every human being on the planet. By then, our accumulated digital universe of data will grow from 4.4 zettabyets today to around 44 zettabytes, or 44 trillion gigabytes.\” (International Data Corp/IDC.)

2015: \”The NSA says new data equal to the Library of Congress is being produced every six hours.\” (David Baldacci, \”The Escape.\”)

2019: \”The amount of data generated in the world will grow to 175 zettabytes by 2025, putting the compound growth rate at 28 percent over the next five years.\” (IDC adds that going forward, much of the new data will be generated by IoT devices, metadata and video surveillance.)

From a Deluge of Data-Digital Pixie Dust

Businesses today are caught in a deluge of data. Most every company of any size has initiated online and on-site customer tracking, including purchasing history, personal information, social media and other online activities. The challenge in the retail industry is how to best maximize use of this data to actually increase sales. All that data has created the new businesses of data storage (mostly in the Cloud) and data analytics on how to interpret the data. A recent survey found that \”83 percent of all companies invest in big data projects,\” and \”79 percent of executives believe that failing to embrace big data will lead to bankruptcy.\” (77+ Big Data Stats for the Big Future Ahead/Wikibon.) With ownership of large amounts of data also come security concerns, spawning the booming industry of cybersecurity: protection from data thieves and hackers.

According to technology giant Cisco, \”99.5 percent of collected data never gets used or analyzed.\” It is clear that in the majority of cases, collection abilities far exceed utilization abilities. To avoid a data supernova, we will have to become more and more dependent on artificial intelligence (AI) and machine learning (ML).

Poster Child for Analytics

Among retailers, online shopping giant Amazon turns data into industrial strength digital pixie dust. The company uses its Amazon Web Services (AWS) computing platform to create a 360-degree view profile of each customer. Amazon did not turn profitable until after they initiated the AWS program in 2016. In 2017, the company reported earnings of $3 billion, and in 2018, annual net income rose to $11.5 billion. Amazon\’s Prime membership program, first introduced in 2005, that offers perks such as free, speedy shipping for an annual fee (currently $119), was predicated on the power of AWS. Amazon has been a star performer during the pandemic, with its second quarter, 2020, net income doubling to $5.24 billion, compared to 2019, mainly due to a tripling of its online grocery sales.

Walmart Is Amazon\’s Biggest Headache

While Amazon has been the pioneer in big data for the retail industry, Walmart is aggressively moving forward with plans to challenge the digital forerunner. Its new Walmart+ membership program is similar to Amazon\’s Prime, and the Walmart $98 annual fee membership will provide free delivery (for orders of $35 or more) and fuel discounts. This program will enable the company to dive deeper into developing consumer data profiles. Walmart has struggled with e-commerce-according to Recode, Walmart\’s online business was on track to lose about $1 billion in 2019. CEO Doug McMillon surprised retail observers when he announced that he had an interest in making a joint bid for Chinese-owned video-sharing app TikTok, along with Oracle. TikTok would allow access to data on millions of new [mostly younger] customers and would be a new platform for advertising.

Haves and Have-Nots

Unfortunately, all of this technology creates significant cost barriers for smaller and startup retail businesses. Not only do they have to have digital payment systems, scannable merchandising labeling, computerized inventory control and ordering, social media sites and online sales, but they also must address heightened security features-and in the pandemic era, unprecedented health and sanitation protocols. Fifty years ago, few retailers were burdened with any of these expenses. The deeper into the data sphere we go, the bigger advantage large retailers will have over smaller competitors, and the closer we will move toward monopolies. The pandemic has accelerated this process.

Big Data: Speed Bumps

1. Security is an absolute. Businesses that cannot guarantee the safety of consumer data, should not be collecting it. And that includes legacy protection in the future if a company is sold or declares bankruptcy. Many retailers have experienced serious data breaches in recent years, including Target, Home Depot, TJX, Hudson Bay (owner of Saks Fifth Avenue and other properties), and Neiman Marcus. Customers are becoming increasingly concerned about their personal data ending up in the hands of third parties with criminal or unknown intentions.

2. Privacy concerns will only grow in importance. While there is a need for transparency and honesty between businesses and customers, consumers are growing increasingly apprehensive about being \”spied on.\” Companies need to find a balance between their data needs and consumers\’ rights to privacy. Some monitoring and advertising activities are well tolerated, even welcomed when customers feel they benefit, but there is a fine line in digital surveillance that often passes into territory that consumers call \”creepy.\” An InMoment survey reported that a \”whopping 75 percent of consumers said they find personalized advertising and branding at least somewhat creepy.\”

3. Be sure targeted advertising is actually on target. An article in eMarketer reported that 7 out of 10 customers would welcome personalized ads, and research by Segment concluded that, \”Consumers expect highly personalized shopping experiences from retailers and are willing to spend more when brands deliver targeted recommendations.\” However, the majority of consumers are \”disappointed with the lack of personalization in their shopping experiences,\” and \”on average 71 percent express some level of frustration with their personalization experiences.\”

The Datafication of Everything

According to Forbes, \”Over 150 trillion gigabytes (150 zettabytes) of data will need analysis by 2025.\” That will include much more than tracking consumer purchases and NSA surveillance. As we add facial recognition, embedded medical devices, geolocation, and \”smart\” IoT items, ranging from homes, cars and spy cameras in Teddy bears, to Ring doorbells, light bulbs and toothbrushes, nearly everything in the modern world is becoming datafied.

The data sphere is awash in data, with more coming in every millisecond, and the rate of speed is constantly accelerating. Some retailers are questioning if consumer data collected pre-pandemic will even be relevant in the post-pandemic era. It is a race to keep up and an even bigger challenge to assure that all that data produces meaningful, beneficial new knowledge.



Scroll to Top
the Daily Report

Insights + Interviews right to your inbox.