danny flamberg
danny flamberg 12 April 2017
Categories Data & Analytics

The Best Kept Secret in Data Science

Big Data is a big pain. Humans are generating data five times faster than the rate of economic growth. Each day we create 2.5 Exabytes of new data, ninety percent of which is unstructured. That’s roughly half a gigabyte of new data each day per person

Big Data is a big pain. Humans are generating data five times faster than the rate of economic growth. Each day we create 2.5 Exabytes of new data, ninety percent of which is unstructured. That’s roughly half a gigabyte of new data each day per person as we surf the web, play music or videos, text friends, open emails, buy stuff, take photos, make phone calls, travel, visit the doctor or go about our lives. Snails leave a residue in their wake. We leave data.

Collecting, assessing, sorting, filtering, segmenting and storing all that data is a pain; not to mention an incredible analytics challenge. And yet data scientists in every sector are madly tying to get their arms around this data gusher, find relevant patterns and discern market-changing insights. They are simultaneously empowered and encumbered by the array of deployed technologies, most of which do one or two things well but, cannot handle the volume or make the underlying connections between data elements.

DPMs store media and third party data. CRM platforms collect and store psycho-demographic, marketing and purchase data. Marketing cloud offerings use data to deploy campaigns. DSPs manage media targeting and buys. Business Intelligence tools extract data from storage systems but don’t work across platforms. Data lakes, data marts and data warehouses store enormous volumes of information one field at a time. And even though many of these tools connect to each other, none offer a single source of robust, validated information tied to each individual.

Turning Big Data into big value faces three distinct challenges. First is getting a grip on whom to address. Assembling data from multiple sources, in many formats, into a single “golden record” representing everything we know about an individual or a business is difficult but necessary. The stickiest part is resolving identities. We have to connect disparate online, email, social and mobile accounts accessed with different handles and be sure they all correspond to the same person.

Second, finding patterns and extruding actionable and competitive intelligence from mountains of data requires extensive cleansing and preparation and a series of discrete analytical skills. In most cases, 80 percent of analytic time is spent preparing the data with just 20 percent left for analysis.

Third, storing, maintaining and updating data in formats that can be transported, imported, ingested, used or reused in multiple existing and future systems or structures requires agile formatting and forward thinking to hedge against evolving technology. Too much data is trapped in legacy systems and outdated programming languages unavailable for analysis. 

The answer to these challenges is an ingenious approach to data analytics which transforms, pre-sorts, pre-codes and pre-fabricates raw data into signals. Signals turn massive data sets into manageable, relevant pieces that expedite analysis. Signals are useful information about events, customers, systems and interactions. They describe behaviors, events and attributes, and they can predict future outcomes.

A typical business can be profiled with 3000 baseline signals, which are essentially the family jewels of a business. Signals are combinations of raw data. Often they mirror patterns and dynamics in a business or an industry. Once signals are created they can be reused or recombined like LEGO building blocks. Signals can be constructed, connected and reconfigured easily and quickly. They eliminate the need to go back to or process raw data every time a new use case comes up.

Signals come in two flavors. Descriptive signals explain who (White collar man 35+ living in Cleveland) what (bought a business class ticket for a window seat to LA credited to his airline loyalty account) when (Thursday March 30 at 1:45pm GMT) and how (using MasterCard #### with a credit limit of $15K). Signals document interest, intent, preferences and actions that can be used to trigger interactions.

Prescriptive signals represent the application of machine learning algorithms to basic signals. They are propensity or model scores applied to individual records, which tell the likelihood of a given event or transaction happening in the future. They express preferences, identify look-alikes and represent experience scores, price sensitivity and metrics predicting sustained loyalty or early defection. They are the building blocks for predictive analytics.

Signals create an intermediary stage in Big Data technology because they sit above the raw data stored in marts, warehouses and lakes and below the cloud-based marketing automation systems, CRM platforms, BI tools and other operational structures that create and deploy campaigns. This new middle space in the technology stack is the Signal Layer.

The Signal Layer is the secret sauce of savvy data scientists. The repository of thousands of Signals and billions of raw data components, it automatically updates all the data elements and stores the metadata to make reuse and recalculation easy. The Signal Layer becomes the central nervous system for marketers by drawing basic data from storage, turning it into intelligence, powering specific tasks and use cases and ultimately feeding or directing the systems used to engage and interact with consumers.

Rather than compile mailing lists, develop basic segmentation and deploy one-size-fits-some campaigns, savvy marketers are taking advantage of the signals consumers continuously create. They aim to gain competitive advantage by practicing data-activated marketing based on individual’s real-time needs experiences, preferences, interests and behaviors using signals to power use cases stored, refreshed and analyzed in the emerging Signal Layer.


Please login or register to add a comment.

Contribute Now!

Loving our articles? Do you have an insightful post that you want to shout about? Well, you've come to the right place! We are always looking for fresh Doughnuts to be a part of our community.

Popular Articles

See all
Digital Marketing Vs. Traditional Marketing: Which One Is Better?

Digital Marketing Vs. Traditional Marketing: Which One Is Better?

What's the difference between digital marketing and traditional marketing, and why does it matter? The answers may surprise you.

Julie Cave
Julie Cave 14 July 2016
Read more
4 Important Digital Marketing Channels You Should Know About

4 Important Digital Marketing Channels You Should Know About

It goes without saying that a company can't do without digital marketing in today's world.

Digital Doughnut Contributor
Digital Doughnut Contributor 5 November 2014
Read more
14 Facebook post ideas for each and every brand

14 Facebook post ideas for each and every brand

A lack of creativity – like many things in the world – just happens sometimes. There’s nothing wrong with it, it’s not an irregular event. But nowadays, you need far more than a your average post to attract your audience’s attention instead of losing them to your competitors. And if you don’t enhance your strategy with creative ideas that work towards your business goals, your performance is going to weaken.

Jan Zajac
Jan Zajac 19 June 2017
Read more
How To Write Content That’s Easy To Localize

How To Write Content That’s Easy To Localize

Do you want to know how to reduce costs and get to market faster? Write content that's easy to localize!

David Ost
David Ost 15 June 2017
Read more
5 Marketing Automation Hacks

5 Marketing Automation Hacks

In 2015, it was predicted that marketing automation will be amongst the most piloted marketing technologies, and since then, the term has become a buzz-word for marketers. Now, marketing automation is almost seen as something both necessary for business growth and essential for generating new leads.

Natasha Orme
Natasha Orme 26 June 2017
Read more