Big Data is the key to Artificial Intelligence

By Anthony S. Policastro

Have you ever thought about all the data your business is capturing on a hourly, daily or weekly basis? It is probably incomprehensible in light of the channels and volume of information captured 24/7.

The overall, high-level purpose of mining all this structured and unstructured data from your CRM, sales, marketing and advertising channels and most recently IoT devices is to garner insights into your customers, competitors and potential market trends.

It is not humanly possible to categorize and find insights from these oceans of data quickly enough so that the information is relevant.


With all that data, the teams of data analysts that companies rely on today to interpret the data simply can’t keep pace with the volume.

The real challenge is merging all the analysis together to get a 360-degree contextual picture of your customers, potential purchases and market trends.


Apple's Steve Jobs once said during an interview,

“I remember reading an article when I was about twelve years old. I think it might have been Scientific American, where they measured the efficiency of locomotion for all these species on planet earth. How many kilocalories did they expend to get from point A to point B? And the condor won, came in at the top of the list, surpassed everything else. And humans came in about a third of the way down the list, which was not such a great showing for the crown of creation. But somebody there had the imagination to test the efficiency of a human riding a bicycle. A human riding a bicycle blew away the condor all the way off the top of the list. And it made a really big impression on me that we humans are tool builders. And that we can fashion tools that amplify these inherent abilities that we have to spectacular magnitudes. And so for me, a computer has always been a bicycle of the mind. Something that takes us far beyond our inherent abilities.”

Artificial Intelligence (AI) is the new bicycle bridging the virtual world with the physical and big data is the fuel and lifeblood of AI.

With recent advancements in computer processing, data storage, and better machine-learning algorithms it is possible to ingest and analyze more data than ever before. At the same time, there is a connectivity boom as more and more devices and apps connect to the Internet producing even more data.

With these advances, is it now possible to feed your big data into an AI engine and let machine learning mine the precious insights, predictions and next course of action. We can teach machines through supervised learning now, instead of programming them and they will then learn on their own through trial and error. That’s why having large amounts of data is more important than ever. The more data AI has, the more accurate it will become.

Data is now more valuable than oil

The Economist says the world's most valuable resource is no longer oil, but data.

"As devices from watches to cars connect to the internet, the volume is increasing: some estimate that a self-driving car will generate 100 gigabytes 
per second. Meanwhile, artificial-intelligence (AI) techniques such as machine learning extract more value from data. Algorithms can predict when a customer is ready to buy, a jet-engine needs servicing or a person is at risk of a disease. Industrial giants such as GE and Siemens now sell themselves as data firms."

The International Data Corporation (IDC) predicts that 44 zettabytes will be generated by 2020 (A zettabyte is 1 trillion gigabytes).

Trends that will shape Big Data and AI in 2017

TechRepublic, a resource for IT decision makers, says there are five major big data trends to watch in 2017.

1. AI and machine learning will increase the need for for big data analytics
2. Self-service big data tools even for beginners are hitting the web
3. Analytics is struggling to keep up even with big data warehouses like Hadoop and Spark
4. Data cleansing will become a prominent industry as AI is only as effective as the data it ingests.
5. Democratization of data - server-less, micro architectures will allow data to be accessed, analyzed and managed without servers from anywhere by anyone.

AI is ubiquitous and growing

No matter what you do, AI will eventually touch every aspect of your life. AI, machine learning and deep learning are making big impacts on business and your personal life from simple chatbots to self-driving cars.

Many people use these terms interchangeably, but they are different.

• AI is defined as the capability of a machine to imitate intelligent human behavior.

Examples are computer chess and most chatbots where the AI is programmed to ONLY play chess or answer a specific subset of questions like customer support issues or a back-to-school sale.

• Machine Learning (ML) is a subset of AI and designed to analyze large subsets of data and learn from it. ML allows computers to learn without programming to complete a task.

ML understands speech and can make predictions based on the data it analyzes.

• Deep Learning (DL) is a subset of ML and uses neural networks to learn the characteristics of something like face recognition.

Google's DeepMind AlphGo used Dl to beat 18-time Go world champion Lee Sedol in 2016. AlphaGo studied 30 million human moves in Go and learned by playing against itself.

Google Translations can now teach itself to translate languages it doesn't know using its DL Google Neural Machine Translation (GNMT) system. The new DL improves translation quality, and enables “Zero-Shot Translation” — translation between language pairs never seen explicitly by the system.