Big Data Needs AI; AI Needs Big Data

The Resurgence of Big Data

It’s that time of year again, when people publish their top-10 or top-20 lists of what to expect in the year ahead. As usual, rather than pile on with another list, I’m limiting my contribution to one compelling (or half-baked) trend for the year ahead.

Data is becoming more than the “new oil,” it is becoming the new money. Big data, which became a big deal about a decade ago as analytics hit center stage as the path to business success, faded a bit as big data suddenly was everywhere, making the term irrelevant. Over the past two years, amid all the excitement about generative AI, it almost seemed as if data — or attention to its quality and trustworthiness — took a back seat to all the jazzed-up illustrations and hyper-insightful insights generative AI was delivering.

The Importance of High-Quality Data

Now, with generative AI so critical to business, people are realizing that their AI foundations are built on piles of very loose sand. When AI “hallucinates,” it’s not because it’s mind is wandering, because it has no mind to speak of. It’s simply running probabilities to grab on to the next piece of available and related data to complete a narrative.

Running Out of Data

Now, there’s even concern that we’re starting to run out of data to feed the machines. “Most of the world’s publicly available data — whether it is obtained legally or not — has been exhausted,” said Andy Thurai, senior analyst with Constellation Research. When will the madness end, right?

The Synergistic Relationship between Big Data and AI

Big data and AI “have a synergistic relationship,” states a report out of Qlik. “Big data analytics leverages AI for better data analysis. In turn, AI requires a massive scale of data to learn and improve decision-making processes.” Big data will either make or break AI.

RAG Solutions and the Need for Trustworthy Data

At least 86% of executives report data-related barriers to AI, such as difficulties in gaining meaningful insights and issues with real-time data access, a survey of 1,000 IT executives out of Presidio finds. Half believe they plunged into gen AI before they were fully prepared. The venture capitalist community remains hot on AI, “but guess what? It’s going to take high quality, validated data that doesn’t traipse on privacy or data sovereignty,” Baer said.

The Importance of Open and Trusted Data

Consequently, there’s a growing emphasis on retrieval augmented generative (RAG) solutions, which form the bridge between standard databases and large language models, Baer said. The latest announcements out of the AI Alliance, a consortium of leading technology companies, emphasizes the need for establishing trustworthy data foundations.

Data for AI Must be Transparent, Trusted, Accurate, and Applicable Broadly

“Data is the most important constituent of AI models and systems, yet today data for AI too often has murky provenance, unclear licensing, and large gaps in quality and diversity of languages, modalities, and expert domains represented,” according to a statement announcing the AI Alliance’s Open Trusted Data Initiative.

Conclusion

In conclusion, big data will be back in the spotlight in 2025, as we need high-quality, trusted, and abundant data to power our AI models. The trend is clear: big data needs AI, and AI needs big data. With the increasing focus on open and trusted data, we can expect to see a surge in innovation and adoption in the coming year.

FAQs

Q: What is the relationship between big data and AI?
A: Big data and AI have a synergistic relationship, where big data analytics leverages AI for better data analysis, and AI requires a massive scale of data to learn and improve decision-making processes.

Q: Why is high-quality data important for AI?
A: High-quality data is essential for AI, as it enables accurate and reliable decision-making. Low-quality data can lead to hallucinations and flawed insights.

Q: What is the Open Trusted Data Initiative?
A: The Open Trusted Data Initiative is an initiative by the AI Alliance, a consortium of leading technology companies, to develop better requirements, processes, and tooling for curating high-quality, trustworthy, and open data sets.