The fervor around big data continues to grow. The World Economic Forum and The New York Times are jumping on the bandwagon. While we share their enthusiasm for the potential, big data needs a reality check.
Here are just a few of the how-do-you-get-there-from-here questions for anyone considering big data projects.
Big data means high costs. Data acquisition is a monstrous problem. Much of the best data is under the lock-and-key of the major platforms (search, social, mobile). And “open” data sure isn’t free. It often requires a long series of data transformations and analyses before it can be used effectively.
Figure: Big data complexity
Data-driven applications are inherently complex. Large amounts of source data need to be collected. Thereafter, the data undergoes multiple layers of analysis and transformations. Finally, the application services that are consuming the data need to be integrated before the final result can be presented to end-users.
Poor performance in the long tail
Big data is plagued with small data problems in the long tail of interests. Everything individuals truly care about is in that long tail. Many of the most interesting and lucrative markets are long tail collections of sparse data segments, including local e-commerce, social interest networks, and vertical domains such as health and politics.
Big data approaches need large amounts of detailed data about users and their usage patterns. Consumers are getting fed up with these intrusions into their privacy. Most producers don’t want the headaches or risks of managing this sensitive data.
The vernacular of big data analysis—trends, correlations, inferences, and frames—betray their bias towards generalization over specialization. This may seem academic … unless you want to treat people as individuals.
Perhaps the biggest bottleneck to big data is people. The complexity of big data approaches drives the need for highly specialized (and highly paid) professionals to build and manage the systems.
Time to market
When you add up all the expertise, infrastructure and technology needed to acquire, analyze, and manage big data, time to market for these projects is often measured in years. Big data opportunities are enticing, but pale if they can’t deliver within Internet time frames.
At Primal, we love big data problems. It drives our business. Many of the companies we’re working with are bumping up against the limits of big data approaches and are looking for innovative new approaches.
So here’s a radical thought: If the cost and complexity of big data is so painful, why don’t we look for simpler solutions? If the cost and complexity stems from the acquisition, analysis, and management of data, why don’t we look for solutions that are far less dependent on massive amounts of existing data?
Figure: Primal’s approach
Primal’s approach to big data opportunities is simple: 1) Provide keywords as indicators of your interests and select your sources; 2) Primal synthesizes an interest network that connects your interests to the information sources; 3) Primal delivers your filtered information in the format that suits your needs.
Primal uses a computational approach that avoids the monumental costs of big data. Instead of analyzing large amounts of representative content to derive the data, we synthesize the data directly.
Primal works effectively using only sparse data as input. We support dozens of popular Internet sources out-of-the-box. And a simple and intuitive interface makes it easy for anyone to use Primal, even if you have no prior experience with big data.
Check out our developers site to explore your own Primal-powered solution!