In a recent paper from O’Reilly on the “Big Data Market“, they apply a big data approach to analyse the big data market using live data crawled from “websites, meetups, hiring patterns, business relationships, blogs, press, forums, SEC filings…”.
An interesting approach which revealed some interesting insights including:
Given that most big data analytics projects are developing their own custom siloed pipelines, this is not surprising. The silos often share common design patterns and a common set of technologies from the Hadoop ecosystem, which results in duplication and wasted effort.
Next consider the conclusions from a report from McKinsey & Co – “Big data: The next frontier for innovation, competition, and productivity” :
So there’s clearly a significant imbalance between demand and supply of big data professionals. Building siloed pipelines makes this situation unsustainable and drives the need for a more sustainable approach.
That’s one of the reasons for being for PNDA. PNDA is a simple, scalable open data platform which provides a common set of services for developing analytics applications, accelerating the process of developing big data analytics applications whilst significantly reducing the TCO.