In systems like Spark and FlumeJava, there is a templated, or generic, data structure representing a distributed collection of objects. In Spark, it's called an RDD; In Flume, a PCollection.

New, useful Apache big data projects seem to arrive daily. Rather than relearn your way every time, what if you could go through a unified API? A long-standing joke about the Hadoop ecosystem is that ...

Unlock the full InfoQ experience by logging in! Stay updated with your favorite authors and topics, engage with content, and download exclusive resources. Dany Lepage discusses the architectural ...

The following content is from an older version of this website, and may not display correctly. The system which has made feasible the processing of big data at scales beyond the largest storage volume ...

GOOGLE I/O Google has built a new thing for developers who want to have their data-filled cake and eat it right now, or perhaps set it aside for later and chomp it down at leisure. The Google Cloud ...

Google continues to share the wealth of the uniquely powerful software systems it erected to run its enormous online empire. "Cloud DataFlow is the result of over a decade of experience in data ...

A monthly overview of things you need to know as an architect or aspiring architect. Unlock the full InfoQ experience by logging in! Stay updated with your favorite authors and topics, engage with ...

Jeffrey Dean and Sanjay Ghemawat: “MapReduce: Simplified Data Processing on Large Clusters,” at 6th USENIX Symposium on Operating System Design and Implementation (OSDI), December 2004. Joel Spolsky: ...