(2013-03-05) Linked In Technologies

LinkedIn is creating a lot of original technology, much of which has been Open Source-d. Instead of writing algorithms to make People You Know Know more accurate, he worked on getting LinkedIn’s Hadoop infrastructure in place and built a distributed database called Volde Mort. Since then, he’s built Azka Ban, an open source scheduler for batch processes such as Hadoop jobs, and Kaf Ka, another open source tool that Kreps called “the Big Data equivalent of a Message Broker.” At a high level, Kaf Ka is responsible for managing the company’s real-time data and getting those hundreds of feeds to the apps that subscribe to them with minimal latency... One of the most-important things the company has built is a new database system called Es Presso. Unlike Volde Mort, which is an eventually consistent Key-Value store modeled after (the) Amazon Dynamo database and used to serve certain data at high speeds, Espresso is a transactionally consistent Document Store that’s going to replace legacy Oracle databases across the company’s web operations. It was originally designed to provide a usability boost for LinkedIn’s In Mail messaging service, and the company plans to open source Espresso later this year.