How Yelp Built Their Data Pipeline with Amazon Redshift

Yelp

The tech world has changed dramatically since Yelp was launched back in 2004. Eight years later and Yelp started its change. It transformed from running a huge monolithic application on-premises to one built on microservices running in the AWS cloud. By the end of 2014, there were more than 150 production services running, with over 100 of them owning data. Its main part of the cloud stack is better known as PaaSTA, based on Mesos and Docker, offloading data to warehouses such as Redshift, Salesforce, and Marketo. Data enters the pipeline through Kafka, which in turn receives it from multiple different “producer” sources.

Sources:

• https://engineeringblog.yelp.com/2016/07/billions-of-messages-a-day-yelps-real-time-data-pipeline.html
• https://techcrunch-com.cdn.ampproject.org/v/s/techcrunch.com/2018/06/04/how-yelp-mostly-shut-down-its-own-data-centers-and-moved-to-aws/amp/
• https://techcrunch.com/2018/06/04/how-yelp-mostly-shut-down-its-own-data-centers-and-moved-to-aws/
• https://engineeringblog.yelp.com/2016/11/open-sourcing-yelps-data-pipeline.html