Skip to content

pinterest/memq

Folders and files

NameName
Last commit message
Last commit date

Latest commit

author
Ambud Sharma
Oct 4, 2022
7aa9c6f · Oct 4, 2022

History

10 Commits
May 16, 2022
Jun 8, 2022
May 16, 2022
May 24, 2022
Oct 4, 2022
Oct 4, 2022
Jun 8, 2022
May 16, 2022
Aug 19, 2022
May 16, 2022
May 16, 2022
May 16, 2022
May 16, 2022
May 16, 2022
May 24, 2022

Repository files navigation

MemQ: An efficient, scalable cloud native PubSub system

MemQ is a new PubSub system that augments Kafka at Pinterest. It uses a decoupled storage and serving architecture similar to Apache Pulsar and Facebook Logdevice; however, it relies on a pluggable replicated storage layer i.e. Object Store / DFS / NFS for storing data. The net result is a PubSub system that:

  • Handles GB/s traffic
  • Independently scales, writes, and reads
  • Doesn’t require expensive rebalancing to handle traffic growth
  • Is 90% more cost effective than our Kafka footprint

Design

Getting Started

Checkout our Getting Started Guide

Usage

MemQ can be a valuable PubSub choice for the following use case families:

  • Large scale data ingestion
  • Bulk uploads
  • Near Real Time Analytics

Design

Detailed architecture docs can be found here

Blog

Please checkout the original MemQ blog for details and background.

Authors

Ambud Sharma

Ping Min-Lin

See contributors

License

MemQ distributed under Apache License, Version 2.0