From 1b327898dfcc942be9dfa09beb2622d8ff7fc9ea Mon Sep 17 00:00:00 2001 From: sescobb27 Date: Wed, 9 Jan 2019 09:29:48 -0500 Subject: [PATCH] [ci skip][docs] update poller and jobs general docs --- apps/repo_jobs/README.md | 13 +++++++++++++ apps/repo_poller/README.md | 14 ++++++++++++++ 2 files changed, 27 insertions(+) diff --git a/apps/repo_jobs/README.md b/apps/repo_jobs/README.md index e98a39e..c8dce67 100644 --- a/apps/repo_jobs/README.md +++ b/apps/repo_jobs/README.md @@ -19,3 +19,16 @@ Documentation can be generated with [ExDoc](https://github.com/elixir-lang/ex_do and published on [HexDocs](https://hexdocs.pm). Once published, the docs can be found at [https://hexdocs.pm/repo_jobs](https://hexdocs.pm/repo_jobs). +## General Overview + +- BugsBunny creates a pool of connections to RabbitMQ +- each connection worker traps exits and links the connection process to it +- each connection worker creates a pool of channels and links them to it +- we spawn in the ConsumerSupervisor a given number of GenServers that are going to be our RabbitMQ consumers +- each consumer is going to get a channel out of the channel pool and is going to subscribe itself as a consumer via Basic.Consume using that channel + +## High Level Architecture + +it is really similar to the architecture of `RepoPoller` in the sense that we need a pool of connections and channels to RabbitMQ and a pool of workers, in this case, RabbitMQ Consumers; this is because we also use BugsBunny which is our connection layer that both apps uses. + +![screen shot 2018-08-29 at 7 22 49 am](https://user-images.githubusercontent.com/1157892/44787334-7a2e5d80-ab5c-11e8-86c4-16f7e5de3275.png) diff --git a/apps/repo_poller/README.md b/apps/repo_poller/README.md index 844b142..b2ed34c 100644 --- a/apps/repo_poller/README.md +++ b/apps/repo_poller/README.md @@ -19,3 +19,17 @@ Documentation can be generated with [ExDoc](https://github.com/elixir-lang/ex_do and published on [HexDocs](https://hexdocs.pm). Once published, the docs can be found at [https://hexdocs.pm/repo_poller](https://hexdocs.pm/repo_poller). +## General Overview + +- BugsBunny creates a pool of connections to RabbitMQ +- each connection worker traps exits and links the connection process to it +- each connection worker creates a pool of channels and links them to it +- when a client checks out a channel out of the pool the connection worker monitors that client to return the channel into it in case of a crash +- RepoPoller polls GitHub repos for new tags +- if there are new tags it publishes a message to RabbitMQ using the connection/channels pool + +## High Level Architecture + +when starting a connection worker we are going to start within it a pool of multiplexed channels to RabbitMQ and store them in its state (we can move this later to ets). Then, inside the connection worker we are going to trap exits and link each channel to it. this way if a channel crashes, the connection worker is going to be able to start another channel and if a connection to RabbitMQ crashes we are going to be able to restart that connection, remove all crashed channels and then restart them with a new connection; also we are going to be able to easily monitor client accessing channels, queue an dequeue channels from the pool in order to make them accessible by 1 client at a time making them race condition free. + +![screen shot 2018-08-17 at 7 51 36 am](https://user-images.githubusercontent.com/1157892/44267068-71e83100-a1f2-11e8-8d73-2bc7a1914733.png)