Tracker pool and dirty list

[pzel]

Tracker Pool & dirty_list API

Summary

This work is aimed at increasing the capacity of applications using
Phoenix Presence.

Specifically, it addresses two issues:

1. A single Tracker server becoming a bottleneck under high throughput
2. Tracker.list calls invoking GenServer.call to get a list of
   presences

The first issue is resolved by starting a pool of named Tracker.Shards,
and dispatching calls to them based on the topic in question. The default
pool size is 1, and therefore the current behavior of Phoenix.Presence is
not affected.

The second issue is resolved by introducing a Tracker.dirty_list function
with the same API as Tracker.list, but with much less overhead and less
precise results.

Comments & Caveats

This likely addresses #71.

A corresponding change will need to be made to the Phoenix Framework in order
for the default Phoenix.Presence handler to expose the new
Tracker.dirty_list functionality. In the meantime, applications that need
to use dirty_list can implement it directly, as shown below:

# file: myapp/lib/myapp/presence_tracker.ex
#

defmodule MyApp.PresenceTracker do
  use Phoenix.Presence,
    otp_app: :myapp,
    pubsub_server: MyApp.PubSub,
    pool_size: 128

  def dirty_list(%Phoenix.Socket{topic: topic}), do: dirty_list(topic)
  def dirty_list(topic), do: dirty_list(__MODULE__, topic)

  def dirty_list(module, topic) do
    grouped =
      module
      |> Phoenix.Tracker.dirty_list(topic)
      |> group()
    module.fetch(topic, grouped)
  end

  defp group(presences) do
    presences
    |> Enum.reverse()
    |> Enum.reduce(%{}, fn {key, meta}, acc ->
      Map.update(acc, to_string(key), %{metas: [meta]}, fn %{metas: metas} ->
        %{metas: [meta | metas]}
      end)
    end)
  end
end                                                                                          

Graphs from the application running under target production load

The graphs compare the same scenario with different configurations of the proposed changes, the default setting being the one of them.

Pool Size Load Test Results:

• 150K channels connected via websocket;
• A single channel lifetime uniformly distributed between 0 and 300 seconds;
• Every channel subscribes to presence updates of 100 other channels.

At pool_size = 1, the track time tail latencies are much higher than the corresponding metric at pool_size = 36.

Pool size: 1

Pool size: 36

List Function Load Test Results

• 137.5K channels connected via websocket;
• A single channel lifetime uniformly distributed between 0 and 300 seconds;
• Every channel subscribes to presence updates of 500 other channels.

List times are dramatically lower when using 'dirty_list, due to the complete bypass of GenServer.call`s.

Tracker.list @ pool size = 128

Tracker.dirty_list @ pool size = 128

[pzel]

Hi @chrismccord!
We rebased this PR on top of your clouds work on the Tracker. We think it's ready to go -- please take a look when you find a moment.

Happy 4th! 🇺🇸

[josevalim]

Hi @pzel, let me hijack this thread briefly. I have just watched your excellent ElixirConf talk. At some point in the talk you mentioned the use of phx_requests for tracking subscription and I want to be sure I understood the design. Previously, you would join a channel for every friend you had. Then you changed it to have a single channel that subscribes to the topic of every friend. Is this correct?

[josevalim]

I just got to the questions part of the talk and Chris mentioned fastlaning and I would like to point out you can still fastlane if you pass the proper options on subscribe.

@chrismccord, it may be worth adding a subscribe function to channels that always pass the proper fastlane options. I feel like we have talked about it in the past. Any downsides?

[OvermindDL1]

Then you changed it to have a single channel that subscribes to the topic of every friend.

That is what I do, prevents process-explosion. ^.^;

I just got to the questions part of the talk and Chris mentioned fastlaning and I would like to point out you can still fastlane if you pass the proper options on subscribe.

Ooo, I did not know about those options...

it may be worth adding a subscribe function to channels that always pass the proper fastlane options.

Please yes? :-)

[chrismccord]

@josevalim I implemented this but then reverted because the clients are not aware of the fastlaned topics. Broadcasts are matched on the client via the topic, so the only way to handle fastlaned messages would be to use socket.onMessage and try to parse out what special topics/events you are looking for. So including it as a generalized easy-to-use option option doesn't seem straightforward enough to me as a feature. As you noted, it's all still possible today by passing the options yourself if you have special handling needs. Are there other options for handling on the client that I'm missing?

[chrismccord]

I'll also note that the "subscribe to many topics via a single channel process" is definitely a pattern we promote for different usecases. Some references to this in the guides would be nice.

[josevalim]

@josevalim <https://github.com/josevalim> I implemented this but then

reverted because the clients are not aware of the fastlaned topics Does it mean the client discards them or does it mean the client is unable to know from where it actually came from? -- *José Valimwww.plataformatec.com.br <http://www.plataformatec.com.br/>Founder and Director of R&D*

[chrismccord]

Both but there is a socket.onMessage where you could try your own handling

…

Sent from my iPhone

On Sep 30, 2017, at 12:05 AM, José Valim ***@***.***> wrote: > @josevalim <https://github.com/josevalim> I implemented this but then reverted because the clients are not aware of the fastlaned topics Does it mean the client discards them or does it mean the client is unable to know from where it actually came from? -- *José Valimwww.plataformatec.com.br <http://www.plataformatec.com.br/>Founder and Director of R&D* — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[pzel]

@josevalim: Chris pretty much summed up what we ran into.

We developed a fastlane module that does some sanitization, and plugged it in globally. It worked great, except for the fact that the topic was now hardcoded to the topic of the originator of the broadcast. The clients were not listening to these (they only reacted to messages coming from OurPrefix:MyId, their home-channel topic), and so we had to move back to fastlane: nil to maintain backward compatibility.

This is what our setup looked like before we had to revert:

supervisor(Phoenix.PubSub.PG2,
        [OurApp.PubSub, [fastlane: OurApp.PrivacyPreservingFastLane,
                         broadcast_strategy: Phoenix.PubSub.Strategy.Serial]]),

[lucaspolonio]

Hey @chrismccord, are there any plans to merge this PR? I'd love to see it released. Phoenix.Presence really suffers under high throughput right now and I believe this PR addresses most of the issues.

[chrismccord]

Yes, it is still on my plate!

[chrismccord]

Sorry this took so long to get merged in. Thanks so much!!! Note: I have decided to remove the dirty_list API for now so as not to increase the API surface area. Thank you! ❤️❤️❤️🐥🔥

[pzel]

Sweet! Hope people enjoy the speed benefits!

[indrekj]

@pzel do you have recommendations how to choose a pool size for the tracker?

Also, I understand dirty_list was removed from the tracker API. I still see it in the Shard module. Is it possible to still use it?

[pzel]

@indrekj I've been out of the loop regarding PubSub performance for a while now, so I can't make any recommendations about how to measure total system performance for your use case. However, I think that using a pool size equal to the number of CPUs on your target deployment machine is a reasonable choice. If you don't know this number, use 16 and see how the system behaves ;)

Regarding dirty_list, I think the only way to access it now is to keep a local fork of phoenix_pubsub and add
https://github.com/phoenixframework/phoenix_pubsub/pull/79/files#diff-fc3d1e8efd95176c339eaafe5ab24ad6R232 to the tracker source. Not elegant, I know, but I understand why the Phoenix project wouldn't want a 'broken' function in their api.

[chasers]

If anyone else needs this you don't need to fork it, just implement a dirty_list function in your app that looks like list, except use dirty_list. Like:

def dirty_list(tracker_name, topic) do
    pool_size = your_pool_size

    tracker_name
    |> Phoenix.Tracker.Shard.name_for_topic(topic, pool_size)
    |> Phoenix.Tracker.Shard.dirty_list(topic)
  end

[indrekj]

I also made a PR #127 which removes the need for dirty_list.

[chasers]

@indrekj I saw that! I was trying not to diverge from what's here, as I'm pretty new to all this. I hope they merge it! The whole thing is eventually consistent so why serialize the reads in the first place?