Connection.activate_stream RuntimeError: can't add a new key into hash during iteration

[tboyko]

We're experiencing a very intermittent error using http-2 by way of net-http2 by way of Apnotic

E, [2018-03-27T03:50:05.485152 #657] ERROR -- : Actor crashed!
RuntimeError: can't add a new key into hash during iteration
	/var/www/[removed]/shared/bundle/ruby/2.5.0/gems/http-2-0.8.4/lib/http/2/connection.rb:669:in `activate_stream'
	/var/www/[removed]/shared/bundle/ruby/2.5.0/gems/http-2-0.8.4/lib/http/2/connection.rb:109:in `new_stream'
	/var/www/[removed]/shared/bundle/ruby/2.5.0/gems/net-http2-0.16.0/lib/net-http2/client.rb:86:in `new_stream'
	/var/www/[removed]/shared/bundle/ruby/2.5.0/gems/net-http2-0.16.0/lib/net-http2/client.rb:93:in `new_monitored_stream_for'
	/var/www/[removed]/shared/bundle/ruby/2.5.0/gems/net-http2-0.16.0/lib/net-http2/client.rb:40:in `call_async'
	/var/www/[removed]/shared/bundle/ruby/2.5.0/gems/apnotic-1.3.0/lib/apnotic/connection.rb:85:in `delayed_push_async'
	/var/www/[removed]/shared/bundle/ruby/2.5.0/gems/apnotic-1.3.0/lib/apnotic/connection.rb:49:in `push_async'
	/var/www/[removed]/releases/20170327013745/lib/[removed]/connection.rb:98:in `send_notification'
	/var/www/[removed]/shared/bundle/ruby/2.5.0/gems/celluloid-0.17.3/lib/celluloid/calls.rb:28:in `public_send'
	/var/www/[removed]/shared/bundle/ruby/2.5.0/gems/celluloid-0.17.3/lib/celluloid/calls.rb:28:in `dispatch'
	/var/www/[removed]/shared/bundle/ruby/2.5.0/gems/celluloid-0.17.3/lib/celluloid/call/async.rb:7:in `dispatch'
	/var/www/[removed]/shared/bundle/ruby/2.5.0/gems/celluloid-0.17.3/lib/celluloid/cell.rb:50:in `block in dispatch'
	/var/www/[removed]/shared/bundle/ruby/2.5.0/gems/celluloid-0.17.3/lib/celluloid/cell.rb:76:in `block in task'
	/var/www/[removed]/shared/bundle/ruby/2.5.0/gems/celluloid-0.17.3/lib/celluloid/actor.rb:337:in `block (2 levels) in task'
	/var/www/[removed]/shared/bundle/ruby/2.5.0/gems/celluloid-0.17.3/lib/celluloid/task.rb:97:in `exclusive'
	/var/www/[removed]/shared/bundle/ruby/2.5.0/gems/celluloid-0.17.3/lib/celluloid.rb:421:in `exclusive'
	/var/www/[removed]/shared/bundle/ruby/2.5.0/gems/celluloid-0.17.3/lib/celluloid/actor.rb:337:in `block in task'
	/var/www/[removed]/shared/bundle/ruby/2.5.0/gems/celluloid-0.17.3/lib/celluloid/task.rb:44:in `block in initialize'
	/var/www/[removed]/shared/bundle/ruby/2.5.0/gems/celluloid-0.17.3/lib/celluloid/task/fibered.rb:14:in `block in create'

The issue at connection.rb:669 is not readily apparent to me. Perhaps this is a concurrency problem with some other part of the library iterating over this array at the same time?

[igrigorik]

Hmm.. @ostinelli any insights on what might be the issue here?

[ostinelli]

Unfortunately not, though it does seem an issue related to Celluloid and concurrency used with Http2 / NetHttp2 / Apnotic. Never seen this one myself, though I have not used any of this together with Celluloid.

[igrigorik]

@tboyko I'd suggest routing this to Celluloid tracker.. Without a repro case, it's hard to provide any meaningful feedback here.

[ioquatix]

Are you sharing state between actors? By default they run on their own threads IIRC. I don't think http-2 is thread safe (which is probably to be expected).

[tboyko]

Nope. Each actor has its own instance.

…

On Jun 30, 2018, at 3:05 PM, Samuel Williams ***@***.***> wrote: Are you sharing state between actors? By default they run on their own threads IIRC. I don't think http-2 is thread safe (which is probably to be expected). — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[ioquatix]

Is it possible you are passing something between actors as an argument?

[tboyko]

The only object that Apnotic touches is a P12 string that gets wrapped by a StringIO object within the context of the actor. That and the actual notification that is passed to the actor to be sent. I would assume these do not pose issues but maybe I’m misunderstanding something?

…

On Sat, Jun 30, 2018 at 5:03 PM Samuel Williams ***@***.***> wrote: Is it possible you are passing something between actors as an argument? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#123 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA7LZJAC0ydey2QWOVAjgGSN1gek5rx8ks5uCBHfgaJpZM4S9i0A> .

[ioquatix]

As long as the ownership is passed to the other actor it should be okay. That means once you dispatch the request, e.g. actor.async.make_request(body, notification), you don't touch body or notification again.

How does the read/write loop work?

Because if you are not careful, it might be possible you have multiple fibers entering into the http-2 code and that could be causing problems. Can you point me at the code where you do reading and writing from the network?

[tboyko]

Unfortunately this is from a project that we had to abandon due to this issue (and failing to resolve it ourselves). The top-level library we were using was Apnotic (https://github.com/ostinelli/apnotic) in conjunction with Celluloid. Each actor had its own instance of an apnotic connection and the methods in each actor cell were labeled exclusive. Such being the case, the http-2 interface should have only been interacted with in a synchronous manner. Since I do not have a horse in the race any longer, this ticket can be closed unless anyone else is experiencing this or you’d like to investigate further.

…

On Sat, Jun 30, 2018 at 5:45 PM Samuel Williams ***@***.***> wrote: As long as the ownership is passed to the other actor it should be okay. That means once you dispatch the request, e.g. actor.async.make_request(body, notification), you don't touch body or notification again. How does the read/write loop work? Because if you are not careful, it might be possible you have multiple fibers entering into the http-2 code and that could be causing problems. Can you point me at the code where you do reading and writing from the network? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#123 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA7LZB7aNgvl4iZrYrHppW0PJnko_J0wks5uCBurgaJpZM4S9i0A> .

[ioquatix]

Fair enough.

My understanding that celluloid is no longer being maintained. My last experience with celluloid-io was that it was buggy. So it may well not be an issue with apnotic/http-2.

If you are looking for a solution, I'm working on async-http which might provide the foundation you require for further development. Let me know if you are interested. https://github.com/socketry/async-http

[ioquatix]

Also I see it's quite an old version of http-2. Perhaps updating that would be worth a try.

[ioquatix]

Just for interest, I took a look where it was possible that streams was being enumerated:

http-2/lib/http/2/connection.rb

Lines 540 to 555 in d6dcc7e

	when :local
	@local_window = @local_window - @local_window_limit + v
	@streams.each do |_id, stream|
	stream.emit(:local_window, stream.local_window - @local_window_limit + v)
	end
	@local_window_limit = v
	when :remote
	@remote_window = @remote_window - @remote_window_limit + v
	@streams.each do |_id, stream|
	# Event name is :window, not :remote_window
	stream.emit(:window, stream.remote_window - @remote_window_limit + v)
	end
	@remote_window_limit = v
	end

So it seems odd that it would be in that block at the same time as trying to create a new stream... I'd need to see the celluloid actor code.

[tboyko]

Very interesting, both about the enumeration and about celluloid. Thank you for your insights.

…

On Sun, Jul 1, 2018 at 2:57 PM Samuel Williams ***@***.***> wrote: Just for interest, I took a look where it was possible that streams was being enumerated: https://github.com/igrigorik/http-2/blob/d6dcc7e09ff7d77fced40862f53114f4f7af7665/lib/http/2/connection.rb#L540-L555 So it seems odd that it would be in that block at the same time as trying to create a new stream... I'd need to see the celluloid actor code. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#123 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA7LZEUA-Nn4EmwXD5Jzl_C4jxC86KW3ks5uCUW8gaJpZM4S9i0A> .