Firewalld rule change temporarily breaks new connections

[dylancanfield]

Cookbook version

2.6.2

Chef-client version

12.17

Platform Details

RHEL 7

Scenario:

For a load-balanced, distributed, service adding or removing systems (and therefore firewall rules) causes the firewalld provider to flush all rules and re-add rules one by one. Since the default behavior is 'drop', connections are denied until the relevant rule happens to be added. For our systems this can take 2 seconds or longer.

Additionally, this issue can be triggered intermittently just by re-converging with the same rules when using chef search. For example:

query = 'policy_name:my_web_host_policy'
search(:node, query).each do |n|                                                
  firewall_rule "Allow database traffic from #{n['fqdn']}" do                   
    source n['ipaddress']                                                       
    port 3306                                                                   
    protocol :tcp                                                               
  end                                                                           
end 

In the above scenario, occasionally the chef server will return a list of nodes in a different order. This causes /etc/sysconfig/firewall-chef.rules to be updated (because the content is in a different order) and the firewalld rules are dropped and re-loaded.

Steps to Reproduce:

Add or remove firewall rules. Or use a chef search, which returns 2 or more nodes, to generate firewall rules.

Expected Result:

Existing firewall rules are not dropped if they exist in the current chef run.

Actual Result:

When one or more firewall rule changes due to addition, subtraction, or re-ordering, all firewall rules are temporarily discarded.

[martinb3]

Hi there -- I don't have a great solution for this. You may want to check out one of the firewalld-specific cookbooks, which will go deeper into support for that specific technology (vs. being a catch all for iptables, ufw, firewalld, and Windows).

The only improvement, I've come up with, would be writing a library that can interact with the kernel netfilter subsystem and intelligently add and remove only the changed rules, using the various services' actual configuration file mechanism was the next best thing. firewalld, iptables, ufw often treat a file as a single group of rules to load at once, on startup (and we also support Windows which behave differently). Ideally loading a file of rules wouldn't take 2 full seconds, but I understand sometimes that's a limitation with lots of rules.

I would strongly recommend you sort the results of a Chef search (when using it as a general discovery mechanism), if they're non-deterministic, to avoid shuffling rules, as you're seeing. This would take care of at least that ordering issue causing so much reordering.

[decoyjoe]

This was fixed in #304 (cc: @bmhughes).

The firewall_rule resource on firewalld platforms now implements rules as firewalld rich rules. This allows individual rules to be added without flushing/disturbing any other existing rules in the firewalld zone.