Initial VRF-Lite for NDFC (#181)

* initial policy work

* change to pipe

* more work on templating into module structure

* more work on policies and policy_group buildout under a switch

* refactor template

* refactor template complete

* refactor template again & start of cross-ref rule

* update template & task using policy module

* small updates for policy create

* prototype use case policy

* rebase for only initial policy

* more updates for pr

* resolve lint errors

* resolve lint errors

* resolve lint errors

* resolve lint errors

* resolve lint errors

* remove render roles items

* update initial render tasks

* updated ndfc_vrf_lite_bgp.j2

* adjustment to where switch cmds are generated

* fix lint errors

* udpated j2 file for the vrf lite ebgp

* commit fix for dup names

* Updated MD to MD_Extended for the policy

* adding ndfc_vrf_lite_ospf template

* updates for ndfc_vrf_lite_ospf

* updated ndfc_vrf_lite_bgp.j2

* fixing typo in the vrf lite bgp jinja2

* add initial prepare plugin for vrf lites & remove render role dependency

* update ignore files

* moving vrf lite bgp and ospf jinja2 files to the common role

* expanded vrf-lite-ospf template for interface ospf

* format update

* expanded vrf lite bgp j2

* updated vrf lite bgp jinja2

* updating defaults to match the nac-vxlan repo updates

* fix defaults and update vrf_lite_ospf jinja

* prepare vrf liteupdates:
Updates
- adding logic to update redistribution secion of each switch to use the
  global config
- update ospf secion of each interface is ospf is enabled and ospf
  section is not defined under the interface or default area is not
given

* remove tailing spaces

* lint

* clean up render role

* remove render.py from ignore txt

* fixed a typo in the code

* updated vrf lite bgp jinja2

* fix for ospf default area

* fix a bug

* updated the vrf lite bgp jinja2 template

* updated the vrf lite bgp jinja2 template

* udpated and renamed vrf lite ebgp jinja2 file

* Removed comment from the vrf_lite ebgp jinja2 file

* update ospf vrf-lite template for bfd

* ospf vrf-lite template updates for bfd

* Expanded ndfc_vrf_lite_ebgp.j2

* Updated ndfc_vrf_lite_ebgp.j2

* partial fix for default_area in prep plugin

* minor fix of prepare vrf_lite plugin

* fixes for area number in vrf lite ospf template

* updated ndfc_vrf_lite_ebgp.j2

* update ospf default cost

* update defaults for ospf cost and ebgp_multihop

* updated defauts for lsa_retransit_interval and lsa_transmit_delay

* Updated vrf lite ebgp jinja2

* removed route reflector client config

* update allow_as parameter

Signed-off-by: ccoueffe <[email protected]>

* update allow_as_in_number to 3

Signed-off-by: ccoueffe <[email protected]>

* rename allowas to allow_as

Signed-off-by: ccoueffe <[email protected]>

* update jinja ospf

Signed-off-by: ccoueffe <[email protected]>

* update jinja for ospf cost

* pass defaults to jinja template

* pass defaults to jinja template

* updated jinja template with default references

* add condition for ospf + direct and static redistribution

Signed-off-by: ccoueffe <[email protected]>

* add helper function for hostname to ip address mapping lookup

* remove 108 reference

* update interface name + router_id

Signed-off-by: ccoueffe <[email protected]>

* updated multihop defalut in the vrf lite ebgp j2

* Fixed the bgp_peers indentation in the defaults

* vrf-lite ospf jinja updates: fixed duplication for bfd enabled; fixed redistribution

* removed route-map from the vrf lite ebgp jinja2

* create global bfd and redistribution values and update condition for bfd and ospf redistribution in bgp

* updated vrf lite ebgp jinja2

* Updates in the vrf lite ebgp jinja2

* defaults update in the vrf lite ebgp jinja2

* update wording for VRF-lite ospf template

* removed when condition on policies.yml

* update send-community to false

Signed-off-by: ccoueffe <[email protected]>

* commented distance since its wip

* removed redistribute direct route-map since it's configured by default and we see out of sync

* updated ospf jinja template for area parameters and distance

* Removed direct redistribution

* Prep plugin update for the vrf lite address family and ebgp jinja2 updates

* addressed whitespaces

* addressed whitespaces

* addressed whitespaces

* added check for the bgp in the 107 prep plugin

* updated vrf-lite ospf jinja, fix for area check and remove auth

* updates to use ipaddr filter to get area id in IP format

* updates for ospf interface authentication

* update defaults for interface authentication

* fix for interface auth

* update area id if area is equal to 0. ipaddress(0) equal to False not 0.0.0.0

* Add condition if default-cost if default of 1 (default value) don't push cli (out-of-sync issue)

* fix trailing-whitespace

* fix trailing-whitespace

* Add rule 502 and update typo ospf jinja

Signed-off-by: ccoueffe <[email protected]>

* Add conditions to rule 502

Signed-off-by: ccoueffe <[email protected]>

* clean PEP8 and rewrite with function

* fixed closing braket

* update error message with path

* Update data model path in the comments

* update comment with key data model

* rewrite and fix static function

* move result variable

* Strip outer defaults key (#206) (#207)

Co-authored-by: Mike Wiebe <[email protected]>

* updating the defaults in ndfc_vrf_lite_ebgp.j2

* update for defaults bfd

* fix for bfd and all other fixes for custom defaults

* fix for loopback lowercase in rules

* add rule for loopback and passive-interface

* updates to exclude passive-interface for loopback interfaces

* added bfd in the vrf lite ebgp jinja2

* updates for custom default

* fix for ospf.distance and auth_type

* update for area_type for custom defaults

* add check ospf process

* update script 502

* fix typo

* adding feature ospf to template for vrf-lite ospf

* update prep 107 to allow configuration without vrf-lite and reformat rule 502 to fit PEP8 (long lines, quotes)

* update defaults and ospf template for nssa and default-information originate

* add verification in rule 502 for nssa

* add missing keywork translate type7 for supress-fa, never and and always

* Temp update prep_107_vrf_lites.py

* Temp update prep_107_vrf_lites.py

* Fix typo and order

* Update ndfc_vrf_lite_ospf.j2

remove list for nssa and default_originate

* Update 502_policy_vrf_lite_cross_reference.py

update class check_global_ospf_nssa: remove list

* Update prep_107_vrf_lites.py

Adding to bypass pylint sanity check error

* Update prep_107_vrf_lites.py

Update pylint location to bypass sanity checks

* Update prep_107_vrf_lites.py

removing whitespace

* Update requirements.txt

To use the ipaddr filter in Ansible, you need to install the netaddr Python library.  Required for prep_plugin 107. https://docs.ansible.com/ansible/latest/collections/ansible/utils/docsite/filters_ipaddr.html

* Update ndfc_vrf_lite_ospf.j2

reorder option nssa
no-summary should be before no-redistribution in the show run.

Ex:
router ospf OVERLAY
  vrf nac-vrf01
    router-id 1.1.1.1
    area 0.0.0.10 nssa no-summary no-redistribution default-information-originate
    area 0.0.0.20 nssa translate type7 always supress-fa
    default-information originate

* Update prep_107_vrf_lites.py

update remove lines 40-42 to 41-43

* Update ndfc_vrf_lite_ospf.j2

remote list line 78
 {% set _= nssa_options_flags.append(area.nssa.route_map) %}

* Update prep_107_vrf_lites.py

remove nac_ from unique_name, because it was adding in policy.j2 now.

* Update sub_main.yml

Removing comment for end_play

---------

Signed-off-by: ccoueffe <[email protected]>
Co-authored-by: devegupt <[email protected]>
Co-authored-by: Justin Burnette (juburnet) <[email protected]>
Co-authored-by: mwiebe <[email protected]>
Co-authored-by: Shangxin Du <[email protected]>
Co-authored-by: ccoueffe <[email protected]>
Co-authored-by: Devendra Gupta <[email protected]>
Co-authored-by: Charly Coueffe <[email protected]>
Co-authored-by: juburnet <[email protected]>

plugins/action/common/prepare_plugins/prep_107_vrf_lites.py

@@ -0,0 +1,288 @@
+# Copyright (c) 2024 Cisco Systems, Inc. and its affiliates
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy of
+# this software and associated documentation files (the "Software"), to deal in
+# the Software without restriction, including without limitation the rights to
+# use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of
+# the Software, and to permit persons to whom the Software is furnished to do so,
+# subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in all
+# copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
+# FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
+# COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER
+# IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+# CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+#
+# SPDX-License-Identifier: MIT
+
+from ansible.utils.display import Display
+from jinja2 import ChainableUndefined, Environment, FileSystemLoader
+from ....plugin_utils.helper_functions import hostname_to_ip_mapping
+from ansible_collections.ansible.utils.plugins.filter import ipaddr
+
+display = Display()
+
+
+class PreparePlugin:
+    def __init__(self, **kwargs):
+        self.kwargs = kwargs
+        self.keys = []
+
+    def prepare(self):
+        templates_path = self.kwargs['templates_path']
+        model_data = self.kwargs['results']['model_extended']
+        default_values = self.kwargs['default_values']
+
+        # Remove lines 41-43 after route control that includes route-maps, prefix-lists, etc is merged
+        display.warning("VRF-Lite in VXLAN as Code is currently not supported.")
+        # pylint: disable=unreachable
+        return self.kwargs['results']
+
+        template_filename = "ndfc_vrf_lite.j2"
+        # pylint: enable=unreachable
+        env = Environment(
+            loader=FileSystemLoader(templates_path),
+            undefined=ChainableUndefined,
+            lstrip_blocks=True,
+            trim_blocks=True,
+        )
+
+        env.filters["ipaddr"] = ipaddr.ipaddr
+        template = env.get_template(template_filename)
+        if "overlay_extensions" in model_data["vxlan"]:
+            if "vrf_lites" in model_data["vxlan"]["overlay_extensions"]:
+                for vrf_lite in model_data["vxlan"]["overlay_extensions"]["vrf_lites"]:
+                    ospf_enabled = True if vrf_lite.get(
+                        "ospf") is not None else False
+                    default_area = vrf_lite.get("ospf", {}).get(
+                        "default_area",
+                        default_values["vxlan"]["overlay_extensions"]["vrf_lites"]["ospf"]["default_area"]
+                    )
+                    for switch in vrf_lite["switches"]:
+                        unique_name = f"{vrf_lite['name']}_{switch['name']}"
+
+                        # Adding redistribution secion under the switches with the vrf_lite global config if it is not defined
+                        # for example:
+                        # before:
+                        # vxlan:
+                        #   overlay_extensions:
+                        #     vrf_lites:
+                        #       - name: ospf_vrf_red
+                        #         vrf: vrf_red
+                        #         redistribution:
+                        #           - source: direct
+                        #             route_map: fabric-rmp-redist-direct
+                        #           - source: static
+                        #             route_map: fabric-rmp-redist-static
+                        #         switches:
+                        #           - name: dc-border1
+                        # after:
+                        # vxlan:
+                        #   overlay_extensions:
+                        #     vrf_lites:
+                        #       - name: ospf_vrf_red
+                        #         vrf: vrf_red
+                        #         redistribution:
+                        #           - source: direct
+                        #             route_map: fabric-rmp-redist-direct
+                        #           - source: static
+                        #             route_map: fabric-rmp-redist-static
+                        #         switches:
+                        #           - name: dc-border1
+                        #             redistribution:
+                        #               - source: direct
+                        #                 route_map: fabric-rmp-redist-direct
+                        #               - source: static
+                        #                 route_map: fabric-rmp-redist-static
+
+                        g_redist = vrf_lite.get("redistribution", [])
+                        if switch.get("redistribution", []) == []:
+                            switch["redistribution"] = g_redist
+
+                        # Adding ospf section under the interfaces config if ospf is not define with the default area id
+                        # for example:
+                        # before:
+                        # vxlan:
+                        #   overlay_extensions:
+                        #     vrf_lites:
+                        #       - name: ospf_vrf_red
+                        #         vrf: vrf_red
+                        #         ospf:
+                        #           process: overlay
+                        #           default_area: 0
+                        #         switches:
+                        #           - name: dc-border1
+                        #             interfaces:
+                        #               - name: ethernet1/1
+                        #                 ospf:
+                        #                   area: 1
+                        #               - name: ethernet1/2
+                        #               - name: ethernet1/3
+                        #                 ospf:
+                        #                   area_cost: 55
+                        # after:
+                        # vxlan:
+                        #   overlay_extensions:
+                        #     vrf_lites:
+                        #       - name: ospf_vrf_red
+                        #         vrf: vrf_red
+                        #         ospf:
+                        #           process: overlay
+                        #           default_area: 0
+                        #         switches:
+                        #           - name: dc-border1
+                        #             interfaces:
+                        #               - name: ethernet1/1
+                        #                 ospf:
+                        #                   area: 1
+                        #               - name: ethernet1/2
+                        #                 ospf:
+                        #                   area: 0
+                        #               - name: ethernet1/3
+                        #                 ospf:
+                        #                   area: 0
+                        #                   area_cost: 55
+
+                        for intf_index in range(len(switch.get("interfaces", []))):
+                            intf = switch["interfaces"][intf_index]
+                            if not ospf_enabled or (intf.get("ospf") is not None and intf["ospf"].get("area", -1) != -1):
+                                continue
+                            if intf.get("ospf") is None:
+                                intf["ospf"] = {
+                                    "area": default_area
+                                }
+                            else:
+                                intf["ospf"]["area"] = default_area
+                            switch["interfaces"][intf_index] = intf
+
+                        # Adding address_family_ipv4_unicast and address_family_ipv6_unicast and child keys
+                        # under switches with the vrf_lite global config if it is not defined.
+                        # for example:
+                        # before:
+                        # vxlan:
+                        #   overlay_extensions:
+                        #     vrf_lites:
+                        #       - name: myvrf_50001_vrf_lite
+                        #         vrf: myvrf_50001
+                        #         bgp:
+                        #           local_as: 1111
+                        #           address_family_ipv4_unicast:
+                        #               additional_paths_receive: true
+                        #               additional_paths_send: true
+                        #               additional_paths_selection_route_map: test-map-globalaf4
+                        #               default_originate: true
+                        #               ebgp_distance: 25
+                        #               ibgp_distance: 180
+                        #               local_distance: 200
+                        #           address_family_ipv6_unicast:
+                        #               additional_paths_receive: true
+                        #               additional_paths_send: true
+                        #               additional_paths_selection_route_map: test-map-globalaf6
+                        #               default_originate: true
+                        #               ebgp_distance: 25
+                        #               ibgp_distance: 180
+                        #               local_distance: 200
+                        #         switches:
+                        #           - name: netascode4-ebgp-bl1
+                        #             bgp:
+                        #               local_as: 1111
+                        # after:
+                        # vxlan:
+                        #   overlay_extensions:
+                        #     vrf_lites:
+                        #       - name: myvrf_50001_vrf_lite
+                        #         vrf: myvrf_50001
+                        #         bgp:
+                        #           local_as: 1111
+                        #           address_family_ipv4_unicast:
+                        #               additional_paths_receive: true
+                        #               additional_paths_send: true
+                        #               additional_paths_selection_route_map: test-map-globalaf4
+                        #               default_originate: true
+                        #               ebgp_distance: 25
+                        #               ibgp_distance: 180
+                        #               local_distance: 200
+                        #           address_family_ipv6_unicast:
+                        #               additional_paths_receive: true
+                        #               additional_paths_send: true
+                        #               additional_paths_selection_route_map: test-map-globalaf6
+                        #               default_originate: true
+                        #               ebgp_distance: 25
+                        #               ibgp_distance: 180
+                        #               local_distance: 200
+                        #         switches:
+                        #           - name: netascode4-ebgp-bl1
+                        #             bgp:
+                        #               local_as: 1111
+                        #               address_family_ipv4_unicast:
+                        #                   additional_paths_receive: true
+                        #                   additional_paths_send: true
+                        #                   additional_paths_selection_route_map: test-map-globalaf4
+                        #                   default_originate: true
+                        #                   ebgp_distance: 25
+                        #                   ibgp_distance: 180
+                        #                   local_distance: 200
+                        #               address_family_ipv6_unicast:
+                        #                   additional_paths_receive: true
+                        #                   additional_paths_send: true
+                        #                   additional_paths_selection_route_map: test-map-globalaf6
+                        #                   default_originate: true
+                        #                   ebgp_distance: 25
+                        #                   ibgp_distance: 180
+                        #                   local_distance: 200
+                        if "bgp" in vrf_lite:
+                            for af in ["address_family_ipv4_unicast", "address_family_ipv6_unicast"]:
+                                if af in vrf_lite["bgp"]:
+                                    switch_bgp_af = switch.setdefault("bgp", {}).setdefault(af, {})
+                                    for key, value in vrf_lite["bgp"][af].items():
+                                        if key not in switch_bgp_af:
+                                            switch_bgp_af[key] = value
+
+                        output = template.render(
+                            MD_Extended=model_data, item=vrf_lite, switch_item=switch, defaults=default_values)
+
+                        new_policy = {
+                            "name": unique_name,
+                            "template_name": "switch_freeform",
+                            "template_vars": {
+                                "CONF": output
+                            }
+                        }
+
+                        model_data["vxlan"]["policy"]["policies"].append(new_policy)
+
+                        if any(sw['name'] == switch['name'] for sw in model_data["vxlan"]["policy"]["switches"]):
+                            found_switch = next(([idx, i] for idx, i in enumerate(
+                                model_data["vxlan"]["policy"]["switches"]) if i["name"] == switch['name']))
+                            if "groups" in found_switch[1].keys():
+                                model_data["vxlan"]["policy"]["switches"][found_switch[0]]["groups"].append(
+                                    unique_name)
+                            else:
+                                model_data["vxlan"]["policy"]["switches"][found_switch[0]]["groups"] = [
+                                    unique_name]
+                        else:
+                            new_switch = {
+                                "name": switch["name"],
+                                "groups": [unique_name]
+                            }
+                            model_data["vxlan"]["policy"]["switches"].append(
+                                new_switch)
+
+                        if not any(group['name'] == vrf_lite['name'] for group in model_data["vxlan"]["policy"]["groups"]):
+                            new_group = {
+                                "name": unique_name,
+                                "policies": [
+                                    {"name": unique_name},
+                                ],
+                                "priority": 500
+                            }
+                            model_data["vxlan"]["policy"]["groups"].append(new_group)
+
+            model_data = hostname_to_ip_mapping(model_data)
+
+        self.kwargs['results']['model_extended'] = model_data
+        return self.kwargs['results']

plugins/action/common/prepare_service_model.py

@@ -47,6 +47,8 @@ def run(self, tmp=None, task_vars=None):
         # Get Data from Ansible Task
         ihn = self._task.args['inventory_hostname']
         hvs = self._task.args['hostvars']
+        tp = self._task.args['templates_path']
+        default_values = self._task.args['default_values']
 
         # sm_data contains the golden untouched model data
         sm_data = self._task.args['model_data']
@@ -77,7 +79,11 @@ def run(self, tmp=None, task_vars=None):
                 results['msg'] = f"Plugin {plugin_name} must have a list of keys"
             # Call each plugin in a loop
             results = dict_of_plugins[plugin_name].PreparePlugin(
-                host_name=ihn, hostvars=hvs, results=results).prepare()
+                host_name=ihn,
+                hostvars=hvs,
+                default_values=default_values,
+                templates_path=tp,
+                results=results).prepare()
 
         if results['failed']:
             # If there is a failure, remove the model data to make the failure message more readable

plugins/plugin_utils/helper_functions.py

@@ -55,6 +55,31 @@ def data_model_key_check(tested_object, keys):
     return dm_key_dict
 
 
+def hostname_to_ip_mapping(model_data):
+    """
+    Update in-memory data model with IP address mapping to hostname.
+
+    :Parameters:
+        :model_data (dict): The in-memory data model.
+
+    :Returns:
+        :model_data: The updated in-memory data model with IP address mapping to hostname.
+
+    :Raises:
+        N/A
+    """
+    topology_switches = model_data['vxlan']['topology']['switches']
+    for switch in model_data['vxlan']['policy']['switches']:
+        if any(sw['name'] == switch['name'] for sw in topology_switches):
+            found_switch = next((item for item in topology_switches if item["name"] == switch['name']))
+            if found_switch.get('management').get('management_ipv4_address'):
+                switch['name'] = found_switch['management']['management_ipv4_address']
+            elif found_switch.get('management').get('management_ipv6_address'):
+                switch['name'] = found_switch['management']['management_ipv6_address']
+
+    return model_data
+
+
 def ndfc_get_switch_policy(self, task_vars, tmp, switch_serial_number):
     """
     Get NDFC policy for a given managed switch by the switch's serial number.