[MANOPD-70719] Verify plugins restart after template applying

documentation/Installation.md

@@ -2097,6 +2097,10 @@ For Kubernetes and ETCD to work correctly, it is recommended to configure the sy
 
 *OS specific*: Yes, performs only on the RHEL OS family.
 
+**Warning:** incorrect time synchronization can lead to incorrect operation of the cluster or services. You can validate
+the time synchronization via the [Time difference](Kubecheck.md#218-time-difference) test between the nodes from 
+[PAAS Check procedure](Kubecheck.md#paas-procedure).
+
 To synchronize the system time, you must make a list of NTP servers. All servers must be accessible from any node of the cluster.
 The list should be indicated in the `chrony` section of the` services.ntp` section config file.
 In addition to the NTP server address, you can specify any additional configurations in the same line. 
@@ -2143,6 +2147,10 @@ If the configuration `services.ntp.chrony.servers` is absent, then the task` pre
 
 *OS specific*: Yes, performs only on Debian OS family.
 
+**Warning:** incorrect time synchronization can lead to incorrect operation of the cluster or services. You can validate
+the time synchronization via the [Time difference](Kubecheck.md#218-time-difference) test between the nodes from 
+[PAAS Check procedure](Kubecheck.md#paas-procedure).
+
 To synchronize the system time, you must make a list of NTP servers. All servers must be accessible from any node of the cluster.
 The list should be indicated in the `timesyncd.Time.NTP` parameter of the` services.ntp` section config file.
 In addition to the NTP server address, you can specify any additional configurations in the same line. 

documentation/Kubecheck.md

@@ -4,8 +4,8 @@ This section provides information about the Kubecheck functionality.
 - [Check Procedures](#check-procedures)
   - [IAAS Procedure](#iaas-procedure)
     - [001 Connectivity](#001-connectivity)
-    - [002 Latency - Single Thread](#002-latency-single-thread)
-    - [003 Latency - Multi Thread](#003-latency-multi-thread)
+    - [002 Latency - Single Thread](#002-latency---single-thread)
+    - [003 Latency - Multi Thread](#003-latency---multi-thread)
     - [004 Sudoer Access](#004-sudoer-access)
     - [005 Items Amount](#005-items-amount)
       - [005 VIPs Amount](#005-vips-amount)
@@ -48,7 +48,8 @@ This section provides information about the Kubecheck functionality.
     - [214 Selinux configuration](#214-selinux-configuration)
     - [215 Firewalld status](#215-firewalld-status)
     - [216 Swap state](#216-swap-state)
-    - [217 Modprobe rules](#216-modprobe-rules)
+    - [217 Modprobe rules](#217-modprobe-rules)
+    - [218 Time difference](#218-time-difference)
 - [Report File Generation](#report-file-generation)
   - [HTML Report](#html-report)
   - [CSV Report](#csv-report)
@@ -436,37 +437,50 @@ This test checks the condition `Ready` of the Kubernetes nodes of the cluster.
 
 ###### 213 Selinux security policy
 
-*Task*: `kubemarine.procedures.check_paas.verify_selinux_status`
+*Task*: `services.security.selinux.status`
 
 The test checks the status of Selinux. It must be `enforcing`. It may be `permissive`, but must be explicitly specified
 in the inventory. Otherwise, the test will fail. This test is applicable only for systems of the RHEL family.
 
 ###### 214 Selinux configuration
 
-*Task*: `kubemarine.procedures.check_paas.verify_selinux_config`
+*Task*: `services.security.selinux.config`
 
 The test compares the configuration of Selinux on the nodes with the configuration specified in the inventory or with the
 one by default. If the configuration does not match, the test will fail.
 
 ###### 215 Firewalld status
 
-*Task*: `kubemarine.procedures.check_paas.verify_firewalld_status`
+*Task*: `services.security.firewalld.status`
 
 The test verifies that the FirewallD is disabled on cluster nodes, otherwise the test will fail.
 
 ###### 216 Swap state
 
-*Task*: `kubemarine.procedures.check_paas.verify_swap_state`
+*Task*: `services.system.swap.status`
 
 The test verifies that swap is disabled on all nodes in the cluster, otherwise the test will fail.
 
 ###### 217 Modprobe rules
 
-*Task*: `kubemarine.procedures.check_paas.verify_modprobe_rules`
+*Task*: `services.system.modprobe.rules`
 
 The test compares the modprobe rules on the nodes with the rules specified in the inventory or with default rules. If
 rules does not match, the test will fail.
 
+###### 218 Time difference
+
+*Task*: `services.system.time`
+
+The test verifies that the time between all nodes does not differ by more than the maximum limit value. Otherwise, the 
+cluster may not work properly and the test will be highlighted with a warning.
+
+Maximum limit value: 15000ms
+
+**Note:** this test can give a false-positive result if there are a lot of nodes in the cluster, there is too much delay
+between the deployer node and all the others, or any other conditions of the environment. It is also recommended to be 
+sure to perform latency tests: [002 Latency - Single Thread](#002-latency---single-thread) and 
+[003 Latency - Multi Thread](#003-latency---multi-thread).
 
 
 ### Report File Generation

kubemarine/core/annotations.py

@@ -27,3 +27,17 @@ def wrapper(group: NodeGroup, *args, **kwargs):
             raise Exception(f'Method "{str(fn)}" do not supports multi-os group')
         return fn(group, *args, **kwargs)
     return wrapper
+
+
+def restrict_empty_group(fn: callable):
+    """
+    Method is an annotation that prohibits passing empty groups to the function.
+    :param fn: Origin function to apply annotation validation to
+    :return: Validation wrapper function
+    """
+    def wrapper(group: NodeGroup, *args, **kwargs):
+        # TODO: walk through all nodes in *args, check isinstance NodeGroup and perform validation
+        if group.is_empty():
+            raise Exception(f'Method "{str(fn)}" prohibits passing empty groups to it')
+        return fn(group, *args, **kwargs)
+    return wrapper

kubemarine/core/cluster.py

@@ -194,7 +194,7 @@ def gather_facts(self, step) -> None:
             self.context = t_cluster.context
         elif step == 'after':
             self.remove_invalid_cri_config(self.inventory)
-            # Method "kubetool.system.is_multiple_os_detected" is not used because it detects OS family for new nodes
+            # Method "kubemarine.system.is_multiple_os_detected" is not used because it detects OS family for new nodes
             # only, while package versions caching performs on all nodes.
             if self.nodes['all'].get_nodes_os(suppress_exceptions=True, force_all_nodes=True) != 'multiple':
                 self.cache_package_versions()

kubemarine/core/group.py

@@ -222,9 +222,21 @@ def get_nonzero_nodes_group(self) -> NodeGroup:
         nodes_list = self.get_nonzero_nodes_list()
         return self.cluster.make_group(nodes_list)
 
+    def get_nodes_list_where_value_in_stdout(self, value: str) -> List[fabric.connection.Connection]:
+        """
+        Returns a list of node connections that contains the given string value in results stderr.
+        :param value: The string value to be found in the nodes results stderr.
+        :return: List with nodes connections
+        """
+        nodes_with_stderr_value: List[fabric.connection.Connection] = []
+        for conn, result in self.items():
+            if isinstance(result, fabric.runners.Result) and value in result.stdout:
+                nodes_with_stderr_value.append(conn)
+        return nodes_with_stderr_value
+
     def get_nodes_list_where_value_in_stderr(self, value: str) -> List[fabric.connection.Connection]:
         """
-        Returns a list of node connections that contains the given string value in results stderr
+        Returns a list of node connections that contains the given string value in results stderr.
         :param value: The string value to be found in the nodes results stderr.
         :return: List with nodes connections
         """
@@ -234,15 +246,40 @@ def get_nodes_list_where_value_in_stderr(self, value: str) -> List[fabric.connec
                 nodes_with_stderr_value.append(conn)
         return nodes_with_stderr_value
 
+    def get_nodes_group_where_value_in_stdout(self, value: str) -> NodeGroup:
+        """
+        Forms and returns new NodeGroup of nodes that contains the given string value in results stdout.
+        :param value: The string value to be found in the nodes results stdout.
+        :return: NodeGroup
+        """
+        nodes_list = self.get_nodes_list_where_value_in_stdout(value)
+        return self.cluster.make_group(nodes_list)
+
     def get_nodes_group_where_value_in_stderr(self, value: str) -> NodeGroup:
         """
-        Forms and returns new NodeGroup of nodes that contains the given string value in results stderr
+        Forms and returns new NodeGroup of nodes that contains the given string value in results stderr.
         :param value: The string value to be found in the nodes results stderr.
         :return: NodeGroup
         """
         nodes_list = self.get_nodes_list_where_value_in_stderr(value)
         return self.cluster.make_group(nodes_list)
 
+    def stdout_contains(self, value: str) -> bool:
+        """
+        Checks for the presence of the given string in all results stdout.
+        :param value: The string value to be found in the nodes results stdout.
+        :return: true if string presented
+        """
+        return len(self.get_nodes_list_where_value_in_stdout(value)) > 0
+
+    def stderr_contains(self, value: str) -> bool:
+        """
+        Checks for the presence of the given string in all results stderr.
+        :param value: The string value to be found in the nodes results stderr.
+        :return: true if string presented
+        """
+        return len(self.get_nodes_list_where_value_in_stderr(value)) > 0
+
     def __eq__(self, other) -> bool:
         if self is other:
             return True
@@ -732,7 +769,13 @@ def get_last_member(self, provide_node_configs=False, apply_filter=None):
     def get_any_member(self, provide_node_configs=False, apply_filter=None):
         member = random.choice(self.get_ordered_members_list(provide_node_configs=provide_node_configs,
                                                              apply_filter=apply_filter))
-        self.cluster.log.verbose(f'Selected node {str(member)}')
+        if isinstance(member, NodeGroup):
+            # to avoid "Selected node <kubemarine.core.group.NodeGroup object at 0x7f925625d070>" writing to log,
+            # let's get node ip from selected member and pass to it to log
+            member_str = str(list(member.nodes.keys())[0])
+        else:
+            member_str = str(member)
+        self.cluster.log.verbose(f'Selected node {member_str}')
         return member
 
     def get_member_by_name(self, name, provide_node_configs=False):

kubemarine/kubernetes/daemonset.py

@@ -0,0 +1,46 @@
+# Copyright 2021-2022 NetCracker Technology Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from kubemarine.core.cluster import KubernetesCluster
+from kubemarine.kubernetes.object import KubernetesObject
+
+
+class DaemonSet(KubernetesObject):
+
+    def __init__(self, cluster: KubernetesCluster, name=None, namespace=None, obj=None):
+        super().__init__(cluster, kind='DaemonSet', name=name, namespace=namespace, obj=obj)
+
+    def is_actual_and_ready(self) -> bool:
+        return self.is_ready() and self.is_up_to_date()
+
+    def is_up_to_date(self) -> bool:
+        desired_number_scheduled = self._obj.get('status', {}).get('desiredNumberScheduled')
+        updated_number_scheduled = self._obj.get('status', {}).get('updatedNumberScheduled')
+        return desired_number_scheduled is not None \
+               and updated_number_scheduled is not None \
+               and desired_number_scheduled == updated_number_scheduled
+
+    def is_ready(self) -> bool:
+        desired_number_scheduled = self._obj.get('status', {}).get('desiredNumberScheduled')
+        number_ready = self._obj.get('status', {}).get('numberReady')
+        return desired_number_scheduled is not None \
+               and number_ready is not None \
+               and desired_number_scheduled == number_ready
+
+    def is_scheduled(self) -> bool:
+        desired_number_scheduled = self._obj.get('status', {}).get('desiredNumberScheduled')
+        current_number_scheduled = self._obj.get('status', {}).get('currentNumberScheduled')
+        return desired_number_scheduled is not None \
+               and current_number_scheduled is not None \
+               and desired_number_scheduled == current_number_scheduled

kubemarine/kubernetes/deployment.py

@@ -0,0 +1,39 @@
+# Copyright 2021-2022 NetCracker Technology Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from kubemarine.core.cluster import KubernetesCluster
+from kubemarine.kubernetes.object import KubernetesObject
+
+
+class Deployment(KubernetesObject):
+
+    def __init__(self, cluster: KubernetesCluster, name=None, namespace=None, obj=None):
+        super().__init__(cluster, kind='Deployment', name=name, namespace=namespace, obj=obj)
+
+    def is_actual_and_ready(self) -> bool:
+        return self.is_ready() and self.is_up_to_date()
+
+    def is_up_to_date(self) -> bool:
+        desired_number_scheduled = self._obj.get('spec', {}).get('replicas')
+        updated_number_scheduled = self._obj.get('status', {}).get('updatedReplicas')
+        return desired_number_scheduled is not None \
+               and updated_number_scheduled is not None \
+               and desired_number_scheduled == updated_number_scheduled
+
+    def is_ready(self) -> bool:
+        desired_number_scheduled = self._obj.get('spec', {}).get('replicas')
+        number_ready = self._obj.get('status', {}).get('readyReplicas')
+        return desired_number_scheduled is not None \
+               and number_ready is not None \
+               and desired_number_scheduled == number_ready

kubemarine/kubernetes/object.py

@@ -0,0 +1,101 @@
+# Copyright 2021-2022 NetCracker Technology Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from __future__ import annotations
+
+import io
+import json
+import uuid
+import time
+
+import yaml
+
+from kubemarine.core.cluster import KubernetesCluster
+
+
+class KubernetesObject:
+    def __init__(self, cluster: KubernetesCluster, kind=None, name=None, namespace=None, obj=None):
+
+        self._cluster = cluster
+        self._reload_t = -1
+
+        if not kind and not name and not namespace and not obj:
+            raise RuntimeError('Not enough parameter values to construct the object')
+        if obj:
+            self._obj = obj
+        else:
+            if not kind or not name or not namespace:
+                raise RuntimeError('An unsynchronized object has not enough parameters '
+                                   'to be reloaded')
+            self._obj = {
+                "kind": kind,
+                "metadata": {
+                    "name": name,
+                    "namespace": namespace,
+                }
+            }
+
+    def __str__(self):
+        return self.to_yaml()
+
+    @property
+    def uid(self):
+        uid = self._obj.get('metadata', {}).get('uid')
+        if uid:
+            return uid
+
+        return str(uuid.uuid4())
+
+    @property
+    def kind(self):
+        return self._obj.get('kind').lower()
+
+    @property
+    def namespace(self):
+        return self._obj.get('metadata', {}).get('namespace').lower()
+
+    @property
+    def name(self):
+        return self._obj.get('metadata', {}).get('name').lower()
+
+    def to_json(self):
+        return json.dumps(self._obj)
+
+    def to_yaml(self):
+        return yaml.dump(self._obj)
+
+    def is_reloaded(self):
+        return self._reload_t > -1
+
+    def reload(self, master=None, suppress_exceptions=False) -> KubernetesObject:
+        if not master:
+            master = self._cluster.nodes['master'].get_any_member()
+        cmd = f'kubectl get {self.kind} -n {self.namespace} {self.name} -o json'
+        result = master.sudo(cmd, warn=suppress_exceptions)
+        self._cluster.log.verbose(result)
+        if not result.is_any_failed():
+            self._obj = json.loads(result.get_simple_out())
+            self._reload_t = time.time()
+        return self
+
+    def apply(self, master=None):
+        if not master:
+            master = self._cluster.nodes['master'].get_any_member()
+
+        json_str = self.to_json()
+        obj_filename = "_".join([self.kind, self.namespace, self.name, self.uid]) + '.json'
+        obj_path = f'/tmp/{obj_filename}'
+
+        master.put(io.StringIO(json_str), obj_path, sudo=True)
+        master.sudo(f'kubectl apply -f {obj_path} && sudo rm -f {obj_path}')