Merge pull request #124 from th2-net/release-3.9.0

[TH2-2845] Release 3.9.0
th2-net · Dec 13, 2021 · f52498c · f52498c
2 parents e9aa09b + f4da2af
commit f52498c
Show file tree

Hide file tree

Showing 40 changed files with 3,199 additions and 376 deletions.
diff --git a/README.md b/README.md
@@ -1,12 +1,12 @@
-# th2 check1 (3.8.0)
+# th2 check1 (3.9.0)
 
 ## Overview
 
 The component is responsible for verifying decoded messages.
 
 Communication with the script takes place via grpc, messages are received via rabbit mq.
 
-The component subscribes to queues specified in the configuration and accumulates messages from them in a FIFO buffer. 
+The component subscribes to the queues specified in the configuration and accumulates messages from them in a FIFO buffer. 
 
 The buffer size is configurable, and it is set to 1000 by default.
 
@@ -17,7 +17,53 @@ When the component starts, the grpc server also starts and then the component wa
 Available requests are described in [this repository](https://gitlab.exactpro.com/vivarium/th2/th2-core-open-source/th2-grpc-check1)
 
 - CheckSequenceRuleRequest - prefilters the messages and verify all of them by filter. Order checking configured from request.
+  Depending on the request and check1 configuration **SilenceCheckRule** can be added after the CheckSequenceRule.
+  It verifies that there were not any messages matching the pre-filter in the original request.
+  It awaits for realtime timeout that is equal to clean-up timeout.
+  Reports about unexpected messages only after the timeout is exceeded. Reports nothing if any task is added to the chain.
 - CheckRuleRequest - get message filter from request and check it with messages in the cache or await specified time in case of empty cache or message absence.
+- NoMessageCheckRequest - prefilters messages and verifies that no other messages have been received.
+
+## Request parameters
+
+### Common
+
+#### Required
+
+* **parent_event_id** - all events generated by the rule will be attached to that event
+* **connectivity_id** (the `session_alias` inside `connectivity_id` must not be empty)
+
+#### Optional
+
+* **direction** - the direction of messages to be checked by rule. By default, it is _FIRST_
+* **chain_id** - the id to connect rules (rule starts checking after the previous one in the chain). Considers **connectivity_id**
+* **description** - the description that will be added to the root event produced by the rule
+* **timeout** - defines the allowed timeout for messages matching by real time. If not set the default value from check1 settings will be taken
+* **message_timeout** - defines the allowed timeout for messages matching by the time they were received.
+* **checkpoint** (must be set if `message_timeout` is used and no valid `chain_id` has been provided)
+
+### CheckRuleRequest
+
+#### Required
+
+* **root_filter** or **filter** (please note, that the `filter` parameter is deprecated and will be removed in the future releases)
+
+### CheckSequenceRuleRequest
+
+#### Required
+
+* **root_message_filters** or **message_filters** with at least one filter
+  (please note, that the `message_filters` parameter is deprecated and will be removed in the future releases)
+
+#### Optional
+* **pre_filter** - pre-filtering for messages. Only messages that passed the filter will be checked by the main filters.
+* **check_order** - enables order validation in message's collections
+* **silence_check** - enables auto-check for messages that match the `pre_filter` after the rule has finished
+
+### NoMessageCheckRequest
+
+#### Optional
+* **pre_filter** pre-filtering for messages that should not be received.
 
 ## Quick start
 General view of the component will look like this:
@@ -35,6 +81,10 @@ spec:
     cleanup-older-than: '60'
     cleanup-time-unit: 'SECONDS'
     max-event-batch-content-size: '1048576'
+    rule-execution-timeout: '5000'
+    auto-silence-check-after-sequence-rule: false
+    time-precision: 'PT0.000000001S'
+    decimal-precision: '0.00001'
   type: th2-check1
   pins:
     - name: server
@@ -67,7 +117,12 @@ This block describes the configuration for check1.
   "message-cache-size": 1000,
   "cleanup-older-than": 60,
   "cleanup-time-unit": "SECONDS",
-  "max-event-batch-content-size": "1048576"
+  "max-event-batch-content-size": "1048576",
+  "rule-execution-timeout": 5000,
+  "auto-silence-check-after-sequence-rule": false,
+  "time-precision": "PT0.000000001S",
+  "decimal-precision": 0.00001,
+  "check-null-value-as-empty": false
 }
 ```
 
@@ -86,11 +141,27 @@ The time unit for _cleanup-older-than_ setting. The available values are MILLIS,
 #### max-event-batch-content-size
 The max size in bytes of summary events content in a batch defined in _max-event-batch-content-size_ setting. _The default value is set to 1048576_
 
+#### rule-execution-timeout
+The default rule execution timeout is used if no rule timeout is specified. Measured in milliseconds
+
+#### auto-silence-check-after-sequence-rule
+Defines a default behavior for creating CheckSequenceRule if `silence_check` parameter is not specified in the request. The default value is `false`
+
+#### time-precision
+The time precision is used to compare two time values. It is based on the `ISO-8601` duration format `PnDTnHnMn.nS` with days considered to be exactly 24 hours. Additional information can be found [here](https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/time/Duration.html#parse(java.lang.CharSequence))
+
+#### decimal-precision
+The decimal precision is used to compare the value of two numbers. Can be specified in number or string format. For example `0.0001`, `0.125`, `125E-3`
+
+#### check-null-value-as-empty
+`check-null-value-as-empty` is used for `EMPTY` and `NOT_EMPTY` operations to check if `NULL_VALUE` value is empty. By default, this parameter is set to `false`. For example, if the `checkNullValueAsEmpty` parameter is:
++ `true`, then `NULL_VALUE` is equal to `EMPTY`, otherwise `NULL_VALUE` is equal to `NOT_EMPTY`
+
 ## Required pins
 
 The Check1 component has two types of pin:
 * gRPC server pin: it allows other components to connect via `com.exactpro.th2.check1.grpc.Check1Service` class.
-* MQ pin: it is used for listening to parsed messages. You can link several sources with different directions and session alises to it.
+* MQ pin: it is used for listening to parsed messages. You can link several sources with different directions and session aliases to it.
 
 ```yaml
 apiVersion: th2.exactpro.com/v1
@@ -121,6 +192,31 @@ The `th2_check1_active_tasks_number` metric separate rules with label `rule_type
 
 ## Release Notes
 
+### 3.9.0
+
+#### Added:
++ Implemented NoMessageCheck rule task. Updated CheckRule and CheckSequence rule tasks
++ New configuration parameter `rule-execution-timeout` which is used if the user has not specified a timeout for the rule execution
++ Auto silence check after the CheckSequenceRule.
++ `auto-silence-check-after-sequence-rule` to setup a default behavior for CheckSequenceRule
++ New configuration parameter `time-precision` which is used if the user has not specified a time precision
++ New configuration parameter `decimal-precision` which is used if the user has not specified a number precision
++ New parameter `hint` for verification event which is used to display the reason for the failed field comparison. For example the type mismatch of the compared values
++ New configuration parameter `check-null-value-as-empty` witch us used to configure the `EMPTY` and `NOT_EMPTY` operations
+
+#### Changed:
++ Migrated `common` version from `3.26.4` to `3.31.3`
++ Migrated `grpc-check1` version from `3.4.2` to `3.5.1`
++ Migrated `sailfish-utils` version from `3.9.1` to `3.12.2`
+  + Fixed conversion of `null` values
+  + Add marker for `null` values to determine whether the field was set with `null` value or was not set at all
+  + Allow checking for exact `null` value in message
+  + Added new parameter `checkNullValueAsEmpty` in the `FilterSettings`
++ Corrected verification entry when the `null` value and string `"null"` looked the same for the expected value
++ Fixed setting of the `failUnexpected` parameter while converting a message filter
++ Migrated `sailfish-core` version to `3.2.1752`
+  + Fix incorrect matching in repeating groups with reordered messages
+
 ### 3.8.0
 
 #### Added:

diff --git a/build.gradle b/build.gradle
@@ -1,6 +1,6 @@
 plugins {
     id 'com.palantir.docker' version '0.25.0'
-    id 'org.jetbrains.kotlin.jvm' version '1.3.72'
+    id 'org.jetbrains.kotlin.jvm' version '1.5.30'
     id "io.github.gradle-nexus.publish-plugin" version "1.0.0"
     id 'signing'
     id 'java-library'
@@ -10,7 +10,7 @@ plugins {
 
 ext {
     sharedDir           = file("${project.rootDir}/shared")
-    sailfishVersion     = '3.2.1050'
+    sailfishVersion     = '3.2.1752'
 }
 
 group = 'com.exactpro.th2'
@@ -42,6 +42,7 @@ repositories {
         name 'Sonatype_releases'
         url 'https://s01.oss.sonatype.org/content/repositories/releases/'
     }
+    mavenLocal()
 
     configurations.all {
         resolutionStrategy.cacheChangingModulesFor 0, 'seconds'
@@ -165,21 +166,20 @@ signing {
 
 dependencies {
     api platform('com.exactpro.th2:bom:3.0.0')
-    implementation 'com.exactpro.th2:grpc-check1:3.4.2'
-    implementation 'com.exactpro.th2:common:3.26.4'
-    implementation 'com.exactpro.th2:sailfish-utils:3.9.1'
+    implementation 'com.exactpro.th2:grpc-check1:3.5.1'
+    implementation 'com.exactpro.th2:common:3.31.3'
+    implementation 'com.exactpro.th2:sailfish-utils:3.12.2'
     implementation "org.slf4j:slf4j-log4j12"
     implementation "org.slf4j:slf4j-api"
 
     implementation "com.exactpro.sf:sailfish-core:${sailfishVersion}"
 
-    implementation "com.datastax.cassandra:cassandra-driver-core" //FIXME: Migrate to another library for UUID
-
     implementation "io.reactivex.rxjava2:rxjava:2.2.19" // https://github.com/salesforce/reactive-grpc/issues/202
 
     implementation('io.prometheus:simpleclient') {
         because('metrics from messages and rules')
     }
+    implementation 'org.jetbrains.kotlin:kotlin-reflect'
 
     testImplementation 'org.junit.jupiter:junit-jupiter:5.6.2'
     testImplementation 'org.jetbrains.kotlin:kotlin-test-junit'
@@ -206,4 +206,4 @@ dockerPrepare {
 
 docker {
     copySpec.from(tarTree("$buildDir/distributions/${applicationName}.tar"))
-}
+}
diff --git a/gradle.properties b/gradle.properties
@@ -1,4 +1,4 @@
-release_version = 3.8.0
+release_version = 3.9.0
 
 description = 'th2 check1 box'
 

diff --git a/src/main/java/com/exactpro/th2/check1/Check1Handler.java b/src/main/java/com/exactpro/th2/check1/Check1Handler.java
@@ -1,5 +1,5 @@
 /*
- * Copyright 2020-2020 Exactpro (Exactpro Systems Limited)
+ * Copyright 2020-2021 Exactpro (Exactpro Systems Limited)
  * Licensed under the Apache License, Version 2.0 (the "License");
  * you may not use this file except in compliance with the License.
  * You may obtain a copy of the License at
@@ -14,19 +14,22 @@
 
 import static com.exactpro.th2.common.grpc.RequestStatus.Status.ERROR;
 import static com.exactpro.th2.common.grpc.RequestStatus.Status.SUCCESS;
-import static com.google.protobuf.TextFormat.shortDebugString;
 
+import com.exactpro.th2.check1.utils.ProtoMessageUtilsKt;
+import com.exactpro.th2.common.message.MessageUtils;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
 import com.exactpro.th2.check1.grpc.ChainID;
+import com.exactpro.th2.check1.grpc.CheckpointResponse;
 import com.exactpro.th2.check1.grpc.Check1Grpc.Check1ImplBase;
 import com.exactpro.th2.check1.grpc.CheckRuleRequest;
 import com.exactpro.th2.check1.grpc.CheckRuleResponse;
 import com.exactpro.th2.check1.grpc.CheckSequenceRuleRequest;
 import com.exactpro.th2.check1.grpc.CheckSequenceRuleResponse;
 import com.exactpro.th2.check1.grpc.CheckpointRequest;
-import com.exactpro.th2.check1.grpc.CheckpointResponse;
+import com.exactpro.th2.check1.grpc.NoMessageCheckResponse;
+import com.exactpro.th2.check1.grpc.NoMessageCheckRequest;
 import com.exactpro.th2.common.grpc.RequestStatus;
 
 import io.grpc.stub.StreamObserver;
@@ -45,20 +48,20 @@ public void createCheckpoint(CheckpointRequest request, StreamObserver<Checkpoin
         try {
             var internalCheckpoint = collectorService.createCheckpoint(request);
             CheckpointResponse checkpointResponse = CheckpointResponse.newBuilder()
-                    .setCheckpoint(internalCheckpoint.convert())
+                    .setCheckpoint(ProtoMessageUtilsKt.convert(internalCheckpoint))
                     .build();
             if (logger.isDebugEnabled()) {
-                logger.debug("Created checkpoint internal '{}' proto '{}", internalCheckpoint, shortDebugString(checkpointResponse));
+                logger.debug("Created checkpoint internal '{}' proto '{}", internalCheckpoint, MessageUtils.toJson(checkpointResponse));
             }
             responseObserver.onNext(checkpointResponse);
-            responseObserver.onCompleted();
-        } catch (RuntimeException e) {
+        } catch (Exception e) {
             if (logger.isErrorEnabled()) {
-                logger.error("CheckRule failed. Request " + shortDebugString(request), e);
+                logger.error("CheckRule failed. Request " + MessageUtils.toJson(request), e);
             }
             responseObserver.onNext(CheckpointResponse.newBuilder()
                     .setStatus(RequestStatus.newBuilder().setStatus(ERROR).setMessage("Create checkpoint failed. See the logs.").build())
                     .build());
+        } finally {
             responseObserver.onCompleted();
         }
     }
@@ -67,7 +70,7 @@ public void createCheckpoint(CheckpointRequest request, StreamObserver<Checkpoin
     public void submitCheckRule(CheckRuleRequest request, StreamObserver<CheckRuleResponse> responseObserver) {
         try {
             if (logger.isInfoEnabled()) {
-                logger.info("Submit CheckRule request: " + shortDebugString(request));
+                logger.info("Submit CheckRule request: " + MessageUtils.toJson(request));
             }
 
             CheckRuleResponse.Builder response = CheckRuleResponse.newBuilder();
@@ -77,7 +80,7 @@ public void submitCheckRule(CheckRuleRequest request, StreamObserver<CheckRuleRe
                         .setStatus(RequestStatus.newBuilder().setStatus(SUCCESS));
             } catch (Exception e) {
                 if (logger.isErrorEnabled()) {
-                    logger.error("CheckRule failed in CollectorService. Request " + shortDebugString(request), e);
+                    logger.error("CheckRule failed in CollectorService. Request " + MessageUtils.toJson(request), e);
                 }
                 RequestStatus status = RequestStatus.newBuilder()
                         .setStatus(ERROR)
@@ -86,14 +89,14 @@ public void submitCheckRule(CheckRuleRequest request, StreamObserver<CheckRuleRe
                 response.setStatus(status);
             }
             responseObserver.onNext(response.build());
-            responseObserver.onCompleted();
-        } catch (Throwable e) {
+        } catch (Exception e) {
             if (logger.isErrorEnabled()) {
-                logger.error("CheckRule failed. Request " + shortDebugString(request), e);
+                logger.error("CheckRule failed. Request " + MessageUtils.toJson(request), e);
             }
             responseObserver.onNext(CheckRuleResponse.newBuilder()
                     .setStatus(RequestStatus.newBuilder().setStatus(ERROR).setMessage("CheckRule failed. See the logs.").build())
                     .build());
+        } finally {
             responseObserver.onCompleted();
         }
     }
@@ -104,7 +107,7 @@ public void submitCheckSequenceRule(CheckSequenceRuleRequest request, StreamObse
             CheckSequenceRuleResponse.Builder response = CheckSequenceRuleResponse.newBuilder();
             try {
                 if (logger.isInfoEnabled()) {
-                    logger.info("Submitting sequence rule for request '" + shortDebugString(request) + "' started");
+                    logger.info("Submitting sequence rule for request '" + MessageUtils.toJson(request) + "' started");
                 }
                 ChainID chainID = collectorService.verifyCheckSequenceRule(request);
                 if (logger.isInfoEnabled()) {
@@ -114,7 +117,7 @@ public void submitCheckSequenceRule(CheckSequenceRuleRequest request, StreamObse
                         .setStatus(RequestStatus.newBuilder().setStatus(SUCCESS));
             } catch (Exception e) {
                 if (logger.isErrorEnabled()) {
-                    logger.error("Submitting sequence rule for request '" + shortDebugString(request) + "' failed", e);
+                    logger.error("Submitting sequence rule for request '" + MessageUtils.toJson(request) + "' failed", e);
                 }
                 RequestStatus status = RequestStatus.newBuilder()
                         .setStatus(ERROR)
@@ -123,16 +126,51 @@ public void submitCheckSequenceRule(CheckSequenceRuleRequest request, StreamObse
                 response.setStatus(status);
             }
             responseObserver.onNext(response.build());
-            responseObserver.onCompleted();
-        } catch (RuntimeException e) {
+        } catch (Exception e) {
             if (logger.isErrorEnabled()) {
-                logger.error("Sequence rule task for request '" + shortDebugString(request) + "' isn't submitted", e);
+                logger.error("Sequence rule task for request '" + MessageUtils.toJson(request) + "' isn't submitted", e);
             }
             responseObserver.onNext(CheckSequenceRuleResponse.newBuilder()
                     .setStatus(RequestStatus.newBuilder().setStatus(ERROR)
                             .setMessage("Sequence rule rejected by internal process: " + e.getMessage())
                             .build())
                     .build());
+        } finally {
+            responseObserver.onCompleted();
+        }
+    }
+
+    @Override
+    public void submitNoMessageCheck(NoMessageCheckRequest request, StreamObserver<NoMessageCheckResponse> responseObserver) {
+        try {
+            if (logger.isInfoEnabled()) {
+                logger.info("Submitting no message check rule for request '{}' started", MessageUtils.toJson(request));
+            }
+
+            NoMessageCheckResponse.Builder response = NoMessageCheckResponse.newBuilder();
+            try {
+                ChainID chainID = collectorService.verifyNoMessageCheck(request);
+                response.setChainId(chainID)
+                        .setStatus(RequestStatus.newBuilder().setStatus(SUCCESS));
+            } catch (Exception e) {
+                if (logger.isErrorEnabled()) {
+                    logger.error("No message check rule task for request '{}' isn't submitted", MessageUtils.toJson(request), e);
+                }
+                RequestStatus status = RequestStatus.newBuilder()
+                        .setStatus(ERROR)
+                        .setMessage("No message check rule rejected by internal process: " + e.getMessage())
+                        .build();
+                response.setStatus(status);
+            }
+            responseObserver.onNext(response.build());
+        } catch (Exception e) {
+            if (logger.isErrorEnabled()) {
+                logger.error("No message check rule failed. Request " + MessageUtils.toJson(request), e);
+            }
+            responseObserver.onNext(NoMessageCheckResponse.newBuilder()
+                    .setStatus(RequestStatus.newBuilder().setStatus(ERROR).setMessage("No message check rule failed. See the logs.").build())
+                    .build());
+        } finally {
             responseObserver.onCompleted();
         }
     }