Version 1.5.5

Security

• [PYTHON][DEPENDENCIES] bump numpy version to 1.22.0 (was 1.16.4) for python example filter

Added

• [CORE] -n parameter to keep filter from daemonizing
• [CORE] -o parameter to direct logs to a different file
• [TESTS] test to check the -o parameter
• [SESSION] validate presence of a non-empty token value
• [TESTS][SESSION] 2 new tests to validate that filter now allows tokens with different lengths

Changed

• [CORE] errors in parameters parsing are now directed to stderr
• [TESTS][SESSION] Update input_too_much_parameters test with updated log line check
• [TESTS][SESSION] Replace input_invalid_token_length with new input_empty_token_invalid test

Fixed

• [MANAGER] wait long enough for a filter to start before declaring it dead

Removed

• [SESSION] Do not validate token length anymore
• [SESSION] [CODE] remove openssl dependency

Version 1.5.4

Fixed

• [CORE][LOGGER] properly rotate log file handle, even when new empty file is created between rotation and SIGHUP by the rotation service (such as logrotate on *BSD with the createoption)

Added

• [CORE][TESTS] new test to check correct log rotation when new empty file is created between rotation and SIGHUP

Version 1.5.3

Changelog

Core

• [FIX] REDIS_MANAGER:: misleading log line during pings
• [FIX] ALERT_MANAGER:: don't segfault on log rotation when no alert file was configured in filter's configuration

Manager

• LOGGING:: Use WatchedFileHandler to automatically reopen logging file after system logrotates

Filter

DGA

• [BREAKING] Update filter to use Tensorflow lite models, older Tensorflow build does not work anymore
• [BREAKING] Update compilation to include TFlite (adapted for FreBSD by disabling XNNPACK)

Functional tests

• CORE:: Add tests to ensure proper logging, even after logrotation
• CORE:: Add tests to ensure filters doesn't segfault after logrotation when filter is not configured with an alert log file
• MANAGER:: Add tests to ensure proper logging, even after logrotation

Version 1.5.2

Changelog

Filters

Session

Added

• [SESSION] Can (re)set an expiration on the queried token key
• [SESSION] tests
• [DOCS] Complete Session documentation with expiration input parameter

Security

• [SESSION] Do not validate session on Redis Lookup error

Changed

• [SESSION] refresh key and token expiration on Redis when necessary

Version 1.5

Changelog

Core

Filters

BUFR

• Add 'float' type to valid input formats
• [BREAKING] Buffer checks keys in Redis, won't start if key is of wrong type and will delete the key to avoid using stale/improper data
• [NEW] Add 'sum' buffer
  • Increments a redis key to calculate sum over a time span
  • Can take positive and negative values, represented as integers or floats (handled as floats internally)
  • 'required_log_lines' represent a limit under which not to use the absolute rounded value of the sum
  • Value is never reinserted in case of error -> won't stack sums in different time intervals
  • [FIX] Removing the modification of the wrong statistic value.

Manager

• [FIX] Avoid Manager crashing when updating a configuration with syntax errors
  • Added associated tests
• Add RFC3339 'timestamp' field to stats generated

CMakeLists

• Add VAST file inclusion (proprietary)
• Add VAML file inclusion (proprietary)
• Add option to activate coverage flags

Functional tests

• Add mock test file
• Change tests to include new 'float' input type in entries
• Update anomaly test to adapt to new algorithm
• Add tests for new 'sum' filter
  • test with one value sent
  • test with several (negative, positive, integer) values to test correct summing
  • test with sum under 'required_log_lines'
  • test with existing correct type key overrode at startup
  • test with existing incorrect type key, preventing filter to start

Test framework

• [FEATURE] Add capability to check lines are present in logs
• [FEATURE] Add an ApproxDict class to check float equality with delta in dictionaries
• [FEATURE] Add a generic filter function to validate fields in generated filters' alerts
• Reuse MOCK_PATH into TEST_FILES_DIR to specify runtime temp dir
• Improve Valgrind print returns
• Remove useless sleep after stop
• Add mock test file for VAML/VAST

TEST filter

• Allow ftest to get lists AND simple values
• Simple values are interpreted as string, but can also be numbers

Dockerfile

• Build boost:date_time (required for proprietary VAML)

Version 1.4

Changelog

Core

• added utils to generate UUIDs
• added utils to split strings

Filters

• harmonize date formats in filters' logs to be UTC RFC3339 dateformat

FBuffer (new filter)

• introduction of new FBuffer filter, here to generalize filters with a timed analysis via threads (thanks @GregoireGonzalez for the great work!)

FHostlookup

• Fix empty feed_name for some DB types

FTanomaly

• BREAKING tanomaly now counts properly unique destination ips for a source ip
• BREAKING tanomaly now handles input parsing a bit differently:
  • enforce 4 fields
  • enforce all strings
  • enforce no ";" in fields
  • enforce 4th field to be either "1", "6" or "17"
  • remove old regex validation method
• BREAKING tanomaly tests have changed:
  • Fix wrong stop condition in 'alert_published_test'
  • change 'invalid_ip_ignored_test' to 'invalid_field_ignored_test' -> now check for invalid fields (no ";" for every field, and proto in [1", "6", "17"])
  • decrease alert tests triggers to 10 seconds by using "detection_frequency" parameter
  • update test data:
    • ensure many-hosts detection
    • ensure many-ports detection
    • ensure icmp port is not taken into account
    • uses private IPs
    • use IPv4 and IPv6
    • more test data

Manager

• harmonize date formats in manager logs to be UTC RFC3339 dateformat

Github Workflows

• update workflows to use mlpack version 3.4.0 in docker builds

Version 1.3.1

Changelog

Filters

Fcontent_inspection

• add compatibility with yara lib API v4, necessary for current HardenedBSD build
• fix mistake in dependency insert

Fyara

• add compatibility with yara lib API v4, necessary for current HardenedBSD build

Version 1.3

Changelog

CAUTION: FLOGS IS NOW OBSOLETE

Filters

Core

• BREAKING: The existing alert format is replaced by a more uniform one. Check it !
• BREAKING: Now the option are handled by getopt. Also they are more conform : ./darwin [-h] [-l [DEBUG|INFO|WARNING|ERROR|CRITICAL]] filter_name socket_path config_file monitoring_socket_path pid_file output next_filter_socket_path nb_thread cache_size threshold

Toolkit

• add Yara engine utilities
• add encoders utilities to encode/decode hex/base64
• FIX: generate correct ISO 8601 timestamps in UTC for alerting

YARA (new filter)

• possibility to scan arbitrary data
• possibility to read data as-is, or decode hex or base64-encoded data
• possibility to give a list of files to include during yara rules compilation
• possibility to run yara engine in "fastmode" and to define a "timeout" (see documentation)

HOSTLOOKUP

• Added the possibility to use rsyslog lookup tables as a hostlookup database. Please follow this documentation

CMAKE

• added the -o2 for non-debug compilation.
• update CmakeLists to compile fyara by default

Manager

Now we can set a prefix and a suffix for darwin path, e.g for the log files, if your prefix is 'tmp' and your suffix 'darwin' they will be found in /tmp/logs/darwin ([prefix]/logs[suffix])

They can be set from the manager options :
usage: manager.py [-h] [-l {DEBUG,INFO,WARNING,ERROR,CRITICAL}]
[-p PREFIX_DIRECTORIES]
[-s SUFFIX_DIRECTORIES | --no-suffix-directories]
config_file

By default the prefix is '/var' and the suffix '/darwin'. Like said in the usage, you can't remove the default prefix, but you can remove the default suffix with the '--no-suffix-directories' options.

Update

• when issuing a "update_filters" command with an empty "filters" or none at all, the manager will make a diff of current and newly configured filters in its conf, and start/stop filters according to this diff.
• filters updated in conf won't be taken into account, only filters deleted/added will be taken into account

Fixes

• send a response to client when command has a JSON parsing error
• remove unused restart_all() method
• remove lock on start_all()/stop_all() functions, preventing deadlock
  • those functions are called before/after other threads are active, so locking is useless
• avoid manager raising when starting a wrong filter during update

Docker

• update Armadillo to version 9.900.x
• add yara as a compilation step (current ubuntu 18.04 version is too old for features)

Reconciler

Features:

• can recover alerts from redis, via channel, list, or both (channel+list means triggering pop of alerts on list as soon as a message is received on channel)
• can recover context from redis
• can put "reconciled" alerts in redis (via list, channel, or both) and in a log file
• all redis instances (alert_source, context_source, alert_destination) are configured independently
• there can be multiple redis instances to search for context
• there can be multiple redis and file instances to write reconciled logs
• user can set retry amount and delay when searching for context
• reconciler can work as an imported module, or as an independent executable
• by default, context is added to the alert with a 'context' key
  • this behaviour can be modified by setting a custom update function
• both file and redis managers have functions to test configuration before launching monitoring
• behaviour as an executable uses jsonschema to validate configuration file
• executable can write both to stdout and to a logfile
  • custom logger can be provided

Version 1.2.7

Changelog

!!! CAUTION: LOTS OF BREAKING CHANGES IN THIS RELEASE !!!

Core

• FIX unsigned integer parsing in parameters

Session

• FIX In case of parsing error of the body with a client waiting for response would not respond and go back to listening. Now sends a JSON with an error message and an error code.

Rework of RedisManager

• remove janitor thread -> all connection checks are done by the threads and are time based
• simplify connection management -> no master AND replica connection
• add ability to search and connect to new valid connection if query fails during call
• add rate limiting when doing researches -> no flood of existing Redis replicas
• keep configured and found Redis instances for reconnections/discovery

ThreadManager

• add possibility to change default wake-up interval

Toolkit

Files

• Added filename formating utils

CMake

Boost:

• fully use the cmake module to find required Boost components (core and filters)
• prefer static libs over dynamic ones

Hiredis:

• add a module to find hiredis libs
• prefer static libs over dynamic ones

Mlpack:

• add a module to find mlpack libs and sources
• automatically try to add openmp to compilation if mlpack was compiled with it
• prefer static libs over dynamic ones

Armadillo:

• include modules from Cmake and Armadillo projects to find Armadillo and dependent libs

Faup:

• add a module to find faup libs and sources
• prefer static libs over dynamic ones

Yara/Content_inspection:

• add a module to find yara libs and sources
• silence libcrypto not found on some architectures
• add OpenSSL's crypto lib in dependencies

Tensorflow:

• add a module to find Floopcz's cmake target or custom installation

Misc:

• improve threading library detection

Tests

• FIX single equal sign on tests requirements for redis module
• minor refactor of redis and filter classes -> remove Redis from default Filter class
• BREAKING CHANGE change and complete Redis tests:
  • simple_master_server = test simple master connection
  • master_replica = test simple connection through initial replica
  • master_replica_master_temp_fail = test behaviour during temporary disconnection of master when connected through initial replica
  • master_replica_transfer_no/with_healthcheck = test connection transfer from one server to the other, after master change, without and with healthcheck involved
  • master_replica_failover_no/with_healthcheck = test connection failover from one (failing) server to a new master, without and with healthcheck involved
  • multi_thread_master = test proper behaviour of redis manager with multithreading (one connection per thread)
  • master_replica_discovery_rate_limiting = test discovery rate limiting with multithreading
• Updated Hostlookup tests to cover more failure behavior and new DB format

Manager

• Fix requirements (#182)

Filters

Global

• use new RedisManager to handle reconnection automatically if query fails
• FIX unescaped double quotes in strings inside of the logs formatted as JSON

Tanomaly

• BREAKING CHANGE only start detection thread when filter is connected to a Redis master through an unix socket (filter with a local Redis master will start detection, otherwise will simply add entries to Redis)
• BREAKING CHANGE remove ability to start/stop detection thread through body
• FIX body validation regex
• new parameter 'detection_frequency' -> interval between 2 detection triggers
• new parameter 'start_detection_thread' to manually start it
• don't stop detection thread when unable to query Redis data

Hostlookup

• Added compatibility with a new json db format
  • Added db_type optional field to configuration :
    • text for basic text format
    • json for the new JSON format
    • Default is text
  • Feed name is the file name without extention in text mode
• BREAKING Updated the alert raised to include the feed name and changed the "host" field for "entry" to transition to a more generic filter.

Content Inspection

• FIX example conf

Test filter

• add redis capabilities
  • new parameter 'redis_list_name'
  • new parameter 'redis_channel_name'
  • possibility to trigger a Redis LPUSH with message trigger 'trigger_redis_list'
  • possibility to trigger a Redis channel publish with message trigger 'trigger_redis_channel'

Github

• FIX Moved PR template to .github/ dir, to be used automatically when creating a PR
• add workflow to build docker dev image
  • able to compile Darwin
  • able to launch test (excluding proprietary filters such as DGA)
• add workflow to build release image(s)
  • able to run all open-source filters

Docker

• add dockerfile to compile dependencies and create "development" and "release" images

HOTFIX 1.2.6

HOTFIX 1.2.6

BREAKING

• Update CMake to match new dependency (fanomaly & ftanomaly)
  • lapack -> lapacke
  • blas -> openblas