Malcolm v3.1.0

• Network analyzers

  • Added support for EtherCAT (ICS protocol)
  • Fixed and improved Spicy-based LDAP analyzer
  • Detect VPN protocols IPsec, OpenVPN and WireGuard

• New or improved

  • Updated many Kibana dashboards and added dashbaords for newly-supported network protocols
  • Improved output of debug logs from docker images
  • Many minor improvements to underlying system for ISO installations
  • Massively cut build time for Hedgehog ISO and Zeek Docker container by using .deb packages from released versions rather than building from source
  • During build, install all Zeek plugins via zkg

• Version updates

  • Zeek v4.0.1
  • Spicy v1.0.0
  • Open Distro For Elasticsearch v1.13.2
  • Yara v4.1.0
  • Capa v1.6.3
  • switch from centos:7 to amazonlinux:2 for base Docker image to build Kibana plugins
  • stunnel v5.59
  • NGINX v1.20.0
  • LLVM/clang toolchain v11
  • Flask-Cors v3.0.9 for Hedgehog kiosk interface (dependabot-flagged security alert)
  • latest updates of various Zeek plugins, system and python packages, etc.
  • all Python scripts updated to Python 3

• Bugs fixed

  • When LDAP authentication is used instead of BASIC authentication, show a landing page rather than a server error when attempting to browse to the local authentication management interface
  • Fixed a regression bug where Malcolm fails to start correctly if not using UID/GID 1000:1000
  • Don't automatically expose elasticsearch (and logstash) ports unless explicitly configured to do so
  • freshclam should update the clamav database during docker image build

Malcolm v3.0.1

Malcolm v3.0.1 contains some important version updates for several of its components and fixes a few bugs. Please continue reading for more details.

List of changes in Malcolm v3.0.1:

v3.0.0...v3.0.1

• Version bumps
  • Open Distro for Elastic (v1.13.0), which adds the following functionality over the previous release
    • Reporting
    • Historical data anomaly detection
  • ODFE v1.13.0 is based on the Elastic components 7.10.2 (elasticsearch, kibana, logstash, beats)
  • Zeek 3.0.13
  • NGINX 1.19.7
  • Alpine Linux 3.13 Docker base layer
  • docker-compose 1.28.5 in Malcolm installable ISO version
• Restored the sankey visualization which was temporarily removed in Malcolm v3.0.0 (although there are still a few minor cosmetic issues with it)
• Removed port 8443 for upload (now just use /upload over the regular HTTPS port)
• Fixed issue with ODFE email alerts not being able to use self-signed SMTP certificates by importing CA certs in nginx/ca-trust into the JDK trust store for Elasticsearch and Logstash (see #37)
• Don't expose the Elasticsearch 9200 by default, it must now be explicitly be enabled during install.py -c (see #38)
• For ISO-installed versions of Malcolm and Hedgehog Linux, populate /etc/os-release with information about the build/release version
• Populate user-agent for a few clients (Arkime's moloch-capture, some hedgehog test connection processes) so they're not just sent as blank when communicating with Malcolm
• Added Arkime link to Kibana dashboards' navigation pane
• Fix some issues in control script with older python3 versions (3.6.x) with contextlib.nullcontext not being available
• Fix suggestion for yum-based distributions to install python 3 requests via pip

Malcolm and Hedgehog Linux may be obtained by pulling or building the Docker images and/or building the ISO installer images as described in the documentation. Unofficial ISO installer images for Malcolm and Hedgehog Linux are not hosted on GitHub, but may be downloaded from https://malcolm.fyi/download/.

Malcolm v3.0.0

Malcolm v3.0.0 is a major release with some big replacements in the project's underpinnings, including a few backwards compatibility-breaking changes. Please continue reading for more details.

List of changes in Malcolm v3.0.0:

v2.6.1...v3.0.0

• Change base for Elasticsearch and Kibana Docker images (version 7.6.2) from Elastic.co to Open Distro for Elastic (based on Elastic 7.10.0); see #15. This is a major change which breaks backwards compatibility for several features (listed below). If you are using these features, you will need to back up the data and/or configuration associated with them and migrate them manually to the new tools. No automatic migration or upgrade of these features is performed. It's recommended that you re-run install.py --configure (see System configuration and tuning) prior to running Malcolm v3.0.0.
  • Kibana comments replaced with Notebooks
  • Kibana elastalert plugin replaced with Alerting plugin
  • Elasticsearch curator replaced with Index Management plugin
  • The third-party Sankey visualization plugin has been temporarily removed due to compatibility issues, although it is planned to be reintegrated in a Malcolm point release in the near future (see uniberg/kbn_sankey_vis#15)
  • The third-party Kibana drill-down plugin providing Kibana-to-Moloch pivoting has been temporarily removed due to compatibility issues, although it is planned to be reintegrated in a Malcolm point release in the near future (see goodlabs-studio/kibana-plugin-drilldownmenu#5)
  • In addition to those replacements, the Real Time Anomaly Detection feature is now available:
    • Real Time Anomaly Detection in Open Distro for Elasticsearch blog announcement
    • Anomaly Detection documentation and source code for Elasticsearch and Kibana components
    • Random Cut Forests writeup
  • If you are not up-to-date on the recent developments in Elasticsearch's licensing, here are a few of the official statements from the various parties involved:
    • Elastic.co's original announcement, clarification, Elastic License v2 announcement, "Why we had to change" post and FAQ on 2021 License Change
    • Open Distro for Elasticsearch initial response post, Amazon AWS Open Source Blog post and fork updates post
• Malcolm startup time (especially the Logstash container) has been reduced drastically
• Improvements to Malcolm's prebuilt Kibana dashboards
• Improvements to build scripts
• Minor tweaks and bugfixes for ISO-installed environments for Malcolm and Hedgehog Linux
• Minor other bug fixes and performance improvements
• Version bump
  • Yara v4.0.5

Malcolm and Hedgehog Linux may be obtained by pulling or building the Docker images and/or building the ISO installer images as described in the documentation. Unofficial ISO installer images for Malcolm and Hedgehog Linux are not hosted on GitHub, but may be downloaded from https://malcolm.fyi/download/.

Malcolm v2.6.1

Malcolm v2.6.1 contains the following changes:

v2.6.0...v2.6.1

• Added TFTP Zeek parser and corresponding Logstash parsing, Arkime WISE support and Kibana dashboards
• Provide browser-based access to zeek/extracted-files directory (#34)
• Fix LDAP analyzer not parsing all events (#35)
• Provide more fine-tuned controls for Zeek's node.cfg in Hedgehog sensor (#36, cisagov/pull/158)
• set zeek.uid to conn_uids for files.log entries (#33)
• Modify Zeek build chain to use default GCC compilers instead of LLVM/clang,which reduces build dependencies
• Use Firefox instead of Chromium for browser in ISO-installed versions of Malcolm and in Hedgehog Linux
• Updated copyright notices in text from "2020" to "2021" (which is the bulk of the changed files in this commit)
• Version bumps
  • Yara to 4.0.4

Malcolm v2.6.0

Malcolm v2.6.0 contains the following changes:

v2.5.0...v2.6.0

• Replace some of the Amazon ICS parsers for Zeek with parsers developed at the Idaho National Lab supporting DHS CISA

  • BACnet
  • Ethernet/IP

• Incorporated updates to some default Zeek ICS protocols

  • DNP3
  • Modbus

• Added new parsers for BSAP ICS protocol

  • BSAP IP
  • BSAP Serial

Component version bumps:

• Supercronic 0.1.12 (used in some Malcolm Docker images)
• alpine:3.12 (base layer of some Malcolm Docker images)
• nginx 1.19.6 (the web server handling encryption, authentication and proxying for Malcolm's Docker containers)
• CMake 3.19.3 (for building some Malcolm source code)
• netsniff-ng 0.6.8 (for packet capture)

Malcolm and Hedgehog Linux may be obtained by pulling or building the Docker images and/or building the ISO installer images as described in the documentation. Unofficial ISO installer images for Malcolm and Hedgehog Linux are not hosted on GitHub, but may be downloaded from https://malcolm.fyi/download/.

Malcolm v2.5.0

Malcolm v2.5.0 contains the following changes:

v2.4.2...v2.5.0

• Updated packaged Yara rules (from github.com/Neo23x0/signature-base, originally github.com/fireeye/sunburst_countermeasures) for Yara scanning of carved files for Yara scanning of carved files to detect artifacts from the SolarWinds SUNBURST attack
• Version bumps:
  • Zeek 3.0.12
  • Bison, CMake and LLVM/Clang tools for building Zeek for Docker image and Hedgehog OS ISO

Malcolm and Hedgehog Linux may be obtained by pulling or building the Docker images and/or building the ISO installer images as described in the documentation. Unofficial ISO installer images for Malcolm and Hedgehog Linux are not hosted on GitHub, but may be downloaded from https://malcolm.fyi/download/.

Malcolm v2.4.2

Malcolm v2.4.2 contains the following changes:

v2.4.1...v2.4.2

• Added code to allow periodic updates of Yara and Capa rules in addition to ClamAV rules for file scanners
• Bump to Arkime (Moloch up until recently) 2.7.1 and all possible related user-facing code/documentation changed
• Bump kernel to 5.9.0 for ISO installer
• minor bug fixes and documentation tweaks

Malcolm and Hedgehog Linux may be obtained by pulling or building the Docker images and/or building the ISO installer images as described in the documentation. Unofficial ISO installer images for Malcolm and Hedgehog Linux are not hosted on GitHub, but may be downloaded from https://malcolm.fyi/download/.

Malcolm v2.4.1

Malcolm v2.4.1 contains the following changes:

v2.4.0...v2.4.1

• Zeek

  • added plugin to detect "bad neighbor" (CVE-2020-16898)

• Version bumps

  • supercronic (for Docker images) 0.1.11
  • nginx 1.19.3
  • bison (for Zeek compile) 3.7.2
  • cmake (for Zeek compile) 3.18.4
  • Zeek 3.0.11
  • Moloch 2.4.1
  • Linux Kernel (for ISOs) 5.8.0

Malcolm and Hedgehog Linux may be obtained by pulling or building the Docker images and/or building the ISO installer images as described in the documentation. Unofficial ISO installer images for Malcolm and Hedgehog Linux are not hosted on GitHub, but may be downloaded from https://malcolm.fyi/download/.

Malcolm v2.4.0.1

Malcolm v2.4.0.1 is a repack of the Malcolm v2.4.0 release with one minor fix for the ISO installers for Malcolm and Hedgehog Linux to fix #27. The rest of the code is identical. If you are deploying Malcolm with Docker rather than the ISO-installed version, you can ignore this release.

Malcolm v2.4.0

Malcolm v2.4.0 contains the following new features, improvements and bug fixes:

• Extracted file scanning
  • added Capa as an optional extracted file scanner
  • improvements to the way file scanners work when more than one are enabled
• Version updates
  • updated Moloch to 2.4.1
  • updated Zeek to 3.0.10
  • updated Linux Kernel for ISO installers to 5.7
• Zeek plugins
  • added Corelight's Zerologon plugin to detect CVE-2020-1472
• Tweaks and bug fixes
  • Don't allow docker to mess with firewall rules in Malcolm ISO
  • Fix #26, ISO installers result in blank screen when booting with BIOS
  • Fix #24, install.py won't prompt to change ownership of extracted directory correctly if run as root
  • Leave some development packages in place in Hedgehog ISO so that Spicy plugins can be compiled

v2.3.0...v2.4.0

Malcolm and Hedgehog Linux may be obtained by pulling or building the Docker images and/or building the ISO installer images as described in the documentation. Unofficial ISO installer images for Malcolm and Hedgehog Linux are not hosted on GitHub, but may be downloaded from https://malcolm.fyi/download/.