Malcolm v24.10.0

Malcolm v24.10.0 contains fixes for a few regression bugs, minor improvements, and a few component updates.

v24.09.0...v24.10.0

• Features and enhancements
  • Enable Zeek's parsing of HTTP server and client header names as zeek.http.client_header_names and zeek.http.server_header_names
  • Bumped maximum field limit in OpenSearch templates from 5000 to 6000
  • Some documentation improvements
  • Build improvement: fall back to alternative Zeek .deb download URL (#585)
  • Build improvement: limit threads for spicy build processes during Zeek package installation (#571)
• Component version updates
  • Beats to v8.15.2
  • capa to v7.4.0
  • elasticsearch-dsl Python library to v8.15.4
  • Fluent Bit to v3.1.9
  • Logstash to v8.15.2
  • OpenSearch and OpenSearch Dashboards to v2.17.1
  • osd_transform_vis Dashboards visualization to v2.17.1
  • Spicy to v1.11.3
  • watchdog Python library to v5.0.3
  • Zeek to v7.0.3
• Bug fixes
  • Fix broken dashboards regression from v24.09.0 (#588)
  • Fix Zeek-extracted files not getting saved to correct location for live Zeek capture (#590)
  • Fix for building Hedgehog Linux for Raspberry Pi 4 on an M2 MacBook

Official ISO installer images for Malcolm and Hedgehog Linux can be downloaded from Malcolm's releases page on GitHub. Due to limits on individual files in GitHub releases, these ISO files have been split into 2GB chunks and can be reassembled with scripts provided for both Bash (release_cleaver.sh) and PowerShell (release_cleaver.ps1). See Downloading Malcolm - Installer ISOs for instructions.

Malcolm v24.09.0

Malcolm v24.09.0 contains new features and enhancements, component version updates, and bug fixes.

v24.08.0...v24.09.0

• Features and enhancements
  • When building Docker images and the Hedgehog Linux ISO, allow specifying alternate download URL for MaxMind GeoIP database files (#565)
  • Allow total index size-based pruning for opensearch-remote and elasticsearch-remote database modes (#446)
  • Allow splitting out indexes by other field values (#450)
  • Allow users to use the Arkime Lua plugin without having to create new bind volume mounts manually (#533)
  • Automatically create empty document on startup to avoid "no data" message spamming by Dashboards (#527 and #567)
  • Improvements to documentation and install.py for Linux performance tweaks (#495)
  • Include netbox-topology-views plugin by default (#553)
  • Integrate HART-IP parser (#561)
  • Add option to go backwards in Malcolm's dialog-based install.py installation and configuration script (#487)
  • Added Podman support (#407)
  • Update EtherNet/IP and CIP to account for new packet correlation ID (#558)
  • Update Network Traffic Analysis with Malcolm slides
• Component version updates
  • watchdog Python package to v5.0.x (#550)
  • supercronic to v0.2.32
  • osd_transform_vis to v2.16.0
  • Fluent Bit to v3.1.8
  • OpenSearch and OpenSearch Dashboards to v2.17.0
  • YARA to v4.5.2
  • Zeek to v7.0.1
  • Spicy to v1.11.1
  • elasticsearch-dsl Python package to v8.15.3
  • elasticsearch Python package to v8.15.1
  • Beats to v8.15.1
  • Logstath to 8.15.1
  • flask-cors Python package to v5.0.0 to address CVE-2024-6221
• Bug fixes
  • Filtering on hunt ID in Arkime not working (#554)
  • Hedgehog with OOB/VPN connection sets ARKIME_NODE_HOST incorrectly (#560 and #559, thanks @divinehawk)
  • Offline suricata Docker container does not initialize suricata.yml config file (#564)
• Configuration changes (in environment variables in ./config/) for Malcolm and in control_vars.conf for Hedgehog Linux
  • Malcolm
    • The MALCOLM_NETWORK_INDEX_SUFFIX and MALCOLM_OTHER_INDEX_SUFFIX variables in ./config/opensearch.env now also support expanding dot-delimited field names in ｛｛ ｝｝ (e.g., ｛｛event.provider｝｝％{％y％m％d}).
    • MALCOLM_CONTAINER_RUNTIME has been added to ./config/process.env to indicate docker, podman, or kubernetes. This value only currently used in the install, configuration, and control scripts, not inside the containers themselves.
    • ZEEK_DISABLE_ICS_HART_IP has been added to ./config/zeek.env and can be set to true to disable the new HART-IP protocol parser.
  • Hedgehog Linux
    • ZEEK_DISABLE_ICS_HART_IP has been added to control_vars.conf and can be set to true to disable the new HART-IP protocol parser.

Official ISO installer images for Malcolm and Hedgehog Linux can be downloaded from Malcolm's releases page on GitHub. Due to limits on individual files in GitHub releases, these ISO files have been split into 2GB chunks and can be reassembled with scripts provided for both Bash (release_cleaver.sh) and PowerShell (release_cleaver.ps1). See Downloading Malcolm - Installer ISOs for instructions.

Malcolm v24.08.0

Malcolm v24.08.0 contains minor improvements, some component version updates, and bug fixes.

v24.07.0...v24.08.0

• Features and enhancements
  • in ISO installer, prompt to format other drives for artifact storage rather than just doing it automatically (#529)
  • allow users to more easily add NetBox plugins (#530)
  • run netbox-initializers plugin on startup even if we're doing a netbox database backup preload (#531)
  • during auth_setup "all" operation, do required operations without prompting if the files don't already exist (#536)
  • some containers need resource request specified for Kubernetes (#539)
  • add "public" pseudo-segments for public IP addresses (#542)
  • reworked Windows Event dashboard
  • some documentation updates
  • added netbox tag to any logs that are passed into the netbox_enrich.rb script in the Logstash enrichment pipeline
• Component version updates
  • elasticsearch and elasticsearch-dsl Python libraries to v8.15.0
  • Arkime to v5.4.0
  • Beats to v8.15.0
  • capa to v7.2.0
  • evtx to v0.8.3
  • Fluent Bit to v3.1.6
    • fluent-bit-setup.ps1 helper script needs updated URLs (#541)
  • Logstash to v8.15.0
  • NetBox to v4.0.9
  • OpenSearch and OpenSearch Dashboards to v2.16.0
  • yq to v4.44.3
  • Zeek to v7.0.0 (#535)
• Bug fixes
  • dashboards-helper container's use of curl fails internal container name resolution when host has invalid DNS settings, prevents Malcolm initialization (#499)
  • Netbox service templates not populating (#522)
  • kubernetes manifest for netbox refers to netbox-netmap-json configmap which no longer exists (#540)
  • don't try to expose the OpenSearch port 9200 in docker-compose.yml when the database mode is not opensearch-local
  • improved the liveness check for the offline Zeek container so that it returns "healthy" if the intel thread feeds are still pulling before the monitoring processes start up
  • missing cracklib-runtime package prevents ISO service account password from being updated by non-root user (#548)

Official ISO installer images for Malcolm and Hedgehog Linux can be downloaded from Malcolm's releases page on GitHub. Due to limits on individual files in GitHub releases, these ISO files have been split into 2GB chunks and can be reassembled with scripts provided for both Bash (release_cleaver.sh) and PowerShell (release_cleaver.ps1). See Downloading Malcolm - Installer ISOs for instructions.

Malcolm v24.07.0

Malcolm v24.07.0 contains minor improvements, some component version updates, and a few bug fixes.

v24.06.0...v24.07.0

• Features and enhancements
  • integrated the ICSNPP GE SRTP network analyzer (#516)
  • Changed the way docker compose does bind mounts of files and directories to avoid creating empty directories when the source is missing, returning an error instead (#473)
    • This changed necessitated a switch from Python's built-in YAML library to ruamel.yaml
  • code to pull from MISP feeds should specify JSON as preferred format in HTTP headers (#520)
  • add optional service argument to restart script (#521)
  • replace API link on landing page with extracted-files (#524)
  • exclude private IP space Intel::ADDR items when populating Zeek intel (#528)
  • updated some screenshots for the documentation
• Component version updates
  • Alpine v3.20 for nginx-proxy container (#500)
  • Arkime v5.3.0
  • Beats v8.14.3
  • Fluent Bit to v3.1.4
  • Logstash v8.14.3
  • NetBox v4.0.8
  • osd_transform v2.15.0
  • certifi to v2024.7.4 to address CVE-2024-39689 for Hedgehog Linux
• Bug fixes
  • tarball-based installation should not depend on UID inside of tarball, prevents installation if UID with which tarball's contents were created don't match installing user's (#519)
  • bacnet discovery log not parsed correctly (#523)
  • resolved issue with the build.sh helper script when building non-AMD64 Docker images
• Configuration changes (in environment variables in ./config/)
  • The variable ZEEK_DISABLE_ICS_GE_SRTP has been added to zeek.env and control_vars.conf to control enabling the network analyzer for the GE SRTP protocol. It's default value is true (indicating that the analyzer is disabled) as it is a somewhat uncommon OT protocol that likely won't be needed by most Malcolm users.
• Other
  • Removed long-deprecated net-map.json file support (#517)

Official ISO installer images for Malcolm and Hedgehog Linux can be downloaded from Malcolm's releases page on GitHub. Due to limits on individual files in GitHub releases, these ISO files have been split into 2GB chunks and can be reassembled with scripts provided for both Bash (release_cleaver.sh) and PowerShell (release_cleaver.ps1). See Downloading Malcolm - Installer ISOs for instructions.

Malcolm v24.06.0

Malcolm v24.06.0 contains new features, improvements, component version updates, and a few bug fixes.

v24.05.0...v24.06.0

NetBox: backwards compatibility-breaking change: This release of Malcolm updates NetBox from v3.6.7 to v4.0.6, for bug fixes, security updates, and requirements for Malcolm to support enrichment with multiple NetBox sites. However, NetBox's built-in migrations do not appear to work handle going from v3.6.7 to v4.0.6. It is likely that if you are using NetBox that you will encounter errors upon updating to this release of Malcolm. Prior to upgrading it is recommended that you navigate to Sites, IPAM > Prefixes, DCIM > Devices, and anywhere else you've populated NetBox data and click Export > All Data (CSV) and save those in case you need to recreate your NetBox inventory after upgrading. Malcolm's NetBox backup and restore will not work in this case. If you find NetBox has data errors after upgrading Malcolm, stop Malcolm and clear the NetBox inventory from your Malcolm installation directory (e.g., rm -rf ./netbox/postgres/* ./netbox/redis/*), then start Malcolm and recreate your NetBox inventory.

• Features and enhancements
  • Support for multiple NetBox sites (#449)
    • Malcolm now supports enrichment from a NetBox inventory for asset interaction analysis across multiple sites. The NetBox site can be specified for uploaded PCAP, for a Hedgehog Linux sensor, and for Malcolm live capture.
  • JA4+ replaces the JA3 TLS fingerprinting standard from 2017 (see also this blog post) (#419)
  • Support uploading Windows Event Log evtx files (#465) and update associated dashboard
  • Document using GitHub runners to build Malcolm images (for contributors' guide, #491)
  • Generate new forwarder SSL keys on-the-fly when transferring between Malcolm and Hedgehog Linux (#492)
  • Incorporate ATT&CK-based Control-system Indicator Detection for Zeek (ACID) (#489), a collection of Operational Techonology (OT) protocol indicators developed to alert on specific ATT&CK for ICS behaviors
  • Add platform architecture and machine boot time to Malcolm version API
  • Add links to the navigation pane of most dashboards to "other" dashboards for non-network log data (e.g., resource monitoring, Windows Event logs, etc.)
• Component version updates
  • Alpine to v3.20 for some of the Docker images
  • Beats to v8.14.1
  • capa to v7.1.0
  • elasticsearch-dsl to v8.13.1
  • elasticsearch-py to v8.14.0
  • Fluent Bit to v3.0.7
  • Logstash to v8.14.1
  • NetBox to v4.0.6 (from v3.6.7, #385)
  • OpenSearch and OpenSearch Dashboards to v2.15.0
  • opensearch-py to v2.6.0
  • psutil to v5.9.8
  • Supercronic to v0.2.30
  • urllib3 to v1.26.19
  • yq to v4.44.2
• Bug fixes
  • Arkime viewer not rolling PCAPs (#484)
  • Free up space in GitHub runner environment building ISO images to avoid errors due to exhausted disk space
• Configuration changes in environment variables
  • There are no significant changes or additions to the ./config/*.env environment variable files in Malcolm v24.06.0

Official ISO installer images for Malcolm and Hedgehog Linux can be downloaded from Malcolm's releases page on GitHub. Due to limits on individual files in GitHub releases, these ISO files have been split into 2GB chunks and can be reassembled with scripts provided for both Bash (release_cleaver.sh) and PowerShell (release_cleaver.ps1). See Downloading Malcolm - Installer ISOs for instructions.

Malcolm v24.05.0

Malcolm v24.05.0 contains new features, improvements, bug fixes and component version updates.

v24.04.0...v24.05.0

• Features and enhancements
  • Added ARM64/AArch64 support. Malcolm can now run natively on ARM64 hardware. The ./scripts/configure script should detect the architecture and automatically adjust the image: names in the docker-compose.yml files in Docker deployments, or this can be changed manually by appending -arm64 to the tag for Malcolm's Docker images, e.g., ghcr.io/idaholab/malcolm/zeek:24.05.0-arm64. (#369)
  • Support for new environment variables added to Hedgehog Linux's control_vars.conf for tuning capture-related settings for Arkime on the sensor, bringing them into parity with those that were available in the arkime-live container in Malcolm. (#476)
  • Tweaked some of the default resource-related live capture settings for Suricata and Arkime.
  • Reworked the environment variables used for tuning Zeek live capture resource and performance on both Malcolm and Hedgehog Linux. An in-depth discussion of these tuning parameters can be found in the documentation. (#475)
  • Allow setting the spiDataMaxIndexes variable for Arkime's config.ini file via the ARKIME_SPI_DATA_MAX_INDICES environment variable. (#471)
  • Allow custom tags to be specified at the point of log file ingestion (i.e., FileBeat) on Malcolm and Hedgehog Linux. This makes it easier to specify custom tags used to group network traffic by sensor. (#463)
  • Handle invalid URLs made to the Malcolm web-based UIs better (with a custom 404/502 page). (#461)
  • Switched to official .deb packages for Arkime rather than building from source, reducing build times significantly. (Thanks @awick.)
• Component version updates
  • Suricata to v7.0.5
    • Also, going forward Malcolm will track the latest Suricata release (from the Debian Stable Backports APT repository) rather than what's in the Debian Stable APT repository. (#462)
  • Arkime to v5.2.0
  • OpenSearch and OpenSearch Dashboards to v2.14.0
  • YARA to v4.5.1
  • Beats to v8.13.4
  • Logstash to v8.13.4
  • YQ to v4.44.1
  • Zeek to v6.2.1
  • Fluent Bit to v3.0.6
  • requests Python library to v2.32.0 for CVE-2024-35195
  • flask-cors Python library on Hedgehog Linux to v4.0.1 for CVE-2024-1681
  • Jinja Python library on Hedgehog Linux to v3.1.4 for CVE-2024-34064
  • Werkzeug Python library on Hedgehog linux to v3.0.3 for CVE-2024-34069
• Bug fixes
  • The code that cleans up already-processed Zeek and Suricata logs after a defined period of time was out of date for the current FileBeat registry behavior and would potentially leave log files around longer than they needed to be. This has been remedied. (#479)
  • Fixed issue where the BPF capture filter was not passed to Zeek correctly. (#474)
  • The process which queries threat intelligence feeds and generates the corresponding Zeek intel files will no longer relpace existing intel definitions unless it succeeds in pulling definitions from at least one of the specified feeds. (#472)
  • Fixed calculation of memory and CPU resources used in ./scripts/status for Kubernetes deployment. (#467)
• Configuration changes (in environment variables in ./config/) for Malcolm and in control_vars.conf for Hedgehog Linux
  • Malcolm
    • Added ARKIME_SPI_DATA_MAX_INDICES to arkime.env with a default value of 7, which manifests as spiDataMaxIndexes in Arkime's config.ini. If you are changing the Arkime index period from daily to weekly, hourly, etc., you may wish to adjust this value. (#471)
    • Added EXTRA_TAGS to upload-common.env for specifying custom tags to be associated with logs forwarded to Logstash by FileBeat. (#463)
    • A number of new and modified environment variables are available and can be added to zeek-live.env for tuning the resource utilization and performance of Zeek's live capture, see the documentation for details. (#475)
  • Hedgehog Linux
    • A number of new and modified environment variables are available for control_vars.conf for tuning the resource utilization and performance of Zeek's live capture, see the documentation for details. (#475)
    • Added support for new environment variables added to Hedgehog Linux's control_vars.conf for tuning capture-related settings for Arkime on the sensor, bringing them into parity with those that were available in the arkime-live container in Malcolm. (#476)

Official ISO installer images for Malcolm and Hedgehog Linux can be downloaded from Malcolm's releases page on GitHub. Due to limits on individual files in GitHub releases, these ISO files have been split into 2GB chunks and can be reassembled with scripts provided for both Bash (release_cleaver.sh) and PowerShell (release_cleaver.ps1). See Downloading Malcolm - Installer ISOs for instructions.

Malcolm v24.04.0

Malcolm v24.04.0 contains new features, improvements, bug fixes and component version updates.

v24.03.1...v24.04.0

Because some of the environment variables used for configuring Malcolm have been reorganized in the .env files found in the ./config directory, it is recommended you re-run ./scripts/configure for this release.

• Features and enhancements
  • Zeek-extracted files scanned and preserved on a Hedgehog Linux sensor can now be accessed via the extracted files download user interface (#331).
  • Improvements to creation of index templates, dashboards, and other saved objects on startup (#208) to ensure that saved objects get created correctly upon upgrade (see this comment for more details on this feature).
  • Populating the NetBox inventory via passively-gathered network traffic metadata now uses network traffic logs for DNS, NTLM, and DHCP to identify assets' host names when possible for use when populating device and VM names (#415). Autopopulated devices now have their status field set to Active rather than Stage, and uses tags instead to indicated that they were created through autopopulation.
  • Users can now specify pruning thresholds for carved files so that old files are deleted in order to avoid filling available storage (#453). See a new section of documentation on Managing disk usage for more information about this and similar settings.
  • Users can now specify a prefix that will be prepended to dashboards as they are imported into OpenSearch Dashboards or Kibana, allowing users who have dashboards from other sources to differentiate between those and Malcolm's (#455).
  • The default anomaly detectors created for the OpenSearch Anomaly Detection plugin are now created with category fields for high cardinality to allow for better breakdown of contributing values to anomalies discovered (#464).
  • Include JA4+ plugin in Arkime. See #419 for status on upcoming full JA4+ support in Malcolm.
  • Hedgehog Linux sensors can now periodically refresh their Zeek inteligence files.
    • NOTE: Due to an oversight, a value is missing from the default Hedgehog Linux configuration in this release, preventing the intel refresh cron job from executing. As a workaround, appending the line export INTEL_DIR=/opt/sensor/sensor_ctl/zeek/intel to /opt/sensor/sensor_ctl/control_vars.conf and restarting the sensor services will remedy the situation. This will be corrected in the next Malcolm release.
  • Assorted documentation improvements.
• Component version updates
  • Arkime to v5.1.2
  • OpenSearch and OpenSearch Dashboards to v2.13.0
  • Beats to v8.13.2
  • Logstash to v8.13.2
  • gunicorn to v22.0.0 to address CVE-2024-1135.
  • elasticsearch-dsl to v8.13.0
  • elasticsearch-py to v8.13.0
  • idna to v3.7 to address CVE-2024-3651
  • Fluent Bit to v3.0.3
• Bug fixes
  • The documentation for Windows host system configuration was out of date and has been updated for the latest version of Microsoft Windows Subsystem for Linux (#421).
  • An issue was fixed in which Malcolm's list of users and their password hashes could become corrupted if the file did not initially end with a newline character (#426).
  • The manner in which Zeek intel files are generated has been changed to avoid problems found in Kubernetes deployments when scaling out the number of zeek-live containers (#456). See this comment for more details.
  • Removed the version top-level element from docker-compose.yml files as it is now obsolete and caused a warning message that sometimes was not handled correctly.
  • Fix Malcolm ISO not correctly detecting if it's in a live boot ISO environment or installed mode.
  • Restart live Zeek instances with zeekctl deploy instead of zeekctl restart.
• Configuration changes (in environment variables in ./config/)
  • ARKIME_QUERY_ALL_INDICES in arkime.env can be set to control the queryAllIndices setting in Arkime's config.ini.
  • DASHBOARDS_PREFIX in dashboards-helper.env has been added for #455 (see above in Features and Enhancements).
  • LOGSTASH_NETBOX_ENRICHMENT_DATASETS in logstash.env has been changed to include zeek.dhcp, zeek.dns, and zeek.ntlm to support #415 (see above in Features and Enhancements).
  • LOGSTASH_ZEEK_IGNORED_LOGS in logstash.env has been changed to remove capture_loss and stats so that those diagnostic Zeek logs can be parsed without the user having to manually change this variable.
  • ZEEK_CRON has been removed from zeek-live.env and ZEEK_INTEL_REFRESH_CRON_EXPRESSION was removed from zeek.env and moved to the "offline" version of the container in zeek-offline.env for #456.
  • EXTRACTED_FILE_PRUNE_THRESHOLD_MAX_SIZE, EXTRACTED_FILE_PRUNE_THRESHOLD_TOTAL_DISK_USAGE_PERCENT, and EXTRACTED_FILE_PRUNE_INTERVAL_SECONDS were added to zeek.env for #453. See a new section of documentation on Managing disk usage for more information about these and similar settings.

Official ISO installer images for Malcolm and Hedgehog Linux can now be downloaded from Malcolm's releases page on GitHub. Due to limits on individual files in GitHub releases, these ISO files have been split into 2GB chunks and can be reassembled with scripts provided for both Bash (release_cleaver.sh) and PowerShell (release_cleaver.ps1). See Downloading Malcolm - Installer ISOs for instructions.

Malcolm v24.03.1

Malcolm v24.03.1 contains new features, improvements, bug fixes and component version updates.

v24.03.0...v24.03.1

Because some of the environment variables used for configuring Malcolm have been reorganized in the .env files found in the ./config directory, it is strongly recommended you re-run ./scripts/configure for this release.

• Features and enhancements
  • Malcolm instances created using the installer ISO will now detect and format any large (>100GB) storage devices and automatically set them up for use for storing the OpenSearch data store, PCAP files, and/or log storage, similar to what Hedgehog Linux does. (#266)
  • Since v24.01.0, Malcolm has allowed users to specify custom index patterns for Zeek and Suricata logs (see issue 313). This release now also provides the capability for Arkime to know about those indexes so that those documents also appear in Arkime search results. (#313, arkime/arkime#2705) As this is not released in Arkime yet, Malcolm is using a local patch with these changes, to be released upstream in Arkime v5.0.2.
  • A new setting for Logstash has been added to allow autocreation and assignment of NetBox subnets during enrichment. If "Should Malcolm automatically create missing NetBox subnet prefixes based on observed network traffic?" is answered to the affirmative during configuration, observed traffic that does not fall into any existing NetBox prefix will cause one to automatically be created, creating them one level down (e.g., 8 additional masked bits) from the RFC1918 address space definitions. This replaces an earlier feature (controlled by the NETBOX_PRELOAD_PREFIXES variable, which no longer has any effect) in which three giant buckets (one for each block in the RFC1918 private address definitions) could be created once at startup. (#436). So, for example:
    • 10./16 (255.255.0.0)
      • the IP address 10.9.0.215 would cause us to create and assign it to a 10.9.0.0/16 subnet
    • 192.168./24 (255.255.255.0)
      • the IP address 192.168.100.123 would cause us to create and assign it to a 192.168.100.0/24 subnet
    • 172.16./20 (255.255.240.0)
      • the IP address 172.16.29.10 would cause us to create and assign it to a 172.16.29.10/20 subnet
  • New configuration settings have been added to specify creation and rotation of Suricata's EVE JSON log files, including controls for threaded file output and file rotation. See this comment for a full description of the changes (#445). Most noteworthy are:
    • SURICATA_EVE_THREADED - controls threaded file output (default false)
    • SURICATA_EVE_ROTATE_INTERVAL - controls eve.json file rotation (default 1h)
  • Table visualizations in Malcolm's prebuilt OpenSearch Dashboards were not consistent in the number of rows returned. This has been standardized to 100 and otherBucket: true has been set for all of these table visualizations to ensure that the end user knows that Other rows may also exist outside of the rows shown. (#447)
  • Some some field mappings were moved from malcolm_template.json to the composable template malcolm_common.json
  • Documentation improvements
  • Minor update to slides
  • Some directories named like bro_logs were renamed to zeek_logs on Hedgehog Linux
  • The Community ID field is now being added to Zeek's notice.log
  • Attempt to install necessary Python 3 packages at the beginning of install.py instead of just failing
• Component version updates
  • Zeek to v6.2.0
  • opensearch-py to v2.5.0
  • Fluent Bit to v3.0.0
  • Moved from the no-longer-maintained Salesforce repo for HASSH to Corelight's
• Bug fixes
  • AF_PACKET was not being utilized for capturing traffic on Malcolm in the zeek-live container. This did not affect Suricata, Arkime capture, or any of the Hedgehog Linux processes. (#437)
  • The Packet Capture Statistics dashboard was not correctly computing seen and dropped packets for Suricata. (#442)
  • A STDERR warning from the new Docker Compose v2.25 was messing up the creation of the OpenSearch keystore file. (#452)
  • Fixed an issue in which the Dashboards for non-network data (e.g., temperatures, resource usage, etc.) would not see the correct data if the MALCOLM_OTHER_INDEX_PATTERN variable had been set to something other than the default.
  • Ensure that index names created for use by Logstash sending to OpenSearch/Elasticsearch are lowercase
  • Major cleanup and refactoring of the NetBox enrichment code used by Logstash
• Configuration changes (in environment variables in ./config/)
  • ARKIME_DEBUG_LEVEL=0 has been added to arkime.env to control the debug level for Arkime's config.ini.
  • Additions/deletions in netbox-common.env (also, see below for some existing variables that were moved from logstash.env):
    • NETBOX_PRELOAD_PREFIXES has been removed and replaced with NETBOX_AUTO_CREATE_PREFIX for #436
    • NETBOX_ENRICHMENT_LOOKUP_SERVICE=true has been added to indicate whether or not services (i.e., destination IP/port) should be looked up during NetBox enrichment
  • Comments were added to opensearch.env to give examples of how to set the time bucketing for OpenSearch/Elasticsearch indexes
  • In addition to the new variables mentioned above, some cleanup and organization was done in the environment variable files used for configuring Malcolm:
    • LOG_CLEANUP_MINUTES and ZIP_CLEANUP_MINUTES are now in filebeat.env, moved from upload-common.env
    • Some NetBox related variables have been moved from logstash.env to netbox-common.env and renamed:
      • LOGSTASH_NETBOX_ENRICHMENT is now NETBOX_ENRICHMENT
      • LOGSTASH_NETBOX_AUTO_POPULATE is now NETBOX_AUTO_POPULATE
      • LOGSTASH_NETBOX_CACHE_SIZE is now NETBOX_CACHE_SIZE
      • LOGSTASH_NETBOX_CACHE_TTL is now NETBOX_CACHE_TTL

Official ISO installer images for Malcolm and Hedgehog Linux can now be downloaded from Malcolm's releases page on GitHub. Due to limits on individual files in GitHub releases, these ISO files have been split into 2GB chunks and can be reassembled with scripts provided for both Bash (release_cleaver.sh) and PowerShell (release_cleaver.ps1). See Downloading Malcolm - Installer ISOs for instructions.

Malcolm v24.03.0

Malcolm v24.03.0 contains new features, improvements, bug fixes and component version updates.

v24.02.0...v24.03.0

• Features and enhancements
  • support json-delimited import for Zeek logs (#65)
  • go through list of Trivy security findings (#236)
  • support /attributes and /events enpoints from MISP feed for Zeek intel generation (#336)
  • KEV detections for Unitronics VisiLogic CVE-2023-6448 (#394)
  • create dashboards for other non-network log data (#414)
  • links on landing page should open in a new tab (#427)
  • incorporate ICSNPP Profinet IO CM parser (#429)
• Component version updates
  • Arkime to v5.0.1
  • OpenSearch and OpenSearch Dashboards to v2.12.0
• Bug fixes
  • fix the way we do environment variables in local.zeek (#413)
  • a few issues with the install.py script when installing from GitHub releases (#416)
  • htadmin creating entries without a newline between them in the htpasswd file (#426)
  • hard-coded date value in Kibana pivot links (#428)
  • unencrypted, unzipped extracted file download not working (#431)
• Configuration changes (in environment variables in ./config/)
  • these variables in zeek.env

  # Set to true to indicate that Zeek should output logs in JSON format
  ZEEK_JSON=
  # Whether or not to require SSL certificate verification when querying a TAXII or MISP feed
  ZEEK_INTEL_FEED_SSL_CERTIFICATE_VERIFICATION=false
  # Whether or not to disable the ICSNPP Profinet IO CM parser
  ZEEK_DISABLE_ICS_PROFINET_IO_CM=
  

Malcolm and Hedgehog Linux may be obtained by pulling or building the Docker images and/or building the ISO installer images as described in the documentation. Unofficial ISO installer images for Malcolm and Hedgehog Linux are not hosted on GitHub, but may be downloaded from https://malcolm.fyi/.

Malcolm v24.02.1

Malcolm v24.02.1 is identical to v24.02.0 except for a minor fix to the code that builds the Hedgehog Linux Raspberry Pi image.

The usual build artifacts are not included in this release, and new docker images for Malcolm have not been published.