cdxgen --output option makes 2 different files including the default bom.json

[Nicolas-Peiffer]

I noticed that cdxgen --output myfilename.json creates 2 different files, including the default bom.json and myfilename.json.
Those 2 files are almost identical, except for the "occurrences": field in bom.json, and for the way one file is a prettified JSON and the other bom.json is a one line JSON file.

How to reproduce

cdxgen --version
10.7.0

Java env

$ java --version
openjdk 22 2024-03-19
OpenJDK Runtime Environment (build 22)
OpenJDK 64-Bit Server VM (build 22, mixed mode, sharing)

$ mvn --version
Apache Maven 3.9.8 (36645f6c9b5079805ea5009217e36f2cffd34256)
Maven home: /usr/share/java/maven
Java version: 22, vendor: N/A, runtime: /usr/lib/jvm/java-22-openjdk
Default locale: fr_FR, platform encoding: UTF-8
OS name: "linux", version: "6.6.33-1-lts", arch: "amd64", family: "unix"

get a Java Project

git clone https://github.com/CycloneDX/cyclonedx-core-java.git

Test cdxgen

I create a script timed_command.sh to get start and end datetime.

#!/bin/bash

# clean files
rm bom.json bom-1.6-cyclonedx-core-java.json

# Capture start time
start_time=$(date "+%Y-%m-%d %H:%M:%S")
start_seconds=$(date +%s)
echo "Start time: $start_time"

# Run your command
export CDXGEN_DEBUG_MODE=debug
echo "CDXGEN_DEBUG_MODE is $CDXGEN_DEBUG_MODE"
echo ""
cdxgen -t java -r --include-formulation --include-crypto --spec-version 1.6 --output bom-1.6-cyclonedx-core-java.json
echo ""

# Capture end time
end_time=$(date "+%Y-%m-%d %H:%M:%S")
end_seconds=$(date +%s)
echo "End time: $end_time"

# Calculate elapsed time
elapsed_seconds=$((end_seconds - start_seconds))
echo "Elapsed time: $elapsed_seconds seconds"
echo ""

echo "ls -l --time-style=full-iso bom* | awk '{print \$5, \$6, \$7, \$8, \$9}'"
ls -hl --time-style=full-iso bom* | awk '{print $5, $6, $7, $8, $9}'

chmod +x timed_command.sh
./timed_command.sh

Results:

Start time: 2024-06-21 18:01:56
CDXGEN_DEBUG_MODE is debug

Executing 'mvn -fn org.cyclonedx:cyclonedx-maven-plugin:2.8.0:makeAggregateBom -DoutputName=bom -DincludeTestScope=true' in /tmp/cyclonedx/cyclonedx-core-java
Extracting data from generated bom file /tmp/cyclonedx/cyclonedx-core-java/target/bom.json
IRI failed validation https://github.com:networknt/json-schema-validator.git
IRI failed validation https://github.com:ethlo/itu
Executing /usr/lib/node_modules/@cyclonedx/cdxgen/node_modules/@cyclonedx/cdxgen-plugins-bin/plugins/osquery/osqueryi-linux-amd64 --json select * from deb_packages where name like '%ssl%' OR name like '%crypto%';
Executing /usr/lib/node_modules/@cyclonedx/cdxgen/node_modules/@cyclonedx/cdxgen-plugins-bin/plugins/osquery/osqueryi-linux-amd64 --json select * from portage_packages where package like '%ssl%' OR package like '%crypto%';
Executing /usr/lib/node_modules/@cyclonedx/cdxgen/node_modules/@cyclonedx/cdxgen-plugins-bin/plugins/osquery/osqueryi-linux-amd64 --json select * from rpm_packages where name like '%ssl%' OR name like '%crypto%';
Executing /usr/lib/node_modules/@cyclonedx/cdxgen/node_modules/@cyclonedx/cdxgen-plugins-bin/plugins/osquery/osqueryi-linux-amd64 --json select * from python_packages where name like '%ssl%' OR name like '%crypto%';
Executing /usr/lib/node_modules/@cyclonedx/cdxgen/node_modules/@cyclonedx/cdxgen-plugins-bin/plugins/osquery/osqueryi-linux-amd64 --json select * from programs where name like '%ssl%' OR name like '%crypto%';
Executing /usr/lib/node_modules/@cyclonedx/cdxgen/node_modules/@cyclonedx/cdxgen-plugins-bin/plugins/osquery/osqueryi-linux-amd64 --json SELECT * FROM homebrew_packages where name like '%ssl%' OR name like '%crypto%';
Executing /usr/lib/node_modules/@cyclonedx/cdxgen/node_modules/@cyclonedx/cdxgen-plugins-bin/plugins/osquery/osqueryi-linux-amd64 --json SELECT * FROM authorized_keys;
Executing /usr/lib/node_modules/@cyclonedx/cdxgen/node_modules/@cyclonedx/cdxgen-plugins-bin/plugins/osquery/osqueryi-linux-amd64 --json SELECT * FROM certificates;
Executing /usr/lib/node_modules/@cyclonedx/cdxgen/node_modules/@cyclonedx/cdxgen-plugins-bin/plugins/osquery/osqueryi-linux-amd64 --json SELECT * FROM keychain_items;
Executing /usr/lib/node_modules/@cyclonedx/cdxgen/node_modules/@cyclonedx/cdxgen-plugins-bin/plugins/osquery/osqueryi-linux-amd64 --json SELECT * FROM kernel_keys;
Executing /usr/lib/node_modules/@cyclonedx/cdxgen/node_modules/@cyclonedx/cdxgen-plugins-bin/plugins/osquery/osqueryi-linux-amd64 --json SELECT * FROM yum_sources;
bom.json created successfully.

End time: 2024-06-21 18:02:27
Elapsed time: 31 seconds

ls -l --time-style=full-iso bom* | awk '{print $5, $6, $7, $8, $9}'
1,1M 2024-06-21 18:02:24.721395070 +0200 bom-1.6-cyclonedx-core-java.json
771K 2024-06-21 18:02:26.168061828 +0200 bom.json

Notice the 2 files bom-1.6-cyclonedx-core-java.json and bom.json are of different sizes 1,1M and 771K.

ls -l --time-style=full-iso bom* | awk '{print $5, $6, $7, $8, $9}'
1,1M 2024-06-21 18:02:24.721395070 +0200 bom-1.6-cyclonedx-core-java.json
771K 2024-06-21 18:02:26.168061828 +0200 bom.json

This is because bom.json is a one line JSON file,
whereas bom-1.6-cyclonedx-core-java.json is pretty printed JSON.

Let's prettify the JSON with jq:

cat bom.json | jq > bom-pretty.json

Notice bom-pretty.json is bigger than bom-1.6-cyclonedx-core-java.json,
because of the additional "occurrences": fields.

ls -hl --time-style=full-iso bom* | awk '{print $5, $6, $7, $8, $9}'
1,1M 2024-06-21 18:02:24.721395070 +0200 bom-1.6-cyclonedx-core-java.json
771K 2024-06-21 18:02:26.168061828 +0200 bom.json
1,2M 2024-06-21 18:05:00.784738272 +0200 bom-pretty.json

Notice also below the version field and the timestamp that are different.

[prabhu]

@Nicolas-Peiffer Thank you for the detailed report. cdxgen and evinse are two separate tools. cdxgen generates an SBOM, while evinse uses that sbom and works with atom to attach evidences. The argument --include-crypto automatically enables evinse.

You can see the default bom file name used by evinse is bom.json. The argument to override this is --evinse-output

cdxgen/bin/cdxgen.js

Line 66 in 519545f

default: "bom.json",

The version 2 indicates the fact that the bom was enhanced to create the new version that includes occurrences.

Let me know if there is a better approach to improve this.

[Nicolas-Peiffer]

Thank you @prabhu for this quick answer, I understand your answer.
But let me provide a couple of feedbacks as I am a first time cdxgen user.
This could help to improve cdxgen user documentation.

> cdxgen and evinse are two separate tools"

First, I am not familiar with evinse, I was not aware that it is a separate
tool from cdxgen.
Should have I read this section
Evinse Mode / SaaSBOM
to understand Evinse ?

> The argument to override this is --evinse-output

I was not aware the option flag --evinse-output exist. Are you sure this is
the right flag? Because I can't find it.

cdxgen -v
10.7.0

cdxgen -h | grep -i "evinse"          # nothing
cdxgen -h | grep -i "evinse-output"   # nothing
cdxgen -h | grep -i "evidence"
      --evidence               Generate SBOM with evidence for supported languages.  [booléen] [défaut : false]

I was not aware at all that the option --include-crypto uses evinse.
This should probably be explained in the doc section
Generate Cryptography Bill of Materials (CBOM).

It is not straightforward to have several output options
--output and --evinse-output. I would rather have a cdxgen evinse
sub-command with its own --output flag, or reusing the global --output
flag properly (which means not creating 2 files)

cdxgen evinse --output myoutputfile.json

JSON pretty printing?

I find it weird to have one JSON file that is pretty printed, and the other
that is a oneliner JSON file.

Either every JSON output should not be pretty printed, or every JSON output
should be oneliner.
Or JSON pretty printing or not pretty printing should be an option that you can
control via a flag.

A flag --pretty-json or --pretty can set pretty printing to true.

cdxgen evinse --output myoutputfile.json --pretty-json

Other Feedbacks (not related to evinse or --output)

Disclaimer: I know close to nothing about Javascript's libs and best
practices of codding. I am more familiar with GoLang and python.

Use sub-commands instead of flags?

I find it weird that the cdxgen CLI uses only flags instead of sub-commands.

I think sub-commands would make sense for some features of cdxgen, like for
example running cdxgen as a server.

Below I put an example of a cdxgen server sub-command:

cdxgen -h
cdxgen [commande]

Commandes :
  cdxgen completion  Generate bash/zsh completion
  cdxgen server      Run cdxgen as a server
  [...]

This would make the flags --server-host and --server-port dedicated to the cdxgen server sub-command:

cdxgen server -h

[...]

Options :
      --server-host            Listen address                                                     [défaut : "127.0.0.1"]
      --server-port            Listen port                                                             [défaut : "9090"]
      [...]

Using sub-commands could make CLI easier to use by separating mutually
exclusive flag options.

Completion Scripts: add support for fish?

I am a fish shell user: https://fishshell.com. I find it unfortunate that the
cdxgen completion command does not generate completion scripts for fish
with support for inline help messages.

cdxgen -h
cdxgen [commande]

Commandes :
  cdxgen completion  Generate bash/zsh completion

I am more familiar with CLI frameworks like cobra and typer that natively
support bash, fish, powershell and zsh.

• GoLang cobra CLI: fish completion support
• python typer CLI: typer add help messages to completion

Example with skopeo in Go:

skopeo completion -h

Generate the autocompletion script for skopeo for the specified shell.
See each sub-command's help for details on how to use the generated script.

Usage:
  skopeo completion [command]

Available Commands:
  bash        Generate the autocompletion script for bash
  fish        Generate the autocompletion script for fish
  powershell  Generate the autocompletion script for powershell
  zsh         Generate the autocompletion script for zsh

I ask Chat GPT if this native fish CLI completion feature exits in
Javascript, but I am disappointed by the Javascript CLI frameworks not
supporting fish.

How to generate a Fish shell completion scripts with `oclif` (Javascript)

To generate Fish shell completion scripts for your npm package, you can use the `oclif` framework, which simplifies the creation of CLI tools and supports autocompletion out of the box, including Fish shell. Here’s a step-by-step guide on how to achieve this:

### 1. Set Up Your Project with oclif

1. **Install oclif**:
    ```sh
    npm install -g oclif
    ```

2. **Create a New oclif CLI**:
    ```sh
    oclif generate my-cli
    ```

    Replace `my-cli` with your desired project name.

3. **Navigate to Your Project Directory**:
    ```sh
    cd my-cli
    ```

4. **Install Dependencies**:
    ```sh
    npm install
    ```

### 2. Add Autocomplete Plugin

1. **Install the Autocomplete Plugin**:
    ```sh
    npm install @oclif/plugin-autocomplete
    ```

2. **Add the Plugin to Your oclif Configuration**:
    Edit `package.json` to include the autocomplete plugin:
    ```json
    {
    "name": "my-cli",
    "version": "0.0.0",
    "oclif": {
        "commands": "./src/commands",
        "bin": "my-cli",
        "plugins": [
        "@oclif/plugin-autocomplete"
        ]
    }
    }
    ```

### 3. Generate Fish Completion Script

1. **Generate the Completion Script**:
    Run the following command to generate the Fish completion script:
    ```sh
    ./bin/dev autocomplete fish
    ```

2. **Save the Completion Script**:
    Save the generated script to your Fish completions directory:
    ```sh
    ./bin/dev autocomplete fish > ~/.config/fish/completions/my-cli.fish
    ```

    Make sure to replace `my-cli` with the actual name of your CLI tool.

### 4. Source the Completion Script

1. **Update Your Fish Configuration**:
    Ensure that the Fish shell sources the completion script. This is usually done automatically if you place the script in `~/.config/fish/completions/`.

2. **Restart Your Fish Shell**:
    Open a new terminal or restart the Fish shell to load the new completions.

### Example Project Configuration

Here’s an example of how your `package.json` might look:

```json
{
"name": "my-cli",
"version": "0.0.0",
"description": "My CLI tool",
"bin": "./bin/dev",
"oclif": {
    "commands": "./src/commands",
    "bin": "my-cli",
    "plugins": [
    "@oclif/plugin-autocomplete"
    ]
},
"dependencies": {
    "@oclif/command": "^1.8.0",
    "@oclif/config": "^1.15.0",
    "@oclif/errors": "^1.3.3",
    "@oclif/parser": "^3.8.3",
    "@oclif/plugin-autocomplete": "^0.3.0"
}
}
```

By following these steps, you can generate and use Fish shell completion scripts for your npm CLI tool. This setup uses the `oclif` framework to simplify the process and ensure that your CLI tool supports autocompletion across different shells.

Implementation of cdxgen not following the latest CycloneDX Specs?

Note: this should be a dedicated issue. Let me know if you want me to post the issue.

When I add the JSON schema URL at the top of the JSON file
"$schema": "http://cyclonedx.org/schema/bom-1.6.schema.json"
to the BOM files generated by cdxgen...:

{
  "$schema": "http://cyclonedx.org/schema/bom-1.6.schema.json",
  "bomFormat": "CycloneDX",
  "specVersion": "1.6",

... I notice at least 2 issues with the BOM generated by cdxgen not following
the latest CycloneDX specification:

• using deprecated author field
• for field "cryptographic-asset": in the following bom-pretty.json file, notice the use of "properties" instead of using "cryptoProperties for "cryptographic-asset" of type certificate. This is not following the CBOM spec from CDX 1.6.

I put below the output I generated with:

cdxgen -t java -r --include-formulation --include-crypto --spec-version 1.6 --output bom-1.6-cyclonedx-core-java.json
cat bom.json | jq > bom-pretty.json

bom-pretty.json

And the error message at JSON schema validation:

Disclaimer

Keep in mind that I am biased by GoLang and python best practices. I can not
give advice on how this should be done with Javascript. I would have not used
Javascript for such a tool like cdxgen.

I myself am working on a tool (proof of concept PoC) written in GoLang to help
me write Crypto-BOM. For now, my PoC only converts a .csv file into a CBOM.
So in a way, this is less inteligent than cdxgen which searches for crypto
libs automatically.

But I want my PoC to be able to compute CBOM for projects written in GoLang,
Java, C++, Rust... I am yet to figure out the best approach. And I do not know
if cdxgen is the best solution for my needs.

[prabhu]

Thanks @Nicolas-Peiffer for the detailed feedback. I will digest your comments and come up with some workable issues over time. I am so glad that you have taken an interest in generating CBOMs. CBOM is quite new and a diverse range of implementations, both commercial and open-source, are essential to improve its adoption. All the very best for your project!