Dependencies not populating

[setchy]

I'm noticing that for many of our java repositories, that the dependencies array is empty.

Wanted to seek feedback from others to see if this is localized to my environment, or wider spread

I'd really like to visualize the Dependency Graph within Dependency Track

[prabhu]

@setchy, is the BOM missing dependencies, or is the dependency track not showing the tree? They are two different things since the dependency track doesn't support dangling trees.

[caojinghuan]

I'm experiencing the same problem, used cdxgen.exe -t java -o output_bom.xml，the file uploaded to dependency-track platform. there's no dependency tree... don't have information about primary dependency or subsidiary dependency.
need help...

[caojinghuan]

tried with json. upload to dependency-track, it prompts BOM uploaded in the top-right corner. but the component panel is none. it seems to be not recognizing this file format. my dependency-track is v4.3.6, set-up with docker-compose

if I run cdxgen.exe -t java -o output_bom.xml -p , with the -p to print the tree, I do see there's primary dependency and it's subsidiaries underneath

[prabhu]

Ah. If cdxgen prints the tree, then the issue must be that the root node does not match the parent component.

[caojinghuan]

how to resolve this issue?

[prabhu]

@caojinghuan could you share the BOM or share an example project? You can DM on discord if it is a private repo.

[caojinghuan]

Unfortunately，I can't share the project.

[gbennett-squarespace]

So I don't want to hijack this to talk about my issues with Java and dependencies, but this seems to be a fairly related issue to what I'm currently seeing with Java based repositories (and specifically for our env, Gradle/Spring based). I also want to be clear I'm not a Java dev so this is all what I'm understanding from a particular environment with what research I've done on it, none of this may be standard convention or best practice.

We have a couple of "fun" decisions that were made:

1. We have components missing the build.gradle file or what might be considered other "important" files for Java/Maven/Spring.
2. We don't specify the version of Java or Spring to use anywhere.
3. We have lots of different components all nested in a single repo and at multiple levels, which makes it hard to (programatically) separate what is being used where and for what purpose.

Running cdxgen against these projects gives some pretty similar results:

* What went wrong:
Could not compile settings file '/Users/gbennett/<project>/settings.gradle'.

So after this stacktrace, cdxgen fails and doesn't continue on, I think this is the first problem here is that when it stacktraces and fails to build, we don't try other components within that same repo.

The second is that if we want to build the project in order to better and more accurately identify the components, there's a lot more that goes around that then just running a basic command, especially when folks have all sorts of weirdness in their environment. I know there's a issue that was recently(ish) opened that talks about this, but especially if we have wrappers going on, or don't specify the version of Spring or Java to use, how do you even begin to successfully build the component?

So, at least in our particular case, we do have a fairly decent set of dependencies being used between gradle.properties, settings.properties, and build.gradle. I have no idea if the way we use them is standard convention, but it works for us so it's at least a working convention.

In our environment, we'd have to do the following:

1. Parse all the files and define the variables that they're using (example: morphiaVersion=1.3.2).
2. Parse all files and find their definitions for dependencies being used. Examples:
   implementation "org.mongodb.morphia:morphia:${morphiaVersion}"

dependencies {
    classpath "com.diffplug.spotless:spotless-plugin-gradle:5.7.0"
  }

plugins {
  id "com.org.project" version "${projectPlatformVersion}"
  id "com.org.project2" version "${project2Version}"
}

3. Possibly using the settings.gradle to help build the dependency tree. For example, the root might point to 6 that are 1 level deep, those might have their own settings.gradle that point to deeper dependencies. All sorts of fun with circular logic or dangling branches that could apply here. Example: include 'product1' or include 'integrations:subsubproduct2' (2nd uses : to delve deeper into sub-folders).

The TL;DR: here is that while building the Java project(s) might be the most accurate for determining dependencies, I don't think it's the easiest, simplest, or most reliable. Especially when the build is setup for failure by having unspecified dependencies/versions or super complex wrappers that do a lot of heavy lifting. We run into all sorts of fun stuff like different projects needing very specific versions of Java or some weird combo of Java/Spring/Other Things. This is likely to be very expensive time and processing wise as we'd have to install/uninstall lots of stuff for each run as there's likely to be conflicts and other problems with one component/project and the next one (even inside a single repo).

A flag to set whether to do a full on build of Java or simply scan the files for possible dependencies might be a way to go, with the ability to do a full build being the way to go if you need a super in depth BOM, and those who just want to get a grasp on their basic dependencies go with the "scan my files and see what I reference" route. I think if we want to go the build route we're going to need a lot more customizability/configuration in cdxgen as there's all sorts of fun things there, and the current setup of passing gradle dependency tasks or extra args isn't going to work for a lot of projects, unless you're creating a cdxgen env specific for each individual project (which isn't terribly reusable).

[prabhu]

@gbennett-squarespace, perhaps it's time to think enterprise support? As you point out, it requires some solid customization to make it work for such difficult cases, and what better way is there to reward our team and fund a solid contribution?

[gbennett-squarespace]

That's a different discussion than the root way that cdxgen does Java I think. :)

Looking at other tooling, such as Snyk, it does something very similar to what I'm describing here (going file by file and compiling a list of dependencies). It does not, however, understand versions when variables are used. Other tools such as Github Advanced Security only works if you can compile the Java packages (which is what I think we're running into here).

I don't think we need full fledged compilations as we're not at that level of auditing for dependencies (I understand other orgs may require that).

Regarding the above, do you think that the three steps I outlined would be a reasonable approach for one way to tackle detecting and identifying Java dependencies?

[prabhu]

@gbennett-squarespace we already have the tech that can understand usages of libraries in code with or without sbom. It's called the atom project. You can feel free to look into that. We need a component database to reverse map from imports to packages which is available as db.js in cdxgen and uses SQLite. Feel free to create a separate discussion if you are interested in giving this a go.

[prabhu]

Are there any other problems in the "dependencies" section? I heard recently that for dotnet, there are entries, but all of the dependsOn was empty. Have you seen anything similar to this?