Questions on dependency management

[mqware]

I have a couple of questions around dependency management:

1. The org.example.gradle.base.dependency-rules plugin references the versions platform project, but in turn the versions project includes the org.example.gradle.base.dependency-rules plugin. This looks like a circular reference? Is this correct, and if so, why is it necessary?

2. Why do we have both the versions project and the org.example.gradle.feature.use-all-catalog-versions plugin? It seems like the versions project could be used to declare any constraints, including strict versions. However, looping through the whole catalog and defining everything as a strict version seems like an overkill. For example, I have 80 libraries listed in my catalog, but I only need to define a constraint for two transitive dependencies. So, whether I add these two first to the catalog or not, I would only need to add two constraints to the versions project, which doesn't seem like a big deal. But having 82 constraints might create some overhead, and just doesn't seem necessary.

Thanks,
Peter

[mqware]

For Question 2 above, after looking into it more I see that to be able to control versions of transitive dependencies via the version catalog I would need to manually enter some kind of constraint in the versions platform project. This is what the plugin is trying to automate. (Sorry for stating the obvious, but I am still new to this).
However the following happened in my project. I have 4 jackson libraries in my catalog that are direct dependencies, set at version 2.15.2. These 4 libraries, along with 2 others, are also transitive dependencies of another library, but at version 2.15.3. So, without the plugin, Gradle used version 2.15.3 for all 6 libraries. But with the plugin enabled, my 4 direct dependencies were forced to stay at version 2.15.2, while the other 2 were at 2.15.3, so I ended up with mixed versions.
As I mentioned above, I have 2 transitive dependencies (unrelated to jackson) where I need to control the version. So, one solution I came up with is that I used the prefix transitive in the catalog for my transitive dependency aliases and then I modified the plugin to only apply constraints to a library if its alias name starts with transitive. This seemed to work well, but at the end I just decided to go back to my original solution and add a constraint directly into the versions project, as that looks simpler.
Of course I could have added those 2 jackson transitives to my catalog, thereby forcing them to stay at version 2.15.2 or just upgrade the 4 direct dependencies to 2.15.3 or higher.
I am not really sure what is the best option, but it just seems that using this plugin can have undesired effects and in certain cases it can prevent Gradle from resolving the versions correctly.

Thanks,
Peter

[jjohannes]

Lets see if I can give useful answers.

1. The cycle is there so that we create one consistent dependency resolution context where Gradle knows about all versions of third-party dependencies that appear in the dependency graph everywhere. On the one hand you have independent modules (e.g. :tatooine) on the other hand you know that you only ever use these modules in one larger context (:app). Therefore, it makes sense to have Gradle's conflict resolution mechanism select the same versions of third-party dependencies everywhere. Although from the perspective of the code, the modules further down in the hierarchy (e.g. :tatooine) do not know the modules further up (e.g. :app).
   I plan to write this "pattern" down in more detail in a document to discuss if there are better/other ways to set this up. And maybe even have it in the official Gradle documentation one day. As I think this is a very strong feature of Gradle which is currently almost unknown and could be explained better in documentation with some pictures/diagrams. I'll post another answer here, once I have done that.
2. The idea is that, as a user, you can use the Catalog to write down all versions you care about. No matter if it is for a direct dependency or only an indirect one. To make this happen, we automatically turn the Catalog into a BOM/platform via the use-all-catalog-versions plugin. Not sure what you refer to by "having 82 constraints might create some overhead", but for me there is no overhead. Yes, there are a lot of constraints that are "not needed", but Gradle can handle that and I never observed a negative performance impact or something like that caused by this.
   That being said, if conceptually it is cleaner for you to not put the transitive versions into the Catalog and instead put them into gradle/platform directly that is also a good setup. Or you could put them into the catalog, but then explicitly add them in gradle/platform using the catalog entries. I think if you (and everyone in the team) have a good understanding of this, that might even be the better setup. The setup I have right now is motivated by my experience that, for rather standard projects, it's easier for users if they have all versions in one place and automate as much as possible around this. There are additional thoughts on this in Use TOML version catalogs #5 and the bottom line is that there is not the one perfect setup for everyone, but that there are options to choose from. But there are certain patterns to follow (like not repeating a version no matter where you put it).
3. I am not sure I completely understand how you ended up with "with the plugin enabled, my 4 direct dependencies were forced to stay at version 2.15.2, while the other 2 were at 2.15.3". No matter where you define the versions, that should not happen as long as feature (1) above (consistent resolution with the "cycle") is enabled. If you like, you can reproduce this issue with this project in a fork and share it here. I would like to understand what is going on.

[mqware]

As always, thanks for the detailed replies.

1. I just thought that this would create an infinite loop, but I guess Gradle is smart enough to detect it and not fall into the trap. 😄 However, I also made an experiment and I compared the output of the dependencies task with the project being as-is and with the org.example.gradle.base.dependency-rules plugin removed from the versions project, and the dependency graphs were identical. So, I think a concrete example where this circular setup makes a difference would help in the understanding of the concept.
2. Yes, by "overhead" I meant some potential performance impact, but you're right, it's not like we are talking about millions of extra constraints, and a few hundred (or in my case less than a hundred), shouldn't have any noticeable effect.
   But I still like the approach that you have suggested above the best, which is to put the transitive dependencies into the catalog, and then adding them explicitly in gradle/platform (or what it's now called: gradle/versions) using the catalog entries. In the other discussion that you linked above you also seem to have the same opinion:
   "IMO, the idea that you, no matter what, get the version you defined is not a good way to do this for large dependency graphs. It sounds intuitive, but replacing (especially downgrading) transitive dependencies can break things."
   But this opinion goes directly against using something like the org.example.gradle.feature.use-all-catalog-versions plugin, so this is why I was asking.
3. I was able to recreate the problem in this project, (https://github.com/mqware/gradle-project-setup-howto/tree/jackson), but then I realized that that's because some of the entries in the catalog don't have defined versions. So, to fix this I can either add the missing versions to the catalog, or remove the org.example.gradle.feature.use-all-catalog-versions plugin (or both 😄)
   So, now after my great discovery, I would like to ask if you could explain what is the use-case for having catalog entries without a version? (I tried to search for an answer, but I only found cases where people were doing this, but they didn't say why. 🤷‍♂️)

Thanks,
Peter

[jjohannes]

(1) Yes it is a circle, but a desired one (and not an infinite one). I also talk about this and give an example in this video. You are right though that, when you already use a platform, often the consistent resolution has no additional effect. But it may always have when transitive dependency versions change indirectly through another dependency upgrade.
That being said, you can also achieve consistent resolution only with a platform, if you make sure to:

• Have only strict versions in the platform to make sure they are used and are never upgraded by a transitive dependency.
• Have all versions in the platform. Including all transitive versions. (For this, you would need some setup that makes sure to inform you if transitive versions are not listed. An extreme solution for that is this one: gradle-demos/dependency-constraints)

(2) I can only repeat that (right now) there is not "best solution for all". 😄 As long as you make sure that you...

• (a) get consistent resolution results everwhere (see above)
• (b) have a central place to define versions

...I think it is a good solution. Everything else depends a lot on what gives the best usability for whoever works on the project. The catalog can be used to make things more user friendly, but is completely optional to achieve (a) and (b).

(3) Thank you for sharing the example. I have not thought about this and I think I should change strictly to require in org.example.gradle.feature.use-all-catalog-versions, because this effect is not desired. I will do that change. I guess the "strictly" is only really a "good solution" if you put all versions into the catalog/platform.

[jjohannes]

Here is the document on Consistent Resolution. Details and motivation can be found in the beginning of the document. Comments are welcome!

Global Consistent Resolution with Gradle