Can we make building non-modular libraries easier?

[mtdowling]

I need to integrate with some legacy libraries that have split packages. Updating these libraries and releasing new major versions is one solution, but not practical in my case due to the large number of consumers of the affected libraries and the reluctance of consumers to update this particular library.

In order to write code against these kinds of problematic packages while still using modularized jars, I need to place some jars on the classpath and some jars on the module-path. This plugin currently places all jars on the module-path which wont work for me since I need to be more selective and avoid split-packages breaking my builds.

The approach I've played with which seems to work is to create a project that isn't modularized and loads things from both the classpath and module-path. I scan through all of the dependencies of the project and use a ModuleFinder to determine which dependencies are not automatic modules. These kinds of modules must be on the module-path. All of the dependencies of these modules defined using a ModuleReference#requires are also recursively added to the module-path. Everything else is added to the classpath.

This project could look to see if the package being built defines a module-info.java. If it does, it uses it's current approach of placing everything on the module path. If it doesn't it could use the approach described above to separate the classpath from the module path.

Here's a quick example of what I hacked together to help illustrate what I'm doing: https://gist.github.com/mtdowling/39905d445829ab1ac58d0cafecf6bc9e

Is this a reasonable approach, or are there issues I'm not seeing? Is this something you think could be added to this library?

[tlinkowski]

I'm wondering: is patching modules to prevent split packages not an option for you?

It seems easier than mixing classpath and modulepath (unless I misunderstood your requirements).

[mtdowling]

I did consider patching, but I don’t think it’s a good solution for libraries (unless I misunderstood how it works). Patching is done manually and requires all cascading consumers of a library to also apply the same patches. That won’t work when you’re dealing with tens of thousands of direct consumers and and a massively larger number of transitive consumers.

Splitting the classpath from the module path for classpath based libraries allows for developers to start using modularized libraries today without having to wait for things like rewrites or major version bumps of code that’s incompatible with the module system. It also would work automatically with no explicit configuration. The big trade off is that the code that integrates modules with classpath libraries can themselves not be modularized.

[tlinkowski]

OK, I think I finally understand what you're after (maybe should've read more carefully from the start). I hope I got it right this time (my apologies if I hadn't).

BTW. You have a typo in:

If it does, it uses it's current approach of placing everything on the classpath.

You meant "modulepath", right?

What

As far as I understand:

• you just want to build a "plain old" non-modular Java project (➡️ module-info.java)
• but you want this plugin to place some of its dependencies (i.e. all explicit modules + their dependencies) on the module path.

And what I mentioned before (module patching) would make sense only if you wanted to build a modular Java project.

How

Your presented snippet seems to do this job well *.

* Catch: if you have an unmodularized library (base-unmod-lib) as a dependency to both:

• another unmodularized library (unmod-lib)
• some modularized library (mod-lib)

then:

• mod-lib and base-unmod-lib will end up on module path,
• unmod-lib will end up on classpath.

Hence, unmod-lib won't "see" base-unmod-lib unless you specify --add-modules for it (more details here: Dependency Class Path ~> Module Path).

Why

I wonder why you need it. Strong encapsulation? Service loading mechanism?

I mean, why not put everything on the classpath?

If

Finally, I'm not sure if a plugin whose purpose is to build modular projects should actually support this (but it's for the authors to decide).

[mtdowling]

Typo fixed, thanks

Your understanding of what I'm after is correct. Thanks for summarizing it. I want to build a non-modular java library that depends on modules loaded through the module path while keeping non-modular jars on the classpath.

I wonder why you need it. Strong encapsulation? Service loading mechanism? I mean, why not put everything on the classpath?

Great question that I wished I had answered earlier :)

While you can refer to classes of modular jars loaded through the classpath, you can't find service providers defined in a modular jar's module-info.java, and a modular jar can't even find providers in it's own module.

Here's an example:

1. Jar A has a service named Foo that has multiple providers defined through it's module-info.java.
2. Jar A has a uses definition in it's module-info.java to use Foo.
3. Jar A has multiple provides statements to associated providers with the Foo service.
4. Jar A has a method named JarA#getProviders that loads services and returns them.
5. Jar B uses Jar A on the classpath and calls JarA#getProviders, but this method returns no results.

If you were to place Jar A on the module-path, then Jar B's call to JarA#getProviders would find service providers.

Either I'm doing something wrong or Java treats service providers differently based on if you're loading jars on the module-path or classpath (I hope I screwed something up, btw, because this is a pretty big difference between the module path and classpath).

Finally, I'm not sure if a plugin whose purpose is to build modular projects should actually support this (but it's for the authors to decide).

Definitely up to the authors, but I think this is fair game given the pitch of the project is "This Gradle plugin helps working with the Java Platform Module System". I think this change could help make modularized jars more mainstream too since it will allow modularized jars to be used with libraries that can't be loaded on the module path.

[tlinkowski]

While you can refer to classes of modular jars loaded through the classpath, you can't find service providers defined in a modular jar's module-info.java, and a modular jar can't even find providers in it's own module.

Yes, that's what I suspected when I wrote about "service loading mechanism".

BTW. A modular JAR can't even find providers in its own module, because — on classpath — it's treated as a regular JAR (its module-info.class is ignored).

Either I'm doing something wrong or Java treats service providers differently based on if you're loading jars on the module-path or classpath (I hope I screwed something up, btw, because this is a pretty big difference between the module path and classpath).

To the best of my knowledge, you got it exactly right:

• on classpath, service providers use META-INF/services entries
• on module-path, service providers use module-info.class

BTW. If you have control over those modular JARs, you can provide META-INF/services entries (in addition to the entries in module-info.java) so that this JAR works both on classpath and module-path. I'd suggest trying out Google's @AutoService annotation for this.

[...] I think this is fair game given the pitch of the project is "This Gradle plugin helps working with the Java Platform Module System".

Good point!

[paulbakker]

In most cases you would want non-modular JARs to become automatic modules I think, that way they can be used by other modules (e.g. your code that's already modular). This means it's not an option to place anything that doesn't have a module-info on the classpath.

I'm also still not completely sure what the use case is. @mtdowling you mention split packages, but patch-modules are the preferred solution for that. Is the remaining use case just service providers like @tlinkowski suggests? Are these JARS under your control? My guess is it's pretty rare to find these in the wild.

[nipafx]

Hi, @mtdowling lured me here from Twitter to comment on this issue, particularly his message from yesterday.

If I got this correctly, the initial motivation to develop a non-modular project comes from dependencies causing trouble on the module path. I find it preferable to only have JARs on the module path that are direct dependencies of another modular JAR on the module path, starting with the module under compilation. This approach reduces the number of potentially problematic JARs (i.e. plain JARs) on the module path to a minimum. Maven chose this approach for good reason. AFAIK, Gradle puts all dependencies on the module path and I consider that a shortcoming of the current implementation.

This means it's not an option to place anything that doesn't have a module-info on the classpath.

Maybe I got lost in translation, but I think I disagree. Not all plain JARs can go on the class path, but many can and as many as possible should.

In that light, creating a non-modular library against a mixture of module and class path really seems like a workaround to me. A hypothetical use case would be the requirement to use JPMS for its features (e.g. strong encapsulation) during development, but eventually ship a non-modular JAR, but that would be easier achieved by stripping out the module-info.class at an appropriate time (before integration tests?).

All of that said, regarding the service loading mechanism, @mtdowling and @tlinkowski got it right. In this example Jar A should really use both module-info and META-INF/services to define the same set of modules, so it works on module path and class path. If it only relies on module-info it effectively can't be used on the class path, which on today's level of JPMS adoption would be pretty suicidal. 😉

[mtdowling]

First, this thread has been very enlightening for me and hopefully others in the future. Thanks everyone for clearing things up and demystifying the module system for me.

I'm also still not completely sure what the use case is

At the same time, I need to use modular JARs that define services in a module-info.java and I need to use JARs that suffer from split packages. Placing everything on the module-path doesn't work nor does placing everything on the classpath. Updating the split packages and campaigning for tens of thousands of consumers to migrate will take far more effort than virtually any other option. Adding META-INF/services to modular JARs is a good strategy, but it's not a general purpose solution -- what if I needed to use a modular JAR that I don't own and can't add META-INF/services too?

META-INF/services

Creating META-INF/services for modular JARs is a brilliant idea! This should be standard practice. I'll do that to make sure my library can work on the classpath and will be compatible with tons of existing tools that only support the classpath (including things where the extension mechanism is to drop JARs into a libs/ folder).

While this seems like a best practice for modular library authors, it doesn't seem like a general purpose solution for tool vendors like Gradle and Maven:

1. Not all JARs will adhere to this.
2. It prevents modular library authors from using the provides static factory method to implement a service provider. This isn't supported on the classpath.
3. It requires library authors to repeat themselves: first by curating a module-info.java and then by adding things to the META-INF/services file or using @AutoService annotations.

Generating META-INF/services

Given how important it is to make JARs work on the classpath and how poorly supported in general JPMS is today (excluding this library!), I think it would be very useful for this library to have the option to generate META-INF/services files for each service provider detected in a module-info.java. If the library detects that META-INF/services already exist or that any of the service providers use the static provides method, then it would fail the build. I created an issue for this: #85

--patch-module isn't the answer to most problems

I don't think --patch-module should be encouraged for anything other than applications. It should never be used when developing a library that you expect others to depend on. The problem with --patch-module is that it requires every consumer of your package, including transitive consumers, to also patch your module. Why? --patch-module happens at build time to make a JAR compile. Once it's compiled, the need to patch is no longer associated with the JAR. (Please correct me if I'm missing something here and this isn't the case).

[paulbakker]

Just to be clear, when I said:

This means it's not an option to place anything that doesn't have a module-info on the classpath.

I meant we can't do this for all JARs that aren't a module. There needs to be an option to either put it on the module path, or on the classpath, so that the developer can make a choice if a JAR should become an automatic module or not. Any ideas how we could control/configure this choice?

[mtdowling]

@paulbakker I think for this to work generally, it would need to be automatic with support for opting out of trying to do the right thing by default. Could you take a similar approach to what I describe in https://gist.github.com/mtdowling/39905d445829ab1ac58d0cafecf6bc9e? Something like:

1. Any provided "roots" are always added to the module-path. These roots are useful, for example, to setup a test runner to be able to read from a module under test.
2. All sources that explicitly define a module-info.java are always added to the module-path. Modularized sources must be run from the module-path or they could fail to find their services.
3. Any dependency of a modularized JAR that is referenced through a "reads" or "opens" in their module-info.java is added to the module-path. This includes sources that use an Automatic-Module-Name and sources that use a derived module name based on the rules defined in ModuleFinder#find.
4. The recursive dependencies of all "roots" and modular JARs are added to the module-path.
5. All other sources are omitted from the module-path and should be used in the class-path.

[tlinkowski]

4. The recursive dependencies of all "roots" and modular JARs are added to the module-path.

I think it's worth remembering here that you don't need to put all recursive dependencies of a modular JAR on the module-path.

You only need to put all its immediate dependencies there because — as @nicolaiparlog writes:

[...] all dependencies of a modular JAR must be placed on the module path. Yes, even non-modular JARs, which will then get turned into automatic modules.

The interesting thing is that automatic modules can read the unnamed module, so their dependencies can go on the class path.

[chriskessel]

Adding a +1 for this ability. Following the recommendation in The Java Module System book, I'm making my project a module, but leaving all the 3rd party jars on the classpath. Then I'll work my way through my direct dependencies, but leaving all transitive dependencies on the classpath.

As it stands, if I'm understanding this thread, there's no way to do that with this plugin. That makes it hard to migrate in small steps.