-
-
Notifications
You must be signed in to change notification settings - Fork 75
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Graal Native Image Support #263
Comments
I don't have real opinion about Graal. Anyway, I'm also not against adding support for this as long as it does not create additional effort for every new dbus-java version (while releasing or for keeping the bunch of json files up-to-date and compatible with future Graal versions). The best solution would be if dbus-java uses the same approach to be Graal compatible as the developer who uses dbus-java itself. What I want to avoid is to have some special logic for e.g. populating One may consider to support multiple annotations. What I'm thinking of is something like: Either you annotate your class with a new annotation (e.g. I don't know which options we have, maybe we can also eliminate the need for If you have some 'experimental' branch to play with, you may share it. |
I won't try to sell you Graal :) Well, Graal Native Image anyway. "Graal" covers a lot more than AOT native compilation.
Indeed. Once written, there would be nothing stopping dbus-java itself using its own annotation processor to generate it's own meta-data.
That sound fine. I am fairly sure that annotation processors cannot find un-annotated interfaces. I'll look into though. If it can't, you'd have to parse Java source itself. Of course there are plenty of libraries out there to do so, but it just seems overkill.
Nothing ready yet, but it shouldn't take long. Watch this space. |
Diving into this a bit more, the requirement to have the annotation processor be used to generate the default Graal meta-data is what is the hard part of this task. Ideally, the following need to be true.
Right now, I can think of a few ways to solve this, none of them are great.
I'll keep thinking about this. The annotation processor itself is pretty much written. I just need to get this arrangement right. |
Interesting investigation and also great timing as I just begun to take a look at this. I have to admit that I'm also not happy with either solution you mentioned. Also providing a new annotation will always require existing code to be changed to get compatible with the Graal stuff. Anyway. As already said, I took some deeper look into this and into the idea of "finding every interface which extends DBusInterface". I then googled around a bit and found a library called "spoon". Spoon can parse Java source and create AST which can be analyzed. It also allows manipulating the code, but that is nothing I want to do. <dependency>
<groupId>fr.inria.gforge.spoon</groupId>
<artifactId>spoon-core</artifactId>
<version>11.0.0</version>
</dependency> package com.github.hypfvieh.dbus.graal;
import spoon.Launcher;
import spoon.SpoonAPI;
import spoon.reflect.declaration.CtType;
import spoon.reflect.visitor.filter.AbstractFilter;
import java.util.LinkedHashSet;
import java.util.Set;
public final class CreateGraalConfig {
private static final Set<String> CANDIDATES = Set.of(
"org.freedesktop.dbus.interfaces.DBusInterface",
"org.freedesktop.dbus.Container",
"org.freedesktop.dbus.interfaces.DBusSigHandler");
private CreateGraalConfig() {}
public static void main(String[] _args) {
SpoonAPI spoon = new Launcher();
spoon.addInputResource("../dbus-java-examples/src/main/java/");
spoon.getEnvironment().setComplianceLevel(17);
spoon.buildModel();
Set<String> found = new LinkedHashSet<>();
spoon.getModel().getElements(new AbstractFilter<CtType<?>>() {
@Override
public boolean matches(CtType<?> _element) {
if (_element.getSuperInterfaces().stream()
.anyMatch(e -> CANDIDATES.contains(e.getQualifiedName()))) {
found.add(_element.getQualifiedName());
return true;
}
return false;
};
});
System.out.println(found);
}
} This code utilizes spoon to find all classes and interfaces extending one of the interfaces found in "CANDIDATES". My idea is to use this to find all we need to create proper JSON files. The only problem I see right now is the dependency between the new module (dbus-java-graal) and dbus-java-core. If there is a dependency (e.g. to include all required interface names as classes instead of strings to be refactoring safe) we have the same circular issue like you stated above. Dbus-java-graal would require dbus-java-core and vice versa. What do you think? |
Oh ok, nice. It would certainly be nice if no annotation was needed at all. But yes, as you say, as soon as you use real The number actually used by dbus-java itself is quite small. If the |
One option would be to stick to those String constants, so no circular dependency would be needed. Another idea I just got, is to use maven-enforcer-plugin:
It may be hacky somehow, but will ensure that no refactoring will break anything because the Maven build will fail. Maybe you can use this as starting point. I know that you can also query the spoon-results to get method names, constructors and parameters for those. So I guess all information required for the Graal config files are already available. |
Yup, that looks like it will do that job. I'll take your branch and add my Json generating code and see what happens. |
Just a little progress report ....
As a bonus, I added a Maven profile to
to create this standalone executable. Spoon has quite a few dependencies, and this is a nice way to bundle them all up. Then a user just needs to GitHub actions allows for Graal compilation on open source projects, so this tool could be automatically built and released. (although I doubt it will change much). I have zero idea how to do this though, I just know it's possible. There is still a little to do.
|
Sounds good so far. I don't think I will add a automatic Graal build. When someone needs that feature, they should use the maven plugin or run the util using java or maven. I'll await your PR as soon as you are sure that everything "fits". |
@brett-smith: any update on this topic? Did you get a proper working solution for the Graal stuff? |
This issue is to discuss possible support for Graal Native Image, and how that might be achieved.
Of late, I have been using Graal Native Image more and more, often along with dbus-java, particularly on Linux. For example, if you add PicoCLI it makes Java a great language for writing DBus based command line utilities that are easy to distribute, fast to start up and use a lot less memory. You can even totally statically link (e.g. with libc or musl). Who'd have thought it.
The main challenge to using native image, is providing the reflection (and other) meta-data to help Graal in it's tasks of examining every possible code path. This has always been a bit painful, but has improved of late with things such as the meta-data repository and better tools to automate this. Lots of libraries now either include Graal meta-data, or there are rebuilds of libraries that have had meta-data added (Quarkus etc).
Getting good Graal support in dbus-java itself is going to involved a few different tasks. Its mainly about some resource files, and a new tool.
Checking Dependencies
To have good support, all 3rd party dependencies will need to be checked if they require meta-data, and if they provide it. Either themselves or via an external means. I know JNA has it, other dependencies (in transports I think primarily) will need to be checked. The pure Java (17) one I don't think requires any additional meta-data.
Library Meta-data
Then there is the library itself. Adding meta-data to the project is done by adding
.json
files tosrc/main/resources/META-INF/native-image/[xxxxxx]/yyyyy.json
, wherexxxxxx
is a project name (e.g.dbus-java
) andyyyyyy
is the one of the classes of meta-data.You can use the native image tracing agent, to generate the meta-data at run-time, but this can generate a lot of platform specific cruft that often is not required. It is very helpful though, and we've found fine in production if a brute-force approach is acceptable.
The classes of meta-data are ..
jni-config.json
- contains data about JNI methods called and libraries usedpredefine-classes-config.json
- not sure about this one, i've not yet seen it populatedproxy-config.json
- data aboutProxy
implementations (dbus-java uses this one)resource-config.json
- data about classpath resources loaded. There are sometimes surprising things loaded as resources.relect-config.json
- data about things that need to be reflectable (dbus-java uses this one)serialization-config.json
- data aboutSerializable
usagesSo far, I've only discovered two types of meta-data that are needed for the basic library.
proxy-config.json
andreflect-config.json
. That is not to say there are not others, but this is all i've needed so far.proxy-config.json
reflect-config.json
Annotation Processor or Code Generator
Lastly, it would be great if dbus-java also provided the tools for users to generate meta-data for their own
DBusInterface
implementations, and other DBus structures.For example,
DBusInterface
implementations must be both declared inproxy-config.json
and must be fully reflectable for dbus-java to function.Struct
must also be fully reflectable.This could be done in
InterfaceCodeGenerator
, with it generating Graal meta-data along with annotated Java source, but I feel this would be better solved using a separate Annotation Processor, applied at a later time. This means hand-written dbus-java interfaces and structures would also be handled.This is how PicoCLI solves Graal integration, by looking for it's own annotations and generating the appropriate json resources at build time. Annotation processors are a Java feature, and tools like Maven and others have well documented features for enabling processors.
However, an "Annotation Processor" is exactly that. It finds annotations. In dbus-java, annotations for example on a
DBusInterface
are optional. For this reason, it may be better to have a new annotation, e.g.@DBusNative
that tags either aDBusInterface
orStruct
as a candidate for generation of Graal meta-data.Risks
None really. Any changes should be entirely non-invasive, and not require any particular version of Java. There would only be a single new annotation.
Some transport providers may require either additional meta-data.
Graal native image developers should be by now well used to tweaking or providing additional meta-data while everybody catches up.
Conclusion
Graal native image really is awesome. It is going to be a significant part of Javas future, so it would be great to get 1st class support here.
If the idea sounds good to you, I will at some point get a PR together. I have most of the parts to achieve all of the above, it just needs bringing together in dbus-java itself.
The text was updated successfully, but these errors were encountered: