Mac builds failing due to sign_installer Notarization "Not signed" errors #3500
Comments
github-actions
bot
added
aarch
|
The documentation link in the error states that: You can only notarize apps that you sign with a Developer ID certificate. If you use any other certificate — like a Mac App Distribution certificate, or a self-signed certificate — notarization fails with the following message: "path": "OpenJDK17U-jdk_aarch64_mac_hotspot_17.0.8.1_1-5618895206704016833.pkg/net.temurin.17.jdk.pkg Contents/Payload/Library/Java/JavaVirtualMachines/temurin-17.jdk/Contents/Home/jmods/jdk.jdwp.agent.jmod/lib/libdt_socket.dylib",\n "message": "The binary is not signed with a valid Developer ID certificate.",\n "docUrl": "https://developer.apple.com/documentation/security/notarizing_macos_software_before_distribution/resolving_common_notarization_issues#3087721\",\n "architecture": "arm64"\n },\n {\n "severity": "error",\n "code": null,\n "path": "OpenJDK17U-jdk_aarch64_mac_hotspot_17.0.8.1_1-5618895206704016833.pkg/net.temurin.17.jdk.pkg Contents/Payload/Library/Java/JavaVirtualMachines/temurin-17.jdk/Contents/Home/jmods/jdk.jdwp.agent.jmod/lib/libdt_socket.dylib",\n |
|
so it looks like 1 file in the archive is not signed and has no secure timestamp? Contents/Payload/Library/Java/JavaVirtualMachines/temurin-17.jdk/Contents/Home/jmods/jdk.jdwp.agent.jmod/lib/libdt_socket.dylib |
|
I've started a new nightly jdk21u build here: https://ci.adoptium.net/job/build-scripts/job/jobs/job/jdk21u/job/jdk21u-mac-aarch64-temurin/10/ |
|
From the build script I see that you first use the code signing api at https://cbi.eclipse.org/macos/codesign/sign to sign the file and then notarize the archive. Maybe something in the signing went wrong, inspecting the logs. Do we also have a trace if the file in question libdt_socket.dylib is actually being signed? |
|
In the error logs of the codesign service I dont find anything related to the libdt_socket.dylib file. The access logs dont indicate the file to be signed. |
|
This line selects all dylib with certain permissions to be setup for signing: could it be that this specific file ha different permissions and is thus not included? |
|
Here is the build log showing the signing of libdt_socket.dylib by that loop: |
|
I could see the request in the access log, and there was no error during signing. I could retrieve the full log when using notarytool to notarize the package: |
|
so its not only a single file but many, but all of them are inside jmods. Could it be that the dylibs that are packaged in the jmods have not been signed, and notarytool does now also check if they are signed as well, which it did not do before? |
|
I've unzipped the archive OpenJDK17U-jdk_aarch64_mac_hotspot_17.0.8.1_1.tar.gz: However ! The jdk-17.0.8.1+1/Contents/Home/jmods/jdk.jdwp.agent.jmod(libdt_socket.dylib) does not: |
That sounds like the jmods do not include the signed dylibs after signing. The currently running build has this set: 13:59:51 BUILD_CONFIG[ASSEMBLE_EXPLODED_IMAGE]="true" So from my naive understanding of the build where I see that the jmods are built quite early on, you would have to sign the dylibs that are used to create them before building the jmods, or rebuild them again later on after signing. |
|
I have a bit of a crazy idea, but I tested it and seemed to work fine. In the sign.sh script, we extract the archive and sign all encountered dylib files. To check if the archive contains all signed dylib's in the /Contents/Home/lib directory that are also contained in the various jmod files, I extracted all files from all jmods and compared the list. They are identical as far as I could see. Now my idea is to regenerate the jmod's by replacing the unsigned dylibs with the signed version. The standard jmod tool does not support that, but I had the idea to use proguard as it can read / write jmods. With a configuration like that (name replace.pro), I could replace any dylib in a jmod with the signed version: and then run That replaces the dylib' contained in a single jmod. One would dynamically create such a config file for all jmod files. I use that beta version of proguard as this is the first to add support for JDK 21. |
|
One could also quite easily write something in java to replace a file inside a jmod to avoid using proguard. Was using this to illustrate that it would be possible imho. |
|
hmm the jmods will need to be post-processed as the hashes change and the module-info.class inside them will not match anymore. Running jmod hash might help though. |
|
I have worked on some bytecode manipulation library that would be able to do that:
it would have to be smart to process all jmods in a directory in one go and determine dependencies between them as they depend on each other, and changing a jmod changes its hash ofc. But that could be doable. Link to the library: https://github.com/TinyGearsOrg/bat |
So yes this is possible. |
|
What is suspicious is, it seems the Mac x64 builds sign fine most of the time, it's only the Mac aarch64 that fail all the time... |
|
Do you have a trace of a x64 build that is getting signed and notarized fine so I can check the notarization log? |
So yes that's standard, openjdk has a utility called "jlink" for creating custom JREs which creates new bundles from the required jmods, and and dylib's in each |
|
I cant access the notarization build to retrieve the uuid. However, I downloaded the artifact that succeeded, extracted all files from the jmods and compared them to the dylib's in the Contents/Home/lib folder and the files are identical. This is different to the other build that failed, where the jmods contained slightly different versions of the same dylibs (apparently signed vs unsigned). |
|
|
the aarch64 build has 14:08:38 BUILD_CONFIG[MAKE_EXPLODED]="true" while the x86 has 14:06:28 BUILD_CONFIG[MAKE_EXPLODED]="false" that seems to change the build quite a lot from the logs. |
You're probably looking at the 2nd build within the log, if you do a find of the first instance of MAKE_EXPLODED in both they will be "true" |
|
I think this is the problem, as part of the 2nd build we are seeing : This did not used to occur. |
|
Could it be that aarch64 is built with a different Xcode (12.4) compared to x86 which has a different default settings to the code signing identity that is being used? I can see the following code in mac.sh to explicitly set a code sign identity in some cases. Maybe with older XCode versions the default was to not touch signatures, but with XCode 12.4 it overwrites them or removes them when no identity is set. Btw. I also checked older builds of aarch64 and the same message was there, it was probably not noticed as the previous notarization did not care. |
The point on "concurrent" codesigning from within the build, is due to the openjdk build being concurrent, typically 8 concurrent make jobs on our Mac nodes. |
|
To prevent the unnecessary debug codesigning, we'd need to specify the --without-macosx-codesign configure arg. |
|
Makes sense, I will look into signing jmod's post build. That is a topic that I have some experience with as I have worked on byte code processing which will be required to do as the module-info.class files inside the jmods contains the hashes of referenced module that will need to be updated accordingly. The jmod tool does not seem to support updating a jmod file, but you will have to update them taking into account cross-references between them which will require analysing the module contents anyways (module requires). Having a common way to do that signing for Windows / Macos is certainly a convincing argument. |
|
actually the jmod tool should support updating the hashes for all jmods taking dependencies into account. However, I tried used the way as described in the doc by running it in the jmod directory: tested that with different distributions and versions, always resulting in the same output. Testing with |
|
It is just java.base that needs re-hashing, eg: https://github.com/openjdk/jdk21u/blob/65941f8d31094511a5bf8dff110645f83cfc07ef/make/CreateJmods.gmk#L156 |
|
@netomi hmm, i'm a bit wary of "reproducibility" here in our re-construction of the JMOD, and we might introduce a regression if we don't re-create with jmod as per the build did..... Thoughts? |
|
Yeah that was also my thinking about supporting the in-build for windows, but that would mean that you would have to support a patch. Not sure how this would work in case of temurin, but maybe its already done like that in other cases. I am testing out the updating of the jmod and re-hashing it so you can take a look how this would look like. I got the jmod command now working with your input, ty. |
|
I worked on a script that can be used for testing purposes. If executed inside the respective jmods directory, it will analyse all jmods and replace all encountered dynamic library and executables with the version that is located outside (usually in the ../lib folder but can also be the ../bin folder for window builds). The script has been tested for linux (not that it makes sense there, just for making sure it would also work there), mac and windows. replace-signed-files-within-jmods.zip Edit: updated the script to only write the jmod if it has been updated. |
|
Output of the script for a macosx build is something like that: |
Looks good The other thing i'm going to try is see if we can "fix" the dual pass build issue, as I don't quite see why the need for make -t .. As I can't see why in theory you can't do the exploded make, sign "just" the dylib's, ensure "just" the dylib's timestamps are correct, and then make images. I don't see why gmake should need to rebuild the dylib's, as their dependent recipe targets won't have changed...I am suspecting it maybe because we "stash" the whole support directory, losing timestamp info maybe.... |
|
It's possible this condition dependent on gmake version >= 4.0 maybe related to why it functions differently |
|
@netomi I ran a little test with your script and from the reproducible build perspective it was fine. However, i'm rather nervous on assuming the jmod archive is "zip" format. It is currently due to the Java implementation of Jmod, but that can't be guaranteed, we really should use jmod extract, jmod create,... However, we would need to ensure using jmod with specify the correct extra parameters for each jmod (eg.--class-path, --man-pages, ...) I did some tests without the make -t, and trying to stash differently, but it does seem make triggers a dylib re-build. I don't like the way we currently relying on that not happening. |
|
My understanding is that the jmod is specified like that, however I understand your concern. All tools I know of that work with jmod do more or less the same, skip the first 4 bytes and then read a zip archive. This is a quick and dirty way to being able to update files inside the jmods for quick evaluation if this would be a viable way to go. Using jmod extract / create would be the best way to do it, need to fiddle around with it to get it working though. Initially I was concerned whether I have all knowledge of the parameters that jmod create might expect for creating specific jmods, but need to test this out. Edit: I remember in some older version of the relevant JEP it was mentioned that the content is basically a zip archive apart from the first 4 bytes. In the latest version I cant find any reference to it anymore: https://openjdk.org/jeps/261 So yeah, to be future-proof we should use the jmod tool itself to update files. |
what targets do you use for the second build? Maybe building with trace enable allows to identify why the dylibs are being rebuilt. |
Yeah I tried that and generated a 0.5Gb console file :-) |
|
so I managed to re-create a jmod after extracting it with this command: the only difference was in the module-info.class file, but it is just the order of the attributes (I verified with my dump tool):
its a bit annoying to have to specify all different options for all possible content of a jmod file, but the module-target was not needed (in fact if you specify it, the jmod tool will add an additional ModuleTarget attribute to the existing one) |
|
So thinking about the way we would update the jmod file, I think treating it as an archive and updating specific files is a more fail-safe method imho. If the jmod format changes, you would immediately know that it does not work anymore, the build fails and you would have to adapt. With the jmod create approach, if something changes wrt the content of a jmod, e.g. a new type of content is added, you would not notice a failure during the build, it would just be missing in the re-generated jmod. |
|
The other approach i'm thinking about is doing a new upstream contribution, to add a new openjdk configure option, something like: --with-signtool=<Path to custom signtool>. Which would then get invoked during the make process like the adhoc codesign is at the moment. |
|
I did not think about that, but I guess thats the way it should work. The downside of that approach would be that you would have to support older jdks in a different way as the one where this is integrated, but that would be acceptable imho. Edit: would be happy to help on that contribution. |
|
@andrew-m-leonard @netomi Is this still a problem, and do we need to move forward on a fix before the January release? There's a lot of stuff in this issue and I haven't digested all of it so it would be good to understand the current status of this and decide on a plan. |
|
The problem was triggered by a more recent version of gmake that highlighted some fundamental problem with the way the signing is happening. For the time being this seems to be resolved by reverting to an older version of gmake, but in the long term this will need to be addressed imho. |
|
@netomi So your summary is correct, for the short term the make version resolves, but ideally coming up with a better method would be advtangeous. However, I would not say it's a priority currently. |
https://ci.adoptium.net/job/build-scripts/job/release/job/sign_installer/9894/console
Examing the .dylib's show they look as though they are signed, but the Notarization service seems to not think so....
The text was updated successfully, but these errors were encountered: