Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

JDK23 Linux reproducible comparing tests failure #3920

Closed
sophia-guo opened this issue Aug 21, 2024 · 10 comments · Fixed by #3921
Closed

JDK23 Linux reproducible comparing tests failure #3920

sophia-guo opened this issue Aug 21, 2024 · 10 comments · Fixed by #3921
Assignees
@sophia-guo
Labels
jenkins Issues that enhance or fix our jenkins server testing Issues that enhance or fix our test suites

Comments

@sophia-guo
Copy link
Contributor

jdk23 linux reproducible comparing test failed with the diff https://ci.adoptium.net/job/Test_openjdk23_hs_special.system_x86-64_linux_testList_0/13/

        Total time: 28 seconds
        OpenJDK SBOM will be /home/jenkins/temurin-build/workspace/target/OpenJDK23U-sbom_x64_linux_hotspot_23_36-ea.json.
        Adding GLIBC version to SBOM: 2.17
        Adding GCC version to SBOM: 11.3.0
        BootJDK java : /usr/lib/jvm/jdk-22.0.2+9/bin/java
        Adding BOOTJDK to SBOM: 22.0.2+9
        Adding ALSA version to SBOM: 1.1.8
        FREETYPE_TO_USE=bundled
        Checking for FreeType include /home/jenkins/temurin-build/workspace/./build//src/src/java.desktop/share/native/libfreetype/include/freetype/freetype.h
        Found /home/jenkins/temurin-build/workspace/./build//src/src/java.desktop/share/native/libfreetype/include/freetype/freetype.h
        Checking and getting Tool Summary info:
        CycloneDX SBOM has been created in /home/jenkins/temurin-build/workspace/target/OpenJDK23U-sbom_x64_linux_hotspot_23_36-ea.json
        build.sh : 02:37:41 : All done!
        Comparing ...
        Only in /home/jenkins/jdkbinary/bin: jar.debuginfo
        Only in /home/jenkins/jdkbinary/bin: jarsigner.debuginfo
        Only in /home/jenkins/jdkbinary/bin: java.debuginfo
        Only in /home/jenkins/jdkbinary/bin: javac.debuginfo
        Only in /home/jenkins/jdkbinary/bin: javadoc.debuginfo
        Only in /home/jenkins/jdkbinary/bin: javap.debuginfo
        Only in /home/jenkins/jdkbinary/bin: jcmd.debuginfo
        Only in /home/jenkins/jdkbinary/bin: jconsole.debuginfo
        Only in /home/jenkins/jdkbinary/bin: jdb.debuginfo
        Only in /home/jenkins/jdkbinary/bin: jdeprscan.debuginfo
        Only in /home/jenkins/jdkbinary/bin: jdeps.debuginfo
        Only in /home/jenkins/jdkbinary/bin: jfr.debuginfo
        Only in /home/jenkins/jdkbinary/bin: jhsdb.debuginfo
        Only in /home/jenkins/jdkbinary/bin: jimage.debuginfo
        Only in /home/jenkins/jdkbinary/bin: jinfo.debuginfo
        Only in /home/jenkins/jdkbinary/bin: jlink.debuginfo
        Only in /home/jenkins/jdkbinary/bin: jmap.debuginfo
        Only in /home/jenkins/jdkbinary/bin: jmod.debuginfo
        Only in /home/jenkins/jdkbinary/bin: jpackage.debuginfo
        Only in /home/jenkins/jdkbinary/bin: jps.debuginfo
        Only in /home/jenkins/jdkbinary/bin: jrunscript.debuginfo
        Only in /home/jenkins/jdkbinary/bin: jshell.debuginfo
        Only in /home/jenkins/jdkbinary/bin: jstack.debuginfo
        Only in /home/jenkins/jdkbinary/bin: jstat.debuginfo
        Only in /home/jenkins/jdkbinary/bin: jstatd.debuginfo
        Only in /home/jenkins/jdkbinary/bin: jwebserver.debuginfo
        Only in /home/jenkins/jdkbinary/bin: keytool.debuginfo
        Only in /home/jenkins/jdkbinary/bin: rmiregistry.debuginfo
        Only in /home/jenkins/jdkbinary/bin: serialver.debuginfo
        Binary files /home/jenkins/jdkbinary/jmods/java.base.jmod and compare.1/jdk-23+36/jmods/java.base.jmod differ
        diff -r /home/jenkins/jdkbinary/lib/classlist compare.1/jdk-23+36/lib/classlist
        425d424
        < java/net/DefaultInterface
        428,430d426
        < java/net/Inet6Address
        < java/net/Inet6Address$Inet6AddressHolder
        < java/net/Inet6AddressImpl
        436,437d431
        < java/net/InterfaceAddress
        < java/net/NetworkInterface
        Only in /home/jenkins/jdkbinary/lib: jexec.debuginfo
        Only in /home/jenkins/jdkbinary/lib: jspawnhelper.debuginfo
        Only in /home/jenkins/jdkbinary/lib: libattach.debuginfo
        Only in /home/jenkins/jdkbinary/lib: libawt.debuginfo
        Only in /home/jenkins/jdkbinary/lib: libawt_headless.debuginfo
        Only in /home/jenkins/jdkbinary/lib: libawt_xawt.debuginfo
        Only in /home/jenkins/jdkbinary/lib: libdt_socket.debuginfo
        Only in /home/jenkins/jdkbinary/lib: libextnet.debuginfo
        Only in /home/jenkins/jdkbinary/lib: libfontmanager.debuginfo
        Only in /home/jenkins/jdkbinary/lib: libfreetype.debuginfo
        Only in /home/jenkins/jdkbinary/lib: libinstrument.debuginfo
        Only in /home/jenkins/jdkbinary/lib: libj2gss.debuginfo
        Only in /home/jenkins/jdkbinary/lib: libj2pcsc.debuginfo
        Only in /home/jenkins/jdkbinary/lib: libj2pkcs11.debuginfo
        Only in /home/jenkins/jdkbinary/lib: libjaas.debuginfo
        Only in /home/jenkins/jdkbinary/lib: libjava.debuginfo
        Only in /home/jenkins/jdkbinary/lib: libjavajpeg.debuginfo
        Only in /home/jenkins/jdkbinary/lib: libjawt.debuginfo
        Only in /home/jenkins/jdkbinary/lib: libjdwp.debuginfo
        Only in /home/jenkins/jdkbinary/lib: libjimage.debuginfo
        Only in /home/jenkins/jdkbinary/lib: libjli.debuginfo
        Only in /home/jenkins/jdkbinary/lib: libjsig.debuginfo
        Only in /home/jenkins/jdkbinary/lib: libjsound.debuginfo
        Only in /home/jenkins/jdkbinary/lib: libjsvml.debuginfo
        Only in /home/jenkins/jdkbinary/lib: liblcms.debuginfo
        Only in /home/jenkins/jdkbinary/lib: libmanagement.debuginfo
        Only in /home/jenkins/jdkbinary/lib: libmanagement_agent.debuginfo
        Only in /home/jenkins/jdkbinary/lib: libmanagement_ext.debuginfo
        Only in /home/jenkins/jdkbinary/lib: libmlib_image.debuginfo
        Only in /home/jenkins/jdkbinary/lib: libnet.debuginfo
        Only in /home/jenkins/jdkbinary/lib: libnio.debuginfo
        Only in /home/jenkins/jdkbinary/lib: libprefs.debuginfo
        Only in /home/jenkins/jdkbinary/lib: librmi.debuginfo
        Only in /home/jenkins/jdkbinary/lib: libsaproc.debuginfo
        Only in /home/jenkins/jdkbinary/lib: libsctp.debuginfo
        Only in /home/jenkins/jdkbinary/lib: libsimdsort.debuginfo
        Only in /home/jenkins/jdkbinary/lib: libsplashscreen.debuginfo
        Only in /home/jenkins/jdkbinary/lib: libsyslookup.debuginfo
        Only in /home/jenkins/jdkbinary/lib: libverify.debuginfo
        Only in /home/jenkins/jdkbinary/lib: libzip.debuginfo
        Binary files /home/jenkins/jdkbinary/lib/modules and compare.1/jdk-23+36/lib/modules differ
        Binary files /home/jenkins/jdkbinary/lib/server/classes.jsa and compare.1/jdk-23+36/lib/server/classes.jsa differ
        Binary files /home/jenkins/jdkbinary/lib/server/classes_nocoops.jsa and compare.1/jdk-23+36/lib/server/classes_nocoops.jsa differ
        Only in /home/jenkins/jdkbinary/lib/server: libjvm.debuginfo
        diff -r /home/jenkins/jdkbinary/release compare.1/jdk-23+36/release
        12c12
        < BUILD_SOURCE_REPO="https://github.com/adoptium/temurin-build.git"
        ---
        > BUILD_SOURCE_REPO="https://github.com/adoptium/temurin-build"
        Differences found..., logged in: reprotest.diff
        -----------------------------------
        Rebuild_Same_JDK_Reproducibility_Test_0_FAILED
@github-actions github-actions bot added jenkins Issues that enhance or fix our jenkins server testing Issues that enhance or fix our test suites labels Aug 21, 2024
@sophia-guo sophia-guo moved this to In Progress in 2024 3Q Adoptium Plan Aug 21, 2024
@sophia-guo sophia-guo mentioned this issue Aug 21, 2024
Merged
@sxa
Copy link
Member

sxa commented Aug 22, 2024

A few comments/questions:

  1. Is this specific to jdk23? I see that the JDK21 equivalents do not appear to have run the reproducibility test successfully recently, although they didn't get as far as running the build: https://ci.adoptium.net/job/Test_openjdk21_hs_special.system_x86-64_linux_testList_0/
  2. I'm surprised by the presence of the debuginfo files. They should not be present in the normal JDK tarball as we ship them separately. Is that expected with the current process?
  3. Do we have an equivalent run using the original _reproduce_compare jobs to compare with what's coming out from these new AQA jobs (for any JDK version)? I think it's important that we understand if it is the environment or a failure in the codebase which is causing the failures and I'm not sure we have that information at the moment.

@sxa
Copy link
Member

sxa commented Aug 22, 2024

FYI I got the same results running the jdk23 job on an Ubuntu 24.04 host in this job, which eliminates the older kernel version on the ubuntu 16.04 machine as the cause of the differences:

@sophia-guo
Copy link
Contributor Author

Is this specific to jdk23? I see that the JDK21 equivalents do not appear to have run the reproducibility test successfully recently, although they didn't get as far as running the build: https://ci.adoptium.net/job/Test_openjdk21_hs_special.system_x86-64_linux_testList_0/

Yes, this is for jdk23, jdk21 I haven't got the chance to rerun due to the centos OEL issue.

I'm surprised by the presence of the debuginfo files. They should not be present in the normal JDK tarball as we ship them separately. Is that expected with the current process?

I'm working on this #3921 to ignore the debuginfo

Do we have an equivalent run using the original _reproduce_compare jobs to compare with what's coming out from these new AQA jobs (for any JDK version)? I think it's important that we understand if it is the environment or a failure in the codebase which is causing the failures and I'm not sure we have that information at the moment.

Jobs are here https://ci.adoptium.net/view/ReproducibleBuild/, but we didn't enable for jdk22+, as suppose those jobs will be disabled. adoptium/ci-jenkins-pipelines#1093

@sxa
Copy link
Member

sxa commented Aug 22, 2024

Jobs are here https://ci.adoptium.net/view/ReproducibleBuild/, but we didn't enable for jdk22+, as suppose those jobs will be disabled. adoptium/ci-jenkins-pipelines#1093

Yeah unfortunately they only have up to jdk21u and you're currently using 23 in the new runs so the output isn't directly comparable (Also the jdk21u for xLinux seems to have been a bit broken on those for a while).

It might be worth trying a special.system run with this test against jdk21u/linux/aarch64 if that's feasible to see if it looks ok when compared with the green runs of https://ci.adoptium.net/view/ReproducibleBuild/job/jdk21u-linux-aarch64-temurin_reproduce_compare/

@andrew-m-leonard
Copy link
Contributor

@sophia-guo a couple of things, it seems the /home/jenkins/jdkbinary contains .debuginfo files, however the actual JDK being reproduced (https://ci.adoptium.net/job/build-scripts/job/jobs/job/jdk23/job/jdk23-linux-x64-temurin/19/artifact/workspace/target/OpenJDK23U-jdk_x64_linux_hotspot_23_37-ea.tar.gz) does not...? Which makes me think the /home/jenkins/jdkbinary is not what it is supposed to be?

I also notice at the end of the console, that other processes are running on the same machine?

21:47:09  PROCESSCATCH: Terminating any hung/left over test processes:
[Pipeline] sh
21:47:10  + aqa-tests/terminateTestProcesses.sh jenkins
21:47:10  Unix type machine..
21:47:10  Running on a Linux host
21:47:10  -                           23024 D ?        00:00:01 /home/jenkins/workspace/Test_openjdk11_hs_dev.openjdk_x86-64_linux_testList_0/jdkbinary/j2sdk-image/jdk-11.0.25+4/bin/../bin/java -cp /home/jenkins/workspace/Test_openjdk11_hs_dev.openjdk_x86-64_linux_testList_0/aqa-tests/TKG/output_17243123622521/hotspot_serviceability_0/work/classes/2/serviceability/sa/ClhsdbCDSCore.d:/home/jenkins/workspace/Test_openjdk11_hs_dev.openjdk_x86-64_linux_testList_0/aqa-tests/openjdk/openjdk-jdk/test/hotspot/jtreg/serviceability/sa:/home/jenkins/workspace/Test_openjdk11_hs_dev.openjdk_x86-64_linux_testList_0/aqa-tests/TKG/output_17243123622521/hotspot_serviceability_0/work/classes/2/test/lib:/home/jenkins/workspace/Test_openjdk11_hs_dev.openjdk_x86-64_linux_testList_0/aqa-tests/openjdk/openjdk-jdk/test/lib:/home/jenkins/workspace/Test_openjdk11_hs_dev.openjdk_x86-64_linux_testList_0/jvmtest/openjdk/jtreg/lib/javatest.jar:/home/jenkins/workspace/Test_openjdk11_hs_dev.openjdk_x86-64_linux_testList_0/jvmtest/openjdk/jtreg/lib/jtreg.jar -ea -esa -Xmx512m -Xmx512m -XX:+UnlockDiagnosticVMOptions -XX:SharedArchiveFile=ArchiveForClhsdbCDSCore.jsa -XX:+CreateCoredumpOnCrash -Xshare:auto -XX:+ProfileInterpreter --add-exports=java.base/jdk.internal.misc=ALL-UNNAMED -XX:-AlwaysPreTouch CrashApp

Could the /home/jenkins/jdkbinary folder be being populated from one of these other test jobs running on this host?
@steelhead31 This might imply https://ci.adoptium.net/computer/test%2Dibmcloud%2Dubuntu1604%2Dx64%2D1/ is a "Docker host" machine, which has other test "containers" running on it? or maybe this node should not be labelled as a ci.role.test node itself?

@steelhead31
Copy link
Contributor

@andrew-m-leonard it does appear to be a physical machine with docker installed, it doesn't host any static containers (in the way the dockerhost machines do ) but it does appear to create test containers to run tests dynamically. I wonder if we need a different way to identify machines suitable for reproducible build testing..?,

@sxa
Copy link
Member

sxa commented Sep 2, 2024 •
edited
Loading

There are a number of AQA tests (mostly in the external suite, but some have been added to the dev suite in recent months too) which need to run docker and spin up containers on the fly for running certain tests - and these reproducible build jobs are now in that categorty - they spin up a container with the correct verions of things to do the comparison build. The process you have identified will almost certainly have been leftover from a previous test run on the machine and is not indicative of multiple jobs attempting to run simultaneously on the machine.

It looks like things are working as designed (other than some of the dev test jobs not clearing up after themselves). Note that @sophia-guo has access to that machine for debugging some permissions issues on there at the moment: adoptium/infrastructure#3722

I wonder if we need a different way to identify machines suitable for reproducible build testing

Any machine that is tagged in a way that lets test use docker should be suitable for this, in the same way as they are for the dev and external ones referenced above - they all use the sw.tool.docker label to identify suitable ones.

@sophia-guo
Copy link
Contributor Author

it seems the /home/jenkins/jdkbinary contains .debuginfo files, however the actual JDK being reproduced (https://ci.adoptium.net/job/build-scripts/job/jobs/job/jdk23/job/jdk23-linux-x64-temurin/19/artifact/workspace/target/OpenJDK23U-jdk_x64_linux_hotspot_23_37-ea.tar.gz) does not...? Which makes me think the /home/jenkins/jdkbinary is not what it is supposed to be?

This is interesting. /home/jenkins/jdkbinary is the place of the untarred the jdk. The debuginfor comes from debugimage tar. Aqa-test job triggered by upstream will grab all available jar files and untar them including debugimage. https://github.com/adoptium/aqa-tests/blob/master/get.sh#L327-L343

The reason we didn't notice it before is when tests were enabled we use customized jdk ( jdk link and json link). Hence no this problem. Also our former reproducible jenkins jobs are setup to filter the unnecessary files and only grab the jdk and sbom files so no this issue either.

So if we don't care about the debuginfo at all https://adoptium.net/docs/reproducible-verification-builds/reproduce-linux-x64/ the easiest way to do is ignore the debuginfo if they are available. https://github.com/adoptium/temurin-build/pull/3921/files#diff-65e4993f80b74f0a4078e853b798cbf6770215b4c31ae1377aa2704eec01ae16R340-R342 @andrew-m-leonard thoughts?

As @sxa mentioned that linux reproducible comparison testing are under the category of special system so those leftover processes are other test jobs under same category, but reproducible comparing tests will by no mean be affected by them.

@andrew-m-leonard
Copy link
Contributor

So if we don't care about the debuginfo at all https://adoptium.net/docs/reproducible-verification-builds/reproduce-linux-x64/ the easiest way to do is ignore the debuginfo if they are available. https://github.com/adoptium/temurin-build/pull/3921/files#diff-65e4993f80b74f0a4078e853b798cbf6770215b4c31ae1377aa2704eec01ae16R340-R342 @andrew-m-leonard thoughts?

I don't think we should be changing the reproducible compare logic to cater for whatever extra stuff maybe put into the JDK dir, if get.sh adds other stuff in the future we're back to the same problem.
Can we specifically only extract the jdk.tar ? If not then I think just put the logic to remove any extra "aqa-test" stuff in https://github.com/adoptium/temurin-build/blob/master/test/system/reproducibleCompare/reproducible.mk ?

@andrew-m-leonard
Copy link
Contributor

@sophia-guo it looks like it is only the debugimage that gets "overlaid" currently, so lets just add the remove logic in reproducible.mk

18:05:31  get jdk binary...
18:05:31  --sdkdir is set to upstream. Therefore, skip download jdk binary
18:05:31  Uncompressing file: OpenJDK23U-jdk-sources_23_36-ea.tar.gz ...
18:05:36  Uncompressing file: OpenJDK23U-jdk_x64_linux_hotspot_23_36-ea.tar.gz ...
18:05:38  Uncompressing file: OpenJDK23U-jre_x64_linux_hotspot_23_36-ea.tar.gz ...
18:05:39  Uncompressing file: OpenJDK23U-static-libs-glibc_x64_linux_hotspot_23_36-ea.tar.gz ...
18:05:39  Uncompressing file: OpenJDK23U-testimage_x64_linux_hotspot_23_36-ea.tar.gz ...
18:05:42  Uncompressing OpenJDK23U-debugimage_x64_linux_hotspot_23_36-ea.tar.gz over ./j2sdk-image...

@andrew-m-leonard andrew-m-leonard removed their assignment Sep 5, 2024
@github-project-automation github-project-automation bot moved this from In Progress to Done in 2024 3Q Adoptium Plan Sep 6, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@sophia-guo sophia-guo

Labels
jenkins Issues that enhance or fix our jenkins server testing Issues that enhance or fix our test suites
Projects
No open projects
2024 3Q Adoptium Plan
Status: Done
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Update reproducible comparing on linux sophia-guo/openjdk-build
4 participants
@sxa @sophia-guo @andrew-m-leonard @steelhead31