Jmxterm hangs when executed in a loop (java.lang.Thread.State: WAITING (on object monitor))

[ITCBSOPS]

Hi !

Thank you for making Jmxterm. The tool is useful and awesome. I use it to collect metrics of running JVMs.
That being said, I want to mention an issue with Jmxterm 1.0.2 that I discovered yesterday.

When a Jmxterm command-script is executed many times in a loop, at some point, one of the instance of Jmxterm hangs, and waits indefenitely.

Here is how to reproduce the issue:

• launch manu instances of a java program with JMX enabled. I started 16 times our internally developped Java FIX Engine based on QuickFIX/J.
• make a list of the running JVMs (VMIDs), using jps. Or make a list of the PIDs.
• Using the list of VMIDs, generate a Jmxterm command script which will look like this :
  o open <jvmid#1> --> get -q -s -b java.lang:type=OperatingSystem ProcessCpuLoad  close
  o open <jvmid#2>  --> get -q -s -b java.lang:type=OperatingSystem ProcessCpuLoad  close
  o …
  o quit
• make a shell script loop that launches Jmxterm with the command-script again and again...
• command-script.txt | java -jar jmxterm-1.0.2-uber.jar -e -n -v verbose

After a few loops, the current instance of Jmxterm hangs.
In a separate shell, use the command jstack against the blocked Jmxterm instance. This displays the current thread stack.
Here is an example:

`2023-11-14 15:49:04
Full thread dump OpenJDK 64-Bit Server VM (25.392-b08 mixed mode):

"RMI Scheduler(0)" #13 daemon prio=5 os_prio=0 tid=0x0000027b80c03000 nid=0x49e0 waiting on condition [0x000000ecbc4ff000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000008017c8f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)

"GC Daemon" #12 daemon prio=2 os_prio=-2 tid=0x0000027b80dbf800 nid=0x1718 in Object.wait() [0x000000ecbc3fe000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x0000000080002d98> (a sun.misc.GC$LatencyLock)
at sun.misc.GC$Daemon.run(GC.java:117)
- locked <0x0000000080002d98> (a sun.misc.GC$LatencyLock)

"RMI RenewClean-[172.21.4.60:49993]" #11 daemon prio=5 os_prio=0 tid=0x0000027b805a6000 nid=0x5c1c in Object.wait() [0x000000ecbc2fe000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000800123d0> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
- locked <0x00000000800123d0> (a java.lang.ref.ReferenceQueue$Lock)
at sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCClient.java:563)
at java.lang.Thread.run(Thread.java:750)

"Service Thread" #9 daemon prio=9 os_prio=0 tid=0x0000027bff5fc000 nid=0x20fc runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

"C1 CompilerThread2" #8 daemon prio=9 os_prio=2 tid=0x0000027bff5c7000 nid=0x3654 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

"C2 CompilerThread1" #7 daemon prio=9 os_prio=2 tid=0x0000027bff5c1800 nid=0x15b4 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

"C2 CompilerThread0" #6 daemon prio=9 os_prio=2 tid=0x0000027bff5c6000 nid=0x8524 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

"Attach Listener" #5 daemon prio=5 os_prio=2 tid=0x0000027bff5b6800 nid=0x3a8c waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" #4 daemon prio=9 os_prio=2 tid=0x0000027bff5b5800 nid=0x1120 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

"Finalizer" #3 daemon prio=8 os_prio=1 tid=0x0000027bfee58000 nid=0x254 in Object.wait() [0x000000ecbbafe000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x000000008000a908> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
- locked <0x000000008000a908> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:188)

"Reference Handler" #2 daemon prio=10 os_prio=2 tid=0x0000027bfee51800 nid=0x5024 in Object.wait() [0x000000ecbb9ff000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x0000000080012ff0> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:502)
at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
- locked <0x0000000080012ff0> (a java.lang.ref.Reference$Lock)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)

"main" #1 prio=5 os_prio=0 tid=0x0000027bea014000 nid=0x1c08 runnable [0x000000ecbb3fe000]
java.lang.Thread.State: RUNNABLE
at sun.tools.attach.WindowsVirtualMachine.connectPipe(Native Method)
at sun.tools.attach.WindowsVirtualMachine.execute(WindowsVirtualMachine.java:101)
at sun.tools.attach.HotSpotVirtualMachine.executeCommand(HotSpotVirtualMachine.java:261)
at sun.tools.attach.HotSpotVirtualMachine.getAgentProperties(HotSpotVirtualMachine.java:156)
at sun.tools.jconsole.LocalVirtualMachine.getAttachableVMs(LocalVirtualMachine.java:174)
at sun.tools.jconsole.LocalVirtualMachine.getAllVirtualMachines(LocalVirtualMachine.java:125)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.cyclopsgroup.jmxterm.utils.WeakCastUtils$2.invoke(WeakCastUtils.java:136)
at com.sun.proxy.$Proxy5.getAllVirtualMachines(Unknown Source)
at org.cyclopsgroup.jmxterm.jdk6.Jdk6JavaProcessManager.get(Jdk6JavaProcessManager.java:28)
at org.cyclopsgroup.jmxterm.SyntaxUtils.getUrl(SyntaxUtils.java:41)
at org.cyclopsgroup.jmxterm.cmd.OpenCommand.execute(OpenCommand.java:64)
at org.cyclopsgroup.jmxterm.cc.CommandCenter.doExecute(CommandCenter.java:161)
at org.cyclopsgroup.jmxterm.cc.CommandCenter.doExecute(CommandCenter.java:134)
at org.cyclopsgroup.jmxterm.cc.CommandCenter.execute(CommandCenter.java:176)
at org.cyclopsgroup.jmxterm.boot.CliMain.execute(CliMain.java:149)
at org.cyclopsgroup.jmxterm.boot.CliMain.main(CliMain.java:41)

"VM Thread" os_prio=2 tid=0x0000027bff571800 nid=0x3af0 runnable

"GC task thread#0 (ParallelGC)" os_prio=0 tid=0x0000027bea02c000 nid=0x67b8 runnable

"GC task thread#1 (ParallelGC)" os_prio=0 tid=0x0000027bea02d000 nid=0x6084 runnable

"GC task thread#2 (ParallelGC)" os_prio=0 tid=0x0000027bea02e800 nid=0x89b0 runnable

"GC task thread#3 (ParallelGC)" os_prio=0 tid=0x0000027bea030000 nid=0x5034 runnable

"VM Periodic Task Thread" os_prio=2 tid=0x0000027bff5ff800 nid=0x2f30 waiting on condition

JNI global references: 259
`

[jiaqi]

Good observation. Jmxterm uses some unpublished Java libraries to attach to a running JVM with PID. It's very possible the library is not designed to scale. If anyone find a way or another tool to get around this problem please update this issue. Thanks.