Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve the documentation for the Jetty Server Dump #12205

Open
wants to merge 7 commits into
base: jetty-12.0.x
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -281,4 +281,13 @@ public void beanRemoved(Container parent, Object child)
parent.addBean(younger);
// end::containerSiblings[]
}

public void dumpExample()
{
Server server = new Server();
// tag::dumpExample[]
server.setDumpAfterStart(true);
server.setDumpBeforeStop(true);
// end::dumpExample[]
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -15,101 +15,15 @@

To troubleshoot Jetty when used as a standalone server, there are two main tools: the Jetty Server Dump and enabling DEBUG level logging.

Jetty is based on components organized as a tree, with the `Server` instance at the root of the tree.

As explained in the xref:jmx/index.adoc[JMX section], these components can be exported as JMX MBeans and therefore be accessible from JMX Consoles such as Java Missions Control (JMC).

Being able to take a snapshot of the state of Jetty while it is running is the most useful information that can be attached when reporting an issue.
Such state includes:

* The thread pool configuration and its current state, including how many threads are in use, and their stack trace.
* The TLS configuration.
* The I/O configuration and its current state, including the ports Jetty listens to, how many connections are currently open, and he state of each connection, and the state of the request/response handling for each connection.
* The `Handler` structure and its configuration.
* The web applications deployed and their configurations, including the class loader information.

The prerequisite for troubleshooting is to enable JMX, so that Jetty -- possibly a production server -- can be accessed from a remote location to obtain the information exported via JMX, and possibly be able to reconfigure Jetty to solve the issue.
For troubleshooting, it can be helpful to enable JMX, so that Jetty -- possibly a production server -- can be accessed from a remote location to obtain the information exported via JMX, and possibly be able to reconfigure Jetty to solve the issue.

IMPORTANT: Make sure you read about how to secure the access to Jetty when using xref:jmx/index.adoc#remote[remote JMX].

[[dump]]
== Server Dump

The Jetty Server Dump is obtained by invoking, via JMX, the `Server.dump()` operation, as shown below using https://adoptium.net/jmc.html[Java Mission Control (JMC)]:

image::jmc-server-dump.png[]

Find the `Server` MBean in the MBean Tree, under `org.eclipse.jetty.server:type=server,id=0`.
Then click on the "Operations" tab, select the `dump()` operation, and then click the `Execute` button.
In the bottom panel you will see the result of the invocation, that you can copy into a text editor and save to your file system.

[CAUTION]
====
Taking a Jetty Server Dump is a relatively expensive operation, as it dumps the state of all connections (which can be thousands), and the state of all threads.

The result of the invocation may produce a large string, possibly few MiB, that may impact the server memory usage.

Furthermore, dumping the state of the I/O Jetty components takes a little CPU time off the handling of the actual I/O, possibly slowing it down temporarily.

While the slow-down caused by taking the Jetty Server Dump may be noticeable on highly loaded systems, it is typically a very small price to pay to obtain the information about the Jetty state that may be critical to the resolution of an issue.
====

[NOTE]
====
The format of the Jetty Server Dump output is subject to change at any time, as Jetty developers modify the Jetty code and decide to include more state, or remove state that is no longer relevant.

The Jetty Server Dump is organized in a tree whose structure is similar to the runtime Jetty component tree.

At the end of the dump output there is a legend that explains the type of tree node: whether it is a node that represent a _managed_ component, or an _array_ node (or a _map_ node) that represent some component state, etc.
====

[[dump-start-stop]]
=== Dump at Server Start/Stop

The `Server.dump()` operation may also be invoked just after the `Server` starts (to log the state of the freshly started server), and just before the `Server` stops (which may be useful to log the state of server that is not working properly).

You can temporarily enable the Jetty Server Dump at start time by overriding the `jetty.server.dumpAfterStart` property on the command line:

[source,subs=+quotes]
----
$ java -jar $JETTY_HOME/start.jar *jetty.server.dumpAfterStart=true*
----

To make this change persistent across server restarts, see the xref:modules/standard.adoc#server[`server` module] configuration for more information about how to configure the server to dump at start/stop time.

[[dump-detailed]]
=== Detailed ThreadPool Information

By default, the dump of the thread pool will only dump the topmost stack frame of each thread.
It is possible to configure the thread pool to dump the whole stack trace for each thread; while this may be a little more expensive, it provides complete information about the state of each thread, which may be important to diagnose the issue.

See the xref:modules/standard.adoc#threadpool[`threadpool` module] configuration for more information about how to configure the thread pool to dump detailed thread information.

Detailed thread pool information can also be turned on/off on-the-fly via JMX, by finding the `ThreadPool` MBean under `org.eclipse.jetty.util.thread:type=queuedthreadpool,id=0`, then selecting the `detailedDump` attribute and setting it to `true`. You can now perform the `Server.dump()` operation as explained above, and then set `detailedDump` back to `false`.

[[dump-example]]
=== Dump Example

Below you can find a simple example of a Jetty Server Dump, with annotations for the principal components:

[jetty.small%nowrap]
....
[jetty]
setupArgs=--add-modules=http
args=jetty.http.selectors=1 jetty.http.acceptors=1 jetty.threadPool.minThreads=4 jetty.server.dumpAfterStart=true
delete=^[0-9]\{4}
callouts= <$N>,Server@,= QueuedThreadPool,HandlerCollection@,= ServerConnector,ManagedSelector@,keys @,startJarLoader@,unmanaged
....
ifdef::run-jetty[]
<1> The `Server` instance at the root of the tree
<2> The thread pool component
<3> The root of the `Handler` structure
<4> The connector listening on port `8080` for the HTTP/1.1 protocol
<5> A selector component that manages connections
<6> The connections currently managed by the selector component
<7> The server `ClassLoader` and its classpath
<8> The legend for the dump nodes
endif::[]
The Jetty Server Dump is a useful tool for debugging the state of the server.
See the xref:programming-guide:troubleshooting/jetty-server-dump.adoc[Jetty Server Dump] section for detailed information.

[[logging]]
== Enabling DEBUG Logging
Expand All @@ -125,7 +39,7 @@ Enabling DEBUG level logging for `org.eclipse.jetty` is very, *very* expensive.
Your server could be slowed down to almost a halt, especially if it is under heavy load.
Furthermore, the log file could quickly fill up the entire filesystem (unless configured to roll over), so you want to be really careful using DEBUG logging.

For production servers, consider using the <<dump,Jetty Server Dump>> first, and enable DEBUG logging only as a last resort.
For production servers, consider using the xref:programming-guide:troubleshooting/jetty-server-dump.adoc[Jetty Server Dump] first, and enable DEBUG logging only as a last resort.
====

However, sometimes issues are such that only DEBUG logging can really tell what's going on in the system, and enabling DEBUG logging is your best chance to figure the issue out.
Expand Down
2 changes: 1 addition & 1 deletion documentation/jetty/modules/programming-guide/nav.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@
** xref:troubleshooting/logging.adoc[]
** xref:troubleshooting/thread-dump.adoc[]
** xref:troubleshooting/state-tracking.adoc[]
** xref:troubleshooting/component-dump.adoc[]
** xref:troubleshooting/jetty-server-dump.adoc[]
** xref:troubleshooting/debugging.adoc[]
* Migration Guides
** xref:migration/94-to-10.adoc[]
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ In server application the root of the component tree is a `Server` instance, whi

Having all the Jetty components in a tree is beneficial in a number of use cases.
It makes possible to register the components in the tree as xref:arch/jmx.adoc[JMX MBeans] so that a JMX console can look at the internal state of the components.
It also makes possible to xref:troubleshooting/component-dump.adoc[dump the component tree] (and therefore each component's internal state) to a log file or to the console for xref:troubleshooting/index.adoc[troubleshooting purposes].
It also makes possible to xref:troubleshooting/jetty-server-dump.adoc[dump the component tree] (and therefore each component's internal state) to a log file or to the console for xref:troubleshooting/index.adoc[troubleshooting purposes].
// TODO: add a section on Dumpable?

[[lifecycle]]
Expand Down Expand Up @@ -104,7 +104,7 @@ The component tree should be used for long-lived or medium-lived components such

It is not recommended adding to, and removing from, the component tree short-lived objects such as HTTP requests or TCP connections, for performance reasons.

If you need component tree features such as automatic xref:arch/jmx.adoc[export to JMX] or xref:troubleshooting/component-dump.adoc[dump capabilities] for short-lived objects, consider having a long-lived container in the component tree instead.
If you need component tree features such as automatic xref:arch/jmx.adoc[export to JMX] or xref:troubleshooting/jetty-server-dump.adoc[dump capabilities] for short-lived objects, consider having a long-lived container in the component tree instead.
You can make the long-lived container efficient at adding/removing the short-lived components using a data structure that is not part of the component tree, and make the long-lived container handle the JMX and dump features for the short-lived components.
====

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -271,4 +271,4 @@ include::code:example$src/main/java/org/eclipse/jetty/docs/programming/Architect

Despite the name, `VirtualThreadPool` does not pool virtual threads, but allows you to impose a limit on the maximum number of current virtual threads, in order to limit resource consumption.

Furthermore, you can configure it to track virtual threads so that a xref:troubleshooting/component-dump.adoc[Jetty component dump] will show all virtual threads, including those that are unmounted.
Furthermore, you can configure it to track virtual threads so that a xref:troubleshooting/jetty-server-dump.adoc[Jetty component dump] will show all virtual threads, including those that are unmounted.

This file was deleted.

Original file line number Diff line number Diff line change
@@ -0,0 +1,116 @@
//
// ========================================================================
// Copyright (c) 1995 Mort Bay Consulting Pty Ltd and others.
//
// This program and the accompanying materials are made available under the
// terms of the Eclipse Public License v. 2.0 which is available at
// https://www.eclipse.org/legal/epl-2.0, or the Apache License, Version 2.0
// which is available at https://www.apache.org/licenses/LICENSE-2.0.
//
// SPDX-License-Identifier: EPL-2.0 OR Apache-2.0
// ========================================================================
//

[[dump]]
= Jetty Server Dump

Jetty is based on components organized as a tree, with the `Server` instance at the root of the tree.
lachlan-roberts marked this conversation as resolved.
Show resolved Hide resolved

As explained in the xref:operations-guide:jmx/index.adoc[JMX section], these components can be exported as JMX MBeans and therefore be accessible from JMX Consoles such as Java Missions Control (JMC).

Being able to take a snapshot of the state of Jetty while it is running is the most useful information that can be attached when reporting an issue.
lachlan-roberts marked this conversation as resolved.
Show resolved Hide resolved
Such state includes:
lachlan-roberts marked this conversation as resolved.
Show resolved Hide resolved

* The thread pool configuration and its current state, including how many threads are in use, and their stack trace.
* The TLS configuration.
lachlan-roberts marked this conversation as resolved.
Show resolved Hide resolved
* The I/O configuration and its current state, including the ports Jetty listens to, how many connections are currently open, and he state of each connection, and the state of the request/response handling for each connection.
* The `Handler` structure and its configuration.
* The web applications deployed and their configurations, including the class loader information.

[[dump-start-stop]]
== Dump at Server Start/Stop

The `Server.dump()` operation may also be invoked just after the `Server` starts (to log the state of the freshly started server), and just before the `Server` stops (which may be useful to log the state of server that is not working properly).

You can temporarily enable the Jetty Server Dump at start time by overriding the `jetty.server.dumpAfterStart` property on the command line:

For embedded usage this can be used by calling the setters directly.
[,java,indent=0]
----
include::code:example$src/main/java/org/eclipse/jetty/docs/programming/ComponentDocs.java[tags=dumpExample]
----

Standalone Jetty uses two properties to control this behaviour which are referenced in `jetty.xml` to call these setters.
These properties are `jetty.server.dumpAfterStart` and `jetty.server.dumpBeforeStop`.

These can be temporarily enabled by supplying these properties as command line arguments,
or to make this change persistent across server restarts they can be enabled via the `server.ini` file (see xref:operations-guide:modules/standard.adoc#server-config[Server Configuration]).

[source,subs=+quotes]
----
$ java -jar $JETTY_HOME/start.jar *jetty.server.dumpAfterStart=true*
----

[NOTE]
====
The format of the Jetty Server Dump output is subject to change at any time, as Jetty developers modify the Jetty code and decide to include more state, or remove state that is no longer relevant.

The Jetty Server Dump is organized in a tree whose structure is similar to the runtime Jetty component tree.

At the end of the dump output there is a legend that explains the type of tree node: whether it is a node that represent a _managed_ component, or an _array_ node (or a _map_ node) that represent some component state, etc.
====

== Obtaining Dump via JMX

The Jetty Server Dump can be obtained by invoking, via JMX, the `Server.dump()` operation, as shown below using https://adoptium.net/jmc.html[Java Mission Control (JMC)]:

image::jmc-server-dump.png[]

Find the `Server` MBean in the MBean Tree, under `org.eclipse.jetty.server:type=server,id=0`.
Then click on the "Operations" tab, select the `dump()` operation, and then click the `Execute` button.
In the bottom panel you will see the result of the invocation, that you can copy into a text editor and save to your file system.

[CAUTION]
====
Taking a Jetty Server Dump is a relatively expensive operation, as it dumps the state of all connections (which can be thousands), and the state of all threads.

The result of the invocation may produce a large string, possibly few MiB, that may impact the server memory usage.

Furthermore, dumping the state of the I/O Jetty components takes a little CPU time off the handling of the actual I/O, possibly slowing it down temporarily.

While the slow-down caused by taking the Jetty Server Dump may be noticeable on highly loaded systems, it is typically a very small price to pay to obtain the information about the Jetty state that may be critical to the resolution of an issue.
====

[[dump-detailed]]
== Detailed ThreadPool Information

By default, the dump of the thread pool will only dump the topmost stack frame of each thread.
It is possible to configure the thread pool to dump the whole stack trace for each thread; while this may be a little more expensive, it provides complete information about the state of each thread, which may be important to diagnose the issue.

See the xref:operations-guide:modules/standard.adoc#threadpool[`threadpool` module] configuration for more information about how to configure the thread pool to dump detailed thread information.

Detailed thread pool information can also be turned on/off on-the-fly via JMX, by finding the `ThreadPool` MBean under `org.eclipse.jetty.util.thread:type=queuedthreadpool,id=0`, then selecting the `detailedDump` attribute and setting it to `true`. You can now perform the `Server.dump()` operation as explained above, and then set `detailedDump` back to `false`.

[[dump-example]]
== Dump Example

Below you can find a simple example of a Jetty Server Dump, with annotations for the principal components:

[jetty.small%nowrap]
....
[jetty]
setupArgs=--add-modules=http
args=jetty.http.selectors=1 jetty.http.acceptors=1 jetty.threadPool.minThreads=4 jetty.server.dumpAfterStart=true
delete=^[0-9]\{4}
callouts= <$N>,Server@,= QueuedThreadPool,HandlerCollection@,= ServerConnector,ManagedSelector@,keys @,startJarLoader@,unmanaged
....
ifdef::run-jetty[]
<1> The `Server` instance at the root of the tree
<2> The thread pool component
<3> The root of the `Handler` structure
<4> The connector listening on port `8080` for the HTTP/1.1 protocol
<5> A selector component that manages connections
<6> The connections currently managed by the selector component
<7> The server `ClassLoader` and its classpath
<8> The legend for the dump nodes
endif::[]
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@

`StateTrackingHandler` (described xref:server/http.adoc#handler-use-state-tracking[here]) is a troubleshooting `Handler` that can be inserted in the `Handler` chain to track usages of `Handler`/`Request`/`Response` asynchronous APIs.

xref:troubleshooting/component-dump.adoc[Dumping the Jetty component tree] will dump the `StateTrackingHandler`, which will dump the state of the current requests.
xref:troubleshooting/jetty-server-dump.adoc[Dumping the Jetty component tree] will dump the `StateTrackingHandler`, which will dump the state of the current requests.

This will help detecting whether requests are not completed due to callbacks not being completed, or whether callback code is stuck while invoking blocking APIs, etc.

Expand Down
Loading