KAFKA-8115: Reduce flakiness in Trogdor JsonRestServer shutdown (apac…

…he#12830)

The GRACEFUL_SHUTDOWN_TIMEOUT_MS for the Trogdor JsonRestServer is 100ms.
In heavily loaded CI environments, this timeout can be exceeded. When this happens,
it causes the jettyServer.stop() and jettyServer.destroy() calls to throw exceptions, which
prevents shutdownExecutor.shutdown() from running. This has the effect of causing the JsonRestServer::waitForShutdown method to block for 1 day, which exceeds the 120s
timeout on the CoordinatorTest (and any other test relying on MiniTrogdorCluster).

This change makes it such that the graceful shutdown timeout is less likely to be exceeded,
and when it is, the timeout does not cause the waitForShutdown method to block for much
longer than the graceful shutdown timeout.

Reviewers: Ismael Juma <[email protected]>

trogdor/src/main/java/org/apache/kafka/trogdor/rest/JsonRestServer.java

@@ -56,7 +56,7 @@
 public class JsonRestServer {
     private static final Logger log = LoggerFactory.getLogger(JsonRestServer.class);
 
-    private static final long GRACEFUL_SHUTDOWN_TIMEOUT_MS = 100;
+    private static final long GRACEFUL_SHUTDOWN_TIMEOUT_MS = 10 * 1000;
 
     private final ScheduledExecutorService shutdownExecutor;
 
@@ -142,9 +142,13 @@ public void beginShutdown() {
                 } catch (Exception e) {
                     log.error("Unable to stop REST server", e);
                 } finally {
-                    jettyServer.destroy();
+                    try {
+                        jettyServer.destroy();
+                    } catch (Exception e) {
+                        log.error("Unable to destroy REST server", e);
+                    }
+                    shutdownExecutor.shutdown();
                 }
-                shutdownExecutor.shutdown();
                 return null;
             });
         }