Skip to content
This repository has been archived by the owner on Jan 8, 2019. It is now read-only.

Nodes page not showing JVM usage for one of the nodes #1673

Open
evanccnyc opened this issue Nov 5, 2015 · 12 comments
Open

Nodes page not showing JVM usage for one of the nodes #1673

evanccnyc opened this issue Nov 5, 2015 · 12 comments

Comments

@evanccnyc
Copy link

Graylog: 1.2.2

On a brand new setup on the nodes page one of the nodes doesnt show JVM usage like the other one does.

If I click on the actual node, I see the data however.

Screenshot attached:

screen shot 2015-11-05 at 11 27 08 am
@evanccnyc
Copy link
Author

After posting this I got an error stacktrace posted below:

! @6o37bk041 - Internal server error, for (GET) [/system/nodes] ->

play.api.Application$$anon$1: Execution exception[[NullPointerException: null]]
    at play.api.Application$class.handleError(Application.scala:296) ~[com.typesafe.play.play_2.10-2.3.10.jar:2.3.10]
    at play.api.DefaultApplication.handleError(Application.scala:402) [com.typesafe.play.play_2.10-2.3.10.jar:2.3.10]
    at play.core.server.netty.PlayDefaultUpstreamHandler$$anonfun$3$$anonfun$applyOrElse$4.apply(PlayDefaultUpstreamHandler.scala:320) [com.typesafe.play.play_2.10-2.3.10.jar:2.3.10]
    at play.core.server.netty.PlayDefaultUpstreamHandler$$anonfun$3$$anonfun$applyOrElse$4.apply(PlayDefaultUpstreamHandler.scala:320) [com.typesafe.play.play_2.10-2.3.10.jar:2.3.10]
    at scala.Option.map(Option.scala:145) [org.scala-lang.scala-library-2.10.4.jar:na]
    at play.core.server.netty.PlayDefaultUpstreamHandler$$anonfun$3.applyOrElse(PlayDefaultUpstreamHandler.scala:320) [com.typesafe.play.play_2.10-2.3.10.jar:2.3.10]
    at play.core.server.netty.PlayDefaultUpstreamHandler$$anonfun$3.applyOrElse(PlayDefaultUpstreamHandler.scala:316) [com.typesafe.play.play_2.10-2.3.10.jar:2.3.10]
    at scala.concurrent.Future$$anonfun$recoverWith$1.apply(Future.scala:344) [org.scala-lang.scala-library-2.10.4.jar:na]
    at scala.concurrent.Future$$anonfun$recoverWith$1.apply(Future.scala:343) [org.scala-lang.scala-library-2.10.4.jar:na]
    at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32) [org.scala-lang.scala-library-2.10.4.jar:na]
    at play.api.libs.iteratee.Execution$trampoline$.execute(Execution.scala:46) [com.typesafe.play.play-iteratees_2.10-2.3.10.jar:2.3.10]
    at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:40) [org.scala-lang.scala-library-2.10.4.jar:na]
    at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:248) [org.scala-lang.scala-library-2.10.4.jar:na]
    at scala.concurrent.Promise$class.complete(Promise.scala:55) [org.scala-lang.scala-library-2.10.4.jar:na]
    at scala.concurrent.impl.Promise$DefaultPromise.complete(Promise.scala:153) [org.scala-lang.scala-library-2.10.4.jar:na]
    at scala.concurrent.Future$$anonfun$map$1.apply(Future.scala:235) [org.scala-lang.scala-library-2.10.4.jar:na]
    at scala.concurrent.Future$$anonfun$map$1.apply(Future.scala:235) [org.scala-lang.scala-library-2.10.4.jar:na]
    at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32) [org.scala-lang.scala-library-2.10.4.jar:na]
    at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.processBatch$1(BatchingExecutor.scala:67) [com.typesafe.akka.akka-actor_2.10-2.3.5.jar:na]
    at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:82) [com.typesafe.akka.akka-actor_2.10-2.3.5.jar:na]
    at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59) [com.typesafe.akka.akka-actor_2.10-2.3.5.jar:na]
    at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59) [com.typesafe.akka.akka-actor_2.10-2.3.5.jar:na]
    at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72) [org.scala-lang.scala-library-2.10.4.jar:na]
    at akka.dispatch.BatchingExecutor$Batch.run(BatchingExecutor.scala:58) [com.typesafe.akka.akka-actor_2.10-2.3.5.jar:na]
    at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:41) [com.typesafe.akka.akka-actor_2.10-2.3.5.jar:na]
    at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393) [com.typesafe.akka.akka-actor_2.10-2.3.5.jar:na]
    at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) [org.scala-lang.scala-library-2.10.4.jar:na]
    at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) [org.scala-lang.scala-library-2.10.4.jar:na]
    at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) [org.scala-lang.scala-library-2.10.4.jar:na]
    at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) [org.scala-lang.scala-library-2.10.4.jar:na]
Caused by: java.lang.NullPointerException: null
    at views.html.partials.node_title_link$.apply(node_title_link.template.scala:31) ~[graylog-web-interface.graylog-web-interface-1.2.2.jar:1.2.2]
    at views.html.system.nodes.index$$anonfun$1.apply(index.template.scala:73) ~[graylog-web-interface.graylog-web-interface-1.2.2.jar:1.2.2]
    at views.html.system.nodes.index$$anonfun$1.apply(index.template.scala:68) ~[graylog-web-interface.graylog-web-interface-1.2.2.jar:1.2.2]
    at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) ~[org.scala-lang.scala-library-2.10.4.jar:na]
    at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) ~[org.scala-lang.scala-library-2.10.4.jar:na]
    at scala.collection.Iterator$class.foreach(Iterator.scala:727) ~[org.scala-lang.scala-library-2.10.4.jar:na]
    at scala.collection.AbstractIterator.foreach(Iterator.scala:1157) ~[org.scala-lang.scala-library-2.10.4.jar:na]
    at scala.collection.IterableLike$class.foreach(IterableLike.scala:72) ~[org.scala-lang.scala-library-2.10.4.jar:na]
    at scala.collection.AbstractIterable.foreach(Iterable.scala:54) ~[org.scala-lang.scala-library-2.10.4.jar:na]
    at scala.collection.TraversableLike$class.map(TraversableLike.scala:244) ~[org.scala-lang.scala-library-2.10.4.jar:na]
    at scala.collection.AbstractTraversable.map(Traversable.scala:105) ~[org.scala-lang.scala-library-2.10.4.jar:na]
    at views.html.system.nodes.index$.apply(index.template.scala:68) ~[graylog-web-interface.graylog-web-interface-1.2.2.jar:1.2.2]
    at views.html.system.nodes.index$.render(index.template.scala:319) ~[graylog-web-interface.graylog-web-interface-1.2.2.jar:1.2.2]
    at views.html.system.nodes.index.render(index.template.scala) ~[graylog-web-interface.graylog-web-interface-1.2.2.jar:1.2.2]
    at controllers.NodesController.nodes(NodesController.java:92) ~[graylog-web-interface.graylog-web-interface-1.2.2.jar:1.2.2]
    at Routes$$anonfun$routes$1$$anonfun$applyOrElse$27$$anonfun$apply$515.apply(routes_routing.scala:1805) ~[graylog-web-interface.graylog-web-interface-1.2.2.jar:na]
    at Routes$$anonfun$routes$1$$anonfun$applyOrElse$27$$anonfun$apply$515.apply(routes_routing.scala:1805) ~[graylog-web-interface.graylog-web-interface-1.2.2.jar:na]
    at play.core.Router$HandlerInvokerFactory$$anon$4.resultCall(Router.scala:264) ~[com.typesafe.play.play_2.10-2.3.10.jar:2.3.10]
    at play.core.Router$HandlerInvokerFactory$JavaActionInvokerFactory$$anon$15$$anon$1.invocation(Router.scala:255) ~[com.typesafe.play.play_2.10-2.3.10.jar:2.3.10]
    at play.core.j.JavaAction$$anon$1.call(JavaAction.scala:55) ~[com.typesafe.play.play_2.10-2.3.10.jar:2.3.10]
    at play.GlobalSettings$1.call(GlobalSettings.java:67) ~[com.typesafe.play.play_2.10-2.3.10.jar:2.3.10]
    at play.mvc.Security$AuthenticatedAction.call(Security.java:44) ~[com.typesafe.play.play_2.10-2.3.10.jar:2.3.10]
    at play.core.j.JavaAction$$anonfun$11.apply(JavaAction.scala:82) ~[com.typesafe.play.play_2.10-2.3.10.jar:2.3.10]
    at play.core.j.JavaAction$$anonfun$11.apply(JavaAction.scala:82) ~[com.typesafe.play.play_2.10-2.3.10.jar:2.3.10]
    at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24) ~[org.scala-lang.scala-library-2.10.4.jar:na]
    at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24) ~[org.scala-lang.scala-library-2.10.4.jar:na]
    at play.core.j.HttpExecutionContext$$anon$2.run(HttpExecutionContext.scala:40) ~[com.typesafe.play.play_2.10-2.3.10.jar:2.3.10]
    at play.api.libs.iteratee.Execution$trampoline$.execute(Execution.scala:46) [com.typesafe.play.play-iteratees_2.10-2.3.10.jar:2.3.10]
    at play.core.j.HttpExecutionContext.execute(HttpExecutionContext.scala:32) ~[com.typesafe.play.play_2.10-2.3.10.jar:2.3.10]
    at scala.concurrent.impl.Future$.apply(Future.scala:31) ~[org.scala-lang.scala-library-2.10.4.jar:na]
    at scala.concurrent.Future$.apply(Future.scala:485) ~[org.scala-lang.scala-library-2.10.4.jar:na]
    at play.core.j.JavaAction$class.apply(JavaAction.scala:82) ~[com.typesafe.play.play_2.10-2.3.10.jar:2.3.10]
    at play.core.Router$HandlerInvokerFactory$JavaActionInvokerFactory$$anon$15$$anon$1.apply(Router.scala:252) ~[com.typesafe.play.play_2.10-2.3.10.jar:2.3.10]
    at play.api.mvc.Action$$anonfun$apply$1$$anonfun$apply$4$$anonfun$apply$5.apply(Action.scala:130) ~[com.typesafe.play.play_2.10-2.3.10.jar:2.3.10]
    at play.api.mvc.Action$$anonfun$apply$1$$anonfun$apply$4$$anonfun$apply$5.apply(Action.scala:130) ~[com.typesafe.play.play_2.10-2.3.10.jar:2.3.10]
    at play.utils.Threads$.withContextClassLoader(Threads.scala:21) ~[com.typesafe.play.play_2.10-2.3.10.jar:2.3.10]
    at play.api.mvc.Action$$anonfun$apply$1$$anonfun$apply$4.apply(Action.scala:129) ~[com.typesafe.play.play_2.10-2.3.10.jar:2.3.10]
    at play.api.mvc.Action$$anonfun$apply$1$$anonfun$apply$4.apply(Action.scala:128) ~[com.typesafe.play.play_2.10-2.3.10.jar:2.3.10]
    at scala.Option.map(Option.scala:145) [org.scala-lang.scala-library-2.10.4.jar:na]
    at play.api.mvc.Action$$anonfun$apply$1.apply(Action.scala:128) ~[com.typesafe.play.play_2.10-2.3.10.jar:2.3.10]
    at play.api.mvc.Action$$anonfun$apply$1.apply(Action.scala:121) ~[com.typesafe.play.play_2.10-2.3.10.jar:2.3.10]
    at play.api.libs.iteratee.Iteratee$$anonfun$mapM$1.apply(Iteratee.scala:483) ~[com.typesafe.play.play-iteratees_2.10-2.3.10.jar:2.3.10]
    at play.api.libs.iteratee.Iteratee$$anonfun$mapM$1.apply(Iteratee.scala:483) ~[com.typesafe.play.play-iteratees_2.10-2.3.10.jar:2.3.10]
    at play.api.libs.iteratee.Iteratee$$anonfun$flatMapM$1.apply(Iteratee.scala:519) ~[com.typesafe.play.play-iteratees_2.10-2.3.10.jar:2.3.10]
    at play.api.libs.iteratee.Iteratee$$anonfun$flatMapM$1.apply(Iteratee.scala:519) ~[com.typesafe.play.play-iteratees_2.10-2.3.10.jar:2.3.10]
    at play.api.libs.iteratee.Iteratee$$anonfun$flatMap$1$$anonfun$apply$14.apply(Iteratee.scala:496) ~[com.typesafe.play.play-iteratees_2.10-2.3.10.jar:2.3.10]
    at play.api.libs.iteratee.Iteratee$$anonfun$flatMap$1$$anonfun$apply$14.apply(Iteratee.scala:496) ~[com.typesafe.play.play-iteratees_2.10-2.3.10.jar:2.3.10]
    at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24) ~[org.scala-lang.scala-library-2.10.4.jar:na]
    at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24) ~[org.scala-lang.scala-library-2.10.4.jar:na]
    ... 6 common frames omitted

@bernd bernd self-assigned this Nov 9, 2015
@bernd
Copy link
Contributor

bernd commented Nov 9, 2015

@evanccnyc Do you get that every time? I am unable to reproduce this with two Graylog server nodes.

@evanccnyc
Copy link
Author

Not everytime. I had to do a reinstall for unrelated reasons and it seems to have gone away.

@bernd
Copy link
Contributor

bernd commented Nov 11, 2015

Thank you for the feedback! I am closing this now. Please re-open or create a new ticket if you run into this again. Thank you!

@bernd bernd closed this as completed Nov 11, 2015
@evanccnyc
Copy link
Author

This has reappeared.

@evanccnyc
Copy link
Author

Log from web interface

! @6o57k2c5f - Internal server error, for (GET) [/system/nodes] ->

play.api.Application$$anon$1: Execution exception[[NullPointerException: null]]
        at play.api.Application$class.handleError(Application.scala:296) ~[com.typesafe.play.play_2.10-2.3.10.jar:2.3.10]
        at play.api.DefaultApplication.handleError(Application.scala:402) [com.typesafe.play.play_2.10-2.3.10.jar:2.3.10]
        at play.core.server.netty.PlayDefaultUpstreamHandler$$anonfun$3$$anonfun$applyOrElse$4.apply(PlayDefaultUpstreamHandler.scala:320) [com.typesafe.play.play_2.10-2.3.10.jar:2.3.10]
        at play.core.server.netty.PlayDefaultUpstreamHandler$$anonfun$3$$anonfun$applyOrElse$4.apply(PlayDefaultUpstreamHandler.scala:320) [com.typesafe.play.play_2.10-2.3.10.jar:2.3.10]
        at scala.Option.map(Option.scala:145) [org.scala-lang.scala-library-2.10.4.jar:na]
        at play.core.server.netty.PlayDefaultUpstreamHandler$$anonfun$3.applyOrElse(PlayDefaultUpstreamHandler.scala:320) [com.typesafe.play.play_2.10-2.3.10.jar:2.3.10]
        at play.core.server.netty.PlayDefaultUpstreamHandler$$anonfun$3.applyOrElse(PlayDefaultUpstreamHandler.scala:316) [com.typesafe.play.play_2.10-2.3.10.jar:2.3.10]
        at scala.concurrent.Future$$anonfun$recoverWith$1.apply(Future.scala:344) [org.scala-lang.scala-library-2.10.4.jar:na]
        at scala.concurrent.Future$$anonfun$recoverWith$1.apply(Future.scala:343) [org.scala-lang.scala-library-2.10.4.jar:na]
        at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32) [org.scala-lang.scala-library-2.10.4.jar:na]
        at play.api.libs.iteratee.Execution$trampoline$.execute(Execution.scala:46) [com.typesafe.play.play-iteratees_2.10-2.3.10.jar:2.3.10]
        at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:40) [org.scala-lang.scala-library-2.10.4.jar:na]
        at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:248) [org.scala-lang.scala-library-2.10.4.jar:na]
        at scala.concurrent.Promise$class.complete(Promise.scala:55) [org.scala-lang.scala-library-2.10.4.jar:na]
        at scala.concurrent.impl.Promise$DefaultPromise.complete(Promise.scala:153) [org.scala-lang.scala-library-2.10.4.jar:na]
        at scala.concurrent.Future$$anonfun$map$1.apply(Future.scala:235) [org.scala-lang.scala-library-2.10.4.jar:na]
        at scala.concurrent.Future$$anonfun$map$1.apply(Future.scala:235) [org.scala-lang.scala-library-2.10.4.jar:na]
        at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32) [org.scala-lang.scala-library-2.10.4.jar:na]
        at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.processBatch$1(BatchingExecutor.scala:67) [com.typesafe.akka.akka-actor_2.10-2.3.5.jar:na]
        at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:82) [com.typesafe.akka.akka-actor_2.10-2.3.5.jar:na]
        at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59) [com.typesafe.akka.akka-actor_2.10-2.3.5.jar:na]
        at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59) [com.typesafe.akka.akka-actor_2.10-2.3.5.jar:na]
        at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72) [org.scala-lang.scala-library-2.10.4.jar:na]
        at akka.dispatch.BatchingExecutor$Batch.run(BatchingExecutor.scala:58) [com.typesafe.akka.akka-actor_2.10-2.3.5.jar:na]
        at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:41) [com.typesafe.akka.akka-actor_2.10-2.3.5.jar:na]
        at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393) [com.typesafe.akka.akka-actor_2.10-2.3.5.jar:na]
        at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) [org.scala-lang.scala-library-2.10.4.jar:na]
        at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) [org.scala-lang.scala-library-2.10.4.jar:na]
        at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) [org.scala-lang.scala-library-2.10.4.jar:na]
        at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) [org.scala-lang.scala-library-2.10.4.jar:na]
Caused by: java.lang.NullPointerException: null
        at views.html.partials.node_title_link$.apply(node_title_link.template.scala:31) ~[graylog-web-interface.graylog-web-interface-1.2.2.jar:1.2.2]
        at views.html.system.nodes.index$$anonfun$1.apply(index.template.scala:73) ~[graylog-web-interface.graylog-web-interface-1.2.2.jar:1.2.2]
        at views.html.system.nodes.index$$anonfun$1.apply(index.template.scala:68) ~[graylog-web-interface.graylog-web-interface-1.2.2.jar:1.2.2]
        at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) ~[org.scala-lang.scala-library-2.10.4.jar:na]
        at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) ~[org.scala-lang.scala-library-2.10.4.jar:na]
        at scala.collection.Iterator$class.foreach(Iterator.scala:727) ~[org.scala-lang.scala-library-2.10.4.jar:na]
        at scala.collection.AbstractIterator.foreach(Iterator.scala:1157) ~[org.scala-lang.scala-library-2.10.4.jar:na]
        at scala.collection.IterableLike$class.foreach(IterableLike.scala:72) ~[org.scala-lang.scala-library-2.10.4.jar:na]
        at scala.collection.AbstractIterable.foreach(Iterable.scala:54) ~[org.scala-lang.scala-library-2.10.4.jar:na]
        at scala.collection.TraversableLike$class.map(TraversableLike.scala:244) ~[org.scala-lang.scala-library-2.10.4.jar:na]
        at scala.collection.AbstractTraversable.map(Traversable.scala:105) ~[org.scala-lang.scala-library-2.10.4.jar:na]
        at views.html.system.nodes.index$.apply(index.template.scala:68) ~[graylog-web-interface.graylog-web-interface-1.2.2.jar:1.2.2]
        at views.html.system.nodes.index$.render(index.template.scala:319) ~[graylog-web-interface.graylog-web-interface-1.2.2.jar:1.2.2]
        at views.html.system.nodes.index.render(index.template.scala) ~[graylog-web-interface.graylog-web-interface-1.2.2.jar:1.2.2]
        at controllers.NodesController.nodes(NodesController.java:92) ~[graylog-web-interface.graylog-web-interface-1.2.2.jar:1.2.2]
        at Routes$$anonfun$routes$1$$anonfun$applyOrElse$27$$anonfun$apply$515.apply(routes_routing.scala:1805) ~[graylog-web-interface.graylog-web-interface-1.2.2.jar:na]
        at Routes$$anonfun$routes$1$$anonfun$applyOrElse$27$$anonfun$apply$515.apply(routes_routing.scala:1805) ~[graylog-web-interface.graylog-web-interface-1.2.2.jar:na]
        at play.core.Router$HandlerInvokerFactory$$anon$4.resultCall(Router.scala:264) ~[com.typesafe.play.play_2.10-2.3.10.jar:2.3.10]
        at play.core.Router$HandlerInvokerFactory$JavaActionInvokerFactory$$anon$15$$anon$1.invocation(Router.scala:255) ~[com.typesafe.play.play_2.10-2.3.10.jar:2.3.10]
        at play.core.j.JavaAction$$anon$1.call(JavaAction.scala:55) ~[com.typesafe.play.play_2.10-2.3.10.jar:2.3.10]
        at play.GlobalSettings$1.call(GlobalSettings.java:67) ~[com.typesafe.play.play_2.10-2.3.10.jar:2.3.10]
        at play.mvc.Security$AuthenticatedAction.call(Security.java:44) ~[com.typesafe.play.play_2.10-2.3.10.jar:2.3.10]
        at play.core.j.JavaAction$$anonfun$11.apply(JavaAction.scala:82) ~[com.typesafe.play.play_2.10-2.3.10.jar:2.3.10]
        at play.core.j.JavaAction$$anonfun$11.apply(JavaAction.scala:82) ~[com.typesafe.play.play_2.10-2.3.10.jar:2.3.10]
        at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24) ~[org.scala-lang.scala-library-2.10.4.jar:na]
        at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24) ~[org.scala-lang.scala-library-2.10.4.jar:na]
        at play.core.j.HttpExecutionContext$$anon$2.run(HttpExecutionContext.scala:40) ~[com.typesafe.play.play_2.10-2.3.10.jar:2.3.10]
        at play.api.libs.iteratee.Execution$trampoline$.execute(Execution.scala:46) [com.typesafe.play.play-iteratees_2.10-2.3.10.jar:2.3.10]
        at play.core.j.HttpExecutionContext.execute(HttpExecutionContext.scala:32) ~[com.typesafe.play.play_2.10-2.3.10.jar:2.3.10]
        at scala.concurrent.impl.Future$.apply(Future.scala:31) ~[org.scala-lang.scala-library-2.10.4.jar:na]
        at scala.concurrent.Future$.apply(Future.scala:485) ~[org.scala-lang.scala-library-2.10.4.jar:na]
        at play.core.j.JavaAction$class.apply(JavaAction.scala:82) ~[com.typesafe.play.play_2.10-2.3.10.jar:2.3.10]
        at play.core.Router$HandlerInvokerFactory$JavaActionInvokerFactory$$anon$15$$anon$1.apply(Router.scala:252) ~[com.typesafe.play.play_2.10-2.3.10.jar:2.3.10]
        at play.api.mvc.Action$$anonfun$apply$1$$anonfun$apply$4$$anonfun$apply$5.apply(Action.scala:130) ~[com.typesafe.play.play_2.10-2.3.10.jar:2.3.10]
        at play.api.mvc.Action$$anonfun$apply$1$$anonfun$apply$4$$anonfun$apply$5.apply(Action.scala:130) ~[com.typesafe.play.play_2.10-2.3.10.jar:2.3.10]
        at play.utils.Threads$.withContextClassLoader(Threads.scala:21) ~[com.typesafe.play.play_2.10-2.3.10.jar:2.3.10]
        at play.api.mvc.Action$$anonfun$apply$1$$anonfun$apply$4.apply(Action.scala:129) ~[com.typesafe.play.play_2.10-2.3.10.jar:2.3.10]
        at play.api.mvc.Action$$anonfun$apply$1$$anonfun$apply$4.apply(Action.scala:128) ~[com.typesafe.play.play_2.10-2.3.10.jar:2.3.10]
        at scala.Option.map(Option.scala:145) [org.scala-lang.scala-library-2.10.4.jar:na]
        at play.api.mvc.Action$$anonfun$apply$1.apply(Action.scala:128) ~[com.typesafe.play.play_2.10-2.3.10.jar:2.3.10]
        at play.api.mvc.Action$$anonfun$apply$1.apply(Action.scala:121) ~[com.typesafe.play.play_2.10-2.3.10.jar:2.3.10]
        at play.api.libs.iteratee.Iteratee$$anonfun$mapM$1.apply(Iteratee.scala:483) ~[com.typesafe.play.play-iteratees_2.10-2.3.10.jar:2.3.10]
        at play.api.libs.iteratee.Iteratee$$anonfun$mapM$1.apply(Iteratee.scala:483) ~[com.typesafe.play.play-iteratees_2.10-2.3.10.jar:2.3.10]
        at play.api.libs.iteratee.Iteratee$$anonfun$flatMapM$1.apply(Iteratee.scala:519) ~[com.typesafe.play.play-iteratees_2.10-2.3.10.jar:2.3.10]
        at play.api.libs.iteratee.Iteratee$$anonfun$flatMapM$1.apply(Iteratee.scala:519) ~[com.typesafe.play.play-iteratees_2.10-2.3.10.jar:2.3.10]
        at play.api.libs.iteratee.Iteratee$$anonfun$flatMap$1$$anonfun$apply$14.apply(Iteratee.scala:496) ~[com.typesafe.play.play-iteratees_2.10-2.3.10.jar:2.3.10]
        at play.api.libs.iteratee.Iteratee$$anonfun$flatMap$1$$anonfun$apply$14.apply(Iteratee.scala:496) ~[com.typesafe.play.play-iteratees_2.10-2.3.10.jar:2.3.10]
        at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24) ~[org.scala-lang.scala-library-2.10.4.jar:na]
        at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24) ~[org.scala-lang.scala-library-2.10.4.jar:na]
        ... 6 common frames omitted

@evanccnyc
Copy link
Author

In one of the server nodes logs I saw this:

2015-11-12T21:33:22.751Z WARN  [NodePingThread] Did not find meta info of this node. Re-registering.
2015-11-12T21:33:23.751Z WARN  [NodePingThread] Did not find meta info of this node. Re-registering.
2015-11-12T21:33:24.751Z WARN  [NodePingThread] Did not find meta info of this node. Re-registering.
2015-11-12T21:33:25.751Z WARN  [NodePingThread] Did not find meta info of this node. Re-registering.
2015-11-12T21:33:26.751Z WARN  [NodePingThread] Did not find meta info of this node. Re-registering.
2015-11-12T21:33:27.775Z WARN  [NodePingThread] Did not find meta info of this node. Re-registering.
2015-11-12T21:33:30.369Z WARN  [NodePingThread] Did not find meta info of this node. Re-registering.

Once i ran ntpdate I was able to access the node web interface again.

@evanccnyc
Copy link
Author

Can we reopen this bug?

@bernd
Copy link
Contributor

bernd commented Nov 13, 2015

I will re-open the issue. Thanks for the updates.

@bernd bernd reopened this Nov 13, 2015
@bernd bernd removed their assignment Nov 13, 2015
@MattDevUK
Copy link

I am also getting this issue, alongside a similar one to #1517 where randomly if I try access the nodes page, I get "Oops, an error occured". It only appeared today, after the cluster has been running fine for over 24 hours. I have 2 nodes, node 1 (master) which works fine and the jvm stats are correct, then the second node which displays JVM stats sometimes, and other times it's blank. If I turn off the second node (The one not displaying the stats) then it works fine, even going through the second node's web address to view it.

In the Web server logs when I receive the error I am seeing;

2015-11-18T12:37:27.676Z - [ERROR] - from application in play-akka.actor.default-dispatcher-16
Could not load node information
org.graylog2.restclient.models.NodeService$NodeNotFoundException: null
        at org.graylog2.restclient.models.NodeService.loadNode(NodeService.java:53) ~[org.graylog2.graylog2-rest-client--1.1.6-1.1.6.jar:na]
        at controllers.NodesController.nodes(NodesController.java:67) ~[graylog-web-interface.graylog-web-interface-1.1.6.jar:1.1.6]
        at Routes$$anonfun$routes$1$$anonfun$applyOrElse$42$$anonfun$apply$564.apply(routes_routing.scala:1939) [graylog-web-interface.graylog-web-interface-1.1.6.jar:na]
        at Routes$$anonfun$routes$1$$anonfun$applyOrElse$42$$anonfun$apply$564.apply(routes_routing.scala:1939) [graylog-web-interface.graylog-web-interface-1.1.6.jar:na]
        at play.core.Router$HandlerInvokerFactory$$anon$4.resultCall(Router.scala:264) [com.typesafe.play.play_2.10-2.3.9.jar:2.3.9]
        at play.core.Router$HandlerInvokerFactory$JavaActionInvokerFactory$$anon$15$$anon$1.invocation(Router.scala:255) [com.typesafe.play.play_2.10-2.3.9.jar:2.3.9]
        at play.core.j.JavaAction$$anon$1.call(JavaAction.scala:55) [com.typesafe.play.play_2.10-2.3.9.jar:2.3.9]
        at play.GlobalSettings$1.call(GlobalSettings.java:67) [com.typesafe.play.play_2.10-2.3.9.jar:2.3.9]
        at play.mvc.Security$AuthenticatedAction.call(Security.java:44) [com.typesafe.play.play_2.10-2.3.9.jar:2.3.9]
        at play.core.j.JavaAction$$anonfun$11.apply(JavaAction.scala:82) [com.typesafe.play.play_2.10-2.3.9.jar:2.3.9]
        at play.core.j.JavaAction$$anonfun$11.apply(JavaAction.scala:82) [com.typesafe.play.play_2.10-2.3.9.jar:2.3.9]
        at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24) [org.scala-lang.scala-library-2.10.4.jar:na]
        at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24) [org.scala-lang.scala-library-2.10.4.jar:na]
        at play.core.j.HttpExecutionContext$$anon$2.run(HttpExecutionContext.scala:40) [com.typesafe.play.play_2.10-2.3.9.jar:2.3.9]
        at play.api.libs.iteratee.Execution$trampoline$.execute(Execution.scala:46) [com.typesafe.play.play-iteratees_2.10-2.3.9.jar:2.3.9]
        at play.core.j.HttpExecutionContext.execute(HttpExecutionContext.scala:32) [com.typesafe.play.play_2.10-2.3.9.jar:2.3.9]
        at scala.concurrent.impl.Future$.apply(Future.scala:31) [org.scala-lang.scala-library-2.10.4.jar:na]
        at scala.concurrent.Future$.apply(Future.scala:485) [org.scala-lang.scala-library-2.10.4.jar:na]
        at play.core.j.JavaAction$class.apply(JavaAction.scala:82) [com.typesafe.play.play_2.10-2.3.9.jar:2.3.9]
        at play.core.Router$HandlerInvokerFactory$JavaActionInvokerFactory$$anon$15$$anon$1.apply(Router.scala:252) [com.typesafe.play.play_2.10-2.3.9.jar:2.3.9]
        at play.api.mvc.Action$$anonfun$apply$1$$anonfun$apply$4$$anonfun$apply$5.apply(Action.scala:130) [com.typesafe.play.play_2.10-2.3.9.jar:2.3.9]
        at play.api.mvc.Action$$anonfun$apply$1$$anonfun$apply$4$$anonfun$apply$5.apply(Action.scala:130) [com.typesafe.play.play_2.10-2.3.9.jar:2.3.9]
        at play.utils.Threads$.withContextClassLoader(Threads.scala:21) [com.typesafe.play.play_2.10-2.3.9.jar:2.3.9]
        at play.api.mvc.Action$$anonfun$apply$1$$anonfun$apply$4.apply(Action.scala:129) [com.typesafe.play.play_2.10-2.3.9.jar:2.3.9]
        at play.api.mvc.Action$$anonfun$apply$1$$anonfun$apply$4.apply(Action.scala:128) [com.typesafe.play.play_2.10-2.3.9.jar:2.3.9]
        at scala.Option.map(Option.scala:145) [org.scala-lang.scala-library-2.10.4.jar:na]
        at play.api.mvc.Action$$anonfun$apply$1.apply(Action.scala:128) [com.typesafe.play.play_2.10-2.3.9.jar:2.3.9]
        at play.api.mvc.Action$$anonfun$apply$1.apply(Action.scala:121) [com.typesafe.play.play_2.10-2.3.9.jar:2.3.9]
        at play.api.libs.iteratee.Iteratee$$anonfun$mapM$1.apply(Iteratee.scala:483) [com.typesafe.play.play-iteratees_2.10-2.3.9.jar:2.3.9]
        at play.api.libs.iteratee.Iteratee$$anonfun$mapM$1.apply(Iteratee.scala:483) [com.typesafe.play.play-iteratees_2.10-2.3.9.jar:2.3.9]
        at play.api.libs.iteratee.Iteratee$$anonfun$flatMapM$1.apply(Iteratee.scala:519) [com.typesafe.play.play-iteratees_2.10-2.3.9.jar:2.3.9]
        at play.api.libs.iteratee.Iteratee$$anonfun$flatMapM$1.apply(Iteratee.scala:519) [com.typesafe.play.play-iteratees_2.10-2.3.9.jar:2.3.9]
        at play.api.libs.iteratee.Iteratee$$anonfun$flatMap$1$$anonfun$apply$14.apply(Iteratee.scala:496) [com.typesafe.play.play-iteratees_2.10-2.3.9.jar:2.3.9]
        at play.api.libs.iteratee.Iteratee$$anonfun$flatMap$1$$anonfun$apply$14.apply(Iteratee.scala:496) [com.typesafe.play.play-iteratees_2.10-2.3.9.jar:2.3.9]
        at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24) [org.scala-lang.scala-library-2.10.4.jar:na]
        at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24) [org.scala-lang.scala-library-2.10.4.jar:na]
        at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:41) [com.typesafe.akka.akka-actor_2.10-2.3.5.jar:na]
        at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393) [com.typesafe.akka.akka-actor_2.10-2.3.5.jar:na]
        at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) [org.scala-lang.scala-library-2.10.4.jar:na]
        at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) [org.scala-lang.scala-library-2.10.4.jar:na]
        at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) [org.scala-lang.scala-library-2.10.4.jar:na]
        at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) [org.scala-lang.scala-library-2.10.4.jar:na]

2015-11-18T12:37:27.698Z - [ERROR] - from play in play-akka.actor.default-dispatcher-16
Cannot invoke the action, eventually got an error: java.lang.NullPointerException

2015-11-18T12:37:27.700Z - [ERROR] - from application in play-akka.actor.default-dispatcher-16


! @6o6mkm76c - Internal server error, for (GET) [/system/nodes] ->

play.api.Application$$anon$1: Execution exception[[NullPointerException: null]]
        at play.api.Application$class.handleError(Application.scala:296) ~[com.typesafe.play.play_2.10-2.3.9.jar:2.3.9]
        at play.api.DefaultApplication.handleError(Application.scala:402) [com.typesafe.play.play_2.10-2.3.9.jar:2.3.9]
        at play.core.server.netty.PlayDefaultUpstreamHandler$$anonfun$3$$anonfun$applyOrElse$4.apply(PlayDefaultUpstreamHandler.scala:320) [com.typesafe.play.play_2.10-2.3.9.jar:2.3.9]
        at play.core.server.netty.PlayDefaultUpstreamHandler$$anonfun$3$$anonfun$applyOrElse$4.apply(PlayDefaultUpstreamHandler.scala:320) [com.typesafe.play.play_2.10-2.3.9.jar:2.3.9]
        at scala.Option.map(Option.scala:145) [org.scala-lang.scala-library-2.10.4.jar:na]
        at play.core.server.netty.PlayDefaultUpstreamHandler$$anonfun$3.applyOrElse(PlayDefaultUpstreamHandler.scala:320) [com.typesafe.play.play_2.10-2.3.9.jar:2.3.9]
        at play.core.server.netty.PlayDefaultUpstreamHandler$$anonfun$3.applyOrElse(PlayDefaultUpstreamHandler.scala:316) [com.typesafe.play.play_2.10-2.3.9.jar:2.3.9]
        at scala.concurrent.Future$$anonfun$recoverWith$1.apply(Future.scala:344) [org.scala-lang.scala-library-2.10.4.jar:na]
        at scala.concurrent.Future$$anonfun$recoverWith$1.apply(Future.scala:343) [org.scala-lang.scala-library-2.10.4.jar:na]
        at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32) [org.scala-lang.scala-library-2.10.4.jar:na]
        at play.api.libs.iteratee.Execution$trampoline$.execute(Execution.scala:46) [com.typesafe.play.play-iteratees_2.10-2.3.9.jar:2.3.9]
        at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:40) [org.scala-lang.scala-library-2.10.4.jar:na]
        at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:248) [org.scala-lang.scala-library-2.10.4.jar:na]
        at scala.concurrent.Promise$class.complete(Promise.scala:55) [org.scala-lang.scala-library-2.10.4.jar:na]
        at scala.concurrent.impl.Promise$DefaultPromise.complete(Promise.scala:153) [org.scala-lang.scala-library-2.10.4.jar:na]
        at scala.concurrent.Future$$anonfun$map$1.apply(Future.scala:235) [org.scala-lang.scala-library-2.10.4.jar:na]
        at scala.concurrent.Future$$anonfun$map$1.apply(Future.scala:235) [org.scala-lang.scala-library-2.10.4.jar:na]
        at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32) [org.scala-lang.scala-library-2.10.4.jar:na]
        at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.processBatch$1(BatchingExecutor.scala:67) [com.typesafe.akka.akka-actor_2.10-2.3.5.jar:na]
        at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:82) [com.typesafe.akka.akka-actor_2.10-2.3.5.jar:na]
        at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59) [com.typesafe.akka.akka-actor_2.10-2.3.5.jar:na]
        at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59) [com.typesafe.akka.akka-actor_2.10-2.3.5.jar:na]
        at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72) [org.scala-lang.scala-library-2.10.4.jar:na]
        at akka.dispatch.BatchingExecutor$Batch.run(BatchingExecutor.scala:58) [com.typesafe.akka.akka-actor_2.10-2.3.5.jar:na]
        at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:41) [com.typesafe.akka.akka-actor_2.10-2.3.5.jar:na]
        at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393) [com.typesafe.akka.akka-actor_2.10-2.3.5.jar:na]
        at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) [org.scala-lang.scala-library-2.10.4.jar:na]
        at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) [org.scala-lang.scala-library-2.10.4.jar:na]
        at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) [org.scala-lang.scala-library-2.10.4.jar:na]
        at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) [org.scala-lang.scala-library-2.10.4.jar:na]
Caused by: java.lang.NullPointerException: null
        at views.html.partials.node_title_link$.apply(node_title_link.template.scala:31) ~[graylog-web-interface.graylog-web-interface-1.1.6.jar:1.1.6]
        at views.html.system.nodes.index$$anonfun$1.apply(index.template.scala:73) ~[graylog-web-interface.graylog-web-interface-1.1.6.jar:1.1.6]
        at views.html.system.nodes.index$$anonfun$1.apply(index.template.scala:68) ~[graylog-web-interface.graylog-web-interface-1.1.6.jar:1.1.6]
        at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) ~[org.scala-lang.scala-library-2.10.4.jar:na]
        at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) ~[org.scala-lang.scala-library-2.10.4.jar:na]
        at scala.collection.Iterator$class.foreach(Iterator.scala:727) ~[org.scala-lang.scala-library-2.10.4.jar:na]
        at scala.collection.AbstractIterator.foreach(Iterator.scala:1157) ~[org.scala-lang.scala-library-2.10.4.jar:na]
        at scala.collection.IterableLike$class.foreach(IterableLike.scala:72) ~[org.scala-lang.scala-library-2.10.4.jar:na]
        at scala.collection.AbstractIterable.foreach(Iterable.scala:54) ~[org.scala-lang.scala-library-2.10.4.jar:na]
        at scala.collection.TraversableLike$class.map(TraversableLike.scala:244) ~[org.scala-lang.scala-library-2.10.4.jar:na]
        at scala.collection.AbstractTraversable.map(Traversable.scala:105) ~[org.scala-lang.scala-library-2.10.4.jar:na]
        at views.html.system.nodes.index$.apply(index.template.scala:68) ~[graylog-web-interface.graylog-web-interface-1.1.6.jar:1.1.6]
        at views.html.system.nodes.index$.render(index.template.scala:320) ~[graylog-web-interface.graylog-web-interface-1.1.6.jar:1.1.6]
        at views.html.system.nodes.index.render(index.template.scala) ~[graylog-web-interface.graylog-web-interface-1.1.6.jar:1.1.6]
        at controllers.NodesController.nodes(NodesController.java:92) ~[graylog-web-interface.graylog-web-interface-1.1.6.jar:1.1.6]
        at Routes$$anonfun$routes$1$$anonfun$applyOrElse$42$$anonfun$apply$564.apply(routes_routing.scala:1939) ~[graylog-web-interface.graylog-web-interface-1.1.6.jar:na]
        at Routes$$anonfun$routes$1$$anonfun$applyOrElse$42$$anonfun$apply$564.apply(routes_routing.scala:1939) ~[graylog-web-interface.graylog-web-interface-1.1.6.jar:na]
        at play.core.Router$HandlerInvokerFactory$$anon$4.resultCall(Router.scala:264) ~[com.typesafe.play.play_2.10-2.3.9.jar:2.3.9]
        at play.core.Router$HandlerInvokerFactory$JavaActionInvokerFactory$$anon$15$$anon$1.invocation(Router.scala:255) ~[com.typesafe.play.play_2.10-2.3.9.jar:2.3.9]
        at play.core.j.JavaAction$$anon$1.call(JavaAction.scala:55) ~[com.typesafe.play.play_2.10-2.3.9.jar:2.3.9]
        at play.GlobalSettings$1.call(GlobalSettings.java:67) ~[com.typesafe.play.play_2.10-2.3.9.jar:2.3.9]
        at play.mvc.Security$AuthenticatedAction.call(Security.java:44) ~[com.typesafe.play.play_2.10-2.3.9.jar:2.3.9]
        at play.core.j.JavaAction$$anonfun$11.apply(JavaAction.scala:82) ~[com.typesafe.play.play_2.10-2.3.9.jar:2.3.9]
        at play.core.j.JavaAction$$anonfun$11.apply(JavaAction.scala:82) ~[com.typesafe.play.play_2.10-2.3.9.jar:2.3.9]
        at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24) ~[org.scala-lang.scala-library-2.10.4.jar:na]
        at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24) ~[org.scala-lang.scala-library-2.10.4.jar:na]
        at play.core.j.HttpExecutionContext$$anon$2.run(HttpExecutionContext.scala:40) ~[com.typesafe.play.play_2.10-2.3.9.jar:2.3.9]
        at play.api.libs.iteratee.Execution$trampoline$.execute(Execution.scala:46) [com.typesafe.play.play-iteratees_2.10-2.3.9.jar:2.3.9]
        at play.core.j.HttpExecutionContext.execute(HttpExecutionContext.scala:32) ~[com.typesafe.play.play_2.10-2.3.9.jar:2.3.9]
        at scala.concurrent.impl.Future$.apply(Future.scala:31) ~[org.scala-lang.scala-library-2.10.4.jar:na]
        at scala.concurrent.Future$.apply(Future.scala:485) ~[org.scala-lang.scala-library-2.10.4.jar:na]
        at play.core.j.JavaAction$class.apply(JavaAction.scala:82) ~[com.typesafe.play.play_2.10-2.3.9.jar:2.3.9]
        at play.core.Router$HandlerInvokerFactory$JavaActionInvokerFactory$$anon$15$$anon$1.apply(Router.scala:252) ~[com.typesafe.play.play_2.10-2.3.9.jar:2.3.9]
        at play.api.mvc.Action$$anonfun$apply$1$$anonfun$apply$4$$anonfun$apply$5.apply(Action.scala:130) ~[com.typesafe.play.play_2.10-2.3.9.jar:2.3.9]
        at play.api.mvc.Action$$anonfun$apply$1$$anonfun$apply$4$$anonfun$apply$5.apply(Action.scala:130) ~[com.typesafe.play.play_2.10-2.3.9.jar:2.3.9]
        at play.utils.Threads$.withContextClassLoader(Threads.scala:21) ~[com.typesafe.play.play_2.10-2.3.9.jar:2.3.9]
        at play.api.mvc.Action$$anonfun$apply$1$$anonfun$apply$4.apply(Action.scala:129) ~[com.typesafe.play.play_2.10-2.3.9.jar:2.3.9]
        at play.api.mvc.Action$$anonfun$apply$1$$anonfun$apply$4.apply(Action.scala:128) ~[com.typesafe.play.play_2.10-2.3.9.jar:2.3.9]
        at scala.Option.map(Option.scala:145) [org.scala-lang.scala-library-2.10.4.jar:na]
        at play.api.mvc.Action$$anonfun$apply$1.apply(Action.scala:128) ~[com.typesafe.play.play_2.10-2.3.9.jar:2.3.9]
        at play.api.mvc.Action$$anonfun$apply$1.apply(Action.scala:121) ~[com.typesafe.play.play_2.10-2.3.9.jar:2.3.9]
        at play.api.libs.iteratee.Iteratee$$anonfun$mapM$1.apply(Iteratee.scala:483) ~[com.typesafe.play.play-iteratees_2.10-2.3.9.jar:2.3.9]
        at play.api.libs.iteratee.Iteratee$$anonfun$mapM$1.apply(Iteratee.scala:483) ~[com.typesafe.play.play-iteratees_2.10-2.3.9.jar:2.3.9]
        at play.api.libs.iteratee.Iteratee$$anonfun$flatMapM$1.apply(Iteratee.scala:519) ~[com.typesafe.play.play-iteratees_2.10-2.3.9.jar:2.3.9]
        at play.api.libs.iteratee.Iteratee$$anonfun$flatMapM$1.apply(Iteratee.scala:519) ~[com.typesafe.play.play-iteratees_2.10-2.3.9.jar:2.3.9]
        at play.api.libs.iteratee.Iteratee$$anonfun$flatMap$1$$anonfun$apply$14.apply(Iteratee.scala:496) ~[com.typesafe.play.play-iteratees_2.10-2.3.9.jar:2.3.9]
        at play.api.libs.iteratee.Iteratee$$anonfun$flatMap$1$$anonfun$apply$14.apply(Iteratee.scala:496) ~[com.typesafe.play.play-iteratees_2.10-2.3.9.jar:2.3.9]
        at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24) ~[org.scala-lang.scala-library-2.10.4.jar:na]
        at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24) ~[org.scala-lang.scala-library-2.10.4.jar:na]
        ... 6 common frames omitted

There doesn't appear to be any errors when the JVM stats don't work. I am also seeing the 2015-11-18T12:30:34.987Z WARN [NodePingThread] Did not find meta info of this node. Re-registering. being sent every second.

My boxes are running with Graylog 1.1.6, Elasticsearch 1.3.9, CentOS7, Java 1.8

@kroepke
Copy link
Contributor

kroepke commented Nov 18, 2015

@MattDevUK The NodePingThread warning means one of the nodes is having garbage collection (or other) pauses of over 5 seconds, which leads to all kinds of different errors.
You might have to increase the heap size of the server, it seems overloaded.

@MattDevUK
Copy link

@kroepke I've tripled the heapsize and checked with jmap, it's using about 50% of the space allocated but still getting that warning in the logs every second. also, only the second node is getting this and it's the exact same setup as the master.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@bernd @kroepke @MattDevUK @evanccnyc