NoClassDefFoundError on run time

[gonsan]

I have this problem.. :(

this module was deployed well.

But, sometimes I have a follow problem on run time.

java.lang.NoClassDefFoundError: com/github/mauricio/async/db/util/ByteBufferUtils$
at com.github.mauricio.async.db.mysql.codec.MySQLFrameDecoder.decode(MySQLFrameDecoder.scala:63)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:241)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:149)
at io.netty.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChannelHandlerContext.java:341)
at io.netty.channel.DefaultChannelHandlerContext.fireChannelRead(DefaultChannelHandlerContext.java:327)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:785)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:126)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:507)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:464)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:378)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:350)
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: com.github.mauricio.async.db.util.ByteBufferUtils$
at org.vertx.java.platform.impl.ModuleClassLoader.loadFromModule(ModuleClassLoader.java:127)
at org.vertx.java.platform.impl.ModuleClassLoader.loadClass(ModuleClassLoader.java:108)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
... 13 more

I'm using

• Vert.x 2.1
• Ubuntu 14.0.4 (Windwos is working well..)
• scala=io.vertxlang-scala1.0.0:org.vertx.scala.platform.impl.ScalaVerticleFactory

[Zwergal]

Could you please try to give us a reproducer of this problem? When does this occur and with what commands are you working? I guess you are using MySQL?

[Narigo]

Do you think this might be the same issue as #27 ?

[gonsan]

@Narigo yes, I think it's same issue as #27
Do you know how to I solve this problem?

[Narigo]

@gonsan did you try the latest version of mod-mysql-postgresql? Does it still occur with 0.3.1?

[raphaelsampaio]

Hi! I'm experiencing this issue sometimes as well.

My setup is:

• vert.x: 2.1.2
• lang-scala: 2.10-1.1.0-M1
• mod-mysql-postgresql: 2.10-0.3.1 (MySQL)

I tried on both Ubuntu 14.04 and OSX 10.9.5

[raphaelsampaio]

Hi guys. We just figured out that we were redeploying the module before each test in our integration tests suite. Deploying it before all tests solved the issue.

[Narigo]

@raphaelsampaio thanks for that hint!

Could it be that you were not waiting for complete deployment of the module? Otherwise there might still be a race condition somewhere but it doesn't occur very often, I guess.

[raphaelsampaio]

We are actually using vert.x AsyncResultHandler to check if the module was completely deployed. After the first connection to the database the exception is thrown. We verified this is happening when we try to deploy this module as part of another module's deployment. The scenarios below should explain it better:

Scenario 1
Module A deploys mod-mysql-postgresql and completes its deployment procedure.

Scenario 2
mod-mysql-postgresql is deployed before Module A.

The exception gets thrown only in Scenario 1 and in specific environments. For example, the same test suite would fail because of that exception in Jenkins, but pass in my computer.

[litch]

I'm CONSTANTLY getting these now. Not sure why.

The particular class that seems to be hanging things up in the current setup is:

java.lang.NoClassDefFoundError: scala/collection/immutable/Range$Partial
    at java.lang.Class.getDeclaredMethods0(Native Method)
    at java.lang.Class.privateGetDeclaredMethods(java/lang/Class.java:2688)
    at java.lang.Class.getDeclaredMethods(java/lang/Class.java:1962)
    at com.fasterxml.jackson.databind.introspect.AnnotatedClass._addMemberMethods(com/fasterxml/jackson/databind/introspect/AnnotatedClass.java:593)
    at com.fasterxml.jackson.databind.introspect.AnnotatedClass.resolveMemberMethods(com/fasterxml/jackson/databind/introspect/AnnotatedClass.java:416)
    at com.fasterxml.jackson.databind.introspect.AnnotatedClass.memberMethods(com/fasterxml/jackson/databind/introspect/AnnotatedClass.java:243)
    at com.fasterxml.jackson.databind.introspect.POJOPropertiesCollector._addMethods(com/fasterxml/jackson/databind/introspect/POJOPropertiesCollector.java:457)
    at com.fasterxml.jackson.databind.introspect.POJOPropertiesCollector.collect(com/fasterxml/jackson/databind/introspect/POJOPropertiesCollector.java:233)
    at com.fasterxml.jackson.databind.introspect.BasicClassIntrospector.collectProperties(com/fasterxml/jackson/databind/introspect/BasicClassIntrospector.java:142)

I've tried making sure the module loads first - in my main verticle doing:

Vertx.deploy_module('io.vertx~mod-mysql-postgresql_2.10~0.3.1', config['async_pg'], 8) do
  other verticle...
  other verticle...
  other verticle...
end

Or even:

Vertx.deploy_module('io.vertx~mod-mysql-postgresql_2.10~0.3.1', config['async_pg'], 8) do
  Vertx.set_timer(30_000) do
    other verticle...
    other verticle...
    other verticle...
  end
end

However, if I start it in a totally separate vert.x instance that's clustered, it works fine.

Pretty annoying =/

[adri0]

I also have it happening from time to time for apparently no reason. I noticed the error seems to happen more often when my application is under heavy load, and generally in the first seconds/minutes after the start. The warmup of the server seems to have some influence over it too. I'm not really sure.

It's indeed annoying.

Sometimes the error reports different classes (NoClassDefFoundError). This is what I saw the last time it happened:

2015-03-25 17:03:47.111 WARN  i.n.c.DefaultChannelPipeline - An exception was thrown by a user handler's exceptionCaught() method while handling the following exception:
java.lang.NoClassDefFoundError: scala/util/Success$$anonfun$map$1
    at scala.util.Success.map(Try.scala:206)
    at scala.concurrent.Future$$anonfun$map$1.apply(Future.scala:235)
    at scala.concurrent.Future$$anonfun$map$1.apply(Future.scala:235)
    at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
    at org.vertx.scala.core.VertxExecutionContext$VertxExecutionContextImpl.execute(VertxExecutionContext.scala:49)
    at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:40)
    at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:248)
    at scala.concurrent.Promise$class.complete(Promise.scala:55)
    at scala.concurrent.impl.Promise$DefaultPromise.complete(Promise.scala:153)
    at scala.concurrent.Future$$anonfun$recoverWith$1.apply(Future.scala:345)
    at scala.concurrent.Future$$anonfun$recoverWith$1.apply(Future.scala:343)
    at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
    at org.vertx.scala.core.VertxExecutionContext$VertxExecutionContextImpl.execute(VertxExecutionContext.scala:49)
    at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:40)
    at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:248)
    at scala.concurrent.Promise$class.complete(Promise.scala:55)
    at scala.concurrent.impl.Promise$DefaultPromise.complete(Promise.scala:153)
    at scala.concurrent.Promise$class.success(Promise.scala:89)
    at scala.concurrent.impl.Promise$DefaultPromise.success(Promise.scala:153)
    at com.github.mauricio.async.db.mysql.MySQLConnection.onOk(MySQLConnection.scala:137)
    at com.github.mauricio.async.db.mysql.codec.MySQLConnectionHandler.channelRead0(MySQLConnectionHandler.scala:97)

[artur-w]

I have the same problem. Deploying on my computer works fine, deploying on Jenkins produces:

java.lang.NoClassDefFoundError: com/github/mauricio/async/db/postgresql/messages/frontend/StartupMessage
        at com.github.mauricio.async.db.postgresql.codec.PostgreSQLConnectionHandler.channelActive(PostgreSQLConnectionHandler.scala:123)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:208)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelActive(AbstractChannelHandlerContext.java:194)
        at io.netty.channel.ChannelInboundHandlerAdapter.channelActive(ChannelInboundHandlerAdapter.java:64)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:208)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelActive(AbstractChannelHandlerContext.java:194)
        at io.netty.channel.DefaultChannelPipeline.fireChannelActive(DefaultChannelPipeline.java:758)
        at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:259)
        at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:288)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:528)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
        at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: com.github.mauricio.async.db.postgresql.messages.frontend.StartupMessage
        at org.vertx.java.platform.impl.ModuleClassLoader.loadFromModule(ModuleClassLoader.java:127)
        at org.vertx.java.platform.impl.ModuleClassLoader.loadClass(ModuleClassLoader.java:108)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        ... 15 more

any ideas?

[litch]

I never did make any progress on this - I found it easier to just leave an async_db module running in a separate vert.x instance in the cluster. The best I could tell was that it was from it starting to handle messages before the classes were totally loaded.

[artur-w]

I suppose so. When I add 2000ms delay after async_db module initialized but before other verticles initialization it started to work correctly.
I wonder if there is small bug here: https://github.com/vert-x/mod-mysql-postgresql/blob/master/src/main/scala/io/vertx/asyncsql/Starter.scala#L34

Handler is added without any wait until registration is confirmed. When there are few vertx instances in cluster registration can take some time. Maybe then other modules start to query not-totally loaded async_db module?

[litch]

That looks likely to me, though I have very little understanding of this project.

[Narigo]

@EKOSYSTEM I guess you are right here. Do you want to provide a pull request for this?

[Narigo]

@EKOSYSTEM can you try out the fix here?
https://github.com/vert-x/mod-mysql-postgresql/tree/wait-for-registered-handler