Tests that use mongodb replica sets fail on some machines.

[vizio360]

Hello, a bit of a weird one that I cannot really solve.

We use MongoMemoryReplSet in our tests and these tests work on some machines and fail on others.
Here is the combination of setups we have tested:

• windows + node 20.18.1 + mongodb-inmemory-server 8.16.1 + mongodb 6.0.14 = ✅
• macos + node 20.18.1 + mongodb-inmemory-server 8.16.1 + mongodb 6.0.14 = ❌

I managed to reproduce the issue with a small repo but in that case adding writeConcern: { wtimeoutMS: 5000} seems to fix the problem. But it does not in our code base.

We are also using NestJS and Mongoose.

We get errors like:

MongoServerError: not primary
MongoServerError: operation was interrupted
OverwriteModelError: Cannot overwritecollectionOne model once compiled.

From what I've noticed it seems to be a timing issue; when using the MongoMemoryReplSet mongodb takes longer to startup and so mongoose then takes longer to setup its models, indexes etc. and the tests then fail because we are trying to access the models when they are not ready yet.

Are there any special precautions you need to take when using MongoMemoryReplSet ?
Are there any known issues with Mac OS or more recent versions of Node?

I know this is a bit vague but I cannot really share our code and the sample app I did is fixed with the extra parameter (I could share that anyway if needed).

Maybe you can ask me some questions that will then help me look at other options/ideas.

[hasezoey]

OverwriteModelError: Cannot overwritecollectionOne model once compiled.

This is completely a mongoose error, you are trying to create another model with the same name on the same connection / mongoose instance. Either dont register a new model, or delete the old one first.

We get errors like:

Aside from the mongoose specific error, in what context do they happen? (in MMS start / stop, or your operations?)

mongodb-inmemory-server 8.16.1

Quite a old version, did you already try the latest? (currently its 10.1.3)

[vizio360]

Quite a old version, did you already try the latest? (currently its 10.1.3)

Yes tried with that one and still got failures of the like MongoServerError: not primary, MongoServerError: operation was interrupted

Aside from the mongoose specific error, in what context do they happen? (in MMS start / stop, or your operations?)

MMS start seems to work fine, we start the server in the beforeAll and then tests start running, so that seems to work.
MMS stop the same, it seems to work.

It seems to be an issue with knowing when MMS is ready. Is there a way to know when MMS is completely up and running? Not sure if this is exacerbated by the performance of the machines we run our tests on (my local machine in my case).

I'm guessing it's a MMS readiness issue which means Mongoose has not got all models setup yet (or possibly a Mongoose issue being not able to figure out when the server is ready) because some of the errors are like:

TypeError: Cannot read properties of undefined (reading 'deleteMany')

      191 |
      192 |     afterEach(async () => {
    > 193 |         await OurModel.deleteMany({}).exec()
          |                               ^
      194 |         await OurOtherModel.deleteMany({}).exec()
      195 |     })
      196 |

others come back with MongoServerError: not primary which again makes me think that we are trying to interact with the DB while MMS is still trying to get itself ready with primary promotion etc.

The test is a pretty quick one, we just check if an instance has been defined correctly. So it will check that and then quickly call the afterEach to do the clean up.

If I add an arbitrary sleep at the end of the beforeEach all tests pass.

I'll also head to the Mongoose repo and see if people have similar issues there.

[hasezoey]

It seems to be an issue with knowing when MMS is ready. Is there a way to know when MMS is completely up and running?

For MongoMemoryServer we will wait for the instanceReady event (among some others, like Closed or Error), which gets fired when finding line waiting for connections. MongoReplSet on the other hand also waits for the instancePrimary event after setup, which gets fired when finding line transition to primary complete; database writes are now permitted.
At least that is the theory in MongoReplSet, but if there are more than 1 servers, there is no guarantee that it might just get demoted right after and not have a primary for some time, at least i dont know of a more reliable way.

In case you dont await MongoReplset.start or somehow dont have access to the promise, but do to the server instance, you can call .waitUntilRunning, which is not necessary when awaiting .start.
(this might be the case when having started once, but then called .stop and then .start again)

The only recommendation i can confidently give, is to check that you are using a uneven number for the replset instances (ie 1,3,5,etc), see ReplSetOpts#count.
Also check that you are using MongoReplSet.getUri for the url, which should include all started servers, which the driver then uses to find a instance it wants / can use. (including finding primary for writes)

means Mongoose has not got all models setup yet (or possibly a Mongoose issue being not able to figure out when the server is ready) because some of the errors are like:
TypeError: Cannot read properties of undefined (reading 'deleteMany')

If you are using vanilla mongoose, this should not happen, for more debugging there i would need access to the code or representative code examples.

Also to further debug the not primary and operation was interrupted errors, i would need some mongodb-memory-server debug logs, which you might need to audit as it might contain stuff you dont wanna share (saying this because of the but I cannot really share our code)

[vizio360]

@hasezoey thank you so much for you prompt replies, I'll investigate further and test the odd number of server option (I think atm we set the replicas to 2) and also will provide some logs.