World changes do not appear on bluemap.

[Samg381]

What I did / Steps to reproduce

I simply installed the latest version of BlueMap (bluemap-5.5-paper.jar) on my paper server, restarted, and accessed bluemap using the LAN IP of the server (gaming PC and server are separate machines). I then joined the server, and could see my character moving around on bluemap. As a final test, I built a test structure on my world.

Expected result

I expected the new structure to appear on bluemap.

Actual result

The new structure does not appear on bluemap.

Things I tried:

• waiting over an hour (per this FAQ)
• clicking 'Update Map' within bluemap
• used a different browser (to rule out cache issue)
• running /save-all flush
• restarting the server

Context

Bluemap:
[19:05:12 INFO]: Version: 5.5
[19:05:12 INFO]: Commit: 2c63d76
[19:05:12 INFO]: Implementation: paper
[19:05:12 INFO]: Minecraft: 1.21.3
[19:05:12 INFO]: Render-threads: 0
[19:05:12 INFO]: Available processors: 6
[19:05:12 INFO]: Available memory: 10240 MiB

Paper:
Paper version 1.21.3-67-master@d38624b (2024-11-27T20:28:58Z) (Implementing API version 1.21.3-R0.1-SNAPSHOT)
1.0.0-fabric-1.16.1

Plugins
I am not running any plugins other than bluemap.

Other info
Server is running Windows 10 22H2. Map generated organically.

Screenshot showing map discrepancy after 1+ hour and a server restart

I do not use discord, but am more than happy to provide any logs / debugging steps requested of me.

[TBlueF]

Check /bluemap maps .. is the map frozen?

[Samg381]

Check /bluemap maps .. is the map frozen?

Doesn't appear so- here is that output:

[19:31:38 INFO]: Maps loaded by BlueMap:
[19:31:38 INFO]: - world (world (overworld))
[19:31:38 INFO]:    World: world#minecraft:overworld
[19:31:38 INFO]: - world_the_end (world_the_end (the_end))
[19:31:38 INFO]:    World: world_the_end#minecraft:the_end
[19:31:38 INFO]: - world_nether (world_nether (the_nether))
[19:31:38 INFO]:    World: world_nether#minecraft:the_nether

[TBlueF]

/bluemap debug map world while standing on a block that has changed but is not visible on the map

[Samg381]

[TBlueF]

Okay .. and what is /bluemap showing rn?

(Chunk current hash and last hash are different -> bluemap WILL re-render this map-tile latest with the next server-restart/bluemap reload)

[Samg381]

[TBlueF]

your render-threads are stopped -> bluemap doesn't render
you must have stopped them at some point with /bluemap stop
run /bluemap start to start the render threads again, and you will see the updates come in :)

[Samg381]

Hmm! I don't remember doing that.. odd! Maybe I fatfingered it when testing earlier..

That seemed to fix it! Thanks a ton!

[Samg381]

Okay.. false alarm (maybe).. the structure seemed to appear after we ran bluemap start, however new changes are not appearing.

For instance, I tried to destroy that structure, and it is still present on bluemap.

I ran bluemap again:

bluemap
[19:53:57 INFO]: BlueMap - Status:
[19:53:57 INFO]: Render-Threads are idle!
[19:53:57 INFO]: Last time running: 2024-12-16 19:47:50

The current time is 19:55 - is there a way to check / configure when the render threads will run again?

[Samg381]

Update after 30 minutes- it seems as if the renderer still hasn't run.

[Samg381]

I did a bunch of experiments. It seems as if the only thing that will get bluemap to see new changes is a full re-render, or a server restart. Running bluemap update world 1 1 1 also seems to properly force an update.

Based on my loose understanding of how bluemap works, it seems like a hash is generated for each chunk every x ticks. This hash is compared to the previous hash of said chunk, and if they are not identical, a render task is created for that chunk/block.

My superficial observation is that the hashing / change detection is working properly (given I can consistently observe discrepancies in hashes after making changes), however something is preventing these changes from triggering a render / update of the changed area.

For instance, here is the output of bluemap debug map world when I freshly restart the server (i.e. all changes rendered). The hashes match, as we would expect.

Then, I place a block on the ground, and immediately run bluemap debug block. It looks like bluemap hasn't registered it yet, and thinks it is air:

After waiting a minute, the block is indeed recognized as stone:

Now we run bluemap debug map world again, and as expected, the hash has changed:

Based on my understanding of bluemap, this should have triggered an update. But if we run bluemap, there are no updates queued, and the last update occurred 10 minutes ago when the server restarted :(

I hope this helps.

[TechnicJelle]

is there a way to check / configure when the render threads will run again?

BlueMap needs to wait until the server saves the world data to disk, which it detects with File Watchers.

The server software (in this case, Paper) decides when to save the world to disk.
Although a /save-all flush should also trigger them.

Maybe those File Watchers aren't working properly?

Have you checked BlueMap's debug log file yet? (So not the output in your server console, but BlueMap's own log, which has more detail)
Maybe that will show some useful information?
I vaguely remember that it can show File Watcher crash logs, maybe.

You could also try setting the full-update-interval option to something shorter than 24 hours (which is the default) to see if that helps.

[Samg381]

I gave /save-all flush a try- it does succeed in forcing bluemap to update- of course this is not really a permanent solution.

I just checked the Bluemap log. I am not sure what 'normal' would look like, but it seems like the actual updates aren't firing?

[TBlueF]

If /save-all flush makes the map update, then from bluemaps side everything is working.. All that command does is forcing minecraft to save stuff to the world-files, bluemap doesn't detect that command, it just watches the actual files and updates based on that. So if you ran that command and then bluemap updated the world, that means bluemap is detecting the changes correctly and is updating correctly.
So your issue lies with your minecraft server not saving the chunks to the actual files as fast as you would like it to. E.g. if you are in the spawn-chunks or are constantly loading the chunks you changed something in, then minecraft doesn't unload them and doesn't save them to the world-files for a long time.

Out of experience from a lot of similar support-discussions: If you just let bluemap and minecraft do it's thing, bluemap will update eventually (as long as its not stopped/frozen). Usually within an hour often a lot less. On normal gameplay -> moving around loading/unloading new chunks, etc.. It's not a real-time map and never will be.

[Samg381]

Thanks for getting back. What you described does makes sense, and I can believe that is how bluemap should ordinarily function, however I have left my server for 8+ hours and have not once seen the map update itself without a manual re-render or server restart. This is consistent across multiple tests, all while moving around, loading / unloading chunks, etc.

Out of curiosity, does bluemap debug block pull information about the block directly from the server? Or does it query the saved chunk in the world-file like the bluemap change detection does? There seems to be a delay in bluemap debug block being able to detect changes to a block (see my post with air vs. stone), so I suspect it is the latter, and is actually querying the worldfile in the same way you describe bluemap watches the world file.

If it is indeed the latter, than what you described seems to somewhat conflict with the experiment I conducted in my previous post. Bluemap seems to recognize that the block is new, and according to the logs it even schedules an update, but the update never occurs. I am claiming this based on running the bluemap command, which reports the 'last render' time. I have never seen this time be after a server restart or manual re-render.

So your issue lies with your minecraft server not saving the chunks to the actual files as fast as you would like it to

I looked into how paper handles automatic saving. Paper's world config file allows auto-save-interval to be manually adjusted. When this value is left default, like mine is, it uses the Bukkit/Spigot default value of 6000 ticks, or 5 minutes. So theoretically, I should not have to wait any longer than 5 minutes for a render / update to start. In my case, they never start.

[TBlueF]

Out of curiosity, does bluemap debug block pull information about the block directly from the server?

Yes that command loads the block from the world-files..

So theoretically, I should not have to wait any longer than 5 minutes for a render / update to start. In my case, they never start.

In theory, but there also is some extra IO-caching going on with paper/spigot that might mess with those values.
Anyways this would be something to discuss with the paper/spigot devs ^^'

---

Okay, sorry to be like this,.. but what do you want me to do now? I can't reproduce your issue, it's working fine for me on my own paper server (default values) and seems to work fine for the over 8000 other servers using bluemap too. Your logs and debug commands are fine, and the best i can do right now is explaining to you how bluemap's update system works..

I am glad to explain every detail to you, but a GitHub issue is kinda the wrong place for this.

[Samg381]

what do you want me to do now

I have to admit, this is the first time I've been asked that in a bug report.. I always assumed that was the developer's job 😁

I am running a completely default paper / bluemap install too.. so I provided every detail I could figuring this might be of interest to investigate, but perhaps not. Thanks for your time I suppose.

[TBlueF]

I mean .. i am literally out of ideas how i can investigate this further ^^'

I am still not convinced this is actually a bug - please excuse me - you wouldn't believe how many times people were claiming their updated were not working and after some investigation it always just turned out to be some cache, frozen-map, stopped bluemap-renders or impatience on the user-end .. mostly a combination of multiple of those..

But alright,.. I'll spin up another fresh paper testserver in a bit and see if i can reproduce your issue somehow 😅

[Samg381]

I'm definitely open to the possibility that this is impatience on my end- I will admit I am prone to this type of disposition- but feel like I should have definitely been able to observe a normal update by now, and that something is off..

If you do spin up a fresh paper server (I am running 1.21.3), and you cannot reproduce the error- perhaps you could send me the entire server directory so I can test it on my side? That is one potential troubleshooting route.

Also, for the sake of scientific consistency, let's make the test simple - placing a single block of dirt on the map and seeing how long it takes to appear on bluemap without any commands or server restarts.

I have run the test and am not able to get the block to appear at all. According to the paper docs, it should take no longer than (roughly) 5 minutes, give or take the IO stuff as you said.

[TBlueF]

Alright, i gave it a spin, here are my results...

First test:

I could reproduce the issue you were having. I created the server, joined, went to somewhere outside the spawn-chunks for consistency, let everything run until idle.. made sure the map is up to date.

• Then i placed a block, and waited 5 minutes -> nothing
• I teleported my player to a completely different location to not load the chunks and waited 5 minutes -> it rendered the newly generated regions, but didn't update the block i placed
• I disconnected and waited 5 minutes -> still nothing

But:
I also monitored the region-file timestamps in my file-explorer, especially the region file that I placed the block in.
It also never changed after placing the block!
Only after actually running save-all flush the region-file including the timestamp of the region-file changed, and bluemap instantly started rendering the region and the block was finally visible.

Second Test:

• I restarted the server, joined again and went back to the block i placed .. (outside the spawn-chunks)
• I ran save-all flush once more and waited for everything to render and be updated to have a clean start
• I placed a second block
• I picked a direction and started flying ...

After ~2000 blocks of flying bluemap started rendering the regions i left behind ... including the one i placed the second block in
-> the block was visible on the map.

Here is my guess of what is going on:

When paper is loading a region-file, it holds the file open for as long as possible -> it writes chunks it unloads to the file, but never "closes"/"releases" the file, until it has a specific amount of region-files open, and then it starts closing the least recently accessed or least recently opened files. This is done to maximize IO-speed for loading/saving chunks.
The problem: Only when the file is closed/released, the OS is actually updating the meta (-> timestamp) and notifying the file-watchers (-> bluemap) that there were changes to said file.
This will happen more frequently the busier the server is and the more players move around, since the threshold for closing files will be reached more often. However in our simple test-scenario this is not happening at all, since we are not moving/doing enough to fill the caches.
Running save-all flush causes paper to close/release all files, and reopen them. -> bluemap updates.
This is also why we see the changes in the debug-commands earlier then when they are actually rendered. Since the data is already written to the world-file, its just not closed yet so the file-watcher didn't trigger yet.

Conclusion:

BlueMap's update-system is working fine / as-designed, its just very difficult to cause paper to consistently finalize a region-file change frequently, especially in a test-scenario.
At some point paper will always have to close the file, and bluemap will update, even if it sometimes takes a long time.

The only way to detect changes and render them without waiting for the file-watchers is to make bluemap actively re-scann every single region-file and every chunk-timestamp for changes. Which is happening when bluemap loads, with a /bluemap update or every time-interval defined by the full-update-interval config in the plugin.conf (by default every 24h).

[Samg381]

Thank you very much for the thorough write-up and replication of my environment + issue. This seems to track with what I am observing on my side. I am indeed running a very small server (3-6 players, rarely concurrent, most of them around the spawn point) - so it is conceivable that paper rarely has a need to fully close the file handle on the spawn chunk.

I ran a similar experiment at midnight last night, where I placed a new block one chunk away from spawn. It is now 10 hours later, and bluemap still hasn't updated. No players have joined since I placed the blocks.

I am a little surprised that paper wouldn't periodically close / reopen the file handle, but I suppose it makes sense given the data is still being written to the file, meaning it is still being backed up in case of a crash.

I did a deeper dive on the paper backend. It turns out the term 'flush' means saving the chunk to the region file AND closing the file descriptor/handle, which I had misunderstood. It makes sense that bluemap is able to pick up on changes when save-all flush is run, since the descriptors are closed.

I had previously mentioned a paper flag called auto-save-interval that defaults to Spigot's 6000 ticks (5 minutes). It turns out, this flag is meaningless without mentioning paper's other flag called flush-regions-on-save, which is set to false.

I am going to try setting flush-regions-on-save, to true to see if that can workaround this issue.

---

Closing thoughts

In my eyes, this situation is neither paper, nor bluemap's fault. Both are technically working as designed. I would, however, posit that bluemap could likely implement it's own, fairly lightweight, change detection, by checking the hashes of each region file at it's own pace instead of waiting for the file descriptors to close- thereby determining when an update is needed completely on it's own. There would obviously need to be some logic implemented to prevent checks during saves, but I think this might be worth overcoming.

The end goal would be an abstracted update-interval parameter for bluemap that would allow server operators to specify how often each chunk is queried for changes. As it stands, there is no direct way to configure this behavior, leaving the update interval to be rather long and unpredictable as seen in my long duration experiments. In my eyes, this is a fairly confusing and unclear arrangement (again, not bluemap's fault) that leads to inconsistencies in behavior across server sizes, platforms, and use cases.

Perhaps the above idea could be filed in the idea board / roadmap.

[TBlueF]

Whats the difference between your suggestion and the existing plugin.conf -> full-update-interval?

[Samg381]

full-update-interval forces a full re-render of the entire map, no?

My suggestion would allow bluemap to perform the more granular 'updates' without the need to rely on paper's shoddy and unpredictable file descriptor open/close behavior.

[TBlueF]

full-update-interval forces a full re-render of the entire map, no?

No, it does a full scan of all region-files/chunks for changes, and only renders the changes -> Simplified: it compares the timestamp of each chunk with the last rendered one and renders only if the timestamp diverges.
The same thing /bluemap update <map> does.

Fully re-rendering the entire map on each start/every 24h? .. you must be insane ;D

[Samg381]

No, it does a full scan of all region-files/chunks for changes, and only renders the changes -> Simplified: it compares the timestamp of each chunk with the last rendered one and renders only if the timestamp diverges.

Okay, but this still relies on the file descriptors/handles being closed, right? In other words, it is still at the mercy of whether paper has decided to flush the changes to disk- which as we've observed is very unpredictable.

[TBlueF]

No, the chunk-timestamps are not dependant on the files being closed, they are dependant on the actual data in the files, since they are actual data in the region-files.
Nothing to do with the file-timestamps.

Each time minecraft writes a chunk to the region-file, it also updates a timestamp for that chunk. Baked into the region-file format. That's the only thing bluemap ever uses to determine if a chunk has changed or not. Hashing the entire files or individual chunks would be too slow.

[Samg381]

Okay, well I clearly need to do a deeper dive into how bluemap handles this. Fair enough.

I suppose my suggestion really just boils down to addressing whatever is causing my world to not update for 10+ hours with a completely vanilla paper + bluemap install. Perhaps this is simply changing full-update-interval as you said, maybe to something like five minutes, or something more in line with how a novice user might expect bluemap updates to take. Yes, it is paper's fault that the file watchers can take a very long time to get updated. I get that. I am just trying to think if there are any easy ways for bluemap to take the stock behavior of paper into account.

I'll step off my soapbox 😶

[TBlueF]

I am just trying to think if there are any easy ways for bluemap to take the stock behavior of paper into account.

Sure! :D .. just .. I spent a ton of time and multiple update-system reworks to think about this exact question. And I can't think of any better way to handle updates than bluemap does right now. And to this time no-one has presented an (in my opinion) better solution to me either.

[TBlueF]

I hope it's ok if I'll close this for now :)