Bug: Cache Preloading not always kicking in after cache purge from publishing event

[porg]

I debugged W3TC 's Cache Preload feature intensively over the weekend & came to these findings:

1. Cache Preload's initial buildup from Automatically prime the page cache works fine.
   • Both after clicking Save Settings & Purge Caches in the settings.
   • or a manual SSH/FTP deletion of /wp-content/cach/page_enhanced/domain-directory/.
2. But Preload the post cache upon publish events works not reliably. In some tests it kicked in:
   • ✅ reliably within what I calculated as the total Preloading Loop Duration (see in: Debugging Setup)
     • though still too late to make a difference for editors, better would be:
       Regenerating edited page immediately with top priority would benefit UX for the first requestor of the updated page #686
   • ❌ not at all, also not hours later.

My suspicion

• The offset with which the Preload Cache stops seems to play a role whether at all and how it continues.
• From my observations I deduced that the offset can possible get stuck at the loop end also for later loops.

My Website Usability and Performance Goals

My Environment + My Goals in full there and summarized here:

3. Don’t want the first visitor of a new or purged cached page to await page cache generation. Hence:
   • a) ☑︎ Automatically prime the page cache
   • b) ☑︎ Preload the post cache upon publish events
4. Want my users to always have a chance to get the most recent version of a page/post. Hence:
   • Browser Cache → HTML/XML → Cache-Policy: “cache with validation”
     • PRO: Reliably always get most recent content with only a minimal overhead: One roundtrip of HTTP metadata exchange, no content download if unchanged.

Debugging Setup

• Cache Preload
  • ☑︎ Automatically prime the page cache
  • Update interval: 15 secs
  • Pages per interval: 5
  • Sitemap URL: https://mydomain.com/mini-sitemap.xml
  • ☑︎ Preload the post cache upon publish events
• /mini-sitemap.xml manually created for debugging, containing only 30 URLs in total:
  • Only pages, no other content-types.
  • Level 0 /: 5 top level menu entires (/blog/ omitted)
    • Level 1: 15 pages in /skills/ and /work/
      • Level 2: 10 pages in /skills/profile/ and /skills/analysis/

Preloading Loop Duration: 30 URLs in batches of 5 = 6 batches total * 15secs per batch = 90secs = 1.5 mins

Video Recording

Window setup used throughout the video:

Nicely documented

• I put a lot of effort into presenting this as compact and comprehensible as possible in a video screen recording.

  • Cut out any pauses or repetitions, only pure information.
  • Divided into different chapters, for efficient access & reference.
  • Only problem: I have to provide you the video(s) with some compromise.

• ▶️ Youtube video with chapters + machine transcript

  • ❌ Has choppy audio here & then from chapter 2 onwards. Still intelligible but possibly annoying.
  • ❌ Chapters as individual compressed MP4s suffer the same choppy audio.
    • I regret that I had cut the individual files with QuickTime Player (and once had an intermittent crash)
    • I guess it introduced some markers/encodings on which ffmpeg / Handbrake / FinalCut / Youtube all choked.

Alternative: Each chapters as standalone video file

• ℹ️ The pure .MOV files are possibly but not necessarily are better audio-wise.

01 Windows and Setup Explained

04.Deleting.entire.domain.page.cache.then.observing.one.full.Cache.Preload.cycle.mov

02 Edited then Purged but Preloading never

01.Windows.and.Setup.Explained.mov

03 Cached pages outside of sitemap overnight

05.Edited.then.Purged.then.Preloading.worked.mov

04 Deleting entire domain page cache then observing one full Cache Preload cycle

02.Edited.then.Purged.but.Preloading.never.mov

05 Edited then Purged then Preloading worked

03.Cached.pages.outside.of.sitemap.overnight.mov

06 Observing Preloading in Command Line with wp-cli

• As MP4 with choppy audio (11 MB), as the original MOV (126 MB) was above Github's maximum of 100 MB

W3TC--06-Observing-Preloading-in-Command-Line-with-wp-cli.mp4

[porg]

I'd appreciate to get a response to this!

If you don't have the time to read the original report, let's make it short:

• There is a Purge Log (in the PRO version, which I now have)
  • This shows clearly that a page's cache is immediately purged when updating a page in the Block Editor.
• But there is no Cache Preload (Priming) log.
  • It's impossible to debug the Preload Loop.
  • I would love to have such a log too, which shows at what time wp-cron kicks in another round of Cache Preload, and at what offset the loop currently is.
  • And understand why an updated page does not get recreated immediately at utmost priority, but instead only minutes/hours later with the next preload loop, or by request of that poor user that requests the updated page (and has to await the full PHP page generation time).

Besides all the debugging I still recommend to give the recreation of a purged page utmost priority (temporarily queue in the updates page(s) in another "urgent preload loop", or temporarily change the main loop offset to there, and then back to the last offset number), however you may do that technically, as I proposed in:

• Regenerating edited page immediately with top priority would benefit UX for the first requestor of the updated page #686

[porg]

After updating /my-page/ I do not want to perform any of these workarounds:

• Request /my-page/ anonymously so that the page cache is ready immediately for new visitors.
• Run the wp-cli preload cache from position 1 to 10000 (effecticely all pages of my website) just to also regenerate /my-page/ with that. That feels like shooting with a huge bomb on a tiny insect.

This should work more intelligently. Regenerate what was recently purged. I mean that priority is really obvious. Please.

[porg]

So my report has been confirmed as reproducible? And the symptoms could be fixed?

[cssjoe]

@porg This issue has been attached to a pull request that was merged and is staged to be released in W3 Total Cache 2.7.2. We have added a couple of settings that can be enabled to prime the cache for pushed and updated posts/pages.