Zimit 2 tests: mesquartierschinois

[benoit74]

A new Zimit2 ZIM is ready for testing

Scraper: warc2zim 2.0.0-dev9 + zimit 2.0.0-dev5 + Browsertrix crawler 1.1.3
ZIM: https://mirror.download.kiwix.org/zim/.hidden/dev/mes-quartiers-chinois_fr_all_2024-05.zim
Library link: https://dev.library.kiwix.org/viewer#mes-quartiers-chinois_fr_all_2024-05

Suggested test plan:

• Open the ZIM, check homepage and 5 subpages (randomly chosen) loads properly
• Perform a suggestion search and open any suggestion
• Perform a full-text search, check result quality and open any result
• Confirm Youtube video plays correctly (audio + video), see e.g. Girona article

Known issues:

• only first seconds of Vimeo videos are working (this is not planned to be fixed soon)

@Jaifroid, still welcomed (I sent you an invite to this repo with Read rights so that we can at least tag you correctly, if you don't mind of course)

[Popolechien]

Kiwix-server: infinite scroll does not work and new entries do not load. Link on the right (categories) return an error but this I suspect is more a dev.library.kiwix.org issue as it is intermittent. Clicking on "related posts" at the bottom of an individual entry returns an error Not found

macOS: file does not open at all Unable to load the article requested.

Kiwix-JS (Chrome): video not showing (still black screen), rest looks great. I do get an error trying to click Related posts at the beginning of an individual entry. It first tells me that I'm trying to access an external page, and when I accept it actually returns an empty search.

Android: infinite scroll does not work and new entries do not load (or rather: it loads to a certain point and where I should see a Previous page link I just get the endlessly rotating circle.

[Jaifroid]

@Jaifroid, still welcomed (I sent you an invite to this repo with Read rights so that we can at least tag you correctly, if you don't mind of course)

Accepted, thanks.

[Jaifroid]

In PWA and Browser Extension (installed via Chromium browser) the video is working for me on Windows. @Popolechien, for the Borwser Extension, please check it says v4.0.1 top left?

EDIT: The Vimeo video also working (first 11 secs) in both as well. That's a big advance, because Vimeo wasn't working at all in previous test ZIMs! 😃 Same with Firefox Browser Extension (4.0.1) -- all working for me on Windows.

[Popolechien]

Ah no it's the 4.0.0

How come this does not update automatically? I feel cheated!
Let me look into this and revert back.

[Jaifroid]

@Popolechien No, not your fault. Please check in Configuration that you're using ServiceWorker mode, not ServiceWorkerLocal mode. To run a lot of inline JS in the browser, the app needs to be in ServiceWorker mode, which will access the server to cache the latest code. That should then show v4.0.1.

[Popolechien]

Yup we're good. But then now I also get the endless wheel of loading at the bottom of the page:

[Jaifroid]

Is that on the Home / landing page? I don't get it there, but I do get it on other pages. I think it's a scraper issue.

[Popolechien]

Home page, but also on individual posts. I think it's a scraper issue yes because on one of the pages I can see the Chargement (loading) being hardcoded text

[Popolechien]

Ah found another issue when clicking on Related posts at the bottom. Updated my earlier comment.

[Jaifroid]

For me, all other tests completed satisfactorily in Chromium and Firefox on Windows. Infinite scroll works fine EDIT: works partially from Home page, but not on some other pages in either browser (I think this may be because the crawler may not have scrolled those pages to end? -- I don't think it's an intrinsic issue).

Also in the PWA (Chrome) on Android -- both videos working there too! Again, another advance, because video was broken on Android in many previous versions due to differences between videos requested on desktop and mobile.

In the PWA installed via Safari on an iPad in iOS 17 (via Browser Stack), YouTube videos appear to start loading, but then don't get the correct version for iPad and bail out. However, the good news is that the Vimeo video works in this context (first 11 secs). Infinite scrolling works EDIT: works partially on landing page, but not on other pages (it pulls several articles in but then stops after about a page).

I'm having difficulty getting a console.log readout on the iPad due to flakey BrowserStack implementation of DevTools on iOS... (to debug video). I'll persevere.

[Jaifroid]

@Popolechien I see what you mean about infinite scrolling on Landing Page, sorry -- it gets a long way, as far as "Supersize Me", and then stops. I suspect the crawler stopped there. When I said infinite scrolling was working, it was because I hadn't scrolled so far down!

[Jaifroid]

Ah found another issue when clicking on Related posts at the bottom. Updated my earlier comment.

I confirm "Articles similaires" are not scraped, and appear as external links. This may be a domain scope issue.

[Jaifroid]

Finally got a console readout for the PWA on iPhone 15 Pro Max with iOS 17 Safari (text below image - this shows issue with YouTube video Girona). Note that in this format (iPhone as opposed to iPad) the Vimeo video is also not found. I suspect a specific iPhone version is being requested by the player, and this is not scraped. This is going to be a significant issue I think, unless we can find a fallback mechanism to forcibly inject the MP4 that was scraped into the video player (this is something that @mgautierfr tried with some success previously).

urlRewriten:
	- current_url: https://pwa.kiwix.org/mes-quartiers-chinois_fr_all_2024-05.zim/C/youtube.fuzzy.replayweb.page/embed/1T3u9jjXcGM
	- orig_host: www.youtube.com
	- orig_scheme: https
	- orig_url: https://www.youtube.com/embed/1T3u9jjXcGM?version=3&rel=1&showsearch=0&showinfo=1&iv_load_policy=1&fs=1&hl=fr&autohide=2&wmode=transparent
	- prefix: https://pwa.kiwix.org/mes-quartiers-chinois_fr_all_2024-05.zim/C/
	- url: https://i.ytimg.com/vi/1T3u9jjXcGM/sddefault.jpg?sqp=-oaymwEmCIAFEOAD8quKqQMa8AEB-AG-B4AC0AWKAgwIABABGHIgVig-MA8=&rs=AOn4CLBoBLbDfVnzoGC5qO5p5z-AKuvyIg
	- useRel: false
	- mod: im_
	- doc: undefined
	- finalUrl: https://pwa.kiwix.org/mes-quartiers-chinois_fr_all_2024-05.zim/C/i.ytimg.com.fuzzy.replayweb.page/vi/1T3u9jjXcGM/thumbnail.jpg
	
VM20 wombatSetup.js:3 urlRewriten:
	- current_url: https://pwa.kiwix.org/mes-quartiers-chinois_fr_all_2024-05.zim/C/youtube.fuzzy.replayweb.page/embed/1T3u9jjXcGM
	- orig_host: www.youtube.com
	- orig_scheme: https
	- orig_url: https://www.youtube.com/embed/1T3u9jjXcGM?version=3&rel=1&showsearch=0&showinfo=1&iv_load_policy=1&fs=1&hl=fr&autohide=2&wmode=transparent
	- prefix: https://pwa.kiwix.org/mes-quartiers-chinois_fr_all_2024-05.zim/C/
	- url: https://i.ytimg.com/vi/1T3u9jjXcGM/sddefault.jpg?sqp=-oaymwEmCIAFEOAD8quKqQMa8AEB-AG-B4AC0AWKAgwIABABGHIgVig-MA8=&rs=AOn4CLBoBLbDfVnzoGC5qO5p5z-AKuvyIg
	- useRel: undefined
	- mod: undefined
	- doc: undefined
	- finalUrl: https://pwa.kiwix.org/mes-quartiers-chinois_fr_all_2024-05.zim/C/i.ytimg.com.fuzzy.replayweb.page/vi/1T3u9jjXcGM/thumbnail.jpg
	
bundle.min.js:24 Title C/www.youtube.com/api/stats/qoe?fmt=18&cpn=yqsCyjsNQbmf6n6y&el=embedded&ns=yt&fexp=v1,24004644,76094,54572,304051,104308,16747,6271,26443548,7111,36343,9954,119... not found in archive.
bundle.min.js:24 Title C/i.ytimg.com.fuzzy.replayweb.page/vi/uapAbBOcy6U/thumbnail.jpg... not found in archive.

[benoit74]

Thank you!

infinite scroll not working

This is not a zimit2 issue, this is an bad recipe configuration. Autoscroll is needed on this website, never been configured, never worked.

Link on the right (categories) return an error but this I suspect is more a dev.library.kiwix.org issue as it is intermittent.

I can't repro this, is anyone else impacted?

macOS: file does not open at all Unable to load the article requested.

Issue is probably same as fas-military-medicine, I commented in this one at least: kiwix/kiwix-apple#779

Safari Youtube video

Errors you get are not related AFAIK, all missing thumbnails are just thumbnails (i.e. not the video itself) and come from the preview of linked videos that Youtube player would like to show, crawler did not fetched them. Maybe we should add a generic custom CSS to hide all these preview which are not available anyway. And the error on /api/stats looks a lot like a tracker (which obviously is not in the ZIM ...).

I reproduce the error on pwa.kiwix.org on Safari on MacOS, so this is an issue of PWA on safari as far as I can tell. Especially since same video in same ZIM in same Safari on same machine works well. And it also works on PWA on Firefox on same machine. No reason for the JS player to choose a different URL only because it is running inside a PWA on Safari. Probably many reason for the PWA to do something silly on Safari due to all Safari subtleties.

I also reproduced the error with other ZIMs, so not specific to mesquartierschinois.

I do not trust the argument about specific video format for specific reader anymore, did not saw it, so far problems on Youtube videos was always related to ZIM content (improper URL rewriting or Wombat injection). And again, does not work on PWA while it works on kiwix-serve.

I opened kiwix/kiwix-js#1256

Clicking on "related posts" at the bottom of an individual entry returns an error Not found

This is an interesting one... opened openzim/warc2zim#291

[benoit74]

Nota: shouldn't we get rid via custom CSS of these two sections:

Both are mostly meaningless in an offline context

[benoit74]

Autoscroll is needed on this website, never been configured, never worked.

Recipe updated, not relaunched since we have too many other things to solve first

[Jaifroid]

Just to confirm that the Girona video works when served from Kiwix Serve on iOS and macOS Safari (tested also on iPhone and iPad via BrowserStack). So the problems with the PWA and Browser Extension playing Zimit2 YouTube video in Safari (only!) is indeed a weird issue on my side. ☹️

[Popolechien]

shouldn't we get rid via custom CSS of these two sections:

Not for this specific website as this is only a test one. If however we can make it a regular feature for all other wordpress blogs out there, then yes.