Scraping dies with "Input buffer contains unsupported image format" if logo returns 301

[benoit74]

Zimfarm recipe: https://farm.openzim.org/recipes/encyclopediaofmath.org_en_all

Zim-request details: openzim/zim-requests#964 (comment)

Log:

[error] [2024-05-22T12:56:44.462Z] Failed to run mwoffliner after [18s]: {
	"stack": "Error: Input buffer contains unsupported image format",
	"message": "Input buffer contains unsupported image format"
}
[error] [2024-05-22T12:56:44.462Z] 

**********

Input buffer contains unsupported image format

**********

It is pretty hard to tell which image has been grabbed and failed to be read, making the fix even more delicate.

[audiodude]

From googling, it looks like that's an error message from the sharp library.

Probably happening here?

mwoffliner/src/Downloader.ts

Line 488 in 9cc613f

.buffer(await sharp(resp.data).toColorspace('srgb').toBuffer(), imageminOptions.get('webp').get(resp.headers['content-type']))

Might just be a bad URL that's getting erroneously read as image data.

[audiodude]

The proximate cause of the error is the fact that the URL:

https://encyclopediaofmath.org/common/spr_logo.gif

Does a 301 to https://encyclopediaofmath.org/wiki/Main_Page

Presumably, this "logo" link is somewhere in the initial metadata that mwoffliner gathers about the wiki. So the bug is that mwoffliner is hardcoded to download this as image data and doesn't consider 301 to be an error status. It then crashes early on in the scraping process.

[audiodude]

If the folks putting in the request would like to fix their problem without waiting for mwoffliner, I would suggest putting an image (even a 1x1 PNG) at that URL.

[benoit74]

Thank you!

Would specifying a custom favicon with --customZimFavicon allow to bypass the code trying to download the "logo"?

[audiodude]

Yes, I believe that would work

[kelson42]

Here the solution is IMHO to test the format of the image at the same time (early) like other ZIM metadata.