Add Xdelta support for softpatching

[JesseTG]

Description

Adds support for the xdelta format to softpatching. Works the same as IPS, UPS, and BPS, in that xdelta patches are applied to ROMs with the same name (in supported emulators).

I've tested this patch with a Nintendo 64 ROM hack, but before it's merged I'd like to get some feedback on it to ensure I'm not missing any edge cases.

[JesseTG]

Whoops, I noticed that there are some compiler errors on Linux. (I've been developing this patch on Windows.)

It has to do with xdelta's SIZEOF_UNSIGNED_LONG_LONG macro. unsigned long long isn't available in C90, so I'll just go ahead and remove uses of it.

[lgtm-com]

This pull request introduces 2 alerts when merging a9756fe into 2f7c298 - view on LGTM.com

new alerts:

• 2 for Comparison of narrow type with wide type in loop condition

---

Heads-up: LGTM.com's PR analysis will be disabled on the 5th of December, and LGTM.com will be shut down ⏻ completely on the 16th of December 2022. Please enable GitHub code scanning, which uses the same CodeQL engine ⚙️ that powers LGTM.com. For more information, please check out our post on the GitHub blog.

[lgtm-com]

This pull request introduces 6 alerts when merging 62d81e5 into 2f7c298 - view on LGTM.com

new alerts:

• 3 for Wrong type of arguments to formatting function
• 2 for Comparison of narrow type with wide type in loop condition
• 1 for Use of goto

---

Heads-up: LGTM.com's PR analysis will be disabled on the 5th of December, and LGTM.com will be shut down ⏻ completely on the 16th of December 2022. Please enable GitHub code scanning, which uses the same CodeQL engine ⚙️ that powers LGTM.com. For more information, please check out our post on the GitHub blog.

[lgtm-com]

This pull request introduces 6 alerts when merging 594b48e into 2f7c298 - view on LGTM.com

new alerts:

• 3 for Wrong type of arguments to formatting function
• 2 for Comparison of narrow type with wide type in loop condition
• 1 for Use of goto

---

Heads-up: LGTM.com's PR analysis will be disabled on the 5th of December, and LGTM.com will be shut down ⏻ completely on the 16th of December 2022. Please enable GitHub code scanning, which uses the same CodeQL engine ⚙️ that powers LGTM.com. For more information, please check out our post on the GitHub blog.

[i30817]

Personally, if there is no maximum limit, which i tend to agree with, i believe there should by a soft limit, based on a percentage of the computer physical memory (not 'virtual' memory).

After all, even if the core is outdated, dolphin is a core, so the maximum size of a 'patched' file is 8gb or so.

Unless this patching is done with temporary files, in which case i don't want to use it.

[lgtm-com]

This pull request introduces 6 alerts when merging dbb4af4 into 2f7c298 - view on LGTM.com

new alerts:

• 3 for Wrong type of arguments to formatting function
• 2 for Comparison of narrow type with wide type in loop condition
• 1 for Use of goto

---

Heads-up: LGTM.com's PR analysis will be disabled on the 5th of December, and LGTM.com will be shut down ⏻ completely on the 16th of December 2022. Please enable GitHub code scanning, which uses the same CodeQL engine ⚙️ that powers LGTM.com. For more information, please check out our post on the GitHub blog.

[lgtm-com]

This pull request introduces 6 alerts when merging 97efcc3 into 2f7c298 - view on LGTM.com

new alerts:

• 3 for Wrong type of arguments to formatting function
• 2 for Comparison of narrow type with wide type in loop condition
• 1 for Use of goto

---

Heads-up: LGTM.com's PR analysis will be disabled on the 5th of December, and LGTM.com will be shut down ⏻ completely on the 16th of December 2022. Please enable GitHub code scanning, which uses the same CodeQL engine ⚙️ that powers LGTM.com. For more information, please check out our post on the GitHub blog.

[lgtm-com]

This pull request introduces 6 alerts when merging 0c04762 into 2f7c298 - view on LGTM.com

new alerts:

• 3 for Wrong type of arguments to formatting function
• 2 for Comparison of narrow type with wide type in loop condition
• 1 for Use of goto

---

Heads-up: LGTM.com's PR analysis will be disabled on the 5th of December, and LGTM.com will be shut down ⏻ completely on the 16th of December 2022. Please enable GitHub code scanning, which uses the same CodeQL engine ⚙️ that powers LGTM.com. For more information, please check out our post on the GitHub blog.

[JesseTG]

Could someone please grant me approval (at least within this PR) to run workflows without having to await approval? There are a lot of environments I need to validate this patch against, and it's difficult to iterate locally.

[JesseTG]

@i30817 Sorry, I missed your message over the @lgtm-com spam.

Personally, if there is no maximum limit, which i tend to agree with, i believe there should by a soft limit, based on a percentage of the computer physical memory (not 'virtual' memory).

After all, even if the core is outdated, dolphin is a core, so the maximum size of a 'patched' file is 8gb or so.

This seems sensible, but I'm not convinced that it would be the best use of my time right now. The cores that load games from ISOs (or other large disc image formats) don't support softpatching yet, so such an exercise would largely be academic. That mostly leaves Nintendo 64, DS, and Game Boy Advance games, which don't usually exceed 128MB (and that's pushing it). A device that can't spare that is unlikely to be able to run such games well anyway. On top of that, games for older/simpler platforms like the Genesis or Game Boy usually don't get patches in xdelta format (IPS, UPS, or BPS are more common).

I'd be happy to revisit this once a few of the major disk-based cores that are known to run well on legacy/underpowered devices get softpatching support.

But if the maintainers demand it now, I'll oblige.

Unless this patching is done with temporary files, in which case i don't want to use it.

Nope, it's all in-memory. The steps are as follows:

1. Use Xdelta to make a pass over the patch file to compute the exact size of the patched file. This is non-trivial, because there isn't a single "target file size" number within Xdelta patches. It's all computed on a per-block basis. This does not emit patched data, and it doesn't allocate much memory (a few kilobytes).
2. Once we know exactly how big the final ROM will be, allocate a buffer for it with malloc.
3. Make a second pass over the patch file, this time actually patching the ROM.

If there's not enough memory to allocate the patch (i.e. if malloc returns NULL), I clean up after myself and return an error. So if a patching a file would chew up more memory than is available, we'll find out before allocating anything. At this point, the game should proceed to run unpatched (assuming there's enough memory for that).

[lgtm-com]

This pull request introduces 6 alerts when merging 3c57abd into 7b09cb2 - view on LGTM.com

new alerts:

• 3 for Wrong type of arguments to formatting function
• 2 for Comparison of narrow type with wide type in loop condition
• 1 for Use of goto

---

Heads-up: LGTM.com's PR analysis will be disabled on the 5th of December, and LGTM.com will be shut down ⏻ completely on the 16th of December 2022. Please enable GitHub code scanning, which uses the same CodeQL engine ⚙️ that powers LGTM.com. For more information, please check out our post on the GitHub blog.

[LibretroAdmin]

I keep stating this - it remains a huge issue that for first-time contributors or any other contributor outside the organization on GItlab, it remains impossible to run a CI lab (even on request by say me). This is a huge blindspot, huge issue, and it keeps screwing us up even despite the numerous Github action CI jobs we already have implemented, because it is just not possible to catch every single platform and toolchain out there.

I really hope in the future our Gitlab CI can become more user friendly and just allow us to run a custom job on request when we need to check if a PR builds against our buildbot targets. That would be a ton better than finding out it doesn't compile, is non-salvageable, and then having to revert all these commits later on.

[hizzlekizzle]

it takes some effort on our end, but it seems like setting up a branch and merging the PR to that instead of directly to main/master does an okay job of letting us run the CI jobs for testing (like we're doing with HSM's PR)

[JesseTG]

As I mentioned to @LibretroAdmin privately, I think act solves that problem pretty nicely. It lets you run workflows locally. Over the next few days I'll submit a PR that ensures the workflow files don't break in act; they're mostly fine, I just need to get them across the finish line.

[Magicrafter13]

Is this still something in consideration, or was this basically axed after being reverted?

[JesseTG]

Is this still something in consideration, or was this basically axed after being reverted?

So glad you asked!