Releases: Nexus-Mods/NexusMods.Archives.Nx
0.6.2
Today I am thrilled to announce the release of NexusMods.Archives.Nx 0.5.0,
now with a stable file format!
Major Features and Changes
Archive Repacking ⚡
This release introduces a powerful new capability: 'Repacking'.
This feature allows users to create new derivative archives, using the data of existing archives
as base. This works by copying data from existing SOLID Blocks, Chunked Files and partial chunks of
SOLID blocks.
In simpler terms, this works by copying already compressed data from existing archives, and
only compressing new data whenever necessary.
Practical Use: File Deletion
It is now possible to delete files from an archive in a very efficient manner.
Moreover, this can be efficiently done without the risks of corruption should the archive be
modified in-place and the system were to lose power, the process be or the application were to crash.
This feature in particular enables a feature called 'Garbage Collection' within the
Nexus Mods app, which allows us to reclaim unused
space from files that are no longer needed.
Practical Use: Archive Merging
This feature can also be used to merge multiple archives in an extremely efficient manner.
This feature was used to benchmark the performance of the repacking feature during development
and is available via the CLI.
Merging enables the following practical potential use cases in the Nexus Mods app:
Quickly sharing a work-in-progress mod to your friend:
In the Nexus Mods app, each archive stores a set of unique files. These archives can now be
merged to output a single archive that another user can then install a mod from.
Don't waste 4 minutes of your life, compressing your 2GB texture mod. Get a working,
shareable Nx archive in 0.5 seconds instead.
'Compacting' data during 'Garbage Collection':
The introduction of 'Garbage Collection' will eventually lead to 'Fragmentation' of archives in the
Nexus Mods app.
Deleting older versions of mods and performing 'Garbage Collection' risks leaving tiny small archives
that will all need to be read, parsed and processed independently in order to apply a single mod
from a loadout.
With the new merging feature, now it will be possible to merge multiple archives belonging to a single
mod as part of the 'Garbage Collection' process. This will improve the efficiency of the Apply operation
used to switch to a loadout.
Practical Use: Accelerated Archive Packing
Since the repacking feature copies data from existing archives, it can also be used to create entirely new
archives faster, if you know the location of some data you want to pack again.
To provide a practical example, suppose you have Mod 0.1.0, and you want to release an update,
Mod 0.2.0. This update is mostly the same, but with a few new files.
Now it is possible to use the repacking feature to copy the compressed data from Mod 0.1.0
and only compress the new files for Mod 0.2.0.
Using the Repacking Functionality from Code
This is very easy!!
Use the NxRepackerBuilder in place of the NxPackerBuilder API in your code.
// Use any IFileDataProvider to provide the existing archive.
// Ideally memory mapped provider, i.e. FromFilePathProvider
// if file is on disk.
var provider = new FromFilePathProvider() {
FilePath = "existing.nx"
};
var header = HeaderParser.ParseHeader(provider);
var repackerBuilder = new NxRepackerBuilder();
// Add files from existing archive
repackerBuilder.AddFilesFromNxArchive(nxSource, header, header.Entries.AsSpan());
// Configure output
repackerBuilder.WithOutput(new FileStream("repacked.nx", FileMode.Create));
// Build the repacked archive
using var outputStream = repackerBuilder.Build();If you want to merge archives, use the NxDeduplicatingRepackerBuilder instead,
this will automatically deduplicate files as they are added.
Performance Preview (How fast is Repacking?)
We will use the merge feature from the CLI.
Merging Skyrim 202X 10.0.1 (Architecture) and Skyrim 202X 9.0 (Architecture):
// NexusMods.Archives.Nx.Cli merge --output "Skyrim202X-merged.nx" --sources "Skyrim202X 9.0.nx" "Skyrim202X 10.0.1.nx" --deduplicate-chunked false --deduplicate-solid false
Merged in 6439ms
Input Size: 19432.68 MiB
Output Size: 11972.45 MiB
Compression Ratio: 61.61 %
Throughput 3955.06MiB/s
Repacking also works with the new deduplication feature, which is enabled by default
for merging.
// NexusMods.Archives.Nx.Cli merge --output "Skyrim202X-merged.nx" --sources "Skyrim202X 9.0.nx" "Skyrim202X 10.0.1.nx"
Merged in 5471ms
Input Size: 19432.68 MiB
Output Size: 11290.83 MiB
Compression Ratio: 58.10 %
Throughput 4654.84MiB/s
The new repacking functionality runs as fast as the write speed of my NVMe drive.
Deduplication
This release of NexusMods.Archives.Nx adds the ability to deduplicate files in real time on the fly.
That is, detect duplicates and only store them once in the archive under multiple names.
This feature is particularly useful when exporting mods to be shared with other people, for example:
- Sharing a work-in-progress mod to your friend.
- For example, a mod you made with the Nexus Mods app.
- Uploading a mod in the Nx format to the web.
Deduplication works in real time, with deduplication of SOLID blocks being free (<0.2% overhead).
For chunked files, the overhead is (at worst) 5% in practice if no duplicates are found.
Below tests use a 1MiB chunk size and Block Size. The default for the Nexus Mods app.
Example: Skyrim 202X 9.0 - Architecture
Part of Skyrim's Most Popular Texture Pack.
- 651 textures, total 11.6GiB in size.
| Scenario | Throughput | Throughput | Throughput | Size (MiB) |
|---|---|---|---|---|
| Solid Only | 302.97 MiB/s | 307.27 MiB/s | 309.29 MiB/s | 9,264.76 |
| Dedupe All | 309.06 MiB/s | 304.67 MiB/s | 310.13 MiB/s | 8,749.81 |
Example: 'Adachi Over Everyone'
This is an (unreleased) test/meme mod (by Mudkip) that I often use to test
duplicate file handling in various archive formats. All models in a game
are replaced with slightly tweaked variations of a single character model.
- 5,657 items, contains 57 duplicated files (several MB each), which are
15.8GiBtotal. - Remaining ~400MB are unique files.
| Scenario | Time (ms) | Throughput | Size (MiB) |
|---|---|---|---|
| Solid Only | 43,572 | 394.93 MiB/s | 5,349.69 |
| Dedupe All | 8,005 | 2135.97 MiB/s | 277.67 |
My NVMe has a throughput of around 3000MiB/s.
Taking into account the ~400MB of unique data, deduplication can be said
to work with nearly no overhead.
Enabling Deduplication
Deduplication can be enabled via the NxPackerBuilder API.
var builder = new NxPackerBuilder();
builder.WithChunkedDeduplication(deduplicateChunked);
builder.WithSolidDeduplication(deduplicateSolid);Or if you are more low level.
You can set PackerSettings.ChunkedDeduplicationState and PackerSettings.SolidDeduplicationState
to non-null.
Final Notes on Deduplication
This was developed during my own free time, and I hope it will eventually be useful to the community.
Deduplication is available for use today. It has been extensively tested, and brute forced with
a variety of mods. Several TiB of data has been packed in testing, and the results have been
consistent.
Currently, deduplication of SOLID blocks is enabled by default, as it is free.
While ded uplication of chunked blocks is opt-in.
For merging archives via archive repacking,
deduplication is automatically enabled.
Stabilized Archive Format
The format of Nx archives has now been formally stabilized as 1.0.0.
Various parts of the specification have been cleared up, a new 'terminology' section has been added,
and there is now a changelog to track changes to the format.
The format changes for this version are documented below.
Feature: Runtime Detection for Unsupported Libraries
With the new stabilized format, there is now proper error handling to detect incompatible versions.
Opening an archive that's too recent, should display an error message that looks like:
Unsupported archive version 1.
The most recent supported version is 0.
Please update your library.
In practical terms, suppose a user needed to downgrade the Nexus Mods app,
due to an unexpected breaking change.
If the App encounters an archive that's too new, it will now display a helpful error message;
rather than potentially trying to read the archive, and either crashing or potentially performing
invalid operations.
Slightly Improved Packing Performance
Performance of Packing archives in a multithreaded scneario has been improved by an approximate 1%,
through the introduction of improved block sorting in the packer. (Descending by time needed to compress.)
This means that overall thread utilization is improved, with less time spent waiting on other threads.
Bug Fixes
Zero Sized Memory Mapped Files
NexusMods.Archives.Nx now correctly writes zero-sized files when using memory mapped files to
extract files.
This fixes Nexus-Mods/NexusMods.App#1783
Miscellaneous
- Updated the 010editor template to reflect the new format changes.
- Added benchmarks for the new functionalities.
- Added various sanity checks that run in debug builds in order to catch potential issues early.
- Added .NET 9 support to the CLI.
- Reduced lock lock tim...
0.6.1
Today I am thrilled to announce the release of NexusMods.Archives.Nx 0.5.0,
now with a stable file format!
Major Features and Changes
Archive Repacking ⚡
This release introduces a powerful new capability: 'Repacking'.
This feature allows users to create new derivative archives, using the data of existing archives
as base. This works by copying data from existing SOLID Blocks, Chunked Files and partial chunks of
SOLID blocks.
In simpler terms, this works by copying already compressed data from existing archives, and
only compressing new data whenever necessary.
Practical Use: File Deletion
It is now possible to delete files from an archive in a very efficient manner.
Moreover, this can be efficiently done without the risks of corruption should the archive be
modified in-place and the system were to lose power, the process be or the application were to crash.
This feature in particular enables a feature called 'Garbage Collection' within the
Nexus Mods app, which allows us to reclaim unused
space from files that are no longer needed.
Practical Use: Archive Merging
This feature can also be used to merge multiple archives in an extremely efficient manner.
This feature was used to benchmark the performance of the repacking feature during development
and is available via the CLI.
Merging enables the following practical potential use cases in the Nexus Mods app:
Quickly sharing a work-in-progress mod to your friend:
In the Nexus Mods app, each archive stores a set of unique files. These archives can now be
merged to output a single archive that another user can then install a mod from.
Don't waste 4 minutes of your life, compressing your 2GB texture mod. Get a working,
shareable Nx archive in 0.5 seconds instead.
'Compacting' data during 'Garbage Collection':
The introduction of 'Garbage Collection' will eventually lead to 'Fragmentation' of archives in the
Nexus Mods app.
Deleting older versions of mods and performing 'Garbage Collection' risks leaving tiny small archives
that will all need to be read, parsed and processed independently in order to apply a single mod
from a loadout.
With the new merging feature, now it will be possible to merge multiple archives belonging to a single
mod as part of the 'Garbage Collection' process. This will improve the efficiency of the Apply operation
used to switch to a loadout.
Practical Use: Accelerated Archive Packing
Since the repacking feature copies data from existing archives, it can also be used to create entirely new
archives faster, if you know the location of some data you want to pack again.
To provide a practical example, suppose you have Mod 0.1.0, and you want to release an update,
Mod 0.2.0. This update is mostly the same, but with a few new files.
Now it is possible to use the repacking feature to copy the compressed data from Mod 0.1.0
and only compress the new files for Mod 0.2.0.
Using the Repacking Functionality from Code
This is very easy!!
Use the NxRepackerBuilder in place of the NxPackerBuilder API in your code.
// Use any IFileDataProvider to provide the existing archive.
// Ideally memory mapped provider, i.e. FromFilePathProvider
// if file is on disk.
var provider = new FromFilePathProvider() {
FilePath = "existing.nx"
};
var header = HeaderParser.ParseHeader(provider);
var repackerBuilder = new NxRepackerBuilder();
// Add files from existing archive
repackerBuilder.AddFilesFromNxArchive(nxSource, header, header.Entries.AsSpan());
// Configure output
repackerBuilder.WithOutput(new FileStream("repacked.nx", FileMode.Create));
// Build the repacked archive
using var outputStream = repackerBuilder.Build();If you want to merge archives, use the NxDeduplicatingRepackerBuilder instead,
this will automatically deduplicate files as they are added.
Performance Preview (How fast is Repacking?)
We will use the merge feature from the CLI.
Merging Skyrim 202X 10.0.1 (Architecture) and Skyrim 202X 9.0 (Architecture):
// NexusMods.Archives.Nx.Cli merge --output "Skyrim202X-merged.nx" --sources "Skyrim202X 9.0.nx" "Skyrim202X 10.0.1.nx" --deduplicate-chunked false --deduplicate-solid false
Merged in 6439ms
Input Size: 19432.68 MiB
Output Size: 11972.45 MiB
Compression Ratio: 61.61 %
Throughput 3955.06MiB/s
Repacking also works with the new deduplication feature, which is enabled by default
for merging.
// NexusMods.Archives.Nx.Cli merge --output "Skyrim202X-merged.nx" --sources "Skyrim202X 9.0.nx" "Skyrim202X 10.0.1.nx"
Merged in 5471ms
Input Size: 19432.68 MiB
Output Size: 11290.83 MiB
Compression Ratio: 58.10 %
Throughput 4654.84MiB/s
The new repacking functionality runs as fast as the write speed of my NVMe drive.
Deduplication
This release of NexusMods.Archives.Nx adds the ability to deduplicate files in real time on the fly.
That is, detect duplicates and only store them once in the archive under multiple names.
This feature is particularly useful when exporting mods to be shared with other people, for example:
- Sharing a work-in-progress mod to your friend.
- For example, a mod you made with the Nexus Mods app.
- Uploading a mod in the Nx format to the web.
Deduplication works in real time, with deduplication of SOLID blocks being free (<0.2% overhead).
For chunked files, the overhead is (at worst) 5% in practice if no duplicates are found.
Below tests use a 1MiB chunk size and Block Size. The default for the Nexus Mods app.
Example: Skyrim 202X 9.0 - Architecture
Part of Skyrim's Most Popular Texture Pack.
- 651 textures, total 11.6GiB in size.
| Scenario | Throughput | Throughput | Throughput | Size (MiB) |
|---|---|---|---|---|
| Solid Only | 302.97 MiB/s | 307.27 MiB/s | 309.29 MiB/s | 9,264.76 |
| Dedupe All | 309.06 MiB/s | 304.67 MiB/s | 310.13 MiB/s | 8,749.81 |
Example: 'Adachi Over Everyone'
This is an (unreleased) test/meme mod (by Mudkip) that I often use to test
duplicate file handling in various archive formats. All models in a game
are replaced with slightly tweaked variations of a single character model.
- 5,657 items, contains 57 duplicated files (several MB each), which are
15.8GiBtotal. - Remaining ~400MB are unique files.
| Scenario | Time (ms) | Throughput | Size (MiB) |
|---|---|---|---|
| Solid Only | 43,572 | 394.93 MiB/s | 5,349.69 |
| Dedupe All | 8,005 | 2135.97 MiB/s | 277.67 |
My NVMe has a throughput of around 3000MiB/s.
Taking into account the ~400MB of unique data, deduplication can be said
to work with nearly no overhead.
Enabling Deduplication
Deduplication can be enabled via the NxPackerBuilder API.
var builder = new NxPackerBuilder();
builder.WithChunkedDeduplication(deduplicateChunked);
builder.WithSolidDeduplication(deduplicateSolid);Or if you are more low level.
You can set PackerSettings.ChunkedDeduplicationState and PackerSettings.SolidDeduplicationState
to non-null.
Final Notes on Deduplication
This was developed during my own free time, and I hope it will eventually be useful to the community.
Deduplication is available for use today. It has been extensively tested, and brute forced with
a variety of mods. Several TiB of data has been packed in testing, and the results have been
consistent.
Currently, deduplication of SOLID blocks is enabled by default, as it is free.
While ded uplication of chunked blocks is opt-in.
For merging archives via archive repacking,
deduplication is automatically enabled.
Stabilized Archive Format
The format of Nx archives has now been formally stabilized as 1.0.0.
Various parts of the specification have been cleared up, a new 'terminology' section has been added,
and there is now a changelog to track changes to the format.
The format changes for this version are documented below.
Feature: Runtime Detection for Unsupported Libraries
With the new stabilized format, there is now proper error handling to detect incompatible versions.
Opening an archive that's too recent, should display an error message that looks like:
Unsupported archive version 1.
The most recent supported version is 0.
Please update your library.
In practical terms, suppose a user needed to downgrade the Nexus Mods app,
due to an unexpected breaking change.
If the App encounters an archive that's too new, it will now display a helpful error message;
rather than potentially trying to read the archive, and either crashing or potentially performing
invalid operations.
Slightly Improved Packing Performance
Performance of Packing archives in a multithreaded scneario has been improved by an approximate 1%,
through the introduction of improved block sorting in the packer. (Descending by time needed to compress.)
This means that overall thread utilization is improved, with less time spent waiting on other threads.
Bug Fixes
Zero Sized Memory Mapped Files
NexusMods.Archives.Nx now correctly writes zero-sized files when using memory mapped files to
extract files.
This fixes Nexus-Mods/NexusMods.App#1783
Miscellaneous
- Updated the 010editor template to reflect the new format changes.
- Added benchmarks for the new functionalities.
- Added various sanity checks that run in debug builds in order to catch potential issues early.
- Added .NET 9 support to the CLI.
- Reduced lock lock tim...
0.6.0
Today I am thrilled to announce the release of NexusMods.Archives.Nx 0.5.0,
now with a stable file format!
Major Features and Changes
Archive Repacking ⚡
This release introduces a powerful new capability: 'Repacking'.
This feature allows users to create new derivative archives, using the data of existing archives
as base. This works by copying data from existing SOLID Blocks, Chunked Files and partial chunks of
SOLID blocks.
In simpler terms, this works by copying already compressed data from existing archives, and
only compressing new data whenever necessary.
Practical Use: File Deletion
It is now possible to delete files from an archive in a very efficient manner.
Moreover, this can be efficiently done without the risks of corruption should the archive be
modified in-place and the system were to lose power, the process be or the application were to crash.
This feature in particular enables a feature called 'Garbage Collection' within the
Nexus Mods app, which allows us to reclaim unused
space from files that are no longer needed.
Practical Use: Archive Merging
This feature can also be used to merge multiple archives in an extremely efficient manner.
This feature was used to benchmark the performance of the repacking feature during development
and is available via the CLI.
Merging enables the following practical potential use cases in the Nexus Mods app:
Quickly sharing a work-in-progress mod to your friend:
In the Nexus Mods app, each archive stores a set of unique files. These archives can now be
merged to output a single archive that another user can then install a mod from.
Don't waste 4 minutes of your life, compressing your 2GB texture mod. Get a working,
shareable Nx archive in 0.5 seconds instead.
'Compacting' data during 'Garbage Collection':
The introduction of 'Garbage Collection' will eventually lead to 'Fragmentation' of archives in the
Nexus Mods app.
Deleting older versions of mods and performing 'Garbage Collection' risks leaving tiny small archives
that will all need to be read, parsed and processed independently in order to apply a single mod
from a loadout.
With the new merging feature, now it will be possible to merge multiple archives belonging to a single
mod as part of the 'Garbage Collection' process. This will improve the efficiency of the Apply operation
used to switch to a loadout.
Practical Use: Accelerated Archive Packing
Since the repacking feature copies data from existing archives, it can also be used to create entirely new
archives faster, if you know the location of some data you want to pack again.
To provide a practical example, suppose you have Mod 0.1.0, and you want to release an update,
Mod 0.2.0. This update is mostly the same, but with a few new files.
Now it is possible to use the repacking feature to copy the compressed data from Mod 0.1.0
and only compress the new files for Mod 0.2.0.
Using the Repacking Functionality from Code
This is very easy!!
Use the NxRepackerBuilder in place of the NxPackerBuilder API in your code.
// Use any IFileDataProvider to provide the existing archive.
// Ideally memory mapped provider, i.e. FromFilePathProvider
// if file is on disk.
var provider = new FromFilePathProvider() {
FilePath = "existing.nx"
};
var header = HeaderParser.ParseHeader(provider);
var repackerBuilder = new NxRepackerBuilder();
// Add files from existing archive
repackerBuilder.AddFilesFromNxArchive(nxSource, header, header.Entries.AsSpan());
// Configure output
repackerBuilder.WithOutput(new FileStream("repacked.nx", FileMode.Create));
// Build the repacked archive
using var outputStream = repackerBuilder.Build();If you want to merge archives, use the NxDeduplicatingRepackerBuilder instead,
this will automatically deduplicate files as they are added.
Performance Preview (How fast is Repacking?)
We will use the merge feature from the CLI.
Merging Skyrim 202X 10.0.1 (Architecture) and Skyrim 202X 9.0 (Architecture):
// NexusMods.Archives.Nx.Cli merge --output "Skyrim202X-merged.nx" --sources "Skyrim202X 9.0.nx" "Skyrim202X 10.0.1.nx" --deduplicate-chunked false --deduplicate-solid false
Merged in 6439ms
Input Size: 19432.68 MiB
Output Size: 11972.45 MiB
Compression Ratio: 61.61 %
Throughput 3955.06MiB/s
Repacking also works with the new deduplication feature, which is enabled by default
for merging.
// NexusMods.Archives.Nx.Cli merge --output "Skyrim202X-merged.nx" --sources "Skyrim202X 9.0.nx" "Skyrim202X 10.0.1.nx"
Merged in 5471ms
Input Size: 19432.68 MiB
Output Size: 11290.83 MiB
Compression Ratio: 58.10 %
Throughput 4654.84MiB/s
The new repacking functionality runs as fast as the write speed of my NVMe drive.
Deduplication
This release of NexusMods.Archives.Nx adds the ability to deduplicate files in real time on the fly.
That is, detect duplicates and only store them once in the archive under multiple names.
This feature is particularly useful when exporting mods to be shared with other people, for example:
- Sharing a work-in-progress mod to your friend.
- For example, a mod you made with the Nexus Mods app.
- Uploading a mod in the Nx format to the web.
Deduplication works in real time, with deduplication of SOLID blocks being free (<0.2% overhead).
For chunked files, the overhead is (at worst) 5% in practice if no duplicates are found.
Below tests use a 1MiB chunk size and Block Size. The default for the Nexus Mods app.
Example: Skyrim 202X 9.0 - Architecture
Part of Skyrim's Most Popular Texture Pack.
- 651 textures, total 11.6GiB in size.
| Scenario | Throughput | Throughput | Throughput | Size (MiB) |
|---|---|---|---|---|
| Solid Only | 302.97 MiB/s | 307.27 MiB/s | 309.29 MiB/s | 9,264.76 |
| Dedupe All | 309.06 MiB/s | 304.67 MiB/s | 310.13 MiB/s | 8,749.81 |
Example: 'Adachi Over Everyone'
This is an (unreleased) test/meme mod (by Mudkip) that I often use to test
duplicate file handling in various archive formats. All models in a game
are replaced with slightly tweaked variations of a single character model.
- 5,657 items, contains 57 duplicated files (several MB each), which are
15.8GiBtotal. - Remaining ~400MB are unique files.
| Scenario | Time (ms) | Throughput | Size (MiB) |
|---|---|---|---|
| Solid Only | 43,572 | 394.93 MiB/s | 5,349.69 |
| Dedupe All | 8,005 | 2135.97 MiB/s | 277.67 |
My NVMe has a throughput of around 3000MiB/s.
Taking into account the ~400MB of unique data, deduplication can be said
to work with nearly no overhead.
Enabling Deduplication
Deduplication can be enabled via the NxPackerBuilder API.
var builder = new NxPackerBuilder();
builder.WithChunkedDeduplication(deduplicateChunked);
builder.WithSolidDeduplication(deduplicateSolid);Or if you are more low level.
You can set PackerSettings.ChunkedDeduplicationState and PackerSettings.SolidDeduplicationState
to non-null.
Final Notes on Deduplication
This was developed during my own free time, and I hope it will eventually be useful to the community.
Deduplication is available for use today. It has been extensively tested, and brute forced with
a variety of mods. Several TiB of data has been packed in testing, and the results have been
consistent.
Currently, deduplication of SOLID blocks is enabled by default, as it is free.
While ded uplication of chunked blocks is opt-in.
For merging archives via archive repacking,
deduplication is automatically enabled.
Stabilized Archive Format
The format of Nx archives has now been formally stabilized as 1.0.0.
Various parts of the specification have been cleared up, a new 'terminology' section has been added,
and there is now a changelog to track changes to the format.
The format changes for this version are documented below.
Feature: Runtime Detection for Unsupported Libraries
With the new stabilized format, there is now proper error handling to detect incompatible versions.
Opening an archive that's too recent, should display an error message that looks like:
Unsupported archive version 1.
The most recent supported version is 0.
Please update your library.
In practical terms, suppose a user needed to downgrade the Nexus Mods app,
due to an unexpected breaking change.
If the App encounters an archive that's too new, it will now display a helpful error message;
rather than potentially trying to read the archive, and either crashing or potentially performing
invalid operations.
Slightly Improved Packing Performance
Performance of Packing archives in a multithreaded scneario has been improved by an approximate 1%,
through the introduction of improved block sorting in the packer. (Descending by time needed to compress.)
This means that overall thread utilization is improved, with less time spent waiting on other threads.
Bug Fixes
Zero Sized Memory Mapped Files
NexusMods.Archives.Nx now correctly writes zero-sized files when using memory mapped files to
extract files.
This fixes Nexus-Mods/NexusMods.App#1783
Miscellaneous
- Updated the 010editor template to reflect the new format changes.
- Added benchmarks for the new functionalities.
- Added various sanity checks that run in debug builds in order to catch potential issues early.
- Added .NET 9 support to the CLI.
- Reduced lock lock tim...
0.5.0
Today I am thrilled to announce the release of NexusMods.Archives.Nx 0.5.0,
now with a stable file format!
Major Features and Changes
Archive Repacking ⚡
This release introduces a powerful new capability: 'Repacking'.
This feature allows users to create new derivative archives, using the data of existing archives
as base. This works by copying data from existing SOLID Blocks, Chunked Files and partial chunks of
SOLID blocks.
In simpler terms, this works by copying already compressed data from existing archives, and
only compressing new data whenever necessary.
Practical Use: File Deletion
It is now possible to delete files from an archive in a very efficient manner.
Moreover, this can be efficiently done without the risks of corruption should the archive be
modified in-place and the system were to lose power, the process be or the application were to crash.
This feature in particular enables a feature called 'Garbage Collection' within the
Nexus Mods app, which allows us to reclaim unused
space from files that are no longer needed.
Practical Use: Archive Merging
This feature can also be used to merge multiple archives in an extremely efficient manner.
This feature was used to benchmark the performance of the repacking feature during development
and is available via the CLI.
Merging enables the following practical potential use cases in the Nexus Mods app:
Quickly sharing a work-in-progress mod to your friend:
In the Nexus Mods app, each archive stores a set of unique files. These archives can now be
merged to output a single archive that another user can then install a mod from.
Don't waste 4 minutes of your life, compressing your 2GB texture mod. Get a working,
shareable Nx archive in 0.5 seconds instead.
'Compacting' data during 'Garbage Collection':
The introduction of 'Garbage Collection' will eventually lead to 'Fragmentation' of archives in the
Nexus Mods app.
Deleting older versions of mods and performing 'Garbage Collection' risks leaving tiny small archives
that will all need to be read, parsed and processed independently in order to apply a single mod
from a loadout.
With the new merging feature, now it will be possible to merge multiple archives belonging to a single
mod as part of the 'Garbage Collection' process. This will improve the efficiency of the Apply operation
used to switch to a loadout.
Practical Use: Accelerated Archive Packing
Since the repacking feature copies data from existing archives, it can also be used to create entirely new
archives faster, if you know the location of some data you want to pack again.
To provide a practical example, suppose you have Mod 0.1.0, and you want to release an update,
Mod 0.2.0. This update is mostly the same, but with a few new files.
Now it is possible to use the repacking feature to copy the compressed data from Mod 0.1.0
and only compress the new files for Mod 0.2.0.
Using the Repacking Functionality from Code
This is very easy!!
Use the NxRepackerBuilder in place of the NxPackerBuilder API in your code.
// Use any IFileDataProvider to provide the existing archive.
// Ideally memory mapped provider, i.e. FromFilePathProvider
// if file is on disk.
var provider = new FromFilePathProvider() {
FilePath = "existing.nx"
};
var header = HeaderParser.ParseHeader(provider);
var repackerBuilder = new NxRepackerBuilder();
// Add files from existing archive
repackerBuilder.AddFilesFromNxArchive(nxSource, header, header.Entries.AsSpan());
// Configure output
repackerBuilder.WithOutput(new FileStream("repacked.nx", FileMode.Create));
// Build the repacked archive
using var outputStream = repackerBuilder.Build();If you want to merge archives, use the NxDeduplicatingRepackerBuilder instead,
this will automatically deduplicate files as they are added.
Performance Preview (How fast is Repacking?)
We will use the merge feature from the CLI.
Merging Skyrim 202X 10.0.1 (Architecture) and Skyrim 202X 9.0 (Architecture):
// NexusMods.Archives.Nx.Cli merge --output "Skyrim202X-merged.nx" --sources "Skyrim202X 9.0.nx" "Skyrim202X 10.0.1.nx" --deduplicate-chunked false --deduplicate-solid false
Merged in 6439ms
Input Size: 19432.68 MiB
Output Size: 11972.45 MiB
Compression Ratio: 61.61 %
Throughput 3955.06MiB/s
Repacking also works with the new deduplication feature, which is enabled by default
for merging.
// NexusMods.Archives.Nx.Cli merge --output "Skyrim202X-merged.nx" --sources "Skyrim202X 9.0.nx" "Skyrim202X 10.0.1.nx"
Merged in 5471ms
Input Size: 19432.68 MiB
Output Size: 11290.83 MiB
Compression Ratio: 58.10 %
Throughput 4654.84MiB/s
The new repacking functionality runs as fast as the write speed of my NVMe drive.
Deduplication
This release of NexusMods.Archives.Nx adds the ability to deduplicate files in real time on the fly.
That is, detect duplicates and only store them once in the archive under multiple names.
This feature is particularly useful when exporting mods to be shared with other people, for example:
- Sharing a work-in-progress mod to your friend.
- For example, a mod you made with the Nexus Mods app.
- Uploading a mod in the Nx format to the web.
Deduplication works in real time, with deduplication of SOLID blocks being free (<0.2% overhead).
For chunked files, the overhead is (at worst) 5% in practice if no duplicates are found.
Below tests use a 1MiB chunk size and Block Size. The default for the Nexus Mods app.
Example: Skyrim 202X 9.0 - Architecture
Part of Skyrim's Most Popular Texture Pack.
- 651 textures, total 11.6GiB in size.
| Scenario | Throughput | Throughput | Throughput | Size (MiB) |
|---|---|---|---|---|
| Solid Only | 302.97 MiB/s | 307.27 MiB/s | 309.29 MiB/s | 9,264.76 |
| Dedupe All | 309.06 MiB/s | 304.67 MiB/s | 310.13 MiB/s | 8,749.81 |
Example: 'Adachi Over Everyone'
This is an (unreleased) test/meme mod (by Mudkip) that I often use to test
duplicate file handling in various archive formats. All models in a game
are replaced with slightly tweaked variations of a single character model.
- 5,657 items, contains 57 duplicated files (several MB each), which are
15.8GiBtotal. - Remaining ~400MB are unique files.
| Scenario | Time (ms) | Throughput | Size (MiB) |
|---|---|---|---|
| Solid Only | 43,572 | 394.93 MiB/s | 5,349.69 |
| Dedupe All | 8,005 | 2135.97 MiB/s | 277.67 |
My NVMe has a throughput of around 3000MiB/s.
Taking into account the ~400MB of unique data, deduplication can be said
to work with nearly no overhead.
Enabling Deduplication
Deduplication can be enabled via the NxPackerBuilder API.
var builder = new NxPackerBuilder();
builder.WithChunkedDeduplication(deduplicateChunked);
builder.WithSolidDeduplication(deduplicateSolid);Or if you are more low level.
You can set PackerSettings.ChunkedDeduplicationState and PackerSettings.SolidDeduplicationState
to non-null.
Final Notes on Deduplication
This was developed during my own free time, and I hope it will eventually be useful to the community.
Deduplication is available for use today. It has been extensively tested, and brute forced with
a variety of mods. Several TiB of data has been packed in testing, and the results have been
consistent.
Currently, deduplication of SOLID blocks is enabled by default, as it is free.
While ded uplication of chunked blocks is opt-in.
For merging archives via archive repacking,
deduplication is automatically enabled.
Stabilized Archive Format
The format of Nx archives has now been formally stabilized as 1.0.0.
Various parts of the specification have been cleared up, a new 'terminology' section has been added,
and there is now a changelog to track changes to the format.
The format changes for this version are documented below.
Feature: Runtime Detection for Unsupported Libraries
With the new stabilized format, there is now proper error handling to detect incompatible versions.
Opening an archive that's too recent, should display an error message that looks like:
Unsupported archive version 1.
The most recent supported version is 0.
Please update your library.
In practical terms, suppose a user needed to downgrade the Nexus Mods app,
due to an unexpected breaking change.
If the App encounters an archive that's too new, it will now display a helpful error message;
rather than potentially trying to read the archive, and either crashing or potentially performing
invalid operations.
Slightly Improved Packing Performance
Performance of Packing archives in a multithreaded scneario has been improved by an approximate 1%,
through the introduction of improved block sorting in the packer. (Descending by time needed to compress.)
This means that overall thread utilization is improved, with less time spent waiting on other threads.
Bug Fixes
Zero Sized Memory Mapped Files
NexusMods.Archives.Nx now correctly writes zero-sized files when using memory mapped files to
extract files.
This fixes Nexus-Mods/NexusMods.App#1783
Miscellaneous
- Updated the 010editor...
0.4.0
NexusMods.Archives.Nx 0.4.0
Breaking Changes
- The header format has been amended to increase the size of block and chunk ranges.
- Blocks are now
4K-64M(previously32K-64M)- Achieved by shifting the base value from
32Kto4K
- Achieved by shifting the base value from
- Chunks are now
32K-1G(previously1M-128M)- Achieved by shifting the base value to
32Kand using 1 more bit in the header - The version field had 1 bit taken away to achieve this
- Default Chunk Size is now
1M(to help packing speed a bit and reduce latency for App use)
- Achieved by shifting the base value to
- Blocks are now
This is a breaking change, previous archives will no longer work.
No future breaking changes are expected however, this is just about rearranging some bits for better flexibility based on existing usage experience.
Performance Improvements
- Packing now detects hyperthreads and chooses NOT to use them by default due to performance regressions on tested hardware (when using ZStandard). Unpacking still uses all threads as normal.
- The code now provides OS hints that memory-mapped data will be accessed 'soon'. Tiny speedup.
- Slightly faster
MemoryMappedFilecreation by optimizing around .NET's implementation of MMF.- There still might be a small additional improvement on Windows if we used WinAPI directly. Maybe in the future.
- Asynchronous flushing of Memory Mapped data to Disk is now implemented on non-Windows.
- Windows was already doing this as the default OS
MemoryMappedFilebehavior. - On Linux,
msync(MS_SYNC)was previously used (by the Runtime), which synchronously wrote the data in a blocking fashion. - This reduces the time spent from approx 240 seconds to 3 seconds to extract 60K ~100 byte files, making it ~2x the speed of Windows.
- Windows was already doing this as the default OS
Other Notes
- Expect around 30-40% speed improvement for packing in the case of highly compressible data, at expense of very small amount of compression ratio.
- For non-compressible data, expect no change.
0.3.8
Changelog
All notable changes to this project will be documented in this file.
The format is based on Keep a Changelog
and this project adheres to Semantic Versioning.
Unreleased
0.3.8 - 2024-02-13
Commits
- Bumped: Package Version
24d0e61
0.3.7
Changelog
All notable changes to this project will be documented in this file.
The format is based on Keep a Changelog
and this project adheres to Semantic Versioning.
Unreleased
0.3.7 - 2023-09-26
Merged
- Fixed: Incorrect ZStandard API Usage for Partial Decompression of Blocks
#13
0.3.6
Changelog
All notable changes to this project will be documented in this file.
The format is based on Keep a Changelog
and this project adheres to Semantic Versioning.
Unreleased
0.3.6 - 2023-09-20
Commits
- Update NexusMods.Archives.Nx.csproj
b2a1cef
0.3.5
Changelog
All notable changes to this project will be documented in this file.
The format is based on Keep a Changelog
and this project adheres to Semantic Versioning.
Unreleased
0.3.5 - 2023-09-19
Commits
- Update NexusMods.Archives.Nx.csproj
d75bfe9
0.3.4
Changelog
All notable changes to this project will be documented in this file.
The format is based on Keep a Changelog
and this project adheres to Semantic Versioning.
Unreleased
0.3.4 - 2023-08-31
Merged
- Miscellaneous Docs Fixes
#11