Skip to content

Commit

Permalink
✨feat: Improved docs
Browse files Browse the repository at this point in the history
  • Loading branch information
masonmark committed Dec 22, 2024
1 parent b694e00 commit 64aa9c8
Show file tree
Hide file tree
Showing 6 changed files with 51 additions and 53 deletions.
10 changes: 8 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
# write-new-file
# @axhxrx/write-new-file

A library and CLI tool for writing new files to a directory. If the destination file already exists, it is not overwritten, but rather a unique, lexicographically sequential suffix is added to the file name (before the file extension).
> **TL;DR** — I want to write files to a directory, perhaps concurrently from multiple processes, but I don't want to overwrite existing files, and I don't want to have to worry about file naming collisions. I also might want to have other processes watching for new files and consuming them, but I don't want to have to worry about partial file reads. I also want the files to be listed in order in `ls -l` or a GUI file browser, so it is obvious at a glance what is going on.
This is a library and CLI tool for writing new files to a directory. If the destination file already exists, it is not overwritten, but rather a unique, lexicographically sequential suffix is added to the file name (before the file extension).

The specialization here is to write "new" files (that is, files that do not already exist) to a single directory, without needing write access to any other directory, and to have both file writes and file reads (presumably executed by external processes) be atomic — no partial writes *or* partial-file reads.

Expand All @@ -24,6 +26,10 @@ However, it probably doesn't work on Windows. Such is life...😭

## Happenings

### 👹 2024-12-22: v0.0.3

No functionality changes, just various improvements to the docs and formatting.

### 👹 2024-12-22: v0.0.2

No changes other than fixing the JSR publishing automation.
Expand Down
17 changes: 6 additions & 11 deletions WriteNewOptions.ts
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ export type WriteNewOptions = {
};

/**
Private class that manages the default options. You shouldn't need to (or be able to, if coding normally) use this directly - use WriteNewOptions.default instead.
Private class that manages the default options. You shouldn't need to (or be able to, if coding normally) use this directly - use `WriteNewOptions.default` instead.
*/
class WriteNewOptionsDefaults
{
Expand All @@ -24,7 +24,6 @@ class WriteNewOptionsDefaults
{
return this._defaultOptions ?? {
outputDirectory: Deno.cwd(),
// mode: '0666',
};
}

Expand All @@ -35,27 +34,23 @@ class WriteNewOptionsDefaults
}

/**
THIS DID NOT WORK OUT 😅 (it did show up in some docs)
This type isn't supposed to show up anywhere. This is my latest workaround for JSR's slow-types checking. I actually like the slow-types checking, and this workaround isn't as onerous as the previous ones I came up with.
But, not well tested either so I hope "SlowTypesDefeater" doesn't show up in in-editor help popups, etc. (The problem at hand is that slow-types checking doesn't allow this old TypeScript saw of exporting a type and a const with the same name to get class-like convenience without class warfare.)
*/
type SlowTypesDefeater = WriteNewOptions & { default: WriteNewOptions };
// type SlowTypesDefeater = WriteNewOptions & { default: WriteNewOptions };

/**
Configuration object for writeNewFile(). Access default options via WriteNewOptions.default. You can also set the default options for your own app so that they needn't be passed to each call to `writeNewFile()`. (Useful if your app only writes to a single directory, for example.)
Configuration object for `writeNewFile()`. Access default options via `WriteNewOptions.default`. You can also set the default options for your own app so that they needn't be passed to each call to `writeNewFile()`. (Useful if your app only writes to a single directory, for example.)
*/
export const WriteNewOptions: SlowTypesDefeater = {
/**
Gets the default options.
*/
export const WriteNewOptions: WriteNewOptions & { default: WriteNewOptions } = {
get default(): WriteNewOptions
{
return WriteNewOptionsDefaults.defaultOptions;
},

/**
Sets the default options.
*/
set default(options: WriteNewOptions | null)
{
WriteNewOptionsDefaults.defaultOptions = options;
Expand Down
2 changes: 1 addition & 1 deletion deno.jsonc
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "@axhxrx/write-new-file",
"version": "0.0.2",
"version": "0.0.3",
"exports": "./mod.ts",
"license": "MIT",
"imports": {
Expand Down
42 changes: 21 additions & 21 deletions tryCreateFile.ts
Original file line number Diff line number Diff line change
@@ -1,20 +1,20 @@
/**
Attempt to create the file with the specified path. If it fails because it already exists, catch the error, and try again. If it fails for something else, just rethrow.
Attempt to create the file with the specified path. If it fails because it already exists, catch the error, and try again. If it fails for something else, just rethrow.
If it succeeds, write `data` and return `true`.
If it succeeds, write `data` and return `true`.
NOTE: This function is concurrency-safe, including atomicity for the file creation, and for consumers reading the output file. The files are written atomically to a temp file, and then renamed to the final path. However, consumers reading files from the directory (e.g. external watcher processes, etc.) must ignore the temp files, which are prefixed with `.__temp__`.
NOTE: This function is concurrency-safe, including atomicity for the file creation, and for consumers reading the output file. The files are written atomically to a temp file, and then renamed to the final path. However, consumers reading files from the directory (e.g. external watcher processes, etc.) must ignore the temp files, which are prefixed with `.__temp__`.
(We write them to the destination directory because the primary use case for this library is writing files to a directory that is read by external processes, and as such this writer may not have permission to write to any other location.)
(We write them to the destination directory because the primary use case for this library is writing files to a directory that is read by external processes, and as such this writer may not have permission to write to any other location.)
@param path The full path to the file we will try to create
@param path The full path to the file we will try to create
@param content The data to write (string or Uint8Array)
@param content The data to write (string or Uint8Array)
@returns `true` if the file was successfully created, `false` if it already existed
@returns `true` if the file was successfully created, `false` if it already existed
@throws Any error other than `Deno.errors.AlreadyExists` that occurs while attempting to create the file — e.g. disk full, permission error, etc
*/
@throws Any error other than `Deno.errors.AlreadyExists` that occurs while attempting to create the file — e.g. disk full, permission error, etc
*/
export async function tryCreateFile(
path: string,
content: string | Uint8Array,
Expand Down Expand Up @@ -49,28 +49,28 @@ export async function tryCreateFile(
file.close();
}
/*
@masonmark 2024-12-22: Wow, I was under a pretty major misapprehension about atomic writes in Deno. I thought `Deno.rename()` would be our jam, but in fact it only allows atomic writes — there's no way to detect/avoid overwriting an existing file.
@masonmark 2024-12-22: Wow, I was under a pretty major misapprehension about atomic writes in Deno. I thought `Deno.rename()` would be our jam, but in fact it only allows atomic writes — there's no way to detect/avoid overwriting an existing file.
Thank science for unit tests!
Thank science for unit tests!
I almost bailed on writing this in Deno — because there *are* other ways to do it, although they smell like fermented soybeans...
I almost bailed on writing this in Deno — because there *are* other ways to do it, although they smell like fermented soybeans...
But in this specific use case, I think Deno.link() meets all our requirements. It's atomic, and it fails if the target file already exists.
But in this specific use case, I think Deno.link() meets all our requirements. It's atomic, and it fails if the target file already exists.
The main downsides of link() are:
The main downsides of link() are:
- Can't reliably create hard links across different filesystems/mount points - but our temp files and the final file are in the same directory
- Can't reliably create hard links across different filesystems/mount points - but our temp files and the final file are in the same directory
- Some filesystems don't support hard links (like FAT32) - but most modern Unix filesystems do, so...
- Some filesystems don't support hard links (like FAT32) - but most modern Unix filesystems do, so...
- There may be filesystem-level limits on number of hard links per file - but this is OK since we immediately delete the temp file
- There may be filesystem-level limits on number of hard links per file - but this is OK since we immediately delete the temp file
- Hard links share inode/permissions with source file - I think we don't care, especially since we're deleting the temp file immediately, but that failing and something I am not thinking of will probably be the reason you find out in 2027 that some North Korean hacker has all your BTC (sorry (T_T)... )
- Hard links share inode/permissions with source file - I think we don't care, especially since we're deleting the temp file immediately, but that failing and something I am not thinking of will probably be the reason you find out in 2027 that some North Korean hacker has all your BTC (sorry (T_T)... )
- Some very restrictive systems might not allow hard links for security reasons (though they'd probably also restrict rename, so that's fine — not our target demographic)
- Some very restrictive systems might not allow hard links for security reasons (though they'd probably also restrict rename, so that's fine — not our target demographic)
So, since our mission here is to write temp files within a single directory, and then The atomic guarantee from link() is worth these (hopefully-)theoretical downsides.
*/
So, since our mission here is to write temp files within a single directory, and then The atomic guarantee from link() is worth these (hopefully-)theoretical downsides.
*/

await Deno.link(tempPath, path);
return true;
Expand Down
8 changes: 4 additions & 4 deletions writeNewFile.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -55,12 +55,12 @@ Deno.test('writeNewFile: basic usage', async () =>
That simulates the situation where not only is the proposed file name already in use by a different file, so the our initial attempt to write to the proposed filename should fail.
2) We manually create a file called something like 'example~2024-12-22-02-30-42.txt'
2) We manually create a file called something like 'example~2024-12-22-02-30-42.txt'
That simulates the situation where `writeNewFile()` has already been called during the current second, either by our process or some other process, meaning that writing to the first proposed unique name should also fail.
That simulates the situation where `writeNewFile()` has already been called during the current second, either by our process or some other process, meaning that writing to the first proposed unique name should also fail.
Given that setup, we can assert that `tryCreateFile()` returns false as expected, but that `writeNewFile()` succeeds (because it keeps trying until it does), and for good measure one more `writeNewFile()` also succeeds, and to verify they have done so wee check that there are 4 total files with names like what we expect and the contents we expect. (Whew!)
*/
Given that setup, we can assert that `tryCreateFile()` returns false as expected, but that `writeNewFile()` succeeds (because it keeps trying until it does), and for good measure one more `writeNewFile()` also succeeds, and to verify they have done so wee check that there are 4 total files with names like what we expect and the contents we expect. (Whew!)
*/
Deno.test('tryCreateFile: some other process wrote a file that is in our way', async () =>
{
const filename = 'example.txt';
Expand Down
25 changes: 11 additions & 14 deletions writeNewFile.ts
Original file line number Diff line number Diff line change
Expand Up @@ -7,19 +7,16 @@ import { WriteNewOptions } from './WriteNewOptions.ts';
Writes data to a unique file path using a date-based suffix, in a concurrency-safe manner. That is, if some other process or thread writes a file at the same time, we will fail, increment the suffix and try again, until we succeed at writing a new and uniquely-named file.
The logic is:
1. For the "current second" of the clock, try a suffix like:
"basename~YYYY-MM-DD HH:mm:ss.ext"
2. If that fails because it already exists, repeatedly:
a. Sleep 50ms
b. If the second is still the same, we try:
"basename~YYYY-MM-DD HH:mm:ssSSS.ext"
(where SSS is the millisecond portion)
c. If that also exists, keep looping in 50ms increments
until either we succeed or the clock moves on to a new second
3. Once the clock changes to a new second, start over at (1)
with the fresh second string.
This yields filenames that, in lexicographic order, generally tend to match their creation order (even across different platforms), in GUI file browsers or results of `ls`, etc.
1. If we can write the file with the proposed filename as-is, we're done. Otherwise:
2. For the "current second" of the clock, try a suffix like:
`'basename~YYYY-MM-DD HH:mm:ss.ext'`
3. If that fails because it already exists, repeatedly:
- Sleep 50ms
- If the second is still the same, we try: `'basename~YYYY-MM-DD HH:mm:ss+SSS.ext'` (where `SSS` is the millisecond portion)
- If that also exists, keep looping in 50ms increments until either we succeed or the clock moves on to a new second
4. Once the clock changes to a new second, start over at (1) with the fresh seconds-only string.
This yields filenames that, in lexicographic order (on most OS ), generally tend to match their creation order (even across different platforms), in GUI file browsers or results of `ls`, etc.
@param proposedFilename The proposed file name to write, including extension (if any), e.g. `'example.txt'`, `'foo.json'`, or `'config'`. If no file exists with that name yet (otherwise, it will have a lexicographically higher suffix appended, so that it is unique and is sorted after the existing files in the default sort order of most OSes)
Expand All @@ -30,7 +27,7 @@ import { WriteNewOptions } from './WriteNewOptions.ts';
@returns The full path to the newly created file, including the unique suffix
@throws Any error from the underlying file operations except for `AlreadyExists` which is handled internally
*/
*/
export async function writeNewFile(
proposedFilename: string,
content: string | Uint8Array,
Expand Down

0 comments on commit 64aa9c8

Please sign in to comment.