[Design] Handling untitled files

[MichaReiser]

This is a proposal for the design changes needed to support untitled (technically, any non-file://) files.

Background

LSP specification

The LSP specifies that a document is identified by a URI. Technically, the URI can use any schema, although the only two used by VS code today are:

• file://: For files stored on disk (there are no guarantees that the file hasn't been deleted after opening it). This convention is explained in the DocumentFilter example.
• untitled://: This seems VS Code specific, but other editors (e.g. neovim) might adopt it too. VS Code uses this schema for untitled documents.

There's an open issue to standardize a buffer:// schema.

Resources

• Improve support for unwritten buffers microsoft/vscode-eslint#1560 (comment)
• LSP untitled:// URI scheme neovim/neovim#21276

Existing Red Knot concepts

The following Red Knot concepts are relevant for this design proposal:

• System is an abstraction of the environment in which Red Knot runs. It exposes methods to read files and directories, get the current working directory, and more. The LSP, CLI, and tests have custom System implementations. Most methods have a SystemPath (a thin wrapper around Path) argument.
• File: A file that Red Knot supports. It stores metadata about the file and its path. The FilePath is either a SystemPath (see System) or a VendoredPath.
• VendoredPath: A path to a file that is vendored with Ruff (included in Ruff's binary).

Proposal

My key insights from the LSP specification are:

• Files that exist on disk use the file schema.
• The set of supported schemas isn't fixed and can change in future LSP specifications. Today, untitled is the only other specified schema.

Ruff/Red Knot wants special handling for file-paths because module resolution, configuration discovery etc. is only possible for files stored on disk.

I propose that we extend FilePath with a new FilePath::SystemVirtual (stores a SystemVirtualPathBuf) variant that is an opaque wrapper around String (or &str). Other than SystemPath, this path will not expose any query or mutation methods like parent(), ancestors, join, etc. Handling all non file schema as an opaque string means we can easily support more schemas in the future.

The two main downsides of this approach that I see are:

• We need to add read_virtual_path_to_string, read_virtual_path_to_notebook, and virtual_path_metadata methods to System that match their SystemPath counterpart. I considered introducing a new AnySystemPath enum that's either a SystemPath or VirtualSystemPath,, but I prefer different method names because not all methods will have virtual counterparts (e.g., `walk_directory).
• There's one more pair of path variants

Required changes

I quickly added a new variant. Most changes look trivial. The only exception is that the module resolver uses FilePath in places where it only supports SystemPath and VendoredPath. The solution here is easy, introduce a new module-resolver FilePath enum that only supports these two variants.

Alternatives

Make SystemPath an enum

Instead of adding a new variant to FilePath, change SystemPath to an enum with two variants. I prefer extending FilePath instead because:

• Making SystemPath an enum means we can no longer implement AsRef on SystemPathBuf, which complicates the implementation a lot.
• The semantics of join, parent, ancestors, etc., for virtual paths are unclear. It shouldn't matter because we should never call these functions on a virtual path, but I would rather have this statically enforced.
• This approach has the benefit of not requiring the introduction of new methods on System, but it does have the disadvantage that all System methods accept virtual paths, even if these operations aren't supported.

[carljm]

Thanks for writing this up!

It would help my understanding to have a bit of higher-level discussion of what behavior we want to present around untitled files, which might influence how we want to represent them internally. (Maybe some assumptions are made about the behavior we want, which aren't clear to me.) For example, here is one possible set of behaviors:

• Untitled files should never be importable by any other file.
• Absolute imports in an untitled file should work the same as in any other file.
• Relative imports in an untitled file always resolve to Unknown.

The second point makes the implicit assumption that we always know which project an untitled file is part of. Do we? Does LSP provide enough information for us to know this, given that our usual methods would rely on locating the file in a tree relative to configs and package directories? If not, I'm not sure how we can resolve any imports (other than perhaps stdlib ones, with some guess at a Python version) from an untitled file.

The third point is also debatable. It would also be possible to try to use some heuristic to decide where in the project structure we think the untitled file might belong, which would allow us to guess at a resolution for relative imports. I'm not sure if we want to do this or not. (How does pyright handle relative imports in untitled files?)

[AlexWaygood]

(How does pyright handle relative imports in untitled files?)

Pyright seems to treat these all as Unknown until you've saved the file to a specific location. Here I've opened a local clone of typeshed-stats in VSCode and then clicked through File > New file > Python file

The same import is resolved fine by pyright if I put it in src/typeshed_stats/serialize.py (or I save the file to the same directory).

[MichaReiser]

Yeah sorry. I should have written this down in my proposal.

• Untitled files should never be importable by any other file.
• Absolute imports in an untitled file should work the same as in any other file.
• Relative imports in an untitled file always resolve to Unknown.

This is precisely the behavior I have in mind and is about as good as we can do without knowing the location of a file (which is inherent to untitled files)

The second point makes the implicit assumption that we always know which project an untitled file is part of. Do we?

This depends on what you understand by a project but even then, no we aren't always able to precisely know to which workspace or package (workspace member) a file belongs:

• Package: We can't know to which package a file belongs because we don't know the file path.
• Workspace: We may know the workspace if only one workspace is open in the editor. See

  ruff/crates/ruff_server/src/session/index.rs

  Lines 401 to 415 in 49cbacc

  	fn settings_for_url(&self, url: &Url) -> Option<&WorkspaceSettings> {
  	if let Ok(path) = url.to_file_path() {
  	self.settings_for_path(&path)
  	} else {
  	// If there's only a single workspace, use that configuration for an untitled document.
  	if self.settings.len() == 1 {
  	tracing::debug!(
  	"Falling back to configuration of the only active workspace for the new document '{url}'."
  	);
  	self.settings.values().next()
  	} else {
  	None
  	}
  	}
  	}

  for how we handle this today

We have a few different options when we can't determine the workspace:

• Just assume one. The chance that it is the correct one is 1/n
• Don't assume any workspace and create an ad-hoc workspace similar to when a user opens a file from outside the workspace.

I consider the logic on how we identify the right workspace for a file to be an LSP problem (still our problem, but outside of a single database)

[carljm]

That all makes sense, thanks for clarifying! The proposal looks reasonable to me.

[dhruvmanila]

Thanks for writing this proposal.

Is this proposal specifically to handle the "untitled" scheme i.e., non-existant file in an editor? Or, is this more so for any non-"file" scheme document? My main motivation for this question is that the "untitled" scheme is specific to VS Code but other editors could provide a different scheme for either non-existant files or any other unknown files. Another motivation is the following comment from one of the linked reference:

https://github.com/microsoft/vscode-eslint/blob/b63a4433b4b5ccfa5f79fa48010552fd4f2e6dd7/server/src/eslint.ts#L1060

). This check should actually be:

if (uri.scheme !== 'file')

since the working directory makes only sense if the file exists on disk.

I'm assuming it's the later based on this:

Handling all non file schema as an opaque string means we can easily support more schemas in the future.

I think this might be as a follow-up but we also need to think about untitled notebooks but they probably need to be handled in a similar, if not the same, way as untitled files.

I might have a question or two regarding the implementation when I'm working on it but going through the existing types based on your implementation proposal, it looks good to me.

[MichaReiser]

Is this proposal specifically to handle the "untitled" scheme i.e., non-existant file in an editor? Or, is this more so for any non-"file" scheme document?

The idea is that this handles all non-file schemas. Or looking at it from a different perspective. Red Knot will have specific support for the file schema, but handles all other schemas the same.

I think this might be as a follow-up but we also need to think about untitled notebooks but they probably need to be handled in a similar, if not the same, way as untitled files.

Whether a File is a notebook or text document is handled at a later stage. First, a File is created with a path (a SystemPath if it is a notebook with a file scheme and a SystemVirtualPath otherwise). That path is then passed to source_code(file) to load the content which either calls read_to_string or read_to_notebook, depending on whether the file is a notebook.

[dhruvmanila]

I see, thanks for clarifying. This makes sense.