Added new build options and shader file inclusion overrides. #3

sinnwrig · 2024-04-28T08:37:30Z

Description

Multiple new options have been added to the build system in order to build mach-dxcompiler for more various use cases, and an addition to the API has been made in order to override default DXC file inclusion behavior.

Additional Build Features

Four new build options have been added to build.zig:

-Dshared Build machdxcompiler as a shared library alongside the static library.
-Dspirv Build SPIR-V codegen support for machdxcompiler.
-Dskip_executables Skip building machdxcompiler executable (Implicitly skips tests)
-Dskip_tests Skip building machdxcompiler-tests executable

API Changes

machDxcCompile function signature has been changed, accepting a MachDxcCompileOptions struct.
- MachDxcCompileOptions struct typedef has been added, containing the following fields:
  - char const* code` (UTF-8)
  - size_t code_len
  - char const* const* args (UTF-8)
  - size_t args_len
  - MachDxcIncludeCallbacks* include_callbacks (null falls back to default behavior)
The ability to override the default DXC file includer has been added to the C API.
- MachDxcIncludeCallbacks struct typedef has been added, containing the following fields:
  - MachDxcIncludeFunc* include_func
  - MachDxcFreeIncludeFunc* free_func
  - void* context
- MachDxcIncludeResult struct typedef has been added, containing the following fields:
  - const char* header_data (UTF-8)
  - size_t header_length
- MachDxcIncludeFunc function type definition has been added.
  - Receives void* as context.
  - Receives const char* as header name in UTF-8.
  - Returns MachDxcIncludeResult*.
- MachDxcFreeIncludeFunc function type definition has been added.
  - Receives void* as context.
  - Receives MachDxcIncludeResult* to free.
  - Returns int exit code.

Implementation details

Shared library building links the static library when compiling a minimal .cpp file.
Executable and test generation are now behind compilation flags.
SPIR-V codegen support builds tools/clang/lib/SPIRV sources in addition to core DXC sources, and links to an external dependency on hexops/spirv-tools.
DXC inclusion behavior has been routed through the provided function pointers using a minimal class inheriting from IDxcIncludeHandler.

Known Issues

There appears to be no issues with SPIR-V compilation or shared library support. However, certain combinations of input arguments, such as mixing '-Zi' with '-spirv' may cause segmentation faults.

Added ability to build shared library

Refactored build.zig to ensure format matches more similarly to the original build.zig.

Removed a python build file created by mistake.

slimsag · 2024-06-02T20:59:45Z

@sinnwrig can you rebase onto latest main which updates the zig version? I'll review this soon

slimsag · 2024-06-02T21:00:27Z

.gitignore

@@ -18,6 +18,7 @@ zig-out/
 /docgen_tmp/

 libs/DirectXShaderCompiler
+generated-include/spirv-tools/


what's the reason for the addition of this? do we end up needing generated spirv-tools stuff?

Can we include it from https://github.com/hexops/spirv-tools instead?

Oh, I didn't know that fork existed! Using that would probably work better than jamming together the SPIR-V headers with everything else. That folder just held the build-generated headers from SPIRV-Tools, so I figured it would be redundant to include them in the repo.

slimsag · 2024-06-02T21:01:02Z

build.zig

@@ -138,6 +153,82 @@ pub fn build(b: *Build) !void {
                lib_support_c_sources ++
                lib_dxilcompression_c_sources;

+            // Build and link SPIRV-Tools


definitely think we should be using https://github.com/hexops/spirv-tools here

I'll definitely be using it instead.

slimsag · 2024-06-02T21:01:55Z

build.zig

-                    try std.fs.cwd().copyFile(lib_path, std.fs.cwd(), "atls.lib", .{});
-                    try std.fs.cwd().copyFile(pdb_path, std.fs.cwd(), pdb_name, .{});
-                    // This is probably a bug in the Zig linker.
+            if (!skip_executables)


do you build spirv support for dxc.exe? or are you not testing that?

curious how I'll be able to test this works if dxc.exe doesn't work with it

I added the option to skip executables since I wanted to clean up the folder output as I was directly copying zig-out in another project. dxc.exe works fine with it.

slimsag · 2024-06-02T21:04:09Z

build.zig

+
+fn ensurePython(allocator: std.mem.Allocator) void
+{
+    if (!ensureCommandExists(allocator, "python3", "--version"))


we can't have a dependency on python here

This is a bit unfortunate; I am guessing we would need a fork of https://github.com/KhronosGroup/SPIRV-Headers packaged for build.zig with the headers pregenerated

Got it. I'm pretty sure hexops/SPIRV-Tools makes the stuff here a bit redundant, and I'll just use that repo instead.

slimsag · 2024-06-02T21:05:12Z

build.zig

+    buildSPIRVVersion(allocator);
+}
+
+const BuildSPIRVGrammarStep = struct {


I have a goal to move away from having any custom build steps like this; so this will be a no-go

No problem, this was only meant to generate spirv-headers, so if those are pregenerated it should be unneccesary.

slimsag · 2024-06-02T21:05:50Z

getting this merged upstream here might be a fair bit of work - we can chat about it though

sinnwrig · 2024-06-02T23:36:50Z

getting this merged upstream here might be a fair bit of work - we can chat about it though

Looking at the reviews, a lot of problems should be resolved if an external repo of SPIRV-Tools built with zig is used instead.

sinnwrig · 2024-06-03T09:15:48Z

spirv compilation now depends on hexops/spirv-tools in the .zon file.

PaperPrototype · 2024-06-21T08:51:20Z

why is this not merged yet

slimsag · 2024-10-09T22:51:10Z

Closing in favor of #7 which I promise I will actually merge soon.

sinnwrig added 4 commits April 28, 2024 01:30

Update build.zig

b5e33ef

Added ability to build shared library

Added shared library build option

447fea8

Moved shared library building to seperate function

f6cf5e1

build now supports SPIR-V codegen

dfa77b0

sinnwrig changed the title ~~Added shared library build option.~~ Added shared library build option and SPIR-V codegen option. May 2, 2024

sinnwrig and others added 4 commits May 1, 2024 22:49

Update build.zig

dc6c1cb

Refactored build.zig to ensure format matches more similarly to the original build.zig.

thing

625388e

Removed file created on accident.

b629000

Removed a python build file created by mistake.

Added compilation flags

b72cda5

sinnwrig changed the title ~~Added shared library build option and SPIR-V codegen option.~~ Added shared library option, SPIR-V codegen option, and executable/test skipping option. May 3, 2024

sinnwrig and others added 5 commits May 3, 2024 18:20

Moved SPIR-V generator output path to generated-include/spirv-tools

6f00bb8

Edited build.zig

d610831

Edited gitignore

947ded3

Edited build.zig

71ab699

Edited build.zig

a3a8405

sinnwrig marked this pull request as draft May 6, 2024 05:12

Fixed occasional segfault when including files with spir-v targets.

fa3f388

sinnwrig marked this pull request as ready for review May 6, 2024 05:32

Added file inclusion handlers.

53912bd

sinnwrig changed the title ~~Added shared library option, SPIR-V codegen option, and executable/test skipping option.~~ Added new build options and file inclusion overrides. May 21, 2024

sinnwrig changed the title ~~Added new build options and file inclusion overrides.~~ Added new build options and shader file inclusion overrides. May 21, 2024

Compile arguments have been condensed into struct

affef9e

slimsag reviewed Jun 2, 2024

View reviewed changes

Merge remote-tracking branch 'mach-dxcompiler/main'

8e4b1ae

sinnwrig added 2 commits June 3, 2024 00:19

Moved spirv-tools to external dependency

60d6a67

Fixed function signature for nominated zig version.

0fb981b

sinnwrig added 3 commits June 3, 2024 20:47

Fixed older zig API usage

a87cace

Moved macro defines to build.zig.

bc7ddfe

Fixed export issue

eb186a9

sinnwrig added 2 commits July 9, 2024 02:12

Removed unused write file additions and moved to newer zig release

bf01738

Reverted to nominated mach version

c3dfe92

sinnwrig force-pushed the main branch from 4da5ab0 to c3dfe92 Compare July 13, 2024 23:54

slimsag mentioned this pull request Oct 4, 2024

dxcompiler: update to Zig to version 0.14.0-dev.1710+8ee52f99c hexops/mach#1278

Closed

RonaldZielaznicki mentioned this pull request Oct 9, 2024

Update to zig 2024.10 and Add new build options and shader inclusion overrides #7

Merged

1 task

slimsag closed this Oct 9, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added new build options and shader file inclusion overrides. #3

Added new build options and shader file inclusion overrides. #3

sinnwrig commented Apr 28, 2024 •

edited

Loading

slimsag commented Jun 2, 2024

slimsag Jun 2, 2024

sinnwrig Jun 2, 2024

slimsag Jun 2, 2024

sinnwrig Jun 2, 2024

slimsag Jun 2, 2024

sinnwrig Jun 2, 2024

slimsag Jun 2, 2024

sinnwrig Jun 2, 2024 •

edited

Loading

slimsag Jun 2, 2024

sinnwrig Jun 2, 2024

slimsag commented Jun 2, 2024

sinnwrig commented Jun 2, 2024 •

edited

Loading

sinnwrig commented Jun 3, 2024 •

edited

Loading

PaperPrototype commented Jun 21, 2024

slimsag commented Oct 9, 2024

Added new build options and shader file inclusion overrides. #3

Added new build options and shader file inclusion overrides. #3

Conversation

sinnwrig commented Apr 28, 2024 • edited Loading

Description

Additional Build Features

API Changes

Implementation details

Known Issues

slimsag commented Jun 2, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sinnwrig Jun 2, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

slimsag commented Jun 2, 2024

sinnwrig commented Jun 2, 2024 • edited Loading

sinnwrig commented Jun 3, 2024 • edited Loading

PaperPrototype commented Jun 21, 2024

slimsag commented Oct 9, 2024

sinnwrig commented Apr 28, 2024 •

edited

Loading

sinnwrig Jun 2, 2024 •

edited

Loading

sinnwrig commented Jun 2, 2024 •

edited

Loading

sinnwrig commented Jun 3, 2024 •

edited

Loading