ABI & Alignment woes

[Hugobros3]

So over the winter break I started this small chess program in Artic as some kind of fun thing/language demo/benchmark, and encountered two kinds of C ABI troubles:

The first, which we touched on lightly with @leissa on today's call has to do with the poor mapping between C and Artic types when it comes to empty datatypes: These simply do not exist in (standard) C, or are defined has having a size of one in C++, causing alignment issues:
http://161.97.116.199:10240/z/hYYMTn Working around this is simple: just avoid zero-sized data types (we could ban them from being present in exported functions, just like we ban exporting higher order ones)

The other, more problematic one in my opinion, has to do with what Arsène was talking about with LLVM being a "leaky" IR. Despite what the LLVM documentation claims, the C calling rules are (partially) implemented in Clang, and not available to other front-ends as a library. (See https://lists.llvm.org/pipermail/llvm-dev/2020-June/142055.html) In particular, the code for mangling functions with "large" parameters, for an example please see how a function passing a structure S is done in Clang versus Thorin

We see the C code putting the whole S struct on the stack, and the Thorin code exploding S and passing its contents in 6 registers + the last one on the stack. The LLVM IR is also different:

Thorin:

%0 = type { i64, i64, i64, { i8, i8, i16, i32 } }
...
declare void @f(%0)

Clang:

%struct.S = type { i64, i64, i64, %struct.tuple_unsigned_char_unsigned_char_unsigned_short_unsigned_int }
%struct.tuple_unsigned_char_unsigned_char_unsigned_short_unsigned_int = type { i8, i8, i16, i32 }
define dso_local void @f(%struct.S* byval(%struct.S) align 8 %0) ...

The real difference here is that Clang uses this byval pointer attribute, while Thorin "naively" just passes S by value. Of course once the actual x86_64 code is emitted, stuff that does not fit into physical registers obviously is made to fit by exploding structs and/or allocating them on the stack, but the fact that the LLVM IR signatures did not match makes it so this "second" round of C ABI conversion can and does produce different results and effectively breaks the ability to link Thorin and C code properly when passing large-ish structs.

[leissa]

There are a ton of these problems. There are ABI requirements and the whole Clang/LLVM pipeline implements different aspects of it in different spots in the pipeline. The signext parameter attribute, e.g., is another one of these ABI requirements.

[madmann91]

So maybe to try to solve this issue at least, we can either:

1. Implement what clang does. A lot of work, error prone, machine-dependent, or
2. Only allow exporting functions that have primitive types (pointers, ixx, uxx, fxx).

I personally think we should go for option 2.

[leissa]

I'm also more in the camp of option 2. This is more or less what we do right now. It basically means
"We only support ABI compatibility for primitive types. Other things may or may not work."