Memory layout in Zig with formulas

▲ Memory layout in Zig with formulas(raymondtana.github.io)

74 points by raymondtana 9 hours ago | 20 comments

▲ thechao 4 hours ago | parent | next [-]

I know this is a bit cursed; but, I always wanted a bitfield-on-steroids construct:

    struct Dang : bits 64    // 64 bits wide, int total
    {
        foo : bits 5 @ 0;    // 5 bits wide at bit offset 0
        bar : bits 5 @ 0;
        baz : bits 16 @ 4;   // 16 bits wide at bit offset 4
        tom : bits 11 @ 32;
    };

▲ titzer 23 minutes ago | parent | next [-]

You might want to have a look at the unboxing and packing annotations that are proposed for Virgil. The unboxing mechanism is implemented and there was a prototype of the packing mechanism implemented by Bradley for his thesis. I am working on making a more robust implementation that I can land.

https://arxiv.org/abs/2410.11094

I'm not sure I understand your example; if I am looking at it right, it has overlapping bitfields.

But supposing you didn't want overlapping fields, you could write:

    type Dang(tom: u11, baz: u16, bar: u5, foo: u5) #packed;

And the compiler would smash the bits together (highest order bits first).

If you wanted more control, you can specify where every bit of every field goes using a bit pattern:

    type Dang(tom: u11, baz: u16, bar: u5, foo: u5) #packed 0bTTTTTTTT_TTTbbbbb_bbbbbbbb_bbbzzzzz_????fffff

Where each of T, b, z, and r represent a bit of each respective field.

▲ Lvl999Noob 3 hours ago | parent | prev | next [-]

Are you saying you want foo and bar to completely overlap? And baz and foo / bar to partially overlap? And have lots of unused bits in there too?

	▲	metaltyphoon 34 minutes ago \| parent [-]
		C# can do this with structs. Its kind of very nice to unpack wire data.

▲ sestep 3 hours ago | parent | prev | next [-]

I think you can do this with Virgil, but I'm having trouble finding the exact doc page at the moment: https://github.com/titzer/virgil

	▲	titzer 22 minutes ago \| parent [-]
		The description is in the paper, but not all of it is implemented. https://arxiv.org/abs/2410.11094 Bradley implemented a prototype of the packing solver, but it doesn't do the full generality of what is proposed in the paper.

▲ _bohm 3 hours ago | parent | prev | next [-]

You can kinda do this with Zig’s packed structs and arbitrary-width integers

▲ bsder an hour ago | parent | prev [-]

Look at Erlang bit syntax: https://www.erlang.org/doc/system/bit_syntax.html

It can even be used for pattern matching.

I don't know whether Gleam or Elixir inherited it.

▲ raymondtana 9 hours ago | parent | prev | next [-]

I've been learning Zig, and needed a refresher on memory layout (@sizeOf and @alignOf).

Wrote this blog post to summarize what I think are the right ways to understand alignment and size for various data types in Zig, just through experimentation.

Let me know any and all feedback!

▲

dnautics 5 hours ago | parent [-]

i could be wrong but i believe the zig compiler reserves the right to lay things out differently depending on compilation mode? especially debug. unless it's extern or packed, in which case the layout will be defined.

▲

rvrb 5 hours ago | parent [-]

`extern` and `packed` container types have well defined layouts. a regular `struct` is an "auto" layout - and the compiler can and will rearrange whenever it wants.

if you need a well defined layout, use `extern`. if your struct makes sense to represent as an integer, use `packed`. I think it is often ill advisable to use `packed` otherwise.

you can explore this yourself on the Type info returned from @TypeInfo(T):

https://ziglang.org/documentation/master/std/#std.builtin.Ty...

▲

LexiMax 5 hours ago | parent | next [-]

To wit: https://ziglang.org/documentation/master/#extern-struct

> An extern struct has in-memory layout matching the C ABI for the target.

Zig is really good at speaking the C ABI of the target, but the upshot seems to be that it appears there is no stable Zig-native ABI.

If I'm correct, I wonder if there are plans to settle on a stable ABI at some point in the future. I do know that in other languages the lack of a stable ABI is brought up as a downside, and although I've been burned by C++ ABI stability too many times to agree, I can understand why people would want one.

▲

Cloudef 4 hours ago | parent | next [-]

I doubt zig will have stable abi any time soon. It may have some sort of "zig extern" when it gets mature. But stable abi isnt very usful if no-one else can talk it. I have project that uses codegen to effectively implement zig like ABI on top of the C abi.

Heres the kind of code it generates https://zigbin.io/6dba68

It can also generate javascript, heres doom running on browser: https://cloudef.pw/sorvi/#doom.wasm

▲

peesem 4 hours ago | parent | prev [-]

Andrew Kelley has said relatively recently that there are no plans to introduce a Zig ABI: https://github.com/ziglang/zig/issues/3786#issuecomment-2646...

▲

LexiMax 3 hours ago | parent [-]

What's interesting is that the scope of the proposal isn't a Zig-specific ABI, but a codified way of expressing certain Zig concepts using the existing C ABI.

That could be an interesting middle ground.

	▲	Cloudef 3 hours ago \| parent [-]
		Yeah the new translate-c package already kind of does that.

▲

dnautics 3 hours ago | parent | prev [-]

in practice, as long as you match the version and release mode, it's fine (though you are playing with fire). I pass raw pointers to zig structs/unions/etc from the zig compiler into a dynamically loaded .so file (via dlload) and as long as my .so file is compiled with the same compiler as the parent (both LLVM, in my case) it's peachy keen.

	▲	Cloudef 2 hours ago \| parent [-]
		You are still playing with fire as the data inside those pointers may be different even if they are the same type. Zig is free to optimize them in anyway it likes depending on the code that touches them (aka its free to assume they never leave the program).

▲ ivanjermakov 2 hours ago | parent | prev | next [-]

I also had to learn struct alignment the hard way working on WebGPU path tracer and struggling to understand why struct fields not aligning (ironically).

▲ bk496 5 hours ago | parent | prev [-]

useful!