EtherStruct

Repository: https://github.com/vikingfacer/EtherStruct
API: https://vikingfacer.github.io/EtherStruct/#EtherStruct

If you plan on working with network packets you will probably need to serialize and de-serialize them. Otherwise you’ll be operating on byte strings. This post is to show off how I’m serializing and de-serializing packets in Zig with my EtherStruct library.

Background

two important concepts need to be laid out. After that the library can be discussed.

Structure of IP/Ethernet packets
Byte alignment of data

IP/Ethernet/Internet/(whatever) Packets

Network packets physically are just strings of bytes, but logically are nesting dolls of structured data. For instance each Ethernet frame consists of four parts: preamble, header, payload, and frame check sequence. EtherStruct library uses libpcap so it can only observer header and payload. The payload contains all the other Layers, and those layers contain other Layers. Ethernet packets are like a nesting doll.

Each Layer starts with a header then immediately starts with payload section. There is no padding between the header and payload. This goes for each layer.

Byte Alignment

Byte alignment is how the data is stored in memory. To be honest byte alignment is a detail that the typical developer/project should not be concerned with. The compiler is better at making byte alignment decisions than developers. It should be seen as a sign of misfortune that we are concerned with it in this library. Since we are mapping an bit level API to structured data. We can not just used what the compiler gives us.

Byte alignment is simple and easy when dealing with primitive types (int,float,etc). The complexity of byte alignment comes in when building structured data like structs and bitfields. With ints, floats, and chars (in C/Cpp) the alignment is easy the size of each type is 4, 4, 1.

struct foo
{
  char a;
  int c;
  char b;
};

In the struct above the size is compiler dependent because of the byte alignment of the fields. looking at the structure a naive guess would be 6 bytes. 1<a> + 4<b> + 1<c> = 6. Most likely, the int changes the alignment of each field to 4 bytes, so the struct is 12 bytes. 4<a> + 4<b> + 4<c> = 12.

Small != Fast

One reason for the compiler to dictate a larger alignment, is speed. To illustrate this here is an example. Image a CPU with 32 bit registers. I want to add a and b from the struct above together. with the struct fields having a byte alignment of 4.

The CPU would need to do the following with alignment of 4

load the memory -> registers
add
store

At a high level that is 3~4 instructions. If the fields where packed together so field alignment is 1. there would need to be additional bit shifting done on the register.

The CPU would need to do the following with alignment of 1

load the memory -> registers
shift the register around
add
store

The Plan

The plan for serialization / de-serialization is two steps.

Build packed structures for each Layers header.
Byte swap for endianess on each field in the struct

We will look at the code and unit tests written to show “the plan”.

Step 1

header -> struct

The struct defines the ethernet header. It is a packed struct not a struct struct. The packed keyword changes the byte alignment of the struct to be 1 byte. the packed keyword makes sure there is no padding between fields.

/// Ethernet frame structure
pub const ethFrame = packed struct {
    dst: u48,
    src: u48,
    frameType: u16,
};

We can check the size of the ethFrame to show that the total size is 14.
6 bytes for source, 6 bytes for destination, 2 bytes for type.

test "Ethernet header size == 14" {
    try testing.expect(packedSize(ethFrame) == 14);
}

u48 is not in the set of 8 * 2^n. Types not in the 8 * 2^n, alignment to the smallest 8 * 2^n that fits in. in u48’s case that is u64. With u48 aligning to u64 the total size would be 8 + 8 + 2 = 18.

In Zig the builtin function @sizeOf states it May contain padding. We need to use @bitSizeOf / 8 to fine the packed sizeOf. BitSizeOf(T) returns the number of bits it takes to store T in memory. Since it returns bits we just need to divide it by 8 to get the packed size.

pub fn packedSize(item: anytype) comptime_int {
    return @bitSizeOf(item) / 8;
}

Step 2

byte swap fields

Byte swapping is done per field. First, bytes are copied into the struct with std.mem.copyForward. Second, compApplyToStruct applies std.mem.bigToNative on each field in structReturned.

pub fn toNativeValue(comptime T: type, bytes: []const u8) T {
    var structReturned: T = undefined;

    std.mem.copyForwards(u8, std.mem.asBytes(&structReturned), bytes);
    compApplyToStruct(T, std.mem.bigToNative, &structReturned);

    return structReturned;
}

std.meta.fields provides compile time reflection which allows the function to iterate through each field. If the filed is at least a byte the bytes can be swapped. (if its a single byte its not swapped). Using std.meta.fields is awesome cause allows this function to be generic.

fn compApplyToStruct(comptime T: type, func: anytype, d: *T) void {
    inline for (std.meta.fields(T)) |f| {
        if (@bitSizeOf(f.type) % 8 == 0) {
            @field(d, f.name) = func(f.type, @field(d, f.name));
        }
    }

With these functions in place we can test if the struct is serialized, and if it can be de-serialized. toNetworkBytes de-serializes the struct to network bytes. it does the opposites toNativeValue except the last line. (I’ll explain in the misc section)

pub fn toNetworkBytes(comptime T: type, nativeStruct: T) []const u8 {
    var bytes: T = nativeStruct;
    compApplyToStruct(T, std.mem.nativeToBig, &bytes);
    return std.mem.toBytes(bytes)[0 .. @bitSizeOf(T) / 8];
}

Tests

With the test below we can serialize the byte string to a struct with toNativeValue. Then check the fields, and finally de-serialize it back into the original byte string.

const TCPAckBytes = [_]u8{
    0xdc, 0x62, 0x79, 0xed, 0xd3, 0x14, 0x18, //
    0x5e, 0x0f, 0xc9, 0x6f, 0x0d, 0x08, 0x00, //
    ... }
test "Ethernet Frame" {
    const localhostEth = TCPAckBytes[0..packedSize(ethFrame)];

    const eth = toNativeValue(ethFrame, localhostEth);

    try testing.expectEqual(0xdc6279edd314, eth.dst);
    try testing.expectEqual(0x0185e0fc96f0d, eth.src);
    try testing.expect(0x0800 == eth.frameType);

    try testing.expectEqualSlices(u8, localhostEth, toNetworkBytes(ethFrame, eth));
}

Misc.

“std.mem.toBytes(bytes)[0 .. @bitSizeOf(T) / 8]”

This line looks odd to me and need more explanation. The expectation I originally had when converting packed struct to bytes. Was the size of the byte string would be the exact size of the packaged struct. However, in Zig packed structs are just an abstraction. They have a backing integer/block on memory, so they are padded.

Unit Test Bytes

The byte strings in the unit tests were captured with wireshark. All the other unit tests in the lib use the same Capture except the UDP unit tests.

Future

Writing this post has served as self code review for this library. It can definitely be improved. Here is a list of things I plan on improving

Add more comments for each field
Type checking (compApplyToField)
Add ?to serialize/de-serialize functions
Utilize Enums
Test and checksum functionality
Add more headers (as needed)

Contributing

If you read this and want to use this library
- Do it. Give me feedback
If you think its missing something and want to add it
- Add it. Make a pull request

The hope with this library is that it can and will grow with use. Contributing helps that happen.