Binary Data Packer

The Value and Format Types

• Text representations like JSON are easy to read but wasteful
  • The integer 1000000 takes seven bytes as ASCII but only four bytes as a 32-bit integer
  • And good luck representing a movie as characters
• Two types work together:

pub type Value {
  VInt(Int)
  VStr(String)
}

pub type Fmt {
  FInt
  FStr
}

• Value represents a single value that can be packed: either an integer or a string
• Fmt is the schema descriptor: it says what kind of value to expect at each position
• FInt and FStr carry no payload: they are pure tags
• Keeping them separate means the same format list can pack different value lists without mixing concerns
• The closest equivalent in Python is the struct format string ">I7s"
  • But that is an untyped string with no compiler help
  • Here the types enforce that every FInt in the format list lines up with a VInt in the value list

Packing Values

• pack encodes a list of values guided by a format list:

pub fn pack(formats: List(Fmt), values: List(Value)) -> BitArray {
  case formats, values {
    [], [] -> <<>>
    [FInt, ..frest], [VInt(n), ..vrest] ->
      <<n:32-big, pack(frest, vrest):bits>>
    [FStr, ..frest], [VStr(s), ..vrest] -> {
      let bytes = string_to_bytes(s)
      let len = bit_array.byte_size(bytes)
      <<len:32-big, bytes:bits, pack(frest, vrest):bits>>
    }
    _, _ -> <<>>
  }
}

• <<>> is an empty BitArray
• <<n:32-big, pack(frest, vrest):bits>> concatenates a 32-bit big-endian integer with the recursive result
  • :bits splices a BitArray into a larger one
• For strings, string_to_bytes converts the String to UTF-8 bytes
  • bit_array.byte_size measures it
  • Then the length is written as a 32-bit prefix before the bytes
• The final _, _ -> <<>> handles mismatched lists
  • In production code this would return a Result
• The BitArray literal syntax mirrors pattern-matching syntax
• The same annotations (32-big, utf8, bits) appear on both sides
• This symmetry makes encoding and decoding code look parallel

Unpacking Values

• Decoding reverses the process by consuming bytes guided by the same format list:

fn unpack_loop(
  formats: List(Fmt),
  data: BitArray,
  acc: List(Value),
) -> Result(List(Value), String) {
  case formats, data {
    [], _ -> Ok(list.reverse(acc))
    [FInt, ..frest], <<n:32-big, rest:bits>> ->
      unpack_loop(frest, rest, [VInt(n), ..acc])
    [FStr, ..frest], <<len:32-big, str_data:bytes-size(len), rest:bits>> -> {
      let s = str_data |> bytes_to_string
      unpack_loop(frest, rest, [VStr(s), ..acc])
    }
    _, _ -> Error("unexpected end of data")
  }
}

• <<n:32-big, rest:bits>> binds the first four bytes as an integer and rest as the remaining bytes
• <<len:32-big, str_data:bytes-size(len), rest:bits>> reads the length prefix, then reads exactly len bytes into str_data
• bit_array.to_string(str_data) converts those bytes back to a String and returns Ok(s) or Error(Nil) if the bytes are not valid UTF-8
• Values are accumulated in reverse and reversed at the end
• Error("unexpected end of data") fires when the pattern match fails, meaning the BitArray ran out of bytes before the format list did
• The bytes-size(len) annotation is the key to variable-length fields
  • It reads exactly len bytes, where len was bound by the preceding len:32-big

Handling Corrupt Data

• Decoding real-world data requires dealing with truncation and corruption

pub fn main() {
  let formats = [FStr, FInt]
  let values = [VStr("Ada"), VInt(30)]

  let packed = pack(formats, values)
  io.println("packed byte size: " <> int.to_string(bit_array.byte_size(packed)))

  let unpacked = unpack(formats, packed)
  io.println(string.inspect(unpacked))

  let corrupted = <<packed:bits, 255>>
  let failed = unpack(formats, corrupted)
  io.println(string.inspect(failed))
}

• <<packed:bits, 255>> appends an extra byte to simulate a corrupted or over-long message
• unpack is called with the same format list and the corrupted data
• Because the format list is exhausted first (all values decoded), the extra byte is silently ignored
  • Ok(list.reverse(acc)) fires when the format list is empty, regardless of remaining bytes
• To reject trailing bytes, change the base case to [], <<>> -> Ok(list.reverse(acc)) and [], _ -> Error("trailing bytes")
• The debug output shows that the round-trip succeeds
  • I.e., the extra byte does not corrupt the result

BitArray Literal Syntax Reference

These annotations appear identically in construction (<<n:32-big>>) and in pattern matching (<<n:32-big, rest:bits>>).

Testing

pub fn roundtrip_int_test() {
  let formats = [FInt]
  let values = [VInt(42)]
  pack(formats, values)
  |> unpack(formats, _)
  |> should.equal(Ok([VInt(42)]))
}

pub fn roundtrip_string_test() {
  let formats = [FStr]
  let values = [VStr("Gleam")]
  pack(formats, values)
  |> unpack(formats, _)
  |> should.equal(Ok([VStr("Gleam")]))
}

pub fn roundtrip_mixed_test() {
  let formats = [FInt, FStr, FInt]
  let values = [VInt(1), VStr("hello"), VInt(2)]
  pack(formats, values)
  |> unpack(formats, _)
  |> should.equal(Ok(values))
}

• Each test packs a value and immediately unpacks it, checking that the decoded list equals the original
• truncated_data_test passes only one byte for an integer field
  • The pattern <<n:32-big, rest:bits>> cannot match, so Error("unexpected end of data") is returned

Check Understanding

Exercises