V Binary Encoder/Decoder

see lib/data/encoder

A high-performance binary encoder/decoder module for V that provides efficient serialization and deserialization of data structures. The encoder supports automatic encoding/decoding of structs using V's compile-time reflection capabilities.

Features

• Automatic struct encoding/decoding using compile-time reflection
• Support for primitive types, arrays, maps, and nested structs
• Compact binary format with length prefixing
• Size limits to prevent memory issues (64KB for strings/lists)
• Comprehensive error handling
• Built-in versioning support

Format

The binary format starts with a version byte (currently v1), followed by the encoded data:

[version_byte][encoded_data...]

Supported Types

Primitive Types

• string
• int (32-bit)
• i64 (64-bit integer)
• f64 (64-bit float)
• bool
• u8
• u16
• u32
• u64
• time.Time
• ourtime.OurTime (native support)
• percentage (u8 between 0-100)
• currency.Amount (currency amount with value)
• gid.GID (Global ID)
• []byte (raw byte arrays)

Arrays

• []string
• []int
• []u8
• []u16
• []u32
• []u64

Maps

• map[string]string
• map[string][]u8

Structs

• Nested struct support with automatic encoding/decoding

Usage

Basic Encoding

import freeflowuniverse.herolib.data.encoder

// Create a new encoder
mut e := encoder.new()

// Add primitive values
e.add_string('hello')
e.add_int(42)
e.add_bool(true)
e.add_u8(255)
e.add_u16(65535)
e.add_u32(4294967295)
e.add_u64(18446744073709551615)

// Add percentage (u8 between 0-100)
e.add_percentage(75)

// Add float64 value
e.add_f64(3.14159)

// Add int64 value
e.add_i64(-9223372036854775807)

// Add raw bytes
e.add_bytes('raw data'.bytes())

// Add time value
import time
e.add_time(time.now())

// Add OurTime (native time format)
import freeflowuniverse.herolib.data.ourtime
my_time := ourtime.OurTime.now()
e.add_ourtime(my_time)

// Add GID
import freeflowuniverse.herolib.data.gid
my_gid := gid.new('project:123')!
e.add_gid(my_gid)

// Add currency amount
import freeflowuniverse.herolib.data.currency
usd := currency.get('USD')!
amount := currency.Amount{
    currency: usd
    val: 99.95
}
e.add_currency(amount)

// Add arrays
e.add_list_string(['one', 'two', 'three'])
e.add_list_int([1, 2, 3])
e.add_list_u8([u8(1), 2, 3])
e.add_list_u16([u16(1), 2, 3])
e.add_list_u32([u32(1), 2, 3])
e.add_list_u64([u64(1), 2, 3])

// Add maps
e.add_map_string({
    'key1': 'value1'
    'key2': 'value2'
})
e.add_map_bytes({
    'key1': 'value1'.bytes()
    'key2': 'value2'.bytes()
})

// Get encoded bytes
encoded := e.data

Basic Decoding

// Create decoder from bytes
mut d := encoder.decoder_new(encoded)

// Read values in same order as encoded
str := d.get_string()!
num := d.get_int()!
bool_val := d.get_bool()!
byte := d.get_u8()!
u16_val := d.get_u16()!
u32_val := d.get_u32()!
u64_val := d.get_u64()!

// Read percentage value
percentage := d.get_percentage()! // u8 value between 0-100

// Read float64 value
f64_val := d.get_f64()!

// Read int64 value
i64_val := d.get_i64()!

// Read raw bytes
bytes_data := d.get_bytes()!

// Read time value
import time
time_val := d.get_time()!

// Read OurTime value
import freeflowuniverse.herolib.data.ourtime
my_time := d.get_ourtime()!

// Read GID
import freeflowuniverse.herolib.data.gid
my_gid := d.get_gid()!

// Read currency amount
import freeflowuniverse.herolib.data.currency
amount := d.get_currency()!

// Read arrays
strings := d.get_list_string()!
ints := d.get_list_int()!
bytes_list := d.get_list_u8()!
u16_list := d.get_list_u16()!
u32_list := d.get_list_u32()!
u64_list := d.get_list_u64()!

// Read maps
str_map := d.get_map_string()!
bytes_map := d.get_map_bytes()!

Automatic Struct Encoding/Decoding

struct Person {
    name string
    age  int
    tags []string
    meta map[string]string
}

// Create struct instance
person := Person{
    name: 'John'
    age: 30
    tags: ['developer', 'v']
    meta: {
        'location': 'NYC'
        'role': 'engineer'
    }
}

// Encode struct
encoded := encoder.encode(person)!

// Decode back to struct
decoded := encoder.decode[Person](encoded)!

Example

Here's a complete example showing how to encode nested structs:

import freeflowuniverse.herolib.data.encoder

// Define some nested structs
struct Address {
    street string
    number int
    country string
}

struct Person {
    name string
    age int
    addresses []Address    // nested array of structs
    metadata map[string]string
}

// Example usage
fn main() {
    // Create test data
    mut person := Person{
        name: 'John Doe'
        age: 30
        addresses: [
            Address{
                street: 'Main St'
                number: 123
                country: 'USA'
            },
            Address{
                street: 'Side St'
                number: 456
                country: 'Canada'
            }
        ]
        metadata: {
            'id': 'abc123'
            'type': 'customer'
        }
    }

    // Encode the data
    mut e := encoder.new()

    // Add version byte (v1)
    e.add_u8(1)

    // Encode the Person struct
    e.add_string(person.name)
    e.add_int(person.age)

    // Encode the addresses array
    e.add_u16(u16(person.addresses.len))  // number of addresses
    for addr in person.addresses {
        e.add_string(addr.street)
        e.add_int(addr.number)
        e.add_string(addr.country)
    }

    // Encode the metadata map
    e.add_map_string(person.metadata)

    // The binary data is now in e.data
    encoded := e.data

    // Later, when decoding, first byte tells us the version
    version := encoded[0]
    assert version == 1
}

Binary Format Details

For the example above, the binary layout would be:

[1]                     // version byte (v1)
[len][John Doe]         // name (u16 length + bytes)
[30]                    // age (int/u32)
[2]                     // number of addresses (u16)
  [len][Main St]        // address 1 street
  [123]                 // address 1 number
  [len][USA]           // address 1 country
  [len][Side St]       // address 2 street
  [456]                // address 2 number
  [len][Canada]        // address 2 country
[2]                     // number of metadata entries (u16)
  [len][id]            // key 1
  [len][abc123]        // value 1
  [len][type]          // key 2
  [len][customer]      // value 2

Implementation Details

Binary Format

The encoded data follows this format for different types:

Primitive Types

• string: u16 length prefix + raw string bytes
• int (32-bit): 4 bytes in little-endian format
• i64 (64-bit): 8 bytes in little-endian format
• f64: 8 bytes (IEEE-754 double precision) in little-endian format
• bool: Single byte (1 for true, 0 for false)
• u8: Single byte
• u16: 2 bytes in little-endian format
• u32: 4 bytes in little-endian format
• u64: 8 bytes in little-endian format
• percentage: Single byte (0-100)

Special Types

• time.Time: Encoded as u32 Unix timestamp (seconds since epoch)
• ourtime.OurTime: Encoded as u32 Unix timestamp
• gid.GID: Encoded as string in format "circle:id"
• currency.Amount: Encoded as a string (currency name) followed by f64 (value)
• []byte (raw byte arrays): u32 length prefix + raw bytes

Collections

• Arrays ([]T):

• u16 length prefix (number of elements)

• Each element encoded according to its type

• Maps:

• u16 count of entries

• For each entry:

• Key encoded according to its type

• Value encoded according to its type

Size Limits

• Strings and arrays are limited to 64KB in length (u16 max)
• This limit helps prevent memory issues and ensures efficient processing

fn decode[T](data []u8) !T

fn decoder_new(data []u8) Decoder

fn encode[T](obj T) ![]u8

example see https://github.com/vlang/v/blob/master/examples/compiletime/reflection.v

fn new() Encoder

enum DataType{ string int bytes u8 u16 u32 u64 time list_string list_int list_u8 list_u16 list_u32 list_u64 map_string map_bytes }

struct Decoder {
pub mut:
	version u8 = 1 // is important
	data    []u8
}

fn (mut d Decoder) get_string() !string

fn (mut d Decoder) get_int() !int

fn (mut d Decoder) get_bytes() ![]u8

fn (mut d Decoder) get_bool() !bool

fn (mut d Decoder) get_u8() !u8

adds u16 length of string in bytes + the bytes

fn (mut d Decoder) get_u16() !u16

fn (mut d Decoder) get_u32() !u32

fn (mut d Decoder) get_u64() !u64

fn (mut d Decoder) get_i64() !i64

fn (mut d Decoder) get_f64() !f64

fn (mut d Decoder) get_time() !time.Time

fn (mut d Decoder) get_ourtime() !ourtime.OurTime

fn (mut d Decoder) get_currency() !currency.Amount

fn (mut d Decoder) get_percentage() !u8

fn (mut d Decoder) get_list_string() ![]string

fn (mut d Decoder) get_list_int() ![]int

fn (mut d Decoder) get_list_u8() ![]u8

fn (mut d Decoder) get_list_u16() ![]u16

fn (mut d Decoder) get_list_u32() ![]u32

fn (mut d Decoder) get_list_u64() ![]u64

fn (mut d Decoder) get_map_string() !map[string]string

fn (mut d Decoder) get_map_bytes() !map[string][]u8

fn (mut d Decoder) get_gid() !gid.GID

Gets GID from encoded string

struct Encoder {
pub mut:
	data []u8
	// datatypes []DataType
}

fn (mut b Encoder) add_string(data string)

adds u16 length of string in bytes + the bytes

fn (mut b Encoder) add_int(data int)

Please note that unlike C and Go, int is always a 32 bit integer. We borrow the add_u32() function to handle the encoding of a 32 bit type

fn (mut b Encoder) add_bytes(data []u8)

add bytes or bytestring

fn (mut b Encoder) add_bool(data bool)

fn (mut b Encoder) add_u8(data u8)

fn (mut b Encoder) add_u16(data u16)

fn (mut b Encoder) add_u32(data u32)

fn (mut b Encoder) add_u64(data u64)

fn (mut b Encoder) add_i64(data i64)

fn (mut b Encoder) add_time(data time.Time)

fn (mut b Encoder) add_ourtime(data ourtime.OurTime)

fn (mut b Encoder) add_currency(data currency.Amount)

fn (mut b Encoder) add_f64(data f64)

adds a float64 value

fn (mut b Encoder) add_gid(data gid.GID)

adds gid as a string

fn (mut b Encoder) add_percentage(data u8)

fn (mut b Encoder) add_list_string(data []string)

fn (mut b Encoder) add_list_int(data []int)

fn (mut b Encoder) add_list_u8(data []u8)

fn (mut b Encoder) add_list_u16(data []u16)

fn (mut b Encoder) add_list_u32(data []u32)

fn (mut b Encoder) add_list_u64(data []u64)

fn (mut b Encoder) add_map_string(data map[string]string)

when complicated hash e.g. map of other object need to serialize each sub object

fn (mut b Encoder) add_map_bytes(data map[string][]u8)

when complicated hash e.g. map of other object need to serialize each sub object