OurDB Module

OurDB is a lightweight, efficient key-value database implementation in V that provides data persistence with history tracking capabilities. It's designed for scenarios where you need fast key-value storage with the ability to track changes over time.

Usage Example

//record_nr_max u32 = 16777216 - 1    // max number of records
//record_size_max u32 = 1024*4        // max record size (4KB default)
//file_size u32 = 500 * (1 << 20)     // file size (500MB default)
//path string                         // storage directory

import freeflowuniverse.crystallib.data.ourdb

mut db := ourdb.new(path:'/tmp/mydb')!

// Store data (note: set() takes []u8 as value)
db.set(1, 'Hello World'.bytes())!

// Retrieve data
data := db.get(1)! // Returns []u8

// Get history
history := db.get_history(1, 5)! // Get last 5 versions

// Delete data
db.delete(1)!

Features

• Efficient key-value storage
• History tracking for values
• Data integrity verification using CRC32
• Support for multiple backend files
• Configurable record sizes and counts
• Memory and disk-based lookup tables

Architecture

OurDB consists of three main components working together in a layered architecture:

1. Frontend (db.v)

• Provides the public API for database operations
• Handles high-level operations (set, get, delete, history)
• Coordinates between lookup and backend components
• Located in db.v

2. Lookup Table (lookup.v)

• Maps keys to physical locations in the backend storage
• Supports both memory and disk-based lookup tables
• Configurable key sizes for optimization
• Handles sparse data efficiently
• Located in lookup.v

3. Backend Storage (backend.v)

• Manages the actual data storage in files
• Handles data integrity with CRC32 checksums
• Supports multiple file backends for large datasets
• Implements the low-level read/write operations
• Located in backend.v

File Structure

• db.v: Frontend interface providing the public API
• lookup.v: Implementation of the lookup table system
• backend.v: Low-level data storage implementation
• factory.v: Database initialization and configuration
• db_test.v: Test suite for verifying functionality

How It Works

1. Frontend Operations

• When you call set(key, value), the frontend:

1. Gets the storage location from the lookup table

2. Passes the data to the backend for storage

3. Updates the lookup table with any new location

4. Lookup Table

• Maintains a mapping between keys and physical locations
• Optimizes key size based on maximum record count
• Can be memory-based for speed or disk-based for large datasets
• Supports sparse data storage for efficient space usage

3. Backend Storage

• Stores data in one or multiple files
• Each record includes:
• Data size
• CRC32 checksum
• Previous record location (for history)
• Actual data
• Automatically handles file selection and management

Implementation Details

Record Format

Each record in the backend storage includes:- 2 bytes: Data size

• 4 bytes: CRC32 checksum
• 6 bytes: Previous record location
• N bytes: Actual data

Lookup Table Optimization

The lookup table automatically optimizes its key size based on:- Total number of records (affects address space)

• Record size and count (determines file splitting)
• Available memory (can switch to disk-based lookup)

File Management

• Supports splitting data across multiple files when needed
• Each file is limited to 500MB by default (configurable)
• Automatic file selection based on record location
• Files are created as needed with format: ${path}/${file_nr}.db

fn new(args OurDBConfig) !OurDB

new_memdb creates a new memory database with the given path and lookup table

struct Location {
pub mut:
	file_nr  u16
	position u32
}

in lookuptable.keysize we specify what nr of bytes are we use as id they are encoded big_endian if + 2^32 then we know we store the data in multiple files, the most significant parts define the filenr, the others the id

struct LookupConfig {
pub:
	size       u32    // size of the table
	keysize    u8     // size of each entry in bytes (2-6), 6 means we store data over multiple files
	lookuppath string // if set, use disk-based lookup

	incremental_mode bool = true
}

struct LookupTable {
	keysize    u8
	lookuppath string
mut:
	data        []u8
	incremental ?u32 // points to next empty slot in the lookup table if incremental mode is enabled
}

struct OurDB {
mut:
	lookup &LookupTable
pub:
	path             string // is the directory in which we will have the lookup db as well as all the backend
	incremental_mode bool
	file_size        u32 = 500 * (1 << 20) // 500MB
pub mut:
	file              os.File
	file_nr           u16 // the file which is open
	last_used_file_nr u16
}

OurDB represents a binary database with variable-length records

fn (mut db OurDB) delete(x u32) !

delete removes the data at the specified key position This operation zeros out the record but maintains the space in the file Use condense() to reclaim space from deleted records (happens in step after)

fn (mut db OurDB) get(x u32) ![]u8

get retrieves data stored at the specified key position Returns error if the key doesn't exist or data is corrupted

fn (mut db OurDB) get_history(x u32, depth u8) ![][]u8

get_history retrieves a list of previous values for the specified key depth parameter controls how many historical values to retrieve (max) Returns error if key doesn't exist or if there's an issue accessing the data

fn (mut db OurDB) set(args OurDBSetArgs) !u32

fn (mut db OurDB) set_(x u32, old_location Location, data []u8) !

set stores data at position x

struct OurDBConfig {
pub:
	record_nr_max   u32 = 16777216 - 1    // max size of records
	record_size_max u32 = 1024 * 4        // max size in bytes of a record, is 4 KB default
	file_size       u32 = 500 * (1 << 20) // 500MB
	path            string // directory where we will stor the DB

	incremental_mode bool = true
}