a OurDBFS: filesystem interface on top of ourbd

The OurDBFS manages files and directories using unique identifiers (u32) as keys and binary data ([]u8) as values.

Architecture

Storage Backend (the ourdb)

• Uses a key-value store where keys are u32 and values are []u8 (bytes)
• Stores both metadata and file data in the same database
• Example usage of underlying database:

import crystallib.data.ourdb

mut db_meta := ourdb.new(path:'/tmp/mydb')!

// Store data
db_meta.set(1, 'Hello World'.bytes())!

// Retrieve data
data := db_meta.get(1)! // Returns []u8

// Delete data
db_meta.delete(1)!

Core Components

1. Common Metadata (common.v)

All filesystem entries (files and directories) share common metadata:

pub struct Metadata {
    id          u32    // unique identifier used as key in DB
    name        string // name of file or directory
    file_type   FileType
    size        u64
    created_at  i64    // unix epoch timestamp
    modified_at i64    // unix epoch timestamp
    accessed_at i64    // unix epoch timestamp
    mode        u32    // file permissions
    owner       string
    group       string
}

2. Files (file.v)

Files are represented as:

pub struct File {
    metadata    Metadata  // Common metadata
    parent_id   u32      // ID of parent directory
    data_blocks []u32    // List of block IDs containing file data
}

3. Directories (directory.v)

Directories are represented as:

pub struct Directory {
    metadata    Metadata  // Common metadata
    parent_id   u32      // ID of parent directory
    children    []u32    // List of child IDs (files and directories)
}

4. Data Storage (data.v)

File data is stored in blocks:

pub struct DataBlock {
    id    u32   // Block ID
    data  []u8  // Actual data content
    size  u32   // Size of data in bytes
    next  u32   // ID of next block (0 if last block)
}

Features

1. Hierarchical Structure

• Files and directories are organized in a tree structure
• Each entry maintains a reference to its parent directory
• Directories maintain a list of child entries

2. Metadata Management

• Comprehensive metadata tracking including:
• Creation, modification, and access timestamps
• File permissions
• Owner and group information
• File size and type

3. File Operations

• File creation and deletion
• Data block management for file content
• Future support for read/write operations

4. Directory Operations

• Directory creation and deletion
• Listing directory contents (recursive and non-recursive)
• Child management

Implementation Details

1. File Types

pub enum FileType {
    file
    directory
    symlink
}

2. Data Block Management

• File data is split into blocks
• Blocks are linked using the 'next' pointer
• Each block has a unique ID for retrieval

3. Directory Traversal

• Supports both recursive and non-recursive listing
• Uses child IDs for efficient navigation

TODO Items

TODO: what is implemented and what not?

1. Directory Implementation

• Implement recursive listing functionality
• Proper cleanup of children during deletion
• ID generation system

2. File Implementation

• Proper cleanup of data blocks
• Data block management system
• Read/Write operations

3. General Improvements

• Transaction support
• Error handling
• Performance optimizations
• Concurrency support

use @encoder dir to see how to encode/decode

make an efficient encoder for Directory add a id u32 to directory this will be the key of the keyvalue stor used

try to use as few as possible bytes when doing the encoding

the first byte is a version nr, so we know if we change the encoding format we can still decode

we will only store directories

fn decode_directory(data []u8) !Directory

decode_directory decodes a binary format back to Directory

fn decode_file(data []u8) !File

decode_file decodes a binary format back to File

fn decode_symlink(data []u8) !Symlink

decode_symlink decodes a binary format back to Symlink

fn new(params VFSParams) !&OurDBFS

Factory method for creating a new OurDBFS instance

type FSEntry = Directory | File | Symlink

FSEntry represents any type of filesystem entry

enum FileType {
	file
	directory
	symlink
}

FileType represents the type of a filesystem entry

struct DataBlock {
pub mut:
	id   u32  // Block ID
	data []u8 // Actual data content
	size u32  // Size of data in bytes
	next u32  // ID of next block (0 if last block)
}

DataBlock represents a block of file data

struct Directory {
pub mut:
	metadata  Metadata // Metadata from models_common.v
	children  []u32    // List of child entry IDs (instead of actual entries)
	parent_id u32      // ID of parent directory (0 for root)
	myvfs     &OurDBFS @[skip]
}

Directory represents a directory in the virtual filesystem

fn (mut dir Directory) add_symlink(symlink Symlink) !

add_symlink adds an existing symlink to this directory

fn (mut dir Directory) children(recursive bool) ![]FSEntry

get_children returns all immediate children as FSEntry objects

fn (mut dir Directory) delete()

fn (dir Directory) encode() []u8

encode encodes a Directory to binary format

fn (mut dir Directory) mkdir(name string) !&Directory

mkdir creates a new directory with default permissions

fn (mut dir Directory) printall(indent string) !string

printall prints the directory structure recursively

fn (mut dir Directory) read(name string) !string

read reads content from a file

fn (mut dir Directory) rm(name string) !

rm removes a file or directory by name

fn (mut self Directory) save() !

fn (mut dir Directory) str() string

str returns a formatted string of directory contents (non-recursive)

fn (mut dir Directory) touch(name string) !&File

touch creates a new empty file with default permissions

fn (mut dir Directory) write(name string, content string) !&File

write creates a new file or writes to an existing file

struct File {
pub mut:
	metadata  Metadata // Metadata from models_common.v
	data      string   // File content stored in DB
	parent_id u32      // ID of parent directory
	myvfs     &OurDBFS @[skip]
}

File represents a file in the virtual filesystem

fn (f File) encode() []u8

encode encodes a File to binary format

fn (mut f File) read() !string

read returns the file's content

fn (mut f File) save() !

fn (mut f File) write(content string) !

write updates the file's content

struct Metadata {
pub mut:
	id          u32    // unique identifier used as key in DB
	name        string // name of file or directory
	file_type   FileType
	size        u64
	created_at  i64 // unix epoch timestamp
	modified_at i64 // unix epoch timestamp
	accessed_at i64 // unix epoch timestamp
	mode        u32 // file permissions
	owner       string
	group       string
}

Metadata represents the common metadata for both files and directories

fn (m Metadata) created_time() time.Time

Get time.Time objects from epochs

fn (m Metadata) modified_time() time.Time

fn (m Metadata) accessed_time() time.Time

struct OurDBFS {
pub mut:
	root_id      u32          // ID of root directory
	block_size   u32          // Size of data blocks in bytes
	data_dir     string       // Directory to store OurDBFS data
	metadata_dir string       // Directory where we store the metadata
	db_data      &ourdb.OurDB // Database instance for persistent storage
	db_meta      &ourdb.OurDB // Database instance for metadata storage
}

OurDBFS represents the virtual filesystem

fn (mut fs OurDBFS) get_root() !&Directory

get_root returns the root directory

fn (mut fs OurDBFS) save_entry(entry FSEntry) !u32

save_entry saves an entry to the database

fn (mut fs OurDBFS) delete_entry(id u32) !

delete_entry deletes an entry from the database

struct Symlink {
pub mut:
	metadata  Metadata // Metadata from models_common.v
	target    string   // Path that this symlink points to
	parent_id u32      // ID of parent directory
	myvfs     &OurDBFS @[skip]
}

Symlink represents a symbolic link in the virtual filesystem

fn (sl Symlink) encode() []u8

encode encodes a Symlink to binary format

fn (mut sl Symlink) get_target() !string

get_target returns the current target path

fn (mut sl Symlink) save() !

fn (mut sl Symlink) update_target(new_target string) !

update_target changes the symlink's target path

struct VFSParams {
pub:
	data_dir     string // Directory to store OurDBFS data
	metadata_dir string // Directory to store OurDBFS metadata
}

Factory method for creating a new OurDBFS instance