Database Filesystem Implementation (vfs_db)

A virtual filesystem implementation that uses OurDB as its storage backend, providing a complete filesystem interface with database-backed storage.

Features

• Persistent storage in OurDB database
• Full support for files, directories, and symlinks
• Transactional operations
• Structured metadata storage
• Hierarchical filesystem structure
• Thread-safe operations

Implementation Details

Structure

vfs_db/
├── factory.v               # VFS factory implementation
├── vfs_implementation.v    # Core VFS interface implementation
├── vfs.v                  # DatabaseVFS type definition
├── model_file.v           # File type implementation
├── model_directory.v      # Directory type implementation
├── model_symlink.v        # Symlink type implementation
├── model_fsentry.v        # Common FSEntry interface
├── metadata.v             # Metadata structure
├── encoder.v              # Data encoding utilities
├── vfs_directory.v        # Directory operations
├── vfs_getters.v         # Common getter methods
└── *_test.v              # Implementation tests

Key Components

• DatabaseVFS: Main implementation struct

pub struct DatabaseVFS {
pub mut:
    root_id          u32    
    block_size       u32    
    data_dir         string 
    metadata_dir     string 
    db_data          &Database
    last_inserted_id u32
}

• FSEntry implementations:

pub type FSEntry = Directory | File | Symlink

Data Storage

Metadata Structure

struct Metadata {
    id          u32    // Unique identifier
    name        string // Entry name
    file_type   FileType
    size        u64
    created_at  i64    // Unix timestamp
    modified_at i64
    accessed_at i64
    mode        u32    // Permissions
    owner       string
    group       string
}

Database Interface

pub interface Database {
mut:
    get(id u32) ![]u8
    set(ourdb.OurDBSetArgs) !u32
    delete(id u32)!
}

Usage

import freeflowuniverse.herolib.vfs.vfs_db

// Create separate databases for data and metadata
mut db_data := ourdb.new(
    path: os.join_path(test_data_dir, 'data')
    incremental_mode: false
)!

mut db_metadata := ourdb.new(
    path: os.join_path(test_data_dir, 'metadata')
    incremental_mode: false
)!

// Create VFS with separate databases for data and metadata
mut fs := new(mut db_data, mut db_metadata)!

// Create directory structure
fs.dir_create('documents')!
fs.dir_create('documents/reports')!

// Create and write files
fs.file_create('documents/reports/q1.txt')!
fs.file_write('documents/reports/q1.txt', 'Q1 Report Content'.bytes())!

// Create symbolic links
fs.link_create('documents/reports/q1.txt', 'documents/latest.txt')!

// List directory contents
entries := fs.dir_list('documents')!
for entry in entries {
    println('${entry.get_path()} (${entry.get_metadata().size} bytes)')
}

// Clean up
fs.destroy()!

Implementation Notes

1. Data Encoding:

• Version byte for format compatibility
• Entry type indicator
• Entry-specific binary data
• Efficient storage format

2. Thread Safety:

• Mutex protection for concurrent access
• Atomic operations
• Clear ownership semantics

3. Error Handling:

• Descriptive error messages
• Proper error propagation
• Recovery mechanisms
• Consistency checks

Limitations

• Performance overhead compared to direct filesystem access
• Database size grows with filesystem usage
• Requires proper database maintenance
• Limited by database backend capabilities

Testing

The implementation includes tests for:- Basic operations (create, read, write, delete)

• Directory operations and traversal
• Symlink handling
• Concurrent access
• Error conditions
• Edge cases
• Data consistency

Run tests with:

v test vfs/vfs_db/

Future Improvements

1. Performance Optimizations:

• Entry caching
• Batch operations
• Improved traversal algorithms

2. Feature Additions:

• Extended attributes
• Access control lists
• Quota management
• Transaction support

3. Robustness:

• Automated recovery
• Consistency verification
• Better error handling
• Backup/restore capabilities

fn decode_directory(data []u8) !Directory

decode_directory decodes a binary format back to Directory

fn decode_file_metadata(data []u8) !File

decode_file decodes a binary format back to File (without the actual file data) returns file without data and the sequence of keys of chunks of data in data db

fn decode_symlink(data []u8) !Symlink

decode_symlink decodes a binary format back to Symlink

fn new(mut data_db Database, mut metadata_db Database) !&DatabaseVFS

Factory method for creating a new DatabaseVFS instance

interface Database {
mut:
	get(id u32) ![]u8
	set(ourdb.OurDBSetArgs) !u32
	delete(id u32) !
}

type FSEntry = Directory | File | Symlink

FSEntry represents any type of filesystem entry

struct CopyDirArgs {
pub mut:
	src_entry_name string     @[required] // source entry name
	dst_entry_name string     @[required] // destination entry name
	dst_parent_dir &Directory @[required] // destination Directory
}

struct DatabaseVFS {
pub mut:
	root_id          u32 // ID of root directory
	block_size       u32 // Size of data blocks in bytes
	db_data          &Database @[str: skip] // Database instance for file data storage
	db_metadata      &Database @[str: skip] // Database instance for metadata storage
	last_inserted_id u32
	id_table         map[u32]u32
}

DatabaseVFS represents the virtual filesystem

fn (mut self DatabaseVFS) copy(src_path_ string, dst_path_ string) !vfs.FSEntry

fn (mut fs DatabaseVFS) copy_directory(dir Directory) !&Directory

copy_directory creates a new directory with the same metadata as the source

fn (mut self DatabaseVFS) copy_file(file File) !&File

copy_file creates a copy of a file

fn (mut self DatabaseVFS) delete(path_ string) !

fn (mut self DatabaseVFS) destroy() !

fn (mut self DatabaseVFS) dir_create(path_ string) !vfs.FSEntry

fn (mut self DatabaseVFS) dir_delete(path string) !

fn (mut self DatabaseVFS) dir_list(path string) ![]vfs.FSEntry

fn (mut fs DatabaseVFS) directory_add_symlink(mut dir Directory, mut symlink Symlink) !

add_symlink adds an existing symlink to this directory

fn (mut fs DatabaseVFS) directory_children(mut dir Directory, recursive bool) ![]FSEntry

get_children returns all immediate children as FSEntry objects

fn (mut fs DatabaseVFS) directory_copy(mut dir Directory, args_ CopyDirArgs) !&Directory

fn (mut fs DatabaseVFS) directory_mkdir(mut dir Directory, name_ string) !&Directory

mkdir creates a new directory with default permissions

fn (mut fs DatabaseVFS) directory_move(dir_ Directory, args_ MoveDirArgs) !&Directory

fn (mut fs DatabaseVFS) directory_print(dir Directory) string

str returns a formatted string of directory contents (non-recursive)

fn (mut fs DatabaseVFS) directory_printall(dir Directory, indent string) !string

printall prints the directory structure recursively

fn (mut fs DatabaseVFS) directory_rename(dir Directory, src_name string, dst_name string) !FSEntry

fn (mut fs DatabaseVFS) directory_rm(mut dir Directory, name string) !

rm removes a file or directory by name

fn (mut fs DatabaseVFS) directory_touch(mut dir Directory, name_ string) !&File

touch creates a new empty file with default permissions

fn (mut self DatabaseVFS) exists(path_ string) bool

fn (mut self DatabaseVFS) file_concatenate(path_ string, data []u8) !

fn (mut self DatabaseVFS) file_create(path_ string) !vfs.FSEntry

fn (mut self DatabaseVFS) file_delete(path string) !

fn (mut self DatabaseVFS) file_read(path_ string) ![]u8

fn (mut self DatabaseVFS) file_write(path_ string, data []u8) !

fn (mut fs DatabaseVFS) get(path_ string) !vfs.FSEntry

fn (mut fs DatabaseVFS) get_next_id() u32

Get the next ID, it should be some kind of auto-incrementing ID

fn (mut self DatabaseVFS) get_path(entry_ &vfs.FSEntry) !string

fn (mut self DatabaseVFS) link_create(target_path string, link_path string) !vfs.FSEntry

fn (mut self DatabaseVFS) link_delete(path string) !

fn (mut self DatabaseVFS) link_read(path string) !string

fn (mut self DatabaseVFS) move(src_path string, dst_path string) !vfs.FSEntry

fn (mut fs DatabaseVFS) new_directory(dir NewDirectory) !&Directory

mkdir creates a new directory with default permissions

fn (mut fs DatabaseVFS) print() !

fn (mut self DatabaseVFS) rename(old_path string, new_path string) !vfs.FSEntry

fn (mut fs DatabaseVFS) root_get() !vfs.FSEntry

Implementation of VFSImplementation interface

fn (mut fs DatabaseVFS) root_get_as_dir() !&Directory

Implementation of VFSImplementation interface

fn (mut fs DatabaseVFS) save_entry(entry FSEntry) !

save_entry saves an entry to the database

fn (mut fs DatabaseVFS) save_file(file_ File, data []u8) !u32

save_entry saves an entry to the database

struct Directory {
pub mut:
	metadata  vfs.Metadata // vfs.Metadata from models_common.v
	children  []u32        // List of child entry IDs (instead of actual entries)
	parent_id u32          // ID of parent directory (0 for root)
}

Directory represents a directory in the virtual filesystem

fn (dir Directory) encode() []u8

encode encodes a Directory to binary format

fn (d &Directory) is_dir() bool

is_dir returns true if the entry is a directory

fn (d &Directory) is_file() bool

is_file returns true if the entry is a file

fn (d &Directory) is_symlink() bool

is_symlink returns true if the entry is a symlink

struct File {
pub mut:
	metadata  vfs.Metadata // vfs.Metadata from models_common.v
	parent_id u32          // ID of parent directory
	chunk_ids []u32        // a list of data addresses for chunks of 64 kb in data_db
}

File represents a file in the virtual filesystem

fn (f File) encode() []u8

File encoding/decoding encode encodes a File metadata to binary format (without the actual file data)

fn (f &File) is_dir() bool

is_dir returns true if the entry is a directory

fn (f &File) is_file() bool

is_file returns true if the entry is a file

fn (f &File) is_symlink() bool

is_symlink returns true if the entry is a symlink

struct MoveDirArgs {
pub mut:
	src_entry_name string     @[required] // source entry name
	dst_entry_name string     @[required] // destination entry name
	dst_parent_dir &Directory @[required] // destination OurDBFSDirectory
}

struct NewDirectory {
pub:
	name      string @[required] // name of file or directory
	mode      u32    = 0o755 // file permissions
	owner     string = 'user'
	group     string = 'user'
	parent_id u32
	children  []u32
}

struct NewFile {
pub:
	name      string @[required] // name of file or directory
	path      string @[required] // path of file or directory
	data      string
	mode      u32    = 0o644 // file permissions
	owner     string = 'user'
	group     string = 'user'
	parent_id u32
}

struct Symlink {
pub mut:
	metadata  vfs.Metadata // vfs.Metadata from models_common.v
	target    string       // Path that this symlink points to
	parent_id u32          // ID of parent directory
}

Symlink represents a symbolic link in the virtual filesystem

fn (sl Symlink) encode() []u8

encode encodes a Symlink to binary format

fn (mut sl Symlink) get_target() !string

get_target returns the current target path

fn (self &Symlink) is_dir() bool

is_dir returns true if the entry is a directory

fn (self &Symlink) is_file() bool

is_file returns true if the entry is a file

fn (self &Symlink) is_symlink() bool

is_symlink returns true if the entry is a symlink

fn (mut sl Symlink) update_target(new_target string) !

update_target changes the symlink's target path