Builder Module

The Builder module is a powerful system automation and remote execution framework that provides a unified interface for executing commands and managing files across both local and remote systems.

Overview

The Builder module consists of several key components:- BuilderFactory: Creates and manages builder instances

• Node: Represents a target system (local or remote) with its properties and state
• Executor: Interface for command execution and file operations (SSH or Local)
• NodeDB: Key-value store at the node level for persistent state

Getting Started

Basic Initialization

import freeflowuniverse.herolib.builder

// Create a new builder instance
mut b := builder.new()!

// Create a node for remote execution
mut n := b.node_new(ipaddr: 'root@195.192.213.2:2222')!

// Or create a local node
mut local_node := builder.node_local()!

Node Configuration

Nodes can be configured with various properties:

// Full node configuration
mut n := b.node_new(
    name: 'myserver',      // Optional name for the node
    ipaddr: 'root@server.example.com:22',  // SSH connection string
    platform: .ubuntu,     // Target platform type
    debug: true           // Enable debug output
)!

Node Properties

Each node maintains information about:- Platform type (OSX, Ubuntu, Alpine, Arch)

• CPU architecture (Intel, ARM)
• Environment variables
• System state
• Execution history

The node automatically detects and caches system information for better performance.

Executor Interface

The executor provides a unified interface for both local and remote operations:

Command Execution

// Execute command and get output
result := n.exec('ls -la')!

// Execute silently (no output)
n.exec_silent('mkdir -p /tmp/test')!

// Interactive shell
n.shell('bash')!

File Operations

// Write file
n.file_write('/path/to/file', 'content')!

// Read file
content := n.file_read('/path/to/file')!

// Check existence
exists := n.file_exists('/path/to/file')

// Delete file/directory
n.delete('/path/to/delete')!

// List directory contents
files := n.list('/path/to/dir')!

// File transfers
n.download('http://example.com/file', '/local/path')!
n.upload('/local/file', '/remote/path')!

Environment Management

// Get all environment variables
env := n.environ_get()!

// Get node information
info := n.info()

Node Database (NodeDB)

The NodeDB provides persistent key-value storage at the node level:

// Store a value
n.done['key'] = 'value'
n.save()!

// Load stored values
n.load()!
value := n.done['key']

This is useful for:- Caching system information

• Storing configuration state
• Tracking execution history
• Maintaining persistent data between sessions

Best Practices

1. Error Handling: Always use the ! operator for methods that can fail and handle errors appropriately.

2. Resource Management: Close connections and clean up resources when done:

defer {
    n.close()
}

3. Debug Mode: Enable debug mode when troubleshooting:

n.debug_on()  // Enable debug output
n.debug_off() // Disable debug output

4. Platform Awareness: Check platform compatibility before executing commands:

if n.platform == .ubuntu {
    // Ubuntu-specific commands
} else if n.platform == .osx {
    // macOS-specific commands
}

Examples

See complete examples in:- Simple usage: examples/builder/simple.vsh

• Remote execution: examples/builder/remote_executor/
• Platform-specific examples:
• IPv4: examples/builder/simple_ip4.vsh
• IPv6: examples/builder/simple_ip6.vsh

Implementation Details

The Builder module uses:- Redis for caching node information

• SSH for secure remote execution
• MD5 hashing for unique node identification
• JSON for data serialization
• Environment detection for platform-specific behavior

For more detailed implementation information, refer to the source code in the lib/builder/ directory.

fn bootstrapper() BootStrapper

to use do something like: export NODES="195.192.213.3" .

fn new() !BuilderFactory

fn node_local(args NodeLocalArgs) !&Node

fn portforward_to_local(args_ ForwardArgsToLocal) !

forward a remote port on ssh host to a local port

fn this_remote_exec(args_ ThisRemoteArgs) !bool

to use do something like: export NODES="195.192.213.3" .

enum CPUType {
	unknown
	intel
	arm
	intel32
	arm32
}

enum PlatformType {
	unknown
	osx
	ubuntu
	alpine
	arch
}

struct BootStrapper {
pub mut:
	embedded_files map[string]embed_file.EmbedFileData @[skip; str: skip]
}

fn (mut bs BootStrapper) run(args_ BootstrapperArgs) !

struct BootstrapperArgs {
pub mut:
	name  string = 'bootstrap'
	addr  string // format:  root@something:33, 192.168.7.7:222, 192.168.7.7, despiegk@something
	reset bool
	debug bool
}

struct BuilderFactory {
}

fn (mut bldr BuilderFactory) node_local() !&Node

get node connection to local machine

fn (mut bldr BuilderFactory) node_new(args_ NodeArguments) !&Node

the factory which returns an node, based on the arguments will chose ssh executor or the local one .

 - format ipaddr: localhost:7777
 - format ipaddr: myuser@192.168.6.6:7777
 - format ipaddr: 192.168.6.6
 - format ipaddr: any ipv6 addr
 - if only name used then is localhost with localhost executor

its possible to put a user as user@ .. in front of an ip addr . .

 pub struct NodeArguments {
    ipaddr string
    name   string //if not filled in will come from ipaddr
    user   string = "root"
    debug  bool
    reset bool
    }

struct EnvGetParams {
pub mut:
	reload bool
}

struct ExecArgs {
pub mut:
	cmd    string
	stdout bool = true
}

struct ExecRetryArgs {
pub:
	cmd          string
	retrymax     int  = 10  // how may times maximum to retry
	period_milli int  = 100 // sleep in between retry in milliseconds
	timeout      int  = 2   // timeout for al the tries together
	stdout       bool = true
}

pub fn (mut node Node) exec_file(args_ ExecFileArgs) !string { mut args:=args_ if args.path == '' { now := ourtime.now() args.path="/tmp/myexec_${now.key()}.sh" } if args.cmd == '' { return error('need to specify cmd') } args.cmd = texttools.dedent(args.cmd) node.file_write(args.path, args.cmd)! return node.exec_silent('chmod +x ${args.path} && bash ${args.path} && rm -f ${args.path}')! // }

struct ExecutorLocal {
	retry int = 1 // nr of times something will be retried before failing, need to check also what error is, only things which should be retried need to be done, default 1 because is local
pub mut:
	debug bool
}

fn (mut executor ExecutorLocal) exec(args ExecArgs) !string

fn (mut executor ExecutorLocal) exec_interactive(args ExecArgs) !

fn (mut executor ExecutorLocal) file_write(path string, text string) !

fn (mut executor ExecutorLocal) file_read(path string) !string

fn (mut executor ExecutorLocal) file_exists(path string) bool

fn (mut executor ExecutorLocal) debug_on()

fn (mut executor ExecutorLocal) debug_off()

fn (mut executor ExecutorLocal) delete(path string) !

carefull removes everything

fn (mut executor ExecutorLocal) environ_get() !map[string]string

get environment variables from the executor

fn (mut executor ExecutorLocal) info() map[string]string

fn (mut executor ExecutorLocal) upload(args SyncArgs) !

upload from local FS to executor FS

fn (mut executor ExecutorLocal) download(args SyncArgs) !

download from executor FS to local FS

fn (mut executor ExecutorLocal) shell(cmd string) !

fn (mut executor ExecutorLocal) list(path string) ![]string

fn (mut executor ExecutorLocal) dir_exists(path string) bool

struct ExecutorNewArguments {
pub mut:
	local  bool // if this set then will always be the local machine
	ipaddr string
	user   string = 'root'
	debug  bool
}

struct ExecutorSSH {
pub mut:
	ipaddr      ipaddress.IPAddress
	sshkey      string
	user        string = 'root' // default will be root
	initialized bool
	retry       int  = 1 // nr of times something will be retried before failing, need to check also what error is, only things which should be retried need to be done
	debug       bool = true
}

fn (mut executor ExecutorSSH) debug_on()

fn (mut executor ExecutorSSH) debug_off()

fn (mut executor ExecutorSSH) exec(args_ ExecArgs) !string

fn (mut executor ExecutorSSH) exec_interactive(args_ ExecArgs) !

fn (mut executor ExecutorSSH) file_write(path string, text string) !

fn (mut executor ExecutorSSH) file_read(path string) !string

fn (mut executor ExecutorSSH) file_exists(path string) bool

fn (mut executor ExecutorSSH) delete(path string) !

carefull removes everything

fn (mut executor ExecutorSSH) download(args SyncArgs) !

upload from local FS to executor FS

fn (mut executor ExecutorSSH) upload(args SyncArgs) !

download from executor FS to local FS

fn (mut executor ExecutorSSH) environ_get() !map[string]string

get environment variables from the executor

fn (mut executor ExecutorSSH) info() map[string]string

fn (mut executor ExecutorSSH) shell(cmd string) !

ssh shell on the node default ssh port, or any custom port that may be forwarding ssh traffic to certain container

fn (mut executor ExecutorSSH) list(path string) ![]string

fn (mut executor ExecutorSSH) dir_exists(path string) bool

struct ForwardArgsToLocal {
pub mut:
	name        string @[required]
	address     string @[required]
	remote_port int    @[required]
	local_port  int
	user        string = 'root'
}

struct HeroInstallArgs {
pub mut:
	reset bool
}

struct HeroUpdateArgs {
pub mut:
	sync_from_local bool // will sync local hero lib to the remote, then cannot use git
	sync_full       bool // sync the full herolib repo
	sync_fast       bool = true // don't hash the files, there is small chance on error
	git_reset       bool // will get the code from github at remote and reset changes
	git_pull        bool // will pull the code but not reset, will give error if it can't reset	
	branch          string
}

' console.print_debug('executing cmd ${cmd}') node.exec_cmd(cmd: cmd)! }

struct Node {
mut:
	factory &BuilderFactory @[skip; str: skip]
pub mut:
	name        string = 'unknown'
	executor    Executor @[skip; str: skip]
	platform    PlatformType
	cputype     CPUType
	done        map[string]string
	environment map[string]string
	params      Params
	hostname    string
}

fn (mut node Node) cmd_exists(cmd string) bool

check command exists on the platform, knows how to deal with different platforms

fn (mut node Node) command_exists(cmd string) bool

checks if given executable exists in node

fn (mut node Node) dagu_install() !

fn (mut node Node) debug_off()

fn (mut node Node) debug_on()

fn (mut node Node) delete(path string) !

fn (mut node Node) dir_exists(path string) bool

fn (mut node Node) done_exists(key string) bool

fn (mut node Node) done_get(key string) ?string

fn (mut node Node) done_get_int(key string) int

will return 0 if it doesnt exist

fn (mut node Node) done_get_str(key string) string

will return empty string if it doesnt exist

fn (mut node Node) done_print()

fn (mut node Node) done_reset() !

fn (mut node Node) done_set(key string, val string) !

fn (mut node Node) download(args_ SyncArgs) !

download files using rsync (can be ssh or local) . args: .

source string
dest string
delete bool //do we want to delete the destination
ignore []string //arguments to ignore e.g. ['*.pyc','*.bak']
ignore_default bool = true //if set will ignore a common set
stdout bool = true

.

fn (mut node Node) environ_get(args EnvGetParams) !map[string]string

fn (mut node Node) exec(args ExecArgs) !string

exec(cmd string) !string exec_silent(cmd string) !string file_write(path string, text string) ! file_read(path string) !string file_exists(path string) bool delete(path string) !

fn (mut node Node) exec_cmd(args_ NodeExecCmd) !string

cmd: cmd to execute . period in sec, e.g. if 3600, means will only execute this if not done yet within the hour . . ARGS: .

 struct NodeExecCmd{
    cmd string
    period int //period in which we check when this was done last, if 0 then period is indefinite
    reset bool = true
    description string
    checkkey string //if used will use this one in stead of hash of cmd, to check if was executed already
 }

fn (mut node Node) exec_interactive(cmd_ string) !

fn (mut node Node) exec_ok(cmd string) bool

check if we can execute and there is not errorcode

fn (mut node Node) exec_retry(args ExecRetryArgs) !string

a cool way to execute something until it succeeds params: cmd string retrymax int = 10 //how may times maximum to retry period_milli int = 100 //sleep in between retry in milliseconds timeout int = 2 //timeout for al the tries together

fn (mut node Node) exec_silent(cmd string) !string

silently execute a command

fn (mut node Node) file_exists(path string) bool

fn (mut node Node) file_read(path string) !string

fn (mut node Node) file_write(path string, text string) !

fn (mut node Node) hero_compile() !

fn (mut node Node) hero_compile_sync() !

sync local hero code to rmote and then compile hero

fn (mut node Node) hero_install() !

fn (mut node Node) hero_update(args_ HeroUpdateArgs) !

execute vscript on remote node

fn (mut node Node) info() map[string]string

fn (mut node Node) ipaddr_pub_get() !string

return the ipaddress as known on the public side is using resolver4.opendns.com

fn (mut node Node) key() string

get unique key for the node, as used in caching environment

fn (mut node Node) list(path string) ![]string

list(path string) ![]string dir_exists(path string) bool debug_off() debug_on()

fn (mut node Node) load() !bool

load the node from redis cache, if not there will load from system . return true if the data was in redis (cache)

fn (mut node Node) package_install(package Package) !

fn (mut node Node) package_refresh() !

fn (mut node Node) readfromsystem() !

get remote environment arguments in memory

fn (mut node Node) save() !

get remote environment arguments in memory

fn (mut node Node) shell(cmd string) !

fn (mut node Node) sync_code(name string, src_ string, dest string, fast_rsync bool) !

fn (mut node Node) upgrade() !

fn (mut node Node) upload(args_ SyncArgs) !

upload files using rsync (can be ssh or local) args: .

source string
dest string
delete bool //do we want to delete the destination
ignore []string //arguments to ignore e.g. ['*.pyc','*.bak']
ignore_default bool = true //if set will ignore a common set
stdout bool = true

.

fn (mut node Node) vscript(args_ VScriptArgs) !

struct NodeArguments {
pub mut:
	ipaddr string
	name   string
	user   string = 'root'
	debug  bool
	reload bool
}

format ipaddr: localhost:7777 . format ipaddr: 192.168.6.6:7777 . format ipaddr: 192.168.6.6 . format ipaddr: any ipv6 addr . format ipaddr: if only name used then is localhost .

struct NodeExecCmd {
pub mut:
	name               string = 'default'
	cmd                string
	period             int // period in which we check when this was done last, if 0 then period is indefinite
	reset              bool = true // means do again or not
	remove_installer   bool = true // delete the installer
	description        string
	stdout             bool = true
	checkkey           string // if used will use this one in stead of hash of cmd, to check if was executed already
	tmpdir             string
	ignore_error_codes []int
}

struct NodeLocalArgs {
pub:
	reload bool
}

struct Package {
	name        string
	description string
	version     string
	aliases     []PackageAlias
}

is e.g. an ubuntu packagedapp, it needs to be packaged by the package maintainers !

fn (mut package Package) name_get(platformtype PlatformType) string

get the right name depending the platform type

fn (mut package Package) version_get(platformtype PlatformType) string

get the right name depending the platform type

struct PackageAlias {
	name         string
	platformtype PlatformType
	version      string
}

if there is an exception of how package needs to be installed (alias) e.g. on ubuntu something is called myapp but on alpine its my_app

struct SyncArgs {
pub mut:
	source         string
	dest           string
	delete         bool     // do we want to delete the destination
	ipaddr         string   // e.g. root@192.168.5.5:33 (can be without root@ or :port)
	ignore         []string // arguments to ignore e.g. ['*.pyc','*.bak']
	ignore_default bool = true // if set will ignore a common set
	stdout         bool = true
	fast_rsync     bool = true
}

struct ThisRemoteArgs {
pub mut:
	name            string = 'remote'
	nodes           string
	script          string
	sync_from_local bool
}

struct VScriptArgs {
pub mut:
	path            string
	sync_from_local bool   // will sync local hero lib to the remote
	git_reset       bool   // will get the code from github at remote and reset changes
	git_pull        bool   // will pull the code but not reset, will give error if it can't reset
	branch          string // can only be used when git used
}