Doctree Module

The doctree module is a V language library designed for scanning, processing, and exporting collections of documents. It provides a structured way to manage document-based content, making it suitable for generating documentation, building static websites, or processing any content organized into collections.

Purpose

The primary goal of this module is to transform structured document collections into a format suitable for various outputs. It handles the complexities of finding collections, loading their content, processing includes, definitions, and macros, and exporting the final result while managing assets like images and files.

Key Concepts

• Tree: The central component (doctree.Tree) that holds one or more Collection instances. It orchestrates the scanning, processing, and exporting of all contained collections.
• Collection: A directory that is marked as a collection by the presence of a .collection file. A collection groups related documents (pages, images, files) and can have its own configuration defined within the .collection file.
• .collection file: A file placed in a directory to designate it as a collection. This file can optionally contain parameters (using the paramsparser format) such as a custom name for the collection.

How it Works (Workflow)

The typical workflow involves creating a Tree, scanning for collections, and then exporting the processed content.å

1. Create Tree: Initialize a doctree.Tree instance using doctree.new().
2. Scan: Use the tree.scan() or tree.scan_concurrent() method, providing a path to a directory or a Git repository URL. The scanner recursively looks for directories containing a .collection file.
3. Load Content: For each identified collection, the module loads its content, including markdown pages, images, and other files.
4. Process Content: The loaded content is processed. This includes handling definitions, includes (content from other files), and macros (dynamic content generation or transformation).
5. Generate Output Paths: The module determines the final paths for all processed files and assets in the destination directory.
6. Export: The tree.export() method writes the processed content and assets to the specified destination directory, maintaining the desired structure.

Usage (For Developers)

Here's a basic example of how to use the doctree module in your V project:

import freeflowuniverse.herolib.data.doctree
// 1. Create a new Tree instance
mut tree := doctree.new(name: 'my_documentation')!

// 2. Scan a directory containing your collections
// Replace './docs' with the actual path to your document collections
tree.scan(path: './docs')!

// use from URL
//git_url   string
//git_reset bool
//git_pull  bool
tree.scan(git_url: 'https://git.threefold.info/tfgrid/docs_tfgrid4/src/branch/main/collections')!

// 3. Export the processed content to a destination directory
// Replace './output' with your desired output path
// if redis then the metadata will be put in redis
tree.export(destination: './output', redis:true)!

println('Documentation successfully exported to ./output')

Structure of a Collection

A collection is a directory containing a .collection file. Inside a collection directory, you would typically organize your content like this:

my_collection/
├── .collection
├── page1.md
├── page2.md
├── images/
│   ├── image1.png
│   └── image2.jpg
└── files/
    ├── document.pdf
    └── data.csv

Markdown files (.md) are treated as pages.

Redis Structure

when using the export redis:true argument, which is default

in redis we will find

#doctree:$collectionname $pagename   $rel_path_in_collection
doctree:$collectionname $filename.$ext  $rel_path_in_collection
doctree:meta            $collectionname     $collectionpath_on_disk

fn new(args_ TreeArgsGet) !&Tree

new creates a new tree and stores it in global map

fn tree_get(name string) !&Tree

tree_get gets tree from global map

fn tree_set(tree Tree)

tree_set stores tree in global map

enum TreeState {
	init
	ok
	error
}

struct CollectionNewArgs {
mut:
	name          string @[required]
	path          string @[required]
	heal          bool = true // healing means we fix images, if selected will automatically load, remove stale links
	load          bool = true
	fail_on_error bool
}

struct CollectionNotFound {
	Error
pub:
	pointer pointer.Pointer
	msg     string
}

fn (err CollectionNotFound) msg() string

struct GraphResponse {
pub:
	graph  map[string]map[string]bool
	errors []CollectionError
}

struct MacroGetArgs {
pub mut:
	actor string
	name  string
}

struct NoOrTooManyObjFound {
	Error
pub:
	tree    &Tree
	pointer pointer.Pointer
	nr      int
}

the next is our custom error for objects not found

fn (err NoOrTooManyObjFound) msg() string

struct ObjNotFound {
	Error
pub:
	name       string
	collection string
	info       string
}

fn (err ObjNotFound) msg() string

struct Tree {
pub:
	name          string
	fail_on_error bool
pub mut:
	collections map[string]&collection.Collection
	defs        map[string]&data.Page
	state       TreeState
	// context context.Context
	cid      string = '000'
	replacer ?regext.ReplaceInstructions
}

fn (mut tree Tree) add_collection(args_ CollectionNewArgs) !

get a new collection

fn (mut tree Tree) export(args TreeExportArgs) !

export all collections to chosen directory . all names will be in name_fixed mode . all images in img/

fn (tree Tree) file_exists(pointerstr string) bool

fn (tree Tree) get_collection(name string) !&collection.Collection

fn (tree Tree) get_collection_with_pointer(p pointer.Pointer) !&collection.Collection

fn (tree Tree) get_file(pointerstr string) !&data.File

get the file from pointer string: $tree:$collection:$name or $collection:$name or $name

fn (tree Tree) get_image(pointerstr string) !&data.File

get the page from pointer string: $tree:$collection:$name or $collection:$name or $name

fn (tree Tree) image_exists(pointerstr string) bool

fn (tree Tree) key() string

the unique key to remember a tree . is unique per circle (based on cid)

fn (tree Tree) page_exists(pointerstr string) bool

fn (tree Tree) page_get(pointerstr string) !&data.Page

get the page from pointer string: $tree:$collection:$name or $collection:$name or $name

fn (mut tree Tree) process_actions_and_macros() !

adds all action elements to a playbook, calls playmacros.play on the playbook, which processes the macros, then reprocesses every page with the actions' new content

fn (mut tree Tree) process_defs() !

process definitions (!!wiki.def actions, elements.Def elements) this must be done before processing includes.

fn (mut tree Tree) process_includes() !

fn (mut tree Tree) scan(args_ TreeScannerArgs) !

walk over directory find dirs with .book or .collection inside and add to the tree . a path will not be added unless .collection is in the path of a collection dir or .book in a book

    path string
    heal bool // healing means we fix images, if selected will automatically load, remove stale links
    git_url   string
    git_reset bool
    git_root  string
    git_pull  bool
```	

fn (mut tree Tree) scan_concurrent(args_ TreeScannerArgs) !

struct TreeArgsGet {
pub mut:
	name          string = 'default'
	fail_on_error bool
}

struct TreeExportArgs {
pub mut:
	destination    string @[required]
	reset          bool = true
	keep_structure bool // wether the structure of the src collection will be preserved or not
	exclude_errors bool // wether error reporting should be exported as well
	toreplace      string
	concurrent     bool = true
	redis          bool = true
}

struct TreeScannerArgs {
pub mut:
	path      string
	heal      bool = true // healing means we fix images
	git_url   string
	git_reset bool
	git_root  string
	git_pull  bool
	load      bool = true // means we scan automatically the added collection
}