KnowledgeTree

this tool can be seen like a pre-processor for collections of markdown docs which

• fixes links
• re-sizes images
• checks on errors in the pages and report on them in errors.md file e.g. checks links in collections
• can find pages, ... in other collections
• when a image is in other collection then it get's copied in the local collection (no re-use of images from other collections)
• exports collections to a new dir all prepared to be used in another tool (e.g. mdbook)
• make it easy for other tools to find an image, page, from out of memory
• macro's get processed eg. html macro's (heroscript to nice html)
• includes are execute, include pages in other pages

the result is structure in memory or exported files in a new dir, representing the collection but processed

Tree

A tree is our main object which holds all the collections.

Collections

Are X nr of heroscripts, pages and images grouped, each collection has a name and location on normally a version controlled directory e.g. a github repo directory.

Content is

• heroscripts = action statements, which can also represent data objects
• md pages (the source as can be used for websites, books, ...)
• images (in multiple formats) or video's
• files

At the root of such a dir you have a .collection file

in this file there can be name = 'mycollectionname' if you want to specify another name as the dir, if not specified the collection name will be the dirname.

Example

import freeflowuniverse.crystallib.data.doctree

mut tree:=doctree.new(name:'test')!

// path      string
// heal      bool = true // healing means we fix images
// git_url   string
// git_reset bool
// git_root  string
// git_pull  bool
// load      bool = true // means we scan automatically the added collection
for project in 'projectinca, legal, why, web4,tfgrid3,companies'.split(',').map(it.trim_space()) {
    tree.scan(
        git_url: 'https://git.ourworld.tf/tfgrid/info_tfgrid/src/branch/development/collections/${project}'
        git_pull: false
    )!
}

tree.export(dest:'/tmp/test',reset:true,keep_structure:true,exclude_errors:false,production:true)!

heroscript example

A good example how to use hero

see https://git.ourworld.tf/tfgrid/info_tfgrid/src/branch/development/heroscript/exporter/run.sh

what are the features of the healing (fixing)

• fix all links so they point to existing path following is supported
• [mylink](MyFile .md) --> [mylink](myfile.md)
• rename the file so its always normalized: 'My__File .md' -> 'my_file.md'
• [mylink](MyFile) --> [mylink](myfile.md) //put.md at end
• [mylink](anothercollection:MyFile) --> [mylink](myfile.md) //check the collection exists, make error if not
• images get resized (see crystallib/docs/tools.imagemagick.html) and potentially converted to jpeg
• then we will go over all links to see if e.g. myimage.png became myimage.jpeg and fix the link
• an errors.md file is written so we can easily see what cannot be fixed
• if a markdown file or image is in same collection then we replace link to [mylink](pathincollection/.../mylink.md)
• this makes it easy for editing files in editor like visual studio code, we will see e.g. the images right away
• fixing of markdown
• we use the markdown parser to read the markdown and write the markdown again, this will rewrite actions, ...
• the markdown will now have reproduceable format

what are the features of the processing

• collections can be expored to specified dir
• when exporting the following will happen
• healing (even if it was not done at source), see above
• re-write markdown
• process the markdown macro's which are in the markdown (next phase, ask Kristof how)

what is the link with the mdbook osal tool

• once the collections are exported in a new structure, then we know they are clean to be processed by the mdbook tool
• that tool will read the collections again as well as summary.md and produce a book

known issues

it might be following is needed on OSX

ulimit -n 1024

fn new(args_ TreeArgsGet) !&Tree

get a new tree initialized, doesn't get remembered in global . will create a new tree instance .

 name string = 'default'
 cid  string
```	
all arguments are optional

fn pointer_new(txt_ string) !Pointer

will return a clean pointer to a page, image or file

 input is e.g. mycollection:filename.jpg
    or filename.jpg
    or mypage.md

fn pointerpath_new(args PointerPathArgs) !PointerPath

will return a pointer to filepath (needs to be a file, cannot be a dir)

fn tree_get(name string) !&Tree

get sheet from global

fn tree_set(tree Tree)

remember sheet in global

enum CollectionErrorCat {
	unknown
	image_double
	file_double
	file_not_found
	image_not_found
	page_double
	page_not_found
	sidebar
	circular_import
	def
	summary
	include
}

enum CollectionState {
	init
	initdone
	scanned
	fixed
	ok
}

enum FileStatus {
	unknown
	ok
	error
}

enum FileType {
	file
	image
}

enum PageStatus {
	unknown
	ok
	error
}

enum PointerCat {
	page
	image
	video
	file
	html
}

enum TreeState {
	init
	ok
	error
}

struct Collection {
pub:
	name string
pub mut:
	title  string
	pages  map[string]&Page // markdown pages in collection
	files  map[string]&File
	images map[string]&File
	path   Path
	errors []CollectionError
	state  CollectionState
	tree   &Tree @[str: skip]
	heal   bool = true
}

fn (mut collection Collection) error(args CollectionError)

fn (collection Collection) file_exists(name string) bool

fn (collection Collection) file_get(name_ string) !&File

fn (mut collection Collection) file_new(mut p Path) !

add a file to the collection, specify existing path

fn (collection Collection) image_exists(name string) bool

fn (collection Collection) image_get(name_ string) !&File

fn (mut collection Collection) image_new(mut p Path) !

add a image to the collection, specify existing path

fn (collection Collection) page_exists(name string) bool

fn (collection Collection) page_get(name_ string) !&Page

format of name is $collectionname:$pagename or $pagename look if we can find page in the local collection is collection name not specified if collectionname specified will look for page in that specific collection

fn (mut collection Collection) page_new(mut p Path) !

add a page to the collection, specify existing path the page will be parsed as markdown

fn (collection Collection) pagenames() []string

return all pagenames for a collection

fn (mut collection Collection) scan() !

walk over one specific collection, find all files and pages

struct CollectionError {
	Error
pub mut:
	path Path
	msg  string
	cat  CollectionErrorCat
}

struct CollectionNewArgs {
mut:
	name string @[required]
	path string @[required]
	heal bool = true // healing means we fix images, if selected will automatically load, remove stale links
	load bool = true
}

struct CollectionNotFound {
	Error
pub:
	pointer Pointer
	msg     string
}

fn (err CollectionNotFound) msg() string

struct File {
pub mut:
	collection   &Collection @[str: skip]
	name         string // received a name fix
	ext          string
	path         pathlib.Path
	pathrel      string
	state        FileStatus
	pages_linked []&Page // pointer to pages which use this file
	ftype        FileType
}

fn (file File) file_name() string

fn (mut file File) copy(dest string) !

struct MacroGetArgs {
pub mut:
	actor string
	name  string
}

struct NoOrTooManyObjFound {
	Error
pub:
	tree    &Tree
	pointer Pointer
	nr      int
}

the next is our custom error for objects not found

fn (err NoOrTooManyObjFound) msg() string

struct ObjNotFound {
	Error
pub:
	name       string
	collection string
	info       string
}

fn (err ObjNotFound) msg() string

struct Page {
	tree &Tree @[str: skip]
pub mut:
	name    string // received a name fix
	alias   string // a proper name for e.g. def
	path    pathlib.Path
	pathrel string // relative path in the collection
	state   PageStatus
	// pages_included  []&Page      @[str: skip]
	// pages_linked    []&Page      @[str: skip]
	// files_linked    []&File      @[str: skip]
	categories      []string
	readonly        bool
	changed         bool
	collection_name string
	doc_            ?&Doc @[str: skip]
}

fn (mut page Page) doc() !&Doc

fn (mut page Page) doc_process() !

reparse the markdown

fn (mut page Page) export(args_ PageExportArgs) !&Doc

save the page on the requested dest make sure the macro's are being executed

fn (page Page) key() string

struct PageExportArgs {
pub mut:
	dest     string @[required]
	replacer ?regext.ReplaceInstructions
}

fn (mut page Page) fix() ! { page.fix_links()! // TODO: do includes if page.changed { $if debug { console.print_debug('CHANGED: ${page.path}') } page.save()! page.changed = false } }

struct Pointer {
pub mut:
	collection string // is the key of a collection
	name       string // is name without extension, all namefixed (lowercase...)
	cat        PointerCat
	extension  string // e.g. jpg
	error      string // if there is an error on the pointer, then will be visible in this property
	tree       string
}

links to a page, image or file

fn (p Pointer) is_image() bool

if p.extension.len > 0 { out += '.${p.extension}' } return out }

fn (p Pointer) is_file_video_html() bool

struct PointerPath {
pub mut:
	pointer Pointer
	path    pathlib.Path
}

fn (p PointerPath) is_image() bool

fn (p PointerPath) is_file_video_html() bool

struct PointerPathArgs {
pub mut:
	path           string
	path_normalize bool = true // if this is set, it means it will change the path so the filename is clean
	needs_to_exist bool // if set will check the file exists
}

struct Tree {
pub:
	name string
pub mut:
	collections map[string]&Collection
	defs        map[string]&Page
	state       TreeState
	// context context.Context
	cid      string = '000'
	replacer ?regext.ReplaceInstructions
}

fn (tree Tree) collection_exists(name string) bool

fn (tree Tree) collection_get(name string) !Collection

fn (mut tree Tree) collection_new(args_ CollectionNewArgs) !&Collection

get a new collection

fn (tree Tree) collectionnames() []string

fn (mut tree Tree) export(args_ TreeExportArgs) !

export all collections to chosen directory . all names will be in name_fixed mode . all images in img/

fn (tree Tree) file_exists(name string) bool

exists or too many

fn (tree Tree) file_get(pointerstr string) !&File

get the file from pointer string: $tree:$collection:$name or $collection:$name or $name

fn (mut tree Tree) get_actions(args_ MacroGetArgs) ![]&Action

fn (tree Tree) image_exists(name string) bool

exists or too many

fn (tree Tree) image_get(pointerstr string) !&File

get the page from pointer string: $tree:$collection:$name or $collection:$name or $name

fn (tree Tree) key() string

the unique key to remember a tree . is unique per circle (based on cid)

fn (tree Tree) page_exists(name string) bool

exists or too many

fn (tree Tree) page_get(pointerstr string) !&Page

get the page from pointer string: $tree:$collection:$name or $collection:$name or $name

fn (mut tree Tree) process_defs() !

fn (mut tree Tree) process_includes() !

fn (mut tree Tree) process_macros() !

fn (mut tree Tree) scan(args_ TreeScannerArgs) !

walk over directory find dirs with .book or .collection inside and add to the tree . a path will not be added unless .collection is in the path of a collection dir or .book in a book

path string
heal bool // healing means we fix images, if selected will automatically load, remove stale links
git_url   string
git_reset bool
git_root  string
git_pull  bool
```	

struct TreeArgsGet {
pub mut:
	name string = 'default'
}

struct TreeExportArgs {
pub mut:
	dest           string @[required]
	reset          bool = true
	keep_structure bool // wether the structure of the src collection will be preserved or not
	exclude_errors bool // wether error reporting should be exported as well
	production     bool = true
	toreplace      string
}

struct TreeScannerArgs {
pub mut:
	path      string
	heal      bool = true // healing means we fix images
	git_url   string
	git_reset bool
	git_root  string
	git_pull  bool
	load      bool = true // means we scan automatically the added collection
}