Token and AST Guide

A token consists of a token type, a token value, and the index in the source code that it was taken from.

A token looks like this:

["IDENTIFIER", "this_is_my_identifier", 27]
 ^             ^                        ^
 token type    token value              character index

The parser uses these types and values to create a JSON-like syntax tree.

A syntax tree consists of expressions contaning nodes.

A syntax tree looks like this:

{
	"Expression @Idx[0]": {
		"Binary Operation +": [
			{"Numerical Constant" : 12}, 
			{"Numerical Constant" : 5}
		]
	}
}

Which can be generated by code that looks like this:

12*5

Since the code eventually boils down into a syntax tree before becoming assembly, it is possible to actually write code in a JSON format. This will be available with the addition of astimport and astinsert in the future.

astimport <mylib.json>
{
	"Expression1" as "ast_expression",
}

int var foo = astinsert {"ast_expression"}

NOTE: astimport and astinsert are resolved statically at compile time, so user input would not be able to modify their behavior. All astinsert does is replace the current working node in the parser with the provided node.

Go Back