Introduction

Project repository: github.com/PsichiX/Intuicio

crates-io version

What is Intuicio?

In short words: Intuicio is a set of building blocks designed to build your own scripting solution in Rust.

Every complete scripting solution built on Intuicio is split into:

Script

Scripts are all the information that defines a script in form of data.

It is an interface between frontends and backends to allow modularity and relatively universal way of communication between them.

Scripts are produced by frontends for backends to "run" them (more precisely to just use them, usually backends are used to execute scripts, but one can create something like a nativizer to transpile script into native code to run).

Scripts data is defined in intuicio-core crate, script module.

Frontend

Frontends are used to convert some data into scripts data.

Usually when we talk about scripting frontend, we are talking about compilers/transpilers for example - in general frontends can parse text code file for particular scripting language and turn it into Intuicio scripts that will be later used by backends. This doesn't mean we are forced to parsing text code files - one can create a node-graph-like scripting language and frontend for converting that into scripts, or even turn images into scripts, only limit is imagination, all Intuicio needs is just scripts data, and how we get them doesn't really matter.

There are examples of few frontends in Intuicio repositiory frontends folder, these are:

intuicio-frontend-assembler - simple assembler-like language that has the closest representation to script data one can get.
intuicio-frontend-serde - this one allows to represent scripts in any of the file formats that serde can support, for example JSON and YAML, or even LEXPR.
intuicio-frontend-vault - an attempt to create strongly typed language written with LEXPR syntax.
intuicio-frontend-simpleton - a full example of simple dynamically typed scripting language with rich standard library.

Backend

Backends are used to "run" scripts, or more specifically to use scripts produced by frontends in any way they want.

An obvious backend that anyone can think of is a Virtual Machine, already made as intuicio-backend-vm crate. This one grabs script data and executes it directly in VmScopes that are self-contained units that executes set of Intuicio script operations.

Another example of backend can be "nativizer" - nativizers are units that transpile scripts into native code, in this case Rust code. Nativizers are great way to speed up scripts execution, by removing the need for constructing and loading scripts at runtime, rather to just get native code that would do what scripts do, but without extra overhead. Although there is an intuicio-backend-rust crate that aims to do that, it is still incomplete, mostly non-functional until it gets proper definition, but experiments are being made and eventually Intuicio will have its own default nativizer.

Host

Host is basically an application-space (native side) of scripting logic, where user creates libraries of native functions and structs to bind into Registry, which later can be called or accessed by script operations or other native side code within given Context, shared between scripting and native side and treated equally as same.

The goal here is to allow for seamless interoperability between scripting and native sides of program logic, not forcing users to focus all their effort onto one particular side, something that is quite unique for scripting solutions, and this design decision was borrowed from Unreal Engine where it proven to be at least quite useful.

Scripting pipeline explained

Here we will attempt to explain entire scripting pipeline using examples of building blocks already made.

First things first

Now let's pick some building blocks. For backend we will use intuicio-backend-vm, and for frontend we will use intuicio-frontend-assembler (this may look quite unintuitive but it will allow us better explain how bindings work with scripts, so please bare with me for a little).

Create new Cargo project and add these dependencies:

[dependencies]
intuicio-data = "*"
intuicio-core = "*"
intuicio-derive = "*"
intuicio-backend-vm = "*"
intuicio-frontend-assembler = "*"

Once we have created new project, now in main.rs import these:

#![allow(unused)]
fn main() {
use intuicio_backend_vm::prelude::*;
use intuicio_core::prelude::*;
use intuicio_derive::*;
use intuicio_frontend_assembler::*;
}

Now let's define a simple native function that our script will call:

#![allow(unused)]
fn main() {
#[intuicio_function(module_name = "lib")]
fn add(a: i32, b: i32) -> i32 {
    a + b
}
}

We use procedural attribute macro that will generate and fill Function type with all the information about this function. We could also construct it by hand, or even with procedural macro like this to register its result into Registry:

#![allow(unused)]
fn main() {
registry.add_function(define_function! {
    registry => mod lib fn add(a: i32, b: i32) -> (result: i32) {
        (a + b,)
    }
});
}

But for the sake of simplicity, just stay with intuicio_function macro.

Then in main function first thing we should create is Registry to store all definitions of structures and functions, both native and script side ones, we also add basic types to not have to add them manually, and register defined function into that registry:

#![allow(unused)]
fn main() {
let mut registry = Registry::default().with_basic_types();
registry.add_function(add::define_function());
}

Like we have said before, registry should hold all the structures and functions information that scripts can interact with, and because Registry turns immutable once it gets into execution phase, better to do that from the get go - you won't be able to modify registry later.

Next step is to load, parse, compile and install assembly script into registry:

#![allow(unused)]
fn main() {
let mut content_provider = FileContentProvider::new("iasm", AsmContentParser);
AsmPackage::new("./resources/main.iasm", &mut content_provider)
    .unwrap()
    .compile()
    .install::<VmScope<AsmExpression>>(&mut registry, None);
}

FileContentProvider allows to pull script content from file system, AsmContentParser parses assembly script content into AsmPackage (intermediate representation of assembly scripts), VmScope is a container for script operations compiled from AsmPackage and finally AsmExpression is a set of custom operations that assembly scripting language performs. Expressions are an essential part of Intuicio scripting, since bare bones script data has very limited, universal set of basic operations required to make scripts call functions and move data between stack and registers - expressions allow extending set of operations, in case of AsmExpression scripts can push literals into stack and drop and forget value from stack.

Last step is to construct Context and put it with Registry into Host that will allow calling any function from the registry:

#![allow(unused)]
fn main() {
let context = Context::new(
    // stack bytes capacity.
    10240,
    // registers bytes capacity.
    10240,
);
let mut host = Host::new(context, RegistryHandle::new(registry));
let (result,) = host
    .call_function::<(i32,), _>(
        // function name.
        "main",
        // module name.
        "test",
        // structure name if function belongs to one.
        None,
    )
    .unwrap()
    .run(());
assert_eq!(result, 42);
}

Now the only thing what's left is to create ./resources/main.iasm file and fill it with script code:

mod test {
    fn main() -> (result: struct i32) {
        literal 40 i32;
        literal 2 i32;
        call mod lib fn add;
    }
}

We can see all this script is doing is pushing data on stack and calling a function - this will help us explain what is happening much easier.

How data moves

Let's start with showing how our add function actually looks like to Host:

#![allow(unused)]
fn main() {
fn add(context: &mut Context, registry: &Registry) {
    let a = context.stack().pop::<i32>().unwrap();
    let b = context.stack().pop::<i32>().unwrap();
    let result = a + b;
    context.stack().push(result);
}
}

Now from the example above we can see that execution starts from application-space when we tell Host to find main function of test module in host's registry, once is found, host passes its Context and Registry to function generated from script.

This is how test::main script function would look like to Host when converted to Rust:

fn main(context: &mut Context, registry: &Registry) {
    context.stack().push(40_i32);
    context.stack().push(2_i32);
    registry
        .find_function(FunctionQuery {
            name: Some("add".into()),
            module_name: Some("lib".into()),
            ..Default::default()
        })
        .unwrap()
        .invoke(context, registry);
}

So both together simply reduce to:

fn main(context: &mut Context, registry: &Registry) {
    context.stack().push(40_i32);
    context.stack().push(2_i32);
    add(context, registry);
}

And this is basically exactly how and why Intuicio doesn't care about which side (script or native) calls what side - both script and native sides are calling each other the same way, and VmScope is just a container for script operations in the middle of interactions to make script side look the same as native side to host - this opens quite interesting opportunity, if expression type matches, to use different frontends together in one application. You can think of it as something like .NET platform for Rust.

Conclusion

To summarize, Intuicio is actually just an engine to move data from place to place via stack, it doesn't care what is the data, it doesn't specify limits on the types of data (other than data has to be owned), it also doesn't care what is the place this data moves to, all it does is moves the data and both script and native side tells it when to move what data - simple as that!

Anatomy of a Host

Host is just a handy container for Context and Registry that simplifies calling functions registered in Registry within given Context.

Registry

Setup

Registry contains all structures and functions that both scripting and native side expose for both sides to interact with.

Here is an example of how to register functions both from native and script sides:

#![allow(unused)]
fn main() {
#[intuicio_function(module_name = "lib")]
fn add(a: i32, b: i32) -> i32 {
    a + b
}

let mut registry = Registry::default().with_basic_types();
registry.add_function(add::define_function());
let mut content_provider = FileContentProvider::new("iasm", AsmContentParser);
AsmPackage::new("main.iasm", &mut content_provider)
    .unwrap()
    .compile()
    .install::<VmScope<AsmExpression>>(&mut registry, None);
}

Of course we don't actually require frontends to register script-side functions and structures, here is an example of how one could create raw scripted function, knowing the backend it's gonna use (VmScope here):

registry.add_function(
    Function::new(
        function_signature! {
            registry => mod test fn main() -> (result: i32)
        },
        VmScope::<AsmExpression>::generate_function_body(
            ScriptBuilder::<AsmExpression>::default()
                .literal(AsmExpression::Literal(AsmLiteral::I32(2)))
                .literal(AsmExpression::Literal(AsmLiteral::I32(40)))
                .call_function(FunctionQuery {
                    name: Some("add".into()),
                    module_name: Some("lib".into()),
                    ..Default::default()
                })
                .build(),
            &registry,
            None,
        )
        .unwrap()
        .0,
    ),
);

And as we can see above, we can use completely different backends for each function we want to register, therefore in principle, one application can run multiple backends, not forcing user to use only one, which is yet another deliberate design decision to allow interoperability between different backends.

Also remember that since Intuicio is all about moving data into and out of function calls, every function that we want to call, has to be always registered in the registry and registry cannot be modified when used in function calls!

Queries

If user wants to call function from registry, user has to find it first and to do that we use FunctionQuery that defines search parameters for registry to filter functions with:

#![allow(unused)]
fn main() {
let (result,) = registry.find_function(FunctionQuery {
    name: Some("add".into()),
    module_name: Some("lib".into()),
    ..Default::default()
})
.expect("`lib::add` function not found!")
.call::<(i32,), _>(&mut context, &registry, (40_i32, 2_i32), true);
assert_eq!(result, 42);
}

When we want to find and call registered function from within another native function, we should be able as long as native-side function has access to context and registry:

#![allow(unused)]
fn main() {
#[intuicio_function(module_name = "script", use_context, use_registry)]
fn script_add(context: &mut Context, registry: &Registry, a: i32, b: i32) -> i32 {
    context.stack().push(b);
    context.stack().push(a);
    registry.find_function(FunctionQuery {
        name: Some("add".into()),
        module_name: Some("lib".into()),
        ..Default::default()
    })
    .expect("`lib::add` function not found!")
    .invoke(context, registry);
    context.stack().pop::<i32>().expect("Expected to return `i32`!")
}
}

Btw. in snippet above we perform function invoke instead of a call, and notice order of pushing values into stack - by design native functions expect to pop their arguments from first to last argument, and push its result in reverse order to match later function calls proper argument pop order. For convienience it is advised to perform function calls instead of invokes, because function calls keep proper stack push and pop order on their own.

Context

Context is a container that holds:

Stack

Used to move data between function calls.

#![allow(unused)]
fn main() {
context.stack().push(42_i32);
assert_eq!(context.stack().pop::<i32>().unwrap(), 42);
}

Registers

Indexed data storage, the closest analogue for local function variables.

#![allow(unused)]
fn main() {
let (stack, registers) = context.stack_and_registers();
let index = registers.push_register().unwrap();
stack.push(42_i32);
let mut register = registers.access_register(index).unwrap();;
registers.pop_to_register(&mut register);
stack.push_from_register(&mut register);
assert_eq!(stack.pop::<i32>().unwrap(), 42);
}

Please remember that with registers, just like with stack, data can be only moved in and out of registers, registers operations does not copy/clone their data - this design choice was dictated by master rule of Intuicio: "data can only be moved", copy/clone is a special operation that given structure has to provide a dedicated function for it to push duplicated source data into stack, from which original data gets moved back to register and its clone stays on the stack - this is what for example simpleton frontend does when it has to copy Reference from local variable to stack for later use.

Heap

Used to store dynamically allocated data in case user wants to ensure that data lifetime to be bound to the context (die along with context death). To be honest, this is not widely used piece of context, since data stored in any of the other pieces of context has no requirement to come from context's heap, it works perfectly fine with data allocated purely on rust-side, although at some point there might be a scenario where having boxed data owned by context is beneficial, therefore it is exposed to the user.
```
#![allow(unused)]
fn main() {
let mut value = context.heap().alloc(0_i32);
*value.write().uwnrap() = 42;
assert_eq!(*value.read().unwrap(), 42);
}
```
It's worth noting that memory allocated by heap box gets automatically returned to the heap once heap box drops, so there is no explicit heap box deallocation.
Custom data

Now this is the interesting bit, it is basically a hash map of Box<Any + Send + Sync> objects that does not fit to any of the other context pieces. It is useful for storing any meta information. For example simpleton frontend stores there its HostProducer that is used to construct new Host for any spawned Jobs worker thread, so each worker thread can execute closures passed into it.
```
#![allow(unused)]
fn main() {
context.set_custom("foo", 42_i32);
assert_eq!(*context.custom::<i32>().unwrap(), 42);
}
```

Anatomy of a Script

Entire Script structure is split into package, modules, functions/strucs/enums and at it's core: operations.

Example:

Package:
- module: test
  - struct: Foo
    - field: a: bool
  - enum: Bar
    - variant: A
    - variant: B
      - field: a: bool
  - function: main
    - output: result: i32
    - operations:
      - literal: 40: i32
      - literal: 2: i32
      - call function: lib::add

Operations

Intuicio scripts by default have a very limited set of operations, because of being an universal interface between frontends and backends, these operations have to be the lowest denominator across most possible frontend and backend designs, therefore operations it haves are:

Expression

This is an additional set of possible operations defined by frontends. It accepts types that implement ScriptExpression that allows to evaluate additional operation using only context and registry. Common example of script expression is pushing literals onto stack - stack manipulation is not part of the script operations set, because Intuicio is completely generic over the data it moves, and specific frontends most likely do have very specific assumptions about the data they allow to use, so it's better to leave this kind of operations to be provided by them.

Example of custom script expression:

#![allow(unused)]
fn main() {
pub enum CustomExpression {
    Literal(i32),
    StackDrop,
}

impl ScriptExpression for CustomExpression {
    fn evaluate(&self, context: &mut Context, _: &Registry) {
        match self {
            Self::Literal(value) => {
                context.stack().push(*value);
            }
            Self::StackDrop => {
                context.stack().drop();
            }
        }
    }
}
}

Define register

Allocates space for new register of given type defined by TypeQuery. Remember to always define register before it gets used! Registers gets removed when scope where they were defined gets dropped.
Drop register

Drops data behind given register.
Push from register

Takes data from given register and pushes it onto stack.
Pop to register

Takes data from stack and moves it to given register.
Move register

Moves data between two registers.
Call function

Uses associated FunctionQuery to find and invoke function from registry.
Branch scope

Takes bool from stack and if it's true, it will execute Success operations set, otherwise Failure operations set if present.
Loop scope

Executes set of operations in an loop so that before start of iteration it takes bool from stack, then if it's true, set of operations gets executed. Important note here is to remember to always push bool value (telling if next iteration should be executed), before we exit child scope.
Push scope

Pushes new set of operations for execution, returning further execution to its parent only when child scope stops its execution.
Pop scope

Closest analogue can be return keyword, which forces scope to stop further operations execution.
Continue scope conditionally

Does what pop scope does, but first takes bool from the stack and if it is false, scope gets dropped.

Usually we leave producing script operations to frontends, but one can create them at any time by any means. One can even create them at runtime and call newly created function in place, without adding it to registry:

let function = Function::new(
    function_signature! {
        registry => mod test fn main() -> (result: i32)
    },
    VmScope::<AsmExpression>::generate_function_body(
        ScriptBuilder::<AsmExpression>::default()
            .literal(AsmExpression::Literal(AsmLiteral::I32(2)))
            .literal(AsmExpression::Literal(AsmLiteral::I32(40)))
            .call_function(FunctionQuery {
                name: Some("add".into()),
                module_name: Some("lib".into()),
                ..Default::default()
            })
            .build(),
        &registry,
        None,
    )
    .unwrap()
    .0
);

function.invoke(&mut context, &registry);
assert_eq!(context.stack().pop::<i32>().unwrap(), 42);

This example shows precisely how short is the path between scripts and backends - backends constructing script functions directly from script operations, and in case of VmScope every time we call function generated by it, we directly execute these operations.

Tutorial

In this tutorial we will be building entire custom and very simple scripting pipeline step by step, in order:

We have choosen this particular order because each next part uses things made in its previous steps.

This entire tutorial sits in /demos/custom/ project on repository, if you want to look at complete project.

Before we start tutorials, let's provide native-side library of functions we will use in the scripts right away, as library.rs file:

#![allow(unused)]
fn main() {
use intuicio_core::prelude::*;
use intuicio_derive::*;

#[intuicio_function(module_name = "lib")]
fn add(a: i32, b: i32) -> i32 {
    a + b
}

#[intuicio_function(module_name = "lib")]
fn sub(a: i32, b: i32) -> i32 {
    a - b
}

#[intuicio_function(module_name = "lib")]
fn mul(a: i32, b: i32) -> i32 {
    a * b
}

#[intuicio_function(module_name = "lib")]
fn div(a: i32, b: i32) -> i32 {
    a / b
}

pub fn install(registry: &mut Registry) {
    registry.add_function(add::define_function(registry));
    registry.add_function(sub::define_function(registry));
    registry.add_function(mul::define_function(registry));
    registry.add_function(div::define_function(registry));
}
}

So whenever you'll see in next tutorials this line, remember it calls install function from library.rs provided above:

#![allow(unused)]
fn main() {
crate::library::install(&mut registry);
}

Building custom frontend

This is gonna be basically a pretty much stripped down version of assembler frontend just to get intuition on creating frontends. For more advanced examples of frontends please take a look at frontends already made, located in frontends folder on repository.

Let's start with defining goals for this frontend to achieve:

scripts operate only on i32 values.
scripts will have two operations:
- push value on stack.
- call functions that takes values from stack, performs operations on them and push results back on stack.
syntax of this language has to be simple, so that:
- each line is an operation or comment.
- we put operations in reverse order so it is easier to read script as a hierarchy of function calls with its arguments indented and in ascending order.

Frontend syntax:

call lib div
    call lib mul
        push 3
        call lib sub
            call lib add
                push 40
                push 2
            push 10
    push 2

So now let's create new project and add Intuicio dependencies:

[dependencies]
intuicio-data = "*"
intuicio-core = "*"
intuicio-derive = "*"
intuicio-backend-vm = "*"

Then create frontend.rs file, where we will heep all frontend-related code, and first import these dependencies:

#![allow(unused)]
fn main() {
use intuicio_core::prelude::*;
use std::{error::Error, str::FromStr};
}

intuicio_core holds types related to script information, we use Error trait for errors propagation and FromStr for parsing.

The most important thing to make is custom Intuiocio expression:

#![allow(unused)]
fn main() {
#[derive(Debug)]
pub enum CustomExpression {
    Literal(i32),
}

impl ScriptExpression for CustomExpression {
    fn evaluate(&self, context: &mut Context, _: &Registry) {
        match self {
            Self::Literal(value) => {
                context.stack().push(*value);
            }
        }
    }
}
}

Expressions allow to extend available operations set of Intuicio scripts to enable features specific to given frontend - here we just allow to push i32 literals onto stack.

Now we will define intermediate script types for our Custom scripting language:

#![allow(unused)]
fn main() {
pub type CustomScript = Vec<CustomOperation>;

pub enum CustomOperation {
    Comment { content: String },
    Push { value: i32 },
    Call { name: String, module_name: String },
}
}

Next we need to implement parsing of operations from string lines to intermediate script data:

#![allow(unused)]
fn main() {
impl FromStr for CustomOperation {
    type Err = CustomOperationError;

    fn from_str(line: &str) -> Result<Self, Self::Err> {
        let line = line.trim();
        if line.is_empty() {
            return Ok(Self::Comment {
                content: "".to_owned(),
            });
        }
        if line.starts_with("#") {
            return Ok(Self::Comment {
                content: line.to_owned(),
            });
        }
        let mut tokens = line.split_ascii_whitespace();
        match tokens.next() {
            Some("push") => {
                let value = tokens.next().unwrap().parse::<i32>().unwrap();
                Ok(Self::Push { value })
            }
            Some("call") => {
                let module_name = tokens.next().unwrap().to_owned();
                let name = tokens.next().unwrap().to_owned();
                Ok(Self::Call { name, module_name })
            }
            _ => Err(CustomOperationError {
                operation: line.to_owned(),
            }),
        }
    }
}
}

Also don't forget to implement our parsing error type:

#![allow(unused)]
fn main() {
#[derive(Debug)]
pub struct CustomOperationError {
    pub operation: String,
}

impl std::fmt::Display for CustomOperationError {
    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
        write!(f, "Unsupported operation: `{}`", self.operation)
    }
}

impl Error for CustomOperationError {}
}

After that we need to implement script compilation from intermediate to Intuicio scripts data, so scripts will be understood by Intuicio backend:

#![allow(unused)]
fn main() {
impl CustomOperation {
    pub fn compile_operation(&self) -> Option<ScriptOperation<'static, CustomExpression>> {
        match self {
            Self::Comment { .. } => None,
            Self::Push { value } => Some(ScriptOperation::Expression {
                expression: CustomExpression::Literal(*value),
            }),
            Self::Call { name, module_name } => Some(ScriptOperation::CallFunction {
                query: FunctionQuery {
                    name: Some(name.to_owned().into()),
                    module_name: Some(module_name.to_owned().into()),
                    ..Default::default()
                },
            }),
        }
    }

    pub fn compile_script(
        operations: &[CustomOperation],
    ) -> ScriptHandle<'static, CustomExpression> {
        operations
            .iter()
            .rev()
            .filter_map(|operation| operation.compile_operation())
            .collect::<Vec<_>>()
            .into()
    }
}
}

In compile_script method we iterate over operations in reverse order, because human-readable side of the scripts expects function call and then its parameters, while Intuicio scripts expect computer-readable order of arguments first, then function call.

Finally we create scripts content parser so it can be used to parse byte strings into intermediate type scripts:

#![allow(unused)]
fn main() {
pub struct CustomContentParser;

impl BytesContentParser<CustomScript> for CustomContentParser {
    fn parse(&self, bytes: Vec<u8>) -> Result<CustomScript, Box<dyn Error>> {
        Ok(String::from_utf8(bytes)?
            .lines()
            .filter(|line| !line.is_empty())
            .map(|line| CustomOperation::from_str(line))
            .collect::<Result<CustomScript, _>>()?)
    }
}
}

Now let's create main.rs file, where we will test this frontend, first import dependencies:

mod frontend;
mod library;

use crate::frontend::*;
use intuicio_backend_vm::prelude::*;
use intuicio_core::prelude::*;

fn main() {
    // next steps go here.
}

Then parse and compile some script:

#![allow(unused)]
fn main() {
let script = b"
call lib div
    call lib mul
        push 3
        call lib sub
            call lib add
                push 40
                push 2
            push 10
    push 2
";
let script = CustomContentParser.parse(script.to_vec()).unwrap();
let script = CustomOperation::compile_script(&script);
}

Next, create and setup registry:

let mut registry = Registry::default().with_basic_types();
crate::library::install(&mut registry);
registry.add_function(Function::new(
    function_signature! {
        registry => mod main fn main() -> (result: i32)
    },
    VmScope::<CustomExpression>::generate_function_body(script, None)
        .unwrap()
        .0,
));

As you can see, our scripts do not define functions, rather operations that belong to single one, so we create new main function and add it to the registry. We also use VmScope from VM backend to test this frontend in already existing VM backend, until we create dedicated backend ourselves.

Final thing to do is to create host and test frontend:

#![allow(unused)]
fn main() {
let mut host = Host::new(Context::new(10240, 10240), registry.into());
let (result,) = host
    .call_function::<(i32,), _>("main", "main", None)
    .unwrap()
    .run(());
assert_eq!(result, 48);
}

Building custom backend

Goal for our custom backend is to simplify Virtual Machine by limiting its execution to only expressions and function calls, everything else will be treated with errors when used.

You might ask:

Ok, but our frontend already works on official VM backend, why creating custom one?

And you're completely right, there is no need for custom backend, although showcasing how one could create custom backend might help gain understanding and spark some ideas or even make someone improve if not create completely new backend that's gonna execute scripts much faster than what official VM backend offers!

So yeah, this part of tutorial is educational only, there is no need to create custom backends for custom frontends.

Now let's start with creating backend.rs as part of our existing project and import dependencies, and create custom VM scope type:

#![allow(unused)]
fn main() {
use intuicio_core::prelude::*;

pub struct CustomScope<'a, SE: ScriptExpression> {
    handle: ScriptHandle<'a, SE>,
    position: usize,
}

impl<'a, SE: ScriptExpression> CustomScope<'a, SE> {
    pub fn new(handle: ScriptHandle<'a, SE>) -> Self {
        Self {
            handle,
            position: 0,
        }
    }
}
}

As you can see, it is generic over the expression type of scripts so this backend could be used by any frontend, limiting scripts execution to expressions and function calls only.

Next we implement simple VM execution of entire script operations set:

#![allow(unused)]
fn main() {
impl<'a, SE: ScriptExpression> CustomScope<'a, SE> {
    pub fn run(&mut self, context: &mut Context, registry: &Registry) {
        while let Some(operation) = self.handle.get(self.position) {
            match operation {
                ScriptOperation::None => {
                    self.position += 1;
                }
                ScriptOperation::Expression { expression } => {
                    expression.evaluate(context, registry);
                    self.position += 1;
                }
                ScriptOperation::CallFunction { query } => {
                    let handle = registry
                        .functions()
                        .find(|handle| query.is_valid(handle.signature()))
                        .unwrap_or_else(|| {
                            panic!("Could not call non-existent function: {:#?}", query)
                        });
                    handle.invoke(context, registry);
                    self.position += 1;
                }
                _ => unreachable!("Trying to perform unsupported operation!"),
            }
        }
    }
}
}

And last thing for this file is to implement ScriptFunctionGenerator for custom VM scope, so it will take any script and turn it into function body that will be provided later to function definitions:

#![allow(unused)]
fn main() {
impl<SE: ScriptExpression + 'static> ScriptFunctionGenerator<SE> for CustomScope<'static, SE> {
    type Input = ();
    type Output = ();

    fn generate_function_body(
        script: ScriptHandle<'static, SE>,
        ignore: Self::Input,
    ) -> Option<(FunctionBody, Self::Output)> {
        Some((
            FunctionBody::closure(move |context, registry| {
                Self::new(script.clone()).run(context, registry);
            }),
            ignore,
        ))
    }
}
}

Finally we need to change our main.rs file slightly to use custom VM scope instead of official VM scope.

First we need to make new dependency imports in place of old ones:

#![allow(unused)]
fn main() {
mod backend;
mod frontend;
mod library;

use crate::backend::*;
use crate::frontend::*;
use intuicio_core::prelude::*;
}

We can also remove intuicio-backend-vm dependency from Cargo.toml since it won't be used anymore.

Next the only thing we change in main function is we just replace our main scripting function definition into:

registry.add_function(Function::new(
    function_signature! {
        registry => mod main fn main() -> (result: i32)
    },
    CustomScope::<CustomExpression>::generate_function_body(script, ())
        .unwrap()
        .0,
));

So it will generate function body that runs custom VM scope with previously compiled script.

Building custom runner

In this part of tutorial we will be creating REPL solution that will prompts users to type operations in each next line and issue execution of collected operations as single script.

First clear entire main.rs file, and start with importing new set of dependencies:

#![allow(unused)]
fn main() {
mod backend;
mod frontend;
mod library;

use crate::backend::*;
use crate::frontend::*;
use intuicio_core::prelude::*;
use std::str::FromStr;
}

Then we create our REPL structure that will hold both collected script operations and host that will run provided scripts:

#![allow(unused)]
fn main() {
struct Repl {
    script: CustomScript,
    host: Host,
}

impl Default for Repl {
    fn default() -> Self {
        let mut registry = Registry::default().with_basic_types();
        crate::library::install(&mut registry);
        let context = Context::new(10240, 10240);
        Self {
            script: Default::default(),
            host: Host::new(context, registry.into()),
        }
    }
}
}

Next we need to implement feeding lines functionality, that will ask user for next operation, parse, compile and either collect into script or execute collected script if user types empty line:

impl Repl {
    fn feed_line(&mut self) {
        let mut line = String::default();
        if let Err(error) = std::io::stdin().read_line(&mut line) {
            println!("* Could not read line: {}", error);
        }
        if line.trim().is_empty() {
            let (context, registry) = self.host.context_and_registry();
            let script = CustomOperation::compile_script(&self.script);
            let body = match CustomScope::<CustomExpression>::generate_function_body(script, ()) {
                Some(body) => body.0,
                None => {
                    println!("Could not generate custom function body!");
                    return;
                }
            };
            let function = Function::new(
                function_signature! {
                    registry => mod main fn main() -> (result: i32)
                },
                body,
            );
            function.invoke(context, registry);
            if let Some(value) = context.stack().pop::<i32>() {
                println!("* Completed with result: {}", value);
            } else {
                println!("* Completed!");
            }
            self.script.clear();
        } else {
            match CustomOperation::from_str(&line) {
                Ok(operation) => self.script.push(operation),
                Err(error) => println!("* Could not parse operation: {}", error),
            }
        }
    }
}

Finally here comes main function that runs REPL in a loop:

fn main() {
    let mut repl = Repl::default();
    println!("Custom REPL.\nPlease feed operation per line or type empty line to execute:");
    loop {
        repl.feed_line();
    }
}

Let's test it:

$ cargo run
Custom REPL.
Please feed operation per line or type empty line to execute:
call lib add
push 40
push 2

* Completed with result: 42

And with that our simple tutorial completes - i hope you have learned something new today and i hope it gave you some intuition on how to create entire scripting solution with Intuicio!

Official frontends

Intuicio platform provides set of official frontends both as languages dedicated to specific domains and serving as an example of how more complex frontends are being designed:

Assembler

Low level text-based language that is the closest one to Intuicio script information data format. Useful for in-game operation systems.

Serde

Low level data-based language to supports scripts that can be defined in any format supported by serde (for example: json, yaml, toml, lexpr and more). Useful as intermediate stage between editors and scripting.

Vault

An attempt to high level text-based strongly typed scripting language with lexpr syntax.

Simpleton

High level text-based scripting language.

Key features:

Dynamically typed.
Influenced by JavaScript and Python.
Simple to understand and learn.
Rich standard library.

Useful for scripting tools and game logic, where simplicity and ergonomics is vastly more important than performance.

Goals

Simpleton was created for simplicity in mind as priority, mostly for allowing to create tools and simple game logic the easiest way possible.

It also has an educational value to showcase Intuicio users how one can create fully fledged dynamically typed scripting language - something that might be counter intuitive since Intuicio is rather strongly typed scripting platform. We achieve that by introducing Reference, Type and Function as fundamental Simpleton types, so that entire frontend is built around interactions with these types.

Syntax

mod main {
    func main(args) {
        console::log_line("Hello World!");
        
        var fib = main::fib(10);
        console::log_line(debug::debug(fib, false));
    }

    func fib(n) {
        if math::less_than(n, 2) {
            return n;
        } else {
            return math::add(
                main::fib(math::sub(n, 1)),
                main::fib(math::sub(n, 2)),
            );
        }
    }
}

Language reference

Let's start with saying that Simpleton is not an OOP language, you won't also see any class methods, inheritance or dynamic method dispatch seen in other OOP scripting languages.

Simpleton is rather functional/procedural scripting language that operates on structures for data transformations.

Script file structure

Every script file consists of module definition and module items such as structures and functions:

mod vec2 {
    struct Vec2 { x, y }

    func new(x, y) {
        return vec2::Vec2 { x, y };
    }

    func add(a, b) {
        return vec2::Vec2 {
            x: math::add(a.x, b.x),
            y: math::add(a.y, b.y),
        };
    }
}

General rule of thumb is that one file describes one module and preferably one structure (if any) so functions in that module are working in context of that structure (somewhat similarly to how GDscript treats its files to some extent).

So later these module types are operated on like this:

mod main {
    import "vec2";

    func main() {
        var a = vec2::new(0.0, 1.0);
        var b = vec2::new(1.0, 0.0);
        var c = vec2::add(a, b);
        // vec2::Vec2 { x: 1.0, y: 1.0 }
        console::log_line(debug::debug(c, false));
    }
}

First question that comes into mind is:

Why there are add function calls instead of operators being used?

Simpleton does not have operators - Simpleton believes in being explicit. If something is doing what function calls does, it should be called as function.

The only exceptions are array and map accessors:

var array = [0, 1, 2];
var array_item = array[1];

var map = { a: 0, b: 1, c: 2 };
var map_item = map{"b"};

Structures

Every structure is defined by its name and list of field names:

struct Vec2 { x, y }

The reason for that is that structures defined in simpleton are concrete objects that take up space defined by number of references they hold, they aren't dynamically sized bags of properties like in many scripting languages.

If we have an object of vec2::Vec2 type and we want to access its x field for reads or writes, it is guaranteed this field exists in the object and we get or set its reference.

And the opposite is true too - if we try to access object field that is not defined in its type, we will get runtime error.

Since Simpleton is not an OOP language, we not only do not have class methods, but also we do not have constructors, therefore we construct objects in-place like this:

var v = vec2::Vec2 { x: 42.0 };

What this does is Simpleton creates default object of type vec2::Vec2 and then applies values to fields listed in brackets - this also means that if we omit some fields, object will have their references nulled.

Therefore if object expects some specific constraints on the object fields, it's good practice to make functions that return new object in-place with fields filled with arguments validated by that function:

func new(x, y) {
    debug::assert(
        math::equals(reflect::type_of(x), <struct math::Real>),
        "`x` is not a number!",
    );
    debug::assert(
        math::equals(reflect::type_of(y), <struct math::Real>),
        "`y` is not a number!",
    );
    return vec2::Vec2 { x, y };
}

Additionally in rare situations when we do not know object type at compile-time, we can construct objects by Type object found at runtime:

var v = reflect::new(<struct vec2::Vec2>, { x: 42 });

This is useful especially in case of deserialization, where type is part of deserialized data:

var data = {
    type_name: "Vec2",
    type_module_name: "vec2",
    properties: {
        x: 42.0,
    },
};
var type = reflect::find_type_by_name(data.type_name, data.type_module_name);
var v = reflect::new(type, data.properties);

And finally to get or set value from object field we use . delimiter between object and its field name:

var v = vec2::Vec2 { x: 42 };
v.x = math::add(v.x, 10);

Functions

Every function is defined by its name, arguments list and function statements as its body:

func sum(a, b, c) {
    console::log_line(
        text::format("sum: {0}, {1}, {2}", [
            reflect::to_text(a),
            reflect::to_text(b),
            reflect::to_text(c),
        ]),
    );

    return math::add(
        math::add(a, b),
        c,
    );
}

As you can see there is return keyword - it is used to inform Simpleton that we want to exit current function with given value.

Functions always return some reference, if function does not have return statement in there, it implicitly returns null. We can also return null; if we want to exit function without value.

Later functions can be called by providing their module name, function name and arguments:

var v = main::sum(1, 2, 3);

If we don't know function at compile-time, we can call it at runtime by Function object:

var v = reflect::call(<func main::sum>, [1, 2, 3]);

We can also find function type by its name and module name:

var function = reflect::find_function_by_name("sum", "main");
var v = reflect::call(function, [1, 2, 3]);

Closures

Closures are special anonymous functions that can also capture variables from outer scope:

var a = 40;
var closure = @[a](b) {
    return math::add(a, b);
};
var v = closure::call(closure, [b]);

Under the hood, closures are objects that store reference to Function object and list of captured references, and are compiled into actual functions like this:

mod _closures {
    func _0(a, b) {
        return math::add(a, b);
    }
}

Primitive types

null - reference that points to nothing.
Boolean - can hold either true or false.
Integer - their literals are numbers without rational part: 42. Additionally there are hex (#A8) and binary ($1011) integer literals.
Real - their literals are numbers with rational part: 4.2.
Text - also known in other languages as strings of UTF8 characters: "Hello World!".
Array - sequence of value references: [0, 1, 2]. We can access its items with: array[0].
Map - unordered table of key-value pairs: { a: 0, b: 1, c: 2 }. We can access its items with: map{"a"}.

It's worth noting that all objects, even Boolean, Integer and Real are boxed object - choice made for the sake of simplicity of language implementation, but in the future they might endup unboxed to improve performance - for now we don't mind them being boxed, at this point in development, performance is not the priority.

Variables

Variables are local to the function, they are name aliases for references to values. You define variable with its value like this:

var answer = 42;

You can also assign new values to existing variables:

answer = 10;

Since variables are just named references local to given function, you can assign different type values to them than what was stored there at creation:

var v = 42;
v = "Hello World!";

To get value behind variable you just use its name:

var a = 42;
console::log_line(reflect::to_text(a));

If-else branching

Branching allows to execute some scope if condition succeeds:

if math::equals(a, 42) {
    console::log_line("`a` equals 42!");
}

One can also specify else scope in case condition fails:

if math::equals(a, 42) {
    console::log_line("`a` equals 42!");
} else {
    console::log_line("`a` does not equals 42!");
}

While loops

While loops allows to execute some scope repeatedly as long as condition succeeds:

var a = 0;
while math::less_than(a, 42) {
    a = math::add(a, 1);
}

For loops

For loops combined with iterators allow to iterate on values yielded by them until null value is yielded, which signals end of iteration:

// 2, 3, 4
for value in iter::walk(2, 3) {
    console::log_line(debug::debug(value, false));
}

// 0, 1, 2
for value in iter::range(0, 3) {
    console::log_line(debug::debug(value, false));
}

var v = [0, 1, 2];
// 2, 1, 0
for value in array::iter(v, true) {
    console::log_line(debug::debug(value, false));
}

var v = { a: 0, b: 1, c: 2};
// { key: "a", value: 1 }, { key: "b", value: 2 }, { key: "c", value: 3 }
for pair in map::iter(v) {
    console::log_line(debug::debug(pair, false));
}

var iter = iter::build([
    iter::range(0, 10),
    [<func iter::filter>, @[](value) {
        return math::equals(
            math::modulo(value, 2),
            0,
        );
    }],
]);
// 0, 2, 4, 6, 8
for pair in iter {
    console::log_line(debug::debug(pair, false));
}

Imports

Packages are built by traversing modules tree - this tree is defined by entry module and its dependencies specified by imports:

mod main {
    import "vec2.simp";

    func main() {
        var a = vec2::Vec2 { x: 1, y: 0 };
        var b = vec2::Vec2 { x: 0, y: 1 };
        // vec2::Vec2 { x: 1, y: 1 }
        var c = vec2::add(a, b);
    }
}

mod vec2 {
    struct Vec2 { x, y }

    func add(a, b) {
        return vec2::Vec2 {
            x: math::add(a.x, b.x),
            y: math::add(a.y, b.y),
        };
    }
}

We provide relative path to another module file with import keyword.

It's worth noting that file extensions can e omited there, compiler will assume simp extension then.

Intuicio - Modular scripting platform for Rust