1. DRAFT dylib-skel-1

  • State "DRAFT" from [2023-11-05 Sun 22:23]

1.1. Overview

Our core languages are Rust and Lisp - this is the killer combo which will allow NAS-T to rapidly develop high-quality software. As such, it's crucial that these two very different languages (i.e. compilers) are able to interoperate seamlessly.

Some interop methods are easy to accomodate via the OS - such as IPC or data sharing, but others are a bit more difficult.

In this 2-part series we'll build a FFI bridge between Rust and Lisp, which is something that can be difficult, due to some complications with Rust and because this is not the most popular software stack (yet ;). This is an experiment and may not make it to our code-base, but it's definitely something worth adding to the toolbox in case we need it.

1.2. FFI

The level of interop we're after in this case is FFI.

Basically, calling Rust code from Lisp and vice-versa. There's an article about calling Rust from Common Lisp here which shows the basics and serves as a great starting point for those interested.

1.2.1. Rust != C

The complication(s) with Rust I mentioned early is really just that it is not C. C is old, i.e. well-supported with a stable ABI, making the process of creating bindings for a C library a breeze in many languages.

For a Rust library we need to first appease the compiler, as explained in this section of the Rustonomicon. Among other things it involves changing the calling-convention of functions with a type signature and editing the Cargo.toml file to produce a C-compatible ABI binary. The Rust default ABI is unstable and can't reliably be used like the C ABI can.

1.2.2. Overhead

Using FFI involves some overhead. Check here for an example benchmark across a few languages. While building the NAS-T core, I'm very much aware of this, and will need a few sanity benchmarks to make sure the cost doesn't outweigh the benefit. In particular, I'm concerned about crossing multiple language barriers (Rust<->C<->Lisp).

1.3. Rust -> C -> Lisp

1.3.1. Setup

For starters, I'm going to assume we all have Rust (via rustup) and Lisp (sbcl only) installed on our GNU/Linux system (some tweaks needed for Darwin/Windows, not covered in this post).

  1. Cargo

    Create a new library crate. For this example we're focusing on a 'skeleton' for dynamic libraries only, so our experiment will be called dylib-skel or dysk for short. cargo init dysk --lib && cd dysk

    A src/lib.rs will be generated for you. Go ahead and delete that. We're going to be making our own lib.rs file directly in the root directory (just to be cool).

    The next step is to edit your Cargo.toml file. Add these lines after the [package] section and before [dependencies]:

    [lib]
    crate-type = ["cdylib","rlib"]
    path = "lib.rs"
    [[bin]]
    name="dysk-test"
    path="test.rs"
    

    This tells Rust to generate a shared C-compatible object with a .so extension which we can open using dlopen.

  2. cbindgen
    1. install

      Next, we want the cbindgen program which we'll use to generate header files for C/C++. This step isn't necessary at all, we just want it for further experimentation.

      cargo install --force cbindgen

      We append the cbindgen crate as a build dependency to our Cargo.toml like so:

      [build-dependencies]
      cbindgen = "0.24"
      
    2. cbindgen.toml
      language = "C"
      autogen_warning = "/* Warning, this file is autogenerated by cbindgen. Don't modify this manually. */"
      include_version = true
      namespace = "dysk"
      cpp_compat = true
      after_includes = "#define DYSK_VERSION \"0.1.0\""
      line_length = 88
      tab_width = 2
      documentation = true
      documentation_style = "c99"
      usize_is_size_t = true
      [cython]
      header = '"dysk.h"'
      
    3. build.rs
      fn main() -> Result<(), cbindgen::Error> {
        if let Ok(b) = cbindgen::generate(std::env::var("CARGO_MANIFEST_DIR").unwrap()) {
          b.write_to_file("dysk.h"); Ok(())}
        else { panic!("failed to generate dysk.h from cbindgen.toml") } }
      

1.3.2. lib.rs

//! lib.rs --- dysk library
use std::ffi::{c_char, c_int, CString};
#[no_mangle]
pub extern "C" fn dysk_hello() -> *const c_char {
  CString::new("hello from rust").unwrap().into_raw()}
#[no_mangle]
pub extern "C" fn dysk_plus(a:c_int,b:c_int) -> c_int {a+b}
#[no_mangle]
pub extern "C" fn dysk_plus1(n:c_int) -> c_int {n+1}

1.3.3. test.rs

//! test.rs --- dysk test
fn main() { let mut i = 0u32; while i < 500000000 {i+=1; dysk::dysk_plus1(2 as core::ffi::c_int);}}

1.3.4. compile

cargo build --release

1.3.5. load from SBCL

(load-shared-object #P"target/release/libdysk.so")
(define-alien-routine dysk-hello c-string)
(define-alien-routine dysk-plus int (a int) (b int))
(define-alien-routine dysk-plus1 int (n int))
(dysk-hello) ;; => "hello from rust"

1.3.6. benchmark

time target/release/dysk-test
(time (dotimes (_ 500000000) (dysk-plus1 2)))