Shared Library Skeletons

1. Overview

CODE
packy/stash/dysk

Our core languages are Rust and Lisp - this is the killer combo which will allow NAS-T to rapidly develop high-quality software. As such, it's crucial that these two very different languages (i.e. compilers) are able to interoperate seamlessly.

Some interop methods are easy to accomodate via the OS - such as IPC or data sharing, but others are a bit more difficult.

In this 2-part series we'll build a FFI bridge between Rust and Lisp, which is something that can be difficult, due to some complications with Rust and because this is not the most popular software stack (yet ;). This is an experiment and may not make it to our code-base, but it's definitely something worth adding to the toolbox in case we need it.

1.1. FFI

The level of interop we're after in this case is FFI.

Basically, calling Rust code from Lisp and vice-versa. There's an article about calling Rust from Common Lisp here which shows the basics and serves as a great starting point for those interested.

1.1.1. Rust ABI

The complication(s) with Rust I mentioned early is really just that it is not C. C is old, i.e. well-supported with a stable ABI, making the process of creating bindings for a C library a breeze in many languages.

For a Rust library we need to first appease the compiler, as explained in this section of the Rustonomicon. Among other things it involves changing the calling-convention of functions with a type signature and editing the Cargo.toml file to produce a C-compatible ABI binary. The Rust default ABI is unstable and can't reliably be used like the C ABI can.

abi_stable_crates is a project which addresses some of the ABI concerns, presenting a sort of ABI-API as a Rust library. Perhaps this is the direction the ecosystem will go with in order to maintain an unstable ABI, but for now there is no 'clear' pathway for a friction-less FFI development experience in Rust.

1.1.2. Overhead

Using FFI involves some overhead. Check here for an example benchmark across a few languages. While building the NAS-T core, I'm very much aware of this, and will need a few sanity benchmarks to make sure the cost doesn't outweigh the benefit. In particular, I'm concerned about crossing multiple language barriers (Rust<->C<->Lisp).

2. basic example

2.1. Setup

For starters, I'm going to assume we all have Rust (via rustup) and Lisp (sbcl only) installed on our GNU/Linux system (some tweaks needed for Darwin/Windows, not covered in this post).

2.1.1. Cargo

Create a new library crate. For this example we're focusing on a 'skeleton' for dynamic libraries only, so our experiment will be called dylib-skel or dysk for short. cargo init dysk --lib && cd dysk

A src/lib.rs will be generated for you. Go ahead and delete that. We're going to be making our own lib.rs file directly in the root directory (just to be cool).

The next step is to edit your Cargo.toml file. Add these lines after the [package] section and before [dependencies]:

[lib]
crate-type = ["cdylib","rlib"]
path = "lib.rs"
[[bin]]
name="dysk-test"
path="test.rs"

This tells Rust to generate a shared C-compatible object with a .so extension which we can open using dlopen.

2.1.2. cbindgen

  1. install

    Next, we want the cbindgen program which we'll use to generate header files for C/C++. This step isn't necessary at all, we just want it for further experimentation.

    cargo install --force cbindgen

    We append the cbindgen crate as a build dependency to our Cargo.toml like so:

    [build-dependencies]
    cbindgen = "0.24"
    
  2. cbindgen.toml
    language = "C"
    autogen_warning = "/* Warning, this file is autogenerated by cbindgen. Don't modify this manually. */"
    include_version = true
    namespace = "dysk"
    cpp_compat = true
    after_includes = "#define DYSK_VERSION \"0.1.0\""
    line_length = 88
    tab_width = 2
    documentation = true
    documentation_style = "c99"
    usize_is_size_t = true
    [cython]
    header = "dysk.h"
    
  3. build.rs
    fn main() -> Result<(), cbindgen::Error> {
      if let Ok(b) = cbindgen::generate(std::env::var("CARGO_MANIFEST_DIR").unwrap()) {
        b.write_to_file("dysk.h"); Ok(())}
      else { panic!("failed to generate dysk.h from cbindgen.toml") } }
    

2.2. lib.rs

//! lib.rs --- dysk library
use std::ffi::{c_char, c_int, CString};
#[no_mangle]
pub extern "C" fn hello() -> *const c_char {
  CString::new("hello from rust").unwrap().into_raw()}
#[no_mangle]
pub extern "C" fn plus(a:c_int,b:c_int) -> c_int {a+b}
#[no_mangle]
pub extern "C" fn plus1(n:c_int) -> c_int {n+1}

2.3. test.rs

//! test.rs --- dysk test
fn main() { let mut i = 0u32; while i < 500000000 {i+=1; dysk::plus1(2 as core::ffi::c_int);}}

2.4. compile

cargo build --release

2.5. load from SBCL

;;; dysk.lisp
;; (dysk:hello) ;; => "hello from rust"
(defpackage :dysk
  (:use :cl :sb-alien)
  (:export :hello :plus :plus1))
(in-package :dysk)
(load-shared-object #P"target/release/libdysk.so")
(define-alien-routine hello c-string)
(define-alien-routine plus int (a int) (b int))
(define-alien-routine plus1 int (n int))

2.6. benchmark

time target/release/dysk-test
(time (dotimes (_ 500000000) (dysk:plus1 2)))