QRust (3) Framework

This article is machine translated

Linking Rust as a dynamic or static library to the Qt environment is already a complex task, and introducing QRust on top of it is even more difficult, so in this chapter, I will guide you step by step forward and across the holes in the ground.

Programming Environment

Qt environment: Qt6, yes, Qt5 is not supported. Because I found that the type derivation of struct is wrong in Qt5 environment.

Rust environment: There is no theoretical limit, but there is a problem with Qt linking Rust static libraries when the link symbol cannot be found in windows environments, this problem occurs on higher versions of Rust, so my Rust version has remained at 1.65.0-nightly, afraid to upgrade, and I have not found the problem in MacOS and Linux environments.

The following is a list of the environments I have tested, specifically stating that not being in the table does not mean that it is not supported, but that I personally do not have the ability to conduct more tests.

The installation and related configuration of Qt6 in the above environment are slightly more complex, especially the community version of Qt6, each installation may encounter different problems, patience and more attempts are the key to success.

Build project

There are two projects in the QRust source code:

• The directory ‘rust’ is a Rust-side project that allows you to build directly.
• The directory ‘qt-window ‘ is the Qt side project, and in order to better demonstrate asynchronous calls to QRust, this project is a window program, not a console program.

These two projects together build the demo program that demonstrates Qrust, and it is highly recommended that your first implementation be built on top of both projects.

Rust side project

The serde_qrust directory is a subcrate contained in the current crate and is the serialization and deserialization implementation of serde. If you are not interested in the serde implementation, you can skip the code here and focus on the current crate, starting with the project configuration file Cargo.toml:

……
[dependencies]
libc = "*"

log = "0.4"
log4rs = "0.10.0"

serde = {version = "1.0", features = ["derive"]}
serde_qrust = {path = "./serde_qrust"}

[lib]
crate-type = ["staticlib"]
name = "qrust"
[profile.release]
panic = "abort"
[profile.dev]
panic = "abort"

log4rs can be replaced with your preferred log library, please focus on the configuration under [lib] :

• crate-type = [“staticlib”] This statement indicates that the ultimate goal of compilation is to statically libraries
• name = “qrust” The output path is in the target directory. The library extension may vary on different systems. In windows, the final output file is qrust.lib

There are two files in the src directory, lib.rs and api.rs

api.rs is completely decouples from QRust, which is the implementation of the demo function and the definition of the struct passed, there is nothing special, when you run the program, combined with Qt side calls, it is easy to understand every line of code in the file. Here, for the sake of convenience, the functions in api.rs are called “business functions”, and the business functions are the parts that you need to implement according to your own requirements, focusing on lib.rs, which is closely related to QRust.

fn invoke(fun_name: &str, mut args: Vec<&[u8]>) -> Result<Option<Vec<u8>>>
{
    match fun_name
    {
        "foo" =>    //Example of a function call with no arguments and no returns
        {
            api::foo();     //Calling function
            Ok(None)
        } 
        "foo1" =>   //Example of a function call with one argument and no return
        {
            let a1 = de::from_pack(args.pop().unwrap())?; //Deserialization gets the parameters
            api::foo1(a1);  
            Ok(None)
        }
        "foo2" =>   //There are multiple arguments and one return
        {
            let a1 = de::from_pack(args.pop().unwrap())?;
            let a2 = de::from_pack(args.pop().unwrap())?;
            let a3 = de::from_pack(args.pop().unwrap())?;

            let ret = api::foo2(a1, a2, a3); 
            let pack = ser::to_pack(&ret)?;     //Return value packaging (serialization)
            Ok(Some(pack))
        }
        "foo_struct" => //Example of a custom struct type
        {
            let arg = de::from_pack(args.pop().unwrap())?;
            let count = de::from_pack(args.pop().unwrap())?;

            let ret = api::foo_struct(arg, count);
            let pack = ser::to_pack(&ret)?; 
            Ok(Some(pack))
        }
       ……
}

The invoke function, which I call a “manual runtime reflection function,” does three things here:

• match the corresponding business function according to the name string of the representative function of the Qt request
• Before executing a business function, the function’s arguments are parsed through from_pack() (deserialization)
• After executing the business function, pack the return value via to_pack() (serialization)

For some languages that support runtime reflection, such as Java, matching and calling functions can be fully automated, and the api.foo() function can be automatically called in a reflective manner upon receiving an “api.foo()” string. Rust doesn’t have this capability, so you need to manually code match for each business function here.

If the number of business functions is too large, the match branch is too long, or the business function needs to be processed by modules, and it is not suitable to write in a match uniformly. In OnTHeSSH, I deal with it like this:

pub fn invoke(fun_name: &str, args: Vec<&[u8]>) -> Result<Option<Vec<u8>>>
{
    if fun_name.starts_with("rhost::")              //Remote host
    {
        crate::rhost::api_host::invoke_host(fun_name, args)
    }
    else if fun_name.starts_with("group::")        //Remote host group
    {
        crate::rhost::api_group::invoke_group(fun_name, args)
    }
    else if fun_name.starts_with("workhosts::")     //working multi remote hosts
    {
        crate::rhost::api_workhosts::invoke(fun_name, args)
    }
    else if fun_name.starts_with("base::")        //Basic ability
    {
        crate::base::api::invoke(fun_name, args)
    }
    ......
}

Add the module prefix before the Qt request function name string, and the request that starts with “rhost::” is forwarded to the rhost::api_host::invoke_host function, which implements the sub-module processing.

Take a closer look at these lines of code (screenshot from vscode) :

The deserialization code for the three parameters a1,a2, and a3 is exactly the same, but the resolved variable types are Vec,HashMap, and HashMap. How is this done? In fact, it is determined by the following api::foo2(a1, a2,a3), because a1,a2,a3 as the parameters of the business function foo2, and the function definition of foo2 determines their data type, so in the previous deserialization process according to the corresponding type to evaluate, this change history behavior is wonderful?

This isn’t quantum mechanics, it’s called type inference in Rust. Type inference not during the run time of the program, but during compilation, which is one reason Rust compiles slowly because the compiler does a lot of additional work.

Similarly, the Qt side of QRust is slow to compile due to the use of generic templates.

FFI

Once the Rust side is compiled into a link library, there are five defined interfaces for the Qt side, disguised as C functions, and their implementation code is in the lib.rs file:

• pub unsafe extern “C” fn qrust_init()
• pub unsafe extern “C” fn qrust_call(in_ptr: *const c_uchar, size: c_int) -> Ret
• pub unsafe extern “C” fn qrust_free_ret(ret: Ret)
• pub unsafe extern “C” fn qrust_free_str(ptr: *const c_char)
• pub unsafe extern “C” fn qrust_free_bytes(ptr: *const u8, len: c_int)

These 5 interfaces are the original FFI-compliant functions that support C calls to Rust, so you can see a little bit of FFI complexity:)

If you are just a QRust user and do not care about the design and implementation of QRust, these five interface functions only need to master the first one.

1）qrust_init()  – The purpose of this function is to do something up front before calling the Rust side business function. In the demo, the Qt side calls this function in main() at the beginning, passes a path to Rust through the process environment variable, and Rust outputs the log generated by log4rs to the log file under this path. Qt and Rust have their own logging frameworks, and it’s not possible to merge Rust’s logs into Qt’s qDebug(), so the log output goes its own way.

2） qrust_call() and qrust_free_ret()  –  These two functions are the core of QRust. When the Qt side calls the Rust side business function, the name string of the business function and the serialized parameters are binary packaged, and the address of the binary package is sent to the Rust side as parameters. After the return value of the business function is packaged (serialized), the binary package address is wrapped into the Ret structure and returned to the Qt caller by the qrust_call() function.

QRust’s function request package and return package are passed as addresses, the request package is generated by the Qt side, the return package is generated by the Rust side, in the memory layout of these two packages are on the heap, so the memory they occupy needs to be released after the function call. Qt and Rust may have different mechanisms for memory allocation and release, so do not free memory requested by the other. The request package is generated by the Qt side and can be released by the Qt side when the function returns, but the return package is generated by the Rust side, and the Rust side has no way of knowing when the return package has been used on the Qt side, so it cannot be released. So you need Qt to tell Rust: “I’m done, you can safely free memory”, qrust_free_ret() function is used for this.

Memory operations are always complex and error-prone, which is one of the difficulties of C calling Rust, but QRust encapsulates these operations internally so that the user does not touch the underlying code.

3) qrust_free_str() and qrust_free_bytes() – These two interface functions are not fully implemented in the current version of QRust, they are interfaces for big data calls. For example, when hundreds of megabytes of string or binary data are passed between Qt and Rust, it is not suitable to use serialization mechanism, and should be passed directly in the address way, which is free memory use.

Qt project

Let’s focus on the Qt side:

metastruct.h，qrust_de.h|.cpp， qrust_ser.h|.cpp，rust.h|.cpp – These 7 files are QRust in the Qt side of the implementation source, if you are not interested in the underlying implementation of QRust, you can ignore them, just need to import these 7 files into the project, and make sure not to edit modification.

First, focus on CMake’s configuration file, CMakeLists.txt.

1) Add a ‘Concurrent’ component to find_package, which supports asynchronous calls.

2) Add the LINK_DIRECTORIES configuration and define the qrust static library path, noting that this path is in the compiled output path of the Rust project.

find_package(QT NAMES Qt6 Qt5 REQUIRED COMPONENTS Widgets Concurrent)
find_package(Qt${QT_VERSION_MAJOR} REQUIRED COMPONENTS Widgets Concurrent)

LINK_DIRECTORIES(D:/MySelf/project/QRust/code/rust/target/release)

3) Add Concurrent components and qrust static libraries to target_link_libraries accordingly:

target_link_libraries(qt_window PRIVATE
    Qt${QT_VERSION_MAJOR}::Widgets
    Qt${QT_VERSION_MAJOR}::Concurrent
    qrust
)

4) Copy metastruct.h, qrust_de.h|.cpp, qrust_ser.h|.cpp, rust.h|.cpp 7 files into the project, and load them into the cmake configuration file:

set(PROJECT_SOURCES
        main.cpp
        mainwindow.cpp
        mainwindow.h
        mainwindow.ui

        metastruct.h
        qrust_de.cpp qrust_de.h
        qrust_ser.cpp qrust_ser.h
        rust.cpp rust.h
)

Refocus on the main.cpp file:

1) In main.cpp, ws2_32.lib and Bcrypt.lib need to be loaded, which are the library files that qrust relies on when compiling in windows environment. Note that these two lines need to be commented out in non-Windows environments.

2) Call the Rust-side function qrust_init() in main(). This call is not necessary, but if you want to do some initialization work, such as log configuration on the rust side.

#include "mainwindow.h"
#include <QApplication>

#pragma comment (lib, "ws2_32.lib")
#pragma comment (lib, "Bcrypt.lib")

//qrust_init() is a rust function, so declared extern "C"
//qrust_init() is rust function, so declared extern "C"
extern "C" {
    void qrust_init();
}

int main(int argc, char *argv[])
{
    QApplication a(argc, argv);

    //Here, set rust's log output path by calling qrust_init()
    QString log_path = "d:/TMP/log";  //This path has to be changed to your own
    qputenv("LOG_PATH", log_path.toStdString().c_str());  //Setting environment variables
    qrust_init();  //qrust initialization

    MainWindow w;
    w.show();
    return a.exec();
}

The Mainwindow.h|.cpp file is a demo of the QRust tape, which I’ll cover in the next chapter.

Compile and run:

The Rust side needs to be compiled first, because it will be used as a link library on the Qt side. Rust compilation command:

cargo build --release

Note that Qt does not recognize changes to static libraries generated by Rust during compilation and debugging. When you recompile the Rust side and then compile Qt, Qt still runs as the old version. This problem can be solved by adding or removing a space in the code file that calls Qrust on the Qt side, causing the file to change, triggering Qt to recompile the file and relink to the link library.

When a Qt project is compiled, CMake on Linux may throw the following error:

  undefined reference to symbol ‘dlsym@@GLIBC_2.4’

The workaround is to add the dl library under target_link_libraries as follows:

target_link_libraries(qt_window PRIVATE
    Qt${QT_VERSION_MAJOR}::Widgets
    Qt${QT_VERSION_MAJOR}::Concurrent
    qrust
    dl
)