Skip to content

Latest commit

 

History

History
582 lines (443 loc) · 25.3 KB

CONTRIBUTING.adoc

File metadata and controls

582 lines (443 loc) · 25.3 KB

Contributing

Table of Contents

Contributions are welcome! However, please open an issue first and discuss the details with me, preferably before you start working on a pull request! And if possible, avoid very large PRs. I usually can’t find the time to process large PRs.

1. Basics

ReaLearn is written in the programming language Rust. It makes heavy use of reaper-rs, which provides Rust bindings for the REAPER C++ API. reaper-rs was developed together with ReaLearn but is designed as independent library that can be used for REAPER plug-ins of all sorts.

Another noteworthy dependency and byproduct of ReaLearn is helgoboss-learn, a crate which provides DAW-agnostic MIDI learn logic (basically the source and mode parts of ReaLearn). Like reaper-rs, it’s designed as independent library and could be used to provide similar MIDI-learn functionality in other DAWs.

2. Code statistics

Use Tokei or Onefetch to display some interesting statistics about the code, e.g. the lines of code and used languages. The file .tokeignore contains code to be ignored when counting the lines of code.

3. Architecture

See here.

4. Directory structure

Directory entry Content

/

Workspace root

/allocator

A custom global allocator for deferring deallocation in real-time threads

/api

Data structures of ReaLearn’s preset API (which is e.g. used in the Lua-based ReaLearn Script)

/base

Very generic utility code used by many crates in the workspace

/csi

Code for interfacing with the ControlSurfaceIntegrator (CSI) project

/dialogs

The single source of truth for ReaLearn’s GUI dialogs

/doc

Documentation

/extension

Helgobox REAPER extension (provides some additional convenience around the actual ReaLearn plug-in)

/helgoboss-license-processor

Contains code for license processing (currently a private submodule)

/macros

Various Rust macros for usage in this project only

/main

Main crate: The actual ReaLearn instrument plug-in (realearn)

/playtime-api

Playtime data structures for describing e.g. clip engine presets

/playtime-clip-engine

Playtime Clip Engine for playing/recording clips (currently a private submodule). Is a workspace member because that makes it much simpler to use the same dependency versions everywhere.

/playtime-clip-engine-placeholder

A placeholder crate for the Playtime Clip Engine. Users who don’t have access to the Playtime Clip Engine submodule, must rename this directory to playtime-clip-engine in order to be able to build ReaLearn without feature playtime.

/pot

Core logic behind Pot Browser, also powers the Pot targets

/pot-browser

The actual Pot Browser user interface

/resources

REAPER projects for manual testing, controller preset files, etc.

/rx-util

Some reactive programming helpers

/swell-ui

Minimalistic UI framework based on SWELL (swell-ui)

5. Build

5.1. All operating systems

Update Cockos WDL (optional)

cd main/lib/WDL
git checkout main
git pull
# After updating Cockos WDL, regenerate Rust bindings (because we use WDL's EEL code, for example)
cargo build --features generate
cargo fmt

Regenerate Luau language bindings (optional)

Luau language bindings should be regenerated from Rust after changing something in api or playtime-api. This is done simply by executing all tests like this:

RUST_MIN_STACK=5242880 cargo test --package helgobox-api --lib bindings::luau::export_luau

Regenerate artwork (optional)

Artwork such as toolbar icons can be regenerated by running a crate:

cargo run helgobox-artwork-processor

3 different approaches for generating code …​ yes, maybe it’s time to unify this ;)

Regenerate diagrams in documentation

This is about the diagrams in the Antora documentation, e.g. the glue signal flow.

sh regenerate-doc-diagrams.sh

Building with Playtime (can only be done by Helgoboss)

Add this to $HOME/.cargo/config (otherwise Cargo will have issues fetching the private submodules):

[net]
git-fetch-with-cli = true

5.2. Windows

In the following, you will find the complete instructions for Windows 10/11, including Rust setup. Points where you have to consider the target architecture (for example, REAPER 32-bit vs. 64-bit) are marked with ⭐.

  1. Enable "Developer mode" in the Windows settings (this is needed because ReaLearn uses Symlinks within its Git repository)

  2. Setup "Visual Studio" (currently tested with version 2022)

    • Rust uses native build toolchains. On Windows, it’s necessary to use the MSVC (Microsoft Visual Studio C++) toolchain because REAPER plug-ins only work with that.

    • Visual Studio downloads → All downloads → Tools for Visual Studio 2022 → Build Tools for Visual Studio 2022

    • Start it and follow the installer instructions

    • Required components

      • Workloads tab

        • "C++ build tools" (large box on the left)

        • Make sure "Windows 10 SDK" is checked on the right side (usually it is)

        • If on Windows ARM64: Make sure "C++ Clang-Tools" is checked on the right side (normally not checked!, currently necessary for compilation of ring dependency)

      • Language packs

        • English

  3. Setup Rust

    • Download and execute rustup-init.exe

    • Accept the defaults

    • Set the correct toolchain default ⭐

      rustup default 1.77.2-x86_64-pc-windows-msvc

  4. Download and install Git for Windows

  5. Clone the ReaLearn Git repository

    git clone https://github.com/helgoboss/helgobox.git`
cd helgobox
git checkout v2.16.0 # or any other release tag

# ONLY IF YOU ARE HELGOBOSS
git submodule update --init

# OTHERWISE
git submodule update --init main/lib/WDL main/lib/helgoboss-learn
rmdir playtime-clip-engine
rename playtime-clip-engine-placeholder playtime-clip-engine

  6. Build ReaLearn (after that you should have a helgobox.dll in target\debug)

    cargo build --features egui

5.3. Linux

Complete instructions to build ReaLearn from a fresh Ubuntu 18.04.3 LTS installation, including Rust setup:

# Install native dependencies
sudo apt update
sudo apt install -y curl git build-essential pkg-config php nasm llvm-dev libclang-dev clang libudev-dev libxdo-dev libx11-dev  libxcursor-dev libxcb-dri2-0-dev libxcb-icccm4-dev libx11-xcb-dev mesa-common-dev libgl1-mesa-dev libglu1-mesa-dev libspeechd-dev libgtk-3-dev


# Install Rust (copied from the official Linux installation instructions)
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh # choose 1 (default)
source $HOME/.cargo/env

# Set the correct toolchain default
rustup default 1.81.0-x86_64-unknown-linux-gnu

# Clone ReaLearn repository
git clone https://github.com/helgoboss/helgobox.git
cd helgobox
git checkout v2.16.0 # or any other release tag

# ONLY IF YOU ARE HELGOBOSS
git submodule update --init

# OTHERWISE
git submodule update --init main/lib/WDL main/lib/helgoboss-learn
rmdir playtime-clip-engine
mv playtime-clip-engine-placeholder playtime-clip-engine

# Build (after that you should have a "libhelgobox.so" in "target/debug")
cargo build --features egui

Some words about the native dependencies:

  • curl git build-essential pkg-config are bare essentials.

  • php is needed to translate the ReaLearn dialog resource file to C so it can be processed by the SWELL dialog generator. It's also necessary for generating the 64-bit EEL assembler code. All of this is the typical WDL C way of doing things, no Rust specifics here.

  • nasm is needed for assembling the 64-bit EEL assembler code to produce asm-nseel-x64.o, which is necessary to make the custom EEL control and feedback transformations in ReaLearn’s absolute mode work.

  • llvm-dev libclang-dev clang are necessary for building with feature generate (to generate bindings to C).

  • libxdo-dev is needed to control the mouse (see target "Global: Mouse")

  • libudev-dev is needed for connecting to Stream Deck via HID API

  • libx11-dev libxcursor-dev libxcb-dri2-0-dev libxcb-icccm4-dev libx11-xcb-dev mesa-common-dev libgl1-mesa-dev libglu1-mesa-dev are necessary for egui-baseview (egui is the GUI framework used for ReaLearn’s control transformation editor)

  • libspeechd-dev is necessary for the speech source

  • libgtk-3-dev is necessary to obtain the X window and X display from a SWELL OS window, in order to fire up OpenGL/egui in it

5.4. macOS

The following instructions include Rust setup. However, it’s very well possible that some native toolchain setup instructions are missing, because I don’t have a bare macOS installation at my disposal. The Rust installation script should provide you with the necessary instructions if something is missing.

# Install Rust
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh # choose 1 (default)
source $HOME/.cargo/env
rustup default 1.81.0-x86_64-apple-darwin

# Clone ReaLearn
cd Downloads
git clone https://github.com/helgoboss/helgobox.git
cd helgobox
git checkout v2.16.0 # or any other release tag

# ONLY IF YOU ARE HELGOBOSS
git submodule update --init

# OTHERWISE
git submodule update --init main/lib/WDL main/lib/helgoboss-learn
rmdir playtime-clip-engine
mv playtime-clip-engine-placeholder playtime-clip-engine

# Install build dependencies
brew install php

# Build ReaLearn
cargo build --features egui

6. GUI

The GUI dialogs are defined in the dialogs directory. Whenever ReaLearn is built, the code there generates an old-school Windows dialog resource file (target/generated/msvc.rc) and a Rust file which contains all the resource ID constants (main/src/infrastructure/ui/bindings.rs).

Previously I used the Visual Studio C++ 2019 resource editor to WYSIWYG-edit this file as part of the solution msvc.sln, but this was too tedious.

Warning
 You can still preview the generated file in Visual Studio but don’t edit the RC file, the changes will be overwritten at build time! Adjust the Rust code in the dialogs directory instead.

On macOS and Linux, an extra step will happen at build time: It will try to use a PHP script (part of Cockos SWELL) to generate target/generated/msvc.rc_mac_dlg, which is a translation of the RC file to C code using SWELL. So make sure you have PHP installed on these platforms!

7. Test

Yes, there are tests but there should be more. While ReaLearn’s basic building blocks helgoboss-learn and reaper-rs are tested pretty thoroughly, ReaLearn itself has room for improvement in that aspect.

7.1. Unit tests

Unit tests should be executed with a higher stack size because there’s one unit test that generates and formats Lua code and this currently overflows the stack in debug builds.

RUST_MIN_STACK=104857600 cargo test

7.2. Integration tests

There’s a growing built-in integration test, launchable via action [developer] ReaLearn: Run integration test. In future, it would be nice to run this integration test during continuous integration, just like in reaper-rs.

8. Log

It’s possible to make ReaLearn output log messages to stdout by setting the HELGOBOX_LOG environment variable, e.g. to debug,vst=info. It follows this format. Beware that e.g. on Windows, stdout is not shown, not even when executing REAPER from the command line. One way to make it visible is to execute REAPER with a debugger.

9. Metrics

It’s possible to make ReaLearn expose execution metrics.

9.1. Prometheus endpoint

  • If the projection server is running, metrics will then be exposed at /realearn/metrics in the popular Prometheus format. That’s great for visualization.

    • Just add this to your prometheus.yml (you might need to adjust the port):

scrape_configs:
  - job_name: 'realearn'
    metrics_path: '/realearn/metrics'
    static_configs:
      - targets: ['localhost:39080']

  • If you don’t have any metrics enabled, this will show zeros only.

Prometheus is usually available at http://localhost:9090/.

9.2. ReaLearn metrics

  • You can turn on ReaLearn metrics by setting the environment variable HELGOBOX_METRICS (value doesn’t matter).

  • If this environment variable is set (value doesn’t matter), ReaLearn will record some metrics and expose them on the Prometheus endpoint mentioned above.

  • If ReaLearn is built with the Playtime Clip Engine, this flag will also enable Clip Engine metrics. This can negatively effect clip playing performance because many clip engine metrics are captured in real-time threads.

10. Debug

10.1. Debug REAPER scanning ReaLearn

Set vst_scan=1 in the [reaper] section of reaper.ini. That makes the debugged REAPER process itself do the scanning.

10.2. Obtain debug symbols

Debug symbols are stripped from release builds but stored as build artifact of the GitHub Actions "Create release" workflow. If you want the symbols for a specific build, proceed as follows:

  1. Open the list of ReaLearn "Create release" workflows.

  2. Use the branch filter to show all releases builds made for a specific version, e.g. "v1.11.0".

  3. Click the desired workflow.

    • GitHub seems to do a fuzzy search, so if there are pre-releases (e.g. "v1.11.0-pre2"), you will see them, too.

    • In that case, just choose the latest one.

  4. You will see a list of artifacts, one for each OS-architecture combination.

  5. Download the one you need and unzip it.

    • You will find both the library file and the symbol file (e.g. realearn.pdb for a Windows build).

10.3. Turn on complete backtraces

As soon as you have the debug symbols, you can make ReaLearn print full backtraces (including line number etc.) in the REAPER ReaScript console. Here’s how you do it.

Windows

  1. Set the environment variable _NT_ALT_SYMBOL_PATH to some directory of your choice.

  2. Copy the PDB file in there.

  3. Fire up REAPER with ReaLearn an make it panic. You should see a complete backtrace now.

10.4. Lookup symbols for symbol-less backtraces

The problem with release builds is that they don’t contain debug symbols and therefore backtraces usually contain not much more than memory addresses. Especially backtraces generated by Windows release builds leave a lot to be desired.

ReaLearn has a built-in REAPER action which attempts to look up symbol information for a given error report: "ReaLearn: Resolve symbols from clipboard". Works on Windows only. To be used like this:

  1. Make sure the PDB for the release build in question is on the search path (see section above).

  2. Fire up an ReaLearn using exactly that release build.

  3. Copy the error report to the clipboard.

  4. Execute the action.

10.5. Differences between debug levels

macOS

Insights:

  • The size difference between debug = 1 and debug = 2 is almost nothing (both 58 MB), and there’s nothing to gain from debug = 2 in terms of stack traces.

  • The size difference between debug = 0 and debug = 1 is around 5 MB (53 MB vs. 58 MB), and debug = 1 only makes a difference if the source files exist (showing line numbers), and only for panics.

  • Hard crash stack traces are completely independent of the debug value. They are always helpful except when stripping the symbols.

  • strip = symbols leads to the smallest binaries (38 MB) but also to completely useless stack traces, both in soft and hard crashes. However, split-debuginfo = "packed" seems to fix this at least for hard crashes, at a similar-sized binary (not for panics though) …​ even if the DSYM directories are not on disk. What also fixes this for hard crashes is stripping via `strip -u -r. We shouldn’t do that anymore!

  • strip = debuginfo leads to an okay size reduction (53 MB) but removes line numbers even if source files exist. However, split-debuginfo = "packed" solves this by creating dSYM directories, as long as they are there.

Takeaway:

  • We should build with debug = 2

    • While debug = 1 is actually enough for most purposes, it can’t hurt building with debug = 2 since we strip debuginfo anyway. So there’s no size difference for the final binary. That way we have more debuginfo on the server whenever we need it.

  • strip = debuginfo is the max we can strip away if we want panics to contain something useful

  • strip = symbols is only okay if we are fine with bogus stack traces in panics. In that case, we must use split-debuginfo = "packed" to get at least detailed stack traces in case of hard crashes.

  • We should use split-debuginfo = "packed" in all cases.

  • Would be good to find a way to leverage symbols for panic stack traces, but I think there is none.

"debug = 0"
Panic (sources don’t matter)
7:        0x120531450 - helgobox::infrastructure::plugin::sandbox::execute::h9608b1370cb08106
Hard crash (sources don’t matter)
4   helgobox-arm64.vst.dylib      	       0x120527df4 _$LT$helgobox..domain..targets..track_volume_target..TrackVolumeTarget$u20$as$u20$helgoboss_learn..mode..target..Target$GT$::current_value::hff7df2fe68cec5d5 + 20
"debug = 1"
Panic if sources exist
7:        0x13071ffdc - helgobox::infrastructure::plugin::sandbox::execute::hb564445c2d9211ee
                               at /Users/helgoboss/Documents/projects/dev/realearn/main/src/infrastructure/plugin/sandbox.rs:3:5
Panic if sources are gone
7:        0x140f1ffdc - helgobox::infrastructure::plugin::sandbox::execute::hb564445c2d9211ee
Hard crash (sources don’t matter)
6   helgobox-arm64.vst.dylib      	       0x1302d4a64 _$LT$helgobox..domain..mapping..CompoundMappingTarget$u20$as$u20$helgoboss_learn..mode..target..Target$GT$::current_value::hadfcb1be993900b3 + 20 (mapping.rs:2498) [inlined]
"debug = 2"
Panic if sources exist
7:        0x12160dafc - helgobox::infrastructure::plugin::sandbox::execute::h18fb689d4112e2d3
                               at /Users/helgoboss/Documents/projects/dev/realearn/main/src/infrastructure/plugin/sandbox.rs:3:5
Panic if sources are gone
7:        0x153f81afc - helgobox::infrastructure::plugin::sandbox::execute::h18fb689d4112e2d3
Hard crash (sources don’t matter)
6   helgobox-arm64.vst.dylib      	       0x1212d8148 _$LT$helgobox..domain..mapping..CompoundMappingTarget$u20$as$u20$helgoboss_learn..mode..target..Target$GT$::current_value::h15041e3226fa455d + 20 (mapping.rs:2498) [inlined]
"debug = 2; strip = debuginfo"
Panic (sources don’t matter)
7:        0x138616570 - helgobox::infrastructure::plugin::sandbox::execute::hd0d406afe4d62df9
Hard crash (sources don’t matter)
4   helgobox-arm64.vst.dylib      	       0x13885aab4 _$LT$helgobox..domain..targets..track_volume_target..TrackVolumeTarget$u20$as$u20$helgoboss_learn..mode..target..Target$GT$::current_value::h35c07d80eb0a312b + 20
"debug = 2; strip = symbols"
Soft crash (sources don’t matter)
0:        0x14bdef0ec - _NSEEL_HOSTSTUB_EnterMutex
1:        0x14bd56f08 - _NSEEL_HOSTSTUB_EnterMutex
2:        0x14bfbe4e4 - _cpp_to_rust_ProjectStateContext_SetTempFlag
3:        0x14bfbddcc - _cpp_to_rust_ProjectStateContext_SetTempFlag
4:        0x14bfbca08 - _cpp_to_rust_ProjectStateContext_SetTempFlag
5:        0x14bfbdabc - _cpp_to_rust_ProjectStateContext_SetTempFlag
6:        0x14c075710 - _cpp_to_rust_ProjectStateContext_SetTempFlag
7:        0x14af50aa0 - _ReaperPluginEntry
8:        0x14bd58fcc - _NSEEL_HOSTSTUB_EnterMutex
9:        0x14bd82844 - _NSEEL_HOSTSTUB_EnterMutex
Hard crash (sources don’t matter)
7   helgobox-arm64.vst.dylib      	       0x14b34b5e0 0x14a65c000 + 13563360

11. Documentation

All documentation is written in AsciiDoc.

Some SVGs embedded in the architecture documentation are generated via NodeJS / SVG.js in doc/svg-gen/index.js. After modifying this file, you need to execute the following command in the project root:

node doc/svg-gen/index.js

12. License check

It’s important to make sure that the licenses of all dependencies are compatible with the final license. We use cargo-deny for this.

Installation:

cargo install --locked cargo-deny

Check:

cargo deny check licenses

13. License report

It’s important to make sure that the licenses of all dependencies are compatible with the final license. We use cargo-deny for this.

Installation:

cargo install --locked cargo-about

Generate the report:

cargo about generate --fail --workspace --all-features --threshold 0.93 about.hbs > about.html

14. Release

This serves mainly as a checklist for Helgobox’s author.

  1. Check licenses via cargo deny check licenses and make sure the outcome is "licenses ok"

  2. Update license report (see above)

  3. Take care of app versioning

    • Plug-in repository: Adjust HOST_API_VERSION and MIN_APP_API_VERSION

    • App repository: Adjust appApiVersion (macOS, Swift), APP_API_VERSION (Windows, C++) and _minHostApiVersionString (Dart)

  4. Bump up the app version number in pubspec.yaml.

  5. Bump up the plug-in version number in main/Cargo.toml.

    • Either to a prerelease (e.g. 2.0.0-pre1) or a final release (e.g. 2.0.0).

    • This is important for having the correct version number displayed in ReaLearn UI.

  6. Build at least once via cargo build --features playtime,egui.

    • This updates Cargo.lock and is important for not having the -dirty display in ReaLearn UI.

  7. Update the user guide if not done already.

  8. Create a version tag via git tag v2.0.0-pre1.

  9. Push via git push origin master --tags.

  10. While GitHub Actions executes the release job, take care of the following.

    • Can only be done by @helgoboss because it needs access to the helgoboss.org website repository.

    • If it’s a prerelease, make sure we are on a prerelease cycle branch of the website repository.

    • Add a changelog entry in data.yaml.

    • In src/snippets/projects/realearn/repo, enter git checkout master and git pull to pull the latest user guide changes.

    • Push via git push origin HEAD and wait until Netlify deployed the page.

    • All the following stuff needs to be done using Netlify’s branch preview if it’s a prerelease!

    • Update helgoboss ReaPack index.

      • Generate ReaLearn-only ReaPack index by requesting /projects/realearn/reapack.txt.

      • Integrate the generated index by copying everything from <category name="Extensions"> and pasting it to the helgoboss ReaPack index without overwriting the preset categories on the top of the file.

      • Don’t push the index yet!

    • Author a REAPER forum ReaLearn thread entry with help of /projects/realearn/reaper-forum.txt but don’t submit yet!

    • Download the user guide by requesting /projects/realearn/user-guide.

    • Copy the corresponding changelog entry in markdown format by requesting /projects/realearn/changelog.md.

  11. Once the release job has finished successfully, edit the not-yet-published release that has been created.

    • Paste the copied changelog entry to the release notes.

    • Manually add the previously downloaded user guide as release artifact named realearn-user-guide.pdf.

  12. Publish the release.

  13. Push the helgoboss ReaPack index.

  14. Submit the REAPER forum ReaLearn thread entry.

  15. Check if synchronization of the ReaPack repository works.

15. Troubleshooting

15.1. Windows: ReaLearn DLL doesn’t unload

In REAPER for Windows it’s possible to enable complete unload of VST plug-ins (Preferences → Plug-ins → VST → Allow complete unload of VST plug-ins). This also affects ReaLearn. Removing the last ReaLearn instance should work with and without this flag enabled, it’s important to test this.

I ran into a case in which Windows was not unloading ReaLearn even though that option was enabled. The reason turned out to be a registry entry that Windows must have created automatically at some point:

HKEY_CURRENT_USER\SOFTWARE\Microsoft\Windows NT\CurrentVersion\AppCompatFlags\Layers → C:\REAPER\reaper.exe with value $ IgnoreFreeLibrary<realearn.dll>

Removing this entry made unloading work again. What a nasty trap!