Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rust bindings refactor #6309

Closed
wants to merge 93 commits into from
Closed

Rust bindings refactor #6309

wants to merge 93 commits into from

Conversation

emesare
Copy link
Member

@emesare emesare commented Jan 11, 2025

Opening this PR for public discussion. This needs more testing and also lacks a few major changes that make sense to land alongside the already existing set of breaking changes.

The Rust bindings are being heavily refactored for the 4.3 release. This is to address many of the shortcomings and historical issues, namely around maintaining and safety.

Some of the additions of this branch include:

  • Revised project structure
  • Consistent formatting (with rustfmt)
  • Many UB fixes
  • Unit tests
  • More/updated documentation
  • Real rust examples (i.e. cargo run --example)
  • Many of the already opened PR's
  • Additional APIs
  • Updated and idiomatic linking to the core
  • Consolidation of API's (removal of historical API's)
  • Rust main thread handler
  • Type driven API's (ex. RegisterId)
  • Updated CI

Reviewer Guide

TODO

  • Explain the CI and add a bunch of links to relevant GitHub Actions documentation
  • Explain the project structure (this isn't really necessary for consumers)
  • Explain the usage of Ref
  • Explain the usage of BnString (and strings in general)
  • Explain how to review PR's
  • Explain how to run tests

emesare and others added 30 commits December 22, 2024 01:21
Moves a bunch of stuff out of src/types.rs that did not belong:
- Confidence
- Variable
- Function specific stuff

- Refactored InstructionInfo, see the msp430 and riscv examples.
- Renamed Function::from_raw to Function::ref_from_raw and fixed places where the ref was incremented twice
- Fixed FunctionRecognizer leaking functions (see above)
- Fixed some apis incorrectly returning Result where Option is clearer
- Started to move destructured types to the From trait for converting to an from ffi types, see Location for an example
- Started to remove bad module level imports (importing std modules like mem all over the place)
- Moved some wrapper handle types to named handle field (this improves readability), see CoreArchitecture for an example
- Removed some unintuitive getters, this is bad practice for Rust code, just access the field directly, see DataVariable for an example
- General code cleanup, purposely did not run rustfmt, that will be a single seperate commit
- Fixed invalid views being able to invoke UB when dealing with databases
- Cleaned up some helper code in dwarf_import
- Fixed inverted is_null checks causing crashes! Oops!
Still a WIP, I think branch info is still invalid, need to figure out the issue there.

- Fixed some invalid Ref lifetimes when constructing indirectly, see Array<DataVariable> for example
- Added some more comments
- Removed some "magic" functions like MLIL Function::ssa_variables

There are still a bunch of invalid lifetimes that aren't crashing us due to the usage of those API's not living long enough. But they ARE an issue.
Trying to comment more TODO's as I go along.

- Renamed llil::Function to llil::LowLevelILFunction for clarity and consistency
- Take base structures by ref in StructureBuilder::set_base_structures to prevent premature drops
- Added more raw to wrapper conversions
- Removed UB prone apis
- Getting rid of more core module references, use std!
- Removed improper Guard usage in array impls for wrapper types with no context
- Untangling the UB of the Label api, still only half done :/
- Misc formatting
- Made Logger ref counted
- Fixed leaking name of logger every time something was logged
- Fixed the last (hopefully) of the unresolved labels
- Simplified some code
componentArray was never freed
When linking you must depend on the -sys crate.

This is because linker arguments (what says where to find binaryninjacore) are NOT transitive. The top level application crate MUST provide it.

For more information see:
- rust-lang/cargo#9554 (comment)
- oxidecomputer/omicron#225
Use cargo to manage the git repo ref instead
This is where all shipped public plugins that are not arch/view/platform/lang will be at from now on

Originally they were in the rust workspace, meaning they all shared a Cargo.lock which is ill-advised.
- More clarification on plugin/executable requirements
- Made examples actually rust examples
- Add Display impl for InstructionTextToken
- Renamed feature "noexports" to "no_exports"
We don't register a compatible log sink so they will just get sent into the void
This is being done in the hopes of curbing the multi-thousand line files that will occur once we flesh out the tests
Still need to add support for running tests in ci
- Architecture id's are now typed accordingly
- Fix some clippy lints
- Make instruction index public in LLIL
- Removed useless helper functions
- LLIL expressions and instruction indexes are now typed accordingly
This should show binaryninjacore-sys alongside binaryninja crate
- Remove lazy_static dependency
- Remove hacky impl Debug for Type and just use the display impl
- Add more debug impls
- Reorder some top level namespace items
- Add first type test
- Added main thread handler api
- Register a headless main thread handler by default in initialization
- Refactor QualifiedName to be properly owned
- Loosened some type constraints on some apis involving QualifiedName
- Fixed some apis that were crashing due to incorrect param types
- Removed extern crate cruft for log crate
- Simplified headless initialization using more wrapper apis
- Fixed segments leaking because of no ref wrapper, see BinaryViewExt::segment_at
- Added rstest to manage headless init in unit tests
- Added some more unit tests
- Refactored demangler api to be more ergonomic
- Fixed minidump plugin not building
- Fixup usage of QualifiedName in plugins
- Make QualifiedName more usable
- Make EdgeStyle type not wrap raw
- Regression tests for WARP will run on all bins in the out dir now
rbran and others added 3 commits January 22, 2025 20:58
Makes collaboration more in line with the refactor. Still a lot of work to do.

Namely still need:
- Proper errors
- _with_opts functions
- More ergonomic api
- Better connection procedure
- Updated documentation
- A LOT of unit tests
- An example
- Typed id's for everything (i dont want BnString as the id!!!)
- NEED to refactor the progress callbacks into the new progress api, but we should pull in some of the stuff the collab progress has
- Elimination of apis that are dumb helpers
@emesare
Copy link
Member Author

emesare commented Jan 23, 2025

Because I have tomorrow and Friday to finish this up I went ahead and pulled in #5773, I think I should have enough time to refactor it and add some tests and examples.

@rbran
Copy link
Contributor

rbran commented Jan 23, 2025

@emesare Thanks that solves my use case.

BTW I hope in the future we can add a function to the coreAPI to recover the BinaryView from a BinaryReader/BinaryWriter, this allow rust to implement #[repr(transparent)] for BinaryReader/Writer to BNBinaryReader/Writer. But that's a task for other PR.

pull_request_target allows PR's to access the headless license, for this to be safe we need to prevent people from running the job.

To prevent the job from being ran we add an environment requirement on testing that a reviewer must review the code and then manually approve it to run.
- Use GroupId instead of u64
- Use ProgressCallback in place of ProgressExecutor
- Misc cleanup of FileMetadata
- Add `save_to_path` and `save_to_accessor` to save modified binaries
- Added binary_view unit tests
- Added collaboration unit tests
- Fixed a few issues with the collaboration apis
- Renamed Command registration functions so that there is no import ambiguity
- Split out RemoteUndoEntry
- Collaboration apis now have a explicit `_with_progress` set of apis
- Misc clippy lint fixes
- Add extra info to README.md
- Refactor components api
- Add components unit test
@emesare
Copy link
Member Author

emesare commented Jan 25, 2025

The rolled up commit will be on https://github.com/Vector35/binaryninja-api/tree/rust_refactor

@emesare
Copy link
Member Author

emesare commented Jan 25, 2025

@emesare
Copy link
Member Author

emesare commented Jan 25, 2025

Merged with 6a63c17

@emesare emesare closed this Jan 25, 2025
@blacktop
Copy link

@emesare are you going to add a load with progress func for the headless session?

@emesare
Copy link
Member Author

emesare commented Jan 25, 2025

@emesare are you going to add a load with progress func for the headless session?

That sounds like a good idea, I'll add that today! Thank you.

@emesare
Copy link
Member Author

emesare commented Jan 25, 2025

@blacktop added with 9827603

I should note that it appears the core only calls that progress callback for loading BNDBs, IIRC there was a comment somewhere about that in the rust code. I could not find that comment so I added it to the respective functions with 0263957

@blacktop
Copy link

Thank you!

I know this is most likely an inappropriate place to ask, but I'm trying to understand how to run a plugin via the Rust API (BinExport) specifically. Are there any examples anywhere on how to do this? I can also ask on Slack if that's a better place?

@emesare
Copy link
Member Author

emesare commented Jan 25, 2025

No problem, by run you mean run a command that BinExport registered?

Like one of these:
image

If so than we would need to add the rust equivalent of PluginCommand

binaryninja-api/binaryninjaapi.h

Lines 14146 to 14168 in 8767400

/*!
The PluginCommand class is used for registering "commands" for Plugins, corresponding to code in those plugins
to be executed.
\ingroup plugin
The proper way to use this class is via one of the \c "Register*" static methods.
*/
class PluginCommand
{
BNPluginCommand m_command;
struct RegisteredDefaultCommand
{
std::function<void(BinaryView*)> action;
std::function<bool(BinaryView*)> isValid;
};
struct RegisteredAddressCommand
{
std::function<void(BinaryView*, uint64_t)> action;
std::function<bool(BinaryView*, uint64_t)> isValid;
};

In rust we can register commands but we cannot call them yet.

I unfortunately have a few things in front of adding that, specifically I need to implement the BinaryDataNotification API. If you would like to make an issue for fleshing out plugin commands to support calling them from rust, I can assign it to myself.

@emesare
Copy link
Member Author

emesare commented Jan 25, 2025

Also this is assuming the plugin command that you want to call does not then invoke some user interaction, that would require us to also add the interaction callback API.

There is a TODO for that in the flow graph example

// TODO: Register BNInteractionHandlerCallbacks with showGraphReport pointing at our function
// TODO: Idea: register showGraphReport that dumps a dotgraph to stdin

@blacktop
Copy link

@emesare I guess this is how it currently works in python? https://infosec.exchange/@binaryninja/113827475796582876

@blacktop
Copy link

No problem, by run you mean run a command that BinExport registered?

Like one of these: image

If so than we would need to add the rust equivalent of PluginCommand

binaryninja-api/binaryninjaapi.h

Lines 14146 to 14168 in 8767400

/*!
The PluginCommand class is used for registering "commands" for Plugins, corresponding to code in those plugins
to be executed.
\ingroup plugin
The proper way to use this class is via one of the \c "Register*" static methods.
*/
class PluginCommand
{
BNPluginCommand m_command;
struct RegisteredDefaultCommand
{
std::function<void(BinaryView*)> action;
std::function<bool(BinaryView*)> isValid;
};
struct RegisteredAddressCommand
{
std::function<void(BinaryView*, uint64_t)> action;
std::function<bool(BinaryView*, uint64_t)> isValid;
};

In rust we can register commands but we cannot call them yet.

I unfortunately have a few things in front of adding that, specifically I need to implement the BinaryDataNotification API. If you would like to make an issue for fleshing out plugin commands to support calling them from rust, I can assign it to myself.

Ah I see, thank you for this explanation. I will wait for PluginCommand running to be added to the Rust API. Thanks again!

@emesare
Copy link
Member Author

emesare commented Jan 25, 2025

Yea that would be through the PluginCommand, if this is something you need now than you can call BNGetAllPluginCommands yourself until I or someone else can get to it.

Here are the issues:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Component: Rust API Issue needs changes to the Rust API Effort: High Issue should take > 1 month Impact: Medium Issue is impactful with a bad, or no, workaround Type: Enhancement Issue is a small enhancement to existing functionality
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add binaryninjacore-sys to online docs
4 participants