Proposal: Scanner Modularization

[rumpelsepp]

TL;DR: This breaks every API, so this qualifies for gallia 2.0.

Problem Statement

Startup Performance

gallia suffers from a very slow startup experience; which causes a bad user experience, especially on slow devices like the raspberry pi. The reason for this lies in gallia's architecture which is strongly coupled with the CLI interface. The whole codebase is imported when --help is issued; resulting in a lot of syscalls and filesystem traversal.

Developer Experience

Further, the developer experience for modules that use multiple scanner classes in sequential order (e.g. session scanner depending on the results of a discover scanner) is bad. This bad experience is also caused by the tight coupling of the cli and the scanner modules. It is indeed impossible to run a scanner from within Python code without specifying a command line invocation string.

Another problem with using scanners as modules is the creation of an event loop. .entry_point() creates an own event loop. Multiple event loops in a single CPython invocation cause odd problems due to garbage collection.

Testing

Testing scanner modules is cumbersome, since they rely on cli arguments and untyped argparse.Namespace containers. Further, there is no definition of scanner outputs/results in the sence of a function return value.

Configuration

The configuration file gallia.toml has no type checks. A moved config value was not discovered by a tester and silently broke a test setup; see also #377.

Logging

The logging module uses exit handlers with atexit. This causes problems when multiple scanners are used and a logfile per scanner is desired.

Proposed Solution

Separating CLI and Scanners

As a first step, the scanner classes need to be separated from any code related to the cli. Further, this code should be replaced by typed configuration containers. These containers should carry all information that is required to:

1. Generate the CLI with only importing a minimal set of gallia modules.
2. Use the same configuration system for the gallia.toml configuration file and benefit from the typing feature.
3. Use gallia scanner modules as a library without a CLI interface.

Perhaps while at it, the source code tree under commands could be restructured, since it will be separated from the cli.

Event Loop

Due to the event loop problem, entry_point() must be async.

Testing

Define outputs of scanners.

Alternatives Considered

Regarding slow tab completion: Tab completion scripts could be generated for each shell. Since we often use gallia from git and rely on the tab completion, this would create an additional maintenance cost. I assume that there even won't be any volunteers for this.^^

Implementation Draft

Living draft; will be modified as needed

# command/config.py
from pathlib import Path

from pydantic import BaseModel


class CommandConfig(BaseModel):
    # Shared configuration options for all scanners, such as:
    artifacts_dir: Path


class FooConfig(CommandConfig):
    foo_setting: str

# command/base.py

from abc import ABC

from gallia.command.config import CommandConfig


class BaseCommand(ABC):
    def __init__(self, config: CommandConfig) -> None:
        self.config = config
        # Plus several other things, such as creating the artifacts_dir etc.

    # Plus several other methods, such as entry_point().

# commands/scan/foo.py

# Using the public API which is defined using __all__ in command/__init__.py
from gallia.command import BaseCommand, FooConfig


class FooCommand(BaseCommand):
    def __init__(self, config: FooConfig) -> None:
        # TODO: Does this work with typing?
        super().__init__(config)

# cli.py
import asyncio

from gallia.command import BaseCommand, BaseConfig, FooConfig

# Contains all configuration classes.
config_registry: list[type[BaseConfig]] = [FooConfig]


def create_parser(registry: list[type[BaseConfig]]) -> argparse.ArgumentParser:
    # TODO: Figure out how to do this.
    # The defaults of the CLI should be populated from gallia.toml.
    return parser


def lookup_command_class(config: BaseConfig) -> type[BaseCommand]:
    # TODO: Figure out how to do this.
    #
    # There is importlib.import_module() which can be used to
    # import a module based on a string. importlib is the
    # official implementation if the import statement, just
    # as a library.
    return _do_magic_stuff_and_find_the_class(config)


def lookup_command(config: BaseConfig) -> BaseCommand:
    command_class = lookup_command_class(config)
    return command_class(config)


def main() -> None:
    parser = create_parser(config_registry)
    args = parser.parse_args()
    # args.config should now contain the populated config container.
    # Otherwise a method for converting a argparse.Namespace container
    # to a configuration object needs to be created.
    command = lookup_command(args.config)
    asyncio.run(command.entry_point())

From within Python Code, this could be used standalone; simply do:

import asyncio

from gallia.command import FooCommand, FooConfig


# Set config params as arguments to the FooConfig constructor.
config = FooConfig()
command = FooCommand(config)
asyncio.run(command.entry_point())

Testing is currently not existant for scanners (ok, there is the vecu runner, but this only tests if the scanners do not crash). There is pytest-mock to mimic the network. AFAIC this could be used somehow like this.

# tests/scanners/discover/test_uds.py

from gallia.command import FooCommand, FooConfig

# TODO: Find out how to specify a list of packets that should be returned by e.g. gallia.transports.doip.read.
# With predefined answers the scanner modules can be tested.

async def test_uds_discover(mocker) -> None:
    mocker.patch('gallia.transports.doip.write')
    mocker.patch('gallia.transports.doip.read')
    
    config = FooConfig()
    command = FooCommand(config)
    await command.entry_point()

    # Check expected output, that is not yet defined how it looks like.
    # Something like this:
    assert command.result == expected

Filenames

• ? Change logfilename to log.jsonl.zst to conform to the jsonline standard: https://jsonlines.org/

Open for Discussion

• How could lookup_command_class() work without importing the whole codebase?
• Where do we place the old settings for the command line, such as COMMAND, SUBCOMMAND, and so on? An idea might be an embedded class in the proposed CommandConfig -> CommandConfig.CLIConfig.
• How could scanner outputs look like? Perhaps also a typed container for every scanner, the same as the proposed configuration?
• How can we integrate logging setup?
• How can plugins specify a dumpcap invocation? Dumpcap defaults to ethernet traffic.
• Document settings from config file in artifacts.

References

• https://github.com/SupImDos/pydantic-argparse
• https://github.com/sander76/clipstick
• https://github.com/brentyi/tyro