Roadmap

This document sets out the high-level tasks which the vg development team hopes to accomplish in the next few versions of vg and beyond.

By Time

These are the things we hope to achieve on several planning horizons:

End of Summer (9/22)

• LR Giraffe prototype (Adam, Xian, Stephen)
  • Giraffe chain seed fill-in for long reads (Adam)
  • Stop speeding up DI2 (Xian)
  • Good understanding of/models for how many/which minimizers to use for long reads in Giraffe, for nanopore and PacBio (Stephen)
• Releases for vg libraries (libhandlegraph, libbdsg, libvgio)
  • Enhanced release acceptance testing so a release means production quality, like we would do for a paper. Proven to be able to run some number of genomes. (Tag paper-validated hashes as release?)
• Actual tutorials and examples for using libvgio/libbdsg for DV use cases (see pysam's progressively increasing examples)
• See if snarl normalization improves mapping/calling (Robin)
• Delete at least one index each from vg index #3144 (Adam, Jordan)
• Eliminate intermediate Alignment as surject output, go graph Alignment -> BAM record(s) to make supplementary alignments work (Jordan)
• Drop pinchesAndCacti and sonlib
  • Drop Cactus-library-based snarl finder (Adam)
• Subset HPRC graphs with PanGenie using kmers and map against that, as an alternative to frequency filtering.
• Speed up snarl normalizer (Robin)

End of Year (12/22)

• Giraffe pretty good on long reads
• Better developer documentation (model: pysam docs) for vg library,
  • Links from libvgio Doxygen section to Protobuf-derived doc pages
• Stability guarantees for vg libraries (like htslib, 1-2 releases per year requiring a code change)
• Support DV use cases in vg libraries (libhandlegraph/libvgio)
  • Query the graph (for nodes/edges)
  • GRCh38 location -> retrieve aligned reads near there
  • Get read attributes and know what they mean
• User-facing, under-test Giraffe docs for HPRC graphs (Jordan)
• Delete old snarl manager (except as a backward-compatible load?) and use snarl-only DI2 everywhere
• Figure out if we need a libsnarls actually (Adam, Xian, Jordan)
• No cruft in vg index #3144 (Adam, Jordan)
• Example tutorial under test for idiomatic Giraffe on long-node rGFA on long reads with auto-chopping (autoindex already does the work probably) and close #3126
  • Queries (vg find) in rGFA space
• Generalize spliced alignment code to let vg surject handle long deletions vs. target path (and generate 30000D CIGARs)

Next Summer (6/23)

• Libraries free of cryptic error messages when the user or inputs are wrong
• Just front-end the autoindex system with a sensible CLI for single indexes in vg index. How do we deprecate things? 3 levels of command? Clear input vs. output distinction.
• Let surject generate supplementary alignments for e.g. mappings over inversions
• Use memory-mapped graphs (Adam)
  • For tube map, to enable interactive whole-genome use (Future data vis enthusiast)
  • For Giraffe
• Giraffe optimizations/presets for Ultima single-end short reads (A student)
• Giraffe actually competitive on long reads
• Completely new paradigm for pangenome variant calling so we don't need population-specific reference hacks
  • Variant calling against rGFA references, allowing new variants all over
  • Think about PanGenie fitting in here
  • Will need to involve non-independent read mapping
  • Solve CNVs with EM (Jordan)

Wishlist

These are things we would like to do eventually.

• Eliminate vg::VG (Jordan)

  • Steal all the things only it can do away from it

• Default everything to GAF instead of GAM

  • mpGAF (Jordan, Jonas)
  • Also pgvf (Graph to graph)
  • Calls and snarls in one of these?

• Python bindings for libhandlegraph algorithms

  • Are they the right algorithms?

• Use of MCMC techniques in the genotyper with multipath alignments

• vg deconstruct and Beagle to impute genotypes into partially-mapped and called data, as a PanGenie alternative (Erik, Andrea)

• Alignment

  • Adoption of the multipath alignment paradigm as the default
  • Graph-to-graph mapping (Xian)

• Variant Calling

  • Implementation of an HHGA-like machine learning based variant caller
  • Integration of variant calling and assembly polishing processes
  • Prune the zoo of TraversalFinders, and expose the useful ones to Python

• Visualization

  • Browser-free tube map
  • Better tube map handling of edge cases
    • No haplotypes on a node
    • Starting on a rare haplotype

• Infrastructure

  • Destructively modernize and unify IO
    • Eliminate VPKG framing if possible in favor of magic numbers everywhere
      • Resolve ensuing questions about GAM format
        • Just use GAF?
      • Handle things like GFA that need to manually sniff
    • Just save from the object; no more save_handle_graph
    • Magic format registration for libvgio magic numbers for loading
    • Depend on libvgio in libbdsg to do the IO there and pick the right handle graph implementation
  • Replace Protobuf internal formats with faster ones
  • Revision of ID assignment logic to allow deterministic node breaking
  • Accept gzipped GFA if practical (can't mmap)
  • Improved HandleGraph API
    • Abstract away node boundaries
    • View all sequence as C++17 string_views instead of sequence-owning strings
    • O(1) reverse complement DNAStringView
  • CMake-ify the main vg build
  • Eliminate old systems and their associated submodules, or factor them out into their own projects
    • vg vectorize could be its own project
      • Update vg vectorize to modern, system Vowpal Wabbit
      • Or pull it out into its own submodule and remove Vowpal Wabbit dependency from vg
    • Eliminate RocksDB from vg; everybody using vg map uses GCSA indexes now.
    • vg genotype
    • vg srpe
  • More cross-language support
    • Interoperate with Rust handle graph users/providers
    • Interoperate with Java handle graph users/providers