General data Getters for outsourced data retrievability

[vgonkivs]

We define the Getter interface to abstract away protocol clients for retrieving data. We also define CascadeGetter as means of composing multiple getters and allowing nodes to have multiple data sources. Currently, cascading is internal to the node construction, and we want to give developers and node operators the ability to choose, turn on/off, prioritize, or include their own implemented getters for composition. Users will want to customize custom Getters to connect to third-party networks for storage and data retrievability, like KYVE, Filecoin, Arweave, etc.

[Wondertan]

Rough design ideas and we need more:

• Supplying Getters programmatically
  • This is the easiest option
  • But it is limiting as it forces custom light node builds for users wishing to use external retrievability/storage solutions
• Define a registry of Getter implementations
  • Every community member could add/propose a new getter.
  • Node operators would then be able to choose getters from registry in the config and configure them separately
  • May require all the getters included in binary
• Define a plugging system for Getters using Golang plugins.
  • Doesn't require a registry(which still might be usefull)
  • Anyone can then permissionless integrate their Getter
  • Very minimal binary overhead

[walldiss]

Let's discuss some practical considerations to figure out which way to go:

1. Multiple Custom Getters: Can users plug in multiple custom Getters simultaneously with each approach? This is essential for versatility in data retrieval.
2. Configuration: How would custom Getters be configured? It seems uniform across the suggested options that Getters might be configured by reading their configuration files upon startup. However, is it feasible to have runtime configurations, such as parameters passed directly from the node?
3. Telemetry Integration: Will the chosen option facilitate passing telemetry to the Getter constructor for metrics and trace collection? This aspect is crucial for system monitoring.

Addressing these questions will provide clarity on the most suitable approach for our goals.

[walldiss]

Great suggestion! Leveraging pluggable getters not only allows users to retrieve data from various storage implementations but also opens up possibilities for deeper integration within the DA network servers. By reworking protocols like shrex and shwap to rely on the Getter interface, nodes can serve data to the entire network based on custom getters. This means data can be fetched from one network and seamlessly relayed to another, enhancing the network's flexibility and interoperability.