Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEATURE] Implement Disk-Based Buffering with Outbox Pattern Support for Elastic.Serilog.Sinks #423

Open
HaikAsatryan opened this issue Aug 21, 2024 · 1 comment
Labels
enhancement New feature or request

Comments

@HaikAsatryan
Copy link

I would like to request a feature enhancement for the Elastic.Serilog.Sinks NuGet package. The current implementation lacks disk-based buffering, which is crucial for ensuring log durability, especially in distributed applications.

Proposed Feature:

  • Disk-Based Buffering: Implement disk-based buffering (as opposed to RAM) to enhance persistence and avoid potential data loss during high-throughput scenarios or application crashes.

  • Outbox Pattern: Integrate the outbox pattern to ensure maximum durability when sending logs to Elasticsearch. This will prevent the need for additional tools like Filebeat for log durability, simplifying the logging pipeline.

Importance:

  • Distributed Applications: Given the rise of distributed systems, it is essential to consider scenarios where multiple instances of an application might be logging concurrently. Ensuring that logs are reliably captured and forwarded to Elasticsearch is critical to maintaining observability in such environments.

By incorporating these features, the Elastic.Serilog.Sinks package can significantly improve its robustness and become a more reliable logging solution in distributed and high-throughput applications.

Thank you for considering this request!

@HaikAsatryan HaikAsatryan added the enhancement New feature or request label Aug 21, 2024
@guillaume86
Copy link

I have the same durability requirement and trying to work around this as recommended with filebeat is sub-optimal,
it's missing the logs-dotnet template bootstrapping and I get some fields conflicts if I just use the default filebeat index:

Image

A durable mode would really be appreciated, and baring that, some easy way to bootstrap the logs-dotnet data stream from filebeat would fill the void in my case but I guess it's out of scope.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants