[NETOBSERV-44] Consumming goflow2 as a library.

[OlivierCazade]

Using goflow2 as a library to directly receive netflow event with the kube-enricher

[jotak]

Just a first-pass superficial review, I still want to review & test more in deep, thanks @OlivierCazade for this PR!

[jotak]

The linter shows an issue with a lock being copied on RenderMessage (the issue was actually there prior to this PR). The state in goflow's FlowMessage seems to contain a lock that shouldn't be copied. Passing it as a reference in RenderMessage should fix it.

There's also a second lint error about deprecated github.com/golang/protobuf/proto

[jotak]

Looks good to me for a first approach (there's just the lint error that should be fixed).
As we discussed, there is still protobuf unmarshalling in the process, that perhaps we can try getting rid of in a next iteration.

For a follow-up task, we should probably be able to avoid the unnecessary PB step that happens between Formatter stage and Transport/Enrichment.

If I understand correctly, goflow2 works with a Formatter and a Transporter, and we previously had:
(In goflow) Formatter => Transporter => (in kube-enricher) Enricher => stdout

Now we basically have:
(In kube-enricher) Formatter => Transporter => Enricher => stdout

Next step could be to try having something like:
(In kube-enricher) Enricher => Formatter => Transporter
so that enrichment takes place before there's any encoding, working from goflow's go model directly?

It's unclear yet to me if the Formatter and Transporter stages will still be around when we'll integrate the loki client, or if they will be like no-ops... (it's just an implementation detail anyway).

[OlivierCazade]

Yes we could try to implement the enricher as a goflow2 Formatter.

Goflow2 API use []bytes to pass data between the formatter and the transporter so if we want to enrich this datas without having to unmarshal them, we have to do it before passing them to the transporter.

What is not clear to me is if we want to use this enricher only with goflow2, or if other collectors like the eBPF are also going to use it. Keeping the Formatter and Transporter stages will make interactions with goflow2 easier, but I don't know how this is going to work if we also want to use another source.

[jotak]

I opened https://issues.redhat.com/browse/NETOBSERV-64 for a follow-up

[mariomac]

Overall LGTM. Just dropped some comments about few "styling" advices. Feel free to implement or push back them.

[jotak]

this will surely change in the future if we want to include vendors in sources as Mario did for the loki client. But we can do that as a follow-up.

[jotak]

Just noting / realizing that if we stick to the redhat-supported tool chain we have to use this older version of go. Generics will wait :)

[jotak]

lgtm but if @mariomac you can take another look, as you wrote most of the comments