trying to use sqlite filter

[Alisher-Nabiev]

hi,
i am trying to create a DB that compares incoming IP from netflow and compare it with the customer name on the DB

i instilled the filter :
npm install @pastash/filter_sqlite

and when i tried to run the command with the filter i got an error related to the filter module missing:
log1.txt

this is my conf:
conf.txt

any suggestions?

[lmangani]

You need to install paStash plugins as globals with npm install -g

[Alisher-Nabiev]

like this?
npm install -g @pastash/filter_sqlite

if yes it's not shown here:

[Alisher-Nabiev]

ok i got it running but i have another issue

i am trying to compare the port numbers from netflow with the port numbers name on the db and output it to elastic
i am now not seeing this on Elastic at all.

  sqlite {
    db => '/home/******/userdata_qa.db'
    query => 'SELECT service_name from port_list WHERE port_number=(?) limit 1;'
    source_field => 'l4_dst_port'
    target_field => 'service'
  }

db:

s```
qlite> .tables
port_list userdata
sqlite> PRAGMA table_info(port_list);
0|id|INTEGER|0||1
1|port_number|INTEGER|0||0
2|service_name|TEXT|1||0
sqlite> SELECT * FROM port_list;
1|53|dns
2|8883|mqtt
3|123|ntp
4|80|http
5|443|https
6|1194|ovpn

[lmangani]

Possibly a type issue. Try to force the input type to be the same format as the stored column you're checking against.

[yaniv999]

Hello lmangani

I have changed the format in the sqlite table to be the same and it's started working.

but now it's sending the same value all the time, like it takes the first source_field => 'l4_dst_port' and use it all the time.

log looks lie that:

all results are 443 even there are a lot of other port numbers

Do we need to change something in the config?

sqlite {
db => '/home/******/userdata_qa.db'
query => 'SELECT service_name from port_list WHERE port_number=(?) limit 1;'
source_field => 'l4_dst_port'
target_field => 'service'
}

[Alisher-Nabiev]

Hi Lorenzo and thank you for your quick reply.

1. you had a filter name ipproto that added the protocol name from the field protocol which holds the protocol number (TCP, UDP ext..). I don't see it anymore now on your documentation.

2. I added the config under filter :

ip_proto {
source_field => l4_dst_port
target_field => proto
custom => "8883:mqtt,1194:ovpn"
}

but now nothing is logged into the elastic, so we had to comment it out

(p.s. - Maybe it's better to name this filter ip_service since the l4_dst_port is the service that is used)

[lmangani]

@Alisher-Nabiev you need to attach the logs from pastash or we won't be able to assist at all
consider adding an stdout {} block in the output to see the messages being sent out in their entirety

[Alisher-Nabiev]

here are the pm2 log whene we trying to start the process
pastash-error.txt

also conf file:
apps:

• script : /usr/local/bin/pastash
  name: 'pastash-logs'
  args: ['--config_file=/home/monogoto/pastash.conf']

pastash-conf.txt

[lmangani]

Loading npm module... @pastash/filter_ip_proto

The module cannot be loaded. Have you installed the plugin with -g global option?

[Alisher-Nabiev]

yes
root:/home/****# npm list -g
/usr/local/lib
├── @pastash/[email protected]
├── @pastash/[email protected]
├── @pastash/[email protected]
├── @pastash/[email protected]
├── [email protected]
├── [email protected]
└── [email protected]

[lmangani]

I see the issue. Install as follows and try again:

npm install -g @pastash/filter_ip_proto

note: make sure you're using version 1.0.2 or higher

[Alisher-Nabiev]

still same
pastash-error.txt

/usr/local/lib
├── @pastash/[email protected]
├── @pastash/[email protected]
├── @pastash/[email protected]
├── @pastash/[email protected]
├── [email protected]
├── [email protected]
└── [email protected]

[lmangani]

If you read my comment above

note: make sure you're using version 1.0.2 or higher

[Alisher-Nabiev]

@lmangani same error

[Alisher-Nabiev]

Loading npm module... @pastash/filter_ip_proto
[Wed, 19 Jun 2024 10:50:41 GMT] INFO Initializing filter ip_proto
[Wed, 19 Jun 2024 10:50:41 GMT] ERROR Unable to load urls from command line
[Wed, 19 Jun 2024 10:50:41 GMT] ERROR TypeError: ipProto.push is not a function
at /usr/local/lib/node_modules/@pastash/filter_ip_proto/filter_ip_proto.js:55:17
at Array.forEach ()
at FilterIPProto.start (/usr/local/lib/node_modules/@pastash/filter_ip_proto/filter_ip_proto.js:53:11)
at FilterIPProto. (/usr/local/lib/node_modules/@pastash/pastash/lib/lib/base_component.js:213:12)
at /usr/local/lib/node_modules/@pastash/pastash/node_modules/async/dist/async.js:3113:16
at replenish (/usr/local/lib/node_modules/@pastash/pastash/node_modules/async/dist/async.js:1014:17)
at /usr/local/lib/node_modules/@pastash/pastash/node_modules/async/dist/async.js:1019:9
at eachLimit$1 (/usr/local/lib/node_modules/@pastash/pastash/node_modules/async/dist/async.js:3199:24)
at Object. (/usr/local/lib/node_modules/@pastash/pastash/node_modules/async/dist/async.js:1049:16)
at FilterIPProto. (/usr/local/lib/node_modules/@pastash/pastash/lib/lib/base_component.js:212:11)

[Alisher-Nabiev]

hi lorenzo, any update? as i write on git hub the error is still the same after updating the version.

…

On Wed, 19 Jun 2024 at 13:47, Lorenzo Mangani ***@***.***> wrote: If you read my comment above note: make sure you're using version 1.0.2 or higher — Reply to this email directly, view it on GitHub <#209 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AICGB3PM54KS3KDLXC3OVRDZIFOTNAVCNFSM6AAAAABIDMOBM6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNZYGM3DEOBUGU> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[lmangani]

@pastash/[email protected]