Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Question about possibility of monitoring multiple tenant domains #26

Open
ipninichuck opened this issue Dec 10, 2019 · 14 comments
Open
Labels
enhancement New feature or request

Comments

@ipninichuck
Copy link

For my current project I am required to pull logs from multiple tenant domains and output each to a separate index. My current solution is running an instance of o365beat per domain that I am pulling logs from. Just curious if I can somehow configure the beat to pull from each domain and use conditionals to send the output to the different indices. In my experience with beats in the past this was done with multiple prospectors, but not sure if that is possible with o365beat. If it is not I will just continue running multiple instances.

@chris-counteractive
Copy link
Collaborator

I hadn't thought of this use-case, thanks for bringing it up, I can see why it'd be helpful. It's not currently supported, but I'll tag this as an enhancement request - until we work through the implications you can of course run multiple instances (you'll have to fiddle with the service registration(s), auto-starting, logging, etc., to make it all play nice, but it sounds like you've worked through that before).

To implement this I'll have to break out the configs for the various tenancies, but it's doable ... I'll target it for the 2.0 release, with any other breaking changes. Thanks for the suggestion!

@chris-counteractive chris-counteractive added the enhancement New feature or request label Dec 10, 2019
@GenCr
Copy link

GenCr commented Dec 24, 2019

@ipninichuck I also have a similar use case except my data can all go into the same index.

May I ask how you configured multiple instances of the beat to pull data from multiple tenants?

@ipninichuck
Copy link
Author

ipninichuck commented Dec 24, 2019 via email

@ipninichuck
Copy link
Author

ipninichuck commented Dec 24, 2019 via email

@GenCr
Copy link

GenCr commented Dec 24, 2019

That's awesome, thanks! I've managed to copy the service and configure for the second tenant with the help of your script. The two instances run alongside each other successfully.

Looking forward to having this feature supported without having to run multiple instances.

@scaruso
Copy link

scaruso commented Oct 8, 2020

Hi @ipninichuck ,

I follow the steps in your script manually and I run this:

o365beat -e -c /etc/itmx-o365beat/o365beat.yml -path.config /etc/itmx-o365beat -path.data /var/lib/itmx-o365beat -path.logs /var/log/itmx-o365beat

but I get this ERROR:

2020-10-08T16:04:56.431Z ERROR instance/beat.go:916 Exiting: non-200 status during api request.
newly enabled or newly subscribed feeds can take 12 hours or more to provide data.
confirm audit log searching is enabled for the target tenancy (https://docs.microsoft.com/en-us/microsoft-365/compliance/turn-audit-log-search-on-or-off#turn-on-audit-log-search).
req: &{POST https://manage.office.com/api/v1.0/47d8db9b-4dd3-4ab1-8dea-4892453bf581/activity/feed/subscriptions/start?PublisherIdentifier=47d8db9b-4dd3-4ab1-8dea-4892453bf581&contentType=Audit.AzureActiveDirectory HTTP/1.1 1 1 map[Authorization:[Bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsIng1dCI6ImtnMkxZczJUMENUaklmajRydDZKSXluZW4zOCIsImtpZCI6ImtnMkxZczJUMENUaklmajRydDZKSXluZW4zOCJ9.eyJhdWQiOiJodHRwczovL21hbmFnZS5vZmZpY2UuY29tIiwiaXNzIjoiaHR0cHM6Ly9zdHMud2luZG93cy5uZXQvNDdkOGRiOWItNGRkMy00YWIxLThkZWEtNDg5MjQ1M2JmNTgxLyIsImlhdCI6MTYwMjE3Mjc5NSwibmJmIjoxNjAyMTcyNzk1LCJleHAiOjE2MDIxNzY2OTUsImFpbyI6IkUyUmdZTGl1TCtZajgyMzE3MEQrRXp2eTVRL3lBUUE9IiwiYXBwaWQiOiJmMjg4YzQ4Ni1hMjg3LTRkZTAtYWVkYy0yODk0M2EwZmI0NzAiLCJhcHBpZGFjciI6IjEiLCJpZHAiOiJodHRwczovL3N0cy53aW5kb3dzLm5ldC80N2Q4ZGI5Yi00ZGQzLTRhYjEtOGRlYS00ODkyNDUzYmY1ODEvIiwib2lkIjoiZjhjYWMxODEtMmI4My00NTE2LWI0ODQtNjExYjY0MTYyNDNhIiwicmgiOiIwLkFBQUFtOXZZUjlOTnNVcU42a2lTUlR2MWdZYkVpUEtIb3VCTnJ0d29sRG9QdEhCSEFBQS4iLCJyb2xlcyI6WyJBY3Rpdml0eUZlZWQuUmVhZERscCIsIlNlcnZpY2VIZWFsdGguUmVhZCJdLCJzdWIiOiJmOGNhYzE4MS0yYjgzLTQ1MTYtYjQ4NC02MTFiNjQxNjI0M2EiLCJ0aWQiOiI0N2Q4ZGI5Yi00ZGQzLTRhYjEtOGRlYS00ODkyNDUzYmY1ODEiLCJ1dGkiOiJfRk9BNVUtSlJVMnBYNXdlQzdKeUFBIiwidmVyIjoiMS4wIn0.bWBoyHzIiLC_g-wdNABq9Y3VfZNWaajqNsiPKinq7tlGjDSFrJUtncN5FhF204gMJyLizww8kZ09URpH7fh2vhvQqbsJj7XYB-aqxSGnsHvcexfWYgH7ENP_w4B-UK2jB6whkg0jmE8XgPwHjZU061kGRFUJhTcAhd9jR9wJMapbISts-SzkTJzyzkfNXocoChHxp_z51q6HkK1zDt2JTMAHCgS-GkioBZWxUKKJUx_ZKIo99nqhdSuF295zV1QweSDxVjn6QATUXCINX8xTrXVCsPWCx2EXusDskshAJWTh9VsZfHArsz0vJBigFNyd5otN9EHNkZd-0vochZ3IQw]] {} 0x13fae20 0 [] false manage.office.com map[] map[] map[] }
res: &{401 Unauthorized 401 HTTP/2.0 2 0 map[Cache-Control:[no-cache] Content-Length:[124] Content-Type:[application/json; charset=utf-8] Date:[Thu, 08 Oct 2020 16:04:56 GMT] Expires:[-1] Pragma:[no-cache] Server:[Microsoft-IIS/10.0] Www-Authenticate:[Bearer] X-Aspnet-Version:[4.0.30319] X-Powered-By:[ASP.NET]] 0xc0001fe120 124 [] false false map[] 0xc0004ec700 0xc0000c28f0}
{"error":{"code":"AF10001","message":"The permission set () sent in the request does not include the expected permission."}}
Exiting: non-200 status during api request.
newly enabled or newly subscribed feeds can take 12 hours or more to provide data.
confirm audit log searching is enabled for the target tenancy (https://docs.microsoft.com/en-us/microsoft-365/compliance/turn-audit-log-search-on-or-off#turn-on-audit-log-search).
req: &{POST https://manage.office.com/api/v1.0/47d8db9b-4dd3-4ab1-8dea-4892453bf581/activity/feed/subscriptions/start?PublisherIdentifier=47d8db9b-4dd3-4ab1-8dea-4892453bf581&contentType=Audit.AzureActiveDirectory HTTP/1.1 1 1 map[Authorization:[Bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsIng1dCI6ImtnMkxZczJUMENUaklmajRydDZKSXluZW4zOCIsImtpZCI6ImtnMkxZczJUMENUaklmajRydDZKSXluZW4zOCJ9.eyJhdWQiOiJodHRwczovL21hbmFnZS5vZmZpY2UuY29tIiwiaXNzIjoiaHR0cHM6Ly9zdHMud2luZG93cy5uZXQvNDdkOGRiOWItNGRkMy00YWIxLThkZWEtNDg5MjQ1M2JmNTgxLyIsImlhdCI6MTYwMjE3Mjc5NSwibmJmIjoxNjAyMTcyNzk1LCJleHAiOjE2MDIxNzY2OTUsImFpbyI6IkUyUmdZTGl1TCtZajgyMzE3MEQrRXp2eTVRL3lBUUE9IiwiYXBwaWQiOiJmMjg4YzQ4Ni1hMjg3LTRkZTAtYWVkYy0yODk0M2EwZmI0NzAiLCJhcHBpZGFjciI6IjEiLCJpZHAiOiJodHRwczovL3N0cy53aW5kb3dzLm5ldC80N2Q4ZGI5Yi00ZGQzLTRhYjEtOGRlYS00ODkyNDUzYmY1ODEvIiwib2lkIjoiZjhjYWMxODEtMmI4My00NTE2LWI0ODQtNjExYjY0MTYyNDNhIiwicmgiOiIwLkFBQUFtOXZZUjlOTnNVcU42a2lTUlR2MWdZYkVpUEtIb3VCTnJ0d29sRG9QdEhCSEFBQS4iLCJyb2xlcyI6WyJBY3Rpdml0eUZlZWQuUmVhZERscCIsIlNlcnZpY2VIZWFsdGguUmVhZCJdLCJzdWIiOiJmOGNhYzE4MS0yYjgzLTQ1MTYtYjQ4NC02MTFiNjQxNjI0M2EiLCJ0aWQiOiI0N2Q4ZGI5Yi00ZGQzLTRhYjEtOGRlYS00ODkyNDUzYmY1ODEiLCJ1dGkiOiJfRk9BNVUtSlJVMnBYNXdlQzdKeUFBIiwidmVyIjoiMS4wIn0.bWBoyHzIiLC_g-wdNABq9Y3VfZNWaajqNsiPKinq7tlGjDSFrJUtncN5FhF204gMJyLizww8kZ09URpH7fh2vhvQqbsJj7XYB-aqxSGnsHvcexfWYgH7ENP_w4B-UK2jB6whkg0jmE8XgPwHjZU061kGRFUJhTcAhd9jR9wJMapbISts-SzkTJzyzkfNXocoChHxp_z51q6HkK1zDt2JTMAHCgS-GkioBZWxUKKJUx_ZKIo99nqhdSuF295zV1QweSDxVjn6QATUXCINX8xTrXVCsPWCx2EXusDskshAJWTh9VsZfHArsz0vJBigFNyd5otN9EHNkZd-0vochZ3IQw]] {} 0x13fae20 0 [] false manage.office.com map[] map[] map[] }
res: &{401 Unauthorized 401 HTTP/2.0 2 0 map[Cache-Control:[no-cache] Content-Length:[124] Content-Type:[application/json; charset=utf-8] Date:[Thu, 08 Oct 2020 16:04:56 GMT] Expires:[-1] Pragma:[no-cache] Server:[Microsoft-IIS/10.0] Www-Authenticate:[Bearer] X-Aspnet-Version:[4.0.30319] X-Powered-By:[ASP.NET]] 0xc0001fe120 124 [] false false map[] 0xc0004ec700 0xc0000c28f0}
{"error":{"code":"AF10001","message":"The permission set () sent in the request does not include the expected permission."}}

I am not very expert at this, can you help me to understand where it stucks?

Thank you,
Sara

@ipninichuck
Copy link
Author

ipninichuck commented Oct 8, 2020 via email

@scaruso
Copy link

scaruso commented Oct 8, 2020

Hello, From the error message, it looks like o365 is replying that the application is not authorized. Did you complete the previous steps in the beat setup process of registering the app in Active Directory and giving it the needed permissions. I believe they are listed in the documentation on the Github page for the beat.

Hi,

thank you for your reply.

I don't have direct access to o365 management. I have to configure a log collector to receive logs from o365.
The customer gives me all required information that I put in o365beat.yml (tenant, client id, directory id, secret)

So do you think that this is not an error depending on running two instances of o365 beat?

@ipninichuck
Copy link
Author

ipninichuck commented Oct 8, 2020 via email

@scaruso
Copy link

scaruso commented Oct 9, 2020

Hi,

the client already provide us with tenant id, app id, directory id and secret. So I suppose that they already accomplished to your suggestion. Or am I wrong?

I ran

o365beat -e -c /etc/itmx-o365beat/o365beat.yml -path.config /etc/itmx-o365beat -path.data /var/lib/itmx-o365beat -path.logs /var/log/itmx-o365beat

but as output of

ps -ax | grep o365beat

I can't see two line as result but only one regarding the older istance.

Is that a good sign?

@ipninichuck
Copy link
Author

ipninichuck commented Oct 11, 2020 via email

@Vetpeet
Copy link

Vetpeet commented Mar 1, 2021

Hi @chris-counteractive,

Is the multi tenant support enhancement likely to be added in the near future?

We are currently using the o365 module within filebeat with multi tenant support but as with many others we are looking to move to AWS open distro and the o365 module is currently not included with the oss compatible version of filebeat.

o365beat will fill this gap for us but we do have quite a few tenancies that we currently monitor.

Thanks!

@chris-counteractive
Copy link
Collaborator

@Vetpeet thanks for the question! Short answer: we hadn't planned to add any features to o365beat since the "official" filebeat 365 module dropped in 7.7.0. Even though the o365 module is under x-pack, I don't know that there's any restriction in filebeat that requires a paid license for any specific modules. That is, I don't think there's any reason you wouldn't be able to ship to AWS-flavored elasticsearch, right? Does filebeat complain if you're trying to send to an oss-compatible ES instance? I've honestly not tried it.

And even if it did, it's might be a more reliable workaround to use the Elastic-licensed filebeat to dump to a jsonl file that you can re-ship with unencumbered filebeat. Certainly kludgy and a bit wasteful, but workable?

If there's an angle on this that I'm not seeing I'm definitely happy to re-assess and perhaps try to get back to feature-parity, definitely not opposed - it just didn't seem to make much sense when the elastic-sponsored filebeat gets most people where they need to go.

@Vetpeet
Copy link

Vetpeet commented Mar 12, 2021

@chris-counteractive thanks for the reply,
The AWS-forked elasticsearch is still under the Apache 2.0 license, the standard Filebeat is now under the new Elastic License, and as such the standard Filebeat will not work with the AWS-forked elasticsearch. There is a Filebeat version (Filebeat -oss) https://www.elastic.co/downloads/beats/filebeat-oss still under the Apache 2.0 license that will work with the AWS-fork, but not all the standard modules are included, I will add screenshot of included modules, hence why we had to turn to o365beat as an alternative in a nutshell the compatible filebeat distro for the AWS-forked elasticsearch don't include the o365 module.

Thanks.

Capture

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

5 participants