Long duration for de-/serialization searchUser-Result

[sschum]

In our project we have many (round about 10,000) huge user data sets. They have many extension field values. We use the "searchUsers"-API to check if a user has access rights to specific parts of our software. So we have many rules. These rules are just search queries.

We have the feeling that the de-/serialization takes a lot of time. For example: if we use the searchUsers query directly against osiam (http-request) we got an answer in ~20sec. But if we use the searchUsers (same query, same parameters) against the connector4java we got an answer in ~30sec. For me it suggests that the deserialization took about ~10sec.

BTW: The query above delivered ~2,000 Users. But not the whole user data set! Only the externalId and userName.

[tkrille]

Hello @sschum,

we are aware of this issue, but until now no one had this amount of data to be deserialized. I think we can easily optimize some things here, and then you can check if the situation got better. Please be aware that it's not a good idea to retrieve that much data via a search request. First of all, we're doing many Joins to retrieve that data from the database, but you also have to transmit all this stuff on the wire. It's most often better to either

1. restrict the search to 1 entity, if you just want to know something in correlation to that entity
2. page the result set

In fact, early versions of OSIAM didn't allow you to retrieve more than 100 resources. I don't know why we ever changed that value to infinity. Can you explain your use case in more detail? Maybe we can find another, more suited solution.

BTW: The query above delivered ~2,000 Users. But not the whole user data set! Only the externalId and userName.

Please create a new issue for that including the query string or some example code that reproduces the issue (preferably in the connector4java-integration-tests).

[sschum]

Why we do so: At the first we have implemented a restricted search:
''(<complicated expression>) AND externalId eq <userUUID>'''
So we got no more than one result. But in our project we must check an user against many rules. That causes that the whole request takes a lot of time. So we implemented a cache. This cache contains all uuids for a specific rule. If a user will be check against a rule, we will check the cache and no more osiam directly. Thats the reason why we get all users for one rule.

For us it doesn't matter if we get all users in one query or less users in many queries. Because the initialization of the cache will get all users either way.

Please create a new issue for that including the query string or some example code that reproduces the issue (preferably in the connector4java-integration-tests).

This is not a bug, its a feature ;) Because we send the fields explicity:

queryBuilder.setAttributes("userName, externalId");

I thought that is a nice-to-have information for this issue.

[tkrille]

Please create a new issue for that including the query string or some example code that reproduces the issue (preferably in the connector4java-integration-tests).

This is not a bug, its a feature ;) Because we send the fields explicity:

queryBuilder.setAttributes("userName, externalId");

I thought that is a nice-to-have information for this issue.

Alright, I misunderstood you :)

Why we do so: At the first we have implemented a restricted search:
''(<complicated expression>) AND externalId eq <userUUID>'''
So we got no more than one result. But in our project we must check an user against many rules. That causes that the whole request takes a lot of time. So we implemented a cache. This cache contains all uuids for a specific rule. If a user will be check against a rule, we will check the cache and no more osiam directly. Thats the reason why we get all users for one rule.

You could also store the User in their session (actually, you cannot, because of osiam/scim-schema#129, but you could store a copied subset of the data in the session), including their groups and extension fields, and just do the checks in place with out ever needing to search for all that stuff. I assume, that you need these checks to calculate the permissions a user has. This way you can also, from time to time, re-fetch the User and thereby update the permissions.

[tkrille]

Fixed with #151 (and subsequently with osiam/scim-schema#136)