Unexpected memory usage

[Brad-IMS]

I have a Blazor WASM .Net7 app using Csla 7 to load datasets into Radzen grids. I am using virtualization with the grids. I am using ADO.Net with stored procedures for data access. The app is currently hosted on a test server with 2 cores and 8GB of RAM.

I am encountering a problem only when accessing large datasets. The access appears to go into a memory-consuming loop on the server and eventually times out. As an example, I have a dataset with 514,417 records occupying 106MB of storage. When I try to fetch that dataset, the IIS Worker Process grabs all the available CPU and steadily increasing amounts of RAM, reaching 5GB by the time it times out.

Has anyone encountered something similar to this or have any idea why it is happening? Suggestions for debugging?

[hurcane]

500,000 rows is a lot of data to show in the web browser. Without knowing your business needs, the first question I would be asking is whether we can show this data in smaller pages that don't overwhelm memory.

I would use a memory profiler to see what kind of data is being generated. Since you're working with a third-party grid control, I would consider the possibility that the control is generating a lot of its own data for keeping track of the data you supply to it. Maybe that control has not been intentionally designed to handle large amounts of data.

[Brad-IMS]

I am using the Virtualization feature of Blazor, so only a screen or two worth of records is actually passed to the web browser. The grid control has to cooperate with the server logic as to what block of records to pass down, but I would assume the calculations at either end would not consume a great deal of memory.

[Bowman74]

I wouldn't consider this a CSLA issue per se. As @hurcane mentions, 500k records is a lot of records to try and load into memory.

It does appear the Radzen grid handles paging (fetching a small "page" of data at a time as the user looks through the grid)

https://www.radzen.com/documentation/blazor/custom-datagrid-paging/

CSLA can do paging, but there is some manual coding. You'd have to do something like add page size and current position as parameters to your fetch and pass those through to the data portal. The in the data access layer use SQL syntax for OFFSET and FETCH NEXT to get the specified page of data and only the specified page.

Having said all that, here is the question I have. Is this UI design usable? If there are 500k records in a grid who is going to start scrolling through that to find the records they want? It seems from a design perspective you first may want to ask the user some information about what information in the table they are looking for and only fetch and show them those records.

[Brad-IMS]

I am familiar with Radzen's paging feature, but Radzen, like most web control vendors, have implemented the more user-friendly built-in paging ability of Blazor's Virtualization capability, introduced in .Net5 (https://www.daveabrock.com/2020/10/20/blazor-component-virtualization/). I assume that with Virtualization the Radzen grid is working with .Net code, rather than Radzen code, at the server end to determine what block of pages to send down to the browser.

Radzen's paging function also does not work well with their grid's sorting and grouping capabilities, whereas it is seamless with Virtualization.

As for the UI design, every grid has a search function where the user can build a search string to only bring down a subset of the data, plus some prebuilt searches (ex. Get the Transactions for a specified Gift Card). Some clients want to see the entire table, however, and the customer is always right, as they say.

It is a lot of records, as you point out, but if the entire table only occupies 106MB of storage. What is it doing with that 5GB of memory?

[hurcane]

The only way to know what is happening is by using a memory profiler on the process. Once you have an understanding of what data is taking up so much space, you can work out a plan to avoid that issue. Without a memory profiler, you can only guess what might be the cause, and those guesses are frequently wrong.

[Bowman74]

You definitely need to put on a memory profiler. There could be some infinite recursion or even non infinite recursion. There is also the question of what you are loading that data into. Is it a simple array of strings or something more complex where each row has an overhead that is much greater that the base storage size. For example, are you loading CSLA objects and having them execute business rules as they are loading.

As far as the customer always being right, that is definitely not a good idea. The customer hires experts partly because those experts can tell them if their ideas are a good fit for the technical limitations and UX realities. I've working in consulting for over 30 years and I've defiantly told my clients no something is not a good idea and explained why and sometimes even that what they are asking for isn't possible. They can ask for seven red lines to be drawn in invisible ink, all perpendicular to each other, all they want, it doesn't mean I'm going to try and do it. :)