Performance Improvement

[BenoitJ76]

Hello,

First, thanks for your wonderful lggr!

Currently, I use lggr with an hundred equipments. I have added a month table and a year table. I also added a live page with specific equipments (Indeed with the number of equipments I have up to 1000 entries per second in normal conditions).

Everything works very well.

However, the pages are long to load. (Once loaded the data are fluid).

In addition, if I exceed the week (in index.php), the stats page no longer displays the graphs (perhaps I forgot a configuration in stats.php concerning the tables month and year).

For information, my CPU does not exceed 20-30% and I have 4GB of ram.

Any ideas to improve performance and solve the stats problem ?

Thank you for your help

[kkretsch]

Thanks for using my lggr tool.

If the stats page doesn't work with loads of data it might be a timeout problem of php. Standard might be 30 secs of runtime, if a job exceeds that time the resulting page is empty which might be the cause here.
You should see some warnings about that in the log files of php or apache.

If timing itself gets too slow, have a look at the database and perhaps the indexing of some columns.

My live server has 4GB RAM too and only one virtual cpu core as a virtual machine. Which is enough most of the time. I have aroung 4 million loglines from 12 servers, not that much.

How fast is the daily purge of old entries? I have about 80k of messages per day to delete which takes about 12 secs.

[BenoitJ76]

Thank you for your reply.

I have 17,400,097 events in my DB and my oldest entry is 2017-01-16 17:11:31.

Regarding the page stats.php I have this error: PHP Notice: Undefined variable: searchvalueprog in /var/www/lggr/tpl/nav.inc.php on line 21, referer: http: //IPAddrress/live.php

But nothing about timeout, what about a problem from Redis?

How can I see the purge time? However I changed this parameter: function purgeOldMessages ($ maxage = 672) to purgeOldMessages function ($ maxage = 8064) to keep logs 1 year.

Can you explain me how the archive function works?

[BenoitJ76]

"In addition, if I exceed the week (in index.php), the stats page no longer displays the graphs (perhaps I forgot a configuration in stats.php concerning the tables month and year)."

-> Fixed with a trim because of special character in the dataCloudWords

About the loading time of the pages, it's because of the count function. Is this mandatory?

[kkretsch]

The statistics isn't mandatory, as isn't the count function. It just displays some usage counts which might be of interest. I myself often have a look about the messages per hour chart which normaly should be the same every hour. In case it goes down my logging might be interrupted, in case it goes up some system might be running awild.

If you want to make the statistic optional just add some configure variable to the source and I will happily integrate any additions. The same for performanca optimizsations. Perhaps not every index on that table is active?