Statistics on N-d grids--Calculation speed weird

[lkcao]

Hi there,
I have a large dataset of about 20 million samples. The task is simple: I want to get sum of a column.
The weird thing is like this: when I run codes below, the time estimate is 19 s.

but when I run sum command individually, it takes about 1s.

But, the steps before sum calculation takes only 0.3 s.....so I don't know what happens/what makes the difference between speed in the two figures.
Thoughts? solutions?

[JovanVeljanoski]

Hi,

First of all i moved this thread to the discussion section since this is not an issue/bug but a question.

Ok now about your question:

• First of all, do not use apply. You really do not need it here. Just write out the expressions, and you will likely get a nice speed-up
• In the first code block you shared you:
  • filter a dataframe
  • create 2 virtual column (the lines with apply).
  • Both of the above two points need to be actually executed for you to calculate the mean.
• The 2nd time you calculate the mean (the 2nd code block), the machine has already cached some parts of the dataset in ram, so execution is much faster the 2nd time around.

This is all in the basics of Vaex, and i really think you should at least quickly go through the tutorial to understand how Vaex works, since under the hood the behavior is different from other dataframe libraries.