Fail to allocate bitmap

[pkubryk]

Hello,

I have the following code that systematically ends up in the following error after 396 charts are saved.

**Fail to allocate bitmap

Process finished with exit code -2147483645**

I tried several combinations including with plt.close all or plt.close of the figure.
I tried with jpeg and png extensions
Although the memory does not seem to be cleaned up ( linear growth ) it does not seem to be the reason since it can fail with plenty of memory left to allocate.
I checked that it was not some kind of murky limitation of the disk, by moving the creation of the file to an NVME disk.
Also updated my graphic card drivers, because why not.
As you see I also tried to reload the module every time in the loop ( desperate move :) )

Really not sure what I am doing wrong here

I updated mplfinance to 0.12.7a17, matplotlib to 3.4.1.
Python version 3.9.2

for index in df.index :
    try :
        reload(mplfinance)
        i=i+1
      
        titleString =(df.iloc[index]['TimeStamp']).strftime('%Y-%m-%d')+' '+(df.iloc[index]['Symbol'])
        print ('processing  '+   str(index)+ ' ' + titleString)
        minRange = df.iloc[index]['ChainPosition'] - chartingDayAddition
        maxRange = df.iloc[index]['ChainPosition'] + tradeDuration + chartingDayAddition

      
        prices = pd.DataFrame(get1dDB().aggregate(pipeline=[
        {
                '$match' : {
                    'Symbol' : df.iloc[index]['Symbol'] ,
                    'ChainPosition': {
                        '$gte': minRange.item() ,
                        '$lte': maxRange.item()
                             }
                        }
         }]))


        BuyLine = prices.loc[prices['ChainPosition'] == df.iloc[index]['ChainPosition']]
        SellLine = prices.loc[prices['ChainPosition'] == (df.iloc[index]['ChainPosition'] + tradeDuration)]

        vls =[(BuyLine.TimeStamp).iloc[0].strftime('%Y-%m-%d'),(SellLine.TimeStamp).iloc[0].strftime('%Y-%m-%d')]

        prices.set_index('TimeStamp', inplace=True)
        fig = mplfinance.plot( prices, type='candle',volume=True, title=titleString, savefig=dict(fname=detailsPath+titleString+'.jpg', dpi=150), closefig=True)

        plt.close('all')
    except :
        print ('error')

[pkubryk]

Something very odd happened when I tried to debug in mplfinance.plot on the iteration that usually fails -> it went past it.
It gives me the impression there is a race condition somewhere.

Edit : this is consistently reproducible on my case. going in debug and doing a step by step works. After that, the plot window remains opened ( although blank ) for each subsequent chart,. The save is still happening. It then fails on bitmap number 740

[DanielGoldfarb]

The are several items of interest in your code and in your report of the issue:

1. First and foremost, you say you get a "**Fail to allocate bitmap" error. I would presume that there would be some kind of a Python Traceback as a result of that exception. Please provide the complete traceback, and/or other entire error messages (that may have been output before the application ended) in order to be able to know exactly where in the code "**Fail to allocate bitmap" is happening.
2. I don't understand your comment " After that, the plot window remains opened" ... since you are doing savefig no plot window should be opening.
3. Please note that mplfinance.plot() does not return anything, unless you set kwarg returnfig=True, so there is no point to setting fig = mplfinance.plot(...). Also, if you do set returnfig=True then it will return both the Figure and all Axes objects. (Click here for more information on gaining access to mplfinance's Figure and Axes objects).
4. I doubt there is a race condition going on (unless your are doing something with threads or multiprocess), however it is a plausible suspicion so, for now, I won't rule it out as a possibility.
5. It's very difficult for me to see what you are trying to accomplish with your code since you have not provided all of the code, nor the data, nor even a few of the plots that were saved (and perhaps the last two or three just before the failure). I would strongly suggest that you try to reproduce the error with very a simple, possibly hard-coded, data set, or simple data set in a csv file, and provide a complete code and data set with which anyone should be able to reproduce the error. This process of trying to reproduce the error in a simple manner may even lead you to the answer on its own.
6. Definitely you should not need to reload the module. I would suggest your reverse that "desperate move :)". Again, try to gradually make the code and the data simpler and simpler, while continuing to reproduce the problem. Either you will get to a point where the problem goes away, or you will be able to provide a simple reproducible code set so that others can help debug.
7. Please focus on the above first, but if all else fails I have found that explicitly deleting the Figure object sometimes works better than calling close. You can do this by not doing savefig nor closefig in mpf.plot(). Rather set returnfig=True and then use Figure.savefig() to save the plot, followed by deleting the Figure. The code might look something like this:

for ix in df.index:
    ...
    fig, axlist = mpf.plot(data, type='candle',volume=True,title=titleString,returnfig=True)
    fig.savefig(detailsPath+titleString+'.jpg',dpi=150)  
    # notice in `fig.savefig()` that `fname` kwarg is ***not*** necessary, just pass file name as first argument
    del fig

If you need more help, please provide the information requested.
Thanks for reporting this issue. Seems an interesting/challenging problem to solve.
All the best. --Daniel

[pkubryk]

Daniel,

Thank you for you detailed answer.
Below some feedback

1- I could not find any. I tried several things to try to catch the exception or skip it for that particular item without success, ( see the except at the end of the code snippet). That makes me think it is a very low level error. Is there a better way to capture it ?

2- You are absolutely right, the plot does not remain open unless I run the code in debug. As mentioned, in debug I manage to get past the item causing problem ( without crashing the process), but after that the plot windows remain open. see screenshots below

I actually figured out that no matter where I place the breakpoint , and no matter which iteration of the loop I am in, it stops closing the figure if I break.

Sometimes the plots are empty, sometimes not :

If I do a step by step in the close method, the closing does work and the figure gets destroyed in

If I play with the debugger stopping randomly the process not through a breakpoint but through the pause button in pycharm, I see that pymongo does use different threads. I don't see any relationship between that and the memory problem, the chart not closing or the save crashing at a specific point in time.

3-point taken.

5/6 - agree. for now I am having a difficulty to reduce it to a small test case. It will take time.

7- I tried this workaround, with the same result

 fig, ax = mplfinance.plot(prices, type='candle', volume=True, title=titleString,
                                  vlines=dict(vlines=vls, colors='b'),
                                  returnfig=True)
        fig.savefig(detailsPath+titleString+'.jpg',dpi=150)
        del fig

However I got the following warning after the 20th plot . that warning did not prevent the process to continue the loop

C:\Users\Pierre\AppData\Local\Programs\Python\Python39\lib\site-packages\mplfinance\plotting.py:351: RuntimeWarning: More than 20 figures have been opened. Figures created through the pyplot interface (`matplotlib.pyplot.figure`) are retained until explicitly closed and may consume too much memory. (To control this warning, see the rcParam `figure.max_open_warning`).
  fig = plt.figure()

new developments to the story
I tried to skip item 369

        if i == 369:
            print('369')
            print (prices)
            print (vls)
            continue

It fails with item 370.
This makes me think this is linked to the state of the memory.
to make sure it was not the previous item I skipped item 368 too, it then fails on item 371

Also adding that my memory load looks like this ( red bubble is the failure point) - it really looks like the memory is not getting freed. I am having that memory load both with and without the workaround:

Finding a way to prevent this might be the key

[pkubryk]

If I break on the plot call, the charts that were kept opened when entering the previous breakpoint are all being closed and the memory gets freed. When the windows gets closed in bulk, other windows appear in the background as if they were there all the time but not getting displayed - see screenshot.
Note that the memory allocated before I hit the breakpoint first ( so before I start seeing charts staying opened) also seem to get freed. This is why entering in debug was enabling to get past the failing items.
This is not a race condition but a memory problem.

Looks like breaking is giving the opportunity to python to clean up its state.

Now I do not understand why it is crashing before reaching or even getting close to max physical memory.
Are there other memory limitations in Python ? I ran much bigger processes without problem before.

[pkubryk]

I'll try triggering the garbage collector regularly tomorrow. see if it makes a difference.

[DanielGoldfarb]

@pkubryk
Thanks for the specific feedback. There are a number of similar postings online (for example this one) with lots of suggestions, but no definitive answer. However after reading through them, the following ideas seem to me to have a high probability of working:

1. Regarding my suggestion to del fig (item 7 above), perhaps do `plt.close('all') before deleting the Figure each time.
2. If you are running inside pycharm or any other IDE or notebook, try running your program simply as script in a terminal.
3. Clearly the debugger is altering memory. I therefore don't think it will isolate the problem. Probably better off with print statements to try to see where it is happening. However I agree that this is probably very low level. I cannot find "Fail to allocate bitmap" anywhere in the matplotlib library. It's possibly coming from some lower level CPython code. I am inclined, unless we can find a simpler reproducible case, to just take shots at making the problem go away; for example, the previous two suggestions, and the ones that follow:
4. Try setting environment variable mplbackend to "Agg": export MPLBACKEND="Agg" and/or call matplotlib.use("Agg") at the very beginning of your program.
5. Alternatively, instead of deleting the figure, try reusing the same figure. To do this you set returnfig=True only on your first call to mpf.plot(). Then between plots call fig.clear(). Then on all subsequent calls to mpf.plot() pass in the Axes objects that came from the first call: mpf.plot(prices,...,ax=axlist[0],volume=axlist[2],...), followed by fig.savefig(...)
6. Is it possible to run your job in batches? Pehaps have the script exit after 360 plots. And then kick it off again with some arguments that tell it to pick up where it left off?
7. Someone suggested import gc and then calling gc.collect() after calling plt.close(all) (or alternatively after del fig) each time.

Let me know if any of these ideas work. And if you are able to create a more simple reproducible case.

All the best. --Daniel

[hawl666]

@pkubryk
according to @DanielGoldfarb advice, i add

import matplotlib
matplotlib.use("Agg")

it works now.

[EricTheRed20]

The following code creates the problem every time:

import numpy
import matplotlib
from matplotlib import pyplot

#matplotlib.use("Agg")  # This does indeed fix the problem but it would be better to fix the problem properly
image = numpy.zeros((256,256,3))
for i in range(500):
	print(i)
	pyplot.subplot(1, 1, 1) # rows, cols, itemno
	pyplot.axis('off')
	pyplot.imshow(image)
	pyplot.savefig('plot_%d.png' % (i), dpi=250)
	pyplot.close()

This code causes the "Fail to allocate bitmap" every time when i = 369

[DanielGoldfarb]

@EricSchwerzel
Thank you very much for providing a simple, reproducible example.
Unfortunately its running fine on my system (WSL2 Ubuntu).

I have a couple other systems to test on; will try those too.

Please let me know what system you are running.

Please also add the following before your loop and report here the results:

print('matplotlib backend=',matplotlib.get_backend())
print('matplotlib.__version__ =',matplotlib.__version__)
print('numpy.__version__ =',numpy.__version__)

Thanks. --Daniel

[DanielGoldfarb]

working ok for me on WSL2 Ubuntu:

backend= Qt5Agg
matplotlib.__version__ = 3.3.4
numpy.__version__ = 1.19.2

and on Windows Powershell python:

backend= Qt5Agg
matplotlib.__version__ = 3.3.3
numpy.__version__ = 1.20.0rc1

[EvilDuncan]

The following code creates the problem every time:

import numpy
import matplotlib
from matplotlib import pyplot

#matplotlib.use("Agg")  # This does indeed fix the problem but it would be better to fix the problem properly
image = numpy.zeros((256,256,3))
for i in range(500):
	print(i)
	pyplot.subplot(1, 1, 1) # rows, cols, itemno
	pyplot.axis('off')
	pyplot.imshow(image)
	pyplot.savefig('plot_%d.png' % (i), dpi=250)
	pyplot.close()

This code causes the "Fail to allocate bitmap" every time when i = 369

I am experiencing the exact same issue with this down to the error on iteration 369. The code I'm using is here: (https://github.com/dmitryr1234/Phonon-Explorer/blob/master/Python%20Code/plotDataWithFit.py), but requires some other scripts to run. I'm using Python 3.7.8 through the Windows Command Prompt in Windows Terminal.

I have found that matplotlib.use("Agg") fixes my issue when nothing else would, but I still don't know what the issue was other than a memory buildup.

[fujiawei-dev]

More than 370 items must cause fail to allocate bitmap.

[DanielGoldfarb]

I am still unable to reproduce this issue. Although thank you @EvilDuncan for the simplified use-case that for you reproduces the problem.

To those who can reproduce, can you please post here your OS information, amount of memory, and matplotlib version?
Thank you.

Also, indicate if you are running inside any kind of virtual environment, or vm, and if so which one and what your settings are.

[fujiawei-dev]

Windows10, 64GB, Python 3.8.8(Anaconda), matplotlib==3.4.2

[EricTheRed20]

As a software developer, I know that these sorts of bugs are a real pain. If you email me, I'd be happy to do a TeamViewer session by phone so that you can see it on my computer and do what ever tests you want.

[tacaswell]

#matplotlib.use("Agg") # This does indeed fix the problem but it would be better to fix the problem properly

I would argue that this is the proper solution, if you do not need / want the GUI windows, then do not even create the GUI windows!

Setting the env MPLBACKEND=agg is another reliably way to get the effect without changing your code.

---

I think the core of the problem here is that when you import pyplot and have not explicitly selected a backend (ether through rcparams or through the MPLBACKEND env), we check if you have already imported a GUI framework we support and if you have we pick that backend and if not we go through a fixed list of backends until we find one that will import and the use that one.

All of the GUI toolkits are implemented in c or c++ and their Python wrappers proxy the underlying objects up to us on the Python side. In order for this to segfault or leak memory there has to be some careful coordination between the Python and c++ sides. In this case when we "close" a Qt window from Python, we drop the Python side references to it, but the final clean up of the window and its resources is done via a Qt Signal/slot.

The way that Qt's Signal/Slots work is that there is an internal queue, when you emit a Signal it goes on the queue and the main Qt loop will pop things off and dispatch out to the correct lots. In the tight loop of the OP, the Qt event loop is never given a chance to run so even though we may be gc'ing the Python there are still 350+ zombie windows in the background. I strongly suspect that you are running into some sort of soft resource limit.

I suspect that the reason the behavior changes on debugging is that as part of the debugging process you allowed the Qt main loop to run (which solved your problem!).

I think the options are:

1. set the backend to Agg (or svg, pdf, ps) so the GUI windows never get made
2. put sacrificial plt.pause(.001) in your tight loop (to let the event loop fully reap the GUI objects)

See https://matplotlib.org/stable/users/interactive_guide.html and
https://matplotlib.org/stable/users/interactive.html for lots more details.

---

There is a third rather radical path (that may require changes to mplfinance) which is to not use pyplot at all and create matplotlib.figure.Figure objects which will be fully managed by your code, but you lose the trivial "and now put this on the screen!" path), some ideas on this are at https://github.com/tacaswell/mpl-gui

[EricTheRed20]

You say to "do not even create the GUI windows". The source code I gave doesn't use the Windows GUI. It saves a file to disk!

[tacaswell]

The source code I gave doesn't use the Windows GUI. It saves a file to disk!

Yes it does, if your backend is 'QtAgg' then when you do

        # This creates:
        #  - matplotlib.figure.Figure
        #  - matplotlib.axes.Axes
        #  - PyQt5.XXX.QMainWindow  (I do not remember the Qt modules off the top of my head)
        #  - ...ToolBar
        #  - ...Qwidget
	pyplot.subplot(1, 1, 1) # rows, cols, itemno

What

matplotlib.use("Agg")

does is select the Agg backend which does not wrap at GUI framework, see https://matplotlib.org/stable/tutorials/introductory/usage.html#backends for more details.

[fujiawei-dev]

Thanks.

matplotlib.use("Agg") fix my problem.

[Alexoko12]

The following code creates the problem every time:

import numpy
import matplotlib
from matplotlib import pyplot

#matplotlib.use("Agg")  # This does indeed fix the problem but it would be better to fix the problem properly
image = numpy.zeros((256,256,3))
for i in range(500):
	print(i)
	pyplot.subplot(1, 1, 1) # rows, cols, itemno
	pyplot.axis('off')
	pyplot.imshow(image)
	pyplot.savefig('plot_%d.png' % (i), dpi=250)
	pyplot.close()

This code causes the "Fail to allocate bitmap" every time when i = 369

I got the same failure on Windows 10, 16GB, Python 3.9.6, Matplotlib 3.4.2 (backend TkAgg), Numpy 1.20.2
I changed the dpi to 5 to speed it a little bit up but the failure was the same at i=369.

Then I tried to define the figure directly and use garbage collector even with reimporting matplotlib.

import gc
import sys
import numpy as np
import matplotlib.pyplot as plt

image = np.zeros((16,16,3))
for i in range(500):
    print(i)
    fig = plt.figure()
    ax = fig.add_subplot(1, 1, 1)  # single axes plot
    ax.imshow(image)
    ax.axis("off")
    fig.savefig('plot_%d.png' % (i), dpi=5)
    plt.close()
    if i == 367:
        modules = list(sys.modules.keys())
        for modul in modules:
            if "matplotlib" in modul or "tkinter" in modul:
                del sys.modules[modul]
        del plt
        gc.collect()
        import matplotlib.pyplot as plt

but still i=369.

Then I tried using an easier code again, but with range(300) and paste it two times manually in the terminal. This works and I could also see that the memory gets free in the time I copied the lines again. But as we want to use it in a script this doesn't help much, because copying this block two times in a script results again in a big loop which fails.

image = np.zeros((16,16,3))
i = 0
for i in range(300):
    fig = plt.figure()
    print(i)
    ax = fig.add_subplot(1, 1, 1)  # single axes plot
    ax.imshow(image)
    ax.axis("off")
    fig.savefig('plot_%d.png' % (i), dpi=5)
    plt.close(fig)

Last thing I tried was to put fig outside the loop and it works I stoped after 8000 iterations and it takes constantly 50MB Ram.

import numpy as np
import matplotlib.pyplot as plt

image = np.zeros((16,16,3))
fig = plt.figure()
i = 0
while True:
    print(i)
    ax = fig.add_subplot(1, 1, 1)  # single axes plot
    ax.imshow(image)
    ax.axis("off")
    fig.savefig('plot_%d.png' % (i), dpi=5)
    ax.clear()
    fig.clf()
    i+= 1

Edit:
To create this error is even easier:

import matplotlib.pyplot as plt
i = 0
while True:
    print(i)
    fig = plt.figure(0)  # Using always same index as it should then reuse this figure
    plt.close(fig)          # Closing the figure creates the problem (same with plt.close(0))
    i += 1

Fails at i = 365 for plt.close(fig) and i = 369 for plt.close(0)

instead:

import matplotlib.pyplot as plt
i = 0
while True:
    print(i)
    fig = plt.figure(0)  # using always same index as it should then reuse this figure
    fig.clf()                  # Clearing is much faster and doesn't fails
    i += 1

[NicoHambauer]

Thank you @ALL! I also had the same issue and it worked out using matplotlib.use("Agg")

[KingOtto123]

Fantastic code example, and great workaround/solution. Both work (re-using figure, using "Agg").

Can someone explain (a) why this happens (370 is a particularly weird integer - why does it crash particularly there??), and (b) what does .use("Agg") do to prevent that?

[DanielGoldfarb]

@KingOtto123
The explanation is here and here and in the links/reference therein.

[alsaleem00]

The code has a problem when i run in a new machine were i installed a new python/matplotlib
It stops at 369. with Python 3.9.7 / matplotlib 3.4.3
however, it works with: python 3.9.6 / matplotlib 3.4.2

That might give a clue.

[poeticlinuxer]

from pathlib import Path
import matplotlib
import mplfinance as mpf
import pandas as pd

matplotlib.use('agg')

It can solve my problem .

[nepomnyi]

It is September 2023 and I am also having this issue. In my case, the magic number is 185 images, just like in this StackOverflow post.

Only matplotlib.use('agg') put right after the import section of my program helped me out. Nothing else could resolve my issue.

What interesting is that I obtained 14 sets of 500 images in the middle of August and processed them without any issues. Yesterday (i.e. in about a month), I obtained 3 more sets of 500 images, tried to process one of them and got that error. Between August and September I did not upgrade anything related to Python or Anaconda or my Hardware. The only thing that received updates was my Windows.

The bottom line is that the issues seems to pop up randomly.

I am using Python 3.11.4, matplotlib 3.7.1. I have Windows 11 Enterprise 22H2, 32 GB of RAM, Intel Xeon on HP Z240.

@alsaleem00 brought up an intersting suggestion that a matplotlib's upgrade could be responsible for the issue.

matplotlib team, I would be gratefull to you if you could resolve this issue. Even though @tacaswell suggested that agg is a solution, I would argue it is not. From a user's perspective it is just a work-around to patch an issue that creates a lot of headache (imaging debugging a code that takes hours to process such a big dataset of images).

[JRouvinen]

I am experiencing similar issues as described above -> my magic number on my code seems to be somewhere around 820.
The matplotlib.use('agg') doesn't seem to fix the problem.
Now i ran little experiment with plt.close('all') and fig.clf() calls and something strange happened:

• plt.close('all') test results 1000 figures-
  Real time: 116.47 seconds
  CPU time: 110.36 seconds

• fig.clf() test results 1000 figures -
  Real time: 0.60 seconds
  CPU time: 0.41 seconds

the fig.clf() call is lot faster and doesn't seem to fail (at least not so fast as plt.close call), i ran 1 000 000 figures and it didn't fail.