python - Error saving and loading a list of matrices

Question

I have a list "data_list", and I would save it in order to load it in another script. First of all I converted it in an array, in this way:

data_array = np.array(data_list)

Then I saved it:

np.savez("File", data_array)

Then, in another script I want to access to "File"; so:

a = np.load("File.npz") 
b = a['arr_0']

I used this code until two weeks ago and it worked fine. In these days I am trying to work with my program, but it ends with an error identified in the line

b = a['arr_0']

"File" is a 300 MB file. The strangest thing is that it has stopped suddenly to work. Any idea about what can be happened?

Ps: I give you some information. My list contains 180 matrices 511x511. Each matrix contains decimal numbers (I tried to create 180 matrices of zeros, and the error occurs in the same way). If I reduce the number of matrices, the script works fine: in particular down to 130 matrices it is ok, while up to the program doesn't work. Here I report the error message

        b = a['arr_0']
    File "C:Python27libsite-packages
umpylib
pyio.py", line 241, in     
    __getitem__
        return format.read_array(value)
    File "C:Python27libsite-packages
umpylibformat.py", line 459, in   
    read_array
        array = numpy.fromstring(data, dtype=dtype, count=count)
    MemoryError 

Answer 1

MemoryError is an out of memory condition. This explains why it happens with objects of at least a certain size - more and bigger arrays, as you would expect, require more memory. What the max size is, and why it seems to have changed, is harder. This can be highly specific to your system, especially in regard to considerations like:

• How much memory (physical RAM and swap space) exists and is available to the operating system
• How much virtual memory the OS gives to Python
• How much of that you're already using
• The implementation of the C library, especially of its malloc function, which can affect how Python uses the memory it is allocated

And possibly quite a few other things.

Per the comments, it seems the biggest problem here is that you are running a 32 bit build of Python. On Windows, 32 bit processes apparently have an effective maximum memory address space of around 2GB. By my tests, the list of arrays you are using by itself might take around a quarter of that. The fact that your error only comes up when reading the file back in suggests that numpy deserialisation is relatively memory intensive, but I don't know enough about its implementation to be able to say why that would be. In any case, it seems like installing a 64 bit build of Python is your best bet.