Making Python Objects Small
Very often python consumes a stupid amount of memory, and it make suck processing scripts or mass processing annoying. TO summarize the tips from the blogpost:
- Data classes are smaller than dicts. Quote:
Starting in Python 3.3, the shared space is used to store keys in the dictionary for all instances of the class. This reduces the size of the instance trace in RAM.
Which seems a bit magical, but I will take it. - slots helps to remove dict and weakref from a class and it is noted that it’s the main memory saving technique for pure python classes. Quote:
This reduction is achieved by the fact that in the memory after the title of the object, object references are stored — the attribute values, and access to them is carried out using special descriptors that are in the class dictionar
y. - There are some additional classes we can use in the RecordClass library, which involves tuples not part of the cyclic GC process. This can minimally reduce size of the pure python class object size.
- Cython is not just for speed. It’s for memory footprint too.
- Numpy for arrays (as usual