Как записать памяти эффективную программу Python?

Главный виновник, как упоминалось выше, - вызов range (). Он создаст список с 15 миллионами участников, и это займет 200 МБ вашей памяти, а с 15 процессами это 3 ГБ.

Но также не считывайте весь 15-мегабайтный файл в data (), read bit по битам из ответа. Вставка этих 15 МБ в переменную будет использовать на 15 МБ памяти больше, чем побитовое чтение из ответа.

Возможно, вы захотите просто извлечь данные, пока не закончатся indata, и сравнить количество извлеченных данных с что должно быть в первых байтах. Тогда вам не нужны ни range (), ни xrange (). Мне кажется более питоническим. :)

Consider using xrange() instead of range(), I believe that xrange is a generator whereas range() expands the whole list.

I'd say either don't read the whole file into memory, or don't keep the whole unpacked structure in memory.

Currently you keep both in memory, at the same time, this is going to be quite big. So you've got at least two copies of your data in memory, plus some metadata.

Also the final line

    f.write(os.linesep.join(data))

May actually mean you've temporarily got a third copy in memory (a big string with the entire output file).

So I'd say you're doing it in quite an inefficient way, keeping the entire input file, entire output file and a fair amount of intermediate data in memory at once.

Using the generator to parse it is quite a nice idea. Consider writing each record out after you've generated it (it can then be discarded and the memory reused), or if that causes too many write requests, batch them into, say, 100 rows at once.

Likewise, reading the response could be done in chunks. As they're fixed records this should be reasonably easy.

The last line should surely be f.close()? Those trailing parens are kinda important.

You can make this program more memory efficient by not reading all 15MB from the TCP connection, but instead processing each line as it is read. This will make the remote servers wait for you, of course, but that's okay.

Python is just not very memory efficient. It wasn't built for that.

You could do more of your work in compiled C code if you convert this to a list comprehension:

data = []
items = GetData(url)
for item in items:
    data.append(';'.join(item))

to:

data = [';'.join(items) for items in GetData(url)]

This is actually slightly different from your original code. In your version, GetData returns a 3-tuple, which comes back in items. You then iterate over this triplet, and append ';'.join(item) for each item in it. This means that you get 3 entries added to data for every triplet read from GetData, each one ';'.join'ed. If the items are just strings, then ';'.join will give you back a string with every other character a ';' - that is ';'.join("ABC") will give back "A;B;C". I think what you actually wanted was to have each triplet saved back to the data list as the 3 values of the triplet, separated by semicolons. That is what my version generates.

This may also help somewhat with your original memory problem, as you are no longer creating as many Python values. Remember that a variable in Python has much more overhead than one in a language like C. Since each value is itself an object, and add the overhead of each name reference to that object, you can easily expand the theoretical storage requirement several-fold. In your case, reading 15Mb X 15 = 225Mb + the overhead of each item of each triple being stored as a string entry in your data list could quickly grow to your 2Gb observed size. At minimum, my version of your data list will have only 1/3 the entries in it, plus the separate item references are skipped, plus the iteration is done in compiled code.

Есть 2 очевидных места, где вы храните большие объекты данных в памяти (переменная data в GetData () и data в MyThread.run () - эти два займут около 500Мб) и, вероятно, в пропущенном коде есть другие места. Есть оба способа сделать память более эффективной. Используйте response.read (4) вместо чтения всего ответа сразу и сделайте то же самое в коде позади РАСПАКОВАТЬ ФИКСИРОВАННУЮ ДЛИНУ ДВОИЧНЫХ ДАННЫХ ЗДЕСЬ . Измените data.append (...) в MyThread.run () на

if not first:
    f.write(os.linesep)
f.write(';'.join(item))

Эти изменения сэкономят вам много памяти.

Убедитесь, что вы удалили потоки после их остановки. (с использованием del )