問(wèn)題描述

我使用 this answer 以便在 Linux 機(jī)器上運(yùn)行 Python 中的多處理并行命令.

I was using this answer in order to run parallel commands with multiprocessing in Python on a Linux box.

我的代碼做了類似的事情:

My code did something like:

import multiprocessing
import logging

def cycle(offset):
    # Do stuff

def run():
    for nprocess in process_per_cycle:
        logger.info("Start cycle with %d processes", nprocess)
        offsets = list(range(nprocess))
        pool = multiprocessing.Pool(nprocess)
        pool.map(cycle, offsets)

但是我收到了這個(gè)錯(cuò)誤:OSError: [Errno 24] Too many open files
因此，代碼打開(kāi)了太多的文件描述符，即:它啟動(dòng)了太多的進(jìn)程而沒(méi)有終止它們.

But I was getting this error: OSError: [Errno 24] Too many open files
So, the code was opening too many file descriptor, i.e.: it was starting too many processes and not terminating them.

我修復(fù)了它，用這些行替換了最后兩行:

I fixed it replacing the last two lines with these lines:

    with multiprocessing.Pool(nprocess) as pool:
        pool.map(cycle, offsets)

但我不知道這些行修復(fù)它的確切原因.

But I do not know exactly why those lines fixed it.

with 下面發(fā)生了什么?

推薦答案

您正在循環(huán)中創(chuàng)建新進(jìn)程，然后在完成后忘記關(guān)閉它們.結(jié)果，您有太多打開(kāi)的進(jìn)程.這是個(gè)壞主意.

You're creating new processes inside a loop, and then forgetting to close them once you're done with them. As a result, there comes a point where you have too many open processes. This is a bad idea.

您可以通過(guò)使用自動(dòng)調(diào)用 pool.terminate 的上下文管理器來(lái)解決此問(wèn)題，或者您自己手動(dòng)調(diào)用 pool.terminate.或者，你為什么不在循環(huán)外創(chuàng)建一個(gè)池一次，然后將任務(wù)發(fā)送到里面的進(jìn)程?

You could fix this by using a context manager which automatically calls pool.terminate, or manually call pool.terminate yourself. Alternatively, why don't you create a pool outside the loop just once, and then send tasks to the processes inside?

pool = multiprocessing.Pool(nprocess) # initialise your pool
for nprocess in process_per_cycle:
    ...       
    pool.map(cycle, offsets) # delegate work inside your loop

pool.close() # shut down the pool

有關(guān)更多信息，您可以仔細(xì)閱讀 multiprocessing.Pool 文檔.

For more information, you could peruse the multiprocessing.Pool documentation.

這篇關(guān)于多處理返回“打開(kāi)的文件太多"；但是使用 `with...as` 可以解決這個(gè)問(wèn)題.為什么?的文章就介紹到這了，希望我們推薦的答案對(duì)大家有所幫助，也希望大家多多支持html5模板網(wǎng)！