問(wèn)題描述

我有這樣一個(gè)簡(jiǎn)單的任務(wù):

I have a simple task like that:

def worker(queue):
    while True:
        try:
            _ = queue.get_nowait()
        except Queue.Empty:
            break

if __name__ == '__main__':
    manager = multiprocessing.Manager()
    # queue = multiprocessing.Queue()
    queue = manager.Queue()

    for i in range(5):
        queue.put(i)

    processes = []

    for i in range(2):
        proc = multiprocessing.Process(target=worker, args=(queue,))
        processes.append(proc)
        proc.start()

    for proc in processes:
        proc.join()

似乎 multiprocessing.Queue 可以完成我需要的所有工作，但另一方面，我看到許多 manager().Queue() 的示例并且無(wú)法理解我真正需要什么.看起來(lái) Manager().Queue() 使用某種代理對(duì)象，但我不明白這些目的，因?yàn)?multiprocessing.Queue() 在沒(méi)有任何代理對(duì)象的情況下做同樣的工作.

It seems that multiprocessing.Queue can do all work that i needed, but on the other hand I see many examples of manager().Queue() and can't understand what I really need. Looks like Manager().Queue() use some sort of proxy objects, but I doesn't understand those purpose, because multiprocessing.Queue() do the same work without any proxy objects.

所以，我的問(wèn)題是:

1) multiprocessing.Queue 和 multiprocessing.manager().Queue() 返回的對(duì)象有什么區(qū)別?

1) What really difference between multiprocessing.Queue and object returned by multiprocessing.manager().Queue()?

2) 我需要使用什么?

2) What do I need to use?

推薦答案

雖然我對(duì)這個(gè)主題的理解有限，但從我所做的我可以看出 multiprocessing.Queue() 和 multiprocessing.Manager() 之間的一個(gè)主要區(qū)別.Queue():

Though my understanding is limited about this subject, from what I did I can tell there is one main difference between multiprocessing.Queue() and multiprocessing.Manager().Queue():

• multiprocessing.Queue() 是一個(gè)對(duì)象，而 multiprocessing.Manager().Queue() 是一個(gè)地址(代理)，指向由 multiprocessing.Manager() 對(duì)象管理的共享隊(duì)列.
• 因此您不能將普通的 multiprocessing.Queue() 對(duì)象傳遞給 Pool 方法，因?yàn)樗荒鼙浑缰?
• 此外，python 文檔告訴我們?cè)谑褂枚嗵幚頃r(shí)要特別注意.Queue() 因?yàn)樗赡軙?huì)產(chǎn)生不良影響

• multiprocessing.Queue() is an object whereas multiprocessing.Manager().Queue() is an address (proxy) pointing to shared queue managed by the multiprocessing.Manager() object.
• therefore you can't pass normal multiprocessing.Queue() objects to Pool methods, because it can't be pickled.
• Moreover the python doc tells us to pay particular attention when using multiprocessing.Queue() because it can have undesired effects

注意當(dāng)一個(gè)對(duì)象被放入隊(duì)列時(shí)，該對(duì)象被腌制并且后臺(tái)線程稍后將腌制數(shù)據(jù)刷新到底層管道.這會(huì)產(chǎn)生一些令人驚訝的后果，但不會(huì)造成任何實(shí)際困難——如果它們真的困擾您，那么您可以改用由經(jīng)理創(chuàng)建的隊(duì)列.將對(duì)象放入空隊(duì)列后，隊(duì)列的 empty() 方法返回 False 和 get_nowait() 可以在不引發(fā) Queue.Empty 的情況下返回之前可能會(huì)有一個(gè)無(wú)限小的延遲.如果多個(gè)進(jìn)程正在對(duì)對(duì)象進(jìn)行排隊(duì)，則對(duì)象可能會(huì)在另一端亂序接收.但是，由同一進(jìn)程排隊(duì)的對(duì)象將始終按預(yù)期順序排列.

Note When an object is put on a queue, the object is pickled and a background thread later flushes the pickled data to an underlying pipe. This has some consequences which are a little surprising, but should not cause any practical difficulties – if they really bother you then you can instead use a queue created with a manager. After putting an object on an empty queue there may be an infinitesimal delay before the queue’s empty() method returns False and get_nowait() can return without raising Queue.Empty. If multiple processes are enqueuing objects, it is possible for the objects to be received at the other end out-of-order. However, objects enqueued by the same process will always be in the expected order with respect to each other.

警告 如上所述，如果子進(jìn)程已將項(xiàng)目放入隊(duì)列(并且它沒(méi)有使用 JoinableQueue.cancel_join_thread)，那么該進(jìn)程將不會(huì)終止，直到所有緩沖的項(xiàng)目都被刷新到管道.這意味著如果您嘗試加入該進(jìn)程，您可能會(huì)遇到死鎖，除非您確定已放入隊(duì)列的所有項(xiàng)目都已被消耗.類似地，如果子進(jìn)程是非守護(hù)進(jìn)程，則父進(jìn)程在嘗試加入其所有非守護(hù)子進(jìn)程時(shí)可能會(huì)掛起退出.請(qǐng)注意，使用管理器創(chuàng)建的隊(duì)列不存在此問(wèn)題.

Warning As mentioned above, if a child process has put items on a queue (and it has not used JoinableQueue.cancel_join_thread), then that process will not terminate until all buffered items have been flushed to the pipe. This means that if you try joining that process you may get a deadlock unless you are sure that all items which have been put on the queue have been consumed. Similarly, if the child process is non-daemonic then the parent process may hang on exit when it tries to join all its non-daemonic children. Note that a queue created using a manager does not have this issue.

通過(guò)將隊(duì)列設(shè)置為全局變量并在初始化時(shí)為所有進(jìn)程設(shè)置它，可以將 multiprocessing.Queue() 與 Pool 一起使用:

There is a workaround to use multiprocessing.Queue() with Pool by setting the queue as a global variable and setting it for all processes at initialization :

queue = multiprocessing.Queue()
def initialize_shared(q):
    global queue
    queue=q

pool= Pool(nb_process,initializer=initialize_shared, initargs(queue,))

將創(chuàng)建具有正確共享隊(duì)列的池進(jìn)程，但我們可以爭(zhēng)辯說(shuō) multiprocessing.Queue() 對(duì)象不是為此用途而創(chuàng)建的.

will create pool processes with correctly shared queues but we can argue that the multiprocessing.Queue() objects were not created for this use.

另一方面，manager.Queue() 可以在池子進(jìn)程之間共享，方法是將它作為函數(shù)的普通參數(shù)傳遞.

On the other hand the manager.Queue() can be shared between pool subprocesses by passing it as normal argument of a function.

在我看來(lái)，使用 multiprocessing.Manager().Queue() 在任何情況下都很好，而且麻煩更少.使用經(jīng)理可能會(huì)有一些缺點(diǎn)，但我不知道.

In my opinion, using multiprocessing.Manager().Queue() is fine in every case and less troublesome. There might be some drawbacks using a manager but I'm not aware of it.

這篇關(guān)于Python multiprocessing.Queue 與 multiprocessing.manager().Queue()的文章就介紹到這了，希望我們推薦的答案對(duì)大家有所幫助，也希望大家多多支持html5模板網(wǎng)！