Skip to main content
 首页 » 编程设计

Python 多处理 : losing counts after pool. 加入()

2025年05月04日10zengkefu

我试图解决这个问题,我存储了给定长度的子字符串的位置和数量。由于字符串可能很长(基因组序列),我正在尝试使用多个进程来加速它。当程序运行时,一旦线程结束,存储对象的变量似乎会丢失所有信息。

import numpy 
import multiprocessing 
from multiprocessing.managers import BaseManager, DictProxy 
from collections import defaultdict, namedtuple, Counter 
from functools import partial 
import ctypes as c 
 
class MyManager(BaseManager): 
        pass 
 
MyManager.register('defaultdict', defaultdict, DictProxy) 
 
def gc_count(seq): 
        return int(100 * ((seq.upper().count('G') + seq.upper().count('C') + 0.0) / len(seq))) 
 
def getreads(length, table, counts, genome): 
        genome_len = len(genome) 
        for start in range(0,genome_len):  
                gc = gc_count(genome[start:start+length]) 
                table[ (length, gc) ].append( (start) ) 
                counts[length,gc] +=1 
 
if __name__ == "__main__": 
    g = 'ACTACGACTACGACTACGCATCAGCACATACGCATACGCATCAACGACTACGCATACGACCATCAGATCACGACATCAGCATCAGCATCACAGCATCAGCATCAGCACTACAGCATCAGCATCAGCATCAG' 
    genome_len = len(g) 
 
    mgr = MyManager() 
    mgr.start() 
    m = mgr.defaultdict(list) 
    mp_arr = multiprocessing.Array(c.c_double, 10*101) 
    arr = numpy.frombuffer(mp_arr.get_obj()) 
    count = arr.reshape(10,101) 
 
    pool = multiprocessing.Pool(9) 
    partial_getreads = partial(getreads, table=m, counts=count, genome=g) 
    pool.map(partial_getreads, range(1, 10)) 
    pool.close() 
    pool.join() 
 
    for i in range(1, 10): 
            for j in range(0,101): 
                    print count[i,j] 
    for i in range(1, 10): 
            for j in  range(0,101): 
                    print len(m[(i,j)]) 

最后的循环只会打印出0.0对于 count 中的每个元素和 0对于 m 中的每个列表,所以不知何故我失去了所有的计数。如果我在 getreads(...) 中打印计数函数,我可以看到值正在增加。反之,打印len(table[ (length, gc) ])在getreads(...)或 len(m[(i,j)])在主体中只会产生 0 .

请您参考如下方法:

您还可以将您的问题表述为 map-reduce 问题,这样您就可以避免在多个进程之间共享数据(我想这会加快计算速度)。您只需要从函数 (map) 返回结果表和计数,并合并所有进程的结果 (reduce)。

回到你原来的问题...

在Managers的底部有一个相关的说明 修改字典和列表中的可变值或项目。基本上,你 需要将修改后的对象重新赋值给容器代理。

l = table[ (length, gc) ] 
l.append( (start) ) 
table[ (length, gc) ] = l 

还有一篇关于 combining pool map with Array 的相关 Stackoverflow 帖子.

考虑到这两个因素,您可以执行以下操作:

def getreads(length, table, genome): 
        genome_len = len(genome) 
 
        arr = numpy.frombuffer(mp_arr.get_obj()) 
        counts = arr.reshape(10,101) 
 
        for start in range(0,genome_len):  
                gc = gc_count(genome[start:start+length]) 
                l = table[ (length, gc) ] 
                l.append( (start) ) 
                table[ (length, gc) ] = l 
                counts[length,gc] +=1 
 
 
if __name__ == "__main__": 
    g = 'ACTACGACTACGACTACGCATCAGCACATACGCATACGCATCAACGACTACGCATACGACCATCAGATCACGACATCAGCATCAGCATCACAGCATCAGCATCAGCACTACAGCATCAGCATCAGCATCAG' 
    genome_len = len(g) 
 
    mgr = MyManager() 
    mgr.start() 
    m = mgr.defaultdict(list) 
    mp_arr = multiprocessing.Array(c.c_double, 10*101) 
    arr = numpy.frombuffer(mp_arr.get_obj()) 
    count = arr.reshape(10,101) 
 
    pool = multiprocessing.Pool(9) 
    partial_getreads = partial(getreads, table=m, genome=g) 
 
    pool.map(partial_getreads, range(1, 10)) 
    pool.close() 
    pool.join() 
 
    arr = numpy.frombuffer(mp_arr.get_obj()) 
    count = arr.reshape(10,101)