我试图解决这个问题,我存储了给定长度的子字符串的位置和数量。由于字符串可能很长(基因组序列),我正在尝试使用多个进程来加速它。当程序运行时,一旦线程结束,存储对象的变量似乎会丢失所有信息。
import numpy
import multiprocessing
from multiprocessing.managers import BaseManager, DictProxy
from collections import defaultdict, namedtuple, Counter
from functools import partial
import ctypes as c
class MyManager(BaseManager):
pass
MyManager.register('defaultdict', defaultdict, DictProxy)
def gc_count(seq):
return int(100 * ((seq.upper().count('G') + seq.upper().count('C') + 0.0) / len(seq)))
def getreads(length, table, counts, genome):
genome_len = len(genome)
for start in range(0,genome_len):
gc = gc_count(genome[start:start+length])
table[ (length, gc) ].append( (start) )
counts[length,gc] +=1
if __name__ == "__main__":
g = 'ACTACGACTACGACTACGCATCAGCACATACGCATACGCATCAACGACTACGCATACGACCATCAGATCACGACATCAGCATCAGCATCACAGCATCAGCATCAGCACTACAGCATCAGCATCAGCATCAG'
genome_len = len(g)
mgr = MyManager()
mgr.start()
m = mgr.defaultdict(list)
mp_arr = multiprocessing.Array(c.c_double, 10*101)
arr = numpy.frombuffer(mp_arr.get_obj())
count = arr.reshape(10,101)
pool = multiprocessing.Pool(9)
partial_getreads = partial(getreads, table=m, counts=count, genome=g)
pool.map(partial_getreads, range(1, 10))
pool.close()
pool.join()
for i in range(1, 10):
for j in range(0,101):
print count[i,j]
for i in range(1, 10):
for j in range(0,101):
print len(m[(i,j)])
最后的循环只会打印出0.0对于 count 中的每个元素和 0对于 m 中的每个列表,所以不知何故我失去了所有的计数。如果我在 getreads(...) 中打印计数函数,我可以看到值正在增加。反之,打印len(table[ (length, gc) ])在getreads(...)或 len(m[(i,j)])在主体中只会产生 0 .
请您参考如下方法:
您还可以将您的问题表述为 map-reduce 问题,这样您就可以避免在多个进程之间共享数据(我想这会加快计算速度)。您只需要从函数 (map) 返回结果表和计数,并合并所有进程的结果 (reduce)。
回到你原来的问题...
在Managers的底部有一个相关的说明 修改字典和列表中的可变值或项目。基本上,你 需要将修改后的对象重新赋值给容器代理。
l = table[ (length, gc) ]
l.append( (start) )
table[ (length, gc) ] = l
还有一篇关于 combining pool map with Array 的相关 Stackoverflow 帖子.
考虑到这两个因素,您可以执行以下操作:
def getreads(length, table, genome):
genome_len = len(genome)
arr = numpy.frombuffer(mp_arr.get_obj())
counts = arr.reshape(10,101)
for start in range(0,genome_len):
gc = gc_count(genome[start:start+length])
l = table[ (length, gc) ]
l.append( (start) )
table[ (length, gc) ] = l
counts[length,gc] +=1
if __name__ == "__main__":
g = 'ACTACGACTACGACTACGCATCAGCACATACGCATACGCATCAACGACTACGCATACGACCATCAGATCACGACATCAGCATCAGCATCACAGCATCAGCATCAGCACTACAGCATCAGCATCAGCATCAG'
genome_len = len(g)
mgr = MyManager()
mgr.start()
m = mgr.defaultdict(list)
mp_arr = multiprocessing.Array(c.c_double, 10*101)
arr = numpy.frombuffer(mp_arr.get_obj())
count = arr.reshape(10,101)
pool = multiprocessing.Pool(9)
partial_getreads = partial(getreads, table=m, genome=g)
pool.map(partial_getreads, range(1, 10))
pool.close()
pool.join()
arr = numpy.frombuffer(mp_arr.get_obj())
count = arr.reshape(10,101)
