我有一个 3 维数组 (np.ndarray),其中大部分为 0。现在我想在第一维上对它们求和,但这相当慢。我查看了 csr_matrix,但 csr 不支持 3 维数组。有没有更快的方法来对几乎稀疏的 nd 数组求和?以下是我当前代码的摘录。
相关问题: sparse 3d matrix/array in Python? (创建一个自制的稀疏 ndarray 类,矫枉过正?)
r = np.array([ [[1, 0, 0, 0],
[1, 0, 0, 0],
[0, 0, 1, 0]],
[[0, 1, 0, 0],
[0, 0, 0, 1],
[0, 0, 2, 0]],
[[0, 1, 0, 0],
[0, 0, 0, 0],
[0, 0, 0, 0]],
[[0, 0, 0, 1],
[0, 0, 0, 0],
[0, 0, 0, 0]]], dtype=int)
np.sum(r,axis=0)
Out[35]:
array([[1, 2, 0, 1],
[1, 0, 0, 1],
[0, 0, 3, 0]])
编辑
在下面 hpaulj 的回答之后,我做了更多的计时测试,见下文。似乎 reshape 对总和没有多大好处,而将它们转换为 csr_matrix 并返回 numpy 会降低性能。我仍在考虑直接使用索引(下面称为 rand_persons、rand_articles 和 rand_days,因为在我原来的问题中,我做了大ndarray 使用这些索引。
from timeit import timeit
from scipy.sparse import csr_matrix
import numpy as np
def create_test_data():
'''
dtype = int64
1% nonzero, 1000x1000x100: 1.3 s,
1% nonzero, 10000x1000x100: 13.3 s
0.1% nonzero, 10000x1000x100: 2.7 s
1ppm nonzero, 10000x1000x100: 0.007 s
'''
global purchases
N_persons = 10000
N_articles = 1000
N_days = 100
purchases = np.zeros(shape=(N_days, N_persons, N_articles), dtype=int)
N_elements = N_persons * N_articles * N_days
rand_persons = np.random.choice(a=range(N_persons), size=N_elements / 1e6, replace=True)
rand_articles = np.random.choice(a=range(N_articles), size=N_elements / 1e6, replace=True)
rand_days = np.random.choice(a=range(N_days), size=N_elements / 1e6, replace=True)
for (i, j, k) in zip(rand_persons, rand_articles, rand_days):
purchases[k, i, j] += 1
def sum_over_first_dim_A():
'''
0.1% nonzero, 10000x1000x99: 1.57s (average over 10)
1ppm nonzero, 10000x1000x99: 1.70s (average over 10)
'''
global purchases
d = purchases[:99, :, :]
np.sum(d, axis=0)
def sum_over_first_dim_B():
'''
0.1% nonzero, 10000x1000x99: 1.55s (average over 10)
1ppm nonzero, 10000x1000x99: 1.37s (average over 10)
'''
global purchases
d = purchases[:99, :, :]
(N_days, N_persons, N_articles) = d.shape
d.reshape(N_days, -1).sum(0).reshape(N_persons, N_articles)
def sum_over_first_dim_C():
'''
0.1% nonzero, 10000x1000x99: 7.54s (average over 10)
1ppm nonzero, 10000x1000x99: 7.44s (average over 10)
'''
global purchases
d = purchases[:99, :, :]
(N_days, N_persons, N_articles) = d.shape
r = csr_matrix(d.reshape(N_days, -1))
t = r.sum(axis=0)
np.reshape(t, newshape=(N_persons, N_articles))
if __name__ == '__main__':
print (timeit(create_test_data, number=10))
print (timeit(sum_over_first_dim_A, number=10))
print (timeit(sum_over_first_dim_B, number=10))
print (timeit(sum_over_first_dim_C, number=10))
编辑 2
我现在找到了一种更快的求和方法:我用稀疏矩阵创建了一个 numpy 数组。但是,这些矩阵的初始创建还有一些时间。我现在用一个循环来做这个。有什么办法可以加快速度吗?
def create_test_data():
[ ... ]
'''
0.1% nonzero, 10000x1000x100: 2.1 s
1ppm nonzero, 10000x1000x100: 0.45 s
'''
global sp_purchases
sp_purchases = np.empty(N_days, dtype=lil_matrix)
for i in range(N_days):
sp_purchases[i] = lil_matrix((N_persons, N_articles))
for (i, j, k) in zip(rand_persons, rand_articles, rand_days):
sp_purchases[k][i, j] += 1
def sum_over_first_dim_D():
'''
0.1% nonzero, 10000x1000x99: 0.47s (average over 10)
1ppm nonzero, 10000x1000x99: 0.41s (average over 10)
'''
global sp_purchases
d = sp_purchases[:99]
np.sum(d)
请您参考如下方法:
您可以 reshape 数组使其成为 2d,求和,然后再整形
r.reshape(4,-1).sum(0).reshape(3,4) # == r.sum(0)
reshape 不会增加太多处理时间。您可以将 2d 转换为稀疏,看看是否可以节省时间。我的猜测是您的数组必须非常大且非常稀疏,才能击败直接的 numpy 总和。如果您有其他理由使用稀疏格式,这可能是值得的,但只是为了求和,不行。但是你自己测试一下。
