Skip to main content
 首页 » 编程设计

python之在 Python 中加速一对多相关性计算

2025年04月02日14jpfss

我想计算 Python 中向量与数组每一行之间的 Pearson 相关系数(假定为 numpy 和/或 scipy)。由于实际数据数组的大小和内存限制,将无法使用标准相关矩阵计算函数。这是我天真的实现:

import numpy as np 
import scipy.stats as sps 
 
np.random.seed(0) 
 
def correlateOneWithMany(one, many): 
    """Return Pearson's correlation coef of 'one' with each row of 'many'.""" 
    pr_arr = np.zeros((many.shape[0], 2), dtype=np.float64) 
    pr_arr[:] = np.nan 
    for row_num in np.arange(many.shape[0]): 
        pr_arr[row_num, :] = sps.pearsonr(one, many[row_num, :]) 
    return pr_arr 
 
obs, varz = 10 ** 3, 500 
X = np.random.uniform(size=(obs, varz)) 
 
pr = correlateOneWithMany(X[0, :], X) 
 
%timeit correlateOneWithMany(X[0, :], X) 
# 10 loops, best of 3: 38.9 ms per loop 

如有任何关于加快这一进程的想法,我们将不胜感激!

请您参考如下方法:

模块scipy.spatial.distance实现“相关距离”,它只是一个减去相关系数。您可以使用函数 cdist计算一对多距离,用1减去结果得到相关系数。

这是您的脚本的修改版本,其中包括使用 cdist 计算相关系数:

import numpy as np 
import scipy.stats as sps 
from scipy.spatial.distance import cdist 
 
np.random.seed(0) 
 
def correlateOneWithMany(one, many): 
    """Return Pearson's correlation coef of 'one' with each row of 'many'.""" 
    pr_arr = np.zeros((many.shape[0], 2), dtype=np.float64) 
    pr_arr[:] = np.nan 
    for row_num in np.arange(many.shape[0]): 
        pr_arr[row_num, :] = sps.pearsonr(one, many[row_num, :]) 
    return pr_arr 
 
obs, varz = 10 ** 3, 500 
X = np.random.uniform(size=(obs, varz)) 
 
pr = correlateOneWithMany(X[0, :], X) 
 
c = 1 - cdist(X[0:1, :], X, metric='correlation')[0] 
 
print(np.allclose(c, pr[:, 0])) 

时间:

In [133]: %timeit correlateOneWithMany(X[0, :], X) 
10 loops, best of 3: 37.7 ms per loop 
 
In [134]: %timeit 1 - cdist(X[0:1, :], X, metric='correlation')[0] 
1000 loops, best of 3: 1.11 ms per loop