python之在 Python 中通过正则表达式分解 CSV_编程设计

我有一个格式如下的文件:

S1A23 
0.01,0.01 
0.02,0.02 
0.03,0.03 
S25A123 
0.05,0.06 
0.07,0.08 
S3034A1 
1000,0.04 
2000,0.08 
3000,0.1

我想按每个“S_A_”分解它，并计算下面数据的相关系数。到目前为止，我有:

import re 
import pandas as pd 
 
test = pd.read_csv("predict.csv",sep=('S\d+A\d+')) 
 
print test

但这只会给我:

  Unnamed: 0     , 
0  0.01,0.01  None 
1  0.02,0.02  None 
2  0.03,0.03  None 
3        NaN     , 
4  0.05,0.06  None 
5  0.07,0.08  None 
6        NaN     , 
7  1000,0.04  None 
8  2000,0.08  None 
9   3000,0.1  None 
 
[10 rows x 2 columns]

理想情况下，我喜欢保留正则表达式定界符，并有类似的内容:

S1A23: 1.0 
S2A123: 0.86 
S303A1: 0.75

这可能吗？

编辑
运行大文件(~250k 行)时，我收到以下错误。这不是数据问题，因为当我将 ~250k 行分成更小的 block 时，所有部分都运行良好。

Traceback (most recent call last): 
  File "/Users/adamg/PycharmProjects/Subj_AnswerCorrCoef/GetCorrCoef.py", line 15, in <module> 
    print(result) 
  File "/Users/adamg/anaconda/lib/python2.7/site-packages/pandas/core/base.py", line 35, in __str__ 
    return self.__bytes__() 
  File "/Users/adamg/anaconda/lib/python2.7/site-packages/pandas/core/base.py", line 47, in __bytes__ 
    return self.__unicode__().encode(encoding, 'replace') 
  File "/Users/adamg/anaconda/lib/python2.7/site-packages/pandas/core/series.py", line 857, in __unicode__ 
    result = self._tidy_repr(min(30, max_rows - 4)) 
TypeError: unsupported operand type(s) for -: 'NoneType' and 'int'

我的确切代码是:

import numpy as np 
import pandas as pd 
import csv 
pd.options.display.max_rows = None 
fileName = 'keyStrokeFourgram/TESTING1' 
 
df = pd.read_csv(fileName, names=['pause', 'probability']) 
mask = df['pause'].str.match('^S\d+_A\d+') 
df['S/A'] = (df['pause'] 
              .where(mask, np.nan) 
              .fillna(method='ffill')) 
df = df.loc[~mask] 
 
result = df.groupby(['S/A']).apply(lambda grp: grp['pause'].corr(grp['probability'])) 
print(result)

请您参考如下方法:

sep 参数用于指定在同一行上分隔值的模式。它不能用于将 csv 的行分隔成单独的数据帧。

编辑:有一种方法可以使用 read_csv 将 csv 读入 DataFrame。自 read_csv should be faster 以来，这比使用 Python 循环(如我的原始答案中所做的那样)更可取。 .这可能很重要——尤其是对于大型 csv 文件。

import numpy as np 
import pandas as pd 
df = pd.read_csv("data", names=['x', 'y']) 
mask = df['x'].str.match('^S\d+A\d+')         # 1 
df['type'] = (df['x'] 
              .where(mask, np.nan)            # 2 
              .fillna(method='ffill'))        # 3 
df = df.loc[~mask]                            # 4 
 
result = df.groupby(['type']).apply(lambda grp: grp['x'].corr(grp['y'])) 
print(result)

产量

type 
S1A23      1.000000 
S25A123    1.000000 
S3034A1    0.981981 
dtype: float64

掩码在 'x' 列中具有“类型”的行上为 True。

In [139]: mask 
Out[139]:  
0      True 
1     False 
2     False 
3     False 
4      True 
5     False 
6     False 
7      True 
8     False 
9     False 
10    False 
Name: x, dtype: bool

df['x'].where(mask, np.nan) 返回一个 Series，等于 df['x'] where 掩码为 True，否则为 np.nan。

用货币值向前填充 nans

In [141]: df['x'].where(mask, np.nan).fillna(method='ffill') 
Out[141]:  
0       S1A23 
1       S1A23 
2       S1A23 
3       S1A23 
4     S25A123 
5     S25A123 
6     S25A123 
7     S3034A1 
8     S3034A1 
9     S3034A1 
10    S3034A1 
Name: x, dtype: object

仅选择掩码为 False 的那些行

原答案:

不幸的是，我没有找到将您的数据文件直接读入适当的 DataFrame 的方法。您需要使用 Python 循环对行进行一些调整，使其成为正确的形式。

import pandas as pd 
import csv 
 
def to_columns(f): 
    val = None 
    for row in csv.reader(f): 
        if len(row) == 1: 
            val = row[0] 
        else: 
            yield [val] + row 
 
with open('data') as f: 
    df = pd.DataFrame.from_records(to_columns(f), columns=['type', 'x', 'y']) 
 
print(df) 
result = df.groupby(['type']).apply(lambda grp: grp['x'].corr(grp['y'])) 
print(result)