Skip to main content
 首页 » 编程设计

python之解析大日志文件

2025年05月04日16kenshinobiy

我有一个包含 1,001,623 行格式的日志文件:

[02/Jan/2012:09:07:32] "GET /click?id=162&prod=5475 HTTP/1.1" 200 4352 

每一个都换行

我使用正则表达式对其进行循环并提取我需要的信息(日期、ID、产品)

for txt in logfile: 
    m = rg.search(txt) 
    if m: 
        l1=m.group(1) 
        l2=m.group(2) 
        l3=m.group(3) 
        dt=dt.append(pd.Series([l1])) 
        art=art.append(pd.Series([l2])) 
        usr=usr.append(pd.Series([l3])) 

这在测试中效果很好,我只使用了一个小样本,但当我使用整个集合时,它已经运行了 12 个小时,没有显示任何进展。然后我将创建一个数据框来进行一些分析。有更好的方法吗?

编辑:

这就是我打开日志文件的方式。

logfile = open("data/access.log", "r") 

正则表达式

re1='.*?'   # Non-greedy match on filler 
re2='((?:(?:[0-2]?\\d{1})|(?:[3][01]{1}))[-:\\/.](?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?|May|Jun(?:e)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?|Sept|Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)[-:\\/.](?:(?:[1]{1}\\d{1}\\d{1}\\d{1})|(?:[2]{1}\\d{3})))(?![\\d])'  # DDMMMYYYY 1 
re3='.*?'   # Non-greedy match on filler 
re4='\\d+'  # Uninteresting: int 
re5='.*?'   # Non-greedy match on filler 
re6='\\d+'  # Uninteresting: int 
re7='.*?'   # Non-greedy match on filler 
re8='\\d+'  # Uninteresting: int 
re9='.*?'   # Non-greedy match on filler 
re10='(\\d+)'   # Integer Number 1 
re11='.*?'  # Non-greedy match on filler 
re12='(\\d+)'   # Integer Number 2 
 
rg =  re.compile(re1+re2+re3+re4+re5+re6+re7+re8+re9+re10+re11+re12,re.IGNORECASE|re.DOTALL) 
m = rg.search(txt) 

请您参考如下方法:

您可以使用 pandas。首先通过 strip 剥离 []然后转换 to_datetime .

然后解析id 和prod 最后通过concat 合并在一起:

import pandas as pd 
import io 
 
temp=u"""[02/Jan/2012:09:07:32] "GET /click?id=162&prod=5475 HTTP/1.1" 200 4352 
[02/Jan/2012:09:07:32] "GET /click?id=162&prod=5475 HTTP/1.1" 200 4352 
[02/Jan/2012:09:07:32] "GET /click?id=162&prod=5475 HTTP/1.1" 200 4352 
[02/Jan/2012:09:07:32] "GET /click?id=162&prod=5475 HTTP/1.1" 200 4352 
[02/Jan/2012:09:07:32] "GET /click?id=162&prod=5475 HTTP/1.1" 200 4352""" 
 
#change io.StringIO(temp) to 'filename.csv' 
df = pd.read_csv(io.StringIO(temp), sep="\s*", engine='python', header=None,  
                                    names=['date','get','data','http','no1','no2']) 
 
#format - http://strftime.org/ 
df['date'] = pd.to_datetime(df['date'].str.strip('[]'), format="%d/%b/%Y:%H:%M:%S") 
 
#split Dataframe 
df1 = pd.DataFrame([ x.split('=') for x in df['data'].tolist() ], columns=['c','id','prod']) 
 
#split Dataframe 
df2 = pd.DataFrame([ x.split('&') for x in df1['id'].tolist() ], columns=['id', 'no3']) 
print df 
 
                 date   get                     data       http  no1   no2 
0 2012-01-02 09:07:32  "GET  /click?id=162&prod=5475  HTTP/1.1"  200  4352 
1 2012-01-02 09:07:32  "GET  /click?id=162&prod=5475  HTTP/1.1"  200  4352 
2 2012-01-02 09:07:32  "GET  /click?id=162&prod=5475  HTTP/1.1"  200  4352 
3 2012-01-02 09:07:32  "GET  /click?id=162&prod=5475  HTTP/1.1"  200  4352 
4 2012-01-02 09:07:32  "GET  /click?id=162&prod=5475  HTTP/1.1"  200  4352 
print df1 
 
           c        id  prod 
0  /click?id  162&prod  5475 
1  /click?id  162&prod  5475 
2  /click?id  162&prod  5475 
3  /click?id  162&prod  5475 
4  /click?id  162&prod  5475 
print df2 
 
    id   no3 
0  162  prod 
1  162  prod 
2  162  prod 
3  162  prod 
4  162  prod 
 
df = pd.concat([df['date'], df1['prod'], df2['id']], axis=1) 
print df 
 
                 date  prod   id 
0 2012-01-02 09:07:32  5475  162 
1 2012-01-02 09:07:32  5475  162 
2 2012-01-02 09:07:32  5475  162 
3 2012-01-02 09:07:32  5475  162 
4 2012-01-02 09:07:32  5475  162