Skip to main content
 首页 » 编程设计

python之如何从 CSV 中删除重复的行

2025年05月04日37lonelyxmas

数据:

    112343  The data point was created on 1903. 
    112344  The data point was created on 1909. 
    112345  The data point was created on 1919. 
    112346  The data point was created on 1911. 
    112346  The data point was created on 1911-12. 
    112346  The data point was created on 1911-12. 
    112347  The data point was created on 1911. 
    112348  The data point was created on 1911. 

这里重复的是id。我希望删除重复项,但我想保留最长的行 [1](如理想输出所示)。

这是我尝试过的:

import sys 
import csv 
import re 
import string 
 
df = csv.reader(‘fil.csv’, delimiter = ',') 
 
for r in df: 
    dup = next(df) 
    if r[0] == dup[0]: 
        r[1] < dup[1]: #I am checking if the text is larger then the previous 
            print dup[0], dup[1] 
    else: 
        print r[0], r[1] 

但是我得到的输出是,

112343  The data point was created on 1903. 
112346  The data point was created on 1911-12. 
112346  The data point was created on 1911-12. 
112346  The data point was created on 1911. 
112348  The data point was created on 1911. 

行丢失了!

理想的输出是

112343  The data point was created on 1903. 
112344  The data point was created on 1909. 
112345  The data point was created on 1919. 
112346  The data point was created on 1911-12. 
112347  The data point was created on 1911. 
112348  The data point was created on 1911. 

如何做到这一点?我可以使用什么条件或关键字?或者我可以有两个重复文件并比较它们之间的行以消除重复项吗?

请您参考如下方法:

我的尝试:

import csv 
import collections 
 
csv_input = """    112343,  The data point was created on 1903. 
    112344,  The data point was created on 1909. 
    112345,  The data point was created on 1919. 
    112346,  The data point was created on 1911. 
    112346,  The data point was created on 1911-12. 
    112346,  The data point was created on 1911-12. 
    112347,  The data point was created on 1911. 
    112348,  The data point was created on 1911.""" 
 
reader = csv.reader(csv_input.split('\n'))     
 
result = collections.OrderedDict() 
for row_id, data in reader: 
    if len(result.get(row_id, ''))<len(data): 
        result[row_id] = data 
 
for row_id, data in result.items(): 
    print "{},{}".format(row_id, data)