Skip to main content
 首页 » 编程设计

python之UTF-8 转换为真实字母

2025年05月04日19Leo_wl

我的一个项目需要帮助。我正在清理大量数据以批量插入到 Microsoft SQL 中。数据就像 1000 万行。但是我创建了一个脚本,只是为了提取前 1000 个进行清洁,假设其余的都相同。我注意到有很多 UTF-8 字符,所以我将它转换为最接近的真实字符。但是在我提取它以查看前 100000 行之后,我注意到需要完成更多的 UTF-8 转换,我正在手动转换它们,这非常详尽。我想知道是否有比手动输入所有内容更简单的方法来执行此操作。这是我的代码:

import re 
 
infile = r"C:\\Users\\Dave\\Desktop\\database\\page-links_en.txt" 
outfile = r"C:\\Users\\Dave\\Desktop\\database\\Complete\\cleanedpagelinks_file.txt" 
 
fin = open(infile) 
fout = open(outfile, "w+") 
 
rex = re.compile(r'/([^/>]+)>') 
 
for line in fin: 
#for word in delete_list: 
#    line = line.replace(word, "") 
line = line.replace("%C3%A9","e") 
line = line.replace("%C3%B3","o") 
line = line.replace("%E2%80%93","-") 
line = line.replace("%C3%A6","e") 
line = line.replace("%C3%A8","e") 
line = line.replace("_"," ") 
line = line.replace("%C3%A0","e") 
line = line.replace("%C3%A1","i") 
line = line.replace("%C5%82","l") 
line = line.replace("%C5%84","n") 
line = line.replace("%C3%BF", "y") 
line = line.replace("%C3%BE", "p") 
line = line.replace("%C3%BD", "y") 
line = line.replace("%C3%BC", "u") 
line = line.replace("%C3%BB", "u") 
line = line.replace("%C3%BA", "u") 
line = line.replace("%C3%B9", "o") 
line = line.replace("%C3%B6", "o") 
line = line.replace("%C3%B5", "o") 
line = line.replace("%C3%B4", "o") 
line = line.replace("%C3%B3", "o") 
line = line.replace("%C3%B2", "o") 
line = line.replace("%C3%B1", "n") 
line = line.replace("%C3%B0", "e") 
line = line.replace("%C3%AC", "i") 
line = line.replace("%C3%AD", "i") 
line = line.replace("%C3%AE", "i") 
line = line.replace("%C3%AF", "i") 
line = line.replace("%C3%81","A") 
line = line.replace("%C3%82","A") 
line = line.replace("%C3%83","A") 
line = line.replace("%C3%84","A") 
line = line.replace("%C3%85","A") 
line = line.replace("%C3%86","AE") 
line = line.replace("%C3%87","C") 
line = line.replace("%C3%88","E") 
line = line.replace("%C3%89","E") 
line = line.replace("%C3%8A","E") 
line = line.replace("%C3%8B","E") 
line = line.replace("%C3%8C","I") 
line = line.replace("%C3%8D","I") 
line = line.replace("%C3%8E","I") 
line = line.replace("%C3%8F","I") 
line = line.replace("%C3%90","D") 
line = line.replace("%C3%91","N") 
line = line.replace("%C3%92","O") 
line = line.replace("%C3%93","O") 
line = line.replace("%C3%94","O") 
line = line.replace("%C3%95","O") 
line = line.replace("%C3%96","O") 
line = line.replace("%C3%98","O") 
line = line.replace("%C3%99","U") 
line = line.replace("%C3%9A","U") 
line = line.replace("%C3%9B","U") 
line = line.replace("%C3%9C","U") 
line = line.replace("%C3%9D","Y") 
line = line.replace("%C3%9F","B") 
line = line.replace("%C3%a0","a") 
line = line.replace("%C3%a1","a") 
line = line.replace("%C3%a2","a") 
line = line.replace("%C3%a3","a") 
line = line.replace("%C3%a4","a") 
line = line.replace("%C3%a5","a") 
line = line.replace("%C3%a6","ae") 
line = line.replace("%C3%a7","c") 
line = line.replace("%C3%a8","e") 
line = line.replace("%C3%a9","e") 
line = line.replace("%C3%aa","e") 
line = line.replace("%C3%ab","e") 
 
 
match = rex.search(line) 
if match: 
    newline = match.group(1) 
else: newline = '' 
fout.write(newline + '\n') 
fin.close() 
fout.close() 

正如您在我的代码中看到的那样,我正在手动替换为真实的字符值。 这是我意识到需要转换的文本文件中的示例行。

B%E1%BA%A3o %C4%90%E1%BA%A1i 

请您参考如下方法:

你可以使用 unidecode与 urllib.parse.unquote :

In [8]: from unidecode import  unidecode 
 
In [9]: from urllib.parse import unquote 
 
In [10]: unidecode(unquote("Gotterd%C3%A4mmerung")) 
Out[10]: 'Gotterdammerung' 

unidecode 会将非 ascii 字符转换为它们的 ascii 等效字符。