Skip to main content
 首页 » 编程设计

python之如何使用 NLTK snowball 词干提取器来提取西类牙语单词列表 Python

2025年05月04日10pengyingh

我正在尝试使用 NLTK 雪球词干提取器来词干西类牙语,但我遇到了一些我不知道的编码问题。

这是我要操作的例句:

En diciembre, los precios de la energía subieron un 1,4 por ciento, los de la vivienda aumentaron un 0,1 por ciento y los precios de la vestimenta se mantuvieron sin cambios, mientras que los de los automóviles nuevos bajaron un 0,1 por ciento y los de los pasajes de avión cayeron el 0,7 por ciento.

首先,我使用代码从 xml 文件中读取句子:

from nltk.stem.snowball import SnowballStemmer 
import xml.etree.ElementTree as ET 
 
stemmer = SnowballStemmer("spanish") 
sentence = ET.tostring(context, encoding='utf-8', method="text").lower() 

然后在将句子标记化以获得单词列表之后,我尝试对每个单词进行词干处理:

stem = stemmer.stem(words[headIndex - index]) 

错误来自这一行:

Traceback (most recent call last): 
  File "main.py", line 150, in <module> 
    main() 
  File "main.py", line 142, in main 
    vectorDict, vocabulary = englishXml(language) 
  File "main.py", line 86, in englishXml 
    stem = stemmer.stem(words[headIndex - index]) 
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nltk/stem/snowball.py", line 3404, in stem 
    r1, r2 = self._r1r2_standard(word, self.__vowels) 
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nltk/stem/snowball.py", line 232, in _r1r2_standard 
    if word[i] not in vowels and word[i-1] in vowels: 
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 0: ordinal not in range(128) 

我也尝试从没有“utf-8”编码的xml文件中读取句子,但问题是“.lower()”在那种情况下不起作用:

sentence = ET.tostring(context, method="text").lower() 

在这种情况下错误变为:

Traceback (most recent call last): 
  File "main.py", line 154, in <module> 
    main() 
  File "main.py", line 146, in main 
    vectorDict, vocabulary = englishXml(language) 
  File "main.py", line 63, in englishXml 
    sentence = ET.tostring(context, method="text").lower() 
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/xml/etree/ElementTree.py", line 1126, in tostring 
    ElementTree(element).write(file, encoding, method=method) 
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/xml/etree/ElementTree.py", line 814, in write 
    _serialize_text(write, self._root, encoding) 
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/xml/etree/ElementTree.py", line 1006, in _serialize_text 
    write(part.encode(encoding)) 
UnicodeEncodeError: 'ascii' codec can't encode character u'\xed' in position 18: ordinal not in range(128) 

提前致谢!

请您参考如下方法:

尝试在词干提取之前添加这个

sentence = sentence.decode('utf8')