Skip to main content
 首页 » 编程设计

python之NLTK 标记化搭配

2025年05月04日12zhujiabin

我正在使用 NLTK 并想根据搭配对文本进行分词:例如,“New York”应该是单个分词,而简单的分词会拆分“New”和“York”。

我知道如何找到搭配以及如何标记化,但找不到如何将两者结合起来...

谢谢。

请您参考如下方法:

方法,似乎适合您,称为命名实体识别。有许多链接专门介绍 NLTK for Named Entity Recognition。我只是给你一个来自 here 的例子

from nltk import sent_tokenize, word_tokenize, pos_tag, ne_chunk 
 
 
def extract_entities(text): 
    entities = [] 
    for sentence in sent_tokenize(text): 
        chunks = ne_chunk(pos_tag(word_tokenize(sentence))) 
        entities.extend([chunk for chunk in chunks if hasattr(chunk, 'node')]) 
    return entities 
 
 
if __name__ == '__main__': 
    text = """ 
A multi-agency manhunt is under way across several states and Mexico after 
police say the former Los Angeles police officer suspected in the murders of a 
college basketball coach and her fiancé last weekend is following through on 
his vow to kill police officers after he opened fire Wednesday night on three 
police officers, killing one. 
"In this case, we're his target," Sgt. Rudy Lopez from the Corona Police 
Department said at a press conference. 
The suspect has been identified as Christopher Jordan Dorner, 33, and he is 
considered extremely dangerous and armed with multiple weapons, authorities 
say. The killings appear to be retribution for his 2009 termination from the 
 Los Angeles Police Department for making false statements, authorities say. 
Dorner posted an online manifesto that warned, "I will bring unconventional 
and asymmetrical warfare to those in LAPD uniform whether on or off duty." 
""" 
    print extract_entities(text) 

输出:

[Tree('GPE', [('Mexico', 'NNP')]), Tree('GPE', [('Los', 'NNP'), ('Angeles', 'NNP')]), Tree('PERSON', [('Rudy', 'NNP')]), Tree('ORGANIZATION', [('Lopez', 'NNP')]), Tree('ORGANIZATION', [('Corona', 'NNP')]), Tree('PERSON', [('Christopher', 'NNP'), ('Jordan', 'NNP'), ('Dorner', 'NNP')]), Tree('GPE', [('Los', 'NNP'), ('Angeles', 'NNP')]), Tree('PERSON', [('Dorner', 'NNP')]), Tree('GPE', [('LAPD', 'NNP')])] 

另一种方法——对两者之间的信息重叠使用不同的度量 随机变量,例如 Mutual Information、Pointwise Mutual 信息、t检验等。 << Foundations of Statistical Natural Language Processing中有很好的介绍>> 作者:Christopher D. Manning 和 Hinrich Schütze。第 5 章搭配可供下载。这link - 使用 NLTK 提取搭配的示例。