python之如何使用编译的正则表达式和/或列表理解为机器学习管道准备文本

我正在尝试在 Python 函数中为机器学习管道准备文本，但我无法获得正确的输出。所以，我想将所有单词小写，用空格替换符号，删除符号并从 nltk 中删除停用词。我尝试了从列表理解到正则表达式模式匹配的各种不同方法，但我仍然做不好。请帮忙!以下是必要的导入和基本功能:

import nltk 
nltk.download('stopwords') 
from nltk.corpus import stopwords 
 
import re 

函数如下:

REPLACE_BY_SPACE_RE = re.compile('[/(){}\[\]\|@,;]') 
BAD_SYMBOLS_RE = re.compile('[^0-9a-z #+_]') 
STOPWORDS = set(stopwords.words('english')) 
 
def text_prepare(text): 
""" 
    text: a string 
 
    return: modified initial string 
""" 
 
lower = text.lower() # lowercase text 
space_replace = REPLACE_BY_SPACE_RE.sub(" ",lower) #replace REPLACE_BY_SPACE_RE symbols by space in text 
nosymb = BAD_SYMBOLS_RE.sub("",space_replace) # delete symbols which are in BAD_SYMBOLS_RE from text 
text = [word for word in nosymb if word not in STOPWORDS] # delete                                                                                                             stopwords from text 
 
return text 

这是一个测试函数:

def test_text_prepare(): 
    examples = ["SQL Server - any equivalent of Excel's CHOOSE function?", 
            "How to free c++ memory vector<int> * arr?"] 
    answers = ["sql server equivalent excels choose function",  
           "free c++ memory vectorint arr"] 
    for ex, ans in zip(examples, answers): 
    if text_prepare(ex) != ans: 
        return "Wrong answer for the case: '%s'" % ex 
return 'Basic tests are passed.' 

这是我的测试结果:

print(test_text_prepare()) 
Wrong answer for the case: 'SQL Server - any equivalent of Excel's CHOOSE function?' 

请您参考如下方法:

您将单词视为 [如果单词不在 STOPWORDS 中则在 nosymb 中逐字逐句] 中的单独符号。此外，您不会去除前导/尾随空格，也不会“缩小”之前清理操作产生的多余空间。

这是一个更新的方法:

def text_prepare(text): 
    """ 
    text: a string 
        return: modified initial string 
    """ 
    lower = text.lower() # lowercase text 
    space_replace = REPLACE_BY_SPACE_RE.sub(" ",lower) #replace REPLACE_BY_SPACE_RE symbols by space in text 
    nosymb = BAD_SYMBOLS_RE.sub("",space_replace) # delete symbols which are in BAD_SYMBOLS_RE from text 
    text = re.sub(r"\s*\b(?:{})\b".format("|".join(STOPWORDS)), "", nosymb) # delete STOPWORDS 
    return re.sub(r" {2,}", " ", text.strip()) 

re.sub(r"\s*\b(?:{})\b".format("|".join(STOPWORDS)), "", nosymb) 部分删除所有作为整个单词匹配的停用词(\b 是单词边界)。

re.sub(r"{2,}", "", text.strip()) 部分修剪字符串并将字符串中的所有空格缩小为单个空格。