Skip to main content
 首页 » 编程设计

python之scikit 学习 : update countvectorizer after selecting k best features

2025年05月04日18unruledboy

我有一个包含大量特征的计数向量化器,我希望能够从转换后的集中选择 k 个最佳特征,然后更新 count_vectorizer 以仅包含这些特征。这可能吗?

import pandas as pd 
import numpy as np 
import scipy as sp 
import scipy.stats as ss 
import re 
from sklearn.feature_extraction.text import CountVectorizer 
from sklearn.ensemble import RandomForestClassifier 
 
merge=re.compile('\*\|.+?\|\*') 
def stripmerge(sub): 
    for i in merge.findall(sub): 
        j=i 
        j=j.replace('*|','mcopen') 
        j=j.replace('|*','mcclose') 
        j=re.sub('[^0-9a-zA-Z]','',j) 
        sub=sub.replace(i,j) 
    return sub 
 
input=pd.read_csv('subject_tool_test_23.csv') 
input.subject[input.subject.isnull()]=' ' 
 
 
subjects=np.asarray([stripmerge(i) for i in input.subject]) 
count_vectorizer = CountVectorizer(strip_accents='unicode', ngram_range=(1,1), binary=True, stop_words='english', max_features=500) 
counts=count_vectorizer.fit_transform(subjects) 
 
#see the first output example here 
 
from sklearn.feature_selection import SelectKBest 
from sklearn.feature_selection import chi2 
 
good=np.asarray(input.unique_open_performance>0) 
 
count_new = SelectKBest(chi2, k=250).fit_transform(counts, good) 

第一个输出示例,特征有意义

>>> counts[1] 
<1x500 sparse matrix of type '<type 'numpy.int64'>' 
    with 3 stored elements in Compressed Sparse Row format> 
>>> subjects[1] 
"Lake Group Media's Thursday Target" 
>>> count_vectorizer.inverse_transform(counts[1]) 
[array([u'group', u'media', u'thursday'],  
      dtype='<U18')] 

第二个输出示例,特征不再匹配。

>>> count_new = SelectKBest(chi2, k=250).fit_transform(counts, good) 
>>> count_new.shape 
(992979, 250) 
>>> count_new[1] 
<1x250 sparse matrix of type '<type 'numpy.int64'>' 
    with 2 stored elements in Compressed Sparse Row format> 
>>> count_vectorizer.inverse_transform(count_new[1]) 
[array([u'independence', u'easy'],  
      dtype='<U18')] 
>>> subjects[1] 
"Lake Group Media's Thursday Target" 

有没有办法将特征选择结果应用到我的计数向量化器,以便我可以生成仅包含重要特征的新向量?

请您参考如下方法:

我解决这个问题的方法是运行特征选择,确定从原始集合中选择了哪些列,从中创建一个字典,然后运行一个仅限于该字典的新计数向量化器。处理大型数据集需要更长的时间,但它确实有效。

ch2 = SelectKBest(chi2, k = 3000) 
 
count_new = ch2.fit_transform(counts, good) 
dict=np.asarray(count_vectorizer.get_feature_names())[ch2.get_support()] 
count_vectorizer=CountVectorizer(strip_accents='unicode', ngram_range=(1,1), binary=True,  vocabulary=dict)