我有一个包含大量特征的计数向量化器,我希望能够从转换后的集中选择 k 个最佳特征,然后更新 count_vectorizer 以仅包含这些特征。这可能吗?
import pandas as pd
import numpy as np
import scipy as sp
import scipy.stats as ss
import re
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.ensemble import RandomForestClassifier
merge=re.compile('\*\|.+?\|\*')
def stripmerge(sub):
for i in merge.findall(sub):
j=i
j=j.replace('*|','mcopen')
j=j.replace('|*','mcclose')
j=re.sub('[^0-9a-zA-Z]','',j)
sub=sub.replace(i,j)
return sub
input=pd.read_csv('subject_tool_test_23.csv')
input.subject[input.subject.isnull()]=' '
subjects=np.asarray([stripmerge(i) for i in input.subject])
count_vectorizer = CountVectorizer(strip_accents='unicode', ngram_range=(1,1), binary=True, stop_words='english', max_features=500)
counts=count_vectorizer.fit_transform(subjects)
#see the first output example here
from sklearn.feature_selection import SelectKBest
from sklearn.feature_selection import chi2
good=np.asarray(input.unique_open_performance>0)
count_new = SelectKBest(chi2, k=250).fit_transform(counts, good)
第一个输出示例,特征有意义
>>> counts[1]
<1x500 sparse matrix of type '<type 'numpy.int64'>'
with 3 stored elements in Compressed Sparse Row format>
>>> subjects[1]
"Lake Group Media's Thursday Target"
>>> count_vectorizer.inverse_transform(counts[1])
[array([u'group', u'media', u'thursday'],
dtype='<U18')]
第二个输出示例,特征不再匹配。
>>> count_new = SelectKBest(chi2, k=250).fit_transform(counts, good)
>>> count_new.shape
(992979, 250)
>>> count_new[1]
<1x250 sparse matrix of type '<type 'numpy.int64'>'
with 2 stored elements in Compressed Sparse Row format>
>>> count_vectorizer.inverse_transform(count_new[1])
[array([u'independence', u'easy'],
dtype='<U18')]
>>> subjects[1]
"Lake Group Media's Thursday Target"
有没有办法将特征选择结果应用到我的计数向量化器,以便我可以生成仅包含重要特征的新向量?
请您参考如下方法:
我解决这个问题的方法是运行特征选择,确定从原始集合中选择了哪些列,从中创建一个字典,然后运行一个仅限于该字典的新计数向量化器。处理大型数据集需要更长的时间,但它确实有效。
ch2 = SelectKBest(chi2, k = 3000)
count_new = ch2.fit_transform(counts, good)
dict=np.asarray(count_vectorizer.get_feature_names())[ch2.get_support()]
count_vectorizer=CountVectorizer(strip_accents='unicode', ngram_range=(1,1), binary=True, vocabulary=dict)
