我正在尝试使用 Stanford NER 和 Stanford POS Tagger 来解析大约 23000 个文档。我已经使用以下伪代码实现了它 -
`for each in document:
eachSentences = PunktTokenize(each)
#code to generate NER Tagger
#code to generate POS Taggers on the above output`
对于具有 15 GB RAM 的 4 核机器，NER 的运行时间约为 945 小时。我试图通过使用“线程”库来加强事情，但我收到以下错误-
`Exception in thread Thread-2:
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
self.run()
File "/usr/lib/python2.7/threading.py", line 754, in run
self.__target(*self.__args, **self.__kwargs)
File "removeStopWords.py", line 75, in partofspeechRecognition
listOfRes_new = namedEntityRecognition(listRes[min:max])
File "removeStopWords.py", line 63, in namedEntityRecognition
listRes_ner.append(namedEntityRecognitionResume(eachResSentence))
File "removeStopWords.py", line 50, in namedEntityRecognitionResume
ner2Tags = ner2.tag(each.title().split())
File "/home/datascience/pythonEnv/local/lib/python2.7/site-packages/nltk/tag/stanford.py", line 71, in tag
return sum(self.tag_sents([tokens]), [])
File "/home/datascience/pythonEnv/local/lib/python2.7/site-packages/nltk/tag/stanford.py", line 98, in tag_sents
os.unlink(self._input_file_path)
OSError: [Errno 2] No such file or directory: '/tmp/tmpvMNqwB'`
我使用的是 NLTK 版本 - 3.2.1，Stanford NER,POS - 3.7.0 jar 文件，以及线程模块。据我所知，这可能是由于/tmp 上的线程锁定。 如果我错了，请纠正我，还有什么是使用线程运行上述内容的最佳方式或更好的实现方式。
我正在使用 3 Class Classifier for NER和 Maxent POS Tagger
附言- 请忽略 Python 文件的名称，我仍然没有从原文中删除停用词或标点符号。
编辑 - 使用 cProfile，并按累计时间排序，我得到了以下前 20 个调用
600792 function calls (595912 primitive calls) in 60.795 seconds
Ordered by: cumulative time
List reduced from 3357 to 20 due to restriction <20>
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 60.811 60.811 removeStopWords.py:1(<module>)
1 0.000 0.000 58.923 58.923 removeStopWords.py:76(partofspeechRecognition)
28 0.001 0.000 58.883 2.103 /home/datascience/pythonEnv/local/lib/python2.7/site-packages/nltk/tag/stanford.py:69(tag)
28 0.004 0.000 58.883 2.103 /home/datascience/pythonEnv/local/lib/python2.7/site-packages/nltk/tag/stanford.py:73(tag_sents)
28 0.001 0.000 56.927 2.033 /home/datascience/pythonEnv/local/lib/python2.7/site-packages/nltk/internals.py:63(java)
141 0.001 0.000 56.532 0.401 /usr/lib/python2.7/subprocess.py:769(communicate)
140 0.002 0.000 56.530 0.404 /usr/lib/python2.7/subprocess.py:1408(_communicate)
140 0.008 0.000 56.492 0.404 /usr/lib/python2.7/subprocess.py:1441(_communicate_with_poll)
400 56.474 0.141 56.474 0.141 {built-in method poll}
1 0.001 0.001 43.522 43.522 removeStopWords.py:69(partofspeechRecognitionRes)
1 0.000 0.000 15.401 15.401 removeStopWords.py:62(namedEntityRecognition)
1 0.001 0.001 15.367 15.367 removeStopWords.py:46(namedEntityRecognitionRes)
141 0.004 0.000 2.302 0.016 /usr/lib/python2.7/subprocess.py:651(__init__)
141 0.020 0.000 2.287 0.016 /usr/lib/python2.7/subprocess.py:1199(_execute_child)
56 0.002 0.000 1.933 0.035 /home/datascience/pythonEnv/local/lib/python2.7/site-packages/nltk/internals.py:38(config_java)
56 0.001 0.000 1.931 0.034 /home/datascience/pythonEnv/local/lib/python2.7/site-packages/nltk/internals.py:599(find_binary)
112 0.002 0.000 1.930 0.017 /home/datascience/pythonEnv/local/lib/python2.7/site-packages/nltk/internals.py:582(find_binary_iter)
118 0.009 0.000 1.928 0.016 /home/datascience/pythonEnv/local/lib/python2.7/site-packages/nltk/internals.py:453(find_file_iter)
1 0.001 0.001 1.318 1.318 /usr/lib/python2.7/pickle.py:1383(load)
1 0.046 0.046 1.317 1.317 /usr/lib/python2.7/pickle.py:851(load)
请您参考如下方法:
似乎 Python 包装器是这里的罪魁祸首。 Java 实现没有那么多时间。这大约需要@Gabor Angeli 提到的内容。试试吧。
希望对您有所帮助!
