Skip to main content
 首页 » 编程设计

python之在 Mac OS X 上使用 NLTK 在 Celery 线程中使用 python-javabridge JVM 时出现 fatal error

2025年05月04日22zhujiabin

我正在使用 Python wrapper for Weka这是基于 python-javabridge .我有一项很长的任务要执行,因此,我正在使用 Celery这样做。问题是我得到

A fatal error has been detected by the Java Runtime Environment: 
 
  SIGSEGV (0xb) at pc=0x00007fff91a3c16f, pid=11698, tid=3587 
 
JRE version:  (8.0_31-b13) (build ) 
Java VM: Java HotSpot(TM) 64-Bit Server VM (25.31-b07 mixed mode bsd-amd64 compressed oops) 
Problematic frame: 
C  [libdispatch.dylib+0x616f]  _dispatch_async_f_slow+0x18b 
 
Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again 
 
If you would like to submit a bug report, please visit: 
    http://bugreport.java.com/bugreport/crash.jsp 
The crash happened outside the Java Virtual Machine in native code. 
See problematic frame for where to report the bug. 

在线程内启动 JVM 时。为此使用了这两行代码(来自 weka.core.jvm ):

javabridge.start_vm(run_headless=True) 
javabridge.attach() 

据我了解,这可能是由于 JVM 未附加到 Celery 线程。但是,javabridge.attach() 确实在其中运行。

我错过了什么?


编辑:我确定了导致问​​题的代码。它与 NLTK 有关分词器。以下代码(根据 Vebjorn's answer )将重现错误:

# hello.py 
from nltk.tokenize import RegexpTokenizer 
import javabridge 
from celery import Celery 
 
app = Celery('hello', broker='amqp://guest@localhost//', backend='amqp') 
 
started = False     
 
@app.task 
def hello(): 
    global started 
    if not started: 
        print 'Starting the VM' 
        javabridge.start_vm(run_headless=True) 
        started = True 
 
    sentence = "This is a sentence with some numbers like 1, 2 or and some weird symbols like @, $ or ! :)" 
    tokenizer = RegexpTokenizer(r'\w+') 
    tokenized_sentence = tokenizer.tokenize(sentence.lower()) 
    print "Tokens:", tokenized_sentence 
 
    return javabridge.run_script('java.lang.String.format("Hello, %s!", greetee);', 
                             dict(greetee='world')) 

在不启动 JVM 的情况下,代码可以正常运行。它在不作为 Celery 任务运行时也有效。我不明白为什么它会崩溃。


编辑 2: 它实际上可以在干净的 Ubuntu 环境 ( Dockerized ) 中运行,但不能在 Mac OS X Yosemite (v10.3) 上运行。


编辑 3: 如评论中所述,如果 from nltk.tokenize import RegexpTokenizer 在任务包装器内完成,即在 hello () 函数。

请您参考如下方法:

默认情况下,Celery 启动四个独立的工作进程。 (请参阅 celery worker 的 -c 命令行选项。)您需要确保在所有这些中启动 JVM。这个例子对我有用:

# hello.py 
import os 
import threading 
from celery import Celery 
import javabridge 
 
app = Celery('hello', broker='amqp://guest@localhost//', backend='amqp') 
 
started = False 
 
@app.task 
def hello(): 
    global started 
    if not started: 
        print 'Starting the VM' 
        javabridge.start_vm(run_headless=True) 
        started = True 
    return javabridge.run_script('java.lang.String.format("Hello, %s!", greetee);', 
                                 dict(greetee='world')) 

和

# client.py 
from hello import hello 
 
r = hello.delay() 
print r.get(timeout=1) 
  1. 在原始 Ubuntu 14.04 机器上安装:

    $ sudo apt-get update -y 
    $ sudo apt-get install -y openjdk-7-jdk python-pip python-numpy python-dev rabbitmq-server 
    $ sudo pip install celery javabridge 
    $ sudo /etc/init.d/rabbitmq-server start 
    
  2. 开始工作:

    $ celery -A hello worker 
    ... 
     -------------- celery@a7cc1bedc40d v3.1.17 (Cipater) 
    ---- **** -----  
    --- * ***  * -- Linux-3.16.7-tinycore64-x86_64-with-Ubuntu-14.04-trusty 
    -- * - **** ---  
    - ** ---------- [config] 
    - ** ---------- .> app:         hello:0x7f5464766b50 
    - ** ---------- .> transport:   amqp://guest:**@localhost:5672// 
    - ** ---------- .> results:     amqp 
    - *** --- * --- .> concurrency: 4 (prefork) 
    -- ******* ----  
    --- ***** ----- [queues] 
     -------------- .> celery           exchange=celery(direct) key=celery 
     
     
    [2015-04-21 10:04:31,262: WARNING/MainProcess] celery@a7cc1bedc40d ready. 
    
  3. 在另一个窗口中,运行客户端五次:

     $ python client.py  
     Hello, world! 
     $ python client.py  
     Hello, world! 
     $ python client.py  
     Hello, world! 
     $ python client.py  
     Hello, world! 
     $ python client.py  
     Hello, world! 
    
  4. 在工作窗口中观察,JVM 在来自客户端的前四次调用(转到四个不同的进程)上启动,但在第五次调用时没有启动:

    [2015-04-21 10:05:53,491: WARNING/Worker-1] Starting the VM 
    [2015-04-21 10:05:55,028: WARNING/Worker-2] Starting the VM 
    [2015-04-21 10:05:56,411: WARNING/Worker-3] Starting the VM 
    [2015-04-21 10:05:57,318: WARNING/Worker-4] Starting the VM