When embedding CPython in Java, why does this hang?

I'm embedding CPython into a JVM using Jepp, but when I run

import numpy; numpy.finfo(float)

the process hangs. gdb says something's blocking a semaphore/lock acquisition, and the stack trace suggests something floating point-related:

$ LD_PRELOAD=/usr/lib/libpython2.6.so gdb --args java -jar jepp.jar -i /tmp/go.py
GNU gdb (GDB) 7.1-ubuntu
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/bin/java...(no debugging symbols found)...done.
(gdb) r
Starting program: /usr/bin/java -jar jepp.jar -i /tmp/go.py
[Thread debugging using libthread_db enabled]
process 20736 is executing new program: /usr/lib/jvm/java-6-sun-1.6.0.26/jre/bin/java
[Thread debugging using libthread_db enabled]
[New Thread 0x7ffff58fd700 (LWP 20739)]
[New Thread 0x7ffff16ca700 (LWP 20740)]
[New Thread 0x7ffff15c9700 (LWP 20741)]
[New Thread 0x7ffff14c8700 (LWP 20742)]
[New Thread 0x7ffff13c7700 (LWP 20743)]
[New Thread 0x7ffff0bf5700 (LWP 20744)]
[New Thread 0x7ffff0af4700 (LWP 20745)]
[New Thread 0x7ffff09f3700 (LWP 20746)]
[New Thread 0x7ffff08b3700 (LWP 20747)]
[New Thread 0x7ffff07b2700 (LWP 20748)]
[New Thread 0x7ffff06b1700 (LWP 20749)]
[New Thread 0x7ffff05b0700 (LWP 20750)]
[New Thread 0x7ffff04af700 (LWP 20751)]
>>> import numpy; numpy.finfo(float)
^C
Program received signal SIGINT, Interrupt.
0x00007ffff771703d in pthread_join () from /lib/libpthread.so.0
(gdb) bt
#0  0x00007ffff771703d in pthread_join () from /lib/libpthread.so.0
#1  0x000000004000591c in ContinueInNewThread ()
#2  0x0000000040001ef4 in main ()
(gdb) i threads
  14 Thread 0x7ffff04af700 (LWP 20751)  0x00007ffff771abc9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
  13 Thread 0x7ffff05b0700 (LWP 20750)  0x00007ffff771a85c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
  12 Thread 0x7ffff06b1700 (LWP 20749)  0x00007ffff771a85c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
  11 Thread 0x7ffff07b2700 (LWP 20748)  0x00007ffff771a85c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
  10 Thread 0x7ffff08b3700 (LWP 20747)  0x00007ffff771cb50 in sem_wait ()
   from /lib/libpthread.so.0
  9 Thread 0x7ffff09f3700 (LWP 20746)  0x00007ffff771a85c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
  8 Thread 0x7ffff0af4700 (LWP 20745)  0x00007ffff771a85c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
  7 Thread 0x7ffff0bf5开发者_运维知识库700 (LWP 20744)  0x00007ffff771abc9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
  6 Thread 0x7ffff13c7700 (LWP 20743)  0x00007ffff771a85c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
  5 Thread 0x7ffff14c8700 (LWP 20742)  0x00007ffff771a85c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
  4 Thread 0x7ffff15c9700 (LWP 20741)  0x00007ffff771a85c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
  3 Thread 0x7ffff16ca700 (LWP 20740)  0x00007ffff771a85c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
  2 Thread 0x7ffff58fd700 (LWP 20739)  0x00007ffff771cb50 in sem_wait ()
   from /lib/libpthread.so.0
* 1 Thread 0x7ffff7ed8700 (LWP 20736)  0x00007ffff771703d in pthread_join ()
   from /lib/libpthread.so.0
(gdb) thread 2
[Switching to thread 2 (Thread 0x7ffff58fd700 (LWP 20739))]#0  0x00007ffff771cb50 in sem_wait () from /lib/libpthread.so.0
(gdb) bt
#0  0x00007ffff771cb50 in sem_wait () from /lib/libpthread.so.0
#1  0x00007ffff7a4db28 in PyThread_acquire_lock ()
   from /usr/lib/libpython2.6.so
#2  0x00007ffff7a1cbb4 in PyEval_RestoreThread () from /usr/lib/libpython2.6.so
#3  0x00007ffff7a41008 in PyGILState_Ensure () from /usr/lib/libpython2.6.so
#4  0x00007fffeb1e1ff6 in _error_handler (method=-334489136, errobj=0x80,
    errtype=0x0, retstatus=-1, first=0x0)
    at numpy/core/src/umath/ufunc_object.c:95
#5  0x00007fffeb1e2379 in PyUFunc_handlefperr (errmask=521,
    errobj=0x7fffec502950, retstatus=4, first=0x7ffff58fa5fc)
    at numpy/core/src/umath/ufunc_object.c:189
#6  0x00007fffeb1e7cd1 in PyUFunc_GenericFunction (self=<value optimized out>,
    args=<value optimized out>, kwds=<value optimized out>, op=0x7ffff58faaa0)
    at numpy/core/src/umath/ufunc_object.c:2734
#7  0x00007fffeb1e7ff0 in ufunc_generic_call (self=0x7fffec1df790,
    args=<value optimized out>, kwds=<value optimized out>)
    at numpy/core/src/umath/ufunc_object.c:4180
#8  0x00007ffff797c2e3 in PyObject_Call () from /usr/lib/libpython2.6.so
#9  0x00007ffff797ed9f in _PyObject_CallFunction_SizeT ()
   from /usr/lib/libpython2.6.so
#10 0x00007ffff797c9aa in ?? () from /usr/lib/libpython2.6.so
#11 0x00007ffff797d698 in PyNumber_Multiply () from /usr/lib/libpython2.6.so
#12 0x00007ffff7a1f2bc in PyEval_EvalFrameEx () from /usr/lib/libpython2.6.so
#13 0x00007ffff7a22928 in PyEval_EvalFrameEx () from /usr/lib/libpython2.6.so
#14 0x00007ffff7a23d60 in PyEval_EvalCodeEx () from /usr/lib/libpython2.6.so
#15 0x00007ffff79a9d60 in ?? () from /usr/lib/libpython2.6.so
#16 0x00007ffff797c2e3 in PyObject_Call () from /usr/lib/libpython2.6.so
#17 0x00007ffff798dcef in ?? () from /usr/lib/libpython2.6.so
#18 0x00007ffff797c2e3 in PyObject_Call () from /usr/lib/libpython2.6.so
#19 0x00007ffff79e2cd0 in ?? () from /usr/lib/libpython2.6.so
#20 0x00007ffff79d8f98 in ?? () from /usr/lib/libpython2.6.so
#21 0x00007ffff797c2e3 in PyObject_Call () from /usr/lib/libpython2.6.so
#22 0x00007ffff7a21771 in PyEval_EvalFrameEx () from /usr/lib/libpython2.6.so
#23 0x00007ffff7a23d60 in PyEval_EvalCodeEx () from /usr/lib/libpython2.6.so
#24 0x00007ffff7a220ab in PyEval_EvalFrameEx () from /usr/lib/libpython2.6.so
#25 0x00007ffff7a23d60 in PyEval_EvalCodeEx () from /usr/lib/libpython2.6.so
#26 0x00007ffff79a9d60 in ?? () from /usr/lib/libpython2.6.so
#27 0x00007ffff797c2e3 in PyObject_Call () from /usr/lib/libpython2.6.so
#28 0x00007ffff79d928d in ?? () from /usr/lib/libpython2.6.so
#29 0x00007ffff79d8f53 in ?? () from /usr/lib/libpython2.6.so
#30 0x00007ffff797c2e3 in PyObject_Call () from /usr/lib/libpython2.6.so
#31 0x00007ffff7a21771 in PyEval_EvalFrameEx () from /usr/lib/libpython2.6.so
#32 0x00007ffff7a23d60 in PyEval_EvalCodeEx () from /usr/lib/libpython2.6.so
#33 0x00007ffff7a23e32 in PyEval_EvalCode () from /usr/lib/libpython2.6.so
#34 0x00007ffff7a41d1c in PyRun_StringFlags () from /usr/lib/libpython2.6.so
#35 0x00007ffff019d1e7 in pyembed_eval (env=0x401121d0,
    _jepThread=140737154335376,
    str=0x7fffec117390 "import numpy; numpy.finfo(float)") at pyembed.c:929
#36 0x00007ffff019c29a in Java_jep_Jep_eval (env=0x401121d0,
---Type <return> to continue, or q <return> to quit---
    obj=<value optimized out>, tstate=140737154335376, jstr=0x7ffff58fbdf8)
    at jep.c:137
#37 0x00007ffff1a67d6e in ?? ()
#38 0x0000000000000000 in ?? ()
(gdb) thr 10
[Switching to thread 10 (Thread 0x7ffff08b3700 (LWP 20747))]#0  0x00007ffff771cb50 in sem_wait () from /lib/libpthread.so.0
(gdb) bt
#0  0x00007ffff771cb50 in sem_wait () from /lib/libpthread.so.0
#1  0x00007ffff600eea5 in check_pending_signals(bool) ()
   from /usr/lib/jvm/java-6-sun-1.6.0.26/jre/lib/amd64/server/libjvm.so
#2  0x00007ffff600745c in signal_thread_entry(JavaThread*, Thread*) ()
   from /usr/lib/jvm/java-6-sun-1.6.0.26/jre/lib/amd64/server/libjvm.so
#3  0x00007ffff61170f1 in JavaThread::run() ()
   from /usr/lib/jvm/java-6-sun-1.6.0.26/jre/lib/amd64/server/libjvm.so
#4  0x00007ffff600eadf in java_start(Thread*) ()
   from /usr/lib/jvm/java-6-sun-1.6.0.26/jre/lib/amd64/server/libjvm.so
#5  0x00007ffff77159ca in start_thread () from /lib/libpthread.so.0
#6  0x00007ffff726e70d in clone () from /lib/libc.so.6
#7  0x0000000000000000 in ?? ()

From delta debugging, it appears the following loop runs 9 times (should run max_iterN=10000 times), and then things halt on the temp = z*t line:

   for _ in xrange(max_iterN):
       y = z
       z = y*y
       a = z*one # Check here for underflow
       temp = z*t
       if any(a+a == zero) or any(abs(z)>=y):
           break
       temp1 = temp * betain
       if any(temp1*beta == z):
           break
       i = i + 1
       k = k + k
   else:
       print 'baz'
       raise RuntimeError, msg % (_, one.dtype)

Any ideas? To fully reproduce:

mkdir /tmp/test/

cd /tmp/test/

wget 'http://sourceforge.net/projects/numpy/files/NumPy/1.6.1/numpy-1.6.1.tar.gz'

tar xzf numpy-1.6.1.tar.gz

cd numpy-1.6.1/

python setup.py install --prefix=/tmp/test/env

cd

git clone https://github.com/mrj0/jepp.git

cd jepp/jep/

export JAVA_HOME=/usr/lib/jvm/java-6-sun/

./configure --prefix=/tmp/jepp && make && make install

touch /tmp/go.py

PYTHONPATH=/tmp/test/env/lib/python2.6/site-packages/
LD_PRELOAD=/usr/lib/libpython2.6.so java -jar jepp.jar -i /tmp/go.py

>>> import numpy
>>> numpy.finfo(float)

I've been able to reproduce this on my machines, which are Ubuntu x86_64 boxes. It's been a while but IIRC I don't experience this with the Ubuntu numpy package, only with a from-source build as instructed above.

Thanks in advance. This has been driving me up the wall.

See this post by John Wright who ran into the same problem: http://mail.scipy.org/pipermail/numpy-discussion/2009-July/044046.html

Basically, numpy is using the Python C api in a way that's incompatible with Jep. Jep uses sub interpreters to allow multiple interpreter environments. It could be possible to change Jep so that it uses the GIL state functions, but that would really limit Jep's functionality in Java.

Is there any particular reason you cant use Jython ? My understand is that is pretty much a real subsitute for the real thing.

If you have more than one thread calling into CPython, and one of those threads is "busy", then the global lock will probably be locked, and CPython has no choice except to hang until that lock is released.

What threads do you have calling into python - additionally - what is python doing in those threads?