openMPI bug using mpirun, i get :

Since 2hours, my openMPI installation is out of order... i don't get why! I got 3 serveur (call clusterX) that was running like a charm until today.

The thing i try was to lanch my MPI job using Nohup command in order to logout my SSH session and let my process run over night.

It worked.

But now, i can't run any mpi jobs, i get all the time this message :

*** buffer overflow detected ***: /mirror/mpiu/Xmipp-2.4-src/bin/xmipp_mpi_angular_project_library terminated
======= Backtrace: =========
/lib/libc.so.6(__fortify_fail+0x37)[0x7f9ceb7b9217]
/lib/libc.so.6(+0xfe0d0)[0x7f9ceb7b80d0]
/lib/libc.so.6(+0xfd539)[0x7f9ceb7b7539]
/lib/libc.so.6(_IO_default_xsputn+0xcc)[0x7f9ceb72fd1c]
/lib/libc.so.6(_IO_vfprintf+0xf3e)[0x7f9ceb7002de]
/lib/libc.so.6(__vsprintf_chk+0x99)[0x7f9ceb7b75d9]
/lib/libc.so.6(__sprintf_chk+0x7f)[0x7f9ceb7b751f]
/mirror/mpiu/Xmipp-2.4-src/lib/libXmippData.so(_ZN8FileName7composeERKSsiS1_+0x1a4)[0x7f9ceda56fb4]
/mirror/mpiu/Xmipp-2.4-src/bin/xmipp_mpi_angular_project_library(_ZN43Prog_mpi_angular_project_library_Parameters3runEv+0x224)[0x40ff74]
/mirror/mpiu/Xmipp-2.4-src/bin/xmipp_mpi_angular_project_library(main+0x180)[0x40b3b0]
/lib/libc.so.6(__libc_start_main+0xfd)[0x7f9ceb6d8c4d]
/mirror/mpiu/Xmipp-2.4-src/bin/xmipp_mpi_angular_project_library[0x40b079]
======= Memory map: ========
00400000-00415000 r-xp 00000000 08:11 3672090                            /mirror/mpiu/Xmipp-2.4-src/bin/xmipp_mpi_angular_project_library
00614000-00615000 r--p 00014000 08:11 3672090                            /mirror/mpiu/Xmipp-2.4-src/bin/xmipp_mpi_angular_project_library
00615000-00616000 rw-p 00015000 08:11 3672090                            /mirror/mpiu/Xmipp-2.4-src/bin/xmipp_mpi_angular_project_library
00616000-00617000 rw-p 00000000 00:00 0 
015ee000-0649b000 rw-p 00000000 00:00 0                                  [heap]
7f9ce02dc000-7f9ce1add000 rw-p 00000000 00:00 0 
7f9ce26de000-7f9ce66e0000 rw-s 00000000 08:11 3697721                    /tmp/openmpi-sessions-mpiu@cluster1_0/58742/1/shared_mem_pool.cluster1
7f9ce66e0000-7f9ce66ed000 r-xp 00000000 08:11 792400                     /usr/lib/openmpi/lib/openmpi/mca_osc_rdma.so
7f9ce66ed000-7f9ce68ec000 ---p 0000d000 08:11 792400                     /usr/lib/openmpi/lib/openmpi/mca_osc_rdma.so
7f9ce68ec000-7f9ce68ed000 r--p 0000c000 08:11 792400                     /usr/lib/openmpi/lib/openmpi/mca_osc_rdma.so
7f9ce68ed000-7f9ce68ee000 rw-p 0000d000 08:11 792400                     /usr/lib/openmpi/lib/openmpi/mca_osc_rdma.so
7f9ce68ee000-7f9ce68f8000 r-xp 00000000 08:11 792399                     /usr/lib/openmpi/lib/openmpi/mca_osc_pt2pt.so
7f9ce68f8000-7f9ce6af7000 ---p 0000a000 08:11 792399                     /usr/lib/openmpi/lib/openmpi/mca_osc_pt2pt.so
7f9ce6af7000-7f9ce6af8000 r--p 00009000 08:11 792399                     /usr/lib/openmpi/lib/openmpi/mca_osc_pt2pt.so
7f9ce6af8000-7f9ce6af9000 rw-p 0000a000 08:11 792399                     /usr/lib/openmpi/lib/openmpi/mca_osc_pt2pt.so
7f9ce6af9000-7f9ce6b11000 r-xp 00000000 08:11 792373                     /usr/lib/openmpi/lib/openmpi/mca_coll_tuned.so
7f9ce6b11000-7f9ce6d11000 ---p 00018000 08:11 792373                     /usr/lib/openmpi/lib/openmpi/mca_coll_tuned.so
7f9ce6d11000-7f9ce6d12000 r--p 00018000 08:11 792373                     /usr/lib/openmpi/lib/openmpi/mca_coll_tuned.so
7f9ce6d12000-7f9ce6d13000 rw-p 00019000 08:11 792373                     /usr/lib/openmpi/lib/openmpi/mca_coll_tuned.so
7f9ce6d13000-7f9ce6d16000 r-xp 00000000 08:11 792372                     /usr/lib/openmpi/lib/openmpi/mca_coll_sync.so
7f9ce6d16000-7f9ce6f15000 ---p 00003000 08:11 792372                     /usr/lib/openmpi/lib/openmpi/mca_coll_sync.so
7f9ce6f15000-7f9ce6f16000 r--p 00002000 08:11 792372                     /usr/lib/openmpi/lib/openmpi/mca_coll_sync.so
7f9ce6f16000-7f9ce6f17000 rw-p 00003000 08:11 792372                     /usr/lib/openmpi/lib/openmpi/mca_coll_sync.so
7f9ce6f17000-7f9ce6f1c000 r-xp 00000000 08:11 792371                     /usr/lib/openmpi/lib/openmpi/mca_coll_sm.so
7f9ce6f1c000-7f9ce711b000 ---p 00005000 08:11 792371                     /usr/lib/openmpi/lib/openmpi/mca_coll_sm.so
7f9ce711b000-7f9ce711c000 r--p 00004000 08:11 792371                     /usr/lib/openmpi/lib/openmpi/mca_coll_sm.so
7f9ce711c000-7f9ce711d000 rw-p 00005000 08:11 792371                     /usr/lib/openmpi/lib/openmpi/mca_coll_sm.so
7f9ce711d000-7f9ce711f000 r-xp 00000000 08:11 792370                     /usr/lib/openmpi/lib/openmpi/mca_coll_self.so
7f9ce711f000-7f9ce731e000 ---p 00002000 08:11 792370                     /usr/lib/openmpi/lib/openmpi/mca_coll_self.so
7f9ce731e000-7f9ce731f000 r--p 00001000 08:11 792370                     /usr/lib/openmpi/lib/openmpi/mca_coll_self.so
7f9ce731f000-7f9ce7320000 rw-p 00002000 08:11 792370                     /usr/lib/openmpi/lib/openmpi/mca_coll_self.so
7f9ce7320000-7f9ce7323000 r-xp 00000000 08:11 792369                     /usr/lib/openmpi/lib/openmpi/mca_coll_inter.so
7f9ce7323000-7f9ce7522000 ---p 00003000 08:11 792369                     /usr/lib/openmpi/lib/openmpi/mca_coll_inter.so
7f9ce7522000-7f9ce7523000 r--p 00002000 08:11 792369                     /usr/lib/openmpi/lib/openmpi/mca_coll_inter.so
7f9ce7523000-7f9ce7524000 rw-p 00003000 08:11 792369                     /usr/lib/openmpi/lib/openmpi/mca_coll_inter.so
7f9ce7524000-7f9ce7528000 r-xp 00000000 08:11 792368                     /usr/lib/openmpi/lib/openmpi/mca_coll_hierarch.so
7f9ce7528000-7f9ce772700开发者_JAVA技巧0 ---p 00004000 08:11 792368                     /usr/lib/openmpi/lib/openmpi/mca_coll_hierarch.so
7f9ce7727000-7f9ce7728000 r--p 00003000 08:11 792368                     /usr/lib/openmpi/lib/openmpi/mca_coll_hierarch.so
7f9ce7728000-7f9ce7729000 rw-p 00004000 08:11 792368                     /usr/lib/openmpi/lib/openmpi/mca_coll_hierarch.so
7f9ce7729000-7f9ce7731000 r-xp 00000000 08:11 792367                     /usr/lib/openmpi/lib/openmpi/mca_coll_basic.so
7f9ce7731000-7f9ce7930000 ---p 00008000 08:11 792367                     /usr/lib/openmpi/lib/openmpi/mca_coll_basic.so
7f9ce7930000-7f9ce7931000 r--p 00007000 08:11 792367                     /usr/lib/openmpi/lib/openmpi/mca_coll_basic.so
7f9ce7931000-7f9ce7932000 rw-p 00008000 08:11 792367                     /usr/lib/openmpi/lib/openmpi/mca_coll_basic.so
7f9ce7932000-7f9ce793d000 r-xp 00000000 08:11 792364                     /usr/lib/openmpi/lib/openmpi/mca_btl_tcp.so
7f9ce793d000-7f9ce7b3c000 ---p 0000b000 08:11 792364                     /usr/lib/openmpi/lib/openmpi/mca_btl_tcp.so
7f9ce7b3c000-7f9ce7b3d000 r--p 0000a000 08:11 792364                     /usr/lib/openmpi/lib/openmpi/mca_btl_tcp.so
7f9ce7b3d000-7f9ce7b3e000 rw-p 0000b000 08:11 792364                     /usr/lib/openmpi/lib/openmpi/mca_btl_tcp.so
7f9ce7b3e000-7f9ce7bbe000 rw-p 00000000 00:00 0 
7f9ce7bbe000-7f9ce7bc4000 r-xp 00000000 08:11 792363                     /usr/lib/openmpi/lib/openmpi/mca_btl_sm.so
7f9ce7bc4000-7f9ce7dc3000 ---p 00006000 08:11 792363                     /usr/lib/openmpi/lib/openmpi/mca_btl_sm.so[cluster1:02114] *** Process received signal ***
[cluster1:02114] Signal: Aborted (6)
[cluster1:02114] Signal code:  (-6)
[cluster1:02114] [ 0] /lib/libpthread.so.0(+0xf8f0) [0x7f9ceba4c8f0]
[cluster1:02114] [ 1] /lib/libc.so.6(gsignal+0x35) [0x7f9ceb6eda75]
[cluster1:02114] [ 2] /lib/libc.so.6(abort+0x180) [0x7f9ceb6f15c0]
[cluster1:02114] [ 3] /lib/libc.so.6(+0x6d4fb) [0x7f9ceb7274fb]
[cluster1:02114] [ 4] /lib/libc.so.6(__fortify_fail+0x37) [0x7f9ceb7b9217]
[cluster1:02114] [ 5] /lib/libc.so.6(+0xfe0d0) [0x7f9ceb7b80d0]
[cluster1:02114] [ 6] /lib/libc.so.6(+0xfd539) [0x7f9ceb7b7539]
[cluster1:02114] [ 7] /lib/libc.so.6(_IO_default_xsputn+0xcc) [0x7f9ceb72fd1c]
[cluster1:02114] [ 8] /lib/libc.so.6(_IO_vfprintf+0xf3e) [0x7f9ceb7002de]
[cluster1:02114] [ 9] /lib/libc.so.6(__vsprintf_chk+0x99) [0x7f9ceb7b75d9]
[cluster1:02114] [10] /lib/libc.so.6(__sprintf_chk+0x7f) [0x7f9ceb7b751f]
[cluster1:02114] [11] /mirror/mpiu/Xmipp-2.4-src/lib/libXmippData.so(_ZN8FileName7composeERKSsiS1_+0x1a4) [0x7f9ceda56fb4]
[cluster1:02114] [12] /mirror/mpiu/Xmipp-2.4-src/bin/xmipp_mpi_angular_project_library(_ZN43Prog_mpi_angular_project_library_Parameters3runEv+0x224) [0x40ff74]
[cluster1:02114] [13] /mirror/mpiu/Xmipp-2.4-src/bin/xmipp_mpi_angular_project_library(main+0x180) [0x40b3b0]
[cluster1:02114] [14] /lib/libc.so.6(__libc_start_main+0xfd) [0x7f9ceb6d8c4d]
[cluster1:02114] [15] /mirror/mpiu/Xmipp-2.4-src/bin/xmipp_mpi_angular_project_library() [0x40b079]
[cluster1:02114] *** End of error message ***

if someone as a idea to what i have to do?

I am running all the nodes under Ubuntu 10.04 64Bits

Thank's in advance.

Finally i found out that there is no problem ^^ just have to make a new folder and lanch my app again, and it worked...

don't know exactly what i have modify into this folder to make my app doesn't run but it's ok now.