Bizarre behavior in gdb

I'm experiencing a very strange bug in my code.

I use a variable m_requestedStructures in my program, and the debugger is accessing the wrong address when inspecting it. For example, I have in the code:

qDebug() << "requested: " << m_requestedStructures << " " << &m_requestedStructures

and I set a breakpoint here. When the breakpoint is reached, I compare the printed value and address of m_requestedStructures with those that gdb is accessing:

requested:  0   0xfce640 
Breakpoint 1
157         qDebug() << "requested: " << m_requestedStructures << " " << &m_requestedStructures;
(gdb) p m_requestedStructures
$5 = -1167616
(gdb) p &m_requestedStructures
$6 = (int *) 0xfce644
(gdb)

As can be seen, both the value and the address differ.

Another symptom I'm seeing is that I had a function, QueueManager::checkPopulation_() which I removed quite a while ago, but gdb still tries to tab-complete it, and if I set a breakpoint on it, it will break at the point where the function used to be.

It appears to me that the symbols aren't read correctly. I've tried rebuilding from a clean directory, but the problem persists. Does anyone have any idea what might be causing this or how to fix it?

Thanks in advance!

Edit: I just tried setting a breakpoint on a new function I've added. The function QueueManager::run() appears in gdb's tab complete, but when I try to set the breakpoint, I get

the class GlobalSearch::QueueManager does not have any method named run

Also, inspecting other variables, both locally scoped and members of QueueManager, seems to work fine.

Here is the compiler/linker output. The file containing the class is queuemanager.cpp, which is part of a static library libglobalsearch.a, which is used in a dynamic library xtalopt.so:

$ make VERBOSE=1 xtalopt
/usr/bin/cmake -H/git/xtalopt-public -B/git/xtalopt-public/build --check-build-system CMakeFiles/Makefile.cmake 0
make -f CMakeFiles/Makefile2 xtalopt
make[1]: Entering directory `/git/xtalopt-public/build'
/usr/bin/cmake -H/git/xtalopt-public -B/git/xtalopt-public/build --check-build-system CMakeFiles/Makefile.cmake 0
/usr/bin/cmake -E cmake_progress_start /git/xtalopt-public/build/CMakeFiles 63
make -f CMakeFiles/Makefile2 src/xtalopt/CMakeFiles/xtalopt.dir/all
make[2]: Entering directory `/git/xtalopt-public/build'
make -f src/spglib/CMakeFiles/spglib.dir/build.make src/spglib/CMakeFiles/spglib.dir/depend
make[3]: Entering directory `/git/xtalopt-public/build'
cd /git/xtalopt-public/build && /usr/bin/cmake -E cmake_depends "Unix Makefiles" /git/xtalopt-public /git/xtalopt-public/src/spglib /git/xtalopt-public/build /git/xtalopt-public/build/src/spglib /git/xtalopt-public/build/src/spglib/CMakeFiles/spglib.dir/DependInfo.cmake --color=
make[3]: Leaving directory `/git/xtalopt-public/build'
make -f src/spglib/CMakeFiles/spglib.dir/build.make src/spglib/CMakeFiles/spglib.dir/build
make[3]: Entering directory `/git/xtalopt-public/build'
make[3]: Nothing to be done for `src/spglib/CMakeFiles/spglib.dir/build'.
make[3]: Leaving directory `/git/xtalopt-public/build'
/usr/bin/cmake -E cmake_progress_report /git/xtalopt-public/build/CMakeFiles  18 19 20 21 22 23
[  9%] Built target spglib
make -f src/libssh/CMakeFiles/ssh_static.dir/build.make src/libssh/CMakeFiles/ssh_static.dir/depend
make[3]: Entering directory `/git/xtalopt-public/build'
cd /git/xtalopt-public/build && /usr/bin/cmake -E cmake_depends "Unix Makefiles" /git/xtalopt-public /git/xtalopt-public/src/libssh /git/xtalopt-public/build /git/xtalopt-public/build/src/libssh /git/xtalopt-public/build/src/libssh/CMakeFiles/ssh_static.dir/DependInfo.cmake --color=
make[3]: Leaving directory `/git/xtalopt-public/build'
make -f src/libssh/CMakeFiles/ssh_static.dir/build.make src/libssh/CMakeFiles/ssh_static.dir/build
make[3]: Entering directory `/git/xtalopt-public/build'
make[3]: Nothing to be done for `src/libssh/CMakeFiles/ssh_static.dir/build'.
make[3]: Leaving directory `/git/xtalopt-public/build'
/usr/bin/cmake -E cmake_progress_report /git/xtalopt-public/build/CMakeFiles  26 27 28 29 30 31 32 33 34 35 36 37 38 39 40
[ 33%] Built target ssh_static
make -f src/globalsearch/CMakeFiles/globalsearch.dir/build.make src/globalsearch/CMakeFiles/globalsearch.dir/depend
make[3]: Entering directory `/git/xtalopt-public/build'
cd /git/xtalopt-public/build && /usr/bin/cmake -E cmake_depends "Unix Makefiles" /git/xtalopt-public /git/xtalopt-public/src/globalsearch /git/xtalopt-public/build /git/xtalopt-public/build/src/globalsearch /git/xtalopt-public/build/src/globalsearch/CMakeFiles/globalsearch.dir/DependInfo.cmake --color=
make[3]: Leaving directory `/git/xtalopt-public/build'
make -f src/globalsearch/CMakeFiles/globalsearch.dir/build.make src/globalsearch/CMakeFiles/globalsearch.dir/build
make[3]: Entering directory `/git/xtalopt-public/build'
/usr/bin/cmake -E cmake_progress_report /git/xtalopt-public/build/CMakeFiles 
[ 33%] Building CXX object src/globalsearch/CMakeFiles/globalsearch.dir/queuemanager.cpp.o
cd /git/xtalopt-public/build/src/globalsearch && /usr/bin/c++   -DQT_OPENGL_LIB -DQT_GUI_LIB -DQT_CORE_LIB -DQT_PLUGIN -DQT_SHARED -D_HAVE_EXECINFO_H_ -DQT_DEBUG -fPIC -g -I/usr/include/QtOpenGL -I/usr/include/QtGui -I/usr/include/QtCore -I/usr/include/eigen2 -I/git/xtalopt-public/build -I/usr/include/openbabel-2.0 -I/git/xtalopt-public/src   -o CMakeFiles/globalsearch.dir/queuemanager.cpp.o -c /git/xtalopt-public/src/globalsearch/queuemanager.cpp
Linking CXX static library libglobalsearch.a
cd /git/xtalopt-public/build/src/globalsearch && /usr/bin/cmake -P CMakeFiles/globalsearch.dir/cmake_clean_target.cmake
cd /git/xtalopt-public/build/src/globalsearch && /usr/bin/cmake -E cmake_link_script CMakeFiles/globalsearch.dir/link.txt --verbose=1
/usr/bin/ar cr libglobalsearch.a  CMakeFiles/globalsearch.dir/optbase.cpp.o CMakeFiles/globalsearch.dir/queuemanager.cpp.o CMakeFiles/globalsearch.dir/structure.cpp.o CMakeFiles/globalsearch.dir/tracker.cpp.o CMakeFiles/globalsearch.dir/optimizer.cpp.o CMakeFiles/globalsearch.dir/bt.cpp.o CMakeFiles/globalsearch.dir/sshconnection.cpp.o CMakeFiles/globalsearch.dir/sshmanager.cpp.o CMakeFiles/globalsearch.dir/random.cpp.o CMakeFiles/globalsearch.dir/ui/abstractdialog.cpp.o CMakeFiles/globalsearch.dir/ui/abstracttab.cpp.o CMakeFiles/globalsearch.dir/moc_optbase.cxx.o CMakeFiles/globalsearch.dir/moc_queuemanager.cxx.o CMakeFiles/globalsearch.dir/moc_structure.cxx.o CMakeFiles/globalsearch.dir/moc_tracker.cxx.o CMakeFiles/globalsearch.dir/moc_optimizer.cxx.o CMakeFiles/globalsearch.dir/moc_bt.cxx.o CMakeFiles/globalsearch.dir/moc_sshconnection.cxx.o CMakeFiles/globalsearch.dir/moc_sshmanager.cxx.o CMakeFiles/globalsearch.dir/moc_random.cxx.o CMakeFiles/globalsearch.dir/ui/moc_abstractdialog.cxx.o CMakeFiles/globalsearch.dir/ui/moc_abstracttab.cxx.o
/usr/bin/ranlib libglobalsearch.a
make[3]: Leaving directory `/git/xtalopt-public/build'
/usr/bin/cmake -E cmake_progress_report /git/xtalopt-public/build/CMakeFiles  1 2 3 4 5 6 7 8 9 10 11 12 13 14
[ 55%] Built target globalsearch
make -f src/xtalopt/CMakeFiles/xtalopt.dir/build.make src/xtalopt/CMakeFiles/xtalopt.dir/depend
make[3]: Entering directory `/git/xtalopt-public/build'
cd /git/xtalopt-public/build &&开发者_开发技巧 /usr/bin/cmake -E cmake_depends "Unix Makefiles" /git/xtalopt-public /git/xtalopt-public/src/xtalopt /git/xtalopt-public/build /git/xtalopt-public/build/src/xtalopt /git/xtalopt-public/build/src/xtalopt/CMakeFiles/xtalopt.dir/DependInfo.cmake --color=
make[3]: Leaving directory `/git/xtalopt-public/build'
make -f src/xtalopt/CMakeFiles/xtalopt.dir/build.make src/xtalopt/CMakeFiles/xtalopt.dir/build
make[3]: Entering directory `/git/xtalopt-public/build'
Linking CXX shared library xtalopt.so
cd /git/xtalopt-public/build/src/xtalopt && /usr/bin/cmake -E cmake_link_script CMakeFiles/xtalopt.dir/link.txt --verbose=1
/usr/bin/c++  -fPIC  -fPIC -g  -shared -Wl,-soname,xtalopt.so -o xtalopt.so CMakeFiles/xtalopt.dir/extension.cpp.o CMakeFiles/xtalopt.dir/xtalopt.cpp.o CMakeFiles/xtalopt.dir/genetic.cpp.o CMakeFiles/xtalopt.dir/structures/xtal.cpp.o CMakeFiles/xtalopt.dir/optimizers/xtaloptoptimizer.cpp.o CMakeFiles/xtalopt.dir/optimizers/vasp.cpp.o CMakeFiles/xtalopt.dir/optimizers/gulp.cpp.o CMakeFiles/xtalopt.dir/optimizers/pwscf.cpp.o CMakeFiles/xtalopt.dir/optimizers/castep.cpp.o CMakeFiles/xtalopt.dir/testing/xtalopttest.cpp.o CMakeFiles/xtalopt.dir/ui/dialog.cpp.o CMakeFiles/xtalopt.dir/ui/tab_init.cpp.o CMakeFiles/xtalopt.dir/ui/tab_edit.cpp.o CMakeFiles/xtalopt.dir/ui/tab_opt.cpp.o CMakeFiles/xtalopt.dir/ui/tab_sys.cpp.o CMakeFiles/xtalopt.dir/ui/tab_progress.cpp.o CMakeFiles/xtalopt.dir/ui/tab_plot.cpp.o CMakeFiles/xtalopt.dir/ui/tab_log.cpp.o CMakeFiles/xtalopt.dir/moc_extension.cxx.o CMakeFiles/xtalopt.dir/moc_xtalopt.cxx.o CMakeFiles/xtalopt.dir/moc_genetic.cxx.o CMakeFiles/xtalopt.dir/structures/moc_xtal.cxx.o CMakeFiles/xtalopt.dir/optimizers/moc_xtaloptoptimizer.cxx.o CMakeFiles/xtalopt.dir/optimizers/moc_vasp.cxx.o CMakeFiles/xtalopt.dir/optimizers/moc_gulp.cxx.o CMakeFiles/xtalopt.dir/optimizers/moc_pwscf.cxx.o CMakeFiles/xtalopt.dir/optimizers/moc_castep.cxx.o CMakeFiles/xtalopt.dir/testing/moc_xtalopttest.cxx.o CMakeFiles/xtalopt.dir/ui/moc_dialog.cxx.o CMakeFiles/xtalopt.dir/ui/moc_tab_init.cxx.o CMakeFiles/xtalopt.dir/ui/moc_tab_edit.cxx.o CMakeFiles/xtalopt.dir/ui/moc_tab_opt.cxx.o CMakeFiles/xtalopt.dir/ui/moc_tab_sys.cxx.o CMakeFiles/xtalopt.dir/ui/moc_tab_progress.cxx.o CMakeFiles/xtalopt.dir/ui/moc_tab_plot.cxx.o CMakeFiles/xtalopt.dir/ui/moc_tab_log.cxx.o -lavogadro ../globalsearch/libglobalsearch.a ../spglib/libspglib.a -lQtOpenGL -lGLU -lGL -lSM -lICE -lX11 -lXext -lopenbabel -lQtGui -lQtCore ../libssh/libssh_static.a -lnsl -lresolv -lrt -lz -lssl -lcrypto 
make[3]: Leaving directory `/git/xtalopt-public/build'
/usr/bin/cmake -E cmake_progress_report /git/xtalopt-public/build/CMakeFiles  45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72
[100%] Built target xtalopt
make[2]: Leaving directory `/git/xtalopt-public/build'
/usr/bin/cmake -E cmake_progress_start /git/xtalopt-public/build/CMakeFiles 0
make[1]: Leaving directory `/git/xtalopt-public/build'

I finally managed to figure this out! Here's how:

In gdb, I printed all the shared libraries in memory with info shared. I copied this to a text file shared.txt and extracted the library names with:

cat shared.txt | grep -v \*|grep -v quit---|grep -v Syms\ Read|grep -v \(gdb\)|awk '{print $4}'>libs

From there, I checked the symbols in each library with:

for i in `cat libs|xargs`;do nm --demangle $i|grep checkPopulation; if [ $? -eq 0 ]; then echo \($i\); fi;done

Which prints all symbols containing "checkPopulation" followed by the name of the library that contains each. My output:

000adb18 T GlobalSearch::QueueManager::checkPopulation()
000adb8a T GlobalSearch::QueueManager::checkPopulation_()
(/usr/lib/avogadro/1_1/contrib/randomdock.so)
000db682 T GlobalSearch::QueueManager::checkPopulation()
(/usr/lib/avogadro/1_1/contrib/xtalopt.so)

So this "phantom" checkPopulation_ symbol is coming from randomdock.so, which was built a long time ago against an older version of the static libglobalsearch.a, which contains the now removed GlobalSearch::QueueManager::checkPopulation_(). Removing randomdock.so causes everything to behave as it should.

Thanks for all the suggestions -- I certainly learned a few new gdb tricks while tracking this down. There's also a lesson here about breaking API, too ;-)

Just FYI:

Grep has a -e switch, so you can handle multiple expressions at once. E.g:

grep -v -e \* -e quit--- -e Syms\ Read -e \(gdb\) | awk '{print $4}' > libs

nm has a -A switch (print filename). E.g:

cat libs | xargs nm -A --demangle | grep checkPopulation