how to find the cause of stack smash

I am trying to figure out why im getting stack smashed when using FFMPEG libraries. I've recompiled FFMPEG with debugging info and from the backtrace, the offending line is libavformat/mpegtsenc.c:800. However that line is just the closing } brace of a function mpegts_write_pes.

• So far i tried valgrind, but it crashes when using memptr tool.
• Tried efence but it doesnt seem to work with ffmpeg, cause they use their own malloc/free routines to get the memory aligned blocks
• Placed two large arrays at the beginning and end of the `mpegts_write_pes` declaration

I believe the function return address is being overwritte开发者_运维问答n (hence gdb points at the curly brace).

Here's a link to the source, http://git.ffmpeg.org/?p=ffmpeg;a=blob;f=libavformat/mpegtsenc.c;hb=1a50ca867a57d7dc74d414b5adb9a0264c9a6c6c

The line there would be 814.

thanks

*** stack smashing detected ***: /home/victor/workspace/segmenter/segmenter terminated
======= Backtrace: =========
/lib/libc.so.6(__fortify_fail+0x50)[0x419970]
/lib/libc.so.6(+0xe591a)[0x41991a]
/home/victor/workspace/segmenter/segmenter[0x80be004]
======= Memory map: ========
00110000-0012c000 r-xp 00000000 08:01 158181     /lib/ld-2.12.1.so
0012c000-0012d000 r--p 0001b000 08:01 158181     /lib/ld-2.12.1.so
0012d000-0012e000 rw-p 0001c000 08:01 158181     /lib/ld-2.12.1.so
0012e000-0012f000 r-xp 00000000 00:00 0          [vdso]
0012f000-0013f000 r-xp 00000000 08:01 130866     /lib/libbz2.so.1.0.4
0013f000-00140000 r--p 0000f000 08:01 130866     /lib/libbz2.so.1.0.4
00140000-00141000 rw-p 00010000 08:01 130866     /lib/libbz2.so.1.0.4
00141000-00180000 r-xp 00000000 08:01 277294     /usr/lib/libmp3lame.so.0.0.0
00180000-00181000 r--p 0003e000 08:01 277294     /usr/lib/libmp3lame.so.0.0.0
00181000-00182000 rw-p 0003f000 08:01 277294     /usr/lib/libmp3lame.so.0.0.0
00182000-001b6000 rw-p 00000000 00:00 0
001b6000-001cb000 r-xp 00000000 08:01 158998     /lib/libpthread-2.12.1.so
001cb000-001cc000 ---p 00015000 08:01 158998     /lib/libpthread-2.12.1.so
001cc000-001cd000 r--p 00015000 08:01 158998     /lib/libpthread-2.12.1.so
001cd000-001ce000 rw-p 00016000 08:01 158998     /lib/libpthread-2.12.1.so
001ce000-001d0000 rw-p 00000000 00:00 0
001d0000-001d7000 r-xp 00000000 08:01 158382     /lib/librt-2.12.1.so
001d7000-001d8000 r--p 00006000 08:01 158382     /lib/librt-2.12.1.so
001d8000-001d9000 rw-p 00007000 08:01 158382     /lib/librt-2.12.1.so
001d9000-001ec000 r-xp 00000000 08:01 131016     /lib/libz.so.1.2.3.4
001ec000-001ed000 r--p 00012000 08:01 131016     /lib/libz.so.1.2.3.4
001ed000-001ee000 rw-p 00013000 08:01 131016     /lib/libz.so.1.2.3.4
001ee000-001fc000 r-xp 00000000 08:01 277205     /usr/lib/libfaac.so.0.0.0
001fc000-001fd000 r--p 0000d000 08:01 277205     /usr/lib/libfaac.so.0.0.0
001fd000-00200000 rw-p 0000e000 08:01 277205     /usr/lib/libfaac.so.0.0.0
00200000-00224000 r-xp 00000000 08:01 158974     /lib/libm-2.12.1.so
00224000-00225000 r--p 00023000 08:01 158974     /lib/libm-2.12.1.so
00225000-00226000 rw-p 00024000 08:01 158974     /lib/libm-2.12.1.so
00226000-002bf000 r-xp 00000000 08:01 277352     /usr/lib/libxvidcore.so.4.2
002bf000-002c0000 r--p 00098000 08:01 277352     /usr/lib/libxvidcore.so.4.2
002c0000-002ca000 rw-p 00099000 08:01 277352     /usr/lib/libxvidcore.so.4.2
002ca000-00334000 rw-p 00000000 00:00 0
00334000-0048b000 r-xp 00000000 08:01 158997     /lib/libc-2.12.1.so
0048b000-0048d000 r--p 00157000 08:01 158997     /lib/libc-2.12.1.so
0048d000-0048e000 rw-p 00159000 08:01 158997     /lib/libc-2.12.1.so
0048e000-00491000 rw-p 00000000 00:00 0
00491000-004ab000 r-xp 00000000 08:01 130901     /lib/libgcc_s.so.1
004ab000-004ac000 r--p 00019000 08:01 130901     /lib/libgcc_s.so.1
004ac000-004ad000 rw-p 0001a000 08:01 130901     /lib/libgcc_s.so.1
08048000-086b1000 r-xp 00000000 08:01 276922     /home/victor/workspace/segmenter/segmenter
086b1000-086b2000 r--p 00668000 08:01 276922     /home/victor/workspace/segmenter/segmenter
086b2000-086c3000 rw-p 00669000 08:01 276922     /home/victor/workspace/segmenter/segmenter
086c3000-08c99000 rw-p 00000000 00:00 0          [heap]
b7fec000-b7fef000 rw-p 00000000 00:00 0
b7ffd000-b8000000 rw-p 00000000 00:00 0
bffdf000-c0000000 rw-p 00000000 00:00 0          [stack]

Program received signal SIGABRT, Aborted.
0x0012e416 in __kernel_vsyscall ()
(gdb) where
#0  0x0012e416 in __kernel_vsyscall ()
#1  0x0035e941 in raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
#2  0x00361e42 in abort () at abort.c:92
#3  0x00396305 in __libc_message (do_abort=2, fmt=0x46c36a "*** %s ***: %s terminated\n") at ../sysdeps/unix/sysv/linux/libc_fatal.c:189
#4  0x00419970 in __fortify_fail (msg=<value optimized out>) at fortify_fail.c:32
#5  0x0041991a in __stack_chk_fail () at stack_chk_fail.c:29
#6  0x080be004 in mpegts_write_pes (s=<value optimized out>, st=<value optimized out>, payload=<value optimized out>, payload_size=0, pts=147149074, dts=17992800) at libavformat/mpegtsenc.c:800
#7  0x080be2ed in mpegts_write_packet (s=0x8bcf3d0, pkt=0xbffff1c0) at libavformat/mpegtsenc.c:883
#8  0x080ff9aa in av_interleaved_write_frame (s=0x8bcf3d0, pkt=0xbffff26c) at libavformat/utils.c:2917
#9  0x080704d1 in main (argc=6, argv=0xbffff414) at segmenter.c:662
(gdb) 

`

If you look at the stack in your debugger, is the data that overwrote your return address identifiable? If it is (like a string, for example), that can help you narrow down the offending writer.

Another good technique to try is to use watchpoints in your debugger. If your crash is reproducible, you can get the address of this stack value that gets overwritten. Assuming that it's consistent, you can set a watchpoint at that address and track all of the writes to that location. In no time flat, you'll find out the code that's writing where it shouldn't.