I was wondering why perf record -g don’t show proper stack traces for my programs in production environment. First I thought that kernel was too old, but after performing few experiments I have found out that it wasn’t the case. Problem was that when you compile with optimizations (-O3), gcc automatically omits frame pointers. And it is not easy to unwind stack traces without frame pointers. But gcc can do it somehow. Read more

Sometimes your program uses a lot of system time. Let’s say 90%. You fire up your favorite profiling tool and it tells you which system call it is. If you are experiences and maybe lucky, you can say straight away which part of your program is to blame. But it’s not always so obvious. GDB comes to the rescue. You can use catch syscall <syscall> and it will break when particular syscall is called. Read more