Some time ago YouTube removed the option of receiving new videos from channels that you are subscribed to via email. I loved using email because it was easy to sift through new videos, delete those that are not interested and leave unread those that I want to watch a bit later. Unfortunately, I had to start using the YouTube web interface. A Major drawback of using web interface is that you have to check Subscriptions page often (once every few days) or you would lose the threshold between what you already saw and what you did not see. Read more

This is a bunch of notes taken from “DNS over HTTPS considerations” by Bert Hubert. It’s an amazing talk and I recommend watching it. It covers the history of DNS security, privacy considerations, state-of-the-art and what awaits us in the future. DNS is one of the oldest text protocols that are still widely used. Almost everything else we encrypted already. There have been four tries to encrypt DNS: Read more

I’ve decided to rekindle my blog as my New Year’s resolution. I am going to leave Google’s Blogger behind and switch to a static site generated by Hugo. You are reading a first post in a new blog. Welcome!

At Badoo we have a distributed system that uses ssh on start to connect to each node and exchange some data. Sometimes I’ve seen ssh fail with error ssh_exchange_identification: read: Connection reset by peer error Usually, If you see this error, it means that server didn’t like your certificate or you were banned from accessing this server. The best way to debug this problem is to append -vvv while executing ssh client to increase verbosity and to do the same on server. Read more

We have some very strange memory corruption problem in one of our daemons. As one of the many approaches to find the bug, we have built and pushed to production this daemon, but without jemalloc, memory allocator library that we use in all our projects. Few hours after the release monitoring team reported an unusal amount of errors returned from the system. It happens that few percent of the requests returned “out of memory” error. Read more

I was wondering why perf record -g don’t show proper stack traces for my programs in production environment. First I thought that kernel was too old, but after performing few experiments I have found out that it wasn’t the case. Problem was that when you compile with optimizations (-O3), gcc automatically omits frame pointers. And it is not easy to unwind stack traces without frame pointers. But gcc can do it somehow. Read more

Sometimes your program uses a lot of system time. Let’s say 90%. You fire up your favorite profiling tool and it tells you which system call it is. If you are experiences and maybe lucky, you can say straight away which part of your program is to blame. But it’s not always so obvious. GDB comes to the rescue. You can use catch syscall <syscall> and it will break when particular syscall is called. Read more

I have being playing today with perf on latest available kernel in Fedora (3.13). These are just some observations and thoughts. perf trace This tool is similar to strace, but almost without overhead. Contrary to strace, you can use perf trace to watch syscalls system wide or system calls generated by processes owned by certain user. perf top Function names are shown correctly and -g parameter is now available. It means that you can get call traces in a real time (without perf record). Read more

Sometimes you will see processes on your linux box that are in D state as shown by ps, top, htop or similar. D means uninterruptible sleep. As opposed to normal sleep, you can’t do anything with these processes (i.e. kill them). Additional information could be found on LWN. Usually this means that process is stuck reading or writing something to disk or NFS and remote site does not answer. But not always. Read more

In a deamon that I have created in Badoo I need to search for a positions of a bits in a 32 bit variable. The simplest solution is to look at each bit, but there is a faster solution using GCC built in function __builtin_ctzl: Returns the number of trailing 0-bits in x, starting at the least significant bit position. If x is 0, the result is undefined. So I wrote test to compare speed of these two approaches: Read more