Certificate debugging with strace

This is a short post about a neat tool that just saved the day. A colleague was having trouble getting his Ubuntu virtual machine to connect to any web hosts that are served over HTTPS. This was to be expected as there is a corporate proxy that unencrypts all traffic to inspect it, then re-encrypts it with its own certificate. To view this re-encrypted data, we need to get the system to trust the new root certificate. There were lots of guides on how to do this but none of them were working.

Trying a different angle, I decided to play with strace. If you run a command through strace then you can see all the syscalls that it makes. There are a lot so you'll no-doubt want to filter them.

We ran curl to attempt to reach an HTTPS site. The -e openat flag filters strace's output to just show files that are opened during the course of curl executing.

strace -e openat curl https://example.com

On my own machine (that had the custom root certificate installed) the bottom of strace's output looked like this:

openat(AT_FDCWD, "/usr/lib/ssl/openssl.cnf", O_RDONLY) = 3
openat(AT_FDCWD, "/usr/lib/locale/locale-archive", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/home/damian/.curlrc", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/dev/urandom", O_RDONLY|O_NOCTTY|O_NONBLOCK) = 4
openat(AT_FDCWD, "/etc/ssl/certs/ca-certificates.crt", O_RDONLY) = 4
openat(AT_FDCWD, "/etc/localtime", O_RDONLY|O_CLOEXEC) = 4

We see that /etc/ssl/certs/ca-certificates.crt is the system's combined certificate store that is being loaded to check against. On my colleague's machine, this wasn't the case - instead, it was opening a file called cacerts.crt located somewhere in the current user's Downloads directory. It was apparent that an application (Anaconda) had modified the location for certificate authority checking (at least for the current user).

I was pleasantly surprised by how easy it was to use strace for this purpose and how quickly it solved our problem. I remember playing with demos of DTrace under Solaris and thinking it was cool so I'm happy similar functionality is available and ready to run on modern Linux without having to install a thing. Hopefully it will help with future debugging too.