Connect to multiple VPNs at once using NetworkManager and systemd-resolved
from DNS Haiku
The dark side
When I started at synyx I chose to get a MacBook in order to be productive as quick as possible. I still had the cumbersome Linux desktop experience from my previous employer in mind. 😉 At synyx, however, things are different: we have a highly skilled admin team that provides a very convenient environment to use Linux on laptops. So, a few weeks ago I decided to move away from MacOS to Linux (again).
One of the trickier parts of that migration effort was to setup my VPNs with the same comfort as I had on MacOS. Since we are working as consultants/external developers for several customers there is the need to connect to synyx’ VPN as well as our customers’ VPNs. Thus I need to connect to at least two VPNs with working DNS resolution into all connected networks.
DNS configuration with /etc/resolv.conf
Traditionally on Linux systems the DNS servers used to resolve hostnames to IP addresses are listed in the configuration file /etc/resolv.conf :
When OpenVPN establishes a connection the server side returns information about the connected network. A part of this are the nameserver(s) that are used to resolve names that are part of the connected network. In the simplest possible setup the nameserver of the VPN is appended to the list of nameservers that have been written to that file for the normal network connection by your network manager. For the first connection this simple approach is sufficient in most of the cases. As soon as you configure and connect the second VPN a harsh limitation of libnss_dns library that parses resolv.conf kicks in: only three nameserver entries in /etc/resolv.conf are supported. So only the names of the VPN you connected at first are resolvable. If your network manager has for any reasons written 3 nameservers your VPN’s nameservers are completely ignored.
Alternative DNS configuration mechanisms
Luckily name resolution in libc general is configurable via the /etc/nsswitch.conf configuration file. Besides other mechanisms the file defines how to lookup hostnames. In its default setting the section responsible for resolving hostnames points to ‘files’ and ‘dns’. The first one uses /etc/hosts and the latter uses /etc/resolv.conf for resolution. As the /etc/resolv.conf is limited to 3 nameserver entries this is not suitable for connecting to multiple VPNs simultaneously.
In the past I had a setup with a handcrafted Dnsmasq installation that was congfigured to lookup names from the VPN via the VPN’s nameserver. This time, however, I wanted to try something else. In recent systemd versions a new service called systemd-resolved was introduced. It can be used to dynamically configure DNS servers dependending on the active network connection(s).
So, if the libnss_dns plugin in libc is limited to three server entries in /etc/resolv.conf some other mechanism has to be used. One idea is that you statically configure the address of a locally installed nameserver that is (re-)configured by OpenVPN connections. Dnsmasq is one of the tools many people (including past-me) used for this task. In this article however, I want to focus on the already mentioned systemd-resolved and how it can be integrated into the hostname lookup on a modern Linux system.
OpenVPN and SELinux
Before we get into the details of connecting systemd-resolved with OpenVPN I want to point out a dirty detail of convincing OpenVPN to work together with SELinux (which is active by default on Fedora). So, I use Gnome’s NetworkManager as a network manager not only for WiFi and fixed networks but also for VPNs (and 4G as well). So I used the very convenient UI to import my existing VPN configuration and tried to connect. Nothing happened. 😐
After looking into the logs I realized that the NetworkManager OpenVPN plugin was not permitted to access the VPN certificate files for some reasons. For quite some time I tried to fix permissions but nothing helped. After some googling I found this article which describes that OpenVPN may only access files in the ~/.cert directory and restore the SELinux security context of the files with the restorecon command.
Configure and activate systemd-resolved
Before we can use systemd-resolved it needs to be configured and activated. The config file is located at /etc/systemd/resolved.conf:
The first line defines a fallback DNS if all of your dynamically configured nameservers fail. At least for me this was useful. The second line is more important – it deactivates DNSSEC. DNSSEC defines security extensions on top of the DNS protocol. Although this sounds like a very reasonable thing to do you’ll – especially in private networks – encounter nameservers who do not (properly) support DNSSEC. Sadly it can not be configured per connected network but only globally. So for my setup I deactivated the whole thing.
After configuration systemd-resolved can be enabled and started:
Now that systemd-resolved is active we need to push new nameserver configurations to systemd-resoved everytime a new VPN is connected (and remove them if the VPN is disconnected). If you do not use NetworkManager’s OpenVPN integration but operate OpenVPN directly you can use Jonathan Wright’s very useful update-systemd-resolved script. Just configure it in your OpenVPN config file to be run on connect and disconnect. When you use NetworkManager things are even simpler: Just activete systemd-resolved in the configuration by adding /etc/NetworkManager/conf.d/resolved.conf:
And then restart NetworkManager:
Upon each connect and disconnect of standard networks (wifi, ethernet and 4G) or VPNs NetworkManager will update the nameserver information in systemd-resolved.
Let your software use systemd-resolved for DNS resolution
I assume most of the software on Linux is using libc for name resolution. In libc the name resolution is configured via a file called /etc/nsswitch.conf. Besides other sections nsswitch.conf has a section for hostname resolution. Here is the relevant line from the file:
The important part is the ‘resolve’ behind ‘files’. It tells libc to load a plugin called libnss_resolve and try to use for hostname resolution before other plugins like the traditional libnss_dns are tried. By this all software using libc’s name resolution will automatically use systemd-resolved from now on.
There are, however, other applications that parse your /etc/resolv.conf on their own and circumvent systemd-resolved unconsciously. For this applications /etc/resolv.conf needs to be replaced with a symlink to either
The first file just contains all nameserver entries that have been configured for systemd-resolved. This is again not very helpful for our scenario since it will definitly contain more than three entries and you cannot be sure whether other applications parsing that file have the same limitation as the libnss_dns library.
The second file is better for our scenario. It just contains a single nameserver entry which points to 127.0.0.53 which is a local DNS server that is part of systemd-resolved and just forwards name resolution requests to the configured nameservers.
So execute the following on your system:
Gnome Software and systemd-resolved
For some reason Gnome Software stopped working after I activated systemd-resolved in /etc/nsswitch.conf – it failed to resolve hostnames when downloading packages. I was able to fix this by changing the order in /etc/nsswitch.conf so that dns is before resolve. I did not dig into details here. If you happen to know what could be the reason for this – feel free to get in touch with me. 😉
As you can see from the length of this article, it takes a significant amount of effort and knowledge to configure your Linux desktop to support multiple and simultaneously connected VPN with working DNS resolution. Nevertheless, once configured correctly everything works really well! 😀
So, I hope everything was understandable and helpful for you. If you have any remarks drop a line on Twitter @robinjayasinghe.
In the meantime some time has passed and for various reasons I switched through some distros. In Debian 10 the systemd-resolved approach described in this article worked well. Only Gnome Software did not work reliably when resolved was active. Afterwards I switched back to Fedora and used Fedora 31 for a while. Here I had the option to use systemd-resolved in NetworkManager but decided to use the alternatively offered dnsmasq plugin for NetworkManager. This worked like a charm and there was nothing more to configure than activating the plugin. However, Gnome Software still had issues for me. I did not do any research on that. Due to a crashed upgrade to Fedora 32 I rage-migrated to Ubuntu 20.04 LTS. I am not yet sure what Ubuntu did beneath the surface but connecting to multiple VPNs at the same time including proper DNS resolution works OOTB. Kudos to the folks at Canonical. Gnome Software works (even when connected to VPNs). :partyhat: