Wireguard vpn with network namespace on NixOS

it works pretty well
December 18, 2018 (wireguard, linux, nixos, i3)

Background

Wireguard is a shiny new VPN option. Interestingly, they recommend using network namespaces to set up routing.

Roughly speaking, the traditional model is to have encrypted tunnel and the hardware interfaces visible to programs at the same time, and setting up routing rules to make sure everything goes into the tunnel.

The newer approach is to hide away the physical interfaces in a namespace that’s invisible to most programs, and only expose the wireguard tunnel by default. Only programs that explicitly deal with the physical interfaces, like dhcpcd, even need to know they exist. Conceptually, this is simpler than complicated routing rules, but implementing it has some practical hurdles, since existing programs and configurations may need tweaking to work with the network namespace.

NixOS is an awesome and pioneering linux distribution.

Here’s how I’ve gotten a fully-functional, somehwat-rough-around-the-edges wireguard client setup on my NixOS laptop.

Code is on github.

Description

Fist off, we have a systemd service which creates a network namespace, with a hardcoded name physical. It’s not necessary to break this out, but I’d like to keep the namespace around even when I shut down the wireguard tunnel, just in case I’d like to keep anything active inside it (in particular, i3status continually keeps tabs on the physical interfaces).

systemd.services = {
  physical-netns = {
    description = "physical namespace, for use with wireguard";
    wantedBy = [ "default.target" ];
    before = [ "display-manager.service" "network.target" ];
    serviceConfig = {
      Type = "oneshot";
      RemainAfterExit = true;
      ExecStart = "${pkgs.iproute}/bin/ip netns add physical";
      ExecStop = "${pkgs.iproute}/bin/ip netns del physical";
    };
  };
};

Then we have a service that actually brings up and down the tunnel, and redirects traffic through it. This is heavily based on the wireguard tutorial, though it’s not exactly the same. It depends on the namespace existing (reflected in the systemd requires and after clauses).

systemd.services = {
  wg0 = {
    description = "Wireguard interface, and vpn";
    requires = [ "physical-netns.service" ];
    after = [ "physical-netns.service" ];
    serviceConfig = {
      Type = "oneshot";
      RemainAfterExit = true;
      ExecStart = pkgs.writeScript "wgup" ''
        #! ${pkgs.bash}/bin/bash
        ${pkgs.iproute}/bin/ip -n physical link add wgvpn0 type wireguard
        ${pkgs.iproute}/bin/ip -n physical link set wgvpn0 netns 1
        ${pkgs.wireguard}/bin/wg set wgvpn0 \
          private-key /root/wireguard-keys/private \
          peer ghK62ZFGd9zkRPfF6JehK7OMAW6HMdy68RNalq9FVUo= \
          allowed-ips 0.0.0.0/0 \
          endpoint thufir:51820
        ${pkgs.iproute}/bin/ip link set wgvpn0 up
        ${pkgs.iproute}/bin/ip addr add 10.100.0.2/24 dev wgvpn0
        ${pkgs.iproute}/bin/ip route add default dev wgvpn0
        ${pkgs.iproute}/bin/ip link set enp0s25 netns physical
        ${pkgs.iw}/bin/iw phy phy0 set netns name physical
        ${pkgs.systemd}/bin/systemctl restart --no-block wpa_supplicant dhcpcd
      '';
      ExecStop =  pkgs.writeScript "wgdown" ''
        #! ${pkgs.bash}/bin/bash
        ${pkgs.iproute}/bin/ip -n physical link set enp0s25 netns 1
        ${pkgs.iproute}/bin/ip netns exec physical ${pkgs.iw}/bin/iw phy phy0 set netns 1
        ${pkgs.iproute}/bin/ip link del wgvpn0
        ${pkgs.systemd}/bin/systemctl restart --no-block wpa_supplicant dhcpcd
      '';
    };
  };
};

Then we have a small helper physexec, analogous to the shell script described on the wireguard page.

physexec = pkgs.writeScriptBin "physexec" ''
    #! ${pkgs.bash}/bin/bash
    exec sudo -E ${pkgs.iproute}/bin/ip netns exec physical \
         sudo -E -u \#$(${pkgs.coreutils}/bin/id -u) \
                 -g \#$(${pkgs.coreutils}/bin/id -g) \
                 "$@"
  '';

Now we come to the ugliest part of the whole thing. dhcpcd and wpa_supplicant need to be inside the netns when I have the vpn on, and outside of it when the vpn is off. To handle this, we start with a small wrapper phys-aware. Unlike physexec, this wraps a program in the netns only if the wireguard tunnel exists. We’ll use it to wrap the ExecStart of the two systemd services.

phys-aware = pkgs.writeScript "phys-aware" ''
    #! ${pkgs.bash}/bin/bash
    if ${pkgs.iproute}/bin/ip link | ${pkgs.gnugrep}/bin/grep -q wgvpn0;
    then exec ${pkgs.iproute}/bin/ip netns exec physical "$@"
    else exec "$@"
    fi
  '';

Unfortunately, there isn’t a great way to incorporate this into the two services. They’re defined in nixpkgs, they aren’t natively flexible enough for our needs, and there is no tooling that does for modules what overlays do for packages. The least-bad approach is to copy-paste the module definitions into my configuration repo, make my changes, and disable the originals.

imports = [
  ./copied/dhcpcd.nix
  ./copied/wpa_supplicant.nix
];
disabledModules = [
  "services/networking/dhcpcd.nix"
  "services/networking/wpa_supplicant.nix"
];

Even when running inside the netns, I had some issues with the two services not quite knowing which interfaces to work with. Therefore I set this list explicitly, and further tweaked the ExecStart of the two services to explicitly pass this through.

networking.wireless.interfaces = [ "wlp4s0" ];

Note from earlier that in our main service, both the ExecStart and ExecStop conclude with a line that restarts these two services. They only switch in or out of the namespace when they are restarted, so we have to do that explicitly.

${pkgs.systemd}/bin/systemctl restart --no-block wpa_supplicant dhcpcd

Finally, a bit of a detour. i3status, the status bar of my window manager, displays information about the current wireless connection. However it doesn’t know how to deal with namespaces, so some extra work is necessary. I make a custom i3status script, which internally launches two variants of the original, one inside the netns and one outside. Then there’s some string munging to combine them back together.

anders-i3status = pkgs.writeScriptBin "anders-i3status" ''
    #! ${pkgs.python3}/bin/python -u
    import os
    import signal
    import subprocess
    # https://stackoverflow.com/questions/320232/ensuring-subprocesses-are-dead-on-exiting-python-program
    os.setpgrp() # create new process group, become its leader
    try:
      p1 = subprocess.Popen(['physexec', 'i3status', '-c', '/etc/i3/status-netns'],
                            stdout=subprocess.PIPE)
      p2 = subprocess.Popen([            'i3status', '-c', '/etc/i3/status'],
                            stdout=subprocess.PIPE)
      for i in range(2):
        line1 = p1.stdout.readline().decode('utf-8').strip()
        line2 = p2.stdout.readline().decode('utf-8').strip()
        print(line1)
      while True:
        line1 = p1.stdout.readline().decode('utf-8').strip()
        line2 = p2.stdout.readline().decode('utf-8').strip()
        print(line1.split(']')[0] + ', ' + line2.split('[')[1])
    finally:
      os.killpg(0, signal.SIGKILL) # kill all processes in my group
  '';

Thoughts

Overall I’m pretty satisfied. By far my biggest wishlist item to come out of this would be a cleaner way to wrap existing services. It’s not reasonable to have a flag on every service for whether or not to wrap it in a user namespace, and so there should be a way to apply this type of mechanical change to a service even when it’s not explicitly provided for. A related (rejected) feature request at the systemd level is here.