Firewall

2022-10-16

Ansible inventory

With Ansible You can create an inventory of different computers grouped together based on various aspects. When You write a playbook You can define which group should be affected by the playbook, this way everything runs only on those machines where it should.

My inventory is quite simple:

kistasli ansible_host=139.144.72.56
puruttya ansible_host=192.168.0.200

For example when I want to modify firewall rules for kistasli (which is the hostname for my admin server) I just make sure the playbook host file only contains that.

Modifying firewall rules

WARNING: Please, please! Do it the right way!

I encourage You to always edit the playbook instead of direct iptables calls on the host. It is easy to forget what You have changed and it could cause You hard time trying to figure out what You have done to make something work or not work.

To upgrade firewall rules simply call (do not forget to remove header comments, so You can use it as a playbook):

ansible-playbook -u root -i inventory admin-rules.yml

Firewall basics

The iptables process flow looks like this:

I do not try to explain the internals of iptables (because I can not), but here are a couple of things to know.

tables: There are different tables for dealing with firewall rules. The two most frequent is the nat (purple) and filter (green) tables. Latter is the default when You do not specify the table.
chains: Each table has various chains. For example the filter table has INPUT, FORWARD and OUTPUT chains. To make it worse, You can define Your own chains.

Checking firewall rules

You can list firewall rules for filter table with the following command:

doas iptables -nvL

If You would like to see the nat table instead, try:

doas iptables -t nat -nvL

This command will show the policy and rules (in order!) for each chain. Order is important, as iptables checks the rules in order. So when You have a DROP everything rule, anything below that will not be reached. It will also print the number of packets and bytes matched by a given rule.

Policy

Policy is defined on a chain-basis. Usually the default policy is ACCEPT which means that each package gets accepted unless there is a rule to not accept it. When You setup a firewall it is pretty common to set INPUT and FORWARD chains policy to DROP. DROP (as You probably guessed) just drops packages. Sometimes instead of DROP they use REJECT. Reject also drops the package, but sends back a rejected message to the client to make it obvious that it’s request is rejected. When You DROP packages client just reach a timeout sooner or later.

Docker and iptables

When You setup a firewall (without Docker) You usually deal with the filter table, especially the INPUT and FORWARD chains as OUTPUT traffic is usually considered safe. (On a home network, a corporate network is a different story.)

Bypassing INPUT rules

Docker default firewall rules help to make it easy to deploy on development machines, but it usually means that it is exposed to everywhere. It is not a desired state when You run Docker images on a server and You want to restrict access to those services.

When You list iptables rules after starting Docker You will notice some extra chains beside the default ones: DOCKER, DOCKER-ISOLATION-STAGE-1, DOCKER-USER to name a few.

If You also check the nat table You will see something similar:

Chain PREROUTING (policy ACCEPT 9276 packets, 618K bytes)
 pkts bytes target     prot opt in     out     source               destination
 294K   32M DOCKER     all  --  *      *       0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL

The PREROUTING rule shown above sends everything to the DOCKER chain before they can reach the INPUT filter table.

The DOCKER chain in nat table rewrites every packet’s destination IP to the corresponding Docker container’s IP. For example the DNAT rule below matches all TCP packets (because of !) not coming from the Docker bridge network interface going from anywhere to anywhere with destination port being 8888 and rewrites its destination to 172.22.0.4 and port number 80.

Chain DOCKER (2 references)
 pkts bytes target     prot opt in     out     source               destination
    5   300 RETURN     all  --  br-f540a6981c87 *       0.0.0.0/0            0.0.0.0/0
    0     0 RETURN     all  --  docker0 *       0.0.0.0/0            0.0.0.0/0
    0     0 DNAT       tcp  --  !br-f540a6981c87 *       0.0.0.0/0            0.0.0.0/0            tcp dpt:8888 to:172.22.0.4:80

After PREROUTING(nat)->DOCKER(nat) we have a packet targeting 172.22.0.4. If You check the process flow above it is time to decide if the packet is for this host (and send it to INPUT chain) or targeting a different host (and has to be sent through the FORWARD chain). Because of PREROUTING the destination address differs from the host’s address, so this packet will be sent to the FORWARD chain.

If You check routing table with ip route there will be a similar line:

172.22.0.0/16 dev br-f540a6981c87 proto kernel scope link src 172.22.0.1

Which means we route the packet to br-f540a6981c87 to reach the desired container. Time to check the FORWARD rules (reduced for simplification):

Chain FORWARD (policy DROP 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
 263K   57M DOCKER-USER  all  --  *      *       0.0.0.0/0            0.0.0.0/0
  128  7873 DOCKER     all  --  *      br-f540a6981c87  0.0.0.0/0            0.0.0.0/0

Chain DOCKER (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 ACCEPT     tcp  --  !br-f540a6981c87 br-f540a6981c87  0.0.0.0/0            172.22.0.4           tcp dpt:80

We send every packet to DOCKER chain which is targeting br-f540a6981c87. The DOCKER chain finally ACCEPTs the package as IP and PORT both matches.

(iptables will also take care of reply packets to this request, so they will reach the original source IP of the request thanks to POSTROUTING nat chain and MASQUERADE.)

If You follow the steps in this example You will see that it will never reach INPUT filter rules. So even if You DROP all packets in INPUT chain Your server will serve every request coming from anywhere. This is probably not what You want :-(

DOCKER-USER filter chain for the rescue

In the previous example I have cheated a bit. I have ignored the DOCKER-USER target in FORWARD filter chain. By default it does not really matter as it contains only this rule:

Chain DOCKER-USER (1 references)
 1974  128K RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0

RETURN just returns the previous chain without doing anything with the packet. In this case that is the FORWARD chain, so the DOCKER target kicks in and everything else I have said before stands.

However DOCKER-USER is the place where You can place Your user defined rules. If those rules prohibit accepting packets, You are safe.

Here is mine for example:

Chain DOCKER-USER (1 references)
 pkts bytes target     prot opt in     out     source               destination
 257K   56M ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
  803 45181 ACCEPT     tcp  --  eth0   br+     0.0.0.0/0            0.0.0.0/0            ctstate NEW tcp dpt:443
    8   828 ACCEPT     udp  --  eth0   br+     0.0.0.0/0            0.0.0.0/0            udp dpt:51821
 5162  619K LOG        all  --  eth0   *       0.0.0.0/0            0.0.0.0/0            LOG flags 0 level 4 prefix "DOCKER-USER: "
 5162  619K DROP       all  --  eth0   *       0.0.0.0/0            0.0.0.0/0
 1974  128K RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0

As You can see the DROP part kicks in, so packets coming from outside (via the eth0 interface) get dropped unless they are RELATED/ESTABLISHED, targeting TCP port 443 (https) or UDP port 51821 (Netmaker/wireguard).

My firewall rules

I do not want to explain the rules line-by-line, but here are some highlights. You can find my current firewall setup here.

I prefer to keep ACCEPT as my default INPUT policy. The reasoning is simple. When You use DROP You have to take extra care before flushing records, it is easy to lock Yourself out from the server. For FORWARD it is simpler to go with DROP.

There are two types of action in these rules: append and insert, just like in iptables. Iptables rules match in order so take special care when appending/inserting rules.

The first step is to disable IPv6. I did not set up an IPv6 firewall yet, so the meantime it is easier to just disable it.
Make sure that filter policies are set properly.
Flush and set DOCKER-USER and INPUT filter rules.
Generic rules:
- accept ESTABLISHED,RELATED connections (DOCKER-USER and INPUT)
- accept loopback interface communication
- LOG and DROP: Log and drop every remaining packets in DOCKER-USER and INPUT (DOCKER-USER actually only drops eth0 to make it simpler). Log prefixes let You see which chain drops a given packet, so You can change You firewall rules when necessary.

Open ports (update: 2022-10-24)

This setup opens up the following ports:

ssh(22): It has nothing to do with Netmaker, just a secure way of reaching the VPS.
https(443): Traefik will listen on this port as a reverse proxy. Reverse proxies listen on a port and based on the router rules (domain, subdir etc.) will send requests to the proper service. In this case api calls go to netmaker, dashboard calls go to netmaker-ui and broker raw TCP packets sent to mosquitto.
Netmaker(51821-51822): Each Netmaker VPN needs an open UDP port on the server. I have two ports open because I will run a client on the same host and that needs to communicate somewhere separately from the server.

Checking firewall logs

Make sure You have started klogd:

doas /etc/init.d/klogd start
tail -f /var/log/messages

You will see something similar to this:

[169752.681823] DOCKER-USER: IN=eth0 OUT=br-f540a6981c87 MAC=f2:3c:93:15:ee:5a:fe:ff:ff:ff:ff:ff:08:00 SRC=190.167.253.241 DST=172.22.0.4 LEN=135 TOS=0x00 PREC=0x00 TTL=48 ID=20691 DF PROTO=UDP SPT=55678 DPT=53 LEN=115
[169761.068944] INPUT: IN=eth0 OUT= MAC=f2:3c:93:15:ee:5a:fe:ff:ff:ff:ff:ff:08:00 SRC=5.61.253.157 DST=170.187.188.160 LEN=44 TOS=0x08 PREC=0x20 TTL=56 ID=29643 DF PROTO=TCP SPT=80 DPT=49198 WINDOW=1460 RES=0x00 ACK SYN URGP=0

You can use grep to filter the logs. If You want to keep the logging on You might want to setup logrotate.

Finally, time to setup Netmaker.

Room for improvement (TODO)

Security concerns

When You apply these firewall rules in the time between flushing the rules and appending the last DROP rule You are not protected in that particular chain at all. It perfectly fits my needs, but maybe not good enough for Yours.

Legacy

Iptables is the legacy firewall interface for the Linux kernel. It is now superseded by nftables. Some distributions package a new version of iptables which works with nftables under the hood, but Alpine Linux still uses the old version.

There are also modern firewalls, like firewalld which interacts via D-BUS.