Solved Use DoT/DoH but keep UDP Port 53 free on Debian

Harhkl · Apr 14, 2024

Did not work unfortunately. The ssh option for you would still be possible FYI

CaffeineAddict · Apr 14, 2024

Harhkl said:
Did not work unfortunately. The ssh option for you would still be possible FYI

Thank you but I'm not sure about remote control help, public forums usually don't allow such kind of help.

But what do you mean by "Did not work"? I would appreciate outputs from unbound status and dig google.com
It's possible unbound works but something else doesn't so I need output to confirm to know what we're troubleshooting.

Your previous log shows routing errors and we confirmed this with your routing table, this might be caused by your VPN modifying routing table unless you did add routes yourself?

Harhkl · Apr 14, 2024

The routes are all from a fresh AWS EC2 instance. No VPN there.

unbound output:

Code:

systemctl status unbound
sudo: unable to resolve host ip-172-16-24-121: Temporary failure in name resolution
● unbound.service - Unbound DNS server
     Loaded: loaded (/lib/systemd/system/unbound.service; enabled; preset: enabled)
     Active: active (running) since Sun 2024-04-14 11:41:48 UTC; 12min ago
       Docs: man:unbound(8)
    Process: 410 ExecStartPre=/usr/libexec/unbound-helper chroot_setup (code=exited, status=0/SUCCESS)
    Process: 416 ExecStartPre=/usr/libexec/unbound-helper root_trust_anchor_update (code=exited, status=0/SUCCESS)
   Main PID: 418 (unbound)
      Tasks: 1 (limit: 515)
     Memory: 14.6M
        CPU: 171ms
     CGroup: /system.slice/unbound.service
             └─418 /usr/sbin/unbound -d -p

Apr 14 11:41:48 ip-172-16-24-121 systemd[1]: Starting unbound.service - Unbound DNS server...
Apr 14 11:41:48 ip-172-16-24-121 unbound[418]: [418:0] notice: init module 0: subnetcache
Apr 14 11:41:48 ip-172-16-24-121 unbound[418]: [418:0] warning: subnetcache: prefetch is set but not working for data originating from the subnet module cache.
Apr 14 11:41:48 ip-172-16-24-121 unbound[418]: [418:0] notice: init module 1: validator
Apr 14 11:41:48 ip-172-16-24-121 unbound[418]: [418:0] notice: init module 2: iterator
Apr 14 11:41:48 ip-172-16-24-121 unbound[418]: [418:0] info: start of service (unbound 1.17.1).
Apr 14 11:41:48 ip-172-16-24-121 systemd[1]: Started unbound.service - Unbound DNS server.

Code:

dig google.com
;; communications error to ::1#53: connection refused
;; communications error to ::1#53: connection refused
;; communications error to ::1#53: connection refused
;; communications error to 127.0.0.1#53: connection refused

; <<>> DiG 9.18.24-1-Debian <<>> google.com
;; global options: +cmd
;; no servers could be reached

Code:

netstat -tulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name   
tcp        0      0 127.0.0.1:5555          0.0.0.0:*               LISTEN      418/unbound         
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      414/sshd: /usr/sbin
tcp6       0      0 :::22                   :::*                    LISTEN      414/sshd: /usr/sbin
udp        0      0 127.0.0.1:5555          0.0.0.0:*                           418/unbound         
udp        0      0 172.16.24.121:68        0.0.0.0:*                           366/systemd-network

CaffeineAddict · Apr 14, 2024

I spotted something is missing in your config file:

Under server: clause add this:

Bash:

    # The following line will configure unbound to perform cryptographic
    # DNSSEC validation using the root trust anchor.
    # Unbound needs rw access to this directory
    auto-trust-anchor-file: "/var/lib/unbound/root.key"

    # Enable or disable whether IPv6 queries are answered or issued
    do-ip6: no

Your dig command is querying IPv6 loopback, this must not be the case, lines above should handle that.

Please also add this to server: clause:
access-control: 127.0.0.1/8 allow

Your netstat -tulpn command does not show that unbound is listening to 5555, please share output of:

sudo ss -tunlp

EDIT:
Sorry I misread, it does listen to 5555

Harhkl · Apr 14, 2024

The do-ip6: no is already there.

here the ss output:

Code:

ss -tunlp
sudo: unable to resolve host ip-172-16-24-121: Temporary failure in name resolution
Netid           State            Recv-Q           Send-Q                          Local Address:Port                       Peer Address:Port           Process                                            
udp             UNCONN           0                0                                   127.0.0.1:5555                            0.0.0.0:*               users:(("unbound",pid=418,fd=3))                  
udp             UNCONN           0                0                          172.16.24.121%ens5:68                              0.0.0.0:*               users:(("systemd-network",pid=366,fd=18))          
tcp             LISTEN           0                256                                 127.0.0.1:5555                            0.0.0.0:*               users:(("unbound",pid=418,fd=4))                  
tcp             LISTEN           0                128                                   0.0.0.0:22                              0.0.0.0:*               users:(("sshd",pid=414,fd=3))                      
tcp             LISTEN           0                128                                      [::]:22                                 [::]:*               users:(("sshd",pid=414,fd=4))

Harhkl · Apr 14, 2024

Nothing changed, same outputs

CaffeineAddict · Apr 14, 2024

@Harhkl
Sorry my mistake about listening port, I edited my post in the meantime.

Here is my complete config file adjusted for your setup, please replace yours with mine:

Bash:

# Configuration file: /etc/unbound/unbound.conf
# Includes files: /etc/unbound/unbound.conf.d/unbound.conf
# https://nlnetlabs.nl/documentation/unbound/unbound.conf

# sudo systemctl restart unbound

# Server Options
server:

    #
    # General options
    #

    # Interface to use to connect to the network
    # This interface is listened to for queries from clients, and answers to clients are given from it
    # If an interface name is used instead of an ip address, the list of ip addresses on that interface are used.
    # the default is to listen to localhost
    interface: 127.0.0.1

    # The port number, default 53, on which the server responds to queries
    port: 5555

    # By default only localhost is allowed, the rest is refused
    access-control: 127.0.0.1/8 allow

    # If given, after binding the port the user privileges are dropped (Default is "unbound")
    username: "unbound"

    # The Unbound server forks into the background as a daemon
    # Set the value to no when Unbound runs as systemd service (Default is yes)
    do-daemonize: no

    # The number of threads to create to serve clients
    num-threads: 2

    # Default is nothing, using builtin hints for the IN class
    # The default may become outdated, when servers change, therefore it is good practice to use a root-hints file
    root-hints: "/usr/share/dns/root.hints"

    #
    # Privacy
    #

    # If enabled id.server and hostname.bind queries are refused
    hide-identity: yes

    # If enabled version.server and version.bind queries are refused
    hide-version: yes

    # If enabled the HTTP header User-Agent is not set
    hide-http-user-agent: no

    # Send minimum amount of information to upstream servers
    qname-minimisation: yes

    #
    # Hardening
    #

    # Very small EDNS buffer sizes from queries are ignored
    harden-short-bufsize: yes

    # Will trust glue only if it is within the servers authority.
    harden-glue: yes

    # Require DNSSEC data for trust-anchored zones, if such data is absent, the zone becomes bogus.
    # If turned off you run the risk of a downgrade attack that disables security for a zone.
    harden-dnssec-stripped: yes

    # The nxdomain must be secure
    harden-below-nxdomain: yes

    #
    # Logging
    #

    # Level 0 means no verbosity, only errors
    # Level 1 (default) gives operational information
    verbosity: 1

    # Send log messages to log facility LOG_DAEMON (rsyslog)
    use-syslog: yes

    # Prints one line per query to the log
    log-queries: no

    # Prints one line per reply to the log
    log-replies: no

    # Prints the word "query" and "reply" with log-queries and log-replies.
    log-tag-queryreply: yes

    # Print log lines that say why queries return SERVFAIL to clients
    log-servfail: yes

    # Print log lines to inform about local zone actions.
    log-local-actions: no

    # Have the validator print validation failures to the log (Default is 0, off)
    val-log-level: 1

    #
    # Caching (in seconds)
    #

    # Time to live maximum for RRsets and messages in the cache
    cache-max-ttl: 21600 # 6h (Default is 86400 - 1day)

    # Time to live minimum for RRsets and messages in the cache
    cache-min-ttl: 0

    # Time to live maximum for negative responses
    cache-max-negative-ttl: 3600 # 1h

    #
    # Connection
    #

    # Enable or disable whether IPv4 queries are answered or issued
    do-ip4: yes

    # Enable or disable whether IPv6 queries are answered or issued
    do-ip6: no

    # Enable or disable whether TCP queries are answered or issued
    do-tcp: yes

    # Enable or disable whether UDP queries are answered or issued
    do-udp: yes

    #
    # TLS
    #

    # The upstream queries use TLS only for transport
    tls-upstream: yes

    # The port number on which to provide TCP TLS service,
    tls-port: 853

    # These certificates are used for authenticating connections made to outside peers.
    tls-cert-bundle: "/etc/ssl/certs/ca-certificates.crt"

    #
    # DNSSEC
    #

    # Uses the DNSSEC NSEC chain to synthesize NXDOMAIN and other denials, using information from previous NXDOMAINs answers.
    # It helps to reduce the query rate towards targets that get a very high nonexistent name lookup rate.
    aggressive-nsec: yes

    # This check sees if RRSIGs are present in the answer, when dnssec is expected,
    # and retries another authority if RRSIGs are unexpectedly missing.
    disable-dnssec-lame-check: no

    # "validator iterator" will turn on DNSSEC validation
    module-config: "validator iterator"

    # Mark bogus messages as indeterminate, otherwise SERVFAIL
    # For messages that are found to be secure the AD bit is set in replies
    val-permissive-mode: no

    # Count of validation restarts with another authority in case of failed validation
    val-max-restart: 5

    # TTL for data that has failed validation
    val-bogus-ttl: 60

    # The following line will configure unbound to perform cryptographic
    # DNSSEC validation using the root trust anchor.
    # Unbound needs rw access to this directory
    auto-trust-anchor-file: "/var/lib/unbound/root.key"
    #auto-trust-anchor-file: "/usr/share/dns/root.key"

# There may be multiple forward-zone: clauses.
# Each with a name: and zero or more hostnames or IP addresses.
# For the forward zone this list of nameservers is used to forward the queries to.
forward-zone:

    # Name of the forward zone.
    # This is the full domain name of the zone.
    name: "."

    # Enabled or disable whether the queries to this forwarder use TLS for transport
    forward-tls-upstream: yes

    # If enabled, data inside the forward is not cached
    # This is useful when you want immediate changes to be visible (Default is no)
    forward-no-cache: no

    # The servers listed as forward-host: and forward-addr: have to handle further recursion for the query
    # https://github.com/DigitaleGesellschaft/DNS-Resolver
    # https://www.quad9.net/support/faq/

    # IPv4 primary
    forward-addr: 185.95.218.42@853#dns.digitale-gesellschaft.ch
    forward-addr: 9.9.9.9@853#dns.quad9.net

    # IPv4 secondary
    forward-addr: 185.95.218.43@853#dns.digitale-gesellschaft.ch
    forward-addr: 149.112.112.112@853#dns.quad9.net

    # IPv6 primary
    forward-addr: 2a05:fc84::42@853#dns.digitale-gesellschaft.ch

    # Secure IPv6 primary: Blocklist, DNSSEC, No EDNS Client-Subnet
    forward-addr: 2620:fe::fe@853#dns.quad9.net

    # IPv6 secondary
    forward-addr: 2a05:fc84::43@853#dns.digitale-gesellschaft.ch

    # Secure IPv6 secondary: Blocklist, DNSSEC, No EDNS Client-Subnet
    forward-addr: 2620:fe::9@853#dns.quad9.net

# If this is enabled, the unbound-control(8) utility can be used to send commands to the running Unbound server
remote-control:

    # The option is used to enable remote control (default is no)
    control-enable: yes

    # Give IPv4 or IPv6 addresses or local socket path to listen on for control commands.
    # If you set it to an absolute path, a unix domain socket is used.
    # This socket does not use the certificates and keys, so those files need not be present.
    # To restrict access, Unbound sets permissions on the file to the user and group that is configured,
    # the access bits are set to allow the group members to access the control socket file.
    # By default localhost (127.0.0.1 and ::1) is listened to.
    control-interface: /run/unbound.ctl

    # The port number to listen on for IPv4 or IPv6 control interfaces, (default is 8953)
    control-port: 8953

You must restart unbound every time you change something, so after using my config do:

Bash:

sudo systemctl restart unbound

And try again please.

Harhkl · Apr 14, 2024

I acutally do systemctl restart each time and after that a full reboot, just to make sure.

Code:

journalctl -b -u unbound | cat
Apr 14 12:14:19 ip-172-16-24-121 systemd[1]: Starting unbound.service - Unbound DNS server...
Apr 14 12:14:20 ip-172-16-24-121 unbound[426]: [426:0] notice: init module 0: validator
Apr 14 12:14:20 ip-172-16-24-121 unbound[426]: [426:0] notice: init module 1: iterator
Apr 14 12:14:20 ip-172-16-24-121 systemd[1]: Started unbound.service - Unbound DNS server.
Apr 14 12:14:20 ip-172-16-24-121 unbound[426]: [426:0] info: start of service (unbound 1.17.1).

Still not working, even with your config. I am assuming it has to do with systemd network not using the configurations correctly. Here the output of that service again: https://pastebin.com/raw/t8Tpkvf9

CaffeineAddict · Apr 14, 2024

Harhkl said:
Still not working, even with your config. I am assuming it has to do with systemd network not using the configurations correctly. Here the output of that service again: https://pastebin.com/raw/t8Tpkvf9

While I inspect the log please tell whether you can surf the web on that system?

Harhkl · Apr 14, 2024

CaffeineAddict said:
While I inspect the log please tell whether you can surf the web on that system?

If one ignores the missing DNS resolution, then yes:

Code:

curl -v http://93.184.216.34
*   Trying 93.184.216.34:80...
* Connected to 93.184.216.34 (93.184.216.34) port 80 (#0)
> GET / HTTP/1.1
> Host: 93.184.216.34
> User-Agent: curl/7.88.1
> Accept: */*
>
< HTTP/1.1 404 Not Found
< Content-Type: text/html
< Date: Sun, 14 Apr 2024 12:22:54 GMT
< Server: ECS (dce/2694)
< Content-Length: 345
<
<?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
         "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
    <head>
        <title>404 - Not Found</title>
    </head>
    <body>
        <h1>404 - Not Found</h1>
    </body>
</html>
* Connection #0 to host 93.184.216.34 left intact

CaffeineAddict · Apr 14, 2024

ens5: found matching network '/run/systemd/network/10-netplan-ens5.network'.
Apr 14 12:17:26 ip-172-16-24-121 systemd-networkd[369]: ens5: Configuring with /run/systemd/network/10-netplan-ens5.network.
Apr 14 12:17:26 ip-172-16-24-121 systemd[1]: Started systemd-networkd.service - Network Configuration.

Your interface ens5 is being put down due to using configuration that is not the one I gave you.

Flags change: -UP -LOWER_UP -RUNNING
Apr 14 12:17:26 ip-172-16-24-121 systemd-networkd[369]: ens5: Link DOWN
Apr 14 12:17:26 ip-172-16-24-121 systemd-networkd[369]: ens5: Lost carrier

The log does not show at all that the file we configured is being used.

Harhkl · Apr 14, 2024

Thats what I mean. The config is not being used.

CaffeineAddict · Apr 14, 2024

Harhkl said:
Thats what I mean. The config is not being used.

Please do:

Bash:

cat /run/systemd/network/10-netplan-ens5.network

Harhkl · Apr 14, 2024

CaffeineAddict said:
Please do:

Bash:

cat /run/systemd/network/10-netplan-ens5.network

Code:

[Match]
PermanentMACAddress=02:3b:35:e9:3d:2b
Name=ens5

[Network]
DHCP=ipv4
LinkLocalAddressing=ipv6

[DHCP]
RouteMetric=100
UseMTU=true

CaffeineAddict · Apr 14, 2024

@Harhkl
Nice, so we found the bad guy.

I would first backup this file just in case something goes wrong so that you can restore it:

Bash:

sudo cp /run/systemd/network/10-netplan-ens5.network /run/systemd/network/10-netplan-ens5.network.backup

And then edit the file with what we want:

Bash:

sudo nano /run/systemd/network/10-netplan-ens5.network

Replace contents with:

Bash:

[Match]
Name=ens5
Type=ether

[Network]
DHCP=ipv4
DNS=127.0.0.1:5555

Restart service:

Bash:

sudo systemctl restart systemd-networkd

And then try again testing, share output of dig google.com

Harhkl · Apr 14, 2024

Did not work, I tried this already this morning.

Code:

dig google.com
;; communications error to ::1#53: connection refused
;; communications error to ::1#53: connection refused
;; communications error to ::1#53: connection refused
;; communications error to 127.0.0.1#53: connection refused

; <<>> DiG 9.18.24-1-Debian <<>> google.com
;; global options: +cmd
;; no servers could be reached

CaffeineAddict · Apr 14, 2024

Harhkl said:

Did not work, I tried this already this morning.

Code:

dig google.com
;; communications error to ::1#53: connection refused
;; communications error to ::1#53: connection refused
;; communications error to ::1#53: connection refused
;; communications error to 127.0.0.1#53: connection refused

; <<>> DiG 9.18.24-1-Debian <<>> google.com
;; global options: +cmd
;; no servers could be reached

Please add this line to server: clause, restart unbound and try again:

Bash:

interface: ::1
do-ip6: yes

This will enable IPv6 listener locally.

Harhkl · Apr 14, 2024

Same dig output

Code:

dig google.com
;; communications error to ::1#53: connection refused
;; communications error to ::1#53: connection refused
;; communications error to ::1#53: connection refused
;; communications error to 127.0.0.1#53: connection refused

; <<>> DiG 9.18.24-1-Debian <<>> google.com
;; global options: +cmd
;; no servers could be reached

At this point it may be faster for you to try to set it up from scratch yourself and send me the files. Maybe it is not possible with an AWS EC2 debian instance as well.

CaffeineAddict · Apr 14, 2024

I don't know why the heck are requests being sent to port 53, this is obvious problem because unbound is listening to 5555

Troubleshooting should continue by investigating why this happens, but I have zero XP with AWS and cloud solutions.

CaffeineAddict · Apr 14, 2024

Harhkl said:
At this point it may be faster for you to try to set it up from scratch yourself and send me the files.

Good news for you because I tested this scenario on my machine and after experiencing same issue was able to solve the problem by doing loopback NAT using nftables

You do not have nftables installed therefore first step is to install it to be able to redirect DNS traffic to custom port:

Bash:

# See if available
sudo apt search nftables --names-only

# Install
sudo apt install nftables

Next enable and auto start nftables at boot:

Bash:

sudo systemctl enable nftables
sudo systemctl start nftables

# Verify status OK
sudo systemctl status nftables

When done, the following is nft script to load ruleset, save it to some directory on your server.

Filename: unbound.nft
Usage: sudo nft -f ./unbound.nft

Bash:

#!/usr/sbin/nft -f

#
# ruleset to redirect local machine IPv4 DNS requests to custom port on local machine
#

# NAT table
add table nat_4

# Sees packets that originated from processes in the local machine and allows DNAT
add chain nat_4 output_nat_4 {
    # -165 = After conntrack and before mangle
    type nat hook output priority -165; policy accept;
}

# Redirect local machine DNS requests to unbound custom port 5555
add rule nat_4 output_nat_4 meta oiftype loopback udp dport 53 redirect to 5555
add rule nat_4 output_nat_4 meta oiftype loopback tcp dport 53 redirect to 5555

Verify ruleset loaded:

Bash:

sudo nft list ruleset

Next step is to ensure this ruleset loads at system boot, the following is a script to export ruleset to default location:

Filename: export.sh
Usage: sudo ./export.sh

Bash:

#!/bin/bash

#
# Export nftables ruleset to default location /etc/nftables.conf
#

printf "%s\n" "#!/usr/sbin/nft -f" "" "flush ruleset" "" > /etc/nftables.conf
nft list ruleset >> /etc/nftables.conf

Yesterday however we messed up your systemd-networkd configuration because addresses were specified without network suffix (netmask), so we should fix that too because sooner or later your system might loose access to internet.

NOTE: Make sure there are no trailing white spaces in the config files below!
NOTE: Make sure between equal sign there are no white spaces ex. Name=Value instead of Name = Value

Configure loopback interface as follows:

Bash:

sudo nano /etc/systemd/network/lo.network
# Paste loopback config below here (replace or verify)

Bash:

[Match]
Name=lo
Type=loopback

[Network]
Address=127.0.0.1/8

Also configure physical NIC:

Bash:

sudo nano /run/systemd/network/10-netplan-ens5.network
# Paste ens5 config below here (replace or verify)

Bash:

[Match]
Name=ens5
Type=ether
PermanentMACAddress=02:3b:35:e9:3d:2b

[Network]
DHCP=ipv4
DNS=127.0.0.1:5555

[DHCP]
RouteMetric=100
UseMTU=true

Apply same config above (ens5) to the following file:

Bash:

sudo nano /etc/systemd/network/ens5.network

Restart service to apply

Bash:

sudo systemctl restart systemd-networkd

# Verify status OK
sudo systemctl status systemd-networkd

For more information about systemd-networkd configuration options see:

systemd.network

Make sure you use unbound config file I gave you in my post #27 then restart unbound to apply and clear local cache

Bash:

sudo systemctl restart unbound

# Verify status OK
sudo systemctl status unbound

This setup should immediately make digging work:

Bash:

dig google.com

Note that in the output of the dig command you'll see that port 53 is contacted, but behind the scene it's redirected to 5555 as specified in the config file.
Please share output of the dig command.

If this doesn't work for what ever reason then we'll modify ruleset to also handle IPv6 loopback but I omitted this part for now.

Solved Use DoT/DoH but keep UDP Port 53 free on Debian

New Member

Well-Known Member

New Member

Well-Known Member

New Member

New Member

Well-Known Member

New Member

Well-Known Member

New Member

Well-Known Member

New Member

Well-Known Member

New Member

Well-Known Member

New Member

Well-Known Member

New Member

Well-Known Member

Well-Known Member