Transmit packet drop on wireguard interface

[jhalim10]

Hi guys,

Have you guys encountered transmit packet drop on wireguard interface? I have been trying to look around for the reason online, but I don't see a concrete cause.

I am running Linux 10/buster. I use default wireguard mtu which is 1420. I have about 80 wireguard peers. From my testing, more or less I can confirm that it is not related to the number of wireguard peers, but it is more to traffic related. How did I do the test? I keep the same number of wireguard peers, and playing around with the traffic with denying inbound and outbound traffic via BGP to/from some peers.

# ip -s -s link show wg1
wg1: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1420 qdisc pfifo_fast state UNKNOWN mode DEFAULT group default qlen 500
link/none
RX: bytes packets errors dropped overrun mcast
98410500814053 186491495929 0 0 0 0
RX errors: length crc frame fifo missed
0 0 0 0 0
TX: bytes packets errors dropped carrier collsns
30899039353106 139542572870 0 3309054509 0 0 =======> huge TX drop
TX errors: aborted fifo window heartbeat transns
0 0 0 0 0

Have you guys encountered the same issue? Do we ever know what the maximum bandwidth/throughput that is allowed in a wireguard interface? I don't see much outbound traffic by the way. Thanks in advance!

 

[GatorsFan]

most of the time this packet drop is due to having more then one device connected to the same point if anything else is on (cell phone) disconnect them and see if that helps

 

[jhalim10]

@GatorsFan thanks for your insight. Actually I am setting it as wireguard hub with just 1 wg interface to provide connectivity to more than 80 peers. Do you think it is causing issue? From my observation, it looks to be traffic/throughput related.

 

[jpnilson]

well its possible you are over running buffers on your nic or aggregating device resource issues. Are you seeing data loss on specific peers by chance or is it at the aggragating interface? I have had multiple instances where the culprit was a problem in the interweb rather than at one end or another. Not enough information. So many factors. Interactive traffic? Data transfers?

 

[jhalim10]

@jpnilson it looks like I am getting packet loss pinging to any peer tunnels IP when the TX packet drop is huge on wg interface. I confirm there is no packet drop in eth port that is used to transmit traffic of wg interface. Also I do not see packet loss when pinging the peer public IP endpoints.

Here is the stat of eth port that wg interface is using, as we can see, we have not much transmit traffic:

Packet loss only happens on tunnel IPs and this caused BGP sessions with peers are flapping since we are using tunnel IPs to peer

Is there any additional configuration needed in wg conf to allow a smoother transmit via wg interface without dropping packet?

 

[CaffeineAddict]

I confirm there is no packet drop in eth port that is used to transmit traffic of wg interface. Also I do not see packet loss when pinging the peer public IP endpoints.

Therefore the most likely case is that no gateway in between wg interface and end point (peer) has MTU lower than 1420.

The opposite is however not true, a peer with an MTU larger than 1420 doing the same thing as you may receive ICMP type 3 (destination unreachable) with code 4 (fragmentation needed) by any gateway that's between wg interface and themself.

You'll however never notice this in any of your logs because wg interface won't generate this traffic, this is up to a peer to discover.

Usual MTU is 1500, most peers and gateways will have this value set, but to know which value to actually set for your wg interface you can probe your closest node, the default gateway for MTU as follows:

# Get default gateway ip (replace wg with actual wg NIC name)
ip route show default dev wg

# Obtain first hoop MTU value, replace ip below with your default gateway IP
ping -M do -s 8900 192.168.1.1

MTU will be reported, you need to configure your wg NIC to the reported value in order to prevent IP fragmentation

---

If you want to fine tune MTU value toward specific system (ex. a peer) you can ping them with specific packet size, e.g.:

ping -M do -s 8900 <peer IP>

Which ever the first node to report MTU problem, you then ping again with that value minus 28, and so on until you figure out lowest MTU toward destination, the final value is that you can set to wg NIC. (this assumes peers connect to you, and not otherwise).

More details are below:

How to check supported MTU value for destination system and/or intermediate network ? - Red Hat Customer Portal

How to test a particular MTU value is support ? How I can check if this MTU size is supported by each Network Card ? Is there any checklist to verify our environment is prepared for enable MTU change ? How to test the custom Jumbo Frames are supported ?

access.redhat.com

However if you're VPN administrator you won't find this method useful to determine as to what MTU your wg interface should have, but the method is useful to figure out lowest MTU of gateways on route to target, between you and end peer, which is useful to troubleshoot issues that are outside of your control or if a peer is reporting a problem.

---

If this doesn't work, ex. doesn't fix UDP packet loss, next step is to configure packet queue length because with 80 peers you should be having high bandwidth connection, but first let us know if the above works.

 

[jhalim10]

@CaffeineAddict thanks for your comment.

MTU is configured as default on wg hub and wg clients (1420). We don't configure jumbo MTU on the interfaces.

As for default gw, I believe wg is point to point, hence there won't have any gw:

# ip route show default dev wg1
#

Example pinging to 1 of wg client:

# ping -M do -s 8900 100.100.1.51
PING 100.100.1.51 (100.96.1.51) 8900(8928) bytes of data.
ping: local error: Message too long, mtu=1420
ping: local error: Message too long, mtu=1420
ping: local error: Message too long, mtu=1420
^C
--- 100.100.1.51 ping statistics ---
3 packets transmitted, 0 received, +3 errors, 100% packet loss, time 32ms

Example pinging to 1 of endpoint (public IP is fake for this example):

# ping -M do -s 8900 189.125.82.72
PING 189.125.82.72 (189.125.82.72) 8900(8928) bytes of data.
ping: local error: Message too long, mtu=1500
ping: local error: Message too long, mtu=1500
^C
--- 189.125.82.72 ping statistics ---
2 packets transmitted, 0 received, +2 errors, 100% packet loss, time 19ms

Example of packet loss detected via tunnel:

n1 (100.100.1.1) 2024-09-30T03:34:15+0000
Keys: Help Display mode Restart statistics Order of fields
quit Packets Pings
Host Loss% Snt Last Avg Best Wrst StDev
1. 100.100.1.29 25.6% 18 81.5 82.7 26.1 125.3 19.4

Example of no packet loss detected pinging the endpoint of tunnel to 100.100.1.29 (Take note the source and destination public IP is fake for this example):

n1 (60.18.250.210) 2024-09-30T03:36:21+0000
Keys: Help Display mode Restart statistics Order of fields
quit Packets Pings
Host Loss% Snt Last Avg Best Wrst StDev
1. 100.122.241.116 0.0% 21 0.8 1.1 0.8 5.3 1.0
2. 100.124.112.27 0.0% 21 0.7 0.6 0.5 0.7 0.1
3. 100.98.167.2 0.0% 21 0.8 0.8 0.8 0.9 0.0
4. 100.98.3.160 0.0% 20 1.2 1.2 1.2 1.3 0.0
5. 126.141.28.146 0.0% 20 2.4 2.4 2.0 5.0 0.7
6. 181.21.247.147 0.0% 20 24.4 24.5 24.3 25.7 0.3
7. 181.21.247.147 0.0% 20 24.0 24.1 24.0 24.3 0.1
8. 181.21.245.213 0.0% 20 24.4 24.4 24.4 24.5 0.0
9. 181.21.245.213 0.0% 20 23.8 23.9 23.8 23.9 0.0
10. 181.21.246.195 0.0% 20 23.9 23.9 23.8 23.9 0.0
11. 189.46.70.197 0.0% 20 24.0 24.0 23.9 24.1 0.1
12. 189.46.80.131 0.0% 20 23.6 23.6 23.6 23.7 0.0

The packet loss or high round trip delay is observed to all wg client tunnel IPs when there is a heavy packet drop on TX wg1 interface of wg hub. Do we have bandwidth or buffer limit on wg interface, and are these adjustable? Thanks.

 

[CaffeineAddict]

MTU is configured as default on wg hub and wg clients (1420)

How do you know clients also use MTU of 1420?
Aren't clients other peers who connect to your VPN server?

When you say "Transmit packet drop" what exactly do you mean? transmit into which direction? from which node toward which node exactly?

and playing around with the traffic with denying inbound and outbound traffic via BGP to/from some peers.

I'm really confused about your usage of BGP (Border Gateway Protocol), BGP is normally used by large networks, ex tier 1 networks between 2 countries and similar.

We don't configure jumbo MTU on the interfaces.

Purpose of my previous post was to discover lowest MTU on path between you and a node toward which packet drop is observed, for which you need to send large packets with DF flag set and resend then with reported MTU until lowest is found.

UDP packets are subject to loss, if they're fragmented due to MTU that means packet loss possibility is even greater because only one fragmented datagram will make whole packet useless and will be dropped by a gateway.

Packet fragmentation itself is however not what's causing drops.

As for default gw, I believe wg is point to point, hence there won't have any gw:

It still has to travel over some network, internet, unless you physically connect directly peers which I don't think you do.

You can run sudo ss -tunlp state all to figure out if SOCKS port is open locally, and to figure out where packets are sent from VPN interface using the SOCKS port.

Doing this over wg interface might no be useful since MTU is already lower than expected, but over normal NIC it could provide you with more info, all you need is next hop which you can obtain from the command above.

Unfortunally the IP's you provided mean nothing to me to be able to distinguish nodes for you and give hints, the only thing known is that your peer which you pinged against is behind CGNAT which means you'll not be able to reach them directly.

Do we have bandwidth or buffer limit on wg interface

I'd say buffer issue because if the buffer is filled then subsequent packets are dropped until the queue gets some space.

A queue of 500 might be too low and is useful for real time applications like video streaming, online gaming etc. to reduce latency, larger queues are however needed to prevent packet loss.

You can get queue values with:

# Get kernel backlog queue
sudo sysctl net.core.netdev_max_backlog

# Get txqueuelen of NIC
ip link show dev wg0

Depending on your bandwidth and actual usage of it you might need queues of 5000 or more.

e.g. to set it to 5000:

# Set txqueuelen
sudo ip link set dev wg0 txqueuelen 5000

# Set kernel backlog queue
echo "net.core.netdev_max_backlog = 5000" | sudo tee /etc/sysctl.d/10-netdev_max_backlog.conf > /dev/null

# And to load config run
sudo sysctl -p /etc/sysctl.d/10-netdev_max_backlog.conf

 

[jhalim10]

How do you know clients also use MTU of 1420?
Aren't clients other peers who connect to your VPN server?

When you say "Transmit packet drop" what exactly do you mean? transmit into which direction? from which node toward which node exactly?

It is because the clients site wg are managed by me as well. TX packet drop is observed on wg interface:

ifconfig wg1 | grep TX
TX packets 184731151624 bytes 40316597535617 (36.6 TiB)
TX errors 0 dropped 3332631943 ====> this one overruns 0 carrier 0 collisions 0

I'm really confused about your usage of BGP (Border Gateway Protocol), BGP is normally used by large networks, ex tier 1 networks between 2 countries and similar.

Basically we use BGP to exchange the prefixes between each site and hub

Depending on your bandwidth and actual usage of it you might need queues of 5000 or more.

e.g. to set it to 5000:

The txqueuelen was 500 before on wg1 interface, and I just set it to 5000, this stopped the TX dropped in wg1 interface!

ifconfig wg1 | egrep "TX|tx"
unspec 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00 txqueuelen 5000 (UNSPEC)
TX packets 184757205865 bytes 40322019959137 (36.6 TiB)
TX errors 0 dropped 3332631943 overruns 0 carrier 0 collisions 0

However I am still packet loss pinging the tunnel endpoints though lol , I am monitoring this closely.

kernel backlog has been set to 50000 before , so I don't think I need to configure anything here right?

sysctl net.core.netdev_max_backlog
net.core.netdev_max_backlog = 50000

@CaffeineAddict do you think I need to change any other parameter? This is what the wg1 interface is now. How about qdisc? Right now it is set to default pfifo_fast. Thanks.

ip link show dev wg1
7: wg1: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc pfifo_fast state UNKNOWN mode DEFAULT group default qlen 5000

 

[CaffeineAddict]

The txqueuelen was 500 before on wg1 interface, and I just set it to 5000, this stopped the TX dropped in wg1 interface!

However I am still packet loss pinging the tunnel endpoints though lol , I am monitoring this closely.

do you think I need to change any other parameter?

Good news because now we know what's is the likely problem, since the txqueuelen increases qdisc queue which applies to transmit, txqueuelen is measured in packets, however there is also driver queue which is limited by the interface.

You can see current driver queue by running:

sudo ethtool -g wg1

In that output you'll see 2 sections:

• Pre-set maximums
• Current hardware settings

Current hardware settings is what's currently set, and the Pre-set maximums is the maximum value you can set to be applied to "Current hardware settings"

Each of these sections has TX and RX values, which correspond to driver queue measured in descriptors.
If reported TX in "Current hardware settings" is lower than "Pre-set maximums` you can increase it as follows, paying attention that the value you set is not larger than equivalent in "Pre-set maximums":

sudo ethtool -G wg1 tx 4096

You can do the same with RX for consistency:

sudo ethtool -G wg1 rx 4096

Consequences of setting too big value is performance, so set only as much as needed to prevent packet drop.

How about qdisc? Right now it is set to default pfifo_fast.

pfifo_fast is not optimal, I suggest you to set it to fq_codel.

To set it on your NIC use command:

sudo tc qdisc add dev wg0 root fq_codel

---

Keep in mind that all configuration above ethtool and tc are temporary and will be reset upon reboot.
Let me know if increasing driver queue and qdisc worked.

Question: How is you wg0 NIC managed?

• NetworkManager
• systemd-networkd
• networking
• manual

 

[jhalim10]

@CaffeineAddict over the night, I saw we are transmitting more on wg1 interface, and actually I still see TX dropped during that higher traffic. I didn't see packet drop when the traffic is below 50MiB. I am monitoring closely.

Also I am not able to get output for ethtool wg1. However it is running on top of eth2, and I can see that TX is set to 1024. Do you think this might be the issue? Should I increase it to 8192? Also will it bounce eth2 when I appy this? Will it be service impacting? Thanks

#ethtool -g wg1
netlink error: Operation not supported

# ethtool -g eth2
Ring parameters for eth2:
Pre-set maximums:
RX: 8192
RX Mini: n/a
RX Jumbo: n/a
TX: 8192
Current hardware settings:
RX: 8192
RX Mini: n/a
RX Jumbo: n/a
TX: 1024
RX Buf Len: n/a
CQE Size: n/a
TX Push: n/a

 

[CaffeineAddict]

Also I am not able to get output for ethtool wg1. However it is running on top of eth2, and I can see that TX is set to 1024. Do you think this might be the issue? Should I increase it to 8192?

I see, we have set qdisc length and type for wg interface but not for eth2 interface, and since one runs over the other that means both need to be set equally otherwise one of them will bottleneck (in this instance eth2 is bottleneck and wg nic can't take advantage of our modified config).

So first step is is to set qdisc for both as follows (substitute NIC names below as required):

sudo tc qdisc add dev wg1 root fq_codel
sudo tc qdisc add dev eth2 root fq_codel

sudo ip link set dev wg1 txqueuelen 8192
sudo ip link set dev eth2 txqueuelen 8192

The above is only regarding qdisc, note that I've set it to 8192 (corresponding to maximum hardware capacity), next step is to correlate driver queue to the qdisc AND to be the same for both TX and RX as follows:

sudo ethtool -G wg1 rx 8192
sudo ethtool -G wg1 tx 8192

Wireguard NIC is virtual NIC so I guess that's why hardware properties can't be configured.

Note that the above settings will be be lost upon system reboot!
You did not answer to my question on about how your NIC's are managed so that I could give you steps to persist reboots.

Also will it bounce eth2 when I appy this? Will it be service impacting?

I don't know what you mean by bouncing but the only consequence is slightly higher CPU load and memory usage, but the actual amount should still be low and you shouldn't wory about that because packet loss is bigger issue.

 

[jhalim10]

Bash:
sudo tc qdisc add dev wg1 root fq_codel
sudo tc qdisc add dev eth2 root fq_codel

sudo ip link set dev wg1 txqueuelen 8192
sudo ip link set dev eth2 txqueuelen 8192

I have changed the txqueuelen of wg1 and eth2 to 8192 and changed the qdisc to fq_codel. It looks like after awhile the qdisc of eth2 is reverted back to the previous value which is mq:

#ip link show wg1
9: wg1: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1420 qdisc fq_codel state UNKNOWN mode DEFAULT group default qlen 8192
link/none
#ip link show eth2
4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 8192
link/ether

Bash:
sudo ethtool -G wg1 rx 8192
sudo ethtool -G wg1 tx 8192

Will this flap wg1 interface? Those peers that are connected to this wireguard hub is running traffic, hence I would like to know whether it is service impacting or not.

Do we need to do the same to eth2 tx? The tx is set with 1024 right now, but I don't see any tx dropped using the following command, and I believe the eth2 will flap if we change it?

# ethtool -S eth2 | grep drop | grep tx
tx_queue_dropped: 0
tx0_dropped: 0
tx1_dropped: 0
tx2_dropped: 0
tx3_dropped: 0
tx4_dropped: 0
tx5_dropped: 0
tx6_dropped: 0
tx7_dropped: 0
tx8_dropped: 0
tx9_dropped: 0
tx10_dropped: 0
tx11_dropped: 0
tx12_dropped: 0
tx13_dropped: 0
tx14_dropped: 0
tx15_dropped: 0
tx16_dropped: 0
tx17_dropped: 0
tx18_dropped: 0
tx19_dropped: 0
tx20_dropped: 0
tx21_dropped: 0
tx22_dropped: 0
tx23_dropped: 0
tx24_dropped: 0
tx25_dropped: 0
tx26_dropped: 0
tx27_dropped: 0
tx28_dropped: 0
tx29_dropped: 0
tx30_dropped: 0
tx31_dropped: 0
tx32_dropped: 0
tx33_dropped: 0
tx34_dropped: 0
tx35_dropped: 0
tx36_dropped: 0
tx37_dropped: 0
tx38_dropped: 0
tx39_dropped: 0
tx40_dropped: 0
tx41_dropped: 0
tx42_dropped: 0
tx43_dropped: 0
tx44_dropped: 0
tx45_dropped: 0
tx46_dropped: 0
tx47_dropped: 0
tx48_dropped: 0
tx49_dropped: 0
tx50_dropped: 0
tx51_dropped: 0
tx52_dropped: 0
tx53_dropped: 0
tx54_dropped: 0
tx55_dropped: 0
tx56_dropped: 0
tx57_dropped: 0
tx58_dropped: 0
tx59_dropped: 0
tx60_dropped: 0
tx61_dropped: 0
tx62_dropped: 0

 

[CaffeineAddict]

@jhalim10

If you had read my replies to you so far you would know answers to your questions, I stated the same thing twice already and even marked it in bold so that you can clearly see but all my effort seems to be in vain because it seems you don't read what I write.

Also I asked you same question not once but twice, which you ignored in both of my replies so I can't help if you're not giving input that I'm asking about.

I'm sure somebody else will be able to help because I have no patience to repeat same things over and over again and asking questions without getting answers to them.