How to calculate TCP socket memory usage?

mitsos1os

New Member
Joined
Dec 15, 2020
Messages
4
Reaction score
1
Credits
58
Hi everyone,

I am running a Debian GNU/Linux 9.5 (stretch) with kernel: 4.9.0-7-amd64.

I am trying to figure out TCP sockets memory usage, but with my calculations I seem to be missing something.

According to the following outputs

/proc/net/sockstat:

Code:
sockets: used 779
TCP: inuse 23 orphan 0 tw 145 alloc 177 mem 4451
UDP: inuse 5 mem 2
UDPLITE: inuse 0
RAW: inuse 0
FRAG: inuse 0 memory 0

mem metric is the number of pages (4K) used by TCP memory. So TCP Memory usage equals 4451 * 4 = 17804k

ss -atmp:


Code:
State      Recv-Q Send-Q Local Address:Port                 Peer Address:Port              
LISTEN     0      128        *:ssh                      *:*                     users:(("sshd",pid=559,fd=3))
     skmem:(r0,rb12582912,t0,tb12582912,f0,w0,o0,bl0,d712)
LISTEN     0      4096   127.0.0.1:8125                     *:*                     users:(("netdata",pid=21419,fd=33))
     skmem:(r0,rb33554432,t0,tb12582912,f0,w0,o0,bl0,d0)
LISTEN     0      4096       *:19999                    *:*                     users:(("netdata",pid=21419,fd=4))
     skmem:(r0,rb33554432,t0,tb12582912,f0,w0,o0,bl0,d0)
LISTEN     0      32768      *:3999                     *:*                     users:(("protokube",pid=3504,fd=9))
     skmem:(r0,rb12582912,t0,tb12582912,f0,w0,o0,bl0,d0)
LISTEN     0      32768  127.0.0.1:19365                    *:*                     users:(("kubelet",pid=2607,fd=10))
     skmem:(r0,rb12582912,t0,tb12582912,f0,w0,o0,bl0,d0)
LISTEN     0      32768  127.0.0.1:10248                    *:*                     users:(("kubelet",pid=2607,fd=29))
     skmem:(r0,rb12582912,t0,tb12582912,f0,w0,o0,bl0,d0)
LISTEN     0      32768  127.0.0.1:10249                    *:*                     users:(("kube-proxy",pid=3250,fd=10))
     skmem:(r0,rb12582912,t0,tb12582912,f0,w0,o0,bl0,d0)
LISTEN     0      128        *:sunrpc                   *:*                     users:(("rpcbind",pid=232,fd=8))
     skmem:(r0,rb12582912,t0,tb12582912,f0,w0,o0,bl0,d0)
ESTAB      0      0      172.18.25.47:ssh                  46.198.221.224:35084                 users:(("sshd",pid=20049,fd=3),("sshd",pid=20042,fd=3))
     skmem:(r0,rb12582912,t0,tb12582912,f0,w0,o0,bl0,d0)
TIME-WAIT  0      0      100.96.18.1:48226                100.96.18.110:3006               
  
ESTAB      0      0      172.18.25.47:62641                172.18.18.165:3999                  users:(("protokube",pid=3504,fd=11))
     skmem:(r0,rb12582912,t0,tb12582912,f0,w0,o0,bl0,d15390)
ESTAB      0      0      172.18.25.47:3999                 172.18.63.198:46453                 users:(("protokube",pid=3504,fd=17))
     skmem:(r0,rb12582912,t0,tb12582912,f0,w0,o0,bl0,d0)
SYN-SENT   0      1      172.18.25.47:28870                172.18.23.194:4000                  users:(("protokube",pid=3504,fd=3))
     skmem:(r0,rb12582912,t1280,tb12582912,f2816,w1280,o0,bl0,d0)
TIME-WAIT  0      0      100.96.18.1:34744                100.96.18.108:3008               
  
ESTAB      0      0      172.18.25.47:3999                 172.18.18.165:23733                 users:(("protokube",pid=3504,fd=8))
     skmem:(r0,rb12582912,t0,tb12582912,f0,w0,o0,bl0,d0)
TIME-WAIT  0      0      100.96.18.1:12992                100.96.18.105:3007               
  
TIME-WAIT  0      0      100.96.18.1:48198                100.96.18.110:3006               
  
TIME-WAIT  0      0      100.96.18.1:63502                100.96.18.102:8001               
  
ESTAB      0      0      127.0.0.1:10249                127.0.0.1:53868                 users:(("kube-proxy",pid=3250,fd=5))
     skmem:(r0,rb12582912,t0,tb12582912,f4096,w0,o0,bl0,d0)
TIME-WAIT  0      0      100.96.18.1:58032                100.96.18.101:3000               
  
TIME-WAIT  0      0      100.96.18.1:17158                100.96.18.104:8000               
  
ESTAB      0      0      172.18.25.47:38474                172.18.18.165:https                 users:(("kubelet",pid=2607,fd=38))
     skmem:(r0,rb12582912,t0,tb12582912,f0,w0,o0,bl0,d112)
TIME-WAIT  0      0      100.96.18.1:17308                100.96.18.104:8000               
  
ESTAB      0      0      127.0.0.1:32888                127.0.0.1:10255                 users:(("go.d.plugin",pid=21570,fd=8))
     skmem:(r0,rb12582912,t0,tb12582912,f20480,w0,o0,bl0,d3)
TIME-WAIT  0      0      100.96.18.1:57738                100.96.18.101:3000               
  
TIME-WAIT  0      0      100.96.18.1:23650                100.96.18.97:3004               
  
TIME-WAIT  0      0      100.96.18.1:34518                100.96.18.103:3001               
  
ESTAB      0      0      127.0.0.1:53868                127.0.0.1:10249                 users:(("go.d.plugin",pid=21570,fd=6))
     skmem:(r0,rb12582912,t0,tb12582912,f8192,w0,o0,bl0,d1)
TIME-WAIT  0      0      100.96.18.1:23000                100.96.18.98:3002               
  
ESTAB      0      0      172.18.25.47:38498                172.18.18.165:https                 users:(("kube-proxy",pid=3250,fd=7))
     skmem:(r0,rb12582912,t0,tb12582912,f8192,w0,o0,bl0,d0)
TIME-WAIT  0      0      100.96.18.1:26430                100.96.18.100:3005               
  
TIME-WAIT  0      0      100.96.18.1:34882                100.96.18.103:3001               
  
ESTAB      0      0      172.18.25.47:3999                 172.18.44.34:57033                 users:(("protokube",pid=3504,fd=14))
     skmem:(r0,rb12582912,t0,tb12582912,f0,w0,o0,bl0,d0)
ESTAB      0      0      172.18.25.47:3999                 172.18.25.148:60423                 users:(("protokube",pid=3504,fd=18))
     skmem:(r0,rb12582912,t0,tb12582912,f0,w0,o0,bl0,d0)
ESTAB      0      0      172.18.25.47:61568                35.196.244.138:https                 users:(("netdata",pid=21419,fd=70))
     skmem:(r0,rb12582912,t0,tb262176,f0,w0,o0,bl0,d0)
TIME-WAIT  0      0      100.96.18.1:13154                100.96.18.105:3007               
  
ESTAB      0      0      172.18.25.47:54289                172.18.30.39:3999                  users:(("protokube",pid=3504,fd=12))
     skmem:(r0,rb12582912,t0,tb12582912,f4096,w0,o0,bl0,d15392)
TIME-WAIT  0      0      100.96.18.1:34718                100.96.18.108:3008               
  
TIME-WAIT  0      0      100.96.18.1:24078                100.96.18.97:3004               
  
LISTEN     0      128       :::ssh                     :::*                     users:(("sshd",pid=559,fd=4))
     skmem:(r0,rb12582912,t0,tb12582912,f0,w0,o0,bl0,d0)
LISTEN     0      4096      :::19999                   :::*                     users:(("netdata",pid=21419,fd=5))
     skmem:(r0,rb33554432,t0,tb12582912,f0,w0,o0,bl0,d0)
LISTEN     0      32768     :::4000                    :::*                     users:(("protokube",pid=3504,fd=5))
     skmem:(r0,rb12582912,t0,tb12582912,f0,w0,o0,bl0,d0)
LISTEN     0      32768     :::32003                   :::*                     users:(("kube-proxy",pid=3250,fd=13))
     skmem:(r0,rb12582912,t0,tb12582912,f0,w0,o0,bl0,d0)
LISTEN     0      32768     :::31719                   :::*                     users:(("kube-proxy",pid=3250,fd=12))
     skmem:(r0,rb12582912,t0,tb12582912,f0,w0,o0,bl0,d0)
LISTEN     0      32768     :::10250                   :::*                     users:(("kubelet",pid=2607,fd=24))
     skmem:(r0,rb12582912,t0,tb12582912,f0,w0,o0,bl0,d23)
LISTEN     0      32768     :::9100                    :::*                     users:(("node_exporter",pid=11027,fd=3))
     skmem:(r0,rb12582912,t0,tb12582912,f0,w0,o0,bl0,d0)
LISTEN     0      32768     :::31532                   :::*                     users:(("kube-proxy",pid=3250,fd=11))
     skmem:(r0,rb12582912,t0,tb12582912,f0,w0,o0,bl0,d0)
LISTEN     0      32768     :::30892                   :::*                     users:(("kube-proxy",pid=3250,fd=9))
     skmem:(r0,rb12582912,t0,tb12582912,f0,w0,o0,bl0,d0)
LISTEN     0      32768     :::10255                   :::*                     users:(("kubelet",pid=2607,fd=26))
     skmem:(r0,rb12582912,t0,tb12582912,f0,w0,o0,bl0,d0)
LISTEN     0      128       :::sunrpc                  :::*                     users:(("rpcbind",pid=232,fd=11))
     skmem:(r0,rb12582912,t0,tb12582912,f0,w0,o0,bl0,d0)
LISTEN     0      32768     :::10256                   :::*                     users:(("kube-proxy",pid=3250,fd=8))
     skmem:(r0,rb12582912,t0,tb12582912,f0,w0,o0,bl0,d0)
TIME-WAIT  0      0        ::ffff:172.18.25.47:9100                   ::ffff:172.18.25.47:13492              
  
ESTAB      0      0        ::ffff:172.18.25.47:10250                 ::ffff:172.18.25.148:55670                 users:(("kubelet",pid=2607,fd=40))
     skmem:(r0,rb12582912,t0,tb12582912,f0,w0,o0,bl0,d15400)
TIME-WAIT  0      0        ::ffff:172.18.25.47:9100                   ::ffff:172.18.25.47:13096              
  
TIME-WAIT  0      0        ::ffff:172.18.25.47:9100                   ::ffff:172.18.25.47:13384              
  
ESTAB      0      0        ::ffff:172.18.25.47:10250                  ::ffff:172.18.44.34:49454                 users:(("kubelet",pid=2607,fd=59))
     skmem:(r0,rb12582912,t0,tb12582912,f0,w0,o0,bl0,d7698)
TIME-WAIT  0      0        ::ffff:172.18.25.47:9100                   ::ffff:172.18.25.47:13200              
  
TIME-WAIT  0      0        ::ffff:172.18.25.47:9100                   ::ffff:172.18.25.47:13502              
  
TIME-WAIT  0      0        ::ffff:172.18.25.47:4000                  ::ffff:172.18.63.198:25438              
  
TIME-WAIT  0      0        ::ffff:172.18.25.47:9100                   ::ffff:172.18.25.47:13586              
  
TIME-WAIT  0      0        ::ffff:172.18.25.47:9100                   ::ffff:172.18.25.47:13298              
  
ESTAB      0      0        ::ffff:172.18.25.47:9100                  ::ffff:172.18.25.148:45776                 users:(("node_exporter",pid=11027,fd=7))
     skmem:(r0,rb12582912,t0,tb12582912,f0,w0,o0,bl0,d15419)
TIME-WAIT  0      0        ::ffff:172.18.25.47:9100                   ::ffff:172.18.25.47:13292              
  
ESTAB      0      0       ::ffff:127.0.0.1:10255                 ::ffff:127.0.0.1:32888                 users:(("kubelet",pid=2607,fd=5))
     skmem:(r0,rb12582912,t0,tb12582912,f4096,w0,o0,bl0,d0)
TIME-WAIT  0      0        ::ffff:172.18.25.47:9100                   ::ffff:172.18.25.47:13206              
  
ESTAB      0      0        ::ffff:172.18.25.47:10250                 ::ffff:172.18.18.165:33482                 users:(("kubelet",pid=2607,fd=32))
     skmem:(r0,rb12582912,t0,tb12582912,f0,w0,o0,bl0,d7707)
TIME-WAIT  0      0        ::ffff:172.18.25.47:4000                   ::ffff:172.18.30.39:45200              
  
TIME-WAIT  0      0        ::ffff:172.18.25.47:9100                   ::ffff:172.18.25.47:13594              
  
TIME-WAIT  0      0        ::ffff:172.18.25.47:9100                   ::ffff:172.18.25.47:13390              
  
TIME-WAIT  0      0        ::ffff:172.18.25.47:9100                   ::ffff:172.18.25.47:13090              
  
ESTAB      0      0        ::ffff:172.18.25.47:10250                 ::ffff:172.18.25.148:55590                 users:(("kubelet",pid=2607,fd=41))
     skmem:(r0,rb12582912,t0,tb12582912,f0,w0,o0,bl0,d15418)
ESTAB      0      0        ::ffff:172.18.25.47:10250                 ::ffff:172.18.25.148:55536                 users:(("kubelet",pid=2607,fd=11))
     skmem:(r0,rb12582912,t0,tb12582912,f0,w0,o0,bl0,d15401)
ESTAB      0      0        ::ffff:172.18.25.47:10250                 ::ffff:172.18.25.148:55762                 users:(("kubelet",pid=2607,fd=43))
     skmem:(r0,rb12582912,t0,tb12582912,f0,w0,o0,bl0,d15407)


According to ss manual:

Code:
              skmem:(r<rmem_alloc>,rb<rcv_buf>,t<wmem_alloc>,tb<snd_buf>,
                            f<fwd_alloc>,w<wmem_queued>,o<opt_mem>,
                            bl<back_log>,d<sock_drop>)

              <rmem_alloc>
                     the memory allocated for receiving packet

              <rcv_buf>
                     the total memory can be allocated for receiving packet

              <wmem_alloc>
                     the memory used for sending packet (which has  been  sent
                     to layer 3)

              <snd_buf>
                     the total memory can be allocated for sending packet

              <fwd_alloc>
                     the memory allocated by the socket as cache, but not used
                     for receiving/sending  packet  yet.  If  need  memory  to
                     send/receive  packet,  the  memory  in this cache will be
                     used before allocate additional memory.

              <wmem_queued>
                     The memory allocated for sending packet  (which  has  not
                     been sent to layer 3)

              <ropt_mem>
                     The  memory used for storing socket option, e.g., the key
                     for TCP MD5 signature

              <back_log>
                     The memory used for the sk backlog queue.  On  a  process
                     context,  if  the  process is receiving packet, and a new
                     packet is received, it will be put into  the  sk  backlog
                     queue, so it can be received by the process immediately

              <sock_drop>
                     the  number  of packets dropped before they are de-multi‐
                     plexed into the socket



Adding all the skmem values except for the rb and tb (because they are the maximum amount that can be allocated) and d for dropped packets, I should get a value pretty close to /proc/net/sockstat value. However the value I get is 53k which is nowhere near the 17804k.


Is my logic correct? So what am I missing here?
 
Last edited:


After a lot of searching around, I finally came to a conclusion.

My understanding of how to calculate TCP memory usage is correct.

For every socket add
Code:
socket_memory = rmem_alloc + wmem_alloc + fwd_alloc + wmem_queued + opt_mem + back_log
(the r,t, f, w, bl, o fields in skmem)

The reason that my total captured sockets memory above does not add up, is that a lot of the connections are running inside docker containers which are not displayed in the main system ss output, however they are displayed in kernel output of /proc/net/sockstat.

More info in this helpful stackoverflow question: https://stackoverflow.com/questions...stablished-connections-dont-appear-in-netstat

That explains the difference.

For a host-only-running processes situation the memory summed would be a match.
 

Members online


Top