Linux Server sends TCP RST

KapK

New Member
Joined
Jul 21, 2021
Messages
10
Reaction score
4
Credits
130
Hi All,

***This is my first post in this forum. If there are any mistakes , beg your pardon in advance. Let's go to the issue.***

I am maintaining a server (one out of many) with the IP 10.10.13.26 and renders the service via port 7781. All my servers are located behind a firewall and I allow access for external servers (yet inside our company network ) from this FW. One such node is 10.12.13.215 ( which access my 10.10.13.26 via 7781).

The IP 10.10.13.26 is the physical IP of one node in two nodes cluster (This is the master node). The IP of the other node is 10.10.13.27 and the virtual IP is 10.10.13.25

Currently, a migration is happening in this department and they are moving their nodes to a cloud environment ( 10.30.31.0/24) located behind a firewall which does NAT (to IP 10.250.200.12 ).

The issue is that the new nodes cannot access the service on my 10.10.13.26 via port 7781. (Actually they cannot access none of the cluster )
  • I have allowed the NAT IP 10.250.200.12 in my FW
  • They can ping 10.10.13.26
  • They can access the other nodes in my side and the services rendered by them as well
I used tcpdump to trace the flow from both 10.12.13.215 (existing) and 10.250.200.12 (proposed) and found that my server is sending TCP RST for the proposed node. (A network diagram , screenshots of both traces are attached.)

What could be the reason ?

The whole migration process is with held due to this and need to solve this ASAP and I am in desperate situation. TIA.
 

Attachments

  • Network.jpg
    Network.jpg
    97 KB · Views: 332
  • from Inside.jpg
    from Inside.jpg
    286.2 KB · Views: 379
  • From Cloud.jpg
    From Cloud.jpg
    278.9 KB · Views: 384


I haven't gotta clue, but I did find a bunch of reasons why it may be sending a RST.

 
It's not a good comparison that traffic from 10.12.13.215 does reach 10.0.13.25 on 7781 and that traffic from being NATed coming from 10.250.200.12 to 10.10.13.26 on 7781 doesn't arrive because in the first situation no NATing is being done and the section situation there is.
  • They can access the other nodes in my side and the services rendered by them as well
With other nodes do you mean the traffic coming from 10.13.31.0/24 being NATed to 10.250.200.12 being able to connect to 10.12.13.215 7781? So my first conclusion from the situation you describe is that there is a configuration error in FW2.

Lastly the "Cloud/Proposed Network", the "Existing Network" and "Your side of the Network" are all using private ip ranges. Why are you using NAT for communicating with different private ip ranges? source NATing is used so that private ip's can be translated to a public ip so that they can access the public ip ranges or when a server has a private ip and needs to be accessed by systems that have a public ip then destination NATing is used.
In conclusion NATing is not needed in the situation you sketched, remove the NAT and just have normal firewall rules for access to the different networks so that the situation of traffic coming from the "Existing Network" and the "Cloud/Proposed Network" are the same. If that's not an option then I stick my conclusion that it's a configuration error in FW2.
 
Last edited:
First, your connection from the existing 13.215 network is not relevant except to the point that it does actually work given that your issue is emanating from your 31.0/24 network.

When you run tcpdump (or wireshark or whatever) on the 10.10.13.26 server while trying to connect from your 10.13.31.0/24 network. What exactly are you seeing? You showed what the 31.0/24 was showing, but we need to see both sides. (or all four if you include FW1 and FW2 since they are both are also in the equation) We can see the 31.0/24 is sending something, but not receiving a response. You said you can ping it, but is the ping responding from the server or FW1? My firewalls do not pass pings through the firewall. You likely are seeing a response for FW1 and not the server that is behind FW1. Well, unless you are physically allowing that to occur. Of course, that begs the question, Is FW1 in firewall mode or acting solely as a router without performing any firewalling services? What is FW1's IP address? It's not performing NATing?

I suspect f33dm3bits is correct in that your FW2 is likely misconfigured. I don't know the requirements of your cloud network as it pertains to NATing, so I cannot say if NATing is required or not. That said, the act of NATing itself isn't a problem in itself, and hence why I think the FW2 NAT configuration is likely the issue. Perhaps if NAT transparency is enabled it's causing issues evoking asymmetrical routing of some sort.
 
Thanks and appreciate your responses, f33dm3bits & dcbrown73 !!!

By looking at your feedbacks, I think it would be better to give more details for better understanding so that I can get solution to my problem.
  • My side network (10.10.13.0/24) and FW1 is handled by me
  • The existing network and the proposed network (both) trying to access my network (especially 10.10.13.26)
    • The existing network is legacy network with physical servers.
    • The proposed network is provided by a third party with cloud service.
    • All the services running in the existing network will be migrated to the proposed network in near future.
    • They are doing Preliminary Acceptance Tests from the servers in the proposed network
    • One particular service need to access my server 10.10.13.26 port 7781 and currently it is unable to perform due to this connectivity issue.
  • The above mentioned two networks are not handled by me and have no control over them.
  • The FW2 is provided by the same company who provided the cloud service and I don't know why are they doing the NAT or the actual setup in that side.
    • I just receive traffic from 10.250.200.12
  • The other applications/services run in the proposed network can access other particular nodes in my side(eg 10.10.13.67 P5019 , 10.10.13.31 P22) just like their counterparts in the existing network
Well, I hope I have given a better picture about here. (I have made a small addition to the network diagram and it is attached here with.)

Now, I'll give my responses for you two in one feedback in order to maintain the continuity.

@f33dm3bits

With other nodes do you mean the traffic coming from 10.13.31.0/24 being NATed to 10.250.200.12 being able to connect to 10.12.13.215 7781? So my first conclusion from the situation you describe is that there is a configuration error in FW2.

The whole block 10.30.31.0/24 is being NATed to 10.250.200.12 . Some other services on that side are already accessing my other nodes (eg 10.10.13.67 port 5019, 10.10.13.31 port 22) successfully. Only 10.10.13.26 port 7781 is having this issue.
As mentioned in my original post, they can ping 10.10.13.26 from the proposed network.
No, they are not trying to access 10.12.13.215 because this node is in the existing network and soon it too will be migrated to 10.30.31.0/24 network

Lastly the "Cloud/Proposed Network", the "Existing Network" and "Your side of the Network" are all using private ip ranges. Why are you using NAT for communicating with different private ip ranges?

TRUE. Well as I mentioned previously, this Proposed Network is provided by a third party company and may be it was their suggestion or requested by our particular department to install a firewall (FW2) with NAT. I don't know and I'm helpless here. :|

In conclusion NATing is not needed in the situation you sketched, remove the NAT and just have normal firewall rules for access to the different networks so that the situation of traffic coming from the "Existing Network" and the "Cloud/Proposed Network" are the same.

So now you can understand that I cannot remove the NAT service in FW2.

If that's not an option then I stick my conclusion that it's a configuration error in FW2.

My knowledge is low here. How is it possible ? My server 10.10.13.26 receives the requests from the proposed network and it sends back the TCP RST to 10.250.2.12 and only this server (10.10.13.26 port 7781) having this issue.
They can ping 10.10.13.26 and currently have no issue in accessing other nodes or the ports in my side (10.10.13.67 p5019, 10.10.13.31 p22)

Thanks again f33dm3bits for your insights. Pls go through my feedbacks.


@dcbrown73

When you run tcpdump (or wireshark or whatever) on the 10.10.13.26 server while trying to connect from your 10.13.31.0/24 network. What exactly are you seeing? You showed what the 31.0/24 was showing, but we need to see both sides.

I run the tcpdump with the -host 10.250.200.12 option. Do you mean that I should get the same with -host 10.10.13.26 ?
Well, I never thought of that. I will get that as well and feedback.

Well, unless you are physically allowing that to occur. Of course, that begs the question, Is FW1 in firewall mode or acting solely as a router without performing any firewalling services? What is FW1's IP address? It's not performing NATing?

Well, I have allowed ping for this in order to resolve this issue. FW1 IP is 10.10.13.254 and it DOES NOT perform NAT.

Thanks again dcbrown73 for your insights. Pls go through my feedbacks.
 

Attachments

  • Network_Flow.jpg
    Network_Flow.jpg
    119.3 KB · Views: 246
I haven't gotta clue, but I did find a bunch of reasons why it may be sending a RST.


@KGIII
Thanks KGIII, That was a great reference.
 
Thanks for going into more detail about your situation and that's why I mentioned in my first reply with the information you gave in your first post about NATing that from the information you gave it didn't seem necessary but now I understand why it is the way it is.

I currently think the best option is to run(or have someone else run) a tcpdump on FW2(10.250.200.12)while connecting from 10.13.31.0/24 to 10.10.13.26 7781. I would think you will get the information you need then because all the traffic is going through FW2, since I suspect there will likely be two possibilities.
1. It will verify that that the problem occurs on FW2.
2. It will verify that the problem doesn't occur on FW2.
If it verifies the first possibility then you will have to look into the configuration of FW2 and if it verifies the latter you will have cancelled out FW2 and then can continue the next step which is somewhere on the "Company Network" which could be another switch or router which is not mentioned in the diagram you shared.
 
Thanks again f33dm3bits! I will check. Unfortunately, will have to wait till tomorrow (Monday) to get the required details. ( I will try my best to get them today, but can't be sure).

I thought, since the TCP RST is sending from my server 10.10.13.26, there is a config issue in my Linux server. How can we conclude that 10.10.13.26 server side is ok ?

Thanks and Regards.
 
Yes I see that too but in your first capture where it shows 10.10.13.26 is sending a RST it's only one side, once you see what's happening on the other side you will have the full picture and have a better idea of what the problem could be and you will either be able to cancel something out and have some new information. I don't find it likely there is a configuration issue on your Linux server because from the "Existing Network" it's working or are you also using a local firewall on your server and/or and an acl that restricts what sources can access the application?
 
No. There are no other firewalls. Only FW1 is there.
Tks.
 
Hi @f33dm3bits !
No wonder you have the Gold membership with 21,745 credits. You were right. There was a config issue in FW2 side. Since it belongs to a different team, I don't know what was the exact reason (neither would they tell..) but after having a conf call with all teams, I got to know that there is a config issue.

WOW @f33dm3bits !!! you are a life saver. Seriously, I thought the issue was in my side because of the tcpdump traces. Thanks a lot. Words not enough to convey my gratitude.

Thanks @dcbrown73 and @KGIII for your responses as well.
 
Glad to have helped out and that you were able to solve your problem, "Gold membership" just means I support linux.org nothing more and you get credits per topic started and for each reply to a topic.
 
Last edited:

Staff online

Members online


Top