Need Help Resolving ?Asymmetric Routing? Issue in a Network with pfSense and Netgear Managed Switch (GS724Tv4)
-
@oliverus000 still not seeing any SA blocks.. I see a FA there, which could close the states..
If your flow was asymmetrical, your SA (syn,ack) would be seen on pfsense on the wrong interface for where the state was created.. Not saying you don't have asymmetrical flow.. But I do not see a smoking gun that shows that..
if your flow was truly asymmetrical you wouldn't be able to actually make a connection, because the syn,ack wouldn't be allowed because of lack of state.
Its quite possible your states are just being reset.. I believe out of the box pfsense will reset states on a wan IP change, or loss of wan, etc.
NIC2 and NIC3 are bridged in pfSense
You have a bridge setup in pfsense? Why? Or do you have a "bridge" in your VM software to the physical nic on the box?
-
@johnpoz
I really wanted to dig deeper on the states situation and tried to look at the state in pfsense but when I look for the state that has my destination 10.76.100.221 included the result shows no matches (while I had my browser running the usecase described above):
Why is there no State at all? Am I missing out on something? A state should be there until it really gets reset, right?
To the bridge topic: pfsense runs as a VM on proxmox and therefore I need to bridge the NICs to make them available to the VM.
-
@oliverus000 if you see no states for an IP that is part of a active conversation that is working, then that screams the traffic is not going through pfsense at all.. Not possible for traffic to be flowing through pfsense without a state..
So either the state just got flushed and you haven't noticed that conversation isn't working - or your traffic is not flowing through pfsense like you want it too..
If I was to guess, your issue would be something related to your VM and bridging setup where you don't actually have stuff isolated like you think you do.
-
@johnpoz
I tried it now intensively and I could reproduce the behaviour: When I open a pure novnc window shell, nothing else in the background it creates a couple of states (different ports to the same ip, dont ask me why proxmox or novnc does that)
Then suddenly when something bad happens - ALL of them are gone and the table is empty
Any chance to find out why out of a sudden all states are getting flushed, even the closed ones??
-
@oliverus000 is your wan IP changing?
There is this setting..
Under advanced / misc
Then there is this setting on specific gateway under routing
I would look in your logs - do you see anything around the time you see the states go away?
But yeah if your states go away, that would for sure explain why your seeing the blocks in your firewall.
But also keep in mind, closed states will drop off the states list after the specific timing..
-
@johnpoz
These two settings are set to do not kill states...To the logs:
I cleared all logs before running my test. Performed the test and checked ALL logs:
System/General: nothing
System/Gateway: nothing
System/Routing: nothing
System/GUI Service: nothing specialFirewall: the same as already posted, a lot of blocks
DHCP: nothing special
Rest is empty.
:-(
-
@oliverus000
If you are running L3 switching then look at your gateways.
If you are not running L3 switching then it is not asymmetric routing as routing is layer 3.
By the way I am doing asymmetrical routing and it works on my current setup. I use Cisco for my L3 switching. -
@coxhaus
Yes my Switch is running on L3. Can you elaborate a little bit more what you mean by Gateways? As of now i have not specifically assigned any gateway information on the switch itself. I have only created the VLANs and the port assignments (PVID and Untagged/Tagged) on the switch itself. I have not touched any routing configuration on the switch since this should be handled via pfsense. The connected clients get the gateway info from DHCP which tells all the clients to use the specified VLAN gateway x.x.100.1 and x.x.28.1@johnpoz I have checked the proxmox network and bridge config and I am clueless what can be improved :-(
-
@coxhaus said in Need Help Resolving ?Asymmetric Routing? Issue in a Network with pfSense and Netgear Managed Switch (GS724Tv4):
By the way I am doing asymmetrical routing and it works on my current setup
Why would anyone do asymmetrical routing on purpose? Please explain..
I have done it when there is no other way, you can do host routing to work around it.. But why would anyone design a network to be asymmetrical? My answer to that would be your doing it wrong.
My switch is in L3 mode, but I am currently not routing anything on the switch, but I could if I wanted to - but routing on the switch does not mean your doing asymmetrical routing.. You would use a transit network
A switch with a trunkport and then ports in access mode doesn't say asymmetrical routing - Do you have svi set on the switch for these vlans and then pointing to them as gateways on the devices in these vlans vs the IPs on pfsense in those vlans?
-
@johnpoz said in Need Help Resolving ?Asymmetric Routing? Issue in a Network with pfSense and Netgear Managed Switch (GS724Tv4):
A switch with a trunkport and then ports in access mode doesn't say asymmetrical routing - Do you have svi set on the switch for these vlans and then pointing to them as gateways on the devices in these vlans vs the IPs on pfsense in those vlans?
If this question was for me here what I have setup (no routing for VLAN at all)
Routing in pfsense defined as follows:
Same for the other vlan10.
Firewall for 10 and 100, Pass all traffic:
DHCPs for 10 and 100:
-
@oliverus000 So you have no other IPs on the switch, other than its management IP, and your not pointing the gateway on any clients to these IPs on the switch..
If your not doing anything like that, then you wouldn't have asymmetrical routing.. You say the states just go away? That would be problematic..
Asymmetrical routing on a firewall causes issues when return traffic hits the firewall, but there is no state to allow the traffic..
This is a typical scenario where you would have asymmetrical traffic...
Client via some other router that has connection sends the syn to the destination.. But the device sending the syn,ack back to some other router.. When this router is a firewall as well.. Since it never saw the syn, it has no state to allow the return syn,ack - and would block this traffic.
Normally you would see this..
So when you send a syn, and the firewall allows it creates a state.. And sends the traffic on.. The syn,ack back is allowed by the state.. Now you have traffic flowing in both directs, just normal acks.. if the state goes away.. Traffic in either direction would be blocked.
Until a new state is created via a syn..
If in your blocks on your firewall you were seeing SA blocked, that would scream there is an asymmetrical flow that the firewall is not going to allow.. When you see just acks blocked, this points to just a removal of a state..
Either they just timed out because there was no traffic keeping them open, or they were deleted/killed. If devices are talking to each other an there is no traffic being sent, the state will timeout and close.. Now if one of the clients says hey I wasn't done talking here is some data and sends an ack, that ack will be blocked because there is no state.. Doesn't matter which end is sending the ack..
edit: once a handshake has been completed, ie the syn / syn,ack / ack - now all traffic between these devices wil have the ack flag on them..
If there is no existing state - this traffic will be blocked in either direction.. Just seeing blocks for Acks - where a connection was working before points to a loss of state.. You can see this with phones or wifi devices quite often where they will say wake up out of standby or something and try to continue a conversation they were using before.. But by this time the state has expired on the firewall, and is blocked..
edit
you can use pftop to see age of states, etc.. when they will expire, etc.. You can filter this for specific IPs, etc.
-
@johnpoz said in Need Help Resolving ?Asymmetric Routing? Issue in a Network with pfSense and Netgear Managed Switch (GS724Tv4):
You say the states just go away? That would be problematic..
I would love to record it and its a very weird behaviour, let me describe EXACTLY what happens:
-
I wait until there are no states available any more for any connection to the server x.x.100.221 on pfsense
-
I refresh the window with a connection to x.x.100.221 which has a shell opened to the server via novnc.
-
I have around 20 new states on different ports:
-
I type in stuff in my shell (really just interacting with the server, nothing fancy, just typing in text or even not doing anything, just looking at the shell and then out of nowhere BAM:
-
ALL STATES ARE GONE just at the time when I got kicked out of my connection to the shell. ALL OF THEM
Now what comes to mind mind are two things:
- Is my pfsense detecting something and then flushes all the states and that is really disconnecting me (pfsense is the enemy)
- Is my connection somewhere breaking because something is bad and that leads to the flush of all states. (proxmox is doing some stuipid novnc stuff that pfsense does not like)
The reason why I cant let it go is because my IT head is not liking the fact that this could also happen to any other connection from VLAN10 to VLAN100 (not only me using a novnc shell)
WHY is pfsense flushing all states without telling me the reason? I cant imagine this is happening because they are all expired at the same time, especially when I have a window open connecting to the shell via novnc?
What i did now is a hping3 -S 10.76.100.221 -p 80 -c 1000 from a client in 10.76.28.x which should send TCP-SYN packages to port 80.
I have a packages loss of 10%
Is this related???
-
-
@oliverus000 said in Need Help Resolving ?Asymmetric Routing? Issue in a Network with pfSense and Netgear Managed Switch (GS724Tv4):
I have a packages loss of 10%
I wouldn't expect there to be any packet loss on something your just talking to locally - 10% is quite a lot.. Does it come in a bunch, ie see a bunch of loss and then its all back to normal - or is it a packet here, packet there out of 1000 for example.. That adds up to 10%
How are you determining that you have 10% packet loss? (edit: oh I see) Is that in clumps all together now and then or just random here or there..
If all of the states you see are in closing or closed - then yeah I would expect them to all go away at like the same time.. But if your saying your loosing all states, even active ones - that points to something flushing the state table..
But if your sending data, and getting an answer the state should be active - unless you are not flowing traffic through pfsense??
Those states you show - don't show any response they are all just one sided.. 8/0 etc... that is not what a normal active conversation would look like..
ESTABLISHED:ESTABLISHED
And you should see packets on both sides of the / like
-
@johnpoz
I changed from wifi to a cable and paket loss reduced to almost 0%. So most probably not really connected to my issue.BUT your comment most probably leads to something.... You are absolutely right. There are only one sided states and it never shows "established" when I am connected with a browser to my server... WHAT could this mean???
I only see something like this but this looks also very one sided:
-
@oliverus000 and your answer is not going back through pfsense.
So in my above example if client A talking B sends its syn through pfsense it will open a state if the firewall rules allow the traffic. But if the answers do not flow back through pfsense then the would never be an established connection.. And even if you continue to send traffic from A through pfsense.. At some point this state will close, and now traffic from A to B would be blocked..
So this points to symmetrical flow - but in the other direction.. So you could have something like this..
pfsense will open the state and send your traffic on - but since it never sees any return traffic.. At some point these states will expire.. And now your sender sending traffic will be blocked until he sends a new syn to open up a new state.
This some examples of why asymmetrical flow is almost never a good idea.. That @coxhaus mentions he is doing it - on purpose?? That is horrible design.. And can be very problematic - especially when you have a stateful firewall doing the routing..
You can see this sort of issue with multi homed devices.. As well
So for example my client on 192.168.1.x sends traffic to 192.168.2.x through pfsense.. But the device on 192.168.2 also has a connection in the 192.168.1 network and answers via this path then at some point pfsense will kill off the states.. And further traffic will be blocked until a new syn opens a new state..
Asymmetrical flow, mult-homed devices is just asking for problematic issues.. They should almost always be avoided..
Now you would hope that the client sending the traffic would be smart enough to figure out, hey I sent to 192.168.2.x via my gateway mac of xyz... Why is the response coming from 192.168.1.Y from mac abc.. Because such a response could be of security concern.. But many clients are stupid, and will just accept the answer.. Hey I sent to 192.168.2.x from port 4000 to port 443.. And the response even though from different IP and different mac address is to my port 4000 from a port 443..
Is this device your talking to multihomed? Ie does it have an IP in both networks?
If your going to talk to a device that has interfaces in network A and B from a device in network A.. You should talk to the device IP in network A.. Not B - if you talk to its B address, you are yes most likely going to have issues..
Multihoming can be very problematic.. And also a security concern.. Because your firewall has no control over this device talking to other devices in other networks - because it has a leg in multiple networks. And this can be used to circumvent firewall controls of what can talk to what.
-
@johnpoz
AND it was exactly what you have pointed out. One of my proxmox instances in the LAN had 2 ip addresses assigned to the host setup, one in VLAN10 and one in VLAN100 which basically let to your described behavior.Thanks a lot for taking so much time and reading through all of these messages and screenshot. I have learned a lot and I am so thankful that people like you exist. Please keep up the great work you are doing.
-
@oliverus000 glad you got sorted, and I could help you figure it out.
edit
If you need/want to put vms on multiple networks, that is fine.. And sure ok the Host is actually connected to multiple networks. But the host itself doesn't really need an IP in these multiple networks.. Just the VMs do.. I do same sort of thing.. I have multiple vlans trunked to my VM host via an interface.. So I can put a VM on network A or B or C even.. But to manage the host, to talk to the host to do HOST things, the host only has an IP in network A for example..All of these networks are isolated at layer 2.. An no VM actually has interfaces in more than one network. So there is no concern that when talking to a VM that is on network B from a device in network A, would the vm be able to answer back via network A.
Now could this be a security issue.. Yeah it could be, if the HOST itself was compromised.. It does have interface that could be used in multiple networks.. But in day to day operation this is not a issue. And this host is not exposed to the public internet or anything.. The only way to talk to this host management IP is via being on the management network.
-
J johnpoz referenced this topic on
-
@johnpoz
I only have 1 LAN port setup on pfsense with no DHCP. All VLANs are defined in my Cisco L3 switch and use DHCP off the Cisco L3 switch. The gateway for the shared VLAN connecting to my Pfsense is defined as a VLAN for my Cisco L3 switch and defined only as a LAN port, no VLANs on Pfsense. I use the gateway on the shared VLAN pointing for the clients as the L3 switch and the gateway for Pfsense, no VLAN defined, pointing to my WAN port. The default routing on the L3 switch is pointing to the Pfsense LAN port so unknown IPs are routed to Pfsense. This works fine.It is easier to setup. On my last setup I had a point-to-point network to transfer from my L3 switch to Pfsense. At home I see no difference.
-
@coxhaus yeah that is a transit network, if you have not hosts on it.. That has zero anything to do with this thread.
Why not just put everything in 1 vlans to be honest if your not firewalling between your vlans.. Other than different broadcast domain there is little advantage to that sort of setup.. Do you have any acls setup between these vlans? if your not filtering traffic between the vlans.. Why not just put them all on 1.. Your making your life more complicated for no actual benefit of being able to easy firewall between your segments.
-
@johnpoz
I do, I use ACLs for my guest VLAN on my L3 switch. I have my printer defined on my guest VLAN and share a 29-bit mask so my guests can print and my main LAN. I also run3 Cisco wireless APs since I have a large home with 12-foot ceiling. I have separate VLANs with separate SSID setup for roaming for guest and my main LAN. Cisco 150ax wireless APs setup as 1 virtual wireless AP all grouped together.Using the firewall on my Pfsense router would be slow compared to my line speed Cisco L3 switch. Plus, that would mean Pfsense is doing the routing not my switch. If you use a trunk port you are not doing L3 switching, you are doing L2 switching.
PS
I had a server rack on a separate VLAN, but when I retired I turned off my rack.