Right. Chances are the ISP was intermittently forwarding the traffic to you so you were therefore intermittently able to forward the traffic inward. Can only work on what you receive. Glad it looks like they found it.

@Jeonetgate Having this same issue as well on 2.4.4p3. No flush settings are enabled. Traffic is getting blocked by this default rule. ICMP traffic about 75% of it gets lost due to this. Do have two WANs configured.

I've switched over to AON NAT now, I didn't originally see all of the rules. Turned out that I didn't have an upstream gateway set. Once Set I see the rules automatically generated. I added my rule to disable NAT to create a routed network from my two servers.

I still see an issue however. Can anyone suggest a fix or maybe a better way to achieve what I'm trying to do? I'll try and detail what the issue is...

Screen Shot 2020-03-25 at 3.41.24 PM.png

So it seems like it's an L2 issue...

Here are the test details... Ping from Site1 Nested ESXi VM fails...

The path is: Request: "n-esx1" -> "p-esx1" -> "p-esx2" -> "n-esx7" Reply: "n-esx7" -> "p-esx2" -> dfGW.... where I need it to go back to "p-esx1"
n=Nested
p=Physical

[root@stevelab-n-esx1:/vmfs/volumes] ping 192.168.2.17
PING 192.168.2.17 (192.168.2.17): 56 data bytes

For reference...
Site1 WAN: 60:ac:a6
Site2 WAN: 7b:81:88
WAN GW: ff:fd:90

[root@stevelab-n-esx1:/vmfs/volumes] ping 192.168.2.17
PING 192.168.2.17 (192.168.2.17): 56 data bytes

Destination Nested ESXi sees the request and replies to the request.

[root@stevelab-n-esx7:/vmfs/volumes/3a3b5bc8-88bd4760] pktcap-uw --uplink vmnic0 --dir 2 --ip 192.168.1.11 -o - | tcpdump-uw -enr -
22:23:35.839598 00:0c:29:7b:81:9c > 00:0c:29:35:9b:0d, ethertype IPv4 (0x0800), length 98: 192.168.1.11 > 192.168.2.17: ICMP echo request, id 55660, seq 0, length 64
22:23:35.839758 00:0c:29:35:9b:0d > 00:0c:29:7b:81:9c, ethertype IPv4 (0x0800), length 98: 192.168.2.17 > 192.168.1.11: ICMP echo reply, id 55660, seq 0, length 64

Then the Physical host is sending the reply to the WAN GW. It doesn't send it back from where it came...

[root@stevelab-p-esx2:/vmfs/volumes] pktcap-uw --uplink vmnic0 --dir 2 --ip 192.168.1.11 -o - | tcpdump-uw -enr –
22:24:09.239994 00:0c:29:60:ac:a6 > 00:0c:29:7b:81:88, ethertype IPv4 (0x0800), length 98: 192.168.1.11 > 192.168.2.17: ICMP echo request, id 55660, seq 18, length 64
22:24:09.240589 00:0c:29:7b:81:88 > 00:08:e3:ff:fd:90, ethertype IPv4 (0x0800), length 98: 192.168.2.17 > 192.168.1.11: ICMP echo reply, id 55660, seq 18, length 64
22:24:10.241698 00:0c:29:60:ac:a6 > 00:0c:29:7b:81:88, ethertype IPv4 (0x0800), length 98: 192.168.1.11 > 192.168.2.17: ICMP echo request, id 55660, seq 19, length 64
22:24:10.242331 00:0c:29:7b:81:88 > 00:08:e3:ff:fd:90, ethertype IPv4 (0x0800), length 98: 192.168.2.17 > 192.168.1.11: ICMP echo reply, id 55660, seq 19, length 64

[root@stevelab-p-esx2:/vmfs/volumes] esxcli network ip neighbor list
Neighbor Mac Address Vmknic Expiry State Type
10.33.72.65 00:0c:29:60:ac:a6 vmk0 928 sec Unknown
10.33.72.64 00:0c:29:8c:15:73 vmk0 1196 sec Unknown
10.33.72.62 00:11:32:a6:9a:3f vmk0 1020 sec Unknown
10.33.75.253 00:08:e3:ff:fd:90 vmk0 1198 sec Unknown

Reply never arrives back at Site1 (Of course, because the packet went to the WAN GW of Site2.

[root@stevelab-p-esx1:~] pktcap-uw --uplink vmnic0 --dir 2 --ip 192.168.1.11 -o - | tcpdump-uw -enr -
22:24:24.270369 00:0c:29:60:ac:a6 > 00:0c:29:7b:81:88, ethertype IPv4 (0x0800), length 98: 192.168.1.11 > 192.168.2.17: ICMP echo request, id 55660, seq 33, length 64
22:24:25.271133 00:0c:29:60:ac:a6 > 00:0c:29:7b:81:88, ethertype IPv4 (0x0800), length 98: 192.168.1.11 > 192.168.2.17: ICMP echo request, id 55660, seq 34, length 64
22:24:26.271396 00:0c:29:60:ac:a6 > 00:0c:29:7b:81:88, ethertype IPv4 (0x0800), length 98: 192.168.1.11 > 192.168.2.17: ICMP echo request, id 55660, seq 35, length 64

Willing to try just about anything...

Thanks.

Well whatever was going on seems to be transient as things seem to be working now, although with the current situation extremely slow and laggy.

@Derelict Thanks bud, we just tested what you said and it worked beautifully. The process was a bit tricky to understand because they used Unifi on the opposite side but after letting go of preconceptions and just tried everything just popped into place.

ok, thanks for the help. i have it sorted. and Vlans worked PERFECTLY.

Cheers

Jason

Sorry for making you so upset. TBH, I thought the info above was OK, and you initially seem to grasp that, but obviously I have annoyed you for some reason, and for that I can only apologise

As mentioned, the issue was at the layer 2 level with ARP and the response for some reason not making it back to client when querying the gateway. This was a genuine query but I fear it has has deteriated.

Once again, thank your for you input.

Actually, that's default behavior of the XG-7100. Lagg0 is an aggregate of ix2 and ix3, interfaces internal to the chassis that provide trunking for the 8 ports on the front face.

lagg.png

@noplan

done the same again like the pen and paper check

took the same switch (old habits die hard)

same problem --> wtf

took a brand new switch

workin like a charm

checked the old switch .... some crazy folk just done some MAC ACL testing on some random ports
reset the old switch now workin like a charm
so SOLVED

This one on your proxmox - is this doing nat? What your doing is correct.

I would go through the troubleshooting doc.
https://docs.netgate.com/pfsense/en/latest/nat/port-forward-troubleshooting.html

What your doing is fine you can have multiple IPs sending to port 80 behind... I would validate that traffic is actually getting to pfsense wan, and then sending it on... This can be done with packet captures on pfsense, under the diag menu..

If I had to guess its your proxmox setup - firewall maybe on it? And access from other than your local network?

Did you setup the vip correctly? When you do a vip, it should be available via your dropdown when you do port portward..

example..

vip.jpg

And the mask should be what your network on your wan is using.. Do you have like a /29 or something? Where this address block is coming from?

Have a cocktail.

Done and Done Much appreciated