PfSense behind NAT, wrong output



  • Hi,

    I tried to setup pfSense this morning but I still have a few problems (related to the way my network is organised), I know there (already) are many posts about that but I couldn't make my setup work.
    First of all, I installed pfSense inside a VM, connected to my hypervisor on a virtual bridge, this hypervisor has a single public IP address.

    I created a subnet between the hypervisor and pfSense: 10.0.3.1/24, pfSense is at 10.0.3.2 (WAN side), in order for this to worked, I added two iptables rules on the hypervisor:

    *filter
    :INPUT ACCEPT [302861686:19938597916]
    :FORWARD ACCEPT [119629620:95546130061]
    :OUTPUT ACCEPT [103744971:1152060249475]
    COMMIT
    *nat
    :PREROUTING ACCEPT [3852:325870]
    :INPUT ACCEPT [377:40959]
    :OUTPUT ACCEPT [1087:66380]
    :POSTROUTING ACCEPT [1691:100596]
    -A PREROUTING -p tcp -m multiport ! --dports 22,8000 -j DNAT --to-destination 10.0.3.2
    -A POSTROUTING -s 10.0.3.0/24 -o vmbr0 -j MASQUERADE
    COMMIT
    

    The goal is to redirect all ports (tcp currently, but it will be similar with udp) except 22 (hypervisor ssh server) and 8000 (hypervisor http server) to pfSense.

    The LAN interface of pfSense is vtnet1 with the network 10.0.2.1/24, and I want to forward the port number 1197 to 10.0.2.100:22 hence I added a NAT rules on WAN:

    WAN  TCP  *  *  WAN ADDRESS  1197  10.0.2.100  22
    

    Basically, this will redirect all connection to the wan address (10.0.3.2:1197) to 10.0.2.100:22, pfSense automaticaly added a firewall rule on WAN:

    IPV4 TCP *  *  10.0.2.100  22  *  None
    

    Here, I'm pretty sure this rule is incorrect, I don't see how 10.0.2.100 could be access outside the LAN part of pfSense, I kept this rules, but I added the same one with This firewall as destination
    instead of the LAN IP.

    Everything should be fine now, but this is not the case, I cannot ssh into 10.0.2.100 using ssh username@publicIP -p 1197
    The next step was to analyse the traffic on each end of the network, for the next step I will call public the public interface of the hypervisor, hyper the private interface of the hypervisor (10.0.3.1/24) connected to the WAN of pfSense WAN and LAN for the private network of pfSense.

    I ran the following command on public: tcpdump -i eth0 port 1197 (watching connection on port 1197 of the public IP), and here is the output during a SSH session:

    15:02:42.228011 IP tsf-444-wpa-0-215.myISP.ch.59056 > publicIP.1197: Flags [s], seq 3539741134, win 65535, options [mss 1250,sackOK,eol], length 0
    15:02:42.228320 IP 10.0.3.2.1197 > tsf-444-wpa-0-215.myISP.59056: Flags [S.E], seq 3891171878, ack 3539741135, win 28960, options [mss 1460,sackOK,TS val 1876163144 ecr 1058968724,nop,wscale 7], length 0
    
    [size][b]There seems to be an issue here, why is the source IP of the response `10.0.3.2.1197` ??? (this is the IP address of pfSense WAN)[/b][/size]
    
    Let's continue to inspect the other interfaces to see what happens:
    
    Running the same command, but this time on the `vmbr2` interface of the hypervisor (`hyper`) : [i]tcpdump -i vmbr2 port 1197 
    [/i]
    The source port is not the same as I logged the network on multiple connection to write this post
    
    [code]
    15:05:48.997784 IP tsf-444-wpa-0-215.myISP.ch.59091 > 10.0.3.2.1197: Flags [s], seq 719565286, win 65535, options [mss 1250,nop,wscale 5,nop,nop,TS val 1059158749 ecr 0,sackOK,eol], length 0
    15:05:49.511199 IP 10.0.3.2.1197 > tsf-444-wpa-0-215.myISP.ch.59091: Flags [S.], seq 2676845414, ack 719565287, win 28960, options [mss 1460,sackOK,TS val 1876209965 ecr 1059157457,nop,wscale 7], length 0
    
    The first line is the natted connection to pfSense WAN interface, and the second one is the answer from pfSense to the hypervisor, everything seems to be normal here....
    
    Let's continue, I now run [i]tcpdump -i vtnet0 port 1197[/i] on the WAN interface of pfSense:
    
    [code]
    14:09:07.167193 IP tsf-444-wpa-0-215.myISP.ch.59102 > 10.0.3.2.1197: Flags [s], seq 2594385049, win 65535, options [mss 1250,nop,wscale 5,nop,nop,TS val 1059356101 ecr 0,sackOK,eol], length 0
    14:09:07.683779 IP 10.0.3.2.1197 > tsf-444-wpa-0-215.myISP.ch.59102: Flags [S.E], seq 2535098774, ack 2594385050, win 28960, options [mss 1460,sackOK,TS val 1876259508 ecr 1059354846,nop,wscale 7], length 0
    
    Nothing special here, this is the same as the previous dump
    
    Let's see what happens on the LAN interface tcpdump -i vtnet1 port 1197
    
    [code]
    Nothing, blank, empty...
    [/code]
    
    This is normal, the nat should transform request from port 1197 to 22, [i]tcpdump -i vtnet1 port 22[/i]
    
    [code]14:12:42.888440 IP tsf-444-wpa-0-215.myISP.ch.59138 > 10.0.2.100.ssh: Flags [s], seq 3683235951, win 65535, options [mss 1250,nop,wscale 5,nop,nop,TS val 1059570705 ecr 0,sackOK,eol], length 0
    14:12:43.403714 IP 10.0.2.100.ssh > tsf-444-wpa-0-215.myISP.ch.59138: Flags [S.], seq 1555856048, ack 3683235952, win 28960, options [mss 1460,sackOK,TS val 1876313438 ecr 1059569460,nop,wscale 7], length 0
    
    Nothing strange here, we have the connection in both way, the last machin we can check is the machine hosting the ssh server: `10.0.2.100`
    
    tcpdump -i ens18 port 22
    
    [code]15:14:59.260847 IP tsf-444-wpa-0-215.myISP.ch.59173 > 10.0.2.100.ssh: Flags [s], seq 1462488176, win 65535, options [mss 1250,nop,wscale 5,nop,nop,TS val 1059705314 ecr 0,sackOK,eol], length 0
    15:14:59.260867 IP 10.0.2.100.ssh > tsf-444-wpa-0-215.myISP.ch.59173: Flags [S.], seq 906627863, ack 1462488177, win 28960, options [mss 1460,sackOK,TS val 1876347402 ecr 1059704506,nop,wscale 7], length 0
    
    Once again, no issue here (at least no obvious one), the connection between all the component seems to be ok, but the client timeout, I'm pretty sure this is because of the 'double natting' and the pfSense WAN ip that is not rewrote by the hypervisor.
    
    Could you guys help me please !
    
    Thank you[/s][/code][/s][/code][/s][/code][/s][/code][/s]
    

Log in to reply