AWS lan to wan problem



  • I tried to read all forums posts of matter, and try all known solutions before posting, so I would not create unneccessary rtfm post. Unfortunately, I've been stuck couple of days implementing pfsense as NAT instance in AWS :P

    But I cannot get out from the private subnet (ping 8.8.8.8, apt-get update). I have outbound nat rule but things are odd. I have tried to disable all AWS security features for all hosts, so eveything is in allow everything security group.

    I created port forwarding rule from public ip/port 8000 to forward ssh to debian running in VPC private subnet. It works fine. I can see the traffic passed in tcpdump properly.

    I have very plain AWS VPC with CIDR 10.0.0.0/16

    public subnet (10.0.0.0/24) has routes
    10.0.0.0/16 local
    0.0.0.0/0 aws-provided internet-gateway

    private subnet (10.0.1.0/24) has routes
    10.0.0.0/16 local
    0.0.0.0/0 interface-id of LAN facing interface

    Public subnet has Pfsense 2.3.2 running with two interfaces, (10.0.0.20 public, 10.0.1.140 private) and elastic ip attached. One patch is installed to pfsense for chrome netblock problem fix.

    I can connect to elasticIP, and reach admin console and ssh to fw with no problem.

    AWS gives my private lan ip 10.0.1.140, pinging it is ok as tcpdump is concerned

    ping 10.0.1.1 results in lan:
    64 bytes from 10.0.1.1: icmp_seq=1 ttl=64 time=0.414 ms
    64 bytes from 10.0.1.1: icmp_seq=2 ttl=64 time=0.445 ms
    64 bytes from 10.0.1.1: icmp_seq=3 ttl=64 time=0.456 ms

    tcpdump -vv -i xn1 on pfsense

    23:37:41.796911 IP (tos 0x0, ttl 64, id 14383, offset 0, flags [none], proto ICMP (1), length 28)
        pfSense.localdomain > 10.0.1.1: ICMP echo request, id 56379, seq 48849, length 8
    23:37:41.797212 IP (tos 0x0, ttl 64, id 39768, offset 0, flags [none], proto ICMP (1), length 28)
        10.0.1.1 > pfSense.localdomain: ICMP echo reply, id 56379, seq 48849, length 8

    this is great - but then, wget www.google.com on client, and it's all quiet on tcpdump, cant get anything by filterin on source or so. Tried to put full logging on all filters, but can't get apt working inside LAN yet.

    interface configurations are as follows

    pflog0: flags=100 <promisc>metric 0 mtu 33160
    pfsync0: flags=0<> metric 0 mtu 1500
            syncpeer: 224.0.0.240 maxupd: 128 defer: on
            syncok: 1
    enc0: flags=0<> metric 0 mtu 1536
            nd6 options=21 <performnud,auto_linklocal>lo0: flags=8049 <up,loopback,running,multicast>metric 0 mtu 16384
            options=600003 <rxcsum,txcsum,rxcsum_ipv6,txcsum_ipv6>inet 127.0.0.1 netmask 0xff000000
            inet6 ::1 prefixlen 128
            inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4
            inet 172.19.0.1 netmask 0xffffff00
            nd6 options=21 <performnud,auto_linklocal>xn0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
            options=3 <rxcsum,txcsum>ether 02:4e:d2:8b:33:7f
            inet6 fe80::4e:d2ff:fe8b:337f%xn0 prefixlen 64 scopeid 0x5
            inet 10.0.0.20 netmask 0xffffff00 broadcast 10.0.0.255
            nd6 options=21 <performnud,auto_linklocal>media: Ethernet manual
            status: active
    xn1: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
            options=3 <rxcsum,txcsum>ether 02:e5:c2:30:2b:85
            inet6 fe80::e5:c2ff:fe30:2b85%xn1 prefixlen 64 scopeid 0x6
            inet 10.0.1.140 netmask 0xffffff00 broadcast 10.0.1.255
            nd6 options=21 <performnud,auto_linklocal>media: Ethernet manual
            status: active

    and the inside VPC debian has like this

    eth0      Link encap:Ethernet  HWaddr 02:75:9b:08:e5:2b
              inet addr:10.0.1.34  Bcast:10.0.1.255  Mask:255.255.255.0
              inet6 addr: fe80::75:9bff:fe08:e52b/64 Scope:Link
              UP BROADCAST RUNNING MULTICAST  MTU:9001  Metric:1
              RX packets:1949 errors:0 dropped:0 overruns:0 frame:0
              TX packets:2088 errors:0 dropped:0 overruns:0 carrier:0
              collisions:0 txqueuelen:1000
              RX bytes:128910 (125.8 KiB)  TX bytes:217848 (212.7 KiB)

    lo        Link encap:Local Loopback
              inet addr:127.0.0.1  Mask:255.0.0.0
              inet6 addr: ::1/128 Scope:Host
              UP LOOPBACK RUNNING  MTU:65536  Metric:1
              RX packets:0 errors:0 dropped:0 overruns:0 frame:0
              TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
              collisions:0 txqueuelen:0
              RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

    with routes as this

    root@ip-10-0-1-34:~# route
    Kernel IP routing table
    Destination    Gateway        Genmask        Flags Metric Ref    Use Iface
    default        10.0.1.1        0.0.0.0        UG    0      0        0 eth0
    10.0.1.0        *              255.255.255.0  U    0      0        0 eth0

    here are all nat rules from pfsense:

    no nat proto carp all
    nat-anchor "natearly/" all
    nat-anchor "natrules/
    " all
    nat on xn0 inet from <networks_to_nat>to any -> 10.0.0.20 port 1024:65535
    nat on xn1 inet from <networks_to_nat>to any -> 10.0.1.140 port 1024:65535
    nat on xn0 inet from 127.0.0.0/8 to any port = isakmp -> 10.0.0.20 static-port
    nat on xn0 inet from 127.0.0.0/8 to any -> 10.0.0.20 port 1024:65535
    nat on xn1 inet from 127.0.0.0/8 to any port = isakmp -> 10.0.1.140 static-port
    nat on xn1 inet from 127.0.0.0/8 to any -> 10.0.1.140 port 1024:65535
    nat on xn0 inet from 172.25.53.0/24 to any port = isakmp -> 10.0.0.20 static-port
    nat on xn0 inet from 172.25.53.0/24 to any -> 10.0.0.20 port 1024:65535
    nat on xn1 inet from 172.25.53.0/24 to any port = isakmp -> 10.0.1.140 static-port
    nat on xn1 inet from 172.25.53.0/24 to any -> 10.0.1.140 port 1024:65535
    no rdr proto carp all
    rdr-anchor "relayd/" all
    rdr-anchor "tftp-proxy/
    " all
    rdr on xn0 inet proto tcp from any to 10.0.0.20 port = 8000 -> 10.0.1.34 port 22
    rdr-anchor "miniupnpd" all

    related fw rules

    pass out inet all flags S/SA keep state allow-opts label "let out anything IPv4 from firewall host itself"
    pass out inet6 all flags S/SA keep state allow-opts label "let out anything IPv6 from firewall host itself"
    pass out route-to (xn0 10.0.0.1) inet from 10.0.0.20 to ! 10.0.0.0/24 flags S/SA keep state allow-opts label "let out anything from firewall host itself"
    pass out route-to (xn1 10.0.1.1) inet from 10.0.1.140 to ! 10.0.1.0/24 flags S/SA keep state allow-opts label "let out anything from firewall host itself"
    anchor "userrules/" all
    pass in log quick on xn0 reply-to (xn0 10.0.0.1) inet proto icmp from any to 10.0.0.20 keep state label "USER_RULE: Default ICMP rule"
    pass in quick on xn0 reply-to (xn0 10.0.0.1) inet proto tcp from any to 10.0.0.20 port = ssh flags S/SA keep state label "USER_RULE: Default SSH rule replace_src_with_mgmtnet"
    pass in log quick on xn0 reply-to (xn0 10.0.0.1) inet proto tcp from any to 10.0.0.20 port = https flags S/SA keep state label "USER_RULE: Default HTTPS rule replace_src_with_mgmtnet"
    pass in log quick on xn0 reply-to (xn0 10.0.0.1) inet proto tcp from any to 10.0.0.20 port = http flags S/SA keep state label "USER_RULE: Default HTTP rule replace_src_with_mgmtnet"
    pass in quick on xn0 reply-to (xn0 10.0.0.1) inet proto tcp from any to 10.0.1.34 port = ssh flags S/SA keep state label "USER_RULE: NAT 8000 to zabbix 22"
    pass in log quick on xn1 reply-to (xn1 10.0.1.1) inet from 10.0.1.0/24 to any flags S/SA keep state label "USER_RULE: Default allow LAN to any rule"
    pass in log quick on xn1 reply-to (xn1 10.0.1.1) inet from 10.0.1.1 to 10.0.1.140 flags S/SA keep state label "USER_RULE: Easy Rule: Passed from Firewall Log View"
    anchor "tftp-proxy/
    " all</networks_to_nat></networks_to_nat></performnud,auto_linklocal></rxcsum,txcsum></up,broadcast,running,simplex,multicast></performnud,auto_linklocal></rxcsum,txcsum></up,broadcast,running,simplex,multicast></performnud,auto_linklocal></rxcsum,txcsum,rxcsum_ipv6,txcsum_ipv6></up,loopback,running,multicast></performnud,auto_linklocal></promisc>


Log in to reply