AWS lan to wan problem
-
I tried to read all forums posts of matter, and try all known solutions before posting, so I would not create unneccessary rtfm post. Unfortunately, I've been stuck couple of days implementing pfsense as NAT instance in AWS :P
But I cannot get out from the private subnet (ping 8.8.8.8, apt-get update). I have outbound nat rule but things are odd. I have tried to disable all AWS security features for all hosts, so eveything is in allow everything security group.
I created port forwarding rule from public ip/port 8000 to forward ssh to debian running in VPC private subnet. It works fine. I can see the traffic passed in tcpdump properly.
I have very plain AWS VPC with CIDR 10.0.0.0/16
public subnet (10.0.0.0/24) has routes
10.0.0.0/16 local
0.0.0.0/0 aws-provided internet-gatewayprivate subnet (10.0.1.0/24) has routes
10.0.0.0/16 local
0.0.0.0/0 interface-id of LAN facing interfacePublic subnet has Pfsense 2.3.2 running with two interfaces, (10.0.0.20 public, 10.0.1.140 private) and elastic ip attached. One patch is installed to pfsense for chrome netblock problem fix.
I can connect to elasticIP, and reach admin console and ssh to fw with no problem.
AWS gives my private lan ip 10.0.1.140, pinging it is ok as tcpdump is concerned
ping 10.0.1.1 results in lan:
64 bytes from 10.0.1.1: icmp_seq=1 ttl=64 time=0.414 ms
64 bytes from 10.0.1.1: icmp_seq=2 ttl=64 time=0.445 ms
64 bytes from 10.0.1.1: icmp_seq=3 ttl=64 time=0.456 mstcpdump -vv -i xn1 on pfsense
23:37:41.796911 IP (tos 0x0, ttl 64, id 14383, offset 0, flags [none], proto ICMP (1), length 28)
pfSense.localdomain > 10.0.1.1: ICMP echo request, id 56379, seq 48849, length 8
23:37:41.797212 IP (tos 0x0, ttl 64, id 39768, offset 0, flags [none], proto ICMP (1), length 28)
10.0.1.1 > pfSense.localdomain: ICMP echo reply, id 56379, seq 48849, length 8this is great - but then, wget www.google.com on client, and it's all quiet on tcpdump, cant get anything by filterin on source or so. Tried to put full logging on all filters, but can't get apt working inside LAN yet.
interface configurations are as follows
pflog0: flags=100 <promisc>metric 0 mtu 33160
pfsync0: flags=0<> metric 0 mtu 1500
syncpeer: 224.0.0.240 maxupd: 128 defer: on
syncok: 1
enc0: flags=0<> metric 0 mtu 1536
nd6 options=21 <performnud,auto_linklocal>lo0: flags=8049 <up,loopback,running,multicast>metric 0 mtu 16384
options=600003 <rxcsum,txcsum,rxcsum_ipv6,txcsum_ipv6>inet 127.0.0.1 netmask 0xff000000
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4
inet 172.19.0.1 netmask 0xffffff00
nd6 options=21 <performnud,auto_linklocal>xn0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
options=3 <rxcsum,txcsum>ether 02:4e:d2:8b:33:7f
inet6 fe80::4e:d2ff:fe8b:337f%xn0 prefixlen 64 scopeid 0x5
inet 10.0.0.20 netmask 0xffffff00 broadcast 10.0.0.255
nd6 options=21 <performnud,auto_linklocal>media: Ethernet manual
status: active
xn1: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
options=3 <rxcsum,txcsum>ether 02:e5:c2:30:2b:85
inet6 fe80::e5:c2ff:fe30:2b85%xn1 prefixlen 64 scopeid 0x6
inet 10.0.1.140 netmask 0xffffff00 broadcast 10.0.1.255
nd6 options=21 <performnud,auto_linklocal>media: Ethernet manual
status: activeand the inside VPC debian has like this
eth0 Link encap:Ethernet HWaddr 02:75:9b:08:e5:2b
inet addr:10.0.1.34 Bcast:10.0.1.255 Mask:255.255.255.0
inet6 addr: fe80::75:9bff:fe08:e52b/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:9001 Metric:1
RX packets:1949 errors:0 dropped:0 overruns:0 frame:0
TX packets:2088 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:128910 (125.8 KiB) TX bytes:217848 (212.7 KiB)lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)with routes as this
root@ip-10-0-1-34:~# route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default 10.0.1.1 0.0.0.0 UG 0 0 0 eth0
10.0.1.0 * 255.255.255.0 U 0 0 0 eth0here are all nat rules from pfsense:
no nat proto carp all
nat-anchor "natearly/" all
nat-anchor "natrules/" all
nat on xn0 inet from <networks_to_nat>to any -> 10.0.0.20 port 1024:65535
nat on xn1 inet from <networks_to_nat>to any -> 10.0.1.140 port 1024:65535
nat on xn0 inet from 127.0.0.0/8 to any port = isakmp -> 10.0.0.20 static-port
nat on xn0 inet from 127.0.0.0/8 to any -> 10.0.0.20 port 1024:65535
nat on xn1 inet from 127.0.0.0/8 to any port = isakmp -> 10.0.1.140 static-port
nat on xn1 inet from 127.0.0.0/8 to any -> 10.0.1.140 port 1024:65535
nat on xn0 inet from 172.25.53.0/24 to any port = isakmp -> 10.0.0.20 static-port
nat on xn0 inet from 172.25.53.0/24 to any -> 10.0.0.20 port 1024:65535
nat on xn1 inet from 172.25.53.0/24 to any port = isakmp -> 10.0.1.140 static-port
nat on xn1 inet from 172.25.53.0/24 to any -> 10.0.1.140 port 1024:65535
no rdr proto carp all
rdr-anchor "relayd/" all
rdr-anchor "tftp-proxy/" all
rdr on xn0 inet proto tcp from any to 10.0.0.20 port = 8000 -> 10.0.1.34 port 22
rdr-anchor "miniupnpd" allrelated fw rules
pass out inet all flags S/SA keep state allow-opts label "let out anything IPv4 from firewall host itself"
pass out inet6 all flags S/SA keep state allow-opts label "let out anything IPv6 from firewall host itself"
pass out route-to (xn0 10.0.0.1) inet from 10.0.0.20 to ! 10.0.0.0/24 flags S/SA keep state allow-opts label "let out anything from firewall host itself"
pass out route-to (xn1 10.0.1.1) inet from 10.0.1.140 to ! 10.0.1.0/24 flags S/SA keep state allow-opts label "let out anything from firewall host itself"
anchor "userrules/" all
pass in log quick on xn0 reply-to (xn0 10.0.0.1) inet proto icmp from any to 10.0.0.20 keep state label "USER_RULE: Default ICMP rule"
pass in quick on xn0 reply-to (xn0 10.0.0.1) inet proto tcp from any to 10.0.0.20 port = ssh flags S/SA keep state label "USER_RULE: Default SSH rule replace_src_with_mgmtnet"
pass in log quick on xn0 reply-to (xn0 10.0.0.1) inet proto tcp from any to 10.0.0.20 port = https flags S/SA keep state label "USER_RULE: Default HTTPS rule replace_src_with_mgmtnet"
pass in log quick on xn0 reply-to (xn0 10.0.0.1) inet proto tcp from any to 10.0.0.20 port = http flags S/SA keep state label "USER_RULE: Default HTTP rule replace_src_with_mgmtnet"
pass in quick on xn0 reply-to (xn0 10.0.0.1) inet proto tcp from any to 10.0.1.34 port = ssh flags S/SA keep state label "USER_RULE: NAT 8000 to zabbix 22"
pass in log quick on xn1 reply-to (xn1 10.0.1.1) inet from 10.0.1.0/24 to any flags S/SA keep state label "USER_RULE: Default allow LAN to any rule"
pass in log quick on xn1 reply-to (xn1 10.0.1.1) inet from 10.0.1.1 to 10.0.1.140 flags S/SA keep state label "USER_RULE: Easy Rule: Passed from Firewall Log View"
anchor "tftp-proxy/" all</networks_to_nat></networks_to_nat></performnud,auto_linklocal></rxcsum,txcsum></up,broadcast,running,simplex,multicast></performnud,auto_linklocal></rxcsum,txcsum></up,broadcast,running,simplex,multicast></performnud,auto_linklocal></rxcsum,txcsum,rxcsum_ipv6,txcsum_ipv6></up,loopback,running,multicast></performnud,auto_linklocal></promisc>