2 LAN 1 WAN - Banging my head against a wall

ksuclipse

Ok so i've googled all over the place and as far as i can tell this should be working but its not. Here is my Topo:

LAN - 10.0.0.0/16 - network works fine
OPT1 - 192.168.0.0/24 - DHCP is working but no net, cant ping 192.168.0.1 (opt1 gateway)

Firewall Rules:

LAN -
* * * LAN Address 443,80,22 * * Anti-Lockout Rule
IPv4 * LAN net * * * * none Default allow LAN to any rule

OPT1 -

iPv4 * OPT1 net * LAN net * * none
IPv4 * OPT1 net * * * * none

i can ping the OPT1 gateway from LAN but devices on OPT1 cannot ping opt1 gateway nor anything else. i feel like i must be missing something simple but cant see it.

Also, the networks are segemented on two separate physical LANS as well.

johnpoz

well if you can not ping opt1 interface from devices on opt1 network and you have a any any rule… Would seem to be a layer 1/2 problem.. From a client on opt1 when you ping the IP does it get valid mac? Does it get dhcp address via dhcp server running on pfsense?

ksuclipse

yes dhcp assigns properly, i can see the assignment in the logs with the correct MAC. i've tried with multiple machines as well.

KOM

Post a screen of your OPT1 interface details.

johnpoz

post a screen shot of the firewall rules of opt1

ksuclipse

Here is the info. different ip range but same settings…

![Screen Shot 2015-12-06 at 9.25.04 PM.png](/public/imported_attachments/1/Screen Shot 2015-12-06 at 9.25.04 PM.png)
![Screen Shot 2015-12-06 at 9.25.04 PM.png_thumb](/public/imported_attachments/1/Screen Shot 2015-12-06 at 9.25.04 PM.png_thumb)
![Screen Shot 2015-12-06 at 9.25.22 PM.png](/public/imported_attachments/1/Screen Shot 2015-12-06 at 9.25.22 PM.png)
![Screen Shot 2015-12-06 at 9.25.22 PM.png_thumb](/public/imported_attachments/1/Screen Shot 2015-12-06 at 9.25.22 PM.png_thumb)

johnpoz

So your saying you get the correct mac address when you ping and look in arp table.. But you don't get any any responses.. Does pfsense see the pings? Do a simple sniff.. Does it send back response?

Your logging traffic.. so should see the traffic to opt1

ksuclipse

i see some pass logs in the firewall log:

pass/1449420124
Dec 7 18:21:06	OPT1	Icon Reverse Resolve with DNS  Icon Easy Rule: Add to Block List 172.16.0.100:57866	Icon Reverse Resolve with DNS  Icon Easy Rule: Pass this traffic 8.8.8.8:53	UDP
 pass/1449420124
Dec 7 18:21:06	OPT1	Icon Reverse Resolve with DNS  Icon Easy Rule: Add to Block List 172.16.0.100:53998	Icon Reverse Resolve with DNS  Icon Easy Rule: Pass this traffic 8.8.8.8:53	UDP
 pass/1449420124
Dec 7 18:21:02	OPT1	Icon Reverse Resolve with DNS  Icon Easy Rule: Add to Block List 172.16.0.100:57866	Icon Reverse Resolve with DNS  Icon Easy Rule: Pass this traffic 8.8.4.4:53	UDP
 pass/1449420124
Dec 7 18:21:02	OPT1	Icon Reverse Resolve with DNS  Icon Easy Rule: Add to Block List 172.16.0.100:53998	Icon Reverse Resolve with DNS  Icon Easy Rule: Pass this traffic 8.8.4.4:53	UDP
 pass/1449420124
Dec 7 18:21:01	OPT1	Icon Reverse Resolve with DNS  Icon Easy Rule: Add to Block List 172.16.0.100:60188	Icon Reverse Resolve with DNS  Icon Easy Rule: Pass this traffic 8.8.8.8:53	UDP
 pass/1449420124
Dec 7 18:20:54	OPT1	Icon Reverse Resolve with DNS  Icon Easy Rule: Add to Block List 172.16.0.100:55318	Icon Reverse Resolve with DNS  Icon Easy Rule: Pass this traffic 8.8.8.8:53	UDP
 pass/1449420124
Dec 7 18:20:54	OPT1	Icon Reverse Resolve with DNS  Icon Easy Rule: Add to Block List 172.16.0.100:62422	Icon Reverse Resolve with DNS  Icon Easy Rule: Pass this traffic 8.8.8.8:53	UDP
 pass/1449420124
Dec 7 18:20:53	OPT1	Icon Reverse Resolve with DNS  Icon Easy Rule: Add to Block List 172.16.0.100:62383	Icon Reverse Resolve with DNS  Icon Easy Rule: Pass this traffic 8.8.8.8:53	UDP
 pass/1449420124
Dec 7 18:20:51	OPT1	Icon Reverse Resolve with DNS  Icon Easy Rule: Add to Block List 172.16.0.100:18188	Icon Reverse Resolve with DNS  Icon Easy Rule: Pass this traffic 8.8.4.4:53	UDP
 pass/1449420124
Dec 7 18:20:51	OPT1	Icon Reverse Resolve with DNS  Icon Easy Rule: Add to Block List 172.16.0.100:43060	Icon Reverse Resolve with DNS  Icon Easy Rule: Pass this traffic 8.8.4.4:53	UDP
 pass/1449420124
Dec 7 18:20:50	OPT1	Icon Reverse Resolve with DNS  Icon Easy Rule: Add to Block List 172.16.0.100:55318	Icon Reverse Resolve with DNS  Icon Easy Rule: Pass this traffic 8.8.4.4:53	UDP

however strangely enough even though the client has 172.16.0.100, it does not show up in the arp table, only the gateway at 172.16.0.1.

straight packet capture does not show much either:


18:18:41.087386 IP 0.0.0.0.68 > 255.255.255.255.67: UDP, length 300
18:18:42.089387 IP 172.16.0.1.67 > 172.16.0.100.68: UDP, length 300
18:18:43.090801 IP 0.0.0.0.68 > 255.255.255.255.67: UDP, length 300
18:18:43.092746 IP 172.16.0.1.67 > 172.16.0.100.68: UDP, length 300
18:18:43.093828 ARP, Request who-has 172.16.0.100 tell 0.0.0.0, length 46
18:18:43.418942 ARP, Request who-has 172.16.0.100 tell 0.0.0.0, length 46
18:18:43.798065 ARP, Request who-has 172.16.0.100 tell 0.0.0.0, length 46
18:18:44.120205 ARP, Request who-has 172.16.0.100 tell 172.16.0.100, length 46
18:18:44.464153 ARP, Request who-has 172.16.0.100 tell 172.16.0.100, length 46
18:18:44.799418 ARP, Request who-has 172.16.0.100 tell 172.16.0.100, length 46
18:18:44.800130 ARP, Request who-has 172.16.0.1 tell 172.16.0.100, length 46
18:18:44.800136 ARP, Reply 172.16.0.1 is-at {mac address}, length 28
18:18:44.800785 ARP, Request who-has 169.254.255.255 tell 172.16.0.100, length 46
18:18:44.835724 ARP, Request who-has 172.16.0.1 tell 172.16.0.100, length 46
18:18:44.835736 ARP, Reply 172.16.0.1 is-at {mac address}, length 28
18:18:44.835921 IP 172.16.0.100.53002 > 8.8.8.8.53: UDP, length 47
18:18:44.836843 IP 172.16.0.100.63320 > 8.8.8.8.53: UDP, length 43
18:18:44.912535 IP 172.16.0.100.5353 > 224.0.0.251.5353: UDP, length 107
18:18:45.120844 ARP, Request who-has 169.254.255.255 tell 172.16.0.100, length 46
18:18:45.219301 IP 172.16.0.100.55664 > 8.8.8.8.53: UDP, length 42
18:18:45.220147 IP 172.16.0.100.64176 > 8.8.8.8.53: UDP, length 31
18:18:45.227750 IP 172.16.0.100.50821 > 8.8.8.8.53: UDP, length 43
18:18:45.227820 IP 172.16.0.100.58701 > 8.8.8.8.53: UDP, length 34
18:18:45.227956 IP 172.16.0.100.53571 > 8.8.8.8.53: UDP, length 27
18:18:45.228154 IP 172.16.0.100.59094 > 8.8.8.8.53: UDP, length 33
18:18:45.228442 IP 172.16.0.100.60160 > 8.8.8.8.53: UDP, length 50
18:18:45.231955 IP 172.16.0.100.54951 > 8.8.8.8.53: UDP, length 43
18:18:45.234764 IP 172.16.0.100.56886 > 8.8.8.8.53: UDP, length 44
18:18:45.404680 IP 172.16.0.100.5353 > 224.0.0.251.5353: UDP, length 113

Ping from OPT1


PING 172.16.0.100 (172.16.0.100) from 172.16.0.1: 56 data bytes

--- 172.16.0.100 ping statistics ---
3 packets transmitted, 0 packets received, 100.0% packet loss

Ping from LAN


PING 172.16.0.100 (172.16.0.100) from 10.10.0.1: 56 data bytes

--- 172.16.0.100 ping statistics ---
3 packets transmitted, 0 packets received, 100.0% packet loss

KOM

Let's simplify just to rule out gremlins. Can you disable the block rule and then set the allow rule's Source from OPT1 net to *? It shouldn't make any difference, but it's easy enough to verify. Does the client at 172.16.0.100 have a firewall blocking ICMP?

ksuclipse

No change with only having the allow rule. The client has all firewall disabled and is pingable on the LAN net. At least I'm not going crazy on this one, it should work! If it makes a difference the hardware is a vk-t40e

Derelict

Are we dealing with a USB NIC?

ksuclipse

No on the interface, yes on one of the client's but also tested on another machine.

KOM

The client has all firewall disabled and is pingable on the LAN net.

Can you explain what you mean here? I thought the problem was that nothing could ping anything in OPT1? And OPT1 can't ping anything in LAN?

ksuclipse

Correct. But if I move the client to the primary LAN (not opt1) it connects fine and is pingable through the LAN interface.

johnpoz

Why is your client .100 arping for APIPA broadcast address?

169.254.255.255 tell 172.16.0.100

So I see the dhcp process, then the client .100 testing if anyone else has the .100 IP address..

I see him arp for the gateway

18:18:44.835724 ARP, Request who-has 172.16.0.1 tell 172.16.0.100, length 46
18:18:44.835736 ARP, Reply 172.16.0.1 is-at {mac address}, length 28
18:18:44.835921 IP 172.16.0.100.53002 > 8.8.8.8.53: UDP, length 47
18:18:44.836843 IP 172.16.0.100.63320 > 8.8.8.8.53: UDP, length 43

And then send traffic to googldns.. But where is this ping??? I see no PING in that sniff.. How can pfsense answering if it never sees a ping???

Where is your ping the gateway 172.16.0.1… And WTF you sniffing out the mac address??? Really?? Is that the ACTUAL mac address of pfsense opt1 interface.. I would assume so since your saying your seeing traffic to googledns.. Your saying pfsense does not see the mac address of the client in its arp table?? Even when you try and ping it from pfsense?