Inconsistant pinging across OPT (again but different)
-
TL:DR;
I can ping OPT1 but not OPT3 from LAN. This is a brand new PFSense install (all I did is configure interfaces & DHCP) with "allow all" on LAN and "block all" on WAN as supplied by PFSense (all OPT have no rules, so block all).
The ONLY difference between OPT1 & OPT3 that I am aware of (besides names & IP addresses) is that OPT3 is (intentionally) set as "gateway none" on the DHCP Server setup
Full Background:
I have a Linux box with multiple ethernet ports. I am trying to put the ports on different LANs within PFSense (where only 1 has internet access). Both interfaces are receiving IP addresses via DHCP (from PFSense), but I can only ping the interface on OPT1...not on OPT3. I can ping either interface from their respective interface
And for reference, this is the Linux box (headless ubuntu):
Interfaces & IPs are listed here (WAN is removed but is DHCP, all others are static /24 routes)
My firewall rules are default (no floating rules)
The ONLY difference between OPT1 & OPT3 that I am aware of (besides names & IP addresses) is that OPT3 is (intentionally) set as "gateway none" on the DHCP Server setup
So to summarize:
What works:
LAN Interface --> OPT1 Interface (192.168.2.1)
LAN Interface --> OPT1 Device (192.168.2.21)
LAN Interface --> OPT3 Interface (192.168.4.1)What fails
LAN Interface --> OPT3 Device (192.168.4.21)"interface" as tested via PFSense Ping Diagnostic
Note: This is the same title (and similar problem) as a previous post of mine. However the solution I found there (clearing bad routes) did not work here.
Thanks in advance!
-
Well not being able to ping something from a different network, screams host firewall on what your trying to ping.. Or what your trying to ping is using something else for its gateway..
Or what your pinging is just not getting the ping..
I would do a simple sniff on the opt interface when you ping - do you see the ping go out? To the correct mac? If no response - then you have to figure out if the device is just not getting it or just not answering because of firewall on the device.
If you can ping from pfsense opt interface IP, but not lan IP - really screams firewall on the device.. 10 second sniff will give you the answer your looking for.
-
@johnpoz said in Inconsistant pinging across OPT (again but different):
If you can ping from pfsense opt interface IP, but not lan IP - really screams firewall on the device.. 10 second sniff will give you the answer your looking for.
Firewall certainly makes sense--but the host firewall (ufw) is disabled >_<
I just found PFSense Packet Capture (diag_packet_capture) and if I'm reading this correctly, it looks like the 4 pings I sent from a device on LAN are indeed passing through the OPT3 interface? This is the response from diag_packet_capture when listening on OPT3
13:41:45.675801 IP 192.168.1.40 > 192.168.4.21: ICMP echo request, id 296, seq 1, length 64 13:41:46.677297 IP 192.168.1.40 > 192.168.4.21: ICMP echo request, id 296, seq 2, length 64 13:41:47.679253 IP 192.168.1.40 > 192.168.4.21: ICMP echo request, id 296, seq 3, length 64 13:41:48.679932 IP 192.168.1.40 > 192.168.4.21: ICMP echo request, id 296, seq 4, length 64 -
yeah you sent them.. So validate its the right mac.. But like I said screams device, either the device has a firewall.. or isn't using pfsense opt IP as its gateway and is sending its response somewhere else.
Sniff on the device - does it get these request? Does it answer sending somewhere else? Wrong mask on the device and it thinks that is local so doesn't send answer back to pfsense.
maybe you have /16 vs /24 for its mask?
This server is multihomed?? Just noticed that - yeah that is going to be problematic for sure!! That is going to be asymmetrical nightmare.. Why do you users insist on doing such nonsense..
What is this devices default gateway?
So this device has legs in 2 different networks that route on pfsense.. If it trys and send its answer to a different interface of pfsense than it got the traffic on. Pfsense is not going to send that on, etc.
-
If the device sends a responed it will send it to the OP1 interface address appropriately to its route for 192.168.1.40.
Since pfSense has no state on LAN for the response packets, it will drop theme. -
@johnpoz I checked MAC & mask, they are all correct. I have some capture information I'll post separately, but I'm happy to discuss design decisions if that will fix the problem.
I am treating this Linux box as a sort of web server to run a bunch of self hosted stuff, and will be exposing it to the internet. The motivation for 2 ports was to have 1 on DMZ with access to internet and another port on a separate network for administration. I was under the impression that this was not unusual?
I'd be happy to discuss more either here or PM if you want to convince me this is a terrible idea ;)
-
@johnpoz as promised, the tcpdump from pinging. It looks like you're on the right path, the multihome seems to be causing some routing issues. AS you can see, it seems to eventually start to sort itself out, but even then I still see 0 packets from the tx side
-
@viragomann said in Inconsistant pinging across OPT (again but different):
Since pfSense has no state on LAN for the response packets, it will drop theme.
Not sure I follow, the TX packet coming from LAN hits "allow all" on the LAN firewall and is allowed through. This packet creates a "state" so the response will be allowed. What am I missing?
-
@coatmaker618 said in Inconsistant pinging across OPT (again but different):
nd will be exposing it to the internet.
That is the WORSE case scenario to multihome.. If the box was compromised - it would have direct access to any network it directly attached too, without having to go through firewall.
-
@coatmaker618
I meant, response will go back on OP1.
I correct it above. -
@johnpoz the other network would also be isolated too. It would NOT be a shortcut to the LAN or NAS or other private stuff. However I see your point. The goal of all this redesign I'm working on was to get stuff like this off the trusted LAN!! But it seems I'm taking that separation too far with multiple interfaces?
So your approach would be a single interface on DMZ for this server and just add a rule to LAN to say "allow ssh" & other debug stuff?
-
@viragomann that would certainly be a problem if Linux decided to respond over the default interface rather than the interface which received the packet.
That seems like a bad plan to me, but it would seem that perhaps that's what happening?
Also, sorry for the delay in response viragomann . Apparently I'm still too sketchy (not enough reputation) to post more than once every 2 minutes
-
@coatmaker618 said in Inconsistant pinging across OPT (again but different):
So your approach would be a single interface on DMZ for this server and just add a rule to LAN to say "allow ssh" & other debug stuff?
Yes.. what you would do is pinhole access from stuff in your dmz into your lan.. But why would dmz need access to ssh to something in trusted vlan? I can see trusted to dmz.. But stuff dmz should have access to should really be very limited.
My dmz has zero access into any of my other vlans.. you could make the argument that dmz stuff doesn't even have access to your local dns.. Why would dmz need to initiate traffic to any of your local stuff... Local stuff too dmz sure.. If your going to run some service it needs access to - think about putting that resource in the dmz as well. Or yeah a pinhole into another isolated vlan for say sql access or something.
-
@johnpoz my bad, I meant ssh FROM LAN to DMZ.
My network will eventually have several DMZ, and sure, some of them will have limited connectivity (eg: access a database over SQL or mounted drives from a NAS)--but no. The goal is to have LAN be the only "trusted" network, so the only source for ssh (or RDP, or VNC, or whatever, for debugging).
And certainly no DMZ will be able to reach INTO the LAN!
-
@johnpoz I've tried not allowing DNS for DMZ, but I find that web access & DNS is essential for updating services. Anything exposed to the web, be in Linux or a web service, should be kept up to date, no?
How do you keep OSes & services & such up to date on your DMZ without DNS? If there's a good approach, I may use this myself!
-
well you would have to point to dns, just external like googledns, or cloudflare.. I am talking about internal dns so if the box was compromised they wouldn't even be able to resolve your internal hosts on other vlans, and couldn't take down your local dns..
-
@johnpoz OH!
I gotcha! That's an interesting approach, you just have PFSense forward the DNS? I've never played around with that.
-
No not forward.. You would set say 8.8.8.8 for dns on the device in the dmz, and just allow that.. And block it from even talking to pfsense IP on 53..
If your box in the dmz can not talk to anything on the rest of your network on its own - why would it need to resolve any of your local stuff by name ;)
-
@johnpoz gotcha, that makes sense! Interesting point :)
-
ok, can someone remind me how to change the name of the topic...since I'm writing it off as solved at this point.
- It seems PFSense is indeed passing the communication along
- So it's an issue with the host having multiple active interfaces. The answer seems to be "stop it"
