How can I tell my pfsense box a route to my upstream gateway?
-
How can I tell my pfsense box a route to my upstream gateway?
System | Routing
is where you setup gateways. -
If you've disabled NAT then you'll have to add a route back to the upstream router as you found.
PfSense routes between all it's interfaces by default. You do not have to bridge anything.
Steve
Those two comments seem to be mutually incompatible and my empirical evidence suggests the second is false (at least in my rfc/bogon system).
HOW can I add an upstream route? The System-Routing options doesn't seem to be working. I already posted details in the original post, but here's a more verbatim excerpt from the page:
–---
Interface: WAN
Address Family: IPv4 (I've disabled ipv6 everywhere to eliminate complications for testing).
Name: defgw
Gateway: 192.168.101.1 <-----
Default Gateway : checked.
Disable Gateway Monitoring: Checked.
Monitor IP: blank
Advanced: No changes from defaults
Description: default gwNothing seems obviously wrong and that is the ONLY gateway specified on the pfsense box. Again, if I turn on bridging for those two, it works. That's a kludge though and I'd rather "do it right".
Do you not see the upstream router responding to pfSenses arp request? Why not?
Yes, the upstream router sees the arp and can arp the pfWAN back as expected:
If I log into the upstream router itself (101.1), I can NOT ping the WAN IP (101.3).
If I do an ARP check from the upstream router for that IP though, I DO get a response for the correct pfWAN mac in question.
If I then log into the pfsense and run an arp 192.168.101.1 I get the correct upstream MAC.
If I clear the upstream router's arp table and then re arp from the pfWAN, it still works and the arp entry is created upstream as well.
If I again try to ping from upstream to the pfWAN it still fails though.As mentioned, a PC on the WAN side running wireshark see's ARP requests stating that 101.3 wants to know where 101.1 is… but 101.3 never seems to learn the hw address. If I bridge it, it works instantly so I;'m assuming a failed route in pfsense, but, since I have only the one route, I don't know how to troubleshoot that further at the moment. If I have tio bridge it to get it done I'll do that, but I 'd like to know WHY it isn't working. I assume it's either something bizarre or stupid.I've banged away at it for three days now so I have tunnel vision to whatever the stupid reason may be so I'm hoping to have someone else whose banged their head similarly and solved it chime in. Plus this way this result will come up in google next time I need to fix it.
Is there a diagnostic I can run from the pfsense GUI itself to check the reverse? I don't see any such tool. While I can always log in to the console and manually build routing tables, that's just leaving a grenade for the next guy that will need to support this (even if the next guy is me at 2AM six months from now...). As I said, ARP works but ping does not (from WAN).
It SEEMS like ARP is being routed but IP isn't? I've disabled everything in the firewall and, just in case, I have pass any any on both WAN and LAN groups.
Is there some other command I can run? Some filter to apply to use to look for something in wireshark? I've got bumpkiss other solutions coming to mind for solving the "Why" issue. Workarounds I got plenty, but I'd like to know why it isn't working so I can file a bug if needed and save myself from this again the next time it comes up.
How can I tell my pfsense box a route to my upstream gateway?
System | Routing
is where you setup gateways.Thanks, but as I stated I already tried that and it isn't working in my specific case. See the details previously posted if curious.
-
You are way overcomplicating things, man.
I'll get fancy later, but right now my goals are just to pass basic tests, have PC1 ping PC2 and vice versa, but I can't even get from one pfsense nic to the other.
Another, non-pfsense firewall/router upstream at 192.168.101.1 255.255.255.0 GW Public WAN IP via modem bridge
pfsense WANIP 192.168.101.3 /24 (also tried /32) GW 192.168.101.1 (or no GW specified in pfsense GUI - I've tried both ways no difference).
pfsense LANIP 192.168.102.1 /24 (also tried /32) no GW specified in pfsense GUI
PC1 (Wan side (bogon) on 192.168.101.2 255.255.255.0 GW 192.168.101.1
PC2 (Lan side (bogon) on 192.168.102.2 255.255.255.0 GW 192.168.102.1In UNBRIDGED mode:
PC1 can NOT ping pfWANIP 101.3
PC2 CAN ping pfWANIP 101.3
Note that tracert reports only 1 hop there, it doesn't say 102.1 is the first hop and then 101.3 as I (incorrectly?) would expect.pfSense is a firewall. You cannot ping its WAN interface from OUTSIDE (meaning from 192.168.101.0/24) unless you PASS ICMP TRAFFIC INTO WAN with destination WAN address. This is done with a rule on the WAN interface. There is a default PASS IP ANY ANY rule on LAN so it CAN ping the WAN ip.
If you want to just route traffic between two pfSense interfaces (like 192.168.101.0/24 and 192.168.102.0/24) you are going to want to put PASS IP ANY ANY rules on WAN and LAN and disable NAT.
How does your non-pfSense router at 192.168.101.1 know how to get to 192.168.102.0/24? How do any devices on 192.168.101.0/24 know how to get to 192.168.102.0/24?
-
If you've disabled NAT then you'll have to add a route back to the upstream router as you found.
HOW can I add an upstream route?
Sorry I phrased that in a particularly ambiguous way. ::) What I meant was you have to add a route to the pfSense LAN side subnet on the upstream WAN side router otherwise it won't have a route and won't be able respond to pings etc. You had already done that.
I think almost everything you're seeing is related to the fact that pfSense is continually sending arp requests for 192.168.101.1. You haven't said if you're seeing the arp replies in the packet capture. If you aren't then why isn't the up stream router sending them? If you are then why is pfSense asking repeatedly, especially when it already has the MAC in it's arp table?
You definitely don't have to bridge WAN and LAN.
ARP is not a routable protocol so looking for a routing problem is probably not the right way to go. ;)
If you switch back to automatic outbound NAT and allow pfSense to NAT between WAN and LAN does it work? If the arp problem still exists then it still won't be able to reach the upstream gateway. If it does work then start looking for an IP routing problem when NAT is disabled.
I might be tempted to start again here. Set pfSense up in it's default configuration, firewall and NAT, and ensure that everything is working there first.
Steve
-
How can I tell my pfsense box a route to my upstream gateway?
I think there is no need to. By default behaviour of TCP/IP: a request to something outside it's subnet will be sent to the default gw of the local config.
So, your pc will send that to pfSense, and if pfSense does not know it it will pass it on to it's default gateway. And that GW, is responsable for passing it on the upstream network you are trying to reach.
That is, if it is running in FW mode. No bridging, you don't want a switch I suppose.As Derelict already said, pfSense is a FW and you need to add rules if you want icmp (or parts of it -what is better practice-) through it.
On top of that, if you are using nat, most probably your pc on the wan side wil not know the internal (lan) subnet, as it should, it only knows the ip of the wan adapter. If you should want that, you need to add a route in the pc connected to WAN side.If you want the box to be a router, not a fw, you might want to avoid WAN as tittle of your IF (you can name it what you want), and disable NAT for that IF.
HTH? If I totally misunderstood the problem, a drawing might help understand better what the goal of the setup is (not sure I fully understood)
-
I might be tempted to start again here. Set pfSense up in it's default configuration, firewall and NAT, and ensure that everything is working there first.
Steve
I've actually formatted and reinstalled a few times at this point. 32bit, 64 bit, no luck.
Sorry I phrased that in a particularly ambiguous way. ::) What I meant was you have to add a route to the pfSense LAN side subnet on the upstream WAN side router otherwise it won't have a route and won't be able respond to pings etc. You had already done that.
Correct (and thanks for reading what I posted before replying )
Routing List
Destination IP Subnet Mask Gateway Interface Metric Route Mode Type
192.168.102.0 255.255.255.0 192.168.103.1 LAN 5 Transport Manual
192.168.102.22 255.255.255.255 192.168.102.1 LAN 3 Transport ManualI've also added (as part of testing) this line which didn't help:
192.168.103.1 255.255.255.255 192.168.102.1 LAN 5 Transport ManualYou definitely don't have to bridge WAN and LAN.
ARP is not a routable protocol so looking for a routing problem is probably not the right way to go. ;)
Yrah, I've been chasing my tail on this for a while on this so imaginary straws are about all I've got left.
I think almost everything you're seeing is related to the fact that pfSense is continually sending arp requests for 192.168.101.1. You haven't said if you're seeing the arp replies in the packet capture. If you aren't then why isn't the up stream router sending them? If you are then why is pfSense asking repeatedly, especially when it already has the MAC in it's arp table?
I'll have to figure out how to get them to show up in the pfsense packet capture - I'm not seeing anything when I try - Arp or otherwise for the WAN/OTHER side. Even with packet filtering and NAT all off. This is despite the fact that I CAN manually send and ARP request by IP and get the correct mac back from either side.
Let me ask this from another direction though:
Why would the configuration work as expected/desired when in bridged mode but not when unbridged?That fact makes me think it is not the upstream router…
-
Dude. ARP does not cross router ports. ARP is a broadcast protocol. Routers don't forward broadcasts. WTF are you trying to do? Stop overthinking it.
Routing List
Destination IP Subnet Mask Gateway Interface Metric Route Mode Type
192.168.102.0 255.255.255.0 192.168.103.1 LAN 5 Transport Manual
192.168.102.22 255.255.255.255 192.168.102.1 LAN 3 Transport Manual
192.168.103.1 255.255.255.255 192.168.102.1 LAN 5 Transport ManualHow do you expect to get to the 192.168.102.1 AND 192.168.103.1 over the LAN interface?
-
Dude. ARP does not cross router ports. ARP is a broadcast protocol. Routers don't forward broadcasts. WTF are you trying to do? Stop overthinking it.
Thanks, but Steve already pointed that out.
Routing List
Destination IP Subnet Mask Gateway Interface Metric Route Mode Type
192.168.102.0 255.255.255.0 192.168.103.1 LAN 5 Transport Manual
192.168.102.22 255.255.255.255 192.168.102.1 LAN 3 Transport Manual
192.168.103.1 255.255.255.255 192.168.102.1 LAN 5 Transport ManualHow do you expect to get to the 192.168.102.1 AND 192.168.103.1 over the LAN interface?
I put full details in my original post - including how I already HAD an always pass any to any from any rule - as well as having disable packet filtering entirely via the System/Advanced/Fw/nat checkbox. I always put in the upstream router rules and how they corrected the actual routing issues as I see them.
In the most recent reinstall test I moved the pfsense WAN side to 103.1 from 101.3, in case the upstream 101.1 gateway was interfering in some fashion. That third entry, as I pointed out, was added as I grasp at straws.
Again, if I put it in bridge mode, then it does work, so that's the behavior I'd like to focus on.
-
There is no need to grasp at straws. You need to un-fuck your network and you apparently need help to do so. Draw a diagram of what you're trying to do complete with interface IP addresses and netmasks.
-
There is no need to grasp at straws. You need to un-fuck your network and you apparently need help to do so. Draw a diagram of what you're trying to do complete with interface IP addresses and netmasks.
Here's a fresh install, new IPs, same issues.
[PC1]–[UpstreamRouter]
I
[PFsenseDMZ]
[PC2]–[PFsenseLAN1]UR:UpstreamRouter 10.10.10.1 255.255.255.0 (static)
PC1 on DMZ 10.10.10.20 255.255.255.0 (dhcp via UR gw 10.10.10.1)PFsenseDMZ 192.168.1.1 255.255.255.0 UpGW 10.10.10.1
PFsenseLAN1 192.168.2.1 255.255.255.0 No GW
PC2 on LAN2 192.168.2.2 255.255.255.0 (dhcp via pfsense gw 192.168.2.1)In bridge mode, everyone can ping everyone else.
Unbridged, PC2 can ping only to pfWAN, not PC1 or UR and PC1 can ping to UR but not PC2 or ANY PF Nic.PFSense has a floating, quick "Apply the action immediately on match.", allow any from any to any rule applied. The "Disable all packet filtering" checkbox is also checked and NAT is set to manaual with all rules disabled. There are no blocking rules in place - at least none visible via the GUI. The bogon and RFC network non-exclusions are checked.
There are no pfsense routes in the gui. The ones I assume are automatic don't appear to work unbridged and if I try to manual add them it complains of IP conflict vs the IP GW. I'd be glad to input a route there if someone can provide the complete IP data (w.x.y.z and subnet slash value that SHOULD work).
I'm also glad to run any command lines and post that. To repeat the obvious, bridging DMZ-LAN lets ping and arp work. Unbridged, arp works but ping does not.
Again, bridged, it works. Unbridged it does not. This seems like a key detail to me. I'd also welcome an explanation that explains BOTH why that fixes it and simultaneously explains why the automatic routes don't.
Interface, Gateway and FWRules screenshots follow.
-
Why the hell are you messing around with FLOATING RULES!?!
That's not in any walkthrough I know of.
STOP OVERTHINKING IT and put your pass rule on LAN.
That's not a diagram. Draw one in crayon and take a picture if you have to.
-
Why the hell are you messing around with FLOATING RULES!?!
That's not in any walkthrough I know of.
STOP OVERTHINKING IT and put your pass rule on LAN.
That's not a diagram. Draw one in crayon and take a picture if you have to.
The floating rule was to eliminate the other tabs as concerns.
In any case, I do have pass any any on both wan/dmz and lan. See attached.
-
Your diagram has pfSense with DMZ and LAN interfaces, and no interface IP addresses, yet your screenshot has WAN and LAN interfaces? Which is it?
Annotate your diagram with IP addresses and subnet masks of the various interfaces and make it as matchymatchy with what your screenshots say as possible so people know what they heck you're trying to do.
-
When you've bridged the LAN and WAN and you have completely open firewall rules your downstream clients are effectively talking directly to your upstream router. That clearly removes whatever arp issue is causing problems.
Steve
-
When you've bridged the LAN and WAN and you have completely open firewall rules your downstream clients are effectively talking directly to your upstream router. That clearly removes whatever arp issue is causing problems.
Steve
My thought exactly, Steve. I just don't know what else to try on the pfsense box to resolve that. It's as if it is ignoring the default gateway or has some other hidden automatic route somehow or there is a hidden filter still being applied (despite the disable packet filtering being checked). Any thoughts on how I can diagnose that diagnose that?
Your diagram has pfSense with DMZ and LAN interfaces, and no interface IP addresses, yet your screenshot has WAN and LAN interfaces? Which is it?
Annotate your diagram with IP addresses and subnet masks of the various interfaces and make it as matchymatchy with what your screenshots say as possible so people know what they heck you're trying to do.
Here's a new diagram with the IPs on the diagram itself rather than in the comment.
I haven't renamed interfaces from the default for the GUI. The "WAN" from the pfsense screenshots is the DMZ.
Green lines are working links. The red line is a link that works only in bridged mode and the yellow line within the pfsense box is where I suspect there is an issue.
-
You have the pfSenseDMZ interface set with an interface address of 192.168.1.1/24. You are expecting it to be able to send traffic to 10.10.10.1. You can't do that. The gateway for an interface MUST be on the same subnet/segment as the interface itself.
This is basic IP routing / subnetting.
-
You have the pfSenseDMZ interface set with an interface address of 192.168.1.1/24. You are expecting it to be able to send traffic to 10.10.10.1. You can't do that. The gateway for an interface MUST be on the same subnet/segment as the interface itself.
This is basic IP routing / subnetting.
Actually it's a typo in the pic since I've wiped and reset it a number of times. The pfsense DMZ side IP is 10.10.10.2
Corrected pic attached.
-
So what's not working?
What is it you want to have happen? Be specific and let's work one thing at a time.
-
So what's not working?
What is it you want to have happen? Be specific and let's work one thing at a time.
For starters, I want PC1 1.10.10.20 to be able to ping the DMZ/WAN IP 10.10.10.2 on the pfsense box when it is NOT in bridged mode. If it is bridged, I can ping it. If not I can't. Bridged, or not, I can ARP from either side.
-
Start from scratch on pfSense. Enable your WAN (DMZ) interface as 10.10.10.2/24. Create a gateway of 10.10.10.1. Uncheck block private networks.
Set your LAN interface to 192.168.2.2/24.
Edit your firewall rules on WAN/DMZ. Add a pass rule for IPv4 ICMP any Source any Dest WAN address.
DON'T DO ANYTHING ELSE! 10.10.10.1 and 10.10.10.20 will be able to ping 10.10.10.2.
See screen shot. You won't have the block private rule….
