Internal DNS Not Working
-
@aiden21c The counters for your first 2 rules show no traffic, just pointing it out.
When I asked about the FW rules, I meant (mostly) Floating, WAN and WAN2. -
@aiden21c If you have private IPs on WAN and WAN2, that means that PfSense is behind a NAT. Routers should not (generally) connect to each other via the private space or behind NAT.
-
@nightlyshark I don't have any fw rules on either Wan interface, nor any floating rules. I believe this is the issue I'm having where I have a double NAT on each interface. As per my network diagram above I do have routers on both WANs which are acting as a gateway for the pfsense. How would I go about avoiding this issue as I don't really have any other option for my setup.
-
@aiden21c said in Internal DNS Not Working:
I dont understand how removing the IPv4 route rules
Your NZ_Office rule is before the others (including DNS) and covers everything, making those rules unusable. The more general rules (any->any) go after any rules with restrictions (443, 80, 53).
Also, your second rule has a destination of 192.168.6.1/30. Is this correct? Maybe you meant 192.168.6.0/30?
Also, also, you have two routing rules for WAN, but a single gateway. Each WAN that routes somewhere should get a gateway.
Also, also, also you have a routing rule for a private IPv4, 10.118.32.0/20:
Is that the office subnet? I didn't see an interface on PfSense with an address in that subnet. -
@aiden21c Do you mean CPE routers when you say "routers"? If so, you must contact your ISP(s) and get them to activate PPPoE passthrough on their CPEs, in order for you to be able to connect with a public IPv4.
-
This is how I'm set up:
-
@aiden21c If you don't have administrative access to those routers, your only option (to sneak your network) would be using a VPN. If you have administrative access on the routers and they are DSL and FTTH, turn PPPoE passthrough on on the CPE modem (call your ISP's tech support) and use a switch where you connect the PON, PfSense WAN and CPE of the FTTH ISP. The FTTH ISP would have to remove ACL MAC restrictions on your line and give you 2 dynamic IPv4s, a CG-NATed one for the CPE (telephony, TV...) and a no NAT one for PfSense.
-
That way, you also get IPv6 prefixes. (Provided your ISPs support IPv6)
-
@nightlyshark I'm not going to lie, I think I understood about half of what you have said. I do have administrative access to the office router via the site IT team, who has been more than happy to enable anything for me on the site.
The 10.118.32.0 network is my mesh Sim network I am attempting to give my OpenVPN clients. access to. I do not require any IPv6 networking, or any DHCP on the LAN of my pfsense.
I am struggling a little bit to understand exactly what you mean by the solution, but basically in the office the IT team has installed a basic cisco router setup which connects to the ISP. I believe this would be the CPE. I can get PPPoE enabled on this router which might help with my situation.
Im bit unsure what you mean when you talk about the PON and such, but basically how I have it currently setup is just one singular link from the office router to the pfsense box's wan, as the pfsense is located in an office away from the main network rack. Are you talking about a managed or unmanaged switch in the case you mentioned above? -
@aiden21c No,
- CPE is Customer Premises Equipment, it is the "router" ISPs give to domestic and small professional clients
- PON is the small box that turns fiber to ethernet (if you don't connect a device directly to fiber)
- FTTH is Fiber To The Home (meaning, no DSL)
- PPPoE passthrough is being able to connect PfSense like a CPE does (PPPoE) through the CPE itself (used on DSL, where you must either do that, or have a separate DSL modem and lose telephony)
- ACL MAC restrictions is the ISP only allowing their routers (CPE) to connect directly to the internet by blocking other MAC addresses
- CG-NAT is Carrier-Grade Network Address Translation, the "solution" ISPs have for the depletion of IPv4s, ie grouping whole buildings or even neighborhoods and passing them through a single public IPv4/32 (it's cheaper for them, less allocated public addresses)
Your IT team should have made quick work of this. Have them review the situation. Perhaps, in your case, you cannot even use PfSense and all you need (besides setting up VPN and subnetting/VLANs on the Cisco routers) is a Managed Layer 2 switch, in order to distribute the VLANs to your LAN devices.
The switch I proposed was an unmanaged one, just to get both the ISP equipment that has telephony and such and PfSense to be able to "talk" to the ISP PPP server over the fiber ethernet connection (Point to Point Over Ethernet - PfSense to ISP, CPE to ISP). That was proposed in order to both connect PfSense and not lose any telephony (VoIP) or TV you might be getting from the fiber ISP.
I will whip up a diagram.
-
@nightlyshark said in Internal DNS Not Working:
@gertjan I must confess I don't have your full network diagram, though. If you get a chance, just to help anyone researching this in the future, make us a mock-up.
Something like this :
{The Internet} <=> {My ISP} <=> fiber <=> [WAN pfSense] //pfSense// ( 3 LAN interfaces with dumb devices)
So, nothing special. Fully compliant to the rule : "keep it simple".
@nightlyshark said in Internal DNS Not Working:
Perhaps you were missing the accompanying FW rule?
I showed a NAT rule.
It's a safety net for my captive portal users, based on the pfSense official documentation :
https://docs.netgate.com/pfsense/en/latest/recipes/dns-redirect.html
It's not mandatory, and has nothing to do with forwarding/resolving.I just wanted to show-case forwarding => it works for me.
( and Resolving : far better ;) )No "help me" PM's please. Use the forum, the community will thank you.
Edit : and where are the logs ?? -
@aiden21c
Just to not get confused, VLAN trunk is the connection from VLAN switch to Router, the smaller orange circles are physical interfaces (the leftmost one is on the switch, the other 3 on the router, the purple represents virtual interfaces). The pipes are tunnels, the secure one the outer (encrypted) and the other the innermost (VTI-IPsec or Routed OpenVPN). -
@gertjan Nice. Thank you!
-
Made it better. I hope.
-
@aiden21c Just a note,
- The WAN interface is on the fiber and takes its IP (connects) via PPPoE
- The WAN 2 interface uses either:
-
- The mesh network as a single gateway (the SIM modems of the mesh have a master device on which you connect?)
-
- The mesh network as multiple gateways (each modem) and performs its own load balancing. That would give you WAN2, 3 and 4. That means, including the fiber, you now have 4 WAN connections (each WAN has it's own public IPv4).
-
@aiden21c
Just another note, your LAN (192.168.0.0), in that topology, gets it's own VLAN tag (on the switch and the router) and subnet (on the router, if you use a layer 2 managed switch) and you just connect the link from the managed switch to a dumb switch (that is in the room 192.168.0.0 is in) and all other devices (besides wireless), to that dumb switch. -
@aiden21c
For example, from my setup:
The LAGG 0 interface is a group of 3 ethernet (RJ-45) interfaces that work together and connect the PfSense Server to the Managed switch.
Another port (bge3) has PPPoE configured on it and connects to the ISP.
Everything else is VLANs on the LAGG 0.
The important example here is the EDGEROUTER interface. That connects to another managed switch that then distributes 3 different VLANs to another part of the house (TV with VPN, TV without and LAN for the Living Room PC (which is on the TV, haha)). None of those devices mess with each other, because of VLANs. Each VLAN acts like a different switch connected to a different LAN port on the router (PfSense in my case, but Cisco, of course, does 802.11q VLANs as well).The "bridge" interfaces connect PfSense (which is in a VM) to other virtual servers on the same machine (on other VMs).
-
@nightlyshark Thank you so much for everything you managed to help me with. Turns out the solution was nothing to do with double natting or anything complex like that.
Basically, I was ready to give up and was extremely worried I wouldn't have any luck, so as a last resort I tried just reinstalling pfSense from scratch. I vaguely remembered in my testing, at some point this issue did not exist and then something I changed made it occur, but I wasn't sure what.
Essentially, I rebuilt the pfSense from the ground up. I started by simply assigning interfaces so I just did a basic single WAN setup with 1 LAN. Then I assigned the interfaces IPs. From this point, the ping tool on the shell was able to resolve host names no problem.
I then logged into the web interface and did not touch anything I didn't need to. I downloaded the VPN client config export package which I was able to do cause the pfSense was now actually able to reach the internet. I then added a second WAN and set up the routing rules. Everything was still working.
It was only when I went into general setup -> gateways and removed the default gateway. At first, everything was still working, but only after a reboot did this issue show itself. Setting the default IPv4 gateway as "none" and then rebooting seems to cause the entire pfsense to not be able to access the internet, but this does not effect clients. The catch is that the change on this page is only applied at reboot/reroot.
Changing my default IPv4 gateway back to the office gateway and then rebooting completely fixed this issue. I was actually able to get the VPN completely online with only a port forward to my OVPN port on the office router, and everything was working exactly as intended.
Using my tracert from within a VPN client, I was able to verify the traffic was adhering to my routing rules and only traffic directed towards the private Sim network was going through my 4G gateway.
I honestly thought all hope was lost, but taking it step by step seemed to enable me to find the point of failure from the initial setup. -
@aiden21c Good! I still think that some good came out of this whole situation, though.
- For one, even if your current setup works well, the ideal setup for your whole company network is still with VLANs
- The order of the firewall rules needs to be held in mind (PfSense processes firewall packet rules from top to bottom):
Rule 3 catches all traffic filtered by rules 4, 5, 6. It needs to be last. Rules 5 and 6 have destination address "Any" instead of "LAN Address". A way that helps (me personally) to keep fw rules tidy is to add 4 separators, the top one named "GENERAL BLOCK" (for entire protocols, for example, no need to allow GRE, ESP, AH, OSFP... on a LAN with interconnected servers if there is no explicit need), a second separator named "INCOMING", a third separator named "LOCAL TO FW" and a fourth one named "OUTGOING". I also add separators named "PASS" and "BLOCK", with that order, under each main separator. - Even if no further network changes seem necessary, it is best to avoid NAT. In the future, in order to reduce latencies or enable certain UDP services that cannot be NATed, you can check if the Cisco Router can do PPPoE passthrough for PfSense. Because PPPoE is a separate interface in PfSense, you can have both a PfSense-to-Cisco connection (OFFICE - 192.168.20.40/24, not as a Gateway) and a PPPoE adapter as a direct PfSense Gateway (because PPPoE is a Layer 2 protocol, doesn't use IPs, that is why its Point-To-Point, so it doesn't interfere with the 192.168.20.0/24 subnet at all) with a public IPv4 for PfSense.
- At some point, instead of having separate rules for each gateway and traffic type, you might want to implement Multi-WAN Load Balancing and Traffic Shaping to control which traffic type uses what Gateway.
- It is best to set static IPs for LAN through the DHCP server (without a dynamic address pool) and set your private IPs as Static Mappings. That way, you can use Host Overrides on Unbound, which would allow you to use hostnames (and no IPs) in your setups, and avoid unnecessary config nightmares in case, say, you want to put everything in Docker. You can just change the IPs in the Static Mappings of PfSense Unbound, add a BIND container to Docker (just to handle the inter-container IPs using the same hostnames) and be done with it.
