Internal DNS Not Working
-
@aiden21c No,
- CPE is Customer Premises Equipment, it is the "router" ISPs give to domestic and small professional clients
- PON is the small box that turns fiber to ethernet (if you don't connect a device directly to fiber)
- FTTH is Fiber To The Home (meaning, no DSL)
- PPPoE passthrough is being able to connect PfSense like a CPE does (PPPoE) through the CPE itself (used on DSL, where you must either do that, or have a separate DSL modem and lose telephony)
- ACL MAC restrictions is the ISP only allowing their routers (CPE) to connect directly to the internet by blocking other MAC addresses
- CG-NAT is Carrier-Grade Network Address Translation, the "solution" ISPs have for the depletion of IPv4s, ie grouping whole buildings or even neighborhoods and passing them through a single public IPv4/32 (it's cheaper for them, less allocated public addresses)
Your IT team should have made quick work of this. Have them review the situation. Perhaps, in your case, you cannot even use PfSense and all you need (besides setting up VPN and subnetting/VLANs on the Cisco routers) is a Managed Layer 2 switch, in order to distribute the VLANs to your LAN devices.
The switch I proposed was an unmanaged one, just to get both the ISP equipment that has telephony and such and PfSense to be able to "talk" to the ISP PPP server over the fiber ethernet connection (Point to Point Over Ethernet - PfSense to ISP, CPE to ISP). That was proposed in order to both connect PfSense and not lose any telephony (VoIP) or TV you might be getting from the fiber ISP.
I will whip up a diagram.
-
@nightlyshark said in Internal DNS Not Working:
@gertjan I must confess I don't have your full network diagram, though. If you get a chance, just to help anyone researching this in the future, make us a mock-up.
Something like this :
{The Internet} <=> {My ISP} <=> fiber <=> [WAN pfSense] //pfSense// ( 3 LAN interfaces with dumb devices)
So, nothing special. Fully compliant to the rule : "keep it simple".
@nightlyshark said in Internal DNS Not Working:
Perhaps you were missing the accompanying FW rule?
I showed a NAT rule.
It's a safety net for my captive portal users, based on the pfSense official documentation :
https://docs.netgate.com/pfsense/en/latest/recipes/dns-redirect.html
It's not mandatory, and has nothing to do with forwarding/resolving.I just wanted to show-case forwarding => it works for me.
( and Resolving : far better ;) )No "help me" PM's please. Use the forum, the community will thank you.
Edit : and where are the logs ?? -
@aiden21c
Just to not get confused, VLAN trunk is the connection from VLAN switch to Router, the smaller orange circles are physical interfaces (the leftmost one is on the switch, the other 3 on the router, the purple represents virtual interfaces). The pipes are tunnels, the secure one the outer (encrypted) and the other the innermost (VTI-IPsec or Routed OpenVPN). -
@gertjan Nice. Thank you!
-
Made it better. I hope.
-
@aiden21c Just a note,
- The WAN interface is on the fiber and takes its IP (connects) via PPPoE
- The WAN 2 interface uses either:
-
- The mesh network as a single gateway (the SIM modems of the mesh have a master device on which you connect?)
-
- The mesh network as multiple gateways (each modem) and performs its own load balancing. That would give you WAN2, 3 and 4. That means, including the fiber, you now have 4 WAN connections (each WAN has it's own public IPv4).
-
@aiden21c
Just another note, your LAN (192.168.0.0), in that topology, gets it's own VLAN tag (on the switch and the router) and subnet (on the router, if you use a layer 2 managed switch) and you just connect the link from the managed switch to a dumb switch (that is in the room 192.168.0.0 is in) and all other devices (besides wireless), to that dumb switch. -
@aiden21c
For example, from my setup:
The LAGG 0 interface is a group of 3 ethernet (RJ-45) interfaces that work together and connect the PfSense Server to the Managed switch.
Another port (bge3) has PPPoE configured on it and connects to the ISP.
Everything else is VLANs on the LAGG 0.
The important example here is the EDGEROUTER interface. That connects to another managed switch that then distributes 3 different VLANs to another part of the house (TV with VPN, TV without and LAN for the Living Room PC (which is on the TV, haha)). None of those devices mess with each other, because of VLANs. Each VLAN acts like a different switch connected to a different LAN port on the router (PfSense in my case, but Cisco, of course, does 802.11q VLANs as well).The "bridge" interfaces connect PfSense (which is in a VM) to other virtual servers on the same machine (on other VMs).
-
@nightlyshark Thank you so much for everything you managed to help me with. Turns out the solution was nothing to do with double natting or anything complex like that.
Basically, I was ready to give up and was extremely worried I wouldn't have any luck, so as a last resort I tried just reinstalling pfSense from scratch. I vaguely remembered in my testing, at some point this issue did not exist and then something I changed made it occur, but I wasn't sure what.
Essentially, I rebuilt the pfSense from the ground up. I started by simply assigning interfaces so I just did a basic single WAN setup with 1 LAN. Then I assigned the interfaces IPs. From this point, the ping tool on the shell was able to resolve host names no problem.
I then logged into the web interface and did not touch anything I didn't need to. I downloaded the VPN client config export package which I was able to do cause the pfSense was now actually able to reach the internet. I then added a second WAN and set up the routing rules. Everything was still working.
It was only when I went into general setup -> gateways and removed the default gateway. At first, everything was still working, but only after a reboot did this issue show itself. Setting the default IPv4 gateway as "none" and then rebooting seems to cause the entire pfsense to not be able to access the internet, but this does not effect clients. The catch is that the change on this page is only applied at reboot/reroot.
Changing my default IPv4 gateway back to the office gateway and then rebooting completely fixed this issue. I was actually able to get the VPN completely online with only a port forward to my OVPN port on the office router, and everything was working exactly as intended.
Using my tracert from within a VPN client, I was able to verify the traffic was adhering to my routing rules and only traffic directed towards the private Sim network was going through my 4G gateway.
I honestly thought all hope was lost, but taking it step by step seemed to enable me to find the point of failure from the initial setup. -
@aiden21c Good! I still think that some good came out of this whole situation, though.
- For one, even if your current setup works well, the ideal setup for your whole company network is still with VLANs
- The order of the firewall rules needs to be held in mind (PfSense processes firewall packet rules from top to bottom):
Rule 3 catches all traffic filtered by rules 4, 5, 6. It needs to be last. Rules 5 and 6 have destination address "Any" instead of "LAN Address". A way that helps (me personally) to keep fw rules tidy is to add 4 separators, the top one named "GENERAL BLOCK" (for entire protocols, for example, no need to allow GRE, ESP, AH, OSFP... on a LAN with interconnected servers if there is no explicit need), a second separator named "INCOMING", a third separator named "LOCAL TO FW" and a fourth one named "OUTGOING". I also add separators named "PASS" and "BLOCK", with that order, under each main separator. - Even if no further network changes seem necessary, it is best to avoid NAT. In the future, in order to reduce latencies or enable certain UDP services that cannot be NATed, you can check if the Cisco Router can do PPPoE passthrough for PfSense. Because PPPoE is a separate interface in PfSense, you can have both a PfSense-to-Cisco connection (OFFICE - 192.168.20.40/24, not as a Gateway) and a PPPoE adapter as a direct PfSense Gateway (because PPPoE is a Layer 2 protocol, doesn't use IPs, that is why its Point-To-Point, so it doesn't interfere with the 192.168.20.0/24 subnet at all) with a public IPv4 for PfSense.
- At some point, instead of having separate rules for each gateway and traffic type, you might want to implement Multi-WAN Load Balancing and Traffic Shaping to control which traffic type uses what Gateway.
- It is best to set static IPs for LAN through the DHCP server (without a dynamic address pool) and set your private IPs as Static Mappings. That way, you can use Host Overrides on Unbound, which would allow you to use hostnames (and no IPs) in your setups, and avoid unnecessary config nightmares in case, say, you want to put everything in Docker. You can just change the IPs in the Static Mappings of PfSense Unbound, add a BIND container to Docker (just to handle the inter-container IPs using the same hostnames) and be done with it.
