[Resolved] Trouble with new pfSense+FiOS Actiontec router install
-
Major update in this post below.
I'm still looking for help on how to use the firewall logs to see where traffic is coming from/going, and also I'd greatly appreciate some help with how to configure chrome to trust the webconfig certificate.
–---
As the title says, I'm a newbie trying to set up pfsense for the first time. I don't think I'm a moron, but I'm running into serious problems. I'm really at my wits end here. I don't think I've ever been so frustrated. Full disclosure- I'm on pfsense v2.2-rc, since I don't think my hardware is supported by v2.1. However, I really don't my problems are related to using v2.2, which is why I'm posting here.I'm trying to set up a really basic pfsense firewall/router behind my Verizon FIOS actiontec router. At least for now I'm just trying to put the pfsense router directly behind the Actiontec router as the DMZ host. I've done that for a few years now with an Asus router and it seemed to work fine- both for TV and for Internet.
Again, I'm going for a really basic setup. I assigned a static IP on the WAN, using the Google public DNS servers (originally I tried Verizon's, but changed it when it didn't work). I disabled IPv6 on the LAN and WAN. I enabled DNS Query Forwarding (again, originally disabled, but I enabled it when things weren't working).
pfsense is up and running, but I can't seem to get traffic to get through to my clients. Well, most traffic, that is. It's weird. I can go to some websites, like facebook.com and Google.com, but almost no other sites will load. Then I tried pinging some sites from pfsense diagnostics page. Google.com works fine. Facebook.com (despite the site working), fails with 100% loss. As does just about every other site. At first I wondered whether it was a weird DNS configuration problem, the ping does show the right IP address for hosts (e.g., if I try to ping CNN.com, it says its trying 157.166.226.25).
The firewall logs don't seem to help. I'm not sure what I need to do there. First of all, I've been assuming that pfsense by default blocks incoming connections, but allows outgoing connections. Second, the logs don't make any sense to me. The only things that appear in the logs is blocked traffic. I don't see anything in there from the traffic that actually does go through (like connections to facebook and google). Actually, lately it seems like I don't see much of anything particularly relevant in the firewall logs.
I can't identify any patterns for what sites work and what sites don't. At first I thought only sites that I recently visited worked (hence why Google and Facebook work). But then I discovered a handful of other, more obscure sites also seemed to work.
Any ideas? I'm really, really struggling here.
-
I'm still seriously baffled.
I realized that I hadn't disabled blocking private networks on my WAN. I think that should have been required to get any traffic out, but somehow I was still able to get out to Google.com from behind my pfsense router even with that blocking enabled. Turning that option off has had no impact. I still can't get out to most sites. But I'm still somehow able to get out to Google and Facebook.
I still can't figure out what's going on with the firewall rules. I'm seeing a lot of weird stuff in the logs that I can't make heads or tails of.
For example, I see something was blocked coming from my wireless AP to my pfsense router (192.168.1.130:50255 to 192.168.1.1:2189). Why was it blocked? No idea- the log says "@51(10000001570) block drop in log on ! re0 inet from 128.0.0.0/1 to any"
I have no idea if this is related to the problems I'm having.
In general, I also think its very odd that I don't seem to see traffic from my desktop computer showing up in the logs. I turned on all the logging options under Status-System Logs- Settings. If I go to Google.com from my desktop, I expect to see firewall logs showing traffic passing from my desktop to a google server. I don't see that, though. I see a whole bunch of traffic from my WAN IP going out to the internet. I can't easily tell where its going (most of the IPs don't resolve to anything), although it does seem to include some some Google addresses and domains (e.g., 1e100.net). Assuming I can actually get things functionally working, am I going to be able to tell what machines behind my router are sending/receiving traffic from those IPs (rather than just seeing the traffic tied to the WAN address)?
Edit to Add: For what its worth, I tried disabling the DNS resolver (unbound) and went back to the DNS forwarder. No impact whatsoever. I still can't get to sites. I guess I should have expected that. This doesn't seem to be a DNS problem. I can't ping IP addresses for the sites I can't visit.
Edit to Add #2: I'm trying to do whatever I can here… pfsense.org is a site that won't load. However, an nslookup from my windows box behind pfsense does get me to 208.123.73.69. A ping to pfsense.org, though, completely fails. 100% loss. Pinging google works fine- no loss. I've tried whatever I can think of with DNS. I really don't think this is a DNS problem, but I guess I don't know for sure.
-
With the information given the only advice can be to check your ip addresses/masks carefully again and make sure you only have a single DHCP-server per network.
If you want better help you need to explain what networking equipment you have, exactly how it is connected and what ip addresses you use at all networking interfaces. A network diagram often says more than words.
-
Thanks, P3R.
I double checked the IPs and netmasks. At some point pfSense changed the DHCP range (starting at 192.168.10 instead of what I configured earlier, 192.168.1.126), but that doesn't seem to have had an impact. The only reason I was doing that is because I have some static reservations set up between .100 and .125.
There's a DHCP server running on the Verizon router (on the 192.168.0.1/24 subnet), in front of the pfsense router. But, the pfsense router is the only DHCP server on 192.168.1.1/24. My wireless AP is running in AP-only mode, with NAT and DHCP off.
I've attached a network diagram that hopefully provides some other relevant, helpful details.
Note: I'm noticing the same behavior on the laptop and desktop. Some external hosts work, some don't. nslookups on those boxes do seem to give me the right answers, but I still can't ping those hosts (either by IP or hostname).
The Verizon Wireless network extender (a femtocell provided by my mobile phone provider) does seem to work. I have some NAT/firewall rules in place to open holes to that device. However, that device is really only talking to a couple public IPs, so it's not like its getting to hosts that my desktop/laptop can't get to.
![network_diagram (2).png_thumb](/public/imported_attachments/1/network_diagram (2).png_thumb)
![network_diagram (2).png](/public/imported_attachments/1/network_diagram (2).png) -
The lack of internet access on my networked computers was driving me and my wife nuts. I went back to the Asus router, which works perfectly fine behind the Actiontec router (I know there are a lot of Actiontec haters out there).
I'm going to set up the pfSense box behind the Asus router (gasp: triple-NATing) and so some more troubleshooting. I have to admit, though: I am completely out of ideas.
-
Major update: I resolved the most serious problem- the problem with some websites working and others not. I reconfigured pfsense to grab an IP from the Actiontec router via DHCP, rather than getting a static IP. I have no idea why that fixed that problem, but it did.
I'm still running to into a couple other issues that I'd really like to resolve.
- Traffic Monitoring using firewall logs
The main reason I wanted to move to pfsense was to keep a closer eye on what traffic entering/leaving my network. I figured the firewall logs ought to give me that information. By default, traffic which passes rules doesn't seem to be logged, so I modified the logging settings to show that. However, the logs (almost) always show my pfsense router as the source of outgoing traffic, rather than showing the IP address of the client generating that traffic.
How do I tell what client generated that traffic?
- Certificate Warnings in Chrome
The certificate warnings in Chrome are driving me nuts. Sometimes it keeps kicking me out to the cert warning screen every time I go to a different page in the WebGUI. I tried to add the webconfig certificate to the Chrome trust store, but it didn't seem to work. Is there a how-to for that somewhere?
- Traffic Monitoring using firewall logs
-
I have no idea why that fixed that problem, but it did.
Me neither, but that's great!
By default, traffic which passes rules doesn't seem to be logged,…
Correct.
…so I modified the logging settings to show that.
Yes, that's the way to do it.
How do I tell what client generated that traffic?
Rules in pfSense affect traffic coming IN on it's interfaces only. To log what I think you want, both the local source address and the destination (website or other) address in your logs, you should apply logging on the LAN interface rule allowing the traffic. The response to those requests will not, as far as I know be possible to log (and since the request is the interesting part, I see no reason).
If you also want log incoming traffic on the WAN interface (usually not useful), you probably need to use the GUI filtering in the Status, System Logs, Firewall tab to make the logs useful.
If you for explicit troubleshooting reasons need to see the actual incoming response packets to each request also, you have to use Diagnostics, Packet Capture on the WAN interface. Beware, you probably need to make extensive use of filtering to get any useful output from a packet capture on the WAN interface!
- Certificate Warnings in Chrome
The certificate warnings in Chrome are driving me nuts. Sometimes it keeps kicking me out to the cert warning screen every time I go to a different page in the WebGUI. I tried to add the webconfig certificate to the Chrome trust store, but it didn't seem to work. Is there a how-to for that somewhere?
I'm sorry I have no idea. I almost always use Chrome (later more specifically Iron browser) and can't recall ever having experienced what you report. I wouldn't expect that to be the cause for your issues but I don't use 2.2 yet, only 2.1.5.
- Certificate Warnings in Chrome
-
@P3R:
Rules in pfSense affect traffic coming IN on it's interfaces only. To log what I think you want, both the local source address and the destination (website or other) address in your logs, you should apply logging on the LAN interface rule allowing the traffic. The response to those requests will not, as far as I know be possible to log (and since the request is the interesting part, I see no reason).
Excellent. Thanks! You're right- I needed to enable logging in LAN rules.
@P3R:
I'm sorry I have no idea. I almost always use Chrome (later more specifically Iron browser) and can't recall ever having experienced what you report. I wouldn't expect that to be the cause for your issues but I don't use 2.2 yet, only 2.1.5.
After a bit of playing around I (mostly) figured out how to solve this problem.
First of all, I was having problems with the constant reminders because I had too many different machines/browsers connected to the webGUI at once (more than 3).
Second, I figured out how to get the browsers to trust the TLS certificate. I created a new CA certificate and added it to my Trusted Root CA store. Then I created a new server certificate off that root with pfsense. <domain.xxx>as the CN, being sure to also set that as a subject alt name. It took a couple reloads of the page after that (who knows why), but then my browsers now accept and trust the pfsense TLS certificate.</domain.xxx>