Intermittent Problems Reaching Anything Beyond pfSense Firewall



  • Forgive the long post, but I'd rather be told I included too much than that I omitted something important!

    I suspect this issue is with my internet router more than pfSense, but I could use some help and guidance on this. This has gotten worse and worse over the past few weeks, to the point where we frequently can't get to websites and can't stream. (Which means I'm in trouble with family until I fix it.)

    My setup: I have an internet connection through a wireless broadband data reseller. It's quite possible the problem is their router or that AT&T is messing with us because we streamed too much or I shouldn't have downloaded a few Linux distro images or something like that. I have CAT5e running from the wireless router to my Netgate pfSense firewall. No devices in between. On the LAN side of the firewall, it's connected to an ethernet hub and our computers and devices are connected to that. I use wifi only for mobile devices, so if it's a computer, Blu-Ray player, streaming video device, or anything that doesn't move a lot, it's hardwired.

    I used to have an old firewall, a Soekris net5501 with pfSense on it. I don't think I had updated it in at least 8 or 9 years. (Multiple family deaths and a stressful decade - not a lot got done.)

    I finally got to ordering an SG-1100 and set it up. I took screen shots of the old configurations on the Soekris box, including the DHCP settings, static mappings, and so on. I basically used my old device as a firewall, DNS server, and DHCP server. (I'd forward the DNS requests to Google's name servers, like 8.8.8.8.) I copied over all the static DHCP mappings to the new device and triple checked them. I did use a different name for the new firewall than the old one. I also changed the domain from an Oz theme (using Oz names and using house.oz as the LAN domain) to a Tolkien theme (using rivendell.me as the new LAN domain).

    I did have one issue or question about my setup that I posted here.

    What's wrong: In short, intermittent internet access, in weird ways. It usually goes out for 1-2 minutes, then comes back up. I was pretty sure it was the internet router, but now I'm seeing a few things that make me wonder if I've done something that could create caching issues or something else like that.

    During one short outage, I tried pinging. I could ping my firewall. I could ping the LAN side of the internet router. I could not ping beyond that. But it's not that simple and now it's like taking the red pill or going down the rabbit hole. Things are just inconsistent and don't make sense. The problems started just about when I changed to the new firewall.

    Now, I know that sounds like it's clear it's the internet router. But then, this evening, my wife was having serious problems - she couldn't do anything on her computer on the internet. But I was having almost no problem on my computer. (Both hardwired.) I tried with my cell phone, using our wifi and it wouldn't work. We went through this repeatedly this evening, finding that often one computer could connect to internet sites and others couldn't. I got the router IP address and, when it was down, used my iPad to ping our external IP address through a cell connection and it worked. I even found that when my wife was having trouble, I could easily ping websites.

    To be clear: We found that, frequently, one hardwired computer could reach internet sites and another could not at the same time. I found that sites I visited often came up faster than ones I didn't visit often. (I don't think that's got a thing to do with this issue, but I'm mentioning it, just in case.) When internet was down for several computers, I could still go through my cell connection and ping our external IP address.

    There are times where I can ping external sites and can't load a site in my browser.

    We've also seen a weird problem where we can go to a streaming site like Netflix and see video descriptions and not see the thumbnails.

    I'm at the point where I'm thinking I need to put in the old firewall and see if the problems still keep happening.


    My ISP said this could be a problem with my SIMM card in the router or something similar. Since this issues started when I swapped firewalls, I want to be sure I could not have done something in setting up the firewall that could create DNS delays (which might explain why we sometimes can't load websites or get things to stream).

    I'd like to either find out I may have messed up something simple in pfSense without realizing it or be able to verify this is not something that pfSense could be causing. That would make it easier to be firmer with the ISP to be clear it's not something on our end.



  • @TangoOversway said in Intermittent Problems Reaching Anything Beyond pfSense Firewall:

    I tried pinging. I could ping my firewall. I could ping the LAN side of the internet router. I could not ping beyond that

    Can you reach the upstream ISP router ?
    When you experience an outage, what do you see in the
    System logs ?
    DNS logs ?
    DHCP log ?

    @TangoOversway said in Intermittent Problems Reaching Anything Beyond pfSense Firewall:

    (I'd forward the DNS requests to Google's name servers, like 8.8.8.8.)

    Not needed, except if you signed a contract with them like "here are all my private DNS requests".
    During testing, switch to default DNS behaviour (works just fine for decades).



  • @TangoOversway said in Intermittent Problems Reaching Anything Beyond pfSense Firewall:

    On the LAN side of the firewall, it's connected to an ethernet hub and our computers and devices are connected to that.

    You're mentioning equipment that hasn't been upgraded in nearly a decade, so when you use the word ethernet hub here I'm thinking it might actually be a hub and not a misuse of words. Please double check and then triple check that this piece of equipment is actually a hub (not the same as a switch). If it is actually a hub, disconnect everything that is plugged into it and throw the hub in the garbage because that could definitely cause the problems you're describing. Replace it with any decent consumer grade gigabit SWITCH which are under $20 these days. Alternatively, you can try wifi instead of hardwiring those clients that went into the hub.

    Raffi



  • @Raffi_

    A hub? Those started disappearing over 20 years ago. Switches started to become popular in the mid '90s. If it actually is a hub, he'd be limited to 10 Mb, half duplex, unless he has one of those rare 100 Mb hubs.

    If I dig around through my junk closet, I may be able to find a 10 port hub.



  • @JKnott I hope I'm wrong. We somehow still had hubs in the office less than 10 years ago. If you find one in your junk closet, throw it out :)

    Edit: We still have one in the office laying around :o
    Dammit, I can't throw it out to help better the world because it's not mine.



  • @Raffi_

    Outside of my closet, the last time I saw a hub was at work over 15 years ago. At the time, I was working for a really CHEAP company, where the network was a collection of consumer grade hubs and a couple of switches. The guy who was responsible for the network wasn't the brightest. He had a hub next to the server, and switches were closest to the users. He should have had one of them next to the server. When I tried telling the manager that, I was told the IT guy told her otherwise. At that time, I had already been a Novell CNA for a few years, but this guy apparently knew better.



  • On second thought, don't throw it out. It could be donated to a school. It could be good for instructional purposes to demonstrate traffic collisions on a network. Or donate it to a tech museum to help keep waste out of the landfill :)



  • @Raffi_

    After I stopped using it as a hub, I used it with Wireshark, to monitor traffic. However, that function has since been replaced with a managed switch configured for port monitoring. Plugging a 10 Mb hub into a Gb connection causes a bit of a performance hit.

    BTW, you shouldn't be seeing collisions, unless someone misconfigured a switch or NIC to be half duplex.



  • Still catching up, but I realized I made a few mistakes and want to clarify those quickly.

    I was posting with very little sleep over a good while, so I probably should fix a few points:

    1. When we moved into the new house, in September, 2017, I used all new equipment. The whole house was wired with CAT5e (I wanted CAT6 - long story there) and thoroughly tested. Also, remember, this wiring has been in use that long without a problem like this. The only thing left was the old Soekris/pfSense firewall - something I was going to replace "next week" for that full 2 1/2 years.

    2. It's a switch. Sorry, I know I said hub. And a newer one.

    I'm still reading everything and will add more, but thought I should correct those points ASAP.



  • @TangoOversway said in Intermittent Problems Reaching Anything Beyond pfSense Firewall:

    The whole house was wired with CAT5e (I wanted CAT6 - long story there)

    CAT6 won't provide much benefit. Gb Ethernet was designed to work over 100 M (330') of CAT5, before there even was CAT5e. As I doubt you have many 100M runs in your house, it's even less of an issue.

    I have plain CAT5 in my condo. It was installed¹ by my cable company when I got my cable modem, over 20 years ago.

    1. When the modem was installed, they had to run the coax from where it comes into my living room to my "office" at the other end of my unit. Since I didn't want cable stapled to the baseboards, around doors, etc., they sent 2 guys and they took about 3 hours to run it inside the walls, ceiling etc., so that it's completely hidden, except where it crosses over my laundry room ceiling, next to an air duct. I asked them to pull in a couple of runs of CAT5 (I provided) when they did that.


  • @JKnott said in Intermittent Problems Reaching Anything Beyond pfSense Firewall:

    @TangoOversway said in Intermittent Problems Reaching Anything Beyond pfSense Firewall:

    The whole house was wired with CAT5e (I wanted CAT6 - long story there)

    CAT6 won't provide much benefit. Gb Ethernet was designed to work over 100 M (330') of CAT5, before there even was CAT5e. As I doubt you have many 100M runs in your house, it's even less of an issue.

    Well, I do have a 500' run in this system - but that's underground fiber. Did it all myself, except for help with the electrician (who's also a friend) when we needed someone to pull and someone to feed. It was cool watching 500' of jetline zip into the black poly pipe when he turned on the shop vac on the other end! (And by, "Did it all myself," that includes running it under the house and barn/guest house foundations, 475' or so of 2' or deeper trenching, running it under our new roadway at the creek crossing, and running the ethernet cables from the converter locations to the switches in either building.)

    I have plain CAT5 in my condo. It was installed¹ by my cable company when I got my cable modem, over 20 years ago.

    1. When the modem was installed, they had to run the coax from where it comes into my living room to my "office" at the other end of my unit. Since I didn't want cable stapled to the baseboards, around doors, etc., they sent 2 guys and they took about 3 hours to run it inside the walls, ceiling etc., so that it's completely hidden, except where it crosses over my laundry room ceiling, next to an air duct. I asked them to pull in a couple of runs of CAT5 (I provided) when they did that.

    Did they use a cable gun? Those things are cool and I would have loved to have used one here, but they're just too expensive.

    In my old house, I had done a lot of the AC rewiring on it in the 1990s. (The house was from the 1940s - WAY outdated!) I had to spend a lot of time in the attic and crawlspace. But that made it easy to run the ethernet cables in that house when it came time to do so. In this house, I had a great general contractor (also a friend) and closely supervised everything. I ran the HDMI, coax, and ethernet cables here and in the barn by myself.



  • @TangoOversway If you go to System/Routing/Gateways, what do you have set as the monitor IP for the primary WAN gateway? By default this is blank and will monitor your modem IP. I would suggest putting in something like google DNS (8.8.8.8 or 8.8.4.4) as a monitoring IP. This would give you a better understanding of your internet connection in the monitoring graphs and logs.



  • @Gertjan said in Intermittent Problems Reaching Anything Beyond pfSense Firewall:

    @TangoOversway said in Intermittent Problems Reaching Anything Beyond pfSense Firewall:

    I tried pinging. I could ping my firewall. I could ping the LAN side of the internet router. I could not ping beyond that

    Can you reach the upstream ISP router ?
    When you experience an outage, what do you see in the
    System logs ?

    Nothing new in the system logs since March 10th.

    DNS logs ?

    At end of text.

    DHCP log ?

    Also at end.

    Both these logs show that the services are extremely active and constantly doing something. Is this normal for a LAN that covers two buildings? Not many computers, but several mobile devices on wifi, plus a number of Sonos speakers and things like blu-ray players, Apple TVs, and so on.

    @TangoOversway said in Intermittent Problems Reaching Anything Beyond pfSense Firewall:

    (I'd forward the DNS requests to Google's name servers, like 8.8.8.8.)

    Not needed, except if you signed a contract with them like "here are all my private DNS requests".
    During testing, switch to default DNS behaviour (works just fine for decades).

    Sorry - in regards to my other note, I am operating on very little sleep and have been careless with my language. I'm using Google's name servers, specified under System/General Setup. Sloppy wording and sloppy way of thinking from exhaustion.

    Would it help if I post a screenshot of DHCP or DNS Resolver settings?

    DNS Resolver Log:

    For recent entries - Is it supposed to be this active? The whole log only covered under 30 minutes, so there was no way of getting data from last night when we were testing everything. And, of course, with all intermittent issues, it's behaving right now.


    Mar 13 15:14:04 pelorimts unbound: [56562:0] info: generate keytag query _ta-4f66. NULL IN
    Mar 13 15:14:04 pelorimts unbound: [56562:0] info: service stopped (unbound 1.9.1).
    Mar 13 15:14:04 pelorimts unbound: [56562:0] info: server stats for thread 0: 2 queries, 0 answers from cache, 2 recursions, 0 prefetch, 0 rejected by ip ratelimiting
    Mar 13 15:14:04 pelorimts unbound: [56562:0] info: server stats for thread 0: requestlist max 6 avg 3 exceeded 0 jostled 0
    Mar 13 15:14:04 pelorimts unbound: [56562:0] info: average recursion processing time 0.260167 sec
    Mar 13 15:14:04 pelorimts unbound: [56562:0] info: histogram of recursion processing times
    Mar 13 15:14:04 pelorimts unbound: [56562:0] info: [25%]=0 median[50%]=0 [75%]=0
    Mar 13 15:14:04 pelorimts unbound: [56562:0] info: lower(secs) upper(secs) recursions
    Mar 13 15:14:04 pelorimts unbound: [56562:0] info: 0.131072 0.262144 1
    Mar 13 15:14:04 pelorimts unbound: [56562:0] info: 0.262144 0.524288 1
    Mar 13 15:14:04 pelorimts unbound: [56562:0] info: server stats for thread 1: 1 queries, 0 answers from cache, 1 recursions, 0 prefetch, 0 rejected by ip ratelimiting
    Mar 13 15:14:04 pelorimts unbound: [56562:0] info: server stats for thread 1: requestlist max 0 avg 0 exceeded 0 jostled 0
    Mar 13 15:14:04 pelorimts unbound: [56562:0] info: average recursion processing time 0.484469 sec
    Mar 13 15:14:04 pelorimts unbound: [56562:0] info: histogram of recursion processing times
    Mar 13 15:14:04 pelorimts unbound: [56562:0] info: [25%]=0 median[50%]=0 [75%]=0
    Mar 13 15:14:04 pelorimts unbound: [56562:0] info: lower(secs) upper(secs) recursions
    Mar 13 15:14:04 pelorimts unbound: [56562:0] info: 0.262144 0.524288 1
    Mar 13 15:14:04 pelorimts unbound: [56562:0] notice: Restart of unbound 1.9.1.
    Mar 13 15:14:04 pelorimts unbound: [56562:0] notice: init module 0: validator
    Mar 13 15:14:04 pelorimts unbound: [56562:0] notice: init module 1: iterator
    Mar 13 15:14:04 pelorimts unbound: [56562:0] info: start of service (unbound 1.9.1).
    Mar 13 15:14:05 pelorimts unbound: [56562:0] info: service stopped (unbound 1.9.1).
    Mar 13 15:14:05 pelorimts unbound: [56562:0] info: server stats for thread 0: 0 queries, 0 answers from cache, 0 recursions, 0 prefetch, 0 rejected by ip ratelimiting
    Mar 13 15:14:05 pelorimts unbound: [56562:0] info: server stats for thread 0: requestlist max 0 avg 0 exceeded 0 jostled 0
    Mar 13 15:14:05 pelorimts unbound: [56562:0] info: server stats for thread 1: 0 queries, 0 answers from cache, 0 recursions, 0 prefetch, 0 rejected by ip ratelimiting
    Mar 13 15:14:05 pelorimts unbound: [56562:0] info: server stats for thread 1: requestlist max 0 avg 0 exceeded 0 jostled 0
    Mar 13 15:14:05 pelorimts unbound: [56562:0] notice: Restart of unbound 1.9.1.
    Mar 13 15:14:05 pelorimts unbound: [56562:0] notice: init module 0: validator
    Mar 13 15:14:05 pelorimts unbound: [56562:0] notice: init module 1: iterator
    Mar 13 15:14:05 pelorimts unbound: [56562:0] info: start of service (unbound 1.9.1).
    Mar 13 15:14:05 pelorimts unbound: [56562:0] info: service stopped (unbound 1.9.1).
    Mar 13 15:14:05 pelorimts unbound: [56562:0] info: server stats for thread 0: 0 queries, 0 answers from cache, 0 recursions, 0 prefetch, 0 rejected by ip ratelimiting
    Mar 13 15:14:05 pelorimts unbound: [56562:0] info: server stats for thread 0: requestlist max 0 avg 0 exceeded 0 jostled 0
    Mar 13 15:14:05 pelorimts unbound: [56562:0] info: server stats for thread 1: 0 queries, 0 answers from cache, 0 recursions, 0 prefetch, 0 rejected by ip ratelimiting
    Mar 13 15:14:05 pelorimts unbound: [56562:0] info: server stats for thread 1: requestlist max 0 avg 0 exceeded 0 jostled 0
    Mar 13 15:14:05 pelorimts unbound: [56562:0] notice: Restart of unbound 1.9.1.
    Mar 13 15:14:05 pelorimts unbound: [56562:0] notice: init module 0: validator
    Mar 13 15:14:05 pelorimts unbound: [56562:0] notice: init module 1: iterator
    Mar 13 15:14:05 pelorimts unbound: [56562:0] info: start of service (unbound 1.9.1).
    Mar 13 15:14:08 pelorimts unbound: [56562:0] info: service stopped (unbound 1.9.1).
    Mar 13 15:14:08 pelorimts unbound: [56562:0] info: server stats for thread 0: 0 queries, 0 answers from cache, 0 recursions, 0 prefetch, 0 rejected by ip ratelimiting
    Mar 13 15:14:08 pelorimts unbound: [56562:0] info: server stats for thread 0: requestlist max 0 avg 0 exceeded 0 jostled 0
    Mar 13 15:14:08 pelorimts unbound: [56562:0] info: server stats for thread 1: 0 queries, 0 answers from cache, 0 recursions, 0 prefetch, 0 rejected by ip ratelimiting
    Mar 13 15:14:08 pelorimts unbound: [56562:0] info: server stats for thread 1: requestlist max 0 avg 0 exceeded 0 jostled 0
    Mar 13 15:14:08 pelorimts unbound: [56562:0] notice: Restart of unbound 1.9.1.
    Mar 13 15:14:08 pelorimts unbound: [56562:0] notice: init module 0: validator
    Mar 13 15:14:08 pelorimts unbound: [56562:0] notice: init module 1: iterator
    Mar 13 15:14:08 pelorimts unbound: [56562:0] info: start of service (unbound 1.9.1).
    Mar 13 15:14:09 pelorimts unbound: [56562:0] info: service stopped (unbound 1.9.1).
    Mar 13 15:14:09 pelorimts unbound: [56562:0] info: server stats for thread 0: 0 queries, 0 answers from cache, 0 recursions, 0 prefetch, 0 rejected by ip ratelimiting
    Mar 13 15:14:09 pelorimts unbound: [56562:0] info: server stats for thread 0: requestlist max 0 avg 0 exceeded 0 jostled 0
    Mar 13 15:14:09 pelorimts unbound: [56562:0] info: server stats for thread 1: 0 queries, 0 answers from cache, 0 recursions, 0 prefetch, 0 rejected by ip ratelimiting
    Mar 13 15:14:09 pelorimts unbound: [56562:0] info: server stats for thread 1: requestlist max 0 avg 0 exceeded 0 jostled 0
    Mar 13 15:14:09 pelorimts unbound: [56562:0] notice: Restart of unbound 1.9.1.
    Mar 13 15:14:09 pelorimts unbound: [56562:0] notice: init module 0: validator
    Mar 13 15:14:09 pelorimts unbound: [56562:0] notice: init module 1: iterator


    DHCP Log:

    Same as with DNS Resolver - so much going on not much time covered in the log. This is more recent.


    Mar 13 15:29:11 pelorimts dhcpleases: Sending HUP signal to dns daemon(87942)
    Mar 13 15:29:12 pelorimts dhcpd: DHCPREQUEST for 172.16.7.233 from c8:e0:eb:56:c0:71 (bagend-2) via mvneta0.4091
    Mar 13 15:29:12 pelorimts dhcpd: DHCPACK on 172.16.7.233 to c8:e0:eb:56:c0:71 (bagend-2) via mvneta0.4091
    Mar 13 15:29:12 pelorimts dhcpd: DHCPREQUEST for 172.16.7.16 from 78:7b:8a:c7:73:ec via mvneta0.4091
    Mar 13 15:29:12 pelorimts dhcpd: DHCPACK on 172.16.7.16 to 78:7b:8a:c7:73:ec via mvneta0.4091
    Mar 13 15:29:13 pelorimts dhcpleases: Sending HUP signal to dns daemon(87942)
    Mar 13 15:29:15 pelorimts dhcpd: DHCPREQUEST for 172.16.7.226 from ac:87:a3:2b:39:f4 (Roseys-iMac-5) via mvneta0.4091
    Mar 13 15:29:15 pelorimts dhcpd: DHCPACK on 172.16.7.226 to ac:87:a3:2b:39:f4 (Roseys-iMac-5) via mvneta0.4091
    Mar 13 15:29:15 pelorimts dhcpleases: Sending HUP signal to dns daemon(87942)
    Mar 13 15:29:16 pelorimts dhcpd: DHCPREQUEST for 172.16.7.224 from b8:09:8a:cc:2c:8b (Roseys-iMac-5) via mvneta0.4091
    Mar 13 15:29:16 pelorimts dhcpd: DHCPACK on 172.16.7.224 to b8:09:8a:cc:2c:8b (Roseys-iMac-5) via mvneta0.4091
    Mar 13 15:29:16 pelorimts dhcpleases: Sending HUP signal to dns daemon(87942)
    Mar 13 15:29:19 pelorimts dhcpd: DHCPREQUEST for 172.16.7.187 from 48:a6:b8:a0:b6:dc (SonosZP) via mvneta0.4091
    Mar 13 15:29:19 pelorimts dhcpd: DHCPACK on 172.16.7.187 to 48:a6:b8:a0:b6:dc (SonosZP) via mvneta0.4091
    Mar 13 15:29:19 pelorimts dhcpleases: Sending HUP signal to dns daemon(87942)
    Mar 13 15:29:21 pelorimts dhcpd: uid lease 172.16.7.238 for client 00:18:dd:07:8f:ec is duplicate on 172.16.7.0/24
    Mar 13 15:29:21 pelorimts dhcpd: DHCPREQUEST for 172.16.7.246 (172.16.7.1) from 00:18:dd:07:8f:ec via mvneta0.4091
    Mar 13 15:29:21 pelorimts dhcpd: DHCPACK on 172.16.7.246 to 00:18:dd:07:8f:ec via mvneta0.4091
    Mar 13 15:29:21 pelorimts dhcpd: DHCPREQUEST for 172.16.7.236 from 78:28:ca:a1:73:4e (SonosZP) via mvneta0.4091
    Mar 13 15:29:21 pelorimts dhcpd: DHCPACK on 172.16.7.236 to 78:28:ca:a1:73:4e (SonosZP) via mvneta0.4091
    Mar 13 15:29:21 pelorimts dhcpd: DHCPREQUEST for 172.16.7.235 from 00:0e:58:fb:ce:a2 (SonosZP) via mvneta0.4091
    Mar 13 15:29:21 pelorimts dhcpd: DHCPACK on 172.16.7.235 to 00:0e:58:fb:ce:a2 (SonosZP) via mvneta0.4091
    Mar 13 15:29:21 pelorimts dhcpleases: Sending HUP signal to dns daemon(87942)
    Mar 13 15:29:21 pelorimts dhcpleases: Sending HUP signal to dns daemon(87942)
    Mar 13 15:29:22 pelorimts dhcpd: DHCPREQUEST for 172.16.7.245 (172.16.7.1) from 00:18:dd:07:8e:71 via mvneta0.4091
    Mar 13 15:29:22 pelorimts dhcpd: DHCPACK on 172.16.7.245 to 00:18:dd:07:8e:71 via mvneta0.4091
    Mar 13 15:29:22 pelorimts dhcpd: DHCPREQUEST for 172.16.7.231 from 00:0e:58:fa:2e:40 (SonosZP) via mvneta0.4091
    Mar 13 15:29:22 pelorimts dhcpd: DHCPACK on 172.16.7.231 to 00:0e:58:fa:2e:40 (SonosZP) via mvneta0.4091
    Mar 13 15:29:22 pelorimts dhcpleases: Sending HUP signal to dns daemon(87942)
    Mar 13 15:29:24 pelorimts dhcpd: DHCPREQUEST for 172.16.7.244 from 00:21:b9:02:53:64 via mvneta0.4091
    Mar 13 15:29:24 pelorimts dhcpd: DHCPACK on 172.16.7.244 to 00:21:b9:02:53:64 via mvneta0.4091
    Mar 13 15:29:29 pelorimts dhcpd: DHCPREQUEST for 172.16.7.241 from e4:95:6e:42:f6:b3 via mvneta0.4091
    Mar 13 15:29:29 pelorimts dhcpd: DHCPACK on 172.16.7.241 to e4:95:6e:42:f6:b3 via mvneta0.4091
    Mar 13 15:29:32 pelorimts dhcpd: DHCPREQUEST for 172.16.7.4 from 84:39:be:6d:d1:7c via mvneta0.4091
    Mar 13 15:29:32 pelorimts dhcpd: DHCPACK on 172.16.7.4 to 84:39:be:6d:d1:7c via mvneta0.4091
    Mar 13 15:29:36 pelorimts dhcpd: DHCPREQUEST for 172.16.7.201 from 90:dd:5d🇩🇪73:f0 (Barn-Cinema-ATV) via mvneta0.4091
    Mar 13 15:29:36 pelorimts dhcpd: DHCPACK on 172.16.7.201 to 90:dd:5d🇩🇪73:f0 (Barn-Cinema-ATV) via mvneta0.4091
    Mar 13 15:29:36 pelorimts dhcpleases: Sending HUP signal to dns daemon(87942)
    Mar 13 15:29:40 pelorimts dhcpd: DHCPREQUEST for 172.16.7.189 from 34:7e:5c:7e:66:a0 (SonosZP) via mvneta0.4091
    Mar 13 15:29:40 pelorimts dhcpd: DHCPACK on 172.16.7.189 to 34:7e:5c:7e:66:a0 (SonosZP) via mvneta0.4091
    Mar 13 15:29:40 pelorimts dhcpleases: Sending HUP signal to dns daemon(87942)
    Mar 13 15:29:41 pelorimts dhcpd: DHCPREQUEST for 172.16.7.185 from 28:ff:3c:b2:3e:4d (BarnBedroomATV) via mvneta0.4091
    Mar 13 15:29:41 pelorimts dhcpd: DHCPACK on 172.16.7.185 to 28:ff:3c:b2:3e:4d (BarnBedroomATV) via mvneta0.4091
    Mar 13 15:29:41 pelorimts dhcpleases: Sending HUP signal to dns daemon(87942)
    Mar 13 15:29:43 pelorimts dhcpd: DHCPREQUEST for 172.16.7.213 from e4:9a:dc:60:3d:d3 (iPhone-Samwise) via mvneta0.4091
    Mar 13 15:29:43 pelorimts dhcpd: DHCPACK on 172.16.7.213 to e4:9a:dc:60:3d:d3 (iPhone-Samwise) via mvneta0.4091
    Mar 13 15:29:43 pelorimts dhcpleases: Sending HUP signal to dns daemon(87942)
    Mar 13 15:29:45 pelorimts dhcpd: DHCPREQUEST for 172.16.7.195 from d8:0d:17:5c:a4:d0 (TL-SG1024DE) via mvneta0.4091
    Mar 13 15:29:45 pelorimts dhcpd: DHCPACK on 172.16.7.195 to d8:0d:17:5c:a4:d0 (TL-SG1024DE) via mvneta0.4091
    Mar 13 15:29:45 pelorimts dhcpd: DHCPREQUEST for 172.16.7.191 from 48:a6:b8:13:1a:9a (SonosZP) via mvneta0.4091
    Mar 13 15:29:45 pelorimts dhcpd: DHCPACK on 172.16.7.191 to 48:a6:b8:13:1a:9a (SonosZP) via mvneta0.4091
    Mar 13 15:29:45 pelorimts dhcpd: DHCPREQUEST for 172.16.7.190 from 78:28:ca:e7:d7:2e (SonosZP) via mvneta0.4091




  • @TangoOversway after you make the change to your monitor IP, keep an eye on your WAN gateway monitoring graphs when the problem comes up.

    This is what those graphs looks like. A tiny bit of packet loss can be ignored like the ones seen in mine, but when you see the issue, the packet loss would increase if your connection to the web is the issue.

    5f111d18-d7fa-4999-809f-8d6add6c878c-image.png



  • @Raffi_ said in Intermittent Problems Reaching Anything Beyond pfSense Firewall:

    @TangoOversway If you go to System/Routing/Gateways, what do you have set as the monitor IP for the primary WAN gateway? By default this is blank and will monitor your modem IP. I would suggest putting in something like google DNS (8.8.8.8 or 8.8.4.4) as a monitoring IP. This would give you a better understanding of your internet connection in the monitoring graphs and logs.

    Here's a screenshot of what I have in System/Routing/Gateways. Odd, but it still has the old firewall name there and I thought that was gone.

    Screen Shot 2020-03-13 at 3.37.50 PM.png

    I have changed the Monitor IP for "WAN_DHCP (default)" to 8.8.8.8

    Could you go more in depth on what you mean about monitoring? I see it mentioned in the docs, but not clearly explained. I would assume that pfSense pings the monitoring IP periodically to monitor the status of the connection and if it's working or down?



  • If you have anything under gateway you don't expect such as the old firewall you're mentioning, then it should probably be removed. Before changing anything though, save all your settings just in case. Go to Diagnostics/Backup & Restore, then hit the download configuration as xml button.

    It may not be needed once you clean up the gateways, but I would still manually change the default IPV4 and IPV6 off of automatic and manually select the gateway you want being used. That removes any ambiguity with which gateway is being used.

    Could you go more in depth on what you mean about monitoring? I see it mentioned in the docs, but not clearly explained. I would assume that pfSense pings the monitoring IP periodically to monitor the status of the connection and if it's working or down?

    Yes, it pings the monitor ip twice a second by default. If the packet loss reaches a certain threshold (10% by default) it will mark the gateway as being down.



  • @Raffi_ said in Intermittent Problems Reaching Anything Beyond pfSense Firewall:

    @TangoOversway after you make the change to your monitor IP, keep an eye on your WAN gateway monitoring graphs when the problem comes up.

    This is what those graphs looks like. A tiny bit of packet loss can be ignored like the ones seen in mine, but when you're see the issue, the packet loss would increase if your connection to the web is the issue.

    I had no idea the interface could be monitored like that. Blame that on lack of knowledge, but I also thought I knew the old system pretty well. I don't know if all this has been added in the past decade, or if I was just always a lot more ignorant of all the things pfSense can do!

    Here's my graph for 24 hours. And over this time, I figure it was about 21:00 last night when my wife and I were running all the tests and couldn't get her computer t pull up much of anything. I don't see much going on at that time. I do see spikes in delays at 5:00, when I was trying to install Linux on a system that had just recently got messed up. I had a lot of problems then, with the Debian installer not being able to download files and I had to restart stages in the install a number of times. Another notable set of peaks about 7:45, which is when I had trouble downloading several files to install Channels DVR. The install script uses curl and wget a number of times to get the files it needs.

    I checked back, just now, in my bash history, and found that it was during that install session, around 5:00 this morning, when I could not ping Google by using the IP address.

    Screen Shot 2020-03-13 at 3.53.12 PM.png



  • The monitoring graph as it is currently doesn't tell us a whole lot other than pfSense was able to ping your modem without issues. Going forward though, since the monitoring IP has been changed to something out on the web it should be more useful.



  • @Raffi_ said in Intermittent Problems Reaching Anything Beyond pfSense Firewall:

    If you have anything under gateway you don't expect such as the old firewall you're mentioning, then it should probably be removed.

    You're not talking physically, right? As in nearby or still hooked up, right? None of that. I did try to change the gateway name, but that's not allowed.

    Before changing anything though, save all your settings just in case. Go to Diagnostics/Backup & Restore, then hit the download configuration as xml button.

    I'm paranoid. I like backups! But thank you - I am spotty and there are things I know well, but I often miss basics.

    It may not be needed once you clean up the gateways, but I would still manually change the default IPV4 and IPV6 off of automatic and manually select the gateway you want being used. That removes any ambiguity with which gateway is being used.

    I'm assuming, by "clean up," you mean changing that name to what I'm using for everything else at this point (like the host name for the firewall) and changing the Monitor IP. But the system won't let me change the name of a gateway. I'm also not clear what you mean by changing them off automatic. Do you mean to specify to use one in particular (like not using DHCP6 at this point)?



  • I'm a little confused by that tiktok gateway. What is that? Is it an interface which provides web access?



  • @Raffi_ said in Intermittent Problems Reaching Anything Beyond pfSense Firewall:

    The monitoring graph as it is currently doesn't tell us a whole lot other than pfSense was able to ping your modem without issues.

    Then is it having delays while I was downloading a lot of data normal, since the interface was in use at that point?

    Going forward though, since the monitoring IP has been changed to something out on the web it should be more useful.

    I'm hoping it gives me something good. I take it that if it shows the gateway as down, or packet losses, or anything like that, that will prove it's the internet router and not something I've done wrong in the firewall?



  • @Raffi_ said in Intermittent Problems Reaching Anything Beyond pfSense Firewall:

    I'm a little confused by that tiktok gateway. What is that? Is it an interface which provides web access?

    tiktok was the host name of the old firewall. I may have typed that in, out of habit, when I set things up. (I used to have an Oz theme to all the names on my LAN, now it's Tolkien.) pfSense won't let me change it and says something about how it can't allow changing gateway names. It's to the LAN. I use the 172.16.7.xxx address space on my LAN. The 192.168.0.xxx space is the no-man's-land zone between the internet router and my firewall. What I don't get is that the router is 192.168.0.1 and serves as the DHCP for that zone (which is only the LAN side of the internet router and the WAN side of the firewall).

    Just to clarify - Internet comes into the wireless router, then the router's LAN interface at 192.168.0.1, then to the firewall's WAN interface at 192.168.0.180, then on the LAN side of the firewall is 172.16.7.1.



  • Then is it having delays while I was downloading a lot of data normal, since the interface was in use at that point?

    I wouldn't worry about that. Those numbers looked fine. If you look at my graph my peak standard deviation was around 4ms even higher than yours. That's not really telling us much. It's definitely not an issue and nothing to worry about.

    I'm hoping it gives me something good. I take it that if it shows the gateway as down, or packet losses, or anything like that, that will prove it's the internet router and not something I've done wrong in the firewall?

    We'll have to wait and see, but basically yes. If you can't ping past the modem then you're modem or something upstream is likely the issue.

    tiktok was the host name of the old firewall. I may have typed that in, out of habit, when I set things up. (I used to have an Oz theme to all the names on my LAN, now it's Tolkien.) pfSense won't let me change it and says something about how it can't allow changing gateway names. It's to the LAN. I use the 172.16.7.xxx address space on my LAN. The 192.168.0.xxx space is the no-man's-land zone between the internet router and my firewall. What I don't get is that the router is 192.168.0.1 and serves as the DHCP for that zone (which is only the LAN side of the internet router and the WAN side of the firewall).

    It sound to me like tiktok should not be a gateway at all. Don't try to rename it, delete it completely from that list. It sounds like that was something that might have been copied over from the old setup but shouldn't be there.



  • @Raffi_

    @Raffi_ said in Intermittent Problems Reaching Anything Beyond pfSense Firewall:

    It sound to me like tiktok should not be a gateway at all. Don't try to rename it, delete it completely from that list. It sounds like that was something that might have been copied over from the old setup but shouldn't be there.

    Quite possible. I don't remember timing, but at some point in the past, I was updating my old pfSense regularly. Then there was some change, I forgot what it was, but instead of just saving the config, then updating it through the web interface, there was more to do. I don't remember what, but I remember part of the issue was that I'd have to make a new image file (and use a serial cable - and mine had busted) and that the config file format had changed or something.

    So when I installed this firewall, I didn't even bother to save the config and load it. Since there was some reason that wouldn't have worked 10 years ago, I didn't even try it now. I just took screenshots of everything and copied it all by hand. I could have just entered 'tiktok' accidentally somewhere without thinking it over.



  • @Raffi_ :

    Here's what I get when I pick related settings for the interfaces. It indicates tiktok is on the LAN interface. Did I set something up wrong or is it just the gateway from the firewall to the LAN?

    Screen Shot 2020-03-13 at 4.45.13 PM.png



  • 172.16.7.1 is your LAN interface? If so, then tiktok should be deleted from Gateways.



  • @Raffi_ said in Intermittent Problems Reaching Anything Beyond pfSense Firewall:

    172.16.7.1 is your LAN interface? If so, then tiktok should be deleted from Gateways.

    Could that be causing problems with stuff being routed incorrectly or something like that? (I'm deleting it - just wondering if it could be part of the issue.)



  • I'm seeing better results now than before. I've also found out that the Google server I was pinging as my way to check on my internet router is not always returning data even when everything else is okay. (I tried pinging several different sites, by domain and IP address, including a different Google server and I found that even when I could ping everything else, sometimes that server I had been using to test was down.)

    With @Raffi_'s help, I've been monitoring the connection with a Monitor IP address and when I have problems, I check the graph and see that there has been no packet loss at all during those times. That makes me think it is the firewall after all, and likely DNS issues, so I'm going to start another thread, since that's an entirely different topic.



  • Could that be causing problems with stuff being routed incorrectly or something like that? (I'm deleting it - just wondering if it could be part of the issue.)

    It could have been part of your problem. Maybe not the source of your problem, but it was definitely a configuration error which needed to be corrected.

    If you're suspecting the issue is a firewall configuration problem, then you might be better off setting up pfSense from scratch using the wizard. The settings right out of the box should work fine. Ask yourself, what specific settings did you copy over from the old firewall that are actually important. Then reconfigure them later if needed. Manually copying settings from a 9 year old firewall could have very well been your problem all along.

    Edit:
    You could still be having issues with that IPV6 gateway if that's not configured right. Why don't you try removing that as well. It might solve your problem.



  • @Raffi_ : I figured that issue out and am having good internet access now. The problem I'm still running into is getting DNS to work on my LAN with the DHCP. I must have copied something wrong from the original firewall when I changed over. Anyway, it comes down to things working well when I stopped using the DNS Resolver. (That's why I think I must have copied something wrong or set it up wrong when I activated DNS on pfSense.)

    So I've posted a new thread here that starts with the DNS issues, overall, and mentions that now the DNS just doesn't seem to want to behave.


  • Netgate Administrator

    @TangoOversway said in Intermittent Problems Reaching Anything Beyond pfSense Firewall:

    172.16.7.1 is your LAN interface? If so, then tiktok should be deleted from Gateways.

    Could that be causing problems with stuff being routed incorrectly or something like that? (I'm deleting it - just wondering if it could be part of the issue.)

    In 2.4.X the system default gateway is set to automatic by default, yours is in the screenshot there. If you have more than one IPv4 gateway and the main WAN fails it will failover to any others available. Here you have the LAN interface itself set as a gateway, if the system sets that as default you will end up with no connectivity.
    You should set the system IPv4 default gateway to WAN_DHCP to avoid that ever happening.

    The only reason you might have a gateway set as the LAN interface is to route traffic over IPSec. This:
    https://docs.netgate.com/pfsense/en/latest/vpn/ipsec/accessing-firewall-services-over-ipsec-vpns.html

    If you were/are not doing that it's probably an invalid gateway.

    Steve



  • @stephenw10 said in Intermittent Problems Reaching Anything Beyond pfSense Firewall:

    @TangoOversway said in Intermittent Problems Reaching Anything Beyond pfSense Firewall:

    172.16.7.1 is your LAN interface? If so, then tiktok should be deleted from Gateways.

    Could that be causing problems with stuff being routed incorrectly or something like that? (I'm deleting it - just wondering if it could be part of the issue.)

    In 2.4.X the system default gateway is set to automatic by default, yours is in the screenshot there. If you have more than one IPv4 gateway and the main WAN fails it will failover to any others available. Here you have the LAN interface itself set as a gateway, if the system sets that as default you will end up with no connectivity.

    Could that have resulted in some kind of intermittent non-connectivity? If so, that might explain a lot. But I don't see how I could have created that gateway accidentally!

    You should set the system IPv4 default gateway to WAN_DHCP to avoid that ever happening.

    I reset to factory defaults and that fixed that and some other issues. But it helps to know what was going wrong, since it was such a puzzle and so frustrating!

    The only reason you might have a gateway set as the LAN interface is to route traffic over IPSec. This:
    https://docs.netgate.com/pfsense/en/latest/vpn/ipsec/accessing-firewall-services-over-ipsec-vpns.html

    I didn't set that up and know I wouldn't have done that at all until I had the basic system working first, so I'm figuring that extra gateway was because I checked a box I didn't mean to or something like that.

    If you were/are not doing that it's probably an invalid gateway.

    If it cause the problems, then it would be nice to know the culprit was found.


  • Netgate Administrator

    The most common way people add a LAN gateway by mistake is if they add a new internal interface in the webgui or they set a new IP address on the existing LAN from the console menu. In both those situations you are presented with an option to add a gateway. There is text guidance explaining that only 'WAN' type interfaces should have a gateway but it's easy to think you are entering the gateway clients should use and add the LAN IP as a gateway. That's incorrect but we see a lot of people do that. 😉
    Only WAN interfaces should have a gateway defined on them directly. That is adding a gateway for the firewall itself not a gateway for clients to use. pfSense uses the presence of a gateway on an interface to identify it as a WAN and will add automatic outbound NAT rules to it.

    Steve


Log in to reply