Always Wan-ip but gateway is 100% packet loss

AcidSleeper

@jrey said in Always Wan-ip but gateway is 100% packet loss:

rebooted what? possible both devices need a power cycle.

MC first, give it time to start up, then pfSense

Done that. Still no change.

from your "can't sleep" post above it appears it was working? (or am I in need of more coffee and reading that wrong)

Yes it was working and have been working but a reboot can make it go away any second.

then let's look at a current set of logs from that sequence.

To bad the logs from that time is long gone. Only 500 rows in log. =/

what's the make/model of the network card in the pfSense box?

Its 4x Intel i225-V 2.5Gbs

AcidSleeper

@jrey I have now attached my asus router to MC and pfsense to asus.

MC --> Asus --> Pfsense.

It seems like it can handle everything I do to it. Reboots, remove wan-cable (pfsense/asus), reboot without wan-cable and attach it when pfsense is up and running.

Every time the gateway goes online and I can browse the internet.

Talked to my ISP and they say the only setting I need to do is DHCP. Nothing else and I even mentioned that Im using pfsense but no settings for pfsense either.

jrey

@AcidSleeper

Interesting
so this the summary

MC - Asus works
MC - pfSense fails
MC - Asus - pfSense works

it's almost like the network card in the pfSense is not compatible with the MC (similar cases have been noted)

do you have a small hub/switch that you can put between the MC and pfSense to test?

AcidSleeper

@jrey Yes I have a dumb switch that I can place in between. Will try that as soon as family leaves me alone with the internet! =)

Thanks yet again.

AcidSleeper

@jrey Hey, tried with a dumb switch (TP-Link SG105) between pfsense and Inteno (MC) but no change. Same behavior. Its like the hardware inside Pfsense cant tolerate (I know, I cant tolerate some people sometimes either) the Inteno relaying DHCP OR my ISP dhcp to Pfsense. I dont know, just throwing out ideas.

Here are som logs anyways:

Logs from General and DHCP - pfsense-random logs.txt
Log from General, show booting, no Internet, pull wan-cabel, plug it back in and got Internet (full functionality - pfsense-Boot-NoInternet-Internet.txt

jrey

@AcidSleeper

So there have been a bunch of previous discussions about the I225-V
needing certain hardware revisions (you can search the forum for those)

setting parameters (
Disable hardware checksum offload
Disable hardware TCP segmentation offload
Disable hardware large receive offload
)

patches etc (all patches I would have thought are in 2.7.1 )

I don't have one and isn't an option for me to emulate, so..

you might want to run this on a command prompt

pciconf -vl

I'm not seeing an error on the icg loads however.
The boot sequence does imply you are getting an IP (.21) in that sequence and gateway (.1)

the .2 and .3 DHCP sequence seems odd, (the sequence at 7:43 is one example)
although it appears you get a OFFER from .2 and .3 after several DHCPNAK from .3 - .2 is always giving you the IP.

AcidSleeper

@jrey said in Always Wan-ip but gateway is 100% packet loss:

@AcidSleeper

So there have been a bunch of previous discussions about the I225-V
needing certain hardware revisions (you can search the forum for those)

setting parameters (
Disable hardware checksum offload
Disable hardware TCP segmentation offload
Disable hardware large receive offload
)

Didnt have "Disable hardware checksum offload" crossed. A friend also said that and I have tried that before but not with a dumb switch. Now it is CHECKED. Will try again with a dumb switch in between Inteno and Pfsense.

patches etc (all patches I would have thought are in 2.7.1 )

I don't have one and isn't an option for me to emulate, so..

I have read much of it too but it seems like the Intel i225-V is working with pfsense 2.7.1, according to what I read. But maybe I am the exception!

you might want to run this on a command prompt
pciconf -vl

hostb0@pci0:0:0:0:	class=0x060000 rev=0x01 hdr=0x00 vendor=0x8086 device=0x9a04 subvendor=0x8086 subdevice=0x7270
    vendor     = 'Intel Corporation'
    class      = bridge
    subclass   = HOST-PCI
vgapci0@pci0:0:2:0:	class=0x030000 rev=0x01 hdr=0x00 vendor=0x8086 device=0x9a78 subvendor=0x8086 subdevice=0x2212
    vendor     = 'Intel Corporation'
    device     = 'Tiger Lake-LP GT2 [UHD Graphics G4]'
    class      = display
    subclass   = VGA
none0@pci0:0:4:0:	class=0x118000 rev=0x01 hdr=0x00 vendor=0x8086 device=0x9a03 subvendor=0x8086 subdevice=0x7270
    vendor     = 'Intel Corporation'
    device     = 'TigerLake-LP Dynamic Tuning Processor Participant'
    class      = dasp
pcib1@pci0:0:6:0:	class=0x060400 rev=0x01 hdr=0x01 vendor=0x8086 device=0x9a09 subvendor=0x8086 subdevice=0x7270
    vendor     = 'Intel Corporation'
    device     = '11th Gen Core Processor PCIe Controller'
    class      = bridge
    subclass   = PCI-PCI
xhci0@pci0:0:13:0:	class=0x0c0330 rev=0x01 hdr=0x00 vendor=0x8086 device=0x9a13 subvendor=0x8086 subdevice=0x7270
    vendor     = 'Intel Corporation'
    device     = 'Tiger Lake-LP Thunderbolt 4 USB Controller'
    class      = serial bus
    subclass   = USB
xhci1@pci0:0:20:0:	class=0x0c0330 rev=0x20 hdr=0x00 vendor=0x8086 device=0xa0ed subvendor=0x8086 subdevice=0x7270
    vendor     = 'Intel Corporation'
    device     = 'Tiger Lake-LP USB 3.2 Gen 2x1 xHCI Host Controller'
    class      = serial bus
    subclass   = USB
none1@pci0:0:20:2:	class=0x050000 rev=0x20 hdr=0x00 vendor=0x8086 device=0xa0ef subvendor=0x8086 subdevice=0x7270
    vendor     = 'Intel Corporation'
    device     = 'Tiger Lake-LP Shared SRAM'
    class      = memory
    subclass   = RAM
none2@pci0:0:22:0:	class=0x078000 rev=0x20 hdr=0x00 vendor=0x8086 device=0xa0e0 subvendor=0x8086 subdevice=0x7270
    vendor     = 'Intel Corporation'
    device     = 'Tiger Lake-LP Management Engine Interface'
    class      = simple comms
ahci0@pci0:0:23:0:	class=0x010601 rev=0x20 hdr=0x00 vendor=0x8086 device=0xa0d3 subvendor=0x8086 subdevice=0x7270
    vendor     = 'Intel Corporation'
    device     = 'Tiger Lake-LP SATA Controller'
    class      = mass storage
    subclass   = SATA
pcib2@pci0:0:28:0:	class=0x060400 rev=0x20 hdr=0x01 vendor=0x8086 device=0xa0bc subvendor=0x8086 subdevice=0x7270
    vendor     = 'Intel Corporation'
    device     = 'Tiger Lake-LP PCI Express Root Port'
    class      = bridge
    subclass   = PCI-PCI
pcib3@pci0:0:28:5:	class=0x060400 rev=0x20 hdr=0x01 vendor=0x8086 device=0xa0bd subvendor=0x8086 subdevice=0x7270
    vendor     = 'Intel Corporation'
    device     = 'Tigerlake PCH-LP PCI Express Root Port'
    class      = bridge
    subclass   = PCI-PCI
pcib4@pci0:0:28:6:	class=0x060400 rev=0x20 hdr=0x01 vendor=0x8086 device=0xa0be subvendor=0x8086 subdevice=0x7270
    vendor     = 'Intel Corporation'
    device     = 'Tiger Lake-LP PCI Express Root Port'
    class      = bridge
    subclass   = PCI-PCI
pcib5@pci0:0:28:7:	class=0x060400 rev=0x20 hdr=0x01 vendor=0x8086 device=0xa0bf subvendor=0x8086 subdevice=0x7270
    vendor     = 'Intel Corporation'
    device     = 'Tiger Lake-LP PCI Express Root Port'
    class      = bridge
    subclass   = PCI-PCI
isab0@pci0:0:31:0:	class=0x060100 rev=0x20 hdr=0x00 vendor=0x8086 device=0xa082 subvendor=0x8086 subdevice=0x7270
    vendor     = 'Intel Corporation'
    device     = 'Tiger Lake-LP LPC Controller'
    class      = bridge
    subclass   = PCI-ISA
hdac0@pci0:0:31:3:	class=0x040300 rev=0x20 hdr=0x00 vendor=0x8086 device=0xa0c8 subvendor=0x10ec subdevice=0x3000
    vendor     = 'Intel Corporation'
    device     = 'Tiger Lake-LP Smart Sound Technology Audio Controller'
    class      = multimedia
    subclass   = HDA
ichsmb0@pci0:0:31:4:	class=0x0c0500 rev=0x20 hdr=0x00 vendor=0x8086 device=0xa0a3 subvendor=0x8086 subdevice=0x7270
    vendor     = 'Intel Corporation'
    device     = 'Tiger Lake-LP SMBus Controller'
    class      = serial bus
    subclass   = SMBus
none3@pci0:0:31:5:	class=0x0c8000 rev=0x20 hdr=0x00 vendor=0x8086 device=0xa0a4 subvendor=0x8086 subdevice=0x7270
    vendor     = 'Intel Corporation'
    device     = 'Tiger Lake-LP SPI Controller'
    class      = serial bus
nvme0@pci0:1:0:0:	class=0x010802 rev=0x03 hdr=0x00 vendor=0x126f device=0x2263 subvendor=0x126f subdevice=0x2263
    vendor     = 'Silicon Motion, Inc.'
    device     = 'SM2263EN/SM2263XT SSD Controller'
    class      = mass storage
    subclass   = NVM
igc0@pci0:2:0:0:	class=0x020000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x15f3 subvendor=0x8086 subdevice=0x0000
    vendor     = 'Intel Corporation'
    device     = 'Ethernet Controller I225-V'
    class      = network
    subclass   = ethernet
igc1@pci0:3:0:0:	class=0x020000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x15f3 subvendor=0x8086 subdevice=0x0000
    vendor     = 'Intel Corporation'
    device     = 'Ethernet Controller I225-V'
    class      = network
    subclass   = ethernet
igc2@pci0:4:0:0:	class=0x020000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x15f3 subvendor=0x8086 subdevice=0x0000
    vendor     = 'Intel Corporation'
    device     = 'Ethernet Controller I225-V'
    class      = network
    subclass   = ethernet
igc3@pci0:5:0:0:	class=0x020000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x15f3 subvendor=0x8086 subdevice=0x0000
    vendor     = 'Intel Corporation'
    device     = 'Ethernet Controller I225-V'
    class      = network
    subclass   = ethernet

I'm not seeing an error on the icg loads however.
The boot sequence does imply you are getting an IP (.21) in that sequence and gateway (.1)

the .2 and .3 DHCP sequence seems odd, (the sequence at 7:43 is one example)
although it appears you get a OFFER from .2 and .3 after several DHCPNAK from .3 - .2 is always giving you the IP.

Did find out the my Asus is using dnsmasq and Pfsense if using Unbound. Just a thought. Can DNS Resolver (Unbound) be the problem that its no configured right or must I use DNS Forwarder?

AcidSleeper

@jrey

jrey

@AcidSleeper

Do you have a DNS Server set on the System -> General Settings page ?
what is it?

as for DNS Resolver it will resolve directly out of the box. No changes required. unbound is great.

Not being able to Resolve would not bring the gateway down. would just mean when the gateway is up you wouldn't be able to find sites by name --- a direct ping to an IP should go through.

The rev=0x03 on the network cards from what I understand regarding that card is a good thing.

I sent you an IM on another item (unrelated to this) earlier - did you see that?

AcidSleeper

@jrey said in Always Wan-ip but gateway is 100% packet loss:

Do you have a DNS Server set on the System -> General Settings page ?
what is it?

jrey

@AcidSleeper said in Always Wan-ip but gateway is 100% packet loss:

General Settings

good

jrey

@AcidSleeper said in Always Wan-ip but gateway is 100% packet loss:

Status > Services
anything showing stopped?

Status -> Gateways (should show you the same as Dashboard)
from there top right corner start or restart the gateway,
if it is running
Screen Shot 2023-12-03 at 12.51.07 PM.png

if is is stopped or pending
Screen Shot 2023-12-03 at 12.51.56 PM.png

what happens?

AcidSleeper

@jrey said in Always Wan-ip but gateway is 100% packet loss:

@AcidSleeper said in Always Wan-ip but gateway is 100% packet loss:

Status > Services
anything showing stopped?

Nothing stopped / pending.

Status -> Gateways (should show you the same as Dashboard)
from there top right corner start or restart the gateway,
if it is running
if is is stopped or pending
what happens?

Did that nothing.

BUT, I ran with my idea that the DNS was faulty somehow. I checked Services -> DNS Resolver and found that the headline "ZONE" had nothing in it. So I stopped it and ran it. Nothing happens when I have 100% packet loss, but if gateways says "Online" but I still cant browse the Internet I checked Services -> Dns Resolver and Zone was still not populating. Restarted and now I can surf.

But still the main problem remains. Why do I get 100% packet loss at Gateway and how to not get it?

johnpoz

@AcidSleeper said in Always Wan-ip but gateway is 100% packet loss:

friend told me to add

Your friend told you to put a any any rule on your firewalls wan address? Good thing you behind another nat router or you would of opened your self up to someone access your pfsense directly..

Advice - don't take networking advice from someone who clearly doesn't have a clue.

Oh your not behind a nat router your IP is 192.121.x.x Yeah you should really remove that any any rule from your wan!!

jrey

@johnpoz
Thanks for taking a look,

from what I can see this is the summary

MC - Asus works
MC - pfSense fails
MC - Asus - pfSense works

from the DHCP snippets of log, it looks like it is being assigned and IP and gateway, and it varies from connection to connection.

I'm still not convinced it isn't somehow MAC related. but the OP says he tried that. Not sure both devices where power cycled at that point. Fibre connection.

The ISP appears to have two DHCP servers responding sometime the address comes from the .2 and others times from the .3

I was just going to suggest that the OP make sure the Named gateway is selected on system Routing Gateways, only to have the system Save the config again.

jrey

@AcidSleeper said in Always Wan-ip but gateway is 100% packet loss:

but if gateways says "Online" but I still cant browse the Internet I checked Services -> Dns Resolver and Zone was still not populating. Restarted and now I can surf.

sounds like 2 issues when you are in this state.

a) so you can get "online" and it stays that way ? or does it just go away on it's own.

then when you are "ONLINE"
b) DNS issue (because you can't browse).

The lack of a running DNS when ONLINE just means you can't browse, but your monitor should still ping the IP.

So here I have turned the resolver off,
Status -> services
unbound isn't even listed.
( Screen Shot 2023-12-04 at 6.54.40 AM.png

the gateway and monitor IP are still ONLINE and the remote IP (in this case I am monitoring 1.1.1.1) is reachable
Screen Shot 2023-12-04 at 6.55.51 AM.png

but I can't browse it will timeout and tell me it can't get there.
Screen Shot 2023-12-04 at 6.58.35 AM.png

So the gateway being up / and online has nothing to do with ability to browse. Of course if it is OFFLINE, the second problem doesn't matter for external browsing.

if you can't reliably get and keep a gateway, that's 1 thing (the first problem)
once you have an online gateway that stays up/online, and you can't browse, that's the second problem.

Step 1 get to the point where your gateway is online and stays that way. Are we there?

Step 2 fix the DNS (if it is broken) out of the box (DNS Resolver) should just resolve. On the Resolver screen you have shown above, uncheck DNSSEC and check Enable Forwarding Mode. You already have DNS servers listed on your General Settings page so when online that's who it should forward to, instead of root servers. See if that changes anything. (of course your gateway has to be online/stable first)

Sorjal

One thing that I've encountered with my ISP is that their gateways tend to drop pings, etc. when they become congested and after a recent "upgrade" they're basically dropping all icmp, pings, almost 100% of the time. I've done a few things.

First in each Gateway (I have 1 IPv4 and 1 IPv6) I set up a different IP in each in the area "Monitor IP". For me I used two of Cloudflare's DNS addresses as they're pretty much always up (1.1.1.1 and 2606:4700:4700::1111). This makes the gateway monitoring a little more accurate as sometimes say the icmp request will make it through the ISP's gateway to the monitor IP and it will return those statistics. Still, as above, the gateways can be so bad that even requests to different addresses never make it there and back in time.

More importantly I checked off the box next to "Gateway Action" to 'disable gateway monitoring action' so that if their stops responding to echo requests, the gateway is still considered up and pfsense doesn't take any actions such as Kill states on gateway failure. You could instead also try changing that option to 'Do no kill states on gateway failure'.

I'm not savvy enough to know a better and more accurate way of interacting with my ISP's gateways (Comcast/Xfinity), this just cuts down on additional down time when the connection is congested but still up.

AcidSleeper

@jrey said in Always Wan-ip but gateway is 100% packet loss:

@johnpoz
Thanks for taking a look,

from what I can see this is the summary

MC - Asus works
MC - pfSense fails
MC - Asus - pfSense works

That is correct! and where MC- Pfsense meet there is 3 occations:

Online (everything work):
- Here Im happy, but worring if next restart will be offline.
Online (ping works on ip, browsing dont work):
- Have figured out that if I stop Unbound and start it again I can browse.
Offline (100% packet loss, nothing works)
- Here is the real dilemma. It seems like its only luck that desides if I get a Online (1 or 2) on the gateway.

I was just going to suggest that the OP make sure the Named gateway is selected on system Routing Gateways, only to have the system Save the config again.

Checked the settings and my gateway is selected as default.

from the DHCP snippets of log, it looks like it is being assigned and IP and gateway, and it varies from connection to connection.

I'm still not convinced it isn't somehow MAC related. but the OP says he tried that. Not sure both devices where power cycled at that point. Fibre connection.

DNSSeq is also turned OFF as off today.

I have now tested Mac-spoofing again:
I copied my Asus mac-adress in Pfsense and saved. I halted Pfsense. Pfsense not plugged into MC yet. I let the MC be turned off a couple of minutes then started it and let it sit without any device connected to it. Then I plugged in the pfsense and started pfsense.

I got Offline 100% packet loss. So I did this:

Status -> Interfaces: Checked Relinquish Lease and pressed Release Wan.
Diagnostics -> Command Prompt: ifconfig eth0 down. Waited 30seconds. Wrote ifconfig eth0 up.
Status -> Interfaces: Looked at it and I was getting an IP but In/out packets (block) was in numbers equal to In/out packets (pass). Tried to browse Internet but no browsing possible.
Status -> Services and stopped Unbound and started it back up after a few seconds. Browsing works.

Even if this works its a pretty long way to go just to start pfsense.

Now I have restarted pfsense 2 times and its up and running on both occassions.
Now I unplugged the wan-cabel between Pfsense and MC. Result:

Plugg back the wan-cable and everything starts back up. Everything works.

Must try another reboot. Nope, Offline with 100% pack loss.

The logs for this:
pfsense-general-log231205.txt
pfsense-gateways-log231205.txt
pfsense-resolver-log231205.txt
pfsense-dhcp-log231205.txt

With the gateway offline I did what I did above:

Status -> Interfaces: Checked Relinquish Lease and pressed Release Wan.
Diagnostics -> Command Prompt: ifconfig eth0 down. Waited 30seconds. Wrote ifconfig eth0 up.
Status -> Interfaces: Looked at it and I was getting an IP but In/out packets (block) was in numbers equal to In/out packets (pass). Tried to browse Internet but no browsing possible.
Status -> Services and stopped Unbound and started it back up after a few seconds. Browsing works.

Then it works again.

AcidSleeper

@Sorjal said in Always Wan-ip but gateway is 100% packet loss:

First in each Gateway (I have 1 IPv4 and 1 IPv6) I set up a different IP in each in the area "Monitor IP". For me I used two of Cloudflare's DNS addresses as they're pretty much always up (1.1.1.1 and 2606:4700:4700::1111). This makes the gateway monitoring a little more accurate as sometimes say the icmp request will make it through the ISP's gateway to the monitor IP and it will return those statistics. Still, as above, the gateways can be so bad that even requests to different addresses never make it there and back in time.

I only have IPv4 and I have added 8.8.8.8 to gateway monitoring.

More importantly I checked off the box next to "Gateway Action" to 'disable gateway monitoring action' so that if their stops responding to echo requests, the gateway is still considered up and pfsense doesn't take any actions such as Kill states on gateway failure.

I havent this box checked but I try anything. Will wait to do that until my answear to @jrey is looked at, so I dont mess up anything.

You could instead also try changing that option to 'Do no kill states on gateway failure'.

My settings:

AcidSleeper

A idea from a swedish forum.

What if there´s a problem when the link-speed is to be negotiated between MC and pfSense?

MC (Inteno) maximum 1GbE and PFsense 2.5GbE

Should I set the Speed and Duplex in Inferface -> Wan?