Super Confused - LAN Gateway

Jarhead

@bearhntr How can it have internet when it's not connected???
Read my previous post.

rcoleman-netgate

Windows says it doesn't have an connection... but that is because they use pings and DNS lookups to verify the connectivity.

Do other devices report similar things? Non-Windows, if you have any (tablets, phones, etc.)

Also your WAN needs a gateway:

viragomann

@bearhntr said in Super Confused - LAN Gateway:

Ping 8.8.8.8 from where?

From the Windows VM, of course.
Failing internet access on the Windows VM is the only one issue you've reported in your first post.

bearhntr

@rcoleman-netgate

I cannot make any sense out of this at all.

I have just RESET the VM pfSense back to factory defaults. I during the setup, - WAN set to Static 10.10.1.2/24. LAN set to static 10.9.28.254/24 and DHCP server enabled.

On my working pfSense OPT1 is set to 10.10.1.1/24 and there is a cable from there to the WAN port on the VM host. There is no cable now in the LAN port on the VM host - as all of the LAN testing I am doing is from a VM on the same back using virtual bridge to LAN that pfSense is using. The Windows machine will pull an IP Address and ask me to identify the network - I choose HOME (Windows 7 = Private in Windows 10).

I have attempted to set a GATEWAY on the WAN - and I chose 10.10.1.1 (which is the port on the working pfSense for OPT 1). What does this need to be? One would think that if PING is failing between the two - nothing else it gonna work either.

I then go to working pfSense Diagnostics and ping 10.10.1.2 - get no response (100% fail). but I can ping and resolve all day long anything on the Internet and on my working LAN network from that box 192.168.10.xxx/24.

This should not be this hard - I am no idiot when it comes to networking - but this is making me re-think that.

rcoleman-netgate

@bearhntr said in Super Confused - LAN Gateway:

I have attempted to set a GATEWAY on the WAN - and I chose 10.10.1.1

Your WAN gateway should be the IP address of the next device upstream. If you don't know what that is set your WAN to DHCP.

Jarhead

@rcoleman-netgate said in Super Confused - LAN Gateway:

@jarhead

Windows says it doesn't have an connection... but that is because they use pings and DNS lookups to verify the connectivity.

Do other devices report similar things? Non-Windows, if you have any (tablets, phones, etc.)

Also your WAN needs a gateway:

No, the WAN of the VM is not connected. If you read several posts back you'll see he can't ping either pfSense from the other. Probably a virtual switch problem but he doesn't want to answer any questions so it's impossible to help him.

bearhntr

@viragomann

The Windows VM can ping itself and the gateway 10.9.28.254 -- nothing else.

Jarhead

@bearhntr
Do this.
Set a pc to 10.10.1.2/30, connect it to the OPT port, can you ping 10.10.1.1?

Then set that pc to 10.10.1.1/30 and connect it to the VM WAN. Can you ping it 10.10.1.2?

You'll see you can ping the existing pfSense but you won't ping the VM. FIX THAT.

bearhntr

@jarhead said in Super Confused - LAN Gateway:

@rcoleman-netgate said in Super Confused - LAN Gateway:

No, the WAN of the VM is not connected. If you read several posts back you'll see he can't ping either pfSense from the other. Probably a virtual switch problem but he doesn't want to answer any questions so it's impossible to help him.

I do not know how you state that I am not answering your questions. I did:

They're not connected.
Are you using a virtual switch?
How are you connecting the two routers?
Is the pc you were connecting to the VM a physical machine? If so, disconnect it and use that cable to connect to OPT on router 1. Does it ping that way?

I gave you this:


The PF2 (will be 10.9.28.254/24) is the new one on the Proxmox. There are 5 ports on this box (on-board NIC is the console port for Proxmox and is set to 192.168.10.250/24 (this will change once I get 10.9.28.xxx/24 working) and connects to one port on the ORBI. The 4-port card in the PCIe slot is as follows:

*port 0 = (to be the new WAN - is vmbr1 (Linux Virtual Bridge) to this port {I have another posting to see if this should be virtualized or or IOMMU PCI port into pfSense VM.

port 1= (is to be the new LAN - is vmbr2 (Linux Virtual Bridge) to this port.*

That leaves me with 2 ports not in use.

From the LAN port on the Proxmox - I have a cable plugged into a hub, in turn from there another cable in to the OPT1 port on the PF1 box (which is static 10.9.28.250/24) - have even tried a cable directly from OPT1 to PF2-LAN made no difference. I put the HUB there in case I wanted to plug a laptop in there to test as well. When I get his working - the HP T620+ will be OFF and stored incase I need a replacement some day.

The Proxmox is setup to use VirtIO Paravirualized ports (bridged in Proxmox to the native ports)

bearhntr

@rcoleman-netgate said in Super Confused - LAN Gateway:

@bearhntr said in Super Confused - LAN Gateway:

I have attempted to set a GATEWAY on the WAN - and I chose 10.10.1.1

Your WAN gateway should be the IP address of the next device upstream. If you don't know what that is set your WAN to DHCP.

So the next device upstream would be the pfSense (calling it PF1) that is my working pfSense box on the HP T620 ThinClient - it there is a cable from its on-board NIC (set as OPT1 in PF1) - and static at 10.10.1.1/24

When I set that - I get this in PF2 (the VM)

FIREWALL RULES on (PF1 - OPT1)

FIREWALL RULES on (PF2 - VM)

chpalmer

@bearhntr can you access the internet from a laptop plugged into opt1 of pf1?

I can find nowhere that this has been answered..

Jarhead

@bearhntr said in Super Confused - LAN Gateway:

@jarhead said in Super Confused - LAN Gateway:
I do not know how you state that I am not answering your questions. I did:

I asked this:

@jarhead said in Super Confused - LAN Gateway:

@bearhntr What's the LANGW and where did you add it?
Should be a WAN gateway, not LAN

No answer.

I asked again.

@jarhead said in Super Confused - LAN Gateway:

@bearhntr
I'm asking you what the LANGW is.
You shouldn't add a gateway on the LAN, so leave it at none as in the picture you posted. But the question stands, what are you considering LANGW??

No answer.

I asked this:

@jarhead said in Super Confused - LAN Gateway:

@bearhntr
From orig pfSense, can you ping the new vm pfSense 10.9.28.254?

No answer.

@jarhead said in Super Confused - LAN Gateway:

@bearhntr said in Super Confused - LAN Gateway:
They're not connected.
Are you using a virtual switch?
How are you connecting the two routers?
Is the pc you were connecting to the VM a physical machine? If so, disconnect it and use that cable to connect to OPT on router 1. Does it ping that way?

No answer.

Again.

@jarhead said in Super Confused - LAN Gateway:

@bearhntr
So, again, they aren't connected. Fix that first.

Set a pc to 10.10.1.2/30, connect it to the OPT port, can you ping 10.10.1.1?

Then set that pc to 10.10.1.1/30 and connect it to the VM WAN. Can you ping it 10.10.1.2?

No answer.

This is very simple. You have a problem with the VM but you refuse to acknowledge that.

Try the ping I suggested and you'll see it's not connecting.

Can you connect any physical machine to the VM at all?

bearhntr

@jarhead said in Super Confused - LAN Gateway:

@bearhntr
Do this.
Set a pc to 10.10.1.2/30, connect it to the OPT port, can you ping 10.10.1.1?

Then set that pc to 10.10.1.1/30 and connect it to the VM WAN. Can you ping it 10.10.1.2?

You'll see you can ping the existing pfSense but you won't ping the VM. FIX THAT.

OK - I did this, very difficult as there is no space for another PC where this box is.

Set the PC to 10.10.1.2/30 and plugged cable into OPT1 on PF1 - it immediately connected to the Internet and PING was successful.

Move the wire to the WAN port on the PF2 - set PC to 10.10.1.1/30 and PING fails.

As I am thinking given all the errors that I am seeing in the 'dmesg' in Proxmox Shell - that the 4-port card is bad or has something wrong with it.

[ 1248.474520] pcieport 0000:00:1d.0: AER: Multiple Corrected error received: 0000:00:1d.0
[ 1248.524181] pcieport 0000:00:1d.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, (Receiver ID)
[ 1248.524207] pcieport 0000:00:1d.0:   device [8086:a118] error status/mask=00002001/00002000
[ 1248.524231] pcieport 0000:00:1d.0:    [ 0] RxErr                 
[ 1248.667371] pcieport 0000:00:1d.0: AER: Multiple Corrected error received: 0000:00:1d.0
[ 1248.691962] pcieport 0000:00:1d.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, (Receiver ID)
[ 1248.691989] pcieport 0000:00:1d.0:   device [8086:a118] error status/mask=00000001/00002000
[ 1248.692011] pcieport 0000:00:1d.0:    [ 0] RxErr                 
[ 1252.456633] pcieport 0000:00:1d.0: AER: Multiple Corrected error received: 0000:00:1d.0
[ 1252.456677] pcieport 0000:00:1d.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, (Receiver ID)
[ 1252.456703] pcieport 0000:00:1d.0:   device [8086:a118] error status/mask=00002001/00002000
[ 1252.456725] pcieport 0000:00:1d.0:    [ 0] RxErr                 
[ 1260.319756] tg3 0000:01:00.1 enp1s0f1: Link is down
[ 1260.319878] vmbr2: port 1(enp1s0f1) entered disabled state
[ 1299.343586] pcieport 0000:00:1d.0: AER: Multiple Corrected error received: 0000:00:1d.0
[ 1299.392764] pcieport 0000:00:1d.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, (Receiver ID)
[ 1299.392790] pcieport 0000:00:1d.0:   device [8086:a118] error status/mask=00002001/00002000
[ 1299.392814] pcieport 0000:00:1d.0:    [ 0] RxErr                 
[ 1299.486874] pcieport 0000:00:1d.0: AER: Multiple Corrected error received: 0000:00:1d.0
[ 1299.535945] pcieport 0000:00:1d.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, (Receiver ID)
[ 1299.535970] pcieport 0000:00:1d.0:   device [8086:a118] error status/mask=00002001/00002000
[ 1299.535994] pcieport 0000:00:1d.0:    [ 0] RxErr                 
[ 1373.798280] pcieport 0000:00:1d.0: AER: Multiple Corrected error received: 0000:00:1d.0
[ 1373.822409] pcieport 0000:00:1d.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, (Receiver ID)
[ 1373.822435] pcieport 0000:00:1d.0:   device [8086:a118] error status/mask=00000001/00002000
[ 1373.822458] pcieport 0000:00:1d.0:    [ 0] RxErr                 
[ 1376.440381] pcieport 0000:00:1d.0: AER: Multiple Corrected error received: 0000:00:1d.0
[ 1376.489879] pcieport 0000:00:1d.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, (Receiver ID)
[ 1376.489905] pcieport 0000:00:1d.0:   device [8086:a118] error status/mask=00002001/00002000
[ 1376.489928] pcieport 0000:00:1d.0:    [ 0] RxErr

I will have to wait for the new card that I ordered (which I was advised would be better for virtualization -- INTEL i350 T4V2) to get here mid-week.

I appreciate all the group-head-banging....as this was a strange one. It made no sense as I have had Proxmox setup with pfSense before, on a differnt box with a 2-port PCIe NIC card - and had no problems. I got this new box, as the motherboard on that other one died and I could not get a replacement.

For the moment - the PF1 box is working and I will leave things be.

Again - thanks for all the extra brain-cells, as mine were about to take a Christmas Vacation.

viragomann

@bearhntr
Did you disable "Hardware Checksum Offloading" in System > Advanced > Networking?
If not add a check there.

bearhntr

@viragomann said in Super Confused - LAN Gateway:

@bearhntr
Did you disable "Hardware Checksum Offloading" in System > Advanced > Networking?
If not add a check there.

I did do this - even remembered all 3 times I FACTORY RESET it.

I am thinking that I have a bad 4-port card.

viragomann

@bearhntr
The error above when pinging IPs outside of the Windows subnet indicates a failure on Windows itself for me.
If it would get no response due to blocking or miss-routing on another network device you would get a simple timeout.

What virtual network cards are you using?
What does an "ipconfig /all" show?
Also post the ARP table, please.

bearhntr

@viragomann said in Super Confused - LAN Gateway:

@bearhntr
The error above when pinging IPs outside of the Windows subnet indicates a failure on Windows itself for me.
If it would get no response due to blocking or miss-routing on another network device you would get a simple timeout.

What virtual network cards are you using?
What does an "ipconfig /all" show?
Also post the ARP table, please.

I have just blown away the PROXMOX installation and attempting to install DEBIAN 11 and UBUNTU 20.04 both fail when I tell them to use any of the 4-ports on that card. While they act like they are working - they actually get no IP Address and cannot connect to the Internet.

I am just gonna scrap that card and wait for the new INTEL i250-T4V2 gets here mid-week.

Thanks again everyone for your assistance. Just ODD that the card in another PC with Windows 2019 Server installed works just fine.

bearhntr

**** UPDATE ****

I found that there was a new FIRMWARE update for the HP 331T-Quad card, and I applied it. It is like 2 versions higher now than where it was.

Did a brand new install of Proxmox, and enabled the IOMMU settings.

Still seeing the PCI Bus errors, and found an article to add 'pci=nommconf' to the loader.conf section.

Been running more than 2 hours - no PCI Bus errors.

So I built a new pfSense VM and added the PCI directly for 0000:01:00.0 and 0000:01:00.1 as two NICs to the VM (direct access not virtualized ports).

Software installs - but only one Interface is seen (not 2)....and just repeated errors about a Firmware Link - over and over and over.

Oh Well - according to UPS, my INTEL i5-T4V2 card should be here tomorrow.