Super Confused - LAN Gateway

bearhntr

The PF1 (192.168.10.254/24) is an HP T620+ ThinClient with a 2-port NIC installed in the expansion slot. The built in NIC is used for OPT1, and the port 0 on the 2-port card is WAN to my cable modem, port 1 is LAN to my Wireless AP (Netgear ORBI).

The PF2 (will be 10.9.28.254/24) is the new one on the Proxmox. There are 5 ports on this box (on-board NIC is the console port for Proxmox and is set to 192.168.10.250/24 (this will change once I get 10.9.28.xxx/24 working) and connects to one port on the ORBI. The 4-port card in the PCIe slot is as follows:

*port 0 = (to be the new WAN - is vmbr1 (Linux Virtual Bridge) to this port {I have another posting to see if this should be virtualized or or IOMMU PCI port into pfSense VM.

port 1= (is to be the new LAN - is vmbr2 (Linux Virtual Bridge) to this port.*

That leaves me with 2 ports not in use.

From the LAN port on the Proxmox - I have a cable plugged into a hub, in turn from there another cable in to the OPT1 port on the PF1 box (which is static 10.9.28.250/24) - have even tried a cable directly from OPT1 to PF2-LAN made no difference. I put the HUB there in case I wanted to plug a laptop in there to test as well. When I get his working - the HP T620+ will be OFF and stored incase I need a replacement some day.

See if this helps:

bearhntr

I am wondering if I have a bad or crazed network card.

I am getting tons of these in the Proxmox SHELL - running 'dmesg'

[ 1248.474520] pcieport 0000:00:1d.0: AER: Multiple Corrected error received: 0000:00:1d.0
[ 1248.524181] pcieport 0000:00:1d.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, (Receiver ID)
[ 1248.524207] pcieport 0000:00:1d.0:   device [8086:a118] error status/mask=00002001/00002000
[ 1248.524231] pcieport 0000:00:1d.0:    [ 0] RxErr                 
[ 1248.667371] pcieport 0000:00:1d.0: AER: Multiple Corrected error received: 0000:00:1d.0
[ 1248.691962] pcieport 0000:00:1d.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, (Receiver ID)
[ 1248.691989] pcieport 0000:00:1d.0:   device [8086:a118] error status/mask=00000001/00002000
[ 1248.692011] pcieport 0000:00:1d.0:    [ 0] RxErr                 
[ 1252.456633] pcieport 0000:00:1d.0: AER: Multiple Corrected error received: 0000:00:1d.0
[ 1252.456677] pcieport 0000:00:1d.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, (Receiver ID)
[ 1252.456703] pcieport 0000:00:1d.0:   device [8086:a118] error status/mask=00002001/00002000
[ 1252.456725] pcieport 0000:00:1d.0:    [ 0] RxErr                 
[ 1260.319756] tg3 0000:01:00.1 enp1s0f1: Link is down
[ 1260.319878] vmbr2: port 1(enp1s0f1) entered disabled state
[ 1299.343586] pcieport 0000:00:1d.0: AER: Multiple Corrected error received: 0000:00:1d.0
[ 1299.392764] pcieport 0000:00:1d.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, (Receiver ID)
[ 1299.392790] pcieport 0000:00:1d.0:   device [8086:a118] error status/mask=00002001/00002000
[ 1299.392814] pcieport 0000:00:1d.0:    [ 0] RxErr                 
[ 1299.486874] pcieport 0000:00:1d.0: AER: Multiple Corrected error received: 0000:00:1d.0
[ 1299.535945] pcieport 0000:00:1d.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, (Receiver ID)
[ 1299.535970] pcieport 0000:00:1d.0:   device [8086:a118] error status/mask=00002001/00002000
[ 1299.535994] pcieport 0000:00:1d.0:    [ 0] RxErr                 
[ 1373.798280] pcieport 0000:00:1d.0: AER: Multiple Corrected error received: 0000:00:1d.0
[ 1373.822409] pcieport 0000:00:1d.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, (Receiver ID)
[ 1373.822435] pcieport 0000:00:1d.0:   device [8086:a118] error status/mask=00000001/00002000
[ 1373.822458] pcieport 0000:00:1d.0:    [ 0] RxErr                 
[ 1376.440381] pcieport 0000:00:1d.0: AER: Multiple Corrected error received: 0000:00:1d.0
[ 1376.489879] pcieport 0000:00:1d.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, (Receiver ID)
[ 1376.489905] pcieport 0000:00:1d.0:   device [8086:a118] error status/mask=00002001/00002000
[ 1376.489928] pcieport 0000:00:1d.0:    [ 0] RxErr

bearhntr

This post is deleted!

bearhntr

This post is deleted!

bearhntr

@jarhead said in Super Confused - LAN Gateway:

@bearhntr Just do this.
Make the OPT 10.10.1.1/30
Make the VM WAN 10.10.1.2/30
Connect the two. Make sure to uncheck block private networks on VM WAN.
You'll now have internet on the VM.
You can allow the original LAN through the VM firewall if you want or just configure it from the VM LAN.

I did as you suggested -- not only is the Web Interface even slower now -- I also get this when I go to PF1 and ping PF2 (WAN Address)

Jarhead

@bearhntr
So, again, they aren't connected. Fix that first.

Set a pc to 10.10.1.2/30, connect it to the OPT port, can you ping 10.10.1.1?

Then set that pc to 10.10.1.1/30 and connect it to the VM WAN. Can you ping it 10.10.1.2?

bearhntr

@jarhead

I have reset the network on the Win7-VM and rebooted - it is pulling a DHCP Address from the PF2 - but still has no INTERNET.

the RULES for OPT1 (on the PF1)

From PF2 (VM) --- WAN

From PF2 (VM) --- LAN

FIREWALL - PF2

Jarhead

@bearhntr How can it have internet when it's not connected???
Read my previous post.

rcoleman-netgate

@jarhead

Windows says it doesn't have an connection... but that is because they use pings and DNS lookups to verify the connectivity.

Do other devices report similar things? Non-Windows, if you have any (tablets, phones, etc.)

Also your WAN needs a gateway:

viragomann

@bearhntr said in Super Confused - LAN Gateway:

Ping 8.8.8.8 from where?

From the Windows VM, of course.
Failing internet access on the Windows VM is the only one issue you've reported in your first post.

bearhntr

@rcoleman-netgate

I cannot make any sense out of this at all.

I have just RESET the VM pfSense back to factory defaults. I during the setup, - WAN set to Static 10.10.1.2/24. LAN set to static 10.9.28.254/24 and DHCP server enabled.

On my working pfSense OPT1 is set to 10.10.1.1/24 and there is a cable from there to the WAN port on the VM host. There is no cable now in the LAN port on the VM host - as all of the LAN testing I am doing is from a VM on the same back using virtual bridge to LAN that pfSense is using. The Windows machine will pull an IP Address and ask me to identify the network - I choose HOME (Windows 7 = Private in Windows 10).

I have attempted to set a GATEWAY on the WAN - and I chose 10.10.1.1 (which is the port on the working pfSense for OPT 1). What does this need to be? One would think that if PING is failing between the two - nothing else it gonna work either.

I then go to working pfSense Diagnostics and ping 10.10.1.2 - get no response (100% fail). but I can ping and resolve all day long anything on the Internet and on my working LAN network from that box 192.168.10.xxx/24.

This should not be this hard - I am no idiot when it comes to networking - but this is making me re-think that.

rcoleman-netgate

@bearhntr said in Super Confused - LAN Gateway:

I have attempted to set a GATEWAY on the WAN - and I chose 10.10.1.1

Your WAN gateway should be the IP address of the next device upstream. If you don't know what that is set your WAN to DHCP.

Jarhead

@rcoleman-netgate said in Super Confused - LAN Gateway:

@jarhead

Windows says it doesn't have an connection... but that is because they use pings and DNS lookups to verify the connectivity.

Do other devices report similar things? Non-Windows, if you have any (tablets, phones, etc.)

Also your WAN needs a gateway:

No, the WAN of the VM is not connected. If you read several posts back you'll see he can't ping either pfSense from the other. Probably a virtual switch problem but he doesn't want to answer any questions so it's impossible to help him.

bearhntr

@viragomann

The Windows VM can ping itself and the gateway 10.9.28.254 -- nothing else.

Jarhead

@bearhntr
Do this.
Set a pc to 10.10.1.2/30, connect it to the OPT port, can you ping 10.10.1.1?

Then set that pc to 10.10.1.1/30 and connect it to the VM WAN. Can you ping it 10.10.1.2?

You'll see you can ping the existing pfSense but you won't ping the VM. FIX THAT.

bearhntr

@jarhead said in Super Confused - LAN Gateway:

@rcoleman-netgate said in Super Confused - LAN Gateway:

No, the WAN of the VM is not connected. If you read several posts back you'll see he can't ping either pfSense from the other. Probably a virtual switch problem but he doesn't want to answer any questions so it's impossible to help him.

I do not know how you state that I am not answering your questions. I did:

They're not connected.
Are you using a virtual switch?
How are you connecting the two routers?
Is the pc you were connecting to the VM a physical machine? If so, disconnect it and use that cable to connect to OPT on router 1. Does it ping that way?

I gave you this:


The PF2 (will be 10.9.28.254/24) is the new one on the Proxmox. There are 5 ports on this box (on-board NIC is the console port for Proxmox and is set to 192.168.10.250/24 (this will change once I get 10.9.28.xxx/24 working) and connects to one port on the ORBI. The 4-port card in the PCIe slot is as follows:

*port 0 = (to be the new WAN - is vmbr1 (Linux Virtual Bridge) to this port {I have another posting to see if this should be virtualized or or IOMMU PCI port into pfSense VM.

port 1= (is to be the new LAN - is vmbr2 (Linux Virtual Bridge) to this port.*

That leaves me with 2 ports not in use.

From the LAN port on the Proxmox - I have a cable plugged into a hub, in turn from there another cable in to the OPT1 port on the PF1 box (which is static 10.9.28.250/24) - have even tried a cable directly from OPT1 to PF2-LAN made no difference. I put the HUB there in case I wanted to plug a laptop in there to test as well. When I get his working - the HP T620+ will be OFF and stored incase I need a replacement some day.

The Proxmox is setup to use VirtIO Paravirualized ports (bridged in Proxmox to the native ports)

bearhntr

@rcoleman-netgate said in Super Confused - LAN Gateway:

@bearhntr said in Super Confused - LAN Gateway:

I have attempted to set a GATEWAY on the WAN - and I chose 10.10.1.1

Your WAN gateway should be the IP address of the next device upstream. If you don't know what that is set your WAN to DHCP.

So the next device upstream would be the pfSense (calling it PF1) that is my working pfSense box on the HP T620 ThinClient - it there is a cable from its on-board NIC (set as OPT1 in PF1) - and static at 10.10.1.1/24

When I set that - I get this in PF2 (the VM)

FIREWALL RULES on (PF1 - OPT1)

FIREWALL RULES on (PF2 - VM)

chpalmer

@bearhntr can you access the internet from a laptop plugged into opt1 of pf1?

I can find nowhere that this has been answered..

Jarhead

@bearhntr said in Super Confused - LAN Gateway:

@jarhead said in Super Confused - LAN Gateway:
I do not know how you state that I am not answering your questions. I did:

I asked this:

@jarhead said in Super Confused - LAN Gateway:

@bearhntr What's the LANGW and where did you add it?
Should be a WAN gateway, not LAN

No answer.

I asked again.

@jarhead said in Super Confused - LAN Gateway:

@bearhntr
I'm asking you what the LANGW is.
You shouldn't add a gateway on the LAN, so leave it at none as in the picture you posted. But the question stands, what are you considering LANGW??

No answer.

I asked this:

@jarhead said in Super Confused - LAN Gateway:

@bearhntr
From orig pfSense, can you ping the new vm pfSense 10.9.28.254?

No answer.

@jarhead said in Super Confused - LAN Gateway:

@bearhntr said in Super Confused - LAN Gateway:
They're not connected.
Are you using a virtual switch?
How are you connecting the two routers?
Is the pc you were connecting to the VM a physical machine? If so, disconnect it and use that cable to connect to OPT on router 1. Does it ping that way?

No answer.

Again.

@jarhead said in Super Confused - LAN Gateway:

@bearhntr
So, again, they aren't connected. Fix that first.

Set a pc to 10.10.1.2/30, connect it to the OPT port, can you ping 10.10.1.1?

Then set that pc to 10.10.1.1/30 and connect it to the VM WAN. Can you ping it 10.10.1.2?

No answer.

This is very simple. You have a problem with the VM but you refuse to acknowledge that.

Try the ping I suggested and you'll see it's not connecting.

Can you connect any physical machine to the VM at all?

bearhntr

@jarhead said in Super Confused - LAN Gateway:

@bearhntr
Do this.
Set a pc to 10.10.1.2/30, connect it to the OPT port, can you ping 10.10.1.1?

Then set that pc to 10.10.1.1/30 and connect it to the VM WAN. Can you ping it 10.10.1.2?

You'll see you can ping the existing pfSense but you won't ping the VM. FIX THAT.

OK - I did this, very difficult as there is no space for another PC where this box is.

Set the PC to 10.10.1.2/30 and plugged cable into OPT1 on PF1 - it immediately connected to the Internet and PING was successful.

Move the wire to the WAN port on the PF2 - set PC to 10.10.1.1/30 and PING fails.

As I am thinking given all the errors that I am seeing in the 'dmesg' in Proxmox Shell - that the 4-port card is bad or has something wrong with it.

[ 1248.474520] pcieport 0000:00:1d.0: AER: Multiple Corrected error received: 0000:00:1d.0
[ 1248.524181] pcieport 0000:00:1d.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, (Receiver ID)
[ 1248.524207] pcieport 0000:00:1d.0:   device [8086:a118] error status/mask=00002001/00002000
[ 1248.524231] pcieport 0000:00:1d.0:    [ 0] RxErr                 
[ 1248.667371] pcieport 0000:00:1d.0: AER: Multiple Corrected error received: 0000:00:1d.0
[ 1248.691962] pcieport 0000:00:1d.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, (Receiver ID)
[ 1248.691989] pcieport 0000:00:1d.0:   device [8086:a118] error status/mask=00000001/00002000
[ 1248.692011] pcieport 0000:00:1d.0:    [ 0] RxErr                 
[ 1252.456633] pcieport 0000:00:1d.0: AER: Multiple Corrected error received: 0000:00:1d.0
[ 1252.456677] pcieport 0000:00:1d.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, (Receiver ID)
[ 1252.456703] pcieport 0000:00:1d.0:   device [8086:a118] error status/mask=00002001/00002000
[ 1252.456725] pcieport 0000:00:1d.0:    [ 0] RxErr                 
[ 1260.319756] tg3 0000:01:00.1 enp1s0f1: Link is down
[ 1260.319878] vmbr2: port 1(enp1s0f1) entered disabled state
[ 1299.343586] pcieport 0000:00:1d.0: AER: Multiple Corrected error received: 0000:00:1d.0
[ 1299.392764] pcieport 0000:00:1d.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, (Receiver ID)
[ 1299.392790] pcieport 0000:00:1d.0:   device [8086:a118] error status/mask=00002001/00002000
[ 1299.392814] pcieport 0000:00:1d.0:    [ 0] RxErr                 
[ 1299.486874] pcieport 0000:00:1d.0: AER: Multiple Corrected error received: 0000:00:1d.0
[ 1299.535945] pcieport 0000:00:1d.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, (Receiver ID)
[ 1299.535970] pcieport 0000:00:1d.0:   device [8086:a118] error status/mask=00002001/00002000
[ 1299.535994] pcieport 0000:00:1d.0:    [ 0] RxErr                 
[ 1373.798280] pcieport 0000:00:1d.0: AER: Multiple Corrected error received: 0000:00:1d.0
[ 1373.822409] pcieport 0000:00:1d.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, (Receiver ID)
[ 1373.822435] pcieport 0000:00:1d.0:   device [8086:a118] error status/mask=00000001/00002000
[ 1373.822458] pcieport 0000:00:1d.0:    [ 0] RxErr                 
[ 1376.440381] pcieport 0000:00:1d.0: AER: Multiple Corrected error received: 0000:00:1d.0
[ 1376.489879] pcieport 0000:00:1d.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, (Receiver ID)
[ 1376.489905] pcieport 0000:00:1d.0:   device [8086:a118] error status/mask=00002001/00002000
[ 1376.489928] pcieport 0000:00:1d.0:    [ 0] RxErr

I will have to wait for the new card that I ordered (which I was advised would be better for virtualization -- INTEL i350 T4V2) to get here mid-week.

I appreciate all the group-head-banging....as this was a strange one. It made no sense as I have had Proxmox setup with pfSense before, on a differnt box with a 2-port PCIe NIC card - and had no problems. I got this new box, as the motherboard on that other one died and I could not get a replacement.

For the moment - the PF1 box is working and I will leave things be.

Again - thanks for all the extra brain-cells, as mine were about to take a Christmas Vacation.