Direct connection says host is down

williamsilverstein

@stephenw10 The /8 was only connected to the other port. I put a switch in, but no difference.

This is the original:

I inserted a router, just in case I was wrong. No difference.

stephenw10

So you are not using 10.x.x.x. on any other interfaces there?

How are those ports configured in PVE? Are you passing them through to the pfSense VM(s)?

williamsilverstein

@stephenw10 There ports are not configured in the proxmox or the VM. The network device is passed through to the VM

stephenw10

Ok, well those subnets are killing me! But they should work.

So are you sure the ports you have linked are actually the ports you have passed through?

Do you see it lose link when you unplug it?

williamsilverstein

@stephenw10 Yes. I checked it before, and I just checked it again.

stephenw10

Ok so how are you testing the connectivity?

'Host is down' implies it's ARPing for the target device and not seeing any replies. Can we assume neither host appears in the ARP table of the other?

williamsilverstein

@stephenw10 You are correct. I plugged a windows laptop into the same switch and set an IP address of 10.1.2.50 (netmask 255.0.0.0) and it would not see either 10.1.1.50 or 10.1.0.50..

The other network does not show up in the arp list.

stephenw10

Hmm. How are you actually testing? Just trying to ping the other IPs?

Try running a packet capture on that interface and see is anything is arriving at all. One both VMs.

It looks like it's not passing the NIC through correctly IMO. Some hardware off loading perhaps.

Are you connecting to the VMs using the LAN? The vtnet interface?

Can the VMs connect out using the other passed through NIC on the WAN?

williamsilverstein

@stephenw10

The 2nd ports on each machine did not.

I changed the PCI passthrough to allow only one passthrough on each machine with all functions checked, which now provides two Ethernet devices in PfSense. Now, one works, not the other. One machine works (10.1.0.50), the other shows that the 2nd port (on 10.1.1.50) is down on the interface status screen.

stephenw10

Hmm, well it sounds like definitely some pass-through issue in PVE then. Anything in the Proxmox logs?

So on the failing machine the NIC that's passed through never shows link? Have you tested the other ports to be sure it's not just passing the wrong one?

williamsilverstein

@stephenw10 You are correct, but it confuses me even more. The Idrac says port 4 is down.

From the idrac8 software:
port 1, identified as 10gb, mac ending with 6e.
port 2, identified as 10gb, mac ending with 6f
port 3, identified as 1gb, mac ending with 6c
port 4, identified as 1gb, mac ending with 6d

From the arp table (either in pfsense machine or router)
WAN 192.168.0.89 :6e (pfsense wan address, passthrough)
WAN 192.168.0.90 :6c (proxmox address)
OPT2 10.1.1.50 :6d (direct connection)
OPT1 10.1.1.50 (not active now) :6f

Also what is confusing is that the interfaces widget indicates the wrong speeds for some reason.

Part of the current idea is to have a 10gb directly wired to a 10gb so that i can use that for fast backups. When I move into the COLO, I will connect OPT1/2 (whichever works) into a local switch tied to 5 other machines.

stephenw10

Hmm, now I'm confused. Your screenshot above shows two igb NICs. That's a 1G Intel NIC. It can't link at 10G. Are you using different NICs there now?

I expect the idrac to show the real hardware MAC for each port. The internals only vtnet ports should show a MAC generated by Proxmox.