Direct connection says host is down
-
@williamsilverstein OPT1 is the interface the two computer are connected to? And you configured the OPT1 interface with an IP address of 10.1.0.0/8? And is there a switch on the OPT1 interface onto which the two computers are connected?
If yes then these two machine can communicate with each other directly and won't need pfSense. What OS are installed on them? And does that OS have a firewall enabled?
It is very unsual to have an /8 subnet, that gives you 16'777'216 addresses and most people don't need that many. It's way more common to use an /24 subnet.
-
@patient0 I have two almost identical systems. Both proxmox with Pfsense as a VM on Dell 730xds. No switch, just a patch cable. I had done this before with my 2950.
I am setting up a PBE on the second server so I can do rapid backups (10gb) and eventually move the 1st system to a colo facility with a switch on 10.1.X.X for a local network.
-
@williamsilverstein I don't understand your setup.
Both are Proxmox VE servers or is one a VE and the other a Proxmox Backup server? If bothj PVE, is a pfSense VM on each PVE? How are the PVE bridges configured? And why a huge 10.1/8 network? Is the PVE firewall enabled?
-
@patient0 Yes, both Pfsense are VMs on PVE. I am doing a PCI passthrough on the network cards so I can use the hardware offloading.
The reason for the /8 is to match the local network I'll eventually connect to, allowing me to maintain the IP addressing convention.
At first, I was thinking I may need the switch, here, but the hardware indicates a valid connection.
Yes, I tried and confirmed the firewalls are off.
-
@williamsilverstein both VMs have a WAN connection in PCI passthrough and OPT1 as a local network interface as passthrough? Are the 10.1.0.50/51 the IPs of the pfSense OPT1 interfaces, like are they part of the same local network, not WAN.
And btw: when you write "Direct connection says host is down", what do you mean be that? Are the interfaces marked as down in pfSense Dashboard?
Maybe a quick diagram (by hand even) would help?
-
@williamsilverstein said in Direct connection says host is down:
10.1.X.X for a local network.
That's /16.
I agree seeing /8 like that is a massive red flag for me. It would be very easy to get a subnet conflict with such a huge subnet on one interface. It's also almost always the result of an initial setup by someone who didn't really understand the available private subnets at the time.
But, yes, how are the VMs configured inside PVE?
-
@stephenw10 The /8 was only connected to the other port. I put a switch in, but no difference.
This is the original:
I inserted a router, just in case I was wrong. No difference.
-
So you are not using 10.x.x.x. on any other interfaces there?
How are those ports configured in PVE? Are you passing them through to the pfSense VM(s)?
-
@stephenw10 There ports are not configured in the proxmox or the VM. The network device is passed through to the VM
-
Ok, well those subnets are killing me! But they should work.
So are you sure the ports you have linked are actually the ports you have passed through?
Do you see it lose link when you unplug it?
-
@stephenw10 Yes. I checked it before, and I just checked it again.
-
Ok so how are you testing the connectivity?
'Host is down' implies it's ARPing for the target device and not seeing any replies. Can we assume neither host appears in the ARP table of the other?
-
@stephenw10 You are correct. I plugged a windows laptop into the same switch and set an IP address of 10.1.2.50 (netmask 255.0.0.0) and it would not see either 10.1.1.50 or 10.1.0.50..
The other network does not show up in the arp list.
-
Hmm. How are you actually testing? Just trying to ping the other IPs?
Try running a packet capture on that interface and see is anything is arriving at all. One both VMs.
It looks like it's not passing the NIC through correctly IMO. Some hardware off loading perhaps.
Are you connecting to the VMs using the LAN? The vtnet interface?
Can the VMs connect out using the other passed through NIC on the WAN?
-
The 2nd ports on each machine did not.
I changed the PCI passthrough to allow only one passthrough on each machine with all functions checked, which now provides two Ethernet devices in PfSense. Now, one works, not the other. One machine works (10.1.0.50), the other shows that the 2nd port (on 10.1.1.50) is down on the interface status screen.
-
Hmm, well it sounds like definitely some pass-through issue in PVE then. Anything in the Proxmox logs?
So on the failing machine the NIC that's passed through never shows link? Have you tested the other ports to be sure it's not just passing the wrong one?
-
@stephenw10 You are correct, but it confuses me even more. The Idrac says port 4 is down.
From the idrac8 software:
port 1, identified as 10gb, mac ending with 6e.
port 2, identified as 10gb, mac ending with 6f
port 3, identified as 1gb, mac ending with 6c
port 4, identified as 1gb, mac ending with 6dFrom the arp table (either in pfsense machine or router)
WAN 192.168.0.89 :6e (pfsense wan address, passthrough)
WAN 192.168.0.90 :6c (proxmox address)
OPT2 10.1.1.50 :6d (direct connection)
OPT1 10.1.1.50 (not active now) :6fAlso what is confusing is that the interfaces widget indicates the wrong speeds for some reason.
Part of the current idea is to have a 10gb directly wired to a 10gb so that i can use that for fast backups. When I move into the COLO, I will connect OPT1/2 (whichever works) into a local switch tied to 5 other machines.
-
Hmm, now I'm confused. Your screenshot above shows two igb NICs. That's a 1G Intel NIC. It can't link at 10G. Are you using different NICs there now?
I expect the idrac to show the real hardware MAC for each port. The internals only vtnet ports should show a MAC generated by Proxmox.
