[SOLVED] Proxmox/KVM additional IP and subnet->ping works, but no internet/ssh

nut 0 · Feb 20, 2014, 3:59 PM

**[SOLVED]
Virtio network cards drivers caused the error!

Had enabled virtio drivers in pfSense i386 and used virtio-drivers in Proxmox 3.1,
which obviously caused the error.

virtio -> vtnet0, vtnet1, vtnet2
now
e1000 -> em0, em1, em2

In Proxmox use e1000 drivers for NetCards and reassign the interfaces
at next pfSense boot.

Other hardware virtualizations are working:
https://doc.pfsense.org/index.php/VirtIO_Driver_Support

The setup described below works perfectly.
[/SOLVED]**

Hello network and pfSense professionals,

I need your help!
I'm not a network pro, but do these network specialties for interest.

[EDIT]
To be more specific:
What I'm missing is a scenario to lead through my x.y.73.16/29 official subnet IPs requests from WAN to SUBNET Interface and back without NAT or anything else.
Only classic Firewall Rules I want to use in first Step.
I have red about virual IPs or NAT 1:1, but isn't there a way to do it more simple?
[/EDIT]

I have built a Profmox / pfSense configuration at Hetzner with these instructions:
http://www.hetzner-doku.tiahost.de/index.php?title=Zusammenfassung#Firewall_-_pfSense

Functioning is also available with Proxmox VE 3.1 with ESXi instead (because I just want to get away from) as a hypervisor:
http://forum.hetzner.de/wbb2/thread.php?threadid=20403

I am currently working already on a different server with pfSense ( ESXi )
but only with WAN / LAN to protect the internal network.

BUT it hooks yet …

coarse:
Traceroute / ping go from inside and outside with very good times.
data transmission/services such as Internet,ssh in both directions does NOT work.
The DNS resolution is well, when I run apt-get from the inside it looks like this:
[ img] http://erde2012.de/apt-get0proz.jpg [/ img]
… stuck at 0 %.

Same effect with WinSCP from the outside, he finds the host, hang at "connecting to host".

My Proxmox/KVM Architecture:

WAN / Internet
:
: Hetzner Network
LAN : x.y.48.192/27
routed LAN : x.y.73.16/29 auf Hetzner zusatz IP x.y.48.220
:
.–----------+---------------------------------------------------------------.
| | "Proxmox/KVM Host" IP x.y.48.201 |
| .---------+-------. |
| | vmbr0 - vSwitch | |
| '---------+-------' |
| | |
| LAN | x.y.48.192/27 |
| routed LAN | x.y.73.16/29 over x.y.48.220 |
| | |
| .-----+---------------------------------------. |
| | WAN IP | |
| | x.y.48.220 (incl. Hetzner MAC) | |
| | | |
| | | |
| | VM pfSense - i386 virtIO | |
| | (x64 hat Probleme mit KVM) | |
| | | |
| | SUBNET IP INTRANET IP | |
| | x.y.73.17/29 192.168.2.1/24 | |
| '-----+----------------------------+----------' |
| | | |
| LAN | x.y.73.16/29 LAN | 192.168.2.0/24 |
| | | |
| .---------+-------. .---------+-------. |
| | vmbr1 - vSwitch | | vmbr2 - vSwitch | |
| '---------+-------' '---------+-------' |
| | | |
| .-----+-------. .-----|---------. |
| | : | : |
| | (Clients/Servers) : (internal Clients/Server VMs) |
| | |
| .---+---. |
| | VM 1 | x.y.73.18/29 <-- from outside/inside ping/tracerout |
| '-------' <-- NO apt-get, wget, Internet-Browsing |
| <-- from outside NO ssh (winSCP) |
| |
'----------------------------------------------------------------------------'

VM 1: (oder Ubuntu Server, Windows)
Xubuntu Network over DHCP:
IP x.y.73.21
Broadcast x.y.73.23
Subnetz 255.255.255.248
Gateway x.y.73.17
prim DNS x.y.73.17
All VM operating systems same effects:

Traceroute/Ping possible from outside and inside
Connect from outside via ssh or browseing does not work.
apt-get update does not work.
to the neighbors everything works as expected

Windows VM, same problems -> no internet, internally ssh (WinSCP) to neighbors goes.

Questions / uncertainties:

What about you with the Broadcasting address (xy48.223)?
DNS forwarder is on by default,
the IPs from network xy73.16/29 should be visible from outside.
No NAT may be required and how to configure it in pfSense?

Network Hetzner:


Proxmox 3.1 + additional IP + subnetz/29

mainIP:  x.y.48.201
  gateway x.y.48.193
  netzmaske 255.255.255.224
  broadcast x.y.48.223

additional IP:  x.y.48.220 incl. own MAC
  gateway x.y.48.193
  netzmaske 255.255.255.224
  broadcast x.y.48.23

Subnetz: x.y.73.16 / 29
  gateway x.y.48.220 -> Subnetz is routed here
  netzmaske 255.255.255.248
  broadcast x.y.73.23

Proxmox 3.1 Netzwerk Logik:


Erste VM: 
  pfsense vmbr0=WAN
  vmbr1=SUBNET
  vmbr2=INTRANET

Restlichen VM:
vmbr1=SUBNET
vmbr2=INTERANET

Proxmox 3.1 /etc/interfaces:


# forwarding in /etc/sysctl.conf -> OFF
# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
# auto eth0
iface eth0 inet manual

################################
#### The first vswitch interface
# vswitch fuer den pfsense WAN "vtnet0" Adapter mit der Zusatz-IP "x.y.48.220" incl. Hetzner MAC Adresse
# MAC Adresse in PROXMOX dem pfsense adapter vmbr0 vergeben
# Auf die Zusatz-IP sollte das Subnet geroutet sein. 
# Berechnung von Netzwerk-Geometrie: http://www.trinler.net/de/service/tools/ipcalc.html
auto vmbr0
iface vmbr0 inet static
    # Hetzner Haupt IP Adresse des Servers
    address x.y.48.201
    netmask 255.255.255.255
    # broadcast x.y.48.223
    network x.y.48.192

    # Hetzner Haupt Gateway Adresse des Servers
    gateway x.y.48.193
    pointopoint x.y.48.193

    # Virtual Bridge Einstellungen
    bridge_ports eth0
    bridge_stp off
    bridge_fd 0

################################
#### The second vswitch interface
# vswitch fuer den pfsense SUBNET "vtnet1" Adapter)
auto vmbr1
iface vmbr1 inet manual
    bridge_ports none
    bridge_stp off
    bridge_fd 0

################################
#### The third vswitch interface
# vswitch fuer den pfsense OPT1/LOCAL/LAN "vtnet2" Adapter)
auto vmbr2
iface vmbr2 inet manual
    bridge_ports none
    bridge_stp off
    bridge_fd 0

```   

Konfiguration pfSense:

nach dieser Anleitung:
http://www.hetzner-doku.tiahost.de/index.php?title=Zusammenfassung#Firewall_-_pfSense

1. ####
System -> General Setup:
127.0.0.1
8.8.8.8
208.67.220.220
213.133.100.100

2. ####
System -> Routing:
Gateways:
Name Interface Gateway Monitor IP Description
Hetzner_v6 WAN fe80::1 fe80::1 Hetzner IPv6 Gateway
Hetzner_v4 WAN x.y.48.193 x.y.48.193 Hetzner IPv4 Gateway

Routes:
Network Gateway Interface Description
x.y.48.192/28 Hetzner_v4 WAN Route zu Switchnachbarn, untere Hälfte
x.y.48.208/28 Hetzner_v4 WAN Route zu Switchnachbarn, obere Hälfte

SUBNET IPv4: static | IPv4 adress: x.y.73.17/29 | Gateway: None
SUBNET IPv6: static | IPv6 adress: xx1::2/80 | Gateway: None

INTRANET IPv4: static | IPv4 adress: 192.168.1.1/24 | Gateway: None
INTRANET IPv6: static | IPv6 adress: xx2::2/80 | Gateway: None

4. ####
Firewall -> Rules -> WAN:
dir ID Proto Source Port Dest Port Gateway Queue Schedule Description
block * RFC 1918 networks * * * * * * Block private networks
block * Reserved/not assigned by IANA * * * * * * Block bogon networks
pass IP4* * * * * * none TEST ALLE IPv4 AUF DURCHZUG

Firewall -> Rules -> SUBNET:
dir ID Proto Source Port Dest Port Gateway Queue Schedule Description
pass IP4* * * * * * none TEST ALLE IPv4 AUF DURCHZUG

5. ####
Firewall -> Rules -> INTRANET:
dir ID Proto Source Port Dest Port Gateway Queue Schedule Description
pass * ManAccsess * * MPorts * none * pass management - hosts on ports
reject * * * * MPorts * none * reject management - other on ports
pass IP4* * * * * * none TEST ALLE IPv4 AUF DURCHZUG

6. ####
Firewall -> NAT: (alles Voreinstellungen)
Port Rorward: KEINE EINTRÄGE
1:1: KEINE EINTRÄGE
Outbound: KEINE EINTRÄGE, Mode: Automatic outbound NAT rule generation
NPt: KEINE EINTRÄGE

7. ####
DNS forwarder:

Enable DNS forwarder
Register DHCP leases in DNS forwarder
Register DHCP static mappings in DNS forwarder
Resolve DHCP mappings first
Interfaces: All

8. ####
DHCP server:
WAN: OFF
SUBNET: ON -> x.y.73.18 to x.y.73.22
INTRANET: ON -> 192.168.1.100 to 192.168.1.200


From outside xy48.220 (pfSense firewall) accessibility is as expected:
ifconfig pfSense:

vtnet0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
options=c02bb <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso6,vlan_hwtso,linkstate>ether aa:bb:cc:dd:ee:aa
inet x.y.48.220 netmask 0xffffffe0 broadcast x.y.48.223
inet6 cc:d:5723%vtnet0 prefixlen 64 scopeid 0x1
inet6 ac:d::2 prefixlen 64
nd6 options=1 <performnud>media: Ethernet 1000baseT <full-duplex>status: active
vtnet1: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
options=c02bb <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso6,vlan_hwtso,linkstate>ether aa:bb:cc:dd:ee:bb
inet x.y.73.17 netmask 0xfffffff8 broadcast x.y.73.23
inet6 fe80::7c81:62ff:fe5a:170%vtnet1 prefixlen 64 scopeid 0x2
inet6 ac:d:1::2 prefixlen 80
nd6 options=1 <performnud>media: Ethernet 1000baseT <full-duplex>status: active
vtnet2: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
options=c02bb <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso6,vlan_hwtso,linkstate>ether aa:bb:cc:dd:ee:cc
inet 192.168.2.1 netmask 0xffffff00 broadcast 192.168.2.255
inet6 bc:d:6676%vtnet2 prefixlen 64 scopeid 0x3
inet6 ac:d:2::2 prefixlen 80
nd6 options=1 <performnud>media: Ethernet 1000baseT <full-duplex>status: active
lo0: flags=8049 <up,loopback,running,multicast>metric 0 mtu 16384
options=3 <rxcsum,txcsum>inet 127.0.0.1 netmask 0xff000000
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4
nd6 options=3 <performnud,accept_rtadv>pfsync0: flags=0<> metric 0 mtu 1460
syncpeer: 224.0.0.240 maxupd: 128 syncok: 1
enc0: flags=0<> metric 0 mtu 1536
pflog0: flags=100 <promisc>metric 0 mtu 33192</promisc></performnud,accept_rtadv></rxcsum,txcsum></up,loopback,running,multicast></full-duplex></performnud></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso6,vlan_hwtso,linkstate></up,broadcast,running,simplex,multicast></full-duplex></performnud></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso6,vlan_hwtso,linkstate></up,broadcast,running,simplex,multicast></full-duplex></performnud></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso6,vlan_hwtso,linkstate></up,broadcast,running,simplex,multicast>


Thanks in advance.

Helmut

mcdiesel · Jul 23, 2016, 11:33 AM

This is an incredibly useful post.

I've just started deploying pfSense on more powerful yet still small Jetway NF9HG boards. Plenty of power to run pfSense, so I'ved added Proxmox underneath as well.

No end of troubles with traffic - ping works, voip works, ssh doesn't. Configs all good, no errors. Users experience good downloads, awful or non-existent uploads.

Wireshark captures show lots of retransmission.

Taken hours to find this post, since the issue isn't particularly obvious. Wondering now if it was related to 'disable hardware checksum offloading', as I can't imagine the FreeBSD vtnet drivers are this broken.

Tolman · Dec 6, 2016, 5:36 PM

nut, seriously dude … i just love u ;)

I lost 2 evenings on my configuration because of this !!!!

Ok, it has been mentioned at the very end but it has to be written on 32 font size blinking red on this page :
https://pve.proxmox.com/wiki/PfSense_Guest_Notes

And apparently you can still use virtIO driver following steps on the link :
https://doc.pfsense.org/index.php/VirtIO_Driver_Support

Thanks again

booruszuru · Aug 24, 2020, 5:00 PM

pls show me your full /etc/network/interfaces file from proxmox host
i cant seem to make it work plssssssssssssssssssssssssss