[SOLVED] Proxmox/KVM additional IP and subnet->ping works, but no internet/ssh

Installation and Upgrades
Log in to reply
  • N

    **[SOLVED]
      Virtio network cards drivers caused the error!

    Had enabled virtio drivers in pfSense i386 and used virtio-drivers in Proxmox 3.1,
      which obviously caused the error.

    virtio -> vtnet0, vtnet1, vtnet2
      now
      e1000 -> em0, em1, em2

    In Proxmox use e1000 drivers for NetCards and reassign the interfaces
      at next pfSense boot.

    Other hardware virtualizations are working:
      https://doc.pfsense.org/index.php/VirtIO_Driver_Support

    The setup described below works perfectly.
    [/SOLVED]**

    Hello network and pfSense professionals,

    I need your help!
    I'm not a network pro, but do these network specialties for interest.

    [EDIT]
    To be more specific:
    What I'm missing is a scenario to lead through my x.y.73.16/29 official subnet IPs requests from WAN to SUBNET Interface and back without NAT or anything else.
    Only classic Firewall Rules I want to use in first Step.
    I have red about virual IPs or NAT 1:1, but isn't there a way to do it more simple?
    [/EDIT]

    I have built a Profmox / pfSense configuration at Hetzner with these instructions:
    http://www.hetzner-doku.tiahost.de/index.php?title=Zusammenfassung#Firewall_-_pfSense

    Functioning is also available with Proxmox VE 3.1 with ESXi instead (because I just want to get away from) as a hypervisor:
    http://forum.hetzner.de/wbb2/thread.php?threadid=20403

    I am currently working already on a different server with pfSense ( ESXi )
    but only with WAN / LAN to protect the internal network.

    BUT it hooks yet …

    coarse:
    Traceroute / ping go from inside and outside with very good times.
    data transmission/services such as Internet,ssh in both directions does NOT work.
    The DNS resolution is well, when I run apt-get from the inside it looks like this:
    [ img] http://erde2012.de/apt-get0proz.jpg [/ img]
    … stuck at 0 %.

    Same effect with WinSCP from the outside, he finds the host, hang at "connecting to host".

    My Proxmox/KVM Architecture:

    WAN / Internet
                :
                : Hetzner Network
            LAN : x.y.48.192/27             
      routed LAN : x.y.73.16/29 auf Hetzner zusatz IP x.y.48.220
                :
    .–----------+---------------------------------------------------------------.
    |            |              "Proxmox/KVM Host" IP x.y.48.201                |
    |  .---------+-------.                                                      |
    |  | vmbr0 - vSwitch |                                                      |
    |  '---------+-------'                                                      |
    |            |                                                              |
    |        LAN | x.y.48.192/27                                                |
    | routed LAN | x.y.73.16/29 over x.y.48.220                                  |
    |            |                                                              |
    |      .-----+---------------------------------------.                      |
    |      |  WAN IP                                    |                      |
    |      |  x.y.48.220 (incl. Hetzner MAC)            |                      |
    |      |                                            |                      |
    |      |                                            |                      |
    |      |  VM pfSense - i386 virtIO                  |                      |
    |      | (x64 hat Probleme mit KVM)                  |                      |
    |      |                                            |                      |
    |      |  SUBNET IP                INTRANET IP      |                      |
    |      |  x.y.73.17/29            192.168.2.1/24    |                      |
    |      '-----+----------------------------+----------'                      |
    |            |                            |                                  |
    |        LAN | x.y.73.16/29          LAN | 192.168.2.0/24                  |
    |            |                            |                                  |
    |  .---------+-------.          .---------+-------.                          |
    |  | vmbr1 - vSwitch |          | vmbr2 - vSwitch |                          |
    |  '---------+-------'          '---------+-------'                          |
    |            |                            |                                  |
    |      .-----+-------.              .-----|---------.                        |
    |      |            :              |              :                        |
    |      |      (Clients/Servers)    :      (internal Clients/Server VMs)    |
    |      |                                                                    |
    |  .---+---.                                                                |                     
    |  | VM 1  | x.y.73.18/29  <-- from outside/inside ping/tracerout          |
    |  '-------'                <-- NO apt-get, wget, Internet-Browsing          |
    |                          <-- from outside NO ssh (winSCP)                |
    |                                                                            |
    '----------------------------------------------------------------------------'

    VM 1: (oder Ubuntu Server, Windows)
    Xubuntu Network over DHCP:
    IP        x.y.73.21
    Broadcast  x.y.73.23
    Subnetz    255.255.255.248
    Gateway    x.y.73.17
    prim DNS  x.y.73.17
    All VM operating systems same effects:

    • Traceroute/Ping possible from outside and inside
    • Connect from outside via ssh or browseing does not work.
    • apt-get update does not work.
    • to the neighbors everything works as expected

    Windows VM, same problems -> no internet, internally ssh (WinSCP) to neighbors goes.

    Questions / uncertainties:

    • What about you with the Broadcasting address (xy48.223)?
    • DNS forwarder is on by default,
      the IPs from network xy73.16/29 should be visible from outside.
      No NAT may be required and how to configure it in pfSense?

    Network Hetzner:

    
Proxmox 3.1 + additional IP + subnetz/29

mainIP:  x.y.48.201
  gateway x.y.48.193
  netzmaske 255.255.255.224
  broadcast x.y.48.223

additional IP:  x.y.48.220 incl. own MAC
  gateway x.y.48.193
  netzmaske 255.255.255.224
  broadcast x.y.48.23

Subnetz: x.y.73.16 / 29
  gateway x.y.48.220 -> Subnetz is routed here
  netzmaske 255.255.255.248
  broadcast x.y.73.23

    Proxmox 3.1 Netzwerk Logik:

    
Erste VM: 
  pfsense vmbr0=WAN
  vmbr1=SUBNET
  vmbr2=INTRANET

Restlichen VM:
vmbr1=SUBNET
vmbr2=INTERANET

    Proxmox 3.1 /etc/interfaces:

    
# forwarding in /etc/sysctl.conf -> OFF
# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
# auto eth0
iface eth0 inet manual

################################
#### The first vswitch interface
# vswitch fuer den pfsense WAN "vtnet0" Adapter mit der Zusatz-IP "x.y.48.220" incl. Hetzner MAC Adresse
# MAC Adresse in PROXMOX dem pfsense adapter vmbr0 vergeben
# Auf die Zusatz-IP sollte das Subnet geroutet sein. 
# Berechnung von Netzwerk-Geometrie: http://www.trinler.net/de/service/tools/ipcalc.html
auto vmbr0
iface vmbr0 inet static
    # Hetzner Haupt IP Adresse des Servers
    address x.y.48.201
    netmask 255.255.255.255
    # broadcast x.y.48.223
    network x.y.48.192

    # Hetzner Haupt Gateway Adresse des Servers
    gateway x.y.48.193
    pointopoint x.y.48.193

    # Virtual Bridge Einstellungen
    bridge_ports eth0
    bridge_stp off
    bridge_fd 0

################################
#### The second vswitch interface
# vswitch fuer den pfsense SUBNET "vtnet1" Adapter)
auto vmbr1
iface vmbr1 inet manual
    bridge_ports none
    bridge_stp off
    bridge_fd 0

################################
#### The third vswitch interface
# vswitch fuer den pfsense OPT1/LOCAL/LAN "vtnet2" Adapter)
auto vmbr2
iface vmbr2 inet manual
    bridge_ports none
    bridge_stp off
    bridge_fd 0

```   

Konfiguration pfSense:

    nach dieser Anleitung:
    http://www.hetzner-doku.tiahost.de/index.php?title=Zusammenfassung#Firewall_-_pfSense

    1. ####
    System -> General Setup:
    127.0.0.1
    8.8.8.8
    208.67.220.220
    213.133.100.100

    2. ####
    System -> Routing:
    Gateways:
    Name        Interface    Gateway      Monitor IP    Description
    Hetzner_v6  WAN          fe80::1      fe80::1        Hetzner IPv6 Gateway
    Hetzner_v4  WAN          x.y.48.193    x.y.48.193    Hetzner IPv4 Gateway

    Routes:
    Network        Gateway      Interface    Description
    x.y.48.192/28  Hetzner_v4  WAN          Route zu Switchnachbarn, untere Hälfte
    x.y.48.208/28  Hetzner_v4  WAN          Route zu Switchnachbarn, obere Hälfte

    3. ####
    Interfaces:
    WAN IPv4: static        | IPv4 adress: x.y.48.220/29    | Gateway: x.y.48.193    | Block private und bogon
    WAN IPv6: static        | IPv6 adress: x❌x❌:2        | Gateway: fe80::1

    SUBNET IPv4: static    | IPv4 adress: x.y.73.17/29      | Gateway: None
    SUBNET IPv6: static    | IPv6 adress: x❌x❌1::2/80  | Gateway: None

    INTRANET IPv4: static  | IPv4 adress: 192.168.1.1/24    | Gateway: None
    INTRANET IPv6: static  | IPv6 adress: x❌x❌2::2/80  | Gateway: None

    4. ####
    Firewall -> Rules -> WAN:
    dir        ID    Proto    Source                  Port Dest  Port  Gateway  Queue  Schedule  Description   
    block      *      RFC 1918 networks                *    *    *      *        *      *        Block private networks   
    block      *      Reserved/not assigned by IANA    *    *    *      *        *      *        Block bogon networks   
    pass      IP4*  *                                *    *    *      *        none            TEST ALLE IPv4 AUF DURCHZUG

    Firewall -> Rules -> SUBNET:
    dir        ID    Proto    Source                  Port Dest  Port  Gateway  Queue  Schedule  Description   
    pass      IP4*  *                                *    *    *      *        none            TEST ALLE IPv4 AUF DURCHZUG

    5. ####
    Firewall -> Rules -> INTRANET:
    dir        ID    Proto    Source                  Port Dest  Port  Gateway  Queue  Schedule  Description   
    pass      *      ManAccsess                        *    *    MPorts *        none  *        pass management - hosts on ports   
    reject    *      *                                *    *    MPorts *        none  *        reject management - other on ports
    pass      IP4*  *                                *    *    *      *        none            TEST ALLE IPv4 AUF DURCHZUG

    6. ####
    Firewall -> NAT: (alles Voreinstellungen)
    Port Rorward:    KEINE EINTRÄGE
    1:1:            KEINE EINTRÄGE
    Outbound:        KEINE EINTRÄGE, Mode: Automatic outbound NAT rule generation
    NPt:            KEINE EINTRÄGE

    7. ####
    DNS forwarder:

    • Enable DNS forwarder
    • Register DHCP leases in DNS forwarder
    • Register DHCP static mappings in DNS forwarder
    • Resolve DHCP mappings first
      Interfaces: All

    8. ####
    DHCP server:
    WAN:        OFF
    SUBNET:      ON -> x.y.73.18      to  x.y.73.22
    INTRANET:    ON -> 192.168.1.100  to  192.168.1.200

    
From outside xy48.220 (pfSense firewall) accessibility is as expected:
ifconfig pfSense:

    vtnet0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
            options=c02bb <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso6,vlan_hwtso,linkstate>ether aa:bb:cc:dd:ee:aa
            inet x.y.48.220 netmask 0xffffffe0 broadcast x.y.48.223
            inet6 c🅱c:d:5723%vtnet0 prefixlen 64 scopeid 0x1
            inet6 a🅱c:d::2 prefixlen 64
            nd6 options=1 <performnud>media: Ethernet 1000baseT <full-duplex>status: active
    vtnet1: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
            options=c02bb <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso6,vlan_hwtso,linkstate>ether aa:bb:cc:dd:ee:bb
            inet x.y.73.17 netmask 0xfffffff8 broadcast x.y.73.23
            inet6 fe80::7c81:62ff:fe5a:170%vtnet1 prefixlen 64 scopeid 0x2
            inet6 a🅱c:d:1::2 prefixlen 80
            nd6 options=1 <performnud>media: Ethernet 1000baseT <full-duplex>status: active
    vtnet2: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
            options=c02bb <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso6,vlan_hwtso,linkstate>ether aa:bb:cc:dd:ee:cc
            inet 192.168.2.1 netmask 0xffffff00 broadcast 192.168.2.255
            inet6 b🅱c:d:6676%vtnet2 prefixlen 64 scopeid 0x3
            inet6 a🅱c:d:2::2 prefixlen 80
            nd6 options=1 <performnud>media: Ethernet 1000baseT <full-duplex>status: active
    lo0: flags=8049 <up,loopback,running,multicast>metric 0 mtu 16384
            options=3 <rxcsum,txcsum>inet 127.0.0.1 netmask 0xff000000
            inet6 ::1 prefixlen 128
            inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4
            nd6 options=3 <performnud,accept_rtadv>pfsync0: flags=0<> metric 0 mtu 1460
            syncpeer: 224.0.0.240 maxupd: 128 syncok: 1
    enc0: flags=0<> metric 0 mtu 1536
    pflog0: flags=100 <promisc>metric 0 mtu 33192</promisc></performnud,accept_rtadv></rxcsum,txcsum></up,loopback,running,multicast></full-duplex></performnud></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso6,vlan_hwtso,linkstate></up,broadcast,running,simplex,multicast></full-duplex></performnud></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso6,vlan_hwtso,linkstate></up,broadcast,running,simplex,multicast></full-duplex></performnud></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso6,vlan_hwtso,linkstate></up,broadcast,running,simplex,multicast>

    
Thanks in advance.

Helmut
    0
  • M

    This is an incredibly useful post.

    I've just started deploying pfSense on more powerful yet still small Jetway NF9HG boards.  Plenty of power to run pfSense, so I'ved added Proxmox underneath as well.

    No end of troubles with traffic - ping works, voip works, ssh doesn't. Configs all good, no errors.  Users experience good downloads, awful or non-existent uploads.

    Wireshark captures show lots of retransmission.

    Taken hours to find this post, since the issue isn't particularly obvious.  Wondering now if it was related to 'disable hardware checksum offloading', as I can't imagine the FreeBSD vtnet drivers are this broken.

    0
  • T

    nut, seriously dude … i just love u  ;)

    I lost 2 evenings on my configuration because of this !!!!

    Ok, it has been mentioned at the very end but it has to be written on 32 font size blinking red on this page :
    https://pve.proxmox.com/wiki/PfSense_Guest_Notes

    And apparently you can still use virtIO driver following steps on the link :
    https://doc.pfsense.org/index.php/VirtIO_Driver_Support

    Thanks again

    0
  • B

    pls show me your full /etc/network/interfaces file from proxmox host
    i cant seem to make it work plssssssssssssssssssssssssss 😢 😢 😢 😢

    0

Products

Services

Support

News

Resources

Company

Our Mission

We provide leading-edge network security at a fair price - regardless of organizational size or network sophistication. We believe that an open-source security model offers disruptive pricing along with the agility required to quickly address emerging threats.

Subscribe to our Newsletter

Product information, software announcements, and special offers. See our newsletter archive to sign up for future newsletters and to read past announcements.

© 2020 Rubicon Communications, LLC | Privacy Policy