Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    [SOLVED] Proxmox/KVM additional IP and subnet->ping works, but no internet/ssh

    Scheduled Pinned Locked Moved Problems Installing or Upgrading pfSense Software
    4 Posts 4 Posters 13.1k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • N
      nut 0
      last edited by

      **[SOLVED]
        Virtio network cards drivers caused the error!

      Had enabled virtio drivers in pfSense i386 and used virtio-drivers in Proxmox 3.1,
        which obviously caused the error.

      virtio -> vtnet0, vtnet1, vtnet2
        now
        e1000 -> em0, em1, em2

      In Proxmox use e1000 drivers for NetCards and reassign the interfaces
        at next pfSense boot.

      Other hardware virtualizations are working:
        https://doc.pfsense.org/index.php/VirtIO_Driver_Support

      The setup described below works perfectly.
      [/SOLVED]**

      Hello network and pfSense professionals,

      I need your help!
      I'm not a network pro, but do these network specialties for interest.

      [EDIT]
      To be more specific:
      What I'm missing is a scenario to lead through my x.y.73.16/29 official subnet IPs requests from WAN to SUBNET Interface and back without NAT or anything else.
      Only classic Firewall Rules I want to use in first Step.
      I have red about virual IPs or NAT 1:1, but isn't there a way to do it more simple?
      [/EDIT]

      I have built a Profmox / pfSense configuration at Hetzner with these instructions:
      http://www.hetzner-doku.tiahost.de/index.php?title=Zusammenfassung#Firewall_-_pfSense

      Functioning is also available with Proxmox VE 3.1 with ESXi instead (because I just want to get away from) as a hypervisor:
      http://forum.hetzner.de/wbb2/thread.php?threadid=20403

      I am currently working already on a different server with pfSense ( ESXi )
      but only with WAN / LAN to protect the internal network.

      BUT it hooks yet …

      coarse:
      Traceroute / ping go from inside and outside with very good times.
      data transmission/services such as Internet,ssh in both directions does NOT work.
      The DNS resolution is well, when I run apt-get from the inside it looks like this:
      [ img] http://erde2012.de/apt-get0proz.jpg [/ img]
      … stuck at 0 %.

      Same effect with WinSCP from the outside, he finds the host, hang at "connecting to host".

      My Proxmox/KVM Architecture:

      WAN / Internet
                  :
                  : Hetzner Network
              LAN : x.y.48.192/27             
        routed LAN : x.y.73.16/29 auf Hetzner zusatz IP x.y.48.220
                  :
      .–----------+---------------------------------------------------------------.
      |            |              "Proxmox/KVM Host" IP x.y.48.201                |
      |  .---------+-------.                                                      |
      |  | vmbr0 - vSwitch |                                                      |
      |  '---------+-------'                                                      |
      |            |                                                              |
      |        LAN | x.y.48.192/27                                                |
      | routed LAN | x.y.73.16/29 over x.y.48.220                                  |
      |            |                                                              |
      |      .-----+---------------------------------------.                      |
      |      |  WAN IP                                    |                      |
      |      |  x.y.48.220 (incl. Hetzner MAC)            |                      |
      |      |                                            |                      |
      |      |                                            |                      |
      |      |  VM pfSense - i386 virtIO                  |                      |
      |      | (x64 hat Probleme mit KVM)                  |                      |
      |      |                                            |                      |
      |      |  SUBNET IP                INTRANET IP      |                      |
      |      |  x.y.73.17/29            192.168.2.1/24    |                      |
      |      '-----+----------------------------+----------'                      |
      |            |                            |                                  |
      |        LAN | x.y.73.16/29          LAN | 192.168.2.0/24                  |
      |            |                            |                                  |
      |  .---------+-------.          .---------+-------.                          |
      |  | vmbr1 - vSwitch |          | vmbr2 - vSwitch |                          |
      |  '---------+-------'          '---------+-------'                          |
      |            |                            |                                  |
      |      .-----+-------.              .-----|---------.                        |
      |      |            :              |              :                        |
      |      |      (Clients/Servers)    :      (internal Clients/Server VMs)    |
      |      |                                                                    |
      |  .---+---.                                                                |                     
      |  | VM 1  | x.y.73.18/29  <-- from outside/inside ping/tracerout          |
      |  '-------'                <-- NO apt-get, wget, Internet-Browsing          |
      |                          <-- from outside NO ssh (winSCP)                |
      |                                                                            |
      '----------------------------------------------------------------------------'

      VM 1: (oder Ubuntu Server, Windows)
      Xubuntu Network over DHCP:
      IP        x.y.73.21
      Broadcast  x.y.73.23
      Subnetz    255.255.255.248
      Gateway    x.y.73.17
      prim DNS  x.y.73.17
      All VM operating systems same effects:

      • Traceroute/Ping possible from outside and inside
      • Connect from outside via ssh or browseing does not work.
      • apt-get update does not work.
      • to the neighbors everything works as expected

      Windows VM, same problems -> no internet, internally ssh (WinSCP) to neighbors goes.

      Questions / uncertainties:

      • What about you with the Broadcasting address (xy48.223)?
      • DNS forwarder is on by default,
        the IPs from network xy73.16/29 should be visible from outside.
        No NAT may be required and how to configure it in pfSense?

      Network Hetzner:

      
      Proxmox 3.1 + additional IP + subnetz/29
      
      mainIP:  x.y.48.201
        gateway x.y.48.193
        netzmaske 255.255.255.224
        broadcast x.y.48.223
      
      additional IP:  x.y.48.220 incl. own MAC
        gateway x.y.48.193
        netzmaske 255.255.255.224
        broadcast x.y.48.23
      
      Subnetz: x.y.73.16 / 29
        gateway x.y.48.220 -> Subnetz is routed here
        netzmaske 255.255.255.248
        broadcast x.y.73.23
      
      

      Proxmox 3.1 Netzwerk Logik:

      
      Erste VM: 
        pfsense vmbr0=WAN
        vmbr1=SUBNET
        vmbr2=INTRANET
      
      Restlichen VM:
      vmbr1=SUBNET
      vmbr2=INTERANET
      
      

      Proxmox 3.1 /etc/interfaces:

      
      # forwarding in /etc/sysctl.conf -> OFF
      # The loopback network interface
      auto lo
      iface lo inet loopback
      
      # The primary network interface
      # auto eth0
      iface eth0 inet manual
      
      ################################
      #### The first vswitch interface
      # vswitch fuer den pfsense WAN "vtnet0" Adapter mit der Zusatz-IP "x.y.48.220" incl. Hetzner MAC Adresse
      # MAC Adresse in PROXMOX dem pfsense adapter vmbr0 vergeben
      # Auf die Zusatz-IP sollte das Subnet geroutet sein. 
      # Berechnung von Netzwerk-Geometrie: http://www.trinler.net/de/service/tools/ipcalc.html
      auto vmbr0
      iface vmbr0 inet static
          # Hetzner Haupt IP Adresse des Servers
          address x.y.48.201
          netmask 255.255.255.255
          # broadcast x.y.48.223
          network x.y.48.192
      
          # Hetzner Haupt Gateway Adresse des Servers
          gateway x.y.48.193
          pointopoint x.y.48.193
      
          # Virtual Bridge Einstellungen
          bridge_ports eth0
          bridge_stp off
          bridge_fd 0
      
      ################################
      #### The second vswitch interface
      # vswitch fuer den pfsense SUBNET "vtnet1" Adapter)
      auto vmbr1
      iface vmbr1 inet manual
          bridge_ports none
          bridge_stp off
          bridge_fd 0
      
      ################################
      #### The third vswitch interface
      # vswitch fuer den pfsense OPT1/LOCAL/LAN "vtnet2" Adapter)
      auto vmbr2
      iface vmbr2 inet manual
          bridge_ports none
          bridge_stp off
          bridge_fd 0
      
      ```   
      
      Konfiguration pfSense:
      

      nach dieser Anleitung:
      http://www.hetzner-doku.tiahost.de/index.php?title=Zusammenfassung#Firewall_-_pfSense

      1. ####
      System -> General Setup:
      127.0.0.1
      8.8.8.8
      208.67.220.220
      213.133.100.100

      2. ####
      System -> Routing:
      Gateways:
      Name        Interface    Gateway      Monitor IP    Description
      Hetzner_v6  WAN          fe80::1      fe80::1        Hetzner IPv6 Gateway
      Hetzner_v4  WAN          x.y.48.193    x.y.48.193    Hetzner IPv4 Gateway

      Routes:
      Network        Gateway      Interface    Description
      x.y.48.192/28  Hetzner_v4  WAN          Route zu Switchnachbarn, untere Hälfte
      x.y.48.208/28  Hetzner_v4  WAN          Route zu Switchnachbarn, obere Hälfte

      3. ####
      Interfaces:
      WAN IPv4: static        | IPv4 adress: x.y.48.220/29    | Gateway: x.y.48.193    | Block private und bogon
      WAN IPv6: static        | IPv6 adress: x❌x❌:2        | Gateway: fe80::1

      SUBNET IPv4: static    | IPv4 adress: x.y.73.17/29      | Gateway: None
      SUBNET IPv6: static    | IPv6 adress: x❌x❌1::2/80  | Gateway: None

      INTRANET IPv4: static  | IPv4 adress: 192.168.1.1/24    | Gateway: None
      INTRANET IPv6: static  | IPv6 adress: x❌x❌2::2/80  | Gateway: None

      4. ####
      Firewall -> Rules -> WAN:
      dir        ID    Proto    Source                  Port Dest  Port  Gateway  Queue  Schedule  Description   
      block      *      RFC 1918 networks                *    *    *      *        *      *        Block private networks   
      block      *      Reserved/not assigned by IANA    *    *    *      *        *      *        Block bogon networks   
      pass      IP4*  *                                *    *    *      *        none            TEST ALLE IPv4 AUF DURCHZUG

      Firewall -> Rules -> SUBNET:
      dir        ID    Proto    Source                  Port Dest  Port  Gateway  Queue  Schedule  Description   
      pass      IP4*  *                                *    *    *      *        none            TEST ALLE IPv4 AUF DURCHZUG

      5. ####
      Firewall -> Rules -> INTRANET:
      dir        ID    Proto    Source                  Port Dest  Port  Gateway  Queue  Schedule  Description   
      pass      *      ManAccsess                        *    *    MPorts *        none  *        pass management - hosts on ports   
      reject    *      *                                *    *    MPorts *        none  *        reject management - other on ports
      pass      IP4*  *                                *    *    *      *        none            TEST ALLE IPv4 AUF DURCHZUG

      6. ####
      Firewall -> NAT: (alles Voreinstellungen)
      Port Rorward:    KEINE EINTRÄGE
      1:1:            KEINE EINTRÄGE
      Outbound:        KEINE EINTRÄGE, Mode: Automatic outbound NAT rule generation
      NPt:            KEINE EINTRÄGE

      7. ####
      DNS forwarder:

      • Enable DNS forwarder
      • Register DHCP leases in DNS forwarder
      • Register DHCP static mappings in DNS forwarder
      • Resolve DHCP mappings first
        Interfaces: All

      8. ####
      DHCP server:
      WAN:        OFF
      SUBNET:      ON -> x.y.73.18      to  x.y.73.22
      INTRANET:    ON -> 192.168.1.100  to  192.168.1.200

      
      From outside xy48.220 (pfSense firewall) accessibility is as expected:
      ifconfig pfSense:
      

      vtnet0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
              options=c02bb <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso6,vlan_hwtso,linkstate>ether aa:bb:cc:dd:ee:aa
              inet x.y.48.220 netmask 0xffffffe0 broadcast x.y.48.223
              inet6 c🅱c:d:5723%vtnet0 prefixlen 64 scopeid 0x1
              inet6 a🅱c:d::2 prefixlen 64
              nd6 options=1 <performnud>media: Ethernet 1000baseT <full-duplex>status: active
      vtnet1: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
              options=c02bb <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso6,vlan_hwtso,linkstate>ether aa:bb:cc:dd:ee:bb
              inet x.y.73.17 netmask 0xfffffff8 broadcast x.y.73.23
              inet6 fe80::7c81:62ff:fe5a:170%vtnet1 prefixlen 64 scopeid 0x2
              inet6 a🅱c:d:1::2 prefixlen 80
              nd6 options=1 <performnud>media: Ethernet 1000baseT <full-duplex>status: active
      vtnet2: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
              options=c02bb <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso6,vlan_hwtso,linkstate>ether aa:bb:cc:dd:ee:cc
              inet 192.168.2.1 netmask 0xffffff00 broadcast 192.168.2.255
              inet6 b🅱c:d:6676%vtnet2 prefixlen 64 scopeid 0x3
              inet6 a🅱c:d:2::2 prefixlen 80
              nd6 options=1 <performnud>media: Ethernet 1000baseT <full-duplex>status: active
      lo0: flags=8049 <up,loopback,running,multicast>metric 0 mtu 16384
              options=3 <rxcsum,txcsum>inet 127.0.0.1 netmask 0xff000000
              inet6 ::1 prefixlen 128
              inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4
              nd6 options=3 <performnud,accept_rtadv>pfsync0: flags=0<> metric 0 mtu 1460
              syncpeer: 224.0.0.240 maxupd: 128 syncok: 1
      enc0: flags=0<> metric 0 mtu 1536
      pflog0: flags=100 <promisc>metric 0 mtu 33192</promisc></performnud,accept_rtadv></rxcsum,txcsum></up,loopback,running,multicast></full-duplex></performnud></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso6,vlan_hwtso,linkstate></up,broadcast,running,simplex,multicast></full-duplex></performnud></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso6,vlan_hwtso,linkstate></up,broadcast,running,simplex,multicast></full-duplex></performnud></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso6,vlan_hwtso,linkstate></up,broadcast,running,simplex,multicast>

      
      Thanks in advance.
      
      Helmut
      1 Reply Last reply Reply Quote 0
      • M
        mcdiesel
        last edited by

        This is an incredibly useful post.

        I've just started deploying pfSense on more powerful yet still small Jetway NF9HG boards.  Plenty of power to run pfSense, so I'ved added Proxmox underneath as well.

        No end of troubles with traffic - ping works, voip works, ssh doesn't. Configs all good, no errors.  Users experience good downloads, awful or non-existent uploads.

        Wireshark captures show lots of retransmission.

        Taken hours to find this post, since the issue isn't particularly obvious.  Wondering now if it was related to 'disable hardware checksum offloading', as I can't imagine the FreeBSD vtnet drivers are this broken.

        1 Reply Last reply Reply Quote 0
        • T
          Tolman
          last edited by

          nut, seriously dude … i just love u  ;)

          I lost 2 evenings on my configuration because of this !!!!

          Ok, it has been mentioned at the very end but it has to be written on 32 font size blinking red on this page :
          https://pve.proxmox.com/wiki/PfSense_Guest_Notes

          And apparently you can still use virtIO driver following steps on the link :
          https://doc.pfsense.org/index.php/VirtIO_Driver_Support

          Thanks again

          1 Reply Last reply Reply Quote 0
          • B
            booruszuru
            last edited by

            pls show me your full /etc/network/interfaces file from proxmox host
            i cant seem to make it work plssssssssssssssssssssssssss 😢 😢 😢 😢

            1 Reply Last reply Reply Quote 0
            • First post
              Last post
            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.