Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    pfSense not enabling port

    Scheduled Pinned Locked Moved
    General pfSense Questions
    4
    145
    3.9k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • georgelzaG
      georgelza @stephenw10
      last edited by

      @stephenw10
      well it seems I can now ping 172.16.30.1, from both the pmox shell, I also created a ubuntu guest vm, i can from inside here ping the 30.1 also

      so we def making progress.

      i've down a ifconfig <interface> down and a back up, not picking up a ip yet.

      G

      G 1 Reply Last reply Reply Quote 0
      • G
        Gblenn @georgelza
        last edited by

        @georgelza said in pfSense not enabling port:

        @Gblenn said in pfSense not enabling port:

        dmesg | grep ixgbe

        [175483.486743] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Down
[175484.110776] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[175484.214719] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Down
[175485.462768] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[175485.670734] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Down
[175485.879762] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[175485.982711] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Down
[175486.398837] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[175486.502809] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Down
[175486.606757] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[175486.710853] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Down
[175487.230749] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[175487.438681] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Down
[175489.830832] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[175489.934714] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Down
[175498.150577] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[175498.255548] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Down
[175502.414503] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[175502.518579] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Down

        It would be interesting to see what it sais before it start flapping like that... Of there is anything that could provide a clue whether there is some module inocmpatibility for example. Some cards check the module to see if it is in the list of "supported devices", which it will announce here if that is the case...

        I wonder if you remove the module, issue dmesg | grep ixgbe again.
        Then plug it in and immediately run dmesg | grep ixgbe

        1 Reply Last reply Reply Quote 0
        • G
          Gblenn @georgelza
          last edited by Gblenn

          @georgelza said in pfSense not enabling port:

          @stephenw10
          well it seems I can now ping 172.16.30.1, from both the pmox shell, I also created a ubuntu guest vm, i can from inside here ping the 30.1 also

          so we def making progress.

          i've down a ifconfig <interface> down and a back up, not picking up a ip yet.

          G

          Proxmox should be able to ping 172.16.30.1, since that is allowed according to your rules, and it is pfsense responding on that IP. You are likely pinging from 172.16.10 which is the Proxmox host, and not the ix0 interface you are trying to fix.
          From the VM however, if you are able to ping, you should be good to go.

          What does dmesg | grep ixgbe on Proxmox host show you now?

          georgelzaG 1 Reply Last reply Reply Quote 0
          • georgelzaG
            georgelza @Gblenn
            last edited by georgelza

            @Gblenn

            [178207.262591] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Down
[178207.470686] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[178207.574588] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Down
[178207.782573] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[178207.991549] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Down
[178208.094751] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[178208.198712] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Down
[178209.342607] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[178209.446782] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Down
[178210.902621] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[178211.006577] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Down
[178212.462795] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[178212.566761] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Down
[178215.478660] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[178215.790564] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Down
[178218.702566] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[178218.910620] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Down
[178219.118550] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[178219.222534] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Down
[178219.430550] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX

            this is the guest vm created

            ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: ens18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether bc:24:11:c0:e8:1e brd ff:ff:ff:ff:ff:ff
    inet 172.16.10.201/24 metric 100 brd 172.16.10.255 scope global dynamic ens18
       valid_lft 5516sec preferred_lft 5516sec
    inet6 fe80::be24:11ff:fec0:e81e/64 scope link
       valid_lft forever preferred_lft forever
4: ens19: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether bc:24:11:8f:4d:2a brd ff:ff:ff:ff:ff:ff
    inet6 fe80::be24:11ff:fe8f:4d2a/64 scope link
       valid_lft forever preferred_lft forever

            and then the pmox shell

            root@pmox1:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UP group default qlen 1000
    link/ether a8:b8:e0:02:a3:71 brd ff:ff:ff:ff:ff:ff
3: enp3s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether a8:b8:e0:02:a3:72 brd ff:ff:ff:ff:ff:ff
4: enp5s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether a8:b8:e0:02:a3:73 brd ff:ff:ff:ff:ff:ff
5: enp6s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether a8:b8:e0:02:a3:74 brd ff:ff:ff:ff:ff:ff
6: enp4s0f0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master vmbr30 state DOWN group default qlen 1000
    link/ether a8:b8:e0:05:f0:91 brd ff:ff:ff:ff:ff:ff
7: enp4s0f1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether a8:b8:e0:05:f0:92 brd ff:ff:ff:ff:ff:ff
8: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether a8:b8:e0:02:a3:71 brd ff:ff:ff:ff:ff:ff
    inet 172.16.10.51/24 scope global vmbr0
       valid_lft forever preferred_lft forever
    inet6 fe80::aab8:e0ff:fe02:a371/64 scope link 
       valid_lft forever preferred_lft forever
10: vmbr30: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether a8:b8:e0:05:f0:91 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::aab8:e0ff:fe05:f091/64 scope link 
       valid_lft forever preferred_lft forever
15: tap100i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master fwbr100i0 state UNKNOWN group default qlen 1000
    link/ether fe:6b:ee:5d:ac:58 brd ff:ff:ff:ff:ff:ff
16: fwbr100i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether ba:54:5b:a0:3b:26 brd ff:ff:ff:ff:ff:ff
17: fwpr100p0@fwln100i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP group default qlen 1000
    link/ether de:8e:79:f1:ec:bd brd ff:ff:ff:ff:ff:ff
18: fwln100i0@fwpr100p0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr100i0 state UP group default qlen 1000
    link/ether ba:54:5b:a0:3b:26 brd ff:ff:ff:ff:ff:ff
19: tap100i1: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master fwbr100i1 state UNKNOWN group default qlen 1000
    link/ether 46:27:5f:a9:be:cc brd ff:ff:ff:ff:ff:ff
20: fwbr100i1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether b6:76:d9:f3:3c:fe brd ff:ff:ff:ff:ff:ff
21: fwpr100p1@fwln100i1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr30 state UP group default qlen 1000
    link/ether 96:3e:4f:c9:f2:84 brd ff:ff:ff:ff:ff:ff
22: fwln100i1@fwpr100p1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr100i1 state UP group default qlen 1000
    link/ether b6:76:d9:f3:3c:fe brd ff:ff:ff:ff:ff:ff
            G georgelzaG 3 Replies Last reply Reply Quote 0
            • G
              Gblenn @georgelza
              last edited by

              Ok this is new and positive...

              10: vmbr30: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
              link/ether a8:b8:e0:05:f0:91 brd ff:ff:ff:ff:ff:ff

              However it is strange the network device still sais NO-CARRIER...

              6: enp4s0f0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master vmbr30 state DOWN group default qlen 1000
              link/ether a8:b8:e0:05:f0:91 brd ff:ff:ff:ff:ff:ff

              1 Reply Last reply Reply Quote 1
              • georgelzaG
                georgelza @georgelza
                last edited by

                from the guest vm

                ping 172.16.30.1
PING 172.16.30.1 (172.16.30.1) 56(84) bytes of data.
64 bytes from 172.16.30.1: icmp_seq=1 ttl=64 time=0.264 ms
64 bytes from 172.16.30.1: icmp_seq=2 ttl=64 time=0.282 ms
64 bytes from 172.16.30.1: icmp_seq=3 ttl=64 time=0.389 ms
64 bytes from 172.16.30.1: icmp_seq=4 ttl=64 time=0.338 ms
64 bytes from 172.16.30.1: icmp_seq=5 ttl=64 time=0.276 ms
64 bytes from 172.16.30.1: icmp_seq=6 ttl=64 time=0.204 ms
^C
--- 172.16.30.1 ping statistics ---
6 packets transmitted, 6 received, 0% packet loss, time 5110ms
rtt min/avg/max/mdev = 0.204/0.292/0.389/0.058 ms
ifconfig -a
docker0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 172.17.0.1  netmask 255.255.0.0  broadcast 172.17.255.255
        ether 02:42:ca:03:51:5d  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ens18: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.16.10.201  netmask 255.255.255.0  broadcast 172.16.10.255
        inet6 fe80::be24:11ff:fec0:e81e  prefixlen 64  scopeid 0x20<link>
        ether bc:24:11:c0:e8:1e  txqueuelen 1000  (Ethernet)
        RX packets 167843  bytes 270486832 (270.4 MB)
        RX errors 0  dropped 44  overruns 0  frame 0
        TX packets 44953  bytes 3636555 (3.6 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ens19: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::be24:11ff:fe8f:4d2a  prefixlen 64  scopeid 0x20<link>
        ether bc:24:11:8f:4d:2a  txqueuelen 1000  (Ethernet)
        RX packets 34  bytes 2090 (2.0 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 26  bytes 2012 (2.0 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 152  bytes 13984 (13.9 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 152  bytes 13984 (13.9 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

                from my the pmox1 shell

                Linux pmox1 6.8.4-2-pve #1 SMP PREEMPT_DYNAMIC PMX 6.8.4-2 (2024-04-10T17:36Z) x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Thu Oct 31 19:44:44 SAST 2024 on pts/1
root@pmox1:~# ping 172.16.30.1
PING 172.16.30.1 (172.16.30.1) 56(84) bytes of data.
64 bytes from 172.16.30.1: icmp_seq=1 ttl=64 time=0.184 ms
64 bytes from 172.16.30.1: icmp_seq=2 ttl=64 time=0.129 ms
^C
--- 172.16.30.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1062ms
rtt min/avg/max/mdev = 0.129/0.156/0.184/0.027 ms
root@pmox1:~# 
root@pmox1:~# 
root@pmox1:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UP group default qlen 1000
    link/ether a8:b8:e0:02:a3:71 brd ff:ff:ff:ff:ff:ff
3: enp3s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether a8:b8:e0:02:a3:72 brd ff:ff:ff:ff:ff:ff
4: enp5s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether a8:b8:e0:02:a3:73 brd ff:ff:ff:ff:ff:ff
5: enp6s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether a8:b8:e0:02:a3:74 brd ff:ff:ff:ff:ff:ff
6: enp4s0f0: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master vmbr30 state UP group default qlen 1000
    link/ether a8:b8:e0:05:f0:91 brd ff:ff:ff:ff:ff:ff
7: enp4s0f1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether a8:b8:e0:05:f0:92 brd ff:ff:ff:ff:ff:ff
8: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether a8:b8:e0:02:a3:71 brd ff:ff:ff:ff:ff:ff
    inet 172.16.10.51/24 scope global vmbr0
       valid_lft forever preferred_lft forever
    inet6 fe80::aab8:e0ff:fe02:a371/64 scope link 
       valid_lft forever preferred_lft forever
10: vmbr30: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether a8:b8:e0:05:f0:91 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::aab8:e0ff:fe05:f091/64 scope link 
       valid_lft forever preferred_lft forever
15: tap100i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master fwbr100i0 state UNKNOWN group default qlen 1000
    link/ether fe:6b:ee:5d:ac:58 brd ff:ff:ff:ff:ff:ff
16: fwbr100i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether ba:54:5b:a0:3b:26 brd ff:ff:ff:ff:ff:ff
17: fwpr100p0@fwln100i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP group default qlen 1000
    link/ether de:8e:79:f1:ec:bd brd ff:ff:ff:ff:ff:ff
18: fwln100i0@fwpr100p0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr100i0 state UP group default qlen 1000
    link/ether ba:54:5b:a0:3b:26 brd ff:ff:ff:ff:ff:ff
19: tap100i1: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master fwbr100i1 state UNKNOWN group default qlen 1000
    link/ether 46:27:5f:a9:be:cc brd ff:ff:ff:ff:ff:ff
20: fwbr100i1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether b6:76:d9:f3:3c:fe brd ff:ff:ff:ff:ff:ff
21: fwpr100p1@fwln100i1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr30 state UP group default qlen 1000
    link/ether 96:3e:4f:c9:f2:84 brd ff:ff:ff:ff:ff:ff
22: fwln100i1@fwpr100p1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr100i1 state UP group default qlen 1000
    link/ether b6:76:d9:f3:3c:fe brd ff:ff:ff:ff:ff:ff
root@pmox1:~#
                1 Reply Last reply Reply Quote 0
                • G
                  Gblenn @georgelza
                  last edited by Gblenn

                  @georgelza BTW, you'r VM is showing two interfaces... did you assign more than one?
                  The one with an IP is on the 10.1 network, so perhaps it's the 2.5G bridge that is working.

                  No IP on the other one though...

                  ip a
                  1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
                  link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
                  inet 127.0.0.1/8 scope host lo
                  valid_lft forever preferred_lft forever
                  inet6 ::1/128 scope host
                  valid_lft forever preferred_lft forever
                  2: ens18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
                  link/ether bc:24:11:c0:e8:1e brd ff:ff:ff:ff:ff:ff
                  inet 172.16.10.201/24 metric 100 brd 172.16.10.255 scope global dynamic ens18
                  valid_lft 5516sec preferred_lft 5516sec
                  inet6 fe80::be24:11ff:fec0:e81e/64 scope link
                  valid_lft forever preferred_lft forever
                  4: ens19: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
                  link/ether bc:24:11:8f:4d:2a brd ff:ff:ff:ff:ff:ff
                  inet6 fe80::be24:11ff:fe8f:4d2a/64 scope link
                  valid_lft forever preferred_lft forever

                  georgelzaG 1 Reply Last reply Reply Quote 0
                  • georgelzaG
                    georgelza @Gblenn
                    last edited by

                    @Gblenn

                    correct.
                    172.16.10.51 is the pmox1 address,
                    172.16.10.201 is the guest vm, ubuntu1 dynamic assigned ip.
                    these live/transverse the 2.5GbE links.

                    we're trying to get the fiber working which would sit on
                    172.16.30.0 network. as you noticed, we can ping the 172.6.30.1 gw now, but not getting a ip assigned as expected.

                    let me check pfSense settings for this vlan quickly.

                    G

                    1 Reply Last reply Reply Quote 0
                    • georgelzaG
                      georgelza
                      last edited by

                      dmesg | grep ixgbe is still flapping

                      G

                      1 Reply Last reply Reply Quote 0
                      • georgelzaG
                        georgelza
                        last edited by

                        pfSense is def enabled to run dhcp server on the vlan and got a ranged assigned.

                        once it picks up a ip then i can, reassign/static reserve a ip.

                        G

                        1 Reply Last reply Reply Quote 0
                        • georgelzaG
                          georgelza
                          last edited by

                          lets park this for the evening, think we've made awesome progress.

                          got the link activated. :) :)

                          now to get the ip assignment working, which sits with pfSense.

                          G

                          1 Reply Last reply Reply Quote 0
                          • georgelzaG
                            georgelza
                            last edited by georgelza

                            morning all...

                            ok... as we made some progress and learned some things along the way, or make that me learn... did do some changes.

                            port0 / en2ps0 on topton is now 172.16.10.50, configured into vmbr0
                            created a vlan30 which is configured to run on vmbr0, with a 172.16.30.0/24 range. this will be the cluster backplane, and vmotion network.
                            The above is on the 2.5GbE Cat5

                            then enp4s0f0 is port0 on the 10GbE, configured not vmbr40 to host 172.16.40.0/24

                            I can ping out from the guest vm ubuntu01, configured with ip 172.16.10.51
                            default gw îs 172.16.10.1

                            issue... neither the vlan30 nor vlan40, the fiber is getting a ip from dhcp from the pfsense, and yes dhcp is enabled and a range is assigned ;)

                            root@ubuntu-01:~# ping 172.16.10.1
PING 172.16.10.1 (172.16.10.1) 56(84) bytes of data.
64 bytes from 172.16.10.1: icmp_seq=1 ttl=64 time=0.301 ms
64 bytes from 172.16.10.1: icmp_seq=2 ttl=64 time=0.384 ms
^C
--- 172.16.10.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1028ms
rtt min/avg/max/mdev = 0.301/0.342/0.384/0.041 ms
root@ubuntu-01:~# ping 172.16.30.1
PING 172.16.30.1 (172.16.30.1) 56(84) bytes of data.
64 bytes from 172.16.30.1: icmp_seq=1 ttl=64 time=0.240 ms
64 bytes from 172.16.30.1: icmp_seq=2 ttl=64 time=0.412 ms
64 bytes from 172.16.30.1: icmp_seq=3 ttl=64 time=0.404 ms

64 bytes from 172.16.30.1: icmp_seq=4 ttl=64 time=0.385 ms
64 bytes from 172.16.30.1: icmp_seq=5 ttl=64 time=0.365 ms
^C
--- 172.16.30.1 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4093ms
rtt min/avg/max/mdev = 0.240/0.361/0.412/0.062 ms
root@ubuntu-01:~# ping 172.16.40.1
PING 172.16.40.1 (172.16.40.1) 56(84) bytes of data.
64 bytes from 172.16.40.1: icmp_seq=1 ttl=64 time=0.282 ms
64 bytes from 172.16.40.1: icmp_seq=2 ttl=64 time=0.366 ms
64 bytes from 172.16.40.1: icmp_seq=3 ttl=64 time=0.409 ms
64 bytes from 172.16.40.1: icmp_seq=4 ttl=64 time=0.438 ms
^C
--- 172.16.40.1 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3077ms
rtt min/avg/max/mdev = 0.282/0.373/0.438/0.058 ms
root@ubuntu-01:~#
root@ubuntu-01:~#
root@ubuntu-01:~#
root@ubuntu-01:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: ens18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether bc:24:11:c0:e8:1e brd ff:ff:ff:ff:ff:ff
    inet 172.16.10.51/24 metric 100 brd 172.16.10.255 scope global dynamic ens18
       valid_lft 6409sec preferred_lft 6409sec
    inet6 fe80::be24:11ff:fec0:e81e/64 scope link
       valid_lft forever preferred_lft forever
3: ens19: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether bc:24:11:81:2c:cb brd ff:ff:ff:ff:ff:ff
4: ens20: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether bc:24:11:1e:20:ac brd ff:ff:ff:ff:ff:ff
5: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
    link/ether 02:42:cb:d9:fb:26 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever
root@ubuntu-01:~#
root@ubuntu-01:~#
root@ubuntu-01:~#
root@ubuntu-01:~# ifconfig -a
docker0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 172.17.0.1  netmask 255.255.0.0  broadcast 172.17.255.255
        ether 02:42:cb:d9:fb:26  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ens18: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.16.10.51  netmask 255.255.255.0  broadcast 172.16.10.255
        inet6 fe80::be24:11ff:fec0:e81e  prefixlen 64  scopeid 0x20<link>
        ether bc:24:11:c0:e8:1e  txqueuelen 1000  (Ethernet)
        RX packets 77062  bytes 87887828 (87.8 MB)
        RX errors 0  dropped 29  overruns 0  frame 0
        TX packets 18581  bytes 1826196 (1.8 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ens19: flags=4098<BROADCAST,MULTICAST>  mtu 1500
        ether bc:24:11:81:2c:cb  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ens20: flags=4098<BROADCAST,MULTICAST>  mtu 1500
        ether bc:24:11:1e:20:ac  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 124  bytes 10936 (10.9 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 124  bytes 10936 (10.9 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

root@ubuntu-01:~#

                            15505677-9b9b-4fd8-95b0-105e3e2c4ddd-image.png

                            ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute
       valid_lft forever preferred_lft forever
2: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UP group default qlen 1000
    link/ether a8:b8:e0:02:a3:71 brd ff:ff:ff:ff:ff:ff
3: enp3s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether a8:b8:e0:02:a3:72 brd ff:ff:ff:ff:ff:ff
4: enp5s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether a8:b8:e0:02:a3:73 brd ff:ff:ff:ff:ff:ff
5: enp6s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether a8:b8:e0:02:a3:74 brd ff:ff:ff:ff:ff:ff
6: enp4s0f0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master vmbr40 state DOWN group default qlen 1000
    link/ether a8:b8:e0:05:f0:91 brd ff:ff:ff:ff:ff:ff
7: enp4s0f1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether a8:b8:e0:05:f0:92 brd ff:ff:ff:ff:ff:ff
8: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether a8:b8:e0:02:a3:71 brd ff:ff:ff:ff:ff:ff
    inet 172.16.10.50/24 scope global vmbr0
       valid_lft forever preferred_lft forever
    inet6 fe80::aab8:e0ff:fe02:a371/64 scope link
       valid_lft forever preferred_lft forever
9: vmbr40: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether a8:b8:e0:05:f0:91 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::aab8:e0ff:fe05:f091/64 scope link
       valid_lft forever preferred_lft forever
10: tap100i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master fwbr100i0 state UNKNOWN group default qlen 1000
    link/ether c6:2d:5a:7a:41:d7 brd ff:ff:ff:ff:ff:ff
11: fwbr100i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether da:0c:52:e1:39:89 brd ff:ff:ff:ff:ff:ff
12: fwpr100p0@fwln100i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP group default qlen 1000
    link/ether 76:c4:56:d3:6a:93 brd ff:ff:ff:ff:ff:ff
13: fwln100i0@fwpr100p0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr100i0 state UP group default qlen 1000
    link/ether da:0c:52:e1:39:89 brd ff:ff:ff:ff:ff:ff
14: tap100i1: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master fwbr100i1 state UNKNOWN group default qlen 1000
    link/ether 76:c9:86:22:cc:30 brd ff:ff:ff:ff:ff:ff
15: fwbr100i1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 52:9a:1c:4d:76:a6 brd ff:ff:ff:ff:ff:ff
16: fwpr100p1@fwln100i1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP group default qlen 1000
    link/ether 26:2d:af:5a:9a:b4 brd ff:ff:ff:ff:ff:ff
17: fwln100i1@fwpr100p1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr100i1 state UP group default qlen 1000
    link/ether 52:9a:1c:4d:76:a6 brd ff:ff:ff:ff:ff:ff
18: tap100i2: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master fwbr100i2 state UNKNOWN group default qlen 1000
    link/ether c2:9f:1f:4e:c6:2e brd ff:ff:ff:ff:ff:ff
19: fwbr100i2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 7a:a8:67:a9:29:14 brd ff:ff:ff:ff:ff:ff
20: fwpr100p2@fwln100i2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr40 state UP group default qlen 1000
    link/ether 72:43:cc:4b:b3:40 brd ff:ff:ff:ff:ff:ff
21: fwln100i2@fwpr100p2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr100i2 state UP group default qlen 1000
    link/ether 7a:a8:67:a9:29:14 brd ff:ff:ff:ff:ff:ff
22: vlan30@vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether a8:b8:e0:02:a3:71 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::aab8:e0ff:fe02:a371/64 scope link
       valid_lft forever preferred_lft forever
                            G 1 Reply Last reply Reply Quote 0
                            • G
                              Gblenn @georgelza
                              last edited by

                              @georgelza Yes it seems like the problem exists also in Proxmox. Where I was going with the dmesg command was to figure out if there was any additional information about e.g. incompatible module. Do you know what chipset Topton is using for the NIC's?
                              There is a (small) chance that the nw-card will not accept an "unsupported" module. This can be solved as per the first post in this thread: https://forum.proxmox.com/threads/unsupported-sfp-module.84555/

                              Basically it's an override that tells the card not to check the module...

                              Simply add the file /etc/modprobe.d/ixgbe.conf if it doesn't exist. And add the one row
                              options ixgbe allow_unsupported_sfp=1
                              Not sure if it should be =1 or =1,1 (since it's a dual port card).

                              georgelzaG 2 Replies Last reply Reply Quote 0
                              • georgelzaG
                                georgelza @Gblenn
                                last edited by

                                @Gblenn said in pfSense not enabling port:

                                ou know what chipset Topton is using for the NIC's?

                                trying hard to find out, but talking to AliExpress chinese not easy. all i got was a install the following driver, and a rar file, which then extracts to a exe... slight problem i'm on a mac, and there is no windows machines in the house, and i'm not sure building a windows vm would work.

                                G

                                1 Reply Last reply Reply Quote 0
                                • georgelzaG
                                  georgelza @Gblenn
                                  last edited by

                                  @Gblenn

                                  after the above.

                                  what i have noticed though, my port on the switch has gone red, complaining of packet loss etc.
                                  wonder if we need to go back to auto negotiate.
                                  ip a
                                  i did have to bring the interfaces up, both the vlan30 that sits on vmbr0 and vmbr40 was down.

                                  1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute
       valid_lft forever preferred_lft forever
2: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UP group default qlen 1000
    link/ether a8:b8:e0:02:a3:71 brd ff:ff:ff:ff:ff:ff
3: enp3s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether a8:b8:e0:02:a3:72 brd ff:ff:ff:ff:ff:ff
4: enp5s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether a8:b8:e0:02:a3:73 brd ff:ff:ff:ff:ff:ff
5: enp6s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether a8:b8:e0:02:a3:74 brd ff:ff:ff:ff:ff:ff
8: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether a8:b8:e0:02:a3:71 brd ff:ff:ff:ff:ff:ff
    inet 172.16.10.50/24 scope global vmbr0
       valid_lft forever preferred_lft forever
    inet6 fe80::aab8:e0ff:fe02:a371/64 scope link
       valid_lft forever preferred_lft forever
9: vmbr40: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether a8:b8:e0:05:f0:91 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::aab8:e0ff:fe05:f091/64 scope link
       valid_lft forever preferred_lft forever
10: tap100i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master fwbr100i0 state UNKNOWN group default qlen 1000
    link/ether c6:2d:5a:7a:41:d7 brd ff:ff:ff:ff:ff:ff
11: fwbr100i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether da:0c:52:e1:39:89 brd ff:ff:ff:ff:ff:ff
12: fwpr100p0@fwln100i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP group default qlen 1000
    link/ether 76:c4:56:d3:6a:93 brd ff:ff:ff:ff:ff:ff
13: fwln100i0@fwpr100p0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr100i0 state UP group default qlen 1000
    link/ether da:0c:52:e1:39:89 brd ff:ff:ff:ff:ff:ff
14: tap100i1: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master fwbr100i1 state UNKNOWN group default qlen 1000
    link/ether 76:c9:86:22:cc:30 brd ff:ff:ff:ff:ff:ff
15: fwbr100i1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 52:9a:1c:4d:76:a6 brd ff:ff:ff:ff:ff:ff
16: fwpr100p1@fwln100i1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP group default qlen 1000
    link/ether 26:2d:af:5a:9a:b4 brd ff:ff:ff:ff:ff:ff
17: fwln100i1@fwpr100p1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr100i1 state UP group default qlen 1000
    link/ether 52:9a:1c:4d:76:a6 brd ff:ff:ff:ff:ff:ff
18: tap100i2: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master fwbr100i2 state UNKNOWN group default qlen 1000
    link/ether c2:9f:1f:4e:c6:2e brd ff:ff:ff:ff:ff:ff
19: fwbr100i2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 7a:a8:67:a9:29:14 brd ff:ff:ff:ff:ff:ff
20: fwpr100p2@fwln100i2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr40 state UP group default qlen 1000
    link/ether 72:43:cc:4b:b3:40 brd ff:ff:ff:ff:ff:ff
21: fwln100i2@fwpr100p2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr100i2 state UP group default qlen 1000
    link/ether 7a:a8:67:a9:29:14 brd ff:ff:ff:ff:ff:ff
22: vlan30@vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether a8:b8:e0:02:a3:71 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::aab8:e0ff:fe02:a371/64 scope link
       valid_lft forever preferred_lft forever
23: enp4s0f0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether a8:b8:e0:05:f0:91 brd ff:ff:ff:ff:ff:ff
24: enp4s0f1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether a8:b8:e0:05:f0:92 brd ff:ff:ff:ff:ff:ff

                                  dmesg | grep ixgbe

                                  [19185.217533] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[19185.321424] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Down
[19185.529459] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[19185.633445] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Down
[19185.862491] ixgbe 0000:04:00.1: complete
[19185.862751] ixgbe 0000:04:00.0: removed PHC on enp4s0f0
[19185.922172] ixgbe 0000:04:00.0 enp4s0f0 (unregistering): left allmulticast mode
[19185.922174] ixgbe 0000:04:00.0 enp4s0f0 (unregistering): left promiscuous mode
[19185.984591] ixgbe 0000:04:00.0: complete
[19186.014865] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver
[19186.014867] ixgbe: Copyright (c) 1999-2016 Intel Corporation.
[19186.183424] ixgbe 0000:04:00.0: Multiqueue Enabled: Rx Queue count = 12, Tx Queue count = 12 XDP Queue count = 0
[19186.183731] ixgbe 0000:04:00.0: 16.000 Gb/s available PCIe bandwidth, limited by 5.0 GT/s PCIe x4 link at 0000:00:1c.4 (capable of 32.000 Gb/s with 5.0 GT/s PCIe x8 link)
[19186.183816] ixgbe 0000:04:00.0: MAC: 2, PHY: 19, SFP+: 5, PBA No: FFFFFF-0FF
[19186.183817] ixgbe 0000:04:00.0: a8:b8:e0:05:f0:91
[19186.184842] ixgbe 0000:04:00.0: Intel(R) 10 Gigabit Network Connection
[19186.195416] ixgbe 0000:04:00.0 enp4s0f0: renamed from eth0
[19187.336359] ixgbe 0000:04:00.1: Multiqueue Enabled: Rx Queue count = 12, Tx Queue count = 12 XDP Queue count = 0
[19187.336664] ixgbe 0000:04:00.1: 16.000 Gb/s available PCIe bandwidth, limited by 5.0 GT/s PCIe x4 link at 0000:00:1c.4 (capable of 32.000 Gb/s with 5.0 GT/s PCIe x8 link)
[19187.336749] ixgbe 0000:04:00.1: MAC: 2, PHY: 1, PBA No: FFFFFF-0FF
[19187.336750] ixgbe 0000:04:00.1: a8:b8:e0:05:f0:92
[19187.337749] ixgbe 0000:04:00.1: Intel(R) 10 Gigabit Network Connection
[19187.353403] ixgbe 0000:04:00.1 enp4s0f1: renamed from eth0
                                  georgelzaG 1 Reply Last reply Reply Quote 0
                                  • georgelzaG
                                    georgelza @georgelza
                                    last edited by

                                    on my switch side i run

                                    172.16.10.0/24 as my primary on the physical port
                                    then have the following vlans defined atm
                                    (running on the same above physical port)
                                    vlan20 - 172.16.20.0/24 -> all wifi devices
                                    vlan30 - 172.16.30.0/24 -> will be "vmotion/k8s cluster interconnect"

                                    vlan40 - 172.16.40.0/24 -> will be storage/io to be used by all to access the TrueNAS. 10GbE based on dedicated port.

                                    i've configured vmbr0 and vmbr40.
                                    i ended configuring vlan30 as a vlan on vmbr0.

                                    G

                                    G 1 Reply Last reply Reply Quote 0
                                    • G
                                      Gblenn @georgelza
                                      last edited by

                                      @georgelza Did you unplug the module from the SPF+ port on the Proxmox machine? What happens if you plug it in again, only the Proxmox side and leave the switch side hanging. Anything new that shows up from dmesg?

                                      georgelzaG 1 Reply Last reply Reply Quote 0
                                      • georgelzaG
                                        georgelza @Gblenn
                                        last edited by

                                        @Gblenn

                                        did not touch the physical side.

                                        i've since rechecked the switch and reset to manual and redefined it as 10GbE DFX

                                        re-issued the ethtool command on the pmox level.

                                        G

                                        georgelzaG 1 Reply Last reply Reply Quote 0
                                        • georgelzaG
                                          georgelza @georgelza
                                          last edited by georgelza

                                          @georgelza

                                          ok, got it back active on the switch.

                                          noticed that in pmox under network it was active=no, switched it to autostart and it seemed to have figured out to start itself.

                                          G

                                          root@ubuntu-01:~# ifconfig -a
docker0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 172.17.0.1  netmask 255.255.0.0  broadcast 172.17.255.255
        ether 02:42:cb:d9:fb:26  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ens18: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.16.10.51  netmask 255.255.255.0  broadcast 172.16.10.255
        inet6 fe80::be24:11ff:fec0:e81e  prefixlen 64  scopeid 0x20<link>
        ether bc:24:11:c0:e8:1e  txqueuelen 1000  (Ethernet)
        RX packets 124398  bytes 94163032 (94.1 MB)
        RX errors 0  dropped 64  overruns 0  frame 0
        TX packets 22096  bytes 2425895 (2.4 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ens19: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::be24:11ff:fe81:2ccb  prefixlen 64  scopeid 0x20<link>
        ether bc:24:11:81:2c:cb  txqueuelen 1000  (Ethernet)
        RX packets 22  bytes 1566 (1.5 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 15  bytes 1146 (1.1 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ens20: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::be24:11ff:fe1e:20ac  prefixlen 64  scopeid 0x20<link>
        ether bc:24:11:1e:20:ac  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 15  bytes 1146 (1.1 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 128  bytes 11444 (11.4 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 128  bytes 11444 (11.4 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

root@ubuntu-01:~

                                          ip a

                                          root@ubuntu-01:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: ens18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether bc:24:11:c0:e8:1e brd ff:ff:ff:ff:ff:ff
    inet 172.16.10.51/24 metric 100 brd 172.16.10.255 scope global dynamic ens18
       valid_lft 5917sec preferred_lft 5917sec
    inet6 fe80::be24:11ff:fec0:e81e/64 scope link
       valid_lft forever preferred_lft forever
3: ens19: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether bc:24:11:81:2c:cb brd ff:ff:ff:ff:ff:ff
    inet6 fe80::be24:11ff:fe81:2ccb/64 scope link
       valid_lft forever preferred_lft forever
4: ens20: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether bc:24:11:1e:20:ac brd ff:ff:ff:ff:ff:ff
    inet6 fe80::be24:11ff:fe1e:20ac/64 scope link
       valid_lft forever preferred_lft forever
5: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
    link/ether 02:42:cb:d9:fb:26 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever
root@ubuntu-01:~#

                                          dmesg, flapping again

                                          [23193.684199] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Down
[23196.284232] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[23196.492219] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Down
[23196.700248] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[23196.804186] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Down
[23197.012247] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[23197.116191] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Down
[23197.325189] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[23197.428371] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Down
[23199.924253] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[23200.028227] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Down
[23200.756251] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[23200.860223] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Down
[23200.964189] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[23201.068135] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Down
[23201.172183] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[23201.276115] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Down
[23201.484251] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[23201.588222] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Down
[23203.668163] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[23203.772131] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Down
[23204.916258] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[23205.020120] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Down
[23205.228162] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[23205.332137] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Down
[23205.540156] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[23205.644121] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Down
[23205.748154] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[23205.956212] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Down
[23206.060156] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[23206.268107] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Down

                                          then issued the ethtool command, which made my switch port go red.

                                          then dmesg changed to

                                          [23352.570211] ixgbe 0000:04:00.0: complete
[23352.608227] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver
[23352.608229] ixgbe: Copyright (c) 1999-2016 Intel Corporation.
[23352.776181] ixgbe 0000:04:00.0: Multiqueue Enabled: Rx Queue count = 12, Tx Queue count = 12 XDP Queue count = 0
[23352.776488] ixgbe 0000:04:00.0: 16.000 Gb/s available PCIe bandwidth, limited by 5.0 GT/s PCIe x4 link at 0000:00:1c.4 (capable of 32.000 Gb/s with 5.0 GT/s PCIe x8 link)
[23352.776573] ixgbe 0000:04:00.0: MAC: 2, PHY: 19, SFP+: 5, PBA No: FFFFFF-0FF
[23352.776574] ixgbe 0000:04:00.0: a8:b8:e0:05:f0:91
[23352.777559] ixgbe 0000:04:00.0: Intel(R) 10 Gigabit Network Connection
[23352.785028] ixgbe 0000:04:00.0 enp4s0f0: renamed from eth0
[23353.930084] ixgbe 0000:04:00.1: Multiqueue Enabled: Rx Queue count = 12, Tx Queue count = 12 XDP Queue count = 0
[23353.930388] ixgbe 0000:04:00.1: 16.000 Gb/s available PCIe bandwidth, limited by 5.0 GT/s PCIe x4 link at 0000:00:1c.4 (capable of 32.000 Gb/s with 5.0 GT/s PCIe x8 link)
[23353.930472] ixgbe 0000:04:00.1: MAC: 2, PHY: 1, PBA No: FFFFFF-0FF
[23353.930473] ixgbe 0000:04:00.1: a8:b8:e0:05:f0:92
[23353.931466] ixgbe 0000:04:00.1: Intel(R) 10 Gigabit Network Connection
[23353.947017] ixgbe 0000:04:00.1 enp4s0f1: renamed from eth0
[23367.436987] ixgbe 0000:04:00.1: complete
[23367.446889] ixgbe 0000:04:00.0: complete
[23367.483630] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver
[23367.483632] ixgbe: Copyright (c) 1999-2016 Intel Corporation.
[23367.652922] ixgbe 0000:04:00.0: Multiqueue Enabled: Rx Queue count = 12, Tx Queue count = 12 XDP Queue count = 0
[23367.653228] ixgbe 0000:04:00.0: 16.000 Gb/s available PCIe bandwidth, limited by 5.0 GT/s PCIe x4 link at 0000:00:1c.4 (capable of 32.000 Gb/s with 5.0 GT/s PCIe x8 link)
[23367.653313] ixgbe 0000:04:00.0: MAC: 2, PHY: 19, SFP+: 5, PBA No: FFFFFF-0FF
[23367.653314] ixgbe 0000:04:00.0: a8:b8:e0:05:f0:91
[23367.654319] ixgbe 0000:04:00.0: Intel(R) 10 Gigabit Network Connection
[23367.662770] ixgbe 0000:04:00.0 enp4s0f0: renamed from eth0
[23368.800920] ixgbe 0000:04:00.1: Multiqueue Enabled: Rx Queue count = 12, Tx Queue count = 12 XDP Queue count = 0
[23368.801226] ixgbe 0000:04:00.1: 16.000 Gb/s available PCIe bandwidth, limited by 5.0 GT/s PCIe x4 link at 0000:00:1c.4 (capable of 32.000 Gb/s with 5.0 GT/s PCIe x8 link)
[23368.801311] ixgbe 0000:04:00.1: MAC: 2, PHY: 1, PBA No: FFFFFF-0FF
[23368.801311] ixgbe 0000:04:00.1: a8:b8:e0:05:f0:92
[23368.802549] ixgbe 0000:04:00.1: Intel(R) 10 Gigabit Network Connection
[23368.812911] ixgbe 0000:04:00.1 enp4s0f1: renamed from eth0
                                          1 Reply Last reply Reply Quote 0
                                          • georgelzaG
                                            georgelza
                                            last edited by

                                            thinking need to figure out how to address the port...

                                            tried

                                            options ixgbe allow_unsupported_sfp=1
                                            and
                                            options ixgbe allow_unsupported_sfp=1.1
                                            and
                                            options ixgbe allow_unsupported_sfp=0.1
                                            and
                                            options ixgbe allow_unsupported_sfp=0.0

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post