Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    SG-3100 switch weird behavior (resolved)

    Official Netgate® Hardware
    sg-3100 switch arp syslog
    3
    85
    18.9k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • stephenw10S
      stephenw10 Netgate Administrator
      last edited by

      It looks like you changed the IP address of the desktop from .254 to .251. I assume that made no difference?

      In the redmine you show ping traffic to .253 going to the wrong port. But above you said:

      The problem stops for a few minutes after pinging 192.168.255.253 from pfsense, and it starts again.

      That's confusing. I can only imagine one of those to be true.

      You also said that removing ramdisks appeared to resolve it. Did that turn out to be incorrect? When you add or remove ramdisks the firewall has to reboot. Does rebooting normally also correct it for some time?

      The only thing I can imagine creating this issue is if the MAC table in the switch in the 3100 is somehow incorrect or being loaded with a bad value. pfSense has no control over that dircetly though. I've never seen that before.

      Steve

      M 1 Reply Last reply Reply Quote 0
      • stephenw10S
        stephenw10 Netgate Administrator @mcury
        last edited by

        @mcury said in Removed by user.:

        First ping I get the no response found as shown in packet 6645, then the following ICMP requests goes to the correct host (192.168.255.253).
        The problem gets solved for a few minutes and then it starts again.

        Could be something redirecting it. What does a pcap on the RasPi show when that's happening?

        Almost feels like a bad subnet mask except pfSense is sending to correct MAC.

        M 1 Reply Last reply Reply Quote 0
        • M
          mcury @stephenw10
          last edited by

          @stephenw10 said in SG-3100 switch weird behavior:

          It looks like you changed the IP address of the desktop from .254 to .251. I assume that made no difference?

          correct, same behavior

          @stephenw10 said in SG-3100 switch weird behavior:

          You also said that removing ramdisks appeared to resolve it. Did that turn out to be incorrect?

          Unfortunately it worked for a while, but the problem started again, same as before.

          @stephenw10 said in SG-3100 switch weird behavior:

          When you add or remove ramdisks the firewall has to reboot. Does rebooting normally also correct it for some time?

          Yes, it rebooted normally and it was fine for a while, except that the problem happened again.

          dead on arrival, nowhere to be found.

          1 Reply Last reply Reply Quote 0
          • M
            mcury @stephenw10
            last edited by mcury

            @stephenw10 said in SG-3100 switch weird behavior:

            Could be something redirecting it. What does a pcap on the RasPi show when that's happening?

            I'll tcpdump it, let me wait for the problem to happen again

            Almost feels like a bad subnet mask except pfSense is sending to correct MAC.

            raspberry pi 4b 192.168.255.253 eth0:

            2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
                link/ether dc:a6:32:a5:47:19 brd ff:ff:ff:ff:ff:ff
                inet 192.168.255.253/29 brd 192.168.255.255 scope global dynamic eth0
            

            957e1af7-2859-4859-bcb0-cb593ccbbcf8-image.png

            22e533ec-38df-497a-b1d7-c235d0ba8bf3-image.png

            dead on arrival, nowhere to be found.

            1 Reply Last reply Reply Quote 0
            • M
              mcury @johnpoz
              last edited by mcury

              @johnpoz said in SG-3100 switch weird behavior:

              Or wait this is a 3100, so switch ports. If I had to guess your mini is leaking, this is a flex mini?
              So you send traffic to 255.253, and your PC is seeing it.. Which you are correct it never should. Even in prosc mode, those packets should only be sent out the port that 255.253 is connected to - not all of them.
              You have some sort of leak if you ask me.. Could be pfsense sending it out all the lan interfaces in on the switch.. Hmmm

              Sorry, I missed that part
              Yes, its a SG-3100, the problem started, let me tcpdump it from rapsberry pi 4b, one sec

              rpi4.pcap

              dead on arrival, nowhere to be found.

              M 1 Reply Last reply Reply Quote 0
              • M
                mcury @mcury
                last edited by mcury

                yes, hehe, its kind of a mess

                Packet capture in host 192.168.255.251
                00fff5e7-5e34-4e1b-9c52-d9eed102f54d-image.png

                dead on arrival, nowhere to be found.

                M johnpozJ 2 Replies Last reply Reply Quote 0
                • M
                  mcury @mcury
                  last edited by

                  I'll reinstall my pfsense from scratch to test, but I can't do it right now.
                  Then, if the problem happens again, I'll replace this switch.

                  dead on arrival, nowhere to be found.

                  M 1 Reply Last reply Reply Quote 0
                  • M
                    mcury @mcury
                    last edited by

                    So weird man, a ping fixes it temporarily..
                    9313c58e-10e9-49be-9ca2-482c64fb77f6-image.png

                    dead on arrival, nowhere to be found.

                    1 Reply Last reply Reply Quote 0
                    • johnpozJ
                      johnpoz LAYER 8 Global Moderator @mcury
                      last edited by johnpoz

                      @mcury yeah you shouldn't be seeing those. Hmmmm Even if your nic was in promiscuous mode, that mac shouldn't be sent down the port where the mac is not listed.

                      If you had some sort of leak or bridge where the mac was being learned on multiple interfaces that could happen..

                      So proper destination mac is your down the trunk (lan4) to the flex mini. But pfsense is also sending it out lan1? But the only place the mac of that pi4 should be seen by pfsense is the lan4 interface, it should never send that mac out lan1, unless there was bridge setup.

                      hmmmm strange...

                      This a good question for @stephenw10 he would know way more than me on the inter workings of the switch in the 3100. But typically a switch would only send traffic down the interface that the mac is on.

                      An intelligent man is sometimes forced to be drunk to spend time with his fools
                      If you get confused: Listen to the Music Play
                      Please don't Chat/PM me for help, unless mod related
                      SG-4860 24.11 | Lab VMs 2.7.2, 24.11

                      M 1 Reply Last reply Reply Quote 0
                      • M
                        mcury @johnpoz
                        last edited by

                        @johnpoz Exactly, its so weird, that packet should never go to pfsense's LAN1 ..

                        I'll try to fix it tonight by reinstalling my pfsense from scratch..
                        Then, if the problem happens again, I'll replace this switch..

                        dead on arrival, nowhere to be found.

                        1 Reply Last reply Reply Quote 0
                        • stephenw10S
                          stephenw10 Netgate Administrator
                          last edited by

                          Yeah, it's a pretty basic switch and there's no control over things like the MAC table. That's the only thing I could imagine causing that though.

                          If you haven't already try power cycling the 3100 entirely. That should completely reset the switch if it's somehow managed to toggle some flag.

                          Steve

                          M 1 Reply Last reply Reply Quote 0
                          • M
                            mcury @stephenw10
                            last edited by

                            @stephenw10 hm, I'll try it now a shutdown, remove the power cable, one sec, let me see who is here using the Internet

                            dead on arrival, nowhere to be found.

                            M 1 Reply Last reply Reply Quote 0
                            • M
                              mcury @mcury
                              last edited by

                              Done, the problem persists..

                              1. Halt system and once the shutdown process ended, removed the power cable for a few seconds.

                              0c31ac53-951d-4c89-89a8-4fadaed63ab6-image.png

                              6910718c-7557-49a5-918d-8e19068f4198-image.png

                              dead on arrival, nowhere to be found.

                              1 Reply Last reply Reply Quote 0
                              • stephenw10S
                                stephenw10 Netgate Administrator
                                last edited by stephenw10

                                Hmm, the only other thing I could imagine causing this is if something feeding bad data into the switch MAC table. That would have to be the desktop machine.

                                If you run a continuous ping from the RasPi to somewhere that has to be accessed through the 3100 switch, does that prevent the issue?

                                If it does I'd try to find something sending the RasPi MAC from the desktop. Hard to say what that might be.... something reflected perhaps?

                                If you run a pcap on the desktop and filter by the RasPi MAC address whilst the problem is not happening and wait for it to start. The first thing that happens there might be the offending packet.

                                Steve

                                M 1 Reply Last reply Reply Quote 0
                                • M
                                  mcury @stephenw10
                                  last edited by mcury

                                  @stephenw10 said in SG-3100 switch weird behavior:

                                  If you run a continuous ping from the RasPi to somewhere that has to be accessed through the 3100 switch, does that prevent the issue?

                                  Testing now, ping is running from RPI4 to pfsense.
                                  It seems to have stopped, but it may start again soon, so I'll wait a little longer this time.

                                  Packet capture set:
                                  a0a5c924-fc4a-43f3-9466-2aa3c860e74e-image.png

                                  Edit:

                                  This is my ARP table (desktop)

                                  $ cat /proc/net/arp
                                  IP address       HW type     Flags       HW address            Mask     Device
                                  192.168.255.252  0x1         0x2         00:11:32:9f:ee:93     *        enp7s0
                                  192.168.255.249  0x1         0x2         00:08:a2:0c:c4:1c     *        enp7s0
                                  192.168.255.250  0x1         0x2         b8:27:eb:ea:f8:65     *        enp7s0
                                  192.168.255.253  0x1         0x2         dc:a6:32:a5:47:19     *        enp7s0
                                  

                                  dead on arrival, nowhere to be found.

                                  M 1 Reply Last reply Reply Quote 0
                                  • M
                                    mcury @mcury
                                    last edited by mcury

                                    40 minutes pinging from raspberry pi 4b (192.168.255.253) to pfsense (192.168.255.249) and no problem so far.

                                    I have two wireshark windows opened, one monitoring:
                                    eth.src == dc:a6:32:a5:47:19 and not tcp.port == 22 and not tcp.port == 9000

                                    And the second one monitoring:
                                    ip.addr == 192.168.255.253 and not tcp.port == 9000 and not tcp.port == 22

                                    dead on arrival, nowhere to be found.

                                    M 1 Reply Last reply Reply Quote 0
                                    • M
                                      mcury @mcury
                                      last edited by mcury

                                      Dropped the ping and one minute later (or less), the problem starts again:

                                      84f90579-b4d5-4df7-b823-f091eb706d61-image.png

                                      desktop ARP table:
                                      $ cat /proc/net/arp
                                      IP address HW type Flags HW address Mask Device
                                      192.168.255.252 0x1 0x2 00:11:32:9f:ee:93 * enp7s0
                                      192.168.255.249 0x1 0x2 00:08:a2:0c:c4:1c * enp7s0
                                      192.168.255.250 0x1 0x2 b8:27:eb:ea:f8:65 * enp7s0
                                      192.168.255.253 0x1 0x2 dc:a6:32:a5:47:19 * enp7s0

                                      dead on arrival, nowhere to be found.

                                      1 Reply Last reply Reply Quote 0
                                      • stephenw10S
                                        stephenw10 Netgate Administrator
                                        last edited by

                                        Hmm, so nothing from the RasPi MAC address at the desktop that might be inserting invalid entries into the switch.

                                        It might be worth re-running that test using the RasPi MAC as destination in the filter (or as either).
                                        You might catch something arriving using that but a different IP address.

                                        Also when this happens do you see traffic being sent only to the desktop? Or is the syslog traffic sent to all the 3100 switch ports? Does it also arrive at the RasPi?

                                        Steve

                                        M 1 Reply Last reply Reply Quote 0
                                        • M
                                          mcury @stephenw10
                                          last edited by

                                          @stephenw10 It seems that its only going to LAN1..

                                          d6978265-5e91-4305-bf2d-0c79ff092142-image.png

                                          raspberry pi 3 in which you see the tcpdump above is connected to the switch unifi mini.

                                          Let me perform this test again, but in the NAS which is connected to LAN2 of pfsense, one sec.

                                          dead on arrival, nowhere to be found.

                                          M 1 Reply Last reply Reply Quote 0
                                          • M
                                            mcury @mcury
                                            last edited by mcury

                                            Hmmm, its going to port LAN2 of pfsense too:
                                            NAS IP is 192.168.255.252 (tcpdump) (LAN2 of pfsense)

                                            On the right, wireshark running on desktop (LAN1 of pfsense)

                                            9b27a73e-270d-451c-88e1-ea4b7c604009-image.png

                                            dead on arrival, nowhere to be found.

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.