• Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Search
  • Register
  • Login
Netgate Discussion Forum
  • Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Search
  • Register
  • Login

Another Netgate with storage failure, 6 in total so far

Scheduled Pinned Locked Moved Official Netgate® Hardware
302 Posts 37 Posters 66.1k Views
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • S
    stephenw10 Netgate Administrator
    last edited by Feb 27, 2025, 3:52 PM

    Yup when ramdisks are enabled it triggers a bunch of things at boot and shutdown. But it might be possible....

    1 Reply Last reply Reply Quote 0
    • M
      mer @JonathanLee
      last edited by Feb 27, 2025, 4:23 PM

      @JonathanLee said in Another Netgate with storage failure, 6 in total so far:

      needs of zfs are burning up onboard components

      I'm not trying to say people aren't having problems, I'm trying to understand "why" they are.

      What needs of ZFS are causing this?

      Burning up these kinds of components are related to the number of erase cycles, which circles back around to writes.
      What parts of pfSense are doing a lot of writes? I can see persistent logging, maybe "check for updates", maybe atime property enabled? Updating persistant store for block lists?

      The way I look at it, once the system is configured and running, it should be doing mostly read from the filesystem and running from memory.

      S 1 Reply Last reply Feb 27, 2025, 4:51 PM Reply Quote 0
      • D
        dennypage @stephenw10
        last edited by Feb 27, 2025, 4:38 PM

        @stephenw10 said in Another Netgate with storage failure, 6 in total so far:

        Be good if you could set it for /tmp only..... 🤔

        It would be really good if you could simply do ramdisk for /tmp only. No need to save/restore.

        1 Reply Last reply Reply Quote 0
        • S
          SteveITS Galactic Empire @mer
          last edited by Feb 27, 2025, 4:51 PM

          @mer said in Another Netgate with storage failure, 6 in total so far:

          What parts of pfSense are doing a lot of writes?

          Some packages (https://www.netgate.com/supported-pfsense-plus-packages), logging of default block rules, IGMP block logging, logging set in packages, updates of block lists and country lists, nginx access log (dashboard widgets), and similar.

          Pre-2.7.2/23.09: Only install packages for your version, or risk breaking it. Select your branch in System/Update/Update Settings.
          When upgrading, allow 10-15 minutes to restart, or more depending on packages and device speed.
          Upvote 👍 helpful posts!

          J W D S 4 Replies Last reply Feb 27, 2025, 5:05 PM Reply Quote 1
          • J
            JonathanLee @SteveITS
            last edited by Feb 27, 2025, 5:05 PM

            @SteveITS what files can use a linker file to direct to a usb drive? That model can use a usb drive right ?

            Make sure to upvote

            S 1 Reply Last reply Feb 27, 2025, 6:59 PM Reply Quote 0
            • W
              w0w @SteveITS
              last edited by Feb 27, 2025, 6:17 PM

              @SteveITS
              Hmm, so enabling compression "on the fly" in case of logs can significantly reduce writes, yes?

              zfs set compression=lz4 pfsense/var/log
              

              Some log compression options can be enabled via the GUI, but I don’t think they use "on-the-fly" compression.

              1 Reply Last reply Reply Quote 0
              • D
                dennypage @SteveITS
                last edited by Feb 27, 2025, 6:21 PM

                @SteveITS said in Another Netgate with storage failure, 6 in total so far:

                logging of default block rules, IGMP block logging

                These two can be quite voluminous, but are easy to address:

                • Add a rule on Local to pass IPv4+IPv6 IGMP with IP options set. I think this should actually be a default rule in pfSense.
                • Disable logging of packets blocked by the default rule in Firewall Logs. There are often thousands of these every day, and the individual log entries really aren't of much value.
                1 Reply Last reply Reply Quote 1
                • S
                  SteveITS Galactic Empire @JonathanLee
                  last edited by Feb 27, 2025, 6:59 PM

                  @JonathanLee Oh I have no idea. :)

                  @w0w said in Another Netgate with storage failure, 6 in total so far:

                  Some log compression options can be enabled via the GUI, but I don’t think they use "on-the-fly" compression.

                  Yes it does: https://docs.netgate.com/pfsense/en/latest/monitoring/logs/index.html#log-format

                  "ZFS already compresses this data"

                  @dennypage said in Another Netgate with storage failure, 6 in total so far:

                  easy to address

                  Yep, mentioned above. In a link maybe, it's been a long thread. We actually don't pass the IGMP, since it's "supposed" to be blocked (always has been) we add a block rule that is set to not log. Otherwise IGMP is logged even if the logging for the default block rule is off.

                  Pre-2.7.2/23.09: Only install packages for your version, or risk breaking it. Select your branch in System/Update/Update Settings.
                  When upgrading, allow 10-15 minutes to restart, or more depending on packages and device speed.
                  Upvote 👍 helpful posts!

                  D 1 Reply Last reply Feb 27, 2025, 7:19 PM Reply Quote 1
                  • D
                    dennypage @SteveITS
                    last edited by Feb 27, 2025, 7:19 PM

                    @SteveITS said in Another Netgate with storage failure, 6 in total so far:

                    We actually don't pass the IGMP, since it's "supposed" to be blocked (always has been) we add a block rule that is set to not log.

                    I would not say IGMP is supposed to be blocked, and it's rather inefficient to do. Multicast flooding is not desirable, even if it's only mDNS.

                    Of course, if IGMP is completely disabled in your switches, it doesn't matter. But if it is disabled in your switches, you won't see the IGMP messages to begin with. 😊

                    S 1 Reply Last reply Feb 27, 2025, 7:28 PM Reply Quote 1
                    • S
                      SteveITS Galactic Empire @dennypage
                      last edited by Feb 27, 2025, 7:28 PM

                      @dennypage rephrasing, pfSense blocks them by default.
                      https://docs.netgate.com/pfsense/en/latest/troubleshooting/log-filter-blocked.html#packets-with-ip-options

                      Pre-2.7.2/23.09: Only install packages for your version, or risk breaking it. Select your branch in System/Update/Update Settings.
                      When upgrading, allow 10-15 minutes to restart, or more depending on packages and device speed.
                      Upvote 👍 helpful posts!

                      D 1 Reply Last reply Feb 27, 2025, 7:36 PM Reply Quote 0
                      • D
                        dennypage @SteveITS
                        last edited by Feb 27, 2025, 7:36 PM

                        @SteveITS said in Another Netgate with storage failure, 6 in total so far:

                        rephrasing, pfSense blocks them by default.

                        Yea, that's why I called out that pfSense should add a default pass rule for IGMP.

                        Blocking packets with IP options that are to be forwarded is a good default, however IGMP isn't forwarded. Blocking IGMP by default makes little sense.

                        A 1 Reply Last reply Feb 27, 2025, 9:11 PM Reply Quote 1
                        • A
                          andrew_cb @dennypage
                          last edited by andrew_cb Feb 27, 2025, 9:36 PM Feb 27, 2025, 9:11 PM

                          @dennypage @SteveITS I had commented on redmine 15400 but since it was closed I guess that my message went unnoticed.

                          I have created a new redmine 16068 for adding options to disable logging of packets with IP options.

                          1 Reply Last reply Reply Quote 2
                          • K
                            kingsleyadam
                            last edited by Mar 10, 2025, 7:18 PM

                            Thanks to @andrew_cb and others for bringing awareness to this. I had no idea my 6100 has limitations due to the eMMC. I went out of my way to buy a 6100 over my own router build because I just wanted to setup my router and forget about it. As someone who is fully remote the router is the last thing I can have fail.

                            I saw the threads on Reddit and did a quick check. Just over 1.5 years of having my 6100, it’s already at 70% wear.

                            I bought 2x16GB Intel Optane Drives which you can get for less than 5 euro a piece and managed to get them installed and set up in a mirror for redundancy (the drives are so cheap, I think it’s silly not too). I also 0’d out my eMMC drive to ensure it does’t cause any conflicts.

                            I’m not thrilled that I had to do this, I’m thankful there were M.2 ports on the 6100 that I could use. But my biggest take-away is that installing your own SSD is not “supported” and could void your warranty is unacceptable. I think there should be a well documented SSD upgrade for any device that has an available slot, it should not void the warranty, and most definitely shouldn’t be discouraged.

                            As a side note, I really wish the installer was offline. I was sweating bullets attempting to configure the WAN in the Installer with PPoE and VLAN tagging (don’t get me wrong, it was easy, but if it didn’t work I’d be SOL).

                            A 1 Reply Last reply Mar 20, 2025, 11:24 PM Reply Quote 2
                            • A
                              andrew_cb @kingsleyadam
                              last edited by Mar 20, 2025, 11:24 PM

                              @kingsleyadam I am glad you discovered the storage wear on your 6100 and installed an SSD before you experienced a sudden failure!

                              I had no idea my 6100 has limitations due to the eMMC. I went out of my way to buy a 6100 over my own router build because I just wanted to setup my router and forget about it. As someone who is fully remote the router is the last thing I can have fail.

                              Your comment is exactly what this thread is about.

                              There have been many good suggestions in this thread on ways to reduce the wear of the onboard eMMC, but they do not address the main point of this thread:

                              If any usage assumptions or limitations are not clearly stated upfront or in the documentation, then it does not matter what the technical reasons are, how valid they are, or what workarounds are available!

                              You cannot advertise a ladder as great for construction work and then not disclose that it has a 100-pound weight limit, just like you cannot sell a manual transmission vehicle without an instrument panel and then say it is the user's fault when the engine blows up.

                              If Netgate sold ladders like they sell firewalls, what kind of chaos would result from people using the Netgate Ladder-4100 BASE version?

                              If there are limitations, recommendations, or "best practices" regarding firewalls with eMMC storage, then state them clearly on the product page and conspicuously in the documentation! That would significantly reduce or even eliminate this entire problem.

                              It has been nearly two months since Netgate acknowledged the issue, and there have been no changes. I do not understand why Netgate refuses to spend an hour copying and pasting an informational blurb to the store product pages.

                              J 2 Replies Last reply Mar 21, 2025, 11:33 PM Reply Quote 0
                              • A andrew_cb referenced this topic on Mar 20, 2025, 11:24 PM
                              • J
                                jwt Netgate @andrew_cb
                                last edited by Mar 21, 2025, 11:33 PM

                                It has been nearly two months since Netgate acknowledged the issue, and there have been no changes.

                                [literally responding from a Starbucks in So Colorado at 5:30 on a Friday]

                                You are wrong, there are a lot of changes in-progress, but I’m not getting I to this with you, here, right now.

                                A 1 Reply Last reply Mar 22, 2025, 1:32 AM Reply Quote 0
                                • J
                                  jwt Netgate @andrew_cb
                                  last edited by Mar 21, 2025, 11:37 PM

                                  @andrew_cb said in Another Netgate with storage failure, 6 in total so far:

                                  just like you cannot sell a manual transmission vehicle without an instrument panel and then say it is the user's fault when the engine blows up.

                                  I have several old Toyota trucks with manual transmissions that do not have tachometers.

                                  A 1 Reply Last reply Mar 22, 2025, 1:01 AM Reply Quote 0
                                  • A
                                    andrew_cb @jwt
                                    last edited by Mar 22, 2025, 1:01 AM

                                    @jwt said in Another Netgate with storage failure, 6 in total so far:

                                    I have several old Toyota trucks with manual transmissions that do not have tachometers.

                                    A 22R requires some patience and effort to get it revving high enough to blow... not to mention the noise it makes will provide some obvious auditory clues.
                                    Most 22RE are limited to 5800 RPM.
                                    In both cases, I am pretty sure that Toyota has the maximum safe rev limit specified in the owner's manual.

                                    Are 40-year-old Toyota trucks the best comparison for Netgate firewalls? Both offer legacy/carb and EFI options, but the evidence suggests that Toyota is the more reliable of the two.

                                    J 1 Reply Last reply Mar 22, 2025, 1:31 AM Reply Quote 0
                                    • J
                                      jwt Netgate @andrew_cb
                                      last edited by Mar 22, 2025, 1:31 AM

                                      @andrew_cb said in Another Netgate with storage failure, 6 in total so far:

                                      @jwt said in Another Netgate with storage failure, 6 in total so far:

                                      I have several old Toyota trucks with manual transmissions that do not have tachometers.

                                      A 22R requires some patience and effort to get it revving high enough to blow... not to mention the noise it makes will provide some obvious auditory clues.
                                      Most 22RE are limited to 5800 RPM.
                                      In both cases, I am pretty sure that Toyota has the maximum safe rev limit specified in the owner's manual.

                                      Are 40-year-old Toyota trucks the best comparison for Netgate firewalls? Both offer legacy/carb and EFI options, but the evidence suggests that Toyota is the more reliable of the two.

                                      [Now close to New Mexico]

                                      @andrew_cb said in Another Netgate with storage failure, 6 in total so far:

                                      @jwt said in Another Netgate with storage failure, 6 in total so far:

                                      I have several old Toyota trucks with manual transmissions that do not have tachometers.

                                      A 22R requires some patience and effort to get it revving high enough to blow... not to mention the noise it makes will provide some obvious auditory clues.
                                      Most 22RE are limited to 5800 RPM.
                                      In both cases, I am pretty sure that Toyota has the maximum safe rev limit specified in the owner's manual.

                                      Nothing I own has a 22R. Some have F or 2F engines. One has a 3F (belongs to the CEO, technically), but that one has an auto, and a tach.

                                      You were talking about instrumentation on the dash, now you’re bailing for the owner’s manual.

                                      Are 40-year-old Toyota trucks the best comparison for Netgate firewalls? Both offer legacy/carb and EFI options, but the evidence suggests that Toyota is the more reliable of the two.

                                      Go ahead, show us all the proof in this FJ40 manual

                                      https://www.slideshare.net/slideshow/toyota-land-cruiser-owners-manual-1968-1971-fj40-fj43-fj45-pdf/269783892

                                      Seems like you still don’t know what you’re talking about, and anre only here to fight.

                                      A 1 Reply Last reply Mar 22, 2025, 5:36 AM Reply Quote 0
                                      • A
                                        andrew_cb @jwt
                                        last edited by Mar 22, 2025, 1:32 AM

                                        @jwt said in Another Netgate with storage failure, 6 in total so far:

                                        It has been nearly two months since Netgate acknowledged the issue, and there have been no changes.

                                        [literally responding from a Starbucks in So Colorado at 5:30 on a Friday]

                                        You are wrong, there are a lot of changes in-progress, but I’m not getting I to this with you, here, right now.

                                        Nice of you to chime in with vague and unverifiable claims.

                                        Perhaps your time would have been better spent doing something to help protect your 'valued' customers from premature, catastrophic failures of your hardware, such as adding warnings and/or disclaimers to the product pages in the Netgate store?

                                        1 Reply Last reply Reply Quote 0
                                        • A
                                          andrew_cb @jwt
                                          last edited by andrew_cb Mar 22, 2025, 6:01 AM Mar 22, 2025, 5:36 AM

                                          @jwt said in Another Netgate with storage failure, 6 in total so far:

                                          Seems like you still don’t know what you’re talking about, and anre only here to fight.

                                          You are right - I am here fighting to protect current and prospective Netgate customers against sudden, premature failure of Netgate firewalls that they are spending their hard-earned money on. My posts on Reddit and this forum have helped many users avoid disaster, particularly those who became aware that their Netgate firewall was terminally damaged and at risk of imminent failure. How many users have you helped avoid storage failure today, this week, this month, this year?

                                          You seem passionate about discussing old Toyotas - unfortunately, you do not seem to share the same passion for educating and protecting Netgate customers from storage failure.

                                          I will concede that the 1968-1971 Toyota FJ40 did not come with a tachometer and the owner's manual does not specify a maximum RPM for the engine. Your example of a 55-year-old vehicle proves that everything else posted in the nearly 200 comments on this thread is wrong. Clearly, nothing is wrong, and all the premature failures of Netgate devices must be some sort of shared mass-hallucination.

                                          On the subject of Toyota LandCruisers, do you ever go out cruising with fellow enthusiasts Netgate's marketing director or gonzopancho from Reddit? Gonzopancho also lives in Colorado, and he is the co-owner of Netgate and head of engineering. You and gonzopancho have very similar writing styles, boy, it sure would be a surprise if you turned out to be him!

                                          Maybe you guys can meet up sometime and discuss solutions to improve storage health monitoring and lifetime, and ways to better educate customers on the storage limitations of the BASE/eMMC versions of Netgate firewalls?

                                          J 1 Reply Last reply Mar 22, 2025, 11:35 PM Reply Quote 0
                                          • First post
                                            Last post
                                          Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.
                                            [[user:consent.lead]]
                                            [[user:consent.not_received]]