RESOLVED~Squid different cache location and management.

JonathanLee

Hello fellow Netgate community members can you please help?

/pkg_edit.php: [squid] Will NOT delete cache dir '/nvme/LOGS_Optane/Squid_Cache' since it is not located under /var/squid. Delete manually if required.

Does anyone know how to make sure that Squid can manage a different cache location? I had mine fill up 30GBs but it would evict any old items once it is full. It works it just wont manage a different location.

I have already done a link file to the original location it will delete the linked file and place the cache on the original drive. That path did not work.

JonathanLee

pfSense Squid Cache on NVMe (Persistent Across Reboots)
======================================================

Goal:
-----
Use an existing directory on an already-mounted NVMe drive
as Squid’s cache, while satisfying pfSense’s requirement
that the cache live under /var/squid.

This method:
- Works across reboots
- Survives pfSense & Squid upgrades
- Does NOT use symlinks (which do not work)
- Does NOT destroy existing data on the drive

--------------------------------------------------------

Prerequisites:
--------------
- NVMe drive already mounted (example: /nvme)
- Existing cache directory on the drive:
 /nvme/LOGS_Optane/Squid_Cache
- Squid package installed

--------------------------------------------------------

Step 1: Create the mount point under /var/squid
------------------------------------------------
(Only creates an empty directory if it doesn’t exist)

mkdir -p /var/squid/cache

--------------------------------------------------------

Step 2: Bind-mount (nullfs) the existing directory
--------------------------------------------------
(This makes the NVMe directory appear inside /var/squid)

mount_nullfs /nvme/LOGS_Optane/Squid_Cache /var/squid/cache

NOTE:
- No data is moved or copied
- Existing cache contents are used directly

--------------------------------------------------------

Step 3: Make the mount persistent (IMPORTANT)
------------------------------------------------
pfSense GUI:

add cron for script below

4. Save
5. Apply Changes

--------------------------------------------------------

Step 4: Configure Squid to use the new path
-------------------------------------------
pfSense GUI:

Services → Squid Proxy Server → Local Cache

Set:
Cache directory:
 /var/squid/cache

Save and restart Squid.

--------------------------------------------------------

Step 5: Verify
--------------
Check that the mount is active:

df -h | grep squid
mount | grep nullfs

You should see the NVMe filesystem mounted at:
 /var/squid/cache

--------------------------------------------------------

Behavior After Reboot:
----------------------
- Mount persists across reboots
- pfSense package manager no longer complains
- Squid cache cleanup works normally
- No manual remounting required

--------------------------------------------------------

Important Notes:
----------------
- Symlinks do NOT work (pfSense resolves real paths)
- nullfs is the correct and supported workaround
- MSDOS/FAT filesystems work but are not ideal long-term
- UFS or ZFS is recommended if you ever reformat

--------------------------------------------------------

script

#!/bin/sh

TAG="squid-nullfs"
NVME_DEV="/dev/nda0p2"
NVME_MNT="/nvme/LOGS_Optane"
CACHE_SRC="${NVME_MNT}/Squid_Cache"
CACHE_DST="/var/squid/cache"

logger -t ${TAG} "Starting Squid nullfs mount sequence"

# 1. Ensure NVMe filesystem is mounted
if ! mount | grep -q "on ${NVME_MNT} "; then
    logger -t ${TAG} "Mounting NVMe filesystem"
    mount_msdosfs ${NVME_DEV} ${NVME_MNT} || {
        logger -t ${TAG} "ERROR: NVMe mount failed"
        exit 1
    }
else
    logger -t ${TAG} "NVMe already mounted"
fi

# 2. Stop squid if running
if pgrep -x squid >/dev/null; then
    logger -t ${TAG} "Stopping squid"
    /usr/local/sbin/pfSsh.php playback svc stop squid
    sleep 2
fi

# 3. Ensure directories exist
mkdir -p "${CACHE_SRC}"
mkdir -p "${CACHE_DST}"

# 4. Mount nullfs if not already mounted
if ! mount | grep -q "on ${CACHE_DST} "; then
    logger -t ${TAG} "Mounting nullfs cache"
    mount -t nullfs "${CACHE_SRC}" "${CACHE_DST}" || {
        logger -t ${TAG} "ERROR: nullfs mount failed"
        exit 1
    }
else
    logger -t ${TAG} "nullfs already mounted"
fi

# 5. Start squid
logger -t ${TAG} "Starting squid"
/usr/local/sbin/pfSsh.php playback svc start squid

logger -t ${TAG} "Squid nullfs mount completed"

JonathanLee

Screenshot 2026-01-09 at 12.13.51.png Screenshot 2026-01-09 at 12.12.22.png Screenshot 2026-01-09 at 11.59.23.png

it works

JonathanLee

WARNING!!!!
DO NOT USE THIS AT IT WILL NOT WORK even with late
you must use ,late or it will lock up example
/nvme/LOGS_Optane/Squid_Cache /var/squid/cache nullfs rw,late 0 0

# Device		Mountpoint	FStype	Options		Dump	Pass#
/dev/msdosfs/EFISYS	/boot/efi	msdosfs	rw,noatime,noauto	0	0
/dev/msdosfs/DTBFAT0	/boot/msdos	msdosfs	rw,noatime,noauto	0	0
/dev/gpt/swapUSB.eli		none	swap	sw		0	0
/nvme/LOGS_Optane/Squid_Cache  /var/squid/cache  nullfs  rw,late  0  0

This also did not resolve it it as the system boots but it is to late and squid makes its own file system because it didn't mount

tinfoilmatt

Interestinggg. Entirely unrelated—but I feel like this (i.e., the mount_nullfs part) could maaaybe be made to work to store all pfBlockerNG logging on an expendable/burner drive?

I would love to be able to log all Unbound queries in pfBlockerNG's Python Mode format, or all pfBlockerNG logging for that matter. I once tried to do it via symlink to a mounted USB drive, but gave up when I couldn't get the log files to be created.

JonathanLee

@tinfoilmatt DONT DO IT I AM having issues with this right now I cant do the fstab because nvme driver loads at a different time and cron it is to late for

JonathanLee

@tinfoilmatt this worked

cron job for @reboot /root/mount_squid_nullfs.sh

and the script

#!/bin/sh

TAG="squid-nullfs"
NVME_DEV="/dev/nda0p2"
NVME_MNT="/nvme/LOGS_Optane"
CACHE_SRC="${NVME_MNT}/Squid_Cache"
CACHE_DST="/var/squid/cache"

logger -t ${TAG} "Starting Squid nullfs mount sequence"

# 1. Ensure NVMe filesystem is mounted
if ! mount | grep -q "on ${NVME_MNT} "; then
    logger -t ${TAG} "Mounting NVMe filesystem"
    mount_msdosfs ${NVME_DEV} ${NVME_MNT} || {
        logger -t ${TAG} "ERROR: NVMe mount failed"
        exit 1
    }
else
    logger -t ${TAG} "NVMe already mounted"
fi

# 2. Stop squid if running
if pgrep -x squid >/dev/null; then
    logger -t ${TAG} "Stopping squid"
    /usr/local/sbin/pfSsh.php playback svc stop squid
    sleep 2
fi

# 3. Ensure directories exist
mkdir -p "${CACHE_SRC}"
mkdir -p "${CACHE_DST}"

# 4. Mount nullfs if not already mounted
if ! mount | grep -q "on ${CACHE_DST} "; then
    logger -t ${TAG} "Mounting nullfs cache"
    mount -t nullfs "${CACHE_SRC}" "${CACHE_DST}" || {
        logger -t ${TAG} "ERROR: nullfs mount failed"
        exit 1
    }
else
    logger -t ${TAG} "nullfs already mounted"
fi

# 5. Start squid
logger -t ${TAG} "Starting squid"
/usr/local/sbin/pfSsh.php playback svc start squid

logger -t ${TAG} "Squid nullfs mount completed"

JonathanLee

@tinfoilmatt this one is better the logger doesn't have errors thrown

#!/bin/sh

TAG="squid-nullfs"
NVME_DEV="/dev/nda0p2"
NVME_MNT="/nvme/LOGS_Optane"
CACHE_SRC="${NVME_MNT}/Squid_Cache"
CACHE_DST="/var/squid/cache"

# --- Helper function to log safely to system.log using pfSense PHP ---
log_sys() {
    MESSAGE="$1"
    logger -t "$TAG" "$MESSAGE"
}

log_sys "Starting Squid nullfs mount sequence"

# 1. Ensure NVMe filesystem is mounted
if ! mount | grep -q "on ${NVME_MNT} "; then
    log_sys "Mounting NVMe filesystem"
    mount_msdosfs "${NVME_DEV}" "${NVME_MNT}" || {
        log_sys "ERROR: NVMe mount failed"
        exit 1
    }
else
    log_sys "NVMe already mounted"
fi

# 2. Stop squid if running
if pgrep -x squid >/dev/null; then
    log_sys "Stopping squid"
    /usr/local/sbin/pfSsh.php playback svc stop squid
    sleep 2
fi

# 3. Ensure directories exist
mkdir -p "${CACHE_SRC}" "${CACHE_DST}"

# 4. Mount nullfs if not already mounted
if ! mount | grep -q "on ${CACHE_DST} "; then
    log_sys "Mounting nullfs cache"
    mount -t nullfs "${CACHE_SRC}" "${CACHE_DST}" || {
        log_sys "ERROR: nullfs mount failed"
        exit 1
    }
else
    log_sys "nullfs already mounted"
fi

# 5. Start squid
log_sys "Starting squid"
/usr/local/sbin/pfSsh.php playback svc start squid

log_sys "Squid nullfs mount completed"

shows the logs in system logs

Screenshot 2026-01-09 at 15.22.16.png

Done YEAH!!!! I had to do so much VI and ai help with this to find out a way to fix the race conditions

JonathanLee

and again... it's all useless for me because the cache still needs manual deletion. The "clear disk cache now" button still doesn't work. It does allow you to use that path with the overlay so kind of a win but same issues as before... so why go and do all of that .... head hits table .....

JonathanLee

@tinfoilmatt what about nullfs for var/logs?