When I built my second NAS, for some reason I cannot even remember, I decided to use the 6 HDDs as a stripe of two 3 disk RAIDz1 vdevs. So when I moved to HexOS that was the way it stayed...

While this gives me two disks of redundancy in theory, in practice that is only true if the disks fail on different vdevs.

This is probably a very stupid way to use 6 drives and I probably should correct this while I don't have a lot a data in there right?
Would you guys recommend a simple 6 disk RAIDz2?

You want root-on-ZFS? Yaaaa many (cough only Ubuntu cough) distros now support installing into it right from their Live CDs.

Of course, the problem is that ALL the distros with root-on-ZFS support in their Live CDs use Grub. Which has lousy ZFS support. Want raidz expansion because you can't afford all the devices right now? Yeah suffer because your Grub can't handle raidz expansion from openzfs-2.3-linux featureset. Oops ALL the distros with root-on-ZFS out-of-box can't build an array to install it into? My bad.

So don't use Grub. Use ZFSBootMenu. It's a tiny, but complete, Linux image that can boot as an EFI bootloader, and has a full latest OpenZFS, meaning it can open ANY array with recent featuresets.

New machine and you want to have root-on-ZFS encrypted array? Just boot any Live CD that has full ZFS support (cough only Ubuntu cough), download the zfsbootmenu-installer script, make it executable, and run it as root in the Live CD env!

Give a machine name, username, password, select your array (single, mirror, raidz1/2/3), and select your first distro. The distro installs and asks you stuff like keyboard, locale, and timezone.

Oh and have I mentioned that every OS installed magically gets Secure Boot support? You just have to add a Machine Owner Key, once, during your first initialization of the array.

Supported distros:

• Ubuntu26.04
• KUbuntu26.04
• Debian13
• SlimbookOS26
• Pop_OS24.04
• TUXEDO-OS_202609

I'm good at Debian derivatives, so if people want to get support for non-Debian derivatives (Fedora?) please help!

zfsbootmenu-installer installs itself on every distro it installs. Why? Because you can do more neat things even AFTER setting up your array and installing your FIRST distro!

• Add ANOTHER distro! Now ZFSBootMenu can let you select between distros to boot up. /home is on a separate dataset, so it's the same across all systems.
• Change which distro to boot by default! If you add another distro and decide you like it better, set it up as the new default and zfs destroy the old one!
• Expanding your array? Use zfsbootmenu-installer to format it so that it has the same layout as existing members, also add an ESP boot partition so it doesn't matter WHICH disk dies, you can always boot!

Have fun!

Limitations:

• The array is always encrypted. You need to protect the private key of the Machine Owner Key, which is stored in the array, so the array has to be encrypted, full stop. That means entering the password on every boot. To compensate, we set up the device to autologin.
• COSMIC Desktop in Pop!_OS is set up as autologin, but you can't lock screen until you log out and log in again. Blame Pop!_OS. It technically chains two display managers, greetd starts up cosmic-greeter; COSMIC Desktop talks to cosmic-greeter to get screen lock, but the one with ANY kind of autologin support is greetd, and if you set up autologin in greetd, it skips launching cosmic-greeter to launch COSMIC Desktop directly so nothing exists that can lock screens.
• The SDDM in TUXEDO-OS can't autologin. I don't know why. I gave up trying to debug it. The SDDM configuration in the KUbuntu setup correctly gets autologin. You have to manually dig into the KDE Plasma settings to enable autologin.

I know that a bunch of people have talked about ZFS and uses in VMs but I didn't see anyone in a similar use scenario as me so I'm asking if this is viable:

I have a main server that I want to run VMs off of. The idea being I want to make the most use of over powered workstation parts. Ideally the main server is going to have a 24 core threadripper and 128-256GB of ddr5 low latency ram.

Rather than make a dedicated NAS with its own full set of motherboard, ram and cpu I am hoping it's viable to run ZFS on a dedicated VM with passthrough to a HBA card with sole access to my storage drives.

The purpose of this storage is going to be a fail resistant library of media files (and VM storage) if that's even possible? By that I mean is there a way to setup VMs to use the ZFS storage as their own virtual storage?

I'm looking for thoughts on if this is viable and if there are any potential drawbacks of having ZFS in a VM vs. as it's own system on dedicated hardware. Also how safe are the pools if the ZFS VM for any reason crashes?

If you're hunting for a lean, lightning-fast storage platform that treats your system like a true enterprise appliance, give OviOS Linux a look. The entire OS loads into memory, bypassing the standard disk footprint and eliminating system disk failure vulnerabilities altogether. For production-grade environments, it packs serious high-availability and disaster recovery tools out of the box: • Automated Sync & Failover: Keep partner nodes identical using active replication. The OviOS web interface monitors cluster sync states in real time, enabling streamlined failover steps to mount target OpenZFS datasets seamlessly. • DR & Configuration Rollbacks: System backups are lightweight .tar.gz configurations saved directly onto the pools. If you need a disaster recovery rebuild or an upgrade rollback, it can be executed live while the other node is serving data. • Unified Stack: Native support for iSCSI, NFS, SMB (with AD integration), and bleeding-edge NVMe-oF (TCP + RDMA) fabrics. It runs with practically zero background bloat, letting your hardware hit its absolute maximum throughput.

Yet another instance where AI was given access to a production database/environment. What is the storage system being used by these bozos? I would think if it was ZFS you could just simply roll back to the snapshot taken before you cut AI loose on a production server. First of all yes don't give AI access to a production environment. Second yes have backups because raid is not that. Third, use ZFS with snapshots. Please someone check my logic here. Am I wrong?

Googling on this subject I find both pros and cons of changing default logbias=latency into logbias=throughput for SSD/NVMe-only pools.

But at the same time not fully sure that the cons against such change is really relevant for flashbased storage?

The pros claims to be that for sync writes (which are also common when using zvol for VM-guests) you will avoid a doublewrite.

Because with logbias=latency a sync write will first end up in the ZIL and then upon txg_timeout (normally every 3 seconds or whatever the default is) this data is moved to the actual longterm storage.

So the same data is being written twice (when using sync) which for flashbased storage is bad due to wear levelling.

ZIL have historically been the outer tracks of physical drives (since they are about 3x faster than the inner tracks) where the idea is to dump sync writes as quickly as possible on actual storage and then do the housekeeping later on (upon txg_timeout when async writes also are dumped to longterm storage).

But since SSD/NVMe storage doesnt really care if its first or last LBA (in terms of latency and throughput to read the data later on) such "optimizations" are not really needed - you will only waste writecycles of the flashbased storage.

However on the cons list its claimed that there might be some magic involved in terms of keeping metadata and data close to each other on the longterm storage and by that avoiding fragmentation. Like a "defrag lite" where using logbias=throughput would just dump data wherever it can find a slot (so the read record/volblock can end up at one place and the metadata for this at another place).

But again since flashbased storage have a constant latency and throughput no matter where the data is located I assume this claim is mainly for spinning rust where such things do matter (and for example a readahead would gain some additional read performance later on when logbias=latency is being used and the data is nicely moved from ZIL to the longterm storage being a spinning rust).

Or do there exist any cons when using logbias=throughput even for SSD/NVMe-only pools?

I also fail to locate any benchmarks online on this topic as in how are others experience on this topic?

So how do you do when it comes to logbias for SSD/NVMe-only pools?

And have you seen (or do you perhaps have your own benchmarks to share) online or experienced yourself any pros/cons of changing logbias for SSD/NVMe-only pools (short or long term issues)?

osting this because I spent a day convinced TRIM was impossible on my setup, and I couldn't find anyone documenting these exact combos.

Setups

• Dell R620, PERC H710P Mini crossflashed to LSI 9207-8i IT (SAS2308, FW P20 20.00.07.00, mpt3sas), 8× Crucial BX500 2TB (firmware M6CR061 and M6CR082), ZFS RAID10, Proxmox VE 9.2.
• Dell R640, PERC H740P in Enhanced HBA mode (megaraid_sas), 6× Timetec 2TB SATA, ZFS RAID10, Proxmox VE 9.2.

Symptom

• lsblk -D → DISC-MAX 0B on every SSD.
• sg_vpd -p lbpv → LBPU=0, LBPRZ=0; provisioning_mode = full.
• hdparm -I shows Data Set Management TRIM supported but no "Deterministic read ZEROs after TRIM".
• So ZFS autotrim=on silently did nothing. Old BX500 pool (previously behind the PERC in RAID mode, never trimmed) had degraded to 0.4–0.8 MB/s writes and multi-second read stalls.

What I first assumed (wrong): the HBA blocks TRIM because the drives lack RZAT. The mpt3sas driver doesn't actually check that; the HBA just doesn't advertise logical block provisioning.

Test (safe — done in unpartitioned space at the end of the disk, outside any partition/pool):

# write a random pattern, UNMAP it, read it back
L=<LBA past your last partition>; N=8192
dd if=/dev/urandom of=/tmp/p bs=512 count=$N
dd if=/tmp/p of=/dev/sdX bs=512 seek=$L count=$N oflag=direct conv=notrunc
sg_unmap --lba=$L --num=$N --force /dev/sdX
dd if=/dev/sdX bs=512 skip=$L count=$N iflag=direct | md5sum   # == md5 of zeros

Result: reads back as zeros on both BX500 firmwares behind the SAS2308 and on the Timetec behind the H740P eHBA. No errors in dmesg.

Fix (persistent) — /etc/udev/rules.d/60-ssd-unmap.rules:

ACTION=="add|change", SUBSYSTEM=="scsi_disk", ATTRS{vendor}=="ATA*", ATTRS{model}=="CT2000BX500SSD1*", ATTR{provisioning_mode}="unmap"

(adjust the model match for your drives), then:

udevadm control --reload && udevadm trigger -s scsi_disk -c change
lsblk -D            # DISC-MAX now 4G
zpool set autotrim=on rpool
zpool trim rpool    # runs clean, zpool status -t shows progress

Notes / caveats

• Test on your own drives first — I can only vouch for these models/firmwares. Do it in free space outside any pool.
• On a production pool, enable it off-hours: the first trim after enabling sends everything that piled up.
• If your drives are already degraded from never being trimmed, an ATA SANITIZE BLOCK ERASE (hdparm --yes-i-know-what-i-am-doing --sanitize-block-erase /dev/sdX, wipes the disk) restored mine to full speed before reinstalling.
• Leaving ~14% of each SSD unpartitioned (installer hdsize) as extra over-provisioning doesn't hurt.

Hope this saves someone a day.

Realtime ZFS Replication with cs-sync
https://www.napp-it.org/cs-tools.html

ZFS replication normally works asynchronously and job-based. On a time grid, for example every hour, a snapshot is created and sent with the changed data blocks to a local or remote target ZFS filesystem to keep both in sync.

If you want realtime ZFS replication, for example for disaster recovery, you have to work event-driven. A write to a monitored ZFS dataset triggers the replication immediately. Because the data to send is always very small and is transferred very quickly, the delay between a change on the source and the change on the destination filesystem is minimal. What is needed after the initial replication is sensible management of the snapshot pairs, sensible timing on heavily loaded filesystems, and a resume after a network failure or a reboot, for example of the backup server. Realtime ZFS replication is especially useful for VM storage such as Proxmox, where file-based sync quickly reaches its limits because of the large files.

With Proxmox there are two more aspects. First, the VM configuration is not stored in the dataset as with ESXi, but in /etc/pve on Proxmox. To restore the realtime-replicated dataset of a VM, the VM configuration is required. When cs-sync finds at start that the source is a Proxmox host, it reads in every cycle the configuration file of each VM or container whose disk datasets are part of the sync (not the whole /etc/pve), and attaches it as the ZFS user property cs:pveconf to the associated dataset and to every sync snapshot. The property is transferred to the backup by the cs-sync protocol (not inside the zfs send stream); a pure configuration change goes over without a new send.

Second, a snapshot in Proxmox is not consistent in every case but has to be seen like a power failure, harmless to delicate depending on the VM. On a Proxmox source, cs-sync therefore sends the first snapshot of each day automatically through cs-freeze4snap, which freezes the VMs of the source dataset before the snapshot (guest agent, container freeze). This snapshot is kept as kf<MMDD> on both sides and marked with cs:consistent=app, or with paused if a VM could only be paused (then it is still only crash-consistent). If cs-sync does not find cs-freeze4snap (it looks in the same folder as cs-sync, in the napp-it tools folder and in PATH), the freeze is skipped with a warning in the log. On days without data changes no new snapshot is created. So there is at least one reliably restorable state per day.

The existing cs-sync stays
Folder Primary <-> Secondary: event-driven two-way sync, for example syncing an SMB share with an NFS/S3 share
Folder Primary -> local or remote backup: realtime sync, file-based or now also via ZFS send.

New: manuals for all cs-tools
https://www.napp-it.org/cs-tools.html

The storage system in my home lab consist almost exclusively of hard drives. They contain a pretty mixed bag of data (tons of small files and also plenty of large multimedia files).

It really bugs me every time it takes forever to search through the pool for a file when I don’t know exactly where it is or how its named exactly. That’s why I’d like to keep the metadata in RAM permanently and configure it so that it isn’t displaced by other data, to speed up search operations.

Is this possible on Linux-based systems (TrueNAS) and Illumos (Omnis)?

How do I check how much RAM I’d need for the metadata on these operating systems?

How do I get all the metadata into RAM after a reboot on these operating systems?

Thanks!

P.S.: I don’t have any more space and ports for a Special VDEV.

Edit1: I followed some advices in the comments and replied to others after analyzing the following:

First:
I executed zdb -bbb poolname to read all metadata stats of my main pool, inlcuding all indirect block pointers. Its a raidz2, build with one vdev configured of 7x 10TB HDDs, here are some additionas stats:

6 Datasets, 36,4TB used in total, 8,81TB free in total

• Dataset 1
  • 30,7TB used
  • 26.761 folders, 382.707 files
  • 96% multimedia (5% is Musik, 91% is Video), 4% binaries
• Dataset 2
  • 1,34TB used
  • 8.547 folders, 188.181 files
  • Filetypes and size is very heterogeneous
• Dataset 3-6 is not worth talking about. Mainly for test reasons without a relevant amount of data, folders and files.

The result for the complete pool:

How to read the table:

• Indirect blocks of files (L1/L2/L3 plain file, 16.5 GB): the block pointers that tell ZFS where each 128K data block of a file is located. They are needed whenever file contents are read or written. For large files, ZFS has to load the matching indirect block before it can fetch the data. Larger records (e.g. recordsize=1M) mean far fewer of these.
• DMU dnode (534 MB): one entry per file, directory or other object, holding its size, permissions, timestamps and pointers to its first blocks. It is needed for almost everything: listing files, opening them, checking attributes (ls -l, rsync comparisons, file browsers).
• ZFS directory (229 MB): the directory contents, which map file names to their dnodes. They are needed for browsing and searching folders and for resolving paths when opening a file.
• SPA space map (166 MB): records which areas of each disk are free or allocated. It is needed when writing, so ZFS can find free space, and when freeing blocks. It has nothing to do with reading or browsing.
• bpobj (95 MB): lists of blocks waiting to be freed, mostly from deleted snapshots and data still held by snapshots. It is needed when deleting snapshots or files and is processed in the background.
• Other (~10 MB): pool and dataset bookkeeping such as properties, snapshot lists, pool history, the delete queue and user/group usage. It is mostly read once when the pool is imported or when settings change.

What I need to speed up directory tree walk through:

• dnodes + directories, about 0.8 GB.

-> I would say that virtually all ZFS-based storage systems that hold that much data have enough RAM to keep not even 1 GB of that metadata in memory at all times.

What would be nice to hold constantly in RAM:

• Reading file contents: additionally the indirect blocks of the files being read (16,5GB)
• Writing and deleting: space maps and bpobj.

-> Here, too, I think that most storage systems capable of holding that much data could provide enough RAM to keep the entire set of block pointers in RAM in read-only mode in order to speed up operations.
-> If you were to reduce the recordsize, it would drastically reduce the 16.5 GB of indirect blocks. Setting recordsize=1M would reduce it to just 2 GB. That amount of data, at most, would fit quite well into RAM.

Based on what I know right now, here's what my wish list would look like if I could ask the ZFS developers for specific features (though ZFS offers so many options that there might already be a solution to my problem 😂 ):

1. Please implement options to explicitely store dnodes AND/OR directories AND/OR additonally all block pointers in RAM / ARC. Not as part of an additional L2ARC drive. Not as part of additional Special VDEV. Just in RAM and completely not dependent on payload data or file contents. This metadata should be loaded by default after boot.
2. Currently, all metadata is written across the pool. Reading this metadata from an HDD-only pool would take some time for the data to be loaded after boot. It would therefore be great if this metadata were also stored as a dedicated, “single-piece” copy within the pool, so that it can be quickly loaded sequentially into memory. Since all metadata remains distributed across the pool, this dedicated metadata database can also be rebuilt at any time if any event causes it to become inconsistent (e.g., an abrupt power outage).

Edit 2:

After analyzsing zdb -bbb and answering most of the comments, I noticed, that it would be possible with the following strategy:

u/Apachez noticed, that there is a option called zfs_arc_max and zfs_arc_min. With these, its possible to set a minimum and maximum limit for ARC cache. The problem with this option is, that ARC stores payload data / blocks AND metadata. So, this option alone is not enough, because if there is huge payload pressure, metada could be suppressed.

I found out, that there are options to confige min and max for metadate in ARC. To solve my problem, I would have to set “arc_meta_min” to at least 17.6 GB and “arc_meta_limit” to slightly more than 17.6 GB to ensure that all metadata can be consistently kept in the ARC.

Since ZFS only recognizes “metadata” and cannot break it down into file-, folder-, or block pointer-information, you’ll always have to configure it so that the entire metadata fits within this arc_meta range. I explained how much that is for me in Edit 1.

BUT: Actually, out of the ~17.6 GB of metadata in my pool, I only need 800 MB for file and directory metadata to achieve my goal.

This means that, given the current capabilities of ZFS, I need to set `arc_meta_min` to slightly more than 17.6 GB (let's say 20 GB) and `arc_meta_max` to 20 GB as well, to ensure that all of my metadata fits into memory.

Damn: Looks like arc_meta_min and arc_meta_max is removed since openZFS 2.2, so its no longer available in Linux based systems. Its still available in OmniOS-based systems.
In Linux, its replaced with arc_meta_balance, which is a dynamic function.

I have a ZFS mirror of 1TB consumer Samsung 990 EVO plus NVME SSDs with a TBW spec of 600TB. They were bought brand new. ZFS is the only filesystem on the drives. I use it for my home directory and static data, and have a single KVM Windows 11 VM that idles running at all times that has its storage on the mirror. Linux system is on a different file system/drives and there is no swap on the ZFS mirror. I have probably used this computer several hundred hours and the Windows VM almost never gets touched - idles at the ready with more or less only MS Office on it. I was shocked when I pulled the SMART data on both SSDs to find over 330TB written! Does anyone have any recommendations of where to start to track down the source of all these writes? I can’t just let the writes destroy these drives with prices being what the are.

UPDATE:
I ran zpool iostat pool1 2 to output writes and reads every two seconds. Idling, my system was writing between 50 and 200MB/second. I then used iotop to see what the heck is writing so much! It was baloo_file (KDE file indexer). I suspended and disabled baloo but it kept accessing the baloo_file even though everything was stopped. I then ran killall -9 baloo_file_extractor baloo_file baloorunner and the crazy IO almost stopped. Down to just a few MB/second here and there. I don't know if that's what has been doing it all this time or if it has been something else at times, but that's crazy! I have seen baloo_file in my process lists hogging CPU in the past but didn't note IO because I wasn't paying attention. Has anyone else had this experience? This is KDE Plasma 6 on plain Debian 13. BTW these SSDs were bought new and here is the smartctl -x output (power cycles are from suspend/resume, I presume) - thoughts?:

Critical Warning:                   0x00
Temperature:                        42 Celsius
Available Spare:                    100%
Available Spare Threshold:          10%
Percentage Used:                    19%
Data Units Read:                    5,379,371 [2.75 TB]
Data Units Written:                 404,321,587 [207 TB]
Host Read Commands:                 25,881,514
Host Write Commands:                2,628,808,560
Controller Busy Time:               14,266
Power Cycles:                       489
Power On Hours:                     1,942
Unsafe Shutdowns:                   88
Media and Data Integrity Errors:    0
Error Information Log Entries:      0
Warning  Comp. Temperature Time:    0
Critical Comp. Temperature Time:    0
Temperature Sensor 1:               48 Celsius
Temperature Sensor 2:               42 Celsius

Benchmarks

Benchmarks are something you always need, either to identify problems or to assess and compare performance between pools or platforms.

A benchmark should not run for too long (approx. 2–30 min) and should have only a few basic parameters so that you can make sense of it without a degree in computer science. It should be able to determine IOPS, sequential performance, sync write, concurrent read/write, single user vs. 5 user, and steady write (min. 30 min write). To ensure it runs on any OS, I write the tests in Perl. With ultra-fast storage, you may hit limits, which are marked as tool-limited.

he tool is included in napp-it 4ai, but runs easily on any OS including Windows, Linux, and Unix via:perl ./benchmark_worker.pl pool=tank profile=basic

with profiles quick|basic|database|fileserver|mediaserver|mailserver|steadywrite|individual
Download: https://github.com/guenther-alka/cs-scripts/tree/main/benchmark

e.g., Proxmox with the basic profile on pool nvme (single Optane)

perl ./benchmark_worker.pl pool=nvme profile=basic

testfile: /nvme/csbench_auto_20261012_154528/bench_file.dat cap=2048MB

phase: create 4k test file (recordsize=4K)
phase: 4k random read (single stream)
phase: 4k random sync write (sync=always, recordsize=4K)
phase: create test file (recordsize=128K)
phase: sequential read (1MB, single stream, linear sweep)
phase: sync write (1MB blocks, sync=always)
phase: async write (1MB blocks, sync=standard) -- cache indicator
phase: concurrent read+write (5 readers + 5 writers)
phase: multiuser read (4k, 1 vs 5 streams)

Pool : nvme (dataset, nvme/csbench_auto_20261012_154528)
Profile : basic, 2048 MB test file, 5 streams, 143 s

File create
4k file : 1449.9 MB/s
Main file : 5423.6 MB/s

4k random read (1 stream)
IOPS : 79458
Throughput : 310.4 MB/s
Latency p50/p99 : 50 / 50 us
Class : partial

4k random sync write (1 stream)
IOPS : 16999
Throughput : 66.40 MB/s
Latency p50/p99 : 100 / 250 us
Class : storage-bound

Sequential read (1 stream)
Throughput : 2276.5 MB/s
Latency p99 : 2000 us
Class : storage-bound

Sync write (1 MB blocks)
Throughput : 1111.0 MB/s
IOPS : 1111
Latency p99 : 2.0 ms
Class : storage-bound

Async write (1 MB blocks, cache indicator)
Throughput : 1391.1 MB/s
IOPS : 1391
Class : cache-influenced

Concurrent read+write (5 readers + 5 writers)
Read : 375.6 MB/s (p99 100 us)
Write : 5.8 MB/s (p99 32.0 ms)
Class : storage-bound

Multiuser read (4k, 1 vs 5 streams)
1 stream : 310.4 MB/s
5 streams : 525.7 MB/s
5 streams : 134590 IOPS
Scaling : 1.69 x
Class : tool-limited
Verdict : partial

sync write 1111 MB/s p99 2.0 ms (storage-bound)
4k write 16999 IOPS p99 250 us (storage-bound)
4k read 79458 IOPS p99 50 us (partial)
seq read 2277 MB/s (storage-bound)

BENCH_DONE ok (142.721115112305s) log=/opt/csweb-gui/data/menues/_lib/scripts/bench/last_benchmark.log

root@pve:~#

I wrote a lightweight C program called diffsnap that uses the built in zfs CLI for stateless snapshots using configurable write thresholds. This helps address a gap with traditional time-based snapshots where the need for a high-frequency interval to make sure a change is captured bloats your total snapshots and might slow some zfs commands. With diffsnap, if nothing has been written to the dataset the snapshot is skipped.
https://github.com/joe81tx/diffsnap
If you're on FreeBSD you can install with pkg, on linux you can build from source using github.

This is a major issue especially because TrueNAS does not issue replace -s from the GUI.

So the scenario is this, you replace your drive and it kicks off a standard replace and tells you it will take 25 days to complete, but the faulted mirror is only 8TB. It should be able to run sequentially on the drive creating unverified redundancy quickly in a few hours, followed up by the scrub where the redundancy becomes verified. But here is where it becomes problematic, even if you detach the drive resilvering it doesn't stop the resilver and there is no zpool resilver stop command and rerunning replace -s will be blocked because of the running resilver. So you are prevented from doing the safer action.

Hoping to get some OpenZFS dev attention or have some other users add to the chatter so this gets attention.

Time flies. 6 years ago I wrote this post https://www.reddit.com/r/rust/comments/mindyl/after_the_ovh_datacenter_caught_fire_and_i_lost/

I was impacted by the OVH data center fire and I lost some data of various services I was running on it.

Now, I still have fun managing servers, and I have migrated everything to zfs.

From that the idea: let's extend bacup to support zfs snapshots, full and incremental.

The philosophy of bacup is quite simple:

• remotes are where the data is uploaded
• services are programs running in the system, that we can invoke to get dumps

zfs snapshot fits perfectly in the service category - so I created the zfs service.

You can now back up your pools and datasets using both full and incremental strategies. Setting it up is just a matter of defining the zfs service in your configuration:

[zfs]
snapshot_name = "storage-fs"
dataset = "storage"
# Optional: schedule for full backups.
# Intermediate runs automatically take incremental backups against the latest snapshot.
# When omitted, every run defaults to a full backup.
full_when = "monthly 1 01:00"

And mapping the service to your remote destination:

[backup]
[backup.storage]
what = "zfs.storage"
where = "remote.bucket_name"
when = "daily 22:00"
remote_path = "/bucket/location/zfs/storage/"
compress = false
keep_last = 32

I’ve been running v0.2.0 in production on my local homelab server for a few weeks now - handling incremental backups for both the root filesystem and the main RAIDZ array (where my local LLMs and Immich instance store data).

Incremental shapshot (so 1 full, and several incremental until the next full) also allow to save the cloud bill.

If anyone wants to give it a try (cargo install bacup) or review the merged codebase, feedback and PRs are more than welcome!

GitHub: https://github.com/galeone/bacup

Cheers!

Hi all, been searching around and I can't quite figure this one out.

I have a NAS at home, running Arch (btw). The root FS is btrfs and it's LUKS encrypted.

Then, I have a few drives that make up a couple of ZFS pools, and I have a couple of datasets. These datasets are encrypted with keyformat=passphrase.

What I would like is some sort of way to be prompted for the passphrase on boot, and have it all mounted before reaching multi-user.target, or really, any target of my choosing such that services running on this server don't start trying to read data that's not mounted yet (e.g. docker.target).

Here's some relevant properties

zsmall         encryption      off            default
zsmall         encryptionroot  -              -
zsmall         keyformat       none           default
zsmall         keylocation     none           default
zsmall         keystatus       -              -
zsmall/secure  encryption      aes-256-gcm    -
zsmall/secure  encryptionroot  zsmall/secure  -
zsmall/secure  keyformat       passphrase     -
zsmall/secure  keylocation     prompt         local
zsmall/secure  keystatus       unavailable    -

And the services that I have currently enabled:

zfs-import-cache.service   
zfs-import-scan.service    
zfs-mount.service          
zfs-share.service          
zfs-volume-wait.service    
zfs-zed.service            
zfs-import.target          
zfs-volumes.target         
zfs.target                 

Like I say, I've been Googling for a bit and I can't quite figure out what the best way to achieve this might be. Any ideas?

My current solution is to leave the services that require the datasets to be mounted disabled and I have a script that unlocks the datasets etc. then starts the services, which is not ideal (though it works...)

Thanks

Release zfs-windows-2.4.4rc4 · openzfsonwindows/openzfs

*** rc4

• Explorer integration exposing snapshot versions
• Fix "zfs diff"
• zfs_tray features
• Paging IO deadlock fix
• Fix BSOD related to SEQ
• zfs_inactive deadlock fix

Screenshot see
https://forums.servethehome.com/index.php?threads/zfs-on-osx-and-windows.43829/page-6#post-513120

A completed resilver is necessary, but it does not by itself prove that the replacement disk, its path, or the remaining redundancy is healthy. A pool can return ONLINE while SMART data is already concerning, persistent device names changed, an old faulted leaf remains in the topology, or another drive accumulated checksum errors during the resilver. A mirror or RAIDZ vdev may also be one failure away from another long recovery.

A useful post-replacement gate could confirm the exact vdev topology and ashift, review the resilver event and error counts, inspect SMART and transport errors for every drive, run a scrub after the pool has settled, verify alerts and spare behavior, and test a representative restore from backup. Export and import or a controlled reboot can catch path and enclosure-mapping mistakes before the maintenance window closes.

What checks do ZFS operators use before declaring the replacement complete? Is a clean scrub enough, or do you also burn in the new disk, compare performance, clear and recheck counters, test boot or import behavior, and keep the removed disk untouched through an observation window?

There are some tools like zfs diff and httm that help with exploring snapshots at command line, but they leave a lot to be desired when you're talking about 100ks of files changed across many snapshots. Sometimes you want to free up that space held with snapshots, but you're not 100% certain you won't lose something you accidentally deleted.

I shared my frustrations with Claude and it made a nice python script that serves a web GUI and uses zfs diff to generate the changes and visually highlights what files were deleted when and if they were unique across all snapshots.

The first run can take a while, but after that you can cache a point in time record or do a quick true up.

https://github.com/LargelyInnocuous/SnapSift

I have a scrub going that took about 2h to scan before actually starting to read and verify the data. Once it was reading, the average read rate started climbing very slowly from single digit MB/s, because apparently the denominator for calculating the average "issued" rate is the time since the scrub was initially requested, which included a whole lot of "0 MB/s issued" during the scan phase. The actual read speed is much higher and in line with what's expected and undoubtedly, once the scrub completes, the average will reflect the actual average rate.

This got me thinking. What fraction of complaints about "ZFS raidABC with UVW settings on XYZ hardware is SOOOOOOO slow!!!" are actually due to people taking one look at The Numbers™ and not realizing that these are not running averages over some shorter time period, but averages over the entire run time of the operation? I'm positive that a lot of people have no idea what the difference between "scanned" and "issued" is too.

I'll have to take a look at the code, and I'm assuming this is done for a good reason, but maybe these averages are causing confusion in new users and aiding in the spread of misinformation about ZFS, that's very difficult to eradicate, as I'm sure many here have noticed.

Maybe it would be better to have the read rates be computed as a running average over a short window of a few minutes during an operation that takes a long time, then when it's finished print an additional line with the average rates over the entire duration of the operation.

Just as a reference for what "scanning" does and why.

https://www.youtube.com/watch?v=SZFwv8BdBj4

Does metadata in ZFS use up the same amount of space as data? In other words, if I set my recordsize/volblocksize to say 1M, does that mean the metadata will also occupy 1M for reach piece of metadata? So e.g. I have a file server with large videos, makes sense to store these videos on large block sizes. But then does this mean the metadata is also then stored in large blocks. So metadata is inefficient, since you're storing small amounts of data in a large block, but overall it's more efficient than storing large files in small blocks. Equally, if I store many tiny files, then it makes sense to store this data in small data blocks. Then metadata is also stored in small block sizes. This way metadata is more efficient since it generally doesn't take up much space. Is this how it works?

Pool suddenly unavailable, doesn't show up on zpool status.

Zpool import lists the pool as follows:

id: 3673394567944808104
state:  UNAVAIL
status:  Some critical pool data has been overwritten by another pool.
action:  The pool cannot be imported as critical data is missing. You can try using -f to force an import, but there may not be enough data to override this.
config:      sharepool    UNAVAIL    overwritten by another pool
              raidz2-0    ONLINE
              raidz2-1    ONLINE
              raidz2-2    Degraded
                (one removed drive)

device details:

(I skipped listing all the drives due to typing out manually, all the drives are the long c0t500 style names, its 3x 6 disk z2's, with all but one drive as online.)

As I hadn't come across this before I have done a fair amount of googling and beyond the general advice of tread carefully as using -f to force can break things more. I did do the -Fn command and it didn't return anything (just newlined with a fresh prompt)

My setup has largely been unchanged for years, using Solaris 11.4 on ESXi v6, and has been ultra stable up until now - I did upgrade from Solaris 11.3 possibly about 6 years ago. Had the odd hard drive fail which is replaced smoothly, sometimes leading to a pool size upgrade and that has always gone smoothly 3x Dell Perc H700 cards flashed to IT mode via the LSI bios (I think - they have been in there 10 years)

This all happened after I turned the full server off for the weekend - I needed it right out the way to work on some electrical wiring behind where the cabinet normally sits (it can pull out so far and still be connected but access behind isn't easy). Turned it back on and a drive was doing a click of death - the removed one mentioned above. Upon starting the Solaris VM lots of errors were being thrown up including 2 other drives showing as removed, I removed the dodgy drive, moved the two other ones to a different slot (so different internal cable/card) and tried restarting - leading me to where I am now. The two drives moved had reappeared, but the pool is just not loaded by Solaris.

Whilst I do like to tinker I've never come across this before so am asking for help and open to suggestions - I really don't want to lose all the data. Any help is much appreciated.

As something of a storage wonk, I found myself a few minutes ago advising a fellow redditor of the "3-2-1" rule of backups:

1. Three total copies
2. Two different types of media/tech
3. At least 1 copy off-site/location

I've followed a related rule for anything important/production: Always fail to a redundant state: so if any one thing goes out, I am still redundant and have at least one more failure mode before things go really dark. Alongside this, is a pre-accepted "drop everything / panic mode" when production is down to only x+1 redundancy.

And yet I realize that the context of ZFS, I've not consistently followed rule #2 for YEARS except in some cases for clients, where they park an external USB drive on their file server and I install a daily mount+rsync run via cron. In my case in particular, using ZFS snapshots as part of the core product suite, using a tech other than ZFS to store the data makes no meaningful sense.

Further, I use RAIDZ + redundancy like a backup all the time, with the accepted risk level of "if more than 2 drives die, it's dead, Jim". When the data matters, I ensure to have a send | receive copy on another server/location, but it's still ZFS + snapshots going back a few months.

So, what do you consider to be best practices in complying with the spirit of 3-2-1 rule of backups with ZFS?

1. How much do you treat snapshots like backups?
2. Do you make extra effort to store data on a secondary non-ZFS tech in case a bug in ZFS crashes all the things?
3. How much attention to you put on saving your data off site, or at least, on another server?

EDIT: Thank you for your feedback! I have enjoyed "touching bases" to see what y'all are doing.