vdev

The answer to changing the sector size of a vdev is similar, just like it is for removing a disk from a vdev, or reconfiguring your redundancy in most cases.

However, in cases where you are using SSDs for ZIL/L2ARC, vdev spam isn't as big of a factor.

Best practice is to build a new vdev with the 4x4TB drives and move the entire dataset across.

One thing to note: the '-o ashift' option is a per-drive (vdev) attribute according to the manual.

So it's generally only a problem if you are replacing an existing drive in a vdev after it failed.

If you make one 90 drive vdev, you are more or less limiting yourself to the iops of a single disk.

You can just add a new raidz (or any type of vdev) to the zpool.

As noted above by BrainInAJar, vdev removal just hit the Illumos ZFS tree.

You could add another 4x3TB vdev to the pool to add more storage to the pool, but once you did you couldn't remove it.

So, when replacing a vdev, be sure to specify this option again if you do not use the default: zpool replace -o ashift=12 old new.

Zfs sector size (ashift parameter) is set at vdev creation time.

I'd add one more: it is impossible to remove a vdev from a pool.

Syntax is: zdb -R poolname vdev:offset:size:d The "d" at the end says to decompress.

"losing" three disks in a raidz2 vdev, my guess is that you would just replace the faulty card and reimport the pool and chances are things would be OK...