This chapter describes how to create and administer ZFS storage pools.The following sections are provided in this chapter:Components of a ZFS Storage Pool Creating and Destroying ZFS Storage Pools Managing Devices in ZFS Storage Pools Managing ZFS Storage Pool Properties Querying ZFS Storage Pool Status Migrating ZFS Storage Pools Upgrading ZFS Storage Pools The following sections provide detailed information about the following storage pool components:Using Disks in a ZFS Storage Pool Using Files in a ZFS Storage Pool Identifying Virtual Devices in a Storage Pool The most basic element of a storage pool is a piece of physical storage. Physical storage can be any block device of at least 128 Mbytes in size. Typically, this device is a hard drive that is visible to the system in the /dev/dsk directory.A storage device can be a whole disk (c1t0d0) or an individual slice (c0t0d0s7). The recommended mode of operation is to use an entire disk, in which case the disk does not need to be specially formatted. ZFS formats the disk using an EFI label to contain a single, large slice. When used in this way, the partition table that is displayed by the format command appears similar to the following:Current partition table (original): Total disk sectors available: 71670953 + 16384 (reserved sectors) Part Tag Flag First Sector Size Last Sector 0 usr wm 34 34.18GB 71670953 1 unassigned wm 0 0 0 2 unassigned wm 0 0 0 3 unassigned wm 0 0 0 4 unassigned wm 0 0 0 5 unassigned wm 0 0 0 6 unassigned wm 0 0 0 7 unassigned wm 0 0 0 8 reserved wm 71670954 8.00MB 71687337To use whole disks, the disks must be named using the standard Solaris convention, such as /dev/dsk/cXtXdXsX. Some third-party drivers use a different naming convention or place disks in a location other than the /dev/dsk directory. To use these disks, you must manually label the disk and provide a slice to ZFS. ZFS applies an EFI label when you create a storage pool with whole disks. Disks can be labeled with a traditional Solaris VTOC label when you create a storage pool with a disk slice.Slices should only be used under the following conditions:The device name is nonstandard. A single disk is shared between ZFS and another file system, such as UFS. A disk is used as a swap or a dump device. Disks can be specified by using either the full path, such as /dev/dsk/c1t0d0, or a shorthand name that consists of the device name within the /dev/dsk directory, such as c1t0d0. For example, the following are valid disk names:c1t0d0 /dev/dsk/c1t0d0 c0t0d6s2 /dev/foo/disk Using whole physical disks is the simplest way to create ZFS storage pools. ZFS configurations become progressively more complex, from management, reliability, and performance perspectives, when you build pools from disk slices, LUNs in hardware RAID arrays, or volumes presented by software-based volume managers. The following considerations might help you determine how to configure ZFS with other hardware or software storage solutions:If you construct ZFS configurations on top of LUNs from hardware RAID arrays, you need to understand the relationship between ZFS redundancy features and the redundancy features offered by the array. Certain configurations might provide adequate redundancy and performance, but other configurations might not. You can construct logical devices for ZFS using volumes presented by software-based volume managers, such as Solaris Volume Manager (SVM) or Veritas Volume Manager (VxVM). However, these configurations are not recommended. While ZFS functions properly on such devices, less-than-optimal performance might be the result. For additional information about storage pool recommendations, see the ZFS best practices site:http://www.solarisinternals.com/wiki/index.php/ZFS_Best_Practices_GuideDisks are identified both by their path and by their device ID, if available. This method allows devices to be reconfigured on a system without having to update any ZFS state. If a disk is switched between controller 1 and controller 2, ZFS uses the device ID to detect that the disk has moved and should now be accessed using controller 2. The device ID is unique to the drive's firmware. While unlikely, some firmware updates have been known to change device IDs. If this situation happens, ZFS can still access the device by path and update the stored device ID automatically. If you inadvertently change both the path and the ID of the device, then export and re-import the pool in order to use it. ZFS also allows you to use UFS files as virtual devices in your storage pool. This feature is aimed primarily at testing and enabling simple experimentation, not for production use. The reason is that any use of files relies on the underlying file system for consistency. If you create a ZFS pool backed by files on a UFS file system, then you are implicitly relying on UFS to guarantee correctness and synchronous semantics.However, files can be quite useful when you are first trying out ZFS or experimenting with more complicated layouts when not enough physical devices are present. All files must be specified as complete paths and must be at least 64 Mbytes in size. If a file is moved or renamed, the pool must be exported and re-imported in order to use it, as no device ID is associated with files by which they can be located. Each storage pool is comprised of one or more virtual devices. A virtual device is an internal representation of the storage pool that describes the layout of physical storage and its fault characteristics. As such, a virtual device represents the disk devices or files that are used to create the storage pool. Two top-level virtual devices provide data redundancy: mirror and RAID-Z virtual devices. These virtual devices consist of disks, disk slices, or files.Disks, disk slices, or files that are used in pools outside of mirrors and RAID-Z virtual devices, function as top-level virtual devices themselves.Storage pools typically contain multiple top-level virtual devices. ZFS dynamically stripes data among all of the top-level virtual devices in a pool. ZFS provides data redundancy, as well as self-healing properties, in a mirrored and a RAID-Z configuration.Mirrored Storage Pool Configuration RAID-Z Storage Pool Configuration Self-Healing Data in a Redundant Configuration Dynamic Striping in a Storage Pool A mirrored storage pool configuration requires at least two disks, preferably on separate controllers. Many disks can be used in a mirrored configuration. In addition, you can create more than one mirror in each pool. Conceptually, a simple mirrored configuration would look similar to the following:mirror c1t0d0 c2t0d0Conceptually, a more complex mirrored configuration would look similar to the following:mirror c1t0d0 c2t0d0 c3t0d0 mirror c4t0d0 c5t0d0 c6t0d0For information about creating a mirrored storage pool, see Creating a Mirrored Storage Pool. In addition to a mirrored storage pool configuration, ZFS provides a RAID-Z configuration with either single or double parity fault tolerance. Single-parity RAID-Z is similar to RAID-5. Double-parity RAID-Z is similar to RAID-6.All traditional RAID-5-like algorithms (RAID-4. RAID-5. RAID-6, RDP, and EVEN-ODD, for example) suffer from a problem known as the “RAID-5 write hole.” If only part of a RAID-5 stripe is written, and power is lost before all blocks have made it to disk, the parity will remain out of sync with the data, and therefore useless, forever (unless a subsequent full-stripe write overwrites it). In RAID-Z, ZFS uses variable-width RAID stripes so that all writes are full-stripe writes. This design is only possible because ZFS integrates file system and device management in such a way that the file system's metadata has enough information about the underlying data redundancy model to handle variable-width RAID stripes. RAID-Z is the world's first software-only solution to the RAID-5 write hole.You need at least two disks for a RAID-Z configuration. Otherwise, no special hardware is required to create a RAID-Z configuration. Currently, RAID-Z provides single parity. For example, if you have three disks in a RAID-Z configuration, parity data occupies space equal to one of the three disks. A RAID-Z configuration with N disks of size X with P parity disks can hold approximately (N-P)*X bytes and can withstand P device(s) failing before data integrity is compromised. You need at least two disks for a single-parity RAID-Z configuration and at least three disks for a double-parity RAID-Z configuration. For example, if you have three disks in a single-parity RAID-Z configuration, parity data occupies space equal to one of the three disks. Otherwise, no special hardware is required to create a RAID-Z configuration. Conceptually, a RAID-Z configuration with three disks would look similar to the following:raidz c1t0d0 c2t0d0 c3t0d0A more complex conceptual RAID-Z configuration would look similar to the following:raidz c1t0d0 c2t0d0 c3t0d0 c4t0d0 c5t0d0 c6t0d0 c7t0d0 raidz c8t0d0 c9t0d0 c10t0d0 c11t0d0 c12t0d0 c13t0d0 c14t0d0If you are creating a RAID-Z configuration with many disks, as in this example, a RAID-Z configuration with 14 disks is better split into a two 7-disk groupings. RAID-Z configurations with single-digit groupings of disks should perform better.For information about creating a RAID-Z storage pool, see Creating RAID-Z Storage Pools.For more information about choosing between a mirrored configuration or a RAID-Z configuration based on performance and space considerations, see the following blog:http://blogs.sun.com/roller/page/roch?entry=when_to_and_not_toFor additional information on RAID-Z storage pool recommendations, see the ZFS best practices site:http://www.solarisinternals.com/wiki/index.php/ZFS_Best_Practices_Guide ZFS provides for self-healing data in a mirrored or RAID-Z configuration.When a bad data block is detected, not only does ZFS fetch the correct data from another redundant copy, but it also repairs the bad data by replacing it with the good copy. For each virtual device that is added to the pool, ZFS dynamically stripes data across all available devices. The decision about where to place data is done at write time, so no fixed width stripes are created at allocation time.When virtual devices are added to a pool, ZFS gradually allocates data to the new device in order to maintain performance and space allocation policies. Each virtual device can also be a mirror or a RAID-Z device that contains other disk devices or files. This configuration allows for flexibility in controlling the fault characteristics of your pool. For example, you could create the following configurations out of 4 disks:Four disks using dynamic striping One four-way RAID-Z configuration Two two-way mirrors using dynamic striping While ZFS supports combining different types of virtual devices within the same pool, this practice is not recommended. For example, you can create a pool with a two-way mirror and a three-way RAID-Z configuration. However, your fault tolerance is as good as your worst virtual device, RAID-Z in this case. The recommended practice is to use top-level virtual devices of the same type with the same redundancy level in each device. The following sections describe different scenarios for creating and destroying ZFS storage pools.Creating a ZFS Storage Pool Handling ZFS Storage Pool Creation Errors Destroying ZFS Storage Pools By design, creating and destroying pools is fast and easy. However, be cautious when doing these operations. Although checks are performed to prevent using devices known to be in use in a new pool, ZFS cannot always know when a device is already in use. Destroying a pool is even easier. Use zpool destroy with caution. This is a simple command with significant consequences. To create a storage pool, use the zpool create command. This command takes a pool name and any number of virtual devices as arguments. The pool name must satisfy the naming conventions outlined in ZFS Component Naming Requirements.The following command creates a new pool named tank that consists of the disks c1t0d0 and c1t1d0:# zpool create tank c1t0d0 c1t1d0These whole disks are found in the /dev/dsk directory and are labelled appropriately by ZFS to contain a single, large slice. Data is dynamically striped across both disks. To create a mirrored pool, use the mirror keyword, followed by any number of storage devices that will comprise the mirror. Multiple mirrors can be specified by repeating the mirror keyword on the command line. The following command creates a pool with two, two-way mirrors:# zpool create tank mirror c1d0 c2d0 mirror c3d0 c4d0The second mirror keyword indicates that a new top-level virtual device is being specified. Data is dynamically striped across both mirrors, with data being redundant between each disk appropriately.Currently, the following operations are supported on a ZFS mirrored configuration:Adding another set of disks for an additional top-level vdev to an existing mirrored configuration. For more information, see Adding Devices to a Storage Pool. Attaching additional disks to an existing mirrored configuration. Or, attaching additional disks to a non-replicated configuration to create a mirrored configuration. For more information, see Attaching and Detaching Devices in a Storage Pool. Replace a disk or disks in an existing mirrored configuration as long as the replacement disks are greater than or equal to the device to be replaced. For more information, see Replacing Devices in a Storage Pool. Detach a disk or disk in a mirrored configuration as long as the remaining devices provide adequate redundancy for the configuration. For more information, see Attaching and Detaching Devices in a Storage Pool. Currently, the following operations are not supported on a mirrored configuration:You cannot outright remove a device from a mirrored storage pool. An RFE is filed for this feature. You cannot split or break a mirror for backup purposes. An RFE is filed for this feature. Creating a single-parity RAID-Z pool is identical to creating a mirrored pool, except that the raidz or raidz1 keyword is used instead of mirror. The following example shows how to create a pool with a single RAID-Z device that consists of five disks:# zpool create tank raidz c1t0d0 c2t0d0 c3t0d0 c4t0d0 /dev/dsk/c5t0d0This example demonstrates that disks can be specified by using their full paths. The /dev/dsk/c5t0d0 device is identical to the c5t0d0 device.A similar configuration could be created with disk slices. For example:# zpool create tank raidz c1t0d0s0 c2t0d0s0 c3t0d0s0 c4t0d0s0 c5t0d0s0However, the disks must be preformatted to have an appropriately sized slice zero.You can create a double-parity RAID-Z configuration by using the raidz2 keyword when the pool is created. For example:# zpool create tank raidz2 c1t0d0 c2t0d0 c3t0d0 # zpool status -v tank pool: tank state: ONLINE scrub: none requested config: NAME STATE READ WRITE CKSUM tank ONLINE 0 0 0 raidz2 ONLINE 0 0 0 c1t0d0 ONLINE 0 0 0 c2t0d0 ONLINE 0 0 0 c3t0d0 ONLINE 0 0 0 errors: No known data errorsCurrently, the following operations are supported on a ZFS RAID-Z configuration:Add another set of disks for an additional top-level vdev to an existing RAID-Z configuration. For more information, see Adding Devices to a Storage Pool. Replace a disk or disks in an existing RAID-Z configuration as long as the replacement disks are greater than or equal to the device to be replaced. For more information, see Replacing Devices in a Storage Pool. Currently, the following operations are not supported on a RAID-Z configuration:Attach an additional disk to an existing RAID-Z configuration. Detach a disk from a RAID-Z configuration. You cannot outright remove a device from a RAID-Z configuration. An RFE is filed for this feature. For more information about a RAID-Z configuration, see RAID-Z Storage Pool Configuration. By default, the ZIL is allocated from blocks within the main pool. However, better performance might be possible by using separate intent log devices, such as NVRAM or a dedicated disk. For more information about ZFS log devices, see Setting Up Separate ZFS Logging Devices.You can set up a ZFS logging device when the storage pool is created or after the pool is created.For example, create a mirrored storage pool with mirrored log devices.# zpool create datap mirror c1t1d0 c1t2d0 mirror c1t3d0 c1t4d0 log mirror c1t5d0 c1t8d0 # zpool status pool: datap state: ONLINE scrub: none requested config: NAME STATE READ WRITE CKSUM datap ONLINE 0 0 0 mirror ONLINE 0 0 0 c1t1d0 ONLINE 0 0 0 c1t2d0 ONLINE 0 0 0 mirror ONLINE 0 0 0 c1t3d0 ONLINE 0 0 0 c1t4d0 ONLINE 0 0 0 logs ONLINE 0 0 0 mirror ONLINE 0 0 0 c1t5d0 ONLINE 0 0 0 c1t8d0 ONLINE 0 0 0 errors: No known data errors You can create a storage pool with cache devices to cache storage pool data. For example:# zpool create tank mirror c2t0d0 c2t1d0 c2t3d0 cache c2t5d0 c2t8d0 # zpool status tank pool: tank state: ONLINE scrub: none requested config: NAME STATE READ WRITE CKSUM tank ONLINE 0 0 0 mirror ONLINE 0 0 0 c2t0d0 ONLINE 0 0 0 c2t1d0 ONLINE 0 0 0 c2t3d0 ONLINE 0 0 0 cache c2t5d0 ONLINE 0 0 0 c2t8d0 ONLINE 0 0 0Review the following points when considering whether to create a ZFS storage pool with cache devices:Using cache devices provide the greatest performance improvement for random read-workloads of mostly static content. Capacity and reads can be monitored by using the zpool iostat command. Single or multiple cache devices can be added when the pool is created or added and removed after the pool is created. For more information, see Example 4–3. Cache devices cannot be mirrored or be part of a RAID-Z configuration. If a read error is encountered on a cache device, that read I/O is reissued to the original storage pool device, which might be part of a mirrored or RAID-Z configuration. The content of the cache devices is considered volatile, as is the case with other system caches. Pool creation errors can occur for many reasons. Some of these reasons are obvious, such as when a specified device doesn't exist, while other reasons are more subtle.Before formatting a device, ZFS first determines if the disk is in use by ZFS or some other part of the operating system. If the disk is in use, you might see errors such as the following:# zpool create tank c1t0d0 c1t1d0 invalid vdev specification use '-f' to override the following errors: /dev/dsk/c1t0d0s0 is currently mounted on /. Please see umount(1M). /dev/dsk/c1t0d0s1 is currently mounted on swap. Please see swap(1M). /dev/dsk/c1t1d0s0 is part of active ZFS pool zeepool. Please see zpool(1M).Some of these errors can be overridden by using the f option, but most errors cannot. The following uses cannot be overridden by using the f option, and you must manually correct them:Mounted file systemThe disk or one of its slices contains a file system that is currently mounted. To correct this error, use the umount command. File system in /etc/vfstabThe disk contains a file system that is listed in the /etc/vfstab file, but the file system is not currently mounted. To correct this error, remove or comment out the line in the /etc/vfstab file. Dedicated dump deviceThe disk is in use as the dedicated dump device for the system. To correct this error, use the dumpadm command. Part of a ZFS poolThe disk or file is part of an active ZFS storage pool. To correct this error, use the zpool command to destroy the pool. The following in-use checks serve as helpful warnings and can be overridden by using the f option to create the pool:Contains a file systemThe disk contains a known file system, though it is not mounted and doesn't appear to be in use. Part of volumeThe disk is part of an SVM volume. Live upgradeThe disk is in use as an alternate boot environment for Solaris Live Upgrade. Part of exported ZFS poolThe disk is part of a storage pool that has been exported or manually removed from a system. In the latter case, the pool is reported as potentially active, as the disk might or might not be a network-attached drive in use by another system. Be cautious when overriding a potentially active pool. The following example demonstrates how the f option is used:# zpool create tank c1t0d0 invalid vdev specification use '-f' to override the following errors: /dev/dsk/c1t0d0s0 contains a ufs filesystem. # zpool create -f tank c1t0d0Ideally, correct the errors rather than use the f option. Creating pools with virtual devices of different replication levels is not recommended. The zpool command tries to prevent you from accidentally creating a pool with mismatched levels of redundancy. If you try to create a pool with such a configuration, you see errors similar to the following:# zpool create tank c1t0d0 mirror c2t0d0 c3t0d0 invalid vdev specification use '-f' to override the following errors: mismatched replication level: both disk and mirror vdevs are present # zpool create tank mirror c1t0d0 c2t0d0 mirror c3t0d0 c4t0d0 c5t0d0 invalid vdev specification use '-f' to override the following errors: mismatched replication level: 2-way mirror and 3-way mirror vdevs are presentYou can override these errors with the f option, though this practice is not recommended. The command also warns you about creating a mirrored or RAID-Z pool using devices of different sizes. While this configuration is allowed, mismatched levels of redundancy result in unused space on the larger device, and requires the f option to override the warning. Because creating a pool can fail unexpectedly in different ways, and because formatting disks is such a potentially harmful action, the zpool create command has an additional option, n, which simulates creating the pool without actually writing data to disk. This option performs the device in-use checking and replication level validation, and reports any errors in the process. If no errors are found, you see output similar to the following:# zpool create -n tank mirror c1t0d0 c1t1d0 would create 'tank' with the following layout: tank mirror c1t0d0 c1t1d0Some errors cannot be detected without actually creating the pool. The most common example is specifying the same device twice in the same configuration. This error cannot be reliably detected without writing the data itself, so the create -n command can report success and yet fail to create the pool when run for real. When a pool is created, the default mount point for the root dataset is /pool-name. This directory must either not exist or be empty. If the directory does not exist, it is automatically created. If the directory is empty, the root dataset is mounted on top of the existing directory. To create a pool with a different default mount point, use the m option of the zpool create command:# zpool create home c1t0d0 default mountpoint '/home' exists and is not empty use '-m' option to specify a different default # zpool create -m /export/zfs home c1t0d0# zpool create home c1t0d0 default mountpoint '/home' exists and is not empty use '-m' option to provide a different default # zpool create -m /export/zfs home c1t0d0This command creates a new pool home and the home dataset with a mount point of /export/zfs.For more information about mount points, see Managing ZFS Mount Points. Pools are destroyed by using the zpool destroy command. This command destroys the pool even if it contains mounted datasets.# zpool destroy tankBe very careful when you destroy a pool. Make sure you are destroying the right pool and you always have copies of your data. If you accidentally destroy the wrong pool, you can attempt to recover the pool. For more information, see Recovering Destroyed ZFS Storage Pools. The act of destroying a pool requires that data be written to disk to indicate that the pool is no longer valid. This state information prevents the devices from showing up as a potential pool when you perform an import. If one or more devices are unavailable, the pool can still be destroyed. However, the necessary state information won't be written to these damaged devices.These devices, when suitably repaired, are reported as potentially active when you create a new pool, and appear as valid devices when you search for pools to import. If a pool has enough faulted devices such that the pool itself is faulted (meaning that a top-level virtual device is faulted), then the command prints a warning and cannot complete without the f option. This option is necessary because the pool cannot be opened, so whether data is stored there or not is unknown. For example:# zpool destroy tank cannot destroy 'tank': pool is faulted use '-f' to force destruction anyway # zpool destroy -f tankFor more information about pool and device health, see Determining the Health Status of ZFS Storage Pools.For more information about importing pools, see Importing ZFS Storage Pools. Most of the basic information regarding devices is covered in Components of a ZFS Storage Pool. Once a pool has been created, you can perform several tasks to manage the physical devices within the pool.Adding Devices to a Storage Pool Attaching and Detaching Devices in a Storage Pool Onlining and Offlining Devices in a Storage Pool Clearing Storage Pool Devices Replacing Devices in a Storage Pool Designating Hot Spares in Your Storage Pool You can dynamically add space to a pool by adding a new top-level virtual device. This space is immediately available to all datasets within the pool. To add a new virtual device to a pool, use the zpool add command. For example:# zpool add zeepool mirror c2t1d0 c2t2d0The format of the virtual devices is the same as for the zpool create command, and the same rules apply. Devices are checked to determine if they are in use, and the command cannot change the level of redundancy without the f option. The command also supports the n option so that you can perform a dry run. For example:# zpool add -n zeepool mirror c3t1d0 c3t2d0 would update 'zeepool' to the following configuration: zeepool mirror c1t0d0 c1t1d0 mirror c2t1d0 c2t2d0 mirror c3t1d0 c3t2d0This command syntax would add mirrored devices c3t1d0 and c3t2d0 to zeepool's existing configuration.For more information about how virtual device validation is done, see Detecting in Use Devices.Additional disks can be added similarly to a RAID-Z configuration. The following example shows how to convert a storage pool with one RAID–Z device comprised of 3 disks to a storage pool with two RAID-Z devices comprised of 3 disks.# zpool status pool: rpool state: ONLINE scrub: none requested config: NAME STATE READ WRITE CKSUM rpool ONLINE 0 0 0 raidz1 ONLINE 0 0 0 c1t2d0 ONLINE 0 0 0 c1t3d0 ONLINE 0 0 0 c1t4d0 ONLINE 0 0 0 errors: No known data errors # zpool add rpool raidz c2t2d0 c2t3d0 c2t4d0 # zpool status pool: rpool state: ONLINE scrub: none requested config: NAME STATE READ WRITE CKSUM rpool ONLINE 0 0 0 raidz1 ONLINE 0 0 0 c1t2d0 ONLINE 0 0 0 c1t3d0 ONLINE 0 0 0 c1t4d0 ONLINE 0 0 0 raidz1 ONLINE 0 0 0 c2t2d0 ONLINE 0 0 0 c2t3d0 ONLINE 0 0 0 c2t4d0 ONLINE 0 0 0 errors: No known data errors The following example shows how to add a mirrored log device to mirrored storage pool.For more information about using log devices in your storage pool, see Setting Up Separate ZFS Logging Devices.# zpool status newpool pool: newpool state: ONLINE scrub: none requested config: NAME STATE READ WRITE CKSUM newpool ONLINE 0 0 0 mirror ONLINE 0 0 0 c1t9d0 ONLINE 0 0 0 c1t10d0 ONLINE 0 0 0 errors: No known data errors # zpool add newpool log mirror c1t11d0 c1t12d0 # zpool status newpool pool: newpool state: ONLINE scrub: none requested config: NAME STATE READ WRITE CKSUM newpool ONLINE 0 0 0 mirror ONLINE 0 0 0 c1t9d0 ONLINE 0 0 0 c1t10d0 ONLINE 0 0 0 logs ONLINE 0 0 0 mirror ONLINE 0 0 0 c1t11d0 ONLINE 0 0 0 c1t12d0 ONLINE 0 0 0 errors: No known data errorsYou can attach a log device to an existing log device to create a mirrored log device. This operation is identical to attaching a device in a unmirrored storage pool. You can add and remove cache devices to your ZFS storage pool.Use the zpool add command to add cache devices. For example:# zpool add tank cache c2t5d0 c2t8d0 # zpool status tank pool: tank state: ONLINE scrub: none requested config: NAME STATE READ WRITE CKSUM tank ONLINE 0 0 0 mirror ONLINE 0 0 0 c2t0d0 ONLINE 0 0 0 c2t1d0 ONLINE 0 0 0 c2t3d0 ONLINE 0 0 0 cache c2t5d0 ONLINE 0 0 0 c2t8d0 ONLINE 0 0 0 errors: No known data errorsCache devices cannot be mirrored or be part of a RAID-Z configuration.Use the zpool remove command to remove cache devices. For example:E zpool remove tank c2t5d0 c2t8d0 # zpool status tank pool: tank state: ONLINE scrub: none requested config: NAME STATE READ WRITE CKSUM tank ONLINE 0 0 0 mirror ONLINE 0 0 0 c2t0d0 ONLINE 0 0 0 c2t1d0 ONLINE 0 0 0 c2t3d0 ONLINE 0 0 0 errors: No known data errorsCurrently, the zpool remove command only supports removing hot spares and cache devices. Devices that are part of the main mirrored pool configuration can be removed by using the zpool detach command. Non-redundant and RAID-Z devices cannot be removed from a pool.For more information about using cache devices in a ZFS storage pool, see Creating a ZFS Storage Pool with Cache Devices. In addition to the zpool add command, you can use the zpool attach command to add a new device to an existing mirrored or non-mirrored device. In this example, zeepool is an existing two-way mirror that is transformed to a three-way mirror by attaching c2t1d0, the new device, to the existing device, c1t1d0.# zpool status pool: zeepool state: ONLINE scrub: none requested config: NAME STATE READ WRITE CKSUM zeepool ONLINE 0 0 0 mirror ONLINE 0 0 0 c0t1d0 ONLINE 0 0 0 c1t1d0 ONLINE 0 0 0 errors: No known data errors # zpool attach zeepool c1t1d0 c2t1d0 # zpool status pool: zeepool state: ONLINE scrub: resilver completed with 0 errors on Fri Jan 12 14:47:36 2007 config: NAME STATE READ WRITE CKSUM zeepool ONLINE 0 0 0 mirror ONLINE 0 0 0 c0t1d0 ONLINE 0 0 0 c1t1d0 ONLINE 0 0 0 c2t1d0 ONLINE 0 0 0If the existing device is part of a two-way mirror, attaching the new device, creates a three-way mirror, and so on. In either case, the new device begins to resilver immediately. In addition, you can convert a non-redundant storage pool into a redundant storage pool by using the zpool attach command. For example:# zpool create tank c0t1d0 # zpool status pool: tank state: ONLINE scrub: none requested config: NAME STATE READ WRITE CKSUM tank ONLINE 0 0 0 c0t1d0 ONLINE 0 0 0 errors: No known data errors # zpool attach tank c0t1d0 c1t1d0 # zpool status pool: tank state: ONLINE scrub: resilver completed with 0 errors on Fri Jan 12 14:55:48 2007 config: NAME STATE READ WRITE CKSUM tank ONLINE 0 0 0 mirror ONLINE 0 0 0 c0t1d0 ONLINE 0 0 0 c1t1d0 ONLINE 0 0 0 You can use the zpool detach command to detach a device from a mirrored storage pool. For example:# zpool detach zeepool c2t1d0However, this operation is refused if there are no other valid replicas of the data. For example:# zpool detach newpool c1t2d0 cannot detach c1t2d0: only applicable to mirror and replacing vdevs ZFS allows individual devices to be taken offline or brought online. When hardware is unreliable or not functioning properly, ZFS continues to read or write data to the device, assuming the condition is only temporary. If the condition is not temporary, it is possible to instruct ZFS to ignore the device by bringing it offline. ZFS does not send any requests to an offlined device.Devices do not need to be taken offline in order to replace them. You can use the offline command when you need to temporarily disconnect storage. For example, if you need to physically disconnect an array from one set of Fibre Channel switches and connect the array to a different set, you could take the LUNs offline from the array that was used in ZFS storage pools. After the array was reconnected and operational on the new set of switches, you could then bring the same LUNs online. Data that had been added to the storage pools while the LUNs were offline would resilver to the LUNs after they were brought back online.This scenario is possible assuming that the systems in question see the storage once it is attached to the new switches, possibly through different controllers than before, and your pools are set up as RAID-Z or mirrored configurations.You can take a device offline by using the zpool offline command. The device can be specified by path or by short name, if the device is a disk. For example:# zpool offline tank c1t0d0 bringing device c1t0d0 offlineKeep the following points in mind when taking a device offline:You cannot take a pool offline to the point where it becomes faulted. For example, you cannot take offline two devices out of a RAID-Z configuration, nor can you take offline a top-level virtual device.# zpool offline tank c1t0d0 cannot offline c1t0d0: no valid replicas By default, the offline state is persistent. The device remains offline when the system is rebooted. To temporarily take a device offline, use the zpool offline t option. For example:# zpool offline -t tank c1t0d0 bringing device 'c1t0d0' offlineWhen the system is rebooted, this device is automatically returned to the ONLINE state. When a device is taken offline, it is not detached from the storage pool. If you attempt to use the offlined device in another pool, even after the original pool is destroyed, you will see a message similar to the following:device is part of exported or potentially active ZFS pool. Please see zpool(1M)If you want to use the offlined device in another storage pool after destroying the original storage pool, first bring the device back online, then destroy the original storage pool.Another way to use a device from another storage pool if you want to keep the original storage pool is to replace the existing device in the original storage pool with another comparable device. For information about replacing devices, see Replacing Devices in a Storage Pool. Offlined devices show up in the OFFLINE state when you query pool status. For information about querying pool status, see Querying ZFS Storage Pool Status.For more information on device health, see Determining the Health Status of ZFS Storage Pools. Once a device is taken offline, it can be restored by using the zpool online command:# zpool online tank c1t0d0 bringing device c1t0d0 onlineWhen a device is brought online, any data that has been written to the pool is resynchronized to the newly available device. Note that you cannot use device onlining to replace a disk. If you offline a device, replace the drive, and try to bring it online, it remains in the faulted state.If you attempt to online a faulted device, a message similar to the following is displayed from fmd:# zpool online tank c1t0d0 Bringing device c1t0d0 online # SUNW-MSG-ID: ZFS-8000-D3, TYPE: Fault, VER: 1, SEVERITY: Major EVENT-TIME: Thu Aug 31 11:13:59 MDT 2006 PLATFORM: SUNW,Ultra-60, CSN: -, HOSTNAME: neo SOURCE: zfs-diagnosis, REV: 1.0 EVENT-ID: e11d8245-d76a-e152-80c6-e63763ed7e4f DESC: A ZFS device failed. Refer to http://sun.com/msg/ZFS-8000-D3 for more information. AUTO-RESPONSE: No automated response will occur. IMPACT: Fault tolerance of the pool may be compromised. REC-ACTION: Run 'zpool status -x' and replace the bad device.For more information on replacing a faulted device, see Repairing a Missing Device. If a device is taken offline due to a failure that causes errors to be listed in the zpool status output, you can clear the error counts with the zpool clear command. If specified with no arguments, this command clears all device errors within the pool. For example:# zpool clear tankIf one or more devices are specified, this command only clear errors associated with the specified devices. For example:# zpool clear tank c1t0d0For more information on clearing zpool errors, see Clearing Transient Errors. You can replace a device in a storage pool by using the zpool replace command.If you are physically replacing a device with another device in the same location in a redundant pool, then you only need identify the replaced device. ZFS recognizes that it is a different disk in the same location. For example, to replace a failed disk (c1t1d0) by removing the disk and replacing it in the same location, use the syntax similar to the following:# zpool replace tank c1t1d0If you are replacing a device in a non-redundant storage pool that contains only one device, you will need to specify both devices. For example:# zpool replace tank c1t1d0 c1t2d0Keep the following considerations in mind when replacing devices in a ZFS storage pool:The replacement device must be greater than or equal to the minimum size of all the devices in a mirrored or RAID-Z configuration. If the replacement device is larger, the pool capacity is increased when the replacement is complete. Currently, you must export and import the pool to see the expanded capacity. For example:# zpool list tank NAME SIZE USED AVAIL CAP HEALTH ALTROOT tank 16.8G 94K 16.7G 0% ONLINE - # zpool replace tank c0t0d0 c0t4d0 # zpool list tank NAME SIZE USED AVAIL CAP HEALTH ALTROOT tank 16.8G 112K 16.7G 0% ONLINE - # zpool export tank # zpool import tank # zpool list tank NAME SIZE USED AVAIL CAP HEALTH ALTROOT tank 33.9G 114K 33.9G 0% ONLINE -For more information about exporting and importing pools, see Migrating ZFS Storage Pools. Currently, you must also perform the export and import steps when growing the size of an existing LUN that is part of a storage pool to see the expanded capacity. Replacing many disks in a large pool is time consuming due to resilvering the data onto the new disks. In addition, you might consider running the zpool scrub command between disk replacements to ensure that the replacement devices are operational and the data is written correctly. For more information about replacing devices, see Repairing a Missing Device and Repairing a Damaged Device. The hot spares feature enables you to identify disks that could be used to replace a failed or faulted device in one or more storage pools. Designating a device as a hot spare means that the device is not an active device in a pool, but if an active device in the pool fails, the hot spare automatically replaces the failed device.Devices can be designated as hot spares in the following ways:When the pool is created with the zpool create command After the pool is created with the zpool add command Hot spare devices can be shared between multiple pools Designate devices as hot spares when the pool is created. For example:# zpool create zeepool mirror c1t1d0 c2t1d0 spare c1t2d0 c2t2d0 # zpool status zeepool pool: zeepool state: ONLINE scrub: none requested config: NAME STATE READ WRITE CKSUM zeepool ONLINE 0 0 0 mirror ONLINE 0 0 0 c1t1d0 ONLINE 0 0 0 c2t1d0 ONLINE 0 0 0 spares c1t2d0 AVAIL c2t2d0 AVAIL Designate hot spares by adding them to a pool after the pool is created. For example:# zpool add -f zeepool spare c1t3d0 c2t3d0 # zpool status zeepool pool: zeepool state: ONLINE scrub: none requested config: NAME STATE READ WRITE CKSUM zeepool ONLINE 0 0 0 mirror ONLINE 0 0 0 c1t1d0 ONLINE 0 0 0 c2t1d0 ONLINE 0 0 0 spares c1t3d0 AVAIL c2t3d0 AVAIL Multiple pools can share devices that are designated as hot spares. For example:# zpool create zeepool mirror c1t1d0 c2t1d0 spare c1t2d0 c2t2d0 # zpool create tank raidz c3t1d0 c4t1d0 spare c1t2d0 c2t2d0Hot spares can be removed from a storage pool by using the zpool remove command. For example:# zpool remove zeepool c1t2d0 # zpool status zeepool pool: zeepool state: ONLINE scrub: none requested config: NAME STATE READ WRITE CKSUM zeepool ONLINE 0 0 0 mirror ONLINE 0 0 0 c1t1d0 ONLINE 0 0 0 c2t1d0 ONLINE 0 0 0 spares c1t3d0 AVAILA hot spare cannot be removed if it is currently used by the storage pool.Keep the following points in mind when using ZFS hot spares:Currently, the zpool remove command can only be used to remove hot spares. Add a disk as a spare that is equal to or larger than the size of the largest disk in the pool. Adding a smaller disk as a spare to a pool is allowed. However, when the smaller spare disk is activated, either automatically or with the zpool replace command, the operation fails with an error similar to the following:cannot replace disk3 with disk4: device is too small Hot spares are activated in the following ways:Manually replacement – Replace a failed device in a storage pool with a hot spare by using the zpool replace command. Automatic replacement – When a fault is received, an FMA agent examines the pool to see if it has any available hot spares. If so, it replaces the faulted device with an available spare.If a hot spare that is currently in use fails, the agent detaches the spare and thereby cancels the replacement. The agent then attempts to replace the device with another hot spare, if one is available. This feature is currently limited by the fact that the ZFS diagnosis engine only emits faults when a device disappears from the system. Currently, no automated response is available to bring the original device back online. You must explicitly take one of the actions described in the example below. A future enhancement will allow ZFS to subscribe to hotplug events and automatically replace the affected device when it is replaced on the system. Manually replace a device with a hot spare by using the zpool replace command. For example:# zpool replace zeepool c2t1d0 c2t3d0 # zpool status zeepool pool: zeepool state: ONLINE scrub: resilver completed with 0 errors on Fri Jun 2 13:44:40 2006 config: NAME STATE READ WRITE CKSUM zeepool ONLINE 0 0 0 mirror ONLINE 0 0 0 c1t2d0 ONLINE 0 0 0 spare ONLINE 0 0 0 c2t1d0 ONLINE 0 0 0 c2t3d0 ONLINE 0 0 0 spares c1t3d0 AVAIL c2t3d0 INUSE currently in use errors: No known data errorsA faulted device is automatically replaced if a hot spare is available. For example:# zpool status -x pool: zeepool state: DEGRADED status: One or more devices could not be opened. Sufficient replicas exist for the pool to continue functioning in a degraded state. action: Attach the missing device and online it using 'zpool online'. see: http://www.sun.com/msg/ZFS-8000-D3 scrub: resilver completed with 0 errors on Fri Jun 2 13:56:49 2006 config: NAME STATE READ WRITE CKSUM zeepool DEGRADED 0 0 0 mirror DEGRADED 0 0 0 c1t2d0 ONLINE 0 0 0 spare DEGRADED 0 0 0 c2t1d0 UNAVAIL 0 0 0 cannot open c2t3d0 ONLINE 0 0 0 spares c1t3d0 AVAIL c2t3d0 INUSE currently in use errors: No known data errorsCurrently, three ways to deactivate hot spares are available:Canceling the hot spare by removing it from the storage pool Replacing the original device with a hot spare Permanently swapping in the hot spare After the faulted device is replaced, use the zpool detach command to return the hot spare back to the spare set. For example:# zpool detach zeepool c2t3d0 # zpool status zeepool pool: zeepool state: ONLINE scrub: resilver completed with 0 errors on Fri Jun 2 13:58:35 2006 config: NAME STATE READ WRITE CKSUM zeepool ONLINE 0 0 0 mirror ONLINE 0 0 0 c1t2d0 ONLINE 0 0 0 c2t1d0 ONLINE 0 0 0 spares c1t3d0 AVAIL c2t3d0 AVAIL errors: No known data errors You can use the zpool get command to display pool property information. For example:# zpool get all tank2 NAME PROPERTY VALUE SOURCE tank2 size 33.8G - tank2 used 158K - tank2 available 33.7G - tank2 capacity 0% - tank2 altroot - default tank2 health ONLINE - tank2 guid 8032621780930948264 - tank2 version 8 default tank2 bootfs - default tank2 delegation on default tank2 autoreplace off default tank2 temporary off default tank2 failmode wait defaultStorage pool properties can be set with the zpool set command. For example:# zpool set autoreplace=on tank # zpool get autoreplace tank NAME PROPERTY VALUE SOURCE tank autoreplace on defaultProperty Name Type Default Value Description altroot String off Identifies an alternate root directory. If set, this directory is prepended to any mount points within the pool. This property can be used when examining an unknown pool, if the mount points cannot be trusted, or in an alternate boot environment, where the typical paths are not valid. Setting this property implies that the temporary property is also set. available Number N/A Read-only value that identifies the amount of storage that is available within the pool.This property can also be referred to by its shortened column name, avail. autoreplace Boolean off Controls automatic device replacement. If set to off, device replacement must be initiated by the administrator by using the zpool replace command. If set to on, any new device, found in the same physical location as a device that previously belonged to the pool, is automatically formatted and replaced. The default behavior is off. This property can also be referred to by its shortened column name, replace. bootfs Boolean N/A Identifies the default bootable dataset for the root pool. This property is expected to be set mainly by the installation and upgrade programs. capacity Number N/A Read-only value that identifies the percentage of pool space used.This property can also be referred to by its shortened column name, cap. delegation Boolean on Controls whether a non-privileged user can be granted access permissions that are defined for the dataset. For more information, see Chapter 8, ZFS Delegated Administration. guid String N/A Read-only property that identifies the unique identifier for the pool. health String N/A Read-only property that identifies the current health of the pool, as either ONLINE, DEGRADED, FAULTED, OFFLINE, REMOVED, or UNAVAIL. size Number N/A Read-only property that identifies the total size of the storage pool. used Number N/A Read-only property that identifies the amount of storage space used within the pool. temporary Boolean off Controls whether the pool is available temporarily. By default, all pools are persistent, and are automatically opened when the system is rebooted. Setting this property to on causes the pool to exist only while the system is up. If the system is rebooted, the pool has to be manually imported by using the zpool import command. Setting this property is helpful when using pools on removable media, where the devices might not be present when the system reboots.This property can also be referred to by its shortened column name, temp. version Number N/A Identifies the current on-disk version of the pool. The value of this property can be increased, but never decreased. The preferred method of updating pools is with the zpool upgrade command, although this property can be used when a specific version is needed for backwards compatibility. This property can be set to any number between 1 and the current version reported by the zpool upgrade v command. The current value is an alias for the latest supported version. The zpool list command provides a number of ways to request information regarding pool status. The information available generally falls into three categories: basic usage information, I/O statistics, and health status. All three types of storage pool information are covered in this section.Displaying Basic ZFS Storage Pool Information Viewing ZFS Storage Pool I/O Statistics Determining the Health Status of ZFS Storage Pools You can use the zpool list command to display basic information about pools.With no arguments, the command displays all the fields for all pools on the system. For example:# zpool list NAME SIZE USED AVAIL CAP HEALTH ALTROOT tank 80.0G 22.3G 47.7G 28% ONLINE - dozer 1.2T 384G 816G 32% ONLINE -This output displays the following information:NAMEThe name of the pool. SIZEThe total size of the pool, equal to the sum of the size of all top-level virtual devices. USEDThe amount of space allocated by all datasets and internal metadata. Note that this amount is different from the amount of space as reported at the file system level.For more information about determining available file system space, see ZFS Space Accounting. AVAILABLEThe amount of unallocated space in the pool. CAPACITY (CAP)The amount of space used, expressed as a percentage of total space. HEALTHThe current health status of the pool.For more information about pool health, see Determining the Health Status of ZFS Storage Pools. ALTROOTThe alternate root of the pool, if any.For more information about alternate root pools, see Using ZFS Alternate Root Pools. You can also gather statistics for a specific pool by specifying the pool name. For example:# zpool list tank NAME SIZE USED AVAIL CAP HEALTH ALTROOT tank 80.0G 22.3G 47.7G 28% ONLINE - Specific statistics can be requested by using the o option. This option allows for custom reports or a quick way to list pertinent information. For example, to list only the name and size of each pool, you use the following syntax:# zpool list -o name,size NAME SIZE tank 80.0G dozer 1.2TThe column names correspond to the properties that are listed in Listing Information About All Storage Pools. The default output for the zpool list command is designed for readability, and is not easy to use as part of a shell script. To aid programmatic uses of the command, the H option can be used to suppress the column headings and separate fields by tabs, rather than by spaces. For example, to request a simple list of all pool names on the system:# zpool list -Ho name tank dozerHere is another example:# zpool list -H -o name,size tank 80.0G dozer 1.2T To request I/O statistics for a pool or specific virtual devices, use the zpool iostat command. Similar to the iostat command, this command can display a static snapshot of all I/O activity so far, as well as updated statistics for every specified interval. The following statistics are reported:USED CAPACITYThe amount of data currently stored in the pool or device. This figure differs from the amount of space available to actual file systems by a small amount due to internal implementation details.For more information about the difference between pool space and dataset space, see ZFS Space Accounting. AVAILABLE CAPACITYThe amount of space available in the pool or device. As with the used statistic, this amount differs from the amount of space available to datasets by a small margin. READ OPERATIONSThe number of read I/O operations sent to the pool or device, including metadata requests. WRITE OPERATIONSThe number of write I/O operations sent to the pool or device. READ BANDWIDTHThe bandwidth of all read operations (including metadata), expressed as units per second. WRITE BANDWIDTHThe bandwidth of all write operations, expressed as units per second. With no options, the zpool iostat command displays the accumulated statistics since boot for all pools on the system. For example:# zpool iostat capacity operations bandwidth pool used avail read write read write ---------- ----- ----- ----- ----- ----- ----- tank 100G 20.0G 1.2M 102K 1.2M 3.45K dozer 12.3G 67.7G 132K 15.2K 32.1K 1.20KBecause these statistics are cumulative since boot, bandwidth might appear low if the pool is relatively idle. You can request a more accurate view of current bandwidth usage by specifying an interval. For example:# zpool iostat tank 2 capacity operations bandwidth pool used avail read write read write ---------- ----- ----- ----- ----- ----- ----- tank 100G 20.0G 1.2M 102K 1.2M 3.45K tank 100G 20.0G 134 0 1.34K 0 tank 100G 20.0G 94 342 1.06K 4.1MIn this example, the command displays usage statistics only for the pool tank every two seconds until you type Ctrl-C. Alternately, you can specify an additional count parameter, which causes the command to terminate after the specified number of iterations. For example, zpool iostat 2 3 would print a summary every two seconds for three iterations, for a total of six seconds. If there is a single pool, then the statistics are displayed on consecutive lines. If more than one pool exists, then an additional dashed line delineates each iteration to provide visual separation. In addition to pool-wide I/O statistics, the zpool iostat command can display statistics for specific virtual devices. This command can be used to identify abnormally slow devices, or simply to observe the distribution of I/O generated by ZFS. To request the complete virtual device layout as well as all I/O statistics, use the zpool iostat -v command. For example:# zpool iostat -v capacity operations bandwidth tank used avail read write read write ---------- ----- ----- ----- ----- ----- ----- mirror 20.4G 59.6G 0 22 0 6.00K c1t0d0 - - 1 295 11.2K 148K c1t1d0 - - 1 299 11.2K 148K ---------- ----- ----- ----- ----- ----- ----- total 24.5K 149M 0 22 0 6.00KNote two important things when viewing I/O statistics on a virtual device basis.First, space usage is only available for top-level virtual devices. The way in which space is allocated among mirror and RAID-Z virtual devices is particular to the implementation and not easily expressed as a single number. Second, the numbers might not add up exactly as you would expect them to. In particular, operations across RAID-Z and mirrored devices will not be exactly equal. This difference is particularly noticeable immediately after a pool is created, as a significant amount of I/O is done directly to the disks as part of pool creation that is not accounted for at the mirror level. Over time, these numbers should gradually equalize, although broken, unresponsive, or offlined devices can affect this symmetry as well. You can use the same set of options (interval and count) when examining virtual device statistics. ZFS provides an integrated method of examining pool and device health. The health of a pool is determined from the state of all its devices. This state information is displayed by using the zpool status command. In addition, potential pool and device failures are reported by fmd and are displayed on the system console and the /var/adm/messages file. This section describes how to determine pool and device health. This chapter does not document how to repair or recover from unhealthy pools. For more information on troubleshooting and data recovery, see Chapter 10, ZFS Troubleshooting and Data Recovery.Each device can fall into one of the following states:ONLINEThe device is in normal working order. While some transient errors might still occur, the device is otherwise in working order. DEGRADEDThe virtual device has experienced failure but is still able to function. This state is most common when a mirror or RAID-Z device has lost one or more constituent devices. The fault tolerance of the pool might be compromised, as a subsequent fault in another device might be unrecoverable. FAULTEDThe virtual device is completely inaccessible. This status typically indicates total failure of the device, such that ZFS is incapable of sending or receiving data from it. If a top-level virtual device is in this state, then the pool is completely inaccessible. OFFLINEThe virtual device has been explicitly taken offline by the administrator. UNAVAILABLEThe device or virtual device cannot be opened. In some cases, pools with UNAVAILABLE devices appear in DEGRADED mode. If a top-level virtual device is unavailable, then nothing in the pool can be accessed. REMOVEDThe device was physically removed while the system was running. Device removal detection is hardware-dependent and might not be supported on all platforms. The health of a pool is determined from the health of all its top-level virtual devices. If all virtual devices are ONLINE, then the pool is also ONLINE. If any one of the virtual devices is DEGRADED or UNAVAILABLE, then the pool is also DEGRADED. If a top-level virtual device is FAULTED or OFFLINE, then the pool is also FAULTED. A pool in the faulted state is completely inaccessible. No data can be recovered until the necessary devices are attached or repaired. A pool in the degraded state continues to run, but you might not achieve the same level of data redundancy or data throughput than if the pool were online.The simplest way to request a quick overview of pool health status is to use the zpool status command:# zpool status -x all pools are healthySpecific pools can be examined by specifying a pool name to the command. Any pool that is not in the ONLINE state should be investigated for potential problems, as described in the next section. You can request a more detailed health summary by using the v option. For example:# zpool status -v tank pool: tank state: DEGRADED status: One or more devices could not be opened. Sufficient replicas exist for the pool to continue functioning in a degraded state. action: Attach the missing device and online it using 'zpool online'. see: http://www.sun.com/msg/ZFS-8000-2Q scrub: none requested config: NAME STATE READ WRITE CKSUM tank DEGRADED 0 0 0 mirror DEGRADED 0 0 0 c1t0d0 FAULTED 0 0 0 cannot open c1t1d0 ONLINE 0 0 0 errors: No known data errorsThis output displays a complete description of why the pool is in its current state, including a readable description of the problem and a link to a knowledge article for more information. Each knowledge article provides up-to-date information on the best way to recover from your current problem. Using the detailed configuration information, you should be able to determine which device is damaged and how to repair the pool.In the above example, the faulted device should be replaced. After the device is replaced, use the zpool online command to bring the device back online. For example:# zpool online tank c1t0d0 Bringing device c1t0d0 online # zpool status -x all pools are healthyIf a pool has an offlined device, the command output identifies the problem pool. For example:# zpool status -x pool: tank state: DEGRADED status: One or more devices has been taken offline by the adminstrator. Sufficient replicas exist for the pool to continue functioning in a degraded state. action: Online the device using 'zpool online' or replace the device with 'zpool replace'. scrub: none requested config: NAME STATE READ WRITE CKSUM tank DEGRADED 0 0 0 mirror DEGRADED 0 0 0 c1t0d0 ONLINE 0 0 0 c1t1d0 OFFLINE 0 0 0 errors: No known data errorsThe READ and WRITE columns provides a count of I/O errors seen on the device, while the CKSUM column provides a count of uncorrectable checksum errors that occurred on the device. Both of these error counts likely indicate potential device failure, and some corrective action is needed. If non-zero errors are reported for a top-level virtual device, portions of your data might have become inaccessible. The errors count identifies any known data errors.In the example output above, the offlined device is not causing data errors.For more information about diagnosing and repairing faulted pools and data, see Chapter 10, ZFS Troubleshooting and Data Recovery. Occasionally, you might need to move a storage pool between machines. To do so, the storage devices must be disconnected from the original machine and reconnected to the destination machine. This task can be accomplished by physically recabling the devices, or by using multiported devices such as the devices on a SAN. ZFS enables you to export the pool from one machine and import it on the destination machine, even if the machines are of different endianness. For information about replicating or migrating file systems between different storage pools, which might reside on different machines, see Saving and Restoring ZFS Data.Preparing for ZFS Storage Pool Migration Exporting a ZFS Storage Pool Determining Available Storage Pools to Import Finding ZFS Storage Pools From Alternate Directories Importing ZFS Storage Pools Recovering Destroyed ZFS Storage Pools Upgrading ZFS Storage Pools Storage pools should be explicitly exported to indicate that they are ready to be migrated. This operation flushes any unwritten data to disk, writes data to the disk indicating that the export was done, and removes all knowledge of the pool from the system.If you do not explicitly export the pool, but instead remove the disks manually, you can still import the resulting pool on another system. However, you might lose the last few seconds of data transactions, and the pool will appear faulted on the original machine because the devices are no longer present. By default, the destination machine refuses to import a pool that has not been explicitly exported. This condition is necessary to prevent accidentally importing an active pool that consists of network attached storage that is still in use on another system. To export a pool, use the zpool export command. For example:# zpool export tankOnce this command is executed, the pool tank is no longer visible on the system. The command attempts to unmount any mounted file systems within the pool before continuing. If any of the file systems fail to unmount, you can forcefully unmount them by using the f option. For example:# zpool export tank cannot unmount '/export/home/eschrock': Device busy # zpool export -f tankIf devices are unavailable at the time of export, the disks cannot be specified as cleanly exported. If one of these devices is later attached to a system without any of the working devices, it appears as “potentially active.” If ZFS volumes are in use in the pool, the pool cannot be exported, even with the f option. To export a pool with an ZFS volume, first make sure that all consumers of the volume are no longer active.For more information about ZFS volumes, see ZFS Volumes. Once the pool has been removed from the system (either through export or by forcefully removing the devices), attach the devices to the target system. Although ZFS can handle some situations in which only a portion of the devices is available, all devices within the pool must be moved between the systems. The devices do not necessarily have to be attached under the same device name. ZFS detects any moved or renamed devices, and adjusts the configuration appropriately. To discover available pools, run the zpool import command with no options. For example:# zpool import pool: tank id: 3778921145927357706 state: ONLINE action: The pool can be imported using its name or numeric identifier. config: tank ONLINE mirror ONLINE c1t0d0 ONLINE c1t1d0 ONLINEIn this example, the pool tank is available to be imported on the target system. Each pool is identified by a name as well as a unique numeric identifier. If multiple pools available to import have the same name, you can use the numeric identifier to distinguish between them.Similar to the zpool status command, the zpool import command refers to a knowledge article available on the web with the most up-to-date information regarding repair procedures for a problem that is preventing a pool from being imported. In this case, the user can force the pool to be imported. However, importing a pool that is currently in use by another system over a storage network can result in data corruption and panics as both systems attempt to write to the same storage. If some devices in the pool are not available but enough redundancy is available to have a usable pool, the pool appears in the DEGRADED state. For example:# zpool import pool: tank id: 3778921145927357706 state: DEGRADED status: One or more devices are missing from the system. action: The pool can be imported despite missing or damaged devices. The fault tolerance of the pool may be compromised if imported. see: http://www.sun.com/msg/ZFS-8000-2Q config: tank DEGRADED mirror DEGRADED c1t0d0 UNAVAIL cannot open c1t1d0 ONLINEIn this example, the first disk is damaged or missing, though you can still import the pool because the mirrored data is still accessible. If too many faulted or missing devices are present, the pool cannot be imported. For example:# zpool import pool: dozer id: 12090808386336829175 state: FAULTED action: The pool cannot be imported. Attach the missing devices and try again. see: http://www.sun.com/msg/ZFS-8000-6X config: raidz FAULTED c1t0d0 ONLINE c1t1d0 FAULTED c1t2d0 ONLINE c1t3d0 FAULTEDIn this example, two disks are missing from a RAID-Z virtual device, which means that sufficient redundant data is not available to reconstruct the pool. In some cases, not enough devices are present to determine the complete configuration. In this case, ZFS doesn't know what other devices were part of the pool, though ZFS does report as much information as possible about the situation. For example:# zpool import pool: dozer id: 12090808386336829175 state: FAULTED status: One or more devices are missing from the system. action: The pool cannot be imported. Attach the missing devices and try again. see: http://www.sun.com/msg/ZFS-8000-6X config: dozer FAULTED missing device raidz ONLINE c1t0d0 ONLINE c1t1d0 ONLINE c1t2d0 ONLINE c1t3d0 ONLINE Additional devices are known to be part of this pool, though their exact configuration cannot be determined. By default, the zpool import command only searches devices within the /dev/dsk directory. If devices exist in another directory, or you are using pools backed by files, you must use the d option to search different directories. For example:# zpool create dozer mirror /file/a /file/b # zpool export dozer # zpool import -d /file pool: dozer id: 10952414725867935582 state: ONLINE action: The pool can be imported using its name or numeric identifier. config: dozer ONLINE mirror ONLINE /file/a ONLINE /file/b ONLINE # zpool import -d /file dozerIf devices exist in multiple directories, you can specify multiple d options. Once a pool has been identified for import, you can import it by specifying the name of the pool or its numeric identifier as an argument to the zpool import command. For example:# zpool import tankIf multiple available pools have the same name, you can specify which pool to import using the numeric identifier. For example:# zpool import pool: dozer id: 2704475622193776801 state: ONLINE action: The pool can be imported using its name or numeric identifier. config: dozer ONLINE c1t9d0 ONLINE pool: dozer id: 6223921996155991199 state: ONLINE action: The pool can be imported using its name or numeric identifier. config: dozer ONLINE c1t8d0 ONLINE # zpool import dozer cannot import 'dozer': more than one matching pool import by numeric ID instead # zpool import 6223921996155991199If the pool name conflicts with an existing pool name, you can import the pool under a different name. For example:# zpool import dozer zeepoolThis command imports the exported pool dozer using the new name zeepool. If the pool was not cleanly exported, ZFS requires the f flag to prevent users from accidentally importing a pool that is still in use on another system. For example:# zpool import dozer cannot import 'dozer': pool may be in use on another system use '-f' to import anyway # zpool import -f dozerPools can also be imported under an alternate root by using the R option. For more information on alternate root pools, see Using ZFS Alternate Root Pools. You can use the zpool import D command to recover a storage pool that has been destroyed. For example:# zpool destroy tank # zpool import -D pool: tank id: 3778921145927357706 state: ONLINE (DESTROYED) action: The pool can be imported using its name or numeric identifier. The pool was destroyed, but can be imported using the '-Df' flags. config: tank ONLINE mirror ONLINE c1t0d0 ONLINE c1t1d0 ONLINEIn the above zpool import output, you can identify this pool as the destroyed pool because of the following state information:state: ONLINE (DESTROYED)To recover the destroyed pool, issue the zpool import D command again with the pool to be recovered and the f option. For example:# zpool import -Df tank # zpool status tank pool: tank state: ONLINE scrub: none requested config: NAME STATE READ WRITE CKSUM tank ONLINE 0 0 0 mirror ONLINE 0 0 0 c1t0d0 ONLINE 0 0 0 c1t1d0 ONLINE 0 0 0 errors: No known data errorsIf one of the devices in the destroyed pool is faulted or unavailable, you might be able to recover the destroyed pool anyway. In this scenario, import the degraded pool and then attempt to fix the device failure. For example:# zpool destroy dozer # zpool import -D pool: dozer id: state: DEGRADED (DESTROYED) status: One or more devices are missing from the system. action: The pool can be imported despite missing or damaged devices. The fault tolerance of the pool may be compromised if imported. The pool was destroyed, but can be imported using the '-Df' flags. see: http://www.sun.com/msg/ZFS-8000-2Q config: dozer DEGRADED raidz ONLINE c1t0d0 ONLINE c1t1d0 ONLINE c1t2d0 UNAVAIL cannot open c1t3d0 ONLINE # zpool import -Df dozer # zpool status -x pool: dozer state: DEGRADED status: One or more devices could not be opened. Sufficient replicas exist for the pool to continue functioning in a degraded state. action: Attach the missing device and online it using 'zpool online'. see: http://www.sun.com/msg/ZFS-8000-D3 scrub: resilver completed with 0 errors on Fri Mar 17 16:11:35 2006 config: NAME STATE READ WRITE CKSUM dozer DEGRADED 0 0 0 raidz ONLINE 0 0 0 c1t0d0 ONLINE 0 0 0 c1t1d0 ONLINE 0 0 0 c1t2d0 UNAVAIL 0 0 0 cannot open c1t3d0 ONLINE 0 0 0 errors: No known data errors # zpool online dozer c1t2d0 Bringing device c1t2d0 online # zpool status -x all pools are healthy If you have ZFS storage pools from a previous Solaris release, such as the Solaris 10 6/06 release, you can upgrade your pools with the zpool upgrade command to take advantage of the pool features in the Solaris 10 11/06 release. In addition, the zpool status command has been modified to notify you when your pools are running older versions. For example:# zpool status pool: test state: ONLINE status: The pool is formatted using an older on-disk format. The pool can still be used, but some features are unavailable. action: Upgrade the pool using 'zpool upgrade'. Once this is done, the pool will no longer be accessible on older software versions. scrub: none requested config: NAME STATE READ WRITE CKSUM test ONLINE 0 0 0 c1t27d0 ONLINE 0 0 0 errors: No known data errorsYou can use the following syntax to identify additional information about a particular version and supported releases.# zpool upgrade -v This system is currently running ZFS version 3. The following versions are supported: VER DESCRIPTION --- -------------------------------------------------------- 1 Initial ZFS version 2 Ditto blocks (replicated metadata) 3 Hot spares and double parity RAID-Z For more information on a particular version, including supported releases, see: http://www.opensolaris.org/os/community/zfs/version/N Where 'N' is the version number.Then, you can run the zpool upgrade command to upgrade your pools. For example:# zpool upgrade -aIf you upgrade your pools to the latest version, they will not be accessible on systems that run older ZFS versions.