AIX MPIO and SDDPCM multipath options with DS3000/DS4000/DS5000

Where do I find information about the AIX MPIO and SDDPCM multipath options with DS3000/DS4000/DS5000?

References for AIX MPIO and SDDPCM multipath options with DS3000/DS4000/DS5000

While the RDAC driver is being deprecated with AIX, it is still the default driver on AIX 5.2 and 5.3 for DS4000 devices. With AIX 6.1, the default driver is MPIO for DS4000/DS5000 devices. All future DS4000/DS5000 models will only be supported with MPIO. The DS3K products are always configured as MPIO devices. The MPIO support for these devices removes the limitation of connecting only one adapter to each DS3/4K controller port and allows more than two paths to be configured to the device. This simplifies zoning and improves performance.

DS3000: native MPIO only
DS4000: RDAC, native MPIO, MPIO w/SDDPCM
DS5000: native MPIO, MPIO w/SDDPCM

For more details about prerequisites and supported OS and driver levels take a look at the particular interoperability matrices:

DS3000 family products interoperability matrix
http://www.ibm.com/systems/storage/disk/ds3000/pdf/interop.pdf

DS4000 family products interoperability matrix
http://www.ibm.com/systems/resources/systems_storage_disk_ds4000_pdf_interop-matrix.pdf

DS5000 family products interoperability matrix
http://www.ibm.com/systems/resources/systems_storage_disk_ds5000_interop-matrix.pdf

AIX MPIO is supported on AIX starting from AIX 5.2 TL10, AIX 5.3 TL6 and AIX 6.1 SP2 though various PTFs and fixes may be required (see interoperability matrix and AIX MPIO documentation below). It is supported with DS4000 firmware 6.60.xx.xx and higher on DS4800, DS4700, DS4200, DS4500 and DS4300. MPIO is the strategic driver for all current DS4000 platforms in AIX 6.1 and up - except for DS4400 which is RDAC only. RDAC is the default driver on all levels of AIX 5.2 and 5.3.

In AIX Version 6.1, the DS4K products are configured as Multipath I/O (MPIO) devices by default. A DS4K product using the FCPARRAY (RDAC) driver may be migrated to the MPIO driver by using the instructions found in the documents below, or all DS4K devices may be migrated by uninstalling the devices.fcp.disk.array.rte package and then running the cfgmgr command or rebooting.

For more information on how to manage MPIO on AIX with DS4000 subsystems, the required DS4000 NVSRAM settings and the new manage_disk_driver and mpio_get_config commands, take a look at these documents

Procedures for AIX 6.1 and DS4000 Interoperability
http://www.ibm.com/systems/resources/systems_storage_disk_ds4000_pdf_aix.pdf

AIX61 TL02 Release Notes (p.19ff)
http://publib.boulder.ibm.com/infocenter/systems/scope/aix/topic/com.ibm.aix.resources/RELNOTES/SC23662903.pdf

The manage_disk_drivers command displays a list of DS4K storage models and the driver that manages each model. All disks of a storage model must be managed by the same driver. If you specify the -c flag, the command changes the driver selection to an alternate supported driver for the storage model.

# manage_disk_drivers
1: DS4100: currently MPIO; supported: RDAC/fcparray, MPIO
2: DS4300: currently MPIO; supported: RDAC/fcparray, MPIO
3: DS4500: currently MPIO; supported: RDAC/fcparray, MPIO
4: DS4700/DS4200: currently MPIO; supported: RDAC/fcparray, MPIO
5: DS4800: currently MPIO; supported: RDAC/fcparray, MPIO

According to these documents (and the DS4k/DS5k interoperability matrix) the following NVSRAM changes have to be applied to the DS4000 for MPIO support:

set controller [a] HostNVSRAMBYTE [0x06,0x27] = 0;
set controller HostNVSRAMBYTE [0x06,0x27] = 0;

Please note, that for the NVSRAM changes to take effect a reboot of the DS4K controller is required!

The default settings on a DS4800 with firmware 07.36.14.00 and NVSRAM version N1815D48R1036V12 are

show controller [a] HostNVSRAMBYTE [0x06,0x27];
show controller HostNVSRAMBYTE [0x06,0x27];
Executing script...
Controller "a" Host Type Index 6 NVSRAM offset 0x27 = 0x8. (Default value is '8')
Controller "b" Host Type Index 6 NVSRAM offset 0x27 = 0x8.

Please check and note the default settings first by running the show controller command before actually applying the NVSRAM changes!

These settings change bit 3 in NVSRAM for the host type 'AIX' to 0 for MPIO support. Bit 3 of the host type has the following meaning:

Bit 3 = 0 A header is returned, along with a reservation key set to zero. This is not is in accordance with the current T10
standards for the persistent reservations command set, but preserves compatibility with the legacy implementation.
Bit 3 = 1 A header is returned (but no reservation key). This is in accordance with the current T10 standards for the persistent reservations command set.

Examples for SAN attachment and zoning with AIX native MPIO and more than two paths can be found in the

IBM System Storage DS3000 - Storage Manager v2 Installation and Support Guide for IBM AIX and Linux on POWER
http://www.ibm.com/systems/support/supportsite.wss/docdisplay?lndocid=MIGR-5073850&brandind=5000028

in chapter 3, Preparing for a SAN attachment (p25ff):

The following examples are a small subset of the connectivity configurations that are supported using the DS3400. The best practice is to use 2 to 4 adapters providing 4 to 8 paths.

Example 1 Create one zone that contains two Fibre Channel HBA ports, one Fibre Channel port from controller A and one Fibre Channel port from controller B of the DS3400. This configuration provides four paths to hdisks. Two of the paths that are associated with the preferred storage controller will service I/Os when the DS3400 is optimal. The other two paths are used if the preferred controller is not accessible by the host. Note: The hdisk attribute “algorithm” should be set to round_robin. If the attribute is set to round_robin, the hdisk attribute “reserve_policy” must be set to no_reserve, pr_exclusive, or pr_shared.

Example 2 Create separate zones for the connection between the HBA port and one port of the DS3400 controller. Two zones are required for a dual-controller storage configuration. One zone contains an HBA port and controller port from controller A. The other zone contains a different HBA port and controller port from controller B. This configuration provides two paths to hdisks. Note: Attributes “algorithm” and “algorithm of hdisks” do not need to be altered from default values.

Example 3 Create one zone that contains two Fibre Channel HBA ports and all four Fibre Channel ports from the DS3400. This configuration provides eight paths to hdisks. Four of the paths that are associated with the preferred storage controller will service I/Os when the DS3400 is Optimal. The other four paths are used if the preferred controller is inoperable. Note: The hdisk attribute ″algorithm″ should be set to round_robin. When this attribute is set to round_robin the hdisk attribute ″reserve_policy″ must be set to no_reserve, pr_exclusive, or pr_shared.

The default multipath algorithm for DS4K/DS5K devices with MPIO on AIX is fail_over with a reserve_policy of single_path.

# lsattr -El hdisk3
PCM          PCM/friend/otherapdisk                Path Control Module             False
PR_key_value none                                  Persistant Reserve Key Value    True
algorithm    fail_over                            Algorithm                      True
autorecovery no                                  Path/Ownership Autorecovery    True
clr_q          no                                  Device CLEARS its Queue on error True
cntl_delay_time 0                                     Controller Delay Time          True
cntl_hcheck_int 0                                     Controller Health Check Interval True
dist_err_pcnt 0                                     Distributed Error Percentage    True
dist_tw_width 50                                  Distributed Error Sample Time True
hcheck_cmd    inquiry                               Health Check Command          True
hcheck_interval 60                                  Health Check Interval          True
hcheck_mode    nonactive                            Health Check Mode             True
location                                              Location Label                True
lun_id       0x0                                  Logical Unit Number ID          False
lun_reset_spt yes                                  LUN Reset Supported             True
max_retry_delay 60                                  Maximum Quiesce Time          True
max_transfer 0x40000                               Maximum TRANSFER Size          True
node_name    0x200600a0b81163b8                   FC Node Name                   False
pvid          none                                  Physical volume identifier    False
q_err          yes                                  Use QERR bit                   True
q_type       simple                               Queuing TYPE                   True
queue_depth    10                                  Queue DEPTH                   True
reassign_to    120                                  REASSIGN time out value       True
reserve_policy  single_path                         Reserve Policy                True
rw_timeout    30                                  READ/WRITE time out value       True
scsi_id       0xda                                  SCSI ID                         False
start_timeout 60                                  START unit time out value       True
unique_id    3E213600A0B8000118FD000009A1F49D1EDB20F1815 FAStT03IBMfcp Unique device id False
ww_name       0x202700a0b81143a0                   FC World Wide Name             False

In order to use multiple paths to the preferred controller with native AIX MPIO it is required to set the hdisk's algorithm attribute to round_robin and the reserve_policy attribute to no_reserve (no_reserve will be fine for most situations but is depending on your specific SCSI reservation policy) using:

# chdev -l <hdisk#> -a algorithm=round_robin -a reserve_policy=no_reserve -P

This change requires a reboot. Alternatively one can stop using the hdisks (varyoff any VG with the disks) and eliminate the -P flag to go into effect without a reboot.

The MPIO reserve_policy defines whether a reservation methodology is employed when the device is opened. The values are as follows:

# lsattr -Rl <hdisk#> -a reserve_policy
no_reserve
single_path
PR_exclusive
PR_shared

no_reserve Does not apply a reservation methodology for the device. The device might be accessed by other initiators, and these initiators might be on other host systems.

single_path Applies a SCSI2 reserve methodology for the device, which means the device can be accessed only by the initiator that issued the reserve. This policy prevents other initiators in the same host or on other hosts from accessing the device. This policy uses the SCSI2 reserve to lock the device to a single initiator (path), and commands routed through any other path result in a reservation conflict.
Path selection algorithms that alternate commands among multiple paths can result in thrashing when the single_path reserve value is selected. As an example, assume a device-specific PCM has a required attribute that is set to a value that distributes I/O across multiple paths. When single_path reserve is in effect, the disk driver must issue a bus device reset (BDR) and then issue a reserve using a new path for sending the next command to break the previous reservation. Each time a different path is selected, thrashing occurs and performance degrades because of the overhead of sending a BDR and issuing a reserve to the target device. (The AIX® PCM does not allow you to select an algorithm that could cause thrashing.)

PR_exclusive Applies a SCSI3 persistent-reserve, exclusive-host methodology when the device is opened. The PR_key attribute value must be unique for each host system. The PR_key attribute is used to prevent access to the device by initiators from other host systems.

PR_shared Applies a SCSI3 persistent-reserve, shared-host methodology when the device is opened. The PR_key value must be a unique value for each host system. Initiators from other host systems are required to register before they can access the device.

The PR_key_value is required only if the device supports any of the persistent reserve policies (PR_exclusive or PR_shared).

For more information on native AIX MPIO and the related device attributes please take a look at:

System p and AIX Information Center - Multiple Path I/O
http://publib.boulder.ibm.com/infocenter/pseries/v5r3/index.jsp?topic=/com.ibm.aix.baseadmn/doc/baseadmndita/devmpio.htm

With the release of SDDPCM version 2.4.0.0 DS4K and DS5K now are also supported with the additional SDDPCM path control module. SDDPCM is compliant with the MPIO architecture and as such, is an alternative to using the default MPIO PCM that comes with AIX, and is preferable to just using MPIO. RDAC is not an option for the DS5000 or DS3000, and one uses MPIO or preferably SDDPCM which adheres to the MPIO architecture.

Be aware that the default path selection algorithm with SDDPCM is load_balance with the device reserve policy set to no_reservewhich means that no reservation methodology for the device is applied. A SCSI-2 reservation which is typically in place when doing a varyonvg of an AIX volume group is _not_ applied actually with SDDPCM default settings for the device (here the hdisks in that VG) which are reserve_policy=no_reserve and algorithm=load_balance. Such an AIX volume group might be varied on and accessed by other initiators and even other host systems in this case!

If the AIX volume group is shared between hosts (not using any cluster services like PowerHA with enhanced concurrent volume groups) and should _not_ be accessed by other hosts using a SCSI-2 reservation methodology the device reserve_policy can be set set to single_path ( SCSI-2 reserve) after setting the device path selection algorithm to fail_over. Any attempt to set the algorithm to round_robin, load_balance, or load_balance_port with single_path reserve policy will fail.

For a general documentation of SDDPCM support with DS4000/DS5000 take a look at the

Multipath Subsystem Device Driver User's Guide
http://www.ibm.com/support/docview.wss?rs=540&context=ST52G7&uid=ssg1S7000303
ftp://ftp.software.ibm.com/storage/subsystem/UG/1.8--2.4/SDD_1.8--2.4_User_Guide_English_version.pdf

and also the SDDPCM readme

SDDPCM 2.4.0.0 for AIX Readme
ftp://ftp.software.ibm.com/storage/subsystem/aix/2.4.0.0/sddpcm.readme.2.4.0.0.txt

which always provides latest information about prerequisites, APARs and other restrictions that currently apply.

Please note, the following restrictions (taken from the SDDPCM 2.4.0.0 readme) currently apply with SDDPCM and DS4000/DS5000:
VIOS and HACMP are not supported with SDDPCM on DS4000 and DS5000 subsystem devices
...the SAN boot function is not supported with DS4000 and DS5000 subsystem devices at this release

Furthermore, the SDDPCM packages also provides the additional pcmpath query command to display and manage SDDPCM devices:

# pcmpath query device

Total Dual Active and Active/Asymmetrc Devices : 2

DEV#: 0  DEVICE NAME: hdisk1  TYPE: 2107900  ALGORITHM:  Load Balance
SERIAL: 75KA3628201
==========================================================================
Path#    Adapter/Path Name       State    Mode    Select    Errors
0          fscsi0/path0          OPEN NORMAL    844158       0
1          fscsi0/path1          OPEN NORMAL    843779       0
2          fscsi1/path2          OPEN NORMAL    833541       0
3          fscsi1/path3          OPEN NORMAL    833459       0

DEV#: 1  DEVICE NAME: hdisk2  TYPE: 2145  ALGORITHM:  Load Balance
SERIAL: 60050768018082E29000000000000008
==========================================================================
Path#    Adapter/Path Name       State    Mode    Select    Errors
0*       fscsi0/path0          OPEN NORMAL       6       0
1          fscsi0/path1          OPEN NORMAL 2124784       0
2          fscsi1/path2          OPEN NORMAL 2148331       0
3*       fscsi1/path3          OPEN NORMAL       8       0

Total Active/Passive Devices : 1

DEV#: 3  DEVICE NAME: hdisk3  TYPE: 1815    ALGORITHM:  Load Balance
SERIAL: 600A0B8000118FD00000878E49C33524
==========================================================================
Path#    Adapter/Path Name       State    Mode    Select    Errors
0*       fscsi0/path0          OPEN NORMAL       0       0
1*       fscsi1/path2          OPEN NORMAL       0       0
2          fscsi0/path1          OPEN NORMAL    94440       0
3          fscsi1/path3          OPEN NORMAL    94168       0

The DS4000 and DS5000 products are dual array controller disk subsystems. Each LUN is assigned to one controller, which is considered the owner, or the active (primary) controller, of a particular LUN. The other controller is considered as an alternate, or passive (secondary), controller. The '*' next to the path ID in the pcmpath command output indicates that the related path is a non-preferred path. The indicator '*' for preferred and non-preferred paths are only shown after the device is opened once.

SDDPCM distinguishes the following paths to a given LUN on a DS4000 and DS5000 subsystem:
Paths to the primary (active) controller that currently owns the LUN and processes the I/Os to that LUN
Paths to the alternate (passive/secondary) controller

With this type of active/passive dual-controller subsystem device, I/O can be sent only to the controller that currently owns the LUN. When the SDDPCM selects paths for I/O, it selects paths that are connected only to the active controller for a given LUN. If there is no path on the active controller that can be used, SDDPCM changes the LUN controller ownership to an alternate controller, switches the paths that were passive to active, and then selects these active paths for I/O.

As shown in the example above for a DS4800 (model type 1825) the '*' indicates the paths to the secondary or alternate controller for the LUN. That's why there were no I/Os seen on these paths. Here the I/O load for the given LUN is balanced only across the paths to the primary DS4K/DS5K controller that currently owns the LUN. With two host FC adapters (HBAs) in the server and each host adapter seeing both DS4K/DS5k controllers the load-balancing algorithm with SDDPCM will balance I/Os across both host FC adapters for all I/Os to all DS4K/DS5K LUNs.

The SDDPCM fileset also provides the sddpcm_get_config command for DS4000/DS5000 devices which displays information about all MPIO-based DS4K/DS5K subsystems and the hdisks associated with them. Specifically, it displays information about the frame (subsystem), including the frame's assigned name and worldwide name, and a list of hdisks (only those currently in the Available state) associated with that subsystem, including the hdisk name, LUN number, current ownership, preferred path, and the user-assigned label for that volume.

# sddpcm_get_config -Av
Frame id 0:
Storage Subsystem worldwide name: 60ab8001143a0000049b88968
Controller count: 2
Partition count: 1
Partition 0:
Storage Subsystem Name = 'VIOS_DS4800'
      hdisk    LUN # Ownership       User Label
      hdisk3       0 A (non-preferred)  lpar3_vol01

The SDDPCM support matrix and the SDDPCM filesets (devices.sddpcm.61.rte) for DS4000/DS5000 are available here:

Support Matrix for Subsystem Device Driver, Subsystem Device Driver Path Control Module
http://www.ibm.com/support/docview.wss?rs=540&context=ST52G7&dc=DA400&uid=ssg1S7001350

Next >

SEO by Artio