iSCSI Fundamentals: Targets, LUNs and Multipathing

iSCSI carries SCSI commands over TCP/IP. That single sentence explains most of its behaviour, including the parts that surprise people coming from file sharing protocols. The client is not asking a server for a file. It is issuing block level commands to what it believes is a locally attached disk, and the network is only the transport.

Initiators, targets and LUNs

The initiator is the client. It runs in software on the host operating system, or in hardware on an iSCSI capable adapter, and it is the side that issues commands.

The target is the storage side. It presents one or more logical units, each identified by a logical unit number, or LUN. A LUN is the thing the host sees as a raw disk. It can be backed by a file, a logical volume, a ZFS zvol, a whole physical disk or an array volume, and the initiator neither knows nor cares which.

Both ends are named with an iSCSI qualified name, or IQN. The format is a fixed prefix, a reversed DNS domain, a date component establishing that the domain was owned at that time, and a unique suffix chosen by the administrator. IQNs are identifiers, not addresses, and they are how targets decide which initiators are allowed to see which LUNs. A worked example of that mapping on a NAS-class box, including which backing store to choose, is in Unraid iSCSI target setup.

Discovery and authentication

The initiator connects to a portal, which is an IP address and port on the target, and performs discovery. The target responds with the list of targets it is willing to advertise. The initiator then logs in to a specific target and begins a session.

Access control has two independent layers and both matter. Initiator masking restricts which IQNs may access a given LUN. CHAP provides authentication during login. One way CHAP authenticates the initiator to the target. Mutual CHAP additionally authenticates the target to the initiator, which is what prevents a host from attaching to an impostor target.

Discovery and login being separate exchanges is the most useful diagnostic fact in the protocol: a target that lists but refuses to log in has an identity or authentication problem, not a network problem. That split, and how to read it, is the starting point of iSCSI troubleshooting for login, timeout and path errors.

Neither layer encrypts traffic. iSCSI data crosses the wire in the clear unless you wrap it in IPsec or confine it to a trusted network segment. Confining it is the usual answer.

Network design

Treat the storage network as a separate network. A dedicated VLAN or dedicated physical interfaces keep general traffic from competing with storage traffic, and keep the storage fabric out of reach of clients that have no business on it.

Flow control and switch buffering matter more here than on general networks, because TCP retransmission on a storage path shows up to applications as latency spikes on a disk. Larger frame sizes can reduce per packet overhead, but every device in the path must agree on the setting, and a mismatch produces silent fragmentation or dropped frames that are painful to diagnose.

Before adding interfaces, it is worth knowing whether the network or the media is the ceiling. The iSCSI SAN bandwidth and MPIO sizer works that arithmetic for a given path count, block size and storage class.

MPIO is not link aggregation

This is a recurring architectural error. Link aggregation bonds physical links into one logical link, and a single TCP connection still traverses one member link, so a single iSCSI session gains no bandwidth from it.

Multipath I/O works at the SCSI layer instead. The initiator establishes independent sessions over independent paths, recognises that the LUNs arriving on each path are the same device, and presents one device to the operating system. The multipath layer then distributes I/O across paths and fails over when one dies. The standard design gives each path its own subnet and its own physical interface, so that path selection is unambiguous and a switch failure removes exactly one path.

Common mistakes

Mounting the same LUN read write on two hosts with an ordinary filesystem. Block storage does no arbitration, so both hosts cache and write independently and the filesystem corrupts. Sharing a LUN requires a cluster aware filesystem or a clustered volume manager. When several machines genuinely need the same data at once, a file protocol is the right tool instead, and the ownership models are compared in iSCSI vs NFS vs SMB.

Others worth avoiding: putting all paths on one subnet, which makes multipathing ineffective; leaving CHAP disabled on a routable network; thin provisioning past physical capacity with no monitoring, which turns a full backing store into write failures on every attached host; and forgetting that snapshots on the target are crash consistent unless the host is quiesced first.

Where to go next

• Build a target on consumer NAS hardware: Unraid iSCSI target setup.
• Decide whether block storage is the right choice at all: iSCSI vs NFS vs SMB.
• Diagnose a session that will not come up, or a path that keeps dropping: iSCSI troubleshooting: login, timeout and path errors.
• Size the fabric: iSCSI SAN bandwidth and MPIO sizer.