iSCSI Troubleshooting: Login, Timeout and Path Errors

iSCSI faults are diagnosed badly because the symptom always arrives at the wrong layer. An MTU mismatch on a switch presents as a database timing out. A CHAP typo presents as a disk that will not appear. A dead path presents, minutes later, as a filesystem going read-only. Working from the symptom backwards wastes time; working up the stack in a fixed order does not.

The order below is the one that resolves fastest, because each step rules out everything beneath it.

1. Is the portal reachable at all?

Before anything protocol-specific, confirm TCP reachability to the target portal on port 3260 from the host, on the interface that is supposed to carry storage traffic. Not from a management interface, and not by pinging the array’s management address. A surprising share of “iSCSI is broken” turns out to be a host that has one route to the storage subnet and it goes out the wrong NIC.

Microsoft’s iSCSI troubleshooting checklist opens in the same place, asking whether iSCSI, management and client networks are segregated and correctly routed, and whether MTU, VLAN, jumbo frame and flow control settings are consistent.

If the port does not answer, stop. Nothing further in this list is relevant.

2. Does discovery succeed but login fail?

These are two separate exchanges and separating them is the most valuable single diagnostic in iSCSI.

Discovery is a SendTargets request to the portal. If it returns a target list, the network path and the target service are both alive. Login is the session establishment against a specific target, and it is where identity and authentication are evaluated.

Discovery succeeding while login fails narrows the fault to three things, in descending order of likelihood:

• ACL mismatch. The target does not have this initiator’s IQN mapped to any LUN. Read the initiator name from the client rather than from documentation: /etc/iscsi/initiatorname.iscsi on Linux, the Configuration tab of the iSCSI Initiator control panel on Windows. A single transposed character produces exactly this symptom.
• CHAP mismatch. Wrong secret, or one-way CHAP configured on one side and mutual on the other, or a secret that violates the target’s length rules. RFC 7143 is where that floor comes from: a CHAP secret must be at least 96 bits, which is 12 bytes, and the specification recommends at least 128 bits. A target that enforces the minimum rejects a short secret at login rather than at configuration time, so the error surfaces nowhere near the mistake.
• Discovery-time versus session-time authentication. open-iscsi treats these as separate settings, discovery.sendtargets.auth.* and node.session.auth.*, both defaulting to no authentication. Configuring one and not the other yields a target that lists fine and refuses to log in.

Where the target enforces CHAP on discovery too, the failure moves one step earlier and discovery itself returns nothing, which is worth knowing before concluding the portal is unreachable.

3. Login succeeds but the disk misbehaves

Once a session is up, the remaining faults are about data, not identity.

MTU mismatch is the signature failure here. Small packets get through, so discovery and login both succeed, and then anything that fills a frame stalls or retries. If jumbo frames are enabled anywhere in the path they must be enabled everywhere in it, including the switch, and Microsoft’s guidance repeats the point in both its checklist and its resolution steps: make sure MTUs and jumbo frames are consistent end to end. The cheapest test is to revert every hop to 1500 and see whether the problem disappears.

One disk per path means MPIO is not claiming the device. Multiple sessions to the same LUN without a multipath layer above them present as several independent disks with identical contents, and writing through two of them corrupts the volume. Aggregation is the whole job of that layer: Microsoft’s description of the initiator’s Devices tab is that with MPIO the iSCSI Initiator can log in on multiple sessions to the same target and aggregate the duplicate devices into a single device exposed to Windows, each session using different network adapters, network infrastructure and target ports. Without it, nothing merges them. On Windows the checks are Install-WindowsFeature Multipath-IO for the feature itself and mpclaim -s -d to see whether the Microsoft DSM has claimed the disks; on Linux the equivalent is that multipath -ll shows one map with several paths rather than nothing at all.

Paths on one subnet is the design error behind most of the above. Each path wants its own subnet and its own interface so that path selection is unambiguous, and Microsoft’s MPIO guidance makes the hardware half of that explicit: for redundancy, each iSCSI connection should use a different network adapter, and a system that detects only one path needs its iSCSI connections rebuilt. The reasoning is set out in iSCSI fundamentals: targets, LUNs and multipathing.

4. The timers that decide when I/O fails

This is where “it recovered but the application died anyway” gets explained, and the defaults are worth memorising.

open-iscsi pings each session with NOP-Out requests. The shipped iscsid.conf sets node.conn[0].timeo.noop_out_interval = 5 and node.conn[0].timeo.noop_out_timeout = 5, both documented as five seconds. When a NOP-Out times out, the iSCSI layer fails the running commands and instructs the SCSI layer to requeue them.

The session then has a grace period before failure is passed upward. node.session.timeo.replacement_timeout = 120 controls how long the iSCSI layer waits for a timed-out path or session to re-establish itself before failing the commands on it. Red Hat’s description is the same: replacement timeout governs how long to wait for a path to come back before giving up on it.

Those defaults suit a single-path host, where waiting two minutes for the path to return beats failing I/O. They are wrong for a multipath host. With MPIO in place the correct behaviour is to fail fast on a dead path so the multipath layer can move I/O to a live one. Red Hat puts a number on it: with queue_if_no_path set in multipath.conf, replacement_timeout belongs at 15 to 20 seconds, so that I/O is handed to a new path quickly while the iSCSI layer keeps trying to re-establish the failed one underneath. Leaving it at 120 on a multipath host produces the worst outcome available: two minutes of stalled I/O on a fabric that had a perfectly good second path the whole time.

On Windows the equivalent lever is the disk timeout. Microsoft’s guidance for surprise-removal and failover problems includes setting TimeOutValue under HKLM\SYSTEM\CurrentControlSet\Services\disk to a larger number, such as 179. Change it deliberately: a longer disk timeout hides transient path loss from applications, and also hides real path loss for the same interval.

The other open-iscsi default worth knowing is node.startup = manual. A session that was working perfectly and vanishes after a reboot is usually this, not a fault.

5. Reading the Windows event log

Microsoft’s troubleshooting article lists the events that matter, and they are far more specific than the generic disk errors that surround them:

• Event ID 157, “Disk X has been surprise removed”, is the classic signature of a path or session dropping under an active volume.
• Event IDs 9, 20, 27, 39 and 153 cover the iSCSI side, including “Target did not respond”, “Initiator failed to connect” and retried I/O at a logical block address.
• The documented causes are network instability, MPIO configuration errors, adapters or NIC teams not being ready when the iSCSI service starts so ports cannot bind, mismatched VLAN or MTU or jumbo settings, outdated firmware and drivers, and resource exhaustion on the array.

For state rather than history, Microsoft points at Get-IscsiConnection, Get-IscsiSession and Get-MSDSMAutomaticClaimSettings to gather path and session status, and Get-Disk and Get-PhysicalDisk to review the resulting disk mappings.

Two items in that list are easy to miss and expensive to rediscover. First, applications and scripts must not rely on disk numbers, because path failovers can change them. Second, LBFO NIC teaming is deprecated for Hyper-V deployments as of Windows Server 2022, with switch embedded teaming as the replacement, so a teaming configuration inherited from an older build is a legitimate suspect rather than a stable baseline.

6. When the volume itself is damaged

If a volume has gone RAW or a filesystem is reporting checksum errors, the protocol layer is no longer the problem and the priority order changes: back up the affected volume first, repair second. Microsoft’s sequence is explicit about that ordering, with chkdsk /f and chkdsk /r for NTFS and refsutil salvage for ReFS, and with the log and recovery folders required to be on a different volume from the damaged one.

Two causes deserve checking before the repair, because both will recreate the damage: a LUN attached read-write by more than one host without a cluster-aware filesystem, and a thin-provisioned LUN whose backing store filled up. Neither is a filesystem bug. The first is an architectural mistake described in iSCSI vs NFS vs SMB, where the comparison of ownership models makes clear why block storage arbitrates nothing.

A short checklist

1. TCP 3260 answers from the storage interface, not the management one.
2. Discovery returns a target list.
3. The target’s ACL contains the initiator’s actual IQN, copied from the client.
4. CHAP is configured on the same side, in the same direction, at both discovery and session scope.
5. MTU is identical on host, switch and target, or 1500 everywhere.
6. Each path has its own subnet and its own interface.
7. The host shows one multipath device, not one disk per path.
8. Session timers match the topology: fail fast with MPIO, wait with a single path.
9. Node startup is automatic if the LUN is expected to survive a reboot.

Most persistent iSCSI faults are one of items 3, 5 and 6. For the setup-side decisions that avoid them on a NAS-class target, see Unraid iSCSI target setup, and to check whether the number of paths in the design can carry the workload at all, the iSCSI SAN bandwidth and MPIO sizer works the arithmetic.