iSCSI vs NFS vs SMB: Choosing a Storage Protocol

Most comparisons of these three protocols open with throughput numbers. That is the least useful place to start, because on a healthy network with the same disks behind it, all three will saturate a link long before the protocol becomes the limit. The decision that actually matters is architectural, and it is decided before any of them is configured.

One question decides it: who owns the filesystem?

iSCSI is a transport for SCSI. RFC 7143 defines it as a protocol that carries SCSI over TCP while remaining compliant with the SCSI Architecture Model. What crosses the wire is block commands, not file requests. The target exposes a LUN; the client formats it, mounts it, and owns the filesystem entirely. The storage box has no idea what is on it.

NFS and SMB are file protocols. The server owns the filesystem, resolves paths, enforces permissions and arbitrates concurrent access. The client asks for operations on files and the server decides what happens.

Everything else in this comparison follows from that one difference.

Concurrency and locking

Because the storage side of an iSCSI target has no filesystem, it has no lock manager. Two hosts can attach the same LUN simultaneously and both will believe they own it. With an ordinary single-host filesystem on that LUN, both will cache metadata, both will write, and the volume will be destroyed. Sharing a LUN safely requires a cluster-aware filesystem or a clustered volume manager on top of it. This is the single most common architectural mistake with block storage and it is covered further in iSCSI fundamentals: targets, LUNs and multipathing.

Virtualization platforms solve this by supplying the cluster filesystem themselves. In vSphere, a block datastore is formatted with VMFS so multiple hosts can share one volume; Broadcom’s NFS documentation contrasts this directly, noting that you do not need to format an NFS volume with a local filesystem such as VMFS, and instead mount the volume directly on the ESXi hosts.

File protocols arbitrate by design, but not identically. Broadcom’s own NFS 3 versus NFS 4.1 comparison lists NFS 3 as using proprietary client-side locking, while NFS 4.1 uses server-side locking. SMB places locking on the server as well, with oplocks and leases governing what a client is allowed to cache.

Path redundancy

This is where the protocols diverge sharply, and where a lot of “NFS is slower” folklore comes from.

iSCSI redundancy is multipath I/O. The initiator opens independent sessions over independent paths, recognises that the same LUN has arrived on each, and presents one device with a policy layer distributing I/O across paths and failing over when one dies. It works at the SCSI layer, above TCP, which is why it aggregates where link bonding does not.

NFS depends on the version. Broadcom documents NFS 3 as not supporting multipathing, and NFS 4.1 as supporting it through session trunking. An NFS 3 datastore therefore has exactly one active path per mount, no matter how many interfaces the host has.

SMB 3 has Multichannel, which discovers multiple network paths between client and server and uses them concurrently. Microsoft’s file-server overview groups it with SMB Direct (RDMA) as the performance and connectivity features that dependent services such as SQL Server over SMB inherit.

Authentication and encryption

None of these three is secure by default in the way people assume.

iSCSI authenticates with CHAP. Red Hat’s description is exact: CHAP lets you protect the target with a password that the initiator must know in order to connect. Add mutual CHAP and the target proves itself to the initiator as well. Access control is separate, applied as an ACL keyed on the initiator’s IQN. What is absent is confidentiality: RFC 7143 defines a SCSI transport, and payload crosses the wire in the clear unless it is wrapped in IPsec or confined to a trusted segment.

NFS again splits by version. Broadcom lists NFS 3 as supporting AUTH_SYS only, which is to say the client asserts a UID and the server believes it. NFS 4.1 adds Kerberos, documented as krb5 and krb5i, with AES256-CTS-HMAC-SHA1-96 and AES128-CTS-HMAC-SHA1-96 among the supported encryption algorithms.

SMB is the strongest of the three out of the box on a Windows estate, because identity is Kerberos or NTLM against a directory rather than an asserted UID, and because Microsoft documents encryption, signing, NTLM blocking and authentication rate limiting as first-class protocol features rather than as add-ons.

Side by side

	iSCSI	NFS	SMB
Unit of access	Block (LUN)	File	File
Filesystem owner	Client	Server	Server
Safe multi-host write	Only with a cluster filesystem	Yes	Yes
Multiple paths	MPIO at the SCSI layer	None in NFS 3; session trunking in NFS 4.1	SMB Multichannel
Authentication	CHAP, plus IQN ACLs	AUTH_SYS in NFS 3; Kerberos added in NFS 4.1	Kerberos or NTLM, with signing
Encryption in protocol	None; needs IPsec	None in NFS 3; Kerberos krb5i integrity in NFS 4.1	SMB encryption
vSphere datastore	Formatted with VMFS	Mounted directly, no VMFS	Not a vSphere datastore type
Typical failure mode	Host sees disk errors or a dead device	Mount hangs or goes stale	Share disconnects, handles recover

One entry in that table deserves emphasis. Broadcom’s comparison table also records Storage I/O Control as supported on NFS 3 but not on NFS 4.1, and Storage DRS on NFS 4.1 as limited to initial placement and load balancing based on space utilisation. Moving from NFS 3 to NFS 4.1 to gain multipathing therefore costs vSphere features, which is a real tradeoff and not an obvious one.

Failure behaviour

The protocols fail differently, and the difference is felt by applications rather than by administrators.

When an iSCSI path dies, the initiator’s SCSI layer is in charge. It fails commands, requeues them, and if no path recovers before the replacement timeout expires, the block device starts returning errors upward. Filesystems react to that as a disk fault: read-only remounts, journal aborts, Windows marking a volume RAW. The recovery timers and the events that fire along the way are worked through in iSCSI troubleshooting: login, timeout and path errors.

When an NFS server goes away, a hard mount blocks. Processes sit in uninterruptible sleep and the box looks hung rather than broken, which is ugly but is usually recoverable without data loss when the server returns.

When an SMB server goes away, clients disconnect and, with transparent failover on a clustered file server, reconnect and resume open handles. Microsoft documents transparent failover and Scale-Out File Server as part of the high-availability feature set for exactly this case.

Choosing

Pick iSCSI when the workload wants a disk rather than a share: a single VM’s storage, a database volume, a boot LUN, a backup target that a client-side agent will format itself, or anything that needs SCSI-level semantics such as persistent reservations. Accept in exchange that you own the filesystem, the single-writer rule and the multipath design. Sizing the path count and spotting whether the network or the media is the ceiling is what the iSCSI SAN bandwidth and MPIO sizer is for.

Pick NFS when many hosts need the same data at the same time on a Unix or virtualization estate, and when operational simplicity matters more than block semantics. Choose the version deliberately rather than by default; the multipathing and security gap between 3 and 4.1 is wide.

Pick SMB when the clients are Windows, when identity should come from the directory rather than from a UID the client asserts, or when the workload is user data rather than machine data.

A mixed estate is normal and not a sign of indecision. The failure is using one protocol where the other one’s ownership model was needed, and the classic version of that failure is a LUN mounted read-write by two machines that both thought they were alone. Setting one up correctly on a NAS-class box, including where the backing store should live, is covered in Unraid iSCSI target setup.