Two Ports, One 200G: Cabling a DGX Spark Cluster from 2 Nodes to 8
Summary. A DGX Spark has 2 QSFP ports rated at 200 Gb/s each, but they are not 400 Gb/s of network. The ConnectX-7 reaches the SoC over 2 PCIe Gen5 x4 links, so the whole box gets about 200 Gb/s per direction, and a single cable only delivers it if you load both links. That one fact sets every layout: 1 cable for 2 Sparks, 3 cables in a triangle for 3, a ring or a switch for 4, and a switch beyond that. On cables, 12 of the 13 I tested on my own pair gave an identical 185.1 Gb/s. The 13th ran clean for 101 minutes at idle, then dropped the link 3 minutes into a GPU burn. ...