Common GPU Compute Server Faults Analysed: GPU Drop-off, Not Detected and TRAY Board Issues
With the large-scale deployment of data-centre-class compute servers such as the H100, H200 and H800, the demand for related fault handling has grown rapidly. This class of equipment carries high unit value and long lead times, and its fault symptoms frequently overlap, which makes the diagnostic method especially important.
1. Common Fault Symptoms
- GPU not detected: the system or out-of-band management cannot see all GPUs
- GPU drop-off: a GPU disappears during operation, interrupting the service
- Indicator lit but card not readable: TRAY board indicators are normal, but the GPU cannot be read
- Power-on self-test failure: GPU or PCIe related errors reported during the POST stage
- Frequent throttling or computation errors: detected, but with abnormal performance or incorrect computation results
2. Which Layer the Fault May Lie In
The same symptom can have a completely different root cause:
| Fault layer | Possible cause |
|---|---|
| GPU module | Chip itself, HBM memory, power module failure |
| TRAY board | On-board power delivery, signal traces and routing, poor connector contact |
| PCIe Switch | Switch chip functional module failure, firmware abnormality |
| High-speed interconnect | NVLink/NVSwitch link abnormality |
| Power supply and cooling | Abnormal power module output, insufficient cooling leading to throttling |
Special note: in the SXM architecture, multiple GPUs are interconnected through the TRAY board and the PCIe Switch. A GPU that is not detected therefore does not mean the GPU is faulty - in many cases the root cause is ultimately found on the TRAY board or in the PCIe Switch chip.
3. Diagnostic Approach
1. Read the out-of-band management log first
BMC event logs usually record the time of the first hardware-level anomaly and the location of the component involved, which makes them the most efficient starting point. Focus on: GPU presence detection, power, temperature and PCIe link errors.
2. Cross-validation method
This is the most reliable method of locating a fault, and its core principle is changing only one variable at a time:
- Move the suspect GPU to a known-good slot → the fault follows the card = the card is the problem
- Still faulty after moving to a known-good slot → investigate the TRAY board or the switch
- Insert a known-good GPU into the suspect slot → still faulty = the board or the slot is the problem
- Recovery after replacing the TRAY board → located at board level
3. Check links and power delivery
- Whether PCIe link width and speed negotiate correctly (for example x16 dropping to x8/x4)
- Whether the supply voltage is stable and free of dips
- Check high-speed differential pair connectors for oxidation, cold solder joints and abnormal impedance
4. Rule out environmental factors
- Whether the inlet temperature exceeds the specified limit and causes throttling
- Whether the airflow path is obstructed
- Whether the rack power supply meets full-load requirements
4. Repair Level and Decision
| Level | Approach | Applicable situation |
|---|---|---|
| Whole system / whole card replacement | Replace directly with new | Within the warranty period, time-critical |
| Board-level repair | Replace the TRAY board, power module, fans and so on | Modules can be replaced independently and spare parts are available |
| Component-level repair | Replace the PCIe Switch chip, BGA reballing, repair traces | End of life (EOL), out of stock, cost-sensitive |
Why component-level repair is worth considering: equipment such as the H100 and H200 carries a high per-card cost and long original manufacturer lead times. When a TRAY board or switch chip fails, the cost and waiting time of a whole-board replacement are both considerable. A service provider with chip-level capability can complete the repair within a shorter cycle, and the value of that to business continuity is very direct.
5. Recommendations for Operations Teams
1. Keep complete fault records: time of the first alarm, out-of-band logs and cross-validation results.
2. A spare parts strategy should be tiered: for critical models, prepare TRAY board or whole-card-level spares.
3. Do not treat "replaced and working" as "root cause found": confirm the root cause, otherwise the same batch of equipment may fail repeatedly.
4. Read health data regularly: trends in GPU temperature, power consumption and ECC error counts are more meaningful than any single reading.
The core competitiveness of compute equipment repair does not lie in parts, but in whether a fault can be accurately located down to board level and component level.
Note: This article is provided for technical exchange and experience sharing. Always refer to the original manufacturer's technical documentation and the conditions of your own environment. Maintenance and change operations should be carried out by qualified engineers.