RTX 5090 16-pin connector thermal watchdog (Pi Pico)#

Status: Proposal (draft): designed and bench-simulated, not yet built on a node Scope: one watchdog per RTX 5090 service node (PC1, PC2, future nodes) Code: ../connector-watchdog/: Pico firmware main.py, host connector_watchdog.py, connector-watchdog.service Why: connector-melt evidence (connector melts are silent until damage is done) and the PSU decision

What it does#

A thin-film temperature sensor sits between the wires just behind the GPU-end 16-pin plug, and a second sensor stands on the board in the case air as a reference. A Raspberry Pi Pico reads both once a second and reports to the node over USB. The GPU-plug sensor drives every action:

The host also alerts on:

• Detached sensor: GPU above 400 W for 5 min while the plug sensor reads within DETACH_DELTA_C (default 8 °C) of the case air. A sensor touching loaded wires always reads well above the air; one that has fallen away doesn't.
• Airflow problem: case air ≥ 60 °C.
• Sensor fault: either sensor reads open or shorted.
• Lost Pico: no data for 15 s.

It adds no connection to the power path: the sensor only touches the outside of the wire insulation.

What it can and cannot catch#

• Why the sensor sits on the wires, not the housing. The fault starts at a metal terminal inside the plug. The plastic housing insulates, so its outer surface reads lower and later than the terminal. Copper carries terminal heat back along the wire faster than it crosses the plastic wall, and wire insulation is thinner. That's where the sensor in Thermal Grizzly's June 2024 demo sat too.
• Slow degradation is caught well, and it's the common case. In the incident data, owners typically saw days or weeks of crashes or discolouration before a melt. A contact whose resistance creeps up over hours or days shows on the wires long before plastic melts.
• Sudden faults are caught later. If a pin suddenly loses contact and its neighbours jump to 20 A or more, the inside can reach damaging temperatures before the outside crosses a fixed threshold. The rate-of-rise trigger fires early in such an event, while the absolute temperature is still modest. In a sudden fault the watchdog should be expected to limit the damage (stop a fire, save the GPU socket) rather than prevent every melt.
• Only per-pin current sensing has no thermal lag (a Thermal Grizzly WireView, or an ASUS Astral card's own monitoring). An inline WireView adds a connection to the power path, so it's optional. It's worth using once to check a new cable's per-pin balance at commissioning.
• Unverified, possible zero-lag signal. Windows tools read a "16-pin input voltage" from the card, which sags as contacts degrade: 11.16 V and 11.4 V in two incidents, against about 11.8 V healthy. Whether Linux exposes it (e.g. via NVML) has not been checked. If it does, it's a free extra trigger.
• Accepted gap: the PSU end is not monitored. In the incident data, 9 of 26 native-cable melts were at the PSU end only, including the Hardware Unboxed and Club386 cables. Those usually spare the GPU, which is what this protects, so the PSU end is left to the periodic IR check. Adding it later costs one sensor and one 10 kΩ resistor on the free input GP28 (pin 34).

Parts (per node)#

Buying tips:

• Resistors: a 50-pack of 330 Ω (one per node, spares for years), plus 10 kΩ 1% metal film (2 per node). Or one 1/4 W 1% metal-film assortment kit, which covers everything. Power rating is a non-issue: nothing here dissipates more than about 15 mW.
• Check every resistor with the multimeter before fitting. Colour bands on blue metal-film resistors are easy to misread.

Why a 10k resistor with a 100k thermistor#

The Pico measures voltage, not resistance, so each thermistor is paired with a fixed resistor as a voltage divider. The divider is most precise where the thermistor's resistance is close to the fixed resistor's. These 100k thermistors fall to about 10 kΩ near 90 °C, the range we care about:

The sensor self-heats less than 0.5 °C at these currents.

Wiring#

Temperature sensors#

 Pico pin 36  3V3(OUT) ──┬─────────────┐
                         │             │
                      [10k 1%]      [10k 1%]
                         │             │
 Pico pin 31  GP26/ADC0 ─┤             ├─ Pico pin 32  GP27/ADC1
                         │             │
              (100nF)────┤             ├────(100nF)        optional, cap to AGND
                         │             │
                  [film NTC 100k]  [bead NTC 100k]
                  GPU PLUG WIRES     CASE AIR        ← film on Dupont leads; bead on the board
                         │             │
 Pico pin 33  AGND ──────┴─────────────┘

• Circuit: each thermistor sits between its ADC input and AGND, with the 10 kΩ resistor between 3V3(OUT) and the ADC input. The firmware computes R_ntc = 10k × raw / (65535 − raw), so the exact 3.3 V value cancels out. It prints the resistance too, for calibration.
• Faults: a disconnected sensor reads as open and a shorted one as short; both raise alerts, and neither can cause a false shutdown.
• Noise: keep the film sensor's leads away from the 12 V wires once they leave the bundle. Twisting the lead pair helps.

Forced power-off (PC817 across the power-switch header)#

 Pico pin 20  GP15 ──[330R]──► PC817 pin 1 (anode, dot side)
 Pico pin 18  GND  ──────────► PC817 pin 2 (cathode)

 PC817 pin 4 (collector) ──► JFP1 POWER SW  "signal" pin (the one at ~3.3–5 V to ground)
 PC817 pin 3 (emitter)   ──► JFP1 POWER SW  ground pin
                              ▲
 Case power-button leads ─────┘ same two pins, via the Y-splitter (button still works)

 PC817, top view:     1 ● ┌───┐ 4
                          │817│
                      2   └───┘ 3        pins 1–2 = Pico side, 3–4 = motherboard side

• Find the right pins on your board. On both boards here (MAG X870E Tomahawk WiFi, PRO X870E-S EVO WiFi) the front-panel header is JFP1, and the manual shows the pins only as a diagram. MSI boards normally put Power Switch on pins 6 and 8. With the PSU switched on and the PC off, measure each of the two power-switch pins against a chassis screw: the one at about 3.3–5 V is the signal pin and gets the collector (pin 4). The one at 0 V gets the emitter (pin 3). A mechanical button has no polarity; the optocoupler does, and if it's reversed it simply never triggers.
• What the presses do. A 0.3 s press (the TEST command) is a normal button press: Linux does a clean shutdown if HandlePowerKey=poweroff in logind.conf, which is the default. A 6 s hold is the hardware forced-off.
• The latch prevents turning the machine back on. If the motherboard keeps USB powered while the PC is off, pressing the switch again would start the PC. So after its one forced-off, the Pico latches and never presses again until it loses power.

Pico pin summary#

Board layout#

Breadboard (400-point), for building and testing:

• The Pico straddles the centre channel; its pins land in rows c and h, leaving two free holes beside every pin.
• Resistors: a jumper from pin 36 (3V3) to a + rail, then each 10 kΩ resistor from that rail to its ADC pin's column.
• Ground: a jumper from pin 33 (AGND) to the − rail.
• Film sensor: a 2-pin male header in two free holes, one hole in the GPU ADC column (pin 31), the other in the − rail. The sensor's Dupont plug pushes onto it.
• Air bead: one leg in the air ADC column (pin 32), the other in the − rail. Bend the legs at least 2 mm from the glass so the bead stands about 1 cm above the board, a few cm from the Pico's chip. If the thin legs grip loosely, fold each tip over once.
• The PC817 straddles the channel at the far end, dot to the top-left, with pins 1–2 on the Pico's side. Female-male jumpers run from pins 3–4 to the JFP1 splitter.
• Once it's all working, fix the PC817, its resistor, the header jumpers and the bead with a dab of hot glue or tape. A loose sensor or resistor raises an alert. A loose optocoupler lead is silent: the hardware cut-off just stops working, which only the periodic TEST reveals.

Perfboard (5 × 7 cm), the optional permanent version:

• Solder the two female header strips 7 holes apart, so the Pico H plugs in along the 7 cm side.
• Place the resistors, the PC817, the sensor header and the bead in the ~9 free columns beside it, following the same connections. It's about 20 solder joints.
• Mount the board on M3 nylon standoffs or foam tape, so the solder side can't touch metal.

Preparing the thin-film sensor#

The MF5B strip has only short bare legs, so it needs leads. A beginner-friendly method:

1. Take a 2-wire female-female Dupont jumper pair, 30–50 cm, and cut one end off. The remaining plug end pushes onto the 2-pin header on the board.
2. Prepare the wire ends: strip about 3 mm, twist the strands, and tin them (melt a little solder in).
3. Slide a small piece of heat-shrink onto each wire first, and a larger piece over both.
4. Solder one wire to each leg with a brief touch, 2–3 seconds. Hold the leg with tweezers between the joint and the film, so the heat goes into the tweezers instead of the sensing element.
5. Shrink the small pieces over each joint, so the two joints can't touch. Then shrink the larger piece over both joints and the base of the strip, for strain relief.
6. Check with the multimeter: about 100 kΩ at room temperature (98–102 kΩ for the 1% part), falling when you pinch the tip.

Sensor placement#

1. GPU-plug sensor. Slide the tip of the film strip (the dark dot) between the wires in the middle of the bundle, 5–10 mm behind the back of the plug housing, with the film flat against the wires.
   • Why the middle: the ground-return wires carry the same current as the 12 V wires and heat just as much when a contact fails, so any wire is relevant. The middle of the bundle touches the most.
   • Getting it in: open a gap gently with a plastic spudger or toothpick, without levering.
   • Leave the plug alone: no pulling, pushing or sideways force on it. Sideways force on the plug is itself a known cause of melts. If a cable comb sits right at the plug, work around it.
   • Securing it: hold the sensor with one or two loose turns of Kapton around the bundle, snug but not squeezing the wires. Tie the lead to the cable a few cm further back, so a tug lands there. The polyimide film is insulated on both sides, so touching the wires is safe.
2. Air sensor: the bead standing on the board. Put the board in moving case air, not a dead corner and not in the path of the PSU or GPU exhaust.
3. Mounting the board: somewhere cool and non-conductive. The breadboard has an adhesive back. Nothing on it may touch metal.

Software setup#

1. Pico: flash MicroPython (the official RP2040 .uf2: hold BOOTSEL while plugging in USB, then drag the file onto the drive that appears). Copy main.py with mpremote cp main.py :main.py.
2. Node: copy connector_watchdog.py to /opt/connector-watchdog/ and connector-watchdog.service to /etc/systemd/system/.
   • Set WATCHDOG_DEVICE from ls /dev/serial/by-id/.
   • Set WORKER_UNIT to the ComfyUI service.
   • Set ALERT_WEBHOOK to a Discord webhook if wanted.
   • Then systemctl enable --now connector-watchdog.
3. Language: the host script uses only the Python standard library, so a bare node needs nothing installed. That's a deliberate exception to the house TypeScript rule: this is a small on-node daemon, not an operator CLI.

Build, calibration and commissioning (test each stage before the next)#

1. Pico only. Flash MicroPython and run mpremote repl. The board should respond.
2. Air bead on the breadboard. Copy main.py; mpremote shows a line every second. air should read room temperature, and gpu shows nan / open until that sensor is fitted.
3. Film sensor. Prepare its leads and plug it in. Both should now read room temperature within about 1 °C of each other.
4. Calibration check (5 minutes). Put both sensors, with a kitchen thermometer, in a cup of hot water at around 70–80 °C. Keep the joints and the bead's legs dry: put the sensors in a small sealed plastic bag.
   • Pass: each reading agrees with the thermometer within about 2 °C.
   • If one is off by several °C at 80 °C but fine at room temperature, its B value differs from the listing. Take the r= resistance the Pico prints for that sensor at a known temperature T (in kelvin) and compute B = ln(R25 / R) / (1/298.15 − 1/T). Then set that sensor's B in PROBES in main.py.
5. PC817. Find the JFP1 signal and ground pins with the multimeter as above, then wire them. With the node idle, send echo TEST > /dev/serial/by-id/...: the node should shut down cleanly. This proves the wiring without the 6 s force.
6. Fit the sensors in the node as described above, and start the host service.
7. Baseline. Run the GPU at sustained full load (575 W class) for 30 min. Log the plug-sensor and air plateaus, and check the plug with the IR thermometer. Also note how much the plug reading drifts per minute once it has settled.
   • WARN_C = plug plateau + 10–15 °C, capped at 85 °C.
   • DETACH_DELTA_C = about half of (plug plateau − air plateau).
   • RISE_C_PER_MIN = comfortably above the settled drift; the default 4 °C/min suits a drift under 1 °C/min.
   • If the plug plateau is already above 70 °C, stop and re-check seating and airflow before relying on the connector.
8. Test the chain end to end. Temporarily set WARN_C 5 °C above the current reading and warm the plug sensor gently (fingers, or a hair-dryer at a distance; never a heat gun on the connector). Confirm the worker stops, the power cap applies and the alert arrives. Restore WARN_C.
9. Record the node's values in its build record: baseline, WARN_C, DETACH_DELTA_C, RISE_C_PER_MIN, each sensor's B, and a photo of the sensor position.

Ongoing#

• Every few months: check both ends of the cable with the IR thermometer (this is also the PSU-end check) and compare against the baseline. Repeat the TEST press to prove the optocoupler path. Do all of it without unplugging the 16-pin.
• Any warning or rate-of-rise alert means replacing the cable, not re-seating it. Inspect both ends and the GPU socket before re-enabling the worker.
• Separate from this device: a certified photoelectric smoke alarm in the room, which also covers non-connector fires such as board or PSU failures. Optionally, one with a relay output driving a contactor on the node circuit downstream of the UPS, installed by a licensed wireman. Don't use hobby MQ-series gas sensors: they drift with heat and humidity and their heater runs continuously.

Deferred until the above is verified on a node#

These are planned but not built, so that the core watchdog is proven first.

• Status LEDs. A green and a red LED, each with a 330 Ω resistor (100 Ω for blue or white LEDs), on GP12 (pin 16) and GP13 (pin 17). Long leg (+) to the resistor, short leg (flat rim) to GND. Optionally mounted in 5 mm panel holders on the case. Planned patterns:
  • green slow blink: alive and host connected;
  • red fast blink: warning;
  • red double blink: sensor fault, detached sensor, or host not responding;
  • red solid: cut-off.
• Buzzer. An "active low trigger" module (MH-FMD, 3 pins, with an onboard PNP transistor). Wire VCC to 3.3 V (pin 36), not 5 V, because a 3.3 V pin can't fully silence a low-trigger module powered from 5 V. GND to GND, I/O to GP14 (pin 19); pin low = beep. Peel off the "remove seal after washing" sticker for more volume. A short chirp at power-up is normal.
• Host → Pico state messages. The host sends OK / WARN / FAULT every few seconds so the Pico can show states only the host knows (warning, detached sensor). Silence from the host itself becomes a "host not responding" signal. The buzzer and LED code must never block or crash the hardware cut-off loop.
• Remote history. Write both temperatures to node-exporter's textfile collector, for Grafana history and alerting.
• 16-pin voltage via NVML, if Linux exposes it (see above).
• PSU-end sensor on GP28, if the accepted gap is revisited.

Known limits#

• Heat lag. The sensor reads the outside of the wire insulation, which lags the terminal inside. The rate-of-rise and baseline-relative thresholds reduce but don't remove this for sudden faults (see above).
• The PSU end is not watched. Only the IR checks cover it.
• Detached-sensor detection needs load. It only works while the GPU is under load, and takes 5 minutes to alert.
• Rate of rise needs steady power. It only arms after 5 minutes of steady GPU power, so the normal warm-up after load starts can't trigger it. A fault that begins during a load change is left to the absolute thresholds.
• The Pico needs USB power. The hardware cut-off needs the Pico powered. It is powered whenever the PC is running, which is when protection is needed.
• Tested so far: the code is bench-simulated only.
  • Temperature math: round-trip checked.
  • Host logic: exercised with the system commands stubbed. Warning, power-off, sensor-fault, airflow and detached-sensor alerts all fire. Rate of rise catches a 6 °C/min climb at steady power at 69 °C, and ignores both a normal warm-up and a climb after a jump in GPU power.
  • Not yet run: no real Pico, sensor or node. Commissioning is the first real test.