The hottest thermal zone is not your CPU

A guard that stops work at 85 °C read max() over the /sys thermal zones. On this laptop that is a chassis probe running up to 26 °C above the die — and it killed 8 of 14 overnight QA runs while the CPU sat in the fifties.

November 13, 2026 Rebel Studios Software
A terminal panel showing max over the sys thermal zones returning 78000 for acpitz, the chassis sensor the guard read, against 52000 for x86_pkg_temp, the actual CPU die, at the same instant on an idle laptop: 26 degrees apart.

Our WordPress plugins get a full end-to-end sweep every night at 03:00 — build, a fresh WordPress, fixtures, admin screens, licence activation, Plugin Check, then browser flows. It runs on a laptop, unattended, so it is wrapped in a temperature guard that tears everything down if the machine gets hot.

This is the line I kept finding in the morning log:

2026-11-13 03:06:09 !! 85C >= 85C -- stopping + tearing down (partial results above)

Eight of the last fourteen runs ended that way. The browser flows — the part that actually drives a checkout — had run on five of fourteen nights. And every one of those torn-down runs was recorded by systemd as Result=success, so nothing ever complained.

The number that gave it away

Before touching the guard I read what the logs said about each run, which is where it stopped being a heat problem:

RunStarted atTripped atElapsed
11-0665 °C87 °C30 seconds
11-1066 °C85 °C60 seconds
11-0963 °C88 °C3 minutes

Thirty seconds in, the suite is still pulling Docker images. It is also running under nice -n 15. That workload does not add 22 °C to a CPU die in half a minute.

And then the line directly underneath the 11-06 teardown:

03:00:30 !! 87C >= 85C -- stopping + tearing down
03:00:45 === done at 68C -- results above

Nineteen degrees, in fifteen seconds, on a passively cooled laptop with the fans already spun up. Silicon does cool quickly, but not like that, and not while the machine is still tearing down containers. Two readings fifteen seconds apart were not describing the same thing.

One line, and the word that hid it

Here is the whole guard:

pkgtemp(){ cat /sys/class/thermal/thermal_zone*/temp 2>/dev/null \
             | sort -rn | head -1 | awk '{print int($1/1000)}'; }

It is called pkgtemp, which is what everyone reading it assumed it returned. What it actually does is read every thermal zone the kernel exposes and return the highest. Those are not the same number, and on this machine they are not even close.

This box has six zones. Sampled three seconds apart, at idle:

acpitz=72C  TCPU=55C  pkg=55C   load=7.02
acpitz=78C  TCPU=55C  pkg=56C   load=6.78
acpitz=78C  TCPU=54C  pkg=52C   load=6.64

acpitz climbed six degrees while the die fell three. It is not a CPU sensor with an offset you could subtract — it moves independently. Later the same morning, under real load, the two swapped places entirely: acpitz read 59 °C while the die hit 71 °C.

That independence is the reason the failures looked random. The guard did not fire when the machine got hot. It fired whenever acpitz happened to be drifting high, which has very little to do with what the suite was doing.

The zones tell you this themselves

Every thermal zone publishes its own trip points, and they are worth reading before you pick a ceiling:

acpitz        active 70C   hot 99C    critical 120C
TCPU          active 103C  passive 110C  critical 110C
x86_pkg_temp  (no trips; Tjmax on this part is ~100C)

A chassis zone whose firmware marks 70 °C as the point to spin a fan is expected to sit in the seventies. Its own "hot" mark is 99 °C. So 85 on that zone is unremarkable, while 85 on the die is a real if conservative ceiling. One number, two completely different meanings, decided by which file you happened to read.

For completeness I checked whether the CPU had ever actually been throttling, since that is the thing a guard is there to pre-empt:

$ cat /sys/devices/system/cpu/cpu0/thermal_throttle/core_throttle_count   # t0
11653
$ cat /sys/devices/system/cpu/cpu0/thermal_throttle/core_throttle_count   # t+8s
11653

No movement. The hardware was not in difficulty.

The fix is a refusal, not a lookup

Resolving the right zone is three lines. The part that matters is what happens when it is not there:

ZONE=""
for z in /sys/class/thermal/thermal_zone*; do
  case "$(cat "$z/type" 2>/dev/null)" in
    x86_pkg_temp) ZONE="$z"; break ;;
    TCPU)         [ -z "$ZONE" ] && ZONE="$z" ;;
  esac
done
[ -n "$ZONE" ] || { log "no CPU thermal zone found -- refusing to run blind"; exit 2; }

The original bug was not really that the guard picked the wrong zone. It was that a wrong zone was available, plausible, and silently acceptable. sort -rn | head -1 always returns something, so the guard could never report that it had no idea what it was measuring. Code that cannot fail to find an answer will hand you a wrong one indefinitely.

I left the ceiling at 85. The temptation when a guard misfires is to raise the constant, and it would not even have been unreasonable here — 85 on the die leaves 15 °C before this part throttles itself. But the constant was never the problem, and tuning it would have buried the actual one.

The second bug, which is the worse one

The teardown path ended like this:

kill -TERM "$QA"; sleep 3; kill -KILL "$QA"; teardown; break

— and then fell through to the end of the script, which exited 0. So systemd recorded Result=success, the dashboard went green, and a run that had completed one of nine sections was indistinguishable from a clean pass. For eight nights nobody had any reason to look.

A thermal stop now exits 1. That one change is worth more than the sensor fix, because it is what would have surfaced the sensor fix eight nights earlier. Partial work that reports success is more expensive than work that fails loudly: failure costs you a run, and a false pass costs you every run after it.

With both fixed, the next run reached the browser flows and got through seven of them instead of one, with the die sitting at 71 °C against an 85 °C ceiling.

What I don’t know

Which zone comes out highest is board-specific. On a machine whose hottest zone really is the package, this guard would have worked correctly for years, and the bug would only appear when the script moved. acpitz is an ACPI thermal zone whose meaning is whatever the firmware decided — probably a skin or VRM probe here, but I did not pull the DSDT apart to confirm it, because for the purpose of fixing this it only had to be not the CPU. And I have not tested the guard against a genuinely hot machine since the change; the honest claim is that it now reads the die, not that I have watched it stop a real overheat.

← Back to Blog