Thermal Perception: honest about the sensor gap

Picture a fixed camera watching a loading dock at 2 a.m., or a corridor filling with smoke during an incident, or a yard sitting under thick fog. A visible-light camera is nearly useless in all three cases: there is not enough reflected light for it to form an image, and smoke scatters what little light exists. A thermal (infrared) sensor keeps working, because it is not imaging reflected light at all, it is imaging emitted heat. A person, a running engine, a fire, all keep radiating in the long-wave infrared band regardless of ambient light. That is the entire case for thermal perception: it is the only sensor modality that stays useful exactly when darkness, smoke, or fog take the others offline.

The cost is that thermal sensors are not "RGB but darker." A microbolometer's raw output is a 16-bit radiometric count array, and a real scene at night typically fills only a narrow slice of that sensor's full dynamic range, our own synthetic generator's default scene spans roughly 14,000 to 43,000 counts out of a 16-bit sensor's full 0 to 65,535 span. Which normalization squeezes that slice into a model's expected 0 to 1 input, naive global min-max stretch, a percentile clip, plateau-limited histogram equalization, tiled CLAHE, is not a cosmetic preprocessing step. It decides how much real scene contrast a downstream model ever sees, and whether two different camera units even produce visually comparable frames for the same scene. The section below, "How it works," shows the actual pixel-level difference this choice makes on one frame.

Layered on top of that is a data problem, not just a preprocessing problem: there is vastly less labeled thermal training data in the world than labeled RGB training data, by orders of magnitude. The two largest thermal-specific public annotation sets we could locate without an interactive registration gate, FLIR ADAS v2 and KAIST Multispectral Pedestrian, together offer roughly 110,000 to 120,000 usable real thermal frames, itself still short of what our own pretraining recipe's floor requires (see "Measured results" below). The practical consequence for a buyer: a team under deadline pressure often takes an RGB-pretrained perception model and simply runs it on thermal frames after some ad hoc contrast stretch. That model was never shown thermal statistics during pretraining, has no reason to have learned what a genuine thermal hot spot looks like versus a normalization artifact, and can produce confident, wrong detections that look plausible to an operator who has no easy way to tell the difference. This product exists to close that specific gap honestly, by building the normalization-aware, thermal-native pretraining groundwork first, and by refusing to claim a trained model exists before one actually does.

The failure mode is not exotic, it is mundane and therefore easy to miss in a demo. An RGB-pretrained backbone has never seen a frame where absolute pixel brightness correlates with which physical camera captured it rather than with scene content, because RGB cameras are broadly white-balanced and exposure-corrected before a model ever sees them. Thermal sensors are not: two cameras of the same model, on the same wall, at the same moment, can report meaningfully different absolute counts for the same physical temperature because of gain drift, ambient calibration offset, and column fixed-pattern noise. A model that was never trained to be indifferent to that will quietly fold "which camera" into what it thinks is "how hot," and there is no way for an operator watching a dashboard to see that failure mode from the outside, it looks exactly like a normal detection until the numbers are checked against ground truth. That is precisely why this product's research half spends its effort on per-patch target normalization and cross-sensor quantile matching before it spends a single GPU-hour on a backbone: the fix has to happen in the data and pretraining objective, not be patched in afterward.

thermalcore is two halves of one product. The perception engine is the shipped half: a standalone, market-agnostic engine that takes any detector's outputs (boxes, tracks, classes, as JSON, schema tp/1) and returns thermal geometry metrics, native thermal detections, and plain-language explanations, no GPU and no training required to run it. The research half is the complete non-GPU groundwork for pretraining a thermal-native backbone: the radiometric data engine, the self-supervised (masked autoencoding) pretraining harness with its math CPU-verified, and the evaluation protocols the eventual backbone and depth models will be judged by. It is explicitly not a trained model: there is no checkpoint in this repository and no claim that one exists. The GPU run is emitted as a costed recipe (thermalcore emit-recipe), the exact training configuration a torch loop gets written against once compute is connected, not shipped as weights.

Perception engine input and output

Input is one JSON envelope, schema tp/1, from any detector: image dimensions, camera mount height and pitch, a radiometry mode, one or more frames, and a list of detections (box, class, confidence, track id, timestamp). Output is one tp/1 artifact with five channels, each degrading gracefully when its inputs are missing:

• geometry (pinhole plus flat-ground model): each box's foot point becomes a metric ground position, range, slant range, azimuth and elevation, and real-world width and height via foot-point depth. Feet at or above the horizon are flagged, not silently solved; implausible sizes warn about calibration instead of being reported as fact.
• tracks: timestamped ground fixes per track id become path length, displacement, average and max speed, heading, and stationarity; missing timestamps degrade to counts plus a warning, speeds over 250 km/h warn about identity switches.
• thermal: raw radiometric matrices through a radiometry spec (counts-linear, kelvin, celsius) into per-box min/max/mean/p95 temperature, contrast against a background ring, and hot-pixel fraction.
• native_detections: the zero-training radiometric detector, absolute or percentile threshold, union-find connected-component labeling, core-area filtering (so a hot-pixel defect cannot dilate its way past the area test), then soft-falloff box expansion. This is the detection the engine ships today; a learned detector is a future recipe, not a present claim.
• explanations: every detection and track described in plain English. Deterministic templates by default; an optional --explainer ollama mode rewrites the same facts with a local qwen3.5:2b-q4_K_M model via Ollama's native /api/chat endpoint with think: false, falling back to the template text on any failure, the artifact's own engine field discloses which one produced a given explanation.

Research half: the pretraining groundwork

• synth, a physics-flavored radiometric scene generator: warm bodies (people around 305 K, vehicle and engine blobs around 310 to 330 K) with soft elliptical falloff over a cool background with slow spatial gradients, plus the sensor artifacts a real microbolometer adds, temporal NETD noise, per-column fixed-pattern noise, hot-pixel defects. Every frame ships with ground-truth boxes and a true temperature map, so every downstream module is tested against physics, not against another model's opinion.
• agc, the normalization zoo: minmax, percentile, plateau (the FLIR-style default), and tile_clahe, all pure numpy, all returning NaN-free float32 in [0, 1]. The corpus manifest records which normalizer built each frame, because the choice is a corpus-level hyperparameter, not a preprocessing detail that can be swapped after the fact.
• sensor, cross-sensor domain tooling: a sensor profile tracks a camera's robust count quantiles online and its column fixed-pattern-noise signature, then builds a monotone piecewise-linear map from one sensor's quantiles onto a reference sensor's, so the same scene lands in the same count range regardless of which fleet camera saw it. The measured test for this closes a simulated gain-plus-offset sensor gap to under 15% of its original size.
• corpus, the pretraining dataset engine: a coarse average-hash dedup on the percentile-normalized frame drops near-duplicate temporal neighbors while keeping genuinely new content, frames are tagged by activity bucket (empty, sparse, busy) so training can pull rare busy minutes at parity with empty-scene hours, and everything lands in a JSONL manifest that is itself the dataset definition.
• ssl, the masked-autoencoding (MAE) pretraining harness, math only: patchify/unpatchify round-trip exactly, random masking hides exactly round(mask_ratio * n_patches) patches by construction (with an optional activity-weighted variant that biases which patches stay visible toward warm structure), per-patch target normalization stops the backbone from learning absolute sensor brightness as content, and a masked-reconstruction loss scores only the hidden patches. PretrainRecipe.min_corpus_frames = 200,000 is a hard-coded floor: below it, the documented, honest move is distilling from an RGB teacher instead of pretraining a thermal backbone from scratch.
• evalx, the evaluation protocols fixed before the models exist: a closed-form ridge linear probe on frozen features (no sklearn or torch dependency, so the protocol itself is auditable), field-standard depth metrics (AbsRel, RMSE, delta-under-1.25^n) with explicit median-scaling control, and a deliberately simple single-threshold class-agnostic precision/recall detector scorer for COCO-format thermal annotations.
• cli, three commands: build-corpus (index raw frames into a deduped, bucketed corpus), emit-recipe (write the GPU-phase pretraining recipe JSON for a corpus), and selfcheck (the "is this install sane" command used throughout this page).

Integrating a real detector

The engine has exactly one input coupling: the tp/1 JSON envelope. If a detector can emit boxes, it can feed this engine, no SDK, no imports, no platform dependency. The most common integration is an Ultralytics-style (YOLO family) detector: anything that yields pixel boxes plus class names and confidences converts directly.

The camera dict is the one piece of information a detector does not have on its own, it describes the mount, not the model: image width and height, horizontal field of view from the lens datasheet, mount height in meters from a tape measure, and pitch in degrees from an inclinometer or an installer app. If a deployment also captures raw thermal matrices (16-bit counts, kelvin, or celsius, in .npy, .csv, or .pgm), the same envelope references them per frame and the engine additionally returns per-box temperature stats and its own native hotspot detections. With no detector at all, the native hotspot pass needs nothing but a frame:

The diagram below traces one frame from raw sensor output to a disclosed result on this page. Scroll horizontally on narrow screens; the boxes and arrows do not reflow, they stay legible and you pan across them.

Two hands-on pieces below. The first shows the exact pixel-level effect of the AGC choice described above on one real synthetic frame from this session. The second is a browser-only toy that visualizes the masking algorithm the pretraining harness uses, with no model behind it.

The self-check confirms all four AGC methods in agc.py run correctly; this section shows what they actually do to a frame and why the choice is not cosmetic. Same synthetic scene (seed 2, 4 warm bodies, 1 vehicle, ambient 285 K, the repository's default 4 hot pixels), run this session through two of the four normalizers with no other change:

"99 tests, all green, no GPU required" is easy to say and easy to leave vague. Here is the actual breakdown by file, collected fresh this session via pytest tests/ --collect-only -q, so the number is not just a total to trust, it is a map of what is actually under test:

thermalcore.cli demo --outdir demo_out --seed 0 is the product demo and its own referee: it synthesizes ground truth, plays "any detector" by serializing those ground-truth boxes as a tp/1 envelope, adds a true-projected walking track (foot and head each projected through the full camera model, so the recovered numbers are a genuine round trip, not hand-authored pixels), runs the full engine, then scores the engine's own hotspot pass against ground truth. Rerun fresh on 2026-07-09, this session, on a plain Mac CPU:

No GPU was available in this session, so the numbers in this subsection are reported as committed document statements from TRAINING_RESULTS.md and REAL_DATA_RESULTS.md in the private codebase, not reproduced here. Two short pilots of scripts/train_mae.py and scripts/train_mae_real.py ran on a rented RTX 5060 Ti: one on the repository's own synthetic generator, one on a real public infrared dataset (LLVIP). Both are explicitly labeled training-mechanics checks, not evidence of learned-feature quality, and both sit far below the pretraining corpus floor.

The LLVIP pilot used huggingface_hub.hf_hub_download to fetch a public, ungated mirror of the dataset's infrared half only (12,025 train plus 3,463 test JPEGs, 15,488 total, 1280x1024, near-grayscale LWIR frames), after three other candidate sources were checked and rejected (one turned out to be visible-light images only, one was a 551-pair sample far too small, one at ~4,200 pairs was still under the requested band). FLIR ADAS v2 and KAIST Multispectral were both confirmed login/registration gated and excluded per the task's own instruction to avoid them.

The repository's perceptual-hash dedup exists to drop literal duplicate frames from a continuous fixed-camera stream. On a 600-frame LLVIP sample, it was measured directly before the decision was made:

LLVIP (Jia, Chen, Zhu, Zheng, Wu, "LLVIP: A Visible-infrared Paired Dataset for Low-light Vision," ICCV 2021 Workshops) is a real visible/infrared paired pedestrian dataset captured on real streets and campus in low light, 15,488 genuine LWIR frames used here (infrared half only). This is the complete, un-truncated masked-patch MSE loss log from that pilot, every point recorded in REAL_DATA_RESULTS.md's loss trajectory table, not start and end numbers alone. Hover or tap a point in the chart for its exact step, elapsed time, and learning rate.

The first pilot, run before the LLVIP pilot, used the repository's own synthetic generator because FLIR ADAS v2 and KAIST Multispectral were both registration-gated. Its loss curve, in full, for direct comparison against the real-data curve above:

The recipe the two pilots above trained a small slice of is fully specified in src/thermalcore/ssl.py's PretrainRecipe dataclass: vit_small_patch16 architecture, 224x224 input, patch size 16 (196 patches per image), mask ratio 0.75 (only 49 of 196 patches reach the encoder), activity-weighted masking on, per-patch target normalization on, plateau normalization baked into corpus selection, 400 epochs, batch size 1,024, base LR 1.5e-4, 20-epoch warmup, weight decay 0.05, and the min_corpus_frames = 200,000 floor this whole page keeps returning to. docs/COMPUTE_BUDGET.md concretizes that recipe into an actual compute ask, arithmetic derived from the recipe's own tested defaults, not a guess:

No published single-GPU throughput benchmark exists for MAE at ViT-S/16 with 75% masking, so the throughput estimate is derived, not measured: anchored to Meta's own published MAE/ViT-Large run (64 V100s, 42 hours, ImageNet-1k), scaled down to ViT-S/16 by GFLOPs ratio, then scaled from V100 to RTX 4090 by generational throughput improvement, landing at an estimated 1,700 to 4,200 images/sec on a single RTX 4090. This is flagged in the source document as a derived estimate, not a citation, to be corrected with a short timing pilot before committing a full job.

Every command below is the actual command that produced a number quoted somewhere on this page, not a simplified stand-in. CPU commands ran fresh this session; GPU commands are quoted as documented in the private codebase and were not rerun (no GPU was available in this session).

• The data floor is not met by any run to date. The repository's own PretrainRecipe.min_corpus_frames = 200,000 is a hard-coded threshold; the synthetic pilot used 2,503 frames (about 80x below the floor) and the real LLVIP pilot used 15,488 frames (about 13x below the floor). Even combining the two largest ungated-enough public thermal annotation sets we identified, FLIR ADAS v2 and KAIST Multispectral, yields only roughly 110,000 to 120,000 usable frames, still short of the floor. Both GPU pilots on record are explicitly labeled mechanics checks, not quality evidence, in their own source documents.
• No trained checkpoint exists, in this repository or anywhere on the Hugging Face Hub, for this product, on purpose. It would be easy to publish a thermal "foundation model" checkpoint and a headline accuracy number; we deliberately do not, because we have not trained one at scale. The GPU pretraining run is emitted as a recipe (thermalcore emit-recipe) with its compute costed line by line, not shipped as weights. There is no model repo on the Hub to point a buyer to, and none of the numbers on this page should be read as a stand-in for one.
• The perception engine's native detector (hotspot.py) is a zero-training threshold and connected-component method, not a learned model. Its precision/recall numbers on this page describe engine correctness against synthetic ground truth, not learned detection accuracy on real scenes.
• The only pilot training on genuine sensor data used a small (15,488-frame) public academic dataset (LLVIP), which is 8-bit AGC-processed JPEG, not the 16-bit radiometric format the rest of the pipeline is built for, and is not live sensor capture from a deployed camera. The "absolute counts encode sensor identity" framing behind per-patch target normalization applies less cleanly to data that has already been camera-ISP processed.
• All CPU-verified math (AGC, masking, corpus dedup, geometry, evaluation protocols) is tested against the repository's own synthetic generator, not against an independent real-world benchmark.
• The dedup measurement on a 600-frame LLVIP sample showed the repository's default perceptual-hash threshold keeps only 4.8% of frames on this kind of near-static night footage, and even exact-hash matching only keeps 43.7%; dedup was measured, then explicitly turned off for the LLVIP pilot rather than silently changed in the underlying code.
• LLVIP frames were resized to 256x256 before being written to the corpus, to keep the pilot's 15,488-frame corpus within a shared GPU box's disk budget (native 1280x1024 would have run roughly 40 GB); the trainer's own random-resized-crop to 224 still has real room to move at that stored resolution, but this is a disclosed storage compromise, not the dataset's native resolution.
• The actual hardware sensor decision for real-world capture is still open; nothing on this page reflects a chosen, deployed thermal camera model.
• Full-scale GPU pretraining, thermal-depth distillation, and edge (TensorRT/Jetson) latency numbers are future roadmap items. They are not present in this repository today, and no number on this page should be read as a preview of them.