RadiaCode-Spectrometer/tools/xml_to_replay_csv.py
vhadmin 7e0a78337f Exclude the detector's overflow channel from the likelihood
The RC-110's top channel is an overflow bin: every event above the last
calibrated energy is counted there, so it is not spectral content at that
energy. It shows up as a single-channel spike around a hundred times its
neighbours - 186 counts of 91224 in the indoor background capture, and 618 of
81225 (0.76% of all counts, in one channel) in a new 2.5 h forest background.

No template can reproduce a one-channel spike, so leaving it in the Poisson
likelihood does not invent a nuclide - it hides them. The only way the fit can
cover the spike is by lifting the topmost continuum bump, and that raised
continuum then absorbs genuine counts across the whole high-energy region.

Measured, toggling FIT_OVERFLOW_CHANNELS 0/1 over the real captures:

  capture       residual        reduced chi2    overflow bin
  background    6.8 -> 6.4%     2.18 -> 1.69    0.20% of counts
  forest        9.7 -> 7.2%     3.72 -> 1.28    0.76% of counts
  thorium      11.7 -> 11.5%    1.94 -> 1.84    -
  uranium       8.7 -> 8.5%     2.01 -> 1.84    -

Goodness-of-fit improves on all four and by far the most on the capture with
the largest artefact, which is what ties the effect to this cause. Verdicts
and time-to-identification are unchanged: background 900 s, thorium 120 s,
uranium 300 s, forest 3600 s.

Significance moves modestly in both directions - K-40 rises 3.7 -> 4.6 sigma
on the forest capture but falls 6.8 -> 6.2 on the indoor one. The change is
justified by the artefact being an artefact and by the fit becoming
well-specified (chi2 1.3-1.8 where it was 1.9-3.7), not by larger numbers.
The post-fix figures are the trustworthy ones precisely because the model now
fits.

The peak search already skipped this channel; the template fit, which is what
actually decides identification now, did not.

Also adds:
- host/spectra/forest.csv - a real 9002 s / 81225 count outdoor background
  from a forest, 9.0 cps against the indoor capture's 21.7, now a permanent
  regression case for the low-rate end.
- tools/xml_to_replay_csv.py - converts a RadiaCode XML export into the
  replay harness format. tools/radiacode_to_background.py writes
  counts/second references for the device's background library, which replay
  cannot consume, so there was no way to get an XML export under test.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-10-04 12:04:21 +02:00

98 lines
3.2 KiB
Python

#!/usr/bin/env python3
"""Convert a RadiaCode XML export into the replay-harness CSV format.
The harness reads what SdStorage::exportSpectrum() writes: duration and
calibration in header comments, then absolute counts per channel. The existing
tools/radiacode_to_background.py writes counts/SECOND references for the
device's background library instead, which replay cannot consume.
python xml2replay.py in.xml out.csv [--zero-last] [--dose D]
--zero-last clears the final channel, which on the RC-110 is an overflow bin
holding every event above the last calibrated energy rather than real spectral
content at that energy.
"""
import re
import sys
import xml.etree.ElementTree as ET
def strip_ns(tag):
return tag.split('}', 1)[-1]
def find_first(root, name):
for el in root.iter():
if strip_ns(el.tag) == name:
return el
return None
def main():
args = [a for a in sys.argv[1:]]
zero_last = '--zero-last' in args
if zero_last:
args.remove('--zero-last')
dose = None
if '--dose' in args:
i = args.index('--dose')
dose = float(args[i + 1])
del args[i:i + 2]
src, dst = args[0], args[1]
root = ET.parse(src).getroot()
spec = find_first(root, 'Spectrum')
counts = [int(float(dp.text)) for dp in spec
if strip_ns(dp.tag) == 'DataPoint']
cal = find_first(root, 'EnergyCalibration')
coeffs = [float(c.text) for c in cal.iter()
if strip_ns(c.tag) == 'Coefficient']
while len(coeffs) < 3:
coeffs.append(0.0)
a0, a1, a2 = coeffs[:3]
mt = find_first(root, 'MeasurementTime')
duration = int(float(mt.text))
if len(counts) != 1024:
sys.exit('expected 1024 channels, got %d' % len(counts))
overflow = counts[-1]
if zero_last:
counts[-1] = 0
total = sum(counts)
cps = total / duration if duration else 0.0
# The RC-110's published sensitivity is ~77 cps per uSv/h for Cs-137; only
# used when the caller does not supply a measured dose.
if dose is None:
dose = cps / 77.0
with open(dst, 'w', newline='\n') as f:
f.write('# RadiaCode-110 spectrum export (from %s)\n' % src.replace('\\', '/').split('/')[-1])
f.write('# duration_s,%d\n' % duration)
f.write('# total_counts,%d\n' % total)
f.write('# cps,%.3f\n' % cps)
f.write('# dose_uSv_h,%.4f\n' % dose)
f.write('# calib_a0,%.6f\n' % a0)
f.write('# calib_a1,%.6f\n' % a1)
f.write('# calib_a2,%.8f\n' % a2)
f.write('# background_subtracted,0\n')
f.write('channel,energy_keV,counts\n')
for ch, c in enumerate(counts):
e = a0 + a1 * ch + a2 * ch * ch
f.write('%d,%.2f,%d\n' % (ch, e, c))
print('%s -> %s' % (src, dst))
print(' %d ch, %d s, %d counts, %.2f cps, dose %.4f uSv/h' %
(len(counts), duration, total, cps, dose))
print(' calib a0=%.4f a1=%.5f a2=%.7f' % (a0, a1, a2))
print(' last-channel overflow bin held %d counts (%.2f%% of total)%s' %
(overflow, 100.0 * overflow / max(total + (overflow if zero_last else 0), 1),
' - ZEROED' if zero_last else ''))
if __name__ == '__main__':
main()