## Highlights
- **Per-tensor INT8** activations + **per-channel INT8** weights via
embedl-deploy's PTQ flow on top of TensorRT's fused MHA kernel.
No QAT or distillation needed.
- **Drop-in replacement** for `amazon/chronos-2` inference: same
`(context, group_ids) → quantile_preds` signature; 21 evenly
spaced quantile levels with the median at index 10.
- **Validated** on the [GIFT-Eval](https://huggingface.co/datasets/Salesforce/GiftEval)
benchmark across 125 task configurations. See Accuracy below.
- **Two ctx variants** so you can pick the latency/history-window
trade-off that fits your deployment.
## Quick Start
```bash
pip install tensorrt pycuda numpy
python infer_trt.py --ctx 512 # 1.2× faster than FP16 on Orin
python infer_trt.py --ctx 2048 # 1.3× faster than FP16 on Orin
```
The `infer_trt.py` helper script builds a TensorRT engine from the
ONNX on first run (cached as `*.engine` next to the artifact) and
feeds a synthetic seasonal context for demonstration. Replace the
context generator with your own series of the right length.
## Files
| File | Purpose |
|---|---|
| `embedl_chronos_2_ctx512_int8.onnx` | INT8 ONNX with Q/DQ — ctx=512, 1024-step horizon. |
| `embedl_chronos_2_ctx2048_int8.onnx` | INT8 ONNX with Q/DQ — ctx=2048, 1024-step horizon. |
| `infer_trt.py` | ONNX Runtime / TensorRT inference example. |
Both artifacts emit a `(1, 21, 1024)` quantile tensor (21 quantile
levels × 64 output patches × 16 steps-per-patch = 1024 horizon
steps). Slice the median (`preds[0, 10]`) for a point forecast and
clip to your needed prediction length.
## Performance
Latency measured with TensorRT + `trtexec`, GPU compute time only
(`--noDataTransfers`), CUDA Graph + Spin Wait enabled, clocks locked
(`nvpmodel -m 0 && jetson_clocks` on Jetson).
### Jetson AGX Orin (MAXN)
#### ctx=512
