2026-04-16 19:52:41,835 INFO use char tokenizer
2026-04-16 19:52:41,835 INFO training on multiple gpus, this gpu 0, rank 0, world_size 1
2026-04-16 19:52:42,250 INFO [Rank 0] Checkpoint: save to checkpoint exp/conformer_mongolian/init.pt
2026-04-16 19:52:42,365 INFO Epoch 0 Step 0 TRAIN info lr 8.0000e-08 rank 0
2026-04-16 19:52:42,383 INFO using accumulate grad, new batch size is 1 times larger than before

Epoch 0 TRAIN: 0it [00:00, ?it/s]/home/rookie/Documents/batuka/projects/wenet/wenet/wenet/utils/train_utils.py:689: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  amp_autocast(enabled=dtype is not None,
/home/rookie/Documents/batuka/projects/wenet/wenet/wenet/utils/train_utils.py:693: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  amp_autocast(enabled=scaler is not None),
[rank0]:[W416 19:52:47.411323494 reducer.cpp:1400] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())

Epoch 0 TRAIN: 0it [00:05, ?it/s, epoch=0, grad_norm=926.298, loss=638.1240, lr=8.00e-08]
Epoch 0 TRAIN: 1it [00:05,  5.26s/it, epoch=0, grad_norm=926.298, loss=638.1240, lr=8.00e-08]
Epoch 0 TRAIN: 1it [00:05,  5.26s/it, epoch=0, grad_norm=992.981, loss=637.5255, lr=1.60e-07]
Epoch 0 TRAIN: 1it [00:05,  5.26s/it, epoch=0, grad_norm=977.971, loss=683.0901, lr=2.40e-07]
Epoch 0 TRAIN: 1it [00:05,  5.26s/it, epoch=0, grad_norm=996.669, loss=655.2043, lr=3.20e-07]
Epoch 0 TRAIN: 1it [00:05,  5.26s/it, epoch=0, grad_norm=1068.044, loss=639.4346, lr=4.00e-07]
Epoch 0 TRAIN: 5it [00:05,  1.12it/s, epoch=0, grad_norm=1068.044, loss=639.4346, lr=4.00e-07]
Epoch 0 TRAIN: 5it [00:05,  1.12it/s, epoch=0, grad_norm=906.760, loss=605.3057, lr=4.80e-07] 
Epoch 0 TRAIN: 5it [00:06,  1.12it/s, epoch=0, grad_norm=1022.865, loss=611.7023, lr=5.60e-07]
Epoch 0 TRAIN: 5it [00:06,  1.12it/s, epoch=0, grad_norm=910.587, loss=681.9362, lr=6.40e-07] 
Epoch 0 TRAIN: 5it [00:06,  1.12it/s, epoch=0, grad_norm=946.675, loss=652.0659, lr=7.20e-07]
Epoch 0 TRAIN: 9it [00:06,  2.09it/s, epoch=0, grad_norm=946.675, loss=652.0659, lr=7.20e-07]
Epoch 0 TRAIN: 9it [00:06,  2.09it/s, epoch=0, grad_norm=1130.374, loss=706.6662, lr=8.00e-07]
Epoch 0 TRAIN: 9it [00:06,  2.09it/s, epoch=0, grad_norm=998.064, loss=650.3630, lr=8.80e-07] 
Epoch 0 TRAIN: 9it [00:06,  2.09it/s, epoch=0, grad_norm=988.106, loss=646.7837, lr=9.60e-07]
Epoch 0 TRAIN: 9it [00:06,  2.09it/s, epoch=0, grad_norm=1193.629, loss=648.5864, lr=1.04e-06]
Epoch 0 TRAIN: 13it [00:06,  3.09it/s, epoch=0, grad_norm=1193.629, loss=648.5864, lr=1.04e-06]
Epoch 0 TRAIN: 13it [00:06,  3.09it/s, epoch=0, grad_norm=1023.404, loss=671.2255, lr=1.12e-06]
Epoch 0 TRAIN: 13it [00:07,  3.09it/s, epoch=0, grad_norm=1263.773, loss=661.2456, lr=1.20e-06]
Epoch 0 TRAIN: 13it [00:07,  3.09it/s, epoch=0, grad_norm=1190.967, loss=657.4905, lr=1.28e-06]
Epoch 0 TRAIN: 13it [00:07,  3.09it/s, epoch=0, grad_norm=1121.275, loss=612.0612, lr=1.36e-06]
Epoch 0 TRAIN: 17it [00:07,  3.99it/s, epoch=0, grad_norm=1121.275, loss=612.0612, lr=1.36e-06]
Epoch 0 TRAIN: 17it [00:07,  3.99it/s, epoch=0, grad_norm=1514.131, loss=665.5478, lr=1.44e-06]
Epoch 0 TRAIN: 17it [00:07,  3.99it/s, epoch=0, grad_norm=1602.619, loss=619.3444, lr=1.52e-06]
Epoch 0 TRAIN: 17it [00:07,  3.99it/s, epoch=0, grad_norm=1565.864, loss=630.3460, lr=1.60e-06]
Epoch 0 TRAIN: 17it [00:07,  3.99it/s, epoch=0, grad_norm=1720.920, loss=621.9006, lr=1.68e-06]
Epoch 0 TRAIN: 21it [00:07,  4.79it/s, epoch=0, grad_norm=1720.920, loss=621.9006, lr=1.68e-06]
Epoch 0 TRAIN: 21it [00:08,  4.79it/s, epoch=0, grad_norm=1584.154, loss=680.7437, lr=1.76e-06]
Epoch 0 TRAIN: 21it [00:08,  4.79it/s, epoch=0, grad_norm=1658.726, loss=684.2361, lr=1.84e-06]
Epoch 0 TRAIN: 21it [00:08,  4.79it/s, epoch=0, grad_norm=1586.082, loss=643.9550, lr=1.92e-06]
Epoch 0 TRAIN: 21it [00:08,  4.79it/s, epoch=0, grad_norm=1671.008, loss=637.3904, lr=2.00e-06]
Epoch 0 TRAIN: 25it [00:08,  5.47it/s, epoch=0, grad_norm=1671.008, loss=637.3904, lr=2.00e-06]
Epoch 0 TRAIN: 25it [00:08,  5.47it/s, epoch=0, grad_norm=2115.991, loss=641.5280, lr=2.08e-06]
Epoch 0 TRAIN: 25it [00:08,  5.47it/s, epoch=0, grad_norm=1686.839, loss=672.2903, lr=2.16e-06]
Epoch 0 TRAIN: 25it [00:08,  5.47it/s, epoch=0, grad_norm=1857.319, loss=651.2874, lr=2.24e-06]
Epoch 0 TRAIN: 25it [00:08,  5.47it/s, epoch=0, grad_norm=2173.224, loss=631.9431, lr=2.32e-06]
Epoch 0 TRAIN: 29it [00:08,  6.04it/s, epoch=0, grad_norm=2173.224, loss=631.9431, lr=2.32e-06]
Epoch 0 TRAIN: 29it [00:09,  6.04it/s, epoch=0, grad_norm=2082.918, loss=631.9880, lr=2.40e-06]
Epoch 0 TRAIN: 29it [00:09,  6.04it/s, epoch=0, grad_norm=2223.663, loss=601.3704, lr=2.48e-06]
Epoch 0 TRAIN: 29it [00:09,  6.04it/s, epoch=0, grad_norm=1928.032, loss=612.6840, lr=2.56e-06]
Epoch 0 TRAIN: 29it [00:09,  6.04it/s, epoch=0, grad_norm=2489.374, loss=603.3690, lr=2.64e-06]
Epoch 0 TRAIN: 33it [00:09,  6.46it/s, epoch=0, grad_norm=2489.374, loss=603.3690, lr=2.64e-06]
Epoch 0 TRAIN: 33it [00:09,  6.46it/s, epoch=0, grad_norm=2687.037, loss=605.9614, lr=2.72e-06]
Epoch 0 TRAIN: 33it [00:09,  6.46it/s, epoch=0, grad_norm=2411.289, loss=642.3756, lr=2.80e-06]
Epoch 0 TRAIN: 33it [00:09,  6.46it/s, epoch=0, grad_norm=2224.177, loss=597.1509, lr=2.88e-06]
Epoch 0 TRAIN: 33it [00:10,  6.46it/s, epoch=0, grad_norm=2423.902, loss=585.6821, lr=2.96e-06]
Epoch 0 TRAIN: 37it [00:10,  6.75it/s, epoch=0, grad_norm=2423.902, loss=585.6821, lr=2.96e-06]
Epoch 0 TRAIN: 37it [00:10,  6.75it/s, epoch=0, grad_norm=2116.568, loss=532.5958, lr=3.04e-06]
Epoch 0 TRAIN: 37it [00:10,  6.75it/s, epoch=0, grad_norm=2429.042, loss=552.8181, lr=3.12e-06]
Epoch 0 TRAIN: 37it [00:10,  6.75it/s, epoch=0, grad_norm=2214.772, loss=530.7015, lr=3.20e-06]
Epoch 0 TRAIN: 37it [00:10,  6.75it/s, epoch=0, grad_norm=1900.028, loss=604.6144, lr=3.28e-06]
Epoch 0 TRAIN: 41it [00:10,  6.96it/s, epoch=0, grad_norm=1900.028, loss=604.6144, lr=3.28e-06]
Epoch 0 TRAIN: 41it [00:10,  6.96it/s, epoch=0, grad_norm=1978.599, loss=575.6720, lr=3.36e-06]
Epoch 0 TRAIN: 41it [00:10,  6.96it/s, epoch=0, grad_norm=2026.278, loss=515.3434, lr=3.44e-06]
Epoch 0 TRAIN: 41it [00:10,  6.96it/s, epoch=0, grad_norm=1779.731, loss=504.6479, lr=3.52e-06]
Epoch 0 TRAIN: 41it [00:11,  6.96it/s, epoch=0, grad_norm=1719.675, loss=496.8148, lr=3.60e-06]
Epoch 0 TRAIN: 45it [00:11,  7.14it/s, epoch=0, grad_norm=1719.675, loss=496.8148, lr=3.60e-06]
Epoch 0 TRAIN: 45it [00:11,  7.14it/s, epoch=0, grad_norm=1726.525, loss=430.4977, lr=3.68e-06]
Epoch 0 TRAIN: 45it [00:11,  7.14it/s, epoch=0, grad_norm=1488.317, loss=516.7393, lr=3.76e-06]
Epoch 0 TRAIN: 45it [00:11,  7.14it/s, epoch=0, grad_norm=1077.220, loss=495.2209, lr=3.84e-06]
Epoch 0 TRAIN: 45it [00:11,  7.14it/s, epoch=0, grad_norm=1130.226, loss=480.8752, lr=3.92e-06]
Epoch 0 TRAIN: 49it [00:11,  7.18it/s, epoch=0, grad_norm=1130.226, loss=480.8752, lr=3.92e-06]
Epoch 0 TRAIN: 49it [00:11,  7.18it/s, epoch=0, grad_norm=968.026, loss=487.2144, lr=4.00e-06] 
Epoch 0 TRAIN: 49it [00:11,  7.18it/s, epoch=0, grad_norm=629.367, loss=492.3438, lr=4.08e-06]
Epoch 0 TRAIN: 49it [00:12,  7.18it/s, epoch=0, grad_norm=736.209, loss=479.8107, lr=4.16e-06]
Epoch 0 TRAIN: 49it [00:12,  7.18it/s, epoch=0, grad_norm=582.512, loss=486.5223, lr=4.24e-06]
Epoch 0 TRAIN: 53it [00:12,  7.34it/s, epoch=0, grad_norm=582.512, loss=486.5223, lr=4.24e-06]
Epoch 0 TRAIN: 53it [00:12,  7.34it/s, epoch=0, grad_norm=587.407, loss=479.9575, lr=4.32e-06]
Epoch 0 TRAIN: 53it [00:12,  7.34it/s, epoch=0, grad_norm=392.346, loss=515.6404, lr=4.40e-06]
Epoch 0 TRAIN: 53it [00:12,  7.34it/s, epoch=0, grad_norm=450.239, loss=464.7114, lr=4.48e-06]
Epoch 0 TRAIN: 53it [00:12,  7.34it/s, epoch=0, grad_norm=492.085, loss=468.1231, lr=4.56e-06]
Epoch 0 TRAIN: 57it [00:12,  7.42it/s, epoch=0, grad_norm=492.085, loss=468.1231, lr=4.56e-06]
Epoch 0 TRAIN: 57it [00:12,  7.42it/s, epoch=0, grad_norm=337.314, loss=381.2465, lr=4.64e-06]
Epoch 0 TRAIN: 57it [00:12,  7.42it/s, epoch=0, grad_norm=402.237, loss=488.9800, lr=4.72e-06]
Epoch 0 TRAIN: 57it [00:13,  7.42it/s, epoch=0, grad_norm=396.961, loss=418.4325, lr=4.80e-06]
Epoch 0 TRAIN: 57it [00:13,  7.42it/s, epoch=0, grad_norm=329.547, loss=489.9693, lr=4.88e-06]
Epoch 0 TRAIN: 61it [00:13,  7.43it/s, epoch=0, grad_norm=329.547, loss=489.9693, lr=4.88e-06]
Epoch 0 TRAIN: 61it [00:13,  7.43it/s, epoch=0, grad_norm=343.415, loss=482.5835, lr=4.96e-06]
Epoch 0 TRAIN: 61it [00:13,  7.43it/s, epoch=0, grad_norm=362.787, loss=458.0842, lr=5.04e-06]
Epoch 0 TRAIN: 61it [00:13,  7.43it/s, epoch=0, grad_norm=476.676, loss=525.5262, lr=5.12e-06]
Epoch 0 TRAIN: 61it [00:13,  7.43it/s, epoch=0, grad_norm=212.825, loss=406.7229, lr=5.20e-06]
Epoch 0 TRAIN: 65it [00:13,  7.44it/s, epoch=0, grad_norm=212.825, loss=406.7229, lr=5.20e-06]
Epoch 0 TRAIN: 65it [00:13,  7.44it/s, epoch=0, grad_norm=297.438, loss=461.5645, lr=5.28e-06]
Epoch 0 TRAIN: 65it [00:14,  7.44it/s, epoch=0, grad_norm=237.144, loss=359.6244, lr=5.36e-06]
Epoch 0 TRAIN: 65it [00:14,  7.44it/s, epoch=0, grad_norm=267.661, loss=485.2396, lr=5.44e-06]
Epoch 0 TRAIN: 65it [00:14,  7.44it/s, epoch=0, grad_norm=399.563, loss=508.1697, lr=5.52e-06]
Epoch 0 TRAIN: 69it [00:14,  7.34it/s, epoch=0, grad_norm=399.563, loss=508.1697, lr=5.52e-06]
Epoch 0 TRAIN: 69it [00:14,  7.34it/s, epoch=0, grad_norm=354.242, loss=399.6199, lr=5.60e-06]
Epoch 0 TRAIN: 69it [00:14,  7.34it/s, epoch=0, grad_norm=324.806, loss=436.9661, lr=5.68e-06]
Epoch 0 TRAIN: 69it [00:14,  7.34it/s, epoch=0, grad_norm=283.265, loss=439.7288, lr=5.76e-06]
Epoch 0 TRAIN: 69it [00:14,  7.34it/s, epoch=0, grad_norm=702.109, loss=359.9575, lr=5.84e-06]
Epoch 0 TRAIN: 73it [00:14,  7.13it/s, epoch=0, grad_norm=702.109, loss=359.9575, lr=5.84e-06]
Epoch 0 TRAIN: 73it [00:15,  7.13it/s, epoch=0, grad_norm=604.575, loss=394.9626, lr=5.92e-06]
Epoch 0 TRAIN: 73it [00:15,  7.13it/s, epoch=0, grad_norm=251.583, loss=447.1705, lr=6.00e-06]
Epoch 0 TRAIN: 73it [00:15,  7.13it/s, epoch=0, grad_norm=272.663, loss=476.9738, lr=6.08e-06]
Epoch 0 TRAIN: 73it [00:15,  7.13it/s, epoch=0, grad_norm=369.053, loss=442.3559, lr=6.16e-06]
Epoch 0 TRAIN: 77it [00:15,  7.20it/s, epoch=0, grad_norm=369.053, loss=442.3559, lr=6.16e-06]
Epoch 0 TRAIN: 77it [00:15,  7.20it/s, epoch=0, grad_norm=177.709, loss=460.9892, lr=6.24e-06]
Epoch 0 TRAIN: 77it [00:15,  7.20it/s, epoch=0, grad_norm=311.088, loss=439.9316, lr=6.32e-06]
Epoch 0 TRAIN: 77it [00:15,  7.20it/s, epoch=0, grad_norm=228.235, loss=450.2321, lr=6.40e-06]
Epoch 0 TRAIN: 77it [00:15,  7.20it/s, epoch=0, grad_norm=227.997, loss=502.1472, lr=6.48e-06]
Epoch 0 TRAIN: 81it [00:16,  7.18it/s, epoch=0, grad_norm=227.997, loss=502.1472, lr=6.48e-06]
Epoch 0 TRAIN: 81it [00:16,  7.18it/s, epoch=0, grad_norm=571.883, loss=367.5209, lr=6.56e-06]
Epoch 0 TRAIN: 81it [00:16,  7.18it/s, epoch=0, grad_norm=246.524, loss=478.5785, lr=6.64e-06]
Epoch 0 TRAIN: 81it [00:16,  7.18it/s, epoch=0, grad_norm=285.590, loss=434.1410, lr=6.72e-06]
Epoch 0 TRAIN: 81it [00:16,  7.18it/s, epoch=0, grad_norm=244.235, loss=400.5124, lr=6.80e-06]
Epoch 0 TRAIN: 85it [00:16,  6.98it/s, epoch=0, grad_norm=244.235, loss=400.5124, lr=6.80e-06]
Epoch 0 TRAIN: 85it [00:16,  6.98it/s, epoch=0, grad_norm=197.852, loss=438.8212, lr=6.88e-06]
Epoch 0 TRAIN: 85it [00:16,  6.98it/s, epoch=0, grad_norm=189.905, loss=444.6791, lr=6.96e-06]
Epoch 0 TRAIN: 85it [00:17,  6.98it/s, epoch=0, grad_norm=304.052, loss=374.5923, lr=7.04e-06]
Epoch 0 TRAIN: 85it [00:17,  6.98it/s, epoch=0, grad_norm=448.586, loss=354.1235, lr=7.12e-06]
Epoch 0 TRAIN: 89it [00:17,  6.98it/s, epoch=0, grad_norm=448.586, loss=354.1235, lr=7.12e-06]
Epoch 0 TRAIN: 89it [00:17,  6.98it/s, epoch=0, grad_norm=297.824, loss=409.8843, lr=7.20e-06]
Epoch 0 TRAIN: 89it [00:17,  6.98it/s, epoch=0, grad_norm=265.790, loss=353.2458, lr=7.28e-06]
Epoch 0 TRAIN: 89it [00:17,  6.98it/s, epoch=0, grad_norm=178.069, loss=377.3312, lr=7.36e-06]
Epoch 0 TRAIN: 89it [00:17,  6.98it/s, epoch=0, grad_norm=207.597, loss=354.4006, lr=7.44e-06]
Epoch 0 TRAIN: 93it [00:17,  7.10it/s, epoch=0, grad_norm=207.597, loss=354.4006, lr=7.44e-06]
Epoch 0 TRAIN: 93it [00:17,  7.10it/s, epoch=0, grad_norm=278.059, loss=423.6377, lr=7.52e-06]
Epoch 0 TRAIN: 93it [00:17,  7.10it/s, epoch=0, grad_norm=287.033, loss=346.7193, lr=7.60e-06]
Epoch 0 TRAIN: 93it [00:18,  7.10it/s, epoch=0, grad_norm=284.286, loss=435.8151, lr=7.68e-06]
Epoch 0 TRAIN: 93it [00:18,  7.10it/s, epoch=0, grad_norm=157.424, loss=347.6880, lr=7.76e-06]
Epoch 0 TRAIN: 97it [00:18,  7.08it/s, epoch=0, grad_norm=157.424, loss=347.6880, lr=7.76e-06]
Epoch 0 TRAIN: 97it [00:18,  7.08it/s, epoch=0, grad_norm=206.541, loss=439.0459, lr=7.84e-06]
Epoch 0 TRAIN: 97it [00:18,  7.08it/s, epoch=0, grad_norm=211.181, loss=312.6534, lr=7.92e-06]
Epoch 0 TRAIN: 97it [00:18,  7.08it/s, epoch=0, grad_norm=153.684, loss=381.5858, lr=8.00e-06]2026-04-16 19:53:01,132 DEBUG TRAIN | steps/sec 5.280| Batch 0/100 loss 381.585785 loss_att 371.019135 loss_ctc 406.241302 th_accuracy 0.051075 lr 8.0000e-06 grad_norm 153.683929 rank 0

Epoch 0 TRAIN: 97it [00:18,  7.08it/s, epoch=0, grad_norm=169.667, loss=400.3729, lr=8.08e-06]
Epoch 0 TRAIN: 101it [00:18,  6.90it/s, epoch=0, grad_norm=169.667, loss=400.3729, lr=8.08e-06]
Epoch 0 TRAIN: 101it [00:19,  6.90it/s, epoch=0, grad_norm=146.476, loss=371.9395, lr=8.16e-06]
Epoch 0 TRAIN: 101it [00:19,  6.90it/s, epoch=0, grad_norm=158.290, loss=357.2529, lr=8.24e-06]
Epoch 0 TRAIN: 101it [00:19,  6.90it/s, epoch=0, grad_norm=341.189, loss=533.7515, lr=8.32e-06]
Epoch 0 TRAIN: 101it [00:19,  6.90it/s, epoch=0, grad_norm=237.057, loss=491.7017, lr=8.40e-06]
Epoch 0 TRAIN: 105it [00:19,  7.02it/s, epoch=0, grad_norm=237.057, loss=491.7017, lr=8.40e-06]
Epoch 0 TRAIN: 105it [00:19,  7.02it/s, epoch=0, grad_norm=273.167, loss=496.0121, lr=8.48e-06]
Epoch 0 TRAIN: 105it [00:19,  7.02it/s, epoch=0, grad_norm=235.342, loss=464.7297, lr=8.56e-06]
Epoch 0 TRAIN: 105it [00:19,  7.02it/s, epoch=0, grad_norm=569.803, loss=461.5146, lr=8.64e-06]
Epoch 0 TRAIN: 105it [00:19,  7.02it/s, epoch=0, grad_norm=254.866, loss=442.5222, lr=8.72e-06]
Epoch 0 TRAIN: 109it [00:19,  7.22it/s, epoch=0, grad_norm=254.866, loss=442.5222, lr=8.72e-06]
Epoch 0 TRAIN: 109it [00:20,  7.22it/s, epoch=0, grad_norm=330.764, loss=440.5875, lr=8.80e-06]
Epoch 0 TRAIN: 109it [00:20,  7.22it/s, epoch=0, grad_norm=163.950, loss=493.9410, lr=8.88e-06]
Epoch 0 TRAIN: 109it [00:20,  7.22it/s, epoch=0, grad_norm=186.109, loss=483.6877, lr=8.96e-06]
Epoch 0 TRAIN: 109it [00:20,  7.22it/s, epoch=0, grad_norm=158.826, loss=453.2689, lr=9.04e-06]
Epoch 0 TRAIN: 113it [00:20,  6.99it/s, epoch=0, grad_norm=158.826, loss=453.2689, lr=9.04e-06]
Epoch 0 TRAIN: 113it [00:20,  6.99it/s, epoch=0, grad_norm=226.001, loss=470.9177, lr=9.12e-06]
Epoch 0 TRAIN: 113it [00:20,  6.99it/s, epoch=0, grad_norm=133.595, loss=494.8362, lr=9.20e-06]
Epoch 0 TRAIN: 113it [00:20,  6.99it/s, epoch=0, grad_norm=156.068, loss=523.1016, lr=9.28e-06]
Epoch 0 TRAIN: 113it [00:21,  6.99it/s, epoch=0, grad_norm=192.100, loss=561.2066, lr=9.36e-06]
Epoch 0 TRAIN: 117it [00:21,  7.18it/s, epoch=0, grad_norm=192.100, loss=561.2066, lr=9.36e-06]
Epoch 0 TRAIN: 117it [00:21,  7.18it/s, epoch=0, grad_norm=138.766, loss=470.1357, lr=9.44e-06]
Epoch 0 TRAIN: 117it [00:21,  7.18it/s, epoch=0, grad_norm=222.032, loss=374.3672, lr=9.52e-06]
Epoch 0 TRAIN: 117it [00:21,  7.18it/s, epoch=0, grad_norm=399.326, loss=316.7515, lr=9.60e-06]
Epoch 0 TRAIN: 117it [00:21,  7.18it/s, epoch=0, grad_norm=246.843, loss=445.8704, lr=9.68e-06]
Epoch 0 TRAIN: 121it [00:21,  7.06it/s, epoch=0, grad_norm=246.843, loss=445.8704, lr=9.68e-06]
Epoch 0 TRAIN: 121it [00:21,  7.06it/s, epoch=0, grad_norm=225.469, loss=402.3617, lr=9.76e-06]
Epoch 0 TRAIN: 121it [00:21,  7.06it/s, epoch=0, grad_norm=137.632, loss=440.3280, lr=9.84e-06]
Epoch 0 TRAIN: 121it [00:22,  7.06it/s, epoch=0, grad_norm=187.080, loss=436.8904, lr=9.92e-06]
Epoch 0 TRAIN: 121it [00:22,  7.06it/s, epoch=0, grad_norm=147.266, loss=396.0627, lr=1.00e-05]
Epoch 0 TRAIN: 121it [00:22,  7.06it/s, epoch=0, grad_norm=319.739, loss=536.9909, lr=1.01e-05]
Epoch 0 TRAIN: 126it [00:22,  7.32it/s, epoch=0, grad_norm=319.739, loss=536.9909, lr=1.01e-05]
Epoch 0 TRAIN: 126it [00:22,  7.32it/s, epoch=0, grad_norm=163.436, loss=445.7943, lr=1.02e-05]
Epoch 0 TRAIN: 126it [00:22,  7.32it/s, epoch=0, grad_norm=218.851, loss=469.9691, lr=1.02e-05]
Epoch 0 TRAIN: 126it [00:22,  7.32it/s, epoch=0, grad_norm=141.942, loss=362.0862, lr=1.03e-05]
Epoch 0 TRAIN: 126it [00:22,  7.32it/s, epoch=0, grad_norm=202.847, loss=526.8591, lr=1.04e-05]
Epoch 0 TRAIN: 126it [00:22,  7.32it/s, epoch=0, grad_norm=179.164, loss=474.3831, lr=1.05e-05]
Epoch 0 TRAIN: 131it [00:22,  7.67it/s, epoch=0, grad_norm=179.164, loss=474.3831, lr=1.05e-05]
Epoch 0 TRAIN: 131it [00:23,  7.67it/s, epoch=0, grad_norm=167.620, loss=503.6517, lr=1.06e-05]
Epoch 0 TRAIN: 132it [00:23,  5.72it/s, epoch=0, grad_norm=167.620, loss=503.6517, lr=1.06e-05]

CV: 0it [00:00, ?it/s]/home/rookie/Documents/batuka/projects/wenet/wenet/wenet/utils/train_utils.py:689: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  amp_autocast(enabled=dtype is not None,

CV: 1it [00:01,  1.20s/it]
                          
2026-04-16 19:53:06,766 INFO Epoch 0 Step 132 CV info lr 1.0560e-05 cv_loss 493.0354031649503 rank 0 acc 0.08543461126585801
2026-04-16 19:53:06,768 INFO [Rank 0] Checkpoint: save to checkpoint exp/conformer_mongolian/epoch_0.pt
2026-04-16 19:53:06,881 INFO Epoch 1 Step 132 TRAIN info lr 1.0560e-05 rank 0
2026-04-16 19:53:06,894 INFO using accumulate grad, new batch size is 1 times larger than before

Epoch 1 TRAIN:   0%|          | 0/132 [00:00<?, ?it/s]
Epoch 1 TRAIN:   0%|          | 0/132 [00:03<?, ?it/s, epoch=1, grad_norm=532.381, loss=450.3934, lr=1.06e-05]
Epoch 1 TRAIN:   1%|          | 1/132 [00:03<08:01,  3.67s/it, epoch=1, grad_norm=532.381, loss=450.3934, lr=1.06e-05]
Epoch 1 TRAIN:   1%|          | 1/132 [00:04<08:01,  3.67s/it, epoch=1, grad_norm=700.272, loss=546.1603, lr=1.07e-05]
Epoch 1 TRAIN:   2%|▏         | 2/132 [00:04<04:01,  1.86s/it, epoch=1, grad_norm=700.272, loss=546.1603, lr=1.07e-05]
Epoch 1 TRAIN:   2%|▏         | 2/132 [00:04<04:01,  1.86s/it, epoch=1, grad_norm=549.759, loss=493.0074, lr=1.08e-05]
Epoch 1 TRAIN:   2%|▏         | 2/132 [00:04<04:01,  1.86s/it, epoch=1, grad_norm=580.324, loss=543.3506, lr=1.09e-05]
Epoch 1 TRAIN:   2%|▏         | 2/132 [00:04<04:01,  1.86s/it, epoch=1, grad_norm=529.618, loss=550.4033, lr=1.10e-05]
Epoch 1 TRAIN:   2%|▏         | 2/132 [00:04<04:01,  1.86s/it, epoch=1, grad_norm=271.666, loss=464.8093, lr=1.10e-05]
Epoch 1 TRAIN:   2%|▏         | 2/132 [00:04<04:01,  1.86s/it, epoch=1, grad_norm=182.932, loss=474.3825, lr=1.11e-05]
Epoch 1 TRAIN:   2%|▏         | 2/132 [00:04<04:01,  1.86s/it, epoch=1, grad_norm=181.515, loss=503.0537, lr=1.12e-05]
Epoch 1 TRAIN:   2%|▏         | 2/132 [00:04<04:01,  1.86s/it, epoch=1, grad_norm=347.705, loss=429.4198, lr=1.13e-05]
Epoch 1 TRAIN:   7%|▋         | 9/132 [00:04<00:40,  3.02it/s, epoch=1, grad_norm=347.705, loss=429.4198, lr=1.13e-05]
Epoch 1 TRAIN:   7%|▋         | 9/132 [00:04<00:40,  3.02it/s, epoch=1, grad_norm=524.627, loss=399.2037, lr=1.14e-05]
Epoch 1 TRAIN:   7%|▋         | 9/132 [00:05<00:40,  3.02it/s, epoch=1, grad_norm=551.572, loss=420.0698, lr=1.14e-05]
Epoch 1 TRAIN:   7%|▋         | 9/132 [00:05<00:40,  3.02it/s, epoch=1, grad_norm=538.843, loss=419.5546, lr=1.15e-05]
Epoch 1 TRAIN:   7%|▋         | 9/132 [00:05<00:40,  3.02it/s, epoch=1, grad_norm=384.739, loss=492.5327, lr=1.16e-05]
Epoch 1 TRAIN:  10%|▉         | 13/132 [00:05<00:29,  3.99it/s, epoch=1, grad_norm=384.739, loss=492.5327, lr=1.16e-05]
Epoch 1 TRAIN:  10%|▉         | 13/132 [00:05<00:29,  3.99it/s, epoch=1, grad_norm=437.744, loss=428.3743, lr=1.17e-05]
Epoch 1 TRAIN:  10%|▉         | 13/132 [00:05<00:29,  3.99it/s, epoch=1, grad_norm=471.981, loss=394.0956, lr=1.18e-05]
Epoch 1 TRAIN:  10%|▉         | 13/132 [00:05<00:29,  3.99it/s, epoch=1, grad_norm=288.043, loss=432.8463, lr=1.18e-05]
Epoch 1 TRAIN:  10%|▉         | 13/132 [00:05<00:29,  3.99it/s, epoch=1, grad_norm=376.671, loss=371.7540, lr=1.19e-05]
Epoch 1 TRAIN:  13%|█▎        | 17/132 [00:05<00:23,  4.84it/s, epoch=1, grad_norm=376.671, loss=371.7540, lr=1.19e-05]
Epoch 1 TRAIN:  13%|█▎        | 17/132 [00:05<00:23,  4.84it/s, epoch=1, grad_norm=146.895, loss=438.5561, lr=1.20e-05]
Epoch 1 TRAIN:  13%|█▎        | 17/132 [00:06<00:23,  4.84it/s, epoch=1, grad_norm=166.101, loss=397.4515, lr=1.21e-05]
Epoch 1 TRAIN:  13%|█▎        | 17/132 [00:06<00:23,  4.84it/s, epoch=1, grad_norm=203.977, loss=451.1616, lr=1.22e-05]
Epoch 1 TRAIN:  13%|█▎        | 17/132 [00:06<00:23,  4.84it/s, epoch=1, grad_norm=300.504, loss=484.3538, lr=1.22e-05]
Epoch 1 TRAIN:  16%|█▌        | 21/132 [00:06<00:20,  5.47it/s, epoch=1, grad_norm=300.504, loss=484.3538, lr=1.22e-05]
Epoch 1 TRAIN:  16%|█▌        | 21/132 [00:06<00:20,  5.47it/s, epoch=1, grad_norm=292.751, loss=469.0961, lr=1.23e-05]
Epoch 1 TRAIN:  16%|█▌        | 21/132 [00:06<00:20,  5.47it/s, epoch=1, grad_norm=233.395, loss=426.4777, lr=1.24e-05]
Epoch 1 TRAIN:  16%|█▌        | 21/132 [00:06<00:20,  5.47it/s, epoch=1, grad_norm=256.646, loss=449.0944, lr=1.25e-05]
Epoch 1 TRAIN:  16%|█▌        | 21/132 [00:06<00:20,  5.47it/s, epoch=1, grad_norm=214.473, loss=459.5690, lr=1.26e-05]
Epoch 1 TRAIN:  19%|█▉        | 25/132 [00:06<00:18,  5.94it/s, epoch=1, grad_norm=214.473, loss=459.5690, lr=1.26e-05]
Epoch 1 TRAIN:  19%|█▉        | 25/132 [00:07<00:18,  5.94it/s, epoch=1, grad_norm=121.385, loss=395.4513, lr=1.26e-05]
Epoch 1 TRAIN:  19%|█▉        | 25/132 [00:07<00:18,  5.94it/s, epoch=1, grad_norm=107.097, loss=382.5357, lr=1.27e-05]
Epoch 1 TRAIN:  19%|█▉        | 25/132 [00:07<00:18,  5.94it/s, epoch=1, grad_norm=120.956, loss=422.2874, lr=1.28e-05]
Epoch 1 TRAIN:  19%|█▉        | 25/132 [00:07<00:18,  5.94it/s, epoch=1, grad_norm=134.913, loss=454.1078, lr=1.29e-05]
Epoch 1 TRAIN:  22%|██▏       | 29/132 [00:07<00:16,  6.34it/s, epoch=1, grad_norm=134.913, loss=454.1078, lr=1.29e-05]
Epoch 1 TRAIN:  22%|██▏       | 29/132 [00:07<00:16,  6.34it/s, epoch=1, grad_norm=157.435, loss=387.0605, lr=1.30e-05]
Epoch 1 TRAIN:  22%|██▏       | 29/132 [00:07<00:16,  6.34it/s, epoch=1, grad_norm=133.994, loss=421.7093, lr=1.30e-05]
Epoch 1 TRAIN:  22%|██▏       | 29/132 [00:07<00:16,  6.34it/s, epoch=1, grad_norm=136.009, loss=442.0594, lr=1.31e-05]
Epoch 1 TRAIN:  22%|██▏       | 29/132 [00:08<00:16,  6.34it/s, epoch=1, grad_norm=215.044, loss=408.9920, lr=1.32e-05]
Epoch 1 TRAIN:  25%|██▌       | 33/132 [00:08<00:14,  6.61it/s, epoch=1, grad_norm=215.044, loss=408.9920, lr=1.32e-05]
Epoch 1 TRAIN:  25%|██▌       | 33/132 [00:08<00:14,  6.61it/s, epoch=1, grad_norm=123.407, loss=453.6834, lr=1.33e-05]
Epoch 1 TRAIN:  25%|██▌       | 33/132 [00:08<00:14,  6.61it/s, epoch=1, grad_norm=230.960, loss=392.3174, lr=1.34e-05]
Epoch 1 TRAIN:  25%|██▌       | 33/132 [00:08<00:14,  6.61it/s, epoch=1, grad_norm=105.510, loss=424.5005, lr=1.34e-05]
Epoch 1 TRAIN:  25%|██▌       | 33/132 [00:08<00:14,  6.61it/s, epoch=1, grad_norm=103.600, loss=406.2471, lr=1.35e-05]
Epoch 1 TRAIN:  28%|██▊       | 37/132 [00:08<00:13,  6.83it/s, epoch=1, grad_norm=103.600, loss=406.2471, lr=1.35e-05]
Epoch 1 TRAIN:  28%|██▊       | 37/132 [00:08<00:13,  6.83it/s, epoch=1, grad_norm=129.421, loss=388.4724, lr=1.36e-05]
Epoch 1 TRAIN:  28%|██▊       | 37/132 [00:08<00:13,  6.83it/s, epoch=1, grad_norm=124.284, loss=402.7646, lr=1.37e-05]
Epoch 1 TRAIN:  28%|██▊       | 37/132 [00:08<00:13,  6.83it/s, epoch=1, grad_norm=164.876, loss=434.5948, lr=1.38e-05]
Epoch 1 TRAIN:  28%|██▊       | 37/132 [00:09<00:13,  6.83it/s, epoch=1, grad_norm=119.112, loss=412.2685, lr=1.38e-05]
Epoch 1 TRAIN:  31%|███       | 41/132 [00:09<00:13,  7.00it/s, epoch=1, grad_norm=119.112, loss=412.2685, lr=1.38e-05]
Epoch 1 TRAIN:  31%|███       | 41/132 [00:09<00:13,  7.00it/s, epoch=1, grad_norm=117.447, loss=393.4797, lr=1.39e-05]
Epoch 1 TRAIN:  31%|███       | 41/132 [00:09<00:13,  7.00it/s, epoch=1, grad_norm=111.458, loss=396.3036, lr=1.40e-05]
Epoch 1 TRAIN:  31%|███       | 41/132 [00:09<00:13,  7.00it/s, epoch=1, grad_norm=137.633, loss=417.1057, lr=1.41e-05]
Epoch 1 TRAIN:  31%|███       | 41/132 [00:09<00:13,  7.00it/s, epoch=1, grad_norm=183.657, loss=453.2611, lr=1.42e-05]
Epoch 1 TRAIN:  34%|███▍      | 45/132 [00:09<00:12,  7.02it/s, epoch=1, grad_norm=183.657, loss=453.2611, lr=1.42e-05]
Epoch 1 TRAIN:  34%|███▍      | 45/132 [00:09<00:12,  7.02it/s, epoch=1, grad_norm=176.580, loss=469.0706, lr=1.42e-05]
Epoch 1 TRAIN:  34%|███▍      | 45/132 [00:09<00:12,  7.02it/s, epoch=1, grad_norm=132.538, loss=395.9438, lr=1.43e-05]
Epoch 1 TRAIN:  34%|███▍      | 45/132 [00:10<00:12,  7.02it/s, epoch=1, grad_norm=214.021, loss=344.5364, lr=1.44e-05]
Epoch 1 TRAIN:  34%|███▍      | 45/132 [00:10<00:12,  7.02it/s, epoch=1, grad_norm=248.314, loss=375.9923, lr=1.45e-05]
Epoch 1 TRAIN:  37%|███▋      | 49/132 [00:10<00:11,  7.10it/s, epoch=1, grad_norm=248.314, loss=375.9923, lr=1.45e-05]
Epoch 1 TRAIN:  37%|███▋      | 49/132 [00:10<00:11,  7.10it/s, epoch=1, grad_norm=119.327, loss=411.1581, lr=1.46e-05]
Epoch 1 TRAIN:  37%|███▋      | 49/132 [00:10<00:11,  7.10it/s, epoch=1, grad_norm=144.866, loss=414.0189, lr=1.46e-05]
Epoch 1 TRAIN:  37%|███▋      | 49/132 [00:10<00:11,  7.10it/s, epoch=1, grad_norm=116.954, loss=403.3265, lr=1.47e-05]
Epoch 1 TRAIN:  37%|███▋      | 49/132 [00:10<00:11,  7.10it/s, epoch=1, grad_norm=146.147, loss=443.9620, lr=1.48e-05]
Epoch 1 TRAIN:  40%|████      | 53/132 [00:10<00:11,  7.13it/s, epoch=1, grad_norm=146.147, loss=443.9620, lr=1.48e-05]
Epoch 1 TRAIN:  40%|████      | 53/132 [00:10<00:11,  7.13it/s, epoch=1, grad_norm=98.421, loss=401.0244, lr=1.49e-05] 
Epoch 1 TRAIN:  40%|████      | 53/132 [00:11<00:11,  7.13it/s, epoch=1, grad_norm=131.190, loss=440.2570, lr=1.50e-05]
Epoch 1 TRAIN:  40%|████      | 53/132 [00:11<00:11,  7.13it/s, epoch=1, grad_norm=105.937, loss=392.7990, lr=1.50e-05]
Epoch 1 TRAIN:  40%|████      | 53/132 [00:11<00:11,  7.13it/s, epoch=1, grad_norm=250.034, loss=315.2540, lr=1.51e-05]
Epoch 1 TRAIN:  43%|████▎     | 57/132 [00:11<00:10,  7.07it/s, epoch=1, grad_norm=250.034, loss=315.2540, lr=1.51e-05]
Epoch 1 TRAIN:  43%|████▎     | 57/132 [00:11<00:10,  7.07it/s, epoch=1, grad_norm=107.758, loss=402.6362, lr=1.52e-05]
Epoch 1 TRAIN:  43%|████▎     | 57/132 [00:11<00:10,  7.07it/s, epoch=1, grad_norm=145.137, loss=349.7819, lr=1.53e-05]
Epoch 1 TRAIN:  43%|████▎     | 57/132 [00:11<00:10,  7.07it/s, epoch=1, grad_norm=138.464, loss=427.4655, lr=1.54e-05]
Epoch 1 TRAIN:  43%|████▎     | 57/132 [00:11<00:10,  7.07it/s, epoch=1, grad_norm=163.262, loss=423.9892, lr=1.54e-05]
Epoch 1 TRAIN:  46%|████▌     | 61/132 [00:11<00:10,  7.07it/s, epoch=1, grad_norm=163.262, loss=423.9892, lr=1.54e-05]
Epoch 1 TRAIN:  46%|████▌     | 61/132 [00:12<00:10,  7.07it/s, epoch=1, grad_norm=98.174, loss=377.5705, lr=1.55e-05] 
Epoch 1 TRAIN:  46%|████▌     | 61/132 [00:12<00:10,  7.07it/s, epoch=1, grad_norm=139.356, loss=449.0046, lr=1.56e-05]
Epoch 1 TRAIN:  46%|████▌     | 61/132 [00:12<00:10,  7.07it/s, epoch=1, grad_norm=114.241, loss=414.4142, lr=1.57e-05]
Epoch 1 TRAIN:  46%|████▌     | 61/132 [00:12<00:10,  7.07it/s, epoch=1, grad_norm=112.281, loss=429.5023, lr=1.58e-05]
Epoch 1 TRAIN:  49%|████▉     | 65/132 [00:12<00:09,  7.08it/s, epoch=1, grad_norm=112.281, loss=429.5023, lr=1.58e-05]
Epoch 1 TRAIN:  49%|████▉     | 65/132 [00:12<00:09,  7.08it/s, epoch=1, grad_norm=137.808, loss=376.4321, lr=1.58e-05]
Epoch 1 TRAIN:  49%|████▉     | 65/132 [00:12<00:09,  7.08it/s, epoch=1, grad_norm=202.270, loss=386.4391, lr=1.59e-05]
Epoch 1 TRAIN:  49%|████▉     | 65/132 [00:12<00:09,  7.08it/s, epoch=1, grad_norm=210.746, loss=369.4850, lr=1.60e-05]
Epoch 1 TRAIN:  49%|████▉     | 65/132 [00:13<00:09,  7.08it/s, epoch=1, grad_norm=117.486, loss=379.2134, lr=1.61e-05]
Epoch 1 TRAIN:  52%|█████▏    | 69/132 [00:13<00:08,  7.17it/s, epoch=1, grad_norm=117.486, loss=379.2134, lr=1.61e-05]
Epoch 1 TRAIN:  52%|█████▏    | 69/132 [00:13<00:08,  7.17it/s, epoch=1, grad_norm=93.080, loss=372.1108, lr=1.62e-05] 
Epoch 1 TRAIN:  52%|█████▏    | 69/132 [00:13<00:08,  7.17it/s, epoch=1, grad_norm=126.537, loss=333.9422, lr=1.62e-05]
Epoch 1 TRAIN:  52%|█████▏    | 69/132 [00:13<00:08,  7.17it/s, epoch=1, grad_norm=108.407, loss=339.2478, lr=1.63e-05]
Epoch 1 TRAIN:  52%|█████▏    | 69/132 [00:13<00:08,  7.17it/s, epoch=1, grad_norm=142.379, loss=395.0834, lr=1.64e-05]
Epoch 1 TRAIN:  55%|█████▌    | 73/132 [00:13<00:08,  7.08it/s, epoch=1, grad_norm=142.379, loss=395.0834, lr=1.64e-05]
Epoch 1 TRAIN:  55%|█████▌    | 73/132 [00:13<00:08,  7.08it/s, epoch=1, grad_norm=202.843, loss=420.4676, lr=1.65e-05]
Epoch 1 TRAIN:  55%|█████▌    | 73/132 [00:13<00:08,  7.08it/s, epoch=1, grad_norm=170.590, loss=333.2113, lr=1.66e-05]
Epoch 1 TRAIN:  55%|█████▌    | 73/132 [00:14<00:08,  7.08it/s, epoch=1, grad_norm=214.635, loss=399.8603, lr=1.66e-05]
Epoch 1 TRAIN:  55%|█████▌    | 73/132 [00:14<00:08,  7.08it/s, epoch=1, grad_norm=230.805, loss=387.3589, lr=1.67e-05]
Epoch 1 TRAIN:  58%|█████▊    | 77/132 [00:14<00:07,  7.05it/s, epoch=1, grad_norm=230.805, loss=387.3589, lr=1.67e-05]
Epoch 1 TRAIN:  58%|█████▊    | 77/132 [00:14<00:07,  7.05it/s, epoch=1, grad_norm=188.230, loss=377.9527, lr=1.68e-05]
Epoch 1 TRAIN:  58%|█████▊    | 77/132 [00:14<00:07,  7.05it/s, epoch=1, grad_norm=172.001, loss=370.0881, lr=1.69e-05]
Epoch 1 TRAIN:  58%|█████▊    | 77/132 [00:14<00:07,  7.05it/s, epoch=1, grad_norm=137.042, loss=375.0284, lr=1.70e-05]
Epoch 1 TRAIN:  58%|█████▊    | 77/132 [00:14<00:07,  7.05it/s, epoch=1, grad_norm=308.173, loss=317.8282, lr=1.70e-05]
Epoch 1 TRAIN:  61%|██████▏   | 81/132 [00:14<00:07,  7.15it/s, epoch=1, grad_norm=308.173, loss=317.8282, lr=1.70e-05]
Epoch 1 TRAIN:  61%|██████▏   | 81/132 [00:14<00:07,  7.15it/s, epoch=1, grad_norm=166.792, loss=454.6361, lr=1.71e-05]
Epoch 1 TRAIN:  61%|██████▏   | 81/132 [00:15<00:07,  7.15it/s, epoch=1, grad_norm=257.567, loss=391.0308, lr=1.72e-05]
Epoch 1 TRAIN:  61%|██████▏   | 81/132 [00:15<00:07,  7.15it/s, epoch=1, grad_norm=221.561, loss=349.5658, lr=1.73e-05]
Epoch 1 TRAIN:  61%|██████▏   | 81/132 [00:15<00:07,  7.15it/s, epoch=1, grad_norm=209.198, loss=465.2584, lr=1.74e-05]
Epoch 1 TRAIN:  64%|██████▍   | 85/132 [00:15<00:06,  7.09it/s, epoch=1, grad_norm=209.198, loss=465.2584, lr=1.74e-05]
Epoch 1 TRAIN:  64%|██████▍   | 85/132 [00:15<00:06,  7.09it/s, epoch=1, grad_norm=101.447, loss=442.6868, lr=1.74e-05]
Epoch 1 TRAIN:  64%|██████▍   | 85/132 [00:15<00:06,  7.09it/s, epoch=1, grad_norm=87.806, loss=390.4747, lr=1.75e-05] 
Epoch 1 TRAIN:  64%|██████▍   | 85/132 [00:15<00:06,  7.09it/s, epoch=1, grad_norm=204.827, loss=387.3405, lr=1.76e-05]
Epoch 1 TRAIN:  64%|██████▍   | 85/132 [00:15<00:06,  7.09it/s, epoch=1, grad_norm=524.829, loss=353.8021, lr=1.77e-05]
Epoch 1 TRAIN:  67%|██████▋   | 89/132 [00:15<00:06,  7.06it/s, epoch=1, grad_norm=524.829, loss=353.8021, lr=1.77e-05]
Epoch 1 TRAIN:  67%|██████▋   | 89/132 [00:16<00:06,  7.06it/s, epoch=1, grad_norm=137.706, loss=418.3854, lr=1.78e-05]
Epoch 1 TRAIN:  67%|██████▋   | 89/132 [00:16<00:06,  7.06it/s, epoch=1, grad_norm=382.886, loss=405.6108, lr=1.78e-05]
Epoch 1 TRAIN:  67%|██████▋   | 89/132 [00:16<00:06,  7.06it/s, epoch=1, grad_norm=156.942, loss=373.9964, lr=1.79e-05]
Epoch 1 TRAIN:  67%|██████▋   | 89/132 [00:16<00:06,  7.06it/s, epoch=1, grad_norm=149.137, loss=399.0746, lr=1.80e-05]
Epoch 1 TRAIN:  70%|███████   | 93/132 [00:16<00:05,  6.91it/s, epoch=1, grad_norm=149.137, loss=399.0746, lr=1.80e-05]
Epoch 1 TRAIN:  70%|███████   | 93/132 [00:16<00:05,  6.91it/s, epoch=1, grad_norm=87.039, loss=344.8499, lr=1.81e-05] 
Epoch 1 TRAIN:  70%|███████   | 93/132 [00:16<00:05,  6.91it/s, epoch=1, grad_norm=303.649, loss=456.1444, lr=1.82e-05]
Epoch 1 TRAIN:  70%|███████   | 93/132 [00:16<00:05,  6.91it/s, epoch=1, grad_norm=86.393, loss=333.8616, lr=1.82e-05] 
Epoch 1 TRAIN:  70%|███████   | 93/132 [00:17<00:05,  6.91it/s, epoch=1, grad_norm=221.221, loss=406.4156, lr=1.83e-05]
Epoch 1 TRAIN:  73%|███████▎  | 97/132 [00:17<00:05,  6.85it/s, epoch=1, grad_norm=221.221, loss=406.4156, lr=1.83e-05]
Epoch 1 TRAIN:  73%|███████▎  | 97/132 [00:17<00:05,  6.85it/s, epoch=1, grad_norm=102.798, loss=351.0565, lr=1.84e-05]
Epoch 1 TRAIN:  73%|███████▎  | 97/132 [00:17<00:05,  6.85it/s, epoch=1, grad_norm=109.464, loss=395.8022, lr=1.85e-05]
Epoch 1 TRAIN:  73%|███████▎  | 97/132 [00:17<00:05,  6.85it/s, epoch=1, grad_norm=138.687, loss=282.4952, lr=1.86e-05]2026-04-16 19:53:24,400 DEBUG TRAIN | steps/sec 5.673| Batch 1/100 loss 282.495178 loss_att 263.319672 loss_ctc 327.238037 th_accuracy 0.185535 lr 1.8560e-05 grad_norm 138.687317 rank 0

Epoch 1 TRAIN:  73%|███████▎  | 97/132 [00:17<00:05,  6.85it/s, epoch=1, grad_norm=364.810, loss=459.7681, lr=1.86e-05]
Epoch 1 TRAIN:  77%|███████▋  | 101/132 [00:17<00:04,  6.87it/s, epoch=1, grad_norm=364.810, loss=459.7681, lr=1.86e-05]
Epoch 1 TRAIN:  77%|███████▋  | 101/132 [00:17<00:04,  6.87it/s, epoch=1, grad_norm=338.186, loss=458.3467, lr=1.87e-05]
Epoch 1 TRAIN:  77%|███████▋  | 101/132 [00:17<00:04,  6.87it/s, epoch=1, grad_norm=295.363, loss=400.1642, lr=1.88e-05]
Epoch 1 TRAIN:  77%|███████▋  | 101/132 [00:18<00:04,  6.87it/s, epoch=1, grad_norm=88.235, loss=316.3632, lr=1.89e-05] 
Epoch 1 TRAIN:  77%|███████▋  | 101/132 [00:18<00:04,  6.87it/s, epoch=1, grad_norm=111.229, loss=293.6785, lr=1.90e-05]
Epoch 1 TRAIN:  80%|███████▉  | 105/132 [00:18<00:03,  7.01it/s, epoch=1, grad_norm=111.229, loss=293.6785, lr=1.90e-05]
Epoch 1 TRAIN:  80%|███████▉  | 105/132 [00:18<00:03,  7.01it/s, epoch=1, grad_norm=134.639, loss=399.6331, lr=1.90e-05]
Epoch 1 TRAIN:  80%|███████▉  | 105/132 [00:18<00:03,  7.01it/s, epoch=1, grad_norm=129.099, loss=324.9528, lr=1.91e-05]
Epoch 1 TRAIN:  80%|███████▉  | 105/132 [00:18<00:03,  7.01it/s, epoch=1, grad_norm=172.173, loss=366.6722, lr=1.92e-05]
Epoch 1 TRAIN:  80%|███████▉  | 105/132 [00:18<00:03,  7.01it/s, epoch=1, grad_norm=125.364, loss=300.2394, lr=1.93e-05]
Epoch 1 TRAIN:  83%|████████▎ | 109/132 [00:18<00:03,  7.03it/s, epoch=1, grad_norm=125.364, loss=300.2394, lr=1.93e-05]
Epoch 1 TRAIN:  83%|████████▎ | 109/132 [00:18<00:03,  7.03it/s, epoch=1, grad_norm=183.472, loss=365.5330, lr=1.94e-05]
Epoch 1 TRAIN:  83%|████████▎ | 109/132 [00:19<00:03,  7.03it/s, epoch=1, grad_norm=195.455, loss=448.3725, lr=1.94e-05]
Epoch 1 TRAIN:  83%|████████▎ | 109/132 [00:19<00:03,  7.03it/s, epoch=1, grad_norm=91.755, loss=359.2759, lr=1.95e-05] 
Epoch 1 TRAIN:  83%|████████▎ | 109/132 [00:19<00:03,  7.03it/s, epoch=1, grad_norm=142.085, loss=473.2764, lr=1.96e-05]
Epoch 1 TRAIN:  86%|████████▌ | 113/132 [00:19<00:02,  6.94it/s, epoch=1, grad_norm=142.085, loss=473.2764, lr=1.96e-05]
Epoch 1 TRAIN:  86%|████████▌ | 113/132 [00:19<00:02,  6.94it/s, epoch=1, grad_norm=87.006, loss=384.0017, lr=1.97e-05] 
Epoch 1 TRAIN:  86%|████████▌ | 113/132 [00:19<00:02,  6.94it/s, epoch=1, grad_norm=117.620, loss=310.4609, lr=1.98e-05]
Epoch 1 TRAIN:  86%|████████▌ | 113/132 [00:19<00:02,  6.94it/s, epoch=1, grad_norm=221.535, loss=283.6201, lr=1.98e-05]
Epoch 1 TRAIN:  86%|████████▌ | 113/132 [00:19<00:02,  6.94it/s, epoch=1, grad_norm=98.704, loss=328.2282, lr=1.99e-05] 
Epoch 1 TRAIN:  89%|████████▊ | 117/132 [00:19<00:02,  6.82it/s, epoch=1, grad_norm=98.704, loss=328.2282, lr=1.99e-05]
Epoch 1 TRAIN:  89%|████████▊ | 117/132 [00:20<00:02,  6.82it/s, epoch=1, grad_norm=169.213, loss=483.1382, lr=2.00e-05]
Epoch 1 TRAIN:  89%|████████▊ | 117/132 [00:20<00:02,  6.82it/s, epoch=1, grad_norm=103.300, loss=383.6798, lr=2.01e-05]
Epoch 1 TRAIN:  89%|████████▊ | 117/132 [00:20<00:02,  6.82it/s, epoch=1, grad_norm=267.134, loss=363.6924, lr=2.02e-05]
Epoch 1 TRAIN:  89%|████████▊ | 117/132 [00:20<00:02,  6.82it/s, epoch=1, grad_norm=128.575, loss=454.5852, lr=2.02e-05]
Epoch 1 TRAIN:  92%|█████████▏| 121/132 [00:20<00:01,  6.94it/s, epoch=1, grad_norm=128.575, loss=454.5852, lr=2.02e-05]
Epoch 1 TRAIN:  92%|█████████▏| 121/132 [00:20<00:01,  6.94it/s, epoch=1, grad_norm=100.397, loss=351.8896, lr=2.03e-05]
Epoch 1 TRAIN:  92%|█████████▏| 121/132 [00:20<00:01,  6.94it/s, epoch=1, grad_norm=257.989, loss=497.5266, lr=2.04e-05]
Epoch 1 TRAIN:  92%|█████████▏| 121/132 [00:20<00:01,  6.94it/s, epoch=1, grad_norm=184.932, loss=403.0951, lr=2.05e-05]
Epoch 1 TRAIN:  92%|█████████▏| 121/132 [00:21<00:01,  6.94it/s, epoch=1, grad_norm=146.934, loss=470.8504, lr=2.06e-05]
Epoch 1 TRAIN:  95%|█████████▍| 125/132 [00:21<00:01,  6.95it/s, epoch=1, grad_norm=146.934, loss=470.8504, lr=2.06e-05]
Epoch 1 TRAIN:  95%|█████████▍| 125/132 [00:21<00:01,  6.95it/s, epoch=1, grad_norm=123.291, loss=492.7487, lr=2.06e-05]
Epoch 1 TRAIN:  95%|█████████▍| 125/132 [00:21<00:01,  6.95it/s, epoch=1, grad_norm=270.436, loss=350.1666, lr=2.07e-05]
Epoch 1 TRAIN:  95%|█████████▍| 125/132 [00:21<00:01,  6.95it/s, epoch=1, grad_norm=113.518, loss=487.9394, lr=2.08e-05]
Epoch 1 TRAIN:  95%|█████████▍| 125/132 [00:21<00:01,  6.95it/s, epoch=1, grad_norm=194.251, loss=344.5317, lr=2.09e-05]
Epoch 1 TRAIN:  98%|█████████▊| 129/132 [00:21<00:00,  7.06it/s, epoch=1, grad_norm=194.251, loss=344.5317, lr=2.09e-05]
Epoch 1 TRAIN:  98%|█████████▊| 129/132 [00:21<00:00,  7.06it/s, epoch=1, grad_norm=252.538, loss=427.1947, lr=2.10e-05]
Epoch 1 TRAIN:  98%|█████████▊| 129/132 [00:21<00:00,  7.06it/s, epoch=1, grad_norm=455.371, loss=360.1992, lr=2.10e-05]
Epoch 1 TRAIN:  98%|█████████▊| 129/132 [00:21<00:00,  7.06it/s, epoch=1, grad_norm=151.207, loss=462.4988, lr=2.11e-05]
Epoch 1 TRAIN:  98%|█████████▊| 129/132 [00:22<00:00,  7.06it/s, epoch=1, grad_norm=154.601, loss=448.2256, lr=2.12e-05]
Epoch 1 TRAIN: 133it [00:22,  7.31it/s, epoch=1, grad_norm=154.601, loss=448.2256, lr=2.12e-05]                         
Epoch 1 TRAIN: 133it [00:22,  7.31it/s, epoch=1, grad_norm=161.363, loss=420.6894, lr=2.13e-05]
Epoch 1 TRAIN: 133it [00:22,  7.31it/s, epoch=1, grad_norm=119.333, loss=395.0858, lr=2.14e-05]
Epoch 1 TRAIN: 133it [00:22,  7.31it/s, epoch=1, grad_norm=221.169, loss=355.9395, lr=2.14e-05]
Epoch 1 TRAIN: 133it [00:22,  7.31it/s, epoch=1, grad_norm=202.700, loss=509.8112, lr=2.15e-05]
Epoch 1 TRAIN: 137it [00:22,  7.50it/s, epoch=1, grad_norm=202.700, loss=509.8112, lr=2.15e-05]
Epoch 1 TRAIN: 137it [00:22,  6.05it/s, epoch=1, grad_norm=202.700, loss=509.8112, lr=2.15e-05]

CV:   0%|          | 0/6 [00:00<?, ?it/s]/home/rookie/Documents/batuka/projects/wenet/wenet/wenet/utils/train_utils.py:689: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  amp_autocast(enabled=dtype is not None,

CV:  17%|█▋        | 1/6 [00:00<00:03,  1.33it/s]
                                                 
2026-04-16 19:53:30,378 INFO Epoch 1 Step 269 CV info lr 2.1520e-05 cv_loss 451.75680264559657 rank 0 acc 0.1971041684349378
2026-04-16 19:53:30,380 INFO [Rank 0] Checkpoint: save to checkpoint exp/conformer_mongolian/epoch_1.pt
2026-04-16 19:53:30,491 INFO Epoch 2 Step 269 TRAIN info lr 2.1520e-05 rank 0
2026-04-16 19:53:30,512 INFO using accumulate grad, new batch size is 1 times larger than before

Epoch 2 TRAIN:   0%|          | 0/137 [00:00<?, ?it/s]
Epoch 2 TRAIN:   0%|          | 0/137 [00:04<?, ?it/s, epoch=2, grad_norm=637.370, loss=485.5900, lr=2.16e-05]
Epoch 2 TRAIN:   1%|          | 1/137 [00:04<09:04,  4.00s/it, epoch=2, grad_norm=637.370, loss=485.5900, lr=2.16e-05]
Epoch 2 TRAIN:   1%|          | 1/137 [00:04<09:04,  4.00s/it, epoch=2, grad_norm=616.940, loss=465.7344, lr=2.17e-05]
Epoch 2 TRAIN:   1%|          | 1/137 [00:04<09:04,  4.00s/it, epoch=2, grad_norm=647.878, loss=486.6750, lr=2.18e-05]
Epoch 2 TRAIN:   1%|          | 1/137 [00:04<09:04,  4.00s/it, epoch=2, grad_norm=305.421, loss=289.5607, lr=2.18e-05]
Epoch 2 TRAIN:   1%|          | 1/137 [00:04<09:04,  4.00s/it, epoch=2, grad_norm=605.543, loss=465.3098, lr=2.19e-05]
Epoch 2 TRAIN:   1%|          | 1/137 [00:04<09:04,  4.00s/it, epoch=2, grad_norm=418.828, loss=378.0216, lr=2.20e-05]
Epoch 2 TRAIN:   1%|          | 1/137 [00:04<09:04,  4.00s/it, epoch=2, grad_norm=426.987, loss=392.6709, lr=2.21e-05]
Epoch 2 TRAIN:   1%|          | 1/137 [00:04<09:04,  4.00s/it, epoch=2, grad_norm=315.075, loss=360.5075, lr=2.22e-05]
Epoch 2 TRAIN:   1%|          | 1/137 [00:04<09:04,  4.00s/it, epoch=2, grad_norm=376.706, loss=421.5246, lr=2.22e-05]
Epoch 2 TRAIN:   7%|▋         | 9/137 [00:04<00:50,  2.55it/s, epoch=2, grad_norm=376.706, loss=421.5246, lr=2.22e-05]
Epoch 2 TRAIN:   7%|▋         | 9/137 [00:04<00:50,  2.55it/s, epoch=2, grad_norm=262.446, loss=395.8859, lr=2.23e-05]
Epoch 2 TRAIN:   7%|▋         | 9/137 [00:04<00:50,  2.55it/s, epoch=2, grad_norm=80.983, loss=351.6578, lr=2.24e-05] 
Epoch 2 TRAIN:   7%|▋         | 9/137 [00:05<00:50,  2.55it/s, epoch=2, grad_norm=90.279, loss=375.0465, lr=2.25e-05]
Epoch 2 TRAIN:   7%|▋         | 9/137 [00:05<00:50,  2.55it/s, epoch=2, grad_norm=111.165, loss=380.7955, lr=2.26e-05]
Epoch 2 TRAIN:   9%|▉         | 13/137 [00:05<00:36,  3.43it/s, epoch=2, grad_norm=111.165, loss=380.7955, lr=2.26e-05]
Epoch 2 TRAIN:   9%|▉         | 13/137 [00:05<00:36,  3.43it/s, epoch=2, grad_norm=124.686, loss=396.2214, lr=2.26e-05]
Epoch 2 TRAIN:   9%|▉         | 13/137 [00:05<00:36,  3.43it/s, epoch=2, grad_norm=179.120, loss=384.1317, lr=2.27e-05]
Epoch 2 TRAIN:   9%|▉         | 13/137 [00:05<00:36,  3.43it/s, epoch=2, grad_norm=256.305, loss=374.8235, lr=2.28e-05]
Epoch 2 TRAIN:   9%|▉         | 13/137 [00:05<00:36,  3.43it/s, epoch=2, grad_norm=325.126, loss=360.2586, lr=2.29e-05]
Epoch 2 TRAIN:  12%|█▏        | 17/137 [00:05<00:28,  4.22it/s, epoch=2, grad_norm=325.126, loss=360.2586, lr=2.29e-05]
Epoch 2 TRAIN:  12%|█▏        | 17/137 [00:05<00:28,  4.22it/s, epoch=2, grad_norm=220.688, loss=389.9032, lr=2.30e-05]
Epoch 2 TRAIN:  12%|█▏        | 17/137 [00:05<00:28,  4.22it/s, epoch=2, grad_norm=214.807, loss=364.6744, lr=2.30e-05]
Epoch 2 TRAIN:  12%|█▏        | 17/137 [00:06<00:28,  4.22it/s, epoch=2, grad_norm=131.643, loss=401.3435, lr=2.31e-05]
Epoch 2 TRAIN:  12%|█▏        | 17/137 [00:06<00:28,  4.22it/s, epoch=2, grad_norm=75.820, loss=434.2256, lr=2.32e-05] 
Epoch 2 TRAIN:  15%|█▌        | 21/137 [00:06<00:23,  4.90it/s, epoch=2, grad_norm=75.820, loss=434.2256, lr=2.32e-05]
Epoch 2 TRAIN:  15%|█▌        | 21/137 [00:06<00:23,  4.90it/s, epoch=2, grad_norm=92.061, loss=365.9359, lr=2.33e-05]
Epoch 2 TRAIN:  15%|█▌        | 21/137 [00:06<00:23,  4.90it/s, epoch=2, grad_norm=72.126, loss=364.0867, lr=2.34e-05]
Epoch 2 TRAIN:  15%|█▌        | 21/137 [00:06<00:23,  4.90it/s, epoch=2, grad_norm=76.791, loss=350.1964, lr=2.34e-05]
Epoch 2 TRAIN:  15%|█▌        | 21/137 [00:06<00:23,  4.90it/s, epoch=2, grad_norm=177.419, loss=417.0098, lr=2.35e-05]
Epoch 2 TRAIN:  18%|█▊        | 25/137 [00:06<00:20,  5.45it/s, epoch=2, grad_norm=177.419, loss=417.0098, lr=2.35e-05]
Epoch 2 TRAIN:  18%|█▊        | 25/137 [00:06<00:20,  5.45it/s, epoch=2, grad_norm=73.355, loss=302.2961, lr=2.36e-05] 
Epoch 2 TRAIN:  18%|█▊        | 25/137 [00:07<00:20,  5.45it/s, epoch=2, grad_norm=98.363, loss=365.1572, lr=2.37e-05]
Epoch 2 TRAIN:  18%|█▊        | 25/137 [00:07<00:20,  5.45it/s, epoch=2, grad_norm=150.787, loss=369.4916, lr=2.38e-05]
Epoch 2 TRAIN:  18%|█▊        | 25/137 [00:07<00:20,  5.45it/s, epoch=2, grad_norm=202.605, loss=412.0719, lr=2.38e-05]
Epoch 2 TRAIN:  21%|██        | 29/137 [00:07<00:18,  5.85it/s, epoch=2, grad_norm=202.605, loss=412.0719, lr=2.38e-05]
Epoch 2 TRAIN:  21%|██        | 29/137 [00:07<00:18,  5.85it/s, epoch=2, grad_norm=94.664, loss=349.9521, lr=2.39e-05] 
Epoch 2 TRAIN:  21%|██        | 29/137 [00:07<00:18,  5.85it/s, epoch=2, grad_norm=80.135, loss=351.2919, lr=2.40e-05]
Epoch 2 TRAIN:  21%|██        | 29/137 [00:07<00:18,  5.85it/s, epoch=2, grad_norm=170.276, loss=410.9509, lr=2.41e-05]
Epoch 2 TRAIN:  21%|██        | 29/137 [00:07<00:18,  5.85it/s, epoch=2, grad_norm=96.090, loss=410.0779, lr=2.42e-05] 
Epoch 2 TRAIN:  24%|██▍       | 33/137 [00:07<00:16,  6.22it/s, epoch=2, grad_norm=96.090, loss=410.0779, lr=2.42e-05]
Epoch 2 TRAIN:  24%|██▍       | 33/137 [00:08<00:16,  6.22it/s, epoch=2, grad_norm=76.010, loss=380.5703, lr=2.42e-05]
Epoch 2 TRAIN:  24%|██▍       | 33/137 [00:08<00:16,  6.22it/s, epoch=2, grad_norm=97.986, loss=393.2389, lr=2.43e-05]
Epoch 2 TRAIN:  24%|██▍       | 33/137 [00:08<00:16,  6.22it/s, epoch=2, grad_norm=151.352, loss=367.9479, lr=2.44e-05]
Epoch 2 TRAIN:  24%|██▍       | 33/137 [00:08<00:16,  6.22it/s, epoch=2, grad_norm=284.146, loss=348.5240, lr=2.45e-05]
Epoch 2 TRAIN:  27%|██▋       | 37/137 [00:08<00:15,  6.50it/s, epoch=2, grad_norm=284.146, loss=348.5240, lr=2.45e-05]
Epoch 2 TRAIN:  27%|██▋       | 37/137 [00:08<00:15,  6.50it/s, epoch=2, grad_norm=168.239, loss=361.1327, lr=2.46e-05]
Epoch 2 TRAIN:  27%|██▋       | 37/137 [00:08<00:15,  6.50it/s, epoch=2, grad_norm=116.178, loss=405.1331, lr=2.46e-05]
Epoch 2 TRAIN:  27%|██▋       | 37/137 [00:08<00:15,  6.50it/s, epoch=2, grad_norm=67.362, loss=400.6379, lr=2.47e-05] 
Epoch 2 TRAIN:  27%|██▋       | 37/137 [00:09<00:15,  6.50it/s, epoch=2, grad_norm=70.896, loss=398.3441, lr=2.48e-05]
Epoch 2 TRAIN:  30%|██▉       | 41/137 [00:09<00:14,  6.69it/s, epoch=2, grad_norm=70.896, loss=398.3441, lr=2.48e-05]
Epoch 2 TRAIN:  30%|██▉       | 41/137 [00:09<00:14,  6.69it/s, epoch=2, grad_norm=84.622, loss=389.4428, lr=2.49e-05]
Epoch 2 TRAIN:  30%|██▉       | 41/137 [00:09<00:14,  6.69it/s, epoch=2, grad_norm=68.090, loss=379.0897, lr=2.50e-05]
Epoch 2 TRAIN:  30%|██▉       | 41/137 [00:09<00:14,  6.69it/s, epoch=2, grad_norm=124.526, loss=303.1729, lr=2.50e-05]
Epoch 2 TRAIN:  30%|██▉       | 41/137 [00:09<00:14,  6.69it/s, epoch=2, grad_norm=68.222, loss=358.8251, lr=2.51e-05] 
Epoch 2 TRAIN:  33%|███▎      | 45/137 [00:09<00:13,  6.76it/s, epoch=2, grad_norm=68.222, loss=358.8251, lr=2.51e-05]
Epoch 2 TRAIN:  33%|███▎      | 45/137 [00:09<00:13,  6.76it/s, epoch=2, grad_norm=179.867, loss=411.2051, lr=2.52e-05]
Epoch 2 TRAIN:  33%|███▎      | 45/137 [00:09<00:13,  6.76it/s, epoch=2, grad_norm=152.127, loss=402.3429, lr=2.53e-05]
Epoch 2 TRAIN:  33%|███▎      | 45/137 [00:10<00:13,  6.76it/s, epoch=2, grad_norm=117.223, loss=380.6246, lr=2.54e-05]
Epoch 2 TRAIN:  33%|███▎      | 45/137 [00:10<00:13,  6.76it/s, epoch=2, grad_norm=69.645, loss=352.3043, lr=2.54e-05] 
Epoch 2 TRAIN:  36%|███▌      | 49/137 [00:10<00:12,  6.82it/s, epoch=2, grad_norm=69.645, loss=352.3043, lr=2.54e-05]
Epoch 2 TRAIN:  36%|███▌      | 49/137 [00:10<00:12,  6.82it/s, epoch=2, grad_norm=69.551, loss=369.2302, lr=2.55e-05]
Epoch 2 TRAIN:  36%|███▌      | 49/137 [00:10<00:12,  6.82it/s, epoch=2, grad_norm=73.497, loss=379.7113, lr=2.56e-05]
Epoch 2 TRAIN:  36%|███▌      | 49/137 [00:10<00:12,  6.82it/s, epoch=2, grad_norm=109.824, loss=414.8331, lr=2.57e-05]
Epoch 2 TRAIN:  36%|███▌      | 49/137 [00:10<00:12,  6.82it/s, epoch=2, grad_norm=90.772, loss=427.9661, lr=2.58e-05] 
Epoch 2 TRAIN:  39%|███▊      | 53/137 [00:10<00:12,  6.84it/s, epoch=2, grad_norm=90.772, loss=427.9661, lr=2.58e-05]
Epoch 2 TRAIN:  39%|███▊      | 53/137 [00:10<00:12,  6.84it/s, epoch=2, grad_norm=70.073, loss=405.0877, lr=2.58e-05]
Epoch 2 TRAIN:  39%|███▊      | 53/137 [00:11<00:12,  6.84it/s, epoch=2, grad_norm=202.546, loss=356.3073, lr=2.59e-05]
Epoch 2 TRAIN:  39%|███▊      | 53/137 [00:11<00:12,  6.84it/s, epoch=2, grad_norm=194.214, loss=347.5725, lr=2.60e-05]
Epoch 2 TRAIN:  39%|███▊      | 53/137 [00:11<00:12,  6.84it/s, epoch=2, grad_norm=189.331, loss=381.1722, lr=2.61e-05]
Epoch 2 TRAIN:  42%|████▏     | 57/137 [00:11<00:11,  6.80it/s, epoch=2, grad_norm=189.331, loss=381.1722, lr=2.61e-05]
Epoch 2 TRAIN:  42%|████▏     | 57/137 [00:11<00:11,  6.80it/s, epoch=2, grad_norm=191.970, loss=350.7731, lr=2.62e-05]
Epoch 2 TRAIN:  42%|████▏     | 57/137 [00:11<00:11,  6.80it/s, epoch=2, grad_norm=204.595, loss=336.0725, lr=2.62e-05]
Epoch 2 TRAIN:  42%|████▏     | 57/137 [00:11<00:11,  6.80it/s, epoch=2, grad_norm=93.457, loss=364.4900, lr=2.63e-05] 
Epoch 2 TRAIN:  42%|████▏     | 57/137 [00:12<00:11,  6.80it/s, epoch=2, grad_norm=72.098, loss=373.3419, lr=2.64e-05]
Epoch 2 TRAIN:  45%|████▍     | 61/137 [00:12<00:11,  6.82it/s, epoch=2, grad_norm=72.098, loss=373.3419, lr=2.64e-05]
Epoch 2 TRAIN:  45%|████▍     | 61/137 [00:12<00:11,  6.82it/s, epoch=2, grad_norm=159.667, loss=412.1547, lr=2.65e-05]
Epoch 2 TRAIN:  45%|████▍     | 61/137 [00:12<00:11,  6.82it/s, epoch=2, grad_norm=78.380, loss=345.9730, lr=2.66e-05] 
Epoch 2 TRAIN:  45%|████▍     | 61/137 [00:12<00:11,  6.82it/s, epoch=2, grad_norm=82.847, loss=341.3025, lr=2.66e-05]
Epoch 2 TRAIN:  45%|████▍     | 61/137 [00:12<00:11,  6.82it/s, epoch=2, grad_norm=142.996, loss=387.8516, lr=2.67e-05]
Epoch 2 TRAIN:  47%|████▋     | 65/137 [00:12<00:10,  6.98it/s, epoch=2, grad_norm=142.996, loss=387.8516, lr=2.67e-05]
Epoch 2 TRAIN:  47%|████▋     | 65/137 [00:12<00:10,  6.98it/s, epoch=2, grad_norm=205.715, loss=411.6080, lr=2.68e-05]
Epoch 2 TRAIN:  47%|████▋     | 65/137 [00:12<00:10,  6.98it/s, epoch=2, grad_norm=67.921, loss=323.6416, lr=2.69e-05] 
Epoch 2 TRAIN:  47%|████▋     | 65/137 [00:12<00:10,  6.98it/s, epoch=2, grad_norm=173.621, loss=406.1861, lr=2.70e-05]
Epoch 2 TRAIN:  47%|████▋     | 65/137 [00:13<00:10,  6.98it/s, epoch=2, grad_norm=89.183, loss=379.2585, lr=2.70e-05] 
Epoch 2 TRAIN:  50%|█████     | 69/137 [00:13<00:09,  7.00it/s, epoch=2, grad_norm=89.183, loss=379.2585, lr=2.70e-05]
Epoch 2 TRAIN:  50%|█████     | 69/137 [00:13<00:09,  7.00it/s, epoch=2, grad_norm=89.231, loss=393.0845, lr=2.71e-05]
Epoch 2 TRAIN:  50%|█████     | 69/137 [00:13<00:09,  7.00it/s, epoch=2, grad_norm=129.862, loss=338.3445, lr=2.72e-05]
Epoch 2 TRAIN:  50%|█████     | 69/137 [00:13<00:09,  7.00it/s, epoch=2, grad_norm=75.008, loss=398.5533, lr=2.73e-05] 
Epoch 2 TRAIN:  50%|█████     | 69/137 [00:13<00:09,  7.00it/s, epoch=2, grad_norm=119.484, loss=368.7182, lr=2.74e-05]
Epoch 2 TRAIN:  53%|█████▎    | 73/137 [00:13<00:09,  6.94it/s, epoch=2, grad_norm=119.484, loss=368.7182, lr=2.74e-05]
Epoch 2 TRAIN:  53%|█████▎    | 73/137 [00:13<00:09,  6.94it/s, epoch=2, grad_norm=63.844, loss=393.2104, lr=2.74e-05] 
Epoch 2 TRAIN:  53%|█████▎    | 73/137 [00:14<00:09,  6.94it/s, epoch=2, grad_norm=241.164, loss=367.8821, lr=2.75e-05]
Epoch 2 TRAIN:  53%|█████▎    | 73/137 [00:14<00:09,  6.94it/s, epoch=2, grad_norm=72.647, loss=409.5721, lr=2.76e-05] 
Epoch 2 TRAIN:  53%|█████▎    | 73/137 [00:14<00:09,  6.94it/s, epoch=2, grad_norm=109.136, loss=385.7116, lr=2.77e-05]
Epoch 2 TRAIN:  56%|█████▌    | 77/137 [00:14<00:08,  6.78it/s, epoch=2, grad_norm=109.136, loss=385.7116, lr=2.77e-05]
Epoch 2 TRAIN:  56%|█████▌    | 77/137 [00:14<00:08,  6.78it/s, epoch=2, grad_norm=223.562, loss=373.3127, lr=2.78e-05]
Epoch 2 TRAIN:  56%|█████▌    | 77/137 [00:14<00:08,  6.78it/s, epoch=2, grad_norm=244.233, loss=294.2661, lr=2.78e-05]
Epoch 2 TRAIN:  56%|█████▌    | 77/137 [00:14<00:08,  6.78it/s, epoch=2, grad_norm=113.526, loss=304.1758, lr=2.79e-05]
Epoch 2 TRAIN:  56%|█████▌    | 77/137 [00:14<00:08,  6.78it/s, epoch=2, grad_norm=102.099, loss=347.2802, lr=2.80e-05]
Epoch 2 TRAIN:  59%|█████▉    | 81/137 [00:14<00:08,  6.78it/s, epoch=2, grad_norm=102.099, loss=347.2802, lr=2.80e-05]
Epoch 2 TRAIN:  59%|█████▉    | 81/137 [00:15<00:08,  6.78it/s, epoch=2, grad_norm=212.416, loss=398.8804, lr=2.81e-05]
Epoch 2 TRAIN:  59%|█████▉    | 81/137 [00:15<00:08,  6.78it/s, epoch=2, grad_norm=110.248, loss=379.3452, lr=2.82e-05]
Epoch 2 TRAIN:  59%|█████▉    | 81/137 [00:15<00:08,  6.78it/s, epoch=2, grad_norm=180.333, loss=393.9031, lr=2.82e-05]
Epoch 2 TRAIN:  59%|█████▉    | 81/137 [00:15<00:08,  6.78it/s, epoch=2, grad_norm=98.676, loss=350.9676, lr=2.83e-05] 
Epoch 2 TRAIN:  62%|██████▏   | 85/137 [00:15<00:07,  6.82it/s, epoch=2, grad_norm=98.676, loss=350.9676, lr=2.83e-05]
Epoch 2 TRAIN:  62%|██████▏   | 85/137 [00:15<00:07,  6.82it/s, epoch=2, grad_norm=128.973, loss=344.4559, lr=2.84e-05]
Epoch 2 TRAIN:  62%|██████▏   | 85/137 [00:15<00:07,  6.82it/s, epoch=2, grad_norm=117.245, loss=318.4995, lr=2.85e-05]
Epoch 2 TRAIN:  62%|██████▏   | 85/137 [00:15<00:07,  6.82it/s, epoch=2, grad_norm=110.594, loss=358.5454, lr=2.86e-05]
Epoch 2 TRAIN:  62%|██████▏   | 85/137 [00:16<00:07,  6.82it/s, epoch=2, grad_norm=210.999, loss=243.6878, lr=2.86e-05]
Epoch 2 TRAIN:  65%|██████▍   | 89/137 [00:16<00:07,  6.82it/s, epoch=2, grad_norm=210.999, loss=243.6878, lr=2.86e-05]
Epoch 2 TRAIN:  65%|██████▍   | 89/137 [00:16<00:07,  6.82it/s, epoch=2, grad_norm=81.397, loss=356.1828, lr=2.87e-05] 
Epoch 2 TRAIN:  65%|██████▍   | 89/137 [00:16<00:07,  6.82it/s, epoch=2, grad_norm=195.199, loss=306.1130, lr=2.88e-05]
Epoch 2 TRAIN:  65%|██████▍   | 89/137 [00:16<00:07,  6.82it/s, epoch=2, grad_norm=163.789, loss=438.7961, lr=2.89e-05]
Epoch 2 TRAIN:  65%|██████▍   | 89/137 [00:16<00:07,  6.82it/s, epoch=2, grad_norm=132.421, loss=322.5223, lr=2.90e-05]
Epoch 2 TRAIN:  68%|██████▊   | 93/137 [00:16<00:06,  6.88it/s, epoch=2, grad_norm=132.421, loss=322.5223, lr=2.90e-05]
Epoch 2 TRAIN:  68%|██████▊   | 93/137 [00:16<00:06,  6.88it/s, epoch=2, grad_norm=95.473, loss=422.6489, lr=2.90e-05] 
Epoch 2 TRAIN:  68%|██████▊   | 93/137 [00:16<00:06,  6.88it/s, epoch=2, grad_norm=210.746, loss=293.8253, lr=2.91e-05]
Epoch 2 TRAIN:  68%|██████▊   | 93/137 [00:17<00:06,  6.88it/s, epoch=2, grad_norm=142.380, loss=391.0229, lr=2.92e-05]
Epoch 2 TRAIN:  68%|██████▊   | 93/137 [00:17<00:06,  6.88it/s, epoch=2, grad_norm=153.112, loss=274.3636, lr=2.93e-05]
Epoch 2 TRAIN:  71%|███████   | 97/137 [00:17<00:05,  6.67it/s, epoch=2, grad_norm=153.112, loss=274.3636, lr=2.93e-05]
Epoch 2 TRAIN:  71%|███████   | 97/137 [00:17<00:05,  6.67it/s, epoch=2, grad_norm=88.816, loss=351.8722, lr=2.94e-05] 
Epoch 2 TRAIN:  71%|███████   | 97/137 [00:17<00:05,  6.67it/s, epoch=2, grad_norm=137.585, loss=311.3215, lr=2.94e-05]
Epoch 2 TRAIN:  71%|███████   | 97/137 [00:17<00:05,  6.67it/s, epoch=2, grad_norm=78.809, loss=364.2263, lr=2.95e-05] 2026-04-16 19:53:48,238 DEBUG TRAIN | steps/sec 5.747| Batch 2/100 loss 364.226318 loss_att 330.482849 loss_ctc 442.961029 th_accuracy 0.187793 lr 2.9520e-05 grad_norm 78.808853 rank 0

Epoch 2 TRAIN:  71%|███████   | 97/137 [00:17<00:05,  6.67it/s, epoch=2, grad_norm=146.430, loss=423.8324, lr=2.96e-05]
Epoch 2 TRAIN:  74%|███████▎  | 101/137 [00:17<00:05,  6.72it/s, epoch=2, grad_norm=146.430, loss=423.8324, lr=2.96e-05]
Epoch 2 TRAIN:  74%|███████▎  | 101/137 [00:18<00:05,  6.72it/s, epoch=2, grad_norm=97.425, loss=333.1378, lr=2.97e-05] 
Epoch 2 TRAIN:  74%|███████▎  | 101/137 [00:18<00:05,  6.72it/s, epoch=2, grad_norm=80.254, loss=355.5944, lr=2.98e-05]
Epoch 2 TRAIN:  74%|███████▎  | 101/137 [00:18<00:05,  6.72it/s, epoch=2, grad_norm=64.512, loss=315.3494, lr=2.98e-05]
Epoch 2 TRAIN:  74%|███████▎  | 101/137 [00:18<00:05,  6.72it/s, epoch=2, grad_norm=84.888, loss=351.8209, lr=2.99e-05]
Epoch 2 TRAIN:  77%|███████▋  | 105/137 [00:18<00:04,  6.74it/s, epoch=2, grad_norm=84.888, loss=351.8209, lr=2.99e-05]
Epoch 2 TRAIN:  77%|███████▋  | 105/137 [00:18<00:04,  6.74it/s, epoch=2, grad_norm=107.836, loss=294.1891, lr=3.00e-05]
Epoch 2 TRAIN:  77%|███████▋  | 105/137 [00:18<00:04,  6.74it/s, epoch=2, grad_norm=310.660, loss=279.7253, lr=3.01e-05]
Epoch 2 TRAIN:  77%|███████▋  | 105/137 [00:18<00:04,  6.74it/s, epoch=2, grad_norm=106.144, loss=324.7871, lr=3.02e-05]
Epoch 2 TRAIN:  77%|███████▋  | 105/137 [00:19<00:04,  6.74it/s, epoch=2, grad_norm=84.406, loss=313.9230, lr=3.02e-05] 
Epoch 2 TRAIN:  80%|███████▉  | 109/137 [00:19<00:04,  6.66it/s, epoch=2, grad_norm=84.406, loss=313.9230, lr=3.02e-05]
Epoch 2 TRAIN:  80%|███████▉  | 109/137 [00:19<00:04,  6.66it/s, epoch=2, grad_norm=148.825, loss=418.4496, lr=3.03e-05]
Epoch 2 TRAIN:  80%|███████▉  | 109/137 [00:19<00:04,  6.66it/s, epoch=2, grad_norm=135.394, loss=344.6219, lr=3.04e-05]
Epoch 2 TRAIN:  80%|███████▉  | 109/137 [00:19<00:04,  6.66it/s, epoch=2, grad_norm=143.361, loss=346.2707, lr=3.05e-05]
Epoch 2 TRAIN:  80%|███████▉  | 109/137 [00:19<00:04,  6.66it/s, epoch=2, grad_norm=132.070, loss=257.5270, lr=3.06e-05]
Epoch 2 TRAIN:  82%|████████▏ | 113/137 [00:19<00:03,  6.73it/s, epoch=2, grad_norm=132.070, loss=257.5270, lr=3.06e-05]
Epoch 2 TRAIN:  82%|████████▏ | 113/137 [00:19<00:03,  6.73it/s, epoch=2, grad_norm=75.811, loss=320.6458, lr=3.06e-05] 
Epoch 2 TRAIN:  82%|████████▏ | 113/137 [00:19<00:03,  6.73it/s, epoch=2, grad_norm=135.725, loss=414.3160, lr=3.07e-05]
Epoch 2 TRAIN:  82%|████████▏ | 113/137 [00:20<00:03,  6.73it/s, epoch=2, grad_norm=187.641, loss=435.8717, lr=3.08e-05]
Epoch 2 TRAIN:  82%|████████▏ | 113/137 [00:20<00:03,  6.73it/s, epoch=2, grad_norm=75.496, loss=334.1492, lr=3.09e-05] 
Epoch 2 TRAIN:  85%|████████▌ | 117/137 [00:20<00:02,  6.81it/s, epoch=2, grad_norm=75.496, loss=334.1492, lr=3.09e-05]
Epoch 2 TRAIN:  85%|████████▌ | 117/137 [00:20<00:02,  6.81it/s, epoch=2, grad_norm=97.338, loss=462.8492, lr=3.10e-05]
Epoch 2 TRAIN:  85%|████████▌ | 117/137 [00:20<00:02,  6.81it/s, epoch=2, grad_norm=209.610, loss=281.5270, lr=3.10e-05]
Epoch 2 TRAIN:  85%|████████▌ | 117/137 [00:20<00:02,  6.81it/s, epoch=2, grad_norm=90.474, loss=405.3513, lr=3.11e-05] 
Epoch 2 TRAIN:  85%|████████▌ | 117/137 [00:20<00:02,  6.81it/s, epoch=2, grad_norm=92.625, loss=349.0737, lr=3.12e-05]
Epoch 2 TRAIN:  88%|████████▊ | 121/137 [00:20<00:02,  6.72it/s, epoch=2, grad_norm=92.625, loss=349.0737, lr=3.12e-05]
Epoch 2 TRAIN:  88%|████████▊ | 121/137 [00:20<00:02,  6.72it/s, epoch=2, grad_norm=109.487, loss=420.4062, lr=3.13e-05]
Epoch 2 TRAIN:  88%|████████▊ | 121/137 [00:21<00:02,  6.72it/s, epoch=2, grad_norm=158.371, loss=363.9236, lr=3.14e-05]
Epoch 2 TRAIN:  88%|████████▊ | 121/137 [00:21<00:02,  6.72it/s, epoch=2, grad_norm=175.854, loss=352.3251, lr=3.14e-05]
Epoch 2 TRAIN:  88%|████████▊ | 121/137 [00:21<00:02,  6.72it/s, epoch=2, grad_norm=114.021, loss=380.7573, lr=3.15e-05]
Epoch 2 TRAIN:  91%|█████████ | 125/137 [00:21<00:01,  6.73it/s, epoch=2, grad_norm=114.021, loss=380.7573, lr=3.15e-05]
Epoch 2 TRAIN:  91%|█████████ | 125/137 [00:21<00:01,  6.73it/s, epoch=2, grad_norm=358.412, loss=344.0544, lr=3.16e-05]
Epoch 2 TRAIN:  91%|█████████ | 125/137 [00:21<00:01,  6.73it/s, epoch=2, grad_norm=132.448, loss=383.0330, lr=3.17e-05]
Epoch 2 TRAIN:  91%|█████████ | 125/137 [00:21<00:01,  6.73it/s, epoch=2, grad_norm=107.619, loss=411.7889, lr=3.18e-05]
Epoch 2 TRAIN:  91%|█████████ | 125/137 [00:21<00:01,  6.73it/s, epoch=2, grad_norm=103.397, loss=321.3163, lr=3.18e-05]
Epoch 2 TRAIN:  94%|█████████▍| 129/137 [00:21<00:01,  6.87it/s, epoch=2, grad_norm=103.397, loss=321.3163, lr=3.18e-05]
Epoch 2 TRAIN:  94%|█████████▍| 129/137 [00:22<00:01,  6.87it/s, epoch=2, grad_norm=97.577, loss=404.8799, lr=3.19e-05] 
Epoch 2 TRAIN:  94%|█████████▍| 129/137 [00:22<00:01,  6.87it/s, epoch=2, grad_norm=140.465, loss=423.6797, lr=3.20e-05]
Epoch 2 TRAIN:  94%|█████████▍| 129/137 [00:22<00:01,  6.87it/s, epoch=2, grad_norm=134.636, loss=417.5811, lr=3.21e-05]
Epoch 2 TRAIN:  94%|█████████▍| 129/137 [00:22<00:01,  6.87it/s, epoch=2, grad_norm=86.195, loss=356.3398, lr=3.22e-05] 
Epoch 2 TRAIN:  97%|█████████▋| 133/137 [00:22<00:00,  6.97it/s, epoch=2, grad_norm=86.195, loss=356.3398, lr=3.22e-05]
Epoch 2 TRAIN:  97%|█████████▋| 133/137 [00:22<00:00,  6.97it/s, epoch=2, grad_norm=204.190, loss=496.3423, lr=3.22e-05]
Epoch 2 TRAIN:  97%|█████████▋| 133/137 [00:22<00:00,  6.97it/s, epoch=2, grad_norm=167.411, loss=398.8883, lr=3.23e-05]
Epoch 2 TRAIN:  97%|█████████▋| 133/137 [00:22<00:00,  6.97it/s, epoch=2, grad_norm=133.843, loss=447.3652, lr=3.24e-05]
Epoch 2 TRAIN:  99%|█████████▉| 136/137 [00:22<00:00,  5.94it/s, epoch=2, grad_norm=133.843, loss=447.3652, lr=3.24e-05]

CV:   0%|          | 0/6 [00:00<?, ?it/s]/home/rookie/Documents/batuka/projects/wenet/wenet/wenet/utils/train_utils.py:689: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  amp_autocast(enabled=dtype is not None,

CV:  17%|█▋        | 1/6 [00:00<00:03,  1.31it/s]
                                                 
2026-04-16 19:53:54,272 INFO Epoch 2 Step 405 CV info lr 3.2400e-05 cv_loss 414.07164140181106 rank 0 acc 0.20954926063617071
2026-04-16 19:53:54,273 INFO [Rank 0] Checkpoint: save to checkpoint exp/conformer_mongolian/epoch_2.pt
2026-04-16 19:53:54,383 INFO Epoch 3 Step 405 TRAIN info lr 3.2400e-05 rank 0
2026-04-16 19:53:54,396 INFO using accumulate grad, new batch size is 1 times larger than before

Epoch 3 TRAIN:   0%|          | 0/136 [00:00<?, ?it/s]
Epoch 3 TRAIN:   0%|          | 0/136 [00:04<?, ?it/s, epoch=3, grad_norm=681.082, loss=517.4402, lr=3.25e-05]
Epoch 3 TRAIN:   1%|          | 1/136 [00:04<09:27,  4.20s/it, epoch=3, grad_norm=681.082, loss=517.4402, lr=3.25e-05]
Epoch 3 TRAIN:   1%|          | 1/136 [00:04<09:27,  4.20s/it, epoch=3, grad_norm=574.188, loss=480.9809, lr=3.26e-05]
Epoch 3 TRAIN:   1%|          | 1/136 [00:04<09:27,  4.20s/it, epoch=3, grad_norm=482.984, loss=412.4927, lr=3.26e-05]
Epoch 3 TRAIN:   1%|          | 1/136 [00:04<09:27,  4.20s/it, epoch=3, grad_norm=421.182, loss=395.7204, lr=3.27e-05]
Epoch 3 TRAIN:   1%|          | 1/136 [00:04<09:27,  4.20s/it, epoch=3, grad_norm=395.126, loss=401.8115, lr=3.28e-05]
Epoch 3 TRAIN:   1%|          | 1/136 [00:04<09:27,  4.20s/it, epoch=3, grad_norm=150.145, loss=302.0592, lr=3.29e-05]
Epoch 3 TRAIN:   1%|          | 1/136 [00:04<09:27,  4.20s/it, epoch=3, grad_norm=260.755, loss=379.5174, lr=3.30e-05]
Epoch 3 TRAIN:   1%|          | 1/136 [00:04<09:27,  4.20s/it, epoch=3, grad_norm=96.513, loss=306.3658, lr=3.30e-05] 
Epoch 3 TRAIN:   1%|          | 1/136 [00:04<09:27,  4.20s/it, epoch=3, grad_norm=96.476, loss=382.8897, lr=3.31e-05]
Epoch 3 TRAIN:   7%|▋         | 9/136 [00:04<00:51,  2.46it/s, epoch=3, grad_norm=96.476, loss=382.8897, lr=3.31e-05]
Epoch 3 TRAIN:   7%|▋         | 9/136 [00:04<00:51,  2.46it/s, epoch=3, grad_norm=61.052, loss=384.2447, lr=3.32e-05]
Epoch 3 TRAIN:   7%|▋         | 9/136 [00:05<00:51,  2.46it/s, epoch=3, grad_norm=148.102, loss=369.4876, lr=3.33e-05]
Epoch 3 TRAIN:   7%|▋         | 9/136 [00:05<00:51,  2.46it/s, epoch=3, grad_norm=208.658, loss=365.7505, lr=3.34e-05]
Epoch 3 TRAIN:   7%|▋         | 9/136 [00:05<00:51,  2.46it/s, epoch=3, grad_norm=190.274, loss=379.0261, lr=3.34e-05]
Epoch 3 TRAIN:  10%|▉         | 13/136 [00:05<00:37,  3.30it/s, epoch=3, grad_norm=190.274, loss=379.0261, lr=3.34e-05]
Epoch 3 TRAIN:  10%|▉         | 13/136 [00:05<00:37,  3.30it/s, epoch=3, grad_norm=317.467, loss=327.7883, lr=3.35e-05]
Epoch 3 TRAIN:  10%|▉         | 13/136 [00:05<00:37,  3.30it/s, epoch=3, grad_norm=196.692, loss=361.6382, lr=3.36e-05]
Epoch 3 TRAIN:  10%|▉         | 13/136 [00:05<00:37,  3.30it/s, epoch=3, grad_norm=119.902, loss=396.8271, lr=3.37e-05]
Epoch 3 TRAIN:  10%|▉         | 13/136 [00:05<00:37,  3.30it/s, epoch=3, grad_norm=159.825, loss=398.7548, lr=3.38e-05]
Epoch 3 TRAIN:  12%|█▎        | 17/136 [00:05<00:28,  4.11it/s, epoch=3, grad_norm=159.825, loss=398.7548, lr=3.38e-05]
Epoch 3 TRAIN:  12%|█▎        | 17/136 [00:06<00:28,  4.11it/s, epoch=3, grad_norm=144.020, loss=379.2409, lr=3.38e-05]
Epoch 3 TRAIN:  12%|█▎        | 17/136 [00:06<00:28,  4.11it/s, epoch=3, grad_norm=137.130, loss=359.6187, lr=3.39e-05]
Epoch 3 TRAIN:  12%|█▎        | 17/136 [00:06<00:28,  4.11it/s, epoch=3, grad_norm=114.514, loss=336.4821, lr=3.40e-05]
Epoch 3 TRAIN:  12%|█▎        | 17/136 [00:06<00:28,  4.11it/s, epoch=3, grad_norm=48.141, loss=347.6840, lr=3.41e-05] 
Epoch 3 TRAIN:  15%|█▌        | 21/136 [00:06<00:24,  4.77it/s, epoch=3, grad_norm=48.141, loss=347.6840, lr=3.41e-05]
Epoch 3 TRAIN:  15%|█▌        | 21/136 [00:06<00:24,  4.77it/s, epoch=3, grad_norm=239.797, loss=421.1093, lr=3.42e-05]
Epoch 3 TRAIN:  15%|█▌        | 21/136 [00:06<00:24,  4.77it/s, epoch=3, grad_norm=211.193, loss=390.1653, lr=3.42e-05]
Epoch 3 TRAIN:  15%|█▌        | 21/136 [00:06<00:24,  4.77it/s, epoch=3, grad_norm=219.424, loss=381.9709, lr=3.43e-05]
Epoch 3 TRAIN:  15%|█▌        | 21/136 [00:07<00:24,  4.77it/s, epoch=3, grad_norm=61.737, loss=317.1701, lr=3.44e-05] 
Epoch 3 TRAIN:  18%|█▊        | 25/136 [00:07<00:20,  5.39it/s, epoch=3, grad_norm=61.737, loss=317.1701, lr=3.44e-05]
Epoch 3 TRAIN:  18%|█▊        | 25/136 [00:07<00:20,  5.39it/s, epoch=3, grad_norm=145.137, loss=368.8906, lr=3.45e-05]
Epoch 3 TRAIN:  18%|█▊        | 25/136 [00:07<00:20,  5.39it/s, epoch=3, grad_norm=163.334, loss=378.8928, lr=3.46e-05]
Epoch 3 TRAIN:  18%|█▊        | 25/136 [00:07<00:20,  5.39it/s, epoch=3, grad_norm=75.728, loss=342.6912, lr=3.46e-05] 
Epoch 3 TRAIN:  18%|█▊        | 25/136 [00:07<00:20,  5.39it/s, epoch=3, grad_norm=109.702, loss=284.6640, lr=3.47e-05]
Epoch 3 TRAIN:  21%|██▏       | 29/136 [00:07<00:18,  5.85it/s, epoch=3, grad_norm=109.702, loss=284.6640, lr=3.47e-05]
Epoch 3 TRAIN:  21%|██▏       | 29/136 [00:07<00:18,  5.85it/s, epoch=3, grad_norm=57.501, loss=349.7367, lr=3.48e-05] 
Epoch 3 TRAIN:  21%|██▏       | 29/136 [00:07<00:18,  5.85it/s, epoch=3, grad_norm=76.815, loss=339.5324, lr=3.49e-05]
Epoch 3 TRAIN:  21%|██▏       | 29/136 [00:08<00:18,  5.85it/s, epoch=3, grad_norm=100.492, loss=382.5876, lr=3.50e-05]
Epoch 3 TRAIN:  21%|██▏       | 29/136 [00:08<00:18,  5.85it/s, epoch=3, grad_norm=76.134, loss=395.1140, lr=3.50e-05] 
Epoch 3 TRAIN:  24%|██▍       | 33/136 [00:08<00:16,  6.21it/s, epoch=3, grad_norm=76.134, loss=395.1140, lr=3.50e-05]
Epoch 3 TRAIN:  24%|██▍       | 33/136 [00:08<00:16,  6.21it/s, epoch=3, grad_norm=97.046, loss=359.2779, lr=3.51e-05]
Epoch 3 TRAIN:  24%|██▍       | 33/136 [00:08<00:16,  6.21it/s, epoch=3, grad_norm=170.871, loss=319.0045, lr=3.52e-05]
Epoch 3 TRAIN:  24%|██▍       | 33/136 [00:08<00:16,  6.21it/s, epoch=3, grad_norm=221.419, loss=317.7894, lr=3.53e-05]
Epoch 3 TRAIN:  24%|██▍       | 33/136 [00:08<00:16,  6.21it/s, epoch=3, grad_norm=99.685, loss=343.5377, lr=3.54e-05] 
Epoch 3 TRAIN:  27%|██▋       | 37/136 [00:08<00:15,  6.44it/s, epoch=3, grad_norm=99.685, loss=343.5377, lr=3.54e-05]
Epoch 3 TRAIN:  27%|██▋       | 37/136 [00:08<00:15,  6.44it/s, epoch=3, grad_norm=78.617, loss=348.2677, lr=3.54e-05]
Epoch 3 TRAIN:  27%|██▋       | 37/136 [00:09<00:15,  6.44it/s, epoch=3, grad_norm=65.400, loss=373.9006, lr=3.55e-05]
Epoch 3 TRAIN:  27%|██▋       | 37/136 [00:09<00:15,  6.44it/s, epoch=3, grad_norm=60.139, loss=351.7820, lr=3.56e-05]
Epoch 3 TRAIN:  27%|██▋       | 37/136 [00:09<00:15,  6.44it/s, epoch=3, grad_norm=64.953, loss=326.7189, lr=3.57e-05]
Epoch 3 TRAIN:  30%|███       | 41/136 [00:09<00:14,  6.63it/s, epoch=3, grad_norm=64.953, loss=326.7189, lr=3.57e-05]
Epoch 3 TRAIN:  30%|███       | 41/136 [00:09<00:14,  6.63it/s, epoch=3, grad_norm=135.061, loss=390.2552, lr=3.58e-05]
Epoch 3 TRAIN:  30%|███       | 41/136 [00:09<00:14,  6.63it/s, epoch=3, grad_norm=117.121, loss=394.7774, lr=3.58e-05]
Epoch 3 TRAIN:  30%|███       | 41/136 [00:09<00:14,  6.63it/s, epoch=3, grad_norm=62.096, loss=376.4026, lr=3.59e-05] 
Epoch 3 TRAIN:  30%|███       | 41/136 [00:09<00:14,  6.63it/s, epoch=3, grad_norm=92.490, loss=319.6982, lr=3.60e-05]
Epoch 3 TRAIN:  33%|███▎      | 45/136 [00:09<00:13,  6.71it/s, epoch=3, grad_norm=92.490, loss=319.6982, lr=3.60e-05]
Epoch 3 TRAIN:  33%|███▎      | 45/136 [00:10<00:13,  6.71it/s, epoch=3, grad_norm=102.819, loss=317.8561, lr=3.61e-05]
Epoch 3 TRAIN:  33%|███▎      | 45/136 [00:10<00:13,  6.71it/s, epoch=3, grad_norm=73.326, loss=361.5347, lr=3.62e-05] 
Epoch 3 TRAIN:  33%|███▎      | 45/136 [00:10<00:13,  6.71it/s, epoch=3, grad_norm=62.921, loss=312.5287, lr=3.62e-05]
Epoch 3 TRAIN:  33%|███▎      | 45/136 [00:10<00:13,  6.71it/s, epoch=3, grad_norm=46.634, loss=354.6075, lr=3.63e-05]
Epoch 3 TRAIN:  36%|███▌      | 49/136 [00:10<00:12,  6.76it/s, epoch=3, grad_norm=46.634, loss=354.6075, lr=3.63e-05]
Epoch 3 TRAIN:  36%|███▌      | 49/136 [00:10<00:12,  6.76it/s, epoch=3, grad_norm=89.898, loss=376.9139, lr=3.64e-05]
Epoch 3 TRAIN:  36%|███▌      | 49/136 [00:10<00:12,  6.76it/s, epoch=3, grad_norm=103.395, loss=392.2015, lr=3.65e-05]
Epoch 3 TRAIN:  36%|███▌      | 49/136 [00:10<00:12,  6.76it/s, epoch=3, grad_norm=67.437, loss=368.1650, lr=3.66e-05] 
Epoch 3 TRAIN:  36%|███▌      | 49/136 [00:11<00:12,  6.76it/s, epoch=3, grad_norm=150.436, loss=316.1014, lr=3.66e-05]
Epoch 3 TRAIN:  39%|███▉      | 53/136 [00:11<00:12,  6.77it/s, epoch=3, grad_norm=150.436, loss=316.1014, lr=3.66e-05]
Epoch 3 TRAIN:  39%|███▉      | 53/136 [00:11<00:12,  6.77it/s, epoch=3, grad_norm=63.866, loss=362.3130, lr=3.67e-05] 
Epoch 3 TRAIN:  39%|███▉      | 53/136 [00:11<00:12,  6.77it/s, epoch=3, grad_norm=61.129, loss=343.2296, lr=3.68e-05]
Epoch 3 TRAIN:  39%|███▉      | 53/136 [00:11<00:12,  6.77it/s, epoch=3, grad_norm=55.728, loss=355.6208, lr=3.69e-05]
Epoch 3 TRAIN:  39%|███▉      | 53/136 [00:11<00:12,  6.77it/s, epoch=3, grad_norm=95.558, loss=353.5991, lr=3.70e-05]
Epoch 3 TRAIN:  42%|████▏     | 57/136 [00:11<00:11,  6.84it/s, epoch=3, grad_norm=95.558, loss=353.5991, lr=3.70e-05]
Epoch 3 TRAIN:  42%|████▏     | 57/136 [00:11<00:11,  6.84it/s, epoch=3, grad_norm=58.038, loss=366.8364, lr=3.70e-05]
Epoch 3 TRAIN:  42%|████▏     | 57/136 [00:11<00:11,  6.84it/s, epoch=3, grad_norm=65.176, loss=365.7650, lr=3.71e-05]
Epoch 3 TRAIN:  42%|████▏     | 57/136 [00:12<00:11,  6.84it/s, epoch=3, grad_norm=87.929, loss=351.3873, lr=3.72e-05]
Epoch 3 TRAIN:  42%|████▏     | 57/136 [00:12<00:11,  6.84it/s, epoch=3, grad_norm=57.311, loss=336.8495, lr=3.73e-05]
Epoch 3 TRAIN:  45%|████▍     | 61/136 [00:12<00:10,  6.89it/s, epoch=3, grad_norm=57.311, loss=336.8495, lr=3.73e-05]
Epoch 3 TRAIN:  45%|████▍     | 61/136 [00:12<00:10,  6.89it/s, epoch=3, grad_norm=93.204, loss=370.7350, lr=3.74e-05]
Epoch 3 TRAIN:  45%|████▍     | 61/136 [00:12<00:10,  6.89it/s, epoch=3, grad_norm=131.302, loss=372.5170, lr=3.74e-05]
Epoch 3 TRAIN:  45%|████▍     | 61/136 [00:12<00:10,  6.89it/s, epoch=3, grad_norm=137.577, loss=360.5223, lr=3.75e-05]
Epoch 3 TRAIN:  45%|████▍     | 61/136 [00:12<00:10,  6.89it/s, epoch=3, grad_norm=69.453, loss=308.5343, lr=3.76e-05] 
Epoch 3 TRAIN:  48%|████▊     | 65/136 [00:12<00:10,  6.93it/s, epoch=3, grad_norm=69.453, loss=308.5343, lr=3.76e-05]
Epoch 3 TRAIN:  48%|████▊     | 65/136 [00:12<00:10,  6.93it/s, epoch=3, grad_norm=93.856, loss=373.5446, lr=3.77e-05]
Epoch 3 TRAIN:  48%|████▊     | 65/136 [00:13<00:10,  6.93it/s, epoch=3, grad_norm=78.303, loss=329.9224, lr=3.78e-05]
Epoch 3 TRAIN:  48%|████▊     | 65/136 [00:13<00:10,  6.93it/s, epoch=3, grad_norm=82.346, loss=420.3594, lr=3.78e-05]
Epoch 3 TRAIN:  48%|████▊     | 65/136 [00:13<00:10,  6.93it/s, epoch=3, grad_norm=161.017, loss=297.0679, lr=3.79e-05]
Epoch 3 TRAIN:  51%|█████     | 69/136 [00:13<00:09,  6.89it/s, epoch=3, grad_norm=161.017, loss=297.0679, lr=3.79e-05]
Epoch 3 TRAIN:  51%|█████     | 69/136 [00:13<00:09,  6.89it/s, epoch=3, grad_norm=98.768, loss=328.6157, lr=3.80e-05] 
Epoch 3 TRAIN:  51%|█████     | 69/136 [00:13<00:09,  6.89it/s, epoch=3, grad_norm=202.316, loss=413.9185, lr=3.81e-05]
Epoch 3 TRAIN:  51%|█████     | 69/136 [00:13<00:09,  6.89it/s, epoch=3, grad_norm=94.295, loss=359.3658, lr=3.82e-05] 
Epoch 3 TRAIN:  51%|█████     | 69/136 [00:13<00:09,  6.89it/s, epoch=3, grad_norm=134.173, loss=400.2898, lr=3.82e-05]
Epoch 3 TRAIN:  54%|█████▎    | 73/136 [00:13<00:09,  6.87it/s, epoch=3, grad_norm=134.173, loss=400.2898, lr=3.82e-05]
Epoch 3 TRAIN:  54%|█████▎    | 73/136 [00:14<00:09,  6.87it/s, epoch=3, grad_norm=145.796, loss=300.8536, lr=3.83e-05]
Epoch 3 TRAIN:  54%|█████▎    | 73/136 [00:14<00:09,  6.87it/s, epoch=3, grad_norm=279.558, loss=277.5640, lr=3.84e-05]
Epoch 3 TRAIN:  54%|█████▎    | 73/136 [00:14<00:09,  6.87it/s, epoch=3, grad_norm=156.674, loss=400.6224, lr=3.85e-05]
Epoch 3 TRAIN:  54%|█████▎    | 73/136 [00:14<00:09,  6.87it/s, epoch=3, grad_norm=147.197, loss=302.5926, lr=3.86e-05]
Epoch 3 TRAIN:  57%|█████▋    | 77/136 [00:14<00:08,  6.83it/s, epoch=3, grad_norm=147.197, loss=302.5926, lr=3.86e-05]
Epoch 3 TRAIN:  57%|█████▋    | 77/136 [00:14<00:08,  6.83it/s, epoch=3, grad_norm=93.206, loss=352.2131, lr=3.86e-05] 
Epoch 3 TRAIN:  57%|█████▋    | 77/136 [00:14<00:08,  6.83it/s, epoch=3, grad_norm=122.960, loss=359.5521, lr=3.87e-05]
Epoch 3 TRAIN:  57%|█████▋    | 77/136 [00:14<00:08,  6.83it/s, epoch=3, grad_norm=150.581, loss=368.8956, lr=3.88e-05]
Epoch 3 TRAIN:  57%|█████▋    | 77/136 [00:15<00:08,  6.83it/s, epoch=3, grad_norm=60.999, loss=333.6922, lr=3.89e-05] 
Epoch 3 TRAIN:  60%|█████▉    | 81/136 [00:15<00:08,  6.76it/s, epoch=3, grad_norm=60.999, loss=333.6922, lr=3.89e-05]
Epoch 3 TRAIN:  60%|█████▉    | 81/136 [00:15<00:08,  6.76it/s, epoch=3, grad_norm=93.676, loss=357.2338, lr=3.90e-05]
Epoch 3 TRAIN:  60%|█████▉    | 81/136 [00:15<00:08,  6.76it/s, epoch=3, grad_norm=58.371, loss=362.3820, lr=3.90e-05]
Epoch 3 TRAIN:  60%|█████▉    | 81/136 [00:15<00:08,  6.76it/s, epoch=3, grad_norm=291.909, loss=321.9001, lr=3.91e-05]
Epoch 3 TRAIN:  60%|█████▉    | 81/136 [00:15<00:08,  6.76it/s, epoch=3, grad_norm=88.341, loss=376.6595, lr=3.92e-05] 
Epoch 3 TRAIN:  62%|██████▎   | 85/136 [00:15<00:07,  6.78it/s, epoch=3, grad_norm=88.341, loss=376.6595, lr=3.92e-05]
Epoch 3 TRAIN:  62%|██████▎   | 85/136 [00:15<00:07,  6.78it/s, epoch=3, grad_norm=133.795, loss=439.6610, lr=3.93e-05]
Epoch 3 TRAIN:  62%|██████▎   | 85/136 [00:16<00:07,  6.78it/s, epoch=3, grad_norm=81.419, loss=376.4736, lr=3.94e-05] 
Epoch 3 TRAIN:  62%|██████▎   | 85/136 [00:16<00:07,  6.78it/s, epoch=3, grad_norm=91.232, loss=365.6507, lr=3.94e-05]
Epoch 3 TRAIN:  62%|██████▎   | 85/136 [00:16<00:07,  6.78it/s, epoch=3, grad_norm=62.812, loss=375.0868, lr=3.95e-05]
Epoch 3 TRAIN:  65%|██████▌   | 89/136 [00:16<00:06,  6.76it/s, epoch=3, grad_norm=62.812, loss=375.0868, lr=3.95e-05]
Epoch 3 TRAIN:  65%|██████▌   | 89/136 [00:16<00:06,  6.76it/s, epoch=3, grad_norm=292.304, loss=251.4745, lr=3.96e-05]
Epoch 3 TRAIN:  65%|██████▌   | 89/136 [00:16<00:06,  6.76it/s, epoch=3, grad_norm=277.821, loss=344.7906, lr=3.97e-05]
Epoch 3 TRAIN:  65%|██████▌   | 89/136 [00:16<00:06,  6.76it/s, epoch=3, grad_norm=378.196, loss=357.8521, lr=3.98e-05]
Epoch 3 TRAIN:  65%|██████▌   | 89/136 [00:16<00:06,  6.76it/s, epoch=3, grad_norm=497.037, loss=269.8371, lr=3.98e-05]
Epoch 3 TRAIN:  68%|██████▊   | 93/136 [00:16<00:06,  6.73it/s, epoch=3, grad_norm=497.037, loss=269.8371, lr=3.98e-05]
Epoch 3 TRAIN:  68%|██████▊   | 93/136 [00:17<00:06,  6.73it/s, epoch=3, grad_norm=272.221, loss=343.6060, lr=3.99e-05]
Epoch 3 TRAIN:  68%|██████▊   | 93/136 [00:17<00:06,  6.73it/s, epoch=3, grad_norm=216.803, loss=274.6760, lr=4.00e-05]
Epoch 3 TRAIN:  68%|██████▊   | 93/136 [00:17<00:06,  6.73it/s, epoch=3, grad_norm=81.896, loss=325.4493, lr=4.01e-05] 
Epoch 3 TRAIN:  68%|██████▊   | 93/136 [00:17<00:06,  6.73it/s, epoch=3, grad_norm=107.057, loss=328.9566, lr=4.02e-05]
Epoch 3 TRAIN:  71%|███████▏  | 97/136 [00:17<00:05,  6.75it/s, epoch=3, grad_norm=107.057, loss=328.9566, lr=4.02e-05]
Epoch 3 TRAIN:  71%|███████▏  | 97/136 [00:17<00:05,  6.75it/s, epoch=3, grad_norm=85.843, loss=311.5074, lr=4.02e-05] 
Epoch 3 TRAIN:  71%|███████▏  | 97/136 [00:17<00:05,  6.75it/s, epoch=3, grad_norm=138.662, loss=264.9487, lr=4.03e-05]
Epoch 3 TRAIN:  71%|███████▏  | 97/136 [00:17<00:05,  6.75it/s, epoch=3, grad_norm=138.514, loss=291.6838, lr=4.04e-05]2026-04-16 19:54:12,352 DEBUG TRAIN | steps/sec 5.640| Batch 3/100 loss 291.683838 loss_att 258.652191 loss_ctc 368.757629 th_accuracy 0.256267 lr 4.0400e-05 grad_norm 138.514389 rank 0

Epoch 3 TRAIN:  71%|███████▏  | 97/136 [00:18<00:05,  6.75it/s, epoch=3, grad_norm=79.016, loss=342.8719, lr=4.05e-05] 
Epoch 3 TRAIN:  74%|███████▍  | 101/136 [00:18<00:05,  6.68it/s, epoch=3, grad_norm=79.016, loss=342.8719, lr=4.05e-05]
Epoch 3 TRAIN:  74%|███████▍  | 101/136 [00:18<00:05,  6.68it/s, epoch=3, grad_norm=58.378, loss=327.2508, lr=4.06e-05]
Epoch 3 TRAIN:  74%|███████▍  | 101/136 [00:18<00:05,  6.68it/s, epoch=3, grad_norm=460.108, loss=483.2861, lr=4.06e-05]
Epoch 3 TRAIN:  74%|███████▍  | 101/136 [00:18<00:05,  6.68it/s, epoch=3, grad_norm=256.575, loss=344.5706, lr=4.07e-05]
Epoch 3 TRAIN:  74%|███████▍  | 101/136 [00:18<00:05,  6.68it/s, epoch=3, grad_norm=215.886, loss=356.7969, lr=4.08e-05]
Epoch 3 TRAIN:  77%|███████▋  | 105/136 [00:18<00:04,  6.66it/s, epoch=3, grad_norm=215.886, loss=356.7969, lr=4.08e-05]
Epoch 3 TRAIN:  77%|███████▋  | 105/136 [00:18<00:04,  6.66it/s, epoch=3, grad_norm=72.844, loss=272.6462, lr=4.09e-05] 
Epoch 3 TRAIN:  77%|███████▋  | 105/136 [00:19<00:04,  6.66it/s, epoch=3, grad_norm=305.408, loss=460.7867, lr=4.10e-05]
Epoch 3 TRAIN:  77%|███████▋  | 105/136 [00:19<00:04,  6.66it/s, epoch=3, grad_norm=183.819, loss=283.7702, lr=4.10e-05]
Epoch 3 TRAIN:  77%|███████▋  | 105/136 [00:19<00:04,  6.66it/s, epoch=3, grad_norm=127.835, loss=437.7385, lr=4.11e-05]
Epoch 3 TRAIN:  80%|████████  | 109/136 [00:19<00:04,  6.70it/s, epoch=3, grad_norm=127.835, loss=437.7385, lr=4.11e-05]
Epoch 3 TRAIN:  80%|████████  | 109/136 [00:19<00:04,  6.70it/s, epoch=3, grad_norm=144.026, loss=331.0197, lr=4.12e-05]
Epoch 3 TRAIN:  80%|████████  | 109/136 [00:19<00:04,  6.70it/s, epoch=3, grad_norm=65.904, loss=346.0414, lr=4.13e-05] 
Epoch 3 TRAIN:  80%|████████  | 109/136 [00:19<00:04,  6.70it/s, epoch=3, grad_norm=159.035, loss=244.5631, lr=4.14e-05]
Epoch 3 TRAIN:  80%|████████  | 109/136 [00:19<00:04,  6.70it/s, epoch=3, grad_norm=152.147, loss=275.1184, lr=4.14e-05]
Epoch 3 TRAIN:  83%|████████▎ | 113/136 [00:19<00:03,  6.77it/s, epoch=3, grad_norm=152.147, loss=275.1184, lr=4.14e-05]
Epoch 3 TRAIN:  83%|████████▎ | 113/136 [00:20<00:03,  6.77it/s, epoch=3, grad_norm=118.039, loss=270.9801, lr=4.15e-05]
Epoch 3 TRAIN:  83%|████████▎ | 113/136 [00:20<00:03,  6.77it/s, epoch=3, grad_norm=150.740, loss=332.1989, lr=4.16e-05]
Epoch 3 TRAIN:  83%|████████▎ | 113/136 [00:20<00:03,  6.77it/s, epoch=3, grad_norm=235.340, loss=450.6721, lr=4.17e-05]
Epoch 3 TRAIN:  83%|████████▎ | 113/136 [00:20<00:03,  6.77it/s, epoch=3, grad_norm=143.772, loss=400.8386, lr=4.18e-05]
Epoch 3 TRAIN:  83%|████████▎ | 113/136 [00:20<00:03,  6.77it/s, epoch=3, grad_norm=63.619, loss=298.9455, lr=4.18e-05] 
Epoch 3 TRAIN:  87%|████████▋ | 118/136 [00:20<00:02,  7.00it/s, epoch=3, grad_norm=63.619, loss=298.9455, lr=4.18e-05]
Epoch 3 TRAIN:  87%|████████▋ | 118/136 [00:20<00:02,  7.00it/s, epoch=3, grad_norm=140.779, loss=295.8980, lr=4.19e-05]
Epoch 3 TRAIN:  87%|████████▋ | 118/136 [00:20<00:02,  7.00it/s, epoch=3, grad_norm=116.984, loss=278.0939, lr=4.20e-05]
Epoch 3 TRAIN:  87%|████████▋ | 118/136 [00:20<00:02,  7.00it/s, epoch=3, grad_norm=148.617, loss=296.8158, lr=4.21e-05]
Epoch 3 TRAIN:  87%|████████▋ | 118/136 [00:21<00:02,  7.00it/s, epoch=3, grad_norm=217.417, loss=382.3969, lr=4.22e-05]
Epoch 3 TRAIN:  90%|████████▉ | 122/136 [00:21<00:02,  6.95it/s, epoch=3, grad_norm=217.417, loss=382.3969, lr=4.22e-05]
Epoch 3 TRAIN:  90%|████████▉ | 122/136 [00:21<00:02,  6.95it/s, epoch=3, grad_norm=226.701, loss=352.8510, lr=4.22e-05]
Epoch 3 TRAIN:  90%|████████▉ | 122/136 [00:21<00:02,  6.95it/s, epoch=3, grad_norm=116.579, loss=299.8134, lr=4.23e-05]
Epoch 3 TRAIN:  90%|████████▉ | 122/136 [00:21<00:02,  6.95it/s, epoch=3, grad_norm=155.945, loss=381.5203, lr=4.24e-05]
Epoch 3 TRAIN:  90%|████████▉ | 122/136 [00:21<00:02,  6.95it/s, epoch=3, grad_norm=350.570, loss=335.5278, lr=4.25e-05]
Epoch 3 TRAIN:  93%|█████████▎| 126/136 [00:21<00:01,  6.96it/s, epoch=3, grad_norm=350.570, loss=335.5278, lr=4.25e-05]
Epoch 3 TRAIN:  93%|█████████▎| 126/136 [00:21<00:01,  6.96it/s, epoch=3, grad_norm=92.825, loss=324.4547, lr=4.26e-05] 
Epoch 3 TRAIN:  93%|█████████▎| 126/136 [00:22<00:01,  6.96it/s, epoch=3, grad_norm=89.147, loss=363.2501, lr=4.26e-05]
Epoch 3 TRAIN:  93%|█████████▎| 126/136 [00:22<00:01,  6.96it/s, epoch=3, grad_norm=108.005, loss=381.4269, lr=4.27e-05]
Epoch 3 TRAIN:  93%|█████████▎| 126/136 [00:22<00:01,  6.96it/s, epoch=3, grad_norm=135.404, loss=336.0997, lr=4.28e-05]
Epoch 3 TRAIN:  96%|█████████▌| 130/136 [00:22<00:00,  6.67it/s, epoch=3, grad_norm=135.404, loss=336.0997, lr=4.28e-05]
Epoch 3 TRAIN:  96%|█████████▌| 130/136 [00:22<00:00,  6.67it/s, epoch=3, grad_norm=109.981, loss=409.9700, lr=4.29e-05]
Epoch 3 TRAIN:  96%|█████████▌| 130/136 [00:22<00:00,  6.67it/s, epoch=3, grad_norm=176.107, loss=421.6933, lr=4.30e-05]
Epoch 3 TRAIN:  96%|█████████▌| 130/136 [00:22<00:00,  6.67it/s, epoch=3, grad_norm=107.682, loss=353.5533, lr=4.30e-05]
Epoch 3 TRAIN:  96%|█████████▌| 130/136 [00:22<00:00,  6.67it/s, epoch=3, grad_norm=119.161, loss=447.1113, lr=4.31e-05]
Epoch 3 TRAIN:  99%|█████████▊| 134/136 [00:22<00:00,  6.97it/s, epoch=3, grad_norm=119.161, loss=447.1113, lr=4.31e-05]
Epoch 3 TRAIN:  99%|█████████▊| 134/136 [00:23<00:00,  6.97it/s, epoch=3, grad_norm=175.824, loss=343.0852, lr=4.32e-05]
Epoch 3 TRAIN:  99%|█████████▊| 134/136 [00:23<00:00,  6.97it/s, epoch=3, grad_norm=116.124, loss=387.1721, lr=4.33e-05]
Epoch 3 TRAIN:  99%|█████████▊| 134/136 [00:23<00:00,  6.97it/s, epoch=3, grad_norm=153.063, loss=469.7009, lr=4.34e-05]
Epoch 3 TRAIN: 137it [00:23,  5.88it/s, epoch=3, grad_norm=153.063, loss=469.7009, lr=4.34e-05]                         

CV:   0%|          | 0/6 [00:00<?, ?it/s]/home/rookie/Documents/batuka/projects/wenet/wenet/wenet/utils/train_utils.py:689: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  amp_autocast(enabled=dtype is not None,

CV:  17%|█▋        | 1/6 [00:00<00:03,  1.32it/s]
                                                 
2026-04-16 19:54:18,567 INFO Epoch 3 Step 542 CV info lr 4.3360e-05 cv_loss 403.10521073774856 rank 0 acc 0.21412897606690726
2026-04-16 19:54:18,569 INFO [Rank 0] Checkpoint: save to checkpoint exp/conformer_mongolian/epoch_3.pt
2026-04-16 19:54:18,681 INFO Epoch 4 Step 542 TRAIN info lr 4.3360e-05 rank 0
2026-04-16 19:54:18,695 INFO using accumulate grad, new batch size is 1 times larger than before

Epoch 4 TRAIN:   0%|          | 0/137 [00:00<?, ?it/s]
Epoch 4 TRAIN:   0%|          | 0/137 [00:03<?, ?it/s, epoch=4, grad_norm=517.545, loss=406.3187, lr=4.34e-05]
Epoch 4 TRAIN:   1%|          | 1/137 [00:03<08:36,  3.80s/it, epoch=4, grad_norm=517.545, loss=406.3187, lr=4.34e-05]
Epoch 4 TRAIN:   1%|          | 1/137 [00:03<08:36,  3.80s/it, epoch=4, grad_norm=573.123, loss=442.2686, lr=4.35e-05]
Epoch 4 TRAIN:   1%|          | 1/137 [00:04<08:36,  3.80s/it, epoch=4, grad_norm=467.919, loss=385.6799, lr=4.36e-05]
Epoch 4 TRAIN:   1%|          | 1/137 [00:04<08:36,  3.80s/it, epoch=4, grad_norm=524.497, loss=434.1726, lr=4.37e-05]
Epoch 4 TRAIN:   1%|          | 1/137 [00:04<08:36,  3.80s/it, epoch=4, grad_norm=477.464, loss=408.4778, lr=4.38e-05]
Epoch 4 TRAIN:   1%|          | 1/137 [00:04<08:36,  3.80s/it, epoch=4, grad_norm=398.179, loss=386.3316, lr=4.38e-05]
Epoch 4 TRAIN:   1%|          | 1/137 [00:04<08:36,  3.80s/it, epoch=4, grad_norm=460.501, loss=422.7311, lr=4.39e-05]
Epoch 4 TRAIN:   5%|▌         | 7/137 [00:04<01:01,  2.11it/s, epoch=4, grad_norm=460.501, loss=422.7311, lr=4.39e-05]
Epoch 4 TRAIN:   5%|▌         | 7/137 [00:04<01:01,  2.11it/s, epoch=4, grad_norm=343.541, loss=395.1434, lr=4.40e-05]
Epoch 4 TRAIN:   5%|▌         | 7/137 [00:04<01:01,  2.11it/s, epoch=4, grad_norm=192.626, loss=343.9089, lr=4.41e-05]
Epoch 4 TRAIN:   5%|▌         | 7/137 [00:04<01:01,  2.11it/s, epoch=4, grad_norm=164.225, loss=356.7634, lr=4.42e-05]
Epoch 4 TRAIN:   5%|▌         | 7/137 [00:04<01:01,  2.11it/s, epoch=4, grad_norm=163.730, loss=403.4260, lr=4.42e-05]
Epoch 4 TRAIN:   5%|▌         | 7/137 [00:04<01:01,  2.11it/s, epoch=4, grad_norm=121.393, loss=328.8390, lr=4.43e-05]
Epoch 4 TRAIN:   9%|▉         | 12/137 [00:04<00:36,  3.39it/s, epoch=4, grad_norm=121.393, loss=328.8390, lr=4.43e-05]
Epoch 4 TRAIN:   9%|▉         | 12/137 [00:05<00:36,  3.39it/s, epoch=4, grad_norm=235.699, loss=308.0747, lr=4.44e-05]
Epoch 4 TRAIN:   9%|▉         | 12/137 [00:05<00:36,  3.39it/s, epoch=4, grad_norm=174.625, loss=357.0411, lr=4.45e-05]
Epoch 4 TRAIN:   9%|▉         | 12/137 [00:05<00:36,  3.39it/s, epoch=4, grad_norm=91.724, loss=396.3810, lr=4.46e-05] 
Epoch 4 TRAIN:   9%|▉         | 12/137 [00:05<00:36,  3.39it/s, epoch=4, grad_norm=187.032, loss=344.3555, lr=4.46e-05]
Epoch 4 TRAIN:  12%|█▏        | 16/137 [00:05<00:28,  4.18it/s, epoch=4, grad_norm=187.032, loss=344.3555, lr=4.46e-05]
Epoch 4 TRAIN:  12%|█▏        | 16/137 [00:05<00:28,  4.18it/s, epoch=4, grad_norm=208.292, loss=358.8900, lr=4.47e-05]
Epoch 4 TRAIN:  12%|█▏        | 16/137 [00:05<00:28,  4.18it/s, epoch=4, grad_norm=194.147, loss=351.5411, lr=4.48e-05]
Epoch 4 TRAIN:  12%|█▏        | 16/137 [00:05<00:28,  4.18it/s, epoch=4, grad_norm=68.239, loss=408.7634, lr=4.49e-05] 
Epoch 4 TRAIN:  12%|█▏        | 16/137 [00:06<00:28,  4.18it/s, epoch=4, grad_norm=192.901, loss=313.5878, lr=4.50e-05]
Epoch 4 TRAIN:  15%|█▍        | 20/137 [00:06<00:24,  4.84it/s, epoch=4, grad_norm=192.901, loss=313.5878, lr=4.50e-05]
Epoch 4 TRAIN:  15%|█▍        | 20/137 [00:06<00:24,  4.84it/s, epoch=4, grad_norm=207.284, loss=299.4866, lr=4.50e-05]
Epoch 4 TRAIN:  15%|█▍        | 20/137 [00:06<00:24,  4.84it/s, epoch=4, grad_norm=53.675, loss=360.6864, lr=4.51e-05] 
Epoch 4 TRAIN:  15%|█▍        | 20/137 [00:06<00:24,  4.84it/s, epoch=4, grad_norm=97.940, loss=352.2633, lr=4.52e-05]
Epoch 4 TRAIN:  15%|█▍        | 20/137 [00:06<00:24,  4.84it/s, epoch=4, grad_norm=135.642, loss=373.9767, lr=4.53e-05]
Epoch 4 TRAIN:  18%|█▊        | 24/137 [00:06<00:21,  5.37it/s, epoch=4, grad_norm=135.642, loss=373.9767, lr=4.53e-05]
Epoch 4 TRAIN:  18%|█▊        | 24/137 [00:06<00:21,  5.37it/s, epoch=4, grad_norm=52.694, loss=331.9211, lr=4.54e-05] 
Epoch 4 TRAIN:  18%|█▊        | 24/137 [00:06<00:21,  5.37it/s, epoch=4, grad_norm=90.455, loss=349.5272, lr=4.54e-05]
Epoch 4 TRAIN:  18%|█▊        | 24/137 [00:07<00:21,  5.37it/s, epoch=4, grad_norm=129.773, loss=384.5538, lr=4.55e-05]
Epoch 4 TRAIN:  18%|█▊        | 24/137 [00:07<00:21,  5.37it/s, epoch=4, grad_norm=63.895, loss=356.1486, lr=4.56e-05] 
Epoch 4 TRAIN:  20%|██        | 28/137 [00:07<00:18,  5.79it/s, epoch=4, grad_norm=63.895, loss=356.1486, lr=4.56e-05]
Epoch 4 TRAIN:  20%|██        | 28/137 [00:07<00:18,  5.79it/s, epoch=4, grad_norm=48.346, loss=336.3288, lr=4.57e-05]
Epoch 4 TRAIN:  20%|██        | 28/137 [00:07<00:18,  5.79it/s, epoch=4, grad_norm=163.966, loss=431.2499, lr=4.58e-05]
Epoch 4 TRAIN:  20%|██        | 28/137 [00:07<00:18,  5.79it/s, epoch=4, grad_norm=56.166, loss=358.5115, lr=4.58e-05] 
Epoch 4 TRAIN:  20%|██        | 28/137 [00:07<00:18,  5.79it/s, epoch=4, grad_norm=99.486, loss=352.8849, lr=4.59e-05]
Epoch 4 TRAIN:  23%|██▎       | 32/137 [00:07<00:17,  6.11it/s, epoch=4, grad_norm=99.486, loss=352.8849, lr=4.59e-05]
Epoch 4 TRAIN:  23%|██▎       | 32/137 [00:07<00:17,  6.11it/s, epoch=4, grad_norm=88.042, loss=357.7615, lr=4.60e-05]
Epoch 4 TRAIN:  23%|██▎       | 32/137 [00:08<00:17,  6.11it/s, epoch=4, grad_norm=198.234, loss=295.3677, lr=4.61e-05]
Epoch 4 TRAIN:  23%|██▎       | 32/137 [00:08<00:17,  6.11it/s, epoch=4, grad_norm=60.793, loss=395.6448, lr=4.62e-05] 
Epoch 4 TRAIN:  23%|██▎       | 32/137 [00:08<00:17,  6.11it/s, epoch=4, grad_norm=132.026, loss=338.8443, lr=4.62e-05]
Epoch 4 TRAIN:  26%|██▋       | 36/137 [00:08<00:15,  6.38it/s, epoch=4, grad_norm=132.026, loss=338.8443, lr=4.62e-05]
Epoch 4 TRAIN:  26%|██▋       | 36/137 [00:08<00:15,  6.38it/s, epoch=4, grad_norm=127.988, loss=323.1892, lr=4.63e-05]
Epoch 4 TRAIN:  26%|██▋       | 36/137 [00:08<00:15,  6.38it/s, epoch=4, grad_norm=121.278, loss=298.9666, lr=4.64e-05]
Epoch 4 TRAIN:  26%|██▋       | 36/137 [00:08<00:15,  6.38it/s, epoch=4, grad_norm=46.235, loss=326.3987, lr=4.65e-05] 
Epoch 4 TRAIN:  26%|██▋       | 36/137 [00:08<00:15,  6.38it/s, epoch=4, grad_norm=49.512, loss=343.3616, lr=4.66e-05]
Epoch 4 TRAIN:  29%|██▉       | 40/137 [00:08<00:14,  6.56it/s, epoch=4, grad_norm=49.512, loss=343.3616, lr=4.66e-05]
Epoch 4 TRAIN:  29%|██▉       | 40/137 [00:09<00:14,  6.56it/s, epoch=4, grad_norm=44.844, loss=326.5479, lr=4.66e-05]
Epoch 4 TRAIN:  29%|██▉       | 40/137 [00:09<00:14,  6.56it/s, epoch=4, grad_norm=183.684, loss=405.2094, lr=4.67e-05]
Epoch 4 TRAIN:  29%|██▉       | 40/137 [00:09<00:14,  6.56it/s, epoch=4, grad_norm=55.593, loss=279.7314, lr=4.68e-05] 
Epoch 4 TRAIN:  29%|██▉       | 40/137 [00:09<00:14,  6.56it/s, epoch=4, grad_norm=53.756, loss=314.0621, lr=4.69e-05]
Epoch 4 TRAIN:  32%|███▏      | 44/137 [00:09<00:14,  6.64it/s, epoch=4, grad_norm=53.756, loss=314.0621, lr=4.69e-05]
Epoch 4 TRAIN:  32%|███▏      | 44/137 [00:09<00:14,  6.64it/s, epoch=4, grad_norm=81.069, loss=338.0066, lr=4.70e-05]
Epoch 4 TRAIN:  32%|███▏      | 44/137 [00:09<00:14,  6.64it/s, epoch=4, grad_norm=134.711, loss=358.2349, lr=4.70e-05]
Epoch 4 TRAIN:  32%|███▏      | 44/137 [00:10<00:14,  6.64it/s, epoch=4, grad_norm=134.673, loss=359.4730, lr=4.71e-05]
Epoch 4 TRAIN:  32%|███▏      | 44/137 [00:10<00:14,  6.64it/s, epoch=4, grad_norm=99.411, loss=364.8747, lr=4.72e-05] 
Epoch 4 TRAIN:  35%|███▌      | 48/137 [00:10<00:13,  6.73it/s, epoch=4, grad_norm=99.411, loss=364.8747, lr=4.72e-05]
Epoch 4 TRAIN:  35%|███▌      | 48/137 [00:10<00:13,  6.73it/s, epoch=4, grad_norm=87.760, loss=383.8420, lr=4.73e-05]
Epoch 4 TRAIN:  35%|███▌      | 48/137 [00:10<00:13,  6.73it/s, epoch=4, grad_norm=75.636, loss=339.6764, lr=4.74e-05]
Epoch 4 TRAIN:  35%|███▌      | 48/137 [00:10<00:13,  6.73it/s, epoch=4, grad_norm=78.413, loss=400.1106, lr=4.74e-05]
Epoch 4 TRAIN:  35%|███▌      | 48/137 [00:10<00:13,  6.73it/s, epoch=4, grad_norm=242.516, loss=286.7656, lr=4.75e-05]
Epoch 4 TRAIN:  38%|███▊      | 52/137 [00:10<00:12,  6.79it/s, epoch=4, grad_norm=242.516, loss=286.7656, lr=4.75e-05]
Epoch 4 TRAIN:  38%|███▊      | 52/137 [00:10<00:12,  6.79it/s, epoch=4, grad_norm=128.254, loss=361.1725, lr=4.76e-05]
Epoch 4 TRAIN:  38%|███▊      | 52/137 [00:11<00:12,  6.79it/s, epoch=4, grad_norm=183.948, loss=328.1827, lr=4.77e-05]
Epoch 4 TRAIN:  38%|███▊      | 52/137 [00:11<00:12,  6.79it/s, epoch=4, grad_norm=144.459, loss=327.0595, lr=4.78e-05]
Epoch 4 TRAIN:  38%|███▊      | 52/137 [00:11<00:12,  6.79it/s, epoch=4, grad_norm=141.837, loss=374.6311, lr=4.78e-05]
Epoch 4 TRAIN:  41%|████      | 56/137 [00:11<00:11,  6.77it/s, epoch=4, grad_norm=141.837, loss=374.6311, lr=4.78e-05]
Epoch 4 TRAIN:  41%|████      | 56/137 [00:11<00:11,  6.77it/s, epoch=4, grad_norm=172.290, loss=322.3595, lr=4.79e-05]
Epoch 4 TRAIN:  41%|████      | 56/137 [00:11<00:11,  6.77it/s, epoch=4, grad_norm=92.635, loss=349.3371, lr=4.80e-05] 
Epoch 4 TRAIN:  41%|████      | 56/137 [00:11<00:11,  6.77it/s, epoch=4, grad_norm=60.712, loss=321.3912, lr=4.81e-05]
Epoch 4 TRAIN:  41%|████      | 56/137 [00:11<00:11,  6.77it/s, epoch=4, grad_norm=95.517, loss=358.3812, lr=4.82e-05]
Epoch 4 TRAIN:  44%|████▍     | 60/137 [00:11<00:11,  6.87it/s, epoch=4, grad_norm=95.517, loss=358.3812, lr=4.82e-05]
Epoch 4 TRAIN:  44%|████▍     | 60/137 [00:12<00:11,  6.87it/s, epoch=4, grad_norm=129.525, loss=355.1038, lr=4.82e-05]
Epoch 4 TRAIN:  44%|████▍     | 60/137 [00:12<00:11,  6.87it/s, epoch=4, grad_norm=173.544, loss=367.6862, lr=4.83e-05]
Epoch 4 TRAIN:  44%|████▍     | 60/137 [00:12<00:11,  6.87it/s, epoch=4, grad_norm=140.627, loss=345.7102, lr=4.84e-05]
Epoch 4 TRAIN:  44%|████▍     | 60/137 [00:12<00:11,  6.87it/s, epoch=4, grad_norm=58.579, loss=333.3123, lr=4.85e-05] 
Epoch 4 TRAIN:  47%|████▋     | 64/137 [00:12<00:10,  6.88it/s, epoch=4, grad_norm=58.579, loss=333.3123, lr=4.85e-05]
Epoch 4 TRAIN:  47%|████▋     | 64/137 [00:12<00:10,  6.88it/s, epoch=4, grad_norm=69.292, loss=349.5429, lr=4.86e-05]
Epoch 4 TRAIN:  47%|████▋     | 64/137 [00:12<00:10,  6.88it/s, epoch=4, grad_norm=87.648, loss=365.9892, lr=4.86e-05]
Epoch 4 TRAIN:  47%|████▋     | 64/137 [00:12<00:10,  6.88it/s, epoch=4, grad_norm=53.803, loss=338.7477, lr=4.87e-05]
Epoch 4 TRAIN:  47%|████▋     | 64/137 [00:13<00:10,  6.88it/s, epoch=4, grad_norm=226.622, loss=270.0756, lr=4.88e-05]
Epoch 4 TRAIN:  50%|████▉     | 68/137 [00:13<00:10,  6.77it/s, epoch=4, grad_norm=226.622, loss=270.0756, lr=4.88e-05]
Epoch 4 TRAIN:  50%|████▉     | 68/137 [00:13<00:10,  6.77it/s, epoch=4, grad_norm=192.186, loss=288.9959, lr=4.89e-05]
Epoch 4 TRAIN:  50%|████▉     | 68/137 [00:13<00:10,  6.77it/s, epoch=4, grad_norm=122.486, loss=408.9264, lr=4.90e-05]
Epoch 4 TRAIN:  50%|████▉     | 68/137 [00:13<00:10,  6.77it/s, epoch=4, grad_norm=52.595, loss=334.7438, lr=4.90e-05] 
Epoch 4 TRAIN:  50%|████▉     | 68/137 [00:13<00:10,  6.77it/s, epoch=4, grad_norm=78.228, loss=378.4137, lr=4.91e-05]
Epoch 4 TRAIN:  53%|█████▎    | 72/137 [00:13<00:09,  6.79it/s, epoch=4, grad_norm=78.228, loss=378.4137, lr=4.91e-05]
Epoch 4 TRAIN:  53%|█████▎    | 72/137 [00:13<00:09,  6.79it/s, epoch=4, grad_norm=212.670, loss=313.7776, lr=4.92e-05]
Epoch 4 TRAIN:  53%|█████▎    | 72/137 [00:13<00:09,  6.79it/s, epoch=4, grad_norm=87.789, loss=362.8165, lr=4.93e-05] 
Epoch 4 TRAIN:  53%|█████▎    | 72/137 [00:14<00:09,  6.79it/s, epoch=4, grad_norm=70.953, loss=370.1320, lr=4.94e-05]
Epoch 4 TRAIN:  53%|█████▎    | 72/137 [00:14<00:09,  6.79it/s, epoch=4, grad_norm=58.820, loss=358.9733, lr=4.94e-05]
Epoch 4 TRAIN:  55%|█████▌    | 76/137 [00:14<00:08,  6.89it/s, epoch=4, grad_norm=58.820, loss=358.9733, lr=4.94e-05]
Epoch 4 TRAIN:  55%|█████▌    | 76/137 [00:14<00:08,  6.89it/s, epoch=4, grad_norm=96.433, loss=306.5247, lr=4.95e-05]
Epoch 4 TRAIN:  55%|█████▌    | 76/137 [00:14<00:08,  6.89it/s, epoch=4, grad_norm=177.328, loss=396.7706, lr=4.96e-05]
Epoch 4 TRAIN:  55%|█████▌    | 76/137 [00:14<00:08,  6.89it/s, epoch=4, grad_norm=132.577, loss=357.3135, lr=4.97e-05]
Epoch 4 TRAIN:  55%|█████▌    | 76/137 [00:14<00:08,  6.89it/s, epoch=4, grad_norm=221.844, loss=446.3591, lr=4.98e-05]
Epoch 4 TRAIN:  58%|█████▊    | 80/137 [00:14<00:08,  6.90it/s, epoch=4, grad_norm=221.844, loss=446.3591, lr=4.98e-05]
Epoch 4 TRAIN:  58%|█████▊    | 80/137 [00:14<00:08,  6.90it/s, epoch=4, grad_norm=150.294, loss=326.6678, lr=4.98e-05]
Epoch 4 TRAIN:  58%|█████▊    | 80/137 [00:15<00:08,  6.90it/s, epoch=4, grad_norm=52.559, loss=370.8121, lr=4.99e-05] 
Epoch 4 TRAIN:  58%|█████▊    | 80/137 [00:15<00:08,  6.90it/s, epoch=4, grad_norm=164.654, loss=278.7960, lr=5.00e-05]
Epoch 4 TRAIN:  58%|█████▊    | 80/137 [00:15<00:08,  6.90it/s, epoch=4, grad_norm=131.320, loss=302.0669, lr=5.01e-05]
Epoch 4 TRAIN:  61%|██████▏   | 84/137 [00:15<00:07,  6.89it/s, epoch=4, grad_norm=131.320, loss=302.0669, lr=5.01e-05]
Epoch 4 TRAIN:  61%|██████▏   | 84/137 [00:15<00:07,  6.89it/s, epoch=4, grad_norm=83.452, loss=327.8225, lr=5.02e-05] 
Epoch 4 TRAIN:  61%|██████▏   | 84/137 [00:15<00:07,  6.89it/s, epoch=4, grad_norm=153.422, loss=370.1074, lr=5.02e-05]
Epoch 4 TRAIN:  61%|██████▏   | 84/137 [00:15<00:07,  6.89it/s, epoch=4, grad_norm=208.200, loss=366.5730, lr=5.03e-05]
Epoch 4 TRAIN:  61%|██████▏   | 84/137 [00:15<00:07,  6.89it/s, epoch=4, grad_norm=230.504, loss=234.1406, lr=5.04e-05]
Epoch 4 TRAIN:  64%|██████▍   | 88/137 [00:15<00:07,  6.79it/s, epoch=4, grad_norm=230.504, loss=234.1406, lr=5.04e-05]
Epoch 4 TRAIN:  64%|██████▍   | 88/137 [00:16<00:07,  6.79it/s, epoch=4, grad_norm=386.889, loss=182.9659, lr=5.05e-05]
Epoch 4 TRAIN:  64%|██████▍   | 88/137 [00:16<00:07,  6.79it/s, epoch=4, grad_norm=64.741, loss=336.3634, lr=5.06e-05] 
Epoch 4 TRAIN:  64%|██████▍   | 88/137 [00:16<00:07,  6.79it/s, epoch=4, grad_norm=113.099, loss=358.9099, lr=5.06e-05]
Epoch 4 TRAIN:  64%|██████▍   | 88/137 [00:16<00:07,  6.79it/s, epoch=4, grad_norm=78.668, loss=274.4202, lr=5.07e-05] 
Epoch 4 TRAIN:  67%|██████▋   | 92/137 [00:16<00:06,  6.77it/s, epoch=4, grad_norm=78.668, loss=274.4202, lr=5.07e-05]
Epoch 4 TRAIN:  67%|██████▋   | 92/137 [00:16<00:06,  6.77it/s, epoch=4, grad_norm=122.253, loss=298.5251, lr=5.08e-05]
Epoch 4 TRAIN:  67%|██████▋   | 92/137 [00:16<00:06,  6.77it/s, epoch=4, grad_norm=197.007, loss=368.7456, lr=5.09e-05]
Epoch 4 TRAIN:  67%|██████▋   | 92/137 [00:17<00:06,  6.77it/s, epoch=4, grad_norm=259.119, loss=373.5170, lr=5.10e-05]
Epoch 4 TRAIN:  67%|██████▋   | 92/137 [00:17<00:06,  6.77it/s, epoch=4, grad_norm=78.100, loss=339.7650, lr=5.10e-05] 
Epoch 4 TRAIN:  70%|███████   | 96/137 [00:17<00:06,  6.46it/s, epoch=4, grad_norm=78.100, loss=339.7650, lr=5.10e-05]
Epoch 4 TRAIN:  70%|███████   | 96/137 [00:17<00:06,  6.46it/s, epoch=4, grad_norm=247.638, loss=257.1666, lr=5.11e-05]
Epoch 4 TRAIN:  70%|███████   | 96/137 [00:17<00:06,  6.46it/s, epoch=4, grad_norm=60.387, loss=318.9168, lr=5.12e-05] 
Epoch 4 TRAIN:  70%|███████   | 96/137 [00:17<00:06,  6.46it/s, epoch=4, grad_norm=100.803, loss=339.3060, lr=5.13e-05]
Epoch 4 TRAIN:  70%|███████   | 96/137 [00:17<00:06,  6.46it/s, epoch=4, grad_norm=107.914, loss=358.9617, lr=5.14e-05]2026-04-16 19:54:36,556 DEBUG TRAIN | steps/sec 5.660| Batch 4/100 loss 358.961670 loss_att 314.422821 loss_ctc 462.885620 th_accuracy 0.264657 lr 5.1360e-05 grad_norm 107.913582 rank 0

Epoch 4 TRAIN:  73%|███████▎  | 100/137 [00:17<00:05,  6.52it/s, epoch=4, grad_norm=107.914, loss=358.9617, lr=5.14e-05]
Epoch 4 TRAIN:  73%|███████▎  | 100/137 [00:18<00:05,  6.52it/s, epoch=4, grad_norm=182.862, loss=300.9818, lr=5.14e-05]
Epoch 4 TRAIN:  73%|███████▎  | 100/137 [00:18<00:05,  6.52it/s, epoch=4, grad_norm=89.026, loss=382.4231, lr=5.15e-05] 
Epoch 4 TRAIN:  73%|███████▎  | 100/137 [00:18<00:05,  6.52it/s, epoch=4, grad_norm=132.950, loss=356.4807, lr=5.16e-05]
Epoch 4 TRAIN:  73%|███████▎  | 100/137 [00:18<00:05,  6.52it/s, epoch=4, grad_norm=248.453, loss=307.8246, lr=5.17e-05]
Epoch 4 TRAIN:  76%|███████▌  | 104/137 [00:18<00:05,  6.53it/s, epoch=4, grad_norm=248.453, loss=307.8246, lr=5.17e-05]
Epoch 4 TRAIN:  76%|███████▌  | 104/137 [00:18<00:05,  6.53it/s, epoch=4, grad_norm=262.157, loss=360.9938, lr=5.18e-05]
Epoch 4 TRAIN:  76%|███████▌  | 104/137 [00:18<00:05,  6.53it/s, epoch=4, grad_norm=250.011, loss=314.1531, lr=5.18e-05]
Epoch 4 TRAIN:  76%|███████▌  | 104/137 [00:18<00:05,  6.53it/s, epoch=4, grad_norm=190.684, loss=310.2311, lr=5.19e-05]
Epoch 4 TRAIN:  76%|███████▌  | 104/137 [00:19<00:05,  6.53it/s, epoch=4, grad_norm=276.149, loss=226.7654, lr=5.20e-05]
Epoch 4 TRAIN:  79%|███████▉  | 108/137 [00:19<00:04,  6.53it/s, epoch=4, grad_norm=276.149, loss=226.7654, lr=5.20e-05]
Epoch 4 TRAIN:  79%|███████▉  | 108/137 [00:19<00:04,  6.53it/s, epoch=4, grad_norm=111.615, loss=340.5982, lr=5.21e-05]
Epoch 4 TRAIN:  79%|███████▉  | 108/137 [00:19<00:04,  6.53it/s, epoch=4, grad_norm=98.102, loss=396.2946, lr=5.22e-05] 
Epoch 4 TRAIN:  79%|███████▉  | 108/137 [00:19<00:04,  6.53it/s, epoch=4, grad_norm=83.240, loss=300.8667, lr=5.22e-05]
Epoch 4 TRAIN:  79%|███████▉  | 108/137 [00:19<00:04,  6.53it/s, epoch=4, grad_norm=71.500, loss=313.2578, lr=5.23e-05]
Epoch 4 TRAIN:  82%|████████▏ | 112/137 [00:19<00:03,  6.54it/s, epoch=4, grad_norm=71.500, loss=313.2578, lr=5.23e-05]
Epoch 4 TRAIN:  82%|████████▏ | 112/137 [00:19<00:03,  6.54it/s, epoch=4, grad_norm=159.262, loss=380.4867, lr=5.24e-05]
Epoch 4 TRAIN:  82%|████████▏ | 112/137 [00:20<00:03,  6.54it/s, epoch=4, grad_norm=171.909, loss=380.7945, lr=5.25e-05]
Epoch 4 TRAIN:  82%|████████▏ | 112/137 [00:20<00:03,  6.54it/s, epoch=4, grad_norm=121.354, loss=320.9640, lr=5.26e-05]
Epoch 4 TRAIN:  82%|████████▏ | 112/137 [00:20<00:03,  6.54it/s, epoch=4, grad_norm=166.835, loss=321.6686, lr=5.26e-05]
Epoch 4 TRAIN:  85%|████████▍ | 116/137 [00:20<00:03,  6.39it/s, epoch=4, grad_norm=166.835, loss=321.6686, lr=5.26e-05]
Epoch 4 TRAIN:  85%|████████▍ | 116/137 [00:20<00:03,  6.39it/s, epoch=4, grad_norm=126.284, loss=367.1535, lr=5.27e-05]
Epoch 4 TRAIN:  85%|████████▍ | 116/137 [00:20<00:03,  6.39it/s, epoch=4, grad_norm=199.642, loss=278.4605, lr=5.28e-05]
Epoch 4 TRAIN:  85%|████████▍ | 116/137 [00:20<00:03,  6.39it/s, epoch=4, grad_norm=67.203, loss=266.5556, lr=5.29e-05] 
Epoch 4 TRAIN:  85%|████████▍ | 116/137 [00:20<00:03,  6.39it/s, epoch=4, grad_norm=110.162, loss=361.9933, lr=5.30e-05]
Epoch 4 TRAIN:  88%|████████▊ | 120/137 [00:20<00:02,  6.59it/s, epoch=4, grad_norm=110.162, loss=361.9933, lr=5.30e-05]
Epoch 4 TRAIN:  88%|████████▊ | 120/137 [00:21<00:02,  6.59it/s, epoch=4, grad_norm=207.311, loss=418.6705, lr=5.30e-05]
Epoch 4 TRAIN:  88%|████████▊ | 120/137 [00:21<00:02,  6.59it/s, epoch=4, grad_norm=102.744, loss=306.2950, lr=5.31e-05]
Epoch 4 TRAIN:  88%|████████▊ | 120/137 [00:21<00:02,  6.59it/s, epoch=4, grad_norm=77.304, loss=360.2326, lr=5.32e-05] 
Epoch 4 TRAIN:  88%|████████▊ | 120/137 [00:21<00:02,  6.59it/s, epoch=4, grad_norm=88.728, loss=356.5698, lr=5.33e-05]
Epoch 4 TRAIN:  91%|█████████ | 124/137 [00:21<00:01,  6.56it/s, epoch=4, grad_norm=88.728, loss=356.5698, lr=5.33e-05]
Epoch 4 TRAIN:  91%|█████████ | 124/137 [00:21<00:01,  6.56it/s, epoch=4, grad_norm=122.295, loss=425.0037, lr=5.34e-05]
Epoch 4 TRAIN:  91%|█████████ | 124/137 [00:21<00:01,  6.56it/s, epoch=4, grad_norm=80.382, loss=380.7732, lr=5.34e-05] 
Epoch 4 TRAIN:  91%|█████████ | 124/137 [00:21<00:01,  6.56it/s, epoch=4, grad_norm=123.072, loss=362.9079, lr=5.35e-05]
Epoch 4 TRAIN:  91%|█████████ | 124/137 [00:22<00:01,  6.56it/s, epoch=4, grad_norm=101.498, loss=406.6332, lr=5.36e-05]
Epoch 4 TRAIN:  93%|█████████▎| 128/137 [00:22<00:01,  6.70it/s, epoch=4, grad_norm=101.498, loss=406.6332, lr=5.36e-05]
Epoch 4 TRAIN:  93%|█████████▎| 128/137 [00:22<00:01,  6.70it/s, epoch=4, grad_norm=97.245, loss=337.6667, lr=5.37e-05] 
Epoch 4 TRAIN:  93%|█████████▎| 128/137 [00:22<00:01,  6.70it/s, epoch=4, grad_norm=204.461, loss=339.4749, lr=5.38e-05]
Epoch 4 TRAIN:  93%|█████████▎| 128/137 [00:22<00:01,  6.70it/s, epoch=4, grad_norm=124.479, loss=375.4126, lr=5.38e-05]
Epoch 4 TRAIN:  93%|█████████▎| 128/137 [00:22<00:01,  6.70it/s, epoch=4, grad_norm=120.114, loss=432.6721, lr=5.39e-05]
Epoch 4 TRAIN:  96%|█████████▋| 132/137 [00:22<00:00,  6.86it/s, epoch=4, grad_norm=120.114, loss=432.6721, lr=5.39e-05]
Epoch 4 TRAIN:  96%|█████████▋| 132/137 [00:22<00:00,  6.86it/s, epoch=4, grad_norm=133.370, loss=386.3966, lr=5.40e-05]
Epoch 4 TRAIN:  96%|█████████▋| 132/137 [00:22<00:00,  6.86it/s, epoch=4, grad_norm=111.130, loss=367.5522, lr=5.41e-05]
Epoch 4 TRAIN:  96%|█████████▋| 132/137 [00:23<00:00,  6.86it/s, epoch=4, grad_norm=103.141, loss=397.6368, lr=5.42e-05]
Epoch 4 TRAIN:  96%|█████████▋| 132/137 [00:23<00:00,  6.86it/s, epoch=4, grad_norm=148.694, loss=343.4644, lr=5.42e-05]
Epoch 4 TRAIN:  99%|█████████▉| 136/137 [00:23<00:00,  6.90it/s, epoch=4, grad_norm=148.694, loss=343.4644, lr=5.42e-05]
Epoch 4 TRAIN:  99%|█████████▉| 136/137 [00:23<00:00,  6.90it/s, epoch=4, grad_norm=176.144, loss=446.1581, lr=5.43e-05]
Epoch 4 TRAIN: 100%|██████████| 137/137 [00:23<00:00,  5.87it/s, epoch=4, grad_norm=176.144, loss=446.1581, lr=5.43e-05]

CV:   0%|          | 0/6 [00:00<?, ?it/s]/home/rookie/Documents/batuka/projects/wenet/wenet/wenet/utils/train_utils.py:689: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  amp_autocast(enabled=dtype is not None,

CV:  17%|█▋        | 1/6 [00:00<00:03,  1.33it/s]
                                                 
2026-04-16 19:54:42,876 INFO Epoch 4 Step 679 CV info lr 5.4320e-05 cv_loss 407.8834505948153 rank 0 acc 0.23314850529034933
2026-04-16 19:54:42,878 INFO [Rank 0] Checkpoint: save to checkpoint exp/conformer_mongolian/epoch_4.pt
2026-04-16 19:54:42,991 INFO Epoch 5 Step 679 TRAIN info lr 5.4320e-05 rank 0
2026-04-16 19:54:43,009 INFO using accumulate grad, new batch size is 1 times larger than before

Epoch 5 TRAIN:   0%|          | 0/137 [00:00<?, ?it/s]
Epoch 5 TRAIN:   0%|          | 0/137 [00:03<?, ?it/s, epoch=5, grad_norm=399.119, loss=339.5668, lr=5.44e-05]
Epoch 5 TRAIN:   1%|          | 1/137 [00:03<08:45,  3.87s/it, epoch=5, grad_norm=399.119, loss=339.5668, lr=5.44e-05]
Epoch 5 TRAIN:   1%|          | 1/137 [00:03<08:45,  3.87s/it, epoch=5, grad_norm=512.333, loss=403.5243, lr=5.45e-05]
Epoch 5 TRAIN:   1%|          | 1/137 [00:04<08:45,  3.87s/it, epoch=5, grad_norm=603.825, loss=458.5149, lr=5.46e-05]
Epoch 5 TRAIN:   1%|          | 1/137 [00:04<08:45,  3.87s/it, epoch=5, grad_norm=438.941, loss=362.3834, lr=5.46e-05]
Epoch 5 TRAIN:   1%|          | 1/137 [00:04<08:45,  3.87s/it, epoch=5, grad_norm=335.144, loss=300.9341, lr=5.47e-05]
Epoch 5 TRAIN:   1%|          | 1/137 [00:04<08:45,  3.87s/it, epoch=5, grad_norm=490.649, loss=400.7198, lr=5.48e-05]
Epoch 5 TRAIN:   1%|          | 1/137 [00:04<08:45,  3.87s/it, epoch=5, grad_norm=519.838, loss=452.4745, lr=5.49e-05]
Epoch 5 TRAIN:   5%|▌         | 7/137 [00:04<01:02,  2.09it/s, epoch=5, grad_norm=519.838, loss=452.4745, lr=5.49e-05]
Epoch 5 TRAIN:   5%|▌         | 7/137 [00:04<01:02,  2.09it/s, epoch=5, grad_norm=396.505, loss=397.7198, lr=5.50e-05]
Epoch 5 TRAIN:   5%|▌         | 7/137 [00:04<01:02,  2.09it/s, epoch=5, grad_norm=262.422, loss=348.3112, lr=5.50e-05]
Epoch 5 TRAIN:   5%|▌         | 7/137 [00:04<01:02,  2.09it/s, epoch=5, grad_norm=287.170, loss=393.6608, lr=5.51e-05]
Epoch 5 TRAIN:   5%|▌         | 7/137 [00:04<01:02,  2.09it/s, epoch=5, grad_norm=205.637, loss=368.6994, lr=5.52e-05]
Epoch 5 TRAIN:   5%|▌         | 7/137 [00:05<01:02,  2.09it/s, epoch=5, grad_norm=147.626, loss=379.1715, lr=5.53e-05]
Epoch 5 TRAIN:   9%|▉         | 12/137 [00:05<00:37,  3.36it/s, epoch=5, grad_norm=147.626, loss=379.1715, lr=5.53e-05]
Epoch 5 TRAIN:   9%|▉         | 12/137 [00:05<00:37,  3.36it/s, epoch=5, grad_norm=75.085, loss=366.3247, lr=5.54e-05] 
Epoch 5 TRAIN:   9%|▉         | 12/137 [00:05<00:37,  3.36it/s, epoch=5, grad_norm=187.169, loss=309.2120, lr=5.54e-05]
Epoch 5 TRAIN:   9%|▉         | 12/137 [00:05<00:37,  3.36it/s, epoch=5, grad_norm=120.545, loss=375.5622, lr=5.55e-05]
Epoch 5 TRAIN:   9%|▉         | 12/137 [00:05<00:37,  3.36it/s, epoch=5, grad_norm=170.081, loss=402.7223, lr=5.56e-05]
Epoch 5 TRAIN:  12%|█▏        | 16/137 [00:05<00:29,  4.13it/s, epoch=5, grad_norm=170.081, loss=402.7223, lr=5.56e-05]
Epoch 5 TRAIN:  12%|█▏        | 16/137 [00:05<00:29,  4.13it/s, epoch=5, grad_norm=234.190, loss=377.1212, lr=5.57e-05]
Epoch 5 TRAIN:  12%|█▏        | 16/137 [00:05<00:29,  4.13it/s, epoch=5, grad_norm=280.917, loss=350.5442, lr=5.58e-05]
Epoch 5 TRAIN:  12%|█▏        | 16/137 [00:06<00:29,  4.13it/s, epoch=5, grad_norm=392.830, loss=305.1340, lr=5.58e-05]
Epoch 5 TRAIN:  12%|█▏        | 16/137 [00:06<00:29,  4.13it/s, epoch=5, grad_norm=252.658, loss=360.0731, lr=5.59e-05]
Epoch 5 TRAIN:  15%|█▍        | 20/137 [00:06<00:24,  4.75it/s, epoch=5, grad_norm=252.658, loss=360.0731, lr=5.59e-05]
Epoch 5 TRAIN:  15%|█▍        | 20/137 [00:06<00:24,  4.75it/s, epoch=5, grad_norm=282.563, loss=330.2566, lr=5.60e-05]
Epoch 5 TRAIN:  15%|█▍        | 20/137 [00:06<00:24,  4.75it/s, epoch=5, grad_norm=212.487, loss=341.4759, lr=5.61e-05]
Epoch 5 TRAIN:  15%|█▍        | 20/137 [00:06<00:24,  4.75it/s, epoch=5, grad_norm=133.026, loss=355.7923, lr=5.62e-05]
Epoch 5 TRAIN:  15%|█▍        | 20/137 [00:06<00:24,  4.75it/s, epoch=5, grad_norm=115.296, loss=357.7638, lr=5.62e-05]
Epoch 5 TRAIN:  18%|█▊        | 24/137 [00:06<00:21,  5.29it/s, epoch=5, grad_norm=115.296, loss=357.7638, lr=5.62e-05]
Epoch 5 TRAIN:  18%|█▊        | 24/137 [00:06<00:21,  5.29it/s, epoch=5, grad_norm=100.664, loss=320.2861, lr=5.63e-05]
Epoch 5 TRAIN:  18%|█▊        | 24/137 [00:07<00:21,  5.29it/s, epoch=5, grad_norm=105.553, loss=370.5960, lr=5.64e-05]
Epoch 5 TRAIN:  18%|█▊        | 24/137 [00:07<00:21,  5.29it/s, epoch=5, grad_norm=53.713, loss=315.0847, lr=5.65e-05] 
Epoch 5 TRAIN:  18%|█▊        | 24/137 [00:07<00:21,  5.29it/s, epoch=5, grad_norm=162.087, loss=364.7735, lr=5.66e-05]
Epoch 5 TRAIN:  20%|██        | 28/137 [00:07<00:19,  5.70it/s, epoch=5, grad_norm=162.087, loss=364.7735, lr=5.66e-05]
Epoch 5 TRAIN:  20%|██        | 28/137 [00:07<00:19,  5.70it/s, epoch=5, grad_norm=50.512, loss=293.5340, lr=5.66e-05] 
Epoch 5 TRAIN:  20%|██        | 28/137 [00:07<00:19,  5.70it/s, epoch=5, grad_norm=118.649, loss=327.8970, lr=5.67e-05]
Epoch 5 TRAIN:  20%|██        | 28/137 [00:07<00:19,  5.70it/s, epoch=5, grad_norm=224.129, loss=369.0662, lr=5.68e-05]
Epoch 5 TRAIN:  20%|██        | 28/137 [00:07<00:19,  5.70it/s, epoch=5, grad_norm=71.232, loss=320.0336, lr=5.69e-05] 
Epoch 5 TRAIN:  23%|██▎       | 32/137 [00:07<00:17,  6.06it/s, epoch=5, grad_norm=71.232, loss=320.0336, lr=5.69e-05]
Epoch 5 TRAIN:  23%|██▎       | 32/137 [00:08<00:17,  6.06it/s, epoch=5, grad_norm=150.824, loss=371.6043, lr=5.70e-05]
Epoch 5 TRAIN:  23%|██▎       | 32/137 [00:08<00:17,  6.06it/s, epoch=5, grad_norm=77.630, loss=343.4093, lr=5.70e-05] 
Epoch 5 TRAIN:  23%|██▎       | 32/137 [00:08<00:17,  6.06it/s, epoch=5, grad_norm=103.814, loss=295.7839, lr=5.71e-05]
Epoch 5 TRAIN:  23%|██▎       | 32/137 [00:08<00:17,  6.06it/s, epoch=5, grad_norm=103.091, loss=367.3093, lr=5.72e-05]
Epoch 5 TRAIN:  26%|██▋       | 36/137 [00:08<00:16,  6.25it/s, epoch=5, grad_norm=103.091, loss=367.3093, lr=5.72e-05]
Epoch 5 TRAIN:  26%|██▋       | 36/137 [00:08<00:16,  6.25it/s, epoch=5, grad_norm=61.340, loss=346.4486, lr=5.73e-05] 
Epoch 5 TRAIN:  26%|██▋       | 36/137 [00:08<00:16,  6.25it/s, epoch=5, grad_norm=67.138, loss=364.0182, lr=5.74e-05]
Epoch 5 TRAIN:  26%|██▋       | 36/137 [00:08<00:16,  6.25it/s, epoch=5, grad_norm=93.607, loss=332.8881, lr=5.74e-05]
Epoch 5 TRAIN:  26%|██▋       | 36/137 [00:09<00:16,  6.25it/s, epoch=5, grad_norm=53.286, loss=372.1318, lr=5.75e-05]
Epoch 5 TRAIN:  29%|██▉       | 40/137 [00:09<00:15,  6.41it/s, epoch=5, grad_norm=53.286, loss=372.1318, lr=5.75e-05]
Epoch 5 TRAIN:  29%|██▉       | 40/137 [00:09<00:15,  6.41it/s, epoch=5, grad_norm=113.525, loss=332.3595, lr=5.76e-05]
Epoch 5 TRAIN:  29%|██▉       | 40/137 [00:09<00:15,  6.41it/s, epoch=5, grad_norm=54.775, loss=396.3614, lr=5.77e-05] 
Epoch 5 TRAIN:  29%|██▉       | 40/137 [00:09<00:15,  6.41it/s, epoch=5, grad_norm=150.994, loss=312.2378, lr=5.78e-05]
Epoch 5 TRAIN:  29%|██▉       | 40/137 [00:09<00:15,  6.41it/s, epoch=5, grad_norm=84.726, loss=330.6450, lr=5.78e-05] 
Epoch 5 TRAIN:  32%|███▏      | 44/137 [00:09<00:14,  6.50it/s, epoch=5, grad_norm=84.726, loss=330.6450, lr=5.78e-05]
Epoch 5 TRAIN:  32%|███▏      | 44/137 [00:09<00:14,  6.50it/s, epoch=5, grad_norm=48.000, loss=335.1043, lr=5.79e-05]
Epoch 5 TRAIN:  32%|███▏      | 44/137 [00:09<00:14,  6.50it/s, epoch=5, grad_norm=62.359, loss=358.6723, lr=5.80e-05]
Epoch 5 TRAIN:  32%|███▏      | 44/137 [00:10<00:14,  6.50it/s, epoch=5, grad_norm=58.572, loss=320.3160, lr=5.81e-05]
Epoch 5 TRAIN:  32%|███▏      | 44/137 [00:10<00:14,  6.50it/s, epoch=5, grad_norm=84.082, loss=366.0954, lr=5.82e-05]
Epoch 5 TRAIN:  35%|███▌      | 48/137 [00:10<00:13,  6.61it/s, epoch=5, grad_norm=84.082, loss=366.0954, lr=5.82e-05]
Epoch 5 TRAIN:  35%|███▌      | 48/137 [00:10<00:13,  6.61it/s, epoch=5, grad_norm=91.226, loss=367.8507, lr=5.82e-05]
Epoch 5 TRAIN:  35%|███▌      | 48/137 [00:10<00:13,  6.61it/s, epoch=5, grad_norm=106.966, loss=380.5995, lr=5.83e-05]
Epoch 5 TRAIN:  35%|███▌      | 48/137 [00:10<00:13,  6.61it/s, epoch=5, grad_norm=117.948, loss=285.2916, lr=5.84e-05]
Epoch 5 TRAIN:  35%|███▌      | 48/137 [00:10<00:13,  6.61it/s, epoch=5, grad_norm=131.597, loss=385.5752, lr=5.85e-05]
Epoch 5 TRAIN:  38%|███▊      | 52/137 [00:10<00:12,  6.63it/s, epoch=5, grad_norm=131.597, loss=385.5752, lr=5.85e-05]
Epoch 5 TRAIN:  38%|███▊      | 52/137 [00:11<00:12,  6.63it/s, epoch=5, grad_norm=50.343, loss=319.8300, lr=5.86e-05] 
Epoch 5 TRAIN:  38%|███▊      | 52/137 [00:11<00:12,  6.63it/s, epoch=5, grad_norm=108.842, loss=344.9939, lr=5.86e-05]
Epoch 5 TRAIN:  38%|███▊      | 52/137 [00:11<00:12,  6.63it/s, epoch=5, grad_norm=45.349, loss=323.7201, lr=5.87e-05] 
Epoch 5 TRAIN:  38%|███▊      | 52/137 [00:11<00:12,  6.63it/s, epoch=5, grad_norm=57.978, loss=353.0991, lr=5.88e-05]
Epoch 5 TRAIN:  41%|████      | 56/137 [00:11<00:12,  6.64it/s, epoch=5, grad_norm=57.978, loss=353.0991, lr=5.88e-05]
Epoch 5 TRAIN:  41%|████      | 56/137 [00:11<00:12,  6.64it/s, epoch=5, grad_norm=52.303, loss=355.3931, lr=5.89e-05]
Epoch 5 TRAIN:  41%|████      | 56/137 [00:11<00:12,  6.64it/s, epoch=5, grad_norm=48.378, loss=359.7076, lr=5.90e-05]
Epoch 5 TRAIN:  41%|████      | 56/137 [00:11<00:12,  6.64it/s, epoch=5, grad_norm=96.554, loss=381.2279, lr=5.90e-05]
Epoch 5 TRAIN:  41%|████      | 56/137 [00:12<00:12,  6.64it/s, epoch=5, grad_norm=54.832, loss=326.5303, lr=5.91e-05]
Epoch 5 TRAIN:  44%|████▍     | 60/137 [00:12<00:11,  6.68it/s, epoch=5, grad_norm=54.832, loss=326.5303, lr=5.91e-05]
Epoch 5 TRAIN:  44%|████▍     | 60/137 [00:12<00:11,  6.68it/s, epoch=5, grad_norm=66.886, loss=317.2753, lr=5.92e-05]
Epoch 5 TRAIN:  44%|████▍     | 60/137 [00:12<00:11,  6.68it/s, epoch=5, grad_norm=67.804, loss=381.1664, lr=5.93e-05]
Epoch 5 TRAIN:  44%|████▍     | 60/137 [00:12<00:11,  6.68it/s, epoch=5, grad_norm=58.473, loss=347.7180, lr=5.94e-05]
Epoch 5 TRAIN:  44%|████▍     | 60/137 [00:12<00:11,  6.68it/s, epoch=5, grad_norm=102.125, loss=306.5703, lr=5.94e-05]
Epoch 5 TRAIN:  47%|████▋     | 64/137 [00:12<00:10,  6.78it/s, epoch=5, grad_norm=102.125, loss=306.5703, lr=5.94e-05]
Epoch 5 TRAIN:  47%|████▋     | 64/137 [00:12<00:10,  6.78it/s, epoch=5, grad_norm=106.186, loss=384.3104, lr=5.95e-05]
Epoch 5 TRAIN:  47%|████▋     | 64/137 [00:12<00:10,  6.78it/s, epoch=5, grad_norm=104.978, loss=322.0450, lr=5.96e-05]
Epoch 5 TRAIN:  47%|████▋     | 64/137 [00:13<00:10,  6.78it/s, epoch=5, grad_norm=82.280, loss=353.2017, lr=5.97e-05] 
Epoch 5 TRAIN:  47%|████▋     | 64/137 [00:13<00:10,  6.78it/s, epoch=5, grad_norm=62.376, loss=351.0290, lr=5.98e-05]
Epoch 5 TRAIN:  50%|████▉     | 68/137 [00:13<00:10,  6.75it/s, epoch=5, grad_norm=62.376, loss=351.0290, lr=5.98e-05]
Epoch 5 TRAIN:  50%|████▉     | 68/137 [00:13<00:10,  6.75it/s, epoch=5, grad_norm=91.655, loss=345.7038, lr=5.98e-05]
Epoch 5 TRAIN:  50%|████▉     | 68/137 [00:13<00:10,  6.75it/s, epoch=5, grad_norm=288.909, loss=287.1123, lr=5.99e-05]
Epoch 5 TRAIN:  50%|████▉     | 68/137 [00:13<00:10,  6.75it/s, epoch=5, grad_norm=77.063, loss=373.7057, lr=6.00e-05] 
Epoch 5 TRAIN:  50%|████▉     | 68/137 [00:13<00:10,  6.75it/s, epoch=5, grad_norm=71.726, loss=319.1727, lr=6.01e-05]
Epoch 5 TRAIN:  53%|█████▎    | 72/137 [00:13<00:09,  6.62it/s, epoch=5, grad_norm=71.726, loss=319.1727, lr=6.01e-05]
Epoch 5 TRAIN:  53%|█████▎    | 72/137 [00:14<00:09,  6.62it/s, epoch=5, grad_norm=117.192, loss=300.0023, lr=6.02e-05]
Epoch 5 TRAIN:  53%|█████▎    | 72/137 [00:14<00:09,  6.62it/s, epoch=5, grad_norm=75.671, loss=391.6168, lr=6.02e-05] 
Epoch 5 TRAIN:  53%|█████▎    | 72/137 [00:14<00:09,  6.62it/s, epoch=5, grad_norm=110.204, loss=383.8359, lr=6.03e-05]
Epoch 5 TRAIN:  53%|█████▎    | 72/137 [00:14<00:09,  6.62it/s, epoch=5, grad_norm=126.296, loss=297.8131, lr=6.04e-05]
Epoch 5 TRAIN:  55%|█████▌    | 76/137 [00:14<00:09,  6.50it/s, epoch=5, grad_norm=126.296, loss=297.8131, lr=6.04e-05]
Epoch 5 TRAIN:  55%|█████▌    | 76/137 [00:14<00:09,  6.50it/s, epoch=5, grad_norm=79.838, loss=368.8651, lr=6.05e-05] 
Epoch 5 TRAIN:  55%|█████▌    | 76/137 [00:14<00:09,  6.50it/s, epoch=5, grad_norm=277.302, loss=340.4093, lr=6.06e-05]
Epoch 5 TRAIN:  55%|█████▌    | 76/137 [00:15<00:09,  6.50it/s, epoch=5, grad_norm=105.069, loss=318.7050, lr=6.06e-05]
Epoch 5 TRAIN:  55%|█████▌    | 76/137 [00:15<00:09,  6.50it/s, epoch=5, grad_norm=51.243, loss=341.8324, lr=6.07e-05] 
Epoch 5 TRAIN:  58%|█████▊    | 80/137 [00:15<00:08,  6.44it/s, epoch=5, grad_norm=51.243, loss=341.8324, lr=6.07e-05]
Epoch 5 TRAIN:  58%|█████▊    | 80/137 [00:15<00:08,  6.44it/s, epoch=5, grad_norm=70.642, loss=376.2422, lr=6.08e-05]
Epoch 5 TRAIN:  58%|█████▊    | 80/137 [00:15<00:08,  6.44it/s, epoch=5, grad_norm=126.703, loss=312.1327, lr=6.09e-05]
Epoch 5 TRAIN:  58%|█████▊    | 80/137 [00:15<00:08,  6.44it/s, epoch=5, grad_norm=119.547, loss=371.2736, lr=6.10e-05]
Epoch 5 TRAIN:  58%|█████▊    | 80/137 [00:15<00:08,  6.44it/s, epoch=5, grad_norm=62.978, loss=318.1713, lr=6.10e-05] 
Epoch 5 TRAIN:  61%|██████▏   | 84/137 [00:15<00:08,  6.46it/s, epoch=5, grad_norm=62.978, loss=318.1713, lr=6.10e-05]
Epoch 5 TRAIN:  61%|██████▏   | 84/137 [00:15<00:08,  6.46it/s, epoch=5, grad_norm=156.686, loss=369.0183, lr=6.11e-05]
Epoch 5 TRAIN:  61%|██████▏   | 84/137 [00:16<00:08,  6.46it/s, epoch=5, grad_norm=400.595, loss=277.5059, lr=6.12e-05]
Epoch 5 TRAIN:  61%|██████▏   | 84/137 [00:16<00:08,  6.46it/s, epoch=5, grad_norm=86.171, loss=365.6575, lr=6.13e-05] 
Epoch 5 TRAIN:  61%|██████▏   | 84/137 [00:16<00:08,  6.46it/s, epoch=5, grad_norm=90.588, loss=345.1818, lr=6.14e-05]
Epoch 5 TRAIN:  64%|██████▍   | 88/137 [00:16<00:07,  6.37it/s, epoch=5, grad_norm=90.588, loss=345.1818, lr=6.14e-05]
Epoch 5 TRAIN:  64%|██████▍   | 88/137 [00:16<00:07,  6.37it/s, epoch=5, grad_norm=276.818, loss=262.5080, lr=6.14e-05]
Epoch 5 TRAIN:  64%|██████▍   | 88/137 [00:16<00:07,  6.37it/s, epoch=5, grad_norm=318.679, loss=233.2170, lr=6.15e-05]
Epoch 5 TRAIN:  64%|██████▍   | 88/137 [00:16<00:07,  6.37it/s, epoch=5, grad_norm=63.218, loss=315.8664, lr=6.16e-05] 
Epoch 5 TRAIN:  64%|██████▍   | 88/137 [00:17<00:07,  6.37it/s, epoch=5, grad_norm=76.373, loss=367.6860, lr=6.17e-05]
Epoch 5 TRAIN:  67%|██████▋   | 92/137 [00:17<00:06,  6.46it/s, epoch=5, grad_norm=76.373, loss=367.6860, lr=6.17e-05]
Epoch 5 TRAIN:  67%|██████▋   | 92/137 [00:17<00:06,  6.46it/s, epoch=5, grad_norm=75.998, loss=285.0434, lr=6.18e-05]
Epoch 5 TRAIN:  67%|██████▋   | 92/137 [00:17<00:06,  6.46it/s, epoch=5, grad_norm=210.621, loss=355.2426, lr=6.18e-05]
Epoch 5 TRAIN:  67%|██████▋   | 92/137 [00:17<00:06,  6.46it/s, epoch=5, grad_norm=173.228, loss=270.7447, lr=6.19e-05]
Epoch 5 TRAIN:  67%|██████▋   | 92/137 [00:17<00:06,  6.46it/s, epoch=5, grad_norm=124.596, loss=342.5986, lr=6.20e-05]
Epoch 5 TRAIN:  70%|███████   | 96/137 [00:17<00:06,  6.63it/s, epoch=5, grad_norm=124.596, loss=342.5986, lr=6.20e-05]
Epoch 5 TRAIN:  70%|███████   | 96/137 [00:17<00:06,  6.63it/s, epoch=5, grad_norm=90.800, loss=313.9784, lr=6.21e-05] 
Epoch 5 TRAIN:  70%|███████   | 96/137 [00:17<00:06,  6.63it/s, epoch=5, grad_norm=73.528, loss=320.7441, lr=6.22e-05]
Epoch 5 TRAIN:  70%|███████   | 96/137 [00:18<00:06,  6.63it/s, epoch=5, grad_norm=57.530, loss=279.4834, lr=6.22e-05]
Epoch 5 TRAIN:  70%|███████   | 96/137 [00:18<00:06,  6.63it/s, epoch=5, grad_norm=213.473, loss=388.2848, lr=6.23e-05]2026-04-16 19:55:01,265 DEBUG TRAIN | steps/sec 5.544| Batch 5/100 loss 388.284821 loss_att 334.292877 loss_ctc 514.266052 th_accuracy 0.232339 lr 6.2320e-05 grad_norm 213.473480 rank 0

Epoch 5 TRAIN:  73%|███████▎  | 100/137 [00:18<00:05,  6.45it/s, epoch=5, grad_norm=213.473, loss=388.2848, lr=6.23e-05]
Epoch 5 TRAIN:  73%|███████▎  | 100/137 [00:18<00:05,  6.45it/s, epoch=5, grad_norm=150.900, loss=324.0854, lr=6.24e-05]
Epoch 5 TRAIN:  73%|███████▎  | 100/137 [00:18<00:05,  6.45it/s, epoch=5, grad_norm=120.448, loss=439.7205, lr=6.25e-05]
Epoch 5 TRAIN:  73%|███████▎  | 100/137 [00:18<00:05,  6.45it/s, epoch=5, grad_norm=60.679, loss=323.5779, lr=6.26e-05] 
Epoch 5 TRAIN:  73%|███████▎  | 100/137 [00:18<00:05,  6.45it/s, epoch=5, grad_norm=112.799, loss=331.7977, lr=6.26e-05]
Epoch 5 TRAIN:  76%|███████▌  | 104/137 [00:18<00:05,  6.45it/s, epoch=5, grad_norm=112.799, loss=331.7977, lr=6.26e-05]
Epoch 5 TRAIN:  76%|███████▌  | 104/137 [00:19<00:05,  6.45it/s, epoch=5, grad_norm=81.801, loss=378.6025, lr=6.27e-05] 
Epoch 5 TRAIN:  76%|███████▌  | 104/137 [00:19<00:05,  6.45it/s, epoch=5, grad_norm=170.551, loss=309.2124, lr=6.28e-05]
Epoch 5 TRAIN:  76%|███████▌  | 104/137 [00:19<00:05,  6.45it/s, epoch=5, grad_norm=127.041, loss=351.4320, lr=6.29e-05]
Epoch 5 TRAIN:  76%|███████▌  | 104/137 [00:19<00:05,  6.45it/s, epoch=5, grad_norm=82.838, loss=353.6518, lr=6.30e-05] 
Epoch 5 TRAIN:  79%|███████▉  | 108/137 [00:19<00:04,  6.24it/s, epoch=5, grad_norm=82.838, loss=353.6518, lr=6.30e-05]
Epoch 5 TRAIN:  79%|███████▉  | 108/137 [00:19<00:04,  6.24it/s, epoch=5, grad_norm=279.712, loss=196.1689, lr=6.30e-05]
Epoch 5 TRAIN:  79%|███████▉  | 108/137 [00:19<00:04,  6.24it/s, epoch=5, grad_norm=383.765, loss=346.1918, lr=6.31e-05]
Epoch 5 TRAIN:  79%|███████▉  | 108/137 [00:19<00:04,  6.24it/s, epoch=5, grad_norm=111.532, loss=364.3677, lr=6.32e-05]
Epoch 5 TRAIN:  79%|███████▉  | 108/137 [00:20<00:04,  6.24it/s, epoch=5, grad_norm=90.273, loss=294.9568, lr=6.33e-05] 
Epoch 5 TRAIN:  82%|████████▏ | 112/137 [00:20<00:03,  6.43it/s, epoch=5, grad_norm=90.273, loss=294.9568, lr=6.33e-05]
Epoch 5 TRAIN:  82%|████████▏ | 112/137 [00:20<00:03,  6.43it/s, epoch=5, grad_norm=101.714, loss=425.4795, lr=6.34e-05]
Epoch 5 TRAIN:  82%|████████▏ | 112/137 [00:20<00:03,  6.43it/s, epoch=5, grad_norm=157.481, loss=325.0859, lr=6.34e-05]
Epoch 5 TRAIN:  82%|████████▏ | 112/137 [00:20<00:03,  6.43it/s, epoch=5, grad_norm=125.857, loss=236.2825, lr=6.35e-05]
Epoch 5 TRAIN:  82%|████████▏ | 112/137 [00:20<00:03,  6.43it/s, epoch=5, grad_norm=70.549, loss=313.6975, lr=6.36e-05] 
Epoch 5 TRAIN:  85%|████████▍ | 116/137 [00:20<00:03,  6.37it/s, epoch=5, grad_norm=70.549, loss=313.6975, lr=6.36e-05]
Epoch 5 TRAIN:  85%|████████▍ | 116/137 [00:20<00:03,  6.37it/s, epoch=5, grad_norm=84.164, loss=254.2592, lr=6.37e-05]
Epoch 5 TRAIN:  85%|████████▍ | 116/137 [00:21<00:03,  6.37it/s, epoch=5, grad_norm=105.846, loss=342.9258, lr=6.38e-05]
Epoch 5 TRAIN:  85%|████████▍ | 116/137 [00:21<00:03,  6.37it/s, epoch=5, grad_norm=341.228, loss=424.5979, lr=6.38e-05]
Epoch 5 TRAIN:  85%|████████▍ | 116/137 [00:21<00:03,  6.37it/s, epoch=5, grad_norm=89.572, loss=300.8244, lr=6.39e-05] 
Epoch 5 TRAIN:  88%|████████▊ | 120/137 [00:21<00:02,  6.47it/s, epoch=5, grad_norm=89.572, loss=300.8244, lr=6.39e-05]
Epoch 5 TRAIN:  88%|████████▊ | 120/137 [00:21<00:02,  6.47it/s, epoch=5, grad_norm=169.420, loss=371.7058, lr=6.40e-05]
Epoch 5 TRAIN:  88%|████████▊ | 120/137 [00:21<00:02,  6.47it/s, epoch=5, grad_norm=141.426, loss=311.3051, lr=6.41e-05]
Epoch 5 TRAIN:  88%|████████▊ | 120/137 [00:21<00:02,  6.47it/s, epoch=5, grad_norm=79.170, loss=302.3276, lr=6.42e-05] 
Epoch 5 TRAIN:  88%|████████▊ | 120/137 [00:21<00:02,  6.47it/s, epoch=5, grad_norm=335.844, loss=419.7790, lr=6.42e-05]
Epoch 5 TRAIN:  91%|█████████ | 124/137 [00:21<00:01,  6.58it/s, epoch=5, grad_norm=335.844, loss=419.7790, lr=6.42e-05]
Epoch 5 TRAIN:  91%|█████████ | 124/137 [00:22<00:01,  6.58it/s, epoch=5, grad_norm=91.571, loss=335.2056, lr=6.43e-05] 
Epoch 5 TRAIN:  91%|█████████ | 124/137 [00:22<00:01,  6.58it/s, epoch=5, grad_norm=81.907, loss=309.1614, lr=6.44e-05]
Epoch 5 TRAIN:  91%|█████████ | 124/137 [00:22<00:01,  6.58it/s, epoch=5, grad_norm=81.718, loss=393.8626, lr=6.45e-05]
Epoch 5 TRAIN:  91%|█████████ | 124/137 [00:22<00:01,  6.58it/s, epoch=5, grad_norm=178.308, loss=370.3712, lr=6.46e-05]
Epoch 5 TRAIN:  93%|█████████▎| 128/137 [00:22<00:01,  6.66it/s, epoch=5, grad_norm=178.308, loss=370.3712, lr=6.46e-05]
Epoch 5 TRAIN:  93%|█████████▎| 128/137 [00:22<00:01,  6.66it/s, epoch=5, grad_norm=204.444, loss=421.7602, lr=6.46e-05]
Epoch 5 TRAIN:  93%|█████████▎| 128/137 [00:22<00:01,  6.66it/s, epoch=5, grad_norm=193.967, loss=391.2272, lr=6.47e-05]
Epoch 5 TRAIN:  93%|█████████▎| 128/137 [00:22<00:01,  6.66it/s, epoch=5, grad_norm=103.889, loss=367.4167, lr=6.48e-05]
Epoch 5 TRAIN:  93%|█████████▎| 128/137 [00:23<00:01,  6.66it/s, epoch=5, grad_norm=143.996, loss=473.3311, lr=6.49e-05]
Epoch 5 TRAIN:  96%|█████████▋| 132/137 [00:23<00:00,  6.86it/s, epoch=5, grad_norm=143.996, loss=473.3311, lr=6.49e-05]
Epoch 5 TRAIN:  96%|█████████▋| 132/137 [00:23<00:00,  6.86it/s, epoch=5, grad_norm=92.644, loss=359.8723, lr=6.50e-05] 
Epoch 5 TRAIN:  96%|█████████▋| 132/137 [00:23<00:00,  6.86it/s, epoch=5, grad_norm=171.733, loss=475.2291, lr=6.50e-05]
Epoch 5 TRAIN:  96%|█████████▋| 132/137 [00:23<00:00,  6.86it/s, epoch=5, grad_norm=136.920, loss=361.8075, lr=6.51e-05]
Epoch 5 TRAIN:  96%|█████████▋| 132/137 [00:23<00:00,  6.86it/s, epoch=5, grad_norm=392.054, loss=321.6708, lr=6.52e-05]
Epoch 5 TRAIN:  99%|█████████▉| 136/137 [00:23<00:00,  7.16it/s, epoch=5, grad_norm=392.054, loss=321.6708, lr=6.52e-05]
Epoch 5 TRAIN:  99%|█████████▉| 136/137 [00:23<00:00,  5.76it/s, epoch=5, grad_norm=392.054, loss=321.6708, lr=6.52e-05]

CV:   0%|          | 0/6 [00:00<?, ?it/s]/home/rookie/Documents/batuka/projects/wenet/wenet/wenet/utils/train_utils.py:689: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  amp_autocast(enabled=dtype is not None,

CV:  17%|█▋        | 1/6 [00:00<00:03,  1.34it/s]
                                                 
2026-04-16 19:55:07,453 INFO Epoch 5 Step 815 CV info lr 6.5200e-05 cv_loss 396.7141834605824 rank 0 acc 0.23826897889375687
2026-04-16 19:55:07,455 INFO [Rank 0] Checkpoint: save to checkpoint exp/conformer_mongolian/epoch_5.pt
2026-04-16 19:55:07,565 INFO Epoch 6 Step 815 TRAIN info lr 6.5200e-05 rank 0
2026-04-16 19:55:07,582 INFO using accumulate grad, new batch size is 1 times larger than before

Epoch 6 TRAIN:   0%|          | 0/136 [00:00<?, ?it/s]
Epoch 6 TRAIN:   0%|          | 0/136 [00:03<?, ?it/s, epoch=6, grad_norm=418.545, loss=384.1346, lr=6.53e-05]
Epoch 6 TRAIN:   1%|          | 1/136 [00:03<08:32,  3.80s/it, epoch=6, grad_norm=418.545, loss=384.1346, lr=6.53e-05]
Epoch 6 TRAIN:   1%|          | 1/136 [00:03<08:32,  3.80s/it, epoch=6, grad_norm=324.742, loss=326.9811, lr=6.54e-05]
Epoch 6 TRAIN:   1%|          | 1/136 [00:04<08:32,  3.80s/it, epoch=6, grad_norm=383.981, loss=367.6492, lr=6.54e-05]
Epoch 6 TRAIN:   1%|          | 1/136 [00:04<08:32,  3.80s/it, epoch=6, grad_norm=228.473, loss=281.6220, lr=6.55e-05]
Epoch 6 TRAIN:   1%|          | 1/136 [00:04<08:32,  3.80s/it, epoch=6, grad_norm=319.908, loss=341.1357, lr=6.56e-05]
Epoch 6 TRAIN:   1%|          | 1/136 [00:04<08:32,  3.80s/it, epoch=6, grad_norm=424.285, loss=398.5135, lr=6.57e-05]
Epoch 6 TRAIN:   4%|▍         | 6/136 [00:04<01:12,  1.80it/s, epoch=6, grad_norm=424.285, loss=398.5135, lr=6.57e-05]
Epoch 6 TRAIN:   4%|▍         | 6/136 [00:04<01:12,  1.80it/s, epoch=6, grad_norm=278.229, loss=323.5786, lr=6.58e-05]
Epoch 6 TRAIN:   4%|▍         | 6/136 [00:04<01:12,  1.80it/s, epoch=6, grad_norm=427.153, loss=432.7069, lr=6.58e-05]
Epoch 6 TRAIN:   4%|▍         | 6/136 [00:04<01:12,  1.80it/s, epoch=6, grad_norm=202.242, loss=365.3525, lr=6.59e-05]
Epoch 6 TRAIN:   4%|▍         | 6/136 [00:04<01:12,  1.80it/s, epoch=6, grad_norm=106.068, loss=343.8589, lr=6.60e-05]
Epoch 6 TRAIN:   4%|▍         | 6/136 [00:04<01:12,  1.80it/s, epoch=6, grad_norm=62.560, loss=311.5047, lr=6.61e-05] 
Epoch 6 TRAIN:   8%|▊         | 11/136 [00:04<00:38,  3.22it/s, epoch=6, grad_norm=62.560, loss=311.5047, lr=6.61e-05]
Epoch 6 TRAIN:   8%|▊         | 11/136 [00:05<00:38,  3.22it/s, epoch=6, grad_norm=46.410, loss=338.6637, lr=6.62e-05]
Epoch 6 TRAIN:   8%|▊         | 11/136 [00:05<00:38,  3.22it/s, epoch=6, grad_norm=89.603, loss=330.2929, lr=6.62e-05]
Epoch 6 TRAIN:   8%|▊         | 11/136 [00:05<00:38,  3.22it/s, epoch=6, grad_norm=79.530, loss=372.2302, lr=6.63e-05]
Epoch 6 TRAIN:   8%|▊         | 11/136 [00:05<00:38,  3.22it/s, epoch=6, grad_norm=60.088, loss=378.9061, lr=6.64e-05]
Epoch 6 TRAIN:  11%|█         | 15/136 [00:05<00:30,  4.03it/s, epoch=6, grad_norm=60.088, loss=378.9061, lr=6.64e-05]
Epoch 6 TRAIN:  11%|█         | 15/136 [00:05<00:30,  4.03it/s, epoch=6, grad_norm=53.687, loss=420.5802, lr=6.65e-05]
Epoch 6 TRAIN:  11%|█         | 15/136 [00:05<00:30,  4.03it/s, epoch=6, grad_norm=255.134, loss=315.0592, lr=6.66e-05]
Epoch 6 TRAIN:  11%|█         | 15/136 [00:05<00:30,  4.03it/s, epoch=6, grad_norm=339.918, loss=306.7872, lr=6.66e-05]
Epoch 6 TRAIN:  11%|█         | 15/136 [00:06<00:30,  4.03it/s, epoch=6, grad_norm=250.684, loss=310.4320, lr=6.67e-05]
Epoch 6 TRAIN:  14%|█▍        | 19/136 [00:06<00:24,  4.70it/s, epoch=6, grad_norm=250.684, loss=310.4320, lr=6.67e-05]
Epoch 6 TRAIN:  14%|█▍        | 19/136 [00:06<00:24,  4.70it/s, epoch=6, grad_norm=125.572, loss=361.6190, lr=6.68e-05]
Epoch 6 TRAIN:  14%|█▍        | 19/136 [00:06<00:24,  4.70it/s, epoch=6, grad_norm=103.492, loss=358.0749, lr=6.69e-05]
Epoch 6 TRAIN:  14%|█▍        | 19/136 [00:06<00:24,  4.70it/s, epoch=6, grad_norm=78.183, loss=355.2709, lr=6.70e-05] 
Epoch 6 TRAIN:  14%|█▍        | 19/136 [00:06<00:24,  4.70it/s, epoch=6, grad_norm=66.733, loss=365.1771, lr=6.70e-05]
Epoch 6 TRAIN:  17%|█▋        | 23/136 [00:06<00:21,  5.20it/s, epoch=6, grad_norm=66.733, loss=365.1771, lr=6.70e-05]
Epoch 6 TRAIN:  17%|█▋        | 23/136 [00:06<00:21,  5.20it/s, epoch=6, grad_norm=38.957, loss=326.1598, lr=6.71e-05]
Epoch 6 TRAIN:  17%|█▋        | 23/136 [00:06<00:21,  5.20it/s, epoch=6, grad_norm=76.857, loss=302.0659, lr=6.72e-05]
Epoch 6 TRAIN:  17%|█▋        | 23/136 [00:07<00:21,  5.20it/s, epoch=6, grad_norm=104.040, loss=372.5174, lr=6.73e-05]
Epoch 6 TRAIN:  17%|█▋        | 23/136 [00:07<00:21,  5.20it/s, epoch=6, grad_norm=87.244, loss=334.3253, lr=6.74e-05] 
Epoch 6 TRAIN:  20%|█▉        | 27/136 [00:07<00:19,  5.59it/s, epoch=6, grad_norm=87.244, loss=334.3253, lr=6.74e-05]
Epoch 6 TRAIN:  20%|█▉        | 27/136 [00:07<00:19,  5.59it/s, epoch=6, grad_norm=137.268, loss=354.8734, lr=6.74e-05]
Epoch 6 TRAIN:  20%|█▉        | 27/136 [00:07<00:19,  5.59it/s, epoch=6, grad_norm=134.623, loss=353.5985, lr=6.75e-05]
Epoch 6 TRAIN:  20%|█▉        | 27/136 [00:07<00:19,  5.59it/s, epoch=6, grad_norm=120.285, loss=349.6113, lr=6.76e-05]
Epoch 6 TRAIN:  20%|█▉        | 27/136 [00:07<00:19,  5.59it/s, epoch=6, grad_norm=113.752, loss=359.0378, lr=6.77e-05]
Epoch 6 TRAIN:  23%|██▎       | 31/136 [00:07<00:17,  5.92it/s, epoch=6, grad_norm=113.752, loss=359.0378, lr=6.77e-05]
Epoch 6 TRAIN:  23%|██▎       | 31/136 [00:08<00:17,  5.92it/s, epoch=6, grad_norm=53.637, loss=329.2001, lr=6.78e-05] 
Epoch 6 TRAIN:  23%|██▎       | 31/136 [00:08<00:17,  5.92it/s, epoch=6, grad_norm=50.205, loss=350.9483, lr=6.78e-05]
Epoch 6 TRAIN:  23%|██▎       | 31/136 [00:08<00:17,  5.92it/s, epoch=6, grad_norm=157.248, loss=327.1053, lr=6.79e-05]
Epoch 6 TRAIN:  23%|██▎       | 31/136 [00:08<00:17,  5.92it/s, epoch=6, grad_norm=108.024, loss=349.4598, lr=6.80e-05]
Epoch 6 TRAIN:  26%|██▌       | 35/136 [00:08<00:16,  6.18it/s, epoch=6, grad_norm=108.024, loss=349.4598, lr=6.80e-05]
Epoch 6 TRAIN:  26%|██▌       | 35/136 [00:08<00:16,  6.18it/s, epoch=6, grad_norm=52.095, loss=368.7137, lr=6.81e-05] 
Epoch 6 TRAIN:  26%|██▌       | 35/136 [00:08<00:16,  6.18it/s, epoch=6, grad_norm=86.727, loss=335.0067, lr=6.82e-05]
Epoch 6 TRAIN:  26%|██▌       | 35/136 [00:08<00:16,  6.18it/s, epoch=6, grad_norm=94.268, loss=335.5784, lr=6.82e-05]
Epoch 6 TRAIN:  26%|██▌       | 35/136 [00:09<00:16,  6.18it/s, epoch=6, grad_norm=126.291, loss=309.1729, lr=6.83e-05]
Epoch 6 TRAIN:  29%|██▊       | 39/136 [00:09<00:15,  6.38it/s, epoch=6, grad_norm=126.291, loss=309.1729, lr=6.83e-05]
Epoch 6 TRAIN:  29%|██▊       | 39/136 [00:09<00:15,  6.38it/s, epoch=6, grad_norm=46.542, loss=339.9987, lr=6.84e-05] 
Epoch 6 TRAIN:  29%|██▊       | 39/136 [00:09<00:15,  6.38it/s, epoch=6, grad_norm=85.036, loss=330.3197, lr=6.85e-05]
Epoch 6 TRAIN:  29%|██▊       | 39/136 [00:09<00:15,  6.38it/s, epoch=6, grad_norm=64.946, loss=338.9869, lr=6.86e-05]
Epoch 6 TRAIN:  29%|██▊       | 39/136 [00:09<00:15,  6.38it/s, epoch=6, grad_norm=69.767, loss=357.6673, lr=6.86e-05]
Epoch 6 TRAIN:  32%|███▏      | 43/136 [00:09<00:14,  6.46it/s, epoch=6, grad_norm=69.767, loss=357.6673, lr=6.86e-05]
Epoch 6 TRAIN:  32%|███▏      | 43/136 [00:09<00:14,  6.46it/s, epoch=6, grad_norm=80.151, loss=287.4424, lr=6.87e-05]
Epoch 6 TRAIN:  32%|███▏      | 43/136 [00:09<00:14,  6.46it/s, epoch=6, grad_norm=115.953, loss=354.8586, lr=6.88e-05]
Epoch 6 TRAIN:  32%|███▏      | 43/136 [00:10<00:14,  6.46it/s, epoch=6, grad_norm=157.282, loss=225.9918, lr=6.89e-05]
Epoch 6 TRAIN:  32%|███▏      | 43/136 [00:10<00:14,  6.46it/s, epoch=6, grad_norm=129.113, loss=329.7941, lr=6.90e-05]
Epoch 6 TRAIN:  35%|███▍      | 47/136 [00:10<00:13,  6.55it/s, epoch=6, grad_norm=129.113, loss=329.7941, lr=6.90e-05]
Epoch 6 TRAIN:  35%|███▍      | 47/136 [00:10<00:13,  6.55it/s, epoch=6, grad_norm=183.390, loss=360.1082, lr=6.90e-05]
Epoch 6 TRAIN:  35%|███▍      | 47/136 [00:10<00:13,  6.55it/s, epoch=6, grad_norm=168.041, loss=359.3484, lr=6.91e-05]
Epoch 6 TRAIN:  35%|███▍      | 47/136 [00:10<00:13,  6.55it/s, epoch=6, grad_norm=128.903, loss=351.7486, lr=6.92e-05]
Epoch 6 TRAIN:  35%|███▍      | 47/136 [00:10<00:13,  6.55it/s, epoch=6, grad_norm=171.714, loss=379.9444, lr=6.93e-05]
Epoch 6 TRAIN:  38%|███▊      | 51/136 [00:10<00:12,  6.68it/s, epoch=6, grad_norm=171.714, loss=379.9444, lr=6.93e-05]
Epoch 6 TRAIN:  38%|███▊      | 51/136 [00:10<00:12,  6.68it/s, epoch=6, grad_norm=99.930, loss=354.7786, lr=6.94e-05] 
Epoch 6 TRAIN:  38%|███▊      | 51/136 [00:11<00:12,  6.68it/s, epoch=6, grad_norm=78.902, loss=359.4924, lr=6.94e-05]
Epoch 6 TRAIN:  38%|███▊      | 51/136 [00:11<00:12,  6.68it/s, epoch=6, grad_norm=66.859, loss=371.3759, lr=6.95e-05]
Epoch 6 TRAIN:  38%|███▊      | 51/136 [00:11<00:12,  6.68it/s, epoch=6, grad_norm=75.157, loss=334.0870, lr=6.96e-05]
Epoch 6 TRAIN:  40%|████      | 55/136 [00:11<00:12,  6.64it/s, epoch=6, grad_norm=75.157, loss=334.0870, lr=6.96e-05]
Epoch 6 TRAIN:  40%|████      | 55/136 [00:11<00:12,  6.64it/s, epoch=6, grad_norm=100.685, loss=351.1515, lr=6.97e-05]
Epoch 6 TRAIN:  40%|████      | 55/136 [00:11<00:12,  6.64it/s, epoch=6, grad_norm=147.890, loss=367.4008, lr=6.98e-05]
Epoch 6 TRAIN:  40%|████      | 55/136 [00:11<00:12,  6.64it/s, epoch=6, grad_norm=130.784, loss=386.7469, lr=6.98e-05]
Epoch 6 TRAIN:  40%|████      | 55/136 [00:12<00:12,  6.64it/s, epoch=6, grad_norm=162.656, loss=345.8452, lr=6.99e-05]
Epoch 6 TRAIN:  43%|████▎     | 59/136 [00:12<00:11,  6.68it/s, epoch=6, grad_norm=162.656, loss=345.8452, lr=6.99e-05]
Epoch 6 TRAIN:  43%|████▎     | 59/136 [00:12<00:11,  6.68it/s, epoch=6, grad_norm=77.295, loss=372.9185, lr=7.00e-05] 
Epoch 6 TRAIN:  43%|████▎     | 59/136 [00:12<00:11,  6.68it/s, epoch=6, grad_norm=85.718, loss=364.0873, lr=7.01e-05]
Epoch 6 TRAIN:  43%|████▎     | 59/136 [00:12<00:11,  6.68it/s, epoch=6, grad_norm=56.377, loss=353.4928, lr=7.02e-05]
Epoch 6 TRAIN:  43%|████▎     | 59/136 [00:12<00:11,  6.68it/s, epoch=6, grad_norm=67.208, loss=359.3481, lr=7.02e-05]
Epoch 6 TRAIN:  46%|████▋     | 63/136 [00:12<00:10,  6.70it/s, epoch=6, grad_norm=67.208, loss=359.3481, lr=7.02e-05]
Epoch 6 TRAIN:  46%|████▋     | 63/136 [00:12<00:10,  6.70it/s, epoch=6, grad_norm=60.222, loss=307.3467, lr=7.03e-05]
Epoch 6 TRAIN:  46%|████▋     | 63/136 [00:12<00:10,  6.70it/s, epoch=6, grad_norm=196.005, loss=273.2498, lr=7.04e-05]
Epoch 6 TRAIN:  46%|████▋     | 63/136 [00:13<00:10,  6.70it/s, epoch=6, grad_norm=141.974, loss=283.6632, lr=7.05e-05]
Epoch 6 TRAIN:  46%|████▋     | 63/136 [00:13<00:10,  6.70it/s, epoch=6, grad_norm=77.528, loss=339.5060, lr=7.06e-05] 
Epoch 6 TRAIN:  49%|████▉     | 67/136 [00:13<00:10,  6.65it/s, epoch=6, grad_norm=77.528, loss=339.5060, lr=7.06e-05]
Epoch 6 TRAIN:  49%|████▉     | 67/136 [00:13<00:10,  6.65it/s, epoch=6, grad_norm=119.889, loss=325.9507, lr=7.06e-05]
Epoch 6 TRAIN:  49%|████▉     | 67/136 [00:13<00:10,  6.65it/s, epoch=6, grad_norm=198.457, loss=373.0526, lr=7.07e-05]
Epoch 6 TRAIN:  49%|████▉     | 67/136 [00:13<00:10,  6.65it/s, epoch=6, grad_norm=228.246, loss=389.6771, lr=7.08e-05]
Epoch 6 TRAIN:  49%|████▉     | 67/136 [00:13<00:10,  6.65it/s, epoch=6, grad_norm=267.812, loss=395.1635, lr=7.09e-05]
Epoch 6 TRAIN:  52%|█████▏    | 71/136 [00:13<00:09,  6.68it/s, epoch=6, grad_norm=267.812, loss=395.1635, lr=7.09e-05]
Epoch 6 TRAIN:  52%|█████▏    | 71/136 [00:13<00:09,  6.68it/s, epoch=6, grad_norm=242.031, loss=392.9334, lr=7.10e-05]
Epoch 6 TRAIN:  52%|█████▏    | 71/136 [00:14<00:09,  6.68it/s, epoch=6, grad_norm=58.125, loss=356.0232, lr=7.10e-05] 
Epoch 6 TRAIN:  52%|█████▏    | 71/136 [00:14<00:09,  6.68it/s, epoch=6, grad_norm=151.655, loss=314.0908, lr=7.11e-05]
Epoch 6 TRAIN:  52%|█████▏    | 71/136 [00:14<00:09,  6.68it/s, epoch=6, grad_norm=73.683, loss=360.8496, lr=7.12e-05] 
Epoch 6 TRAIN:  55%|█████▌    | 75/136 [00:14<00:09,  6.70it/s, epoch=6, grad_norm=73.683, loss=360.8496, lr=7.12e-05]
Epoch 6 TRAIN:  55%|█████▌    | 75/136 [00:14<00:09,  6.70it/s, epoch=6, grad_norm=81.427, loss=314.5395, lr=7.13e-05]
Epoch 6 TRAIN:  55%|█████▌    | 75/136 [00:14<00:09,  6.70it/s, epoch=6, grad_norm=70.764, loss=366.8548, lr=7.14e-05]
Epoch 6 TRAIN:  55%|█████▌    | 75/136 [00:14<00:09,  6.70it/s, epoch=6, grad_norm=127.779, loss=416.1308, lr=7.14e-05]
Epoch 6 TRAIN:  55%|█████▌    | 75/136 [00:15<00:09,  6.70it/s, epoch=6, grad_norm=148.900, loss=276.0632, lr=7.15e-05]
Epoch 6 TRAIN:  58%|█████▊    | 79/136 [00:15<00:08,  6.61it/s, epoch=6, grad_norm=148.900, loss=276.0632, lr=7.15e-05]
Epoch 6 TRAIN:  58%|█████▊    | 79/136 [00:15<00:08,  6.61it/s, epoch=6, grad_norm=60.398, loss=355.9385, lr=7.16e-05] 
Epoch 6 TRAIN:  58%|█████▊    | 79/136 [00:15<00:08,  6.61it/s, epoch=6, grad_norm=145.388, loss=407.7412, lr=7.17e-05]
Epoch 6 TRAIN:  58%|█████▊    | 79/136 [00:15<00:08,  6.61it/s, epoch=6, grad_norm=278.313, loss=316.6507, lr=7.18e-05]
Epoch 6 TRAIN:  58%|█████▊    | 79/136 [00:15<00:08,  6.61it/s, epoch=6, grad_norm=63.654, loss=399.1331, lr=7.18e-05] 
Epoch 6 TRAIN:  61%|██████    | 83/136 [00:15<00:08,  6.60it/s, epoch=6, grad_norm=63.654, loss=399.1331, lr=7.18e-05]
Epoch 6 TRAIN:  61%|██████    | 83/136 [00:15<00:08,  6.60it/s, epoch=6, grad_norm=221.203, loss=289.9001, lr=7.19e-05]
Epoch 6 TRAIN:  61%|██████    | 83/136 [00:15<00:08,  6.60it/s, epoch=6, grad_norm=133.383, loss=316.3297, lr=7.20e-05]
Epoch 6 TRAIN:  61%|██████    | 83/136 [00:16<00:08,  6.60it/s, epoch=6, grad_norm=62.894, loss=332.2376, lr=7.21e-05] 
Epoch 6 TRAIN:  61%|██████    | 83/136 [00:16<00:08,  6.60it/s, epoch=6, grad_norm=60.977, loss=313.7367, lr=7.22e-05]
Epoch 6 TRAIN:  64%|██████▍   | 87/136 [00:16<00:07,  6.63it/s, epoch=6, grad_norm=60.977, loss=313.7367, lr=7.22e-05]
Epoch 6 TRAIN:  64%|██████▍   | 87/136 [00:16<00:07,  6.63it/s, epoch=6, grad_norm=60.208, loss=307.2421, lr=7.22e-05]
Epoch 6 TRAIN:  64%|██████▍   | 87/136 [00:16<00:07,  6.63it/s, epoch=6, grad_norm=286.921, loss=276.8898, lr=7.23e-05]
Epoch 6 TRAIN:  64%|██████▍   | 87/136 [00:16<00:07,  6.63it/s, epoch=6, grad_norm=188.638, loss=280.0017, lr=7.24e-05]
Epoch 6 TRAIN:  64%|██████▍   | 87/136 [00:16<00:07,  6.63it/s, epoch=6, grad_norm=61.732, loss=299.6204, lr=7.25e-05] 
Epoch 6 TRAIN:  67%|██████▋   | 91/136 [00:16<00:06,  6.73it/s, epoch=6, grad_norm=61.732, loss=299.6204, lr=7.25e-05]
Epoch 6 TRAIN:  67%|██████▋   | 91/136 [00:16<00:06,  6.73it/s, epoch=6, grad_norm=139.157, loss=368.2777, lr=7.26e-05]
Epoch 6 TRAIN:  67%|██████▋   | 91/136 [00:17<00:06,  6.73it/s, epoch=6, grad_norm=146.148, loss=323.5463, lr=7.26e-05]
Epoch 6 TRAIN:  67%|██████▋   | 91/136 [00:17<00:06,  6.73it/s, epoch=6, grad_norm=198.707, loss=347.7904, lr=7.27e-05]
Epoch 6 TRAIN:  67%|██████▋   | 91/136 [00:17<00:06,  6.73it/s, epoch=6, grad_norm=199.615, loss=343.8216, lr=7.28e-05]
Epoch 6 TRAIN:  70%|██████▉   | 95/136 [00:17<00:06,  6.68it/s, epoch=6, grad_norm=199.615, loss=343.8216, lr=7.28e-05]
Epoch 6 TRAIN:  70%|██████▉   | 95/136 [00:17<00:06,  6.68it/s, epoch=6, grad_norm=104.589, loss=317.2703, lr=7.29e-05]
Epoch 6 TRAIN:  70%|██████▉   | 95/136 [00:17<00:06,  6.68it/s, epoch=6, grad_norm=89.692, loss=337.2992, lr=7.30e-05] 
Epoch 6 TRAIN:  70%|██████▉   | 95/136 [00:17<00:06,  6.68it/s, epoch=6, grad_norm=104.046, loss=381.7601, lr=7.30e-05]
Epoch 6 TRAIN:  70%|██████▉   | 95/136 [00:18<00:06,  6.68it/s, epoch=6, grad_norm=83.644, loss=344.7240, lr=7.31e-05] 
Epoch 6 TRAIN:  73%|███████▎  | 99/136 [00:18<00:05,  6.55it/s, epoch=6, grad_norm=83.644, loss=344.7240, lr=7.31e-05]
Epoch 6 TRAIN:  73%|███████▎  | 99/136 [00:18<00:05,  6.55it/s, epoch=6, grad_norm=82.024, loss=353.4694, lr=7.32e-05]2026-04-16 19:55:25,784 DEBUG TRAIN | steps/sec 5.547| Batch 6/100 loss 353.469360 loss_att 310.364319 loss_ctc 454.047821 th_accuracy 0.246637 lr 7.3200e-05 grad_norm 82.023544 rank 0

Epoch 6 TRAIN:  73%|███████▎  | 99/136 [00:18<00:05,  6.55it/s, epoch=6, grad_norm=197.926, loss=252.2987, lr=7.33e-05]
Epoch 6 TRAIN:  73%|███████▎  | 99/136 [00:18<00:05,  6.55it/s, epoch=6, grad_norm=80.883, loss=319.0270, lr=7.34e-05] 
Epoch 6 TRAIN:  73%|███████▎  | 99/136 [00:18<00:05,  6.55it/s, epoch=6, grad_norm=120.657, loss=353.8828, lr=7.34e-05]
Epoch 6 TRAIN:  76%|███████▌  | 103/136 [00:18<00:05,  6.56it/s, epoch=6, grad_norm=120.657, loss=353.8828, lr=7.34e-05]
Epoch 6 TRAIN:  76%|███████▌  | 103/136 [00:18<00:05,  6.56it/s, epoch=6, grad_norm=84.565, loss=314.4537, lr=7.35e-05] 
Epoch 6 TRAIN:  76%|███████▌  | 103/136 [00:18<00:05,  6.56it/s, epoch=6, grad_norm=154.838, loss=367.7688, lr=7.36e-05]
Epoch 6 TRAIN:  76%|███████▌  | 103/136 [00:19<00:05,  6.56it/s, epoch=6, grad_norm=104.187, loss=369.9387, lr=7.37e-05]
Epoch 6 TRAIN:  76%|███████▌  | 103/136 [00:19<00:05,  6.56it/s, epoch=6, grad_norm=149.093, loss=310.9646, lr=7.38e-05]
Epoch 6 TRAIN:  79%|███████▊  | 107/136 [00:19<00:04,  6.41it/s, epoch=6, grad_norm=149.093, loss=310.9646, lr=7.38e-05]
Epoch 6 TRAIN:  79%|███████▊  | 107/136 [00:19<00:04,  6.41it/s, epoch=6, grad_norm=139.130, loss=271.6907, lr=7.38e-05]
Epoch 6 TRAIN:  79%|███████▊  | 107/136 [00:19<00:04,  6.41it/s, epoch=6, grad_norm=205.029, loss=399.8973, lr=7.39e-05]
Epoch 6 TRAIN:  79%|███████▊  | 107/136 [00:19<00:04,  6.41it/s, epoch=6, grad_norm=57.321, loss=290.5527, lr=7.40e-05] 
Epoch 6 TRAIN:  79%|███████▊  | 107/136 [00:19<00:04,  6.41it/s, epoch=6, grad_norm=131.952, loss=321.8234, lr=7.41e-05]
Epoch 6 TRAIN:  82%|████████▏ | 111/136 [00:19<00:03,  6.41it/s, epoch=6, grad_norm=131.952, loss=321.8234, lr=7.41e-05]
Epoch 6 TRAIN:  82%|████████▏ | 111/136 [00:20<00:03,  6.41it/s, epoch=6, grad_norm=89.003, loss=299.0891, lr=7.42e-05] 
Epoch 6 TRAIN:  82%|████████▏ | 111/136 [00:20<00:03,  6.41it/s, epoch=6, grad_norm=144.541, loss=459.1711, lr=7.42e-05]
Epoch 6 TRAIN:  82%|████████▏ | 111/136 [00:20<00:03,  6.41it/s, epoch=6, grad_norm=172.114, loss=293.4055, lr=7.43e-05]
Epoch 6 TRAIN:  82%|████████▏ | 111/136 [00:20<00:03,  6.41it/s, epoch=6, grad_norm=152.003, loss=367.1801, lr=7.44e-05]
Epoch 6 TRAIN:  85%|████████▍ | 115/136 [00:20<00:03,  6.77it/s, epoch=6, grad_norm=152.003, loss=367.1801, lr=7.44e-05]
Epoch 6 TRAIN:  85%|████████▍ | 115/136 [00:20<00:03,  6.77it/s, epoch=6, grad_norm=78.559, loss=313.8271, lr=7.45e-05] 
Epoch 6 TRAIN:  85%|████████▍ | 115/136 [00:20<00:03,  6.77it/s, epoch=6, grad_norm=79.447, loss=281.2276, lr=7.46e-05]
Epoch 6 TRAIN:  85%|████████▍ | 115/136 [00:20<00:03,  6.77it/s, epoch=6, grad_norm=105.607, loss=249.1609, lr=7.46e-05]
Epoch 6 TRAIN:  85%|████████▍ | 115/136 [00:21<00:03,  6.77it/s, epoch=6, grad_norm=80.070, loss=246.2355, lr=7.47e-05] 
Epoch 6 TRAIN:  88%|████████▊ | 119/136 [00:21<00:02,  6.61it/s, epoch=6, grad_norm=80.070, loss=246.2355, lr=7.47e-05]
Epoch 6 TRAIN:  88%|████████▊ | 119/136 [00:21<00:02,  6.61it/s, epoch=6, grad_norm=191.315, loss=368.0862, lr=7.48e-05]
Epoch 6 TRAIN:  88%|████████▊ | 119/136 [00:21<00:02,  6.61it/s, epoch=6, grad_norm=113.098, loss=346.8254, lr=7.49e-05]
Epoch 6 TRAIN:  88%|████████▊ | 119/136 [00:21<00:02,  6.61it/s, epoch=6, grad_norm=213.616, loss=441.8222, lr=7.50e-05]
Epoch 6 TRAIN:  88%|████████▊ | 119/136 [00:21<00:02,  6.61it/s, epoch=6, grad_norm=163.772, loss=387.4611, lr=7.50e-05]
Epoch 6 TRAIN:  90%|█████████ | 123/136 [00:21<00:01,  6.95it/s, epoch=6, grad_norm=163.772, loss=387.4611, lr=7.50e-05]
Epoch 6 TRAIN:  90%|█████████ | 123/136 [00:21<00:01,  6.95it/s, epoch=6, grad_norm=68.579, loss=309.0146, lr=7.51e-05] 
Epoch 6 TRAIN:  90%|█████████ | 123/136 [00:21<00:01,  6.95it/s, epoch=6, grad_norm=62.397, loss=307.9369, lr=7.52e-05]
Epoch 6 TRAIN:  90%|█████████ | 123/136 [00:22<00:01,  6.95it/s, epoch=6, grad_norm=63.033, loss=301.9708, lr=7.53e-05]
Epoch 6 TRAIN:  90%|█████████ | 123/136 [00:22<00:01,  6.95it/s, epoch=6, grad_norm=128.670, loss=398.6862, lr=7.54e-05]
Epoch 6 TRAIN:  93%|█████████▎| 127/136 [00:22<00:01,  6.78it/s, epoch=6, grad_norm=128.670, loss=398.6862, lr=7.54e-05]
Epoch 6 TRAIN:  93%|█████████▎| 127/136 [00:22<00:01,  6.78it/s, epoch=6, grad_norm=234.678, loss=353.0685, lr=7.54e-05]
Epoch 6 TRAIN:  93%|█████████▎| 127/136 [00:22<00:01,  6.78it/s, epoch=6, grad_norm=132.155, loss=405.3026, lr=7.55e-05]
Epoch 6 TRAIN:  93%|█████████▎| 127/136 [00:22<00:01,  6.78it/s, epoch=6, grad_norm=118.793, loss=366.3846, lr=7.56e-05]
Epoch 6 TRAIN:  93%|█████████▎| 127/136 [00:22<00:01,  6.78it/s, epoch=6, grad_norm=145.978, loss=430.3080, lr=7.57e-05]
Epoch 6 TRAIN:  96%|█████████▋| 131/136 [00:22<00:00,  6.95it/s, epoch=6, grad_norm=145.978, loss=430.3080, lr=7.57e-05]
Epoch 6 TRAIN:  96%|█████████▋| 131/136 [00:22<00:00,  6.95it/s, epoch=6, grad_norm=94.969, loss=437.0344, lr=7.58e-05] 
Epoch 6 TRAIN:  96%|█████████▋| 131/136 [00:23<00:00,  6.95it/s, epoch=6, grad_norm=153.975, loss=417.8876, lr=7.58e-05]
Epoch 6 TRAIN:  96%|█████████▋| 131/136 [00:23<00:00,  6.95it/s, epoch=6, grad_norm=246.949, loss=394.3230, lr=7.59e-05]
Epoch 6 TRAIN:  96%|█████████▋| 131/136 [00:23<00:00,  6.95it/s, epoch=6, grad_norm=344.139, loss=343.9485, lr=7.60e-05]
Epoch 6 TRAIN:  99%|█████████▉| 135/136 [00:23<00:00,  6.96it/s, epoch=6, grad_norm=344.139, loss=343.9485, lr=7.60e-05]
Epoch 6 TRAIN:  99%|█████████▉| 135/136 [00:23<00:00,  6.96it/s, epoch=6, grad_norm=182.854, loss=391.4412, lr=7.61e-05]
Epoch 6 TRAIN:  99%|█████████▉| 135/136 [00:23<00:00,  6.96it/s, epoch=6, grad_norm=289.218, loss=323.6234, lr=7.62e-05]
Epoch 6 TRAIN:  99%|█████████▉| 135/136 [00:23<00:00,  6.96it/s, epoch=6, grad_norm=142.104, loss=409.5797, lr=7.62e-05]
Epoch 6 TRAIN: 138it [00:23,  5.80it/s, epoch=6, grad_norm=142.104, loss=409.5797, lr=7.62e-05]                         

CV:   0%|          | 0/6 [00:00<?, ?it/s]/home/rookie/Documents/batuka/projects/wenet/wenet/wenet/utils/train_utils.py:689: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  amp_autocast(enabled=dtype is not None,

CV:  17%|█▋        | 1/6 [00:00<00:03,  1.33it/s]
                                                 
2026-04-16 19:55:32,200 INFO Epoch 6 Step 953 CV info lr 7.6240e-05 cv_loss 392.3556657270952 rank 0 acc 0.23613625764846802
2026-04-16 19:55:32,202 INFO [Rank 0] Checkpoint: save to checkpoint exp/conformer_mongolian/epoch_6.pt
2026-04-16 19:55:32,314 INFO Epoch 7 Step 953 TRAIN info lr 7.6240e-05 rank 0
2026-04-16 19:55:32,332 INFO using accumulate grad, new batch size is 1 times larger than before

Epoch 7 TRAIN:   0%|          | 0/138 [00:00<?, ?it/s]
Epoch 7 TRAIN:   0%|          | 0/138 [00:03<?, ?it/s, epoch=7, grad_norm=380.947, loss=332.6306, lr=7.63e-05]
Epoch 7 TRAIN:   1%|          | 1/138 [00:03<08:57,  3.92s/it, epoch=7, grad_norm=380.947, loss=332.6306, lr=7.63e-05]
Epoch 7 TRAIN:   1%|          | 1/138 [00:04<08:57,  3.92s/it, epoch=7, grad_norm=292.497, loss=285.7220, lr=7.64e-05]
Epoch 7 TRAIN:   1%|          | 1/138 [00:04<08:57,  3.92s/it, epoch=7, grad_norm=451.021, loss=362.5055, lr=7.65e-05]
Epoch 7 TRAIN:   1%|          | 1/138 [00:04<08:57,  3.92s/it, epoch=7, grad_norm=369.207, loss=313.8119, lr=7.66e-05]
Epoch 7 TRAIN:   1%|          | 1/138 [00:04<08:57,  3.92s/it, epoch=7, grad_norm=616.336, loss=467.2911, lr=7.66e-05]
Epoch 7 TRAIN:   1%|          | 1/138 [00:04<08:57,  3.92s/it, epoch=7, grad_norm=476.303, loss=383.7285, lr=7.67e-05]
Epoch 7 TRAIN:   1%|          | 1/138 [00:04<08:57,  3.92s/it, epoch=7, grad_norm=360.681, loss=327.0081, lr=7.68e-05]
Epoch 7 TRAIN:   5%|▌         | 7/138 [00:04<01:04,  2.03it/s, epoch=7, grad_norm=360.681, loss=327.0081, lr=7.68e-05]
Epoch 7 TRAIN:   5%|▌         | 7/138 [00:04<01:04,  2.03it/s, epoch=7, grad_norm=372.599, loss=351.7071, lr=7.69e-05]
Epoch 7 TRAIN:   5%|▌         | 7/138 [00:04<01:04,  2.03it/s, epoch=7, grad_norm=304.249, loss=342.3080, lr=7.70e-05]
Epoch 7 TRAIN:   5%|▌         | 7/138 [00:04<01:04,  2.03it/s, epoch=7, grad_norm=261.110, loss=338.3641, lr=7.70e-05]
Epoch 7 TRAIN:   5%|▌         | 7/138 [00:04<01:04,  2.03it/s, epoch=7, grad_norm=210.895, loss=329.8181, lr=7.71e-05]
Epoch 7 TRAIN:   5%|▌         | 7/138 [00:05<01:04,  2.03it/s, epoch=7, grad_norm=151.393, loss=319.4862, lr=7.72e-05]
Epoch 7 TRAIN:   9%|▊         | 12/138 [00:05<00:38,  3.28it/s, epoch=7, grad_norm=151.393, loss=319.4862, lr=7.72e-05]
Epoch 7 TRAIN:   9%|▊         | 12/138 [00:05<00:38,  3.28it/s, epoch=7, grad_norm=118.547, loss=338.0882, lr=7.73e-05]
Epoch 7 TRAIN:   9%|▊         | 12/138 [00:05<00:38,  3.28it/s, epoch=7, grad_norm=115.444, loss=363.5727, lr=7.74e-05]
Epoch 7 TRAIN:   9%|▊         | 12/138 [00:05<00:38,  3.28it/s, epoch=7, grad_norm=70.320, loss=371.8746, lr=7.74e-05] 
Epoch 7 TRAIN:   9%|▊         | 12/138 [00:05<00:38,  3.28it/s, epoch=7, grad_norm=122.757, loss=349.0116, lr=7.75e-05]
Epoch 7 TRAIN:  12%|█▏        | 16/138 [00:05<00:30,  4.06it/s, epoch=7, grad_norm=122.757, loss=349.0116, lr=7.75e-05]
Epoch 7 TRAIN:  12%|█▏        | 16/138 [00:05<00:30,  4.06it/s, epoch=7, grad_norm=305.210, loss=311.2175, lr=7.76e-05]
Epoch 7 TRAIN:  12%|█▏        | 16/138 [00:06<00:30,  4.06it/s, epoch=7, grad_norm=121.991, loss=387.1953, lr=7.77e-05]
Epoch 7 TRAIN:  12%|█▏        | 16/138 [00:06<00:30,  4.06it/s, epoch=7, grad_norm=296.612, loss=322.7491, lr=7.78e-05]
Epoch 7 TRAIN:  12%|█▏        | 16/138 [00:06<00:30,  4.06it/s, epoch=7, grad_norm=372.469, loss=303.6906, lr=7.78e-05]
Epoch 7 TRAIN:  14%|█▍        | 20/138 [00:06<00:25,  4.70it/s, epoch=7, grad_norm=372.469, loss=303.6906, lr=7.78e-05]
Epoch 7 TRAIN:  14%|█▍        | 20/138 [00:06<00:25,  4.70it/s, epoch=7, grad_norm=201.043, loss=367.2902, lr=7.79e-05]
Epoch 7 TRAIN:  14%|█▍        | 20/138 [00:06<00:25,  4.70it/s, epoch=7, grad_norm=121.289, loss=384.5840, lr=7.80e-05]
Epoch 7 TRAIN:  14%|█▍        | 20/138 [00:06<00:25,  4.70it/s, epoch=7, grad_norm=112.101, loss=383.4729, lr=7.81e-05]
Epoch 7 TRAIN:  14%|█▍        | 20/138 [00:06<00:25,  4.70it/s, epoch=7, grad_norm=174.027, loss=351.3834, lr=7.82e-05]
Epoch 7 TRAIN:  17%|█▋        | 24/138 [00:06<00:21,  5.20it/s, epoch=7, grad_norm=174.027, loss=351.3834, lr=7.82e-05]
Epoch 7 TRAIN:  17%|█▋        | 24/138 [00:07<00:21,  5.20it/s, epoch=7, grad_norm=243.913, loss=302.9088, lr=7.82e-05]
Epoch 7 TRAIN:  17%|█▋        | 24/138 [00:07<00:21,  5.20it/s, epoch=7, grad_norm=149.625, loss=307.9456, lr=7.83e-05]
Epoch 7 TRAIN:  17%|█▋        | 24/138 [00:07<00:21,  5.20it/s, epoch=7, grad_norm=74.257, loss=318.1992, lr=7.84e-05] 
Epoch 7 TRAIN:  17%|█▋        | 24/138 [00:07<00:21,  5.20it/s, epoch=7, grad_norm=61.115, loss=311.6131, lr=7.85e-05]
Epoch 7 TRAIN:  20%|██        | 28/138 [00:07<00:19,  5.68it/s, epoch=7, grad_norm=61.115, loss=311.6131, lr=7.85e-05]
Epoch 7 TRAIN:  20%|██        | 28/138 [00:07<00:19,  5.68it/s, epoch=7, grad_norm=80.881, loss=348.7303, lr=7.86e-05]
Epoch 7 TRAIN:  20%|██        | 28/138 [00:07<00:19,  5.68it/s, epoch=7, grad_norm=218.784, loss=382.6257, lr=7.86e-05]
Epoch 7 TRAIN:  20%|██        | 28/138 [00:07<00:19,  5.68it/s, epoch=7, grad_norm=236.906, loss=380.7723, lr=7.87e-05]
Epoch 7 TRAIN:  20%|██        | 28/138 [00:08<00:19,  5.68it/s, epoch=7, grad_norm=208.071, loss=370.6176, lr=7.88e-05]
Epoch 7 TRAIN:  23%|██▎       | 32/138 [00:08<00:17,  5.96it/s, epoch=7, grad_norm=208.071, loss=370.6176, lr=7.88e-05]
Epoch 7 TRAIN:  23%|██▎       | 32/138 [00:08<00:17,  5.96it/s, epoch=7, grad_norm=200.284, loss=376.7778, lr=7.89e-05]
Epoch 7 TRAIN:  23%|██▎       | 32/138 [00:08<00:17,  5.96it/s, epoch=7, grad_norm=94.263, loss=318.7452, lr=7.90e-05] 
Epoch 7 TRAIN:  23%|██▎       | 32/138 [00:08<00:17,  5.96it/s, epoch=7, grad_norm=118.165, loss=343.3779, lr=7.90e-05]
Epoch 7 TRAIN:  23%|██▎       | 32/138 [00:08<00:17,  5.96it/s, epoch=7, grad_norm=232.746, loss=412.3015, lr=7.91e-05]
Epoch 7 TRAIN:  26%|██▌       | 36/138 [00:08<00:16,  6.20it/s, epoch=7, grad_norm=232.746, loss=412.3015, lr=7.91e-05]
Epoch 7 TRAIN:  26%|██▌       | 36/138 [00:08<00:16,  6.20it/s, epoch=7, grad_norm=84.776, loss=368.1811, lr=7.92e-05] 
Epoch 7 TRAIN:  26%|██▌       | 36/138 [00:08<00:16,  6.20it/s, epoch=7, grad_norm=60.266, loss=316.4524, lr=7.93e-05]
Epoch 7 TRAIN:  26%|██▌       | 36/138 [00:09<00:16,  6.20it/s, epoch=7, grad_norm=50.919, loss=363.8648, lr=7.94e-05]
Epoch 7 TRAIN:  26%|██▌       | 36/138 [00:09<00:16,  6.20it/s, epoch=7, grad_norm=62.607, loss=353.9507, lr=7.94e-05]
Epoch 7 TRAIN:  29%|██▉       | 40/138 [00:09<00:15,  6.37it/s, epoch=7, grad_norm=62.607, loss=353.9507, lr=7.94e-05]
Epoch 7 TRAIN:  29%|██▉       | 40/138 [00:09<00:15,  6.37it/s, epoch=7, grad_norm=99.893, loss=367.5670, lr=7.95e-05]
Epoch 7 TRAIN:  29%|██▉       | 40/138 [00:09<00:15,  6.37it/s, epoch=7, grad_norm=177.250, loss=305.9952, lr=7.96e-05]
Epoch 7 TRAIN:  29%|██▉       | 40/138 [00:09<00:15,  6.37it/s, epoch=7, grad_norm=209.462, loss=283.9214, lr=7.97e-05]
Epoch 7 TRAIN:  29%|██▉       | 40/138 [00:09<00:15,  6.37it/s, epoch=7, grad_norm=156.414, loss=310.1292, lr=7.98e-05]
Epoch 7 TRAIN:  32%|███▏      | 44/138 [00:09<00:14,  6.42it/s, epoch=7, grad_norm=156.414, loss=310.1292, lr=7.98e-05]
Epoch 7 TRAIN:  32%|███▏      | 44/138 [00:10<00:14,  6.42it/s, epoch=7, grad_norm=88.553, loss=349.2687, lr=7.98e-05] 
Epoch 7 TRAIN:  32%|███▏      | 44/138 [00:10<00:14,  6.42it/s, epoch=7, grad_norm=51.333, loss=347.2045, lr=7.99e-05]
Epoch 7 TRAIN:  32%|███▏      | 44/138 [00:10<00:14,  6.42it/s, epoch=7, grad_norm=114.286, loss=393.1896, lr=8.00e-05]
Epoch 7 TRAIN:  32%|███▏      | 44/138 [00:10<00:14,  6.42it/s, epoch=7, grad_norm=55.292, loss=356.7444, lr=8.01e-05] 
Epoch 7 TRAIN:  35%|███▍      | 48/138 [00:10<00:13,  6.44it/s, epoch=7, grad_norm=55.292, loss=356.7444, lr=8.01e-05]
Epoch 7 TRAIN:  35%|███▍      | 48/138 [00:10<00:13,  6.44it/s, epoch=7, grad_norm=65.679, loss=350.1127, lr=8.02e-05]
Epoch 7 TRAIN:  35%|███▍      | 48/138 [00:10<00:13,  6.44it/s, epoch=7, grad_norm=90.239, loss=365.3937, lr=8.02e-05]
Epoch 7 TRAIN:  35%|███▍      | 48/138 [00:10<00:13,  6.44it/s, epoch=7, grad_norm=122.802, loss=384.0950, lr=8.03e-05]
Epoch 7 TRAIN:  35%|███▍      | 48/138 [00:11<00:13,  6.44it/s, epoch=7, grad_norm=52.323, loss=324.2289, lr=8.04e-05] 
Epoch 7 TRAIN:  38%|███▊      | 52/138 [00:11<00:13,  6.44it/s, epoch=7, grad_norm=52.323, loss=324.2289, lr=8.04e-05]
Epoch 7 TRAIN:  38%|███▊      | 52/138 [00:11<00:13,  6.44it/s, epoch=7, grad_norm=52.203, loss=359.3065, lr=8.05e-05]
Epoch 7 TRAIN:  38%|███▊      | 52/138 [00:11<00:13,  6.44it/s, epoch=7, grad_norm=71.820, loss=319.9810, lr=8.06e-05]
Epoch 7 TRAIN:  38%|███▊      | 52/138 [00:11<00:13,  6.44it/s, epoch=7, grad_norm=55.000, loss=334.4702, lr=8.06e-05]
Epoch 7 TRAIN:  38%|███▊      | 52/138 [00:11<00:13,  6.44it/s, epoch=7, grad_norm=66.208, loss=363.7595, lr=8.07e-05]
Epoch 7 TRAIN:  41%|████      | 56/138 [00:11<00:12,  6.54it/s, epoch=7, grad_norm=66.208, loss=363.7595, lr=8.07e-05]
Epoch 7 TRAIN:  41%|████      | 56/138 [00:11<00:12,  6.54it/s, epoch=7, grad_norm=155.464, loss=296.1592, lr=8.08e-05]
Epoch 7 TRAIN:  41%|████      | 56/138 [00:12<00:12,  6.54it/s, epoch=7, grad_norm=89.408, loss=376.0165, lr=8.09e-05] 
Epoch 7 TRAIN:  41%|████      | 56/138 [00:12<00:12,  6.54it/s, epoch=7, grad_norm=57.681, loss=342.3156, lr=8.10e-05]
Epoch 7 TRAIN:  41%|████      | 56/138 [00:12<00:12,  6.54it/s, epoch=7, grad_norm=61.299, loss=359.6825, lr=8.10e-05]
Epoch 7 TRAIN:  43%|████▎     | 60/138 [00:12<00:11,  6.54it/s, epoch=7, grad_norm=61.299, loss=359.6825, lr=8.10e-05]
Epoch 7 TRAIN:  43%|████▎     | 60/138 [00:12<00:11,  6.54it/s, epoch=7, grad_norm=80.366, loss=363.8027, lr=8.11e-05]
Epoch 7 TRAIN:  43%|████▎     | 60/138 [00:12<00:11,  6.54it/s, epoch=7, grad_norm=77.515, loss=393.9217, lr=8.12e-05]
Epoch 7 TRAIN:  43%|████▎     | 60/138 [00:12<00:11,  6.54it/s, epoch=7, grad_norm=72.329, loss=320.9464, lr=8.13e-05]
Epoch 7 TRAIN:  43%|████▎     | 60/138 [00:12<00:11,  6.54it/s, epoch=7, grad_norm=52.378, loss=360.0256, lr=8.14e-05]
Epoch 7 TRAIN:  46%|████▋     | 64/138 [00:12<00:11,  6.53it/s, epoch=7, grad_norm=52.378, loss=360.0256, lr=8.14e-05]
Epoch 7 TRAIN:  46%|████▋     | 64/138 [00:13<00:11,  6.53it/s, epoch=7, grad_norm=64.163, loss=376.1538, lr=8.14e-05]
Epoch 7 TRAIN:  46%|████▋     | 64/138 [00:13<00:11,  6.53it/s, epoch=7, grad_norm=83.549, loss=372.9063, lr=8.15e-05]
Epoch 7 TRAIN:  46%|████▋     | 64/138 [00:13<00:11,  6.53it/s, epoch=7, grad_norm=121.952, loss=395.9506, lr=8.16e-05]
Epoch 7 TRAIN:  46%|████▋     | 64/138 [00:13<00:11,  6.53it/s, epoch=7, grad_norm=268.114, loss=231.5658, lr=8.17e-05]
Epoch 7 TRAIN:  49%|████▉     | 68/138 [00:13<00:10,  6.56it/s, epoch=7, grad_norm=268.114, loss=231.5658, lr=8.17e-05]
Epoch 7 TRAIN:  49%|████▉     | 68/138 [00:13<00:10,  6.56it/s, epoch=7, grad_norm=203.717, loss=295.1060, lr=8.18e-05]
Epoch 7 TRAIN:  49%|████▉     | 68/138 [00:13<00:10,  6.56it/s, epoch=7, grad_norm=68.563, loss=357.5015, lr=8.18e-05] 
Epoch 7 TRAIN:  49%|████▉     | 68/138 [00:13<00:10,  6.56it/s, epoch=7, grad_norm=92.179, loss=349.6104, lr=8.19e-05]
Epoch 7 TRAIN:  49%|████▉     | 68/138 [00:14<00:10,  6.56it/s, epoch=7, grad_norm=55.650, loss=340.4301, lr=8.20e-05]
Epoch 7 TRAIN:  52%|█████▏    | 72/138 [00:14<00:10,  6.54it/s, epoch=7, grad_norm=55.650, loss=340.4301, lr=8.20e-05]
Epoch 7 TRAIN:  52%|█████▏    | 72/138 [00:14<00:10,  6.54it/s, epoch=7, grad_norm=78.570, loss=385.5971, lr=8.21e-05]
Epoch 7 TRAIN:  52%|█████▏    | 72/138 [00:14<00:10,  6.54it/s, epoch=7, grad_norm=144.174, loss=392.5265, lr=8.22e-05]
Epoch 7 TRAIN:  52%|█████▏    | 72/138 [00:14<00:10,  6.54it/s, epoch=7, grad_norm=67.054, loss=327.6077, lr=8.22e-05] 
Epoch 7 TRAIN:  52%|█████▏    | 72/138 [00:14<00:10,  6.54it/s, epoch=7, grad_norm=91.976, loss=309.0241, lr=8.23e-05]
Epoch 7 TRAIN:  55%|█████▌    | 76/138 [00:14<00:09,  6.49it/s, epoch=7, grad_norm=91.976, loss=309.0241, lr=8.23e-05]
Epoch 7 TRAIN:  55%|█████▌    | 76/138 [00:14<00:09,  6.49it/s, epoch=7, grad_norm=113.012, loss=355.4532, lr=8.24e-05]
Epoch 7 TRAIN:  55%|█████▌    | 76/138 [00:15<00:09,  6.49it/s, epoch=7, grad_norm=170.010, loss=306.0487, lr=8.25e-05]
Epoch 7 TRAIN:  55%|█████▌    | 76/138 [00:15<00:09,  6.49it/s, epoch=7, grad_norm=49.219, loss=314.6786, lr=8.26e-05] 
Epoch 7 TRAIN:  55%|█████▌    | 76/138 [00:15<00:09,  6.49it/s, epoch=7, grad_norm=175.132, loss=265.4904, lr=8.26e-05]
Epoch 7 TRAIN:  58%|█████▊    | 80/138 [00:15<00:08,  6.49it/s, epoch=7, grad_norm=175.132, loss=265.4904, lr=8.26e-05]
Epoch 7 TRAIN:  58%|█████▊    | 80/138 [00:15<00:08,  6.49it/s, epoch=7, grad_norm=237.280, loss=254.9935, lr=8.27e-05]
Epoch 7 TRAIN:  58%|█████▊    | 80/138 [00:15<00:08,  6.49it/s, epoch=7, grad_norm=218.197, loss=393.1284, lr=8.28e-05]
Epoch 7 TRAIN:  58%|█████▊    | 80/138 [00:15<00:08,  6.49it/s, epoch=7, grad_norm=149.156, loss=367.0773, lr=8.29e-05]
Epoch 7 TRAIN:  58%|█████▊    | 80/138 [00:16<00:08,  6.49it/s, epoch=7, grad_norm=195.774, loss=201.5703, lr=8.30e-05]
Epoch 7 TRAIN:  61%|██████    | 84/138 [00:16<00:08,  6.46it/s, epoch=7, grad_norm=195.774, loss=201.5703, lr=8.30e-05]
Epoch 7 TRAIN:  61%|██████    | 84/138 [00:16<00:08,  6.46it/s, epoch=7, grad_norm=160.749, loss=244.8778, lr=8.30e-05]
Epoch 7 TRAIN:  61%|██████    | 84/138 [00:16<00:08,  6.46it/s, epoch=7, grad_norm=67.092, loss=339.0309, lr=8.31e-05] 
Epoch 7 TRAIN:  61%|██████    | 84/138 [00:16<00:08,  6.46it/s, epoch=7, grad_norm=180.311, loss=323.8862, lr=8.32e-05]
Epoch 7 TRAIN:  61%|██████    | 84/138 [00:16<00:08,  6.46it/s, epoch=7, grad_norm=69.720, loss=267.2415, lr=8.33e-05] 
Epoch 7 TRAIN:  64%|██████▍   | 88/138 [00:16<00:07,  6.39it/s, epoch=7, grad_norm=69.720, loss=267.2415, lr=8.33e-05]
Epoch 7 TRAIN:  64%|██████▍   | 88/138 [00:16<00:07,  6.39it/s, epoch=7, grad_norm=67.865, loss=308.7791, lr=8.34e-05]
Epoch 7 TRAIN:  64%|██████▍   | 88/138 [00:16<00:07,  6.39it/s, epoch=7, grad_norm=256.160, loss=380.6118, lr=8.34e-05]
Epoch 7 TRAIN:  64%|██████▍   | 88/138 [00:17<00:07,  6.39it/s, epoch=7, grad_norm=48.060, loss=248.0617, lr=8.35e-05] 
Epoch 7 TRAIN:  64%|██████▍   | 88/138 [00:17<00:07,  6.39it/s, epoch=7, grad_norm=198.340, loss=359.4174, lr=8.36e-05]
Epoch 7 TRAIN:  67%|██████▋   | 92/138 [00:17<00:07,  6.43it/s, epoch=7, grad_norm=198.340, loss=359.4174, lr=8.36e-05]
Epoch 7 TRAIN:  67%|██████▋   | 92/138 [00:17<00:07,  6.43it/s, epoch=7, grad_norm=88.389, loss=280.0375, lr=8.37e-05] 
Epoch 7 TRAIN:  67%|██████▋   | 92/138 [00:17<00:07,  6.43it/s, epoch=7, grad_norm=234.122, loss=407.8747, lr=8.38e-05]
Epoch 7 TRAIN:  67%|██████▋   | 92/138 [00:17<00:07,  6.43it/s, epoch=7, grad_norm=184.303, loss=336.7451, lr=8.38e-05]
Epoch 7 TRAIN:  67%|██████▋   | 92/138 [00:17<00:07,  6.43it/s, epoch=7, grad_norm=60.426, loss=307.3017, lr=8.39e-05] 
Epoch 7 TRAIN:  70%|██████▉   | 96/138 [00:17<00:06,  6.50it/s, epoch=7, grad_norm=60.426, loss=307.3017, lr=8.39e-05]
Epoch 7 TRAIN:  70%|██████▉   | 96/138 [00:18<00:06,  6.50it/s, epoch=7, grad_norm=151.219, loss=317.7263, lr=8.40e-05]
Epoch 7 TRAIN:  70%|██████▉   | 96/138 [00:18<00:06,  6.50it/s, epoch=7, grad_norm=62.447, loss=331.0852, lr=8.41e-05] 
Epoch 7 TRAIN:  70%|██████▉   | 96/138 [00:18<00:06,  6.50it/s, epoch=7, grad_norm=238.084, loss=404.9109, lr=8.42e-05]
Epoch 7 TRAIN:  70%|██████▉   | 96/138 [00:18<00:06,  6.50it/s, epoch=7, grad_norm=43.521, loss=327.5491, lr=8.42e-05] 2026-04-16 19:55:50,826 DEBUG TRAIN | steps/sec 5.511| Batch 7/100 loss 327.549133 loss_att 289.794800 loss_ctc 415.642639 th_accuracy 0.244165 lr 8.4240e-05 grad_norm 43.521378 rank 0

Epoch 7 TRAIN:  72%|███████▏  | 100/138 [00:18<00:05,  6.45it/s, epoch=7, grad_norm=43.521, loss=327.5491, lr=8.42e-05]
Epoch 7 TRAIN:  72%|███████▏  | 100/138 [00:18<00:05,  6.45it/s, epoch=7, grad_norm=113.546, loss=344.1096, lr=8.43e-05]
Epoch 7 TRAIN:  72%|███████▏  | 100/138 [00:18<00:05,  6.45it/s, epoch=7, grad_norm=435.719, loss=212.7807, lr=8.44e-05]
Epoch 7 TRAIN:  72%|███████▏  | 100/138 [00:18<00:05,  6.45it/s, epoch=7, grad_norm=146.378, loss=375.2901, lr=8.45e-05]
Epoch 7 TRAIN:  72%|███████▏  | 100/138 [00:19<00:05,  6.45it/s, epoch=7, grad_norm=73.250, loss=387.2548, lr=8.46e-05] 
Epoch 7 TRAIN:  75%|███████▌  | 104/138 [00:19<00:05,  6.33it/s, epoch=7, grad_norm=73.250, loss=387.2548, lr=8.46e-05]
Epoch 7 TRAIN:  75%|███████▌  | 104/138 [00:19<00:05,  6.33it/s, epoch=7, grad_norm=109.993, loss=374.7568, lr=8.46e-05]
Epoch 7 TRAIN:  75%|███████▌  | 104/138 [00:19<00:05,  6.33it/s, epoch=7, grad_norm=83.641, loss=394.4950, lr=8.47e-05] 
Epoch 7 TRAIN:  75%|███████▌  | 104/138 [00:19<00:05,  6.33it/s, epoch=7, grad_norm=118.424, loss=322.8338, lr=8.48e-05]
Epoch 7 TRAIN:  75%|███████▌  | 104/138 [00:19<00:05,  6.33it/s, epoch=7, grad_norm=153.740, loss=309.6033, lr=8.49e-05]
Epoch 7 TRAIN:  78%|███████▊  | 108/138 [00:19<00:04,  6.34it/s, epoch=7, grad_norm=153.740, loss=309.6033, lr=8.49e-05]
Epoch 7 TRAIN:  78%|███████▊  | 108/138 [00:19<00:04,  6.34it/s, epoch=7, grad_norm=142.935, loss=328.7501, lr=8.50e-05]
Epoch 7 TRAIN:  78%|███████▊  | 108/138 [00:20<00:04,  6.34it/s, epoch=7, grad_norm=109.679, loss=371.4357, lr=8.50e-05]
Epoch 7 TRAIN:  78%|███████▊  | 108/138 [00:20<00:04,  6.34it/s, epoch=7, grad_norm=193.994, loss=377.0475, lr=8.51e-05]
Epoch 7 TRAIN:  78%|███████▊  | 108/138 [00:20<00:04,  6.34it/s, epoch=7, grad_norm=87.908, loss=422.5624, lr=8.52e-05] 
Epoch 7 TRAIN:  81%|████████  | 112/138 [00:20<00:04,  6.45it/s, epoch=7, grad_norm=87.908, loss=422.5624, lr=8.52e-05]
Epoch 7 TRAIN:  81%|████████  | 112/138 [00:20<00:04,  6.45it/s, epoch=7, grad_norm=181.880, loss=443.3683, lr=8.53e-05]
Epoch 7 TRAIN:  81%|████████  | 112/138 [00:20<00:04,  6.45it/s, epoch=7, grad_norm=112.302, loss=308.2973, lr=8.54e-05]
Epoch 7 TRAIN:  81%|████████  | 112/138 [00:20<00:04,  6.45it/s, epoch=7, grad_norm=68.642, loss=321.7404, lr=8.54e-05] 
Epoch 7 TRAIN:  81%|████████  | 112/138 [00:21<00:04,  6.45it/s, epoch=7, grad_norm=133.877, loss=266.8690, lr=8.55e-05]
Epoch 7 TRAIN:  84%|████████▍ | 116/138 [00:21<00:03,  6.43it/s, epoch=7, grad_norm=133.877, loss=266.8690, lr=8.55e-05]
Epoch 7 TRAIN:  84%|████████▍ | 116/138 [00:21<00:03,  6.43it/s, epoch=7, grad_norm=152.789, loss=292.0050, lr=8.56e-05]
Epoch 7 TRAIN:  84%|████████▍ | 116/138 [00:21<00:03,  6.43it/s, epoch=7, grad_norm=196.391, loss=326.5041, lr=8.57e-05]
Epoch 7 TRAIN:  84%|████████▍ | 116/138 [00:21<00:03,  6.43it/s, epoch=7, grad_norm=72.165, loss=279.1064, lr=8.58e-05] 
Epoch 7 TRAIN:  84%|████████▍ | 116/138 [00:21<00:03,  6.43it/s, epoch=7, grad_norm=172.734, loss=431.4617, lr=8.58e-05]
Epoch 7 TRAIN:  87%|████████▋ | 120/138 [00:21<00:02,  6.54it/s, epoch=7, grad_norm=172.734, loss=431.4617, lr=8.58e-05]
Epoch 7 TRAIN:  87%|████████▋ | 120/138 [00:21<00:02,  6.54it/s, epoch=7, grad_norm=93.888, loss=373.3961, lr=8.59e-05] 
Epoch 7 TRAIN:  87%|████████▋ | 120/138 [00:21<00:02,  6.54it/s, epoch=7, grad_norm=73.674, loss=292.5744, lr=8.60e-05]
Epoch 7 TRAIN:  87%|████████▋ | 120/138 [00:22<00:02,  6.54it/s, epoch=7, grad_norm=74.231, loss=309.9228, lr=8.61e-05]
Epoch 7 TRAIN:  87%|████████▋ | 120/138 [00:22<00:02,  6.54it/s, epoch=7, grad_norm=210.663, loss=409.1859, lr=8.62e-05]
Epoch 7 TRAIN:  90%|████████▉ | 124/138 [00:22<00:02,  6.29it/s, epoch=7, grad_norm=210.663, loss=409.1859, lr=8.62e-05]
Epoch 7 TRAIN:  90%|████████▉ | 124/138 [00:22<00:02,  6.29it/s, epoch=7, grad_norm=81.470, loss=329.3629, lr=8.62e-05] 
Epoch 7 TRAIN:  90%|████████▉ | 124/138 [00:22<00:02,  6.29it/s, epoch=7, grad_norm=85.339, loss=366.1299, lr=8.63e-05]
Epoch 7 TRAIN:  90%|████████▉ | 124/138 [00:22<00:02,  6.29it/s, epoch=7, grad_norm=309.963, loss=420.6306, lr=8.64e-05]
Epoch 7 TRAIN:  90%|████████▉ | 124/138 [00:22<00:02,  6.29it/s, epoch=7, grad_norm=153.251, loss=320.7295, lr=8.65e-05]
Epoch 7 TRAIN:  93%|█████████▎| 128/138 [00:22<00:01,  6.51it/s, epoch=7, grad_norm=153.251, loss=320.7295, lr=8.65e-05]
Epoch 7 TRAIN:  93%|█████████▎| 128/138 [00:22<00:01,  6.51it/s, epoch=7, grad_norm=146.567, loss=401.2959, lr=8.66e-05]
Epoch 7 TRAIN:  93%|█████████▎| 128/138 [00:23<00:01,  6.51it/s, epoch=7, grad_norm=82.711, loss=391.2878, lr=8.66e-05] 
Epoch 7 TRAIN:  93%|█████████▎| 128/138 [00:23<00:01,  6.51it/s, epoch=7, grad_norm=91.360, loss=368.0807, lr=8.67e-05]
Epoch 7 TRAIN:  93%|█████████▎| 128/138 [00:23<00:01,  6.51it/s, epoch=7, grad_norm=111.806, loss=332.5944, lr=8.68e-05]
Epoch 7 TRAIN:  96%|█████████▌| 132/138 [00:23<00:00,  6.69it/s, epoch=7, grad_norm=111.806, loss=332.5944, lr=8.68e-05]
Epoch 7 TRAIN:  96%|█████████▌| 132/138 [00:23<00:00,  6.69it/s, epoch=7, grad_norm=170.744, loss=218.3713, lr=8.69e-05]
Epoch 7 TRAIN:  96%|█████████▌| 132/138 [00:23<00:00,  6.69it/s, epoch=7, grad_norm=137.297, loss=440.8066, lr=8.70e-05]
Epoch 7 TRAIN:  96%|█████████▌| 132/138 [00:23<00:00,  6.69it/s, epoch=7, grad_norm=162.803, loss=329.7058, lr=8.70e-05]
Epoch 7 TRAIN:  96%|█████████▌| 132/138 [00:23<00:00,  6.69it/s, epoch=7, grad_norm=125.152, loss=350.6481, lr=8.71e-05]
Epoch 7 TRAIN:  99%|█████████▊| 136/138 [00:23<00:00,  7.00it/s, epoch=7, grad_norm=125.152, loss=350.6481, lr=8.71e-05]
Epoch 7 TRAIN:  99%|█████████▊| 136/138 [00:24<00:00,  7.00it/s, epoch=7, grad_norm=83.950, loss=330.0974, lr=8.72e-05] 
Epoch 7 TRAIN:  99%|█████████▊| 136/138 [00:24<00:00,  7.00it/s, epoch=7, grad_norm=140.307, loss=417.5545, lr=8.73e-05]
Epoch 7 TRAIN: 100%|██████████| 138/138 [00:24<00:00,  5.70it/s, epoch=7, grad_norm=140.307, loss=417.5545, lr=8.73e-05]

CV:   0%|          | 0/6 [00:00<?, ?it/s]/home/rookie/Documents/batuka/projects/wenet/wenet/wenet/utils/train_utils.py:689: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  amp_autocast(enabled=dtype is not None,

CV:  17%|█▋        | 1/6 [00:00<00:04,  1.14it/s]
                                                 
2026-04-16 19:55:57,523 INFO Epoch 7 Step 1091 CV info lr 8.7280e-05 cv_loss 399.21309037642044 rank 0 acc 0.235211414595445
2026-04-16 19:55:57,524 INFO [Rank 0] Checkpoint: save to checkpoint exp/conformer_mongolian/epoch_7.pt
2026-04-16 19:55:57,635 INFO Epoch 8 Step 1091 TRAIN info lr 8.7280e-05 rank 0
2026-04-16 19:55:57,650 INFO using accumulate grad, new batch size is 1 times larger than before

Epoch 8 TRAIN:   0%|          | 0/138 [00:00<?, ?it/s]
Epoch 8 TRAIN:   0%|          | 0/138 [00:04<?, ?it/s, epoch=8, grad_norm=458.610, loss=371.1638, lr=8.74e-05]
Epoch 8 TRAIN:   1%|          | 1/138 [00:04<09:30,  4.16s/it, epoch=8, grad_norm=458.610, loss=371.1638, lr=8.74e-05]
Epoch 8 TRAIN:   1%|          | 1/138 [00:04<09:30,  4.16s/it, epoch=8, grad_norm=194.955, loss=225.7021, lr=8.74e-05]
Epoch 8 TRAIN:   1%|          | 1/138 [00:04<09:30,  4.16s/it, epoch=8, grad_norm=530.888, loss=416.7795, lr=8.75e-05]
Epoch 8 TRAIN:   1%|          | 1/138 [00:04<09:30,  4.16s/it, epoch=8, grad_norm=455.137, loss=391.7627, lr=8.76e-05]
Epoch 8 TRAIN:   1%|          | 1/138 [00:04<09:30,  4.16s/it, epoch=8, grad_norm=391.170, loss=344.4148, lr=8.77e-05]
Epoch 8 TRAIN:   1%|          | 1/138 [00:04<09:30,  4.16s/it, epoch=8, grad_norm=496.491, loss=430.7976, lr=8.78e-05]
Epoch 8 TRAIN:   1%|          | 1/138 [00:04<09:30,  4.16s/it, epoch=8, grad_norm=343.329, loss=363.4424, lr=8.78e-05]
Epoch 8 TRAIN:   1%|          | 1/138 [00:04<09:30,  4.16s/it, epoch=8, grad_norm=354.102, loss=378.1650, lr=8.79e-05]
Epoch 8 TRAIN:   6%|▌         | 8/138 [00:04<00:57,  2.25it/s, epoch=8, grad_norm=354.102, loss=378.1650, lr=8.79e-05]
Epoch 8 TRAIN:   6%|▌         | 8/138 [00:04<00:57,  2.25it/s, epoch=8, grad_norm=237.385, loss=362.2548, lr=8.80e-05]
Epoch 8 TRAIN:   6%|▌         | 8/138 [00:04<00:57,  2.25it/s, epoch=8, grad_norm=221.265, loss=366.9266, lr=8.81e-05]
Epoch 8 TRAIN:   6%|▌         | 8/138 [00:05<00:57,  2.25it/s, epoch=8, grad_norm=164.546, loss=368.4520, lr=8.82e-05]
Epoch 8 TRAIN:   6%|▌         | 8/138 [00:05<00:57,  2.25it/s, epoch=8, grad_norm=57.498, loss=312.0276, lr=8.82e-05] 
Epoch 8 TRAIN:   6%|▌         | 8/138 [00:05<00:57,  2.25it/s, epoch=8, grad_norm=194.644, loss=258.1183, lr=8.83e-05]
Epoch 8 TRAIN:   9%|▉         | 13/138 [00:05<00:37,  3.32it/s, epoch=8, grad_norm=194.644, loss=258.1183, lr=8.83e-05]
Epoch 8 TRAIN:   9%|▉         | 13/138 [00:05<00:37,  3.32it/s, epoch=8, grad_norm=70.079, loss=370.5463, lr=8.84e-05] 
Epoch 8 TRAIN:   9%|▉         | 13/138 [00:05<00:37,  3.32it/s, epoch=8, grad_norm=126.943, loss=343.2739, lr=8.85e-05]
Epoch 8 TRAIN:   9%|▉         | 13/138 [00:05<00:37,  3.32it/s, epoch=8, grad_norm=59.916, loss=376.4500, lr=8.86e-05] 
Epoch 8 TRAIN:   9%|▉         | 13/138 [00:06<00:37,  3.32it/s, epoch=8, grad_norm=320.768, loss=285.3755, lr=8.86e-05]
Epoch 8 TRAIN:  12%|█▏        | 17/138 [00:06<00:30,  3.99it/s, epoch=8, grad_norm=320.768, loss=285.3755, lr=8.86e-05]
Epoch 8 TRAIN:  12%|█▏        | 17/138 [00:06<00:30,  3.99it/s, epoch=8, grad_norm=245.230, loss=304.2647, lr=8.87e-05]
Epoch 8 TRAIN:  12%|█▏        | 17/138 [00:06<00:30,  3.99it/s, epoch=8, grad_norm=148.429, loss=340.0277, lr=8.88e-05]
Epoch 8 TRAIN:  12%|█▏        | 17/138 [00:06<00:30,  3.99it/s, epoch=8, grad_norm=98.254, loss=349.9724, lr=8.89e-05] 
Epoch 8 TRAIN:  12%|█▏        | 17/138 [00:06<00:30,  3.99it/s, epoch=8, grad_norm=44.643, loss=368.6773, lr=8.90e-05]
Epoch 8 TRAIN:  15%|█▌        | 21/138 [00:06<00:26,  4.44it/s, epoch=8, grad_norm=44.643, loss=368.6773, lr=8.90e-05]
Epoch 8 TRAIN:  15%|█▌        | 21/138 [00:06<00:26,  4.44it/s, epoch=8, grad_norm=167.082, loss=294.0198, lr=8.90e-05]
Epoch 8 TRAIN:  15%|█▌        | 21/138 [00:07<00:26,  4.44it/s, epoch=8, grad_norm=52.809, loss=331.0963, lr=8.91e-05] 
Epoch 8 TRAIN:  15%|█▌        | 21/138 [00:07<00:26,  4.44it/s, epoch=8, grad_norm=224.624, loss=432.1397, lr=8.92e-05]
Epoch 8 TRAIN:  15%|█▌        | 21/138 [00:07<00:26,  4.44it/s, epoch=8, grad_norm=103.247, loss=391.1508, lr=8.93e-05]
Epoch 8 TRAIN:  18%|█▊        | 25/138 [00:07<00:23,  4.83it/s, epoch=8, grad_norm=103.247, loss=391.1508, lr=8.93e-05]
Epoch 8 TRAIN:  18%|█▊        | 25/138 [00:07<00:23,  4.83it/s, epoch=8, grad_norm=55.779, loss=332.2644, lr=8.94e-05] 
Epoch 8 TRAIN:  18%|█▊        | 25/138 [00:07<00:23,  4.83it/s, epoch=8, grad_norm=88.562, loss=353.0345, lr=8.94e-05]
Epoch 8 TRAIN:  18%|█▊        | 25/138 [00:07<00:23,  4.83it/s, epoch=8, grad_norm=51.290, loss=334.0134, lr=8.95e-05]
Epoch 8 TRAIN:  18%|█▊        | 25/138 [00:08<00:23,  4.83it/s, epoch=8, grad_norm=86.937, loss=353.8264, lr=8.96e-05]
Epoch 8 TRAIN:  21%|██        | 29/138 [00:08<00:21,  5.18it/s, epoch=8, grad_norm=86.937, loss=353.8264, lr=8.96e-05]
Epoch 8 TRAIN:  21%|██        | 29/138 [00:08<00:21,  5.18it/s, epoch=8, grad_norm=73.202, loss=318.7802, lr=8.97e-05]
Epoch 8 TRAIN:  21%|██        | 29/138 [00:08<00:21,  5.18it/s, epoch=8, grad_norm=94.438, loss=378.6350, lr=8.98e-05]
Epoch 8 TRAIN:  21%|██        | 29/138 [00:08<00:21,  5.18it/s, epoch=8, grad_norm=110.386, loss=378.1446, lr=8.98e-05]
Epoch 8 TRAIN:  21%|██        | 29/138 [00:08<00:21,  5.18it/s, epoch=8, grad_norm=102.944, loss=339.5662, lr=8.99e-05]
Epoch 8 TRAIN:  24%|██▍       | 33/138 [00:08<00:19,  5.40it/s, epoch=8, grad_norm=102.944, loss=339.5662, lr=8.99e-05]
Epoch 8 TRAIN:  24%|██▍       | 33/138 [00:08<00:19,  5.40it/s, epoch=8, grad_norm=133.048, loss=327.8886, lr=9.00e-05]
Epoch 8 TRAIN:  24%|██▍       | 33/138 [00:09<00:19,  5.40it/s, epoch=8, grad_norm=63.085, loss=347.8516, lr=9.01e-05] 
Epoch 8 TRAIN:  24%|██▍       | 33/138 [00:09<00:19,  5.40it/s, epoch=8, grad_norm=54.102, loss=372.8104, lr=9.02e-05]
Epoch 8 TRAIN:  24%|██▍       | 33/138 [00:09<00:19,  5.40it/s, epoch=8, grad_norm=54.991, loss=349.3666, lr=9.02e-05]
Epoch 8 TRAIN:  27%|██▋       | 37/138 [00:09<00:18,  5.57it/s, epoch=8, grad_norm=54.991, loss=349.3666, lr=9.02e-05]
Epoch 8 TRAIN:  27%|██▋       | 37/138 [00:09<00:18,  5.57it/s, epoch=8, grad_norm=97.280, loss=333.3260, lr=9.03e-05]
Epoch 8 TRAIN:  27%|██▋       | 37/138 [00:09<00:18,  5.57it/s, epoch=8, grad_norm=84.257, loss=317.5280, lr=9.04e-05]
Epoch 8 TRAIN:  27%|██▋       | 37/138 [00:09<00:18,  5.57it/s, epoch=8, grad_norm=99.943, loss=369.8401, lr=9.05e-05]
Epoch 8 TRAIN:  27%|██▋       | 37/138 [00:10<00:18,  5.57it/s, epoch=8, grad_norm=46.367, loss=363.7052, lr=9.06e-05]
Epoch 8 TRAIN:  30%|██▉       | 41/138 [00:10<00:17,  5.57it/s, epoch=8, grad_norm=46.367, loss=363.7052, lr=9.06e-05]
Epoch 8 TRAIN:  30%|██▉       | 41/138 [00:10<00:17,  5.57it/s, epoch=8, grad_norm=122.523, loss=299.1078, lr=9.06e-05]
Epoch 8 TRAIN:  30%|██▉       | 41/138 [00:10<00:17,  5.57it/s, epoch=8, grad_norm=123.293, loss=379.2332, lr=9.07e-05]
Epoch 8 TRAIN:  30%|██▉       | 41/138 [00:10<00:17,  5.57it/s, epoch=8, grad_norm=91.210, loss=306.8260, lr=9.08e-05] 
Epoch 8 TRAIN:  32%|███▏      | 44/138 [00:10<00:16,  5.63it/s, epoch=8, grad_norm=91.210, loss=306.8260, lr=9.08e-05]
Epoch 8 TRAIN:  32%|███▏      | 44/138 [00:10<00:16,  5.63it/s, epoch=8, grad_norm=98.328, loss=362.8988, lr=9.09e-05]
Epoch 8 TRAIN:  32%|███▏      | 44/138 [00:10<00:16,  5.63it/s, epoch=8, grad_norm=71.493, loss=349.1332, lr=9.10e-05]
Epoch 8 TRAIN:  32%|███▏      | 44/138 [00:11<00:16,  5.63it/s, epoch=8, grad_norm=91.049, loss=285.1376, lr=9.10e-05]
Epoch 8 TRAIN:  34%|███▍      | 47/138 [00:11<00:16,  5.67it/s, epoch=8, grad_norm=91.049, loss=285.1376, lr=9.10e-05]
Epoch 8 TRAIN:  34%|███▍      | 47/138 [00:11<00:16,  5.67it/s, epoch=8, grad_norm=51.127, loss=291.7734, lr=9.11e-05]
Epoch 8 TRAIN:  34%|███▍      | 47/138 [00:11<00:16,  5.67it/s, epoch=8, grad_norm=51.643, loss=320.8227, lr=9.12e-05]
Epoch 8 TRAIN:  34%|███▍      | 47/138 [00:11<00:16,  5.67it/s, epoch=8, grad_norm=58.006, loss=327.6798, lr=9.13e-05]
Epoch 8 TRAIN:  36%|███▌      | 50/138 [00:11<00:15,  5.66it/s, epoch=8, grad_norm=58.006, loss=327.6798, lr=9.13e-05]
Epoch 8 TRAIN:  36%|███▌      | 50/138 [00:11<00:15,  5.66it/s, epoch=8, grad_norm=50.484, loss=308.0695, lr=9.14e-05]
Epoch 8 TRAIN:  36%|███▌      | 50/138 [00:12<00:15,  5.66it/s, epoch=8, grad_norm=115.222, loss=372.1749, lr=9.14e-05]
Epoch 8 TRAIN:  36%|███▌      | 50/138 [00:12<00:15,  5.66it/s, epoch=8, grad_norm=98.029, loss=360.9832, lr=9.15e-05] 
Epoch 8 TRAIN:  38%|███▊      | 53/138 [00:12<00:14,  5.73it/s, epoch=8, grad_norm=98.029, loss=360.9832, lr=9.15e-05]
Epoch 8 TRAIN:  38%|███▊      | 53/138 [00:12<00:14,  5.73it/s, epoch=8, grad_norm=59.649, loss=340.1610, lr=9.16e-05]
Epoch 8 TRAIN:  38%|███▊      | 53/138 [00:12<00:14,  5.73it/s, epoch=8, grad_norm=62.406, loss=295.8732, lr=9.17e-05]
Epoch 8 TRAIN:  38%|███▊      | 53/138 [00:12<00:14,  5.73it/s, epoch=8, grad_norm=136.077, loss=362.1271, lr=9.18e-05]
Epoch 8 TRAIN:  41%|████      | 56/138 [00:12<00:14,  5.75it/s, epoch=8, grad_norm=136.077, loss=362.1271, lr=9.18e-05]
Epoch 8 TRAIN:  41%|████      | 56/138 [00:12<00:14,  5.75it/s, epoch=8, grad_norm=60.252, loss=365.4404, lr=9.18e-05] 
Epoch 8 TRAIN:  41%|████      | 56/138 [00:13<00:14,  5.75it/s, epoch=8, grad_norm=52.955, loss=356.0886, lr=9.19e-05]
Epoch 8 TRAIN:  41%|████      | 56/138 [00:13<00:14,  5.75it/s, epoch=8, grad_norm=58.926, loss=346.1667, lr=9.20e-05]
Epoch 8 TRAIN:  41%|████      | 56/138 [00:13<00:14,  5.75it/s, epoch=8, grad_norm=111.319, loss=329.2034, lr=9.21e-05]
Epoch 8 TRAIN:  43%|████▎     | 60/138 [00:13<00:13,  5.89it/s, epoch=8, grad_norm=111.319, loss=329.2034, lr=9.21e-05]
Epoch 8 TRAIN:  43%|████▎     | 60/138 [00:13<00:13,  5.89it/s, epoch=8, grad_norm=74.762, loss=352.3689, lr=9.22e-05] 
Epoch 8 TRAIN:  43%|████▎     | 60/138 [00:13<00:13,  5.89it/s, epoch=8, grad_norm=89.206, loss=376.1642, lr=9.22e-05]
Epoch 8 TRAIN:  43%|████▎     | 60/138 [00:13<00:13,  5.89it/s, epoch=8, grad_norm=50.672, loss=342.7768, lr=9.23e-05]
Epoch 8 TRAIN:  43%|████▎     | 60/138 [00:13<00:13,  5.89it/s, epoch=8, grad_norm=70.523, loss=346.7531, lr=9.24e-05]
Epoch 8 TRAIN:  46%|████▋     | 64/138 [00:13<00:12,  6.04it/s, epoch=8, grad_norm=70.523, loss=346.7531, lr=9.24e-05]
Epoch 8 TRAIN:  46%|████▋     | 64/138 [00:14<00:12,  6.04it/s, epoch=8, grad_norm=59.203, loss=399.7379, lr=9.25e-05]
Epoch 8 TRAIN:  46%|████▋     | 64/138 [00:14<00:12,  6.04it/s, epoch=8, grad_norm=115.221, loss=320.0082, lr=9.26e-05]
Epoch 8 TRAIN:  46%|████▋     | 64/138 [00:14<00:12,  6.04it/s, epoch=8, grad_norm=96.892, loss=315.7202, lr=9.26e-05] 
Epoch 8 TRAIN:  46%|████▋     | 64/138 [00:14<00:12,  6.04it/s, epoch=8, grad_norm=160.995, loss=279.2516, lr=9.27e-05]
Epoch 8 TRAIN:  49%|████▉     | 68/138 [00:14<00:11,  6.20it/s, epoch=8, grad_norm=160.995, loss=279.2516, lr=9.27e-05]
Epoch 8 TRAIN:  49%|████▉     | 68/138 [00:14<00:11,  6.20it/s, epoch=8, grad_norm=91.149, loss=360.9438, lr=9.28e-05] 
Epoch 8 TRAIN:  49%|████▉     | 68/138 [00:14<00:11,  6.20it/s, epoch=8, grad_norm=126.132, loss=290.3289, lr=9.29e-05]
Epoch 8 TRAIN:  49%|████▉     | 68/138 [00:15<00:11,  6.20it/s, epoch=8, grad_norm=191.117, loss=372.9212, lr=9.30e-05]
Epoch 8 TRAIN:  49%|████▉     | 68/138 [00:15<00:11,  6.20it/s, epoch=8, grad_norm=122.497, loss=316.2491, lr=9.30e-05]
Epoch 8 TRAIN:  52%|█████▏    | 72/138 [00:15<00:10,  6.30it/s, epoch=8, grad_norm=122.497, loss=316.2491, lr=9.30e-05]
Epoch 8 TRAIN:  52%|█████▏    | 72/138 [00:15<00:10,  6.30it/s, epoch=8, grad_norm=116.861, loss=380.4040, lr=9.31e-05]
Epoch 8 TRAIN:  52%|█████▏    | 72/138 [00:15<00:10,  6.30it/s, epoch=8, grad_norm=128.171, loss=366.9645, lr=9.32e-05]
Epoch 8 TRAIN:  52%|█████▏    | 72/138 [00:15<00:10,  6.30it/s, epoch=8, grad_norm=110.020, loss=352.1904, lr=9.33e-05]
Epoch 8 TRAIN:  52%|█████▏    | 72/138 [00:15<00:10,  6.30it/s, epoch=8, grad_norm=213.277, loss=435.9394, lr=9.34e-05]
Epoch 8 TRAIN:  55%|█████▌    | 76/138 [00:15<00:09,  6.26it/s, epoch=8, grad_norm=213.277, loss=435.9394, lr=9.34e-05]
Epoch 8 TRAIN:  55%|█████▌    | 76/138 [00:16<00:09,  6.26it/s, epoch=8, grad_norm=120.487, loss=323.4132, lr=9.34e-05]
Epoch 8 TRAIN:  55%|█████▌    | 76/138 [00:16<00:09,  6.26it/s, epoch=8, grad_norm=319.200, loss=273.6115, lr=9.35e-05]
Epoch 8 TRAIN:  55%|█████▌    | 76/138 [00:16<00:09,  6.26it/s, epoch=8, grad_norm=71.018, loss=328.4523, lr=9.36e-05] 
Epoch 8 TRAIN:  55%|█████▌    | 76/138 [00:16<00:09,  6.26it/s, epoch=8, grad_norm=55.757, loss=332.0094, lr=9.37e-05]
Epoch 8 TRAIN:  58%|█████▊    | 80/138 [00:16<00:09,  6.20it/s, epoch=8, grad_norm=55.757, loss=332.0094, lr=9.37e-05]
Epoch 8 TRAIN:  58%|█████▊    | 80/138 [00:16<00:09,  6.20it/s, epoch=8, grad_norm=278.603, loss=297.6898, lr=9.38e-05]
Epoch 8 TRAIN:  58%|█████▊    | 80/138 [00:16<00:09,  6.20it/s, epoch=8, grad_norm=110.541, loss=322.8976, lr=9.38e-05]
Epoch 8 TRAIN:  58%|█████▊    | 80/138 [00:17<00:09,  6.20it/s, epoch=8, grad_norm=74.704, loss=343.9567, lr=9.39e-05] 
Epoch 8 TRAIN:  58%|█████▊    | 80/138 [00:17<00:09,  6.20it/s, epoch=8, grad_norm=65.354, loss=362.3930, lr=9.40e-05]
Epoch 8 TRAIN:  61%|██████    | 84/138 [00:17<00:08,  6.23it/s, epoch=8, grad_norm=65.354, loss=362.3930, lr=9.40e-05]
Epoch 8 TRAIN:  61%|██████    | 84/138 [00:17<00:08,  6.23it/s, epoch=8, grad_norm=133.232, loss=322.1359, lr=9.41e-05]
Epoch 8 TRAIN:  61%|██████    | 84/138 [00:17<00:08,  6.23it/s, epoch=8, grad_norm=292.637, loss=255.6177, lr=9.42e-05]
Epoch 8 TRAIN:  61%|██████    | 84/138 [00:17<00:08,  6.23it/s, epoch=8, grad_norm=75.627, loss=338.8617, lr=9.42e-05] 
Epoch 8 TRAIN:  61%|██████    | 84/138 [00:17<00:08,  6.23it/s, epoch=8, grad_norm=206.530, loss=372.1863, lr=9.43e-05]
Epoch 8 TRAIN:  64%|██████▍   | 88/138 [00:17<00:08,  6.21it/s, epoch=8, grad_norm=206.530, loss=372.1863, lr=9.43e-05]
Epoch 8 TRAIN:  64%|██████▍   | 88/138 [00:17<00:08,  6.21it/s, epoch=8, grad_norm=277.297, loss=268.9724, lr=9.44e-05]
Epoch 8 TRAIN:  64%|██████▍   | 88/138 [00:18<00:08,  6.21it/s, epoch=8, grad_norm=126.100, loss=278.0951, lr=9.45e-05]
Epoch 8 TRAIN:  64%|██████▍   | 88/138 [00:18<00:08,  6.21it/s, epoch=8, grad_norm=181.259, loss=382.7689, lr=9.46e-05]
Epoch 8 TRAIN:  64%|██████▍   | 88/138 [00:18<00:08,  6.21it/s, epoch=8, grad_norm=110.403, loss=346.7461, lr=9.46e-05]
Epoch 8 TRAIN:  67%|██████▋   | 92/138 [00:18<00:07,  6.18it/s, epoch=8, grad_norm=110.403, loss=346.7461, lr=9.46e-05]
Epoch 8 TRAIN:  67%|██████▋   | 92/138 [00:18<00:07,  6.18it/s, epoch=8, grad_norm=110.948, loss=314.0043, lr=9.47e-05]
Epoch 8 TRAIN:  67%|██████▋   | 92/138 [00:18<00:07,  6.18it/s, epoch=8, grad_norm=128.072, loss=303.2477, lr=9.48e-05]
Epoch 8 TRAIN:  67%|██████▋   | 92/138 [00:18<00:07,  6.18it/s, epoch=8, grad_norm=98.687, loss=360.7664, lr=9.49e-05] 
Epoch 8 TRAIN:  67%|██████▋   | 92/138 [00:19<00:07,  6.18it/s, epoch=8, grad_norm=247.999, loss=359.9509, lr=9.50e-05]
Epoch 8 TRAIN:  70%|██████▉   | 96/138 [00:19<00:06,  6.31it/s, epoch=8, grad_norm=247.999, loss=359.9509, lr=9.50e-05]
Epoch 8 TRAIN:  70%|██████▉   | 96/138 [00:19<00:06,  6.31it/s, epoch=8, grad_norm=85.229, loss=325.3998, lr=9.50e-05] 
Epoch 8 TRAIN:  70%|██████▉   | 96/138 [00:19<00:06,  6.31it/s, epoch=8, grad_norm=164.085, loss=366.6022, lr=9.51e-05]
Epoch 8 TRAIN:  70%|██████▉   | 96/138 [00:19<00:06,  6.31it/s, epoch=8, grad_norm=71.644, loss=269.3531, lr=9.52e-05] 
Epoch 8 TRAIN:  70%|██████▉   | 96/138 [00:19<00:06,  6.31it/s, epoch=8, grad_norm=90.035, loss=274.6969, lr=9.53e-05]2026-04-16 19:56:17,405 DEBUG TRAIN | steps/sec 5.192| Batch 8/100 loss 274.696899 loss_att 237.957001 loss_ctc 360.423340 th_accuracy 0.269639 lr 9.5280e-05 grad_norm 90.034538 rank 0

Epoch 8 TRAIN:  72%|███████▏  | 100/138 [00:19<00:06,  6.14it/s, epoch=8, grad_norm=90.035, loss=274.6969, lr=9.53e-05]
Epoch 8 TRAIN:  72%|███████▏  | 100/138 [00:19<00:06,  6.14it/s, epoch=8, grad_norm=239.034, loss=247.7794, lr=9.54e-05]
Epoch 8 TRAIN:  72%|███████▏  | 100/138 [00:20<00:06,  6.14it/s, epoch=8, grad_norm=147.522, loss=297.8229, lr=9.54e-05]
Epoch 8 TRAIN:  72%|███████▏  | 100/138 [00:20<00:06,  6.14it/s, epoch=8, grad_norm=120.106, loss=368.0958, lr=9.55e-05]
Epoch 8 TRAIN:  72%|███████▏  | 100/138 [00:20<00:06,  6.14it/s, epoch=8, grad_norm=242.915, loss=349.1013, lr=9.56e-05]
Epoch 8 TRAIN:  75%|███████▌  | 104/138 [00:20<00:05,  6.10it/s, epoch=8, grad_norm=242.915, loss=349.1013, lr=9.56e-05]
Epoch 8 TRAIN:  75%|███████▌  | 104/138 [00:20<00:05,  6.10it/s, epoch=8, grad_norm=108.816, loss=380.0330, lr=9.57e-05]
Epoch 8 TRAIN:  75%|███████▌  | 104/138 [00:20<00:05,  6.10it/s, epoch=8, grad_norm=78.370, loss=308.4368, lr=9.58e-05] 
Epoch 8 TRAIN:  75%|███████▌  | 104/138 [00:20<00:05,  6.10it/s, epoch=8, grad_norm=239.734, loss=408.5520, lr=9.58e-05]
Epoch 8 TRAIN:  75%|███████▌  | 104/138 [00:21<00:05,  6.10it/s, epoch=8, grad_norm=70.569, loss=322.4857, lr=9.59e-05] 
Epoch 8 TRAIN:  78%|███████▊  | 108/138 [00:21<00:04,  6.15it/s, epoch=8, grad_norm=70.569, loss=322.4857, lr=9.59e-05]
Epoch 8 TRAIN:  78%|███████▊  | 108/138 [00:21<00:04,  6.15it/s, epoch=8, grad_norm=128.683, loss=337.8041, lr=9.60e-05]
Epoch 8 TRAIN:  78%|███████▊  | 108/138 [00:21<00:04,  6.15it/s, epoch=8, grad_norm=114.946, loss=346.7867, lr=9.61e-05]
Epoch 8 TRAIN:  78%|███████▊  | 108/138 [00:21<00:04,  6.15it/s, epoch=8, grad_norm=104.086, loss=326.4097, lr=9.62e-05]
Epoch 8 TRAIN:  78%|███████▊  | 108/138 [00:21<00:04,  6.15it/s, epoch=8, grad_norm=65.778, loss=256.4194, lr=9.62e-05] 
Epoch 8 TRAIN:  81%|████████  | 112/138 [00:21<00:04,  6.36it/s, epoch=8, grad_norm=65.778, loss=256.4194, lr=9.62e-05]
Epoch 8 TRAIN:  81%|████████  | 112/138 [00:21<00:04,  6.36it/s, epoch=8, grad_norm=85.735, loss=351.3833, lr=9.63e-05]
Epoch 8 TRAIN:  81%|████████  | 112/138 [00:21<00:04,  6.36it/s, epoch=8, grad_norm=218.451, loss=413.6682, lr=9.64e-05]
Epoch 8 TRAIN:  81%|████████  | 112/138 [00:22<00:04,  6.36it/s, epoch=8, grad_norm=160.422, loss=248.5303, lr=9.65e-05]
Epoch 8 TRAIN:  81%|████████  | 112/138 [00:22<00:04,  6.36it/s, epoch=8, grad_norm=145.087, loss=382.8105, lr=9.66e-05]
Epoch 8 TRAIN:  84%|████████▍ | 116/138 [00:22<00:03,  6.29it/s, epoch=8, grad_norm=145.087, loss=382.8105, lr=9.66e-05]
Epoch 8 TRAIN:  84%|████████▍ | 116/138 [00:22<00:03,  6.29it/s, epoch=8, grad_norm=183.180, loss=381.0764, lr=9.66e-05]
Epoch 8 TRAIN:  84%|████████▍ | 116/138 [00:22<00:03,  6.29it/s, epoch=8, grad_norm=95.132, loss=385.6162, lr=9.67e-05] 
Epoch 8 TRAIN:  84%|████████▍ | 116/138 [00:22<00:03,  6.29it/s, epoch=8, grad_norm=227.444, loss=289.7851, lr=9.68e-05]
Epoch 8 TRAIN:  84%|████████▍ | 116/138 [00:22<00:03,  6.29it/s, epoch=8, grad_norm=192.400, loss=354.9332, lr=9.69e-05]
Epoch 8 TRAIN:  87%|████████▋ | 120/138 [00:22<00:02,  6.41it/s, epoch=8, grad_norm=192.400, loss=354.9332, lr=9.69e-05]
Epoch 8 TRAIN:  87%|████████▋ | 120/138 [00:23<00:02,  6.41it/s, epoch=8, grad_norm=166.484, loss=374.3372, lr=9.70e-05]
Epoch 8 TRAIN:  87%|████████▋ | 120/138 [00:23<00:02,  6.41it/s, epoch=8, grad_norm=83.877, loss=274.3875, lr=9.70e-05] 
Epoch 8 TRAIN:  87%|████████▋ | 120/138 [00:23<00:02,  6.41it/s, epoch=8, grad_norm=133.382, loss=401.1179, lr=9.71e-05]
Epoch 8 TRAIN:  87%|████████▋ | 120/138 [00:23<00:02,  6.41it/s, epoch=8, grad_norm=119.533, loss=351.1080, lr=9.72e-05]
Epoch 8 TRAIN:  90%|████████▉ | 124/138 [00:23<00:02,  6.36it/s, epoch=8, grad_norm=119.533, loss=351.1080, lr=9.72e-05]
Epoch 8 TRAIN:  90%|████████▉ | 124/138 [00:23<00:02,  6.36it/s, epoch=8, grad_norm=116.617, loss=395.3204, lr=9.73e-05]
Epoch 8 TRAIN:  90%|████████▉ | 124/138 [00:23<00:02,  6.36it/s, epoch=8, grad_norm=163.705, loss=372.4139, lr=9.74e-05]
Epoch 8 TRAIN:  90%|████████▉ | 124/138 [00:23<00:02,  6.36it/s, epoch=8, grad_norm=66.592, loss=291.8772, lr=9.74e-05] 
Epoch 8 TRAIN:  90%|████████▉ | 124/138 [00:24<00:02,  6.36it/s, epoch=8, grad_norm=272.725, loss=312.9945, lr=9.75e-05]
Epoch 8 TRAIN:  93%|█████████▎| 128/138 [00:24<00:01,  6.50it/s, epoch=8, grad_norm=272.725, loss=312.9945, lr=9.75e-05]
Epoch 8 TRAIN:  93%|█████████▎| 128/138 [00:24<00:01,  6.50it/s, epoch=8, grad_norm=79.896, loss=362.5435, lr=9.76e-05] 
Epoch 8 TRAIN:  93%|█████████▎| 128/138 [00:24<00:01,  6.50it/s, epoch=8, grad_norm=117.062, loss=387.1672, lr=9.77e-05]
Epoch 8 TRAIN:  93%|█████████▎| 128/138 [00:24<00:01,  6.50it/s, epoch=8, grad_norm=112.151, loss=417.5254, lr=9.78e-05]
Epoch 8 TRAIN:  93%|█████████▎| 128/138 [00:24<00:01,  6.50it/s, epoch=8, grad_norm=200.225, loss=255.9540, lr=9.78e-05]
Epoch 8 TRAIN:  96%|█████████▌| 132/138 [00:24<00:00,  6.41it/s, epoch=8, grad_norm=200.225, loss=255.9540, lr=9.78e-05]
Epoch 8 TRAIN:  96%|█████████▌| 132/138 [00:24<00:00,  6.41it/s, epoch=8, grad_norm=115.011, loss=430.5604, lr=9.79e-05]
Epoch 8 TRAIN:  96%|█████████▌| 132/138 [00:24<00:00,  6.41it/s, epoch=8, grad_norm=233.839, loss=400.2585, lr=9.80e-05]
Epoch 8 TRAIN:  96%|█████████▌| 132/138 [00:25<00:00,  6.41it/s, epoch=8, grad_norm=385.786, loss=386.2012, lr=9.81e-05]
Epoch 8 TRAIN:  96%|█████████▌| 132/138 [00:25<00:00,  6.41it/s, epoch=8, grad_norm=118.388, loss=379.6588, lr=9.82e-05]
Epoch 8 TRAIN:  99%|█████████▊| 136/138 [00:25<00:00,  6.70it/s, epoch=8, grad_norm=118.388, loss=379.6588, lr=9.82e-05]
Epoch 8 TRAIN:  99%|█████████▊| 136/138 [00:25<00:00,  6.70it/s, epoch=8, grad_norm=114.832, loss=371.4394, lr=9.82e-05]
Epoch 8 TRAIN:  99%|█████████▉| 137/138 [00:25<00:00,  5.38it/s, epoch=8, grad_norm=114.832, loss=371.4394, lr=9.82e-05]

CV:   0%|          | 0/6 [00:00<?, ?it/s]/home/rookie/Documents/batuka/projects/wenet/wenet/wenet/utils/train_utils.py:689: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  amp_autocast(enabled=dtype is not None,

CV:  17%|█▋        | 1/6 [00:00<00:03,  1.32it/s]
                                                 
2026-04-16 19:56:23,952 INFO Epoch 8 Step 1228 CV info lr 9.8240e-05 cv_loss 392.3195329145952 rank 0 acc 0.2457417075832685
2026-04-16 19:56:23,954 INFO [Rank 0] Checkpoint: save to checkpoint exp/conformer_mongolian/epoch_8.pt
2026-04-16 19:56:24,066 INFO Epoch 9 Step 1228 TRAIN info lr 9.8240e-05 rank 0
2026-04-16 19:56:24,079 INFO using accumulate grad, new batch size is 1 times larger than before

Epoch 9 TRAIN:   0%|          | 0/137 [00:00<?, ?it/s]
Epoch 9 TRAIN:   0%|          | 0/137 [00:03<?, ?it/s, epoch=9, grad_norm=579.197, loss=445.9645, lr=9.83e-05]
Epoch 9 TRAIN:   1%|          | 1/137 [00:03<09:03,  3.99s/it, epoch=9, grad_norm=579.197, loss=445.9645, lr=9.83e-05]
Epoch 9 TRAIN:   1%|          | 1/137 [00:04<09:03,  3.99s/it, epoch=9, grad_norm=239.531, loss=245.9878, lr=9.84e-05]
Epoch 9 TRAIN:   1%|          | 1/137 [00:04<09:03,  3.99s/it, epoch=9, grad_norm=313.872, loss=290.6801, lr=9.85e-05]
Epoch 9 TRAIN:   1%|          | 1/137 [00:04<09:03,  3.99s/it, epoch=9, grad_norm=470.205, loss=378.4406, lr=9.86e-05]
Epoch 9 TRAIN:   1%|          | 1/137 [00:04<09:03,  3.99s/it, epoch=9, grad_norm=555.424, loss=457.3835, lr=9.86e-05]
Epoch 9 TRAIN:   1%|          | 1/137 [00:04<09:03,  3.99s/it, epoch=9, grad_norm=539.056, loss=439.5878, lr=9.87e-05]
Epoch 9 TRAIN:   1%|          | 1/137 [00:04<09:03,  3.99s/it, epoch=9, grad_norm=366.483, loss=373.8736, lr=9.88e-05]
Epoch 9 TRAIN:   5%|▌         | 7/137 [00:04<01:04,  2.01it/s, epoch=9, grad_norm=366.483, loss=373.8736, lr=9.88e-05]
Epoch 9 TRAIN:   5%|▌         | 7/137 [00:04<01:04,  2.01it/s, epoch=9, grad_norm=300.081, loss=344.8320, lr=9.89e-05]
Epoch 9 TRAIN:   5%|▌         | 7/137 [00:04<01:04,  2.01it/s, epoch=9, grad_norm=257.081, loss=360.1360, lr=9.90e-05]
Epoch 9 TRAIN:   5%|▌         | 7/137 [00:04<01:04,  2.01it/s, epoch=9, grad_norm=276.345, loss=395.0346, lr=9.90e-05]
Epoch 9 TRAIN:   5%|▌         | 7/137 [00:05<01:04,  2.01it/s, epoch=9, grad_norm=167.180, loss=369.9406, lr=9.91e-05]
Epoch 9 TRAIN:   8%|▊         | 11/137 [00:05<00:41,  3.05it/s, epoch=9, grad_norm=167.180, loss=369.9406, lr=9.91e-05]
Epoch 9 TRAIN:   8%|▊         | 11/137 [00:05<00:41,  3.05it/s, epoch=9, grad_norm=109.674, loss=329.6077, lr=9.92e-05]
Epoch 9 TRAIN:   8%|▊         | 11/137 [00:05<00:41,  3.05it/s, epoch=9, grad_norm=69.943, loss=344.6478, lr=9.93e-05] 
Epoch 9 TRAIN:   8%|▊         | 11/137 [00:05<00:41,  3.05it/s, epoch=9, grad_norm=56.047, loss=343.6317, lr=9.94e-05]
Epoch 9 TRAIN:   8%|▊         | 11/137 [00:05<00:41,  3.05it/s, epoch=9, grad_norm=102.351, loss=309.0444, lr=9.94e-05]
Epoch 9 TRAIN:  11%|█         | 15/137 [00:05<00:31,  3.87it/s, epoch=9, grad_norm=102.351, loss=309.0444, lr=9.94e-05]
Epoch 9 TRAIN:  11%|█         | 15/137 [00:05<00:31,  3.87it/s, epoch=9, grad_norm=129.711, loss=308.0544, lr=9.95e-05]
Epoch 9 TRAIN:  11%|█         | 15/137 [00:05<00:31,  3.87it/s, epoch=9, grad_norm=75.245, loss=381.2753, lr=9.96e-05] 
Epoch 9 TRAIN:  11%|█         | 15/137 [00:06<00:31,  3.87it/s, epoch=9, grad_norm=99.357, loss=303.0626, lr=9.97e-05]
Epoch 9 TRAIN:  11%|█         | 15/137 [00:06<00:31,  3.87it/s, epoch=9, grad_norm=80.197, loss=367.1079, lr=9.98e-05]
Epoch 9 TRAIN:  14%|█▍        | 19/137 [00:06<00:26,  4.54it/s, epoch=9, grad_norm=80.197, loss=367.1079, lr=9.98e-05]
Epoch 9 TRAIN:  14%|█▍        | 19/137 [00:06<00:26,  4.54it/s, epoch=9, grad_norm=131.568, loss=272.3440, lr=9.98e-05]
Epoch 9 TRAIN:  14%|█▍        | 19/137 [00:06<00:26,  4.54it/s, epoch=9, grad_norm=47.180, loss=334.9297, lr=9.99e-05] 
Epoch 9 TRAIN:  14%|█▍        | 19/137 [00:06<00:26,  4.54it/s, epoch=9, grad_norm=46.915, loss=324.3987, lr=1.00e-04]
Epoch 9 TRAIN:  14%|█▍        | 19/137 [00:06<00:26,  4.54it/s, epoch=9, grad_norm=69.407, loss=350.2169, lr=1.00e-04]
Epoch 9 TRAIN:  17%|█▋        | 23/137 [00:06<00:22,  5.01it/s, epoch=9, grad_norm=69.407, loss=350.2169, lr=1.00e-04]
Epoch 9 TRAIN:  17%|█▋        | 23/137 [00:07<00:22,  5.01it/s, epoch=9, grad_norm=54.222, loss=328.1032, lr=1.00e-04]
Epoch 9 TRAIN:  17%|█▋        | 23/137 [00:07<00:22,  5.01it/s, epoch=9, grad_norm=61.142, loss=357.7733, lr=1.00e-04]
Epoch 9 TRAIN:  17%|█▋        | 23/137 [00:07<00:22,  5.01it/s, epoch=9, grad_norm=66.981, loss=368.8579, lr=1.00e-04]
Epoch 9 TRAIN:  17%|█▋        | 23/137 [00:07<00:22,  5.01it/s, epoch=9, grad_norm=138.566, loss=303.1461, lr=1.00e-04]
Epoch 9 TRAIN:  20%|█▉        | 27/137 [00:07<00:20,  5.39it/s, epoch=9, grad_norm=138.566, loss=303.1461, lr=1.00e-04]
Epoch 9 TRAIN:  20%|█▉        | 27/137 [00:07<00:20,  5.39it/s, epoch=9, grad_norm=90.701, loss=331.1483, lr=1.00e-04] 
Epoch 9 TRAIN:  20%|█▉        | 27/137 [00:07<00:20,  5.39it/s, epoch=9, grad_norm=60.689, loss=347.9065, lr=1.01e-04]
Epoch 9 TRAIN:  20%|█▉        | 27/137 [00:07<00:20,  5.39it/s, epoch=9, grad_norm=69.303, loss=357.6441, lr=1.01e-04]
Epoch 9 TRAIN:  20%|█▉        | 27/137 [00:08<00:20,  5.39it/s, epoch=9, grad_norm=59.493, loss=340.0648, lr=1.01e-04]
Epoch 9 TRAIN:  23%|██▎       | 31/137 [00:08<00:18,  5.78it/s, epoch=9, grad_norm=59.493, loss=340.0648, lr=1.01e-04]
Epoch 9 TRAIN:  23%|██▎       | 31/137 [00:08<00:18,  5.78it/s, epoch=9, grad_norm=55.284, loss=346.9175, lr=1.01e-04]
Epoch 9 TRAIN:  23%|██▎       | 31/137 [00:08<00:18,  5.78it/s, epoch=9, grad_norm=77.963, loss=364.4956, lr=1.01e-04]
Epoch 9 TRAIN:  23%|██▎       | 31/137 [00:08<00:18,  5.78it/s, epoch=9, grad_norm=69.588, loss=373.4245, lr=1.01e-04]
Epoch 9 TRAIN:  23%|██▎       | 31/137 [00:08<00:18,  5.78it/s, epoch=9, grad_norm=54.840, loss=349.5209, lr=1.01e-04]
Epoch 9 TRAIN:  26%|██▌       | 35/137 [00:08<00:16,  6.02it/s, epoch=9, grad_norm=54.840, loss=349.5209, lr=1.01e-04]
Epoch 9 TRAIN:  26%|██▌       | 35/137 [00:08<00:16,  6.02it/s, epoch=9, grad_norm=87.560, loss=317.3293, lr=1.01e-04]
Epoch 9 TRAIN:  26%|██▌       | 35/137 [00:09<00:16,  6.02it/s, epoch=9, grad_norm=61.751, loss=310.2072, lr=1.01e-04]
Epoch 9 TRAIN:  26%|██▌       | 35/137 [00:09<00:16,  6.02it/s, epoch=9, grad_norm=68.201, loss=351.6833, lr=1.01e-04]
Epoch 9 TRAIN:  26%|██▌       | 35/137 [00:09<00:16,  6.02it/s, epoch=9, grad_norm=57.761, loss=336.7122, lr=1.01e-04]
Epoch 9 TRAIN:  28%|██▊       | 39/137 [00:09<00:16,  6.11it/s, epoch=9, grad_norm=57.761, loss=336.7122, lr=1.01e-04]
Epoch 9 TRAIN:  28%|██▊       | 39/137 [00:09<00:16,  6.11it/s, epoch=9, grad_norm=56.950, loss=310.0867, lr=1.01e-04]
Epoch 9 TRAIN:  28%|██▊       | 39/137 [00:09<00:16,  6.11it/s, epoch=9, grad_norm=66.298, loss=324.8687, lr=1.02e-04]
Epoch 9 TRAIN:  28%|██▊       | 39/137 [00:09<00:16,  6.11it/s, epoch=9, grad_norm=52.048, loss=314.6001, lr=1.02e-04]
Epoch 9 TRAIN:  28%|██▊       | 39/137 [00:09<00:16,  6.11it/s, epoch=9, grad_norm=53.034, loss=318.5804, lr=1.02e-04]
Epoch 9 TRAIN:  31%|███▏      | 43/137 [00:09<00:15,  6.20it/s, epoch=9, grad_norm=53.034, loss=318.5804, lr=1.02e-04]
Epoch 9 TRAIN:  31%|███▏      | 43/137 [00:10<00:15,  6.20it/s, epoch=9, grad_norm=85.540, loss=272.4398, lr=1.02e-04]
Epoch 9 TRAIN:  31%|███▏      | 43/137 [00:10<00:15,  6.20it/s, epoch=9, grad_norm=50.789, loss=316.9230, lr=1.02e-04]
Epoch 9 TRAIN:  31%|███▏      | 43/137 [00:10<00:15,  6.20it/s, epoch=9, grad_norm=132.936, loss=367.7398, lr=1.02e-04]
Epoch 9 TRAIN:  31%|███▏      | 43/137 [00:10<00:15,  6.20it/s, epoch=9, grad_norm=55.044, loss=294.5062, lr=1.02e-04] 
Epoch 9 TRAIN:  34%|███▍      | 47/137 [00:10<00:14,  6.26it/s, epoch=9, grad_norm=55.044, loss=294.5062, lr=1.02e-04]
Epoch 9 TRAIN:  34%|███▍      | 47/137 [00:10<00:14,  6.26it/s, epoch=9, grad_norm=82.807, loss=348.2175, lr=1.02e-04]
Epoch 9 TRAIN:  34%|███▍      | 47/137 [00:10<00:14,  6.26it/s, epoch=9, grad_norm=71.499, loss=358.4438, lr=1.02e-04]
Epoch 9 TRAIN:  34%|███▍      | 47/137 [00:11<00:14,  6.26it/s, epoch=9, grad_norm=94.064, loss=375.4211, lr=1.02e-04]
Epoch 9 TRAIN:  34%|███▍      | 47/137 [00:11<00:14,  6.26it/s, epoch=9, grad_norm=185.406, loss=317.1297, lr=1.02e-04]
Epoch 9 TRAIN:  37%|███▋      | 51/137 [00:11<00:13,  6.35it/s, epoch=9, grad_norm=185.406, loss=317.1297, lr=1.02e-04]
Epoch 9 TRAIN:  37%|███▋      | 51/137 [00:11<00:13,  6.35it/s, epoch=9, grad_norm=93.801, loss=365.9235, lr=1.02e-04] 
Epoch 9 TRAIN:  37%|███▋      | 51/137 [00:11<00:13,  6.35it/s, epoch=9, grad_norm=81.516, loss=315.8921, lr=1.02e-04]
Epoch 9 TRAIN:  37%|███▋      | 51/137 [00:11<00:13,  6.35it/s, epoch=9, grad_norm=96.150, loss=365.4108, lr=1.03e-04]
Epoch 9 TRAIN:  37%|███▋      | 51/137 [00:11<00:13,  6.35it/s, epoch=9, grad_norm=87.658, loss=359.8171, lr=1.03e-04]
Epoch 9 TRAIN:  40%|████      | 55/137 [00:11<00:13,  6.26it/s, epoch=9, grad_norm=87.658, loss=359.8171, lr=1.03e-04]
Epoch 9 TRAIN:  40%|████      | 55/137 [00:12<00:13,  6.26it/s, epoch=9, grad_norm=73.925, loss=337.2621, lr=1.03e-04]
Epoch 9 TRAIN:  40%|████      | 55/137 [00:12<00:13,  6.26it/s, epoch=9, grad_norm=53.689, loss=366.4922, lr=1.03e-04]
Epoch 9 TRAIN:  40%|████      | 55/137 [00:12<00:13,  6.26it/s, epoch=9, grad_norm=60.276, loss=339.6772, lr=1.03e-04]
Epoch 9 TRAIN:  40%|████      | 55/137 [00:12<00:13,  6.26it/s, epoch=9, grad_norm=151.464, loss=344.8084, lr=1.03e-04]
Epoch 9 TRAIN:  43%|████▎     | 59/137 [00:12<00:12,  6.20it/s, epoch=9, grad_norm=151.464, loss=344.8084, lr=1.03e-04]
Epoch 9 TRAIN:  43%|████▎     | 59/137 [00:12<00:12,  6.20it/s, epoch=9, grad_norm=67.547, loss=336.1866, lr=1.03e-04] 
Epoch 9 TRAIN:  43%|████▎     | 59/137 [00:12<00:12,  6.20it/s, epoch=9, grad_norm=89.907, loss=318.4817, lr=1.03e-04]
Epoch 9 TRAIN:  43%|████▎     | 59/137 [00:12<00:12,  6.20it/s, epoch=9, grad_norm=60.653, loss=340.7667, lr=1.03e-04]
Epoch 9 TRAIN:  43%|████▎     | 59/137 [00:13<00:12,  6.20it/s, epoch=9, grad_norm=72.428, loss=349.3992, lr=1.03e-04]
Epoch 9 TRAIN:  46%|████▌     | 63/137 [00:13<00:11,  6.31it/s, epoch=9, grad_norm=72.428, loss=349.3992, lr=1.03e-04]
Epoch 9 TRAIN:  46%|████▌     | 63/137 [00:13<00:11,  6.31it/s, epoch=9, grad_norm=124.092, loss=373.2910, lr=1.03e-04]
Epoch 9 TRAIN:  46%|████▌     | 63/137 [00:13<00:11,  6.31it/s, epoch=9, grad_norm=135.026, loss=337.8556, lr=1.03e-04]
Epoch 9 TRAIN:  46%|████▌     | 63/137 [00:13<00:11,  6.31it/s, epoch=9, grad_norm=58.801, loss=335.2719, lr=1.04e-04] 
Epoch 9 TRAIN:  46%|████▌     | 63/137 [00:13<00:11,  6.31it/s, epoch=9, grad_norm=177.038, loss=304.8082, lr=1.04e-04]
Epoch 9 TRAIN:  49%|████▉     | 67/137 [00:13<00:10,  6.43it/s, epoch=9, grad_norm=177.038, loss=304.8082, lr=1.04e-04]
Epoch 9 TRAIN:  49%|████▉     | 67/137 [00:13<00:10,  6.43it/s, epoch=9, grad_norm=89.370, loss=337.9601, lr=1.04e-04] 
Epoch 9 TRAIN:  49%|████▉     | 67/137 [00:14<00:10,  6.43it/s, epoch=9, grad_norm=77.730, loss=322.4258, lr=1.04e-04]
Epoch 9 TRAIN:  49%|████▉     | 67/137 [00:14<00:10,  6.43it/s, epoch=9, grad_norm=64.146, loss=365.8925, lr=1.04e-04]
Epoch 9 TRAIN:  49%|████▉     | 67/137 [00:14<00:10,  6.43it/s, epoch=9, grad_norm=85.153, loss=344.4922, lr=1.04e-04]
Epoch 9 TRAIN:  52%|█████▏    | 71/137 [00:14<00:10,  6.31it/s, epoch=9, grad_norm=85.153, loss=344.4922, lr=1.04e-04]
Epoch 9 TRAIN:  52%|█████▏    | 71/137 [00:14<00:10,  6.31it/s, epoch=9, grad_norm=96.198, loss=365.0472, lr=1.04e-04]
Epoch 9 TRAIN:  52%|█████▏    | 71/137 [00:14<00:10,  6.31it/s, epoch=9, grad_norm=126.112, loss=326.2171, lr=1.04e-04]
Epoch 9 TRAIN:  52%|█████▏    | 71/137 [00:14<00:10,  6.31it/s, epoch=9, grad_norm=96.973, loss=363.4310, lr=1.04e-04] 
Epoch 9 TRAIN:  52%|█████▏    | 71/137 [00:15<00:10,  6.31it/s, epoch=9, grad_norm=76.030, loss=385.7356, lr=1.04e-04]
Epoch 9 TRAIN:  55%|█████▍    | 75/137 [00:15<00:09,  6.21it/s, epoch=9, grad_norm=76.030, loss=385.7356, lr=1.04e-04]
Epoch 9 TRAIN:  55%|█████▍    | 75/137 [00:15<00:09,  6.21it/s, epoch=9, grad_norm=99.062, loss=345.5002, lr=1.04e-04]
Epoch 9 TRAIN:  55%|█████▍    | 75/137 [00:15<00:09,  6.21it/s, epoch=9, grad_norm=125.711, loss=321.7124, lr=1.04e-04]
Epoch 9 TRAIN:  55%|█████▍    | 75/137 [00:15<00:09,  6.21it/s, epoch=9, grad_norm=66.694, loss=348.0609, lr=1.04e-04] 
Epoch 9 TRAIN:  55%|█████▍    | 75/137 [00:15<00:09,  6.21it/s, epoch=9, grad_norm=69.379, loss=321.0172, lr=1.05e-04]
Epoch 9 TRAIN:  58%|█████▊    | 79/137 [00:15<00:09,  6.22it/s, epoch=9, grad_norm=69.379, loss=321.0172, lr=1.05e-04]
Epoch 9 TRAIN:  58%|█████▊    | 79/137 [00:15<00:09,  6.22it/s, epoch=9, grad_norm=85.716, loss=265.2827, lr=1.05e-04]
Epoch 9 TRAIN:  58%|█████▊    | 79/137 [00:16<00:09,  6.22it/s, epoch=9, grad_norm=225.993, loss=235.8278, lr=1.05e-04]
Epoch 9 TRAIN:  58%|█████▊    | 79/137 [00:16<00:09,  6.22it/s, epoch=9, grad_norm=176.239, loss=334.0792, lr=1.05e-04]
Epoch 9 TRAIN:  58%|█████▊    | 79/137 [00:16<00:09,  6.22it/s, epoch=9, grad_norm=64.940, loss=316.3210, lr=1.05e-04] 
Epoch 9 TRAIN:  61%|██████    | 83/137 [00:16<00:08,  6.09it/s, epoch=9, grad_norm=64.940, loss=316.3210, lr=1.05e-04]
Epoch 9 TRAIN:  61%|██████    | 83/137 [00:16<00:08,  6.09it/s, epoch=9, grad_norm=110.386, loss=334.0702, lr=1.05e-04]
Epoch 9 TRAIN:  61%|██████    | 83/137 [00:16<00:08,  6.09it/s, epoch=9, grad_norm=52.932, loss=292.9876, lr=1.05e-04] 
Epoch 9 TRAIN:  61%|██████    | 83/137 [00:16<00:08,  6.09it/s, epoch=9, grad_norm=76.493, loss=343.3356, lr=1.05e-04]
Epoch 9 TRAIN:  61%|██████    | 83/137 [00:16<00:08,  6.09it/s, epoch=9, grad_norm=73.016, loss=324.1238, lr=1.05e-04]
Epoch 9 TRAIN:  64%|██████▎   | 87/137 [00:16<00:07,  6.28it/s, epoch=9, grad_norm=73.016, loss=324.1238, lr=1.05e-04]
Epoch 9 TRAIN:  64%|██████▎   | 87/137 [00:17<00:07,  6.28it/s, epoch=9, grad_norm=94.681, loss=404.6573, lr=1.05e-04]
Epoch 9 TRAIN:  64%|██████▎   | 87/137 [00:17<00:07,  6.28it/s, epoch=9, grad_norm=206.299, loss=354.3310, lr=1.05e-04]
Epoch 9 TRAIN:  64%|██████▎   | 87/137 [00:17<00:07,  6.28it/s, epoch=9, grad_norm=58.302, loss=341.3492, lr=1.05e-04] 
Epoch 9 TRAIN:  64%|██████▎   | 87/137 [00:17<00:07,  6.28it/s, epoch=9, grad_norm=405.303, loss=306.9264, lr=1.06e-04]
Epoch 9 TRAIN:  66%|██████▋   | 91/137 [00:17<00:07,  5.92it/s, epoch=9, grad_norm=405.303, loss=306.9264, lr=1.06e-04]
Epoch 9 TRAIN:  66%|██████▋   | 91/137 [00:17<00:07,  5.92it/s, epoch=9, grad_norm=347.299, loss=307.6303, lr=1.06e-04]
Epoch 9 TRAIN:  66%|██████▋   | 91/137 [00:18<00:07,  5.92it/s, epoch=9, grad_norm=274.056, loss=264.2952, lr=1.06e-04]
Epoch 9 TRAIN:  66%|██████▋   | 91/137 [00:18<00:07,  5.92it/s, epoch=9, grad_norm=81.318, loss=370.8543, lr=1.06e-04] 
Epoch 9 TRAIN:  69%|██████▊   | 94/137 [00:18<00:07,  5.88it/s, epoch=9, grad_norm=81.318, loss=370.8543, lr=1.06e-04]
Epoch 9 TRAIN:  69%|██████▊   | 94/137 [00:18<00:07,  5.88it/s, epoch=9, grad_norm=82.629, loss=267.0912, lr=1.06e-04]
Epoch 9 TRAIN:  69%|██████▊   | 94/137 [00:18<00:07,  5.88it/s, epoch=9, grad_norm=90.060, loss=232.4883, lr=1.06e-04]
Epoch 9 TRAIN:  69%|██████▊   | 94/137 [00:18<00:07,  5.88it/s, epoch=9, grad_norm=104.282, loss=325.9022, lr=1.06e-04]
Epoch 9 TRAIN:  69%|██████▊   | 94/137 [00:18<00:07,  5.88it/s, epoch=9, grad_norm=287.595, loss=342.3331, lr=1.06e-04]
Epoch 9 TRAIN:  72%|███████▏  | 98/137 [00:18<00:06,  5.99it/s, epoch=9, grad_norm=287.595, loss=342.3331, lr=1.06e-04]
Epoch 9 TRAIN:  72%|███████▏  | 98/137 [00:19<00:06,  5.99it/s, epoch=9, grad_norm=95.437, loss=315.4474, lr=1.06e-04] 
Epoch 9 TRAIN:  72%|███████▏  | 98/137 [00:19<00:06,  5.99it/s, epoch=9, grad_norm=178.001, loss=347.6010, lr=1.06e-04]2026-04-16 19:56:43,336 DEBUG TRAIN | steps/sec 5.283| Batch 9/100 loss 347.601013 loss_att 308.005219 loss_ctc 439.991211 th_accuracy 0.264192 lr 1.0624e-04 grad_norm 178.000809 rank 0

Epoch 9 TRAIN:  72%|███████▏  | 98/137 [00:19<00:06,  5.99it/s, epoch=9, grad_norm=189.585, loss=370.8708, lr=1.06e-04]
Epoch 9 TRAIN:  72%|███████▏  | 98/137 [00:19<00:06,  5.99it/s, epoch=9, grad_norm=80.941, loss=256.4491, lr=1.06e-04] 
Epoch 9 TRAIN:  74%|███████▍  | 102/137 [00:19<00:05,  5.94it/s, epoch=9, grad_norm=80.941, loss=256.4491, lr=1.06e-04]
Epoch 9 TRAIN:  74%|███████▍  | 102/137 [00:19<00:05,  5.94it/s, epoch=9, grad_norm=231.277, loss=434.7838, lr=1.06e-04]
Epoch 9 TRAIN:  74%|███████▍  | 102/137 [00:19<00:05,  5.94it/s, epoch=9, grad_norm=112.089, loss=245.1816, lr=1.07e-04]
Epoch 9 TRAIN:  74%|███████▍  | 102/137 [00:20<00:05,  5.94it/s, epoch=9, grad_norm=112.511, loss=300.7892, lr=1.07e-04]
Epoch 9 TRAIN:  74%|███████▍  | 102/137 [00:20<00:05,  5.94it/s, epoch=9, grad_norm=50.608, loss=273.5209, lr=1.07e-04] 
Epoch 9 TRAIN:  77%|███████▋  | 106/137 [00:20<00:05,  5.84it/s, epoch=9, grad_norm=50.608, loss=273.5209, lr=1.07e-04]
Epoch 9 TRAIN:  77%|███████▋  | 106/137 [00:20<00:05,  5.84it/s, epoch=9, grad_norm=170.896, loss=357.8454, lr=1.07e-04]
Epoch 9 TRAIN:  77%|███████▋  | 106/137 [00:20<00:05,  5.84it/s, epoch=9, grad_norm=180.301, loss=310.5587, lr=1.07e-04]
Epoch 9 TRAIN:  77%|███████▋  | 106/137 [00:20<00:05,  5.84it/s, epoch=9, grad_norm=62.240, loss=345.8097, lr=1.07e-04] 
Epoch 9 TRAIN:  80%|███████▉  | 109/137 [00:20<00:04,  5.63it/s, epoch=9, grad_norm=62.240, loss=345.8097, lr=1.07e-04]
Epoch 9 TRAIN:  80%|███████▉  | 109/137 [00:21<00:04,  5.63it/s, epoch=9, grad_norm=62.429, loss=295.4597, lr=1.07e-04]
Epoch 9 TRAIN:  80%|███████▉  | 109/137 [00:21<00:04,  5.63it/s, epoch=9, grad_norm=65.773, loss=287.4798, lr=1.07e-04]
Epoch 9 TRAIN:  80%|███████▉  | 109/137 [00:21<00:04,  5.63it/s, epoch=9, grad_norm=83.318, loss=267.3251, lr=1.07e-04]
Epoch 9 TRAIN:  80%|███████▉  | 109/137 [00:21<00:04,  5.63it/s, epoch=9, grad_norm=73.117, loss=308.9628, lr=1.07e-04]
Epoch 9 TRAIN:  82%|████████▏ | 113/137 [00:21<00:04,  5.82it/s, epoch=9, grad_norm=73.117, loss=308.9628, lr=1.07e-04]
Epoch 9 TRAIN:  82%|████████▏ | 113/137 [00:21<00:04,  5.82it/s, epoch=9, grad_norm=119.297, loss=288.2449, lr=1.07e-04]
Epoch 9 TRAIN:  82%|████████▏ | 113/137 [00:21<00:04,  5.82it/s, epoch=9, grad_norm=104.921, loss=339.7993, lr=1.07e-04]
Epoch 9 TRAIN:  82%|████████▏ | 113/137 [00:22<00:04,  5.82it/s, epoch=9, grad_norm=106.686, loss=302.4158, lr=1.08e-04]
Epoch 9 TRAIN:  85%|████████▍ | 116/137 [00:22<00:03,  5.71it/s, epoch=9, grad_norm=106.686, loss=302.4158, lr=1.08e-04]
Epoch 9 TRAIN:  85%|████████▍ | 116/137 [00:22<00:03,  5.71it/s, epoch=9, grad_norm=170.415, loss=428.6101, lr=1.08e-04]
Epoch 9 TRAIN:  85%|████████▍ | 116/137 [00:22<00:03,  5.71it/s, epoch=9, grad_norm=132.230, loss=354.7449, lr=1.08e-04]
Epoch 9 TRAIN:  85%|████████▍ | 116/137 [00:22<00:03,  5.71it/s, epoch=9, grad_norm=129.231, loss=254.5517, lr=1.08e-04]
Epoch 9 TRAIN:  87%|████████▋ | 119/137 [00:22<00:03,  5.77it/s, epoch=9, grad_norm=129.231, loss=254.5517, lr=1.08e-04]
Epoch 9 TRAIN:  87%|████████▋ | 119/137 [00:22<00:03,  5.77it/s, epoch=9, grad_norm=112.952, loss=361.0170, lr=1.08e-04]
Epoch 9 TRAIN:  87%|████████▋ | 119/137 [00:22<00:03,  5.77it/s, epoch=9, grad_norm=86.943, loss=303.4085, lr=1.08e-04] 
Epoch 9 TRAIN:  87%|████████▋ | 119/137 [00:23<00:03,  5.77it/s, epoch=9, grad_norm=96.354, loss=298.0056, lr=1.08e-04]
Epoch 9 TRAIN:  87%|████████▋ | 119/137 [00:23<00:03,  5.77it/s, epoch=9, grad_norm=141.029, loss=444.8751, lr=1.08e-04]
Epoch 9 TRAIN:  90%|████████▉ | 123/137 [00:23<00:02,  5.91it/s, epoch=9, grad_norm=141.029, loss=444.8751, lr=1.08e-04]
Epoch 9 TRAIN:  90%|████████▉ | 123/137 [00:23<00:02,  5.91it/s, epoch=9, grad_norm=130.586, loss=440.1954, lr=1.08e-04]
Epoch 9 TRAIN:  90%|████████▉ | 123/137 [00:23<00:02,  5.91it/s, epoch=9, grad_norm=103.132, loss=348.6777, lr=1.08e-04]
Epoch 9 TRAIN:  90%|████████▉ | 123/137 [00:23<00:02,  5.91it/s, epoch=9, grad_norm=89.271, loss=381.4892, lr=1.08e-04] 
Epoch 9 TRAIN:  90%|████████▉ | 123/137 [00:23<00:02,  5.91it/s, epoch=9, grad_norm=195.276, loss=340.8283, lr=1.08e-04]
Epoch 9 TRAIN:  93%|█████████▎| 127/137 [00:23<00:01,  6.14it/s, epoch=9, grad_norm=195.276, loss=340.8283, lr=1.08e-04]
Epoch 9 TRAIN:  93%|█████████▎| 127/137 [00:24<00:01,  6.14it/s, epoch=9, grad_norm=84.494, loss=316.3114, lr=1.08e-04] 
Epoch 9 TRAIN:  93%|█████████▎| 127/137 [00:24<00:01,  6.14it/s, epoch=9, grad_norm=170.469, loss=392.0305, lr=1.09e-04]
Epoch 9 TRAIN:  93%|█████████▎| 127/137 [00:24<00:01,  6.14it/s, epoch=9, grad_norm=67.300, loss=322.0881, lr=1.09e-04] 
Epoch 9 TRAIN:  93%|█████████▎| 127/137 [00:24<00:01,  6.14it/s, epoch=9, grad_norm=76.114, loss=313.4887, lr=1.09e-04]
Epoch 9 TRAIN:  96%|█████████▌| 131/137 [00:24<00:01,  5.82it/s, epoch=9, grad_norm=76.114, loss=313.4887, lr=1.09e-04]
Epoch 9 TRAIN:  96%|█████████▌| 131/137 [00:24<00:01,  5.82it/s, epoch=9, grad_norm=116.408, loss=378.3718, lr=1.09e-04]
Epoch 9 TRAIN:  96%|█████████▌| 131/137 [00:25<00:01,  5.82it/s, epoch=9, grad_norm=96.445, loss=351.7486, lr=1.09e-04] 
Epoch 9 TRAIN:  97%|█████████▋| 133/137 [00:25<00:00,  5.32it/s, epoch=9, grad_norm=96.445, loss=351.7486, lr=1.09e-04]

CV:   0%|          | 0/6 [00:00<?, ?it/s]/home/rookie/Documents/batuka/projects/wenet/wenet/wenet/utils/train_utils.py:689: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  amp_autocast(enabled=dtype is not None,

CV:  17%|█▋        | 1/6 [00:00<00:03,  1.32it/s]
                                                 
2026-04-16 19:56:49,941 INFO Epoch 9 Step 1361 CV info lr 1.0888e-04 cv_loss 394.89182212136006 rank 0 acc 0.24636398504177728
2026-04-16 19:56:49,943 INFO [Rank 0] Checkpoint: save to checkpoint exp/conformer_mongolian/epoch_9.pt
2026-04-16 19:56:50,055 INFO Epoch 10 Step 1361 TRAIN info lr 1.0888e-04 rank 0
2026-04-16 19:56:50,076 INFO using accumulate grad, new batch size is 1 times larger than before

Epoch 10 TRAIN:   0%|          | 0/133 [00:00<?, ?it/s]
Epoch 10 TRAIN:   0%|          | 0/133 [00:04<?, ?it/s, epoch=10, grad_norm=543.559, loss=448.5774, lr=1.09e-04]
Epoch 10 TRAIN:   1%|          | 1/133 [00:04<08:57,  4.07s/it, epoch=10, grad_norm=543.559, loss=448.5774, lr=1.09e-04]
Epoch 10 TRAIN:   1%|          | 1/133 [00:04<08:57,  4.07s/it, epoch=10, grad_norm=414.378, loss=382.3507, lr=1.09e-04]
Epoch 10 TRAIN:   1%|          | 1/133 [00:04<08:57,  4.07s/it, epoch=10, grad_norm=280.859, loss=287.2879, lr=1.09e-04]
Epoch 10 TRAIN:   1%|          | 1/133 [00:04<08:57,  4.07s/it, epoch=10, grad_norm=355.599, loss=350.4791, lr=1.09e-04]
Epoch 10 TRAIN:   1%|          | 1/133 [00:04<08:57,  4.07s/it, epoch=10, grad_norm=277.278, loss=312.8462, lr=1.09e-04]
Epoch 10 TRAIN:   1%|          | 1/133 [00:04<08:57,  4.07s/it, epoch=10, grad_norm=324.713, loss=397.5517, lr=1.09e-04]
Epoch 10 TRAIN:   1%|          | 1/133 [00:04<08:57,  4.07s/it, epoch=10, grad_norm=239.362, loss=386.3574, lr=1.09e-04]
Epoch 10 TRAIN:   1%|          | 1/133 [00:04<08:57,  4.07s/it, epoch=10, grad_norm=149.812, loss=279.9944, lr=1.10e-04]
Epoch 10 TRAIN:   1%|          | 1/133 [00:04<08:57,  4.07s/it, epoch=10, grad_norm=150.634, loss=332.2673, lr=1.10e-04]
Epoch 10 TRAIN:   7%|▋         | 9/133 [00:04<00:49,  2.50it/s, epoch=10, grad_norm=150.634, loss=332.2673, lr=1.10e-04]
Epoch 10 TRAIN:   7%|▋         | 9/133 [00:04<00:49,  2.50it/s, epoch=10, grad_norm=72.990, loss=363.0722, lr=1.10e-04] 
Epoch 10 TRAIN:   7%|▋         | 9/133 [00:05<00:49,  2.50it/s, epoch=10, grad_norm=65.643, loss=359.7686, lr=1.10e-04]
Epoch 10 TRAIN:   7%|▋         | 9/133 [00:05<00:49,  2.50it/s, epoch=10, grad_norm=47.965, loss=331.8528, lr=1.10e-04]
Epoch 10 TRAIN:   7%|▋         | 9/133 [00:05<00:49,  2.50it/s, epoch=10, grad_norm=54.242, loss=338.9448, lr=1.10e-04]
Epoch 10 TRAIN:  10%|▉         | 13/133 [00:05<00:36,  3.30it/s, epoch=10, grad_norm=54.242, loss=338.9448, lr=1.10e-04]
Epoch 10 TRAIN:  10%|▉         | 13/133 [00:05<00:36,  3.30it/s, epoch=10, grad_norm=73.322, loss=316.0645, lr=1.10e-04]
Epoch 10 TRAIN:  10%|▉         | 13/133 [00:05<00:36,  3.30it/s, epoch=10, grad_norm=65.228, loss=347.7570, lr=1.10e-04]
Epoch 10 TRAIN:  10%|▉         | 13/133 [00:05<00:36,  3.30it/s, epoch=10, grad_norm=56.271, loss=318.6259, lr=1.10e-04]
Epoch 10 TRAIN:  10%|▉         | 13/133 [00:05<00:36,  3.30it/s, epoch=10, grad_norm=83.096, loss=289.5712, lr=1.10e-04]
Epoch 10 TRAIN:  13%|█▎        | 17/133 [00:05<00:28,  4.02it/s, epoch=10, grad_norm=83.096, loss=289.5712, lr=1.10e-04]
Epoch 10 TRAIN:  13%|█▎        | 17/133 [00:06<00:28,  4.02it/s, epoch=10, grad_norm=85.337, loss=360.1469, lr=1.10e-04]
Epoch 10 TRAIN:  13%|█▎        | 17/133 [00:06<00:28,  4.02it/s, epoch=10, grad_norm=105.519, loss=367.0945, lr=1.10e-04]
Epoch 10 TRAIN:  13%|█▎        | 17/133 [00:06<00:28,  4.02it/s, epoch=10, grad_norm=76.826, loss=350.3431, lr=1.10e-04] 
Epoch 10 TRAIN:  13%|█▎        | 17/133 [00:06<00:28,  4.02it/s, epoch=10, grad_norm=52.028, loss=345.7060, lr=1.11e-04]
Epoch 10 TRAIN:  16%|█▌        | 21/133 [00:06<00:24,  4.60it/s, epoch=10, grad_norm=52.028, loss=345.7060, lr=1.11e-04]
Epoch 10 TRAIN:  16%|█▌        | 21/133 [00:06<00:24,  4.60it/s, epoch=10, grad_norm=157.247, loss=310.3626, lr=1.11e-04]
Epoch 10 TRAIN:  16%|█▌        | 21/133 [00:06<00:24,  4.60it/s, epoch=10, grad_norm=58.514, loss=387.3819, lr=1.11e-04] 
Epoch 10 TRAIN:  16%|█▌        | 21/133 [00:07<00:24,  4.60it/s, epoch=10, grad_norm=56.454, loss=353.5117, lr=1.11e-04]
Epoch 10 TRAIN:  16%|█▌        | 21/133 [00:07<00:24,  4.60it/s, epoch=10, grad_norm=193.881, loss=297.6064, lr=1.11e-04]
Epoch 10 TRAIN:  19%|█▉        | 25/133 [00:07<00:21,  5.10it/s, epoch=10, grad_norm=193.881, loss=297.6064, lr=1.11e-04]
Epoch 10 TRAIN:  19%|█▉        | 25/133 [00:07<00:21,  5.10it/s, epoch=10, grad_norm=71.907, loss=380.4429, lr=1.11e-04] 
Epoch 10 TRAIN:  19%|█▉        | 25/133 [00:07<00:21,  5.10it/s, epoch=10, grad_norm=83.053, loss=327.2148, lr=1.11e-04]
Epoch 10 TRAIN:  19%|█▉        | 25/133 [00:07<00:21,  5.10it/s, epoch=10, grad_norm=96.386, loss=299.0868, lr=1.11e-04]
Epoch 10 TRAIN:  19%|█▉        | 25/133 [00:07<00:21,  5.10it/s, epoch=10, grad_norm=63.675, loss=367.4121, lr=1.11e-04]
Epoch 10 TRAIN:  22%|██▏       | 29/133 [00:07<00:18,  5.50it/s, epoch=10, grad_norm=63.675, loss=367.4121, lr=1.11e-04]
Epoch 10 TRAIN:  22%|██▏       | 29/133 [00:07<00:18,  5.50it/s, epoch=10, grad_norm=50.130, loss=332.6740, lr=1.11e-04]
Epoch 10 TRAIN:  22%|██▏       | 29/133 [00:08<00:18,  5.50it/s, epoch=10, grad_norm=47.455, loss=321.2639, lr=1.11e-04]
Epoch 10 TRAIN:  22%|██▏       | 29/133 [00:08<00:18,  5.50it/s, epoch=10, grad_norm=58.619, loss=339.9062, lr=1.11e-04]
Epoch 10 TRAIN:  22%|██▏       | 29/133 [00:08<00:18,  5.50it/s, epoch=10, grad_norm=74.362, loss=351.2918, lr=1.12e-04]
Epoch 10 TRAIN:  25%|██▍       | 33/133 [00:08<00:17,  5.83it/s, epoch=10, grad_norm=74.362, loss=351.2918, lr=1.12e-04]
Epoch 10 TRAIN:  25%|██▍       | 33/133 [00:08<00:17,  5.83it/s, epoch=10, grad_norm=97.237, loss=394.2203, lr=1.12e-04]
Epoch 10 TRAIN:  25%|██▍       | 33/133 [00:08<00:17,  5.83it/s, epoch=10, grad_norm=64.750, loss=393.5957, lr=1.12e-04]
Epoch 10 TRAIN:  25%|██▍       | 33/133 [00:08<00:17,  5.83it/s, epoch=10, grad_norm=159.557, loss=277.0735, lr=1.12e-04]
Epoch 10 TRAIN:  25%|██▍       | 33/133 [00:08<00:17,  5.83it/s, epoch=10, grad_norm=84.047, loss=373.4622, lr=1.12e-04] 
Epoch 10 TRAIN:  28%|██▊       | 37/133 [00:08<00:15,  6.07it/s, epoch=10, grad_norm=84.047, loss=373.4622, lr=1.12e-04]
Epoch 10 TRAIN:  28%|██▊       | 37/133 [00:09<00:15,  6.07it/s, epoch=10, grad_norm=67.534, loss=365.6588, lr=1.12e-04]
Epoch 10 TRAIN:  28%|██▊       | 37/133 [00:09<00:15,  6.07it/s, epoch=10, grad_norm=73.985, loss=339.0203, lr=1.12e-04]
Epoch 10 TRAIN:  28%|██▊       | 37/133 [00:09<00:15,  6.07it/s, epoch=10, grad_norm=87.274, loss=304.5038, lr=1.12e-04]
Epoch 10 TRAIN:  28%|██▊       | 37/133 [00:09<00:15,  6.07it/s, epoch=10, grad_norm=63.251, loss=324.2356, lr=1.12e-04]
Epoch 10 TRAIN:  31%|███       | 41/133 [00:09<00:14,  6.20it/s, epoch=10, grad_norm=63.251, loss=324.2356, lr=1.12e-04]
Epoch 10 TRAIN:  31%|███       | 41/133 [00:09<00:14,  6.20it/s, epoch=10, grad_norm=57.660, loss=292.0236, lr=1.12e-04]
Epoch 10 TRAIN:  31%|███       | 41/133 [00:09<00:14,  6.20it/s, epoch=10, grad_norm=48.538, loss=296.7581, lr=1.12e-04]
Epoch 10 TRAIN:  31%|███       | 41/133 [00:10<00:14,  6.20it/s, epoch=10, grad_norm=50.755, loss=263.5891, lr=1.12e-04]
Epoch 10 TRAIN:  31%|███       | 41/133 [00:10<00:14,  6.20it/s, epoch=10, grad_norm=107.448, loss=341.0161, lr=1.12e-04]
Epoch 10 TRAIN:  34%|███▍      | 45/133 [00:10<00:14,  6.24it/s, epoch=10, grad_norm=107.448, loss=341.0161, lr=1.12e-04]
Epoch 10 TRAIN:  34%|███▍      | 45/133 [00:10<00:14,  6.24it/s, epoch=10, grad_norm=52.498, loss=294.0508, lr=1.13e-04] 
Epoch 10 TRAIN:  34%|███▍      | 45/133 [00:10<00:14,  6.24it/s, epoch=10, grad_norm=109.231, loss=355.2562, lr=1.13e-04]
Epoch 10 TRAIN:  34%|███▍      | 45/133 [00:10<00:14,  6.24it/s, epoch=10, grad_norm=60.607, loss=293.2693, lr=1.13e-04] 
Epoch 10 TRAIN:  34%|███▍      | 45/133 [00:10<00:14,  6.24it/s, epoch=10, grad_norm=64.608, loss=318.9565, lr=1.13e-04]
Epoch 10 TRAIN:  37%|███▋      | 49/133 [00:10<00:13,  6.21it/s, epoch=10, grad_norm=64.608, loss=318.9565, lr=1.13e-04]
Epoch 10 TRAIN:  37%|███▋      | 49/133 [00:11<00:13,  6.21it/s, epoch=10, grad_norm=63.928, loss=302.7873, lr=1.13e-04]
Epoch 10 TRAIN:  37%|███▋      | 49/133 [00:11<00:13,  6.21it/s, epoch=10, grad_norm=60.737, loss=347.8912, lr=1.13e-04]
Epoch 10 TRAIN:  37%|███▋      | 49/133 [00:11<00:13,  6.21it/s, epoch=10, grad_norm=86.379, loss=355.2868, lr=1.13e-04]
Epoch 10 TRAIN:  37%|███▋      | 49/133 [00:11<00:13,  6.21it/s, epoch=10, grad_norm=118.099, loss=286.3323, lr=1.13e-04]
Epoch 10 TRAIN:  40%|███▉      | 53/133 [00:11<00:12,  6.27it/s, epoch=10, grad_norm=118.099, loss=286.3323, lr=1.13e-04]
Epoch 10 TRAIN:  40%|███▉      | 53/133 [00:11<00:12,  6.27it/s, epoch=10, grad_norm=145.722, loss=290.5498, lr=1.13e-04]
Epoch 10 TRAIN:  40%|███▉      | 53/133 [00:11<00:12,  6.27it/s, epoch=10, grad_norm=60.862, loss=342.2010, lr=1.13e-04] 
Epoch 10 TRAIN:  40%|███▉      | 53/133 [00:11<00:12,  6.27it/s, epoch=10, grad_norm=111.081, loss=356.5418, lr=1.13e-04]
Epoch 10 TRAIN:  40%|███▉      | 53/133 [00:12<00:12,  6.27it/s, epoch=10, grad_norm=99.533, loss=374.0601, lr=1.13e-04] 
Epoch 10 TRAIN:  43%|████▎     | 57/133 [00:12<00:12,  6.21it/s, epoch=10, grad_norm=99.533, loss=374.0601, lr=1.13e-04]
Epoch 10 TRAIN:  43%|████▎     | 57/133 [00:12<00:12,  6.21it/s, epoch=10, grad_norm=57.704, loss=315.8983, lr=1.14e-04]
Epoch 10 TRAIN:  43%|████▎     | 57/133 [00:12<00:12,  6.21it/s, epoch=10, grad_norm=163.618, loss=297.1639, lr=1.14e-04]
Epoch 10 TRAIN:  43%|████▎     | 57/133 [00:12<00:12,  6.21it/s, epoch=10, grad_norm=77.428, loss=344.1985, lr=1.14e-04] 
Epoch 10 TRAIN:  43%|████▎     | 57/133 [00:12<00:12,  6.21it/s, epoch=10, grad_norm=72.052, loss=342.4113, lr=1.14e-04]
Epoch 10 TRAIN:  46%|████▌     | 61/133 [00:12<00:11,  6.12it/s, epoch=10, grad_norm=72.052, loss=342.4113, lr=1.14e-04]
Epoch 10 TRAIN:  46%|████▌     | 61/133 [00:12<00:11,  6.12it/s, epoch=10, grad_norm=115.018, loss=322.6729, lr=1.14e-04]
Epoch 10 TRAIN:  46%|████▌     | 61/133 [00:13<00:11,  6.12it/s, epoch=10, grad_norm=57.457, loss=322.7480, lr=1.14e-04] 
Epoch 10 TRAIN:  46%|████▌     | 61/133 [00:13<00:11,  6.12it/s, epoch=10, grad_norm=73.287, loss=326.2512, lr=1.14e-04]
Epoch 10 TRAIN:  46%|████▌     | 61/133 [00:13<00:11,  6.12it/s, epoch=10, grad_norm=68.012, loss=341.1811, lr=1.14e-04]
Epoch 10 TRAIN:  49%|████▉     | 65/133 [00:13<00:11,  6.11it/s, epoch=10, grad_norm=68.012, loss=341.1811, lr=1.14e-04]
Epoch 10 TRAIN:  49%|████▉     | 65/133 [00:13<00:11,  6.11it/s, epoch=10, grad_norm=55.147, loss=314.4545, lr=1.14e-04]
Epoch 10 TRAIN:  49%|████▉     | 65/133 [00:13<00:11,  6.11it/s, epoch=10, grad_norm=68.366, loss=359.1516, lr=1.14e-04]
Epoch 10 TRAIN:  49%|████▉     | 65/133 [00:13<00:11,  6.11it/s, epoch=10, grad_norm=85.595, loss=355.1643, lr=1.14e-04]
Epoch 10 TRAIN:  49%|████▉     | 65/133 [00:14<00:11,  6.11it/s, epoch=10, grad_norm=87.021, loss=297.7330, lr=1.14e-04]
Epoch 10 TRAIN:  52%|█████▏    | 69/133 [00:14<00:10,  6.13it/s, epoch=10, grad_norm=87.021, loss=297.7330, lr=1.14e-04]
Epoch 10 TRAIN:  52%|█████▏    | 69/133 [00:14<00:10,  6.13it/s, epoch=10, grad_norm=86.224, loss=372.2506, lr=1.14e-04]
Epoch 10 TRAIN:  52%|█████▏    | 69/133 [00:14<00:10,  6.13it/s, epoch=10, grad_norm=75.878, loss=340.8703, lr=1.15e-04]
Epoch 10 TRAIN:  52%|█████▏    | 69/133 [00:14<00:10,  6.13it/s, epoch=10, grad_norm=124.076, loss=357.5616, lr=1.15e-04]
Epoch 10 TRAIN:  52%|█████▏    | 69/133 [00:14<00:10,  6.13it/s, epoch=10, grad_norm=59.874, loss=334.5989, lr=1.15e-04] 
Epoch 10 TRAIN:  55%|█████▍    | 73/133 [00:14<00:09,  6.00it/s, epoch=10, grad_norm=59.874, loss=334.5989, lr=1.15e-04]
Epoch 10 TRAIN:  55%|█████▍    | 73/133 [00:15<00:09,  6.00it/s, epoch=10, grad_norm=67.531, loss=328.6622, lr=1.15e-04]
Epoch 10 TRAIN:  55%|█████▍    | 73/133 [00:15<00:09,  6.00it/s, epoch=10, grad_norm=135.049, loss=330.8191, lr=1.15e-04]
Epoch 10 TRAIN:  55%|█████▍    | 73/133 [00:15<00:09,  6.00it/s, epoch=10, grad_norm=133.259, loss=270.2946, lr=1.15e-04]
Epoch 10 TRAIN:  55%|█████▍    | 73/133 [00:15<00:09,  6.00it/s, epoch=10, grad_norm=69.004, loss=341.6284, lr=1.15e-04] 
Epoch 10 TRAIN:  58%|█████▊    | 77/133 [00:15<00:09,  5.93it/s, epoch=10, grad_norm=69.004, loss=341.6284, lr=1.15e-04]
Epoch 10 TRAIN:  58%|█████▊    | 77/133 [00:15<00:09,  5.93it/s, epoch=10, grad_norm=126.560, loss=350.1136, lr=1.15e-04]
Epoch 10 TRAIN:  58%|█████▊    | 77/133 [00:15<00:09,  5.93it/s, epoch=10, grad_norm=105.502, loss=362.5159, lr=1.15e-04]
Epoch 10 TRAIN:  58%|█████▊    | 77/133 [00:16<00:09,  5.93it/s, epoch=10, grad_norm=62.716, loss=280.4492, lr=1.15e-04] 
Epoch 10 TRAIN:  60%|██████    | 80/133 [00:16<00:09,  5.76it/s, epoch=10, grad_norm=62.716, loss=280.4492, lr=1.15e-04]
Epoch 10 TRAIN:  60%|██████    | 80/133 [00:16<00:09,  5.76it/s, epoch=10, grad_norm=71.592, loss=315.0738, lr=1.15e-04]
Epoch 10 TRAIN:  60%|██████    | 80/133 [00:16<00:09,  5.76it/s, epoch=10, grad_norm=73.493, loss=299.0363, lr=1.15e-04]
Epoch 10 TRAIN:  60%|██████    | 80/133 [00:16<00:09,  5.76it/s, epoch=10, grad_norm=107.526, loss=313.5746, lr=1.16e-04]
Epoch 10 TRAIN:  60%|██████    | 80/133 [00:16<00:09,  5.76it/s, epoch=10, grad_norm=63.809, loss=299.3468, lr=1.16e-04] 
Epoch 10 TRAIN:  63%|██████▎   | 84/133 [00:16<00:08,  5.92it/s, epoch=10, grad_norm=63.809, loss=299.3468, lr=1.16e-04]
Epoch 10 TRAIN:  63%|██████▎   | 84/133 [00:16<00:08,  5.92it/s, epoch=10, grad_norm=77.422, loss=332.8777, lr=1.16e-04]
Epoch 10 TRAIN:  63%|██████▎   | 84/133 [00:17<00:08,  5.92it/s, epoch=10, grad_norm=151.669, loss=289.5787, lr=1.16e-04]
Epoch 10 TRAIN:  63%|██████▎   | 84/133 [00:17<00:08,  5.92it/s, epoch=10, grad_norm=125.267, loss=400.2603, lr=1.16e-04]
Epoch 10 TRAIN:  65%|██████▌   | 87/133 [00:17<00:08,  5.64it/s, epoch=10, grad_norm=125.267, loss=400.2603, lr=1.16e-04]
Epoch 10 TRAIN:  65%|██████▌   | 87/133 [00:17<00:08,  5.64it/s, epoch=10, grad_norm=164.025, loss=364.5620, lr=1.16e-04]
Epoch 10 TRAIN:  65%|██████▌   | 87/133 [00:17<00:08,  5.64it/s, epoch=10, grad_norm=129.221, loss=257.4208, lr=1.16e-04]
Epoch 10 TRAIN:  65%|██████▌   | 87/133 [00:17<00:08,  5.64it/s, epoch=10, grad_norm=131.512, loss=324.3586, lr=1.16e-04]
Epoch 10 TRAIN:  68%|██████▊   | 90/133 [00:17<00:07,  5.66it/s, epoch=10, grad_norm=131.512, loss=324.3586, lr=1.16e-04]
Epoch 10 TRAIN:  68%|██████▊   | 90/133 [00:18<00:07,  5.66it/s, epoch=10, grad_norm=202.806, loss=292.5639, lr=1.16e-04]
Epoch 10 TRAIN:  68%|██████▊   | 90/133 [00:18<00:07,  5.66it/s, epoch=10, grad_norm=58.273, loss=298.6747, lr=1.16e-04] 
Epoch 10 TRAIN:  68%|██████▊   | 90/133 [00:18<00:07,  5.66it/s, epoch=10, grad_norm=93.254, loss=340.6505, lr=1.16e-04]
Epoch 10 TRAIN:  70%|██████▉   | 93/133 [00:18<00:07,  5.71it/s, epoch=10, grad_norm=93.254, loss=340.6505, lr=1.16e-04]
Epoch 10 TRAIN:  70%|██████▉   | 93/133 [00:18<00:07,  5.71it/s, epoch=10, grad_norm=83.838, loss=323.8593, lr=1.16e-04]
Epoch 10 TRAIN:  70%|██████▉   | 93/133 [00:18<00:07,  5.71it/s, epoch=10, grad_norm=118.683, loss=276.5597, lr=1.16e-04]
Epoch 10 TRAIN:  70%|██████▉   | 93/133 [00:18<00:07,  5.71it/s, epoch=10, grad_norm=63.528, loss=279.3894, lr=1.17e-04] 
Epoch 10 TRAIN:  72%|███████▏  | 96/133 [00:18<00:06,  5.48it/s, epoch=10, grad_norm=63.528, loss=279.3894, lr=1.17e-04]
Epoch 10 TRAIN:  72%|███████▏  | 96/133 [00:19<00:06,  5.48it/s, epoch=10, grad_norm=110.554, loss=274.2939, lr=1.17e-04]
Epoch 10 TRAIN:  72%|███████▏  | 96/133 [00:19<00:06,  5.48it/s, epoch=10, grad_norm=80.793, loss=254.8205, lr=1.17e-04] 
Epoch 10 TRAIN:  72%|███████▏  | 96/133 [00:19<00:06,  5.48it/s, epoch=10, grad_norm=196.252, loss=387.5804, lr=1.17e-04]
Epoch 10 TRAIN:  72%|███████▏  | 96/133 [00:19<00:06,  5.48it/s, epoch=10, grad_norm=219.026, loss=351.6993, lr=1.17e-04]2026-04-16 19:57:09,727 DEBUG TRAIN | steps/sec 5.040| Batch 10/100 loss 351.699341 loss_att 311.232025 loss_ctc 446.123077 th_accuracy 0.252046 lr 1.1688e-04 grad_norm 219.026367 rank 0

Epoch 10 TRAIN:  75%|███████▌  | 100/133 [00:19<00:05,  5.60it/s, epoch=10, grad_norm=219.026, loss=351.6993, lr=1.17e-04]
Epoch 10 TRAIN:  75%|███████▌  | 100/133 [00:19<00:05,  5.60it/s, epoch=10, grad_norm=161.467, loss=331.1387, lr=1.17e-04]
Epoch 10 TRAIN:  75%|███████▌  | 100/133 [00:20<00:05,  5.60it/s, epoch=10, grad_norm=111.229, loss=390.5512, lr=1.17e-04]
Epoch 10 TRAIN:  75%|███████▌  | 100/133 [00:20<00:05,  5.60it/s, epoch=10, grad_norm=138.383, loss=334.6245, lr=1.17e-04]
Epoch 10 TRAIN:  77%|███████▋  | 103/133 [00:20<00:05,  5.61it/s, epoch=10, grad_norm=138.383, loss=334.6245, lr=1.17e-04]
Epoch 10 TRAIN:  77%|███████▋  | 103/133 [00:20<00:05,  5.61it/s, epoch=10, grad_norm=184.997, loss=235.1402, lr=1.17e-04]
Epoch 10 TRAIN:  77%|███████▋  | 103/133 [00:20<00:05,  5.61it/s, epoch=10, grad_norm=163.321, loss=248.6643, lr=1.17e-04]
Epoch 10 TRAIN:  77%|███████▋  | 103/133 [00:20<00:05,  5.61it/s, epoch=10, grad_norm=136.578, loss=343.4856, lr=1.17e-04]
Epoch 10 TRAIN:  77%|███████▋  | 103/133 [00:20<00:05,  5.61it/s, epoch=10, grad_norm=141.306, loss=338.4933, lr=1.17e-04]
Epoch 10 TRAIN:  80%|████████  | 107/133 [00:20<00:04,  5.76it/s, epoch=10, grad_norm=141.306, loss=338.4933, lr=1.17e-04]
Epoch 10 TRAIN:  80%|████████  | 107/133 [00:21<00:04,  5.76it/s, epoch=10, grad_norm=141.653, loss=347.3163, lr=1.18e-04]
Epoch 10 TRAIN:  80%|████████  | 107/133 [00:21<00:04,  5.76it/s, epoch=10, grad_norm=91.418, loss=265.8680, lr=1.18e-04] 
Epoch 10 TRAIN:  80%|████████  | 107/133 [00:21<00:04,  5.76it/s, epoch=10, grad_norm=74.704, loss=327.1675, lr=1.18e-04]
Epoch 10 TRAIN:  83%|████████▎ | 110/133 [00:21<00:03,  5.77it/s, epoch=10, grad_norm=74.704, loss=327.1675, lr=1.18e-04]
Epoch 10 TRAIN:  83%|████████▎ | 110/133 [00:21<00:03,  5.77it/s, epoch=10, grad_norm=127.273, loss=322.9361, lr=1.18e-04]
Epoch 10 TRAIN:  83%|████████▎ | 110/133 [00:21<00:03,  5.77it/s, epoch=10, grad_norm=109.889, loss=256.3398, lr=1.18e-04]
Epoch 10 TRAIN:  83%|████████▎ | 110/133 [00:21<00:03,  5.77it/s, epoch=10, grad_norm=78.055, loss=304.0599, lr=1.18e-04] 
Epoch 10 TRAIN:  85%|████████▍ | 113/133 [00:21<00:03,  5.47it/s, epoch=10, grad_norm=78.055, loss=304.0599, lr=1.18e-04]
Epoch 10 TRAIN:  85%|████████▍ | 113/133 [00:22<00:03,  5.47it/s, epoch=10, grad_norm=99.130, loss=298.8554, lr=1.18e-04]
Epoch 10 TRAIN:  85%|████████▍ | 113/133 [00:22<00:03,  5.47it/s, epoch=10, grad_norm=111.266, loss=291.9951, lr=1.18e-04]
Epoch 10 TRAIN:  85%|████████▍ | 113/133 [00:22<00:03,  5.47it/s, epoch=10, grad_norm=62.553, loss=268.8458, lr=1.18e-04] 
Epoch 10 TRAIN:  87%|████████▋ | 116/133 [00:22<00:03,  5.46it/s, epoch=10, grad_norm=62.553, loss=268.8458, lr=1.18e-04]
Epoch 10 TRAIN:  87%|████████▋ | 116/133 [00:22<00:03,  5.46it/s, epoch=10, grad_norm=77.517, loss=280.8455, lr=1.18e-04]
Epoch 10 TRAIN:  87%|████████▋ | 116/133 [00:22<00:03,  5.46it/s, epoch=10, grad_norm=137.203, loss=354.4154, lr=1.18e-04]
Epoch 10 TRAIN:  87%|████████▋ | 116/133 [00:23<00:03,  5.46it/s, epoch=10, grad_norm=281.640, loss=426.8464, lr=1.18e-04]
Epoch 10 TRAIN:  87%|████████▋ | 116/133 [00:23<00:03,  5.46it/s, epoch=10, grad_norm=131.144, loss=329.6373, lr=1.18e-04]
Epoch 10 TRAIN:  90%|█████████ | 120/133 [00:23<00:02,  5.76it/s, epoch=10, grad_norm=131.144, loss=329.6373, lr=1.18e-04]
Epoch 10 TRAIN:  90%|█████████ | 120/133 [00:23<00:02,  5.76it/s, epoch=10, grad_norm=114.113, loss=277.9429, lr=1.19e-04]
Epoch 10 TRAIN:  90%|█████████ | 120/133 [00:23<00:02,  5.76it/s, epoch=10, grad_norm=91.693, loss=348.2118, lr=1.19e-04] 
Epoch 10 TRAIN:  90%|█████████ | 120/133 [00:23<00:02,  5.76it/s, epoch=10, grad_norm=126.092, loss=390.2004, lr=1.19e-04]
Epoch 10 TRAIN:  90%|█████████ | 120/133 [00:23<00:02,  5.76it/s, epoch=10, grad_norm=135.453, loss=374.7435, lr=1.19e-04]
Epoch 10 TRAIN:  93%|█████████▎| 124/133 [00:23<00:01,  6.00it/s, epoch=10, grad_norm=135.453, loss=374.7435, lr=1.19e-04]
Epoch 10 TRAIN:  93%|█████████▎| 124/133 [00:23<00:01,  6.00it/s, epoch=10, grad_norm=106.506, loss=254.8194, lr=1.19e-04]
Epoch 10 TRAIN:  93%|█████████▎| 124/133 [00:24<00:01,  6.00it/s, epoch=10, grad_norm=127.914, loss=429.5862, lr=1.19e-04]
Epoch 10 TRAIN:  93%|█████████▎| 124/133 [00:24<00:01,  6.00it/s, epoch=10, grad_norm=186.041, loss=327.0988, lr=1.19e-04]
Epoch 10 TRAIN:  93%|█████████▎| 124/133 [00:24<00:01,  6.00it/s, epoch=10, grad_norm=111.351, loss=305.4189, lr=1.19e-04]
Epoch 10 TRAIN:  96%|█████████▌| 128/133 [00:24<00:00,  6.09it/s, epoch=10, grad_norm=111.351, loss=305.4189, lr=1.19e-04]
Epoch 10 TRAIN:  96%|█████████▌| 128/133 [00:24<00:00,  6.09it/s, epoch=10, grad_norm=160.044, loss=263.6974, lr=1.19e-04]
Epoch 10 TRAIN:  96%|█████████▌| 128/133 [00:24<00:00,  6.09it/s, epoch=10, grad_norm=98.206, loss=317.7144, lr=1.19e-04] 
Epoch 10 TRAIN:  96%|█████████▌| 128/133 [00:24<00:00,  6.09it/s, epoch=10, grad_norm=195.998, loss=378.2422, lr=1.19e-04]
Epoch 10 TRAIN:  96%|█████████▌| 128/133 [00:25<00:00,  6.09it/s, epoch=10, grad_norm=140.434, loss=385.1695, lr=1.19e-04]
Epoch 10 TRAIN:  99%|█████████▉| 132/133 [00:25<00:00,  6.07it/s, epoch=10, grad_norm=140.434, loss=385.1695, lr=1.19e-04]
Epoch 10 TRAIN:  99%|█████████▉| 132/133 [00:25<00:00,  6.07it/s, epoch=10, grad_norm=166.623, loss=400.1276, lr=1.20e-04]
Epoch 10 TRAIN:  99%|█████████▉| 132/133 [00:25<00:00,  6.07it/s, epoch=10, grad_norm=114.453, loss=299.1165, lr=1.20e-04]
Epoch 10 TRAIN:  99%|█████████▉| 132/133 [00:25<00:00,  6.07it/s, epoch=10, grad_norm=148.277, loss=425.8273, lr=1.20e-04]
Epoch 10 TRAIN: 135it [00:25,  5.28it/s, epoch=10, grad_norm=148.277, loss=425.8273, lr=1.20e-04]                         

CV:   0%|          | 0/6 [00:00<?, ?it/s]/home/rookie/Documents/batuka/projects/wenet/wenet/wenet/utils/train_utils.py:689: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  amp_autocast(enabled=dtype is not None,

CV:  17%|█▋        | 1/6 [00:00<00:03,  1.33it/s]
                                                 
2026-04-16 19:57:16,475 INFO Epoch 10 Step 1496 CV info lr 1.1968e-04 cv_loss 379.40226606889206 rank 0 acc 0.2472728987534841
2026-04-16 19:57:16,476 INFO [Rank 0] Checkpoint: save to checkpoint exp/conformer_mongolian/epoch_10.pt
2026-04-16 19:57:16,587 INFO Epoch 11 Step 1496 TRAIN info lr 1.1968e-04 rank 0
2026-04-16 19:57:16,607 INFO using accumulate grad, new batch size is 1 times larger than before

Epoch 11 TRAIN:   0%|          | 0/135 [00:00<?, ?it/s]
Epoch 11 TRAIN:   0%|          | 0/135 [00:04<?, ?it/s, epoch=11, grad_norm=299.014, loss=310.8687, lr=1.20e-04]
Epoch 11 TRAIN:   1%|          | 1/135 [00:04<09:01,  4.04s/it, epoch=11, grad_norm=299.014, loss=310.8687, lr=1.20e-04]
Epoch 11 TRAIN:   1%|          | 1/135 [00:04<09:01,  4.04s/it, epoch=11, grad_norm=432.422, loss=410.1504, lr=1.20e-04]
Epoch 11 TRAIN:   1%|          | 1/135 [00:04<09:01,  4.04s/it, epoch=11, grad_norm=320.846, loss=353.9737, lr=1.20e-04]
Epoch 11 TRAIN:   1%|          | 1/135 [00:04<09:01,  4.04s/it, epoch=11, grad_norm=270.030, loss=340.6840, lr=1.20e-04]
Epoch 11 TRAIN:   1%|          | 1/135 [00:04<09:01,  4.04s/it, epoch=11, grad_norm=88.401, loss=246.8023, lr=1.20e-04] 
Epoch 11 TRAIN:   1%|          | 1/135 [00:04<09:01,  4.04s/it, epoch=11, grad_norm=226.640, loss=364.5148, lr=1.20e-04]
Epoch 11 TRAIN:   1%|          | 1/135 [00:04<09:01,  4.04s/it, epoch=11, grad_norm=104.437, loss=311.6731, lr=1.20e-04]
Epoch 11 TRAIN:   1%|          | 1/135 [00:04<09:01,  4.04s/it, epoch=11, grad_norm=115.897, loss=372.5482, lr=1.20e-04]
Epoch 11 TRAIN:   6%|▌         | 8/135 [00:04<00:55,  2.30it/s, epoch=11, grad_norm=115.897, loss=372.5482, lr=1.20e-04]
Epoch 11 TRAIN:   6%|▌         | 8/135 [00:04<00:55,  2.30it/s, epoch=11, grad_norm=70.953, loss=355.5790, lr=1.20e-04] 
Epoch 11 TRAIN:   6%|▌         | 8/135 [00:04<00:55,  2.30it/s, epoch=11, grad_norm=66.111, loss=356.0335, lr=1.20e-04]
Epoch 11 TRAIN:   6%|▌         | 8/135 [00:05<00:55,  2.30it/s, epoch=11, grad_norm=73.415, loss=354.2159, lr=1.21e-04]
Epoch 11 TRAIN:   6%|▌         | 8/135 [00:05<00:55,  2.30it/s, epoch=11, grad_norm=88.923, loss=332.9251, lr=1.21e-04]
Epoch 11 TRAIN:   9%|▉         | 12/135 [00:05<00:38,  3.17it/s, epoch=11, grad_norm=88.923, loss=332.9251, lr=1.21e-04]
Epoch 11 TRAIN:   9%|▉         | 12/135 [00:05<00:38,  3.17it/s, epoch=11, grad_norm=173.499, loss=296.8066, lr=1.21e-04]
Epoch 11 TRAIN:   9%|▉         | 12/135 [00:05<00:38,  3.17it/s, epoch=11, grad_norm=68.466, loss=325.6021, lr=1.21e-04] 
Epoch 11 TRAIN:   9%|▉         | 12/135 [00:05<00:38,  3.17it/s, epoch=11, grad_norm=52.672, loss=325.6021, lr=1.21e-04]
Epoch 11 TRAIN:   9%|▉         | 12/135 [00:05<00:38,  3.17it/s, epoch=11, grad_norm=57.909, loss=310.4795, lr=1.21e-04]
Epoch 11 TRAIN:  12%|█▏        | 16/135 [00:05<00:30,  3.92it/s, epoch=11, grad_norm=57.909, loss=310.4795, lr=1.21e-04]
Epoch 11 TRAIN:  12%|█▏        | 16/135 [00:05<00:30,  3.92it/s, epoch=11, grad_norm=55.216, loss=300.5232, lr=1.21e-04]
Epoch 11 TRAIN:  12%|█▏        | 16/135 [00:06<00:30,  3.92it/s, epoch=11, grad_norm=66.105, loss=307.0394, lr=1.21e-04]
Epoch 11 TRAIN:  12%|█▏        | 16/135 [00:06<00:30,  3.92it/s, epoch=11, grad_norm=86.496, loss=327.6031, lr=1.21e-04]
Epoch 11 TRAIN:  12%|█▏        | 16/135 [00:06<00:30,  3.92it/s, epoch=11, grad_norm=45.626, loss=321.3894, lr=1.21e-04]
Epoch 11 TRAIN:  15%|█▍        | 20/135 [00:06<00:25,  4.56it/s, epoch=11, grad_norm=45.626, loss=321.3894, lr=1.21e-04]
Epoch 11 TRAIN:  15%|█▍        | 20/135 [00:06<00:25,  4.56it/s, epoch=11, grad_norm=70.795, loss=330.8865, lr=1.21e-04]
Epoch 11 TRAIN:  15%|█▍        | 20/135 [00:06<00:25,  4.56it/s, epoch=11, grad_norm=53.326, loss=329.2733, lr=1.21e-04]
Epoch 11 TRAIN:  15%|█▍        | 20/135 [00:06<00:25,  4.56it/s, epoch=11, grad_norm=65.580, loss=357.5518, lr=1.22e-04]
Epoch 11 TRAIN:  15%|█▍        | 20/135 [00:07<00:25,  4.56it/s, epoch=11, grad_norm=50.849, loss=324.9995, lr=1.22e-04]
Epoch 11 TRAIN:  18%|█▊        | 24/135 [00:07<00:21,  5.06it/s, epoch=11, grad_norm=50.849, loss=324.9995, lr=1.22e-04]
Epoch 11 TRAIN:  18%|█▊        | 24/135 [00:07<00:21,  5.06it/s, epoch=11, grad_norm=55.629, loss=347.8055, lr=1.22e-04]
Epoch 11 TRAIN:  18%|█▊        | 24/135 [00:07<00:21,  5.06it/s, epoch=11, grad_norm=69.998, loss=377.0548, lr=1.22e-04]
Epoch 11 TRAIN:  18%|█▊        | 24/135 [00:07<00:21,  5.06it/s, epoch=11, grad_norm=77.142, loss=313.5501, lr=1.22e-04]
Epoch 11 TRAIN:  18%|█▊        | 24/135 [00:07<00:21,  5.06it/s, epoch=11, grad_norm=56.726, loss=341.8307, lr=1.22e-04]
Epoch 11 TRAIN:  21%|██        | 28/135 [00:07<00:19,  5.46it/s, epoch=11, grad_norm=56.726, loss=341.8307, lr=1.22e-04]
Epoch 11 TRAIN:  21%|██        | 28/135 [00:07<00:19,  5.46it/s, epoch=11, grad_norm=93.432, loss=321.4443, lr=1.22e-04]
Epoch 11 TRAIN:  21%|██        | 28/135 [00:07<00:19,  5.46it/s, epoch=11, grad_norm=91.796, loss=296.9868, lr=1.22e-04]
Epoch 11 TRAIN:  21%|██        | 28/135 [00:08<00:19,  5.46it/s, epoch=11, grad_norm=80.226, loss=359.3423, lr=1.22e-04]
Epoch 11 TRAIN:  21%|██        | 28/135 [00:08<00:19,  5.46it/s, epoch=11, grad_norm=67.236, loss=297.4881, lr=1.22e-04]
Epoch 11 TRAIN:  24%|██▎       | 32/135 [00:08<00:17,  5.82it/s, epoch=11, grad_norm=67.236, loss=297.4881, lr=1.22e-04]
Epoch 11 TRAIN:  24%|██▎       | 32/135 [00:08<00:17,  5.82it/s, epoch=11, grad_norm=66.259, loss=318.8530, lr=1.22e-04]
Epoch 11 TRAIN:  24%|██▎       | 32/135 [00:08<00:17,  5.82it/s, epoch=11, grad_norm=108.866, loss=363.7869, lr=1.22e-04]
Epoch 11 TRAIN:  24%|██▎       | 32/135 [00:08<00:17,  5.82it/s, epoch=11, grad_norm=65.248, loss=309.6073, lr=1.22e-04] 
Epoch 11 TRAIN:  24%|██▎       | 32/135 [00:08<00:17,  5.82it/s, epoch=11, grad_norm=117.321, loss=383.0731, lr=1.23e-04]
Epoch 11 TRAIN:  27%|██▋       | 36/135 [00:08<00:16,  6.04it/s, epoch=11, grad_norm=117.321, loss=383.0731, lr=1.23e-04]
Epoch 11 TRAIN:  27%|██▋       | 36/135 [00:08<00:16,  6.04it/s, epoch=11, grad_norm=69.498, loss=364.9595, lr=1.23e-04] 
Epoch 11 TRAIN:  27%|██▋       | 36/135 [00:09<00:16,  6.04it/s, epoch=11, grad_norm=115.013, loss=336.9662, lr=1.23e-04]
Epoch 11 TRAIN:  27%|██▋       | 36/135 [00:09<00:16,  6.04it/s, epoch=11, grad_norm=136.965, loss=313.8151, lr=1.23e-04]
Epoch 11 TRAIN:  27%|██▋       | 36/135 [00:09<00:16,  6.04it/s, epoch=11, grad_norm=91.154, loss=348.2044, lr=1.23e-04] 
Epoch 11 TRAIN:  30%|██▉       | 40/135 [00:09<00:15,  6.05it/s, epoch=11, grad_norm=91.154, loss=348.2044, lr=1.23e-04]
Epoch 11 TRAIN:  30%|██▉       | 40/135 [00:09<00:15,  6.05it/s, epoch=11, grad_norm=73.163, loss=335.8563, lr=1.23e-04]
Epoch 11 TRAIN:  30%|██▉       | 40/135 [00:09<00:15,  6.05it/s, epoch=11, grad_norm=62.964, loss=310.1472, lr=1.23e-04]
Epoch 11 TRAIN:  30%|██▉       | 40/135 [00:10<00:15,  6.05it/s, epoch=11, grad_norm=54.478, loss=294.0907, lr=1.23e-04]
Epoch 11 TRAIN:  30%|██▉       | 40/135 [00:10<00:15,  6.05it/s, epoch=11, grad_norm=120.116, loss=344.1424, lr=1.23e-04]
Epoch 11 TRAIN:  33%|███▎      | 44/135 [00:10<00:15,  6.02it/s, epoch=11, grad_norm=120.116, loss=344.1424, lr=1.23e-04]
Epoch 11 TRAIN:  33%|███▎      | 44/135 [00:10<00:15,  6.02it/s, epoch=11, grad_norm=149.129, loss=364.0149, lr=1.23e-04]
Epoch 11 TRAIN:  33%|███▎      | 44/135 [00:10<00:15,  6.02it/s, epoch=11, grad_norm=56.629, loss=306.1457, lr=1.23e-04] 
Epoch 11 TRAIN:  33%|███▎      | 44/135 [00:10<00:15,  6.02it/s, epoch=11, grad_norm=74.635, loss=338.4691, lr=1.23e-04]
Epoch 11 TRAIN:  33%|███▎      | 44/135 [00:10<00:15,  6.02it/s, epoch=11, grad_norm=90.298, loss=369.5996, lr=1.24e-04]
Epoch 11 TRAIN:  36%|███▌      | 48/135 [00:10<00:14,  5.95it/s, epoch=11, grad_norm=90.298, loss=369.5996, lr=1.24e-04]
Epoch 11 TRAIN:  36%|███▌      | 48/135 [00:11<00:14,  5.95it/s, epoch=11, grad_norm=59.062, loss=346.7584, lr=1.24e-04]
Epoch 11 TRAIN:  36%|███▌      | 48/135 [00:11<00:14,  5.95it/s, epoch=11, grad_norm=129.941, loss=304.6230, lr=1.24e-04]
Epoch 11 TRAIN:  36%|███▌      | 48/135 [00:11<00:14,  5.95it/s, epoch=11, grad_norm=71.138, loss=347.1699, lr=1.24e-04] 
Epoch 11 TRAIN:  36%|███▌      | 48/135 [00:11<00:14,  5.95it/s, epoch=11, grad_norm=52.002, loss=361.3456, lr=1.24e-04]
Epoch 11 TRAIN:  39%|███▊      | 52/135 [00:11<00:14,  5.90it/s, epoch=11, grad_norm=52.002, loss=361.3456, lr=1.24e-04]
Epoch 11 TRAIN:  39%|███▊      | 52/135 [00:11<00:14,  5.90it/s, epoch=11, grad_norm=73.028, loss=327.8536, lr=1.24e-04]
Epoch 11 TRAIN:  39%|███▊      | 52/135 [00:11<00:14,  5.90it/s, epoch=11, grad_norm=180.704, loss=279.8495, lr=1.24e-04]
Epoch 11 TRAIN:  39%|███▊      | 52/135 [00:12<00:14,  5.90it/s, epoch=11, grad_norm=98.651, loss=305.5377, lr=1.24e-04] 
Epoch 11 TRAIN:  39%|███▊      | 52/135 [00:12<00:14,  5.90it/s, epoch=11, grad_norm=76.806, loss=298.5131, lr=1.24e-04]
Epoch 11 TRAIN:  41%|████▏     | 56/135 [00:12<00:13,  5.78it/s, epoch=11, grad_norm=76.806, loss=298.5131, lr=1.24e-04]
Epoch 11 TRAIN:  41%|████▏     | 56/135 [00:12<00:13,  5.78it/s, epoch=11, grad_norm=191.753, loss=384.7003, lr=1.24e-04]
Epoch 11 TRAIN:  41%|████▏     | 56/135 [00:12<00:13,  5.78it/s, epoch=11, grad_norm=115.369, loss=343.8792, lr=1.24e-04]
Epoch 11 TRAIN:  41%|████▏     | 56/135 [00:12<00:13,  5.78it/s, epoch=11, grad_norm=197.494, loss=372.5220, lr=1.24e-04]
Epoch 11 TRAIN:  44%|████▎     | 59/135 [00:12<00:13,  5.78it/s, epoch=11, grad_norm=197.494, loss=372.5220, lr=1.24e-04]
Epoch 11 TRAIN:  44%|████▎     | 59/135 [00:13<00:13,  5.78it/s, epoch=11, grad_norm=91.384, loss=313.3858, lr=1.24e-04] 
Epoch 11 TRAIN:  44%|████▎     | 59/135 [00:13<00:13,  5.78it/s, epoch=11, grad_norm=165.530, loss=374.9306, lr=1.25e-04]
Epoch 11 TRAIN:  44%|████▎     | 59/135 [00:13<00:13,  5.78it/s, epoch=11, grad_norm=98.646, loss=287.3412, lr=1.25e-04] 
Epoch 11 TRAIN:  46%|████▌     | 62/135 [00:13<00:12,  5.62it/s, epoch=11, grad_norm=98.646, loss=287.3412, lr=1.25e-04]
Epoch 11 TRAIN:  46%|████▌     | 62/135 [00:13<00:12,  5.62it/s, epoch=11, grad_norm=81.547, loss=329.5952, lr=1.25e-04]
Epoch 11 TRAIN:  46%|████▌     | 62/135 [00:13<00:12,  5.62it/s, epoch=11, grad_norm=63.984, loss=324.1617, lr=1.25e-04]
Epoch 11 TRAIN:  46%|████▌     | 62/135 [00:13<00:12,  5.62it/s, epoch=11, grad_norm=86.418, loss=359.8976, lr=1.25e-04]
Epoch 11 TRAIN:  46%|████▌     | 62/135 [00:14<00:12,  5.62it/s, epoch=11, grad_norm=119.052, loss=318.1553, lr=1.25e-04]
Epoch 11 TRAIN:  49%|████▉     | 66/135 [00:14<00:11,  5.85it/s, epoch=11, grad_norm=119.052, loss=318.1553, lr=1.25e-04]
Epoch 11 TRAIN:  49%|████▉     | 66/135 [00:14<00:11,  5.85it/s, epoch=11, grad_norm=84.491, loss=311.1151, lr=1.25e-04] 
Epoch 11 TRAIN:  49%|████▉     | 66/135 [00:14<00:11,  5.85it/s, epoch=11, grad_norm=53.204, loss=326.1044, lr=1.25e-04]
Epoch 11 TRAIN:  49%|████▉     | 66/135 [00:14<00:11,  5.85it/s, epoch=11, grad_norm=71.995, loss=299.4419, lr=1.25e-04]
Epoch 11 TRAIN:  51%|█████     | 69/135 [00:14<00:12,  5.49it/s, epoch=11, grad_norm=71.995, loss=299.4419, lr=1.25e-04]
Epoch 11 TRAIN:  51%|█████     | 69/135 [00:14<00:12,  5.49it/s, epoch=11, grad_norm=128.186, loss=270.4300, lr=1.25e-04]
Epoch 11 TRAIN:  51%|█████     | 69/135 [00:14<00:12,  5.49it/s, epoch=11, grad_norm=166.259, loss=389.5561, lr=1.25e-04]
Epoch 11 TRAIN:  51%|█████     | 69/135 [00:15<00:12,  5.49it/s, epoch=11, grad_norm=71.786, loss=266.8311, lr=1.25e-04] 
Epoch 11 TRAIN:  51%|█████     | 69/135 [00:15<00:12,  5.49it/s, epoch=11, grad_norm=146.064, loss=360.5806, lr=1.26e-04]
Epoch 11 TRAIN:  54%|█████▍    | 73/135 [00:15<00:10,  5.77it/s, epoch=11, grad_norm=146.064, loss=360.5806, lr=1.26e-04]
Epoch 11 TRAIN:  54%|█████▍    | 73/135 [00:15<00:10,  5.77it/s, epoch=11, grad_norm=53.951, loss=273.9579, lr=1.26e-04] 
Epoch 11 TRAIN:  54%|█████▍    | 73/135 [00:15<00:10,  5.77it/s, epoch=11, grad_norm=107.240, loss=322.9429, lr=1.26e-04]
Epoch 11 TRAIN:  54%|█████▍    | 73/135 [00:15<00:10,  5.77it/s, epoch=11, grad_norm=143.105, loss=368.6078, lr=1.26e-04]
Epoch 11 TRAIN:  56%|█████▋    | 76/135 [00:15<00:10,  5.61it/s, epoch=11, grad_norm=143.105, loss=368.6078, lr=1.26e-04]
Epoch 11 TRAIN:  56%|█████▋    | 76/135 [00:16<00:10,  5.61it/s, epoch=11, grad_norm=151.711, loss=255.1037, lr=1.26e-04]
Epoch 11 TRAIN:  56%|█████▋    | 76/135 [00:16<00:10,  5.61it/s, epoch=11, grad_norm=88.263, loss=334.4461, lr=1.26e-04] 
Epoch 11 TRAIN:  56%|█████▋    | 76/135 [00:16<00:10,  5.61it/s, epoch=11, grad_norm=101.233, loss=391.6414, lr=1.26e-04]
Epoch 11 TRAIN:  59%|█████▊    | 79/135 [00:16<00:09,  5.70it/s, epoch=11, grad_norm=101.233, loss=391.6414, lr=1.26e-04]
Epoch 11 TRAIN:  59%|█████▊    | 79/135 [00:16<00:09,  5.70it/s, epoch=11, grad_norm=121.355, loss=322.0529, lr=1.26e-04]
Epoch 11 TRAIN:  59%|█████▊    | 79/135 [00:16<00:09,  5.70it/s, epoch=11, grad_norm=71.509, loss=312.3786, lr=1.26e-04] 
Epoch 11 TRAIN:  59%|█████▊    | 79/135 [00:16<00:09,  5.70it/s, epoch=11, grad_norm=103.405, loss=383.2057, lr=1.26e-04]
Epoch 11 TRAIN:  61%|██████    | 82/135 [00:16<00:09,  5.72it/s, epoch=11, grad_norm=103.405, loss=383.2057, lr=1.26e-04]
Epoch 11 TRAIN:  61%|██████    | 82/135 [00:17<00:09,  5.72it/s, epoch=11, grad_norm=62.722, loss=317.9256, lr=1.26e-04] 
Epoch 11 TRAIN:  61%|██████    | 82/135 [00:17<00:09,  5.72it/s, epoch=11, grad_norm=77.830, loss=326.8415, lr=1.26e-04]
Epoch 11 TRAIN:  61%|██████    | 82/135 [00:17<00:09,  5.72it/s, epoch=11, grad_norm=129.247, loss=305.9687, lr=1.26e-04]
Epoch 11 TRAIN:  63%|██████▎   | 85/135 [00:17<00:08,  5.57it/s, epoch=11, grad_norm=129.247, loss=305.9687, lr=1.26e-04]
Epoch 11 TRAIN:  63%|██████▎   | 85/135 [00:17<00:08,  5.57it/s, epoch=11, grad_norm=401.567, loss=208.5221, lr=1.27e-04]
Epoch 11 TRAIN:  63%|██████▎   | 85/135 [00:17<00:08,  5.57it/s, epoch=11, grad_norm=145.165, loss=297.3091, lr=1.27e-04]
Epoch 11 TRAIN:  63%|██████▎   | 85/135 [00:17<00:08,  5.57it/s, epoch=11, grad_norm=115.146, loss=398.7713, lr=1.27e-04]
Epoch 11 TRAIN:  63%|██████▎   | 85/135 [00:18<00:08,  5.57it/s, epoch=11, grad_norm=73.184, loss=301.7867, lr=1.27e-04] 
Epoch 11 TRAIN:  66%|██████▌   | 89/135 [00:18<00:07,  5.81it/s, epoch=11, grad_norm=73.184, loss=301.7867, lr=1.27e-04]
Epoch 11 TRAIN:  66%|██████▌   | 89/135 [00:18<00:07,  5.81it/s, epoch=11, grad_norm=98.511, loss=290.9713, lr=1.27e-04]
Epoch 11 TRAIN:  66%|██████▌   | 89/135 [00:18<00:07,  5.81it/s, epoch=11, grad_norm=81.813, loss=280.5488, lr=1.27e-04]
Epoch 11 TRAIN:  66%|██████▌   | 89/135 [00:18<00:07,  5.81it/s, epoch=11, grad_norm=57.377, loss=253.6351, lr=1.27e-04]
Epoch 11 TRAIN:  68%|██████▊   | 92/135 [00:18<00:08,  5.34it/s, epoch=11, grad_norm=57.377, loss=253.6351, lr=1.27e-04]
Epoch 11 TRAIN:  68%|██████▊   | 92/135 [00:18<00:08,  5.34it/s, epoch=11, grad_norm=101.117, loss=223.0754, lr=1.27e-04]
Epoch 11 TRAIN:  68%|██████▊   | 92/135 [00:19<00:08,  5.34it/s, epoch=11, grad_norm=125.840, loss=273.9191, lr=1.27e-04]
Epoch 11 TRAIN:  68%|██████▊   | 92/135 [00:19<00:08,  5.34it/s, epoch=11, grad_norm=69.067, loss=285.8557, lr=1.27e-04] 
Epoch 11 TRAIN:  68%|██████▊   | 92/135 [00:19<00:08,  5.34it/s, epoch=11, grad_norm=67.254, loss=282.2681, lr=1.27e-04]
Epoch 11 TRAIN:  71%|███████   | 96/135 [00:19<00:06,  5.62it/s, epoch=11, grad_norm=67.254, loss=282.2681, lr=1.27e-04]
Epoch 11 TRAIN:  71%|███████   | 96/135 [00:19<00:06,  5.62it/s, epoch=11, grad_norm=102.985, loss=290.1166, lr=1.27e-04]
Epoch 11 TRAIN:  71%|███████   | 96/135 [00:19<00:06,  5.62it/s, epoch=11, grad_norm=63.726, loss=276.8297, lr=1.28e-04] 
Epoch 11 TRAIN:  71%|███████   | 96/135 [00:19<00:06,  5.62it/s, epoch=11, grad_norm=249.966, loss=395.4613, lr=1.28e-04]
Epoch 11 TRAIN:  73%|███████▎  | 99/135 [00:19<00:06,  5.59it/s, epoch=11, grad_norm=249.966, loss=395.4613, lr=1.28e-04]
Epoch 11 TRAIN:  73%|███████▎  | 99/135 [00:20<00:06,  5.59it/s, epoch=11, grad_norm=56.621, loss=235.5534, lr=1.28e-04] 2026-04-16 19:57:36,719 DEBUG TRAIN | steps/sec 5.002| Batch 11/100 loss 235.553391 loss_att 214.124420 loss_ctc 285.554352 th_accuracy 0.256659 lr 1.2768e-04 grad_norm 56.620640 rank 0

Epoch 11 TRAIN:  73%|███████▎  | 99/135 [00:20<00:06,  5.59it/s, epoch=11, grad_norm=124.397, loss=376.8930, lr=1.28e-04]
Epoch 11 TRAIN:  73%|███████▎  | 99/135 [00:20<00:06,  5.59it/s, epoch=11, grad_norm=95.185, loss=403.7236, lr=1.28e-04] 
Epoch 11 TRAIN:  76%|███████▌  | 102/135 [00:20<00:05,  5.68it/s, epoch=11, grad_norm=95.185, loss=403.7236, lr=1.28e-04]
Epoch 11 TRAIN:  76%|███████▌  | 102/135 [00:20<00:05,  5.68it/s, epoch=11, grad_norm=83.726, loss=347.7530, lr=1.28e-04]
Epoch 11 TRAIN:  76%|███████▌  | 102/135 [00:20<00:05,  5.68it/s, epoch=11, grad_norm=154.676, loss=327.8306, lr=1.28e-04]
Epoch 11 TRAIN:  76%|███████▌  | 102/135 [00:21<00:05,  5.68it/s, epoch=11, grad_norm=74.055, loss=339.5554, lr=1.28e-04] 
Epoch 11 TRAIN:  78%|███████▊  | 105/135 [00:21<00:05,  5.58it/s, epoch=11, grad_norm=74.055, loss=339.5554, lr=1.28e-04]
Epoch 11 TRAIN:  78%|███████▊  | 105/135 [00:21<00:05,  5.58it/s, epoch=11, grad_norm=163.438, loss=347.3768, lr=1.28e-04]
Epoch 11 TRAIN:  78%|███████▊  | 105/135 [00:21<00:05,  5.58it/s, epoch=11, grad_norm=128.944, loss=353.5498, lr=1.28e-04]
Epoch 11 TRAIN:  78%|███████▊  | 105/135 [00:21<00:05,  5.58it/s, epoch=11, grad_norm=107.126, loss=239.1150, lr=1.28e-04]
Epoch 11 TRAIN:  80%|████████  | 108/135 [00:21<00:04,  5.62it/s, epoch=11, grad_norm=107.126, loss=239.1150, lr=1.28e-04]
Epoch 11 TRAIN:  80%|████████  | 108/135 [00:21<00:04,  5.62it/s, epoch=11, grad_norm=101.059, loss=330.0148, lr=1.28e-04]
Epoch 11 TRAIN:  80%|████████  | 108/135 [00:21<00:04,  5.62it/s, epoch=11, grad_norm=143.390, loss=363.5910, lr=1.28e-04]
Epoch 11 TRAIN:  80%|████████  | 108/135 [00:22<00:04,  5.62it/s, epoch=11, grad_norm=76.653, loss=338.3116, lr=1.29e-04] 
Epoch 11 TRAIN:  82%|████████▏ | 111/135 [00:22<00:04,  5.65it/s, epoch=11, grad_norm=76.653, loss=338.3116, lr=1.29e-04]
Epoch 11 TRAIN:  82%|████████▏ | 111/135 [00:22<00:04,  5.65it/s, epoch=11, grad_norm=111.924, loss=293.2409, lr=1.29e-04]
Epoch 11 TRAIN:  82%|████████▏ | 111/135 [00:22<00:04,  5.65it/s, epoch=11, grad_norm=85.977, loss=268.4800, lr=1.29e-04] 
Epoch 11 TRAIN:  82%|████████▏ | 111/135 [00:22<00:04,  5.65it/s, epoch=11, grad_norm=213.689, loss=394.5438, lr=1.29e-04]
Epoch 11 TRAIN:  82%|████████▏ | 111/135 [00:22<00:04,  5.65it/s, epoch=11, grad_norm=196.677, loss=340.3240, lr=1.29e-04]
Epoch 11 TRAIN:  85%|████████▌ | 115/135 [00:22<00:03,  5.85it/s, epoch=11, grad_norm=196.677, loss=340.3240, lr=1.29e-04]
Epoch 11 TRAIN:  85%|████████▌ | 115/135 [00:22<00:03,  5.85it/s, epoch=11, grad_norm=75.748, loss=261.7841, lr=1.29e-04] 
Epoch 11 TRAIN:  85%|████████▌ | 115/135 [00:23<00:03,  5.85it/s, epoch=11, grad_norm=105.074, loss=342.0010, lr=1.29e-04]
Epoch 11 TRAIN:  85%|████████▌ | 115/135 [00:23<00:03,  5.85it/s, epoch=11, grad_norm=179.739, loss=314.0862, lr=1.29e-04]
Epoch 11 TRAIN:  87%|████████▋ | 118/135 [00:23<00:02,  5.87it/s, epoch=11, grad_norm=179.739, loss=314.0862, lr=1.29e-04]
Epoch 11 TRAIN:  87%|████████▋ | 118/135 [00:23<00:02,  5.87it/s, epoch=11, grad_norm=100.245, loss=377.3416, lr=1.29e-04]
Epoch 11 TRAIN:  87%|████████▋ | 118/135 [00:23<00:02,  5.87it/s, epoch=11, grad_norm=102.817, loss=413.3599, lr=1.29e-04]
Epoch 11 TRAIN:  87%|████████▋ | 118/135 [00:23<00:02,  5.87it/s, epoch=11, grad_norm=92.126, loss=395.2579, lr=1.29e-04] 
Epoch 11 TRAIN:  90%|████████▉ | 121/135 [00:23<00:02,  5.71it/s, epoch=11, grad_norm=92.126, loss=395.2579, lr=1.29e-04]
Epoch 11 TRAIN:  90%|████████▉ | 121/135 [00:23<00:02,  5.71it/s, epoch=11, grad_norm=171.658, loss=302.4619, lr=1.29e-04]
Epoch 11 TRAIN:  90%|████████▉ | 121/135 [00:24<00:02,  5.71it/s, epoch=11, grad_norm=109.962, loss=252.8147, lr=1.30e-04]
Epoch 11 TRAIN:  90%|████████▉ | 121/135 [00:24<00:02,  5.71it/s, epoch=11, grad_norm=68.992, loss=288.4645, lr=1.30e-04] 
Epoch 11 TRAIN:  90%|████████▉ | 121/135 [00:24<00:02,  5.71it/s, epoch=11, grad_norm=126.921, loss=344.3549, lr=1.30e-04]
Epoch 11 TRAIN:  93%|█████████▎| 125/135 [00:24<00:01,  5.69it/s, epoch=11, grad_norm=126.921, loss=344.3549, lr=1.30e-04]
Epoch 11 TRAIN:  93%|█████████▎| 125/135 [00:24<00:01,  5.69it/s, epoch=11, grad_norm=154.449, loss=372.1665, lr=1.30e-04]
Epoch 11 TRAIN:  93%|█████████▎| 125/135 [00:24<00:01,  5.69it/s, epoch=11, grad_norm=114.543, loss=376.0253, lr=1.30e-04]
Epoch 11 TRAIN:  93%|█████████▎| 125/135 [00:24<00:01,  5.69it/s, epoch=11, grad_norm=127.629, loss=376.9629, lr=1.30e-04]
Epoch 11 TRAIN:  93%|█████████▎| 125/135 [00:25<00:01,  5.69it/s, epoch=11, grad_norm=82.197, loss=297.4842, lr=1.30e-04] 
Epoch 11 TRAIN:  96%|█████████▌| 129/135 [00:25<00:01,  5.85it/s, epoch=11, grad_norm=82.197, loss=297.4842, lr=1.30e-04]
Epoch 11 TRAIN:  96%|█████████▌| 129/135 [00:25<00:01,  5.85it/s, epoch=11, grad_norm=77.559, loss=280.2294, lr=1.30e-04]
Epoch 11 TRAIN:  96%|█████████▌| 129/135 [00:25<00:01,  5.85it/s, epoch=11, grad_norm=162.968, loss=393.6340, lr=1.30e-04]
Epoch 11 TRAIN:  96%|█████████▌| 129/135 [00:25<00:01,  5.85it/s, epoch=11, grad_norm=141.867, loss=409.3735, lr=1.30e-04]
Epoch 11 TRAIN:  96%|█████████▌| 129/135 [00:25<00:01,  5.85it/s, epoch=11, grad_norm=164.028, loss=389.7429, lr=1.30e-04]
Epoch 11 TRAIN:  99%|█████████▊| 133/135 [00:25<00:00,  6.03it/s, epoch=11, grad_norm=164.028, loss=389.7429, lr=1.30e-04]
Epoch 11 TRAIN:  99%|█████████▊| 133/135 [00:25<00:00,  6.03it/s, epoch=11, grad_norm=146.009, loss=258.5634, lr=1.30e-04]
Epoch 11 TRAIN:  99%|█████████▊| 133/135 [00:26<00:00,  6.03it/s, epoch=11, grad_norm=117.397, loss=360.2414, lr=1.30e-04]
Epoch 11 TRAIN:  99%|█████████▊| 133/135 [00:26<00:00,  6.03it/s, epoch=11, grad_norm=76.770, loss=306.0777, lr=1.31e-04] 
Epoch 11 TRAIN:  99%|█████████▊| 133/135 [00:26<00:00,  6.03it/s, epoch=11, grad_norm=116.153, loss=379.6063, lr=1.31e-04]
Epoch 11 TRAIN: 137it [00:26,  5.64it/s, epoch=11, grad_norm=116.153, loss=379.6063, lr=1.31e-04]                         
Epoch 11 TRAIN: 137it [00:26,  5.16it/s, epoch=11, grad_norm=116.153, loss=379.6063, lr=1.31e-04]

CV:   0%|          | 0/6 [00:00<?, ?it/s]/home/rookie/Documents/batuka/projects/wenet/wenet/wenet/utils/train_utils.py:689: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  amp_autocast(enabled=dtype is not None,

CV:  17%|█▋        | 1/6 [00:00<00:03,  1.33it/s]
                                                 
2026-04-16 19:57:44,043 INFO Epoch 11 Step 1633 CV info lr 1.3064e-04 cv_loss 381.4534745649858 rank 0 acc 0.2559954871733983
2026-04-16 19:57:44,045 INFO [Rank 0] Checkpoint: save to checkpoint exp/conformer_mongolian/epoch_11.pt
2026-04-16 19:57:44,158 INFO Epoch 12 Step 1633 TRAIN info lr 1.3064e-04 rank 0
2026-04-16 19:57:44,179 INFO using accumulate grad, new batch size is 1 times larger than before

Epoch 12 TRAIN:   0%|          | 0/137 [00:00<?, ?it/s]
Epoch 12 TRAIN:   0%|          | 0/137 [00:03<?, ?it/s, epoch=12, grad_norm=303.978, loss=330.8511, lr=1.31e-04]
Epoch 12 TRAIN:   1%|          | 1/137 [00:03<08:58,  3.96s/it, epoch=12, grad_norm=303.978, loss=330.8511, lr=1.31e-04]
Epoch 12 TRAIN:   1%|          | 1/137 [00:04<08:58,  3.96s/it, epoch=12, grad_norm=593.201, loss=539.0478, lr=1.31e-04]
Epoch 12 TRAIN:   1%|          | 1/137 [00:04<08:58,  3.96s/it, epoch=12, grad_norm=282.751, loss=305.6211, lr=1.31e-04]
Epoch 12 TRAIN:   1%|          | 1/137 [00:04<08:58,  3.96s/it, epoch=12, grad_norm=444.093, loss=433.7631, lr=1.31e-04]
Epoch 12 TRAIN:   1%|          | 1/137 [00:04<08:58,  3.96s/it, epoch=12, grad_norm=265.710, loss=327.0016, lr=1.31e-04]
Epoch 12 TRAIN:   1%|          | 1/137 [00:04<08:58,  3.96s/it, epoch=12, grad_norm=317.811, loss=384.9334, lr=1.31e-04]
Epoch 12 TRAIN:   1%|          | 1/137 [00:04<08:58,  3.96s/it, epoch=12, grad_norm=216.377, loss=372.1391, lr=1.31e-04]
Epoch 12 TRAIN:   1%|          | 1/137 [00:04<08:58,  3.96s/it, epoch=12, grad_norm=185.501, loss=331.6061, lr=1.31e-04]
Epoch 12 TRAIN:   6%|▌         | 8/137 [00:04<00:54,  2.35it/s, epoch=12, grad_norm=185.501, loss=331.6061, lr=1.31e-04]
Epoch 12 TRAIN:   6%|▌         | 8/137 [00:04<00:54,  2.35it/s, epoch=12, grad_norm=61.144, loss=351.8269, lr=1.31e-04] 
Epoch 12 TRAIN:   6%|▌         | 8/137 [00:04<00:54,  2.35it/s, epoch=12, grad_norm=77.039, loss=344.4247, lr=1.31e-04]
Epoch 12 TRAIN:   6%|▌         | 8/137 [00:04<00:54,  2.35it/s, epoch=12, grad_norm=133.970, loss=345.2097, lr=1.32e-04]
Epoch 12 TRAIN:   6%|▌         | 8/137 [00:05<00:54,  2.35it/s, epoch=12, grad_norm=177.672, loss=312.3002, lr=1.32e-04]
Epoch 12 TRAIN:   6%|▌         | 8/137 [00:05<00:54,  2.35it/s, epoch=12, grad_norm=138.327, loss=329.3301, lr=1.32e-04]
Epoch 12 TRAIN:   9%|▉         | 13/137 [00:05<00:36,  3.38it/s, epoch=12, grad_norm=138.327, loss=329.3301, lr=1.32e-04]
Epoch 12 TRAIN:   9%|▉         | 13/137 [00:05<00:36,  3.38it/s, epoch=12, grad_norm=105.291, loss=333.4389, lr=1.32e-04]
Epoch 12 TRAIN:   9%|▉         | 13/137 [00:05<00:36,  3.38it/s, epoch=12, grad_norm=62.054, loss=325.5387, lr=1.32e-04] 
Epoch 12 TRAIN:   9%|▉         | 13/137 [00:05<00:36,  3.38it/s, epoch=12, grad_norm=68.787, loss=345.1212, lr=1.32e-04]
Epoch 12 TRAIN:   9%|▉         | 13/137 [00:05<00:36,  3.38it/s, epoch=12, grad_norm=61.776, loss=342.7182, lr=1.32e-04]
Epoch 12 TRAIN:  12%|█▏        | 17/137 [00:05<00:29,  4.07it/s, epoch=12, grad_norm=61.776, loss=342.7182, lr=1.32e-04]
Epoch 12 TRAIN:  12%|█▏        | 17/137 [00:06<00:29,  4.07it/s, epoch=12, grad_norm=85.096, loss=358.3392, lr=1.32e-04]
Epoch 12 TRAIN:  12%|█▏        | 17/137 [00:06<00:29,  4.07it/s, epoch=12, grad_norm=87.444, loss=324.3641, lr=1.32e-04]
Epoch 12 TRAIN:  12%|█▏        | 17/137 [00:06<00:29,  4.07it/s, epoch=12, grad_norm=142.742, loss=285.6744, lr=1.32e-04]
Epoch 12 TRAIN:  12%|█▏        | 17/137 [00:06<00:29,  4.07it/s, epoch=12, grad_norm=70.759, loss=347.4442, lr=1.32e-04] 
Epoch 12 TRAIN:  15%|█▌        | 21/137 [00:06<00:25,  4.63it/s, epoch=12, grad_norm=70.759, loss=347.4442, lr=1.32e-04]
Epoch 12 TRAIN:  15%|█▌        | 21/137 [00:06<00:25,  4.63it/s, epoch=12, grad_norm=72.180, loss=307.1158, lr=1.32e-04]
Epoch 12 TRAIN:  15%|█▌        | 21/137 [00:06<00:25,  4.63it/s, epoch=12, grad_norm=54.641, loss=328.6312, lr=1.32e-04]
Epoch 12 TRAIN:  15%|█▌        | 21/137 [00:06<00:25,  4.63it/s, epoch=12, grad_norm=86.341, loss=366.6313, lr=1.33e-04]
Epoch 12 TRAIN:  15%|█▌        | 21/137 [00:07<00:25,  4.63it/s, epoch=12, grad_norm=73.537, loss=357.0726, lr=1.33e-04]
Epoch 12 TRAIN:  18%|█▊        | 25/137 [00:07<00:22,  5.06it/s, epoch=12, grad_norm=73.537, loss=357.0726, lr=1.33e-04]
Epoch 12 TRAIN:  18%|█▊        | 25/137 [00:07<00:22,  5.06it/s, epoch=12, grad_norm=106.556, loss=385.0031, lr=1.33e-04]
Epoch 12 TRAIN:  18%|█▊        | 25/137 [00:07<00:22,  5.06it/s, epoch=12, grad_norm=67.123, loss=332.0611, lr=1.33e-04] 
Epoch 12 TRAIN:  18%|█▊        | 25/137 [00:07<00:22,  5.06it/s, epoch=12, grad_norm=101.574, loss=328.8750, lr=1.33e-04]
Epoch 12 TRAIN:  18%|█▊        | 25/137 [00:07<00:22,  5.06it/s, epoch=12, grad_norm=70.966, loss=355.8019, lr=1.33e-04] 
Epoch 12 TRAIN:  21%|██        | 29/137 [00:07<00:19,  5.47it/s, epoch=12, grad_norm=70.966, loss=355.8019, lr=1.33e-04]
Epoch 12 TRAIN:  21%|██        | 29/137 [00:07<00:19,  5.47it/s, epoch=12, grad_norm=123.732, loss=291.4839, lr=1.33e-04]
Epoch 12 TRAIN:  21%|██        | 29/137 [00:08<00:19,  5.47it/s, epoch=12, grad_norm=75.736, loss=300.9920, lr=1.33e-04] 
Epoch 12 TRAIN:  21%|██        | 29/137 [00:08<00:19,  5.47it/s, epoch=12, grad_norm=62.323, loss=322.1112, lr=1.33e-04]
Epoch 12 TRAIN:  21%|██        | 29/137 [00:08<00:19,  5.47it/s, epoch=12, grad_norm=116.819, loss=340.4151, lr=1.33e-04]
Epoch 12 TRAIN:  24%|██▍       | 33/137 [00:08<00:18,  5.75it/s, epoch=12, grad_norm=116.819, loss=340.4151, lr=1.33e-04]
Epoch 12 TRAIN:  24%|██▍       | 33/137 [00:08<00:18,  5.75it/s, epoch=12, grad_norm=96.439, loss=352.1777, lr=1.33e-04] 
Epoch 12 TRAIN:  24%|██▍       | 33/137 [00:08<00:18,  5.75it/s, epoch=12, grad_norm=94.028, loss=349.8447, lr=1.33e-04]
Epoch 12 TRAIN:  24%|██▍       | 33/137 [00:08<00:18,  5.75it/s, epoch=12, grad_norm=82.523, loss=361.1919, lr=1.34e-04]
Epoch 12 TRAIN:  24%|██▍       | 33/137 [00:08<00:18,  5.75it/s, epoch=12, grad_norm=77.582, loss=315.3773, lr=1.34e-04]
Epoch 12 TRAIN:  27%|██▋       | 37/137 [00:08<00:16,  5.95it/s, epoch=12, grad_norm=77.582, loss=315.3773, lr=1.34e-04]
Epoch 12 TRAIN:  27%|██▋       | 37/137 [00:09<00:16,  5.95it/s, epoch=12, grad_norm=76.903, loss=315.9415, lr=1.34e-04]
Epoch 12 TRAIN:  27%|██▋       | 37/137 [00:09<00:16,  5.95it/s, epoch=12, grad_norm=68.665, loss=344.2036, lr=1.34e-04]
Epoch 12 TRAIN:  27%|██▋       | 37/137 [00:09<00:16,  5.95it/s, epoch=12, grad_norm=71.911, loss=356.8854, lr=1.34e-04]
Epoch 12 TRAIN:  27%|██▋       | 37/137 [00:09<00:16,  5.95it/s, epoch=12, grad_norm=101.749, loss=335.2053, lr=1.34e-04]
Epoch 12 TRAIN:  30%|██▉       | 41/137 [00:09<00:16,  5.79it/s, epoch=12, grad_norm=101.749, loss=335.2053, lr=1.34e-04]
Epoch 12 TRAIN:  30%|██▉       | 41/137 [00:09<00:16,  5.79it/s, epoch=12, grad_norm=67.610, loss=360.0992, lr=1.34e-04] 
Epoch 12 TRAIN:  30%|██▉       | 41/137 [00:10<00:16,  5.79it/s, epoch=12, grad_norm=113.353, loss=319.2080, lr=1.34e-04]
Epoch 12 TRAIN:  30%|██▉       | 41/137 [00:10<00:16,  5.79it/s, epoch=12, grad_norm=69.631, loss=363.2469, lr=1.34e-04] 
Epoch 12 TRAIN:  30%|██▉       | 41/137 [00:10<00:16,  5.79it/s, epoch=12, grad_norm=85.797, loss=304.2679, lr=1.34e-04]
Epoch 12 TRAIN:  33%|███▎      | 45/137 [00:10<00:15,  5.80it/s, epoch=12, grad_norm=85.797, loss=304.2679, lr=1.34e-04]
Epoch 12 TRAIN:  33%|███▎      | 45/137 [00:10<00:15,  5.80it/s, epoch=12, grad_norm=70.804, loss=334.5612, lr=1.34e-04]
Epoch 12 TRAIN:  33%|███▎      | 45/137 [00:10<00:15,  5.80it/s, epoch=12, grad_norm=159.044, loss=370.3582, lr=1.34e-04]
Epoch 12 TRAIN:  33%|███▎      | 45/137 [00:10<00:15,  5.80it/s, epoch=12, grad_norm=76.951, loss=287.6089, lr=1.34e-04] 
Epoch 12 TRAIN:  33%|███▎      | 45/137 [00:11<00:15,  5.80it/s, epoch=12, grad_norm=62.641, loss=290.2198, lr=1.35e-04]
Epoch 12 TRAIN:  36%|███▌      | 49/137 [00:11<00:15,  5.70it/s, epoch=12, grad_norm=62.641, loss=290.2198, lr=1.35e-04]
Epoch 12 TRAIN:  36%|███▌      | 49/137 [00:11<00:15,  5.70it/s, epoch=12, grad_norm=59.357, loss=304.3033, lr=1.35e-04]
Epoch 12 TRAIN:  36%|███▌      | 49/137 [00:11<00:15,  5.70it/s, epoch=12, grad_norm=68.421, loss=285.4762, lr=1.35e-04]
Epoch 12 TRAIN:  36%|███▌      | 49/137 [00:11<00:15,  5.70it/s, epoch=12, grad_norm=80.597, loss=317.7628, lr=1.35e-04]
Epoch 12 TRAIN:  38%|███▊      | 52/137 [00:11<00:14,  5.74it/s, epoch=12, grad_norm=80.597, loss=317.7628, lr=1.35e-04]
Epoch 12 TRAIN:  38%|███▊      | 52/137 [00:11<00:14,  5.74it/s, epoch=12, grad_norm=95.077, loss=320.9377, lr=1.35e-04]
Epoch 12 TRAIN:  38%|███▊      | 52/137 [00:11<00:14,  5.74it/s, epoch=12, grad_norm=85.854, loss=334.2688, lr=1.35e-04]
Epoch 12 TRAIN:  38%|███▊      | 52/137 [00:12<00:14,  5.74it/s, epoch=12, grad_norm=69.874, loss=297.1408, lr=1.35e-04]
Epoch 12 TRAIN:  38%|███▊      | 52/137 [00:12<00:14,  5.74it/s, epoch=12, grad_norm=69.606, loss=314.9008, lr=1.35e-04]
Epoch 12 TRAIN:  41%|████      | 56/137 [00:12<00:13,  5.92it/s, epoch=12, grad_norm=69.606, loss=314.9008, lr=1.35e-04]
Epoch 12 TRAIN:  41%|████      | 56/137 [00:12<00:13,  5.92it/s, epoch=12, grad_norm=78.996, loss=331.3133, lr=1.35e-04]
Epoch 12 TRAIN:  41%|████      | 56/137 [00:12<00:13,  5.92it/s, epoch=12, grad_norm=87.058, loss=371.4841, lr=1.35e-04]
Epoch 12 TRAIN:  41%|████      | 56/137 [00:12<00:13,  5.92it/s, epoch=12, grad_norm=72.685, loss=332.2098, lr=1.35e-04]
Epoch 12 TRAIN:  41%|████      | 56/137 [00:13<00:13,  5.92it/s, epoch=12, grad_norm=140.295, loss=308.9120, lr=1.35e-04]
Epoch 12 TRAIN:  44%|████▍     | 60/137 [00:13<00:13,  5.64it/s, epoch=12, grad_norm=140.295, loss=308.9120, lr=1.35e-04]
Epoch 12 TRAIN:  44%|████▍     | 60/137 [00:13<00:13,  5.64it/s, epoch=12, grad_norm=64.325, loss=322.2291, lr=1.36e-04] 
Epoch 12 TRAIN:  44%|████▍     | 60/137 [00:13<00:13,  5.64it/s, epoch=12, grad_norm=89.781, loss=297.5874, lr=1.36e-04]
Epoch 12 TRAIN:  44%|████▍     | 60/137 [00:13<00:13,  5.64it/s, epoch=12, grad_norm=123.940, loss=256.6367, lr=1.36e-04]
Epoch 12 TRAIN:  46%|████▌     | 63/137 [00:13<00:13,  5.64it/s, epoch=12, grad_norm=123.940, loss=256.6367, lr=1.36e-04]
Epoch 12 TRAIN:  46%|████▌     | 63/137 [00:13<00:13,  5.64it/s, epoch=12, grad_norm=109.999, loss=289.6235, lr=1.36e-04]
Epoch 12 TRAIN:  46%|████▌     | 63/137 [00:13<00:13,  5.64it/s, epoch=12, grad_norm=69.049, loss=343.5471, lr=1.36e-04] 
Epoch 12 TRAIN:  46%|████▌     | 63/137 [00:14<00:13,  5.64it/s, epoch=12, grad_norm=115.799, loss=321.9751, lr=1.36e-04]
Epoch 12 TRAIN:  48%|████▊     | 66/137 [00:14<00:12,  5.67it/s, epoch=12, grad_norm=115.799, loss=321.9751, lr=1.36e-04]
Epoch 12 TRAIN:  48%|████▊     | 66/137 [00:14<00:12,  5.67it/s, epoch=12, grad_norm=83.214, loss=305.8878, lr=1.36e-04] 
Epoch 12 TRAIN:  48%|████▊     | 66/137 [00:14<00:12,  5.67it/s, epoch=12, grad_norm=76.859, loss=301.4226, lr=1.36e-04]
Epoch 12 TRAIN:  48%|████▊     | 66/137 [00:14<00:12,  5.67it/s, epoch=12, grad_norm=107.110, loss=319.9281, lr=1.36e-04]
Epoch 12 TRAIN:  50%|█████     | 69/137 [00:14<00:12,  5.63it/s, epoch=12, grad_norm=107.110, loss=319.9281, lr=1.36e-04]
Epoch 12 TRAIN:  50%|█████     | 69/137 [00:14<00:12,  5.63it/s, epoch=12, grad_norm=70.261, loss=304.6641, lr=1.36e-04] 
Epoch 12 TRAIN:  50%|█████     | 69/137 [00:14<00:12,  5.63it/s, epoch=12, grad_norm=58.938, loss=298.5811, lr=1.36e-04]
Epoch 12 TRAIN:  50%|█████     | 69/137 [00:15<00:12,  5.63it/s, epoch=12, grad_norm=71.510, loss=328.6851, lr=1.36e-04]
Epoch 12 TRAIN:  53%|█████▎    | 72/137 [00:15<00:11,  5.63it/s, epoch=12, grad_norm=71.510, loss=328.6851, lr=1.36e-04]
Epoch 12 TRAIN:  53%|█████▎    | 72/137 [00:15<00:11,  5.63it/s, epoch=12, grad_norm=182.816, loss=289.0715, lr=1.36e-04]
Epoch 12 TRAIN:  53%|█████▎    | 72/137 [00:15<00:11,  5.63it/s, epoch=12, grad_norm=109.397, loss=336.1455, lr=1.37e-04]
Epoch 12 TRAIN:  53%|█████▎    | 72/137 [00:15<00:11,  5.63it/s, epoch=12, grad_norm=86.917, loss=360.7067, lr=1.37e-04] 
Epoch 12 TRAIN:  55%|█████▍    | 75/137 [00:15<00:11,  5.59it/s, epoch=12, grad_norm=86.917, loss=360.7067, lr=1.37e-04]
Epoch 12 TRAIN:  55%|█████▍    | 75/137 [00:15<00:11,  5.59it/s, epoch=12, grad_norm=81.012, loss=333.1902, lr=1.37e-04]
Epoch 12 TRAIN:  55%|█████▍    | 75/137 [00:16<00:11,  5.59it/s, epoch=12, grad_norm=159.365, loss=237.2391, lr=1.37e-04]
Epoch 12 TRAIN:  55%|█████▍    | 75/137 [00:16<00:11,  5.59it/s, epoch=12, grad_norm=94.754, loss=338.8981, lr=1.37e-04] 
Epoch 12 TRAIN:  57%|█████▋    | 78/137 [00:16<00:10,  5.62it/s, epoch=12, grad_norm=94.754, loss=338.8981, lr=1.37e-04]
Epoch 12 TRAIN:  57%|█████▋    | 78/137 [00:16<00:10,  5.62it/s, epoch=12, grad_norm=80.316, loss=328.6304, lr=1.37e-04]
Epoch 12 TRAIN:  57%|█████▋    | 78/137 [00:16<00:10,  5.62it/s, epoch=12, grad_norm=94.847, loss=373.7595, lr=1.37e-04]
Epoch 12 TRAIN:  57%|█████▋    | 78/137 [00:16<00:10,  5.62it/s, epoch=12, grad_norm=145.500, loss=317.1393, lr=1.37e-04]
Epoch 12 TRAIN:  59%|█████▉    | 81/137 [00:16<00:10,  5.58it/s, epoch=12, grad_norm=145.500, loss=317.1393, lr=1.37e-04]
Epoch 12 TRAIN:  59%|█████▉    | 81/137 [00:16<00:10,  5.58it/s, epoch=12, grad_norm=88.477, loss=358.4301, lr=1.37e-04] 
Epoch 12 TRAIN:  59%|█████▉    | 81/137 [00:17<00:10,  5.58it/s, epoch=12, grad_norm=99.914, loss=289.8560, lr=1.37e-04]
Epoch 12 TRAIN:  59%|█████▉    | 81/137 [00:17<00:10,  5.58it/s, epoch=12, grad_norm=120.795, loss=366.2709, lr=1.37e-04]
Epoch 12 TRAIN:  61%|██████▏   | 84/137 [00:17<00:09,  5.37it/s, epoch=12, grad_norm=120.795, loss=366.2709, lr=1.37e-04]
Epoch 12 TRAIN:  61%|██████▏   | 84/137 [00:17<00:09,  5.37it/s, epoch=12, grad_norm=128.785, loss=371.6695, lr=1.37e-04]
Epoch 12 TRAIN:  61%|██████▏   | 84/137 [00:17<00:09,  5.37it/s, epoch=12, grad_norm=69.537, loss=290.8788, lr=1.38e-04] 
Epoch 12 TRAIN:  61%|██████▏   | 84/137 [00:18<00:09,  5.37it/s, epoch=12, grad_norm=91.687, loss=338.5002, lr=1.38e-04]
Epoch 12 TRAIN:  64%|██████▎   | 87/137 [00:18<00:09,  5.18it/s, epoch=12, grad_norm=91.687, loss=338.5002, lr=1.38e-04]
Epoch 12 TRAIN:  64%|██████▎   | 87/137 [00:18<00:09,  5.18it/s, epoch=12, grad_norm=149.291, loss=281.4214, lr=1.38e-04]
Epoch 12 TRAIN:  64%|██████▎   | 87/137 [00:18<00:09,  5.18it/s, epoch=12, grad_norm=202.970, loss=258.6702, lr=1.38e-04]
Epoch 12 TRAIN:  64%|██████▎   | 87/137 [00:18<00:09,  5.18it/s, epoch=12, grad_norm=172.639, loss=285.5135, lr=1.38e-04]
Epoch 12 TRAIN:  66%|██████▌   | 90/137 [00:18<00:08,  5.26it/s, epoch=12, grad_norm=172.639, loss=285.5135, lr=1.38e-04]
Epoch 12 TRAIN:  66%|██████▌   | 90/137 [00:18<00:08,  5.26it/s, epoch=12, grad_norm=91.633, loss=345.0854, lr=1.38e-04] 
Epoch 12 TRAIN:  66%|██████▌   | 90/137 [00:18<00:08,  5.26it/s, epoch=12, grad_norm=128.826, loss=252.6224, lr=1.38e-04]
Epoch 12 TRAIN:  66%|██████▌   | 90/137 [00:19<00:08,  5.26it/s, epoch=12, grad_norm=92.247, loss=301.0275, lr=1.38e-04] 
Epoch 12 TRAIN:  68%|██████▊   | 93/137 [00:19<00:08,  5.24it/s, epoch=12, grad_norm=92.247, loss=301.0275, lr=1.38e-04]
Epoch 12 TRAIN:  68%|██████▊   | 93/137 [00:19<00:08,  5.24it/s, epoch=12, grad_norm=190.696, loss=355.2101, lr=1.38e-04]
Epoch 12 TRAIN:  68%|██████▊   | 93/137 [00:19<00:08,  5.24it/s, epoch=12, grad_norm=218.540, loss=359.7068, lr=1.38e-04]
Epoch 12 TRAIN:  68%|██████▊   | 93/137 [00:19<00:08,  5.24it/s, epoch=12, grad_norm=78.294, loss=247.4122, lr=1.38e-04] 
Epoch 12 TRAIN:  70%|███████   | 96/137 [00:19<00:07,  5.18it/s, epoch=12, grad_norm=78.294, loss=247.4122, lr=1.38e-04]
Epoch 12 TRAIN:  70%|███████   | 96/137 [00:19<00:07,  5.18it/s, epoch=12, grad_norm=72.832, loss=282.0939, lr=1.38e-04]
Epoch 12 TRAIN:  70%|███████   | 96/137 [00:20<00:07,  5.18it/s, epoch=12, grad_norm=108.203, loss=258.4239, lr=1.38e-04]
Epoch 12 TRAIN:  70%|███████   | 96/137 [00:20<00:07,  5.18it/s, epoch=12, grad_norm=116.860, loss=247.0290, lr=1.39e-04]
Epoch 12 TRAIN:  70%|███████   | 96/137 [00:20<00:07,  5.18it/s, epoch=12, grad_norm=178.103, loss=358.5486, lr=1.39e-04]2026-04-16 19:58:04,705 DEBUG TRAIN | steps/sec 4.895| Batch 12/100 loss 358.548615 loss_att 321.840576 loss_ctc 444.200684 th_accuracy 0.253138 lr 1.3864e-04 grad_norm 178.103058 rank 0

Epoch 12 TRAIN:  73%|███████▎  | 100/137 [00:20<00:07,  5.18it/s, epoch=12, grad_norm=178.103, loss=358.5486, lr=1.39e-04]
Epoch 12 TRAIN:  73%|███████▎  | 100/137 [00:20<00:07,  5.18it/s, epoch=12, grad_norm=106.463, loss=282.0074, lr=1.39e-04]
Epoch 12 TRAIN:  73%|███████▎  | 100/137 [00:20<00:07,  5.18it/s, epoch=12, grad_norm=181.404, loss=369.3384, lr=1.39e-04]
Epoch 12 TRAIN:  73%|███████▎  | 100/137 [00:21<00:07,  5.18it/s, epoch=12, grad_norm=79.669, loss=244.3567, lr=1.39e-04] 
Epoch 12 TRAIN:  73%|███████▎  | 100/137 [00:21<00:07,  5.18it/s, epoch=12, grad_norm=99.834, loss=296.5853, lr=1.39e-04]
Epoch 12 TRAIN:  76%|███████▌  | 104/137 [00:21<00:05,  5.50it/s, epoch=12, grad_norm=99.834, loss=296.5853, lr=1.39e-04]
Epoch 12 TRAIN:  76%|███████▌  | 104/137 [00:21<00:05,  5.50it/s, epoch=12, grad_norm=192.125, loss=248.7520, lr=1.39e-04]
Epoch 12 TRAIN:  76%|███████▌  | 104/137 [00:21<00:05,  5.50it/s, epoch=12, grad_norm=78.953, loss=315.2836, lr=1.39e-04] 
Epoch 12 TRAIN:  76%|███████▌  | 104/137 [00:21<00:05,  5.50it/s, epoch=12, grad_norm=75.845, loss=297.2979, lr=1.39e-04]
Epoch 12 TRAIN:  78%|███████▊  | 107/137 [00:21<00:05,  5.50it/s, epoch=12, grad_norm=75.845, loss=297.2979, lr=1.39e-04]
Epoch 12 TRAIN:  78%|███████▊  | 107/137 [00:21<00:05,  5.50it/s, epoch=12, grad_norm=105.231, loss=283.6781, lr=1.39e-04]
Epoch 12 TRAIN:  78%|███████▊  | 107/137 [00:22<00:05,  5.50it/s, epoch=12, grad_norm=128.911, loss=326.4331, lr=1.39e-04]
Epoch 12 TRAIN:  78%|███████▊  | 107/137 [00:22<00:05,  5.50it/s, epoch=12, grad_norm=99.357, loss=303.7496, lr=1.39e-04] 
Epoch 12 TRAIN:  80%|████████  | 110/137 [00:22<00:05,  5.31it/s, epoch=12, grad_norm=99.357, loss=303.7496, lr=1.39e-04]
Epoch 12 TRAIN:  80%|████████  | 110/137 [00:22<00:05,  5.31it/s, epoch=12, grad_norm=65.217, loss=237.4632, lr=1.40e-04]
Epoch 12 TRAIN:  80%|████████  | 110/137 [00:22<00:05,  5.31it/s, epoch=12, grad_norm=83.127, loss=276.5039, lr=1.40e-04]
Epoch 12 TRAIN:  80%|████████  | 110/137 [00:22<00:05,  5.31it/s, epoch=12, grad_norm=106.311, loss=322.7200, lr=1.40e-04]
Epoch 12 TRAIN:  82%|████████▏ | 113/137 [00:22<00:04,  5.48it/s, epoch=12, grad_norm=106.311, loss=322.7200, lr=1.40e-04]
Epoch 12 TRAIN:  82%|████████▏ | 113/137 [00:23<00:04,  5.48it/s, epoch=12, grad_norm=88.138, loss=343.6362, lr=1.40e-04] 
Epoch 12 TRAIN:  82%|████████▏ | 113/137 [00:23<00:04,  5.48it/s, epoch=12, grad_norm=103.350, loss=303.6852, lr=1.40e-04]
Epoch 12 TRAIN:  82%|████████▏ | 113/137 [00:23<00:04,  5.48it/s, epoch=12, grad_norm=136.529, loss=410.0623, lr=1.40e-04]
Epoch 12 TRAIN:  85%|████████▍ | 116/137 [00:23<00:03,  5.35it/s, epoch=12, grad_norm=136.529, loss=410.0623, lr=1.40e-04]
Epoch 12 TRAIN:  85%|████████▍ | 116/137 [00:23<00:03,  5.35it/s, epoch=12, grad_norm=84.822, loss=305.9509, lr=1.40e-04] 
Epoch 12 TRAIN:  85%|████████▍ | 116/137 [00:23<00:03,  5.35it/s, epoch=12, grad_norm=94.422, loss=348.1735, lr=1.40e-04]
Epoch 12 TRAIN:  85%|████████▍ | 116/137 [00:24<00:03,  5.35it/s, epoch=12, grad_norm=115.265, loss=409.1454, lr=1.40e-04]
Epoch 12 TRAIN:  87%|████████▋ | 119/137 [00:24<00:03,  5.29it/s, epoch=12, grad_norm=115.265, loss=409.1454, lr=1.40e-04]
Epoch 12 TRAIN:  87%|████████▋ | 119/137 [00:24<00:03,  5.29it/s, epoch=12, grad_norm=104.481, loss=376.3438, lr=1.40e-04]
Epoch 12 TRAIN:  87%|████████▋ | 119/137 [00:24<00:03,  5.29it/s, epoch=12, grad_norm=203.928, loss=379.0018, lr=1.40e-04]
Epoch 12 TRAIN:  87%|████████▋ | 119/137 [00:24<00:03,  5.29it/s, epoch=12, grad_norm=250.844, loss=410.8696, lr=1.40e-04]
Epoch 12 TRAIN:  87%|████████▋ | 119/137 [00:24<00:03,  5.29it/s, epoch=12, grad_norm=143.005, loss=384.5354, lr=1.40e-04]
Epoch 12 TRAIN:  90%|████████▉ | 123/137 [00:24<00:02,  5.50it/s, epoch=12, grad_norm=143.005, loss=384.5354, lr=1.40e-04]
Epoch 12 TRAIN:  90%|████████▉ | 123/137 [00:24<00:02,  5.50it/s, epoch=12, grad_norm=214.254, loss=278.7767, lr=1.41e-04]
Epoch 12 TRAIN:  90%|████████▉ | 123/137 [00:25<00:02,  5.50it/s, epoch=12, grad_norm=96.076, loss=371.0203, lr=1.41e-04] 
Epoch 12 TRAIN:  90%|████████▉ | 123/137 [00:25<00:02,  5.50it/s, epoch=12, grad_norm=112.922, loss=353.5880, lr=1.41e-04]
Epoch 12 TRAIN:  92%|█████████▏| 126/137 [00:25<00:02,  5.45it/s, epoch=12, grad_norm=112.922, loss=353.5880, lr=1.41e-04]
Epoch 12 TRAIN:  92%|█████████▏| 126/137 [00:25<00:02,  5.45it/s, epoch=12, grad_norm=78.376, loss=272.2419, lr=1.41e-04] 
Epoch 12 TRAIN:  92%|█████████▏| 126/137 [00:25<00:02,  5.45it/s, epoch=12, grad_norm=130.852, loss=394.6146, lr=1.41e-04]
Epoch 12 TRAIN:  92%|█████████▏| 126/137 [00:25<00:02,  5.45it/s, epoch=12, grad_norm=143.864, loss=390.0692, lr=1.41e-04]
Epoch 12 TRAIN:  94%|█████████▍| 129/137 [00:25<00:01,  5.57it/s, epoch=12, grad_norm=143.864, loss=390.0692, lr=1.41e-04]
Epoch 12 TRAIN:  94%|█████████▍| 129/137 [00:25<00:01,  5.57it/s, epoch=12, grad_norm=150.939, loss=377.4366, lr=1.41e-04]
Epoch 12 TRAIN:  94%|█████████▍| 129/137 [00:26<00:01,  5.57it/s, epoch=12, grad_norm=168.396, loss=370.3687, lr=1.41e-04]
Epoch 12 TRAIN:  94%|█████████▍| 129/137 [00:26<00:01,  5.57it/s, epoch=12, grad_norm=95.040, loss=281.0503, lr=1.41e-04] 
Epoch 12 TRAIN:  94%|█████████▍| 129/137 [00:26<00:01,  5.57it/s, epoch=12, grad_norm=70.029, loss=301.2725, lr=1.41e-04]
Epoch 12 TRAIN:  97%|█████████▋| 133/137 [00:26<00:00,  5.89it/s, epoch=12, grad_norm=70.029, loss=301.2725, lr=1.41e-04]
Epoch 12 TRAIN:  97%|█████████▋| 133/137 [00:26<00:00,  5.89it/s, epoch=12, grad_norm=91.376, loss=355.9145, lr=1.41e-04]
Epoch 12 TRAIN:  97%|█████████▋| 133/137 [00:26<00:00,  5.89it/s, epoch=12, grad_norm=133.350, loss=360.5591, lr=1.41e-04]
Epoch 12 TRAIN:  97%|█████████▋| 133/137 [00:26<00:00,  5.89it/s, epoch=12, grad_norm=148.196, loss=312.7324, lr=1.42e-04]
Epoch 12 TRAIN:  99%|█████████▉| 136/137 [00:26<00:00,  5.79it/s, epoch=12, grad_norm=148.196, loss=312.7324, lr=1.42e-04]
Epoch 12 TRAIN:  99%|█████████▉| 136/137 [00:27<00:00,  5.79it/s, epoch=12, grad_norm=144.659, loss=313.3102, lr=1.42e-04]
Epoch 12 TRAIN: 100%|██████████| 137/137 [00:27<00:00,  5.07it/s, epoch=12, grad_norm=144.659, loss=313.3102, lr=1.42e-04]

CV:   0%|          | 0/6 [00:00<?, ?it/s]/home/rookie/Documents/batuka/projects/wenet/wenet/wenet/utils/train_utils.py:689: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  amp_autocast(enabled=dtype is not None,

CV:  17%|█▋        | 1/6 [00:00<00:03,  1.29it/s]
                                                 
2026-04-16 19:58:12,060 INFO Epoch 12 Step 1770 CV info lr 1.4160e-04 cv_loss 406.0129560990767 rank 0 acc 0.2466669057806333
2026-04-16 19:58:12,061 INFO [Rank 0] Checkpoint: save to checkpoint exp/conformer_mongolian/epoch_12.pt
2026-04-16 19:58:12,174 INFO Epoch 13 Step 1770 TRAIN info lr 1.4160e-04 rank 0
2026-04-16 19:58:12,194 INFO using accumulate grad, new batch size is 1 times larger than before

Epoch 13 TRAIN:   0%|          | 0/137 [00:00<?, ?it/s]
Epoch 13 TRAIN:   0%|          | 0/137 [00:03<?, ?it/s, epoch=13, grad_norm=595.436, loss=504.2571, lr=1.42e-04]
Epoch 13 TRAIN:   1%|          | 1/137 [00:03<08:26,  3.72s/it, epoch=13, grad_norm=595.436, loss=504.2571, lr=1.42e-04]
Epoch 13 TRAIN:   1%|          | 1/137 [00:03<08:26,  3.72s/it, epoch=13, grad_norm=386.749, loss=361.8398, lr=1.42e-04]
Epoch 13 TRAIN:   1%|          | 1/137 [00:04<08:26,  3.72s/it, epoch=13, grad_norm=438.358, loss=364.5555, lr=1.42e-04]
Epoch 13 TRAIN:   2%|▏         | 3/137 [00:04<02:34,  1.15s/it, epoch=13, grad_norm=438.358, loss=364.5555, lr=1.42e-04]
Epoch 13 TRAIN:   2%|▏         | 3/137 [00:04<02:34,  1.15s/it, epoch=13, grad_norm=475.150, loss=410.8054, lr=1.42e-04]
Epoch 13 TRAIN:   2%|▏         | 3/137 [00:04<02:34,  1.15s/it, epoch=13, grad_norm=329.574, loss=313.6078, lr=1.42e-04]
Epoch 13 TRAIN:   2%|▏         | 3/137 [00:04<02:34,  1.15s/it, epoch=13, grad_norm=233.256, loss=284.2267, lr=1.42e-04]
Epoch 13 TRAIN:   2%|▏         | 3/137 [00:04<02:34,  1.15s/it, epoch=13, grad_norm=367.876, loss=389.8443, lr=1.42e-04]
Epoch 13 TRAIN:   2%|▏         | 3/137 [00:04<02:34,  1.15s/it, epoch=13, grad_norm=464.721, loss=461.3960, lr=1.42e-04]
Epoch 13 TRAIN:   2%|▏         | 3/137 [00:04<02:34,  1.15s/it, epoch=13, grad_norm=209.592, loss=362.1947, lr=1.42e-04]
Epoch 13 TRAIN:   2%|▏         | 3/137 [00:04<02:34,  1.15s/it, epoch=13, grad_norm=92.760, loss=314.0698, lr=1.42e-04] 
Epoch 13 TRAIN:   7%|▋         | 10/137 [00:04<00:40,  3.15it/s, epoch=13, grad_norm=92.760, loss=314.0698, lr=1.42e-04]
Epoch 13 TRAIN:   7%|▋         | 10/137 [00:05<00:40,  3.15it/s, epoch=13, grad_norm=106.666, loss=299.3815, lr=1.42e-04]
Epoch 13 TRAIN:   7%|▋         | 10/137 [00:05<00:40,  3.15it/s, epoch=13, grad_norm=89.613, loss=353.1417, lr=1.43e-04] 
Epoch 13 TRAIN:   7%|▋         | 10/137 [00:05<00:40,  3.15it/s, epoch=13, grad_norm=110.633, loss=331.3566, lr=1.43e-04]
Epoch 13 TRAIN:   7%|▋         | 10/137 [00:05<00:40,  3.15it/s, epoch=13, grad_norm=74.130, loss=354.4694, lr=1.43e-04] 
Epoch 13 TRAIN:  10%|█         | 14/137 [00:05<00:31,  3.90it/s, epoch=13, grad_norm=74.130, loss=354.4694, lr=1.43e-04]
Epoch 13 TRAIN:  10%|█         | 14/137 [00:05<00:31,  3.90it/s, epoch=13, grad_norm=97.059, loss=328.2275, lr=1.43e-04]
Epoch 13 TRAIN:  10%|█         | 14/137 [00:05<00:31,  3.90it/s, epoch=13, grad_norm=59.139, loss=335.0433, lr=1.43e-04]
Epoch 13 TRAIN:  10%|█         | 14/137 [00:05<00:31,  3.90it/s, epoch=13, grad_norm=79.066, loss=335.7256, lr=1.43e-04]
Epoch 13 TRAIN:  10%|█         | 14/137 [00:06<00:31,  3.90it/s, epoch=13, grad_norm=103.566, loss=310.1561, lr=1.43e-04]
Epoch 13 TRAIN:  13%|█▎        | 18/137 [00:06<00:26,  4.56it/s, epoch=13, grad_norm=103.566, loss=310.1561, lr=1.43e-04]
Epoch 13 TRAIN:  13%|█▎        | 18/137 [00:06<00:26,  4.56it/s, epoch=13, grad_norm=56.537, loss=321.3253, lr=1.43e-04] 
Epoch 13 TRAIN:  13%|█▎        | 18/137 [00:06<00:26,  4.56it/s, epoch=13, grad_norm=74.891, loss=284.2201, lr=1.43e-04]
Epoch 13 TRAIN:  13%|█▎        | 18/137 [00:06<00:26,  4.56it/s, epoch=13, grad_norm=90.762, loss=317.3707, lr=1.43e-04]
Epoch 13 TRAIN:  13%|█▎        | 18/137 [00:06<00:26,  4.56it/s, epoch=13, grad_norm=152.373, loss=322.7214, lr=1.43e-04]
Epoch 13 TRAIN:  16%|█▌        | 22/137 [00:06<00:22,  5.06it/s, epoch=13, grad_norm=152.373, loss=322.7214, lr=1.43e-04]
Epoch 13 TRAIN:  16%|█▌        | 22/137 [00:06<00:22,  5.06it/s, epoch=13, grad_norm=273.654, loss=417.7565, lr=1.43e-04]
Epoch 13 TRAIN:  16%|█▌        | 22/137 [00:07<00:22,  5.06it/s, epoch=13, grad_norm=162.377, loss=333.7683, lr=1.44e-04]
Epoch 13 TRAIN:  16%|█▌        | 22/137 [00:07<00:22,  5.06it/s, epoch=13, grad_norm=84.511, loss=325.4689, lr=1.44e-04] 
Epoch 13 TRAIN:  16%|█▌        | 22/137 [00:07<00:22,  5.06it/s, epoch=13, grad_norm=69.624, loss=323.9277, lr=1.44e-04]
Epoch 13 TRAIN:  19%|█▉        | 26/137 [00:07<00:20,  5.43it/s, epoch=13, grad_norm=69.624, loss=323.9277, lr=1.44e-04]
Epoch 13 TRAIN:  19%|█▉        | 26/137 [00:07<00:20,  5.43it/s, epoch=13, grad_norm=61.073, loss=357.5927, lr=1.44e-04]
Epoch 13 TRAIN:  19%|█▉        | 26/137 [00:07<00:20,  5.43it/s, epoch=13, grad_norm=131.957, loss=306.8570, lr=1.44e-04]
Epoch 13 TRAIN:  19%|█▉        | 26/137 [00:07<00:20,  5.43it/s, epoch=13, grad_norm=88.022, loss=357.6158, lr=1.44e-04] 
Epoch 13 TRAIN:  19%|█▉        | 26/137 [00:08<00:20,  5.43it/s, epoch=13, grad_norm=67.036, loss=357.7765, lr=1.44e-04]
Epoch 13 TRAIN:  22%|██▏       | 30/137 [00:08<00:18,  5.68it/s, epoch=13, grad_norm=67.036, loss=357.7765, lr=1.44e-04]
Epoch 13 TRAIN:  22%|██▏       | 30/137 [00:08<00:18,  5.68it/s, epoch=13, grad_norm=160.398, loss=274.5545, lr=1.44e-04]
Epoch 13 TRAIN:  22%|██▏       | 30/137 [00:08<00:18,  5.68it/s, epoch=13, grad_norm=74.470, loss=324.8301, lr=1.44e-04] 
Epoch 13 TRAIN:  22%|██▏       | 30/137 [00:08<00:18,  5.68it/s, epoch=13, grad_norm=72.348, loss=355.9833, lr=1.44e-04]
Epoch 13 TRAIN:  22%|██▏       | 30/137 [00:08<00:18,  5.68it/s, epoch=13, grad_norm=93.307, loss=303.4409, lr=1.44e-04]
Epoch 13 TRAIN:  25%|██▍       | 34/137 [00:08<00:18,  5.71it/s, epoch=13, grad_norm=93.307, loss=303.4409, lr=1.44e-04]
Epoch 13 TRAIN:  25%|██▍       | 34/137 [00:08<00:18,  5.71it/s, epoch=13, grad_norm=56.660, loss=314.5654, lr=1.44e-04]
Epoch 13 TRAIN:  25%|██▍       | 34/137 [00:09<00:18,  5.71it/s, epoch=13, grad_norm=121.933, loss=360.3802, lr=1.44e-04]
Epoch 13 TRAIN:  25%|██▍       | 34/137 [00:09<00:18,  5.71it/s, epoch=13, grad_norm=142.870, loss=376.2484, lr=1.45e-04]
Epoch 13 TRAIN:  25%|██▍       | 34/137 [00:09<00:18,  5.71it/s, epoch=13, grad_norm=146.837, loss=354.3063, lr=1.45e-04]
Epoch 13 TRAIN:  28%|██▊       | 38/137 [00:09<00:16,  5.86it/s, epoch=13, grad_norm=146.837, loss=354.3063, lr=1.45e-04]
Epoch 13 TRAIN:  28%|██▊       | 38/137 [00:09<00:16,  5.86it/s, epoch=13, grad_norm=93.365, loss=309.9766, lr=1.45e-04] 
Epoch 13 TRAIN:  28%|██▊       | 38/137 [00:09<00:16,  5.86it/s, epoch=13, grad_norm=103.219, loss=346.3291, lr=1.45e-04]
Epoch 13 TRAIN:  28%|██▊       | 38/137 [00:09<00:16,  5.86it/s, epoch=13, grad_norm=196.918, loss=240.5226, lr=1.45e-04]
Epoch 13 TRAIN:  28%|██▊       | 38/137 [00:10<00:16,  5.86it/s, epoch=13, grad_norm=82.176, loss=340.3658, lr=1.45e-04] 
Epoch 13 TRAIN:  31%|███       | 42/137 [00:10<00:16,  5.77it/s, epoch=13, grad_norm=82.176, loss=340.3658, lr=1.45e-04]
Epoch 13 TRAIN:  31%|███       | 42/137 [00:10<00:16,  5.77it/s, epoch=13, grad_norm=89.503, loss=330.0929, lr=1.45e-04]
Epoch 13 TRAIN:  31%|███       | 42/137 [00:10<00:16,  5.77it/s, epoch=13, grad_norm=74.743, loss=341.1946, lr=1.45e-04]
Epoch 13 TRAIN:  31%|███       | 42/137 [00:10<00:16,  5.77it/s, epoch=13, grad_norm=77.344, loss=335.6344, lr=1.45e-04]
Epoch 13 TRAIN:  31%|███       | 42/137 [00:10<00:16,  5.77it/s, epoch=13, grad_norm=93.380, loss=278.2745, lr=1.45e-04]
Epoch 13 TRAIN:  34%|███▎      | 46/137 [00:10<00:16,  5.67it/s, epoch=13, grad_norm=93.380, loss=278.2745, lr=1.45e-04]
Epoch 13 TRAIN:  34%|███▎      | 46/137 [00:10<00:16,  5.67it/s, epoch=13, grad_norm=105.587, loss=333.7468, lr=1.45e-04]
Epoch 13 TRAIN:  34%|███▎      | 46/137 [00:11<00:16,  5.67it/s, epoch=13, grad_norm=90.749, loss=298.0100, lr=1.45e-04] 
Epoch 13 TRAIN:  34%|███▎      | 46/137 [00:11<00:16,  5.67it/s, epoch=13, grad_norm=71.867, loss=332.1358, lr=1.46e-04]
Epoch 13 TRAIN:  36%|███▌      | 49/137 [00:11<00:15,  5.67it/s, epoch=13, grad_norm=71.867, loss=332.1358, lr=1.46e-04]
Epoch 13 TRAIN:  36%|███▌      | 49/137 [00:11<00:15,  5.67it/s, epoch=13, grad_norm=90.019, loss=358.8913, lr=1.46e-04]
Epoch 13 TRAIN:  36%|███▌      | 49/137 [00:11<00:15,  5.67it/s, epoch=13, grad_norm=90.568, loss=288.0150, lr=1.46e-04]
Epoch 13 TRAIN:  36%|███▌      | 49/137 [00:11<00:15,  5.67it/s, epoch=13, grad_norm=75.991, loss=276.5967, lr=1.46e-04]
Epoch 13 TRAIN:  38%|███▊      | 52/137 [00:11<00:15,  5.65it/s, epoch=13, grad_norm=75.991, loss=276.5967, lr=1.46e-04]
Epoch 13 TRAIN:  38%|███▊      | 52/137 [00:12<00:15,  5.65it/s, epoch=13, grad_norm=81.173, loss=319.3172, lr=1.46e-04]
Epoch 13 TRAIN:  38%|███▊      | 52/137 [00:12<00:15,  5.65it/s, epoch=13, grad_norm=86.098, loss=298.2673, lr=1.46e-04]
Epoch 13 TRAIN:  38%|███▊      | 52/137 [00:12<00:15,  5.65it/s, epoch=13, grad_norm=98.710, loss=319.8970, lr=1.46e-04]
Epoch 13 TRAIN:  40%|████      | 55/137 [00:12<00:14,  5.64it/s, epoch=13, grad_norm=98.710, loss=319.8970, lr=1.46e-04]
Epoch 13 TRAIN:  40%|████      | 55/137 [00:12<00:14,  5.64it/s, epoch=13, grad_norm=130.331, loss=350.0808, lr=1.46e-04]
Epoch 13 TRAIN:  40%|████      | 55/137 [00:12<00:14,  5.64it/s, epoch=13, grad_norm=117.871, loss=365.8762, lr=1.46e-04]
Epoch 13 TRAIN:  40%|████      | 55/137 [00:12<00:14,  5.64it/s, epoch=13, grad_norm=99.501, loss=309.9155, lr=1.46e-04] 
Epoch 13 TRAIN:  42%|████▏     | 58/137 [00:12<00:14,  5.61it/s, epoch=13, grad_norm=99.501, loss=309.9155, lr=1.46e-04]
Epoch 13 TRAIN:  42%|████▏     | 58/137 [00:13<00:14,  5.61it/s, epoch=13, grad_norm=91.866, loss=353.0518, lr=1.46e-04]
Epoch 13 TRAIN:  42%|████▏     | 58/137 [00:13<00:14,  5.61it/s, epoch=13, grad_norm=98.582, loss=365.9754, lr=1.46e-04]
Epoch 13 TRAIN:  42%|████▏     | 58/137 [00:13<00:14,  5.61it/s, epoch=13, grad_norm=70.056, loss=355.1080, lr=1.46e-04]
Epoch 13 TRAIN:  45%|████▍     | 61/137 [00:13<00:14,  5.38it/s, epoch=13, grad_norm=70.056, loss=355.1080, lr=1.46e-04]
Epoch 13 TRAIN:  45%|████▍     | 61/137 [00:13<00:14,  5.38it/s, epoch=13, grad_norm=88.682, loss=354.0208, lr=1.47e-04]
Epoch 13 TRAIN:  45%|████▍     | 61/137 [00:13<00:14,  5.38it/s, epoch=13, grad_norm=75.271, loss=319.0041, lr=1.47e-04]
Epoch 13 TRAIN:  45%|████▍     | 61/137 [00:14<00:14,  5.38it/s, epoch=13, grad_norm=129.426, loss=273.6383, lr=1.47e-04]
Epoch 13 TRAIN:  47%|████▋     | 64/137 [00:14<00:13,  5.33it/s, epoch=13, grad_norm=129.426, loss=273.6383, lr=1.47e-04]
Epoch 13 TRAIN:  47%|████▋     | 64/137 [00:14<00:13,  5.33it/s, epoch=13, grad_norm=189.426, loss=309.8145, lr=1.47e-04]
Epoch 13 TRAIN:  47%|████▋     | 64/137 [00:14<00:13,  5.33it/s, epoch=13, grad_norm=148.297, loss=304.6547, lr=1.47e-04]
Epoch 13 TRAIN:  47%|████▋     | 64/137 [00:14<00:13,  5.33it/s, epoch=13, grad_norm=219.676, loss=242.1285, lr=1.47e-04]
Epoch 13 TRAIN:  49%|████▉     | 67/137 [00:14<00:12,  5.50it/s, epoch=13, grad_norm=219.676, loss=242.1285, lr=1.47e-04]
Epoch 13 TRAIN:  49%|████▉     | 67/137 [00:14<00:12,  5.50it/s, epoch=13, grad_norm=101.006, loss=349.2343, lr=1.47e-04]
Epoch 13 TRAIN:  49%|████▉     | 67/137 [00:14<00:12,  5.50it/s, epoch=13, grad_norm=96.598, loss=297.5446, lr=1.47e-04] 
Epoch 13 TRAIN:  49%|████▉     | 67/137 [00:15<00:12,  5.50it/s, epoch=13, grad_norm=169.460, loss=346.5020, lr=1.47e-04]
Epoch 13 TRAIN:  49%|████▉     | 67/137 [00:15<00:12,  5.50it/s, epoch=13, grad_norm=230.785, loss=357.1326, lr=1.47e-04]
Epoch 13 TRAIN:  52%|█████▏    | 71/137 [00:15<00:11,  5.62it/s, epoch=13, grad_norm=230.785, loss=357.1326, lr=1.47e-04]
Epoch 13 TRAIN:  52%|█████▏    | 71/137 [00:15<00:11,  5.62it/s, epoch=13, grad_norm=123.266, loss=293.3816, lr=1.47e-04]
Epoch 13 TRAIN:  52%|█████▏    | 71/137 [00:15<00:11,  5.62it/s, epoch=13, grad_norm=79.339, loss=274.5281, lr=1.47e-04] 
Epoch 13 TRAIN:  52%|█████▏    | 71/137 [00:15<00:11,  5.62it/s, epoch=13, grad_norm=67.293, loss=309.6139, lr=1.48e-04]
Epoch 13 TRAIN:  54%|█████▍    | 74/137 [00:15<00:11,  5.39it/s, epoch=13, grad_norm=67.293, loss=309.6139, lr=1.48e-04]
Epoch 13 TRAIN:  54%|█████▍    | 74/137 [00:16<00:11,  5.39it/s, epoch=13, grad_norm=76.191, loss=307.7661, lr=1.48e-04]
Epoch 13 TRAIN:  54%|█████▍    | 74/137 [00:16<00:11,  5.39it/s, epoch=13, grad_norm=114.488, loss=385.1199, lr=1.48e-04]
Epoch 13 TRAIN:  54%|█████▍    | 74/137 [00:16<00:11,  5.39it/s, epoch=13, grad_norm=97.677, loss=268.3126, lr=1.48e-04] 
Epoch 13 TRAIN:  56%|█████▌    | 77/137 [00:16<00:10,  5.52it/s, epoch=13, grad_norm=97.677, loss=268.3126, lr=1.48e-04]
Epoch 13 TRAIN:  56%|█████▌    | 77/137 [00:16<00:10,  5.52it/s, epoch=13, grad_norm=165.666, loss=334.5278, lr=1.48e-04]
Epoch 13 TRAIN:  56%|█████▌    | 77/137 [00:16<00:10,  5.52it/s, epoch=13, grad_norm=101.234, loss=312.8383, lr=1.48e-04]
Epoch 13 TRAIN:  56%|█████▌    | 77/137 [00:17<00:10,  5.52it/s, epoch=13, grad_norm=69.412, loss=302.5748, lr=1.48e-04] 
Epoch 13 TRAIN:  58%|█████▊    | 80/137 [00:17<00:10,  5.44it/s, epoch=13, grad_norm=69.412, loss=302.5748, lr=1.48e-04]
Epoch 13 TRAIN:  58%|█████▊    | 80/137 [00:17<00:10,  5.44it/s, epoch=13, grad_norm=239.639, loss=331.6386, lr=1.48e-04]
Epoch 13 TRAIN:  58%|█████▊    | 80/137 [00:17<00:10,  5.44it/s, epoch=13, grad_norm=200.041, loss=319.8484, lr=1.48e-04]
Epoch 13 TRAIN:  58%|█████▊    | 80/137 [00:17<00:10,  5.44it/s, epoch=13, grad_norm=155.708, loss=308.7979, lr=1.48e-04]
Epoch 13 TRAIN:  61%|██████    | 83/137 [00:17<00:10,  5.36it/s, epoch=13, grad_norm=155.708, loss=308.7979, lr=1.48e-04]
Epoch 13 TRAIN:  61%|██████    | 83/137 [00:17<00:10,  5.36it/s, epoch=13, grad_norm=121.689, loss=327.4375, lr=1.48e-04]
Epoch 13 TRAIN:  61%|██████    | 83/137 [00:17<00:10,  5.36it/s, epoch=13, grad_norm=76.903, loss=326.1998, lr=1.48e-04] 
Epoch 13 TRAIN:  61%|██████    | 83/137 [00:18<00:10,  5.36it/s, epoch=13, grad_norm=116.007, loss=337.3620, lr=1.48e-04]
Epoch 13 TRAIN:  63%|██████▎   | 86/137 [00:18<00:09,  5.33it/s, epoch=13, grad_norm=116.007, loss=337.3620, lr=1.48e-04]
Epoch 13 TRAIN:  63%|██████▎   | 86/137 [00:18<00:09,  5.33it/s, epoch=13, grad_norm=128.150, loss=293.9337, lr=1.49e-04]
Epoch 13 TRAIN:  63%|██████▎   | 86/137 [00:18<00:09,  5.33it/s, epoch=13, grad_norm=162.123, loss=341.4045, lr=1.49e-04]
Epoch 13 TRAIN:  63%|██████▎   | 86/137 [00:18<00:09,  5.33it/s, epoch=13, grad_norm=171.517, loss=282.1000, lr=1.49e-04]
Epoch 13 TRAIN:  65%|██████▍   | 89/137 [00:18<00:09,  5.06it/s, epoch=13, grad_norm=171.517, loss=282.1000, lr=1.49e-04]
Epoch 13 TRAIN:  65%|██████▍   | 89/137 [00:18<00:09,  5.06it/s, epoch=13, grad_norm=153.418, loss=267.8824, lr=1.49e-04]
Epoch 13 TRAIN:  65%|██████▍   | 89/137 [00:19<00:09,  5.06it/s, epoch=13, grad_norm=121.470, loss=239.0720, lr=1.49e-04]
Epoch 13 TRAIN:  65%|██████▍   | 89/137 [00:19<00:09,  5.06it/s, epoch=13, grad_norm=121.041, loss=327.3160, lr=1.49e-04]
Epoch 13 TRAIN:  67%|██████▋   | 92/137 [00:19<00:08,  5.23it/s, epoch=13, grad_norm=121.041, loss=327.3160, lr=1.49e-04]
Epoch 13 TRAIN:  67%|██████▋   | 92/137 [00:19<00:08,  5.23it/s, epoch=13, grad_norm=162.836, loss=360.3707, lr=1.49e-04]
Epoch 13 TRAIN:  67%|██████▋   | 92/137 [00:19<00:08,  5.23it/s, epoch=13, grad_norm=140.704, loss=305.7688, lr=1.49e-04]
Epoch 13 TRAIN:  67%|██████▋   | 92/137 [00:19<00:08,  5.23it/s, epoch=13, grad_norm=299.005, loss=392.8313, lr=1.49e-04]
Epoch 13 TRAIN:  69%|██████▉   | 95/137 [00:19<00:08,  5.18it/s, epoch=13, grad_norm=299.005, loss=392.8313, lr=1.49e-04]
Epoch 13 TRAIN:  69%|██████▉   | 95/137 [00:20<00:08,  5.18it/s, epoch=13, grad_norm=262.808, loss=390.9507, lr=1.49e-04]
Epoch 13 TRAIN:  69%|██████▉   | 95/137 [00:20<00:08,  5.18it/s, epoch=13, grad_norm=117.603, loss=339.3857, lr=1.49e-04]
Epoch 13 TRAIN:  69%|██████▉   | 95/137 [00:20<00:08,  5.18it/s, epoch=13, grad_norm=214.467, loss=285.3718, lr=1.49e-04]
Epoch 13 TRAIN:  72%|███████▏  | 98/137 [00:20<00:07,  5.12it/s, epoch=13, grad_norm=214.467, loss=285.3718, lr=1.49e-04]
Epoch 13 TRAIN:  72%|███████▏  | 98/137 [00:20<00:07,  5.12it/s, epoch=13, grad_norm=253.519, loss=210.7565, lr=1.50e-04]
Epoch 13 TRAIN:  72%|███████▏  | 98/137 [00:20<00:07,  5.12it/s, epoch=13, grad_norm=185.141, loss=307.9016, lr=1.50e-04]2026-04-16 19:58:33,174 DEBUG TRAIN | steps/sec 4.812| Batch 13/100 loss 307.901642 loss_att 277.367218 loss_ctc 379.148682 th_accuracy 0.219451 lr 1.4960e-04 grad_norm 185.141052 rank 0

Epoch 13 TRAIN:  72%|███████▏  | 98/137 [00:21<00:07,  5.12it/s, epoch=13, grad_norm=115.265, loss=290.0578, lr=1.50e-04]
Epoch 13 TRAIN:  74%|███████▎  | 101/137 [00:21<00:07,  5.05it/s, epoch=13, grad_norm=115.265, loss=290.0578, lr=1.50e-04]
Epoch 13 TRAIN:  74%|███████▎  | 101/137 [00:21<00:07,  5.05it/s, epoch=13, grad_norm=79.103, loss=289.7017, lr=1.50e-04] 
Epoch 13 TRAIN:  74%|███████▎  | 101/137 [00:21<00:07,  5.05it/s, epoch=13, grad_norm=180.271, loss=335.3818, lr=1.50e-04]
Epoch 13 TRAIN:  74%|███████▎  | 101/137 [00:21<00:07,  5.05it/s, epoch=13, grad_norm=178.348, loss=378.2352, lr=1.50e-04]
Epoch 13 TRAIN:  76%|███████▌  | 104/137 [00:21<00:06,  5.14it/s, epoch=13, grad_norm=178.348, loss=378.2352, lr=1.50e-04]
Epoch 13 TRAIN:  76%|███████▌  | 104/137 [00:21<00:06,  5.14it/s, epoch=13, grad_norm=100.218, loss=358.6979, lr=1.50e-04]
Epoch 13 TRAIN:  76%|███████▌  | 104/137 [00:22<00:06,  5.14it/s, epoch=13, grad_norm=98.898, loss=328.5543, lr=1.50e-04] 
Epoch 13 TRAIN:  76%|███████▌  | 104/137 [00:22<00:06,  5.14it/s, epoch=13, grad_norm=120.209, loss=262.9621, lr=1.50e-04]
Epoch 13 TRAIN:  78%|███████▊  | 107/137 [00:22<00:05,  5.32it/s, epoch=13, grad_norm=120.209, loss=262.9621, lr=1.50e-04]
Epoch 13 TRAIN:  78%|███████▊  | 107/137 [00:22<00:05,  5.32it/s, epoch=13, grad_norm=80.962, loss=297.1761, lr=1.50e-04] 
Epoch 13 TRAIN:  78%|███████▊  | 107/137 [00:22<00:05,  5.32it/s, epoch=13, grad_norm=125.174, loss=225.4706, lr=1.50e-04]
Epoch 13 TRAIN:  78%|███████▊  | 107/137 [00:22<00:05,  5.32it/s, epoch=13, grad_norm=125.535, loss=302.2651, lr=1.50e-04]
Epoch 13 TRAIN:  80%|████████  | 110/137 [00:22<00:04,  5.41it/s, epoch=13, grad_norm=125.535, loss=302.2651, lr=1.50e-04]
Epoch 13 TRAIN:  80%|████████  | 110/137 [00:22<00:04,  5.41it/s, epoch=13, grad_norm=169.846, loss=311.5133, lr=1.50e-04]
Epoch 13 TRAIN:  80%|████████  | 110/137 [00:23<00:04,  5.41it/s, epoch=13, grad_norm=91.347, loss=307.7833, lr=1.51e-04] 
Epoch 13 TRAIN:  80%|████████  | 110/137 [00:23<00:04,  5.41it/s, epoch=13, grad_norm=135.607, loss=400.2057, lr=1.51e-04]
Epoch 13 TRAIN:  82%|████████▏ | 113/137 [00:23<00:04,  5.25it/s, epoch=13, grad_norm=135.607, loss=400.2057, lr=1.51e-04]
Epoch 13 TRAIN:  82%|████████▏ | 113/137 [00:23<00:04,  5.25it/s, epoch=13, grad_norm=107.278, loss=368.7101, lr=1.51e-04]
Epoch 13 TRAIN:  82%|████████▏ | 113/137 [00:23<00:04,  5.25it/s, epoch=13, grad_norm=90.154, loss=340.9111, lr=1.51e-04] 
Epoch 13 TRAIN:  82%|████████▏ | 113/137 [00:23<00:04,  5.25it/s, epoch=13, grad_norm=189.965, loss=274.0401, lr=1.51e-04]
Epoch 13 TRAIN:  85%|████████▍ | 116/137 [00:23<00:04,  5.12it/s, epoch=13, grad_norm=189.965, loss=274.0401, lr=1.51e-04]
Epoch 13 TRAIN:  85%|████████▍ | 116/137 [00:24<00:04,  5.12it/s, epoch=13, grad_norm=165.846, loss=300.0555, lr=1.51e-04]
Epoch 13 TRAIN:  85%|████████▍ | 116/137 [00:24<00:04,  5.12it/s, epoch=13, grad_norm=95.352, loss=352.1804, lr=1.51e-04] 
Epoch 13 TRAIN:  85%|████████▍ | 116/137 [00:24<00:04,  5.12it/s, epoch=13, grad_norm=96.014, loss=232.2411, lr=1.51e-04]
Epoch 13 TRAIN:  87%|████████▋ | 119/137 [00:24<00:03,  5.10it/s, epoch=13, grad_norm=96.014, loss=232.2411, lr=1.51e-04]
Epoch 13 TRAIN:  87%|████████▋ | 119/137 [00:24<00:03,  5.10it/s, epoch=13, grad_norm=121.157, loss=328.6341, lr=1.51e-04]
Epoch 13 TRAIN:  87%|████████▋ | 119/137 [00:24<00:03,  5.10it/s, epoch=13, grad_norm=178.904, loss=454.6140, lr=1.51e-04]
Epoch 13 TRAIN:  87%|████████▋ | 119/137 [00:25<00:03,  5.10it/s, epoch=13, grad_norm=113.064, loss=358.4827, lr=1.51e-04]
Epoch 13 TRAIN:  87%|████████▋ | 119/137 [00:25<00:03,  5.10it/s, epoch=13, grad_norm=101.339, loss=413.5457, lr=1.51e-04]
Epoch 13 TRAIN:  90%|████████▉ | 123/137 [00:25<00:02,  5.42it/s, epoch=13, grad_norm=101.339, loss=413.5457, lr=1.51e-04]
Epoch 13 TRAIN:  90%|████████▉ | 123/137 [00:25<00:02,  5.42it/s, epoch=13, grad_norm=70.509, loss=333.4307, lr=1.52e-04] 
Epoch 13 TRAIN:  90%|████████▉ | 123/137 [00:25<00:02,  5.42it/s, epoch=13, grad_norm=155.258, loss=319.1801, lr=1.52e-04]
Epoch 13 TRAIN:  90%|████████▉ | 123/137 [00:25<00:02,  5.42it/s, epoch=13, grad_norm=170.371, loss=303.1385, lr=1.52e-04]
Epoch 13 TRAIN:  92%|█████████▏| 126/137 [00:25<00:02,  5.22it/s, epoch=13, grad_norm=170.371, loss=303.1385, lr=1.52e-04]
Epoch 13 TRAIN:  92%|█████████▏| 126/137 [00:26<00:02,  5.22it/s, epoch=13, grad_norm=119.035, loss=316.1996, lr=1.52e-04]
Epoch 13 TRAIN:  92%|█████████▏| 126/137 [00:26<00:02,  5.22it/s, epoch=13, grad_norm=235.528, loss=312.1773, lr=1.52e-04]
Epoch 13 TRAIN:  92%|█████████▏| 126/137 [00:26<00:02,  5.22it/s, epoch=13, grad_norm=117.779, loss=329.3335, lr=1.52e-04]
Epoch 13 TRAIN:  94%|█████████▍| 129/137 [00:26<00:01,  5.12it/s, epoch=13, grad_norm=117.779, loss=329.3335, lr=1.52e-04]
Epoch 13 TRAIN:  94%|█████████▍| 129/137 [00:26<00:01,  5.12it/s, epoch=13, grad_norm=182.266, loss=365.9691, lr=1.52e-04]
Epoch 13 TRAIN:  94%|█████████▍| 129/137 [00:26<00:01,  5.12it/s, epoch=13, grad_norm=104.585, loss=293.2396, lr=1.52e-04]
Epoch 13 TRAIN:  94%|█████████▍| 129/137 [00:27<00:01,  5.12it/s, epoch=13, grad_norm=210.949, loss=393.2461, lr=1.52e-04]
Epoch 13 TRAIN:  96%|█████████▋| 132/137 [00:27<00:00,  5.25it/s, epoch=13, grad_norm=210.949, loss=393.2461, lr=1.52e-04]
Epoch 13 TRAIN:  96%|█████████▋| 132/137 [00:27<00:00,  5.25it/s, epoch=13, grad_norm=119.222, loss=359.8134, lr=1.52e-04]
Epoch 13 TRAIN:  96%|█████████▋| 132/137 [00:27<00:00,  5.25it/s, epoch=13, grad_norm=179.882, loss=432.3026, lr=1.52e-04]
Epoch 13 TRAIN:  96%|█████████▋| 132/137 [00:27<00:00,  5.25it/s, epoch=13, grad_norm=153.541, loss=413.0212, lr=1.52e-04]
Epoch 13 TRAIN:  99%|█████████▊| 135/137 [00:27<00:00,  5.35it/s, epoch=13, grad_norm=153.541, loss=413.0212, lr=1.52e-04]
Epoch 13 TRAIN:  99%|█████████▊| 135/137 [00:27<00:00,  5.35it/s, epoch=13, grad_norm=131.271, loss=385.9242, lr=1.52e-04]
Epoch 13 TRAIN:  99%|█████████▉| 136/137 [00:27<00:00,  4.90it/s, epoch=13, grad_norm=131.271, loss=385.9242, lr=1.52e-04]

CV:   0%|          | 0/6 [00:00<?, ?it/s]/home/rookie/Documents/batuka/projects/wenet/wenet/wenet/utils/train_utils.py:689: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  amp_autocast(enabled=dtype is not None,

CV:  17%|█▋        | 1/6 [00:00<00:03,  1.30it/s]
                                                 
2026-04-16 19:58:40,825 INFO Epoch 13 Step 1906 CV info lr 1.5248e-04 cv_loss 383.2944308194247 rank 0 acc 0.24509660402933756
2026-04-16 19:58:40,827 INFO [Rank 0] Checkpoint: save to checkpoint exp/conformer_mongolian/epoch_13.pt
2026-04-16 19:58:40,941 INFO Epoch 14 Step 1906 TRAIN info lr 1.5248e-04 rank 0
2026-04-16 19:58:40,953 INFO using accumulate grad, new batch size is 1 times larger than before

Epoch 14 TRAIN:   0%|          | 0/136 [00:00<?, ?it/s]
Epoch 14 TRAIN:   0%|          | 0/136 [00:04<?, ?it/s, epoch=14, grad_norm=453.574, loss=462.3707, lr=1.53e-04]
Epoch 14 TRAIN:   1%|          | 1/136 [00:04<09:22,  4.17s/it, epoch=14, grad_norm=453.574, loss=462.3707, lr=1.53e-04]
Epoch 14 TRAIN:   1%|          | 1/136 [00:04<09:22,  4.17s/it, epoch=14, grad_norm=189.163, loss=275.2576, lr=1.53e-04]
Epoch 14 TRAIN:   1%|          | 1/136 [00:04<09:22,  4.17s/it, epoch=14, grad_norm=427.304, loss=468.8223, lr=1.53e-04]
Epoch 14 TRAIN:   1%|          | 1/136 [00:04<09:22,  4.17s/it, epoch=14, grad_norm=215.392, loss=336.2111, lr=1.53e-04]
Epoch 14 TRAIN:   1%|          | 1/136 [00:04<09:22,  4.17s/it, epoch=14, grad_norm=251.052, loss=327.9527, lr=1.53e-04]
Epoch 14 TRAIN:   1%|          | 1/136 [00:04<09:22,  4.17s/it, epoch=14, grad_norm=173.783, loss=335.7730, lr=1.53e-04]
Epoch 14 TRAIN:   1%|          | 1/136 [00:04<09:22,  4.17s/it, epoch=14, grad_norm=169.778, loss=224.4518, lr=1.53e-04]
Epoch 14 TRAIN:   1%|          | 1/136 [00:04<09:22,  4.17s/it, epoch=14, grad_norm=150.251, loss=347.4870, lr=1.53e-04]
Epoch 14 TRAIN:   1%|          | 1/136 [00:04<09:22,  4.17s/it, epoch=14, grad_norm=93.851, loss=345.5825, lr=1.53e-04] 
Epoch 14 TRAIN:   7%|▋         | 9/136 [00:04<00:51,  2.45it/s, epoch=14, grad_norm=93.851, loss=345.5825, lr=1.53e-04]
Epoch 14 TRAIN:   7%|▋         | 9/136 [00:04<00:51,  2.45it/s, epoch=14, grad_norm=103.083, loss=326.8956, lr=1.53e-04]
Epoch 14 TRAIN:   7%|▋         | 9/136 [00:05<00:51,  2.45it/s, epoch=14, grad_norm=66.101, loss=371.9150, lr=1.53e-04] 
Epoch 14 TRAIN:   7%|▋         | 9/136 [00:05<00:51,  2.45it/s, epoch=14, grad_norm=73.059, loss=363.7808, lr=1.53e-04]
Epoch 14 TRAIN:   7%|▋         | 9/136 [00:05<00:51,  2.45it/s, epoch=14, grad_norm=84.331, loss=330.7519, lr=1.54e-04]
Epoch 14 TRAIN:  10%|▉         | 13/136 [00:05<00:38,  3.20it/s, epoch=14, grad_norm=84.331, loss=330.7519, lr=1.54e-04]
Epoch 14 TRAIN:  10%|▉         | 13/136 [00:05<00:38,  3.20it/s, epoch=14, grad_norm=126.979, loss=275.1231, lr=1.54e-04]
Epoch 14 TRAIN:  10%|▉         | 13/136 [00:05<00:38,  3.20it/s, epoch=14, grad_norm=77.291, loss=356.3595, lr=1.54e-04] 
Epoch 14 TRAIN:  10%|▉         | 13/136 [00:05<00:38,  3.20it/s, epoch=14, grad_norm=71.920, loss=329.5734, lr=1.54e-04]
Epoch 14 TRAIN:  10%|▉         | 13/136 [00:06<00:38,  3.20it/s, epoch=14, grad_norm=71.321, loss=341.6521, lr=1.54e-04]
Epoch 14 TRAIN:  12%|█▎        | 17/136 [00:06<00:30,  3.91it/s, epoch=14, grad_norm=71.321, loss=341.6521, lr=1.54e-04]
Epoch 14 TRAIN:  12%|█▎        | 17/136 [00:06<00:30,  3.91it/s, epoch=14, grad_norm=70.723, loss=311.5267, lr=1.54e-04]
Epoch 14 TRAIN:  12%|█▎        | 17/136 [00:06<00:30,  3.91it/s, epoch=14, grad_norm=66.728, loss=317.1544, lr=1.54e-04]
Epoch 14 TRAIN:  12%|█▎        | 17/136 [00:06<00:30,  3.91it/s, epoch=14, grad_norm=68.680, loss=343.4239, lr=1.54e-04]
Epoch 14 TRAIN:  12%|█▎        | 17/136 [00:06<00:30,  3.91it/s, epoch=14, grad_norm=92.465, loss=297.4930, lr=1.54e-04]
Epoch 14 TRAIN:  15%|█▌        | 21/136 [00:06<00:25,  4.48it/s, epoch=14, grad_norm=92.465, loss=297.4930, lr=1.54e-04]
Epoch 14 TRAIN:  15%|█▌        | 21/136 [00:06<00:25,  4.48it/s, epoch=14, grad_norm=90.317, loss=343.4246, lr=1.54e-04]
Epoch 14 TRAIN:  15%|█▌        | 21/136 [00:07<00:25,  4.48it/s, epoch=14, grad_norm=95.526, loss=365.6920, lr=1.54e-04]
Epoch 14 TRAIN:  15%|█▌        | 21/136 [00:07<00:25,  4.48it/s, epoch=14, grad_norm=87.014, loss=355.6710, lr=1.54e-04]
Epoch 14 TRAIN:  15%|█▌        | 21/136 [00:07<00:25,  4.48it/s, epoch=14, grad_norm=67.210, loss=328.3864, lr=1.54e-04]
Epoch 14 TRAIN:  18%|█▊        | 25/136 [00:07<00:22,  4.88it/s, epoch=14, grad_norm=67.210, loss=328.3864, lr=1.54e-04]
Epoch 14 TRAIN:  18%|█▊        | 25/136 [00:07<00:22,  4.88it/s, epoch=14, grad_norm=63.717, loss=335.3593, lr=1.55e-04]
Epoch 14 TRAIN:  18%|█▊        | 25/136 [00:07<00:22,  4.88it/s, epoch=14, grad_norm=94.825, loss=327.4842, lr=1.55e-04]
Epoch 14 TRAIN:  18%|█▊        | 25/136 [00:07<00:22,  4.88it/s, epoch=14, grad_norm=86.282, loss=326.5908, lr=1.55e-04]
Epoch 14 TRAIN:  18%|█▊        | 25/136 [00:08<00:22,  4.88it/s, epoch=14, grad_norm=69.844, loss=348.4714, lr=1.55e-04]
Epoch 14 TRAIN:  21%|██▏       | 29/136 [00:08<00:20,  5.23it/s, epoch=14, grad_norm=69.844, loss=348.4714, lr=1.55e-04]
Epoch 14 TRAIN:  21%|██▏       | 29/136 [00:08<00:20,  5.23it/s, epoch=14, grad_norm=81.774, loss=343.7948, lr=1.55e-04]
Epoch 14 TRAIN:  21%|██▏       | 29/136 [00:08<00:20,  5.23it/s, epoch=14, grad_norm=66.034, loss=310.5919, lr=1.55e-04]
Epoch 14 TRAIN:  21%|██▏       | 29/136 [00:08<00:20,  5.23it/s, epoch=14, grad_norm=101.869, loss=283.7204, lr=1.55e-04]
Epoch 14 TRAIN:  21%|██▏       | 29/136 [00:08<00:20,  5.23it/s, epoch=14, grad_norm=74.381, loss=304.5253, lr=1.55e-04] 
Epoch 14 TRAIN:  24%|██▍       | 33/136 [00:08<00:19,  5.34it/s, epoch=14, grad_norm=74.381, loss=304.5253, lr=1.55e-04]
Epoch 14 TRAIN:  24%|██▍       | 33/136 [00:08<00:19,  5.34it/s, epoch=14, grad_norm=63.338, loss=317.2973, lr=1.55e-04]
Epoch 14 TRAIN:  24%|██▍       | 33/136 [00:09<00:19,  5.34it/s, epoch=14, grad_norm=92.043, loss=292.5290, lr=1.55e-04]
Epoch 14 TRAIN:  24%|██▍       | 33/136 [00:09<00:19,  5.34it/s, epoch=14, grad_norm=90.320, loss=343.7500, lr=1.55e-04]
Epoch 14 TRAIN:  24%|██▍       | 33/136 [00:09<00:19,  5.34it/s, epoch=14, grad_norm=89.552, loss=348.7638, lr=1.55e-04]
Epoch 14 TRAIN:  27%|██▋       | 37/136 [00:09<00:18,  5.42it/s, epoch=14, grad_norm=89.552, loss=348.7638, lr=1.55e-04]
Epoch 14 TRAIN:  27%|██▋       | 37/136 [00:09<00:18,  5.42it/s, epoch=14, grad_norm=61.279, loss=278.7124, lr=1.56e-04]
Epoch 14 TRAIN:  27%|██▋       | 37/136 [00:09<00:18,  5.42it/s, epoch=14, grad_norm=82.190, loss=328.4164, lr=1.56e-04]
Epoch 14 TRAIN:  27%|██▋       | 37/136 [00:09<00:18,  5.42it/s, epoch=14, grad_norm=68.278, loss=346.0964, lr=1.56e-04]
Epoch 14 TRAIN:  29%|██▉       | 40/136 [00:09<00:17,  5.53it/s, epoch=14, grad_norm=68.278, loss=346.0964, lr=1.56e-04]
Epoch 14 TRAIN:  29%|██▉       | 40/136 [00:10<00:17,  5.53it/s, epoch=14, grad_norm=59.218, loss=318.7094, lr=1.56e-04]
Epoch 14 TRAIN:  29%|██▉       | 40/136 [00:10<00:17,  5.53it/s, epoch=14, grad_norm=58.895, loss=348.8742, lr=1.56e-04]
Epoch 14 TRAIN:  29%|██▉       | 40/136 [00:10<00:17,  5.53it/s, epoch=14, grad_norm=92.016, loss=285.4041, lr=1.56e-04]
Epoch 14 TRAIN:  32%|███▏      | 43/136 [00:10<00:17,  5.33it/s, epoch=14, grad_norm=92.016, loss=285.4041, lr=1.56e-04]
Epoch 14 TRAIN:  32%|███▏      | 43/136 [00:10<00:17,  5.33it/s, epoch=14, grad_norm=93.546, loss=318.3502, lr=1.56e-04]
Epoch 14 TRAIN:  32%|███▏      | 43/136 [00:10<00:17,  5.33it/s, epoch=14, grad_norm=80.377, loss=335.8373, lr=1.56e-04]
Epoch 14 TRAIN:  32%|███▏      | 43/136 [00:11<00:17,  5.33it/s, epoch=14, grad_norm=124.960, loss=349.1247, lr=1.56e-04]
Epoch 14 TRAIN:  34%|███▍      | 46/136 [00:11<00:16,  5.32it/s, epoch=14, grad_norm=124.960, loss=349.1247, lr=1.56e-04]
Epoch 14 TRAIN:  34%|███▍      | 46/136 [00:11<00:16,  5.32it/s, epoch=14, grad_norm=114.957, loss=364.6874, lr=1.56e-04]
Epoch 14 TRAIN:  34%|███▍      | 46/136 [00:11<00:16,  5.32it/s, epoch=14, grad_norm=68.623, loss=334.0482, lr=1.56e-04] 
Epoch 14 TRAIN:  34%|███▍      | 46/136 [00:11<00:16,  5.32it/s, epoch=14, grad_norm=62.922, loss=301.1884, lr=1.56e-04]
Epoch 14 TRAIN:  36%|███▌      | 49/136 [00:11<00:16,  5.28it/s, epoch=14, grad_norm=62.922, loss=301.1884, lr=1.56e-04]
Epoch 14 TRAIN:  36%|███▌      | 49/136 [00:12<00:16,  5.28it/s, epoch=14, grad_norm=77.643, loss=300.2554, lr=1.56e-04]
Epoch 14 TRAIN:  36%|███▌      | 49/136 [00:12<00:16,  5.28it/s, epoch=14, grad_norm=81.479, loss=328.2393, lr=1.57e-04]
Epoch 14 TRAIN:  36%|███▌      | 49/136 [00:12<00:16,  5.28it/s, epoch=14, grad_norm=55.703, loss=336.6544, lr=1.57e-04]
Epoch 14 TRAIN:  38%|███▊      | 52/136 [00:12<00:15,  5.31it/s, epoch=14, grad_norm=55.703, loss=336.6544, lr=1.57e-04]
Epoch 14 TRAIN:  38%|███▊      | 52/136 [00:12<00:15,  5.31it/s, epoch=14, grad_norm=96.881, loss=313.0948, lr=1.57e-04]
Epoch 14 TRAIN:  38%|███▊      | 52/136 [00:12<00:15,  5.31it/s, epoch=14, grad_norm=72.671, loss=333.9303, lr=1.57e-04]
Epoch 14 TRAIN:  38%|███▊      | 52/136 [00:12<00:15,  5.31it/s, epoch=14, grad_norm=55.756, loss=279.3854, lr=1.57e-04]
Epoch 14 TRAIN:  40%|████      | 55/136 [00:12<00:15,  5.11it/s, epoch=14, grad_norm=55.756, loss=279.3854, lr=1.57e-04]
Epoch 14 TRAIN:  40%|████      | 55/136 [00:13<00:15,  5.11it/s, epoch=14, grad_norm=63.500, loss=315.4775, lr=1.57e-04]
Epoch 14 TRAIN:  40%|████      | 55/136 [00:13<00:15,  5.11it/s, epoch=14, grad_norm=50.068, loss=274.2796, lr=1.57e-04]
Epoch 14 TRAIN:  40%|████      | 55/136 [00:13<00:15,  5.11it/s, epoch=14, grad_norm=112.977, loss=317.0117, lr=1.57e-04]
Epoch 14 TRAIN:  43%|████▎     | 58/136 [00:13<00:14,  5.27it/s, epoch=14, grad_norm=112.977, loss=317.0117, lr=1.57e-04]
Epoch 14 TRAIN:  43%|████▎     | 58/136 [00:13<00:14,  5.27it/s, epoch=14, grad_norm=137.425, loss=354.8931, lr=1.57e-04]
Epoch 14 TRAIN:  43%|████▎     | 58/136 [00:13<00:14,  5.27it/s, epoch=14, grad_norm=75.744, loss=292.9878, lr=1.57e-04] 
Epoch 14 TRAIN:  43%|████▎     | 58/136 [00:14<00:14,  5.27it/s, epoch=14, grad_norm=62.012, loss=354.7523, lr=1.57e-04]
Epoch 14 TRAIN:  45%|████▍     | 61/136 [00:14<00:14,  5.18it/s, epoch=14, grad_norm=62.012, loss=354.7523, lr=1.57e-04]
Epoch 14 TRAIN:  45%|████▍     | 61/136 [00:14<00:14,  5.18it/s, epoch=14, grad_norm=67.685, loss=315.6762, lr=1.57e-04]
Epoch 14 TRAIN:  45%|████▍     | 61/136 [00:14<00:14,  5.18it/s, epoch=14, grad_norm=84.086, loss=353.7227, lr=1.58e-04]
Epoch 14 TRAIN:  45%|████▍     | 61/136 [00:14<00:14,  5.18it/s, epoch=14, grad_norm=74.305, loss=342.1450, lr=1.58e-04]
Epoch 14 TRAIN:  47%|████▋     | 64/136 [00:14<00:13,  5.33it/s, epoch=14, grad_norm=74.305, loss=342.1450, lr=1.58e-04]
Epoch 14 TRAIN:  47%|████▋     | 64/136 [00:14<00:13,  5.33it/s, epoch=14, grad_norm=67.463, loss=369.6599, lr=1.58e-04]
Epoch 14 TRAIN:  47%|████▋     | 64/136 [00:15<00:13,  5.33it/s, epoch=14, grad_norm=152.995, loss=278.4365, lr=1.58e-04]
Epoch 14 TRAIN:  47%|████▋     | 64/136 [00:15<00:13,  5.33it/s, epoch=14, grad_norm=136.634, loss=328.2258, lr=1.58e-04]
Epoch 14 TRAIN:  49%|████▉     | 67/136 [00:15<00:12,  5.35it/s, epoch=14, grad_norm=136.634, loss=328.2258, lr=1.58e-04]
Epoch 14 TRAIN:  49%|████▉     | 67/136 [00:15<00:12,  5.35it/s, epoch=14, grad_norm=336.805, loss=323.0166, lr=1.58e-04]
Epoch 14 TRAIN:  49%|████▉     | 67/136 [00:15<00:12,  5.35it/s, epoch=14, grad_norm=137.181, loss=298.3908, lr=1.58e-04]
Epoch 14 TRAIN:  49%|████▉     | 67/136 [00:15<00:12,  5.35it/s, epoch=14, grad_norm=68.781, loss=281.4298, lr=1.58e-04] 
Epoch 14 TRAIN:  51%|█████▏    | 70/136 [00:15<00:12,  5.37it/s, epoch=14, grad_norm=68.781, loss=281.4298, lr=1.58e-04]
Epoch 14 TRAIN:  51%|█████▏    | 70/136 [00:15<00:12,  5.37it/s, epoch=14, grad_norm=76.639, loss=335.9255, lr=1.58e-04]
Epoch 14 TRAIN:  51%|█████▏    | 70/136 [00:16<00:12,  5.37it/s, epoch=14, grad_norm=56.938, loss=297.3713, lr=1.58e-04]
Epoch 14 TRAIN:  51%|█████▏    | 70/136 [00:16<00:12,  5.37it/s, epoch=14, grad_norm=147.341, loss=353.1887, lr=1.58e-04]
Epoch 14 TRAIN:  54%|█████▎    | 73/136 [00:16<00:12,  5.25it/s, epoch=14, grad_norm=147.341, loss=353.1887, lr=1.58e-04]
Epoch 14 TRAIN:  54%|█████▎    | 73/136 [00:16<00:12,  5.25it/s, epoch=14, grad_norm=96.207, loss=323.4786, lr=1.58e-04] 
Epoch 14 TRAIN:  54%|█████▎    | 73/136 [00:16<00:12,  5.25it/s, epoch=14, grad_norm=71.992, loss=337.4226, lr=1.58e-04]
Epoch 14 TRAIN:  54%|█████▎    | 73/136 [00:16<00:12,  5.25it/s, epoch=14, grad_norm=122.771, loss=295.4125, lr=1.59e-04]
Epoch 14 TRAIN:  56%|█████▌    | 76/136 [00:16<00:11,  5.26it/s, epoch=14, grad_norm=122.771, loss=295.4125, lr=1.59e-04]
Epoch 14 TRAIN:  56%|█████▌    | 76/136 [00:17<00:11,  5.26it/s, epoch=14, grad_norm=109.350, loss=387.4810, lr=1.59e-04]
Epoch 14 TRAIN:  56%|█████▌    | 76/136 [00:17<00:11,  5.26it/s, epoch=14, grad_norm=105.977, loss=342.4850, lr=1.59e-04]
Epoch 14 TRAIN:  56%|█████▌    | 76/136 [00:17<00:11,  5.26it/s, epoch=14, grad_norm=97.093, loss=328.8774, lr=1.59e-04] 
Epoch 14 TRAIN:  58%|█████▊    | 79/136 [00:17<00:11,  5.11it/s, epoch=14, grad_norm=97.093, loss=328.8774, lr=1.59e-04]
Epoch 14 TRAIN:  58%|█████▊    | 79/136 [00:17<00:11,  5.11it/s, epoch=14, grad_norm=74.672, loss=311.3936, lr=1.59e-04]
Epoch 14 TRAIN:  58%|█████▊    | 79/136 [00:17<00:11,  5.11it/s, epoch=14, grad_norm=63.727, loss=323.9995, lr=1.59e-04]
Epoch 14 TRAIN:  58%|█████▊    | 79/136 [00:18<00:11,  5.11it/s, epoch=14, grad_norm=87.161, loss=290.3849, lr=1.59e-04]
Epoch 14 TRAIN:  60%|██████    | 82/136 [00:18<00:10,  5.06it/s, epoch=14, grad_norm=87.161, loss=290.3849, lr=1.59e-04]
Epoch 14 TRAIN:  60%|██████    | 82/136 [00:18<00:10,  5.06it/s, epoch=14, grad_norm=195.377, loss=329.7727, lr=1.59e-04]
Epoch 14 TRAIN:  60%|██████    | 82/136 [00:18<00:10,  5.06it/s, epoch=14, grad_norm=174.013, loss=284.2352, lr=1.59e-04]
Epoch 14 TRAIN:  60%|██████    | 82/136 [00:18<00:10,  5.06it/s, epoch=14, grad_norm=91.270, loss=323.3654, lr=1.59e-04] 
Epoch 14 TRAIN:  62%|██████▎   | 85/136 [00:18<00:09,  5.19it/s, epoch=14, grad_norm=91.270, loss=323.3654, lr=1.59e-04]
Epoch 14 TRAIN:  62%|██████▎   | 85/136 [00:18<00:09,  5.19it/s, epoch=14, grad_norm=88.106, loss=331.1629, lr=1.59e-04]
Epoch 14 TRAIN:  62%|██████▎   | 85/136 [00:19<00:09,  5.19it/s, epoch=14, grad_norm=123.741, loss=319.8092, lr=1.59e-04]
Epoch 14 TRAIN:  62%|██████▎   | 85/136 [00:19<00:09,  5.19it/s, epoch=14, grad_norm=141.749, loss=367.7808, lr=1.60e-04]
Epoch 14 TRAIN:  65%|██████▍   | 88/136 [00:19<00:09,  5.32it/s, epoch=14, grad_norm=141.749, loss=367.7808, lr=1.60e-04]
Epoch 14 TRAIN:  65%|██████▍   | 88/136 [00:19<00:09,  5.32it/s, epoch=14, grad_norm=113.739, loss=331.7791, lr=1.60e-04]
Epoch 14 TRAIN:  65%|██████▍   | 88/136 [00:19<00:09,  5.32it/s, epoch=14, grad_norm=130.411, loss=370.1165, lr=1.60e-04]
Epoch 14 TRAIN:  65%|██████▍   | 88/136 [00:19<00:09,  5.32it/s, epoch=14, grad_norm=211.507, loss=314.0035, lr=1.60e-04]
Epoch 14 TRAIN:  65%|██████▍   | 88/136 [00:19<00:09,  5.32it/s, epoch=14, grad_norm=209.536, loss=287.3387, lr=1.60e-04]
Epoch 14 TRAIN:  68%|██████▊   | 92/136 [00:19<00:07,  5.61it/s, epoch=14, grad_norm=209.536, loss=287.3387, lr=1.60e-04]
Epoch 14 TRAIN:  68%|██████▊   | 92/136 [00:20<00:07,  5.61it/s, epoch=14, grad_norm=179.146, loss=329.6336, lr=1.60e-04]
Epoch 14 TRAIN:  68%|██████▊   | 92/136 [00:20<00:07,  5.61it/s, epoch=14, grad_norm=78.631, loss=281.9245, lr=1.60e-04] 
Epoch 14 TRAIN:  68%|██████▊   | 92/136 [00:20<00:07,  5.61it/s, epoch=14, grad_norm=72.730, loss=313.7957, lr=1.60e-04]
Epoch 14 TRAIN:  70%|██████▉   | 95/136 [00:20<00:07,  5.42it/s, epoch=14, grad_norm=72.730, loss=313.7957, lr=1.60e-04]
Epoch 14 TRAIN:  70%|██████▉   | 95/136 [00:20<00:07,  5.42it/s, epoch=14, grad_norm=71.173, loss=324.2501, lr=1.60e-04]
Epoch 14 TRAIN:  70%|██████▉   | 95/136 [00:20<00:07,  5.42it/s, epoch=14, grad_norm=88.547, loss=313.5517, lr=1.60e-04]
Epoch 14 TRAIN:  70%|██████▉   | 95/136 [00:21<00:07,  5.42it/s, epoch=14, grad_norm=69.452, loss=242.5526, lr=1.60e-04]
Epoch 14 TRAIN:  72%|███████▏  | 98/136 [00:21<00:07,  5.35it/s, epoch=14, grad_norm=69.452, loss=242.5526, lr=1.60e-04]
Epoch 14 TRAIN:  72%|███████▏  | 98/136 [00:21<00:07,  5.35it/s, epoch=14, grad_norm=231.353, loss=303.9523, lr=1.60e-04]
Epoch 14 TRAIN:  72%|███████▏  | 98/136 [00:21<00:07,  5.35it/s, epoch=14, grad_norm=109.730, loss=353.8949, lr=1.60e-04]2026-04-16 19:59:02,247 DEBUG TRAIN | steps/sec 4.678| Batch 14/100 loss 353.894867 loss_att 320.049713 loss_ctc 432.866913 th_accuracy 0.300813 lr 1.6048e-04 grad_norm 109.729828 rank 0

Epoch 14 TRAIN:  72%|███████▏  | 98/136 [00:21<00:07,  5.35it/s, epoch=14, grad_norm=86.089, loss=386.4932, lr=1.61e-04] 
Epoch 14 TRAIN:  72%|███████▏  | 98/136 [00:21<00:07,  5.35it/s, epoch=14, grad_norm=73.679, loss=279.5323, lr=1.61e-04]
Epoch 14 TRAIN:  75%|███████▌  | 102/136 [00:21<00:06,  5.57it/s, epoch=14, grad_norm=73.679, loss=279.5323, lr=1.61e-04]
Epoch 14 TRAIN:  75%|███████▌  | 102/136 [00:21<00:06,  5.57it/s, epoch=14, grad_norm=64.430, loss=243.9361, lr=1.61e-04]
Epoch 14 TRAIN:  75%|███████▌  | 102/136 [00:22<00:06,  5.57it/s, epoch=14, grad_norm=70.245, loss=236.7051, lr=1.61e-04]
Epoch 14 TRAIN:  75%|███████▌  | 102/136 [00:22<00:06,  5.57it/s, epoch=14, grad_norm=94.151, loss=229.5909, lr=1.61e-04]
Epoch 14 TRAIN:  77%|███████▋  | 105/136 [00:22<00:05,  5.33it/s, epoch=14, grad_norm=94.151, loss=229.5909, lr=1.61e-04]
Epoch 14 TRAIN:  77%|███████▋  | 105/136 [00:22<00:05,  5.33it/s, epoch=14, grad_norm=95.423, loss=265.5494, lr=1.61e-04]
Epoch 14 TRAIN:  77%|███████▋  | 105/136 [00:22<00:05,  5.33it/s, epoch=14, grad_norm=134.907, loss=301.9879, lr=1.61e-04]
Epoch 14 TRAIN:  77%|███████▋  | 105/136 [00:22<00:05,  5.33it/s, epoch=14, grad_norm=243.962, loss=451.6887, lr=1.61e-04]
Epoch 14 TRAIN:  77%|███████▋  | 105/136 [00:22<00:05,  5.33it/s, epoch=14, grad_norm=87.903, loss=256.7838, lr=1.61e-04] 
Epoch 14 TRAIN:  80%|████████  | 109/136 [00:22<00:04,  5.50it/s, epoch=14, grad_norm=87.903, loss=256.7838, lr=1.61e-04]
Epoch 14 TRAIN:  80%|████████  | 109/136 [00:23<00:04,  5.50it/s, epoch=14, grad_norm=104.791, loss=292.8625, lr=1.61e-04]
Epoch 14 TRAIN:  80%|████████  | 109/136 [00:23<00:04,  5.50it/s, epoch=14, grad_norm=116.965, loss=248.0040, lr=1.61e-04]
Epoch 14 TRAIN:  80%|████████  | 109/136 [00:23<00:04,  5.50it/s, epoch=14, grad_norm=169.721, loss=392.1302, lr=1.61e-04]
Epoch 14 TRAIN:  82%|████████▏ | 112/136 [00:23<00:04,  5.56it/s, epoch=14, grad_norm=169.721, loss=392.1302, lr=1.61e-04]
Epoch 14 TRAIN:  82%|████████▏ | 112/136 [00:23<00:04,  5.56it/s, epoch=14, grad_norm=98.010, loss=302.3386, lr=1.62e-04] 
Epoch 14 TRAIN:  82%|████████▏ | 112/136 [00:23<00:04,  5.56it/s, epoch=14, grad_norm=110.593, loss=362.2043, lr=1.62e-04]
Epoch 14 TRAIN:  82%|████████▏ | 112/136 [00:24<00:04,  5.56it/s, epoch=14, grad_norm=94.415, loss=301.5616, lr=1.62e-04] 
Epoch 14 TRAIN:  85%|████████▍ | 115/136 [00:24<00:03,  5.51it/s, epoch=14, grad_norm=94.415, loss=301.5616, lr=1.62e-04]
Epoch 14 TRAIN:  85%|████████▍ | 115/136 [00:24<00:03,  5.51it/s, epoch=14, grad_norm=98.760, loss=293.5540, lr=1.62e-04]
Epoch 14 TRAIN:  85%|████████▍ | 115/136 [00:24<00:03,  5.51it/s, epoch=14, grad_norm=92.045, loss=323.4682, lr=1.62e-04]
Epoch 14 TRAIN:  85%|████████▍ | 115/136 [00:24<00:03,  5.51it/s, epoch=14, grad_norm=111.835, loss=296.0153, lr=1.62e-04]
Epoch 14 TRAIN:  87%|████████▋ | 118/136 [00:24<00:03,  5.47it/s, epoch=14, grad_norm=111.835, loss=296.0153, lr=1.62e-04]
Epoch 14 TRAIN:  87%|████████▋ | 118/136 [00:24<00:03,  5.47it/s, epoch=14, grad_norm=92.191, loss=260.0859, lr=1.62e-04] 
Epoch 14 TRAIN:  87%|████████▋ | 118/136 [00:25<00:03,  5.47it/s, epoch=14, grad_norm=91.889, loss=323.8668, lr=1.62e-04]
Epoch 14 TRAIN:  87%|████████▋ | 118/136 [00:25<00:03,  5.47it/s, epoch=14, grad_norm=100.783, loss=347.6107, lr=1.62e-04]
Epoch 14 TRAIN:  89%|████████▉ | 121/136 [00:25<00:02,  5.10it/s, epoch=14, grad_norm=100.783, loss=347.6107, lr=1.62e-04]
Epoch 14 TRAIN:  89%|████████▉ | 121/136 [00:25<00:02,  5.10it/s, epoch=14, grad_norm=141.067, loss=389.6385, lr=1.62e-04]
Epoch 14 TRAIN:  89%|████████▉ | 121/136 [00:25<00:02,  5.10it/s, epoch=14, grad_norm=180.573, loss=380.3603, lr=1.62e-04]
Epoch 14 TRAIN:  89%|████████▉ | 121/136 [00:25<00:02,  5.10it/s, epoch=14, grad_norm=101.701, loss=341.4937, lr=1.62e-04]
Epoch 14 TRAIN:  89%|████████▉ | 121/136 [00:25<00:02,  5.10it/s, epoch=14, grad_norm=83.230, loss=313.0424, lr=1.62e-04] 
Epoch 14 TRAIN:  92%|█████████▏| 125/136 [00:25<00:02,  5.45it/s, epoch=14, grad_norm=83.230, loss=313.0424, lr=1.62e-04]
Epoch 14 TRAIN:  92%|█████████▏| 125/136 [00:26<00:02,  5.45it/s, epoch=14, grad_norm=128.976, loss=367.6932, lr=1.63e-04]
Epoch 14 TRAIN:  92%|█████████▏| 125/136 [00:26<00:02,  5.45it/s, epoch=14, grad_norm=129.825, loss=378.7883, lr=1.63e-04]
Epoch 14 TRAIN:  92%|█████████▏| 125/136 [00:26<00:02,  5.45it/s, epoch=14, grad_norm=149.007, loss=270.1869, lr=1.63e-04]
Epoch 14 TRAIN:  94%|█████████▍| 128/136 [00:26<00:01,  5.17it/s, epoch=14, grad_norm=149.007, loss=270.1869, lr=1.63e-04]
Epoch 14 TRAIN:  94%|█████████▍| 128/136 [00:26<00:01,  5.17it/s, epoch=14, grad_norm=109.850, loss=361.8707, lr=1.63e-04]
Epoch 14 TRAIN:  94%|█████████▍| 128/136 [00:26<00:01,  5.17it/s, epoch=14, grad_norm=95.960, loss=291.0640, lr=1.63e-04] 
Epoch 14 TRAIN:  94%|█████████▍| 128/136 [00:27<00:01,  5.17it/s, epoch=14, grad_norm=181.516, loss=298.0492, lr=1.63e-04]
Epoch 14 TRAIN:  94%|█████████▍| 128/136 [00:27<00:01,  5.17it/s, epoch=14, grad_norm=143.224, loss=273.7553, lr=1.63e-04]
Epoch 14 TRAIN:  97%|█████████▋| 132/136 [00:27<00:00,  5.43it/s, epoch=14, grad_norm=143.224, loss=273.7553, lr=1.63e-04]
Epoch 14 TRAIN:  97%|█████████▋| 132/136 [00:27<00:00,  5.43it/s, epoch=14, grad_norm=104.676, loss=360.0009, lr=1.63e-04]
Epoch 14 TRAIN:  97%|█████████▋| 132/136 [00:27<00:00,  5.43it/s, epoch=14, grad_norm=270.448, loss=386.6660, lr=1.63e-04]
Epoch 14 TRAIN:  97%|█████████▋| 132/136 [00:27<00:00,  5.43it/s, epoch=14, grad_norm=161.211, loss=348.2061, lr=1.63e-04]
Epoch 14 TRAIN:  99%|█████████▉| 135/136 [00:27<00:00,  5.53it/s, epoch=14, grad_norm=161.211, loss=348.2061, lr=1.63e-04]
Epoch 14 TRAIN:  99%|█████████▉| 135/136 [00:27<00:00,  5.53it/s, epoch=14, grad_norm=190.743, loss=404.5904, lr=1.63e-04]
Epoch 14 TRAIN:  99%|█████████▉| 135/136 [00:28<00:00,  5.53it/s, epoch=14, grad_norm=195.571, loss=417.5578, lr=1.63e-04]
Epoch 14 TRAIN: 137it [00:28,  4.88it/s, epoch=14, grad_norm=195.571, loss=417.5578, lr=1.63e-04]                         

CV:   0%|          | 0/6 [00:00<?, ?it/s]/home/rookie/Documents/batuka/projects/wenet/wenet/wenet/utils/train_utils.py:689: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  amp_autocast(enabled=dtype is not None,

CV:  17%|█▋        | 1/6 [00:00<00:03,  1.33it/s]
                                                 
2026-04-16 19:59:09,924 INFO Epoch 14 Step 2043 CV info lr 1.6344e-04 cv_loss 407.3760195645419 rank 0 acc 0.2521771267056465
2026-04-16 19:59:09,925 INFO [Rank 0] Checkpoint: save to checkpoint exp/conformer_mongolian/epoch_14.pt
2026-04-16 19:59:10,045 INFO Epoch 15 Step 2043 TRAIN info lr 1.6344e-04 rank 0
2026-04-16 19:59:10,064 INFO using accumulate grad, new batch size is 1 times larger than before

Epoch 15 TRAIN:   0%|          | 0/137 [00:00<?, ?it/s]
Epoch 15 TRAIN:   0%|          | 0/137 [00:04<?, ?it/s, epoch=15, grad_norm=450.350, loss=404.5778, lr=1.64e-04]
Epoch 15 TRAIN:   1%|          | 1/137 [00:04<09:16,  4.09s/it, epoch=15, grad_norm=450.350, loss=404.5778, lr=1.64e-04]
Epoch 15 TRAIN:   1%|          | 1/137 [00:04<09:16,  4.09s/it, epoch=15, grad_norm=444.515, loss=389.0749, lr=1.64e-04]
Epoch 15 TRAIN:   1%|          | 1/137 [00:04<09:16,  4.09s/it, epoch=15, grad_norm=328.955, loss=324.3954, lr=1.64e-04]
Epoch 15 TRAIN:   1%|          | 1/137 [00:04<09:16,  4.09s/it, epoch=15, grad_norm=376.632, loss=385.6828, lr=1.64e-04]
Epoch 15 TRAIN:   1%|          | 1/137 [00:04<09:16,  4.09s/it, epoch=15, grad_norm=344.816, loss=408.6258, lr=1.64e-04]
Epoch 15 TRAIN:   1%|          | 1/137 [00:04<09:16,  4.09s/it, epoch=15, grad_norm=308.936, loss=426.3036, lr=1.64e-04]
Epoch 15 TRAIN:   1%|          | 1/137 [00:04<09:16,  4.09s/it, epoch=15, grad_norm=146.595, loss=312.8981, lr=1.64e-04]
Epoch 15 TRAIN:   1%|          | 1/137 [00:04<09:16,  4.09s/it, epoch=15, grad_norm=140.674, loss=353.8936, lr=1.64e-04]
Epoch 15 TRAIN:   6%|▌         | 8/137 [00:04<00:57,  2.26it/s, epoch=15, grad_norm=140.674, loss=353.8936, lr=1.64e-04]
Epoch 15 TRAIN:   6%|▌         | 8/137 [00:04<00:57,  2.26it/s, epoch=15, grad_norm=108.116, loss=304.5935, lr=1.64e-04]
Epoch 15 TRAIN:   6%|▌         | 8/137 [00:04<00:57,  2.26it/s, epoch=15, grad_norm=78.084, loss=375.4308, lr=1.64e-04] 
Epoch 15 TRAIN:   6%|▌         | 8/137 [00:05<00:57,  2.26it/s, epoch=15, grad_norm=109.686, loss=348.0404, lr=1.64e-04]
Epoch 15 TRAIN:   6%|▌         | 8/137 [00:05<00:57,  2.26it/s, epoch=15, grad_norm=115.123, loss=330.5516, lr=1.64e-04]
Epoch 15 TRAIN:   9%|▉         | 12/137 [00:05<00:40,  3.12it/s, epoch=15, grad_norm=115.123, loss=330.5516, lr=1.64e-04]
Epoch 15 TRAIN:   9%|▉         | 12/137 [00:05<00:40,  3.12it/s, epoch=15, grad_norm=112.633, loss=340.4822, lr=1.64e-04]
Epoch 15 TRAIN:   9%|▉         | 12/137 [00:05<00:40,  3.12it/s, epoch=15, grad_norm=82.781, loss=328.7794, lr=1.65e-04] 
Epoch 15 TRAIN:   9%|▉         | 12/137 [00:05<00:40,  3.12it/s, epoch=15, grad_norm=73.927, loss=355.3556, lr=1.65e-04]
Epoch 15 TRAIN:   9%|▉         | 12/137 [00:05<00:40,  3.12it/s, epoch=15, grad_norm=102.457, loss=305.9702, lr=1.65e-04]
Epoch 15 TRAIN:  12%|█▏        | 16/137 [00:05<00:31,  3.84it/s, epoch=15, grad_norm=102.457, loss=305.9702, lr=1.65e-04]
Epoch 15 TRAIN:  12%|█▏        | 16/137 [00:06<00:31,  3.84it/s, epoch=15, grad_norm=55.089, loss=315.7897, lr=1.65e-04] 
Epoch 15 TRAIN:  12%|█▏        | 16/137 [00:06<00:31,  3.84it/s, epoch=15, grad_norm=104.826, loss=275.4616, lr=1.65e-04]
Epoch 15 TRAIN:  12%|█▏        | 16/137 [00:06<00:31,  3.84it/s, epoch=15, grad_norm=56.969, loss=297.3360, lr=1.65e-04] 
Epoch 15 TRAIN:  12%|█▏        | 16/137 [00:06<00:31,  3.84it/s, epoch=15, grad_norm=111.542, loss=350.2200, lr=1.65e-04]
Epoch 15 TRAIN:  15%|█▍        | 20/137 [00:06<00:26,  4.42it/s, epoch=15, grad_norm=111.542, loss=350.2200, lr=1.65e-04]
Epoch 15 TRAIN:  15%|█▍        | 20/137 [00:06<00:26,  4.42it/s, epoch=15, grad_norm=107.597, loss=347.7293, lr=1.65e-04]
Epoch 15 TRAIN:  15%|█▍        | 20/137 [00:06<00:26,  4.42it/s, epoch=15, grad_norm=125.998, loss=367.6413, lr=1.65e-04]
Epoch 15 TRAIN:  15%|█▍        | 20/137 [00:07<00:26,  4.42it/s, epoch=15, grad_norm=149.715, loss=377.3519, lr=1.65e-04]
Epoch 15 TRAIN:  15%|█▍        | 20/137 [00:07<00:26,  4.42it/s, epoch=15, grad_norm=72.726, loss=315.0724, lr=1.65e-04] 
Epoch 15 TRAIN:  18%|█▊        | 24/137 [00:07<00:22,  4.93it/s, epoch=15, grad_norm=72.726, loss=315.0724, lr=1.65e-04]
Epoch 15 TRAIN:  18%|█▊        | 24/137 [00:07<00:22,  4.93it/s, epoch=15, grad_norm=65.842, loss=320.8920, lr=1.65e-04]
Epoch 15 TRAIN:  18%|█▊        | 24/137 [00:07<00:22,  4.93it/s, epoch=15, grad_norm=64.966, loss=354.0069, lr=1.66e-04]
Epoch 15 TRAIN:  18%|█▊        | 24/137 [00:07<00:22,  4.93it/s, epoch=15, grad_norm=156.683, loss=293.6782, lr=1.66e-04]
Epoch 15 TRAIN:  18%|█▊        | 24/137 [00:07<00:22,  4.93it/s, epoch=15, grad_norm=60.815, loss=347.5053, lr=1.66e-04] 
Epoch 15 TRAIN:  20%|██        | 28/137 [00:07<00:20,  5.34it/s, epoch=15, grad_norm=60.815, loss=347.5053, lr=1.66e-04]
Epoch 15 TRAIN:  20%|██        | 28/137 [00:07<00:20,  5.34it/s, epoch=15, grad_norm=113.670, loss=322.7771, lr=1.66e-04]
Epoch 15 TRAIN:  20%|██        | 28/137 [00:08<00:20,  5.34it/s, epoch=15, grad_norm=73.722, loss=354.4155, lr=1.66e-04] 
Epoch 15 TRAIN:  20%|██        | 28/137 [00:08<00:20,  5.34it/s, epoch=15, grad_norm=115.597, loss=312.0569, lr=1.66e-04]
Epoch 15 TRAIN:  20%|██        | 28/137 [00:08<00:20,  5.34it/s, epoch=15, grad_norm=72.167, loss=336.1763, lr=1.66e-04] 
Epoch 15 TRAIN:  23%|██▎       | 32/137 [00:08<00:18,  5.68it/s, epoch=15, grad_norm=72.167, loss=336.1763, lr=1.66e-04]
Epoch 15 TRAIN:  23%|██▎       | 32/137 [00:08<00:18,  5.68it/s, epoch=15, grad_norm=67.262, loss=307.4583, lr=1.66e-04]
Epoch 15 TRAIN:  23%|██▎       | 32/137 [00:08<00:18,  5.68it/s, epoch=15, grad_norm=61.357, loss=332.2427, lr=1.66e-04]
Epoch 15 TRAIN:  23%|██▎       | 32/137 [00:08<00:18,  5.68it/s, epoch=15, grad_norm=71.151, loss=355.3365, lr=1.66e-04]
Epoch 15 TRAIN:  23%|██▎       | 32/137 [00:09<00:18,  5.68it/s, epoch=15, grad_norm=75.054, loss=317.2698, lr=1.66e-04]
Epoch 15 TRAIN:  26%|██▋       | 36/137 [00:09<00:17,  5.73it/s, epoch=15, grad_norm=75.054, loss=317.2698, lr=1.66e-04]
Epoch 15 TRAIN:  26%|██▋       | 36/137 [00:09<00:17,  5.73it/s, epoch=15, grad_norm=124.045, loss=287.3265, lr=1.66e-04]
Epoch 15 TRAIN:  26%|██▋       | 36/137 [00:09<00:17,  5.73it/s, epoch=15, grad_norm=94.118, loss=285.0983, lr=1.66e-04] 
Epoch 15 TRAIN:  26%|██▋       | 36/137 [00:09<00:17,  5.73it/s, epoch=15, grad_norm=88.332, loss=319.5023, lr=1.67e-04]
Epoch 15 TRAIN:  26%|██▋       | 36/137 [00:09<00:17,  5.73it/s, epoch=15, grad_norm=95.477, loss=340.3323, lr=1.67e-04]
Epoch 15 TRAIN:  29%|██▉       | 40/137 [00:09<00:17,  5.64it/s, epoch=15, grad_norm=95.477, loss=340.3323, lr=1.67e-04]
Epoch 15 TRAIN:  29%|██▉       | 40/137 [00:10<00:17,  5.64it/s, epoch=15, grad_norm=121.846, loss=366.8957, lr=1.67e-04]
Epoch 15 TRAIN:  29%|██▉       | 40/137 [00:10<00:17,  5.64it/s, epoch=15, grad_norm=54.899, loss=316.6711, lr=1.67e-04] 
Epoch 15 TRAIN:  29%|██▉       | 40/137 [00:10<00:17,  5.64it/s, epoch=15, grad_norm=68.679, loss=325.3422, lr=1.67e-04]
Epoch 15 TRAIN:  29%|██▉       | 40/137 [00:10<00:17,  5.64it/s, epoch=15, grad_norm=90.179, loss=311.7294, lr=1.67e-04]
Epoch 15 TRAIN:  32%|███▏      | 44/137 [00:10<00:16,  5.56it/s, epoch=15, grad_norm=90.179, loss=311.7294, lr=1.67e-04]
Epoch 15 TRAIN:  32%|███▏      | 44/137 [00:10<00:16,  5.56it/s, epoch=15, grad_norm=93.507, loss=336.7048, lr=1.67e-04]
Epoch 15 TRAIN:  32%|███▏      | 44/137 [00:10<00:16,  5.56it/s, epoch=15, grad_norm=67.059, loss=297.1940, lr=1.67e-04]
Epoch 15 TRAIN:  32%|███▏      | 44/137 [00:11<00:16,  5.56it/s, epoch=15, grad_norm=98.144, loss=308.4595, lr=1.67e-04]
Epoch 15 TRAIN:  34%|███▍      | 47/137 [00:11<00:16,  5.60it/s, epoch=15, grad_norm=98.144, loss=308.4595, lr=1.67e-04]
Epoch 15 TRAIN:  34%|███▍      | 47/137 [00:11<00:16,  5.60it/s, epoch=15, grad_norm=81.507, loss=287.0812, lr=1.67e-04]
Epoch 15 TRAIN:  34%|███▍      | 47/137 [00:11<00:16,  5.60it/s, epoch=15, grad_norm=125.231, loss=340.4445, lr=1.67e-04]
Epoch 15 TRAIN:  34%|███▍      | 47/137 [00:11<00:16,  5.60it/s, epoch=15, grad_norm=71.746, loss=328.7828, lr=1.67e-04] 
Epoch 15 TRAIN:  36%|███▋      | 50/137 [00:11<00:15,  5.61it/s, epoch=15, grad_norm=71.746, loss=328.7828, lr=1.67e-04]
Epoch 15 TRAIN:  36%|███▋      | 50/137 [00:11<00:15,  5.61it/s, epoch=15, grad_norm=62.654, loss=309.1345, lr=1.68e-04]
Epoch 15 TRAIN:  36%|███▋      | 50/137 [00:12<00:15,  5.61it/s, epoch=15, grad_norm=122.822, loss=336.3881, lr=1.68e-04]
Epoch 15 TRAIN:  36%|███▋      | 50/137 [00:12<00:15,  5.61it/s, epoch=15, grad_norm=75.768, loss=341.6332, lr=1.68e-04] 
Epoch 15 TRAIN:  39%|███▊      | 53/137 [00:12<00:15,  5.47it/s, epoch=15, grad_norm=75.768, loss=341.6332, lr=1.68e-04]
Epoch 15 TRAIN:  39%|███▊      | 53/137 [00:12<00:15,  5.47it/s, epoch=15, grad_norm=154.269, loss=387.9663, lr=1.68e-04]
Epoch 15 TRAIN:  39%|███▊      | 53/137 [00:12<00:15,  5.47it/s, epoch=15, grad_norm=69.788, loss=329.5079, lr=1.68e-04] 
Epoch 15 TRAIN:  39%|███▊      | 53/137 [00:12<00:15,  5.47it/s, epoch=15, grad_norm=56.756, loss=335.9071, lr=1.68e-04]
Epoch 15 TRAIN:  41%|████      | 56/137 [00:12<00:15,  5.32it/s, epoch=15, grad_norm=56.756, loss=335.9071, lr=1.68e-04]
Epoch 15 TRAIN:  41%|████      | 56/137 [00:13<00:15,  5.32it/s, epoch=15, grad_norm=80.331, loss=350.3137, lr=1.68e-04]
Epoch 15 TRAIN:  41%|████      | 56/137 [00:13<00:15,  5.32it/s, epoch=15, grad_norm=89.943, loss=362.9957, lr=1.68e-04]
Epoch 15 TRAIN:  41%|████      | 56/137 [00:13<00:15,  5.32it/s, epoch=15, grad_norm=91.929, loss=328.0258, lr=1.68e-04]
Epoch 15 TRAIN:  43%|████▎     | 59/137 [00:13<00:14,  5.37it/s, epoch=15, grad_norm=91.929, loss=328.0258, lr=1.68e-04]
Epoch 15 TRAIN:  43%|████▎     | 59/137 [00:13<00:14,  5.37it/s, epoch=15, grad_norm=135.955, loss=291.0793, lr=1.68e-04]
Epoch 15 TRAIN:  43%|████▎     | 59/137 [00:13<00:14,  5.37it/s, epoch=15, grad_norm=119.558, loss=325.4747, lr=1.68e-04]
Epoch 15 TRAIN:  43%|████▎     | 59/137 [00:13<00:14,  5.37it/s, epoch=15, grad_norm=149.346, loss=337.1501, lr=1.68e-04]
Epoch 15 TRAIN:  45%|████▌     | 62/137 [00:13<00:13,  5.48it/s, epoch=15, grad_norm=149.346, loss=337.1501, lr=1.68e-04]
Epoch 15 TRAIN:  45%|████▌     | 62/137 [00:14<00:13,  5.48it/s, epoch=15, grad_norm=93.235, loss=320.1096, lr=1.68e-04] 
Epoch 15 TRAIN:  45%|████▌     | 62/137 [00:14<00:13,  5.48it/s, epoch=15, grad_norm=171.338, loss=360.5108, lr=1.69e-04]
Epoch 15 TRAIN:  45%|████▌     | 62/137 [00:14<00:13,  5.48it/s, epoch=15, grad_norm=115.053, loss=307.9681, lr=1.69e-04]
Epoch 15 TRAIN:  47%|████▋     | 65/137 [00:14<00:13,  5.39it/s, epoch=15, grad_norm=115.053, loss=307.9681, lr=1.69e-04]
Epoch 15 TRAIN:  47%|████▋     | 65/137 [00:14<00:13,  5.39it/s, epoch=15, grad_norm=80.729, loss=295.5254, lr=1.69e-04] 
Epoch 15 TRAIN:  47%|████▋     | 65/137 [00:14<00:13,  5.39it/s, epoch=15, grad_norm=67.083, loss=328.1595, lr=1.69e-04]
Epoch 15 TRAIN:  47%|████▋     | 65/137 [00:15<00:13,  5.39it/s, epoch=15, grad_norm=81.566, loss=310.0793, lr=1.69e-04]
Epoch 15 TRAIN:  50%|████▉     | 68/137 [00:15<00:12,  5.37it/s, epoch=15, grad_norm=81.566, loss=310.0793, lr=1.69e-04]
Epoch 15 TRAIN:  50%|████▉     | 68/137 [00:15<00:12,  5.37it/s, epoch=15, grad_norm=186.147, loss=306.2811, lr=1.69e-04]
Epoch 15 TRAIN:  50%|████▉     | 68/137 [00:15<00:12,  5.37it/s, epoch=15, grad_norm=100.098, loss=280.9230, lr=1.69e-04]
Epoch 15 TRAIN:  50%|████▉     | 68/137 [00:15<00:12,  5.37it/s, epoch=15, grad_norm=75.664, loss=311.7455, lr=1.69e-04] 
Epoch 15 TRAIN:  52%|█████▏    | 71/137 [00:15<00:12,  5.46it/s, epoch=15, grad_norm=75.664, loss=311.7455, lr=1.69e-04]
Epoch 15 TRAIN:  52%|█████▏    | 71/137 [00:15<00:12,  5.46it/s, epoch=15, grad_norm=252.380, loss=417.6182, lr=1.69e-04]
Epoch 15 TRAIN:  52%|█████▏    | 71/137 [00:15<00:12,  5.46it/s, epoch=15, grad_norm=140.307, loss=285.4470, lr=1.69e-04]
Epoch 15 TRAIN:  52%|█████▏    | 71/137 [00:16<00:12,  5.46it/s, epoch=15, grad_norm=225.486, loss=384.5907, lr=1.69e-04]
Epoch 15 TRAIN:  54%|█████▍    | 74/137 [00:16<00:11,  5.50it/s, epoch=15, grad_norm=225.486, loss=384.5907, lr=1.69e-04]
Epoch 15 TRAIN:  54%|█████▍    | 74/137 [00:16<00:11,  5.50it/s, epoch=15, grad_norm=106.830, loss=274.4398, lr=1.69e-04]
Epoch 15 TRAIN:  54%|█████▍    | 74/137 [00:16<00:11,  5.50it/s, epoch=15, grad_norm=145.643, loss=319.5395, lr=1.70e-04]
Epoch 15 TRAIN:  54%|█████▍    | 74/137 [00:16<00:11,  5.50it/s, epoch=15, grad_norm=144.505, loss=333.3353, lr=1.70e-04]
Epoch 15 TRAIN:  56%|█████▌    | 77/137 [00:16<00:11,  5.22it/s, epoch=15, grad_norm=144.505, loss=333.3353, lr=1.70e-04]
Epoch 15 TRAIN:  56%|█████▌    | 77/137 [00:16<00:11,  5.22it/s, epoch=15, grad_norm=166.997, loss=336.1441, lr=1.70e-04]
Epoch 15 TRAIN:  56%|█████▌    | 77/137 [00:17<00:11,  5.22it/s, epoch=15, grad_norm=75.450, loss=294.0363, lr=1.70e-04] 
Epoch 15 TRAIN:  56%|█████▌    | 77/137 [00:17<00:11,  5.22it/s, epoch=15, grad_norm=124.865, loss=310.5225, lr=1.70e-04]
Epoch 15 TRAIN:  58%|█████▊    | 80/137 [00:17<00:10,  5.33it/s, epoch=15, grad_norm=124.865, loss=310.5225, lr=1.70e-04]
Epoch 15 TRAIN:  58%|█████▊    | 80/137 [00:17<00:10,  5.33it/s, epoch=15, grad_norm=116.366, loss=286.0260, lr=1.70e-04]
Epoch 15 TRAIN:  58%|█████▊    | 80/137 [00:17<00:10,  5.33it/s, epoch=15, grad_norm=142.851, loss=365.1396, lr=1.70e-04]
Epoch 15 TRAIN:  58%|█████▊    | 80/137 [00:17<00:10,  5.33it/s, epoch=15, grad_norm=103.113, loss=321.7971, lr=1.70e-04]
Epoch 15 TRAIN:  61%|██████    | 83/137 [00:17<00:10,  5.02it/s, epoch=15, grad_norm=103.113, loss=321.7971, lr=1.70e-04]
Epoch 15 TRAIN:  61%|██████    | 83/137 [00:18<00:10,  5.02it/s, epoch=15, grad_norm=131.149, loss=293.1284, lr=1.70e-04]
Epoch 15 TRAIN:  61%|██████    | 83/137 [00:18<00:10,  5.02it/s, epoch=15, grad_norm=225.734, loss=299.7657, lr=1.70e-04]
Epoch 15 TRAIN:  61%|██████    | 83/137 [00:18<00:10,  5.02it/s, epoch=15, grad_norm=119.327, loss=330.5327, lr=1.70e-04]
Epoch 15 TRAIN:  63%|██████▎   | 86/137 [00:18<00:10,  4.92it/s, epoch=15, grad_norm=119.327, loss=330.5327, lr=1.70e-04]
Epoch 15 TRAIN:  63%|██████▎   | 86/137 [00:18<00:10,  4.92it/s, epoch=15, grad_norm=186.433, loss=243.8573, lr=1.70e-04]
Epoch 15 TRAIN:  63%|██████▎   | 86/137 [00:18<00:10,  4.92it/s, epoch=15, grad_norm=163.976, loss=285.6659, lr=1.70e-04]
Epoch 15 TRAIN:  63%|██████▎   | 86/137 [00:19<00:10,  4.92it/s, epoch=15, grad_norm=107.778, loss=311.1022, lr=1.71e-04]
Epoch 15 TRAIN:  65%|██████▍   | 89/137 [00:19<00:09,  5.12it/s, epoch=15, grad_norm=107.778, loss=311.1022, lr=1.71e-04]
Epoch 15 TRAIN:  65%|██████▍   | 89/137 [00:19<00:09,  5.12it/s, epoch=15, grad_norm=136.859, loss=288.4355, lr=1.71e-04]
Epoch 15 TRAIN:  65%|██████▍   | 89/137 [00:19<00:09,  5.12it/s, epoch=15, grad_norm=124.299, loss=258.4346, lr=1.71e-04]
Epoch 15 TRAIN:  65%|██████▍   | 89/137 [00:19<00:09,  5.12it/s, epoch=15, grad_norm=136.117, loss=281.0799, lr=1.71e-04]
Epoch 15 TRAIN:  67%|██████▋   | 92/137 [00:19<00:08,  5.31it/s, epoch=15, grad_norm=136.117, loss=281.0799, lr=1.71e-04]
Epoch 15 TRAIN:  67%|██████▋   | 92/137 [00:19<00:08,  5.31it/s, epoch=15, grad_norm=84.715, loss=255.8479, lr=1.71e-04] 
Epoch 15 TRAIN:  67%|██████▋   | 92/137 [00:19<00:08,  5.31it/s, epoch=15, grad_norm=190.704, loss=308.7318, lr=1.71e-04]
Epoch 15 TRAIN:  67%|██████▋   | 92/137 [00:20<00:08,  5.31it/s, epoch=15, grad_norm=181.193, loss=293.8955, lr=1.71e-04]
Epoch 15 TRAIN:  69%|██████▉   | 95/137 [00:20<00:08,  5.15it/s, epoch=15, grad_norm=181.193, loss=293.8955, lr=1.71e-04]
Epoch 15 TRAIN:  69%|██████▉   | 95/137 [00:20<00:08,  5.15it/s, epoch=15, grad_norm=101.603, loss=241.0875, lr=1.71e-04]
Epoch 15 TRAIN:  69%|██████▉   | 95/137 [00:20<00:08,  5.15it/s, epoch=15, grad_norm=248.666, loss=359.0535, lr=1.71e-04]
Epoch 15 TRAIN:  69%|██████▉   | 95/137 [00:20<00:08,  5.15it/s, epoch=15, grad_norm=140.540, loss=350.8022, lr=1.71e-04]
Epoch 15 TRAIN:  72%|███████▏  | 98/137 [00:20<00:07,  5.24it/s, epoch=15, grad_norm=140.540, loss=350.8022, lr=1.71e-04]
Epoch 15 TRAIN:  72%|███████▏  | 98/137 [00:20<00:07,  5.24it/s, epoch=15, grad_norm=149.195, loss=319.1199, lr=1.71e-04]
Epoch 15 TRAIN:  72%|███████▏  | 98/137 [00:21<00:07,  5.24it/s, epoch=15, grad_norm=92.520, loss=344.3963, lr=1.71e-04] 2026-04-16 19:59:31,341 DEBUG TRAIN | steps/sec 4.709| Batch 15/100 loss 344.396301 loss_att 315.888733 loss_ctc 410.913910 th_accuracy 0.259657 lr 1.7144e-04 grad_norm 92.519966 rank 0

Epoch 15 TRAIN:  72%|███████▏  | 98/137 [00:21<00:07,  5.24it/s, epoch=15, grad_norm=241.407, loss=293.5526, lr=1.72e-04]
Epoch 15 TRAIN:  74%|███████▎  | 101/137 [00:21<00:07,  5.03it/s, epoch=15, grad_norm=241.407, loss=293.5526, lr=1.72e-04]
Epoch 15 TRAIN:  74%|███████▎  | 101/137 [00:21<00:07,  5.03it/s, epoch=15, grad_norm=135.828, loss=316.4577, lr=1.72e-04]
Epoch 15 TRAIN:  74%|███████▎  | 101/137 [00:21<00:07,  5.03it/s, epoch=15, grad_norm=322.258, loss=187.3481, lr=1.72e-04]
Epoch 15 TRAIN:  74%|███████▎  | 101/137 [00:22<00:07,  5.03it/s, epoch=15, grad_norm=200.324, loss=275.5517, lr=1.72e-04]
Epoch 15 TRAIN:  76%|███████▌  | 104/137 [00:22<00:06,  5.14it/s, epoch=15, grad_norm=200.324, loss=275.5517, lr=1.72e-04]
Epoch 15 TRAIN:  76%|███████▌  | 104/137 [00:22<00:06,  5.14it/s, epoch=15, grad_norm=112.377, loss=334.8803, lr=1.72e-04]
Epoch 15 TRAIN:  76%|███████▌  | 104/137 [00:22<00:06,  5.14it/s, epoch=15, grad_norm=165.016, loss=288.2682, lr=1.72e-04]
Epoch 15 TRAIN:  76%|███████▌  | 104/137 [00:22<00:06,  5.14it/s, epoch=15, grad_norm=218.053, loss=203.8377, lr=1.72e-04]
Epoch 15 TRAIN:  78%|███████▊  | 107/137 [00:22<00:05,  5.19it/s, epoch=15, grad_norm=218.053, loss=203.8377, lr=1.72e-04]
Epoch 15 TRAIN:  78%|███████▊  | 107/137 [00:22<00:05,  5.19it/s, epoch=15, grad_norm=125.510, loss=324.0018, lr=1.72e-04]
Epoch 15 TRAIN:  78%|███████▊  | 107/137 [00:23<00:05,  5.19it/s, epoch=15, grad_norm=156.032, loss=354.8535, lr=1.72e-04]
Epoch 15 TRAIN:  78%|███████▊  | 107/137 [00:23<00:05,  5.19it/s, epoch=15, grad_norm=313.644, loss=393.0530, lr=1.72e-04]
Epoch 15 TRAIN:  80%|████████  | 110/137 [00:23<00:05,  4.86it/s, epoch=15, grad_norm=313.644, loss=393.0530, lr=1.72e-04]
Epoch 15 TRAIN:  80%|████████  | 110/137 [00:23<00:05,  4.86it/s, epoch=15, grad_norm=312.666, loss=408.3384, lr=1.72e-04]
Epoch 15 TRAIN:  80%|████████  | 110/137 [00:23<00:05,  4.86it/s, epoch=15, grad_norm=110.694, loss=305.8574, lr=1.72e-04]
Epoch 15 TRAIN:  80%|████████  | 110/137 [00:23<00:05,  4.86it/s, epoch=15, grad_norm=110.326, loss=292.4399, lr=1.72e-04]
Epoch 15 TRAIN:  82%|████████▏ | 113/137 [00:23<00:05,  4.73it/s, epoch=15, grad_norm=110.326, loss=292.4399, lr=1.72e-04]
Epoch 15 TRAIN:  82%|████████▏ | 113/137 [00:24<00:05,  4.73it/s, epoch=15, grad_norm=70.639, loss=280.9659, lr=1.73e-04] 
Epoch 15 TRAIN:  82%|████████▏ | 113/137 [00:24<00:05,  4.73it/s, epoch=15, grad_norm=97.142, loss=251.6843, lr=1.73e-04]
Epoch 15 TRAIN:  82%|████████▏ | 113/137 [00:24<00:05,  4.73it/s, epoch=15, grad_norm=126.917, loss=316.8578, lr=1.73e-04]
Epoch 15 TRAIN:  85%|████████▍ | 116/137 [00:24<00:04,  4.79it/s, epoch=15, grad_norm=126.917, loss=316.8578, lr=1.73e-04]
Epoch 15 TRAIN:  85%|████████▍ | 116/137 [00:24<00:04,  4.79it/s, epoch=15, grad_norm=142.636, loss=411.1116, lr=1.73e-04]
Epoch 15 TRAIN:  85%|████████▍ | 116/137 [00:24<00:04,  4.79it/s, epoch=15, grad_norm=136.240, loss=315.2684, lr=1.73e-04]
Epoch 15 TRAIN:  85%|████████▍ | 116/137 [00:25<00:04,  4.79it/s, epoch=15, grad_norm=105.638, loss=337.9638, lr=1.73e-04]
Epoch 15 TRAIN:  87%|████████▋ | 119/137 [00:25<00:03,  4.98it/s, epoch=15, grad_norm=105.638, loss=337.9638, lr=1.73e-04]
Epoch 15 TRAIN:  87%|████████▋ | 119/137 [00:25<00:03,  4.98it/s, epoch=15, grad_norm=168.683, loss=340.1455, lr=1.73e-04]
Epoch 15 TRAIN:  87%|████████▋ | 119/137 [00:25<00:03,  4.98it/s, epoch=15, grad_norm=209.901, loss=335.7048, lr=1.73e-04]
Epoch 15 TRAIN:  87%|████████▋ | 119/137 [00:25<00:03,  4.98it/s, epoch=15, grad_norm=166.793, loss=401.6699, lr=1.73e-04]
Epoch 15 TRAIN:  89%|████████▉ | 122/137 [00:25<00:03,  4.95it/s, epoch=15, grad_norm=166.793, loss=401.6699, lr=1.73e-04]
Epoch 15 TRAIN:  89%|████████▉ | 122/137 [00:25<00:03,  4.95it/s, epoch=15, grad_norm=175.773, loss=349.8253, lr=1.73e-04]
Epoch 15 TRAIN:  89%|████████▉ | 122/137 [00:26<00:03,  4.95it/s, epoch=15, grad_norm=139.001, loss=368.4404, lr=1.73e-04]
Epoch 15 TRAIN:  89%|████████▉ | 122/137 [00:26<00:03,  4.95it/s, epoch=15, grad_norm=155.780, loss=353.5861, lr=1.73e-04]
Epoch 15 TRAIN:  91%|█████████ | 125/137 [00:26<00:02,  4.99it/s, epoch=15, grad_norm=155.780, loss=353.5861, lr=1.73e-04]
Epoch 15 TRAIN:  91%|█████████ | 125/137 [00:26<00:02,  4.99it/s, epoch=15, grad_norm=105.107, loss=377.1773, lr=1.74e-04]
Epoch 15 TRAIN:  91%|█████████ | 125/137 [00:26<00:02,  4.99it/s, epoch=15, grad_norm=175.148, loss=326.2813, lr=1.74e-04]
Epoch 15 TRAIN:  91%|█████████ | 125/137 [00:26<00:02,  4.99it/s, epoch=15, grad_norm=210.007, loss=314.0580, lr=1.74e-04]
Epoch 15 TRAIN:  91%|█████████ | 125/137 [00:26<00:02,  4.99it/s, epoch=15, grad_norm=113.625, loss=301.2536, lr=1.74e-04]
Epoch 15 TRAIN:  94%|█████████▍| 129/137 [00:26<00:01,  5.42it/s, epoch=15, grad_norm=113.625, loss=301.2536, lr=1.74e-04]
Epoch 15 TRAIN:  94%|█████████▍| 129/137 [00:27<00:01,  5.42it/s, epoch=15, grad_norm=97.563, loss=284.1854, lr=1.74e-04] 
Epoch 15 TRAIN:  94%|█████████▍| 129/137 [00:27<00:01,  5.42it/s, epoch=15, grad_norm=199.565, loss=358.5481, lr=1.74e-04]
Epoch 15 TRAIN:  94%|█████████▍| 129/137 [00:27<00:01,  5.42it/s, epoch=15, grad_norm=230.643, loss=401.6631, lr=1.74e-04]
Epoch 15 TRAIN:  96%|█████████▋| 132/137 [00:27<00:00,  5.47it/s, epoch=15, grad_norm=230.643, loss=401.6631, lr=1.74e-04]
Epoch 15 TRAIN:  96%|█████████▋| 132/137 [00:27<00:00,  5.47it/s, epoch=15, grad_norm=381.642, loss=455.0357, lr=1.74e-04]
Epoch 15 TRAIN:  96%|█████████▋| 132/137 [00:27<00:00,  5.47it/s, epoch=15, grad_norm=266.383, loss=402.8218, lr=1.74e-04]
Epoch 15 TRAIN:  96%|█████████▋| 132/137 [00:27<00:00,  5.47it/s, epoch=15, grad_norm=179.249, loss=316.9539, lr=1.74e-04]
Epoch 15 TRAIN:  99%|█████████▊| 135/137 [00:27<00:00,  4.84it/s, epoch=15, grad_norm=179.249, loss=316.9539, lr=1.74e-04]

CV:   0%|          | 0/6 [00:00<?, ?it/s]/home/rookie/Documents/batuka/projects/wenet/wenet/wenet/utils/train_utils.py:689: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  amp_autocast(enabled=dtype is not None,

CV:  17%|█▋        | 1/6 [00:00<00:03,  1.33it/s]
                                                 
2026-04-16 19:59:38,781 INFO Epoch 15 Step 2178 CV info lr 1.7424e-04 cv_loss 397.3881752707741 rank 0 acc 0.24942893286546072
2026-04-16 19:59:38,783 INFO [Rank 0] Checkpoint: save to checkpoint exp/conformer_mongolian/epoch_15.pt
2026-04-16 19:59:38,899 INFO Epoch 16 Step 2178 TRAIN info lr 1.7424e-04 rank 0
2026-04-16 19:59:38,912 INFO using accumulate grad, new batch size is 1 times larger than before

Epoch 16 TRAIN:   0%|          | 0/135 [00:00<?, ?it/s]
Epoch 16 TRAIN:   0%|          | 0/135 [00:04<?, ?it/s, epoch=16, grad_norm=396.956, loss=379.9088, lr=1.74e-04]
Epoch 16 TRAIN:   1%|          | 1/135 [00:04<09:03,  4.06s/it, epoch=16, grad_norm=396.956, loss=379.9088, lr=1.74e-04]
Epoch 16 TRAIN:   1%|          | 1/135 [00:04<09:03,  4.06s/it, epoch=16, grad_norm=449.347, loss=402.8849, lr=1.74e-04]
Epoch 16 TRAIN:   1%|          | 1/135 [00:04<09:03,  4.06s/it, epoch=16, grad_norm=451.012, loss=381.6380, lr=1.74e-04]
Epoch 16 TRAIN:   1%|          | 1/135 [00:04<09:03,  4.06s/it, epoch=16, grad_norm=456.314, loss=436.6144, lr=1.75e-04]
Epoch 16 TRAIN:   1%|          | 1/135 [00:04<09:03,  4.06s/it, epoch=16, grad_norm=449.229, loss=430.7731, lr=1.75e-04]
Epoch 16 TRAIN:   1%|          | 1/135 [00:04<09:03,  4.06s/it, epoch=16, grad_norm=451.100, loss=443.2166, lr=1.75e-04]
Epoch 16 TRAIN:   1%|          | 1/135 [00:04<09:03,  4.06s/it, epoch=16, grad_norm=191.188, loss=317.7628, lr=1.75e-04]
Epoch 16 TRAIN:   1%|          | 1/135 [00:04<09:03,  4.06s/it, epoch=16, grad_norm=125.723, loss=289.6573, lr=1.75e-04]
Epoch 16 TRAIN:   6%|▌         | 8/135 [00:04<00:56,  2.26it/s, epoch=16, grad_norm=125.723, loss=289.6573, lr=1.75e-04]
Epoch 16 TRAIN:   6%|▌         | 8/135 [00:04<00:56,  2.26it/s, epoch=16, grad_norm=114.322, loss=324.5421, lr=1.75e-04]
Epoch 16 TRAIN:   6%|▌         | 8/135 [00:04<00:56,  2.26it/s, epoch=16, grad_norm=84.936, loss=318.2264, lr=1.75e-04] 
Epoch 16 TRAIN:   6%|▌         | 8/135 [00:05<00:56,  2.26it/s, epoch=16, grad_norm=85.262, loss=338.8299, lr=1.75e-04]
Epoch 16 TRAIN:   6%|▌         | 8/135 [00:05<00:56,  2.26it/s, epoch=16, grad_norm=96.142, loss=345.7087, lr=1.75e-04]
Epoch 16 TRAIN:   9%|▉         | 12/135 [00:05<00:39,  3.12it/s, epoch=16, grad_norm=96.142, loss=345.7087, lr=1.75e-04]
Epoch 16 TRAIN:   9%|▉         | 12/135 [00:05<00:39,  3.12it/s, epoch=16, grad_norm=67.795, loss=306.9749, lr=1.75e-04]
Epoch 16 TRAIN:   9%|▉         | 12/135 [00:05<00:39,  3.12it/s, epoch=16, grad_norm=92.928, loss=292.3044, lr=1.75e-04]
Epoch 16 TRAIN:   9%|▉         | 12/135 [00:05<00:39,  3.12it/s, epoch=16, grad_norm=81.232, loss=303.9669, lr=1.75e-04]
Epoch 16 TRAIN:   9%|▉         | 12/135 [00:05<00:39,  3.12it/s, epoch=16, grad_norm=132.959, loss=372.4050, lr=1.76e-04]
Epoch 16 TRAIN:  12%|█▏        | 16/135 [00:05<00:30,  3.88it/s, epoch=16, grad_norm=132.959, loss=372.4050, lr=1.76e-04]
Epoch 16 TRAIN:  12%|█▏        | 16/135 [00:06<00:30,  3.88it/s, epoch=16, grad_norm=63.298, loss=333.8827, lr=1.76e-04] 
Epoch 16 TRAIN:  12%|█▏        | 16/135 [00:06<00:30,  3.88it/s, epoch=16, grad_norm=99.450, loss=318.3823, lr=1.76e-04]
Epoch 16 TRAIN:  12%|█▏        | 16/135 [00:06<00:30,  3.88it/s, epoch=16, grad_norm=68.169, loss=337.1973, lr=1.76e-04]
Epoch 16 TRAIN:  12%|█▏        | 16/135 [00:06<00:30,  3.88it/s, epoch=16, grad_norm=80.794, loss=383.8378, lr=1.76e-04]
Epoch 16 TRAIN:  15%|█▍        | 20/135 [00:06<00:25,  4.45it/s, epoch=16, grad_norm=80.794, loss=383.8378, lr=1.76e-04]
Epoch 16 TRAIN:  15%|█▍        | 20/135 [00:06<00:25,  4.45it/s, epoch=16, grad_norm=108.897, loss=384.3978, lr=1.76e-04]
Epoch 16 TRAIN:  15%|█▍        | 20/135 [00:06<00:25,  4.45it/s, epoch=16, grad_norm=80.893, loss=314.3966, lr=1.76e-04] 
Epoch 16 TRAIN:  15%|█▍        | 20/135 [00:06<00:25,  4.45it/s, epoch=16, grad_norm=82.949, loss=328.5910, lr=1.76e-04]
Epoch 16 TRAIN:  15%|█▍        | 20/135 [00:07<00:25,  4.45it/s, epoch=16, grad_norm=77.148, loss=377.6541, lr=1.76e-04]
Epoch 16 TRAIN:  18%|█▊        | 24/135 [00:07<00:22,  4.92it/s, epoch=16, grad_norm=77.148, loss=377.6541, lr=1.76e-04]
Epoch 16 TRAIN:  18%|█▊        | 24/135 [00:07<00:22,  4.92it/s, epoch=16, grad_norm=96.530, loss=310.1504, lr=1.76e-04]
Epoch 16 TRAIN:  18%|█▊        | 24/135 [00:07<00:22,  4.92it/s, epoch=16, grad_norm=66.107, loss=346.0617, lr=1.76e-04]
Epoch 16 TRAIN:  18%|█▊        | 24/135 [00:07<00:22,  4.92it/s, epoch=16, grad_norm=65.956, loss=363.1199, lr=1.76e-04]
Epoch 16 TRAIN:  18%|█▊        | 24/135 [00:07<00:22,  4.92it/s, epoch=16, grad_norm=74.304, loss=314.6151, lr=1.76e-04]
Epoch 16 TRAIN:  21%|██        | 28/135 [00:07<00:20,  5.35it/s, epoch=16, grad_norm=74.304, loss=314.6151, lr=1.76e-04]
Epoch 16 TRAIN:  21%|██        | 28/135 [00:07<00:20,  5.35it/s, epoch=16, grad_norm=65.441, loss=335.4340, lr=1.77e-04]
Epoch 16 TRAIN:  21%|██        | 28/135 [00:08<00:20,  5.35it/s, epoch=16, grad_norm=104.605, loss=275.7192, lr=1.77e-04]
Epoch 16 TRAIN:  21%|██        | 28/135 [00:08<00:20,  5.35it/s, epoch=16, grad_norm=112.528, loss=266.5274, lr=1.77e-04]
Epoch 16 TRAIN:  21%|██        | 28/135 [00:08<00:20,  5.35it/s, epoch=16, grad_norm=105.057, loss=281.8422, lr=1.77e-04]
Epoch 16 TRAIN:  24%|██▎       | 32/135 [00:08<00:18,  5.46it/s, epoch=16, grad_norm=105.057, loss=281.8422, lr=1.77e-04]
Epoch 16 TRAIN:  24%|██▎       | 32/135 [00:08<00:18,  5.46it/s, epoch=16, grad_norm=87.104, loss=325.9165, lr=1.77e-04] 
Epoch 16 TRAIN:  24%|██▎       | 32/135 [00:08<00:18,  5.46it/s, epoch=16, grad_norm=170.316, loss=366.5392, lr=1.77e-04]
Epoch 16 TRAIN:  24%|██▎       | 32/135 [00:09<00:18,  5.46it/s, epoch=16, grad_norm=146.413, loss=356.4380, lr=1.77e-04]
Epoch 16 TRAIN:  24%|██▎       | 32/135 [00:09<00:18,  5.46it/s, epoch=16, grad_norm=55.601, loss=298.8101, lr=1.77e-04] 
Epoch 16 TRAIN:  27%|██▋       | 36/135 [00:09<00:17,  5.51it/s, epoch=16, grad_norm=55.601, loss=298.8101, lr=1.77e-04]
Epoch 16 TRAIN:  27%|██▋       | 36/135 [00:09<00:17,  5.51it/s, epoch=16, grad_norm=96.414, loss=327.0149, lr=1.77e-04]
Epoch 16 TRAIN:  27%|██▋       | 36/135 [00:09<00:17,  5.51it/s, epoch=16, grad_norm=84.349, loss=332.4575, lr=1.77e-04]
Epoch 16 TRAIN:  27%|██▋       | 36/135 [00:09<00:17,  5.51it/s, epoch=16, grad_norm=71.508, loss=334.9827, lr=1.77e-04]
Epoch 16 TRAIN:  27%|██▋       | 36/135 [00:09<00:17,  5.51it/s, epoch=16, grad_norm=77.706, loss=328.2893, lr=1.77e-04]
Epoch 16 TRAIN:  30%|██▉       | 40/135 [00:09<00:17,  5.33it/s, epoch=16, grad_norm=77.706, loss=328.2893, lr=1.77e-04]
Epoch 16 TRAIN:  30%|██▉       | 40/135 [00:10<00:17,  5.33it/s, epoch=16, grad_norm=72.094, loss=315.2761, lr=1.78e-04]
Epoch 16 TRAIN:  30%|██▉       | 40/135 [00:10<00:17,  5.33it/s, epoch=16, grad_norm=96.108, loss=318.8680, lr=1.78e-04]
Epoch 16 TRAIN:  30%|██▉       | 40/135 [00:10<00:17,  5.33it/s, epoch=16, grad_norm=74.039, loss=342.4443, lr=1.78e-04]
Epoch 16 TRAIN:  32%|███▏      | 43/135 [00:10<00:16,  5.44it/s, epoch=16, grad_norm=74.039, loss=342.4443, lr=1.78e-04]
Epoch 16 TRAIN:  32%|███▏      | 43/135 [00:10<00:16,  5.44it/s, epoch=16, grad_norm=87.991, loss=378.6940, lr=1.78e-04]
Epoch 16 TRAIN:  32%|███▏      | 43/135 [00:10<00:16,  5.44it/s, epoch=16, grad_norm=86.918, loss=282.5846, lr=1.78e-04]
Epoch 16 TRAIN:  32%|███▏      | 43/135 [00:11<00:16,  5.44it/s, epoch=16, grad_norm=64.498, loss=289.5154, lr=1.78e-04]
Epoch 16 TRAIN:  34%|███▍      | 46/135 [00:11<00:16,  5.28it/s, epoch=16, grad_norm=64.498, loss=289.5154, lr=1.78e-04]
Epoch 16 TRAIN:  34%|███▍      | 46/135 [00:11<00:16,  5.28it/s, epoch=16, grad_norm=56.268, loss=309.0847, lr=1.78e-04]
Epoch 16 TRAIN:  34%|███▍      | 46/135 [00:11<00:16,  5.28it/s, epoch=16, grad_norm=141.878, loss=312.3578, lr=1.78e-04]
Epoch 16 TRAIN:  34%|███▍      | 46/135 [00:11<00:16,  5.28it/s, epoch=16, grad_norm=97.767, loss=345.0650, lr=1.78e-04] 
Epoch 16 TRAIN:  36%|███▋      | 49/135 [00:11<00:16,  5.31it/s, epoch=16, grad_norm=97.767, loss=345.0650, lr=1.78e-04]
Epoch 16 TRAIN:  36%|███▋      | 49/135 [00:11<00:16,  5.31it/s, epoch=16, grad_norm=80.701, loss=303.7837, lr=1.78e-04]
Epoch 16 TRAIN:  36%|███▋      | 49/135 [00:12<00:16,  5.31it/s, epoch=16, grad_norm=116.871, loss=352.6805, lr=1.78e-04]
Epoch 16 TRAIN:  36%|███▋      | 49/135 [00:12<00:16,  5.31it/s, epoch=16, grad_norm=83.019, loss=362.7734, lr=1.78e-04] 
Epoch 16 TRAIN:  39%|███▊      | 52/135 [00:12<00:15,  5.33it/s, epoch=16, grad_norm=83.019, loss=362.7734, lr=1.78e-04]
Epoch 16 TRAIN:  39%|███▊      | 52/135 [00:12<00:15,  5.33it/s, epoch=16, grad_norm=106.436, loss=330.1034, lr=1.78e-04]
Epoch 16 TRAIN:  39%|███▊      | 52/135 [00:12<00:15,  5.33it/s, epoch=16, grad_norm=77.274, loss=305.5852, lr=1.79e-04] 
Epoch 16 TRAIN:  39%|███▊      | 52/135 [00:12<00:15,  5.33it/s, epoch=16, grad_norm=66.091, loss=357.5568, lr=1.79e-04]
Epoch 16 TRAIN:  41%|████      | 55/135 [00:12<00:15,  5.24it/s, epoch=16, grad_norm=66.091, loss=357.5568, lr=1.79e-04]
Epoch 16 TRAIN:  41%|████      | 55/135 [00:13<00:15,  5.24it/s, epoch=16, grad_norm=80.037, loss=369.1111, lr=1.79e-04]
Epoch 16 TRAIN:  41%|████      | 55/135 [00:13<00:15,  5.24it/s, epoch=16, grad_norm=105.553, loss=332.5220, lr=1.79e-04]
Epoch 16 TRAIN:  41%|████      | 55/135 [00:13<00:15,  5.24it/s, epoch=16, grad_norm=129.451, loss=292.1150, lr=1.79e-04]
Epoch 16 TRAIN:  43%|████▎     | 58/135 [00:13<00:14,  5.34it/s, epoch=16, grad_norm=129.451, loss=292.1150, lr=1.79e-04]
Epoch 16 TRAIN:  43%|████▎     | 58/135 [00:13<00:14,  5.34it/s, epoch=16, grad_norm=142.672, loss=295.9521, lr=1.79e-04]
Epoch 16 TRAIN:  43%|████▎     | 58/135 [00:13<00:14,  5.34it/s, epoch=16, grad_norm=80.924, loss=333.7667, lr=1.79e-04] 
Epoch 16 TRAIN:  43%|████▎     | 58/135 [00:14<00:14,  5.34it/s, epoch=16, grad_norm=54.950, loss=290.7135, lr=1.79e-04]
Epoch 16 TRAIN:  45%|████▌     | 61/135 [00:14<00:14,  5.10it/s, epoch=16, grad_norm=54.950, loss=290.7135, lr=1.79e-04]
Epoch 16 TRAIN:  45%|████▌     | 61/135 [00:14<00:14,  5.10it/s, epoch=16, grad_norm=183.065, loss=362.7692, lr=1.79e-04]
Epoch 16 TRAIN:  45%|████▌     | 61/135 [00:14<00:14,  5.10it/s, epoch=16, grad_norm=75.522, loss=291.8638, lr=1.79e-04] 
Epoch 16 TRAIN:  45%|████▌     | 61/135 [00:14<00:14,  5.10it/s, epoch=16, grad_norm=146.068, loss=364.3317, lr=1.79e-04]
Epoch 16 TRAIN:  47%|████▋     | 64/135 [00:14<00:13,  5.21it/s, epoch=16, grad_norm=146.068, loss=364.3317, lr=1.79e-04]
Epoch 16 TRAIN:  47%|████▋     | 64/135 [00:14<00:13,  5.21it/s, epoch=16, grad_norm=77.536, loss=273.2506, lr=1.79e-04] 
Epoch 16 TRAIN:  47%|████▋     | 64/135 [00:14<00:13,  5.21it/s, epoch=16, grad_norm=111.088, loss=378.0328, lr=1.80e-04]
Epoch 16 TRAIN:  47%|████▋     | 64/135 [00:15<00:13,  5.21it/s, epoch=16, grad_norm=195.038, loss=266.2758, lr=1.80e-04]
Epoch 16 TRAIN:  50%|████▉     | 67/135 [00:15<00:12,  5.32it/s, epoch=16, grad_norm=195.038, loss=266.2758, lr=1.80e-04]
Epoch 16 TRAIN:  50%|████▉     | 67/135 [00:15<00:12,  5.32it/s, epoch=16, grad_norm=88.947, loss=342.5859, lr=1.80e-04] 
Epoch 16 TRAIN:  50%|████▉     | 67/135 [00:15<00:12,  5.32it/s, epoch=16, grad_norm=121.144, loss=308.3156, lr=1.80e-04]
Epoch 16 TRAIN:  50%|████▉     | 67/135 [00:15<00:12,  5.32it/s, epoch=16, grad_norm=60.655, loss=312.4771, lr=1.80e-04] 
Epoch 16 TRAIN:  52%|█████▏    | 70/135 [00:15<00:12,  5.24it/s, epoch=16, grad_norm=60.655, loss=312.4771, lr=1.80e-04]
Epoch 16 TRAIN:  52%|█████▏    | 70/135 [00:15<00:12,  5.24it/s, epoch=16, grad_norm=59.700, loss=328.1956, lr=1.80e-04]
Epoch 16 TRAIN:  52%|█████▏    | 70/135 [00:16<00:12,  5.24it/s, epoch=16, grad_norm=79.699, loss=299.8578, lr=1.80e-04]
Epoch 16 TRAIN:  52%|█████▏    | 70/135 [00:16<00:12,  5.24it/s, epoch=16, grad_norm=88.924, loss=306.4501, lr=1.80e-04]
Epoch 16 TRAIN:  54%|█████▍    | 73/135 [00:16<00:11,  5.33it/s, epoch=16, grad_norm=88.924, loss=306.4501, lr=1.80e-04]
Epoch 16 TRAIN:  54%|█████▍    | 73/135 [00:16<00:11,  5.33it/s, epoch=16, grad_norm=95.330, loss=398.2716, lr=1.80e-04]
Epoch 16 TRAIN:  54%|█████▍    | 73/135 [00:16<00:11,  5.33it/s, epoch=16, grad_norm=154.331, loss=261.9340, lr=1.80e-04]
Epoch 16 TRAIN:  54%|█████▍    | 73/135 [00:16<00:11,  5.33it/s, epoch=16, grad_norm=79.473, loss=267.0878, lr=1.80e-04] 
Epoch 16 TRAIN:  56%|█████▋    | 76/135 [00:16<00:11,  5.11it/s, epoch=16, grad_norm=79.473, loss=267.0878, lr=1.80e-04]
Epoch 16 TRAIN:  56%|█████▋    | 76/135 [00:17<00:11,  5.11it/s, epoch=16, grad_norm=91.984, loss=307.8343, lr=1.80e-04]
Epoch 16 TRAIN:  56%|█████▋    | 76/135 [00:17<00:11,  5.11it/s, epoch=16, grad_norm=108.099, loss=280.9006, lr=1.80e-04]
Epoch 16 TRAIN:  56%|█████▋    | 76/135 [00:17<00:11,  5.11it/s, epoch=16, grad_norm=76.684, loss=310.9357, lr=1.81e-04] 
Epoch 16 TRAIN:  59%|█████▊    | 79/135 [00:17<00:10,  5.30it/s, epoch=16, grad_norm=76.684, loss=310.9357, lr=1.81e-04]
Epoch 16 TRAIN:  59%|█████▊    | 79/135 [00:17<00:10,  5.30it/s, epoch=16, grad_norm=143.029, loss=347.9698, lr=1.81e-04]
Epoch 16 TRAIN:  59%|█████▊    | 79/135 [00:17<00:10,  5.30it/s, epoch=16, grad_norm=70.952, loss=316.5832, lr=1.81e-04] 
Epoch 16 TRAIN:  59%|█████▊    | 79/135 [00:18<00:10,  5.30it/s, epoch=16, grad_norm=70.631, loss=339.6983, lr=1.81e-04]
Epoch 16 TRAIN:  61%|██████    | 82/135 [00:18<00:10,  5.16it/s, epoch=16, grad_norm=70.631, loss=339.6983, lr=1.81e-04]
Epoch 16 TRAIN:  61%|██████    | 82/135 [00:18<00:10,  5.16it/s, epoch=16, grad_norm=124.483, loss=285.6288, lr=1.81e-04]
Epoch 16 TRAIN:  61%|██████    | 82/135 [00:18<00:10,  5.16it/s, epoch=16, grad_norm=75.698, loss=326.4594, lr=1.81e-04] 
Epoch 16 TRAIN:  61%|██████    | 82/135 [00:18<00:10,  5.16it/s, epoch=16, grad_norm=134.533, loss=252.9450, lr=1.81e-04]
Epoch 16 TRAIN:  63%|██████▎   | 85/135 [00:18<00:09,  5.11it/s, epoch=16, grad_norm=134.533, loss=252.9450, lr=1.81e-04]
Epoch 16 TRAIN:  63%|██████▎   | 85/135 [00:18<00:09,  5.11it/s, epoch=16, grad_norm=72.784, loss=268.6164, lr=1.81e-04] 
Epoch 16 TRAIN:  63%|██████▎   | 85/135 [00:19<00:09,  5.11it/s, epoch=16, grad_norm=86.082, loss=317.5687, lr=1.81e-04]
Epoch 16 TRAIN:  63%|██████▎   | 85/135 [00:19<00:09,  5.11it/s, epoch=16, grad_norm=77.411, loss=319.9424, lr=1.81e-04]
Epoch 16 TRAIN:  65%|██████▌   | 88/135 [00:19<00:09,  5.10it/s, epoch=16, grad_norm=77.411, loss=319.9424, lr=1.81e-04]
Epoch 16 TRAIN:  65%|██████▌   | 88/135 [00:19<00:09,  5.10it/s, epoch=16, grad_norm=122.039, loss=363.6534, lr=1.81e-04]
Epoch 16 TRAIN:  65%|██████▌   | 88/135 [00:19<00:09,  5.10it/s, epoch=16, grad_norm=112.356, loss=358.3614, lr=1.81e-04]
Epoch 16 TRAIN:  65%|██████▌   | 88/135 [00:19<00:09,  5.10it/s, epoch=16, grad_norm=74.149, loss=309.7353, lr=1.82e-04] 
Epoch 16 TRAIN:  67%|██████▋   | 91/135 [00:19<00:08,  5.12it/s, epoch=16, grad_norm=74.149, loss=309.7353, lr=1.82e-04]
Epoch 16 TRAIN:  67%|██████▋   | 91/135 [00:19<00:08,  5.12it/s, epoch=16, grad_norm=112.688, loss=254.2092, lr=1.82e-04]
Epoch 16 TRAIN:  67%|██████▋   | 91/135 [00:20<00:08,  5.12it/s, epoch=16, grad_norm=77.856, loss=339.2605, lr=1.82e-04] 
Epoch 16 TRAIN:  67%|██████▋   | 91/135 [00:20<00:08,  5.12it/s, epoch=16, grad_norm=90.830, loss=290.9399, lr=1.82e-04]
Epoch 16 TRAIN:  70%|██████▉   | 94/135 [00:20<00:07,  5.16it/s, epoch=16, grad_norm=90.830, loss=290.9399, lr=1.82e-04]
Epoch 16 TRAIN:  70%|██████▉   | 94/135 [00:20<00:07,  5.16it/s, epoch=16, grad_norm=63.113, loss=301.6245, lr=1.82e-04]
Epoch 16 TRAIN:  70%|██████▉   | 94/135 [00:20<00:07,  5.16it/s, epoch=16, grad_norm=130.378, loss=281.3462, lr=1.82e-04]
Epoch 16 TRAIN:  70%|██████▉   | 94/135 [00:21<00:07,  5.16it/s, epoch=16, grad_norm=125.847, loss=280.6328, lr=1.82e-04]
Epoch 16 TRAIN:  72%|███████▏  | 97/135 [00:21<00:07,  4.77it/s, epoch=16, grad_norm=125.847, loss=280.6328, lr=1.82e-04]
Epoch 16 TRAIN:  72%|███████▏  | 97/135 [00:21<00:07,  4.77it/s, epoch=16, grad_norm=78.646, loss=302.1243, lr=1.82e-04] 
Epoch 16 TRAIN:  72%|███████▏  | 97/135 [00:21<00:07,  4.77it/s, epoch=16, grad_norm=85.700, loss=341.0167, lr=1.82e-04]
Epoch 16 TRAIN:  72%|███████▏  | 97/135 [00:21<00:07,  4.77it/s, epoch=16, grad_norm=84.048, loss=306.0054, lr=1.82e-04]2026-04-16 20:00:00,627 DEBUG TRAIN | steps/sec 4.610| Batch 16/100 loss 306.005371 loss_att 277.607330 loss_ctc 372.267426 th_accuracy 0.250460 lr 1.8224e-04 grad_norm 84.048477 rank 0

Epoch 16 TRAIN:  74%|███████▍  | 100/135 [00:21<00:07,  4.80it/s, epoch=16, grad_norm=84.048, loss=306.0054, lr=1.82e-04]
Epoch 16 TRAIN:  74%|███████▍  | 100/135 [00:21<00:07,  4.80it/s, epoch=16, grad_norm=72.187, loss=264.4725, lr=1.82e-04]
Epoch 16 TRAIN:  74%|███████▍  | 100/135 [00:22<00:07,  4.80it/s, epoch=16, grad_norm=80.001, loss=267.0361, lr=1.82e-04]
Epoch 16 TRAIN:  74%|███████▍  | 100/135 [00:22<00:07,  4.80it/s, epoch=16, grad_norm=151.669, loss=316.0517, lr=1.82e-04]
Epoch 16 TRAIN:  76%|███████▋  | 103/135 [00:22<00:06,  4.96it/s, epoch=16, grad_norm=151.669, loss=316.0517, lr=1.82e-04]
Epoch 16 TRAIN:  76%|███████▋  | 103/135 [00:22<00:06,  4.96it/s, epoch=16, grad_norm=101.607, loss=289.3425, lr=1.83e-04]
Epoch 16 TRAIN:  76%|███████▋  | 103/135 [00:22<00:06,  4.96it/s, epoch=16, grad_norm=120.519, loss=325.2827, lr=1.83e-04]
Epoch 16 TRAIN:  76%|███████▋  | 103/135 [00:22<00:06,  4.96it/s, epoch=16, grad_norm=76.189, loss=263.5417, lr=1.83e-04] 
Epoch 16 TRAIN:  79%|███████▊  | 106/135 [00:22<00:05,  4.86it/s, epoch=16, grad_norm=76.189, loss=263.5417, lr=1.83e-04]
Epoch 16 TRAIN:  79%|███████▊  | 106/135 [00:23<00:05,  4.86it/s, epoch=16, grad_norm=120.613, loss=275.6225, lr=1.83e-04]
Epoch 16 TRAIN:  79%|███████▊  | 106/135 [00:23<00:05,  4.86it/s, epoch=16, grad_norm=86.743, loss=279.3085, lr=1.83e-04] 
Epoch 16 TRAIN:  79%|███████▊  | 106/135 [00:23<00:05,  4.86it/s, epoch=16, grad_norm=115.900, loss=244.8113, lr=1.83e-04]
Epoch 16 TRAIN:  81%|████████  | 109/135 [00:23<00:05,  4.81it/s, epoch=16, grad_norm=115.900, loss=244.8113, lr=1.83e-04]
Epoch 16 TRAIN:  81%|████████  | 109/135 [00:23<00:05,  4.81it/s, epoch=16, grad_norm=143.836, loss=310.5102, lr=1.83e-04]
Epoch 16 TRAIN:  81%|████████  | 109/135 [00:23<00:05,  4.81it/s, epoch=16, grad_norm=66.839, loss=311.6271, lr=1.83e-04] 
Epoch 16 TRAIN:  81%|████████  | 109/135 [00:24<00:05,  4.81it/s, epoch=16, grad_norm=199.757, loss=397.7695, lr=1.83e-04]
Epoch 16 TRAIN:  83%|████████▎ | 112/135 [00:24<00:04,  5.09it/s, epoch=16, grad_norm=199.757, loss=397.7695, lr=1.83e-04]
Epoch 16 TRAIN:  83%|████████▎ | 112/135 [00:24<00:04,  5.09it/s, epoch=16, grad_norm=161.960, loss=350.4194, lr=1.83e-04]
Epoch 16 TRAIN:  83%|████████▎ | 112/135 [00:24<00:04,  5.09it/s, epoch=16, grad_norm=108.878, loss=272.7577, lr=1.83e-04]
Epoch 16 TRAIN:  83%|████████▎ | 112/135 [00:24<00:04,  5.09it/s, epoch=16, grad_norm=72.320, loss=302.8728, lr=1.83e-04] 
Epoch 16 TRAIN:  85%|████████▌ | 115/135 [00:24<00:04,  4.86it/s, epoch=16, grad_norm=72.320, loss=302.8728, lr=1.83e-04]
Epoch 16 TRAIN:  85%|████████▌ | 115/135 [00:24<00:04,  4.86it/s, epoch=16, grad_norm=80.989, loss=326.6010, lr=1.84e-04]
Epoch 16 TRAIN:  85%|████████▌ | 115/135 [00:25<00:04,  4.86it/s, epoch=16, grad_norm=91.543, loss=404.3798, lr=1.84e-04]
Epoch 16 TRAIN:  85%|████████▌ | 115/135 [00:25<00:04,  4.86it/s, epoch=16, grad_norm=116.263, loss=273.9074, lr=1.84e-04]
Epoch 16 TRAIN:  87%|████████▋ | 118/135 [00:25<00:03,  5.10it/s, epoch=16, grad_norm=116.263, loss=273.9074, lr=1.84e-04]
Epoch 16 TRAIN:  87%|████████▋ | 118/135 [00:25<00:03,  5.10it/s, epoch=16, grad_norm=66.514, loss=286.4794, lr=1.84e-04] 
Epoch 16 TRAIN:  87%|████████▋ | 118/135 [00:25<00:03,  5.10it/s, epoch=16, grad_norm=136.750, loss=282.1815, lr=1.84e-04]
Epoch 16 TRAIN:  87%|████████▋ | 118/135 [00:25<00:03,  5.10it/s, epoch=16, grad_norm=121.980, loss=385.8694, lr=1.84e-04]
Epoch 16 TRAIN:  90%|████████▉ | 121/135 [00:25<00:02,  5.31it/s, epoch=16, grad_norm=121.980, loss=385.8694, lr=1.84e-04]
Epoch 16 TRAIN:  90%|████████▉ | 121/135 [00:26<00:02,  5.31it/s, epoch=16, grad_norm=78.938, loss=301.9317, lr=1.84e-04] 
Epoch 16 TRAIN:  90%|████████▉ | 121/135 [00:26<00:02,  5.31it/s, epoch=16, grad_norm=96.116, loss=355.1500, lr=1.84e-04]
Epoch 16 TRAIN:  90%|████████▉ | 121/135 [00:26<00:02,  5.31it/s, epoch=16, grad_norm=122.106, loss=335.4889, lr=1.84e-04]
Epoch 16 TRAIN:  92%|█████████▏| 124/135 [00:26<00:02,  5.45it/s, epoch=16, grad_norm=122.106, loss=335.4889, lr=1.84e-04]
Epoch 16 TRAIN:  92%|█████████▏| 124/135 [00:26<00:02,  5.45it/s, epoch=16, grad_norm=105.147, loss=292.7589, lr=1.84e-04]
Epoch 16 TRAIN:  92%|█████████▏| 124/135 [00:26<00:02,  5.45it/s, epoch=16, grad_norm=71.500, loss=267.1570, lr=1.84e-04] 
Epoch 16 TRAIN:  92%|█████████▏| 124/135 [00:26<00:02,  5.45it/s, epoch=16, grad_norm=213.831, loss=384.3670, lr=1.84e-04]
Epoch 16 TRAIN:  94%|█████████▍| 127/135 [00:26<00:01,  5.35it/s, epoch=16, grad_norm=213.831, loss=384.3670, lr=1.84e-04]
Epoch 16 TRAIN:  94%|█████████▍| 127/135 [00:27<00:01,  5.35it/s, epoch=16, grad_norm=195.486, loss=437.9433, lr=1.84e-04]
Epoch 16 TRAIN:  94%|█████████▍| 127/135 [00:27<00:01,  5.35it/s, epoch=16, grad_norm=198.236, loss=404.7936, lr=1.85e-04]
Epoch 16 TRAIN:  94%|█████████▍| 127/135 [00:27<00:01,  5.35it/s, epoch=16, grad_norm=131.854, loss=416.3786, lr=1.85e-04]
Epoch 16 TRAIN:  96%|█████████▋| 130/135 [00:27<00:00,  5.40it/s, epoch=16, grad_norm=131.854, loss=416.3786, lr=1.85e-04]
Epoch 16 TRAIN:  96%|█████████▋| 130/135 [00:27<00:00,  5.40it/s, epoch=16, grad_norm=116.995, loss=392.1749, lr=1.85e-04]
Epoch 16 TRAIN:  96%|█████████▋| 130/135 [00:27<00:00,  5.40it/s, epoch=16, grad_norm=143.489, loss=334.4288, lr=1.85e-04]
Epoch 16 TRAIN:  96%|█████████▋| 130/135 [00:27<00:00,  5.40it/s, epoch=16, grad_norm=147.013, loss=448.6377, lr=1.85e-04]
Epoch 16 TRAIN:  99%|█████████▊| 133/135 [00:27<00:00,  5.56it/s, epoch=16, grad_norm=147.013, loss=448.6377, lr=1.85e-04]
Epoch 16 TRAIN:  99%|█████████▊| 133/135 [00:28<00:00,  5.56it/s, epoch=16, grad_norm=135.765, loss=346.0657, lr=1.85e-04]
Epoch 16 TRAIN:  99%|█████████▊| 133/135 [00:28<00:00,  5.56it/s, epoch=16, grad_norm=195.694, loss=350.8045, lr=1.85e-04]
Epoch 16 TRAIN:  99%|█████████▊| 133/135 [00:28<00:00,  5.56it/s, epoch=16, grad_norm=187.924, loss=302.5084, lr=1.85e-04]
Epoch 16 TRAIN: 136it [00:28,  5.42it/s, epoch=16, grad_norm=187.924, loss=302.5084, lr=1.85e-04]                         
Epoch 16 TRAIN: 136it [00:28,  4.77it/s, epoch=16, grad_norm=187.924, loss=302.5084, lr=1.85e-04]

CV:   0%|          | 0/6 [00:00<?, ?it/s]/home/rookie/Documents/batuka/projects/wenet/wenet/wenet/utils/train_utils.py:689: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  amp_autocast(enabled=dtype is not None,

CV:  17%|█▋        | 1/6 [00:00<00:03,  1.33it/s]
                                                 
2026-04-16 20:00:08,285 INFO Epoch 16 Step 2314 CV info lr 1.8512e-04 cv_loss 383.5077597878196 rank 0 acc 0.24574621270100275
2026-04-16 20:00:08,287 INFO [Rank 0] Checkpoint: save to checkpoint exp/conformer_mongolian/epoch_16.pt
2026-04-16 20:00:08,402 INFO Epoch 17 Step 2314 TRAIN info lr 1.8512e-04 rank 0
2026-04-16 20:00:08,420 INFO using accumulate grad, new batch size is 1 times larger than before

Epoch 17 TRAIN:   0%|          | 0/136 [00:00<?, ?it/s]
Epoch 17 TRAIN:   0%|          | 0/136 [00:03<?, ?it/s, epoch=17, grad_norm=470.628, loss=461.0970, lr=1.85e-04]
Epoch 17 TRAIN:   1%|          | 1/136 [00:03<08:53,  3.95s/it, epoch=17, grad_norm=470.628, loss=461.0970, lr=1.85e-04]
Epoch 17 TRAIN:   1%|          | 1/136 [00:04<08:53,  3.95s/it, epoch=17, grad_norm=435.992, loss=415.4482, lr=1.85e-04]
Epoch 17 TRAIN:   1%|          | 1/136 [00:04<08:53,  3.95s/it, epoch=17, grad_norm=302.921, loss=334.4395, lr=1.85e-04]
Epoch 17 TRAIN:   1%|          | 1/136 [00:04<08:53,  3.95s/it, epoch=17, grad_norm=294.940, loss=356.9672, lr=1.85e-04]
Epoch 17 TRAIN:   1%|          | 1/136 [00:04<08:53,  3.95s/it, epoch=17, grad_norm=305.001, loss=339.7513, lr=1.86e-04]
Epoch 17 TRAIN:   1%|          | 1/136 [00:04<08:53,  3.95s/it, epoch=17, grad_norm=95.392, loss=212.4086, lr=1.86e-04] 
Epoch 17 TRAIN:   1%|          | 1/136 [00:04<08:53,  3.95s/it, epoch=17, grad_norm=329.966, loss=382.1852, lr=1.86e-04]
Epoch 17 TRAIN:   1%|          | 1/136 [00:04<08:53,  3.95s/it, epoch=17, grad_norm=340.073, loss=411.9834, lr=1.86e-04]
Epoch 17 TRAIN:   6%|▌         | 8/136 [00:04<00:54,  2.36it/s, epoch=17, grad_norm=340.073, loss=411.9834, lr=1.86e-04]
Epoch 17 TRAIN:   6%|▌         | 8/136 [00:04<00:54,  2.36it/s, epoch=17, grad_norm=180.316, loss=346.5482, lr=1.86e-04]
Epoch 17 TRAIN:   6%|▌         | 8/136 [00:04<00:54,  2.36it/s, epoch=17, grad_norm=114.722, loss=337.0608, lr=1.86e-04]
Epoch 17 TRAIN:   6%|▌         | 8/136 [00:04<00:54,  2.36it/s, epoch=17, grad_norm=75.094, loss=304.6332, lr=1.86e-04] 
Epoch 17 TRAIN:   6%|▌         | 8/136 [00:05<00:54,  2.36it/s, epoch=17, grad_norm=179.071, loss=293.0489, lr=1.86e-04]
Epoch 17 TRAIN:   6%|▌         | 8/136 [00:05<00:54,  2.36it/s, epoch=17, grad_norm=91.033, loss=330.2033, lr=1.86e-04] 
Epoch 17 TRAIN:  10%|▉         | 13/136 [00:05<00:36,  3.39it/s, epoch=17, grad_norm=91.033, loss=330.2033, lr=1.86e-04]
Epoch 17 TRAIN:  10%|▉         | 13/136 [00:05<00:36,  3.39it/s, epoch=17, grad_norm=93.375, loss=332.4986, lr=1.86e-04]
Epoch 17 TRAIN:  10%|▉         | 13/136 [00:05<00:36,  3.39it/s, epoch=17, grad_norm=169.602, loss=309.4103, lr=1.86e-04]
Epoch 17 TRAIN:  10%|▉         | 13/136 [00:05<00:36,  3.39it/s, epoch=17, grad_norm=83.209, loss=336.6871, lr=1.86e-04] 
Epoch 17 TRAIN:  10%|▉         | 13/136 [00:05<00:36,  3.39it/s, epoch=17, grad_norm=58.698, loss=379.6562, lr=1.86e-04]
Epoch 17 TRAIN:  12%|█▎        | 17/136 [00:05<00:29,  4.06it/s, epoch=17, grad_norm=58.698, loss=379.6562, lr=1.86e-04]
Epoch 17 TRAIN:  12%|█▎        | 17/136 [00:06<00:29,  4.06it/s, epoch=17, grad_norm=120.568, loss=316.4635, lr=1.87e-04]
Epoch 17 TRAIN:  12%|█▎        | 17/136 [00:06<00:29,  4.06it/s, epoch=17, grad_norm=72.259, loss=380.3192, lr=1.87e-04] 
Epoch 17 TRAIN:  12%|█▎        | 17/136 [00:06<00:29,  4.06it/s, epoch=17, grad_norm=58.463, loss=353.1779, lr=1.87e-04]
Epoch 17 TRAIN:  12%|█▎        | 17/136 [00:06<00:29,  4.06it/s, epoch=17, grad_norm=65.498, loss=328.7680, lr=1.87e-04]
Epoch 17 TRAIN:  15%|█▌        | 21/136 [00:06<00:25,  4.55it/s, epoch=17, grad_norm=65.498, loss=328.7680, lr=1.87e-04]
Epoch 17 TRAIN:  15%|█▌        | 21/136 [00:06<00:25,  4.55it/s, epoch=17, grad_norm=70.717, loss=362.2295, lr=1.87e-04]
Epoch 17 TRAIN:  15%|█▌        | 21/136 [00:06<00:25,  4.55it/s, epoch=17, grad_norm=86.255, loss=291.0304, lr=1.87e-04]
Epoch 17 TRAIN:  15%|█▌        | 21/136 [00:07<00:25,  4.55it/s, epoch=17, grad_norm=87.244, loss=284.6623, lr=1.87e-04]
Epoch 17 TRAIN:  15%|█▌        | 21/136 [00:07<00:25,  4.55it/s, epoch=17, grad_norm=87.007, loss=304.9320, lr=1.87e-04]
Epoch 17 TRAIN:  18%|█▊        | 25/136 [00:07<00:22,  4.97it/s, epoch=17, grad_norm=87.007, loss=304.9320, lr=1.87e-04]
Epoch 17 TRAIN:  18%|█▊        | 25/136 [00:07<00:22,  4.97it/s, epoch=17, grad_norm=107.366, loss=278.2108, lr=1.87e-04]
Epoch 17 TRAIN:  18%|█▊        | 25/136 [00:07<00:22,  4.97it/s, epoch=17, grad_norm=103.570, loss=355.7019, lr=1.87e-04]
Epoch 17 TRAIN:  18%|█▊        | 25/136 [00:07<00:22,  4.97it/s, epoch=17, grad_norm=77.753, loss=332.9194, lr=1.87e-04] 
Epoch 17 TRAIN:  18%|█▊        | 25/136 [00:07<00:22,  4.97it/s, epoch=17, grad_norm=120.068, loss=343.4633, lr=1.87e-04]
Epoch 17 TRAIN:  21%|██▏       | 29/136 [00:07<00:20,  5.11it/s, epoch=17, grad_norm=120.068, loss=343.4633, lr=1.87e-04]
Epoch 17 TRAIN:  21%|██▏       | 29/136 [00:08<00:20,  5.11it/s, epoch=17, grad_norm=92.867, loss=256.2484, lr=1.88e-04] 
Epoch 17 TRAIN:  21%|██▏       | 29/136 [00:08<00:20,  5.11it/s, epoch=17, grad_norm=81.124, loss=314.2338, lr=1.88e-04]
Epoch 17 TRAIN:  21%|██▏       | 29/136 [00:08<00:20,  5.11it/s, epoch=17, grad_norm=49.551, loss=291.7631, lr=1.88e-04]
Epoch 17 TRAIN:  21%|██▏       | 29/136 [00:08<00:20,  5.11it/s, epoch=17, grad_norm=71.287, loss=298.0881, lr=1.88e-04]
Epoch 17 TRAIN:  24%|██▍       | 33/136 [00:08<00:19,  5.25it/s, epoch=17, grad_norm=71.287, loss=298.0881, lr=1.88e-04]
Epoch 17 TRAIN:  24%|██▍       | 33/136 [00:08<00:19,  5.25it/s, epoch=17, grad_norm=136.129, loss=367.0692, lr=1.88e-04]
Epoch 17 TRAIN:  24%|██▍       | 33/136 [00:08<00:19,  5.25it/s, epoch=17, grad_norm=126.317, loss=251.7078, lr=1.88e-04]
Epoch 17 TRAIN:  24%|██▍       | 33/136 [00:09<00:19,  5.25it/s, epoch=17, grad_norm=51.944, loss=320.9482, lr=1.88e-04] 
Epoch 17 TRAIN:  26%|██▋       | 36/136 [00:09<00:18,  5.36it/s, epoch=17, grad_norm=51.944, loss=320.9482, lr=1.88e-04]
Epoch 17 TRAIN:  26%|██▋       | 36/136 [00:09<00:18,  5.36it/s, epoch=17, grad_norm=88.732, loss=351.0576, lr=1.88e-04]
Epoch 17 TRAIN:  26%|██▋       | 36/136 [00:09<00:18,  5.36it/s, epoch=17, grad_norm=104.411, loss=288.1114, lr=1.88e-04]
Epoch 17 TRAIN:  26%|██▋       | 36/136 [00:09<00:18,  5.36it/s, epoch=17, grad_norm=142.021, loss=346.0051, lr=1.88e-04]
Epoch 17 TRAIN:  29%|██▊       | 39/136 [00:09<00:18,  5.34it/s, epoch=17, grad_norm=142.021, loss=346.0051, lr=1.88e-04]
Epoch 17 TRAIN:  29%|██▊       | 39/136 [00:09<00:18,  5.34it/s, epoch=17, grad_norm=100.757, loss=340.9875, lr=1.88e-04]
Epoch 17 TRAIN:  29%|██▊       | 39/136 [00:10<00:18,  5.34it/s, epoch=17, grad_norm=89.654, loss=320.7535, lr=1.88e-04] 
Epoch 17 TRAIN:  29%|██▊       | 39/136 [00:10<00:18,  5.34it/s, epoch=17, grad_norm=73.705, loss=344.3625, lr=1.88e-04]
Epoch 17 TRAIN:  31%|███       | 42/136 [00:10<00:17,  5.43it/s, epoch=17, grad_norm=73.705, loss=344.3625, lr=1.88e-04]
Epoch 17 TRAIN:  31%|███       | 42/136 [00:10<00:17,  5.43it/s, epoch=17, grad_norm=52.571, loss=350.0454, lr=1.89e-04]
Epoch 17 TRAIN:  31%|███       | 42/136 [00:10<00:17,  5.43it/s, epoch=17, grad_norm=168.484, loss=269.0847, lr=1.89e-04]
Epoch 17 TRAIN:  31%|███       | 42/136 [00:10<00:17,  5.43it/s, epoch=17, grad_norm=73.883, loss=327.1569, lr=1.89e-04] 
Epoch 17 TRAIN:  33%|███▎      | 45/136 [00:10<00:17,  5.28it/s, epoch=17, grad_norm=73.883, loss=327.1569, lr=1.89e-04]
Epoch 17 TRAIN:  33%|███▎      | 45/136 [00:11<00:17,  5.28it/s, epoch=17, grad_norm=74.808, loss=355.6518, lr=1.89e-04]
Epoch 17 TRAIN:  33%|███▎      | 45/136 [00:11<00:17,  5.28it/s, epoch=17, grad_norm=60.851, loss=318.3920, lr=1.89e-04]
Epoch 17 TRAIN:  33%|███▎      | 45/136 [00:11<00:17,  5.28it/s, epoch=17, grad_norm=85.759, loss=317.9472, lr=1.89e-04]
Epoch 17 TRAIN:  35%|███▌      | 48/136 [00:11<00:16,  5.30it/s, epoch=17, grad_norm=85.759, loss=317.9472, lr=1.89e-04]
Epoch 17 TRAIN:  35%|███▌      | 48/136 [00:11<00:16,  5.30it/s, epoch=17, grad_norm=70.133, loss=343.5065, lr=1.89e-04]
Epoch 17 TRAIN:  35%|███▌      | 48/136 [00:11<00:16,  5.30it/s, epoch=17, grad_norm=65.027, loss=323.0777, lr=1.89e-04]
Epoch 17 TRAIN:  35%|███▌      | 48/136 [00:12<00:16,  5.30it/s, epoch=17, grad_norm=74.612, loss=364.5679, lr=1.89e-04]
Epoch 17 TRAIN:  38%|███▊      | 51/136 [00:12<00:16,  5.14it/s, epoch=17, grad_norm=74.612, loss=364.5679, lr=1.89e-04]
Epoch 17 TRAIN:  38%|███▊      | 51/136 [00:12<00:16,  5.14it/s, epoch=17, grad_norm=128.210, loss=365.9328, lr=1.89e-04]
Epoch 17 TRAIN:  38%|███▊      | 51/136 [00:12<00:16,  5.14it/s, epoch=17, grad_norm=133.758, loss=318.8230, lr=1.89e-04]
Epoch 17 TRAIN:  38%|███▊      | 51/136 [00:12<00:16,  5.14it/s, epoch=17, grad_norm=90.247, loss=369.6457, lr=1.89e-04] 
Epoch 17 TRAIN:  40%|███▉      | 54/136 [00:12<00:16,  5.04it/s, epoch=17, grad_norm=90.247, loss=369.6457, lr=1.89e-04]
Epoch 17 TRAIN:  40%|███▉      | 54/136 [00:12<00:16,  5.04it/s, epoch=17, grad_norm=63.802, loss=338.1184, lr=1.90e-04]
Epoch 17 TRAIN:  40%|███▉      | 54/136 [00:13<00:16,  5.04it/s, epoch=17, grad_norm=104.830, loss=297.7794, lr=1.90e-04]
Epoch 17 TRAIN:  40%|███▉      | 54/136 [00:13<00:16,  5.04it/s, epoch=17, grad_norm=101.598, loss=363.0708, lr=1.90e-04]
Epoch 17 TRAIN:  42%|████▏     | 57/136 [00:13<00:15,  5.08it/s, epoch=17, grad_norm=101.598, loss=363.0708, lr=1.90e-04]
Epoch 17 TRAIN:  42%|████▏     | 57/136 [00:13<00:15,  5.08it/s, epoch=17, grad_norm=62.981, loss=328.7593, lr=1.90e-04] 
Epoch 17 TRAIN:  42%|████▏     | 57/136 [00:13<00:15,  5.08it/s, epoch=17, grad_norm=94.903, loss=328.9217, lr=1.90e-04]
Epoch 17 TRAIN:  42%|████▏     | 57/136 [00:13<00:15,  5.08it/s, epoch=17, grad_norm=231.031, loss=233.2601, lr=1.90e-04]
Epoch 17 TRAIN:  44%|████▍     | 60/136 [00:13<00:15,  5.00it/s, epoch=17, grad_norm=231.031, loss=233.2601, lr=1.90e-04]
Epoch 17 TRAIN:  44%|████▍     | 60/136 [00:14<00:15,  5.00it/s, epoch=17, grad_norm=75.646, loss=338.3025, lr=1.90e-04] 
Epoch 17 TRAIN:  44%|████▍     | 60/136 [00:14<00:15,  5.00it/s, epoch=17, grad_norm=107.719, loss=270.2346, lr=1.90e-04]
Epoch 17 TRAIN:  44%|████▍     | 60/136 [00:14<00:15,  5.00it/s, epoch=17, grad_norm=131.795, loss=337.8983, lr=1.90e-04]
Epoch 17 TRAIN:  46%|████▋     | 63/136 [00:14<00:14,  5.14it/s, epoch=17, grad_norm=131.795, loss=337.8983, lr=1.90e-04]
Epoch 17 TRAIN:  46%|████▋     | 63/136 [00:14<00:14,  5.14it/s, epoch=17, grad_norm=93.267, loss=325.5356, lr=1.90e-04] 
Epoch 17 TRAIN:  46%|████▋     | 63/136 [00:14<00:14,  5.14it/s, epoch=17, grad_norm=192.644, loss=380.5109, lr=1.90e-04]
Epoch 17 TRAIN:  46%|████▋     | 63/136 [00:14<00:14,  5.14it/s, epoch=17, grad_norm=70.750, loss=328.5853, lr=1.90e-04] 
Epoch 17 TRAIN:  49%|████▊     | 66/136 [00:14<00:13,  5.17it/s, epoch=17, grad_norm=70.750, loss=328.5853, lr=1.90e-04]
Epoch 17 TRAIN:  49%|████▊     | 66/136 [00:15<00:13,  5.17it/s, epoch=17, grad_norm=103.904, loss=281.9815, lr=1.90e-04]
Epoch 17 TRAIN:  49%|████▊     | 66/136 [00:15<00:13,  5.17it/s, epoch=17, grad_norm=116.290, loss=321.3299, lr=1.91e-04]
Epoch 17 TRAIN:  49%|████▊     | 66/136 [00:15<00:13,  5.17it/s, epoch=17, grad_norm=133.457, loss=347.2829, lr=1.91e-04]
Epoch 17 TRAIN:  51%|█████     | 69/136 [00:15<00:13,  5.13it/s, epoch=17, grad_norm=133.457, loss=347.2829, lr=1.91e-04]
Epoch 17 TRAIN:  51%|█████     | 69/136 [00:15<00:13,  5.13it/s, epoch=17, grad_norm=98.753, loss=318.5665, lr=1.91e-04] 
Epoch 17 TRAIN:  51%|█████     | 69/136 [00:16<00:13,  5.13it/s, epoch=17, grad_norm=106.235, loss=349.4058, lr=1.91e-04]
Epoch 17 TRAIN:  51%|█████     | 69/136 [00:16<00:13,  5.13it/s, epoch=17, grad_norm=73.524, loss=324.1679, lr=1.91e-04] 
Epoch 17 TRAIN:  53%|█████▎    | 72/136 [00:16<00:12,  4.93it/s, epoch=17, grad_norm=73.524, loss=324.1679, lr=1.91e-04]
Epoch 17 TRAIN:  53%|█████▎    | 72/136 [00:16<00:12,  4.93it/s, epoch=17, grad_norm=78.151, loss=305.9091, lr=1.91e-04]
Epoch 17 TRAIN:  53%|█████▎    | 72/136 [00:16<00:12,  4.93it/s, epoch=17, grad_norm=99.873, loss=338.7928, lr=1.91e-04]
Epoch 17 TRAIN:  53%|█████▎    | 72/136 [00:16<00:12,  4.93it/s, epoch=17, grad_norm=165.097, loss=409.9619, lr=1.91e-04]
Epoch 17 TRAIN:  55%|█████▌    | 75/136 [00:16<00:12,  4.99it/s, epoch=17, grad_norm=165.097, loss=409.9619, lr=1.91e-04]
Epoch 17 TRAIN:  55%|█████▌    | 75/136 [00:17<00:12,  4.99it/s, epoch=17, grad_norm=106.305, loss=280.1649, lr=1.91e-04]
Epoch 17 TRAIN:  55%|█████▌    | 75/136 [00:17<00:12,  4.99it/s, epoch=17, grad_norm=115.391, loss=354.6410, lr=1.91e-04]
Epoch 17 TRAIN:  55%|█████▌    | 75/136 [00:17<00:12,  4.99it/s, epoch=17, grad_norm=79.419, loss=313.1413, lr=1.91e-04] 
Epoch 17 TRAIN:  57%|█████▋    | 78/136 [00:17<00:11,  5.14it/s, epoch=17, grad_norm=79.419, loss=313.1413, lr=1.91e-04]
Epoch 17 TRAIN:  57%|█████▋    | 78/136 [00:17<00:11,  5.14it/s, epoch=17, grad_norm=87.365, loss=272.4998, lr=1.91e-04]
Epoch 17 TRAIN:  57%|█████▋    | 78/136 [00:17<00:11,  5.14it/s, epoch=17, grad_norm=105.609, loss=321.4903, lr=1.92e-04]
Epoch 17 TRAIN:  57%|█████▋    | 78/136 [00:17<00:11,  5.14it/s, epoch=17, grad_norm=159.142, loss=285.5717, lr=1.92e-04]
Epoch 17 TRAIN:  60%|█████▉    | 81/136 [00:17<00:10,  5.06it/s, epoch=17, grad_norm=159.142, loss=285.5717, lr=1.92e-04]
Epoch 17 TRAIN:  60%|█████▉    | 81/136 [00:18<00:10,  5.06it/s, epoch=17, grad_norm=160.355, loss=367.7115, lr=1.92e-04]
Epoch 17 TRAIN:  60%|█████▉    | 81/136 [00:18<00:10,  5.06it/s, epoch=17, grad_norm=110.344, loss=351.2381, lr=1.92e-04]
Epoch 17 TRAIN:  60%|█████▉    | 81/136 [00:18<00:10,  5.06it/s, epoch=17, grad_norm=189.419, loss=232.1259, lr=1.92e-04]
Epoch 17 TRAIN:  62%|██████▏   | 84/136 [00:18<00:10,  5.02it/s, epoch=17, grad_norm=189.419, loss=232.1259, lr=1.92e-04]
Epoch 17 TRAIN:  62%|██████▏   | 84/136 [00:18<00:10,  5.02it/s, epoch=17, grad_norm=134.081, loss=301.0702, lr=1.92e-04]
Epoch 17 TRAIN:  62%|██████▏   | 84/136 [00:18<00:10,  5.02it/s, epoch=17, grad_norm=95.391, loss=323.6356, lr=1.92e-04] 
Epoch 17 TRAIN:  62%|██████▏   | 84/136 [00:19<00:10,  5.02it/s, epoch=17, grad_norm=113.563, loss=337.8258, lr=1.92e-04]
Epoch 17 TRAIN:  64%|██████▍   | 87/136 [00:19<00:09,  5.05it/s, epoch=17, grad_norm=113.563, loss=337.8258, lr=1.92e-04]
Epoch 17 TRAIN:  64%|██████▍   | 87/136 [00:19<00:09,  5.05it/s, epoch=17, grad_norm=131.469, loss=241.6834, lr=1.92e-04]
Epoch 17 TRAIN:  64%|██████▍   | 87/136 [00:19<00:09,  5.05it/s, epoch=17, grad_norm=168.193, loss=352.4500, lr=1.92e-04]
Epoch 17 TRAIN:  64%|██████▍   | 87/136 [00:19<00:09,  5.05it/s, epoch=17, grad_norm=81.730, loss=323.7570, lr=1.92e-04] 
Epoch 17 TRAIN:  66%|██████▌   | 90/136 [00:19<00:09,  4.88it/s, epoch=17, grad_norm=81.730, loss=323.7570, lr=1.92e-04]
Epoch 17 TRAIN:  66%|██████▌   | 90/136 [00:20<00:09,  4.88it/s, epoch=17, grad_norm=129.315, loss=290.9061, lr=1.92e-04]
Epoch 17 TRAIN:  66%|██████▌   | 90/136 [00:20<00:09,  4.88it/s, epoch=17, grad_norm=145.673, loss=242.1029, lr=1.92e-04]
Epoch 17 TRAIN:  66%|██████▌   | 90/136 [00:20<00:09,  4.88it/s, epoch=17, grad_norm=80.175, loss=280.1492, lr=1.93e-04] 
Epoch 17 TRAIN:  68%|██████▊   | 93/136 [00:20<00:08,  4.93it/s, epoch=17, grad_norm=80.175, loss=280.1492, lr=1.93e-04]
Epoch 17 TRAIN:  68%|██████▊   | 93/136 [00:20<00:08,  4.93it/s, epoch=17, grad_norm=110.887, loss=255.1816, lr=1.93e-04]
Epoch 17 TRAIN:  68%|██████▊   | 93/136 [00:20<00:08,  4.93it/s, epoch=17, grad_norm=92.932, loss=285.0358, lr=1.93e-04] 
Epoch 17 TRAIN:  68%|██████▊   | 93/136 [00:21<00:08,  4.93it/s, epoch=17, grad_norm=184.991, loss=370.0198, lr=1.93e-04]
Epoch 17 TRAIN:  71%|███████   | 96/136 [00:21<00:08,  4.91it/s, epoch=17, grad_norm=184.991, loss=370.0198, lr=1.93e-04]
Epoch 17 TRAIN:  71%|███████   | 96/136 [00:21<00:08,  4.91it/s, epoch=17, grad_norm=93.963, loss=318.5419, lr=1.93e-04] 
Epoch 17 TRAIN:  71%|███████   | 96/136 [00:21<00:08,  4.91it/s, epoch=17, grad_norm=175.299, loss=211.1489, lr=1.93e-04]
Epoch 17 TRAIN:  71%|███████   | 96/136 [00:21<00:08,  4.91it/s, epoch=17, grad_norm=205.405, loss=342.1467, lr=1.93e-04]
Epoch 17 TRAIN:  73%|███████▎  | 99/136 [00:21<00:07,  5.06it/s, epoch=17, grad_norm=205.405, loss=342.1467, lr=1.93e-04]
Epoch 17 TRAIN:  73%|███████▎  | 99/136 [00:21<00:07,  5.06it/s, epoch=17, grad_norm=117.247, loss=273.1171, lr=1.93e-04]2026-04-16 20:00:30,228 DEBUG TRAIN | steps/sec 4.594| Batch 17/100 loss 273.117126 loss_att 249.959290 loss_ctc 327.152130 th_accuracy 0.261580 lr 1.9312e-04 grad_norm 117.247375 rank 0

Epoch 17 TRAIN:  73%|███████▎  | 99/136 [00:21<00:07,  5.06it/s, epoch=17, grad_norm=226.891, loss=375.4178, lr=1.93e-04]
Epoch 17 TRAIN:  73%|███████▎  | 99/136 [00:22<00:07,  5.06it/s, epoch=17, grad_norm=80.029, loss=284.9201, lr=1.93e-04] 
Epoch 17 TRAIN:  75%|███████▌  | 102/136 [00:22<00:06,  5.11it/s, epoch=17, grad_norm=80.029, loss=284.9201, lr=1.93e-04]
Epoch 17 TRAIN:  75%|███████▌  | 102/136 [00:22<00:06,  5.11it/s, epoch=17, grad_norm=80.955, loss=249.4425, lr=1.93e-04]
Epoch 17 TRAIN:  75%|███████▌  | 102/136 [00:22<00:06,  5.11it/s, epoch=17, grad_norm=91.870, loss=274.4497, lr=1.93e-04]
Epoch 17 TRAIN:  75%|███████▌  | 102/136 [00:22<00:06,  5.11it/s, epoch=17, grad_norm=109.232, loss=292.6330, lr=1.94e-04]
Epoch 17 TRAIN:  77%|███████▋  | 105/136 [00:22<00:06,  4.81it/s, epoch=17, grad_norm=109.232, loss=292.6330, lr=1.94e-04]
Epoch 17 TRAIN:  77%|███████▋  | 105/136 [00:23<00:06,  4.81it/s, epoch=17, grad_norm=140.049, loss=375.8004, lr=1.94e-04]
Epoch 17 TRAIN:  77%|███████▋  | 105/136 [00:23<00:06,  4.81it/s, epoch=17, grad_norm=117.908, loss=333.2741, lr=1.94e-04]
Epoch 17 TRAIN:  77%|███████▋  | 105/136 [00:23<00:06,  4.81it/s, epoch=17, grad_norm=87.374, loss=330.5852, lr=1.94e-04] 
Epoch 17 TRAIN:  79%|███████▉  | 108/136 [00:23<00:05,  5.02it/s, epoch=17, grad_norm=87.374, loss=330.5852, lr=1.94e-04]
Epoch 17 TRAIN:  79%|███████▉  | 108/136 [00:23<00:05,  5.02it/s, epoch=17, grad_norm=73.164, loss=310.1933, lr=1.94e-04]
Epoch 17 TRAIN:  79%|███████▉  | 108/136 [00:23<00:05,  5.02it/s, epoch=17, grad_norm=129.180, loss=330.5441, lr=1.94e-04]
Epoch 17 TRAIN:  79%|███████▉  | 108/136 [00:24<00:05,  5.02it/s, epoch=17, grad_norm=99.353, loss=303.0871, lr=1.94e-04] 
Epoch 17 TRAIN:  82%|████████▏ | 111/136 [00:24<00:05,  4.63it/s, epoch=17, grad_norm=99.353, loss=303.0871, lr=1.94e-04]
Epoch 17 TRAIN:  82%|████████▏ | 111/136 [00:24<00:05,  4.63it/s, epoch=17, grad_norm=167.581, loss=262.9242, lr=1.94e-04]
Epoch 17 TRAIN:  82%|████████▏ | 111/136 [00:24<00:05,  4.63it/s, epoch=17, grad_norm=116.653, loss=256.0638, lr=1.94e-04]
Epoch 17 TRAIN:  82%|████████▏ | 111/136 [00:24<00:05,  4.63it/s, epoch=17, grad_norm=175.473, loss=242.6331, lr=1.94e-04]
Epoch 17 TRAIN:  84%|████████▍ | 114/136 [00:24<00:04,  4.89it/s, epoch=17, grad_norm=175.473, loss=242.6331, lr=1.94e-04]
Epoch 17 TRAIN:  84%|████████▍ | 114/136 [00:24<00:04,  4.89it/s, epoch=17, grad_norm=134.092, loss=290.3523, lr=1.94e-04]
Epoch 17 TRAIN:  84%|████████▍ | 114/136 [00:25<00:04,  4.89it/s, epoch=17, grad_norm=133.586, loss=391.0973, lr=1.94e-04]
Epoch 17 TRAIN:  84%|████████▍ | 114/136 [00:25<00:04,  4.89it/s, epoch=17, grad_norm=105.269, loss=337.9163, lr=1.94e-04]
Epoch 17 TRAIN:  86%|████████▌ | 117/136 [00:25<00:03,  4.80it/s, epoch=17, grad_norm=105.269, loss=337.9163, lr=1.94e-04]
Epoch 17 TRAIN:  86%|████████▌ | 117/136 [00:25<00:03,  4.80it/s, epoch=17, grad_norm=194.634, loss=344.6240, lr=1.95e-04]
Epoch 17 TRAIN:  86%|████████▌ | 117/136 [00:25<00:03,  4.80it/s, epoch=17, grad_norm=130.849, loss=391.0807, lr=1.95e-04]
Epoch 17 TRAIN:  86%|████████▌ | 117/136 [00:25<00:03,  4.80it/s, epoch=17, grad_norm=63.444, loss=231.3032, lr=1.95e-04] 
Epoch 17 TRAIN:  88%|████████▊ | 120/136 [00:25<00:03,  4.87it/s, epoch=17, grad_norm=63.444, loss=231.3032, lr=1.95e-04]
Epoch 17 TRAIN:  88%|████████▊ | 120/136 [00:26<00:03,  4.87it/s, epoch=17, grad_norm=116.503, loss=312.8688, lr=1.95e-04]
Epoch 17 TRAIN:  88%|████████▊ | 120/136 [00:26<00:03,  4.87it/s, epoch=17, grad_norm=120.260, loss=270.0088, lr=1.95e-04]
Epoch 17 TRAIN:  88%|████████▊ | 120/136 [00:26<00:03,  4.87it/s, epoch=17, grad_norm=340.247, loss=329.2541, lr=1.95e-04]
Epoch 17 TRAIN:  88%|████████▊ | 120/136 [00:26<00:03,  4.87it/s, epoch=17, grad_norm=186.795, loss=326.2269, lr=1.95e-04]
Epoch 17 TRAIN:  91%|█████████ | 124/136 [00:26<00:02,  5.15it/s, epoch=17, grad_norm=186.795, loss=326.2269, lr=1.95e-04]
Epoch 17 TRAIN:  91%|█████████ | 124/136 [00:26<00:02,  5.15it/s, epoch=17, grad_norm=141.168, loss=405.8460, lr=1.95e-04]
Epoch 17 TRAIN:  91%|█████████ | 124/136 [00:27<00:02,  5.15it/s, epoch=17, grad_norm=129.362, loss=386.1859, lr=1.95e-04]
Epoch 17 TRAIN:  91%|█████████ | 124/136 [00:27<00:02,  5.15it/s, epoch=17, grad_norm=115.820, loss=348.0728, lr=1.95e-04]
Epoch 17 TRAIN:  93%|█████████▎| 127/136 [00:27<00:01,  5.08it/s, epoch=17, grad_norm=115.820, loss=348.0728, lr=1.95e-04]
Epoch 17 TRAIN:  93%|█████████▎| 127/136 [00:27<00:01,  5.08it/s, epoch=17, grad_norm=249.201, loss=273.1637, lr=1.95e-04]
Epoch 17 TRAIN:  93%|█████████▎| 127/136 [00:27<00:01,  5.08it/s, epoch=17, grad_norm=134.267, loss=315.1860, lr=1.95e-04]
Epoch 17 TRAIN:  93%|█████████▎| 127/136 [00:27<00:01,  5.08it/s, epoch=17, grad_norm=139.356, loss=376.2810, lr=1.96e-04]
Epoch 17 TRAIN:  96%|█████████▌| 130/136 [00:27<00:01,  4.92it/s, epoch=17, grad_norm=139.356, loss=376.2810, lr=1.96e-04]
Epoch 17 TRAIN:  96%|█████████▌| 130/136 [00:28<00:01,  4.92it/s, epoch=17, grad_norm=116.957, loss=401.5925, lr=1.96e-04]
Epoch 17 TRAIN:  96%|█████████▌| 130/136 [00:28<00:01,  4.92it/s, epoch=17, grad_norm=101.234, loss=321.0270, lr=1.96e-04]
Epoch 17 TRAIN:  96%|█████████▌| 130/136 [00:28<00:01,  4.92it/s, epoch=17, grad_norm=165.668, loss=353.6589, lr=1.96e-04]
Epoch 17 TRAIN:  98%|█████████▊| 133/136 [00:28<00:00,  4.86it/s, epoch=17, grad_norm=165.668, loss=353.6589, lr=1.96e-04]
Epoch 17 TRAIN:  98%|█████████▊| 133/136 [00:28<00:00,  4.86it/s, epoch=17, grad_norm=149.088, loss=428.8519, lr=1.96e-04]
Epoch 17 TRAIN:  98%|█████████▊| 133/136 [00:28<00:00,  4.86it/s, epoch=17, grad_norm=169.845, loss=323.4536, lr=1.96e-04]
Epoch 17 TRAIN:  99%|█████████▉| 135/136 [00:28<00:00,  4.67it/s, epoch=17, grad_norm=169.845, loss=323.4536, lr=1.96e-04]

CV:   0%|          | 0/6 [00:00<?, ?it/s]/home/rookie/Documents/batuka/projects/wenet/wenet/wenet/utils/train_utils.py:689: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  amp_autocast(enabled=dtype is not None,

CV:  17%|█▋        | 1/6 [00:00<00:03,  1.32it/s]
                                                 
2026-04-16 20:00:38,180 INFO Epoch 17 Step 2449 CV info lr 1.9592e-04 cv_loss 389.37384310635656 rank 0 acc 0.2488065312306086
2026-04-16 20:00:38,182 INFO [Rank 0] Checkpoint: save to checkpoint exp/conformer_mongolian/epoch_17.pt
2026-04-16 20:00:38,294 INFO Epoch 18 Step 2449 TRAIN info lr 1.9592e-04 rank 0
2026-04-16 20:00:38,312 INFO using accumulate grad, new batch size is 1 times larger than before

Epoch 18 TRAIN:   0%|          | 0/135 [00:00<?, ?it/s]
Epoch 18 TRAIN:   0%|          | 0/135 [00:04<?, ?it/s, epoch=18, grad_norm=435.474, loss=422.7616, lr=1.96e-04]
Epoch 18 TRAIN:   1%|          | 1/135 [00:04<09:01,  4.04s/it, epoch=18, grad_norm=435.474, loss=422.7616, lr=1.96e-04]
Epoch 18 TRAIN:   1%|          | 1/135 [00:04<09:01,  4.04s/it, epoch=18, grad_norm=397.012, loss=387.5165, lr=1.96e-04]
Epoch 18 TRAIN:   1%|          | 1/135 [00:04<09:01,  4.04s/it, epoch=18, grad_norm=313.731, loss=360.4973, lr=1.96e-04]
Epoch 18 TRAIN:   1%|          | 1/135 [00:04<09:01,  4.04s/it, epoch=18, grad_norm=408.643, loss=381.5883, lr=1.96e-04]
Epoch 18 TRAIN:   1%|          | 1/135 [00:04<09:01,  4.04s/it, epoch=18, grad_norm=236.429, loss=312.6707, lr=1.96e-04]
Epoch 18 TRAIN:   1%|          | 1/135 [00:04<09:01,  4.04s/it, epoch=18, grad_norm=340.920, loss=263.8135, lr=1.96e-04]
Epoch 18 TRAIN:   1%|          | 1/135 [00:04<09:01,  4.04s/it, epoch=18, grad_norm=255.365, loss=432.4845, lr=1.96e-04]
Epoch 18 TRAIN:   1%|          | 1/135 [00:04<09:01,  4.04s/it, epoch=18, grad_norm=159.147, loss=361.0210, lr=1.97e-04]
Epoch 18 TRAIN:   6%|▌         | 8/135 [00:04<00:54,  2.31it/s, epoch=18, grad_norm=159.147, loss=361.0210, lr=1.97e-04]
Epoch 18 TRAIN:   6%|▌         | 8/135 [00:04<00:54,  2.31it/s, epoch=18, grad_norm=107.321, loss=333.7542, lr=1.97e-04]
Epoch 18 TRAIN:   6%|▌         | 8/135 [00:04<00:54,  2.31it/s, epoch=18, grad_norm=114.032, loss=348.9174, lr=1.97e-04]
Epoch 18 TRAIN:   6%|▌         | 8/135 [00:05<00:54,  2.31it/s, epoch=18, grad_norm=164.836, loss=353.4538, lr=1.97e-04]
Epoch 18 TRAIN:   6%|▌         | 8/135 [00:05<00:54,  2.31it/s, epoch=18, grad_norm=160.200, loss=330.6258, lr=1.97e-04]
Epoch 18 TRAIN:   6%|▌         | 8/135 [00:05<00:54,  2.31it/s, epoch=18, grad_norm=107.446, loss=345.6668, lr=1.97e-04]
Epoch 18 TRAIN:  10%|▉         | 13/135 [00:05<00:36,  3.32it/s, epoch=18, grad_norm=107.446, loss=345.6668, lr=1.97e-04]
Epoch 18 TRAIN:  10%|▉         | 13/135 [00:05<00:36,  3.32it/s, epoch=18, grad_norm=118.022, loss=294.1860, lr=1.97e-04]
Epoch 18 TRAIN:  10%|▉         | 13/135 [00:05<00:36,  3.32it/s, epoch=18, grad_norm=75.140, loss=322.7478, lr=1.97e-04] 
Epoch 18 TRAIN:  10%|▉         | 13/135 [00:05<00:36,  3.32it/s, epoch=18, grad_norm=89.530, loss=339.2554, lr=1.97e-04]
Epoch 18 TRAIN:  10%|▉         | 13/135 [00:05<00:36,  3.32it/s, epoch=18, grad_norm=177.669, loss=326.4170, lr=1.97e-04]
Epoch 18 TRAIN:  13%|█▎        | 17/135 [00:05<00:29,  3.99it/s, epoch=18, grad_norm=177.669, loss=326.4170, lr=1.97e-04]
Epoch 18 TRAIN:  13%|█▎        | 17/135 [00:06<00:29,  3.99it/s, epoch=18, grad_norm=84.019, loss=319.0206, lr=1.97e-04] 
Epoch 18 TRAIN:  13%|█▎        | 17/135 [00:06<00:29,  3.99it/s, epoch=18, grad_norm=76.974, loss=291.6883, lr=1.97e-04]
Epoch 18 TRAIN:  13%|█▎        | 17/135 [00:06<00:29,  3.99it/s, epoch=18, grad_norm=107.767, loss=350.1448, lr=1.98e-04]
Epoch 18 TRAIN:  13%|█▎        | 17/135 [00:06<00:29,  3.99it/s, epoch=18, grad_norm=124.302, loss=338.4240, lr=1.98e-04]
Epoch 18 TRAIN:  16%|█▌        | 21/135 [00:06<00:25,  4.48it/s, epoch=18, grad_norm=124.302, loss=338.4240, lr=1.98e-04]
Epoch 18 TRAIN:  16%|█▌        | 21/135 [00:06<00:25,  4.48it/s, epoch=18, grad_norm=100.005, loss=346.3808, lr=1.98e-04]
Epoch 18 TRAIN:  16%|█▌        | 21/135 [00:06<00:25,  4.48it/s, epoch=18, grad_norm=101.631, loss=299.6411, lr=1.98e-04]
Epoch 18 TRAIN:  16%|█▌        | 21/135 [00:07<00:25,  4.48it/s, epoch=18, grad_norm=83.297, loss=358.4566, lr=1.98e-04] 
Epoch 18 TRAIN:  16%|█▌        | 21/135 [00:07<00:25,  4.48it/s, epoch=18, grad_norm=78.963, loss=383.0570, lr=1.98e-04]
Epoch 18 TRAIN:  19%|█▊        | 25/135 [00:07<00:22,  4.89it/s, epoch=18, grad_norm=78.963, loss=383.0570, lr=1.98e-04]
Epoch 18 TRAIN:  19%|█▊        | 25/135 [00:07<00:22,  4.89it/s, epoch=18, grad_norm=156.406, loss=306.3018, lr=1.98e-04]
Epoch 18 TRAIN:  19%|█▊        | 25/135 [00:07<00:22,  4.89it/s, epoch=18, grad_norm=182.405, loss=290.5965, lr=1.98e-04]
Epoch 18 TRAIN:  19%|█▊        | 25/135 [00:07<00:22,  4.89it/s, epoch=18, grad_norm=91.452, loss=317.0251, lr=1.98e-04] 
Epoch 18 TRAIN:  19%|█▊        | 25/135 [00:08<00:22,  4.89it/s, epoch=18, grad_norm=117.332, loss=311.4605, lr=1.98e-04]
Epoch 18 TRAIN:  21%|██▏       | 29/135 [00:08<00:21,  5.01it/s, epoch=18, grad_norm=117.332, loss=311.4605, lr=1.98e-04]
Epoch 18 TRAIN:  21%|██▏       | 29/135 [00:08<00:21,  5.01it/s, epoch=18, grad_norm=75.196, loss=302.6822, lr=1.98e-04] 
Epoch 18 TRAIN:  21%|██▏       | 29/135 [00:08<00:21,  5.01it/s, epoch=18, grad_norm=115.629, loss=289.6819, lr=1.98e-04]
Epoch 18 TRAIN:  21%|██▏       | 29/135 [00:08<00:21,  5.01it/s, epoch=18, grad_norm=105.318, loss=299.6356, lr=1.98e-04]
Epoch 18 TRAIN:  21%|██▏       | 29/135 [00:08<00:21,  5.01it/s, epoch=18, grad_norm=156.931, loss=324.0904, lr=1.99e-04]
Epoch 18 TRAIN:  24%|██▍       | 33/135 [00:08<00:19,  5.20it/s, epoch=18, grad_norm=156.931, loss=324.0904, lr=1.99e-04]
Epoch 18 TRAIN:  24%|██▍       | 33/135 [00:09<00:19,  5.20it/s, epoch=18, grad_norm=134.862, loss=352.7117, lr=1.99e-04]
Epoch 18 TRAIN:  24%|██▍       | 33/135 [00:09<00:19,  5.20it/s, epoch=18, grad_norm=66.002, loss=301.9896, lr=1.99e-04] 
Epoch 18 TRAIN:  24%|██▍       | 33/135 [00:09<00:19,  5.20it/s, epoch=18, grad_norm=114.707, loss=333.8516, lr=1.99e-04]
Epoch 18 TRAIN:  27%|██▋       | 36/135 [00:09<00:19,  5.04it/s, epoch=18, grad_norm=114.707, loss=333.8516, lr=1.99e-04]
Epoch 18 TRAIN:  27%|██▋       | 36/135 [00:09<00:19,  5.04it/s, epoch=18, grad_norm=146.842, loss=375.5181, lr=1.99e-04]
Epoch 18 TRAIN:  27%|██▋       | 36/135 [00:09<00:19,  5.04it/s, epoch=18, grad_norm=119.619, loss=299.5685, lr=1.99e-04]
Epoch 18 TRAIN:  27%|██▋       | 36/135 [00:09<00:19,  5.04it/s, epoch=18, grad_norm=203.778, loss=379.6766, lr=1.99e-04]
Epoch 18 TRAIN:  29%|██▉       | 39/135 [00:09<00:18,  5.14it/s, epoch=18, grad_norm=203.778, loss=379.6766, lr=1.99e-04]
Epoch 18 TRAIN:  29%|██▉       | 39/135 [00:10<00:18,  5.14it/s, epoch=18, grad_norm=103.135, loss=326.6212, lr=1.99e-04]
Epoch 18 TRAIN:  29%|██▉       | 39/135 [00:10<00:18,  5.14it/s, epoch=18, grad_norm=158.116, loss=315.5139, lr=1.99e-04]
Epoch 18 TRAIN:  29%|██▉       | 39/135 [00:10<00:18,  5.14it/s, epoch=18, grad_norm=172.772, loss=306.2453, lr=1.99e-04]
Epoch 18 TRAIN:  31%|███       | 42/135 [00:10<00:17,  5.20it/s, epoch=18, grad_norm=172.772, loss=306.2453, lr=1.99e-04]
Epoch 18 TRAIN:  31%|███       | 42/135 [00:10<00:17,  5.20it/s, epoch=18, grad_norm=233.463, loss=301.9053, lr=1.99e-04]
Epoch 18 TRAIN:  31%|███       | 42/135 [00:11<00:17,  5.20it/s, epoch=18, grad_norm=110.411, loss=300.0193, lr=1.99e-04]
Epoch 18 TRAIN:  31%|███       | 42/135 [00:11<00:17,  5.20it/s, epoch=18, grad_norm=126.254, loss=310.5612, lr=2.00e-04]
Epoch 18 TRAIN:  33%|███▎      | 45/135 [00:11<00:17,  5.05it/s, epoch=18, grad_norm=126.254, loss=310.5612, lr=2.00e-04]
Epoch 18 TRAIN:  33%|███▎      | 45/135 [00:11<00:17,  5.05it/s, epoch=18, grad_norm=103.756, loss=348.9159, lr=2.00e-04]
Epoch 18 TRAIN:  33%|███▎      | 45/135 [00:11<00:17,  5.05it/s, epoch=18, grad_norm=156.120, loss=373.8050, lr=2.00e-04]
Epoch 18 TRAIN:  33%|███▎      | 45/135 [00:11<00:17,  5.05it/s, epoch=18, grad_norm=112.614, loss=324.4409, lr=2.00e-04]
Epoch 18 TRAIN:  36%|███▌      | 48/135 [00:11<00:17,  5.12it/s, epoch=18, grad_norm=112.614, loss=324.4409, lr=2.00e-04]
Epoch 18 TRAIN:  36%|███▌      | 48/135 [00:11<00:17,  5.12it/s, epoch=18, grad_norm=106.994, loss=285.8954, lr=2.00e-04]
Epoch 18 TRAIN:  36%|███▌      | 48/135 [00:12<00:17,  5.12it/s, epoch=18, grad_norm=83.307, loss=330.6546, lr=2.00e-04] 
Epoch 18 TRAIN:  36%|███▌      | 48/135 [00:12<00:17,  5.12it/s, epoch=18, grad_norm=68.263, loss=325.5580, lr=2.00e-04]
Epoch 18 TRAIN:  38%|███▊      | 51/135 [00:12<00:16,  5.08it/s, epoch=18, grad_norm=68.263, loss=325.5580, lr=2.00e-04]
Epoch 18 TRAIN:  38%|███▊      | 51/135 [00:12<00:16,  5.08it/s, epoch=18, grad_norm=85.487, loss=317.1931, lr=2.00e-04]
Epoch 18 TRAIN:  38%|███▊      | 51/135 [00:12<00:16,  5.08it/s, epoch=18, grad_norm=307.611, loss=374.2205, lr=2.00e-04]
Epoch 18 TRAIN:  38%|███▊      | 51/135 [00:12<00:16,  5.08it/s, epoch=18, grad_norm=86.813, loss=297.1664, lr=2.00e-04] 
Epoch 18 TRAIN:  40%|████      | 54/135 [00:12<00:15,  5.16it/s, epoch=18, grad_norm=86.813, loss=297.1664, lr=2.00e-04]
Epoch 18 TRAIN:  40%|████      | 54/135 [00:13<00:15,  5.16it/s, epoch=18, grad_norm=117.665, loss=292.3908, lr=2.00e-04]
Epoch 18 TRAIN:  40%|████      | 54/135 [00:13<00:15,  5.16it/s, epoch=18, grad_norm=106.206, loss=298.5611, lr=2.00e-04]
Epoch 18 TRAIN:  40%|████      | 54/135 [00:13<00:15,  5.16it/s, epoch=18, grad_norm=97.045, loss=342.1513, lr=2.00e-04] 
Epoch 18 TRAIN:  42%|████▏     | 57/135 [00:13<00:15,  5.11it/s, epoch=18, grad_norm=97.045, loss=342.1513, lr=2.00e-04]
Epoch 18 TRAIN:  42%|████▏     | 57/135 [00:13<00:15,  5.11it/s, epoch=18, grad_norm=129.253, loss=338.8415, lr=2.01e-04]
Epoch 18 TRAIN:  42%|████▏     | 57/135 [00:13<00:15,  5.11it/s, epoch=18, grad_norm=136.520, loss=307.7016, lr=2.01e-04]
Epoch 18 TRAIN:  42%|████▏     | 57/135 [00:14<00:15,  5.11it/s, epoch=18, grad_norm=97.670, loss=330.1279, lr=2.01e-04] 
Epoch 18 TRAIN:  44%|████▍     | 60/135 [00:14<00:14,  5.06it/s, epoch=18, grad_norm=97.670, loss=330.1279, lr=2.01e-04]
Epoch 18 TRAIN:  44%|████▍     | 60/135 [00:14<00:14,  5.06it/s, epoch=18, grad_norm=145.422, loss=363.0392, lr=2.01e-04]
Epoch 18 TRAIN:  44%|████▍     | 60/135 [00:14<00:14,  5.06it/s, epoch=18, grad_norm=69.467, loss=300.2029, lr=2.01e-04] 
Epoch 18 TRAIN:  44%|████▍     | 60/135 [00:14<00:14,  5.06it/s, epoch=18, grad_norm=136.210, loss=299.9860, lr=2.01e-04]
Epoch 18 TRAIN:  47%|████▋     | 63/135 [00:14<00:14,  4.97it/s, epoch=18, grad_norm=136.210, loss=299.9860, lr=2.01e-04]
Epoch 18 TRAIN:  47%|████▋     | 63/135 [00:14<00:14,  4.97it/s, epoch=18, grad_norm=125.870, loss=284.7468, lr=2.01e-04]
Epoch 18 TRAIN:  47%|████▋     | 63/135 [00:15<00:14,  4.97it/s, epoch=18, grad_norm=123.132, loss=314.2130, lr=2.01e-04]
Epoch 18 TRAIN:  47%|████▋     | 63/135 [00:15<00:14,  4.97it/s, epoch=18, grad_norm=116.162, loss=346.6907, lr=2.01e-04]
Epoch 18 TRAIN:  49%|████▉     | 66/135 [00:15<00:13,  5.06it/s, epoch=18, grad_norm=116.162, loss=346.6907, lr=2.01e-04]
Epoch 18 TRAIN:  49%|████▉     | 66/135 [00:15<00:13,  5.06it/s, epoch=18, grad_norm=144.905, loss=302.7662, lr=2.01e-04]
Epoch 18 TRAIN:  49%|████▉     | 66/135 [00:15<00:13,  5.06it/s, epoch=18, grad_norm=104.467, loss=339.3528, lr=2.01e-04]
Epoch 18 TRAIN:  49%|████▉     | 66/135 [00:15<00:13,  5.06it/s, epoch=18, grad_norm=103.949, loss=362.6526, lr=2.01e-04]
Epoch 18 TRAIN:  51%|█████     | 69/135 [00:15<00:12,  5.13it/s, epoch=18, grad_norm=103.949, loss=362.6526, lr=2.01e-04]
Epoch 18 TRAIN:  51%|█████     | 69/135 [00:16<00:12,  5.13it/s, epoch=18, grad_norm=72.468, loss=312.7391, lr=2.02e-04] 
Epoch 18 TRAIN:  51%|█████     | 69/135 [00:16<00:12,  5.13it/s, epoch=18, grad_norm=146.730, loss=342.3348, lr=2.02e-04]
Epoch 18 TRAIN:  51%|█████     | 69/135 [00:16<00:12,  5.13it/s, epoch=18, grad_norm=230.495, loss=301.4963, lr=2.02e-04]
Epoch 18 TRAIN:  53%|█████▎    | 72/135 [00:16<00:11,  5.34it/s, epoch=18, grad_norm=230.495, loss=301.4963, lr=2.02e-04]
Epoch 18 TRAIN:  53%|█████▎    | 72/135 [00:16<00:11,  5.34it/s, epoch=18, grad_norm=100.424, loss=371.1435, lr=2.02e-04]
Epoch 18 TRAIN:  53%|█████▎    | 72/135 [00:16<00:11,  5.34it/s, epoch=18, grad_norm=201.095, loss=312.7935, lr=2.02e-04]
Epoch 18 TRAIN:  53%|█████▎    | 72/135 [00:16<00:11,  5.34it/s, epoch=18, grad_norm=83.086, loss=399.2537, lr=2.02e-04] 
Epoch 18 TRAIN:  56%|█████▌    | 75/135 [00:16<00:11,  5.19it/s, epoch=18, grad_norm=83.086, loss=399.2537, lr=2.02e-04]
Epoch 18 TRAIN:  56%|█████▌    | 75/135 [00:17<00:11,  5.19it/s, epoch=18, grad_norm=105.497, loss=348.7330, lr=2.02e-04]
Epoch 18 TRAIN:  56%|█████▌    | 75/135 [00:17<00:11,  5.19it/s, epoch=18, grad_norm=97.909, loss=313.6306, lr=2.02e-04] 
Epoch 18 TRAIN:  56%|█████▌    | 75/135 [00:17<00:11,  5.19it/s, epoch=18, grad_norm=131.982, loss=268.8933, lr=2.02e-04]
Epoch 18 TRAIN:  58%|█████▊    | 78/135 [00:17<00:11,  5.05it/s, epoch=18, grad_norm=131.982, loss=268.8933, lr=2.02e-04]
Epoch 18 TRAIN:  58%|█████▊    | 78/135 [00:17<00:11,  5.05it/s, epoch=18, grad_norm=158.991, loss=372.2301, lr=2.02e-04]
Epoch 18 TRAIN:  58%|█████▊    | 78/135 [00:18<00:11,  5.05it/s, epoch=18, grad_norm=63.965, loss=304.3368, lr=2.02e-04] 
Epoch 18 TRAIN:  58%|█████▊    | 78/135 [00:18<00:11,  5.05it/s, epoch=18, grad_norm=224.245, loss=239.3300, lr=2.02e-04]
Epoch 18 TRAIN:  60%|██████    | 81/135 [00:18<00:10,  5.04it/s, epoch=18, grad_norm=224.245, loss=239.3300, lr=2.02e-04]
Epoch 18 TRAIN:  60%|██████    | 81/135 [00:18<00:10,  5.04it/s, epoch=18, grad_norm=229.058, loss=235.3670, lr=2.02e-04]
Epoch 18 TRAIN:  60%|██████    | 81/135 [00:18<00:10,  5.04it/s, epoch=18, grad_norm=78.792, loss=334.2516, lr=2.03e-04] 
Epoch 18 TRAIN:  60%|██████    | 81/135 [00:18<00:10,  5.04it/s, epoch=18, grad_norm=224.642, loss=377.1671, lr=2.03e-04]
Epoch 18 TRAIN:  62%|██████▏   | 84/135 [00:18<00:09,  5.12it/s, epoch=18, grad_norm=224.642, loss=377.1671, lr=2.03e-04]
Epoch 18 TRAIN:  62%|██████▏   | 84/135 [00:18<00:09,  5.12it/s, epoch=18, grad_norm=100.872, loss=340.3300, lr=2.03e-04]
Epoch 18 TRAIN:  62%|██████▏   | 84/135 [00:19<00:09,  5.12it/s, epoch=18, grad_norm=130.416, loss=268.7673, lr=2.03e-04]
Epoch 18 TRAIN:  62%|██████▏   | 84/135 [00:19<00:09,  5.12it/s, epoch=18, grad_norm=92.562, loss=248.5481, lr=2.03e-04] 
Epoch 18 TRAIN:  64%|██████▍   | 87/135 [00:19<00:09,  4.92it/s, epoch=18, grad_norm=92.562, loss=248.5481, lr=2.03e-04]
Epoch 18 TRAIN:  64%|██████▍   | 87/135 [00:19<00:09,  4.92it/s, epoch=18, grad_norm=184.945, loss=369.9826, lr=2.03e-04]
Epoch 18 TRAIN:  64%|██████▍   | 87/135 [00:19<00:09,  4.92it/s, epoch=18, grad_norm=175.896, loss=259.9204, lr=2.03e-04]
Epoch 18 TRAIN:  64%|██████▍   | 87/135 [00:20<00:09,  4.92it/s, epoch=18, grad_norm=144.426, loss=289.8317, lr=2.03e-04]
Epoch 18 TRAIN:  67%|██████▋   | 90/135 [00:20<00:09,  4.93it/s, epoch=18, grad_norm=144.426, loss=289.8317, lr=2.03e-04]
Epoch 18 TRAIN:  67%|██████▋   | 90/135 [00:20<00:09,  4.93it/s, epoch=18, grad_norm=206.067, loss=246.6044, lr=2.03e-04]
Epoch 18 TRAIN:  67%|██████▋   | 90/135 [00:20<00:09,  4.93it/s, epoch=18, grad_norm=126.753, loss=298.6369, lr=2.03e-04]
Epoch 18 TRAIN:  67%|██████▋   | 90/135 [00:20<00:09,  4.93it/s, epoch=18, grad_norm=135.599, loss=274.5502, lr=2.03e-04]
Epoch 18 TRAIN:  69%|██████▉   | 93/135 [00:20<00:08,  4.99it/s, epoch=18, grad_norm=135.599, loss=274.5502, lr=2.03e-04]
Epoch 18 TRAIN:  69%|██████▉   | 93/135 [00:20<00:08,  4.99it/s, epoch=18, grad_norm=123.284, loss=279.3174, lr=2.03e-04]
Epoch 18 TRAIN:  69%|██████▉   | 93/135 [00:21<00:08,  4.99it/s, epoch=18, grad_norm=204.382, loss=338.5905, lr=2.04e-04]
Epoch 18 TRAIN:  69%|██████▉   | 93/135 [00:21<00:08,  4.99it/s, epoch=18, grad_norm=90.005, loss=256.3129, lr=2.04e-04] 
Epoch 18 TRAIN:  71%|███████   | 96/135 [00:21<00:08,  4.82it/s, epoch=18, grad_norm=90.005, loss=256.3129, lr=2.04e-04]
Epoch 18 TRAIN:  71%|███████   | 96/135 [00:21<00:08,  4.82it/s, epoch=18, grad_norm=157.733, loss=260.3708, lr=2.04e-04]
Epoch 18 TRAIN:  71%|███████   | 96/135 [00:21<00:08,  4.82it/s, epoch=18, grad_norm=99.901, loss=304.6825, lr=2.04e-04] 
Epoch 18 TRAIN:  71%|███████   | 96/135 [00:21<00:08,  4.82it/s, epoch=18, grad_norm=89.016, loss=332.0496, lr=2.04e-04]
Epoch 18 TRAIN:  73%|███████▎  | 99/135 [00:21<00:07,  4.92it/s, epoch=18, grad_norm=89.016, loss=332.0496, lr=2.04e-04]
Epoch 18 TRAIN:  73%|███████▎  | 99/135 [00:22<00:07,  4.92it/s, epoch=18, grad_norm=171.006, loss=323.9738, lr=2.04e-04]2026-04-16 20:01:00,378 DEBUG TRAIN | steps/sec 4.478| Batch 18/100 loss 323.973755 loss_att 289.108429 loss_ctc 405.326172 th_accuracy 0.221034 lr 2.0392e-04 grad_norm 171.005661 rank 0

Epoch 18 TRAIN:  73%|███████▎  | 99/135 [00:22<00:07,  4.92it/s, epoch=18, grad_norm=177.418, loss=269.8110, lr=2.04e-04]
Epoch 18 TRAIN:  73%|███████▎  | 99/135 [00:22<00:07,  4.92it/s, epoch=18, grad_norm=154.930, loss=313.3055, lr=2.04e-04]
Epoch 18 TRAIN:  76%|███████▌  | 102/135 [00:22<00:06,  4.88it/s, epoch=18, grad_norm=154.930, loss=313.3055, lr=2.04e-04]
Epoch 18 TRAIN:  76%|███████▌  | 102/135 [00:22<00:06,  4.88it/s, epoch=18, grad_norm=165.913, loss=298.7550, lr=2.04e-04]
Epoch 18 TRAIN:  76%|███████▌  | 102/135 [00:22<00:06,  4.88it/s, epoch=18, grad_norm=157.025, loss=359.5751, lr=2.04e-04]
Epoch 18 TRAIN:  76%|███████▌  | 102/135 [00:23<00:06,  4.88it/s, epoch=18, grad_norm=89.249, loss=295.0833, lr=2.04e-04] 
Epoch 18 TRAIN:  78%|███████▊  | 105/135 [00:23<00:06,  4.86it/s, epoch=18, grad_norm=89.249, loss=295.0833, lr=2.04e-04]
Epoch 18 TRAIN:  78%|███████▊  | 105/135 [00:23<00:06,  4.86it/s, epoch=18, grad_norm=149.544, loss=321.6592, lr=2.04e-04]
Epoch 18 TRAIN:  78%|███████▊  | 105/135 [00:23<00:06,  4.86it/s, epoch=18, grad_norm=97.049, loss=369.4804, lr=2.04e-04] 
Epoch 18 TRAIN:  78%|███████▊  | 105/135 [00:23<00:06,  4.86it/s, epoch=18, grad_norm=127.805, loss=316.4581, lr=2.05e-04]
Epoch 18 TRAIN:  80%|████████  | 108/135 [00:23<00:05,  5.11it/s, epoch=18, grad_norm=127.805, loss=316.4581, lr=2.05e-04]
Epoch 18 TRAIN:  80%|████████  | 108/135 [00:23<00:05,  5.11it/s, epoch=18, grad_norm=120.904, loss=356.7636, lr=2.05e-04]
Epoch 18 TRAIN:  80%|████████  | 108/135 [00:24<00:05,  5.11it/s, epoch=18, grad_norm=223.460, loss=240.3879, lr=2.05e-04]
Epoch 18 TRAIN:  80%|████████  | 108/135 [00:24<00:05,  5.11it/s, epoch=18, grad_norm=101.022, loss=278.7911, lr=2.05e-04]
Epoch 18 TRAIN:  82%|████████▏ | 111/135 [00:24<00:04,  5.03it/s, epoch=18, grad_norm=101.022, loss=278.7911, lr=2.05e-04]
Epoch 18 TRAIN:  82%|████████▏ | 111/135 [00:24<00:04,  5.03it/s, epoch=18, grad_norm=94.114, loss=268.4229, lr=2.05e-04] 
Epoch 18 TRAIN:  82%|████████▏ | 111/135 [00:24<00:04,  5.03it/s, epoch=18, grad_norm=83.632, loss=310.8023, lr=2.05e-04]
Epoch 18 TRAIN:  82%|████████▏ | 111/135 [00:24<00:04,  5.03it/s, epoch=18, grad_norm=225.905, loss=423.5166, lr=2.05e-04]
Epoch 18 TRAIN:  84%|████████▍ | 114/135 [00:24<00:04,  5.12it/s, epoch=18, grad_norm=225.905, loss=423.5166, lr=2.05e-04]
Epoch 18 TRAIN:  84%|████████▍ | 114/135 [00:25<00:04,  5.12it/s, epoch=18, grad_norm=108.171, loss=375.7008, lr=2.05e-04]
Epoch 18 TRAIN:  84%|████████▍ | 114/135 [00:25<00:04,  5.12it/s, epoch=18, grad_norm=132.145, loss=252.1187, lr=2.05e-04]
Epoch 18 TRAIN:  84%|████████▍ | 114/135 [00:25<00:04,  5.12it/s, epoch=18, grad_norm=169.748, loss=387.6577, lr=2.05e-04]
Epoch 18 TRAIN:  87%|████████▋ | 117/135 [00:25<00:03,  5.21it/s, epoch=18, grad_norm=169.748, loss=387.6577, lr=2.05e-04]
Epoch 18 TRAIN:  87%|████████▋ | 117/135 [00:25<00:03,  5.21it/s, epoch=18, grad_norm=157.154, loss=324.3371, lr=2.05e-04]
Epoch 18 TRAIN:  87%|████████▋ | 117/135 [00:25<00:03,  5.21it/s, epoch=18, grad_norm=85.153, loss=312.8223, lr=2.05e-04] 
Epoch 18 TRAIN:  87%|████████▋ | 117/135 [00:26<00:03,  5.21it/s, epoch=18, grad_norm=382.768, loss=333.7905, lr=2.06e-04]
Epoch 18 TRAIN:  89%|████████▉ | 120/135 [00:26<00:02,  5.08it/s, epoch=18, grad_norm=382.768, loss=333.7905, lr=2.06e-04]
Epoch 18 TRAIN:  89%|████████▉ | 120/135 [00:26<00:02,  5.08it/s, epoch=18, grad_norm=99.174, loss=354.8151, lr=2.06e-04] 
Epoch 18 TRAIN:  89%|████████▉ | 120/135 [00:26<00:02,  5.08it/s, epoch=18, grad_norm=229.210, loss=339.4814, lr=2.06e-04]
Epoch 18 TRAIN:  89%|████████▉ | 120/135 [00:26<00:02,  5.08it/s, epoch=18, grad_norm=187.731, loss=336.0976, lr=2.06e-04]
Epoch 18 TRAIN:  91%|█████████ | 123/135 [00:26<00:02,  5.01it/s, epoch=18, grad_norm=187.731, loss=336.0976, lr=2.06e-04]
Epoch 18 TRAIN:  91%|█████████ | 123/135 [00:26<00:02,  5.01it/s, epoch=18, grad_norm=156.305, loss=384.5032, lr=2.06e-04]
Epoch 18 TRAIN:  91%|█████████ | 123/135 [00:27<00:02,  5.01it/s, epoch=18, grad_norm=189.105, loss=283.7441, lr=2.06e-04]
Epoch 18 TRAIN:  91%|█████████ | 123/135 [00:27<00:02,  5.01it/s, epoch=18, grad_norm=95.353, loss=387.4337, lr=2.06e-04] 
Epoch 18 TRAIN:  93%|█████████▎| 126/135 [00:27<00:01,  4.91it/s, epoch=18, grad_norm=95.353, loss=387.4337, lr=2.06e-04]
Epoch 18 TRAIN:  93%|█████████▎| 126/135 [00:27<00:01,  4.91it/s, epoch=18, grad_norm=205.609, loss=351.7407, lr=2.06e-04]
Epoch 18 TRAIN:  93%|█████████▎| 126/135 [00:27<00:01,  4.91it/s, epoch=18, grad_norm=166.391, loss=393.7327, lr=2.06e-04]
Epoch 18 TRAIN:  93%|█████████▎| 126/135 [00:27<00:01,  4.91it/s, epoch=18, grad_norm=122.167, loss=279.6016, lr=2.06e-04]
Epoch 18 TRAIN:  96%|█████████▌| 129/135 [00:27<00:01,  5.15it/s, epoch=18, grad_norm=122.167, loss=279.6016, lr=2.06e-04]
Epoch 18 TRAIN:  96%|█████████▌| 129/135 [00:27<00:01,  5.15it/s, epoch=18, grad_norm=144.458, loss=312.1324, lr=2.06e-04]
Epoch 18 TRAIN:  96%|█████████▌| 129/135 [00:28<00:01,  5.15it/s, epoch=18, grad_norm=217.180, loss=406.7635, lr=2.06e-04]
Epoch 18 TRAIN:  96%|█████████▌| 129/135 [00:28<00:01,  5.15it/s, epoch=18, grad_norm=205.352, loss=375.8085, lr=2.06e-04]
Epoch 18 TRAIN:  96%|█████████▌| 129/135 [00:28<00:01,  5.15it/s, epoch=18, grad_norm=146.628, loss=385.8975, lr=2.07e-04]
Epoch 18 TRAIN:  99%|█████████▊| 133/135 [00:28<00:00,  5.73it/s, epoch=18, grad_norm=146.628, loss=385.8975, lr=2.07e-04]
Epoch 18 TRAIN:  99%|█████████▊| 133/135 [00:28<00:00,  5.73it/s, epoch=18, grad_norm=102.221, loss=302.6999, lr=2.07e-04]
Epoch 18 TRAIN:  99%|█████████▊| 133/135 [00:28<00:00,  5.73it/s, epoch=18, grad_norm=79.979, loss=311.1483, lr=2.07e-04] 
Epoch 18 TRAIN:  99%|█████████▊| 133/135 [00:28<00:00,  5.73it/s, epoch=18, grad_norm=142.908, loss=385.8222, lr=2.07e-04]
Epoch 18 TRAIN:  99%|█████████▊| 133/135 [00:29<00:00,  5.73it/s, epoch=18, grad_norm=101.840, loss=302.4320, lr=2.07e-04]
Epoch 18 TRAIN: 137it [00:29,  5.63it/s, epoch=18, grad_norm=101.840, loss=302.4320, lr=2.07e-04]                         
Epoch 18 TRAIN: 137it [00:29,  4.71it/s, epoch=18, grad_norm=101.840, loss=302.4320, lr=2.07e-04]

CV:   0%|          | 0/6 [00:00<?, ?it/s]/home/rookie/Documents/batuka/projects/wenet/wenet/wenet/utils/train_utils.py:689: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  amp_autocast(enabled=dtype is not None,

CV:  17%|█▋        | 1/6 [00:00<00:03,  1.33it/s]
                                                 
2026-04-16 20:01:08,245 INFO Epoch 18 Step 2586 CV info lr 2.0688e-04 cv_loss 392.1363026012074 rank 0 acc 0.24636402477820715
2026-04-16 20:01:08,246 INFO [Rank 0] Checkpoint: save to checkpoint exp/conformer_mongolian/epoch_18.pt
2026-04-16 20:01:08,358 INFO Epoch 19 Step 2586 TRAIN info lr 2.0688e-04 rank 0
2026-04-16 20:01:08,380 INFO using accumulate grad, new batch size is 1 times larger than before

Epoch 19 TRAIN:   0%|          | 0/137 [00:00<?, ?it/s]
Epoch 19 TRAIN:   0%|          | 0/137 [00:04<?, ?it/s, epoch=19, grad_norm=408.762, loss=371.1499, lr=2.07e-04]
Epoch 19 TRAIN:   1%|          | 1/137 [00:04<09:04,  4.00s/it, epoch=19, grad_norm=408.762, loss=371.1499, lr=2.07e-04]
Epoch 19 TRAIN:   1%|          | 1/137 [00:04<09:04,  4.00s/it, epoch=19, grad_norm=187.648, loss=241.6574, lr=2.07e-04]
Epoch 19 TRAIN:   1%|          | 1/137 [00:04<09:04,  4.00s/it, epoch=19, grad_norm=484.399, loss=424.2084, lr=2.07e-04]
Epoch 19 TRAIN:   1%|          | 1/137 [00:04<09:04,  4.00s/it, epoch=19, grad_norm=243.844, loss=291.1060, lr=2.07e-04]
Epoch 19 TRAIN:   3%|▎         | 4/137 [00:04<01:58,  1.12it/s, epoch=19, grad_norm=243.844, loss=291.1060, lr=2.07e-04]
Epoch 19 TRAIN:   3%|▎         | 4/137 [00:04<01:58,  1.12it/s, epoch=19, grad_norm=571.378, loss=454.7202, lr=2.07e-04]
Epoch 19 TRAIN:   3%|▎         | 4/137 [00:04<01:58,  1.12it/s, epoch=19, grad_norm=237.852, loss=340.3383, lr=2.07e-04]
Epoch 19 TRAIN:   3%|▎         | 4/137 [00:04<01:58,  1.12it/s, epoch=19, grad_norm=137.803, loss=312.5723, lr=2.07e-04]
Epoch 19 TRAIN:   3%|▎         | 4/137 [00:04<01:58,  1.12it/s, epoch=19, grad_norm=161.682, loss=343.7071, lr=2.08e-04]
Epoch 19 TRAIN:   3%|▎         | 4/137 [00:04<01:58,  1.12it/s, epoch=19, grad_norm=118.954, loss=334.1494, lr=2.08e-04]
Epoch 19 TRAIN:   3%|▎         | 4/137 [00:05<01:58,  1.12it/s, epoch=19, grad_norm=79.741, loss=313.0109, lr=2.08e-04] 
Epoch 19 TRAIN:   7%|▋         | 10/137 [00:05<00:42,  2.97it/s, epoch=19, grad_norm=79.741, loss=313.0109, lr=2.08e-04]
Epoch 19 TRAIN:   7%|▋         | 10/137 [00:05<00:42,  2.97it/s, epoch=19, grad_norm=98.083, loss=349.5859, lr=2.08e-04]
Epoch 19 TRAIN:   7%|▋         | 10/137 [00:05<00:42,  2.97it/s, epoch=19, grad_norm=234.641, loss=301.8404, lr=2.08e-04]
Epoch 19 TRAIN:   7%|▋         | 10/137 [00:05<00:42,  2.97it/s, epoch=19, grad_norm=116.381, loss=370.0391, lr=2.08e-04]
Epoch 19 TRAIN:   7%|▋         | 10/137 [00:05<00:42,  2.97it/s, epoch=19, grad_norm=225.561, loss=314.1841, lr=2.08e-04]
Epoch 19 TRAIN:  10%|█         | 14/137 [00:05<00:32,  3.75it/s, epoch=19, grad_norm=225.561, loss=314.1841, lr=2.08e-04]
Epoch 19 TRAIN:  10%|█         | 14/137 [00:05<00:32,  3.75it/s, epoch=19, grad_norm=170.835, loss=299.3358, lr=2.08e-04]
Epoch 19 TRAIN:  10%|█         | 14/137 [00:06<00:32,  3.75it/s, epoch=19, grad_norm=105.553, loss=378.9254, lr=2.08e-04]
Epoch 19 TRAIN:  10%|█         | 14/137 [00:06<00:32,  3.75it/s, epoch=19, grad_norm=118.907, loss=293.7807, lr=2.08e-04]
Epoch 19 TRAIN:  10%|█         | 14/137 [00:06<00:32,  3.75it/s, epoch=19, grad_norm=72.439, loss=328.7924, lr=2.08e-04] 
Epoch 19 TRAIN:  13%|█▎        | 18/137 [00:06<00:27,  4.35it/s, epoch=19, grad_norm=72.439, loss=328.7924, lr=2.08e-04]
Epoch 19 TRAIN:  13%|█▎        | 18/137 [00:06<00:27,  4.35it/s, epoch=19, grad_norm=110.809, loss=339.3036, lr=2.08e-04]
Epoch 19 TRAIN:  13%|█▎        | 18/137 [00:06<00:27,  4.35it/s, epoch=19, grad_norm=94.867, loss=316.4730, lr=2.08e-04] 
Epoch 19 TRAIN:  13%|█▎        | 18/137 [00:06<00:27,  4.35it/s, epoch=19, grad_norm=109.897, loss=350.0479, lr=2.09e-04]
Epoch 19 TRAIN:  13%|█▎        | 18/137 [00:07<00:27,  4.35it/s, epoch=19, grad_norm=91.550, loss=339.5789, lr=2.09e-04] 
Epoch 19 TRAIN:  16%|█▌        | 22/137 [00:07<00:23,  4.85it/s, epoch=19, grad_norm=91.550, loss=339.5789, lr=2.09e-04]
Epoch 19 TRAIN:  16%|█▌        | 22/137 [00:07<00:23,  4.85it/s, epoch=19, grad_norm=113.437, loss=301.5073, lr=2.09e-04]
Epoch 19 TRAIN:  16%|█▌        | 22/137 [00:07<00:23,  4.85it/s, epoch=19, grad_norm=102.877, loss=320.6123, lr=2.09e-04]
Epoch 19 TRAIN:  16%|█▌        | 22/137 [00:07<00:23,  4.85it/s, epoch=19, grad_norm=87.844, loss=352.4961, lr=2.09e-04] 
Epoch 19 TRAIN:  16%|█▌        | 22/137 [00:07<00:23,  4.85it/s, epoch=19, grad_norm=77.335, loss=347.5184, lr=2.09e-04]
Epoch 19 TRAIN:  19%|█▉        | 26/137 [00:07<00:21,  5.25it/s, epoch=19, grad_norm=77.335, loss=347.5184, lr=2.09e-04]
Epoch 19 TRAIN:  19%|█▉        | 26/137 [00:07<00:21,  5.25it/s, epoch=19, grad_norm=64.933, loss=345.5194, lr=2.09e-04]
Epoch 19 TRAIN:  19%|█▉        | 26/137 [00:07<00:21,  5.25it/s, epoch=19, grad_norm=72.611, loss=324.5249, lr=2.09e-04]
Epoch 19 TRAIN:  19%|█▉        | 26/137 [00:08<00:21,  5.25it/s, epoch=19, grad_norm=169.152, loss=365.5146, lr=2.09e-04]
Epoch 19 TRAIN:  19%|█▉        | 26/137 [00:08<00:21,  5.25it/s, epoch=19, grad_norm=81.933, loss=342.8545, lr=2.09e-04] 
Epoch 19 TRAIN:  22%|██▏       | 30/137 [00:08<00:20,  5.30it/s, epoch=19, grad_norm=81.933, loss=342.8545, lr=2.09e-04]
Epoch 19 TRAIN:  22%|██▏       | 30/137 [00:08<00:20,  5.30it/s, epoch=19, grad_norm=79.569, loss=367.1452, lr=2.09e-04]
Epoch 19 TRAIN:  22%|██▏       | 30/137 [00:08<00:20,  5.30it/s, epoch=19, grad_norm=113.358, loss=310.7325, lr=2.09e-04]
Epoch 19 TRAIN:  22%|██▏       | 30/137 [00:08<00:20,  5.30it/s, epoch=19, grad_norm=116.696, loss=321.3704, lr=2.10e-04]
Epoch 19 TRAIN:  22%|██▏       | 30/137 [00:09<00:20,  5.30it/s, epoch=19, grad_norm=74.452, loss=314.4080, lr=2.10e-04] 
Epoch 19 TRAIN:  25%|██▍       | 34/137 [00:09<00:18,  5.63it/s, epoch=19, grad_norm=74.452, loss=314.4080, lr=2.10e-04]
Epoch 19 TRAIN:  25%|██▍       | 34/137 [00:09<00:18,  5.63it/s, epoch=19, grad_norm=99.376, loss=326.3567, lr=2.10e-04]
Epoch 19 TRAIN:  25%|██▍       | 34/137 [00:09<00:18,  5.63it/s, epoch=19, grad_norm=107.364, loss=303.2213, lr=2.10e-04]
Epoch 19 TRAIN:  25%|██▍       | 34/137 [00:09<00:18,  5.63it/s, epoch=19, grad_norm=142.589, loss=375.6380, lr=2.10e-04]
Epoch 19 TRAIN:  25%|██▍       | 34/137 [00:09<00:18,  5.63it/s, epoch=19, grad_norm=139.183, loss=347.9346, lr=2.10e-04]
Epoch 19 TRAIN:  28%|██▊       | 38/137 [00:09<00:18,  5.41it/s, epoch=19, grad_norm=139.183, loss=347.9346, lr=2.10e-04]
Epoch 19 TRAIN:  28%|██▊       | 38/137 [00:10<00:18,  5.41it/s, epoch=19, grad_norm=83.074, loss=321.9193, lr=2.10e-04] 
Epoch 19 TRAIN:  28%|██▊       | 38/137 [00:10<00:18,  5.41it/s, epoch=19, grad_norm=110.652, loss=363.5233, lr=2.10e-04]
Epoch 19 TRAIN:  28%|██▊       | 38/137 [00:10<00:18,  5.41it/s, epoch=19, grad_norm=160.295, loss=343.5666, lr=2.10e-04]
Epoch 19 TRAIN:  30%|██▉       | 41/137 [00:10<00:17,  5.39it/s, epoch=19, grad_norm=160.295, loss=343.5666, lr=2.10e-04]
Epoch 19 TRAIN:  30%|██▉       | 41/137 [00:10<00:17,  5.39it/s, epoch=19, grad_norm=95.702, loss=293.7577, lr=2.10e-04] 
Epoch 19 TRAIN:  30%|██▉       | 41/137 [00:10<00:17,  5.39it/s, epoch=19, grad_norm=129.247, loss=288.3256, lr=2.10e-04]
Epoch 19 TRAIN:  30%|██▉       | 41/137 [00:10<00:17,  5.39it/s, epoch=19, grad_norm=66.578, loss=319.7535, lr=2.10e-04] 
Epoch 19 TRAIN:  32%|███▏      | 44/137 [00:10<00:17,  5.45it/s, epoch=19, grad_norm=66.578, loss=319.7535, lr=2.10e-04]
Epoch 19 TRAIN:  32%|███▏      | 44/137 [00:11<00:17,  5.45it/s, epoch=19, grad_norm=126.253, loss=348.4833, lr=2.10e-04]
Epoch 19 TRAIN:  32%|███▏      | 44/137 [00:11<00:17,  5.45it/s, epoch=19, grad_norm=64.690, loss=300.4115, lr=2.11e-04] 
Epoch 19 TRAIN:  32%|███▏      | 44/137 [00:11<00:17,  5.45it/s, epoch=19, grad_norm=111.353, loss=347.8980, lr=2.11e-04]
Epoch 19 TRAIN:  34%|███▍      | 47/137 [00:11<00:16,  5.49it/s, epoch=19, grad_norm=111.353, loss=347.8980, lr=2.11e-04]
Epoch 19 TRAIN:  34%|███▍      | 47/137 [00:11<00:16,  5.49it/s, epoch=19, grad_norm=82.618, loss=296.4233, lr=2.11e-04] 
Epoch 19 TRAIN:  34%|███▍      | 47/137 [00:11<00:16,  5.49it/s, epoch=19, grad_norm=144.607, loss=366.9783, lr=2.11e-04]
Epoch 19 TRAIN:  34%|███▍      | 47/137 [00:12<00:16,  5.49it/s, epoch=19, grad_norm=86.466, loss=287.6559, lr=2.11e-04] 
Epoch 19 TRAIN:  36%|███▋      | 50/137 [00:12<00:16,  5.24it/s, epoch=19, grad_norm=86.466, loss=287.6559, lr=2.11e-04]
Epoch 19 TRAIN:  36%|███▋      | 50/137 [00:12<00:16,  5.24it/s, epoch=19, grad_norm=110.207, loss=371.4870, lr=2.11e-04]
Epoch 19 TRAIN:  36%|███▋      | 50/137 [00:12<00:16,  5.24it/s, epoch=19, grad_norm=79.739, loss=307.0266, lr=2.11e-04] 
Epoch 19 TRAIN:  36%|███▋      | 50/137 [00:12<00:16,  5.24it/s, epoch=19, grad_norm=89.016, loss=354.1699, lr=2.11e-04]
Epoch 19 TRAIN:  39%|███▊      | 53/137 [00:12<00:16,  5.15it/s, epoch=19, grad_norm=89.016, loss=354.1699, lr=2.11e-04]
Epoch 19 TRAIN:  39%|███▊      | 53/137 [00:12<00:16,  5.15it/s, epoch=19, grad_norm=118.442, loss=284.1589, lr=2.11e-04]
Epoch 19 TRAIN:  39%|███▊      | 53/137 [00:13<00:16,  5.15it/s, epoch=19, grad_norm=107.454, loss=301.9496, lr=2.11e-04]
Epoch 19 TRAIN:  39%|███▊      | 53/137 [00:13<00:16,  5.15it/s, epoch=19, grad_norm=78.181, loss=349.5259, lr=2.11e-04] 
Epoch 19 TRAIN:  41%|████      | 56/137 [00:13<00:15,  5.24it/s, epoch=19, grad_norm=78.181, loss=349.5259, lr=2.11e-04]
Epoch 19 TRAIN:  41%|████      | 56/137 [00:13<00:15,  5.24it/s, epoch=19, grad_norm=112.344, loss=269.6264, lr=2.11e-04]
Epoch 19 TRAIN:  41%|████      | 56/137 [00:13<00:15,  5.24it/s, epoch=19, grad_norm=85.012, loss=278.0113, lr=2.12e-04] 
Epoch 19 TRAIN:  41%|████      | 56/137 [00:13<00:15,  5.24it/s, epoch=19, grad_norm=71.919, loss=287.5060, lr=2.12e-04]
Epoch 19 TRAIN:  43%|████▎     | 59/137 [00:13<00:14,  5.22it/s, epoch=19, grad_norm=71.919, loss=287.5060, lr=2.12e-04]
Epoch 19 TRAIN:  43%|████▎     | 59/137 [00:13<00:14,  5.22it/s, epoch=19, grad_norm=160.324, loss=358.4908, lr=2.12e-04]
Epoch 19 TRAIN:  43%|████▎     | 59/137 [00:14<00:14,  5.22it/s, epoch=19, grad_norm=85.135, loss=281.3767, lr=2.12e-04] 
Epoch 19 TRAIN:  43%|████▎     | 59/137 [00:14<00:14,  5.22it/s, epoch=19, grad_norm=110.479, loss=314.5677, lr=2.12e-04]
Epoch 19 TRAIN:  45%|████▌     | 62/137 [00:14<00:13,  5.40it/s, epoch=19, grad_norm=110.479, loss=314.5677, lr=2.12e-04]
Epoch 19 TRAIN:  45%|████▌     | 62/137 [00:14<00:13,  5.40it/s, epoch=19, grad_norm=104.804, loss=294.8911, lr=2.12e-04]
Epoch 19 TRAIN:  45%|████▌     | 62/137 [00:14<00:13,  5.40it/s, epoch=19, grad_norm=191.430, loss=333.0712, lr=2.12e-04]
Epoch 19 TRAIN:  45%|████▌     | 62/137 [00:14<00:13,  5.40it/s, epoch=19, grad_norm=166.000, loss=359.2390, lr=2.12e-04]
Epoch 19 TRAIN:  47%|████▋     | 65/137 [00:14<00:13,  5.21it/s, epoch=19, grad_norm=166.000, loss=359.2390, lr=2.12e-04]
Epoch 19 TRAIN:  47%|████▋     | 65/137 [00:15<00:13,  5.21it/s, epoch=19, grad_norm=121.272, loss=329.8467, lr=2.12e-04]
Epoch 19 TRAIN:  47%|████▋     | 65/137 [00:15<00:13,  5.21it/s, epoch=19, grad_norm=117.705, loss=308.2338, lr=2.12e-04]
Epoch 19 TRAIN:  47%|████▋     | 65/137 [00:15<00:13,  5.21it/s, epoch=19, grad_norm=118.684, loss=328.1367, lr=2.12e-04]
Epoch 19 TRAIN:  50%|████▉     | 68/137 [00:15<00:13,  5.06it/s, epoch=19, grad_norm=118.684, loss=328.1367, lr=2.12e-04]
Epoch 19 TRAIN:  50%|████▉     | 68/137 [00:15<00:13,  5.06it/s, epoch=19, grad_norm=88.818, loss=350.7208, lr=2.12e-04] 
Epoch 19 TRAIN:  50%|████▉     | 68/137 [00:15<00:13,  5.06it/s, epoch=19, grad_norm=112.905, loss=300.2198, lr=2.12e-04]
Epoch 19 TRAIN:  50%|████▉     | 68/137 [00:16<00:13,  5.06it/s, epoch=19, grad_norm=122.960, loss=340.1350, lr=2.13e-04]
Epoch 19 TRAIN:  52%|█████▏    | 71/137 [00:16<00:12,  5.13it/s, epoch=19, grad_norm=122.960, loss=340.1350, lr=2.13e-04]
Epoch 19 TRAIN:  52%|█████▏    | 71/137 [00:16<00:12,  5.13it/s, epoch=19, grad_norm=74.078, loss=381.6569, lr=2.13e-04] 
Epoch 19 TRAIN:  52%|█████▏    | 71/137 [00:16<00:12,  5.13it/s, epoch=19, grad_norm=93.700, loss=342.4279, lr=2.13e-04]
Epoch 19 TRAIN:  52%|█████▏    | 71/137 [00:16<00:12,  5.13it/s, epoch=19, grad_norm=121.417, loss=268.2449, lr=2.13e-04]
Epoch 19 TRAIN:  54%|█████▍    | 74/137 [00:16<00:12,  5.06it/s, epoch=19, grad_norm=121.417, loss=268.2449, lr=2.13e-04]
Epoch 19 TRAIN:  54%|█████▍    | 74/137 [00:16<00:12,  5.06it/s, epoch=19, grad_norm=129.850, loss=368.0499, lr=2.13e-04]
Epoch 19 TRAIN:  54%|█████▍    | 74/137 [00:17<00:12,  5.06it/s, epoch=19, grad_norm=110.881, loss=290.3299, lr=2.13e-04]
Epoch 19 TRAIN:  54%|█████▍    | 74/137 [00:17<00:12,  5.06it/s, epoch=19, grad_norm=71.418, loss=322.5270, lr=2.13e-04] 
Epoch 19 TRAIN:  56%|█████▌    | 77/137 [00:17<00:11,  5.13it/s, epoch=19, grad_norm=71.418, loss=322.5270, lr=2.13e-04]
Epoch 19 TRAIN:  56%|█████▌    | 77/137 [00:17<00:11,  5.13it/s, epoch=19, grad_norm=75.744, loss=263.3615, lr=2.13e-04]
Epoch 19 TRAIN:  56%|█████▌    | 77/137 [00:17<00:11,  5.13it/s, epoch=19, grad_norm=153.394, loss=358.0483, lr=2.13e-04]
Epoch 19 TRAIN:  56%|█████▌    | 77/137 [00:17<00:11,  5.13it/s, epoch=19, grad_norm=108.002, loss=288.7726, lr=2.13e-04]
Epoch 19 TRAIN:  58%|█████▊    | 80/137 [00:17<00:11,  5.17it/s, epoch=19, grad_norm=108.002, loss=288.7726, lr=2.13e-04]
Epoch 19 TRAIN:  58%|█████▊    | 80/137 [00:18<00:11,  5.17it/s, epoch=19, grad_norm=122.896, loss=324.1963, lr=2.13e-04]
Epoch 19 TRAIN:  58%|█████▊    | 80/137 [00:18<00:11,  5.17it/s, epoch=19, grad_norm=213.748, loss=379.6445, lr=2.13e-04]
Epoch 19 TRAIN:  58%|█████▊    | 80/137 [00:18<00:11,  5.17it/s, epoch=19, grad_norm=70.848, loss=295.5112, lr=2.14e-04] 
Epoch 19 TRAIN:  61%|██████    | 83/137 [00:18<00:10,  5.04it/s, epoch=19, grad_norm=70.848, loss=295.5112, lr=2.14e-04]
Epoch 19 TRAIN:  61%|██████    | 83/137 [00:18<00:10,  5.04it/s, epoch=19, grad_norm=124.561, loss=343.0394, lr=2.14e-04]
Epoch 19 TRAIN:  61%|██████    | 83/137 [00:18<00:10,  5.04it/s, epoch=19, grad_norm=130.936, loss=332.2996, lr=2.14e-04]
Epoch 19 TRAIN:  61%|██████    | 83/137 [00:19<00:10,  5.04it/s, epoch=19, grad_norm=143.001, loss=246.8975, lr=2.14e-04]
Epoch 19 TRAIN:  63%|██████▎   | 86/137 [00:19<00:09,  5.10it/s, epoch=19, grad_norm=143.001, loss=246.8975, lr=2.14e-04]
Epoch 19 TRAIN:  63%|██████▎   | 86/137 [00:19<00:09,  5.10it/s, epoch=19, grad_norm=107.237, loss=362.4442, lr=2.14e-04]
Epoch 19 TRAIN:  63%|██████▎   | 86/137 [00:19<00:09,  5.10it/s, epoch=19, grad_norm=182.832, loss=270.9977, lr=2.14e-04]
Epoch 19 TRAIN:  63%|██████▎   | 86/137 [00:19<00:09,  5.10it/s, epoch=19, grad_norm=115.767, loss=271.8805, lr=2.14e-04]
Epoch 19 TRAIN:  65%|██████▍   | 89/137 [00:19<00:09,  5.14it/s, epoch=19, grad_norm=115.767, loss=271.8805, lr=2.14e-04]
Epoch 19 TRAIN:  65%|██████▍   | 89/137 [00:19<00:09,  5.14it/s, epoch=19, grad_norm=98.753, loss=348.5163, lr=2.14e-04] 
Epoch 19 TRAIN:  65%|██████▍   | 89/137 [00:20<00:09,  5.14it/s, epoch=19, grad_norm=121.905, loss=278.8562, lr=2.14e-04]
Epoch 19 TRAIN:  65%|██████▍   | 89/137 [00:20<00:09,  5.14it/s, epoch=19, grad_norm=111.713, loss=310.5049, lr=2.14e-04]
Epoch 19 TRAIN:  67%|██████▋   | 92/137 [00:20<00:08,  5.16it/s, epoch=19, grad_norm=111.713, loss=310.5049, lr=2.14e-04]
Epoch 19 TRAIN:  67%|██████▋   | 92/137 [00:20<00:08,  5.16it/s, epoch=19, grad_norm=166.886, loss=234.5231, lr=2.14e-04]
Epoch 19 TRAIN:  67%|██████▋   | 92/137 [00:20<00:08,  5.16it/s, epoch=19, grad_norm=124.965, loss=315.1829, lr=2.14e-04]
Epoch 19 TRAIN:  67%|██████▋   | 92/137 [00:20<00:08,  5.16it/s, epoch=19, grad_norm=79.190, loss=223.5923, lr=2.14e-04] 
Epoch 19 TRAIN:  69%|██████▉   | 95/137 [00:20<00:08,  5.05it/s, epoch=19, grad_norm=79.190, loss=223.5923, lr=2.14e-04]
Epoch 19 TRAIN:  69%|██████▉   | 95/137 [00:21<00:08,  5.05it/s, epoch=19, grad_norm=138.172, loss=333.0932, lr=2.15e-04]
Epoch 19 TRAIN:  69%|██████▉   | 95/137 [00:21<00:08,  5.05it/s, epoch=19, grad_norm=83.183, loss=269.2150, lr=2.15e-04] 
Epoch 19 TRAIN:  69%|██████▉   | 95/137 [00:21<00:08,  5.05it/s, epoch=19, grad_norm=91.945, loss=260.4517, lr=2.15e-04]
Epoch 19 TRAIN:  72%|███████▏  | 98/137 [00:21<00:07,  5.22it/s, epoch=19, grad_norm=91.945, loss=260.4517, lr=2.15e-04]
Epoch 19 TRAIN:  72%|███████▏  | 98/137 [00:21<00:07,  5.22it/s, epoch=19, grad_norm=190.315, loss=387.4988, lr=2.15e-04]
Epoch 19 TRAIN:  72%|███████▏  | 98/137 [00:21<00:07,  5.22it/s, epoch=19, grad_norm=114.156, loss=297.2246, lr=2.15e-04]2026-04-16 20:01:30,075 DEBUG TRAIN | steps/sec 4.613| Batch 19/100 loss 297.224609 loss_att 269.116302 loss_ctc 362.810638 th_accuracy 0.292308 lr 2.1488e-04 grad_norm 114.156403 rank 0

Epoch 19 TRAIN:  72%|███████▏  | 98/137 [00:21<00:07,  5.22it/s, epoch=19, grad_norm=100.331, loss=290.5858, lr=2.15e-04]
Epoch 19 TRAIN:  72%|███████▏  | 98/137 [00:22<00:07,  5.22it/s, epoch=19, grad_norm=162.316, loss=329.1522, lr=2.15e-04]
Epoch 19 TRAIN:  74%|███████▍  | 102/137 [00:22<00:06,  5.24it/s, epoch=19, grad_norm=162.316, loss=329.1522, lr=2.15e-04]
Epoch 19 TRAIN:  74%|███████▍  | 102/137 [00:22<00:06,  5.24it/s, epoch=19, grad_norm=130.005, loss=241.1921, lr=2.15e-04]
Epoch 19 TRAIN:  74%|███████▍  | 102/137 [00:22<00:06,  5.24it/s, epoch=19, grad_norm=154.131, loss=228.1113, lr=2.15e-04]
Epoch 19 TRAIN:  74%|███████▍  | 102/137 [00:22<00:06,  5.24it/s, epoch=19, grad_norm=179.250, loss=406.3345, lr=2.15e-04]
Epoch 19 TRAIN:  77%|███████▋  | 105/137 [00:22<00:05,  5.37it/s, epoch=19, grad_norm=179.250, loss=406.3345, lr=2.15e-04]
Epoch 19 TRAIN:  77%|███████▋  | 105/137 [00:22<00:05,  5.37it/s, epoch=19, grad_norm=118.656, loss=210.2581, lr=2.15e-04]
Epoch 19 TRAIN:  77%|███████▋  | 105/137 [00:23<00:05,  5.37it/s, epoch=19, grad_norm=112.627, loss=297.8846, lr=2.15e-04]
Epoch 19 TRAIN:  77%|███████▋  | 105/137 [00:23<00:05,  5.37it/s, epoch=19, grad_norm=224.045, loss=301.4467, lr=2.16e-04]
Epoch 19 TRAIN:  79%|███████▉  | 108/137 [00:23<00:05,  5.25it/s, epoch=19, grad_norm=224.045, loss=301.4467, lr=2.16e-04]
Epoch 19 TRAIN:  79%|███████▉  | 108/137 [00:23<00:05,  5.25it/s, epoch=19, grad_norm=105.896, loss=287.9498, lr=2.16e-04]
Epoch 19 TRAIN:  79%|███████▉  | 108/137 [00:23<00:05,  5.25it/s, epoch=19, grad_norm=220.410, loss=348.2708, lr=2.16e-04]
Epoch 19 TRAIN:  79%|███████▉  | 108/137 [00:23<00:05,  5.25it/s, epoch=19, grad_norm=156.955, loss=292.7033, lr=2.16e-04]
Epoch 19 TRAIN:  81%|████████  | 111/137 [00:23<00:05,  5.05it/s, epoch=19, grad_norm=156.955, loss=292.7033, lr=2.16e-04]
Epoch 19 TRAIN:  81%|████████  | 111/137 [00:24<00:05,  5.05it/s, epoch=19, grad_norm=232.130, loss=371.6337, lr=2.16e-04]
Epoch 19 TRAIN:  81%|████████  | 111/137 [00:24<00:05,  5.05it/s, epoch=19, grad_norm=261.004, loss=336.8719, lr=2.16e-04]
Epoch 19 TRAIN:  81%|████████  | 111/137 [00:24<00:05,  5.05it/s, epoch=19, grad_norm=149.606, loss=287.9786, lr=2.16e-04]
Epoch 19 TRAIN:  83%|████████▎ | 114/137 [00:24<00:04,  4.89it/s, epoch=19, grad_norm=149.606, loss=287.9786, lr=2.16e-04]
Epoch 19 TRAIN:  83%|████████▎ | 114/137 [00:24<00:04,  4.89it/s, epoch=19, grad_norm=117.141, loss=326.3926, lr=2.16e-04]
Epoch 19 TRAIN:  83%|████████▎ | 114/137 [00:25<00:04,  4.89it/s, epoch=19, grad_norm=157.833, loss=387.1076, lr=2.16e-04]
Epoch 19 TRAIN:  83%|████████▎ | 114/137 [00:25<00:04,  4.89it/s, epoch=19, grad_norm=134.477, loss=335.7570, lr=2.16e-04]
Epoch 19 TRAIN:  85%|████████▌ | 117/137 [00:25<00:04,  4.98it/s, epoch=19, grad_norm=134.477, loss=335.7570, lr=2.16e-04]
Epoch 19 TRAIN:  85%|████████▌ | 117/137 [00:25<00:04,  4.98it/s, epoch=19, grad_norm=103.167, loss=372.3243, lr=2.16e-04]
Epoch 19 TRAIN:  85%|████████▌ | 117/137 [00:25<00:04,  4.98it/s, epoch=19, grad_norm=127.331, loss=330.6938, lr=2.16e-04]
Epoch 19 TRAIN:  85%|████████▌ | 117/137 [00:25<00:04,  4.98it/s, epoch=19, grad_norm=146.048, loss=286.9851, lr=2.16e-04]
Epoch 19 TRAIN:  88%|████████▊ | 120/137 [00:25<00:03,  5.03it/s, epoch=19, grad_norm=146.048, loss=286.9851, lr=2.16e-04]
Epoch 19 TRAIN:  88%|████████▊ | 120/137 [00:25<00:03,  5.03it/s, epoch=19, grad_norm=76.042, loss=253.0435, lr=2.17e-04] 
Epoch 19 TRAIN:  88%|████████▊ | 120/137 [00:26<00:03,  5.03it/s, epoch=19, grad_norm=262.231, loss=316.8310, lr=2.17e-04]
Epoch 19 TRAIN:  88%|████████▊ | 120/137 [00:26<00:03,  5.03it/s, epoch=19, grad_norm=86.746, loss=256.0726, lr=2.17e-04] 
Epoch 19 TRAIN:  90%|████████▉ | 123/137 [00:26<00:02,  5.06it/s, epoch=19, grad_norm=86.746, loss=256.0726, lr=2.17e-04]
Epoch 19 TRAIN:  90%|████████▉ | 123/137 [00:26<00:02,  5.06it/s, epoch=19, grad_norm=84.014, loss=393.7858, lr=2.17e-04]
Epoch 19 TRAIN:  90%|████████▉ | 123/137 [00:26<00:02,  5.06it/s, epoch=19, grad_norm=235.152, loss=387.2231, lr=2.17e-04]
Epoch 19 TRAIN:  90%|████████▉ | 123/137 [00:26<00:02,  5.06it/s, epoch=19, grad_norm=221.387, loss=317.6612, lr=2.17e-04]
Epoch 19 TRAIN:  92%|█████████▏| 126/137 [00:26<00:02,  5.25it/s, epoch=19, grad_norm=221.387, loss=317.6612, lr=2.17e-04]
Epoch 19 TRAIN:  92%|█████████▏| 126/137 [00:27<00:02,  5.25it/s, epoch=19, grad_norm=219.702, loss=388.9070, lr=2.17e-04]
Epoch 19 TRAIN:  92%|█████████▏| 126/137 [00:27<00:02,  5.25it/s, epoch=19, grad_norm=135.523, loss=348.9501, lr=2.17e-04]
Epoch 19 TRAIN:  92%|█████████▏| 126/137 [00:27<00:02,  5.25it/s, epoch=19, grad_norm=69.624, loss=256.6802, lr=2.17e-04] 
Epoch 19 TRAIN:  94%|█████████▍| 129/137 [00:27<00:01,  5.22it/s, epoch=19, grad_norm=69.624, loss=256.6802, lr=2.17e-04]
Epoch 19 TRAIN:  94%|█████████▍| 129/137 [00:27<00:01,  5.22it/s, epoch=19, grad_norm=218.546, loss=417.5058, lr=2.17e-04]
Epoch 19 TRAIN:  94%|█████████▍| 129/137 [00:27<00:01,  5.22it/s, epoch=19, grad_norm=99.214, loss=295.6067, lr=2.17e-04] 
Epoch 19 TRAIN:  94%|█████████▍| 129/137 [00:27<00:01,  5.22it/s, epoch=19, grad_norm=123.560, loss=308.6538, lr=2.17e-04]
Epoch 19 TRAIN:  94%|█████████▍| 129/137 [00:28<00:01,  5.22it/s, epoch=19, grad_norm=114.953, loss=327.9175, lr=2.18e-04]
Epoch 19 TRAIN:  97%|█████████▋| 133/137 [00:28<00:00,  5.64it/s, epoch=19, grad_norm=114.953, loss=327.9175, lr=2.18e-04]
Epoch 19 TRAIN:  97%|█████████▋| 133/137 [00:28<00:00,  5.64it/s, epoch=19, grad_norm=215.947, loss=405.5095, lr=2.18e-04]
Epoch 19 TRAIN:  97%|█████████▋| 133/137 [00:28<00:00,  5.64it/s, epoch=19, grad_norm=118.391, loss=321.5652, lr=2.18e-04]
Epoch 19 TRAIN:  97%|█████████▋| 133/137 [00:28<00:00,  5.64it/s, epoch=19, grad_norm=153.793, loss=359.2827, lr=2.18e-04]
Epoch 19 TRAIN:  99%|█████████▉| 136/137 [00:28<00:00,  5.34it/s, epoch=19, grad_norm=153.793, loss=359.2827, lr=2.18e-04]
Epoch 19 TRAIN:  99%|█████████▉| 136/137 [00:28<00:00,  5.34it/s, epoch=19, grad_norm=227.474, loss=431.2874, lr=2.18e-04]
Epoch 19 TRAIN: 100%|██████████| 137/137 [00:28<00:00,  4.75it/s, epoch=19, grad_norm=227.474, loss=431.2874, lr=2.18e-04]

CV:   0%|          | 0/6 [00:00<?, ?it/s]/home/rookie/Documents/batuka/projects/wenet/wenet/wenet/utils/train_utils.py:689: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  amp_autocast(enabled=dtype is not None,

CV:  17%|█▋        | 1/6 [00:00<00:03,  1.31it/s]
                                                 
2026-04-16 20:01:38,063 INFO Epoch 19 Step 2723 CV info lr 2.1784e-04 cv_loss 377.3046320134943 rank 0 acc 0.24661835034688315
2026-04-16 20:01:38,065 INFO [Rank 0] Checkpoint: save to checkpoint exp/conformer_mongolian/epoch_19.pt
2026-04-16 20:01:38,189 INFO Epoch 20 Step 2723 TRAIN info lr 2.1784e-04 rank 0
2026-04-16 20:01:38,202 INFO using accumulate grad, new batch size is 1 times larger than before

Epoch 20 TRAIN:   0%|          | 0/137 [00:00<?, ?it/s]
Epoch 20 TRAIN:   0%|          | 0/137 [00:05<?, ?it/s, epoch=20, grad_norm=491.818, loss=443.8596, lr=2.18e-04]
Epoch 20 TRAIN:   1%|          | 1/137 [00:05<11:59,  5.29s/it, epoch=20, grad_norm=491.818, loss=443.8596, lr=2.18e-04]
Epoch 20 TRAIN:   1%|          | 1/137 [00:05<11:59,  5.29s/it, epoch=20, grad_norm=653.049, loss=479.6167, lr=2.18e-04]
Epoch 20 TRAIN:   1%|          | 1/137 [00:05<11:59,  5.29s/it, epoch=20, grad_norm=474.459, loss=458.3308, lr=2.18e-04]
Epoch 20 TRAIN:   1%|          | 1/137 [00:05<11:59,  5.29s/it, epoch=20, grad_norm=281.577, loss=345.4128, lr=2.18e-04]
Epoch 20 TRAIN:   1%|          | 1/137 [00:05<11:59,  5.29s/it, epoch=20, grad_norm=249.498, loss=329.0561, lr=2.18e-04]
Epoch 20 TRAIN:   1%|          | 1/137 [00:05<11:59,  5.29s/it, epoch=20, grad_norm=114.040, loss=284.0483, lr=2.18e-04]
Epoch 20 TRAIN:   1%|          | 1/137 [00:05<11:59,  5.29s/it, epoch=20, grad_norm=145.424, loss=375.6498, lr=2.18e-04]
Epoch 20 TRAIN:   5%|▌         | 7/137 [00:05<01:23,  1.56it/s, epoch=20, grad_norm=145.424, loss=375.6498, lr=2.18e-04]
Epoch 20 TRAIN:   5%|▌         | 7/137 [00:05<01:23,  1.56it/s, epoch=20, grad_norm=268.877, loss=374.5457, lr=2.18e-04]
Epoch 20 TRAIN:   5%|▌         | 7/137 [00:06<01:23,  1.56it/s, epoch=20, grad_norm=76.733, loss=362.2727, lr=2.19e-04] 
Epoch 20 TRAIN:   5%|▌         | 7/137 [00:06<01:23,  1.56it/s, epoch=20, grad_norm=105.824, loss=337.7304, lr=2.19e-04]
Epoch 20 TRAIN:   5%|▌         | 7/137 [00:06<01:23,  1.56it/s, epoch=20, grad_norm=141.658, loss=311.4739, lr=2.19e-04]
Epoch 20 TRAIN:   8%|▊         | 11/137 [00:06<00:52,  2.39it/s, epoch=20, grad_norm=141.658, loss=311.4739, lr=2.19e-04]
Epoch 20 TRAIN:   8%|▊         | 11/137 [00:06<00:52,  2.39it/s, epoch=20, grad_norm=109.441, loss=302.3493, lr=2.19e-04]
Epoch 20 TRAIN:   8%|▊         | 11/137 [00:06<00:52,  2.39it/s, epoch=20, grad_norm=86.771, loss=353.1099, lr=2.19e-04] 
Epoch 20 TRAIN:   8%|▊         | 11/137 [00:07<00:52,  2.39it/s, epoch=20, grad_norm=64.903, loss=340.5047, lr=2.19e-04]
Epoch 20 TRAIN:   8%|▊         | 11/137 [00:07<00:52,  2.39it/s, epoch=20, grad_norm=75.328, loss=303.3736, lr=2.19e-04]
Epoch 20 TRAIN:  11%|█         | 15/137 [00:07<00:39,  3.09it/s, epoch=20, grad_norm=75.328, loss=303.3736, lr=2.19e-04]
Epoch 20 TRAIN:  11%|█         | 15/137 [00:07<00:39,  3.09it/s, epoch=20, grad_norm=137.969, loss=377.8033, lr=2.19e-04]
Epoch 20 TRAIN:  11%|█         | 15/137 [00:07<00:39,  3.09it/s, epoch=20, grad_norm=76.716, loss=339.1696, lr=2.19e-04] 
Epoch 20 TRAIN:  11%|█         | 15/137 [00:07<00:39,  3.09it/s, epoch=20, grad_norm=62.774, loss=333.0887, lr=2.19e-04]
Epoch 20 TRAIN:  13%|█▎        | 18/137 [00:07<00:34,  3.48it/s, epoch=20, grad_norm=62.774, loss=333.0887, lr=2.19e-04]
Epoch 20 TRAIN:  13%|█▎        | 18/137 [00:08<00:34,  3.48it/s, epoch=20, grad_norm=74.480, loss=325.2256, lr=2.19e-04]
Epoch 20 TRAIN:  13%|█▎        | 18/137 [00:08<00:34,  3.48it/s, epoch=20, grad_norm=96.214, loss=297.4784, lr=2.19e-04]
Epoch 20 TRAIN:  13%|█▎        | 18/137 [00:08<00:34,  3.48it/s, epoch=20, grad_norm=79.553, loss=346.3266, lr=2.20e-04]
Epoch 20 TRAIN:  15%|█▌        | 21/137 [00:08<00:29,  3.90it/s, epoch=20, grad_norm=79.553, loss=346.3266, lr=2.20e-04]
Epoch 20 TRAIN:  15%|█▌        | 21/137 [00:08<00:29,  3.90it/s, epoch=20, grad_norm=101.635, loss=345.8914, lr=2.20e-04]
Epoch 20 TRAIN:  15%|█▌        | 21/137 [00:08<00:29,  3.90it/s, epoch=20, grad_norm=81.017, loss=282.6321, lr=2.20e-04] 
Epoch 20 TRAIN:  15%|█▌        | 21/137 [00:08<00:29,  3.90it/s, epoch=20, grad_norm=86.007, loss=338.6877, lr=2.20e-04]
Epoch 20 TRAIN:  18%|█▊        | 24/137 [00:08<00:26,  4.31it/s, epoch=20, grad_norm=86.007, loss=338.6877, lr=2.20e-04]
Epoch 20 TRAIN:  18%|█▊        | 24/137 [00:09<00:26,  4.31it/s, epoch=20, grad_norm=64.698, loss=346.7507, lr=2.20e-04]
Epoch 20 TRAIN:  18%|█▊        | 24/137 [00:09<00:26,  4.31it/s, epoch=20, grad_norm=62.583, loss=343.7310, lr=2.20e-04]
Epoch 20 TRAIN:  18%|█▊        | 24/137 [00:09<00:26,  4.31it/s, epoch=20, grad_norm=101.759, loss=350.7079, lr=2.20e-04]
Epoch 20 TRAIN:  20%|█▉        | 27/137 [00:09<00:23,  4.68it/s, epoch=20, grad_norm=101.759, loss=350.7079, lr=2.20e-04]
Epoch 20 TRAIN:  20%|█▉        | 27/137 [00:09<00:23,  4.68it/s, epoch=20, grad_norm=113.845, loss=283.4084, lr=2.20e-04]
Epoch 20 TRAIN:  20%|█▉        | 27/137 [00:09<00:23,  4.68it/s, epoch=20, grad_norm=58.364, loss=349.1884, lr=2.20e-04] 
Epoch 20 TRAIN:  20%|█▉        | 27/137 [00:09<00:23,  4.68it/s, epoch=20, grad_norm=79.949, loss=320.5730, lr=2.20e-04]
Epoch 20 TRAIN:  22%|██▏       | 30/137 [00:09<00:21,  4.93it/s, epoch=20, grad_norm=79.949, loss=320.5730, lr=2.20e-04]
Epoch 20 TRAIN:  22%|██▏       | 30/137 [00:10<00:21,  4.93it/s, epoch=20, grad_norm=80.513, loss=373.8721, lr=2.20e-04]
Epoch 20 TRAIN:  22%|██▏       | 30/137 [00:10<00:21,  4.93it/s, epoch=20, grad_norm=78.130, loss=310.9029, lr=2.20e-04]
Epoch 20 TRAIN:  22%|██▏       | 30/137 [00:10<00:21,  4.93it/s, epoch=20, grad_norm=101.855, loss=316.8538, lr=2.20e-04]
Epoch 20 TRAIN:  24%|██▍       | 33/137 [00:10<00:20,  5.12it/s, epoch=20, grad_norm=101.855, loss=316.8538, lr=2.20e-04]
Epoch 20 TRAIN:  24%|██▍       | 33/137 [00:10<00:20,  5.12it/s, epoch=20, grad_norm=88.454, loss=374.6862, lr=2.21e-04] 
Epoch 20 TRAIN:  24%|██▍       | 33/137 [00:10<00:20,  5.12it/s, epoch=20, grad_norm=87.682, loss=324.1581, lr=2.21e-04]
Epoch 20 TRAIN:  24%|██▍       | 33/137 [00:11<00:20,  5.12it/s, epoch=20, grad_norm=118.739, loss=278.4737, lr=2.21e-04]
Epoch 20 TRAIN:  26%|██▋       | 36/137 [00:11<00:19,  5.21it/s, epoch=20, grad_norm=118.739, loss=278.4737, lr=2.21e-04]
Epoch 20 TRAIN:  26%|██▋       | 36/137 [00:11<00:19,  5.21it/s, epoch=20, grad_norm=75.369, loss=335.4410, lr=2.21e-04] 
Epoch 20 TRAIN:  26%|██▋       | 36/137 [00:11<00:19,  5.21it/s, epoch=20, grad_norm=72.440, loss=302.6001, lr=2.21e-04]
Epoch 20 TRAIN:  26%|██▋       | 36/137 [00:11<00:19,  5.21it/s, epoch=20, grad_norm=130.771, loss=347.8748, lr=2.21e-04]
Epoch 20 TRAIN:  28%|██▊       | 39/137 [00:11<00:18,  5.33it/s, epoch=20, grad_norm=130.771, loss=347.8748, lr=2.21e-04]
Epoch 20 TRAIN:  28%|██▊       | 39/137 [00:11<00:18,  5.33it/s, epoch=20, grad_norm=166.810, loss=347.5223, lr=2.21e-04]
Epoch 20 TRAIN:  28%|██▊       | 39/137 [00:11<00:18,  5.33it/s, epoch=20, grad_norm=63.679, loss=323.9130, lr=2.21e-04] 
Epoch 20 TRAIN:  28%|██▊       | 39/137 [00:12<00:18,  5.33it/s, epoch=20, grad_norm=60.406, loss=283.8781, lr=2.21e-04]
Epoch 20 TRAIN:  31%|███       | 42/137 [00:12<00:18,  5.23it/s, epoch=20, grad_norm=60.406, loss=283.8781, lr=2.21e-04]
Epoch 20 TRAIN:  31%|███       | 42/137 [00:12<00:18,  5.23it/s, epoch=20, grad_norm=109.728, loss=274.9599, lr=2.21e-04]
Epoch 20 TRAIN:  31%|███       | 42/137 [00:12<00:18,  5.23it/s, epoch=20, grad_norm=183.081, loss=418.5721, lr=2.21e-04]
Epoch 20 TRAIN:  31%|███       | 42/137 [00:12<00:18,  5.23it/s, epoch=20, grad_norm=75.442, loss=334.7941, lr=2.21e-04] 
Epoch 20 TRAIN:  33%|███▎      | 45/137 [00:12<00:18,  5.05it/s, epoch=20, grad_norm=75.442, loss=334.7941, lr=2.21e-04]
Epoch 20 TRAIN:  33%|███▎      | 45/137 [00:12<00:18,  5.05it/s, epoch=20, grad_norm=112.612, loss=317.3504, lr=2.22e-04]
Epoch 20 TRAIN:  33%|███▎      | 45/137 [00:13<00:18,  5.05it/s, epoch=20, grad_norm=75.449, loss=299.3388, lr=2.22e-04] 
Epoch 20 TRAIN:  33%|███▎      | 45/137 [00:13<00:18,  5.05it/s, epoch=20, grad_norm=119.711, loss=358.9832, lr=2.22e-04]
Epoch 20 TRAIN:  35%|███▌      | 48/137 [00:13<00:17,  5.20it/s, epoch=20, grad_norm=119.711, loss=358.9832, lr=2.22e-04]
Epoch 20 TRAIN:  35%|███▌      | 48/137 [00:13<00:17,  5.20it/s, epoch=20, grad_norm=113.858, loss=290.9527, lr=2.22e-04]
Epoch 20 TRAIN:  35%|███▌      | 48/137 [00:13<00:17,  5.20it/s, epoch=20, grad_norm=133.055, loss=262.5133, lr=2.22e-04]
Epoch 20 TRAIN:  35%|███▌      | 48/137 [00:13<00:17,  5.20it/s, epoch=20, grad_norm=51.772, loss=295.4725, lr=2.22e-04] 
Epoch 20 TRAIN:  37%|███▋      | 51/137 [00:13<00:16,  5.32it/s, epoch=20, grad_norm=51.772, loss=295.4725, lr=2.22e-04]
Epoch 20 TRAIN:  37%|███▋      | 51/137 [00:14<00:16,  5.32it/s, epoch=20, grad_norm=91.705, loss=331.0622, lr=2.22e-04]
Epoch 20 TRAIN:  37%|███▋      | 51/137 [00:14<00:16,  5.32it/s, epoch=20, grad_norm=90.321, loss=318.6185, lr=2.22e-04]
Epoch 20 TRAIN:  37%|███▋      | 51/137 [00:14<00:16,  5.32it/s, epoch=20, grad_norm=54.128, loss=314.2012, lr=2.22e-04]
Epoch 20 TRAIN:  39%|███▉      | 54/137 [00:14<00:16,  5.07it/s, epoch=20, grad_norm=54.128, loss=314.2012, lr=2.22e-04]
Epoch 20 TRAIN:  39%|███▉      | 54/137 [00:14<00:16,  5.07it/s, epoch=20, grad_norm=87.158, loss=333.2872, lr=2.22e-04]
Epoch 20 TRAIN:  39%|███▉      | 54/137 [00:14<00:16,  5.07it/s, epoch=20, grad_norm=75.321, loss=270.3829, lr=2.22e-04]
Epoch 20 TRAIN:  39%|███▉      | 54/137 [00:15<00:16,  5.07it/s, epoch=20, grad_norm=100.293, loss=355.6293, lr=2.22e-04]
Epoch 20 TRAIN:  42%|████▏     | 57/137 [00:15<00:15,  5.19it/s, epoch=20, grad_norm=100.293, loss=355.6293, lr=2.22e-04]
Epoch 20 TRAIN:  42%|████▏     | 57/137 [00:15<00:15,  5.19it/s, epoch=20, grad_norm=74.895, loss=343.9942, lr=2.22e-04] 
Epoch 20 TRAIN:  42%|████▏     | 57/137 [00:15<00:15,  5.19it/s, epoch=20, grad_norm=68.400, loss=318.0722, lr=2.23e-04]
Epoch 20 TRAIN:  42%|████▏     | 57/137 [00:15<00:15,  5.19it/s, epoch=20, grad_norm=69.818, loss=321.6333, lr=2.23e-04]
Epoch 20 TRAIN:  44%|████▍     | 60/137 [00:15<00:14,  5.33it/s, epoch=20, grad_norm=69.818, loss=321.6333, lr=2.23e-04]
Epoch 20 TRAIN:  44%|████▍     | 60/137 [00:15<00:14,  5.33it/s, epoch=20, grad_norm=163.877, loss=291.9230, lr=2.23e-04]
Epoch 20 TRAIN:  44%|████▍     | 60/137 [00:16<00:14,  5.33it/s, epoch=20, grad_norm=81.792, loss=361.9390, lr=2.23e-04] 
Epoch 20 TRAIN:  44%|████▍     | 60/137 [00:16<00:14,  5.33it/s, epoch=20, grad_norm=68.904, loss=326.7914, lr=2.23e-04]
Epoch 20 TRAIN:  46%|████▌     | 63/137 [00:16<00:14,  5.17it/s, epoch=20, grad_norm=68.904, loss=326.7914, lr=2.23e-04]
Epoch 20 TRAIN:  46%|████▌     | 63/137 [00:16<00:14,  5.17it/s, epoch=20, grad_norm=99.150, loss=276.1003, lr=2.23e-04]
Epoch 20 TRAIN:  46%|████▌     | 63/137 [00:16<00:14,  5.17it/s, epoch=20, grad_norm=63.241, loss=315.9405, lr=2.23e-04]
Epoch 20 TRAIN:  46%|████▌     | 63/137 [00:16<00:14,  5.17it/s, epoch=20, grad_norm=101.678, loss=329.3513, lr=2.23e-04]
Epoch 20 TRAIN:  46%|████▌     | 63/137 [00:16<00:14,  5.17it/s, epoch=20, grad_norm=101.157, loss=299.4296, lr=2.23e-04]
Epoch 20 TRAIN:  49%|████▉     | 67/137 [00:16<00:12,  5.52it/s, epoch=20, grad_norm=101.157, loss=299.4296, lr=2.23e-04]
Epoch 20 TRAIN:  49%|████▉     | 67/137 [00:17<00:12,  5.52it/s, epoch=20, grad_norm=60.051, loss=332.4082, lr=2.23e-04] 
Epoch 20 TRAIN:  49%|████▉     | 67/137 [00:17<00:12,  5.52it/s, epoch=20, grad_norm=114.399, loss=353.8730, lr=2.23e-04]
Epoch 20 TRAIN:  49%|████▉     | 67/137 [00:17<00:12,  5.52it/s, epoch=20, grad_norm=83.718, loss=258.8015, lr=2.23e-04] 
Epoch 20 TRAIN:  51%|█████     | 70/137 [00:17<00:12,  5.20it/s, epoch=20, grad_norm=83.718, loss=258.8015, lr=2.23e-04]
Epoch 20 TRAIN:  51%|█████     | 70/137 [00:17<00:12,  5.20it/s, epoch=20, grad_norm=59.884, loss=273.0139, lr=2.24e-04]
Epoch 20 TRAIN:  51%|█████     | 70/137 [00:17<00:12,  5.20it/s, epoch=20, grad_norm=110.735, loss=380.3505, lr=2.24e-04]
Epoch 20 TRAIN:  51%|█████     | 70/137 [00:18<00:12,  5.20it/s, epoch=20, grad_norm=62.299, loss=311.4824, lr=2.24e-04] 
Epoch 20 TRAIN:  53%|█████▎    | 73/137 [00:18<00:12,  5.10it/s, epoch=20, grad_norm=62.299, loss=311.4824, lr=2.24e-04]
Epoch 20 TRAIN:  53%|█████▎    | 73/137 [00:18<00:12,  5.10it/s, epoch=20, grad_norm=69.983, loss=321.4117, lr=2.24e-04]
Epoch 20 TRAIN:  53%|█████▎    | 73/137 [00:18<00:12,  5.10it/s, epoch=20, grad_norm=77.589, loss=322.1701, lr=2.24e-04]
Epoch 20 TRAIN:  53%|█████▎    | 73/137 [00:18<00:12,  5.10it/s, epoch=20, grad_norm=100.519, loss=307.5428, lr=2.24e-04]
Epoch 20 TRAIN:  55%|█████▌    | 76/137 [00:18<00:11,  5.16it/s, epoch=20, grad_norm=100.519, loss=307.5428, lr=2.24e-04]
Epoch 20 TRAIN:  55%|█████▌    | 76/137 [00:18<00:11,  5.16it/s, epoch=20, grad_norm=154.417, loss=267.5449, lr=2.24e-04]
Epoch 20 TRAIN:  55%|█████▌    | 76/137 [00:19<00:11,  5.16it/s, epoch=20, grad_norm=201.654, loss=350.1522, lr=2.24e-04]
Epoch 20 TRAIN:  55%|█████▌    | 76/137 [00:19<00:11,  5.16it/s, epoch=20, grad_norm=58.290, loss=285.3462, lr=2.24e-04] 
Epoch 20 TRAIN:  58%|█████▊    | 79/137 [00:19<00:11,  5.05it/s, epoch=20, grad_norm=58.290, loss=285.3462, lr=2.24e-04]
Epoch 20 TRAIN:  58%|█████▊    | 79/137 [00:19<00:11,  5.05it/s, epoch=20, grad_norm=88.656, loss=364.4542, lr=2.24e-04]
Epoch 20 TRAIN:  58%|█████▊    | 79/137 [00:19<00:11,  5.05it/s, epoch=20, grad_norm=85.146, loss=328.4463, lr=2.24e-04]
Epoch 20 TRAIN:  58%|█████▊    | 79/137 [00:19<00:11,  5.05it/s, epoch=20, grad_norm=71.178, loss=278.1028, lr=2.24e-04]
Epoch 20 TRAIN:  60%|█████▉    | 82/137 [00:19<00:10,  5.08it/s, epoch=20, grad_norm=71.178, loss=278.1028, lr=2.24e-04]
Epoch 20 TRAIN:  60%|█████▉    | 82/137 [00:20<00:10,  5.08it/s, epoch=20, grad_norm=88.043, loss=275.1559, lr=2.24e-04]
Epoch 20 TRAIN:  60%|█████▉    | 82/137 [00:20<00:10,  5.08it/s, epoch=20, grad_norm=108.840, loss=316.1962, lr=2.25e-04]
Epoch 20 TRAIN:  60%|█████▉    | 82/137 [00:20<00:10,  5.08it/s, epoch=20, grad_norm=106.524, loss=310.8731, lr=2.25e-04]
Epoch 20 TRAIN:  62%|██████▏   | 85/137 [00:20<00:10,  5.12it/s, epoch=20, grad_norm=106.524, loss=310.8731, lr=2.25e-04]
Epoch 20 TRAIN:  62%|██████▏   | 85/137 [00:20<00:10,  5.12it/s, epoch=20, grad_norm=188.623, loss=317.9105, lr=2.25e-04]
Epoch 20 TRAIN:  62%|██████▏   | 85/137 [00:20<00:10,  5.12it/s, epoch=20, grad_norm=162.816, loss=354.9517, lr=2.25e-04]
Epoch 20 TRAIN:  62%|██████▏   | 85/137 [00:21<00:10,  5.12it/s, epoch=20, grad_norm=73.071, loss=282.4578, lr=2.25e-04] 
Epoch 20 TRAIN:  64%|██████▍   | 88/137 [00:21<00:09,  5.21it/s, epoch=20, grad_norm=73.071, loss=282.4578, lr=2.25e-04]
Epoch 20 TRAIN:  64%|██████▍   | 88/137 [00:21<00:09,  5.21it/s, epoch=20, grad_norm=120.163, loss=299.4418, lr=2.25e-04]
Epoch 20 TRAIN:  64%|██████▍   | 88/137 [00:21<00:09,  5.21it/s, epoch=20, grad_norm=111.273, loss=245.8332, lr=2.25e-04]
Epoch 20 TRAIN:  64%|██████▍   | 88/137 [00:21<00:09,  5.21it/s, epoch=20, grad_norm=67.880, loss=318.4122, lr=2.25e-04] 
Epoch 20 TRAIN:  66%|██████▋   | 91/137 [00:21<00:09,  4.87it/s, epoch=20, grad_norm=67.880, loss=318.4122, lr=2.25e-04]
Epoch 20 TRAIN:  66%|██████▋   | 91/137 [00:21<00:09,  4.87it/s, epoch=20, grad_norm=117.868, loss=339.5932, lr=2.25e-04]
Epoch 20 TRAIN:  66%|██████▋   | 91/137 [00:22<00:09,  4.87it/s, epoch=20, grad_norm=151.286, loss=298.1947, lr=2.25e-04]
Epoch 20 TRAIN:  66%|██████▋   | 91/137 [00:22<00:09,  4.87it/s, epoch=20, grad_norm=96.144, loss=242.9837, lr=2.25e-04] 
Epoch 20 TRAIN:  69%|██████▊   | 94/137 [00:22<00:08,  4.84it/s, epoch=20, grad_norm=96.144, loss=242.9837, lr=2.25e-04]
Epoch 20 TRAIN:  69%|██████▊   | 94/137 [00:22<00:08,  4.84it/s, epoch=20, grad_norm=83.249, loss=305.1152, lr=2.25e-04]
Epoch 20 TRAIN:  69%|██████▊   | 94/137 [00:22<00:08,  4.84it/s, epoch=20, grad_norm=89.639, loss=324.8175, lr=2.26e-04]
Epoch 20 TRAIN:  69%|██████▊   | 94/137 [00:23<00:08,  4.84it/s, epoch=20, grad_norm=59.030, loss=280.5092, lr=2.26e-04]
Epoch 20 TRAIN:  71%|███████   | 97/137 [00:23<00:08,  4.81it/s, epoch=20, grad_norm=59.030, loss=280.5092, lr=2.26e-04]
Epoch 20 TRAIN:  71%|███████   | 97/137 [00:23<00:08,  4.81it/s, epoch=20, grad_norm=94.348, loss=198.9521, lr=2.26e-04]
Epoch 20 TRAIN:  71%|███████   | 97/137 [00:23<00:08,  4.81it/s, epoch=20, grad_norm=82.769, loss=275.7521, lr=2.26e-04]
Epoch 20 TRAIN:  71%|███████   | 97/137 [00:23<00:08,  4.81it/s, epoch=20, grad_norm=114.856, loss=275.7736, lr=2.26e-04]2026-04-16 20:02:01,787 DEBUG TRAIN | steps/sec 4.320| Batch 20/100 loss 275.773590 loss_att 250.286957 loss_ctc 335.242371 th_accuracy 0.245283 lr 2.2584e-04 grad_norm 114.856331 rank 0

Epoch 20 TRAIN:  73%|███████▎  | 100/137 [00:23<00:07,  4.95it/s, epoch=20, grad_norm=114.856, loss=275.7736, lr=2.26e-04]
Epoch 20 TRAIN:  73%|███████▎  | 100/137 [00:23<00:07,  4.95it/s, epoch=20, grad_norm=171.269, loss=296.6902, lr=2.26e-04]
Epoch 20 TRAIN:  73%|███████▎  | 100/137 [00:23<00:07,  4.95it/s, epoch=20, grad_norm=443.891, loss=267.3056, lr=2.26e-04]
Epoch 20 TRAIN:  73%|███████▎  | 100/137 [00:24<00:07,  4.95it/s, epoch=20, grad_norm=209.488, loss=301.5868, lr=2.26e-04]
Epoch 20 TRAIN:  75%|███████▌  | 103/137 [00:24<00:06,  5.01it/s, epoch=20, grad_norm=209.488, loss=301.5868, lr=2.26e-04]
Epoch 20 TRAIN:  75%|███████▌  | 103/137 [00:24<00:06,  5.01it/s, epoch=20, grad_norm=81.685, loss=263.9834, lr=2.26e-04] 
Epoch 20 TRAIN:  75%|███████▌  | 103/137 [00:24<00:06,  5.01it/s, epoch=20, grad_norm=140.186, loss=325.5594, lr=2.26e-04]
Epoch 20 TRAIN:  75%|███████▌  | 103/137 [00:24<00:06,  5.01it/s, epoch=20, grad_norm=158.165, loss=361.3577, lr=2.26e-04]
Epoch 20 TRAIN:  77%|███████▋  | 106/137 [00:24<00:05,  5.19it/s, epoch=20, grad_norm=158.165, loss=361.3577, lr=2.26e-04]
Epoch 20 TRAIN:  77%|███████▋  | 106/137 [00:24<00:05,  5.19it/s, epoch=20, grad_norm=203.807, loss=266.4405, lr=2.26e-04]
Epoch 20 TRAIN:  77%|███████▋  | 106/137 [00:25<00:05,  5.19it/s, epoch=20, grad_norm=300.312, loss=344.1437, lr=2.26e-04]
Epoch 20 TRAIN:  77%|███████▋  | 106/137 [00:25<00:05,  5.19it/s, epoch=20, grad_norm=101.341, loss=310.2462, lr=2.27e-04]
Epoch 20 TRAIN:  80%|███████▉  | 109/137 [00:25<00:05,  5.11it/s, epoch=20, grad_norm=101.341, loss=310.2462, lr=2.27e-04]
Epoch 20 TRAIN:  80%|███████▉  | 109/137 [00:25<00:05,  5.11it/s, epoch=20, grad_norm=110.000, loss=332.9767, lr=2.27e-04]
Epoch 20 TRAIN:  80%|███████▉  | 109/137 [00:25<00:05,  5.11it/s, epoch=20, grad_norm=179.963, loss=345.5403, lr=2.27e-04]
Epoch 20 TRAIN:  80%|███████▉  | 109/137 [00:25<00:05,  5.11it/s, epoch=20, grad_norm=139.927, loss=306.6904, lr=2.27e-04]
Epoch 20 TRAIN:  82%|████████▏ | 112/137 [00:25<00:05,  4.98it/s, epoch=20, grad_norm=139.927, loss=306.6904, lr=2.27e-04]
Epoch 20 TRAIN:  82%|████████▏ | 112/137 [00:26<00:05,  4.98it/s, epoch=20, grad_norm=216.136, loss=277.7516, lr=2.27e-04]
Epoch 20 TRAIN:  82%|████████▏ | 112/137 [00:26<00:05,  4.98it/s, epoch=20, grad_norm=150.796, loss=274.2070, lr=2.27e-04]
Epoch 20 TRAIN:  82%|████████▏ | 112/137 [00:26<00:05,  4.98it/s, epoch=20, grad_norm=156.583, loss=331.9239, lr=2.27e-04]
Epoch 20 TRAIN:  84%|████████▍ | 115/137 [00:26<00:04,  5.15it/s, epoch=20, grad_norm=156.583, loss=331.9239, lr=2.27e-04]
Epoch 20 TRAIN:  84%|████████▍ | 115/137 [00:26<00:04,  5.15it/s, epoch=20, grad_norm=194.806, loss=293.1382, lr=2.27e-04]
Epoch 20 TRAIN:  84%|████████▍ | 115/137 [00:26<00:04,  5.15it/s, epoch=20, grad_norm=116.671, loss=320.6335, lr=2.27e-04]
Epoch 20 TRAIN:  84%|████████▍ | 115/137 [00:27<00:04,  5.15it/s, epoch=20, grad_norm=137.486, loss=309.4446, lr=2.27e-04]
Epoch 20 TRAIN:  86%|████████▌ | 118/137 [00:27<00:03,  4.93it/s, epoch=20, grad_norm=137.486, loss=309.4446, lr=2.27e-04]
Epoch 20 TRAIN:  86%|████████▌ | 118/137 [00:27<00:03,  4.93it/s, epoch=20, grad_norm=247.923, loss=325.9246, lr=2.27e-04]
Epoch 20 TRAIN:  86%|████████▌ | 118/137 [00:27<00:03,  4.93it/s, epoch=20, grad_norm=136.124, loss=283.1248, lr=2.27e-04]
Epoch 20 TRAIN:  86%|████████▌ | 118/137 [00:27<00:03,  4.93it/s, epoch=20, grad_norm=340.684, loss=385.4352, lr=2.28e-04]
Epoch 20 TRAIN:  88%|████████▊ | 121/137 [00:27<00:03,  4.89it/s, epoch=20, grad_norm=340.684, loss=385.4352, lr=2.28e-04]
Epoch 20 TRAIN:  88%|████████▊ | 121/137 [00:27<00:03,  4.89it/s, epoch=20, grad_norm=343.012, loss=392.4870, lr=2.28e-04]
Epoch 20 TRAIN:  88%|████████▊ | 121/137 [00:28<00:03,  4.89it/s, epoch=20, grad_norm=252.017, loss=325.4272, lr=2.28e-04]
Epoch 20 TRAIN:  88%|████████▊ | 121/137 [00:28<00:03,  4.89it/s, epoch=20, grad_norm=366.322, loss=291.6064, lr=2.28e-04]
Epoch 20 TRAIN:  91%|█████████ | 124/137 [00:28<00:02,  4.99it/s, epoch=20, grad_norm=366.322, loss=291.6064, lr=2.28e-04]
Epoch 20 TRAIN:  91%|█████████ | 124/137 [00:28<00:02,  4.99it/s, epoch=20, grad_norm=143.489, loss=370.9590, lr=2.28e-04]
Epoch 20 TRAIN:  91%|█████████ | 124/137 [00:28<00:02,  4.99it/s, epoch=20, grad_norm=179.747, loss=351.9827, lr=2.28e-04]
Epoch 20 TRAIN:  91%|█████████ | 124/137 [00:28<00:02,  4.99it/s, epoch=20, grad_norm=128.552, loss=333.6553, lr=2.28e-04]
Epoch 20 TRAIN:  93%|█████████▎| 127/137 [00:28<00:02,  4.93it/s, epoch=20, grad_norm=128.552, loss=333.6553, lr=2.28e-04]
Epoch 20 TRAIN:  93%|█████████▎| 127/137 [00:29<00:02,  4.93it/s, epoch=20, grad_norm=126.395, loss=346.9072, lr=2.28e-04]
Epoch 20 TRAIN:  93%|█████████▎| 127/137 [00:29<00:02,  4.93it/s, epoch=20, grad_norm=131.923, loss=367.7287, lr=2.28e-04]
Epoch 20 TRAIN:  93%|█████████▎| 127/137 [00:29<00:02,  4.93it/s, epoch=20, grad_norm=162.482, loss=304.1606, lr=2.28e-04]
Epoch 20 TRAIN:  95%|█████████▍| 130/137 [00:29<00:01,  4.98it/s, epoch=20, grad_norm=162.482, loss=304.1606, lr=2.28e-04]
Epoch 20 TRAIN:  95%|█████████▍| 130/137 [00:29<00:01,  4.98it/s, epoch=20, grad_norm=134.512, loss=368.9845, lr=2.28e-04]
Epoch 20 TRAIN:  95%|█████████▍| 130/137 [00:29<00:01,  4.98it/s, epoch=20, grad_norm=121.108, loss=443.0696, lr=2.28e-04]
Epoch 20 TRAIN:  95%|█████████▍| 130/137 [00:30<00:01,  4.98it/s, epoch=20, grad_norm=262.539, loss=428.8931, lr=2.28e-04]
Epoch 20 TRAIN:  97%|█████████▋| 133/137 [00:30<00:00,  5.07it/s, epoch=20, grad_norm=262.539, loss=428.8931, lr=2.28e-04]
Epoch 20 TRAIN:  97%|█████████▋| 133/137 [00:30<00:00,  5.07it/s, epoch=20, grad_norm=218.271, loss=281.0306, lr=2.29e-04]
Epoch 20 TRAIN:  97%|█████████▋| 133/137 [00:30<00:00,  5.07it/s, epoch=20, grad_norm=140.875, loss=309.1802, lr=2.29e-04]
Epoch 20 TRAIN:  97%|█████████▋| 133/137 [00:30<00:00,  5.07it/s, epoch=20, grad_norm=260.451, loss=387.2028, lr=2.29e-04]
Epoch 20 TRAIN:  97%|█████████▋| 133/137 [00:30<00:00,  5.07it/s, epoch=20, grad_norm=230.981, loss=388.4598, lr=2.29e-04]
Epoch 20 TRAIN:  97%|█████████▋| 133/137 [00:30<00:00,  5.07it/s, epoch=20, grad_norm=196.706, loss=388.2050, lr=2.29e-04]
Epoch 20 TRAIN: 138it [00:30,  5.87it/s, epoch=20, grad_norm=196.706, loss=388.2050, lr=2.29e-04]                         
Epoch 20 TRAIN: 138it [00:30,  4.48it/s, epoch=20, grad_norm=196.706, loss=388.2050, lr=2.29e-04]

CV:   0%|          | 0/6 [00:00<?, ?it/s]/home/rookie/Documents/batuka/projects/wenet/wenet/wenet/utils/train_utils.py:689: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  amp_autocast(enabled=dtype is not None,

CV:  17%|█▋        | 1/6 [00:00<00:03,  1.29it/s]
                                                 
2026-04-16 20:02:09,870 INFO Epoch 20 Step 2861 CV info lr 2.2888e-04 cv_loss 396.9213340065696 rank 0 acc 0.2412357156475385
2026-04-16 20:02:09,871 INFO [Rank 0] Checkpoint: save to checkpoint exp/conformer_mongolian/epoch_20.pt
2026-04-16 20:02:09,986 INFO Epoch 21 Step 2861 TRAIN info lr 2.2888e-04 rank 0
2026-04-16 20:02:10,003 INFO using accumulate grad, new batch size is 1 times larger than before

Epoch 21 TRAIN:   0%|          | 0/138 [00:00<?, ?it/s]
Epoch 21 TRAIN:   0%|          | 0/138 [00:04<?, ?it/s, epoch=21, grad_norm=442.825, loss=385.5495, lr=2.29e-04]
Epoch 21 TRAIN:   1%|          | 1/138 [00:04<09:37,  4.22s/it, epoch=21, grad_norm=442.825, loss=385.5495, lr=2.29e-04]
Epoch 21 TRAIN:   1%|          | 1/138 [00:04<09:37,  4.22s/it, epoch=21, grad_norm=272.717, loss=233.4218, lr=2.29e-04]
Epoch 21 TRAIN:   1%|          | 1/138 [00:04<09:37,  4.22s/it, epoch=21, grad_norm=483.309, loss=390.4742, lr=2.29e-04]
Epoch 21 TRAIN:   1%|          | 1/138 [00:04<09:37,  4.22s/it, epoch=21, grad_norm=275.394, loss=293.3815, lr=2.29e-04]
Epoch 21 TRAIN:   1%|          | 1/138 [00:04<09:37,  4.22s/it, epoch=21, grad_norm=423.148, loss=412.9383, lr=2.29e-04]
Epoch 21 TRAIN:   1%|          | 1/138 [00:04<09:37,  4.22s/it, epoch=21, grad_norm=269.346, loss=308.6281, lr=2.29e-04]
Epoch 21 TRAIN:   1%|          | 1/138 [00:04<09:37,  4.22s/it, epoch=21, grad_norm=373.180, loss=420.4751, lr=2.29e-04]
Epoch 21 TRAIN:   1%|          | 1/138 [00:04<09:37,  4.22s/it, epoch=21, grad_norm=273.626, loss=336.1180, lr=2.30e-04]
Epoch 21 TRAIN:   1%|          | 1/138 [00:04<09:37,  4.22s/it, epoch=21, grad_norm=177.275, loss=351.8833, lr=2.30e-04]
Epoch 21 TRAIN:   7%|▋         | 9/138 [00:04<00:53,  2.43it/s, epoch=21, grad_norm=177.275, loss=351.8833, lr=2.30e-04]
Epoch 21 TRAIN:   7%|▋         | 9/138 [00:04<00:53,  2.43it/s, epoch=21, grad_norm=73.629, loss=315.5609, lr=2.30e-04] 
Epoch 21 TRAIN:   7%|▋         | 9/138 [00:05<00:53,  2.43it/s, epoch=21, grad_norm=84.730, loss=333.2592, lr=2.30e-04]
Epoch 21 TRAIN:   7%|▋         | 9/138 [00:05<00:53,  2.43it/s, epoch=21, grad_norm=132.057, loss=375.8647, lr=2.30e-04]
Epoch 21 TRAIN:   7%|▋         | 9/138 [00:05<00:53,  2.43it/s, epoch=21, grad_norm=126.514, loss=360.9242, lr=2.30e-04]
Epoch 21 TRAIN:   9%|▉         | 13/138 [00:05<00:39,  3.20it/s, epoch=21, grad_norm=126.514, loss=360.9242, lr=2.30e-04]
Epoch 21 TRAIN:   9%|▉         | 13/138 [00:05<00:39,  3.20it/s, epoch=21, grad_norm=222.376, loss=323.2664, lr=2.30e-04]
Epoch 21 TRAIN:   9%|▉         | 13/138 [00:05<00:39,  3.20it/s, epoch=21, grad_norm=123.056, loss=352.3462, lr=2.30e-04]
Epoch 21 TRAIN:   9%|▉         | 13/138 [00:05<00:39,  3.20it/s, epoch=21, grad_norm=147.765, loss=323.7246, lr=2.30e-04]
Epoch 21 TRAIN:   9%|▉         | 13/138 [00:06<00:39,  3.20it/s, epoch=21, grad_norm=119.890, loss=336.3308, lr=2.30e-04]
Epoch 21 TRAIN:  12%|█▏        | 17/138 [00:06<00:30,  3.92it/s, epoch=21, grad_norm=119.890, loss=336.3308, lr=2.30e-04]
Epoch 21 TRAIN:  12%|█▏        | 17/138 [00:06<00:30,  3.92it/s, epoch=21, grad_norm=129.696, loss=301.7715, lr=2.30e-04]
Epoch 21 TRAIN:  12%|█▏        | 17/138 [00:06<00:30,  3.92it/s, epoch=21, grad_norm=108.410, loss=339.0637, lr=2.30e-04]
Epoch 21 TRAIN:  12%|█▏        | 17/138 [00:06<00:30,  3.92it/s, epoch=21, grad_norm=78.989, loss=335.2583, lr=2.30e-04] 
Epoch 21 TRAIN:  12%|█▏        | 17/138 [00:06<00:30,  3.92it/s, epoch=21, grad_norm=69.222, loss=331.5056, lr=2.31e-04]
Epoch 21 TRAIN:  15%|█▌        | 21/138 [00:06<00:26,  4.48it/s, epoch=21, grad_norm=69.222, loss=331.5056, lr=2.31e-04]
Epoch 21 TRAIN:  15%|█▌        | 21/138 [00:06<00:26,  4.48it/s, epoch=21, grad_norm=109.671, loss=357.2101, lr=2.31e-04]
Epoch 21 TRAIN:  15%|█▌        | 21/138 [00:07<00:26,  4.48it/s, epoch=21, grad_norm=76.211, loss=327.4654, lr=2.31e-04] 
Epoch 21 TRAIN:  15%|█▌        | 21/138 [00:07<00:26,  4.48it/s, epoch=21, grad_norm=107.823, loss=324.7222, lr=2.31e-04]
Epoch 21 TRAIN:  15%|█▌        | 21/138 [00:07<00:26,  4.48it/s, epoch=21, grad_norm=98.437, loss=286.7874, lr=2.31e-04] 
Epoch 21 TRAIN:  18%|█▊        | 25/138 [00:07<00:23,  4.77it/s, epoch=21, grad_norm=98.437, loss=286.7874, lr=2.31e-04]
Epoch 21 TRAIN:  18%|█▊        | 25/138 [00:07<00:23,  4.77it/s, epoch=21, grad_norm=66.891, loss=305.9747, lr=2.31e-04]
Epoch 21 TRAIN:  18%|█▊        | 25/138 [00:07<00:23,  4.77it/s, epoch=21, grad_norm=90.431, loss=282.0640, lr=2.31e-04]
Epoch 21 TRAIN:  18%|█▊        | 25/138 [00:08<00:23,  4.77it/s, epoch=21, grad_norm=87.262, loss=322.9548, lr=2.31e-04]
Epoch 21 TRAIN:  18%|█▊        | 25/138 [00:08<00:23,  4.77it/s, epoch=21, grad_norm=72.987, loss=297.8694, lr=2.31e-04]
Epoch 21 TRAIN:  21%|██        | 29/138 [00:08<00:21,  5.03it/s, epoch=21, grad_norm=72.987, loss=297.8694, lr=2.31e-04]
Epoch 21 TRAIN:  21%|██        | 29/138 [00:08<00:21,  5.03it/s, epoch=21, grad_norm=113.860, loss=336.7398, lr=2.31e-04]
Epoch 21 TRAIN:  21%|██        | 29/138 [00:08<00:21,  5.03it/s, epoch=21, grad_norm=63.417, loss=307.2895, lr=2.31e-04] 
Epoch 21 TRAIN:  21%|██        | 29/138 [00:08<00:21,  5.03it/s, epoch=21, grad_norm=62.422, loss=297.8720, lr=2.31e-04]
Epoch 21 TRAIN:  21%|██        | 29/138 [00:08<00:21,  5.03it/s, epoch=21, grad_norm=78.287, loss=291.8211, lr=2.32e-04]
Epoch 21 TRAIN:  24%|██▍       | 33/138 [00:08<00:19,  5.27it/s, epoch=21, grad_norm=78.287, loss=291.8211, lr=2.32e-04]
Epoch 21 TRAIN:  24%|██▍       | 33/138 [00:09<00:19,  5.27it/s, epoch=21, grad_norm=71.365, loss=293.7845, lr=2.32e-04]
Epoch 21 TRAIN:  24%|██▍       | 33/138 [00:09<00:19,  5.27it/s, epoch=21, grad_norm=109.051, loss=344.6908, lr=2.32e-04]
Epoch 21 TRAIN:  24%|██▍       | 33/138 [00:09<00:19,  5.27it/s, epoch=21, grad_norm=119.473, loss=368.0175, lr=2.32e-04]
Epoch 21 TRAIN:  26%|██▌       | 36/138 [00:09<00:19,  5.26it/s, epoch=21, grad_norm=119.473, loss=368.0175, lr=2.32e-04]
Epoch 21 TRAIN:  26%|██▌       | 36/138 [00:09<00:19,  5.26it/s, epoch=21, grad_norm=94.316, loss=334.4763, lr=2.32e-04] 
Epoch 21 TRAIN:  26%|██▌       | 36/138 [00:09<00:19,  5.26it/s, epoch=21, grad_norm=67.507, loss=329.0101, lr=2.32e-04]
Epoch 21 TRAIN:  26%|██▌       | 36/138 [00:10<00:19,  5.26it/s, epoch=21, grad_norm=71.467, loss=351.0158, lr=2.32e-04]
Epoch 21 TRAIN:  28%|██▊       | 39/138 [00:10<00:19,  5.10it/s, epoch=21, grad_norm=71.467, loss=351.0158, lr=2.32e-04]
Epoch 21 TRAIN:  28%|██▊       | 39/138 [00:10<00:19,  5.10it/s, epoch=21, grad_norm=69.871, loss=350.6682, lr=2.32e-04]
Epoch 21 TRAIN:  28%|██▊       | 39/138 [00:10<00:19,  5.10it/s, epoch=21, grad_norm=120.372, loss=345.8673, lr=2.32e-04]
Epoch 21 TRAIN:  28%|██▊       | 39/138 [00:10<00:19,  5.10it/s, epoch=21, grad_norm=113.711, loss=345.4686, lr=2.32e-04]
Epoch 21 TRAIN:  30%|███       | 42/138 [00:10<00:18,  5.15it/s, epoch=21, grad_norm=113.711, loss=345.4686, lr=2.32e-04]
Epoch 21 TRAIN:  30%|███       | 42/138 [00:10<00:18,  5.15it/s, epoch=21, grad_norm=80.355, loss=364.8183, lr=2.32e-04] 
Epoch 21 TRAIN:  30%|███       | 42/138 [00:11<00:18,  5.15it/s, epoch=21, grad_norm=126.533, loss=339.1292, lr=2.32e-04]
Epoch 21 TRAIN:  30%|███       | 42/138 [00:11<00:18,  5.15it/s, epoch=21, grad_norm=86.112, loss=326.2104, lr=2.32e-04] 
Epoch 21 TRAIN:  33%|███▎      | 45/138 [00:11<00:17,  5.25it/s, epoch=21, grad_norm=86.112, loss=326.2104, lr=2.32e-04]
Epoch 21 TRAIN:  33%|███▎      | 45/138 [00:11<00:17,  5.25it/s, epoch=21, grad_norm=89.054, loss=310.4948, lr=2.33e-04]
Epoch 21 TRAIN:  33%|███▎      | 45/138 [00:11<00:17,  5.25it/s, epoch=21, grad_norm=92.608, loss=280.4746, lr=2.33e-04]
Epoch 21 TRAIN:  33%|███▎      | 45/138 [00:11<00:17,  5.25it/s, epoch=21, grad_norm=69.747, loss=302.6416, lr=2.33e-04]
Epoch 21 TRAIN:  35%|███▍      | 48/138 [00:11<00:17,  5.10it/s, epoch=21, grad_norm=69.747, loss=302.6416, lr=2.33e-04]
Epoch 21 TRAIN:  35%|███▍      | 48/138 [00:12<00:17,  5.10it/s, epoch=21, grad_norm=116.753, loss=328.9276, lr=2.33e-04]
Epoch 21 TRAIN:  35%|███▍      | 48/138 [00:12<00:17,  5.10it/s, epoch=21, grad_norm=53.410, loss=276.7958, lr=2.33e-04] 
Epoch 21 TRAIN:  35%|███▍      | 48/138 [00:12<00:17,  5.10it/s, epoch=21, grad_norm=141.557, loss=342.9805, lr=2.33e-04]
Epoch 21 TRAIN:  37%|███▋      | 51/138 [00:12<00:17,  5.07it/s, epoch=21, grad_norm=141.557, loss=342.9805, lr=2.33e-04]
Epoch 21 TRAIN:  37%|███▋      | 51/138 [00:12<00:17,  5.07it/s, epoch=21, grad_norm=94.572, loss=311.0975, lr=2.33e-04] 
Epoch 21 TRAIN:  37%|███▋      | 51/138 [00:12<00:17,  5.07it/s, epoch=21, grad_norm=86.486, loss=286.5248, lr=2.33e-04]
Epoch 21 TRAIN:  37%|███▋      | 51/138 [00:13<00:17,  5.07it/s, epoch=21, grad_norm=115.318, loss=349.7747, lr=2.33e-04]
Epoch 21 TRAIN:  39%|███▉      | 54/138 [00:13<00:16,  5.04it/s, epoch=21, grad_norm=115.318, loss=349.7747, lr=2.33e-04]
Epoch 21 TRAIN:  39%|███▉      | 54/138 [00:13<00:16,  5.04it/s, epoch=21, grad_norm=93.056, loss=329.8911, lr=2.33e-04] 
Epoch 21 TRAIN:  39%|███▉      | 54/138 [00:13<00:16,  5.04it/s, epoch=21, grad_norm=115.844, loss=341.2388, lr=2.33e-04]
Epoch 21 TRAIN:  39%|███▉      | 54/138 [00:13<00:16,  5.04it/s, epoch=21, grad_norm=194.270, loss=278.0411, lr=2.33e-04]
Epoch 21 TRAIN:  41%|████▏     | 57/138 [00:13<00:16,  5.04it/s, epoch=21, grad_norm=194.270, loss=278.0411, lr=2.33e-04]
Epoch 21 TRAIN:  41%|████▏     | 57/138 [00:13<00:16,  5.04it/s, epoch=21, grad_norm=98.253, loss=360.2238, lr=2.34e-04] 
Epoch 21 TRAIN:  41%|████▏     | 57/138 [00:13<00:16,  5.04it/s, epoch=21, grad_norm=166.449, loss=292.9142, lr=2.34e-04]
Epoch 21 TRAIN:  41%|████▏     | 57/138 [00:14<00:16,  5.04it/s, epoch=21, grad_norm=133.517, loss=295.5208, lr=2.34e-04]
Epoch 21 TRAIN:  43%|████▎     | 60/138 [00:14<00:15,  5.09it/s, epoch=21, grad_norm=133.517, loss=295.5208, lr=2.34e-04]
Epoch 21 TRAIN:  43%|████▎     | 60/138 [00:14<00:15,  5.09it/s, epoch=21, grad_norm=70.138, loss=359.0147, lr=2.34e-04] 
Epoch 21 TRAIN:  43%|████▎     | 60/138 [00:14<00:15,  5.09it/s, epoch=21, grad_norm=86.575, loss=371.6420, lr=2.34e-04]
Epoch 21 TRAIN:  43%|████▎     | 60/138 [00:14<00:15,  5.09it/s, epoch=21, grad_norm=102.884, loss=321.4600, lr=2.34e-04]
Epoch 21 TRAIN:  46%|████▌     | 63/138 [00:14<00:14,  5.25it/s, epoch=21, grad_norm=102.884, loss=321.4600, lr=2.34e-04]
Epoch 21 TRAIN:  46%|████▌     | 63/138 [00:15<00:14,  5.25it/s, epoch=21, grad_norm=126.619, loss=266.3265, lr=2.34e-04]
Epoch 21 TRAIN:  46%|████▌     | 63/138 [00:15<00:14,  5.25it/s, epoch=21, grad_norm=91.686, loss=342.8896, lr=2.34e-04] 
Epoch 21 TRAIN:  46%|████▌     | 63/138 [00:15<00:14,  5.25it/s, epoch=21, grad_norm=96.956, loss=282.0948, lr=2.34e-04]
Epoch 21 TRAIN:  48%|████▊     | 66/138 [00:15<00:14,  5.08it/s, epoch=21, grad_norm=96.956, loss=282.0948, lr=2.34e-04]
Epoch 21 TRAIN:  48%|████▊     | 66/138 [00:15<00:14,  5.08it/s, epoch=21, grad_norm=67.772, loss=292.9318, lr=2.34e-04]
Epoch 21 TRAIN:  48%|████▊     | 66/138 [00:15<00:14,  5.08it/s, epoch=21, grad_norm=107.998, loss=329.5700, lr=2.34e-04]
Epoch 21 TRAIN:  48%|████▊     | 66/138 [00:15<00:14,  5.08it/s, epoch=21, grad_norm=101.462, loss=301.0618, lr=2.34e-04]
Epoch 21 TRAIN:  50%|█████     | 69/138 [00:15<00:13,  4.97it/s, epoch=21, grad_norm=101.462, loss=301.0618, lr=2.34e-04]
Epoch 21 TRAIN:  50%|█████     | 69/138 [00:16<00:13,  4.97it/s, epoch=21, grad_norm=198.244, loss=366.3444, lr=2.34e-04]
Epoch 21 TRAIN:  50%|█████     | 69/138 [00:16<00:13,  4.97it/s, epoch=21, grad_norm=112.604, loss=328.0939, lr=2.35e-04]
Epoch 21 TRAIN:  50%|█████     | 69/138 [00:16<00:13,  4.97it/s, epoch=21, grad_norm=116.352, loss=307.8175, lr=2.35e-04]
Epoch 21 TRAIN:  52%|█████▏    | 72/138 [00:16<00:13,  4.82it/s, epoch=21, grad_norm=116.352, loss=307.8175, lr=2.35e-04]
Epoch 21 TRAIN:  52%|█████▏    | 72/138 [00:16<00:13,  4.82it/s, epoch=21, grad_norm=110.844, loss=273.9428, lr=2.35e-04]
Epoch 21 TRAIN:  52%|█████▏    | 72/138 [00:16<00:13,  4.82it/s, epoch=21, grad_norm=148.473, loss=243.3603, lr=2.35e-04]
Epoch 21 TRAIN:  52%|█████▏    | 72/138 [00:17<00:13,  4.82it/s, epoch=21, grad_norm=70.856, loss=312.6146, lr=2.35e-04] 
Epoch 21 TRAIN:  54%|█████▍    | 75/138 [00:17<00:13,  4.76it/s, epoch=21, grad_norm=70.856, loss=312.6146, lr=2.35e-04]
Epoch 21 TRAIN:  54%|█████▍    | 75/138 [00:17<00:13,  4.76it/s, epoch=21, grad_norm=160.480, loss=367.5966, lr=2.35e-04]
Epoch 21 TRAIN:  54%|█████▍    | 75/138 [00:17<00:13,  4.76it/s, epoch=21, grad_norm=66.329, loss=273.8958, lr=2.35e-04] 
Epoch 21 TRAIN:  54%|█████▍    | 75/138 [00:17<00:13,  4.76it/s, epoch=21, grad_norm=75.578, loss=272.0038, lr=2.35e-04]
Epoch 21 TRAIN:  57%|█████▋    | 78/138 [00:17<00:11,  5.04it/s, epoch=21, grad_norm=75.578, loss=272.0038, lr=2.35e-04]
Epoch 21 TRAIN:  57%|█████▋    | 78/138 [00:18<00:11,  5.04it/s, epoch=21, grad_norm=100.658, loss=304.3682, lr=2.35e-04]
Epoch 21 TRAIN:  57%|█████▋    | 78/138 [00:18<00:11,  5.04it/s, epoch=21, grad_norm=214.282, loss=414.2051, lr=2.35e-04]
Epoch 21 TRAIN:  57%|█████▋    | 78/138 [00:18<00:11,  5.04it/s, epoch=21, grad_norm=213.328, loss=238.4190, lr=2.35e-04]
Epoch 21 TRAIN:  59%|█████▊    | 81/138 [00:18<00:11,  4.93it/s, epoch=21, grad_norm=213.328, loss=238.4190, lr=2.35e-04]
Epoch 21 TRAIN:  59%|█████▊    | 81/138 [00:18<00:11,  4.93it/s, epoch=21, grad_norm=77.987, loss=338.2766, lr=2.35e-04] 
Epoch 21 TRAIN:  59%|█████▊    | 81/138 [00:18<00:11,  4.93it/s, epoch=21, grad_norm=129.651, loss=322.1798, lr=2.36e-04]
Epoch 21 TRAIN:  59%|█████▊    | 81/138 [00:19<00:11,  4.93it/s, epoch=21, grad_norm=97.855, loss=343.0461, lr=2.36e-04] 
Epoch 21 TRAIN:  61%|██████    | 84/138 [00:19<00:11,  4.89it/s, epoch=21, grad_norm=97.855, loss=343.0461, lr=2.36e-04]
Epoch 21 TRAIN:  61%|██████    | 84/138 [00:19<00:11,  4.89it/s, epoch=21, grad_norm=130.781, loss=287.9510, lr=2.36e-04]
Epoch 21 TRAIN:  61%|██████    | 84/138 [00:19<00:11,  4.89it/s, epoch=21, grad_norm=89.146, loss=362.5571, lr=2.36e-04] 
Epoch 21 TRAIN:  61%|██████    | 84/138 [00:19<00:11,  4.89it/s, epoch=21, grad_norm=203.148, loss=374.8446, lr=2.36e-04]
Epoch 21 TRAIN:  63%|██████▎   | 87/138 [00:19<00:10,  5.00it/s, epoch=21, grad_norm=203.148, loss=374.8446, lr=2.36e-04]
Epoch 21 TRAIN:  63%|██████▎   | 87/138 [00:19<00:10,  5.00it/s, epoch=21, grad_norm=76.329, loss=275.1840, lr=2.36e-04] 
Epoch 21 TRAIN:  63%|██████▎   | 87/138 [00:20<00:10,  5.00it/s, epoch=21, grad_norm=78.835, loss=374.5779, lr=2.36e-04]
Epoch 21 TRAIN:  63%|██████▎   | 87/138 [00:20<00:10,  5.00it/s, epoch=21, grad_norm=94.821, loss=371.3304, lr=2.36e-04]
Epoch 21 TRAIN:  65%|██████▌   | 90/138 [00:20<00:10,  4.74it/s, epoch=21, grad_norm=94.821, loss=371.3304, lr=2.36e-04]
Epoch 21 TRAIN:  65%|██████▌   | 90/138 [00:20<00:10,  4.74it/s, epoch=21, grad_norm=189.866, loss=304.5053, lr=2.36e-04]
Epoch 21 TRAIN:  65%|██████▌   | 90/138 [00:20<00:10,  4.74it/s, epoch=21, grad_norm=91.624, loss=311.7176, lr=2.36e-04] 
Epoch 21 TRAIN:  65%|██████▌   | 90/138 [00:20<00:10,  4.74it/s, epoch=21, grad_norm=142.665, loss=309.7880, lr=2.36e-04]
Epoch 21 TRAIN:  67%|██████▋   | 93/138 [00:20<00:09,  4.73it/s, epoch=21, grad_norm=142.665, loss=309.7880, lr=2.36e-04]
Epoch 21 TRAIN:  67%|██████▋   | 93/138 [00:21<00:09,  4.73it/s, epoch=21, grad_norm=176.827, loss=297.3033, lr=2.36e-04]
Epoch 21 TRAIN:  67%|██████▋   | 93/138 [00:21<00:09,  4.73it/s, epoch=21, grad_norm=240.872, loss=354.9688, lr=2.36e-04]
Epoch 21 TRAIN:  67%|██████▋   | 93/138 [00:21<00:09,  4.73it/s, epoch=21, grad_norm=146.196, loss=341.1577, lr=2.37e-04]
Epoch 21 TRAIN:  70%|██████▉   | 96/138 [00:21<00:08,  4.88it/s, epoch=21, grad_norm=146.196, loss=341.1577, lr=2.37e-04]
Epoch 21 TRAIN:  70%|██████▉   | 96/138 [00:21<00:08,  4.88it/s, epoch=21, grad_norm=72.870, loss=275.2560, lr=2.37e-04] 
Epoch 21 TRAIN:  70%|██████▉   | 96/138 [00:21<00:08,  4.88it/s, epoch=21, grad_norm=116.666, loss=244.6082, lr=2.37e-04]
Epoch 21 TRAIN:  70%|██████▉   | 96/138 [00:22<00:08,  4.88it/s, epoch=21, grad_norm=110.375, loss=319.2958, lr=2.37e-04]
Epoch 21 TRAIN:  72%|███████▏  | 99/138 [00:22<00:07,  5.03it/s, epoch=21, grad_norm=110.375, loss=319.2958, lr=2.37e-04]
Epoch 21 TRAIN:  72%|███████▏  | 99/138 [00:22<00:07,  5.03it/s, epoch=21, grad_norm=163.716, loss=299.9257, lr=2.37e-04]2026-04-16 20:02:32,309 DEBUG TRAIN | steps/sec 4.521| Batch 21/100 loss 299.925690 loss_att 267.148010 loss_ctc 376.406921 th_accuracy 0.279126 lr 2.3688e-04 grad_norm 163.715759 rank 0

Epoch 21 TRAIN:  72%|███████▏  | 99/138 [00:22<00:07,  5.03it/s, epoch=21, grad_norm=126.743, loss=284.6915, lr=2.37e-04]
Epoch 21 TRAIN:  72%|███████▏  | 99/138 [00:22<00:07,  5.03it/s, epoch=21, grad_norm=53.893, loss=247.7666, lr=2.37e-04] 
Epoch 21 TRAIN:  74%|███████▍  | 102/138 [00:22<00:07,  4.93it/s, epoch=21, grad_norm=53.893, loss=247.7666, lr=2.37e-04]
Epoch 21 TRAIN:  74%|███████▍  | 102/138 [00:22<00:07,  4.93it/s, epoch=21, grad_norm=219.940, loss=344.9313, lr=2.37e-04]
Epoch 21 TRAIN:  74%|███████▍  | 102/138 [00:23<00:07,  4.93it/s, epoch=21, grad_norm=93.602, loss=263.3111, lr=2.37e-04] 
Epoch 21 TRAIN:  74%|███████▍  | 102/138 [00:23<00:07,  4.93it/s, epoch=21, grad_norm=109.429, loss=258.1376, lr=2.37e-04]
Epoch 21 TRAIN:  76%|███████▌  | 105/138 [00:23<00:06,  4.94it/s, epoch=21, grad_norm=109.429, loss=258.1376, lr=2.37e-04]
Epoch 21 TRAIN:  76%|███████▌  | 105/138 [00:23<00:06,  4.94it/s, epoch=21, grad_norm=204.373, loss=362.2956, lr=2.37e-04]
Epoch 21 TRAIN:  76%|███████▌  | 105/138 [00:23<00:06,  4.94it/s, epoch=21, grad_norm=121.093, loss=412.4290, lr=2.37e-04]
Epoch 21 TRAIN:  76%|███████▌  | 105/138 [00:24<00:06,  4.94it/s, epoch=21, grad_norm=205.064, loss=301.8164, lr=2.38e-04]
Epoch 21 TRAIN:  78%|███████▊  | 108/138 [00:24<00:06,  4.82it/s, epoch=21, grad_norm=205.064, loss=301.8164, lr=2.38e-04]
Epoch 21 TRAIN:  78%|███████▊  | 108/138 [00:24<00:06,  4.82it/s, epoch=21, grad_norm=235.901, loss=243.9110, lr=2.38e-04]
Epoch 21 TRAIN:  78%|███████▊  | 108/138 [00:24<00:06,  4.82it/s, epoch=21, grad_norm=100.627, loss=339.7155, lr=2.38e-04]
Epoch 21 TRAIN:  78%|███████▊  | 108/138 [00:24<00:06,  4.82it/s, epoch=21, grad_norm=79.925, loss=313.4025, lr=2.38e-04] 
Epoch 21 TRAIN:  80%|████████  | 111/138 [00:24<00:05,  4.79it/s, epoch=21, grad_norm=79.925, loss=313.4025, lr=2.38e-04]
Epoch 21 TRAIN:  80%|████████  | 111/138 [00:24<00:05,  4.79it/s, epoch=21, grad_norm=134.996, loss=256.5305, lr=2.38e-04]
Epoch 21 TRAIN:  80%|████████  | 111/138 [00:25<00:05,  4.79it/s, epoch=21, grad_norm=102.014, loss=342.0436, lr=2.38e-04]
Epoch 21 TRAIN:  80%|████████  | 111/138 [00:25<00:05,  4.79it/s, epoch=21, grad_norm=101.034, loss=350.0363, lr=2.38e-04]
Epoch 21 TRAIN:  83%|████████▎ | 114/138 [00:25<00:05,  4.50it/s, epoch=21, grad_norm=101.034, loss=350.0363, lr=2.38e-04]
Epoch 21 TRAIN:  83%|████████▎ | 114/138 [00:25<00:05,  4.50it/s, epoch=21, grad_norm=173.561, loss=413.4119, lr=2.38e-04]
Epoch 21 TRAIN:  83%|████████▎ | 114/138 [00:25<00:05,  4.50it/s, epoch=21, grad_norm=140.719, loss=347.3695, lr=2.38e-04]
Epoch 21 TRAIN:  83%|████████▎ | 114/138 [00:25<00:05,  4.50it/s, epoch=21, grad_norm=187.440, loss=304.0884, lr=2.38e-04]
Epoch 21 TRAIN:  83%|████████▎ | 114/138 [00:26<00:05,  4.50it/s, epoch=21, grad_norm=104.823, loss=323.0214, lr=2.38e-04]
Epoch 21 TRAIN:  86%|████████▌ | 118/138 [00:26<00:04,  4.95it/s, epoch=21, grad_norm=104.823, loss=323.0214, lr=2.38e-04]
Epoch 21 TRAIN:  86%|████████▌ | 118/138 [00:26<00:04,  4.95it/s, epoch=21, grad_norm=134.314, loss=327.0686, lr=2.38e-04]
Epoch 21 TRAIN:  86%|████████▌ | 118/138 [00:26<00:04,  4.95it/s, epoch=21, grad_norm=98.191, loss=258.6671, lr=2.38e-04] 
Epoch 21 TRAIN:  86%|████████▌ | 118/138 [00:26<00:04,  4.95it/s, epoch=21, grad_norm=105.129, loss=294.3554, lr=2.39e-04]
Epoch 21 TRAIN:  88%|████████▊ | 121/138 [00:26<00:03,  4.92it/s, epoch=21, grad_norm=105.129, loss=294.3554, lr=2.39e-04]
Epoch 21 TRAIN:  88%|████████▊ | 121/138 [00:26<00:03,  4.92it/s, epoch=21, grad_norm=224.778, loss=421.4297, lr=2.39e-04]
Epoch 21 TRAIN:  88%|████████▊ | 121/138 [00:27<00:03,  4.92it/s, epoch=21, grad_norm=140.468, loss=316.8362, lr=2.39e-04]
Epoch 21 TRAIN:  88%|████████▊ | 121/138 [00:27<00:03,  4.92it/s, epoch=21, grad_norm=130.978, loss=316.9764, lr=2.39e-04]
Epoch 21 TRAIN:  90%|████████▉ | 124/138 [00:27<00:02,  5.10it/s, epoch=21, grad_norm=130.978, loss=316.9764, lr=2.39e-04]
Epoch 21 TRAIN:  90%|████████▉ | 124/138 [00:27<00:02,  5.10it/s, epoch=21, grad_norm=160.055, loss=363.5869, lr=2.39e-04]
Epoch 21 TRAIN:  90%|████████▉ | 124/138 [00:27<00:02,  5.10it/s, epoch=21, grad_norm=145.126, loss=217.6720, lr=2.39e-04]
Epoch 21 TRAIN:  90%|████████▉ | 124/138 [00:27<00:02,  5.10it/s, epoch=21, grad_norm=137.133, loss=303.9449, lr=2.39e-04]
Epoch 21 TRAIN:  92%|█████████▏| 127/138 [00:27<00:02,  4.96it/s, epoch=21, grad_norm=137.133, loss=303.9449, lr=2.39e-04]
Epoch 21 TRAIN:  92%|█████████▏| 127/138 [00:28<00:02,  4.96it/s, epoch=21, grad_norm=136.660, loss=344.4657, lr=2.39e-04]
Epoch 21 TRAIN:  92%|█████████▏| 127/138 [00:28<00:02,  4.96it/s, epoch=21, grad_norm=126.245, loss=353.4830, lr=2.39e-04]
Epoch 21 TRAIN:  92%|█████████▏| 127/138 [00:28<00:02,  4.96it/s, epoch=21, grad_norm=137.109, loss=403.8635, lr=2.39e-04]
Epoch 21 TRAIN:  94%|█████████▍| 130/138 [00:28<00:01,  4.93it/s, epoch=21, grad_norm=137.109, loss=403.8635, lr=2.39e-04]
Epoch 21 TRAIN:  94%|█████████▍| 130/138 [00:28<00:01,  4.93it/s, epoch=21, grad_norm=154.569, loss=430.0309, lr=2.39e-04]
Epoch 21 TRAIN:  94%|█████████▍| 130/138 [00:28<00:01,  4.93it/s, epoch=21, grad_norm=303.965, loss=333.3492, lr=2.39e-04]
Epoch 21 TRAIN:  94%|█████████▍| 130/138 [00:29<00:01,  4.93it/s, epoch=21, grad_norm=211.556, loss=299.4007, lr=2.40e-04]
Epoch 21 TRAIN:  96%|█████████▋| 133/138 [00:29<00:00,  5.02it/s, epoch=21, grad_norm=211.556, loss=299.4007, lr=2.40e-04]
Epoch 21 TRAIN:  96%|█████████▋| 133/138 [00:29<00:00,  5.02it/s, epoch=21, grad_norm=177.613, loss=378.6236, lr=2.40e-04]
Epoch 21 TRAIN:  96%|█████████▋| 133/138 [00:29<00:00,  5.02it/s, epoch=21, grad_norm=108.746, loss=357.9784, lr=2.40e-04]
Epoch 21 TRAIN:  96%|█████████▋| 133/138 [00:29<00:00,  5.02it/s, epoch=21, grad_norm=393.903, loss=239.0609, lr=2.40e-04]
Epoch 21 TRAIN:  99%|█████████▊| 136/138 [00:29<00:00,  4.61it/s, epoch=21, grad_norm=393.903, loss=239.0609, lr=2.40e-04]

CV:   0%|          | 0/6 [00:00<?, ?it/s]/home/rookie/Documents/batuka/projects/wenet/wenet/wenet/utils/train_utils.py:689: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  amp_autocast(enabled=dtype is not None,

CV:  17%|█▋        | 1/6 [00:00<00:03,  1.32it/s]
                                                 
2026-04-16 20:02:40,375 INFO Epoch 21 Step 2997 CV info lr 2.3976e-04 cv_loss 374.93737238103694 rank 0 acc 0.25305086622635525
2026-04-16 20:02:40,377 INFO [Rank 0] Checkpoint: save to checkpoint exp/conformer_mongolian/epoch_21.pt
2026-04-16 20:02:40,493 INFO Epoch 22 Step 2997 TRAIN info lr 2.3976e-04 rank 0
2026-04-16 20:02:40,514 INFO using accumulate grad, new batch size is 1 times larger than before

Epoch 22 TRAIN:   0%|          | 0/136 [00:00<?, ?it/s]
Epoch 22 TRAIN:   0%|          | 0/136 [00:03<?, ?it/s, epoch=22, grad_norm=508.119, loss=481.3121, lr=2.40e-04]
Epoch 22 TRAIN:   1%|          | 1/136 [00:03<08:52,  3.95s/it, epoch=22, grad_norm=508.119, loss=481.3121, lr=2.40e-04]
Epoch 22 TRAIN:   1%|          | 1/136 [00:04<08:52,  3.95s/it, epoch=22, grad_norm=441.709, loss=458.5062, lr=2.40e-04]
Epoch 22 TRAIN:   1%|          | 1/136 [00:04<08:52,  3.95s/it, epoch=22, grad_norm=315.832, loss=341.3924, lr=2.40e-04]
Epoch 22 TRAIN:   1%|          | 1/136 [00:04<08:52,  3.95s/it, epoch=22, grad_norm=358.834, loss=340.0115, lr=2.40e-04]
Epoch 22 TRAIN:   1%|          | 1/136 [00:04<08:52,  3.95s/it, epoch=22, grad_norm=289.942, loss=336.5120, lr=2.40e-04]
Epoch 22 TRAIN:   1%|          | 1/136 [00:04<08:52,  3.95s/it, epoch=22, grad_norm=380.151, loss=408.5930, lr=2.40e-04]
Epoch 22 TRAIN:   4%|▍         | 6/136 [00:04<01:14,  1.74it/s, epoch=22, grad_norm=380.151, loss=408.5930, lr=2.40e-04]
Epoch 22 TRAIN:   4%|▍         | 6/136 [00:04<01:14,  1.74it/s, epoch=22, grad_norm=126.019, loss=296.5756, lr=2.40e-04]
Epoch 22 TRAIN:   4%|▍         | 6/136 [00:04<01:14,  1.74it/s, epoch=22, grad_norm=193.503, loss=403.4039, lr=2.40e-04]
Epoch 22 TRAIN:   4%|▍         | 6/136 [00:04<01:14,  1.74it/s, epoch=22, grad_norm=68.354, loss=296.6472, lr=2.40e-04] 
Epoch 22 TRAIN:   4%|▍         | 6/136 [00:04<01:14,  1.74it/s, epoch=22, grad_norm=164.756, loss=317.1284, lr=2.41e-04]
Epoch 22 TRAIN:   4%|▍         | 6/136 [00:05<01:14,  1.74it/s, epoch=22, grad_norm=79.692, loss=338.4554, lr=2.41e-04] 
Epoch 22 TRAIN:   8%|▊         | 11/136 [00:05<00:40,  3.10it/s, epoch=22, grad_norm=79.692, loss=338.4554, lr=2.41e-04]
Epoch 22 TRAIN:   8%|▊         | 11/136 [00:05<00:40,  3.10it/s, epoch=22, grad_norm=111.963, loss=315.5561, lr=2.41e-04]
Epoch 22 TRAIN:   8%|▊         | 11/136 [00:05<00:40,  3.10it/s, epoch=22, grad_norm=73.454, loss=326.8808, lr=2.41e-04] 
Epoch 22 TRAIN:   8%|▊         | 11/136 [00:05<00:40,  3.10it/s, epoch=22, grad_norm=101.688, loss=313.0142, lr=2.41e-04]
Epoch 22 TRAIN:   8%|▊         | 11/136 [00:05<00:40,  3.10it/s, epoch=22, grad_norm=59.022, loss=335.5203, lr=2.41e-04] 
Epoch 22 TRAIN:  11%|█         | 15/136 [00:05<00:31,  3.86it/s, epoch=22, grad_norm=59.022, loss=335.5203, lr=2.41e-04]
Epoch 22 TRAIN:  11%|█         | 15/136 [00:05<00:31,  3.86it/s, epoch=22, grad_norm=64.090, loss=314.0324, lr=2.41e-04]
Epoch 22 TRAIN:  11%|█         | 15/136 [00:06<00:31,  3.86it/s, epoch=22, grad_norm=87.221, loss=329.0033, lr=2.41e-04]
Epoch 22 TRAIN:  11%|█         | 15/136 [00:06<00:31,  3.86it/s, epoch=22, grad_norm=116.714, loss=368.4679, lr=2.41e-04]
Epoch 22 TRAIN:  11%|█         | 15/136 [00:06<00:31,  3.86it/s, epoch=22, grad_norm=70.154, loss=315.7968, lr=2.41e-04] 
Epoch 22 TRAIN:  14%|█▍        | 19/136 [00:06<00:26,  4.47it/s, epoch=22, grad_norm=70.154, loss=315.7968, lr=2.41e-04]
Epoch 22 TRAIN:  14%|█▍        | 19/136 [00:06<00:26,  4.47it/s, epoch=22, grad_norm=81.198, loss=342.0677, lr=2.41e-04]
Epoch 22 TRAIN:  14%|█▍        | 19/136 [00:06<00:26,  4.47it/s, epoch=22, grad_norm=86.690, loss=368.0155, lr=2.41e-04]
Epoch 22 TRAIN:  14%|█▍        | 19/136 [00:06<00:26,  4.47it/s, epoch=22, grad_norm=117.792, loss=298.7685, lr=2.42e-04]
Epoch 22 TRAIN:  14%|█▍        | 19/136 [00:07<00:26,  4.47it/s, epoch=22, grad_norm=122.787, loss=311.1212, lr=2.42e-04]
Epoch 22 TRAIN:  17%|█▋        | 23/136 [00:07<00:23,  4.79it/s, epoch=22, grad_norm=122.787, loss=311.1212, lr=2.42e-04]
Epoch 22 TRAIN:  17%|█▋        | 23/136 [00:07<00:23,  4.79it/s, epoch=22, grad_norm=76.405, loss=326.8139, lr=2.42e-04] 
Epoch 22 TRAIN:  17%|█▋        | 23/136 [00:07<00:23,  4.79it/s, epoch=22, grad_norm=121.159, loss=305.6237, lr=2.42e-04]
Epoch 22 TRAIN:  17%|█▋        | 23/136 [00:07<00:23,  4.79it/s, epoch=22, grad_norm=94.865, loss=311.8592, lr=2.42e-04] 
Epoch 22 TRAIN:  17%|█▋        | 23/136 [00:07<00:23,  4.79it/s, epoch=22, grad_norm=60.146, loss=287.6582, lr=2.42e-04]
Epoch 22 TRAIN:  20%|█▉        | 27/136 [00:07<00:21,  5.05it/s, epoch=22, grad_norm=60.146, loss=287.6582, lr=2.42e-04]
Epoch 22 TRAIN:  20%|█▉        | 27/136 [00:07<00:21,  5.05it/s, epoch=22, grad_norm=134.994, loss=360.1644, lr=2.42e-04]
Epoch 22 TRAIN:  20%|█▉        | 27/136 [00:08<00:21,  5.05it/s, epoch=22, grad_norm=57.794, loss=278.1282, lr=2.42e-04] 
Epoch 22 TRAIN:  20%|█▉        | 27/136 [00:08<00:21,  5.05it/s, epoch=22, grad_norm=144.091, loss=325.4845, lr=2.42e-04]
Epoch 22 TRAIN:  20%|█▉        | 27/136 [00:08<00:21,  5.05it/s, epoch=22, grad_norm=147.213, loss=335.4028, lr=2.42e-04]
Epoch 22 TRAIN:  23%|██▎       | 31/136 [00:08<00:20,  5.17it/s, epoch=22, grad_norm=147.213, loss=335.4028, lr=2.42e-04]
Epoch 22 TRAIN:  23%|██▎       | 31/136 [00:08<00:20,  5.17it/s, epoch=22, grad_norm=134.197, loss=341.5529, lr=2.42e-04]
Epoch 22 TRAIN:  23%|██▎       | 31/136 [00:08<00:20,  5.17it/s, epoch=22, grad_norm=77.328, loss=258.3804, lr=2.42e-04] 
Epoch 22 TRAIN:  23%|██▎       | 31/136 [00:09<00:20,  5.17it/s, epoch=22, grad_norm=83.652, loss=308.5268, lr=2.42e-04]
Epoch 22 TRAIN:  25%|██▌       | 34/136 [00:09<00:19,  5.35it/s, epoch=22, grad_norm=83.652, loss=308.5268, lr=2.42e-04]
Epoch 22 TRAIN:  25%|██▌       | 34/136 [00:09<00:19,  5.35it/s, epoch=22, grad_norm=70.793, loss=303.0743, lr=2.43e-04]
Epoch 22 TRAIN:  25%|██▌       | 34/136 [00:09<00:19,  5.35it/s, epoch=22, grad_norm=65.252, loss=314.5011, lr=2.43e-04]
Epoch 22 TRAIN:  25%|██▌       | 34/136 [00:09<00:19,  5.35it/s, epoch=22, grad_norm=47.961, loss=315.9520, lr=2.43e-04]
Epoch 22 TRAIN:  27%|██▋       | 37/136 [00:09<00:19,  5.12it/s, epoch=22, grad_norm=47.961, loss=315.9520, lr=2.43e-04]
Epoch 22 TRAIN:  27%|██▋       | 37/136 [00:09<00:19,  5.12it/s, epoch=22, grad_norm=71.124, loss=307.9900, lr=2.43e-04]
Epoch 22 TRAIN:  27%|██▋       | 37/136 [00:10<00:19,  5.12it/s, epoch=22, grad_norm=162.890, loss=385.9109, lr=2.43e-04]
Epoch 22 TRAIN:  27%|██▋       | 37/136 [00:10<00:19,  5.12it/s, epoch=22, grad_norm=58.644, loss=340.1277, lr=2.43e-04] 
Epoch 22 TRAIN:  29%|██▉       | 40/136 [00:10<00:18,  5.13it/s, epoch=22, grad_norm=58.644, loss=340.1277, lr=2.43e-04]
Epoch 22 TRAIN:  29%|██▉       | 40/136 [00:10<00:18,  5.13it/s, epoch=22, grad_norm=93.539, loss=350.2376, lr=2.43e-04]
Epoch 22 TRAIN:  29%|██▉       | 40/136 [00:10<00:18,  5.13it/s, epoch=22, grad_norm=77.846, loss=323.6822, lr=2.43e-04]
Epoch 22 TRAIN:  29%|██▉       | 40/136 [00:10<00:18,  5.13it/s, epoch=22, grad_norm=163.085, loss=266.5384, lr=2.43e-04]
Epoch 22 TRAIN:  32%|███▏      | 43/136 [00:10<00:18,  5.08it/s, epoch=22, grad_norm=163.085, loss=266.5384, lr=2.43e-04]
Epoch 22 TRAIN:  32%|███▏      | 43/136 [00:11<00:18,  5.08it/s, epoch=22, grad_norm=118.514, loss=392.9620, lr=2.43e-04]
Epoch 22 TRAIN:  32%|███▏      | 43/136 [00:11<00:18,  5.08it/s, epoch=22, grad_norm=73.111, loss=331.4088, lr=2.43e-04] 
Epoch 22 TRAIN:  32%|███▏      | 43/136 [00:11<00:18,  5.08it/s, epoch=22, grad_norm=62.640, loss=307.7711, lr=2.43e-04]
Epoch 22 TRAIN:  34%|███▍      | 46/136 [00:11<00:17,  5.07it/s, epoch=22, grad_norm=62.640, loss=307.7711, lr=2.43e-04]
Epoch 22 TRAIN:  34%|███▍      | 46/136 [00:11<00:17,  5.07it/s, epoch=22, grad_norm=159.122, loss=377.8549, lr=2.44e-04]
Epoch 22 TRAIN:  34%|███▍      | 46/136 [00:11<00:17,  5.07it/s, epoch=22, grad_norm=84.626, loss=340.0577, lr=2.44e-04] 
Epoch 22 TRAIN:  34%|███▍      | 46/136 [00:11<00:17,  5.07it/s, epoch=22, grad_norm=78.412, loss=305.3567, lr=2.44e-04]
Epoch 22 TRAIN:  36%|███▌      | 49/136 [00:11<00:16,  5.18it/s, epoch=22, grad_norm=78.412, loss=305.3567, lr=2.44e-04]
Epoch 22 TRAIN:  36%|███▌      | 49/136 [00:12<00:16,  5.18it/s, epoch=22, grad_norm=64.530, loss=342.3161, lr=2.44e-04]
Epoch 22 TRAIN:  36%|███▌      | 49/136 [00:12<00:16,  5.18it/s, epoch=22, grad_norm=82.552, loss=319.1404, lr=2.44e-04]
Epoch 22 TRAIN:  36%|███▌      | 49/136 [00:12<00:16,  5.18it/s, epoch=22, grad_norm=81.684, loss=348.3304, lr=2.44e-04]
Epoch 22 TRAIN:  38%|███▊      | 52/136 [00:12<00:17,  4.87it/s, epoch=22, grad_norm=81.684, loss=348.3304, lr=2.44e-04]
Epoch 22 TRAIN:  38%|███▊      | 52/136 [00:12<00:17,  4.87it/s, epoch=22, grad_norm=61.239, loss=323.2493, lr=2.44e-04]
Epoch 22 TRAIN:  38%|███▊      | 52/136 [00:13<00:17,  4.87it/s, epoch=22, grad_norm=90.949, loss=286.3206, lr=2.44e-04]
Epoch 22 TRAIN:  38%|███▊      | 52/136 [00:13<00:17,  4.87it/s, epoch=22, grad_norm=53.654, loss=318.3162, lr=2.44e-04]
Epoch 22 TRAIN:  40%|████      | 55/136 [00:13<00:17,  4.71it/s, epoch=22, grad_norm=53.654, loss=318.3162, lr=2.44e-04]
Epoch 22 TRAIN:  40%|████      | 55/136 [00:13<00:17,  4.71it/s, epoch=22, grad_norm=91.165, loss=302.8325, lr=2.44e-04]
Epoch 22 TRAIN:  40%|████      | 55/136 [00:13<00:17,  4.71it/s, epoch=22, grad_norm=86.979, loss=305.5752, lr=2.44e-04]
Epoch 22 TRAIN:  40%|████      | 55/136 [00:13<00:17,  4.71it/s, epoch=22, grad_norm=123.228, loss=328.2509, lr=2.44e-04]
Epoch 22 TRAIN:  43%|████▎     | 58/136 [00:13<00:15,  4.91it/s, epoch=22, grad_norm=123.228, loss=328.2509, lr=2.44e-04]
Epoch 22 TRAIN:  43%|████▎     | 58/136 [00:14<00:15,  4.91it/s, epoch=22, grad_norm=74.881, loss=319.0152, lr=2.44e-04] 
Epoch 22 TRAIN:  43%|████▎     | 58/136 [00:14<00:15,  4.91it/s, epoch=22, grad_norm=197.261, loss=396.6262, lr=2.45e-04]
Epoch 22 TRAIN:  43%|████▎     | 58/136 [00:14<00:15,  4.91it/s, epoch=22, grad_norm=87.347, loss=329.1432, lr=2.45e-04] 
Epoch 22 TRAIN:  45%|████▍     | 61/136 [00:14<00:15,  4.92it/s, epoch=22, grad_norm=87.347, loss=329.1432, lr=2.45e-04]
Epoch 22 TRAIN:  45%|████▍     | 61/136 [00:14<00:15,  4.92it/s, epoch=22, grad_norm=102.512, loss=339.7382, lr=2.45e-04]
Epoch 22 TRAIN:  45%|████▍     | 61/136 [00:14<00:15,  4.92it/s, epoch=22, grad_norm=68.899, loss=308.2648, lr=2.45e-04] 
Epoch 22 TRAIN:  45%|████▍     | 61/136 [00:15<00:15,  4.92it/s, epoch=22, grad_norm=125.176, loss=309.2410, lr=2.45e-04]
Epoch 22 TRAIN:  47%|████▋     | 64/136 [00:15<00:14,  5.02it/s, epoch=22, grad_norm=125.176, loss=309.2410, lr=2.45e-04]
Epoch 22 TRAIN:  47%|████▋     | 64/136 [00:15<00:14,  5.02it/s, epoch=22, grad_norm=211.188, loss=257.9848, lr=2.45e-04]
Epoch 22 TRAIN:  47%|████▋     | 64/136 [00:15<00:14,  5.02it/s, epoch=22, grad_norm=102.055, loss=371.7888, lr=2.45e-04]
Epoch 22 TRAIN:  47%|████▋     | 64/136 [00:15<00:14,  5.02it/s, epoch=22, grad_norm=146.923, loss=307.6476, lr=2.45e-04]
Epoch 22 TRAIN:  49%|████▉     | 67/136 [00:15<00:13,  5.07it/s, epoch=22, grad_norm=146.923, loss=307.6476, lr=2.45e-04]
Epoch 22 TRAIN:  49%|████▉     | 67/136 [00:15<00:13,  5.07it/s, epoch=22, grad_norm=82.275, loss=334.8562, lr=2.45e-04] 
Epoch 22 TRAIN:  49%|████▉     | 67/136 [00:16<00:13,  5.07it/s, epoch=22, grad_norm=117.017, loss=313.0661, lr=2.45e-04]
Epoch 22 TRAIN:  49%|████▉     | 67/136 [00:16<00:13,  5.07it/s, epoch=22, grad_norm=82.307, loss=295.3041, lr=2.45e-04] 
Epoch 22 TRAIN:  51%|█████▏    | 70/136 [00:16<00:13,  4.99it/s, epoch=22, grad_norm=82.307, loss=295.3041, lr=2.45e-04]
Epoch 22 TRAIN:  51%|█████▏    | 70/136 [00:16<00:13,  4.99it/s, epoch=22, grad_norm=100.419, loss=339.5839, lr=2.45e-04]
Epoch 22 TRAIN:  51%|█████▏    | 70/136 [00:16<00:13,  4.99it/s, epoch=22, grad_norm=122.425, loss=348.9751, lr=2.46e-04]
Epoch 22 TRAIN:  51%|█████▏    | 70/136 [00:16<00:13,  4.99it/s, epoch=22, grad_norm=67.474, loss=332.4238, lr=2.46e-04] 
Epoch 22 TRAIN:  54%|█████▎    | 73/136 [00:16<00:12,  5.04it/s, epoch=22, grad_norm=67.474, loss=332.4238, lr=2.46e-04]
Epoch 22 TRAIN:  54%|█████▎    | 73/136 [00:17<00:12,  5.04it/s, epoch=22, grad_norm=115.902, loss=333.3689, lr=2.46e-04]
Epoch 22 TRAIN:  54%|█████▎    | 73/136 [00:17<00:12,  5.04it/s, epoch=22, grad_norm=89.728, loss=276.1470, lr=2.46e-04] 
Epoch 22 TRAIN:  54%|█████▎    | 73/136 [00:17<00:12,  5.04it/s, epoch=22, grad_norm=95.121, loss=343.3321, lr=2.46e-04]
Epoch 22 TRAIN:  56%|█████▌    | 76/136 [00:17<00:11,  5.15it/s, epoch=22, grad_norm=95.121, loss=343.3321, lr=2.46e-04]
Epoch 22 TRAIN:  56%|█████▌    | 76/136 [00:17<00:11,  5.15it/s, epoch=22, grad_norm=119.226, loss=340.9442, lr=2.46e-04]
Epoch 22 TRAIN:  56%|█████▌    | 76/136 [00:17<00:11,  5.15it/s, epoch=22, grad_norm=75.352, loss=281.3029, lr=2.46e-04] 
Epoch 22 TRAIN:  56%|█████▌    | 76/136 [00:18<00:11,  5.15it/s, epoch=22, grad_norm=90.275, loss=323.2464, lr=2.46e-04]
Epoch 22 TRAIN:  58%|█████▊    | 79/136 [00:18<00:11,  4.96it/s, epoch=22, grad_norm=90.275, loss=323.2464, lr=2.46e-04]
Epoch 22 TRAIN:  58%|█████▊    | 79/136 [00:18<00:11,  4.96it/s, epoch=22, grad_norm=92.497, loss=268.5287, lr=2.46e-04]
Epoch 22 TRAIN:  58%|█████▊    | 79/136 [00:18<00:11,  4.96it/s, epoch=22, grad_norm=79.192, loss=354.5541, lr=2.46e-04]
Epoch 22 TRAIN:  58%|█████▊    | 79/136 [00:18<00:11,  4.96it/s, epoch=22, grad_norm=84.370, loss=302.5444, lr=2.46e-04]
Epoch 22 TRAIN:  60%|██████    | 82/136 [00:18<00:10,  5.04it/s, epoch=22, grad_norm=84.370, loss=302.5444, lr=2.46e-04]
Epoch 22 TRAIN:  60%|██████    | 82/136 [00:18<00:10,  5.04it/s, epoch=22, grad_norm=89.534, loss=354.1474, lr=2.46e-04]
Epoch 22 TRAIN:  60%|██████    | 82/136 [00:19<00:10,  5.04it/s, epoch=22, grad_norm=78.907, loss=307.2823, lr=2.46e-04]
Epoch 22 TRAIN:  60%|██████    | 82/136 [00:19<00:10,  5.04it/s, epoch=22, grad_norm=83.586, loss=327.0910, lr=2.47e-04]
Epoch 22 TRAIN:  62%|██████▎   | 85/136 [00:19<00:10,  5.05it/s, epoch=22, grad_norm=83.586, loss=327.0910, lr=2.47e-04]
Epoch 22 TRAIN:  62%|██████▎   | 85/136 [00:19<00:10,  5.05it/s, epoch=22, grad_norm=77.355, loss=300.1945, lr=2.47e-04]
Epoch 22 TRAIN:  62%|██████▎   | 85/136 [00:19<00:10,  5.05it/s, epoch=22, grad_norm=82.555, loss=329.9755, lr=2.47e-04]
Epoch 22 TRAIN:  62%|██████▎   | 85/136 [00:19<00:10,  5.05it/s, epoch=22, grad_norm=190.896, loss=274.3774, lr=2.47e-04]
Epoch 22 TRAIN:  65%|██████▍   | 88/136 [00:19<00:09,  4.96it/s, epoch=22, grad_norm=190.896, loss=274.3774, lr=2.47e-04]
Epoch 22 TRAIN:  65%|██████▍   | 88/136 [00:20<00:09,  4.96it/s, epoch=22, grad_norm=487.837, loss=260.5591, lr=2.47e-04]
Epoch 22 TRAIN:  65%|██████▍   | 88/136 [00:20<00:09,  4.96it/s, epoch=22, grad_norm=102.084, loss=282.4842, lr=2.47e-04]
Epoch 22 TRAIN:  65%|██████▍   | 88/136 [00:20<00:09,  4.96it/s, epoch=22, grad_norm=119.369, loss=242.9258, lr=2.47e-04]
Epoch 22 TRAIN:  67%|██████▋   | 91/136 [00:20<00:09,  4.90it/s, epoch=22, grad_norm=119.369, loss=242.9258, lr=2.47e-04]
Epoch 22 TRAIN:  67%|██████▋   | 91/136 [00:20<00:09,  4.90it/s, epoch=22, grad_norm=71.038, loss=260.2814, lr=2.47e-04] 
Epoch 22 TRAIN:  67%|██████▋   | 91/136 [00:20<00:09,  4.90it/s, epoch=22, grad_norm=159.276, loss=313.9173, lr=2.47e-04]
Epoch 22 TRAIN:  67%|██████▋   | 91/136 [00:21<00:09,  4.90it/s, epoch=22, grad_norm=370.283, loss=448.0896, lr=2.47e-04]
Epoch 22 TRAIN:  69%|██████▉   | 94/136 [00:21<00:08,  4.91it/s, epoch=22, grad_norm=370.283, loss=448.0896, lr=2.47e-04]
Epoch 22 TRAIN:  69%|██████▉   | 94/136 [00:21<00:08,  4.91it/s, epoch=22, grad_norm=169.986, loss=288.7761, lr=2.47e-04]
Epoch 22 TRAIN:  69%|██████▉   | 94/136 [00:21<00:08,  4.91it/s, epoch=22, grad_norm=68.300, loss=256.4296, lr=2.47e-04] 
Epoch 22 TRAIN:  69%|██████▉   | 94/136 [00:21<00:08,  4.91it/s, epoch=22, grad_norm=233.826, loss=402.0632, lr=2.48e-04]
Epoch 22 TRAIN:  71%|███████▏  | 97/136 [00:21<00:07,  4.96it/s, epoch=22, grad_norm=233.826, loss=402.0632, lr=2.48e-04]
Epoch 22 TRAIN:  71%|███████▏  | 97/136 [00:21<00:07,  4.96it/s, epoch=22, grad_norm=120.597, loss=315.6720, lr=2.48e-04]
Epoch 22 TRAIN:  71%|███████▏  | 97/136 [00:22<00:07,  4.96it/s, epoch=22, grad_norm=177.378, loss=181.5862, lr=2.48e-04]
Epoch 22 TRAIN:  71%|███████▏  | 97/136 [00:22<00:07,  4.96it/s, epoch=22, grad_norm=101.317, loss=299.0027, lr=2.48e-04]2026-04-16 20:03:02,890 DEBUG TRAIN | steps/sec 4.447| Batch 22/100 loss 299.002716 loss_att 270.069031 loss_ctc 366.514648 th_accuracy 0.262211 lr 2.4776e-04 grad_norm 101.316994 rank 0

Epoch 22 TRAIN:  74%|███████▎  | 100/136 [00:22<00:07,  4.85it/s, epoch=22, grad_norm=101.317, loss=299.0027, lr=2.48e-04]
Epoch 22 TRAIN:  74%|███████▎  | 100/136 [00:22<00:07,  4.85it/s, epoch=22, grad_norm=95.912, loss=292.3412, lr=2.48e-04] 
Epoch 22 TRAIN:  74%|███████▎  | 100/136 [00:22<00:07,  4.85it/s, epoch=22, grad_norm=126.236, loss=358.0752, lr=2.48e-04]
Epoch 22 TRAIN:  74%|███████▎  | 100/136 [00:22<00:07,  4.85it/s, epoch=22, grad_norm=234.181, loss=389.7275, lr=2.48e-04]
Epoch 22 TRAIN:  76%|███████▌  | 103/136 [00:22<00:06,  5.05it/s, epoch=22, grad_norm=234.181, loss=389.7275, lr=2.48e-04]
Epoch 22 TRAIN:  76%|███████▌  | 103/136 [00:23<00:06,  5.05it/s, epoch=22, grad_norm=122.801, loss=288.9580, lr=2.48e-04]
Epoch 22 TRAIN:  76%|███████▌  | 103/136 [00:23<00:06,  5.05it/s, epoch=22, grad_norm=100.542, loss=300.3859, lr=2.48e-04]
Epoch 22 TRAIN:  76%|███████▌  | 103/136 [00:23<00:06,  5.05it/s, epoch=22, grad_norm=271.055, loss=189.6787, lr=2.48e-04]
Epoch 22 TRAIN:  78%|███████▊  | 106/136 [00:23<00:05,  5.05it/s, epoch=22, grad_norm=271.055, loss=189.6787, lr=2.48e-04]
Epoch 22 TRAIN:  78%|███████▊  | 106/136 [00:23<00:05,  5.05it/s, epoch=22, grad_norm=147.526, loss=262.3408, lr=2.48e-04]
Epoch 22 TRAIN:  78%|███████▊  | 106/136 [00:23<00:05,  5.05it/s, epoch=22, grad_norm=98.445, loss=326.1172, lr=2.48e-04] 
Epoch 22 TRAIN:  78%|███████▊  | 106/136 [00:24<00:05,  5.05it/s, epoch=22, grad_norm=99.440, loss=297.2371, lr=2.48e-04]
Epoch 22 TRAIN:  80%|████████  | 109/136 [00:24<00:05,  5.04it/s, epoch=22, grad_norm=99.440, loss=297.2371, lr=2.48e-04]
Epoch 22 TRAIN:  80%|████████  | 109/136 [00:24<00:05,  5.04it/s, epoch=22, grad_norm=115.218, loss=306.3207, lr=2.49e-04]
Epoch 22 TRAIN:  80%|████████  | 109/136 [00:24<00:05,  5.04it/s, epoch=22, grad_norm=56.645, loss=227.4925, lr=2.49e-04] 
Epoch 22 TRAIN:  80%|████████  | 109/136 [00:24<00:05,  5.04it/s, epoch=22, grad_norm=89.717, loss=278.8374, lr=2.49e-04]
Epoch 22 TRAIN:  82%|████████▏ | 112/136 [00:24<00:05,  4.80it/s, epoch=22, grad_norm=89.717, loss=278.8374, lr=2.49e-04]
Epoch 22 TRAIN:  82%|████████▏ | 112/136 [00:24<00:05,  4.80it/s, epoch=22, grad_norm=110.848, loss=388.2809, lr=2.49e-04]
Epoch 22 TRAIN:  82%|████████▏ | 112/136 [00:25<00:05,  4.80it/s, epoch=22, grad_norm=92.327, loss=292.3688, lr=2.49e-04] 
Epoch 22 TRAIN:  82%|████████▏ | 112/136 [00:25<00:05,  4.80it/s, epoch=22, grad_norm=114.007, loss=367.5592, lr=2.49e-04]
Epoch 22 TRAIN:  85%|████████▍ | 115/136 [00:25<00:04,  4.89it/s, epoch=22, grad_norm=114.007, loss=367.5592, lr=2.49e-04]
Epoch 22 TRAIN:  85%|████████▍ | 115/136 [00:25<00:04,  4.89it/s, epoch=22, grad_norm=75.136, loss=293.6486, lr=2.49e-04] 
Epoch 22 TRAIN:  85%|████████▍ | 115/136 [00:25<00:04,  4.89it/s, epoch=22, grad_norm=104.415, loss=382.9427, lr=2.49e-04]
Epoch 22 TRAIN:  85%|████████▍ | 115/136 [00:25<00:04,  4.89it/s, epoch=22, grad_norm=145.062, loss=320.7043, lr=2.49e-04]
Epoch 22 TRAIN:  87%|████████▋ | 118/136 [00:25<00:03,  4.95it/s, epoch=22, grad_norm=145.062, loss=320.7043, lr=2.49e-04]
Epoch 22 TRAIN:  87%|████████▋ | 118/136 [00:26<00:03,  4.95it/s, epoch=22, grad_norm=119.994, loss=381.5112, lr=2.49e-04]
Epoch 22 TRAIN:  87%|████████▋ | 118/136 [00:26<00:03,  4.95it/s, epoch=22, grad_norm=153.948, loss=307.7040, lr=2.49e-04]
Epoch 22 TRAIN:  87%|████████▋ | 118/136 [00:26<00:03,  4.95it/s, epoch=22, grad_norm=289.178, loss=329.8105, lr=2.49e-04]
Epoch 22 TRAIN:  89%|████████▉ | 121/136 [00:26<00:03,  4.75it/s, epoch=22, grad_norm=289.178, loss=329.8105, lr=2.49e-04]
Epoch 22 TRAIN:  89%|████████▉ | 121/136 [00:26<00:03,  4.75it/s, epoch=22, grad_norm=116.967, loss=350.3773, lr=2.50e-04]
Epoch 22 TRAIN:  89%|████████▉ | 121/136 [00:27<00:03,  4.75it/s, epoch=22, grad_norm=127.174, loss=368.8431, lr=2.50e-04]
Epoch 22 TRAIN:  89%|████████▉ | 121/136 [00:27<00:03,  4.75it/s, epoch=22, grad_norm=151.020, loss=289.3335, lr=2.50e-04]
Epoch 22 TRAIN:  91%|█████████ | 124/136 [00:27<00:02,  4.83it/s, epoch=22, grad_norm=151.020, loss=289.3335, lr=2.50e-04]
Epoch 22 TRAIN:  91%|█████████ | 124/136 [00:27<00:02,  4.83it/s, epoch=22, grad_norm=232.563, loss=358.8600, lr=2.50e-04]
Epoch 22 TRAIN:  91%|█████████ | 124/136 [00:27<00:02,  4.83it/s, epoch=22, grad_norm=168.081, loss=412.6845, lr=2.50e-04]
Epoch 22 TRAIN:  91%|█████████ | 124/136 [00:27<00:02,  4.83it/s, epoch=22, grad_norm=245.162, loss=377.1368, lr=2.50e-04]
Epoch 22 TRAIN:  91%|█████████ | 124/136 [00:27<00:02,  4.83it/s, epoch=22, grad_norm=90.282, loss=282.1381, lr=2.50e-04] 
Epoch 22 TRAIN:  94%|█████████▍| 128/136 [00:27<00:01,  5.35it/s, epoch=22, grad_norm=90.282, loss=282.1381, lr=2.50e-04]
Epoch 22 TRAIN:  94%|█████████▍| 128/136 [00:28<00:01,  5.35it/s, epoch=22, grad_norm=133.859, loss=350.9781, lr=2.50e-04]
Epoch 22 TRAIN:  94%|█████████▍| 128/136 [00:28<00:01,  5.35it/s, epoch=22, grad_norm=177.092, loss=400.4637, lr=2.50e-04]
Epoch 22 TRAIN:  94%|█████████▍| 128/136 [00:28<00:01,  5.35it/s, epoch=22, grad_norm=159.120, loss=241.7893, lr=2.50e-04]
Epoch 22 TRAIN:  96%|█████████▋| 131/136 [00:28<00:00,  5.17it/s, epoch=22, grad_norm=159.120, loss=241.7893, lr=2.50e-04]
Epoch 22 TRAIN:  96%|█████████▋| 131/136 [00:28<00:00,  5.17it/s, epoch=22, grad_norm=98.859, loss=294.5790, lr=2.50e-04] 
Epoch 22 TRAIN:  96%|█████████▋| 131/136 [00:28<00:00,  5.17it/s, epoch=22, grad_norm=103.264, loss=369.0101, lr=2.50e-04]
Epoch 22 TRAIN:  96%|█████████▋| 131/136 [00:29<00:00,  5.17it/s, epoch=22, grad_norm=105.340, loss=376.5156, lr=2.50e-04]
Epoch 22 TRAIN:  99%|█████████▊| 134/136 [00:29<00:00,  4.99it/s, epoch=22, grad_norm=105.340, loss=376.5156, lr=2.50e-04]
Epoch 22 TRAIN:  99%|█████████▊| 134/136 [00:29<00:00,  4.99it/s, epoch=22, grad_norm=192.435, loss=352.8804, lr=2.51e-04]
Epoch 22 TRAIN:  99%|█████████▊| 134/136 [00:29<00:00,  4.99it/s, epoch=22, grad_norm=97.913, loss=343.4068, lr=2.51e-04] 
Epoch 22 TRAIN: 100%|██████████| 136/136 [00:29<00:00,  4.62it/s, epoch=22, grad_norm=97.913, loss=343.4068, lr=2.51e-04]

CV:   0%|          | 0/6 [00:00<?, ?it/s]/home/rookie/Documents/batuka/projects/wenet/wenet/wenet/utils/train_utils.py:689: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  amp_autocast(enabled=dtype is not None,

CV:  17%|█▋        | 1/6 [00:00<00:03,  1.32it/s]
                                                 
2026-04-16 20:03:10,807 INFO Epoch 22 Step 3133 CV info lr 2.5064e-04 cv_loss 393.3608994917436 rank 0 acc 0.25014213224252063
2026-04-16 20:03:10,808 INFO [Rank 0] Checkpoint: save to checkpoint exp/conformer_mongolian/epoch_22.pt
2026-04-16 20:03:10,920 INFO Epoch 23 Step 3133 TRAIN info lr 2.5064e-04 rank 0
2026-04-16 20:03:10,942 INFO using accumulate grad, new batch size is 1 times larger than before

Epoch 23 TRAIN:   0%|          | 0/136 [00:00<?, ?it/s]
Epoch 23 TRAIN:   0%|          | 0/136 [00:04<?, ?it/s, epoch=23, grad_norm=358.928, loss=358.7519, lr=2.51e-04]
Epoch 23 TRAIN:   1%|          | 1/136 [00:04<09:05,  4.04s/it, epoch=23, grad_norm=358.928, loss=358.7519, lr=2.51e-04]
Epoch 23 TRAIN:   1%|          | 1/136 [00:04<09:05,  4.04s/it, epoch=23, grad_norm=448.175, loss=430.5553, lr=2.51e-04]
Epoch 23 TRAIN:   1%|          | 1/136 [00:04<09:05,  4.04s/it, epoch=23, grad_norm=421.697, loss=428.7736, lr=2.51e-04]
Epoch 23 TRAIN:   1%|          | 1/136 [00:04<09:05,  4.04s/it, epoch=23, grad_norm=437.071, loss=433.5879, lr=2.51e-04]
Epoch 23 TRAIN:   1%|          | 1/136 [00:04<09:05,  4.04s/it, epoch=23, grad_norm=447.208, loss=460.1873, lr=2.51e-04]
Epoch 23 TRAIN:   1%|          | 1/136 [00:04<09:05,  4.04s/it, epoch=23, grad_norm=310.544, loss=384.6538, lr=2.51e-04]
Epoch 23 TRAIN:   1%|          | 1/136 [00:04<09:05,  4.04s/it, epoch=23, grad_norm=111.154, loss=267.8864, lr=2.51e-04]
Epoch 23 TRAIN:   1%|          | 1/136 [00:04<09:05,  4.04s/it, epoch=23, grad_norm=147.774, loss=340.1835, lr=2.51e-04]
Epoch 23 TRAIN:   1%|          | 1/136 [00:04<09:05,  4.04s/it, epoch=23, grad_norm=171.786, loss=354.9942, lr=2.51e-04]
Epoch 23 TRAIN:   7%|▋         | 9/136 [00:04<00:50,  2.51it/s, epoch=23, grad_norm=171.786, loss=354.9942, lr=2.51e-04]
Epoch 23 TRAIN:   7%|▋         | 9/136 [00:04<00:50,  2.51it/s, epoch=23, grad_norm=113.562, loss=328.6232, lr=2.51e-04]
Epoch 23 TRAIN:   7%|▋         | 9/136 [00:05<00:50,  2.51it/s, epoch=23, grad_norm=91.680, loss=339.2239, lr=2.52e-04] 
Epoch 23 TRAIN:   7%|▋         | 9/136 [00:05<00:50,  2.51it/s, epoch=23, grad_norm=73.775, loss=371.2017, lr=2.52e-04]
Epoch 23 TRAIN:   7%|▋         | 9/136 [00:05<00:50,  2.51it/s, epoch=23, grad_norm=137.437, loss=343.3572, lr=2.52e-04]
Epoch 23 TRAIN:  10%|▉         | 13/136 [00:05<00:37,  3.28it/s, epoch=23, grad_norm=137.437, loss=343.3572, lr=2.52e-04]
Epoch 23 TRAIN:  10%|▉         | 13/136 [00:05<00:37,  3.28it/s, epoch=23, grad_norm=132.164, loss=308.9210, lr=2.52e-04]
Epoch 23 TRAIN:  10%|▉         | 13/136 [00:05<00:37,  3.28it/s, epoch=23, grad_norm=126.447, loss=312.0749, lr=2.52e-04]
Epoch 23 TRAIN:  10%|▉         | 13/136 [00:05<00:37,  3.28it/s, epoch=23, grad_norm=70.528, loss=324.1533, lr=2.52e-04] 
Epoch 23 TRAIN:  10%|▉         | 13/136 [00:05<00:37,  3.28it/s, epoch=23, grad_norm=87.812, loss=300.8325, lr=2.52e-04]
Epoch 23 TRAIN:  12%|█▎        | 17/136 [00:05<00:29,  4.00it/s, epoch=23, grad_norm=87.812, loss=300.8325, lr=2.52e-04]
Epoch 23 TRAIN:  12%|█▎        | 17/136 [00:06<00:29,  4.00it/s, epoch=23, grad_norm=68.267, loss=293.1754, lr=2.52e-04]
Epoch 23 TRAIN:  12%|█▎        | 17/136 [00:06<00:29,  4.00it/s, epoch=23, grad_norm=70.400, loss=304.7556, lr=2.52e-04]
Epoch 23 TRAIN:  12%|█▎        | 17/136 [00:06<00:29,  4.00it/s, epoch=23, grad_norm=93.930, loss=306.1500, lr=2.52e-04]
Epoch 23 TRAIN:  12%|█▎        | 17/136 [00:06<00:29,  4.00it/s, epoch=23, grad_norm=67.456, loss=293.3633, lr=2.52e-04]
Epoch 23 TRAIN:  15%|█▌        | 21/136 [00:06<00:25,  4.54it/s, epoch=23, grad_norm=67.456, loss=293.3633, lr=2.52e-04]
Epoch 23 TRAIN:  15%|█▌        | 21/136 [00:06<00:25,  4.54it/s, epoch=23, grad_norm=126.759, loss=330.2798, lr=2.52e-04]
Epoch 23 TRAIN:  15%|█▌        | 21/136 [00:06<00:25,  4.54it/s, epoch=23, grad_norm=57.204, loss=280.2668, lr=2.52e-04] 
Epoch 23 TRAIN:  15%|█▌        | 21/136 [00:07<00:25,  4.54it/s, epoch=23, grad_norm=89.068, loss=322.6611, lr=2.53e-04]
Epoch 23 TRAIN:  15%|█▌        | 21/136 [00:07<00:25,  4.54it/s, epoch=23, grad_norm=90.116, loss=330.0187, lr=2.53e-04]
Epoch 23 TRAIN:  18%|█▊        | 25/136 [00:07<00:22,  5.01it/s, epoch=23, grad_norm=90.116, loss=330.0187, lr=2.53e-04]
Epoch 23 TRAIN:  18%|█▊        | 25/136 [00:07<00:22,  5.01it/s, epoch=23, grad_norm=75.312, loss=333.4867, lr=2.53e-04]
Epoch 23 TRAIN:  18%|█▊        | 25/136 [00:07<00:22,  5.01it/s, epoch=23, grad_norm=80.492, loss=302.6119, lr=2.53e-04]
Epoch 23 TRAIN:  18%|█▊        | 25/136 [00:07<00:22,  5.01it/s, epoch=23, grad_norm=62.246, loss=338.4133, lr=2.53e-04]
Epoch 23 TRAIN:  18%|█▊        | 25/136 [00:07<00:22,  5.01it/s, epoch=23, grad_norm=56.944, loss=346.1694, lr=2.53e-04]
Epoch 23 TRAIN:  21%|██▏       | 29/136 [00:07<00:20,  5.15it/s, epoch=23, grad_norm=56.944, loss=346.1694, lr=2.53e-04]
Epoch 23 TRAIN:  21%|██▏       | 29/136 [00:08<00:20,  5.15it/s, epoch=23, grad_norm=71.554, loss=357.6579, lr=2.53e-04]
Epoch 23 TRAIN:  21%|██▏       | 29/136 [00:08<00:20,  5.15it/s, epoch=23, grad_norm=137.861, loss=319.4099, lr=2.53e-04]
Epoch 23 TRAIN:  21%|██▏       | 29/136 [00:08<00:20,  5.15it/s, epoch=23, grad_norm=63.725, loss=349.3890, lr=2.53e-04] 
Epoch 23 TRAIN:  21%|██▏       | 29/136 [00:08<00:20,  5.15it/s, epoch=23, grad_norm=151.174, loss=276.6673, lr=2.53e-04]
Epoch 23 TRAIN:  24%|██▍       | 33/136 [00:08<00:19,  5.25it/s, epoch=23, grad_norm=151.174, loss=276.6673, lr=2.53e-04]
Epoch 23 TRAIN:  24%|██▍       | 33/136 [00:08<00:19,  5.25it/s, epoch=23, grad_norm=146.620, loss=324.9673, lr=2.53e-04]
Epoch 23 TRAIN:  24%|██▍       | 33/136 [00:09<00:19,  5.25it/s, epoch=23, grad_norm=73.078, loss=317.3660, lr=2.53e-04] 
Epoch 23 TRAIN:  24%|██▍       | 33/136 [00:09<00:19,  5.25it/s, epoch=23, grad_norm=63.208, loss=356.6835, lr=2.54e-04]
Epoch 23 TRAIN:  26%|██▋       | 36/136 [00:09<00:18,  5.40it/s, epoch=23, grad_norm=63.208, loss=356.6835, lr=2.54e-04]
Epoch 23 TRAIN:  26%|██▋       | 36/136 [00:09<00:18,  5.40it/s, epoch=23, grad_norm=81.152, loss=302.1910, lr=2.54e-04]
Epoch 23 TRAIN:  26%|██▋       | 36/136 [00:09<00:18,  5.40it/s, epoch=23, grad_norm=69.102, loss=297.8170, lr=2.54e-04]
Epoch 23 TRAIN:  26%|██▋       | 36/136 [00:09<00:18,  5.40it/s, epoch=23, grad_norm=110.106, loss=313.0433, lr=2.54e-04]
Epoch 23 TRAIN:  26%|██▋       | 36/136 [00:09<00:18,  5.40it/s, epoch=23, grad_norm=111.881, loss=359.4360, lr=2.54e-04]
Epoch 23 TRAIN:  29%|██▉       | 40/136 [00:09<00:18,  5.30it/s, epoch=23, grad_norm=111.881, loss=359.4360, lr=2.54e-04]
Epoch 23 TRAIN:  29%|██▉       | 40/136 [00:10<00:18,  5.30it/s, epoch=23, grad_norm=141.289, loss=358.1447, lr=2.54e-04]
Epoch 23 TRAIN:  29%|██▉       | 40/136 [00:10<00:18,  5.30it/s, epoch=23, grad_norm=60.886, loss=322.3582, lr=2.54e-04] 
Epoch 23 TRAIN:  29%|██▉       | 40/136 [00:10<00:18,  5.30it/s, epoch=23, grad_norm=133.525, loss=386.5397, lr=2.54e-04]
Epoch 23 TRAIN:  29%|██▉       | 40/136 [00:10<00:18,  5.30it/s, epoch=23, grad_norm=108.929, loss=321.7048, lr=2.54e-04]
Epoch 23 TRAIN:  32%|███▏      | 44/136 [00:10<00:16,  5.52it/s, epoch=23, grad_norm=108.929, loss=321.7048, lr=2.54e-04]
Epoch 23 TRAIN:  32%|███▏      | 44/136 [00:10<00:16,  5.52it/s, epoch=23, grad_norm=130.567, loss=319.7865, lr=2.54e-04]
Epoch 23 TRAIN:  32%|███▏      | 44/136 [00:11<00:16,  5.52it/s, epoch=23, grad_norm=105.198, loss=319.0242, lr=2.54e-04]
Epoch 23 TRAIN:  32%|███▏      | 44/136 [00:11<00:16,  5.52it/s, epoch=23, grad_norm=73.870, loss=377.7586, lr=2.54e-04] 
Epoch 23 TRAIN:  35%|███▍      | 47/136 [00:11<00:16,  5.33it/s, epoch=23, grad_norm=73.870, loss=377.7586, lr=2.54e-04]
Epoch 23 TRAIN:  35%|███▍      | 47/136 [00:11<00:16,  5.33it/s, epoch=23, grad_norm=97.855, loss=334.2201, lr=2.54e-04]
Epoch 23 TRAIN:  35%|███▍      | 47/136 [00:11<00:16,  5.33it/s, epoch=23, grad_norm=79.466, loss=323.8657, lr=2.55e-04]
Epoch 23 TRAIN:  35%|███▍      | 47/136 [00:11<00:16,  5.33it/s, epoch=23, grad_norm=117.466, loss=300.6267, lr=2.55e-04]
Epoch 23 TRAIN:  37%|███▋      | 50/136 [00:11<00:16,  5.17it/s, epoch=23, grad_norm=117.466, loss=300.6267, lr=2.55e-04]
Epoch 23 TRAIN:  37%|███▋      | 50/136 [00:12<00:16,  5.17it/s, epoch=23, grad_norm=58.839, loss=326.5627, lr=2.55e-04] 
Epoch 23 TRAIN:  37%|███▋      | 50/136 [00:12<00:16,  5.17it/s, epoch=23, grad_norm=142.147, loss=248.1733, lr=2.55e-04]
Epoch 23 TRAIN:  37%|███▋      | 50/136 [00:12<00:16,  5.17it/s, epoch=23, grad_norm=91.859, loss=329.6502, lr=2.55e-04] 
Epoch 23 TRAIN:  39%|███▉      | 53/136 [00:12<00:16,  5.16it/s, epoch=23, grad_norm=91.859, loss=329.6502, lr=2.55e-04]
Epoch 23 TRAIN:  39%|███▉      | 53/136 [00:12<00:16,  5.16it/s, epoch=23, grad_norm=87.257, loss=316.6131, lr=2.55e-04]
Epoch 23 TRAIN:  39%|███▉      | 53/136 [00:12<00:16,  5.16it/s, epoch=23, grad_norm=52.679, loss=265.5441, lr=2.55e-04]
Epoch 23 TRAIN:  39%|███▉      | 53/136 [00:12<00:16,  5.16it/s, epoch=23, grad_norm=53.025, loss=267.7011, lr=2.55e-04]
Epoch 23 TRAIN:  41%|████      | 56/136 [00:12<00:15,  5.32it/s, epoch=23, grad_norm=53.025, loss=267.7011, lr=2.55e-04]
Epoch 23 TRAIN:  41%|████      | 56/136 [00:13<00:15,  5.32it/s, epoch=23, grad_norm=122.657, loss=326.0024, lr=2.55e-04]
Epoch 23 TRAIN:  41%|████      | 56/136 [00:13<00:15,  5.32it/s, epoch=23, grad_norm=64.001, loss=295.3665, lr=2.55e-04] 
Epoch 23 TRAIN:  41%|████      | 56/136 [00:13<00:15,  5.32it/s, epoch=23, grad_norm=102.171, loss=322.6239, lr=2.55e-04]
Epoch 23 TRAIN:  43%|████▎     | 59/136 [00:13<00:15,  5.03it/s, epoch=23, grad_norm=102.171, loss=322.6239, lr=2.55e-04]
Epoch 23 TRAIN:  43%|████▎     | 59/136 [00:13<00:15,  5.03it/s, epoch=23, grad_norm=62.984, loss=300.3010, lr=2.55e-04] 
Epoch 23 TRAIN:  43%|████▎     | 59/136 [00:14<00:15,  5.03it/s, epoch=23, grad_norm=83.710, loss=332.3968, lr=2.56e-04]
Epoch 23 TRAIN:  43%|████▎     | 59/136 [00:14<00:15,  5.03it/s, epoch=23, grad_norm=78.261, loss=343.7912, lr=2.56e-04]
Epoch 23 TRAIN:  46%|████▌     | 62/136 [00:14<00:14,  5.18it/s, epoch=23, grad_norm=78.261, loss=343.7912, lr=2.56e-04]
Epoch 23 TRAIN:  46%|████▌     | 62/136 [00:14<00:14,  5.18it/s, epoch=23, grad_norm=68.250, loss=347.9645, lr=2.56e-04]
Epoch 23 TRAIN:  46%|████▌     | 62/136 [00:14<00:14,  5.18it/s, epoch=23, grad_norm=177.843, loss=295.4198, lr=2.56e-04]
Epoch 23 TRAIN:  46%|████▌     | 62/136 [00:14<00:14,  5.18it/s, epoch=23, grad_norm=177.966, loss=290.3864, lr=2.56e-04]
Epoch 23 TRAIN:  48%|████▊     | 65/136 [00:14<00:13,  5.30it/s, epoch=23, grad_norm=177.966, loss=290.3864, lr=2.56e-04]
Epoch 23 TRAIN:  48%|████▊     | 65/136 [00:14<00:13,  5.30it/s, epoch=23, grad_norm=82.062, loss=361.2584, lr=2.56e-04] 
Epoch 23 TRAIN:  48%|████▊     | 65/136 [00:15<00:13,  5.30it/s, epoch=23, grad_norm=148.328, loss=299.6785, lr=2.56e-04]
Epoch 23 TRAIN:  48%|████▊     | 65/136 [00:15<00:13,  5.30it/s, epoch=23, grad_norm=154.764, loss=304.1205, lr=2.56e-04]
Epoch 23 TRAIN:  50%|█████     | 68/136 [00:15<00:13,  5.01it/s, epoch=23, grad_norm=154.764, loss=304.1205, lr=2.56e-04]
Epoch 23 TRAIN:  50%|█████     | 68/136 [00:15<00:13,  5.01it/s, epoch=23, grad_norm=143.448, loss=375.9400, lr=2.56e-04]
Epoch 23 TRAIN:  50%|█████     | 68/136 [00:15<00:13,  5.01it/s, epoch=23, grad_norm=69.770, loss=335.4215, lr=2.56e-04] 
Epoch 23 TRAIN:  50%|█████     | 68/136 [00:15<00:13,  5.01it/s, epoch=23, grad_norm=87.783, loss=332.8948, lr=2.56e-04]
Epoch 23 TRAIN:  52%|█████▏    | 71/136 [00:15<00:12,  5.09it/s, epoch=23, grad_norm=87.783, loss=332.8948, lr=2.56e-04]
Epoch 23 TRAIN:  52%|█████▏    | 71/136 [00:16<00:12,  5.09it/s, epoch=23, grad_norm=63.251, loss=336.9857, lr=2.56e-04]
Epoch 23 TRAIN:  52%|█████▏    | 71/136 [00:16<00:12,  5.09it/s, epoch=23, grad_norm=60.884, loss=327.9836, lr=2.56e-04]
Epoch 23 TRAIN:  52%|█████▏    | 71/136 [00:16<00:12,  5.09it/s, epoch=23, grad_norm=86.394, loss=277.1837, lr=2.57e-04]
Epoch 23 TRAIN:  54%|█████▍    | 74/136 [00:16<00:12,  5.15it/s, epoch=23, grad_norm=86.394, loss=277.1837, lr=2.57e-04]
Epoch 23 TRAIN:  54%|█████▍    | 74/136 [00:16<00:12,  5.15it/s, epoch=23, grad_norm=78.219, loss=347.1641, lr=2.57e-04]
Epoch 23 TRAIN:  54%|█████▍    | 74/136 [00:16<00:12,  5.15it/s, epoch=23, grad_norm=107.558, loss=308.3444, lr=2.57e-04]
Epoch 23 TRAIN:  54%|█████▍    | 74/136 [00:17<00:12,  5.15it/s, epoch=23, grad_norm=115.601, loss=334.7465, lr=2.57e-04]
Epoch 23 TRAIN:  57%|█████▋    | 77/136 [00:17<00:11,  5.13it/s, epoch=23, grad_norm=115.601, loss=334.7465, lr=2.57e-04]
Epoch 23 TRAIN:  57%|█████▋    | 77/136 [00:17<00:11,  5.13it/s, epoch=23, grad_norm=103.007, loss=354.2264, lr=2.57e-04]
Epoch 23 TRAIN:  57%|█████▋    | 77/136 [00:17<00:11,  5.13it/s, epoch=23, grad_norm=76.905, loss=325.3422, lr=2.57e-04] 
Epoch 23 TRAIN:  57%|█████▋    | 77/136 [00:17<00:11,  5.13it/s, epoch=23, grad_norm=122.283, loss=355.8907, lr=2.57e-04]
Epoch 23 TRAIN:  59%|█████▉    | 80/136 [00:17<00:11,  4.98it/s, epoch=23, grad_norm=122.283, loss=355.8907, lr=2.57e-04]
Epoch 23 TRAIN:  59%|█████▉    | 80/136 [00:17<00:11,  4.98it/s, epoch=23, grad_norm=97.733, loss=342.7578, lr=2.57e-04] 
Epoch 23 TRAIN:  59%|█████▉    | 80/136 [00:18<00:11,  4.98it/s, epoch=23, grad_norm=135.153, loss=301.5862, lr=2.57e-04]
Epoch 23 TRAIN:  59%|█████▉    | 80/136 [00:18<00:11,  4.98it/s, epoch=23, grad_norm=317.898, loss=223.8924, lr=2.57e-04]
Epoch 23 TRAIN:  61%|██████    | 83/136 [00:18<00:10,  5.05it/s, epoch=23, grad_norm=317.898, loss=223.8924, lr=2.57e-04]
Epoch 23 TRAIN:  61%|██████    | 83/136 [00:18<00:10,  5.05it/s, epoch=23, grad_norm=164.788, loss=311.4727, lr=2.57e-04]
Epoch 23 TRAIN:  61%|██████    | 83/136 [00:18<00:10,  5.05it/s, epoch=23, grad_norm=74.925, loss=323.2948, lr=2.57e-04] 
Epoch 23 TRAIN:  61%|██████    | 83/136 [00:18<00:10,  5.05it/s, epoch=23, grad_norm=82.312, loss=298.0826, lr=2.58e-04]
Epoch 23 TRAIN:  63%|██████▎   | 86/136 [00:18<00:09,  5.06it/s, epoch=23, grad_norm=82.312, loss=298.0826, lr=2.58e-04]
Epoch 23 TRAIN:  63%|██████▎   | 86/136 [00:19<00:09,  5.06it/s, epoch=23, grad_norm=156.890, loss=373.4171, lr=2.58e-04]
Epoch 23 TRAIN:  63%|██████▎   | 86/136 [00:19<00:09,  5.06it/s, epoch=23, grad_norm=90.841, loss=303.3602, lr=2.58e-04] 
Epoch 23 TRAIN:  63%|██████▎   | 86/136 [00:19<00:09,  5.06it/s, epoch=23, grad_norm=210.009, loss=240.7587, lr=2.58e-04]
Epoch 23 TRAIN:  65%|██████▌   | 89/136 [00:19<00:09,  4.89it/s, epoch=23, grad_norm=210.009, loss=240.7587, lr=2.58e-04]
Epoch 23 TRAIN:  65%|██████▌   | 89/136 [00:19<00:09,  4.89it/s, epoch=23, grad_norm=81.496, loss=258.6920, lr=2.58e-04] 
Epoch 23 TRAIN:  65%|██████▌   | 89/136 [00:19<00:09,  4.89it/s, epoch=23, grad_norm=97.325, loss=347.0402, lr=2.58e-04]
Epoch 23 TRAIN:  65%|██████▌   | 89/136 [00:20<00:09,  4.89it/s, epoch=23, grad_norm=114.060, loss=304.1415, lr=2.58e-04]
Epoch 23 TRAIN:  68%|██████▊   | 92/136 [00:20<00:08,  5.01it/s, epoch=23, grad_norm=114.060, loss=304.1415, lr=2.58e-04]
Epoch 23 TRAIN:  68%|██████▊   | 92/136 [00:20<00:08,  5.01it/s, epoch=23, grad_norm=98.961, loss=252.6257, lr=2.58e-04] 
Epoch 23 TRAIN:  68%|██████▊   | 92/136 [00:20<00:08,  5.01it/s, epoch=23, grad_norm=99.876, loss=313.8113, lr=2.58e-04]
Epoch 23 TRAIN:  68%|██████▊   | 92/136 [00:20<00:08,  5.01it/s, epoch=23, grad_norm=54.019, loss=250.7426, lr=2.58e-04]
Epoch 23 TRAIN:  70%|██████▉   | 95/136 [00:20<00:08,  5.06it/s, epoch=23, grad_norm=54.019, loss=250.7426, lr=2.58e-04]
Epoch 23 TRAIN:  70%|██████▉   | 95/136 [00:20<00:08,  5.06it/s, epoch=23, grad_norm=81.104, loss=281.6851, lr=2.58e-04]
Epoch 23 TRAIN:  70%|██████▉   | 95/136 [00:21<00:08,  5.06it/s, epoch=23, grad_norm=102.731, loss=325.2048, lr=2.58e-04]
Epoch 23 TRAIN:  70%|██████▉   | 95/136 [00:21<00:08,  5.06it/s, epoch=23, grad_norm=132.727, loss=225.2147, lr=2.58e-04]
Epoch 23 TRAIN:  72%|███████▏  | 98/136 [00:21<00:07,  5.14it/s, epoch=23, grad_norm=132.727, loss=225.2147, lr=2.58e-04]
Epoch 23 TRAIN:  72%|███████▏  | 98/136 [00:21<00:07,  5.14it/s, epoch=23, grad_norm=94.820, loss=335.3420, lr=2.59e-04] 
Epoch 23 TRAIN:  72%|███████▏  | 98/136 [00:21<00:07,  5.14it/s, epoch=23, grad_norm=125.130, loss=350.3705, lr=2.59e-04]2026-04-16 20:03:32,648 DEBUG TRAIN | steps/sec 4.570| Batch 23/100 loss 350.370483 loss_att 315.740479 loss_ctc 431.173828 th_accuracy 0.268344 lr 2.5864e-04 grad_norm 125.129585 rank 0

Epoch 23 TRAIN:  72%|███████▏  | 98/136 [00:21<00:07,  5.14it/s, epoch=23, grad_norm=223.316, loss=367.6140, lr=2.59e-04]
Epoch 23 TRAIN:  74%|███████▍  | 101/136 [00:21<00:06,  5.21it/s, epoch=23, grad_norm=223.316, loss=367.6140, lr=2.59e-04]
Epoch 23 TRAIN:  74%|███████▍  | 101/136 [00:22<00:06,  5.21it/s, epoch=23, grad_norm=259.627, loss=363.1298, lr=2.59e-04]
Epoch 23 TRAIN:  74%|███████▍  | 101/136 [00:22<00:06,  5.21it/s, epoch=23, grad_norm=163.937, loss=216.8878, lr=2.59e-04]
Epoch 23 TRAIN:  74%|███████▍  | 101/136 [00:22<00:06,  5.21it/s, epoch=23, grad_norm=156.929, loss=316.4556, lr=2.59e-04]
Epoch 23 TRAIN:  76%|███████▋  | 104/136 [00:22<00:06,  5.10it/s, epoch=23, grad_norm=156.929, loss=316.4556, lr=2.59e-04]
Epoch 23 TRAIN:  76%|███████▋  | 104/136 [00:22<00:06,  5.10it/s, epoch=23, grad_norm=78.371, loss=348.7145, lr=2.59e-04] 
Epoch 23 TRAIN:  76%|███████▋  | 104/136 [00:22<00:06,  5.10it/s, epoch=23, grad_norm=106.580, loss=366.6066, lr=2.59e-04]
Epoch 23 TRAIN:  76%|███████▋  | 104/136 [00:23<00:06,  5.10it/s, epoch=23, grad_norm=102.498, loss=296.9383, lr=2.59e-04]
Epoch 23 TRAIN:  79%|███████▊  | 107/136 [00:23<00:05,  5.03it/s, epoch=23, grad_norm=102.498, loss=296.9383, lr=2.59e-04]
Epoch 23 TRAIN:  79%|███████▊  | 107/136 [00:23<00:05,  5.03it/s, epoch=23, grad_norm=120.565, loss=368.9103, lr=2.59e-04]
Epoch 23 TRAIN:  79%|███████▊  | 107/136 [00:23<00:05,  5.03it/s, epoch=23, grad_norm=84.082, loss=340.8895, lr=2.59e-04] 
Epoch 23 TRAIN:  79%|███████▊  | 107/136 [00:23<00:05,  5.03it/s, epoch=23, grad_norm=93.849, loss=331.3868, lr=2.59e-04]
Epoch 23 TRAIN:  81%|████████  | 110/136 [00:23<00:05,  5.07it/s, epoch=23, grad_norm=93.849, loss=331.3868, lr=2.59e-04]
Epoch 23 TRAIN:  81%|████████  | 110/136 [00:23<00:05,  5.07it/s, epoch=23, grad_norm=66.361, loss=297.3235, lr=2.60e-04]
Epoch 23 TRAIN:  81%|████████  | 110/136 [00:24<00:05,  5.07it/s, epoch=23, grad_norm=142.381, loss=332.2116, lr=2.60e-04]
Epoch 23 TRAIN:  81%|████████  | 110/136 [00:24<00:05,  5.07it/s, epoch=23, grad_norm=72.902, loss=259.5712, lr=2.60e-04] 
Epoch 23 TRAIN:  83%|████████▎ | 113/136 [00:24<00:04,  5.16it/s, epoch=23, grad_norm=72.902, loss=259.5712, lr=2.60e-04]
Epoch 23 TRAIN:  83%|████████▎ | 113/136 [00:24<00:04,  5.16it/s, epoch=23, grad_norm=98.126, loss=280.6563, lr=2.60e-04]
Epoch 23 TRAIN:  83%|████████▎ | 113/136 [00:24<00:04,  5.16it/s, epoch=23, grad_norm=100.686, loss=307.6388, lr=2.60e-04]
Epoch 23 TRAIN:  83%|████████▎ | 113/136 [00:24<00:04,  5.16it/s, epoch=23, grad_norm=89.953, loss=304.9366, lr=2.60e-04] 
Epoch 23 TRAIN:  85%|████████▌ | 116/136 [00:24<00:04,  4.77it/s, epoch=23, grad_norm=89.953, loss=304.9366, lr=2.60e-04]
Epoch 23 TRAIN:  85%|████████▌ | 116/136 [00:25<00:04,  4.77it/s, epoch=23, grad_norm=72.014, loss=266.3008, lr=2.60e-04]
Epoch 23 TRAIN:  85%|████████▌ | 116/136 [00:25<00:04,  4.77it/s, epoch=23, grad_norm=126.637, loss=281.2890, lr=2.60e-04]
Epoch 23 TRAIN:  85%|████████▌ | 116/136 [00:25<00:04,  4.77it/s, epoch=23, grad_norm=127.073, loss=356.4169, lr=2.60e-04]
Epoch 23 TRAIN:  88%|████████▊ | 119/136 [00:25<00:03,  4.83it/s, epoch=23, grad_norm=127.073, loss=356.4169, lr=2.60e-04]
Epoch 23 TRAIN:  88%|████████▊ | 119/136 [00:25<00:03,  4.83it/s, epoch=23, grad_norm=97.940, loss=257.8063, lr=2.60e-04] 
Epoch 23 TRAIN:  88%|████████▊ | 119/136 [00:25<00:03,  4.83it/s, epoch=23, grad_norm=115.494, loss=311.4592, lr=2.60e-04]
Epoch 23 TRAIN:  88%|████████▊ | 119/136 [00:26<00:03,  4.83it/s, epoch=23, grad_norm=103.889, loss=318.4777, lr=2.60e-04]
Epoch 23 TRAIN:  90%|████████▉ | 122/136 [00:26<00:02,  4.93it/s, epoch=23, grad_norm=103.889, loss=318.4777, lr=2.60e-04]
Epoch 23 TRAIN:  90%|████████▉ | 122/136 [00:26<00:02,  4.93it/s, epoch=23, grad_norm=114.354, loss=350.1490, lr=2.60e-04]
Epoch 23 TRAIN:  90%|████████▉ | 122/136 [00:26<00:02,  4.93it/s, epoch=23, grad_norm=106.132, loss=247.9604, lr=2.61e-04]
Epoch 23 TRAIN:  90%|████████▉ | 122/136 [00:26<00:02,  4.93it/s, epoch=23, grad_norm=173.291, loss=340.5962, lr=2.61e-04]
Epoch 23 TRAIN:  92%|█████████▏| 125/136 [00:26<00:02,  5.03it/s, epoch=23, grad_norm=173.291, loss=340.5962, lr=2.61e-04]
Epoch 23 TRAIN:  92%|█████████▏| 125/136 [00:26<00:02,  5.03it/s, epoch=23, grad_norm=139.099, loss=324.7947, lr=2.61e-04]
Epoch 23 TRAIN:  92%|█████████▏| 125/136 [00:27<00:02,  5.03it/s, epoch=23, grad_norm=320.066, loss=319.1061, lr=2.61e-04]
Epoch 23 TRAIN:  92%|█████████▏| 125/136 [00:27<00:02,  5.03it/s, epoch=23, grad_norm=139.850, loss=342.0396, lr=2.61e-04]
Epoch 23 TRAIN:  92%|█████████▏| 125/136 [00:27<00:02,  5.03it/s, epoch=23, grad_norm=129.088, loss=410.0563, lr=2.61e-04]
Epoch 23 TRAIN:  95%|█████████▍| 129/136 [00:27<00:01,  5.38it/s, epoch=23, grad_norm=129.088, loss=410.0563, lr=2.61e-04]
Epoch 23 TRAIN:  95%|█████████▍| 129/136 [00:27<00:01,  5.38it/s, epoch=23, grad_norm=93.617, loss=327.3088, lr=2.61e-04] 
Epoch 23 TRAIN:  95%|█████████▍| 129/136 [00:27<00:01,  5.38it/s, epoch=23, grad_norm=185.955, loss=397.2733, lr=2.61e-04]
Epoch 23 TRAIN:  95%|█████████▍| 129/136 [00:27<00:01,  5.38it/s, epoch=23, grad_norm=171.561, loss=301.2831, lr=2.61e-04]
Epoch 23 TRAIN:  95%|█████████▍| 129/136 [00:28<00:01,  5.38it/s, epoch=23, grad_norm=120.168, loss=393.5246, lr=2.61e-04]
Epoch 23 TRAIN:  98%|█████████▊| 133/136 [00:28<00:00,  5.69it/s, epoch=23, grad_norm=120.168, loss=393.5246, lr=2.61e-04]
Epoch 23 TRAIN:  98%|█████████▊| 133/136 [00:28<00:00,  5.69it/s, epoch=23, grad_norm=150.678, loss=317.9573, lr=2.61e-04]
Epoch 23 TRAIN:  98%|█████████▊| 133/136 [00:28<00:00,  5.69it/s, epoch=23, grad_norm=225.917, loss=435.3306, lr=2.61e-04]
Epoch 23 TRAIN:  98%|█████████▊| 133/136 [00:28<00:00,  5.69it/s, epoch=23, grad_norm=206.542, loss=362.0164, lr=2.62e-04]
Epoch 23 TRAIN: 100%|██████████| 136/136 [00:28<00:00,  4.77it/s, epoch=23, grad_norm=206.542, loss=362.0164, lr=2.62e-04]

CV:   0%|          | 0/6 [00:00<?, ?it/s]/home/rookie/Documents/batuka/projects/wenet/wenet/wenet/utils/train_utils.py:689: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  amp_autocast(enabled=dtype is not None,

CV:  17%|█▋        | 1/6 [00:00<00:03,  1.33it/s]
                                                 
2026-04-16 20:03:40,286 INFO Epoch 23 Step 3269 CV info lr 2.6152e-04 cv_loss 537.426924272017 rank 0 acc 0.2367580235004425
2026-04-16 20:03:40,288 INFO [Rank 0] Checkpoint: save to checkpoint exp/conformer_mongolian/epoch_23.pt
2026-04-16 20:03:40,401 INFO Epoch 24 Step 3269 TRAIN info lr 2.6152e-04 rank 0
2026-04-16 20:03:40,414 INFO using accumulate grad, new batch size is 1 times larger than before

Epoch 24 TRAIN:   0%|          | 0/136 [00:00<?, ?it/s]
Epoch 24 TRAIN:   0%|          | 0/136 [00:04<?, ?it/s, epoch=24, grad_norm=309.397, loss=307.0945, lr=2.62e-04]
Epoch 24 TRAIN:   1%|          | 1/136 [00:04<09:02,  4.02s/it, epoch=24, grad_norm=309.397, loss=307.0945, lr=2.62e-04]
Epoch 24 TRAIN:   1%|          | 1/136 [00:04<09:02,  4.02s/it, epoch=24, grad_norm=589.010, loss=464.2688, lr=2.62e-04]
Epoch 24 TRAIN:   1%|          | 1/136 [00:04<09:02,  4.02s/it, epoch=24, grad_norm=463.799, loss=422.1897, lr=2.62e-04]
Epoch 24 TRAIN:   1%|          | 1/136 [00:04<09:02,  4.02s/it, epoch=24, grad_norm=342.813, loss=357.9285, lr=2.62e-04]
Epoch 24 TRAIN:   1%|          | 1/136 [00:04<09:02,  4.02s/it, epoch=24, grad_norm=142.303, loss=272.1513, lr=2.62e-04]
Epoch 24 TRAIN:   1%|          | 1/136 [00:04<09:02,  4.02s/it, epoch=24, grad_norm=104.221, loss=336.6360, lr=2.62e-04]
Epoch 24 TRAIN:   1%|          | 1/136 [00:04<09:02,  4.02s/it, epoch=24, grad_norm=198.587, loss=359.6269, lr=2.62e-04]
Epoch 24 TRAIN:   1%|          | 1/136 [00:04<09:02,  4.02s/it, epoch=24, grad_norm=150.283, loss=351.9524, lr=2.62e-04]
Epoch 24 TRAIN:   1%|          | 1/136 [00:04<09:02,  4.02s/it, epoch=24, grad_norm=97.573, loss=353.3809, lr=2.62e-04] 
Epoch 24 TRAIN:   7%|▋         | 9/136 [00:04<00:50,  2.52it/s, epoch=24, grad_norm=97.573, loss=353.3809, lr=2.62e-04]
Epoch 24 TRAIN:   7%|▋         | 9/136 [00:04<00:50,  2.52it/s, epoch=24, grad_norm=113.760, loss=317.7659, lr=2.62e-04]
Epoch 24 TRAIN:   7%|▋         | 9/136 [00:04<00:50,  2.52it/s, epoch=24, grad_norm=99.170, loss=343.0923, lr=2.62e-04] 
Epoch 24 TRAIN:   7%|▋         | 9/136 [00:05<00:50,  2.52it/s, epoch=24, grad_norm=74.762, loss=354.3581, lr=2.62e-04]
Epoch 24 TRAIN:   7%|▋         | 9/136 [00:05<00:50,  2.52it/s, epoch=24, grad_norm=80.840, loss=349.3247, lr=2.63e-04]
Epoch 24 TRAIN:  10%|▉         | 13/136 [00:05<00:37,  3.30it/s, epoch=24, grad_norm=80.840, loss=349.3247, lr=2.63e-04]
Epoch 24 TRAIN:  10%|▉         | 13/136 [00:05<00:37,  3.30it/s, epoch=24, grad_norm=71.364, loss=345.2693, lr=2.63e-04]
Epoch 24 TRAIN:  10%|▉         | 13/136 [00:05<00:37,  3.30it/s, epoch=24, grad_norm=137.880, loss=276.2313, lr=2.63e-04]
Epoch 24 TRAIN:  10%|▉         | 13/136 [00:05<00:37,  3.30it/s, epoch=24, grad_norm=72.561, loss=319.0979, lr=2.63e-04] 
Epoch 24 TRAIN:  10%|▉         | 13/136 [00:05<00:37,  3.30it/s, epoch=24, grad_norm=101.706, loss=359.9113, lr=2.63e-04]
Epoch 24 TRAIN:  12%|█▎        | 17/136 [00:05<00:29,  4.01it/s, epoch=24, grad_norm=101.706, loss=359.9113, lr=2.63e-04]
Epoch 24 TRAIN:  12%|█▎        | 17/136 [00:06<00:29,  4.01it/s, epoch=24, grad_norm=75.231, loss=326.8260, lr=2.63e-04] 
Epoch 24 TRAIN:  12%|█▎        | 17/136 [00:06<00:29,  4.01it/s, epoch=24, grad_norm=84.394, loss=306.9220, lr=2.63e-04]
Epoch 24 TRAIN:  12%|█▎        | 17/136 [00:06<00:29,  4.01it/s, epoch=24, grad_norm=75.182, loss=328.5979, lr=2.63e-04]
Epoch 24 TRAIN:  12%|█▎        | 17/136 [00:06<00:29,  4.01it/s, epoch=24, grad_norm=68.539, loss=318.8845, lr=2.63e-04]
Epoch 24 TRAIN:  15%|█▌        | 21/136 [00:06<00:25,  4.55it/s, epoch=24, grad_norm=68.539, loss=318.8845, lr=2.63e-04]
Epoch 24 TRAIN:  15%|█▌        | 21/136 [00:06<00:25,  4.55it/s, epoch=24, grad_norm=90.768, loss=311.5457, lr=2.63e-04]
Epoch 24 TRAIN:  15%|█▌        | 21/136 [00:06<00:25,  4.55it/s, epoch=24, grad_norm=83.011, loss=299.4008, lr=2.63e-04]
Epoch 24 TRAIN:  15%|█▌        | 21/136 [00:07<00:25,  4.55it/s, epoch=24, grad_norm=94.352, loss=270.5435, lr=2.63e-04]
Epoch 24 TRAIN:  15%|█▌        | 21/136 [00:07<00:25,  4.55it/s, epoch=24, grad_norm=68.724, loss=308.8969, lr=2.64e-04]
Epoch 24 TRAIN:  18%|█▊        | 25/136 [00:07<00:23,  4.76it/s, epoch=24, grad_norm=68.724, loss=308.8969, lr=2.64e-04]
Epoch 24 TRAIN:  18%|█▊        | 25/136 [00:07<00:23,  4.76it/s, epoch=24, grad_norm=74.576, loss=336.2318, lr=2.64e-04]
Epoch 24 TRAIN:  18%|█▊        | 25/136 [00:07<00:23,  4.76it/s, epoch=24, grad_norm=98.521, loss=335.0223, lr=2.64e-04]
Epoch 24 TRAIN:  18%|█▊        | 25/136 [00:07<00:23,  4.76it/s, epoch=24, grad_norm=114.324, loss=337.1807, lr=2.64e-04]
Epoch 24 TRAIN:  18%|█▊        | 25/136 [00:07<00:23,  4.76it/s, epoch=24, grad_norm=118.172, loss=319.0167, lr=2.64e-04]
Epoch 24 TRAIN:  21%|██▏       | 29/136 [00:07<00:20,  5.15it/s, epoch=24, grad_norm=118.172, loss=319.0167, lr=2.64e-04]
Epoch 24 TRAIN:  21%|██▏       | 29/136 [00:08<00:20,  5.15it/s, epoch=24, grad_norm=73.344, loss=326.0308, lr=2.64e-04] 
Epoch 24 TRAIN:  21%|██▏       | 29/136 [00:08<00:20,  5.15it/s, epoch=24, grad_norm=111.032, loss=354.8220, lr=2.64e-04]
Epoch 24 TRAIN:  21%|██▏       | 29/136 [00:08<00:20,  5.15it/s, epoch=24, grad_norm=66.813, loss=325.2582, lr=2.64e-04] 
Epoch 24 TRAIN:  21%|██▏       | 29/136 [00:08<00:20,  5.15it/s, epoch=24, grad_norm=127.224, loss=286.9884, lr=2.64e-04]
Epoch 24 TRAIN:  24%|██▍       | 33/136 [00:08<00:20,  5.11it/s, epoch=24, grad_norm=127.224, loss=286.9884, lr=2.64e-04]
Epoch 24 TRAIN:  24%|██▍       | 33/136 [00:08<00:20,  5.11it/s, epoch=24, grad_norm=118.910, loss=307.7555, lr=2.64e-04]
Epoch 24 TRAIN:  24%|██▍       | 33/136 [00:09<00:20,  5.11it/s, epoch=24, grad_norm=81.039, loss=345.2065, lr=2.64e-04] 
Epoch 24 TRAIN:  24%|██▍       | 33/136 [00:09<00:20,  5.11it/s, epoch=24, grad_norm=86.737, loss=329.0172, lr=2.64e-04]
Epoch 24 TRAIN:  26%|██▋       | 36/136 [00:09<00:19,  5.20it/s, epoch=24, grad_norm=86.737, loss=329.0172, lr=2.64e-04]
Epoch 24 TRAIN:  26%|██▋       | 36/136 [00:09<00:19,  5.20it/s, epoch=24, grad_norm=91.385, loss=341.4204, lr=2.64e-04]
Epoch 24 TRAIN:  26%|██▋       | 36/136 [00:09<00:19,  5.20it/s, epoch=24, grad_norm=96.556, loss=321.2670, lr=2.65e-04]
Epoch 24 TRAIN:  26%|██▋       | 36/136 [00:09<00:19,  5.20it/s, epoch=24, grad_norm=72.093, loss=316.4441, lr=2.65e-04]
Epoch 24 TRAIN:  29%|██▊       | 39/136 [00:09<00:19,  5.04it/s, epoch=24, grad_norm=72.093, loss=316.4441, lr=2.65e-04]
Epoch 24 TRAIN:  29%|██▊       | 39/136 [00:10<00:19,  5.04it/s, epoch=24, grad_norm=69.106, loss=290.7806, lr=2.65e-04]
Epoch 24 TRAIN:  29%|██▊       | 39/136 [00:10<00:19,  5.04it/s, epoch=24, grad_norm=79.034, loss=317.1873, lr=2.65e-04]
Epoch 24 TRAIN:  29%|██▊       | 39/136 [00:10<00:19,  5.04it/s, epoch=24, grad_norm=96.575, loss=312.4742, lr=2.65e-04]
Epoch 24 TRAIN:  31%|███       | 42/136 [00:10<00:18,  5.18it/s, epoch=24, grad_norm=96.575, loss=312.4742, lr=2.65e-04]
Epoch 24 TRAIN:  31%|███       | 42/136 [00:10<00:18,  5.18it/s, epoch=24, grad_norm=108.549, loss=327.0327, lr=2.65e-04]
Epoch 24 TRAIN:  31%|███       | 42/136 [00:10<00:18,  5.18it/s, epoch=24, grad_norm=84.172, loss=312.6779, lr=2.65e-04] 
Epoch 24 TRAIN:  31%|███       | 42/136 [00:11<00:18,  5.18it/s, epoch=24, grad_norm=88.203, loss=326.3251, lr=2.65e-04]
Epoch 24 TRAIN:  33%|███▎      | 45/136 [00:11<00:18,  5.01it/s, epoch=24, grad_norm=88.203, loss=326.3251, lr=2.65e-04]
Epoch 24 TRAIN:  33%|███▎      | 45/136 [00:11<00:18,  5.01it/s, epoch=24, grad_norm=71.397, loss=354.7493, lr=2.65e-04]
Epoch 24 TRAIN:  33%|███▎      | 45/136 [00:11<00:18,  5.01it/s, epoch=24, grad_norm=68.659, loss=343.7081, lr=2.65e-04]
Epoch 24 TRAIN:  33%|███▎      | 45/136 [00:11<00:18,  5.01it/s, epoch=24, grad_norm=82.085, loss=327.2509, lr=2.65e-04]
Epoch 24 TRAIN:  35%|███▌      | 48/136 [00:11<00:16,  5.19it/s, epoch=24, grad_norm=82.085, loss=327.2509, lr=2.65e-04]
Epoch 24 TRAIN:  35%|███▌      | 48/136 [00:11<00:16,  5.19it/s, epoch=24, grad_norm=83.491, loss=355.9730, lr=2.65e-04]
Epoch 24 TRAIN:  35%|███▌      | 48/136 [00:12<00:16,  5.19it/s, epoch=24, grad_norm=122.374, loss=339.0326, lr=2.66e-04]
Epoch 24 TRAIN:  35%|███▌      | 48/136 [00:12<00:16,  5.19it/s, epoch=24, grad_norm=111.331, loss=319.5703, lr=2.66e-04]
Epoch 24 TRAIN:  38%|███▊      | 51/136 [00:12<00:17,  4.97it/s, epoch=24, grad_norm=111.331, loss=319.5703, lr=2.66e-04]
Epoch 24 TRAIN:  38%|███▊      | 51/136 [00:12<00:17,  4.97it/s, epoch=24, grad_norm=85.162, loss=322.1940, lr=2.66e-04] 
Epoch 24 TRAIN:  38%|███▊      | 51/136 [00:12<00:17,  4.97it/s, epoch=24, grad_norm=107.415, loss=373.1310, lr=2.66e-04]
Epoch 24 TRAIN:  38%|███▊      | 51/136 [00:12<00:17,  4.97it/s, epoch=24, grad_norm=86.118, loss=333.6409, lr=2.66e-04] 
Epoch 24 TRAIN:  40%|███▉      | 54/136 [00:12<00:16,  5.07it/s, epoch=24, grad_norm=86.118, loss=333.6409, lr=2.66e-04]
Epoch 24 TRAIN:  40%|███▉      | 54/136 [00:13<00:16,  5.07it/s, epoch=24, grad_norm=89.093, loss=334.8440, lr=2.66e-04]
Epoch 24 TRAIN:  40%|███▉      | 54/136 [00:13<00:16,  5.07it/s, epoch=24, grad_norm=101.510, loss=339.4202, lr=2.66e-04]
Epoch 24 TRAIN:  40%|███▉      | 54/136 [00:13<00:16,  5.07it/s, epoch=24, grad_norm=112.505, loss=297.3246, lr=2.66e-04]
Epoch 24 TRAIN:  42%|████▏     | 57/136 [00:13<00:15,  5.15it/s, epoch=24, grad_norm=112.505, loss=297.3246, lr=2.66e-04]
Epoch 24 TRAIN:  42%|████▏     | 57/136 [00:13<00:15,  5.15it/s, epoch=24, grad_norm=83.805, loss=339.7481, lr=2.66e-04] 
Epoch 24 TRAIN:  42%|████▏     | 57/136 [00:13<00:15,  5.15it/s, epoch=24, grad_norm=79.313, loss=309.2108, lr=2.66e-04]
Epoch 24 TRAIN:  42%|████▏     | 57/136 [00:14<00:15,  5.15it/s, epoch=24, grad_norm=117.076, loss=328.9310, lr=2.66e-04]
Epoch 24 TRAIN:  44%|████▍     | 60/136 [00:14<00:14,  5.07it/s, epoch=24, grad_norm=117.076, loss=328.9310, lr=2.66e-04]
Epoch 24 TRAIN:  44%|████▍     | 60/136 [00:14<00:14,  5.07it/s, epoch=24, grad_norm=80.071, loss=296.7654, lr=2.66e-04] 
Epoch 24 TRAIN:  44%|████▍     | 60/136 [00:14<00:14,  5.07it/s, epoch=24, grad_norm=80.847, loss=306.9810, lr=2.66e-04]
Epoch 24 TRAIN:  44%|████▍     | 60/136 [00:14<00:14,  5.07it/s, epoch=24, grad_norm=73.402, loss=296.1462, lr=2.67e-04]
Epoch 24 TRAIN:  46%|████▋     | 63/136 [00:14<00:14,  5.04it/s, epoch=24, grad_norm=73.402, loss=296.1462, lr=2.67e-04]
Epoch 24 TRAIN:  46%|████▋     | 63/136 [00:14<00:14,  5.04it/s, epoch=24, grad_norm=94.538, loss=335.7672, lr=2.67e-04]
Epoch 24 TRAIN:  46%|████▋     | 63/136 [00:15<00:14,  5.04it/s, epoch=24, grad_norm=100.631, loss=317.5544, lr=2.67e-04]
Epoch 24 TRAIN:  46%|████▋     | 63/136 [00:15<00:14,  5.04it/s, epoch=24, grad_norm=80.195, loss=338.5717, lr=2.67e-04] 
Epoch 24 TRAIN:  49%|████▊     | 66/136 [00:15<00:13,  5.09it/s, epoch=24, grad_norm=80.195, loss=338.5717, lr=2.67e-04]
Epoch 24 TRAIN:  49%|████▊     | 66/136 [00:15<00:13,  5.09it/s, epoch=24, grad_norm=115.184, loss=331.8883, lr=2.67e-04]
Epoch 24 TRAIN:  49%|████▊     | 66/136 [00:15<00:13,  5.09it/s, epoch=24, grad_norm=147.871, loss=348.5894, lr=2.67e-04]
Epoch 24 TRAIN:  49%|████▊     | 66/136 [00:15<00:13,  5.09it/s, epoch=24, grad_norm=123.247, loss=300.4997, lr=2.67e-04]
Epoch 24 TRAIN:  51%|█████     | 69/136 [00:15<00:13,  5.12it/s, epoch=24, grad_norm=123.247, loss=300.4997, lr=2.67e-04]
Epoch 24 TRAIN:  51%|█████     | 69/136 [00:16<00:13,  5.12it/s, epoch=24, grad_norm=86.352, loss=310.1501, lr=2.67e-04] 
Epoch 24 TRAIN:  51%|█████     | 69/136 [00:16<00:13,  5.12it/s, epoch=24, grad_norm=73.109, loss=303.2953, lr=2.67e-04]
Epoch 24 TRAIN:  51%|█████     | 69/136 [00:16<00:13,  5.12it/s, epoch=24, grad_norm=104.548, loss=307.7968, lr=2.67e-04]
Epoch 24 TRAIN:  53%|█████▎    | 72/136 [00:16<00:12,  5.13it/s, epoch=24, grad_norm=104.548, loss=307.7968, lr=2.67e-04]
Epoch 24 TRAIN:  53%|█████▎    | 72/136 [00:16<00:12,  5.13it/s, epoch=24, grad_norm=131.060, loss=253.8737, lr=2.67e-04]
Epoch 24 TRAIN:  53%|█████▎    | 72/136 [00:16<00:12,  5.13it/s, epoch=24, grad_norm=123.616, loss=336.1211, lr=2.67e-04]
Epoch 24 TRAIN:  53%|█████▎    | 72/136 [00:17<00:12,  5.13it/s, epoch=24, grad_norm=102.540, loss=287.0326, lr=2.68e-04]
Epoch 24 TRAIN:  55%|█████▌    | 75/136 [00:17<00:12,  4.78it/s, epoch=24, grad_norm=102.540, loss=287.0326, lr=2.68e-04]
Epoch 24 TRAIN:  55%|█████▌    | 75/136 [00:17<00:12,  4.78it/s, epoch=24, grad_norm=146.271, loss=319.5004, lr=2.68e-04]
Epoch 24 TRAIN:  55%|█████▌    | 75/136 [00:17<00:12,  4.78it/s, epoch=24, grad_norm=223.104, loss=432.1592, lr=2.68e-04]
Epoch 24 TRAIN:  55%|█████▌    | 75/136 [00:17<00:12,  4.78it/s, epoch=24, grad_norm=98.466, loss=370.6464, lr=2.68e-04] 
Epoch 24 TRAIN:  57%|█████▋    | 78/136 [00:17<00:11,  4.99it/s, epoch=24, grad_norm=98.466, loss=370.6464, lr=2.68e-04]
Epoch 24 TRAIN:  57%|█████▋    | 78/136 [00:17<00:11,  4.99it/s, epoch=24, grad_norm=220.463, loss=271.7330, lr=2.68e-04]
Epoch 24 TRAIN:  57%|█████▋    | 78/136 [00:18<00:11,  4.99it/s, epoch=24, grad_norm=162.900, loss=297.1339, lr=2.68e-04]
Epoch 24 TRAIN:  57%|█████▋    | 78/136 [00:18<00:11,  4.99it/s, epoch=24, grad_norm=168.940, loss=325.6943, lr=2.68e-04]
Epoch 24 TRAIN:  60%|█████▉    | 81/136 [00:18<00:10,  5.09it/s, epoch=24, grad_norm=168.940, loss=325.6943, lr=2.68e-04]
Epoch 24 TRAIN:  60%|█████▉    | 81/136 [00:18<00:10,  5.09it/s, epoch=24, grad_norm=164.180, loss=315.7177, lr=2.68e-04]
Epoch 24 TRAIN:  60%|█████▉    | 81/136 [00:18<00:10,  5.09it/s, epoch=24, grad_norm=104.656, loss=353.6669, lr=2.68e-04]
Epoch 24 TRAIN:  60%|█████▉    | 81/136 [00:18<00:10,  5.09it/s, epoch=24, grad_norm=115.812, loss=312.5984, lr=2.68e-04]
Epoch 24 TRAIN:  62%|██████▏   | 84/136 [00:18<00:10,  4.77it/s, epoch=24, grad_norm=115.812, loss=312.5984, lr=2.68e-04]
Epoch 24 TRAIN:  62%|██████▏   | 84/136 [00:19<00:10,  4.77it/s, epoch=24, grad_norm=133.458, loss=252.2278, lr=2.68e-04]
Epoch 24 TRAIN:  62%|██████▏   | 84/136 [00:19<00:10,  4.77it/s, epoch=24, grad_norm=164.326, loss=346.2690, lr=2.68e-04]
Epoch 24 TRAIN:  62%|██████▏   | 84/136 [00:19<00:10,  4.77it/s, epoch=24, grad_norm=170.084, loss=320.4769, lr=2.68e-04]
Epoch 24 TRAIN:  64%|██████▍   | 87/136 [00:19<00:10,  4.85it/s, epoch=24, grad_norm=170.084, loss=320.4769, lr=2.68e-04]
Epoch 24 TRAIN:  64%|██████▍   | 87/136 [00:19<00:10,  4.85it/s, epoch=24, grad_norm=149.589, loss=296.4355, lr=2.69e-04]
Epoch 24 TRAIN:  64%|██████▍   | 87/136 [00:20<00:10,  4.85it/s, epoch=24, grad_norm=90.221, loss=312.1547, lr=2.69e-04] 
Epoch 24 TRAIN:  64%|██████▍   | 87/136 [00:20<00:10,  4.85it/s, epoch=24, grad_norm=133.166, loss=366.2525, lr=2.69e-04]
Epoch 24 TRAIN:  66%|██████▌   | 90/136 [00:20<00:09,  4.76it/s, epoch=24, grad_norm=133.166, loss=366.2525, lr=2.69e-04]
Epoch 24 TRAIN:  66%|██████▌   | 90/136 [00:20<00:09,  4.76it/s, epoch=24, grad_norm=167.015, loss=298.0139, lr=2.69e-04]
Epoch 24 TRAIN:  66%|██████▌   | 90/136 [00:20<00:09,  4.76it/s, epoch=24, grad_norm=198.209, loss=301.0420, lr=2.69e-04]
Epoch 24 TRAIN:  66%|██████▌   | 90/136 [00:20<00:09,  4.76it/s, epoch=24, grad_norm=228.756, loss=265.6884, lr=2.69e-04]
Epoch 24 TRAIN:  68%|██████▊   | 93/136 [00:20<00:08,  5.07it/s, epoch=24, grad_norm=228.756, loss=265.6884, lr=2.69e-04]
Epoch 24 TRAIN:  68%|██████▊   | 93/136 [00:20<00:08,  5.07it/s, epoch=24, grad_norm=277.906, loss=229.9601, lr=2.69e-04]
Epoch 24 TRAIN:  68%|██████▊   | 93/136 [00:21<00:08,  5.07it/s, epoch=24, grad_norm=133.350, loss=244.4400, lr=2.69e-04]
Epoch 24 TRAIN:  68%|██████▊   | 93/136 [00:21<00:08,  5.07it/s, epoch=24, grad_norm=89.109, loss=276.0797, lr=2.69e-04] 
Epoch 24 TRAIN:  71%|███████   | 96/136 [00:21<00:07,  5.24it/s, epoch=24, grad_norm=89.109, loss=276.0797, lr=2.69e-04]
Epoch 24 TRAIN:  71%|███████   | 96/136 [00:21<00:07,  5.24it/s, epoch=24, grad_norm=172.639, loss=363.3582, lr=2.69e-04]
Epoch 24 TRAIN:  71%|███████   | 96/136 [00:21<00:07,  5.24it/s, epoch=24, grad_norm=105.907, loss=297.3127, lr=2.69e-04]
Epoch 24 TRAIN:  71%|███████   | 96/136 [00:21<00:07,  5.24it/s, epoch=24, grad_norm=240.896, loss=380.7039, lr=2.69e-04]
Epoch 24 TRAIN:  73%|███████▎  | 99/136 [00:21<00:07,  5.09it/s, epoch=24, grad_norm=240.896, loss=380.7039, lr=2.69e-04]
Epoch 24 TRAIN:  73%|███████▎  | 99/136 [00:22<00:07,  5.09it/s, epoch=24, grad_norm=161.060, loss=263.3878, lr=2.70e-04]2026-04-16 20:04:02,523 DEBUG TRAIN | steps/sec 4.552| Batch 24/100 loss 263.387817 loss_att 234.299103 loss_ctc 331.261444 th_accuracy 0.227778 lr 2.6952e-04 grad_norm 161.059814 rank 0

Epoch 24 TRAIN:  73%|███████▎  | 99/136 [00:22<00:07,  5.09it/s, epoch=24, grad_norm=125.976, loss=279.5864, lr=2.70e-04]
Epoch 24 TRAIN:  73%|███████▎  | 99/136 [00:22<00:07,  5.09it/s, epoch=24, grad_norm=138.062, loss=344.1439, lr=2.70e-04]
Epoch 24 TRAIN:  75%|███████▌  | 102/136 [00:22<00:06,  5.04it/s, epoch=24, grad_norm=138.062, loss=344.1439, lr=2.70e-04]
Epoch 24 TRAIN:  75%|███████▌  | 102/136 [00:22<00:06,  5.04it/s, epoch=24, grad_norm=147.618, loss=239.6484, lr=2.70e-04]
Epoch 24 TRAIN:  75%|███████▌  | 102/136 [00:22<00:06,  5.04it/s, epoch=24, grad_norm=241.247, loss=356.8162, lr=2.70e-04]
Epoch 24 TRAIN:  75%|███████▌  | 102/136 [00:23<00:06,  5.04it/s, epoch=24, grad_norm=131.844, loss=232.2146, lr=2.70e-04]
Epoch 24 TRAIN:  77%|███████▋  | 105/136 [00:23<00:06,  5.06it/s, epoch=24, grad_norm=131.844, loss=232.2146, lr=2.70e-04]
Epoch 24 TRAIN:  77%|███████▋  | 105/136 [00:23<00:06,  5.06it/s, epoch=24, grad_norm=148.335, loss=256.9799, lr=2.70e-04]
Epoch 24 TRAIN:  77%|███████▋  | 105/136 [00:23<00:06,  5.06it/s, epoch=24, grad_norm=128.806, loss=272.0987, lr=2.70e-04]
Epoch 24 TRAIN:  77%|███████▋  | 105/136 [00:23<00:06,  5.06it/s, epoch=24, grad_norm=105.579, loss=333.8699, lr=2.70e-04]
Epoch 24 TRAIN:  79%|███████▉  | 108/136 [00:23<00:05,  4.89it/s, epoch=24, grad_norm=105.579, loss=333.8699, lr=2.70e-04]
Epoch 24 TRAIN:  79%|███████▉  | 108/136 [00:23<00:05,  4.89it/s, epoch=24, grad_norm=118.904, loss=281.7228, lr=2.70e-04]
Epoch 24 TRAIN:  79%|███████▉  | 108/136 [00:24<00:05,  4.89it/s, epoch=24, grad_norm=181.752, loss=319.4980, lr=2.70e-04]
Epoch 24 TRAIN:  79%|███████▉  | 108/136 [00:24<00:05,  4.89it/s, epoch=24, grad_norm=179.526, loss=387.6456, lr=2.70e-04]
Epoch 24 TRAIN:  82%|████████▏ | 111/136 [00:24<00:04,  5.02it/s, epoch=24, grad_norm=179.526, loss=387.6456, lr=2.70e-04]
Epoch 24 TRAIN:  82%|████████▏ | 111/136 [00:24<00:04,  5.02it/s, epoch=24, grad_norm=133.390, loss=316.9352, lr=2.70e-04]
Epoch 24 TRAIN:  82%|████████▏ | 111/136 [00:24<00:04,  5.02it/s, epoch=24, grad_norm=116.529, loss=308.1440, lr=2.71e-04]
Epoch 24 TRAIN:  82%|████████▏ | 111/136 [00:24<00:04,  5.02it/s, epoch=24, grad_norm=134.868, loss=360.8626, lr=2.71e-04]
Epoch 24 TRAIN:  84%|████████▍ | 114/136 [00:24<00:04,  4.87it/s, epoch=24, grad_norm=134.868, loss=360.8626, lr=2.71e-04]
Epoch 24 TRAIN:  84%|████████▍ | 114/136 [00:25<00:04,  4.87it/s, epoch=24, grad_norm=352.771, loss=325.1500, lr=2.71e-04]
Epoch 24 TRAIN:  84%|████████▍ | 114/136 [00:25<00:04,  4.87it/s, epoch=24, grad_norm=187.598, loss=350.5093, lr=2.71e-04]
Epoch 24 TRAIN:  84%|████████▍ | 114/136 [00:25<00:04,  4.87it/s, epoch=24, grad_norm=239.528, loss=263.5957, lr=2.71e-04]
Epoch 24 TRAIN:  86%|████████▌ | 117/136 [00:25<00:04,  4.71it/s, epoch=24, grad_norm=239.528, loss=263.5957, lr=2.71e-04]
Epoch 24 TRAIN:  86%|████████▌ | 117/136 [00:25<00:04,  4.71it/s, epoch=24, grad_norm=290.579, loss=328.1630, lr=2.71e-04]
Epoch 24 TRAIN:  86%|████████▌ | 117/136 [00:26<00:04,  4.71it/s, epoch=24, grad_norm=92.121, loss=294.1960, lr=2.71e-04] 
Epoch 24 TRAIN:  86%|████████▌ | 117/136 [00:26<00:04,  4.71it/s, epoch=24, grad_norm=158.146, loss=332.3896, lr=2.71e-04]
Epoch 24 TRAIN:  88%|████████▊ | 120/136 [00:26<00:03,  4.88it/s, epoch=24, grad_norm=158.146, loss=332.3896, lr=2.71e-04]
Epoch 24 TRAIN:  88%|████████▊ | 120/136 [00:26<00:03,  4.88it/s, epoch=24, grad_norm=99.232, loss=279.3935, lr=2.71e-04] 
Epoch 24 TRAIN:  88%|████████▊ | 120/136 [00:26<00:03,  4.88it/s, epoch=24, grad_norm=273.865, loss=432.5881, lr=2.71e-04]
Epoch 24 TRAIN:  88%|████████▊ | 120/136 [00:26<00:03,  4.88it/s, epoch=24, grad_norm=87.109, loss=225.1117, lr=2.71e-04] 
Epoch 24 TRAIN:  88%|████████▊ | 120/136 [00:26<00:03,  4.88it/s, epoch=24, grad_norm=184.856, loss=318.7928, lr=2.71e-04]
Epoch 24 TRAIN:  91%|█████████ | 124/136 [00:26<00:02,  5.12it/s, epoch=24, grad_norm=184.856, loss=318.7928, lr=2.71e-04]
Epoch 24 TRAIN:  91%|█████████ | 124/136 [00:27<00:02,  5.12it/s, epoch=24, grad_norm=162.323, loss=376.4824, lr=2.72e-04]
Epoch 24 TRAIN:  91%|█████████ | 124/136 [00:27<00:02,  5.12it/s, epoch=24, grad_norm=190.211, loss=350.7666, lr=2.72e-04]
Epoch 24 TRAIN:  91%|█████████ | 124/136 [00:27<00:02,  5.12it/s, epoch=24, grad_norm=146.994, loss=273.4168, lr=2.72e-04]
Epoch 24 TRAIN:  93%|█████████▎| 127/136 [00:27<00:01,  5.12it/s, epoch=24, grad_norm=146.994, loss=273.4168, lr=2.72e-04]
Epoch 24 TRAIN:  93%|█████████▎| 127/136 [00:27<00:01,  5.12it/s, epoch=24, grad_norm=168.612, loss=403.8045, lr=2.72e-04]
Epoch 24 TRAIN:  93%|█████████▎| 127/136 [00:27<00:01,  5.12it/s, epoch=24, grad_norm=151.370, loss=404.7387, lr=2.72e-04]
Epoch 24 TRAIN:  93%|█████████▎| 127/136 [00:27<00:01,  5.12it/s, epoch=24, grad_norm=145.087, loss=369.2953, lr=2.72e-04]
Epoch 24 TRAIN:  93%|█████████▎| 127/136 [00:28<00:01,  5.12it/s, epoch=24, grad_norm=258.844, loss=347.0415, lr=2.72e-04]
Epoch 24 TRAIN:  96%|█████████▋| 131/136 [00:28<00:00,  5.24it/s, epoch=24, grad_norm=258.844, loss=347.0415, lr=2.72e-04]
Epoch 24 TRAIN:  96%|█████████▋| 131/136 [00:28<00:00,  5.24it/s, epoch=24, grad_norm=233.990, loss=308.7285, lr=2.72e-04]
Epoch 24 TRAIN:  96%|█████████▋| 131/136 [00:28<00:00,  5.24it/s, epoch=24, grad_norm=126.126, loss=367.7361, lr=2.72e-04]
Epoch 24 TRAIN:  96%|█████████▋| 131/136 [00:28<00:00,  5.24it/s, epoch=24, grad_norm=171.309, loss=416.3636, lr=2.72e-04]
Epoch 24 TRAIN:  96%|█████████▋| 131/136 [00:28<00:00,  5.24it/s, epoch=24, grad_norm=220.679, loss=450.3699, lr=2.72e-04]
Epoch 24 TRAIN:  99%|█████████▉| 135/136 [00:28<00:00,  5.73it/s, epoch=24, grad_norm=220.679, loss=450.3699, lr=2.72e-04]
Epoch 24 TRAIN:  99%|█████████▉| 135/136 [00:28<00:00,  5.73it/s, epoch=24, grad_norm=140.292, loss=298.0763, lr=2.72e-04]
Epoch 24 TRAIN:  99%|█████████▉| 135/136 [00:29<00:00,  5.73it/s, epoch=24, grad_norm=213.999, loss=309.7213, lr=2.72e-04]
Epoch 24 TRAIN: 137it [00:29,  4.70it/s, epoch=24, grad_norm=213.999, loss=309.7213, lr=2.72e-04]                         

CV:   0%|          | 0/6 [00:00<?, ?it/s]/home/rookie/Documents/batuka/projects/wenet/wenet/wenet/utils/train_utils.py:689: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  amp_autocast(enabled=dtype is not None,

CV:  17%|█▋        | 1/6 [00:00<00:03,  1.33it/s]
                                                 
2026-04-16 20:04:10,411 INFO Epoch 24 Step 3406 CV info lr 2.7248e-04 cv_loss 537.8476118607955 rank 0 acc 0.251863457262516
2026-04-16 20:04:10,412 INFO [Rank 0] Checkpoint: save to checkpoint exp/conformer_mongolian/epoch_24.pt
2026-04-16 20:04:10,523 INFO Epoch 25 Step 3406 TRAIN info lr 2.7248e-04 rank 0
2026-04-16 20:04:10,544 INFO using accumulate grad, new batch size is 1 times larger than before

Epoch 25 TRAIN:   0%|          | 0/137 [00:00<?, ?it/s]
Epoch 25 TRAIN:   0%|          | 0/137 [00:03<?, ?it/s, epoch=25, grad_norm=417.981, loss=396.5640, lr=2.73e-04]
Epoch 25 TRAIN:   1%|          | 1/137 [00:03<08:37,  3.80s/it, epoch=25, grad_norm=417.981, loss=396.5640, lr=2.73e-04]
Epoch 25 TRAIN:   1%|          | 1/137 [00:03<08:37,  3.80s/it, epoch=25, grad_norm=451.044, loss=420.0974, lr=2.73e-04]
Epoch 25 TRAIN:   1%|          | 1/137 [00:04<08:37,  3.80s/it, epoch=25, grad_norm=445.040, loss=416.6301, lr=2.73e-04]
Epoch 25 TRAIN:   1%|          | 1/137 [00:04<08:37,  3.80s/it, epoch=25, grad_norm=152.609, loss=216.7280, lr=2.73e-04]
Epoch 25 TRAIN:   1%|          | 1/137 [00:04<08:37,  3.80s/it, epoch=25, grad_norm=463.810, loss=430.5739, lr=2.73e-04]
Epoch 25 TRAIN:   1%|          | 1/137 [00:04<08:37,  3.80s/it, epoch=25, grad_norm=352.507, loss=449.4249, lr=2.73e-04]
Epoch 25 TRAIN:   1%|          | 1/137 [00:04<08:37,  3.80s/it, epoch=25, grad_norm=210.688, loss=373.9474, lr=2.73e-04]
Epoch 25 TRAIN:   5%|▌         | 7/137 [00:04<01:01,  2.11it/s, epoch=25, grad_norm=210.688, loss=373.9474, lr=2.73e-04]
Epoch 25 TRAIN:   5%|▌         | 7/137 [00:04<01:01,  2.11it/s, epoch=25, grad_norm=111.775, loss=300.6738, lr=2.73e-04]
Epoch 25 TRAIN:   5%|▌         | 7/137 [00:04<01:01,  2.11it/s, epoch=25, grad_norm=82.785, loss=319.7310, lr=2.73e-04] 
Epoch 25 TRAIN:   5%|▌         | 7/137 [00:04<01:01,  2.11it/s, epoch=25, grad_norm=117.553, loss=341.2209, lr=2.73e-04]
Epoch 25 TRAIN:   5%|▌         | 7/137 [00:04<01:01,  2.11it/s, epoch=25, grad_norm=176.958, loss=308.7778, lr=2.73e-04]
Epoch 25 TRAIN:   8%|▊         | 11/137 [00:04<00:40,  3.14it/s, epoch=25, grad_norm=176.958, loss=308.7778, lr=2.73e-04]
Epoch 25 TRAIN:   8%|▊         | 11/137 [00:05<00:40,  3.14it/s, epoch=25, grad_norm=64.811, loss=365.5933, lr=2.73e-04] 
Epoch 25 TRAIN:   8%|▊         | 11/137 [00:05<00:40,  3.14it/s, epoch=25, grad_norm=110.308, loss=285.1913, lr=2.74e-04]
Epoch 25 TRAIN:   8%|▊         | 11/137 [00:05<00:40,  3.14it/s, epoch=25, grad_norm=97.523, loss=354.1206, lr=2.74e-04] 
Epoch 25 TRAIN:   8%|▊         | 11/137 [00:05<00:40,  3.14it/s, epoch=25, grad_norm=78.384, loss=295.7652, lr=2.74e-04]
Epoch 25 TRAIN:  11%|█         | 15/137 [00:05<00:31,  3.93it/s, epoch=25, grad_norm=78.384, loss=295.7652, lr=2.74e-04]
Epoch 25 TRAIN:  11%|█         | 15/137 [00:05<00:31,  3.93it/s, epoch=25, grad_norm=112.343, loss=354.2620, lr=2.74e-04]
Epoch 25 TRAIN:  11%|█         | 15/137 [00:05<00:31,  3.93it/s, epoch=25, grad_norm=74.629, loss=295.3812, lr=2.74e-04] 
Epoch 25 TRAIN:  11%|█         | 15/137 [00:05<00:31,  3.93it/s, epoch=25, grad_norm=69.869, loss=311.5887, lr=2.74e-04]
Epoch 25 TRAIN:  11%|█         | 15/137 [00:06<00:31,  3.93it/s, epoch=25, grad_norm=69.633, loss=301.8728, lr=2.74e-04]
Epoch 25 TRAIN:  14%|█▍        | 19/137 [00:06<00:25,  4.56it/s, epoch=25, grad_norm=69.633, loss=301.8728, lr=2.74e-04]
Epoch 25 TRAIN:  14%|█▍        | 19/137 [00:06<00:25,  4.56it/s, epoch=25, grad_norm=76.444, loss=330.3958, lr=2.74e-04]
Epoch 25 TRAIN:  14%|█▍        | 19/137 [00:06<00:25,  4.56it/s, epoch=25, grad_norm=132.745, loss=351.6999, lr=2.74e-04]
Epoch 25 TRAIN:  14%|█▍        | 19/137 [00:06<00:25,  4.56it/s, epoch=25, grad_norm=163.630, loss=377.5923, lr=2.74e-04]
Epoch 25 TRAIN:  14%|█▍        | 19/137 [00:06<00:25,  4.56it/s, epoch=25, grad_norm=98.776, loss=289.5199, lr=2.74e-04] 
Epoch 25 TRAIN:  17%|█▋        | 23/137 [00:06<00:23,  4.90it/s, epoch=25, grad_norm=98.776, loss=289.5199, lr=2.74e-04]
Epoch 25 TRAIN:  17%|█▋        | 23/137 [00:07<00:23,  4.90it/s, epoch=25, grad_norm=66.915, loss=305.8126, lr=2.74e-04]
Epoch 25 TRAIN:  17%|█▋        | 23/137 [00:07<00:23,  4.90it/s, epoch=25, grad_norm=94.417, loss=332.4937, lr=2.74e-04]
Epoch 25 TRAIN:  17%|█▋        | 23/137 [00:07<00:23,  4.90it/s, epoch=25, grad_norm=122.524, loss=294.3198, lr=2.75e-04]
Epoch 25 TRAIN:  17%|█▋        | 23/137 [00:07<00:23,  4.90it/s, epoch=25, grad_norm=143.425, loss=260.0329, lr=2.75e-04]
Epoch 25 TRAIN:  20%|█▉        | 27/137 [00:07<00:21,  5.13it/s, epoch=25, grad_norm=143.425, loss=260.0329, lr=2.75e-04]
Epoch 25 TRAIN:  20%|█▉        | 27/137 [00:07<00:21,  5.13it/s, epoch=25, grad_norm=84.718, loss=331.4802, lr=2.75e-04] 
Epoch 25 TRAIN:  20%|█▉        | 27/137 [00:07<00:21,  5.13it/s, epoch=25, grad_norm=101.340, loss=306.7490, lr=2.75e-04]
Epoch 25 TRAIN:  20%|█▉        | 27/137 [00:08<00:21,  5.13it/s, epoch=25, grad_norm=101.177, loss=353.6270, lr=2.75e-04]
Epoch 25 TRAIN:  20%|█▉        | 27/137 [00:08<00:21,  5.13it/s, epoch=25, grad_norm=73.185, loss=323.9297, lr=2.75e-04] 
Epoch 25 TRAIN:  23%|██▎       | 31/137 [00:08<00:20,  5.21it/s, epoch=25, grad_norm=73.185, loss=323.9297, lr=2.75e-04]
Epoch 25 TRAIN:  23%|██▎       | 31/137 [00:08<00:20,  5.21it/s, epoch=25, grad_norm=80.386, loss=320.2166, lr=2.75e-04]
Epoch 25 TRAIN:  23%|██▎       | 31/137 [00:08<00:20,  5.21it/s, epoch=25, grad_norm=127.572, loss=356.3672, lr=2.75e-04]
Epoch 25 TRAIN:  23%|██▎       | 31/137 [00:08<00:20,  5.21it/s, epoch=25, grad_norm=120.232, loss=314.5103, lr=2.75e-04]
Epoch 25 TRAIN:  23%|██▎       | 31/137 [00:08<00:20,  5.21it/s, epoch=25, grad_norm=96.879, loss=265.8118, lr=2.75e-04] 
Epoch 25 TRAIN:  26%|██▌       | 35/137 [00:08<00:18,  5.54it/s, epoch=25, grad_norm=96.879, loss=265.8118, lr=2.75e-04]
Epoch 25 TRAIN:  26%|██▌       | 35/137 [00:09<00:18,  5.54it/s, epoch=25, grad_norm=98.909, loss=352.1653, lr=2.75e-04]
Epoch 25 TRAIN:  26%|██▌       | 35/137 [00:09<00:18,  5.54it/s, epoch=25, grad_norm=122.764, loss=287.2437, lr=2.75e-04]
Epoch 25 TRAIN:  26%|██▌       | 35/137 [00:09<00:18,  5.54it/s, epoch=25, grad_norm=101.560, loss=334.1625, lr=2.76e-04]
Epoch 25 TRAIN:  26%|██▌       | 35/137 [00:09<00:18,  5.54it/s, epoch=25, grad_norm=88.402, loss=327.9585, lr=2.76e-04] 
Epoch 25 TRAIN:  28%|██▊       | 39/137 [00:09<00:18,  5.25it/s, epoch=25, grad_norm=88.402, loss=327.9585, lr=2.76e-04]
Epoch 25 TRAIN:  28%|██▊       | 39/137 [00:09<00:18,  5.25it/s, epoch=25, grad_norm=81.068, loss=330.4699, lr=2.76e-04]
Epoch 25 TRAIN:  28%|██▊       | 39/137 [00:10<00:18,  5.25it/s, epoch=25, grad_norm=103.710, loss=308.0891, lr=2.76e-04]
Epoch 25 TRAIN:  28%|██▊       | 39/137 [00:10<00:18,  5.25it/s, epoch=25, grad_norm=83.787, loss=318.4577, lr=2.76e-04] 
Epoch 25 TRAIN:  31%|███       | 42/137 [00:10<00:18,  5.23it/s, epoch=25, grad_norm=83.787, loss=318.4577, lr=2.76e-04]
Epoch 25 TRAIN:  31%|███       | 42/137 [00:10<00:18,  5.23it/s, epoch=25, grad_norm=78.724, loss=338.9337, lr=2.76e-04]
Epoch 25 TRAIN:  31%|███       | 42/137 [00:10<00:18,  5.23it/s, epoch=25, grad_norm=76.202, loss=306.3051, lr=2.76e-04]
Epoch 25 TRAIN:  31%|███       | 42/137 [00:10<00:18,  5.23it/s, epoch=25, grad_norm=112.079, loss=281.1765, lr=2.76e-04]
Epoch 25 TRAIN:  33%|███▎      | 45/137 [00:10<00:17,  5.21it/s, epoch=25, grad_norm=112.079, loss=281.1765, lr=2.76e-04]
Epoch 25 TRAIN:  33%|███▎      | 45/137 [00:11<00:17,  5.21it/s, epoch=25, grad_norm=126.212, loss=306.0751, lr=2.76e-04]
Epoch 25 TRAIN:  33%|███▎      | 45/137 [00:11<00:17,  5.21it/s, epoch=25, grad_norm=178.427, loss=364.4752, lr=2.76e-04]
Epoch 25 TRAIN:  33%|███▎      | 45/137 [00:11<00:17,  5.21it/s, epoch=25, grad_norm=115.545, loss=307.1359, lr=2.76e-04]
Epoch 25 TRAIN:  35%|███▌      | 48/137 [00:11<00:16,  5.24it/s, epoch=25, grad_norm=115.545, loss=307.1359, lr=2.76e-04]
Epoch 25 TRAIN:  35%|███▌      | 48/137 [00:11<00:16,  5.24it/s, epoch=25, grad_norm=64.896, loss=293.7932, lr=2.76e-04] 
Epoch 25 TRAIN:  35%|███▌      | 48/137 [00:11<00:16,  5.24it/s, epoch=25, grad_norm=78.512, loss=301.2794, lr=2.76e-04]
Epoch 25 TRAIN:  35%|███▌      | 48/137 [00:12<00:16,  5.24it/s, epoch=25, grad_norm=95.462, loss=360.9430, lr=2.77e-04]
Epoch 25 TRAIN:  37%|███▋      | 51/137 [00:12<00:17,  4.82it/s, epoch=25, grad_norm=95.462, loss=360.9430, lr=2.77e-04]
Epoch 25 TRAIN:  37%|███▋      | 51/137 [00:12<00:17,  4.82it/s, epoch=25, grad_norm=71.010, loss=337.0616, lr=2.77e-04]
Epoch 25 TRAIN:  37%|███▋      | 51/137 [00:12<00:17,  4.82it/s, epoch=25, grad_norm=207.222, loss=277.5917, lr=2.77e-04]
Epoch 25 TRAIN:  37%|███▋      | 51/137 [00:12<00:17,  4.82it/s, epoch=25, grad_norm=88.501, loss=352.3685, lr=2.77e-04] 
Epoch 25 TRAIN:  39%|███▉      | 54/137 [00:12<00:16,  4.89it/s, epoch=25, grad_norm=88.501, loss=352.3685, lr=2.77e-04]
Epoch 25 TRAIN:  39%|███▉      | 54/137 [00:12<00:16,  4.89it/s, epoch=25, grad_norm=74.911, loss=396.6441, lr=2.77e-04]
Epoch 25 TRAIN:  39%|███▉      | 54/137 [00:13<00:16,  4.89it/s, epoch=25, grad_norm=102.347, loss=330.6302, lr=2.77e-04]
Epoch 25 TRAIN:  39%|███▉      | 54/137 [00:13<00:16,  4.89it/s, epoch=25, grad_norm=75.302, loss=352.0496, lr=2.77e-04] 
Epoch 25 TRAIN:  42%|████▏     | 57/137 [00:13<00:15,  5.11it/s, epoch=25, grad_norm=75.302, loss=352.0496, lr=2.77e-04]
Epoch 25 TRAIN:  42%|████▏     | 57/137 [00:13<00:15,  5.11it/s, epoch=25, grad_norm=113.949, loss=303.2984, lr=2.77e-04]
Epoch 25 TRAIN:  42%|████▏     | 57/137 [00:13<00:15,  5.11it/s, epoch=25, grad_norm=123.099, loss=325.5553, lr=2.77e-04]
Epoch 25 TRAIN:  42%|████▏     | 57/137 [00:14<00:15,  5.11it/s, epoch=25, grad_norm=102.035, loss=307.4942, lr=2.77e-04]
Epoch 25 TRAIN:  44%|████▍     | 60/137 [00:14<00:15,  4.84it/s, epoch=25, grad_norm=102.035, loss=307.4942, lr=2.77e-04]
Epoch 25 TRAIN:  44%|████▍     | 60/137 [00:14<00:15,  4.84it/s, epoch=25, grad_norm=95.316, loss=300.4699, lr=2.77e-04] 
Epoch 25 TRAIN:  44%|████▍     | 60/137 [00:14<00:15,  4.84it/s, epoch=25, grad_norm=165.928, loss=354.0122, lr=2.77e-04]
Epoch 25 TRAIN:  44%|████▍     | 60/137 [00:14<00:15,  4.84it/s, epoch=25, grad_norm=156.874, loss=359.1157, lr=2.78e-04]
Epoch 25 TRAIN:  46%|████▌     | 63/137 [00:14<00:15,  4.77it/s, epoch=25, grad_norm=156.874, loss=359.1157, lr=2.78e-04]
Epoch 25 TRAIN:  46%|████▌     | 63/137 [00:14<00:15,  4.77it/s, epoch=25, grad_norm=146.746, loss=353.5437, lr=2.78e-04]
Epoch 25 TRAIN:  46%|████▌     | 63/137 [00:15<00:15,  4.77it/s, epoch=25, grad_norm=110.122, loss=324.1768, lr=2.78e-04]
Epoch 25 TRAIN:  46%|████▌     | 63/137 [00:15<00:15,  4.77it/s, epoch=25, grad_norm=74.848, loss=284.1792, lr=2.78e-04] 
Epoch 25 TRAIN:  48%|████▊     | 66/137 [00:15<00:14,  4.99it/s, epoch=25, grad_norm=74.848, loss=284.1792, lr=2.78e-04]
Epoch 25 TRAIN:  48%|████▊     | 66/137 [00:15<00:14,  4.99it/s, epoch=25, grad_norm=197.747, loss=239.4477, lr=2.78e-04]
Epoch 25 TRAIN:  48%|████▊     | 66/137 [00:15<00:14,  4.99it/s, epoch=25, grad_norm=76.201, loss=295.6948, lr=2.78e-04] 
Epoch 25 TRAIN:  48%|████▊     | 66/137 [00:15<00:14,  4.99it/s, epoch=25, grad_norm=134.890, loss=320.5453, lr=2.78e-04]
Epoch 25 TRAIN:  50%|█████     | 69/137 [00:15<00:12,  5.23it/s, epoch=25, grad_norm=134.890, loss=320.5453, lr=2.78e-04]
Epoch 25 TRAIN:  50%|█████     | 69/137 [00:15<00:12,  5.23it/s, epoch=25, grad_norm=117.765, loss=357.6358, lr=2.78e-04]
Epoch 25 TRAIN:  50%|█████     | 69/137 [00:16<00:12,  5.23it/s, epoch=25, grad_norm=131.519, loss=297.5406, lr=2.78e-04]
Epoch 25 TRAIN:  50%|█████     | 69/137 [00:16<00:12,  5.23it/s, epoch=25, grad_norm=75.771, loss=313.3239, lr=2.78e-04] 
Epoch 25 TRAIN:  53%|█████▎    | 72/137 [00:16<00:12,  5.01it/s, epoch=25, grad_norm=75.771, loss=313.3239, lr=2.78e-04]
Epoch 25 TRAIN:  53%|█████▎    | 72/137 [00:16<00:12,  5.01it/s, epoch=25, grad_norm=60.908, loss=304.8533, lr=2.78e-04]
Epoch 25 TRAIN:  53%|█████▎    | 72/137 [00:16<00:12,  5.01it/s, epoch=25, grad_norm=63.220, loss=322.0724, lr=2.78e-04]
Epoch 25 TRAIN:  53%|█████▎    | 72/137 [00:17<00:12,  5.01it/s, epoch=25, grad_norm=100.717, loss=316.0535, lr=2.78e-04]
Epoch 25 TRAIN:  55%|█████▍    | 75/137 [00:17<00:12,  4.92it/s, epoch=25, grad_norm=100.717, loss=316.0535, lr=2.78e-04]
Epoch 25 TRAIN:  55%|█████▍    | 75/137 [00:17<00:12,  4.92it/s, epoch=25, grad_norm=70.455, loss=300.3669, lr=2.79e-04] 
Epoch 25 TRAIN:  55%|█████▍    | 75/137 [00:17<00:12,  4.92it/s, epoch=25, grad_norm=275.053, loss=238.4608, lr=2.79e-04]
Epoch 25 TRAIN:  55%|█████▍    | 75/137 [00:17<00:12,  4.92it/s, epoch=25, grad_norm=201.747, loss=378.5878, lr=2.79e-04]
Epoch 25 TRAIN:  55%|█████▍    | 75/137 [00:17<00:12,  4.92it/s, epoch=25, grad_norm=154.609, loss=262.8428, lr=2.79e-04]
Epoch 25 TRAIN:  58%|█████▊    | 79/137 [00:17<00:11,  5.21it/s, epoch=25, grad_norm=154.609, loss=262.8428, lr=2.79e-04]
Epoch 25 TRAIN:  58%|█████▊    | 79/137 [00:17<00:11,  5.21it/s, epoch=25, grad_norm=62.447, loss=320.3230, lr=2.79e-04] 
Epoch 25 TRAIN:  58%|█████▊    | 79/137 [00:18<00:11,  5.21it/s, epoch=25, grad_norm=85.169, loss=285.6154, lr=2.79e-04]
Epoch 25 TRAIN:  58%|█████▊    | 79/137 [00:18<00:11,  5.21it/s, epoch=25, grad_norm=165.673, loss=374.2987, lr=2.79e-04]
Epoch 25 TRAIN:  60%|█████▉    | 82/137 [00:18<00:10,  5.07it/s, epoch=25, grad_norm=165.673, loss=374.2987, lr=2.79e-04]
Epoch 25 TRAIN:  60%|█████▉    | 82/137 [00:18<00:10,  5.07it/s, epoch=25, grad_norm=221.378, loss=241.7614, lr=2.79e-04]
Epoch 25 TRAIN:  60%|█████▉    | 82/137 [00:18<00:10,  5.07it/s, epoch=25, grad_norm=86.517, loss=298.9778, lr=2.79e-04] 
Epoch 25 TRAIN:  60%|█████▉    | 82/137 [00:18<00:10,  5.07it/s, epoch=25, grad_norm=247.882, loss=282.5792, lr=2.79e-04]
Epoch 25 TRAIN:  62%|██████▏   | 85/137 [00:18<00:10,  5.17it/s, epoch=25, grad_norm=247.882, loss=282.5792, lr=2.79e-04]
Epoch 25 TRAIN:  62%|██████▏   | 85/137 [00:19<00:10,  5.17it/s, epoch=25, grad_norm=316.274, loss=423.8505, lr=2.79e-04]
Epoch 25 TRAIN:  62%|██████▏   | 85/137 [00:19<00:10,  5.17it/s, epoch=25, grad_norm=141.074, loss=275.6151, lr=2.79e-04]
Epoch 25 TRAIN:  62%|██████▏   | 85/137 [00:19<00:10,  5.17it/s, epoch=25, grad_norm=133.789, loss=336.6134, lr=2.80e-04]
Epoch 25 TRAIN:  64%|██████▍   | 88/137 [00:19<00:09,  5.25it/s, epoch=25, grad_norm=133.789, loss=336.6134, lr=2.80e-04]
Epoch 25 TRAIN:  64%|██████▍   | 88/137 [00:19<00:09,  5.25it/s, epoch=25, grad_norm=77.386, loss=278.8904, lr=2.80e-04] 
Epoch 25 TRAIN:  64%|██████▍   | 88/137 [00:19<00:09,  5.25it/s, epoch=25, grad_norm=74.704, loss=312.5909, lr=2.80e-04]
Epoch 25 TRAIN:  64%|██████▍   | 88/137 [00:20<00:09,  5.25it/s, epoch=25, grad_norm=166.938, loss=294.6665, lr=2.80e-04]
Epoch 25 TRAIN:  66%|██████▋   | 91/137 [00:20<00:09,  4.75it/s, epoch=25, grad_norm=166.938, loss=294.6665, lr=2.80e-04]
Epoch 25 TRAIN:  66%|██████▋   | 91/137 [00:20<00:09,  4.75it/s, epoch=25, grad_norm=175.584, loss=349.5893, lr=2.80e-04]
Epoch 25 TRAIN:  66%|██████▋   | 91/137 [00:20<00:09,  4.75it/s, epoch=25, grad_norm=118.419, loss=341.6590, lr=2.80e-04]
Epoch 25 TRAIN:  66%|██████▋   | 91/137 [00:20<00:09,  4.75it/s, epoch=25, grad_norm=295.649, loss=395.3472, lr=2.80e-04]
Epoch 25 TRAIN:  69%|██████▊   | 94/137 [00:20<00:08,  4.96it/s, epoch=25, grad_norm=295.649, loss=395.3472, lr=2.80e-04]
Epoch 25 TRAIN:  69%|██████▊   | 94/137 [00:20<00:08,  4.96it/s, epoch=25, grad_norm=113.992, loss=331.0221, lr=2.80e-04]
Epoch 25 TRAIN:  69%|██████▊   | 94/137 [00:21<00:08,  4.96it/s, epoch=25, grad_norm=192.662, loss=391.1675, lr=2.80e-04]
Epoch 25 TRAIN:  69%|██████▊   | 94/137 [00:21<00:08,  4.96it/s, epoch=25, grad_norm=171.877, loss=263.4370, lr=2.80e-04]
Epoch 25 TRAIN:  71%|███████   | 97/137 [00:21<00:07,  5.01it/s, epoch=25, grad_norm=171.877, loss=263.4370, lr=2.80e-04]
Epoch 25 TRAIN:  71%|███████   | 97/137 [00:21<00:07,  5.01it/s, epoch=25, grad_norm=104.660, loss=315.0359, lr=2.80e-04]
Epoch 25 TRAIN:  71%|███████   | 97/137 [00:21<00:07,  5.01it/s, epoch=25, grad_norm=237.565, loss=275.6812, lr=2.80e-04]
Epoch 25 TRAIN:  71%|███████   | 97/137 [00:21<00:07,  5.01it/s, epoch=25, grad_norm=91.789, loss=299.7404, lr=2.80e-04] 2026-04-16 20:04:32,468 DEBUG TRAIN | steps/sec 4.575| Batch 25/100 loss 299.740356 loss_att 264.255890 loss_ctc 382.537415 th_accuracy 0.287037 lr 2.8048e-04 grad_norm 91.788628 rank 0

Epoch 25 TRAIN:  73%|███████▎  | 100/137 [00:21<00:07,  5.03it/s, epoch=25, grad_norm=91.789, loss=299.7404, lr=2.80e-04]
Epoch 25 TRAIN:  73%|███████▎  | 100/137 [00:22<00:07,  5.03it/s, epoch=25, grad_norm=175.500, loss=299.1066, lr=2.81e-04]
Epoch 25 TRAIN:  73%|███████▎  | 100/137 [00:22<00:07,  5.03it/s, epoch=25, grad_norm=224.759, loss=334.9149, lr=2.81e-04]
Epoch 25 TRAIN:  73%|███████▎  | 100/137 [00:22<00:07,  5.03it/s, epoch=25, grad_norm=84.762, loss=318.5045, lr=2.81e-04] 
Epoch 25 TRAIN:  75%|███████▌  | 103/137 [00:22<00:07,  4.73it/s, epoch=25, grad_norm=84.762, loss=318.5045, lr=2.81e-04]
Epoch 25 TRAIN:  75%|███████▌  | 103/137 [00:22<00:07,  4.73it/s, epoch=25, grad_norm=156.609, loss=330.3949, lr=2.81e-04]
Epoch 25 TRAIN:  75%|███████▌  | 103/137 [00:23<00:07,  4.73it/s, epoch=25, grad_norm=126.997, loss=305.3843, lr=2.81e-04]
Epoch 25 TRAIN:  75%|███████▌  | 103/137 [00:23<00:07,  4.73it/s, epoch=25, grad_norm=147.692, loss=250.4951, lr=2.81e-04]
Epoch 25 TRAIN:  77%|███████▋  | 106/137 [00:23<00:06,  4.57it/s, epoch=25, grad_norm=147.692, loss=250.4951, lr=2.81e-04]
Epoch 25 TRAIN:  77%|███████▋  | 106/137 [00:23<00:06,  4.57it/s, epoch=25, grad_norm=90.401, loss=374.6537, lr=2.81e-04] 
Epoch 25 TRAIN:  77%|███████▋  | 106/137 [00:23<00:06,  4.57it/s, epoch=25, grad_norm=100.469, loss=311.7115, lr=2.81e-04]
Epoch 25 TRAIN:  77%|███████▋  | 106/137 [00:23<00:06,  4.57it/s, epoch=25, grad_norm=315.149, loss=356.0788, lr=2.81e-04]
Epoch 25 TRAIN:  80%|███████▉  | 109/137 [00:23<00:06,  4.63it/s, epoch=25, grad_norm=315.149, loss=356.0788, lr=2.81e-04]
Epoch 25 TRAIN:  80%|███████▉  | 109/137 [00:24<00:06,  4.63it/s, epoch=25, grad_norm=251.324, loss=346.4686, lr=2.81e-04]
Epoch 25 TRAIN:  80%|███████▉  | 109/137 [00:24<00:06,  4.63it/s, epoch=25, grad_norm=95.618, loss=355.9147, lr=2.81e-04] 
Epoch 25 TRAIN:  80%|███████▉  | 109/137 [00:24<00:06,  4.63it/s, epoch=25, grad_norm=92.895, loss=320.1762, lr=2.81e-04]
Epoch 25 TRAIN:  82%|████████▏ | 112/137 [00:24<00:05,  4.63it/s, epoch=25, grad_norm=92.895, loss=320.1762, lr=2.81e-04]
Epoch 25 TRAIN:  82%|████████▏ | 112/137 [00:24<00:05,  4.63it/s, epoch=25, grad_norm=139.152, loss=385.0752, lr=2.82e-04]
Epoch 25 TRAIN:  82%|████████▏ | 112/137 [00:25<00:05,  4.63it/s, epoch=25, grad_norm=212.922, loss=284.9548, lr=2.82e-04]
Epoch 25 TRAIN:  82%|████████▏ | 112/137 [00:25<00:05,  4.63it/s, epoch=25, grad_norm=172.923, loss=325.5455, lr=2.82e-04]
Epoch 25 TRAIN:  84%|████████▍ | 115/137 [00:25<00:04,  4.44it/s, epoch=25, grad_norm=172.923, loss=325.5455, lr=2.82e-04]
Epoch 25 TRAIN:  84%|████████▍ | 115/137 [00:25<00:04,  4.44it/s, epoch=25, grad_norm=106.963, loss=312.0024, lr=2.82e-04]
Epoch 25 TRAIN:  84%|████████▍ | 115/137 [00:25<00:04,  4.44it/s, epoch=25, grad_norm=278.756, loss=305.8961, lr=2.82e-04]
Epoch 25 TRAIN:  84%|████████▍ | 115/137 [00:26<00:04,  4.44it/s, epoch=25, grad_norm=87.188, loss=233.8499, lr=2.82e-04] 
Epoch 25 TRAIN:  86%|████████▌ | 118/137 [00:26<00:04,  4.26it/s, epoch=25, grad_norm=87.188, loss=233.8499, lr=2.82e-04]
Epoch 25 TRAIN:  86%|████████▌ | 118/137 [00:26<00:04,  4.26it/s, epoch=25, grad_norm=125.552, loss=328.1221, lr=2.82e-04]
Epoch 25 TRAIN:  86%|████████▌ | 118/137 [00:26<00:04,  4.26it/s, epoch=25, grad_norm=169.962, loss=296.7253, lr=2.82e-04]
Epoch 25 TRAIN:  86%|████████▌ | 118/137 [00:26<00:04,  4.26it/s, epoch=25, grad_norm=201.033, loss=220.8624, lr=2.82e-04]
Epoch 25 TRAIN:  86%|████████▌ | 118/137 [00:26<00:04,  4.26it/s, epoch=25, grad_norm=118.598, loss=263.7834, lr=2.82e-04]
Epoch 25 TRAIN:  89%|████████▉ | 122/137 [00:26<00:02,  5.01it/s, epoch=25, grad_norm=118.598, loss=263.7834, lr=2.82e-04]
Epoch 25 TRAIN:  89%|████████▉ | 122/137 [00:26<00:02,  5.01it/s, epoch=25, grad_norm=223.729, loss=378.1636, lr=2.82e-04]
Epoch 25 TRAIN:  89%|████████▉ | 122/137 [00:27<00:02,  5.01it/s, epoch=25, grad_norm=335.979, loss=440.4633, lr=2.82e-04]
Epoch 25 TRAIN:  89%|████████▉ | 122/137 [00:27<00:02,  5.01it/s, epoch=25, grad_norm=200.498, loss=326.4092, lr=2.82e-04]
Epoch 25 TRAIN:  91%|█████████ | 125/137 [00:27<00:02,  5.14it/s, epoch=25, grad_norm=200.498, loss=326.4092, lr=2.82e-04]
Epoch 25 TRAIN:  91%|█████████ | 125/137 [00:27<00:02,  5.14it/s, epoch=25, grad_norm=169.661, loss=408.3603, lr=2.83e-04]
Epoch 25 TRAIN:  91%|█████████ | 125/137 [00:27<00:02,  5.14it/s, epoch=25, grad_norm=98.729, loss=248.7079, lr=2.83e-04] 
Epoch 25 TRAIN:  91%|█████████ | 125/137 [00:27<00:02,  5.14it/s, epoch=25, grad_norm=161.319, loss=343.5113, lr=2.83e-04]
Epoch 25 TRAIN:  93%|█████████▎| 128/137 [00:27<00:01,  4.99it/s, epoch=25, grad_norm=161.319, loss=343.5113, lr=2.83e-04]
Epoch 25 TRAIN:  93%|█████████▎| 128/137 [00:27<00:01,  4.99it/s, epoch=25, grad_norm=220.268, loss=303.6845, lr=2.83e-04]
Epoch 25 TRAIN:  93%|█████████▎| 128/137 [00:28<00:01,  4.99it/s, epoch=25, grad_norm=88.056, loss=311.6799, lr=2.83e-04] 
Epoch 25 TRAIN:  93%|█████████▎| 128/137 [00:28<00:01,  4.99it/s, epoch=25, grad_norm=102.577, loss=315.2537, lr=2.83e-04]
Epoch 25 TRAIN:  96%|█████████▌| 131/137 [00:28<00:01,  5.24it/s, epoch=25, grad_norm=102.577, loss=315.2537, lr=2.83e-04]
Epoch 25 TRAIN:  96%|█████████▌| 131/137 [00:28<00:01,  5.24it/s, epoch=25, grad_norm=229.778, loss=229.5427, lr=2.83e-04]
Epoch 25 TRAIN:  96%|█████████▌| 131/137 [00:28<00:01,  5.24it/s, epoch=25, grad_norm=132.814, loss=359.3552, lr=2.83e-04]
Epoch 25 TRAIN:  96%|█████████▌| 131/137 [00:29<00:01,  5.24it/s, epoch=25, grad_norm=93.568, loss=352.7210, lr=2.83e-04] 
Epoch 25 TRAIN:  98%|█████████▊| 134/137 [00:29<00:00,  5.05it/s, epoch=25, grad_norm=93.568, loss=352.7210, lr=2.83e-04]
Epoch 25 TRAIN:  98%|█████████▊| 134/137 [00:29<00:00,  5.05it/s, epoch=25, grad_norm=168.334, loss=362.5108, lr=2.83e-04]
Epoch 25 TRAIN:  98%|█████████▊| 134/137 [00:29<00:00,  5.05it/s, epoch=25, grad_norm=175.643, loss=413.3835, lr=2.83e-04]
Epoch 25 TRAIN:  99%|█████████▉| 136/137 [00:29<00:00,  4.65it/s, epoch=25, grad_norm=175.643, loss=413.3835, lr=2.83e-04]

CV:   0%|          | 0/6 [00:00<?, ?it/s]/home/rookie/Documents/batuka/projects/wenet/wenet/wenet/utils/train_utils.py:689: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  amp_autocast(enabled=dtype is not None,

CV:  17%|█▋        | 1/6 [00:00<00:03,  1.32it/s]
                                                 
2026-04-16 20:04:40,646 INFO Epoch 25 Step 3542 CV info lr 2.8336e-04 cv_loss 541.2243929776279 rank 0 acc 0.25146681318680447
2026-04-16 20:04:40,647 INFO [Rank 0] Checkpoint: save to checkpoint exp/conformer_mongolian/epoch_25.pt
2026-04-16 20:04:40,759 INFO Epoch 26 Step 3542 TRAIN info lr 2.8336e-04 rank 0
2026-04-16 20:04:40,780 INFO using accumulate grad, new batch size is 1 times larger than before

Epoch 26 TRAIN:   0%|          | 0/136 [00:00<?, ?it/s]
Epoch 26 TRAIN:   0%|          | 0/136 [00:04<?, ?it/s, epoch=26, grad_norm=337.472, loss=317.4302, lr=2.83e-04]
Epoch 26 TRAIN:   1%|          | 1/136 [00:04<09:21,  4.16s/it, epoch=26, grad_norm=337.472, loss=317.4302, lr=2.83e-04]
Epoch 26 TRAIN:   1%|          | 1/136 [00:04<09:21,  4.16s/it, epoch=26, grad_norm=347.578, loss=354.3425, lr=2.84e-04]
Epoch 26 TRAIN:   1%|          | 1/136 [00:04<09:21,  4.16s/it, epoch=26, grad_norm=383.873, loss=376.6098, lr=2.84e-04]
Epoch 26 TRAIN:   1%|          | 1/136 [00:04<09:21,  4.16s/it, epoch=26, grad_norm=493.552, loss=442.1589, lr=2.84e-04]
Epoch 26 TRAIN:   1%|          | 1/136 [00:04<09:21,  4.16s/it, epoch=26, grad_norm=337.386, loss=392.8034, lr=2.84e-04]
Epoch 26 TRAIN:   1%|          | 1/136 [00:04<09:21,  4.16s/it, epoch=26, grad_norm=374.927, loss=411.7950, lr=2.84e-04]
Epoch 26 TRAIN:   1%|          | 1/136 [00:04<09:21,  4.16s/it, epoch=26, grad_norm=270.935, loss=419.8425, lr=2.84e-04]
Epoch 26 TRAIN:   1%|          | 1/136 [00:04<09:21,  4.16s/it, epoch=26, grad_norm=170.468, loss=323.3668, lr=2.84e-04]
Epoch 26 TRAIN:   1%|          | 1/136 [00:04<09:21,  4.16s/it, epoch=26, grad_norm=150.364, loss=372.2395, lr=2.84e-04]
Epoch 26 TRAIN:   7%|▋         | 9/136 [00:04<00:51,  2.44it/s, epoch=26, grad_norm=150.364, loss=372.2395, lr=2.84e-04]
Epoch 26 TRAIN:   7%|▋         | 9/136 [00:04<00:51,  2.44it/s, epoch=26, grad_norm=125.823, loss=309.4081, lr=2.84e-04]
Epoch 26 TRAIN:   7%|▋         | 9/136 [00:05<00:51,  2.44it/s, epoch=26, grad_norm=123.841, loss=340.7446, lr=2.84e-04]
Epoch 26 TRAIN:   7%|▋         | 9/136 [00:05<00:51,  2.44it/s, epoch=26, grad_norm=130.146, loss=325.2943, lr=2.84e-04]
Epoch 26 TRAIN:   7%|▋         | 9/136 [00:05<00:51,  2.44it/s, epoch=26, grad_norm=174.441, loss=306.4287, lr=2.84e-04]
Epoch 26 TRAIN:  10%|▉         | 13/136 [00:05<00:37,  3.24it/s, epoch=26, grad_norm=174.441, loss=306.4287, lr=2.84e-04]
Epoch 26 TRAIN:  10%|▉         | 13/136 [00:05<00:37,  3.24it/s, epoch=26, grad_norm=179.724, loss=307.3429, lr=2.84e-04]
Epoch 26 TRAIN:  10%|▉         | 13/136 [00:05<00:37,  3.24it/s, epoch=26, grad_norm=118.161, loss=314.2802, lr=2.85e-04]
Epoch 26 TRAIN:  10%|▉         | 13/136 [00:05<00:37,  3.24it/s, epoch=26, grad_norm=74.033, loss=361.1307, lr=2.85e-04] 
Epoch 26 TRAIN:  10%|▉         | 13/136 [00:06<00:37,  3.24it/s, epoch=26, grad_norm=63.185, loss=310.1236, lr=2.85e-04]
Epoch 26 TRAIN:  12%|█▎        | 17/136 [00:06<00:30,  3.93it/s, epoch=26, grad_norm=63.185, loss=310.1236, lr=2.85e-04]
Epoch 26 TRAIN:  12%|█▎        | 17/136 [00:06<00:30,  3.93it/s, epoch=26, grad_norm=60.998, loss=299.8862, lr=2.85e-04]
Epoch 26 TRAIN:  12%|█▎        | 17/136 [00:06<00:30,  3.93it/s, epoch=26, grad_norm=180.062, loss=366.5662, lr=2.85e-04]
Epoch 26 TRAIN:  12%|█▎        | 17/136 [00:06<00:30,  3.93it/s, epoch=26, grad_norm=173.329, loss=365.9399, lr=2.85e-04]
Epoch 26 TRAIN:  12%|█▎        | 17/136 [00:06<00:30,  3.93it/s, epoch=26, grad_norm=58.560, loss=279.7616, lr=2.85e-04] 
Epoch 26 TRAIN:  15%|█▌        | 21/136 [00:06<00:26,  4.41it/s, epoch=26, grad_norm=58.560, loss=279.7616, lr=2.85e-04]
Epoch 26 TRAIN:  15%|█▌        | 21/136 [00:06<00:26,  4.41it/s, epoch=26, grad_norm=101.267, loss=357.4576, lr=2.85e-04]
Epoch 26 TRAIN:  15%|█▌        | 21/136 [00:07<00:26,  4.41it/s, epoch=26, grad_norm=53.404, loss=315.1940, lr=2.85e-04] 
Epoch 26 TRAIN:  15%|█▌        | 21/136 [00:07<00:26,  4.41it/s, epoch=26, grad_norm=60.391, loss=327.5260, lr=2.85e-04]
Epoch 26 TRAIN:  15%|█▌        | 21/136 [00:07<00:26,  4.41it/s, epoch=26, grad_norm=109.155, loss=333.3239, lr=2.85e-04]
Epoch 26 TRAIN:  18%|█▊        | 25/136 [00:07<00:24,  4.50it/s, epoch=26, grad_norm=109.155, loss=333.3239, lr=2.85e-04]
Epoch 26 TRAIN:  18%|█▊        | 25/136 [00:07<00:24,  4.50it/s, epoch=26, grad_norm=110.800, loss=318.0286, lr=2.85e-04]
Epoch 26 TRAIN:  18%|█▊        | 25/136 [00:07<00:24,  4.50it/s, epoch=26, grad_norm=86.885, loss=355.8179, lr=2.86e-04] 
Epoch 26 TRAIN:  18%|█▊        | 25/136 [00:08<00:24,  4.50it/s, epoch=26, grad_norm=95.610, loss=321.0630, lr=2.86e-04]
Epoch 26 TRAIN:  21%|██        | 28/136 [00:08<00:23,  4.65it/s, epoch=26, grad_norm=95.610, loss=321.0630, lr=2.86e-04]
Epoch 26 TRAIN:  21%|██        | 28/136 [00:08<00:23,  4.65it/s, epoch=26, grad_norm=90.459, loss=347.6762, lr=2.86e-04]
Epoch 26 TRAIN:  21%|██        | 28/136 [00:08<00:23,  4.65it/s, epoch=26, grad_norm=80.076, loss=328.2061, lr=2.86e-04]
Epoch 26 TRAIN:  21%|██        | 28/136 [00:08<00:23,  4.65it/s, epoch=26, grad_norm=67.464, loss=316.8717, lr=2.86e-04]
Epoch 26 TRAIN:  23%|██▎       | 31/136 [00:08<00:21,  4.91it/s, epoch=26, grad_norm=67.464, loss=316.8717, lr=2.86e-04]
Epoch 26 TRAIN:  23%|██▎       | 31/136 [00:08<00:21,  4.91it/s, epoch=26, grad_norm=79.527, loss=325.6551, lr=2.86e-04]
Epoch 26 TRAIN:  23%|██▎       | 31/136 [00:09<00:21,  4.91it/s, epoch=26, grad_norm=71.389, loss=324.5703, lr=2.86e-04]
Epoch 26 TRAIN:  23%|██▎       | 31/136 [00:09<00:21,  4.91it/s, epoch=26, grad_norm=77.080, loss=345.0077, lr=2.86e-04]
Epoch 26 TRAIN:  25%|██▌       | 34/136 [00:09<00:20,  5.01it/s, epoch=26, grad_norm=77.080, loss=345.0077, lr=2.86e-04]
Epoch 26 TRAIN:  25%|██▌       | 34/136 [00:09<00:20,  5.01it/s, epoch=26, grad_norm=72.780, loss=317.5389, lr=2.86e-04]
Epoch 26 TRAIN:  25%|██▌       | 34/136 [00:09<00:20,  5.01it/s, epoch=26, grad_norm=84.560, loss=288.7401, lr=2.86e-04]
Epoch 26 TRAIN:  25%|██▌       | 34/136 [00:09<00:20,  5.01it/s, epoch=26, grad_norm=74.350, loss=284.8936, lr=2.86e-04]
Epoch 26 TRAIN:  27%|██▋       | 37/136 [00:09<00:20,  4.87it/s, epoch=26, grad_norm=74.350, loss=284.8936, lr=2.86e-04]
Epoch 26 TRAIN:  27%|██▋       | 37/136 [00:10<00:20,  4.87it/s, epoch=26, grad_norm=99.099, loss=334.1333, lr=2.86e-04]
Epoch 26 TRAIN:  27%|██▋       | 37/136 [00:10<00:20,  4.87it/s, epoch=26, grad_norm=58.037, loss=322.3715, lr=2.86e-04]
Epoch 26 TRAIN:  27%|██▋       | 37/136 [00:10<00:20,  4.87it/s, epoch=26, grad_norm=139.004, loss=350.0161, lr=2.87e-04]
Epoch 26 TRAIN:  29%|██▉       | 40/136 [00:10<00:19,  5.02it/s, epoch=26, grad_norm=139.004, loss=350.0161, lr=2.87e-04]
Epoch 26 TRAIN:  29%|██▉       | 40/136 [00:10<00:19,  5.02it/s, epoch=26, grad_norm=74.364, loss=320.5481, lr=2.87e-04] 
Epoch 26 TRAIN:  29%|██▉       | 40/136 [00:10<00:19,  5.02it/s, epoch=26, grad_norm=85.977, loss=316.8170, lr=2.87e-04]
Epoch 26 TRAIN:  29%|██▉       | 40/136 [00:11<00:19,  5.02it/s, epoch=26, grad_norm=127.824, loss=360.1913, lr=2.87e-04]
Epoch 26 TRAIN:  32%|███▏      | 43/136 [00:11<00:18,  5.13it/s, epoch=26, grad_norm=127.824, loss=360.1913, lr=2.87e-04]
Epoch 26 TRAIN:  32%|███▏      | 43/136 [00:11<00:18,  5.13it/s, epoch=26, grad_norm=90.231, loss=315.4855, lr=2.87e-04] 
Epoch 26 TRAIN:  32%|███▏      | 43/136 [00:11<00:18,  5.13it/s, epoch=26, grad_norm=111.361, loss=280.7874, lr=2.87e-04]
Epoch 26 TRAIN:  32%|███▏      | 43/136 [00:11<00:18,  5.13it/s, epoch=26, grad_norm=96.200, loss=341.2051, lr=2.87e-04] 
Epoch 26 TRAIN:  34%|███▍      | 46/136 [00:11<00:17,  5.03it/s, epoch=26, grad_norm=96.200, loss=341.2051, lr=2.87e-04]
Epoch 26 TRAIN:  34%|███▍      | 46/136 [00:11<00:17,  5.03it/s, epoch=26, grad_norm=111.117, loss=327.7856, lr=2.87e-04]
Epoch 26 TRAIN:  34%|███▍      | 46/136 [00:12<00:17,  5.03it/s, epoch=26, grad_norm=81.933, loss=330.8035, lr=2.87e-04] 
Epoch 26 TRAIN:  34%|███▍      | 46/136 [00:12<00:17,  5.03it/s, epoch=26, grad_norm=137.246, loss=351.1352, lr=2.87e-04]
Epoch 26 TRAIN:  36%|███▌      | 49/136 [00:12<00:17,  4.99it/s, epoch=26, grad_norm=137.246, loss=351.1352, lr=2.87e-04]
Epoch 26 TRAIN:  36%|███▌      | 49/136 [00:12<00:17,  4.99it/s, epoch=26, grad_norm=154.200, loss=353.5642, lr=2.87e-04]
Epoch 26 TRAIN:  36%|███▌      | 49/136 [00:12<00:17,  4.99it/s, epoch=26, grad_norm=135.888, loss=278.0341, lr=2.87e-04]
Epoch 26 TRAIN:  36%|███▌      | 49/136 [00:12<00:17,  4.99it/s, epoch=26, grad_norm=89.751, loss=338.4449, lr=2.88e-04] 
Epoch 26 TRAIN:  38%|███▊      | 52/136 [00:12<00:16,  4.98it/s, epoch=26, grad_norm=89.751, loss=338.4449, lr=2.88e-04]
Epoch 26 TRAIN:  38%|███▊      | 52/136 [00:13<00:16,  4.98it/s, epoch=26, grad_norm=89.862, loss=351.5423, lr=2.88e-04]
Epoch 26 TRAIN:  38%|███▊      | 52/136 [00:13<00:16,  4.98it/s, epoch=26, grad_norm=97.135, loss=285.6808, lr=2.88e-04]
Epoch 26 TRAIN:  38%|███▊      | 52/136 [00:13<00:16,  4.98it/s, epoch=26, grad_norm=94.446, loss=321.8032, lr=2.88e-04]
Epoch 26 TRAIN:  40%|████      | 55/136 [00:13<00:16,  4.99it/s, epoch=26, grad_norm=94.446, loss=321.8032, lr=2.88e-04]
Epoch 26 TRAIN:  40%|████      | 55/136 [00:13<00:16,  4.99it/s, epoch=26, grad_norm=111.439, loss=356.5264, lr=2.88e-04]
Epoch 26 TRAIN:  40%|████      | 55/136 [00:13<00:16,  4.99it/s, epoch=26, grad_norm=95.927, loss=362.8522, lr=2.88e-04] 
Epoch 26 TRAIN:  40%|████      | 55/136 [00:14<00:16,  4.99it/s, epoch=26, grad_norm=171.006, loss=311.6185, lr=2.88e-04]
Epoch 26 TRAIN:  43%|████▎     | 58/136 [00:14<00:15,  5.04it/s, epoch=26, grad_norm=171.006, loss=311.6185, lr=2.88e-04]
Epoch 26 TRAIN:  43%|████▎     | 58/136 [00:14<00:15,  5.04it/s, epoch=26, grad_norm=105.444, loss=277.8824, lr=2.88e-04]
Epoch 26 TRAIN:  43%|████▎     | 58/136 [00:14<00:15,  5.04it/s, epoch=26, grad_norm=94.965, loss=280.9012, lr=2.88e-04] 
Epoch 26 TRAIN:  43%|████▎     | 58/136 [00:14<00:15,  5.04it/s, epoch=26, grad_norm=68.456, loss=305.8654, lr=2.88e-04]
Epoch 26 TRAIN:  45%|████▍     | 61/136 [00:14<00:14,  5.08it/s, epoch=26, grad_norm=68.456, loss=305.8654, lr=2.88e-04]
Epoch 26 TRAIN:  45%|████▍     | 61/136 [00:14<00:14,  5.08it/s, epoch=26, grad_norm=84.667, loss=296.8944, lr=2.88e-04]
Epoch 26 TRAIN:  45%|████▍     | 61/136 [00:15<00:14,  5.08it/s, epoch=26, grad_norm=129.437, loss=235.5280, lr=2.88e-04]
Epoch 26 TRAIN:  45%|████▍     | 61/136 [00:15<00:14,  5.08it/s, epoch=26, grad_norm=167.970, loss=320.7165, lr=2.88e-04]
Epoch 26 TRAIN:  47%|████▋     | 64/136 [00:15<00:14,  4.86it/s, epoch=26, grad_norm=167.970, loss=320.7165, lr=2.88e-04]
Epoch 26 TRAIN:  47%|████▋     | 64/136 [00:15<00:14,  4.86it/s, epoch=26, grad_norm=72.510, loss=285.2166, lr=2.89e-04] 
Epoch 26 TRAIN:  47%|████▋     | 64/136 [00:15<00:14,  4.86it/s, epoch=26, grad_norm=163.428, loss=269.5744, lr=2.89e-04]
Epoch 26 TRAIN:  47%|████▋     | 64/136 [00:15<00:14,  4.86it/s, epoch=26, grad_norm=125.836, loss=303.2348, lr=2.89e-04]
Epoch 26 TRAIN:  49%|████▉     | 67/136 [00:15<00:14,  4.77it/s, epoch=26, grad_norm=125.836, loss=303.2348, lr=2.89e-04]
Epoch 26 TRAIN:  49%|████▉     | 67/136 [00:16<00:14,  4.77it/s, epoch=26, grad_norm=139.064, loss=327.6813, lr=2.89e-04]
Epoch 26 TRAIN:  49%|████▉     | 67/136 [00:16<00:14,  4.77it/s, epoch=26, grad_norm=69.801, loss=296.9386, lr=2.89e-04] 
Epoch 26 TRAIN:  49%|████▉     | 67/136 [00:16<00:14,  4.77it/s, epoch=26, grad_norm=134.535, loss=336.4558, lr=2.89e-04]
Epoch 26 TRAIN:  51%|█████▏    | 70/136 [00:16<00:13,  4.83it/s, epoch=26, grad_norm=134.535, loss=336.4558, lr=2.89e-04]
Epoch 26 TRAIN:  51%|█████▏    | 70/136 [00:16<00:13,  4.83it/s, epoch=26, grad_norm=73.520, loss=316.3593, lr=2.89e-04] 
Epoch 26 TRAIN:  51%|█████▏    | 70/136 [00:16<00:13,  4.83it/s, epoch=26, grad_norm=157.151, loss=381.1331, lr=2.89e-04]
Epoch 26 TRAIN:  51%|█████▏    | 70/136 [00:17<00:13,  4.83it/s, epoch=26, grad_norm=91.595, loss=333.3629, lr=2.89e-04] 
Epoch 26 TRAIN:  54%|█████▎    | 73/136 [00:17<00:12,  4.89it/s, epoch=26, grad_norm=91.595, loss=333.3629, lr=2.89e-04]
Epoch 26 TRAIN:  54%|█████▎    | 73/136 [00:17<00:12,  4.89it/s, epoch=26, grad_norm=188.309, loss=399.8105, lr=2.89e-04]
Epoch 26 TRAIN:  54%|█████▎    | 73/136 [00:17<00:12,  4.89it/s, epoch=26, grad_norm=119.133, loss=318.5598, lr=2.89e-04]
Epoch 26 TRAIN:  54%|█████▎    | 73/136 [00:17<00:12,  4.89it/s, epoch=26, grad_norm=126.929, loss=349.9636, lr=2.89e-04]
Epoch 26 TRAIN:  56%|█████▌    | 76/136 [00:17<00:12,  4.81it/s, epoch=26, grad_norm=126.929, loss=349.9636, lr=2.89e-04]
Epoch 26 TRAIN:  56%|█████▌    | 76/136 [00:18<00:12,  4.81it/s, epoch=26, grad_norm=101.461, loss=341.5128, lr=2.90e-04]
Epoch 26 TRAIN:  56%|█████▌    | 76/136 [00:18<00:12,  4.81it/s, epoch=26, grad_norm=76.585, loss=332.1014, lr=2.90e-04] 
Epoch 26 TRAIN:  56%|█████▌    | 76/136 [00:18<00:12,  4.81it/s, epoch=26, grad_norm=135.724, loss=327.7165, lr=2.90e-04]
Epoch 26 TRAIN:  58%|█████▊    | 79/136 [00:18<00:11,  4.85it/s, epoch=26, grad_norm=135.724, loss=327.7165, lr=2.90e-04]
Epoch 26 TRAIN:  58%|█████▊    | 79/136 [00:18<00:11,  4.85it/s, epoch=26, grad_norm=78.927, loss=271.8836, lr=2.90e-04] 
Epoch 26 TRAIN:  58%|█████▊    | 79/136 [00:18<00:11,  4.85it/s, epoch=26, grad_norm=299.417, loss=175.3946, lr=2.90e-04]
Epoch 26 TRAIN:  58%|█████▊    | 79/136 [00:19<00:11,  4.85it/s, epoch=26, grad_norm=322.530, loss=220.5457, lr=2.90e-04]
Epoch 26 TRAIN:  60%|██████    | 82/136 [00:19<00:11,  4.88it/s, epoch=26, grad_norm=322.530, loss=220.5457, lr=2.90e-04]
Epoch 26 TRAIN:  60%|██████    | 82/136 [00:19<00:11,  4.88it/s, epoch=26, grad_norm=135.364, loss=310.4533, lr=2.90e-04]
Epoch 26 TRAIN:  60%|██████    | 82/136 [00:19<00:11,  4.88it/s, epoch=26, grad_norm=119.687, loss=280.1582, lr=2.90e-04]
Epoch 26 TRAIN:  60%|██████    | 82/136 [00:19<00:11,  4.88it/s, epoch=26, grad_norm=141.709, loss=344.9784, lr=2.90e-04]
Epoch 26 TRAIN:  62%|██████▎   | 85/136 [00:19<00:10,  4.82it/s, epoch=26, grad_norm=141.709, loss=344.9784, lr=2.90e-04]
Epoch 26 TRAIN:  62%|██████▎   | 85/136 [00:19<00:10,  4.82it/s, epoch=26, grad_norm=166.750, loss=343.5347, lr=2.90e-04]
Epoch 26 TRAIN:  62%|██████▎   | 85/136 [00:20<00:10,  4.82it/s, epoch=26, grad_norm=109.269, loss=306.3459, lr=2.90e-04]
Epoch 26 TRAIN:  62%|██████▎   | 85/136 [00:20<00:10,  4.82it/s, epoch=26, grad_norm=135.346, loss=300.3777, lr=2.90e-04]
Epoch 26 TRAIN:  65%|██████▍   | 88/136 [00:20<00:09,  4.92it/s, epoch=26, grad_norm=135.346, loss=300.3777, lr=2.90e-04]
Epoch 26 TRAIN:  65%|██████▍   | 88/136 [00:20<00:09,  4.92it/s, epoch=26, grad_norm=85.921, loss=347.0198, lr=2.90e-04] 
Epoch 26 TRAIN:  65%|██████▍   | 88/136 [00:20<00:09,  4.92it/s, epoch=26, grad_norm=188.819, loss=332.6543, lr=2.91e-04]
Epoch 26 TRAIN:  65%|██████▍   | 88/136 [00:20<00:09,  4.92it/s, epoch=26, grad_norm=143.400, loss=338.5868, lr=2.91e-04]
Epoch 26 TRAIN:  67%|██████▋   | 91/136 [00:20<00:09,  4.81it/s, epoch=26, grad_norm=143.400, loss=338.5868, lr=2.91e-04]
Epoch 26 TRAIN:  67%|██████▋   | 91/136 [00:21<00:09,  4.81it/s, epoch=26, grad_norm=213.743, loss=312.3250, lr=2.91e-04]
Epoch 26 TRAIN:  67%|██████▋   | 91/136 [00:21<00:09,  4.81it/s, epoch=26, grad_norm=139.242, loss=289.5609, lr=2.91e-04]
Epoch 26 TRAIN:  67%|██████▋   | 91/136 [00:21<00:09,  4.81it/s, epoch=26, grad_norm=123.287, loss=309.2431, lr=2.91e-04]
Epoch 26 TRAIN:  69%|██████▉   | 94/136 [00:21<00:08,  4.87it/s, epoch=26, grad_norm=123.287, loss=309.2431, lr=2.91e-04]
Epoch 26 TRAIN:  69%|██████▉   | 94/136 [00:21<00:08,  4.87it/s, epoch=26, grad_norm=227.902, loss=283.5457, lr=2.91e-04]
Epoch 26 TRAIN:  69%|██████▉   | 94/136 [00:21<00:08,  4.87it/s, epoch=26, grad_norm=104.116, loss=348.9207, lr=2.91e-04]
Epoch 26 TRAIN:  69%|██████▉   | 94/136 [00:22<00:08,  4.87it/s, epoch=26, grad_norm=129.574, loss=280.3197, lr=2.91e-04]
Epoch 26 TRAIN:  71%|███████▏  | 97/136 [00:22<00:07,  4.89it/s, epoch=26, grad_norm=129.574, loss=280.3197, lr=2.91e-04]
Epoch 26 TRAIN:  71%|███████▏  | 97/136 [00:22<00:07,  4.89it/s, epoch=26, grad_norm=130.953, loss=378.9395, lr=2.91e-04]
Epoch 26 TRAIN:  71%|███████▏  | 97/136 [00:22<00:07,  4.89it/s, epoch=26, grad_norm=248.121, loss=383.2167, lr=2.91e-04]
Epoch 26 TRAIN:  71%|███████▏  | 97/136 [00:22<00:07,  4.89it/s, epoch=26, grad_norm=114.439, loss=252.5494, lr=2.91e-04]2026-04-16 20:05:03,675 DEBUG TRAIN | steps/sec 4.358| Batch 26/100 loss 252.549377 loss_att 225.002609 loss_ctc 316.825134 th_accuracy 0.281250 lr 2.9136e-04 grad_norm 114.438705 rank 0

Epoch 26 TRAIN:  74%|███████▎  | 100/136 [00:22<00:07,  4.54it/s, epoch=26, grad_norm=114.439, loss=252.5494, lr=2.91e-04]
Epoch 26 TRAIN:  74%|███████▎  | 100/136 [00:23<00:07,  4.54it/s, epoch=26, grad_norm=169.415, loss=358.8304, lr=2.91e-04]
Epoch 26 TRAIN:  74%|███████▎  | 100/136 [00:23<00:07,  4.54it/s, epoch=26, grad_norm=144.034, loss=281.1360, lr=2.92e-04]
Epoch 26 TRAIN:  74%|███████▎  | 100/136 [00:23<00:07,  4.54it/s, epoch=26, grad_norm=212.357, loss=383.8047, lr=2.92e-04]
Epoch 26 TRAIN:  76%|███████▌  | 103/136 [00:23<00:07,  4.61it/s, epoch=26, grad_norm=212.357, loss=383.8047, lr=2.92e-04]
Epoch 26 TRAIN:  76%|███████▌  | 103/136 [00:23<00:07,  4.61it/s, epoch=26, grad_norm=139.372, loss=177.2787, lr=2.92e-04]
Epoch 26 TRAIN:  76%|███████▌  | 103/136 [00:23<00:07,  4.61it/s, epoch=26, grad_norm=117.058, loss=309.8157, lr=2.92e-04]
Epoch 26 TRAIN:  76%|███████▌  | 103/136 [00:24<00:07,  4.61it/s, epoch=26, grad_norm=235.375, loss=421.4793, lr=2.92e-04]
Epoch 26 TRAIN:  78%|███████▊  | 106/136 [00:24<00:06,  4.64it/s, epoch=26, grad_norm=235.375, loss=421.4793, lr=2.92e-04]
Epoch 26 TRAIN:  78%|███████▊  | 106/136 [00:24<00:06,  4.64it/s, epoch=26, grad_norm=100.242, loss=295.1492, lr=2.92e-04]
Epoch 26 TRAIN:  78%|███████▊  | 106/136 [00:24<00:06,  4.64it/s, epoch=26, grad_norm=159.094, loss=365.6319, lr=2.92e-04]
Epoch 26 TRAIN:  78%|███████▊  | 106/136 [00:24<00:06,  4.64it/s, epoch=26, grad_norm=140.634, loss=319.0887, lr=2.92e-04]
Epoch 26 TRAIN:  80%|████████  | 109/136 [00:24<00:05,  4.65it/s, epoch=26, grad_norm=140.634, loss=319.0887, lr=2.92e-04]
Epoch 26 TRAIN:  80%|████████  | 109/136 [00:25<00:05,  4.65it/s, epoch=26, grad_norm=120.039, loss=257.3875, lr=2.92e-04]
Epoch 26 TRAIN:  80%|████████  | 109/136 [00:25<00:05,  4.65it/s, epoch=26, grad_norm=336.418, loss=288.8813, lr=2.92e-04]
Epoch 26 TRAIN:  80%|████████  | 109/136 [00:25<00:05,  4.65it/s, epoch=26, grad_norm=130.854, loss=314.5752, lr=2.92e-04]
Epoch 26 TRAIN:  82%|████████▏ | 112/136 [00:25<00:04,  4.82it/s, epoch=26, grad_norm=130.854, loss=314.5752, lr=2.92e-04]
Epoch 26 TRAIN:  82%|████████▏ | 112/136 [00:25<00:04,  4.82it/s, epoch=26, grad_norm=138.888, loss=308.8893, lr=2.92e-04]
Epoch 26 TRAIN:  82%|████████▏ | 112/136 [00:25<00:04,  4.82it/s, epoch=26, grad_norm=193.233, loss=387.2134, lr=2.92e-04]
Epoch 26 TRAIN:  82%|████████▏ | 112/136 [00:25<00:04,  4.82it/s, epoch=26, grad_norm=154.290, loss=294.1803, lr=2.93e-04]
Epoch 26 TRAIN:  85%|████████▍ | 115/136 [00:25<00:04,  4.86it/s, epoch=26, grad_norm=154.290, loss=294.1803, lr=2.93e-04]
Epoch 26 TRAIN:  85%|████████▍ | 115/136 [00:26<00:04,  4.86it/s, epoch=26, grad_norm=202.995, loss=361.4859, lr=2.93e-04]
Epoch 26 TRAIN:  85%|████████▍ | 115/136 [00:26<00:04,  4.86it/s, epoch=26, grad_norm=175.692, loss=348.2739, lr=2.93e-04]
Epoch 26 TRAIN:  85%|████████▍ | 115/136 [00:26<00:04,  4.86it/s, epoch=26, grad_norm=274.721, loss=327.4148, lr=2.93e-04]
Epoch 26 TRAIN:  87%|████████▋ | 118/136 [00:26<00:03,  4.84it/s, epoch=26, grad_norm=274.721, loss=327.4148, lr=2.93e-04]
Epoch 26 TRAIN:  87%|████████▋ | 118/136 [00:26<00:03,  4.84it/s, epoch=26, grad_norm=282.753, loss=358.2943, lr=2.93e-04]
Epoch 26 TRAIN:  87%|████████▋ | 118/136 [00:27<00:03,  4.84it/s, epoch=26, grad_norm=284.808, loss=244.7720, lr=2.93e-04]
Epoch 26 TRAIN:  87%|████████▋ | 118/136 [00:27<00:03,  4.84it/s, epoch=26, grad_norm=222.113, loss=303.1208, lr=2.93e-04]
Epoch 26 TRAIN:  89%|████████▉ | 121/136 [00:27<00:03,  4.70it/s, epoch=26, grad_norm=222.113, loss=303.1208, lr=2.93e-04]
Epoch 26 TRAIN:  89%|████████▉ | 121/136 [00:27<00:03,  4.70it/s, epoch=26, grad_norm=156.090, loss=247.8476, lr=2.93e-04]
Epoch 26 TRAIN:  89%|████████▉ | 121/136 [00:27<00:03,  4.70it/s, epoch=26, grad_norm=232.225, loss=296.6307, lr=2.93e-04]
Epoch 26 TRAIN:  89%|████████▉ | 121/136 [00:27<00:03,  4.70it/s, epoch=26, grad_norm=200.028, loss=371.4189, lr=2.93e-04]
Epoch 26 TRAIN:  91%|█████████ | 124/136 [00:27<00:02,  5.03it/s, epoch=26, grad_norm=200.028, loss=371.4189, lr=2.93e-04]
Epoch 26 TRAIN:  91%|█████████ | 124/136 [00:28<00:02,  5.03it/s, epoch=26, grad_norm=279.801, loss=364.5731, lr=2.93e-04]
Epoch 26 TRAIN:  91%|█████████ | 124/136 [00:28<00:02,  5.03it/s, epoch=26, grad_norm=103.058, loss=317.1802, lr=2.93e-04]
Epoch 26 TRAIN:  91%|█████████ | 124/136 [00:28<00:02,  5.03it/s, epoch=26, grad_norm=167.794, loss=350.2583, lr=2.94e-04]
Epoch 26 TRAIN:  93%|█████████▎| 127/136 [00:28<00:01,  4.98it/s, epoch=26, grad_norm=167.794, loss=350.2583, lr=2.94e-04]
Epoch 26 TRAIN:  93%|█████████▎| 127/136 [00:28<00:01,  4.98it/s, epoch=26, grad_norm=219.693, loss=352.7104, lr=2.94e-04]
Epoch 26 TRAIN:  93%|█████████▎| 127/136 [00:28<00:01,  4.98it/s, epoch=26, grad_norm=123.490, loss=349.3933, lr=2.94e-04]
Epoch 26 TRAIN:  93%|█████████▎| 127/136 [00:28<00:01,  4.98it/s, epoch=26, grad_norm=69.143, loss=243.3582, lr=2.94e-04] 
Epoch 26 TRAIN:  96%|█████████▌| 130/136 [00:28<00:01,  5.11it/s, epoch=26, grad_norm=69.143, loss=243.3582, lr=2.94e-04]
Epoch 26 TRAIN:  96%|█████████▌| 130/136 [00:29<00:01,  5.11it/s, epoch=26, grad_norm=114.403, loss=318.5154, lr=2.94e-04]
Epoch 26 TRAIN:  96%|█████████▌| 130/136 [00:29<00:01,  5.11it/s, epoch=26, grad_norm=161.439, loss=361.5208, lr=2.94e-04]
Epoch 26 TRAIN:  96%|█████████▌| 130/136 [00:29<00:01,  5.11it/s, epoch=26, grad_norm=115.392, loss=329.1525, lr=2.94e-04]
Epoch 26 TRAIN:  98%|█████████▊| 133/136 [00:29<00:00,  5.21it/s, epoch=26, grad_norm=115.392, loss=329.1525, lr=2.94e-04]
Epoch 26 TRAIN:  98%|█████████▊| 133/136 [00:29<00:00,  5.21it/s, epoch=26, grad_norm=90.844, loss=318.0207, lr=2.94e-04] 
Epoch 26 TRAIN:  98%|█████████▊| 133/136 [00:29<00:00,  5.21it/s, epoch=26, grad_norm=163.246, loss=300.2285, lr=2.94e-04]
Epoch 26 TRAIN:  99%|█████████▉| 135/136 [00:29<00:00,  4.52it/s, epoch=26, grad_norm=163.246, loss=300.2285, lr=2.94e-04]

CV:   0%|          | 0/6 [00:00<?, ?it/s]/home/rookie/Documents/batuka/projects/wenet/wenet/wenet/utils/train_utils.py:689: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  amp_autocast(enabled=dtype is not None,

CV:  17%|█▋        | 1/6 [00:00<00:03,  1.32it/s]
                                                 
2026-04-16 20:05:11,594 INFO Epoch 26 Step 3677 CV info lr 2.9416e-04 cv_loss 494.083701393821 rank 0 acc 0.26035622010628384
2026-04-16 20:05:11,596 INFO [Rank 0] Checkpoint: save to checkpoint exp/conformer_mongolian/epoch_26.pt
2026-04-16 20:05:11,718 INFO Epoch 27 Step 3677 TRAIN info lr 2.9416e-04 rank 0
2026-04-16 20:05:11,733 INFO using accumulate grad, new batch size is 1 times larger than before

Epoch 27 TRAIN:   0%|          | 0/135 [00:00<?, ?it/s]
Epoch 27 TRAIN:   0%|          | 0/135 [00:03<?, ?it/s, epoch=27, grad_norm=466.780, loss=431.4283, lr=2.94e-04]
Epoch 27 TRAIN:   1%|          | 1/135 [00:03<08:48,  3.94s/it, epoch=27, grad_norm=466.780, loss=431.4283, lr=2.94e-04]
Epoch 27 TRAIN:   1%|          | 1/135 [00:04<08:48,  3.94s/it, epoch=27, grad_norm=433.215, loss=324.4077, lr=2.94e-04]
Epoch 27 TRAIN:   1%|          | 1/135 [00:04<08:48,  3.94s/it, epoch=27, grad_norm=226.522, loss=273.1845, lr=2.94e-04]
Epoch 27 TRAIN:   1%|          | 1/135 [00:04<08:48,  3.94s/it, epoch=27, grad_norm=322.750, loss=391.4417, lr=2.94e-04]
Epoch 27 TRAIN:   1%|          | 1/135 [00:04<08:48,  3.94s/it, epoch=27, grad_norm=132.692, loss=308.7869, lr=2.95e-04]
Epoch 27 TRAIN:   1%|          | 1/135 [00:04<08:48,  3.94s/it, epoch=27, grad_norm=521.471, loss=260.6314, lr=2.95e-04]
Epoch 27 TRAIN:   1%|          | 1/135 [00:04<08:48,  3.94s/it, epoch=27, grad_norm=221.480, loss=394.9976, lr=2.95e-04]
Epoch 27 TRAIN:   1%|          | 1/135 [00:04<08:48,  3.94s/it, epoch=27, grad_norm=276.549, loss=282.3061, lr=2.95e-04]
Epoch 27 TRAIN:   6%|▌         | 8/135 [00:04<00:53,  2.35it/s, epoch=27, grad_norm=276.549, loss=282.3061, lr=2.95e-04]
Epoch 27 TRAIN:   6%|▌         | 8/135 [00:04<00:53,  2.35it/s, epoch=27, grad_norm=88.011, loss=304.2279, lr=2.95e-04] 
Epoch 27 TRAIN:   6%|▌         | 8/135 [00:04<00:53,  2.35it/s, epoch=27, grad_norm=148.690, loss=330.4892, lr=2.95e-04]
Epoch 27 TRAIN:   6%|▌         | 8/135 [00:04<00:53,  2.35it/s, epoch=27, grad_norm=222.131, loss=389.3487, lr=2.95e-04]
Epoch 27 TRAIN:   6%|▌         | 8/135 [00:05<00:53,  2.35it/s, epoch=27, grad_norm=65.230, loss=283.7569, lr=2.95e-04] 
Epoch 27 TRAIN:   6%|▌         | 8/135 [00:05<00:53,  2.35it/s, epoch=27, grad_norm=115.779, loss=372.0038, lr=2.95e-04]
Epoch 27 TRAIN:  10%|▉         | 13/135 [00:05<00:36,  3.35it/s, epoch=27, grad_norm=115.779, loss=372.0038, lr=2.95e-04]
Epoch 27 TRAIN:  10%|▉         | 13/135 [00:05<00:36,  3.35it/s, epoch=27, grad_norm=175.201, loss=358.1957, lr=2.95e-04]
Epoch 27 TRAIN:  10%|▉         | 13/135 [00:05<00:36,  3.35it/s, epoch=27, grad_norm=181.886, loss=307.4840, lr=2.95e-04]
Epoch 27 TRAIN:  10%|▉         | 13/135 [00:05<00:36,  3.35it/s, epoch=27, grad_norm=77.417, loss=382.7634, lr=2.95e-04] 
Epoch 27 TRAIN:  10%|▉         | 13/135 [00:05<00:36,  3.35it/s, epoch=27, grad_norm=176.555, loss=314.7182, lr=2.96e-04]
Epoch 27 TRAIN:  13%|█▎        | 17/135 [00:05<00:29,  3.99it/s, epoch=27, grad_norm=176.555, loss=314.7182, lr=2.96e-04]
Epoch 27 TRAIN:  13%|█▎        | 17/135 [00:06<00:29,  3.99it/s, epoch=27, grad_norm=220.239, loss=287.8690, lr=2.96e-04]
Epoch 27 TRAIN:  13%|█▎        | 17/135 [00:06<00:29,  3.99it/s, epoch=27, grad_norm=73.261, loss=347.7365, lr=2.96e-04] 
Epoch 27 TRAIN:  13%|█▎        | 17/135 [00:06<00:29,  3.99it/s, epoch=27, grad_norm=85.160, loss=339.7394, lr=2.96e-04]
Epoch 27 TRAIN:  13%|█▎        | 17/135 [00:06<00:29,  3.99it/s, epoch=27, grad_norm=80.737, loss=333.0770, lr=2.96e-04]
Epoch 27 TRAIN:  16%|█▌        | 21/135 [00:06<00:25,  4.43it/s, epoch=27, grad_norm=80.737, loss=333.0770, lr=2.96e-04]
Epoch 27 TRAIN:  16%|█▌        | 21/135 [00:06<00:25,  4.43it/s, epoch=27, grad_norm=86.696, loss=332.3632, lr=2.96e-04]
Epoch 27 TRAIN:  16%|█▌        | 21/135 [00:06<00:25,  4.43it/s, epoch=27, grad_norm=111.495, loss=301.4486, lr=2.96e-04]
Epoch 27 TRAIN:  16%|█▌        | 21/135 [00:07<00:25,  4.43it/s, epoch=27, grad_norm=141.544, loss=360.9780, lr=2.96e-04]
Epoch 27 TRAIN:  16%|█▌        | 21/135 [00:07<00:25,  4.43it/s, epoch=27, grad_norm=72.044, loss=328.5455, lr=2.96e-04] 
Epoch 27 TRAIN:  19%|█▊        | 25/135 [00:07<00:23,  4.70it/s, epoch=27, grad_norm=72.044, loss=328.5455, lr=2.96e-04]
Epoch 27 TRAIN:  19%|█▊        | 25/135 [00:07<00:23,  4.70it/s, epoch=27, grad_norm=83.649, loss=331.9973, lr=2.96e-04]
Epoch 27 TRAIN:  19%|█▊        | 25/135 [00:07<00:23,  4.70it/s, epoch=27, grad_norm=90.305, loss=324.5255, lr=2.96e-04]
Epoch 27 TRAIN:  19%|█▊        | 25/135 [00:07<00:23,  4.70it/s, epoch=27, grad_norm=106.835, loss=311.5743, lr=2.96e-04]
Epoch 27 TRAIN:  19%|█▊        | 25/135 [00:08<00:23,  4.70it/s, epoch=27, grad_norm=136.799, loss=314.0465, lr=2.96e-04]
Epoch 27 TRAIN:  21%|██▏       | 29/135 [00:08<00:21,  4.91it/s, epoch=27, grad_norm=136.799, loss=314.0465, lr=2.96e-04]
Epoch 27 TRAIN:  21%|██▏       | 29/135 [00:08<00:21,  4.91it/s, epoch=27, grad_norm=86.300, loss=317.4973, lr=2.97e-04] 
Epoch 27 TRAIN:  21%|██▏       | 29/135 [00:08<00:21,  4.91it/s, epoch=27, grad_norm=89.992, loss=320.3742, lr=2.97e-04]
Epoch 27 TRAIN:  21%|██▏       | 29/135 [00:08<00:21,  4.91it/s, epoch=27, grad_norm=125.259, loss=325.9026, lr=2.97e-04]
Epoch 27 TRAIN:  24%|██▎       | 32/135 [00:08<00:20,  5.02it/s, epoch=27, grad_norm=125.259, loss=325.9026, lr=2.97e-04]
Epoch 27 TRAIN:  24%|██▎       | 32/135 [00:08<00:20,  5.02it/s, epoch=27, grad_norm=101.185, loss=341.4642, lr=2.97e-04]
Epoch 27 TRAIN:  24%|██▎       | 32/135 [00:09<00:20,  5.02it/s, epoch=27, grad_norm=123.059, loss=335.8246, lr=2.97e-04]
Epoch 27 TRAIN:  24%|██▎       | 32/135 [00:09<00:20,  5.02it/s, epoch=27, grad_norm=165.694, loss=349.6707, lr=2.97e-04]
Epoch 27 TRAIN:  26%|██▌       | 35/135 [00:09<00:19,  5.13it/s, epoch=27, grad_norm=165.694, loss=349.6707, lr=2.97e-04]
Epoch 27 TRAIN:  26%|██▌       | 35/135 [00:09<00:19,  5.13it/s, epoch=27, grad_norm=139.129, loss=338.5429, lr=2.97e-04]
Epoch 27 TRAIN:  26%|██▌       | 35/135 [00:09<00:19,  5.13it/s, epoch=27, grad_norm=188.952, loss=279.2988, lr=2.97e-04]
Epoch 27 TRAIN:  26%|██▌       | 35/135 [00:09<00:19,  5.13it/s, epoch=27, grad_norm=168.843, loss=288.1430, lr=2.97e-04]
Epoch 27 TRAIN:  28%|██▊       | 38/135 [00:09<00:18,  5.20it/s, epoch=27, grad_norm=168.843, loss=288.1430, lr=2.97e-04]
Epoch 27 TRAIN:  28%|██▊       | 38/135 [00:10<00:18,  5.20it/s, epoch=27, grad_norm=146.722, loss=291.1268, lr=2.97e-04]
Epoch 27 TRAIN:  28%|██▊       | 38/135 [00:10<00:18,  5.20it/s, epoch=27, grad_norm=146.261, loss=334.0829, lr=2.97e-04]
Epoch 27 TRAIN:  28%|██▊       | 38/135 [00:10<00:18,  5.20it/s, epoch=27, grad_norm=116.133, loss=281.4180, lr=2.97e-04]
Epoch 27 TRAIN:  30%|███       | 41/135 [00:10<00:18,  4.96it/s, epoch=27, grad_norm=116.133, loss=281.4180, lr=2.97e-04]
Epoch 27 TRAIN:  30%|███       | 41/135 [00:10<00:18,  4.96it/s, epoch=27, grad_norm=100.840, loss=347.2571, lr=2.98e-04]
Epoch 27 TRAIN:  30%|███       | 41/135 [00:10<00:18,  4.96it/s, epoch=27, grad_norm=87.740, loss=280.5127, lr=2.98e-04] 
Epoch 27 TRAIN:  30%|███       | 41/135 [00:11<00:18,  4.96it/s, epoch=27, grad_norm=55.080, loss=298.8471, lr=2.98e-04]
Epoch 27 TRAIN:  33%|███▎      | 44/135 [00:11<00:18,  5.01it/s, epoch=27, grad_norm=55.080, loss=298.8471, lr=2.98e-04]
Epoch 27 TRAIN:  33%|███▎      | 44/135 [00:11<00:18,  5.01it/s, epoch=27, grad_norm=143.203, loss=303.4259, lr=2.98e-04]
Epoch 27 TRAIN:  33%|███▎      | 44/135 [00:11<00:18,  5.01it/s, epoch=27, grad_norm=164.882, loss=328.7859, lr=2.98e-04]
Epoch 27 TRAIN:  33%|███▎      | 44/135 [00:11<00:18,  5.01it/s, epoch=27, grad_norm=97.502, loss=319.6136, lr=2.98e-04] 
Epoch 27 TRAIN:  35%|███▍      | 47/135 [00:11<00:17,  4.97it/s, epoch=27, grad_norm=97.502, loss=319.6136, lr=2.98e-04]
Epoch 27 TRAIN:  35%|███▍      | 47/135 [00:11<00:17,  4.97it/s, epoch=27, grad_norm=75.184, loss=288.5496, lr=2.98e-04]
Epoch 27 TRAIN:  35%|███▍      | 47/135 [00:12<00:17,  4.97it/s, epoch=27, grad_norm=99.488, loss=333.1401, lr=2.98e-04]
Epoch 27 TRAIN:  35%|███▍      | 47/135 [00:12<00:17,  4.97it/s, epoch=27, grad_norm=91.186, loss=327.4266, lr=2.98e-04]
Epoch 27 TRAIN:  37%|███▋      | 50/135 [00:12<00:17,  4.88it/s, epoch=27, grad_norm=91.186, loss=327.4266, lr=2.98e-04]
Epoch 27 TRAIN:  37%|███▋      | 50/135 [00:12<00:17,  4.88it/s, epoch=27, grad_norm=110.301, loss=348.5046, lr=2.98e-04]
Epoch 27 TRAIN:  37%|███▋      | 50/135 [00:12<00:17,  4.88it/s, epoch=27, grad_norm=92.000, loss=344.6369, lr=2.98e-04] 
Epoch 27 TRAIN:  37%|███▋      | 50/135 [00:12<00:17,  4.88it/s, epoch=27, grad_norm=96.835, loss=316.7886, lr=2.98e-04]
Epoch 27 TRAIN:  39%|███▉      | 53/135 [00:12<00:16,  4.93it/s, epoch=27, grad_norm=96.835, loss=316.7886, lr=2.98e-04]
Epoch 27 TRAIN:  39%|███▉      | 53/135 [00:13<00:16,  4.93it/s, epoch=27, grad_norm=184.192, loss=273.3470, lr=2.98e-04]
Epoch 27 TRAIN:  39%|███▉      | 53/135 [00:13<00:16,  4.93it/s, epoch=27, grad_norm=234.377, loss=297.6641, lr=2.99e-04]
Epoch 27 TRAIN:  39%|███▉      | 53/135 [00:13<00:16,  4.93it/s, epoch=27, grad_norm=79.780, loss=304.0809, lr=2.99e-04] 
Epoch 27 TRAIN:  41%|████▏     | 56/135 [00:13<00:15,  4.99it/s, epoch=27, grad_norm=79.780, loss=304.0809, lr=2.99e-04]
Epoch 27 TRAIN:  41%|████▏     | 56/135 [00:13<00:15,  4.99it/s, epoch=27, grad_norm=122.148, loss=382.9307, lr=2.99e-04]
Epoch 27 TRAIN:  41%|████▏     | 56/135 [00:13<00:15,  4.99it/s, epoch=27, grad_norm=102.549, loss=347.5121, lr=2.99e-04]
Epoch 27 TRAIN:  41%|████▏     | 56/135 [00:14<00:15,  4.99it/s, epoch=27, grad_norm=102.604, loss=349.2126, lr=2.99e-04]
Epoch 27 TRAIN:  44%|████▎     | 59/135 [00:14<00:15,  4.88it/s, epoch=27, grad_norm=102.604, loss=349.2126, lr=2.99e-04]
Epoch 27 TRAIN:  44%|████▎     | 59/135 [00:14<00:15,  4.88it/s, epoch=27, grad_norm=99.413, loss=313.0273, lr=2.99e-04] 
Epoch 27 TRAIN:  44%|████▎     | 59/135 [00:14<00:15,  4.88it/s, epoch=27, grad_norm=103.002, loss=359.8172, lr=2.99e-04]
Epoch 27 TRAIN:  44%|████▎     | 59/135 [00:14<00:15,  4.88it/s, epoch=27, grad_norm=122.456, loss=298.1605, lr=2.99e-04]
Epoch 27 TRAIN:  46%|████▌     | 62/135 [00:14<00:14,  4.93it/s, epoch=27, grad_norm=122.456, loss=298.1605, lr=2.99e-04]
Epoch 27 TRAIN:  46%|████▌     | 62/135 [00:14<00:14,  4.93it/s, epoch=27, grad_norm=152.510, loss=274.7787, lr=2.99e-04]
Epoch 27 TRAIN:  46%|████▌     | 62/135 [00:15<00:14,  4.93it/s, epoch=27, grad_norm=78.040, loss=335.0466, lr=2.99e-04] 
Epoch 27 TRAIN:  46%|████▌     | 62/135 [00:15<00:14,  4.93it/s, epoch=27, grad_norm=111.966, loss=357.9117, lr=2.99e-04]
Epoch 27 TRAIN:  48%|████▊     | 65/135 [00:15<00:14,  4.89it/s, epoch=27, grad_norm=111.966, loss=357.9117, lr=2.99e-04]
Epoch 27 TRAIN:  48%|████▊     | 65/135 [00:15<00:14,  4.89it/s, epoch=27, grad_norm=93.203, loss=278.8961, lr=2.99e-04] 
Epoch 27 TRAIN:  48%|████▊     | 65/135 [00:15<00:14,  4.89it/s, epoch=27, grad_norm=153.710, loss=372.2073, lr=3.00e-04]
Epoch 27 TRAIN:  48%|████▊     | 65/135 [00:15<00:14,  4.89it/s, epoch=27, grad_norm=133.058, loss=359.1014, lr=3.00e-04]
Epoch 27 TRAIN:  50%|█████     | 68/135 [00:15<00:13,  4.88it/s, epoch=27, grad_norm=133.058, loss=359.1014, lr=3.00e-04]
Epoch 27 TRAIN:  50%|█████     | 68/135 [00:16<00:13,  4.88it/s, epoch=27, grad_norm=112.425, loss=330.4754, lr=3.00e-04]
Epoch 27 TRAIN:  50%|█████     | 68/135 [00:16<00:13,  4.88it/s, epoch=27, grad_norm=80.020, loss=348.2394, lr=3.00e-04] 
Epoch 27 TRAIN:  50%|█████     | 68/135 [00:16<00:13,  4.88it/s, epoch=27, grad_norm=193.939, loss=260.8193, lr=3.00e-04]
Epoch 27 TRAIN:  53%|█████▎    | 71/135 [00:16<00:13,  4.89it/s, epoch=27, grad_norm=193.939, loss=260.8193, lr=3.00e-04]
Epoch 27 TRAIN:  53%|█████▎    | 71/135 [00:16<00:13,  4.89it/s, epoch=27, grad_norm=93.051, loss=335.6239, lr=3.00e-04] 
Epoch 27 TRAIN:  53%|█████▎    | 71/135 [00:16<00:13,  4.89it/s, epoch=27, grad_norm=81.688, loss=304.5023, lr=3.00e-04]
Epoch 27 TRAIN:  53%|█████▎    | 71/135 [00:17<00:13,  4.89it/s, epoch=27, grad_norm=81.662, loss=313.7820, lr=3.00e-04]
Epoch 27 TRAIN:  55%|█████▍    | 74/135 [00:17<00:12,  4.80it/s, epoch=27, grad_norm=81.662, loss=313.7820, lr=3.00e-04]
Epoch 27 TRAIN:  55%|█████▍    | 74/135 [00:17<00:12,  4.80it/s, epoch=27, grad_norm=99.583, loss=327.5893, lr=3.00e-04]
Epoch 27 TRAIN:  55%|█████▍    | 74/135 [00:17<00:12,  4.80it/s, epoch=27, grad_norm=74.426, loss=341.3688, lr=3.00e-04]
Epoch 27 TRAIN:  55%|█████▍    | 74/135 [00:17<00:12,  4.80it/s, epoch=27, grad_norm=91.038, loss=342.9739, lr=3.00e-04]
Epoch 27 TRAIN:  57%|█████▋    | 77/135 [00:17<00:12,  4.81it/s, epoch=27, grad_norm=91.038, loss=342.9739, lr=3.00e-04]
Epoch 27 TRAIN:  57%|█████▋    | 77/135 [00:18<00:12,  4.81it/s, epoch=27, grad_norm=82.187, loss=332.9466, lr=3.00e-04]
Epoch 27 TRAIN:  57%|█████▋    | 77/135 [00:18<00:12,  4.81it/s, epoch=27, grad_norm=100.376, loss=362.1818, lr=3.00e-04]
Epoch 27 TRAIN:  57%|█████▋    | 77/135 [00:18<00:12,  4.81it/s, epoch=27, grad_norm=249.990, loss=262.0794, lr=3.01e-04]
Epoch 27 TRAIN:  59%|█████▉    | 80/135 [00:18<00:11,  4.88it/s, epoch=27, grad_norm=249.990, loss=262.0794, lr=3.01e-04]
Epoch 27 TRAIN:  59%|█████▉    | 80/135 [00:18<00:11,  4.88it/s, epoch=27, grad_norm=87.842, loss=291.2672, lr=3.01e-04] 
Epoch 27 TRAIN:  59%|█████▉    | 80/135 [00:18<00:11,  4.88it/s, epoch=27, grad_norm=93.358, loss=335.7391, lr=3.01e-04]
Epoch 27 TRAIN:  59%|█████▉    | 80/135 [00:19<00:11,  4.88it/s, epoch=27, grad_norm=133.468, loss=255.4459, lr=3.01e-04]
Epoch 27 TRAIN:  61%|██████▏   | 83/135 [00:19<00:10,  4.93it/s, epoch=27, grad_norm=133.468, loss=255.4459, lr=3.01e-04]
Epoch 27 TRAIN:  61%|██████▏   | 83/135 [00:19<00:10,  4.93it/s, epoch=27, grad_norm=117.081, loss=335.7045, lr=3.01e-04]
Epoch 27 TRAIN:  61%|██████▏   | 83/135 [00:19<00:10,  4.93it/s, epoch=27, grad_norm=114.447, loss=330.5385, lr=3.01e-04]
Epoch 27 TRAIN:  61%|██████▏   | 83/135 [00:19<00:10,  4.93it/s, epoch=27, grad_norm=80.384, loss=290.8439, lr=3.01e-04] 
Epoch 27 TRAIN:  64%|██████▎   | 86/135 [00:19<00:10,  4.55it/s, epoch=27, grad_norm=80.384, loss=290.8439, lr=3.01e-04]
Epoch 27 TRAIN:  64%|██████▎   | 86/135 [00:20<00:10,  4.55it/s, epoch=27, grad_norm=205.604, loss=213.4836, lr=3.01e-04]
Epoch 27 TRAIN:  64%|██████▎   | 86/135 [00:20<00:10,  4.55it/s, epoch=27, grad_norm=111.877, loss=289.3804, lr=3.01e-04]
Epoch 27 TRAIN:  64%|██████▎   | 86/135 [00:20<00:10,  4.55it/s, epoch=27, grad_norm=241.346, loss=366.3615, lr=3.01e-04]
Epoch 27 TRAIN:  66%|██████▌   | 89/135 [00:20<00:09,  4.67it/s, epoch=27, grad_norm=241.346, loss=366.3615, lr=3.01e-04]
Epoch 27 TRAIN:  66%|██████▌   | 89/135 [00:20<00:09,  4.67it/s, epoch=27, grad_norm=283.016, loss=397.4993, lr=3.01e-04]
Epoch 27 TRAIN:  66%|██████▌   | 89/135 [00:20<00:09,  4.67it/s, epoch=27, grad_norm=239.195, loss=383.8011, lr=3.01e-04]
Epoch 27 TRAIN:  66%|██████▌   | 89/135 [00:21<00:09,  4.67it/s, epoch=27, grad_norm=97.107, loss=273.7933, lr=3.02e-04] 
Epoch 27 TRAIN:  68%|██████▊   | 92/135 [00:21<00:09,  4.74it/s, epoch=27, grad_norm=97.107, loss=273.7933, lr=3.02e-04]
Epoch 27 TRAIN:  68%|██████▊   | 92/135 [00:21<00:09,  4.74it/s, epoch=27, grad_norm=208.018, loss=256.0692, lr=3.02e-04]
Epoch 27 TRAIN:  68%|██████▊   | 92/135 [00:21<00:09,  4.74it/s, epoch=27, grad_norm=118.877, loss=359.5035, lr=3.02e-04]
Epoch 27 TRAIN:  68%|██████▊   | 92/135 [00:21<00:09,  4.74it/s, epoch=27, grad_norm=177.931, loss=296.9493, lr=3.02e-04]
Epoch 27 TRAIN:  70%|███████   | 95/135 [00:21<00:08,  4.56it/s, epoch=27, grad_norm=177.931, loss=296.9493, lr=3.02e-04]
Epoch 27 TRAIN:  70%|███████   | 95/135 [00:21<00:08,  4.56it/s, epoch=27, grad_norm=209.795, loss=294.6428, lr=3.02e-04]
Epoch 27 TRAIN:  70%|███████   | 95/135 [00:22<00:08,  4.56it/s, epoch=27, grad_norm=124.129, loss=243.2764, lr=3.02e-04]
Epoch 27 TRAIN:  70%|███████   | 95/135 [00:22<00:08,  4.56it/s, epoch=27, grad_norm=104.923, loss=296.6024, lr=3.02e-04]
Epoch 27 TRAIN:  73%|███████▎  | 98/135 [00:22<00:07,  4.74it/s, epoch=27, grad_norm=104.923, loss=296.6024, lr=3.02e-04]
Epoch 27 TRAIN:  73%|███████▎  | 98/135 [00:22<00:07,  4.74it/s, epoch=27, grad_norm=192.599, loss=375.9206, lr=3.02e-04]
Epoch 27 TRAIN:  73%|███████▎  | 98/135 [00:22<00:07,  4.74it/s, epoch=27, grad_norm=117.542, loss=267.2771, lr=3.02e-04]2026-04-16 20:05:34,409 DEBUG TRAIN | steps/sec 4.392| Batch 27/100 loss 267.277069 loss_att 235.636078 loss_ctc 341.106018 th_accuracy 0.250000 lr 3.0216e-04 grad_norm 117.542458 rank 0

Epoch 27 TRAIN:  73%|███████▎  | 98/135 [00:22<00:07,  4.74it/s, epoch=27, grad_norm=97.859, loss=322.2774, lr=3.02e-04] 
Epoch 27 TRAIN:  75%|███████▍  | 101/135 [00:22<00:06,  4.88it/s, epoch=27, grad_norm=97.859, loss=322.2774, lr=3.02e-04]
Epoch 27 TRAIN:  75%|███████▍  | 101/135 [00:23<00:06,  4.88it/s, epoch=27, grad_norm=90.438, loss=221.6483, lr=3.02e-04]
Epoch 27 TRAIN:  75%|███████▍  | 101/135 [00:23<00:06,  4.88it/s, epoch=27, grad_norm=92.275, loss=318.7046, lr=3.02e-04]
Epoch 27 TRAIN:  75%|███████▍  | 101/135 [00:23<00:06,  4.88it/s, epoch=27, grad_norm=94.138, loss=312.6939, lr=3.02e-04]
Epoch 27 TRAIN:  77%|███████▋  | 104/135 [00:23<00:06,  4.74it/s, epoch=27, grad_norm=94.138, loss=312.6939, lr=3.02e-04]
Epoch 27 TRAIN:  77%|███████▋  | 104/135 [00:23<00:06,  4.74it/s, epoch=27, grad_norm=101.828, loss=313.2591, lr=3.03e-04]
Epoch 27 TRAIN:  77%|███████▋  | 104/135 [00:24<00:06,  4.74it/s, epoch=27, grad_norm=86.108, loss=323.8927, lr=3.03e-04] 
Epoch 27 TRAIN:  77%|███████▋  | 104/135 [00:24<00:06,  4.74it/s, epoch=27, grad_norm=114.728, loss=302.2883, lr=3.03e-04]
Epoch 27 TRAIN:  79%|███████▉  | 107/135 [00:24<00:06,  4.64it/s, epoch=27, grad_norm=114.728, loss=302.2883, lr=3.03e-04]
Epoch 27 TRAIN:  79%|███████▉  | 107/135 [00:24<00:06,  4.64it/s, epoch=27, grad_norm=88.851, loss=278.1896, lr=3.03e-04] 
Epoch 27 TRAIN:  79%|███████▉  | 107/135 [00:24<00:06,  4.64it/s, epoch=27, grad_norm=111.733, loss=306.4268, lr=3.03e-04]
Epoch 27 TRAIN:  79%|███████▉  | 107/135 [00:24<00:06,  4.64it/s, epoch=27, grad_norm=97.710, loss=233.2158, lr=3.03e-04] 
Epoch 27 TRAIN:  81%|████████▏ | 110/135 [00:24<00:05,  4.80it/s, epoch=27, grad_norm=97.710, loss=233.2158, lr=3.03e-04]
Epoch 27 TRAIN:  81%|████████▏ | 110/135 [00:25<00:05,  4.80it/s, epoch=27, grad_norm=81.529, loss=289.5807, lr=3.03e-04]
Epoch 27 TRAIN:  81%|████████▏ | 110/135 [00:25<00:05,  4.80it/s, epoch=27, grad_norm=194.953, loss=350.6490, lr=3.03e-04]
Epoch 27 TRAIN:  81%|████████▏ | 110/135 [00:25<00:05,  4.80it/s, epoch=27, grad_norm=56.953, loss=228.0012, lr=3.03e-04] 
Epoch 27 TRAIN:  84%|████████▎ | 113/135 [00:25<00:04,  4.55it/s, epoch=27, grad_norm=56.953, loss=228.0012, lr=3.03e-04]
Epoch 27 TRAIN:  84%|████████▎ | 113/135 [00:25<00:04,  4.55it/s, epoch=27, grad_norm=242.687, loss=421.3497, lr=3.03e-04]
Epoch 27 TRAIN:  84%|████████▎ | 113/135 [00:25<00:04,  4.55it/s, epoch=27, grad_norm=207.539, loss=380.0705, lr=3.03e-04]
Epoch 27 TRAIN:  84%|████████▎ | 113/135 [00:26<00:04,  4.55it/s, epoch=27, grad_norm=126.648, loss=299.0583, lr=3.03e-04]
Epoch 27 TRAIN:  86%|████████▌ | 116/135 [00:26<00:03,  4.89it/s, epoch=27, grad_norm=126.648, loss=299.0583, lr=3.03e-04]
Epoch 27 TRAIN:  86%|████████▌ | 116/135 [00:26<00:03,  4.89it/s, epoch=27, grad_norm=292.849, loss=311.0641, lr=3.04e-04]
Epoch 27 TRAIN:  86%|████████▌ | 116/135 [00:26<00:03,  4.89it/s, epoch=27, grad_norm=199.277, loss=271.5153, lr=3.04e-04]
Epoch 27 TRAIN:  86%|████████▌ | 116/135 [00:26<00:03,  4.89it/s, epoch=27, grad_norm=181.536, loss=318.0798, lr=3.04e-04]
Epoch 27 TRAIN:  88%|████████▊ | 119/135 [00:26<00:03,  4.46it/s, epoch=27, grad_norm=181.536, loss=318.0798, lr=3.04e-04]
Epoch 27 TRAIN:  88%|████████▊ | 119/135 [00:27<00:03,  4.46it/s, epoch=27, grad_norm=154.204, loss=365.5706, lr=3.04e-04]
Epoch 27 TRAIN:  88%|████████▊ | 119/135 [00:27<00:03,  4.46it/s, epoch=27, grad_norm=85.057, loss=272.3724, lr=3.04e-04] 
Epoch 27 TRAIN:  88%|████████▊ | 119/135 [00:27<00:03,  4.46it/s, epoch=27, grad_norm=170.941, loss=334.3251, lr=3.04e-04]
Epoch 27 TRAIN:  88%|████████▊ | 119/135 [00:27<00:03,  4.46it/s, epoch=27, grad_norm=189.639, loss=284.9893, lr=3.04e-04]
Epoch 27 TRAIN:  91%|█████████ | 123/135 [00:27<00:02,  5.10it/s, epoch=27, grad_norm=189.639, loss=284.9893, lr=3.04e-04]
Epoch 27 TRAIN:  91%|█████████ | 123/135 [00:27<00:02,  5.10it/s, epoch=27, grad_norm=238.290, loss=322.3668, lr=3.04e-04]
Epoch 27 TRAIN:  91%|█████████ | 123/135 [00:27<00:02,  5.10it/s, epoch=27, grad_norm=130.895, loss=253.0580, lr=3.04e-04]
Epoch 27 TRAIN:  91%|█████████ | 123/135 [00:28<00:02,  5.10it/s, epoch=27, grad_norm=223.347, loss=398.0834, lr=3.04e-04]
Epoch 27 TRAIN:  93%|█████████▎| 126/135 [00:28<00:01,  4.84it/s, epoch=27, grad_norm=223.347, loss=398.0834, lr=3.04e-04]
Epoch 27 TRAIN:  93%|█████████▎| 126/135 [00:28<00:01,  4.84it/s, epoch=27, grad_norm=125.144, loss=349.7854, lr=3.04e-04]
Epoch 27 TRAIN:  93%|█████████▎| 126/135 [00:28<00:01,  4.84it/s, epoch=27, grad_norm=172.784, loss=359.8884, lr=3.04e-04]
Epoch 27 TRAIN:  93%|█████████▎| 126/135 [00:28<00:01,  4.84it/s, epoch=27, grad_norm=145.183, loss=367.7125, lr=3.04e-04]
Epoch 27 TRAIN:  96%|█████████▌| 129/135 [00:28<00:01,  4.91it/s, epoch=27, grad_norm=145.183, loss=367.7125, lr=3.04e-04]
Epoch 27 TRAIN:  96%|█████████▌| 129/135 [00:28<00:01,  4.91it/s, epoch=27, grad_norm=98.331, loss=299.6299, lr=3.05e-04] 
Epoch 27 TRAIN:  96%|█████████▌| 129/135 [00:29<00:01,  4.91it/s, epoch=27, grad_norm=174.897, loss=411.3724, lr=3.05e-04]
Epoch 27 TRAIN:  96%|█████████▌| 129/135 [00:29<00:01,  4.91it/s, epoch=27, grad_norm=172.430, loss=382.6854, lr=3.05e-04]
Epoch 27 TRAIN:  98%|█████████▊| 132/135 [00:29<00:00,  5.00it/s, epoch=27, grad_norm=172.430, loss=382.6854, lr=3.05e-04]
Epoch 27 TRAIN:  98%|█████████▊| 132/135 [00:29<00:00,  5.00it/s, epoch=27, grad_norm=142.482, loss=382.1698, lr=3.05e-04]
Epoch 27 TRAIN:  98%|█████████▊| 132/135 [00:29<00:00,  5.00it/s, epoch=27, grad_norm=102.985, loss=299.0286, lr=3.05e-04]
Epoch 27 TRAIN:  98%|█████████▊| 132/135 [00:29<00:00,  5.00it/s, epoch=27, grad_norm=88.026, loss=351.5520, lr=3.05e-04] 
Epoch 27 TRAIN: 100%|██████████| 135/135 [00:29<00:00,  5.09it/s, epoch=27, grad_norm=88.026, loss=351.5520, lr=3.05e-04]
Epoch 27 TRAIN: 100%|██████████| 135/135 [00:29<00:00,  5.09it/s, epoch=27, grad_norm=140.712, loss=306.5207, lr=3.05e-04]
Epoch 27 TRAIN: 136it [00:29,  4.54it/s, epoch=27, grad_norm=140.712, loss=306.5207, lr=3.05e-04]                         

CV:   0%|          | 0/6 [00:00<?, ?it/s]/home/rookie/Documents/batuka/projects/wenet/wenet/wenet/utils/train_utils.py:689: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  amp_autocast(enabled=dtype is not None,

CV:  17%|█▋        | 1/6 [00:00<00:03,  1.31it/s]
                                                 
2026-04-16 20:05:42,559 INFO Epoch 27 Step 3813 CV info lr 3.0504e-04 cv_loss 557.8743341619319 rank 0 acc 0.24893138060967127
2026-04-16 20:05:42,560 INFO [Rank 0] Checkpoint: save to checkpoint exp/conformer_mongolian/epoch_27.pt
2026-04-16 20:05:42,693 INFO Epoch 28 Step 3813 TRAIN info lr 3.0504e-04 rank 0
2026-04-16 20:05:42,709 INFO using accumulate grad, new batch size is 1 times larger than before

Epoch 28 TRAIN:   0%|          | 0/136 [00:00<?, ?it/s]
Epoch 28 TRAIN:   0%|          | 0/136 [00:04<?, ?it/s, epoch=28, grad_norm=464.821, loss=419.8413, lr=3.05e-04]
Epoch 28 TRAIN:   1%|          | 1/136 [00:04<09:07,  4.05s/it, epoch=28, grad_norm=464.821, loss=419.8413, lr=3.05e-04]
Epoch 28 TRAIN:   1%|          | 1/136 [00:04<09:07,  4.05s/it, epoch=28, grad_norm=535.083, loss=385.5999, lr=3.05e-04]
Epoch 28 TRAIN:   1%|          | 1/136 [00:04<09:07,  4.05s/it, epoch=28, grad_norm=167.860, loss=238.9956, lr=3.05e-04]
Epoch 28 TRAIN:   1%|          | 1/136 [00:04<09:07,  4.05s/it, epoch=28, grad_norm=430.101, loss=341.6265, lr=3.05e-04]
Epoch 28 TRAIN:   1%|          | 1/136 [00:04<09:07,  4.05s/it, epoch=28, grad_norm=109.343, loss=280.6899, lr=3.05e-04]
Epoch 28 TRAIN:   1%|          | 1/136 [00:04<09:07,  4.05s/it, epoch=28, grad_norm=353.529, loss=338.7208, lr=3.06e-04]
Epoch 28 TRAIN:   1%|          | 1/136 [00:04<09:07,  4.05s/it, epoch=28, grad_norm=116.145, loss=279.0022, lr=3.06e-04]
Epoch 28 TRAIN:   1%|          | 1/136 [00:04<09:07,  4.05s/it, epoch=28, grad_norm=108.807, loss=347.9457, lr=3.06e-04]
Epoch 28 TRAIN:   6%|▌         | 8/136 [00:04<00:55,  2.30it/s, epoch=28, grad_norm=108.807, loss=347.9457, lr=3.06e-04]
Epoch 28 TRAIN:   6%|▌         | 8/136 [00:04<00:55,  2.30it/s, epoch=28, grad_norm=133.205, loss=295.7172, lr=3.06e-04]
Epoch 28 TRAIN:   6%|▌         | 8/136 [00:04<00:55,  2.30it/s, epoch=28, grad_norm=241.492, loss=351.9926, lr=3.06e-04]
Epoch 28 TRAIN:   6%|▌         | 8/136 [00:05<00:55,  2.30it/s, epoch=28, grad_norm=142.282, loss=355.5206, lr=3.06e-04]
Epoch 28 TRAIN:   6%|▌         | 8/136 [00:05<00:55,  2.30it/s, epoch=28, grad_norm=108.918, loss=303.5602, lr=3.06e-04]
Epoch 28 TRAIN:   6%|▌         | 8/136 [00:05<00:55,  2.30it/s, epoch=28, grad_norm=183.427, loss=306.7115, lr=3.06e-04]
Epoch 28 TRAIN:  10%|▉         | 13/136 [00:05<00:37,  3.30it/s, epoch=28, grad_norm=183.427, loss=306.7115, lr=3.06e-04]
Epoch 28 TRAIN:  10%|▉         | 13/136 [00:05<00:37,  3.30it/s, epoch=28, grad_norm=83.173, loss=364.4817, lr=3.06e-04] 
Epoch 28 TRAIN:  10%|▉         | 13/136 [00:05<00:37,  3.30it/s, epoch=28, grad_norm=93.399, loss=370.7545, lr=3.06e-04]
Epoch 28 TRAIN:  10%|▉         | 13/136 [00:05<00:37,  3.30it/s, epoch=28, grad_norm=172.327, loss=317.0790, lr=3.06e-04]
Epoch 28 TRAIN:  10%|▉         | 13/136 [00:06<00:37,  3.30it/s, epoch=28, grad_norm=105.866, loss=338.2527, lr=3.06e-04]
Epoch 28 TRAIN:  12%|█▎        | 17/136 [00:06<00:30,  3.95it/s, epoch=28, grad_norm=105.866, loss=338.2527, lr=3.06e-04]
Epoch 28 TRAIN:  12%|█▎        | 17/136 [00:06<00:30,  3.95it/s, epoch=28, grad_norm=108.738, loss=328.0339, lr=3.06e-04]
Epoch 28 TRAIN:  12%|█▎        | 17/136 [00:06<00:30,  3.95it/s, epoch=28, grad_norm=77.214, loss=343.5938, lr=3.07e-04] 
Epoch 28 TRAIN:  12%|█▎        | 17/136 [00:06<00:30,  3.95it/s, epoch=28, grad_norm=59.616, loss=307.9957, lr=3.07e-04]
Epoch 28 TRAIN:  12%|█▎        | 17/136 [00:06<00:30,  3.95it/s, epoch=28, grad_norm=82.644, loss=304.8334, lr=3.07e-04]
Epoch 28 TRAIN:  15%|█▌        | 21/136 [00:06<00:25,  4.48it/s, epoch=28, grad_norm=82.644, loss=304.8334, lr=3.07e-04]
Epoch 28 TRAIN:  15%|█▌        | 21/136 [00:06<00:25,  4.48it/s, epoch=28, grad_norm=87.975, loss=324.0093, lr=3.07e-04]
Epoch 28 TRAIN:  15%|█▌        | 21/136 [00:07<00:25,  4.48it/s, epoch=28, grad_norm=121.186, loss=347.1733, lr=3.07e-04]
Epoch 28 TRAIN:  15%|█▌        | 21/136 [00:07<00:25,  4.48it/s, epoch=28, grad_norm=76.699, loss=291.0498, lr=3.07e-04] 
Epoch 28 TRAIN:  15%|█▌        | 21/136 [00:07<00:25,  4.48it/s, epoch=28, grad_norm=84.682, loss=348.9552, lr=3.07e-04]
Epoch 28 TRAIN:  18%|█▊        | 25/136 [00:07<00:23,  4.71it/s, epoch=28, grad_norm=84.682, loss=348.9552, lr=3.07e-04]
Epoch 28 TRAIN:  18%|█▊        | 25/136 [00:07<00:23,  4.71it/s, epoch=28, grad_norm=102.511, loss=363.2527, lr=3.07e-04]
Epoch 28 TRAIN:  18%|█▊        | 25/136 [00:07<00:23,  4.71it/s, epoch=28, grad_norm=102.003, loss=353.9663, lr=3.07e-04]
Epoch 28 TRAIN:  18%|█▊        | 25/136 [00:07<00:23,  4.71it/s, epoch=28, grad_norm=129.885, loss=305.5541, lr=3.07e-04]
Epoch 28 TRAIN:  18%|█▊        | 25/136 [00:08<00:23,  4.71it/s, epoch=28, grad_norm=112.817, loss=303.8962, lr=3.07e-04]
Epoch 28 TRAIN:  21%|██▏       | 29/136 [00:08<00:21,  4.99it/s, epoch=28, grad_norm=112.817, loss=303.8962, lr=3.07e-04]
Epoch 28 TRAIN:  21%|██▏       | 29/136 [00:08<00:21,  4.99it/s, epoch=28, grad_norm=104.187, loss=387.8891, lr=3.07e-04]
Epoch 28 TRAIN:  21%|██▏       | 29/136 [00:08<00:21,  4.99it/s, epoch=28, grad_norm=85.867, loss=330.4570, lr=3.08e-04] 
Epoch 28 TRAIN:  21%|██▏       | 29/136 [00:08<00:21,  4.99it/s, epoch=28, grad_norm=134.219, loss=294.8043, lr=3.08e-04]
Epoch 28 TRAIN:  24%|██▎       | 32/136 [00:08<00:20,  5.07it/s, epoch=28, grad_norm=134.219, loss=294.8043, lr=3.08e-04]
Epoch 28 TRAIN:  24%|██▎       | 32/136 [00:08<00:20,  5.07it/s, epoch=28, grad_norm=125.965, loss=303.7629, lr=3.08e-04]
Epoch 28 TRAIN:  24%|██▎       | 32/136 [00:09<00:20,  5.07it/s, epoch=28, grad_norm=92.799, loss=291.0057, lr=3.08e-04] 
Epoch 28 TRAIN:  24%|██▎       | 32/136 [00:09<00:20,  5.07it/s, epoch=28, grad_norm=65.719, loss=288.0722, lr=3.08e-04]
Epoch 28 TRAIN:  26%|██▌       | 35/136 [00:09<00:20,  4.94it/s, epoch=28, grad_norm=65.719, loss=288.0722, lr=3.08e-04]
Epoch 28 TRAIN:  26%|██▌       | 35/136 [00:09<00:20,  4.94it/s, epoch=28, grad_norm=129.046, loss=326.6815, lr=3.08e-04]
Epoch 28 TRAIN:  26%|██▌       | 35/136 [00:09<00:20,  4.94it/s, epoch=28, grad_norm=53.942, loss=257.2371, lr=3.08e-04] 
Epoch 28 TRAIN:  26%|██▌       | 35/136 [00:09<00:20,  4.94it/s, epoch=28, grad_norm=162.646, loss=348.5380, lr=3.08e-04]
Epoch 28 TRAIN:  26%|██▌       | 35/136 [00:10<00:20,  4.94it/s, epoch=28, grad_norm=182.344, loss=352.0103, lr=3.08e-04]
Epoch 28 TRAIN:  29%|██▊       | 39/136 [00:10<00:18,  5.15it/s, epoch=28, grad_norm=182.344, loss=352.0103, lr=3.08e-04]
Epoch 28 TRAIN:  29%|██▊       | 39/136 [00:10<00:18,  5.15it/s, epoch=28, grad_norm=92.762, loss=323.3145, lr=3.08e-04] 
Epoch 28 TRAIN:  29%|██▊       | 39/136 [00:10<00:18,  5.15it/s, epoch=28, grad_norm=67.060, loss=291.9964, lr=3.08e-04]
Epoch 28 TRAIN:  29%|██▊       | 39/136 [00:10<00:18,  5.15it/s, epoch=28, grad_norm=64.712, loss=304.7845, lr=3.08e-04]
Epoch 28 TRAIN:  31%|███       | 42/136 [00:10<00:18,  4.95it/s, epoch=28, grad_norm=64.712, loss=304.7845, lr=3.08e-04]
Epoch 28 TRAIN:  31%|███       | 42/136 [00:10<00:18,  4.95it/s, epoch=28, grad_norm=110.119, loss=348.9202, lr=3.08e-04]
Epoch 28 TRAIN:  31%|███       | 42/136 [00:11<00:18,  4.95it/s, epoch=28, grad_norm=134.912, loss=382.1479, lr=3.09e-04]
Epoch 28 TRAIN:  31%|███       | 42/136 [00:11<00:18,  4.95it/s, epoch=28, grad_norm=85.778, loss=332.5073, lr=3.09e-04] 
Epoch 28 TRAIN:  33%|███▎      | 45/136 [00:11<00:17,  5.10it/s, epoch=28, grad_norm=85.778, loss=332.5073, lr=3.09e-04]
Epoch 28 TRAIN:  33%|███▎      | 45/136 [00:11<00:17,  5.10it/s, epoch=28, grad_norm=97.506, loss=333.1725, lr=3.09e-04]
Epoch 28 TRAIN:  33%|███▎      | 45/136 [00:11<00:17,  5.10it/s, epoch=28, grad_norm=135.349, loss=307.7878, lr=3.09e-04]
Epoch 28 TRAIN:  33%|███▎      | 45/136 [00:11<00:17,  5.10it/s, epoch=28, grad_norm=150.518, loss=301.2945, lr=3.09e-04]
Epoch 28 TRAIN:  35%|███▌      | 48/136 [00:11<00:16,  5.23it/s, epoch=28, grad_norm=150.518, loss=301.2945, lr=3.09e-04]
Epoch 28 TRAIN:  35%|███▌      | 48/136 [00:12<00:16,  5.23it/s, epoch=28, grad_norm=77.482, loss=355.1339, lr=3.09e-04] 
Epoch 28 TRAIN:  35%|███▌      | 48/136 [00:12<00:16,  5.23it/s, epoch=28, grad_norm=83.488, loss=330.3445, lr=3.09e-04]
Epoch 28 TRAIN:  35%|███▌      | 48/136 [00:12<00:16,  5.23it/s, epoch=28, grad_norm=70.840, loss=291.8047, lr=3.09e-04]
Epoch 28 TRAIN:  38%|███▊      | 51/136 [00:12<00:16,  5.08it/s, epoch=28, grad_norm=70.840, loss=291.8047, lr=3.09e-04]
Epoch 28 TRAIN:  38%|███▊      | 51/136 [00:12<00:16,  5.08it/s, epoch=28, grad_norm=69.937, loss=325.7120, lr=3.09e-04]
Epoch 28 TRAIN:  38%|███▊      | 51/136 [00:12<00:16,  5.08it/s, epoch=28, grad_norm=73.356, loss=328.9082, lr=3.09e-04]
Epoch 28 TRAIN:  38%|███▊      | 51/136 [00:13<00:16,  5.08it/s, epoch=28, grad_norm=133.439, loss=362.0086, lr=3.09e-04]
Epoch 28 TRAIN:  40%|███▉      | 54/136 [00:13<00:15,  5.13it/s, epoch=28, grad_norm=133.439, loss=362.0086, lr=3.09e-04]
Epoch 28 TRAIN:  40%|███▉      | 54/136 [00:13<00:15,  5.13it/s, epoch=28, grad_norm=64.438, loss=298.2132, lr=3.09e-04] 
Epoch 28 TRAIN:  40%|███▉      | 54/136 [00:13<00:15,  5.13it/s, epoch=28, grad_norm=64.373, loss=295.3829, lr=3.10e-04]
Epoch 28 TRAIN:  40%|███▉      | 54/136 [00:13<00:15,  5.13it/s, epoch=28, grad_norm=121.097, loss=369.4473, lr=3.10e-04]
Epoch 28 TRAIN:  42%|████▏     | 57/136 [00:13<00:15,  5.07it/s, epoch=28, grad_norm=121.097, loss=369.4473, lr=3.10e-04]
Epoch 28 TRAIN:  42%|████▏     | 57/136 [00:13<00:15,  5.07it/s, epoch=28, grad_norm=59.180, loss=327.6983, lr=3.10e-04] 
Epoch 28 TRAIN:  42%|████▏     | 57/136 [00:14<00:15,  5.07it/s, epoch=28, grad_norm=87.987, loss=343.2318, lr=3.10e-04]
Epoch 28 TRAIN:  42%|████▏     | 57/136 [00:14<00:15,  5.07it/s, epoch=28, grad_norm=68.466, loss=327.3261, lr=3.10e-04]
Epoch 28 TRAIN:  44%|████▍     | 60/136 [00:14<00:15,  4.97it/s, epoch=28, grad_norm=68.466, loss=327.3261, lr=3.10e-04]
Epoch 28 TRAIN:  44%|████▍     | 60/136 [00:14<00:15,  4.97it/s, epoch=28, grad_norm=92.675, loss=344.4083, lr=3.10e-04]
Epoch 28 TRAIN:  44%|████▍     | 60/136 [00:14<00:15,  4.97it/s, epoch=28, grad_norm=125.661, loss=270.7690, lr=3.10e-04]
Epoch 28 TRAIN:  44%|████▍     | 60/136 [00:14<00:15,  4.97it/s, epoch=28, grad_norm=94.401, loss=282.9015, lr=3.10e-04] 
Epoch 28 TRAIN:  46%|████▋     | 63/136 [00:14<00:14,  5.07it/s, epoch=28, grad_norm=94.401, loss=282.9015, lr=3.10e-04]
Epoch 28 TRAIN:  46%|████▋     | 63/136 [00:15<00:14,  5.07it/s, epoch=28, grad_norm=91.567, loss=284.6609, lr=3.10e-04]
Epoch 28 TRAIN:  46%|████▋     | 63/136 [00:15<00:14,  5.07it/s, epoch=28, grad_norm=112.129, loss=359.9323, lr=3.10e-04]
Epoch 28 TRAIN:  46%|████▋     | 63/136 [00:15<00:14,  5.07it/s, epoch=28, grad_norm=88.914, loss=341.9904, lr=3.10e-04] 
Epoch 28 TRAIN:  49%|████▊     | 66/136 [00:15<00:13,  5.09it/s, epoch=28, grad_norm=88.914, loss=341.9904, lr=3.10e-04]
Epoch 28 TRAIN:  49%|████▊     | 66/136 [00:15<00:13,  5.09it/s, epoch=28, grad_norm=117.183, loss=336.2342, lr=3.10e-04]
Epoch 28 TRAIN:  49%|████▊     | 66/136 [00:15<00:13,  5.09it/s, epoch=28, grad_norm=89.339, loss=317.5016, lr=3.10e-04] 
Epoch 28 TRAIN:  49%|████▊     | 66/136 [00:16<00:13,  5.09it/s, epoch=28, grad_norm=137.993, loss=300.2230, lr=3.11e-04]
Epoch 28 TRAIN:  51%|█████     | 69/136 [00:16<00:13,  4.93it/s, epoch=28, grad_norm=137.993, loss=300.2230, lr=3.11e-04]
Epoch 28 TRAIN:  51%|█████     | 69/136 [00:16<00:13,  4.93it/s, epoch=28, grad_norm=79.829, loss=329.9910, lr=3.11e-04] 
Epoch 28 TRAIN:  51%|█████     | 69/136 [00:16<00:13,  4.93it/s, epoch=28, grad_norm=84.700, loss=322.8599, lr=3.11e-04]
Epoch 28 TRAIN:  51%|█████     | 69/136 [00:16<00:13,  4.93it/s, epoch=28, grad_norm=75.061, loss=298.1153, lr=3.11e-04]
Epoch 28 TRAIN:  53%|█████▎    | 72/136 [00:16<00:12,  5.18it/s, epoch=28, grad_norm=75.061, loss=298.1153, lr=3.11e-04]
Epoch 28 TRAIN:  53%|█████▎    | 72/136 [00:16<00:12,  5.18it/s, epoch=28, grad_norm=74.593, loss=309.5468, lr=3.11e-04]
Epoch 28 TRAIN:  53%|█████▎    | 72/136 [00:16<00:12,  5.18it/s, epoch=28, grad_norm=87.283, loss=346.1665, lr=3.11e-04]
Epoch 28 TRAIN:  53%|█████▎    | 72/136 [00:17<00:12,  5.18it/s, epoch=28, grad_norm=85.006, loss=322.1913, lr=3.11e-04]
Epoch 28 TRAIN:  55%|█████▌    | 75/136 [00:17<00:11,  5.09it/s, epoch=28, grad_norm=85.006, loss=322.1913, lr=3.11e-04]
Epoch 28 TRAIN:  55%|█████▌    | 75/136 [00:17<00:11,  5.09it/s, epoch=28, grad_norm=80.248, loss=325.2411, lr=3.11e-04]
Epoch 28 TRAIN:  55%|█████▌    | 75/136 [00:17<00:11,  5.09it/s, epoch=28, grad_norm=245.144, loss=251.3257, lr=3.11e-04]
Epoch 28 TRAIN:  55%|█████▌    | 75/136 [00:17<00:11,  5.09it/s, epoch=28, grad_norm=83.415, loss=317.7107, lr=3.11e-04] 
Epoch 28 TRAIN:  57%|█████▋    | 78/136 [00:17<00:11,  5.13it/s, epoch=28, grad_norm=83.415, loss=317.7107, lr=3.11e-04]
Epoch 28 TRAIN:  57%|█████▋    | 78/136 [00:17<00:11,  5.13it/s, epoch=28, grad_norm=101.818, loss=309.9417, lr=3.11e-04]
Epoch 28 TRAIN:  57%|█████▋    | 78/136 [00:18<00:11,  5.13it/s, epoch=28, grad_norm=92.715, loss=307.9496, lr=3.11e-04] 
Epoch 28 TRAIN:  57%|█████▋    | 78/136 [00:18<00:11,  5.13it/s, epoch=28, grad_norm=76.504, loss=311.1115, lr=3.12e-04]
Epoch 28 TRAIN:  60%|█████▉    | 81/136 [00:18<00:10,  5.10it/s, epoch=28, grad_norm=76.504, loss=311.1115, lr=3.12e-04]
Epoch 28 TRAIN:  60%|█████▉    | 81/136 [00:18<00:10,  5.10it/s, epoch=28, grad_norm=127.594, loss=312.0992, lr=3.12e-04]
Epoch 28 TRAIN:  60%|█████▉    | 81/136 [00:18<00:10,  5.10it/s, epoch=28, grad_norm=79.594, loss=294.2123, lr=3.12e-04] 
Epoch 28 TRAIN:  60%|█████▉    | 81/136 [00:18<00:10,  5.10it/s, epoch=28, grad_norm=93.980, loss=319.3288, lr=3.12e-04]
Epoch 28 TRAIN:  62%|██████▏   | 84/136 [00:18<00:10,  5.02it/s, epoch=28, grad_norm=93.980, loss=319.3288, lr=3.12e-04]
Epoch 28 TRAIN:  62%|██████▏   | 84/136 [00:19<00:10,  5.02it/s, epoch=28, grad_norm=218.991, loss=319.3588, lr=3.12e-04]
Epoch 28 TRAIN:  62%|██████▏   | 84/136 [00:19<00:10,  5.02it/s, epoch=28, grad_norm=141.792, loss=355.7079, lr=3.12e-04]
Epoch 28 TRAIN:  62%|██████▏   | 84/136 [00:19<00:10,  5.02it/s, epoch=28, grad_norm=200.143, loss=423.9684, lr=3.12e-04]
Epoch 28 TRAIN:  64%|██████▍   | 87/136 [00:19<00:09,  5.07it/s, epoch=28, grad_norm=200.143, loss=423.9684, lr=3.12e-04]
Epoch 28 TRAIN:  64%|██████▍   | 87/136 [00:19<00:09,  5.07it/s, epoch=28, grad_norm=84.053, loss=342.4404, lr=3.12e-04] 
Epoch 28 TRAIN:  64%|██████▍   | 87/136 [00:19<00:09,  5.07it/s, epoch=28, grad_norm=177.736, loss=243.0038, lr=3.12e-04]
Epoch 28 TRAIN:  64%|██████▍   | 87/136 [00:20<00:09,  5.07it/s, epoch=28, grad_norm=314.591, loss=253.9271, lr=3.12e-04]
Epoch 28 TRAIN:  66%|██████▌   | 90/136 [00:20<00:09,  5.05it/s, epoch=28, grad_norm=314.591, loss=253.9271, lr=3.12e-04]
Epoch 28 TRAIN:  66%|██████▌   | 90/136 [00:20<00:09,  5.05it/s, epoch=28, grad_norm=84.613, loss=328.0312, lr=3.12e-04] 
Epoch 28 TRAIN:  66%|██████▌   | 90/136 [00:20<00:09,  5.05it/s, epoch=28, grad_norm=74.112, loss=307.3957, lr=3.12e-04]
Epoch 28 TRAIN:  66%|██████▌   | 90/136 [00:20<00:09,  5.05it/s, epoch=28, grad_norm=249.342, loss=368.8103, lr=3.12e-04]
Epoch 28 TRAIN:  68%|██████▊   | 93/136 [00:20<00:08,  4.92it/s, epoch=28, grad_norm=249.342, loss=368.8103, lr=3.12e-04]
Epoch 28 TRAIN:  68%|██████▊   | 93/136 [00:20<00:08,  4.92it/s, epoch=28, grad_norm=181.365, loss=408.7251, lr=3.13e-04]
Epoch 28 TRAIN:  68%|██████▊   | 93/136 [00:21<00:08,  4.92it/s, epoch=28, grad_norm=204.762, loss=293.8762, lr=3.13e-04]
Epoch 28 TRAIN:  68%|██████▊   | 93/136 [00:21<00:08,  4.92it/s, epoch=28, grad_norm=104.490, loss=321.1534, lr=3.13e-04]
Epoch 28 TRAIN:  71%|███████   | 96/136 [00:21<00:08,  4.87it/s, epoch=28, grad_norm=104.490, loss=321.1534, lr=3.13e-04]
Epoch 28 TRAIN:  71%|███████   | 96/136 [00:21<00:08,  4.87it/s, epoch=28, grad_norm=186.392, loss=373.9367, lr=3.13e-04]
Epoch 28 TRAIN:  71%|███████   | 96/136 [00:21<00:08,  4.87it/s, epoch=28, grad_norm=91.743, loss=333.6880, lr=3.13e-04] 
Epoch 28 TRAIN:  71%|███████   | 96/136 [00:22<00:08,  4.87it/s, epoch=28, grad_norm=106.109, loss=256.3824, lr=3.13e-04]
Epoch 28 TRAIN:  73%|███████▎  | 99/136 [00:22<00:07,  4.85it/s, epoch=28, grad_norm=106.109, loss=256.3824, lr=3.13e-04]
Epoch 28 TRAIN:  73%|███████▎  | 99/136 [00:22<00:07,  4.85it/s, epoch=28, grad_norm=91.434, loss=271.4917, lr=3.13e-04] 2026-04-16 20:06:04,956 DEBUG TRAIN | steps/sec 4.452| Batch 28/100 loss 271.491730 loss_att 240.308029 loss_ctc 344.253662 th_accuracy 0.297959 lr 3.1304e-04 grad_norm 91.434456 rank 0

Epoch 28 TRAIN:  73%|███████▎  | 99/136 [00:22<00:07,  4.85it/s, epoch=28, grad_norm=223.770, loss=314.9109, lr=3.13e-04]
Epoch 28 TRAIN:  73%|███████▎  | 99/136 [00:22<00:07,  4.85it/s, epoch=28, grad_norm=105.746, loss=228.4960, lr=3.13e-04]
Epoch 28 TRAIN:  75%|███████▌  | 102/136 [00:22<00:06,  4.90it/s, epoch=28, grad_norm=105.746, loss=228.4960, lr=3.13e-04]
Epoch 28 TRAIN:  75%|███████▌  | 102/136 [00:22<00:06,  4.90it/s, epoch=28, grad_norm=119.913, loss=304.9031, lr=3.13e-04]
Epoch 28 TRAIN:  75%|███████▌  | 102/136 [00:23<00:06,  4.90it/s, epoch=28, grad_norm=217.716, loss=328.9149, lr=3.13e-04]
Epoch 28 TRAIN:  75%|███████▌  | 102/136 [00:23<00:06,  4.90it/s, epoch=28, grad_norm=108.931, loss=252.3322, lr=3.13e-04]
Epoch 28 TRAIN:  77%|███████▋  | 105/136 [00:23<00:06,  4.98it/s, epoch=28, grad_norm=108.931, loss=252.3322, lr=3.13e-04]
Epoch 28 TRAIN:  77%|███████▋  | 105/136 [00:23<00:06,  4.98it/s, epoch=28, grad_norm=135.769, loss=297.6078, lr=3.14e-04]
Epoch 28 TRAIN:  77%|███████▋  | 105/136 [00:23<00:06,  4.98it/s, epoch=28, grad_norm=117.105, loss=247.1108, lr=3.14e-04]
Epoch 28 TRAIN:  77%|███████▋  | 105/136 [00:23<00:06,  4.98it/s, epoch=28, grad_norm=114.406, loss=273.3266, lr=3.14e-04]
Epoch 28 TRAIN:  79%|███████▉  | 108/136 [00:23<00:05,  4.91it/s, epoch=28, grad_norm=114.406, loss=273.3266, lr=3.14e-04]
Epoch 28 TRAIN:  79%|███████▉  | 108/136 [00:24<00:05,  4.91it/s, epoch=28, grad_norm=76.439, loss=292.5019, lr=3.14e-04] 
Epoch 28 TRAIN:  79%|███████▉  | 108/136 [00:24<00:05,  4.91it/s, epoch=28, grad_norm=102.351, loss=301.3358, lr=3.14e-04]
Epoch 28 TRAIN:  79%|███████▉  | 108/136 [00:24<00:05,  4.91it/s, epoch=28, grad_norm=86.693, loss=311.5995, lr=3.14e-04] 
Epoch 28 TRAIN:  82%|████████▏ | 111/136 [00:24<00:05,  4.91it/s, epoch=28, grad_norm=86.693, loss=311.5995, lr=3.14e-04]
Epoch 28 TRAIN:  82%|████████▏ | 111/136 [00:24<00:05,  4.91it/s, epoch=28, grad_norm=163.627, loss=231.1762, lr=3.14e-04]
Epoch 28 TRAIN:  82%|████████▏ | 111/136 [00:24<00:05,  4.91it/s, epoch=28, grad_norm=156.728, loss=318.7226, lr=3.14e-04]
Epoch 28 TRAIN:  82%|████████▏ | 111/136 [00:25<00:05,  4.91it/s, epoch=28, grad_norm=165.447, loss=206.0758, lr=3.14e-04]
Epoch 28 TRAIN:  84%|████████▍ | 114/136 [00:25<00:04,  4.96it/s, epoch=28, grad_norm=165.447, loss=206.0758, lr=3.14e-04]
Epoch 28 TRAIN:  84%|████████▍ | 114/136 [00:25<00:04,  4.96it/s, epoch=28, grad_norm=90.938, loss=229.4434, lr=3.14e-04] 
Epoch 28 TRAIN:  84%|████████▍ | 114/136 [00:25<00:04,  4.96it/s, epoch=28, grad_norm=89.395, loss=296.8818, lr=3.14e-04]
Epoch 28 TRAIN:  84%|████████▍ | 114/136 [00:25<00:04,  4.96it/s, epoch=28, grad_norm=119.704, loss=312.6830, lr=3.14e-04]
Epoch 28 TRAIN:  86%|████████▌ | 117/136 [00:25<00:03,  4.91it/s, epoch=28, grad_norm=119.704, loss=312.6830, lr=3.14e-04]
Epoch 28 TRAIN:  86%|████████▌ | 117/136 [00:25<00:03,  4.91it/s, epoch=28, grad_norm=152.818, loss=310.2009, lr=3.14e-04]
Epoch 28 TRAIN:  86%|████████▌ | 117/136 [00:26<00:03,  4.91it/s, epoch=28, grad_norm=110.591, loss=289.9291, lr=3.15e-04]
Epoch 28 TRAIN:  86%|████████▌ | 117/136 [00:26<00:03,  4.91it/s, epoch=28, grad_norm=240.376, loss=442.6572, lr=3.15e-04]
Epoch 28 TRAIN:  88%|████████▊ | 120/136 [00:26<00:03,  4.73it/s, epoch=28, grad_norm=240.376, loss=442.6572, lr=3.15e-04]
Epoch 28 TRAIN:  88%|████████▊ | 120/136 [00:26<00:03,  4.73it/s, epoch=28, grad_norm=91.313, loss=307.9348, lr=3.15e-04] 
Epoch 28 TRAIN:  88%|████████▊ | 120/136 [00:26<00:03,  4.73it/s, epoch=28, grad_norm=99.314, loss=333.5452, lr=3.15e-04]
Epoch 28 TRAIN:  88%|████████▊ | 120/136 [00:26<00:03,  4.73it/s, epoch=28, grad_norm=180.629, loss=388.1407, lr=3.15e-04]
Epoch 28 TRAIN:  90%|█████████ | 123/136 [00:26<00:02,  4.95it/s, epoch=28, grad_norm=180.629, loss=388.1407, lr=3.15e-04]
Epoch 28 TRAIN:  90%|█████████ | 123/136 [00:27<00:02,  4.95it/s, epoch=28, grad_norm=176.867, loss=309.5386, lr=3.15e-04]
Epoch 28 TRAIN:  90%|█████████ | 123/136 [00:27<00:02,  4.95it/s, epoch=28, grad_norm=238.532, loss=271.2328, lr=3.15e-04]
Epoch 28 TRAIN:  90%|█████████ | 123/136 [00:27<00:02,  4.95it/s, epoch=28, grad_norm=305.143, loss=392.8445, lr=3.15e-04]
Epoch 28 TRAIN:  93%|█████████▎| 126/136 [00:27<00:02,  4.90it/s, epoch=28, grad_norm=305.143, loss=392.8445, lr=3.15e-04]
Epoch 28 TRAIN:  93%|█████████▎| 126/136 [00:27<00:02,  4.90it/s, epoch=28, grad_norm=152.639, loss=396.1385, lr=3.15e-04]
Epoch 28 TRAIN:  93%|█████████▎| 126/136 [00:27<00:02,  4.90it/s, epoch=28, grad_norm=149.861, loss=303.7562, lr=3.15e-04]
Epoch 28 TRAIN:  93%|█████████▎| 126/136 [00:28<00:02,  4.90it/s, epoch=28, grad_norm=158.782, loss=335.5120, lr=3.15e-04]
Epoch 28 TRAIN:  95%|█████████▍| 129/136 [00:28<00:01,  5.08it/s, epoch=28, grad_norm=158.782, loss=335.5120, lr=3.15e-04]
Epoch 28 TRAIN:  95%|█████████▍| 129/136 [00:28<00:01,  5.08it/s, epoch=28, grad_norm=128.711, loss=336.7957, lr=3.15e-04]
Epoch 28 TRAIN:  95%|█████████▍| 129/136 [00:28<00:01,  5.08it/s, epoch=28, grad_norm=188.695, loss=401.9768, lr=3.16e-04]
Epoch 28 TRAIN:  95%|█████████▍| 129/136 [00:28<00:01,  5.08it/s, epoch=28, grad_norm=209.266, loss=385.0431, lr=3.16e-04]
Epoch 28 TRAIN:  97%|█████████▋| 132/136 [00:28<00:00,  5.15it/s, epoch=28, grad_norm=209.266, loss=385.0431, lr=3.16e-04]
Epoch 28 TRAIN:  97%|█████████▋| 132/136 [00:28<00:00,  5.15it/s, epoch=28, grad_norm=178.915, loss=334.9195, lr=3.16e-04]
Epoch 28 TRAIN:  97%|█████████▋| 132/136 [00:28<00:00,  5.15it/s, epoch=28, grad_norm=170.086, loss=290.6256, lr=3.16e-04]
Epoch 28 TRAIN:  97%|█████████▋| 132/136 [00:29<00:00,  5.15it/s, epoch=28, grad_norm=106.614, loss=316.3599, lr=3.16e-04]
Epoch 28 TRAIN:  97%|█████████▋| 132/136 [00:29<00:00,  5.15it/s, epoch=28, grad_norm=220.417, loss=409.7645, lr=3.16e-04]
Epoch 28 TRAIN: 100%|██████████| 136/136 [00:29<00:00,  5.64it/s, epoch=28, grad_norm=220.417, loss=409.7645, lr=3.16e-04]
Epoch 28 TRAIN: 100%|██████████| 136/136 [00:29<00:00,  5.64it/s, epoch=28, grad_norm=157.717, loss=398.0397, lr=3.16e-04]
Epoch 28 TRAIN: 100%|██████████| 136/136 [00:29<00:00,  5.64it/s, epoch=28, grad_norm=146.992, loss=362.0762, lr=3.16e-04]
Epoch 28 TRAIN: 138it [00:29,  4.67it/s, epoch=28, grad_norm=146.992, loss=362.0762, lr=3.16e-04]                         

CV:   0%|          | 0/6 [00:00<?, ?it/s]/home/rookie/Documents/batuka/projects/wenet/wenet/wenet/utils/train_utils.py:689: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  amp_autocast(enabled=dtype is not None,

CV:  17%|█▋        | 1/6 [00:00<00:03,  1.33it/s]
                                                 
2026-04-16 20:06:13,097 INFO Epoch 28 Step 3951 CV info lr 3.1608e-04 cv_loss 519.3414473100142 rank 0 acc 0.25123825172583264
2026-04-16 20:06:13,098 INFO [Rank 0] Checkpoint: save to checkpoint exp/conformer_mongolian/epoch_28.pt
2026-04-16 20:06:13,213 INFO Epoch 29 Step 3951 TRAIN info lr 3.1608e-04 rank 0
2026-04-16 20:06:13,228 INFO using accumulate grad, new batch size is 1 times larger than before

Epoch 29 TRAIN:   0%|          | 0/138 [00:00<?, ?it/s]
Epoch 29 TRAIN:   0%|          | 0/138 [00:03<?, ?it/s, epoch=29, grad_norm=381.263, loss=345.4266, lr=3.16e-04]
Epoch 29 TRAIN:   1%|          | 1/138 [00:03<09:07,  4.00s/it, epoch=29, grad_norm=381.263, loss=345.4266, lr=3.16e-04]
Epoch 29 TRAIN:   1%|          | 1/138 [00:04<09:07,  4.00s/it, epoch=29, grad_norm=358.563, loss=335.2319, lr=3.16e-04]
Epoch 29 TRAIN:   1%|          | 1/138 [00:04<09:07,  4.00s/it, epoch=29, grad_norm=380.571, loss=380.4738, lr=3.16e-04]
Epoch 29 TRAIN:   1%|          | 1/138 [00:04<09:07,  4.00s/it, epoch=29, grad_norm=486.350, loss=501.2980, lr=3.16e-04]
Epoch 29 TRAIN:   1%|          | 1/138 [00:04<09:07,  4.00s/it, epoch=29, grad_norm=241.809, loss=332.8287, lr=3.16e-04]
Epoch 29 TRAIN:   1%|          | 1/138 [00:04<09:07,  4.00s/it, epoch=29, grad_norm=187.556, loss=261.3151, lr=3.17e-04]
Epoch 29 TRAIN:   1%|          | 1/138 [00:04<09:07,  4.00s/it, epoch=29, grad_norm=210.852, loss=330.3943, lr=3.17e-04]
Epoch 29 TRAIN:   1%|          | 1/138 [00:04<09:07,  4.00s/it, epoch=29, grad_norm=103.917, loss=237.0009, lr=3.17e-04]
Epoch 29 TRAIN:   1%|          | 1/138 [00:04<09:07,  4.00s/it, epoch=29, grad_norm=121.047, loss=345.3619, lr=3.17e-04]
Epoch 29 TRAIN:   7%|▋         | 9/138 [00:04<00:50,  2.53it/s, epoch=29, grad_norm=121.047, loss=345.3619, lr=3.17e-04]
Epoch 29 TRAIN:   7%|▋         | 9/138 [00:04<00:50,  2.53it/s, epoch=29, grad_norm=127.995, loss=292.3852, lr=3.17e-04]
Epoch 29 TRAIN:   7%|▋         | 9/138 [00:04<00:50,  2.53it/s, epoch=29, grad_norm=175.425, loss=322.0853, lr=3.17e-04]
Epoch 29 TRAIN:   7%|▋         | 9/138 [00:05<00:50,  2.53it/s, epoch=29, grad_norm=275.290, loss=305.7757, lr=3.17e-04]
Epoch 29 TRAIN:   7%|▋         | 9/138 [00:05<00:50,  2.53it/s, epoch=29, grad_norm=138.809, loss=320.1758, lr=3.17e-04]
Epoch 29 TRAIN:   9%|▉         | 13/138 [00:05<00:37,  3.32it/s, epoch=29, grad_norm=138.809, loss=320.1758, lr=3.17e-04]
Epoch 29 TRAIN:   9%|▉         | 13/138 [00:05<00:37,  3.32it/s, epoch=29, grad_norm=65.325, loss=349.9684, lr=3.17e-04] 
Epoch 29 TRAIN:   9%|▉         | 13/138 [00:05<00:37,  3.32it/s, epoch=29, grad_norm=128.672, loss=276.6234, lr=3.17e-04]
Epoch 29 TRAIN:   9%|▉         | 13/138 [00:05<00:37,  3.32it/s, epoch=29, grad_norm=88.586, loss=330.3673, lr=3.17e-04] 
Epoch 29 TRAIN:   9%|▉         | 13/138 [00:05<00:37,  3.32it/s, epoch=29, grad_norm=79.139, loss=298.3307, lr=3.17e-04]
Epoch 29 TRAIN:  12%|█▏        | 17/138 [00:05<00:30,  4.02it/s, epoch=29, grad_norm=79.139, loss=298.3307, lr=3.17e-04]
Epoch 29 TRAIN:  12%|█▏        | 17/138 [00:06<00:30,  4.02it/s, epoch=29, grad_norm=88.408, loss=339.6333, lr=3.18e-04]
Epoch 29 TRAIN:  12%|█▏        | 17/138 [00:06<00:30,  4.02it/s, epoch=29, grad_norm=94.050, loss=272.0129, lr=3.18e-04]
Epoch 29 TRAIN:  12%|█▏        | 17/138 [00:06<00:30,  4.02it/s, epoch=29, grad_norm=101.846, loss=313.3299, lr=3.18e-04]
Epoch 29 TRAIN:  12%|█▏        | 17/138 [00:06<00:30,  4.02it/s, epoch=29, grad_norm=69.262, loss=290.4097, lr=3.18e-04] 
Epoch 29 TRAIN:  15%|█▌        | 21/138 [00:06<00:25,  4.56it/s, epoch=29, grad_norm=69.262, loss=290.4097, lr=3.18e-04]
Epoch 29 TRAIN:  15%|█▌        | 21/138 [00:06<00:25,  4.56it/s, epoch=29, grad_norm=185.367, loss=360.9367, lr=3.18e-04]
Epoch 29 TRAIN:  15%|█▌        | 21/138 [00:07<00:25,  4.56it/s, epoch=29, grad_norm=132.702, loss=351.5581, lr=3.18e-04]
Epoch 29 TRAIN:  15%|█▌        | 21/138 [00:07<00:25,  4.56it/s, epoch=29, grad_norm=59.208, loss=298.7780, lr=3.18e-04] 
Epoch 29 TRAIN:  15%|█▌        | 21/138 [00:07<00:25,  4.56it/s, epoch=29, grad_norm=78.822, loss=316.4279, lr=3.18e-04]
Epoch 29 TRAIN:  18%|█▊        | 25/138 [00:07<00:24,  4.68it/s, epoch=29, grad_norm=78.822, loss=316.4279, lr=3.18e-04]
Epoch 29 TRAIN:  18%|█▊        | 25/138 [00:07<00:24,  4.68it/s, epoch=29, grad_norm=91.023, loss=348.3670, lr=3.18e-04]
Epoch 29 TRAIN:  18%|█▊        | 25/138 [00:07<00:24,  4.68it/s, epoch=29, grad_norm=89.800, loss=288.4762, lr=3.18e-04]
Epoch 29 TRAIN:  18%|█▊        | 25/138 [00:07<00:24,  4.68it/s, epoch=29, grad_norm=65.341, loss=317.2773, lr=3.18e-04]
Epoch 29 TRAIN:  20%|██        | 28/138 [00:07<00:22,  4.93it/s, epoch=29, grad_norm=65.341, loss=317.2773, lr=3.18e-04]
Epoch 29 TRAIN:  20%|██        | 28/138 [00:08<00:22,  4.93it/s, epoch=29, grad_norm=87.008, loss=347.0702, lr=3.18e-04]
Epoch 29 TRAIN:  20%|██        | 28/138 [00:08<00:22,  4.93it/s, epoch=29, grad_norm=90.347, loss=316.6681, lr=3.18e-04]
Epoch 29 TRAIN:  20%|██        | 28/138 [00:08<00:22,  4.93it/s, epoch=29, grad_norm=76.454, loss=314.6223, lr=3.19e-04]
Epoch 29 TRAIN:  22%|██▏       | 31/138 [00:08<00:21,  4.96it/s, epoch=29, grad_norm=76.454, loss=314.6223, lr=3.19e-04]
Epoch 29 TRAIN:  22%|██▏       | 31/138 [00:08<00:21,  4.96it/s, epoch=29, grad_norm=92.844, loss=335.4410, lr=3.19e-04]
Epoch 29 TRAIN:  22%|██▏       | 31/138 [00:08<00:21,  4.96it/s, epoch=29, grad_norm=87.604, loss=353.9269, lr=3.19e-04]
Epoch 29 TRAIN:  22%|██▏       | 31/138 [00:08<00:21,  4.96it/s, epoch=29, grad_norm=104.202, loss=318.2340, lr=3.19e-04]
Epoch 29 TRAIN:  25%|██▍       | 34/138 [00:08<00:20,  5.13it/s, epoch=29, grad_norm=104.202, loss=318.2340, lr=3.19e-04]
Epoch 29 TRAIN:  25%|██▍       | 34/138 [00:09<00:20,  5.13it/s, epoch=29, grad_norm=96.489, loss=336.2945, lr=3.19e-04] 
Epoch 29 TRAIN:  25%|██▍       | 34/138 [00:09<00:20,  5.13it/s, epoch=29, grad_norm=76.924, loss=329.0168, lr=3.19e-04]
Epoch 29 TRAIN:  25%|██▍       | 34/138 [00:09<00:20,  5.13it/s, epoch=29, grad_norm=113.764, loss=340.8209, lr=3.19e-04]
Epoch 29 TRAIN:  27%|██▋       | 37/138 [00:09<00:20,  4.95it/s, epoch=29, grad_norm=113.764, loss=340.8209, lr=3.19e-04]
Epoch 29 TRAIN:  27%|██▋       | 37/138 [00:09<00:20,  4.95it/s, epoch=29, grad_norm=139.295, loss=301.9844, lr=3.19e-04]
Epoch 29 TRAIN:  27%|██▋       | 37/138 [00:10<00:20,  4.95it/s, epoch=29, grad_norm=88.159, loss=371.8602, lr=3.19e-04] 
Epoch 29 TRAIN:  27%|██▋       | 37/138 [00:10<00:20,  4.95it/s, epoch=29, grad_norm=164.766, loss=297.3310, lr=3.19e-04]
Epoch 29 TRAIN:  29%|██▉       | 40/138 [00:10<00:19,  5.00it/s, epoch=29, grad_norm=164.766, loss=297.3310, lr=3.19e-04]
Epoch 29 TRAIN:  29%|██▉       | 40/138 [00:10<00:19,  5.00it/s, epoch=29, grad_norm=153.968, loss=300.2347, lr=3.19e-04]
Epoch 29 TRAIN:  29%|██▉       | 40/138 [00:10<00:19,  5.00it/s, epoch=29, grad_norm=79.633, loss=326.7407, lr=3.19e-04] 
Epoch 29 TRAIN:  29%|██▉       | 40/138 [00:10<00:19,  5.00it/s, epoch=29, grad_norm=100.954, loss=353.8532, lr=3.20e-04]
Epoch 29 TRAIN:  31%|███       | 43/138 [00:10<00:18,  5.05it/s, epoch=29, grad_norm=100.954, loss=353.8532, lr=3.20e-04]
Epoch 29 TRAIN:  31%|███       | 43/138 [00:11<00:18,  5.05it/s, epoch=29, grad_norm=82.077, loss=320.8193, lr=3.20e-04] 
Epoch 29 TRAIN:  31%|███       | 43/138 [00:11<00:18,  5.05it/s, epoch=29, grad_norm=131.715, loss=372.5308, lr=3.20e-04]
Epoch 29 TRAIN:  31%|███       | 43/138 [00:11<00:18,  5.05it/s, epoch=29, grad_norm=81.230, loss=316.3204, lr=3.20e-04] 
Epoch 29 TRAIN:  33%|███▎      | 46/138 [00:11<00:18,  5.08it/s, epoch=29, grad_norm=81.230, loss=316.3204, lr=3.20e-04]
Epoch 29 TRAIN:  33%|███▎      | 46/138 [00:11<00:18,  5.08it/s, epoch=29, grad_norm=91.678, loss=332.3763, lr=3.20e-04]
Epoch 29 TRAIN:  33%|███▎      | 46/138 [00:11<00:18,  5.08it/s, epoch=29, grad_norm=68.555, loss=315.6425, lr=3.20e-04]
Epoch 29 TRAIN:  33%|███▎      | 46/138 [00:11<00:18,  5.08it/s, epoch=29, grad_norm=84.988, loss=309.6172, lr=3.20e-04]
Epoch 29 TRAIN:  36%|███▌      | 49/138 [00:11<00:17,  5.09it/s, epoch=29, grad_norm=84.988, loss=309.6172, lr=3.20e-04]
Epoch 29 TRAIN:  36%|███▌      | 49/138 [00:12<00:17,  5.09it/s, epoch=29, grad_norm=104.276, loss=286.5336, lr=3.20e-04]
Epoch 29 TRAIN:  36%|███▌      | 49/138 [00:12<00:17,  5.09it/s, epoch=29, grad_norm=102.736, loss=286.6089, lr=3.20e-04]
Epoch 29 TRAIN:  36%|███▌      | 49/138 [00:12<00:17,  5.09it/s, epoch=29, grad_norm=89.510, loss=302.2056, lr=3.20e-04] 
Epoch 29 TRAIN:  38%|███▊      | 52/138 [00:12<00:17,  4.89it/s, epoch=29, grad_norm=89.510, loss=302.2056, lr=3.20e-04]
Epoch 29 TRAIN:  38%|███▊      | 52/138 [00:12<00:17,  4.89it/s, epoch=29, grad_norm=184.987, loss=386.1367, lr=3.20e-04]
Epoch 29 TRAIN:  38%|███▊      | 52/138 [00:13<00:17,  4.89it/s, epoch=29, grad_norm=124.964, loss=340.5610, lr=3.20e-04]
Epoch 29 TRAIN:  38%|███▊      | 52/138 [00:13<00:17,  4.89it/s, epoch=29, grad_norm=116.882, loss=338.4136, lr=3.20e-04]
Epoch 29 TRAIN:  40%|███▉      | 55/138 [00:13<00:16,  4.92it/s, epoch=29, grad_norm=116.882, loss=338.4136, lr=3.20e-04]
Epoch 29 TRAIN:  40%|███▉      | 55/138 [00:13<00:16,  4.92it/s, epoch=29, grad_norm=85.319, loss=273.0001, lr=3.21e-04] 
Epoch 29 TRAIN:  40%|███▉      | 55/138 [00:13<00:16,  4.92it/s, epoch=29, grad_norm=77.210, loss=324.9275, lr=3.21e-04]
Epoch 29 TRAIN:  40%|███▉      | 55/138 [00:13<00:16,  4.92it/s, epoch=29, grad_norm=101.539, loss=282.9592, lr=3.21e-04]
Epoch 29 TRAIN:  42%|████▏     | 58/138 [00:13<00:15,  5.11it/s, epoch=29, grad_norm=101.539, loss=282.9592, lr=3.21e-04]
Epoch 29 TRAIN:  42%|████▏     | 58/138 [00:14<00:15,  5.11it/s, epoch=29, grad_norm=87.673, loss=322.6169, lr=3.21e-04] 
Epoch 29 TRAIN:  42%|████▏     | 58/138 [00:14<00:15,  5.11it/s, epoch=29, grad_norm=75.148, loss=328.2182, lr=3.21e-04]
Epoch 29 TRAIN:  42%|████▏     | 58/138 [00:14<00:15,  5.11it/s, epoch=29, grad_norm=113.817, loss=364.7270, lr=3.21e-04]
Epoch 29 TRAIN:  44%|████▍     | 61/138 [00:14<00:15,  4.89it/s, epoch=29, grad_norm=113.817, loss=364.7270, lr=3.21e-04]
Epoch 29 TRAIN:  44%|████▍     | 61/138 [00:14<00:15,  4.89it/s, epoch=29, grad_norm=104.866, loss=313.0630, lr=3.21e-04]
Epoch 29 TRAIN:  44%|████▍     | 61/138 [00:14<00:15,  4.89it/s, epoch=29, grad_norm=83.623, loss=334.2467, lr=3.21e-04] 
Epoch 29 TRAIN:  44%|████▍     | 61/138 [00:15<00:15,  4.89it/s, epoch=29, grad_norm=75.794, loss=353.8527, lr=3.21e-04]
Epoch 29 TRAIN:  46%|████▋     | 64/138 [00:15<00:15,  4.92it/s, epoch=29, grad_norm=75.794, loss=353.8527, lr=3.21e-04]
Epoch 29 TRAIN:  46%|████▋     | 64/138 [00:15<00:15,  4.92it/s, epoch=29, grad_norm=142.817, loss=317.4975, lr=3.21e-04]
Epoch 29 TRAIN:  46%|████▋     | 64/138 [00:15<00:15,  4.92it/s, epoch=29, grad_norm=90.570, loss=344.8304, lr=3.21e-04] 
Epoch 29 TRAIN:  46%|████▋     | 64/138 [00:15<00:15,  4.92it/s, epoch=29, grad_norm=113.119, loss=315.1252, lr=3.21e-04]
Epoch 29 TRAIN:  49%|████▊     | 67/138 [00:15<00:14,  4.87it/s, epoch=29, grad_norm=113.119, loss=315.1252, lr=3.21e-04]
Epoch 29 TRAIN:  49%|████▊     | 67/138 [00:15<00:14,  4.87it/s, epoch=29, grad_norm=70.863, loss=353.5873, lr=3.22e-04] 
Epoch 29 TRAIN:  49%|████▊     | 67/138 [00:16<00:14,  4.87it/s, epoch=29, grad_norm=103.717, loss=293.0577, lr=3.22e-04]
Epoch 29 TRAIN:  49%|████▊     | 67/138 [00:16<00:14,  4.87it/s, epoch=29, grad_norm=94.738, loss=323.5584, lr=3.22e-04] 
Epoch 29 TRAIN:  51%|█████     | 70/138 [00:16<00:13,  4.94it/s, epoch=29, grad_norm=94.738, loss=323.5584, lr=3.22e-04]
Epoch 29 TRAIN:  51%|█████     | 70/138 [00:16<00:13,  4.94it/s, epoch=29, grad_norm=90.202, loss=296.1272, lr=3.22e-04]
Epoch 29 TRAIN:  51%|█████     | 70/138 [00:16<00:13,  4.94it/s, epoch=29, grad_norm=75.315, loss=311.8708, lr=3.22e-04]
Epoch 29 TRAIN:  51%|█████     | 70/138 [00:16<00:13,  4.94it/s, epoch=29, grad_norm=103.150, loss=302.7975, lr=3.22e-04]
Epoch 29 TRAIN:  53%|█████▎    | 73/138 [00:16<00:13,  4.89it/s, epoch=29, grad_norm=103.150, loss=302.7975, lr=3.22e-04]
Epoch 29 TRAIN:  53%|█████▎    | 73/138 [00:17<00:13,  4.89it/s, epoch=29, grad_norm=142.045, loss=295.1390, lr=3.22e-04]
Epoch 29 TRAIN:  53%|█████▎    | 73/138 [00:17<00:13,  4.89it/s, epoch=29, grad_norm=105.804, loss=307.2637, lr=3.22e-04]
Epoch 29 TRAIN:  53%|█████▎    | 73/138 [00:17<00:13,  4.89it/s, epoch=29, grad_norm=150.293, loss=229.5168, lr=3.22e-04]
Epoch 29 TRAIN:  55%|█████▌    | 76/138 [00:17<00:12,  4.86it/s, epoch=29, grad_norm=150.293, loss=229.5168, lr=3.22e-04]
Epoch 29 TRAIN:  55%|█████▌    | 76/138 [00:17<00:12,  4.86it/s, epoch=29, grad_norm=107.274, loss=347.6861, lr=3.22e-04]
Epoch 29 TRAIN:  55%|█████▌    | 76/138 [00:17<00:12,  4.86it/s, epoch=29, grad_norm=169.646, loss=391.6474, lr=3.22e-04]
Epoch 29 TRAIN:  55%|█████▌    | 76/138 [00:18<00:12,  4.86it/s, epoch=29, grad_norm=112.721, loss=345.7652, lr=3.22e-04]
Epoch 29 TRAIN:  57%|█████▋    | 79/138 [00:18<00:12,  4.83it/s, epoch=29, grad_norm=112.721, loss=345.7652, lr=3.22e-04]
Epoch 29 TRAIN:  57%|█████▋    | 79/138 [00:18<00:12,  4.83it/s, epoch=29, grad_norm=78.664, loss=321.2910, lr=3.22e-04] 
Epoch 29 TRAIN:  57%|█████▋    | 79/138 [00:18<00:12,  4.83it/s, epoch=29, grad_norm=173.483, loss=290.1441, lr=3.23e-04]
Epoch 29 TRAIN:  57%|█████▋    | 79/138 [00:18<00:12,  4.83it/s, epoch=29, grad_norm=156.820, loss=301.2623, lr=3.23e-04]
Epoch 29 TRAIN:  59%|█████▉    | 82/138 [00:18<00:11,  4.85it/s, epoch=29, grad_norm=156.820, loss=301.2623, lr=3.23e-04]
Epoch 29 TRAIN:  59%|█████▉    | 82/138 [00:18<00:11,  4.85it/s, epoch=29, grad_norm=119.871, loss=305.4333, lr=3.23e-04]
Epoch 29 TRAIN:  59%|█████▉    | 82/138 [00:19<00:11,  4.85it/s, epoch=29, grad_norm=85.712, loss=305.9713, lr=3.23e-04] 
Epoch 29 TRAIN:  59%|█████▉    | 82/138 [00:19<00:11,  4.85it/s, epoch=29, grad_norm=178.743, loss=390.2286, lr=3.23e-04]
Epoch 29 TRAIN:  62%|██████▏   | 85/138 [00:19<00:10,  4.87it/s, epoch=29, grad_norm=178.743, loss=390.2286, lr=3.23e-04]
Epoch 29 TRAIN:  62%|██████▏   | 85/138 [00:19<00:10,  4.87it/s, epoch=29, grad_norm=111.374, loss=360.7917, lr=3.23e-04]
Epoch 29 TRAIN:  62%|██████▏   | 85/138 [00:19<00:10,  4.87it/s, epoch=29, grad_norm=114.327, loss=351.4115, lr=3.23e-04]
Epoch 29 TRAIN:  62%|██████▏   | 85/138 [00:20<00:10,  4.87it/s, epoch=29, grad_norm=181.409, loss=230.8780, lr=3.23e-04]
Epoch 29 TRAIN:  64%|██████▍   | 88/138 [00:20<00:10,  4.88it/s, epoch=29, grad_norm=181.409, loss=230.8780, lr=3.23e-04]
Epoch 29 TRAIN:  64%|██████▍   | 88/138 [00:20<00:10,  4.88it/s, epoch=29, grad_norm=245.859, loss=228.0942, lr=3.23e-04]
Epoch 29 TRAIN:  64%|██████▍   | 88/138 [00:20<00:10,  4.88it/s, epoch=29, grad_norm=232.217, loss=273.9493, lr=3.23e-04]
Epoch 29 TRAIN:  64%|██████▍   | 88/138 [00:20<00:10,  4.88it/s, epoch=29, grad_norm=72.606, loss=328.8451, lr=3.23e-04] 
Epoch 29 TRAIN:  66%|██████▌   | 91/138 [00:20<00:09,  4.95it/s, epoch=29, grad_norm=72.606, loss=328.8451, lr=3.23e-04]
Epoch 29 TRAIN:  66%|██████▌   | 91/138 [00:20<00:09,  4.95it/s, epoch=29, grad_norm=102.677, loss=254.7464, lr=3.23e-04]
Epoch 29 TRAIN:  66%|██████▌   | 91/138 [00:21<00:09,  4.95it/s, epoch=29, grad_norm=87.439, loss=279.0771, lr=3.24e-04] 
Epoch 29 TRAIN:  66%|██████▌   | 91/138 [00:21<00:09,  4.95it/s, epoch=29, grad_norm=246.799, loss=405.2930, lr=3.24e-04]
Epoch 29 TRAIN:  68%|██████▊   | 94/138 [00:21<00:09,  4.85it/s, epoch=29, grad_norm=246.799, loss=405.2930, lr=3.24e-04]
Epoch 29 TRAIN:  68%|██████▊   | 94/138 [00:21<00:09,  4.85it/s, epoch=29, grad_norm=150.437, loss=310.8878, lr=3.24e-04]
Epoch 29 TRAIN:  68%|██████▊   | 94/138 [00:21<00:09,  4.85it/s, epoch=29, grad_norm=127.007, loss=321.4579, lr=3.24e-04]
Epoch 29 TRAIN:  68%|██████▊   | 94/138 [00:21<00:09,  4.85it/s, epoch=29, grad_norm=120.288, loss=322.4147, lr=3.24e-04]
Epoch 29 TRAIN:  70%|███████   | 97/138 [00:21<00:08,  4.81it/s, epoch=29, grad_norm=120.288, loss=322.4147, lr=3.24e-04]
Epoch 29 TRAIN:  70%|███████   | 97/138 [00:22<00:08,  4.81it/s, epoch=29, grad_norm=224.299, loss=258.2348, lr=3.24e-04]
Epoch 29 TRAIN:  70%|███████   | 97/138 [00:22<00:08,  4.81it/s, epoch=29, grad_norm=120.323, loss=263.4297, lr=3.24e-04]
Epoch 29 TRAIN:  70%|███████   | 97/138 [00:22<00:08,  4.81it/s, epoch=29, grad_norm=162.944, loss=263.5898, lr=3.24e-04]2026-04-16 20:06:35,678 DEBUG TRAIN | steps/sec 4.492| Batch 29/100 loss 263.589783 loss_att 233.502136 loss_ctc 333.794250 th_accuracy 0.227139 lr 3.2408e-04 grad_norm 162.943542 rank 0

Epoch 29 TRAIN:  72%|███████▏  | 100/138 [00:22<00:07,  4.93it/s, epoch=29, grad_norm=162.944, loss=263.5898, lr=3.24e-04]
Epoch 29 TRAIN:  72%|███████▏  | 100/138 [00:22<00:07,  4.93it/s, epoch=29, grad_norm=108.557, loss=268.9538, lr=3.24e-04]
Epoch 29 TRAIN:  72%|███████▏  | 100/138 [00:22<00:07,  4.93it/s, epoch=29, grad_norm=142.335, loss=242.2825, lr=3.24e-04]
Epoch 29 TRAIN:  72%|███████▏  | 100/138 [00:23<00:07,  4.93it/s, epoch=29, grad_norm=187.747, loss=361.9697, lr=3.24e-04]
Epoch 29 TRAIN:  75%|███████▍  | 103/138 [00:23<00:06,  5.03it/s, epoch=29, grad_norm=187.747, loss=361.9697, lr=3.24e-04]
Epoch 29 TRAIN:  75%|███████▍  | 103/138 [00:23<00:06,  5.03it/s, epoch=29, grad_norm=190.634, loss=339.8671, lr=3.24e-04]
Epoch 29 TRAIN:  75%|███████▍  | 103/138 [00:23<00:06,  5.03it/s, epoch=29, grad_norm=100.414, loss=279.9016, lr=3.24e-04]
Epoch 29 TRAIN:  75%|███████▍  | 103/138 [00:23<00:06,  5.03it/s, epoch=29, grad_norm=72.835, loss=312.8588, lr=3.25e-04] 
Epoch 29 TRAIN:  77%|███████▋  | 106/138 [00:23<00:06,  4.76it/s, epoch=29, grad_norm=72.835, loss=312.8588, lr=3.25e-04]
Epoch 29 TRAIN:  77%|███████▋  | 106/138 [00:23<00:06,  4.76it/s, epoch=29, grad_norm=84.726, loss=268.6015, lr=3.25e-04]
Epoch 29 TRAIN:  77%|███████▋  | 106/138 [00:24<00:06,  4.76it/s, epoch=29, grad_norm=71.636, loss=264.2579, lr=3.25e-04]
Epoch 29 TRAIN:  77%|███████▋  | 106/138 [00:24<00:06,  4.76it/s, epoch=29, grad_norm=109.150, loss=299.8210, lr=3.25e-04]
Epoch 29 TRAIN:  79%|███████▉  | 109/138 [00:24<00:06,  4.53it/s, epoch=29, grad_norm=109.150, loss=299.8210, lr=3.25e-04]
Epoch 29 TRAIN:  79%|███████▉  | 109/138 [00:24<00:06,  4.53it/s, epoch=29, grad_norm=288.725, loss=391.6108, lr=3.25e-04]
Epoch 29 TRAIN:  79%|███████▉  | 109/138 [00:24<00:06,  4.53it/s, epoch=29, grad_norm=72.475, loss=249.6927, lr=3.25e-04] 
Epoch 29 TRAIN:  79%|███████▉  | 109/138 [00:25<00:06,  4.53it/s, epoch=29, grad_norm=166.930, loss=366.5995, lr=3.25e-04]
Epoch 29 TRAIN:  81%|████████  | 112/138 [00:25<00:05,  4.62it/s, epoch=29, grad_norm=166.930, loss=366.5995, lr=3.25e-04]
Epoch 29 TRAIN:  81%|████████  | 112/138 [00:25<00:05,  4.62it/s, epoch=29, grad_norm=99.321, loss=277.3088, lr=3.25e-04] 
Epoch 29 TRAIN:  81%|████████  | 112/138 [00:25<00:05,  4.62it/s, epoch=29, grad_norm=142.825, loss=405.7063, lr=3.25e-04]
Epoch 29 TRAIN:  81%|████████  | 112/138 [00:25<00:05,  4.62it/s, epoch=29, grad_norm=114.197, loss=290.4492, lr=3.25e-04]
Epoch 29 TRAIN:  83%|████████▎ | 115/138 [00:25<00:04,  4.75it/s, epoch=29, grad_norm=114.197, loss=290.4492, lr=3.25e-04]
Epoch 29 TRAIN:  83%|████████▎ | 115/138 [00:25<00:04,  4.75it/s, epoch=29, grad_norm=106.522, loss=305.2024, lr=3.25e-04]
Epoch 29 TRAIN:  83%|████████▎ | 115/138 [00:26<00:04,  4.75it/s, epoch=29, grad_norm=123.479, loss=357.7651, lr=3.25e-04]
Epoch 29 TRAIN:  83%|████████▎ | 115/138 [00:26<00:04,  4.75it/s, epoch=29, grad_norm=86.971, loss=297.1013, lr=3.26e-04] 
Epoch 29 TRAIN:  86%|████████▌ | 118/138 [00:26<00:04,  4.91it/s, epoch=29, grad_norm=86.971, loss=297.1013, lr=3.26e-04]
Epoch 29 TRAIN:  86%|████████▌ | 118/138 [00:26<00:04,  4.91it/s, epoch=29, grad_norm=113.810, loss=311.9086, lr=3.26e-04]
Epoch 29 TRAIN:  86%|████████▌ | 118/138 [00:26<00:04,  4.91it/s, epoch=29, grad_norm=93.367, loss=310.4784, lr=3.26e-04] 
Epoch 29 TRAIN:  86%|████████▌ | 118/138 [00:26<00:04,  4.91it/s, epoch=29, grad_norm=120.838, loss=345.2058, lr=3.26e-04]
Epoch 29 TRAIN:  88%|████████▊ | 121/138 [00:26<00:03,  5.06it/s, epoch=29, grad_norm=120.838, loss=345.2058, lr=3.26e-04]
Epoch 29 TRAIN:  88%|████████▊ | 121/138 [00:26<00:03,  5.06it/s, epoch=29, grad_norm=102.450, loss=353.6765, lr=3.26e-04]
Epoch 29 TRAIN:  88%|████████▊ | 121/138 [00:27<00:03,  5.06it/s, epoch=29, grad_norm=110.670, loss=344.7764, lr=3.26e-04]
Epoch 29 TRAIN:  88%|████████▊ | 121/138 [00:27<00:03,  5.06it/s, epoch=29, grad_norm=114.432, loss=347.1886, lr=3.26e-04]
Epoch 29 TRAIN:  90%|████████▉ | 124/138 [00:27<00:02,  5.05it/s, epoch=29, grad_norm=114.432, loss=347.1886, lr=3.26e-04]
Epoch 29 TRAIN:  90%|████████▉ | 124/138 [00:27<00:02,  5.05it/s, epoch=29, grad_norm=121.496, loss=400.4651, lr=3.26e-04]
Epoch 29 TRAIN:  90%|████████▉ | 124/138 [00:27<00:02,  5.05it/s, epoch=29, grad_norm=117.816, loss=302.8883, lr=3.26e-04]
Epoch 29 TRAIN:  90%|████████▉ | 124/138 [00:27<00:02,  5.05it/s, epoch=29, grad_norm=242.300, loss=260.4229, lr=3.26e-04]
Epoch 29 TRAIN:  92%|█████████▏| 127/138 [00:27<00:02,  4.99it/s, epoch=29, grad_norm=242.300, loss=260.4229, lr=3.26e-04]
Epoch 29 TRAIN:  92%|█████████▏| 127/138 [00:28<00:02,  4.99it/s, epoch=29, grad_norm=154.765, loss=319.3579, lr=3.26e-04]
Epoch 29 TRAIN:  92%|█████████▏| 127/138 [00:28<00:02,  4.99it/s, epoch=29, grad_norm=132.114, loss=364.4765, lr=3.26e-04]
Epoch 29 TRAIN:  92%|█████████▏| 127/138 [00:28<00:02,  4.99it/s, epoch=29, grad_norm=135.971, loss=354.3808, lr=3.26e-04]
Epoch 29 TRAIN:  94%|█████████▍| 130/138 [00:28<00:01,  5.00it/s, epoch=29, grad_norm=135.971, loss=354.3808, lr=3.26e-04]
Epoch 29 TRAIN:  94%|█████████▍| 130/138 [00:28<00:01,  5.00it/s, epoch=29, grad_norm=141.786, loss=384.7009, lr=3.27e-04]
Epoch 29 TRAIN:  94%|█████████▍| 130/138 [00:28<00:01,  5.00it/s, epoch=29, grad_norm=132.709, loss=331.0672, lr=3.27e-04]
Epoch 29 TRAIN:  94%|█████████▍| 130/138 [00:29<00:01,  5.00it/s, epoch=29, grad_norm=151.882, loss=328.9891, lr=3.27e-04]
Epoch 29 TRAIN:  94%|█████████▍| 130/138 [00:29<00:01,  5.00it/s, epoch=29, grad_norm=180.137, loss=429.3191, lr=3.27e-04]
Epoch 29 TRAIN:  97%|█████████▋| 134/138 [00:29<00:00,  5.10it/s, epoch=29, grad_norm=180.137, loss=429.3191, lr=3.27e-04]
Epoch 29 TRAIN:  97%|█████████▋| 134/138 [00:29<00:00,  5.10it/s, epoch=29, grad_norm=218.116, loss=312.8810, lr=3.27e-04]
Epoch 29 TRAIN:  98%|█████████▊| 135/138 [00:29<00:00,  4.58it/s, epoch=29, grad_norm=218.116, loss=312.8810, lr=3.27e-04]

CV:   0%|          | 0/6 [00:00<?, ?it/s]/home/rookie/Documents/batuka/projects/wenet/wenet/wenet/utils/train_utils.py:689: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  amp_autocast(enabled=dtype is not None,

CV:  17%|█▋        | 1/6 [00:00<00:03,  1.33it/s]
                                                 
2026-04-16 20:06:43,587 INFO Epoch 29 Step 4086 CV info lr 3.2688e-04 cv_loss 519.5730313387784 rank 0 acc 0.2461547628045082
2026-04-16 20:06:43,588 INFO [Rank 0] Checkpoint: save to checkpoint exp/conformer_mongolian/epoch_29.pt
2026-04-16 20:06:43,704 INFO Epoch 30 Step 4086 TRAIN info lr 3.2688e-04 rank 0
2026-04-16 20:06:43,719 INFO using accumulate grad, new batch size is 1 times larger than before

Epoch 30 TRAIN:   0%|          | 0/135 [00:00<?, ?it/s]
Epoch 30 TRAIN:   0%|          | 0/135 [00:04<?, ?it/s, epoch=30, grad_norm=438.682, loss=368.5752, lr=3.27e-04]
Epoch 30 TRAIN:   1%|          | 1/135 [00:04<09:04,  4.07s/it, epoch=30, grad_norm=438.682, loss=368.5752, lr=3.27e-04]
Epoch 30 TRAIN:   1%|          | 1/135 [00:04<09:04,  4.07s/it, epoch=30, grad_norm=353.510, loss=377.9483, lr=3.27e-04]
Epoch 30 TRAIN:   1%|          | 1/135 [00:04<09:04,  4.07s/it, epoch=30, grad_norm=362.609, loss=389.0361, lr=3.27e-04]
Epoch 30 TRAIN:   1%|          | 1/135 [00:04<09:04,  4.07s/it, epoch=30, grad_norm=495.317, loss=436.2486, lr=3.27e-04]
Epoch 30 TRAIN:   1%|          | 1/135 [00:04<09:04,  4.07s/it, epoch=30, grad_norm=103.362, loss=314.1779, lr=3.27e-04]
Epoch 30 TRAIN:   1%|          | 1/135 [00:04<09:04,  4.07s/it, epoch=30, grad_norm=319.217, loss=340.6089, lr=3.27e-04]
Epoch 30 TRAIN:   1%|          | 1/135 [00:04<09:04,  4.07s/it, epoch=30, grad_norm=150.604, loss=431.0438, lr=3.27e-04]
Epoch 30 TRAIN:   5%|▌         | 7/135 [00:04<01:05,  1.97it/s, epoch=30, grad_norm=150.604, loss=431.0438, lr=3.27e-04]
Epoch 30 TRAIN:   5%|▌         | 7/135 [00:04<01:05,  1.97it/s, epoch=30, grad_norm=79.122, loss=221.2911, lr=3.28e-04] 
Epoch 30 TRAIN:   5%|▌         | 7/135 [00:04<01:05,  1.97it/s, epoch=30, grad_norm=85.202, loss=290.7291, lr=3.28e-04]
Epoch 30 TRAIN:   5%|▌         | 7/135 [00:05<01:05,  1.97it/s, epoch=30, grad_norm=145.004, loss=355.1226, lr=3.28e-04]
Epoch 30 TRAIN:   5%|▌         | 7/135 [00:05<01:05,  1.97it/s, epoch=30, grad_norm=85.019, loss=317.5007, lr=3.28e-04] 
Epoch 30 TRAIN:   8%|▊         | 11/135 [00:05<00:41,  2.96it/s, epoch=30, grad_norm=85.019, loss=317.5007, lr=3.28e-04]
Epoch 30 TRAIN:   8%|▊         | 11/135 [00:05<00:41,  2.96it/s, epoch=30, grad_norm=156.472, loss=368.5396, lr=3.28e-04]
Epoch 30 TRAIN:   8%|▊         | 11/135 [00:05<00:41,  2.96it/s, epoch=30, grad_norm=62.606, loss=360.5656, lr=3.28e-04] 
Epoch 30 TRAIN:   8%|▊         | 11/135 [00:05<00:41,  2.96it/s, epoch=30, grad_norm=259.198, loss=314.5745, lr=3.28e-04]
Epoch 30 TRAIN:   8%|▊         | 11/135 [00:05<00:41,  2.96it/s, epoch=30, grad_norm=71.076, loss=355.2689, lr=3.28e-04] 
Epoch 30 TRAIN:  11%|█         | 15/135 [00:05<00:32,  3.71it/s, epoch=30, grad_norm=71.076, loss=355.2689, lr=3.28e-04]
Epoch 30 TRAIN:  11%|█         | 15/135 [00:06<00:32,  3.71it/s, epoch=30, grad_norm=85.178, loss=305.8956, lr=3.28e-04]
Epoch 30 TRAIN:  11%|█         | 15/135 [00:06<00:32,  3.71it/s, epoch=30, grad_norm=69.767, loss=312.5303, lr=3.28e-04]
Epoch 30 TRAIN:  11%|█         | 15/135 [00:06<00:32,  3.71it/s, epoch=30, grad_norm=96.932, loss=320.5473, lr=3.28e-04]
Epoch 30 TRAIN:  11%|█         | 15/135 [00:06<00:32,  3.71it/s, epoch=30, grad_norm=113.634, loss=319.2719, lr=3.28e-04]
Epoch 30 TRAIN:  14%|█▍        | 19/135 [00:06<00:26,  4.33it/s, epoch=30, grad_norm=113.634, loss=319.2719, lr=3.28e-04]
Epoch 30 TRAIN:  14%|█▍        | 19/135 [00:06<00:26,  4.33it/s, epoch=30, grad_norm=63.182, loss=306.2794, lr=3.28e-04] 
Epoch 30 TRAIN:  14%|█▍        | 19/135 [00:06<00:26,  4.33it/s, epoch=30, grad_norm=132.374, loss=328.5766, lr=3.29e-04]
Epoch 30 TRAIN:  14%|█▍        | 19/135 [00:07<00:26,  4.33it/s, epoch=30, grad_norm=72.099, loss=320.3978, lr=3.29e-04] 
Epoch 30 TRAIN:  14%|█▍        | 19/135 [00:07<00:26,  4.33it/s, epoch=30, grad_norm=98.321, loss=336.3676, lr=3.29e-04]
Epoch 30 TRAIN:  17%|█▋        | 23/135 [00:07<00:23,  4.70it/s, epoch=30, grad_norm=98.321, loss=336.3676, lr=3.29e-04]
Epoch 30 TRAIN:  17%|█▋        | 23/135 [00:07<00:23,  4.70it/s, epoch=30, grad_norm=78.785, loss=289.1776, lr=3.29e-04]
Epoch 30 TRAIN:  17%|█▋        | 23/135 [00:07<00:23,  4.70it/s, epoch=30, grad_norm=87.935, loss=325.6442, lr=3.29e-04]
Epoch 30 TRAIN:  17%|█▋        | 23/135 [00:07<00:23,  4.70it/s, epoch=30, grad_norm=95.783, loss=301.4057, lr=3.29e-04]
Epoch 30 TRAIN:  17%|█▋        | 23/135 [00:07<00:23,  4.70it/s, epoch=30, grad_norm=107.029, loss=306.2989, lr=3.29e-04]
Epoch 30 TRAIN:  20%|██        | 27/135 [00:07<00:21,  5.04it/s, epoch=30, grad_norm=107.029, loss=306.2989, lr=3.29e-04]
Epoch 30 TRAIN:  20%|██        | 27/135 [00:08<00:21,  5.04it/s, epoch=30, grad_norm=80.144, loss=283.0394, lr=3.29e-04] 
Epoch 30 TRAIN:  20%|██        | 27/135 [00:08<00:21,  5.04it/s, epoch=30, grad_norm=68.962, loss=275.4383, lr=3.29e-04]
Epoch 30 TRAIN:  20%|██        | 27/135 [00:08<00:21,  5.04it/s, epoch=30, grad_norm=198.869, loss=384.1078, lr=3.29e-04]
Epoch 30 TRAIN:  20%|██        | 27/135 [00:08<00:21,  5.04it/s, epoch=30, grad_norm=97.142, loss=299.3709, lr=3.29e-04] 
Epoch 30 TRAIN:  23%|██▎       | 31/135 [00:08<00:20,  5.19it/s, epoch=30, grad_norm=97.142, loss=299.3709, lr=3.29e-04]
Epoch 30 TRAIN:  23%|██▎       | 31/135 [00:08<00:20,  5.19it/s, epoch=30, grad_norm=103.447, loss=311.5462, lr=3.29e-04]
Epoch 30 TRAIN:  23%|██▎       | 31/135 [00:08<00:20,  5.19it/s, epoch=30, grad_norm=94.994, loss=316.8627, lr=3.30e-04] 
Epoch 30 TRAIN:  23%|██▎       | 31/135 [00:09<00:20,  5.19it/s, epoch=30, grad_norm=73.621, loss=284.8074, lr=3.30e-04]
Epoch 30 TRAIN:  25%|██▌       | 34/135 [00:09<00:18,  5.36it/s, epoch=30, grad_norm=73.621, loss=284.8074, lr=3.30e-04]
Epoch 30 TRAIN:  25%|██▌       | 34/135 [00:09<00:18,  5.36it/s, epoch=30, grad_norm=83.745, loss=305.6852, lr=3.30e-04]
Epoch 30 TRAIN:  25%|██▌       | 34/135 [00:09<00:18,  5.36it/s, epoch=30, grad_norm=98.159, loss=298.9922, lr=3.30e-04]
Epoch 30 TRAIN:  25%|██▌       | 34/135 [00:09<00:18,  5.36it/s, epoch=30, grad_norm=92.530, loss=349.7398, lr=3.30e-04]
Epoch 30 TRAIN:  27%|██▋       | 37/135 [00:09<00:18,  5.17it/s, epoch=30, grad_norm=92.530, loss=349.7398, lr=3.30e-04]
Epoch 30 TRAIN:  27%|██▋       | 37/135 [00:09<00:18,  5.17it/s, epoch=30, grad_norm=106.863, loss=382.6716, lr=3.30e-04]
Epoch 30 TRAIN:  27%|██▋       | 37/135 [00:10<00:18,  5.17it/s, epoch=30, grad_norm=118.758, loss=378.9300, lr=3.30e-04]
Epoch 30 TRAIN:  27%|██▋       | 37/135 [00:10<00:18,  5.17it/s, epoch=30, grad_norm=88.140, loss=324.8901, lr=3.30e-04] 
Epoch 30 TRAIN:  30%|██▉       | 40/135 [00:10<00:18,  5.17it/s, epoch=30, grad_norm=88.140, loss=324.8901, lr=3.30e-04]
Epoch 30 TRAIN:  30%|██▉       | 40/135 [00:10<00:18,  5.17it/s, epoch=30, grad_norm=107.495, loss=322.1483, lr=3.30e-04]
Epoch 30 TRAIN:  30%|██▉       | 40/135 [00:10<00:18,  5.17it/s, epoch=30, grad_norm=132.514, loss=305.6979, lr=3.30e-04]
Epoch 30 TRAIN:  30%|██▉       | 40/135 [00:10<00:18,  5.17it/s, epoch=30, grad_norm=218.101, loss=260.5605, lr=3.30e-04]
Epoch 30 TRAIN:  32%|███▏      | 43/135 [00:10<00:17,  5.15it/s, epoch=30, grad_norm=218.101, loss=260.5605, lr=3.30e-04]
Epoch 30 TRAIN:  32%|███▏      | 43/135 [00:11<00:17,  5.15it/s, epoch=30, grad_norm=73.244, loss=341.8903, lr=3.30e-04] 
Epoch 30 TRAIN:  32%|███▏      | 43/135 [00:11<00:17,  5.15it/s, epoch=30, grad_norm=90.448, loss=336.0983, lr=3.30e-04]
Epoch 30 TRAIN:  32%|███▏      | 43/135 [00:11<00:17,  5.15it/s, epoch=30, grad_norm=108.229, loss=357.3825, lr=3.31e-04]
Epoch 30 TRAIN:  34%|███▍      | 46/135 [00:11<00:18,  4.88it/s, epoch=30, grad_norm=108.229, loss=357.3825, lr=3.31e-04]
Epoch 30 TRAIN:  34%|███▍      | 46/135 [00:11<00:18,  4.88it/s, epoch=30, grad_norm=79.034, loss=309.8925, lr=3.31e-04] 
Epoch 30 TRAIN:  34%|███▍      | 46/135 [00:12<00:18,  4.88it/s, epoch=30, grad_norm=56.899, loss=310.8979, lr=3.31e-04]
Epoch 30 TRAIN:  34%|███▍      | 46/135 [00:12<00:18,  4.88it/s, epoch=30, grad_norm=125.077, loss=348.5017, lr=3.31e-04]
Epoch 30 TRAIN:  36%|███▋      | 49/135 [00:12<00:17,  4.94it/s, epoch=30, grad_norm=125.077, loss=348.5017, lr=3.31e-04]
Epoch 30 TRAIN:  36%|███▋      | 49/135 [00:12<00:17,  4.94it/s, epoch=30, grad_norm=96.302, loss=348.1196, lr=3.31e-04] 
Epoch 30 TRAIN:  36%|███▋      | 49/135 [00:12<00:17,  4.94it/s, epoch=30, grad_norm=116.674, loss=300.5734, lr=3.31e-04]
Epoch 30 TRAIN:  36%|███▋      | 49/135 [00:12<00:17,  4.94it/s, epoch=30, grad_norm=89.118, loss=354.5721, lr=3.31e-04] 
Epoch 30 TRAIN:  39%|███▊      | 52/135 [00:12<00:16,  4.92it/s, epoch=30, grad_norm=89.118, loss=354.5721, lr=3.31e-04]
Epoch 30 TRAIN:  39%|███▊      | 52/135 [00:13<00:16,  4.92it/s, epoch=30, grad_norm=82.480, loss=332.2558, lr=3.31e-04]
Epoch 30 TRAIN:  39%|███▊      | 52/135 [00:13<00:16,  4.92it/s, epoch=30, grad_norm=91.113, loss=305.4687, lr=3.31e-04]
Epoch 30 TRAIN:  39%|███▊      | 52/135 [00:13<00:16,  4.92it/s, epoch=30, grad_norm=119.321, loss=313.0789, lr=3.31e-04]
Epoch 30 TRAIN:  41%|████      | 55/135 [00:13<00:16,  4.93it/s, epoch=30, grad_norm=119.321, loss=313.0789, lr=3.31e-04]
Epoch 30 TRAIN:  41%|████      | 55/135 [00:13<00:16,  4.93it/s, epoch=30, grad_norm=93.860, loss=365.5966, lr=3.31e-04] 
Epoch 30 TRAIN:  41%|████      | 55/135 [00:13<00:16,  4.93it/s, epoch=30, grad_norm=77.828, loss=332.7589, lr=3.31e-04]
Epoch 30 TRAIN:  41%|████      | 55/135 [00:13<00:16,  4.93it/s, epoch=30, grad_norm=89.175, loss=308.9879, lr=3.32e-04]
Epoch 30 TRAIN:  43%|████▎     | 58/135 [00:13<00:15,  5.10it/s, epoch=30, grad_norm=89.175, loss=308.9879, lr=3.32e-04]
Epoch 30 TRAIN:  43%|████▎     | 58/135 [00:14<00:15,  5.10it/s, epoch=30, grad_norm=179.501, loss=299.1523, lr=3.32e-04]
Epoch 30 TRAIN:  43%|████▎     | 58/135 [00:14<00:15,  5.10it/s, epoch=30, grad_norm=59.338, loss=321.8799, lr=3.32e-04] 
Epoch 30 TRAIN:  43%|████▎     | 58/135 [00:14<00:15,  5.10it/s, epoch=30, grad_norm=76.220, loss=296.6744, lr=3.32e-04]
Epoch 30 TRAIN:  45%|████▌     | 61/135 [00:14<00:15,  4.93it/s, epoch=30, grad_norm=76.220, loss=296.6744, lr=3.32e-04]
Epoch 30 TRAIN:  45%|████▌     | 61/135 [00:14<00:15,  4.93it/s, epoch=30, grad_norm=100.767, loss=273.0917, lr=3.32e-04]
Epoch 30 TRAIN:  45%|████▌     | 61/135 [00:15<00:15,  4.93it/s, epoch=30, grad_norm=110.201, loss=332.8472, lr=3.32e-04]
Epoch 30 TRAIN:  45%|████▌     | 61/135 [00:15<00:15,  4.93it/s, epoch=30, grad_norm=84.895, loss=293.2773, lr=3.32e-04] 
Epoch 30 TRAIN:  47%|████▋     | 64/135 [00:15<00:14,  4.92it/s, epoch=30, grad_norm=84.895, loss=293.2773, lr=3.32e-04]
Epoch 30 TRAIN:  47%|████▋     | 64/135 [00:15<00:14,  4.92it/s, epoch=30, grad_norm=72.848, loss=285.9204, lr=3.32e-04]
Epoch 30 TRAIN:  47%|████▋     | 64/135 [00:15<00:14,  4.92it/s, epoch=30, grad_norm=85.956, loss=314.8527, lr=3.32e-04]
Epoch 30 TRAIN:  47%|████▋     | 64/135 [00:15<00:14,  4.92it/s, epoch=30, grad_norm=143.781, loss=320.2340, lr=3.32e-04]
Epoch 30 TRAIN:  50%|████▉     | 67/135 [00:15<00:13,  5.10it/s, epoch=30, grad_norm=143.781, loss=320.2340, lr=3.32e-04]
Epoch 30 TRAIN:  50%|████▉     | 67/135 [00:15<00:13,  5.10it/s, epoch=30, grad_norm=110.568, loss=309.4468, lr=3.32e-04]
Epoch 30 TRAIN:  50%|████▉     | 67/135 [00:16<00:13,  5.10it/s, epoch=30, grad_norm=92.991, loss=304.1050, lr=3.32e-04] 
Epoch 30 TRAIN:  50%|████▉     | 67/135 [00:16<00:13,  5.10it/s, epoch=30, grad_norm=79.627, loss=335.1394, lr=3.32e-04]
Epoch 30 TRAIN:  52%|█████▏    | 70/135 [00:16<00:13,  4.99it/s, epoch=30, grad_norm=79.627, loss=335.1394, lr=3.32e-04]
Epoch 30 TRAIN:  52%|█████▏    | 70/135 [00:16<00:13,  4.99it/s, epoch=30, grad_norm=77.975, loss=351.2350, lr=3.33e-04]
Epoch 30 TRAIN:  52%|█████▏    | 70/135 [00:16<00:13,  4.99it/s, epoch=30, grad_norm=77.304, loss=324.3083, lr=3.33e-04]
Epoch 30 TRAIN:  52%|█████▏    | 70/135 [00:17<00:13,  4.99it/s, epoch=30, grad_norm=152.353, loss=394.3239, lr=3.33e-04]
Epoch 30 TRAIN:  54%|█████▍    | 73/135 [00:17<00:12,  4.93it/s, epoch=30, grad_norm=152.353, loss=394.3239, lr=3.33e-04]
Epoch 30 TRAIN:  54%|█████▍    | 73/135 [00:17<00:12,  4.93it/s, epoch=30, grad_norm=99.844, loss=365.9781, lr=3.33e-04] 
Epoch 30 TRAIN:  54%|█████▍    | 73/135 [00:17<00:12,  4.93it/s, epoch=30, grad_norm=251.633, loss=348.4900, lr=3.33e-04]
Epoch 30 TRAIN:  54%|█████▍    | 73/135 [00:17<00:12,  4.93it/s, epoch=30, grad_norm=107.002, loss=321.5082, lr=3.33e-04]
Epoch 30 TRAIN:  56%|█████▋    | 76/135 [00:17<00:12,  4.88it/s, epoch=30, grad_norm=107.002, loss=321.5082, lr=3.33e-04]
Epoch 30 TRAIN:  56%|█████▋    | 76/135 [00:17<00:12,  4.88it/s, epoch=30, grad_norm=136.544, loss=296.0407, lr=3.33e-04]
Epoch 30 TRAIN:  56%|█████▋    | 76/135 [00:18<00:12,  4.88it/s, epoch=30, grad_norm=79.241, loss=344.8298, lr=3.33e-04] 
Epoch 30 TRAIN:  56%|█████▋    | 76/135 [00:18<00:12,  4.88it/s, epoch=30, grad_norm=96.972, loss=348.8311, lr=3.33e-04]
Epoch 30 TRAIN:  59%|█████▊    | 79/135 [00:18<00:11,  4.89it/s, epoch=30, grad_norm=96.972, loss=348.8311, lr=3.33e-04]
Epoch 30 TRAIN:  59%|█████▊    | 79/135 [00:18<00:11,  4.89it/s, epoch=30, grad_norm=75.969, loss=282.2473, lr=3.33e-04]
Epoch 30 TRAIN:  59%|█████▊    | 79/135 [00:18<00:11,  4.89it/s, epoch=30, grad_norm=149.049, loss=329.1118, lr=3.33e-04]
Epoch 30 TRAIN:  59%|█████▊    | 79/135 [00:18<00:11,  4.89it/s, epoch=30, grad_norm=117.023, loss=286.6521, lr=3.33e-04]
Epoch 30 TRAIN:  61%|██████    | 82/135 [00:18<00:11,  4.78it/s, epoch=30, grad_norm=117.023, loss=286.6521, lr=3.33e-04]
Epoch 30 TRAIN:  61%|██████    | 82/135 [00:19<00:11,  4.78it/s, epoch=30, grad_norm=168.310, loss=276.2984, lr=3.34e-04]
Epoch 30 TRAIN:  61%|██████    | 82/135 [00:19<00:11,  4.78it/s, epoch=30, grad_norm=111.969, loss=293.5697, lr=3.34e-04]
Epoch 30 TRAIN:  61%|██████    | 82/135 [00:19<00:11,  4.78it/s, epoch=30, grad_norm=127.474, loss=323.5485, lr=3.34e-04]
Epoch 30 TRAIN:  63%|██████▎   | 85/135 [00:19<00:10,  4.84it/s, epoch=30, grad_norm=127.474, loss=323.5485, lr=3.34e-04]
Epoch 30 TRAIN:  63%|██████▎   | 85/135 [00:19<00:10,  4.84it/s, epoch=30, grad_norm=78.037, loss=259.1227, lr=3.34e-04] 
Epoch 30 TRAIN:  63%|██████▎   | 85/135 [00:20<00:10,  4.84it/s, epoch=30, grad_norm=117.479, loss=281.8649, lr=3.34e-04]
Epoch 30 TRAIN:  63%|██████▎   | 85/135 [00:20<00:10,  4.84it/s, epoch=30, grad_norm=94.053, loss=274.4579, lr=3.34e-04] 
Epoch 30 TRAIN:  65%|██████▌   | 88/135 [00:20<00:10,  4.68it/s, epoch=30, grad_norm=94.053, loss=274.4579, lr=3.34e-04]
Epoch 30 TRAIN:  65%|██████▌   | 88/135 [00:20<00:10,  4.68it/s, epoch=30, grad_norm=152.030, loss=312.0636, lr=3.34e-04]
Epoch 30 TRAIN:  65%|██████▌   | 88/135 [00:20<00:10,  4.68it/s, epoch=30, grad_norm=80.621, loss=308.4182, lr=3.34e-04] 
Epoch 30 TRAIN:  65%|██████▌   | 88/135 [00:20<00:10,  4.68it/s, epoch=30, grad_norm=153.885, loss=325.0810, lr=3.34e-04]
Epoch 30 TRAIN:  67%|██████▋   | 91/135 [00:20<00:09,  4.70it/s, epoch=30, grad_norm=153.885, loss=325.0810, lr=3.34e-04]
Epoch 30 TRAIN:  67%|██████▋   | 91/135 [00:21<00:09,  4.70it/s, epoch=30, grad_norm=162.756, loss=349.2544, lr=3.34e-04]
Epoch 30 TRAIN:  67%|██████▋   | 91/135 [00:21<00:09,  4.70it/s, epoch=30, grad_norm=165.442, loss=398.5304, lr=3.34e-04]
Epoch 30 TRAIN:  67%|██████▋   | 91/135 [00:21<00:09,  4.70it/s, epoch=30, grad_norm=95.451, loss=306.3953, lr=3.34e-04] 
Epoch 30 TRAIN:  70%|██████▉   | 94/135 [00:21<00:08,  4.88it/s, epoch=30, grad_norm=95.451, loss=306.3953, lr=3.34e-04]
Epoch 30 TRAIN:  70%|██████▉   | 94/135 [00:21<00:08,  4.88it/s, epoch=30, grad_norm=290.492, loss=223.6234, lr=3.34e-04]
Epoch 30 TRAIN:  70%|██████▉   | 94/135 [00:21<00:08,  4.88it/s, epoch=30, grad_norm=80.790, loss=331.4108, lr=3.35e-04] 
Epoch 30 TRAIN:  70%|██████▉   | 94/135 [00:21<00:08,  4.88it/s, epoch=30, grad_norm=90.206, loss=276.2361, lr=3.35e-04]
Epoch 30 TRAIN:  72%|███████▏  | 97/135 [00:21<00:07,  4.95it/s, epoch=30, grad_norm=90.206, loss=276.2361, lr=3.35e-04]
Epoch 30 TRAIN:  72%|███████▏  | 97/135 [00:22<00:07,  4.95it/s, epoch=30, grad_norm=231.294, loss=229.4202, lr=3.35e-04]
Epoch 30 TRAIN:  72%|███████▏  | 97/135 [00:22<00:07,  4.95it/s, epoch=30, grad_norm=219.401, loss=255.1644, lr=3.35e-04]
Epoch 30 TRAIN:  72%|███████▏  | 97/135 [00:22<00:07,  4.95it/s, epoch=30, grad_norm=104.908, loss=287.6404, lr=3.35e-04]2026-04-16 20:07:06,292 DEBUG TRAIN | steps/sec 4.410| Batch 30/100 loss 287.640442 loss_att 253.623337 loss_ctc 367.013733 th_accuracy 0.318436 lr 3.3488e-04 grad_norm 104.908226 rank 0

Epoch 30 TRAIN:  74%|███████▍  | 100/135 [00:22<00:06,  5.01it/s, epoch=30, grad_norm=104.908, loss=287.6404, lr=3.35e-04]
Epoch 30 TRAIN:  74%|███████▍  | 100/135 [00:22<00:06,  5.01it/s, epoch=30, grad_norm=189.469, loss=340.1979, lr=3.35e-04]
Epoch 30 TRAIN:  74%|███████▍  | 100/135 [00:22<00:06,  5.01it/s, epoch=30, grad_norm=103.918, loss=281.3575, lr=3.35e-04]
Epoch 30 TRAIN:  74%|███████▍  | 100/135 [00:23<00:06,  5.01it/s, epoch=30, grad_norm=88.609, loss=282.9130, lr=3.35e-04] 
Epoch 30 TRAIN:  76%|███████▋  | 103/135 [00:23<00:06,  4.87it/s, epoch=30, grad_norm=88.609, loss=282.9130, lr=3.35e-04]
Epoch 30 TRAIN:  76%|███████▋  | 103/135 [00:23<00:06,  4.87it/s, epoch=30, grad_norm=224.632, loss=352.9955, lr=3.35e-04]
Epoch 30 TRAIN:  76%|███████▋  | 103/135 [00:23<00:06,  4.87it/s, epoch=30, grad_norm=82.730, loss=216.5724, lr=3.35e-04] 
Epoch 30 TRAIN:  76%|███████▋  | 103/135 [00:23<00:06,  4.87it/s, epoch=30, grad_norm=83.768, loss=229.6642, lr=3.35e-04]
Epoch 30 TRAIN:  79%|███████▊  | 106/135 [00:23<00:05,  4.96it/s, epoch=30, grad_norm=83.768, loss=229.6642, lr=3.35e-04]
Epoch 30 TRAIN:  79%|███████▊  | 106/135 [00:23<00:05,  4.96it/s, epoch=30, grad_norm=98.912, loss=340.7527, lr=3.35e-04]
Epoch 30 TRAIN:  79%|███████▊  | 106/135 [00:24<00:05,  4.96it/s, epoch=30, grad_norm=165.244, loss=262.2574, lr=3.36e-04]
Epoch 30 TRAIN:  79%|███████▊  | 106/135 [00:24<00:05,  4.96it/s, epoch=30, grad_norm=173.565, loss=347.1508, lr=3.36e-04]
Epoch 30 TRAIN:  81%|████████  | 109/135 [00:24<00:05,  4.90it/s, epoch=30, grad_norm=173.565, loss=347.1508, lr=3.36e-04]
Epoch 30 TRAIN:  81%|████████  | 109/135 [00:24<00:05,  4.90it/s, epoch=30, grad_norm=155.470, loss=324.3784, lr=3.36e-04]
Epoch 30 TRAIN:  81%|████████  | 109/135 [00:24<00:05,  4.90it/s, epoch=30, grad_norm=110.112, loss=296.0555, lr=3.36e-04]
Epoch 30 TRAIN:  81%|████████  | 109/135 [00:25<00:05,  4.90it/s, epoch=30, grad_norm=99.353, loss=272.2746, lr=3.36e-04] 
Epoch 30 TRAIN:  83%|████████▎ | 112/135 [00:25<00:04,  4.89it/s, epoch=30, grad_norm=99.353, loss=272.2746, lr=3.36e-04]
Epoch 30 TRAIN:  83%|████████▎ | 112/135 [00:25<00:04,  4.89it/s, epoch=30, grad_norm=83.880, loss=273.3748, lr=3.36e-04]
Epoch 30 TRAIN:  83%|████████▎ | 112/135 [00:25<00:04,  4.89it/s, epoch=30, grad_norm=134.984, loss=318.9915, lr=3.36e-04]
Epoch 30 TRAIN:  83%|████████▎ | 112/135 [00:25<00:04,  4.89it/s, epoch=30, grad_norm=124.863, loss=355.1945, lr=3.36e-04]
Epoch 30 TRAIN:  85%|████████▌ | 115/135 [00:25<00:04,  4.71it/s, epoch=30, grad_norm=124.863, loss=355.1945, lr=3.36e-04]
Epoch 30 TRAIN:  85%|████████▌ | 115/135 [00:25<00:04,  4.71it/s, epoch=30, grad_norm=204.086, loss=332.6907, lr=3.36e-04]
Epoch 30 TRAIN:  85%|████████▌ | 115/135 [00:26<00:04,  4.71it/s, epoch=30, grad_norm=97.137, loss=262.1484, lr=3.36e-04] 
Epoch 30 TRAIN:  85%|████████▌ | 115/135 [00:26<00:04,  4.71it/s, epoch=30, grad_norm=97.414, loss=282.6295, lr=3.36e-04]
Epoch 30 TRAIN:  87%|████████▋ | 118/135 [00:26<00:03,  4.77it/s, epoch=30, grad_norm=97.414, loss=282.6295, lr=3.36e-04]
Epoch 30 TRAIN:  87%|████████▋ | 118/135 [00:26<00:03,  4.77it/s, epoch=30, grad_norm=110.584, loss=330.8014, lr=3.36e-04]
Epoch 30 TRAIN:  87%|████████▋ | 118/135 [00:26<00:03,  4.77it/s, epoch=30, grad_norm=89.524, loss=296.5132, lr=3.36e-04] 
Epoch 30 TRAIN:  87%|████████▋ | 118/135 [00:27<00:03,  4.77it/s, epoch=30, grad_norm=151.704, loss=372.5616, lr=3.37e-04]
Epoch 30 TRAIN:  90%|████████▉ | 121/135 [00:27<00:03,  4.60it/s, epoch=30, grad_norm=151.704, loss=372.5616, lr=3.37e-04]
Epoch 30 TRAIN:  90%|████████▉ | 121/135 [00:27<00:03,  4.60it/s, epoch=30, grad_norm=131.717, loss=333.4777, lr=3.37e-04]
Epoch 30 TRAIN:  90%|████████▉ | 121/135 [00:27<00:03,  4.60it/s, epoch=30, grad_norm=134.892, loss=362.1469, lr=3.37e-04]
Epoch 30 TRAIN:  90%|████████▉ | 121/135 [00:27<00:03,  4.60it/s, epoch=30, grad_norm=114.183, loss=349.4762, lr=3.37e-04]
Epoch 30 TRAIN:  92%|█████████▏| 124/135 [00:27<00:02,  4.65it/s, epoch=30, grad_norm=114.183, loss=349.4762, lr=3.37e-04]
Epoch 30 TRAIN:  92%|█████████▏| 124/135 [00:27<00:02,  4.65it/s, epoch=30, grad_norm=197.130, loss=428.4121, lr=3.37e-04]
Epoch 30 TRAIN:  92%|█████████▏| 124/135 [00:28<00:02,  4.65it/s, epoch=30, grad_norm=206.556, loss=318.2690, lr=3.37e-04]
Epoch 30 TRAIN:  92%|█████████▏| 124/135 [00:28<00:02,  4.65it/s, epoch=30, grad_norm=194.562, loss=225.8814, lr=3.37e-04]
Epoch 30 TRAIN:  92%|█████████▏| 124/135 [00:28<00:02,  4.65it/s, epoch=30, grad_norm=136.884, loss=344.8323, lr=3.37e-04]
Epoch 30 TRAIN:  95%|█████████▍| 128/135 [00:28<00:01,  5.05it/s, epoch=30, grad_norm=136.884, loss=344.8323, lr=3.37e-04]
Epoch 30 TRAIN:  95%|█████████▍| 128/135 [00:28<00:01,  5.05it/s, epoch=30, grad_norm=109.109, loss=354.9993, lr=3.37e-04]
Epoch 30 TRAIN:  95%|█████████▍| 128/135 [00:28<00:01,  5.05it/s, epoch=30, grad_norm=283.039, loss=401.7040, lr=3.37e-04]
Epoch 30 TRAIN:  95%|█████████▍| 128/135 [00:29<00:01,  5.05it/s, epoch=30, grad_norm=222.372, loss=294.8844, lr=3.37e-04]
Epoch 30 TRAIN:  97%|█████████▋| 131/135 [00:29<00:00,  4.90it/s, epoch=30, grad_norm=222.372, loss=294.8844, lr=3.37e-04]
Epoch 30 TRAIN:  97%|█████████▋| 131/135 [00:29<00:00,  4.90it/s, epoch=30, grad_norm=135.072, loss=350.4569, lr=3.37e-04]
Epoch 30 TRAIN:  97%|█████████▋| 131/135 [00:29<00:00,  4.90it/s, epoch=30, grad_norm=146.897, loss=304.8214, lr=3.38e-04]
Epoch 30 TRAIN:  97%|█████████▋| 131/135 [00:29<00:00,  4.90it/s, epoch=30, grad_norm=217.683, loss=385.3433, lr=3.38e-04]
Epoch 30 TRAIN:  97%|█████████▋| 131/135 [00:29<00:00,  4.90it/s, epoch=30, grad_norm=135.297, loss=313.3822, lr=3.38e-04]
Epoch 30 TRAIN: 100%|██████████| 135/135 [00:29<00:00,  5.25it/s, epoch=30, grad_norm=135.297, loss=313.3822, lr=3.38e-04]
Epoch 30 TRAIN: 100%|██████████| 135/135 [00:29<00:00,  5.25it/s, epoch=30, grad_norm=155.406, loss=371.2219, lr=3.38e-04]
Epoch 30 TRAIN: 100%|██████████| 135/135 [00:30<00:00,  5.25it/s, epoch=30, grad_norm=231.846, loss=418.7064, lr=3.38e-04]
Epoch 30 TRAIN: 137it [00:30,  4.56it/s, epoch=30, grad_norm=231.846, loss=418.7064, lr=3.38e-04]                         

CV:   0%|          | 0/6 [00:00<?, ?it/s]/home/rookie/Documents/batuka/projects/wenet/wenet/wenet/utils/train_utils.py:689: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  amp_autocast(enabled=dtype is not None,

CV:  17%|█▋        | 1/6 [00:00<00:03,  1.33it/s]
                                                 
2026-04-16 20:07:14,588 INFO Epoch 30 Step 4223 CV info lr 3.3784e-04 cv_loss 516.5904651988636 rank 0 acc 0.25985973328351974
2026-04-16 20:07:14,589 INFO [Rank 0] Checkpoint: save to checkpoint exp/conformer_mongolian/epoch_30.pt
2026-04-16 20:07:14,710 INFO Epoch 31 Step 4223 TRAIN info lr 3.3784e-04 rank 0
2026-04-16 20:07:14,730 INFO using accumulate grad, new batch size is 1 times larger than before

Epoch 31 TRAIN:   0%|          | 0/137 [00:00<?, ?it/s]
Epoch 31 TRAIN:   0%|          | 0/137 [00:03<?, ?it/s, epoch=31, grad_norm=345.393, loss=373.9190, lr=3.38e-04]
Epoch 31 TRAIN:   1%|          | 1/137 [00:03<09:01,  3.98s/it, epoch=31, grad_norm=345.393, loss=373.9190, lr=3.38e-04]
Epoch 31 TRAIN:   1%|          | 1/137 [00:04<09:01,  3.98s/it, epoch=31, grad_norm=283.227, loss=285.3129, lr=3.38e-04]
Epoch 31 TRAIN:   1%|          | 1/137 [00:04<09:01,  3.98s/it, epoch=31, grad_norm=290.874, loss=355.0782, lr=3.38e-04]
Epoch 31 TRAIN:   1%|          | 1/137 [00:04<09:01,  3.98s/it, epoch=31, grad_norm=360.281, loss=409.6510, lr=3.38e-04]
Epoch 31 TRAIN:   1%|          | 1/137 [00:04<09:01,  3.98s/it, epoch=31, grad_norm=174.045, loss=316.7663, lr=3.38e-04]
Epoch 31 TRAIN:   1%|          | 1/137 [00:04<09:01,  3.98s/it, epoch=31, grad_norm=327.161, loss=451.1398, lr=3.38e-04]
Epoch 31 TRAIN:   1%|          | 1/137 [00:04<09:01,  3.98s/it, epoch=31, grad_norm=299.332, loss=420.5645, lr=3.38e-04]
Epoch 31 TRAIN:   1%|          | 1/137 [00:04<09:01,  3.98s/it, epoch=31, grad_norm=511.891, loss=296.8994, lr=3.38e-04]
Epoch 31 TRAIN:   6%|▌         | 8/137 [00:04<00:55,  2.31it/s, epoch=31, grad_norm=511.891, loss=296.8994, lr=3.38e-04]
Epoch 31 TRAIN:   6%|▌         | 8/137 [00:04<00:55,  2.31it/s, epoch=31, grad_norm=91.264, loss=311.6447, lr=3.39e-04] 
Epoch 31 TRAIN:   6%|▌         | 8/137 [00:04<00:55,  2.31it/s, epoch=31, grad_norm=211.835, loss=278.3371, lr=3.39e-04]
Epoch 31 TRAIN:   6%|▌         | 8/137 [00:05<00:55,  2.31it/s, epoch=31, grad_norm=196.473, loss=295.4276, lr=3.39e-04]
Epoch 31 TRAIN:   6%|▌         | 8/137 [00:05<00:55,  2.31it/s, epoch=31, grad_norm=107.137, loss=330.4883, lr=3.39e-04]
Epoch 31 TRAIN:   9%|▉         | 12/137 [00:05<00:39,  3.14it/s, epoch=31, grad_norm=107.137, loss=330.4883, lr=3.39e-04]
Epoch 31 TRAIN:   9%|▉         | 12/137 [00:05<00:39,  3.14it/s, epoch=31, grad_norm=117.497, loss=302.1468, lr=3.39e-04]
Epoch 31 TRAIN:   9%|▉         | 12/137 [00:05<00:39,  3.14it/s, epoch=31, grad_norm=82.784, loss=316.2255, lr=3.39e-04] 
Epoch 31 TRAIN:   9%|▉         | 12/137 [00:05<00:39,  3.14it/s, epoch=31, grad_norm=136.949, loss=354.8155, lr=3.39e-04]
Epoch 31 TRAIN:   9%|▉         | 12/137 [00:05<00:39,  3.14it/s, epoch=31, grad_norm=97.510, loss=341.1850, lr=3.39e-04] 
Epoch 31 TRAIN:  12%|█▏        | 16/137 [00:05<00:31,  3.84it/s, epoch=31, grad_norm=97.510, loss=341.1850, lr=3.39e-04]
Epoch 31 TRAIN:  12%|█▏        | 16/137 [00:06<00:31,  3.84it/s, epoch=31, grad_norm=73.920, loss=322.3929, lr=3.39e-04]
Epoch 31 TRAIN:  12%|█▏        | 16/137 [00:06<00:31,  3.84it/s, epoch=31, grad_norm=133.510, loss=276.5889, lr=3.39e-04]
Epoch 31 TRAIN:  12%|█▏        | 16/137 [00:06<00:31,  3.84it/s, epoch=31, grad_norm=104.097, loss=381.0339, lr=3.39e-04]
Epoch 31 TRAIN:  12%|█▏        | 16/137 [00:06<00:31,  3.84it/s, epoch=31, grad_norm=94.336, loss=322.6237, lr=3.39e-04] 
Epoch 31 TRAIN:  15%|█▍        | 20/137 [00:06<00:27,  4.25it/s, epoch=31, grad_norm=94.336, loss=322.6237, lr=3.39e-04]
Epoch 31 TRAIN:  15%|█▍        | 20/137 [00:06<00:27,  4.25it/s, epoch=31, grad_norm=76.495, loss=355.5464, lr=3.40e-04]
Epoch 31 TRAIN:  15%|█▍        | 20/137 [00:06<00:27,  4.25it/s, epoch=31, grad_norm=79.219, loss=343.8215, lr=3.40e-04]
Epoch 31 TRAIN:  15%|█▍        | 20/137 [00:07<00:27,  4.25it/s, epoch=31, grad_norm=73.065, loss=345.8618, lr=3.40e-04]
Epoch 31 TRAIN:  15%|█▍        | 20/137 [00:07<00:27,  4.25it/s, epoch=31, grad_norm=67.416, loss=329.5164, lr=3.40e-04]
Epoch 31 TRAIN:  18%|█▊        | 24/137 [00:07<00:25,  4.43it/s, epoch=31, grad_norm=67.416, loss=329.5164, lr=3.40e-04]
Epoch 31 TRAIN:  18%|█▊        | 24/137 [00:07<00:25,  4.43it/s, epoch=31, grad_norm=78.445, loss=325.0026, lr=3.40e-04]
Epoch 31 TRAIN:  18%|█▊        | 24/137 [00:07<00:25,  4.43it/s, epoch=31, grad_norm=69.386, loss=322.7372, lr=3.40e-04]
Epoch 31 TRAIN:  18%|█▊        | 24/137 [00:08<00:25,  4.43it/s, epoch=31, grad_norm=67.601, loss=291.3505, lr=3.40e-04]
Epoch 31 TRAIN:  20%|█▉        | 27/137 [00:08<00:23,  4.61it/s, epoch=31, grad_norm=67.601, loss=291.3505, lr=3.40e-04]
Epoch 31 TRAIN:  20%|█▉        | 27/137 [00:08<00:23,  4.61it/s, epoch=31, grad_norm=65.487, loss=301.9025, lr=3.40e-04]
Epoch 31 TRAIN:  20%|█▉        | 27/137 [00:08<00:23,  4.61it/s, epoch=31, grad_norm=138.767, loss=334.1044, lr=3.40e-04]
Epoch 31 TRAIN:  20%|█▉        | 27/137 [00:08<00:23,  4.61it/s, epoch=31, grad_norm=110.574, loss=342.6504, lr=3.40e-04]
Epoch 31 TRAIN:  22%|██▏       | 30/137 [00:08<00:22,  4.80it/s, epoch=31, grad_norm=110.574, loss=342.6504, lr=3.40e-04]
Epoch 31 TRAIN:  22%|██▏       | 30/137 [00:08<00:22,  4.80it/s, epoch=31, grad_norm=64.566, loss=280.7021, lr=3.40e-04] 
Epoch 31 TRAIN:  22%|██▏       | 30/137 [00:08<00:22,  4.80it/s, epoch=31, grad_norm=91.207, loss=242.9919, lr=3.40e-04]
Epoch 31 TRAIN:  22%|██▏       | 30/137 [00:09<00:22,  4.80it/s, epoch=31, grad_norm=168.878, loss=364.0704, lr=3.40e-04]
Epoch 31 TRAIN:  24%|██▍       | 33/137 [00:09<00:21,  4.84it/s, epoch=31, grad_norm=168.878, loss=364.0704, lr=3.40e-04]
Epoch 31 TRAIN:  24%|██▍       | 33/137 [00:09<00:21,  4.84it/s, epoch=31, grad_norm=86.817, loss=324.4433, lr=3.41e-04] 
Epoch 31 TRAIN:  24%|██▍       | 33/137 [00:09<00:21,  4.84it/s, epoch=31, grad_norm=69.656, loss=306.7552, lr=3.41e-04]
Epoch 31 TRAIN:  24%|██▍       | 33/137 [00:09<00:21,  4.84it/s, epoch=31, grad_norm=68.331, loss=283.4384, lr=3.41e-04]
Epoch 31 TRAIN:  26%|██▋       | 36/137 [00:09<00:20,  4.85it/s, epoch=31, grad_norm=68.331, loss=283.4384, lr=3.41e-04]
Epoch 31 TRAIN:  26%|██▋       | 36/137 [00:09<00:20,  4.85it/s, epoch=31, grad_norm=75.308, loss=307.7594, lr=3.41e-04]
Epoch 31 TRAIN:  26%|██▋       | 36/137 [00:10<00:20,  4.85it/s, epoch=31, grad_norm=91.805, loss=358.5242, lr=3.41e-04]
Epoch 31 TRAIN:  26%|██▋       | 36/137 [00:10<00:20,  4.85it/s, epoch=31, grad_norm=71.373, loss=321.6150, lr=3.41e-04]
Epoch 31 TRAIN:  28%|██▊       | 39/137 [00:10<00:20,  4.87it/s, epoch=31, grad_norm=71.373, loss=321.6150, lr=3.41e-04]
Epoch 31 TRAIN:  28%|██▊       | 39/137 [00:10<00:20,  4.87it/s, epoch=31, grad_norm=66.109, loss=336.4403, lr=3.41e-04]
Epoch 31 TRAIN:  28%|██▊       | 39/137 [00:10<00:20,  4.87it/s, epoch=31, grad_norm=83.872, loss=326.0058, lr=3.41e-04]
Epoch 31 TRAIN:  28%|██▊       | 39/137 [00:11<00:20,  4.87it/s, epoch=31, grad_norm=149.436, loss=288.1019, lr=3.41e-04]
Epoch 31 TRAIN:  31%|███       | 42/137 [00:11<00:19,  4.84it/s, epoch=31, grad_norm=149.436, loss=288.1019, lr=3.41e-04]
Epoch 31 TRAIN:  31%|███       | 42/137 [00:11<00:19,  4.84it/s, epoch=31, grad_norm=85.080, loss=360.9221, lr=3.41e-04] 
Epoch 31 TRAIN:  31%|███       | 42/137 [00:11<00:19,  4.84it/s, epoch=31, grad_norm=78.788, loss=323.9363, lr=3.41e-04]
Epoch 31 TRAIN:  31%|███       | 42/137 [00:11<00:19,  4.84it/s, epoch=31, grad_norm=83.889, loss=326.9886, lr=3.41e-04]
Epoch 31 TRAIN:  33%|███▎      | 45/137 [00:11<00:18,  4.88it/s, epoch=31, grad_norm=83.889, loss=326.9886, lr=3.41e-04]
Epoch 31 TRAIN:  33%|███▎      | 45/137 [00:11<00:18,  4.88it/s, epoch=31, grad_norm=79.849, loss=347.2039, lr=3.42e-04]
Epoch 31 TRAIN:  33%|███▎      | 45/137 [00:12<00:18,  4.88it/s, epoch=31, grad_norm=114.164, loss=279.3082, lr=3.42e-04]
Epoch 31 TRAIN:  33%|███▎      | 45/137 [00:12<00:18,  4.88it/s, epoch=31, grad_norm=95.431, loss=298.5118, lr=3.42e-04] 
Epoch 31 TRAIN:  35%|███▌      | 48/137 [00:12<00:18,  4.90it/s, epoch=31, grad_norm=95.431, loss=298.5118, lr=3.42e-04]
Epoch 31 TRAIN:  35%|███▌      | 48/137 [00:12<00:18,  4.90it/s, epoch=31, grad_norm=101.940, loss=299.6646, lr=3.42e-04]
Epoch 31 TRAIN:  35%|███▌      | 48/137 [00:12<00:18,  4.90it/s, epoch=31, grad_norm=68.522, loss=328.1453, lr=3.42e-04] 
Epoch 31 TRAIN:  35%|███▌      | 48/137 [00:12<00:18,  4.90it/s, epoch=31, grad_norm=94.242, loss=340.5236, lr=3.42e-04]
Epoch 31 TRAIN:  37%|███▋      | 51/137 [00:12<00:17,  4.79it/s, epoch=31, grad_norm=94.242, loss=340.5236, lr=3.42e-04]
Epoch 31 TRAIN:  37%|███▋      | 51/137 [00:13<00:17,  4.79it/s, epoch=31, grad_norm=148.583, loss=369.9541, lr=3.42e-04]
Epoch 31 TRAIN:  37%|███▋      | 51/137 [00:13<00:17,  4.79it/s, epoch=31, grad_norm=77.123, loss=302.9181, lr=3.42e-04] 
Epoch 31 TRAIN:  37%|███▋      | 51/137 [00:13<00:17,  4.79it/s, epoch=31, grad_norm=92.085, loss=304.2550, lr=3.42e-04]
Epoch 31 TRAIN:  39%|███▉      | 54/137 [00:13<00:16,  4.91it/s, epoch=31, grad_norm=92.085, loss=304.2550, lr=3.42e-04]
Epoch 31 TRAIN:  39%|███▉      | 54/137 [00:13<00:16,  4.91it/s, epoch=31, grad_norm=128.773, loss=287.8177, lr=3.42e-04]
Epoch 31 TRAIN:  39%|███▉      | 54/137 [00:13<00:16,  4.91it/s, epoch=31, grad_norm=93.046, loss=332.4671, lr=3.42e-04] 
Epoch 31 TRAIN:  39%|███▉      | 54/137 [00:14<00:16,  4.91it/s, epoch=31, grad_norm=108.145, loss=369.1313, lr=3.42e-04]
Epoch 31 TRAIN:  42%|████▏     | 57/137 [00:14<00:16,  4.89it/s, epoch=31, grad_norm=108.145, loss=369.1313, lr=3.42e-04]
Epoch 31 TRAIN:  42%|████▏     | 57/137 [00:14<00:16,  4.89it/s, epoch=31, grad_norm=134.123, loss=382.1002, lr=3.42e-04]
Epoch 31 TRAIN:  42%|████▏     | 57/137 [00:14<00:16,  4.89it/s, epoch=31, grad_norm=63.801, loss=293.5180, lr=3.43e-04] 
Epoch 31 TRAIN:  42%|████▏     | 57/137 [00:14<00:16,  4.89it/s, epoch=31, grad_norm=59.823, loss=303.6635, lr=3.43e-04]
Epoch 31 TRAIN:  44%|████▍     | 60/137 [00:14<00:16,  4.80it/s, epoch=31, grad_norm=59.823, loss=303.6635, lr=3.43e-04]
Epoch 31 TRAIN:  44%|████▍     | 60/137 [00:14<00:16,  4.80it/s, epoch=31, grad_norm=94.621, loss=325.6041, lr=3.43e-04]
Epoch 31 TRAIN:  44%|████▍     | 60/137 [00:15<00:16,  4.80it/s, epoch=31, grad_norm=91.405, loss=286.9649, lr=3.43e-04]
Epoch 31 TRAIN:  44%|████▍     | 60/137 [00:15<00:16,  4.80it/s, epoch=31, grad_norm=76.674, loss=350.6760, lr=3.43e-04]
Epoch 31 TRAIN:  46%|████▌     | 63/137 [00:15<00:15,  4.83it/s, epoch=31, grad_norm=76.674, loss=350.6760, lr=3.43e-04]
Epoch 31 TRAIN:  46%|████▌     | 63/137 [00:15<00:15,  4.83it/s, epoch=31, grad_norm=110.367, loss=270.0063, lr=3.43e-04]
Epoch 31 TRAIN:  46%|████▌     | 63/137 [00:15<00:15,  4.83it/s, epoch=31, grad_norm=113.253, loss=362.4856, lr=3.43e-04]
Epoch 31 TRAIN:  46%|████▌     | 63/137 [00:15<00:15,  4.83it/s, epoch=31, grad_norm=76.070, loss=299.0784, lr=3.43e-04] 
Epoch 31 TRAIN:  48%|████▊     | 66/137 [00:15<00:14,  4.88it/s, epoch=31, grad_norm=76.070, loss=299.0784, lr=3.43e-04]
Epoch 31 TRAIN:  48%|████▊     | 66/137 [00:16<00:14,  4.88it/s, epoch=31, grad_norm=94.354, loss=325.0859, lr=3.43e-04]
Epoch 31 TRAIN:  48%|████▊     | 66/137 [00:16<00:14,  4.88it/s, epoch=31, grad_norm=108.774, loss=328.8004, lr=3.43e-04]
Epoch 31 TRAIN:  48%|████▊     | 66/137 [00:16<00:14,  4.88it/s, epoch=31, grad_norm=84.076, loss=305.1187, lr=3.43e-04] 
Epoch 31 TRAIN:  50%|█████     | 69/137 [00:16<00:13,  5.01it/s, epoch=31, grad_norm=84.076, loss=305.1187, lr=3.43e-04]
Epoch 31 TRAIN:  50%|█████     | 69/137 [00:16<00:13,  5.01it/s, epoch=31, grad_norm=91.343, loss=288.6406, lr=3.43e-04]
Epoch 31 TRAIN:  50%|█████     | 69/137 [00:17<00:13,  5.01it/s, epoch=31, grad_norm=263.013, loss=234.0982, lr=3.44e-04]
Epoch 31 TRAIN:  50%|█████     | 69/137 [00:17<00:13,  5.01it/s, epoch=31, grad_norm=85.706, loss=303.1722, lr=3.44e-04] 
Epoch 31 TRAIN:  53%|█████▎    | 72/137 [00:17<00:13,  4.71it/s, epoch=31, grad_norm=85.706, loss=303.1722, lr=3.44e-04]
Epoch 31 TRAIN:  53%|█████▎    | 72/137 [00:17<00:13,  4.71it/s, epoch=31, grad_norm=106.273, loss=288.3113, lr=3.44e-04]
Epoch 31 TRAIN:  53%|█████▎    | 72/137 [00:17<00:13,  4.71it/s, epoch=31, grad_norm=85.581, loss=342.1895, lr=3.44e-04] 
Epoch 31 TRAIN:  53%|█████▎    | 72/137 [00:17<00:13,  4.71it/s, epoch=31, grad_norm=163.513, loss=354.3087, lr=3.44e-04]
Epoch 31 TRAIN:  55%|█████▍    | 75/137 [00:17<00:13,  4.67it/s, epoch=31, grad_norm=163.513, loss=354.3087, lr=3.44e-04]
Epoch 31 TRAIN:  55%|█████▍    | 75/137 [00:18<00:13,  4.67it/s, epoch=31, grad_norm=155.594, loss=328.7665, lr=3.44e-04]
Epoch 31 TRAIN:  55%|█████▍    | 75/137 [00:18<00:13,  4.67it/s, epoch=31, grad_norm=217.611, loss=394.4869, lr=3.44e-04]
Epoch 31 TRAIN:  55%|█████▍    | 75/137 [00:18<00:13,  4.67it/s, epoch=31, grad_norm=117.168, loss=275.7361, lr=3.44e-04]
Epoch 31 TRAIN:  57%|█████▋    | 78/137 [00:18<00:12,  4.68it/s, epoch=31, grad_norm=117.168, loss=275.7361, lr=3.44e-04]
Epoch 31 TRAIN:  57%|█████▋    | 78/137 [00:18<00:12,  4.68it/s, epoch=31, grad_norm=350.579, loss=219.2168, lr=3.44e-04]
Epoch 31 TRAIN:  57%|█████▋    | 78/137 [00:18<00:12,  4.68it/s, epoch=31, grad_norm=86.430, loss=329.0138, lr=3.44e-04] 
Epoch 31 TRAIN:  57%|█████▋    | 78/137 [00:19<00:12,  4.68it/s, epoch=31, grad_norm=198.912, loss=243.1961, lr=3.44e-04]
Epoch 31 TRAIN:  59%|█████▉    | 81/137 [00:19<00:11,  4.76it/s, epoch=31, grad_norm=198.912, loss=243.1961, lr=3.44e-04]
Epoch 31 TRAIN:  59%|█████▉    | 81/137 [00:19<00:11,  4.76it/s, epoch=31, grad_norm=104.371, loss=288.5565, lr=3.44e-04]
Epoch 31 TRAIN:  59%|█████▉    | 81/137 [00:19<00:11,  4.76it/s, epoch=31, grad_norm=62.047, loss=272.9140, lr=3.44e-04] 
Epoch 31 TRAIN:  59%|█████▉    | 81/137 [00:19<00:11,  4.76it/s, epoch=31, grad_norm=92.791, loss=281.1849, lr=3.45e-04]
Epoch 31 TRAIN:  61%|██████▏   | 84/137 [00:19<00:10,  4.92it/s, epoch=31, grad_norm=92.791, loss=281.1849, lr=3.45e-04]
Epoch 31 TRAIN:  61%|██████▏   | 84/137 [00:19<00:10,  4.92it/s, epoch=31, grad_norm=147.031, loss=315.1933, lr=3.45e-04]
Epoch 31 TRAIN:  61%|██████▏   | 84/137 [00:20<00:10,  4.92it/s, epoch=31, grad_norm=152.963, loss=332.8065, lr=3.45e-04]
Epoch 31 TRAIN:  61%|██████▏   | 84/137 [00:20<00:10,  4.92it/s, epoch=31, grad_norm=222.099, loss=267.6361, lr=3.45e-04]
Epoch 31 TRAIN:  64%|██████▎   | 87/137 [00:20<00:10,  4.72it/s, epoch=31, grad_norm=222.099, loss=267.6361, lr=3.45e-04]
Epoch 31 TRAIN:  64%|██████▎   | 87/137 [00:20<00:10,  4.72it/s, epoch=31, grad_norm=112.756, loss=336.2780, lr=3.45e-04]
Epoch 31 TRAIN:  64%|██████▎   | 87/137 [00:20<00:10,  4.72it/s, epoch=31, grad_norm=93.431, loss=337.2879, lr=3.45e-04] 
Epoch 31 TRAIN:  64%|██████▎   | 87/137 [00:21<00:10,  4.72it/s, epoch=31, grad_norm=152.340, loss=267.0783, lr=3.45e-04]
Epoch 31 TRAIN:  66%|██████▌   | 90/137 [00:21<00:09,  4.73it/s, epoch=31, grad_norm=152.340, loss=267.0783, lr=3.45e-04]
Epoch 31 TRAIN:  66%|██████▌   | 90/137 [00:21<00:09,  4.73it/s, epoch=31, grad_norm=86.754, loss=278.8273, lr=3.45e-04] 
Epoch 31 TRAIN:  66%|██████▌   | 90/137 [00:21<00:09,  4.73it/s, epoch=31, grad_norm=136.642, loss=248.8843, lr=3.45e-04]
Epoch 31 TRAIN:  66%|██████▌   | 90/137 [00:21<00:09,  4.73it/s, epoch=31, grad_norm=75.965, loss=241.1194, lr=3.45e-04] 
Epoch 31 TRAIN:  68%|██████▊   | 93/137 [00:21<00:09,  4.84it/s, epoch=31, grad_norm=75.965, loss=241.1194, lr=3.45e-04]
Epoch 31 TRAIN:  68%|██████▊   | 93/137 [00:21<00:09,  4.84it/s, epoch=31, grad_norm=106.391, loss=325.6290, lr=3.45e-04]
Epoch 31 TRAIN:  68%|██████▊   | 93/137 [00:22<00:09,  4.84it/s, epoch=31, grad_norm=128.077, loss=320.0474, lr=3.45e-04]
Epoch 31 TRAIN:  68%|██████▊   | 93/137 [00:22<00:09,  4.84it/s, epoch=31, grad_norm=153.161, loss=344.3742, lr=3.46e-04]
Epoch 31 TRAIN:  70%|███████   | 96/137 [00:22<00:09,  4.51it/s, epoch=31, grad_norm=153.161, loss=344.3742, lr=3.46e-04]
Epoch 31 TRAIN:  70%|███████   | 96/137 [00:22<00:09,  4.51it/s, epoch=31, grad_norm=134.673, loss=365.0015, lr=3.46e-04]
Epoch 31 TRAIN:  70%|███████   | 96/137 [00:22<00:09,  4.51it/s, epoch=31, grad_norm=147.299, loss=258.9854, lr=3.46e-04]
Epoch 31 TRAIN:  70%|███████   | 96/137 [00:22<00:09,  4.51it/s, epoch=31, grad_norm=210.396, loss=358.1069, lr=3.46e-04]
Epoch 31 TRAIN:  72%|███████▏  | 99/137 [00:22<00:08,  4.68it/s, epoch=31, grad_norm=210.396, loss=358.1069, lr=3.46e-04]
Epoch 31 TRAIN:  72%|███████▏  | 99/137 [00:23<00:08,  4.68it/s, epoch=31, grad_norm=149.272, loss=258.2020, lr=3.46e-04]2026-04-16 20:07:37,927 DEBUG TRAIN | steps/sec 4.331| Batch 31/100 loss 258.202026 loss_att 226.008179 loss_ctc 333.321045 th_accuracy 0.269122 lr 3.4584e-04 grad_norm 149.271820 rank 0

Epoch 31 TRAIN:  72%|███████▏  | 99/137 [00:23<00:08,  4.68it/s, epoch=31, grad_norm=202.963, loss=359.0021, lr=3.46e-04]
Epoch 31 TRAIN:  72%|███████▏  | 99/137 [00:23<00:08,  4.68it/s, epoch=31, grad_norm=169.134, loss=396.1647, lr=3.46e-04]
Epoch 31 TRAIN:  74%|███████▍  | 102/137 [00:23<00:07,  4.55it/s, epoch=31, grad_norm=169.134, loss=396.1647, lr=3.46e-04]
Epoch 31 TRAIN:  74%|███████▍  | 102/137 [00:23<00:07,  4.55it/s, epoch=31, grad_norm=268.899, loss=326.5400, lr=3.46e-04]
Epoch 31 TRAIN:  74%|███████▍  | 102/137 [00:24<00:07,  4.55it/s, epoch=31, grad_norm=201.481, loss=293.1249, lr=3.46e-04]
Epoch 31 TRAIN:  74%|███████▍  | 102/137 [00:24<00:07,  4.55it/s, epoch=31, grad_norm=110.139, loss=325.0498, lr=3.46e-04]
Epoch 31 TRAIN:  77%|███████▋  | 105/137 [00:24<00:06,  4.79it/s, epoch=31, grad_norm=110.139, loss=325.0498, lr=3.46e-04]
Epoch 31 TRAIN:  77%|███████▋  | 105/137 [00:24<00:06,  4.79it/s, epoch=31, grad_norm=143.990, loss=329.2576, lr=3.46e-04]
Epoch 31 TRAIN:  77%|███████▋  | 105/137 [00:24<00:06,  4.79it/s, epoch=31, grad_norm=84.328, loss=304.8988, lr=3.46e-04] 
Epoch 31 TRAIN:  77%|███████▋  | 105/137 [00:24<00:06,  4.79it/s, epoch=31, grad_norm=117.896, loss=354.9049, lr=3.46e-04]
Epoch 31 TRAIN:  79%|███████▉  | 108/137 [00:24<00:05,  4.86it/s, epoch=31, grad_norm=117.896, loss=354.9049, lr=3.46e-04]
Epoch 31 TRAIN:  79%|███████▉  | 108/137 [00:25<00:05,  4.86it/s, epoch=31, grad_norm=171.263, loss=342.4683, lr=3.47e-04]
Epoch 31 TRAIN:  79%|███████▉  | 108/137 [00:25<00:05,  4.86it/s, epoch=31, grad_norm=127.937, loss=355.9994, lr=3.47e-04]
Epoch 31 TRAIN:  79%|███████▉  | 108/137 [00:25<00:05,  4.86it/s, epoch=31, grad_norm=128.831, loss=402.1393, lr=3.47e-04]
Epoch 31 TRAIN:  81%|████████  | 111/137 [00:25<00:05,  4.89it/s, epoch=31, grad_norm=128.831, loss=402.1393, lr=3.47e-04]
Epoch 31 TRAIN:  81%|████████  | 111/137 [00:25<00:05,  4.89it/s, epoch=31, grad_norm=185.802, loss=260.0503, lr=3.47e-04]
Epoch 31 TRAIN:  81%|████████  | 111/137 [00:25<00:05,  4.89it/s, epoch=31, grad_norm=189.621, loss=225.4176, lr=3.47e-04]
Epoch 31 TRAIN:  81%|████████  | 111/137 [00:25<00:05,  4.89it/s, epoch=31, grad_norm=181.911, loss=423.7137, lr=3.47e-04]
Epoch 31 TRAIN:  83%|████████▎ | 114/137 [00:25<00:04,  5.09it/s, epoch=31, grad_norm=181.911, loss=423.7137, lr=3.47e-04]
Epoch 31 TRAIN:  83%|████████▎ | 114/137 [00:26<00:04,  5.09it/s, epoch=31, grad_norm=134.038, loss=348.1613, lr=3.47e-04]
Epoch 31 TRAIN:  83%|████████▎ | 114/137 [00:26<00:04,  5.09it/s, epoch=31, grad_norm=213.666, loss=332.4169, lr=3.47e-04]
Epoch 31 TRAIN:  83%|████████▎ | 114/137 [00:26<00:04,  5.09it/s, epoch=31, grad_norm=104.179, loss=304.0646, lr=3.47e-04]
Epoch 31 TRAIN:  83%|████████▎ | 114/137 [00:26<00:04,  5.09it/s, epoch=31, grad_norm=115.316, loss=315.2581, lr=3.47e-04]
Epoch 31 TRAIN:  86%|████████▌ | 118/137 [00:26<00:03,  4.97it/s, epoch=31, grad_norm=115.316, loss=315.2581, lr=3.47e-04]
Epoch 31 TRAIN:  86%|████████▌ | 118/137 [00:27<00:03,  4.97it/s, epoch=31, grad_norm=137.930, loss=383.0226, lr=3.47e-04]
Epoch 31 TRAIN:  86%|████████▌ | 118/137 [00:27<00:03,  4.97it/s, epoch=31, grad_norm=108.934, loss=344.6201, lr=3.47e-04]
Epoch 31 TRAIN:  86%|████████▌ | 118/137 [00:27<00:03,  4.97it/s, epoch=31, grad_norm=206.057, loss=317.1948, lr=3.48e-04]
Epoch 31 TRAIN:  88%|████████▊ | 121/137 [00:27<00:03,  5.21it/s, epoch=31, grad_norm=206.057, loss=317.1948, lr=3.48e-04]
Epoch 31 TRAIN:  88%|████████▊ | 121/137 [00:27<00:03,  5.21it/s, epoch=31, grad_norm=93.536, loss=282.2607, lr=3.48e-04] 
Epoch 31 TRAIN:  88%|████████▊ | 121/137 [00:27<00:03,  5.21it/s, epoch=31, grad_norm=137.128, loss=288.7486, lr=3.48e-04]
Epoch 31 TRAIN:  88%|████████▊ | 121/137 [00:27<00:03,  5.21it/s, epoch=31, grad_norm=157.310, loss=361.3598, lr=3.48e-04]
Epoch 31 TRAIN:  91%|█████████ | 124/137 [00:27<00:02,  5.18it/s, epoch=31, grad_norm=157.310, loss=361.3598, lr=3.48e-04]
Epoch 31 TRAIN:  91%|█████████ | 124/137 [00:28<00:02,  5.18it/s, epoch=31, grad_norm=88.272, loss=291.1837, lr=3.48e-04] 
Epoch 31 TRAIN:  91%|█████████ | 124/137 [00:28<00:02,  5.18it/s, epoch=31, grad_norm=92.967, loss=264.3559, lr=3.48e-04]
Epoch 31 TRAIN:  91%|█████████ | 124/137 [00:28<00:02,  5.18it/s, epoch=31, grad_norm=80.855, loss=265.8231, lr=3.48e-04]
Epoch 31 TRAIN:  93%|█████████▎| 127/137 [00:28<00:02,  4.98it/s, epoch=31, grad_norm=80.855, loss=265.8231, lr=3.48e-04]
Epoch 31 TRAIN:  93%|█████████▎| 127/137 [00:28<00:02,  4.98it/s, epoch=31, grad_norm=133.276, loss=304.8293, lr=3.48e-04]
Epoch 31 TRAIN:  93%|█████████▎| 127/137 [00:28<00:02,  4.98it/s, epoch=31, grad_norm=136.296, loss=332.4542, lr=3.48e-04]
Epoch 31 TRAIN:  93%|█████████▎| 127/137 [00:29<00:02,  4.98it/s, epoch=31, grad_norm=120.675, loss=275.7989, lr=3.48e-04]
Epoch 31 TRAIN:  95%|█████████▍| 130/137 [00:29<00:01,  5.03it/s, epoch=31, grad_norm=120.675, loss=275.7989, lr=3.48e-04]
Epoch 31 TRAIN:  95%|█████████▍| 130/137 [00:29<00:01,  5.03it/s, epoch=31, grad_norm=161.683, loss=355.9894, lr=3.48e-04]
Epoch 31 TRAIN:  95%|█████████▍| 130/137 [00:29<00:01,  5.03it/s, epoch=31, grad_norm=117.246, loss=332.8458, lr=3.48e-04]
Epoch 31 TRAIN:  95%|█████████▍| 130/137 [00:29<00:01,  5.03it/s, epoch=31, grad_norm=134.335, loss=381.7068, lr=3.48e-04]
Epoch 31 TRAIN:  97%|█████████▋| 133/137 [00:29<00:00,  4.89it/s, epoch=31, grad_norm=134.335, loss=381.7068, lr=3.48e-04]
Epoch 31 TRAIN:  97%|█████████▋| 133/137 [00:29<00:00,  4.89it/s, epoch=31, grad_norm=194.399, loss=390.4474, lr=3.49e-04]
Epoch 31 TRAIN:  97%|█████████▋| 133/137 [00:30<00:00,  4.89it/s, epoch=31, grad_norm=222.479, loss=295.9463, lr=3.49e-04]
Epoch 31 TRAIN:  97%|█████████▋| 133/137 [00:30<00:00,  4.89it/s, epoch=31, grad_norm=180.566, loss=262.3390, lr=3.49e-04]
Epoch 31 TRAIN:  99%|█████████▉| 136/137 [00:30<00:00,  5.17it/s, epoch=31, grad_norm=180.566, loss=262.3390, lr=3.49e-04]
Epoch 31 TRAIN:  99%|█████████▉| 136/137 [00:30<00:00,  5.17it/s, epoch=31, grad_norm=225.999, loss=380.1964, lr=3.49e-04]
Epoch 31 TRAIN: 100%|██████████| 137/137 [00:30<00:00,  4.49it/s, epoch=31, grad_norm=225.999, loss=380.1964, lr=3.49e-04]

CV:   0%|          | 0/6 [00:00<?, ?it/s]/home/rookie/Documents/batuka/projects/wenet/wenet/wenet/utils/train_utils.py:689: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  amp_autocast(enabled=dtype is not None,

CV:  17%|█▋        | 1/6 [00:00<00:03,  1.33it/s]
                                                 
2026-04-16 20:07:46,081 INFO Epoch 31 Step 4360 CV info lr 3.4880e-04 cv_loss 483.06436434659093 rank 0 acc 0.2517387121915817
2026-04-16 20:07:46,083 INFO [Rank 0] Checkpoint: save to checkpoint exp/conformer_mongolian/epoch_31.pt
2026-04-16 20:07:46,215 INFO Epoch 32 Step 4360 TRAIN info lr 3.4880e-04 rank 0
2026-04-16 20:07:46,230 INFO using accumulate grad, new batch size is 1 times larger than before

Epoch 32 TRAIN:   0%|          | 0/137 [00:00<?, ?it/s]
Epoch 32 TRAIN:   0%|          | 0/137 [00:04<?, ?it/s, epoch=32, grad_norm=593.402, loss=504.0768, lr=3.49e-04]
Epoch 32 TRAIN:   1%|          | 1/137 [00:04<09:16,  4.09s/it, epoch=32, grad_norm=593.402, loss=504.0768, lr=3.49e-04]
Epoch 32 TRAIN:   1%|          | 1/137 [00:04<09:16,  4.09s/it, epoch=32, grad_norm=539.517, loss=476.2383, lr=3.49e-04]
Epoch 32 TRAIN:   1%|          | 1/137 [00:04<09:16,  4.09s/it, epoch=32, grad_norm=346.920, loss=332.4090, lr=3.49e-04]
Epoch 32 TRAIN:   1%|          | 1/137 [00:04<09:16,  4.09s/it, epoch=32, grad_norm=225.808, loss=344.9897, lr=3.49e-04]
Epoch 32 TRAIN:   1%|          | 1/137 [00:04<09:16,  4.09s/it, epoch=32, grad_norm=405.076, loss=384.5422, lr=3.49e-04]
Epoch 32 TRAIN:   1%|          | 1/137 [00:04<09:16,  4.09s/it, epoch=32, grad_norm=281.850, loss=332.5589, lr=3.49e-04]
Epoch 32 TRAIN:   1%|          | 1/137 [00:04<09:16,  4.09s/it, epoch=32, grad_norm=311.334, loss=369.5018, lr=3.49e-04]
Epoch 32 TRAIN:   1%|          | 1/137 [00:04<09:16,  4.09s/it, epoch=32, grad_norm=267.831, loss=348.3932, lr=3.49e-04]
Epoch 32 TRAIN:   1%|          | 1/137 [00:04<09:16,  4.09s/it, epoch=32, grad_norm=72.590, loss=293.1232, lr=3.50e-04] 
Epoch 32 TRAIN:   7%|▋         | 9/137 [00:04<00:51,  2.49it/s, epoch=32, grad_norm=72.590, loss=293.1232, lr=3.50e-04]
Epoch 32 TRAIN:   7%|▋         | 9/137 [00:04<00:51,  2.49it/s, epoch=32, grad_norm=75.849, loss=317.6505, lr=3.50e-04]
Epoch 32 TRAIN:   7%|▋         | 9/137 [00:05<00:51,  2.49it/s, epoch=32, grad_norm=90.930, loss=317.6760, lr=3.50e-04]
Epoch 32 TRAIN:   7%|▋         | 9/137 [00:05<00:51,  2.49it/s, epoch=32, grad_norm=183.597, loss=306.0424, lr=3.50e-04]
Epoch 32 TRAIN:   7%|▋         | 9/137 [00:05<00:51,  2.49it/s, epoch=32, grad_norm=73.141, loss=319.1304, lr=3.50e-04] 
Epoch 32 TRAIN:   9%|▉         | 13/137 [00:05<00:37,  3.27it/s, epoch=32, grad_norm=73.141, loss=319.1304, lr=3.50e-04]
Epoch 32 TRAIN:   9%|▉         | 13/137 [00:05<00:37,  3.27it/s, epoch=32, grad_norm=138.569, loss=296.6838, lr=3.50e-04]
Epoch 32 TRAIN:   9%|▉         | 13/137 [00:05<00:37,  3.27it/s, epoch=32, grad_norm=109.663, loss=338.0079, lr=3.50e-04]
Epoch 32 TRAIN:   9%|▉         | 13/137 [00:05<00:37,  3.27it/s, epoch=32, grad_norm=85.448, loss=337.2037, lr=3.50e-04] 
Epoch 32 TRAIN:   9%|▉         | 13/137 [00:05<00:37,  3.27it/s, epoch=32, grad_norm=81.236, loss=330.8613, lr=3.50e-04]
Epoch 32 TRAIN:  12%|█▏        | 17/137 [00:05<00:30,  4.00it/s, epoch=32, grad_norm=81.236, loss=330.8613, lr=3.50e-04]
Epoch 32 TRAIN:  12%|█▏        | 17/137 [00:06<00:30,  4.00it/s, epoch=32, grad_norm=73.986, loss=320.1143, lr=3.50e-04]
Epoch 32 TRAIN:  12%|█▏        | 17/137 [00:06<00:30,  4.00it/s, epoch=32, grad_norm=66.739, loss=347.5873, lr=3.50e-04]
Epoch 32 TRAIN:  12%|█▏        | 17/137 [00:06<00:30,  4.00it/s, epoch=32, grad_norm=67.652, loss=357.9375, lr=3.50e-04]
Epoch 32 TRAIN:  12%|█▏        | 17/137 [00:06<00:30,  4.00it/s, epoch=32, grad_norm=61.609, loss=335.7819, lr=3.50e-04]
Epoch 32 TRAIN:  15%|█▌        | 21/137 [00:06<00:25,  4.52it/s, epoch=32, grad_norm=61.609, loss=335.7819, lr=3.50e-04]
Epoch 32 TRAIN:  15%|█▌        | 21/137 [00:06<00:25,  4.52it/s, epoch=32, grad_norm=65.044, loss=324.2993, lr=3.51e-04]
Epoch 32 TRAIN:  15%|█▌        | 21/137 [00:07<00:25,  4.52it/s, epoch=32, grad_norm=68.975, loss=340.2669, lr=3.51e-04]
Epoch 32 TRAIN:  15%|█▌        | 21/137 [00:07<00:25,  4.52it/s, epoch=32, grad_norm=85.442, loss=363.7625, lr=3.51e-04]
Epoch 32 TRAIN:  15%|█▌        | 21/137 [00:07<00:25,  4.52it/s, epoch=32, grad_norm=99.353, loss=330.9891, lr=3.51e-04]
Epoch 32 TRAIN:  18%|█▊        | 25/137 [00:07<00:23,  4.71it/s, epoch=32, grad_norm=99.353, loss=330.9891, lr=3.51e-04]
Epoch 32 TRAIN:  18%|█▊        | 25/137 [00:07<00:23,  4.71it/s, epoch=32, grad_norm=180.829, loss=262.0960, lr=3.51e-04]
Epoch 32 TRAIN:  18%|█▊        | 25/137 [00:07<00:23,  4.71it/s, epoch=32, grad_norm=75.030, loss=355.2147, lr=3.51e-04] 
Epoch 32 TRAIN:  18%|█▊        | 25/137 [00:07<00:23,  4.71it/s, epoch=32, grad_norm=109.796, loss=342.6055, lr=3.51e-04]
Epoch 32 TRAIN:  18%|█▊        | 25/137 [00:08<00:23,  4.71it/s, epoch=32, grad_norm=87.797, loss=298.1181, lr=3.51e-04] 
Epoch 32 TRAIN:  21%|██        | 29/137 [00:08<00:21,  5.03it/s, epoch=32, grad_norm=87.797, loss=298.1181, lr=3.51e-04]
Epoch 32 TRAIN:  21%|██        | 29/137 [00:08<00:21,  5.03it/s, epoch=32, grad_norm=61.218, loss=310.0637, lr=3.51e-04]
Epoch 32 TRAIN:  21%|██        | 29/137 [00:08<00:21,  5.03it/s, epoch=32, grad_norm=86.247, loss=334.1249, lr=3.51e-04]
Epoch 32 TRAIN:  21%|██        | 29/137 [00:08<00:21,  5.03it/s, epoch=32, grad_norm=99.063, loss=336.9144, lr=3.51e-04]
Epoch 32 TRAIN:  21%|██        | 29/137 [00:08<00:21,  5.03it/s, epoch=32, grad_norm=90.989, loss=278.6497, lr=3.51e-04]
Epoch 32 TRAIN:  24%|██▍       | 33/137 [00:08<00:20,  5.07it/s, epoch=32, grad_norm=90.989, loss=278.6497, lr=3.51e-04]
Epoch 32 TRAIN:  24%|██▍       | 33/137 [00:09<00:20,  5.07it/s, epoch=32, grad_norm=61.664, loss=296.8365, lr=3.52e-04]
Epoch 32 TRAIN:  24%|██▍       | 33/137 [00:09<00:20,  5.07it/s, epoch=32, grad_norm=67.762, loss=292.7496, lr=3.52e-04]
Epoch 32 TRAIN:  24%|██▍       | 33/137 [00:09<00:20,  5.07it/s, epoch=32, grad_norm=61.904, loss=273.6740, lr=3.52e-04]
Epoch 32 TRAIN:  26%|██▋       | 36/137 [00:09<00:20,  4.97it/s, epoch=32, grad_norm=61.904, loss=273.6740, lr=3.52e-04]
Epoch 32 TRAIN:  26%|██▋       | 36/137 [00:09<00:20,  4.97it/s, epoch=32, grad_norm=83.625, loss=331.5486, lr=3.52e-04]
Epoch 32 TRAIN:  26%|██▋       | 36/137 [00:09<00:20,  4.97it/s, epoch=32, grad_norm=90.223, loss=309.0806, lr=3.52e-04]
Epoch 32 TRAIN:  26%|██▋       | 36/137 [00:10<00:20,  4.97it/s, epoch=32, grad_norm=88.020, loss=297.9231, lr=3.52e-04]
Epoch 32 TRAIN:  28%|██▊       | 39/137 [00:10<00:19,  5.09it/s, epoch=32, grad_norm=88.020, loss=297.9231, lr=3.52e-04]
Epoch 32 TRAIN:  28%|██▊       | 39/137 [00:10<00:19,  5.09it/s, epoch=32, grad_norm=98.090, loss=317.2941, lr=3.52e-04]
Epoch 32 TRAIN:  28%|██▊       | 39/137 [00:10<00:19,  5.09it/s, epoch=32, grad_norm=69.503, loss=312.3579, lr=3.52e-04]
Epoch 32 TRAIN:  28%|██▊       | 39/137 [00:10<00:19,  5.09it/s, epoch=32, grad_norm=75.421, loss=280.7129, lr=3.52e-04]
Epoch 32 TRAIN:  31%|███       | 42/137 [00:10<00:18,  5.18it/s, epoch=32, grad_norm=75.421, loss=280.7129, lr=3.52e-04]
Epoch 32 TRAIN:  31%|███       | 42/137 [00:10<00:18,  5.18it/s, epoch=32, grad_norm=72.560, loss=325.6495, lr=3.52e-04]
Epoch 32 TRAIN:  31%|███       | 42/137 [00:11<00:18,  5.18it/s, epoch=32, grad_norm=86.128, loss=289.9897, lr=3.52e-04]
Epoch 32 TRAIN:  31%|███       | 42/137 [00:11<00:18,  5.18it/s, epoch=32, grad_norm=91.800, loss=284.1529, lr=3.52e-04]
Epoch 32 TRAIN:  33%|███▎      | 45/137 [00:11<00:18,  4.89it/s, epoch=32, grad_norm=91.800, loss=284.1529, lr=3.52e-04]
Epoch 32 TRAIN:  33%|███▎      | 45/137 [00:11<00:18,  4.89it/s, epoch=32, grad_norm=78.718, loss=323.2995, lr=3.52e-04]
Epoch 32 TRAIN:  33%|███▎      | 45/137 [00:11<00:18,  4.89it/s, epoch=32, grad_norm=66.346, loss=267.6765, lr=3.53e-04]
Epoch 32 TRAIN:  33%|███▎      | 45/137 [00:11<00:18,  4.89it/s, epoch=32, grad_norm=123.869, loss=315.1832, lr=3.53e-04]
Epoch 32 TRAIN:  35%|███▌      | 48/137 [00:11<00:18,  4.94it/s, epoch=32, grad_norm=123.869, loss=315.1832, lr=3.53e-04]
Epoch 32 TRAIN:  35%|███▌      | 48/137 [00:12<00:18,  4.94it/s, epoch=32, grad_norm=94.450, loss=315.3062, lr=3.53e-04] 
Epoch 32 TRAIN:  35%|███▌      | 48/137 [00:12<00:18,  4.94it/s, epoch=32, grad_norm=154.531, loss=369.9787, lr=3.53e-04]
Epoch 32 TRAIN:  35%|███▌      | 48/137 [00:12<00:18,  4.94it/s, epoch=32, grad_norm=76.990, loss=340.0375, lr=3.53e-04] 
Epoch 32 TRAIN:  37%|███▋      | 51/137 [00:12<00:17,  5.03it/s, epoch=32, grad_norm=76.990, loss=340.0375, lr=3.53e-04]
Epoch 32 TRAIN:  37%|███▋      | 51/137 [00:12<00:17,  5.03it/s, epoch=32, grad_norm=99.697, loss=344.2007, lr=3.53e-04]
Epoch 32 TRAIN:  37%|███▋      | 51/137 [00:12<00:17,  5.03it/s, epoch=32, grad_norm=95.975, loss=322.6143, lr=3.53e-04]
Epoch 32 TRAIN:  37%|███▋      | 51/137 [00:13<00:17,  5.03it/s, epoch=32, grad_norm=73.073, loss=333.7974, lr=3.53e-04]
Epoch 32 TRAIN:  39%|███▉      | 54/137 [00:13<00:16,  5.18it/s, epoch=32, grad_norm=73.073, loss=333.7974, lr=3.53e-04]
Epoch 32 TRAIN:  39%|███▉      | 54/137 [00:13<00:16,  5.18it/s, epoch=32, grad_norm=76.682, loss=322.8664, lr=3.53e-04]
Epoch 32 TRAIN:  39%|███▉      | 54/137 [00:13<00:16,  5.18it/s, epoch=32, grad_norm=61.621, loss=329.9512, lr=3.53e-04]
Epoch 32 TRAIN:  39%|███▉      | 54/137 [00:13<00:16,  5.18it/s, epoch=32, grad_norm=90.619, loss=334.6440, lr=3.53e-04]
Epoch 32 TRAIN:  42%|████▏     | 57/137 [00:13<00:16,  4.86it/s, epoch=32, grad_norm=90.619, loss=334.6440, lr=3.53e-04]
Epoch 32 TRAIN:  42%|████▏     | 57/137 [00:13<00:16,  4.86it/s, epoch=32, grad_norm=75.315, loss=348.4072, lr=3.53e-04]
Epoch 32 TRAIN:  42%|████▏     | 57/137 [00:14<00:16,  4.86it/s, epoch=32, grad_norm=99.931, loss=311.5383, lr=3.54e-04]
Epoch 32 TRAIN:  42%|████▏     | 57/137 [00:14<00:16,  4.86it/s, epoch=32, grad_norm=77.986, loss=324.0374, lr=3.54e-04]
Epoch 32 TRAIN:  44%|████▍     | 60/137 [00:14<00:15,  4.86it/s, epoch=32, grad_norm=77.986, loss=324.0374, lr=3.54e-04]
Epoch 32 TRAIN:  44%|████▍     | 60/137 [00:14<00:15,  4.86it/s, epoch=32, grad_norm=76.547, loss=294.7562, lr=3.54e-04]
Epoch 32 TRAIN:  44%|████▍     | 60/137 [00:14<00:15,  4.86it/s, epoch=32, grad_norm=163.357, loss=342.3854, lr=3.54e-04]
Epoch 32 TRAIN:  44%|████▍     | 60/137 [00:14<00:15,  4.86it/s, epoch=32, grad_norm=86.409, loss=290.2964, lr=3.54e-04] 
Epoch 32 TRAIN:  46%|████▌     | 63/137 [00:14<00:14,  4.96it/s, epoch=32, grad_norm=86.409, loss=290.2964, lr=3.54e-04]
Epoch 32 TRAIN:  46%|████▌     | 63/137 [00:15<00:14,  4.96it/s, epoch=32, grad_norm=166.203, loss=354.6976, lr=3.54e-04]
Epoch 32 TRAIN:  46%|████▌     | 63/137 [00:15<00:14,  4.96it/s, epoch=32, grad_norm=81.109, loss=308.5869, lr=3.54e-04] 
Epoch 32 TRAIN:  46%|████▌     | 63/137 [00:15<00:14,  4.96it/s, epoch=32, grad_norm=87.321, loss=296.2023, lr=3.54e-04]
Epoch 32 TRAIN:  48%|████▊     | 66/137 [00:15<00:14,  4.75it/s, epoch=32, grad_norm=87.321, loss=296.2023, lr=3.54e-04]
Epoch 32 TRAIN:  48%|████▊     | 66/137 [00:15<00:14,  4.75it/s, epoch=32, grad_norm=98.549, loss=344.1423, lr=3.54e-04]
Epoch 32 TRAIN:  48%|████▊     | 66/137 [00:16<00:14,  4.75it/s, epoch=32, grad_norm=104.137, loss=346.5757, lr=3.54e-04]
Epoch 32 TRAIN:  48%|████▊     | 66/137 [00:16<00:14,  4.75it/s, epoch=32, grad_norm=133.895, loss=317.0019, lr=3.54e-04]
Epoch 32 TRAIN:  50%|█████     | 69/137 [00:16<00:14,  4.85it/s, epoch=32, grad_norm=133.895, loss=317.0019, lr=3.54e-04]
Epoch 32 TRAIN:  50%|█████     | 69/137 [00:16<00:14,  4.85it/s, epoch=32, grad_norm=98.973, loss=331.0174, lr=3.54e-04] 
Epoch 32 TRAIN:  50%|█████     | 69/137 [00:16<00:14,  4.85it/s, epoch=32, grad_norm=113.238, loss=278.4136, lr=3.54e-04]
Epoch 32 TRAIN:  50%|█████     | 69/137 [00:16<00:14,  4.85it/s, epoch=32, grad_norm=73.013, loss=300.5075, lr=3.55e-04] 
Epoch 32 TRAIN:  53%|█████▎    | 72/137 [00:16<00:13,  4.85it/s, epoch=32, grad_norm=73.013, loss=300.5075, lr=3.55e-04]
Epoch 32 TRAIN:  53%|█████▎    | 72/137 [00:17<00:13,  4.85it/s, epoch=32, grad_norm=73.091, loss=319.6739, lr=3.55e-04]
Epoch 32 TRAIN:  53%|█████▎    | 72/137 [00:17<00:13,  4.85it/s, epoch=32, grad_norm=202.992, loss=261.4624, lr=3.55e-04]
Epoch 32 TRAIN:  53%|█████▎    | 72/137 [00:17<00:13,  4.85it/s, epoch=32, grad_norm=83.518, loss=323.0703, lr=3.55e-04] 
Epoch 32 TRAIN:  55%|█████▍    | 75/137 [00:17<00:12,  4.97it/s, epoch=32, grad_norm=83.518, loss=323.0703, lr=3.55e-04]
Epoch 32 TRAIN:  55%|█████▍    | 75/137 [00:17<00:12,  4.97it/s, epoch=32, grad_norm=165.933, loss=301.1126, lr=3.55e-04]
Epoch 32 TRAIN:  55%|█████▍    | 75/137 [00:17<00:12,  4.97it/s, epoch=32, grad_norm=84.489, loss=339.2610, lr=3.55e-04] 
Epoch 32 TRAIN:  55%|█████▍    | 75/137 [00:18<00:12,  4.97it/s, epoch=32, grad_norm=111.117, loss=319.3533, lr=3.55e-04]
Epoch 32 TRAIN:  57%|█████▋    | 78/137 [00:18<00:12,  4.56it/s, epoch=32, grad_norm=111.117, loss=319.3533, lr=3.55e-04]
Epoch 32 TRAIN:  57%|█████▋    | 78/137 [00:18<00:12,  4.56it/s, epoch=32, grad_norm=101.364, loss=345.9758, lr=3.55e-04]
Epoch 32 TRAIN:  57%|█████▋    | 78/137 [00:18<00:12,  4.56it/s, epoch=32, grad_norm=98.262, loss=276.9383, lr=3.55e-04] 
Epoch 32 TRAIN:  57%|█████▋    | 78/137 [00:18<00:12,  4.56it/s, epoch=32, grad_norm=93.635, loss=358.1337, lr=3.55e-04]
Epoch 32 TRAIN:  59%|█████▉    | 81/137 [00:18<00:11,  4.86it/s, epoch=32, grad_norm=93.635, loss=358.1337, lr=3.55e-04]
Epoch 32 TRAIN:  59%|█████▉    | 81/137 [00:18<00:11,  4.86it/s, epoch=32, grad_norm=311.028, loss=239.2732, lr=3.55e-04]
Epoch 32 TRAIN:  59%|█████▉    | 81/137 [00:19<00:11,  4.86it/s, epoch=32, grad_norm=86.433, loss=370.9238, lr=3.55e-04] 
Epoch 32 TRAIN:  59%|█████▉    | 81/137 [00:19<00:11,  4.86it/s, epoch=32, grad_norm=262.186, loss=295.1597, lr=3.56e-04]
Epoch 32 TRAIN:  61%|██████▏   | 84/137 [00:19<00:11,  4.46it/s, epoch=32, grad_norm=262.186, loss=295.1597, lr=3.56e-04]
Epoch 32 TRAIN:  61%|██████▏   | 84/137 [00:19<00:11,  4.46it/s, epoch=32, grad_norm=138.756, loss=339.7717, lr=3.56e-04]
Epoch 32 TRAIN:  61%|██████▏   | 84/137 [00:19<00:11,  4.46it/s, epoch=32, grad_norm=80.389, loss=310.7907, lr=3.56e-04] 
Epoch 32 TRAIN:  61%|██████▏   | 84/137 [00:20<00:11,  4.46it/s, epoch=32, grad_norm=76.557, loss=322.6855, lr=3.56e-04]
Epoch 32 TRAIN:  64%|██████▎   | 87/137 [00:20<00:10,  4.71it/s, epoch=32, grad_norm=76.557, loss=322.6855, lr=3.56e-04]
Epoch 32 TRAIN:  64%|██████▎   | 87/137 [00:20<00:10,  4.71it/s, epoch=32, grad_norm=121.069, loss=322.8477, lr=3.56e-04]
Epoch 32 TRAIN:  64%|██████▎   | 87/137 [00:20<00:10,  4.71it/s, epoch=32, grad_norm=78.249, loss=279.7086, lr=3.56e-04] 
Epoch 32 TRAIN:  64%|██████▎   | 87/137 [00:20<00:10,  4.71it/s, epoch=32, grad_norm=199.686, loss=273.3720, lr=3.56e-04]
Epoch 32 TRAIN:  66%|██████▌   | 90/137 [00:20<00:10,  4.41it/s, epoch=32, grad_norm=199.686, loss=273.3720, lr=3.56e-04]
Epoch 32 TRAIN:  66%|██████▌   | 90/137 [00:20<00:10,  4.41it/s, epoch=32, grad_norm=100.322, loss=257.7334, lr=3.56e-04]
Epoch 32 TRAIN:  66%|██████▌   | 90/137 [00:21<00:10,  4.41it/s, epoch=32, grad_norm=150.977, loss=306.5329, lr=3.56e-04]
Epoch 32 TRAIN:  66%|██████▌   | 90/137 [00:21<00:10,  4.41it/s, epoch=32, grad_norm=77.316, loss=302.0880, lr=3.56e-04] 
Epoch 32 TRAIN:  68%|██████▊   | 93/137 [00:21<00:09,  4.58it/s, epoch=32, grad_norm=77.316, loss=302.0880, lr=3.56e-04]
Epoch 32 TRAIN:  68%|██████▊   | 93/137 [00:21<00:09,  4.58it/s, epoch=32, grad_norm=146.623, loss=300.2013, lr=3.56e-04]
Epoch 32 TRAIN:  68%|██████▊   | 93/137 [00:21<00:09,  4.58it/s, epoch=32, grad_norm=211.095, loss=331.4308, lr=3.56e-04]
Epoch 32 TRAIN:  68%|██████▊   | 93/137 [00:21<00:09,  4.58it/s, epoch=32, grad_norm=210.553, loss=350.4214, lr=3.56e-04]
Epoch 32 TRAIN:  70%|███████   | 96/137 [00:21<00:08,  4.89it/s, epoch=32, grad_norm=210.553, loss=350.4214, lr=3.56e-04]
Epoch 32 TRAIN:  70%|███████   | 96/137 [00:22<00:08,  4.89it/s, epoch=32, grad_norm=166.929, loss=314.8168, lr=3.57e-04]
Epoch 32 TRAIN:  70%|███████   | 96/137 [00:22<00:08,  4.89it/s, epoch=32, grad_norm=235.142, loss=211.1644, lr=3.57e-04]
Epoch 32 TRAIN:  70%|███████   | 96/137 [00:22<00:08,  4.89it/s, epoch=32, grad_norm=161.236, loss=238.0660, lr=3.57e-04]
Epoch 32 TRAIN:  72%|███████▏  | 99/137 [00:22<00:08,  4.64it/s, epoch=32, grad_norm=161.236, loss=238.0660, lr=3.57e-04]
Epoch 32 TRAIN:  72%|███████▏  | 99/137 [00:22<00:08,  4.64it/s, epoch=32, grad_norm=108.329, loss=342.5137, lr=3.57e-04]2026-04-16 20:08:09,091 DEBUG TRAIN | steps/sec 4.396| Batch 32/100 loss 342.513672 loss_att 302.734772 loss_ctc 435.331146 th_accuracy 0.306034 lr 3.5680e-04 grad_norm 108.329071 rank 0

Epoch 32 TRAIN:  72%|███████▏  | 99/137 [00:23<00:08,  4.64it/s, epoch=32, grad_norm=165.627, loss=265.3666, lr=3.57e-04]
Epoch 32 TRAIN:  72%|███████▏  | 99/137 [00:23<00:08,  4.64it/s, epoch=32, grad_norm=138.138, loss=334.9196, lr=3.57e-04]
Epoch 32 TRAIN:  74%|███████▍  | 102/137 [00:23<00:07,  4.75it/s, epoch=32, grad_norm=138.138, loss=334.9196, lr=3.57e-04]
Epoch 32 TRAIN:  74%|███████▍  | 102/137 [00:23<00:07,  4.75it/s, epoch=32, grad_norm=218.480, loss=380.0283, lr=3.57e-04]
Epoch 32 TRAIN:  74%|███████▍  | 102/137 [00:23<00:07,  4.75it/s, epoch=32, grad_norm=94.908, loss=300.2290, lr=3.57e-04] 
Epoch 32 TRAIN:  74%|███████▍  | 102/137 [00:23<00:07,  4.75it/s, epoch=32, grad_norm=163.949, loss=187.8129, lr=3.57e-04]
Epoch 32 TRAIN:  77%|███████▋  | 105/137 [00:23<00:06,  4.92it/s, epoch=32, grad_norm=163.949, loss=187.8129, lr=3.57e-04]
Epoch 32 TRAIN:  77%|███████▋  | 105/137 [00:24<00:06,  4.92it/s, epoch=32, grad_norm=174.369, loss=262.9754, lr=3.57e-04]
Epoch 32 TRAIN:  77%|███████▋  | 105/137 [00:24<00:06,  4.92it/s, epoch=32, grad_norm=131.061, loss=233.7463, lr=3.57e-04]
Epoch 32 TRAIN:  77%|███████▋  | 105/137 [00:24<00:06,  4.92it/s, epoch=32, grad_norm=94.156, loss=271.4182, lr=3.57e-04] 
Epoch 32 TRAIN:  79%|███████▉  | 108/137 [00:24<00:06,  4.71it/s, epoch=32, grad_norm=94.156, loss=271.4182, lr=3.57e-04]
Epoch 32 TRAIN:  79%|███████▉  | 108/137 [00:24<00:06,  4.71it/s, epoch=32, grad_norm=74.491, loss=292.7821, lr=3.58e-04]
Epoch 32 TRAIN:  79%|███████▉  | 108/137 [00:24<00:06,  4.71it/s, epoch=32, grad_norm=159.012, loss=317.0721, lr=3.58e-04]
Epoch 32 TRAIN:  79%|███████▉  | 108/137 [00:25<00:06,  4.71it/s, epoch=32, grad_norm=138.711, loss=293.8910, lr=3.58e-04]
Epoch 32 TRAIN:  81%|████████  | 111/137 [00:25<00:05,  4.65it/s, epoch=32, grad_norm=138.711, loss=293.8910, lr=3.58e-04]
Epoch 32 TRAIN:  81%|████████  | 111/137 [00:25<00:05,  4.65it/s, epoch=32, grad_norm=187.484, loss=356.7198, lr=3.58e-04]
Epoch 32 TRAIN:  81%|████████  | 111/137 [00:25<00:05,  4.65it/s, epoch=32, grad_norm=177.334, loss=354.6215, lr=3.58e-04]
Epoch 32 TRAIN:  81%|████████  | 111/137 [00:25<00:05,  4.65it/s, epoch=32, grad_norm=118.799, loss=284.6374, lr=3.58e-04]
Epoch 32 TRAIN:  83%|████████▎ | 114/137 [00:25<00:04,  4.83it/s, epoch=32, grad_norm=118.799, loss=284.6374, lr=3.58e-04]
Epoch 32 TRAIN:  83%|████████▎ | 114/137 [00:25<00:04,  4.83it/s, epoch=32, grad_norm=118.702, loss=356.4744, lr=3.58e-04]
Epoch 32 TRAIN:  83%|████████▎ | 114/137 [00:26<00:04,  4.83it/s, epoch=32, grad_norm=157.158, loss=284.8365, lr=3.58e-04]
Epoch 32 TRAIN:  83%|████████▎ | 114/137 [00:26<00:04,  4.83it/s, epoch=32, grad_norm=122.210, loss=395.6425, lr=3.58e-04]
Epoch 32 TRAIN:  85%|████████▌ | 117/137 [00:26<00:04,  4.95it/s, epoch=32, grad_norm=122.210, loss=395.6425, lr=3.58e-04]
Epoch 32 TRAIN:  85%|████████▌ | 117/137 [00:26<00:04,  4.95it/s, epoch=32, grad_norm=206.653, loss=271.8766, lr=3.58e-04]
Epoch 32 TRAIN:  85%|████████▌ | 117/137 [00:26<00:04,  4.95it/s, epoch=32, grad_norm=124.235, loss=320.0338, lr=3.58e-04]
Epoch 32 TRAIN:  85%|████████▌ | 117/137 [00:26<00:04,  4.95it/s, epoch=32, grad_norm=248.450, loss=312.7961, lr=3.58e-04]
Epoch 32 TRAIN:  88%|████████▊ | 120/137 [00:26<00:03,  5.12it/s, epoch=32, grad_norm=248.450, loss=312.7961, lr=3.58e-04]
Epoch 32 TRAIN:  88%|████████▊ | 120/137 [00:27<00:03,  5.12it/s, epoch=32, grad_norm=84.770, loss=338.1871, lr=3.58e-04] 
Epoch 32 TRAIN:  88%|████████▊ | 120/137 [00:27<00:03,  5.12it/s, epoch=32, grad_norm=116.931, loss=308.1400, lr=3.59e-04]
Epoch 32 TRAIN:  88%|████████▊ | 120/137 [00:27<00:03,  5.12it/s, epoch=32, grad_norm=121.016, loss=315.7343, lr=3.59e-04]
Epoch 32 TRAIN:  90%|████████▉ | 123/137 [00:27<00:02,  4.93it/s, epoch=32, grad_norm=121.016, loss=315.7343, lr=3.59e-04]
Epoch 32 TRAIN:  90%|████████▉ | 123/137 [00:27<00:02,  4.93it/s, epoch=32, grad_norm=170.388, loss=412.2532, lr=3.59e-04]
Epoch 32 TRAIN:  90%|████████▉ | 123/137 [00:27<00:02,  4.93it/s, epoch=32, grad_norm=136.952, loss=366.4301, lr=3.59e-04]
Epoch 32 TRAIN:  90%|████████▉ | 123/137 [00:28<00:02,  4.93it/s, epoch=32, grad_norm=112.589, loss=368.2398, lr=3.59e-04]
Epoch 32 TRAIN:  92%|█████████▏| 126/137 [00:28<00:02,  4.72it/s, epoch=32, grad_norm=112.589, loss=368.2398, lr=3.59e-04]
Epoch 32 TRAIN:  92%|█████████▏| 126/137 [00:28<00:02,  4.72it/s, epoch=32, grad_norm=129.664, loss=343.1019, lr=3.59e-04]
Epoch 32 TRAIN:  92%|█████████▏| 126/137 [00:28<00:02,  4.72it/s, epoch=32, grad_norm=141.737, loss=426.0282, lr=3.59e-04]
Epoch 32 TRAIN:  92%|█████████▏| 126/137 [00:28<00:02,  4.72it/s, epoch=32, grad_norm=100.309, loss=362.4909, lr=3.59e-04]
Epoch 32 TRAIN:  94%|█████████▍| 129/137 [00:28<00:01,  5.03it/s, epoch=32, grad_norm=100.309, loss=362.4909, lr=3.59e-04]
Epoch 32 TRAIN:  94%|█████████▍| 129/137 [00:28<00:01,  5.03it/s, epoch=32, grad_norm=125.001, loss=405.0397, lr=3.59e-04]
Epoch 32 TRAIN:  94%|█████████▍| 129/137 [00:29<00:01,  5.03it/s, epoch=32, grad_norm=187.584, loss=251.3526, lr=3.59e-04]
Epoch 32 TRAIN:  94%|█████████▍| 129/137 [00:29<00:01,  5.03it/s, epoch=32, grad_norm=193.843, loss=328.7895, lr=3.59e-04]
Epoch 32 TRAIN:  94%|█████████▍| 129/137 [00:29<00:01,  5.03it/s, epoch=32, grad_norm=148.815, loss=346.7210, lr=3.59e-04]
Epoch 32 TRAIN:  97%|█████████▋| 133/137 [00:29<00:00,  5.53it/s, epoch=32, grad_norm=148.815, loss=346.7210, lr=3.59e-04]
Epoch 32 TRAIN:  97%|█████████▋| 133/137 [00:29<00:00,  5.53it/s, epoch=32, grad_norm=224.783, loss=412.3865, lr=3.60e-04]
Epoch 32 TRAIN:  97%|█████████▋| 133/137 [00:29<00:00,  5.53it/s, epoch=32, grad_norm=119.394, loss=340.6844, lr=3.60e-04]
Epoch 32 TRAIN:  97%|█████████▋| 133/137 [00:29<00:00,  5.53it/s, epoch=32, grad_norm=140.995, loss=293.7041, lr=3.60e-04]
Epoch 32 TRAIN:  97%|█████████▋| 133/137 [00:29<00:00,  5.53it/s, epoch=32, grad_norm=205.421, loss=404.5141, lr=3.60e-04]
Epoch 32 TRAIN: 100%|██████████| 137/137 [00:29<00:00,  6.00it/s, epoch=32, grad_norm=205.421, loss=404.5141, lr=3.60e-04]
Epoch 32 TRAIN: 100%|██████████| 137/137 [00:29<00:00,  4.59it/s, epoch=32, grad_norm=205.421, loss=404.5141, lr=3.60e-04]

CV:   0%|          | 0/6 [00:00<?, ?it/s]/home/rookie/Documents/batuka/projects/wenet/wenet/wenet/utils/train_utils.py:689: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  amp_autocast(enabled=dtype is not None,

CV:  17%|█▋        | 1/6 [00:00<00:03,  1.32it/s]
                                                 
2026-04-16 20:08:16,969 INFO Epoch 32 Step 4497 CV info lr 3.5976e-04 cv_loss 543.3965010209517 rank 0 acc 0.2510703057050705
2026-04-16 20:08:16,971 INFO [Rank 0] Checkpoint: save to checkpoint exp/conformer_mongolian/epoch_32.pt
2026-04-16 20:08:17,093 INFO Epoch 33 Step 4497 TRAIN info lr 3.5976e-04 rank 0
2026-04-16 20:08:17,110 INFO using accumulate grad, new batch size is 1 times larger than before

Epoch 33 TRAIN:   0%|          | 0/137 [00:00<?, ?it/s]
Epoch 33 TRAIN:   0%|          | 0/137 [00:04<?, ?it/s, epoch=33, grad_norm=432.653, loss=380.4128, lr=3.60e-04]
Epoch 33 TRAIN:   1%|          | 1/137 [00:04<09:36,  4.24s/it, epoch=33, grad_norm=432.653, loss=380.4128, lr=3.60e-04]
Epoch 33 TRAIN:   1%|          | 1/137 [00:04<09:36,  4.24s/it, epoch=33, grad_norm=413.999, loss=341.1279, lr=3.60e-04]
Epoch 33 TRAIN:   1%|          | 1/137 [00:04<09:36,  4.24s/it, epoch=33, grad_norm=531.952, loss=439.2413, lr=3.60e-04]
Epoch 33 TRAIN:   1%|          | 1/137 [00:04<09:36,  4.24s/it, epoch=33, grad_norm=289.114, loss=277.4638, lr=3.60e-04]
Epoch 33 TRAIN:   1%|          | 1/137 [00:04<09:36,  4.24s/it, epoch=33, grad_norm=596.854, loss=503.4670, lr=3.60e-04]
Epoch 33 TRAIN:   1%|          | 1/137 [00:04<09:36,  4.24s/it, epoch=33, grad_norm=258.097, loss=292.8058, lr=3.60e-04]
Epoch 33 TRAIN:   1%|          | 1/137 [00:04<09:36,  4.24s/it, epoch=33, grad_norm=374.617, loss=444.1225, lr=3.60e-04]
Epoch 33 TRAIN:   1%|          | 1/137 [00:04<09:36,  4.24s/it, epoch=33, grad_norm=353.338, loss=372.2746, lr=3.60e-04]
Epoch 33 TRAIN:   1%|          | 1/137 [00:04<09:36,  4.24s/it, epoch=33, grad_norm=306.802, loss=377.2531, lr=3.60e-04]
Epoch 33 TRAIN:   7%|▋         | 9/137 [00:04<00:52,  2.44it/s, epoch=33, grad_norm=306.802, loss=377.2531, lr=3.60e-04]
Epoch 33 TRAIN:   7%|▋         | 9/137 [00:04<00:52,  2.44it/s, epoch=33, grad_norm=131.276, loss=275.4339, lr=3.61e-04]
Epoch 33 TRAIN:   7%|▋         | 9/137 [00:05<00:52,  2.44it/s, epoch=33, grad_norm=142.443, loss=275.4850, lr=3.61e-04]
Epoch 33 TRAIN:   7%|▋         | 9/137 [00:05<00:52,  2.44it/s, epoch=33, grad_norm=107.464, loss=307.7980, lr=3.61e-04]
Epoch 33 TRAIN:   7%|▋         | 9/137 [00:05<00:52,  2.44it/s, epoch=33, grad_norm=78.389, loss=323.6760, lr=3.61e-04] 
Epoch 33 TRAIN:   9%|▉         | 13/137 [00:05<00:38,  3.22it/s, epoch=33, grad_norm=78.389, loss=323.6760, lr=3.61e-04]
Epoch 33 TRAIN:   9%|▉         | 13/137 [00:05<00:38,  3.22it/s, epoch=33, grad_norm=308.098, loss=329.6200, lr=3.61e-04]
Epoch 33 TRAIN:   9%|▉         | 13/137 [00:05<00:38,  3.22it/s, epoch=33, grad_norm=59.426, loss=354.8500, lr=3.61e-04] 
Epoch 33 TRAIN:   9%|▉         | 13/137 [00:05<00:38,  3.22it/s, epoch=33, grad_norm=288.640, loss=301.6462, lr=3.61e-04]
Epoch 33 TRAIN:   9%|▉         | 13/137 [00:06<00:38,  3.22it/s, epoch=33, grad_norm=193.775, loss=306.5396, lr=3.61e-04]
Epoch 33 TRAIN:  12%|█▏        | 17/137 [00:06<00:30,  3.93it/s, epoch=33, grad_norm=193.775, loss=306.5396, lr=3.61e-04]
Epoch 33 TRAIN:  12%|█▏        | 17/137 [00:06<00:30,  3.93it/s, epoch=33, grad_norm=72.397, loss=313.2755, lr=3.61e-04] 
Epoch 33 TRAIN:  12%|█▏        | 17/137 [00:06<00:30,  3.93it/s, epoch=33, grad_norm=60.829, loss=282.9659, lr=3.61e-04]
Epoch 33 TRAIN:  12%|█▏        | 17/137 [00:06<00:30,  3.93it/s, epoch=33, grad_norm=177.821, loss=358.4825, lr=3.61e-04]
Epoch 33 TRAIN:  12%|█▏        | 17/137 [00:06<00:30,  3.93it/s, epoch=33, grad_norm=165.480, loss=355.7248, lr=3.61e-04]
Epoch 33 TRAIN:  15%|█▌        | 21/137 [00:06<00:27,  4.28it/s, epoch=33, grad_norm=165.480, loss=355.7248, lr=3.61e-04]
Epoch 33 TRAIN:  15%|█▌        | 21/137 [00:07<00:27,  4.28it/s, epoch=33, grad_norm=187.758, loss=377.8605, lr=3.62e-04]
Epoch 33 TRAIN:  15%|█▌        | 21/137 [00:07<00:27,  4.28it/s, epoch=33, grad_norm=120.999, loss=342.4523, lr=3.62e-04]
Epoch 33 TRAIN:  15%|█▌        | 21/137 [00:07<00:27,  4.28it/s, epoch=33, grad_norm=55.302, loss=292.7224, lr=3.62e-04] 
Epoch 33 TRAIN:  15%|█▌        | 21/137 [00:07<00:27,  4.28it/s, epoch=33, grad_norm=81.517, loss=301.0579, lr=3.62e-04]
Epoch 33 TRAIN:  18%|█▊        | 25/137 [00:07<00:24,  4.62it/s, epoch=33, grad_norm=81.517, loss=301.0579, lr=3.62e-04]
Epoch 33 TRAIN:  18%|█▊        | 25/137 [00:07<00:24,  4.62it/s, epoch=33, grad_norm=67.600, loss=317.0552, lr=3.62e-04]
Epoch 33 TRAIN:  18%|█▊        | 25/137 [00:07<00:24,  4.62it/s, epoch=33, grad_norm=115.734, loss=293.6329, lr=3.62e-04]
Epoch 33 TRAIN:  18%|█▊        | 25/137 [00:08<00:24,  4.62it/s, epoch=33, grad_norm=73.017, loss=314.0538, lr=3.62e-04] 
Epoch 33 TRAIN:  18%|█▊        | 25/137 [00:08<00:24,  4.62it/s, epoch=33, grad_norm=82.950, loss=308.5159, lr=3.62e-04]
Epoch 33 TRAIN:  21%|██        | 29/137 [00:08<00:22,  4.83it/s, epoch=33, grad_norm=82.950, loss=308.5159, lr=3.62e-04]
Epoch 33 TRAIN:  21%|██        | 29/137 [00:08<00:22,  4.83it/s, epoch=33, grad_norm=74.046, loss=317.1559, lr=3.62e-04]
Epoch 33 TRAIN:  21%|██        | 29/137 [00:08<00:22,  4.83it/s, epoch=33, grad_norm=103.932, loss=346.4894, lr=3.62e-04]
Epoch 33 TRAIN:  21%|██        | 29/137 [00:08<00:22,  4.83it/s, epoch=33, grad_norm=89.776, loss=315.3317, lr=3.62e-04] 
Epoch 33 TRAIN:  21%|██        | 29/137 [00:09<00:22,  4.83it/s, epoch=33, grad_norm=64.846, loss=298.7025, lr=3.62e-04]
Epoch 33 TRAIN:  24%|██▍       | 33/137 [00:09<00:20,  5.14it/s, epoch=33, grad_norm=64.846, loss=298.7025, lr=3.62e-04]
Epoch 33 TRAIN:  24%|██▍       | 33/137 [00:09<00:20,  5.14it/s, epoch=33, grad_norm=65.584, loss=310.9261, lr=3.62e-04]
Epoch 33 TRAIN:  24%|██▍       | 33/137 [00:09<00:20,  5.14it/s, epoch=33, grad_norm=105.412, loss=348.3243, lr=3.63e-04]
Epoch 33 TRAIN:  24%|██▍       | 33/137 [00:09<00:20,  5.14it/s, epoch=33, grad_norm=65.235, loss=309.4651, lr=3.63e-04] 
Epoch 33 TRAIN:  26%|██▋       | 36/137 [00:09<00:19,  5.12it/s, epoch=33, grad_norm=65.235, loss=309.4651, lr=3.63e-04]
Epoch 33 TRAIN:  26%|██▋       | 36/137 [00:09<00:19,  5.12it/s, epoch=33, grad_norm=95.995, loss=346.2620, lr=3.63e-04]
Epoch 33 TRAIN:  26%|██▋       | 36/137 [00:09<00:19,  5.12it/s, epoch=33, grad_norm=69.194, loss=309.6974, lr=3.63e-04]
Epoch 33 TRAIN:  26%|██▋       | 36/137 [00:10<00:19,  5.12it/s, epoch=33, grad_norm=71.277, loss=339.1735, lr=3.63e-04]
Epoch 33 TRAIN:  28%|██▊       | 39/137 [00:10<00:18,  5.26it/s, epoch=33, grad_norm=71.277, loss=339.1735, lr=3.63e-04]
Epoch 33 TRAIN:  28%|██▊       | 39/137 [00:10<00:18,  5.26it/s, epoch=33, grad_norm=124.790, loss=280.4472, lr=3.63e-04]
Epoch 33 TRAIN:  28%|██▊       | 39/137 [00:10<00:18,  5.26it/s, epoch=33, grad_norm=115.733, loss=297.7866, lr=3.63e-04]
Epoch 33 TRAIN:  28%|██▊       | 39/137 [00:10<00:18,  5.26it/s, epoch=33, grad_norm=201.211, loss=252.8216, lr=3.63e-04]
Epoch 33 TRAIN:  31%|███       | 42/137 [00:10<00:18,  5.05it/s, epoch=33, grad_norm=201.211, loss=252.8216, lr=3.63e-04]
Epoch 33 TRAIN:  31%|███       | 42/137 [00:10<00:18,  5.05it/s, epoch=33, grad_norm=89.549, loss=346.3652, lr=3.63e-04] 
Epoch 33 TRAIN:  31%|███       | 42/137 [00:11<00:18,  5.05it/s, epoch=33, grad_norm=93.892, loss=335.6254, lr=3.63e-04]
Epoch 33 TRAIN:  31%|███       | 42/137 [00:11<00:18,  5.05it/s, epoch=33, grad_norm=60.942, loss=302.7732, lr=3.63e-04]
Epoch 33 TRAIN:  33%|███▎      | 45/137 [00:11<00:18,  4.98it/s, epoch=33, grad_norm=60.942, loss=302.7732, lr=3.63e-04]
Epoch 33 TRAIN:  33%|███▎      | 45/137 [00:11<00:18,  4.98it/s, epoch=33, grad_norm=93.079, loss=316.9595, lr=3.63e-04]
Epoch 33 TRAIN:  33%|███▎      | 45/137 [00:11<00:18,  4.98it/s, epoch=33, grad_norm=109.788, loss=308.0240, lr=3.64e-04]
Epoch 33 TRAIN:  33%|███▎      | 45/137 [00:12<00:18,  4.98it/s, epoch=33, grad_norm=101.450, loss=321.9417, lr=3.64e-04]
Epoch 33 TRAIN:  35%|███▌      | 48/137 [00:12<00:17,  4.98it/s, epoch=33, grad_norm=101.450, loss=321.9417, lr=3.64e-04]
Epoch 33 TRAIN:  35%|███▌      | 48/137 [00:12<00:17,  4.98it/s, epoch=33, grad_norm=85.441, loss=298.2018, lr=3.64e-04] 
Epoch 33 TRAIN:  35%|███▌      | 48/137 [00:12<00:17,  4.98it/s, epoch=33, grad_norm=106.374, loss=347.6735, lr=3.64e-04]
Epoch 33 TRAIN:  35%|███▌      | 48/137 [00:12<00:17,  4.98it/s, epoch=33, grad_norm=83.373, loss=309.2797, lr=3.64e-04] 
Epoch 33 TRAIN:  37%|███▋      | 51/137 [00:12<00:17,  5.01it/s, epoch=33, grad_norm=83.373, loss=309.2797, lr=3.64e-04]
Epoch 33 TRAIN:  37%|███▋      | 51/137 [00:12<00:17,  5.01it/s, epoch=33, grad_norm=114.868, loss=321.0770, lr=3.64e-04]
Epoch 33 TRAIN:  37%|███▋      | 51/137 [00:13<00:17,  5.01it/s, epoch=33, grad_norm=71.364, loss=343.0474, lr=3.64e-04] 
Epoch 33 TRAIN:  37%|███▋      | 51/137 [00:13<00:17,  5.01it/s, epoch=33, grad_norm=64.081, loss=351.8361, lr=3.64e-04]
Epoch 33 TRAIN:  39%|███▉      | 54/137 [00:13<00:16,  4.96it/s, epoch=33, grad_norm=64.081, loss=351.8361, lr=3.64e-04]
Epoch 33 TRAIN:  39%|███▉      | 54/137 [00:13<00:16,  4.96it/s, epoch=33, grad_norm=86.798, loss=311.3563, lr=3.64e-04]
Epoch 33 TRAIN:  39%|███▉      | 54/137 [00:13<00:16,  4.96it/s, epoch=33, grad_norm=91.295, loss=328.8806, lr=3.64e-04]
Epoch 33 TRAIN:  39%|███▉      | 54/137 [00:13<00:16,  4.96it/s, epoch=33, grad_norm=100.856, loss=347.4163, lr=3.64e-04]
Epoch 33 TRAIN:  42%|████▏     | 57/137 [00:13<00:16,  4.90it/s, epoch=33, grad_norm=100.856, loss=347.4163, lr=3.64e-04]
Epoch 33 TRAIN:  42%|████▏     | 57/137 [00:14<00:16,  4.90it/s, epoch=33, grad_norm=123.925, loss=321.6377, lr=3.64e-04]
Epoch 33 TRAIN:  42%|████▏     | 57/137 [00:14<00:16,  4.90it/s, epoch=33, grad_norm=82.911, loss=340.6861, lr=3.64e-04] 
Epoch 33 TRAIN:  42%|████▏     | 57/137 [00:14<00:16,  4.90it/s, epoch=33, grad_norm=106.557, loss=352.2978, lr=3.65e-04]
Epoch 33 TRAIN:  44%|████▍     | 60/137 [00:14<00:15,  4.85it/s, epoch=33, grad_norm=106.557, loss=352.2978, lr=3.65e-04]
Epoch 33 TRAIN:  44%|████▍     | 60/137 [00:14<00:15,  4.85it/s, epoch=33, grad_norm=166.957, loss=379.5694, lr=3.65e-04]
Epoch 33 TRAIN:  44%|████▍     | 60/137 [00:14<00:15,  4.85it/s, epoch=33, grad_norm=78.265, loss=278.0465, lr=3.65e-04] 
Epoch 33 TRAIN:  44%|████▍     | 60/137 [00:15<00:15,  4.85it/s, epoch=33, grad_norm=136.701, loss=325.3517, lr=3.65e-04]
Epoch 33 TRAIN:  46%|████▌     | 63/137 [00:15<00:15,  4.85it/s, epoch=33, grad_norm=136.701, loss=325.3517, lr=3.65e-04]
Epoch 33 TRAIN:  46%|████▌     | 63/137 [00:15<00:15,  4.85it/s, epoch=33, grad_norm=138.772, loss=319.2847, lr=3.65e-04]
Epoch 33 TRAIN:  46%|████▌     | 63/137 [00:15<00:15,  4.85it/s, epoch=33, grad_norm=112.160, loss=329.6054, lr=3.65e-04]
Epoch 33 TRAIN:  46%|████▌     | 63/137 [00:15<00:15,  4.85it/s, epoch=33, grad_norm=138.468, loss=361.0342, lr=3.65e-04]
Epoch 33 TRAIN:  48%|████▊     | 66/137 [00:15<00:14,  4.88it/s, epoch=33, grad_norm=138.468, loss=361.0342, lr=3.65e-04]
Epoch 33 TRAIN:  48%|████▊     | 66/137 [00:15<00:14,  4.88it/s, epoch=33, grad_norm=130.351, loss=260.0480, lr=3.65e-04]
Epoch 33 TRAIN:  48%|████▊     | 66/137 [00:16<00:14,  4.88it/s, epoch=33, grad_norm=123.669, loss=304.1956, lr=3.65e-04]
Epoch 33 TRAIN:  48%|████▊     | 66/137 [00:16<00:14,  4.88it/s, epoch=33, grad_norm=98.756, loss=326.1617, lr=3.65e-04] 
Epoch 33 TRAIN:  50%|█████     | 69/137 [00:16<00:13,  4.95it/s, epoch=33, grad_norm=98.756, loss=326.1617, lr=3.65e-04]
Epoch 33 TRAIN:  50%|█████     | 69/137 [00:16<00:13,  4.95it/s, epoch=33, grad_norm=71.144, loss=331.3121, lr=3.65e-04]
Epoch 33 TRAIN:  50%|█████     | 69/137 [00:16<00:13,  4.95it/s, epoch=33, grad_norm=89.188, loss=309.4078, lr=3.65e-04]
Epoch 33 TRAIN:  50%|█████     | 69/137 [00:16<00:13,  4.95it/s, epoch=33, grad_norm=112.421, loss=358.6130, lr=3.66e-04]
Epoch 33 TRAIN:  53%|█████▎    | 72/137 [00:16<00:13,  4.89it/s, epoch=33, grad_norm=112.421, loss=358.6130, lr=3.66e-04]
Epoch 33 TRAIN:  53%|█████▎    | 72/137 [00:17<00:13,  4.89it/s, epoch=33, grad_norm=88.617, loss=297.1839, lr=3.66e-04] 
Epoch 33 TRAIN:  53%|█████▎    | 72/137 [00:17<00:13,  4.89it/s, epoch=33, grad_norm=150.342, loss=300.4165, lr=3.66e-04]
Epoch 33 TRAIN:  53%|█████▎    | 72/137 [00:17<00:13,  4.89it/s, epoch=33, grad_norm=95.891, loss=376.0592, lr=3.66e-04] 
Epoch 33 TRAIN:  55%|█████▍    | 75/137 [00:17<00:12,  5.03it/s, epoch=33, grad_norm=95.891, loss=376.0592, lr=3.66e-04]
Epoch 33 TRAIN:  55%|█████▍    | 75/137 [00:17<00:12,  5.03it/s, epoch=33, grad_norm=126.055, loss=384.7441, lr=3.66e-04]
Epoch 33 TRAIN:  55%|█████▍    | 75/137 [00:17<00:12,  5.03it/s, epoch=33, grad_norm=119.536, loss=284.0435, lr=3.66e-04]
Epoch 33 TRAIN:  55%|█████▍    | 75/137 [00:18<00:12,  5.03it/s, epoch=33, grad_norm=102.320, loss=341.1395, lr=3.66e-04]
Epoch 33 TRAIN:  57%|█████▋    | 78/137 [00:18<00:11,  5.05it/s, epoch=33, grad_norm=102.320, loss=341.1395, lr=3.66e-04]
Epoch 33 TRAIN:  57%|█████▋    | 78/137 [00:18<00:11,  5.05it/s, epoch=33, grad_norm=169.380, loss=398.5182, lr=3.66e-04]
Epoch 33 TRAIN:  57%|█████▋    | 78/137 [00:18<00:11,  5.05it/s, epoch=33, grad_norm=102.059, loss=296.3598, lr=3.66e-04]
Epoch 33 TRAIN:  57%|█████▋    | 78/137 [00:18<00:11,  5.05it/s, epoch=33, grad_norm=244.106, loss=243.2327, lr=3.66e-04]
Epoch 33 TRAIN:  59%|█████▉    | 81/137 [00:18<00:11,  4.99it/s, epoch=33, grad_norm=244.106, loss=243.2327, lr=3.66e-04]
Epoch 33 TRAIN:  59%|█████▉    | 81/137 [00:18<00:11,  4.99it/s, epoch=33, grad_norm=112.128, loss=296.2198, lr=3.66e-04]
Epoch 33 TRAIN:  59%|█████▉    | 81/137 [00:19<00:11,  4.99it/s, epoch=33, grad_norm=127.099, loss=276.2816, lr=3.66e-04]
Epoch 33 TRAIN:  59%|█████▉    | 81/137 [00:19<00:11,  4.99it/s, epoch=33, grad_norm=109.019, loss=270.7094, lr=3.66e-04]
Epoch 33 TRAIN:  61%|██████▏   | 84/137 [00:19<00:11,  4.67it/s, epoch=33, grad_norm=109.019, loss=270.7094, lr=3.66e-04]
Epoch 33 TRAIN:  61%|██████▏   | 84/137 [00:19<00:11,  4.67it/s, epoch=33, grad_norm=250.324, loss=348.6012, lr=3.67e-04]
Epoch 33 TRAIN:  61%|██████▏   | 84/137 [00:19<00:11,  4.67it/s, epoch=33, grad_norm=96.811, loss=226.2748, lr=3.67e-04] 
Epoch 33 TRAIN:  61%|██████▏   | 84/137 [00:19<00:11,  4.67it/s, epoch=33, grad_norm=215.814, loss=288.4140, lr=3.67e-04]
Epoch 33 TRAIN:  64%|██████▎   | 87/137 [00:19<00:10,  4.84it/s, epoch=33, grad_norm=215.814, loss=288.4140, lr=3.67e-04]
Epoch 33 TRAIN:  64%|██████▎   | 87/137 [00:20<00:10,  4.84it/s, epoch=33, grad_norm=124.424, loss=274.8405, lr=3.67e-04]
Epoch 33 TRAIN:  64%|██████▎   | 87/137 [00:20<00:10,  4.84it/s, epoch=33, grad_norm=87.591, loss=298.6829, lr=3.67e-04] 
Epoch 33 TRAIN:  64%|██████▎   | 87/137 [00:20<00:10,  4.84it/s, epoch=33, grad_norm=87.273, loss=248.7343, lr=3.67e-04]
Epoch 33 TRAIN:  66%|██████▌   | 90/137 [00:20<00:09,  4.93it/s, epoch=33, grad_norm=87.273, loss=248.7343, lr=3.67e-04]
Epoch 33 TRAIN:  66%|██████▌   | 90/137 [00:20<00:09,  4.93it/s, epoch=33, grad_norm=93.799, loss=342.8326, lr=3.67e-04]
Epoch 33 TRAIN:  66%|██████▌   | 90/137 [00:21<00:09,  4.93it/s, epoch=33, grad_norm=165.362, loss=259.6808, lr=3.67e-04]
Epoch 33 TRAIN:  66%|██████▌   | 90/137 [00:21<00:09,  4.93it/s, epoch=33, grad_norm=261.411, loss=398.2779, lr=3.67e-04]
Epoch 33 TRAIN:  68%|██████▊   | 93/137 [00:21<00:08,  4.89it/s, epoch=33, grad_norm=261.411, loss=398.2779, lr=3.67e-04]
Epoch 33 TRAIN:  68%|██████▊   | 93/137 [00:21<00:08,  4.89it/s, epoch=33, grad_norm=79.068, loss=332.5945, lr=3.67e-04] 
Epoch 33 TRAIN:  68%|██████▊   | 93/137 [00:21<00:08,  4.89it/s, epoch=33, grad_norm=70.115, loss=235.7229, lr=3.67e-04]
Epoch 33 TRAIN:  68%|██████▊   | 93/137 [00:21<00:08,  4.89it/s, epoch=33, grad_norm=73.850, loss=290.6947, lr=3.67e-04]
Epoch 33 TRAIN:  70%|███████   | 96/137 [00:21<00:08,  4.90it/s, epoch=33, grad_norm=73.850, loss=290.6947, lr=3.67e-04]
Epoch 33 TRAIN:  70%|███████   | 96/137 [00:22<00:08,  4.90it/s, epoch=33, grad_norm=224.583, loss=253.4467, lr=3.68e-04]
Epoch 33 TRAIN:  70%|███████   | 96/137 [00:22<00:08,  4.90it/s, epoch=33, grad_norm=101.789, loss=267.9842, lr=3.68e-04]
Epoch 33 TRAIN:  70%|███████   | 96/137 [00:22<00:08,  4.90it/s, epoch=33, grad_norm=85.552, loss=359.4026, lr=3.68e-04] 
Epoch 33 TRAIN:  72%|███████▏  | 99/137 [00:22<00:07,  4.79it/s, epoch=33, grad_norm=85.552, loss=359.4026, lr=3.68e-04]
Epoch 33 TRAIN:  72%|███████▏  | 99/137 [00:22<00:07,  4.79it/s, epoch=33, grad_norm=109.285, loss=261.7050, lr=3.68e-04]2026-04-16 20:08:39,782 DEBUG TRAIN | steps/sec 4.464| Batch 33/100 loss 261.705017 loss_att 230.496643 loss_ctc 334.524628 th_accuracy 0.312155 lr 3.6776e-04 grad_norm 109.285149 rank 0

Epoch 33 TRAIN:  72%|███████▏  | 99/137 [00:22<00:07,  4.79it/s, epoch=33, grad_norm=240.958, loss=368.9291, lr=3.68e-04]
Epoch 33 TRAIN:  72%|███████▏  | 99/137 [00:23<00:07,  4.79it/s, epoch=33, grad_norm=82.370, loss=280.7866, lr=3.68e-04] 
Epoch 33 TRAIN:  74%|███████▍  | 102/137 [00:23<00:07,  4.84it/s, epoch=33, grad_norm=82.370, loss=280.7866, lr=3.68e-04]
Epoch 33 TRAIN:  74%|███████▍  | 102/137 [00:23<00:07,  4.84it/s, epoch=33, grad_norm=263.101, loss=342.8827, lr=3.68e-04]
Epoch 33 TRAIN:  74%|███████▍  | 102/137 [00:23<00:07,  4.84it/s, epoch=33, grad_norm=87.430, loss=224.7955, lr=3.68e-04] 
Epoch 33 TRAIN:  74%|███████▍  | 102/137 [00:23<00:07,  4.84it/s, epoch=33, grad_norm=161.765, loss=372.1205, lr=3.68e-04]
Epoch 33 TRAIN:  77%|███████▋  | 105/137 [00:23<00:06,  4.75it/s, epoch=33, grad_norm=161.765, loss=372.1205, lr=3.68e-04]
Epoch 33 TRAIN:  77%|███████▋  | 105/137 [00:23<00:06,  4.75it/s, epoch=33, grad_norm=92.887, loss=262.0989, lr=3.68e-04] 
Epoch 33 TRAIN:  77%|███████▋  | 105/137 [00:24<00:06,  4.75it/s, epoch=33, grad_norm=88.521, loss=331.6573, lr=3.68e-04]
Epoch 33 TRAIN:  77%|███████▋  | 105/137 [00:24<00:06,  4.75it/s, epoch=33, grad_norm=77.480, loss=299.5906, lr=3.68e-04]
Epoch 33 TRAIN:  79%|███████▉  | 108/137 [00:24<00:06,  4.80it/s, epoch=33, grad_norm=77.480, loss=299.5906, lr=3.68e-04]
Epoch 33 TRAIN:  79%|███████▉  | 108/137 [00:24<00:06,  4.80it/s, epoch=33, grad_norm=169.865, loss=357.8931, lr=3.68e-04]
Epoch 33 TRAIN:  79%|███████▉  | 108/137 [00:24<00:06,  4.80it/s, epoch=33, grad_norm=90.690, loss=244.5250, lr=3.69e-04] 
Epoch 33 TRAIN:  79%|███████▉  | 108/137 [00:24<00:06,  4.80it/s, epoch=33, grad_norm=336.193, loss=394.7077, lr=3.69e-04]
Epoch 33 TRAIN:  81%|████████  | 111/137 [00:24<00:05,  4.82it/s, epoch=33, grad_norm=336.193, loss=394.7077, lr=3.69e-04]
Epoch 33 TRAIN:  81%|████████  | 111/137 [00:25<00:05,  4.82it/s, epoch=33, grad_norm=147.031, loss=345.5367, lr=3.69e-04]
Epoch 33 TRAIN:  81%|████████  | 111/137 [00:25<00:05,  4.82it/s, epoch=33, grad_norm=104.748, loss=351.8112, lr=3.69e-04]
Epoch 33 TRAIN:  81%|████████  | 111/137 [00:25<00:05,  4.82it/s, epoch=33, grad_norm=83.517, loss=296.2349, lr=3.69e-04] 
Epoch 33 TRAIN:  83%|████████▎ | 114/137 [00:25<00:05,  4.52it/s, epoch=33, grad_norm=83.517, loss=296.2349, lr=3.69e-04]
Epoch 33 TRAIN:  83%|████████▎ | 114/137 [00:25<00:05,  4.52it/s, epoch=33, grad_norm=168.251, loss=340.9801, lr=3.69e-04]
Epoch 33 TRAIN:  83%|████████▎ | 114/137 [00:26<00:05,  4.52it/s, epoch=33, grad_norm=164.919, loss=320.2913, lr=3.69e-04]
Epoch 33 TRAIN:  83%|████████▎ | 114/137 [00:26<00:05,  4.52it/s, epoch=33, grad_norm=149.085, loss=273.8588, lr=3.69e-04]
Epoch 33 TRAIN:  85%|████████▌ | 117/137 [00:26<00:04,  4.78it/s, epoch=33, grad_norm=149.085, loss=273.8588, lr=3.69e-04]
Epoch 33 TRAIN:  85%|████████▌ | 117/137 [00:26<00:04,  4.78it/s, epoch=33, grad_norm=133.971, loss=295.1021, lr=3.69e-04]
Epoch 33 TRAIN:  85%|████████▌ | 117/137 [00:26<00:04,  4.78it/s, epoch=33, grad_norm=164.787, loss=320.7612, lr=3.69e-04]
Epoch 33 TRAIN:  85%|████████▌ | 117/137 [00:26<00:04,  4.78it/s, epoch=33, grad_norm=111.499, loss=327.2710, lr=3.69e-04]
Epoch 33 TRAIN:  88%|████████▊ | 120/137 [00:26<00:03,  4.70it/s, epoch=33, grad_norm=111.499, loss=327.2710, lr=3.69e-04]
Epoch 33 TRAIN:  88%|████████▊ | 120/137 [00:27<00:03,  4.70it/s, epoch=33, grad_norm=122.752, loss=355.3217, lr=3.69e-04]
Epoch 33 TRAIN:  88%|████████▊ | 120/137 [00:27<00:03,  4.70it/s, epoch=33, grad_norm=179.871, loss=310.9158, lr=3.70e-04]
Epoch 33 TRAIN:  88%|████████▊ | 120/137 [00:27<00:03,  4.70it/s, epoch=33, grad_norm=131.298, loss=281.4596, lr=3.70e-04]
Epoch 33 TRAIN:  90%|████████▉ | 123/137 [00:27<00:03,  4.50it/s, epoch=33, grad_norm=131.298, loss=281.4596, lr=3.70e-04]
Epoch 33 TRAIN:  90%|████████▉ | 123/137 [00:27<00:03,  4.50it/s, epoch=33, grad_norm=143.147, loss=216.7754, lr=3.70e-04]
Epoch 33 TRAIN:  90%|████████▉ | 123/137 [00:27<00:03,  4.50it/s, epoch=33, grad_norm=162.535, loss=320.1080, lr=3.70e-04]
Epoch 33 TRAIN:  90%|████████▉ | 123/137 [00:28<00:03,  4.50it/s, epoch=33, grad_norm=144.226, loss=303.0421, lr=3.70e-04]
Epoch 33 TRAIN:  92%|█████████▏| 126/137 [00:28<00:02,  4.80it/s, epoch=33, grad_norm=144.226, loss=303.0421, lr=3.70e-04]
Epoch 33 TRAIN:  92%|█████████▏| 126/137 [00:28<00:02,  4.80it/s, epoch=33, grad_norm=236.736, loss=402.2284, lr=3.70e-04]
Epoch 33 TRAIN:  92%|█████████▏| 126/137 [00:28<00:02,  4.80it/s, epoch=33, grad_norm=111.748, loss=285.3284, lr=3.70e-04]
Epoch 33 TRAIN:  92%|█████████▏| 126/137 [00:28<00:02,  4.80it/s, epoch=33, grad_norm=273.406, loss=252.1251, lr=3.70e-04]
Epoch 33 TRAIN:  92%|█████████▏| 126/137 [00:28<00:02,  4.80it/s, epoch=33, grad_norm=140.101, loss=327.2714, lr=3.70e-04]
Epoch 33 TRAIN:  95%|█████████▍| 130/137 [00:28<00:01,  5.48it/s, epoch=33, grad_norm=140.101, loss=327.2714, lr=3.70e-04]
Epoch 33 TRAIN:  95%|█████████▍| 130/137 [00:28<00:01,  5.48it/s, epoch=33, grad_norm=111.727, loss=292.9937, lr=3.70e-04]
Epoch 33 TRAIN:  95%|█████████▍| 130/137 [00:29<00:01,  5.48it/s, epoch=33, grad_norm=154.874, loss=410.4944, lr=3.70e-04]
Epoch 33 TRAIN:  95%|█████████▍| 130/137 [00:29<00:01,  5.48it/s, epoch=33, grad_norm=155.323, loss=376.5249, lr=3.70e-04]
Epoch 33 TRAIN:  95%|█████████▍| 130/137 [00:29<00:01,  5.48it/s, epoch=33, grad_norm=147.643, loss=407.3138, lr=3.70e-04]
Epoch 33 TRAIN:  98%|█████████▊| 134/137 [00:29<00:00,  5.79it/s, epoch=33, grad_norm=147.643, loss=407.3138, lr=3.70e-04]
Epoch 33 TRAIN:  98%|█████████▊| 134/137 [00:29<00:00,  5.79it/s, epoch=33, grad_norm=256.569, loss=312.3111, lr=3.71e-04]
Epoch 33 TRAIN:  98%|█████████▊| 134/137 [00:29<00:00,  5.79it/s, epoch=33, grad_norm=120.736, loss=338.4430, lr=3.71e-04]
Epoch 33 TRAIN:  98%|█████████▊| 134/137 [00:29<00:00,  5.79it/s, epoch=33, grad_norm=179.643, loss=356.5484, lr=3.71e-04]
Epoch 33 TRAIN: 100%|██████████| 137/137 [00:29<00:00,  5.81it/s, epoch=33, grad_norm=179.643, loss=356.5484, lr=3.71e-04]
Epoch 33 TRAIN: 100%|██████████| 137/137 [00:29<00:00,  4.59it/s, epoch=33, grad_norm=179.643, loss=356.5484, lr=3.71e-04]

CV:   0%|          | 0/6 [00:00<?, ?it/s]/home/rookie/Documents/batuka/projects/wenet/wenet/wenet/utils/train_utils.py:689: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  amp_autocast(enabled=dtype is not None,

CV:  17%|█▋        | 1/6 [00:00<00:03,  1.32it/s]
                                                 
2026-04-16 20:08:47,825 INFO Epoch 33 Step 4634 CV info lr 3.7072e-04 cv_loss 523.510312167081 rank 0 acc 0.2536052664120992
2026-04-16 20:08:47,827 INFO [Rank 0] Checkpoint: save to checkpoint exp/conformer_mongolian/epoch_33.pt
2026-04-16 20:08:47,949 INFO Epoch 34 Step 4634 TRAIN info lr 3.7072e-04 rank 0
2026-04-16 20:08:47,965 INFO using accumulate grad, new batch size is 1 times larger than before

Epoch 34 TRAIN:   0%|          | 0/137 [00:00<?, ?it/s]
Epoch 34 TRAIN:   0%|          | 0/137 [00:04<?, ?it/s, epoch=34, grad_norm=471.641, loss=393.1217, lr=3.71e-04]
Epoch 34 TRAIN:   1%|          | 1/137 [00:04<09:08,  4.04s/it, epoch=34, grad_norm=471.641, loss=393.1217, lr=3.71e-04]
Epoch 34 TRAIN:   1%|          | 1/137 [00:04<09:08,  4.04s/it, epoch=34, grad_norm=394.578, loss=385.7405, lr=3.71e-04]
Epoch 34 TRAIN:   1%|          | 1/137 [00:04<09:08,  4.04s/it, epoch=34, grad_norm=261.832, loss=271.4996, lr=3.71e-04]
Epoch 34 TRAIN:   1%|          | 1/137 [00:04<09:08,  4.04s/it, epoch=34, grad_norm=267.510, loss=312.3900, lr=3.71e-04]
Epoch 34 TRAIN:   1%|          | 1/137 [00:04<09:08,  4.04s/it, epoch=34, grad_norm=484.975, loss=444.8389, lr=3.71e-04]
Epoch 34 TRAIN:   1%|          | 1/137 [00:04<09:08,  4.04s/it, epoch=34, grad_norm=144.698, loss=277.9599, lr=3.71e-04]
Epoch 34 TRAIN:   1%|          | 1/137 [00:04<09:08,  4.04s/it, epoch=34, grad_norm=359.376, loss=377.1556, lr=3.71e-04]
Epoch 34 TRAIN:   1%|          | 1/137 [00:04<09:08,  4.04s/it, epoch=34, grad_norm=100.437, loss=225.3966, lr=3.71e-04]
Epoch 34 TRAIN:   6%|▌         | 8/137 [00:04<00:55,  2.31it/s, epoch=34, grad_norm=100.437, loss=225.3966, lr=3.71e-04]
Epoch 34 TRAIN:   6%|▌         | 8/137 [00:04<00:55,  2.31it/s, epoch=34, grad_norm=97.631, loss=328.3782, lr=3.71e-04] 
Epoch 34 TRAIN:   6%|▌         | 8/137 [00:04<00:55,  2.31it/s, epoch=34, grad_norm=66.917, loss=345.5717, lr=3.72e-04]
Epoch 34 TRAIN:   6%|▌         | 8/137 [00:05<00:55,  2.31it/s, epoch=34, grad_norm=109.845, loss=349.0177, lr=3.72e-04]
Epoch 34 TRAIN:   6%|▌         | 8/137 [00:05<00:55,  2.31it/s, epoch=34, grad_norm=101.085, loss=344.2725, lr=3.72e-04]
Epoch 34 TRAIN:   6%|▌         | 8/137 [00:05<00:55,  2.31it/s, epoch=34, grad_norm=253.497, loss=345.6548, lr=3.72e-04]
Epoch 34 TRAIN:   9%|▉         | 13/137 [00:05<00:37,  3.33it/s, epoch=34, grad_norm=253.497, loss=345.6548, lr=3.72e-04]
Epoch 34 TRAIN:   9%|▉         | 13/137 [00:05<00:37,  3.33it/s, epoch=34, grad_norm=87.298, loss=352.6208, lr=3.72e-04] 
Epoch 34 TRAIN:   9%|▉         | 13/137 [00:05<00:37,  3.33it/s, epoch=34, grad_norm=127.665, loss=296.4019, lr=3.72e-04]
Epoch 34 TRAIN:   9%|▉         | 13/137 [00:05<00:37,  3.33it/s, epoch=34, grad_norm=64.467, loss=285.0707, lr=3.72e-04] 
Epoch 34 TRAIN:   9%|▉         | 13/137 [00:05<00:37,  3.33it/s, epoch=34, grad_norm=80.065, loss=310.4122, lr=3.72e-04]
Epoch 34 TRAIN:  12%|█▏        | 17/137 [00:05<00:29,  4.03it/s, epoch=34, grad_norm=80.065, loss=310.4122, lr=3.72e-04]
Epoch 34 TRAIN:  12%|█▏        | 17/137 [00:06<00:29,  4.03it/s, epoch=34, grad_norm=93.262, loss=318.5511, lr=3.72e-04]
Epoch 34 TRAIN:  12%|█▏        | 17/137 [00:06<00:29,  4.03it/s, epoch=34, grad_norm=52.061, loss=274.4739, lr=3.72e-04]
Epoch 34 TRAIN:  12%|█▏        | 17/137 [00:06<00:29,  4.03it/s, epoch=34, grad_norm=100.549, loss=331.6805, lr=3.72e-04]
Epoch 34 TRAIN:  12%|█▏        | 17/137 [00:06<00:29,  4.03it/s, epoch=34, grad_norm=90.178, loss=364.2839, lr=3.72e-04] 
Epoch 34 TRAIN:  15%|█▌        | 21/137 [00:06<00:26,  4.45it/s, epoch=34, grad_norm=90.178, loss=364.2839, lr=3.72e-04]
Epoch 34 TRAIN:  15%|█▌        | 21/137 [00:06<00:26,  4.45it/s, epoch=34, grad_norm=87.042, loss=313.9524, lr=3.72e-04]
Epoch 34 TRAIN:  15%|█▌        | 21/137 [00:07<00:26,  4.45it/s, epoch=34, grad_norm=82.475, loss=316.1107, lr=3.73e-04]
Epoch 34 TRAIN:  15%|█▌        | 21/137 [00:07<00:26,  4.45it/s, epoch=34, grad_norm=91.431, loss=326.4070, lr=3.73e-04]
Epoch 34 TRAIN:  15%|█▌        | 21/137 [00:07<00:26,  4.45it/s, epoch=34, grad_norm=87.253, loss=337.2269, lr=3.73e-04]
Epoch 34 TRAIN:  18%|█▊        | 25/137 [00:07<00:23,  4.67it/s, epoch=34, grad_norm=87.253, loss=337.2269, lr=3.73e-04]
Epoch 34 TRAIN:  18%|█▊        | 25/137 [00:07<00:23,  4.67it/s, epoch=34, grad_norm=76.328, loss=312.6616, lr=3.73e-04]
Epoch 34 TRAIN:  18%|█▊        | 25/137 [00:07<00:23,  4.67it/s, epoch=34, grad_norm=76.467, loss=294.3037, lr=3.73e-04]
Epoch 34 TRAIN:  18%|█▊        | 25/137 [00:07<00:23,  4.67it/s, epoch=34, grad_norm=76.436, loss=274.8202, lr=3.73e-04]
Epoch 34 TRAIN:  18%|█▊        | 25/137 [00:08<00:23,  4.67it/s, epoch=34, grad_norm=152.113, loss=346.2266, lr=3.73e-04]
Epoch 34 TRAIN:  21%|██        | 29/137 [00:08<00:21,  4.99it/s, epoch=34, grad_norm=152.113, loss=346.2266, lr=3.73e-04]
Epoch 34 TRAIN:  21%|██        | 29/137 [00:08<00:21,  4.99it/s, epoch=34, grad_norm=65.413, loss=271.1487, lr=3.73e-04] 
Epoch 34 TRAIN:  21%|██        | 29/137 [00:08<00:21,  4.99it/s, epoch=34, grad_norm=219.775, loss=379.6501, lr=3.73e-04]
Epoch 34 TRAIN:  21%|██        | 29/137 [00:08<00:21,  4.99it/s, epoch=34, grad_norm=91.529, loss=313.0284, lr=3.73e-04] 
Epoch 34 TRAIN:  21%|██        | 29/137 [00:08<00:21,  4.99it/s, epoch=34, grad_norm=73.307, loss=318.3482, lr=3.73e-04]
Epoch 34 TRAIN:  24%|██▍       | 33/137 [00:08<00:20,  4.97it/s, epoch=34, grad_norm=73.307, loss=318.3482, lr=3.73e-04]
Epoch 34 TRAIN:  24%|██▍       | 33/137 [00:09<00:20,  4.97it/s, epoch=34, grad_norm=81.992, loss=305.1626, lr=3.73e-04]
Epoch 34 TRAIN:  24%|██▍       | 33/137 [00:09<00:20,  4.97it/s, epoch=34, grad_norm=74.589, loss=308.9565, lr=3.74e-04]
Epoch 34 TRAIN:  24%|██▍       | 33/137 [00:09<00:20,  4.97it/s, epoch=34, grad_norm=66.638, loss=338.4494, lr=3.74e-04]
Epoch 34 TRAIN:  26%|██▋       | 36/137 [00:09<00:20,  4.95it/s, epoch=34, grad_norm=66.638, loss=338.4494, lr=3.74e-04]
Epoch 34 TRAIN:  26%|██▋       | 36/137 [00:09<00:20,  4.95it/s, epoch=34, grad_norm=81.313, loss=316.2202, lr=3.74e-04]
Epoch 34 TRAIN:  26%|██▋       | 36/137 [00:09<00:20,  4.95it/s, epoch=34, grad_norm=73.991, loss=331.1574, lr=3.74e-04]
Epoch 34 TRAIN:  26%|██▋       | 36/137 [00:10<00:20,  4.95it/s, epoch=34, grad_norm=67.229, loss=319.1143, lr=3.74e-04]
Epoch 34 TRAIN:  28%|██▊       | 39/137 [00:10<00:19,  5.07it/s, epoch=34, grad_norm=67.229, loss=319.1143, lr=3.74e-04]
Epoch 34 TRAIN:  28%|██▊       | 39/137 [00:10<00:19,  5.07it/s, epoch=34, grad_norm=120.920, loss=300.6976, lr=3.74e-04]
Epoch 34 TRAIN:  28%|██▊       | 39/137 [00:10<00:19,  5.07it/s, epoch=34, grad_norm=96.331, loss=315.0414, lr=3.74e-04] 
Epoch 34 TRAIN:  28%|██▊       | 39/137 [00:10<00:19,  5.07it/s, epoch=34, grad_norm=73.449, loss=312.0793, lr=3.74e-04]
Epoch 34 TRAIN:  31%|███       | 42/137 [00:10<00:18,  5.10it/s, epoch=34, grad_norm=73.449, loss=312.0793, lr=3.74e-04]
Epoch 34 TRAIN:  31%|███       | 42/137 [00:10<00:18,  5.10it/s, epoch=34, grad_norm=74.934, loss=284.4408, lr=3.74e-04]
Epoch 34 TRAIN:  31%|███       | 42/137 [00:11<00:18,  5.10it/s, epoch=34, grad_norm=62.606, loss=305.1102, lr=3.74e-04]
Epoch 34 TRAIN:  31%|███       | 42/137 [00:11<00:18,  5.10it/s, epoch=34, grad_norm=96.987, loss=304.1752, lr=3.74e-04]
Epoch 34 TRAIN:  33%|███▎      | 45/137 [00:11<00:18,  4.97it/s, epoch=34, grad_norm=96.987, loss=304.1752, lr=3.74e-04]
Epoch 34 TRAIN:  33%|███▎      | 45/137 [00:11<00:18,  4.97it/s, epoch=34, grad_norm=142.089, loss=326.8756, lr=3.74e-04]
Epoch 34 TRAIN:  33%|███▎      | 45/137 [00:11<00:18,  4.97it/s, epoch=34, grad_norm=159.345, loss=371.9046, lr=3.74e-04]
Epoch 34 TRAIN:  33%|███▎      | 45/137 [00:11<00:18,  4.97it/s, epoch=34, grad_norm=99.778, loss=345.7955, lr=3.75e-04] 
Epoch 34 TRAIN:  35%|███▌      | 48/137 [00:11<00:18,  4.94it/s, epoch=34, grad_norm=99.778, loss=345.7955, lr=3.75e-04]
Epoch 34 TRAIN:  35%|███▌      | 48/137 [00:12<00:18,  4.94it/s, epoch=34, grad_norm=80.137, loss=333.0398, lr=3.75e-04]
Epoch 34 TRAIN:  35%|███▌      | 48/137 [00:12<00:18,  4.94it/s, epoch=34, grad_norm=126.444, loss=288.9214, lr=3.75e-04]
Epoch 34 TRAIN:  35%|███▌      | 48/137 [00:12<00:18,  4.94it/s, epoch=34, grad_norm=90.556, loss=344.4969, lr=3.75e-04] 
Epoch 34 TRAIN:  37%|███▋      | 51/137 [00:12<00:16,  5.09it/s, epoch=34, grad_norm=90.556, loss=344.4969, lr=3.75e-04]
Epoch 34 TRAIN:  37%|███▋      | 51/137 [00:12<00:16,  5.09it/s, epoch=34, grad_norm=96.144, loss=337.9958, lr=3.75e-04]
Epoch 34 TRAIN:  37%|███▋      | 51/137 [00:12<00:16,  5.09it/s, epoch=34, grad_norm=71.114, loss=339.1426, lr=3.75e-04]
Epoch 34 TRAIN:  37%|███▋      | 51/137 [00:13<00:16,  5.09it/s, epoch=34, grad_norm=67.305, loss=336.4102, lr=3.75e-04]
Epoch 34 TRAIN:  39%|███▉      | 54/137 [00:13<00:17,  4.80it/s, epoch=34, grad_norm=67.305, loss=336.4102, lr=3.75e-04]
Epoch 34 TRAIN:  39%|███▉      | 54/137 [00:13<00:17,  4.80it/s, epoch=34, grad_norm=79.907, loss=365.8093, lr=3.75e-04]
Epoch 34 TRAIN:  39%|███▉      | 54/137 [00:13<00:17,  4.80it/s, epoch=34, grad_norm=87.459, loss=326.9489, lr=3.75e-04]
Epoch 34 TRAIN:  39%|███▉      | 54/137 [00:13<00:17,  4.80it/s, epoch=34, grad_norm=123.431, loss=301.5925, lr=3.75e-04]
Epoch 34 TRAIN:  42%|████▏     | 57/137 [00:13<00:16,  4.86it/s, epoch=34, grad_norm=123.431, loss=301.5925, lr=3.75e-04]
Epoch 34 TRAIN:  42%|████▏     | 57/137 [00:13<00:16,  4.86it/s, epoch=34, grad_norm=74.195, loss=337.1548, lr=3.75e-04] 
Epoch 34 TRAIN:  42%|████▏     | 57/137 [00:14<00:16,  4.86it/s, epoch=34, grad_norm=107.274, loss=322.3491, lr=3.75e-04]
Epoch 34 TRAIN:  42%|████▏     | 57/137 [00:14<00:16,  4.86it/s, epoch=34, grad_norm=69.981, loss=334.4148, lr=3.76e-04] 
Epoch 34 TRAIN:  44%|████▍     | 60/137 [00:14<00:15,  4.93it/s, epoch=34, grad_norm=69.981, loss=334.4148, lr=3.76e-04]
Epoch 34 TRAIN:  44%|████▍     | 60/137 [00:14<00:15,  4.93it/s, epoch=34, grad_norm=207.641, loss=394.6712, lr=3.76e-04]
Epoch 34 TRAIN:  44%|████▍     | 60/137 [00:14<00:15,  4.93it/s, epoch=34, grad_norm=77.043, loss=332.1743, lr=3.76e-04] 
Epoch 34 TRAIN:  44%|████▍     | 60/137 [00:14<00:15,  4.93it/s, epoch=34, grad_norm=102.701, loss=319.6662, lr=3.76e-04]
Epoch 34 TRAIN:  46%|████▌     | 63/137 [00:14<00:14,  5.08it/s, epoch=34, grad_norm=102.701, loss=319.6662, lr=3.76e-04]
Epoch 34 TRAIN:  46%|████▌     | 63/137 [00:15<00:14,  5.08it/s, epoch=34, grad_norm=209.428, loss=252.1097, lr=3.76e-04]
Epoch 34 TRAIN:  46%|████▌     | 63/137 [00:15<00:14,  5.08it/s, epoch=34, grad_norm=113.127, loss=327.6345, lr=3.76e-04]
Epoch 34 TRAIN:  46%|████▌     | 63/137 [00:15<00:14,  5.08it/s, epoch=34, grad_norm=96.549, loss=320.0329, lr=3.76e-04] 
Epoch 34 TRAIN:  48%|████▊     | 66/137 [00:15<00:14,  4.76it/s, epoch=34, grad_norm=96.549, loss=320.0329, lr=3.76e-04]
Epoch 34 TRAIN:  48%|████▊     | 66/137 [00:15<00:14,  4.76it/s, epoch=34, grad_norm=113.337, loss=361.8358, lr=3.76e-04]
Epoch 34 TRAIN:  48%|████▊     | 66/137 [00:16<00:14,  4.76it/s, epoch=34, grad_norm=101.363, loss=318.1214, lr=3.76e-04]
Epoch 34 TRAIN:  48%|████▊     | 66/137 [00:16<00:14,  4.76it/s, epoch=34, grad_norm=92.551, loss=261.9435, lr=3.76e-04] 
Epoch 34 TRAIN:  50%|█████     | 69/137 [00:16<00:14,  4.85it/s, epoch=34, grad_norm=92.551, loss=261.9435, lr=3.76e-04]
Epoch 34 TRAIN:  50%|█████     | 69/137 [00:16<00:14,  4.85it/s, epoch=34, grad_norm=95.563, loss=299.7161, lr=3.76e-04]
Epoch 34 TRAIN:  50%|█████     | 69/137 [00:16<00:14,  4.85it/s, epoch=34, grad_norm=90.649, loss=338.8699, lr=3.76e-04]
Epoch 34 TRAIN:  50%|█████     | 69/137 [00:16<00:14,  4.85it/s, epoch=34, grad_norm=93.315, loss=318.0017, lr=3.76e-04]
Epoch 34 TRAIN:  53%|█████▎    | 72/137 [00:16<00:12,  5.02it/s, epoch=34, grad_norm=93.315, loss=318.0017, lr=3.76e-04]
Epoch 34 TRAIN:  53%|█████▎    | 72/137 [00:17<00:12,  5.02it/s, epoch=34, grad_norm=169.647, loss=394.0859, lr=3.77e-04]
Epoch 34 TRAIN:  53%|█████▎    | 72/137 [00:17<00:12,  5.02it/s, epoch=34, grad_norm=79.290, loss=315.4297, lr=3.77e-04] 
Epoch 34 TRAIN:  53%|█████▎    | 72/137 [00:17<00:12,  5.02it/s, epoch=34, grad_norm=91.702, loss=304.6301, lr=3.77e-04]
Epoch 34 TRAIN:  55%|█████▍    | 75/137 [00:17<00:13,  4.67it/s, epoch=34, grad_norm=91.702, loss=304.6301, lr=3.77e-04]
Epoch 34 TRAIN:  55%|█████▍    | 75/137 [00:17<00:13,  4.67it/s, epoch=34, grad_norm=133.711, loss=305.9302, lr=3.77e-04]
Epoch 34 TRAIN:  55%|█████▍    | 75/137 [00:17<00:13,  4.67it/s, epoch=34, grad_norm=112.900, loss=311.8990, lr=3.77e-04]
Epoch 34 TRAIN:  55%|█████▍    | 75/137 [00:18<00:13,  4.67it/s, epoch=34, grad_norm=105.526, loss=293.8828, lr=3.77e-04]
Epoch 34 TRAIN:  57%|█████▋    | 78/137 [00:18<00:12,  4.72it/s, epoch=34, grad_norm=105.526, loss=293.8828, lr=3.77e-04]
Epoch 34 TRAIN:  57%|█████▋    | 78/137 [00:18<00:12,  4.72it/s, epoch=34, grad_norm=114.337, loss=310.1770, lr=3.77e-04]
Epoch 34 TRAIN:  57%|█████▋    | 78/137 [00:18<00:12,  4.72it/s, epoch=34, grad_norm=177.738, loss=311.4720, lr=3.77e-04]
Epoch 34 TRAIN:  57%|█████▋    | 78/137 [00:18<00:12,  4.72it/s, epoch=34, grad_norm=227.821, loss=250.9332, lr=3.77e-04]
Epoch 34 TRAIN:  59%|█████▉    | 81/137 [00:18<00:11,  4.70it/s, epoch=34, grad_norm=227.821, loss=250.9332, lr=3.77e-04]
Epoch 34 TRAIN:  59%|█████▉    | 81/137 [00:18<00:11,  4.70it/s, epoch=34, grad_norm=93.663, loss=297.8819, lr=3.77e-04] 
Epoch 34 TRAIN:  59%|█████▉    | 81/137 [00:19<00:11,  4.70it/s, epoch=34, grad_norm=89.357, loss=287.2344, lr=3.77e-04]
Epoch 34 TRAIN:  59%|█████▉    | 81/137 [00:19<00:11,  4.70it/s, epoch=34, grad_norm=83.098, loss=292.4628, lr=3.77e-04]
Epoch 34 TRAIN:  61%|██████▏   | 84/137 [00:19<00:11,  4.74it/s, epoch=34, grad_norm=83.098, loss=292.4628, lr=3.77e-04]
Epoch 34 TRAIN:  61%|██████▏   | 84/137 [00:19<00:11,  4.74it/s, epoch=34, grad_norm=110.734, loss=270.2123, lr=3.78e-04]
Epoch 34 TRAIN:  61%|██████▏   | 84/137 [00:19<00:11,  4.74it/s, epoch=34, grad_norm=115.721, loss=256.8317, lr=3.78e-04]
Epoch 34 TRAIN:  61%|██████▏   | 84/137 [00:20<00:11,  4.74it/s, epoch=34, grad_norm=149.263, loss=311.8464, lr=3.78e-04]
Epoch 34 TRAIN:  64%|██████▎   | 87/137 [00:20<00:10,  4.65it/s, epoch=34, grad_norm=149.263, loss=311.8464, lr=3.78e-04]
Epoch 34 TRAIN:  64%|██████▎   | 87/137 [00:20<00:10,  4.65it/s, epoch=34, grad_norm=105.139, loss=285.9203, lr=3.78e-04]
Epoch 34 TRAIN:  64%|██████▎   | 87/137 [00:20<00:10,  4.65it/s, epoch=34, grad_norm=97.668, loss=278.7967, lr=3.78e-04] 
Epoch 34 TRAIN:  64%|██████▎   | 87/137 [00:20<00:10,  4.65it/s, epoch=34, grad_norm=93.365, loss=287.8681, lr=3.78e-04]
Epoch 34 TRAIN:  66%|██████▌   | 90/137 [00:20<00:10,  4.67it/s, epoch=34, grad_norm=93.365, loss=287.8681, lr=3.78e-04]
Epoch 34 TRAIN:  66%|██████▌   | 90/137 [00:20<00:10,  4.67it/s, epoch=34, grad_norm=119.915, loss=265.4374, lr=3.78e-04]
Epoch 34 TRAIN:  66%|██████▌   | 90/137 [00:21<00:10,  4.67it/s, epoch=34, grad_norm=87.231, loss=307.2313, lr=3.78e-04] 
Epoch 34 TRAIN:  66%|██████▌   | 90/137 [00:21<00:10,  4.67it/s, epoch=34, grad_norm=308.833, loss=436.2152, lr=3.78e-04]
Epoch 34 TRAIN:  68%|██████▊   | 93/137 [00:21<00:08,  4.93it/s, epoch=34, grad_norm=308.833, loss=436.2152, lr=3.78e-04]
Epoch 34 TRAIN:  68%|██████▊   | 93/137 [00:21<00:08,  4.93it/s, epoch=34, grad_norm=122.058, loss=334.8046, lr=3.78e-04]
Epoch 34 TRAIN:  68%|██████▊   | 93/137 [00:21<00:08,  4.93it/s, epoch=34, grad_norm=131.273, loss=373.5172, lr=3.78e-04]
Epoch 34 TRAIN:  68%|██████▊   | 93/137 [00:21<00:08,  4.93it/s, epoch=34, grad_norm=238.998, loss=320.1505, lr=3.78e-04]
Epoch 34 TRAIN:  70%|███████   | 96/137 [00:21<00:08,  4.63it/s, epoch=34, grad_norm=238.998, loss=320.1505, lr=3.78e-04]
Epoch 34 TRAIN:  70%|███████   | 96/137 [00:22<00:08,  4.63it/s, epoch=34, grad_norm=217.573, loss=275.9326, lr=3.78e-04]
Epoch 34 TRAIN:  70%|███████   | 96/137 [00:22<00:08,  4.63it/s, epoch=34, grad_norm=287.861, loss=236.1665, lr=3.79e-04]
Epoch 34 TRAIN:  70%|███████   | 96/137 [00:22<00:08,  4.63it/s, epoch=34, grad_norm=97.466, loss=335.6639, lr=3.79e-04] 
Epoch 34 TRAIN:  72%|███████▏  | 99/137 [00:22<00:07,  4.80it/s, epoch=34, grad_norm=97.466, loss=335.6639, lr=3.79e-04]
Epoch 34 TRAIN:  72%|███████▏  | 99/137 [00:22<00:07,  4.80it/s, epoch=34, grad_norm=233.089, loss=280.6132, lr=3.79e-04]2026-04-16 20:09:10,714 DEBUG TRAIN | steps/sec 4.429| Batch 34/100 loss 280.613159 loss_att 244.875107 loss_ctc 364.001953 th_accuracy 0.275132 lr 3.7872e-04 grad_norm 233.088959 rank 0

Epoch 34 TRAIN:  72%|███████▏  | 99/137 [00:22<00:07,  4.80it/s, epoch=34, grad_norm=152.120, loss=265.7991, lr=3.79e-04]
Epoch 34 TRAIN:  72%|███████▏  | 99/137 [00:23<00:07,  4.80it/s, epoch=34, grad_norm=97.654, loss=264.3846, lr=3.79e-04] 
Epoch 34 TRAIN:  74%|███████▍  | 102/137 [00:23<00:06,  5.04it/s, epoch=34, grad_norm=97.654, loss=264.3846, lr=3.79e-04]
Epoch 34 TRAIN:  74%|███████▍  | 102/137 [00:23<00:06,  5.04it/s, epoch=34, grad_norm=107.338, loss=321.1696, lr=3.79e-04]
Epoch 34 TRAIN:  74%|███████▍  | 102/137 [00:23<00:06,  5.04it/s, epoch=34, grad_norm=115.153, loss=333.1401, lr=3.79e-04]
Epoch 34 TRAIN:  74%|███████▍  | 102/137 [00:23<00:06,  5.04it/s, epoch=34, grad_norm=137.069, loss=278.6262, lr=3.79e-04]
Epoch 34 TRAIN:  77%|███████▋  | 105/137 [00:23<00:06,  4.78it/s, epoch=34, grad_norm=137.069, loss=278.6262, lr=3.79e-04]
Epoch 34 TRAIN:  77%|███████▋  | 105/137 [00:23<00:06,  4.78it/s, epoch=34, grad_norm=92.595, loss=252.4657, lr=3.79e-04] 
Epoch 34 TRAIN:  77%|███████▋  | 105/137 [00:24<00:06,  4.78it/s, epoch=34, grad_norm=231.943, loss=370.9993, lr=3.79e-04]
Epoch 34 TRAIN:  77%|███████▋  | 105/137 [00:24<00:06,  4.78it/s, epoch=34, grad_norm=154.015, loss=215.7780, lr=3.79e-04]
Epoch 34 TRAIN:  79%|███████▉  | 108/137 [00:24<00:05,  4.92it/s, epoch=34, grad_norm=154.015, loss=215.7780, lr=3.79e-04]
Epoch 34 TRAIN:  79%|███████▉  | 108/137 [00:24<00:05,  4.92it/s, epoch=34, grad_norm=94.259, loss=306.7350, lr=3.79e-04] 
Epoch 34 TRAIN:  79%|███████▉  | 108/137 [00:24<00:05,  4.92it/s, epoch=34, grad_norm=91.600, loss=293.4910, lr=3.80e-04]
Epoch 34 TRAIN:  79%|███████▉  | 108/137 [00:24<00:05,  4.92it/s, epoch=34, grad_norm=113.122, loss=301.0849, lr=3.80e-04]
Epoch 34 TRAIN:  81%|████████  | 111/137 [00:24<00:05,  4.88it/s, epoch=34, grad_norm=113.122, loss=301.0849, lr=3.80e-04]
Epoch 34 TRAIN:  81%|████████  | 111/137 [00:25<00:05,  4.88it/s, epoch=34, grad_norm=164.896, loss=359.8049, lr=3.80e-04]
Epoch 34 TRAIN:  81%|████████  | 111/137 [00:25<00:05,  4.88it/s, epoch=34, grad_norm=82.493, loss=260.3481, lr=3.80e-04] 
Epoch 34 TRAIN:  81%|████████  | 111/137 [00:25<00:05,  4.88it/s, epoch=34, grad_norm=116.308, loss=313.9993, lr=3.80e-04]
Epoch 34 TRAIN:  83%|████████▎ | 114/137 [00:25<00:04,  5.11it/s, epoch=34, grad_norm=116.308, loss=313.9993, lr=3.80e-04]
Epoch 34 TRAIN:  83%|████████▎ | 114/137 [00:25<00:04,  5.11it/s, epoch=34, grad_norm=136.902, loss=354.6922, lr=3.80e-04]
Epoch 34 TRAIN:  83%|████████▎ | 114/137 [00:25<00:04,  5.11it/s, epoch=34, grad_norm=92.769, loss=326.4866, lr=3.80e-04] 
Epoch 34 TRAIN:  83%|████████▎ | 114/137 [00:26<00:04,  5.11it/s, epoch=34, grad_norm=91.348, loss=330.0867, lr=3.80e-04]
Epoch 34 TRAIN:  85%|████████▌ | 117/137 [00:26<00:04,  4.76it/s, epoch=34, grad_norm=91.348, loss=330.0867, lr=3.80e-04]
Epoch 34 TRAIN:  85%|████████▌ | 117/137 [00:26<00:04,  4.76it/s, epoch=34, grad_norm=191.401, loss=220.1249, lr=3.80e-04]
Epoch 34 TRAIN:  85%|████████▌ | 117/137 [00:26<00:04,  4.76it/s, epoch=34, grad_norm=177.856, loss=271.2261, lr=3.80e-04]
Epoch 34 TRAIN:  85%|████████▌ | 117/137 [00:26<00:04,  4.76it/s, epoch=34, grad_norm=110.724, loss=342.9351, lr=3.80e-04]
Epoch 34 TRAIN:  88%|████████▊ | 120/137 [00:26<00:03,  4.79it/s, epoch=34, grad_norm=110.724, loss=342.9351, lr=3.80e-04]
Epoch 34 TRAIN:  88%|████████▊ | 120/137 [00:27<00:03,  4.79it/s, epoch=34, grad_norm=113.022, loss=279.3674, lr=3.80e-04]
Epoch 34 TRAIN:  88%|████████▊ | 120/137 [00:27<00:03,  4.79it/s, epoch=34, grad_norm=92.003, loss=334.3707, lr=3.80e-04] 
Epoch 34 TRAIN:  88%|████████▊ | 120/137 [00:27<00:03,  4.79it/s, epoch=34, grad_norm=131.451, loss=280.0311, lr=3.81e-04]
Epoch 34 TRAIN:  90%|████████▉ | 123/137 [00:27<00:02,  4.89it/s, epoch=34, grad_norm=131.451, loss=280.0311, lr=3.81e-04]
Epoch 34 TRAIN:  90%|████████▉ | 123/137 [00:27<00:02,  4.89it/s, epoch=34, grad_norm=158.457, loss=326.2104, lr=3.81e-04]
Epoch 34 TRAIN:  90%|████████▉ | 123/137 [00:27<00:02,  4.89it/s, epoch=34, grad_norm=102.431, loss=291.0823, lr=3.81e-04]
Epoch 34 TRAIN:  90%|████████▉ | 123/137 [00:28<00:02,  4.89it/s, epoch=34, grad_norm=134.362, loss=334.3066, lr=3.81e-04]
Epoch 34 TRAIN:  92%|█████████▏| 126/137 [00:28<00:02,  4.91it/s, epoch=34, grad_norm=134.362, loss=334.3066, lr=3.81e-04]
Epoch 34 TRAIN:  92%|█████████▏| 126/137 [00:28<00:02,  4.91it/s, epoch=34, grad_norm=94.423, loss=288.4277, lr=3.81e-04] 
Epoch 34 TRAIN:  92%|█████████▏| 126/137 [00:28<00:02,  4.91it/s, epoch=34, grad_norm=165.431, loss=360.5143, lr=3.81e-04]
Epoch 34 TRAIN:  92%|█████████▏| 126/137 [00:28<00:02,  4.91it/s, epoch=34, grad_norm=155.649, loss=414.7322, lr=3.81e-04]
Epoch 34 TRAIN:  94%|█████████▍| 129/137 [00:28<00:01,  4.85it/s, epoch=34, grad_norm=155.649, loss=414.7322, lr=3.81e-04]
Epoch 34 TRAIN:  94%|█████████▍| 129/137 [00:28<00:01,  4.85it/s, epoch=34, grad_norm=207.467, loss=363.6739, lr=3.81e-04]
Epoch 34 TRAIN:  94%|█████████▍| 129/137 [00:29<00:01,  4.85it/s, epoch=34, grad_norm=214.499, loss=294.3377, lr=3.81e-04]
Epoch 34 TRAIN:  94%|█████████▍| 129/137 [00:29<00:01,  4.85it/s, epoch=34, grad_norm=139.574, loss=273.8287, lr=3.81e-04]
Epoch 34 TRAIN:  96%|█████████▋| 132/137 [00:29<00:01,  4.91it/s, epoch=34, grad_norm=139.574, loss=273.8287, lr=3.81e-04]
Epoch 34 TRAIN:  96%|█████████▋| 132/137 [00:29<00:01,  4.91it/s, epoch=34, grad_norm=115.796, loss=398.3258, lr=3.81e-04]
Epoch 34 TRAIN:  96%|█████████▋| 132/137 [00:29<00:01,  4.91it/s, epoch=34, grad_norm=146.622, loss=374.9178, lr=3.81e-04]
Epoch 34 TRAIN:  96%|█████████▋| 132/137 [00:29<00:01,  4.91it/s, epoch=34, grad_norm=182.238, loss=405.5037, lr=3.82e-04]
Epoch 34 TRAIN:  99%|█████████▊| 135/137 [00:29<00:00,  4.84it/s, epoch=34, grad_norm=182.238, loss=405.5037, lr=3.82e-04]
Epoch 34 TRAIN:  99%|█████████▊| 135/137 [00:30<00:00,  4.84it/s, epoch=34, grad_norm=175.148, loss=305.0862, lr=3.82e-04]
Epoch 34 TRAIN:  99%|█████████▊| 135/137 [00:30<00:00,  4.84it/s, epoch=34, grad_norm=199.950, loss=399.2851, lr=3.82e-04]
Epoch 34 TRAIN:  99%|█████████▊| 135/137 [00:30<00:00,  4.84it/s, epoch=34, grad_norm=162.799, loss=343.6997, lr=3.82e-04]
Epoch 34 TRAIN: 138it [00:30,  4.55it/s, epoch=34, grad_norm=162.799, loss=343.6997, lr=3.82e-04]                         

CV:   0%|          | 0/6 [00:00<?, ?it/s]/home/rookie/Documents/batuka/projects/wenet/wenet/wenet/utils/train_utils.py:689: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  amp_autocast(enabled=dtype is not None,

CV:  17%|█▋        | 1/6 [00:00<00:03,  1.33it/s]
                                                 
2026-04-16 20:09:19,158 INFO Epoch 34 Step 4772 CV info lr 3.8176e-04 cv_loss 505.41869562322444 rank 0 acc 0.2610875740647316
2026-04-16 20:09:19,159 INFO [Rank 0] Checkpoint: save to checkpoint exp/conformer_mongolian/epoch_34.pt
2026-04-16 20:09:19,287 INFO Epoch 35 Step 4772 TRAIN info lr 3.8176e-04 rank 0
2026-04-16 20:09:19,305 INFO using accumulate grad, new batch size is 1 times larger than before

Epoch 35 TRAIN:   0%|          | 0/138 [00:00<?, ?it/s]
Epoch 35 TRAIN:   0%|          | 0/138 [00:03<?, ?it/s, epoch=35, grad_norm=334.677, loss=308.8922, lr=3.82e-04]
Epoch 35 TRAIN:   1%|          | 1/138 [00:03<09:01,  3.95s/it, epoch=35, grad_norm=334.677, loss=308.8922, lr=3.82e-04]
Epoch 35 TRAIN:   1%|          | 1/138 [00:04<09:01,  3.95s/it, epoch=35, grad_norm=438.029, loss=393.3132, lr=3.82e-04]
Epoch 35 TRAIN:   1%|          | 1/138 [00:04<09:01,  3.95s/it, epoch=35, grad_norm=177.596, loss=264.8969, lr=3.82e-04]
Epoch 35 TRAIN:   1%|          | 1/138 [00:04<09:01,  3.95s/it, epoch=35, grad_norm=351.120, loss=399.2941, lr=3.82e-04]
Epoch 35 TRAIN:   1%|          | 1/138 [00:04<09:01,  3.95s/it, epoch=35, grad_norm=142.049, loss=308.3699, lr=3.82e-04]
Epoch 35 TRAIN:   1%|          | 1/138 [00:04<09:01,  3.95s/it, epoch=35, grad_norm=365.709, loss=375.1591, lr=3.82e-04]
Epoch 35 TRAIN:   1%|          | 1/138 [00:04<09:01,  3.95s/it, epoch=35, grad_norm=195.741, loss=304.1079, lr=3.82e-04]
Epoch 35 TRAIN:   1%|          | 1/138 [00:04<09:01,  3.95s/it, epoch=35, grad_norm=302.533, loss=342.4655, lr=3.82e-04]
Epoch 35 TRAIN:   6%|▌         | 8/138 [00:04<00:55,  2.33it/s, epoch=35, grad_norm=302.533, loss=342.4655, lr=3.82e-04]
Epoch 35 TRAIN:   6%|▌         | 8/138 [00:04<00:55,  2.33it/s, epoch=35, grad_norm=132.600, loss=337.4014, lr=3.82e-04]
Epoch 35 TRAIN:   6%|▌         | 8/138 [00:04<00:55,  2.33it/s, epoch=35, grad_norm=122.468, loss=351.8488, lr=3.83e-04]
Epoch 35 TRAIN:   6%|▌         | 8/138 [00:04<00:55,  2.33it/s, epoch=35, grad_norm=72.814, loss=358.0158, lr=3.83e-04] 
Epoch 35 TRAIN:   6%|▌         | 8/138 [00:05<00:55,  2.33it/s, epoch=35, grad_norm=166.985, loss=288.1218, lr=3.83e-04]
Epoch 35 TRAIN:   9%|▊         | 12/138 [00:05<00:39,  3.18it/s, epoch=35, grad_norm=166.985, loss=288.1218, lr=3.83e-04]
Epoch 35 TRAIN:   9%|▊         | 12/138 [00:05<00:39,  3.18it/s, epoch=35, grad_norm=247.477, loss=359.2531, lr=3.83e-04]
Epoch 35 TRAIN:   9%|▊         | 12/138 [00:05<00:39,  3.18it/s, epoch=35, grad_norm=211.924, loss=302.5346, lr=3.83e-04]
Epoch 35 TRAIN:   9%|▊         | 12/138 [00:05<00:39,  3.18it/s, epoch=35, grad_norm=194.552, loss=290.1569, lr=3.83e-04]
Epoch 35 TRAIN:   9%|▊         | 12/138 [00:05<00:39,  3.18it/s, epoch=35, grad_norm=78.164, loss=347.1019, lr=3.83e-04] 
Epoch 35 TRAIN:  12%|█▏        | 16/138 [00:05<00:31,  3.87it/s, epoch=35, grad_norm=78.164, loss=347.1019, lr=3.83e-04]
Epoch 35 TRAIN:  12%|█▏        | 16/138 [00:06<00:31,  3.87it/s, epoch=35, grad_norm=73.888, loss=324.4099, lr=3.83e-04]
Epoch 35 TRAIN:  12%|█▏        | 16/138 [00:06<00:31,  3.87it/s, epoch=35, grad_norm=73.554, loss=295.8596, lr=3.83e-04]
Epoch 35 TRAIN:  12%|█▏        | 16/138 [00:06<00:31,  3.87it/s, epoch=35, grad_norm=75.856, loss=320.8200, lr=3.83e-04]
Epoch 35 TRAIN:  12%|█▏        | 16/138 [00:06<00:31,  3.87it/s, epoch=35, grad_norm=136.241, loss=345.9640, lr=3.83e-04]
Epoch 35 TRAIN:  14%|█▍        | 20/138 [00:06<00:28,  4.15it/s, epoch=35, grad_norm=136.241, loss=345.9640, lr=3.83e-04]
Epoch 35 TRAIN:  14%|█▍        | 20/138 [00:06<00:28,  4.15it/s, epoch=35, grad_norm=57.595, loss=293.9834, lr=3.83e-04] 
Epoch 35 TRAIN:  14%|█▍        | 20/138 [00:06<00:28,  4.15it/s, epoch=35, grad_norm=135.965, loss=338.1618, lr=3.84e-04]
Epoch 35 TRAIN:  14%|█▍        | 20/138 [00:07<00:28,  4.15it/s, epoch=35, grad_norm=212.494, loss=346.3646, lr=3.84e-04]
Epoch 35 TRAIN:  17%|█▋        | 23/138 [00:07<00:25,  4.45it/s, epoch=35, grad_norm=212.494, loss=346.3646, lr=3.84e-04]
Epoch 35 TRAIN:  17%|█▋        | 23/138 [00:07<00:25,  4.45it/s, epoch=35, grad_norm=67.345, loss=319.5372, lr=3.84e-04] 
Epoch 35 TRAIN:  17%|█▋        | 23/138 [00:07<00:25,  4.45it/s, epoch=35, grad_norm=134.693, loss=321.2187, lr=3.84e-04]
Epoch 35 TRAIN:  17%|█▋        | 23/138 [00:07<00:25,  4.45it/s, epoch=35, grad_norm=153.622, loss=348.4026, lr=3.84e-04]
Epoch 35 TRAIN:  19%|█▉        | 26/138 [00:07<00:24,  4.58it/s, epoch=35, grad_norm=153.622, loss=348.4026, lr=3.84e-04]
Epoch 35 TRAIN:  19%|█▉        | 26/138 [00:07<00:24,  4.58it/s, epoch=35, grad_norm=144.897, loss=313.0435, lr=3.84e-04]
Epoch 35 TRAIN:  19%|█▉        | 26/138 [00:08<00:24,  4.58it/s, epoch=35, grad_norm=86.380, loss=336.1357, lr=3.84e-04] 
Epoch 35 TRAIN:  19%|█▉        | 26/138 [00:08<00:24,  4.58it/s, epoch=35, grad_norm=131.933, loss=382.7050, lr=3.84e-04]
Epoch 35 TRAIN:  21%|██        | 29/138 [00:08<00:23,  4.64it/s, epoch=35, grad_norm=131.933, loss=382.7050, lr=3.84e-04]
Epoch 35 TRAIN:  21%|██        | 29/138 [00:08<00:23,  4.64it/s, epoch=35, grad_norm=118.552, loss=329.5246, lr=3.84e-04]
Epoch 35 TRAIN:  21%|██        | 29/138 [00:08<00:23,  4.64it/s, epoch=35, grad_norm=87.888, loss=323.4501, lr=3.84e-04] 
Epoch 35 TRAIN:  21%|██        | 29/138 [00:08<00:23,  4.64it/s, epoch=35, grad_norm=109.355, loss=325.5767, lr=3.84e-04]
Epoch 35 TRAIN:  23%|██▎       | 32/138 [00:08<00:21,  4.82it/s, epoch=35, grad_norm=109.355, loss=325.5767, lr=3.84e-04]
Epoch 35 TRAIN:  23%|██▎       | 32/138 [00:09<00:21,  4.82it/s, epoch=35, grad_norm=80.645, loss=363.7272, lr=3.84e-04] 
Epoch 35 TRAIN:  23%|██▎       | 32/138 [00:09<00:21,  4.82it/s, epoch=35, grad_norm=159.552, loss=271.4437, lr=3.84e-04]
Epoch 35 TRAIN:  23%|██▎       | 32/138 [00:09<00:21,  4.82it/s, epoch=35, grad_norm=137.205, loss=292.1029, lr=3.85e-04]
Epoch 35 TRAIN:  25%|██▌       | 35/138 [00:09<00:21,  4.83it/s, epoch=35, grad_norm=137.205, loss=292.1029, lr=3.85e-04]
Epoch 35 TRAIN:  25%|██▌       | 35/138 [00:09<00:21,  4.83it/s, epoch=35, grad_norm=91.833, loss=292.9128, lr=3.85e-04] 
Epoch 35 TRAIN:  25%|██▌       | 35/138 [00:09<00:21,  4.83it/s, epoch=35, grad_norm=85.187, loss=291.6591, lr=3.85e-04]
Epoch 35 TRAIN:  25%|██▌       | 35/138 [00:10<00:21,  4.83it/s, epoch=35, grad_norm=112.410, loss=306.3476, lr=3.85e-04]
Epoch 35 TRAIN:  28%|██▊       | 38/138 [00:10<00:20,  4.93it/s, epoch=35, grad_norm=112.410, loss=306.3476, lr=3.85e-04]
Epoch 35 TRAIN:  28%|██▊       | 38/138 [00:10<00:20,  4.93it/s, epoch=35, grad_norm=87.971, loss=296.9652, lr=3.85e-04] 
Epoch 35 TRAIN:  28%|██▊       | 38/138 [00:10<00:20,  4.93it/s, epoch=35, grad_norm=97.812, loss=309.7400, lr=3.85e-04]
Epoch 35 TRAIN:  28%|██▊       | 38/138 [00:10<00:20,  4.93it/s, epoch=35, grad_norm=99.612, loss=321.9646, lr=3.85e-04]
Epoch 35 TRAIN:  30%|██▉       | 41/138 [00:10<00:19,  4.92it/s, epoch=35, grad_norm=99.612, loss=321.9646, lr=3.85e-04]
Epoch 35 TRAIN:  30%|██▉       | 41/138 [00:10<00:19,  4.92it/s, epoch=35, grad_norm=190.534, loss=360.6510, lr=3.85e-04]
Epoch 35 TRAIN:  30%|██▉       | 41/138 [00:11<00:19,  4.92it/s, epoch=35, grad_norm=120.888, loss=330.8657, lr=3.85e-04]
Epoch 35 TRAIN:  30%|██▉       | 41/138 [00:11<00:19,  4.92it/s, epoch=35, grad_norm=86.558, loss=314.1224, lr=3.85e-04] 
Epoch 35 TRAIN:  32%|███▏      | 44/138 [00:11<00:19,  4.94it/s, epoch=35, grad_norm=86.558, loss=314.1224, lr=3.85e-04]
Epoch 35 TRAIN:  32%|███▏      | 44/138 [00:11<00:19,  4.94it/s, epoch=35, grad_norm=103.337, loss=345.6900, lr=3.85e-04]
Epoch 35 TRAIN:  32%|███▏      | 44/138 [00:11<00:19,  4.94it/s, epoch=35, grad_norm=130.763, loss=259.4792, lr=3.85e-04]
Epoch 35 TRAIN:  32%|███▏      | 44/138 [00:12<00:19,  4.94it/s, epoch=35, grad_norm=106.307, loss=341.6259, lr=3.86e-04]
Epoch 35 TRAIN:  34%|███▍      | 47/138 [00:12<00:18,  4.90it/s, epoch=35, grad_norm=106.307, loss=341.6259, lr=3.86e-04]
Epoch 35 TRAIN:  34%|███▍      | 47/138 [00:12<00:18,  4.90it/s, epoch=35, grad_norm=101.895, loss=353.1573, lr=3.86e-04]
Epoch 35 TRAIN:  34%|███▍      | 47/138 [00:12<00:18,  4.90it/s, epoch=35, grad_norm=199.429, loss=278.7754, lr=3.86e-04]
Epoch 35 TRAIN:  34%|███▍      | 47/138 [00:12<00:18,  4.90it/s, epoch=35, grad_norm=131.870, loss=271.2290, lr=3.86e-04]
Epoch 35 TRAIN:  36%|███▌      | 50/138 [00:12<00:17,  4.92it/s, epoch=35, grad_norm=131.870, loss=271.2290, lr=3.86e-04]
Epoch 35 TRAIN:  36%|███▌      | 50/138 [00:12<00:17,  4.92it/s, epoch=35, grad_norm=92.117, loss=336.3344, lr=3.86e-04] 
Epoch 35 TRAIN:  36%|███▌      | 50/138 [00:13<00:17,  4.92it/s, epoch=35, grad_norm=86.919, loss=295.2743, lr=3.86e-04]
Epoch 35 TRAIN:  36%|███▌      | 50/138 [00:13<00:17,  4.92it/s, epoch=35, grad_norm=96.249, loss=300.7755, lr=3.86e-04]
Epoch 35 TRAIN:  38%|███▊      | 53/138 [00:13<00:16,  5.03it/s, epoch=35, grad_norm=96.249, loss=300.7755, lr=3.86e-04]
Epoch 35 TRAIN:  38%|███▊      | 53/138 [00:13<00:16,  5.03it/s, epoch=35, grad_norm=93.620, loss=298.6574, lr=3.86e-04]
Epoch 35 TRAIN:  38%|███▊      | 53/138 [00:13<00:16,  5.03it/s, epoch=35, grad_norm=110.244, loss=296.5419, lr=3.86e-04]
Epoch 35 TRAIN:  38%|███▊      | 53/138 [00:13<00:16,  5.03it/s, epoch=35, grad_norm=90.098, loss=341.2286, lr=3.86e-04] 
Epoch 35 TRAIN:  41%|████      | 56/138 [00:13<00:17,  4.71it/s, epoch=35, grad_norm=90.098, loss=341.2286, lr=3.86e-04]
Epoch 35 TRAIN:  41%|████      | 56/138 [00:14<00:17,  4.71it/s, epoch=35, grad_norm=157.804, loss=247.8536, lr=3.86e-04]
Epoch 35 TRAIN:  41%|████      | 56/138 [00:14<00:17,  4.71it/s, epoch=35, grad_norm=107.324, loss=297.2622, lr=3.86e-04]
Epoch 35 TRAIN:  41%|████      | 56/138 [00:14<00:17,  4.71it/s, epoch=35, grad_norm=88.262, loss=317.1069, lr=3.86e-04] 
Epoch 35 TRAIN:  43%|████▎     | 59/138 [00:14<00:16,  4.67it/s, epoch=35, grad_norm=88.262, loss=317.1069, lr=3.86e-04]
Epoch 35 TRAIN:  43%|████▎     | 59/138 [00:14<00:16,  4.67it/s, epoch=35, grad_norm=87.835, loss=320.9206, lr=3.87e-04]
Epoch 35 TRAIN:  43%|████▎     | 59/138 [00:14<00:16,  4.67it/s, epoch=35, grad_norm=117.049, loss=348.4075, lr=3.87e-04]
Epoch 35 TRAIN:  43%|████▎     | 59/138 [00:15<00:16,  4.67it/s, epoch=35, grad_norm=109.222, loss=340.2461, lr=3.87e-04]
Epoch 35 TRAIN:  45%|████▍     | 62/138 [00:15<00:16,  4.64it/s, epoch=35, grad_norm=109.222, loss=340.2461, lr=3.87e-04]
Epoch 35 TRAIN:  45%|████▍     | 62/138 [00:15<00:16,  4.64it/s, epoch=35, grad_norm=94.069, loss=347.8709, lr=3.87e-04] 
Epoch 35 TRAIN:  45%|████▍     | 62/138 [00:15<00:16,  4.64it/s, epoch=35, grad_norm=209.279, loss=273.7835, lr=3.87e-04]
Epoch 35 TRAIN:  45%|████▍     | 62/138 [00:15<00:16,  4.64it/s, epoch=35, grad_norm=197.883, loss=312.8384, lr=3.87e-04]
Epoch 35 TRAIN:  47%|████▋     | 65/138 [00:15<00:15,  4.71it/s, epoch=35, grad_norm=197.883, loss=312.8384, lr=3.87e-04]
Epoch 35 TRAIN:  47%|████▋     | 65/138 [00:16<00:15,  4.71it/s, epoch=35, grad_norm=91.321, loss=356.5380, lr=3.87e-04] 
Epoch 35 TRAIN:  47%|████▋     | 65/138 [00:16<00:15,  4.71it/s, epoch=35, grad_norm=95.741, loss=346.4237, lr=3.87e-04]
Epoch 35 TRAIN:  47%|████▋     | 65/138 [00:16<00:15,  4.71it/s, epoch=35, grad_norm=94.269, loss=323.1734, lr=3.87e-04]
Epoch 35 TRAIN:  49%|████▉     | 68/138 [00:16<00:14,  4.78it/s, epoch=35, grad_norm=94.269, loss=323.1734, lr=3.87e-04]
Epoch 35 TRAIN:  49%|████▉     | 68/138 [00:16<00:14,  4.78it/s, epoch=35, grad_norm=105.544, loss=346.6759, lr=3.87e-04]
Epoch 35 TRAIN:  49%|████▉     | 68/138 [00:16<00:14,  4.78it/s, epoch=35, grad_norm=85.875, loss=312.4360, lr=3.87e-04] 
Epoch 35 TRAIN:  49%|████▉     | 68/138 [00:17<00:14,  4.78it/s, epoch=35, grad_norm=115.168, loss=285.5373, lr=3.87e-04]
Epoch 35 TRAIN:  51%|█████▏    | 71/138 [00:17<00:14,  4.75it/s, epoch=35, grad_norm=115.168, loss=285.5373, lr=3.87e-04]
Epoch 35 TRAIN:  51%|█████▏    | 71/138 [00:17<00:14,  4.75it/s, epoch=35, grad_norm=82.440, loss=335.5093, lr=3.88e-04] 
Epoch 35 TRAIN:  51%|█████▏    | 71/138 [00:17<00:14,  4.75it/s, epoch=35, grad_norm=140.529, loss=271.0638, lr=3.88e-04]
Epoch 35 TRAIN:  51%|█████▏    | 71/138 [00:17<00:14,  4.75it/s, epoch=35, grad_norm=149.668, loss=318.2355, lr=3.88e-04]
Epoch 35 TRAIN:  54%|█████▎    | 74/138 [00:17<00:13,  4.81it/s, epoch=35, grad_norm=149.668, loss=318.2355, lr=3.88e-04]
Epoch 35 TRAIN:  54%|█████▎    | 74/138 [00:17<00:13,  4.81it/s, epoch=35, grad_norm=157.388, loss=348.3951, lr=3.88e-04]
Epoch 35 TRAIN:  54%|█████▎    | 74/138 [00:18<00:13,  4.81it/s, epoch=35, grad_norm=114.197, loss=295.9005, lr=3.88e-04]
Epoch 35 TRAIN:  54%|█████▎    | 74/138 [00:18<00:13,  4.81it/s, epoch=35, grad_norm=94.681, loss=316.9340, lr=3.88e-04] 
Epoch 35 TRAIN:  56%|█████▌    | 77/138 [00:18<00:13,  4.61it/s, epoch=35, grad_norm=94.681, loss=316.9340, lr=3.88e-04]
Epoch 35 TRAIN:  56%|█████▌    | 77/138 [00:18<00:13,  4.61it/s, epoch=35, grad_norm=99.924, loss=314.6921, lr=3.88e-04]
Epoch 35 TRAIN:  56%|█████▌    | 77/138 [00:18<00:13,  4.61it/s, epoch=35, grad_norm=97.141, loss=328.0415, lr=3.88e-04]
Epoch 35 TRAIN:  56%|█████▌    | 77/138 [00:19<00:13,  4.61it/s, epoch=35, grad_norm=113.518, loss=330.3610, lr=3.88e-04]
Epoch 35 TRAIN:  58%|█████▊    | 80/138 [00:19<00:12,  4.52it/s, epoch=35, grad_norm=113.518, loss=330.3610, lr=3.88e-04]
Epoch 35 TRAIN:  58%|█████▊    | 80/138 [00:19<00:12,  4.52it/s, epoch=35, grad_norm=252.227, loss=242.0754, lr=3.88e-04]
Epoch 35 TRAIN:  58%|█████▊    | 80/138 [00:19<00:12,  4.52it/s, epoch=35, grad_norm=91.086, loss=279.0863, lr=3.88e-04] 
Epoch 35 TRAIN:  58%|█████▊    | 80/138 [00:19<00:12,  4.52it/s, epoch=35, grad_norm=186.900, loss=389.5645, lr=3.88e-04]
Epoch 35 TRAIN:  60%|██████    | 83/138 [00:19<00:11,  4.60it/s, epoch=35, grad_norm=186.900, loss=389.5645, lr=3.88e-04]
Epoch 35 TRAIN:  60%|██████    | 83/138 [00:19<00:11,  4.60it/s, epoch=35, grad_norm=126.102, loss=239.2440, lr=3.88e-04]
Epoch 35 TRAIN:  60%|██████    | 83/138 [00:20<00:11,  4.60it/s, epoch=35, grad_norm=148.023, loss=307.5800, lr=3.89e-04]
Epoch 35 TRAIN:  60%|██████    | 83/138 [00:20<00:11,  4.60it/s, epoch=35, grad_norm=163.594, loss=344.3053, lr=3.89e-04]
Epoch 35 TRAIN:  62%|██████▏   | 86/138 [00:20<00:11,  4.34it/s, epoch=35, grad_norm=163.594, loss=344.3053, lr=3.89e-04]
Epoch 35 TRAIN:  62%|██████▏   | 86/138 [00:20<00:11,  4.34it/s, epoch=35, grad_norm=84.787, loss=273.9808, lr=3.89e-04] 
Epoch 35 TRAIN:  62%|██████▏   | 86/138 [00:20<00:11,  4.34it/s, epoch=35, grad_norm=178.670, loss=238.4290, lr=3.89e-04]
Epoch 35 TRAIN:  62%|██████▏   | 86/138 [00:21<00:11,  4.34it/s, epoch=35, grad_norm=110.536, loss=316.8535, lr=3.89e-04]
Epoch 35 TRAIN:  64%|██████▍   | 89/138 [00:21<00:11,  4.39it/s, epoch=35, grad_norm=110.536, loss=316.8535, lr=3.89e-04]
Epoch 35 TRAIN:  64%|██████▍   | 89/138 [00:21<00:11,  4.39it/s, epoch=35, grad_norm=284.659, loss=366.2541, lr=3.89e-04]
Epoch 35 TRAIN:  64%|██████▍   | 89/138 [00:21<00:11,  4.39it/s, epoch=35, grad_norm=189.416, loss=331.1658, lr=3.89e-04]
Epoch 35 TRAIN:  64%|██████▍   | 89/138 [00:21<00:11,  4.39it/s, epoch=35, grad_norm=83.652, loss=246.8033, lr=3.89e-04] 
Epoch 35 TRAIN:  67%|██████▋   | 92/138 [00:21<00:10,  4.28it/s, epoch=35, grad_norm=83.652, loss=246.8033, lr=3.89e-04]
Epoch 35 TRAIN:  67%|██████▋   | 92/138 [00:22<00:10,  4.28it/s, epoch=35, grad_norm=101.614, loss=312.2047, lr=3.89e-04]
Epoch 35 TRAIN:  67%|██████▋   | 92/138 [00:22<00:10,  4.28it/s, epoch=35, grad_norm=132.302, loss=302.8978, lr=3.89e-04]
Epoch 35 TRAIN:  67%|██████▋   | 92/138 [00:22<00:10,  4.28it/s, epoch=35, grad_norm=81.236, loss=294.1913, lr=3.89e-04] 
Epoch 35 TRAIN:  69%|██████▉   | 95/138 [00:22<00:09,  4.45it/s, epoch=35, grad_norm=81.236, loss=294.1913, lr=3.89e-04]
Epoch 35 TRAIN:  69%|██████▉   | 95/138 [00:22<00:09,  4.45it/s, epoch=35, grad_norm=187.176, loss=301.6670, lr=3.89e-04]
Epoch 35 TRAIN:  69%|██████▉   | 95/138 [00:22<00:09,  4.45it/s, epoch=35, grad_norm=186.596, loss=261.2378, lr=3.90e-04]
Epoch 35 TRAIN:  69%|██████▉   | 95/138 [00:23<00:09,  4.45it/s, epoch=35, grad_norm=83.337, loss=246.2027, lr=3.90e-04] 
Epoch 35 TRAIN:  71%|███████   | 98/138 [00:23<00:08,  4.48it/s, epoch=35, grad_norm=83.337, loss=246.2027, lr=3.90e-04]
Epoch 35 TRAIN:  71%|███████   | 98/138 [00:23<00:08,  4.48it/s, epoch=35, grad_norm=111.835, loss=308.2792, lr=3.90e-04]
Epoch 35 TRAIN:  71%|███████   | 98/138 [00:23<00:08,  4.48it/s, epoch=35, grad_norm=161.434, loss=337.4805, lr=3.90e-04]2026-04-16 20:09:42,853 DEBUG TRAIN | steps/sec 4.294| Batch 35/100 loss 337.480469 loss_att 297.483582 loss_ctc 430.806458 th_accuracy 0.303371 lr 3.8976e-04 grad_norm 161.434311 rank 0

Epoch 35 TRAIN:  71%|███████   | 98/138 [00:23<00:08,  4.48it/s, epoch=35, grad_norm=103.771, loss=304.0650, lr=3.90e-04]
Epoch 35 TRAIN:  73%|███████▎  | 101/138 [00:23<00:07,  4.64it/s, epoch=35, grad_norm=103.771, loss=304.0650, lr=3.90e-04]
Epoch 35 TRAIN:  73%|███████▎  | 101/138 [00:24<00:07,  4.64it/s, epoch=35, grad_norm=92.977, loss=285.9820, lr=3.90e-04] 
Epoch 35 TRAIN:  73%|███████▎  | 101/138 [00:24<00:07,  4.64it/s, epoch=35, grad_norm=164.876, loss=262.3743, lr=3.90e-04]
Epoch 35 TRAIN:  73%|███████▎  | 101/138 [00:24<00:07,  4.64it/s, epoch=35, grad_norm=156.566, loss=369.5431, lr=3.90e-04]
Epoch 35 TRAIN:  75%|███████▌  | 104/138 [00:24<00:07,  4.66it/s, epoch=35, grad_norm=156.566, loss=369.5431, lr=3.90e-04]
Epoch 35 TRAIN:  75%|███████▌  | 104/138 [00:24<00:07,  4.66it/s, epoch=35, grad_norm=190.870, loss=243.6029, lr=3.90e-04]
Epoch 35 TRAIN:  75%|███████▌  | 104/138 [00:24<00:07,  4.66it/s, epoch=35, grad_norm=111.212, loss=227.7953, lr=3.90e-04]
Epoch 35 TRAIN:  75%|███████▌  | 104/138 [00:25<00:07,  4.66it/s, epoch=35, grad_norm=106.729, loss=300.8047, lr=3.90e-04]
Epoch 35 TRAIN:  78%|███████▊  | 107/138 [00:25<00:06,  4.73it/s, epoch=35, grad_norm=106.729, loss=300.8047, lr=3.90e-04]
Epoch 35 TRAIN:  78%|███████▊  | 107/138 [00:25<00:06,  4.73it/s, epoch=35, grad_norm=130.957, loss=284.1561, lr=3.90e-04]
Epoch 35 TRAIN:  78%|███████▊  | 107/138 [00:25<00:06,  4.73it/s, epoch=35, grad_norm=95.030, loss=306.0815, lr=3.90e-04] 
Epoch 35 TRAIN:  78%|███████▊  | 107/138 [00:25<00:06,  4.73it/s, epoch=35, grad_norm=301.501, loss=421.0848, lr=3.91e-04]
Epoch 35 TRAIN:  80%|███████▉  | 110/138 [00:25<00:05,  4.82it/s, epoch=35, grad_norm=301.501, loss=421.0848, lr=3.91e-04]
Epoch 35 TRAIN:  80%|███████▉  | 110/138 [00:25<00:05,  4.82it/s, epoch=35, grad_norm=233.656, loss=407.9858, lr=3.91e-04]
Epoch 35 TRAIN:  80%|███████▉  | 110/138 [00:26<00:05,  4.82it/s, epoch=35, grad_norm=246.160, loss=323.5474, lr=3.91e-04]
Epoch 35 TRAIN:  80%|███████▉  | 110/138 [00:26<00:05,  4.82it/s, epoch=35, grad_norm=183.537, loss=315.0008, lr=3.91e-04]
Epoch 35 TRAIN:  82%|████████▏ | 113/138 [00:26<00:05,  4.62it/s, epoch=35, grad_norm=183.537, loss=315.0008, lr=3.91e-04]
Epoch 35 TRAIN:  82%|████████▏ | 113/138 [00:26<00:05,  4.62it/s, epoch=35, grad_norm=130.116, loss=260.9413, lr=3.91e-04]
Epoch 35 TRAIN:  82%|████████▏ | 113/138 [00:26<00:05,  4.62it/s, epoch=35, grad_norm=161.136, loss=227.1965, lr=3.91e-04]
Epoch 35 TRAIN:  82%|████████▏ | 113/138 [00:26<00:05,  4.62it/s, epoch=35, grad_norm=104.170, loss=272.4019, lr=3.91e-04]
Epoch 35 TRAIN:  84%|████████▍ | 116/138 [00:26<00:04,  4.63it/s, epoch=35, grad_norm=104.170, loss=272.4019, lr=3.91e-04]
Epoch 35 TRAIN:  84%|████████▍ | 116/138 [00:27<00:04,  4.63it/s, epoch=35, grad_norm=182.321, loss=335.9360, lr=3.91e-04]
Epoch 35 TRAIN:  84%|████████▍ | 116/138 [00:27<00:04,  4.63it/s, epoch=35, grad_norm=95.348, loss=301.5975, lr=3.91e-04] 
Epoch 35 TRAIN:  84%|████████▍ | 116/138 [00:27<00:04,  4.63it/s, epoch=35, grad_norm=126.525, loss=360.1374, lr=3.91e-04]
Epoch 35 TRAIN:  86%|████████▌ | 119/138 [00:27<00:04,  4.48it/s, epoch=35, grad_norm=126.525, loss=360.1374, lr=3.91e-04]
Epoch 35 TRAIN:  86%|████████▌ | 119/138 [00:27<00:04,  4.48it/s, epoch=35, grad_norm=106.395, loss=365.1507, lr=3.91e-04]
Epoch 35 TRAIN:  86%|████████▌ | 119/138 [00:28<00:04,  4.48it/s, epoch=35, grad_norm=124.437, loss=362.5163, lr=3.91e-04]
Epoch 35 TRAIN:  86%|████████▌ | 119/138 [00:28<00:04,  4.48it/s, epoch=35, grad_norm=107.357, loss=384.3308, lr=3.92e-04]
Epoch 35 TRAIN:  88%|████████▊ | 122/138 [00:28<00:03,  4.55it/s, epoch=35, grad_norm=107.357, loss=384.3308, lr=3.92e-04]
Epoch 35 TRAIN:  88%|████████▊ | 122/138 [00:28<00:03,  4.55it/s, epoch=35, grad_norm=147.668, loss=286.7513, lr=3.92e-04]
Epoch 35 TRAIN:  88%|████████▊ | 122/138 [00:28<00:03,  4.55it/s, epoch=35, grad_norm=205.573, loss=299.2214, lr=3.92e-04]
Epoch 35 TRAIN:  88%|████████▊ | 122/138 [00:28<00:03,  4.55it/s, epoch=35, grad_norm=182.810, loss=305.9962, lr=3.92e-04]
Epoch 35 TRAIN:  91%|█████████ | 125/138 [00:28<00:02,  4.62it/s, epoch=35, grad_norm=182.810, loss=305.9962, lr=3.92e-04]
Epoch 35 TRAIN:  91%|█████████ | 125/138 [00:29<00:02,  4.62it/s, epoch=35, grad_norm=170.550, loss=334.4408, lr=3.92e-04]
Epoch 35 TRAIN:  91%|█████████ | 125/138 [00:29<00:02,  4.62it/s, epoch=35, grad_norm=109.161, loss=330.7378, lr=3.92e-04]
Epoch 35 TRAIN:  91%|█████████ | 125/138 [00:29<00:02,  4.62it/s, epoch=35, grad_norm=106.171, loss=310.2769, lr=3.92e-04]
Epoch 35 TRAIN:  93%|█████████▎| 128/138 [00:29<00:02,  4.72it/s, epoch=35, grad_norm=106.171, loss=310.2769, lr=3.92e-04]
Epoch 35 TRAIN:  93%|█████████▎| 128/138 [00:29<00:02,  4.72it/s, epoch=35, grad_norm=94.068, loss=284.2275, lr=3.92e-04] 
Epoch 35 TRAIN:  93%|█████████▎| 128/138 [00:29<00:02,  4.72it/s, epoch=35, grad_norm=131.246, loss=287.9338, lr=3.92e-04]
Epoch 35 TRAIN:  93%|█████████▎| 128/138 [00:30<00:02,  4.72it/s, epoch=35, grad_norm=271.130, loss=389.8563, lr=3.92e-04]
Epoch 35 TRAIN:  95%|█████████▍| 131/138 [00:30<00:01,  4.75it/s, epoch=35, grad_norm=271.130, loss=389.8563, lr=3.92e-04]
Epoch 35 TRAIN:  95%|█████████▍| 131/138 [00:30<00:01,  4.75it/s, epoch=35, grad_norm=405.569, loss=421.0102, lr=3.92e-04]
Epoch 35 TRAIN:  95%|█████████▍| 131/138 [00:30<00:01,  4.75it/s, epoch=35, grad_norm=133.986, loss=345.2597, lr=3.92e-04]
Epoch 35 TRAIN:  95%|█████████▍| 131/138 [00:30<00:01,  4.75it/s, epoch=35, grad_norm=161.383, loss=292.7134, lr=3.92e-04]
Epoch 35 TRAIN:  95%|█████████▍| 131/138 [00:30<00:01,  4.75it/s, epoch=35, grad_norm=114.760, loss=296.0431, lr=3.93e-04]
Epoch 35 TRAIN:  98%|█████████▊| 135/138 [00:30<00:00,  5.26it/s, epoch=35, grad_norm=114.760, loss=296.0431, lr=3.93e-04]
Epoch 35 TRAIN:  98%|█████████▊| 135/138 [00:30<00:00,  5.26it/s, epoch=35, grad_norm=267.530, loss=295.6863, lr=3.93e-04]
Epoch 35 TRAIN:  99%|█████████▊| 136/138 [00:30<00:00,  4.40it/s, epoch=35, grad_norm=267.530, loss=295.6863, lr=3.93e-04]

CV:   0%|          | 0/6 [00:00<?, ?it/s]/home/rookie/Documents/batuka/projects/wenet/wenet/wenet/utils/train_utils.py:689: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  amp_autocast(enabled=dtype is not None,

CV:  17%|█▋        | 1/6 [00:00<00:03,  1.33it/s]
                                                 
2026-04-16 20:09:51,057 INFO Epoch 35 Step 4908 CV info lr 3.9264e-04 cv_loss 496.45581609552556 rank 0 acc 0.251386230190595
2026-04-16 20:09:51,059 INFO [Rank 0] Checkpoint: save to checkpoint exp/conformer_mongolian/epoch_35.pt
2026-04-16 20:09:51,171 INFO Epoch 36 Step 4908 TRAIN info lr 3.9264e-04 rank 0
2026-04-16 20:09:51,188 INFO using accumulate grad, new batch size is 1 times larger than before

Epoch 36 TRAIN:   0%|          | 0/136 [00:00<?, ?it/s]
Epoch 36 TRAIN:   0%|          | 0/136 [00:04<?, ?it/s, epoch=36, grad_norm=338.956, loss=346.9123, lr=3.93e-04]
Epoch 36 TRAIN:   1%|          | 1/136 [00:04<09:22,  4.16s/it, epoch=36, grad_norm=338.956, loss=346.9123, lr=3.93e-04]
Epoch 36 TRAIN:   1%|          | 1/136 [00:04<09:22,  4.16s/it, epoch=36, grad_norm=498.643, loss=441.5216, lr=3.93e-04]
Epoch 36 TRAIN:   1%|          | 1/136 [00:04<09:22,  4.16s/it, epoch=36, grad_norm=209.045, loss=271.2085, lr=3.93e-04]
Epoch 36 TRAIN:   1%|          | 1/136 [00:04<09:22,  4.16s/it, epoch=36, grad_norm=182.689, loss=300.7188, lr=3.93e-04]
Epoch 36 TRAIN:   1%|          | 1/136 [00:04<09:22,  4.16s/it, epoch=36, grad_norm=123.508, loss=279.3067, lr=3.93e-04]
Epoch 36 TRAIN:   1%|          | 1/136 [00:04<09:22,  4.16s/it, epoch=36, grad_norm=283.797, loss=342.5347, lr=3.93e-04]
Epoch 36 TRAIN:   1%|          | 1/136 [00:04<09:22,  4.16s/it, epoch=36, grad_norm=279.631, loss=258.8895, lr=3.93e-04]
Epoch 36 TRAIN:   1%|          | 1/136 [00:04<09:22,  4.16s/it, epoch=36, grad_norm=167.409, loss=301.9156, lr=3.93e-04]
Epoch 36 TRAIN:   1%|          | 1/136 [00:04<09:22,  4.16s/it, epoch=36, grad_norm=201.227, loss=368.8597, lr=3.93e-04]
Epoch 36 TRAIN:   7%|▋         | 9/136 [00:04<00:51,  2.45it/s, epoch=36, grad_norm=201.227, loss=368.8597, lr=3.93e-04]
Epoch 36 TRAIN:   7%|▋         | 9/136 [00:05<00:51,  2.45it/s, epoch=36, grad_norm=89.024, loss=323.5169, lr=3.93e-04] 
Epoch 36 TRAIN:   7%|▋         | 9/136 [00:05<00:51,  2.45it/s, epoch=36, grad_norm=72.464, loss=329.3046, lr=3.94e-04]
Epoch 36 TRAIN:   7%|▋         | 9/136 [00:05<00:51,  2.45it/s, epoch=36, grad_norm=78.733, loss=311.5039, lr=3.94e-04]
Epoch 36 TRAIN:   7%|▋         | 9/136 [00:05<00:51,  2.45it/s, epoch=36, grad_norm=123.838, loss=296.0213, lr=3.94e-04]
Epoch 36 TRAIN:  10%|▉         | 13/136 [00:05<00:38,  3.16it/s, epoch=36, grad_norm=123.838, loss=296.0213, lr=3.94e-04]
Epoch 36 TRAIN:  10%|▉         | 13/136 [00:05<00:38,  3.16it/s, epoch=36, grad_norm=85.554, loss=352.8132, lr=3.94e-04] 
Epoch 36 TRAIN:  10%|▉         | 13/136 [00:05<00:38,  3.16it/s, epoch=36, grad_norm=103.122, loss=387.9597, lr=3.94e-04]
Epoch 36 TRAIN:  10%|▉         | 13/136 [00:05<00:38,  3.16it/s, epoch=36, grad_norm=77.777, loss=336.5219, lr=3.94e-04] 
Epoch 36 TRAIN:  10%|▉         | 13/136 [00:06<00:38,  3.16it/s, epoch=36, grad_norm=92.080, loss=329.8061, lr=3.94e-04]
Epoch 36 TRAIN:  12%|█▎        | 17/136 [00:06<00:30,  3.85it/s, epoch=36, grad_norm=92.080, loss=329.8061, lr=3.94e-04]
Epoch 36 TRAIN:  12%|█▎        | 17/136 [00:06<00:30,  3.85it/s, epoch=36, grad_norm=104.674, loss=304.1937, lr=3.94e-04]
Epoch 36 TRAIN:  12%|█▎        | 17/136 [00:06<00:30,  3.85it/s, epoch=36, grad_norm=82.721, loss=333.4797, lr=3.94e-04] 
Epoch 36 TRAIN:  12%|█▎        | 17/136 [00:06<00:30,  3.85it/s, epoch=36, grad_norm=114.801, loss=363.3680, lr=3.94e-04]
Epoch 36 TRAIN:  12%|█▎        | 17/136 [00:06<00:30,  3.85it/s, epoch=36, grad_norm=68.166, loss=298.0471, lr=3.94e-04] 
Epoch 36 TRAIN:  15%|█▌        | 21/136 [00:06<00:27,  4.12it/s, epoch=36, grad_norm=68.166, loss=298.0471, lr=3.94e-04]
Epoch 36 TRAIN:  15%|█▌        | 21/136 [00:07<00:27,  4.12it/s, epoch=36, grad_norm=104.651, loss=326.4038, lr=3.94e-04]
Epoch 36 TRAIN:  15%|█▌        | 21/136 [00:07<00:27,  4.12it/s, epoch=36, grad_norm=80.018, loss=302.8086, lr=3.94e-04] 
Epoch 36 TRAIN:  15%|█▌        | 21/136 [00:07<00:27,  4.12it/s, epoch=36, grad_norm=61.826, loss=286.7620, lr=3.95e-04]
Epoch 36 TRAIN:  18%|█▊        | 24/136 [00:07<00:26,  4.28it/s, epoch=36, grad_norm=61.826, loss=286.7620, lr=3.95e-04]
Epoch 36 TRAIN:  18%|█▊        | 24/136 [00:07<00:26,  4.28it/s, epoch=36, grad_norm=97.003, loss=339.8337, lr=3.95e-04]
Epoch 36 TRAIN:  18%|█▊        | 24/136 [00:07<00:26,  4.28it/s, epoch=36, grad_norm=72.061, loss=294.3401, lr=3.95e-04]
Epoch 36 TRAIN:  18%|█▊        | 24/136 [00:08<00:26,  4.28it/s, epoch=36, grad_norm=80.853, loss=294.1829, lr=3.95e-04]
Epoch 36 TRAIN:  20%|█▉        | 27/136 [00:08<00:24,  4.45it/s, epoch=36, grad_norm=80.853, loss=294.1829, lr=3.95e-04]
Epoch 36 TRAIN:  20%|█▉        | 27/136 [00:08<00:24,  4.45it/s, epoch=36, grad_norm=81.084, loss=305.4875, lr=3.95e-04]
Epoch 36 TRAIN:  20%|█▉        | 27/136 [00:08<00:24,  4.45it/s, epoch=36, grad_norm=75.191, loss=305.1196, lr=3.95e-04]
Epoch 36 TRAIN:  20%|█▉        | 27/136 [00:08<00:24,  4.45it/s, epoch=36, grad_norm=107.381, loss=341.4631, lr=3.95e-04]
Epoch 36 TRAIN:  22%|██▏       | 30/136 [00:08<00:22,  4.62it/s, epoch=36, grad_norm=107.381, loss=341.4631, lr=3.95e-04]
Epoch 36 TRAIN:  22%|██▏       | 30/136 [00:08<00:22,  4.62it/s, epoch=36, grad_norm=68.950, loss=332.4703, lr=3.95e-04] 
Epoch 36 TRAIN:  22%|██▏       | 30/136 [00:09<00:22,  4.62it/s, epoch=36, grad_norm=78.303, loss=345.3221, lr=3.95e-04]
Epoch 36 TRAIN:  22%|██▏       | 30/136 [00:09<00:22,  4.62it/s, epoch=36, grad_norm=115.601, loss=327.7839, lr=3.95e-04]
Epoch 36 TRAIN:  24%|██▍       | 33/136 [00:09<00:21,  4.76it/s, epoch=36, grad_norm=115.601, loss=327.7839, lr=3.95e-04]
Epoch 36 TRAIN:  24%|██▍       | 33/136 [00:09<00:21,  4.76it/s, epoch=36, grad_norm=131.057, loss=327.9651, lr=3.95e-04]
Epoch 36 TRAIN:  24%|██▍       | 33/136 [00:09<00:21,  4.76it/s, epoch=36, grad_norm=178.924, loss=307.1397, lr=3.95e-04]
Epoch 36 TRAIN:  24%|██▍       | 33/136 [00:10<00:21,  4.76it/s, epoch=36, grad_norm=105.525, loss=283.9880, lr=3.96e-04]
Epoch 36 TRAIN:  26%|██▋       | 36/136 [00:10<00:21,  4.59it/s, epoch=36, grad_norm=105.525, loss=283.9880, lr=3.96e-04]
Epoch 36 TRAIN:  26%|██▋       | 36/136 [00:10<00:21,  4.59it/s, epoch=36, grad_norm=113.918, loss=350.7430, lr=3.96e-04]
Epoch 36 TRAIN:  26%|██▋       | 36/136 [00:10<00:21,  4.59it/s, epoch=36, grad_norm=58.766, loss=306.5758, lr=3.96e-04] 
Epoch 36 TRAIN:  26%|██▋       | 36/136 [00:10<00:21,  4.59it/s, epoch=36, grad_norm=82.068, loss=321.5165, lr=3.96e-04]
Epoch 36 TRAIN:  29%|██▊       | 39/136 [00:10<00:20,  4.80it/s, epoch=36, grad_norm=82.068, loss=321.5165, lr=3.96e-04]
Epoch 36 TRAIN:  29%|██▊       | 39/136 [00:10<00:20,  4.80it/s, epoch=36, grad_norm=75.689, loss=320.5562, lr=3.96e-04]
Epoch 36 TRAIN:  29%|██▊       | 39/136 [00:11<00:20,  4.80it/s, epoch=36, grad_norm=66.713, loss=322.4137, lr=3.96e-04]
Epoch 36 TRAIN:  29%|██▊       | 39/136 [00:11<00:20,  4.80it/s, epoch=36, grad_norm=167.978, loss=269.5433, lr=3.96e-04]
Epoch 36 TRAIN:  31%|███       | 42/136 [00:11<00:19,  4.84it/s, epoch=36, grad_norm=167.978, loss=269.5433, lr=3.96e-04]
Epoch 36 TRAIN:  31%|███       | 42/136 [00:11<00:19,  4.84it/s, epoch=36, grad_norm=78.226, loss=336.6129, lr=3.96e-04] 
Epoch 36 TRAIN:  31%|███       | 42/136 [00:11<00:19,  4.84it/s, epoch=36, grad_norm=64.148, loss=318.3602, lr=3.96e-04]
Epoch 36 TRAIN:  31%|███       | 42/136 [00:11<00:19,  4.84it/s, epoch=36, grad_norm=64.620, loss=313.3245, lr=3.96e-04]
Epoch 36 TRAIN:  33%|███▎      | 45/136 [00:11<00:19,  4.58it/s, epoch=36, grad_norm=64.620, loss=313.3245, lr=3.96e-04]
Epoch 36 TRAIN:  33%|███▎      | 45/136 [00:12<00:19,  4.58it/s, epoch=36, grad_norm=128.200, loss=395.9091, lr=3.96e-04]
Epoch 36 TRAIN:  33%|███▎      | 45/136 [00:12<00:19,  4.58it/s, epoch=36, grad_norm=69.076, loss=305.0161, lr=3.96e-04] 
Epoch 36 TRAIN:  33%|███▎      | 45/136 [00:12<00:19,  4.58it/s, epoch=36, grad_norm=66.787, loss=301.0863, lr=3.96e-04]
Epoch 36 TRAIN:  35%|███▌      | 48/136 [00:12<00:18,  4.71it/s, epoch=36, grad_norm=66.787, loss=301.0863, lr=3.96e-04]
Epoch 36 TRAIN:  35%|███▌      | 48/136 [00:12<00:18,  4.71it/s, epoch=36, grad_norm=65.904, loss=325.3613, lr=3.97e-04]
Epoch 36 TRAIN:  35%|███▌      | 48/136 [00:12<00:18,  4.71it/s, epoch=36, grad_norm=65.502, loss=319.6399, lr=3.97e-04]
Epoch 36 TRAIN:  35%|███▌      | 48/136 [00:13<00:18,  4.71it/s, epoch=36, grad_norm=162.150, loss=260.0844, lr=3.97e-04]
Epoch 36 TRAIN:  38%|███▊      | 51/136 [00:13<00:17,  4.81it/s, epoch=36, grad_norm=162.150, loss=260.0844, lr=3.97e-04]
Epoch 36 TRAIN:  38%|███▊      | 51/136 [00:13<00:17,  4.81it/s, epoch=36, grad_norm=60.993, loss=283.2623, lr=3.97e-04] 
Epoch 36 TRAIN:  38%|███▊      | 51/136 [00:13<00:17,  4.81it/s, epoch=36, grad_norm=99.962, loss=333.7997, lr=3.97e-04]
Epoch 36 TRAIN:  38%|███▊      | 51/136 [00:13<00:17,  4.81it/s, epoch=36, grad_norm=61.873, loss=268.2401, lr=3.97e-04]
Epoch 36 TRAIN:  40%|███▉      | 54/136 [00:13<00:17,  4.81it/s, epoch=36, grad_norm=61.873, loss=268.2401, lr=3.97e-04]
Epoch 36 TRAIN:  40%|███▉      | 54/136 [00:14<00:17,  4.81it/s, epoch=36, grad_norm=77.208, loss=295.1498, lr=3.97e-04]
Epoch 36 TRAIN:  40%|███▉      | 54/136 [00:14<00:17,  4.81it/s, epoch=36, grad_norm=166.509, loss=340.8299, lr=3.97e-04]
Epoch 36 TRAIN:  40%|███▉      | 54/136 [00:14<00:17,  4.81it/s, epoch=36, grad_norm=85.233, loss=313.2365, lr=3.97e-04] 
Epoch 36 TRAIN:  42%|████▏     | 57/136 [00:14<00:16,  4.90it/s, epoch=36, grad_norm=85.233, loss=313.2365, lr=3.97e-04]
Epoch 36 TRAIN:  42%|████▏     | 57/136 [00:14<00:16,  4.90it/s, epoch=36, grad_norm=84.178, loss=320.0573, lr=3.97e-04]
Epoch 36 TRAIN:  42%|████▏     | 57/136 [00:14<00:16,  4.90it/s, epoch=36, grad_norm=94.740, loss=311.0149, lr=3.97e-04]
Epoch 36 TRAIN:  42%|████▏     | 57/136 [00:15<00:16,  4.90it/s, epoch=36, grad_norm=90.223, loss=341.7382, lr=3.97e-04]
Epoch 36 TRAIN:  44%|████▍     | 60/136 [00:15<00:15,  4.78it/s, epoch=36, grad_norm=90.223, loss=341.7382, lr=3.97e-04]
Epoch 36 TRAIN:  44%|████▍     | 60/136 [00:15<00:15,  4.78it/s, epoch=36, grad_norm=109.302, loss=270.7233, lr=3.98e-04]
Epoch 36 TRAIN:  44%|████▍     | 60/136 [00:15<00:15,  4.78it/s, epoch=36, grad_norm=134.055, loss=377.3195, lr=3.98e-04]
Epoch 36 TRAIN:  44%|████▍     | 60/136 [00:15<00:15,  4.78it/s, epoch=36, grad_norm=89.373, loss=295.6443, lr=3.98e-04] 
Epoch 36 TRAIN:  46%|████▋     | 63/136 [00:15<00:15,  4.82it/s, epoch=36, grad_norm=89.373, loss=295.6443, lr=3.98e-04]
Epoch 36 TRAIN:  46%|████▋     | 63/136 [00:15<00:15,  4.82it/s, epoch=36, grad_norm=77.699, loss=338.2371, lr=3.98e-04]
Epoch 36 TRAIN:  46%|████▋     | 63/136 [00:16<00:15,  4.82it/s, epoch=36, grad_norm=125.088, loss=281.0876, lr=3.98e-04]
Epoch 36 TRAIN:  46%|████▋     | 63/136 [00:16<00:15,  4.82it/s, epoch=36, grad_norm=127.893, loss=295.3651, lr=3.98e-04]
Epoch 36 TRAIN:  49%|████▊     | 66/136 [00:16<00:14,  4.80it/s, epoch=36, grad_norm=127.893, loss=295.3651, lr=3.98e-04]
Epoch 36 TRAIN:  49%|████▊     | 66/136 [00:16<00:14,  4.80it/s, epoch=36, grad_norm=145.591, loss=293.2565, lr=3.98e-04]
Epoch 36 TRAIN:  49%|████▊     | 66/136 [00:16<00:14,  4.80it/s, epoch=36, grad_norm=95.065, loss=282.9508, lr=3.98e-04] 
Epoch 36 TRAIN:  49%|████▊     | 66/136 [00:17<00:14,  4.80it/s, epoch=36, grad_norm=172.087, loss=376.8127, lr=3.98e-04]
Epoch 36 TRAIN:  51%|█████     | 69/136 [00:17<00:14,  4.53it/s, epoch=36, grad_norm=172.087, loss=376.8127, lr=3.98e-04]
Epoch 36 TRAIN:  51%|█████     | 69/136 [00:17<00:14,  4.53it/s, epoch=36, grad_norm=93.770, loss=322.2704, lr=3.98e-04] 
Epoch 36 TRAIN:  51%|█████     | 69/136 [00:17<00:14,  4.53it/s, epoch=36, grad_norm=87.068, loss=291.0751, lr=3.98e-04]
Epoch 36 TRAIN:  51%|█████     | 69/136 [00:17<00:14,  4.53it/s, epoch=36, grad_norm=71.096, loss=269.2041, lr=3.98e-04]
Epoch 36 TRAIN:  53%|█████▎    | 72/136 [00:17<00:14,  4.39it/s, epoch=36, grad_norm=71.096, loss=269.2041, lr=3.98e-04]
Epoch 36 TRAIN:  53%|█████▎    | 72/136 [00:17<00:14,  4.39it/s, epoch=36, grad_norm=131.894, loss=335.3301, lr=3.98e-04]
Epoch 36 TRAIN:  53%|█████▎    | 72/136 [00:18<00:14,  4.39it/s, epoch=36, grad_norm=75.523, loss=307.1440, lr=3.99e-04] 
Epoch 36 TRAIN:  53%|█████▎    | 72/136 [00:18<00:14,  4.39it/s, epoch=36, grad_norm=115.955, loss=307.1997, lr=3.99e-04]
Epoch 36 TRAIN:  55%|█████▌    | 75/136 [00:18<00:13,  4.66it/s, epoch=36, grad_norm=115.955, loss=307.1997, lr=3.99e-04]
Epoch 36 TRAIN:  55%|█████▌    | 75/136 [00:18<00:13,  4.66it/s, epoch=36, grad_norm=79.106, loss=296.9668, lr=3.99e-04] 
Epoch 36 TRAIN:  55%|█████▌    | 75/136 [00:18<00:13,  4.66it/s, epoch=36, grad_norm=132.611, loss=349.4427, lr=3.99e-04]
Epoch 36 TRAIN:  55%|█████▌    | 75/136 [00:18<00:13,  4.66it/s, epoch=36, grad_norm=99.654, loss=325.7919, lr=3.99e-04] 
Epoch 36 TRAIN:  57%|█████▋    | 78/136 [00:18<00:12,  4.76it/s, epoch=36, grad_norm=99.654, loss=325.7919, lr=3.99e-04]
Epoch 36 TRAIN:  57%|█████▋    | 78/136 [00:19<00:12,  4.76it/s, epoch=36, grad_norm=81.927, loss=306.2649, lr=3.99e-04]
Epoch 36 TRAIN:  57%|█████▋    | 78/136 [00:19<00:12,  4.76it/s, epoch=36, grad_norm=90.556, loss=331.7739, lr=3.99e-04]
Epoch 36 TRAIN:  57%|█████▋    | 78/136 [00:19<00:12,  4.76it/s, epoch=36, grad_norm=85.823, loss=368.1163, lr=3.99e-04]
Epoch 36 TRAIN:  60%|█████▉    | 81/136 [00:19<00:11,  4.70it/s, epoch=36, grad_norm=85.823, loss=368.1163, lr=3.99e-04]
Epoch 36 TRAIN:  60%|█████▉    | 81/136 [00:19<00:11,  4.70it/s, epoch=36, grad_norm=152.003, loss=301.8290, lr=3.99e-04]
Epoch 36 TRAIN:  60%|█████▉    | 81/136 [00:19<00:11,  4.70it/s, epoch=36, grad_norm=443.619, loss=256.7122, lr=3.99e-04]
Epoch 36 TRAIN:  60%|█████▉    | 81/136 [00:20<00:11,  4.70it/s, epoch=36, grad_norm=149.450, loss=351.2556, lr=3.99e-04]
Epoch 36 TRAIN:  62%|██████▏   | 84/136 [00:20<00:11,  4.50it/s, epoch=36, grad_norm=149.450, loss=351.2556, lr=3.99e-04]
Epoch 36 TRAIN:  62%|██████▏   | 84/136 [00:20<00:11,  4.50it/s, epoch=36, grad_norm=185.885, loss=296.7404, lr=3.99e-04]
Epoch 36 TRAIN:  62%|██████▏   | 84/136 [00:20<00:11,  4.50it/s, epoch=36, grad_norm=175.863, loss=246.9380, lr=4.00e-04]
Epoch 36 TRAIN:  62%|██████▏   | 84/136 [00:20<00:11,  4.50it/s, epoch=36, grad_norm=102.341, loss=282.6335, lr=4.00e-04]
Epoch 36 TRAIN:  64%|██████▍   | 87/136 [00:20<00:10,  4.79it/s, epoch=36, grad_norm=102.341, loss=282.6335, lr=4.00e-04]
Epoch 36 TRAIN:  64%|██████▍   | 87/136 [00:21<00:10,  4.79it/s, epoch=36, grad_norm=180.605, loss=306.7245, lr=4.00e-04]
Epoch 36 TRAIN:  64%|██████▍   | 87/136 [00:21<00:10,  4.79it/s, epoch=36, grad_norm=199.479, loss=324.2044, lr=4.00e-04]
Epoch 36 TRAIN:  64%|██████▍   | 87/136 [00:21<00:10,  4.79it/s, epoch=36, grad_norm=147.049, loss=314.2032, lr=4.00e-04]
Epoch 36 TRAIN:  66%|██████▌   | 90/136 [00:21<00:10,  4.44it/s, epoch=36, grad_norm=147.049, loss=314.2032, lr=4.00e-04]
Epoch 36 TRAIN:  66%|██████▌   | 90/136 [00:21<00:10,  4.44it/s, epoch=36, grad_norm=173.658, loss=237.8609, lr=4.00e-04]
Epoch 36 TRAIN:  66%|██████▌   | 90/136 [00:22<00:10,  4.44it/s, epoch=36, grad_norm=132.281, loss=226.3411, lr=4.00e-04]
Epoch 36 TRAIN:  66%|██████▌   | 90/136 [00:22<00:10,  4.44it/s, epoch=36, grad_norm=107.489, loss=234.4919, lr=4.00e-04]
Epoch 36 TRAIN:  68%|██████▊   | 93/136 [00:22<00:09,  4.55it/s, epoch=36, grad_norm=107.489, loss=234.4919, lr=4.00e-04]
Epoch 36 TRAIN:  68%|██████▊   | 93/136 [00:22<00:09,  4.55it/s, epoch=36, grad_norm=266.353, loss=390.9923, lr=4.00e-04]
Epoch 36 TRAIN:  68%|██████▊   | 93/136 [00:22<00:09,  4.55it/s, epoch=36, grad_norm=128.311, loss=336.2912, lr=4.00e-04]
Epoch 36 TRAIN:  68%|██████▊   | 93/136 [00:22<00:09,  4.55it/s, epoch=36, grad_norm=309.726, loss=401.2510, lr=4.00e-04]
Epoch 36 TRAIN:  71%|███████   | 96/136 [00:22<00:08,  4.73it/s, epoch=36, grad_norm=309.726, loss=401.2510, lr=4.00e-04]
Epoch 36 TRAIN:  71%|███████   | 96/136 [00:23<00:08,  4.73it/s, epoch=36, grad_norm=99.227, loss=287.8441, lr=4.00e-04] 
Epoch 36 TRAIN:  71%|███████   | 96/136 [00:23<00:08,  4.73it/s, epoch=36, grad_norm=91.817, loss=292.2219, lr=4.00e-04]
Epoch 36 TRAIN:  71%|███████   | 96/136 [00:23<00:08,  4.73it/s, epoch=36, grad_norm=145.649, loss=329.2329, lr=4.01e-04]
Epoch 36 TRAIN:  73%|███████▎  | 99/136 [00:23<00:07,  4.86it/s, epoch=36, grad_norm=145.649, loss=329.2329, lr=4.01e-04]
Epoch 36 TRAIN:  73%|███████▎  | 99/136 [00:23<00:07,  4.86it/s, epoch=36, grad_norm=146.059, loss=291.6409, lr=4.01e-04]2026-04-16 20:10:14,813 DEBUG TRAIN | steps/sec 4.255| Batch 36/100 loss 291.640900 loss_att 256.809448 loss_ctc 372.914246 th_accuracy 0.272277 lr 4.0064e-04 grad_norm 146.058517 rank 0

Epoch 36 TRAIN:  73%|███████▎  | 99/136 [00:23<00:07,  4.86it/s, epoch=36, grad_norm=239.050, loss=195.5854, lr=4.01e-04]
Epoch 36 TRAIN:  73%|███████▎  | 99/136 [00:24<00:07,  4.86it/s, epoch=36, grad_norm=81.801, loss=293.9756, lr=4.01e-04] 
Epoch 36 TRAIN:  75%|███████▌  | 102/136 [00:24<00:06,  4.90it/s, epoch=36, grad_norm=81.801, loss=293.9756, lr=4.01e-04]
Epoch 36 TRAIN:  75%|███████▌  | 102/136 [00:24<00:06,  4.90it/s, epoch=36, grad_norm=131.657, loss=361.2707, lr=4.01e-04]
Epoch 36 TRAIN:  75%|███████▌  | 102/136 [00:24<00:06,  4.90it/s, epoch=36, grad_norm=104.787, loss=272.3351, lr=4.01e-04]
Epoch 36 TRAIN:  75%|███████▌  | 102/136 [00:24<00:06,  4.90it/s, epoch=36, grad_norm=74.518, loss=239.4292, lr=4.01e-04] 
Epoch 36 TRAIN:  77%|███████▋  | 105/136 [00:24<00:06,  4.79it/s, epoch=36, grad_norm=74.518, loss=239.4292, lr=4.01e-04]
Epoch 36 TRAIN:  77%|███████▋  | 105/136 [00:24<00:06,  4.79it/s, epoch=36, grad_norm=148.365, loss=367.7278, lr=4.01e-04]
Epoch 36 TRAIN:  77%|███████▋  | 105/136 [00:25<00:06,  4.79it/s, epoch=36, grad_norm=111.172, loss=334.6509, lr=4.01e-04]
Epoch 36 TRAIN:  77%|███████▋  | 105/136 [00:25<00:06,  4.79it/s, epoch=36, grad_norm=91.253, loss=299.1639, lr=4.01e-04] 
Epoch 36 TRAIN:  79%|███████▉  | 108/136 [00:25<00:06,  4.64it/s, epoch=36, grad_norm=91.253, loss=299.1639, lr=4.01e-04]
Epoch 36 TRAIN:  79%|███████▉  | 108/136 [00:25<00:06,  4.64it/s, epoch=36, grad_norm=134.923, loss=272.9399, lr=4.01e-04]
Epoch 36 TRAIN:  79%|███████▉  | 108/136 [00:25<00:06,  4.64it/s, epoch=36, grad_norm=66.250, loss=289.4711, lr=4.01e-04] 
Epoch 36 TRAIN:  79%|███████▉  | 108/136 [00:26<00:06,  4.64it/s, epoch=36, grad_norm=103.268, loss=304.4878, lr=4.02e-04]
Epoch 36 TRAIN:  82%|████████▏ | 111/136 [00:26<00:05,  4.43it/s, epoch=36, grad_norm=103.268, loss=304.4878, lr=4.02e-04]
Epoch 36 TRAIN:  82%|████████▏ | 111/136 [00:26<00:05,  4.43it/s, epoch=36, grad_norm=235.656, loss=395.8691, lr=4.02e-04]
Epoch 36 TRAIN:  82%|████████▏ | 111/136 [00:26<00:05,  4.43it/s, epoch=36, grad_norm=111.020, loss=291.6821, lr=4.02e-04]
Epoch 36 TRAIN:  82%|████████▏ | 111/136 [00:26<00:05,  4.43it/s, epoch=36, grad_norm=104.031, loss=315.3844, lr=4.02e-04]
Epoch 36 TRAIN:  84%|████████▍ | 114/136 [00:26<00:04,  4.62it/s, epoch=36, grad_norm=104.031, loss=315.3844, lr=4.02e-04]
Epoch 36 TRAIN:  84%|████████▍ | 114/136 [00:26<00:04,  4.62it/s, epoch=36, grad_norm=239.349, loss=288.4915, lr=4.02e-04]
Epoch 36 TRAIN:  84%|████████▍ | 114/136 [00:26<00:04,  4.62it/s, epoch=36, grad_norm=98.148, loss=300.6013, lr=4.02e-04] 
Epoch 36 TRAIN:  84%|████████▍ | 114/136 [00:27<00:04,  4.62it/s, epoch=36, grad_norm=115.137, loss=315.7990, lr=4.02e-04]
Epoch 36 TRAIN:  84%|████████▍ | 114/136 [00:27<00:04,  4.62it/s, epoch=36, grad_norm=140.347, loss=328.0323, lr=4.02e-04]
Epoch 36 TRAIN:  87%|████████▋ | 118/136 [00:27<00:03,  5.22it/s, epoch=36, grad_norm=140.347, loss=328.0323, lr=4.02e-04]
Epoch 36 TRAIN:  87%|████████▋ | 118/136 [00:27<00:03,  5.22it/s, epoch=36, grad_norm=124.897, loss=326.9631, lr=4.02e-04]
Epoch 36 TRAIN:  87%|████████▋ | 118/136 [00:27<00:03,  5.22it/s, epoch=36, grad_norm=148.145, loss=313.1055, lr=4.02e-04]
Epoch 36 TRAIN:  87%|████████▋ | 118/136 [00:28<00:03,  5.22it/s, epoch=36, grad_norm=142.774, loss=312.8024, lr=4.02e-04]
Epoch 36 TRAIN:  89%|████████▉ | 121/136 [00:28<00:03,  4.84it/s, epoch=36, grad_norm=142.774, loss=312.8024, lr=4.02e-04]
Epoch 36 TRAIN:  89%|████████▉ | 121/136 [00:28<00:03,  4.84it/s, epoch=36, grad_norm=190.539, loss=252.3806, lr=4.02e-04]
Epoch 36 TRAIN:  89%|████████▉ | 121/136 [00:28<00:03,  4.84it/s, epoch=36, grad_norm=134.720, loss=273.9505, lr=4.02e-04]
Epoch 36 TRAIN:  89%|████████▉ | 121/136 [00:28<00:03,  4.84it/s, epoch=36, grad_norm=110.342, loss=339.7642, lr=4.03e-04]
Epoch 36 TRAIN:  91%|█████████ | 124/136 [00:28<00:02,  5.06it/s, epoch=36, grad_norm=110.342, loss=339.7642, lr=4.03e-04]
Epoch 36 TRAIN:  91%|█████████ | 124/136 [00:28<00:02,  5.06it/s, epoch=36, grad_norm=120.729, loss=329.1306, lr=4.03e-04]
Epoch 36 TRAIN:  91%|█████████ | 124/136 [00:29<00:02,  5.06it/s, epoch=36, grad_norm=153.391, loss=255.0077, lr=4.03e-04]
Epoch 36 TRAIN:  91%|█████████ | 124/136 [00:29<00:02,  5.06it/s, epoch=36, grad_norm=165.409, loss=354.3828, lr=4.03e-04]
Epoch 36 TRAIN:  93%|█████████▎| 127/136 [00:29<00:01,  4.89it/s, epoch=36, grad_norm=165.409, loss=354.3828, lr=4.03e-04]
Epoch 36 TRAIN:  93%|█████████▎| 127/136 [00:29<00:01,  4.89it/s, epoch=36, grad_norm=125.335, loss=359.1954, lr=4.03e-04]
Epoch 36 TRAIN:  93%|█████████▎| 127/136 [00:29<00:01,  4.89it/s, epoch=36, grad_norm=163.160, loss=402.4222, lr=4.03e-04]
Epoch 36 TRAIN:  93%|█████████▎| 127/136 [00:29<00:01,  4.89it/s, epoch=36, grad_norm=108.418, loss=335.3613, lr=4.03e-04]
Epoch 36 TRAIN:  96%|█████████▌| 130/136 [00:29<00:01,  4.79it/s, epoch=36, grad_norm=108.418, loss=335.3613, lr=4.03e-04]
Epoch 36 TRAIN:  96%|█████████▌| 130/136 [00:30<00:01,  4.79it/s, epoch=36, grad_norm=135.711, loss=356.3077, lr=4.03e-04]
Epoch 36 TRAIN:  96%|█████████▌| 130/136 [00:30<00:01,  4.79it/s, epoch=36, grad_norm=116.376, loss=357.5790, lr=4.03e-04]
Epoch 36 TRAIN:  96%|█████████▌| 130/136 [00:30<00:01,  4.79it/s, epoch=36, grad_norm=150.921, loss=349.9260, lr=4.03e-04]
Epoch 36 TRAIN:  98%|█████████▊| 133/136 [00:30<00:00,  4.47it/s, epoch=36, grad_norm=150.921, loss=349.9260, lr=4.03e-04]
Epoch 36 TRAIN:  98%|█████████▊| 133/136 [00:30<00:00,  4.47it/s, epoch=36, grad_norm=180.632, loss=365.0978, lr=4.03e-04]
Epoch 36 TRAIN:  98%|█████████▊| 133/136 [00:31<00:00,  4.47it/s, epoch=36, grad_norm=176.072, loss=273.5461, lr=4.03e-04]
Epoch 36 TRAIN:  98%|█████████▊| 133/136 [00:31<00:00,  4.47it/s, epoch=36, grad_norm=157.981, loss=408.3109, lr=4.04e-04]
Epoch 36 TRAIN: 100%|██████████| 136/136 [00:31<00:00,  4.76it/s, epoch=36, grad_norm=157.981, loss=408.3109, lr=4.04e-04]
Epoch 36 TRAIN: 100%|██████████| 136/136 [00:31<00:00,  4.36it/s, epoch=36, grad_norm=157.981, loss=408.3109, lr=4.04e-04]

CV:   0%|          | 0/6 [00:00<?, ?it/s]/home/rookie/Documents/batuka/projects/wenet/wenet/wenet/utils/train_utils.py:689: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  amp_autocast(enabled=dtype is not None,

CV:  17%|█▋        | 1/6 [00:00<00:03,  1.30it/s]
                                                 
2026-04-16 20:10:23,245 INFO Epoch 36 Step 5044 CV info lr 4.0352e-04 cv_loss 461.08914462002843 rank 0 acc 0.2533595412969589
2026-04-16 20:10:23,246 INFO [Rank 0] Checkpoint: save to checkpoint exp/conformer_mongolian/epoch_36.pt
2026-04-16 20:10:23,370 INFO Epoch 37 Step 5044 TRAIN info lr 4.0352e-04 rank 0
2026-04-16 20:10:23,392 INFO using accumulate grad, new batch size is 1 times larger than before

Epoch 37 TRAIN:   0%|          | 0/136 [00:00<?, ?it/s]
Epoch 37 TRAIN:   0%|          | 0/136 [00:04<?, ?it/s, epoch=37, grad_norm=502.064, loss=448.8105, lr=4.04e-04]
Epoch 37 TRAIN:   1%|          | 1/136 [00:04<09:00,  4.01s/it, epoch=37, grad_norm=502.064, loss=448.8105, lr=4.04e-04]
Epoch 37 TRAIN:   1%|          | 1/136 [00:04<09:00,  4.01s/it, epoch=37, grad_norm=260.806, loss=307.8860, lr=4.04e-04]
Epoch 37 TRAIN:   1%|          | 1/136 [00:04<09:00,  4.01s/it, epoch=37, grad_norm=371.524, loss=377.8302, lr=4.04e-04]
Epoch 37 TRAIN:   1%|          | 1/136 [00:04<09:00,  4.01s/it, epoch=37, grad_norm=344.892, loss=394.9481, lr=4.04e-04]
Epoch 37 TRAIN:   1%|          | 1/136 [00:04<09:00,  4.01s/it, epoch=37, grad_norm=306.066, loss=334.0399, lr=4.04e-04]
Epoch 37 TRAIN:   1%|          | 1/136 [00:04<09:00,  4.01s/it, epoch=37, grad_norm=93.013, loss=232.5250, lr=4.04e-04] 
Epoch 37 TRAIN:   1%|          | 1/136 [00:04<09:00,  4.01s/it, epoch=37, grad_norm=96.739, loss=225.9119, lr=4.04e-04]
Epoch 37 TRAIN:   1%|          | 1/136 [00:04<09:00,  4.01s/it, epoch=37, grad_norm=116.687, loss=382.6777, lr=4.04e-04]
Epoch 37 TRAIN:   6%|▌         | 8/136 [00:04<00:54,  2.33it/s, epoch=37, grad_norm=116.687, loss=382.6777, lr=4.04e-04]
Epoch 37 TRAIN:   6%|▌         | 8/136 [00:04<00:54,  2.33it/s, epoch=37, grad_norm=112.636, loss=336.0213, lr=4.04e-04]
Epoch 37 TRAIN:   6%|▌         | 8/136 [00:04<00:54,  2.33it/s, epoch=37, grad_norm=72.950, loss=308.7309, lr=4.04e-04] 
Epoch 37 TRAIN:   6%|▌         | 8/136 [00:05<00:54,  2.33it/s, epoch=37, grad_norm=89.757, loss=320.2003, lr=4.04e-04]
Epoch 37 TRAIN:   6%|▌         | 8/136 [00:05<00:54,  2.33it/s, epoch=37, grad_norm=136.133, loss=296.2909, lr=4.04e-04]
Epoch 37 TRAIN:   6%|▌         | 8/136 [00:05<00:54,  2.33it/s, epoch=37, grad_norm=88.875, loss=315.8718, lr=4.05e-04] 
Epoch 37 TRAIN:  10%|▉         | 13/136 [00:05<00:37,  3.28it/s, epoch=37, grad_norm=88.875, loss=315.8718, lr=4.05e-04]
Epoch 37 TRAIN:  10%|▉         | 13/136 [00:05<00:37,  3.28it/s, epoch=37, grad_norm=67.344, loss=323.0415, lr=4.05e-04]
Epoch 37 TRAIN:  10%|▉         | 13/136 [00:05<00:37,  3.28it/s, epoch=37, grad_norm=76.151, loss=316.5079, lr=4.05e-04]
Epoch 37 TRAIN:  10%|▉         | 13/136 [00:05<00:37,  3.28it/s, epoch=37, grad_norm=64.766, loss=325.4915, lr=4.05e-04]
Epoch 37 TRAIN:  10%|▉         | 13/136 [00:06<00:37,  3.28it/s, epoch=37, grad_norm=73.086, loss=333.1616, lr=4.05e-04]
Epoch 37 TRAIN:  12%|█▎        | 17/136 [00:06<00:30,  3.94it/s, epoch=37, grad_norm=73.086, loss=333.1616, lr=4.05e-04]
Epoch 37 TRAIN:  12%|█▎        | 17/136 [00:06<00:30,  3.94it/s, epoch=37, grad_norm=54.402, loss=320.1763, lr=4.05e-04]
Epoch 37 TRAIN:  12%|█▎        | 17/136 [00:06<00:30,  3.94it/s, epoch=37, grad_norm=61.540, loss=306.6049, lr=4.05e-04]
Epoch 37 TRAIN:  12%|█▎        | 17/136 [00:06<00:30,  3.94it/s, epoch=37, grad_norm=85.344, loss=282.1032, lr=4.05e-04]
Epoch 37 TRAIN:  12%|█▎        | 17/136 [00:06<00:30,  3.94it/s, epoch=37, grad_norm=91.231, loss=269.5283, lr=4.05e-04]
Epoch 37 TRAIN:  15%|█▌        | 21/136 [00:06<00:26,  4.30it/s, epoch=37, grad_norm=91.231, loss=269.5283, lr=4.05e-04]
Epoch 37 TRAIN:  15%|█▌        | 21/136 [00:06<00:26,  4.30it/s, epoch=37, grad_norm=73.841, loss=311.4576, lr=4.05e-04]
Epoch 37 TRAIN:  15%|█▌        | 21/136 [00:07<00:26,  4.30it/s, epoch=37, grad_norm=66.342, loss=283.1906, lr=4.05e-04]
Epoch 37 TRAIN:  15%|█▌        | 21/136 [00:07<00:26,  4.30it/s, epoch=37, grad_norm=184.793, loss=384.1899, lr=4.05e-04]
Epoch 37 TRAIN:  15%|█▌        | 21/136 [00:07<00:26,  4.30it/s, epoch=37, grad_norm=95.562, loss=332.8122, lr=4.06e-04] 
Epoch 37 TRAIN:  18%|█▊        | 25/136 [00:07<00:24,  4.59it/s, epoch=37, grad_norm=95.562, loss=332.8122, lr=4.06e-04]
Epoch 37 TRAIN:  18%|█▊        | 25/136 [00:07<00:24,  4.59it/s, epoch=37, grad_norm=125.434, loss=350.5001, lr=4.06e-04]
Epoch 37 TRAIN:  18%|█▊        | 25/136 [00:07<00:24,  4.59it/s, epoch=37, grad_norm=114.337, loss=361.6779, lr=4.06e-04]
Epoch 37 TRAIN:  18%|█▊        | 25/136 [00:08<00:24,  4.59it/s, epoch=37, grad_norm=66.115, loss=323.1265, lr=4.06e-04] 
Epoch 37 TRAIN:  21%|██        | 28/136 [00:08<00:22,  4.75it/s, epoch=37, grad_norm=66.115, loss=323.1265, lr=4.06e-04]
Epoch 37 TRAIN:  21%|██        | 28/136 [00:08<00:22,  4.75it/s, epoch=37, grad_norm=78.220, loss=322.3462, lr=4.06e-04]
Epoch 37 TRAIN:  21%|██        | 28/136 [00:08<00:22,  4.75it/s, epoch=37, grad_norm=70.320, loss=325.9665, lr=4.06e-04]
Epoch 37 TRAIN:  21%|██        | 28/136 [00:08<00:22,  4.75it/s, epoch=37, grad_norm=137.577, loss=295.7894, lr=4.06e-04]
Epoch 37 TRAIN:  23%|██▎       | 31/136 [00:08<00:21,  4.80it/s, epoch=37, grad_norm=137.577, loss=295.7894, lr=4.06e-04]
Epoch 37 TRAIN:  23%|██▎       | 31/136 [00:08<00:21,  4.80it/s, epoch=37, grad_norm=87.781, loss=328.2162, lr=4.06e-04] 
Epoch 37 TRAIN:  23%|██▎       | 31/136 [00:09<00:21,  4.80it/s, epoch=37, grad_norm=73.700, loss=348.0430, lr=4.06e-04]
Epoch 37 TRAIN:  23%|██▎       | 31/136 [00:09<00:21,  4.80it/s, epoch=37, grad_norm=65.820, loss=320.9152, lr=4.06e-04]
Epoch 37 TRAIN:  25%|██▌       | 34/136 [00:09<00:20,  4.99it/s, epoch=37, grad_norm=65.820, loss=320.9152, lr=4.06e-04]
Epoch 37 TRAIN:  25%|██▌       | 34/136 [00:09<00:20,  4.99it/s, epoch=37, grad_norm=80.838, loss=299.3768, lr=4.06e-04]
Epoch 37 TRAIN:  25%|██▌       | 34/136 [00:09<00:20,  4.99it/s, epoch=37, grad_norm=86.949, loss=360.3032, lr=4.06e-04]
Epoch 37 TRAIN:  25%|██▌       | 34/136 [00:09<00:20,  4.99it/s, epoch=37, grad_norm=74.390, loss=335.8746, lr=4.06e-04]
Epoch 37 TRAIN:  27%|██▋       | 37/136 [00:09<00:19,  5.00it/s, epoch=37, grad_norm=74.390, loss=335.8746, lr=4.06e-04]
Epoch 37 TRAIN:  27%|██▋       | 37/136 [00:10<00:19,  5.00it/s, epoch=37, grad_norm=67.052, loss=285.8252, lr=4.07e-04]
Epoch 37 TRAIN:  27%|██▋       | 37/136 [00:10<00:19,  5.00it/s, epoch=37, grad_norm=100.535, loss=349.5142, lr=4.07e-04]
Epoch 37 TRAIN:  27%|██▋       | 37/136 [00:10<00:19,  5.00it/s, epoch=37, grad_norm=86.786, loss=333.0751, lr=4.07e-04] 
Epoch 37 TRAIN:  29%|██▉       | 40/136 [00:10<00:19,  4.85it/s, epoch=37, grad_norm=86.786, loss=333.0751, lr=4.07e-04]
Epoch 37 TRAIN:  29%|██▉       | 40/136 [00:10<00:19,  4.85it/s, epoch=37, grad_norm=76.619, loss=305.2550, lr=4.07e-04]
Epoch 37 TRAIN:  29%|██▉       | 40/136 [00:10<00:19,  4.85it/s, epoch=37, grad_norm=66.566, loss=300.8658, lr=4.07e-04]
Epoch 37 TRAIN:  29%|██▉       | 40/136 [00:11<00:19,  4.85it/s, epoch=37, grad_norm=137.995, loss=257.5371, lr=4.07e-04]
Epoch 37 TRAIN:  32%|███▏      | 43/136 [00:11<00:18,  4.96it/s, epoch=37, grad_norm=137.995, loss=257.5371, lr=4.07e-04]
Epoch 37 TRAIN:  32%|███▏      | 43/136 [00:11<00:18,  4.96it/s, epoch=37, grad_norm=93.327, loss=344.0492, lr=4.07e-04] 
Epoch 37 TRAIN:  32%|███▏      | 43/136 [00:11<00:18,  4.96it/s, epoch=37, grad_norm=67.750, loss=332.0794, lr=4.07e-04]
Epoch 37 TRAIN:  32%|███▏      | 43/136 [00:11<00:18,  4.96it/s, epoch=37, grad_norm=59.300, loss=300.2353, lr=4.07e-04]
Epoch 37 TRAIN:  34%|███▍      | 46/136 [00:11<00:18,  4.87it/s, epoch=37, grad_norm=59.300, loss=300.2353, lr=4.07e-04]
Epoch 37 TRAIN:  34%|███▍      | 46/136 [00:11<00:18,  4.87it/s, epoch=37, grad_norm=73.609, loss=354.6330, lr=4.07e-04]
Epoch 37 TRAIN:  34%|███▍      | 46/136 [00:12<00:18,  4.87it/s, epoch=37, grad_norm=82.687, loss=337.6651, lr=4.07e-04]
Epoch 37 TRAIN:  34%|███▍      | 46/136 [00:12<00:18,  4.87it/s, epoch=37, grad_norm=81.483, loss=320.8304, lr=4.07e-04]
Epoch 37 TRAIN:  36%|███▌      | 49/136 [00:12<00:18,  4.79it/s, epoch=37, grad_norm=81.483, loss=320.8304, lr=4.07e-04]
Epoch 37 TRAIN:  36%|███▌      | 49/136 [00:12<00:18,  4.79it/s, epoch=37, grad_norm=73.668, loss=341.1755, lr=4.08e-04]
Epoch 37 TRAIN:  36%|███▌      | 49/136 [00:12<00:18,  4.79it/s, epoch=37, grad_norm=64.251, loss=326.8398, lr=4.08e-04]
Epoch 37 TRAIN:  36%|███▌      | 49/136 [00:13<00:18,  4.79it/s, epoch=37, grad_norm=121.857, loss=274.3634, lr=4.08e-04]
Epoch 37 TRAIN:  38%|███▊      | 52/136 [00:13<00:18,  4.66it/s, epoch=37, grad_norm=121.857, loss=274.3634, lr=4.08e-04]
Epoch 37 TRAIN:  38%|███▊      | 52/136 [00:13<00:18,  4.66it/s, epoch=37, grad_norm=75.639, loss=330.1754, lr=4.08e-04] 
Epoch 37 TRAIN:  38%|███▊      | 52/136 [00:13<00:18,  4.66it/s, epoch=37, grad_norm=91.414, loss=275.3395, lr=4.08e-04]
Epoch 37 TRAIN:  38%|███▊      | 52/136 [00:13<00:18,  4.66it/s, epoch=37, grad_norm=78.003, loss=318.3880, lr=4.08e-04]
Epoch 37 TRAIN:  40%|████      | 55/136 [00:13<00:17,  4.67it/s, epoch=37, grad_norm=78.003, loss=318.3880, lr=4.08e-04]
Epoch 37 TRAIN:  40%|████      | 55/136 [00:13<00:17,  4.67it/s, epoch=37, grad_norm=67.938, loss=294.9079, lr=4.08e-04]
Epoch 37 TRAIN:  40%|████      | 55/136 [00:14<00:17,  4.67it/s, epoch=37, grad_norm=86.082, loss=318.4663, lr=4.08e-04]
Epoch 37 TRAIN:  40%|████      | 55/136 [00:14<00:17,  4.67it/s, epoch=37, grad_norm=116.735, loss=356.0123, lr=4.08e-04]
Epoch 37 TRAIN:  43%|████▎     | 58/136 [00:14<00:16,  4.79it/s, epoch=37, grad_norm=116.735, loss=356.0123, lr=4.08e-04]
Epoch 37 TRAIN:  43%|████▎     | 58/136 [00:14<00:16,  4.79it/s, epoch=37, grad_norm=81.821, loss=314.7430, lr=4.08e-04] 
Epoch 37 TRAIN:  43%|████▎     | 58/136 [00:14<00:16,  4.79it/s, epoch=37, grad_norm=106.168, loss=348.0076, lr=4.08e-04]
Epoch 37 TRAIN:  43%|████▎     | 58/136 [00:14<00:16,  4.79it/s, epoch=37, grad_norm=71.241, loss=331.4703, lr=4.08e-04] 
Epoch 37 TRAIN:  45%|████▍     | 61/136 [00:14<00:15,  4.93it/s, epoch=37, grad_norm=71.241, loss=331.4703, lr=4.08e-04]
Epoch 37 TRAIN:  45%|████▍     | 61/136 [00:15<00:15,  4.93it/s, epoch=37, grad_norm=94.183, loss=320.3431, lr=4.08e-04]
Epoch 37 TRAIN:  45%|████▍     | 61/136 [00:15<00:15,  4.93it/s, epoch=37, grad_norm=177.967, loss=295.5245, lr=4.09e-04]
Epoch 37 TRAIN:  45%|████▍     | 61/136 [00:15<00:15,  4.93it/s, epoch=37, grad_norm=102.325, loss=309.5412, lr=4.09e-04]
Epoch 37 TRAIN:  47%|████▋     | 64/136 [00:15<00:15,  4.59it/s, epoch=37, grad_norm=102.325, loss=309.5412, lr=4.09e-04]
Epoch 37 TRAIN:  47%|████▋     | 64/136 [00:15<00:15,  4.59it/s, epoch=37, grad_norm=240.769, loss=258.3713, lr=4.09e-04]
Epoch 37 TRAIN:  47%|████▋     | 64/136 [00:15<00:15,  4.59it/s, epoch=37, grad_norm=87.338, loss=329.4534, lr=4.09e-04] 
Epoch 37 TRAIN:  47%|████▋     | 64/136 [00:16<00:15,  4.59it/s, epoch=37, grad_norm=79.749, loss=315.1572, lr=4.09e-04]
Epoch 37 TRAIN:  49%|████▉     | 67/136 [00:16<00:14,  4.69it/s, epoch=37, grad_norm=79.749, loss=315.1572, lr=4.09e-04]
Epoch 37 TRAIN:  49%|████▉     | 67/136 [00:16<00:14,  4.69it/s, epoch=37, grad_norm=82.349, loss=314.6475, lr=4.09e-04]
Epoch 37 TRAIN:  49%|████▉     | 67/136 [00:16<00:14,  4.69it/s, epoch=37, grad_norm=112.708, loss=355.6042, lr=4.09e-04]
Epoch 37 TRAIN:  49%|████▉     | 67/136 [00:16<00:14,  4.69it/s, epoch=37, grad_norm=152.187, loss=350.0661, lr=4.09e-04]
Epoch 37 TRAIN:  51%|█████▏    | 70/136 [00:16<00:14,  4.58it/s, epoch=37, grad_norm=152.187, loss=350.0661, lr=4.09e-04]
Epoch 37 TRAIN:  51%|█████▏    | 70/136 [00:17<00:14,  4.58it/s, epoch=37, grad_norm=148.111, loss=240.8046, lr=4.09e-04]
Epoch 37 TRAIN:  51%|█████▏    | 70/136 [00:17<00:14,  4.58it/s, epoch=37, grad_norm=120.537, loss=329.4755, lr=4.09e-04]
Epoch 37 TRAIN:  51%|█████▏    | 70/136 [00:17<00:14,  4.58it/s, epoch=37, grad_norm=112.108, loss=368.7359, lr=4.09e-04]
Epoch 37 TRAIN:  54%|█████▎    | 73/136 [00:17<00:13,  4.70it/s, epoch=37, grad_norm=112.108, loss=368.7359, lr=4.09e-04]
Epoch 37 TRAIN:  54%|█████▎    | 73/136 [00:17<00:13,  4.70it/s, epoch=37, grad_norm=89.805, loss=319.9205, lr=4.09e-04] 
Epoch 37 TRAIN:  54%|█████▎    | 73/136 [00:17<00:13,  4.70it/s, epoch=37, grad_norm=101.653, loss=336.3262, lr=4.10e-04]
Epoch 37 TRAIN:  54%|█████▎    | 73/136 [00:18<00:13,  4.70it/s, epoch=37, grad_norm=88.277, loss=290.2812, lr=4.10e-04] 
Epoch 37 TRAIN:  56%|█████▌    | 76/136 [00:18<00:12,  4.65it/s, epoch=37, grad_norm=88.277, loss=290.2812, lr=4.10e-04]
Epoch 37 TRAIN:  56%|█████▌    | 76/136 [00:18<00:12,  4.65it/s, epoch=37, grad_norm=231.931, loss=222.2754, lr=4.10e-04]
Epoch 37 TRAIN:  56%|█████▌    | 76/136 [00:18<00:12,  4.65it/s, epoch=37, grad_norm=79.575, loss=336.3067, lr=4.10e-04] 
Epoch 37 TRAIN:  56%|█████▌    | 76/136 [00:18<00:12,  4.65it/s, epoch=37, grad_norm=219.090, loss=257.4493, lr=4.10e-04]
Epoch 37 TRAIN:  58%|█████▊    | 79/136 [00:18<00:11,  4.75it/s, epoch=37, grad_norm=219.090, loss=257.4493, lr=4.10e-04]
Epoch 37 TRAIN:  58%|█████▊    | 79/136 [00:18<00:11,  4.75it/s, epoch=37, grad_norm=125.587, loss=247.1955, lr=4.10e-04]
Epoch 37 TRAIN:  58%|█████▊    | 79/136 [00:19<00:11,  4.75it/s, epoch=37, grad_norm=163.418, loss=284.0877, lr=4.10e-04]
Epoch 37 TRAIN:  58%|█████▊    | 79/136 [00:19<00:11,  4.75it/s, epoch=37, grad_norm=74.758, loss=298.5559, lr=4.10e-04] 
Epoch 37 TRAIN:  60%|██████    | 82/136 [00:19<00:11,  4.89it/s, epoch=37, grad_norm=74.758, loss=298.5559, lr=4.10e-04]
Epoch 37 TRAIN:  60%|██████    | 82/136 [00:19<00:11,  4.89it/s, epoch=37, grad_norm=156.176, loss=370.0556, lr=4.10e-04]
Epoch 37 TRAIN:  60%|██████    | 82/136 [00:19<00:11,  4.89it/s, epoch=37, grad_norm=106.708, loss=327.7228, lr=4.10e-04]
Epoch 37 TRAIN:  60%|██████    | 82/136 [00:20<00:11,  4.89it/s, epoch=37, grad_norm=107.847, loss=340.3386, lr=4.10e-04]
Epoch 37 TRAIN:  62%|██████▎   | 85/136 [00:20<00:10,  4.66it/s, epoch=37, grad_norm=107.847, loss=340.3386, lr=4.10e-04]
Epoch 37 TRAIN:  62%|██████▎   | 85/136 [00:20<00:10,  4.66it/s, epoch=37, grad_norm=134.361, loss=352.0303, lr=4.10e-04]
Epoch 37 TRAIN:  62%|██████▎   | 85/136 [00:20<00:10,  4.66it/s, epoch=37, grad_norm=98.553, loss=349.8744, lr=4.10e-04] 
Epoch 37 TRAIN:  62%|██████▎   | 85/136 [00:20<00:10,  4.66it/s, epoch=37, grad_norm=70.176, loss=315.0416, lr=4.11e-04]
Epoch 37 TRAIN:  65%|██████▍   | 88/136 [00:20<00:10,  4.46it/s, epoch=37, grad_norm=70.176, loss=315.0416, lr=4.11e-04]
Epoch 37 TRAIN:  65%|██████▍   | 88/136 [00:20<00:10,  4.46it/s, epoch=37, grad_norm=188.708, loss=314.8289, lr=4.11e-04]
Epoch 37 TRAIN:  65%|██████▍   | 88/136 [00:21<00:10,  4.46it/s, epoch=37, grad_norm=100.655, loss=307.2990, lr=4.11e-04]
Epoch 37 TRAIN:  65%|██████▍   | 88/136 [00:21<00:10,  4.46it/s, epoch=37, grad_norm=157.808, loss=265.2438, lr=4.11e-04]
Epoch 37 TRAIN:  67%|██████▋   | 91/136 [00:21<00:09,  4.80it/s, epoch=37, grad_norm=157.808, loss=265.2438, lr=4.11e-04]
Epoch 37 TRAIN:  67%|██████▋   | 91/136 [00:21<00:09,  4.80it/s, epoch=37, grad_norm=116.783, loss=284.5113, lr=4.11e-04]
Epoch 37 TRAIN:  67%|██████▋   | 91/136 [00:21<00:09,  4.80it/s, epoch=37, grad_norm=147.239, loss=258.7409, lr=4.11e-04]
Epoch 37 TRAIN:  67%|██████▋   | 91/136 [00:22<00:09,  4.80it/s, epoch=37, grad_norm=74.218, loss=257.7057, lr=4.11e-04] 
Epoch 37 TRAIN:  69%|██████▉   | 94/136 [00:22<00:09,  4.60it/s, epoch=37, grad_norm=74.218, loss=257.7057, lr=4.11e-04]
Epoch 37 TRAIN:  69%|██████▉   | 94/136 [00:22<00:09,  4.60it/s, epoch=37, grad_norm=83.683, loss=257.0375, lr=4.11e-04]
Epoch 37 TRAIN:  69%|██████▉   | 94/136 [00:22<00:09,  4.60it/s, epoch=37, grad_norm=132.699, loss=259.1031, lr=4.11e-04]
Epoch 37 TRAIN:  69%|██████▉   | 94/136 [00:22<00:09,  4.60it/s, epoch=37, grad_norm=166.646, loss=336.2672, lr=4.11e-04]
Epoch 37 TRAIN:  71%|███████▏  | 97/136 [00:22<00:08,  4.52it/s, epoch=37, grad_norm=166.646, loss=336.2672, lr=4.11e-04]
Epoch 37 TRAIN:  71%|███████▏  | 97/136 [00:22<00:08,  4.52it/s, epoch=37, grad_norm=183.686, loss=290.0604, lr=4.11e-04]
Epoch 37 TRAIN:  71%|███████▏  | 97/136 [00:23<00:08,  4.52it/s, epoch=37, grad_norm=72.271, loss=217.7120, lr=4.11e-04] 
Epoch 37 TRAIN:  71%|███████▏  | 97/136 [00:23<00:08,  4.52it/s, epoch=37, grad_norm=219.413, loss=342.1583, lr=4.12e-04]2026-04-16 20:10:46,659 DEBUG TRAIN | steps/sec 4.270| Batch 37/100 loss 342.158264 loss_att 295.045227 loss_ctc 452.088623 th_accuracy 0.298729 lr 4.1152e-04 grad_norm 219.412964 rank 0

Epoch 37 TRAIN:  74%|███████▎  | 100/136 [00:23<00:07,  4.73it/s, epoch=37, grad_norm=219.413, loss=342.1583, lr=4.12e-04]
Epoch 37 TRAIN:  74%|███████▎  | 100/136 [00:23<00:07,  4.73it/s, epoch=37, grad_norm=79.469, loss=233.4554, lr=4.12e-04] 
Epoch 37 TRAIN:  74%|███████▎  | 100/136 [00:23<00:07,  4.73it/s, epoch=37, grad_norm=122.355, loss=342.7547, lr=4.12e-04]
Epoch 37 TRAIN:  74%|███████▎  | 100/136 [00:23<00:07,  4.73it/s, epoch=37, grad_norm=85.550, loss=327.1161, lr=4.12e-04] 
Epoch 37 TRAIN:  76%|███████▌  | 103/136 [00:23<00:06,  4.84it/s, epoch=37, grad_norm=85.550, loss=327.1161, lr=4.12e-04]
Epoch 37 TRAIN:  76%|███████▌  | 103/136 [00:24<00:06,  4.84it/s, epoch=37, grad_norm=158.419, loss=347.6474, lr=4.12e-04]
Epoch 37 TRAIN:  76%|███████▌  | 103/136 [00:24<00:06,  4.84it/s, epoch=37, grad_norm=105.218, loss=371.8750, lr=4.12e-04]
Epoch 37 TRAIN:  76%|███████▌  | 103/136 [00:24<00:06,  4.84it/s, epoch=37, grad_norm=122.563, loss=289.9934, lr=4.12e-04]
Epoch 37 TRAIN:  78%|███████▊  | 106/136 [00:24<00:06,  4.53it/s, epoch=37, grad_norm=122.563, loss=289.9934, lr=4.12e-04]
Epoch 37 TRAIN:  78%|███████▊  | 106/136 [00:24<00:06,  4.53it/s, epoch=37, grad_norm=230.969, loss=190.0207, lr=4.12e-04]
Epoch 37 TRAIN:  78%|███████▊  | 106/136 [00:25<00:06,  4.53it/s, epoch=37, grad_norm=109.301, loss=390.0431, lr=4.12e-04]
Epoch 37 TRAIN:  78%|███████▊  | 106/136 [00:25<00:06,  4.53it/s, epoch=37, grad_norm=156.817, loss=309.2739, lr=4.12e-04]
Epoch 37 TRAIN:  80%|████████  | 109/136 [00:25<00:06,  4.46it/s, epoch=37, grad_norm=156.817, loss=309.2739, lr=4.12e-04]
Epoch 37 TRAIN:  80%|████████  | 109/136 [00:25<00:06,  4.46it/s, epoch=37, grad_norm=131.454, loss=295.3064, lr=4.12e-04]
Epoch 37 TRAIN:  80%|████████  | 109/136 [00:25<00:06,  4.46it/s, epoch=37, grad_norm=119.345, loss=351.0548, lr=4.12e-04]
Epoch 37 TRAIN:  80%|████████  | 109/136 [00:25<00:06,  4.46it/s, epoch=37, grad_norm=95.224, loss=302.2154, lr=4.12e-04] 
Epoch 37 TRAIN:  82%|████████▏ | 112/136 [00:25<00:05,  4.52it/s, epoch=37, grad_norm=95.224, loss=302.2154, lr=4.12e-04]
Epoch 37 TRAIN:  82%|████████▏ | 112/136 [00:26<00:05,  4.52it/s, epoch=37, grad_norm=170.274, loss=277.9250, lr=4.13e-04]
Epoch 37 TRAIN:  82%|████████▏ | 112/136 [00:26<00:05,  4.52it/s, epoch=37, grad_norm=100.110, loss=276.2603, lr=4.13e-04]
Epoch 37 TRAIN:  82%|████████▏ | 112/136 [00:26<00:05,  4.52it/s, epoch=37, grad_norm=111.938, loss=268.4578, lr=4.13e-04]
Epoch 37 TRAIN:  85%|████████▍ | 115/136 [00:26<00:04,  4.68it/s, epoch=37, grad_norm=111.938, loss=268.4578, lr=4.13e-04]
Epoch 37 TRAIN:  85%|████████▍ | 115/136 [00:26<00:04,  4.68it/s, epoch=37, grad_norm=201.122, loss=338.1119, lr=4.13e-04]
Epoch 37 TRAIN:  85%|████████▍ | 115/136 [00:27<00:04,  4.68it/s, epoch=37, grad_norm=85.354, loss=276.0936, lr=4.13e-04] 
Epoch 37 TRAIN:  85%|████████▍ | 115/136 [00:27<00:04,  4.68it/s, epoch=37, grad_norm=90.071, loss=319.7758, lr=4.13e-04]
Epoch 37 TRAIN:  87%|████████▋ | 118/136 [00:27<00:03,  4.55it/s, epoch=37, grad_norm=90.071, loss=319.7758, lr=4.13e-04]
Epoch 37 TRAIN:  87%|████████▋ | 118/136 [00:27<00:03,  4.55it/s, epoch=37, grad_norm=141.534, loss=283.8860, lr=4.13e-04]
Epoch 37 TRAIN:  87%|████████▋ | 118/136 [00:27<00:03,  4.55it/s, epoch=37, grad_norm=137.057, loss=317.4950, lr=4.13e-04]
Epoch 37 TRAIN:  87%|████████▋ | 118/136 [00:27<00:03,  4.55it/s, epoch=37, grad_norm=240.404, loss=294.9081, lr=4.13e-04]
Epoch 37 TRAIN:  89%|████████▉ | 121/136 [00:27<00:03,  4.70it/s, epoch=37, grad_norm=240.404, loss=294.9081, lr=4.13e-04]
Epoch 37 TRAIN:  89%|████████▉ | 121/136 [00:28<00:03,  4.70it/s, epoch=37, grad_norm=113.546, loss=316.7403, lr=4.13e-04]
Epoch 37 TRAIN:  89%|████████▉ | 121/136 [00:28<00:03,  4.70it/s, epoch=37, grad_norm=166.197, loss=309.4663, lr=4.13e-04]
Epoch 37 TRAIN:  89%|████████▉ | 121/136 [00:28<00:03,  4.70it/s, epoch=37, grad_norm=121.043, loss=297.4949, lr=4.13e-04]
Epoch 37 TRAIN:  91%|█████████ | 124/136 [00:28<00:02,  4.78it/s, epoch=37, grad_norm=121.043, loss=297.4949, lr=4.13e-04]
Epoch 37 TRAIN:  91%|█████████ | 124/136 [00:28<00:02,  4.78it/s, epoch=37, grad_norm=132.709, loss=376.2760, lr=4.14e-04]
Epoch 37 TRAIN:  91%|█████████ | 124/136 [00:28<00:02,  4.78it/s, epoch=37, grad_norm=179.246, loss=408.0485, lr=4.14e-04]
Epoch 37 TRAIN:  91%|█████████ | 124/136 [00:28<00:02,  4.78it/s, epoch=37, grad_norm=106.599, loss=278.7177, lr=4.14e-04]
Epoch 37 TRAIN:  93%|█████████▎| 127/136 [00:28<00:01,  5.03it/s, epoch=37, grad_norm=106.599, loss=278.7177, lr=4.14e-04]
Epoch 37 TRAIN:  93%|█████████▎| 127/136 [00:29<00:01,  5.03it/s, epoch=37, grad_norm=89.052, loss=298.4910, lr=4.14e-04] 
Epoch 37 TRAIN:  93%|█████████▎| 127/136 [00:29<00:01,  5.03it/s, epoch=37, grad_norm=162.824, loss=376.1714, lr=4.14e-04]
Epoch 37 TRAIN:  93%|█████████▎| 127/136 [00:29<00:01,  5.03it/s, epoch=37, grad_norm=107.897, loss=346.3701, lr=4.14e-04]
Epoch 37 TRAIN:  93%|█████████▎| 127/136 [00:29<00:01,  5.03it/s, epoch=37, grad_norm=93.176, loss=303.0833, lr=4.14e-04] 
Epoch 37 TRAIN:  96%|█████████▋| 131/136 [00:29<00:00,  5.25it/s, epoch=37, grad_norm=93.176, loss=303.0833, lr=4.14e-04]
Epoch 37 TRAIN:  96%|█████████▋| 131/136 [00:29<00:00,  5.25it/s, epoch=37, grad_norm=158.421, loss=301.6472, lr=4.14e-04]
Epoch 37 TRAIN:  96%|█████████▋| 131/136 [00:29<00:00,  5.25it/s, epoch=37, grad_norm=152.256, loss=320.0869, lr=4.14e-04]
Epoch 37 TRAIN:  96%|█████████▋| 131/136 [00:30<00:00,  5.25it/s, epoch=37, grad_norm=106.249, loss=340.1419, lr=4.14e-04]
Epoch 37 TRAIN:  99%|█████████▊| 134/136 [00:30<00:00,  5.38it/s, epoch=37, grad_norm=106.249, loss=340.1419, lr=4.14e-04]
Epoch 37 TRAIN:  99%|█████████▊| 134/136 [00:30<00:00,  5.38it/s, epoch=37, grad_norm=189.040, loss=418.4694, lr=4.14e-04]
Epoch 37 TRAIN:  99%|█████████▊| 134/136 [00:30<00:00,  5.38it/s, epoch=37, grad_norm=124.256, loss=401.9158, lr=4.14e-04]
Epoch 37 TRAIN: 100%|██████████| 136/136 [00:30<00:00,  4.44it/s, epoch=37, grad_norm=124.256, loss=401.9158, lr=4.14e-04]

CV:   0%|          | 0/6 [00:00<?, ?it/s]/home/rookie/Documents/batuka/projects/wenet/wenet/wenet/utils/train_utils.py:689: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  amp_autocast(enabled=dtype is not None,

CV:  17%|█▋        | 1/6 [00:00<00:03,  1.33it/s]
                                                 
2026-04-16 20:10:54,856 INFO Epoch 37 Step 5180 CV info lr 4.1440e-04 cv_loss 474.4666914506392 rank 0 acc 0.2573799292246501
2026-04-16 20:10:54,857 INFO [Rank 0] Checkpoint: save to checkpoint exp/conformer_mongolian/epoch_37.pt
2026-04-16 20:10:54,970 INFO Epoch 38 Step 5180 TRAIN info lr 4.1440e-04 rank 0
2026-04-16 20:10:54,986 INFO using accumulate grad, new batch size is 1 times larger than before

Epoch 38 TRAIN:   0%|          | 0/136 [00:00<?, ?it/s]
Epoch 38 TRAIN:   0%|          | 0/136 [00:03<?, ?it/s, epoch=38, grad_norm=308.804, loss=322.9700, lr=4.14e-04]
Epoch 38 TRAIN:   1%|          | 1/136 [00:03<08:47,  3.91s/it, epoch=38, grad_norm=308.804, loss=322.9700, lr=4.14e-04]
Epoch 38 TRAIN:   1%|          | 1/136 [00:04<08:47,  3.91s/it, epoch=38, grad_norm=251.035, loss=297.9908, lr=4.15e-04]
Epoch 38 TRAIN:   1%|          | 1/136 [00:04<08:47,  3.91s/it, epoch=38, grad_norm=282.478, loss=333.0304, lr=4.15e-04]
Epoch 38 TRAIN:   1%|          | 1/136 [00:04<08:47,  3.91s/it, epoch=38, grad_norm=421.169, loss=424.4235, lr=4.15e-04]
Epoch 38 TRAIN:   1%|          | 1/136 [00:04<08:47,  3.91s/it, epoch=38, grad_norm=244.815, loss=356.2330, lr=4.15e-04]
Epoch 38 TRAIN:   1%|          | 1/136 [00:04<08:47,  3.91s/it, epoch=38, grad_norm=103.739, loss=296.1413, lr=4.15e-04]
Epoch 38 TRAIN:   1%|          | 1/136 [00:04<08:47,  3.91s/it, epoch=38, grad_norm=233.313, loss=382.3250, lr=4.15e-04]
Epoch 38 TRAIN:   5%|▌         | 7/136 [00:04<01:02,  2.05it/s, epoch=38, grad_norm=233.313, loss=382.3250, lr=4.15e-04]
Epoch 38 TRAIN:   5%|▌         | 7/136 [00:04<01:02,  2.05it/s, epoch=38, grad_norm=113.316, loss=271.9843, lr=4.15e-04]
Epoch 38 TRAIN:   5%|▌         | 7/136 [00:04<01:02,  2.05it/s, epoch=38, grad_norm=78.814, loss=340.3740, lr=4.15e-04] 
Epoch 38 TRAIN:   5%|▌         | 7/136 [00:04<01:02,  2.05it/s, epoch=38, grad_norm=110.918, loss=270.2773, lr=4.15e-04]
Epoch 38 TRAIN:   5%|▌         | 7/136 [00:04<01:02,  2.05it/s, epoch=38, grad_norm=82.083, loss=341.8846, lr=4.15e-04] 
Epoch 38 TRAIN:   8%|▊         | 11/136 [00:04<00:40,  3.07it/s, epoch=38, grad_norm=82.083, loss=341.8846, lr=4.15e-04]
Epoch 38 TRAIN:   8%|▊         | 11/136 [00:05<00:40,  3.07it/s, epoch=38, grad_norm=102.121, loss=310.9777, lr=4.15e-04]
Epoch 38 TRAIN:   8%|▊         | 11/136 [00:05<00:40,  3.07it/s, epoch=38, grad_norm=64.979, loss=341.2063, lr=4.15e-04] 
Epoch 38 TRAIN:   8%|▊         | 11/136 [00:05<00:40,  3.07it/s, epoch=38, grad_norm=83.226, loss=315.8684, lr=4.16e-04]
Epoch 38 TRAIN:   8%|▊         | 11/136 [00:05<00:40,  3.07it/s, epoch=38, grad_norm=64.128, loss=332.3998, lr=4.16e-04]
Epoch 38 TRAIN:  11%|█         | 15/136 [00:05<00:31,  3.85it/s, epoch=38, grad_norm=64.128, loss=332.3998, lr=4.16e-04]
Epoch 38 TRAIN:  11%|█         | 15/136 [00:05<00:31,  3.85it/s, epoch=38, grad_norm=60.784, loss=304.7764, lr=4.16e-04]
Epoch 38 TRAIN:  11%|█         | 15/136 [00:05<00:31,  3.85it/s, epoch=38, grad_norm=85.961, loss=353.6856, lr=4.16e-04]
Epoch 38 TRAIN:  11%|█         | 15/136 [00:06<00:31,  3.85it/s, epoch=38, grad_norm=119.763, loss=356.3093, lr=4.16e-04]
Epoch 38 TRAIN:  11%|█         | 15/136 [00:06<00:31,  3.85it/s, epoch=38, grad_norm=108.417, loss=274.7472, lr=4.16e-04]
Epoch 38 TRAIN:  14%|█▍        | 19/136 [00:06<00:25,  4.50it/s, epoch=38, grad_norm=108.417, loss=274.7472, lr=4.16e-04]
Epoch 38 TRAIN:  14%|█▍        | 19/136 [00:06<00:25,  4.50it/s, epoch=38, grad_norm=63.129, loss=327.6519, lr=4.16e-04] 
Epoch 38 TRAIN:  14%|█▍        | 19/136 [00:06<00:25,  4.50it/s, epoch=38, grad_norm=105.624, loss=260.9460, lr=4.16e-04]
Epoch 38 TRAIN:  14%|█▍        | 19/136 [00:06<00:25,  4.50it/s, epoch=38, grad_norm=86.253, loss=312.4274, lr=4.16e-04] 
Epoch 38 TRAIN:  14%|█▍        | 19/136 [00:07<00:25,  4.50it/s, epoch=38, grad_norm=105.449, loss=352.3612, lr=4.16e-04]
Epoch 38 TRAIN:  17%|█▋        | 23/136 [00:07<00:25,  4.48it/s, epoch=38, grad_norm=105.449, loss=352.3612, lr=4.16e-04]
Epoch 38 TRAIN:  17%|█▋        | 23/136 [00:07<00:25,  4.48it/s, epoch=38, grad_norm=66.060, loss=324.3204, lr=4.16e-04] 
Epoch 38 TRAIN:  17%|█▋        | 23/136 [00:07<00:25,  4.48it/s, epoch=38, grad_norm=98.138, loss=308.4532, lr=4.16e-04]
Epoch 38 TRAIN:  17%|█▋        | 23/136 [00:07<00:25,  4.48it/s, epoch=38, grad_norm=76.374, loss=306.1473, lr=4.16e-04]
Epoch 38 TRAIN:  19%|█▉        | 26/136 [00:07<00:23,  4.74it/s, epoch=38, grad_norm=76.374, loss=306.1473, lr=4.16e-04]
Epoch 38 TRAIN:  19%|█▉        | 26/136 [00:07<00:23,  4.74it/s, epoch=38, grad_norm=128.267, loss=306.0187, lr=4.17e-04]
Epoch 38 TRAIN:  19%|█▉        | 26/136 [00:08<00:23,  4.74it/s, epoch=38, grad_norm=93.154, loss=283.5509, lr=4.17e-04] 
Epoch 38 TRAIN:  19%|█▉        | 26/136 [00:08<00:23,  4.74it/s, epoch=38, grad_norm=104.602, loss=330.9454, lr=4.17e-04]
Epoch 38 TRAIN:  21%|██▏       | 29/136 [00:08<00:21,  5.01it/s, epoch=38, grad_norm=104.602, loss=330.9454, lr=4.17e-04]
Epoch 38 TRAIN:  21%|██▏       | 29/136 [00:08<00:21,  5.01it/s, epoch=38, grad_norm=102.162, loss=307.7678, lr=4.17e-04]
Epoch 38 TRAIN:  21%|██▏       | 29/136 [00:08<00:21,  5.01it/s, epoch=38, grad_norm=94.210, loss=330.1151, lr=4.17e-04] 
Epoch 38 TRAIN:  21%|██▏       | 29/136 [00:08<00:21,  5.01it/s, epoch=38, grad_norm=96.327, loss=325.3859, lr=4.17e-04]
Epoch 38 TRAIN:  24%|██▎       | 32/136 [00:08<00:21,  4.93it/s, epoch=38, grad_norm=96.327, loss=325.3859, lr=4.17e-04]
Epoch 38 TRAIN:  24%|██▎       | 32/136 [00:09<00:21,  4.93it/s, epoch=38, grad_norm=74.591, loss=298.8839, lr=4.17e-04]
Epoch 38 TRAIN:  24%|██▎       | 32/136 [00:09<00:21,  4.93it/s, epoch=38, grad_norm=114.870, loss=359.3525, lr=4.17e-04]
Epoch 38 TRAIN:  24%|██▎       | 32/136 [00:09<00:21,  4.93it/s, epoch=38, grad_norm=116.554, loss=311.6543, lr=4.17e-04]
Epoch 38 TRAIN:  26%|██▌       | 35/136 [00:09<00:19,  5.06it/s, epoch=38, grad_norm=116.554, loss=311.6543, lr=4.17e-04]
Epoch 38 TRAIN:  26%|██▌       | 35/136 [00:09<00:19,  5.06it/s, epoch=38, grad_norm=73.040, loss=326.4724, lr=4.17e-04] 
Epoch 38 TRAIN:  26%|██▌       | 35/136 [00:09<00:19,  5.06it/s, epoch=38, grad_norm=68.111, loss=354.3907, lr=4.17e-04]
Epoch 38 TRAIN:  26%|██▌       | 35/136 [00:09<00:19,  5.06it/s, epoch=38, grad_norm=100.479, loss=358.0139, lr=4.17e-04]
Epoch 38 TRAIN:  28%|██▊       | 38/136 [00:09<00:19,  5.10it/s, epoch=38, grad_norm=100.479, loss=358.0139, lr=4.17e-04]
Epoch 38 TRAIN:  28%|██▊       | 38/136 [00:10<00:19,  5.10it/s, epoch=38, grad_norm=99.584, loss=297.1837, lr=4.18e-04] 
Epoch 38 TRAIN:  28%|██▊       | 38/136 [00:10<00:19,  5.10it/s, epoch=38, grad_norm=113.277, loss=257.2962, lr=4.18e-04]
Epoch 38 TRAIN:  28%|██▊       | 38/136 [00:10<00:19,  5.10it/s, epoch=38, grad_norm=92.883, loss=284.1618, lr=4.18e-04] 
Epoch 38 TRAIN:  30%|███       | 41/136 [00:10<00:18,  5.12it/s, epoch=38, grad_norm=92.883, loss=284.1618, lr=4.18e-04]
Epoch 38 TRAIN:  30%|███       | 41/136 [00:10<00:18,  5.12it/s, epoch=38, grad_norm=63.979, loss=309.0401, lr=4.18e-04]
Epoch 38 TRAIN:  30%|███       | 41/136 [00:10<00:18,  5.12it/s, epoch=38, grad_norm=58.308, loss=334.6688, lr=4.18e-04]
Epoch 38 TRAIN:  30%|███       | 41/136 [00:11<00:18,  5.12it/s, epoch=38, grad_norm=81.344, loss=322.7157, lr=4.18e-04]
Epoch 38 TRAIN:  32%|███▏      | 44/136 [00:11<00:17,  5.16it/s, epoch=38, grad_norm=81.344, loss=322.7157, lr=4.18e-04]
Epoch 38 TRAIN:  32%|███▏      | 44/136 [00:11<00:17,  5.16it/s, epoch=38, grad_norm=121.255, loss=361.0142, lr=4.18e-04]
Epoch 38 TRAIN:  32%|███▏      | 44/136 [00:11<00:17,  5.16it/s, epoch=38, grad_norm=62.610, loss=304.8204, lr=4.18e-04] 
Epoch 38 TRAIN:  32%|███▏      | 44/136 [00:11<00:17,  5.16it/s, epoch=38, grad_norm=72.957, loss=307.5754, lr=4.18e-04]
Epoch 38 TRAIN:  35%|███▍      | 47/136 [00:11<00:17,  4.95it/s, epoch=38, grad_norm=72.957, loss=307.5754, lr=4.18e-04]
Epoch 38 TRAIN:  35%|███▍      | 47/136 [00:11<00:17,  4.95it/s, epoch=38, grad_norm=105.447, loss=352.2900, lr=4.18e-04]
Epoch 38 TRAIN:  35%|███▍      | 47/136 [00:12<00:17,  4.95it/s, epoch=38, grad_norm=85.322, loss=312.5380, lr=4.18e-04] 
Epoch 38 TRAIN:  35%|███▍      | 47/136 [00:12<00:17,  4.95it/s, epoch=38, grad_norm=75.838, loss=305.0269, lr=4.18e-04]
Epoch 38 TRAIN:  37%|███▋      | 50/136 [00:12<00:17,  4.83it/s, epoch=38, grad_norm=75.838, loss=305.0269, lr=4.18e-04]
Epoch 38 TRAIN:  37%|███▋      | 50/136 [00:12<00:17,  4.83it/s, epoch=38, grad_norm=153.783, loss=291.7741, lr=4.18e-04]
Epoch 38 TRAIN:  37%|███▋      | 50/136 [00:12<00:17,  4.83it/s, epoch=38, grad_norm=81.802, loss=348.5450, lr=4.19e-04] 
Epoch 38 TRAIN:  37%|███▋      | 50/136 [00:13<00:17,  4.83it/s, epoch=38, grad_norm=75.011, loss=275.1510, lr=4.19e-04]
Epoch 38 TRAIN:  39%|███▉      | 53/136 [00:13<00:17,  4.87it/s, epoch=38, grad_norm=75.011, loss=275.1510, lr=4.19e-04]
Epoch 38 TRAIN:  39%|███▉      | 53/136 [00:13<00:17,  4.87it/s, epoch=38, grad_norm=57.965, loss=293.2820, lr=4.19e-04]
Epoch 38 TRAIN:  39%|███▉      | 53/136 [00:13<00:17,  4.87it/s, epoch=38, grad_norm=60.255, loss=315.2190, lr=4.19e-04]
Epoch 38 TRAIN:  39%|███▉      | 53/136 [00:13<00:17,  4.87it/s, epoch=38, grad_norm=157.090, loss=348.5350, lr=4.19e-04]
Epoch 38 TRAIN:  41%|████      | 56/136 [00:13<00:16,  4.87it/s, epoch=38, grad_norm=157.090, loss=348.5350, lr=4.19e-04]
Epoch 38 TRAIN:  41%|████      | 56/136 [00:13<00:16,  4.87it/s, epoch=38, grad_norm=136.279, loss=351.7228, lr=4.19e-04]
Epoch 38 TRAIN:  41%|████      | 56/136 [00:14<00:16,  4.87it/s, epoch=38, grad_norm=90.338, loss=309.6006, lr=4.19e-04] 
Epoch 38 TRAIN:  41%|████      | 56/136 [00:14<00:16,  4.87it/s, epoch=38, grad_norm=72.082, loss=340.2370, lr=4.19e-04]
Epoch 38 TRAIN:  43%|████▎     | 59/136 [00:14<00:15,  4.83it/s, epoch=38, grad_norm=72.082, loss=340.2370, lr=4.19e-04]
Epoch 38 TRAIN:  43%|████▎     | 59/136 [00:14<00:15,  4.83it/s, epoch=38, grad_norm=64.402, loss=324.3728, lr=4.19e-04]
Epoch 38 TRAIN:  43%|████▎     | 59/136 [00:14<00:15,  4.83it/s, epoch=38, grad_norm=71.823, loss=316.0991, lr=4.19e-04]
Epoch 38 TRAIN:  43%|████▎     | 59/136 [00:14<00:15,  4.83it/s, epoch=38, grad_norm=68.256, loss=327.3750, lr=4.19e-04]
Epoch 38 TRAIN:  46%|████▌     | 62/136 [00:14<00:15,  4.77it/s, epoch=38, grad_norm=68.256, loss=327.3750, lr=4.19e-04]
Epoch 38 TRAIN:  46%|████▌     | 62/136 [00:15<00:15,  4.77it/s, epoch=38, grad_norm=102.669, loss=320.6078, lr=4.19e-04]
Epoch 38 TRAIN:  46%|████▌     | 62/136 [00:15<00:15,  4.77it/s, epoch=38, grad_norm=80.993, loss=327.6990, lr=4.20e-04] 
Epoch 38 TRAIN:  46%|████▌     | 62/136 [00:15<00:15,  4.77it/s, epoch=38, grad_norm=112.285, loss=301.9659, lr=4.20e-04]
Epoch 38 TRAIN:  48%|████▊     | 65/136 [00:15<00:14,  4.87it/s, epoch=38, grad_norm=112.285, loss=301.9659, lr=4.20e-04]
Epoch 38 TRAIN:  48%|████▊     | 65/136 [00:15<00:14,  4.87it/s, epoch=38, grad_norm=89.437, loss=337.0167, lr=4.20e-04] 
Epoch 38 TRAIN:  48%|████▊     | 65/136 [00:15<00:14,  4.87it/s, epoch=38, grad_norm=288.656, loss=243.6113, lr=4.20e-04]
Epoch 38 TRAIN:  48%|████▊     | 65/136 [00:16<00:14,  4.87it/s, epoch=38, grad_norm=98.113, loss=308.9161, lr=4.20e-04] 
Epoch 38 TRAIN:  50%|█████     | 68/136 [00:16<00:14,  4.84it/s, epoch=38, grad_norm=98.113, loss=308.9161, lr=4.20e-04]
Epoch 38 TRAIN:  50%|█████     | 68/136 [00:16<00:14,  4.84it/s, epoch=38, grad_norm=89.923, loss=278.0345, lr=4.20e-04]
Epoch 38 TRAIN:  50%|█████     | 68/136 [00:16<00:14,  4.84it/s, epoch=38, grad_norm=146.896, loss=341.7433, lr=4.20e-04]
Epoch 38 TRAIN:  50%|█████     | 68/136 [00:16<00:14,  4.84it/s, epoch=38, grad_norm=182.808, loss=241.3833, lr=4.20e-04]
Epoch 38 TRAIN:  52%|█████▏    | 71/136 [00:16<00:13,  4.78it/s, epoch=38, grad_norm=182.808, loss=241.3833, lr=4.20e-04]
Epoch 38 TRAIN:  52%|█████▏    | 71/136 [00:16<00:13,  4.78it/s, epoch=38, grad_norm=146.471, loss=319.7321, lr=4.20e-04]
Epoch 38 TRAIN:  52%|█████▏    | 71/136 [00:17<00:13,  4.78it/s, epoch=38, grad_norm=113.292, loss=306.1054, lr=4.20e-04]
Epoch 38 TRAIN:  52%|█████▏    | 71/136 [00:17<00:13,  4.78it/s, epoch=38, grad_norm=71.181, loss=277.5749, lr=4.20e-04] 
Epoch 38 TRAIN:  54%|█████▍    | 74/136 [00:17<00:12,  4.81it/s, epoch=38, grad_norm=71.181, loss=277.5749, lr=4.20e-04]
Epoch 38 TRAIN:  54%|█████▍    | 74/136 [00:17<00:12,  4.81it/s, epoch=38, grad_norm=100.297, loss=314.1143, lr=4.20e-04]
Epoch 38 TRAIN:  54%|█████▍    | 74/136 [00:17<00:12,  4.81it/s, epoch=38, grad_norm=188.765, loss=386.3478, lr=4.20e-04]
Epoch 38 TRAIN:  54%|█████▍    | 74/136 [00:17<00:12,  4.81it/s, epoch=38, grad_norm=105.041, loss=341.0509, lr=4.21e-04]
Epoch 38 TRAIN:  57%|█████▋    | 77/136 [00:17<00:12,  4.90it/s, epoch=38, grad_norm=105.041, loss=341.0509, lr=4.21e-04]
Epoch 38 TRAIN:  57%|█████▋    | 77/136 [00:18<00:12,  4.90it/s, epoch=38, grad_norm=96.216, loss=299.7517, lr=4.21e-04] 
Epoch 38 TRAIN:  57%|█████▋    | 77/136 [00:18<00:12,  4.90it/s, epoch=38, grad_norm=243.822, loss=288.0021, lr=4.21e-04]
Epoch 38 TRAIN:  57%|█████▋    | 77/136 [00:18<00:12,  4.90it/s, epoch=38, grad_norm=103.081, loss=353.4783, lr=4.21e-04]
Epoch 38 TRAIN:  59%|█████▉    | 80/136 [00:18<00:11,  4.91it/s, epoch=38, grad_norm=103.081, loss=353.4783, lr=4.21e-04]
Epoch 38 TRAIN:  59%|█████▉    | 80/136 [00:18<00:11,  4.91it/s, epoch=38, grad_norm=117.781, loss=298.3969, lr=4.21e-04]
Epoch 38 TRAIN:  59%|█████▉    | 80/136 [00:19<00:11,  4.91it/s, epoch=38, grad_norm=165.614, loss=256.6645, lr=4.21e-04]
Epoch 38 TRAIN:  59%|█████▉    | 80/136 [00:19<00:11,  4.91it/s, epoch=38, grad_norm=260.536, loss=293.0800, lr=4.21e-04]
Epoch 38 TRAIN:  61%|██████    | 83/136 [00:19<00:11,  4.79it/s, epoch=38, grad_norm=260.536, loss=293.0800, lr=4.21e-04]
Epoch 38 TRAIN:  61%|██████    | 83/136 [00:19<00:11,  4.79it/s, epoch=38, grad_norm=88.064, loss=349.2862, lr=4.21e-04] 
Epoch 38 TRAIN:  61%|██████    | 83/136 [00:19<00:11,  4.79it/s, epoch=38, grad_norm=129.544, loss=283.3859, lr=4.21e-04]
Epoch 38 TRAIN:  61%|██████    | 83/136 [00:19<00:11,  4.79it/s, epoch=38, grad_norm=131.561, loss=334.4757, lr=4.21e-04]
Epoch 38 TRAIN:  63%|██████▎   | 86/136 [00:19<00:10,  4.87it/s, epoch=38, grad_norm=131.561, loss=334.4757, lr=4.21e-04]
Epoch 38 TRAIN:  63%|██████▎   | 86/136 [00:20<00:10,  4.87it/s, epoch=38, grad_norm=93.066, loss=350.8255, lr=4.21e-04] 
Epoch 38 TRAIN:  63%|██████▎   | 86/136 [00:20<00:10,  4.87it/s, epoch=38, grad_norm=139.472, loss=344.5964, lr=4.21e-04]
Epoch 38 TRAIN:  63%|██████▎   | 86/136 [00:20<00:10,  4.87it/s, epoch=38, grad_norm=110.076, loss=287.7055, lr=4.22e-04]
Epoch 38 TRAIN:  65%|██████▌   | 89/136 [00:20<00:09,  4.80it/s, epoch=38, grad_norm=110.076, loss=287.7055, lr=4.22e-04]
Epoch 38 TRAIN:  65%|██████▌   | 89/136 [00:20<00:09,  4.80it/s, epoch=38, grad_norm=68.842, loss=291.7570, lr=4.22e-04] 
Epoch 38 TRAIN:  65%|██████▌   | 89/136 [00:21<00:09,  4.80it/s, epoch=38, grad_norm=151.401, loss=379.8251, lr=4.22e-04]
Epoch 38 TRAIN:  65%|██████▌   | 89/136 [00:21<00:09,  4.80it/s, epoch=38, grad_norm=71.632, loss=290.8289, lr=4.22e-04] 
Epoch 38 TRAIN:  68%|██████▊   | 92/136 [00:21<00:09,  4.47it/s, epoch=38, grad_norm=71.632, loss=290.8289, lr=4.22e-04]
Epoch 38 TRAIN:  68%|██████▊   | 92/136 [00:21<00:09,  4.47it/s, epoch=38, grad_norm=85.728, loss=328.5031, lr=4.22e-04]
Epoch 38 TRAIN:  68%|██████▊   | 92/136 [00:21<00:09,  4.47it/s, epoch=38, grad_norm=73.006, loss=269.6970, lr=4.22e-04]
Epoch 38 TRAIN:  68%|██████▊   | 92/136 [00:21<00:09,  4.47it/s, epoch=38, grad_norm=229.924, loss=316.3697, lr=4.22e-04]
Epoch 38 TRAIN:  70%|██████▉   | 95/136 [00:21<00:09,  4.48it/s, epoch=38, grad_norm=229.924, loss=316.3697, lr=4.22e-04]
Epoch 38 TRAIN:  70%|██████▉   | 95/136 [00:22<00:09,  4.48it/s, epoch=38, grad_norm=98.495, loss=277.4167, lr=4.22e-04] 
Epoch 38 TRAIN:  70%|██████▉   | 95/136 [00:22<00:09,  4.48it/s, epoch=38, grad_norm=230.513, loss=246.6513, lr=4.22e-04]
Epoch 38 TRAIN:  70%|██████▉   | 95/136 [00:22<00:09,  4.48it/s, epoch=38, grad_norm=209.994, loss=192.7498, lr=4.22e-04]
Epoch 38 TRAIN:  72%|███████▏  | 98/136 [00:22<00:07,  4.77it/s, epoch=38, grad_norm=209.994, loss=192.7498, lr=4.22e-04]
Epoch 38 TRAIN:  72%|███████▏  | 98/136 [00:22<00:07,  4.77it/s, epoch=38, grad_norm=139.202, loss=245.0206, lr=4.22e-04]
Epoch 38 TRAIN:  72%|███████▏  | 98/136 [00:22<00:07,  4.77it/s, epoch=38, grad_norm=142.324, loss=296.4219, lr=4.22e-04]2026-04-16 20:11:17,928 DEBUG TRAIN | steps/sec 4.349| Batch 38/100 loss 296.421875 loss_att 259.988983 loss_ctc 381.431976 th_accuracy 0.277778 lr 4.2240e-04 grad_norm 142.324402 rank 0

Epoch 38 TRAIN:  72%|███████▏  | 98/136 [00:23<00:07,  4.77it/s, epoch=38, grad_norm=109.774, loss=185.3713, lr=4.22e-04]
Epoch 38 TRAIN:  74%|███████▍  | 101/136 [00:23<00:07,  4.69it/s, epoch=38, grad_norm=109.774, loss=185.3713, lr=4.22e-04]
Epoch 38 TRAIN:  74%|███████▍  | 101/136 [00:23<00:07,  4.69it/s, epoch=38, grad_norm=331.121, loss=323.0571, lr=4.23e-04]
Epoch 38 TRAIN:  74%|███████▍  | 101/136 [00:23<00:07,  4.69it/s, epoch=38, grad_norm=119.184, loss=311.0641, lr=4.23e-04]
Epoch 38 TRAIN:  74%|███████▍  | 101/136 [00:23<00:07,  4.69it/s, epoch=38, grad_norm=166.861, loss=284.4354, lr=4.23e-04]
Epoch 38 TRAIN:  76%|███████▋  | 104/136 [00:23<00:06,  4.76it/s, epoch=38, grad_norm=166.861, loss=284.4354, lr=4.23e-04]
Epoch 38 TRAIN:  76%|███████▋  | 104/136 [00:23<00:06,  4.76it/s, epoch=38, grad_norm=181.471, loss=345.3578, lr=4.23e-04]
Epoch 38 TRAIN:  76%|███████▋  | 104/136 [00:24<00:06,  4.76it/s, epoch=38, grad_norm=237.037, loss=351.3096, lr=4.23e-04]
Epoch 38 TRAIN:  76%|███████▋  | 104/136 [00:24<00:06,  4.76it/s, epoch=38, grad_norm=91.132, loss=267.7786, lr=4.23e-04] 
Epoch 38 TRAIN:  79%|███████▊  | 107/136 [00:24<00:05,  4.94it/s, epoch=38, grad_norm=91.132, loss=267.7786, lr=4.23e-04]
Epoch 38 TRAIN:  79%|███████▊  | 107/136 [00:24<00:05,  4.94it/s, epoch=38, grad_norm=87.631, loss=320.6249, lr=4.23e-04]
Epoch 38 TRAIN:  79%|███████▊  | 107/136 [00:24<00:05,  4.94it/s, epoch=38, grad_norm=125.958, loss=351.5936, lr=4.23e-04]
Epoch 38 TRAIN:  79%|███████▊  | 107/136 [00:24<00:05,  4.94it/s, epoch=38, grad_norm=90.670, loss=306.6543, lr=4.23e-04] 
Epoch 38 TRAIN:  81%|████████  | 110/136 [00:24<00:05,  4.73it/s, epoch=38, grad_norm=90.670, loss=306.6543, lr=4.23e-04]
Epoch 38 TRAIN:  81%|████████  | 110/136 [00:25<00:05,  4.73it/s, epoch=38, grad_norm=170.406, loss=354.6591, lr=4.23e-04]
Epoch 38 TRAIN:  81%|████████  | 110/136 [00:25<00:05,  4.73it/s, epoch=38, grad_norm=130.101, loss=307.8515, lr=4.23e-04]
Epoch 38 TRAIN:  81%|████████  | 110/136 [00:25<00:05,  4.73it/s, epoch=38, grad_norm=174.368, loss=358.6076, lr=4.23e-04]
Epoch 38 TRAIN:  83%|████████▎ | 113/136 [00:25<00:04,  4.81it/s, epoch=38, grad_norm=174.368, loss=358.6076, lr=4.23e-04]
Epoch 38 TRAIN:  83%|████████▎ | 113/136 [00:25<00:04,  4.81it/s, epoch=38, grad_norm=81.621, loss=305.9567, lr=4.24e-04] 
Epoch 38 TRAIN:  83%|████████▎ | 113/136 [00:25<00:04,  4.81it/s, epoch=38, grad_norm=316.564, loss=246.8119, lr=4.24e-04]
Epoch 38 TRAIN:  83%|████████▎ | 113/136 [00:26<00:04,  4.81it/s, epoch=38, grad_norm=149.863, loss=381.5195, lr=4.24e-04]
Epoch 38 TRAIN:  83%|████████▎ | 113/136 [00:26<00:04,  4.81it/s, epoch=38, grad_norm=92.677, loss=278.7180, lr=4.24e-04] 
Epoch 38 TRAIN:  86%|████████▌ | 117/136 [00:26<00:03,  5.07it/s, epoch=38, grad_norm=92.677, loss=278.7180, lr=4.24e-04]
Epoch 38 TRAIN:  86%|████████▌ | 117/136 [00:26<00:03,  5.07it/s, epoch=38, grad_norm=178.223, loss=339.1155, lr=4.24e-04]
Epoch 38 TRAIN:  86%|████████▌ | 117/136 [00:26<00:03,  5.07it/s, epoch=38, grad_norm=136.039, loss=408.7043, lr=4.24e-04]
Epoch 38 TRAIN:  86%|████████▌ | 117/136 [00:26<00:03,  5.07it/s, epoch=38, grad_norm=152.660, loss=217.3317, lr=4.24e-04]
Epoch 38 TRAIN:  88%|████████▊ | 120/136 [00:26<00:03,  4.95it/s, epoch=38, grad_norm=152.660, loss=217.3317, lr=4.24e-04]
Epoch 38 TRAIN:  88%|████████▊ | 120/136 [00:27<00:03,  4.95it/s, epoch=38, grad_norm=106.322, loss=385.4537, lr=4.24e-04]
Epoch 38 TRAIN:  88%|████████▊ | 120/136 [00:27<00:03,  4.95it/s, epoch=38, grad_norm=147.441, loss=348.5547, lr=4.24e-04]
Epoch 38 TRAIN:  88%|████████▊ | 120/136 [00:27<00:03,  4.95it/s, epoch=38, grad_norm=154.811, loss=391.5142, lr=4.24e-04]
Epoch 38 TRAIN:  90%|█████████ | 123/136 [00:27<00:02,  5.09it/s, epoch=38, grad_norm=154.811, loss=391.5142, lr=4.24e-04]
Epoch 38 TRAIN:  90%|█████████ | 123/136 [00:27<00:02,  5.09it/s, epoch=38, grad_norm=113.539, loss=292.2701, lr=4.24e-04]
Epoch 38 TRAIN:  90%|█████████ | 123/136 [00:27<00:02,  5.09it/s, epoch=38, grad_norm=118.747, loss=287.2777, lr=4.24e-04]
Epoch 38 TRAIN:  90%|█████████ | 123/136 [00:28<00:02,  5.09it/s, epoch=38, grad_norm=245.552, loss=409.9760, lr=4.24e-04]
Epoch 38 TRAIN:  93%|█████████▎| 126/136 [00:28<00:02,  4.95it/s, epoch=38, grad_norm=245.552, loss=409.9760, lr=4.24e-04]
Epoch 38 TRAIN:  93%|█████████▎| 126/136 [00:28<00:02,  4.95it/s, epoch=38, grad_norm=118.420, loss=275.0391, lr=4.25e-04]
Epoch 38 TRAIN:  93%|█████████▎| 126/136 [00:28<00:02,  4.95it/s, epoch=38, grad_norm=111.906, loss=239.9065, lr=4.25e-04]
Epoch 38 TRAIN:  93%|█████████▎| 126/136 [00:28<00:02,  4.95it/s, epoch=38, grad_norm=161.971, loss=314.0211, lr=4.25e-04]
Epoch 38 TRAIN:  95%|█████████▍| 129/136 [00:28<00:01,  4.98it/s, epoch=38, grad_norm=161.971, loss=314.0211, lr=4.25e-04]
Epoch 38 TRAIN:  95%|█████████▍| 129/136 [00:28<00:01,  4.98it/s, epoch=38, grad_norm=213.986, loss=329.4549, lr=4.25e-04]
Epoch 38 TRAIN:  95%|█████████▍| 129/136 [00:29<00:01,  4.98it/s, epoch=38, grad_norm=224.739, loss=371.6829, lr=4.25e-04]
Epoch 38 TRAIN:  95%|█████████▍| 129/136 [00:29<00:01,  4.98it/s, epoch=38, grad_norm=116.514, loss=285.4534, lr=4.25e-04]
Epoch 38 TRAIN:  97%|█████████▋| 132/136 [00:29<00:00,  5.02it/s, epoch=38, grad_norm=116.514, loss=285.4534, lr=4.25e-04]
Epoch 38 TRAIN:  97%|█████████▋| 132/136 [00:29<00:00,  5.02it/s, epoch=38, grad_norm=122.137, loss=262.4284, lr=4.25e-04]
Epoch 38 TRAIN:  97%|█████████▋| 132/136 [00:29<00:00,  5.02it/s, epoch=38, grad_norm=229.638, loss=302.1610, lr=4.25e-04]
Epoch 38 TRAIN:  97%|█████████▋| 132/136 [00:29<00:00,  5.02it/s, epoch=38, grad_norm=129.903, loss=379.4792, lr=4.25e-04]
Epoch 38 TRAIN:  99%|█████████▉| 135/136 [00:29<00:00,  5.22it/s, epoch=38, grad_norm=129.903, loss=379.4792, lr=4.25e-04]
Epoch 38 TRAIN:  99%|█████████▉| 135/136 [00:30<00:00,  5.22it/s, epoch=38, grad_norm=129.033, loss=346.5295, lr=4.25e-04]
Epoch 38 TRAIN: 100%|██████████| 136/136 [00:30<00:00,  4.53it/s, epoch=38, grad_norm=129.033, loss=346.5295, lr=4.25e-04]

CV:   0%|          | 0/6 [00:00<?, ?it/s]/home/rookie/Documents/batuka/projects/wenet/wenet/wenet/utils/train_utils.py:689: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  amp_autocast(enabled=dtype is not None,

CV:  17%|█▋        | 1/6 [00:00<00:03,  1.30it/s]
                                                 
2026-04-16 20:11:25,906 INFO Epoch 38 Step 5316 CV info lr 4.2528e-04 cv_loss 453.8018243963068 rank 0 acc 0.2607686792810758
2026-04-16 20:11:25,908 INFO [Rank 0] Checkpoint: save to checkpoint exp/conformer_mongolian/epoch_38.pt
2026-04-16 20:11:26,057 INFO Epoch 39 Step 5316 TRAIN info lr 4.2528e-04 rank 0
2026-04-16 20:11:26,073 INFO using accumulate grad, new batch size is 1 times larger than before

Epoch 39 TRAIN:   0%|          | 0/136 [00:00<?, ?it/s]
Epoch 39 TRAIN:   0%|          | 0/136 [00:03<?, ?it/s, epoch=39, grad_norm=524.955, loss=471.0723, lr=4.25e-04]
Epoch 39 TRAIN:   1%|          | 1/136 [00:03<08:51,  3.94s/it, epoch=39, grad_norm=524.955, loss=471.0723, lr=4.25e-04]
Epoch 39 TRAIN:   1%|          | 1/136 [00:04<08:51,  3.94s/it, epoch=39, grad_norm=368.037, loss=342.2447, lr=4.25e-04]
Epoch 39 TRAIN:   1%|          | 1/136 [00:04<08:51,  3.94s/it, epoch=39, grad_norm=232.869, loss=313.1439, lr=4.26e-04]
Epoch 39 TRAIN:   1%|          | 1/136 [00:04<08:51,  3.94s/it, epoch=39, grad_norm=318.130, loss=376.7225, lr=4.26e-04]
Epoch 39 TRAIN:   1%|          | 1/136 [00:04<08:51,  3.94s/it, epoch=39, grad_norm=299.784, loss=319.2637, lr=4.26e-04]
Epoch 39 TRAIN:   1%|          | 1/136 [00:04<08:51,  3.94s/it, epoch=39, grad_norm=124.604, loss=361.9222, lr=4.26e-04]
Epoch 39 TRAIN:   4%|▍         | 6/136 [00:04<01:15,  1.73it/s, epoch=39, grad_norm=124.604, loss=361.9222, lr=4.26e-04]
Epoch 39 TRAIN:   4%|▍         | 6/136 [00:04<01:15,  1.73it/s, epoch=39, grad_norm=104.359, loss=322.8247, lr=4.26e-04]
Epoch 39 TRAIN:   4%|▍         | 6/136 [00:04<01:15,  1.73it/s, epoch=39, grad_norm=145.355, loss=299.4329, lr=4.26e-04]
Epoch 39 TRAIN:   4%|▍         | 6/136 [00:04<01:15,  1.73it/s, epoch=39, grad_norm=159.778, loss=315.6325, lr=4.26e-04]
Epoch 39 TRAIN:   4%|▍         | 6/136 [00:04<01:15,  1.73it/s, epoch=39, grad_norm=172.610, loss=349.6824, lr=4.26e-04]
Epoch 39 TRAIN:   4%|▍         | 6/136 [00:05<01:15,  1.73it/s, epoch=39, grad_norm=116.923, loss=342.6901, lr=4.26e-04]
Epoch 39 TRAIN:   8%|▊         | 11/136 [00:05<00:40,  3.08it/s, epoch=39, grad_norm=116.923, loss=342.6901, lr=4.26e-04]
Epoch 39 TRAIN:   8%|▊         | 11/136 [00:05<00:40,  3.08it/s, epoch=39, grad_norm=87.300, loss=331.2700, lr=4.26e-04] 
Epoch 39 TRAIN:   8%|▊         | 11/136 [00:05<00:40,  3.08it/s, epoch=39, grad_norm=67.618, loss=322.9310, lr=4.26e-04]
Epoch 39 TRAIN:   8%|▊         | 11/136 [00:05<00:40,  3.08it/s, epoch=39, grad_norm=114.452, loss=323.3062, lr=4.26e-04]
Epoch 39 TRAIN:   8%|▊         | 11/136 [00:05<00:40,  3.08it/s, epoch=39, grad_norm=123.826, loss=305.3860, lr=4.26e-04]
Epoch 39 TRAIN:  11%|█         | 15/136 [00:05<00:31,  3.84it/s, epoch=39, grad_norm=123.826, loss=305.3860, lr=4.26e-04]
Epoch 39 TRAIN:  11%|█         | 15/136 [00:05<00:31,  3.84it/s, epoch=39, grad_norm=155.998, loss=278.8089, lr=4.27e-04]
Epoch 39 TRAIN:  11%|█         | 15/136 [00:06<00:31,  3.84it/s, epoch=39, grad_norm=73.426, loss=350.1069, lr=4.27e-04] 
Epoch 39 TRAIN:  11%|█         | 15/136 [00:06<00:31,  3.84it/s, epoch=39, grad_norm=112.319, loss=291.3273, lr=4.27e-04]
Epoch 39 TRAIN:  11%|█         | 15/136 [00:06<00:31,  3.84it/s, epoch=39, grad_norm=72.928, loss=346.8009, lr=4.27e-04] 
Epoch 39 TRAIN:  14%|█▍        | 19/136 [00:06<00:27,  4.26it/s, epoch=39, grad_norm=72.928, loss=346.8009, lr=4.27e-04]
Epoch 39 TRAIN:  14%|█▍        | 19/136 [00:06<00:27,  4.26it/s, epoch=39, grad_norm=88.442, loss=330.3414, lr=4.27e-04]
Epoch 39 TRAIN:  14%|█▍        | 19/136 [00:06<00:27,  4.26it/s, epoch=39, grad_norm=78.987, loss=330.8592, lr=4.27e-04]
Epoch 39 TRAIN:  14%|█▍        | 19/136 [00:07<00:27,  4.26it/s, epoch=39, grad_norm=121.879, loss=350.1303, lr=4.27e-04]
Epoch 39 TRAIN:  14%|█▍        | 19/136 [00:07<00:27,  4.26it/s, epoch=39, grad_norm=62.437, loss=327.6190, lr=4.27e-04] 
Epoch 39 TRAIN:  17%|█▋        | 23/136 [00:07<00:25,  4.51it/s, epoch=39, grad_norm=62.437, loss=327.6190, lr=4.27e-04]
Epoch 39 TRAIN:  17%|█▋        | 23/136 [00:07<00:25,  4.51it/s, epoch=39, grad_norm=61.171, loss=324.6003, lr=4.27e-04]
Epoch 39 TRAIN:  17%|█▋        | 23/136 [00:07<00:25,  4.51it/s, epoch=39, grad_norm=73.418, loss=337.6275, lr=4.27e-04]
Epoch 39 TRAIN:  17%|█▋        | 23/136 [00:07<00:25,  4.51it/s, epoch=39, grad_norm=160.810, loss=289.1196, lr=4.27e-04]
Epoch 39 TRAIN:  17%|█▋        | 23/136 [00:07<00:25,  4.51it/s, epoch=39, grad_norm=161.641, loss=276.4875, lr=4.27e-04]
Epoch 39 TRAIN:  20%|█▉        | 27/136 [00:07<00:22,  4.92it/s, epoch=39, grad_norm=161.641, loss=276.4875, lr=4.27e-04]
Epoch 39 TRAIN:  20%|█▉        | 27/136 [00:08<00:22,  4.92it/s, epoch=39, grad_norm=93.845, loss=310.5776, lr=4.28e-04] 
Epoch 39 TRAIN:  20%|█▉        | 27/136 [00:08<00:22,  4.92it/s, epoch=39, grad_norm=76.334, loss=294.7175, lr=4.28e-04]
Epoch 39 TRAIN:  20%|█▉        | 27/136 [00:08<00:22,  4.92it/s, epoch=39, grad_norm=116.987, loss=336.4679, lr=4.28e-04]
Epoch 39 TRAIN:  22%|██▏       | 30/136 [00:08<00:21,  4.92it/s, epoch=39, grad_norm=116.987, loss=336.4679, lr=4.28e-04]
Epoch 39 TRAIN:  22%|██▏       | 30/136 [00:08<00:21,  4.92it/s, epoch=39, grad_norm=79.143, loss=319.3134, lr=4.28e-04] 
Epoch 39 TRAIN:  22%|██▏       | 30/136 [00:08<00:21,  4.92it/s, epoch=39, grad_norm=92.110, loss=315.8401, lr=4.28e-04]
Epoch 39 TRAIN:  22%|██▏       | 30/136 [00:09<00:21,  4.92it/s, epoch=39, grad_norm=71.929, loss=280.1888, lr=4.28e-04]
Epoch 39 TRAIN:  24%|██▍       | 33/136 [00:09<00:21,  4.76it/s, epoch=39, grad_norm=71.929, loss=280.1888, lr=4.28e-04]
Epoch 39 TRAIN:  24%|██▍       | 33/136 [00:09<00:21,  4.76it/s, epoch=39, grad_norm=58.697, loss=300.9792, lr=4.28e-04]
Epoch 39 TRAIN:  24%|██▍       | 33/136 [00:09<00:21,  4.76it/s, epoch=39, grad_norm=131.726, loss=353.0028, lr=4.28e-04]
Epoch 39 TRAIN:  24%|██▍       | 33/136 [00:09<00:21,  4.76it/s, epoch=39, grad_norm=75.183, loss=331.6582, lr=4.28e-04] 
Epoch 39 TRAIN:  26%|██▋       | 36/136 [00:09<00:20,  4.90it/s, epoch=39, grad_norm=75.183, loss=331.6582, lr=4.28e-04]
Epoch 39 TRAIN:  26%|██▋       | 36/136 [00:09<00:20,  4.90it/s, epoch=39, grad_norm=76.862, loss=325.8721, lr=4.28e-04]
Epoch 39 TRAIN:  26%|██▋       | 36/136 [00:10<00:20,  4.90it/s, epoch=39, grad_norm=64.187, loss=316.9420, lr=4.28e-04]
Epoch 39 TRAIN:  26%|██▋       | 36/136 [00:10<00:20,  4.90it/s, epoch=39, grad_norm=70.515, loss=320.9355, lr=4.28e-04]
Epoch 39 TRAIN:  29%|██▊       | 39/136 [00:10<00:19,  5.10it/s, epoch=39, grad_norm=70.515, loss=320.9355, lr=4.28e-04]
Epoch 39 TRAIN:  29%|██▊       | 39/136 [00:10<00:19,  5.10it/s, epoch=39, grad_norm=75.006, loss=334.8859, lr=4.28e-04]
Epoch 39 TRAIN:  29%|██▊       | 39/136 [00:10<00:19,  5.10it/s, epoch=39, grad_norm=145.925, loss=303.3308, lr=4.29e-04]
Epoch 39 TRAIN:  29%|██▊       | 39/136 [00:11<00:19,  5.10it/s, epoch=39, grad_norm=93.223, loss=332.1296, lr=4.29e-04] 
Epoch 39 TRAIN:  31%|███       | 42/136 [00:11<00:20,  4.70it/s, epoch=39, grad_norm=93.223, loss=332.1296, lr=4.29e-04]
Epoch 39 TRAIN:  31%|███       | 42/136 [00:11<00:20,  4.70it/s, epoch=39, grad_norm=79.165, loss=315.4647, lr=4.29e-04]
Epoch 39 TRAIN:  31%|███       | 42/136 [00:11<00:20,  4.70it/s, epoch=39, grad_norm=90.648, loss=299.8169, lr=4.29e-04]
Epoch 39 TRAIN:  31%|███       | 42/136 [00:11<00:20,  4.70it/s, epoch=39, grad_norm=104.155, loss=351.7640, lr=4.29e-04]
Epoch 39 TRAIN:  33%|███▎      | 45/136 [00:11<00:19,  4.75it/s, epoch=39, grad_norm=104.155, loss=351.7640, lr=4.29e-04]
Epoch 39 TRAIN:  33%|███▎      | 45/136 [00:11<00:19,  4.75it/s, epoch=39, grad_norm=76.893, loss=315.7787, lr=4.29e-04] 
Epoch 39 TRAIN:  33%|███▎      | 45/136 [00:11<00:19,  4.75it/s, epoch=39, grad_norm=61.318, loss=307.1568, lr=4.29e-04]
Epoch 39 TRAIN:  33%|███▎      | 45/136 [00:12<00:19,  4.75it/s, epoch=39, grad_norm=78.651, loss=326.0446, lr=4.29e-04]
Epoch 39 TRAIN:  33%|███▎      | 45/136 [00:12<00:19,  4.75it/s, epoch=39, grad_norm=71.945, loss=342.8297, lr=4.29e-04]
Epoch 39 TRAIN:  36%|███▌      | 49/136 [00:12<00:16,  5.15it/s, epoch=39, grad_norm=71.945, loss=342.8297, lr=4.29e-04]
Epoch 39 TRAIN:  36%|███▌      | 49/136 [00:12<00:16,  5.15it/s, epoch=39, grad_norm=104.923, loss=279.3241, lr=4.29e-04]
Epoch 39 TRAIN:  36%|███▌      | 49/136 [00:12<00:16,  5.15it/s, epoch=39, grad_norm=99.207, loss=254.3129, lr=4.29e-04] 
Epoch 39 TRAIN:  36%|███▌      | 49/136 [00:13<00:16,  5.15it/s, epoch=39, grad_norm=78.486, loss=308.7987, lr=4.29e-04]
Epoch 39 TRAIN:  38%|███▊      | 52/136 [00:13<00:17,  4.71it/s, epoch=39, grad_norm=78.486, loss=308.7987, lr=4.29e-04]
Epoch 39 TRAIN:  38%|███▊      | 52/136 [00:13<00:17,  4.71it/s, epoch=39, grad_norm=153.584, loss=356.7463, lr=4.30e-04]
Epoch 39 TRAIN:  38%|███▊      | 52/136 [00:13<00:17,  4.71it/s, epoch=39, grad_norm=57.243, loss=263.1299, lr=4.30e-04] 
Epoch 39 TRAIN:  38%|███▊      | 52/136 [00:13<00:17,  4.71it/s, epoch=39, grad_norm=121.921, loss=317.1700, lr=4.30e-04]
Epoch 39 TRAIN:  40%|████      | 55/136 [00:13<00:17,  4.73it/s, epoch=39, grad_norm=121.921, loss=317.1700, lr=4.30e-04]
Epoch 39 TRAIN:  40%|████      | 55/136 [00:13<00:17,  4.73it/s, epoch=39, grad_norm=77.447, loss=300.2076, lr=4.30e-04] 
Epoch 39 TRAIN:  40%|████      | 55/136 [00:14<00:17,  4.73it/s, epoch=39, grad_norm=82.820, loss=351.8786, lr=4.30e-04]
Epoch 39 TRAIN:  40%|████      | 55/136 [00:14<00:17,  4.73it/s, epoch=39, grad_norm=64.995, loss=318.1587, lr=4.30e-04]
Epoch 39 TRAIN:  43%|████▎     | 58/136 [00:14<00:16,  4.86it/s, epoch=39, grad_norm=64.995, loss=318.1587, lr=4.30e-04]
Epoch 39 TRAIN:  43%|████▎     | 58/136 [00:14<00:16,  4.86it/s, epoch=39, grad_norm=60.782, loss=293.5380, lr=4.30e-04]
Epoch 39 TRAIN:  43%|████▎     | 58/136 [00:14<00:16,  4.86it/s, epoch=39, grad_norm=83.682, loss=318.7238, lr=4.30e-04]
Epoch 39 TRAIN:  43%|████▎     | 58/136 [00:14<00:16,  4.86it/s, epoch=39, grad_norm=65.622, loss=322.6986, lr=4.30e-04]
Epoch 39 TRAIN:  45%|████▍     | 61/136 [00:14<00:15,  4.86it/s, epoch=39, grad_norm=65.622, loss=322.6986, lr=4.30e-04]
Epoch 39 TRAIN:  45%|████▍     | 61/136 [00:15<00:15,  4.86it/s, epoch=39, grad_norm=78.497, loss=306.8109, lr=4.30e-04]
Epoch 39 TRAIN:  45%|████▍     | 61/136 [00:15<00:15,  4.86it/s, epoch=39, grad_norm=89.093, loss=372.2349, lr=4.30e-04]
Epoch 39 TRAIN:  45%|████▍     | 61/136 [00:15<00:15,  4.86it/s, epoch=39, grad_norm=93.088, loss=336.6010, lr=4.30e-04]
Epoch 39 TRAIN:  47%|████▋     | 64/136 [00:15<00:14,  4.88it/s, epoch=39, grad_norm=93.088, loss=336.6010, lr=4.30e-04]
Epoch 39 TRAIN:  47%|████▋     | 64/136 [00:15<00:14,  4.88it/s, epoch=39, grad_norm=281.223, loss=315.0042, lr=4.30e-04]
Epoch 39 TRAIN:  47%|████▋     | 64/136 [00:15<00:14,  4.88it/s, epoch=39, grad_norm=245.195, loss=283.4826, lr=4.31e-04]
Epoch 39 TRAIN:  47%|████▋     | 64/136 [00:16<00:14,  4.88it/s, epoch=39, grad_norm=219.653, loss=252.6398, lr=4.31e-04]
Epoch 39 TRAIN:  49%|████▉     | 67/136 [00:16<00:14,  4.87it/s, epoch=39, grad_norm=219.653, loss=252.6398, lr=4.31e-04]
Epoch 39 TRAIN:  49%|████▉     | 67/136 [00:16<00:14,  4.87it/s, epoch=39, grad_norm=201.218, loss=246.9662, lr=4.31e-04]
Epoch 39 TRAIN:  49%|████▉     | 67/136 [00:16<00:14,  4.87it/s, epoch=39, grad_norm=82.962, loss=328.3170, lr=4.31e-04] 
Epoch 39 TRAIN:  49%|████▉     | 67/136 [00:16<00:14,  4.87it/s, epoch=39, grad_norm=139.625, loss=359.6474, lr=4.31e-04]
Epoch 39 TRAIN:  51%|█████▏    | 70/136 [00:16<00:13,  4.84it/s, epoch=39, grad_norm=139.625, loss=359.6474, lr=4.31e-04]
Epoch 39 TRAIN:  51%|█████▏    | 70/136 [00:16<00:13,  4.84it/s, epoch=39, grad_norm=74.523, loss=290.6046, lr=4.31e-04] 
Epoch 39 TRAIN:  51%|█████▏    | 70/136 [00:17<00:13,  4.84it/s, epoch=39, grad_norm=103.083, loss=338.4572, lr=4.31e-04]
Epoch 39 TRAIN:  51%|█████▏    | 70/136 [00:17<00:13,  4.84it/s, epoch=39, grad_norm=221.954, loss=282.0428, lr=4.31e-04]
Epoch 39 TRAIN:  54%|█████▎    | 73/136 [00:17<00:12,  4.89it/s, epoch=39, grad_norm=221.954, loss=282.0428, lr=4.31e-04]
Epoch 39 TRAIN:  54%|█████▎    | 73/136 [00:17<00:12,  4.89it/s, epoch=39, grad_norm=173.512, loss=278.0691, lr=4.31e-04]
Epoch 39 TRAIN:  54%|█████▎    | 73/136 [00:17<00:12,  4.89it/s, epoch=39, grad_norm=61.182, loss=278.7581, lr=4.31e-04] 
Epoch 39 TRAIN:  54%|█████▎    | 73/136 [00:18<00:12,  4.89it/s, epoch=39, grad_norm=75.228, loss=306.5814, lr=4.31e-04]
Epoch 39 TRAIN:  56%|█████▌    | 76/136 [00:18<00:12,  4.82it/s, epoch=39, grad_norm=75.228, loss=306.5814, lr=4.31e-04]
Epoch 39 TRAIN:  56%|█████▌    | 76/136 [00:18<00:12,  4.82it/s, epoch=39, grad_norm=78.989, loss=309.0109, lr=4.31e-04]
Epoch 39 TRAIN:  56%|█████▌    | 76/136 [00:18<00:12,  4.82it/s, epoch=39, grad_norm=68.839, loss=280.1910, lr=4.32e-04]
Epoch 39 TRAIN:  56%|█████▌    | 76/136 [00:18<00:12,  4.82it/s, epoch=39, grad_norm=105.435, loss=316.6261, lr=4.32e-04]
Epoch 39 TRAIN:  58%|█████▊    | 79/136 [00:18<00:11,  4.79it/s, epoch=39, grad_norm=105.435, loss=316.6261, lr=4.32e-04]
Epoch 39 TRAIN:  58%|█████▊    | 79/136 [00:18<00:11,  4.79it/s, epoch=39, grad_norm=75.932, loss=308.4939, lr=4.32e-04] 
Epoch 39 TRAIN:  58%|█████▊    | 79/136 [00:19<00:11,  4.79it/s, epoch=39, grad_norm=111.724, loss=310.2238, lr=4.32e-04]
Epoch 39 TRAIN:  58%|█████▊    | 79/136 [00:19<00:11,  4.79it/s, epoch=39, grad_norm=193.612, loss=222.4540, lr=4.32e-04]
Epoch 39 TRAIN:  60%|██████    | 82/136 [00:19<00:11,  4.87it/s, epoch=39, grad_norm=193.612, loss=222.4540, lr=4.32e-04]
Epoch 39 TRAIN:  60%|██████    | 82/136 [00:19<00:11,  4.87it/s, epoch=39, grad_norm=142.611, loss=334.4875, lr=4.32e-04]
Epoch 39 TRAIN:  60%|██████    | 82/136 [00:19<00:11,  4.87it/s, epoch=39, grad_norm=109.007, loss=338.8223, lr=4.32e-04]
Epoch 39 TRAIN:  60%|██████    | 82/136 [00:19<00:11,  4.87it/s, epoch=39, grad_norm=130.929, loss=328.1830, lr=4.32e-04]
Epoch 39 TRAIN:  62%|██████▎   | 85/136 [00:19<00:10,  4.84it/s, epoch=39, grad_norm=130.929, loss=328.1830, lr=4.32e-04]
Epoch 39 TRAIN:  62%|██████▎   | 85/136 [00:20<00:10,  4.84it/s, epoch=39, grad_norm=106.029, loss=307.3649, lr=4.32e-04]
Epoch 39 TRAIN:  62%|██████▎   | 85/136 [00:20<00:10,  4.84it/s, epoch=39, grad_norm=96.357, loss=252.5766, lr=4.32e-04] 
Epoch 39 TRAIN:  62%|██████▎   | 85/136 [00:20<00:10,  4.84it/s, epoch=39, grad_norm=150.426, loss=260.8895, lr=4.32e-04]
Epoch 39 TRAIN:  65%|██████▍   | 88/136 [00:20<00:10,  4.77it/s, epoch=39, grad_norm=150.426, loss=260.8895, lr=4.32e-04]
Epoch 39 TRAIN:  65%|██████▍   | 88/136 [00:20<00:10,  4.77it/s, epoch=39, grad_norm=113.134, loss=351.6718, lr=4.32e-04]
Epoch 39 TRAIN:  65%|██████▍   | 88/136 [00:20<00:10,  4.77it/s, epoch=39, grad_norm=152.291, loss=319.1083, lr=4.32e-04]
Epoch 39 TRAIN:  65%|██████▍   | 88/136 [00:21<00:10,  4.77it/s, epoch=39, grad_norm=76.744, loss=291.6591, lr=4.33e-04] 
Epoch 39 TRAIN:  67%|██████▋   | 91/136 [00:21<00:09,  4.77it/s, epoch=39, grad_norm=76.744, loss=291.6591, lr=4.33e-04]
Epoch 39 TRAIN:  67%|██████▋   | 91/136 [00:21<00:09,  4.77it/s, epoch=39, grad_norm=69.637, loss=277.1396, lr=4.33e-04]
Epoch 39 TRAIN:  67%|██████▋   | 91/136 [00:21<00:09,  4.77it/s, epoch=39, grad_norm=74.860, loss=264.0896, lr=4.33e-04]
Epoch 39 TRAIN:  67%|██████▋   | 91/136 [00:21<00:09,  4.77it/s, epoch=39, grad_norm=159.409, loss=277.5526, lr=4.33e-04]
Epoch 39 TRAIN:  69%|██████▉   | 94/136 [00:21<00:08,  4.79it/s, epoch=39, grad_norm=159.409, loss=277.5526, lr=4.33e-04]
Epoch 39 TRAIN:  69%|██████▉   | 94/136 [00:22<00:08,  4.79it/s, epoch=39, grad_norm=240.365, loss=373.9124, lr=4.33e-04]
Epoch 39 TRAIN:  69%|██████▉   | 94/136 [00:22<00:08,  4.79it/s, epoch=39, grad_norm=196.251, loss=385.8676, lr=4.33e-04]
Epoch 39 TRAIN:  69%|██████▉   | 94/136 [00:22<00:08,  4.79it/s, epoch=39, grad_norm=175.924, loss=305.2156, lr=4.33e-04]
Epoch 39 TRAIN:  71%|███████▏  | 97/136 [00:22<00:08,  4.74it/s, epoch=39, grad_norm=175.924, loss=305.2156, lr=4.33e-04]
Epoch 39 TRAIN:  71%|███████▏  | 97/136 [00:22<00:08,  4.74it/s, epoch=39, grad_norm=263.400, loss=245.2907, lr=4.33e-04]
Epoch 39 TRAIN:  71%|███████▏  | 97/136 [00:22<00:08,  4.74it/s, epoch=39, grad_norm=90.522, loss=279.6975, lr=4.33e-04] 
Epoch 39 TRAIN:  71%|███████▏  | 97/136 [00:23<00:08,  4.74it/s, epoch=39, grad_norm=135.027, loss=268.2844, lr=4.33e-04]2026-04-16 20:11:49,123 DEBUG TRAIN | steps/sec 4.360| Batch 39/100 loss 268.284424 loss_att 235.046051 loss_ctc 345.840637 th_accuracy 0.310881 lr 4.3328e-04 grad_norm 135.026871 rank 0

Epoch 39 TRAIN:  74%|███████▎  | 100/136 [00:23<00:07,  4.81it/s, epoch=39, grad_norm=135.027, loss=268.2844, lr=4.33e-04]
Epoch 39 TRAIN:  74%|███████▎  | 100/136 [00:23<00:07,  4.81it/s, epoch=39, grad_norm=130.442, loss=224.8838, lr=4.33e-04]
Epoch 39 TRAIN:  74%|███████▎  | 100/136 [00:23<00:07,  4.81it/s, epoch=39, grad_norm=90.113, loss=292.3067, lr=4.33e-04] 
Epoch 39 TRAIN:  74%|███████▎  | 100/136 [00:23<00:07,  4.81it/s, epoch=39, grad_norm=87.430, loss=248.6387, lr=4.34e-04]
Epoch 39 TRAIN:  76%|███████▌  | 103/136 [00:23<00:07,  4.67it/s, epoch=39, grad_norm=87.430, loss=248.6387, lr=4.34e-04]
Epoch 39 TRAIN:  76%|███████▌  | 103/136 [00:23<00:07,  4.67it/s, epoch=39, grad_norm=195.375, loss=398.4213, lr=4.34e-04]
Epoch 39 TRAIN:  76%|███████▌  | 103/136 [00:24<00:07,  4.67it/s, epoch=39, grad_norm=104.091, loss=304.6477, lr=4.34e-04]
Epoch 39 TRAIN:  76%|███████▌  | 103/136 [00:24<00:07,  4.67it/s, epoch=39, grad_norm=80.530, loss=333.0109, lr=4.34e-04] 
Epoch 39 TRAIN:  78%|███████▊  | 106/136 [00:24<00:06,  4.58it/s, epoch=39, grad_norm=80.530, loss=333.0109, lr=4.34e-04]
Epoch 39 TRAIN:  78%|███████▊  | 106/136 [00:24<00:06,  4.58it/s, epoch=39, grad_norm=83.403, loss=260.8116, lr=4.34e-04]
Epoch 39 TRAIN:  78%|███████▊  | 106/136 [00:24<00:06,  4.58it/s, epoch=39, grad_norm=165.732, loss=331.6379, lr=4.34e-04]
Epoch 39 TRAIN:  78%|███████▊  | 106/136 [00:24<00:06,  4.58it/s, epoch=39, grad_norm=96.140, loss=274.3587, lr=4.34e-04] 
Epoch 39 TRAIN:  80%|████████  | 109/136 [00:24<00:05,  4.84it/s, epoch=39, grad_norm=96.140, loss=274.3587, lr=4.34e-04]
Epoch 39 TRAIN:  80%|████████  | 109/136 [00:25<00:05,  4.84it/s, epoch=39, grad_norm=271.171, loss=390.4885, lr=4.34e-04]
Epoch 39 TRAIN:  80%|████████  | 109/136 [00:25<00:05,  4.84it/s, epoch=39, grad_norm=93.805, loss=251.6718, lr=4.34e-04] 
Epoch 39 TRAIN:  80%|████████  | 109/136 [00:25<00:05,  4.84it/s, epoch=39, grad_norm=187.339, loss=241.9244, lr=4.34e-04]
Epoch 39 TRAIN:  82%|████████▏ | 112/136 [00:25<00:05,  4.75it/s, epoch=39, grad_norm=187.339, loss=241.9244, lr=4.34e-04]
Epoch 39 TRAIN:  82%|████████▏ | 112/136 [00:25<00:05,  4.75it/s, epoch=39, grad_norm=172.275, loss=315.8984, lr=4.34e-04]
Epoch 39 TRAIN:  82%|████████▏ | 112/136 [00:26<00:05,  4.75it/s, epoch=39, grad_norm=163.570, loss=302.8435, lr=4.34e-04]
Epoch 39 TRAIN:  82%|████████▏ | 112/136 [00:26<00:05,  4.75it/s, epoch=39, grad_norm=95.244, loss=253.8369, lr=4.34e-04] 
Epoch 39 TRAIN:  85%|████████▍ | 115/136 [00:26<00:04,  4.82it/s, epoch=39, grad_norm=95.244, loss=253.8369, lr=4.34e-04]
Epoch 39 TRAIN:  85%|████████▍ | 115/136 [00:26<00:04,  4.82it/s, epoch=39, grad_norm=84.759, loss=264.7245, lr=4.35e-04]
Epoch 39 TRAIN:  85%|████████▍ | 115/136 [00:26<00:04,  4.82it/s, epoch=39, grad_norm=129.960, loss=311.7447, lr=4.35e-04]
Epoch 39 TRAIN:  85%|████████▍ | 115/136 [00:26<00:04,  4.82it/s, epoch=39, grad_norm=235.373, loss=382.2789, lr=4.35e-04]
Epoch 39 TRAIN:  87%|████████▋ | 118/136 [00:26<00:03,  4.73it/s, epoch=39, grad_norm=235.373, loss=382.2789, lr=4.35e-04]
Epoch 39 TRAIN:  87%|████████▋ | 118/136 [00:27<00:03,  4.73it/s, epoch=39, grad_norm=124.007, loss=284.0539, lr=4.35e-04]
Epoch 39 TRAIN:  87%|████████▋ | 118/136 [00:27<00:03,  4.73it/s, epoch=39, grad_norm=87.655, loss=305.2722, lr=4.35e-04] 
Epoch 39 TRAIN:  87%|████████▋ | 118/136 [00:27<00:03,  4.73it/s, epoch=39, grad_norm=97.194, loss=314.8091, lr=4.35e-04]
Epoch 39 TRAIN:  89%|████████▉ | 121/136 [00:27<00:03,  4.91it/s, epoch=39, grad_norm=97.194, loss=314.8091, lr=4.35e-04]
Epoch 39 TRAIN:  89%|████████▉ | 121/136 [00:27<00:03,  4.91it/s, epoch=39, grad_norm=118.373, loss=344.6994, lr=4.35e-04]
Epoch 39 TRAIN:  89%|████████▉ | 121/136 [00:27<00:03,  4.91it/s, epoch=39, grad_norm=106.049, loss=264.4108, lr=4.35e-04]
Epoch 39 TRAIN:  89%|████████▉ | 121/136 [00:28<00:03,  4.91it/s, epoch=39, grad_norm=107.762, loss=351.2926, lr=4.35e-04]
Epoch 39 TRAIN:  91%|█████████ | 124/136 [00:28<00:02,  4.80it/s, epoch=39, grad_norm=107.762, loss=351.2926, lr=4.35e-04]
Epoch 39 TRAIN:  91%|█████████ | 124/136 [00:28<00:02,  4.80it/s, epoch=39, grad_norm=142.630, loss=377.5602, lr=4.35e-04]
Epoch 39 TRAIN:  91%|█████████ | 124/136 [00:28<00:02,  4.80it/s, epoch=39, grad_norm=138.893, loss=315.2542, lr=4.35e-04]
Epoch 39 TRAIN:  91%|█████████ | 124/136 [00:28<00:02,  4.80it/s, epoch=39, grad_norm=133.458, loss=305.2561, lr=4.35e-04]
Epoch 39 TRAIN:  93%|█████████▎| 127/136 [00:28<00:01,  4.82it/s, epoch=39, grad_norm=133.458, loss=305.2561, lr=4.35e-04]
Epoch 39 TRAIN:  93%|█████████▎| 127/136 [00:28<00:01,  4.82it/s, epoch=39, grad_norm=167.940, loss=285.9409, lr=4.36e-04]
Epoch 39 TRAIN:  93%|█████████▎| 127/136 [00:28<00:01,  4.82it/s, epoch=39, grad_norm=137.553, loss=326.4207, lr=4.36e-04]
Epoch 39 TRAIN:  93%|█████████▎| 127/136 [00:29<00:01,  4.82it/s, epoch=39, grad_norm=103.715, loss=295.0271, lr=4.36e-04]
Epoch 39 TRAIN:  93%|█████████▎| 127/136 [00:29<00:01,  4.82it/s, epoch=39, grad_norm=174.974, loss=404.6996, lr=4.36e-04]
Epoch 39 TRAIN:  96%|█████████▋| 131/136 [00:29<00:00,  5.19it/s, epoch=39, grad_norm=174.974, loss=404.6996, lr=4.36e-04]
Epoch 39 TRAIN:  96%|█████████▋| 131/136 [00:29<00:00,  5.19it/s, epoch=39, grad_norm=127.416, loss=323.5684, lr=4.36e-04]
Epoch 39 TRAIN:  96%|█████████▋| 131/136 [00:29<00:00,  5.19it/s, epoch=39, grad_norm=163.810, loss=383.4764, lr=4.36e-04]
Epoch 39 TRAIN:  96%|█████████▋| 131/136 [00:29<00:00,  5.19it/s, epoch=39, grad_norm=157.168, loss=292.3937, lr=4.36e-04]
Epoch 39 TRAIN:  96%|█████████▋| 131/136 [00:30<00:00,  5.19it/s, epoch=39, grad_norm=174.985, loss=397.4556, lr=4.36e-04]
Epoch 39 TRAIN:  99%|█████████▉| 135/136 [00:30<00:00,  5.46it/s, epoch=39, grad_norm=174.985, loss=397.4556, lr=4.36e-04]
Epoch 39 TRAIN:  99%|█████████▉| 135/136 [00:30<00:00,  5.46it/s, epoch=39, grad_norm=156.660, loss=388.5612, lr=4.36e-04]
Epoch 39 TRAIN:  99%|█████████▉| 135/136 [00:30<00:00,  5.46it/s, epoch=39, grad_norm=141.368, loss=422.3212, lr=4.36e-04]
Epoch 39 TRAIN: 137it [00:30,  4.49it/s, epoch=39, grad_norm=141.368, loss=422.3212, lr=4.36e-04]                         

CV:   0%|          | 0/6 [00:00<?, ?it/s]/home/rookie/Documents/batuka/projects/wenet/wenet/wenet/utils/train_utils.py:689: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  amp_autocast(enabled=dtype is not None,

CV:  17%|█▋        | 1/6 [00:00<00:03,  1.31it/s]
                                                 
2026-04-16 20:11:57,424 INFO Epoch 39 Step 5453 CV info lr 4.3624e-04 cv_loss 524.6930597478694 rank 0 acc 0.2668128137787183
2026-04-16 20:11:57,426 INFO [Rank 0] Checkpoint: save to checkpoint exp/conformer_mongolian/epoch_39.pt
2026-04-16 20:11:57,538 INFO Epoch 40 Step 5453 TRAIN info lr 4.3624e-04 rank 0
2026-04-16 20:11:57,557 INFO using accumulate grad, new batch size is 1 times larger than before

Epoch 40 TRAIN:   0%|          | 0/137 [00:00<?, ?it/s]
Epoch 40 TRAIN:   0%|          | 0/137 [00:03<?, ?it/s, epoch=40, grad_norm=470.609, loss=434.4249, lr=4.36e-04]
Epoch 40 TRAIN:   1%|          | 1/137 [00:03<08:39,  3.82s/it, epoch=40, grad_norm=470.609, loss=434.4249, lr=4.36e-04]
Epoch 40 TRAIN:   1%|          | 1/137 [00:04<08:39,  3.82s/it, epoch=40, grad_norm=223.852, loss=265.1035, lr=4.36e-04]
Epoch 40 TRAIN:   1%|          | 1/137 [00:04<08:39,  3.82s/it, epoch=40, grad_norm=377.993, loss=388.9482, lr=4.36e-04]
Epoch 40 TRAIN:   1%|          | 1/137 [00:04<08:39,  3.82s/it, epoch=40, grad_norm=232.730, loss=364.4409, lr=4.37e-04]
Epoch 40 TRAIN:   1%|          | 1/137 [00:04<08:39,  3.82s/it, epoch=40, grad_norm=292.794, loss=364.9073, lr=4.37e-04]
Epoch 40 TRAIN:   1%|          | 1/137 [00:04<08:39,  3.82s/it, epoch=40, grad_norm=308.368, loss=425.0319, lr=4.37e-04]
Epoch 40 TRAIN:   1%|          | 1/137 [00:04<08:39,  3.82s/it, epoch=40, grad_norm=180.725, loss=374.9574, lr=4.37e-04]
Epoch 40 TRAIN:   5%|▌         | 7/137 [00:04<01:02,  2.08it/s, epoch=40, grad_norm=180.725, loss=374.9574, lr=4.37e-04]
Epoch 40 TRAIN:   5%|▌         | 7/137 [00:04<01:02,  2.08it/s, epoch=40, grad_norm=158.523, loss=349.6583, lr=4.37e-04]
Epoch 40 TRAIN:   5%|▌         | 7/137 [00:04<01:02,  2.08it/s, epoch=40, grad_norm=86.184, loss=364.3768, lr=4.37e-04] 
Epoch 40 TRAIN:   5%|▌         | 7/137 [00:04<01:02,  2.08it/s, epoch=40, grad_norm=103.873, loss=348.9763, lr=4.37e-04]
Epoch 40 TRAIN:   5%|▌         | 7/137 [00:04<01:02,  2.08it/s, epoch=40, grad_norm=165.610, loss=290.2106, lr=4.37e-04]
Epoch 40 TRAIN:   8%|▊         | 11/137 [00:04<00:40,  3.10it/s, epoch=40, grad_norm=165.610, loss=290.2106, lr=4.37e-04]
Epoch 40 TRAIN:   8%|▊         | 11/137 [00:05<00:40,  3.10it/s, epoch=40, grad_norm=162.941, loss=300.0803, lr=4.37e-04]
Epoch 40 TRAIN:   8%|▊         | 11/137 [00:05<00:40,  3.10it/s, epoch=40, grad_norm=104.554, loss=323.4730, lr=4.37e-04]
Epoch 40 TRAIN:   8%|▊         | 11/137 [00:05<00:40,  3.10it/s, epoch=40, grad_norm=100.245, loss=378.6803, lr=4.37e-04]
Epoch 40 TRAIN:   8%|▊         | 11/137 [00:05<00:40,  3.10it/s, epoch=40, grad_norm=58.276, loss=329.1097, lr=4.37e-04] 
Epoch 40 TRAIN:  11%|█         | 15/137 [00:05<00:31,  3.89it/s, epoch=40, grad_norm=58.276, loss=329.1097, lr=4.37e-04]
Epoch 40 TRAIN:  11%|█         | 15/137 [00:05<00:31,  3.89it/s, epoch=40, grad_norm=67.146, loss=341.4065, lr=4.38e-04]
Epoch 40 TRAIN:  11%|█         | 15/137 [00:05<00:31,  3.89it/s, epoch=40, grad_norm=102.874, loss=302.6711, lr=4.38e-04]
Epoch 40 TRAIN:  11%|█         | 15/137 [00:06<00:31,  3.89it/s, epoch=40, grad_norm=89.966, loss=321.0706, lr=4.38e-04] 
Epoch 40 TRAIN:  11%|█         | 15/137 [00:06<00:31,  3.89it/s, epoch=40, grad_norm=73.804, loss=336.7928, lr=4.38e-04]
Epoch 40 TRAIN:  14%|█▍        | 19/137 [00:06<00:26,  4.52it/s, epoch=40, grad_norm=73.804, loss=336.7928, lr=4.38e-04]
Epoch 40 TRAIN:  14%|█▍        | 19/137 [00:06<00:26,  4.52it/s, epoch=40, grad_norm=76.679, loss=276.5200, lr=4.38e-04]
Epoch 40 TRAIN:  14%|█▍        | 19/137 [00:06<00:26,  4.52it/s, epoch=40, grad_norm=70.988, loss=292.2646, lr=4.38e-04]
Epoch 40 TRAIN:  14%|█▍        | 19/137 [00:06<00:26,  4.52it/s, epoch=40, grad_norm=77.599, loss=328.8298, lr=4.38e-04]
Epoch 40 TRAIN:  14%|█▍        | 19/137 [00:06<00:26,  4.52it/s, epoch=40, grad_norm=165.591, loss=365.8094, lr=4.38e-04]
Epoch 40 TRAIN:  17%|█▋        | 23/137 [00:06<00:22,  5.01it/s, epoch=40, grad_norm=165.591, loss=365.8094, lr=4.38e-04]
Epoch 40 TRAIN:  17%|█▋        | 23/137 [00:06<00:22,  5.01it/s, epoch=40, grad_norm=79.011, loss=318.1325, lr=4.38e-04] 
Epoch 40 TRAIN:  17%|█▋        | 23/137 [00:07<00:22,  5.01it/s, epoch=40, grad_norm=71.005, loss=331.7942, lr=4.38e-04]
Epoch 40 TRAIN:  17%|█▋        | 23/137 [00:07<00:22,  5.01it/s, epoch=40, grad_norm=56.669, loss=326.6391, lr=4.38e-04]
Epoch 40 TRAIN:  17%|█▋        | 23/137 [00:07<00:22,  5.01it/s, epoch=40, grad_norm=69.478, loss=320.8864, lr=4.38e-04]
Epoch 40 TRAIN:  20%|█▉        | 27/137 [00:07<00:21,  5.08it/s, epoch=40, grad_norm=69.478, loss=320.8864, lr=4.38e-04]
Epoch 40 TRAIN:  20%|█▉        | 27/137 [00:07<00:21,  5.08it/s, epoch=40, grad_norm=89.935, loss=295.0322, lr=4.38e-04]
Epoch 40 TRAIN:  20%|█▉        | 27/137 [00:07<00:21,  5.08it/s, epoch=40, grad_norm=129.257, loss=262.7405, lr=4.39e-04]
Epoch 40 TRAIN:  20%|█▉        | 27/137 [00:08<00:21,  5.08it/s, epoch=40, grad_norm=76.765, loss=317.9689, lr=4.39e-04] 
Epoch 40 TRAIN:  20%|█▉        | 27/137 [00:08<00:21,  5.08it/s, epoch=40, grad_norm=148.645, loss=357.3761, lr=4.39e-04]
Epoch 40 TRAIN:  23%|██▎       | 31/137 [00:08<00:20,  5.28it/s, epoch=40, grad_norm=148.645, loss=357.3761, lr=4.39e-04]
Epoch 40 TRAIN:  23%|██▎       | 31/137 [00:08<00:20,  5.28it/s, epoch=40, grad_norm=64.358, loss=270.0613, lr=4.39e-04] 
Epoch 40 TRAIN:  23%|██▎       | 31/137 [00:08<00:20,  5.28it/s, epoch=40, grad_norm=98.415, loss=364.1661, lr=4.39e-04]
Epoch 40 TRAIN:  23%|██▎       | 31/137 [00:08<00:20,  5.28it/s, epoch=40, grad_norm=79.413, loss=334.0392, lr=4.39e-04]
Epoch 40 TRAIN:  25%|██▍       | 34/137 [00:08<00:20,  5.04it/s, epoch=40, grad_norm=79.413, loss=334.0392, lr=4.39e-04]
Epoch 40 TRAIN:  25%|██▍       | 34/137 [00:09<00:20,  5.04it/s, epoch=40, grad_norm=126.593, loss=260.6494, lr=4.39e-04]
Epoch 40 TRAIN:  25%|██▍       | 34/137 [00:09<00:20,  5.04it/s, epoch=40, grad_norm=128.895, loss=259.6515, lr=4.39e-04]
Epoch 40 TRAIN:  25%|██▍       | 34/137 [00:09<00:20,  5.04it/s, epoch=40, grad_norm=77.188, loss=285.4948, lr=4.39e-04] 
Epoch 40 TRAIN:  25%|██▍       | 34/137 [00:09<00:20,  5.04it/s, epoch=40, grad_norm=104.744, loss=306.3740, lr=4.39e-04]
Epoch 40 TRAIN:  28%|██▊       | 38/137 [00:09<00:18,  5.25it/s, epoch=40, grad_norm=104.744, loss=306.3740, lr=4.39e-04]
Epoch 40 TRAIN:  28%|██▊       | 38/137 [00:09<00:18,  5.25it/s, epoch=40, grad_norm=166.340, loss=354.2422, lr=4.39e-04]
Epoch 40 TRAIN:  28%|██▊       | 38/137 [00:10<00:18,  5.25it/s, epoch=40, grad_norm=138.527, loss=346.5381, lr=4.39e-04]
Epoch 40 TRAIN:  28%|██▊       | 38/137 [00:10<00:18,  5.25it/s, epoch=40, grad_norm=85.975, loss=299.7686, lr=4.40e-04] 
Epoch 40 TRAIN:  30%|██▉       | 41/137 [00:10<00:18,  5.07it/s, epoch=40, grad_norm=85.975, loss=299.7686, lr=4.40e-04]
Epoch 40 TRAIN:  30%|██▉       | 41/137 [00:10<00:18,  5.07it/s, epoch=40, grad_norm=111.767, loss=287.4212, lr=4.40e-04]
Epoch 40 TRAIN:  30%|██▉       | 41/137 [00:10<00:18,  5.07it/s, epoch=40, grad_norm=116.786, loss=289.2205, lr=4.40e-04]
Epoch 40 TRAIN:  30%|██▉       | 41/137 [00:11<00:18,  5.07it/s, epoch=40, grad_norm=79.806, loss=359.5076, lr=4.40e-04] 
Epoch 40 TRAIN:  32%|███▏      | 44/137 [00:11<00:19,  4.80it/s, epoch=40, grad_norm=79.806, loss=359.5076, lr=4.40e-04]
Epoch 40 TRAIN:  32%|███▏      | 44/137 [00:11<00:19,  4.80it/s, epoch=40, grad_norm=104.621, loss=315.8326, lr=4.40e-04]
Epoch 40 TRAIN:  32%|███▏      | 44/137 [00:11<00:19,  4.80it/s, epoch=40, grad_norm=83.747, loss=337.2108, lr=4.40e-04] 
Epoch 40 TRAIN:  32%|███▏      | 44/137 [00:11<00:19,  4.80it/s, epoch=40, grad_norm=111.779, loss=267.9406, lr=4.40e-04]
Epoch 40 TRAIN:  34%|███▍      | 47/137 [00:11<00:17,  5.00it/s, epoch=40, grad_norm=111.779, loss=267.9406, lr=4.40e-04]
Epoch 40 TRAIN:  34%|███▍      | 47/137 [00:11<00:17,  5.00it/s, epoch=40, grad_norm=70.822, loss=280.4176, lr=4.40e-04] 
Epoch 40 TRAIN:  34%|███▍      | 47/137 [00:11<00:17,  5.00it/s, epoch=40, grad_norm=60.343, loss=298.2211, lr=4.40e-04]
Epoch 40 TRAIN:  34%|███▍      | 47/137 [00:12<00:17,  5.00it/s, epoch=40, grad_norm=73.730, loss=293.1602, lr=4.40e-04]
Epoch 40 TRAIN:  36%|███▋      | 50/137 [00:12<00:17,  5.04it/s, epoch=40, grad_norm=73.730, loss=293.1602, lr=4.40e-04]
Epoch 40 TRAIN:  36%|███▋      | 50/137 [00:12<00:17,  5.04it/s, epoch=40, grad_norm=122.261, loss=338.4606, lr=4.40e-04]
Epoch 40 TRAIN:  36%|███▋      | 50/137 [00:12<00:17,  5.04it/s, epoch=40, grad_norm=73.488, loss=326.8110, lr=4.40e-04] 
Epoch 40 TRAIN:  36%|███▋      | 50/137 [00:12<00:17,  5.04it/s, epoch=40, grad_norm=76.197, loss=335.9703, lr=4.40e-04]
Epoch 40 TRAIN:  39%|███▊      | 53/137 [00:12<00:16,  5.02it/s, epoch=40, grad_norm=76.197, loss=335.9703, lr=4.40e-04]
Epoch 40 TRAIN:  39%|███▊      | 53/137 [00:12<00:16,  5.02it/s, epoch=40, grad_norm=60.821, loss=300.0458, lr=4.41e-04]
Epoch 40 TRAIN:  39%|███▊      | 53/137 [00:13<00:16,  5.02it/s, epoch=40, grad_norm=113.502, loss=353.6761, lr=4.41e-04]
Epoch 40 TRAIN:  39%|███▊      | 53/137 [00:13<00:16,  5.02it/s, epoch=40, grad_norm=105.582, loss=295.8846, lr=4.41e-04]
Epoch 40 TRAIN:  41%|████      | 56/137 [00:13<00:16,  5.03it/s, epoch=40, grad_norm=105.582, loss=295.8846, lr=4.41e-04]
Epoch 40 TRAIN:  41%|████      | 56/137 [00:13<00:16,  5.03it/s, epoch=40, grad_norm=103.211, loss=291.1075, lr=4.41e-04]
Epoch 40 TRAIN:  41%|████      | 56/137 [00:13<00:16,  5.03it/s, epoch=40, grad_norm=64.214, loss=326.8246, lr=4.41e-04] 
Epoch 40 TRAIN:  41%|████      | 56/137 [00:13<00:16,  5.03it/s, epoch=40, grad_norm=67.328, loss=300.9589, lr=4.41e-04]
Epoch 40 TRAIN:  43%|████▎     | 59/137 [00:13<00:15,  4.96it/s, epoch=40, grad_norm=67.328, loss=300.9589, lr=4.41e-04]
Epoch 40 TRAIN:  43%|████▎     | 59/137 [00:14<00:15,  4.96it/s, epoch=40, grad_norm=99.243, loss=287.5549, lr=4.41e-04]
Epoch 40 TRAIN:  43%|████▎     | 59/137 [00:14<00:15,  4.96it/s, epoch=40, grad_norm=133.545, loss=253.3531, lr=4.41e-04]
Epoch 40 TRAIN:  43%|████▎     | 59/137 [00:14<00:15,  4.96it/s, epoch=40, grad_norm=70.407, loss=285.8286, lr=4.41e-04] 
Epoch 40 TRAIN:  45%|████▌     | 62/137 [00:14<00:15,  4.99it/s, epoch=40, grad_norm=70.407, loss=285.8286, lr=4.41e-04]
Epoch 40 TRAIN:  45%|████▌     | 62/137 [00:14<00:15,  4.99it/s, epoch=40, grad_norm=131.208, loss=315.3968, lr=4.41e-04]
Epoch 40 TRAIN:  45%|████▌     | 62/137 [00:14<00:15,  4.99it/s, epoch=40, grad_norm=64.511, loss=278.0825, lr=4.41e-04] 
Epoch 40 TRAIN:  45%|████▌     | 62/137 [00:15<00:15,  4.99it/s, epoch=40, grad_norm=76.670, loss=296.7117, lr=4.41e-04]
Epoch 40 TRAIN:  47%|████▋     | 65/137 [00:15<00:14,  4.97it/s, epoch=40, grad_norm=76.670, loss=296.7117, lr=4.41e-04]
Epoch 40 TRAIN:  47%|████▋     | 65/137 [00:15<00:14,  4.97it/s, epoch=40, grad_norm=97.675, loss=329.6283, lr=4.42e-04]
Epoch 40 TRAIN:  47%|████▋     | 65/137 [00:15<00:14,  4.97it/s, epoch=40, grad_norm=132.125, loss=363.7719, lr=4.42e-04]
Epoch 40 TRAIN:  47%|████▋     | 65/137 [00:15<00:14,  4.97it/s, epoch=40, grad_norm=157.786, loss=289.1672, lr=4.42e-04]
Epoch 40 TRAIN:  50%|████▉     | 68/137 [00:15<00:13,  5.01it/s, epoch=40, grad_norm=157.786, loss=289.1672, lr=4.42e-04]
Epoch 40 TRAIN:  50%|████▉     | 68/137 [00:15<00:13,  5.01it/s, epoch=40, grad_norm=119.386, loss=325.5263, lr=4.42e-04]
Epoch 40 TRAIN:  50%|████▉     | 68/137 [00:16<00:13,  5.01it/s, epoch=40, grad_norm=102.282, loss=300.8644, lr=4.42e-04]
Epoch 40 TRAIN:  50%|████▉     | 68/137 [00:16<00:13,  5.01it/s, epoch=40, grad_norm=81.199, loss=288.6283, lr=4.42e-04] 
Epoch 40 TRAIN:  52%|█████▏    | 71/137 [00:16<00:13,  4.93it/s, epoch=40, grad_norm=81.199, loss=288.6283, lr=4.42e-04]
Epoch 40 TRAIN:  52%|█████▏    | 71/137 [00:16<00:13,  4.93it/s, epoch=40, grad_norm=105.723, loss=353.3317, lr=4.42e-04]
Epoch 40 TRAIN:  52%|█████▏    | 71/137 [00:16<00:13,  4.93it/s, epoch=40, grad_norm=95.229, loss=364.8000, lr=4.42e-04] 
Epoch 40 TRAIN:  52%|█████▏    | 71/137 [00:17<00:13,  4.93it/s, epoch=40, grad_norm=126.451, loss=262.5206, lr=4.42e-04]
Epoch 40 TRAIN:  54%|█████▍    | 74/137 [00:17<00:13,  4.79it/s, epoch=40, grad_norm=126.451, loss=262.5206, lr=4.42e-04]
Epoch 40 TRAIN:  54%|█████▍    | 74/137 [00:17<00:13,  4.79it/s, epoch=40, grad_norm=168.496, loss=248.8598, lr=4.42e-04]
Epoch 40 TRAIN:  54%|█████▍    | 74/137 [00:17<00:13,  4.79it/s, epoch=40, grad_norm=108.370, loss=326.4885, lr=4.42e-04]
Epoch 40 TRAIN:  54%|█████▍    | 74/137 [00:17<00:13,  4.79it/s, epoch=40, grad_norm=101.563, loss=334.9460, lr=4.42e-04]
Epoch 40 TRAIN:  56%|█████▌    | 77/137 [00:17<00:12,  4.74it/s, epoch=40, grad_norm=101.563, loss=334.9460, lr=4.42e-04]
Epoch 40 TRAIN:  56%|█████▌    | 77/137 [00:17<00:12,  4.74it/s, epoch=40, grad_norm=78.039, loss=279.7889, lr=4.42e-04] 
Epoch 40 TRAIN:  56%|█████▌    | 77/137 [00:18<00:12,  4.74it/s, epoch=40, grad_norm=210.660, loss=332.8581, lr=4.43e-04]
Epoch 40 TRAIN:  56%|█████▌    | 77/137 [00:18<00:12,  4.74it/s, epoch=40, grad_norm=97.284, loss=278.1886, lr=4.43e-04] 
Epoch 40 TRAIN:  58%|█████▊    | 80/137 [00:18<00:11,  4.87it/s, epoch=40, grad_norm=97.284, loss=278.1886, lr=4.43e-04]
Epoch 40 TRAIN:  58%|█████▊    | 80/137 [00:18<00:11,  4.87it/s, epoch=40, grad_norm=151.816, loss=221.6710, lr=4.43e-04]
Epoch 40 TRAIN:  58%|█████▊    | 80/137 [00:18<00:11,  4.87it/s, epoch=40, grad_norm=156.647, loss=333.7344, lr=4.43e-04]
Epoch 40 TRAIN:  58%|█████▊    | 80/137 [00:18<00:11,  4.87it/s, epoch=40, grad_norm=93.404, loss=283.1900, lr=4.43e-04] 
Epoch 40 TRAIN:  61%|██████    | 83/137 [00:18<00:11,  4.86it/s, epoch=40, grad_norm=93.404, loss=283.1900, lr=4.43e-04]
Epoch 40 TRAIN:  61%|██████    | 83/137 [00:19<00:11,  4.86it/s, epoch=40, grad_norm=91.817, loss=298.0982, lr=4.43e-04]
Epoch 40 TRAIN:  61%|██████    | 83/137 [00:19<00:11,  4.86it/s, epoch=40, grad_norm=107.633, loss=307.6198, lr=4.43e-04]
Epoch 40 TRAIN:  61%|██████    | 83/137 [00:19<00:11,  4.86it/s, epoch=40, grad_norm=88.831, loss=270.5813, lr=4.43e-04] 
Epoch 40 TRAIN:  63%|██████▎   | 86/137 [00:19<00:10,  4.93it/s, epoch=40, grad_norm=88.831, loss=270.5813, lr=4.43e-04]
Epoch 40 TRAIN:  63%|██████▎   | 86/137 [00:19<00:10,  4.93it/s, epoch=40, grad_norm=149.453, loss=337.3491, lr=4.43e-04]
Epoch 40 TRAIN:  63%|██████▎   | 86/137 [00:19<00:10,  4.93it/s, epoch=40, grad_norm=73.820, loss=316.4635, lr=4.43e-04] 
Epoch 40 TRAIN:  63%|██████▎   | 86/137 [00:20<00:10,  4.93it/s, epoch=40, grad_norm=111.815, loss=300.0459, lr=4.43e-04]
Epoch 40 TRAIN:  65%|██████▍   | 89/137 [00:20<00:10,  4.73it/s, epoch=40, grad_norm=111.815, loss=300.0459, lr=4.43e-04]
Epoch 40 TRAIN:  65%|██████▍   | 89/137 [00:20<00:10,  4.73it/s, epoch=40, grad_norm=112.799, loss=305.7298, lr=4.43e-04]
Epoch 40 TRAIN:  65%|██████▍   | 89/137 [00:20<00:10,  4.73it/s, epoch=40, grad_norm=170.078, loss=275.0839, lr=4.44e-04]
Epoch 40 TRAIN:  65%|██████▍   | 89/137 [00:20<00:10,  4.73it/s, epoch=40, grad_norm=249.255, loss=214.9794, lr=4.44e-04]
Epoch 40 TRAIN:  67%|██████▋   | 92/137 [00:20<00:09,  4.90it/s, epoch=40, grad_norm=249.255, loss=214.9794, lr=4.44e-04]
Epoch 40 TRAIN:  67%|██████▋   | 92/137 [00:20<00:09,  4.90it/s, epoch=40, grad_norm=97.715, loss=332.9111, lr=4.44e-04] 
Epoch 40 TRAIN:  67%|██████▋   | 92/137 [00:21<00:09,  4.90it/s, epoch=40, grad_norm=178.221, loss=338.0275, lr=4.44e-04]
Epoch 40 TRAIN:  67%|██████▋   | 92/137 [00:21<00:09,  4.90it/s, epoch=40, grad_norm=149.085, loss=295.7110, lr=4.44e-04]
Epoch 40 TRAIN:  69%|██████▉   | 95/137 [00:21<00:08,  4.88it/s, epoch=40, grad_norm=149.085, loss=295.7110, lr=4.44e-04]
Epoch 40 TRAIN:  69%|██████▉   | 95/137 [00:21<00:08,  4.88it/s, epoch=40, grad_norm=158.323, loss=198.0740, lr=4.44e-04]
Epoch 40 TRAIN:  69%|██████▉   | 95/137 [00:21<00:08,  4.88it/s, epoch=40, grad_norm=126.620, loss=264.9373, lr=4.44e-04]
Epoch 40 TRAIN:  69%|██████▉   | 95/137 [00:22<00:08,  4.88it/s, epoch=40, grad_norm=103.827, loss=294.2491, lr=4.44e-04]
Epoch 40 TRAIN:  72%|███████▏  | 98/137 [00:22<00:08,  4.75it/s, epoch=40, grad_norm=103.827, loss=294.2491, lr=4.44e-04]
Epoch 40 TRAIN:  72%|███████▏  | 98/137 [00:22<00:08,  4.75it/s, epoch=40, grad_norm=170.917, loss=357.7075, lr=4.44e-04]
Epoch 40 TRAIN:  72%|███████▏  | 98/137 [00:22<00:08,  4.75it/s, epoch=40, grad_norm=165.022, loss=219.1056, lr=4.44e-04]2026-04-16 20:12:19,936 DEBUG TRAIN | steps/sec 4.446| Batch 40/100 loss 219.105560 loss_att 188.929489 loss_ctc 289.516418 th_accuracy 0.321429 lr 4.4424e-04 grad_norm 165.022003 rank 0

Epoch 40 TRAIN:  72%|███████▏  | 98/137 [00:22<00:08,  4.75it/s, epoch=40, grad_norm=92.874, loss=344.0857, lr=4.44e-04] 
Epoch 40 TRAIN:  74%|███████▎  | 101/137 [00:22<00:07,  4.86it/s, epoch=40, grad_norm=92.874, loss=344.0857, lr=4.44e-04]
Epoch 40 TRAIN:  74%|███████▎  | 101/137 [00:22<00:07,  4.86it/s, epoch=40, grad_norm=158.787, loss=338.0585, lr=4.44e-04]
Epoch 40 TRAIN:  74%|███████▎  | 101/137 [00:23<00:07,  4.86it/s, epoch=40, grad_norm=119.234, loss=332.7013, lr=4.44e-04]
Epoch 40 TRAIN:  74%|███████▎  | 101/137 [00:23<00:07,  4.86it/s, epoch=40, grad_norm=132.411, loss=378.4261, lr=4.45e-04]
Epoch 40 TRAIN:  76%|███████▌  | 104/137 [00:23<00:06,  4.85it/s, epoch=40, grad_norm=132.411, loss=378.4261, lr=4.45e-04]
Epoch 40 TRAIN:  76%|███████▌  | 104/137 [00:23<00:06,  4.85it/s, epoch=40, grad_norm=215.485, loss=314.0792, lr=4.45e-04]
Epoch 40 TRAIN:  76%|███████▌  | 104/137 [00:23<00:06,  4.85it/s, epoch=40, grad_norm=277.996, loss=252.1867, lr=4.45e-04]
Epoch 40 TRAIN:  76%|███████▌  | 104/137 [00:23<00:06,  4.85it/s, epoch=40, grad_norm=159.333, loss=298.6719, lr=4.45e-04]
Epoch 40 TRAIN:  78%|███████▊  | 107/137 [00:23<00:06,  4.93it/s, epoch=40, grad_norm=159.333, loss=298.6719, lr=4.45e-04]
Epoch 40 TRAIN:  78%|███████▊  | 107/137 [00:24<00:06,  4.93it/s, epoch=40, grad_norm=206.629, loss=285.7835, lr=4.45e-04]
Epoch 40 TRAIN:  78%|███████▊  | 107/137 [00:24<00:06,  4.93it/s, epoch=40, grad_norm=153.146, loss=278.8893, lr=4.45e-04]
Epoch 40 TRAIN:  78%|███████▊  | 107/137 [00:24<00:06,  4.93it/s, epoch=40, grad_norm=134.482, loss=323.6302, lr=4.45e-04]
Epoch 40 TRAIN:  80%|████████  | 110/137 [00:24<00:05,  4.70it/s, epoch=40, grad_norm=134.482, loss=323.6302, lr=4.45e-04]
Epoch 40 TRAIN:  80%|████████  | 110/137 [00:24<00:05,  4.70it/s, epoch=40, grad_norm=123.583, loss=304.4763, lr=4.45e-04]
Epoch 40 TRAIN:  80%|████████  | 110/137 [00:24<00:05,  4.70it/s, epoch=40, grad_norm=89.552, loss=259.1556, lr=4.45e-04] 
Epoch 40 TRAIN:  80%|████████  | 110/137 [00:25<00:05,  4.70it/s, epoch=40, grad_norm=126.665, loss=376.8293, lr=4.45e-04]
Epoch 40 TRAIN:  82%|████████▏ | 113/137 [00:25<00:05,  4.73it/s, epoch=40, grad_norm=126.665, loss=376.8293, lr=4.45e-04]
Epoch 40 TRAIN:  82%|████████▏ | 113/137 [00:25<00:05,  4.73it/s, epoch=40, grad_norm=104.802, loss=314.0091, lr=4.45e-04]
Epoch 40 TRAIN:  82%|████████▏ | 113/137 [00:25<00:05,  4.73it/s, epoch=40, grad_norm=105.126, loss=290.5682, lr=4.45e-04]
Epoch 40 TRAIN:  82%|████████▏ | 113/137 [00:25<00:05,  4.73it/s, epoch=40, grad_norm=118.088, loss=315.6548, lr=4.46e-04]
Epoch 40 TRAIN:  85%|████████▍ | 116/137 [00:25<00:04,  4.73it/s, epoch=40, grad_norm=118.088, loss=315.6548, lr=4.46e-04]
Epoch 40 TRAIN:  85%|████████▍ | 116/137 [00:25<00:04,  4.73it/s, epoch=40, grad_norm=102.387, loss=301.7762, lr=4.46e-04]
Epoch 40 TRAIN:  85%|████████▍ | 116/137 [00:26<00:04,  4.73it/s, epoch=40, grad_norm=88.641, loss=231.8560, lr=4.46e-04] 
Epoch 40 TRAIN:  85%|████████▍ | 116/137 [00:26<00:04,  4.73it/s, epoch=40, grad_norm=79.811, loss=256.2897, lr=4.46e-04]
Epoch 40 TRAIN:  87%|████████▋ | 119/137 [00:26<00:03,  4.91it/s, epoch=40, grad_norm=79.811, loss=256.2897, lr=4.46e-04]
Epoch 40 TRAIN:  87%|████████▋ | 119/137 [00:26<00:03,  4.91it/s, epoch=40, grad_norm=107.573, loss=267.9290, lr=4.46e-04]
Epoch 40 TRAIN:  87%|████████▋ | 119/137 [00:26<00:03,  4.91it/s, epoch=40, grad_norm=130.763, loss=293.0578, lr=4.46e-04]
Epoch 40 TRAIN:  87%|████████▋ | 119/137 [00:26<00:03,  4.91it/s, epoch=40, grad_norm=291.676, loss=418.4777, lr=4.46e-04]
Epoch 40 TRAIN:  89%|████████▉ | 122/137 [00:26<00:02,  5.04it/s, epoch=40, grad_norm=291.676, loss=418.4777, lr=4.46e-04]
Epoch 40 TRAIN:  89%|████████▉ | 122/137 [00:27<00:02,  5.04it/s, epoch=40, grad_norm=135.059, loss=314.7576, lr=4.46e-04]
Epoch 40 TRAIN:  89%|████████▉ | 122/137 [00:27<00:02,  5.04it/s, epoch=40, grad_norm=114.336, loss=374.9544, lr=4.46e-04]
Epoch 40 TRAIN:  89%|████████▉ | 122/137 [00:27<00:02,  5.04it/s, epoch=40, grad_norm=130.130, loss=364.8683, lr=4.46e-04]
Epoch 40 TRAIN:  91%|█████████ | 125/137 [00:27<00:02,  4.69it/s, epoch=40, grad_norm=130.130, loss=364.8683, lr=4.46e-04]
Epoch 40 TRAIN:  91%|█████████ | 125/137 [00:27<00:02,  4.69it/s, epoch=40, grad_norm=153.954, loss=293.5822, lr=4.46e-04]
Epoch 40 TRAIN:  91%|█████████ | 125/137 [00:28<00:02,  4.69it/s, epoch=40, grad_norm=141.425, loss=374.1190, lr=4.46e-04]
Epoch 40 TRAIN:  91%|█████████ | 125/137 [00:28<00:02,  4.69it/s, epoch=40, grad_norm=165.681, loss=364.1240, lr=4.46e-04]
Epoch 40 TRAIN:  93%|█████████▎| 128/137 [00:28<00:01,  4.78it/s, epoch=40, grad_norm=165.681, loss=364.1240, lr=4.46e-04]
Epoch 40 TRAIN:  93%|█████████▎| 128/137 [00:28<00:01,  4.78it/s, epoch=40, grad_norm=105.584, loss=317.9449, lr=4.47e-04]
Epoch 40 TRAIN:  93%|█████████▎| 128/137 [00:28<00:01,  4.78it/s, epoch=40, grad_norm=143.547, loss=289.3835, lr=4.47e-04]
Epoch 40 TRAIN:  93%|█████████▎| 128/137 [00:28<00:01,  4.78it/s, epoch=40, grad_norm=250.851, loss=399.4247, lr=4.47e-04]
Epoch 40 TRAIN:  96%|█████████▌| 131/137 [00:28<00:01,  4.68it/s, epoch=40, grad_norm=250.851, loss=399.4247, lr=4.47e-04]
Epoch 40 TRAIN:  96%|█████████▌| 131/137 [00:29<00:01,  4.68it/s, epoch=40, grad_norm=127.333, loss=297.8137, lr=4.47e-04]
Epoch 40 TRAIN:  96%|█████████▌| 131/137 [00:29<00:01,  4.68it/s, epoch=40, grad_norm=181.283, loss=291.5734, lr=4.47e-04]
Epoch 40 TRAIN:  96%|█████████▌| 131/137 [00:29<00:01,  4.68it/s, epoch=40, grad_norm=133.810, loss=283.5198, lr=4.47e-04]
Epoch 40 TRAIN:  96%|█████████▌| 131/137 [00:29<00:01,  4.68it/s, epoch=40, grad_norm=133.668, loss=334.6336, lr=4.47e-04]
Epoch 40 TRAIN:  99%|█████████▊| 135/137 [00:29<00:00,  5.35it/s, epoch=40, grad_norm=133.668, loss=334.6336, lr=4.47e-04]
Epoch 40 TRAIN:  99%|█████████▊| 135/137 [00:29<00:00,  5.35it/s, epoch=40, grad_norm=143.631, loss=324.0039, lr=4.47e-04]
Epoch 40 TRAIN:  99%|█████████▊| 135/137 [00:29<00:00,  5.35it/s, epoch=40, grad_norm=136.372, loss=389.0821, lr=4.47e-04]
Epoch 40 TRAIN: 100%|██████████| 137/137 [00:29<00:00,  4.59it/s, epoch=40, grad_norm=136.372, loss=389.0821, lr=4.47e-04]

CV:   0%|          | 0/6 [00:00<?, ?it/s]/home/rookie/Documents/batuka/projects/wenet/wenet/wenet/utils/train_utils.py:689: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  amp_autocast(enabled=dtype is not None,

CV:  17%|█▋        | 1/6 [00:00<00:03,  1.32it/s]
                                                 
2026-04-16 20:12:28,249 INFO Epoch 40 Step 5590 CV info lr 4.4720e-04 cv_loss 490.0849526145241 rank 0 acc 0.27638283868630725
2026-04-16 20:12:28,250 INFO [Rank 0] Checkpoint: save to checkpoint exp/conformer_mongolian/epoch_40.pt
2026-04-16 20:12:28,360 INFO Epoch 41 Step 5590 TRAIN info lr 4.4720e-04 rank 0
2026-04-16 20:12:28,372 INFO using accumulate grad, new batch size is 1 times larger than before

Epoch 41 TRAIN:   0%|          | 0/137 [00:00<?, ?it/s]
Epoch 41 TRAIN:   0%|          | 0/137 [00:04<?, ?it/s, epoch=41, grad_norm=632.315, loss=522.4805, lr=4.47e-04]
Epoch 41 TRAIN:   1%|          | 1/137 [00:04<09:13,  4.07s/it, epoch=41, grad_norm=632.315, loss=522.4805, lr=4.47e-04]
Epoch 41 TRAIN:   1%|          | 1/137 [00:04<09:13,  4.07s/it, epoch=41, grad_norm=230.330, loss=265.4319, lr=4.47e-04]
Epoch 41 TRAIN:   1%|          | 1/137 [00:04<09:13,  4.07s/it, epoch=41, grad_norm=159.454, loss=259.5939, lr=4.47e-04]
Epoch 41 TRAIN:   1%|          | 1/137 [00:04<09:13,  4.07s/it, epoch=41, grad_norm=299.669, loss=340.4642, lr=4.48e-04]
Epoch 41 TRAIN:   1%|          | 1/137 [00:04<09:13,  4.07s/it, epoch=41, grad_norm=225.487, loss=320.1329, lr=4.48e-04]
Epoch 41 TRAIN:   1%|          | 1/137 [00:04<09:13,  4.07s/it, epoch=41, grad_norm=405.405, loss=447.8265, lr=4.48e-04]
Epoch 41 TRAIN:   1%|          | 1/137 [00:04<09:13,  4.07s/it, epoch=41, grad_norm=309.128, loss=417.3270, lr=4.48e-04]
Epoch 41 TRAIN:   1%|          | 1/137 [00:04<09:13,  4.07s/it, epoch=41, grad_norm=148.793, loss=329.4763, lr=4.48e-04]
Epoch 41 TRAIN:   1%|          | 1/137 [00:04<09:13,  4.07s/it, epoch=41, grad_norm=75.834, loss=276.1324, lr=4.48e-04] 
Epoch 41 TRAIN:   7%|▋         | 9/137 [00:04<00:51,  2.51it/s, epoch=41, grad_norm=75.834, loss=276.1324, lr=4.48e-04]
Epoch 41 TRAIN:   7%|▋         | 9/137 [00:04<00:51,  2.51it/s, epoch=41, grad_norm=72.333, loss=314.1948, lr=4.48e-04]
Epoch 41 TRAIN:   7%|▋         | 9/137 [00:05<00:51,  2.51it/s, epoch=41, grad_norm=90.730, loss=340.9979, lr=4.48e-04]
Epoch 41 TRAIN:   7%|▋         | 9/137 [00:05<00:51,  2.51it/s, epoch=41, grad_norm=77.543, loss=359.3017, lr=4.48e-04]
Epoch 41 TRAIN:   7%|▋         | 9/137 [00:05<00:51,  2.51it/s, epoch=41, grad_norm=85.191, loss=314.4594, lr=4.48e-04]
Epoch 41 TRAIN:   9%|▉         | 13/137 [00:05<00:37,  3.28it/s, epoch=41, grad_norm=85.191, loss=314.4594, lr=4.48e-04]
Epoch 41 TRAIN:   9%|▉         | 13/137 [00:05<00:37,  3.28it/s, epoch=41, grad_norm=177.239, loss=266.4090, lr=4.48e-04]
Epoch 41 TRAIN:   9%|▉         | 13/137 [00:05<00:37,  3.28it/s, epoch=41, grad_norm=83.908, loss=350.3573, lr=4.48e-04] 
Epoch 41 TRAIN:   9%|▉         | 13/137 [00:05<00:37,  3.28it/s, epoch=41, grad_norm=91.409, loss=310.7194, lr=4.48e-04]
Epoch 41 TRAIN:   9%|▉         | 13/137 [00:05<00:37,  3.28it/s, epoch=41, grad_norm=101.990, loss=292.4951, lr=4.49e-04]
Epoch 41 TRAIN:  12%|█▏        | 17/137 [00:05<00:30,  3.98it/s, epoch=41, grad_norm=101.990, loss=292.4951, lr=4.49e-04]
Epoch 41 TRAIN:  12%|█▏        | 17/137 [00:06<00:30,  3.98it/s, epoch=41, grad_norm=119.395, loss=264.0779, lr=4.49e-04]
Epoch 41 TRAIN:  12%|█▏        | 17/137 [00:06<00:30,  3.98it/s, epoch=41, grad_norm=91.126, loss=304.2053, lr=4.49e-04] 
Epoch 41 TRAIN:  12%|█▏        | 17/137 [00:06<00:30,  3.98it/s, epoch=41, grad_norm=130.385, loss=302.9846, lr=4.49e-04]
Epoch 41 TRAIN:  12%|█▏        | 17/137 [00:06<00:30,  3.98it/s, epoch=41, grad_norm=168.981, loss=319.2290, lr=4.49e-04]
Epoch 41 TRAIN:  15%|█▌        | 21/137 [00:06<00:25,  4.52it/s, epoch=41, grad_norm=168.981, loss=319.2290, lr=4.49e-04]
Epoch 41 TRAIN:  15%|█▌        | 21/137 [00:06<00:25,  4.52it/s, epoch=41, grad_norm=115.796, loss=308.9527, lr=4.49e-04]
Epoch 41 TRAIN:  15%|█▌        | 21/137 [00:07<00:25,  4.52it/s, epoch=41, grad_norm=54.374, loss=270.9987, lr=4.49e-04] 
Epoch 41 TRAIN:  15%|█▌        | 21/137 [00:07<00:25,  4.52it/s, epoch=41, grad_norm=80.237, loss=318.8092, lr=4.49e-04]
Epoch 41 TRAIN:  15%|█▌        | 21/137 [00:07<00:25,  4.52it/s, epoch=41, grad_norm=72.404, loss=303.9746, lr=4.49e-04]
Epoch 41 TRAIN:  18%|█▊        | 25/137 [00:07<00:24,  4.52it/s, epoch=41, grad_norm=72.404, loss=303.9746, lr=4.49e-04]
Epoch 41 TRAIN:  18%|█▊        | 25/137 [00:07<00:24,  4.52it/s, epoch=41, grad_norm=103.032, loss=294.6889, lr=4.49e-04]
Epoch 41 TRAIN:  18%|█▊        | 25/137 [00:07<00:24,  4.52it/s, epoch=41, grad_norm=108.625, loss=369.3060, lr=4.49e-04]
Epoch 41 TRAIN:  18%|█▊        | 25/137 [00:08<00:24,  4.52it/s, epoch=41, grad_norm=89.520, loss=345.1388, lr=4.49e-04] 
Epoch 41 TRAIN:  20%|██        | 28/137 [00:08<00:23,  4.72it/s, epoch=41, grad_norm=89.520, loss=345.1388, lr=4.49e-04]
Epoch 41 TRAIN:  20%|██        | 28/137 [00:08<00:23,  4.72it/s, epoch=41, grad_norm=74.789, loss=357.1264, lr=4.50e-04]
Epoch 41 TRAIN:  20%|██        | 28/137 [00:08<00:23,  4.72it/s, epoch=41, grad_norm=191.817, loss=272.2819, lr=4.50e-04]
Epoch 41 TRAIN:  20%|██        | 28/137 [00:08<00:23,  4.72it/s, epoch=41, grad_norm=196.837, loss=253.9738, lr=4.50e-04]
Epoch 41 TRAIN:  20%|██        | 28/137 [00:08<00:23,  4.72it/s, epoch=41, grad_norm=85.669, loss=350.1983, lr=4.50e-04] 
Epoch 41 TRAIN:  23%|██▎       | 32/137 [00:08<00:20,  5.09it/s, epoch=41, grad_norm=85.669, loss=350.1983, lr=4.50e-04]
Epoch 41 TRAIN:  23%|██▎       | 32/137 [00:08<00:20,  5.09it/s, epoch=41, grad_norm=179.317, loss=265.8278, lr=4.50e-04]
Epoch 41 TRAIN:  23%|██▎       | 32/137 [00:09<00:20,  5.09it/s, epoch=41, grad_norm=94.622, loss=311.1911, lr=4.50e-04] 
Epoch 41 TRAIN:  23%|██▎       | 32/137 [00:09<00:20,  5.09it/s, epoch=41, grad_norm=72.521, loss=325.9351, lr=4.50e-04]
Epoch 41 TRAIN:  26%|██▌       | 35/137 [00:09<00:20,  5.05it/s, epoch=41, grad_norm=72.521, loss=325.9351, lr=4.50e-04]
Epoch 41 TRAIN:  26%|██▌       | 35/137 [00:09<00:20,  5.05it/s, epoch=41, grad_norm=75.227, loss=312.1887, lr=4.50e-04]
Epoch 41 TRAIN:  26%|██▌       | 35/137 [00:09<00:20,  5.05it/s, epoch=41, grad_norm=140.391, loss=362.1343, lr=4.50e-04]
Epoch 41 TRAIN:  26%|██▌       | 35/137 [00:09<00:20,  5.05it/s, epoch=41, grad_norm=105.386, loss=334.4463, lr=4.50e-04]
Epoch 41 TRAIN:  28%|██▊       | 38/137 [00:09<00:19,  5.12it/s, epoch=41, grad_norm=105.386, loss=334.4463, lr=4.50e-04]
Epoch 41 TRAIN:  28%|██▊       | 38/137 [00:10<00:19,  5.12it/s, epoch=41, grad_norm=71.181, loss=324.4817, lr=4.50e-04] 
Epoch 41 TRAIN:  28%|██▊       | 38/137 [00:10<00:19,  5.12it/s, epoch=41, grad_norm=78.684, loss=264.7982, lr=4.50e-04]
Epoch 41 TRAIN:  28%|██▊       | 38/137 [00:10<00:19,  5.12it/s, epoch=41, grad_norm=80.628, loss=333.2877, lr=4.50e-04]
Epoch 41 TRAIN:  30%|██▉       | 41/137 [00:10<00:18,  5.07it/s, epoch=41, grad_norm=80.628, loss=333.2877, lr=4.50e-04]
Epoch 41 TRAIN:  30%|██▉       | 41/137 [00:10<00:18,  5.07it/s, epoch=41, grad_norm=110.520, loss=292.3828, lr=4.51e-04]
Epoch 41 TRAIN:  30%|██▉       | 41/137 [00:10<00:18,  5.07it/s, epoch=41, grad_norm=76.842, loss=316.3943, lr=4.51e-04] 
Epoch 41 TRAIN:  30%|██▉       | 41/137 [00:11<00:18,  5.07it/s, epoch=41, grad_norm=81.706, loss=308.7156, lr=4.51e-04]
Epoch 41 TRAIN:  32%|███▏      | 44/137 [00:11<00:18,  5.07it/s, epoch=41, grad_norm=81.706, loss=308.7156, lr=4.51e-04]
Epoch 41 TRAIN:  32%|███▏      | 44/137 [00:11<00:18,  5.07it/s, epoch=41, grad_norm=97.042, loss=337.8924, lr=4.51e-04]
Epoch 41 TRAIN:  32%|███▏      | 44/137 [00:11<00:18,  5.07it/s, epoch=41, grad_norm=68.143, loss=310.9583, lr=4.51e-04]
Epoch 41 TRAIN:  32%|███▏      | 44/137 [00:11<00:18,  5.07it/s, epoch=41, grad_norm=81.932, loss=339.0187, lr=4.51e-04]
Epoch 41 TRAIN:  34%|███▍      | 47/137 [00:11<00:17,  5.03it/s, epoch=41, grad_norm=81.932, loss=339.0187, lr=4.51e-04]
Epoch 41 TRAIN:  34%|███▍      | 47/137 [00:11<00:17,  5.03it/s, epoch=41, grad_norm=78.344, loss=288.4557, lr=4.51e-04]
Epoch 41 TRAIN:  34%|███▍      | 47/137 [00:12<00:17,  5.03it/s, epoch=41, grad_norm=92.700, loss=303.6518, lr=4.51e-04]
Epoch 41 TRAIN:  34%|███▍      | 47/137 [00:12<00:17,  5.03it/s, epoch=41, grad_norm=71.798, loss=328.6003, lr=4.51e-04]
Epoch 41 TRAIN:  36%|███▋      | 50/137 [00:12<00:17,  5.07it/s, epoch=41, grad_norm=71.798, loss=328.6003, lr=4.51e-04]
Epoch 41 TRAIN:  36%|███▋      | 50/137 [00:12<00:17,  5.07it/s, epoch=41, grad_norm=64.078, loss=295.6061, lr=4.51e-04]
Epoch 41 TRAIN:  36%|███▋      | 50/137 [00:12<00:17,  5.07it/s, epoch=41, grad_norm=109.628, loss=326.8118, lr=4.51e-04]
Epoch 41 TRAIN:  36%|███▋      | 50/137 [00:12<00:17,  5.07it/s, epoch=41, grad_norm=62.491, loss=263.8569, lr=4.51e-04] 
Epoch 41 TRAIN:  39%|███▊      | 53/137 [00:12<00:16,  5.00it/s, epoch=41, grad_norm=62.491, loss=263.8569, lr=4.51e-04]
Epoch 41 TRAIN:  39%|███▊      | 53/137 [00:13<00:16,  5.00it/s, epoch=41, grad_norm=94.943, loss=319.3593, lr=4.52e-04]
Epoch 41 TRAIN:  39%|███▊      | 53/137 [00:13<00:16,  5.00it/s, epoch=41, grad_norm=108.161, loss=342.3330, lr=4.52e-04]
Epoch 41 TRAIN:  39%|███▊      | 53/137 [00:13<00:16,  5.00it/s, epoch=41, grad_norm=115.405, loss=323.0977, lr=4.52e-04]
Epoch 41 TRAIN:  41%|████      | 56/137 [00:13<00:16,  4.95it/s, epoch=41, grad_norm=115.405, loss=323.0977, lr=4.52e-04]
Epoch 41 TRAIN:  41%|████      | 56/137 [00:13<00:16,  4.95it/s, epoch=41, grad_norm=147.495, loss=394.7721, lr=4.52e-04]
Epoch 41 TRAIN:  41%|████      | 56/137 [00:13<00:16,  4.95it/s, epoch=41, grad_norm=124.070, loss=306.5695, lr=4.52e-04]
Epoch 41 TRAIN:  41%|████      | 56/137 [00:14<00:16,  4.95it/s, epoch=41, grad_norm=75.384, loss=314.8299, lr=4.52e-04] 
Epoch 41 TRAIN:  43%|████▎     | 59/137 [00:14<00:16,  4.79it/s, epoch=41, grad_norm=75.384, loss=314.8299, lr=4.52e-04]
Epoch 41 TRAIN:  43%|████▎     | 59/137 [00:14<00:16,  4.79it/s, epoch=41, grad_norm=112.908, loss=345.2578, lr=4.52e-04]
Epoch 41 TRAIN:  43%|████▎     | 59/137 [00:14<00:16,  4.79it/s, epoch=41, grad_norm=134.836, loss=279.9811, lr=4.52e-04]
Epoch 41 TRAIN:  43%|████▎     | 59/137 [00:14<00:16,  4.79it/s, epoch=41, grad_norm=86.940, loss=342.9358, lr=4.52e-04] 
Epoch 41 TRAIN:  45%|████▌     | 62/137 [00:14<00:15,  4.82it/s, epoch=41, grad_norm=86.940, loss=342.9358, lr=4.52e-04]
Epoch 41 TRAIN:  45%|████▌     | 62/137 [00:15<00:15,  4.82it/s, epoch=41, grad_norm=120.704, loss=305.2896, lr=4.52e-04]
Epoch 41 TRAIN:  45%|████▌     | 62/137 [00:15<00:15,  4.82it/s, epoch=41, grad_norm=131.231, loss=257.9016, lr=4.52e-04]
Epoch 41 TRAIN:  45%|████▌     | 62/137 [00:15<00:15,  4.82it/s, epoch=41, grad_norm=133.629, loss=253.0167, lr=4.52e-04]
Epoch 41 TRAIN:  47%|████▋     | 65/137 [00:15<00:14,  4.86it/s, epoch=41, grad_norm=133.629, loss=253.0167, lr=4.52e-04]
Epoch 41 TRAIN:  47%|████▋     | 65/137 [00:15<00:14,  4.86it/s, epoch=41, grad_norm=102.379, loss=293.0149, lr=4.52e-04]
Epoch 41 TRAIN:  47%|████▋     | 65/137 [00:15<00:14,  4.86it/s, epoch=41, grad_norm=114.290, loss=331.7208, lr=4.53e-04]
Epoch 41 TRAIN:  47%|████▋     | 65/137 [00:15<00:14,  4.86it/s, epoch=41, grad_norm=136.059, loss=305.9221, lr=4.53e-04]
Epoch 41 TRAIN:  50%|████▉     | 68/137 [00:15<00:13,  4.94it/s, epoch=41, grad_norm=136.059, loss=305.9221, lr=4.53e-04]
Epoch 41 TRAIN:  50%|████▉     | 68/137 [00:16<00:13,  4.94it/s, epoch=41, grad_norm=86.393, loss=260.7214, lr=4.53e-04] 
Epoch 41 TRAIN:  50%|████▉     | 68/137 [00:16<00:13,  4.94it/s, epoch=41, grad_norm=96.346, loss=287.0434, lr=4.53e-04]
Epoch 41 TRAIN:  50%|████▉     | 68/137 [00:16<00:13,  4.94it/s, epoch=41, grad_norm=113.704, loss=335.5622, lr=4.53e-04]
Epoch 41 TRAIN:  52%|█████▏    | 71/137 [00:16<00:13,  4.96it/s, epoch=41, grad_norm=113.704, loss=335.5622, lr=4.53e-04]
Epoch 41 TRAIN:  52%|█████▏    | 71/137 [00:16<00:13,  4.96it/s, epoch=41, grad_norm=147.732, loss=346.1747, lr=4.53e-04]
Epoch 41 TRAIN:  52%|█████▏    | 71/137 [00:16<00:13,  4.96it/s, epoch=41, grad_norm=170.864, loss=390.8483, lr=4.53e-04]
Epoch 41 TRAIN:  52%|█████▏    | 71/137 [00:17<00:13,  4.96it/s, epoch=41, grad_norm=119.423, loss=321.1772, lr=4.53e-04]
Epoch 41 TRAIN:  54%|█████▍    | 74/137 [00:17<00:12,  5.01it/s, epoch=41, grad_norm=119.423, loss=321.1772, lr=4.53e-04]
Epoch 41 TRAIN:  54%|█████▍    | 74/137 [00:17<00:12,  5.01it/s, epoch=41, grad_norm=149.202, loss=281.3578, lr=4.53e-04]
Epoch 41 TRAIN:  54%|█████▍    | 74/137 [00:17<00:12,  5.01it/s, epoch=41, grad_norm=83.475, loss=332.4334, lr=4.53e-04] 
Epoch 41 TRAIN:  54%|█████▍    | 74/137 [00:17<00:12,  5.01it/s, epoch=41, grad_norm=174.219, loss=267.3403, lr=4.53e-04]
Epoch 41 TRAIN:  56%|█████▌    | 77/137 [00:17<00:12,  4.88it/s, epoch=41, grad_norm=174.219, loss=267.3403, lr=4.53e-04]
Epoch 41 TRAIN:  56%|█████▌    | 77/137 [00:18<00:12,  4.88it/s, epoch=41, grad_norm=97.400, loss=332.0659, lr=4.53e-04] 
Epoch 41 TRAIN:  56%|█████▌    | 77/137 [00:18<00:12,  4.88it/s, epoch=41, grad_norm=151.597, loss=315.7768, lr=4.54e-04]
Epoch 41 TRAIN:  56%|█████▌    | 77/137 [00:18<00:12,  4.88it/s, epoch=41, grad_norm=113.278, loss=297.0528, lr=4.54e-04]
Epoch 41 TRAIN:  58%|█████▊    | 80/137 [00:18<00:12,  4.69it/s, epoch=41, grad_norm=113.278, loss=297.0528, lr=4.54e-04]
Epoch 41 TRAIN:  58%|█████▊    | 80/137 [00:18<00:12,  4.69it/s, epoch=41, grad_norm=91.835, loss=293.5011, lr=4.54e-04] 
Epoch 41 TRAIN:  58%|█████▊    | 80/137 [00:18<00:12,  4.69it/s, epoch=41, grad_norm=105.227, loss=314.9443, lr=4.54e-04]
Epoch 41 TRAIN:  58%|█████▊    | 80/137 [00:19<00:12,  4.69it/s, epoch=41, grad_norm=171.842, loss=348.0317, lr=4.54e-04]
Epoch 41 TRAIN:  61%|██████    | 83/137 [00:19<00:11,  4.69it/s, epoch=41, grad_norm=171.842, loss=348.0317, lr=4.54e-04]
Epoch 41 TRAIN:  61%|██████    | 83/137 [00:19<00:11,  4.69it/s, epoch=41, grad_norm=182.314, loss=344.6801, lr=4.54e-04]
Epoch 41 TRAIN:  61%|██████    | 83/137 [00:19<00:11,  4.69it/s, epoch=41, grad_norm=97.762, loss=298.3423, lr=4.54e-04] 
Epoch 41 TRAIN:  61%|██████    | 83/137 [00:19<00:11,  4.69it/s, epoch=41, grad_norm=158.722, loss=323.1175, lr=4.54e-04]
Epoch 41 TRAIN:  63%|██████▎   | 86/137 [00:19<00:11,  4.63it/s, epoch=41, grad_norm=158.722, loss=323.1175, lr=4.54e-04]
Epoch 41 TRAIN:  63%|██████▎   | 86/137 [00:20<00:11,  4.63it/s, epoch=41, grad_norm=85.040, loss=328.0541, lr=4.54e-04] 
Epoch 41 TRAIN:  63%|██████▎   | 86/137 [00:20<00:11,  4.63it/s, epoch=41, grad_norm=113.394, loss=260.4865, lr=4.54e-04]
Epoch 41 TRAIN:  63%|██████▎   | 86/137 [00:20<00:11,  4.63it/s, epoch=41, grad_norm=77.079, loss=285.4417, lr=4.54e-04] 
Epoch 41 TRAIN:  65%|██████▍   | 89/137 [00:20<00:10,  4.64it/s, epoch=41, grad_norm=77.079, loss=285.4417, lr=4.54e-04]
Epoch 41 TRAIN:  65%|██████▍   | 89/137 [00:20<00:10,  4.64it/s, epoch=41, grad_norm=195.861, loss=236.3934, lr=4.54e-04]
Epoch 41 TRAIN:  65%|██████▍   | 89/137 [00:20<00:10,  4.64it/s, epoch=41, grad_norm=199.632, loss=184.2863, lr=4.54e-04]
Epoch 41 TRAIN:  65%|██████▍   | 89/137 [00:21<00:10,  4.64it/s, epoch=41, grad_norm=188.803, loss=309.3409, lr=4.55e-04]
Epoch 41 TRAIN:  67%|██████▋   | 92/137 [00:21<00:09,  4.78it/s, epoch=41, grad_norm=188.803, loss=309.3409, lr=4.55e-04]
Epoch 41 TRAIN:  67%|██████▋   | 92/137 [00:21<00:09,  4.78it/s, epoch=41, grad_norm=185.253, loss=315.3225, lr=4.55e-04]
Epoch 41 TRAIN:  67%|██████▋   | 92/137 [00:21<00:09,  4.78it/s, epoch=41, grad_norm=239.232, loss=359.7487, lr=4.55e-04]
Epoch 41 TRAIN:  67%|██████▋   | 92/137 [00:21<00:09,  4.78it/s, epoch=41, grad_norm=144.532, loss=233.2674, lr=4.55e-04]
Epoch 41 TRAIN:  69%|██████▉   | 95/137 [00:21<00:08,  4.83it/s, epoch=41, grad_norm=144.532, loss=233.2674, lr=4.55e-04]
Epoch 41 TRAIN:  69%|██████▉   | 95/137 [00:21<00:08,  4.83it/s, epoch=41, grad_norm=84.801, loss=276.5738, lr=4.55e-04] 
Epoch 41 TRAIN:  69%|██████▉   | 95/137 [00:22<00:08,  4.83it/s, epoch=41, grad_norm=167.966, loss=324.9867, lr=4.55e-04]
Epoch 41 TRAIN:  69%|██████▉   | 95/137 [00:22<00:08,  4.83it/s, epoch=41, grad_norm=122.465, loss=262.2448, lr=4.55e-04]
Epoch 41 TRAIN:  72%|███████▏  | 98/137 [00:22<00:08,  4.75it/s, epoch=41, grad_norm=122.465, loss=262.2448, lr=4.55e-04]
Epoch 41 TRAIN:  72%|███████▏  | 98/137 [00:22<00:08,  4.75it/s, epoch=41, grad_norm=108.965, loss=302.9615, lr=4.55e-04]
Epoch 41 TRAIN:  72%|███████▏  | 98/137 [00:22<00:08,  4.75it/s, epoch=41, grad_norm=142.269, loss=317.1888, lr=4.55e-04]2026-04-16 20:12:51,112 DEBUG TRAIN | steps/sec 4.394| Batch 41/100 loss 317.188812 loss_att 277.460327 loss_ctc 409.888580 th_accuracy 0.304054 lr 4.5520e-04 grad_norm 142.268661 rank 0

Epoch 41 TRAIN:  72%|███████▏  | 98/137 [00:22<00:08,  4.75it/s, epoch=41, grad_norm=110.453, loss=310.0793, lr=4.55e-04]
Epoch 41 TRAIN:  74%|███████▎  | 101/137 [00:22<00:07,  4.73it/s, epoch=41, grad_norm=110.453, loss=310.0793, lr=4.55e-04]
Epoch 41 TRAIN:  74%|███████▎  | 101/137 [00:23<00:07,  4.73it/s, epoch=41, grad_norm=175.784, loss=231.9946, lr=4.55e-04]
Epoch 41 TRAIN:  74%|███████▎  | 101/137 [00:23<00:07,  4.73it/s, epoch=41, grad_norm=253.687, loss=297.1193, lr=4.55e-04]
Epoch 41 TRAIN:  74%|███████▎  | 101/137 [00:23<00:07,  4.73it/s, epoch=41, grad_norm=117.235, loss=321.1760, lr=4.56e-04]
Epoch 41 TRAIN:  76%|███████▌  | 104/137 [00:23<00:07,  4.64it/s, epoch=41, grad_norm=117.235, loss=321.1760, lr=4.56e-04]
Epoch 41 TRAIN:  76%|███████▌  | 104/137 [00:23<00:07,  4.64it/s, epoch=41, grad_norm=139.260, loss=240.9678, lr=4.56e-04]
Epoch 41 TRAIN:  76%|███████▌  | 104/137 [00:24<00:07,  4.64it/s, epoch=41, grad_norm=156.510, loss=362.7402, lr=4.56e-04]
Epoch 41 TRAIN:  76%|███████▌  | 104/137 [00:24<00:07,  4.64it/s, epoch=41, grad_norm=84.967, loss=255.5420, lr=4.56e-04] 
Epoch 41 TRAIN:  78%|███████▊  | 107/137 [00:24<00:06,  4.78it/s, epoch=41, grad_norm=84.967, loss=255.5420, lr=4.56e-04]
Epoch 41 TRAIN:  78%|███████▊  | 107/137 [00:24<00:06,  4.78it/s, epoch=41, grad_norm=155.894, loss=329.2741, lr=4.56e-04]
Epoch 41 TRAIN:  78%|███████▊  | 107/137 [00:24<00:06,  4.78it/s, epoch=41, grad_norm=76.434, loss=253.4865, lr=4.56e-04] 
Epoch 41 TRAIN:  78%|███████▊  | 107/137 [00:24<00:06,  4.78it/s, epoch=41, grad_norm=145.081, loss=296.0831, lr=4.56e-04]
Epoch 41 TRAIN:  80%|████████  | 110/137 [00:24<00:05,  4.90it/s, epoch=41, grad_norm=145.081, loss=296.0831, lr=4.56e-04]
Epoch 41 TRAIN:  80%|████████  | 110/137 [00:25<00:05,  4.90it/s, epoch=41, grad_norm=79.502, loss=280.7370, lr=4.56e-04] 
Epoch 41 TRAIN:  80%|████████  | 110/137 [00:25<00:05,  4.90it/s, epoch=41, grad_norm=146.432, loss=316.3876, lr=4.56e-04]
Epoch 41 TRAIN:  80%|████████  | 110/137 [00:25<00:05,  4.90it/s, epoch=41, grad_norm=100.080, loss=245.5829, lr=4.56e-04]
Epoch 41 TRAIN:  82%|████████▏ | 113/137 [00:25<00:05,  4.49it/s, epoch=41, grad_norm=100.080, loss=245.5829, lr=4.56e-04]
Epoch 41 TRAIN:  82%|████████▏ | 113/137 [00:25<00:05,  4.49it/s, epoch=41, grad_norm=171.775, loss=400.8549, lr=4.56e-04]
Epoch 41 TRAIN:  82%|████████▏ | 113/137 [00:26<00:05,  4.49it/s, epoch=41, grad_norm=91.533, loss=321.0509, lr=4.56e-04] 
Epoch 41 TRAIN:  82%|████████▏ | 113/137 [00:26<00:05,  4.49it/s, epoch=41, grad_norm=155.015, loss=379.8553, lr=4.56e-04]
Epoch 41 TRAIN:  85%|████████▍ | 116/137 [00:26<00:04,  4.59it/s, epoch=41, grad_norm=155.015, loss=379.8553, lr=4.56e-04]
Epoch 41 TRAIN:  85%|████████▍ | 116/137 [00:26<00:04,  4.59it/s, epoch=41, grad_norm=153.090, loss=240.5382, lr=4.57e-04]
Epoch 41 TRAIN:  85%|████████▍ | 116/137 [00:26<00:04,  4.59it/s, epoch=41, grad_norm=150.692, loss=242.3666, lr=4.57e-04]
Epoch 41 TRAIN:  85%|████████▍ | 116/137 [00:26<00:04,  4.59it/s, epoch=41, grad_norm=125.826, loss=314.7734, lr=4.57e-04]
Epoch 41 TRAIN:  87%|████████▋ | 119/137 [00:26<00:03,  4.84it/s, epoch=41, grad_norm=125.826, loss=314.7734, lr=4.57e-04]
Epoch 41 TRAIN:  87%|████████▋ | 119/137 [00:26<00:03,  4.84it/s, epoch=41, grad_norm=140.173, loss=315.4788, lr=4.57e-04]
Epoch 41 TRAIN:  87%|████████▋ | 119/137 [00:27<00:03,  4.84it/s, epoch=41, grad_norm=115.141, loss=337.0798, lr=4.57e-04]
Epoch 41 TRAIN:  87%|████████▋ | 119/137 [00:27<00:03,  4.84it/s, epoch=41, grad_norm=95.844, loss=291.8865, lr=4.57e-04] 
Epoch 41 TRAIN:  89%|████████▉ | 122/137 [00:27<00:03,  4.60it/s, epoch=41, grad_norm=95.844, loss=291.8865, lr=4.57e-04]
Epoch 41 TRAIN:  89%|████████▉ | 122/137 [00:27<00:03,  4.60it/s, epoch=41, grad_norm=164.662, loss=352.6765, lr=4.57e-04]
Epoch 41 TRAIN:  89%|████████▉ | 122/137 [00:27<00:03,  4.60it/s, epoch=41, grad_norm=105.834, loss=308.2810, lr=4.57e-04]
Epoch 41 TRAIN:  89%|████████▉ | 122/137 [00:27<00:03,  4.60it/s, epoch=41, grad_norm=159.158, loss=330.6029, lr=4.57e-04]
Epoch 41 TRAIN:  89%|████████▉ | 122/137 [00:28<00:03,  4.60it/s, epoch=41, grad_norm=150.764, loss=390.2950, lr=4.57e-04]
Epoch 41 TRAIN:  92%|█████████▏| 126/137 [00:28<00:02,  5.10it/s, epoch=41, grad_norm=150.764, loss=390.2950, lr=4.57e-04]
Epoch 41 TRAIN:  92%|█████████▏| 126/137 [00:28<00:02,  5.10it/s, epoch=41, grad_norm=124.529, loss=331.6781, lr=4.57e-04]
Epoch 41 TRAIN:  92%|█████████▏| 126/137 [00:28<00:02,  5.10it/s, epoch=41, grad_norm=123.594, loss=373.4067, lr=4.57e-04]
Epoch 41 TRAIN:  92%|█████████▏| 126/137 [00:28<00:02,  5.10it/s, epoch=41, grad_norm=102.108, loss=347.3882, lr=4.58e-04]
Epoch 41 TRAIN:  94%|█████████▍| 129/137 [00:28<00:01,  4.64it/s, epoch=41, grad_norm=102.108, loss=347.3882, lr=4.58e-04]
Epoch 41 TRAIN:  94%|█████████▍| 129/137 [00:29<00:01,  4.64it/s, epoch=41, grad_norm=102.573, loss=317.6780, lr=4.58e-04]
Epoch 41 TRAIN:  94%|█████████▍| 129/137 [00:29<00:01,  4.64it/s, epoch=41, grad_norm=171.077, loss=425.9077, lr=4.58e-04]
Epoch 41 TRAIN:  94%|█████████▍| 129/137 [00:29<00:01,  4.64it/s, epoch=41, grad_norm=322.330, loss=304.8625, lr=4.58e-04]
Epoch 41 TRAIN:  94%|█████████▍| 129/137 [00:29<00:01,  4.64it/s, epoch=41, grad_norm=176.584, loss=351.3380, lr=4.58e-04]
Epoch 41 TRAIN:  97%|█████████▋| 133/137 [00:29<00:00,  4.81it/s, epoch=41, grad_norm=176.584, loss=351.3380, lr=4.58e-04]
Epoch 41 TRAIN:  97%|█████████▋| 133/137 [00:29<00:00,  4.81it/s, epoch=41, grad_norm=184.769, loss=279.5471, lr=4.58e-04]
Epoch 41 TRAIN:  97%|█████████▋| 133/137 [00:30<00:00,  4.81it/s, epoch=41, grad_norm=170.633, loss=302.2972, lr=4.58e-04]
Epoch 41 TRAIN:  97%|█████████▋| 133/137 [00:30<00:00,  4.81it/s, epoch=41, grad_norm=146.154, loss=285.4631, lr=4.58e-04]
Epoch 41 TRAIN:  97%|█████████▋| 133/137 [00:30<00:00,  4.81it/s, epoch=41, grad_norm=123.705, loss=307.9284, lr=4.58e-04]
Epoch 41 TRAIN: 100%|██████████| 137/137 [00:30<00:00,  5.10it/s, epoch=41, grad_norm=123.705, loss=307.9284, lr=4.58e-04]
Epoch 41 TRAIN: 100%|██████████| 137/137 [00:30<00:00,  4.51it/s, epoch=41, grad_norm=123.705, loss=307.9284, lr=4.58e-04]

CV:   0%|          | 0/6 [00:00<?, ?it/s]/home/rookie/Documents/batuka/projects/wenet/wenet/wenet/utils/train_utils.py:689: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  amp_autocast(enabled=dtype is not None,

CV:  17%|█▋        | 1/6 [00:00<00:03,  1.33it/s]
                                                 
2026-04-16 20:12:59,607 INFO Epoch 41 Step 5727 CV info lr 4.5816e-04 cv_loss 510.37043900923294 rank 0 acc 0.259993573029836
2026-04-16 20:12:59,609 INFO [Rank 0] Checkpoint: save to checkpoint exp/conformer_mongolian/epoch_41.pt
2026-04-16 20:12:59,720 INFO Epoch 42 Step 5727 TRAIN info lr 4.5816e-04 rank 0
2026-04-16 20:12:59,738 INFO using accumulate grad, new batch size is 1 times larger than before

Epoch 42 TRAIN:   0%|          | 0/137 [00:00<?, ?it/s]
Epoch 42 TRAIN:   0%|          | 0/137 [00:03<?, ?it/s, epoch=42, grad_norm=478.570, loss=420.5531, lr=4.58e-04]
Epoch 42 TRAIN:   1%|          | 1/137 [00:03<08:16,  3.65s/it, epoch=42, grad_norm=478.570, loss=420.5531, lr=4.58e-04]
Epoch 42 TRAIN:   1%|          | 1/137 [00:04<08:16,  3.65s/it, epoch=42, grad_norm=554.643, loss=458.2459, lr=4.58e-04]
Epoch 42 TRAIN:   1%|          | 1/137 [00:04<08:16,  3.65s/it, epoch=42, grad_norm=282.402, loss=278.7086, lr=4.58e-04]
Epoch 42 TRAIN:   1%|          | 1/137 [00:04<08:16,  3.65s/it, epoch=42, grad_norm=516.700, loss=453.2238, lr=4.58e-04]
Epoch 42 TRAIN:   3%|▎         | 4/137 [00:04<01:50,  1.21it/s, epoch=42, grad_norm=516.700, loss=453.2238, lr=4.58e-04]
Epoch 42 TRAIN:   3%|▎         | 4/137 [00:04<01:50,  1.21it/s, epoch=42, grad_norm=497.385, loss=451.1931, lr=4.59e-04]
Epoch 42 TRAIN:   3%|▎         | 4/137 [00:04<01:50,  1.21it/s, epoch=42, grad_norm=193.225, loss=320.7081, lr=4.59e-04]
Epoch 42 TRAIN:   3%|▎         | 4/137 [00:04<01:50,  1.21it/s, epoch=42, grad_norm=92.955, loss=251.7858, lr=4.59e-04] 
Epoch 42 TRAIN:   3%|▎         | 4/137 [00:04<01:50,  1.21it/s, epoch=42, grad_norm=245.720, loss=359.5349, lr=4.59e-04]
Epoch 42 TRAIN:   3%|▎         | 4/137 [00:04<01:50,  1.21it/s, epoch=42, grad_norm=211.830, loss=347.1816, lr=4.59e-04]
Epoch 42 TRAIN:   3%|▎         | 4/137 [00:04<01:50,  1.21it/s, epoch=42, grad_norm=81.381, loss=316.8039, lr=4.59e-04] 
Epoch 42 TRAIN:   7%|▋         | 10/137 [00:04<00:40,  3.17it/s, epoch=42, grad_norm=81.381, loss=316.8039, lr=4.59e-04]
Epoch 42 TRAIN:   7%|▋         | 10/137 [00:04<00:40,  3.17it/s, epoch=42, grad_norm=68.374, loss=332.3222, lr=4.59e-04]
Epoch 42 TRAIN:   7%|▋         | 10/137 [00:05<00:40,  3.17it/s, epoch=42, grad_norm=148.088, loss=310.6002, lr=4.59e-04]
Epoch 42 TRAIN:   7%|▋         | 10/137 [00:05<00:40,  3.17it/s, epoch=42, grad_norm=124.096, loss=309.7762, lr=4.59e-04]
Epoch 42 TRAIN:   7%|▋         | 10/137 [00:05<00:40,  3.17it/s, epoch=42, grad_norm=96.321, loss=311.3052, lr=4.59e-04] 
Epoch 42 TRAIN:  10%|█         | 14/137 [00:05<00:31,  3.94it/s, epoch=42, grad_norm=96.321, loss=311.3052, lr=4.59e-04]
Epoch 42 TRAIN:  10%|█         | 14/137 [00:05<00:31,  3.94it/s, epoch=42, grad_norm=130.213, loss=290.9947, lr=4.59e-04]
Epoch 42 TRAIN:  10%|█         | 14/137 [00:05<00:31,  3.94it/s, epoch=42, grad_norm=103.672, loss=327.3961, lr=4.59e-04]
Epoch 42 TRAIN:  10%|█         | 14/137 [00:05<00:31,  3.94it/s, epoch=42, grad_norm=122.584, loss=352.7513, lr=4.60e-04]
Epoch 42 TRAIN:  10%|█         | 14/137 [00:06<00:31,  3.94it/s, epoch=42, grad_norm=63.358, loss=266.9743, lr=4.60e-04] 
Epoch 42 TRAIN:  13%|█▎        | 18/137 [00:06<00:26,  4.54it/s, epoch=42, grad_norm=63.358, loss=266.9743, lr=4.60e-04]
Epoch 42 TRAIN:  13%|█▎        | 18/137 [00:06<00:26,  4.54it/s, epoch=42, grad_norm=113.037, loss=336.1695, lr=4.60e-04]
Epoch 42 TRAIN:  13%|█▎        | 18/137 [00:06<00:26,  4.54it/s, epoch=42, grad_norm=61.532, loss=298.5681, lr=4.60e-04] 
Epoch 42 TRAIN:  13%|█▎        | 18/137 [00:06<00:26,  4.54it/s, epoch=42, grad_norm=70.113, loss=320.4729, lr=4.60e-04]
Epoch 42 TRAIN:  13%|█▎        | 18/137 [00:06<00:26,  4.54it/s, epoch=42, grad_norm=100.082, loss=273.1942, lr=4.60e-04]
Epoch 42 TRAIN:  16%|█▌        | 22/137 [00:06<00:24,  4.72it/s, epoch=42, grad_norm=100.082, loss=273.1942, lr=4.60e-04]
Epoch 42 TRAIN:  16%|█▌        | 22/137 [00:06<00:24,  4.72it/s, epoch=42, grad_norm=109.284, loss=343.3079, lr=4.60e-04]
Epoch 42 TRAIN:  16%|█▌        | 22/137 [00:07<00:24,  4.72it/s, epoch=42, grad_norm=80.007, loss=304.5329, lr=4.60e-04] 
Epoch 42 TRAIN:  16%|█▌        | 22/137 [00:07<00:24,  4.72it/s, epoch=42, grad_norm=84.453, loss=301.4081, lr=4.60e-04]
Epoch 42 TRAIN:  16%|█▌        | 22/137 [00:07<00:24,  4.72it/s, epoch=42, grad_norm=63.145, loss=298.2584, lr=4.60e-04]
Epoch 42 TRAIN:  19%|█▉        | 26/137 [00:07<00:21,  5.12it/s, epoch=42, grad_norm=63.145, loss=298.2584, lr=4.60e-04]
Epoch 42 TRAIN:  19%|█▉        | 26/137 [00:07<00:21,  5.12it/s, epoch=42, grad_norm=94.400, loss=295.8078, lr=4.60e-04]
Epoch 42 TRAIN:  19%|█▉        | 26/137 [00:07<00:21,  5.12it/s, epoch=42, grad_norm=100.087, loss=277.3790, lr=4.60e-04]
Epoch 42 TRAIN:  19%|█▉        | 26/137 [00:08<00:21,  5.12it/s, epoch=42, grad_norm=140.314, loss=346.4948, lr=4.60e-04]
Epoch 42 TRAIN:  19%|█▉        | 26/137 [00:08<00:21,  5.12it/s, epoch=42, grad_norm=77.396, loss=289.6910, lr=4.61e-04] 
Epoch 42 TRAIN:  22%|██▏       | 30/137 [00:08<00:20,  5.16it/s, epoch=42, grad_norm=77.396, loss=289.6910, lr=4.61e-04]
Epoch 42 TRAIN:  22%|██▏       | 30/137 [00:08<00:20,  5.16it/s, epoch=42, grad_norm=124.003, loss=315.5860, lr=4.61e-04]
Epoch 42 TRAIN:  22%|██▏       | 30/137 [00:08<00:20,  5.16it/s, epoch=42, grad_norm=116.992, loss=330.0240, lr=4.61e-04]
Epoch 42 TRAIN:  22%|██▏       | 30/137 [00:08<00:20,  5.16it/s, epoch=42, grad_norm=73.510, loss=299.3546, lr=4.61e-04] 
Epoch 42 TRAIN:  24%|██▍       | 33/137 [00:08<00:21,  4.92it/s, epoch=42, grad_norm=73.510, loss=299.3546, lr=4.61e-04]
Epoch 42 TRAIN:  24%|██▍       | 33/137 [00:09<00:21,  4.92it/s, epoch=42, grad_norm=78.909, loss=299.3387, lr=4.61e-04]
Epoch 42 TRAIN:  24%|██▍       | 33/137 [00:09<00:21,  4.92it/s, epoch=42, grad_norm=111.096, loss=330.8549, lr=4.61e-04]
Epoch 42 TRAIN:  24%|██▍       | 33/137 [00:09<00:21,  4.92it/s, epoch=42, grad_norm=72.545, loss=333.1653, lr=4.61e-04] 
Epoch 42 TRAIN:  26%|██▋       | 36/137 [00:09<00:19,  5.14it/s, epoch=42, grad_norm=72.545, loss=333.1653, lr=4.61e-04]
Epoch 42 TRAIN:  26%|██▋       | 36/137 [00:09<00:19,  5.14it/s, epoch=42, grad_norm=78.618, loss=301.6079, lr=4.61e-04]
Epoch 42 TRAIN:  26%|██▋       | 36/137 [00:09<00:19,  5.14it/s, epoch=42, grad_norm=83.598, loss=311.9615, lr=4.61e-04]
Epoch 42 TRAIN:  26%|██▋       | 36/137 [00:10<00:19,  5.14it/s, epoch=42, grad_norm=75.065, loss=310.6698, lr=4.61e-04]
Epoch 42 TRAIN:  28%|██▊       | 39/137 [00:10<00:19,  4.94it/s, epoch=42, grad_norm=75.065, loss=310.6698, lr=4.61e-04]
Epoch 42 TRAIN:  28%|██▊       | 39/137 [00:10<00:19,  4.94it/s, epoch=42, grad_norm=77.775, loss=305.3279, lr=4.61e-04]
Epoch 42 TRAIN:  28%|██▊       | 39/137 [00:10<00:19,  4.94it/s, epoch=42, grad_norm=120.966, loss=298.4472, lr=4.61e-04]
Epoch 42 TRAIN:  28%|██▊       | 39/137 [00:10<00:19,  4.94it/s, epoch=42, grad_norm=82.650, loss=325.4055, lr=4.62e-04] 
Epoch 42 TRAIN:  31%|███       | 42/137 [00:10<00:18,  5.01it/s, epoch=42, grad_norm=82.650, loss=325.4055, lr=4.62e-04]
Epoch 42 TRAIN:  31%|███       | 42/137 [00:10<00:18,  5.01it/s, epoch=42, grad_norm=80.088, loss=288.4521, lr=4.62e-04]
Epoch 42 TRAIN:  31%|███       | 42/137 [00:11<00:18,  5.01it/s, epoch=42, grad_norm=99.253, loss=345.7704, lr=4.62e-04]
Epoch 42 TRAIN:  31%|███       | 42/137 [00:11<00:18,  5.01it/s, epoch=42, grad_norm=103.843, loss=339.9362, lr=4.62e-04]
Epoch 42 TRAIN:  33%|███▎      | 45/137 [00:11<00:19,  4.75it/s, epoch=42, grad_norm=103.843, loss=339.9362, lr=4.62e-04]
Epoch 42 TRAIN:  33%|███▎      | 45/137 [00:11<00:19,  4.75it/s, epoch=42, grad_norm=71.912, loss=296.8494, lr=4.62e-04] 
Epoch 42 TRAIN:  33%|███▎      | 45/137 [00:11<00:19,  4.75it/s, epoch=42, grad_norm=125.401, loss=350.0935, lr=4.62e-04]
Epoch 42 TRAIN:  33%|███▎      | 45/137 [00:12<00:19,  4.75it/s, epoch=42, grad_norm=116.909, loss=333.0091, lr=4.62e-04]
Epoch 42 TRAIN:  35%|███▌      | 48/137 [00:12<00:18,  4.71it/s, epoch=42, grad_norm=116.909, loss=333.0091, lr=4.62e-04]
Epoch 42 TRAIN:  35%|███▌      | 48/137 [00:12<00:18,  4.71it/s, epoch=42, grad_norm=152.323, loss=288.1203, lr=4.62e-04]
Epoch 42 TRAIN:  35%|███▌      | 48/137 [00:12<00:18,  4.71it/s, epoch=42, grad_norm=125.903, loss=268.7383, lr=4.62e-04]
Epoch 42 TRAIN:  35%|███▌      | 48/137 [00:12<00:18,  4.71it/s, epoch=42, grad_norm=88.610, loss=322.1624, lr=4.62e-04] 
Epoch 42 TRAIN:  37%|███▋      | 51/137 [00:12<00:17,  4.79it/s, epoch=42, grad_norm=88.610, loss=322.1624, lr=4.62e-04]
Epoch 42 TRAIN:  37%|███▋      | 51/137 [00:12<00:17,  4.79it/s, epoch=42, grad_norm=91.481, loss=322.4679, lr=4.62e-04]
Epoch 42 TRAIN:  37%|███▋      | 51/137 [00:13<00:17,  4.79it/s, epoch=42, grad_norm=82.542, loss=298.1982, lr=4.62e-04]
Epoch 42 TRAIN:  37%|███▋      | 51/137 [00:13<00:17,  4.79it/s, epoch=42, grad_norm=83.573, loss=297.9991, lr=4.62e-04]
Epoch 42 TRAIN:  39%|███▉      | 54/137 [00:13<00:17,  4.87it/s, epoch=42, grad_norm=83.573, loss=297.9991, lr=4.62e-04]
Epoch 42 TRAIN:  39%|███▉      | 54/137 [00:13<00:17,  4.87it/s, epoch=42, grad_norm=115.818, loss=339.1357, lr=4.63e-04]
Epoch 42 TRAIN:  39%|███▉      | 54/137 [00:13<00:17,  4.87it/s, epoch=42, grad_norm=99.116, loss=256.8582, lr=4.63e-04] 
Epoch 42 TRAIN:  39%|███▉      | 54/137 [00:13<00:17,  4.87it/s, epoch=42, grad_norm=98.743, loss=306.2571, lr=4.63e-04]
Epoch 42 TRAIN:  42%|████▏     | 57/137 [00:13<00:16,  4.88it/s, epoch=42, grad_norm=98.743, loss=306.2571, lr=4.63e-04]
Epoch 42 TRAIN:  42%|████▏     | 57/137 [00:14<00:16,  4.88it/s, epoch=42, grad_norm=98.430, loss=308.5133, lr=4.63e-04]
Epoch 42 TRAIN:  42%|████▏     | 57/137 [00:14<00:16,  4.88it/s, epoch=42, grad_norm=120.034, loss=283.4717, lr=4.63e-04]
Epoch 42 TRAIN:  42%|████▏     | 57/137 [00:14<00:16,  4.88it/s, epoch=42, grad_norm=78.568, loss=324.8860, lr=4.63e-04] 
Epoch 42 TRAIN:  44%|████▍     | 60/137 [00:14<00:15,  4.84it/s, epoch=42, grad_norm=78.568, loss=324.8860, lr=4.63e-04]
Epoch 42 TRAIN:  44%|████▍     | 60/137 [00:14<00:15,  4.84it/s, epoch=42, grad_norm=72.306, loss=300.7079, lr=4.63e-04]
Epoch 42 TRAIN:  44%|████▍     | 60/137 [00:14<00:15,  4.84it/s, epoch=42, grad_norm=82.037, loss=311.8484, lr=4.63e-04]
Epoch 42 TRAIN:  44%|████▍     | 60/137 [00:15<00:15,  4.84it/s, epoch=42, grad_norm=86.301, loss=308.7003, lr=4.63e-04]
Epoch 42 TRAIN:  46%|████▌     | 63/137 [00:15<00:15,  4.83it/s, epoch=42, grad_norm=86.301, loss=308.7003, lr=4.63e-04]
Epoch 42 TRAIN:  46%|████▌     | 63/137 [00:15<00:15,  4.83it/s, epoch=42, grad_norm=78.928, loss=316.3796, lr=4.63e-04]
Epoch 42 TRAIN:  46%|████▌     | 63/137 [00:15<00:15,  4.83it/s, epoch=42, grad_norm=98.742, loss=362.3303, lr=4.63e-04]
Epoch 42 TRAIN:  46%|████▌     | 63/137 [00:15<00:15,  4.83it/s, epoch=42, grad_norm=103.680, loss=288.9867, lr=4.63e-04]
Epoch 42 TRAIN:  48%|████▊     | 66/137 [00:15<00:14,  4.97it/s, epoch=42, grad_norm=103.680, loss=288.9867, lr=4.63e-04]
Epoch 42 TRAIN:  48%|████▊     | 66/137 [00:15<00:14,  4.97it/s, epoch=42, grad_norm=169.892, loss=260.9708, lr=4.64e-04]
Epoch 42 TRAIN:  48%|████▊     | 66/137 [00:16<00:14,  4.97it/s, epoch=42, grad_norm=120.748, loss=379.6635, lr=4.64e-04]
Epoch 42 TRAIN:  48%|████▊     | 66/137 [00:16<00:14,  4.97it/s, epoch=42, grad_norm=78.905, loss=284.2248, lr=4.64e-04] 
Epoch 42 TRAIN:  50%|█████     | 69/137 [00:16<00:13,  4.93it/s, epoch=42, grad_norm=78.905, loss=284.2248, lr=4.64e-04]
Epoch 42 TRAIN:  50%|█████     | 69/137 [00:16<00:13,  4.93it/s, epoch=42, grad_norm=123.464, loss=258.7352, lr=4.64e-04]
Epoch 42 TRAIN:  50%|█████     | 69/137 [00:16<00:13,  4.93it/s, epoch=42, grad_norm=154.733, loss=340.8732, lr=4.64e-04]
Epoch 42 TRAIN:  50%|█████     | 69/137 [00:16<00:13,  4.93it/s, epoch=42, grad_norm=116.335, loss=329.1724, lr=4.64e-04]
Epoch 42 TRAIN:  53%|█████▎    | 72/137 [00:16<00:12,  5.04it/s, epoch=42, grad_norm=116.335, loss=329.1724, lr=4.64e-04]
Epoch 42 TRAIN:  53%|█████▎    | 72/137 [00:17<00:12,  5.04it/s, epoch=42, grad_norm=147.662, loss=318.5092, lr=4.64e-04]
Epoch 42 TRAIN:  53%|█████▎    | 72/137 [00:17<00:12,  5.04it/s, epoch=42, grad_norm=82.845, loss=284.0528, lr=4.64e-04] 
Epoch 42 TRAIN:  53%|█████▎    | 72/137 [00:17<00:12,  5.04it/s, epoch=42, grad_norm=101.226, loss=275.3221, lr=4.64e-04]
Epoch 42 TRAIN:  55%|█████▍    | 75/137 [00:17<00:12,  4.93it/s, epoch=42, grad_norm=101.226, loss=275.3221, lr=4.64e-04]
Epoch 42 TRAIN:  55%|█████▍    | 75/137 [00:17<00:12,  4.93it/s, epoch=42, grad_norm=96.272, loss=300.8264, lr=4.64e-04] 
Epoch 42 TRAIN:  55%|█████▍    | 75/137 [00:17<00:12,  4.93it/s, epoch=42, grad_norm=143.404, loss=320.5456, lr=4.64e-04]
Epoch 42 TRAIN:  55%|█████▍    | 75/137 [00:18<00:12,  4.93it/s, epoch=42, grad_norm=182.805, loss=331.9541, lr=4.64e-04]
Epoch 42 TRAIN:  57%|█████▋    | 78/137 [00:18<00:11,  5.04it/s, epoch=42, grad_norm=182.805, loss=331.9541, lr=4.64e-04]
Epoch 42 TRAIN:  57%|█████▋    | 78/137 [00:18<00:11,  5.04it/s, epoch=42, grad_norm=158.560, loss=307.5112, lr=4.64e-04]
Epoch 42 TRAIN:  57%|█████▋    | 78/137 [00:18<00:11,  5.04it/s, epoch=42, grad_norm=150.607, loss=333.1049, lr=4.65e-04]
Epoch 42 TRAIN:  57%|█████▋    | 78/137 [00:18<00:11,  5.04it/s, epoch=42, grad_norm=159.652, loss=344.5615, lr=4.65e-04]
Epoch 42 TRAIN:  59%|█████▉    | 81/137 [00:18<00:11,  4.70it/s, epoch=42, grad_norm=159.652, loss=344.5615, lr=4.65e-04]
Epoch 42 TRAIN:  59%|█████▉    | 81/137 [00:18<00:11,  4.70it/s, epoch=42, grad_norm=101.526, loss=328.9759, lr=4.65e-04]
Epoch 42 TRAIN:  59%|█████▉    | 81/137 [00:19<00:11,  4.70it/s, epoch=42, grad_norm=155.146, loss=309.7195, lr=4.65e-04]
Epoch 42 TRAIN:  59%|█████▉    | 81/137 [00:19<00:11,  4.70it/s, epoch=42, grad_norm=297.316, loss=265.9349, lr=4.65e-04]
Epoch 42 TRAIN:  61%|██████▏   | 84/137 [00:19<00:11,  4.58it/s, epoch=42, grad_norm=297.316, loss=265.9349, lr=4.65e-04]
Epoch 42 TRAIN:  61%|██████▏   | 84/137 [00:19<00:11,  4.58it/s, epoch=42, grad_norm=103.566, loss=314.9352, lr=4.65e-04]
Epoch 42 TRAIN:  61%|██████▏   | 84/137 [00:19<00:11,  4.58it/s, epoch=42, grad_norm=133.517, loss=293.1395, lr=4.65e-04]
Epoch 42 TRAIN:  61%|██████▏   | 84/137 [00:19<00:11,  4.58it/s, epoch=42, grad_norm=162.199, loss=338.5213, lr=4.65e-04]
Epoch 42 TRAIN:  64%|██████▎   | 87/137 [00:19<00:10,  4.91it/s, epoch=42, grad_norm=162.199, loss=338.5213, lr=4.65e-04]
Epoch 42 TRAIN:  64%|██████▎   | 87/137 [00:20<00:10,  4.91it/s, epoch=42, grad_norm=219.344, loss=403.2045, lr=4.65e-04]
Epoch 42 TRAIN:  64%|██████▎   | 87/137 [00:20<00:10,  4.91it/s, epoch=42, grad_norm=263.771, loss=252.8006, lr=4.65e-04]
Epoch 42 TRAIN:  64%|██████▎   | 87/137 [00:20<00:10,  4.91it/s, epoch=42, grad_norm=112.360, loss=226.6314, lr=4.65e-04]
Epoch 42 TRAIN:  66%|██████▌   | 90/137 [00:20<00:10,  4.66it/s, epoch=42, grad_norm=112.360, loss=226.6314, lr=4.65e-04]
Epoch 42 TRAIN:  66%|██████▌   | 90/137 [00:20<00:10,  4.66it/s, epoch=42, grad_norm=146.051, loss=294.1049, lr=4.65e-04]
Epoch 42 TRAIN:  66%|██████▌   | 90/137 [00:21<00:10,  4.66it/s, epoch=42, grad_norm=99.038, loss=298.5049, lr=4.66e-04] 
Epoch 42 TRAIN:  66%|██████▌   | 90/137 [00:21<00:10,  4.66it/s, epoch=42, grad_norm=142.687, loss=317.0586, lr=4.66e-04]
Epoch 42 TRAIN:  68%|██████▊   | 93/137 [00:21<00:09,  4.70it/s, epoch=42, grad_norm=142.687, loss=317.0586, lr=4.66e-04]
Epoch 42 TRAIN:  68%|██████▊   | 93/137 [00:21<00:09,  4.70it/s, epoch=42, grad_norm=109.816, loss=192.8938, lr=4.66e-04]
Epoch 42 TRAIN:  68%|██████▊   | 93/137 [00:21<00:09,  4.70it/s, epoch=42, grad_norm=268.732, loss=329.1177, lr=4.66e-04]
Epoch 42 TRAIN:  68%|██████▊   | 93/137 [00:21<00:09,  4.70it/s, epoch=42, grad_norm=264.287, loss=350.8000, lr=4.66e-04]
Epoch 42 TRAIN:  70%|███████   | 96/137 [00:21<00:08,  4.85it/s, epoch=42, grad_norm=264.287, loss=350.8000, lr=4.66e-04]
Epoch 42 TRAIN:  70%|███████   | 96/137 [00:22<00:08,  4.85it/s, epoch=42, grad_norm=102.670, loss=286.5400, lr=4.66e-04]
Epoch 42 TRAIN:  70%|███████   | 96/137 [00:22<00:08,  4.85it/s, epoch=42, grad_norm=224.750, loss=338.9572, lr=4.66e-04]
Epoch 42 TRAIN:  70%|███████   | 96/137 [00:22<00:08,  4.85it/s, epoch=42, grad_norm=107.632, loss=278.9634, lr=4.66e-04]
Epoch 42 TRAIN:  72%|███████▏  | 99/137 [00:22<00:07,  4.82it/s, epoch=42, grad_norm=107.632, loss=278.9634, lr=4.66e-04]
Epoch 42 TRAIN:  72%|███████▏  | 99/137 [00:22<00:07,  4.82it/s, epoch=42, grad_norm=102.166, loss=304.9180, lr=4.66e-04]2026-04-16 20:13:22,469 DEBUG TRAIN | steps/sec 4.369| Batch 42/100 loss 304.918030 loss_att 268.204956 loss_ctc 390.581848 th_accuracy 0.326291 lr 4.6616e-04 grad_norm 102.166245 rank 0

Epoch 42 TRAIN:  72%|███████▏  | 99/137 [00:22<00:07,  4.82it/s, epoch=42, grad_norm=103.362, loss=246.3222, lr=4.66e-04]
Epoch 42 TRAIN:  72%|███████▏  | 99/137 [00:23<00:07,  4.82it/s, epoch=42, grad_norm=137.117, loss=242.2000, lr=4.66e-04]
Epoch 42 TRAIN:  74%|███████▍  | 102/137 [00:23<00:07,  4.76it/s, epoch=42, grad_norm=137.117, loss=242.2000, lr=4.66e-04]
Epoch 42 TRAIN:  74%|███████▍  | 102/137 [00:23<00:07,  4.76it/s, epoch=42, grad_norm=188.615, loss=343.1122, lr=4.66e-04]
Epoch 42 TRAIN:  74%|███████▍  | 102/137 [00:23<00:07,  4.76it/s, epoch=42, grad_norm=107.105, loss=222.9290, lr=4.66e-04]
Epoch 42 TRAIN:  74%|███████▍  | 102/137 [00:23<00:07,  4.76it/s, epoch=42, grad_norm=128.815, loss=343.6798, lr=4.67e-04]
Epoch 42 TRAIN:  77%|███████▋  | 105/137 [00:23<00:07,  4.51it/s, epoch=42, grad_norm=128.815, loss=343.6798, lr=4.67e-04]
Epoch 42 TRAIN:  77%|███████▋  | 105/137 [00:24<00:07,  4.51it/s, epoch=42, grad_norm=95.330, loss=244.8387, lr=4.67e-04] 
Epoch 42 TRAIN:  77%|███████▋  | 105/137 [00:24<00:07,  4.51it/s, epoch=42, grad_norm=111.017, loss=340.5584, lr=4.67e-04]
Epoch 42 TRAIN:  77%|███████▋  | 105/137 [00:24<00:07,  4.51it/s, epoch=42, grad_norm=176.727, loss=379.1941, lr=4.67e-04]
Epoch 42 TRAIN:  77%|███████▋  | 105/137 [00:24<00:07,  4.51it/s, epoch=42, grad_norm=95.783, loss=269.0863, lr=4.67e-04] 
Epoch 42 TRAIN:  80%|███████▉  | 109/137 [00:24<00:05,  4.98it/s, epoch=42, grad_norm=95.783, loss=269.0863, lr=4.67e-04]
Epoch 42 TRAIN:  80%|███████▉  | 109/137 [00:24<00:05,  4.98it/s, epoch=42, grad_norm=115.875, loss=312.0776, lr=4.67e-04]
Epoch 42 TRAIN:  80%|███████▉  | 109/137 [00:25<00:05,  4.98it/s, epoch=42, grad_norm=138.921, loss=294.8963, lr=4.67e-04]
Epoch 42 TRAIN:  80%|███████▉  | 109/137 [00:25<00:05,  4.98it/s, epoch=42, grad_norm=122.372, loss=266.0227, lr=4.67e-04]
Epoch 42 TRAIN:  82%|████████▏ | 112/137 [00:25<00:05,  4.61it/s, epoch=42, grad_norm=122.372, loss=266.0227, lr=4.67e-04]
Epoch 42 TRAIN:  82%|████████▏ | 112/137 [00:25<00:05,  4.61it/s, epoch=42, grad_norm=188.165, loss=338.9643, lr=4.67e-04]
Epoch 42 TRAIN:  82%|████████▏ | 112/137 [00:25<00:05,  4.61it/s, epoch=42, grad_norm=175.083, loss=344.3913, lr=4.67e-04]
Epoch 42 TRAIN:  82%|████████▏ | 112/137 [00:25<00:05,  4.61it/s, epoch=42, grad_norm=209.041, loss=348.2708, lr=4.67e-04]
Epoch 42 TRAIN:  84%|████████▍ | 115/137 [00:25<00:04,  4.74it/s, epoch=42, grad_norm=209.041, loss=348.2708, lr=4.67e-04]
Epoch 42 TRAIN:  84%|████████▍ | 115/137 [00:26<00:04,  4.74it/s, epoch=42, grad_norm=169.168, loss=258.1680, lr=4.67e-04]
Epoch 42 TRAIN:  84%|████████▍ | 115/137 [00:26<00:04,  4.74it/s, epoch=42, grad_norm=128.904, loss=328.4741, lr=4.68e-04]
Epoch 42 TRAIN:  84%|████████▍ | 115/137 [00:26<00:04,  4.74it/s, epoch=42, grad_norm=99.880, loss=278.1125, lr=4.68e-04] 
Epoch 42 TRAIN:  86%|████████▌ | 118/137 [00:26<00:04,  4.66it/s, epoch=42, grad_norm=99.880, loss=278.1125, lr=4.68e-04]
Epoch 42 TRAIN:  86%|████████▌ | 118/137 [00:26<00:04,  4.66it/s, epoch=42, grad_norm=118.885, loss=303.1167, lr=4.68e-04]
Epoch 42 TRAIN:  86%|████████▌ | 118/137 [00:27<00:04,  4.66it/s, epoch=42, grad_norm=109.078, loss=283.5068, lr=4.68e-04]
Epoch 42 TRAIN:  86%|████████▌ | 118/137 [00:27<00:04,  4.66it/s, epoch=42, grad_norm=183.832, loss=385.2134, lr=4.68e-04]
Epoch 42 TRAIN:  88%|████████▊ | 121/137 [00:27<00:03,  4.83it/s, epoch=42, grad_norm=183.832, loss=385.2134, lr=4.68e-04]
Epoch 42 TRAIN:  88%|████████▊ | 121/137 [00:27<00:03,  4.83it/s, epoch=42, grad_norm=153.249, loss=351.7220, lr=4.68e-04]
Epoch 42 TRAIN:  88%|████████▊ | 121/137 [00:27<00:03,  4.83it/s, epoch=42, grad_norm=112.859, loss=325.9625, lr=4.68e-04]
Epoch 42 TRAIN:  88%|████████▊ | 121/137 [00:27<00:03,  4.83it/s, epoch=42, grad_norm=145.852, loss=322.4058, lr=4.68e-04]
Epoch 42 TRAIN:  91%|█████████ | 124/137 [00:27<00:02,  4.79it/s, epoch=42, grad_norm=145.852, loss=322.4058, lr=4.68e-04]
Epoch 42 TRAIN:  91%|█████████ | 124/137 [00:28<00:02,  4.79it/s, epoch=42, grad_norm=125.244, loss=306.1806, lr=4.68e-04]
Epoch 42 TRAIN:  91%|█████████ | 124/137 [00:28<00:02,  4.79it/s, epoch=42, grad_norm=273.978, loss=272.1670, lr=4.68e-04]
Epoch 42 TRAIN:  91%|█████████ | 124/137 [00:28<00:02,  4.79it/s, epoch=42, grad_norm=310.576, loss=311.5435, lr=4.68e-04]
Epoch 42 TRAIN:  93%|█████████▎| 127/137 [00:28<00:02,  4.75it/s, epoch=42, grad_norm=310.576, loss=311.5435, lr=4.68e-04]
Epoch 42 TRAIN:  93%|█████████▎| 127/137 [00:28<00:02,  4.75it/s, epoch=42, grad_norm=150.397, loss=340.9948, lr=4.68e-04]
Epoch 42 TRAIN:  93%|█████████▎| 127/137 [00:28<00:02,  4.75it/s, epoch=42, grad_norm=118.732, loss=339.3138, lr=4.68e-04]
Epoch 42 TRAIN:  93%|█████████▎| 127/137 [00:29<00:02,  4.75it/s, epoch=42, grad_norm=136.598, loss=195.6625, lr=4.69e-04]
Epoch 42 TRAIN:  95%|█████████▍| 130/137 [00:29<00:01,  4.81it/s, epoch=42, grad_norm=136.598, loss=195.6625, lr=4.69e-04]
Epoch 42 TRAIN:  95%|█████████▍| 130/137 [00:29<00:01,  4.81it/s, epoch=42, grad_norm=103.249, loss=284.4294, lr=4.69e-04]
Epoch 42 TRAIN:  95%|█████████▍| 130/137 [00:29<00:01,  4.81it/s, epoch=42, grad_norm=177.075, loss=339.4306, lr=4.69e-04]
Epoch 42 TRAIN:  95%|█████████▍| 130/137 [00:29<00:01,  4.81it/s, epoch=42, grad_norm=179.118, loss=289.6867, lr=4.69e-04]
Epoch 42 TRAIN:  97%|█████████▋| 133/137 [00:29<00:00,  4.78it/s, epoch=42, grad_norm=179.118, loss=289.6867, lr=4.69e-04]
Epoch 42 TRAIN:  97%|█████████▋| 133/137 [00:29<00:00,  4.78it/s, epoch=42, grad_norm=299.199, loss=423.8764, lr=4.69e-04]
Epoch 42 TRAIN:  97%|█████████▋| 133/137 [00:30<00:00,  4.78it/s, epoch=42, grad_norm=215.621, loss=354.5087, lr=4.69e-04]
Epoch 42 TRAIN:  99%|█████████▊| 135/137 [00:30<00:00,  4.50it/s, epoch=42, grad_norm=215.621, loss=354.5087, lr=4.69e-04]

CV:   0%|          | 0/6 [00:00<?, ?it/s]/home/rookie/Documents/batuka/projects/wenet/wenet/wenet/utils/train_utils.py:689: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  amp_autocast(enabled=dtype is not None,

CV:  17%|█▋        | 1/6 [00:00<00:03,  1.31it/s]
                                                 
2026-04-16 20:13:30,609 INFO Epoch 42 Step 5862 CV info lr 4.6896e-04 cv_loss 488.420371315696 rank 0 acc 0.26683875918388367
2026-04-16 20:13:30,610 INFO [Rank 0] Checkpoint: save to checkpoint exp/conformer_mongolian/epoch_42.pt
2026-04-16 20:13:30,721 INFO Epoch 43 Step 5862 TRAIN info lr 4.6896e-04 rank 0
2026-04-16 20:13:30,738 INFO using accumulate grad, new batch size is 1 times larger than before

Epoch 43 TRAIN:   0%|          | 0/135 [00:00<?, ?it/s]
Epoch 43 TRAIN:   0%|          | 0/135 [00:04<?, ?it/s, epoch=43, grad_norm=292.658, loss=299.3769, lr=4.69e-04]
Epoch 43 TRAIN:   1%|          | 1/135 [00:04<09:00,  4.03s/it, epoch=43, grad_norm=292.658, loss=299.3769, lr=4.69e-04]
Epoch 43 TRAIN:   1%|          | 1/135 [00:04<09:00,  4.03s/it, epoch=43, grad_norm=353.348, loss=344.1107, lr=4.69e-04]
Epoch 43 TRAIN:   1%|          | 1/135 [00:04<09:00,  4.03s/it, epoch=43, grad_norm=272.588, loss=300.1080, lr=4.69e-04]
Epoch 43 TRAIN:   1%|          | 1/135 [00:04<09:00,  4.03s/it, epoch=43, grad_norm=459.318, loss=439.3078, lr=4.69e-04]
Epoch 43 TRAIN:   1%|          | 1/135 [00:04<09:00,  4.03s/it, epoch=43, grad_norm=240.380, loss=320.3948, lr=4.69e-04]
Epoch 43 TRAIN:   1%|          | 1/135 [00:04<09:00,  4.03s/it, epoch=43, grad_norm=339.129, loss=433.9590, lr=4.69e-04]
Epoch 43 TRAIN:   1%|          | 1/135 [00:04<09:00,  4.03s/it, epoch=43, grad_norm=230.963, loss=367.8768, lr=4.70e-04]
Epoch 43 TRAIN:   1%|          | 1/135 [00:04<09:00,  4.03s/it, epoch=43, grad_norm=94.869, loss=276.6429, lr=4.70e-04] 
Epoch 43 TRAIN:   1%|          | 1/135 [00:04<09:00,  4.03s/it, epoch=43, grad_norm=78.466, loss=314.3097, lr=4.70e-04]
Epoch 43 TRAIN:   7%|▋         | 9/135 [00:04<00:49,  2.52it/s, epoch=43, grad_norm=78.466, loss=314.3097, lr=4.70e-04]
Epoch 43 TRAIN:   7%|▋         | 9/135 [00:04<00:49,  2.52it/s, epoch=43, grad_norm=104.499, loss=339.2803, lr=4.70e-04]
Epoch 43 TRAIN:   7%|▋         | 9/135 [00:04<00:49,  2.52it/s, epoch=43, grad_norm=133.352, loss=303.6331, lr=4.70e-04]
Epoch 43 TRAIN:   7%|▋         | 9/135 [00:05<00:49,  2.52it/s, epoch=43, grad_norm=115.386, loss=324.1274, lr=4.70e-04]
Epoch 43 TRAIN:   7%|▋         | 9/135 [00:05<00:49,  2.52it/s, epoch=43, grad_norm=114.961, loss=321.6345, lr=4.70e-04]
Epoch 43 TRAIN:  10%|▉         | 13/135 [00:05<00:36,  3.31it/s, epoch=43, grad_norm=114.961, loss=321.6345, lr=4.70e-04]
Epoch 43 TRAIN:  10%|▉         | 13/135 [00:05<00:36,  3.31it/s, epoch=43, grad_norm=70.361, loss=333.1098, lr=4.70e-04] 
Epoch 43 TRAIN:  10%|▉         | 13/135 [00:05<00:36,  3.31it/s, epoch=43, grad_norm=91.109, loss=318.3693, lr=4.70e-04]
Epoch 43 TRAIN:  10%|▉         | 13/135 [00:05<00:36,  3.31it/s, epoch=43, grad_norm=76.239, loss=315.6218, lr=4.70e-04]
Epoch 43 TRAIN:  10%|▉         | 13/135 [00:05<00:36,  3.31it/s, epoch=43, grad_norm=167.287, loss=257.6001, lr=4.70e-04]
Epoch 43 TRAIN:  13%|█▎        | 17/135 [00:05<00:29,  4.01it/s, epoch=43, grad_norm=167.287, loss=257.6001, lr=4.70e-04]
Epoch 43 TRAIN:  13%|█▎        | 17/135 [00:06<00:29,  4.01it/s, epoch=43, grad_norm=82.353, loss=282.5471, lr=4.70e-04] 
Epoch 43 TRAIN:  13%|█▎        | 17/135 [00:06<00:29,  4.01it/s, epoch=43, grad_norm=100.800, loss=316.6933, lr=4.70e-04]
Epoch 43 TRAIN:  13%|█▎        | 17/135 [00:06<00:29,  4.01it/s, epoch=43, grad_norm=153.616, loss=361.9995, lr=4.71e-04]
Epoch 43 TRAIN:  13%|█▎        | 17/135 [00:06<00:29,  4.01it/s, epoch=43, grad_norm=134.933, loss=341.8644, lr=4.71e-04]
Epoch 43 TRAIN:  16%|█▌        | 21/135 [00:06<00:24,  4.58it/s, epoch=43, grad_norm=134.933, loss=341.8644, lr=4.71e-04]
Epoch 43 TRAIN:  16%|█▌        | 21/135 [00:06<00:24,  4.58it/s, epoch=43, grad_norm=164.857, loss=340.8100, lr=4.71e-04]
Epoch 43 TRAIN:  16%|█▌        | 21/135 [00:06<00:24,  4.58it/s, epoch=43, grad_norm=111.615, loss=330.7329, lr=4.71e-04]
Epoch 43 TRAIN:  16%|█▌        | 21/135 [00:07<00:24,  4.58it/s, epoch=43, grad_norm=76.136, loss=295.0797, lr=4.71e-04] 
Epoch 43 TRAIN:  16%|█▌        | 21/135 [00:07<00:24,  4.58it/s, epoch=43, grad_norm=73.073, loss=342.9662, lr=4.71e-04]
Epoch 43 TRAIN:  19%|█▊        | 25/135 [00:07<00:24,  4.58it/s, epoch=43, grad_norm=73.073, loss=342.9662, lr=4.71e-04]
Epoch 43 TRAIN:  19%|█▊        | 25/135 [00:07<00:24,  4.58it/s, epoch=43, grad_norm=89.065, loss=345.5489, lr=4.71e-04]
Epoch 43 TRAIN:  19%|█▊        | 25/135 [00:07<00:24,  4.58it/s, epoch=43, grad_norm=109.428, loss=304.3549, lr=4.71e-04]
Epoch 43 TRAIN:  19%|█▊        | 25/135 [00:07<00:24,  4.58it/s, epoch=43, grad_norm=144.696, loss=307.1532, lr=4.71e-04]
Epoch 43 TRAIN:  21%|██        | 28/135 [00:07<00:22,  4.83it/s, epoch=43, grad_norm=144.696, loss=307.1532, lr=4.71e-04]
Epoch 43 TRAIN:  21%|██        | 28/135 [00:08<00:22,  4.83it/s, epoch=43, grad_norm=194.351, loss=239.6900, lr=4.71e-04]
Epoch 43 TRAIN:  21%|██        | 28/135 [00:08<00:22,  4.83it/s, epoch=43, grad_norm=71.246, loss=301.9986, lr=4.71e-04] 
Epoch 43 TRAIN:  21%|██        | 28/135 [00:08<00:22,  4.83it/s, epoch=43, grad_norm=97.033, loss=281.7186, lr=4.71e-04]
Epoch 43 TRAIN:  21%|██        | 28/135 [00:08<00:22,  4.83it/s, epoch=43, grad_norm=124.916, loss=350.9389, lr=4.72e-04]
Epoch 43 TRAIN:  24%|██▎       | 32/135 [00:08<00:20,  5.12it/s, epoch=43, grad_norm=124.916, loss=350.9389, lr=4.72e-04]
Epoch 43 TRAIN:  24%|██▎       | 32/135 [00:08<00:20,  5.12it/s, epoch=43, grad_norm=94.565, loss=318.9355, lr=4.72e-04] 
Epoch 43 TRAIN:  24%|██▎       | 32/135 [00:09<00:20,  5.12it/s, epoch=43, grad_norm=143.910, loss=354.1368, lr=4.72e-04]
Epoch 43 TRAIN:  24%|██▎       | 32/135 [00:09<00:20,  5.12it/s, epoch=43, grad_norm=141.990, loss=347.1469, lr=4.72e-04]
Epoch 43 TRAIN:  26%|██▌       | 35/135 [00:09<00:19,  5.03it/s, epoch=43, grad_norm=141.990, loss=347.1469, lr=4.72e-04]
Epoch 43 TRAIN:  26%|██▌       | 35/135 [00:09<00:19,  5.03it/s, epoch=43, grad_norm=97.407, loss=311.2984, lr=4.72e-04] 
Epoch 43 TRAIN:  26%|██▌       | 35/135 [00:09<00:19,  5.03it/s, epoch=43, grad_norm=100.537, loss=274.1990, lr=4.72e-04]
Epoch 43 TRAIN:  26%|██▌       | 35/135 [00:09<00:19,  5.03it/s, epoch=43, grad_norm=99.271, loss=279.8000, lr=4.72e-04] 
Epoch 43 TRAIN:  28%|██▊       | 38/135 [00:09<00:19,  5.03it/s, epoch=43, grad_norm=99.271, loss=279.8000, lr=4.72e-04]
Epoch 43 TRAIN:  28%|██▊       | 38/135 [00:10<00:19,  5.03it/s, epoch=43, grad_norm=81.108, loss=321.6211, lr=4.72e-04]
Epoch 43 TRAIN:  28%|██▊       | 38/135 [00:10<00:19,  5.03it/s, epoch=43, grad_norm=76.220, loss=334.1398, lr=4.72e-04]
Epoch 43 TRAIN:  28%|██▊       | 38/135 [00:10<00:19,  5.03it/s, epoch=43, grad_norm=77.073, loss=317.7959, lr=4.72e-04]
Epoch 43 TRAIN:  30%|███       | 41/135 [00:10<00:18,  5.09it/s, epoch=43, grad_norm=77.073, loss=317.7959, lr=4.72e-04]
Epoch 43 TRAIN:  30%|███       | 41/135 [00:10<00:18,  5.09it/s, epoch=43, grad_norm=129.163, loss=278.8012, lr=4.72e-04]
Epoch 43 TRAIN:  30%|███       | 41/135 [00:10<00:18,  5.09it/s, epoch=43, grad_norm=105.769, loss=336.3544, lr=4.72e-04]
Epoch 43 TRAIN:  30%|███       | 41/135 [00:11<00:18,  5.09it/s, epoch=43, grad_norm=140.788, loss=275.7651, lr=4.72e-04]
Epoch 43 TRAIN:  33%|███▎      | 44/135 [00:11<00:17,  5.10it/s, epoch=43, grad_norm=140.788, loss=275.7651, lr=4.72e-04]
Epoch 43 TRAIN:  33%|███▎      | 44/135 [00:11<00:17,  5.10it/s, epoch=43, grad_norm=82.857, loss=323.3649, lr=4.73e-04] 
Epoch 43 TRAIN:  33%|███▎      | 44/135 [00:11<00:17,  5.10it/s, epoch=43, grad_norm=72.614, loss=298.5771, lr=4.73e-04]
Epoch 43 TRAIN:  33%|███▎      | 44/135 [00:11<00:17,  5.10it/s, epoch=43, grad_norm=77.584, loss=306.3864, lr=4.73e-04]
Epoch 43 TRAIN:  35%|███▍      | 47/135 [00:11<00:17,  4.92it/s, epoch=43, grad_norm=77.584, loss=306.3864, lr=4.73e-04]
Epoch 43 TRAIN:  35%|███▍      | 47/135 [00:11<00:17,  4.92it/s, epoch=43, grad_norm=78.483, loss=277.9292, lr=4.73e-04]
Epoch 43 TRAIN:  35%|███▍      | 47/135 [00:12<00:17,  4.92it/s, epoch=43, grad_norm=87.953, loss=323.2824, lr=4.73e-04]
Epoch 43 TRAIN:  35%|███▍      | 47/135 [00:12<00:17,  4.92it/s, epoch=43, grad_norm=115.808, loss=260.0609, lr=4.73e-04]
Epoch 43 TRAIN:  37%|███▋      | 50/135 [00:12<00:17,  4.81it/s, epoch=43, grad_norm=115.808, loss=260.0609, lr=4.73e-04]
Epoch 43 TRAIN:  37%|███▋      | 50/135 [00:12<00:17,  4.81it/s, epoch=43, grad_norm=85.143, loss=317.7740, lr=4.73e-04] 
Epoch 43 TRAIN:  37%|███▋      | 50/135 [00:12<00:17,  4.81it/s, epoch=43, grad_norm=84.234, loss=284.5226, lr=4.73e-04]
Epoch 43 TRAIN:  37%|███▋      | 50/135 [00:12<00:17,  4.81it/s, epoch=43, grad_norm=119.605, loss=336.4952, lr=4.73e-04]
Epoch 43 TRAIN:  39%|███▉      | 53/135 [00:12<00:16,  4.99it/s, epoch=43, grad_norm=119.605, loss=336.4952, lr=4.73e-04]
Epoch 43 TRAIN:  39%|███▉      | 53/135 [00:13<00:16,  4.99it/s, epoch=43, grad_norm=157.771, loss=342.0917, lr=4.73e-04]
Epoch 43 TRAIN:  39%|███▉      | 53/135 [00:13<00:16,  4.99it/s, epoch=43, grad_norm=113.933, loss=356.3011, lr=4.73e-04]
Epoch 43 TRAIN:  39%|███▉      | 53/135 [00:13<00:16,  4.99it/s, epoch=43, grad_norm=69.485, loss=288.0722, lr=4.73e-04] 
Epoch 43 TRAIN:  41%|████▏     | 56/135 [00:13<00:16,  4.82it/s, epoch=43, grad_norm=69.485, loss=288.0722, lr=4.73e-04]
Epoch 43 TRAIN:  41%|████▏     | 56/135 [00:13<00:16,  4.82it/s, epoch=43, grad_norm=69.871, loss=274.4897, lr=4.74e-04]
Epoch 43 TRAIN:  41%|████▏     | 56/135 [00:13<00:16,  4.82it/s, epoch=43, grad_norm=81.883, loss=315.4313, lr=4.74e-04]
Epoch 43 TRAIN:  41%|████▏     | 56/135 [00:14<00:16,  4.82it/s, epoch=43, grad_norm=106.422, loss=291.1901, lr=4.74e-04]
Epoch 43 TRAIN:  44%|████▎     | 59/135 [00:14<00:15,  4.84it/s, epoch=43, grad_norm=106.422, loss=291.1901, lr=4.74e-04]
Epoch 43 TRAIN:  44%|████▎     | 59/135 [00:14<00:15,  4.84it/s, epoch=43, grad_norm=101.927, loss=312.1329, lr=4.74e-04]
Epoch 43 TRAIN:  44%|████▎     | 59/135 [00:14<00:15,  4.84it/s, epoch=43, grad_norm=119.278, loss=277.9358, lr=4.74e-04]
Epoch 43 TRAIN:  44%|████▎     | 59/135 [00:14<00:15,  4.84it/s, epoch=43, grad_norm=70.081, loss=312.0545, lr=4.74e-04] 
Epoch 43 TRAIN:  46%|████▌     | 62/135 [00:14<00:15,  4.85it/s, epoch=43, grad_norm=70.081, loss=312.0545, lr=4.74e-04]
Epoch 43 TRAIN:  46%|████▌     | 62/135 [00:14<00:15,  4.85it/s, epoch=43, grad_norm=80.775, loss=299.3643, lr=4.74e-04]
Epoch 43 TRAIN:  46%|████▌     | 62/135 [00:15<00:15,  4.85it/s, epoch=43, grad_norm=91.335, loss=278.5661, lr=4.74e-04]
Epoch 43 TRAIN:  46%|████▌     | 62/135 [00:15<00:15,  4.85it/s, epoch=43, grad_norm=133.918, loss=326.6156, lr=4.74e-04]
Epoch 43 TRAIN:  48%|████▊     | 65/135 [00:15<00:13,  5.00it/s, epoch=43, grad_norm=133.918, loss=326.6156, lr=4.74e-04]
Epoch 43 TRAIN:  48%|████▊     | 65/135 [00:15<00:13,  5.00it/s, epoch=43, grad_norm=93.253, loss=311.2292, lr=4.74e-04] 
Epoch 43 TRAIN:  48%|████▊     | 65/135 [00:15<00:13,  5.00it/s, epoch=43, grad_norm=83.770, loss=320.8489, lr=4.74e-04]
Epoch 43 TRAIN:  48%|████▊     | 65/135 [00:15<00:13,  5.00it/s, epoch=43, grad_norm=106.281, loss=303.6942, lr=4.74e-04]
Epoch 43 TRAIN:  50%|█████     | 68/135 [00:15<00:13,  4.98it/s, epoch=43, grad_norm=106.281, loss=303.6942, lr=4.74e-04]
Epoch 43 TRAIN:  50%|█████     | 68/135 [00:16<00:13,  4.98it/s, epoch=43, grad_norm=108.683, loss=248.8966, lr=4.74e-04]
Epoch 43 TRAIN:  50%|█████     | 68/135 [00:16<00:13,  4.98it/s, epoch=43, grad_norm=185.012, loss=369.8521, lr=4.75e-04]
Epoch 43 TRAIN:  50%|█████     | 68/135 [00:16<00:13,  4.98it/s, epoch=43, grad_norm=96.516, loss=288.6226, lr=4.75e-04] 
Epoch 43 TRAIN:  53%|█████▎    | 71/135 [00:16<00:12,  5.06it/s, epoch=43, grad_norm=96.516, loss=288.6226, lr=4.75e-04]
Epoch 43 TRAIN:  53%|█████▎    | 71/135 [00:16<00:12,  5.06it/s, epoch=43, grad_norm=105.258, loss=294.5555, lr=4.75e-04]
Epoch 43 TRAIN:  53%|█████▎    | 71/135 [00:16<00:12,  5.06it/s, epoch=43, grad_norm=104.317, loss=281.9914, lr=4.75e-04]
Epoch 43 TRAIN:  53%|█████▎    | 71/135 [00:17<00:12,  5.06it/s, epoch=43, grad_norm=155.206, loss=310.5282, lr=4.75e-04]
Epoch 43 TRAIN:  55%|█████▍    | 74/135 [00:17<00:12,  4.92it/s, epoch=43, grad_norm=155.206, loss=310.5282, lr=4.75e-04]
Epoch 43 TRAIN:  55%|█████▍    | 74/135 [00:17<00:12,  4.92it/s, epoch=43, grad_norm=90.063, loss=305.9859, lr=4.75e-04] 
Epoch 43 TRAIN:  55%|█████▍    | 74/135 [00:17<00:12,  4.92it/s, epoch=43, grad_norm=123.374, loss=294.9330, lr=4.75e-04]
Epoch 43 TRAIN:  55%|█████▍    | 74/135 [00:17<00:12,  4.92it/s, epoch=43, grad_norm=120.842, loss=310.8713, lr=4.75e-04]
Epoch 43 TRAIN:  57%|█████▋    | 77/135 [00:17<00:12,  4.68it/s, epoch=43, grad_norm=120.842, loss=310.8713, lr=4.75e-04]
Epoch 43 TRAIN:  57%|█████▋    | 77/135 [00:18<00:12,  4.68it/s, epoch=43, grad_norm=151.191, loss=320.8228, lr=4.75e-04]
Epoch 43 TRAIN:  57%|█████▋    | 77/135 [00:18<00:12,  4.68it/s, epoch=43, grad_norm=193.730, loss=197.1442, lr=4.75e-04]
Epoch 43 TRAIN:  57%|█████▋    | 77/135 [00:18<00:12,  4.68it/s, epoch=43, grad_norm=100.757, loss=266.9756, lr=4.75e-04]
Epoch 43 TRAIN:  59%|█████▉    | 80/135 [00:18<00:11,  4.79it/s, epoch=43, grad_norm=100.757, loss=266.9756, lr=4.75e-04]
Epoch 43 TRAIN:  59%|█████▉    | 80/135 [00:18<00:11,  4.79it/s, epoch=43, grad_norm=101.027, loss=311.6121, lr=4.75e-04]
Epoch 43 TRAIN:  59%|█████▉    | 80/135 [00:18<00:11,  4.79it/s, epoch=43, grad_norm=127.173, loss=268.3554, lr=4.76e-04]
Epoch 43 TRAIN:  59%|█████▉    | 80/135 [00:19<00:11,  4.79it/s, epoch=43, grad_norm=94.527, loss=331.0010, lr=4.76e-04] 
Epoch 43 TRAIN:  61%|██████▏   | 83/135 [00:19<00:11,  4.65it/s, epoch=43, grad_norm=94.527, loss=331.0010, lr=4.76e-04]
Epoch 43 TRAIN:  61%|██████▏   | 83/135 [00:19<00:11,  4.65it/s, epoch=43, grad_norm=186.316, loss=183.7736, lr=4.76e-04]
Epoch 43 TRAIN:  61%|██████▏   | 83/135 [00:19<00:11,  4.65it/s, epoch=43, grad_norm=220.928, loss=338.5736, lr=4.76e-04]
Epoch 43 TRAIN:  61%|██████▏   | 83/135 [00:19<00:11,  4.65it/s, epoch=43, grad_norm=183.644, loss=301.2176, lr=4.76e-04]
Epoch 43 TRAIN:  64%|██████▎   | 86/135 [00:19<00:10,  4.60it/s, epoch=43, grad_norm=183.644, loss=301.2176, lr=4.76e-04]
Epoch 43 TRAIN:  64%|██████▎   | 86/135 [00:20<00:10,  4.60it/s, epoch=43, grad_norm=183.207, loss=319.8818, lr=4.76e-04]
Epoch 43 TRAIN:  64%|██████▎   | 86/135 [00:20<00:10,  4.60it/s, epoch=43, grad_norm=151.005, loss=343.9928, lr=4.76e-04]
Epoch 43 TRAIN:  64%|██████▎   | 86/135 [00:20<00:10,  4.60it/s, epoch=43, grad_norm=189.284, loss=346.7745, lr=4.76e-04]
Epoch 43 TRAIN:  66%|██████▌   | 89/135 [00:20<00:10,  4.58it/s, epoch=43, grad_norm=189.284, loss=346.7745, lr=4.76e-04]
Epoch 43 TRAIN:  66%|██████▌   | 89/135 [00:20<00:10,  4.58it/s, epoch=43, grad_norm=256.728, loss=208.4142, lr=4.76e-04]
Epoch 43 TRAIN:  66%|██████▌   | 89/135 [00:20<00:10,  4.58it/s, epoch=43, grad_norm=185.271, loss=296.7031, lr=4.76e-04]
Epoch 43 TRAIN:  66%|██████▌   | 89/135 [00:21<00:10,  4.58it/s, epoch=43, grad_norm=206.162, loss=282.0334, lr=4.76e-04]
Epoch 43 TRAIN:  68%|██████▊   | 92/135 [00:21<00:08,  4.79it/s, epoch=43, grad_norm=206.162, loss=282.0334, lr=4.76e-04]
Epoch 43 TRAIN:  68%|██████▊   | 92/135 [00:21<00:08,  4.79it/s, epoch=43, grad_norm=140.902, loss=257.3600, lr=4.76e-04]
Epoch 43 TRAIN:  68%|██████▊   | 92/135 [00:21<00:08,  4.79it/s, epoch=43, grad_norm=93.395, loss=250.9496, lr=4.76e-04] 
Epoch 43 TRAIN:  68%|██████▊   | 92/135 [00:21<00:08,  4.79it/s, epoch=43, grad_norm=84.177, loss=317.0733, lr=4.77e-04]
Epoch 43 TRAIN:  70%|███████   | 95/135 [00:21<00:08,  4.79it/s, epoch=43, grad_norm=84.177, loss=317.0733, lr=4.77e-04]
Epoch 43 TRAIN:  70%|███████   | 95/135 [00:21<00:08,  4.79it/s, epoch=43, grad_norm=111.142, loss=287.3610, lr=4.77e-04]
Epoch 43 TRAIN:  70%|███████   | 95/135 [00:22<00:08,  4.79it/s, epoch=43, grad_norm=226.228, loss=360.7070, lr=4.77e-04]
Epoch 43 TRAIN:  70%|███████   | 95/135 [00:22<00:08,  4.79it/s, epoch=43, grad_norm=107.419, loss=327.2749, lr=4.77e-04]
Epoch 43 TRAIN:  73%|███████▎  | 98/135 [00:22<00:07,  4.79it/s, epoch=43, grad_norm=107.419, loss=327.2749, lr=4.77e-04]
Epoch 43 TRAIN:  73%|███████▎  | 98/135 [00:22<00:07,  4.79it/s, epoch=43, grad_norm=91.818, loss=339.5397, lr=4.77e-04] 
Epoch 43 TRAIN:  73%|███████▎  | 98/135 [00:22<00:07,  4.79it/s, epoch=43, grad_norm=141.838, loss=239.3044, lr=4.77e-04]2026-04-16 20:13:53,515 DEBUG TRAIN | steps/sec 4.348| Batch 43/100 loss 239.304428 loss_att 211.095154 loss_ctc 305.126068 th_accuracy 0.315152 lr 4.7696e-04 grad_norm 141.837753 rank 0

Epoch 43 TRAIN:  73%|███████▎  | 98/135 [00:22<00:07,  4.79it/s, epoch=43, grad_norm=91.823, loss=260.0162, lr=4.77e-04] 
Epoch 43 TRAIN:  75%|███████▍  | 101/135 [00:22<00:07,  4.81it/s, epoch=43, grad_norm=91.823, loss=260.0162, lr=4.77e-04]
Epoch 43 TRAIN:  75%|███████▍  | 101/135 [00:23<00:07,  4.81it/s, epoch=43, grad_norm=196.794, loss=317.0179, lr=4.77e-04]
Epoch 43 TRAIN:  75%|███████▍  | 101/135 [00:23<00:07,  4.81it/s, epoch=43, grad_norm=92.396, loss=271.2134, lr=4.77e-04] 
Epoch 43 TRAIN:  75%|███████▍  | 101/135 [00:23<00:07,  4.81it/s, epoch=43, grad_norm=166.223, loss=331.7814, lr=4.77e-04]
Epoch 43 TRAIN:  77%|███████▋  | 104/135 [00:23<00:06,  4.64it/s, epoch=43, grad_norm=166.223, loss=331.7814, lr=4.77e-04]
Epoch 43 TRAIN:  77%|███████▋  | 104/135 [00:23<00:06,  4.64it/s, epoch=43, grad_norm=108.807, loss=286.1502, lr=4.77e-04]
Epoch 43 TRAIN:  77%|███████▋  | 104/135 [00:24<00:06,  4.64it/s, epoch=43, grad_norm=95.708, loss=301.5418, lr=4.77e-04] 
Epoch 43 TRAIN:  77%|███████▋  | 104/135 [00:24<00:06,  4.64it/s, epoch=43, grad_norm=115.504, loss=295.0299, lr=4.78e-04]
Epoch 43 TRAIN:  79%|███████▉  | 107/135 [00:24<00:06,  4.60it/s, epoch=43, grad_norm=115.504, loss=295.0299, lr=4.78e-04]
Epoch 43 TRAIN:  79%|███████▉  | 107/135 [00:24<00:06,  4.60it/s, epoch=43, grad_norm=122.712, loss=308.7001, lr=4.78e-04]
Epoch 43 TRAIN:  79%|███████▉  | 107/135 [00:24<00:06,  4.60it/s, epoch=43, grad_norm=87.457, loss=237.0118, lr=4.78e-04] 
Epoch 43 TRAIN:  79%|███████▉  | 107/135 [00:24<00:06,  4.60it/s, epoch=43, grad_norm=304.758, loss=403.7958, lr=4.78e-04]
Epoch 43 TRAIN:  81%|████████▏ | 110/135 [00:24<00:05,  4.54it/s, epoch=43, grad_norm=304.758, loss=403.7958, lr=4.78e-04]
Epoch 43 TRAIN:  81%|████████▏ | 110/135 [00:25<00:05,  4.54it/s, epoch=43, grad_norm=112.334, loss=269.8313, lr=4.78e-04]
Epoch 43 TRAIN:  81%|████████▏ | 110/135 [00:25<00:05,  4.54it/s, epoch=43, grad_norm=189.627, loss=293.0333, lr=4.78e-04]
Epoch 43 TRAIN:  81%|████████▏ | 110/135 [00:25<00:05,  4.54it/s, epoch=43, grad_norm=149.273, loss=309.4424, lr=4.78e-04]
Epoch 43 TRAIN:  84%|████████▎ | 113/135 [00:25<00:04,  4.63it/s, epoch=43, grad_norm=149.273, loss=309.4424, lr=4.78e-04]
Epoch 43 TRAIN:  84%|████████▎ | 113/135 [00:25<00:04,  4.63it/s, epoch=43, grad_norm=127.886, loss=367.5637, lr=4.78e-04]
Epoch 43 TRAIN:  84%|████████▎ | 113/135 [00:25<00:04,  4.63it/s, epoch=43, grad_norm=158.829, loss=357.1400, lr=4.78e-04]
Epoch 43 TRAIN:  84%|████████▎ | 113/135 [00:26<00:04,  4.63it/s, epoch=43, grad_norm=152.681, loss=334.1193, lr=4.78e-04]
Epoch 43 TRAIN:  84%|████████▎ | 113/135 [00:26<00:04,  4.63it/s, epoch=43, grad_norm=155.156, loss=280.2032, lr=4.78e-04]
Epoch 43 TRAIN:  87%|████████▋ | 117/135 [00:26<00:03,  4.96it/s, epoch=43, grad_norm=155.156, loss=280.2032, lr=4.78e-04]
Epoch 43 TRAIN:  87%|████████▋ | 117/135 [00:26<00:03,  4.96it/s, epoch=43, grad_norm=178.588, loss=349.4433, lr=4.78e-04]
Epoch 43 TRAIN:  87%|████████▋ | 117/135 [00:26<00:03,  4.96it/s, epoch=43, grad_norm=113.599, loss=267.5927, lr=4.78e-04]
Epoch 43 TRAIN:  87%|████████▋ | 117/135 [00:26<00:03,  4.96it/s, epoch=43, grad_norm=121.384, loss=311.8530, lr=4.79e-04]
Epoch 43 TRAIN:  89%|████████▉ | 120/135 [00:26<00:02,  5.04it/s, epoch=43, grad_norm=121.384, loss=311.8530, lr=4.79e-04]
Epoch 43 TRAIN:  89%|████████▉ | 120/135 [00:27<00:02,  5.04it/s, epoch=43, grad_norm=129.849, loss=345.4187, lr=4.79e-04]
Epoch 43 TRAIN:  89%|████████▉ | 120/135 [00:27<00:02,  5.04it/s, epoch=43, grad_norm=184.084, loss=387.4788, lr=4.79e-04]
Epoch 43 TRAIN:  89%|████████▉ | 120/135 [00:27<00:02,  5.04it/s, epoch=43, grad_norm=155.228, loss=371.1295, lr=4.79e-04]
Epoch 43 TRAIN:  91%|█████████ | 123/135 [00:27<00:02,  4.73it/s, epoch=43, grad_norm=155.228, loss=371.1295, lr=4.79e-04]
Epoch 43 TRAIN:  91%|█████████ | 123/135 [00:27<00:02,  4.73it/s, epoch=43, grad_norm=137.651, loss=358.4108, lr=4.79e-04]
Epoch 43 TRAIN:  91%|█████████ | 123/135 [00:28<00:02,  4.73it/s, epoch=43, grad_norm=91.816, loss=261.9052, lr=4.79e-04] 
Epoch 43 TRAIN:  91%|█████████ | 123/135 [00:28<00:02,  4.73it/s, epoch=43, grad_norm=94.123, loss=297.4167, lr=4.79e-04]
Epoch 43 TRAIN:  93%|█████████▎| 126/135 [00:28<00:01,  4.77it/s, epoch=43, grad_norm=94.123, loss=297.4167, lr=4.79e-04]
Epoch 43 TRAIN:  93%|█████████▎| 126/135 [00:28<00:01,  4.77it/s, epoch=43, grad_norm=129.407, loss=333.9017, lr=4.79e-04]
Epoch 43 TRAIN:  93%|█████████▎| 126/135 [00:28<00:01,  4.77it/s, epoch=43, grad_norm=158.785, loss=348.6837, lr=4.79e-04]
Epoch 43 TRAIN:  93%|█████████▎| 126/135 [00:28<00:01,  4.77it/s, epoch=43, grad_norm=106.472, loss=303.0949, lr=4.79e-04]
Epoch 43 TRAIN:  96%|█████████▌| 129/135 [00:28<00:01,  4.72it/s, epoch=43, grad_norm=106.472, loss=303.0949, lr=4.79e-04]
Epoch 43 TRAIN:  96%|█████████▌| 129/135 [00:29<00:01,  4.72it/s, epoch=43, grad_norm=111.142, loss=302.2054, lr=4.79e-04]
Epoch 43 TRAIN:  96%|█████████▌| 129/135 [00:29<00:01,  4.72it/s, epoch=43, grad_norm=150.082, loss=281.7028, lr=4.79e-04]
Epoch 43 TRAIN:  96%|█████████▌| 129/135 [00:29<00:01,  4.72it/s, epoch=43, grad_norm=117.250, loss=310.9469, lr=4.80e-04]
Epoch 43 TRAIN:  98%|█████████▊| 132/135 [00:29<00:00,  4.71it/s, epoch=43, grad_norm=117.250, loss=310.9469, lr=4.80e-04]
Epoch 43 TRAIN:  98%|█████████▊| 132/135 [00:29<00:00,  4.71it/s, epoch=43, grad_norm=130.655, loss=290.7966, lr=4.80e-04]
Epoch 43 TRAIN:  98%|█████████▊| 132/135 [00:29<00:00,  4.71it/s, epoch=43, grad_norm=154.403, loss=341.1399, lr=4.80e-04]
Epoch 43 TRAIN:  98%|█████████▊| 132/135 [00:30<00:00,  4.71it/s, epoch=43, grad_norm=136.539, loss=286.0933, lr=4.80e-04]
Epoch 43 TRAIN: 100%|██████████| 135/135 [00:30<00:00,  4.76it/s, epoch=43, grad_norm=136.539, loss=286.0933, lr=4.80e-04]
Epoch 43 TRAIN: 100%|██████████| 135/135 [00:30<00:00,  4.48it/s, epoch=43, grad_norm=136.539, loss=286.0933, lr=4.80e-04]

CV:   0%|          | 0/6 [00:00<?, ?it/s]/home/rookie/Documents/batuka/projects/wenet/wenet/wenet/utils/train_utils.py:689: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  amp_autocast(enabled=dtype is not None,

CV:  17%|█▋        | 1/6 [00:00<00:03,  1.32it/s]
                                                 
2026-04-16 20:14:01,707 INFO Epoch 43 Step 5997 CV info lr 4.7976e-04 cv_loss 521.4422219016335 rank 0 acc 0.28122467299302417
2026-04-16 20:14:01,709 INFO [Rank 0] Checkpoint: save to checkpoint exp/conformer_mongolian/epoch_43.pt
2026-04-16 20:14:01,844 INFO Epoch 44 Step 5997 TRAIN info lr 4.7976e-04 rank 0
2026-04-16 20:14:01,858 INFO using accumulate grad, new batch size is 1 times larger than before

Epoch 44 TRAIN:   0%|          | 0/135 [00:00<?, ?it/s]
Epoch 44 TRAIN:   0%|          | 0/135 [00:04<?, ?it/s, epoch=44, grad_norm=335.288, loss=299.9702, lr=4.80e-04]
Epoch 44 TRAIN:   1%|          | 1/135 [00:04<09:15,  4.14s/it, epoch=44, grad_norm=335.288, loss=299.9702, lr=4.80e-04]
Epoch 44 TRAIN:   1%|          | 1/135 [00:04<09:15,  4.14s/it, epoch=44, grad_norm=471.964, loss=411.7660, lr=4.80e-04]
Epoch 44 TRAIN:   1%|          | 1/135 [00:04<09:15,  4.14s/it, epoch=44, grad_norm=307.015, loss=296.1499, lr=4.80e-04]
Epoch 44 TRAIN:   1%|          | 1/135 [00:04<09:15,  4.14s/it, epoch=44, grad_norm=386.486, loss=381.9048, lr=4.80e-04]
Epoch 44 TRAIN:   1%|          | 1/135 [00:04<09:15,  4.14s/it, epoch=44, grad_norm=374.112, loss=356.7903, lr=4.80e-04]
Epoch 44 TRAIN:   1%|          | 1/135 [00:04<09:15,  4.14s/it, epoch=44, grad_norm=443.105, loss=429.8383, lr=4.80e-04]
Epoch 44 TRAIN:   1%|          | 1/135 [00:04<09:15,  4.14s/it, epoch=44, grad_norm=170.149, loss=286.0471, lr=4.80e-04]
Epoch 44 TRAIN:   1%|          | 1/135 [00:04<09:15,  4.14s/it, epoch=44, grad_norm=254.956, loss=340.9764, lr=4.80e-04]
Epoch 44 TRAIN:   1%|          | 1/135 [00:04<09:15,  4.14s/it, epoch=44, grad_norm=128.209, loss=297.0596, lr=4.80e-04]
Epoch 44 TRAIN:   7%|▋         | 9/135 [00:04<00:50,  2.48it/s, epoch=44, grad_norm=128.209, loss=297.0596, lr=4.80e-04]
Epoch 44 TRAIN:   7%|▋         | 9/135 [00:04<00:50,  2.48it/s, epoch=44, grad_norm=78.728, loss=287.2944, lr=4.81e-04] 
Epoch 44 TRAIN:   7%|▋         | 9/135 [00:05<00:50,  2.48it/s, epoch=44, grad_norm=76.345, loss=327.1938, lr=4.81e-04]
Epoch 44 TRAIN:   7%|▋         | 9/135 [00:05<00:50,  2.48it/s, epoch=44, grad_norm=117.891, loss=317.7851, lr=4.81e-04]
Epoch 44 TRAIN:   7%|▋         | 9/135 [00:05<00:50,  2.48it/s, epoch=44, grad_norm=91.691, loss=342.4597, lr=4.81e-04] 
Epoch 44 TRAIN:  10%|▉         | 13/135 [00:05<00:37,  3.24it/s, epoch=44, grad_norm=91.691, loss=342.4597, lr=4.81e-04]
Epoch 44 TRAIN:  10%|▉         | 13/135 [00:05<00:37,  3.24it/s, epoch=44, grad_norm=137.480, loss=297.1480, lr=4.81e-04]
Epoch 44 TRAIN:  10%|▉         | 13/135 [00:05<00:37,  3.24it/s, epoch=44, grad_norm=79.845, loss=314.7166, lr=4.81e-04] 
Epoch 44 TRAIN:  10%|▉         | 13/135 [00:05<00:37,  3.24it/s, epoch=44, grad_norm=69.314, loss=309.0958, lr=4.81e-04]
Epoch 44 TRAIN:  10%|▉         | 13/135 [00:06<00:37,  3.24it/s, epoch=44, grad_norm=73.336, loss=302.5804, lr=4.81e-04]
Epoch 44 TRAIN:  13%|█▎        | 17/135 [00:06<00:30,  3.93it/s, epoch=44, grad_norm=73.336, loss=302.5804, lr=4.81e-04]
Epoch 44 TRAIN:  13%|█▎        | 17/135 [00:06<00:30,  3.93it/s, epoch=44, grad_norm=60.575, loss=284.7308, lr=4.81e-04]
Epoch 44 TRAIN:  13%|█▎        | 17/135 [00:06<00:30,  3.93it/s, epoch=44, grad_norm=83.271, loss=310.5728, lr=4.81e-04]
Epoch 44 TRAIN:  13%|█▎        | 17/135 [00:06<00:30,  3.93it/s, epoch=44, grad_norm=96.212, loss=339.1233, lr=4.81e-04]
Epoch 44 TRAIN:  13%|█▎        | 17/135 [00:06<00:30,  3.93it/s, epoch=44, grad_norm=66.597, loss=306.2802, lr=4.81e-04]
Epoch 44 TRAIN:  16%|█▌        | 21/135 [00:06<00:26,  4.35it/s, epoch=44, grad_norm=66.597, loss=306.2802, lr=4.81e-04]
Epoch 44 TRAIN:  16%|█▌        | 21/135 [00:06<00:26,  4.35it/s, epoch=44, grad_norm=79.941, loss=332.6934, lr=4.82e-04]
Epoch 44 TRAIN:  16%|█▌        | 21/135 [00:07<00:26,  4.35it/s, epoch=44, grad_norm=63.785, loss=307.8218, lr=4.82e-04]
Epoch 44 TRAIN:  16%|█▌        | 21/135 [00:07<00:26,  4.35it/s, epoch=44, grad_norm=79.143, loss=362.5089, lr=4.82e-04]
Epoch 44 TRAIN:  16%|█▌        | 21/135 [00:07<00:26,  4.35it/s, epoch=44, grad_norm=164.623, loss=267.8998, lr=4.82e-04]
Epoch 44 TRAIN:  19%|█▊        | 25/135 [00:07<00:23,  4.61it/s, epoch=44, grad_norm=164.623, loss=267.8998, lr=4.82e-04]
Epoch 44 TRAIN:  19%|█▊        | 25/135 [00:07<00:23,  4.61it/s, epoch=44, grad_norm=119.545, loss=307.0204, lr=4.82e-04]
Epoch 44 TRAIN:  19%|█▊        | 25/135 [00:07<00:23,  4.61it/s, epoch=44, grad_norm=80.007, loss=307.5009, lr=4.82e-04] 
Epoch 44 TRAIN:  19%|█▊        | 25/135 [00:08<00:23,  4.61it/s, epoch=44, grad_norm=130.261, loss=289.5525, lr=4.82e-04]
Epoch 44 TRAIN:  21%|██        | 28/135 [00:08<00:23,  4.60it/s, epoch=44, grad_norm=130.261, loss=289.5525, lr=4.82e-04]
Epoch 44 TRAIN:  21%|██        | 28/135 [00:08<00:23,  4.60it/s, epoch=44, grad_norm=98.098, loss=298.6045, lr=4.82e-04] 
Epoch 44 TRAIN:  21%|██        | 28/135 [00:08<00:23,  4.60it/s, epoch=44, grad_norm=70.602, loss=289.4213, lr=4.82e-04]
Epoch 44 TRAIN:  21%|██        | 28/135 [00:08<00:23,  4.60it/s, epoch=44, grad_norm=171.498, loss=368.1125, lr=4.82e-04]
Epoch 44 TRAIN:  23%|██▎       | 31/135 [00:08<00:21,  4.79it/s, epoch=44, grad_norm=171.498, loss=368.1125, lr=4.82e-04]
Epoch 44 TRAIN:  23%|██▎       | 31/135 [00:08<00:21,  4.79it/s, epoch=44, grad_norm=166.903, loss=349.2194, lr=4.82e-04]
Epoch 44 TRAIN:  23%|██▎       | 31/135 [00:09<00:21,  4.79it/s, epoch=44, grad_norm=58.368, loss=268.8252, lr=4.82e-04] 
Epoch 44 TRAIN:  23%|██▎       | 31/135 [00:09<00:21,  4.79it/s, epoch=44, grad_norm=116.649, loss=327.7595, lr=4.82e-04]
Epoch 44 TRAIN:  25%|██▌       | 34/135 [00:09<00:21,  4.76it/s, epoch=44, grad_norm=116.649, loss=327.7595, lr=4.82e-04]
Epoch 44 TRAIN:  25%|██▌       | 34/135 [00:09<00:21,  4.76it/s, epoch=44, grad_norm=101.838, loss=323.3688, lr=4.83e-04]
Epoch 44 TRAIN:  25%|██▌       | 34/135 [00:09<00:21,  4.76it/s, epoch=44, grad_norm=98.806, loss=350.9702, lr=4.83e-04] 
Epoch 44 TRAIN:  25%|██▌       | 34/135 [00:09<00:21,  4.76it/s, epoch=44, grad_norm=88.665, loss=289.9720, lr=4.83e-04]
Epoch 44 TRAIN:  27%|██▋       | 37/135 [00:09<00:19,  5.03it/s, epoch=44, grad_norm=88.665, loss=289.9720, lr=4.83e-04]
Epoch 44 TRAIN:  27%|██▋       | 37/135 [00:10<00:19,  5.03it/s, epoch=44, grad_norm=105.966, loss=289.7406, lr=4.83e-04]
Epoch 44 TRAIN:  27%|██▋       | 37/135 [00:10<00:19,  5.03it/s, epoch=44, grad_norm=107.865, loss=301.9197, lr=4.83e-04]
Epoch 44 TRAIN:  27%|██▋       | 37/135 [00:10<00:19,  5.03it/s, epoch=44, grad_norm=81.106, loss=299.4153, lr=4.83e-04] 
Epoch 44 TRAIN:  30%|██▉       | 40/135 [00:10<00:19,  4.94it/s, epoch=44, grad_norm=81.106, loss=299.4153, lr=4.83e-04]
Epoch 44 TRAIN:  30%|██▉       | 40/135 [00:10<00:19,  4.94it/s, epoch=44, grad_norm=85.976, loss=301.4198, lr=4.83e-04]
Epoch 44 TRAIN:  30%|██▉       | 40/135 [00:10<00:19,  4.94it/s, epoch=44, grad_norm=113.484, loss=289.1342, lr=4.83e-04]
Epoch 44 TRAIN:  30%|██▉       | 40/135 [00:11<00:19,  4.94it/s, epoch=44, grad_norm=132.476, loss=365.7079, lr=4.83e-04]
Epoch 44 TRAIN:  32%|███▏      | 43/135 [00:11<00:18,  5.00it/s, epoch=44, grad_norm=132.476, loss=365.7079, lr=4.83e-04]
Epoch 44 TRAIN:  32%|███▏      | 43/135 [00:11<00:18,  5.00it/s, epoch=44, grad_norm=60.795, loss=334.7380, lr=4.83e-04] 
Epoch 44 TRAIN:  32%|███▏      | 43/135 [00:11<00:18,  5.00it/s, epoch=44, grad_norm=118.060, loss=348.7327, lr=4.83e-04]
Epoch 44 TRAIN:  32%|███▏      | 43/135 [00:11<00:18,  5.00it/s, epoch=44, grad_norm=91.368, loss=320.6974, lr=4.83e-04] 
Epoch 44 TRAIN:  34%|███▍      | 46/135 [00:11<00:18,  4.84it/s, epoch=44, grad_norm=91.368, loss=320.6974, lr=4.83e-04]
Epoch 44 TRAIN:  34%|███▍      | 46/135 [00:11<00:18,  4.84it/s, epoch=44, grad_norm=75.301, loss=317.1846, lr=4.84e-04]
Epoch 44 TRAIN:  34%|███▍      | 46/135 [00:12<00:18,  4.84it/s, epoch=44, grad_norm=85.852, loss=293.5244, lr=4.84e-04]
Epoch 44 TRAIN:  34%|███▍      | 46/135 [00:12<00:18,  4.84it/s, epoch=44, grad_norm=97.916, loss=321.2885, lr=4.84e-04]
Epoch 44 TRAIN:  36%|███▋      | 49/135 [00:12<00:17,  4.89it/s, epoch=44, grad_norm=97.916, loss=321.2885, lr=4.84e-04]
Epoch 44 TRAIN:  36%|███▋      | 49/135 [00:12<00:17,  4.89it/s, epoch=44, grad_norm=147.852, loss=266.9557, lr=4.84e-04]
Epoch 44 TRAIN:  36%|███▋      | 49/135 [00:12<00:17,  4.89it/s, epoch=44, grad_norm=150.080, loss=297.0642, lr=4.84e-04]
Epoch 44 TRAIN:  36%|███▋      | 49/135 [00:12<00:17,  4.89it/s, epoch=44, grad_norm=172.658, loss=257.2694, lr=4.84e-04]
Epoch 44 TRAIN:  39%|███▊      | 52/135 [00:12<00:16,  4.93it/s, epoch=44, grad_norm=172.658, loss=257.2694, lr=4.84e-04]
Epoch 44 TRAIN:  39%|███▊      | 52/135 [00:13<00:16,  4.93it/s, epoch=44, grad_norm=81.675, loss=310.7703, lr=4.84e-04] 
Epoch 44 TRAIN:  39%|███▊      | 52/135 [00:13<00:16,  4.93it/s, epoch=44, grad_norm=68.594, loss=286.3940, lr=4.84e-04]
Epoch 44 TRAIN:  39%|███▊      | 52/135 [00:13<00:16,  4.93it/s, epoch=44, grad_norm=121.841, loss=321.6250, lr=4.84e-04]
Epoch 44 TRAIN:  41%|████      | 55/135 [00:13<00:16,  4.85it/s, epoch=44, grad_norm=121.841, loss=321.6250, lr=4.84e-04]
Epoch 44 TRAIN:  41%|████      | 55/135 [00:13<00:16,  4.85it/s, epoch=44, grad_norm=132.158, loss=321.0932, lr=4.84e-04]
Epoch 44 TRAIN:  41%|████      | 55/135 [00:14<00:16,  4.85it/s, epoch=44, grad_norm=91.428, loss=301.6111, lr=4.84e-04] 
Epoch 44 TRAIN:  41%|████      | 55/135 [00:14<00:16,  4.85it/s, epoch=44, grad_norm=102.017, loss=322.6974, lr=4.84e-04]
Epoch 44 TRAIN:  43%|████▎     | 58/135 [00:14<00:15,  4.83it/s, epoch=44, grad_norm=102.017, loss=322.6974, lr=4.84e-04]
Epoch 44 TRAIN:  43%|████▎     | 58/135 [00:14<00:15,  4.83it/s, epoch=44, grad_norm=76.132, loss=281.8773, lr=4.84e-04] 
Epoch 44 TRAIN:  43%|████▎     | 58/135 [00:14<00:15,  4.83it/s, epoch=44, grad_norm=136.715, loss=295.2776, lr=4.85e-04]
Epoch 44 TRAIN:  43%|████▎     | 58/135 [00:14<00:15,  4.83it/s, epoch=44, grad_norm=92.305, loss=295.9178, lr=4.85e-04] 
Epoch 44 TRAIN:  45%|████▌     | 61/135 [00:14<00:15,  4.79it/s, epoch=44, grad_norm=92.305, loss=295.9178, lr=4.85e-04]
Epoch 44 TRAIN:  45%|████▌     | 61/135 [00:15<00:15,  4.79it/s, epoch=44, grad_norm=117.312, loss=317.2961, lr=4.85e-04]
Epoch 44 TRAIN:  45%|████▌     | 61/135 [00:15<00:15,  4.79it/s, epoch=44, grad_norm=134.546, loss=317.7889, lr=4.85e-04]
Epoch 44 TRAIN:  45%|████▌     | 61/135 [00:15<00:15,  4.79it/s, epoch=44, grad_norm=75.557, loss=288.5472, lr=4.85e-04] 
Epoch 44 TRAIN:  47%|████▋     | 64/135 [00:15<00:14,  4.78it/s, epoch=44, grad_norm=75.557, loss=288.5472, lr=4.85e-04]
Epoch 44 TRAIN:  47%|████▋     | 64/135 [00:15<00:14,  4.78it/s, epoch=44, grad_norm=118.269, loss=368.7383, lr=4.85e-04]
Epoch 44 TRAIN:  47%|████▋     | 64/135 [00:15<00:14,  4.78it/s, epoch=44, grad_norm=84.625, loss=319.8135, lr=4.85e-04] 
Epoch 44 TRAIN:  47%|████▋     | 64/135 [00:16<00:14,  4.78it/s, epoch=44, grad_norm=68.949, loss=333.5222, lr=4.85e-04]
Epoch 44 TRAIN:  50%|████▉     | 67/135 [00:16<00:13,  4.97it/s, epoch=44, grad_norm=68.949, loss=333.5222, lr=4.85e-04]
Epoch 44 TRAIN:  50%|████▉     | 67/135 [00:16<00:13,  4.97it/s, epoch=44, grad_norm=95.497, loss=374.4594, lr=4.85e-04]
Epoch 44 TRAIN:  50%|████▉     | 67/135 [00:16<00:13,  4.97it/s, epoch=44, grad_norm=146.214, loss=278.1960, lr=4.85e-04]
Epoch 44 TRAIN:  50%|████▉     | 67/135 [00:16<00:13,  4.97it/s, epoch=44, grad_norm=196.620, loss=258.6784, lr=4.85e-04]
Epoch 44 TRAIN:  52%|█████▏    | 70/135 [00:16<00:13,  4.86it/s, epoch=44, grad_norm=196.620, loss=258.6784, lr=4.85e-04]
Epoch 44 TRAIN:  52%|█████▏    | 70/135 [00:16<00:13,  4.86it/s, epoch=44, grad_norm=104.036, loss=297.6908, lr=4.85e-04]
Epoch 44 TRAIN:  52%|█████▏    | 70/135 [00:17<00:13,  4.86it/s, epoch=44, grad_norm=145.068, loss=366.8281, lr=4.86e-04]
Epoch 44 TRAIN:  52%|█████▏    | 70/135 [00:17<00:13,  4.86it/s, epoch=44, grad_norm=92.404, loss=321.3956, lr=4.86e-04] 
Epoch 44 TRAIN:  54%|█████▍    | 73/135 [00:17<00:12,  4.81it/s, epoch=44, grad_norm=92.404, loss=321.3956, lr=4.86e-04]
Epoch 44 TRAIN:  54%|█████▍    | 73/135 [00:17<00:12,  4.81it/s, epoch=44, grad_norm=120.301, loss=312.2397, lr=4.86e-04]
Epoch 44 TRAIN:  54%|█████▍    | 73/135 [00:17<00:12,  4.81it/s, epoch=44, grad_norm=127.262, loss=254.0753, lr=4.86e-04]
Epoch 44 TRAIN:  54%|█████▍    | 73/135 [00:18<00:12,  4.81it/s, epoch=44, grad_norm=211.330, loss=258.6248, lr=4.86e-04]
Epoch 44 TRAIN:  56%|█████▋    | 76/135 [00:18<00:12,  4.71it/s, epoch=44, grad_norm=211.330, loss=258.6248, lr=4.86e-04]
Epoch 44 TRAIN:  56%|█████▋    | 76/135 [00:18<00:12,  4.71it/s, epoch=44, grad_norm=101.571, loss=271.3597, lr=4.86e-04]
Epoch 44 TRAIN:  56%|█████▋    | 76/135 [00:18<00:12,  4.71it/s, epoch=44, grad_norm=76.698, loss=316.2170, lr=4.86e-04] 
Epoch 44 TRAIN:  56%|█████▋    | 76/135 [00:18<00:12,  4.71it/s, epoch=44, grad_norm=130.947, loss=348.6712, lr=4.86e-04]
Epoch 44 TRAIN:  59%|█████▊    | 79/135 [00:18<00:11,  4.73it/s, epoch=44, grad_norm=130.947, loss=348.6712, lr=4.86e-04]
Epoch 44 TRAIN:  59%|█████▊    | 79/135 [00:18<00:11,  4.73it/s, epoch=44, grad_norm=82.008, loss=277.3580, lr=4.86e-04] 
Epoch 44 TRAIN:  59%|█████▊    | 79/135 [00:19<00:11,  4.73it/s, epoch=44, grad_norm=136.132, loss=325.3542, lr=4.86e-04]
Epoch 44 TRAIN:  59%|█████▊    | 79/135 [00:19<00:11,  4.73it/s, epoch=44, grad_norm=62.544, loss=280.9626, lr=4.86e-04] 
Epoch 44 TRAIN:  61%|██████    | 82/135 [00:19<00:11,  4.75it/s, epoch=44, grad_norm=62.544, loss=280.9626, lr=4.86e-04]
Epoch 44 TRAIN:  61%|██████    | 82/135 [00:19<00:11,  4.75it/s, epoch=44, grad_norm=153.479, loss=359.9812, lr=4.86e-04]
Epoch 44 TRAIN:  61%|██████    | 82/135 [00:19<00:11,  4.75it/s, epoch=44, grad_norm=145.144, loss=310.0736, lr=4.86e-04]
Epoch 44 TRAIN:  61%|██████    | 82/135 [00:20<00:11,  4.75it/s, epoch=44, grad_norm=141.688, loss=341.6237, lr=4.87e-04]
Epoch 44 TRAIN:  63%|██████▎   | 85/135 [00:20<00:11,  4.41it/s, epoch=44, grad_norm=141.688, loss=341.6237, lr=4.87e-04]
Epoch 44 TRAIN:  63%|██████▎   | 85/135 [00:20<00:11,  4.41it/s, epoch=44, grad_norm=211.292, loss=314.4873, lr=4.87e-04]
Epoch 44 TRAIN:  63%|██████▎   | 85/135 [00:20<00:11,  4.41it/s, epoch=44, grad_norm=220.401, loss=279.0490, lr=4.87e-04]
Epoch 44 TRAIN:  63%|██████▎   | 85/135 [00:20<00:11,  4.41it/s, epoch=44, grad_norm=132.271, loss=345.5223, lr=4.87e-04]
Epoch 44 TRAIN:  65%|██████▌   | 88/135 [00:20<00:10,  4.61it/s, epoch=44, grad_norm=132.271, loss=345.5223, lr=4.87e-04]
Epoch 44 TRAIN:  65%|██████▌   | 88/135 [00:20<00:10,  4.61it/s, epoch=44, grad_norm=128.473, loss=311.9421, lr=4.87e-04]
Epoch 44 TRAIN:  65%|██████▌   | 88/135 [00:21<00:10,  4.61it/s, epoch=44, grad_norm=107.452, loss=299.3024, lr=4.87e-04]
Epoch 44 TRAIN:  65%|██████▌   | 88/135 [00:21<00:10,  4.61it/s, epoch=44, grad_norm=86.116, loss=275.9409, lr=4.87e-04] 
Epoch 44 TRAIN:  67%|██████▋   | 91/135 [00:21<00:09,  4.54it/s, epoch=44, grad_norm=86.116, loss=275.9409, lr=4.87e-04]
Epoch 44 TRAIN:  67%|██████▋   | 91/135 [00:21<00:09,  4.54it/s, epoch=44, grad_norm=168.750, loss=286.7561, lr=4.87e-04]
Epoch 44 TRAIN:  67%|██████▋   | 91/135 [00:21<00:09,  4.54it/s, epoch=44, grad_norm=159.276, loss=263.5055, lr=4.87e-04]
Epoch 44 TRAIN:  67%|██████▋   | 91/135 [00:21<00:09,  4.54it/s, epoch=44, grad_norm=114.113, loss=312.8517, lr=4.87e-04]
Epoch 44 TRAIN:  70%|██████▉   | 94/135 [00:21<00:08,  4.66it/s, epoch=44, grad_norm=114.113, loss=312.8517, lr=4.87e-04]
Epoch 44 TRAIN:  70%|██████▉   | 94/135 [00:22<00:08,  4.66it/s, epoch=44, grad_norm=120.857, loss=300.3116, lr=4.87e-04]
Epoch 44 TRAIN:  70%|██████▉   | 94/135 [00:22<00:08,  4.66it/s, epoch=44, grad_norm=135.951, loss=285.0319, lr=4.87e-04]
Epoch 44 TRAIN:  70%|██████▉   | 94/135 [00:22<00:08,  4.66it/s, epoch=44, grad_norm=87.004, loss=211.0614, lr=4.88e-04] 
Epoch 44 TRAIN:  72%|███████▏  | 97/135 [00:22<00:08,  4.46it/s, epoch=44, grad_norm=87.004, loss=211.0614, lr=4.88e-04]
Epoch 44 TRAIN:  72%|███████▏  | 97/135 [00:22<00:08,  4.46it/s, epoch=44, grad_norm=194.919, loss=310.3406, lr=4.88e-04]
Epoch 44 TRAIN:  72%|███████▏  | 97/135 [00:23<00:08,  4.46it/s, epoch=44, grad_norm=81.920, loss=209.8496, lr=4.88e-04] 
Epoch 44 TRAIN:  72%|███████▏  | 97/135 [00:23<00:08,  4.46it/s, epoch=44, grad_norm=105.504, loss=289.9163, lr=4.88e-04]2026-04-16 20:14:24,997 DEBUG TRAIN | steps/sec 4.288| Batch 44/100 loss 289.916260 loss_att 260.113129 loss_ctc 359.456909 th_accuracy 0.270588 lr 4.8776e-04 grad_norm 105.503899 rank 0

Epoch 44 TRAIN:  72%|███████▏  | 97/135 [00:23<00:08,  4.46it/s, epoch=44, grad_norm=93.249, loss=256.5733, lr=4.88e-04] 
Epoch 44 TRAIN:  75%|███████▍  | 101/135 [00:23<00:07,  4.54it/s, epoch=44, grad_norm=93.249, loss=256.5733, lr=4.88e-04]
Epoch 44 TRAIN:  75%|███████▍  | 101/135 [00:23<00:07,  4.54it/s, epoch=44, grad_norm=112.478, loss=313.2353, lr=4.88e-04]
Epoch 44 TRAIN:  75%|███████▍  | 101/135 [00:23<00:07,  4.54it/s, epoch=44, grad_norm=94.688, loss=225.0313, lr=4.88e-04] 
Epoch 44 TRAIN:  75%|███████▍  | 101/135 [00:24<00:07,  4.54it/s, epoch=44, grad_norm=233.324, loss=329.0994, lr=4.88e-04]
Epoch 44 TRAIN:  77%|███████▋  | 104/135 [00:24<00:06,  4.79it/s, epoch=44, grad_norm=233.324, loss=329.0994, lr=4.88e-04]
Epoch 44 TRAIN:  77%|███████▋  | 104/135 [00:24<00:06,  4.79it/s, epoch=44, grad_norm=75.544, loss=245.1769, lr=4.88e-04] 
Epoch 44 TRAIN:  77%|███████▋  | 104/135 [00:24<00:06,  4.79it/s, epoch=44, grad_norm=175.259, loss=294.6169, lr=4.88e-04]
Epoch 44 TRAIN:  77%|███████▋  | 104/135 [00:24<00:06,  4.79it/s, epoch=44, grad_norm=169.482, loss=303.0494, lr=4.88e-04]
Epoch 44 TRAIN:  79%|███████▉  | 107/135 [00:24<00:05,  4.79it/s, epoch=44, grad_norm=169.482, loss=303.0494, lr=4.88e-04]
Epoch 44 TRAIN:  79%|███████▉  | 107/135 [00:24<00:05,  4.79it/s, epoch=44, grad_norm=131.562, loss=357.7197, lr=4.88e-04]
Epoch 44 TRAIN:  79%|███████▉  | 107/135 [00:25<00:05,  4.79it/s, epoch=44, grad_norm=117.095, loss=353.0851, lr=4.88e-04]
Epoch 44 TRAIN:  79%|███████▉  | 107/135 [00:25<00:05,  4.79it/s, epoch=44, grad_norm=311.580, loss=345.9385, lr=4.89e-04]
Epoch 44 TRAIN:  81%|████████▏ | 110/135 [00:25<00:05,  4.53it/s, epoch=44, grad_norm=311.580, loss=345.9385, lr=4.89e-04]
Epoch 44 TRAIN:  81%|████████▏ | 110/135 [00:25<00:05,  4.53it/s, epoch=44, grad_norm=123.308, loss=326.8960, lr=4.89e-04]
Epoch 44 TRAIN:  81%|████████▏ | 110/135 [00:25<00:05,  4.53it/s, epoch=44, grad_norm=101.969, loss=267.4189, lr=4.89e-04]
Epoch 44 TRAIN:  81%|████████▏ | 110/135 [00:26<00:05,  4.53it/s, epoch=44, grad_norm=210.502, loss=265.2165, lr=4.89e-04]
Epoch 44 TRAIN:  84%|████████▎ | 113/135 [00:26<00:04,  4.65it/s, epoch=44, grad_norm=210.502, loss=265.2165, lr=4.89e-04]
Epoch 44 TRAIN:  84%|████████▎ | 113/135 [00:26<00:04,  4.65it/s, epoch=44, grad_norm=138.672, loss=236.0581, lr=4.89e-04]
Epoch 44 TRAIN:  84%|████████▎ | 113/135 [00:26<00:04,  4.65it/s, epoch=44, grad_norm=102.201, loss=277.7135, lr=4.89e-04]
Epoch 44 TRAIN:  84%|████████▎ | 113/135 [00:26<00:04,  4.65it/s, epoch=44, grad_norm=151.546, loss=321.3986, lr=4.89e-04]
Epoch 44 TRAIN:  86%|████████▌ | 116/135 [00:26<00:04,  4.61it/s, epoch=44, grad_norm=151.546, loss=321.3986, lr=4.89e-04]
Epoch 44 TRAIN:  86%|████████▌ | 116/135 [00:26<00:04,  4.61it/s, epoch=44, grad_norm=102.209, loss=239.9415, lr=4.89e-04]
Epoch 44 TRAIN:  86%|████████▌ | 116/135 [00:27<00:04,  4.61it/s, epoch=44, grad_norm=112.689, loss=248.3593, lr=4.89e-04]
Epoch 44 TRAIN:  86%|████████▌ | 116/135 [00:27<00:04,  4.61it/s, epoch=44, grad_norm=113.553, loss=267.4048, lr=4.89e-04]
Epoch 44 TRAIN:  88%|████████▊ | 119/135 [00:27<00:03,  4.69it/s, epoch=44, grad_norm=113.553, loss=267.4048, lr=4.89e-04]
Epoch 44 TRAIN:  88%|████████▊ | 119/135 [00:27<00:03,  4.69it/s, epoch=44, grad_norm=105.234, loss=278.0572, lr=4.89e-04]
Epoch 44 TRAIN:  88%|████████▊ | 119/135 [00:27<00:03,  4.69it/s, epoch=44, grad_norm=101.264, loss=275.2848, lr=4.89e-04]
Epoch 44 TRAIN:  88%|████████▊ | 119/135 [00:27<00:03,  4.69it/s, epoch=44, grad_norm=98.119, loss=280.0040, lr=4.90e-04] 
Epoch 44 TRAIN:  90%|█████████ | 122/135 [00:27<00:02,  4.68it/s, epoch=44, grad_norm=98.119, loss=280.0040, lr=4.90e-04]
Epoch 44 TRAIN:  90%|█████████ | 122/135 [00:28<00:02,  4.68it/s, epoch=44, grad_norm=259.480, loss=229.2507, lr=4.90e-04]
Epoch 44 TRAIN:  90%|█████████ | 122/135 [00:28<00:02,  4.68it/s, epoch=44, grad_norm=111.704, loss=314.7314, lr=4.90e-04]
Epoch 44 TRAIN:  90%|█████████ | 122/135 [00:28<00:02,  4.68it/s, epoch=44, grad_norm=124.378, loss=273.3602, lr=4.90e-04]
Epoch 44 TRAIN:  93%|█████████▎| 125/135 [00:28<00:02,  4.85it/s, epoch=44, grad_norm=124.378, loss=273.3602, lr=4.90e-04]
Epoch 44 TRAIN:  93%|█████████▎| 125/135 [00:28<00:02,  4.85it/s, epoch=44, grad_norm=167.622, loss=330.4796, lr=4.90e-04]
Epoch 44 TRAIN:  93%|█████████▎| 125/135 [00:28<00:02,  4.85it/s, epoch=44, grad_norm=200.750, loss=375.7930, lr=4.90e-04]
Epoch 44 TRAIN:  93%|█████████▎| 125/135 [00:29<00:02,  4.85it/s, epoch=44, grad_norm=149.286, loss=349.4509, lr=4.90e-04]
Epoch 44 TRAIN:  95%|█████████▍| 128/135 [00:29<00:01,  4.88it/s, epoch=44, grad_norm=149.286, loss=349.4509, lr=4.90e-04]
Epoch 44 TRAIN:  95%|█████████▍| 128/135 [00:29<00:01,  4.88it/s, epoch=44, grad_norm=213.286, loss=402.8480, lr=4.90e-04]
Epoch 44 TRAIN:  95%|█████████▍| 128/135 [00:29<00:01,  4.88it/s, epoch=44, grad_norm=218.288, loss=362.7306, lr=4.90e-04]
Epoch 44 TRAIN:  95%|█████████▍| 128/135 [00:29<00:01,  4.88it/s, epoch=44, grad_norm=324.917, loss=299.6216, lr=4.90e-04]
Epoch 44 TRAIN:  97%|█████████▋| 131/135 [00:29<00:00,  4.90it/s, epoch=44, grad_norm=324.917, loss=299.6216, lr=4.90e-04]
Epoch 44 TRAIN:  97%|█████████▋| 131/135 [00:29<00:00,  4.90it/s, epoch=44, grad_norm=175.903, loss=363.4903, lr=4.90e-04]
Epoch 44 TRAIN:  97%|█████████▋| 131/135 [00:30<00:00,  4.90it/s, epoch=44, grad_norm=169.883, loss=378.2900, lr=4.90e-04]
Epoch 44 TRAIN:  97%|█████████▋| 131/135 [00:30<00:00,  4.90it/s, epoch=44, grad_norm=148.927, loss=400.9711, lr=4.90e-04]
Epoch 44 TRAIN:  99%|█████████▉| 134/135 [00:30<00:00,  5.08it/s, epoch=44, grad_norm=148.927, loss=400.9711, lr=4.90e-04]
Epoch 44 TRAIN:  99%|█████████▉| 134/135 [00:30<00:00,  5.08it/s, epoch=44, grad_norm=165.304, loss=367.3113, lr=4.91e-04]
Epoch 44 TRAIN:  99%|█████████▉| 134/135 [00:30<00:00,  5.08it/s, epoch=44, grad_norm=201.852, loss=350.9251, lr=4.91e-04]
Epoch 44 TRAIN: 136it [00:30,  4.44it/s, epoch=44, grad_norm=201.852, loss=350.9251, lr=4.91e-04]                         

CV:   0%|          | 0/6 [00:00<?, ?it/s]/home/rookie/Documents/batuka/projects/wenet/wenet/wenet/utils/train_utils.py:689: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  amp_autocast(enabled=dtype is not None,

CV:  17%|█▋        | 1/6 [00:00<00:03,  1.33it/s]
                                                 
2026-04-16 20:14:33,333 INFO Epoch 44 Step 6133 CV info lr 4.9064e-04 cv_loss 498.3528775301847 rank 0 acc 0.30134738981723785
2026-04-16 20:14:33,335 INFO [Rank 0] Checkpoint: save to checkpoint exp/conformer_mongolian/epoch_44.pt
2026-04-16 20:14:33,462 INFO Epoch 45 Step 6133 TRAIN info lr 4.9064e-04 rank 0
2026-04-16 20:14:33,482 INFO using accumulate grad, new batch size is 1 times larger than before

Epoch 45 TRAIN:   0%|          | 0/136 [00:00<?, ?it/s]
Epoch 45 TRAIN:   0%|          | 0/136 [00:03<?, ?it/s, epoch=45, grad_norm=433.236, loss=369.3219, lr=4.91e-04]
Epoch 45 TRAIN:   1%|          | 1/136 [00:03<08:30,  3.78s/it, epoch=45, grad_norm=433.236, loss=369.3219, lr=4.91e-04]
Epoch 45 TRAIN:   1%|          | 1/136 [00:03<08:30,  3.78s/it, epoch=45, grad_norm=589.561, loss=512.7096, lr=4.91e-04]
Epoch 45 TRAIN:   1%|          | 1/136 [00:04<08:30,  3.78s/it, epoch=45, grad_norm=288.414, loss=284.0058, lr=4.91e-04]
Epoch 45 TRAIN:   1%|          | 1/136 [00:04<08:30,  3.78s/it, epoch=45, grad_norm=480.066, loss=387.7130, lr=4.91e-04]
Epoch 45 TRAIN:   1%|          | 1/136 [00:04<08:30,  3.78s/it, epoch=45, grad_norm=306.148, loss=329.4003, lr=4.91e-04]
Epoch 45 TRAIN:   1%|          | 1/136 [00:04<08:30,  3.78s/it, epoch=45, grad_norm=297.660, loss=315.3766, lr=4.91e-04]
Epoch 45 TRAIN:   1%|          | 1/136 [00:04<08:30,  3.78s/it, epoch=45, grad_norm=306.701, loss=377.0778, lr=4.91e-04]
Epoch 45 TRAIN:   5%|▌         | 7/136 [00:04<01:01,  2.10it/s, epoch=45, grad_norm=306.701, loss=377.0778, lr=4.91e-04]
Epoch 45 TRAIN:   5%|▌         | 7/136 [00:04<01:01,  2.10it/s, epoch=45, grad_norm=159.779, loss=323.6869, lr=4.91e-04]
Epoch 45 TRAIN:   5%|▌         | 7/136 [00:04<01:01,  2.10it/s, epoch=45, grad_norm=82.426, loss=286.8528, lr=4.91e-04] 
Epoch 45 TRAIN:   5%|▌         | 7/136 [00:04<01:01,  2.10it/s, epoch=45, grad_norm=89.162, loss=300.6560, lr=4.91e-04]
Epoch 45 TRAIN:   5%|▌         | 7/136 [00:04<01:01,  2.10it/s, epoch=45, grad_norm=83.969, loss=321.9803, lr=4.92e-04]
Epoch 45 TRAIN:   8%|▊         | 11/136 [00:04<00:39,  3.16it/s, epoch=45, grad_norm=83.969, loss=321.9803, lr=4.92e-04]
Epoch 45 TRAIN:   8%|▊         | 11/136 [00:05<00:39,  3.16it/s, epoch=45, grad_norm=78.392, loss=337.2115, lr=4.92e-04]
Epoch 45 TRAIN:   8%|▊         | 11/136 [00:05<00:39,  3.16it/s, epoch=45, grad_norm=128.922, loss=340.9698, lr=4.92e-04]
Epoch 45 TRAIN:   8%|▊         | 11/136 [00:05<00:39,  3.16it/s, epoch=45, grad_norm=163.795, loss=317.0643, lr=4.92e-04]
Epoch 45 TRAIN:   8%|▊         | 11/136 [00:05<00:39,  3.16it/s, epoch=45, grad_norm=219.940, loss=259.0047, lr=4.92e-04]
Epoch 45 TRAIN:  11%|█         | 15/136 [00:05<00:30,  3.93it/s, epoch=45, grad_norm=219.940, loss=259.0047, lr=4.92e-04]
Epoch 45 TRAIN:  11%|█         | 15/136 [00:05<00:30,  3.93it/s, epoch=45, grad_norm=112.767, loss=309.2582, lr=4.92e-04]
Epoch 45 TRAIN:  11%|█         | 15/136 [00:05<00:30,  3.93it/s, epoch=45, grad_norm=99.587, loss=343.5285, lr=4.92e-04] 
Epoch 45 TRAIN:  11%|█         | 15/136 [00:05<00:30,  3.93it/s, epoch=45, grad_norm=92.040, loss=290.8339, lr=4.92e-04]
Epoch 45 TRAIN:  11%|█         | 15/136 [00:06<00:30,  3.93it/s, epoch=45, grad_norm=89.120, loss=337.7297, lr=4.92e-04]
Epoch 45 TRAIN:  14%|█▍        | 19/136 [00:06<00:25,  4.51it/s, epoch=45, grad_norm=89.120, loss=337.7297, lr=4.92e-04]
Epoch 45 TRAIN:  14%|█▍        | 19/136 [00:06<00:25,  4.51it/s, epoch=45, grad_norm=74.097, loss=316.1453, lr=4.92e-04]
Epoch 45 TRAIN:  14%|█▍        | 19/136 [00:06<00:25,  4.51it/s, epoch=45, grad_norm=126.570, loss=341.6180, lr=4.92e-04]
Epoch 45 TRAIN:  14%|█▍        | 19/136 [00:06<00:25,  4.51it/s, epoch=45, grad_norm=85.619, loss=331.9131, lr=4.92e-04] 
Epoch 45 TRAIN:  14%|█▍        | 19/136 [00:06<00:25,  4.51it/s, epoch=45, grad_norm=63.553, loss=293.2942, lr=4.92e-04]
Epoch 45 TRAIN:  17%|█▋        | 23/136 [00:06<00:23,  4.81it/s, epoch=45, grad_norm=63.553, loss=293.2942, lr=4.92e-04]
Epoch 45 TRAIN:  17%|█▋        | 23/136 [00:07<00:23,  4.81it/s, epoch=45, grad_norm=88.850, loss=324.5515, lr=4.93e-04]
Epoch 45 TRAIN:  17%|█▋        | 23/136 [00:07<00:23,  4.81it/s, epoch=45, grad_norm=93.819, loss=281.9446, lr=4.93e-04]
Epoch 45 TRAIN:  17%|█▋        | 23/136 [00:07<00:23,  4.81it/s, epoch=45, grad_norm=72.337, loss=286.1001, lr=4.93e-04]
Epoch 45 TRAIN:  17%|█▋        | 23/136 [00:07<00:23,  4.81it/s, epoch=45, grad_norm=109.281, loss=363.9254, lr=4.93e-04]
Epoch 45 TRAIN:  20%|█▉        | 27/136 [00:07<00:21,  5.01it/s, epoch=45, grad_norm=109.281, loss=363.9254, lr=4.93e-04]
Epoch 45 TRAIN:  20%|█▉        | 27/136 [00:07<00:21,  5.01it/s, epoch=45, grad_norm=73.755, loss=317.2297, lr=4.93e-04] 
Epoch 45 TRAIN:  20%|█▉        | 27/136 [00:07<00:21,  5.01it/s, epoch=45, grad_norm=135.770, loss=277.4073, lr=4.93e-04]
Epoch 45 TRAIN:  20%|█▉        | 27/136 [00:08<00:21,  5.01it/s, epoch=45, grad_norm=96.590, loss=289.6896, lr=4.93e-04] 
Epoch 45 TRAIN:  22%|██▏       | 30/136 [00:08<00:20,  5.05it/s, epoch=45, grad_norm=96.590, loss=289.6896, lr=4.93e-04]
Epoch 45 TRAIN:  22%|██▏       | 30/136 [00:08<00:20,  5.05it/s, epoch=45, grad_norm=90.256, loss=320.8589, lr=4.93e-04]
Epoch 45 TRAIN:  22%|██▏       | 30/136 [00:08<00:20,  5.05it/s, epoch=45, grad_norm=128.103, loss=328.4790, lr=4.93e-04]
Epoch 45 TRAIN:  22%|██▏       | 30/136 [00:08<00:20,  5.05it/s, epoch=45, grad_norm=101.540, loss=300.4645, lr=4.93e-04]
Epoch 45 TRAIN:  24%|██▍       | 33/136 [00:08<00:20,  5.05it/s, epoch=45, grad_norm=101.540, loss=300.4645, lr=4.93e-04]
Epoch 45 TRAIN:  24%|██▍       | 33/136 [00:08<00:20,  5.05it/s, epoch=45, grad_norm=86.957, loss=284.5826, lr=4.93e-04] 
Epoch 45 TRAIN:  24%|██▍       | 33/136 [00:09<00:20,  5.05it/s, epoch=45, grad_norm=106.071, loss=306.7215, lr=4.93e-04]
Epoch 45 TRAIN:  24%|██▍       | 33/136 [00:09<00:20,  5.05it/s, epoch=45, grad_norm=111.604, loss=278.1385, lr=4.94e-04]
Epoch 45 TRAIN:  26%|██▋       | 36/136 [00:09<00:19,  5.09it/s, epoch=45, grad_norm=111.604, loss=278.1385, lr=4.94e-04]
Epoch 45 TRAIN:  26%|██▋       | 36/136 [00:09<00:19,  5.09it/s, epoch=45, grad_norm=91.749, loss=260.7794, lr=4.94e-04] 
Epoch 45 TRAIN:  26%|██▋       | 36/136 [00:09<00:19,  5.09it/s, epoch=45, grad_norm=118.039, loss=302.7801, lr=4.94e-04]
Epoch 45 TRAIN:  26%|██▋       | 36/136 [00:09<00:19,  5.09it/s, epoch=45, grad_norm=113.089, loss=291.7093, lr=4.94e-04]
Epoch 45 TRAIN:  29%|██▊       | 39/136 [00:09<00:19,  5.10it/s, epoch=45, grad_norm=113.089, loss=291.7093, lr=4.94e-04]
Epoch 45 TRAIN:  29%|██▊       | 39/136 [00:10<00:19,  5.10it/s, epoch=45, grad_norm=159.629, loss=317.4204, lr=4.94e-04]
Epoch 45 TRAIN:  29%|██▊       | 39/136 [00:10<00:19,  5.10it/s, epoch=45, grad_norm=84.625, loss=307.3029, lr=4.94e-04] 
Epoch 45 TRAIN:  29%|██▊       | 39/136 [00:10<00:19,  5.10it/s, epoch=45, grad_norm=73.743, loss=303.0730, lr=4.94e-04]
Epoch 45 TRAIN:  31%|███       | 42/136 [00:10<00:18,  5.21it/s, epoch=45, grad_norm=73.743, loss=303.0730, lr=4.94e-04]
Epoch 45 TRAIN:  31%|███       | 42/136 [00:10<00:18,  5.21it/s, epoch=45, grad_norm=97.168, loss=321.6429, lr=4.94e-04]
Epoch 45 TRAIN:  31%|███       | 42/136 [00:10<00:18,  5.21it/s, epoch=45, grad_norm=146.029, loss=255.1596, lr=4.94e-04]
Epoch 45 TRAIN:  31%|███       | 42/136 [00:11<00:18,  5.21it/s, epoch=45, grad_norm=175.732, loss=355.7321, lr=4.94e-04]
Epoch 45 TRAIN:  33%|███▎      | 45/136 [00:11<00:18,  4.80it/s, epoch=45, grad_norm=175.732, loss=355.7321, lr=4.94e-04]
Epoch 45 TRAIN:  33%|███▎      | 45/136 [00:11<00:18,  4.80it/s, epoch=45, grad_norm=119.252, loss=290.9925, lr=4.94e-04]
Epoch 45 TRAIN:  33%|███▎      | 45/136 [00:11<00:18,  4.80it/s, epoch=45, grad_norm=65.276, loss=325.8838, lr=4.94e-04] 
Epoch 45 TRAIN:  33%|███▎      | 45/136 [00:11<00:18,  4.80it/s, epoch=45, grad_norm=89.588, loss=330.5457, lr=4.94e-04]
Epoch 45 TRAIN:  35%|███▌      | 48/136 [00:11<00:18,  4.85it/s, epoch=45, grad_norm=89.588, loss=330.5457, lr=4.94e-04]
Epoch 45 TRAIN:  35%|███▌      | 48/136 [00:12<00:18,  4.85it/s, epoch=45, grad_norm=99.902, loss=296.6653, lr=4.95e-04]
Epoch 45 TRAIN:  35%|███▌      | 48/136 [00:12<00:18,  4.85it/s, epoch=45, grad_norm=135.970, loss=289.9867, lr=4.95e-04]
Epoch 45 TRAIN:  35%|███▌      | 48/136 [00:12<00:18,  4.85it/s, epoch=45, grad_norm=107.375, loss=283.2515, lr=4.95e-04]
Epoch 45 TRAIN:  38%|███▊      | 51/136 [00:12<00:18,  4.61it/s, epoch=45, grad_norm=107.375, loss=283.2515, lr=4.95e-04]
Epoch 45 TRAIN:  38%|███▊      | 51/136 [00:12<00:18,  4.61it/s, epoch=45, grad_norm=82.344, loss=291.9387, lr=4.95e-04] 
Epoch 45 TRAIN:  38%|███▊      | 51/136 [00:12<00:18,  4.61it/s, epoch=45, grad_norm=95.176, loss=297.5737, lr=4.95e-04]
Epoch 45 TRAIN:  38%|███▊      | 51/136 [00:13<00:18,  4.61it/s, epoch=45, grad_norm=102.625, loss=299.3592, lr=4.95e-04]
Epoch 45 TRAIN:  40%|███▉      | 54/136 [00:13<00:17,  4.64it/s, epoch=45, grad_norm=102.625, loss=299.3592, lr=4.95e-04]
Epoch 45 TRAIN:  40%|███▉      | 54/136 [00:13<00:17,  4.64it/s, epoch=45, grad_norm=99.950, loss=256.5534, lr=4.95e-04] 
Epoch 45 TRAIN:  40%|███▉      | 54/136 [00:13<00:17,  4.64it/s, epoch=45, grad_norm=156.107, loss=311.1719, lr=4.95e-04]
Epoch 45 TRAIN:  40%|███▉      | 54/136 [00:13<00:17,  4.64it/s, epoch=45, grad_norm=227.086, loss=371.1365, lr=4.95e-04]
Epoch 45 TRAIN:  42%|████▏     | 57/136 [00:13<00:16,  4.77it/s, epoch=45, grad_norm=227.086, loss=371.1365, lr=4.95e-04]
Epoch 45 TRAIN:  42%|████▏     | 57/136 [00:13<00:16,  4.77it/s, epoch=45, grad_norm=151.777, loss=321.3743, lr=4.95e-04]
Epoch 45 TRAIN:  42%|████▏     | 57/136 [00:14<00:16,  4.77it/s, epoch=45, grad_norm=173.417, loss=366.5795, lr=4.95e-04]
Epoch 45 TRAIN:  42%|████▏     | 57/136 [00:14<00:16,  4.77it/s, epoch=45, grad_norm=76.848, loss=328.0811, lr=4.95e-04] 
Epoch 45 TRAIN:  44%|████▍     | 60/136 [00:14<00:15,  4.82it/s, epoch=45, grad_norm=76.848, loss=328.0811, lr=4.95e-04]
Epoch 45 TRAIN:  44%|████▍     | 60/136 [00:14<00:15,  4.82it/s, epoch=45, grad_norm=108.077, loss=290.4445, lr=4.96e-04]
Epoch 45 TRAIN:  44%|████▍     | 60/136 [00:14<00:15,  4.82it/s, epoch=45, grad_norm=80.606, loss=349.5485, lr=4.96e-04] 
Epoch 45 TRAIN:  44%|████▍     | 60/136 [00:14<00:15,  4.82it/s, epoch=45, grad_norm=103.986, loss=290.7713, lr=4.96e-04]
Epoch 45 TRAIN:  46%|████▋     | 63/136 [00:14<00:14,  4.96it/s, epoch=45, grad_norm=103.986, loss=290.7713, lr=4.96e-04]
Epoch 45 TRAIN:  46%|████▋     | 63/136 [00:15<00:14,  4.96it/s, epoch=45, grad_norm=93.383, loss=299.1550, lr=4.96e-04] 
Epoch 45 TRAIN:  46%|████▋     | 63/136 [00:15<00:14,  4.96it/s, epoch=45, grad_norm=112.971, loss=300.6321, lr=4.96e-04]
Epoch 45 TRAIN:  46%|████▋     | 63/136 [00:15<00:14,  4.96it/s, epoch=45, grad_norm=151.055, loss=284.7755, lr=4.96e-04]
Epoch 45 TRAIN:  49%|████▊     | 66/136 [00:15<00:14,  4.92it/s, epoch=45, grad_norm=151.055, loss=284.7755, lr=4.96e-04]
Epoch 45 TRAIN:  49%|████▊     | 66/136 [00:15<00:14,  4.92it/s, epoch=45, grad_norm=89.944, loss=317.6140, lr=4.96e-04] 
Epoch 45 TRAIN:  49%|████▊     | 66/136 [00:16<00:14,  4.92it/s, epoch=45, grad_norm=90.298, loss=340.3290, lr=4.96e-04]
Epoch 45 TRAIN:  49%|████▊     | 66/136 [00:16<00:14,  4.92it/s, epoch=45, grad_norm=85.969, loss=341.2610, lr=4.96e-04]
Epoch 45 TRAIN:  51%|█████     | 69/136 [00:16<00:13,  4.82it/s, epoch=45, grad_norm=85.969, loss=341.2610, lr=4.96e-04]
Epoch 45 TRAIN:  51%|█████     | 69/136 [00:16<00:13,  4.82it/s, epoch=45, grad_norm=91.579, loss=334.3827, lr=4.96e-04]
Epoch 45 TRAIN:  51%|█████     | 69/136 [00:16<00:13,  4.82it/s, epoch=45, grad_norm=82.831, loss=314.3222, lr=4.96e-04]
Epoch 45 TRAIN:  51%|█████     | 69/136 [00:16<00:13,  4.82it/s, epoch=45, grad_norm=69.600, loss=271.1329, lr=4.96e-04]
Epoch 45 TRAIN:  53%|█████▎    | 72/136 [00:16<00:13,  4.81it/s, epoch=45, grad_norm=69.600, loss=271.1329, lr=4.96e-04]
Epoch 45 TRAIN:  53%|█████▎    | 72/136 [00:17<00:13,  4.81it/s, epoch=45, grad_norm=99.613, loss=283.8215, lr=4.96e-04]
Epoch 45 TRAIN:  53%|█████▎    | 72/136 [00:17<00:13,  4.81it/s, epoch=45, grad_norm=75.970, loss=314.3797, lr=4.97e-04]
Epoch 45 TRAIN:  53%|█████▎    | 72/136 [00:17<00:13,  4.81it/s, epoch=45, grad_norm=99.165, loss=316.5339, lr=4.97e-04]
Epoch 45 TRAIN:  55%|█████▌    | 75/136 [00:17<00:12,  4.70it/s, epoch=45, grad_norm=99.165, loss=316.5339, lr=4.97e-04]
Epoch 45 TRAIN:  55%|█████▌    | 75/136 [00:17<00:12,  4.70it/s, epoch=45, grad_norm=141.020, loss=347.5577, lr=4.97e-04]
Epoch 45 TRAIN:  55%|█████▌    | 75/136 [00:17<00:12,  4.70it/s, epoch=45, grad_norm=127.794, loss=346.3311, lr=4.97e-04]
Epoch 45 TRAIN:  55%|█████▌    | 75/136 [00:18<00:12,  4.70it/s, epoch=45, grad_norm=78.041, loss=307.1937, lr=4.97e-04] 
Epoch 45 TRAIN:  57%|█████▋    | 78/136 [00:18<00:12,  4.69it/s, epoch=45, grad_norm=78.041, loss=307.1937, lr=4.97e-04]
Epoch 45 TRAIN:  57%|█████▋    | 78/136 [00:18<00:12,  4.69it/s, epoch=45, grad_norm=118.522, loss=349.6151, lr=4.97e-04]
Epoch 45 TRAIN:  57%|█████▋    | 78/136 [00:18<00:12,  4.69it/s, epoch=45, grad_norm=102.663, loss=287.1877, lr=4.97e-04]
Epoch 45 TRAIN:  57%|█████▋    | 78/136 [00:18<00:12,  4.69it/s, epoch=45, grad_norm=93.361, loss=340.8237, lr=4.97e-04] 
Epoch 45 TRAIN:  60%|█████▉    | 81/136 [00:18<00:11,  4.72it/s, epoch=45, grad_norm=93.361, loss=340.8237, lr=4.97e-04]
Epoch 45 TRAIN:  60%|█████▉    | 81/136 [00:18<00:11,  4.72it/s, epoch=45, grad_norm=144.029, loss=311.2943, lr=4.97e-04]
Epoch 45 TRAIN:  60%|█████▉    | 81/136 [00:19<00:11,  4.72it/s, epoch=45, grad_norm=87.292, loss=329.8392, lr=4.97e-04] 
Epoch 45 TRAIN:  60%|█████▉    | 81/136 [00:19<00:11,  4.72it/s, epoch=45, grad_norm=226.894, loss=303.3950, lr=4.97e-04]
Epoch 45 TRAIN:  62%|██████▏   | 84/136 [00:19<00:11,  4.63it/s, epoch=45, grad_norm=226.894, loss=303.3950, lr=4.97e-04]
Epoch 45 TRAIN:  62%|██████▏   | 84/136 [00:19<00:11,  4.63it/s, epoch=45, grad_norm=117.086, loss=317.3201, lr=4.97e-04]
Epoch 45 TRAIN:  62%|██████▏   | 84/136 [00:19<00:11,  4.63it/s, epoch=45, grad_norm=120.277, loss=275.4158, lr=4.98e-04]
Epoch 45 TRAIN:  62%|██████▏   | 84/136 [00:20<00:11,  4.63it/s, epoch=45, grad_norm=114.447, loss=283.6504, lr=4.98e-04]
Epoch 45 TRAIN:  64%|██████▍   | 87/136 [00:20<00:10,  4.63it/s, epoch=45, grad_norm=114.447, loss=283.6504, lr=4.98e-04]
Epoch 45 TRAIN:  64%|██████▍   | 87/136 [00:20<00:10,  4.63it/s, epoch=45, grad_norm=145.430, loss=305.0660, lr=4.98e-04]
Epoch 45 TRAIN:  64%|██████▍   | 87/136 [00:20<00:10,  4.63it/s, epoch=45, grad_norm=99.668, loss=219.7238, lr=4.98e-04] 
Epoch 45 TRAIN:  64%|██████▍   | 87/136 [00:20<00:10,  4.63it/s, epoch=45, grad_norm=187.100, loss=319.9517, lr=4.98e-04]
Epoch 45 TRAIN:  66%|██████▌   | 90/136 [00:20<00:10,  4.57it/s, epoch=45, grad_norm=187.100, loss=319.9517, lr=4.98e-04]
Epoch 45 TRAIN:  66%|██████▌   | 90/136 [00:20<00:10,  4.57it/s, epoch=45, grad_norm=148.596, loss=193.8627, lr=4.98e-04]
Epoch 45 TRAIN:  66%|██████▌   | 90/136 [00:21<00:10,  4.57it/s, epoch=45, grad_norm=106.402, loss=261.8668, lr=4.98e-04]
Epoch 45 TRAIN:  66%|██████▌   | 90/136 [00:21<00:10,  4.57it/s, epoch=45, grad_norm=151.611, loss=304.5445, lr=4.98e-04]
Epoch 45 TRAIN:  68%|██████▊   | 93/136 [00:21<00:09,  4.58it/s, epoch=45, grad_norm=151.611, loss=304.5445, lr=4.98e-04]
Epoch 45 TRAIN:  68%|██████▊   | 93/136 [00:21<00:09,  4.58it/s, epoch=45, grad_norm=92.004, loss=253.3655, lr=4.98e-04] 
Epoch 45 TRAIN:  68%|██████▊   | 93/136 [00:21<00:09,  4.58it/s, epoch=45, grad_norm=203.937, loss=359.3951, lr=4.98e-04]
Epoch 45 TRAIN:  68%|██████▊   | 93/136 [00:22<00:09,  4.58it/s, epoch=45, grad_norm=126.153, loss=281.2824, lr=4.98e-04]
Epoch 45 TRAIN:  71%|███████   | 96/136 [00:22<00:08,  4.63it/s, epoch=45, grad_norm=126.153, loss=281.2824, lr=4.98e-04]
Epoch 45 TRAIN:  71%|███████   | 96/136 [00:22<00:08,  4.63it/s, epoch=45, grad_norm=147.262, loss=312.4490, lr=4.98e-04]
Epoch 45 TRAIN:  71%|███████   | 96/136 [00:22<00:08,  4.63it/s, epoch=45, grad_norm=156.561, loss=224.8775, lr=4.98e-04]
Epoch 45 TRAIN:  71%|███████   | 96/136 [00:22<00:08,  4.63it/s, epoch=45, grad_norm=82.639, loss=208.4255, lr=4.99e-04] 
Epoch 45 TRAIN:  73%|███████▎  | 99/136 [00:22<00:07,  4.63it/s, epoch=45, grad_norm=82.639, loss=208.4255, lr=4.99e-04]
Epoch 45 TRAIN:  73%|███████▎  | 99/136 [00:22<00:07,  4.63it/s, epoch=45, grad_norm=125.398, loss=305.4542, lr=4.99e-04]2026-04-16 20:14:56,428 DEBUG TRAIN | steps/sec 4.327| Batch 45/100 loss 305.454193 loss_att 269.345795 loss_ctc 389.707153 th_accuracy 0.292566 lr 4.9864e-04 grad_norm 125.398376 rank 0

Epoch 45 TRAIN:  73%|███████▎  | 99/136 [00:23<00:07,  4.63it/s, epoch=45, grad_norm=98.776, loss=244.1622, lr=4.99e-04] 
Epoch 45 TRAIN:  73%|███████▎  | 99/136 [00:23<00:07,  4.63it/s, epoch=45, grad_norm=199.467, loss=344.5791, lr=4.99e-04]
Epoch 45 TRAIN:  75%|███████▌  | 102/136 [00:23<00:07,  4.67it/s, epoch=45, grad_norm=199.467, loss=344.5791, lr=4.99e-04]
Epoch 45 TRAIN:  75%|███████▌  | 102/136 [00:23<00:07,  4.67it/s, epoch=45, grad_norm=112.995, loss=254.2315, lr=4.99e-04]
Epoch 45 TRAIN:  75%|███████▌  | 102/136 [00:23<00:07,  4.67it/s, epoch=45, grad_norm=110.764, loss=307.1929, lr=4.99e-04]
Epoch 45 TRAIN:  75%|███████▌  | 102/136 [00:23<00:07,  4.67it/s, epoch=45, grad_norm=270.737, loss=325.1829, lr=4.99e-04]
Epoch 45 TRAIN:  77%|███████▋  | 105/136 [00:23<00:06,  4.75it/s, epoch=45, grad_norm=270.737, loss=325.1829, lr=4.99e-04]
Epoch 45 TRAIN:  77%|███████▋  | 105/136 [00:24<00:06,  4.75it/s, epoch=45, grad_norm=116.815, loss=301.5433, lr=4.99e-04]
Epoch 45 TRAIN:  77%|███████▋  | 105/136 [00:24<00:06,  4.75it/s, epoch=45, grad_norm=239.067, loss=188.6039, lr=4.99e-04]
Epoch 45 TRAIN:  77%|███████▋  | 105/136 [00:24<00:06,  4.75it/s, epoch=45, grad_norm=142.132, loss=282.4911, lr=4.99e-04]
Epoch 45 TRAIN:  79%|███████▉  | 108/136 [00:24<00:06,  4.65it/s, epoch=45, grad_norm=142.132, loss=282.4911, lr=4.99e-04]
Epoch 45 TRAIN:  79%|███████▉  | 108/136 [00:24<00:06,  4.65it/s, epoch=45, grad_norm=184.160, loss=340.3681, lr=4.99e-04]
Epoch 45 TRAIN:  79%|███████▉  | 108/136 [00:25<00:06,  4.65it/s, epoch=45, grad_norm=123.886, loss=281.1942, lr=4.99e-04]
Epoch 45 TRAIN:  79%|███████▉  | 108/136 [00:25<00:06,  4.65it/s, epoch=45, grad_norm=144.374, loss=280.4599, lr=5.00e-04]
Epoch 45 TRAIN:  82%|████████▏ | 111/136 [00:25<00:05,  4.46it/s, epoch=45, grad_norm=144.374, loss=280.4599, lr=5.00e-04]
Epoch 45 TRAIN:  82%|████████▏ | 111/136 [00:25<00:05,  4.46it/s, epoch=45, grad_norm=330.215, loss=400.2028, lr=5.00e-04]
Epoch 45 TRAIN:  82%|████████▏ | 111/136 [00:25<00:05,  4.46it/s, epoch=45, grad_norm=261.626, loss=330.6716, lr=5.00e-04]
Epoch 45 TRAIN:  82%|████████▏ | 111/136 [00:26<00:05,  4.46it/s, epoch=45, grad_norm=138.732, loss=328.3927, lr=5.00e-04]
Epoch 45 TRAIN:  84%|████████▍ | 114/136 [00:26<00:04,  4.47it/s, epoch=45, grad_norm=138.732, loss=328.3927, lr=5.00e-04]
Epoch 45 TRAIN:  84%|████████▍ | 114/136 [00:26<00:04,  4.47it/s, epoch=45, grad_norm=239.154, loss=192.7662, lr=5.00e-04]
Epoch 45 TRAIN:  84%|████████▍ | 114/136 [00:26<00:04,  4.47it/s, epoch=45, grad_norm=219.782, loss=289.8785, lr=5.00e-04]
Epoch 45 TRAIN:  84%|████████▍ | 114/136 [00:26<00:04,  4.47it/s, epoch=45, grad_norm=141.073, loss=306.4297, lr=5.00e-04]
Epoch 45 TRAIN:  86%|████████▌ | 117/136 [00:26<00:03,  4.76it/s, epoch=45, grad_norm=141.073, loss=306.4297, lr=5.00e-04]
Epoch 45 TRAIN:  86%|████████▌ | 117/136 [00:26<00:03,  4.76it/s, epoch=45, grad_norm=150.638, loss=368.7889, lr=5.00e-04]
Epoch 45 TRAIN:  86%|████████▌ | 117/136 [00:26<00:03,  4.76it/s, epoch=45, grad_norm=98.082, loss=282.8398, lr=5.00e-04] 
Epoch 45 TRAIN:  86%|████████▌ | 117/136 [00:27<00:03,  4.76it/s, epoch=45, grad_norm=86.214, loss=229.6287, lr=5.00e-04]
Epoch 45 TRAIN:  88%|████████▊ | 120/136 [00:27<00:03,  4.75it/s, epoch=45, grad_norm=86.214, loss=229.6287, lr=5.00e-04]
Epoch 45 TRAIN:  88%|████████▊ | 120/136 [00:27<00:03,  4.75it/s, epoch=45, grad_norm=406.962, loss=429.3588, lr=5.00e-04]
Epoch 45 TRAIN:  88%|████████▊ | 120/136 [00:27<00:03,  4.75it/s, epoch=45, grad_norm=205.871, loss=352.7090, lr=5.00e-04]
Epoch 45 TRAIN:  88%|████████▊ | 120/136 [00:27<00:03,  4.75it/s, epoch=45, grad_norm=109.343, loss=284.7146, lr=5.00e-04]
Epoch 45 TRAIN:  90%|█████████ | 123/136 [00:27<00:02,  4.96it/s, epoch=45, grad_norm=109.343, loss=284.7146, lr=5.00e-04]
Epoch 45 TRAIN:  90%|█████████ | 123/136 [00:27<00:02,  4.96it/s, epoch=45, grad_norm=122.723, loss=294.9919, lr=5.01e-04]
Epoch 45 TRAIN:  90%|█████████ | 123/136 [00:28<00:02,  4.96it/s, epoch=45, grad_norm=170.224, loss=351.5276, lr=5.01e-04]
Epoch 45 TRAIN:  90%|█████████ | 123/136 [00:28<00:02,  4.96it/s, epoch=45, grad_norm=134.587, loss=321.6353, lr=5.01e-04]
Epoch 45 TRAIN:  93%|█████████▎| 126/136 [00:28<00:01,  5.03it/s, epoch=45, grad_norm=134.587, loss=321.6353, lr=5.01e-04]
Epoch 45 TRAIN:  93%|█████████▎| 126/136 [00:28<00:01,  5.03it/s, epoch=45, grad_norm=105.117, loss=287.1364, lr=5.01e-04]
Epoch 45 TRAIN:  93%|█████████▎| 126/136 [00:28<00:01,  5.03it/s, epoch=45, grad_norm=87.826, loss=247.9239, lr=5.01e-04] 
Epoch 45 TRAIN:  93%|█████████▎| 126/136 [00:28<00:01,  5.03it/s, epoch=45, grad_norm=291.791, loss=409.9901, lr=5.01e-04]
Epoch 45 TRAIN:  95%|█████████▍| 129/136 [00:28<00:01,  4.91it/s, epoch=45, grad_norm=291.791, loss=409.9901, lr=5.01e-04]
Epoch 45 TRAIN:  95%|█████████▍| 129/136 [00:29<00:01,  4.91it/s, epoch=45, grad_norm=137.897, loss=329.3752, lr=5.01e-04]
Epoch 45 TRAIN:  95%|█████████▍| 129/136 [00:29<00:01,  4.91it/s, epoch=45, grad_norm=205.731, loss=327.7789, lr=5.01e-04]
Epoch 45 TRAIN:  95%|█████████▍| 129/136 [00:29<00:01,  4.91it/s, epoch=45, grad_norm=195.296, loss=270.9139, lr=5.01e-04]
Epoch 45 TRAIN:  97%|█████████▋| 132/136 [00:29<00:00,  5.10it/s, epoch=45, grad_norm=195.296, loss=270.9139, lr=5.01e-04]
Epoch 45 TRAIN:  97%|█████████▋| 132/136 [00:29<00:00,  5.10it/s, epoch=45, grad_norm=237.733, loss=403.3951, lr=5.01e-04]
Epoch 45 TRAIN:  97%|█████████▋| 132/136 [00:29<00:00,  5.10it/s, epoch=45, grad_norm=210.286, loss=284.3214, lr=5.01e-04]
Epoch 45 TRAIN:  97%|█████████▋| 132/136 [00:29<00:00,  5.10it/s, epoch=45, grad_norm=179.957, loss=348.7706, lr=5.01e-04]
Epoch 45 TRAIN:  97%|█████████▋| 132/136 [00:30<00:00,  5.10it/s, epoch=45, grad_norm=137.470, loss=335.7750, lr=5.02e-04]
Epoch 45 TRAIN: 100%|██████████| 136/136 [00:30<00:00,  5.47it/s, epoch=45, grad_norm=137.470, loss=335.7750, lr=5.02e-04]
Epoch 45 TRAIN: 100%|██████████| 136/136 [00:30<00:00,  5.47it/s, epoch=45, grad_norm=178.606, loss=356.7975, lr=5.02e-04]
Epoch 45 TRAIN: 137it [00:30,  4.51it/s, epoch=45, grad_norm=178.606, loss=356.7975, lr=5.02e-04]                         

CV:   0%|          | 0/6 [00:00<?, ?it/s]/home/rookie/Documents/batuka/projects/wenet/wenet/wenet/utils/train_utils.py:689: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  amp_autocast(enabled=dtype is not None,

CV:  17%|█▋        | 1/6 [00:00<00:03,  1.32it/s]
                                                 
2026-04-16 20:15:04,698 INFO Epoch 45 Step 6270 CV info lr 5.0160e-04 cv_loss 492.96236072887075 rank 0 acc 0.2950093001127243
2026-04-16 20:15:04,699 INFO [Rank 0] Checkpoint: save to checkpoint exp/conformer_mongolian/epoch_45.pt
2026-04-16 20:15:04,818 INFO Epoch 46 Step 6270 TRAIN info lr 5.0160e-04 rank 0
2026-04-16 20:15:04,836 INFO using accumulate grad, new batch size is 1 times larger than before

Epoch 46 TRAIN:   0%|          | 0/137 [00:00<?, ?it/s]
Epoch 46 TRAIN:   0%|          | 0/137 [00:04<?, ?it/s, epoch=46, grad_norm=441.668, loss=373.1242, lr=5.02e-04]
Epoch 46 TRAIN:   1%|          | 1/137 [00:04<09:11,  4.05s/it, epoch=46, grad_norm=441.668, loss=373.1242, lr=5.02e-04]
Epoch 46 TRAIN:   1%|          | 1/137 [00:04<09:11,  4.05s/it, epoch=46, grad_norm=354.231, loss=338.2107, lr=5.02e-04]
Epoch 46 TRAIN:   1%|          | 1/137 [00:04<09:11,  4.05s/it, epoch=46, grad_norm=412.148, loss=386.5688, lr=5.02e-04]
Epoch 46 TRAIN:   1%|          | 1/137 [00:04<09:11,  4.05s/it, epoch=46, grad_norm=370.269, loss=411.8772, lr=5.02e-04]
Epoch 46 TRAIN:   1%|          | 1/137 [00:04<09:11,  4.05s/it, epoch=46, grad_norm=284.526, loss=330.0976, lr=5.02e-04]
Epoch 46 TRAIN:   1%|          | 1/137 [00:04<09:11,  4.05s/it, epoch=46, grad_norm=358.988, loss=416.4854, lr=5.02e-04]
Epoch 46 TRAIN:   1%|          | 1/137 [00:04<09:11,  4.05s/it, epoch=46, grad_norm=293.286, loss=393.4322, lr=5.02e-04]
Epoch 46 TRAIN:   1%|          | 1/137 [00:04<09:11,  4.05s/it, epoch=46, grad_norm=131.808, loss=303.9185, lr=5.02e-04]
Epoch 46 TRAIN:   1%|          | 1/137 [00:04<09:11,  4.05s/it, epoch=46, grad_norm=77.858, loss=299.2021, lr=5.02e-04] 
Epoch 46 TRAIN:   7%|▋         | 9/137 [00:04<00:50,  2.53it/s, epoch=46, grad_norm=77.858, loss=299.2021, lr=5.02e-04]
Epoch 46 TRAIN:   7%|▋         | 9/137 [00:04<00:50,  2.53it/s, epoch=46, grad_norm=91.787, loss=338.6925, lr=5.02e-04]
Epoch 46 TRAIN:   7%|▋         | 9/137 [00:04<00:50,  2.53it/s, epoch=46, grad_norm=133.964, loss=323.5792, lr=5.02e-04]
Epoch 46 TRAIN:   7%|▋         | 9/137 [00:05<00:50,  2.53it/s, epoch=46, grad_norm=212.316, loss=243.8547, lr=5.03e-04]
Epoch 46 TRAIN:   7%|▋         | 9/137 [00:05<00:50,  2.53it/s, epoch=46, grad_norm=96.595, loss=337.5580, lr=5.03e-04] 
Epoch 46 TRAIN:   9%|▉         | 13/137 [00:05<00:37,  3.29it/s, epoch=46, grad_norm=96.595, loss=337.5580, lr=5.03e-04]
Epoch 46 TRAIN:   9%|▉         | 13/137 [00:05<00:37,  3.29it/s, epoch=46, grad_norm=73.949, loss=352.9312, lr=5.03e-04]
Epoch 46 TRAIN:   9%|▉         | 13/137 [00:05<00:37,  3.29it/s, epoch=46, grad_norm=65.682, loss=325.6320, lr=5.03e-04]
Epoch 46 TRAIN:   9%|▉         | 13/137 [00:05<00:37,  3.29it/s, epoch=46, grad_norm=66.364, loss=324.9691, lr=5.03e-04]
Epoch 46 TRAIN:   9%|▉         | 13/137 [00:06<00:37,  3.29it/s, epoch=46, grad_norm=78.231, loss=293.8828, lr=5.03e-04]
Epoch 46 TRAIN:  12%|█▏        | 17/137 [00:06<00:31,  3.78it/s, epoch=46, grad_norm=78.231, loss=293.8828, lr=5.03e-04]
Epoch 46 TRAIN:  12%|█▏        | 17/137 [00:06<00:31,  3.78it/s, epoch=46, grad_norm=84.239, loss=307.2433, lr=5.03e-04]
Epoch 46 TRAIN:  12%|█▏        | 17/137 [00:06<00:31,  3.78it/s, epoch=46, grad_norm=107.300, loss=310.5154, lr=5.03e-04]
Epoch 46 TRAIN:  12%|█▏        | 17/137 [00:06<00:31,  3.78it/s, epoch=46, grad_norm=86.829, loss=320.2249, lr=5.03e-04] 
Epoch 46 TRAIN:  12%|█▏        | 17/137 [00:06<00:31,  3.78it/s, epoch=46, grad_norm=92.422, loss=304.8394, lr=5.03e-04]
Epoch 46 TRAIN:  15%|█▌        | 21/137 [00:06<00:27,  4.20it/s, epoch=46, grad_norm=92.422, loss=304.8394, lr=5.03e-04]
Epoch 46 TRAIN:  15%|█▌        | 21/137 [00:07<00:27,  4.20it/s, epoch=46, grad_norm=99.546, loss=314.7424, lr=5.03e-04]
Epoch 46 TRAIN:  15%|█▌        | 21/137 [00:07<00:27,  4.20it/s, epoch=46, grad_norm=56.357, loss=295.1758, lr=5.03e-04]
Epoch 46 TRAIN:  15%|█▌        | 21/137 [00:07<00:27,  4.20it/s, epoch=46, grad_norm=64.356, loss=308.4774, lr=5.04e-04]
Epoch 46 TRAIN:  18%|█▊        | 24/137 [00:07<00:25,  4.42it/s, epoch=46, grad_norm=64.356, loss=308.4774, lr=5.04e-04]
Epoch 46 TRAIN:  18%|█▊        | 24/137 [00:07<00:25,  4.42it/s, epoch=46, grad_norm=93.995, loss=298.4062, lr=5.04e-04]
Epoch 46 TRAIN:  18%|█▊        | 24/137 [00:07<00:25,  4.42it/s, epoch=46, grad_norm=97.319, loss=355.2606, lr=5.04e-04]
Epoch 46 TRAIN:  18%|█▊        | 24/137 [00:08<00:25,  4.42it/s, epoch=46, grad_norm=83.031, loss=336.0740, lr=5.04e-04]
Epoch 46 TRAIN:  20%|█▉        | 27/137 [00:08<00:24,  4.49it/s, epoch=46, grad_norm=83.031, loss=336.0740, lr=5.04e-04]
Epoch 46 TRAIN:  20%|█▉        | 27/137 [00:08<00:24,  4.49it/s, epoch=46, grad_norm=74.280, loss=321.1629, lr=5.04e-04]
Epoch 46 TRAIN:  20%|█▉        | 27/137 [00:08<00:24,  4.49it/s, epoch=46, grad_norm=122.894, loss=276.6447, lr=5.04e-04]
Epoch 46 TRAIN:  20%|█▉        | 27/137 [00:08<00:24,  4.49it/s, epoch=46, grad_norm=104.515, loss=311.5042, lr=5.04e-04]
Epoch 46 TRAIN:  22%|██▏       | 30/137 [00:08<00:22,  4.72it/s, epoch=46, grad_norm=104.515, loss=311.5042, lr=5.04e-04]
Epoch 46 TRAIN:  22%|██▏       | 30/137 [00:08<00:22,  4.72it/s, epoch=46, grad_norm=73.598, loss=288.4122, lr=5.04e-04] 
Epoch 46 TRAIN:  22%|██▏       | 30/137 [00:08<00:22,  4.72it/s, epoch=46, grad_norm=68.901, loss=295.6118, lr=5.04e-04]
Epoch 46 TRAIN:  22%|██▏       | 30/137 [00:09<00:22,  4.72it/s, epoch=46, grad_norm=70.900, loss=287.1921, lr=5.04e-04]
Epoch 46 TRAIN:  24%|██▍       | 33/137 [00:09<00:20,  5.00it/s, epoch=46, grad_norm=70.900, loss=287.1921, lr=5.04e-04]
Epoch 46 TRAIN:  24%|██▍       | 33/137 [00:09<00:20,  5.00it/s, epoch=46, grad_norm=143.515, loss=349.0096, lr=5.04e-04]
Epoch 46 TRAIN:  24%|██▍       | 33/137 [00:09<00:20,  5.00it/s, epoch=46, grad_norm=81.692, loss=315.1285, lr=5.04e-04] 
Epoch 46 TRAIN:  24%|██▍       | 33/137 [00:09<00:20,  5.00it/s, epoch=46, grad_norm=99.619, loss=357.5982, lr=5.04e-04]
Epoch 46 TRAIN:  26%|██▋       | 36/137 [00:09<00:21,  4.80it/s, epoch=46, grad_norm=99.619, loss=357.5982, lr=5.04e-04]
Epoch 46 TRAIN:  26%|██▋       | 36/137 [00:10<00:21,  4.80it/s, epoch=46, grad_norm=134.660, loss=299.6674, lr=5.05e-04]
Epoch 46 TRAIN:  26%|██▋       | 36/137 [00:10<00:21,  4.80it/s, epoch=46, grad_norm=114.271, loss=284.4282, lr=5.05e-04]
Epoch 46 TRAIN:  26%|██▋       | 36/137 [00:10<00:21,  4.80it/s, epoch=46, grad_norm=88.220, loss=329.6210, lr=5.05e-04] 
Epoch 46 TRAIN:  28%|██▊       | 39/137 [00:10<00:20,  4.85it/s, epoch=46, grad_norm=88.220, loss=329.6210, lr=5.05e-04]
Epoch 46 TRAIN:  28%|██▊       | 39/137 [00:10<00:20,  4.85it/s, epoch=46, grad_norm=100.845, loss=290.4731, lr=5.05e-04]
Epoch 46 TRAIN:  28%|██▊       | 39/137 [00:10<00:20,  4.85it/s, epoch=46, grad_norm=87.842, loss=286.8554, lr=5.05e-04] 
Epoch 46 TRAIN:  28%|██▊       | 39/137 [00:11<00:20,  4.85it/s, epoch=46, grad_norm=103.117, loss=284.7070, lr=5.05e-04]
Epoch 46 TRAIN:  31%|███       | 42/137 [00:11<00:19,  4.89it/s, epoch=46, grad_norm=103.117, loss=284.7070, lr=5.05e-04]
Epoch 46 TRAIN:  31%|███       | 42/137 [00:11<00:19,  4.89it/s, epoch=46, grad_norm=73.105, loss=283.1494, lr=5.05e-04] 
Epoch 46 TRAIN:  31%|███       | 42/137 [00:11<00:19,  4.89it/s, epoch=46, grad_norm=95.951, loss=301.0935, lr=5.05e-04]
Epoch 46 TRAIN:  31%|███       | 42/137 [00:11<00:19,  4.89it/s, epoch=46, grad_norm=74.995, loss=276.2412, lr=5.05e-04]
Epoch 46 TRAIN:  33%|███▎      | 45/137 [00:11<00:19,  4.77it/s, epoch=46, grad_norm=74.995, loss=276.2412, lr=5.05e-04]
Epoch 46 TRAIN:  33%|███▎      | 45/137 [00:11<00:19,  4.77it/s, epoch=46, grad_norm=59.837, loss=235.9447, lr=5.05e-04]
Epoch 46 TRAIN:  33%|███▎      | 45/137 [00:12<00:19,  4.77it/s, epoch=46, grad_norm=150.066, loss=320.4341, lr=5.05e-04]
Epoch 46 TRAIN:  33%|███▎      | 45/137 [00:12<00:19,  4.77it/s, epoch=46, grad_norm=230.173, loss=353.4595, lr=5.05e-04]
Epoch 46 TRAIN:  35%|███▌      | 48/137 [00:12<00:18,  4.89it/s, epoch=46, grad_norm=230.173, loss=353.4595, lr=5.05e-04]
Epoch 46 TRAIN:  35%|███▌      | 48/137 [00:12<00:18,  4.89it/s, epoch=46, grad_norm=119.566, loss=342.3948, lr=5.06e-04]
Epoch 46 TRAIN:  35%|███▌      | 48/137 [00:12<00:18,  4.89it/s, epoch=46, grad_norm=87.230, loss=281.7136, lr=5.06e-04] 
Epoch 46 TRAIN:  35%|███▌      | 48/137 [00:12<00:18,  4.89it/s, epoch=46, grad_norm=123.520, loss=351.7959, lr=5.06e-04]
Epoch 46 TRAIN:  37%|███▋      | 51/137 [00:12<00:17,  4.83it/s, epoch=46, grad_norm=123.520, loss=351.7959, lr=5.06e-04]
Epoch 46 TRAIN:  37%|███▋      | 51/137 [00:13<00:17,  4.83it/s, epoch=46, grad_norm=91.381, loss=327.0761, lr=5.06e-04] 
Epoch 46 TRAIN:  37%|███▋      | 51/137 [00:13<00:17,  4.83it/s, epoch=46, grad_norm=136.475, loss=306.1895, lr=5.06e-04]
Epoch 46 TRAIN:  37%|███▋      | 51/137 [00:13<00:17,  4.83it/s, epoch=46, grad_norm=118.142, loss=293.4584, lr=5.06e-04]
Epoch 46 TRAIN:  39%|███▉      | 54/137 [00:13<00:17,  4.76it/s, epoch=46, grad_norm=118.142, loss=293.4584, lr=5.06e-04]
Epoch 46 TRAIN:  39%|███▉      | 54/137 [00:13<00:17,  4.76it/s, epoch=46, grad_norm=91.948, loss=327.9047, lr=5.06e-04] 
Epoch 46 TRAIN:  39%|███▉      | 54/137 [00:14<00:17,  4.76it/s, epoch=46, grad_norm=101.098, loss=332.0428, lr=5.06e-04]
Epoch 46 TRAIN:  39%|███▉      | 54/137 [00:14<00:17,  4.76it/s, epoch=46, grad_norm=88.055, loss=289.7469, lr=5.06e-04] 
Epoch 46 TRAIN:  42%|████▏     | 57/137 [00:14<00:17,  4.64it/s, epoch=46, grad_norm=88.055, loss=289.7469, lr=5.06e-04]
Epoch 46 TRAIN:  42%|████▏     | 57/137 [00:14<00:17,  4.64it/s, epoch=46, grad_norm=157.523, loss=318.4827, lr=5.06e-04]
Epoch 46 TRAIN:  42%|████▏     | 57/137 [00:14<00:17,  4.64it/s, epoch=46, grad_norm=108.689, loss=283.1647, lr=5.06e-04]
Epoch 46 TRAIN:  42%|████▏     | 57/137 [00:15<00:17,  4.64it/s, epoch=46, grad_norm=116.502, loss=299.0865, lr=5.06e-04]
Epoch 46 TRAIN:  44%|████▍     | 60/137 [00:15<00:17,  4.41it/s, epoch=46, grad_norm=116.502, loss=299.0865, lr=5.06e-04]
Epoch 46 TRAIN:  44%|████▍     | 60/137 [00:15<00:17,  4.41it/s, epoch=46, grad_norm=98.402, loss=305.5854, lr=5.06e-04] 
Epoch 46 TRAIN:  44%|████▍     | 60/137 [00:15<00:17,  4.41it/s, epoch=46, grad_norm=142.936, loss=328.7322, lr=5.07e-04]
Epoch 46 TRAIN:  44%|████▍     | 60/137 [00:15<00:17,  4.41it/s, epoch=46, grad_norm=82.973, loss=295.0053, lr=5.07e-04] 
Epoch 46 TRAIN:  44%|████▍     | 60/137 [00:15<00:17,  4.41it/s, epoch=46, grad_norm=101.991, loss=293.6656, lr=5.07e-04]
Epoch 46 TRAIN:  47%|████▋     | 64/137 [00:15<00:14,  4.87it/s, epoch=46, grad_norm=101.991, loss=293.6656, lr=5.07e-04]
Epoch 46 TRAIN:  47%|████▋     | 64/137 [00:16<00:14,  4.87it/s, epoch=46, grad_norm=102.405, loss=292.2286, lr=5.07e-04]
Epoch 46 TRAIN:  47%|████▋     | 64/137 [00:16<00:14,  4.87it/s, epoch=46, grad_norm=89.338, loss=317.6375, lr=5.07e-04] 
Epoch 46 TRAIN:  47%|████▋     | 64/137 [00:16<00:14,  4.87it/s, epoch=46, grad_norm=98.948, loss=319.9079, lr=5.07e-04]
Epoch 46 TRAIN:  49%|████▉     | 67/137 [00:16<00:15,  4.64it/s, epoch=46, grad_norm=98.948, loss=319.9079, lr=5.07e-04]
Epoch 46 TRAIN:  49%|████▉     | 67/137 [00:16<00:15,  4.64it/s, epoch=46, grad_norm=169.492, loss=233.1452, lr=5.07e-04]
Epoch 46 TRAIN:  49%|████▉     | 67/137 [00:16<00:15,  4.64it/s, epoch=46, grad_norm=80.142, loss=268.8518, lr=5.07e-04] 
Epoch 46 TRAIN:  49%|████▉     | 67/137 [00:16<00:15,  4.64it/s, epoch=46, grad_norm=110.547, loss=326.5744, lr=5.07e-04]
Epoch 46 TRAIN:  51%|█████     | 70/137 [00:16<00:13,  4.83it/s, epoch=46, grad_norm=110.547, loss=326.5744, lr=5.07e-04]
Epoch 46 TRAIN:  51%|█████     | 70/137 [00:17<00:13,  4.83it/s, epoch=46, grad_norm=137.213, loss=310.5724, lr=5.07e-04]
Epoch 46 TRAIN:  51%|█████     | 70/137 [00:17<00:13,  4.83it/s, epoch=46, grad_norm=111.313, loss=323.5070, lr=5.07e-04]
Epoch 46 TRAIN:  51%|█████     | 70/137 [00:17<00:13,  4.83it/s, epoch=46, grad_norm=98.251, loss=316.0937, lr=5.07e-04] 
Epoch 46 TRAIN:  53%|█████▎    | 73/137 [00:17<00:13,  4.65it/s, epoch=46, grad_norm=98.251, loss=316.0937, lr=5.07e-04]
Epoch 46 TRAIN:  53%|█████▎    | 73/137 [00:17<00:13,  4.65it/s, epoch=46, grad_norm=113.080, loss=317.1069, lr=5.08e-04]
Epoch 46 TRAIN:  53%|█████▎    | 73/137 [00:18<00:13,  4.65it/s, epoch=46, grad_norm=218.892, loss=285.6948, lr=5.08e-04]
Epoch 46 TRAIN:  53%|█████▎    | 73/137 [00:18<00:13,  4.65it/s, epoch=46, grad_norm=95.817, loss=318.8352, lr=5.08e-04] 
Epoch 46 TRAIN:  55%|█████▌    | 76/137 [00:18<00:13,  4.63it/s, epoch=46, grad_norm=95.817, loss=318.8352, lr=5.08e-04]
Epoch 46 TRAIN:  55%|█████▌    | 76/137 [00:18<00:13,  4.63it/s, epoch=46, grad_norm=145.616, loss=271.4129, lr=5.08e-04]
Epoch 46 TRAIN:  55%|█████▌    | 76/137 [00:18<00:13,  4.63it/s, epoch=46, grad_norm=256.765, loss=211.3709, lr=5.08e-04]
Epoch 46 TRAIN:  55%|█████▌    | 76/137 [00:18<00:13,  4.63it/s, epoch=46, grad_norm=79.887, loss=260.8382, lr=5.08e-04] 
Epoch 46 TRAIN:  58%|█████▊    | 79/137 [00:18<00:12,  4.58it/s, epoch=46, grad_norm=79.887, loss=260.8382, lr=5.08e-04]
Epoch 46 TRAIN:  58%|█████▊    | 79/137 [00:19<00:12,  4.58it/s, epoch=46, grad_norm=114.222, loss=299.7663, lr=5.08e-04]
Epoch 46 TRAIN:  58%|█████▊    | 79/137 [00:19<00:12,  4.58it/s, epoch=46, grad_norm=155.039, loss=305.0795, lr=5.08e-04]
Epoch 46 TRAIN:  58%|█████▊    | 79/137 [00:19<00:12,  4.58it/s, epoch=46, grad_norm=206.989, loss=348.4661, lr=5.08e-04]
Epoch 46 TRAIN:  60%|█████▉    | 82/137 [00:19<00:12,  4.52it/s, epoch=46, grad_norm=206.989, loss=348.4661, lr=5.08e-04]
Epoch 46 TRAIN:  60%|█████▉    | 82/137 [00:19<00:12,  4.52it/s, epoch=46, grad_norm=124.189, loss=230.7434, lr=5.08e-04]
Epoch 46 TRAIN:  60%|█████▉    | 82/137 [00:20<00:12,  4.52it/s, epoch=46, grad_norm=234.589, loss=363.0017, lr=5.08e-04]
Epoch 46 TRAIN:  60%|█████▉    | 82/137 [00:20<00:12,  4.52it/s, epoch=46, grad_norm=90.622, loss=268.7203, lr=5.08e-04] 
Epoch 46 TRAIN:  62%|██████▏   | 85/137 [00:20<00:11,  4.62it/s, epoch=46, grad_norm=90.622, loss=268.7203, lr=5.08e-04]
Epoch 46 TRAIN:  62%|██████▏   | 85/137 [00:20<00:11,  4.62it/s, epoch=46, grad_norm=95.321, loss=316.0375, lr=5.08e-04]
Epoch 46 TRAIN:  62%|██████▏   | 85/137 [00:20<00:11,  4.62it/s, epoch=46, grad_norm=91.197, loss=255.9037, lr=5.09e-04]
Epoch 46 TRAIN:  62%|██████▏   | 85/137 [00:21<00:11,  4.62it/s, epoch=46, grad_norm=113.683, loss=283.1731, lr=5.09e-04]
Epoch 46 TRAIN:  64%|██████▍   | 88/137 [00:21<00:11,  4.45it/s, epoch=46, grad_norm=113.683, loss=283.1731, lr=5.09e-04]
Epoch 46 TRAIN:  64%|██████▍   | 88/137 [00:21<00:11,  4.45it/s, epoch=46, grad_norm=116.184, loss=249.3958, lr=5.09e-04]
Epoch 46 TRAIN:  64%|██████▍   | 88/137 [00:21<00:11,  4.45it/s, epoch=46, grad_norm=116.860, loss=264.5928, lr=5.09e-04]
Epoch 46 TRAIN:  64%|██████▍   | 88/137 [00:21<00:11,  4.45it/s, epoch=46, grad_norm=188.752, loss=235.0695, lr=5.09e-04]
Epoch 46 TRAIN:  66%|██████▋   | 91/137 [00:21<00:10,  4.45it/s, epoch=46, grad_norm=188.752, loss=235.0695, lr=5.09e-04]
Epoch 46 TRAIN:  66%|██████▋   | 91/137 [00:21<00:10,  4.45it/s, epoch=46, grad_norm=125.163, loss=293.5866, lr=5.09e-04]
Epoch 46 TRAIN:  66%|██████▋   | 91/137 [00:22<00:10,  4.45it/s, epoch=46, grad_norm=297.983, loss=370.4370, lr=5.09e-04]
Epoch 46 TRAIN:  66%|██████▋   | 91/137 [00:22<00:10,  4.45it/s, epoch=46, grad_norm=225.803, loss=166.1569, lr=5.09e-04]
Epoch 46 TRAIN:  69%|██████▊   | 94/137 [00:22<00:09,  4.74it/s, epoch=46, grad_norm=225.803, loss=166.1569, lr=5.09e-04]
Epoch 46 TRAIN:  69%|██████▊   | 94/137 [00:22<00:09,  4.74it/s, epoch=46, grad_norm=119.391, loss=271.5100, lr=5.09e-04]
Epoch 46 TRAIN:  69%|██████▊   | 94/137 [00:22<00:09,  4.74it/s, epoch=46, grad_norm=246.870, loss=354.3740, lr=5.09e-04]
Epoch 46 TRAIN:  69%|██████▊   | 94/137 [00:22<00:09,  4.74it/s, epoch=46, grad_norm=128.752, loss=289.7628, lr=5.09e-04]
Epoch 46 TRAIN:  71%|███████   | 97/137 [00:22<00:08,  4.83it/s, epoch=46, grad_norm=128.752, loss=289.7628, lr=5.09e-04]
Epoch 46 TRAIN:  71%|███████   | 97/137 [00:23<00:08,  4.83it/s, epoch=46, grad_norm=110.661, loss=255.8953, lr=5.09e-04]
Epoch 46 TRAIN:  71%|███████   | 97/137 [00:23<00:08,  4.83it/s, epoch=46, grad_norm=143.817, loss=262.3155, lr=5.10e-04]
Epoch 46 TRAIN:  71%|███████   | 97/137 [00:23<00:08,  4.83it/s, epoch=46, grad_norm=130.678, loss=313.9985, lr=5.10e-04]2026-04-16 20:15:28,249 DEBUG TRAIN | steps/sec 4.305| Batch 46/100 loss 313.998474 loss_att 269.510254 loss_ctc 417.804260 th_accuracy 0.336667 lr 5.0960e-04 grad_norm 130.677856 rank 0

Epoch 46 TRAIN:  73%|███████▎  | 100/137 [00:23<00:07,  4.90it/s, epoch=46, grad_norm=130.678, loss=313.9985, lr=5.10e-04]
Epoch 46 TRAIN:  73%|███████▎  | 100/137 [00:23<00:07,  4.90it/s, epoch=46, grad_norm=155.238, loss=335.5334, lr=5.10e-04]
Epoch 46 TRAIN:  73%|███████▎  | 100/137 [00:23<00:07,  4.90it/s, epoch=46, grad_norm=133.566, loss=325.1274, lr=5.10e-04]
Epoch 46 TRAIN:  73%|███████▎  | 100/137 [00:24<00:07,  4.90it/s, epoch=46, grad_norm=232.428, loss=372.6790, lr=5.10e-04]
Epoch 46 TRAIN:  75%|███████▌  | 103/137 [00:24<00:07,  4.59it/s, epoch=46, grad_norm=232.428, loss=372.6790, lr=5.10e-04]
Epoch 46 TRAIN:  75%|███████▌  | 103/137 [00:24<00:07,  4.59it/s, epoch=46, grad_norm=144.581, loss=317.6108, lr=5.10e-04]
Epoch 46 TRAIN:  75%|███████▌  | 103/137 [00:24<00:07,  4.59it/s, epoch=46, grad_norm=104.933, loss=334.2640, lr=5.10e-04]
Epoch 46 TRAIN:  75%|███████▌  | 103/137 [00:24<00:07,  4.59it/s, epoch=46, grad_norm=164.602, loss=302.1272, lr=5.10e-04]
Epoch 46 TRAIN:  77%|███████▋  | 106/137 [00:24<00:06,  4.60it/s, epoch=46, grad_norm=164.602, loss=302.1272, lr=5.10e-04]
Epoch 46 TRAIN:  77%|███████▋  | 106/137 [00:25<00:06,  4.60it/s, epoch=46, grad_norm=213.099, loss=283.5815, lr=5.10e-04]
Epoch 46 TRAIN:  77%|███████▋  | 106/137 [00:25<00:06,  4.60it/s, epoch=46, grad_norm=141.348, loss=248.7361, lr=5.10e-04]
Epoch 46 TRAIN:  77%|███████▋  | 106/137 [00:25<00:06,  4.60it/s, epoch=46, grad_norm=99.568, loss=290.1770, lr=5.10e-04] 
Epoch 46 TRAIN:  80%|███████▉  | 109/137 [00:25<00:06,  4.62it/s, epoch=46, grad_norm=99.568, loss=290.1770, lr=5.10e-04]
Epoch 46 TRAIN:  80%|███████▉  | 109/137 [00:25<00:06,  4.62it/s, epoch=46, grad_norm=103.645, loss=249.4617, lr=5.10e-04]
Epoch 46 TRAIN:  80%|███████▉  | 109/137 [00:25<00:06,  4.62it/s, epoch=46, grad_norm=224.086, loss=312.5338, lr=5.10e-04]
Epoch 46 TRAIN:  80%|███████▉  | 109/137 [00:26<00:06,  4.62it/s, epoch=46, grad_norm=77.906, loss=197.8798, lr=5.11e-04] 
Epoch 46 TRAIN:  82%|████████▏ | 112/137 [00:26<00:05,  4.47it/s, epoch=46, grad_norm=77.906, loss=197.8798, lr=5.11e-04]
Epoch 46 TRAIN:  82%|████████▏ | 112/137 [00:26<00:05,  4.47it/s, epoch=46, grad_norm=227.022, loss=282.2849, lr=5.11e-04]
Epoch 46 TRAIN:  82%|████████▏ | 112/137 [00:26<00:05,  4.47it/s, epoch=46, grad_norm=353.431, loss=363.2269, lr=5.11e-04]
Epoch 46 TRAIN:  82%|████████▏ | 112/137 [00:26<00:05,  4.47it/s, epoch=46, grad_norm=351.395, loss=382.1074, lr=5.11e-04]
Epoch 46 TRAIN:  84%|████████▍ | 115/137 [00:26<00:04,  4.66it/s, epoch=46, grad_norm=351.395, loss=382.1074, lr=5.11e-04]
Epoch 46 TRAIN:  84%|████████▍ | 115/137 [00:26<00:04,  4.66it/s, epoch=46, grad_norm=130.191, loss=261.4968, lr=5.11e-04]
Epoch 46 TRAIN:  84%|████████▍ | 115/137 [00:27<00:04,  4.66it/s, epoch=46, grad_norm=148.232, loss=301.2585, lr=5.11e-04]
Epoch 46 TRAIN:  84%|████████▍ | 115/137 [00:27<00:04,  4.66it/s, epoch=46, grad_norm=208.415, loss=354.5902, lr=5.11e-04]
Epoch 46 TRAIN:  86%|████████▌ | 118/137 [00:27<00:03,  4.85it/s, epoch=46, grad_norm=208.415, loss=354.5902, lr=5.11e-04]
Epoch 46 TRAIN:  86%|████████▌ | 118/137 [00:27<00:03,  4.85it/s, epoch=46, grad_norm=130.899, loss=305.2942, lr=5.11e-04]
Epoch 46 TRAIN:  86%|████████▌ | 118/137 [00:27<00:03,  4.85it/s, epoch=46, grad_norm=144.964, loss=338.7563, lr=5.11e-04]
Epoch 46 TRAIN:  86%|████████▌ | 118/137 [00:27<00:03,  4.85it/s, epoch=46, grad_norm=168.925, loss=290.3874, lr=5.11e-04]
Epoch 46 TRAIN:  88%|████████▊ | 121/137 [00:27<00:03,  4.98it/s, epoch=46, grad_norm=168.925, loss=290.3874, lr=5.11e-04]
Epoch 46 TRAIN:  88%|████████▊ | 121/137 [00:28<00:03,  4.98it/s, epoch=46, grad_norm=244.795, loss=336.3563, lr=5.11e-04]
Epoch 46 TRAIN:  88%|████████▊ | 121/137 [00:28<00:03,  4.98it/s, epoch=46, grad_norm=315.997, loss=305.7511, lr=5.11e-04]
Epoch 46 TRAIN:  88%|████████▊ | 121/137 [00:28<00:03,  4.98it/s, epoch=46, grad_norm=278.865, loss=268.6978, lr=5.12e-04]
Epoch 46 TRAIN:  91%|█████████ | 124/137 [00:28<00:02,  5.18it/s, epoch=46, grad_norm=278.865, loss=268.6978, lr=5.12e-04]
Epoch 46 TRAIN:  91%|█████████ | 124/137 [00:28<00:02,  5.18it/s, epoch=46, grad_norm=224.448, loss=350.9988, lr=5.12e-04]
Epoch 46 TRAIN:  91%|█████████ | 124/137 [00:28<00:02,  5.18it/s, epoch=46, grad_norm=191.911, loss=402.7013, lr=5.12e-04]
Epoch 46 TRAIN:  91%|█████████ | 124/137 [00:29<00:02,  5.18it/s, epoch=46, grad_norm=124.673, loss=292.2282, lr=5.12e-04]
Epoch 46 TRAIN:  93%|█████████▎| 127/137 [00:29<00:02,  4.84it/s, epoch=46, grad_norm=124.673, loss=292.2282, lr=5.12e-04]
Epoch 46 TRAIN:  93%|█████████▎| 127/137 [00:29<00:02,  4.84it/s, epoch=46, grad_norm=108.178, loss=302.7508, lr=5.12e-04]
Epoch 46 TRAIN:  93%|█████████▎| 127/137 [00:29<00:02,  4.84it/s, epoch=46, grad_norm=187.971, loss=306.7339, lr=5.12e-04]
Epoch 46 TRAIN:  93%|█████████▎| 127/137 [00:29<00:02,  4.84it/s, epoch=46, grad_norm=126.809, loss=332.3459, lr=5.12e-04]
Epoch 46 TRAIN:  95%|█████████▍| 130/137 [00:29<00:01,  4.79it/s, epoch=46, grad_norm=126.809, loss=332.3459, lr=5.12e-04]
Epoch 46 TRAIN:  95%|█████████▍| 130/137 [00:29<00:01,  4.79it/s, epoch=46, grad_norm=278.693, loss=288.3318, lr=5.12e-04]
Epoch 46 TRAIN:  95%|█████████▍| 130/137 [00:29<00:01,  4.79it/s, epoch=46, grad_norm=178.044, loss=321.8501, lr=5.12e-04]
Epoch 46 TRAIN:  95%|█████████▍| 130/137 [00:30<00:01,  4.79it/s, epoch=46, grad_norm=144.984, loss=341.6237, lr=5.12e-04]
Epoch 46 TRAIN:  95%|█████████▍| 130/137 [00:30<00:01,  4.79it/s, epoch=46, grad_norm=247.163, loss=403.4785, lr=5.12e-04]
Epoch 46 TRAIN:  98%|█████████▊| 134/137 [00:30<00:00,  5.48it/s, epoch=46, grad_norm=247.163, loss=403.4785, lr=5.12e-04]
Epoch 46 TRAIN:  98%|█████████▊| 134/137 [00:30<00:00,  5.48it/s, epoch=46, grad_norm=227.555, loss=309.0674, lr=5.12e-04]
Epoch 46 TRAIN:  98%|█████████▊| 134/137 [00:30<00:00,  5.48it/s, epoch=46, grad_norm=310.105, loss=291.2967, lr=5.12e-04]
Epoch 46 TRAIN:  98%|█████████▊| 134/137 [00:30<00:00,  5.48it/s, epoch=46, grad_norm=186.353, loss=410.3616, lr=5.13e-04]
Epoch 46 TRAIN:  98%|█████████▊| 134/137 [00:30<00:00,  5.48it/s, epoch=46, grad_norm=244.709, loss=386.1845, lr=5.13e-04]
Epoch 46 TRAIN: 138it [00:30,  6.05it/s, epoch=46, grad_norm=244.709, loss=386.1845, lr=5.13e-04]                         
Epoch 46 TRAIN: 138it [00:30,  6.05it/s, epoch=46, grad_norm=403.819, loss=301.0261, lr=5.13e-04]
Epoch 46 TRAIN: 139it [00:30,  4.49it/s, epoch=46, grad_norm=403.819, loss=301.0261, lr=5.13e-04]

CV:   0%|          | 0/6 [00:00<?, ?it/s]/home/rookie/Documents/batuka/projects/wenet/wenet/wenet/utils/train_utils.py:689: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  amp_autocast(enabled=dtype is not None,

CV:  17%|█▋        | 1/6 [00:00<00:03,  1.32it/s]
                                                 
2026-04-16 20:15:36,667 INFO Epoch 46 Step 6409 CV info lr 5.1272e-04 cv_loss 503.4776888760653 rank 0 acc 0.29349517822265625
2026-04-16 20:15:36,668 INFO [Rank 0] Checkpoint: save to checkpoint exp/conformer_mongolian/epoch_46.pt
2026-04-16 20:15:36,801 INFO Epoch 47 Step 6409 TRAIN info lr 5.1272e-04 rank 0
2026-04-16 20:15:36,821 INFO using accumulate grad, new batch size is 1 times larger than before

Epoch 47 TRAIN:   0%|          | 0/139 [00:00<?, ?it/s]
Epoch 47 TRAIN:   0%|          | 0/139 [00:03<?, ?it/s, epoch=47, grad_norm=460.595, loss=406.3301, lr=5.13e-04]
Epoch 47 TRAIN:   1%|          | 1/139 [00:03<08:58,  3.90s/it, epoch=47, grad_norm=460.595, loss=406.3301, lr=5.13e-04]
Epoch 47 TRAIN:   1%|          | 1/139 [00:04<08:58,  3.90s/it, epoch=47, grad_norm=447.621, loss=374.9543, lr=5.13e-04]
Epoch 47 TRAIN:   1%|          | 1/139 [00:04<08:58,  3.90s/it, epoch=47, grad_norm=283.581, loss=262.7025, lr=5.13e-04]
Epoch 47 TRAIN:   1%|          | 1/139 [00:04<08:58,  3.90s/it, epoch=47, grad_norm=272.920, loss=288.1412, lr=5.13e-04]
Epoch 47 TRAIN:   1%|          | 1/139 [00:04<08:58,  3.90s/it, epoch=47, grad_norm=410.032, loss=396.0536, lr=5.13e-04]
Epoch 47 TRAIN:   1%|          | 1/139 [00:04<08:58,  3.90s/it, epoch=47, grad_norm=375.117, loss=387.7827, lr=5.13e-04]
Epoch 47 TRAIN:   1%|          | 1/139 [00:04<08:58,  3.90s/it, epoch=47, grad_norm=385.533, loss=416.0927, lr=5.13e-04]
Epoch 47 TRAIN:   1%|          | 1/139 [00:04<08:58,  3.90s/it, epoch=47, grad_norm=221.918, loss=287.5532, lr=5.13e-04]
Epoch 47 TRAIN:   6%|▌         | 8/139 [00:04<00:55,  2.36it/s, epoch=47, grad_norm=221.918, loss=287.5532, lr=5.13e-04]
Epoch 47 TRAIN:   6%|▌         | 8/139 [00:04<00:55,  2.36it/s, epoch=47, grad_norm=167.048, loss=336.5163, lr=5.13e-04]
Epoch 47 TRAIN:   6%|▌         | 8/139 [00:04<00:55,  2.36it/s, epoch=47, grad_norm=98.023, loss=307.4827, lr=5.14e-04] 
Epoch 47 TRAIN:   6%|▌         | 8/139 [00:04<00:55,  2.36it/s, epoch=47, grad_norm=87.765, loss=327.9932, lr=5.14e-04]
Epoch 47 TRAIN:   6%|▌         | 8/139 [00:05<00:55,  2.36it/s, epoch=47, grad_norm=135.183, loss=297.9952, lr=5.14e-04]
Epoch 47 TRAIN:   9%|▊         | 12/139 [00:05<00:39,  3.19it/s, epoch=47, grad_norm=135.183, loss=297.9952, lr=5.14e-04]
Epoch 47 TRAIN:   9%|▊         | 12/139 [00:05<00:39,  3.19it/s, epoch=47, grad_norm=90.672, loss=296.8216, lr=5.14e-04] 
Epoch 47 TRAIN:   9%|▊         | 12/139 [00:05<00:39,  3.19it/s, epoch=47, grad_norm=104.026, loss=340.0308, lr=5.14e-04]
Epoch 47 TRAIN:   9%|▊         | 12/139 [00:05<00:39,  3.19it/s, epoch=47, grad_norm=87.319, loss=359.1505, lr=5.14e-04] 
Epoch 47 TRAIN:   9%|▊         | 12/139 [00:05<00:39,  3.19it/s, epoch=47, grad_norm=139.223, loss=290.6885, lr=5.14e-04]
Epoch 47 TRAIN:  12%|█▏        | 16/139 [00:05<00:32,  3.80it/s, epoch=47, grad_norm=139.223, loss=290.6885, lr=5.14e-04]
Epoch 47 TRAIN:  12%|█▏        | 16/139 [00:05<00:32,  3.80it/s, epoch=47, grad_norm=107.567, loss=330.7765, lr=5.14e-04]
Epoch 47 TRAIN:  12%|█▏        | 16/139 [00:06<00:32,  3.80it/s, epoch=47, grad_norm=112.318, loss=341.2342, lr=5.14e-04]
Epoch 47 TRAIN:  12%|█▏        | 16/139 [00:06<00:32,  3.80it/s, epoch=47, grad_norm=134.891, loss=276.8792, lr=5.14e-04]
Epoch 47 TRAIN:  12%|█▏        | 16/139 [00:06<00:32,  3.80it/s, epoch=47, grad_norm=115.505, loss=321.8163, lr=5.14e-04]
Epoch 47 TRAIN:  14%|█▍        | 20/139 [00:06<00:27,  4.36it/s, epoch=47, grad_norm=115.505, loss=321.8163, lr=5.14e-04]
Epoch 47 TRAIN:  14%|█▍        | 20/139 [00:06<00:27,  4.36it/s, epoch=47, grad_norm=103.290, loss=274.7444, lr=5.14e-04]
Epoch 47 TRAIN:  14%|█▍        | 20/139 [00:06<00:27,  4.36it/s, epoch=47, grad_norm=131.242, loss=335.4676, lr=5.14e-04]
Epoch 47 TRAIN:  14%|█▍        | 20/139 [00:07<00:27,  4.36it/s, epoch=47, grad_norm=157.557, loss=321.0764, lr=5.15e-04]
Epoch 47 TRAIN:  14%|█▍        | 20/139 [00:07<00:27,  4.36it/s, epoch=47, grad_norm=122.572, loss=307.4091, lr=5.15e-04]
Epoch 47 TRAIN:  17%|█▋        | 24/139 [00:07<00:25,  4.52it/s, epoch=47, grad_norm=122.572, loss=307.4091, lr=5.15e-04]
Epoch 47 TRAIN:  17%|█▋        | 24/139 [00:07<00:25,  4.52it/s, epoch=47, grad_norm=91.799, loss=291.9350, lr=5.15e-04] 
Epoch 47 TRAIN:  17%|█▋        | 24/139 [00:07<00:25,  4.52it/s, epoch=47, grad_norm=169.731, loss=323.4924, lr=5.15e-04]
Epoch 47 TRAIN:  17%|█▋        | 24/139 [00:07<00:25,  4.52it/s, epoch=47, grad_norm=88.850, loss=309.3675, lr=5.15e-04] 
Epoch 47 TRAIN:  19%|█▉        | 27/139 [00:07<00:23,  4.70it/s, epoch=47, grad_norm=88.850, loss=309.3675, lr=5.15e-04]
Epoch 47 TRAIN:  19%|█▉        | 27/139 [00:08<00:23,  4.70it/s, epoch=47, grad_norm=167.853, loss=238.8641, lr=5.15e-04]
Epoch 47 TRAIN:  19%|█▉        | 27/139 [00:08<00:23,  4.70it/s, epoch=47, grad_norm=82.716, loss=310.5827, lr=5.15e-04] 
Epoch 47 TRAIN:  19%|█▉        | 27/139 [00:08<00:23,  4.70it/s, epoch=47, grad_norm=110.514, loss=275.6888, lr=5.15e-04]
Epoch 47 TRAIN:  22%|██▏       | 30/139 [00:08<00:22,  4.95it/s, epoch=47, grad_norm=110.514, loss=275.6888, lr=5.15e-04]
Epoch 47 TRAIN:  22%|██▏       | 30/139 [00:08<00:22,  4.95it/s, epoch=47, grad_norm=99.092, loss=263.9454, lr=5.15e-04] 
Epoch 47 TRAIN:  22%|██▏       | 30/139 [00:08<00:22,  4.95it/s, epoch=47, grad_norm=85.840, loss=260.5894, lr=5.15e-04]
Epoch 47 TRAIN:  22%|██▏       | 30/139 [00:09<00:22,  4.95it/s, epoch=47, grad_norm=71.120, loss=273.8610, lr=5.15e-04]
Epoch 47 TRAIN:  24%|██▎       | 33/139 [00:09<00:22,  4.76it/s, epoch=47, grad_norm=71.120, loss=273.8610, lr=5.15e-04]
Epoch 47 TRAIN:  24%|██▎       | 33/139 [00:09<00:22,  4.76it/s, epoch=47, grad_norm=86.901, loss=279.8562, lr=5.15e-04]
Epoch 47 TRAIN:  24%|██▎       | 33/139 [00:09<00:22,  4.76it/s, epoch=47, grad_norm=104.479, loss=300.7454, lr=5.16e-04]
Epoch 47 TRAIN:  24%|██▎       | 33/139 [00:09<00:22,  4.76it/s, epoch=47, grad_norm=181.257, loss=313.4356, lr=5.16e-04]
Epoch 47 TRAIN:  24%|██▎       | 33/139 [00:09<00:22,  4.76it/s, epoch=47, grad_norm=161.207, loss=334.1854, lr=5.16e-04]
Epoch 47 TRAIN:  27%|██▋       | 37/139 [00:09<00:19,  5.13it/s, epoch=47, grad_norm=161.207, loss=334.1854, lr=5.16e-04]
Epoch 47 TRAIN:  27%|██▋       | 37/139 [00:09<00:19,  5.13it/s, epoch=47, grad_norm=124.917, loss=314.0098, lr=5.16e-04]
Epoch 47 TRAIN:  27%|██▋       | 37/139 [00:10<00:19,  5.13it/s, epoch=47, grad_norm=86.298, loss=254.9788, lr=5.16e-04] 
Epoch 47 TRAIN:  27%|██▋       | 37/139 [00:10<00:19,  5.13it/s, epoch=47, grad_norm=97.861, loss=278.7778, lr=5.16e-04]
Epoch 47 TRAIN:  29%|██▉       | 40/139 [00:10<00:19,  4.99it/s, epoch=47, grad_norm=97.861, loss=278.7778, lr=5.16e-04]
Epoch 47 TRAIN:  29%|██▉       | 40/139 [00:10<00:19,  4.99it/s, epoch=47, grad_norm=132.173, loss=222.5866, lr=5.16e-04]
Epoch 47 TRAIN:  29%|██▉       | 40/139 [00:10<00:19,  4.99it/s, epoch=47, grad_norm=219.663, loss=327.9094, lr=5.16e-04]
Epoch 47 TRAIN:  29%|██▉       | 40/139 [00:11<00:19,  4.99it/s, epoch=47, grad_norm=201.750, loss=328.3626, lr=5.16e-04]
Epoch 47 TRAIN:  31%|███       | 43/139 [00:11<00:19,  4.92it/s, epoch=47, grad_norm=201.750, loss=328.3626, lr=5.16e-04]
Epoch 47 TRAIN:  31%|███       | 43/139 [00:11<00:19,  4.92it/s, epoch=47, grad_norm=166.692, loss=331.9676, lr=5.16e-04]
Epoch 47 TRAIN:  31%|███       | 43/139 [00:11<00:19,  4.92it/s, epoch=47, grad_norm=123.433, loss=310.6429, lr=5.16e-04]
Epoch 47 TRAIN:  31%|███       | 43/139 [00:11<00:19,  4.92it/s, epoch=47, grad_norm=152.235, loss=335.1473, lr=5.16e-04]
Epoch 47 TRAIN:  33%|███▎      | 46/139 [00:11<00:19,  4.86it/s, epoch=47, grad_norm=152.235, loss=335.1473, lr=5.16e-04]
Epoch 47 TRAIN:  33%|███▎      | 46/139 [00:11<00:19,  4.86it/s, epoch=47, grad_norm=102.371, loss=312.2646, lr=5.16e-04]
Epoch 47 TRAIN:  33%|███▎      | 46/139 [00:12<00:19,  4.86it/s, epoch=47, grad_norm=75.761, loss=338.4108, lr=5.17e-04] 
Epoch 47 TRAIN:  33%|███▎      | 46/139 [00:12<00:19,  4.86it/s, epoch=47, grad_norm=115.943, loss=338.4955, lr=5.17e-04]
Epoch 47 TRAIN:  35%|███▌      | 49/139 [00:12<00:18,  4.92it/s, epoch=47, grad_norm=115.943, loss=338.4955, lr=5.17e-04]
Epoch 47 TRAIN:  35%|███▌      | 49/139 [00:12<00:18,  4.92it/s, epoch=47, grad_norm=131.970, loss=337.3977, lr=5.17e-04]
Epoch 47 TRAIN:  35%|███▌      | 49/139 [00:12<00:18,  4.92it/s, epoch=47, grad_norm=160.414, loss=345.5226, lr=5.17e-04]
Epoch 47 TRAIN:  35%|███▌      | 49/139 [00:12<00:18,  4.92it/s, epoch=47, grad_norm=174.322, loss=328.2368, lr=5.17e-04]
Epoch 47 TRAIN:  37%|███▋      | 52/139 [00:12<00:17,  5.05it/s, epoch=47, grad_norm=174.322, loss=328.2368, lr=5.17e-04]
Epoch 47 TRAIN:  37%|███▋      | 52/139 [00:13<00:17,  5.05it/s, epoch=47, grad_norm=107.960, loss=342.1786, lr=5.17e-04]
Epoch 47 TRAIN:  37%|███▋      | 52/139 [00:13<00:17,  5.05it/s, epoch=47, grad_norm=181.182, loss=282.5382, lr=5.17e-04]
Epoch 47 TRAIN:  37%|███▋      | 52/139 [00:13<00:17,  5.05it/s, epoch=47, grad_norm=80.516, loss=323.1330, lr=5.17e-04] 
Epoch 47 TRAIN:  40%|███▉      | 55/139 [00:13<00:17,  4.76it/s, epoch=47, grad_norm=80.516, loss=323.1330, lr=5.17e-04]
Epoch 47 TRAIN:  40%|███▉      | 55/139 [00:13<00:17,  4.76it/s, epoch=47, grad_norm=102.112, loss=332.2571, lr=5.17e-04]
Epoch 47 TRAIN:  40%|███▉      | 55/139 [00:13<00:17,  4.76it/s, epoch=47, grad_norm=84.028, loss=304.7489, lr=5.17e-04] 
Epoch 47 TRAIN:  40%|███▉      | 55/139 [00:14<00:17,  4.76it/s, epoch=47, grad_norm=89.111, loss=305.9084, lr=5.17e-04]
Epoch 47 TRAIN:  42%|████▏     | 58/139 [00:14<00:16,  4.80it/s, epoch=47, grad_norm=89.111, loss=305.9084, lr=5.17e-04]
Epoch 47 TRAIN:  42%|████▏     | 58/139 [00:14<00:16,  4.80it/s, epoch=47, grad_norm=133.992, loss=333.5249, lr=5.17e-04]
Epoch 47 TRAIN:  42%|████▏     | 58/139 [00:14<00:16,  4.80it/s, epoch=47, grad_norm=85.177, loss=290.6439, lr=5.18e-04] 
Epoch 47 TRAIN:  42%|████▏     | 58/139 [00:14<00:16,  4.80it/s, epoch=47, grad_norm=127.791, loss=280.8930, lr=5.18e-04]
Epoch 47 TRAIN:  44%|████▍     | 61/139 [00:14<00:16,  4.75it/s, epoch=47, grad_norm=127.791, loss=280.8930, lr=5.18e-04]
Epoch 47 TRAIN:  44%|████▍     | 61/139 [00:15<00:16,  4.75it/s, epoch=47, grad_norm=87.307, loss=278.4404, lr=5.18e-04] 
Epoch 47 TRAIN:  44%|████▍     | 61/139 [00:15<00:16,  4.75it/s, epoch=47, grad_norm=115.398, loss=301.3106, lr=5.18e-04]
Epoch 47 TRAIN:  44%|████▍     | 61/139 [00:15<00:16,  4.75it/s, epoch=47, grad_norm=128.442, loss=301.9805, lr=5.18e-04]
Epoch 47 TRAIN:  46%|████▌     | 64/139 [00:15<00:15,  4.77it/s, epoch=47, grad_norm=128.442, loss=301.9805, lr=5.18e-04]
Epoch 47 TRAIN:  46%|████▌     | 64/139 [00:15<00:15,  4.77it/s, epoch=47, grad_norm=134.530, loss=320.2582, lr=5.18e-04]
Epoch 47 TRAIN:  46%|████▌     | 64/139 [00:15<00:15,  4.77it/s, epoch=47, grad_norm=113.793, loss=310.6206, lr=5.18e-04]
Epoch 47 TRAIN:  46%|████▌     | 64/139 [00:16<00:15,  4.77it/s, epoch=47, grad_norm=112.534, loss=279.8824, lr=5.18e-04]
Epoch 47 TRAIN:  48%|████▊     | 67/139 [00:16<00:14,  4.82it/s, epoch=47, grad_norm=112.534, loss=279.8824, lr=5.18e-04]
Epoch 47 TRAIN:  48%|████▊     | 67/139 [00:16<00:14,  4.82it/s, epoch=47, grad_norm=171.492, loss=248.3761, lr=5.18e-04]
Epoch 47 TRAIN:  48%|████▊     | 67/139 [00:16<00:14,  4.82it/s, epoch=47, grad_norm=165.108, loss=329.3687, lr=5.18e-04]
Epoch 47 TRAIN:  48%|████▊     | 67/139 [00:16<00:14,  4.82it/s, epoch=47, grad_norm=105.225, loss=300.1244, lr=5.18e-04]
Epoch 47 TRAIN:  50%|█████     | 70/139 [00:16<00:14,  4.76it/s, epoch=47, grad_norm=105.225, loss=300.1244, lr=5.18e-04]
Epoch 47 TRAIN:  50%|█████     | 70/139 [00:16<00:14,  4.76it/s, epoch=47, grad_norm=169.724, loss=337.9717, lr=5.18e-04]
Epoch 47 TRAIN:  50%|█████     | 70/139 [00:17<00:14,  4.76it/s, epoch=47, grad_norm=109.366, loss=293.4546, lr=5.18e-04]
Epoch 47 TRAIN:  50%|█████     | 70/139 [00:17<00:14,  4.76it/s, epoch=47, grad_norm=126.162, loss=272.1447, lr=5.19e-04]
Epoch 47 TRAIN:  53%|█████▎    | 73/139 [00:17<00:13,  4.93it/s, epoch=47, grad_norm=126.162, loss=272.1447, lr=5.19e-04]
Epoch 47 TRAIN:  53%|█████▎    | 73/139 [00:17<00:13,  4.93it/s, epoch=47, grad_norm=136.525, loss=324.6237, lr=5.19e-04]
Epoch 47 TRAIN:  53%|█████▎    | 73/139 [00:17<00:13,  4.93it/s, epoch=47, grad_norm=169.318, loss=343.5157, lr=5.19e-04]
Epoch 47 TRAIN:  53%|█████▎    | 73/139 [00:18<00:13,  4.93it/s, epoch=47, grad_norm=101.033, loss=326.1270, lr=5.19e-04]
Epoch 47 TRAIN:  55%|█████▍    | 76/139 [00:18<00:14,  4.45it/s, epoch=47, grad_norm=101.033, loss=326.1270, lr=5.19e-04]
Epoch 47 TRAIN:  55%|█████▍    | 76/139 [00:18<00:14,  4.45it/s, epoch=47, grad_norm=302.414, loss=311.5606, lr=5.19e-04]
Epoch 47 TRAIN:  55%|█████▍    | 76/139 [00:18<00:14,  4.45it/s, epoch=47, grad_norm=172.626, loss=276.4785, lr=5.19e-04]
Epoch 47 TRAIN:  55%|█████▍    | 76/139 [00:18<00:14,  4.45it/s, epoch=47, grad_norm=171.677, loss=298.3003, lr=5.19e-04]
Epoch 47 TRAIN:  57%|█████▋    | 79/139 [00:18<00:12,  4.68it/s, epoch=47, grad_norm=171.677, loss=298.3003, lr=5.19e-04]
Epoch 47 TRAIN:  57%|█████▋    | 79/139 [00:18<00:12,  4.68it/s, epoch=47, grad_norm=125.047, loss=286.9518, lr=5.19e-04]
Epoch 47 TRAIN:  57%|█████▋    | 79/139 [00:19<00:12,  4.68it/s, epoch=47, grad_norm=87.136, loss=308.3741, lr=5.19e-04] 
Epoch 47 TRAIN:  57%|█████▋    | 79/139 [00:19<00:12,  4.68it/s, epoch=47, grad_norm=125.977, loss=321.5467, lr=5.19e-04]
Epoch 47 TRAIN:  59%|█████▉    | 82/139 [00:19<00:12,  4.55it/s, epoch=47, grad_norm=125.977, loss=321.5467, lr=5.19e-04]
Epoch 47 TRAIN:  59%|█████▉    | 82/139 [00:19<00:12,  4.55it/s, epoch=47, grad_norm=146.138, loss=339.4573, lr=5.19e-04]
Epoch 47 TRAIN:  59%|█████▉    | 82/139 [00:19<00:12,  4.55it/s, epoch=47, grad_norm=169.264, loss=342.4780, lr=5.19e-04]
Epoch 47 TRAIN:  59%|█████▉    | 82/139 [00:19<00:12,  4.55it/s, epoch=47, grad_norm=215.337, loss=293.2063, lr=5.20e-04]
Epoch 47 TRAIN:  61%|██████    | 85/139 [00:19<00:11,  4.59it/s, epoch=47, grad_norm=215.337, loss=293.2063, lr=5.20e-04]
Epoch 47 TRAIN:  61%|██████    | 85/139 [00:20<00:11,  4.59it/s, epoch=47, grad_norm=165.332, loss=318.8582, lr=5.20e-04]
Epoch 47 TRAIN:  61%|██████    | 85/139 [00:20<00:11,  4.59it/s, epoch=47, grad_norm=124.687, loss=316.7933, lr=5.20e-04]
Epoch 47 TRAIN:  61%|██████    | 85/139 [00:20<00:11,  4.59it/s, epoch=47, grad_norm=128.016, loss=265.2850, lr=5.20e-04]
Epoch 47 TRAIN:  63%|██████▎   | 88/139 [00:20<00:10,  4.67it/s, epoch=47, grad_norm=128.016, loss=265.2850, lr=5.20e-04]
Epoch 47 TRAIN:  63%|██████▎   | 88/139 [00:20<00:10,  4.67it/s, epoch=47, grad_norm=101.191, loss=278.2640, lr=5.20e-04]
Epoch 47 TRAIN:  63%|██████▎   | 88/139 [00:21<00:10,  4.67it/s, epoch=47, grad_norm=182.790, loss=277.7453, lr=5.20e-04]
Epoch 47 TRAIN:  63%|██████▎   | 88/139 [00:21<00:10,  4.67it/s, epoch=47, grad_norm=124.238, loss=259.6994, lr=5.20e-04]
Epoch 47 TRAIN:  65%|██████▌   | 91/139 [00:21<00:10,  4.52it/s, epoch=47, grad_norm=124.238, loss=259.6994, lr=5.20e-04]
Epoch 47 TRAIN:  65%|██████▌   | 91/139 [00:21<00:10,  4.52it/s, epoch=47, grad_norm=157.873, loss=275.7921, lr=5.20e-04]
Epoch 47 TRAIN:  65%|██████▌   | 91/139 [00:21<00:10,  4.52it/s, epoch=47, grad_norm=151.408, loss=254.2014, lr=5.20e-04]
Epoch 47 TRAIN:  65%|██████▌   | 91/139 [00:21<00:10,  4.52it/s, epoch=47, grad_norm=203.601, loss=306.1565, lr=5.20e-04]
Epoch 47 TRAIN:  68%|██████▊   | 94/139 [00:21<00:09,  4.67it/s, epoch=47, grad_norm=203.601, loss=306.1565, lr=5.20e-04]
Epoch 47 TRAIN:  68%|██████▊   | 94/139 [00:22<00:09,  4.67it/s, epoch=47, grad_norm=145.353, loss=240.7895, lr=5.20e-04]
Epoch 47 TRAIN:  68%|██████▊   | 94/139 [00:22<00:09,  4.67it/s, epoch=47, grad_norm=215.224, loss=312.2390, lr=5.20e-04]
Epoch 47 TRAIN:  68%|██████▊   | 94/139 [00:22<00:09,  4.67it/s, epoch=47, grad_norm=158.593, loss=288.3418, lr=5.20e-04]
Epoch 47 TRAIN:  70%|██████▉   | 97/139 [00:22<00:08,  4.76it/s, epoch=47, grad_norm=158.593, loss=288.3418, lr=5.20e-04]
Epoch 47 TRAIN:  70%|██████▉   | 97/139 [00:22<00:08,  4.76it/s, epoch=47, grad_norm=146.498, loss=204.5844, lr=5.21e-04]
Epoch 47 TRAIN:  70%|██████▉   | 97/139 [00:22<00:08,  4.76it/s, epoch=47, grad_norm=111.479, loss=213.7163, lr=5.21e-04]
Epoch 47 TRAIN:  70%|██████▉   | 97/139 [00:23<00:08,  4.76it/s, epoch=47, grad_norm=258.102, loss=398.3102, lr=5.21e-04]2026-04-16 20:15:59,975 DEBUG TRAIN | steps/sec 4.381| Batch 47/100 loss 398.310181 loss_att 342.226227 loss_ctc 529.172729 th_accuracy 0.326279 lr 5.2072e-04 grad_norm 258.101868 rank 0

Epoch 47 TRAIN:  72%|███████▏  | 100/139 [00:23<00:08,  4.66it/s, epoch=47, grad_norm=258.102, loss=398.3102, lr=5.21e-04]
Epoch 47 TRAIN:  72%|███████▏  | 100/139 [00:23<00:08,  4.66it/s, epoch=47, grad_norm=148.896, loss=335.0100, lr=5.21e-04]
Epoch 47 TRAIN:  72%|███████▏  | 100/139 [00:23<00:08,  4.66it/s, epoch=47, grad_norm=87.852, loss=204.5412, lr=5.21e-04] 
Epoch 47 TRAIN:  72%|███████▏  | 100/139 [00:23<00:08,  4.66it/s, epoch=47, grad_norm=154.422, loss=314.3833, lr=5.21e-04]
Epoch 47 TRAIN:  74%|███████▍  | 103/139 [00:23<00:07,  4.58it/s, epoch=47, grad_norm=154.422, loss=314.3833, lr=5.21e-04]
Epoch 47 TRAIN:  74%|███████▍  | 103/139 [00:24<00:07,  4.58it/s, epoch=47, grad_norm=171.850, loss=205.5938, lr=5.21e-04]
Epoch 47 TRAIN:  74%|███████▍  | 103/139 [00:24<00:07,  4.58it/s, epoch=47, grad_norm=149.564, loss=330.3804, lr=5.21e-04]
Epoch 47 TRAIN:  74%|███████▍  | 103/139 [00:24<00:07,  4.58it/s, epoch=47, grad_norm=124.976, loss=288.1529, lr=5.21e-04]
Epoch 47 TRAIN:  76%|███████▋  | 106/139 [00:24<00:07,  4.63it/s, epoch=47, grad_norm=124.976, loss=288.1529, lr=5.21e-04]
Epoch 47 TRAIN:  76%|███████▋  | 106/139 [00:24<00:07,  4.63it/s, epoch=47, grad_norm=135.522, loss=254.1436, lr=5.21e-04]
Epoch 47 TRAIN:  76%|███████▋  | 106/139 [00:24<00:07,  4.63it/s, epoch=47, grad_norm=151.999, loss=387.9719, lr=5.21e-04]
Epoch 47 TRAIN:  76%|███████▋  | 106/139 [00:24<00:07,  4.63it/s, epoch=47, grad_norm=277.061, loss=282.4374, lr=5.21e-04]
Epoch 47 TRAIN:  76%|███████▋  | 106/139 [00:25<00:07,  4.63it/s, epoch=47, grad_norm=127.070, loss=259.8351, lr=5.22e-04]
Epoch 47 TRAIN:  79%|███████▉  | 110/139 [00:25<00:05,  5.00it/s, epoch=47, grad_norm=127.070, loss=259.8351, lr=5.22e-04]
Epoch 47 TRAIN:  79%|███████▉  | 110/139 [00:25<00:05,  5.00it/s, epoch=47, grad_norm=76.048, loss=245.0373, lr=5.22e-04] 
Epoch 47 TRAIN:  79%|███████▉  | 110/139 [00:25<00:05,  5.00it/s, epoch=47, grad_norm=94.456, loss=230.3315, lr=5.22e-04]
Epoch 47 TRAIN:  79%|███████▉  | 110/139 [00:25<00:05,  5.00it/s, epoch=47, grad_norm=102.649, loss=273.0144, lr=5.22e-04]
Epoch 47 TRAIN:  81%|████████▏ | 113/139 [00:25<00:05,  4.90it/s, epoch=47, grad_norm=102.649, loss=273.0144, lr=5.22e-04]
Epoch 47 TRAIN:  81%|████████▏ | 113/139 [00:25<00:05,  4.90it/s, epoch=47, grad_norm=129.322, loss=267.4841, lr=5.22e-04]
Epoch 47 TRAIN:  81%|████████▏ | 113/139 [00:26<00:05,  4.90it/s, epoch=47, grad_norm=194.731, loss=317.9329, lr=5.22e-04]
Epoch 47 TRAIN:  81%|████████▏ | 113/139 [00:26<00:05,  4.90it/s, epoch=47, grad_norm=263.682, loss=350.9637, lr=5.22e-04]
Epoch 47 TRAIN:  83%|████████▎ | 116/139 [00:26<00:04,  5.00it/s, epoch=47, grad_norm=263.682, loss=350.9637, lr=5.22e-04]
Epoch 47 TRAIN:  83%|████████▎ | 116/139 [00:26<00:04,  5.00it/s, epoch=47, grad_norm=194.689, loss=335.9469, lr=5.22e-04]
Epoch 47 TRAIN:  83%|████████▎ | 116/139 [00:26<00:04,  5.00it/s, epoch=47, grad_norm=100.661, loss=269.5748, lr=5.22e-04]
Epoch 47 TRAIN:  83%|████████▎ | 116/139 [00:26<00:04,  5.00it/s, epoch=47, grad_norm=185.894, loss=383.9949, lr=5.22e-04]
Epoch 47 TRAIN:  86%|████████▌ | 119/139 [00:26<00:03,  5.16it/s, epoch=47, grad_norm=185.894, loss=383.9949, lr=5.22e-04]
Epoch 47 TRAIN:  86%|████████▌ | 119/139 [00:27<00:03,  5.16it/s, epoch=47, grad_norm=330.203, loss=326.9442, lr=5.22e-04]
Epoch 47 TRAIN:  86%|████████▌ | 119/139 [00:27<00:03,  5.16it/s, epoch=47, grad_norm=185.115, loss=274.7381, lr=5.22e-04]
Epoch 47 TRAIN:  86%|████████▌ | 119/139 [00:27<00:03,  5.16it/s, epoch=47, grad_norm=304.567, loss=261.9984, lr=5.22e-04]
Epoch 47 TRAIN:  88%|████████▊ | 122/139 [00:27<00:03,  4.79it/s, epoch=47, grad_norm=304.567, loss=261.9984, lr=5.22e-04]
Epoch 47 TRAIN:  88%|████████▊ | 122/139 [00:27<00:03,  4.79it/s, epoch=47, grad_norm=191.672, loss=266.6425, lr=5.23e-04]
Epoch 47 TRAIN:  88%|████████▊ | 122/139 [00:28<00:03,  4.79it/s, epoch=47, grad_norm=129.777, loss=279.0859, lr=5.23e-04]
Epoch 47 TRAIN:  88%|████████▊ | 122/139 [00:28<00:03,  4.79it/s, epoch=47, grad_norm=216.866, loss=347.6204, lr=5.23e-04]
Epoch 47 TRAIN:  90%|████████▉ | 125/139 [00:28<00:02,  4.72it/s, epoch=47, grad_norm=216.866, loss=347.6204, lr=5.23e-04]
Epoch 47 TRAIN:  90%|████████▉ | 125/139 [00:28<00:02,  4.72it/s, epoch=47, grad_norm=230.390, loss=399.8767, lr=5.23e-04]
Epoch 47 TRAIN:  90%|████████▉ | 125/139 [00:28<00:02,  4.72it/s, epoch=47, grad_norm=220.970, loss=327.6293, lr=5.23e-04]
Epoch 47 TRAIN:  90%|████████▉ | 125/139 [00:28<00:02,  4.72it/s, epoch=47, grad_norm=254.941, loss=347.2860, lr=5.23e-04]
Epoch 47 TRAIN:  90%|████████▉ | 125/139 [00:29<00:02,  4.72it/s, epoch=47, grad_norm=189.945, loss=306.9211, lr=5.23e-04]
Epoch 47 TRAIN:  93%|█████████▎| 129/139 [00:29<00:02,  4.75it/s, epoch=47, grad_norm=189.945, loss=306.9211, lr=5.23e-04]
Epoch 47 TRAIN:  93%|█████████▎| 129/139 [00:29<00:02,  4.75it/s, epoch=47, grad_norm=141.069, loss=305.9710, lr=5.23e-04]
Epoch 47 TRAIN:  93%|█████████▎| 129/139 [00:29<00:02,  4.75it/s, epoch=47, grad_norm=125.542, loss=326.6206, lr=5.23e-04]
Epoch 47 TRAIN:  93%|█████████▎| 129/139 [00:29<00:02,  4.75it/s, epoch=47, grad_norm=178.969, loss=310.2972, lr=5.23e-04]
Epoch 47 TRAIN:  93%|█████████▎| 129/139 [00:29<00:02,  4.75it/s, epoch=47, grad_norm=186.842, loss=356.9465, lr=5.23e-04]
Epoch 47 TRAIN:  96%|█████████▌| 133/139 [00:29<00:01,  5.14it/s, epoch=47, grad_norm=186.842, loss=356.9465, lr=5.23e-04]
Epoch 47 TRAIN:  96%|█████████▌| 133/139 [00:29<00:01,  5.14it/s, epoch=47, grad_norm=283.029, loss=412.7374, lr=5.23e-04]
Epoch 47 TRAIN:  96%|█████████▌| 133/139 [00:30<00:01,  5.14it/s, epoch=47, grad_norm=183.891, loss=262.6759, lr=5.24e-04]
Epoch 47 TRAIN:  96%|█████████▌| 133/139 [00:30<00:01,  5.14it/s, epoch=47, grad_norm=312.376, loss=372.4601, lr=5.24e-04]
Epoch 47 TRAIN:  96%|█████████▌| 133/139 [00:30<00:01,  5.14it/s, epoch=47, grad_norm=291.554, loss=382.6613, lr=5.24e-04]
Epoch 47 TRAIN:  99%|█████████▊| 137/139 [00:30<00:00,  5.40it/s, epoch=47, grad_norm=291.554, loss=382.6613, lr=5.24e-04]
Epoch 47 TRAIN:  99%|█████████▊| 137/139 [00:30<00:00,  4.50it/s, epoch=47, grad_norm=291.554, loss=382.6613, lr=5.24e-04]

CV:   0%|          | 0/6 [00:00<?, ?it/s]/home/rookie/Documents/batuka/projects/wenet/wenet/wenet/utils/train_utils.py:689: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  amp_autocast(enabled=dtype is not None,

CV:  17%|█▋        | 1/6 [00:00<00:03,  1.33it/s]
                                                 
2026-04-16 20:16:08,125 INFO Epoch 47 Step 6546 CV info lr 5.2368e-04 cv_loss 473.2072088068182 rank 0 acc 0.32460619509220123
2026-04-16 20:16:08,127 INFO [Rank 0] Checkpoint: save to checkpoint exp/conformer_mongolian/epoch_47.pt
2026-04-16 20:16:08,325 INFO Epoch 48 Step 6546 TRAIN info lr 5.2368e-04 rank 0
2026-04-16 20:16:08,341 INFO using accumulate grad, new batch size is 1 times larger than before

Epoch 48 TRAIN:   0%|          | 0/137 [00:00<?, ?it/s]
Epoch 48 TRAIN:   0%|          | 0/137 [00:03<?, ?it/s, epoch=48, grad_norm=330.954, loss=302.0568, lr=5.24e-04]
Epoch 48 TRAIN:   1%|          | 1/137 [00:03<08:43,  3.85s/it, epoch=48, grad_norm=330.954, loss=302.0568, lr=5.24e-04]
Epoch 48 TRAIN:   1%|          | 1/137 [00:04<08:43,  3.85s/it, epoch=48, grad_norm=195.889, loss=242.6616, lr=5.24e-04]
Epoch 48 TRAIN:   1%|          | 1/137 [00:04<08:43,  3.85s/it, epoch=48, grad_norm=250.688, loss=254.9272, lr=5.24e-04]
Epoch 48 TRAIN:   1%|          | 1/137 [00:04<08:43,  3.85s/it, epoch=48, grad_norm=406.521, loss=346.9658, lr=5.24e-04]
Epoch 48 TRAIN:   1%|          | 1/137 [00:04<08:43,  3.85s/it, epoch=48, grad_norm=196.958, loss=289.2470, lr=5.24e-04]
Epoch 48 TRAIN:   1%|          | 1/137 [00:04<08:43,  3.85s/it, epoch=48, grad_norm=315.422, loss=385.9872, lr=5.24e-04]
Epoch 48 TRAIN:   4%|▍         | 6/137 [00:04<01:14,  1.76it/s, epoch=48, grad_norm=315.422, loss=385.9872, lr=5.24e-04]
Epoch 48 TRAIN:   4%|▍         | 6/137 [00:04<01:14,  1.76it/s, epoch=48, grad_norm=242.148, loss=355.0007, lr=5.24e-04]
Epoch 48 TRAIN:   4%|▍         | 6/137 [00:04<01:14,  1.76it/s, epoch=48, grad_norm=325.958, loss=387.9299, lr=5.24e-04]
Epoch 48 TRAIN:   4%|▍         | 6/137 [00:04<01:14,  1.76it/s, epoch=48, grad_norm=166.684, loss=336.2682, lr=5.24e-04]
Epoch 48 TRAIN:   4%|▍         | 6/137 [00:04<01:14,  1.76it/s, epoch=48, grad_norm=91.514, loss=307.9416, lr=5.24e-04] 
Epoch 48 TRAIN:   4%|▍         | 6/137 [00:04<01:14,  1.76it/s, epoch=48, grad_norm=90.269, loss=329.1594, lr=5.25e-04]
Epoch 48 TRAIN:   8%|▊         | 11/137 [00:04<00:39,  3.15it/s, epoch=48, grad_norm=90.269, loss=329.1594, lr=5.25e-04]
Epoch 48 TRAIN:   8%|▊         | 11/137 [00:05<00:39,  3.15it/s, epoch=48, grad_norm=178.659, loss=342.3395, lr=5.25e-04]
Epoch 48 TRAIN:   8%|▊         | 11/137 [00:05<00:39,  3.15it/s, epoch=48, grad_norm=160.079, loss=300.2624, lr=5.25e-04]
Epoch 48 TRAIN:   8%|▊         | 11/137 [00:05<00:39,  3.15it/s, epoch=48, grad_norm=159.077, loss=256.0168, lr=5.25e-04]
Epoch 48 TRAIN:   8%|▊         | 11/137 [00:05<00:39,  3.15it/s, epoch=48, grad_norm=122.872, loss=354.2466, lr=5.25e-04]
Epoch 48 TRAIN:  11%|█         | 15/137 [00:05<00:31,  3.88it/s, epoch=48, grad_norm=122.872, loss=354.2466, lr=5.25e-04]
Epoch 48 TRAIN:  11%|█         | 15/137 [00:05<00:31,  3.88it/s, epoch=48, grad_norm=111.405, loss=347.4375, lr=5.25e-04]
Epoch 48 TRAIN:  11%|█         | 15/137 [00:05<00:31,  3.88it/s, epoch=48, grad_norm=191.752, loss=284.1094, lr=5.25e-04]
Epoch 48 TRAIN:  11%|█         | 15/137 [00:06<00:31,  3.88it/s, epoch=48, grad_norm=88.036, loss=318.1899, lr=5.25e-04] 
Epoch 48 TRAIN:  11%|█         | 15/137 [00:06<00:31,  3.88it/s, epoch=48, grad_norm=89.147, loss=338.4547, lr=5.25e-04]
Epoch 48 TRAIN:  14%|█▍        | 19/137 [00:06<00:26,  4.46it/s, epoch=48, grad_norm=89.147, loss=338.4547, lr=5.25e-04]
Epoch 48 TRAIN:  14%|█▍        | 19/137 [00:06<00:26,  4.46it/s, epoch=48, grad_norm=72.880, loss=295.2592, lr=5.25e-04]
Epoch 48 TRAIN:  14%|█▍        | 19/137 [00:06<00:26,  4.46it/s, epoch=48, grad_norm=78.756, loss=279.5139, lr=5.25e-04]
Epoch 48 TRAIN:  14%|█▍        | 19/137 [00:06<00:26,  4.46it/s, epoch=48, grad_norm=80.837, loss=283.4524, lr=5.25e-04]
Epoch 48 TRAIN:  14%|█▍        | 19/137 [00:07<00:26,  4.46it/s, epoch=48, grad_norm=113.793, loss=269.3454, lr=5.26e-04]
Epoch 48 TRAIN:  17%|█▋        | 23/137 [00:07<00:24,  4.64it/s, epoch=48, grad_norm=113.793, loss=269.3454, lr=5.26e-04]
Epoch 48 TRAIN:  17%|█▋        | 23/137 [00:07<00:24,  4.64it/s, epoch=48, grad_norm=161.085, loss=338.7923, lr=5.26e-04]
Epoch 48 TRAIN:  17%|█▋        | 23/137 [00:07<00:24,  4.64it/s, epoch=48, grad_norm=75.072, loss=288.2773, lr=5.26e-04] 
Epoch 48 TRAIN:  17%|█▋        | 23/137 [00:07<00:24,  4.64it/s, epoch=48, grad_norm=79.833, loss=307.1378, lr=5.26e-04]
Epoch 48 TRAIN:  17%|█▋        | 23/137 [00:07<00:24,  4.64it/s, epoch=48, grad_norm=89.993, loss=277.3933, lr=5.26e-04]
Epoch 48 TRAIN:  20%|█▉        | 27/137 [00:07<00:22,  4.90it/s, epoch=48, grad_norm=89.993, loss=277.3933, lr=5.26e-04]
Epoch 48 TRAIN:  20%|█▉        | 27/137 [00:08<00:22,  4.90it/s, epoch=48, grad_norm=127.732, loss=339.2404, lr=5.26e-04]
Epoch 48 TRAIN:  20%|█▉        | 27/137 [00:08<00:22,  4.90it/s, epoch=48, grad_norm=99.439, loss=319.3703, lr=5.26e-04] 
Epoch 48 TRAIN:  20%|█▉        | 27/137 [00:08<00:22,  4.90it/s, epoch=48, grad_norm=100.896, loss=347.8176, lr=5.26e-04]
Epoch 48 TRAIN:  22%|██▏       | 30/137 [00:08<00:21,  4.96it/s, epoch=48, grad_norm=100.896, loss=347.8176, lr=5.26e-04]
Epoch 48 TRAIN:  22%|██▏       | 30/137 [00:08<00:21,  4.96it/s, epoch=48, grad_norm=91.527, loss=309.9675, lr=5.26e-04] 
Epoch 48 TRAIN:  22%|██▏       | 30/137 [00:08<00:21,  4.96it/s, epoch=48, grad_norm=145.832, loss=359.3635, lr=5.26e-04]
Epoch 48 TRAIN:  22%|██▏       | 30/137 [00:08<00:21,  4.96it/s, epoch=48, grad_norm=159.438, loss=296.8957, lr=5.26e-04]
Epoch 48 TRAIN:  24%|██▍       | 33/137 [00:08<00:20,  5.15it/s, epoch=48, grad_norm=159.438, loss=296.8957, lr=5.26e-04]
Epoch 48 TRAIN:  24%|██▍       | 33/137 [00:09<00:20,  5.15it/s, epoch=48, grad_norm=110.927, loss=300.6750, lr=5.26e-04]
Epoch 48 TRAIN:  24%|██▍       | 33/137 [00:09<00:20,  5.15it/s, epoch=48, grad_norm=145.886, loss=311.3093, lr=5.26e-04]
Epoch 48 TRAIN:  24%|██▍       | 33/137 [00:09<00:20,  5.15it/s, epoch=48, grad_norm=127.285, loss=299.3037, lr=5.27e-04]
Epoch 48 TRAIN:  26%|██▋       | 36/137 [00:09<00:19,  5.29it/s, epoch=48, grad_norm=127.285, loss=299.3037, lr=5.27e-04]
Epoch 48 TRAIN:  26%|██▋       | 36/137 [00:09<00:19,  5.29it/s, epoch=48, grad_norm=97.964, loss=319.0493, lr=5.27e-04] 
Epoch 48 TRAIN:  26%|██▋       | 36/137 [00:09<00:19,  5.29it/s, epoch=48, grad_norm=182.457, loss=302.1301, lr=5.27e-04]
Epoch 48 TRAIN:  26%|██▋       | 36/137 [00:10<00:19,  5.29it/s, epoch=48, grad_norm=142.774, loss=306.4793, lr=5.27e-04]
Epoch 48 TRAIN:  28%|██▊       | 39/137 [00:10<00:19,  4.99it/s, epoch=48, grad_norm=142.774, loss=306.4793, lr=5.27e-04]
Epoch 48 TRAIN:  28%|██▊       | 39/137 [00:10<00:19,  4.99it/s, epoch=48, grad_norm=168.088, loss=316.0521, lr=5.27e-04]
Epoch 48 TRAIN:  28%|██▊       | 39/137 [00:10<00:19,  4.99it/s, epoch=48, grad_norm=105.182, loss=284.7419, lr=5.27e-04]
Epoch 48 TRAIN:  28%|██▊       | 39/137 [00:10<00:19,  4.99it/s, epoch=48, grad_norm=124.972, loss=250.8833, lr=5.27e-04]
Epoch 48 TRAIN:  31%|███       | 42/137 [00:10<00:18,  5.02it/s, epoch=48, grad_norm=124.972, loss=250.8833, lr=5.27e-04]
Epoch 48 TRAIN:  31%|███       | 42/137 [00:10<00:18,  5.02it/s, epoch=48, grad_norm=173.639, loss=341.3545, lr=5.27e-04]
Epoch 48 TRAIN:  31%|███       | 42/137 [00:11<00:18,  5.02it/s, epoch=48, grad_norm=112.285, loss=307.8339, lr=5.27e-04]
Epoch 48 TRAIN:  31%|███       | 42/137 [00:11<00:18,  5.02it/s, epoch=48, grad_norm=117.308, loss=304.9405, lr=5.27e-04]
Epoch 48 TRAIN:  33%|███▎      | 45/137 [00:11<00:18,  5.05it/s, epoch=48, grad_norm=117.308, loss=304.9405, lr=5.27e-04]
Epoch 48 TRAIN:  33%|███▎      | 45/137 [00:11<00:18,  5.05it/s, epoch=48, grad_norm=88.239, loss=281.5681, lr=5.27e-04] 
Epoch 48 TRAIN:  33%|███▎      | 45/137 [00:11<00:18,  5.05it/s, epoch=48, grad_norm=80.552, loss=300.3085, lr=5.27e-04]
Epoch 48 TRAIN:  33%|███▎      | 45/137 [00:12<00:18,  5.05it/s, epoch=48, grad_norm=145.145, loss=292.7256, lr=5.28e-04]
Epoch 48 TRAIN:  35%|███▌      | 48/137 [00:12<00:18,  4.78it/s, epoch=48, grad_norm=145.145, loss=292.7256, lr=5.28e-04]
Epoch 48 TRAIN:  35%|███▌      | 48/137 [00:12<00:18,  4.78it/s, epoch=48, grad_norm=118.627, loss=302.3604, lr=5.28e-04]
Epoch 48 TRAIN:  35%|███▌      | 48/137 [00:12<00:18,  4.78it/s, epoch=48, grad_norm=113.702, loss=316.0772, lr=5.28e-04]
Epoch 48 TRAIN:  35%|███▌      | 48/137 [00:12<00:18,  4.78it/s, epoch=48, grad_norm=106.727, loss=320.3708, lr=5.28e-04]
Epoch 48 TRAIN:  37%|███▋      | 51/137 [00:12<00:17,  4.82it/s, epoch=48, grad_norm=106.727, loss=320.3708, lr=5.28e-04]
Epoch 48 TRAIN:  37%|███▋      | 51/137 [00:12<00:17,  4.82it/s, epoch=48, grad_norm=136.819, loss=299.5185, lr=5.28e-04]
Epoch 48 TRAIN:  37%|███▋      | 51/137 [00:13<00:17,  4.82it/s, epoch=48, grad_norm=123.326, loss=262.1126, lr=5.28e-04]
Epoch 48 TRAIN:  37%|███▋      | 51/137 [00:13<00:17,  4.82it/s, epoch=48, grad_norm=88.319, loss=298.0012, lr=5.28e-04] 
Epoch 48 TRAIN:  39%|███▉      | 54/137 [00:13<00:17,  4.69it/s, epoch=48, grad_norm=88.319, loss=298.0012, lr=5.28e-04]
Epoch 48 TRAIN:  39%|███▉      | 54/137 [00:13<00:17,  4.69it/s, epoch=48, grad_norm=87.970, loss=316.5198, lr=5.28e-04]
Epoch 48 TRAIN:  39%|███▉      | 54/137 [00:13<00:17,  4.69it/s, epoch=48, grad_norm=173.655, loss=274.5454, lr=5.28e-04]
Epoch 48 TRAIN:  39%|███▉      | 54/137 [00:13<00:17,  4.69it/s, epoch=48, grad_norm=207.336, loss=296.1738, lr=5.28e-04]
Epoch 48 TRAIN:  42%|████▏     | 57/137 [00:13<00:16,  4.97it/s, epoch=48, grad_norm=207.336, loss=296.1738, lr=5.28e-04]
Epoch 48 TRAIN:  42%|████▏     | 57/137 [00:14<00:16,  4.97it/s, epoch=48, grad_norm=124.475, loss=336.8711, lr=5.28e-04]
Epoch 48 TRAIN:  42%|████▏     | 57/137 [00:14<00:16,  4.97it/s, epoch=48, grad_norm=77.925, loss=276.4650, lr=5.28e-04] 
Epoch 48 TRAIN:  42%|████▏     | 57/137 [00:14<00:16,  4.97it/s, epoch=48, grad_norm=186.997, loss=332.5460, lr=5.28e-04]
Epoch 48 TRAIN:  44%|████▍     | 60/137 [00:14<00:16,  4.73it/s, epoch=48, grad_norm=186.997, loss=332.5460, lr=5.28e-04]
Epoch 48 TRAIN:  44%|████▍     | 60/137 [00:14<00:16,  4.73it/s, epoch=48, grad_norm=145.396, loss=304.3061, lr=5.29e-04]
Epoch 48 TRAIN:  44%|████▍     | 60/137 [00:14<00:16,  4.73it/s, epoch=48, grad_norm=98.839, loss=271.5853, lr=5.29e-04] 
Epoch 48 TRAIN:  44%|████▍     | 60/137 [00:15<00:16,  4.73it/s, epoch=48, grad_norm=128.231, loss=312.9282, lr=5.29e-04]
Epoch 48 TRAIN:  46%|████▌     | 63/137 [00:15<00:16,  4.60it/s, epoch=48, grad_norm=128.231, loss=312.9282, lr=5.29e-04]
Epoch 48 TRAIN:  46%|████▌     | 63/137 [00:15<00:16,  4.60it/s, epoch=48, grad_norm=108.884, loss=329.5279, lr=5.29e-04]
Epoch 48 TRAIN:  46%|████▌     | 63/137 [00:15<00:16,  4.60it/s, epoch=48, grad_norm=86.770, loss=302.0859, lr=5.29e-04] 
Epoch 48 TRAIN:  46%|████▌     | 63/137 [00:15<00:16,  4.60it/s, epoch=48, grad_norm=126.401, loss=279.2452, lr=5.29e-04]
Epoch 48 TRAIN:  48%|████▊     | 66/137 [00:15<00:14,  4.93it/s, epoch=48, grad_norm=126.401, loss=279.2452, lr=5.29e-04]
Epoch 48 TRAIN:  48%|████▊     | 66/137 [00:15<00:14,  4.93it/s, epoch=48, grad_norm=149.739, loss=339.1537, lr=5.29e-04]
Epoch 48 TRAIN:  48%|████▊     | 66/137 [00:16<00:14,  4.93it/s, epoch=48, grad_norm=145.453, loss=286.4223, lr=5.29e-04]
Epoch 48 TRAIN:  48%|████▊     | 66/137 [00:16<00:14,  4.93it/s, epoch=48, grad_norm=185.394, loss=307.0346, lr=5.29e-04]
Epoch 48 TRAIN:  50%|█████     | 69/137 [00:16<00:13,  4.93it/s, epoch=48, grad_norm=185.394, loss=307.0346, lr=5.29e-04]
Epoch 48 TRAIN:  50%|█████     | 69/137 [00:16<00:13,  4.93it/s, epoch=48, grad_norm=148.152, loss=253.7844, lr=5.29e-04]
Epoch 48 TRAIN:  50%|█████     | 69/137 [00:16<00:13,  4.93it/s, epoch=48, grad_norm=110.258, loss=290.9936, lr=5.29e-04]
Epoch 48 TRAIN:  50%|█████     | 69/137 [00:17<00:13,  4.93it/s, epoch=48, grad_norm=140.462, loss=326.9400, lr=5.29e-04]
Epoch 48 TRAIN:  53%|█████▎    | 72/137 [00:17<00:13,  4.71it/s, epoch=48, grad_norm=140.462, loss=326.9400, lr=5.29e-04]
Epoch 48 TRAIN:  53%|█████▎    | 72/137 [00:17<00:13,  4.71it/s, epoch=48, grad_norm=155.113, loss=342.2396, lr=5.30e-04]
Epoch 48 TRAIN:  53%|█████▎    | 72/137 [00:17<00:13,  4.71it/s, epoch=48, grad_norm=127.720, loss=281.5128, lr=5.30e-04]
Epoch 48 TRAIN:  53%|█████▎    | 72/137 [00:17<00:13,  4.71it/s, epoch=48, grad_norm=115.030, loss=267.5623, lr=5.30e-04]
Epoch 48 TRAIN:  55%|█████▍    | 75/137 [00:17<00:13,  4.67it/s, epoch=48, grad_norm=115.030, loss=267.5623, lr=5.30e-04]
Epoch 48 TRAIN:  55%|█████▍    | 75/137 [00:17<00:13,  4.67it/s, epoch=48, grad_norm=117.706, loss=356.7261, lr=5.30e-04]
Epoch 48 TRAIN:  55%|█████▍    | 75/137 [00:18<00:13,  4.67it/s, epoch=48, grad_norm=121.030, loss=306.9801, lr=5.30e-04]
Epoch 48 TRAIN:  55%|█████▍    | 75/137 [00:18<00:13,  4.67it/s, epoch=48, grad_norm=111.502, loss=259.7524, lr=5.30e-04]
Epoch 48 TRAIN:  57%|█████▋    | 78/137 [00:18<00:12,  4.79it/s, epoch=48, grad_norm=111.502, loss=259.7524, lr=5.30e-04]
Epoch 48 TRAIN:  57%|█████▋    | 78/137 [00:18<00:12,  4.79it/s, epoch=48, grad_norm=105.163, loss=323.4048, lr=5.30e-04]
Epoch 48 TRAIN:  57%|█████▋    | 78/137 [00:18<00:12,  4.79it/s, epoch=48, grad_norm=155.410, loss=324.8515, lr=5.30e-04]
Epoch 48 TRAIN:  57%|█████▋    | 78/137 [00:18<00:12,  4.79it/s, epoch=48, grad_norm=147.843, loss=347.6794, lr=5.30e-04]
Epoch 48 TRAIN:  59%|█████▉    | 81/137 [00:18<00:12,  4.65it/s, epoch=48, grad_norm=147.843, loss=347.6794, lr=5.30e-04]
Epoch 48 TRAIN:  59%|█████▉    | 81/137 [00:19<00:12,  4.65it/s, epoch=48, grad_norm=225.189, loss=233.4576, lr=5.30e-04]
Epoch 48 TRAIN:  59%|█████▉    | 81/137 [00:19<00:12,  4.65it/s, epoch=48, grad_norm=108.162, loss=308.6935, lr=5.30e-04]
Epoch 48 TRAIN:  59%|█████▉    | 81/137 [00:19<00:12,  4.65it/s, epoch=48, grad_norm=236.920, loss=237.3583, lr=5.30e-04]
Epoch 48 TRAIN:  61%|██████▏   | 84/137 [00:19<00:11,  4.81it/s, epoch=48, grad_norm=236.920, loss=237.3583, lr=5.30e-04]
Epoch 48 TRAIN:  61%|██████▏   | 84/137 [00:19<00:11,  4.81it/s, epoch=48, grad_norm=138.061, loss=294.6471, lr=5.30e-04]
Epoch 48 TRAIN:  61%|██████▏   | 84/137 [00:19<00:11,  4.81it/s, epoch=48, grad_norm=166.715, loss=179.0384, lr=5.31e-04]
Epoch 48 TRAIN:  61%|██████▏   | 84/137 [00:20<00:11,  4.81it/s, epoch=48, grad_norm=268.363, loss=327.7613, lr=5.31e-04]
Epoch 48 TRAIN:  64%|██████▎   | 87/137 [00:20<00:10,  4.80it/s, epoch=48, grad_norm=268.363, loss=327.7613, lr=5.31e-04]
Epoch 48 TRAIN:  64%|██████▎   | 87/137 [00:20<00:10,  4.80it/s, epoch=48, grad_norm=186.317, loss=297.3902, lr=5.31e-04]
Epoch 48 TRAIN:  64%|██████▎   | 87/137 [00:20<00:10,  4.80it/s, epoch=48, grad_norm=118.718, loss=293.3123, lr=5.31e-04]
Epoch 48 TRAIN:  64%|██████▎   | 87/137 [00:20<00:10,  4.80it/s, epoch=48, grad_norm=118.684, loss=275.9550, lr=5.31e-04]
Epoch 48 TRAIN:  66%|██████▌   | 90/137 [00:20<00:10,  4.52it/s, epoch=48, grad_norm=118.684, loss=275.9550, lr=5.31e-04]
Epoch 48 TRAIN:  66%|██████▌   | 90/137 [00:21<00:10,  4.52it/s, epoch=48, grad_norm=133.952, loss=285.7125, lr=5.31e-04]
Epoch 48 TRAIN:  66%|██████▌   | 90/137 [00:21<00:10,  4.52it/s, epoch=48, grad_norm=106.356, loss=319.9985, lr=5.31e-04]
Epoch 48 TRAIN:  66%|██████▌   | 90/137 [00:21<00:10,  4.52it/s, epoch=48, grad_norm=112.712, loss=242.5370, lr=5.31e-04]
Epoch 48 TRAIN:  68%|██████▊   | 93/137 [00:21<00:09,  4.74it/s, epoch=48, grad_norm=112.712, loss=242.5370, lr=5.31e-04]
Epoch 48 TRAIN:  68%|██████▊   | 93/137 [00:21<00:09,  4.74it/s, epoch=48, grad_norm=196.384, loss=214.4000, lr=5.31e-04]
Epoch 48 TRAIN:  68%|██████▊   | 93/137 [00:21<00:09,  4.74it/s, epoch=48, grad_norm=132.372, loss=309.8866, lr=5.31e-04]
Epoch 48 TRAIN:  68%|██████▊   | 93/137 [00:22<00:09,  4.74it/s, epoch=48, grad_norm=119.520, loss=323.2597, lr=5.31e-04]
Epoch 48 TRAIN:  70%|███████   | 96/137 [00:22<00:08,  4.85it/s, epoch=48, grad_norm=119.520, loss=323.2597, lr=5.31e-04]
Epoch 48 TRAIN:  70%|███████   | 96/137 [00:22<00:08,  4.85it/s, epoch=48, grad_norm=95.582, loss=300.8295, lr=5.31e-04] 
Epoch 48 TRAIN:  70%|███████   | 96/137 [00:22<00:08,  4.85it/s, epoch=48, grad_norm=206.456, loss=275.0577, lr=5.32e-04]
Epoch 48 TRAIN:  70%|███████   | 96/137 [00:22<00:08,  4.85it/s, epoch=48, grad_norm=112.512, loss=271.2258, lr=5.32e-04]
Epoch 48 TRAIN:  72%|███████▏  | 99/137 [00:22<00:07,  5.03it/s, epoch=48, grad_norm=112.512, loss=271.2258, lr=5.32e-04]
Epoch 48 TRAIN:  72%|███████▏  | 99/137 [00:22<00:07,  5.03it/s, epoch=48, grad_norm=158.921, loss=229.9604, lr=5.32e-04]2026-04-16 20:16:31,282 DEBUG TRAIN | steps/sec 4.376| Batch 48/100 loss 229.960403 loss_att 196.333160 loss_ctc 308.423950 th_accuracy 0.296970 lr 5.3168e-04 grad_norm 158.920654 rank 0

Epoch 48 TRAIN:  72%|███████▏  | 99/137 [00:23<00:07,  5.03it/s, epoch=48, grad_norm=172.941, loss=291.9750, lr=5.32e-04]
Epoch 48 TRAIN:  72%|███████▏  | 99/137 [00:23<00:07,  5.03it/s, epoch=48, grad_norm=139.135, loss=217.3295, lr=5.32e-04]
Epoch 48 TRAIN:  74%|███████▍  | 102/137 [00:23<00:07,  4.81it/s, epoch=48, grad_norm=139.135, loss=217.3295, lr=5.32e-04]
Epoch 48 TRAIN:  74%|███████▍  | 102/137 [00:23<00:07,  4.81it/s, epoch=48, grad_norm=204.785, loss=331.9996, lr=5.32e-04]
Epoch 48 TRAIN:  74%|███████▍  | 102/137 [00:23<00:07,  4.81it/s, epoch=48, grad_norm=109.892, loss=286.0665, lr=5.32e-04]
Epoch 48 TRAIN:  74%|███████▍  | 102/137 [00:23<00:07,  4.81it/s, epoch=48, grad_norm=106.243, loss=265.9875, lr=5.32e-04]
Epoch 48 TRAIN:  77%|███████▋  | 105/137 [00:23<00:06,  4.68it/s, epoch=48, grad_norm=106.243, loss=265.9875, lr=5.32e-04]
Epoch 48 TRAIN:  77%|███████▋  | 105/137 [00:24<00:06,  4.68it/s, epoch=48, grad_norm=205.406, loss=328.5451, lr=5.32e-04]
Epoch 48 TRAIN:  77%|███████▋  | 105/137 [00:24<00:06,  4.68it/s, epoch=48, grad_norm=209.685, loss=339.2273, lr=5.32e-04]
Epoch 48 TRAIN:  77%|███████▋  | 105/137 [00:24<00:06,  4.68it/s, epoch=48, grad_norm=184.744, loss=347.8751, lr=5.32e-04]
Epoch 48 TRAIN:  79%|███████▉  | 108/137 [00:24<00:06,  4.73it/s, epoch=48, grad_norm=184.744, loss=347.8751, lr=5.32e-04]
Epoch 48 TRAIN:  79%|███████▉  | 108/137 [00:24<00:06,  4.73it/s, epoch=48, grad_norm=142.798, loss=269.8649, lr=5.32e-04]
Epoch 48 TRAIN:  79%|███████▉  | 108/137 [00:25<00:06,  4.73it/s, epoch=48, grad_norm=215.922, loss=234.7089, lr=5.32e-04]
Epoch 48 TRAIN:  79%|███████▉  | 108/137 [00:25<00:06,  4.73it/s, epoch=48, grad_norm=144.533, loss=345.4781, lr=5.33e-04]
Epoch 48 TRAIN:  81%|████████  | 111/137 [00:25<00:05,  4.71it/s, epoch=48, grad_norm=144.533, loss=345.4781, lr=5.33e-04]
Epoch 48 TRAIN:  81%|████████  | 111/137 [00:25<00:05,  4.71it/s, epoch=48, grad_norm=140.107, loss=354.3942, lr=5.33e-04]
Epoch 48 TRAIN:  81%|████████  | 111/137 [00:25<00:05,  4.71it/s, epoch=48, grad_norm=277.798, loss=319.2002, lr=5.33e-04]
Epoch 48 TRAIN:  81%|████████  | 111/137 [00:25<00:05,  4.71it/s, epoch=48, grad_norm=99.161, loss=252.1391, lr=5.33e-04] 
Epoch 48 TRAIN:  83%|████████▎ | 114/137 [00:25<00:04,  4.99it/s, epoch=48, grad_norm=99.161, loss=252.1391, lr=5.33e-04]
Epoch 48 TRAIN:  83%|████████▎ | 114/137 [00:25<00:04,  4.99it/s, epoch=48, grad_norm=103.037, loss=254.7359, lr=5.33e-04]
Epoch 48 TRAIN:  83%|████████▎ | 114/137 [00:26<00:04,  4.99it/s, epoch=48, grad_norm=185.782, loss=309.7835, lr=5.33e-04]
Epoch 48 TRAIN:  83%|████████▎ | 114/137 [00:26<00:04,  4.99it/s, epoch=48, grad_norm=191.192, loss=320.0112, lr=5.33e-04]
Epoch 48 TRAIN:  85%|████████▌ | 117/137 [00:26<00:04,  4.78it/s, epoch=48, grad_norm=191.192, loss=320.0112, lr=5.33e-04]
Epoch 48 TRAIN:  85%|████████▌ | 117/137 [00:26<00:04,  4.78it/s, epoch=48, grad_norm=121.998, loss=246.1874, lr=5.33e-04]
Epoch 48 TRAIN:  85%|████████▌ | 117/137 [00:26<00:04,  4.78it/s, epoch=48, grad_norm=234.189, loss=231.8179, lr=5.33e-04]
Epoch 48 TRAIN:  85%|████████▌ | 117/137 [00:27<00:04,  4.78it/s, epoch=48, grad_norm=153.499, loss=319.4657, lr=5.33e-04]
Epoch 48 TRAIN:  88%|████████▊ | 120/137 [00:27<00:03,  4.76it/s, epoch=48, grad_norm=153.499, loss=319.4657, lr=5.33e-04]
Epoch 48 TRAIN:  88%|████████▊ | 120/137 [00:27<00:03,  4.76it/s, epoch=48, grad_norm=116.120, loss=287.2166, lr=5.33e-04]
Epoch 48 TRAIN:  88%|████████▊ | 120/137 [00:27<00:03,  4.76it/s, epoch=48, grad_norm=142.291, loss=306.4120, lr=5.33e-04]
Epoch 48 TRAIN:  88%|████████▊ | 120/137 [00:27<00:03,  4.76it/s, epoch=48, grad_norm=142.648, loss=314.7717, lr=5.34e-04]
Epoch 48 TRAIN:  90%|████████▉ | 123/137 [00:27<00:02,  4.79it/s, epoch=48, grad_norm=142.648, loss=314.7717, lr=5.34e-04]
Epoch 48 TRAIN:  90%|████████▉ | 123/137 [00:27<00:02,  4.79it/s, epoch=48, grad_norm=154.496, loss=334.8153, lr=5.34e-04]
Epoch 48 TRAIN:  90%|████████▉ | 123/137 [00:27<00:02,  4.79it/s, epoch=48, grad_norm=154.945, loss=267.8342, lr=5.34e-04]
Epoch 48 TRAIN:  90%|████████▉ | 123/137 [00:28<00:02,  4.79it/s, epoch=48, grad_norm=236.609, loss=282.6810, lr=5.34e-04]
Epoch 48 TRAIN:  90%|████████▉ | 123/137 [00:28<00:02,  4.79it/s, epoch=48, grad_norm=292.712, loss=308.3457, lr=5.34e-04]
Epoch 48 TRAIN:  90%|████████▉ | 123/137 [00:28<00:02,  4.79it/s, epoch=48, grad_norm=339.462, loss=430.1528, lr=5.34e-04]
Epoch 48 TRAIN:  93%|█████████▎| 128/137 [00:28<00:01,  5.66it/s, epoch=48, grad_norm=339.462, loss=430.1528, lr=5.34e-04]
Epoch 48 TRAIN:  93%|█████████▎| 128/137 [00:28<00:01,  5.66it/s, epoch=48, grad_norm=143.023, loss=274.0995, lr=5.34e-04]
Epoch 48 TRAIN:  93%|█████████▎| 128/137 [00:28<00:01,  5.66it/s, epoch=48, grad_norm=257.140, loss=365.3083, lr=5.34e-04]
Epoch 48 TRAIN:  93%|█████████▎| 128/137 [00:29<00:01,  5.66it/s, epoch=48, grad_norm=144.267, loss=298.9177, lr=5.34e-04]
Epoch 48 TRAIN:  96%|█████████▌| 131/137 [00:29<00:01,  5.00it/s, epoch=48, grad_norm=144.267, loss=298.9177, lr=5.34e-04]
Epoch 48 TRAIN:  96%|█████████▌| 131/137 [00:29<00:01,  5.00it/s, epoch=48, grad_norm=178.144, loss=372.6677, lr=5.34e-04]
Epoch 48 TRAIN:  96%|█████████▌| 131/137 [00:29<00:01,  5.00it/s, epoch=48, grad_norm=199.112, loss=306.4103, lr=5.34e-04]
Epoch 48 TRAIN:  96%|█████████▌| 131/137 [00:29<00:01,  5.00it/s, epoch=48, grad_norm=226.637, loss=367.2985, lr=5.34e-04]
Epoch 48 TRAIN:  96%|█████████▌| 131/137 [00:29<00:01,  5.00it/s, epoch=48, grad_norm=177.013, loss=333.7000, lr=5.34e-04]
Epoch 48 TRAIN:  99%|█████████▊| 135/137 [00:29<00:00,  5.51it/s, epoch=48, grad_norm=177.013, loss=333.7000, lr=5.34e-04]
Epoch 48 TRAIN:  99%|█████████▊| 135/137 [00:29<00:00,  5.51it/s, epoch=48, grad_norm=216.498, loss=394.8368, lr=5.35e-04]
Epoch 48 TRAIN:  99%|█████████▉| 136/137 [00:29<00:00,  4.56it/s, epoch=48, grad_norm=216.498, loss=394.8368, lr=5.35e-04]

CV:   0%|          | 0/6 [00:00<?, ?it/s]/home/rookie/Documents/batuka/projects/wenet/wenet/wenet/utils/train_utils.py:689: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  amp_autocast(enabled=dtype is not None,

CV:  17%|█▋        | 1/6 [00:00<00:03,  1.32it/s]
                                                 
2026-04-16 20:16:39,048 INFO Epoch 48 Step 6682 CV info lr 5.3456e-04 cv_loss 489.21205832741475 rank 0 acc 0.29210829238096875
2026-04-16 20:16:39,050 INFO [Rank 0] Checkpoint: save to checkpoint exp/conformer_mongolian/epoch_48.pt
2026-04-16 20:16:39,163 INFO Epoch 49 Step 6682 TRAIN info lr 5.3456e-04 rank 0
2026-04-16 20:16:39,184 INFO using accumulate grad, new batch size is 1 times larger than before

Epoch 49 TRAIN:   0%|          | 0/136 [00:00<?, ?it/s]
Epoch 49 TRAIN:   0%|          | 0/136 [00:03<?, ?it/s, epoch=49, grad_norm=345.324, loss=285.3227, lr=5.35e-04]
Epoch 49 TRAIN:   1%|          | 1/136 [00:03<08:27,  3.76s/it, epoch=49, grad_norm=345.324, loss=285.3227, lr=5.35e-04]
Epoch 49 TRAIN:   1%|          | 1/136 [00:03<08:27,  3.76s/it, epoch=49, grad_norm=299.346, loss=315.8968, lr=5.35e-04]
Epoch 49 TRAIN:   1%|          | 1/136 [00:03<08:27,  3.76s/it, epoch=49, grad_norm=337.734, loss=332.0064, lr=5.35e-04]
Epoch 49 TRAIN:   1%|          | 1/136 [00:04<08:27,  3.76s/it, epoch=49, grad_norm=369.779, loss=337.9258, lr=5.35e-04]
Epoch 49 TRAIN:   1%|          | 1/136 [00:04<08:27,  3.76s/it, epoch=49, grad_norm=524.438, loss=426.3920, lr=5.35e-04]
Epoch 49 TRAIN:   1%|          | 1/136 [00:04<08:27,  3.76s/it, epoch=49, grad_norm=440.011, loss=441.8813, lr=5.35e-04]
Epoch 49 TRAIN:   4%|▍         | 6/136 [00:04<01:17,  1.68it/s, epoch=49, grad_norm=440.011, loss=441.8813, lr=5.35e-04]
Epoch 49 TRAIN:   4%|▍         | 6/136 [00:04<01:17,  1.68it/s, epoch=49, grad_norm=228.779, loss=239.4351, lr=5.35e-04]
Epoch 49 TRAIN:   4%|▍         | 6/136 [00:04<01:17,  1.68it/s, epoch=49, grad_norm=255.454, loss=367.8984, lr=5.35e-04]
Epoch 49 TRAIN:   4%|▍         | 6/136 [00:04<01:17,  1.68it/s, epoch=49, grad_norm=147.258, loss=309.2734, lr=5.35e-04]
Epoch 49 TRAIN:   4%|▍         | 6/136 [00:04<01:17,  1.68it/s, epoch=49, grad_norm=110.586, loss=288.6059, lr=5.35e-04]
Epoch 49 TRAIN:   4%|▍         | 6/136 [00:05<01:17,  1.68it/s, epoch=49, grad_norm=143.174, loss=307.4755, lr=5.35e-04]
Epoch 49 TRAIN:   8%|▊         | 11/136 [00:05<00:41,  3.02it/s, epoch=49, grad_norm=143.174, loss=307.4755, lr=5.35e-04]
Epoch 49 TRAIN:   8%|▊         | 11/136 [00:05<00:41,  3.02it/s, epoch=49, grad_norm=105.129, loss=333.9461, lr=5.36e-04]
Epoch 49 TRAIN:   8%|▊         | 11/136 [00:05<00:41,  3.02it/s, epoch=49, grad_norm=90.956, loss=333.0669, lr=5.36e-04] 
Epoch 49 TRAIN:   8%|▊         | 11/136 [00:05<00:41,  3.02it/s, epoch=49, grad_norm=161.460, loss=298.1653, lr=5.36e-04]
Epoch 49 TRAIN:   8%|▊         | 11/136 [00:05<00:41,  3.02it/s, epoch=49, grad_norm=116.880, loss=318.9753, lr=5.36e-04]
Epoch 49 TRAIN:  11%|█         | 15/136 [00:05<00:32,  3.76it/s, epoch=49, grad_norm=116.880, loss=318.9753, lr=5.36e-04]
Epoch 49 TRAIN:  11%|█         | 15/136 [00:05<00:32,  3.76it/s, epoch=49, grad_norm=109.206, loss=315.1558, lr=5.36e-04]
Epoch 49 TRAIN:  11%|█         | 15/136 [00:06<00:32,  3.76it/s, epoch=49, grad_norm=148.663, loss=285.6465, lr=5.36e-04]
Epoch 49 TRAIN:  11%|█         | 15/136 [00:06<00:32,  3.76it/s, epoch=49, grad_norm=109.808, loss=345.0474, lr=5.36e-04]
Epoch 49 TRAIN:  11%|█         | 15/136 [00:06<00:32,  3.76it/s, epoch=49, grad_norm=92.469, loss=336.0963, lr=5.36e-04] 
Epoch 49 TRAIN:  14%|█▍        | 19/136 [00:06<00:27,  4.27it/s, epoch=49, grad_norm=92.469, loss=336.0963, lr=5.36e-04]
Epoch 49 TRAIN:  14%|█▍        | 19/136 [00:06<00:27,  4.27it/s, epoch=49, grad_norm=110.621, loss=306.8833, lr=5.36e-04]
Epoch 49 TRAIN:  14%|█▍        | 19/136 [00:06<00:27,  4.27it/s, epoch=49, grad_norm=81.341, loss=303.0743, lr=5.36e-04] 
Epoch 49 TRAIN:  14%|█▍        | 19/136 [00:07<00:27,  4.27it/s, epoch=49, grad_norm=90.161, loss=309.8521, lr=5.36e-04]
Epoch 49 TRAIN:  16%|█▌        | 22/136 [00:07<00:25,  4.42it/s, epoch=49, grad_norm=90.161, loss=309.8521, lr=5.36e-04]
Epoch 49 TRAIN:  16%|█▌        | 22/136 [00:07<00:25,  4.42it/s, epoch=49, grad_norm=85.396, loss=309.5765, lr=5.36e-04]
Epoch 49 TRAIN:  16%|█▌        | 22/136 [00:07<00:25,  4.42it/s, epoch=49, grad_norm=132.602, loss=335.8412, lr=5.36e-04]
Epoch 49 TRAIN:  16%|█▌        | 22/136 [00:07<00:25,  4.42it/s, epoch=49, grad_norm=189.633, loss=292.4838, lr=5.37e-04]
Epoch 49 TRAIN:  18%|█▊        | 25/136 [00:07<00:24,  4.59it/s, epoch=49, grad_norm=189.633, loss=292.4838, lr=5.37e-04]
Epoch 49 TRAIN:  18%|█▊        | 25/136 [00:07<00:24,  4.59it/s, epoch=49, grad_norm=131.387, loss=336.1686, lr=5.37e-04]
Epoch 49 TRAIN:  18%|█▊        | 25/136 [00:08<00:24,  4.59it/s, epoch=49, grad_norm=162.281, loss=383.3025, lr=5.37e-04]
Epoch 49 TRAIN:  18%|█▊        | 25/136 [00:08<00:24,  4.59it/s, epoch=49, grad_norm=149.337, loss=297.4743, lr=5.37e-04]
Epoch 49 TRAIN:  21%|██        | 28/136 [00:08<00:22,  4.89it/s, epoch=49, grad_norm=149.337, loss=297.4743, lr=5.37e-04]
Epoch 49 TRAIN:  21%|██        | 28/136 [00:08<00:22,  4.89it/s, epoch=49, grad_norm=229.943, loss=306.6998, lr=5.37e-04]
Epoch 49 TRAIN:  21%|██        | 28/136 [00:08<00:22,  4.89it/s, epoch=49, grad_norm=95.918, loss=322.3683, lr=5.37e-04] 
Epoch 49 TRAIN:  21%|██        | 28/136 [00:08<00:22,  4.89it/s, epoch=49, grad_norm=95.812, loss=333.2594, lr=5.37e-04]
Epoch 49 TRAIN:  23%|██▎       | 31/136 [00:08<00:21,  4.94it/s, epoch=49, grad_norm=95.812, loss=333.2594, lr=5.37e-04]
Epoch 49 TRAIN:  23%|██▎       | 31/136 [00:08<00:21,  4.94it/s, epoch=49, grad_norm=110.567, loss=302.9713, lr=5.37e-04]
Epoch 49 TRAIN:  23%|██▎       | 31/136 [00:09<00:21,  4.94it/s, epoch=49, grad_norm=91.958, loss=298.5457, lr=5.37e-04] 
Epoch 49 TRAIN:  23%|██▎       | 31/136 [00:09<00:21,  4.94it/s, epoch=49, grad_norm=100.114, loss=284.5441, lr=5.37e-04]
Epoch 49 TRAIN:  23%|██▎       | 31/136 [00:09<00:21,  4.94it/s, epoch=49, grad_norm=126.933, loss=315.1631, lr=5.37e-04]
Epoch 49 TRAIN:  26%|██▌       | 35/136 [00:09<00:19,  5.28it/s, epoch=49, grad_norm=126.933, loss=315.1631, lr=5.37e-04]
Epoch 49 TRAIN:  26%|██▌       | 35/136 [00:09<00:19,  5.28it/s, epoch=49, grad_norm=122.937, loss=272.0630, lr=5.37e-04]
Epoch 49 TRAIN:  26%|██▌       | 35/136 [00:09<00:19,  5.28it/s, epoch=49, grad_norm=108.175, loss=276.9633, lr=5.38e-04]
Epoch 49 TRAIN:  26%|██▌       | 35/136 [00:10<00:19,  5.28it/s, epoch=49, grad_norm=174.891, loss=311.7347, lr=5.38e-04]
Epoch 49 TRAIN:  28%|██▊       | 38/136 [00:10<00:18,  5.31it/s, epoch=49, grad_norm=174.891, loss=311.7347, lr=5.38e-04]
Epoch 49 TRAIN:  28%|██▊       | 38/136 [00:10<00:18,  5.31it/s, epoch=49, grad_norm=191.692, loss=328.1287, lr=5.38e-04]
Epoch 49 TRAIN:  28%|██▊       | 38/136 [00:10<00:18,  5.31it/s, epoch=49, grad_norm=134.415, loss=300.8015, lr=5.38e-04]
Epoch 49 TRAIN:  28%|██▊       | 38/136 [00:10<00:18,  5.31it/s, epoch=49, grad_norm=142.541, loss=327.0687, lr=5.38e-04]
Epoch 49 TRAIN:  30%|███       | 41/136 [00:10<00:18,  5.11it/s, epoch=49, grad_norm=142.541, loss=327.0687, lr=5.38e-04]
Epoch 49 TRAIN:  30%|███       | 41/136 [00:10<00:18,  5.11it/s, epoch=49, grad_norm=105.986, loss=288.5876, lr=5.38e-04]
Epoch 49 TRAIN:  30%|███       | 41/136 [00:11<00:18,  5.11it/s, epoch=49, grad_norm=141.812, loss=319.8927, lr=5.38e-04]
Epoch 49 TRAIN:  30%|███       | 41/136 [00:11<00:18,  5.11it/s, epoch=49, grad_norm=76.457, loss=282.9924, lr=5.38e-04] 
Epoch 49 TRAIN:  32%|███▏      | 44/136 [00:11<00:18,  5.03it/s, epoch=49, grad_norm=76.457, loss=282.9924, lr=5.38e-04]
Epoch 49 TRAIN:  32%|███▏      | 44/136 [00:11<00:18,  5.03it/s, epoch=49, grad_norm=79.146, loss=259.6407, lr=5.38e-04]
Epoch 49 TRAIN:  32%|███▏      | 44/136 [00:11<00:18,  5.03it/s, epoch=49, grad_norm=105.282, loss=302.7235, lr=5.38e-04]
Epoch 49 TRAIN:  32%|███▏      | 44/136 [00:11<00:18,  5.03it/s, epoch=49, grad_norm=117.877, loss=307.8499, lr=5.38e-04]
Epoch 49 TRAIN:  35%|███▍      | 47/136 [00:11<00:17,  5.03it/s, epoch=49, grad_norm=117.877, loss=307.8499, lr=5.38e-04]
Epoch 49 TRAIN:  35%|███▍      | 47/136 [00:12<00:17,  5.03it/s, epoch=49, grad_norm=110.895, loss=277.9349, lr=5.38e-04]
Epoch 49 TRAIN:  35%|███▍      | 47/136 [00:12<00:17,  5.03it/s, epoch=49, grad_norm=176.798, loss=251.7383, lr=5.38e-04]
Epoch 49 TRAIN:  35%|███▍      | 47/136 [00:12<00:17,  5.03it/s, epoch=49, grad_norm=158.374, loss=249.1365, lr=5.39e-04]
Epoch 49 TRAIN:  37%|███▋      | 50/136 [00:12<00:17,  4.86it/s, epoch=49, grad_norm=158.374, loss=249.1365, lr=5.39e-04]
Epoch 49 TRAIN:  37%|███▋      | 50/136 [00:12<00:17,  4.86it/s, epoch=49, grad_norm=161.630, loss=356.3457, lr=5.39e-04]
Epoch 49 TRAIN:  37%|███▋      | 50/136 [00:12<00:17,  4.86it/s, epoch=49, grad_norm=189.958, loss=331.8489, lr=5.39e-04]
Epoch 49 TRAIN:  37%|███▋      | 50/136 [00:13<00:17,  4.86it/s, epoch=49, grad_norm=124.618, loss=252.9193, lr=5.39e-04]
Epoch 49 TRAIN:  39%|███▉      | 53/136 [00:13<00:16,  4.96it/s, epoch=49, grad_norm=124.618, loss=252.9193, lr=5.39e-04]
Epoch 49 TRAIN:  39%|███▉      | 53/136 [00:13<00:16,  4.96it/s, epoch=49, grad_norm=155.466, loss=259.3176, lr=5.39e-04]
Epoch 49 TRAIN:  39%|███▉      | 53/136 [00:13<00:16,  4.96it/s, epoch=49, grad_norm=130.424, loss=299.7744, lr=5.39e-04]
Epoch 49 TRAIN:  39%|███▉      | 53/136 [00:13<00:16,  4.96it/s, epoch=49, grad_norm=127.627, loss=281.8826, lr=5.39e-04]
Epoch 49 TRAIN:  41%|████      | 56/136 [00:13<00:16,  4.92it/s, epoch=49, grad_norm=127.627, loss=281.8826, lr=5.39e-04]
Epoch 49 TRAIN:  41%|████      | 56/136 [00:13<00:16,  4.92it/s, epoch=49, grad_norm=155.308, loss=285.3010, lr=5.39e-04]
Epoch 49 TRAIN:  41%|████      | 56/136 [00:14<00:16,  4.92it/s, epoch=49, grad_norm=195.921, loss=312.6379, lr=5.39e-04]
Epoch 49 TRAIN:  41%|████      | 56/136 [00:14<00:16,  4.92it/s, epoch=49, grad_norm=125.378, loss=298.9930, lr=5.39e-04]
Epoch 49 TRAIN:  43%|████▎     | 59/136 [00:14<00:15,  4.90it/s, epoch=49, grad_norm=125.378, loss=298.9930, lr=5.39e-04]
Epoch 49 TRAIN:  43%|████▎     | 59/136 [00:14<00:15,  4.90it/s, epoch=49, grad_norm=68.954, loss=269.4546, lr=5.39e-04] 
Epoch 49 TRAIN:  43%|████▎     | 59/136 [00:14<00:15,  4.90it/s, epoch=49, grad_norm=102.414, loss=287.4203, lr=5.39e-04]
Epoch 49 TRAIN:  43%|████▎     | 59/136 [00:14<00:15,  4.90it/s, epoch=49, grad_norm=179.572, loss=342.4084, lr=5.40e-04]
Epoch 49 TRAIN:  46%|████▌     | 62/136 [00:14<00:14,  4.97it/s, epoch=49, grad_norm=179.572, loss=342.4084, lr=5.40e-04]
Epoch 49 TRAIN:  46%|████▌     | 62/136 [00:15<00:14,  4.97it/s, epoch=49, grad_norm=110.698, loss=311.1111, lr=5.40e-04]
Epoch 49 TRAIN:  46%|████▌     | 62/136 [00:15<00:14,  4.97it/s, epoch=49, grad_norm=133.789, loss=281.6232, lr=5.40e-04]
Epoch 49 TRAIN:  46%|████▌     | 62/136 [00:15<00:14,  4.97it/s, epoch=49, grad_norm=155.552, loss=263.7115, lr=5.40e-04]
Epoch 49 TRAIN:  48%|████▊     | 65/136 [00:15<00:14,  4.96it/s, epoch=49, grad_norm=155.552, loss=263.7115, lr=5.40e-04]
Epoch 49 TRAIN:  48%|████▊     | 65/136 [00:15<00:14,  4.96it/s, epoch=49, grad_norm=115.213, loss=288.6948, lr=5.40e-04]
Epoch 49 TRAIN:  48%|████▊     | 65/136 [00:15<00:14,  4.96it/s, epoch=49, grad_norm=119.740, loss=330.1432, lr=5.40e-04]
Epoch 49 TRAIN:  48%|████▊     | 65/136 [00:16<00:14,  4.96it/s, epoch=49, grad_norm=147.100, loss=259.3178, lr=5.40e-04]
Epoch 49 TRAIN:  50%|█████     | 68/136 [00:16<00:13,  5.08it/s, epoch=49, grad_norm=147.100, loss=259.3178, lr=5.40e-04]
Epoch 49 TRAIN:  50%|█████     | 68/136 [00:16<00:13,  5.08it/s, epoch=49, grad_norm=172.884, loss=362.1023, lr=5.40e-04]
Epoch 49 TRAIN:  50%|█████     | 68/136 [00:16<00:13,  5.08it/s, epoch=49, grad_norm=174.814, loss=325.0603, lr=5.40e-04]
Epoch 49 TRAIN:  50%|█████     | 68/136 [00:16<00:13,  5.08it/s, epoch=49, grad_norm=92.867, loss=279.1505, lr=5.40e-04] 
Epoch 49 TRAIN:  52%|█████▏    | 71/136 [00:16<00:13,  4.86it/s, epoch=49, grad_norm=92.867, loss=279.1505, lr=5.40e-04]
Epoch 49 TRAIN:  52%|█████▏    | 71/136 [00:17<00:13,  4.86it/s, epoch=49, grad_norm=133.065, loss=339.5567, lr=5.40e-04]
Epoch 49 TRAIN:  52%|█████▏    | 71/136 [00:17<00:13,  4.86it/s, epoch=49, grad_norm=106.553, loss=309.7022, lr=5.40e-04]
Epoch 49 TRAIN:  52%|█████▏    | 71/136 [00:17<00:13,  4.86it/s, epoch=49, grad_norm=128.863, loss=332.7367, lr=5.40e-04]
Epoch 49 TRAIN:  54%|█████▍    | 74/136 [00:17<00:13,  4.76it/s, epoch=49, grad_norm=128.863, loss=332.7367, lr=5.40e-04]
Epoch 49 TRAIN:  54%|█████▍    | 74/136 [00:17<00:13,  4.76it/s, epoch=49, grad_norm=335.894, loss=181.6431, lr=5.41e-04]
Epoch 49 TRAIN:  54%|█████▍    | 74/136 [00:17<00:13,  4.76it/s, epoch=49, grad_norm=106.355, loss=287.2918, lr=5.41e-04]
Epoch 49 TRAIN:  54%|█████▍    | 74/136 [00:18<00:13,  4.76it/s, epoch=49, grad_norm=149.964, loss=309.4987, lr=5.41e-04]
Epoch 49 TRAIN:  57%|█████▋    | 77/136 [00:18<00:12,  4.85it/s, epoch=49, grad_norm=149.964, loss=309.4987, lr=5.41e-04]
Epoch 49 TRAIN:  57%|█████▋    | 77/136 [00:18<00:12,  4.85it/s, epoch=49, grad_norm=123.653, loss=280.9702, lr=5.41e-04]
Epoch 49 TRAIN:  57%|█████▋    | 77/136 [00:18<00:12,  4.85it/s, epoch=49, grad_norm=113.582, loss=289.9861, lr=5.41e-04]
Epoch 49 TRAIN:  57%|█████▋    | 77/136 [00:18<00:12,  4.85it/s, epoch=49, grad_norm=114.678, loss=322.1712, lr=5.41e-04]
Epoch 49 TRAIN:  59%|█████▉    | 80/136 [00:18<00:11,  4.83it/s, epoch=49, grad_norm=114.678, loss=322.1712, lr=5.41e-04]
Epoch 49 TRAIN:  59%|█████▉    | 80/136 [00:18<00:11,  4.83it/s, epoch=49, grad_norm=129.945, loss=303.3290, lr=5.41e-04]
Epoch 49 TRAIN:  59%|█████▉    | 80/136 [00:19<00:11,  4.83it/s, epoch=49, grad_norm=135.978, loss=273.1308, lr=5.41e-04]
Epoch 49 TRAIN:  59%|█████▉    | 80/136 [00:19<00:11,  4.83it/s, epoch=49, grad_norm=275.095, loss=253.3227, lr=5.41e-04]
Epoch 49 TRAIN:  61%|██████    | 83/136 [00:19<00:11,  4.75it/s, epoch=49, grad_norm=275.095, loss=253.3227, lr=5.41e-04]
Epoch 49 TRAIN:  61%|██████    | 83/136 [00:19<00:11,  4.75it/s, epoch=49, grad_norm=209.566, loss=216.7215, lr=5.41e-04]
Epoch 49 TRAIN:  61%|██████    | 83/136 [00:19<00:11,  4.75it/s, epoch=49, grad_norm=123.585, loss=322.2854, lr=5.41e-04]
Epoch 49 TRAIN:  61%|██████    | 83/136 [00:19<00:11,  4.75it/s, epoch=49, grad_norm=172.016, loss=316.7642, lr=5.41e-04]
Epoch 49 TRAIN:  63%|██████▎   | 86/136 [00:19<00:10,  4.87it/s, epoch=49, grad_norm=172.016, loss=316.7642, lr=5.41e-04]
Epoch 49 TRAIN:  63%|██████▎   | 86/136 [00:20<00:10,  4.87it/s, epoch=49, grad_norm=163.688, loss=281.4905, lr=5.42e-04]
Epoch 49 TRAIN:  63%|██████▎   | 86/136 [00:20<00:10,  4.87it/s, epoch=49, grad_norm=100.693, loss=304.8210, lr=5.42e-04]
Epoch 49 TRAIN:  63%|██████▎   | 86/136 [00:20<00:10,  4.87it/s, epoch=49, grad_norm=143.528, loss=273.1132, lr=5.42e-04]
Epoch 49 TRAIN:  65%|██████▌   | 89/136 [00:20<00:09,  5.02it/s, epoch=49, grad_norm=143.528, loss=273.1132, lr=5.42e-04]
Epoch 49 TRAIN:  65%|██████▌   | 89/136 [00:20<00:09,  5.02it/s, epoch=49, grad_norm=169.077, loss=312.8771, lr=5.42e-04]
Epoch 49 TRAIN:  65%|██████▌   | 89/136 [00:20<00:09,  5.02it/s, epoch=49, grad_norm=173.490, loss=344.6037, lr=5.42e-04]
Epoch 49 TRAIN:  65%|██████▌   | 89/136 [00:21<00:09,  5.02it/s, epoch=49, grad_norm=144.774, loss=317.0163, lr=5.42e-04]
Epoch 49 TRAIN:  68%|██████▊   | 92/136 [00:21<00:08,  4.91it/s, epoch=49, grad_norm=144.774, loss=317.0163, lr=5.42e-04]
Epoch 49 TRAIN:  68%|██████▊   | 92/136 [00:21<00:08,  4.91it/s, epoch=49, grad_norm=132.634, loss=288.3240, lr=5.42e-04]
Epoch 49 TRAIN:  68%|██████▊   | 92/136 [00:21<00:08,  4.91it/s, epoch=49, grad_norm=184.234, loss=313.2860, lr=5.42e-04]
Epoch 49 TRAIN:  68%|██████▊   | 92/136 [00:21<00:08,  4.91it/s, epoch=49, grad_norm=109.493, loss=290.2498, lr=5.42e-04]
Epoch 49 TRAIN:  70%|██████▉   | 95/136 [00:21<00:08,  4.97it/s, epoch=49, grad_norm=109.493, loss=290.2498, lr=5.42e-04]
Epoch 49 TRAIN:  70%|██████▉   | 95/136 [00:21<00:08,  4.97it/s, epoch=49, grad_norm=111.130, loss=279.5237, lr=5.42e-04]
Epoch 49 TRAIN:  70%|██████▉   | 95/136 [00:22<00:08,  4.97it/s, epoch=49, grad_norm=205.019, loss=297.5709, lr=5.42e-04]
Epoch 49 TRAIN:  70%|██████▉   | 95/136 [00:22<00:08,  4.97it/s, epoch=49, grad_norm=97.595, loss=272.5670, lr=5.42e-04] 
Epoch 49 TRAIN:  72%|███████▏  | 98/136 [00:22<00:07,  4.81it/s, epoch=49, grad_norm=97.595, loss=272.5670, lr=5.42e-04]
Epoch 49 TRAIN:  72%|███████▏  | 98/136 [00:22<00:07,  4.81it/s, epoch=49, grad_norm=208.148, loss=260.0485, lr=5.42e-04]
Epoch 49 TRAIN:  72%|███████▏  | 98/136 [00:22<00:07,  4.81it/s, epoch=49, grad_norm=108.107, loss=278.8307, lr=5.43e-04]2026-04-16 20:17:01,922 DEBUG TRAIN | steps/sec 4.439| Batch 49/100 loss 278.830688 loss_att 232.189316 loss_ctc 387.660614 th_accuracy 0.372439 lr 5.4256e-04 grad_norm 108.106583 rank 0

Epoch 49 TRAIN:  72%|███████▏  | 98/136 [00:22<00:07,  4.81it/s, epoch=49, grad_norm=160.840, loss=213.5083, lr=5.43e-04]
Epoch 49 TRAIN:  74%|███████▍  | 101/136 [00:22<00:07,  4.78it/s, epoch=49, grad_norm=160.840, loss=213.5083, lr=5.43e-04]
Epoch 49 TRAIN:  74%|███████▍  | 101/136 [00:23<00:07,  4.78it/s, epoch=49, grad_norm=348.868, loss=335.0168, lr=5.43e-04]
Epoch 49 TRAIN:  74%|███████▍  | 101/136 [00:23<00:07,  4.78it/s, epoch=49, grad_norm=226.337, loss=294.9942, lr=5.43e-04]
Epoch 49 TRAIN:  74%|███████▍  | 101/136 [00:23<00:07,  4.78it/s, epoch=49, grad_norm=118.047, loss=250.5544, lr=5.43e-04]
Epoch 49 TRAIN:  76%|███████▋  | 104/136 [00:23<00:06,  4.71it/s, epoch=49, grad_norm=118.047, loss=250.5544, lr=5.43e-04]
Epoch 49 TRAIN:  76%|███████▋  | 104/136 [00:23<00:06,  4.71it/s, epoch=49, grad_norm=127.515, loss=331.0105, lr=5.43e-04]
Epoch 49 TRAIN:  76%|███████▋  | 104/136 [00:24<00:06,  4.71it/s, epoch=49, grad_norm=160.825, loss=262.7205, lr=5.43e-04]
Epoch 49 TRAIN:  76%|███████▋  | 104/136 [00:24<00:06,  4.71it/s, epoch=49, grad_norm=317.646, loss=328.4399, lr=5.43e-04]
Epoch 49 TRAIN:  79%|███████▊  | 107/136 [00:24<00:06,  4.77it/s, epoch=49, grad_norm=317.646, loss=328.4399, lr=5.43e-04]
Epoch 49 TRAIN:  79%|███████▊  | 107/136 [00:24<00:06,  4.77it/s, epoch=49, grad_norm=149.475, loss=305.4647, lr=5.43e-04]
Epoch 49 TRAIN:  79%|███████▊  | 107/136 [00:24<00:06,  4.77it/s, epoch=49, grad_norm=523.324, loss=363.2540, lr=5.43e-04]
Epoch 49 TRAIN:  79%|███████▊  | 107/136 [00:24<00:06,  4.77it/s, epoch=49, grad_norm=134.047, loss=273.8604, lr=5.43e-04]
Epoch 49 TRAIN:  81%|████████  | 110/136 [00:24<00:05,  4.70it/s, epoch=49, grad_norm=134.047, loss=273.8604, lr=5.43e-04]
Epoch 49 TRAIN:  81%|████████  | 110/136 [00:25<00:05,  4.70it/s, epoch=49, grad_norm=202.129, loss=333.0720, lr=5.43e-04]
Epoch 49 TRAIN:  81%|████████  | 110/136 [00:25<00:05,  4.70it/s, epoch=49, grad_norm=187.440, loss=287.4319, lr=5.44e-04]
Epoch 49 TRAIN:  81%|████████  | 110/136 [00:25<00:05,  4.70it/s, epoch=49, grad_norm=335.279, loss=346.8893, lr=5.44e-04]
Epoch 49 TRAIN:  83%|████████▎ | 113/136 [00:25<00:04,  4.63it/s, epoch=49, grad_norm=335.279, loss=346.8893, lr=5.44e-04]
Epoch 49 TRAIN:  83%|████████▎ | 113/136 [00:25<00:04,  4.63it/s, epoch=49, grad_norm=144.749, loss=288.1870, lr=5.44e-04]
Epoch 49 TRAIN:  83%|████████▎ | 113/136 [00:26<00:04,  4.63it/s, epoch=49, grad_norm=223.134, loss=342.2844, lr=5.44e-04]
Epoch 49 TRAIN:  83%|████████▎ | 113/136 [00:26<00:04,  4.63it/s, epoch=49, grad_norm=95.983, loss=214.6086, lr=5.44e-04] 
Epoch 49 TRAIN:  85%|████████▌ | 116/136 [00:26<00:04,  4.75it/s, epoch=49, grad_norm=95.983, loss=214.6086, lr=5.44e-04]
Epoch 49 TRAIN:  85%|████████▌ | 116/136 [00:26<00:04,  4.75it/s, epoch=49, grad_norm=161.963, loss=297.6003, lr=5.44e-04]
Epoch 49 TRAIN:  85%|████████▌ | 116/136 [00:26<00:04,  4.75it/s, epoch=49, grad_norm=205.740, loss=310.9330, lr=5.44e-04]
Epoch 49 TRAIN:  85%|████████▌ | 116/136 [00:26<00:04,  4.75it/s, epoch=49, grad_norm=165.213, loss=257.5037, lr=5.44e-04]
Epoch 49 TRAIN:  88%|████████▊ | 119/136 [00:26<00:03,  4.76it/s, epoch=49, grad_norm=165.213, loss=257.5037, lr=5.44e-04]
Epoch 49 TRAIN:  88%|████████▊ | 119/136 [00:27<00:03,  4.76it/s, epoch=49, grad_norm=169.730, loss=255.5197, lr=5.44e-04]
Epoch 49 TRAIN:  88%|████████▊ | 119/136 [00:27<00:03,  4.76it/s, epoch=49, grad_norm=147.493, loss=312.6800, lr=5.44e-04]
Epoch 49 TRAIN:  88%|████████▊ | 119/136 [00:27<00:03,  4.76it/s, epoch=49, grad_norm=176.580, loss=308.2962, lr=5.44e-04]
Epoch 49 TRAIN:  90%|████████▉ | 122/136 [00:27<00:03,  4.64it/s, epoch=49, grad_norm=176.580, loss=308.2962, lr=5.44e-04]
Epoch 49 TRAIN:  90%|████████▉ | 122/136 [00:27<00:03,  4.64it/s, epoch=49, grad_norm=220.329, loss=374.1035, lr=5.44e-04]
Epoch 49 TRAIN:  90%|████████▉ | 122/136 [00:27<00:03,  4.64it/s, epoch=49, grad_norm=183.420, loss=353.6389, lr=5.44e-04]
Epoch 49 TRAIN:  90%|████████▉ | 122/136 [00:27<00:03,  4.64it/s, epoch=49, grad_norm=131.508, loss=268.8300, lr=5.45e-04]
Epoch 49 TRAIN:  90%|████████▉ | 122/136 [00:28<00:03,  4.64it/s, epoch=49, grad_norm=170.782, loss=316.3429, lr=5.45e-04]
Epoch 49 TRAIN:  93%|█████████▎| 126/136 [00:28<00:01,  5.00it/s, epoch=49, grad_norm=170.782, loss=316.3429, lr=5.45e-04]
Epoch 49 TRAIN:  93%|█████████▎| 126/136 [00:28<00:01,  5.00it/s, epoch=49, grad_norm=230.821, loss=337.9553, lr=5.45e-04]
Epoch 49 TRAIN:  93%|█████████▎| 126/136 [00:28<00:01,  5.00it/s, epoch=49, grad_norm=129.525, loss=288.9476, lr=5.45e-04]
Epoch 49 TRAIN:  93%|█████████▎| 126/136 [00:28<00:01,  5.00it/s, epoch=49, grad_norm=158.192, loss=357.0849, lr=5.45e-04]
Epoch 49 TRAIN:  95%|█████████▍| 129/136 [00:28<00:01,  4.66it/s, epoch=49, grad_norm=158.192, loss=357.0849, lr=5.45e-04]
Epoch 49 TRAIN:  95%|█████████▍| 129/136 [00:29<00:01,  4.66it/s, epoch=49, grad_norm=206.661, loss=375.6495, lr=5.45e-04]
Epoch 49 TRAIN:  95%|█████████▍| 129/136 [00:29<00:01,  4.66it/s, epoch=49, grad_norm=147.206, loss=302.1255, lr=5.45e-04]
Epoch 49 TRAIN:  95%|█████████▍| 129/136 [00:29<00:01,  4.66it/s, epoch=49, grad_norm=158.156, loss=331.1589, lr=5.45e-04]
Epoch 49 TRAIN:  95%|█████████▍| 129/136 [00:29<00:01,  4.66it/s, epoch=49, grad_norm=181.177, loss=310.4404, lr=5.45e-04]
Epoch 49 TRAIN:  98%|█████████▊| 133/136 [00:29<00:00,  4.97it/s, epoch=49, grad_norm=181.177, loss=310.4404, lr=5.45e-04]
Epoch 49 TRAIN:  98%|█████████▊| 133/136 [00:29<00:00,  4.97it/s, epoch=49, grad_norm=177.840, loss=356.0113, lr=5.45e-04]
Epoch 49 TRAIN:  99%|█████████▊| 134/136 [00:29<00:00,  4.49it/s, epoch=49, grad_norm=177.840, loss=356.0113, lr=5.45e-04]

CV:   0%|          | 0/6 [00:00<?, ?it/s]/home/rookie/Documents/batuka/projects/wenet/wenet/wenet/utils/train_utils.py:689: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  amp_autocast(enabled=dtype is not None,

CV:  17%|█▋        | 1/6 [00:00<00:03,  1.32it/s]
                                                 
2026-04-16 20:17:09,884 INFO Epoch 49 Step 6816 CV info lr 5.4528e-04 cv_loss 485.0980307839134 rank 0 acc 0.30606267352898914
2026-04-16 20:17:09,886 INFO [Rank 0] Checkpoint: save to checkpoint exp/conformer_mongolian/epoch_49.pt
2026-04-16 20:17:10,000 INFO Epoch 50 Step 6816 TRAIN info lr 5.4528e-04 rank 0
2026-04-16 20:17:10,013 INFO using accumulate grad, new batch size is 1 times larger than before

Epoch 50 TRAIN:   0%|          | 0/134 [00:00<?, ?it/s]
Epoch 50 TRAIN:   0%|          | 0/134 [00:03<?, ?it/s, epoch=50, grad_norm=541.004, loss=432.8828, lr=5.45e-04]
Epoch 50 TRAIN:   1%|          | 1/134 [00:03<08:18,  3.75s/it, epoch=50, grad_norm=541.004, loss=432.8828, lr=5.45e-04]
Epoch 50 TRAIN:   1%|          | 1/134 [00:04<08:18,  3.75s/it, epoch=50, grad_norm=334.997, loss=312.7580, lr=5.45e-04]
Epoch 50 TRAIN:   1%|          | 1/134 [00:04<08:18,  3.75s/it, epoch=50, grad_norm=297.633, loss=316.6663, lr=5.46e-04]
Epoch 50 TRAIN:   1%|          | 1/134 [00:04<08:18,  3.75s/it, epoch=50, grad_norm=398.327, loss=330.3097, lr=5.46e-04]
Epoch 50 TRAIN:   3%|▎         | 4/134 [00:04<01:50,  1.18it/s, epoch=50, grad_norm=398.327, loss=330.3097, lr=5.46e-04]
Epoch 50 TRAIN:   3%|▎         | 4/134 [00:04<01:50,  1.18it/s, epoch=50, grad_norm=338.283, loss=344.3069, lr=5.46e-04]
Epoch 50 TRAIN:   3%|▎         | 4/134 [00:04<01:50,  1.18it/s, epoch=50, grad_norm=228.482, loss=283.3216, lr=5.46e-04]
Epoch 50 TRAIN:   3%|▎         | 4/134 [00:04<01:50,  1.18it/s, epoch=50, grad_norm=261.493, loss=351.6544, lr=5.46e-04]
Epoch 50 TRAIN:   3%|▎         | 4/134 [00:04<01:50,  1.18it/s, epoch=50, grad_norm=241.375, loss=340.7433, lr=5.46e-04]
Epoch 50 TRAIN:   3%|▎         | 4/134 [00:04<01:50,  1.18it/s, epoch=50, grad_norm=137.908, loss=299.8225, lr=5.46e-04]
Epoch 50 TRAIN:   3%|▎         | 4/134 [00:04<01:50,  1.18it/s, epoch=50, grad_norm=170.797, loss=244.5340, lr=5.46e-04]
Epoch 50 TRAIN:   7%|▋         | 10/134 [00:04<00:39,  3.11it/s, epoch=50, grad_norm=170.797, loss=244.5340, lr=5.46e-04]
Epoch 50 TRAIN:   7%|▋         | 10/134 [00:04<00:39,  3.11it/s, epoch=50, grad_norm=133.539, loss=339.4177, lr=5.46e-04]
Epoch 50 TRAIN:   7%|▋         | 10/134 [00:05<00:39,  3.11it/s, epoch=50, grad_norm=117.074, loss=324.5572, lr=5.46e-04]
Epoch 50 TRAIN:   7%|▋         | 10/134 [00:05<00:39,  3.11it/s, epoch=50, grad_norm=117.904, loss=337.7882, lr=5.46e-04]
Epoch 50 TRAIN:   7%|▋         | 10/134 [00:05<00:39,  3.11it/s, epoch=50, grad_norm=107.944, loss=321.6435, lr=5.46e-04]
Epoch 50 TRAIN:  10%|█         | 14/134 [00:05<00:30,  3.88it/s, epoch=50, grad_norm=107.944, loss=321.6435, lr=5.46e-04]
Epoch 50 TRAIN:  10%|█         | 14/134 [00:05<00:30,  3.88it/s, epoch=50, grad_norm=101.785, loss=310.0549, lr=5.46e-04]
Epoch 50 TRAIN:  10%|█         | 14/134 [00:05<00:30,  3.88it/s, epoch=50, grad_norm=100.308, loss=320.5455, lr=5.47e-04]
Epoch 50 TRAIN:  10%|█         | 14/134 [00:05<00:30,  3.88it/s, epoch=50, grad_norm=157.590, loss=304.5435, lr=5.47e-04]
Epoch 50 TRAIN:  10%|█         | 14/134 [00:06<00:30,  3.88it/s, epoch=50, grad_norm=127.580, loss=274.8345, lr=5.47e-04]
Epoch 50 TRAIN:  13%|█▎        | 18/134 [00:06<00:25,  4.54it/s, epoch=50, grad_norm=127.580, loss=274.8345, lr=5.47e-04]
Epoch 50 TRAIN:  13%|█▎        | 18/134 [00:06<00:25,  4.54it/s, epoch=50, grad_norm=109.785, loss=289.2493, lr=5.47e-04]
Epoch 50 TRAIN:  13%|█▎        | 18/134 [00:06<00:25,  4.54it/s, epoch=50, grad_norm=71.226, loss=280.4734, lr=5.47e-04] 
Epoch 50 TRAIN:  13%|█▎        | 18/134 [00:06<00:25,  4.54it/s, epoch=50, grad_norm=166.456, loss=333.9324, lr=5.47e-04]
Epoch 50 TRAIN:  13%|█▎        | 18/134 [00:06<00:25,  4.54it/s, epoch=50, grad_norm=124.196, loss=279.5378, lr=5.47e-04]
Epoch 50 TRAIN:  16%|█▋        | 22/134 [00:06<00:23,  4.80it/s, epoch=50, grad_norm=124.196, loss=279.5378, lr=5.47e-04]
Epoch 50 TRAIN:  16%|█▋        | 22/134 [00:06<00:23,  4.80it/s, epoch=50, grad_norm=127.674, loss=296.0556, lr=5.47e-04]
Epoch 50 TRAIN:  16%|█▋        | 22/134 [00:07<00:23,  4.80it/s, epoch=50, grad_norm=203.063, loss=322.4052, lr=5.47e-04]
Epoch 50 TRAIN:  16%|█▋        | 22/134 [00:07<00:23,  4.80it/s, epoch=50, grad_norm=142.268, loss=326.3908, lr=5.47e-04]
Epoch 50 TRAIN:  16%|█▋        | 22/134 [00:07<00:23,  4.80it/s, epoch=50, grad_norm=76.329, loss=277.5278, lr=5.47e-04] 
Epoch 50 TRAIN:  19%|█▉        | 26/134 [00:07<00:21,  5.06it/s, epoch=50, grad_norm=76.329, loss=277.5278, lr=5.47e-04]
Epoch 50 TRAIN:  19%|█▉        | 26/134 [00:07<00:21,  5.06it/s, epoch=50, grad_norm=92.462, loss=278.6089, lr=5.47e-04]
Epoch 50 TRAIN:  19%|█▉        | 26/134 [00:07<00:21,  5.06it/s, epoch=50, grad_norm=106.498, loss=299.0648, lr=5.48e-04]
Epoch 50 TRAIN:  19%|█▉        | 26/134 [00:08<00:21,  5.06it/s, epoch=50, grad_norm=107.403, loss=280.9810, lr=5.48e-04]
Epoch 50 TRAIN:  22%|██▏       | 29/134 [00:08<00:20,  5.07it/s, epoch=50, grad_norm=107.403, loss=280.9810, lr=5.48e-04]
Epoch 50 TRAIN:  22%|██▏       | 29/134 [00:08<00:20,  5.07it/s, epoch=50, grad_norm=111.416, loss=310.1480, lr=5.48e-04]
Epoch 50 TRAIN:  22%|██▏       | 29/134 [00:08<00:20,  5.07it/s, epoch=50, grad_norm=124.097, loss=249.1839, lr=5.48e-04]
Epoch 50 TRAIN:  22%|██▏       | 29/134 [00:08<00:20,  5.07it/s, epoch=50, grad_norm=121.389, loss=309.9724, lr=5.48e-04]
Epoch 50 TRAIN:  24%|██▍       | 32/134 [00:08<00:19,  5.14it/s, epoch=50, grad_norm=121.389, loss=309.9724, lr=5.48e-04]
Epoch 50 TRAIN:  24%|██▍       | 32/134 [00:08<00:19,  5.14it/s, epoch=50, grad_norm=96.653, loss=277.2132, lr=5.48e-04] 
Epoch 50 TRAIN:  24%|██▍       | 32/134 [00:09<00:19,  5.14it/s, epoch=50, grad_norm=129.633, loss=326.1474, lr=5.48e-04]
Epoch 50 TRAIN:  24%|██▍       | 32/134 [00:09<00:19,  5.14it/s, epoch=50, grad_norm=94.263, loss=328.6928, lr=5.48e-04] 
Epoch 50 TRAIN:  26%|██▌       | 35/134 [00:09<00:20,  4.89it/s, epoch=50, grad_norm=94.263, loss=328.6928, lr=5.48e-04]
Epoch 50 TRAIN:  26%|██▌       | 35/134 [00:09<00:20,  4.89it/s, epoch=50, grad_norm=75.461, loss=287.5340, lr=5.48e-04]
Epoch 50 TRAIN:  26%|██▌       | 35/134 [00:09<00:20,  4.89it/s, epoch=50, grad_norm=140.253, loss=337.9719, lr=5.48e-04]
Epoch 50 TRAIN:  26%|██▌       | 35/134 [00:09<00:20,  4.89it/s, epoch=50, grad_norm=83.239, loss=285.3091, lr=5.48e-04] 
Epoch 50 TRAIN:  28%|██▊       | 38/134 [00:09<00:19,  4.92it/s, epoch=50, grad_norm=83.239, loss=285.3091, lr=5.48e-04]
Epoch 50 TRAIN:  28%|██▊       | 38/134 [00:10<00:19,  4.92it/s, epoch=50, grad_norm=87.295, loss=290.9957, lr=5.48e-04]
Epoch 50 TRAIN:  28%|██▊       | 38/134 [00:10<00:19,  4.92it/s, epoch=50, grad_norm=124.678, loss=255.4552, lr=5.48e-04]
Epoch 50 TRAIN:  28%|██▊       | 38/134 [00:10<00:19,  4.92it/s, epoch=50, grad_norm=86.207, loss=327.1732, lr=5.49e-04] 
Epoch 50 TRAIN:  31%|███       | 41/134 [00:10<00:18,  5.03it/s, epoch=50, grad_norm=86.207, loss=327.1732, lr=5.49e-04]
Epoch 50 TRAIN:  31%|███       | 41/134 [00:10<00:18,  5.03it/s, epoch=50, grad_norm=89.484, loss=304.0333, lr=5.49e-04]
Epoch 50 TRAIN:  31%|███       | 41/134 [00:10<00:18,  5.03it/s, epoch=50, grad_norm=79.166, loss=300.1161, lr=5.49e-04]
Epoch 50 TRAIN:  31%|███       | 41/134 [00:11<00:18,  5.03it/s, epoch=50, grad_norm=160.657, loss=341.9672, lr=5.49e-04]
Epoch 50 TRAIN:  33%|███▎      | 44/134 [00:11<00:18,  4.96it/s, epoch=50, grad_norm=160.657, loss=341.9672, lr=5.49e-04]
Epoch 50 TRAIN:  33%|███▎      | 44/134 [00:11<00:18,  4.96it/s, epoch=50, grad_norm=101.300, loss=285.8791, lr=5.49e-04]
Epoch 50 TRAIN:  33%|███▎      | 44/134 [00:11<00:18,  4.96it/s, epoch=50, grad_norm=95.172, loss=271.2526, lr=5.49e-04] 
Epoch 50 TRAIN:  33%|███▎      | 44/134 [00:11<00:18,  4.96it/s, epoch=50, grad_norm=102.671, loss=228.6912, lr=5.49e-04]
Epoch 50 TRAIN:  35%|███▌      | 47/134 [00:11<00:17,  4.97it/s, epoch=50, grad_norm=102.671, loss=228.6912, lr=5.49e-04]
Epoch 50 TRAIN:  35%|███▌      | 47/134 [00:11<00:17,  4.97it/s, epoch=50, grad_norm=154.231, loss=307.4355, lr=5.49e-04]
Epoch 50 TRAIN:  35%|███▌      | 47/134 [00:12<00:17,  4.97it/s, epoch=50, grad_norm=135.665, loss=294.3016, lr=5.49e-04]
Epoch 50 TRAIN:  35%|███▌      | 47/134 [00:12<00:17,  4.97it/s, epoch=50, grad_norm=186.964, loss=330.8144, lr=5.49e-04]
Epoch 50 TRAIN:  37%|███▋      | 50/134 [00:12<00:17,  4.93it/s, epoch=50, grad_norm=186.964, loss=330.8144, lr=5.49e-04]
Epoch 50 TRAIN:  37%|███▋      | 50/134 [00:12<00:17,  4.93it/s, epoch=50, grad_norm=159.516, loss=316.1626, lr=5.49e-04]
Epoch 50 TRAIN:  37%|███▋      | 50/134 [00:12<00:17,  4.93it/s, epoch=50, grad_norm=107.590, loss=315.9412, lr=5.49e-04]
Epoch 50 TRAIN:  37%|███▋      | 50/134 [00:13<00:17,  4.93it/s, epoch=50, grad_norm=133.917, loss=322.9532, lr=5.50e-04]
Epoch 50 TRAIN:  40%|███▉      | 53/134 [00:13<00:16,  4.88it/s, epoch=50, grad_norm=133.917, loss=322.9532, lr=5.50e-04]
Epoch 50 TRAIN:  40%|███▉      | 53/134 [00:13<00:16,  4.88it/s, epoch=50, grad_norm=114.231, loss=303.4242, lr=5.50e-04]
Epoch 50 TRAIN:  40%|███▉      | 53/134 [00:13<00:16,  4.88it/s, epoch=50, grad_norm=121.145, loss=303.5409, lr=5.50e-04]
Epoch 50 TRAIN:  40%|███▉      | 53/134 [00:13<00:16,  4.88it/s, epoch=50, grad_norm=79.395, loss=313.3678, lr=5.50e-04] 
Epoch 50 TRAIN:  42%|████▏     | 56/134 [00:13<00:16,  4.85it/s, epoch=50, grad_norm=79.395, loss=313.3678, lr=5.50e-04]
Epoch 50 TRAIN:  42%|████▏     | 56/134 [00:13<00:16,  4.85it/s, epoch=50, grad_norm=150.044, loss=283.9611, lr=5.50e-04]
Epoch 50 TRAIN:  42%|████▏     | 56/134 [00:14<00:16,  4.85it/s, epoch=50, grad_norm=94.635, loss=292.9692, lr=5.50e-04] 
Epoch 50 TRAIN:  42%|████▏     | 56/134 [00:14<00:16,  4.85it/s, epoch=50, grad_norm=102.098, loss=312.8542, lr=5.50e-04]
Epoch 50 TRAIN:  44%|████▍     | 59/134 [00:14<00:15,  4.86it/s, epoch=50, grad_norm=102.098, loss=312.8542, lr=5.50e-04]
Epoch 50 TRAIN:  44%|████▍     | 59/134 [00:14<00:15,  4.86it/s, epoch=50, grad_norm=138.885, loss=341.8669, lr=5.50e-04]
Epoch 50 TRAIN:  44%|████▍     | 59/134 [00:14<00:15,  4.86it/s, epoch=50, grad_norm=149.245, loss=264.7114, lr=5.50e-04]
Epoch 50 TRAIN:  44%|████▍     | 59/134 [00:14<00:15,  4.86it/s, epoch=50, grad_norm=215.050, loss=338.9194, lr=5.50e-04]
Epoch 50 TRAIN:  46%|████▋     | 62/134 [00:14<00:14,  4.89it/s, epoch=50, grad_norm=215.050, loss=338.9194, lr=5.50e-04]
Epoch 50 TRAIN:  46%|████▋     | 62/134 [00:15<00:14,  4.89it/s, epoch=50, grad_norm=108.842, loss=266.2383, lr=5.50e-04]
Epoch 50 TRAIN:  46%|████▋     | 62/134 [00:15<00:14,  4.89it/s, epoch=50, grad_norm=100.372, loss=301.9584, lr=5.50e-04]
Epoch 50 TRAIN:  46%|████▋     | 62/134 [00:15<00:14,  4.89it/s, epoch=50, grad_norm=127.936, loss=292.9472, lr=5.50e-04]
Epoch 50 TRAIN:  49%|████▊     | 65/134 [00:15<00:14,  4.87it/s, epoch=50, grad_norm=127.936, loss=292.9472, lr=5.50e-04]
Epoch 50 TRAIN:  49%|████▊     | 65/134 [00:15<00:14,  4.87it/s, epoch=50, grad_norm=105.689, loss=301.0554, lr=5.51e-04]
Epoch 50 TRAIN:  49%|████▊     | 65/134 [00:15<00:14,  4.87it/s, epoch=50, grad_norm=91.884, loss=283.2991, lr=5.51e-04] 
Epoch 50 TRAIN:  49%|████▊     | 65/134 [00:16<00:14,  4.87it/s, epoch=50, grad_norm=98.995, loss=307.9080, lr=5.51e-04]
Epoch 50 TRAIN:  51%|█████     | 68/134 [00:16<00:13,  4.86it/s, epoch=50, grad_norm=98.995, loss=307.9080, lr=5.51e-04]
Epoch 50 TRAIN:  51%|█████     | 68/134 [00:16<00:13,  4.86it/s, epoch=50, grad_norm=92.908, loss=328.4634, lr=5.51e-04]
Epoch 50 TRAIN:  51%|█████     | 68/134 [00:16<00:13,  4.86it/s, epoch=50, grad_norm=80.598, loss=317.2619, lr=5.51e-04]
Epoch 50 TRAIN:  51%|█████     | 68/134 [00:16<00:13,  4.86it/s, epoch=50, grad_norm=118.619, loss=304.3368, lr=5.51e-04]
Epoch 50 TRAIN:  53%|█████▎    | 71/134 [00:16<00:13,  4.73it/s, epoch=50, grad_norm=118.619, loss=304.3368, lr=5.51e-04]
Epoch 50 TRAIN:  53%|█████▎    | 71/134 [00:16<00:13,  4.73it/s, epoch=50, grad_norm=96.773, loss=331.4191, lr=5.51e-04] 
Epoch 50 TRAIN:  53%|█████▎    | 71/134 [00:17<00:13,  4.73it/s, epoch=50, grad_norm=290.924, loss=315.3080, lr=5.51e-04]
Epoch 50 TRAIN:  53%|█████▎    | 71/134 [00:17<00:13,  4.73it/s, epoch=50, grad_norm=105.821, loss=320.8969, lr=5.51e-04]
Epoch 50 TRAIN:  55%|█████▌    | 74/134 [00:17<00:12,  4.73it/s, epoch=50, grad_norm=105.821, loss=320.8969, lr=5.51e-04]
Epoch 50 TRAIN:  55%|█████▌    | 74/134 [00:17<00:12,  4.73it/s, epoch=50, grad_norm=158.387, loss=315.1165, lr=5.51e-04]
Epoch 50 TRAIN:  55%|█████▌    | 74/134 [00:17<00:12,  4.73it/s, epoch=50, grad_norm=131.781, loss=346.9274, lr=5.51e-04]
Epoch 50 TRAIN:  55%|█████▌    | 74/134 [00:18<00:12,  4.73it/s, epoch=50, grad_norm=176.714, loss=310.2148, lr=5.51e-04]
Epoch 50 TRAIN:  57%|█████▋    | 77/134 [00:18<00:12,  4.72it/s, epoch=50, grad_norm=176.714, loss=310.2148, lr=5.51e-04]
Epoch 50 TRAIN:  57%|█████▋    | 77/134 [00:18<00:12,  4.72it/s, epoch=50, grad_norm=151.215, loss=308.8914, lr=5.52e-04]
Epoch 50 TRAIN:  57%|█████▋    | 77/134 [00:18<00:12,  4.72it/s, epoch=50, grad_norm=108.618, loss=316.9174, lr=5.52e-04]
Epoch 50 TRAIN:  57%|█████▋    | 77/134 [00:18<00:12,  4.72it/s, epoch=50, grad_norm=99.246, loss=330.9844, lr=5.52e-04] 
Epoch 50 TRAIN:  60%|█████▉    | 80/134 [00:18<00:11,  4.76it/s, epoch=50, grad_norm=99.246, loss=330.9844, lr=5.52e-04]
Epoch 50 TRAIN:  60%|█████▉    | 80/134 [00:18<00:11,  4.76it/s, epoch=50, grad_norm=213.389, loss=309.8116, lr=5.52e-04]
Epoch 50 TRAIN:  60%|█████▉    | 80/134 [00:19<00:11,  4.76it/s, epoch=50, grad_norm=115.568, loss=316.0775, lr=5.52e-04]
Epoch 50 TRAIN:  60%|█████▉    | 80/134 [00:19<00:11,  4.76it/s, epoch=50, grad_norm=119.471, loss=296.8835, lr=5.52e-04]
Epoch 50 TRAIN:  62%|██████▏   | 83/134 [00:19<00:10,  4.78it/s, epoch=50, grad_norm=119.471, loss=296.8835, lr=5.52e-04]
Epoch 50 TRAIN:  62%|██████▏   | 83/134 [00:19<00:10,  4.78it/s, epoch=50, grad_norm=152.539, loss=231.2019, lr=5.52e-04]
Epoch 50 TRAIN:  62%|██████▏   | 83/134 [00:19<00:10,  4.78it/s, epoch=50, grad_norm=103.753, loss=240.9928, lr=5.52e-04]
Epoch 50 TRAIN:  62%|██████▏   | 83/134 [00:20<00:10,  4.78it/s, epoch=50, grad_norm=174.079, loss=288.5502, lr=5.52e-04]
Epoch 50 TRAIN:  64%|██████▍   | 86/134 [00:20<00:11,  4.36it/s, epoch=50, grad_norm=174.079, loss=288.5502, lr=5.52e-04]
Epoch 50 TRAIN:  64%|██████▍   | 86/134 [00:20<00:11,  4.36it/s, epoch=50, grad_norm=285.536, loss=321.5944, lr=5.52e-04]
Epoch 50 TRAIN:  64%|██████▍   | 86/134 [00:20<00:11,  4.36it/s, epoch=50, grad_norm=259.009, loss=358.1805, lr=5.52e-04]
Epoch 50 TRAIN:  64%|██████▍   | 86/134 [00:20<00:11,  4.36it/s, epoch=50, grad_norm=247.520, loss=235.4207, lr=5.52e-04]
Epoch 50 TRAIN:  66%|██████▋   | 89/134 [00:20<00:09,  4.68it/s, epoch=50, grad_norm=247.520, loss=235.4207, lr=5.52e-04]
Epoch 50 TRAIN:  66%|██████▋   | 89/134 [00:20<00:09,  4.68it/s, epoch=50, grad_norm=115.031, loss=279.8090, lr=5.52e-04]
Epoch 50 TRAIN:  66%|██████▋   | 89/134 [00:21<00:09,  4.68it/s, epoch=50, grad_norm=128.719, loss=299.1524, lr=5.53e-04]
Epoch 50 TRAIN:  66%|██████▋   | 89/134 [00:21<00:09,  4.68it/s, epoch=50, grad_norm=191.592, loss=244.9495, lr=5.53e-04]
Epoch 50 TRAIN:  69%|██████▊   | 92/134 [00:21<00:08,  4.86it/s, epoch=50, grad_norm=191.592, loss=244.9495, lr=5.53e-04]
Epoch 50 TRAIN:  69%|██████▊   | 92/134 [00:21<00:08,  4.86it/s, epoch=50, grad_norm=158.701, loss=257.6471, lr=5.53e-04]
Epoch 50 TRAIN:  69%|██████▊   | 92/134 [00:21<00:08,  4.86it/s, epoch=50, grad_norm=97.811, loss=228.0623, lr=5.53e-04] 
Epoch 50 TRAIN:  69%|██████▊   | 92/134 [00:21<00:08,  4.86it/s, epoch=50, grad_norm=277.328, loss=301.4006, lr=5.53e-04]
Epoch 50 TRAIN:  71%|███████   | 95/134 [00:21<00:08,  4.78it/s, epoch=50, grad_norm=277.328, loss=301.4006, lr=5.53e-04]
Epoch 50 TRAIN:  71%|███████   | 95/134 [00:22<00:08,  4.78it/s, epoch=50, grad_norm=294.918, loss=326.8679, lr=5.53e-04]
Epoch 50 TRAIN:  71%|███████   | 95/134 [00:22<00:08,  4.78it/s, epoch=50, grad_norm=217.543, loss=298.2688, lr=5.53e-04]
Epoch 50 TRAIN:  71%|███████   | 95/134 [00:22<00:08,  4.78it/s, epoch=50, grad_norm=118.210, loss=309.2062, lr=5.53e-04]
Epoch 50 TRAIN:  73%|███████▎  | 98/134 [00:22<00:07,  4.58it/s, epoch=50, grad_norm=118.210, loss=309.2062, lr=5.53e-04]
Epoch 50 TRAIN:  73%|███████▎  | 98/134 [00:22<00:07,  4.58it/s, epoch=50, grad_norm=332.542, loss=282.7969, lr=5.53e-04]
Epoch 50 TRAIN:  73%|███████▎  | 98/134 [00:23<00:07,  4.58it/s, epoch=50, grad_norm=309.021, loss=224.9676, lr=5.53e-04]2026-04-16 20:17:33,032 DEBUG TRAIN | steps/sec 4.307| Batch 50/100 loss 224.967560 loss_att 183.938980 loss_ctc 320.700928 th_accuracy 0.298701 lr 5.5328e-04 grad_norm 309.021454 rank 0

Epoch 50 TRAIN:  73%|███████▎  | 98/134 [00:23<00:07,  4.58it/s, epoch=50, grad_norm=124.393, loss=304.7310, lr=5.53e-04]
Epoch 50 TRAIN:  75%|███████▌  | 101/134 [00:23<00:07,  4.41it/s, epoch=50, grad_norm=124.393, loss=304.7310, lr=5.53e-04]
Epoch 50 TRAIN:  75%|███████▌  | 101/134 [00:23<00:07,  4.41it/s, epoch=50, grad_norm=198.920, loss=342.3906, lr=5.53e-04]
Epoch 50 TRAIN:  75%|███████▌  | 101/134 [00:23<00:07,  4.41it/s, epoch=50, grad_norm=87.631, loss=232.8498, lr=5.54e-04] 
Epoch 50 TRAIN:  75%|███████▌  | 101/134 [00:23<00:07,  4.41it/s, epoch=50, grad_norm=234.644, loss=312.5791, lr=5.54e-04]
Epoch 50 TRAIN:  78%|███████▊  | 104/134 [00:23<00:06,  4.43it/s, epoch=50, grad_norm=234.644, loss=312.5791, lr=5.54e-04]
Epoch 50 TRAIN:  78%|███████▊  | 104/134 [00:24<00:06,  4.43it/s, epoch=50, grad_norm=169.716, loss=359.6290, lr=5.54e-04]
Epoch 50 TRAIN:  78%|███████▊  | 104/134 [00:24<00:06,  4.43it/s, epoch=50, grad_norm=118.042, loss=304.6663, lr=5.54e-04]
Epoch 50 TRAIN:  78%|███████▊  | 104/134 [00:24<00:06,  4.43it/s, epoch=50, grad_norm=223.043, loss=262.9473, lr=5.54e-04]
Epoch 50 TRAIN:  80%|███████▉  | 107/134 [00:24<00:05,  4.58it/s, epoch=50, grad_norm=223.043, loss=262.9473, lr=5.54e-04]
Epoch 50 TRAIN:  80%|███████▉  | 107/134 [00:24<00:05,  4.58it/s, epoch=50, grad_norm=254.966, loss=321.5471, lr=5.54e-04]
Epoch 50 TRAIN:  80%|███████▉  | 107/134 [00:25<00:05,  4.58it/s, epoch=50, grad_norm=228.425, loss=287.7655, lr=5.54e-04]
Epoch 50 TRAIN:  80%|███████▉  | 107/134 [00:25<00:05,  4.58it/s, epoch=50, grad_norm=132.958, loss=289.1264, lr=5.54e-04]
Epoch 50 TRAIN:  82%|████████▏ | 110/134 [00:25<00:05,  4.67it/s, epoch=50, grad_norm=132.958, loss=289.1264, lr=5.54e-04]
Epoch 50 TRAIN:  82%|████████▏ | 110/134 [00:25<00:05,  4.67it/s, epoch=50, grad_norm=117.108, loss=280.9593, lr=5.54e-04]
Epoch 50 TRAIN:  82%|████████▏ | 110/134 [00:25<00:05,  4.67it/s, epoch=50, grad_norm=108.319, loss=191.5118, lr=5.54e-04]
Epoch 50 TRAIN:  82%|████████▏ | 110/134 [00:25<00:05,  4.67it/s, epoch=50, grad_norm=224.364, loss=374.1028, lr=5.54e-04]
Epoch 50 TRAIN:  84%|████████▍ | 113/134 [00:25<00:04,  4.87it/s, epoch=50, grad_norm=224.364, loss=374.1028, lr=5.54e-04]
Epoch 50 TRAIN:  84%|████████▍ | 113/134 [00:25<00:04,  4.87it/s, epoch=50, grad_norm=111.059, loss=241.3620, lr=5.54e-04]
Epoch 50 TRAIN:  84%|████████▍ | 113/134 [00:26<00:04,  4.87it/s, epoch=50, grad_norm=205.182, loss=354.4318, lr=5.54e-04]
Epoch 50 TRAIN:  84%|████████▍ | 113/134 [00:26<00:04,  4.87it/s, epoch=50, grad_norm=110.278, loss=277.6252, lr=5.55e-04]
Epoch 50 TRAIN:  87%|████████▋ | 116/134 [00:26<00:03,  5.02it/s, epoch=50, grad_norm=110.278, loss=277.6252, lr=5.55e-04]
Epoch 50 TRAIN:  87%|████████▋ | 116/134 [00:26<00:03,  5.02it/s, epoch=50, grad_norm=128.710, loss=280.3645, lr=5.55e-04]
Epoch 50 TRAIN:  87%|████████▋ | 116/134 [00:26<00:03,  5.02it/s, epoch=50, grad_norm=95.827, loss=248.7361, lr=5.55e-04] 
Epoch 50 TRAIN:  87%|████████▋ | 116/134 [00:27<00:03,  5.02it/s, epoch=50, grad_norm=120.305, loss=274.4596, lr=5.55e-04]
Epoch 50 TRAIN:  89%|████████▉ | 119/134 [00:27<00:03,  4.61it/s, epoch=50, grad_norm=120.305, loss=274.4596, lr=5.55e-04]
Epoch 50 TRAIN:  89%|████████▉ | 119/134 [00:27<00:03,  4.61it/s, epoch=50, grad_norm=304.859, loss=406.8041, lr=5.55e-04]
Epoch 50 TRAIN:  89%|████████▉ | 119/134 [00:27<00:03,  4.61it/s, epoch=50, grad_norm=145.569, loss=303.6634, lr=5.55e-04]
Epoch 50 TRAIN:  89%|████████▉ | 119/134 [00:27<00:03,  4.61it/s, epoch=50, grad_norm=213.043, loss=344.3593, lr=5.55e-04]
Epoch 50 TRAIN:  91%|█████████ | 122/134 [00:27<00:02,  4.76it/s, epoch=50, grad_norm=213.043, loss=344.3593, lr=5.55e-04]
Epoch 50 TRAIN:  91%|█████████ | 122/134 [00:27<00:02,  4.76it/s, epoch=50, grad_norm=269.895, loss=379.9814, lr=5.55e-04]
Epoch 50 TRAIN:  91%|█████████ | 122/134 [00:27<00:02,  4.76it/s, epoch=50, grad_norm=249.230, loss=399.3238, lr=5.55e-04]
Epoch 50 TRAIN:  91%|█████████ | 122/134 [00:28<00:02,  4.76it/s, epoch=50, grad_norm=179.875, loss=400.8039, lr=5.55e-04]
Epoch 50 TRAIN:  91%|█████████ | 122/134 [00:28<00:02,  4.76it/s, epoch=50, grad_norm=134.380, loss=287.6119, lr=5.55e-04]
Epoch 50 TRAIN:  94%|█████████▍| 126/134 [00:28<00:01,  5.25it/s, epoch=50, grad_norm=134.380, loss=287.6119, lr=5.55e-04]
Epoch 50 TRAIN:  94%|█████████▍| 126/134 [00:28<00:01,  5.25it/s, epoch=50, grad_norm=118.968, loss=271.7906, lr=5.55e-04]
Epoch 50 TRAIN:  94%|█████████▍| 126/134 [00:28<00:01,  5.25it/s, epoch=50, grad_norm=96.131, loss=271.9014, lr=5.56e-04] 
Epoch 50 TRAIN:  94%|█████████▍| 126/134 [00:28<00:01,  5.25it/s, epoch=50, grad_norm=172.038, loss=350.9424, lr=5.56e-04]
Epoch 50 TRAIN:  96%|█████████▋| 129/134 [00:28<00:01,  4.96it/s, epoch=50, grad_norm=172.038, loss=350.9424, lr=5.56e-04]
Epoch 50 TRAIN:  96%|█████████▋| 129/134 [00:29<00:01,  4.96it/s, epoch=50, grad_norm=143.794, loss=297.2287, lr=5.56e-04]
Epoch 50 TRAIN:  96%|█████████▋| 129/134 [00:29<00:01,  4.96it/s, epoch=50, grad_norm=359.902, loss=428.0584, lr=5.56e-04]
Epoch 50 TRAIN:  96%|█████████▋| 129/134 [00:29<00:01,  4.96it/s, epoch=50, grad_norm=149.229, loss=331.5480, lr=5.56e-04]
Epoch 50 TRAIN:  99%|█████████▊| 132/134 [00:29<00:00,  4.96it/s, epoch=50, grad_norm=149.229, loss=331.5480, lr=5.56e-04]
Epoch 50 TRAIN:  99%|█████████▊| 132/134 [00:29<00:00,  4.96it/s, epoch=50, grad_norm=169.987, loss=349.9554, lr=5.56e-04]
Epoch 50 TRAIN:  99%|█████████▊| 132/134 [00:29<00:00,  4.96it/s, epoch=50, grad_norm=239.612, loss=293.7404, lr=5.56e-04]
Epoch 50 TRAIN:  99%|█████████▊| 132/134 [00:30<00:00,  4.96it/s, epoch=50, grad_norm=198.063, loss=402.8474, lr=5.56e-04]
Epoch 50 TRAIN:  99%|█████████▊| 132/134 [00:30<00:00,  4.96it/s, epoch=50, grad_norm=406.254, loss=297.7799, lr=5.56e-04]
Epoch 50 TRAIN: 136it [00:30,  5.60it/s, epoch=50, grad_norm=406.254, loss=297.7799, lr=5.56e-04]                         
Epoch 50 TRAIN: 136it [00:30,  5.60it/s, epoch=50, grad_norm=269.158, loss=278.4741, lr=5.56e-04]
Epoch 50 TRAIN: 136it [00:30,  5.60it/s, epoch=50, grad_norm=217.193, loss=298.3688, lr=5.56e-04]
Epoch 50 TRAIN: 138it [00:30,  4.53it/s, epoch=50, grad_norm=217.193, loss=298.3688, lr=5.56e-04]

CV:   0%|          | 0/6 [00:00<?, ?it/s]/home/rookie/Documents/batuka/projects/wenet/wenet/wenet/utils/train_utils.py:689: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  amp_autocast(enabled=dtype is not None,

CV:  17%|█▋        | 1/6 [00:00<00:03,  1.32it/s]
                                                 
2026-04-16 20:17:41,335 INFO Epoch 50 Step 6954 CV info lr 5.5632e-04 cv_loss 471.1755648526278 rank 0 acc 0.3215494205554326
2026-04-16 20:17:41,336 INFO [Rank 0] Checkpoint: save to checkpoint exp/conformer_mongolian/epoch_50.pt
2026-04-16 20:17:41,445 INFO Epoch 51 Step 6954 TRAIN info lr 5.5632e-04 rank 0
2026-04-16 20:17:41,458 INFO using accumulate grad, new batch size is 1 times larger than before

Epoch 51 TRAIN:   0%|          | 0/138 [00:00<?, ?it/s]
Epoch 51 TRAIN:   0%|          | 0/138 [00:04<?, ?it/s, epoch=51, grad_norm=389.644, loss=327.2249, lr=5.56e-04]
Epoch 51 TRAIN:   1%|          | 1/138 [00:04<09:29,  4.16s/it, epoch=51, grad_norm=389.644, loss=327.2249, lr=5.56e-04]
Epoch 51 TRAIN:   1%|          | 1/138 [00:04<09:29,  4.16s/it, epoch=51, grad_norm=539.692, loss=426.2015, lr=5.56e-04]
Epoch 51 TRAIN:   1%|          | 1/138 [00:04<09:29,  4.16s/it, epoch=51, grad_norm=453.246, loss=366.3832, lr=5.57e-04]
Epoch 51 TRAIN:   1%|          | 1/138 [00:04<09:29,  4.16s/it, epoch=51, grad_norm=305.703, loss=266.1357, lr=5.57e-04]
Epoch 51 TRAIN:   1%|          | 1/138 [00:04<09:29,  4.16s/it, epoch=51, grad_norm=392.227, loss=375.3906, lr=5.57e-04]
Epoch 51 TRAIN:   1%|          | 1/138 [00:04<09:29,  4.16s/it, epoch=51, grad_norm=329.874, loss=319.9772, lr=5.57e-04]
Epoch 51 TRAIN:   1%|          | 1/138 [00:04<09:29,  4.16s/it, epoch=51, grad_norm=259.202, loss=275.7001, lr=5.57e-04]
Epoch 51 TRAIN:   1%|          | 1/138 [00:04<09:29,  4.16s/it, epoch=51, grad_norm=366.630, loss=399.9150, lr=5.57e-04]
Epoch 51 TRAIN:   1%|          | 1/138 [00:04<09:29,  4.16s/it, epoch=51, grad_norm=184.709, loss=317.8435, lr=5.57e-04]
Epoch 51 TRAIN:   7%|▋         | 9/138 [00:04<00:52,  2.46it/s, epoch=51, grad_norm=184.709, loss=317.8435, lr=5.57e-04]
Epoch 51 TRAIN:   7%|▋         | 9/138 [00:04<00:52,  2.46it/s, epoch=51, grad_norm=76.888, loss=251.1711, lr=5.57e-04] 
Epoch 51 TRAIN:   7%|▋         | 9/138 [00:05<00:52,  2.46it/s, epoch=51, grad_norm=112.332, loss=318.1162, lr=5.57e-04]
Epoch 51 TRAIN:   7%|▋         | 9/138 [00:05<00:52,  2.46it/s, epoch=51, grad_norm=129.072, loss=357.7549, lr=5.57e-04]
Epoch 51 TRAIN:   7%|▋         | 9/138 [00:05<00:52,  2.46it/s, epoch=51, grad_norm=189.433, loss=331.4771, lr=5.57e-04]
Epoch 51 TRAIN:   9%|▉         | 13/138 [00:05<00:38,  3.24it/s, epoch=51, grad_norm=189.433, loss=331.4771, lr=5.57e-04]
Epoch 51 TRAIN:   9%|▉         | 13/138 [00:05<00:38,  3.24it/s, epoch=51, grad_norm=122.262, loss=327.7977, lr=5.57e-04]
Epoch 51 TRAIN:   9%|▉         | 13/138 [00:05<00:38,  3.24it/s, epoch=51, grad_norm=149.644, loss=298.9808, lr=5.58e-04]
Epoch 51 TRAIN:   9%|▉         | 13/138 [00:05<00:38,  3.24it/s, epoch=51, grad_norm=283.897, loss=258.3003, lr=5.58e-04]
Epoch 51 TRAIN:   9%|▉         | 13/138 [00:06<00:38,  3.24it/s, epoch=51, grad_norm=198.912, loss=301.1409, lr=5.58e-04]
Epoch 51 TRAIN:  12%|█▏        | 17/138 [00:06<00:30,  3.91it/s, epoch=51, grad_norm=198.912, loss=301.1409, lr=5.58e-04]
Epoch 51 TRAIN:  12%|█▏        | 17/138 [00:06<00:30,  3.91it/s, epoch=51, grad_norm=119.005, loss=321.3591, lr=5.58e-04]
Epoch 51 TRAIN:  12%|█▏        | 17/138 [00:06<00:30,  3.91it/s, epoch=51, grad_norm=101.511, loss=349.9460, lr=5.58e-04]
Epoch 51 TRAIN:  12%|█▏        | 17/138 [00:06<00:30,  3.91it/s, epoch=51, grad_norm=79.801, loss=316.3442, lr=5.58e-04] 
Epoch 51 TRAIN:  12%|█▏        | 17/138 [00:06<00:30,  3.91it/s, epoch=51, grad_norm=95.230, loss=312.3088, lr=5.58e-04]
Epoch 51 TRAIN:  15%|█▌        | 21/138 [00:06<00:26,  4.41it/s, epoch=51, grad_norm=95.230, loss=312.3088, lr=5.58e-04]
Epoch 51 TRAIN:  15%|█▌        | 21/138 [00:06<00:26,  4.41it/s, epoch=51, grad_norm=130.370, loss=289.6299, lr=5.58e-04]
Epoch 51 TRAIN:  15%|█▌        | 21/138 [00:07<00:26,  4.41it/s, epoch=51, grad_norm=93.349, loss=290.2045, lr=5.58e-04] 
Epoch 51 TRAIN:  15%|█▌        | 21/138 [00:07<00:26,  4.41it/s, epoch=51, grad_norm=86.280, loss=271.1464, lr=5.58e-04]
Epoch 51 TRAIN:  15%|█▌        | 21/138 [00:07<00:26,  4.41it/s, epoch=51, grad_norm=128.974, loss=295.8787, lr=5.58e-04]
Epoch 51 TRAIN:  18%|█▊        | 25/138 [00:07<00:24,  4.63it/s, epoch=51, grad_norm=128.974, loss=295.8787, lr=5.58e-04]
Epoch 51 TRAIN:  18%|█▊        | 25/138 [00:07<00:24,  4.63it/s, epoch=51, grad_norm=148.482, loss=331.6849, lr=5.58e-04]
Epoch 51 TRAIN:  18%|█▊        | 25/138 [00:07<00:24,  4.63it/s, epoch=51, grad_norm=125.393, loss=337.6073, lr=5.58e-04]
Epoch 51 TRAIN:  18%|█▊        | 25/138 [00:08<00:24,  4.63it/s, epoch=51, grad_norm=113.502, loss=280.3265, lr=5.59e-04]
Epoch 51 TRAIN:  20%|██        | 28/138 [00:08<00:22,  4.83it/s, epoch=51, grad_norm=113.502, loss=280.3265, lr=5.59e-04]
Epoch 51 TRAIN:  20%|██        | 28/138 [00:08<00:22,  4.83it/s, epoch=51, grad_norm=200.490, loss=252.5049, lr=5.59e-04]
Epoch 51 TRAIN:  20%|██        | 28/138 [00:08<00:22,  4.83it/s, epoch=51, grad_norm=132.847, loss=317.9417, lr=5.59e-04]
Epoch 51 TRAIN:  20%|██        | 28/138 [00:08<00:22,  4.83it/s, epoch=51, grad_norm=119.201, loss=305.2689, lr=5.59e-04]
Epoch 51 TRAIN:  22%|██▏       | 31/138 [00:08<00:21,  4.95it/s, epoch=51, grad_norm=119.201, loss=305.2689, lr=5.59e-04]
Epoch 51 TRAIN:  22%|██▏       | 31/138 [00:08<00:21,  4.95it/s, epoch=51, grad_norm=116.595, loss=287.3978, lr=5.59e-04]
Epoch 51 TRAIN:  22%|██▏       | 31/138 [00:09<00:21,  4.95it/s, epoch=51, grad_norm=81.614, loss=300.5698, lr=5.59e-04] 
Epoch 51 TRAIN:  22%|██▏       | 31/138 [00:09<00:21,  4.95it/s, epoch=51, grad_norm=115.853, loss=288.7859, lr=5.59e-04]
Epoch 51 TRAIN:  25%|██▍       | 34/138 [00:09<00:20,  5.04it/s, epoch=51, grad_norm=115.853, loss=288.7859, lr=5.59e-04]
Epoch 51 TRAIN:  25%|██▍       | 34/138 [00:09<00:20,  5.04it/s, epoch=51, grad_norm=200.295, loss=325.7915, lr=5.59e-04]
Epoch 51 TRAIN:  25%|██▍       | 34/138 [00:09<00:20,  5.04it/s, epoch=51, grad_norm=195.690, loss=336.4067, lr=5.59e-04]
Epoch 51 TRAIN:  25%|██▍       | 34/138 [00:09<00:20,  5.04it/s, epoch=51, grad_norm=133.823, loss=260.1345, lr=5.59e-04]
Epoch 51 TRAIN:  27%|██▋       | 37/138 [00:09<00:20,  4.91it/s, epoch=51, grad_norm=133.823, loss=260.1345, lr=5.59e-04]
Epoch 51 TRAIN:  27%|██▋       | 37/138 [00:10<00:20,  4.91it/s, epoch=51, grad_norm=91.808, loss=308.7462, lr=5.59e-04] 
Epoch 51 TRAIN:  27%|██▋       | 37/138 [00:10<00:20,  4.91it/s, epoch=51, grad_norm=99.081, loss=291.2430, lr=5.59e-04]
Epoch 51 TRAIN:  27%|██▋       | 37/138 [00:10<00:20,  4.91it/s, epoch=51, grad_norm=151.445, loss=276.1539, lr=5.60e-04]
Epoch 51 TRAIN:  29%|██▉       | 40/138 [00:10<00:19,  4.98it/s, epoch=51, grad_norm=151.445, loss=276.1539, lr=5.60e-04]
Epoch 51 TRAIN:  29%|██▉       | 40/138 [00:10<00:19,  4.98it/s, epoch=51, grad_norm=108.671, loss=311.1402, lr=5.60e-04]
Epoch 51 TRAIN:  29%|██▉       | 40/138 [00:10<00:19,  4.98it/s, epoch=51, grad_norm=143.940, loss=344.2936, lr=5.60e-04]
Epoch 51 TRAIN:  29%|██▉       | 40/138 [00:11<00:19,  4.98it/s, epoch=51, grad_norm=91.080, loss=308.8559, lr=5.60e-04] 
Epoch 51 TRAIN:  31%|███       | 43/138 [00:11<00:18,  5.01it/s, epoch=51, grad_norm=91.080, loss=308.8559, lr=5.60e-04]
Epoch 51 TRAIN:  31%|███       | 43/138 [00:11<00:18,  5.01it/s, epoch=51, grad_norm=172.738, loss=334.1409, lr=5.60e-04]
Epoch 51 TRAIN:  31%|███       | 43/138 [00:11<00:18,  5.01it/s, epoch=51, grad_norm=146.418, loss=293.8478, lr=5.60e-04]
Epoch 51 TRAIN:  31%|███       | 43/138 [00:11<00:18,  5.01it/s, epoch=51, grad_norm=131.072, loss=290.0242, lr=5.60e-04]
Epoch 51 TRAIN:  33%|███▎      | 46/138 [00:11<00:18,  5.00it/s, epoch=51, grad_norm=131.072, loss=290.0242, lr=5.60e-04]
Epoch 51 TRAIN:  33%|███▎      | 46/138 [00:11<00:18,  5.00it/s, epoch=51, grad_norm=116.075, loss=323.7622, lr=5.60e-04]
Epoch 51 TRAIN:  33%|███▎      | 46/138 [00:12<00:18,  5.00it/s, epoch=51, grad_norm=175.716, loss=358.3636, lr=5.60e-04]
Epoch 51 TRAIN:  33%|███▎      | 46/138 [00:12<00:18,  5.00it/s, epoch=51, grad_norm=103.810, loss=276.3456, lr=5.60e-04]
Epoch 51 TRAIN:  36%|███▌      | 49/138 [00:12<00:17,  5.01it/s, epoch=51, grad_norm=103.810, loss=276.3456, lr=5.60e-04]
Epoch 51 TRAIN:  36%|███▌      | 49/138 [00:12<00:17,  5.01it/s, epoch=51, grad_norm=151.019, loss=330.3692, lr=5.60e-04]
Epoch 51 TRAIN:  36%|███▌      | 49/138 [00:12<00:17,  5.01it/s, epoch=51, grad_norm=194.783, loss=297.2844, lr=5.60e-04]
Epoch 51 TRAIN:  36%|███▌      | 49/138 [00:12<00:17,  5.01it/s, epoch=51, grad_norm=181.912, loss=281.0097, lr=5.60e-04]
Epoch 51 TRAIN:  38%|███▊      | 52/138 [00:12<00:16,  5.09it/s, epoch=51, grad_norm=181.912, loss=281.0097, lr=5.60e-04]
Epoch 51 TRAIN:  38%|███▊      | 52/138 [00:13<00:16,  5.09it/s, epoch=51, grad_norm=224.510, loss=364.0662, lr=5.61e-04]
Epoch 51 TRAIN:  38%|███▊      | 52/138 [00:13<00:16,  5.09it/s, epoch=51, grad_norm=295.541, loss=334.8529, lr=5.61e-04]
Epoch 51 TRAIN:  38%|███▊      | 52/138 [00:13<00:16,  5.09it/s, epoch=51, grad_norm=118.196, loss=281.1497, lr=5.61e-04]
Epoch 51 TRAIN:  40%|███▉      | 55/138 [00:13<00:17,  4.79it/s, epoch=51, grad_norm=118.196, loss=281.1497, lr=5.61e-04]
Epoch 51 TRAIN:  40%|███▉      | 55/138 [00:13<00:17,  4.79it/s, epoch=51, grad_norm=97.962, loss=295.0496, lr=5.61e-04] 
Epoch 51 TRAIN:  40%|███▉      | 55/138 [00:13<00:17,  4.79it/s, epoch=51, grad_norm=109.235, loss=333.4629, lr=5.61e-04]
Epoch 51 TRAIN:  40%|███▉      | 55/138 [00:14<00:17,  4.79it/s, epoch=51, grad_norm=86.311, loss=310.4280, lr=5.61e-04] 
Epoch 51 TRAIN:  42%|████▏     | 58/138 [00:14<00:16,  4.83it/s, epoch=51, grad_norm=86.311, loss=310.4280, lr=5.61e-04]
Epoch 51 TRAIN:  42%|████▏     | 58/138 [00:14<00:16,  4.83it/s, epoch=51, grad_norm=89.328, loss=315.5295, lr=5.61e-04]
Epoch 51 TRAIN:  42%|████▏     | 58/138 [00:14<00:16,  4.83it/s, epoch=51, grad_norm=88.980, loss=287.1581, lr=5.61e-04]
Epoch 51 TRAIN:  42%|████▏     | 58/138 [00:14<00:16,  4.83it/s, epoch=51, grad_norm=80.935, loss=326.3854, lr=5.61e-04]
Epoch 51 TRAIN:  44%|████▍     | 61/138 [00:14<00:15,  4.86it/s, epoch=51, grad_norm=80.935, loss=326.3854, lr=5.61e-04]
Epoch 51 TRAIN:  44%|████▍     | 61/138 [00:14<00:15,  4.86it/s, epoch=51, grad_norm=161.381, loss=291.0496, lr=5.61e-04]
Epoch 51 TRAIN:  44%|████▍     | 61/138 [00:15<00:15,  4.86it/s, epoch=51, grad_norm=103.662, loss=262.2878, lr=5.61e-04]
Epoch 51 TRAIN:  44%|████▍     | 61/138 [00:15<00:15,  4.86it/s, epoch=51, grad_norm=127.756, loss=276.9218, lr=5.61e-04]
Epoch 51 TRAIN:  46%|████▋     | 64/138 [00:15<00:15,  4.80it/s, epoch=51, grad_norm=127.756, loss=276.9218, lr=5.61e-04]
Epoch 51 TRAIN:  46%|████▋     | 64/138 [00:15<00:15,  4.80it/s, epoch=51, grad_norm=106.873, loss=236.3904, lr=5.62e-04]
Epoch 51 TRAIN:  46%|████▋     | 64/138 [00:15<00:15,  4.80it/s, epoch=51, grad_norm=196.253, loss=324.0396, lr=5.62e-04]
Epoch 51 TRAIN:  46%|████▋     | 64/138 [00:15<00:15,  4.80it/s, epoch=51, grad_norm=160.413, loss=319.9354, lr=5.62e-04]
Epoch 51 TRAIN:  49%|████▊     | 67/138 [00:15<00:14,  4.83it/s, epoch=51, grad_norm=160.413, loss=319.9354, lr=5.62e-04]
Epoch 51 TRAIN:  49%|████▊     | 67/138 [00:16<00:14,  4.83it/s, epoch=51, grad_norm=146.047, loss=306.1891, lr=5.62e-04]
Epoch 51 TRAIN:  49%|████▊     | 67/138 [00:16<00:14,  4.83it/s, epoch=51, grad_norm=125.493, loss=258.4683, lr=5.62e-04]
Epoch 51 TRAIN:  49%|████▊     | 67/138 [00:16<00:14,  4.83it/s, epoch=51, grad_norm=214.374, loss=259.7430, lr=5.62e-04]
Epoch 51 TRAIN:  51%|█████     | 70/138 [00:16<00:14,  4.77it/s, epoch=51, grad_norm=214.374, loss=259.7430, lr=5.62e-04]
Epoch 51 TRAIN:  51%|█████     | 70/138 [00:16<00:14,  4.77it/s, epoch=51, grad_norm=133.245, loss=266.0110, lr=5.62e-04]
Epoch 51 TRAIN:  51%|█████     | 70/138 [00:17<00:14,  4.77it/s, epoch=51, grad_norm=141.231, loss=305.8143, lr=5.62e-04]
Epoch 51 TRAIN:  51%|█████     | 70/138 [00:17<00:14,  4.77it/s, epoch=51, grad_norm=111.340, loss=291.0954, lr=5.62e-04]
Epoch 51 TRAIN:  53%|█████▎    | 73/138 [00:17<00:13,  4.84it/s, epoch=51, grad_norm=111.340, loss=291.0954, lr=5.62e-04]
Epoch 51 TRAIN:  53%|█████▎    | 73/138 [00:17<00:13,  4.84it/s, epoch=51, grad_norm=102.278, loss=263.4275, lr=5.62e-04]
Epoch 51 TRAIN:  53%|█████▎    | 73/138 [00:17<00:13,  4.84it/s, epoch=51, grad_norm=132.301, loss=306.4194, lr=5.62e-04]
Epoch 51 TRAIN:  53%|█████▎    | 73/138 [00:17<00:13,  4.84it/s, epoch=51, grad_norm=153.615, loss=333.9599, lr=5.62e-04]
Epoch 51 TRAIN:  55%|█████▌    | 76/138 [00:17<00:12,  4.89it/s, epoch=51, grad_norm=153.615, loss=333.9599, lr=5.62e-04]
Epoch 51 TRAIN:  55%|█████▌    | 76/138 [00:18<00:12,  4.89it/s, epoch=51, grad_norm=121.528, loss=299.5812, lr=5.62e-04]
Epoch 51 TRAIN:  55%|█████▌    | 76/138 [00:18<00:12,  4.89it/s, epoch=51, grad_norm=235.819, loss=215.8464, lr=5.63e-04]
Epoch 51 TRAIN:  55%|█████▌    | 76/138 [00:18<00:12,  4.89it/s, epoch=51, grad_norm=141.860, loss=311.7050, lr=5.63e-04]
Epoch 51 TRAIN:  57%|█████▋    | 79/138 [00:18<00:12,  4.83it/s, epoch=51, grad_norm=141.860, loss=311.7050, lr=5.63e-04]
Epoch 51 TRAIN:  57%|█████▋    | 79/138 [00:18<00:12,  4.83it/s, epoch=51, grad_norm=192.948, loss=279.8508, lr=5.63e-04]
Epoch 51 TRAIN:  57%|█████▋    | 79/138 [00:18<00:12,  4.83it/s, epoch=51, grad_norm=104.274, loss=273.5929, lr=5.63e-04]
Epoch 51 TRAIN:  57%|█████▋    | 79/138 [00:19<00:12,  4.83it/s, epoch=51, grad_norm=147.941, loss=337.3250, lr=5.63e-04]
Epoch 51 TRAIN:  59%|█████▉    | 82/138 [00:19<00:11,  4.79it/s, epoch=51, grad_norm=147.941, loss=337.3250, lr=5.63e-04]
Epoch 51 TRAIN:  59%|█████▉    | 82/138 [00:19<00:11,  4.79it/s, epoch=51, grad_norm=169.696, loss=292.4952, lr=5.63e-04]
Epoch 51 TRAIN:  59%|█████▉    | 82/138 [00:19<00:11,  4.79it/s, epoch=51, grad_norm=164.234, loss=316.4814, lr=5.63e-04]
Epoch 51 TRAIN:  59%|█████▉    | 82/138 [00:19<00:11,  4.79it/s, epoch=51, grad_norm=149.140, loss=289.2731, lr=5.63e-04]
Epoch 51 TRAIN:  62%|██████▏   | 85/138 [00:19<00:11,  4.80it/s, epoch=51, grad_norm=149.140, loss=289.2731, lr=5.63e-04]
Epoch 51 TRAIN:  62%|██████▏   | 85/138 [00:19<00:11,  4.80it/s, epoch=51, grad_norm=211.539, loss=274.7963, lr=5.63e-04]
Epoch 51 TRAIN:  62%|██████▏   | 85/138 [00:20<00:11,  4.80it/s, epoch=51, grad_norm=127.933, loss=287.2406, lr=5.63e-04]
Epoch 51 TRAIN:  62%|██████▏   | 85/138 [00:20<00:11,  4.80it/s, epoch=51, grad_norm=182.603, loss=293.3886, lr=5.63e-04]
Epoch 51 TRAIN:  64%|██████▍   | 88/138 [00:20<00:10,  4.89it/s, epoch=51, grad_norm=182.603, loss=293.3886, lr=5.63e-04]
Epoch 51 TRAIN:  64%|██████▍   | 88/138 [00:20<00:10,  4.89it/s, epoch=51, grad_norm=187.337, loss=199.5936, lr=5.63e-04]
Epoch 51 TRAIN:  64%|██████▍   | 88/138 [00:20<00:10,  4.89it/s, epoch=51, grad_norm=153.942, loss=316.9343, lr=5.64e-04]
Epoch 51 TRAIN:  64%|██████▍   | 88/138 [00:20<00:10,  4.89it/s, epoch=51, grad_norm=167.897, loss=325.3515, lr=5.64e-04]
Epoch 51 TRAIN:  66%|██████▌   | 91/138 [00:20<00:09,  4.82it/s, epoch=51, grad_norm=167.897, loss=325.3515, lr=5.64e-04]
Epoch 51 TRAIN:  66%|██████▌   | 91/138 [00:21<00:09,  4.82it/s, epoch=51, grad_norm=209.647, loss=362.5296, lr=5.64e-04]
Epoch 51 TRAIN:  66%|██████▌   | 91/138 [00:21<00:09,  4.82it/s, epoch=51, grad_norm=120.225, loss=320.1545, lr=5.64e-04]
Epoch 51 TRAIN:  66%|██████▌   | 91/138 [00:21<00:09,  4.82it/s, epoch=51, grad_norm=151.594, loss=266.8407, lr=5.64e-04]
Epoch 51 TRAIN:  68%|██████▊   | 94/138 [00:21<00:08,  4.90it/s, epoch=51, grad_norm=151.594, loss=266.8407, lr=5.64e-04]
Epoch 51 TRAIN:  68%|██████▊   | 94/138 [00:21<00:08,  4.90it/s, epoch=51, grad_norm=152.941, loss=331.3310, lr=5.64e-04]
Epoch 51 TRAIN:  68%|██████▊   | 94/138 [00:21<00:08,  4.90it/s, epoch=51, grad_norm=235.340, loss=311.4800, lr=5.64e-04]
Epoch 51 TRAIN:  68%|██████▊   | 94/138 [00:22<00:08,  4.90it/s, epoch=51, grad_norm=217.413, loss=232.2809, lr=5.64e-04]
Epoch 51 TRAIN:  70%|███████   | 97/138 [00:22<00:08,  4.98it/s, epoch=51, grad_norm=217.413, loss=232.2809, lr=5.64e-04]
Epoch 51 TRAIN:  70%|███████   | 97/138 [00:22<00:08,  4.98it/s, epoch=51, grad_norm=126.922, loss=312.2309, lr=5.64e-04]
Epoch 51 TRAIN:  70%|███████   | 97/138 [00:22<00:08,  4.98it/s, epoch=51, grad_norm=232.377, loss=236.7281, lr=5.64e-04]
Epoch 51 TRAIN:  70%|███████   | 97/138 [00:22<00:08,  4.98it/s, epoch=51, grad_norm=188.879, loss=282.6964, lr=5.64e-04]2026-04-16 20:18:04,268 DEBUG TRAIN | steps/sec 4.418| Batch 51/100 loss 282.696350 loss_att 239.474121 loss_ctc 383.548187 th_accuracy 0.368675 lr 5.6432e-04 grad_norm 188.879349 rank 0

Epoch 51 TRAIN:  72%|███████▏  | 100/138 [00:22<00:07,  4.78it/s, epoch=51, grad_norm=188.879, loss=282.6964, lr=5.64e-04]
Epoch 51 TRAIN:  72%|███████▏  | 100/138 [00:23<00:07,  4.78it/s, epoch=51, grad_norm=167.003, loss=233.5746, lr=5.64e-04]
Epoch 51 TRAIN:  72%|███████▏  | 100/138 [00:23<00:07,  4.78it/s, epoch=51, grad_norm=89.668, loss=250.3029, lr=5.64e-04] 
Epoch 51 TRAIN:  72%|███████▏  | 100/138 [00:23<00:07,  4.78it/s, epoch=51, grad_norm=353.448, loss=375.4084, lr=5.65e-04]
Epoch 51 TRAIN:  75%|███████▍  | 103/138 [00:23<00:07,  4.84it/s, epoch=51, grad_norm=353.448, loss=375.4084, lr=5.65e-04]
Epoch 51 TRAIN:  75%|███████▍  | 103/138 [00:23<00:07,  4.84it/s, epoch=51, grad_norm=214.603, loss=279.2715, lr=5.65e-04]
Epoch 51 TRAIN:  75%|███████▍  | 103/138 [00:23<00:07,  4.84it/s, epoch=51, grad_norm=244.462, loss=290.3476, lr=5.65e-04]
Epoch 51 TRAIN:  75%|███████▍  | 103/138 [00:24<00:07,  4.84it/s, epoch=51, grad_norm=156.324, loss=250.4973, lr=5.65e-04]
Epoch 51 TRAIN:  77%|███████▋  | 106/138 [00:24<00:06,  4.77it/s, epoch=51, grad_norm=156.324, loss=250.4973, lr=5.65e-04]
Epoch 51 TRAIN:  77%|███████▋  | 106/138 [00:24<00:06,  4.77it/s, epoch=51, grad_norm=118.681, loss=260.3715, lr=5.65e-04]
Epoch 51 TRAIN:  77%|███████▋  | 106/138 [00:24<00:06,  4.77it/s, epoch=51, grad_norm=168.583, loss=247.1029, lr=5.65e-04]
Epoch 51 TRAIN:  77%|███████▋  | 106/138 [00:24<00:06,  4.77it/s, epoch=51, grad_norm=154.021, loss=292.0597, lr=5.65e-04]
Epoch 51 TRAIN:  79%|███████▉  | 109/138 [00:24<00:06,  4.57it/s, epoch=51, grad_norm=154.021, loss=292.0597, lr=5.65e-04]
Epoch 51 TRAIN:  79%|███████▉  | 109/138 [00:25<00:06,  4.57it/s, epoch=51, grad_norm=152.801, loss=357.5910, lr=5.65e-04]
Epoch 51 TRAIN:  79%|███████▉  | 109/138 [00:25<00:06,  4.57it/s, epoch=51, grad_norm=123.857, loss=266.6712, lr=5.65e-04]
Epoch 51 TRAIN:  79%|███████▉  | 109/138 [00:25<00:06,  4.57it/s, epoch=51, grad_norm=207.004, loss=318.7020, lr=5.65e-04]
Epoch 51 TRAIN:  81%|████████  | 112/138 [00:25<00:05,  4.72it/s, epoch=51, grad_norm=207.004, loss=318.7020, lr=5.65e-04]
Epoch 51 TRAIN:  81%|████████  | 112/138 [00:25<00:05,  4.72it/s, epoch=51, grad_norm=171.072, loss=329.0153, lr=5.65e-04]
Epoch 51 TRAIN:  81%|████████  | 112/138 [00:25<00:05,  4.72it/s, epoch=51, grad_norm=164.113, loss=241.8354, lr=5.65e-04]
Epoch 51 TRAIN:  81%|████████  | 112/138 [00:26<00:05,  4.72it/s, epoch=51, grad_norm=198.647, loss=289.2574, lr=5.66e-04]
Epoch 51 TRAIN:  83%|████████▎ | 115/138 [00:26<00:05,  4.59it/s, epoch=51, grad_norm=198.647, loss=289.2574, lr=5.66e-04]
Epoch 51 TRAIN:  83%|████████▎ | 115/138 [00:26<00:05,  4.59it/s, epoch=51, grad_norm=139.103, loss=209.4589, lr=5.66e-04]
Epoch 51 TRAIN:  83%|████████▎ | 115/138 [00:26<00:05,  4.59it/s, epoch=51, grad_norm=103.131, loss=243.0368, lr=5.66e-04]
Epoch 51 TRAIN:  83%|████████▎ | 115/138 [00:26<00:05,  4.59it/s, epoch=51, grad_norm=364.422, loss=389.8835, lr=5.66e-04]
Epoch 51 TRAIN:  86%|████████▌ | 118/138 [00:26<00:04,  4.68it/s, epoch=51, grad_norm=364.422, loss=389.8835, lr=5.66e-04]
Epoch 51 TRAIN:  86%|████████▌ | 118/138 [00:26<00:04,  4.68it/s, epoch=51, grad_norm=235.064, loss=287.6105, lr=5.66e-04]
Epoch 51 TRAIN:  86%|████████▌ | 118/138 [00:27<00:04,  4.68it/s, epoch=51, grad_norm=230.501, loss=297.0960, lr=5.66e-04]
Epoch 51 TRAIN:  86%|████████▌ | 118/138 [00:27<00:04,  4.68it/s, epoch=51, grad_norm=96.296, loss=224.1764, lr=5.66e-04] 
Epoch 51 TRAIN:  88%|████████▊ | 121/138 [00:27<00:03,  4.51it/s, epoch=51, grad_norm=96.296, loss=224.1764, lr=5.66e-04]
Epoch 51 TRAIN:  88%|████████▊ | 121/138 [00:27<00:03,  4.51it/s, epoch=51, grad_norm=201.443, loss=307.1363, lr=5.66e-04]
Epoch 51 TRAIN:  88%|████████▊ | 121/138 [00:27<00:03,  4.51it/s, epoch=51, grad_norm=160.303, loss=302.5808, lr=5.66e-04]
Epoch 51 TRAIN:  88%|████████▊ | 121/138 [00:27<00:03,  4.51it/s, epoch=51, grad_norm=110.623, loss=315.1012, lr=5.66e-04]
Epoch 51 TRAIN:  90%|████████▉ | 124/138 [00:27<00:02,  4.75it/s, epoch=51, grad_norm=110.623, loss=315.1012, lr=5.66e-04]
Epoch 51 TRAIN:  90%|████████▉ | 124/138 [00:28<00:02,  4.75it/s, epoch=51, grad_norm=155.269, loss=368.4746, lr=5.66e-04]
Epoch 51 TRAIN:  90%|████████▉ | 124/138 [00:28<00:02,  4.75it/s, epoch=51, grad_norm=125.545, loss=293.4635, lr=5.66e-04]
Epoch 51 TRAIN:  90%|████████▉ | 124/138 [00:28<00:02,  4.75it/s, epoch=51, grad_norm=242.917, loss=395.9103, lr=5.66e-04]
Epoch 51 TRAIN:  92%|█████████▏| 127/138 [00:28<00:02,  4.45it/s, epoch=51, grad_norm=242.917, loss=395.9103, lr=5.66e-04]
Epoch 51 TRAIN:  92%|█████████▏| 127/138 [00:28<00:02,  4.45it/s, epoch=51, grad_norm=215.838, loss=255.1486, lr=5.67e-04]
Epoch 51 TRAIN:  92%|█████████▏| 127/138 [00:29<00:02,  4.45it/s, epoch=51, grad_norm=166.696, loss=338.0026, lr=5.67e-04]
Epoch 51 TRAIN:  92%|█████████▏| 127/138 [00:29<00:02,  4.45it/s, epoch=51, grad_norm=176.229, loss=354.7361, lr=5.67e-04]
Epoch 51 TRAIN:  94%|█████████▍| 130/138 [00:29<00:01,  4.69it/s, epoch=51, grad_norm=176.229, loss=354.7361, lr=5.67e-04]
Epoch 51 TRAIN:  94%|█████████▍| 130/138 [00:29<00:01,  4.69it/s, epoch=51, grad_norm=215.530, loss=300.9684, lr=5.67e-04]
Epoch 51 TRAIN:  94%|█████████▍| 130/138 [00:29<00:01,  4.69it/s, epoch=51, grad_norm=240.099, loss=281.7631, lr=5.67e-04]
Epoch 51 TRAIN:  94%|█████████▍| 130/138 [00:29<00:01,  4.69it/s, epoch=51, grad_norm=311.799, loss=335.9872, lr=5.67e-04]
Epoch 51 TRAIN:  96%|█████████▋| 133/138 [00:29<00:01,  4.89it/s, epoch=51, grad_norm=311.799, loss=335.9872, lr=5.67e-04]
Epoch 51 TRAIN:  96%|█████████▋| 133/138 [00:29<00:01,  4.89it/s, epoch=51, grad_norm=380.749, loss=411.2354, lr=5.67e-04]
Epoch 51 TRAIN:  96%|█████████▋| 133/138 [00:30<00:01,  4.89it/s, epoch=51, grad_norm=349.798, loss=346.7408, lr=5.67e-04]
Epoch 51 TRAIN:  96%|█████████▋| 133/138 [00:30<00:01,  4.89it/s, epoch=51, grad_norm=237.695, loss=330.6311, lr=5.67e-04]
Epoch 51 TRAIN:  99%|█████████▊| 136/138 [00:30<00:00,  4.80it/s, epoch=51, grad_norm=237.695, loss=330.6311, lr=5.67e-04]
Epoch 51 TRAIN:  99%|█████████▊| 136/138 [00:30<00:00,  4.80it/s, epoch=51, grad_norm=200.119, loss=274.0869, lr=5.67e-04]
Epoch 51 TRAIN:  99%|█████████▉| 137/138 [00:30<00:00,  4.47it/s, epoch=51, grad_norm=200.119, loss=274.0869, lr=5.67e-04]

CV:   0%|          | 0/6 [00:00<?, ?it/s]/home/rookie/Documents/batuka/projects/wenet/wenet/wenet/utils/train_utils.py:689: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  amp_autocast(enabled=dtype is not None,

CV:  17%|█▋        | 1/6 [00:00<00:03,  1.32it/s]
                                                 
2026-04-16 20:18:12,949 INFO Epoch 51 Step 7091 CV info lr 5.6728e-04 cv_loss 477.7992442737926 rank 0 acc 0.3279723674058914
2026-04-16 20:18:12,950 INFO [Rank 0] Checkpoint: save to checkpoint exp/conformer_mongolian/epoch_51.pt
2026-04-16 20:18:13,068 INFO Epoch 52 Step 7091 TRAIN info lr 5.6728e-04 rank 0
2026-04-16 20:18:13,082 INFO using accumulate grad, new batch size is 1 times larger than before

Epoch 52 TRAIN:   0%|          | 0/137 [00:00<?, ?it/s]
Epoch 52 TRAIN:   0%|          | 0/137 [00:04<?, ?it/s, epoch=52, grad_norm=508.952, loss=402.4758, lr=5.67e-04]
Epoch 52 TRAIN:   1%|          | 1/137 [00:04<09:13,  4.07s/it, epoch=52, grad_norm=508.952, loss=402.4758, lr=5.67e-04]
Epoch 52 TRAIN:   1%|          | 1/137 [00:04<09:13,  4.07s/it, epoch=52, grad_norm=715.135, loss=442.0535, lr=5.67e-04]
Epoch 52 TRAIN:   1%|          | 1/137 [00:04<09:13,  4.07s/it, epoch=52, grad_norm=324.231, loss=288.4632, lr=5.68e-04]
Epoch 52 TRAIN:   1%|          | 1/137 [00:04<09:13,  4.07s/it, epoch=52, grad_norm=592.607, loss=439.4197, lr=5.68e-04]
Epoch 52 TRAIN:   1%|          | 1/137 [00:04<09:13,  4.07s/it, epoch=52, grad_norm=388.789, loss=360.5229, lr=5.68e-04]
Epoch 52 TRAIN:   1%|          | 1/137 [00:04<09:13,  4.07s/it, epoch=52, grad_norm=253.440, loss=283.9040, lr=5.68e-04]
Epoch 52 TRAIN:   1%|          | 1/137 [00:04<09:13,  4.07s/it, epoch=52, grad_norm=302.460, loss=358.4815, lr=5.68e-04]
Epoch 52 TRAIN:   1%|          | 1/137 [00:04<09:13,  4.07s/it, epoch=52, grad_norm=240.311, loss=324.5785, lr=5.68e-04]
Epoch 52 TRAIN:   1%|          | 1/137 [00:04<09:13,  4.07s/it, epoch=52, grad_norm=149.184, loss=298.8644, lr=5.68e-04]
Epoch 52 TRAIN:   7%|▋         | 9/137 [00:04<00:51,  2.49it/s, epoch=52, grad_norm=149.184, loss=298.8644, lr=5.68e-04]
Epoch 52 TRAIN:   7%|▋         | 9/137 [00:04<00:51,  2.49it/s, epoch=52, grad_norm=97.697, loss=308.0256, lr=5.68e-04] 
Epoch 52 TRAIN:   7%|▋         | 9/137 [00:05<00:51,  2.49it/s, epoch=52, grad_norm=172.607, loss=240.3110, lr=5.68e-04]
Epoch 52 TRAIN:   7%|▋         | 9/137 [00:05<00:51,  2.49it/s, epoch=52, grad_norm=126.241, loss=313.9641, lr=5.68e-04]
Epoch 52 TRAIN:   7%|▋         | 9/137 [00:05<00:51,  2.49it/s, epoch=52, grad_norm=115.842, loss=344.8560, lr=5.68e-04]
Epoch 52 TRAIN:   9%|▉         | 13/137 [00:05<00:38,  3.26it/s, epoch=52, grad_norm=115.842, loss=344.8560, lr=5.68e-04]
Epoch 52 TRAIN:   9%|▉         | 13/137 [00:05<00:38,  3.26it/s, epoch=52, grad_norm=128.088, loss=292.4702, lr=5.68e-04]
Epoch 52 TRAIN:   9%|▉         | 13/137 [00:05<00:38,  3.26it/s, epoch=52, grad_norm=118.594, loss=277.7018, lr=5.68e-04]
Epoch 52 TRAIN:   9%|▉         | 13/137 [00:05<00:38,  3.26it/s, epoch=52, grad_norm=82.980, loss=309.1819, lr=5.69e-04] 
Epoch 52 TRAIN:   9%|▉         | 13/137 [00:05<00:38,  3.26it/s, epoch=52, grad_norm=73.813, loss=324.1256, lr=5.69e-04]
Epoch 52 TRAIN:  12%|█▏        | 17/137 [00:05<00:30,  3.95it/s, epoch=52, grad_norm=73.813, loss=324.1256, lr=5.69e-04]
Epoch 52 TRAIN:  12%|█▏        | 17/137 [00:06<00:30,  3.95it/s, epoch=52, grad_norm=79.203, loss=280.9316, lr=5.69e-04]
Epoch 52 TRAIN:  12%|█▏        | 17/137 [00:06<00:30,  3.95it/s, epoch=52, grad_norm=94.194, loss=297.8839, lr=5.69e-04]
Epoch 52 TRAIN:  12%|█▏        | 17/137 [00:06<00:30,  3.95it/s, epoch=52, grad_norm=108.827, loss=263.8710, lr=5.69e-04]
Epoch 52 TRAIN:  12%|█▏        | 17/137 [00:06<00:30,  3.95it/s, epoch=52, grad_norm=151.493, loss=325.8972, lr=5.69e-04]
Epoch 52 TRAIN:  15%|█▌        | 21/137 [00:06<00:25,  4.55it/s, epoch=52, grad_norm=151.493, loss=325.8972, lr=5.69e-04]
Epoch 52 TRAIN:  15%|█▌        | 21/137 [00:06<00:25,  4.55it/s, epoch=52, grad_norm=181.909, loss=340.0345, lr=5.69e-04]
Epoch 52 TRAIN:  15%|█▌        | 21/137 [00:06<00:25,  4.55it/s, epoch=52, grad_norm=149.767, loss=333.2191, lr=5.69e-04]
Epoch 52 TRAIN:  15%|█▌        | 21/137 [00:07<00:25,  4.55it/s, epoch=52, grad_norm=150.252, loss=337.0790, lr=5.69e-04]
Epoch 52 TRAIN:  15%|█▌        | 21/137 [00:07<00:25,  4.55it/s, epoch=52, grad_norm=125.215, loss=319.5395, lr=5.69e-04]
Epoch 52 TRAIN:  18%|█▊        | 25/137 [00:07<00:22,  4.87it/s, epoch=52, grad_norm=125.215, loss=319.5395, lr=5.69e-04]
Epoch 52 TRAIN:  18%|█▊        | 25/137 [00:07<00:22,  4.87it/s, epoch=52, grad_norm=208.767, loss=252.3283, lr=5.69e-04]
Epoch 52 TRAIN:  18%|█▊        | 25/137 [00:07<00:22,  4.87it/s, epoch=52, grad_norm=122.747, loss=311.5640, lr=5.69e-04]
Epoch 52 TRAIN:  18%|█▊        | 25/137 [00:07<00:22,  4.87it/s, epoch=52, grad_norm=163.675, loss=313.5497, lr=5.70e-04]
Epoch 52 TRAIN:  18%|█▊        | 25/137 [00:08<00:22,  4.87it/s, epoch=52, grad_norm=106.629, loss=339.2140, lr=5.70e-04]
Epoch 52 TRAIN:  21%|██        | 29/137 [00:08<00:21,  4.95it/s, epoch=52, grad_norm=106.629, loss=339.2140, lr=5.70e-04]
Epoch 52 TRAIN:  21%|██        | 29/137 [00:08<00:21,  4.95it/s, epoch=52, grad_norm=140.925, loss=292.6740, lr=5.70e-04]
Epoch 52 TRAIN:  21%|██        | 29/137 [00:08<00:21,  4.95it/s, epoch=52, grad_norm=109.738, loss=344.2513, lr=5.70e-04]
Epoch 52 TRAIN:  21%|██        | 29/137 [00:08<00:21,  4.95it/s, epoch=52, grad_norm=144.976, loss=278.1014, lr=5.70e-04]
Epoch 52 TRAIN:  23%|██▎       | 32/137 [00:08<00:20,  5.06it/s, epoch=52, grad_norm=144.976, loss=278.1014, lr=5.70e-04]
Epoch 52 TRAIN:  23%|██▎       | 32/137 [00:08<00:20,  5.06it/s, epoch=52, grad_norm=166.604, loss=244.1043, lr=5.70e-04]
Epoch 52 TRAIN:  23%|██▎       | 32/137 [00:09<00:20,  5.06it/s, epoch=52, grad_norm=176.104, loss=266.2537, lr=5.70e-04]
Epoch 52 TRAIN:  23%|██▎       | 32/137 [00:09<00:20,  5.06it/s, epoch=52, grad_norm=129.989, loss=278.9140, lr=5.70e-04]
Epoch 52 TRAIN:  26%|██▌       | 35/137 [00:09<00:20,  5.10it/s, epoch=52, grad_norm=129.989, loss=278.9140, lr=5.70e-04]
Epoch 52 TRAIN:  26%|██▌       | 35/137 [00:09<00:20,  5.10it/s, epoch=52, grad_norm=165.155, loss=234.8852, lr=5.70e-04]
Epoch 52 TRAIN:  26%|██▌       | 35/137 [00:09<00:20,  5.10it/s, epoch=52, grad_norm=123.893, loss=306.0001, lr=5.70e-04]
Epoch 52 TRAIN:  26%|██▌       | 35/137 [00:09<00:20,  5.10it/s, epoch=52, grad_norm=134.667, loss=293.0255, lr=5.70e-04]
Epoch 52 TRAIN:  28%|██▊       | 38/137 [00:09<00:19,  5.18it/s, epoch=52, grad_norm=134.667, loss=293.0255, lr=5.70e-04]
Epoch 52 TRAIN:  28%|██▊       | 38/137 [00:09<00:19,  5.18it/s, epoch=52, grad_norm=162.693, loss=303.6828, lr=5.70e-04]
Epoch 52 TRAIN:  28%|██▊       | 38/137 [00:10<00:19,  5.18it/s, epoch=52, grad_norm=144.115, loss=294.1281, lr=5.70e-04]
Epoch 52 TRAIN:  28%|██▊       | 38/137 [00:10<00:19,  5.18it/s, epoch=52, grad_norm=169.788, loss=319.0889, lr=5.71e-04]
Epoch 52 TRAIN:  30%|██▉       | 41/137 [00:10<00:18,  5.22it/s, epoch=52, grad_norm=169.788, loss=319.0889, lr=5.71e-04]
Epoch 52 TRAIN:  30%|██▉       | 41/137 [00:10<00:18,  5.22it/s, epoch=52, grad_norm=192.106, loss=327.8920, lr=5.71e-04]
Epoch 52 TRAIN:  30%|██▉       | 41/137 [00:10<00:18,  5.22it/s, epoch=52, grad_norm=121.773, loss=315.0267, lr=5.71e-04]
Epoch 52 TRAIN:  30%|██▉       | 41/137 [00:11<00:18,  5.22it/s, epoch=52, grad_norm=107.597, loss=298.4881, lr=5.71e-04]
Epoch 52 TRAIN:  32%|███▏      | 44/137 [00:11<00:18,  4.95it/s, epoch=52, grad_norm=107.597, loss=298.4881, lr=5.71e-04]
Epoch 52 TRAIN:  32%|███▏      | 44/137 [00:11<00:18,  4.95it/s, epoch=52, grad_norm=118.546, loss=307.2852, lr=5.71e-04]
Epoch 52 TRAIN:  32%|███▏      | 44/137 [00:11<00:18,  4.95it/s, epoch=52, grad_norm=99.432, loss=273.3412, lr=5.71e-04] 
Epoch 52 TRAIN:  32%|███▏      | 44/137 [00:11<00:18,  4.95it/s, epoch=52, grad_norm=167.963, loss=251.8326, lr=5.71e-04]
Epoch 52 TRAIN:  34%|███▍      | 47/137 [00:11<00:18,  4.99it/s, epoch=52, grad_norm=167.963, loss=251.8326, lr=5.71e-04]
Epoch 52 TRAIN:  34%|███▍      | 47/137 [00:11<00:18,  4.99it/s, epoch=52, grad_norm=160.851, loss=298.9145, lr=5.71e-04]
Epoch 52 TRAIN:  34%|███▍      | 47/137 [00:12<00:18,  4.99it/s, epoch=52, grad_norm=110.154, loss=270.8322, lr=5.71e-04]
Epoch 52 TRAIN:  34%|███▍      | 47/137 [00:12<00:18,  4.99it/s, epoch=52, grad_norm=117.840, loss=342.3801, lr=5.71e-04]
Epoch 52 TRAIN:  36%|███▋      | 50/137 [00:12<00:17,  4.99it/s, epoch=52, grad_norm=117.840, loss=342.3801, lr=5.71e-04]
Epoch 52 TRAIN:  36%|███▋      | 50/137 [00:12<00:17,  4.99it/s, epoch=52, grad_norm=120.658, loss=291.8324, lr=5.71e-04]
Epoch 52 TRAIN:  36%|███▋      | 50/137 [00:12<00:17,  4.99it/s, epoch=52, grad_norm=125.264, loss=329.8977, lr=5.71e-04]
Epoch 52 TRAIN:  36%|███▋      | 50/137 [00:12<00:17,  4.99it/s, epoch=52, grad_norm=107.602, loss=313.2653, lr=5.72e-04]
Epoch 52 TRAIN:  39%|███▊      | 53/137 [00:12<00:17,  4.93it/s, epoch=52, grad_norm=107.602, loss=313.2653, lr=5.72e-04]
Epoch 52 TRAIN:  39%|███▊      | 53/137 [00:13<00:17,  4.93it/s, epoch=52, grad_norm=136.381, loss=325.9957, lr=5.72e-04]
Epoch 52 TRAIN:  39%|███▊      | 53/137 [00:13<00:17,  4.93it/s, epoch=52, grad_norm=107.666, loss=286.7890, lr=5.72e-04]
Epoch 52 TRAIN:  39%|███▊      | 53/137 [00:13<00:17,  4.93it/s, epoch=52, grad_norm=93.064, loss=306.3142, lr=5.72e-04] 
Epoch 52 TRAIN:  41%|████      | 56/137 [00:13<00:16,  4.82it/s, epoch=52, grad_norm=93.064, loss=306.3142, lr=5.72e-04]
Epoch 52 TRAIN:  41%|████      | 56/137 [00:13<00:16,  4.82it/s, epoch=52, grad_norm=118.353, loss=273.7448, lr=5.72e-04]
Epoch 52 TRAIN:  41%|████      | 56/137 [00:13<00:16,  4.82it/s, epoch=52, grad_norm=109.797, loss=298.9272, lr=5.72e-04]
Epoch 52 TRAIN:  41%|████      | 56/137 [00:14<00:16,  4.82it/s, epoch=52, grad_norm=101.347, loss=311.9145, lr=5.72e-04]
Epoch 52 TRAIN:  43%|████▎     | 59/137 [00:14<00:15,  4.93it/s, epoch=52, grad_norm=101.347, loss=311.9145, lr=5.72e-04]
Epoch 52 TRAIN:  43%|████▎     | 59/137 [00:14<00:15,  4.93it/s, epoch=52, grad_norm=129.976, loss=305.2281, lr=5.72e-04]
Epoch 52 TRAIN:  43%|████▎     | 59/137 [00:14<00:15,  4.93it/s, epoch=52, grad_norm=112.872, loss=267.4974, lr=5.72e-04]
Epoch 52 TRAIN:  43%|████▎     | 59/137 [00:14<00:15,  4.93it/s, epoch=52, grad_norm=132.662, loss=289.9734, lr=5.72e-04]
Epoch 52 TRAIN:  45%|████▌     | 62/137 [00:14<00:14,  5.01it/s, epoch=52, grad_norm=132.662, loss=289.9734, lr=5.72e-04]
Epoch 52 TRAIN:  45%|████▌     | 62/137 [00:14<00:14,  5.01it/s, epoch=52, grad_norm=120.445, loss=283.6174, lr=5.72e-04]
Epoch 52 TRAIN:  45%|████▌     | 62/137 [00:15<00:14,  5.01it/s, epoch=52, grad_norm=140.344, loss=329.9055, lr=5.72e-04]
Epoch 52 TRAIN:  45%|████▌     | 62/137 [00:15<00:14,  5.01it/s, epoch=52, grad_norm=112.174, loss=272.3590, lr=5.72e-04]
Epoch 52 TRAIN:  47%|████▋     | 65/137 [00:15<00:14,  4.82it/s, epoch=52, grad_norm=112.174, loss=272.3590, lr=5.72e-04]
Epoch 52 TRAIN:  47%|████▋     | 65/137 [00:15<00:14,  4.82it/s, epoch=52, grad_norm=134.606, loss=286.2322, lr=5.73e-04]
Epoch 52 TRAIN:  47%|████▋     | 65/137 [00:15<00:14,  4.82it/s, epoch=52, grad_norm=131.175, loss=254.1712, lr=5.73e-04]
Epoch 52 TRAIN:  47%|████▋     | 65/137 [00:15<00:14,  4.82it/s, epoch=52, grad_norm=129.471, loss=271.2854, lr=5.73e-04]
Epoch 52 TRAIN:  50%|████▉     | 68/137 [00:15<00:14,  4.83it/s, epoch=52, grad_norm=129.471, loss=271.2854, lr=5.73e-04]
Epoch 52 TRAIN:  50%|████▉     | 68/137 [00:16<00:14,  4.83it/s, epoch=52, grad_norm=93.952, loss=280.5847, lr=5.73e-04] 
Epoch 52 TRAIN:  50%|████▉     | 68/137 [00:16<00:14,  4.83it/s, epoch=52, grad_norm=147.020, loss=284.6451, lr=5.73e-04]
Epoch 52 TRAIN:  50%|████▉     | 68/137 [00:16<00:14,  4.83it/s, epoch=52, grad_norm=152.100, loss=300.7153, lr=5.73e-04]
Epoch 52 TRAIN:  52%|█████▏    | 71/137 [00:16<00:13,  4.84it/s, epoch=52, grad_norm=152.100, loss=300.7153, lr=5.73e-04]
Epoch 52 TRAIN:  52%|█████▏    | 71/137 [00:16<00:13,  4.84it/s, epoch=52, grad_norm=158.915, loss=272.3627, lr=5.73e-04]
Epoch 52 TRAIN:  52%|█████▏    | 71/137 [00:16<00:13,  4.84it/s, epoch=52, grad_norm=128.763, loss=306.5718, lr=5.73e-04]
Epoch 52 TRAIN:  52%|█████▏    | 71/137 [00:17<00:13,  4.84it/s, epoch=52, grad_norm=254.413, loss=244.9761, lr=5.73e-04]
Epoch 52 TRAIN:  54%|█████▍    | 74/137 [00:17<00:12,  4.92it/s, epoch=52, grad_norm=254.413, loss=244.9761, lr=5.73e-04]
Epoch 52 TRAIN:  54%|█████▍    | 74/137 [00:17<00:12,  4.92it/s, epoch=52, grad_norm=227.980, loss=268.0106, lr=5.73e-04]
Epoch 52 TRAIN:  54%|█████▍    | 74/137 [00:17<00:12,  4.92it/s, epoch=52, grad_norm=213.344, loss=309.4147, lr=5.73e-04]
Epoch 52 TRAIN:  54%|█████▍    | 74/137 [00:17<00:12,  4.92it/s, epoch=52, grad_norm=222.345, loss=351.9634, lr=5.73e-04]
Epoch 52 TRAIN:  56%|█████▌    | 77/137 [00:17<00:12,  4.98it/s, epoch=52, grad_norm=222.345, loss=351.9634, lr=5.73e-04]
Epoch 52 TRAIN:  56%|█████▌    | 77/137 [00:17<00:12,  4.98it/s, epoch=52, grad_norm=263.882, loss=337.3071, lr=5.74e-04]
Epoch 52 TRAIN:  56%|█████▌    | 77/137 [00:18<00:12,  4.98it/s, epoch=52, grad_norm=166.291, loss=310.9859, lr=5.74e-04]
Epoch 52 TRAIN:  56%|█████▌    | 77/137 [00:18<00:12,  4.98it/s, epoch=52, grad_norm=125.504, loss=284.0470, lr=5.74e-04]
Epoch 52 TRAIN:  58%|█████▊    | 80/137 [00:18<00:11,  4.94it/s, epoch=52, grad_norm=125.504, loss=284.0470, lr=5.74e-04]
Epoch 52 TRAIN:  58%|█████▊    | 80/137 [00:18<00:11,  4.94it/s, epoch=52, grad_norm=176.932, loss=247.3887, lr=5.74e-04]
Epoch 52 TRAIN:  58%|█████▊    | 80/137 [00:18<00:11,  4.94it/s, epoch=52, grad_norm=141.460, loss=323.8322, lr=5.74e-04]
Epoch 52 TRAIN:  58%|█████▊    | 80/137 [00:19<00:11,  4.94it/s, epoch=52, grad_norm=178.839, loss=310.3679, lr=5.74e-04]
Epoch 52 TRAIN:  61%|██████    | 83/137 [00:19<00:11,  4.84it/s, epoch=52, grad_norm=178.839, loss=310.3679, lr=5.74e-04]
Epoch 52 TRAIN:  61%|██████    | 83/137 [00:19<00:11,  4.84it/s, epoch=52, grad_norm=245.568, loss=211.9746, lr=5.74e-04]
Epoch 52 TRAIN:  61%|██████    | 83/137 [00:19<00:11,  4.84it/s, epoch=52, grad_norm=259.111, loss=361.0183, lr=5.74e-04]
Epoch 52 TRAIN:  61%|██████    | 83/137 [00:19<00:11,  4.84it/s, epoch=52, grad_norm=209.624, loss=341.3272, lr=5.74e-04]
Epoch 52 TRAIN:  63%|██████▎   | 86/137 [00:19<00:10,  4.79it/s, epoch=52, grad_norm=209.624, loss=341.3272, lr=5.74e-04]
Epoch 52 TRAIN:  63%|██████▎   | 86/137 [00:19<00:10,  4.79it/s, epoch=52, grad_norm=216.134, loss=268.4667, lr=5.74e-04]
Epoch 52 TRAIN:  63%|██████▎   | 86/137 [00:19<00:10,  4.79it/s, epoch=52, grad_norm=279.555, loss=317.3422, lr=5.74e-04]
Epoch 52 TRAIN:  63%|██████▎   | 86/137 [00:20<00:10,  4.79it/s, epoch=52, grad_norm=209.141, loss=241.9369, lr=5.74e-04]
Epoch 52 TRAIN:  65%|██████▍   | 89/137 [00:20<00:09,  4.96it/s, epoch=52, grad_norm=209.141, loss=241.9369, lr=5.74e-04]
Epoch 52 TRAIN:  65%|██████▍   | 89/137 [00:20<00:09,  4.96it/s, epoch=52, grad_norm=302.700, loss=360.3250, lr=5.74e-04]
Epoch 52 TRAIN:  65%|██████▍   | 89/137 [00:20<00:09,  4.96it/s, epoch=52, grad_norm=250.616, loss=255.0519, lr=5.75e-04]
Epoch 52 TRAIN:  65%|██████▍   | 89/137 [00:20<00:09,  4.96it/s, epoch=52, grad_norm=137.710, loss=343.1412, lr=5.75e-04]
Epoch 52 TRAIN:  67%|██████▋   | 92/137 [00:20<00:09,  4.64it/s, epoch=52, grad_norm=137.710, loss=343.1412, lr=5.75e-04]
Epoch 52 TRAIN:  67%|██████▋   | 92/137 [00:21<00:09,  4.64it/s, epoch=52, grad_norm=118.780, loss=298.7500, lr=5.75e-04]
Epoch 52 TRAIN:  67%|██████▋   | 92/137 [00:21<00:09,  4.64it/s, epoch=52, grad_norm=104.952, loss=246.5300, lr=5.75e-04]
Epoch 52 TRAIN:  67%|██████▋   | 92/137 [00:21<00:09,  4.64it/s, epoch=52, grad_norm=178.213, loss=267.3857, lr=5.75e-04]
Epoch 52 TRAIN:  69%|██████▉   | 95/137 [00:21<00:08,  4.76it/s, epoch=52, grad_norm=178.213, loss=267.3857, lr=5.75e-04]
Epoch 52 TRAIN:  69%|██████▉   | 95/137 [00:21<00:08,  4.76it/s, epoch=52, grad_norm=243.633, loss=345.7996, lr=5.75e-04]
Epoch 52 TRAIN:  69%|██████▉   | 95/137 [00:21<00:08,  4.76it/s, epoch=52, grad_norm=130.483, loss=295.9684, lr=5.75e-04]
Epoch 52 TRAIN:  69%|██████▉   | 95/137 [00:22<00:08,  4.76it/s, epoch=52, grad_norm=180.736, loss=229.9120, lr=5.75e-04]
Epoch 52 TRAIN:  72%|███████▏  | 98/137 [00:22<00:08,  4.69it/s, epoch=52, grad_norm=180.736, loss=229.9120, lr=5.75e-04]
Epoch 52 TRAIN:  72%|███████▏  | 98/137 [00:22<00:08,  4.69it/s, epoch=52, grad_norm=212.705, loss=312.4902, lr=5.75e-04]
Epoch 52 TRAIN:  72%|███████▏  | 98/137 [00:22<00:08,  4.69it/s, epoch=52, grad_norm=194.464, loss=188.6979, lr=5.75e-04]2026-04-16 20:18:35,728 DEBUG TRAIN | steps/sec 4.355| Batch 52/100 loss 188.697937 loss_att 153.245880 loss_ctc 271.419373 th_accuracy 0.363309 lr 5.7528e-04 grad_norm 194.464294 rank 0

Epoch 52 TRAIN:  72%|███████▏  | 98/137 [00:22<00:08,  4.69it/s, epoch=52, grad_norm=171.432, loss=314.2067, lr=5.75e-04]
Epoch 52 TRAIN:  74%|███████▎  | 101/137 [00:22<00:07,  4.61it/s, epoch=52, grad_norm=171.432, loss=314.2067, lr=5.75e-04]
Epoch 52 TRAIN:  74%|███████▎  | 101/137 [00:23<00:07,  4.61it/s, epoch=52, grad_norm=194.716, loss=278.1453, lr=5.75e-04]
Epoch 52 TRAIN:  74%|███████▎  | 101/137 [00:23<00:07,  4.61it/s, epoch=52, grad_norm=128.496, loss=174.8208, lr=5.76e-04]
Epoch 52 TRAIN:  74%|███████▎  | 101/137 [00:23<00:07,  4.61it/s, epoch=52, grad_norm=301.039, loss=320.7344, lr=5.76e-04]
Epoch 52 TRAIN:  76%|███████▌  | 104/137 [00:23<00:06,  4.72it/s, epoch=52, grad_norm=301.039, loss=320.7344, lr=5.76e-04]
Epoch 52 TRAIN:  76%|███████▌  | 104/137 [00:23<00:06,  4.72it/s, epoch=52, grad_norm=163.489, loss=276.6113, lr=5.76e-04]
Epoch 52 TRAIN:  76%|███████▌  | 104/137 [00:23<00:06,  4.72it/s, epoch=52, grad_norm=148.758, loss=272.0197, lr=5.76e-04]
Epoch 52 TRAIN:  76%|███████▌  | 104/137 [00:23<00:06,  4.72it/s, epoch=52, grad_norm=255.116, loss=352.5504, lr=5.76e-04]
Epoch 52 TRAIN:  78%|███████▊  | 107/137 [00:23<00:05,  5.03it/s, epoch=52, grad_norm=255.116, loss=352.5504, lr=5.76e-04]
Epoch 52 TRAIN:  78%|███████▊  | 107/137 [00:24<00:05,  5.03it/s, epoch=52, grad_norm=126.074, loss=279.5317, lr=5.76e-04]
Epoch 52 TRAIN:  78%|███████▊  | 107/137 [00:24<00:05,  5.03it/s, epoch=52, grad_norm=97.738, loss=323.0842, lr=5.76e-04] 
Epoch 52 TRAIN:  78%|███████▊  | 107/137 [00:24<00:05,  5.03it/s, epoch=52, grad_norm=144.134, loss=282.8057, lr=5.76e-04]
Epoch 52 TRAIN:  80%|████████  | 110/137 [00:24<00:05,  4.96it/s, epoch=52, grad_norm=144.134, loss=282.8057, lr=5.76e-04]
Epoch 52 TRAIN:  80%|████████  | 110/137 [00:24<00:05,  4.96it/s, epoch=52, grad_norm=213.947, loss=263.5498, lr=5.76e-04]
Epoch 52 TRAIN:  80%|████████  | 110/137 [00:25<00:05,  4.96it/s, epoch=52, grad_norm=148.440, loss=309.0193, lr=5.76e-04]
Epoch 52 TRAIN:  80%|████████  | 110/137 [00:25<00:05,  4.96it/s, epoch=52, grad_norm=183.353, loss=325.5370, lr=5.76e-04]
Epoch 52 TRAIN:  82%|████████▏ | 113/137 [00:25<00:05,  4.43it/s, epoch=52, grad_norm=183.353, loss=325.5370, lr=5.76e-04]
Epoch 52 TRAIN:  82%|████████▏ | 113/137 [00:25<00:05,  4.43it/s, epoch=52, grad_norm=262.970, loss=310.3757, lr=5.76e-04]
Epoch 52 TRAIN:  82%|████████▏ | 113/137 [00:25<00:05,  4.43it/s, epoch=52, grad_norm=189.033, loss=294.9173, lr=5.76e-04]
Epoch 52 TRAIN:  82%|████████▏ | 113/137 [00:25<00:05,  4.43it/s, epoch=52, grad_norm=199.966, loss=298.3373, lr=5.77e-04]
Epoch 52 TRAIN:  82%|████████▏ | 113/137 [00:26<00:05,  4.43it/s, epoch=52, grad_norm=112.851, loss=260.0993, lr=5.77e-04]
Epoch 52 TRAIN:  85%|████████▌ | 117/137 [00:26<00:04,  4.89it/s, epoch=52, grad_norm=112.851, loss=260.0993, lr=5.77e-04]
Epoch 52 TRAIN:  85%|████████▌ | 117/137 [00:26<00:04,  4.89it/s, epoch=52, grad_norm=431.228, loss=382.3571, lr=5.77e-04]
Epoch 52 TRAIN:  85%|████████▌ | 117/137 [00:26<00:04,  4.89it/s, epoch=52, grad_norm=293.507, loss=336.5196, lr=5.77e-04]
Epoch 52 TRAIN:  85%|████████▌ | 117/137 [00:26<00:04,  4.89it/s, epoch=52, grad_norm=192.582, loss=291.4698, lr=5.77e-04]
Epoch 52 TRAIN:  88%|████████▊ | 120/137 [00:26<00:03,  4.95it/s, epoch=52, grad_norm=192.582, loss=291.4698, lr=5.77e-04]
Epoch 52 TRAIN:  88%|████████▊ | 120/137 [00:26<00:03,  4.95it/s, epoch=52, grad_norm=207.303, loss=330.9180, lr=5.77e-04]
Epoch 52 TRAIN:  88%|████████▊ | 120/137 [00:27<00:03,  4.95it/s, epoch=52, grad_norm=221.354, loss=354.1293, lr=5.77e-04]
Epoch 52 TRAIN:  88%|████████▊ | 120/137 [00:27<00:03,  4.95it/s, epoch=52, grad_norm=154.753, loss=347.2464, lr=5.77e-04]
Epoch 52 TRAIN:  90%|████████▉ | 123/137 [00:27<00:02,  4.75it/s, epoch=52, grad_norm=154.753, loss=347.2464, lr=5.77e-04]
Epoch 52 TRAIN:  90%|████████▉ | 123/137 [00:27<00:02,  4.75it/s, epoch=52, grad_norm=107.436, loss=265.3964, lr=5.77e-04]
Epoch 52 TRAIN:  90%|████████▉ | 123/137 [00:27<00:02,  4.75it/s, epoch=52, grad_norm=202.351, loss=325.1175, lr=5.77e-04]
Epoch 52 TRAIN:  90%|████████▉ | 123/137 [00:27<00:02,  4.75it/s, epoch=52, grad_norm=305.721, loss=278.8845, lr=5.77e-04]
Epoch 52 TRAIN:  92%|█████████▏| 126/137 [00:27<00:02,  5.05it/s, epoch=52, grad_norm=305.721, loss=278.8845, lr=5.77e-04]
Epoch 52 TRAIN:  92%|█████████▏| 126/137 [00:28<00:02,  5.05it/s, epoch=52, grad_norm=154.924, loss=300.3167, lr=5.77e-04]
Epoch 52 TRAIN:  92%|█████████▏| 126/137 [00:28<00:02,  5.05it/s, epoch=52, grad_norm=144.948, loss=318.5389, lr=5.78e-04]
Epoch 52 TRAIN:  92%|█████████▏| 126/137 [00:28<00:02,  5.05it/s, epoch=52, grad_norm=121.379, loss=248.4631, lr=5.78e-04]
Epoch 52 TRAIN:  94%|█████████▍| 129/137 [00:28<00:01,  4.68it/s, epoch=52, grad_norm=121.379, loss=248.4631, lr=5.78e-04]
Epoch 52 TRAIN:  94%|█████████▍| 129/137 [00:28<00:01,  4.68it/s, epoch=52, grad_norm=303.625, loss=265.8768, lr=5.78e-04]
Epoch 52 TRAIN:  94%|█████████▍| 129/137 [00:29<00:01,  4.68it/s, epoch=52, grad_norm=177.229, loss=376.2220, lr=5.78e-04]
Epoch 52 TRAIN:  94%|█████████▍| 129/137 [00:29<00:01,  4.68it/s, epoch=52, grad_norm=113.760, loss=278.1686, lr=5.78e-04]
Epoch 52 TRAIN:  96%|█████████▋| 132/137 [00:29<00:01,  4.82it/s, epoch=52, grad_norm=113.760, loss=278.1686, lr=5.78e-04]
Epoch 52 TRAIN:  96%|█████████▋| 132/137 [00:29<00:01,  4.82it/s, epoch=52, grad_norm=130.421, loss=352.2777, lr=5.78e-04]
Epoch 52 TRAIN:  96%|█████████▋| 132/137 [00:29<00:01,  4.82it/s, epoch=52, grad_norm=194.961, loss=331.9479, lr=5.78e-04]
Epoch 52 TRAIN:  96%|█████████▋| 132/137 [00:29<00:01,  4.82it/s, epoch=52, grad_norm=129.224, loss=249.5029, lr=5.78e-04]
Epoch 52 TRAIN:  99%|█████████▊| 135/137 [00:29<00:00,  4.89it/s, epoch=52, grad_norm=129.224, loss=249.5029, lr=5.78e-04]
Epoch 52 TRAIN:  99%|█████████▊| 135/137 [00:30<00:00,  4.89it/s, epoch=52, grad_norm=196.838, loss=321.5746, lr=5.78e-04]
Epoch 52 TRAIN:  99%|█████████▊| 135/137 [00:30<00:00,  4.89it/s, epoch=52, grad_norm=158.159, loss=313.3600, lr=5.78e-04]
Epoch 52 TRAIN: 100%|██████████| 137/137 [00:30<00:00,  4.52it/s, epoch=52, grad_norm=158.159, loss=313.3600, lr=5.78e-04]

CV:   0%|          | 0/6 [00:00<?, ?it/s]/home/rookie/Documents/batuka/projects/wenet/wenet/wenet/utils/train_utils.py:689: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  amp_autocast(enabled=dtype is not None,

CV:  17%|█▋        | 1/6 [00:00<00:03,  1.33it/s]
                                                 
2026-04-16 20:18:44,249 INFO Epoch 52 Step 7228 CV info lr 5.7824e-04 cv_loss 458.64304421164775 rank 0 acc 0.331578070918719
2026-04-16 20:18:44,250 INFO [Rank 0] Checkpoint: save to checkpoint exp/conformer_mongolian/epoch_52.pt
2026-04-16 20:18:44,365 INFO Epoch 53 Step 7228 TRAIN info lr 5.7824e-04 rank 0
2026-04-16 20:18:44,378 INFO using accumulate grad, new batch size is 1 times larger than before

Epoch 53 TRAIN:   0%|          | 0/137 [00:00<?, ?it/s]
Epoch 53 TRAIN:   0%|          | 0/137 [00:03<?, ?it/s, epoch=53, grad_norm=474.520, loss=356.5336, lr=5.78e-04]
Epoch 53 TRAIN:   1%|          | 1/137 [00:03<09:02,  3.99s/it, epoch=53, grad_norm=474.520, loss=356.5336, lr=5.78e-04]
Epoch 53 TRAIN:   1%|          | 1/137 [00:04<09:02,  3.99s/it, epoch=53, grad_norm=633.704, loss=397.7574, lr=5.78e-04]
Epoch 53 TRAIN:   1%|          | 1/137 [00:04<09:02,  3.99s/it, epoch=53, grad_norm=352.683, loss=310.4314, lr=5.78e-04]
Epoch 53 TRAIN:   1%|          | 1/137 [00:04<09:02,  3.99s/it, epoch=53, grad_norm=256.455, loss=272.2288, lr=5.79e-04]
Epoch 53 TRAIN:   1%|          | 1/137 [00:04<09:02,  3.99s/it, epoch=53, grad_norm=374.627, loss=376.5298, lr=5.79e-04]
Epoch 53 TRAIN:   1%|          | 1/137 [00:04<09:02,  3.99s/it, epoch=53, grad_norm=357.600, loss=357.2789, lr=5.79e-04]
Epoch 53 TRAIN:   1%|          | 1/137 [00:04<09:02,  3.99s/it, epoch=53, grad_norm=280.617, loss=362.6828, lr=5.79e-04]
Epoch 53 TRAIN:   1%|          | 1/137 [00:04<09:02,  3.99s/it, epoch=53, grad_norm=309.843, loss=367.6652, lr=5.79e-04]
Epoch 53 TRAIN:   6%|▌         | 8/137 [00:04<00:55,  2.31it/s, epoch=53, grad_norm=309.843, loss=367.6652, lr=5.79e-04]
Epoch 53 TRAIN:   6%|▌         | 8/137 [00:04<00:55,  2.31it/s, epoch=53, grad_norm=185.465, loss=334.6257, lr=5.79e-04]
Epoch 53 TRAIN:   6%|▌         | 8/137 [00:04<00:55,  2.31it/s, epoch=53, grad_norm=108.099, loss=333.8634, lr=5.79e-04]
Epoch 53 TRAIN:   6%|▌         | 8/137 [00:05<00:55,  2.31it/s, epoch=53, grad_norm=169.189, loss=291.5056, lr=5.79e-04]
Epoch 53 TRAIN:   6%|▌         | 8/137 [00:05<00:55,  2.31it/s, epoch=53, grad_norm=162.591, loss=318.3979, lr=5.79e-04]
Epoch 53 TRAIN:   9%|▉         | 12/137 [00:05<00:39,  3.16it/s, epoch=53, grad_norm=162.591, loss=318.3979, lr=5.79e-04]
Epoch 53 TRAIN:   9%|▉         | 12/137 [00:05<00:39,  3.16it/s, epoch=53, grad_norm=193.177, loss=278.2069, lr=5.79e-04]
Epoch 53 TRAIN:   9%|▉         | 12/137 [00:05<00:39,  3.16it/s, epoch=53, grad_norm=165.008, loss=273.0006, lr=5.79e-04]
Epoch 53 TRAIN:   9%|▉         | 12/137 [00:05<00:39,  3.16it/s, epoch=53, grad_norm=113.899, loss=320.6048, lr=5.79e-04]
Epoch 53 TRAIN:   9%|▉         | 12/137 [00:05<00:39,  3.16it/s, epoch=53, grad_norm=117.630, loss=313.1189, lr=5.80e-04]
Epoch 53 TRAIN:  12%|█▏        | 16/137 [00:05<00:31,  3.89it/s, epoch=53, grad_norm=117.630, loss=313.1189, lr=5.80e-04]
Epoch 53 TRAIN:  12%|█▏        | 16/137 [00:05<00:31,  3.89it/s, epoch=53, grad_norm=96.040, loss=304.0473, lr=5.80e-04] 
Epoch 53 TRAIN:  12%|█▏        | 16/137 [00:06<00:31,  3.89it/s, epoch=53, grad_norm=162.484, loss=253.6418, lr=5.80e-04]
Epoch 53 TRAIN:  12%|█▏        | 16/137 [00:06<00:31,  3.89it/s, epoch=53, grad_norm=174.967, loss=274.8170, lr=5.80e-04]
Epoch 53 TRAIN:  12%|█▏        | 16/137 [00:06<00:31,  3.89it/s, epoch=53, grad_norm=106.185, loss=297.9899, lr=5.80e-04]
Epoch 53 TRAIN:  15%|█▍        | 20/137 [00:06<00:26,  4.45it/s, epoch=53, grad_norm=106.185, loss=297.9899, lr=5.80e-04]
Epoch 53 TRAIN:  15%|█▍        | 20/137 [00:06<00:26,  4.45it/s, epoch=53, grad_norm=87.361, loss=270.8122, lr=5.80e-04] 
Epoch 53 TRAIN:  15%|█▍        | 20/137 [00:06<00:26,  4.45it/s, epoch=53, grad_norm=107.043, loss=330.7414, lr=5.80e-04]
Epoch 53 TRAIN:  15%|█▍        | 20/137 [00:07<00:26,  4.45it/s, epoch=53, grad_norm=78.363, loss=294.2579, lr=5.80e-04] 
Epoch 53 TRAIN:  15%|█▍        | 20/137 [00:07<00:26,  4.45it/s, epoch=53, grad_norm=95.388, loss=290.3450, lr=5.80e-04]
Epoch 53 TRAIN:  18%|█▊        | 24/137 [00:07<00:23,  4.73it/s, epoch=53, grad_norm=95.388, loss=290.3450, lr=5.80e-04]
Epoch 53 TRAIN:  18%|█▊        | 24/137 [00:07<00:23,  4.73it/s, epoch=53, grad_norm=102.881, loss=296.9919, lr=5.80e-04]
Epoch 53 TRAIN:  18%|█▊        | 24/137 [00:07<00:23,  4.73it/s, epoch=53, grad_norm=91.101, loss=295.1303, lr=5.80e-04] 
Epoch 53 TRAIN:  18%|█▊        | 24/137 [00:07<00:23,  4.73it/s, epoch=53, grad_norm=112.396, loss=290.9348, lr=5.80e-04]
Epoch 53 TRAIN:  18%|█▊        | 24/137 [00:07<00:23,  4.73it/s, epoch=53, grad_norm=147.486, loss=341.1293, lr=5.80e-04]
Epoch 53 TRAIN:  20%|██        | 28/137 [00:07<00:22,  4.91it/s, epoch=53, grad_norm=147.486, loss=341.1293, lr=5.80e-04]
Epoch 53 TRAIN:  20%|██        | 28/137 [00:08<00:22,  4.91it/s, epoch=53, grad_norm=139.976, loss=378.7343, lr=5.81e-04]
Epoch 53 TRAIN:  20%|██        | 28/137 [00:08<00:22,  4.91it/s, epoch=53, grad_norm=194.776, loss=282.0179, lr=5.81e-04]
Epoch 53 TRAIN:  20%|██        | 28/137 [00:08<00:22,  4.91it/s, epoch=53, grad_norm=168.420, loss=309.5918, lr=5.81e-04]
Epoch 53 TRAIN:  23%|██▎       | 31/137 [00:08<00:21,  5.00it/s, epoch=53, grad_norm=168.420, loss=309.5918, lr=5.81e-04]
Epoch 53 TRAIN:  23%|██▎       | 31/137 [00:08<00:21,  5.00it/s, epoch=53, grad_norm=123.096, loss=363.2663, lr=5.81e-04]
Epoch 53 TRAIN:  23%|██▎       | 31/137 [00:08<00:21,  5.00it/s, epoch=53, grad_norm=240.785, loss=219.9096, lr=5.81e-04]
Epoch 53 TRAIN:  23%|██▎       | 31/137 [00:09<00:21,  5.00it/s, epoch=53, grad_norm=130.055, loss=277.8904, lr=5.81e-04]
Epoch 53 TRAIN:  25%|██▍       | 34/137 [00:09<00:20,  5.13it/s, epoch=53, grad_norm=130.055, loss=277.8904, lr=5.81e-04]
Epoch 53 TRAIN:  25%|██▍       | 34/137 [00:09<00:20,  5.13it/s, epoch=53, grad_norm=104.838, loss=318.0139, lr=5.81e-04]
Epoch 53 TRAIN:  25%|██▍       | 34/137 [00:09<00:20,  5.13it/s, epoch=53, grad_norm=111.309, loss=312.5289, lr=5.81e-04]
Epoch 53 TRAIN:  25%|██▍       | 34/137 [00:09<00:20,  5.13it/s, epoch=53, grad_norm=108.724, loss=326.1523, lr=5.81e-04]
Epoch 53 TRAIN:  27%|██▋       | 37/137 [00:09<00:19,  5.03it/s, epoch=53, grad_norm=108.724, loss=326.1523, lr=5.81e-04]
Epoch 53 TRAIN:  27%|██▋       | 37/137 [00:09<00:19,  5.03it/s, epoch=53, grad_norm=77.113, loss=300.1644, lr=5.81e-04] 
Epoch 53 TRAIN:  27%|██▋       | 37/137 [00:10<00:19,  5.03it/s, epoch=53, grad_norm=77.000, loss=299.3447, lr=5.81e-04]
Epoch 53 TRAIN:  27%|██▋       | 37/137 [00:10<00:19,  5.03it/s, epoch=53, grad_norm=72.733, loss=303.6402, lr=5.81e-04]
Epoch 53 TRAIN:  29%|██▉       | 40/137 [00:10<00:19,  4.96it/s, epoch=53, grad_norm=72.733, loss=303.6402, lr=5.81e-04]
Epoch 53 TRAIN:  29%|██▉       | 40/137 [00:10<00:19,  4.96it/s, epoch=53, grad_norm=122.092, loss=284.3574, lr=5.82e-04]
Epoch 53 TRAIN:  29%|██▉       | 40/137 [00:10<00:19,  4.96it/s, epoch=53, grad_norm=103.969, loss=284.6645, lr=5.82e-04]
Epoch 53 TRAIN:  29%|██▉       | 40/137 [00:10<00:19,  4.96it/s, epoch=53, grad_norm=97.609, loss=327.2610, lr=5.82e-04] 
Epoch 53 TRAIN:  31%|███▏      | 43/137 [00:10<00:18,  4.96it/s, epoch=53, grad_norm=97.609, loss=327.2610, lr=5.82e-04]
Epoch 53 TRAIN:  31%|███▏      | 43/137 [00:11<00:18,  4.96it/s, epoch=53, grad_norm=83.632, loss=287.5785, lr=5.82e-04]
Epoch 53 TRAIN:  31%|███▏      | 43/137 [00:11<00:18,  4.96it/s, epoch=53, grad_norm=118.370, loss=307.8756, lr=5.82e-04]
Epoch 53 TRAIN:  31%|███▏      | 43/137 [00:11<00:18,  4.96it/s, epoch=53, grad_norm=104.982, loss=267.9843, lr=5.82e-04]
Epoch 53 TRAIN:  34%|███▎      | 46/137 [00:11<00:18,  4.96it/s, epoch=53, grad_norm=104.982, loss=267.9843, lr=5.82e-04]
Epoch 53 TRAIN:  34%|███▎      | 46/137 [00:11<00:18,  4.96it/s, epoch=53, grad_norm=88.781, loss=273.2271, lr=5.82e-04] 
Epoch 53 TRAIN:  34%|███▎      | 46/137 [00:11<00:18,  4.96it/s, epoch=53, grad_norm=94.784, loss=305.4970, lr=5.82e-04]
Epoch 53 TRAIN:  34%|███▎      | 46/137 [00:12<00:18,  4.96it/s, epoch=53, grad_norm=90.724, loss=291.8776, lr=5.82e-04]
Epoch 53 TRAIN:  36%|███▌      | 49/137 [00:12<00:17,  4.90it/s, epoch=53, grad_norm=90.724, loss=291.8776, lr=5.82e-04]
Epoch 53 TRAIN:  36%|███▌      | 49/137 [00:12<00:17,  4.90it/s, epoch=53, grad_norm=147.455, loss=323.9784, lr=5.82e-04]
Epoch 53 TRAIN:  36%|███▌      | 49/137 [00:12<00:17,  4.90it/s, epoch=53, grad_norm=94.278, loss=354.9856, lr=5.82e-04] 
Epoch 53 TRAIN:  36%|███▌      | 49/137 [00:12<00:17,  4.90it/s, epoch=53, grad_norm=137.912, loss=303.2928, lr=5.82e-04]
Epoch 53 TRAIN:  38%|███▊      | 52/137 [00:12<00:17,  4.88it/s, epoch=53, grad_norm=137.912, loss=303.2928, lr=5.82e-04]
Epoch 53 TRAIN:  38%|███▊      | 52/137 [00:12<00:17,  4.88it/s, epoch=53, grad_norm=92.447, loss=320.6967, lr=5.82e-04] 
Epoch 53 TRAIN:  38%|███▊      | 52/137 [00:13<00:17,  4.88it/s, epoch=53, grad_norm=128.066, loss=281.5898, lr=5.83e-04]
Epoch 53 TRAIN:  38%|███▊      | 52/137 [00:13<00:17,  4.88it/s, epoch=53, grad_norm=116.830, loss=288.2411, lr=5.83e-04]
Epoch 53 TRAIN:  40%|████      | 55/137 [00:13<00:17,  4.82it/s, epoch=53, grad_norm=116.830, loss=288.2411, lr=5.83e-04]
Epoch 53 TRAIN:  40%|████      | 55/137 [00:13<00:17,  4.82it/s, epoch=53, grad_norm=102.207, loss=296.5707, lr=5.83e-04]
Epoch 53 TRAIN:  40%|████      | 55/137 [00:13<00:17,  4.82it/s, epoch=53, grad_norm=116.437, loss=261.5252, lr=5.83e-04]
Epoch 53 TRAIN:  40%|████      | 55/137 [00:13<00:17,  4.82it/s, epoch=53, grad_norm=75.512, loss=278.8349, lr=5.83e-04] 
Epoch 53 TRAIN:  42%|████▏     | 58/137 [00:13<00:15,  4.97it/s, epoch=53, grad_norm=75.512, loss=278.8349, lr=5.83e-04]
Epoch 53 TRAIN:  42%|████▏     | 58/137 [00:14<00:15,  4.97it/s, epoch=53, grad_norm=180.560, loss=370.6973, lr=5.83e-04]
Epoch 53 TRAIN:  42%|████▏     | 58/137 [00:14<00:15,  4.97it/s, epoch=53, grad_norm=74.332, loss=277.4634, lr=5.83e-04] 
Epoch 53 TRAIN:  42%|████▏     | 58/137 [00:14<00:15,  4.97it/s, epoch=53, grad_norm=177.543, loss=347.9673, lr=5.83e-04]
Epoch 53 TRAIN:  45%|████▍     | 61/137 [00:14<00:16,  4.74it/s, epoch=53, grad_norm=177.543, loss=347.9673, lr=5.83e-04]
Epoch 53 TRAIN:  45%|████▍     | 61/137 [00:14<00:16,  4.74it/s, epoch=53, grad_norm=144.237, loss=309.3734, lr=5.83e-04]
Epoch 53 TRAIN:  45%|████▍     | 61/137 [00:15<00:16,  4.74it/s, epoch=53, grad_norm=136.107, loss=318.9960, lr=5.83e-04]
Epoch 53 TRAIN:  45%|████▍     | 61/137 [00:15<00:16,  4.74it/s, epoch=53, grad_norm=189.467, loss=258.4584, lr=5.83e-04]
Epoch 53 TRAIN:  47%|████▋     | 64/137 [00:15<00:15,  4.81it/s, epoch=53, grad_norm=189.467, loss=258.4584, lr=5.83e-04]
Epoch 53 TRAIN:  47%|████▋     | 64/137 [00:15<00:15,  4.81it/s, epoch=53, grad_norm=147.999, loss=262.6701, lr=5.83e-04]
Epoch 53 TRAIN:  47%|████▋     | 64/137 [00:15<00:15,  4.81it/s, epoch=53, grad_norm=123.599, loss=302.7242, lr=5.84e-04]
Epoch 53 TRAIN:  47%|████▋     | 64/137 [00:15<00:15,  4.81it/s, epoch=53, grad_norm=178.848, loss=309.2002, lr=5.84e-04]
Epoch 53 TRAIN:  49%|████▉     | 67/137 [00:15<00:14,  4.81it/s, epoch=53, grad_norm=178.848, loss=309.2002, lr=5.84e-04]
Epoch 53 TRAIN:  49%|████▉     | 67/137 [00:16<00:14,  4.81it/s, epoch=53, grad_norm=111.092, loss=305.9630, lr=5.84e-04]
Epoch 53 TRAIN:  49%|████▉     | 67/137 [00:16<00:14,  4.81it/s, epoch=53, grad_norm=194.007, loss=278.4836, lr=5.84e-04]
Epoch 53 TRAIN:  49%|████▉     | 67/137 [00:16<00:14,  4.81it/s, epoch=53, grad_norm=126.757, loss=256.8905, lr=5.84e-04]
Epoch 53 TRAIN:  51%|█████     | 70/137 [00:16<00:13,  4.81it/s, epoch=53, grad_norm=126.757, loss=256.8905, lr=5.84e-04]
Epoch 53 TRAIN:  51%|█████     | 70/137 [00:16<00:13,  4.81it/s, epoch=53, grad_norm=124.784, loss=257.8484, lr=5.84e-04]
Epoch 53 TRAIN:  51%|█████     | 70/137 [00:16<00:13,  4.81it/s, epoch=53, grad_norm=111.351, loss=262.3360, lr=5.84e-04]
Epoch 53 TRAIN:  51%|█████     | 70/137 [00:17<00:13,  4.81it/s, epoch=53, grad_norm=137.868, loss=294.5587, lr=5.84e-04]
Epoch 53 TRAIN:  53%|█████▎    | 73/137 [00:17<00:13,  4.70it/s, epoch=53, grad_norm=137.868, loss=294.5587, lr=5.84e-04]
Epoch 53 TRAIN:  53%|█████▎    | 73/137 [00:17<00:13,  4.70it/s, epoch=53, grad_norm=207.102, loss=328.8753, lr=5.84e-04]
Epoch 53 TRAIN:  53%|█████▎    | 73/137 [00:17<00:13,  4.70it/s, epoch=53, grad_norm=186.696, loss=342.4616, lr=5.84e-04]
Epoch 53 TRAIN:  53%|█████▎    | 73/137 [00:17<00:13,  4.70it/s, epoch=53, grad_norm=116.255, loss=298.6673, lr=5.84e-04]
Epoch 53 TRAIN:  55%|█████▌    | 76/137 [00:17<00:13,  4.69it/s, epoch=53, grad_norm=116.255, loss=298.6673, lr=5.84e-04]
Epoch 53 TRAIN:  55%|█████▌    | 76/137 [00:18<00:13,  4.69it/s, epoch=53, grad_norm=79.793, loss=272.2793, lr=5.84e-04] 
Epoch 53 TRAIN:  55%|█████▌    | 76/137 [00:18<00:13,  4.69it/s, epoch=53, grad_norm=146.367, loss=330.5670, lr=5.84e-04]
Epoch 53 TRAIN:  55%|█████▌    | 76/137 [00:18<00:13,  4.69it/s, epoch=53, grad_norm=122.121, loss=303.0354, lr=5.85e-04]
Epoch 53 TRAIN:  58%|█████▊    | 79/137 [00:18<00:12,  4.60it/s, epoch=53, grad_norm=122.121, loss=303.0354, lr=5.85e-04]
Epoch 53 TRAIN:  58%|█████▊    | 79/137 [00:18<00:12,  4.60it/s, epoch=53, grad_norm=283.318, loss=208.2950, lr=5.85e-04]
Epoch 53 TRAIN:  58%|█████▊    | 79/137 [00:19<00:12,  4.60it/s, epoch=53, grad_norm=188.457, loss=241.6173, lr=5.85e-04]
Epoch 53 TRAIN:  58%|█████▊    | 79/137 [00:19<00:12,  4.60it/s, epoch=53, grad_norm=113.929, loss=325.0447, lr=5.85e-04]
Epoch 53 TRAIN:  60%|█████▉    | 82/137 [00:19<00:12,  4.41it/s, epoch=53, grad_norm=113.929, loss=325.0447, lr=5.85e-04]
Epoch 53 TRAIN:  60%|█████▉    | 82/137 [00:19<00:12,  4.41it/s, epoch=53, grad_norm=241.985, loss=344.6281, lr=5.85e-04]
Epoch 53 TRAIN:  60%|█████▉    | 82/137 [00:19<00:12,  4.41it/s, epoch=53, grad_norm=88.660, loss=306.0510, lr=5.85e-04] 
Epoch 53 TRAIN:  60%|█████▉    | 82/137 [00:19<00:12,  4.41it/s, epoch=53, grad_norm=103.260, loss=314.2332, lr=5.85e-04]
Epoch 53 TRAIN:  62%|██████▏   | 85/137 [00:19<00:11,  4.61it/s, epoch=53, grad_norm=103.260, loss=314.2332, lr=5.85e-04]
Epoch 53 TRAIN:  62%|██████▏   | 85/137 [00:20<00:11,  4.61it/s, epoch=53, grad_norm=127.081, loss=338.6234, lr=5.85e-04]
Epoch 53 TRAIN:  62%|██████▏   | 85/137 [00:20<00:11,  4.61it/s, epoch=53, grad_norm=148.693, loss=243.9404, lr=5.85e-04]
Epoch 53 TRAIN:  62%|██████▏   | 85/137 [00:20<00:11,  4.61it/s, epoch=53, grad_norm=112.788, loss=302.5093, lr=5.85e-04]
Epoch 53 TRAIN:  64%|██████▍   | 88/137 [00:20<00:10,  4.77it/s, epoch=53, grad_norm=112.788, loss=302.5093, lr=5.85e-04]
Epoch 53 TRAIN:  64%|██████▍   | 88/137 [00:20<00:10,  4.77it/s, epoch=53, grad_norm=95.867, loss=282.7284, lr=5.85e-04] 
Epoch 53 TRAIN:  64%|██████▍   | 88/137 [00:20<00:10,  4.77it/s, epoch=53, grad_norm=109.807, loss=281.3339, lr=5.85e-04]
Epoch 53 TRAIN:  64%|██████▍   | 88/137 [00:21<00:10,  4.77it/s, epoch=53, grad_norm=183.070, loss=276.3748, lr=5.86e-04]
Epoch 53 TRAIN:  66%|██████▋   | 91/137 [00:21<00:09,  4.62it/s, epoch=53, grad_norm=183.070, loss=276.3748, lr=5.86e-04]
Epoch 53 TRAIN:  66%|██████▋   | 91/137 [00:21<00:09,  4.62it/s, epoch=53, grad_norm=91.391, loss=272.4862, lr=5.86e-04] 
Epoch 53 TRAIN:  66%|██████▋   | 91/137 [00:21<00:09,  4.62it/s, epoch=53, grad_norm=157.571, loss=291.2564, lr=5.86e-04]
Epoch 53 TRAIN:  66%|██████▋   | 91/137 [00:21<00:09,  4.62it/s, epoch=53, grad_norm=102.957, loss=247.3080, lr=5.86e-04]
Epoch 53 TRAIN:  69%|██████▊   | 94/137 [00:21<00:09,  4.77it/s, epoch=53, grad_norm=102.957, loss=247.3080, lr=5.86e-04]
Epoch 53 TRAIN:  69%|██████▊   | 94/137 [00:21<00:09,  4.77it/s, epoch=53, grad_norm=108.777, loss=258.9403, lr=5.86e-04]
Epoch 53 TRAIN:  69%|██████▊   | 94/137 [00:22<00:09,  4.77it/s, epoch=53, grad_norm=164.636, loss=310.0430, lr=5.86e-04]
Epoch 53 TRAIN:  69%|██████▊   | 94/137 [00:22<00:09,  4.77it/s, epoch=53, grad_norm=106.475, loss=235.1667, lr=5.86e-04]
Epoch 53 TRAIN:  71%|███████   | 97/137 [00:22<00:08,  4.83it/s, epoch=53, grad_norm=106.475, loss=235.1667, lr=5.86e-04]
Epoch 53 TRAIN:  71%|███████   | 97/137 [00:22<00:08,  4.83it/s, epoch=53, grad_norm=175.809, loss=285.3126, lr=5.86e-04]
Epoch 53 TRAIN:  71%|███████   | 97/137 [00:22<00:08,  4.83it/s, epoch=53, grad_norm=203.495, loss=348.0135, lr=5.86e-04]
Epoch 53 TRAIN:  71%|███████   | 97/137 [00:22<00:08,  4.83it/s, epoch=53, grad_norm=138.157, loss=286.8633, lr=5.86e-04]2026-04-16 20:19:07,271 DEBUG TRAIN | steps/sec 4.343| Batch 53/100 loss 286.863312 loss_att 244.432617 loss_ctc 385.868256 th_accuracy 0.340528 lr 5.8624e-04 grad_norm 138.156662 rank 0

Epoch 53 TRAIN:  73%|███████▎  | 100/137 [00:22<00:07,  4.93it/s, epoch=53, grad_norm=138.157, loss=286.8633, lr=5.86e-04]
Epoch 53 TRAIN:  73%|███████▎  | 100/137 [00:23<00:07,  4.93it/s, epoch=53, grad_norm=236.332, loss=302.8633, lr=5.86e-04]
Epoch 53 TRAIN:  73%|███████▎  | 100/137 [00:23<00:07,  4.93it/s, epoch=53, grad_norm=156.005, loss=296.3145, lr=5.86e-04]
Epoch 53 TRAIN:  73%|███████▎  | 100/137 [00:23<00:07,  4.93it/s, epoch=53, grad_norm=155.037, loss=345.0404, lr=5.86e-04]
Epoch 53 TRAIN:  75%|███████▌  | 103/137 [00:23<00:06,  4.86it/s, epoch=53, grad_norm=155.037, loss=345.0404, lr=5.86e-04]
Epoch 53 TRAIN:  75%|███████▌  | 103/137 [00:23<00:06,  4.86it/s, epoch=53, grad_norm=145.177, loss=277.1381, lr=5.87e-04]
Epoch 53 TRAIN:  75%|███████▌  | 103/137 [00:23<00:06,  4.86it/s, epoch=53, grad_norm=137.807, loss=302.1069, lr=5.87e-04]
Epoch 53 TRAIN:  75%|███████▌  | 103/137 [00:24<00:06,  4.86it/s, epoch=53, grad_norm=125.010, loss=256.9094, lr=5.87e-04]
Epoch 53 TRAIN:  77%|███████▋  | 106/137 [00:24<00:06,  4.81it/s, epoch=53, grad_norm=125.010, loss=256.9094, lr=5.87e-04]
Epoch 53 TRAIN:  77%|███████▋  | 106/137 [00:24<00:06,  4.81it/s, epoch=53, grad_norm=163.209, loss=294.4646, lr=5.87e-04]
Epoch 53 TRAIN:  77%|███████▋  | 106/137 [00:24<00:06,  4.81it/s, epoch=53, grad_norm=166.748, loss=293.3541, lr=5.87e-04]
Epoch 53 TRAIN:  77%|███████▋  | 106/137 [00:24<00:06,  4.81it/s, epoch=53, grad_norm=168.621, loss=269.7783, lr=5.87e-04]
Epoch 53 TRAIN:  80%|███████▉  | 109/137 [00:24<00:06,  4.46it/s, epoch=53, grad_norm=168.621, loss=269.7783, lr=5.87e-04]
Epoch 53 TRAIN:  80%|███████▉  | 109/137 [00:25<00:06,  4.46it/s, epoch=53, grad_norm=259.679, loss=281.5130, lr=5.87e-04]
Epoch 53 TRAIN:  80%|███████▉  | 109/137 [00:25<00:06,  4.46it/s, epoch=53, grad_norm=192.822, loss=271.5203, lr=5.87e-04]
Epoch 53 TRAIN:  80%|███████▉  | 109/137 [00:25<00:06,  4.46it/s, epoch=53, grad_norm=114.383, loss=229.4261, lr=5.87e-04]
Epoch 53 TRAIN:  82%|████████▏ | 112/137 [00:25<00:05,  4.50it/s, epoch=53, grad_norm=114.383, loss=229.4261, lr=5.87e-04]
Epoch 53 TRAIN:  82%|████████▏ | 112/137 [00:25<00:05,  4.50it/s, epoch=53, grad_norm=108.903, loss=261.8702, lr=5.87e-04]
Epoch 53 TRAIN:  82%|████████▏ | 112/137 [00:25<00:05,  4.50it/s, epoch=53, grad_norm=223.103, loss=278.6969, lr=5.87e-04]
Epoch 53 TRAIN:  82%|████████▏ | 112/137 [00:26<00:05,  4.50it/s, epoch=53, grad_norm=226.838, loss=326.7542, lr=5.87e-04]
Epoch 53 TRAIN:  84%|████████▍ | 115/137 [00:26<00:04,  4.80it/s, epoch=53, grad_norm=226.838, loss=326.7542, lr=5.87e-04]
Epoch 53 TRAIN:  84%|████████▍ | 115/137 [00:26<00:04,  4.80it/s, epoch=53, grad_norm=118.265, loss=202.1059, lr=5.88e-04]
Epoch 53 TRAIN:  84%|████████▍ | 115/137 [00:26<00:04,  4.80it/s, epoch=53, grad_norm=119.061, loss=240.3121, lr=5.88e-04]
Epoch 53 TRAIN:  84%|████████▍ | 115/137 [00:26<00:04,  4.80it/s, epoch=53, grad_norm=236.733, loss=367.0250, lr=5.88e-04]
Epoch 53 TRAIN:  86%|████████▌ | 118/137 [00:26<00:04,  4.75it/s, epoch=53, grad_norm=236.733, loss=367.0250, lr=5.88e-04]
Epoch 53 TRAIN:  86%|████████▌ | 118/137 [00:26<00:04,  4.75it/s, epoch=53, grad_norm=118.491, loss=260.3448, lr=5.88e-04]
Epoch 53 TRAIN:  86%|████████▌ | 118/137 [00:27<00:04,  4.75it/s, epoch=53, grad_norm=136.873, loss=276.6677, lr=5.88e-04]
Epoch 53 TRAIN:  86%|████████▌ | 118/137 [00:27<00:04,  4.75it/s, epoch=53, grad_norm=135.712, loss=248.9198, lr=5.88e-04]
Epoch 53 TRAIN:  88%|████████▊ | 121/137 [00:27<00:03,  4.75it/s, epoch=53, grad_norm=135.712, loss=248.9198, lr=5.88e-04]
Epoch 53 TRAIN:  88%|████████▊ | 121/137 [00:27<00:03,  4.75it/s, epoch=53, grad_norm=149.459, loss=248.8127, lr=5.88e-04]
Epoch 53 TRAIN:  88%|████████▊ | 121/137 [00:27<00:03,  4.75it/s, epoch=53, grad_norm=151.352, loss=288.8324, lr=5.88e-04]
Epoch 53 TRAIN:  88%|████████▊ | 121/137 [00:28<00:03,  4.75it/s, epoch=53, grad_norm=110.582, loss=223.1166, lr=5.88e-04]
Epoch 53 TRAIN:  91%|█████████ | 124/137 [00:28<00:02,  4.79it/s, epoch=53, grad_norm=110.582, loss=223.1166, lr=5.88e-04]
Epoch 53 TRAIN:  91%|█████████ | 124/137 [00:28<00:02,  4.79it/s, epoch=53, grad_norm=174.331, loss=269.0039, lr=5.88e-04]
Epoch 53 TRAIN:  91%|█████████ | 124/137 [00:28<00:02,  4.79it/s, epoch=53, grad_norm=258.517, loss=372.6539, lr=5.88e-04]
Epoch 53 TRAIN:  91%|█████████ | 124/137 [00:28<00:02,  4.79it/s, epoch=53, grad_norm=248.920, loss=363.8774, lr=5.88e-04]
Epoch 53 TRAIN:  93%|█████████▎| 127/137 [00:28<00:02,  4.65it/s, epoch=53, grad_norm=248.920, loss=363.8774, lr=5.88e-04]
Epoch 53 TRAIN:  93%|█████████▎| 127/137 [00:28<00:02,  4.65it/s, epoch=53, grad_norm=194.420, loss=329.1311, lr=5.88e-04]
Epoch 53 TRAIN:  93%|█████████▎| 127/137 [00:29<00:02,  4.65it/s, epoch=53, grad_norm=146.069, loss=289.8906, lr=5.89e-04]
Epoch 53 TRAIN:  93%|█████████▎| 127/137 [00:29<00:02,  4.65it/s, epoch=53, grad_norm=157.008, loss=348.0619, lr=5.89e-04]
Epoch 53 TRAIN:  95%|█████████▍| 130/137 [00:29<00:01,  4.77it/s, epoch=53, grad_norm=157.008, loss=348.0619, lr=5.89e-04]
Epoch 53 TRAIN:  95%|█████████▍| 130/137 [00:29<00:01,  4.77it/s, epoch=53, grad_norm=202.174, loss=297.1118, lr=5.89e-04]
Epoch 53 TRAIN:  95%|█████████▍| 130/137 [00:29<00:01,  4.77it/s, epoch=53, grad_norm=195.880, loss=348.9110, lr=5.89e-04]
Epoch 53 TRAIN:  95%|█████████▍| 130/137 [00:29<00:01,  4.77it/s, epoch=53, grad_norm=305.527, loss=315.3594, lr=5.89e-04]
Epoch 53 TRAIN:  97%|█████████▋| 133/137 [00:29<00:00,  4.64it/s, epoch=53, grad_norm=305.527, loss=315.3594, lr=5.89e-04]
Epoch 53 TRAIN:  97%|█████████▋| 133/137 [00:30<00:00,  4.64it/s, epoch=53, grad_norm=166.901, loss=350.7686, lr=5.89e-04]
Epoch 53 TRAIN:  97%|█████████▋| 133/137 [00:30<00:00,  4.64it/s, epoch=53, grad_norm=185.736, loss=396.2117, lr=5.89e-04]
Epoch 53 TRAIN:  97%|█████████▋| 133/137 [00:30<00:00,  4.64it/s, epoch=53, grad_norm=254.699, loss=312.9173, lr=5.89e-04]
Epoch 53 TRAIN:  99%|█████████▉| 136/137 [00:30<00:00,  4.46it/s, epoch=53, grad_norm=254.699, loss=312.9173, lr=5.89e-04]

CV:   0%|          | 0/6 [00:00<?, ?it/s]/home/rookie/Documents/batuka/projects/wenet/wenet/wenet/utils/train_utils.py:689: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  amp_autocast(enabled=dtype is not None,

CV:  17%|█▋        | 1/6 [00:00<00:03,  1.33it/s]
                                                 
2026-04-16 20:19:15,721 INFO Epoch 53 Step 7364 CV info lr 5.8912e-04 cv_loss 492.14139071377843 rank 0 acc 0.31601478656133014
2026-04-16 20:19:15,723 INFO [Rank 0] Checkpoint: save to checkpoint exp/conformer_mongolian/epoch_53.pt
2026-04-16 20:19:15,843 INFO Epoch 54 Step 7364 TRAIN info lr 5.8912e-04 rank 0
2026-04-16 20:19:15,864 INFO using accumulate grad, new batch size is 1 times larger than before

Epoch 54 TRAIN:   0%|          | 0/136 [00:00<?, ?it/s]
Epoch 54 TRAIN:   0%|          | 0/136 [00:03<?, ?it/s, epoch=54, grad_norm=440.696, loss=344.0067, lr=5.89e-04]
Epoch 54 TRAIN:   1%|          | 1/136 [00:03<08:30,  3.78s/it, epoch=54, grad_norm=440.696, loss=344.0067, lr=5.89e-04]
Epoch 54 TRAIN:   1%|          | 1/136 [00:03<08:30,  3.78s/it, epoch=54, grad_norm=363.858, loss=340.2148, lr=5.89e-04]
Epoch 54 TRAIN:   1%|          | 1/136 [00:03<08:30,  3.78s/it, epoch=54, grad_norm=180.417, loss=226.9702, lr=5.89e-04]
Epoch 54 TRAIN:   1%|          | 1/136 [00:04<08:30,  3.78s/it, epoch=54, grad_norm=474.371, loss=382.8544, lr=5.89e-04]
Epoch 54 TRAIN:   1%|          | 1/136 [00:04<08:30,  3.78s/it, epoch=54, grad_norm=350.021, loss=371.6675, lr=5.90e-04]
Epoch 54 TRAIN:   1%|          | 1/136 [00:04<08:30,  3.78s/it, epoch=54, grad_norm=253.298, loss=298.9188, lr=5.90e-04]
Epoch 54 TRAIN:   1%|          | 1/136 [00:04<08:30,  3.78s/it, epoch=54, grad_norm=346.898, loss=388.8735, lr=5.90e-04]
Epoch 54 TRAIN:   1%|          | 1/136 [00:04<08:30,  3.78s/it, epoch=54, grad_norm=331.788, loss=401.5036, lr=5.90e-04]
Epoch 54 TRAIN:   6%|▌         | 8/136 [00:04<00:54,  2.36it/s, epoch=54, grad_norm=331.788, loss=401.5036, lr=5.90e-04]
Epoch 54 TRAIN:   6%|▌         | 8/136 [00:04<00:54,  2.36it/s, epoch=54, grad_norm=115.302, loss=304.8177, lr=5.90e-04]
Epoch 54 TRAIN:   6%|▌         | 8/136 [00:04<00:54,  2.36it/s, epoch=54, grad_norm=145.563, loss=343.5924, lr=5.90e-04]
Epoch 54 TRAIN:   6%|▌         | 8/136 [00:04<00:54,  2.36it/s, epoch=54, grad_norm=98.231, loss=319.3618, lr=5.90e-04] 
Epoch 54 TRAIN:   6%|▌         | 8/136 [00:05<00:54,  2.36it/s, epoch=54, grad_norm=128.392, loss=289.9307, lr=5.90e-04]
Epoch 54 TRAIN:   9%|▉         | 12/136 [00:05<00:39,  3.13it/s, epoch=54, grad_norm=128.392, loss=289.9307, lr=5.90e-04]
Epoch 54 TRAIN:   9%|▉         | 12/136 [00:05<00:39,  3.13it/s, epoch=54, grad_norm=120.713, loss=304.6183, lr=5.90e-04]
Epoch 54 TRAIN:   9%|▉         | 12/136 [00:05<00:39,  3.13it/s, epoch=54, grad_norm=149.502, loss=311.4872, lr=5.90e-04]
Epoch 54 TRAIN:   9%|▉         | 12/136 [00:05<00:39,  3.13it/s, epoch=54, grad_norm=186.355, loss=311.7887, lr=5.90e-04]
Epoch 54 TRAIN:   9%|▉         | 12/136 [00:05<00:39,  3.13it/s, epoch=54, grad_norm=128.864, loss=286.4648, lr=5.90e-04]
Epoch 54 TRAIN:  12%|█▏        | 16/136 [00:05<00:30,  3.87it/s, epoch=54, grad_norm=128.864, loss=286.4648, lr=5.90e-04]
Epoch 54 TRAIN:  12%|█▏        | 16/136 [00:05<00:30,  3.87it/s, epoch=54, grad_norm=137.437, loss=287.9182, lr=5.90e-04]
Epoch 54 TRAIN:  12%|█▏        | 16/136 [00:06<00:30,  3.87it/s, epoch=54, grad_norm=124.800, loss=273.9929, lr=5.91e-04]
Epoch 54 TRAIN:  12%|█▏        | 16/136 [00:06<00:30,  3.87it/s, epoch=54, grad_norm=97.392, loss=281.6228, lr=5.91e-04] 
Epoch 54 TRAIN:  12%|█▏        | 16/136 [00:06<00:30,  3.87it/s, epoch=54, grad_norm=171.556, loss=326.9376, lr=5.91e-04]
Epoch 54 TRAIN:  15%|█▍        | 20/136 [00:06<00:27,  4.23it/s, epoch=54, grad_norm=171.556, loss=326.9376, lr=5.91e-04]
Epoch 54 TRAIN:  15%|█▍        | 20/136 [00:06<00:27,  4.23it/s, epoch=54, grad_norm=153.641, loss=301.0896, lr=5.91e-04]
Epoch 54 TRAIN:  15%|█▍        | 20/136 [00:06<00:27,  4.23it/s, epoch=54, grad_norm=110.235, loss=305.6450, lr=5.91e-04]
Epoch 54 TRAIN:  15%|█▍        | 20/136 [00:07<00:27,  4.23it/s, epoch=54, grad_norm=97.244, loss=297.1024, lr=5.91e-04] 
Epoch 54 TRAIN:  17%|█▋        | 23/136 [00:07<00:24,  4.55it/s, epoch=54, grad_norm=97.244, loss=297.1024, lr=5.91e-04]
Epoch 54 TRAIN:  17%|█▋        | 23/136 [00:07<00:24,  4.55it/s, epoch=54, grad_norm=82.516, loss=296.6371, lr=5.91e-04]
Epoch 54 TRAIN:  17%|█▋        | 23/136 [00:07<00:24,  4.55it/s, epoch=54, grad_norm=107.781, loss=333.4459, lr=5.91e-04]
Epoch 54 TRAIN:  17%|█▋        | 23/136 [00:07<00:24,  4.55it/s, epoch=54, grad_norm=155.626, loss=296.1752, lr=5.91e-04]
Epoch 54 TRAIN:  19%|█▉        | 26/136 [00:07<00:24,  4.49it/s, epoch=54, grad_norm=155.626, loss=296.1752, lr=5.91e-04]
Epoch 54 TRAIN:  19%|█▉        | 26/136 [00:07<00:24,  4.49it/s, epoch=54, grad_norm=153.298, loss=283.0537, lr=5.91e-04]
Epoch 54 TRAIN:  19%|█▉        | 26/136 [00:08<00:24,  4.49it/s, epoch=54, grad_norm=150.055, loss=322.1304, lr=5.91e-04]
Epoch 54 TRAIN:  19%|█▉        | 26/136 [00:08<00:24,  4.49it/s, epoch=54, grad_norm=110.350, loss=320.9408, lr=5.91e-04]
Epoch 54 TRAIN:  21%|██▏       | 29/136 [00:08<00:23,  4.62it/s, epoch=54, grad_norm=110.350, loss=320.9408, lr=5.91e-04]
Epoch 54 TRAIN:  21%|██▏       | 29/136 [00:08<00:23,  4.62it/s, epoch=54, grad_norm=140.336, loss=293.5468, lr=5.92e-04]
Epoch 54 TRAIN:  21%|██▏       | 29/136 [00:08<00:23,  4.62it/s, epoch=54, grad_norm=114.739, loss=291.7167, lr=5.92e-04]
Epoch 54 TRAIN:  21%|██▏       | 29/136 [00:08<00:23,  4.62it/s, epoch=54, grad_norm=82.642, loss=285.8491, lr=5.92e-04] 
Epoch 54 TRAIN:  24%|██▎       | 32/136 [00:08<00:21,  4.94it/s, epoch=54, grad_norm=82.642, loss=285.8491, lr=5.92e-04]
Epoch 54 TRAIN:  24%|██▎       | 32/136 [00:09<00:21,  4.94it/s, epoch=54, grad_norm=143.681, loss=351.3864, lr=5.92e-04]
Epoch 54 TRAIN:  24%|██▎       | 32/136 [00:09<00:21,  4.94it/s, epoch=54, grad_norm=128.378, loss=255.8375, lr=5.92e-04]
Epoch 54 TRAIN:  24%|██▎       | 32/136 [00:09<00:21,  4.94it/s, epoch=54, grad_norm=91.745, loss=296.9340, lr=5.92e-04] 
Epoch 54 TRAIN:  26%|██▌       | 35/136 [00:09<00:20,  4.88it/s, epoch=54, grad_norm=91.745, loss=296.9340, lr=5.92e-04]
Epoch 54 TRAIN:  26%|██▌       | 35/136 [00:09<00:20,  4.88it/s, epoch=54, grad_norm=105.871, loss=300.5815, lr=5.92e-04]
Epoch 54 TRAIN:  26%|██▌       | 35/136 [00:09<00:20,  4.88it/s, epoch=54, grad_norm=94.114, loss=243.2312, lr=5.92e-04] 
Epoch 54 TRAIN:  26%|██▌       | 35/136 [00:10<00:20,  4.88it/s, epoch=54, grad_norm=90.121, loss=287.9034, lr=5.92e-04]
Epoch 54 TRAIN:  28%|██▊       | 38/136 [00:10<00:20,  4.78it/s, epoch=54, grad_norm=90.121, loss=287.9034, lr=5.92e-04]
Epoch 54 TRAIN:  28%|██▊       | 38/136 [00:10<00:20,  4.78it/s, epoch=54, grad_norm=123.560, loss=312.1833, lr=5.92e-04]
Epoch 54 TRAIN:  28%|██▊       | 38/136 [00:10<00:20,  4.78it/s, epoch=54, grad_norm=148.492, loss=269.4746, lr=5.92e-04]
Epoch 54 TRAIN:  28%|██▊       | 38/136 [00:10<00:20,  4.78it/s, epoch=54, grad_norm=120.605, loss=328.5940, lr=5.92e-04]
Epoch 54 TRAIN:  30%|███       | 41/136 [00:10<00:19,  4.91it/s, epoch=54, grad_norm=120.605, loss=328.5940, lr=5.92e-04]
Epoch 54 TRAIN:  30%|███       | 41/136 [00:10<00:19,  4.91it/s, epoch=54, grad_norm=123.289, loss=278.8003, lr=5.92e-04]
Epoch 54 TRAIN:  30%|███       | 41/136 [00:11<00:19,  4.91it/s, epoch=54, grad_norm=115.792, loss=331.2719, lr=5.93e-04]
Epoch 54 TRAIN:  30%|███       | 41/136 [00:11<00:19,  4.91it/s, epoch=54, grad_norm=103.962, loss=281.6670, lr=5.93e-04]
Epoch 54 TRAIN:  32%|███▏      | 44/136 [00:11<00:19,  4.79it/s, epoch=54, grad_norm=103.962, loss=281.6670, lr=5.93e-04]
Epoch 54 TRAIN:  32%|███▏      | 44/136 [00:11<00:19,  4.79it/s, epoch=54, grad_norm=122.457, loss=245.9776, lr=5.93e-04]
Epoch 54 TRAIN:  32%|███▏      | 44/136 [00:11<00:19,  4.79it/s, epoch=54, grad_norm=107.249, loss=299.7466, lr=5.93e-04]
Epoch 54 TRAIN:  32%|███▏      | 44/136 [00:12<00:19,  4.79it/s, epoch=54, grad_norm=84.496, loss=269.9932, lr=5.93e-04] 
Epoch 54 TRAIN:  35%|███▍      | 47/136 [00:12<00:18,  4.77it/s, epoch=54, grad_norm=84.496, loss=269.9932, lr=5.93e-04]
Epoch 54 TRAIN:  35%|███▍      | 47/136 [00:12<00:18,  4.77it/s, epoch=54, grad_norm=94.951, loss=275.9228, lr=5.93e-04]
Epoch 54 TRAIN:  35%|███▍      | 47/136 [00:12<00:18,  4.77it/s, epoch=54, grad_norm=99.421, loss=280.6782, lr=5.93e-04]
Epoch 54 TRAIN:  35%|███▍      | 47/136 [00:12<00:18,  4.77it/s, epoch=54, grad_norm=82.009, loss=273.6166, lr=5.93e-04]
Epoch 54 TRAIN:  37%|███▋      | 50/136 [00:12<00:17,  5.00it/s, epoch=54, grad_norm=82.009, loss=273.6166, lr=5.93e-04]
Epoch 54 TRAIN:  37%|███▋      | 50/136 [00:12<00:17,  5.00it/s, epoch=54, grad_norm=180.101, loss=327.7465, lr=5.93e-04]
Epoch 54 TRAIN:  37%|███▋      | 50/136 [00:12<00:17,  5.00it/s, epoch=54, grad_norm=109.078, loss=297.1749, lr=5.93e-04]
Epoch 54 TRAIN:  37%|███▋      | 50/136 [00:13<00:17,  5.00it/s, epoch=54, grad_norm=119.244, loss=292.2291, lr=5.93e-04]
Epoch 54 TRAIN:  39%|███▉      | 53/136 [00:13<00:16,  4.97it/s, epoch=54, grad_norm=119.244, loss=292.2291, lr=5.93e-04]
Epoch 54 TRAIN:  39%|███▉      | 53/136 [00:13<00:16,  4.97it/s, epoch=54, grad_norm=171.271, loss=348.1807, lr=5.93e-04]
Epoch 54 TRAIN:  39%|███▉      | 53/136 [00:13<00:16,  4.97it/s, epoch=54, grad_norm=74.972, loss=273.8776, lr=5.94e-04] 
Epoch 54 TRAIN:  39%|███▉      | 53/136 [00:13<00:16,  4.97it/s, epoch=54, grad_norm=82.460, loss=278.0949, lr=5.94e-04]
Epoch 54 TRAIN:  41%|████      | 56/136 [00:13<00:16,  4.89it/s, epoch=54, grad_norm=82.460, loss=278.0949, lr=5.94e-04]
Epoch 54 TRAIN:  41%|████      | 56/136 [00:14<00:16,  4.89it/s, epoch=54, grad_norm=114.644, loss=342.4689, lr=5.94e-04]
Epoch 54 TRAIN:  41%|████      | 56/136 [00:14<00:16,  4.89it/s, epoch=54, grad_norm=133.684, loss=303.4878, lr=5.94e-04]
Epoch 54 TRAIN:  41%|████      | 56/136 [00:14<00:16,  4.89it/s, epoch=54, grad_norm=101.532, loss=282.4676, lr=5.94e-04]
Epoch 54 TRAIN:  43%|████▎     | 59/136 [00:14<00:16,  4.60it/s, epoch=54, grad_norm=101.532, loss=282.4676, lr=5.94e-04]
Epoch 54 TRAIN:  43%|████▎     | 59/136 [00:14<00:16,  4.60it/s, epoch=54, grad_norm=149.736, loss=333.8798, lr=5.94e-04]
Epoch 54 TRAIN:  43%|████▎     | 59/136 [00:14<00:16,  4.60it/s, epoch=54, grad_norm=134.579, loss=296.1139, lr=5.94e-04]
Epoch 54 TRAIN:  43%|████▎     | 59/136 [00:15<00:16,  4.60it/s, epoch=54, grad_norm=148.176, loss=281.0120, lr=5.94e-04]
Epoch 54 TRAIN:  46%|████▌     | 62/136 [00:15<00:16,  4.45it/s, epoch=54, grad_norm=148.176, loss=281.0120, lr=5.94e-04]
Epoch 54 TRAIN:  46%|████▌     | 62/136 [00:15<00:16,  4.45it/s, epoch=54, grad_norm=90.719, loss=276.1007, lr=5.94e-04] 
Epoch 54 TRAIN:  46%|████▌     | 62/136 [00:15<00:16,  4.45it/s, epoch=54, grad_norm=160.842, loss=302.2726, lr=5.94e-04]
Epoch 54 TRAIN:  46%|████▌     | 62/136 [00:15<00:16,  4.45it/s, epoch=54, grad_norm=193.113, loss=245.9076, lr=5.94e-04]
Epoch 54 TRAIN:  48%|████▊     | 65/136 [00:15<00:14,  4.80it/s, epoch=54, grad_norm=193.113, loss=245.9076, lr=5.94e-04]
Epoch 54 TRAIN:  48%|████▊     | 65/136 [00:15<00:14,  4.80it/s, epoch=54, grad_norm=87.302, loss=283.8952, lr=5.94e-04] 
Epoch 54 TRAIN:  48%|████▊     | 65/136 [00:16<00:14,  4.80it/s, epoch=54, grad_norm=142.422, loss=279.3001, lr=5.94e-04]
Epoch 54 TRAIN:  48%|████▊     | 65/136 [00:16<00:14,  4.80it/s, epoch=54, grad_norm=112.496, loss=282.0380, lr=5.95e-04]
Epoch 54 TRAIN:  50%|█████     | 68/136 [00:16<00:14,  4.57it/s, epoch=54, grad_norm=112.496, loss=282.0380, lr=5.95e-04]
Epoch 54 TRAIN:  50%|█████     | 68/136 [00:16<00:14,  4.57it/s, epoch=54, grad_norm=214.461, loss=336.9888, lr=5.95e-04]
Epoch 54 TRAIN:  50%|█████     | 68/136 [00:16<00:14,  4.57it/s, epoch=54, grad_norm=142.556, loss=275.1953, lr=5.95e-04]
Epoch 54 TRAIN:  50%|█████     | 68/136 [00:17<00:14,  4.57it/s, epoch=54, grad_norm=115.641, loss=289.3365, lr=5.95e-04]
Epoch 54 TRAIN:  52%|█████▏    | 71/136 [00:17<00:13,  4.86it/s, epoch=54, grad_norm=115.641, loss=289.3365, lr=5.95e-04]
Epoch 54 TRAIN:  52%|█████▏    | 71/136 [00:17<00:13,  4.86it/s, epoch=54, grad_norm=112.013, loss=299.0103, lr=5.95e-04]
Epoch 54 TRAIN:  52%|█████▏    | 71/136 [00:17<00:13,  4.86it/s, epoch=54, grad_norm=183.328, loss=290.6524, lr=5.95e-04]
Epoch 54 TRAIN:  52%|█████▏    | 71/136 [00:17<00:13,  4.86it/s, epoch=54, grad_norm=195.170, loss=262.7946, lr=5.95e-04]
Epoch 54 TRAIN:  54%|█████▍    | 74/136 [00:17<00:12,  4.78it/s, epoch=54, grad_norm=195.170, loss=262.7946, lr=5.95e-04]
Epoch 54 TRAIN:  54%|█████▍    | 74/136 [00:17<00:12,  4.78it/s, epoch=54, grad_norm=129.482, loss=261.3104, lr=5.95e-04]
Epoch 54 TRAIN:  54%|█████▍    | 74/136 [00:18<00:12,  4.78it/s, epoch=54, grad_norm=128.847, loss=286.2581, lr=5.95e-04]
Epoch 54 TRAIN:  54%|█████▍    | 74/136 [00:18<00:12,  4.78it/s, epoch=54, grad_norm=152.711, loss=348.7273, lr=5.95e-04]
Epoch 54 TRAIN:  57%|█████▋    | 77/136 [00:18<00:12,  4.76it/s, epoch=54, grad_norm=152.711, loss=348.7273, lr=5.95e-04]
Epoch 54 TRAIN:  57%|█████▋    | 77/136 [00:18<00:12,  4.76it/s, epoch=54, grad_norm=135.400, loss=313.4376, lr=5.95e-04]
Epoch 54 TRAIN:  57%|█████▋    | 77/136 [00:18<00:12,  4.76it/s, epoch=54, grad_norm=122.476, loss=321.5972, lr=5.95e-04]
Epoch 54 TRAIN:  57%|█████▋    | 77/136 [00:19<00:12,  4.76it/s, epoch=54, grad_norm=140.981, loss=309.9042, lr=5.96e-04]
Epoch 54 TRAIN:  59%|█████▉    | 80/136 [00:19<00:12,  4.61it/s, epoch=54, grad_norm=140.981, loss=309.9042, lr=5.96e-04]
Epoch 54 TRAIN:  59%|█████▉    | 80/136 [00:19<00:12,  4.61it/s, epoch=54, grad_norm=153.251, loss=338.7305, lr=5.96e-04]
Epoch 54 TRAIN:  59%|█████▉    | 80/136 [00:19<00:12,  4.61it/s, epoch=54, grad_norm=136.945, loss=314.4232, lr=5.96e-04]
Epoch 54 TRAIN:  59%|█████▉    | 80/136 [00:19<00:12,  4.61it/s, epoch=54, grad_norm=115.452, loss=325.0800, lr=5.96e-04]
Epoch 54 TRAIN:  61%|██████    | 83/136 [00:19<00:11,  4.67it/s, epoch=54, grad_norm=115.452, loss=325.0800, lr=5.96e-04]
Epoch 54 TRAIN:  61%|██████    | 83/136 [00:19<00:11,  4.67it/s, epoch=54, grad_norm=127.307, loss=270.1612, lr=5.96e-04]
Epoch 54 TRAIN:  61%|██████    | 83/136 [00:20<00:11,  4.67it/s, epoch=54, grad_norm=94.982, loss=318.8185, lr=5.96e-04] 
Epoch 54 TRAIN:  61%|██████    | 83/136 [00:20<00:11,  4.67it/s, epoch=54, grad_norm=270.805, loss=255.1914, lr=5.96e-04]
Epoch 54 TRAIN:  63%|██████▎   | 86/136 [00:20<00:10,  4.58it/s, epoch=54, grad_norm=270.805, loss=255.1914, lr=5.96e-04]
Epoch 54 TRAIN:  63%|██████▎   | 86/136 [00:20<00:10,  4.58it/s, epoch=54, grad_norm=103.546, loss=295.1181, lr=5.96e-04]
Epoch 54 TRAIN:  63%|██████▎   | 86/136 [00:20<00:10,  4.58it/s, epoch=54, grad_norm=136.670, loss=322.7465, lr=5.96e-04]
Epoch 54 TRAIN:  63%|██████▎   | 86/136 [00:20<00:10,  4.58it/s, epoch=54, grad_norm=322.214, loss=210.5078, lr=5.96e-04]
Epoch 54 TRAIN:  65%|██████▌   | 89/136 [00:20<00:10,  4.62it/s, epoch=54, grad_norm=322.214, loss=210.5078, lr=5.96e-04]
Epoch 54 TRAIN:  65%|██████▌   | 89/136 [00:21<00:10,  4.62it/s, epoch=54, grad_norm=167.881, loss=265.5522, lr=5.96e-04]
Epoch 54 TRAIN:  65%|██████▌   | 89/136 [00:21<00:10,  4.62it/s, epoch=54, grad_norm=84.894, loss=249.1959, lr=5.96e-04] 
Epoch 54 TRAIN:  65%|██████▌   | 89/136 [00:21<00:10,  4.62it/s, epoch=54, grad_norm=93.090, loss=255.8952, lr=5.96e-04]
Epoch 54 TRAIN:  68%|██████▊   | 92/136 [00:21<00:09,  4.79it/s, epoch=54, grad_norm=93.090, loss=255.8952, lr=5.96e-04]
Epoch 54 TRAIN:  68%|██████▊   | 92/136 [00:21<00:09,  4.79it/s, epoch=54, grad_norm=195.487, loss=291.8846, lr=5.97e-04]
Epoch 54 TRAIN:  68%|██████▊   | 92/136 [00:21<00:09,  4.79it/s, epoch=54, grad_norm=93.889, loss=245.6833, lr=5.97e-04] 
Epoch 54 TRAIN:  68%|██████▊   | 92/136 [00:22<00:09,  4.79it/s, epoch=54, grad_norm=186.468, loss=238.4896, lr=5.97e-04]
Epoch 54 TRAIN:  70%|██████▉   | 95/136 [00:22<00:08,  4.75it/s, epoch=54, grad_norm=186.468, loss=238.4896, lr=5.97e-04]
Epoch 54 TRAIN:  70%|██████▉   | 95/136 [00:22<00:08,  4.75it/s, epoch=54, grad_norm=262.682, loss=266.5840, lr=5.97e-04]
Epoch 54 TRAIN:  70%|██████▉   | 95/136 [00:22<00:08,  4.75it/s, epoch=54, grad_norm=104.354, loss=310.7705, lr=5.97e-04]
Epoch 54 TRAIN:  70%|██████▉   | 95/136 [00:22<00:08,  4.75it/s, epoch=54, grad_norm=129.208, loss=285.5375, lr=5.97e-04]
Epoch 54 TRAIN:  72%|███████▏  | 98/136 [00:22<00:07,  4.80it/s, epoch=54, grad_norm=129.208, loss=285.5375, lr=5.97e-04]
Epoch 54 TRAIN:  72%|███████▏  | 98/136 [00:22<00:07,  4.80it/s, epoch=54, grad_norm=148.314, loss=184.4691, lr=5.97e-04]
Epoch 54 TRAIN:  72%|███████▏  | 98/136 [00:23<00:07,  4.80it/s, epoch=54, grad_norm=144.104, loss=243.5622, lr=5.97e-04]2026-04-16 20:19:39,009 DEBUG TRAIN | steps/sec 4.285| Batch 54/100 loss 243.562225 loss_att 199.351166 loss_ctc 346.721405 th_accuracy 0.374648 lr 5.9712e-04 grad_norm 144.104340 rank 0

Epoch 54 TRAIN:  72%|███████▏  | 98/136 [00:23<00:07,  4.80it/s, epoch=54, grad_norm=177.444, loss=318.9401, lr=5.97e-04]
Epoch 54 TRAIN:  74%|███████▍  | 101/136 [00:23<00:07,  4.81it/s, epoch=54, grad_norm=177.444, loss=318.9401, lr=5.97e-04]
Epoch 54 TRAIN:  74%|███████▍  | 101/136 [00:23<00:07,  4.81it/s, epoch=54, grad_norm=178.315, loss=307.9770, lr=5.97e-04]
Epoch 54 TRAIN:  74%|███████▍  | 101/136 [00:23<00:07,  4.81it/s, epoch=54, grad_norm=131.107, loss=259.2701, lr=5.97e-04]
Epoch 54 TRAIN:  74%|███████▍  | 101/136 [00:24<00:07,  4.81it/s, epoch=54, grad_norm=141.061, loss=302.8319, lr=5.97e-04]
Epoch 54 TRAIN:  76%|███████▋  | 104/136 [00:24<00:06,  4.71it/s, epoch=54, grad_norm=141.061, loss=302.8319, lr=5.97e-04]
Epoch 54 TRAIN:  76%|███████▋  | 104/136 [00:24<00:06,  4.71it/s, epoch=54, grad_norm=153.324, loss=307.1965, lr=5.98e-04]
Epoch 54 TRAIN:  76%|███████▋  | 104/136 [00:24<00:06,  4.71it/s, epoch=54, grad_norm=102.136, loss=298.5203, lr=5.98e-04]
Epoch 54 TRAIN:  76%|███████▋  | 104/136 [00:24<00:06,  4.71it/s, epoch=54, grad_norm=177.079, loss=244.9388, lr=5.98e-04]
Epoch 54 TRAIN:  79%|███████▊  | 107/136 [00:24<00:06,  4.72it/s, epoch=54, grad_norm=177.079, loss=244.9388, lr=5.98e-04]
Epoch 54 TRAIN:  79%|███████▊  | 107/136 [00:24<00:06,  4.72it/s, epoch=54, grad_norm=157.634, loss=238.3891, lr=5.98e-04]
Epoch 54 TRAIN:  79%|███████▊  | 107/136 [00:25<00:06,  4.72it/s, epoch=54, grad_norm=152.343, loss=294.9603, lr=5.98e-04]
Epoch 54 TRAIN:  79%|███████▊  | 107/136 [00:25<00:06,  4.72it/s, epoch=54, grad_norm=149.582, loss=278.8053, lr=5.98e-04]
Epoch 54 TRAIN:  81%|████████  | 110/136 [00:25<00:05,  4.49it/s, epoch=54, grad_norm=149.582, loss=278.8053, lr=5.98e-04]
Epoch 54 TRAIN:  81%|████████  | 110/136 [00:25<00:05,  4.49it/s, epoch=54, grad_norm=185.027, loss=254.0014, lr=5.98e-04]
Epoch 54 TRAIN:  81%|████████  | 110/136 [00:25<00:05,  4.49it/s, epoch=54, grad_norm=422.012, loss=396.6531, lr=5.98e-04]
Epoch 54 TRAIN:  81%|████████  | 110/136 [00:26<00:05,  4.49it/s, epoch=54, grad_norm=227.979, loss=294.2953, lr=5.98e-04]
Epoch 54 TRAIN:  83%|████████▎ | 113/136 [00:26<00:04,  4.71it/s, epoch=54, grad_norm=227.979, loss=294.2953, lr=5.98e-04]
Epoch 54 TRAIN:  83%|████████▎ | 113/136 [00:26<00:04,  4.71it/s, epoch=54, grad_norm=137.210, loss=240.3543, lr=5.98e-04]
Epoch 54 TRAIN:  83%|████████▎ | 113/136 [00:26<00:04,  4.71it/s, epoch=54, grad_norm=157.061, loss=260.7238, lr=5.98e-04]
Epoch 54 TRAIN:  83%|████████▎ | 113/136 [00:26<00:04,  4.71it/s, epoch=54, grad_norm=145.400, loss=272.7572, lr=5.98e-04]
Epoch 54 TRAIN:  85%|████████▌ | 116/136 [00:26<00:04,  4.64it/s, epoch=54, grad_norm=145.400, loss=272.7572, lr=5.98e-04]
Epoch 54 TRAIN:  85%|████████▌ | 116/136 [00:26<00:04,  4.64it/s, epoch=54, grad_norm=117.007, loss=201.5328, lr=5.98e-04]
Epoch 54 TRAIN:  85%|████████▌ | 116/136 [00:27<00:04,  4.64it/s, epoch=54, grad_norm=313.321, loss=352.5799, lr=5.99e-04]
Epoch 54 TRAIN:  85%|████████▌ | 116/136 [00:27<00:04,  4.64it/s, epoch=54, grad_norm=223.515, loss=353.9241, lr=5.99e-04]
Epoch 54 TRAIN:  88%|████████▊ | 119/136 [00:27<00:03,  4.84it/s, epoch=54, grad_norm=223.515, loss=353.9241, lr=5.99e-04]
Epoch 54 TRAIN:  88%|████████▊ | 119/136 [00:27<00:03,  4.84it/s, epoch=54, grad_norm=125.701, loss=250.4713, lr=5.99e-04]
Epoch 54 TRAIN:  88%|████████▊ | 119/136 [00:27<00:03,  4.84it/s, epoch=54, grad_norm=237.319, loss=371.5038, lr=5.99e-04]
Epoch 54 TRAIN:  88%|████████▊ | 119/136 [00:27<00:03,  4.84it/s, epoch=54, grad_norm=175.328, loss=331.1946, lr=5.99e-04]
Epoch 54 TRAIN:  90%|████████▉ | 122/136 [00:27<00:02,  4.99it/s, epoch=54, grad_norm=175.328, loss=331.1946, lr=5.99e-04]
Epoch 54 TRAIN:  90%|████████▉ | 122/136 [00:28<00:02,  4.99it/s, epoch=54, grad_norm=124.680, loss=286.1289, lr=5.99e-04]
Epoch 54 TRAIN:  90%|████████▉ | 122/136 [00:28<00:02,  4.99it/s, epoch=54, grad_norm=239.793, loss=391.1036, lr=5.99e-04]
Epoch 54 TRAIN:  90%|████████▉ | 122/136 [00:28<00:02,  4.99it/s, epoch=54, grad_norm=145.082, loss=278.1037, lr=5.99e-04]
Epoch 54 TRAIN:  92%|█████████▏| 125/136 [00:28<00:02,  4.95it/s, epoch=54, grad_norm=145.082, loss=278.1037, lr=5.99e-04]
Epoch 54 TRAIN:  92%|█████████▏| 125/136 [00:28<00:02,  4.95it/s, epoch=54, grad_norm=162.190, loss=318.0589, lr=5.99e-04]
Epoch 54 TRAIN:  92%|█████████▏| 125/136 [00:28<00:02,  4.95it/s, epoch=54, grad_norm=313.122, loss=336.3604, lr=5.99e-04]
Epoch 54 TRAIN:  92%|█████████▏| 125/136 [00:28<00:02,  4.95it/s, epoch=54, grad_norm=202.429, loss=305.5862, lr=5.99e-04]
Epoch 54 TRAIN:  92%|█████████▏| 125/136 [00:28<00:02,  4.95it/s, epoch=54, grad_norm=341.204, loss=414.3481, lr=5.99e-04]
Epoch 54 TRAIN:  95%|█████████▍| 129/136 [00:28<00:01,  5.52it/s, epoch=54, grad_norm=341.204, loss=414.3481, lr=5.99e-04]
Epoch 54 TRAIN:  95%|█████████▍| 129/136 [00:29<00:01,  5.52it/s, epoch=54, grad_norm=199.024, loss=290.9600, lr=6.00e-04]
Epoch 54 TRAIN:  95%|█████████▍| 129/136 [00:29<00:01,  5.52it/s, epoch=54, grad_norm=260.269, loss=345.7497, lr=6.00e-04]
Epoch 54 TRAIN:  95%|█████████▍| 129/136 [00:29<00:01,  5.52it/s, epoch=54, grad_norm=166.709, loss=340.9148, lr=6.00e-04]
Epoch 54 TRAIN:  97%|█████████▋| 132/136 [00:29<00:00,  5.53it/s, epoch=54, grad_norm=166.709, loss=340.9148, lr=6.00e-04]
Epoch 54 TRAIN:  97%|█████████▋| 132/136 [00:29<00:00,  5.53it/s, epoch=54, grad_norm=175.159, loss=360.6327, lr=6.00e-04]
Epoch 54 TRAIN:  97%|█████████▋| 132/136 [00:29<00:00,  5.53it/s, epoch=54, grad_norm=215.294, loss=259.2562, lr=6.00e-04]
Epoch 54 TRAIN:  97%|█████████▋| 132/136 [00:30<00:00,  5.53it/s, epoch=54, grad_norm=187.235, loss=400.5583, lr=6.00e-04]
Epoch 54 TRAIN:  99%|█████████▉| 135/136 [00:30<00:00,  5.10it/s, epoch=54, grad_norm=187.235, loss=400.5583, lr=6.00e-04]
Epoch 54 TRAIN:  99%|█████████▉| 135/136 [00:30<00:00,  5.10it/s, epoch=54, grad_norm=188.777, loss=311.4100, lr=6.00e-04]
Epoch 54 TRAIN:  99%|█████████▉| 135/136 [00:30<00:00,  5.10it/s, epoch=54, grad_norm=197.722, loss=383.8611, lr=6.00e-04]
Epoch 54 TRAIN: 137it [00:30,  4.47it/s, epoch=54, grad_norm=197.722, loss=383.8611, lr=6.00e-04]                         

CV:   0%|          | 0/6 [00:00<?, ?it/s]/home/rookie/Documents/batuka/projects/wenet/wenet/wenet/utils/train_utils.py:689: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  amp_autocast(enabled=dtype is not None,

CV:  17%|█▋        | 1/6 [00:00<00:03,  1.32it/s]
                                                 
2026-04-16 20:19:47,381 INFO Epoch 54 Step 7501 CV info lr 6.0008e-04 cv_loss 462.7242709073153 rank 0 acc 0.359381303191185
2026-04-16 20:19:47,383 INFO [Rank 0] Checkpoint: save to checkpoint exp/conformer_mongolian/epoch_54.pt
2026-04-16 20:19:47,502 INFO Epoch 55 Step 7501 TRAIN info lr 6.0008e-04 rank 0
2026-04-16 20:19:47,516 INFO using accumulate grad, new batch size is 1 times larger than before

Epoch 55 TRAIN:   0%|          | 0/137 [00:00<?, ?it/s]
Epoch 55 TRAIN:   0%|          | 0/137 [00:03<?, ?it/s, epoch=55, grad_norm=390.086, loss=354.6140, lr=6.00e-04]
Epoch 55 TRAIN:   1%|          | 1/137 [00:03<08:06,  3.58s/it, epoch=55, grad_norm=390.086, loss=354.6140, lr=6.00e-04]
Epoch 55 TRAIN:   1%|          | 1/137 [00:04<08:06,  3.58s/it, epoch=55, grad_norm=306.127, loss=277.9958, lr=6.00e-04]
Epoch 55 TRAIN:   1%|▏         | 2/137 [00:04<04:12,  1.87s/it, epoch=55, grad_norm=306.127, loss=277.9958, lr=6.00e-04]
Epoch 55 TRAIN:   1%|▏         | 2/137 [00:04<04:12,  1.87s/it, epoch=55, grad_norm=533.411, loss=403.9090, lr=6.00e-04]
Epoch 55 TRAIN:   1%|▏         | 2/137 [00:04<04:12,  1.87s/it, epoch=55, grad_norm=224.691, loss=258.2906, lr=6.00e-04]
Epoch 55 TRAIN:   1%|▏         | 2/137 [00:04<04:12,  1.87s/it, epoch=55, grad_norm=351.010, loss=330.0788, lr=6.00e-04]
Epoch 55 TRAIN:   1%|▏         | 2/137 [00:04<04:12,  1.87s/it, epoch=55, grad_norm=217.004, loss=294.4457, lr=6.01e-04]
Epoch 55 TRAIN:   1%|▏         | 2/137 [00:04<04:12,  1.87s/it, epoch=55, grad_norm=336.067, loss=261.7303, lr=6.01e-04]
Epoch 55 TRAIN:   1%|▏         | 2/137 [00:04<04:12,  1.87s/it, epoch=55, grad_norm=179.723, loss=316.1320, lr=6.01e-04]
Epoch 55 TRAIN:   1%|▏         | 2/137 [00:04<04:12,  1.87s/it, epoch=55, grad_norm=127.484, loss=308.1716, lr=6.01e-04]
Epoch 55 TRAIN:   7%|▋         | 9/137 [00:04<00:44,  2.89it/s, epoch=55, grad_norm=127.484, loss=308.1716, lr=6.01e-04]
Epoch 55 TRAIN:   7%|▋         | 9/137 [00:05<00:44,  2.89it/s, epoch=55, grad_norm=94.035, loss=295.3897, lr=6.01e-04] 
Epoch 55 TRAIN:   7%|▋         | 9/137 [00:05<00:44,  2.89it/s, epoch=55, grad_norm=92.889, loss=306.8572, lr=6.01e-04]
Epoch 55 TRAIN:   7%|▋         | 9/137 [00:05<00:44,  2.89it/s, epoch=55, grad_norm=94.157, loss=294.9907, lr=6.01e-04]
Epoch 55 TRAIN:   7%|▋         | 9/137 [00:05<00:44,  2.89it/s, epoch=55, grad_norm=122.673, loss=316.0046, lr=6.01e-04]
Epoch 55 TRAIN:   9%|▉         | 13/137 [00:05<00:33,  3.70it/s, epoch=55, grad_norm=122.673, loss=316.0046, lr=6.01e-04]
Epoch 55 TRAIN:   9%|▉         | 13/137 [00:05<00:33,  3.70it/s, epoch=55, grad_norm=164.785, loss=275.2717, lr=6.01e-04]
Epoch 55 TRAIN:   9%|▉         | 13/137 [00:05<00:33,  3.70it/s, epoch=55, grad_norm=152.263, loss=290.7583, lr=6.01e-04]
Epoch 55 TRAIN:   9%|▉         | 13/137 [00:05<00:33,  3.70it/s, epoch=55, grad_norm=102.393, loss=278.6369, lr=6.01e-04]
Epoch 55 TRAIN:   9%|▉         | 13/137 [00:06<00:33,  3.70it/s, epoch=55, grad_norm=93.495, loss=303.6485, lr=6.01e-04] 
Epoch 55 TRAIN:  12%|█▏        | 17/137 [00:06<00:28,  4.19it/s, epoch=55, grad_norm=93.495, loss=303.6485, lr=6.01e-04]
Epoch 55 TRAIN:  12%|█▏        | 17/137 [00:06<00:28,  4.19it/s, epoch=55, grad_norm=76.636, loss=298.2081, lr=6.02e-04]
Epoch 55 TRAIN:  12%|█▏        | 17/137 [00:06<00:28,  4.19it/s, epoch=55, grad_norm=91.733, loss=273.6526, lr=6.02e-04]
Epoch 55 TRAIN:  12%|█▏        | 17/137 [00:06<00:28,  4.19it/s, epoch=55, grad_norm=131.735, loss=298.7935, lr=6.02e-04]
Epoch 55 TRAIN:  15%|█▍        | 20/137 [00:06<00:27,  4.32it/s, epoch=55, grad_norm=131.735, loss=298.7935, lr=6.02e-04]
Epoch 55 TRAIN:  15%|█▍        | 20/137 [00:07<00:27,  4.32it/s, epoch=55, grad_norm=191.237, loss=332.7238, lr=6.02e-04]
Epoch 55 TRAIN:  15%|█▍        | 20/137 [00:07<00:27,  4.32it/s, epoch=55, grad_norm=123.252, loss=288.5358, lr=6.02e-04]
Epoch 55 TRAIN:  15%|█▍        | 20/137 [00:07<00:27,  4.32it/s, epoch=55, grad_norm=158.025, loss=310.9135, lr=6.02e-04]
Epoch 55 TRAIN:  17%|█▋        | 23/137 [00:07<00:24,  4.66it/s, epoch=55, grad_norm=158.025, loss=310.9135, lr=6.02e-04]
Epoch 55 TRAIN:  17%|█▋        | 23/137 [00:07<00:24,  4.66it/s, epoch=55, grad_norm=185.833, loss=315.5710, lr=6.02e-04]
Epoch 55 TRAIN:  17%|█▋        | 23/137 [00:07<00:24,  4.66it/s, epoch=55, grad_norm=98.471, loss=293.9434, lr=6.02e-04] 
Epoch 55 TRAIN:  17%|█▋        | 23/137 [00:07<00:24,  4.66it/s, epoch=55, grad_norm=152.626, loss=355.2717, lr=6.02e-04]
Epoch 55 TRAIN:  17%|█▋        | 23/137 [00:08<00:24,  4.66it/s, epoch=55, grad_norm=166.601, loss=327.8144, lr=6.02e-04]
Epoch 55 TRAIN:  20%|█▉        | 27/137 [00:08<00:21,  5.09it/s, epoch=55, grad_norm=166.601, loss=327.8144, lr=6.02e-04]
Epoch 55 TRAIN:  20%|█▉        | 27/137 [00:08<00:21,  5.09it/s, epoch=55, grad_norm=109.001, loss=314.9277, lr=6.02e-04]
Epoch 55 TRAIN:  20%|█▉        | 27/137 [00:08<00:21,  5.09it/s, epoch=55, grad_norm=86.243, loss=336.3463, lr=6.02e-04] 
Epoch 55 TRAIN:  20%|█▉        | 27/137 [00:08<00:21,  5.09it/s, epoch=55, grad_norm=128.387, loss=292.0905, lr=6.02e-04]
Epoch 55 TRAIN:  22%|██▏       | 30/137 [00:08<00:22,  4.84it/s, epoch=55, grad_norm=128.387, loss=292.0905, lr=6.02e-04]
Epoch 55 TRAIN:  22%|██▏       | 30/137 [00:08<00:22,  4.84it/s, epoch=55, grad_norm=127.241, loss=335.7755, lr=6.03e-04]
Epoch 55 TRAIN:  22%|██▏       | 30/137 [00:09<00:22,  4.84it/s, epoch=55, grad_norm=151.475, loss=278.6725, lr=6.03e-04]
Epoch 55 TRAIN:  22%|██▏       | 30/137 [00:09<00:22,  4.84it/s, epoch=55, grad_norm=155.043, loss=301.2053, lr=6.03e-04]
Epoch 55 TRAIN:  24%|██▍       | 33/137 [00:09<00:20,  5.07it/s, epoch=55, grad_norm=155.043, loss=301.2053, lr=6.03e-04]
Epoch 55 TRAIN:  24%|██▍       | 33/137 [00:09<00:20,  5.07it/s, epoch=55, grad_norm=109.653, loss=329.3967, lr=6.03e-04]
Epoch 55 TRAIN:  24%|██▍       | 33/137 [00:09<00:20,  5.07it/s, epoch=55, grad_norm=131.599, loss=306.4700, lr=6.03e-04]
Epoch 55 TRAIN:  24%|██▍       | 33/137 [00:09<00:20,  5.07it/s, epoch=55, grad_norm=115.717, loss=282.4657, lr=6.03e-04]
Epoch 55 TRAIN:  26%|██▋       | 36/137 [00:09<00:20,  4.98it/s, epoch=55, grad_norm=115.717, loss=282.4657, lr=6.03e-04]
Epoch 55 TRAIN:  26%|██▋       | 36/137 [00:10<00:20,  4.98it/s, epoch=55, grad_norm=111.207, loss=249.3397, lr=6.03e-04]
Epoch 55 TRAIN:  26%|██▋       | 36/137 [00:10<00:20,  4.98it/s, epoch=55, grad_norm=137.020, loss=319.3194, lr=6.03e-04]
Epoch 55 TRAIN:  26%|██▋       | 36/137 [00:10<00:20,  4.98it/s, epoch=55, grad_norm=100.307, loss=282.5131, lr=6.03e-04]
Epoch 55 TRAIN:  28%|██▊       | 39/137 [00:10<00:19,  5.04it/s, epoch=55, grad_norm=100.307, loss=282.5131, lr=6.03e-04]
Epoch 55 TRAIN:  28%|██▊       | 39/137 [00:10<00:19,  5.04it/s, epoch=55, grad_norm=141.963, loss=292.6556, lr=6.03e-04]
Epoch 55 TRAIN:  28%|██▊       | 39/137 [00:10<00:19,  5.04it/s, epoch=55, grad_norm=86.513, loss=262.1957, lr=6.03e-04] 
Epoch 55 TRAIN:  28%|██▊       | 39/137 [00:11<00:19,  5.04it/s, epoch=55, grad_norm=115.096, loss=300.6687, lr=6.03e-04]
Epoch 55 TRAIN:  31%|███       | 42/137 [00:11<00:19,  4.88it/s, epoch=55, grad_norm=115.096, loss=300.6687, lr=6.03e-04]
Epoch 55 TRAIN:  31%|███       | 42/137 [00:11<00:19,  4.88it/s, epoch=55, grad_norm=138.814, loss=320.0768, lr=6.04e-04]
Epoch 55 TRAIN:  31%|███       | 42/137 [00:11<00:19,  4.88it/s, epoch=55, grad_norm=210.556, loss=362.6420, lr=6.04e-04]
Epoch 55 TRAIN:  31%|███       | 42/137 [00:11<00:19,  4.88it/s, epoch=55, grad_norm=114.887, loss=268.7411, lr=6.04e-04]
Epoch 55 TRAIN:  33%|███▎      | 45/137 [00:11<00:18,  4.88it/s, epoch=55, grad_norm=114.887, loss=268.7411, lr=6.04e-04]
Epoch 55 TRAIN:  33%|███▎      | 45/137 [00:11<00:18,  4.88it/s, epoch=55, grad_norm=116.363, loss=341.2575, lr=6.04e-04]
Epoch 55 TRAIN:  33%|███▎      | 45/137 [00:12<00:18,  4.88it/s, epoch=55, grad_norm=132.634, loss=238.8493, lr=6.04e-04]
Epoch 55 TRAIN:  33%|███▎      | 45/137 [00:12<00:18,  4.88it/s, epoch=55, grad_norm=99.440, loss=313.3242, lr=6.04e-04] 
Epoch 55 TRAIN:  35%|███▌      | 48/137 [00:12<00:18,  4.72it/s, epoch=55, grad_norm=99.440, loss=313.3242, lr=6.04e-04]
Epoch 55 TRAIN:  35%|███▌      | 48/137 [00:12<00:18,  4.72it/s, epoch=55, grad_norm=97.773, loss=303.7092, lr=6.04e-04]
Epoch 55 TRAIN:  35%|███▌      | 48/137 [00:12<00:18,  4.72it/s, epoch=55, grad_norm=104.235, loss=263.2406, lr=6.04e-04]
Epoch 55 TRAIN:  35%|███▌      | 48/137 [00:12<00:18,  4.72it/s, epoch=55, grad_norm=164.216, loss=226.5724, lr=6.04e-04]
Epoch 55 TRAIN:  35%|███▌      | 48/137 [00:13<00:18,  4.72it/s, epoch=55, grad_norm=100.455, loss=321.9350, lr=6.04e-04]
Epoch 55 TRAIN:  38%|███▊      | 52/137 [00:13<00:16,  5.09it/s, epoch=55, grad_norm=100.455, loss=321.9350, lr=6.04e-04]
Epoch 55 TRAIN:  38%|███▊      | 52/137 [00:13<00:16,  5.09it/s, epoch=55, grad_norm=124.012, loss=321.7567, lr=6.04e-04]
Epoch 55 TRAIN:  38%|███▊      | 52/137 [00:13<00:16,  5.09it/s, epoch=55, grad_norm=106.027, loss=255.0888, lr=6.04e-04]
Epoch 55 TRAIN:  38%|███▊      | 52/137 [00:13<00:16,  5.09it/s, epoch=55, grad_norm=100.369, loss=305.7745, lr=6.04e-04]
Epoch 55 TRAIN:  40%|████      | 55/137 [00:13<00:16,  4.91it/s, epoch=55, grad_norm=100.369, loss=305.7745, lr=6.04e-04]
Epoch 55 TRAIN:  40%|████      | 55/137 [00:14<00:16,  4.91it/s, epoch=55, grad_norm=135.028, loss=290.8704, lr=6.05e-04]
Epoch 55 TRAIN:  40%|████      | 55/137 [00:14<00:16,  4.91it/s, epoch=55, grad_norm=100.545, loss=320.8911, lr=6.05e-04]
Epoch 55 TRAIN:  40%|████      | 55/137 [00:14<00:16,  4.91it/s, epoch=55, grad_norm=139.580, loss=317.0159, lr=6.05e-04]
Epoch 55 TRAIN:  42%|████▏     | 58/137 [00:14<00:16,  4.80it/s, epoch=55, grad_norm=139.580, loss=317.0159, lr=6.05e-04]
Epoch 55 TRAIN:  42%|████▏     | 58/137 [00:14<00:16,  4.80it/s, epoch=55, grad_norm=121.693, loss=302.1768, lr=6.05e-04]
Epoch 55 TRAIN:  42%|████▏     | 58/137 [00:14<00:16,  4.80it/s, epoch=55, grad_norm=106.724, loss=311.9704, lr=6.05e-04]
Epoch 55 TRAIN:  42%|████▏     | 58/137 [00:15<00:16,  4.80it/s, epoch=55, grad_norm=137.637, loss=232.9641, lr=6.05e-04]
Epoch 55 TRAIN:  45%|████▍     | 61/137 [00:15<00:15,  4.83it/s, epoch=55, grad_norm=137.637, loss=232.9641, lr=6.05e-04]
Epoch 55 TRAIN:  45%|████▍     | 61/137 [00:15<00:15,  4.83it/s, epoch=55, grad_norm=145.484, loss=280.9934, lr=6.05e-04]
Epoch 55 TRAIN:  45%|████▍     | 61/137 [00:15<00:15,  4.83it/s, epoch=55, grad_norm=120.077, loss=291.0672, lr=6.05e-04]
Epoch 55 TRAIN:  45%|████▍     | 61/137 [00:15<00:15,  4.83it/s, epoch=55, grad_norm=81.491, loss=270.4266, lr=6.05e-04] 
Epoch 55 TRAIN:  47%|████▋     | 64/137 [00:15<00:15,  4.83it/s, epoch=55, grad_norm=81.491, loss=270.4266, lr=6.05e-04]
Epoch 55 TRAIN:  47%|████▋     | 64/137 [00:15<00:15,  4.83it/s, epoch=55, grad_norm=200.372, loss=350.1458, lr=6.05e-04]
Epoch 55 TRAIN:  47%|████▋     | 64/137 [00:16<00:15,  4.83it/s, epoch=55, grad_norm=171.252, loss=292.7762, lr=6.05e-04]
Epoch 55 TRAIN:  47%|████▋     | 64/137 [00:16<00:15,  4.83it/s, epoch=55, grad_norm=231.120, loss=209.1770, lr=6.05e-04]
Epoch 55 TRAIN:  49%|████▉     | 67/137 [00:16<00:14,  4.79it/s, epoch=55, grad_norm=231.120, loss=209.1770, lr=6.05e-04]
Epoch 55 TRAIN:  49%|████▉     | 67/137 [00:16<00:14,  4.79it/s, epoch=55, grad_norm=120.562, loss=270.8279, lr=6.06e-04]
Epoch 55 TRAIN:  49%|████▉     | 67/137 [00:16<00:14,  4.79it/s, epoch=55, grad_norm=157.126, loss=302.6202, lr=6.06e-04]
Epoch 55 TRAIN:  49%|████▉     | 67/137 [00:16<00:14,  4.79it/s, epoch=55, grad_norm=141.341, loss=287.1629, lr=6.06e-04]
Epoch 55 TRAIN:  51%|█████     | 70/137 [00:16<00:13,  4.87it/s, epoch=55, grad_norm=141.341, loss=287.1629, lr=6.06e-04]
Epoch 55 TRAIN:  51%|█████     | 70/137 [00:17<00:13,  4.87it/s, epoch=55, grad_norm=119.207, loss=306.9413, lr=6.06e-04]
Epoch 55 TRAIN:  51%|█████     | 70/137 [00:17<00:13,  4.87it/s, epoch=55, grad_norm=263.384, loss=358.0925, lr=6.06e-04]
Epoch 55 TRAIN:  51%|█████     | 70/137 [00:17<00:13,  4.87it/s, epoch=55, grad_norm=147.852, loss=272.4899, lr=6.06e-04]
Epoch 55 TRAIN:  53%|█████▎    | 73/137 [00:17<00:13,  4.86it/s, epoch=55, grad_norm=147.852, loss=272.4899, lr=6.06e-04]
Epoch 55 TRAIN:  53%|█████▎    | 73/137 [00:17<00:13,  4.86it/s, epoch=55, grad_norm=125.158, loss=287.3179, lr=6.06e-04]
Epoch 55 TRAIN:  53%|█████▎    | 73/137 [00:17<00:13,  4.86it/s, epoch=55, grad_norm=150.671, loss=337.9227, lr=6.06e-04]
Epoch 55 TRAIN:  53%|█████▎    | 73/137 [00:18<00:13,  4.86it/s, epoch=55, grad_norm=116.764, loss=275.1541, lr=6.06e-04]
Epoch 55 TRAIN:  55%|█████▌    | 76/137 [00:18<00:12,  4.74it/s, epoch=55, grad_norm=116.764, loss=275.1541, lr=6.06e-04]
Epoch 55 TRAIN:  55%|█████▌    | 76/137 [00:18<00:12,  4.74it/s, epoch=55, grad_norm=111.851, loss=276.1254, lr=6.06e-04]
Epoch 55 TRAIN:  55%|█████▌    | 76/137 [00:18<00:12,  4.74it/s, epoch=55, grad_norm=295.492, loss=250.3381, lr=6.06e-04]
Epoch 55 TRAIN:  55%|█████▌    | 76/137 [00:18<00:12,  4.74it/s, epoch=55, grad_norm=122.876, loss=281.6424, lr=6.06e-04]
Epoch 55 TRAIN:  58%|█████▊    | 79/137 [00:18<00:11,  4.90it/s, epoch=55, grad_norm=122.876, loss=281.6424, lr=6.06e-04]
Epoch 55 TRAIN:  58%|█████▊    | 79/137 [00:18<00:11,  4.90it/s, epoch=55, grad_norm=146.716, loss=300.5521, lr=6.06e-04]
Epoch 55 TRAIN:  58%|█████▊    | 79/137 [00:19<00:11,  4.90it/s, epoch=55, grad_norm=158.579, loss=315.4056, lr=6.07e-04]
Epoch 55 TRAIN:  58%|█████▊    | 79/137 [00:19<00:11,  4.90it/s, epoch=55, grad_norm=154.549, loss=343.2593, lr=6.07e-04]
Epoch 55 TRAIN:  60%|█████▉    | 82/137 [00:19<00:11,  4.94it/s, epoch=55, grad_norm=154.549, loss=343.2593, lr=6.07e-04]
Epoch 55 TRAIN:  60%|█████▉    | 82/137 [00:19<00:11,  4.94it/s, epoch=55, grad_norm=92.082, loss=286.2354, lr=6.07e-04] 
Epoch 55 TRAIN:  60%|█████▉    | 82/137 [00:19<00:11,  4.94it/s, epoch=55, grad_norm=136.137, loss=228.9644, lr=6.07e-04]
Epoch 55 TRAIN:  60%|█████▉    | 82/137 [00:20<00:11,  4.94it/s, epoch=55, grad_norm=141.214, loss=279.2688, lr=6.07e-04]
Epoch 55 TRAIN:  62%|██████▏   | 85/137 [00:20<00:10,  4.81it/s, epoch=55, grad_norm=141.214, loss=279.2688, lr=6.07e-04]
Epoch 55 TRAIN:  62%|██████▏   | 85/137 [00:20<00:10,  4.81it/s, epoch=55, grad_norm=148.731, loss=284.0598, lr=6.07e-04]
Epoch 55 TRAIN:  62%|██████▏   | 85/137 [00:20<00:10,  4.81it/s, epoch=55, grad_norm=183.708, loss=233.6014, lr=6.07e-04]
Epoch 55 TRAIN:  62%|██████▏   | 85/137 [00:20<00:10,  4.81it/s, epoch=55, grad_norm=190.435, loss=300.3705, lr=6.07e-04]
Epoch 55 TRAIN:  64%|██████▍   | 88/137 [00:20<00:10,  4.86it/s, epoch=55, grad_norm=190.435, loss=300.3705, lr=6.07e-04]
Epoch 55 TRAIN:  64%|██████▍   | 88/137 [00:20<00:10,  4.86it/s, epoch=55, grad_norm=120.218, loss=310.2480, lr=6.07e-04]
Epoch 55 TRAIN:  64%|██████▍   | 88/137 [00:21<00:10,  4.86it/s, epoch=55, grad_norm=164.069, loss=288.6643, lr=6.07e-04]
Epoch 55 TRAIN:  64%|██████▍   | 88/137 [00:21<00:10,  4.86it/s, epoch=55, grad_norm=215.391, loss=314.8433, lr=6.07e-04]
Epoch 55 TRAIN:  66%|██████▋   | 91/137 [00:21<00:10,  4.60it/s, epoch=55, grad_norm=215.391, loss=314.8433, lr=6.07e-04]
Epoch 55 TRAIN:  66%|██████▋   | 91/137 [00:21<00:10,  4.60it/s, epoch=55, grad_norm=170.776, loss=314.5095, lr=6.07e-04]
Epoch 55 TRAIN:  66%|██████▋   | 91/137 [00:21<00:10,  4.60it/s, epoch=55, grad_norm=147.854, loss=247.5463, lr=6.08e-04]
Epoch 55 TRAIN:  66%|██████▋   | 91/137 [00:22<00:10,  4.60it/s, epoch=55, grad_norm=199.025, loss=248.9698, lr=6.08e-04]
Epoch 55 TRAIN:  69%|██████▊   | 94/137 [00:22<00:09,  4.61it/s, epoch=55, grad_norm=199.025, loss=248.9698, lr=6.08e-04]
Epoch 55 TRAIN:  69%|██████▊   | 94/137 [00:22<00:09,  4.61it/s, epoch=55, grad_norm=116.832, loss=232.1590, lr=6.08e-04]
Epoch 55 TRAIN:  69%|██████▊   | 94/137 [00:22<00:09,  4.61it/s, epoch=55, grad_norm=136.776, loss=292.3394, lr=6.08e-04]
Epoch 55 TRAIN:  69%|██████▊   | 94/137 [00:22<00:09,  4.61it/s, epoch=55, grad_norm=272.413, loss=391.5474, lr=6.08e-04]
Epoch 55 TRAIN:  71%|███████   | 97/137 [00:22<00:08,  4.92it/s, epoch=55, grad_norm=272.413, loss=391.5474, lr=6.08e-04]
Epoch 55 TRAIN:  71%|███████   | 97/137 [00:22<00:08,  4.92it/s, epoch=55, grad_norm=118.499, loss=268.3275, lr=6.08e-04]
Epoch 55 TRAIN:  71%|███████   | 97/137 [00:22<00:08,  4.92it/s, epoch=55, grad_norm=121.116, loss=236.9218, lr=6.08e-04]
Epoch 55 TRAIN:  71%|███████   | 97/137 [00:23<00:08,  4.92it/s, epoch=55, grad_norm=120.251, loss=284.0700, lr=6.08e-04]2026-04-16 20:20:10,595 DEBUG TRAIN | steps/sec 4.337| Batch 55/100 loss 284.070007 loss_att 237.906830 loss_ctc 391.784119 th_accuracy 0.339066 lr 6.0808e-04 grad_norm 120.251221 rank 0

Epoch 55 TRAIN:  73%|███████▎  | 100/137 [00:23<00:07,  5.06it/s, epoch=55, grad_norm=120.251, loss=284.0700, lr=6.08e-04]
Epoch 55 TRAIN:  73%|███████▎  | 100/137 [00:23<00:07,  5.06it/s, epoch=55, grad_norm=116.066, loss=262.5636, lr=6.08e-04]
Epoch 55 TRAIN:  73%|███████▎  | 100/137 [00:23<00:07,  5.06it/s, epoch=55, grad_norm=146.806, loss=243.1246, lr=6.08e-04]
Epoch 55 TRAIN:  73%|███████▎  | 100/137 [00:23<00:07,  5.06it/s, epoch=55, grad_norm=95.727, loss=242.7793, lr=6.08e-04] 
Epoch 55 TRAIN:  75%|███████▌  | 103/137 [00:23<00:06,  4.99it/s, epoch=55, grad_norm=95.727, loss=242.7793, lr=6.08e-04]
Epoch 55 TRAIN:  75%|███████▌  | 103/137 [00:23<00:06,  4.99it/s, epoch=55, grad_norm=128.525, loss=263.5212, lr=6.08e-04]
Epoch 55 TRAIN:  75%|███████▌  | 103/137 [00:24<00:06,  4.99it/s, epoch=55, grad_norm=135.446, loss=294.4918, lr=6.08e-04]
Epoch 55 TRAIN:  75%|███████▌  | 103/137 [00:24<00:06,  4.99it/s, epoch=55, grad_norm=140.763, loss=301.2556, lr=6.09e-04]
Epoch 55 TRAIN:  77%|███████▋  | 106/137 [00:24<00:06,  4.97it/s, epoch=55, grad_norm=140.763, loss=301.2556, lr=6.09e-04]
Epoch 55 TRAIN:  77%|███████▋  | 106/137 [00:24<00:06,  4.97it/s, epoch=55, grad_norm=154.365, loss=306.1176, lr=6.09e-04]
Epoch 55 TRAIN:  77%|███████▋  | 106/137 [00:24<00:06,  4.97it/s, epoch=55, grad_norm=121.406, loss=262.2296, lr=6.09e-04]
Epoch 55 TRAIN:  77%|███████▋  | 106/137 [00:24<00:06,  4.97it/s, epoch=55, grad_norm=244.269, loss=386.5347, lr=6.09e-04]
Epoch 55 TRAIN:  80%|███████▉  | 109/137 [00:24<00:05,  4.87it/s, epoch=55, grad_norm=244.269, loss=386.5347, lr=6.09e-04]
Epoch 55 TRAIN:  80%|███████▉  | 109/137 [00:25<00:05,  4.87it/s, epoch=55, grad_norm=225.681, loss=227.1478, lr=6.09e-04]
Epoch 55 TRAIN:  80%|███████▉  | 109/137 [00:25<00:05,  4.87it/s, epoch=55, grad_norm=290.799, loss=147.5691, lr=6.09e-04]
Epoch 55 TRAIN:  80%|███████▉  | 109/137 [00:25<00:05,  4.87it/s, epoch=55, grad_norm=128.193, loss=293.9991, lr=6.09e-04]
Epoch 55 TRAIN:  82%|████████▏ | 112/137 [00:25<00:05,  4.83it/s, epoch=55, grad_norm=128.193, loss=293.9991, lr=6.09e-04]
Epoch 55 TRAIN:  82%|████████▏ | 112/137 [00:25<00:05,  4.83it/s, epoch=55, grad_norm=144.547, loss=274.0607, lr=6.09e-04]
Epoch 55 TRAIN:  82%|████████▏ | 112/137 [00:25<00:05,  4.83it/s, epoch=55, grad_norm=122.336, loss=271.5702, lr=6.09e-04]
Epoch 55 TRAIN:  82%|████████▏ | 112/137 [00:26<00:05,  4.83it/s, epoch=55, grad_norm=124.511, loss=256.0781, lr=6.09e-04]
Epoch 55 TRAIN:  84%|████████▍ | 115/137 [00:26<00:04,  4.88it/s, epoch=55, grad_norm=124.511, loss=256.0781, lr=6.09e-04]
Epoch 55 TRAIN:  84%|████████▍ | 115/137 [00:26<00:04,  4.88it/s, epoch=55, grad_norm=220.164, loss=386.6006, lr=6.09e-04]
Epoch 55 TRAIN:  84%|████████▍ | 115/137 [00:26<00:04,  4.88it/s, epoch=55, grad_norm=261.919, loss=329.7090, lr=6.09e-04]
Epoch 55 TRAIN:  84%|████████▍ | 115/137 [00:26<00:04,  4.88it/s, epoch=55, grad_norm=178.581, loss=240.1461, lr=6.10e-04]
Epoch 55 TRAIN:  86%|████████▌ | 118/137 [00:26<00:03,  4.85it/s, epoch=55, grad_norm=178.581, loss=240.1461, lr=6.10e-04]
Epoch 55 TRAIN:  86%|████████▌ | 118/137 [00:26<00:03,  4.85it/s, epoch=55, grad_norm=225.138, loss=329.1796, lr=6.10e-04]
Epoch 55 TRAIN:  86%|████████▌ | 118/137 [00:27<00:03,  4.85it/s, epoch=55, grad_norm=153.456, loss=295.9423, lr=6.10e-04]
Epoch 55 TRAIN:  86%|████████▌ | 118/137 [00:27<00:03,  4.85it/s, epoch=55, grad_norm=150.936, loss=308.0439, lr=6.10e-04]
Epoch 55 TRAIN:  88%|████████▊ | 121/137 [00:27<00:03,  4.91it/s, epoch=55, grad_norm=150.936, loss=308.0439, lr=6.10e-04]
Epoch 55 TRAIN:  88%|████████▊ | 121/137 [00:27<00:03,  4.91it/s, epoch=55, grad_norm=132.377, loss=285.0882, lr=6.10e-04]
Epoch 55 TRAIN:  88%|████████▊ | 121/137 [00:27<00:03,  4.91it/s, epoch=55, grad_norm=168.333, loss=309.1521, lr=6.10e-04]
Epoch 55 TRAIN:  88%|████████▊ | 121/137 [00:28<00:03,  4.91it/s, epoch=55, grad_norm=107.137, loss=278.6512, lr=6.10e-04]
Epoch 55 TRAIN:  91%|█████████ | 124/137 [00:28<00:02,  4.57it/s, epoch=55, grad_norm=107.137, loss=278.6512, lr=6.10e-04]
Epoch 55 TRAIN:  91%|█████████ | 124/137 [00:28<00:02,  4.57it/s, epoch=55, grad_norm=136.375, loss=353.0318, lr=6.10e-04]
Epoch 55 TRAIN:  91%|█████████ | 124/137 [00:28<00:02,  4.57it/s, epoch=55, grad_norm=181.543, loss=253.6777, lr=6.10e-04]
Epoch 55 TRAIN:  91%|█████████ | 124/137 [00:28<00:02,  4.57it/s, epoch=55, grad_norm=282.535, loss=303.8102, lr=6.10e-04]
Epoch 55 TRAIN:  93%|█████████▎| 127/137 [00:28<00:02,  4.80it/s, epoch=55, grad_norm=282.535, loss=303.8102, lr=6.10e-04]
Epoch 55 TRAIN:  93%|█████████▎| 127/137 [00:28<00:02,  4.80it/s, epoch=55, grad_norm=162.602, loss=320.5998, lr=6.10e-04]
Epoch 55 TRAIN:  93%|█████████▎| 127/137 [00:29<00:02,  4.80it/s, epoch=55, grad_norm=165.977, loss=307.1280, lr=6.10e-04]
Epoch 55 TRAIN:  93%|█████████▎| 127/137 [00:29<00:02,  4.80it/s, epoch=55, grad_norm=141.641, loss=297.8999, lr=6.10e-04]
Epoch 55 TRAIN:  95%|█████████▍| 130/137 [00:29<00:01,  4.68it/s, epoch=55, grad_norm=141.641, loss=297.8999, lr=6.10e-04]
Epoch 55 TRAIN:  95%|█████████▍| 130/137 [00:29<00:01,  4.68it/s, epoch=55, grad_norm=191.066, loss=392.5289, lr=6.11e-04]
Epoch 55 TRAIN:  95%|█████████▍| 130/137 [00:29<00:01,  4.68it/s, epoch=55, grad_norm=156.472, loss=243.8165, lr=6.11e-04]
Epoch 55 TRAIN:  95%|█████████▍| 130/137 [00:30<00:01,  4.68it/s, epoch=55, grad_norm=256.425, loss=349.9622, lr=6.11e-04]
Epoch 55 TRAIN:  97%|█████████▋| 133/137 [00:30<00:00,  4.74it/s, epoch=55, grad_norm=256.425, loss=349.9622, lr=6.11e-04]
Epoch 55 TRAIN:  97%|█████████▋| 133/137 [00:30<00:00,  4.74it/s, epoch=55, grad_norm=298.109, loss=350.0945, lr=6.11e-04]
Epoch 55 TRAIN:  97%|█████████▋| 133/137 [00:30<00:00,  4.74it/s, epoch=55, grad_norm=291.392, loss=358.1468, lr=6.11e-04]
Epoch 55 TRAIN:  97%|█████████▋| 133/137 [00:30<00:00,  4.74it/s, epoch=55, grad_norm=178.534, loss=310.6167, lr=6.11e-04]
Epoch 55 TRAIN:  99%|█████████▉| 136/137 [00:30<00:00,  4.85it/s, epoch=55, grad_norm=178.534, loss=310.6167, lr=6.11e-04]
Epoch 55 TRAIN:  99%|█████████▉| 136/137 [00:30<00:00,  4.85it/s, epoch=55, grad_norm=211.450, loss=297.5555, lr=6.11e-04]
Epoch 55 TRAIN: 100%|██████████| 137/137 [00:30<00:00,  4.45it/s, epoch=55, grad_norm=211.450, loss=297.5555, lr=6.11e-04]

CV:   0%|          | 0/6 [00:00<?, ?it/s]/home/rookie/Documents/batuka/projects/wenet/wenet/wenet/utils/train_utils.py:689: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  amp_autocast(enabled=dtype is not None,

CV:  17%|█▋        | 1/6 [00:00<00:03,  1.31it/s]
                                                 
2026-04-16 20:20:19,185 INFO Epoch 55 Step 7638 CV info lr 6.1104e-04 cv_loss 463.03382734818894 rank 0 acc 0.3454809288183848
2026-04-16 20:20:19,187 INFO [Rank 0] Checkpoint: save to checkpoint exp/conformer_mongolian/epoch_55.pt
2026-04-16 20:20:19,306 INFO Epoch 56 Step 7638 TRAIN info lr 6.1104e-04 rank 0
2026-04-16 20:20:19,319 INFO using accumulate grad, new batch size is 1 times larger than before

Epoch 56 TRAIN:   0%|          | 0/137 [00:00<?, ?it/s]
Epoch 56 TRAIN:   0%|          | 0/137 [00:04<?, ?it/s, epoch=56, grad_norm=325.846, loss=318.0977, lr=6.11e-04]
Epoch 56 TRAIN:   1%|          | 1/137 [00:04<09:12,  4.06s/it, epoch=56, grad_norm=325.846, loss=318.0977, lr=6.11e-04]
Epoch 56 TRAIN:   1%|          | 1/137 [00:04<09:12,  4.06s/it, epoch=56, grad_norm=287.814, loss=252.2196, lr=6.11e-04]
Epoch 56 TRAIN:   1%|          | 1/137 [00:04<09:12,  4.06s/it, epoch=56, grad_norm=458.677, loss=397.9993, lr=6.11e-04]
Epoch 56 TRAIN:   1%|          | 1/137 [00:04<09:12,  4.06s/it, epoch=56, grad_norm=373.930, loss=337.3810, lr=6.11e-04]
Epoch 56 TRAIN:   1%|          | 1/137 [00:04<09:12,  4.06s/it, epoch=56, grad_norm=495.538, loss=411.6838, lr=6.11e-04]
Epoch 56 TRAIN:   1%|          | 1/137 [00:04<09:12,  4.06s/it, epoch=56, grad_norm=271.091, loss=319.9434, lr=6.12e-04]
Epoch 56 TRAIN:   1%|          | 1/137 [00:04<09:12,  4.06s/it, epoch=56, grad_norm=282.547, loss=302.1971, lr=6.12e-04]
Epoch 56 TRAIN:   1%|          | 1/137 [00:04<09:12,  4.06s/it, epoch=56, grad_norm=267.934, loss=379.2259, lr=6.12e-04]
Epoch 56 TRAIN:   1%|          | 1/137 [00:04<09:12,  4.06s/it, epoch=56, grad_norm=101.327, loss=287.0307, lr=6.12e-04]
Epoch 56 TRAIN:   7%|▋         | 9/137 [00:04<00:51,  2.50it/s, epoch=56, grad_norm=101.327, loss=287.0307, lr=6.12e-04]
Epoch 56 TRAIN:   7%|▋         | 9/137 [00:04<00:51,  2.50it/s, epoch=56, grad_norm=143.342, loss=290.7786, lr=6.12e-04]
Epoch 56 TRAIN:   7%|▋         | 9/137 [00:05<00:51,  2.50it/s, epoch=56, grad_norm=166.701, loss=261.6244, lr=6.12e-04]
Epoch 56 TRAIN:   7%|▋         | 9/137 [00:05<00:51,  2.50it/s, epoch=56, grad_norm=161.589, loss=272.3081, lr=6.12e-04]
Epoch 56 TRAIN:   7%|▋         | 9/137 [00:05<00:51,  2.50it/s, epoch=56, grad_norm=152.630, loss=271.5192, lr=6.12e-04]
Epoch 56 TRAIN:   9%|▉         | 13/137 [00:05<00:37,  3.26it/s, epoch=56, grad_norm=152.630, loss=271.5192, lr=6.12e-04]
Epoch 56 TRAIN:   9%|▉         | 13/137 [00:05<00:37,  3.26it/s, epoch=56, grad_norm=113.911, loss=307.4285, lr=6.12e-04]
Epoch 56 TRAIN:   9%|▉         | 13/137 [00:05<00:37,  3.26it/s, epoch=56, grad_norm=99.473, loss=296.2902, lr=6.12e-04] 
Epoch 56 TRAIN:   9%|▉         | 13/137 [00:05<00:37,  3.26it/s, epoch=56, grad_norm=89.692, loss=301.7316, lr=6.12e-04]
Epoch 56 TRAIN:   9%|▉         | 13/137 [00:05<00:37,  3.26it/s, epoch=56, grad_norm=106.582, loss=282.9327, lr=6.12e-04]
Epoch 56 TRAIN:  12%|█▏        | 17/137 [00:05<00:30,  3.96it/s, epoch=56, grad_norm=106.582, loss=282.9327, lr=6.12e-04]
Epoch 56 TRAIN:  12%|█▏        | 17/137 [00:06<00:30,  3.96it/s, epoch=56, grad_norm=150.552, loss=336.9193, lr=6.12e-04]
Epoch 56 TRAIN:  12%|█▏        | 17/137 [00:06<00:30,  3.96it/s, epoch=56, grad_norm=99.339, loss=335.7555, lr=6.13e-04] 
Epoch 56 TRAIN:  12%|█▏        | 17/137 [00:06<00:30,  3.96it/s, epoch=56, grad_norm=170.588, loss=302.4868, lr=6.13e-04]
Epoch 56 TRAIN:  12%|█▏        | 17/137 [00:06<00:30,  3.96it/s, epoch=56, grad_norm=145.915, loss=283.1808, lr=6.13e-04]
Epoch 56 TRAIN:  15%|█▌        | 21/137 [00:06<00:27,  4.29it/s, epoch=56, grad_norm=145.915, loss=283.1808, lr=6.13e-04]
Epoch 56 TRAIN:  15%|█▌        | 21/137 [00:06<00:27,  4.29it/s, epoch=56, grad_norm=117.535, loss=290.0396, lr=6.13e-04]
Epoch 56 TRAIN:  15%|█▌        | 21/137 [00:07<00:27,  4.29it/s, epoch=56, grad_norm=164.578, loss=356.9118, lr=6.13e-04]
Epoch 56 TRAIN:  15%|█▌        | 21/137 [00:07<00:27,  4.29it/s, epoch=56, grad_norm=122.004, loss=309.4032, lr=6.13e-04]
Epoch 56 TRAIN:  15%|█▌        | 21/137 [00:07<00:27,  4.29it/s, epoch=56, grad_norm=117.371, loss=331.5613, lr=6.13e-04]
Epoch 56 TRAIN:  18%|█▊        | 25/137 [00:07<00:24,  4.55it/s, epoch=56, grad_norm=117.371, loss=331.5613, lr=6.13e-04]
Epoch 56 TRAIN:  18%|█▊        | 25/137 [00:07<00:24,  4.55it/s, epoch=56, grad_norm=72.841, loss=313.1897, lr=6.13e-04] 
Epoch 56 TRAIN:  18%|█▊        | 25/137 [00:07<00:24,  4.55it/s, epoch=56, grad_norm=144.009, loss=293.0752, lr=6.13e-04]
Epoch 56 TRAIN:  18%|█▊        | 25/137 [00:08<00:24,  4.55it/s, epoch=56, grad_norm=124.617, loss=278.9739, lr=6.13e-04]
Epoch 56 TRAIN:  20%|██        | 28/137 [00:08<00:23,  4.66it/s, epoch=56, grad_norm=124.617, loss=278.9739, lr=6.13e-04]
Epoch 56 TRAIN:  20%|██        | 28/137 [00:08<00:23,  4.66it/s, epoch=56, grad_norm=113.460, loss=307.1781, lr=6.13e-04]
Epoch 56 TRAIN:  20%|██        | 28/137 [00:08<00:23,  4.66it/s, epoch=56, grad_norm=80.455, loss=302.9122, lr=6.13e-04] 
Epoch 56 TRAIN:  20%|██        | 28/137 [00:08<00:23,  4.66it/s, epoch=56, grad_norm=112.220, loss=313.0594, lr=6.14e-04]
Epoch 56 TRAIN:  23%|██▎       | 31/137 [00:08<00:22,  4.73it/s, epoch=56, grad_norm=112.220, loss=313.0594, lr=6.14e-04]
Epoch 56 TRAIN:  23%|██▎       | 31/137 [00:08<00:22,  4.73it/s, epoch=56, grad_norm=116.543, loss=292.8038, lr=6.14e-04]
Epoch 56 TRAIN:  23%|██▎       | 31/137 [00:09<00:22,  4.73it/s, epoch=56, grad_norm=157.320, loss=238.3642, lr=6.14e-04]
Epoch 56 TRAIN:  23%|██▎       | 31/137 [00:09<00:22,  4.73it/s, epoch=56, grad_norm=121.245, loss=295.9875, lr=6.14e-04]
Epoch 56 TRAIN:  25%|██▍       | 34/137 [00:09<00:20,  5.00it/s, epoch=56, grad_norm=121.245, loss=295.9875, lr=6.14e-04]
Epoch 56 TRAIN:  25%|██▍       | 34/137 [00:09<00:20,  5.00it/s, epoch=56, grad_norm=148.319, loss=348.7278, lr=6.14e-04]
Epoch 56 TRAIN:  25%|██▍       | 34/137 [00:09<00:20,  5.00it/s, epoch=56, grad_norm=114.154, loss=297.4414, lr=6.14e-04]
Epoch 56 TRAIN:  25%|██▍       | 34/137 [00:09<00:20,  5.00it/s, epoch=56, grad_norm=159.424, loss=314.3381, lr=6.14e-04]
Epoch 56 TRAIN:  27%|██▋       | 37/137 [00:09<00:20,  4.84it/s, epoch=56, grad_norm=159.424, loss=314.3381, lr=6.14e-04]
Epoch 56 TRAIN:  27%|██▋       | 37/137 [00:10<00:20,  4.84it/s, epoch=56, grad_norm=91.003, loss=318.4592, lr=6.14e-04] 
Epoch 56 TRAIN:  27%|██▋       | 37/137 [00:10<00:20,  4.84it/s, epoch=56, grad_norm=105.309, loss=284.2546, lr=6.14e-04]
Epoch 56 TRAIN:  27%|██▋       | 37/137 [00:10<00:20,  4.84it/s, epoch=56, grad_norm=92.118, loss=266.7964, lr=6.14e-04] 
Epoch 56 TRAIN:  29%|██▉       | 40/137 [00:10<00:20,  4.84it/s, epoch=56, grad_norm=92.118, loss=266.7964, lr=6.14e-04]
Epoch 56 TRAIN:  29%|██▉       | 40/137 [00:10<00:20,  4.84it/s, epoch=56, grad_norm=184.768, loss=283.8805, lr=6.14e-04]
Epoch 56 TRAIN:  29%|██▉       | 40/137 [00:10<00:20,  4.84it/s, epoch=56, grad_norm=130.204, loss=306.8370, lr=6.14e-04]
Epoch 56 TRAIN:  29%|██▉       | 40/137 [00:11<00:20,  4.84it/s, epoch=56, grad_norm=107.998, loss=293.0148, lr=6.14e-04]
Epoch 56 TRAIN:  31%|███▏      | 43/137 [00:11<00:18,  5.08it/s, epoch=56, grad_norm=107.998, loss=293.0148, lr=6.14e-04]
Epoch 56 TRAIN:  31%|███▏      | 43/137 [00:11<00:18,  5.08it/s, epoch=56, grad_norm=121.835, loss=321.9921, lr=6.15e-04]
Epoch 56 TRAIN:  31%|███▏      | 43/137 [00:11<00:18,  5.08it/s, epoch=56, grad_norm=124.969, loss=278.8951, lr=6.15e-04]
Epoch 56 TRAIN:  31%|███▏      | 43/137 [00:11<00:18,  5.08it/s, epoch=56, grad_norm=100.027, loss=311.4317, lr=6.15e-04]
Epoch 56 TRAIN:  34%|███▎      | 46/137 [00:11<00:19,  4.74it/s, epoch=56, grad_norm=100.027, loss=311.4317, lr=6.15e-04]
Epoch 56 TRAIN:  34%|███▎      | 46/137 [00:12<00:19,  4.74it/s, epoch=56, grad_norm=135.829, loss=241.0111, lr=6.15e-04]
Epoch 56 TRAIN:  34%|███▎      | 46/137 [00:12<00:19,  4.74it/s, epoch=56, grad_norm=85.734, loss=260.9543, lr=6.15e-04] 
Epoch 56 TRAIN:  34%|███▎      | 46/137 [00:12<00:19,  4.74it/s, epoch=56, grad_norm=90.584, loss=265.0290, lr=6.15e-04]
Epoch 56 TRAIN:  36%|███▌      | 49/137 [00:12<00:18,  4.82it/s, epoch=56, grad_norm=90.584, loss=265.0290, lr=6.15e-04]
Epoch 56 TRAIN:  36%|███▌      | 49/137 [00:12<00:18,  4.82it/s, epoch=56, grad_norm=141.626, loss=309.0639, lr=6.15e-04]
Epoch 56 TRAIN:  36%|███▌      | 49/137 [00:12<00:18,  4.82it/s, epoch=56, grad_norm=297.159, loss=331.8130, lr=6.15e-04]
Epoch 56 TRAIN:  36%|███▌      | 49/137 [00:12<00:18,  4.82it/s, epoch=56, grad_norm=177.112, loss=303.4333, lr=6.15e-04]
Epoch 56 TRAIN:  38%|███▊      | 52/137 [00:12<00:17,  4.95it/s, epoch=56, grad_norm=177.112, loss=303.4333, lr=6.15e-04]
Epoch 56 TRAIN:  38%|███▊      | 52/137 [00:13<00:17,  4.95it/s, epoch=56, grad_norm=137.217, loss=303.5048, lr=6.15e-04]
Epoch 56 TRAIN:  38%|███▊      | 52/137 [00:13<00:17,  4.95it/s, epoch=56, grad_norm=146.611, loss=336.3614, lr=6.15e-04]
Epoch 56 TRAIN:  38%|███▊      | 52/137 [00:13<00:17,  4.95it/s, epoch=56, grad_norm=142.693, loss=299.8127, lr=6.15e-04]
Epoch 56 TRAIN:  40%|████      | 55/137 [00:13<00:16,  4.92it/s, epoch=56, grad_norm=142.693, loss=299.8127, lr=6.15e-04]
Epoch 56 TRAIN:  40%|████      | 55/137 [00:13<00:16,  4.92it/s, epoch=56, grad_norm=100.881, loss=310.6338, lr=6.16e-04]
Epoch 56 TRAIN:  40%|████      | 55/137 [00:14<00:16,  4.92it/s, epoch=56, grad_norm=197.798, loss=244.3303, lr=6.16e-04]
Epoch 56 TRAIN:  40%|████      | 55/137 [00:14<00:16,  4.92it/s, epoch=56, grad_norm=106.162, loss=256.2147, lr=6.16e-04]
Epoch 56 TRAIN:  42%|████▏     | 58/137 [00:14<00:16,  4.75it/s, epoch=56, grad_norm=106.162, loss=256.2147, lr=6.16e-04]
Epoch 56 TRAIN:  42%|████▏     | 58/137 [00:14<00:16,  4.75it/s, epoch=56, grad_norm=183.293, loss=257.5360, lr=6.16e-04]
Epoch 56 TRAIN:  42%|████▏     | 58/137 [00:14<00:16,  4.75it/s, epoch=56, grad_norm=99.255, loss=290.2751, lr=6.16e-04] 
Epoch 56 TRAIN:  42%|████▏     | 58/137 [00:14<00:16,  4.75it/s, epoch=56, grad_norm=144.559, loss=298.0699, lr=6.16e-04]
Epoch 56 TRAIN:  45%|████▍     | 61/137 [00:14<00:15,  4.83it/s, epoch=56, grad_norm=144.559, loss=298.0699, lr=6.16e-04]
Epoch 56 TRAIN:  45%|████▍     | 61/137 [00:15<00:15,  4.83it/s, epoch=56, grad_norm=108.453, loss=312.8409, lr=6.16e-04]
Epoch 56 TRAIN:  45%|████▍     | 61/137 [00:15<00:15,  4.83it/s, epoch=56, grad_norm=83.925, loss=299.5127, lr=6.16e-04] 
Epoch 56 TRAIN:  45%|████▍     | 61/137 [00:15<00:15,  4.83it/s, epoch=56, grad_norm=79.453, loss=289.7925, lr=6.16e-04]
Epoch 56 TRAIN:  47%|████▋     | 64/137 [00:15<00:15,  4.86it/s, epoch=56, grad_norm=79.453, loss=289.7925, lr=6.16e-04]
Epoch 56 TRAIN:  47%|████▋     | 64/137 [00:15<00:15,  4.86it/s, epoch=56, grad_norm=86.957, loss=320.3613, lr=6.16e-04]
Epoch 56 TRAIN:  47%|████▋     | 64/137 [00:15<00:15,  4.86it/s, epoch=56, grad_norm=127.196, loss=296.5240, lr=6.16e-04]
Epoch 56 TRAIN:  47%|████▋     | 64/137 [00:16<00:15,  4.86it/s, epoch=56, grad_norm=149.212, loss=286.6090, lr=6.16e-04]
Epoch 56 TRAIN:  49%|████▉     | 67/137 [00:16<00:13,  5.07it/s, epoch=56, grad_norm=149.212, loss=286.6090, lr=6.16e-04]
Epoch 56 TRAIN:  49%|████▉     | 67/137 [00:16<00:13,  5.07it/s, epoch=56, grad_norm=83.823, loss=272.6279, lr=6.16e-04] 
Epoch 56 TRAIN:  49%|████▉     | 67/137 [00:16<00:13,  5.07it/s, epoch=56, grad_norm=107.141, loss=305.6351, lr=6.17e-04]
Epoch 56 TRAIN:  49%|████▉     | 67/137 [00:16<00:13,  5.07it/s, epoch=56, grad_norm=131.794, loss=293.2513, lr=6.17e-04]
Epoch 56 TRAIN:  51%|█████     | 70/137 [00:16<00:14,  4.78it/s, epoch=56, grad_norm=131.794, loss=293.2513, lr=6.17e-04]
Epoch 56 TRAIN:  51%|█████     | 70/137 [00:16<00:14,  4.78it/s, epoch=56, grad_norm=154.262, loss=270.6801, lr=6.17e-04]
Epoch 56 TRAIN:  51%|█████     | 70/137 [00:17<00:14,  4.78it/s, epoch=56, grad_norm=149.720, loss=266.1074, lr=6.17e-04]
Epoch 56 TRAIN:  51%|█████     | 70/137 [00:17<00:14,  4.78it/s, epoch=56, grad_norm=202.215, loss=266.1364, lr=6.17e-04]
Epoch 56 TRAIN:  53%|█████▎    | 73/137 [00:17<00:13,  4.87it/s, epoch=56, grad_norm=202.215, loss=266.1364, lr=6.17e-04]
Epoch 56 TRAIN:  53%|█████▎    | 73/137 [00:17<00:13,  4.87it/s, epoch=56, grad_norm=104.578, loss=278.3885, lr=6.17e-04]
Epoch 56 TRAIN:  53%|█████▎    | 73/137 [00:17<00:13,  4.87it/s, epoch=56, grad_norm=244.143, loss=212.1906, lr=6.17e-04]
Epoch 56 TRAIN:  53%|█████▎    | 73/137 [00:17<00:13,  4.87it/s, epoch=56, grad_norm=221.389, loss=344.0756, lr=6.17e-04]
Epoch 56 TRAIN:  55%|█████▌    | 76/137 [00:17<00:12,  4.96it/s, epoch=56, grad_norm=221.389, loss=344.0756, lr=6.17e-04]
Epoch 56 TRAIN:  55%|█████▌    | 76/137 [00:18<00:12,  4.96it/s, epoch=56, grad_norm=106.376, loss=289.1511, lr=6.17e-04]
Epoch 56 TRAIN:  55%|█████▌    | 76/137 [00:18<00:12,  4.96it/s, epoch=56, grad_norm=182.159, loss=322.6707, lr=6.17e-04]
Epoch 56 TRAIN:  55%|█████▌    | 76/137 [00:18<00:12,  4.96it/s, epoch=56, grad_norm=338.511, loss=175.9241, lr=6.17e-04]
Epoch 56 TRAIN:  58%|█████▊    | 79/137 [00:18<00:11,  4.92it/s, epoch=56, grad_norm=338.511, loss=175.9241, lr=6.17e-04]
Epoch 56 TRAIN:  58%|█████▊    | 79/137 [00:18<00:11,  4.92it/s, epoch=56, grad_norm=103.909, loss=244.9425, lr=6.17e-04]
Epoch 56 TRAIN:  58%|█████▊    | 79/137 [00:18<00:11,  4.92it/s, epoch=56, grad_norm=176.056, loss=263.4951, lr=6.18e-04]
Epoch 56 TRAIN:  58%|█████▊    | 79/137 [00:19<00:11,  4.92it/s, epoch=56, grad_norm=180.862, loss=328.2544, lr=6.18e-04]
Epoch 56 TRAIN:  60%|█████▉    | 82/137 [00:19<00:11,  4.82it/s, epoch=56, grad_norm=180.862, loss=328.2544, lr=6.18e-04]
Epoch 56 TRAIN:  60%|█████▉    | 82/137 [00:19<00:11,  4.82it/s, epoch=56, grad_norm=174.930, loss=307.3151, lr=6.18e-04]
Epoch 56 TRAIN:  60%|█████▉    | 82/137 [00:19<00:11,  4.82it/s, epoch=56, grad_norm=182.003, loss=317.4466, lr=6.18e-04]
Epoch 56 TRAIN:  60%|█████▉    | 82/137 [00:19<00:11,  4.82it/s, epoch=56, grad_norm=114.542, loss=295.9800, lr=6.18e-04]
Epoch 56 TRAIN:  62%|██████▏   | 85/137 [00:19<00:11,  4.72it/s, epoch=56, grad_norm=114.542, loss=295.9800, lr=6.18e-04]
Epoch 56 TRAIN:  62%|██████▏   | 85/137 [00:20<00:11,  4.72it/s, epoch=56, grad_norm=135.306, loss=287.5862, lr=6.18e-04]
Epoch 56 TRAIN:  62%|██████▏   | 85/137 [00:20<00:11,  4.72it/s, epoch=56, grad_norm=126.839, loss=298.4117, lr=6.18e-04]
Epoch 56 TRAIN:  62%|██████▏   | 85/137 [00:20<00:11,  4.72it/s, epoch=56, grad_norm=131.932, loss=308.9707, lr=6.18e-04]
Epoch 56 TRAIN:  64%|██████▍   | 88/137 [00:20<00:10,  4.50it/s, epoch=56, grad_norm=131.932, loss=308.9707, lr=6.18e-04]
Epoch 56 TRAIN:  64%|██████▍   | 88/137 [00:20<00:10,  4.50it/s, epoch=56, grad_norm=408.620, loss=231.4379, lr=6.18e-04]
Epoch 56 TRAIN:  64%|██████▍   | 88/137 [00:20<00:10,  4.50it/s, epoch=56, grad_norm=117.795, loss=302.5120, lr=6.18e-04]
Epoch 56 TRAIN:  64%|██████▍   | 88/137 [00:21<00:10,  4.50it/s, epoch=56, grad_norm=266.740, loss=260.9675, lr=6.18e-04]
Epoch 56 TRAIN:  66%|██████▋   | 91/137 [00:21<00:09,  4.77it/s, epoch=56, grad_norm=266.740, loss=260.9675, lr=6.18e-04]
Epoch 56 TRAIN:  66%|██████▋   | 91/137 [00:21<00:09,  4.77it/s, epoch=56, grad_norm=82.485, loss=278.1832, lr=6.18e-04] 
Epoch 56 TRAIN:  66%|██████▋   | 91/137 [00:21<00:09,  4.77it/s, epoch=56, grad_norm=123.973, loss=294.1602, lr=6.18e-04]
Epoch 56 TRAIN:  66%|██████▋   | 91/137 [00:21<00:09,  4.77it/s, epoch=56, grad_norm=345.410, loss=229.8057, lr=6.19e-04]
Epoch 56 TRAIN:  69%|██████▊   | 94/137 [00:21<00:09,  4.57it/s, epoch=56, grad_norm=345.410, loss=229.8057, lr=6.19e-04]
Epoch 56 TRAIN:  69%|██████▊   | 94/137 [00:22<00:09,  4.57it/s, epoch=56, grad_norm=159.916, loss=302.4713, lr=6.19e-04]
Epoch 56 TRAIN:  69%|██████▊   | 94/137 [00:22<00:09,  4.57it/s, epoch=56, grad_norm=213.379, loss=348.1287, lr=6.19e-04]
Epoch 56 TRAIN:  69%|██████▊   | 94/137 [00:22<00:09,  4.57it/s, epoch=56, grad_norm=190.799, loss=306.1143, lr=6.19e-04]
Epoch 56 TRAIN:  71%|███████   | 97/137 [00:22<00:08,  4.84it/s, epoch=56, grad_norm=190.799, loss=306.1143, lr=6.19e-04]
Epoch 56 TRAIN:  71%|███████   | 97/137 [00:22<00:08,  4.84it/s, epoch=56, grad_norm=121.933, loss=285.0498, lr=6.19e-04]
Epoch 56 TRAIN:  71%|███████   | 97/137 [00:22<00:08,  4.84it/s, epoch=56, grad_norm=157.005, loss=290.3583, lr=6.19e-04]
Epoch 56 TRAIN:  71%|███████   | 97/137 [00:23<00:08,  4.84it/s, epoch=56, grad_norm=217.081, loss=293.9604, lr=6.19e-04]2026-04-16 20:20:42,341 DEBUG TRAIN | steps/sec 4.316| Batch 56/100 loss 293.960419 loss_att 247.148300 loss_ctc 403.188660 th_accuracy 0.360684 lr 6.1904e-04 grad_norm 217.080826 rank 0

Epoch 56 TRAIN:  73%|███████▎  | 100/137 [00:23<00:07,  4.74it/s, epoch=56, grad_norm=217.081, loss=293.9604, lr=6.19e-04]
Epoch 56 TRAIN:  73%|███████▎  | 100/137 [00:23<00:07,  4.74it/s, epoch=56, grad_norm=205.239, loss=290.4167, lr=6.19e-04]
Epoch 56 TRAIN:  73%|███████▎  | 100/137 [00:23<00:07,  4.74it/s, epoch=56, grad_norm=173.410, loss=313.0773, lr=6.19e-04]
Epoch 56 TRAIN:  73%|███████▎  | 100/137 [00:23<00:07,  4.74it/s, epoch=56, grad_norm=123.918, loss=276.0127, lr=6.19e-04]
Epoch 56 TRAIN:  75%|███████▌  | 103/137 [00:23<00:07,  4.80it/s, epoch=56, grad_norm=123.918, loss=276.0127, lr=6.19e-04]
Epoch 56 TRAIN:  75%|███████▌  | 103/137 [00:23<00:07,  4.80it/s, epoch=56, grad_norm=108.528, loss=258.9002, lr=6.19e-04]
Epoch 56 TRAIN:  75%|███████▌  | 103/137 [00:24<00:07,  4.80it/s, epoch=56, grad_norm=164.862, loss=349.3384, lr=6.19e-04]
Epoch 56 TRAIN:  75%|███████▌  | 103/137 [00:24<00:07,  4.80it/s, epoch=56, grad_norm=107.286, loss=232.0595, lr=6.20e-04]
Epoch 56 TRAIN:  77%|███████▋  | 106/137 [00:24<00:06,  4.82it/s, epoch=56, grad_norm=107.286, loss=232.0595, lr=6.20e-04]
Epoch 56 TRAIN:  77%|███████▋  | 106/137 [00:24<00:06,  4.82it/s, epoch=56, grad_norm=202.301, loss=345.7271, lr=6.20e-04]
Epoch 56 TRAIN:  77%|███████▋  | 106/137 [00:24<00:06,  4.82it/s, epoch=56, grad_norm=124.061, loss=255.4766, lr=6.20e-04]
Epoch 56 TRAIN:  77%|███████▋  | 106/137 [00:24<00:06,  4.82it/s, epoch=56, grad_norm=176.966, loss=283.1384, lr=6.20e-04]
Epoch 56 TRAIN:  80%|███████▉  | 109/137 [00:24<00:05,  4.94it/s, epoch=56, grad_norm=176.966, loss=283.1384, lr=6.20e-04]
Epoch 56 TRAIN:  80%|███████▉  | 109/137 [00:25<00:05,  4.94it/s, epoch=56, grad_norm=169.262, loss=296.0762, lr=6.20e-04]
Epoch 56 TRAIN:  80%|███████▉  | 109/137 [00:25<00:05,  4.94it/s, epoch=56, grad_norm=145.485, loss=298.7638, lr=6.20e-04]
Epoch 56 TRAIN:  80%|███████▉  | 109/137 [00:25<00:05,  4.94it/s, epoch=56, grad_norm=311.371, loss=335.4704, lr=6.20e-04]
Epoch 56 TRAIN:  82%|████████▏ | 112/137 [00:25<00:05,  4.74it/s, epoch=56, grad_norm=311.371, loss=335.4704, lr=6.20e-04]
Epoch 56 TRAIN:  82%|████████▏ | 112/137 [00:25<00:05,  4.74it/s, epoch=56, grad_norm=167.501, loss=298.0774, lr=6.20e-04]
Epoch 56 TRAIN:  82%|████████▏ | 112/137 [00:25<00:05,  4.74it/s, epoch=56, grad_norm=537.281, loss=290.2319, lr=6.20e-04]
Epoch 56 TRAIN:  82%|████████▏ | 112/137 [00:26<00:05,  4.74it/s, epoch=56, grad_norm=137.149, loss=218.8135, lr=6.20e-04]
Epoch 56 TRAIN:  84%|████████▍ | 115/137 [00:26<00:04,  4.96it/s, epoch=56, grad_norm=137.149, loss=218.8135, lr=6.20e-04]
Epoch 56 TRAIN:  84%|████████▍ | 115/137 [00:26<00:04,  4.96it/s, epoch=56, grad_norm=123.374, loss=240.2317, lr=6.20e-04]
Epoch 56 TRAIN:  84%|████████▍ | 115/137 [00:26<00:04,  4.96it/s, epoch=56, grad_norm=144.222, loss=323.0758, lr=6.20e-04]
Epoch 56 TRAIN:  84%|████████▍ | 115/137 [00:26<00:04,  4.96it/s, epoch=56, grad_norm=115.789, loss=281.5246, lr=6.20e-04]
Epoch 56 TRAIN:  86%|████████▌ | 118/137 [00:26<00:03,  4.79it/s, epoch=56, grad_norm=115.789, loss=281.5246, lr=6.20e-04]
Epoch 56 TRAIN:  86%|████████▌ | 118/137 [00:26<00:03,  4.79it/s, epoch=56, grad_norm=212.572, loss=368.3550, lr=6.21e-04]
Epoch 56 TRAIN:  86%|████████▌ | 118/137 [00:27<00:03,  4.79it/s, epoch=56, grad_norm=123.664, loss=254.0557, lr=6.21e-04]
Epoch 56 TRAIN:  86%|████████▌ | 118/137 [00:27<00:03,  4.79it/s, epoch=56, grad_norm=161.717, loss=281.7877, lr=6.21e-04]
Epoch 56 TRAIN:  88%|████████▊ | 121/137 [00:27<00:03,  4.95it/s, epoch=56, grad_norm=161.717, loss=281.7877, lr=6.21e-04]
Epoch 56 TRAIN:  88%|████████▊ | 121/137 [00:27<00:03,  4.95it/s, epoch=56, grad_norm=137.159, loss=271.9849, lr=6.21e-04]
Epoch 56 TRAIN:  88%|████████▊ | 121/137 [00:27<00:03,  4.95it/s, epoch=56, grad_norm=138.789, loss=289.8813, lr=6.21e-04]
Epoch 56 TRAIN:  88%|████████▊ | 121/137 [00:27<00:03,  4.95it/s, epoch=56, grad_norm=124.387, loss=290.9825, lr=6.21e-04]
Epoch 56 TRAIN:  91%|█████████ | 124/137 [00:27<00:02,  4.73it/s, epoch=56, grad_norm=124.387, loss=290.9825, lr=6.21e-04]
Epoch 56 TRAIN:  91%|█████████ | 124/137 [00:28<00:02,  4.73it/s, epoch=56, grad_norm=165.828, loss=216.3055, lr=6.21e-04]
Epoch 56 TRAIN:  91%|█████████ | 124/137 [00:28<00:02,  4.73it/s, epoch=56, grad_norm=159.980, loss=309.2406, lr=6.21e-04]
Epoch 56 TRAIN:  91%|█████████ | 124/137 [00:28<00:02,  4.73it/s, epoch=56, grad_norm=161.709, loss=292.7397, lr=6.21e-04]
Epoch 56 TRAIN:  93%|█████████▎| 127/137 [00:28<00:02,  4.71it/s, epoch=56, grad_norm=161.709, loss=292.7397, lr=6.21e-04]
Epoch 56 TRAIN:  93%|█████████▎| 127/137 [00:28<00:02,  4.71it/s, epoch=56, grad_norm=141.700, loss=285.1209, lr=6.21e-04]
Epoch 56 TRAIN:  93%|█████████▎| 127/137 [00:29<00:02,  4.71it/s, epoch=56, grad_norm=141.146, loss=264.2203, lr=6.21e-04]
Epoch 56 TRAIN:  93%|█████████▎| 127/137 [00:29<00:02,  4.71it/s, epoch=56, grad_norm=223.956, loss=365.0681, lr=6.21e-04]
Epoch 56 TRAIN:  95%|█████████▍| 130/137 [00:29<00:01,  4.58it/s, epoch=56, grad_norm=223.956, loss=365.0681, lr=6.21e-04]
Epoch 56 TRAIN:  95%|█████████▍| 130/137 [00:29<00:01,  4.58it/s, epoch=56, grad_norm=364.675, loss=416.0308, lr=6.22e-04]
Epoch 56 TRAIN:  95%|█████████▍| 130/137 [00:29<00:01,  4.58it/s, epoch=56, grad_norm=335.498, loss=361.4938, lr=6.22e-04]
Epoch 56 TRAIN:  95%|█████████▍| 130/137 [00:29<00:01,  4.58it/s, epoch=56, grad_norm=237.274, loss=294.4210, lr=6.22e-04]
Epoch 56 TRAIN:  97%|█████████▋| 133/137 [00:29<00:00,  4.91it/s, epoch=56, grad_norm=237.274, loss=294.4210, lr=6.22e-04]
Epoch 56 TRAIN:  97%|█████████▋| 133/137 [00:30<00:00,  4.91it/s, epoch=56, grad_norm=142.303, loss=319.8974, lr=6.22e-04]
Epoch 56 TRAIN:  97%|█████████▋| 133/137 [00:30<00:00,  4.91it/s, epoch=56, grad_norm=219.151, loss=337.5787, lr=6.22e-04]
Epoch 56 TRAIN:  97%|█████████▋| 133/137 [00:30<00:00,  4.91it/s, epoch=56, grad_norm=202.404, loss=322.6494, lr=6.22e-04]
Epoch 56 TRAIN:  99%|█████████▉| 136/137 [00:30<00:00,  4.77it/s, epoch=56, grad_norm=202.404, loss=322.6494, lr=6.22e-04]
Epoch 56 TRAIN:  99%|█████████▉| 136/137 [00:30<00:00,  4.46it/s, epoch=56, grad_norm=202.404, loss=322.6494, lr=6.22e-04]

CV:   0%|          | 0/6 [00:00<?, ?it/s]/home/rookie/Documents/batuka/projects/wenet/wenet/wenet/utils/train_utils.py:689: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  amp_autocast(enabled=dtype is not None,

CV:  17%|█▋        | 1/6 [00:00<00:03,  1.32it/s]
                                                 
2026-04-16 20:20:50,679 INFO Epoch 56 Step 7774 CV info lr 6.2192e-04 cv_loss 474.3218883167614 rank 0 acc 0.3285772254069646
2026-04-16 20:20:50,681 INFO [Rank 0] Checkpoint: save to checkpoint exp/conformer_mongolian/epoch_56.pt
2026-04-16 20:20:50,796 INFO Epoch 57 Step 7774 TRAIN info lr 6.2192e-04 rank 0
2026-04-16 20:20:50,812 INFO using accumulate grad, new batch size is 1 times larger than before

Epoch 57 TRAIN:   0%|          | 0/136 [00:00<?, ?it/s]
Epoch 57 TRAIN:   0%|          | 0/136 [00:04<?, ?it/s, epoch=57, grad_norm=464.627, loss=409.5986, lr=6.22e-04]
Epoch 57 TRAIN:   1%|          | 1/136 [00:04<09:14,  4.11s/it, epoch=57, grad_norm=464.627, loss=409.5986, lr=6.22e-04]
Epoch 57 TRAIN:   1%|          | 1/136 [00:04<09:14,  4.11s/it, epoch=57, grad_norm=440.061, loss=404.4312, lr=6.22e-04]
Epoch 57 TRAIN:   1%|          | 1/136 [00:04<09:14,  4.11s/it, epoch=57, grad_norm=226.924, loss=250.0851, lr=6.22e-04]
Epoch 57 TRAIN:   1%|          | 1/136 [00:04<09:14,  4.11s/it, epoch=57, grad_norm=617.332, loss=499.3151, lr=6.22e-04]
Epoch 57 TRAIN:   1%|          | 1/136 [00:04<09:14,  4.11s/it, epoch=57, grad_norm=281.181, loss=305.6503, lr=6.22e-04]
Epoch 57 TRAIN:   1%|          | 1/136 [00:04<09:14,  4.11s/it, epoch=57, grad_norm=376.514, loss=408.9125, lr=6.22e-04]
Epoch 57 TRAIN:   1%|          | 1/136 [00:04<09:14,  4.11s/it, epoch=57, grad_norm=213.858, loss=295.2083, lr=6.22e-04]
Epoch 57 TRAIN:   1%|          | 1/136 [00:04<09:14,  4.11s/it, epoch=57, grad_norm=173.054, loss=295.1816, lr=6.23e-04]
Epoch 57 TRAIN:   6%|▌         | 8/136 [00:04<00:56,  2.26it/s, epoch=57, grad_norm=173.054, loss=295.1816, lr=6.23e-04]
Epoch 57 TRAIN:   6%|▌         | 8/136 [00:04<00:56,  2.26it/s, epoch=57, grad_norm=133.411, loss=278.2701, lr=6.23e-04]
Epoch 57 TRAIN:   6%|▌         | 8/136 [00:04<00:56,  2.26it/s, epoch=57, grad_norm=111.697, loss=271.4834, lr=6.23e-04]
Epoch 57 TRAIN:   6%|▌         | 8/136 [00:05<00:56,  2.26it/s, epoch=57, grad_norm=142.515, loss=306.2126, lr=6.23e-04]
Epoch 57 TRAIN:   6%|▌         | 8/136 [00:05<00:56,  2.26it/s, epoch=57, grad_norm=154.819, loss=323.4037, lr=6.23e-04]
Epoch 57 TRAIN:   9%|▉         | 12/136 [00:05<00:40,  3.10it/s, epoch=57, grad_norm=154.819, loss=323.4037, lr=6.23e-04]
Epoch 57 TRAIN:   9%|▉         | 12/136 [00:05<00:40,  3.10it/s, epoch=57, grad_norm=90.099, loss=317.0205, lr=6.23e-04] 
Epoch 57 TRAIN:   9%|▉         | 12/136 [00:05<00:40,  3.10it/s, epoch=57, grad_norm=114.506, loss=303.7373, lr=6.23e-04]
Epoch 57 TRAIN:   9%|▉         | 12/136 [00:05<00:40,  3.10it/s, epoch=57, grad_norm=115.172, loss=290.6309, lr=6.23e-04]
Epoch 57 TRAIN:   9%|▉         | 12/136 [00:05<00:40,  3.10it/s, epoch=57, grad_norm=163.281, loss=267.7684, lr=6.23e-04]
Epoch 57 TRAIN:  12%|█▏        | 16/136 [00:05<00:31,  3.84it/s, epoch=57, grad_norm=163.281, loss=267.7684, lr=6.23e-04]
Epoch 57 TRAIN:  12%|█▏        | 16/136 [00:06<00:31,  3.84it/s, epoch=57, grad_norm=110.652, loss=311.5429, lr=6.23e-04]
Epoch 57 TRAIN:  12%|█▏        | 16/136 [00:06<00:31,  3.84it/s, epoch=57, grad_norm=102.397, loss=286.6054, lr=6.23e-04]
Epoch 57 TRAIN:  12%|█▏        | 16/136 [00:06<00:31,  3.84it/s, epoch=57, grad_norm=125.153, loss=302.6022, lr=6.23e-04]
Epoch 57 TRAIN:  12%|█▏        | 16/136 [00:06<00:31,  3.84it/s, epoch=57, grad_norm=105.821, loss=316.5980, lr=6.24e-04]
Epoch 57 TRAIN:  15%|█▍        | 20/136 [00:06<00:26,  4.32it/s, epoch=57, grad_norm=105.821, loss=316.5980, lr=6.24e-04]
Epoch 57 TRAIN:  15%|█▍        | 20/136 [00:06<00:26,  4.32it/s, epoch=57, grad_norm=73.743, loss=277.7816, lr=6.24e-04] 
Epoch 57 TRAIN:  15%|█▍        | 20/136 [00:07<00:26,  4.32it/s, epoch=57, grad_norm=94.893, loss=311.1921, lr=6.24e-04]
Epoch 57 TRAIN:  15%|█▍        | 20/136 [00:07<00:26,  4.32it/s, epoch=57, grad_norm=95.122, loss=278.4778, lr=6.24e-04]
Epoch 57 TRAIN:  15%|█▍        | 20/136 [00:07<00:26,  4.32it/s, epoch=57, grad_norm=84.275, loss=269.0666, lr=6.24e-04]
Epoch 57 TRAIN:  18%|█▊        | 24/136 [00:07<00:24,  4.53it/s, epoch=57, grad_norm=84.275, loss=269.0666, lr=6.24e-04]
Epoch 57 TRAIN:  18%|█▊        | 24/136 [00:07<00:24,  4.53it/s, epoch=57, grad_norm=85.433, loss=298.2426, lr=6.24e-04]
Epoch 57 TRAIN:  18%|█▊        | 24/136 [00:07<00:24,  4.53it/s, epoch=57, grad_norm=87.285, loss=306.2248, lr=6.24e-04]
Epoch 57 TRAIN:  18%|█▊        | 24/136 [00:07<00:24,  4.53it/s, epoch=57, grad_norm=110.705, loss=268.6885, lr=6.24e-04]
Epoch 57 TRAIN:  20%|█▉        | 27/136 [00:07<00:23,  4.72it/s, epoch=57, grad_norm=110.705, loss=268.6885, lr=6.24e-04]
Epoch 57 TRAIN:  20%|█▉        | 27/136 [00:08<00:23,  4.72it/s, epoch=57, grad_norm=133.077, loss=317.5628, lr=6.24e-04]
Epoch 57 TRAIN:  20%|█▉        | 27/136 [00:08<00:23,  4.72it/s, epoch=57, grad_norm=86.203, loss=280.0570, lr=6.24e-04] 
Epoch 57 TRAIN:  20%|█▉        | 27/136 [00:08<00:23,  4.72it/s, epoch=57, grad_norm=99.022, loss=314.5687, lr=6.24e-04]
Epoch 57 TRAIN:  22%|██▏       | 30/136 [00:08<00:21,  4.87it/s, epoch=57, grad_norm=99.022, loss=314.5687, lr=6.24e-04]
Epoch 57 TRAIN:  22%|██▏       | 30/136 [00:08<00:21,  4.87it/s, epoch=57, grad_norm=122.654, loss=322.2009, lr=6.24e-04]
Epoch 57 TRAIN:  22%|██▏       | 30/136 [00:08<00:21,  4.87it/s, epoch=57, grad_norm=109.402, loss=291.1498, lr=6.24e-04]
Epoch 57 TRAIN:  22%|██▏       | 30/136 [00:09<00:21,  4.87it/s, epoch=57, grad_norm=114.323, loss=335.0679, lr=6.25e-04]
Epoch 57 TRAIN:  24%|██▍       | 33/136 [00:09<00:21,  4.86it/s, epoch=57, grad_norm=114.323, loss=335.0679, lr=6.25e-04]
Epoch 57 TRAIN:  24%|██▍       | 33/136 [00:09<00:21,  4.86it/s, epoch=57, grad_norm=105.733, loss=278.6956, lr=6.25e-04]
Epoch 57 TRAIN:  24%|██▍       | 33/136 [00:09<00:21,  4.86it/s, epoch=57, grad_norm=117.819, loss=315.0675, lr=6.25e-04]
Epoch 57 TRAIN:  24%|██▍       | 33/136 [00:09<00:21,  4.86it/s, epoch=57, grad_norm=114.318, loss=282.1045, lr=6.25e-04]
Epoch 57 TRAIN:  26%|██▋       | 36/136 [00:09<00:20,  4.80it/s, epoch=57, grad_norm=114.318, loss=282.1045, lr=6.25e-04]
Epoch 57 TRAIN:  26%|██▋       | 36/136 [00:09<00:20,  4.80it/s, epoch=57, grad_norm=100.976, loss=322.0768, lr=6.25e-04]
Epoch 57 TRAIN:  26%|██▋       | 36/136 [00:10<00:20,  4.80it/s, epoch=57, grad_norm=104.414, loss=286.1778, lr=6.25e-04]
Epoch 57 TRAIN:  26%|██▋       | 36/136 [00:10<00:20,  4.80it/s, epoch=57, grad_norm=150.717, loss=270.9445, lr=6.25e-04]
Epoch 57 TRAIN:  29%|██▊       | 39/136 [00:10<00:19,  4.97it/s, epoch=57, grad_norm=150.717, loss=270.9445, lr=6.25e-04]
Epoch 57 TRAIN:  29%|██▊       | 39/136 [00:10<00:19,  4.97it/s, epoch=57, grad_norm=109.875, loss=277.8887, lr=6.25e-04]
Epoch 57 TRAIN:  29%|██▊       | 39/136 [00:10<00:19,  4.97it/s, epoch=57, grad_norm=103.501, loss=295.6915, lr=6.25e-04]
Epoch 57 TRAIN:  29%|██▊       | 39/136 [00:10<00:19,  4.97it/s, epoch=57, grad_norm=138.805, loss=314.1827, lr=6.25e-04]
Epoch 57 TRAIN:  31%|███       | 42/136 [00:10<00:18,  5.04it/s, epoch=57, grad_norm=138.805, loss=314.1827, lr=6.25e-04]
Epoch 57 TRAIN:  31%|███       | 42/136 [00:11<00:18,  5.04it/s, epoch=57, grad_norm=73.676, loss=266.8802, lr=6.25e-04] 
Epoch 57 TRAIN:  31%|███       | 42/136 [00:11<00:18,  5.04it/s, epoch=57, grad_norm=109.826, loss=233.6980, lr=6.25e-04]
Epoch 57 TRAIN:  31%|███       | 42/136 [00:11<00:18,  5.04it/s, epoch=57, grad_norm=162.488, loss=309.9568, lr=6.26e-04]
Epoch 57 TRAIN:  33%|███▎      | 45/136 [00:11<00:18,  4.88it/s, epoch=57, grad_norm=162.488, loss=309.9568, lr=6.26e-04]
Epoch 57 TRAIN:  33%|███▎      | 45/136 [00:11<00:18,  4.88it/s, epoch=57, grad_norm=110.925, loss=311.7075, lr=6.26e-04]
Epoch 57 TRAIN:  33%|███▎      | 45/136 [00:12<00:18,  4.88it/s, epoch=57, grad_norm=103.396, loss=245.7686, lr=6.26e-04]
Epoch 57 TRAIN:  33%|███▎      | 45/136 [00:12<00:18,  4.88it/s, epoch=57, grad_norm=129.523, loss=315.5273, lr=6.26e-04]
Epoch 57 TRAIN:  35%|███▌      | 48/136 [00:12<00:18,  4.88it/s, epoch=57, grad_norm=129.523, loss=315.5273, lr=6.26e-04]
Epoch 57 TRAIN:  35%|███▌      | 48/136 [00:12<00:18,  4.88it/s, epoch=57, grad_norm=164.899, loss=317.5555, lr=6.26e-04]
Epoch 57 TRAIN:  35%|███▌      | 48/136 [00:12<00:18,  4.88it/s, epoch=57, grad_norm=134.719, loss=296.6282, lr=6.26e-04]
Epoch 57 TRAIN:  35%|███▌      | 48/136 [00:12<00:18,  4.88it/s, epoch=57, grad_norm=91.130, loss=279.7545, lr=6.26e-04] 
Epoch 57 TRAIN:  38%|███▊      | 51/136 [00:12<00:17,  4.86it/s, epoch=57, grad_norm=91.130, loss=279.7545, lr=6.26e-04]
Epoch 57 TRAIN:  38%|███▊      | 51/136 [00:13<00:17,  4.86it/s, epoch=57, grad_norm=138.678, loss=316.4933, lr=6.26e-04]
Epoch 57 TRAIN:  38%|███▊      | 51/136 [00:13<00:17,  4.86it/s, epoch=57, grad_norm=148.445, loss=239.5063, lr=6.26e-04]
Epoch 57 TRAIN:  38%|███▊      | 51/136 [00:13<00:17,  4.86it/s, epoch=57, grad_norm=102.801, loss=289.0963, lr=6.26e-04]
Epoch 57 TRAIN:  40%|███▉      | 54/136 [00:13<00:16,  4.98it/s, epoch=57, grad_norm=102.801, loss=289.0963, lr=6.26e-04]
Epoch 57 TRAIN:  40%|███▉      | 54/136 [00:13<00:16,  4.98it/s, epoch=57, grad_norm=107.366, loss=290.1996, lr=6.26e-04]
Epoch 57 TRAIN:  40%|███▉      | 54/136 [00:13<00:16,  4.98it/s, epoch=57, grad_norm=133.007, loss=310.5629, lr=6.26e-04]
Epoch 57 TRAIN:  40%|███▉      | 54/136 [00:14<00:16,  4.98it/s, epoch=57, grad_norm=130.299, loss=316.8980, lr=6.26e-04]
Epoch 57 TRAIN:  42%|████▏     | 57/136 [00:14<00:16,  4.73it/s, epoch=57, grad_norm=130.299, loss=316.8980, lr=6.26e-04]
Epoch 57 TRAIN:  42%|████▏     | 57/136 [00:14<00:16,  4.73it/s, epoch=57, grad_norm=158.635, loss=315.7196, lr=6.27e-04]
Epoch 57 TRAIN:  42%|████▏     | 57/136 [00:14<00:16,  4.73it/s, epoch=57, grad_norm=173.443, loss=324.4615, lr=6.27e-04]
Epoch 57 TRAIN:  42%|████▏     | 57/136 [00:14<00:16,  4.73it/s, epoch=57, grad_norm=183.790, loss=228.1745, lr=6.27e-04]
Epoch 57 TRAIN:  44%|████▍     | 60/136 [00:14<00:15,  4.75it/s, epoch=57, grad_norm=183.790, loss=228.1745, lr=6.27e-04]
Epoch 57 TRAIN:  44%|████▍     | 60/136 [00:14<00:15,  4.75it/s, epoch=57, grad_norm=101.706, loss=293.2009, lr=6.27e-04]
Epoch 57 TRAIN:  44%|████▍     | 60/136 [00:15<00:15,  4.75it/s, epoch=57, grad_norm=140.133, loss=357.1676, lr=6.27e-04]
Epoch 57 TRAIN:  44%|████▍     | 60/136 [00:15<00:15,  4.75it/s, epoch=57, grad_norm=79.444, loss=313.8450, lr=6.27e-04] 
Epoch 57 TRAIN:  46%|████▋     | 63/136 [00:15<00:15,  4.58it/s, epoch=57, grad_norm=79.444, loss=313.8450, lr=6.27e-04]
Epoch 57 TRAIN:  46%|████▋     | 63/136 [00:15<00:15,  4.58it/s, epoch=57, grad_norm=147.551, loss=320.8358, lr=6.27e-04]
Epoch 57 TRAIN:  46%|████▋     | 63/136 [00:15<00:15,  4.58it/s, epoch=57, grad_norm=138.559, loss=300.2573, lr=6.27e-04]
Epoch 57 TRAIN:  46%|████▋     | 63/136 [00:15<00:15,  4.58it/s, epoch=57, grad_norm=131.718, loss=273.8014, lr=6.27e-04]
Epoch 57 TRAIN:  46%|████▋     | 63/136 [00:16<00:15,  4.58it/s, epoch=57, grad_norm=103.977, loss=316.3930, lr=6.27e-04]
Epoch 57 TRAIN:  49%|████▉     | 67/136 [00:16<00:13,  5.01it/s, epoch=57, grad_norm=103.977, loss=316.3930, lr=6.27e-04]
Epoch 57 TRAIN:  49%|████▉     | 67/136 [00:16<00:13,  5.01it/s, epoch=57, grad_norm=114.044, loss=313.7235, lr=6.27e-04]
Epoch 57 TRAIN:  49%|████▉     | 67/136 [00:16<00:13,  5.01it/s, epoch=57, grad_norm=111.366, loss=289.8114, lr=6.27e-04]
Epoch 57 TRAIN:  49%|████▉     | 67/136 [00:16<00:13,  5.01it/s, epoch=57, grad_norm=111.395, loss=335.5454, lr=6.28e-04]
Epoch 57 TRAIN:  51%|█████▏    | 70/136 [00:16<00:13,  5.02it/s, epoch=57, grad_norm=111.395, loss=335.5454, lr=6.28e-04]
Epoch 57 TRAIN:  51%|█████▏    | 70/136 [00:16<00:13,  5.02it/s, epoch=57, grad_norm=98.888, loss=295.4091, lr=6.28e-04] 
Epoch 57 TRAIN:  51%|█████▏    | 70/136 [00:17<00:13,  5.02it/s, epoch=57, grad_norm=103.414, loss=295.5359, lr=6.28e-04]
Epoch 57 TRAIN:  51%|█████▏    | 70/136 [00:17<00:13,  5.02it/s, epoch=57, grad_norm=134.146, loss=277.8748, lr=6.28e-04]
Epoch 57 TRAIN:  54%|█████▎    | 73/136 [00:17<00:13,  4.61it/s, epoch=57, grad_norm=134.146, loss=277.8748, lr=6.28e-04]
Epoch 57 TRAIN:  54%|█████▎    | 73/136 [00:17<00:13,  4.61it/s, epoch=57, grad_norm=94.773, loss=286.7923, lr=6.28e-04] 
Epoch 57 TRAIN:  54%|█████▎    | 73/136 [00:17<00:13,  4.61it/s, epoch=57, grad_norm=240.499, loss=194.4338, lr=6.28e-04]
Epoch 57 TRAIN:  54%|█████▎    | 73/136 [00:18<00:13,  4.61it/s, epoch=57, grad_norm=169.366, loss=288.2038, lr=6.28e-04]
Epoch 57 TRAIN:  56%|█████▌    | 76/136 [00:18<00:12,  4.73it/s, epoch=57, grad_norm=169.366, loss=288.2038, lr=6.28e-04]
Epoch 57 TRAIN:  56%|█████▌    | 76/136 [00:18<00:12,  4.73it/s, epoch=57, grad_norm=103.310, loss=279.6162, lr=6.28e-04]
Epoch 57 TRAIN:  56%|█████▌    | 76/136 [00:18<00:12,  4.73it/s, epoch=57, grad_norm=156.544, loss=301.7558, lr=6.28e-04]
Epoch 57 TRAIN:  56%|█████▌    | 76/136 [00:18<00:12,  4.73it/s, epoch=57, grad_norm=98.903, loss=236.0762, lr=6.28e-04] 
Epoch 57 TRAIN:  58%|█████▊    | 79/136 [00:18<00:12,  4.50it/s, epoch=57, grad_norm=98.903, loss=236.0762, lr=6.28e-04]
Epoch 57 TRAIN:  58%|█████▊    | 79/136 [00:19<00:12,  4.50it/s, epoch=57, grad_norm=100.359, loss=247.8328, lr=6.28e-04]
Epoch 57 TRAIN:  58%|█████▊    | 79/136 [00:19<00:12,  4.50it/s, epoch=57, grad_norm=208.895, loss=314.9304, lr=6.28e-04]
Epoch 57 TRAIN:  58%|█████▊    | 79/136 [00:19<00:12,  4.50it/s, epoch=57, grad_norm=81.383, loss=244.1627, lr=6.28e-04] 
Epoch 57 TRAIN:  60%|██████    | 82/136 [00:19<00:11,  4.57it/s, epoch=57, grad_norm=81.383, loss=244.1627, lr=6.28e-04]
Epoch 57 TRAIN:  60%|██████    | 82/136 [00:19<00:11,  4.57it/s, epoch=57, grad_norm=150.375, loss=328.2612, lr=6.29e-04]
Epoch 57 TRAIN:  60%|██████    | 82/136 [00:19<00:11,  4.57it/s, epoch=57, grad_norm=159.011, loss=264.7231, lr=6.29e-04]
Epoch 57 TRAIN:  60%|██████    | 82/136 [00:20<00:11,  4.57it/s, epoch=57, grad_norm=165.587, loss=274.0063, lr=6.29e-04]
Epoch 57 TRAIN:  62%|██████▎   | 85/136 [00:20<00:10,  4.76it/s, epoch=57, grad_norm=165.587, loss=274.0063, lr=6.29e-04]
Epoch 57 TRAIN:  62%|██████▎   | 85/136 [00:20<00:10,  4.76it/s, epoch=57, grad_norm=118.012, loss=321.4837, lr=6.29e-04]
Epoch 57 TRAIN:  62%|██████▎   | 85/136 [00:20<00:10,  4.76it/s, epoch=57, grad_norm=136.700, loss=296.2915, lr=6.29e-04]
Epoch 57 TRAIN:  62%|██████▎   | 85/136 [00:20<00:10,  4.76it/s, epoch=57, grad_norm=165.487, loss=267.7533, lr=6.29e-04]
Epoch 57 TRAIN:  65%|██████▍   | 88/136 [00:20<00:10,  4.47it/s, epoch=57, grad_norm=165.487, loss=267.7533, lr=6.29e-04]
Epoch 57 TRAIN:  65%|██████▍   | 88/136 [00:21<00:10,  4.47it/s, epoch=57, grad_norm=118.082, loss=297.1336, lr=6.29e-04]
Epoch 57 TRAIN:  65%|██████▍   | 88/136 [00:21<00:10,  4.47it/s, epoch=57, grad_norm=119.267, loss=292.5899, lr=6.29e-04]
Epoch 57 TRAIN:  65%|██████▍   | 88/136 [00:21<00:10,  4.47it/s, epoch=57, grad_norm=124.512, loss=291.8705, lr=6.29e-04]
Epoch 57 TRAIN:  67%|██████▋   | 91/136 [00:21<00:09,  4.54it/s, epoch=57, grad_norm=124.512, loss=291.8705, lr=6.29e-04]
Epoch 57 TRAIN:  67%|██████▋   | 91/136 [00:21<00:09,  4.54it/s, epoch=57, grad_norm=139.949, loss=289.4370, lr=6.29e-04]
Epoch 57 TRAIN:  67%|██████▋   | 91/136 [00:21<00:09,  4.54it/s, epoch=57, grad_norm=158.558, loss=283.3176, lr=6.29e-04]
Epoch 57 TRAIN:  67%|██████▋   | 91/136 [00:21<00:09,  4.54it/s, epoch=57, grad_norm=140.954, loss=229.2256, lr=6.29e-04]
Epoch 57 TRAIN:  69%|██████▉   | 94/136 [00:21<00:08,  4.77it/s, epoch=57, grad_norm=140.954, loss=229.2256, lr=6.29e-04]
Epoch 57 TRAIN:  69%|██████▉   | 94/136 [00:22<00:08,  4.77it/s, epoch=57, grad_norm=190.220, loss=326.8507, lr=6.30e-04]
Epoch 57 TRAIN:  69%|██████▉   | 94/136 [00:22<00:08,  4.77it/s, epoch=57, grad_norm=125.767, loss=257.8149, lr=6.30e-04]
Epoch 57 TRAIN:  69%|██████▉   | 94/136 [00:22<00:08,  4.77it/s, epoch=57, grad_norm=118.343, loss=295.1089, lr=6.30e-04]
Epoch 57 TRAIN:  71%|███████▏  | 97/136 [00:22<00:08,  4.78it/s, epoch=57, grad_norm=118.343, loss=295.1089, lr=6.30e-04]
Epoch 57 TRAIN:  71%|███████▏  | 97/136 [00:22<00:08,  4.78it/s, epoch=57, grad_norm=130.214, loss=244.9734, lr=6.30e-04]
Epoch 57 TRAIN:  71%|███████▏  | 97/136 [00:23<00:08,  4.78it/s, epoch=57, grad_norm=121.929, loss=241.6751, lr=6.30e-04]
Epoch 57 TRAIN:  71%|███████▏  | 97/136 [00:23<00:08,  4.78it/s, epoch=57, grad_norm=117.582, loss=304.1576, lr=6.30e-04]2026-04-16 20:21:14,147 DEBUG TRAIN | steps/sec 4.276| Batch 57/100 loss 304.157623 loss_att 256.226440 loss_ctc 415.997040 th_accuracy 0.359823 lr 6.2992e-04 grad_norm 117.582245 rank 0

Epoch 57 TRAIN:  74%|███████▎  | 100/136 [00:23<00:07,  4.59it/s, epoch=57, grad_norm=117.582, loss=304.1576, lr=6.30e-04]
Epoch 57 TRAIN:  74%|███████▎  | 100/136 [00:23<00:07,  4.59it/s, epoch=57, grad_norm=107.490, loss=247.0812, lr=6.30e-04]
Epoch 57 TRAIN:  74%|███████▎  | 100/136 [00:23<00:07,  4.59it/s, epoch=57, grad_norm=222.600, loss=274.5795, lr=6.30e-04]
Epoch 57 TRAIN:  74%|███████▎  | 100/136 [00:23<00:07,  4.59it/s, epoch=57, grad_norm=113.137, loss=250.0695, lr=6.30e-04]
Epoch 57 TRAIN:  76%|███████▌  | 103/136 [00:23<00:07,  4.66it/s, epoch=57, grad_norm=113.137, loss=250.0695, lr=6.30e-04]
Epoch 57 TRAIN:  76%|███████▌  | 103/136 [00:24<00:07,  4.66it/s, epoch=57, grad_norm=113.270, loss=248.3146, lr=6.30e-04]
Epoch 57 TRAIN:  76%|███████▌  | 103/136 [00:24<00:07,  4.66it/s, epoch=57, grad_norm=133.975, loss=275.0161, lr=6.30e-04]
Epoch 57 TRAIN:  76%|███████▌  | 103/136 [00:24<00:07,  4.66it/s, epoch=57, grad_norm=202.212, loss=312.2664, lr=6.30e-04]
Epoch 57 TRAIN:  78%|███████▊  | 106/136 [00:24<00:06,  4.68it/s, epoch=57, grad_norm=202.212, loss=312.2664, lr=6.30e-04]
Epoch 57 TRAIN:  78%|███████▊  | 106/136 [00:24<00:06,  4.68it/s, epoch=57, grad_norm=135.429, loss=248.2890, lr=6.30e-04]
Epoch 57 TRAIN:  78%|███████▊  | 106/136 [00:24<00:06,  4.68it/s, epoch=57, grad_norm=245.118, loss=318.0079, lr=6.31e-04]
Epoch 57 TRAIN:  78%|███████▊  | 106/136 [00:25<00:06,  4.68it/s, epoch=57, grad_norm=148.492, loss=243.7767, lr=6.31e-04]
Epoch 57 TRAIN:  80%|████████  | 109/136 [00:25<00:05,  4.84it/s, epoch=57, grad_norm=148.492, loss=243.7767, lr=6.31e-04]
Epoch 57 TRAIN:  80%|████████  | 109/136 [00:25<00:05,  4.84it/s, epoch=57, grad_norm=212.267, loss=286.7734, lr=6.31e-04]
Epoch 57 TRAIN:  80%|████████  | 109/136 [00:25<00:05,  4.84it/s, epoch=57, grad_norm=152.522, loss=246.4305, lr=6.31e-04]
Epoch 57 TRAIN:  80%|████████  | 109/136 [00:25<00:05,  4.84it/s, epoch=57, grad_norm=103.521, loss=247.9891, lr=6.31e-04]
Epoch 57 TRAIN:  82%|████████▏ | 112/136 [00:25<00:05,  4.49it/s, epoch=57, grad_norm=103.521, loss=247.9891, lr=6.31e-04]
Epoch 57 TRAIN:  82%|████████▏ | 112/136 [00:26<00:05,  4.49it/s, epoch=57, grad_norm=146.740, loss=270.5275, lr=6.31e-04]
Epoch 57 TRAIN:  82%|████████▏ | 112/136 [00:26<00:05,  4.49it/s, epoch=57, grad_norm=127.147, loss=257.2552, lr=6.31e-04]
Epoch 57 TRAIN:  82%|████████▏ | 112/136 [00:26<00:05,  4.49it/s, epoch=57, grad_norm=105.237, loss=202.6520, lr=6.31e-04]
Epoch 57 TRAIN:  85%|████████▍ | 115/136 [00:26<00:04,  4.73it/s, epoch=57, grad_norm=105.237, loss=202.6520, lr=6.31e-04]
Epoch 57 TRAIN:  85%|████████▍ | 115/136 [00:26<00:04,  4.73it/s, epoch=57, grad_norm=150.709, loss=290.0697, lr=6.31e-04]
Epoch 57 TRAIN:  85%|████████▍ | 115/136 [00:26<00:04,  4.73it/s, epoch=57, grad_norm=148.850, loss=274.5915, lr=6.31e-04]
Epoch 57 TRAIN:  85%|████████▍ | 115/136 [00:27<00:04,  4.73it/s, epoch=57, grad_norm=299.904, loss=412.5774, lr=6.31e-04]
Epoch 57 TRAIN:  87%|████████▋ | 118/136 [00:27<00:03,  4.65it/s, epoch=57, grad_norm=299.904, loss=412.5774, lr=6.31e-04]
Epoch 57 TRAIN:  87%|████████▋ | 118/136 [00:27<00:03,  4.65it/s, epoch=57, grad_norm=145.894, loss=323.5487, lr=6.31e-04]
Epoch 57 TRAIN:  87%|████████▋ | 118/136 [00:27<00:03,  4.65it/s, epoch=57, grad_norm=259.366, loss=284.9402, lr=6.32e-04]
Epoch 57 TRAIN:  87%|████████▋ | 118/136 [00:27<00:03,  4.65it/s, epoch=57, grad_norm=214.194, loss=308.5754, lr=6.32e-04]
Epoch 57 TRAIN:  89%|████████▉ | 121/136 [00:27<00:03,  4.63it/s, epoch=57, grad_norm=214.194, loss=308.5754, lr=6.32e-04]
Epoch 57 TRAIN:  89%|████████▉ | 121/136 [00:27<00:03,  4.63it/s, epoch=57, grad_norm=192.446, loss=401.7490, lr=6.32e-04]
Epoch 57 TRAIN:  89%|████████▉ | 121/136 [00:28<00:03,  4.63it/s, epoch=57, grad_norm=178.045, loss=325.8487, lr=6.32e-04]
Epoch 57 TRAIN:  89%|████████▉ | 121/136 [00:28<00:03,  4.63it/s, epoch=57, grad_norm=275.555, loss=354.5233, lr=6.32e-04]
Epoch 57 TRAIN:  91%|█████████ | 124/136 [00:28<00:02,  4.78it/s, epoch=57, grad_norm=275.555, loss=354.5233, lr=6.32e-04]
Epoch 57 TRAIN:  91%|█████████ | 124/136 [00:28<00:02,  4.78it/s, epoch=57, grad_norm=171.810, loss=283.2353, lr=6.32e-04]
Epoch 57 TRAIN:  91%|█████████ | 124/136 [00:28<00:02,  4.78it/s, epoch=57, grad_norm=157.136, loss=280.4926, lr=6.32e-04]
Epoch 57 TRAIN:  91%|█████████ | 124/136 [00:29<00:02,  4.78it/s, epoch=57, grad_norm=272.844, loss=360.4119, lr=6.32e-04]
Epoch 57 TRAIN:  93%|█████████▎| 127/136 [00:29<00:01,  4.73it/s, epoch=57, grad_norm=272.844, loss=360.4119, lr=6.32e-04]
Epoch 57 TRAIN:  93%|█████████▎| 127/136 [00:29<00:01,  4.73it/s, epoch=57, grad_norm=178.499, loss=322.3744, lr=6.32e-04]
Epoch 57 TRAIN:  93%|█████████▎| 127/136 [00:29<00:01,  4.73it/s, epoch=57, grad_norm=161.197, loss=310.7590, lr=6.32e-04]
Epoch 57 TRAIN:  93%|█████████▎| 127/136 [00:29<00:01,  4.73it/s, epoch=57, grad_norm=347.377, loss=353.1401, lr=6.32e-04]
Epoch 57 TRAIN:  93%|█████████▎| 127/136 [00:29<00:01,  4.73it/s, epoch=57, grad_norm=124.837, loss=318.0367, lr=6.32e-04]
Epoch 57 TRAIN:  96%|█████████▋| 131/136 [00:29<00:00,  5.09it/s, epoch=57, grad_norm=124.837, loss=318.0367, lr=6.32e-04]
Epoch 57 TRAIN:  96%|█████████▋| 131/136 [00:29<00:00,  5.09it/s, epoch=57, grad_norm=162.307, loss=245.0583, lr=6.32e-04]
Epoch 57 TRAIN:  96%|█████████▋| 131/136 [00:30<00:00,  5.09it/s, epoch=57, grad_norm=215.445, loss=306.1150, lr=6.33e-04]
Epoch 57 TRAIN:  96%|█████████▋| 131/136 [00:30<00:00,  5.09it/s, epoch=57, grad_norm=210.359, loss=280.4850, lr=6.33e-04]
Epoch 57 TRAIN:  99%|█████████▊| 134/136 [00:30<00:00,  4.75it/s, epoch=57, grad_norm=210.359, loss=280.4850, lr=6.33e-04]
Epoch 57 TRAIN:  99%|█████████▊| 134/136 [00:30<00:00,  4.75it/s, epoch=57, grad_norm=419.174, loss=277.8260, lr=6.33e-04]
Epoch 57 TRAIN:  99%|█████████▊| 134/136 [00:30<00:00,  4.75it/s, epoch=57, grad_norm=326.111, loss=289.7881, lr=6.33e-04]
Epoch 57 TRAIN: 100%|██████████| 136/136 [00:30<00:00,  4.43it/s, epoch=57, grad_norm=326.111, loss=289.7881, lr=6.33e-04]

CV:   0%|          | 0/6 [00:00<?, ?it/s]/home/rookie/Documents/batuka/projects/wenet/wenet/wenet/utils/train_utils.py:689: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  amp_autocast(enabled=dtype is not None,

CV:  17%|█▋        | 1/6 [00:00<00:03,  1.32it/s]
                                                 
2026-04-16 20:21:22,396 INFO Epoch 57 Step 7910 CV info lr 6.3280e-04 cv_loss 501.0748401988636 rank 0 acc 0.34865205983320874
2026-04-16 20:21:22,398 INFO [Rank 0] Checkpoint: save to checkpoint exp/conformer_mongolian/epoch_57.pt
2026-04-16 20:21:22,516 INFO Epoch 58 Step 7910 TRAIN info lr 6.3280e-04 rank 0
2026-04-16 20:21:22,536 INFO using accumulate grad, new batch size is 1 times larger than before

Epoch 58 TRAIN:   0%|          | 0/136 [00:00<?, ?it/s]
Epoch 58 TRAIN:   0%|          | 0/136 [00:04<?, ?it/s, epoch=58, grad_norm=598.138, loss=430.9284, lr=6.33e-04]
Epoch 58 TRAIN:   1%|          | 1/136 [00:04<09:29,  4.22s/it, epoch=58, grad_norm=598.138, loss=430.9284, lr=6.33e-04]
Epoch 58 TRAIN:   1%|          | 1/136 [00:04<09:29,  4.22s/it, epoch=58, grad_norm=559.975, loss=396.3558, lr=6.33e-04]
Epoch 58 TRAIN:   1%|          | 1/136 [00:04<09:29,  4.22s/it, epoch=58, grad_norm=609.360, loss=402.2562, lr=6.33e-04]
Epoch 58 TRAIN:   1%|          | 1/136 [00:04<09:29,  4.22s/it, epoch=58, grad_norm=657.443, loss=466.4095, lr=6.33e-04]
Epoch 58 TRAIN:   1%|          | 1/136 [00:04<09:29,  4.22s/it, epoch=58, grad_norm=441.402, loss=349.2285, lr=6.33e-04]
Epoch 58 TRAIN:   1%|          | 1/136 [00:04<09:29,  4.22s/it, epoch=58, grad_norm=344.317, loss=324.3473, lr=6.33e-04]
Epoch 58 TRAIN:   1%|          | 1/136 [00:04<09:29,  4.22s/it, epoch=58, grad_norm=312.664, loss=325.3978, lr=6.33e-04]
Epoch 58 TRAIN:   1%|          | 1/136 [00:04<09:29,  4.22s/it, epoch=58, grad_norm=297.868, loss=300.1896, lr=6.33e-04]
Epoch 58 TRAIN:   1%|          | 1/136 [00:04<09:29,  4.22s/it, epoch=58, grad_norm=177.804, loss=284.5994, lr=6.34e-04]
Epoch 58 TRAIN:   7%|▋         | 9/136 [00:04<00:52,  2.43it/s, epoch=58, grad_norm=177.804, loss=284.5994, lr=6.34e-04]
Epoch 58 TRAIN:   7%|▋         | 9/136 [00:05<00:52,  2.43it/s, epoch=58, grad_norm=131.991, loss=282.1124, lr=6.34e-04]
Epoch 58 TRAIN:   7%|▋         | 9/136 [00:05<00:52,  2.43it/s, epoch=58, grad_norm=79.534, loss=267.4234, lr=6.34e-04] 
Epoch 58 TRAIN:   7%|▋         | 9/136 [00:05<00:52,  2.43it/s, epoch=58, grad_norm=123.741, loss=260.7361, lr=6.34e-04]
Epoch 58 TRAIN:   7%|▋         | 9/136 [00:05<00:52,  2.43it/s, epoch=58, grad_norm=148.575, loss=307.1581, lr=6.34e-04]
Epoch 58 TRAIN:  10%|▉         | 13/136 [00:05<00:38,  3.20it/s, epoch=58, grad_norm=148.575, loss=307.1581, lr=6.34e-04]
Epoch 58 TRAIN:  10%|▉         | 13/136 [00:05<00:38,  3.20it/s, epoch=58, grad_norm=118.006, loss=328.2527, lr=6.34e-04]
Epoch 58 TRAIN:  10%|▉         | 13/136 [00:05<00:38,  3.20it/s, epoch=58, grad_norm=164.231, loss=264.4020, lr=6.34e-04]
Epoch 58 TRAIN:  10%|▉         | 13/136 [00:05<00:38,  3.20it/s, epoch=58, grad_norm=128.117, loss=297.1203, lr=6.34e-04]
Epoch 58 TRAIN:  10%|▉         | 13/136 [00:06<00:38,  3.20it/s, epoch=58, grad_norm=136.905, loss=291.9309, lr=6.34e-04]
Epoch 58 TRAIN:  12%|█▎        | 17/136 [00:06<00:30,  3.84it/s, epoch=58, grad_norm=136.905, loss=291.9309, lr=6.34e-04]
Epoch 58 TRAIN:  12%|█▎        | 17/136 [00:06<00:30,  3.84it/s, epoch=58, grad_norm=108.434, loss=325.8474, lr=6.34e-04]
Epoch 58 TRAIN:  12%|█▎        | 17/136 [00:06<00:30,  3.84it/s, epoch=58, grad_norm=110.656, loss=289.6949, lr=6.34e-04]
Epoch 58 TRAIN:  12%|█▎        | 17/136 [00:06<00:30,  3.84it/s, epoch=58, grad_norm=110.088, loss=277.1306, lr=6.34e-04]
Epoch 58 TRAIN:  12%|█▎        | 17/136 [00:06<00:30,  3.84it/s, epoch=58, grad_norm=115.476, loss=270.7959, lr=6.34e-04]
Epoch 58 TRAIN:  15%|█▌        | 21/136 [00:06<00:26,  4.30it/s, epoch=58, grad_norm=115.476, loss=270.7959, lr=6.34e-04]
Epoch 58 TRAIN:  15%|█▌        | 21/136 [00:07<00:26,  4.30it/s, epoch=58, grad_norm=104.442, loss=300.6324, lr=6.35e-04]
Epoch 58 TRAIN:  15%|█▌        | 21/136 [00:07<00:26,  4.30it/s, epoch=58, grad_norm=150.437, loss=342.0023, lr=6.35e-04]
Epoch 58 TRAIN:  15%|█▌        | 21/136 [00:07<00:26,  4.30it/s, epoch=58, grad_norm=177.332, loss=344.5626, lr=6.35e-04]
Epoch 58 TRAIN:  15%|█▌        | 21/136 [00:07<00:26,  4.30it/s, epoch=58, grad_norm=114.742, loss=293.0629, lr=6.35e-04]
Epoch 58 TRAIN:  18%|█▊        | 25/136 [00:07<00:24,  4.51it/s, epoch=58, grad_norm=114.742, loss=293.0629, lr=6.35e-04]
Epoch 58 TRAIN:  18%|█▊        | 25/136 [00:07<00:24,  4.51it/s, epoch=58, grad_norm=97.860, loss=290.0673, lr=6.35e-04] 
Epoch 58 TRAIN:  18%|█▊        | 25/136 [00:08<00:24,  4.51it/s, epoch=58, grad_norm=84.291, loss=283.0229, lr=6.35e-04]
Epoch 58 TRAIN:  18%|█▊        | 25/136 [00:08<00:24,  4.51it/s, epoch=58, grad_norm=163.579, loss=343.8156, lr=6.35e-04]
Epoch 58 TRAIN:  21%|██        | 28/136 [00:08<00:22,  4.75it/s, epoch=58, grad_norm=163.579, loss=343.8156, lr=6.35e-04]
Epoch 58 TRAIN:  21%|██        | 28/136 [00:08<00:22,  4.75it/s, epoch=58, grad_norm=121.217, loss=237.2932, lr=6.35e-04]
Epoch 58 TRAIN:  21%|██        | 28/136 [00:08<00:22,  4.75it/s, epoch=58, grad_norm=87.006, loss=259.2808, lr=6.35e-04] 
Epoch 58 TRAIN:  21%|██        | 28/136 [00:08<00:22,  4.75it/s, epoch=58, grad_norm=88.306, loss=277.8690, lr=6.35e-04]
Epoch 58 TRAIN:  23%|██▎       | 31/136 [00:08<00:21,  4.98it/s, epoch=58, grad_norm=88.306, loss=277.8690, lr=6.35e-04]
Epoch 58 TRAIN:  23%|██▎       | 31/136 [00:08<00:21,  4.98it/s, epoch=58, grad_norm=126.646, loss=280.7547, lr=6.35e-04]
Epoch 58 TRAIN:  23%|██▎       | 31/136 [00:09<00:21,  4.98it/s, epoch=58, grad_norm=118.886, loss=293.9430, lr=6.35e-04]
Epoch 58 TRAIN:  23%|██▎       | 31/136 [00:09<00:21,  4.98it/s, epoch=58, grad_norm=121.173, loss=296.5301, lr=6.36e-04]
Epoch 58 TRAIN:  25%|██▌       | 34/136 [00:09<00:20,  4.99it/s, epoch=58, grad_norm=121.173, loss=296.5301, lr=6.36e-04]
Epoch 58 TRAIN:  25%|██▌       | 34/136 [00:09<00:20,  4.99it/s, epoch=58, grad_norm=108.991, loss=300.7170, lr=6.36e-04]
Epoch 58 TRAIN:  25%|██▌       | 34/136 [00:09<00:20,  4.99it/s, epoch=58, grad_norm=111.314, loss=300.2398, lr=6.36e-04]
Epoch 58 TRAIN:  25%|██▌       | 34/136 [00:09<00:20,  4.99it/s, epoch=58, grad_norm=79.066, loss=289.2310, lr=6.36e-04] 
Epoch 58 TRAIN:  27%|██▋       | 37/136 [00:09<00:19,  4.99it/s, epoch=58, grad_norm=79.066, loss=289.2310, lr=6.36e-04]
Epoch 58 TRAIN:  27%|██▋       | 37/136 [00:10<00:19,  4.99it/s, epoch=58, grad_norm=100.254, loss=280.6014, lr=6.36e-04]
Epoch 58 TRAIN:  27%|██▋       | 37/136 [00:10<00:19,  4.99it/s, epoch=58, grad_norm=91.813, loss=309.7365, lr=6.36e-04] 
Epoch 58 TRAIN:  27%|██▋       | 37/136 [00:10<00:19,  4.99it/s, epoch=58, grad_norm=88.409, loss=270.2427, lr=6.36e-04]
Epoch 58 TRAIN:  29%|██▉       | 40/136 [00:10<00:19,  4.98it/s, epoch=58, grad_norm=88.409, loss=270.2427, lr=6.36e-04]
Epoch 58 TRAIN:  29%|██▉       | 40/136 [00:10<00:19,  4.98it/s, epoch=58, grad_norm=146.986, loss=318.1089, lr=6.36e-04]
Epoch 58 TRAIN:  29%|██▉       | 40/136 [00:10<00:19,  4.98it/s, epoch=58, grad_norm=111.023, loss=318.6583, lr=6.36e-04]
Epoch 58 TRAIN:  29%|██▉       | 40/136 [00:11<00:19,  4.98it/s, epoch=58, grad_norm=133.437, loss=276.8546, lr=6.36e-04]
Epoch 58 TRAIN:  32%|███▏      | 43/136 [00:11<00:19,  4.89it/s, epoch=58, grad_norm=133.437, loss=276.8546, lr=6.36e-04]
Epoch 58 TRAIN:  32%|███▏      | 43/136 [00:11<00:19,  4.89it/s, epoch=58, grad_norm=122.433, loss=283.4138, lr=6.36e-04]
Epoch 58 TRAIN:  32%|███▏      | 43/136 [00:11<00:19,  4.89it/s, epoch=58, grad_norm=151.117, loss=246.2038, lr=6.36e-04]
Epoch 58 TRAIN:  32%|███▏      | 43/136 [00:11<00:19,  4.89it/s, epoch=58, grad_norm=117.711, loss=287.8493, lr=6.36e-04]
Epoch 58 TRAIN:  34%|███▍      | 46/136 [00:11<00:18,  4.87it/s, epoch=58, grad_norm=117.711, loss=287.8493, lr=6.36e-04]
Epoch 58 TRAIN:  34%|███▍      | 46/136 [00:12<00:18,  4.87it/s, epoch=58, grad_norm=108.202, loss=309.9068, lr=6.37e-04]
Epoch 58 TRAIN:  34%|███▍      | 46/136 [00:12<00:18,  4.87it/s, epoch=58, grad_norm=135.500, loss=321.1667, lr=6.37e-04]
Epoch 58 TRAIN:  34%|███▍      | 46/136 [00:12<00:18,  4.87it/s, epoch=58, grad_norm=102.067, loss=278.7851, lr=6.37e-04]
Epoch 58 TRAIN:  36%|███▌      | 49/136 [00:12<00:18,  4.82it/s, epoch=58, grad_norm=102.067, loss=278.7851, lr=6.37e-04]
Epoch 58 TRAIN:  36%|███▌      | 49/136 [00:12<00:18,  4.82it/s, epoch=58, grad_norm=183.405, loss=342.9322, lr=6.37e-04]
Epoch 58 TRAIN:  36%|███▌      | 49/136 [00:12<00:18,  4.82it/s, epoch=58, grad_norm=86.198, loss=261.7157, lr=6.37e-04] 
Epoch 58 TRAIN:  36%|███▌      | 49/136 [00:13<00:18,  4.82it/s, epoch=58, grad_norm=104.452, loss=280.6205, lr=6.37e-04]
Epoch 58 TRAIN:  38%|███▊      | 52/136 [00:13<00:17,  4.77it/s, epoch=58, grad_norm=104.452, loss=280.6205, lr=6.37e-04]
Epoch 58 TRAIN:  38%|███▊      | 52/136 [00:13<00:17,  4.77it/s, epoch=58, grad_norm=106.363, loss=297.9332, lr=6.37e-04]
Epoch 58 TRAIN:  38%|███▊      | 52/136 [00:13<00:17,  4.77it/s, epoch=58, grad_norm=127.246, loss=318.7037, lr=6.37e-04]
Epoch 58 TRAIN:  38%|███▊      | 52/136 [00:13<00:17,  4.77it/s, epoch=58, grad_norm=94.685, loss=256.3990, lr=6.37e-04] 
Epoch 58 TRAIN:  40%|████      | 55/136 [00:13<00:17,  4.73it/s, epoch=58, grad_norm=94.685, loss=256.3990, lr=6.37e-04]
Epoch 58 TRAIN:  40%|████      | 55/136 [00:13<00:17,  4.73it/s, epoch=58, grad_norm=86.787, loss=290.4769, lr=6.37e-04]
Epoch 58 TRAIN:  40%|████      | 55/136 [00:14<00:17,  4.73it/s, epoch=58, grad_norm=192.719, loss=238.4809, lr=6.37e-04]
Epoch 58 TRAIN:  40%|████      | 55/136 [00:14<00:17,  4.73it/s, epoch=58, grad_norm=182.375, loss=320.5656, lr=6.37e-04]
Epoch 58 TRAIN:  43%|████▎     | 58/136 [00:14<00:16,  4.84it/s, epoch=58, grad_norm=182.375, loss=320.5656, lr=6.37e-04]
Epoch 58 TRAIN:  43%|████▎     | 58/136 [00:14<00:16,  4.84it/s, epoch=58, grad_norm=132.962, loss=280.9389, lr=6.38e-04]
Epoch 58 TRAIN:  43%|████▎     | 58/136 [00:14<00:16,  4.84it/s, epoch=58, grad_norm=142.322, loss=280.8540, lr=6.38e-04]
Epoch 58 TRAIN:  43%|████▎     | 58/136 [00:15<00:16,  4.84it/s, epoch=58, grad_norm=210.588, loss=350.4128, lr=6.38e-04]
Epoch 58 TRAIN:  45%|████▍     | 61/136 [00:15<00:16,  4.57it/s, epoch=58, grad_norm=210.588, loss=350.4128, lr=6.38e-04]
Epoch 58 TRAIN:  45%|████▍     | 61/136 [00:15<00:16,  4.57it/s, epoch=58, grad_norm=117.519, loss=321.3852, lr=6.38e-04]
Epoch 58 TRAIN:  45%|████▍     | 61/136 [00:15<00:16,  4.57it/s, epoch=58, grad_norm=99.392, loss=293.1852, lr=6.38e-04] 
Epoch 58 TRAIN:  45%|████▍     | 61/136 [00:15<00:16,  4.57it/s, epoch=58, grad_norm=102.485, loss=295.3414, lr=6.38e-04]
Epoch 58 TRAIN:  47%|████▋     | 64/136 [00:15<00:15,  4.63it/s, epoch=58, grad_norm=102.485, loss=295.3414, lr=6.38e-04]
Epoch 58 TRAIN:  47%|████▋     | 64/136 [00:15<00:15,  4.63it/s, epoch=58, grad_norm=112.552, loss=285.2820, lr=6.38e-04]
Epoch 58 TRAIN:  47%|████▋     | 64/136 [00:16<00:15,  4.63it/s, epoch=58, grad_norm=139.989, loss=323.7342, lr=6.38e-04]
Epoch 58 TRAIN:  47%|████▋     | 64/136 [00:16<00:15,  4.63it/s, epoch=58, grad_norm=130.401, loss=293.8419, lr=6.38e-04]
Epoch 58 TRAIN:  49%|████▉     | 67/136 [00:16<00:14,  4.67it/s, epoch=58, grad_norm=130.401, loss=293.8419, lr=6.38e-04]
Epoch 58 TRAIN:  49%|████▉     | 67/136 [00:16<00:14,  4.67it/s, epoch=58, grad_norm=136.971, loss=261.3644, lr=6.38e-04]
Epoch 58 TRAIN:  49%|████▉     | 67/136 [00:16<00:14,  4.67it/s, epoch=58, grad_norm=88.740, loss=292.2137, lr=6.38e-04] 
Epoch 58 TRAIN:  49%|████▉     | 67/136 [00:16<00:14,  4.67it/s, epoch=58, grad_norm=150.836, loss=327.1048, lr=6.38e-04]
Epoch 58 TRAIN:  51%|█████▏    | 70/136 [00:16<00:14,  4.68it/s, epoch=58, grad_norm=150.836, loss=327.1048, lr=6.38e-04]
Epoch 58 TRAIN:  51%|█████▏    | 70/136 [00:17<00:14,  4.68it/s, epoch=58, grad_norm=111.399, loss=307.3573, lr=6.38e-04]
Epoch 58 TRAIN:  51%|█████▏    | 70/136 [00:17<00:14,  4.68it/s, epoch=58, grad_norm=107.857, loss=341.8305, lr=6.39e-04]
Epoch 58 TRAIN:  51%|█████▏    | 70/136 [00:17<00:14,  4.68it/s, epoch=58, grad_norm=108.294, loss=261.6210, lr=6.39e-04]
Epoch 58 TRAIN:  54%|█████▎    | 73/136 [00:17<00:13,  4.62it/s, epoch=58, grad_norm=108.294, loss=261.6210, lr=6.39e-04]
Epoch 58 TRAIN:  54%|█████▎    | 73/136 [00:17<00:13,  4.62it/s, epoch=58, grad_norm=103.394, loss=308.9382, lr=6.39e-04]
Epoch 58 TRAIN:  54%|█████▎    | 73/136 [00:18<00:13,  4.62it/s, epoch=58, grad_norm=186.825, loss=254.7224, lr=6.39e-04]
Epoch 58 TRAIN:  54%|█████▎    | 73/136 [00:18<00:13,  4.62it/s, epoch=58, grad_norm=152.254, loss=255.0760, lr=6.39e-04]
Epoch 58 TRAIN:  56%|█████▌    | 76/136 [00:18<00:13,  4.57it/s, epoch=58, grad_norm=152.254, loss=255.0760, lr=6.39e-04]
Epoch 58 TRAIN:  56%|█████▌    | 76/136 [00:18<00:13,  4.57it/s, epoch=58, grad_norm=149.487, loss=308.2408, lr=6.39e-04]
Epoch 58 TRAIN:  56%|█████▌    | 76/136 [00:18<00:13,  4.57it/s, epoch=58, grad_norm=143.399, loss=292.7507, lr=6.39e-04]
Epoch 58 TRAIN:  56%|█████▌    | 76/136 [00:18<00:13,  4.57it/s, epoch=58, grad_norm=119.207, loss=224.0211, lr=6.39e-04]
Epoch 58 TRAIN:  58%|█████▊    | 79/136 [00:18<00:12,  4.63it/s, epoch=58, grad_norm=119.207, loss=224.0211, lr=6.39e-04]
Epoch 58 TRAIN:  58%|█████▊    | 79/136 [00:19<00:12,  4.63it/s, epoch=58, grad_norm=108.312, loss=291.0364, lr=6.39e-04]
Epoch 58 TRAIN:  58%|█████▊    | 79/136 [00:19<00:12,  4.63it/s, epoch=58, grad_norm=125.735, loss=321.3215, lr=6.39e-04]
Epoch 58 TRAIN:  58%|█████▊    | 79/136 [00:19<00:12,  4.63it/s, epoch=58, grad_norm=129.027, loss=285.9189, lr=6.39e-04]
Epoch 58 TRAIN:  60%|██████    | 82/136 [00:19<00:11,  4.62it/s, epoch=58, grad_norm=129.027, loss=285.9189, lr=6.39e-04]
Epoch 58 TRAIN:  60%|██████    | 82/136 [00:19<00:11,  4.62it/s, epoch=58, grad_norm=150.625, loss=254.9023, lr=6.39e-04]
Epoch 58 TRAIN:  60%|██████    | 82/136 [00:20<00:11,  4.62it/s, epoch=58, grad_norm=171.718, loss=312.4545, lr=6.40e-04]
Epoch 58 TRAIN:  60%|██████    | 82/136 [00:20<00:11,  4.62it/s, epoch=58, grad_norm=121.954, loss=253.0193, lr=6.40e-04]
Epoch 58 TRAIN:  62%|██████▎   | 85/136 [00:20<00:11,  4.57it/s, epoch=58, grad_norm=121.954, loss=253.0193, lr=6.40e-04]
Epoch 58 TRAIN:  62%|██████▎   | 85/136 [00:20<00:11,  4.57it/s, epoch=58, grad_norm=131.981, loss=291.5838, lr=6.40e-04]
Epoch 58 TRAIN:  62%|██████▎   | 85/136 [00:20<00:11,  4.57it/s, epoch=58, grad_norm=119.490, loss=266.1490, lr=6.40e-04]
Epoch 58 TRAIN:  62%|██████▎   | 85/136 [00:20<00:11,  4.57it/s, epoch=58, grad_norm=121.347, loss=271.4660, lr=6.40e-04]
Epoch 58 TRAIN:  65%|██████▍   | 88/136 [00:20<00:10,  4.49it/s, epoch=58, grad_norm=121.347, loss=271.4660, lr=6.40e-04]
Epoch 58 TRAIN:  65%|██████▍   | 88/136 [00:21<00:10,  4.49it/s, epoch=58, grad_norm=107.477, loss=277.7686, lr=6.40e-04]
Epoch 58 TRAIN:  65%|██████▍   | 88/136 [00:21<00:10,  4.49it/s, epoch=58, grad_norm=112.054, loss=223.4124, lr=6.40e-04]
Epoch 58 TRAIN:  65%|██████▍   | 88/136 [00:21<00:10,  4.49it/s, epoch=58, grad_norm=171.930, loss=224.8732, lr=6.40e-04]
Epoch 58 TRAIN:  65%|██████▍   | 88/136 [00:21<00:10,  4.49it/s, epoch=58, grad_norm=107.766, loss=278.1397, lr=6.40e-04]
Epoch 58 TRAIN:  68%|██████▊   | 92/136 [00:21<00:09,  4.87it/s, epoch=58, grad_norm=107.766, loss=278.1397, lr=6.40e-04]
Epoch 58 TRAIN:  68%|██████▊   | 92/136 [00:21<00:09,  4.87it/s, epoch=58, grad_norm=222.795, loss=332.7733, lr=6.40e-04]
Epoch 58 TRAIN:  68%|██████▊   | 92/136 [00:22<00:09,  4.87it/s, epoch=58, grad_norm=175.767, loss=272.5775, lr=6.40e-04]
Epoch 58 TRAIN:  68%|██████▊   | 92/136 [00:22<00:09,  4.87it/s, epoch=58, grad_norm=260.345, loss=291.6410, lr=6.40e-04]
Epoch 58 TRAIN:  70%|██████▉   | 95/136 [00:22<00:08,  4.77it/s, epoch=58, grad_norm=260.345, loss=291.6410, lr=6.40e-04]
Epoch 58 TRAIN:  70%|██████▉   | 95/136 [00:22<00:08,  4.77it/s, epoch=58, grad_norm=95.878, loss=229.4713, lr=6.40e-04] 
Epoch 58 TRAIN:  70%|██████▉   | 95/136 [00:22<00:08,  4.77it/s, epoch=58, grad_norm=242.129, loss=297.4079, lr=6.41e-04]
Epoch 58 TRAIN:  70%|██████▉   | 95/136 [00:23<00:08,  4.77it/s, epoch=58, grad_norm=130.209, loss=268.5806, lr=6.41e-04]
Epoch 58 TRAIN:  72%|███████▏  | 98/136 [00:23<00:08,  4.58it/s, epoch=58, grad_norm=130.209, loss=268.5806, lr=6.41e-04]
Epoch 58 TRAIN:  72%|███████▏  | 98/136 [00:23<00:08,  4.58it/s, epoch=58, grad_norm=134.629, loss=268.1727, lr=6.41e-04]
Epoch 58 TRAIN:  72%|███████▏  | 98/136 [00:23<00:08,  4.58it/s, epoch=58, grad_norm=168.185, loss=295.6987, lr=6.41e-04]2026-04-16 20:21:46,045 DEBUG TRAIN | steps/sec 4.264| Batch 58/100 loss 295.698669 loss_att 253.606812 loss_ctc 393.912964 th_accuracy 0.319728 lr 6.4080e-04 grad_norm 168.184830 rank 0

Epoch 58 TRAIN:  72%|███████▏  | 98/136 [00:23<00:08,  4.58it/s, epoch=58, grad_norm=146.793, loss=302.8568, lr=6.41e-04]
Epoch 58 TRAIN:  74%|███████▍  | 101/136 [00:23<00:07,  4.50it/s, epoch=58, grad_norm=146.793, loss=302.8568, lr=6.41e-04]
Epoch 58 TRAIN:  74%|███████▍  | 101/136 [00:23<00:07,  4.50it/s, epoch=58, grad_norm=177.762, loss=194.2059, lr=6.41e-04]
Epoch 58 TRAIN:  74%|███████▍  | 101/136 [00:24<00:07,  4.50it/s, epoch=58, grad_norm=95.731, loss=231.8840, lr=6.41e-04] 
Epoch 58 TRAIN:  74%|███████▍  | 101/136 [00:24<00:07,  4.50it/s, epoch=58, grad_norm=132.128, loss=297.0518, lr=6.41e-04]
Epoch 58 TRAIN:  76%|███████▋  | 104/136 [00:24<00:07,  4.30it/s, epoch=58, grad_norm=132.128, loss=297.0518, lr=6.41e-04]
Epoch 58 TRAIN:  76%|███████▋  | 104/136 [00:24<00:07,  4.30it/s, epoch=58, grad_norm=100.356, loss=250.0013, lr=6.41e-04]
Epoch 58 TRAIN:  76%|███████▋  | 104/136 [00:24<00:07,  4.30it/s, epoch=58, grad_norm=115.162, loss=257.1661, lr=6.41e-04]
Epoch 58 TRAIN:  76%|███████▋  | 104/136 [00:24<00:07,  4.30it/s, epoch=58, grad_norm=124.287, loss=298.2076, lr=6.41e-04]
Epoch 58 TRAIN:  76%|███████▋  | 104/136 [00:25<00:07,  4.30it/s, epoch=58, grad_norm=166.680, loss=239.2306, lr=6.41e-04]
Epoch 58 TRAIN:  79%|███████▉  | 108/136 [00:25<00:05,  4.75it/s, epoch=58, grad_norm=166.680, loss=239.2306, lr=6.41e-04]
Epoch 58 TRAIN:  79%|███████▉  | 108/136 [00:25<00:05,  4.75it/s, epoch=58, grad_norm=230.864, loss=337.3766, lr=6.42e-04]
Epoch 58 TRAIN:  79%|███████▉  | 108/136 [00:25<00:05,  4.75it/s, epoch=58, grad_norm=240.424, loss=339.9182, lr=6.42e-04]
Epoch 58 TRAIN:  79%|███████▉  | 108/136 [00:25<00:05,  4.75it/s, epoch=58, grad_norm=151.881, loss=295.0779, lr=6.42e-04]
Epoch 58 TRAIN:  82%|████████▏ | 111/136 [00:25<00:05,  4.68it/s, epoch=58, grad_norm=151.881, loss=295.0779, lr=6.42e-04]
Epoch 58 TRAIN:  82%|████████▏ | 111/136 [00:26<00:05,  4.68it/s, epoch=58, grad_norm=165.531, loss=300.7794, lr=6.42e-04]
Epoch 58 TRAIN:  82%|████████▏ | 111/136 [00:26<00:05,  4.68it/s, epoch=58, grad_norm=121.492, loss=287.5831, lr=6.42e-04]
Epoch 58 TRAIN:  82%|████████▏ | 111/136 [00:26<00:05,  4.68it/s, epoch=58, grad_norm=148.256, loss=283.7686, lr=6.42e-04]
Epoch 58 TRAIN:  84%|████████▍ | 114/136 [00:26<00:04,  4.58it/s, epoch=58, grad_norm=148.256, loss=283.7686, lr=6.42e-04]
Epoch 58 TRAIN:  84%|████████▍ | 114/136 [00:26<00:04,  4.58it/s, epoch=58, grad_norm=259.666, loss=371.4487, lr=6.42e-04]
Epoch 58 TRAIN:  84%|████████▍ | 114/136 [00:26<00:04,  4.58it/s, epoch=58, grad_norm=353.562, loss=271.9720, lr=6.42e-04]
Epoch 58 TRAIN:  84%|████████▍ | 114/136 [00:27<00:04,  4.58it/s, epoch=58, grad_norm=222.167, loss=329.0242, lr=6.42e-04]
Epoch 58 TRAIN:  84%|████████▍ | 114/136 [00:27<00:04,  4.58it/s, epoch=58, grad_norm=117.066, loss=246.7265, lr=6.42e-04]
Epoch 58 TRAIN:  87%|████████▋ | 118/136 [00:27<00:03,  4.96it/s, epoch=58, grad_norm=117.066, loss=246.7265, lr=6.42e-04]
Epoch 58 TRAIN:  87%|████████▋ | 118/136 [00:27<00:03,  4.96it/s, epoch=58, grad_norm=118.418, loss=258.9305, lr=6.42e-04]
Epoch 58 TRAIN:  87%|████████▋ | 118/136 [00:27<00:03,  4.96it/s, epoch=58, grad_norm=120.756, loss=275.9397, lr=6.42e-04]
Epoch 58 TRAIN:  87%|████████▋ | 118/136 [00:27<00:03,  4.96it/s, epoch=58, grad_norm=229.484, loss=302.9536, lr=6.42e-04]
Epoch 58 TRAIN:  89%|████████▉ | 121/136 [00:27<00:03,  4.81it/s, epoch=58, grad_norm=229.484, loss=302.9536, lr=6.42e-04]
Epoch 58 TRAIN:  89%|████████▉ | 121/136 [00:28<00:03,  4.81it/s, epoch=58, grad_norm=208.104, loss=263.6600, lr=6.43e-04]
Epoch 58 TRAIN:  89%|████████▉ | 121/136 [00:28<00:03,  4.81it/s, epoch=58, grad_norm=222.602, loss=329.1795, lr=6.43e-04]
Epoch 58 TRAIN:  89%|████████▉ | 121/136 [00:28<00:03,  4.81it/s, epoch=58, grad_norm=139.524, loss=214.7839, lr=6.43e-04]
Epoch 58 TRAIN:  91%|█████████ | 124/136 [00:28<00:02,  4.88it/s, epoch=58, grad_norm=139.524, loss=214.7839, lr=6.43e-04]
Epoch 58 TRAIN:  91%|█████████ | 124/136 [00:28<00:02,  4.88it/s, epoch=58, grad_norm=113.502, loss=242.3719, lr=6.43e-04]
Epoch 58 TRAIN:  91%|█████████ | 124/136 [00:28<00:02,  4.88it/s, epoch=58, grad_norm=209.319, loss=291.7276, lr=6.43e-04]
Epoch 58 TRAIN:  91%|█████████ | 124/136 [00:29<00:02,  4.88it/s, epoch=58, grad_norm=153.810, loss=292.1658, lr=6.43e-04]
Epoch 58 TRAIN:  93%|█████████▎| 127/136 [00:29<00:01,  4.58it/s, epoch=58, grad_norm=153.810, loss=292.1658, lr=6.43e-04]
Epoch 58 TRAIN:  93%|█████████▎| 127/136 [00:29<00:01,  4.58it/s, epoch=58, grad_norm=217.573, loss=349.7142, lr=6.43e-04]
Epoch 58 TRAIN:  93%|█████████▎| 127/136 [00:29<00:01,  4.58it/s, epoch=58, grad_norm=410.965, loss=318.3602, lr=6.43e-04]
Epoch 58 TRAIN:  93%|█████████▎| 127/136 [00:29<00:01,  4.58it/s, epoch=58, grad_norm=129.382, loss=329.4101, lr=6.43e-04]
Epoch 58 TRAIN:  96%|█████████▌| 130/136 [00:29<00:01,  4.47it/s, epoch=58, grad_norm=129.382, loss=329.4101, lr=6.43e-04]
Epoch 58 TRAIN:  96%|█████████▌| 130/136 [00:30<00:01,  4.47it/s, epoch=58, grad_norm=230.966, loss=317.2395, lr=6.43e-04]
Epoch 58 TRAIN:  96%|█████████▌| 130/136 [00:30<00:01,  4.47it/s, epoch=58, grad_norm=283.055, loss=373.4258, lr=6.43e-04]
Epoch 58 TRAIN:  96%|█████████▌| 130/136 [00:30<00:01,  4.47it/s, epoch=58, grad_norm=191.326, loss=296.9458, lr=6.43e-04]
Epoch 58 TRAIN:  96%|█████████▌| 130/136 [00:30<00:01,  4.47it/s, epoch=58, grad_norm=130.927, loss=318.6641, lr=6.44e-04]
Epoch 58 TRAIN:  99%|█████████▊| 134/136 [00:30<00:00,  4.95it/s, epoch=58, grad_norm=130.927, loss=318.6641, lr=6.44e-04]
Epoch 58 TRAIN:  99%|█████████▊| 134/136 [00:30<00:00,  4.95it/s, epoch=58, grad_norm=128.913, loss=326.6542, lr=6.44e-04]
Epoch 58 TRAIN:  99%|█████████▊| 134/136 [00:31<00:00,  4.95it/s, epoch=58, grad_norm=240.298, loss=369.4755, lr=6.44e-04]
Epoch 58 TRAIN: 100%|██████████| 136/136 [00:31<00:00,  4.38it/s, epoch=58, grad_norm=240.298, loss=369.4755, lr=6.44e-04]

CV:   0%|          | 0/6 [00:00<?, ?it/s]/home/rookie/Documents/batuka/projects/wenet/wenet/wenet/utils/train_utils.py:689: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  amp_autocast(enabled=dtype is not None,

CV:  17%|█▋        | 1/6 [00:00<00:03,  1.34it/s]
                                                 
2026-04-16 20:21:54,443 INFO Epoch 58 Step 8046 CV info lr 6.4368e-04 cv_loss 458.496337890625 rank 0 acc 0.3597683409849803
2026-04-16 20:21:54,445 INFO [Rank 0] Checkpoint: save to checkpoint exp/conformer_mongolian/epoch_58.pt
2026-04-16 20:21:54,632 INFO Epoch 59 Step 8046 TRAIN info lr 6.4368e-04 rank 0
2026-04-16 20:21:54,647 INFO using accumulate grad, new batch size is 1 times larger than before

Epoch 59 TRAIN:   0%|          | 0/136 [00:00<?, ?it/s]
Epoch 59 TRAIN:   0%|          | 0/136 [00:03<?, ?it/s, epoch=59, grad_norm=441.671, loss=358.2269, lr=6.44e-04]
Epoch 59 TRAIN:   1%|          | 1/136 [00:03<08:47,  3.91s/it, epoch=59, grad_norm=441.671, loss=358.2269, lr=6.44e-04]
Epoch 59 TRAIN:   1%|          | 1/136 [00:04<08:47,  3.91s/it, epoch=59, grad_norm=408.470, loss=344.5491, lr=6.44e-04]
Epoch 59 TRAIN:   1%|          | 1/136 [00:04<08:47,  3.91s/it, epoch=59, grad_norm=437.189, loss=392.0834, lr=6.44e-04]
Epoch 59 TRAIN:   1%|          | 1/136 [00:04<08:47,  3.91s/it, epoch=59, grad_norm=343.301, loss=356.0866, lr=6.44e-04]
Epoch 59 TRAIN:   1%|          | 1/136 [00:04<08:47,  3.91s/it, epoch=59, grad_norm=247.112, loss=321.1089, lr=6.44e-04]
Epoch 59 TRAIN:   1%|          | 1/136 [00:04<08:47,  3.91s/it, epoch=59, grad_norm=370.643, loss=340.9501, lr=6.44e-04]
Epoch 59 TRAIN:   1%|          | 1/136 [00:04<08:47,  3.91s/it, epoch=59, grad_norm=171.808, loss=273.8072, lr=6.44e-04]
Epoch 59 TRAIN:   5%|▌         | 7/136 [00:04<01:03,  2.03it/s, epoch=59, grad_norm=171.808, loss=273.8072, lr=6.44e-04]
Epoch 59 TRAIN:   5%|▌         | 7/136 [00:04<01:03,  2.03it/s, epoch=59, grad_norm=117.274, loss=234.0117, lr=6.44e-04]
Epoch 59 TRAIN:   5%|▌         | 7/136 [00:04<01:03,  2.03it/s, epoch=59, grad_norm=95.327, loss=328.5547, lr=6.44e-04] 
Epoch 59 TRAIN:   5%|▌         | 7/136 [00:04<01:03,  2.03it/s, epoch=59, grad_norm=112.727, loss=289.9152, lr=6.44e-04]
Epoch 59 TRAIN:   5%|▌         | 7/136 [00:05<01:03,  2.03it/s, epoch=59, grad_norm=96.572, loss=325.1030, lr=6.45e-04] 
Epoch 59 TRAIN:   8%|▊         | 11/136 [00:05<00:41,  3.02it/s, epoch=59, grad_norm=96.572, loss=325.1030, lr=6.45e-04]
Epoch 59 TRAIN:   8%|▊         | 11/136 [00:05<00:41,  3.02it/s, epoch=59, grad_norm=140.944, loss=300.2588, lr=6.45e-04]
Epoch 59 TRAIN:   8%|▊         | 11/136 [00:05<00:41,  3.02it/s, epoch=59, grad_norm=102.378, loss=272.0099, lr=6.45e-04]
Epoch 59 TRAIN:   8%|▊         | 11/136 [00:05<00:41,  3.02it/s, epoch=59, grad_norm=120.818, loss=283.3959, lr=6.45e-04]
Epoch 59 TRAIN:   8%|▊         | 11/136 [00:05<00:41,  3.02it/s, epoch=59, grad_norm=115.692, loss=271.2236, lr=6.45e-04]
Epoch 59 TRAIN:  11%|█         | 15/136 [00:05<00:31,  3.80it/s, epoch=59, grad_norm=115.692, loss=271.2236, lr=6.45e-04]
Epoch 59 TRAIN:  11%|█         | 15/136 [00:05<00:31,  3.80it/s, epoch=59, grad_norm=88.309, loss=283.6274, lr=6.45e-04] 
Epoch 59 TRAIN:  11%|█         | 15/136 [00:06<00:31,  3.80it/s, epoch=59, grad_norm=80.902, loss=290.0088, lr=6.45e-04]
Epoch 59 TRAIN:  11%|█         | 15/136 [00:06<00:31,  3.80it/s, epoch=59, grad_norm=82.190, loss=275.8854, lr=6.45e-04]
Epoch 59 TRAIN:  11%|█         | 15/136 [00:06<00:31,  3.80it/s, epoch=59, grad_norm=233.572, loss=320.7082, lr=6.45e-04]
Epoch 59 TRAIN:  14%|█▍        | 19/136 [00:06<00:27,  4.25it/s, epoch=59, grad_norm=233.572, loss=320.7082, lr=6.45e-04]
Epoch 59 TRAIN:  14%|█▍        | 19/136 [00:06<00:27,  4.25it/s, epoch=59, grad_norm=148.285, loss=290.4144, lr=6.45e-04]
Epoch 59 TRAIN:  14%|█▍        | 19/136 [00:06<00:27,  4.25it/s, epoch=59, grad_norm=85.317, loss=270.1447, lr=6.45e-04] 
Epoch 59 TRAIN:  14%|█▍        | 19/136 [00:07<00:27,  4.25it/s, epoch=59, grad_norm=76.693, loss=292.9221, lr=6.45e-04]
Epoch 59 TRAIN:  14%|█▍        | 19/136 [00:07<00:27,  4.25it/s, epoch=59, grad_norm=80.010, loss=283.8206, lr=6.46e-04]
Epoch 59 TRAIN:  17%|█▋        | 23/136 [00:07<00:25,  4.51it/s, epoch=59, grad_norm=80.010, loss=283.8206, lr=6.46e-04]
Epoch 59 TRAIN:  17%|█▋        | 23/136 [00:07<00:25,  4.51it/s, epoch=59, grad_norm=81.312, loss=298.1019, lr=6.46e-04]
Epoch 59 TRAIN:  17%|█▋        | 23/136 [00:07<00:25,  4.51it/s, epoch=59, grad_norm=147.798, loss=305.9998, lr=6.46e-04]
Epoch 59 TRAIN:  17%|█▋        | 23/136 [00:07<00:25,  4.51it/s, epoch=59, grad_norm=154.218, loss=293.8550, lr=6.46e-04]
Epoch 59 TRAIN:  19%|█▉        | 26/136 [00:07<00:23,  4.77it/s, epoch=59, grad_norm=154.218, loss=293.8550, lr=6.46e-04]
Epoch 59 TRAIN:  19%|█▉        | 26/136 [00:07<00:23,  4.77it/s, epoch=59, grad_norm=163.309, loss=259.2134, lr=6.46e-04]
Epoch 59 TRAIN:  19%|█▉        | 26/136 [00:08<00:23,  4.77it/s, epoch=59, grad_norm=170.629, loss=277.6037, lr=6.46e-04]
Epoch 59 TRAIN:  19%|█▉        | 26/136 [00:08<00:23,  4.77it/s, epoch=59, grad_norm=92.628, loss=299.3662, lr=6.46e-04] 
Epoch 59 TRAIN:  21%|██▏       | 29/136 [00:08<00:22,  4.70it/s, epoch=59, grad_norm=92.628, loss=299.3662, lr=6.46e-04]
Epoch 59 TRAIN:  21%|██▏       | 29/136 [00:08<00:22,  4.70it/s, epoch=59, grad_norm=120.427, loss=287.4056, lr=6.46e-04]
Epoch 59 TRAIN:  21%|██▏       | 29/136 [00:08<00:22,  4.70it/s, epoch=59, grad_norm=193.304, loss=339.0374, lr=6.46e-04]
Epoch 59 TRAIN:  21%|██▏       | 29/136 [00:09<00:22,  4.70it/s, epoch=59, grad_norm=68.627, loss=256.8887, lr=6.46e-04] 
Epoch 59 TRAIN:  24%|██▎       | 32/136 [00:09<00:21,  4.75it/s, epoch=59, grad_norm=68.627, loss=256.8887, lr=6.46e-04]
Epoch 59 TRAIN:  24%|██▎       | 32/136 [00:09<00:21,  4.75it/s, epoch=59, grad_norm=109.222, loss=307.6819, lr=6.46e-04]
Epoch 59 TRAIN:  24%|██▎       | 32/136 [00:09<00:21,  4.75it/s, epoch=59, grad_norm=97.361, loss=306.1537, lr=6.46e-04] 
Epoch 59 TRAIN:  24%|██▎       | 32/136 [00:09<00:21,  4.75it/s, epoch=59, grad_norm=95.792, loss=279.8230, lr=6.46e-04]
Epoch 59 TRAIN:  26%|██▌       | 35/136 [00:09<00:20,  4.99it/s, epoch=59, grad_norm=95.792, loss=279.8230, lr=6.46e-04]
Epoch 59 TRAIN:  26%|██▌       | 35/136 [00:09<00:20,  4.99it/s, epoch=59, grad_norm=117.304, loss=272.1663, lr=6.47e-04]
Epoch 59 TRAIN:  26%|██▌       | 35/136 [00:10<00:20,  4.99it/s, epoch=59, grad_norm=100.322, loss=300.5916, lr=6.47e-04]
Epoch 59 TRAIN:  26%|██▌       | 35/136 [00:10<00:20,  4.99it/s, epoch=59, grad_norm=117.109, loss=302.7256, lr=6.47e-04]
Epoch 59 TRAIN:  28%|██▊       | 38/136 [00:10<00:20,  4.79it/s, epoch=59, grad_norm=117.109, loss=302.7256, lr=6.47e-04]
Epoch 59 TRAIN:  28%|██▊       | 38/136 [00:10<00:20,  4.79it/s, epoch=59, grad_norm=153.315, loss=268.8668, lr=6.47e-04]
Epoch 59 TRAIN:  28%|██▊       | 38/136 [00:10<00:20,  4.79it/s, epoch=59, grad_norm=94.938, loss=308.4816, lr=6.47e-04] 
Epoch 59 TRAIN:  28%|██▊       | 38/136 [00:10<00:20,  4.79it/s, epoch=59, grad_norm=86.020, loss=296.9245, lr=6.47e-04]
Epoch 59 TRAIN:  30%|███       | 41/136 [00:10<00:19,  4.96it/s, epoch=59, grad_norm=86.020, loss=296.9245, lr=6.47e-04]
Epoch 59 TRAIN:  30%|███       | 41/136 [00:10<00:19,  4.96it/s, epoch=59, grad_norm=105.429, loss=274.1858, lr=6.47e-04]
Epoch 59 TRAIN:  30%|███       | 41/136 [00:11<00:19,  4.96it/s, epoch=59, grad_norm=72.905, loss=289.6056, lr=6.47e-04] 
Epoch 59 TRAIN:  30%|███       | 41/136 [00:11<00:19,  4.96it/s, epoch=59, grad_norm=112.843, loss=286.0216, lr=6.47e-04]
Epoch 59 TRAIN:  32%|███▏      | 44/136 [00:11<00:19,  4.77it/s, epoch=59, grad_norm=112.843, loss=286.0216, lr=6.47e-04]
Epoch 59 TRAIN:  32%|███▏      | 44/136 [00:11<00:19,  4.77it/s, epoch=59, grad_norm=158.000, loss=327.3878, lr=6.47e-04]
Epoch 59 TRAIN:  32%|███▏      | 44/136 [00:11<00:19,  4.77it/s, epoch=59, grad_norm=95.318, loss=290.7901, lr=6.47e-04] 
Epoch 59 TRAIN:  32%|███▏      | 44/136 [00:12<00:19,  4.77it/s, epoch=59, grad_norm=109.778, loss=290.1681, lr=6.47e-04]
Epoch 59 TRAIN:  35%|███▍      | 47/136 [00:12<00:19,  4.67it/s, epoch=59, grad_norm=109.778, loss=290.1681, lr=6.47e-04]
Epoch 59 TRAIN:  35%|███▍      | 47/136 [00:12<00:19,  4.67it/s, epoch=59, grad_norm=164.637, loss=322.3346, lr=6.48e-04]
Epoch 59 TRAIN:  35%|███▍      | 47/136 [00:12<00:19,  4.67it/s, epoch=59, grad_norm=97.419, loss=308.6386, lr=6.48e-04] 
Epoch 59 TRAIN:  35%|███▍      | 47/136 [00:12<00:19,  4.67it/s, epoch=59, grad_norm=155.935, loss=295.4276, lr=6.48e-04]
Epoch 59 TRAIN:  37%|███▋      | 50/136 [00:12<00:17,  4.85it/s, epoch=59, grad_norm=155.935, loss=295.4276, lr=6.48e-04]
Epoch 59 TRAIN:  37%|███▋      | 50/136 [00:12<00:17,  4.85it/s, epoch=59, grad_norm=111.308, loss=328.2966, lr=6.48e-04]
Epoch 59 TRAIN:  37%|███▋      | 50/136 [00:13<00:17,  4.85it/s, epoch=59, grad_norm=116.173, loss=317.5049, lr=6.48e-04]
Epoch 59 TRAIN:  37%|███▋      | 50/136 [00:13<00:17,  4.85it/s, epoch=59, grad_norm=94.565, loss=312.0081, lr=6.48e-04] 
Epoch 59 TRAIN:  39%|███▉      | 53/136 [00:13<00:16,  4.98it/s, epoch=59, grad_norm=94.565, loss=312.0081, lr=6.48e-04]
Epoch 59 TRAIN:  39%|███▉      | 53/136 [00:13<00:16,  4.98it/s, epoch=59, grad_norm=109.283, loss=280.9055, lr=6.48e-04]
Epoch 59 TRAIN:  39%|███▉      | 53/136 [00:13<00:16,  4.98it/s, epoch=59, grad_norm=92.700, loss=273.9589, lr=6.48e-04] 
Epoch 59 TRAIN:  39%|███▉      | 53/136 [00:13<00:16,  4.98it/s, epoch=59, grad_norm=108.920, loss=282.2383, lr=6.48e-04]
Epoch 59 TRAIN:  41%|████      | 56/136 [00:13<00:16,  4.86it/s, epoch=59, grad_norm=108.920, loss=282.2383, lr=6.48e-04]
Epoch 59 TRAIN:  41%|████      | 56/136 [00:14<00:16,  4.86it/s, epoch=59, grad_norm=93.065, loss=298.1935, lr=6.48e-04] 
Epoch 59 TRAIN:  41%|████      | 56/136 [00:14<00:16,  4.86it/s, epoch=59, grad_norm=101.415, loss=322.3302, lr=6.48e-04]
Epoch 59 TRAIN:  41%|████      | 56/136 [00:14<00:16,  4.86it/s, epoch=59, grad_norm=118.759, loss=310.3185, lr=6.48e-04]
Epoch 59 TRAIN:  43%|████▎     | 59/136 [00:14<00:16,  4.77it/s, epoch=59, grad_norm=118.759, loss=310.3185, lr=6.48e-04]
Epoch 59 TRAIN:  43%|████▎     | 59/136 [00:14<00:16,  4.77it/s, epoch=59, grad_norm=121.710, loss=283.9862, lr=6.48e-04]
Epoch 59 TRAIN:  43%|████▎     | 59/136 [00:14<00:16,  4.77it/s, epoch=59, grad_norm=124.864, loss=265.4200, lr=6.49e-04]
Epoch 59 TRAIN:  43%|████▎     | 59/136 [00:15<00:16,  4.77it/s, epoch=59, grad_norm=137.124, loss=257.9823, lr=6.49e-04]
Epoch 59 TRAIN:  46%|████▌     | 62/136 [00:15<00:15,  4.87it/s, epoch=59, grad_norm=137.124, loss=257.9823, lr=6.49e-04]
Epoch 59 TRAIN:  46%|████▌     | 62/136 [00:15<00:15,  4.87it/s, epoch=59, grad_norm=168.544, loss=239.1162, lr=6.49e-04]
Epoch 59 TRAIN:  46%|████▌     | 62/136 [00:15<00:15,  4.87it/s, epoch=59, grad_norm=84.207, loss=286.9768, lr=6.49e-04] 
Epoch 59 TRAIN:  46%|████▌     | 62/136 [00:15<00:15,  4.87it/s, epoch=59, grad_norm=98.022, loss=279.8036, lr=6.49e-04]
Epoch 59 TRAIN:  48%|████▊     | 65/136 [00:15<00:15,  4.72it/s, epoch=59, grad_norm=98.022, loss=279.8036, lr=6.49e-04]
Epoch 59 TRAIN:  48%|████▊     | 65/136 [00:16<00:15,  4.72it/s, epoch=59, grad_norm=126.685, loss=283.7089, lr=6.49e-04]
Epoch 59 TRAIN:  48%|████▊     | 65/136 [00:16<00:15,  4.72it/s, epoch=59, grad_norm=251.425, loss=365.6596, lr=6.49e-04]
Epoch 59 TRAIN:  48%|████▊     | 65/136 [00:16<00:15,  4.72it/s, epoch=59, grad_norm=123.375, loss=292.2538, lr=6.49e-04]
Epoch 59 TRAIN:  50%|█████     | 68/136 [00:16<00:14,  4.78it/s, epoch=59, grad_norm=123.375, loss=292.2538, lr=6.49e-04]
Epoch 59 TRAIN:  50%|█████     | 68/136 [00:16<00:14,  4.78it/s, epoch=59, grad_norm=166.908, loss=339.7017, lr=6.49e-04]
Epoch 59 TRAIN:  50%|█████     | 68/136 [00:16<00:14,  4.78it/s, epoch=59, grad_norm=123.828, loss=242.8113, lr=6.49e-04]
Epoch 59 TRAIN:  50%|█████     | 68/136 [00:17<00:14,  4.78it/s, epoch=59, grad_norm=114.600, loss=313.8606, lr=6.49e-04]
Epoch 59 TRAIN:  52%|█████▏    | 71/136 [00:17<00:13,  4.77it/s, epoch=59, grad_norm=114.600, loss=313.8606, lr=6.49e-04]
Epoch 59 TRAIN:  52%|█████▏    | 71/136 [00:17<00:13,  4.77it/s, epoch=59, grad_norm=104.123, loss=306.7431, lr=6.49e-04]
Epoch 59 TRAIN:  52%|█████▏    | 71/136 [00:17<00:13,  4.77it/s, epoch=59, grad_norm=116.703, loss=332.5708, lr=6.50e-04]
Epoch 59 TRAIN:  52%|█████▏    | 71/136 [00:17<00:13,  4.77it/s, epoch=59, grad_norm=316.126, loss=261.7137, lr=6.50e-04]
Epoch 59 TRAIN:  54%|█████▍    | 74/136 [00:17<00:13,  4.72it/s, epoch=59, grad_norm=316.126, loss=261.7137, lr=6.50e-04]
Epoch 59 TRAIN:  54%|█████▍    | 74/136 [00:17<00:13,  4.72it/s, epoch=59, grad_norm=109.734, loss=307.2588, lr=6.50e-04]
Epoch 59 TRAIN:  54%|█████▍    | 74/136 [00:18<00:13,  4.72it/s, epoch=59, grad_norm=133.352, loss=274.1431, lr=6.50e-04]
Epoch 59 TRAIN:  54%|█████▍    | 74/136 [00:18<00:13,  4.72it/s, epoch=59, grad_norm=109.103, loss=280.2952, lr=6.50e-04]
Epoch 59 TRAIN:  57%|█████▋    | 77/136 [00:18<00:12,  4.58it/s, epoch=59, grad_norm=109.103, loss=280.2952, lr=6.50e-04]
Epoch 59 TRAIN:  57%|█████▋    | 77/136 [00:18<00:12,  4.58it/s, epoch=59, grad_norm=185.875, loss=290.4851, lr=6.50e-04]
Epoch 59 TRAIN:  57%|█████▋    | 77/136 [00:18<00:12,  4.58it/s, epoch=59, grad_norm=136.064, loss=277.0744, lr=6.50e-04]
Epoch 59 TRAIN:  57%|█████▋    | 77/136 [00:19<00:12,  4.58it/s, epoch=59, grad_norm=121.943, loss=293.4509, lr=6.50e-04]
Epoch 59 TRAIN:  59%|█████▉    | 80/136 [00:19<00:11,  4.70it/s, epoch=59, grad_norm=121.943, loss=293.4509, lr=6.50e-04]
Epoch 59 TRAIN:  59%|█████▉    | 80/136 [00:19<00:11,  4.70it/s, epoch=59, grad_norm=150.684, loss=270.3831, lr=6.50e-04]
Epoch 59 TRAIN:  59%|█████▉    | 80/136 [00:19<00:11,  4.70it/s, epoch=59, grad_norm=155.080, loss=262.1229, lr=6.50e-04]
Epoch 59 TRAIN:  59%|█████▉    | 80/136 [00:19<00:11,  4.70it/s, epoch=59, grad_norm=166.610, loss=293.0945, lr=6.50e-04]
Epoch 59 TRAIN:  61%|██████    | 83/136 [00:19<00:10,  4.85it/s, epoch=59, grad_norm=166.610, loss=293.0945, lr=6.50e-04]
Epoch 59 TRAIN:  61%|██████    | 83/136 [00:19<00:10,  4.85it/s, epoch=59, grad_norm=100.137, loss=205.6281, lr=6.50e-04]
Epoch 59 TRAIN:  61%|██████    | 83/136 [00:20<00:10,  4.85it/s, epoch=59, grad_norm=180.731, loss=323.3183, lr=6.50e-04]
Epoch 59 TRAIN:  61%|██████    | 83/136 [00:20<00:10,  4.85it/s, epoch=59, grad_norm=133.727, loss=240.4292, lr=6.51e-04]
Epoch 59 TRAIN:  63%|██████▎   | 86/136 [00:20<00:10,  4.57it/s, epoch=59, grad_norm=133.727, loss=240.4292, lr=6.51e-04]
Epoch 59 TRAIN:  63%|██████▎   | 86/136 [00:20<00:10,  4.57it/s, epoch=59, grad_norm=118.666, loss=234.6742, lr=6.51e-04]
Epoch 59 TRAIN:  63%|██████▎   | 86/136 [00:20<00:10,  4.57it/s, epoch=59, grad_norm=239.201, loss=314.7992, lr=6.51e-04]
Epoch 59 TRAIN:  63%|██████▎   | 86/136 [00:20<00:10,  4.57it/s, epoch=59, grad_norm=248.976, loss=368.9443, lr=6.51e-04]
Epoch 59 TRAIN:  65%|██████▌   | 89/136 [00:20<00:10,  4.59it/s, epoch=59, grad_norm=248.976, loss=368.9443, lr=6.51e-04]
Epoch 59 TRAIN:  65%|██████▌   | 89/136 [00:21<00:10,  4.59it/s, epoch=59, grad_norm=128.042, loss=326.5713, lr=6.51e-04]
Epoch 59 TRAIN:  65%|██████▌   | 89/136 [00:21<00:10,  4.59it/s, epoch=59, grad_norm=119.119, loss=223.7071, lr=6.51e-04]
Epoch 59 TRAIN:  65%|██████▌   | 89/136 [00:21<00:10,  4.59it/s, epoch=59, grad_norm=128.626, loss=233.8472, lr=6.51e-04]
Epoch 59 TRAIN:  68%|██████▊   | 92/136 [00:21<00:09,  4.72it/s, epoch=59, grad_norm=128.626, loss=233.8472, lr=6.51e-04]
Epoch 59 TRAIN:  68%|██████▊   | 92/136 [00:21<00:09,  4.72it/s, epoch=59, grad_norm=159.271, loss=350.5805, lr=6.51e-04]
Epoch 59 TRAIN:  68%|██████▊   | 92/136 [00:22<00:09,  4.72it/s, epoch=59, grad_norm=169.454, loss=237.1874, lr=6.51e-04]
Epoch 59 TRAIN:  68%|██████▊   | 92/136 [00:22<00:09,  4.72it/s, epoch=59, grad_norm=201.677, loss=195.6396, lr=6.51e-04]
Epoch 59 TRAIN:  70%|██████▉   | 95/136 [00:22<00:09,  4.53it/s, epoch=59, grad_norm=201.677, loss=195.6396, lr=6.51e-04]
Epoch 59 TRAIN:  70%|██████▉   | 95/136 [00:22<00:09,  4.53it/s, epoch=59, grad_norm=261.522, loss=347.4941, lr=6.51e-04]
Epoch 59 TRAIN:  70%|██████▉   | 95/136 [00:22<00:09,  4.53it/s, epoch=59, grad_norm=105.908, loss=245.6469, lr=6.51e-04]
Epoch 59 TRAIN:  70%|██████▉   | 95/136 [00:22<00:09,  4.53it/s, epoch=59, grad_norm=123.835, loss=279.1424, lr=6.52e-04]
Epoch 59 TRAIN:  72%|███████▏  | 98/136 [00:22<00:08,  4.67it/s, epoch=59, grad_norm=123.835, loss=279.1424, lr=6.52e-04]
Epoch 59 TRAIN:  72%|███████▏  | 98/136 [00:23<00:08,  4.67it/s, epoch=59, grad_norm=197.753, loss=303.1361, lr=6.52e-04]
Epoch 59 TRAIN:  72%|███████▏  | 98/136 [00:23<00:08,  4.67it/s, epoch=59, grad_norm=214.018, loss=328.7327, lr=6.52e-04]2026-04-16 20:22:18,044 DEBUG TRAIN | steps/sec 4.250| Batch 59/100 loss 328.732727 loss_att 273.709381 loss_ctc 457.120605 th_accuracy 0.405120 lr 6.5168e-04 grad_norm 214.018494 rank 0

Epoch 59 TRAIN:  72%|███████▏  | 98/136 [00:23<00:08,  4.67it/s, epoch=59, grad_norm=107.560, loss=261.4051, lr=6.52e-04]
Epoch 59 TRAIN:  74%|███████▍  | 101/136 [00:23<00:07,  4.48it/s, epoch=59, grad_norm=107.560, loss=261.4051, lr=6.52e-04]
Epoch 59 TRAIN:  74%|███████▍  | 101/136 [00:23<00:07,  4.48it/s, epoch=59, grad_norm=170.566, loss=256.1433, lr=6.52e-04]
Epoch 59 TRAIN:  74%|███████▍  | 101/136 [00:24<00:07,  4.48it/s, epoch=59, grad_norm=157.315, loss=275.8735, lr=6.52e-04]
Epoch 59 TRAIN:  74%|███████▍  | 101/136 [00:24<00:07,  4.48it/s, epoch=59, grad_norm=142.916, loss=307.8129, lr=6.52e-04]
Epoch 59 TRAIN:  76%|███████▋  | 104/136 [00:24<00:07,  4.48it/s, epoch=59, grad_norm=142.916, loss=307.8129, lr=6.52e-04]
Epoch 59 TRAIN:  76%|███████▋  | 104/136 [00:24<00:07,  4.48it/s, epoch=59, grad_norm=132.302, loss=281.2299, lr=6.52e-04]
Epoch 59 TRAIN:  76%|███████▋  | 104/136 [00:24<00:07,  4.48it/s, epoch=59, grad_norm=115.282, loss=279.6395, lr=6.52e-04]
Epoch 59 TRAIN:  76%|███████▋  | 104/136 [00:24<00:07,  4.48it/s, epoch=59, grad_norm=140.007, loss=298.4124, lr=6.52e-04]
Epoch 59 TRAIN:  79%|███████▊  | 107/136 [00:24<00:06,  4.47it/s, epoch=59, grad_norm=140.007, loss=298.4124, lr=6.52e-04]
Epoch 59 TRAIN:  79%|███████▊  | 107/136 [00:25<00:06,  4.47it/s, epoch=59, grad_norm=116.351, loss=224.3460, lr=6.52e-04]
Epoch 59 TRAIN:  79%|███████▊  | 107/136 [00:25<00:06,  4.47it/s, epoch=59, grad_norm=133.836, loss=228.2176, lr=6.52e-04]
Epoch 59 TRAIN:  79%|███████▊  | 107/136 [00:25<00:06,  4.47it/s, epoch=59, grad_norm=148.048, loss=322.4461, lr=6.52e-04]
Epoch 59 TRAIN:  81%|████████  | 110/136 [00:25<00:05,  4.36it/s, epoch=59, grad_norm=148.048, loss=322.4461, lr=6.52e-04]
Epoch 59 TRAIN:  81%|████████  | 110/136 [00:25<00:05,  4.36it/s, epoch=59, grad_norm=139.406, loss=317.0869, lr=6.53e-04]
Epoch 59 TRAIN:  81%|████████  | 110/136 [00:26<00:05,  4.36it/s, epoch=59, grad_norm=166.133, loss=297.1906, lr=6.53e-04]
Epoch 59 TRAIN:  81%|████████  | 110/136 [00:26<00:05,  4.36it/s, epoch=59, grad_norm=162.339, loss=301.1199, lr=6.53e-04]
Epoch 59 TRAIN:  83%|████████▎ | 113/136 [00:26<00:05,  4.20it/s, epoch=59, grad_norm=162.339, loss=301.1199, lr=6.53e-04]
Epoch 59 TRAIN:  83%|████████▎ | 113/136 [00:26<00:05,  4.20it/s, epoch=59, grad_norm=168.780, loss=250.8613, lr=6.53e-04]
Epoch 59 TRAIN:  83%|████████▎ | 113/136 [00:26<00:05,  4.20it/s, epoch=59, grad_norm=119.838, loss=237.7342, lr=6.53e-04]
Epoch 59 TRAIN:  83%|████████▎ | 113/136 [00:27<00:05,  4.20it/s, epoch=59, grad_norm=192.458, loss=309.2226, lr=6.53e-04]
Epoch 59 TRAIN:  85%|████████▌ | 116/136 [00:27<00:04,  4.51it/s, epoch=59, grad_norm=192.458, loss=309.2226, lr=6.53e-04]
Epoch 59 TRAIN:  85%|████████▌ | 116/136 [00:27<00:04,  4.51it/s, epoch=59, grad_norm=214.264, loss=336.5241, lr=6.53e-04]
Epoch 59 TRAIN:  85%|████████▌ | 116/136 [00:27<00:04,  4.51it/s, epoch=59, grad_norm=277.559, loss=328.2674, lr=6.53e-04]
Epoch 59 TRAIN:  85%|████████▌ | 116/136 [00:27<00:04,  4.51it/s, epoch=59, grad_norm=318.211, loss=332.9902, lr=6.53e-04]
Epoch 59 TRAIN:  88%|████████▊ | 119/136 [00:27<00:03,  4.39it/s, epoch=59, grad_norm=318.211, loss=332.9902, lr=6.53e-04]
Epoch 59 TRAIN:  88%|████████▊ | 119/136 [00:28<00:03,  4.39it/s, epoch=59, grad_norm=230.168, loss=260.8001, lr=6.53e-04]
Epoch 59 TRAIN:  88%|████████▊ | 119/136 [00:28<00:03,  4.39it/s, epoch=59, grad_norm=97.744, loss=232.2766, lr=6.53e-04] 
Epoch 59 TRAIN:  88%|████████▊ | 119/136 [00:28<00:03,  4.39it/s, epoch=59, grad_norm=249.683, loss=353.9966, lr=6.53e-04]
Epoch 59 TRAIN:  90%|████████▉ | 122/136 [00:28<00:03,  4.41it/s, epoch=59, grad_norm=249.683, loss=353.9966, lr=6.53e-04]
Epoch 59 TRAIN:  90%|████████▉ | 122/136 [00:28<00:03,  4.41it/s, epoch=59, grad_norm=175.486, loss=267.4849, lr=6.54e-04]
Epoch 59 TRAIN:  90%|████████▉ | 122/136 [00:28<00:03,  4.41it/s, epoch=59, grad_norm=133.126, loss=186.9713, lr=6.54e-04]
Epoch 59 TRAIN:  90%|████████▉ | 122/136 [00:28<00:03,  4.41it/s, epoch=59, grad_norm=144.398, loss=279.8459, lr=6.54e-04]
Epoch 59 TRAIN:  90%|████████▉ | 122/136 [00:29<00:03,  4.41it/s, epoch=59, grad_norm=166.747, loss=386.0377, lr=6.54e-04]
Epoch 59 TRAIN:  93%|█████████▎| 126/136 [00:29<00:02,  4.76it/s, epoch=59, grad_norm=166.747, loss=386.0377, lr=6.54e-04]
Epoch 59 TRAIN:  93%|█████████▎| 126/136 [00:29<00:02,  4.76it/s, epoch=59, grad_norm=234.738, loss=263.2069, lr=6.54e-04]
Epoch 59 TRAIN:  93%|█████████▎| 126/136 [00:29<00:02,  4.76it/s, epoch=59, grad_norm=146.826, loss=319.3040, lr=6.54e-04]
Epoch 59 TRAIN:  93%|█████████▎| 126/136 [00:29<00:02,  4.76it/s, epoch=59, grad_norm=182.102, loss=252.5323, lr=6.54e-04]
Epoch 59 TRAIN:  95%|█████████▍| 129/136 [00:29<00:01,  4.92it/s, epoch=59, grad_norm=182.102, loss=252.5323, lr=6.54e-04]
Epoch 59 TRAIN:  95%|█████████▍| 129/136 [00:29<00:01,  4.92it/s, epoch=59, grad_norm=171.884, loss=329.3229, lr=6.54e-04]
Epoch 59 TRAIN:  95%|█████████▍| 129/136 [00:30<00:01,  4.92it/s, epoch=59, grad_norm=158.002, loss=312.5598, lr=6.54e-04]
Epoch 59 TRAIN:  95%|█████████▍| 129/136 [00:30<00:01,  4.92it/s, epoch=59, grad_norm=176.112, loss=309.6077, lr=6.54e-04]
Epoch 59 TRAIN:  97%|█████████▋| 132/136 [00:30<00:00,  5.03it/s, epoch=59, grad_norm=176.112, loss=309.6077, lr=6.54e-04]
Epoch 59 TRAIN:  97%|█████████▋| 132/136 [00:30<00:00,  5.03it/s, epoch=59, grad_norm=197.709, loss=304.2075, lr=6.54e-04]
Epoch 59 TRAIN:  97%|█████████▋| 132/136 [00:30<00:00,  5.03it/s, epoch=59, grad_norm=355.046, loss=275.7556, lr=6.54e-04]
Epoch 59 TRAIN:  97%|█████████▋| 132/136 [00:30<00:00,  5.03it/s, epoch=59, grad_norm=440.939, loss=342.7955, lr=6.54e-04]
Epoch 59 TRAIN:  97%|█████████▋| 132/136 [00:30<00:00,  5.03it/s, epoch=59, grad_norm=154.705, loss=333.9152, lr=6.55e-04]
Epoch 59 TRAIN: 100%|██████████| 136/136 [00:30<00:00,  5.70it/s, epoch=59, grad_norm=154.705, loss=333.9152, lr=6.55e-04]
Epoch 59 TRAIN: 100%|██████████| 136/136 [00:31<00:00,  5.70it/s, epoch=59, grad_norm=167.030, loss=334.0353, lr=6.55e-04]
Epoch 59 TRAIN: 137it [00:31,  4.41it/s, epoch=59, grad_norm=167.030, loss=334.0353, lr=6.55e-04]                         

CV:   0%|          | 0/6 [00:00<?, ?it/s]/home/rookie/Documents/batuka/projects/wenet/wenet/wenet/utils/train_utils.py:689: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  amp_autocast(enabled=dtype is not None,

CV:  17%|█▋        | 1/6 [00:00<00:03,  1.33it/s]
                                                 
2026-04-16 20:22:26,539 INFO Epoch 59 Step 8183 CV info lr 6.5464e-04 cv_loss 482.88937100497157 rank 0 acc 0.3491149793068568
2026-04-16 20:22:26,541 INFO [Rank 0] Checkpoint: save to checkpoint exp/conformer_mongolian/epoch_59.pt
2026-04-16 20:22:26,660 INFO Epoch 60 Step 8183 TRAIN info lr 6.5464e-04 rank 0
2026-04-16 20:22:26,675 INFO using accumulate grad, new batch size is 1 times larger than before

Epoch 60 TRAIN:   0%|          | 0/137 [00:00<?, ?it/s]
Epoch 60 TRAIN:   0%|          | 0/137 [00:04<?, ?it/s, epoch=60, grad_norm=401.306, loss=327.0379, lr=6.55e-04]
Epoch 60 TRAIN:   1%|          | 1/137 [00:04<09:12,  4.06s/it, epoch=60, grad_norm=401.306, loss=327.0379, lr=6.55e-04]
Epoch 60 TRAIN:   1%|          | 1/137 [00:04<09:12,  4.06s/it, epoch=60, grad_norm=402.030, loss=343.3489, lr=6.55e-04]
Epoch 60 TRAIN:   1%|          | 1/137 [00:04<09:12,  4.06s/it, epoch=60, grad_norm=373.102, loss=330.2567, lr=6.55e-04]
Epoch 60 TRAIN:   1%|          | 1/137 [00:04<09:12,  4.06s/it, epoch=60, grad_norm=356.560, loss=327.4196, lr=6.55e-04]
Epoch 60 TRAIN:   1%|          | 1/137 [00:04<09:12,  4.06s/it, epoch=60, grad_norm=346.910, loss=288.2838, lr=6.55e-04]
Epoch 60 TRAIN:   1%|          | 1/137 [00:04<09:12,  4.06s/it, epoch=60, grad_norm=433.642, loss=406.4845, lr=6.55e-04]
Epoch 60 TRAIN:   1%|          | 1/137 [00:04<09:12,  4.06s/it, epoch=60, grad_norm=366.698, loss=342.6727, lr=6.55e-04]
Epoch 60 TRAIN:   1%|          | 1/137 [00:04<09:12,  4.06s/it, epoch=60, grad_norm=345.532, loss=407.8404, lr=6.55e-04]
Epoch 60 TRAIN:   1%|          | 1/137 [00:04<09:12,  4.06s/it, epoch=60, grad_norm=205.322, loss=306.0356, lr=6.55e-04]
Epoch 60 TRAIN:   7%|▋         | 9/137 [00:04<00:51,  2.50it/s, epoch=60, grad_norm=205.322, loss=306.0356, lr=6.55e-04]
Epoch 60 TRAIN:   7%|▋         | 9/137 [00:04<00:51,  2.50it/s, epoch=60, grad_norm=131.714, loss=306.5745, lr=6.55e-04]
Epoch 60 TRAIN:   7%|▋         | 9/137 [00:05<00:51,  2.50it/s, epoch=60, grad_norm=128.638, loss=308.8012, lr=6.56e-04]
Epoch 60 TRAIN:   7%|▋         | 9/137 [00:05<00:51,  2.50it/s, epoch=60, grad_norm=88.452, loss=329.2108, lr=6.56e-04] 
Epoch 60 TRAIN:   7%|▋         | 9/137 [00:05<00:51,  2.50it/s, epoch=60, grad_norm=147.568, loss=309.6339, lr=6.56e-04]
Epoch 60 TRAIN:   9%|▉         | 13/137 [00:05<00:38,  3.26it/s, epoch=60, grad_norm=147.568, loss=309.6339, lr=6.56e-04]
Epoch 60 TRAIN:   9%|▉         | 13/137 [00:05<00:38,  3.26it/s, epoch=60, grad_norm=185.185, loss=303.0486, lr=6.56e-04]
Epoch 60 TRAIN:   9%|▉         | 13/137 [00:05<00:38,  3.26it/s, epoch=60, grad_norm=143.685, loss=326.6560, lr=6.56e-04]
Epoch 60 TRAIN:   9%|▉         | 13/137 [00:05<00:38,  3.26it/s, epoch=60, grad_norm=164.544, loss=303.0807, lr=6.56e-04]
Epoch 60 TRAIN:   9%|▉         | 13/137 [00:06<00:38,  3.26it/s, epoch=60, grad_norm=138.788, loss=345.4019, lr=6.56e-04]
Epoch 60 TRAIN:  12%|█▏        | 17/137 [00:06<00:30,  3.89it/s, epoch=60, grad_norm=138.788, loss=345.4019, lr=6.56e-04]
Epoch 60 TRAIN:  12%|█▏        | 17/137 [00:06<00:30,  3.89it/s, epoch=60, grad_norm=107.730, loss=306.0928, lr=6.56e-04]
Epoch 60 TRAIN:  12%|█▏        | 17/137 [00:06<00:30,  3.89it/s, epoch=60, grad_norm=109.142, loss=275.6265, lr=6.56e-04]
Epoch 60 TRAIN:  12%|█▏        | 17/137 [00:06<00:30,  3.89it/s, epoch=60, grad_norm=156.617, loss=349.2032, lr=6.56e-04]
Epoch 60 TRAIN:  12%|█▏        | 17/137 [00:06<00:30,  3.89it/s, epoch=60, grad_norm=85.384, loss=239.5746, lr=6.56e-04] 
Epoch 60 TRAIN:  15%|█▌        | 21/137 [00:06<00:27,  4.29it/s, epoch=60, grad_norm=85.384, loss=239.5746, lr=6.56e-04]
Epoch 60 TRAIN:  15%|█▌        | 21/137 [00:06<00:27,  4.29it/s, epoch=60, grad_norm=121.592, loss=289.4566, lr=6.56e-04]
Epoch 60 TRAIN:  15%|█▌        | 21/137 [00:07<00:27,  4.29it/s, epoch=60, grad_norm=90.821, loss=278.5407, lr=6.56e-04] 
Epoch 60 TRAIN:  15%|█▌        | 21/137 [00:07<00:27,  4.29it/s, epoch=60, grad_norm=91.516, loss=294.8828, lr=6.57e-04]
Epoch 60 TRAIN:  15%|█▌        | 21/137 [00:07<00:27,  4.29it/s, epoch=60, grad_norm=88.480, loss=280.5827, lr=6.57e-04]
Epoch 60 TRAIN:  18%|█▊        | 25/137 [00:07<00:24,  4.57it/s, epoch=60, grad_norm=88.480, loss=280.5827, lr=6.57e-04]
Epoch 60 TRAIN:  18%|█▊        | 25/137 [00:07<00:24,  4.57it/s, epoch=60, grad_norm=81.581, loss=273.0459, lr=6.57e-04]
Epoch 60 TRAIN:  18%|█▊        | 25/137 [00:07<00:24,  4.57it/s, epoch=60, grad_norm=85.243, loss=293.6077, lr=6.57e-04]
Epoch 60 TRAIN:  18%|█▊        | 25/137 [00:08<00:24,  4.57it/s, epoch=60, grad_norm=129.337, loss=303.3773, lr=6.57e-04]
Epoch 60 TRAIN:  20%|██        | 28/137 [00:08<00:23,  4.65it/s, epoch=60, grad_norm=129.337, loss=303.3773, lr=6.57e-04]
Epoch 60 TRAIN:  20%|██        | 28/137 [00:08<00:23,  4.65it/s, epoch=60, grad_norm=88.874, loss=326.6999, lr=6.57e-04] 
Epoch 60 TRAIN:  20%|██        | 28/137 [00:08<00:23,  4.65it/s, epoch=60, grad_norm=146.419, loss=310.5622, lr=6.57e-04]
Epoch 60 TRAIN:  20%|██        | 28/137 [00:08<00:23,  4.65it/s, epoch=60, grad_norm=91.553, loss=321.3039, lr=6.57e-04] 
Epoch 60 TRAIN:  23%|██▎       | 31/137 [00:08<00:22,  4.80it/s, epoch=60, grad_norm=91.553, loss=321.3039, lr=6.57e-04]
Epoch 60 TRAIN:  23%|██▎       | 31/137 [00:08<00:22,  4.80it/s, epoch=60, grad_norm=96.038, loss=304.3214, lr=6.57e-04]
Epoch 60 TRAIN:  23%|██▎       | 31/137 [00:09<00:22,  4.80it/s, epoch=60, grad_norm=102.172, loss=316.9059, lr=6.57e-04]
Epoch 60 TRAIN:  23%|██▎       | 31/137 [00:09<00:22,  4.80it/s, epoch=60, grad_norm=124.653, loss=279.8138, lr=6.57e-04]
Epoch 60 TRAIN:  25%|██▍       | 34/137 [00:09<00:21,  4.89it/s, epoch=60, grad_norm=124.653, loss=279.8138, lr=6.57e-04]
Epoch 60 TRAIN:  25%|██▍       | 34/137 [00:09<00:21,  4.89it/s, epoch=60, grad_norm=175.660, loss=235.1822, lr=6.57e-04]
Epoch 60 TRAIN:  25%|██▍       | 34/137 [00:09<00:21,  4.89it/s, epoch=60, grad_norm=119.552, loss=302.7972, lr=6.58e-04]
Epoch 60 TRAIN:  25%|██▍       | 34/137 [00:09<00:21,  4.89it/s, epoch=60, grad_norm=90.339, loss=279.3586, lr=6.58e-04] 
Epoch 60 TRAIN:  27%|██▋       | 37/137 [00:09<00:20,  4.86it/s, epoch=60, grad_norm=90.339, loss=279.3586, lr=6.58e-04]
Epoch 60 TRAIN:  27%|██▋       | 37/137 [00:10<00:20,  4.86it/s, epoch=60, grad_norm=138.518, loss=243.7125, lr=6.58e-04]
Epoch 60 TRAIN:  27%|██▋       | 37/137 [00:10<00:20,  4.86it/s, epoch=60, grad_norm=127.339, loss=304.2096, lr=6.58e-04]
Epoch 60 TRAIN:  27%|██▋       | 37/137 [00:10<00:20,  4.86it/s, epoch=60, grad_norm=139.546, loss=316.9335, lr=6.58e-04]
Epoch 60 TRAIN:  29%|██▉       | 40/137 [00:10<00:20,  4.80it/s, epoch=60, grad_norm=139.546, loss=316.9335, lr=6.58e-04]
Epoch 60 TRAIN:  29%|██▉       | 40/137 [00:10<00:20,  4.80it/s, epoch=60, grad_norm=128.573, loss=303.9724, lr=6.58e-04]
Epoch 60 TRAIN:  29%|██▉       | 40/137 [00:10<00:20,  4.80it/s, epoch=60, grad_norm=132.468, loss=309.0548, lr=6.58e-04]
Epoch 60 TRAIN:  29%|██▉       | 40/137 [00:11<00:20,  4.80it/s, epoch=60, grad_norm=98.515, loss=305.2373, lr=6.58e-04] 
Epoch 60 TRAIN:  31%|███▏      | 43/137 [00:11<00:19,  4.88it/s, epoch=60, grad_norm=98.515, loss=305.2373, lr=6.58e-04]
Epoch 60 TRAIN:  31%|███▏      | 43/137 [00:11<00:19,  4.88it/s, epoch=60, grad_norm=153.785, loss=260.4941, lr=6.58e-04]
Epoch 60 TRAIN:  31%|███▏      | 43/137 [00:11<00:19,  4.88it/s, epoch=60, grad_norm=83.452, loss=298.2991, lr=6.58e-04] 
Epoch 60 TRAIN:  31%|███▏      | 43/137 [00:11<00:19,  4.88it/s, epoch=60, grad_norm=99.262, loss=282.1133, lr=6.58e-04]
Epoch 60 TRAIN:  34%|███▎      | 46/137 [00:11<00:18,  4.90it/s, epoch=60, grad_norm=99.262, loss=282.1133, lr=6.58e-04]
Epoch 60 TRAIN:  34%|███▎      | 46/137 [00:12<00:18,  4.90it/s, epoch=60, grad_norm=108.047, loss=310.2789, lr=6.58e-04]
Epoch 60 TRAIN:  34%|███▎      | 46/137 [00:12<00:18,  4.90it/s, epoch=60, grad_norm=102.677, loss=298.4665, lr=6.58e-04]
Epoch 60 TRAIN:  34%|███▎      | 46/137 [00:12<00:18,  4.90it/s, epoch=60, grad_norm=130.020, loss=293.0644, lr=6.59e-04]
Epoch 60 TRAIN:  36%|███▌      | 49/137 [00:12<00:18,  4.72it/s, epoch=60, grad_norm=130.020, loss=293.0644, lr=6.59e-04]
Epoch 60 TRAIN:  36%|███▌      | 49/137 [00:12<00:18,  4.72it/s, epoch=60, grad_norm=95.585, loss=286.5058, lr=6.59e-04] 
Epoch 60 TRAIN:  36%|███▌      | 49/137 [00:12<00:18,  4.72it/s, epoch=60, grad_norm=120.718, loss=299.0367, lr=6.59e-04]
Epoch 60 TRAIN:  36%|███▌      | 49/137 [00:13<00:18,  4.72it/s, epoch=60, grad_norm=198.182, loss=287.6041, lr=6.59e-04]
Epoch 60 TRAIN:  38%|███▊      | 52/137 [00:13<00:18,  4.66it/s, epoch=60, grad_norm=198.182, loss=287.6041, lr=6.59e-04]
Epoch 60 TRAIN:  38%|███▊      | 52/137 [00:13<00:18,  4.66it/s, epoch=60, grad_norm=152.434, loss=288.9862, lr=6.59e-04]
Epoch 60 TRAIN:  38%|███▊      | 52/137 [00:13<00:18,  4.66it/s, epoch=60, grad_norm=209.512, loss=266.8950, lr=6.59e-04]
Epoch 60 TRAIN:  38%|███▊      | 52/137 [00:13<00:18,  4.66it/s, epoch=60, grad_norm=131.586, loss=332.3937, lr=6.59e-04]
Epoch 60 TRAIN:  40%|████      | 55/137 [00:13<00:18,  4.47it/s, epoch=60, grad_norm=131.586, loss=332.3937, lr=6.59e-04]
Epoch 60 TRAIN:  40%|████      | 55/137 [00:14<00:18,  4.47it/s, epoch=60, grad_norm=170.746, loss=238.0989, lr=6.59e-04]
Epoch 60 TRAIN:  40%|████      | 55/137 [00:14<00:18,  4.47it/s, epoch=60, grad_norm=103.641, loss=265.9573, lr=6.59e-04]
Epoch 60 TRAIN:  40%|████      | 55/137 [00:14<00:18,  4.47it/s, epoch=60, grad_norm=143.892, loss=293.2315, lr=6.59e-04]
Epoch 60 TRAIN:  42%|████▏     | 58/137 [00:14<00:17,  4.48it/s, epoch=60, grad_norm=143.892, loss=293.2315, lr=6.59e-04]
Epoch 60 TRAIN:  42%|████▏     | 58/137 [00:14<00:17,  4.48it/s, epoch=60, grad_norm=138.451, loss=330.8708, lr=6.59e-04]
Epoch 60 TRAIN:  42%|████▏     | 58/137 [00:14<00:17,  4.48it/s, epoch=60, grad_norm=101.556, loss=307.3064, lr=6.59e-04]
Epoch 60 TRAIN:  42%|████▏     | 58/137 [00:15<00:17,  4.48it/s, epoch=60, grad_norm=86.210, loss=243.1661, lr=6.60e-04] 
Epoch 60 TRAIN:  45%|████▍     | 61/137 [00:15<00:16,  4.62it/s, epoch=60, grad_norm=86.210, loss=243.1661, lr=6.60e-04]
Epoch 60 TRAIN:  45%|████▍     | 61/137 [00:15<00:16,  4.62it/s, epoch=60, grad_norm=107.422, loss=238.0755, lr=6.60e-04]
Epoch 60 TRAIN:  45%|████▍     | 61/137 [00:15<00:16,  4.62it/s, epoch=60, grad_norm=152.404, loss=323.9076, lr=6.60e-04]
Epoch 60 TRAIN:  45%|████▍     | 61/137 [00:15<00:16,  4.62it/s, epoch=60, grad_norm=216.803, loss=323.6815, lr=6.60e-04]
Epoch 60 TRAIN:  47%|████▋     | 64/137 [00:15<00:15,  4.70it/s, epoch=60, grad_norm=216.803, loss=323.6815, lr=6.60e-04]
Epoch 60 TRAIN:  47%|████▋     | 64/137 [00:15<00:15,  4.70it/s, epoch=60, grad_norm=132.955, loss=289.3792, lr=6.60e-04]
Epoch 60 TRAIN:  47%|████▋     | 64/137 [00:16<00:15,  4.70it/s, epoch=60, grad_norm=180.014, loss=313.9440, lr=6.60e-04]
Epoch 60 TRAIN:  47%|████▋     | 64/137 [00:16<00:15,  4.70it/s, epoch=60, grad_norm=166.154, loss=325.9410, lr=6.60e-04]
Epoch 60 TRAIN:  49%|████▉     | 67/137 [00:16<00:15,  4.55it/s, epoch=60, grad_norm=166.154, loss=325.9410, lr=6.60e-04]
Epoch 60 TRAIN:  49%|████▉     | 67/137 [00:16<00:15,  4.55it/s, epoch=60, grad_norm=135.398, loss=272.7709, lr=6.60e-04]
Epoch 60 TRAIN:  49%|████▉     | 67/137 [00:16<00:15,  4.55it/s, epoch=60, grad_norm=155.464, loss=226.1996, lr=6.60e-04]
Epoch 60 TRAIN:  49%|████▉     | 67/137 [00:17<00:15,  4.55it/s, epoch=60, grad_norm=239.796, loss=223.4181, lr=6.60e-04]
Epoch 60 TRAIN:  51%|█████     | 70/137 [00:17<00:14,  4.53it/s, epoch=60, grad_norm=239.796, loss=223.4181, lr=6.60e-04]
Epoch 60 TRAIN:  51%|█████     | 70/137 [00:17<00:14,  4.53it/s, epoch=60, grad_norm=112.426, loss=316.7826, lr=6.60e-04]
Epoch 60 TRAIN:  51%|█████     | 70/137 [00:17<00:14,  4.53it/s, epoch=60, grad_norm=115.291, loss=231.1733, lr=6.60e-04]
Epoch 60 TRAIN:  51%|█████     | 70/137 [00:17<00:14,  4.53it/s, epoch=60, grad_norm=96.199, loss=286.1921, lr=6.60e-04] 
Epoch 60 TRAIN:  53%|█████▎    | 73/137 [00:17<00:13,  4.68it/s, epoch=60, grad_norm=96.199, loss=286.1921, lr=6.60e-04]
Epoch 60 TRAIN:  53%|█████▎    | 73/137 [00:17<00:13,  4.68it/s, epoch=60, grad_norm=200.931, loss=290.7614, lr=6.61e-04]
Epoch 60 TRAIN:  53%|█████▎    | 73/137 [00:18<00:13,  4.68it/s, epoch=60, grad_norm=170.882, loss=294.8509, lr=6.61e-04]
Epoch 60 TRAIN:  53%|█████▎    | 73/137 [00:18<00:13,  4.68it/s, epoch=60, grad_norm=200.735, loss=309.1906, lr=6.61e-04]
Epoch 60 TRAIN:  55%|█████▌    | 76/137 [00:18<00:13,  4.58it/s, epoch=60, grad_norm=200.735, loss=309.1906, lr=6.61e-04]
Epoch 60 TRAIN:  55%|█████▌    | 76/137 [00:18<00:13,  4.58it/s, epoch=60, grad_norm=92.032, loss=257.5267, lr=6.61e-04] 
Epoch 60 TRAIN:  55%|█████▌    | 76/137 [00:18<00:13,  4.58it/s, epoch=60, grad_norm=200.385, loss=243.5528, lr=6.61e-04]
Epoch 60 TRAIN:  55%|█████▌    | 76/137 [00:19<00:13,  4.58it/s, epoch=60, grad_norm=159.046, loss=241.0486, lr=6.61e-04]
Epoch 60 TRAIN:  58%|█████▊    | 79/137 [00:19<00:12,  4.55it/s, epoch=60, grad_norm=159.046, loss=241.0486, lr=6.61e-04]
Epoch 60 TRAIN:  58%|█████▊    | 79/137 [00:19<00:12,  4.55it/s, epoch=60, grad_norm=101.896, loss=316.6812, lr=6.61e-04]
Epoch 60 TRAIN:  58%|█████▊    | 79/137 [00:19<00:12,  4.55it/s, epoch=60, grad_norm=83.574, loss=272.1579, lr=6.61e-04] 
Epoch 60 TRAIN:  58%|█████▊    | 79/137 [00:19<00:12,  4.55it/s, epoch=60, grad_norm=111.160, loss=294.7637, lr=6.61e-04]
Epoch 60 TRAIN:  60%|█████▉    | 82/137 [00:19<00:12,  4.47it/s, epoch=60, grad_norm=111.160, loss=294.7637, lr=6.61e-04]
Epoch 60 TRAIN:  60%|█████▉    | 82/137 [00:20<00:12,  4.47it/s, epoch=60, grad_norm=340.861, loss=201.5703, lr=6.61e-04]
Epoch 60 TRAIN:  60%|█████▉    | 82/137 [00:20<00:12,  4.47it/s, epoch=60, grad_norm=302.466, loss=202.2427, lr=6.61e-04]
Epoch 60 TRAIN:  60%|█████▉    | 82/137 [00:20<00:12,  4.47it/s, epoch=60, grad_norm=99.041, loss=255.1803, lr=6.61e-04] 
Epoch 60 TRAIN:  62%|██████▏   | 85/137 [00:20<00:11,  4.54it/s, epoch=60, grad_norm=99.041, loss=255.1803, lr=6.61e-04]
Epoch 60 TRAIN:  62%|██████▏   | 85/137 [00:20<00:11,  4.54it/s, epoch=60, grad_norm=218.587, loss=295.6280, lr=6.62e-04]
Epoch 60 TRAIN:  62%|██████▏   | 85/137 [00:20<00:11,  4.54it/s, epoch=60, grad_norm=187.175, loss=264.6277, lr=6.62e-04]
Epoch 60 TRAIN:  62%|██████▏   | 85/137 [00:20<00:11,  4.54it/s, epoch=60, grad_norm=123.846, loss=263.0924, lr=6.62e-04]
Epoch 60 TRAIN:  64%|██████▍   | 88/137 [00:20<00:10,  4.82it/s, epoch=60, grad_norm=123.846, loss=263.0924, lr=6.62e-04]
Epoch 60 TRAIN:  64%|██████▍   | 88/137 [00:21<00:10,  4.82it/s, epoch=60, grad_norm=169.224, loss=306.2312, lr=6.62e-04]
Epoch 60 TRAIN:  64%|██████▍   | 88/137 [00:21<00:10,  4.82it/s, epoch=60, grad_norm=203.720, loss=314.0286, lr=6.62e-04]
Epoch 60 TRAIN:  64%|██████▍   | 88/137 [00:21<00:10,  4.82it/s, epoch=60, grad_norm=115.750, loss=224.8697, lr=6.62e-04]
Epoch 60 TRAIN:  66%|██████▋   | 91/137 [00:21<00:10,  4.49it/s, epoch=60, grad_norm=115.750, loss=224.8697, lr=6.62e-04]
Epoch 60 TRAIN:  66%|██████▋   | 91/137 [00:21<00:10,  4.49it/s, epoch=60, grad_norm=147.513, loss=289.7646, lr=6.62e-04]
Epoch 60 TRAIN:  66%|██████▋   | 91/137 [00:22<00:10,  4.49it/s, epoch=60, grad_norm=266.942, loss=334.6714, lr=6.62e-04]
Epoch 60 TRAIN:  66%|██████▋   | 91/137 [00:22<00:10,  4.49it/s, epoch=60, grad_norm=190.873, loss=201.0585, lr=6.62e-04]
Epoch 60 TRAIN:  69%|██████▊   | 94/137 [00:22<00:09,  4.42it/s, epoch=60, grad_norm=190.873, loss=201.0585, lr=6.62e-04]
Epoch 60 TRAIN:  69%|██████▊   | 94/137 [00:22<00:09,  4.42it/s, epoch=60, grad_norm=140.037, loss=241.0413, lr=6.62e-04]
Epoch 60 TRAIN:  69%|██████▊   | 94/137 [00:22<00:09,  4.42it/s, epoch=60, grad_norm=147.422, loss=277.9773, lr=6.62e-04]
Epoch 60 TRAIN:  69%|██████▊   | 94/137 [00:23<00:09,  4.42it/s, epoch=60, grad_norm=207.809, loss=317.1689, lr=6.62e-04]
Epoch 60 TRAIN:  71%|███████   | 97/137 [00:23<00:09,  4.40it/s, epoch=60, grad_norm=207.809, loss=317.1689, lr=6.62e-04]
Epoch 60 TRAIN:  71%|███████   | 97/137 [00:23<00:09,  4.40it/s, epoch=60, grad_norm=93.631, loss=201.8683, lr=6.62e-04] 
Epoch 60 TRAIN:  71%|███████   | 97/137 [00:23<00:09,  4.40it/s, epoch=60, grad_norm=223.911, loss=335.2187, lr=6.63e-04]
Epoch 60 TRAIN:  71%|███████   | 97/137 [00:23<00:09,  4.40it/s, epoch=60, grad_norm=153.637, loss=323.3655, lr=6.63e-04]2026-04-16 20:22:50,373 DEBUG TRAIN | steps/sec 4.238| Batch 60/100 loss 323.365540 loss_att 269.976746 loss_ctc 447.939331 th_accuracy 0.384306 lr 6.6264e-04 grad_norm 153.636734 rank 0

Epoch 60 TRAIN:  73%|███████▎  | 100/137 [00:23<00:08,  4.59it/s, epoch=60, grad_norm=153.637, loss=323.3655, lr=6.63e-04]
Epoch 60 TRAIN:  73%|███████▎  | 100/137 [00:23<00:08,  4.59it/s, epoch=60, grad_norm=232.671, loss=330.3237, lr=6.63e-04]
Epoch 60 TRAIN:  73%|███████▎  | 100/137 [00:24<00:08,  4.59it/s, epoch=60, grad_norm=169.859, loss=273.2173, lr=6.63e-04]
Epoch 60 TRAIN:  73%|███████▎  | 100/137 [00:24<00:08,  4.59it/s, epoch=60, grad_norm=141.187, loss=276.9312, lr=6.63e-04]
Epoch 60 TRAIN:  75%|███████▌  | 103/137 [00:24<00:07,  4.75it/s, epoch=60, grad_norm=141.187, loss=276.9312, lr=6.63e-04]
Epoch 60 TRAIN:  75%|███████▌  | 103/137 [00:24<00:07,  4.75it/s, epoch=60, grad_norm=119.151, loss=307.9491, lr=6.63e-04]
Epoch 60 TRAIN:  75%|███████▌  | 103/137 [00:24<00:07,  4.75it/s, epoch=60, grad_norm=191.539, loss=257.2530, lr=6.63e-04]
Epoch 60 TRAIN:  75%|███████▌  | 103/137 [00:24<00:07,  4.75it/s, epoch=60, grad_norm=187.112, loss=238.5375, lr=6.63e-04]
Epoch 60 TRAIN:  77%|███████▋  | 106/137 [00:24<00:06,  4.78it/s, epoch=60, grad_norm=187.112, loss=238.5375, lr=6.63e-04]
Epoch 60 TRAIN:  77%|███████▋  | 106/137 [00:25<00:06,  4.78it/s, epoch=60, grad_norm=151.517, loss=275.2547, lr=6.63e-04]
Epoch 60 TRAIN:  77%|███████▋  | 106/137 [00:25<00:06,  4.78it/s, epoch=60, grad_norm=260.949, loss=324.5989, lr=6.63e-04]
Epoch 60 TRAIN:  77%|███████▋  | 106/137 [00:25<00:06,  4.78it/s, epoch=60, grad_norm=172.472, loss=276.5266, lr=6.63e-04]
Epoch 60 TRAIN:  80%|███████▉  | 109/137 [00:25<00:05,  4.82it/s, epoch=60, grad_norm=172.472, loss=276.5266, lr=6.63e-04]
Epoch 60 TRAIN:  80%|███████▉  | 109/137 [00:25<00:05,  4.82it/s, epoch=60, grad_norm=143.482, loss=307.9766, lr=6.63e-04]
Epoch 60 TRAIN:  80%|███████▉  | 109/137 [00:26<00:05,  4.82it/s, epoch=60, grad_norm=235.979, loss=358.2205, lr=6.64e-04]
Epoch 60 TRAIN:  80%|███████▉  | 109/137 [00:26<00:05,  4.82it/s, epoch=60, grad_norm=188.219, loss=273.9795, lr=6.64e-04]
Epoch 60 TRAIN:  82%|████████▏ | 112/137 [00:26<00:05,  4.55it/s, epoch=60, grad_norm=188.219, loss=273.9795, lr=6.64e-04]
Epoch 60 TRAIN:  82%|████████▏ | 112/137 [00:26<00:05,  4.55it/s, epoch=60, grad_norm=212.443, loss=307.6084, lr=6.64e-04]
Epoch 60 TRAIN:  82%|████████▏ | 112/137 [00:26<00:05,  4.55it/s, epoch=60, grad_norm=228.948, loss=281.1376, lr=6.64e-04]
Epoch 60 TRAIN:  82%|████████▏ | 112/137 [00:26<00:05,  4.55it/s, epoch=60, grad_norm=304.526, loss=253.4395, lr=6.64e-04]
Epoch 60 TRAIN:  84%|████████▍ | 115/137 [00:26<00:04,  4.47it/s, epoch=60, grad_norm=304.526, loss=253.4395, lr=6.64e-04]
Epoch 60 TRAIN:  84%|████████▍ | 115/137 [00:27<00:04,  4.47it/s, epoch=60, grad_norm=150.342, loss=277.4002, lr=6.64e-04]
Epoch 60 TRAIN:  84%|████████▍ | 115/137 [00:27<00:04,  4.47it/s, epoch=60, grad_norm=244.787, loss=338.1976, lr=6.64e-04]
Epoch 60 TRAIN:  84%|████████▍ | 115/137 [00:27<00:04,  4.47it/s, epoch=60, grad_norm=109.742, loss=283.9837, lr=6.64e-04]
Epoch 60 TRAIN:  86%|████████▌ | 118/137 [00:27<00:04,  4.41it/s, epoch=60, grad_norm=109.742, loss=283.9837, lr=6.64e-04]
Epoch 60 TRAIN:  86%|████████▌ | 118/137 [00:27<00:04,  4.41it/s, epoch=60, grad_norm=186.694, loss=264.1415, lr=6.64e-04]
Epoch 60 TRAIN:  86%|████████▌ | 118/137 [00:27<00:04,  4.41it/s, epoch=60, grad_norm=189.655, loss=310.5419, lr=6.64e-04]
Epoch 60 TRAIN:  86%|████████▌ | 118/137 [00:28<00:04,  4.41it/s, epoch=60, grad_norm=93.665, loss=248.5852, lr=6.64e-04] 
Epoch 60 TRAIN:  88%|████████▊ | 121/137 [00:28<00:03,  4.59it/s, epoch=60, grad_norm=93.665, loss=248.5852, lr=6.64e-04]
Epoch 60 TRAIN:  88%|████████▊ | 121/137 [00:28<00:03,  4.59it/s, epoch=60, grad_norm=187.148, loss=290.0537, lr=6.64e-04]
Epoch 60 TRAIN:  88%|████████▊ | 121/137 [00:28<00:03,  4.59it/s, epoch=60, grad_norm=178.384, loss=269.2560, lr=6.64e-04]
Epoch 60 TRAIN:  88%|████████▊ | 121/137 [00:28<00:03,  4.59it/s, epoch=60, grad_norm=105.598, loss=208.3684, lr=6.65e-04]
Epoch 60 TRAIN:  88%|████████▊ | 121/137 [00:28<00:03,  4.59it/s, epoch=60, grad_norm=216.860, loss=276.3909, lr=6.65e-04]
Epoch 60 TRAIN:  91%|█████████ | 125/137 [00:28<00:02,  4.95it/s, epoch=60, grad_norm=216.860, loss=276.3909, lr=6.65e-04]
Epoch 60 TRAIN:  91%|█████████ | 125/137 [00:29<00:02,  4.95it/s, epoch=60, grad_norm=176.247, loss=202.4476, lr=6.65e-04]
Epoch 60 TRAIN:  91%|█████████ | 125/137 [00:29<00:02,  4.95it/s, epoch=60, grad_norm=185.133, loss=316.5797, lr=6.65e-04]
Epoch 60 TRAIN:  91%|█████████ | 125/137 [00:29<00:02,  4.95it/s, epoch=60, grad_norm=190.668, loss=310.4421, lr=6.65e-04]
Epoch 60 TRAIN:  93%|█████████▎| 128/137 [00:29<00:01,  4.95it/s, epoch=60, grad_norm=190.668, loss=310.4421, lr=6.65e-04]
Epoch 60 TRAIN:  93%|█████████▎| 128/137 [00:29<00:01,  4.95it/s, epoch=60, grad_norm=161.323, loss=312.8723, lr=6.65e-04]
Epoch 60 TRAIN:  93%|█████████▎| 128/137 [00:29<00:01,  4.95it/s, epoch=60, grad_norm=212.939, loss=337.7145, lr=6.65e-04]
Epoch 60 TRAIN:  93%|█████████▎| 128/137 [00:30<00:01,  4.95it/s, epoch=60, grad_norm=127.584, loss=243.9918, lr=6.65e-04]
Epoch 60 TRAIN:  96%|█████████▌| 131/137 [00:30<00:01,  4.95it/s, epoch=60, grad_norm=127.584, loss=243.9918, lr=6.65e-04]
Epoch 60 TRAIN:  96%|█████████▌| 131/137 [00:30<00:01,  4.95it/s, epoch=60, grad_norm=127.309, loss=308.1337, lr=6.65e-04]
Epoch 60 TRAIN:  96%|█████████▌| 131/137 [00:30<00:01,  4.95it/s, epoch=60, grad_norm=137.089, loss=348.4556, lr=6.65e-04]
Epoch 60 TRAIN:  96%|█████████▌| 131/137 [00:30<00:01,  4.95it/s, epoch=60, grad_norm=276.802, loss=295.7700, lr=6.65e-04]
Epoch 60 TRAIN:  96%|█████████▌| 131/137 [00:30<00:01,  4.95it/s, epoch=60, grad_norm=170.998, loss=274.6990, lr=6.65e-04]
Epoch 60 TRAIN:  99%|█████████▊| 135/137 [00:30<00:00,  5.22it/s, epoch=60, grad_norm=170.998, loss=274.6990, lr=6.65e-04]
Epoch 60 TRAIN:  99%|█████████▊| 135/137 [00:30<00:00,  5.22it/s, epoch=60, grad_norm=220.763, loss=404.5034, lr=6.66e-04]
Epoch 60 TRAIN:  99%|█████████▊| 135/137 [00:31<00:00,  5.22it/s, epoch=60, grad_norm=220.782, loss=342.8863, lr=6.66e-04]
Epoch 60 TRAIN:  99%|█████████▊| 135/137 [00:31<00:00,  5.22it/s, epoch=60, grad_norm=182.388, loss=338.6660, lr=6.66e-04]
Epoch 60 TRAIN: 138it [00:31,  5.33it/s, epoch=60, grad_norm=182.388, loss=338.6660, lr=6.66e-04]                         
Epoch 60 TRAIN: 138it [00:31,  4.40it/s, epoch=60, grad_norm=182.388, loss=338.6660, lr=6.66e-04]

CV:   0%|          | 0/6 [00:00<?, ?it/s]/home/rookie/Documents/batuka/projects/wenet/wenet/wenet/utils/train_utils.py:689: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  amp_autocast(enabled=dtype is not None,

CV:  17%|█▋        | 1/6 [00:00<00:03,  1.32it/s]
                                                 
2026-04-16 20:22:58,907 INFO Epoch 60 Step 8321 CV info lr 6.6568e-04 cv_loss 473.64508056640625 rank 0 acc 0.3756066958109538
2026-04-16 20:22:58,909 INFO [Rank 0] Checkpoint: save to checkpoint exp/conformer_mongolian/epoch_60.pt
2026-04-16 20:22:59,022 INFO Epoch 61 Step 8321 TRAIN info lr 6.6568e-04 rank 0
2026-04-16 20:22:59,043 INFO using accumulate grad, new batch size is 1 times larger than before

Epoch 61 TRAIN:   0%|          | 0/138 [00:00<?, ?it/s]
Epoch 61 TRAIN:   0%|          | 0/138 [00:03<?, ?it/s, epoch=61, grad_norm=411.919, loss=332.9931, lr=6.66e-04]
Epoch 61 TRAIN:   1%|          | 1/138 [00:03<08:41,  3.81s/it, epoch=61, grad_norm=411.919, loss=332.9931, lr=6.66e-04]
Epoch 61 TRAIN:   1%|          | 1/138 [00:03<08:41,  3.81s/it, epoch=61, grad_norm=338.959, loss=296.9649, lr=6.66e-04]
Epoch 61 TRAIN:   1%|          | 1/138 [00:04<08:41,  3.81s/it, epoch=61, grad_norm=389.678, loss=345.9838, lr=6.66e-04]
Epoch 61 TRAIN:   1%|          | 1/138 [00:04<08:41,  3.81s/it, epoch=61, grad_norm=415.588, loss=375.3705, lr=6.66e-04]
Epoch 61 TRAIN:   1%|          | 1/138 [00:04<08:41,  3.81s/it, epoch=61, grad_norm=369.372, loss=285.6834, lr=6.66e-04]
Epoch 61 TRAIN:   1%|          | 1/138 [00:04<08:41,  3.81s/it, epoch=61, grad_norm=369.148, loss=345.9388, lr=6.66e-04]
Epoch 61 TRAIN:   1%|          | 1/138 [00:04<08:41,  3.81s/it, epoch=61, grad_norm=157.826, loss=259.1127, lr=6.66e-04]
Epoch 61 TRAIN:   5%|▌         | 7/138 [00:04<01:02,  2.10it/s, epoch=61, grad_norm=157.826, loss=259.1127, lr=6.66e-04]
Epoch 61 TRAIN:   5%|▌         | 7/138 [00:04<01:02,  2.10it/s, epoch=61, grad_norm=374.983, loss=355.1487, lr=6.66e-04]
Epoch 61 TRAIN:   5%|▌         | 7/138 [00:04<01:02,  2.10it/s, epoch=61, grad_norm=142.363, loss=296.8964, lr=6.66e-04]
Epoch 61 TRAIN:   5%|▌         | 7/138 [00:04<01:02,  2.10it/s, epoch=61, grad_norm=125.571, loss=300.2120, lr=6.66e-04]
Epoch 61 TRAIN:   5%|▌         | 7/138 [00:04<01:02,  2.10it/s, epoch=61, grad_norm=101.145, loss=265.1192, lr=6.67e-04]
Epoch 61 TRAIN:   8%|▊         | 11/138 [00:04<00:40,  3.15it/s, epoch=61, grad_norm=101.145, loss=265.1192, lr=6.67e-04]
Epoch 61 TRAIN:   8%|▊         | 11/138 [00:05<00:40,  3.15it/s, epoch=61, grad_norm=98.180, loss=292.4782, lr=6.67e-04] 
Epoch 61 TRAIN:   8%|▊         | 11/138 [00:05<00:40,  3.15it/s, epoch=61, grad_norm=93.542, loss=273.5424, lr=6.67e-04]
Epoch 61 TRAIN:   8%|▊         | 11/138 [00:05<00:40,  3.15it/s, epoch=61, grad_norm=93.055, loss=305.1763, lr=6.67e-04]
Epoch 61 TRAIN:   8%|▊         | 11/138 [00:05<00:40,  3.15it/s, epoch=61, grad_norm=98.006, loss=266.1981, lr=6.67e-04]
Epoch 61 TRAIN:  11%|█         | 15/138 [00:05<00:31,  3.90it/s, epoch=61, grad_norm=98.006, loss=266.1981, lr=6.67e-04]
Epoch 61 TRAIN:  11%|█         | 15/138 [00:05<00:31,  3.90it/s, epoch=61, grad_norm=122.479, loss=303.9185, lr=6.67e-04]
Epoch 61 TRAIN:  11%|█         | 15/138 [00:05<00:31,  3.90it/s, epoch=61, grad_norm=105.508, loss=283.8923, lr=6.67e-04]
Epoch 61 TRAIN:  11%|█         | 15/138 [00:06<00:31,  3.90it/s, epoch=61, grad_norm=83.515, loss=271.8434, lr=6.67e-04] 
Epoch 61 TRAIN:  11%|█         | 15/138 [00:06<00:31,  3.90it/s, epoch=61, grad_norm=201.394, loss=329.0643, lr=6.67e-04]
Epoch 61 TRAIN:  14%|█▍        | 19/138 [00:06<00:26,  4.42it/s, epoch=61, grad_norm=201.394, loss=329.0643, lr=6.67e-04]
Epoch 61 TRAIN:  14%|█▍        | 19/138 [00:06<00:26,  4.42it/s, epoch=61, grad_norm=100.509, loss=278.1433, lr=6.67e-04]
Epoch 61 TRAIN:  14%|█▍        | 19/138 [00:06<00:26,  4.42it/s, epoch=61, grad_norm=94.454, loss=258.3054, lr=6.67e-04] 
Epoch 61 TRAIN:  14%|█▍        | 19/138 [00:06<00:26,  4.42it/s, epoch=61, grad_norm=126.686, loss=327.8459, lr=6.67e-04]
Epoch 61 TRAIN:  14%|█▍        | 19/138 [00:07<00:26,  4.42it/s, epoch=61, grad_norm=108.900, loss=281.9931, lr=6.68e-04]
Epoch 61 TRAIN:  17%|█▋        | 23/138 [00:07<00:25,  4.58it/s, epoch=61, grad_norm=108.900, loss=281.9931, lr=6.68e-04]
Epoch 61 TRAIN:  17%|█▋        | 23/138 [00:07<00:25,  4.58it/s, epoch=61, grad_norm=88.183, loss=303.9496, lr=6.68e-04] 
Epoch 61 TRAIN:  17%|█▋        | 23/138 [00:07<00:25,  4.58it/s, epoch=61, grad_norm=186.234, loss=264.2725, lr=6.68e-04]
Epoch 61 TRAIN:  17%|█▋        | 23/138 [00:07<00:25,  4.58it/s, epoch=61, grad_norm=187.804, loss=283.3891, lr=6.68e-04]
Epoch 61 TRAIN:  19%|█▉        | 26/138 [00:07<00:23,  4.67it/s, epoch=61, grad_norm=187.804, loss=283.3891, lr=6.68e-04]
Epoch 61 TRAIN:  19%|█▉        | 26/138 [00:07<00:23,  4.67it/s, epoch=61, grad_norm=155.273, loss=322.5212, lr=6.68e-04]
Epoch 61 TRAIN:  19%|█▉        | 26/138 [00:07<00:23,  4.67it/s, epoch=61, grad_norm=100.287, loss=300.2391, lr=6.68e-04]
Epoch 61 TRAIN:  19%|█▉        | 26/138 [00:08<00:23,  4.67it/s, epoch=61, grad_norm=96.470, loss=312.5229, lr=6.68e-04] 
Epoch 61 TRAIN:  21%|██        | 29/138 [00:08<00:22,  4.78it/s, epoch=61, grad_norm=96.470, loss=312.5229, lr=6.68e-04]
Epoch 61 TRAIN:  21%|██        | 29/138 [00:08<00:22,  4.78it/s, epoch=61, grad_norm=166.910, loss=256.9279, lr=6.68e-04]
Epoch 61 TRAIN:  21%|██        | 29/138 [00:08<00:22,  4.78it/s, epoch=61, grad_norm=137.299, loss=270.6844, lr=6.68e-04]
Epoch 61 TRAIN:  21%|██        | 29/138 [00:08<00:22,  4.78it/s, epoch=61, grad_norm=133.202, loss=310.0157, lr=6.68e-04]
Epoch 61 TRAIN:  23%|██▎       | 32/138 [00:08<00:21,  4.84it/s, epoch=61, grad_norm=133.202, loss=310.0157, lr=6.68e-04]
Epoch 61 TRAIN:  23%|██▎       | 32/138 [00:09<00:21,  4.84it/s, epoch=61, grad_norm=118.655, loss=264.4806, lr=6.68e-04]
Epoch 61 TRAIN:  23%|██▎       | 32/138 [00:09<00:21,  4.84it/s, epoch=61, grad_norm=134.000, loss=280.2057, lr=6.68e-04]
Epoch 61 TRAIN:  23%|██▎       | 32/138 [00:09<00:21,  4.84it/s, epoch=61, grad_norm=132.725, loss=310.5950, lr=6.68e-04]
Epoch 61 TRAIN:  25%|██▌       | 35/138 [00:09<00:20,  4.91it/s, epoch=61, grad_norm=132.725, loss=310.5950, lr=6.68e-04]
Epoch 61 TRAIN:  25%|██▌       | 35/138 [00:09<00:20,  4.91it/s, epoch=61, grad_norm=103.073, loss=289.2794, lr=6.69e-04]
Epoch 61 TRAIN:  25%|██▌       | 35/138 [00:09<00:20,  4.91it/s, epoch=61, grad_norm=188.615, loss=297.9120, lr=6.69e-04]
Epoch 61 TRAIN:  25%|██▌       | 35/138 [00:10<00:20,  4.91it/s, epoch=61, grad_norm=124.720, loss=274.2348, lr=6.69e-04]
Epoch 61 TRAIN:  28%|██▊       | 38/138 [00:10<00:20,  4.82it/s, epoch=61, grad_norm=124.720, loss=274.2348, lr=6.69e-04]
Epoch 61 TRAIN:  28%|██▊       | 38/138 [00:10<00:20,  4.82it/s, epoch=61, grad_norm=186.876, loss=272.7834, lr=6.69e-04]
Epoch 61 TRAIN:  28%|██▊       | 38/138 [00:10<00:20,  4.82it/s, epoch=61, grad_norm=172.513, loss=293.4513, lr=6.69e-04]
Epoch 61 TRAIN:  28%|██▊       | 38/138 [00:10<00:20,  4.82it/s, epoch=61, grad_norm=165.439, loss=347.0920, lr=6.69e-04]
Epoch 61 TRAIN:  30%|██▉       | 41/138 [00:10<00:20,  4.75it/s, epoch=61, grad_norm=165.439, loss=347.0920, lr=6.69e-04]
Epoch 61 TRAIN:  30%|██▉       | 41/138 [00:10<00:20,  4.75it/s, epoch=61, grad_norm=155.468, loss=247.1655, lr=6.69e-04]
Epoch 61 TRAIN:  30%|██▉       | 41/138 [00:11<00:20,  4.75it/s, epoch=61, grad_norm=94.895, loss=311.0256, lr=6.69e-04] 
Epoch 61 TRAIN:  30%|██▉       | 41/138 [00:11<00:20,  4.75it/s, epoch=61, grad_norm=121.131, loss=335.0864, lr=6.69e-04]
Epoch 61 TRAIN:  32%|███▏      | 44/138 [00:11<00:19,  4.72it/s, epoch=61, grad_norm=121.131, loss=335.0864, lr=6.69e-04]
Epoch 61 TRAIN:  32%|███▏      | 44/138 [00:11<00:19,  4.72it/s, epoch=61, grad_norm=127.028, loss=280.9127, lr=6.69e-04]
Epoch 61 TRAIN:  32%|███▏      | 44/138 [00:11<00:19,  4.72it/s, epoch=61, grad_norm=94.936, loss=312.4702, lr=6.69e-04] 
Epoch 61 TRAIN:  32%|███▏      | 44/138 [00:11<00:19,  4.72it/s, epoch=61, grad_norm=128.893, loss=329.0798, lr=6.69e-04]
Epoch 61 TRAIN:  34%|███▍      | 47/138 [00:11<00:19,  4.72it/s, epoch=61, grad_norm=128.893, loss=329.0798, lr=6.69e-04]
Epoch 61 TRAIN:  34%|███▍      | 47/138 [00:12<00:19,  4.72it/s, epoch=61, grad_norm=99.663, loss=306.7172, lr=6.70e-04] 
Epoch 61 TRAIN:  34%|███▍      | 47/138 [00:12<00:19,  4.72it/s, epoch=61, grad_norm=136.556, loss=305.8381, lr=6.70e-04]
Epoch 61 TRAIN:  34%|███▍      | 47/138 [00:12<00:19,  4.72it/s, epoch=61, grad_norm=161.989, loss=279.4598, lr=6.70e-04]
Epoch 61 TRAIN:  36%|███▌      | 50/138 [00:12<00:17,  4.90it/s, epoch=61, grad_norm=161.989, loss=279.4598, lr=6.70e-04]
Epoch 61 TRAIN:  36%|███▌      | 50/138 [00:12<00:17,  4.90it/s, epoch=61, grad_norm=98.727, loss=302.3807, lr=6.70e-04] 
Epoch 61 TRAIN:  36%|███▌      | 50/138 [00:12<00:17,  4.90it/s, epoch=61, grad_norm=146.202, loss=240.7194, lr=6.70e-04]
Epoch 61 TRAIN:  36%|███▌      | 50/138 [00:13<00:17,  4.90it/s, epoch=61, grad_norm=84.706, loss=297.5245, lr=6.70e-04] 
Epoch 61 TRAIN:  38%|███▊      | 53/138 [00:13<00:17,  4.72it/s, epoch=61, grad_norm=84.706, loss=297.5245, lr=6.70e-04]
Epoch 61 TRAIN:  38%|███▊      | 53/138 [00:13<00:17,  4.72it/s, epoch=61, grad_norm=95.338, loss=299.8422, lr=6.70e-04]
Epoch 61 TRAIN:  38%|███▊      | 53/138 [00:13<00:17,  4.72it/s, epoch=61, grad_norm=152.757, loss=309.9409, lr=6.70e-04]
Epoch 61 TRAIN:  38%|███▊      | 53/138 [00:13<00:17,  4.72it/s, epoch=61, grad_norm=118.112, loss=244.1533, lr=6.70e-04]
Epoch 61 TRAIN:  41%|████      | 56/138 [00:13<00:17,  4.81it/s, epoch=61, grad_norm=118.112, loss=244.1533, lr=6.70e-04]
Epoch 61 TRAIN:  41%|████      | 56/138 [00:14<00:17,  4.81it/s, epoch=61, grad_norm=109.800, loss=309.5605, lr=6.70e-04]
Epoch 61 TRAIN:  41%|████      | 56/138 [00:14<00:17,  4.81it/s, epoch=61, grad_norm=170.971, loss=330.6103, lr=6.70e-04]
Epoch 61 TRAIN:  41%|████      | 56/138 [00:14<00:17,  4.81it/s, epoch=61, grad_norm=91.024, loss=290.2078, lr=6.70e-04] 
Epoch 61 TRAIN:  43%|████▎     | 59/138 [00:14<00:16,  4.87it/s, epoch=61, grad_norm=91.024, loss=290.2078, lr=6.70e-04]
Epoch 61 TRAIN:  43%|████▎     | 59/138 [00:14<00:16,  4.87it/s, epoch=61, grad_norm=118.332, loss=264.2214, lr=6.70e-04]
Epoch 61 TRAIN:  43%|████▎     | 59/138 [00:14<00:16,  4.87it/s, epoch=61, grad_norm=172.204, loss=323.7950, lr=6.71e-04]
Epoch 61 TRAIN:  43%|████▎     | 59/138 [00:15<00:16,  4.87it/s, epoch=61, grad_norm=85.091, loss=298.5583, lr=6.71e-04] 
Epoch 61 TRAIN:  45%|████▍     | 62/138 [00:15<00:15,  4.79it/s, epoch=61, grad_norm=85.091, loss=298.5583, lr=6.71e-04]
Epoch 61 TRAIN:  45%|████▍     | 62/138 [00:15<00:15,  4.79it/s, epoch=61, grad_norm=167.879, loss=327.4757, lr=6.71e-04]
Epoch 61 TRAIN:  45%|████▍     | 62/138 [00:15<00:15,  4.79it/s, epoch=61, grad_norm=131.310, loss=320.2201, lr=6.71e-04]
Epoch 61 TRAIN:  45%|████▍     | 62/138 [00:15<00:15,  4.79it/s, epoch=61, grad_norm=129.237, loss=280.7048, lr=6.71e-04]
Epoch 61 TRAIN:  47%|████▋     | 65/138 [00:15<00:15,  4.69it/s, epoch=61, grad_norm=129.237, loss=280.7048, lr=6.71e-04]
Epoch 61 TRAIN:  47%|████▋     | 65/138 [00:15<00:15,  4.69it/s, epoch=61, grad_norm=169.402, loss=243.7000, lr=6.71e-04]
Epoch 61 TRAIN:  47%|████▋     | 65/138 [00:16<00:15,  4.69it/s, epoch=61, grad_norm=92.421, loss=288.9638, lr=6.71e-04] 
Epoch 61 TRAIN:  47%|████▋     | 65/138 [00:16<00:15,  4.69it/s, epoch=61, grad_norm=106.423, loss=282.5014, lr=6.71e-04]
Epoch 61 TRAIN:  49%|████▉     | 68/138 [00:16<00:14,  4.78it/s, epoch=61, grad_norm=106.423, loss=282.5014, lr=6.71e-04]
Epoch 61 TRAIN:  49%|████▉     | 68/138 [00:16<00:14,  4.78it/s, epoch=61, grad_norm=130.711, loss=264.2330, lr=6.71e-04]
Epoch 61 TRAIN:  49%|████▉     | 68/138 [00:16<00:14,  4.78it/s, epoch=61, grad_norm=116.500, loss=260.2569, lr=6.71e-04]
Epoch 61 TRAIN:  49%|████▉     | 68/138 [00:17<00:14,  4.78it/s, epoch=61, grad_norm=110.441, loss=276.6744, lr=6.71e-04]
Epoch 61 TRAIN:  51%|█████▏    | 71/138 [00:17<00:15,  4.45it/s, epoch=61, grad_norm=110.441, loss=276.6744, lr=6.71e-04]
Epoch 61 TRAIN:  51%|█████▏    | 71/138 [00:17<00:15,  4.45it/s, epoch=61, grad_norm=101.924, loss=248.5862, lr=6.71e-04]
Epoch 61 TRAIN:  51%|█████▏    | 71/138 [00:17<00:15,  4.45it/s, epoch=61, grad_norm=147.173, loss=223.3585, lr=6.72e-04]
Epoch 61 TRAIN:  51%|█████▏    | 71/138 [00:17<00:15,  4.45it/s, epoch=61, grad_norm=117.088, loss=217.7824, lr=6.72e-04]
Epoch 61 TRAIN:  54%|█████▎    | 74/138 [00:17<00:13,  4.74it/s, epoch=61, grad_norm=117.088, loss=217.7824, lr=6.72e-04]
Epoch 61 TRAIN:  54%|█████▎    | 74/138 [00:17<00:13,  4.74it/s, epoch=61, grad_norm=201.939, loss=310.6059, lr=6.72e-04]
Epoch 61 TRAIN:  54%|█████▎    | 74/138 [00:18<00:13,  4.74it/s, epoch=61, grad_norm=99.897, loss=238.8206, lr=6.72e-04] 
Epoch 61 TRAIN:  54%|█████▎    | 74/138 [00:18<00:13,  4.74it/s, epoch=61, grad_norm=125.890, loss=268.4570, lr=6.72e-04]
Epoch 61 TRAIN:  56%|█████▌    | 77/138 [00:18<00:13,  4.51it/s, epoch=61, grad_norm=125.890, loss=268.4570, lr=6.72e-04]
Epoch 61 TRAIN:  56%|█████▌    | 77/138 [00:18<00:13,  4.51it/s, epoch=61, grad_norm=113.664, loss=274.5505, lr=6.72e-04]
Epoch 61 TRAIN:  56%|█████▌    | 77/138 [00:18<00:13,  4.51it/s, epoch=61, grad_norm=250.929, loss=327.3705, lr=6.72e-04]
Epoch 61 TRAIN:  56%|█████▌    | 77/138 [00:19<00:13,  4.51it/s, epoch=61, grad_norm=160.572, loss=260.2191, lr=6.72e-04]
Epoch 61 TRAIN:  58%|█████▊    | 80/138 [00:19<00:12,  4.58it/s, epoch=61, grad_norm=160.572, loss=260.2191, lr=6.72e-04]
Epoch 61 TRAIN:  58%|█████▊    | 80/138 [00:19<00:12,  4.58it/s, epoch=61, grad_norm=151.007, loss=322.3841, lr=6.72e-04]
Epoch 61 TRAIN:  58%|█████▊    | 80/138 [00:19<00:12,  4.58it/s, epoch=61, grad_norm=127.544, loss=305.0793, lr=6.72e-04]
Epoch 61 TRAIN:  58%|█████▊    | 80/138 [00:19<00:12,  4.58it/s, epoch=61, grad_norm=168.332, loss=271.1386, lr=6.72e-04]
Epoch 61 TRAIN:  60%|██████    | 83/138 [00:19<00:11,  4.76it/s, epoch=61, grad_norm=168.332, loss=271.1386, lr=6.72e-04]
Epoch 61 TRAIN:  60%|██████    | 83/138 [00:19<00:11,  4.76it/s, epoch=61, grad_norm=184.954, loss=282.1842, lr=6.72e-04]
Epoch 61 TRAIN:  60%|██████    | 83/138 [00:20<00:11,  4.76it/s, epoch=61, grad_norm=97.277, loss=306.4089, lr=6.72e-04] 
Epoch 61 TRAIN:  60%|██████    | 83/138 [00:20<00:11,  4.76it/s, epoch=61, grad_norm=117.357, loss=358.4660, lr=6.73e-04]
Epoch 61 TRAIN:  62%|██████▏   | 86/138 [00:20<00:11,  4.56it/s, epoch=61, grad_norm=117.357, loss=358.4660, lr=6.73e-04]
Epoch 61 TRAIN:  62%|██████▏   | 86/138 [00:20<00:11,  4.56it/s, epoch=61, grad_norm=130.007, loss=310.9561, lr=6.73e-04]
Epoch 61 TRAIN:  62%|██████▏   | 86/138 [00:20<00:11,  4.56it/s, epoch=61, grad_norm=102.877, loss=309.4400, lr=6.73e-04]
Epoch 61 TRAIN:  62%|██████▏   | 86/138 [00:20<00:11,  4.56it/s, epoch=61, grad_norm=261.062, loss=238.2933, lr=6.73e-04]
Epoch 61 TRAIN:  64%|██████▍   | 89/138 [00:20<00:10,  4.56it/s, epoch=61, grad_norm=261.062, loss=238.2933, lr=6.73e-04]
Epoch 61 TRAIN:  64%|██████▍   | 89/138 [00:21<00:10,  4.56it/s, epoch=61, grad_norm=345.613, loss=227.1077, lr=6.73e-04]
Epoch 61 TRAIN:  64%|██████▍   | 89/138 [00:21<00:10,  4.56it/s, epoch=61, grad_norm=171.062, loss=294.6354, lr=6.73e-04]
Epoch 61 TRAIN:  64%|██████▍   | 89/138 [00:21<00:10,  4.56it/s, epoch=61, grad_norm=184.509, loss=229.5369, lr=6.73e-04]
Epoch 61 TRAIN:  67%|██████▋   | 92/138 [00:21<00:10,  4.57it/s, epoch=61, grad_norm=184.509, loss=229.5369, lr=6.73e-04]
Epoch 61 TRAIN:  67%|██████▋   | 92/138 [00:21<00:10,  4.57it/s, epoch=61, grad_norm=88.465, loss=258.6326, lr=6.73e-04] 
Epoch 61 TRAIN:  67%|██████▋   | 92/138 [00:22<00:10,  4.57it/s, epoch=61, grad_norm=180.830, loss=228.4571, lr=6.73e-04]
Epoch 61 TRAIN:  67%|██████▋   | 92/138 [00:22<00:10,  4.57it/s, epoch=61, grad_norm=327.533, loss=365.0290, lr=6.73e-04]
Epoch 61 TRAIN:  69%|██████▉   | 95/138 [00:22<00:09,  4.43it/s, epoch=61, grad_norm=327.533, loss=365.0290, lr=6.73e-04]
Epoch 61 TRAIN:  69%|██████▉   | 95/138 [00:22<00:09,  4.43it/s, epoch=61, grad_norm=327.294, loss=339.7531, lr=6.73e-04]
Epoch 61 TRAIN:  69%|██████▉   | 95/138 [00:22<00:09,  4.43it/s, epoch=61, grad_norm=164.862, loss=267.1447, lr=6.73e-04]
Epoch 61 TRAIN:  69%|██████▉   | 95/138 [00:22<00:09,  4.43it/s, epoch=61, grad_norm=123.125, loss=254.0442, lr=6.74e-04]
Epoch 61 TRAIN:  71%|███████   | 98/138 [00:22<00:08,  4.57it/s, epoch=61, grad_norm=123.125, loss=254.0442, lr=6.74e-04]
Epoch 61 TRAIN:  71%|███████   | 98/138 [00:23<00:08,  4.57it/s, epoch=61, grad_norm=96.450, loss=230.7771, lr=6.74e-04] 
Epoch 61 TRAIN:  71%|███████   | 98/138 [00:23<00:08,  4.57it/s, epoch=61, grad_norm=107.947, loss=221.4453, lr=6.74e-04]2026-04-16 20:23:22,403 DEBUG TRAIN | steps/sec 4.308| Batch 61/100 loss 221.445312 loss_att 183.392456 loss_ctc 310.235352 th_accuracy 0.392330 lr 6.7368e-04 grad_norm 107.947479 rank 0

Epoch 61 TRAIN:  71%|███████   | 98/138 [00:23<00:08,  4.57it/s, epoch=61, grad_norm=109.829, loss=245.5799, lr=6.74e-04]
Epoch 61 TRAIN:  73%|███████▎  | 101/138 [00:23<00:07,  4.71it/s, epoch=61, grad_norm=109.829, loss=245.5799, lr=6.74e-04]
Epoch 61 TRAIN:  73%|███████▎  | 101/138 [00:23<00:07,  4.71it/s, epoch=61, grad_norm=113.474, loss=276.0058, lr=6.74e-04]
Epoch 61 TRAIN:  73%|███████▎  | 101/138 [00:24<00:07,  4.71it/s, epoch=61, grad_norm=265.044, loss=308.6732, lr=6.74e-04]
Epoch 61 TRAIN:  73%|███████▎  | 101/138 [00:24<00:07,  4.71it/s, epoch=61, grad_norm=151.692, loss=304.7001, lr=6.74e-04]
Epoch 61 TRAIN:  75%|███████▌  | 104/138 [00:24<00:07,  4.61it/s, epoch=61, grad_norm=151.692, loss=304.7001, lr=6.74e-04]
Epoch 61 TRAIN:  75%|███████▌  | 104/138 [00:24<00:07,  4.61it/s, epoch=61, grad_norm=170.965, loss=290.6154, lr=6.74e-04]
Epoch 61 TRAIN:  75%|███████▌  | 104/138 [00:24<00:07,  4.61it/s, epoch=61, grad_norm=178.464, loss=324.6935, lr=6.74e-04]
Epoch 61 TRAIN:  75%|███████▌  | 104/138 [00:24<00:07,  4.61it/s, epoch=61, grad_norm=267.043, loss=224.5149, lr=6.74e-04]
Epoch 61 TRAIN:  78%|███████▊  | 107/138 [00:24<00:06,  4.56it/s, epoch=61, grad_norm=267.043, loss=224.5149, lr=6.74e-04]
Epoch 61 TRAIN:  78%|███████▊  | 107/138 [00:25<00:06,  4.56it/s, epoch=61, grad_norm=163.760, loss=243.8096, lr=6.74e-04]
Epoch 61 TRAIN:  78%|███████▊  | 107/138 [00:25<00:06,  4.56it/s, epoch=61, grad_norm=130.155, loss=294.5537, lr=6.74e-04]
Epoch 61 TRAIN:  78%|███████▊  | 107/138 [00:25<00:06,  4.56it/s, epoch=61, grad_norm=167.756, loss=299.3813, lr=6.74e-04]
Epoch 61 TRAIN:  80%|███████▉  | 110/138 [00:25<00:06,  4.36it/s, epoch=61, grad_norm=167.756, loss=299.3813, lr=6.74e-04]
Epoch 61 TRAIN:  80%|███████▉  | 110/138 [00:25<00:06,  4.36it/s, epoch=61, grad_norm=246.208, loss=292.7310, lr=6.75e-04]
Epoch 61 TRAIN:  80%|███████▉  | 110/138 [00:26<00:06,  4.36it/s, epoch=61, grad_norm=141.874, loss=296.8317, lr=6.75e-04]
Epoch 61 TRAIN:  80%|███████▉  | 110/138 [00:26<00:06,  4.36it/s, epoch=61, grad_norm=179.002, loss=296.9941, lr=6.75e-04]
Epoch 61 TRAIN:  80%|███████▉  | 110/138 [00:26<00:06,  4.36it/s, epoch=61, grad_norm=165.662, loss=320.7905, lr=6.75e-04]
Epoch 61 TRAIN:  83%|████████▎ | 114/138 [00:26<00:05,  4.77it/s, epoch=61, grad_norm=165.662, loss=320.7905, lr=6.75e-04]
Epoch 61 TRAIN:  83%|████████▎ | 114/138 [00:26<00:05,  4.77it/s, epoch=61, grad_norm=227.326, loss=340.4099, lr=6.75e-04]
Epoch 61 TRAIN:  83%|████████▎ | 114/138 [00:26<00:05,  4.77it/s, epoch=61, grad_norm=142.679, loss=295.1091, lr=6.75e-04]
Epoch 61 TRAIN:  83%|████████▎ | 114/138 [00:27<00:05,  4.77it/s, epoch=61, grad_norm=166.605, loss=222.9621, lr=6.75e-04]
Epoch 61 TRAIN:  85%|████████▍ | 117/138 [00:27<00:04,  4.49it/s, epoch=61, grad_norm=166.605, loss=222.9621, lr=6.75e-04]
Epoch 61 TRAIN:  85%|████████▍ | 117/138 [00:27<00:04,  4.49it/s, epoch=61, grad_norm=118.567, loss=248.2242, lr=6.75e-04]
Epoch 61 TRAIN:  85%|████████▍ | 117/138 [00:27<00:04,  4.49it/s, epoch=61, grad_norm=162.230, loss=235.6638, lr=6.75e-04]
Epoch 61 TRAIN:  85%|████████▍ | 117/138 [00:27<00:04,  4.49it/s, epoch=61, grad_norm=210.770, loss=343.6740, lr=6.75e-04]
Epoch 61 TRAIN:  87%|████████▋ | 120/138 [00:27<00:03,  4.81it/s, epoch=61, grad_norm=210.770, loss=343.6740, lr=6.75e-04]
Epoch 61 TRAIN:  87%|████████▋ | 120/138 [00:27<00:03,  4.81it/s, epoch=61, grad_norm=175.246, loss=313.8269, lr=6.75e-04]
Epoch 61 TRAIN:  87%|████████▋ | 120/138 [00:27<00:03,  4.81it/s, epoch=61, grad_norm=206.520, loss=360.8123, lr=6.75e-04]
Epoch 61 TRAIN:  87%|████████▋ | 120/138 [00:28<00:03,  4.81it/s, epoch=61, grad_norm=120.985, loss=296.2980, lr=6.76e-04]
Epoch 61 TRAIN:  89%|████████▉ | 123/138 [00:28<00:03,  4.85it/s, epoch=61, grad_norm=120.985, loss=296.2980, lr=6.76e-04]
Epoch 61 TRAIN:  89%|████████▉ | 123/138 [00:28<00:03,  4.85it/s, epoch=61, grad_norm=161.528, loss=350.8659, lr=6.76e-04]
Epoch 61 TRAIN:  89%|████████▉ | 123/138 [00:28<00:03,  4.85it/s, epoch=61, grad_norm=172.607, loss=158.1835, lr=6.76e-04]
Epoch 61 TRAIN:  89%|████████▉ | 123/138 [00:28<00:03,  4.85it/s, epoch=61, grad_norm=212.352, loss=338.5358, lr=6.76e-04]
Epoch 61 TRAIN:  91%|█████████▏| 126/138 [00:28<00:02,  4.75it/s, epoch=61, grad_norm=212.352, loss=338.5358, lr=6.76e-04]
Epoch 61 TRAIN:  91%|█████████▏| 126/138 [00:29<00:02,  4.75it/s, epoch=61, grad_norm=212.607, loss=361.2269, lr=6.76e-04]
Epoch 61 TRAIN:  91%|█████████▏| 126/138 [00:29<00:02,  4.75it/s, epoch=61, grad_norm=184.132, loss=284.6918, lr=6.76e-04]
Epoch 61 TRAIN:  91%|█████████▏| 126/138 [00:29<00:02,  4.75it/s, epoch=61, grad_norm=215.912, loss=339.8110, lr=6.76e-04]
Epoch 61 TRAIN:  91%|█████████▏| 126/138 [00:29<00:02,  4.75it/s, epoch=61, grad_norm=198.158, loss=334.4830, lr=6.76e-04]
Epoch 61 TRAIN:  94%|█████████▍| 130/138 [00:29<00:01,  5.37it/s, epoch=61, grad_norm=198.158, loss=334.4830, lr=6.76e-04]
Epoch 61 TRAIN:  94%|█████████▍| 130/138 [00:29<00:01,  5.37it/s, epoch=61, grad_norm=197.382, loss=324.3685, lr=6.76e-04]
Epoch 61 TRAIN:  94%|█████████▍| 130/138 [00:30<00:01,  5.37it/s, epoch=61, grad_norm=186.451, loss=273.7926, lr=6.76e-04]
Epoch 61 TRAIN:  94%|█████████▍| 130/138 [00:30<00:01,  5.37it/s, epoch=61, grad_norm=178.218, loss=341.5359, lr=6.76e-04]
Epoch 61 TRAIN:  96%|█████████▋| 133/138 [00:30<00:00,  5.01it/s, epoch=61, grad_norm=178.218, loss=341.5359, lr=6.76e-04]
Epoch 61 TRAIN:  96%|█████████▋| 133/138 [00:30<00:00,  5.01it/s, epoch=61, grad_norm=112.692, loss=234.6366, lr=6.76e-04]
Epoch 61 TRAIN:  96%|█████████▋| 133/138 [00:30<00:00,  5.01it/s, epoch=61, grad_norm=180.751, loss=268.3377, lr=6.76e-04]
Epoch 61 TRAIN:  96%|█████████▋| 133/138 [00:30<00:00,  5.01it/s, epoch=61, grad_norm=236.186, loss=263.2132, lr=6.77e-04]
Epoch 61 TRAIN:  99%|█████████▊| 136/138 [00:30<00:00,  5.22it/s, epoch=61, grad_norm=236.186, loss=263.2132, lr=6.77e-04]
Epoch 61 TRAIN:  99%|█████████▊| 136/138 [00:30<00:00,  4.43it/s, epoch=61, grad_norm=236.186, loss=263.2132, lr=6.77e-04]

CV:   0%|          | 0/6 [00:00<?, ?it/s]/home/rookie/Documents/batuka/projects/wenet/wenet/wenet/utils/train_utils.py:689: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  amp_autocast(enabled=dtype is not None,

CV:  17%|█▋        | 1/6 [00:00<00:03,  1.33it/s]
                                                 
2026-04-16 20:23:30,624 INFO Epoch 61 Step 8457 CV info lr 6.7656e-04 cv_loss 470.7385753284801 rank 0 acc 0.35050158699353534
2026-04-16 20:23:30,626 INFO [Rank 0] Checkpoint: save to checkpoint exp/conformer_mongolian/epoch_61.pt
2026-04-16 20:23:30,745 INFO Epoch 62 Step 8457 TRAIN info lr 6.7656e-04 rank 0
2026-04-16 20:23:30,759 INFO using accumulate grad, new batch size is 1 times larger than before

Epoch 62 TRAIN:   0%|          | 0/136 [00:00<?, ?it/s]
Epoch 62 TRAIN:   0%|          | 0/136 [00:03<?, ?it/s, epoch=62, grad_norm=391.291, loss=317.1401, lr=6.77e-04]
Epoch 62 TRAIN:   1%|          | 1/136 [00:03<08:45,  3.90s/it, epoch=62, grad_norm=391.291, loss=317.1401, lr=6.77e-04]
Epoch 62 TRAIN:   1%|          | 1/136 [00:04<08:45,  3.90s/it, epoch=62, grad_norm=407.198, loss=321.3495, lr=6.77e-04]
Epoch 62 TRAIN:   1%|          | 1/136 [00:04<08:45,  3.90s/it, epoch=62, grad_norm=602.565, loss=451.1320, lr=6.77e-04]
Epoch 62 TRAIN:   1%|          | 1/136 [00:04<08:45,  3.90s/it, epoch=62, grad_norm=472.585, loss=361.2683, lr=6.77e-04]
Epoch 62 TRAIN:   1%|          | 1/136 [00:04<08:45,  3.90s/it, epoch=62, grad_norm=436.501, loss=356.0669, lr=6.77e-04]
Epoch 62 TRAIN:   1%|          | 1/136 [00:04<08:45,  3.90s/it, epoch=62, grad_norm=236.965, loss=251.7912, lr=6.77e-04]
Epoch 62 TRAIN:   1%|          | 1/136 [00:04<08:45,  3.90s/it, epoch=62, grad_norm=113.634, loss=218.9539, lr=6.77e-04]
Epoch 62 TRAIN:   5%|▌         | 7/136 [00:04<01:02,  2.06it/s, epoch=62, grad_norm=113.634, loss=218.9539, lr=6.77e-04]
Epoch 62 TRAIN:   5%|▌         | 7/136 [00:04<01:02,  2.06it/s, epoch=62, grad_norm=238.817, loss=312.6903, lr=6.77e-04]
Epoch 62 TRAIN:   5%|▌         | 7/136 [00:04<01:02,  2.06it/s, epoch=62, grad_norm=158.479, loss=283.9777, lr=6.77e-04]
Epoch 62 TRAIN:   5%|▌         | 7/136 [00:04<01:02,  2.06it/s, epoch=62, grad_norm=106.615, loss=262.0338, lr=6.77e-04]
Epoch 62 TRAIN:   5%|▌         | 7/136 [00:04<01:02,  2.06it/s, epoch=62, grad_norm=134.884, loss=323.0614, lr=6.77e-04]
Epoch 62 TRAIN:   8%|▊         | 11/136 [00:04<00:40,  3.06it/s, epoch=62, grad_norm=134.884, loss=323.0614, lr=6.77e-04]
Epoch 62 TRAIN:   8%|▊         | 11/136 [00:05<00:40,  3.06it/s, epoch=62, grad_norm=245.397, loss=307.3100, lr=6.78e-04]
Epoch 62 TRAIN:   8%|▊         | 11/136 [00:05<00:40,  3.06it/s, epoch=62, grad_norm=222.148, loss=304.9286, lr=6.78e-04]
Epoch 62 TRAIN:   8%|▊         | 11/136 [00:05<00:40,  3.06it/s, epoch=62, grad_norm=202.457, loss=285.3177, lr=6.78e-04]
Epoch 62 TRAIN:   8%|▊         | 11/136 [00:05<00:40,  3.06it/s, epoch=62, grad_norm=120.130, loss=354.6182, lr=6.78e-04]
Epoch 62 TRAIN:  11%|█         | 15/136 [00:05<00:31,  3.83it/s, epoch=62, grad_norm=120.130, loss=354.6182, lr=6.78e-04]
Epoch 62 TRAIN:  11%|█         | 15/136 [00:05<00:31,  3.83it/s, epoch=62, grad_norm=85.594, loss=292.5158, lr=6.78e-04] 
Epoch 62 TRAIN:  11%|█         | 15/136 [00:05<00:31,  3.83it/s, epoch=62, grad_norm=118.259, loss=267.3829, lr=6.78e-04]
Epoch 62 TRAIN:  11%|█         | 15/136 [00:06<00:31,  3.83it/s, epoch=62, grad_norm=120.978, loss=311.9844, lr=6.78e-04]
Epoch 62 TRAIN:  11%|█         | 15/136 [00:06<00:31,  3.83it/s, epoch=62, grad_norm=79.233, loss=284.4792, lr=6.78e-04] 
Epoch 62 TRAIN:  14%|█▍        | 19/136 [00:06<00:27,  4.21it/s, epoch=62, grad_norm=79.233, loss=284.4792, lr=6.78e-04]
Epoch 62 TRAIN:  14%|█▍        | 19/136 [00:06<00:27,  4.21it/s, epoch=62, grad_norm=81.706, loss=282.0998, lr=6.78e-04]
Epoch 62 TRAIN:  14%|█▍        | 19/136 [00:06<00:27,  4.21it/s, epoch=62, grad_norm=74.608, loss=263.9323, lr=6.78e-04]
Epoch 62 TRAIN:  14%|█▍        | 19/136 [00:06<00:27,  4.21it/s, epoch=62, grad_norm=107.094, loss=314.3605, lr=6.78e-04]
Epoch 62 TRAIN:  14%|█▍        | 19/136 [00:07<00:27,  4.21it/s, epoch=62, grad_norm=64.525, loss=280.3378, lr=6.78e-04] 
Epoch 62 TRAIN:  17%|█▋        | 23/136 [00:07<00:24,  4.65it/s, epoch=62, grad_norm=64.525, loss=280.3378, lr=6.78e-04]
Epoch 62 TRAIN:  17%|█▋        | 23/136 [00:07<00:24,  4.65it/s, epoch=62, grad_norm=117.799, loss=323.0879, lr=6.78e-04]
Epoch 62 TRAIN:  17%|█▋        | 23/136 [00:07<00:24,  4.65it/s, epoch=62, grad_norm=99.861, loss=308.3801, lr=6.79e-04] 
Epoch 62 TRAIN:  17%|█▋        | 23/136 [00:07<00:24,  4.65it/s, epoch=62, grad_norm=104.717, loss=281.4619, lr=6.79e-04]
Epoch 62 TRAIN:  17%|█▋        | 23/136 [00:07<00:24,  4.65it/s, epoch=62, grad_norm=134.811, loss=248.1747, lr=6.79e-04]
Epoch 62 TRAIN:  20%|█▉        | 27/136 [00:07<00:23,  4.71it/s, epoch=62, grad_norm=134.811, loss=248.1747, lr=6.79e-04]
Epoch 62 TRAIN:  20%|█▉        | 27/136 [00:08<00:23,  4.71it/s, epoch=62, grad_norm=98.518, loss=296.6020, lr=6.79e-04] 
Epoch 62 TRAIN:  20%|█▉        | 27/136 [00:08<00:23,  4.71it/s, epoch=62, grad_norm=108.244, loss=280.6447, lr=6.79e-04]
Epoch 62 TRAIN:  20%|█▉        | 27/136 [00:08<00:23,  4.71it/s, epoch=62, grad_norm=174.698, loss=305.2927, lr=6.79e-04]
Epoch 62 TRAIN:  22%|██▏       | 30/136 [00:08<00:22,  4.79it/s, epoch=62, grad_norm=174.698, loss=305.2927, lr=6.79e-04]
Epoch 62 TRAIN:  22%|██▏       | 30/136 [00:08<00:22,  4.79it/s, epoch=62, grad_norm=107.694, loss=291.4503, lr=6.79e-04]
Epoch 62 TRAIN:  22%|██▏       | 30/136 [00:08<00:22,  4.79it/s, epoch=62, grad_norm=96.818, loss=290.5740, lr=6.79e-04] 
Epoch 62 TRAIN:  22%|██▏       | 30/136 [00:09<00:22,  4.79it/s, epoch=62, grad_norm=91.302, loss=295.3194, lr=6.79e-04]
Epoch 62 TRAIN:  24%|██▍       | 33/136 [00:09<00:20,  4.99it/s, epoch=62, grad_norm=91.302, loss=295.3194, lr=6.79e-04]
Epoch 62 TRAIN:  24%|██▍       | 33/136 [00:09<00:20,  4.99it/s, epoch=62, grad_norm=147.363, loss=306.0533, lr=6.79e-04]
Epoch 62 TRAIN:  24%|██▍       | 33/136 [00:09<00:20,  4.99it/s, epoch=62, grad_norm=116.237, loss=289.5289, lr=6.79e-04]
Epoch 62 TRAIN:  24%|██▍       | 33/136 [00:09<00:20,  4.99it/s, epoch=62, grad_norm=93.789, loss=269.3450, lr=6.79e-04] 
Epoch 62 TRAIN:  26%|██▋       | 36/136 [00:09<00:20,  4.78it/s, epoch=62, grad_norm=93.789, loss=269.3450, lr=6.79e-04]
Epoch 62 TRAIN:  26%|██▋       | 36/136 [00:09<00:20,  4.78it/s, epoch=62, grad_norm=106.274, loss=297.5845, lr=6.80e-04]
Epoch 62 TRAIN:  26%|██▋       | 36/136 [00:10<00:20,  4.78it/s, epoch=62, grad_norm=107.548, loss=292.7278, lr=6.80e-04]
Epoch 62 TRAIN:  26%|██▋       | 36/136 [00:10<00:20,  4.78it/s, epoch=62, grad_norm=151.275, loss=318.7881, lr=6.80e-04]
Epoch 62 TRAIN:  29%|██▊       | 39/136 [00:10<00:20,  4.68it/s, epoch=62, grad_norm=151.275, loss=318.7881, lr=6.80e-04]
Epoch 62 TRAIN:  29%|██▊       | 39/136 [00:10<00:20,  4.68it/s, epoch=62, grad_norm=144.745, loss=317.8491, lr=6.80e-04]
Epoch 62 TRAIN:  29%|██▊       | 39/136 [00:10<00:20,  4.68it/s, epoch=62, grad_norm=175.250, loss=267.5859, lr=6.80e-04]
Epoch 62 TRAIN:  29%|██▊       | 39/136 [00:10<00:20,  4.68it/s, epoch=62, grad_norm=122.204, loss=299.5715, lr=6.80e-04]
Epoch 62 TRAIN:  31%|███       | 42/136 [00:10<00:19,  4.95it/s, epoch=62, grad_norm=122.204, loss=299.5715, lr=6.80e-04]
Epoch 62 TRAIN:  31%|███       | 42/136 [00:11<00:19,  4.95it/s, epoch=62, grad_norm=122.068, loss=311.9718, lr=6.80e-04]
Epoch 62 TRAIN:  31%|███       | 42/136 [00:11<00:19,  4.95it/s, epoch=62, grad_norm=147.846, loss=265.3875, lr=6.80e-04]
Epoch 62 TRAIN:  31%|███       | 42/136 [00:11<00:19,  4.95it/s, epoch=62, grad_norm=116.733, loss=289.3829, lr=6.80e-04]
Epoch 62 TRAIN:  33%|███▎      | 45/136 [00:11<00:19,  4.78it/s, epoch=62, grad_norm=116.733, loss=289.3829, lr=6.80e-04]
Epoch 62 TRAIN:  33%|███▎      | 45/136 [00:11<00:19,  4.78it/s, epoch=62, grad_norm=105.051, loss=332.4476, lr=6.80e-04]
Epoch 62 TRAIN:  33%|███▎      | 45/136 [00:11<00:19,  4.78it/s, epoch=62, grad_norm=90.603, loss=264.0326, lr=6.80e-04] 
Epoch 62 TRAIN:  33%|███▎      | 45/136 [00:12<00:19,  4.78it/s, epoch=62, grad_norm=135.444, loss=311.1097, lr=6.80e-04]
Epoch 62 TRAIN:  35%|███▌      | 48/136 [00:12<00:17,  4.92it/s, epoch=62, grad_norm=135.444, loss=311.1097, lr=6.80e-04]
Epoch 62 TRAIN:  35%|███▌      | 48/136 [00:12<00:17,  4.92it/s, epoch=62, grad_norm=81.545, loss=274.1720, lr=6.80e-04] 
Epoch 62 TRAIN:  35%|███▌      | 48/136 [00:12<00:17,  4.92it/s, epoch=62, grad_norm=114.923, loss=286.9725, lr=6.81e-04]
Epoch 62 TRAIN:  35%|███▌      | 48/136 [00:12<00:17,  4.92it/s, epoch=62, grad_norm=100.486, loss=281.5925, lr=6.81e-04]
Epoch 62 TRAIN:  38%|███▊      | 51/136 [00:12<00:17,  4.76it/s, epoch=62, grad_norm=100.486, loss=281.5925, lr=6.81e-04]
Epoch 62 TRAIN:  38%|███▊      | 51/136 [00:13<00:17,  4.76it/s, epoch=62, grad_norm=99.547, loss=291.2434, lr=6.81e-04] 
Epoch 62 TRAIN:  38%|███▊      | 51/136 [00:13<00:17,  4.76it/s, epoch=62, grad_norm=78.643, loss=265.7381, lr=6.81e-04]
Epoch 62 TRAIN:  38%|███▊      | 51/136 [00:13<00:17,  4.76it/s, epoch=62, grad_norm=153.973, loss=304.0218, lr=6.81e-04]
Epoch 62 TRAIN:  40%|███▉      | 54/136 [00:13<00:16,  4.94it/s, epoch=62, grad_norm=153.973, loss=304.0218, lr=6.81e-04]
Epoch 62 TRAIN:  40%|███▉      | 54/136 [00:13<00:16,  4.94it/s, epoch=62, grad_norm=137.189, loss=292.8363, lr=6.81e-04]
Epoch 62 TRAIN:  40%|███▉      | 54/136 [00:13<00:16,  4.94it/s, epoch=62, grad_norm=149.552, loss=320.2139, lr=6.81e-04]
Epoch 62 TRAIN:  40%|███▉      | 54/136 [00:14<00:16,  4.94it/s, epoch=62, grad_norm=120.522, loss=293.2228, lr=6.81e-04]
Epoch 62 TRAIN:  42%|████▏     | 57/136 [00:14<00:16,  4.77it/s, epoch=62, grad_norm=120.522, loss=293.2228, lr=6.81e-04]
Epoch 62 TRAIN:  42%|████▏     | 57/136 [00:14<00:16,  4.77it/s, epoch=62, grad_norm=138.836, loss=321.1585, lr=6.81e-04]
Epoch 62 TRAIN:  42%|████▏     | 57/136 [00:14<00:16,  4.77it/s, epoch=62, grad_norm=196.770, loss=273.5723, lr=6.81e-04]
Epoch 62 TRAIN:  42%|████▏     | 57/136 [00:14<00:16,  4.77it/s, epoch=62, grad_norm=166.194, loss=275.4354, lr=6.81e-04]
Epoch 62 TRAIN:  44%|████▍     | 60/136 [00:14<00:15,  4.75it/s, epoch=62, grad_norm=166.194, loss=275.4354, lr=6.81e-04]
Epoch 62 TRAIN:  44%|████▍     | 60/136 [00:14<00:15,  4.75it/s, epoch=62, grad_norm=110.938, loss=284.7071, lr=6.81e-04]
Epoch 62 TRAIN:  44%|████▍     | 60/136 [00:15<00:15,  4.75it/s, epoch=62, grad_norm=163.466, loss=241.7795, lr=6.82e-04]
Epoch 62 TRAIN:  44%|████▍     | 60/136 [00:15<00:15,  4.75it/s, epoch=62, grad_norm=195.190, loss=299.0954, lr=6.82e-04]
Epoch 62 TRAIN:  46%|████▋     | 63/136 [00:15<00:15,  4.81it/s, epoch=62, grad_norm=195.190, loss=299.0954, lr=6.82e-04]
Epoch 62 TRAIN:  46%|████▋     | 63/136 [00:15<00:15,  4.81it/s, epoch=62, grad_norm=104.029, loss=277.4408, lr=6.82e-04]
Epoch 62 TRAIN:  46%|████▋     | 63/136 [00:15<00:15,  4.81it/s, epoch=62, grad_norm=112.883, loss=300.5768, lr=6.82e-04]
Epoch 62 TRAIN:  46%|████▋     | 63/136 [00:15<00:15,  4.81it/s, epoch=62, grad_norm=98.584, loss=283.2040, lr=6.82e-04] 
Epoch 62 TRAIN:  49%|████▊     | 66/136 [00:15<00:14,  4.79it/s, epoch=62, grad_norm=98.584, loss=283.2040, lr=6.82e-04]
Epoch 62 TRAIN:  49%|████▊     | 66/136 [00:16<00:14,  4.79it/s, epoch=62, grad_norm=96.353, loss=275.9498, lr=6.82e-04]
Epoch 62 TRAIN:  49%|████▊     | 66/136 [00:16<00:14,  4.79it/s, epoch=62, grad_norm=135.985, loss=316.5069, lr=6.82e-04]
Epoch 62 TRAIN:  49%|████▊     | 66/136 [00:16<00:14,  4.79it/s, epoch=62, grad_norm=114.832, loss=248.8531, lr=6.82e-04]
Epoch 62 TRAIN:  51%|█████     | 69/136 [00:16<00:14,  4.73it/s, epoch=62, grad_norm=114.832, loss=248.8531, lr=6.82e-04]
Epoch 62 TRAIN:  51%|█████     | 69/136 [00:16<00:14,  4.73it/s, epoch=62, grad_norm=96.770, loss=257.7378, lr=6.82e-04] 
Epoch 62 TRAIN:  51%|█████     | 69/136 [00:17<00:14,  4.73it/s, epoch=62, grad_norm=142.686, loss=303.4010, lr=6.82e-04]
Epoch 62 TRAIN:  51%|█████     | 69/136 [00:17<00:14,  4.73it/s, epoch=62, grad_norm=142.556, loss=280.7386, lr=6.82e-04]
Epoch 62 TRAIN:  53%|█████▎    | 72/136 [00:17<00:13,  4.76it/s, epoch=62, grad_norm=142.556, loss=280.7386, lr=6.82e-04]
Epoch 62 TRAIN:  53%|█████▎    | 72/136 [00:17<00:13,  4.76it/s, epoch=62, grad_norm=116.737, loss=278.6321, lr=6.82e-04]
Epoch 62 TRAIN:  53%|█████▎    | 72/136 [00:17<00:13,  4.76it/s, epoch=62, grad_norm=97.704, loss=245.0007, lr=6.82e-04] 
Epoch 62 TRAIN:  53%|█████▎    | 72/136 [00:17<00:13,  4.76it/s, epoch=62, grad_norm=144.539, loss=253.9090, lr=6.83e-04]
Epoch 62 TRAIN:  55%|█████▌    | 75/136 [00:17<00:12,  4.97it/s, epoch=62, grad_norm=144.539, loss=253.9090, lr=6.83e-04]
Epoch 62 TRAIN:  55%|█████▌    | 75/136 [00:17<00:12,  4.97it/s, epoch=62, grad_norm=111.686, loss=282.5517, lr=6.83e-04]
Epoch 62 TRAIN:  55%|█████▌    | 75/136 [00:18<00:12,  4.97it/s, epoch=62, grad_norm=163.411, loss=288.8096, lr=6.83e-04]
Epoch 62 TRAIN:  55%|█████▌    | 75/136 [00:18<00:12,  4.97it/s, epoch=62, grad_norm=197.009, loss=211.2526, lr=6.83e-04]
Epoch 62 TRAIN:  57%|█████▋    | 78/136 [00:18<00:12,  4.62it/s, epoch=62, grad_norm=197.009, loss=211.2526, lr=6.83e-04]
Epoch 62 TRAIN:  57%|█████▋    | 78/136 [00:18<00:12,  4.62it/s, epoch=62, grad_norm=179.814, loss=334.6352, lr=6.83e-04]
Epoch 62 TRAIN:  57%|█████▋    | 78/136 [00:18<00:12,  4.62it/s, epoch=62, grad_norm=109.318, loss=273.7159, lr=6.83e-04]
Epoch 62 TRAIN:  57%|█████▋    | 78/136 [00:19<00:12,  4.62it/s, epoch=62, grad_norm=114.349, loss=300.7966, lr=6.83e-04]
Epoch 62 TRAIN:  60%|█████▉    | 81/136 [00:19<00:12,  4.43it/s, epoch=62, grad_norm=114.349, loss=300.7966, lr=6.83e-04]
Epoch 62 TRAIN:  60%|█████▉    | 81/136 [00:19<00:12,  4.43it/s, epoch=62, grad_norm=147.476, loss=311.9288, lr=6.83e-04]
Epoch 62 TRAIN:  60%|█████▉    | 81/136 [00:19<00:12,  4.43it/s, epoch=62, grad_norm=127.891, loss=274.0800, lr=6.83e-04]
Epoch 62 TRAIN:  60%|█████▉    | 81/136 [00:19<00:12,  4.43it/s, epoch=62, grad_norm=213.388, loss=243.3500, lr=6.83e-04]
Epoch 62 TRAIN:  62%|██████▏   | 84/136 [00:19<00:11,  4.53it/s, epoch=62, grad_norm=213.388, loss=243.3500, lr=6.83e-04]
Epoch 62 TRAIN:  62%|██████▏   | 84/136 [00:20<00:11,  4.53it/s, epoch=62, grad_norm=137.596, loss=290.7951, lr=6.83e-04]
Epoch 62 TRAIN:  62%|██████▏   | 84/136 [00:20<00:11,  4.53it/s, epoch=62, grad_norm=224.847, loss=332.8779, lr=6.83e-04]
Epoch 62 TRAIN:  62%|██████▏   | 84/136 [00:20<00:11,  4.53it/s, epoch=62, grad_norm=100.129, loss=267.6498, lr=6.84e-04]
Epoch 62 TRAIN:  64%|██████▍   | 87/136 [00:20<00:10,  4.79it/s, epoch=62, grad_norm=100.129, loss=267.6498, lr=6.84e-04]
Epoch 62 TRAIN:  64%|██████▍   | 87/136 [00:20<00:10,  4.79it/s, epoch=62, grad_norm=162.981, loss=276.0704, lr=6.84e-04]
Epoch 62 TRAIN:  64%|██████▍   | 87/136 [00:20<00:10,  4.79it/s, epoch=62, grad_norm=135.404, loss=248.4368, lr=6.84e-04]
Epoch 62 TRAIN:  64%|██████▍   | 87/136 [00:21<00:10,  4.79it/s, epoch=62, grad_norm=154.676, loss=312.2452, lr=6.84e-04]
Epoch 62 TRAIN:  66%|██████▌   | 90/136 [00:21<00:10,  4.50it/s, epoch=62, grad_norm=154.676, loss=312.2452, lr=6.84e-04]
Epoch 62 TRAIN:  66%|██████▌   | 90/136 [00:21<00:10,  4.50it/s, epoch=62, grad_norm=244.555, loss=211.3559, lr=6.84e-04]
Epoch 62 TRAIN:  66%|██████▌   | 90/136 [00:21<00:10,  4.50it/s, epoch=62, grad_norm=242.989, loss=234.4066, lr=6.84e-04]
Epoch 62 TRAIN:  66%|██████▌   | 90/136 [00:21<00:10,  4.50it/s, epoch=62, grad_norm=147.670, loss=207.5342, lr=6.84e-04]
Epoch 62 TRAIN:  68%|██████▊   | 93/136 [00:21<00:09,  4.52it/s, epoch=62, grad_norm=147.670, loss=207.5342, lr=6.84e-04]
Epoch 62 TRAIN:  68%|██████▊   | 93/136 [00:22<00:09,  4.52it/s, epoch=62, grad_norm=144.096, loss=291.0165, lr=6.84e-04]
Epoch 62 TRAIN:  68%|██████▊   | 93/136 [00:22<00:09,  4.52it/s, epoch=62, grad_norm=112.220, loss=236.2819, lr=6.84e-04]
Epoch 62 TRAIN:  68%|██████▊   | 93/136 [00:22<00:09,  4.52it/s, epoch=62, grad_norm=278.842, loss=339.3260, lr=6.84e-04]
Epoch 62 TRAIN:  71%|███████   | 96/136 [00:22<00:08,  4.64it/s, epoch=62, grad_norm=278.842, loss=339.3260, lr=6.84e-04]
Epoch 62 TRAIN:  71%|███████   | 96/136 [00:22<00:08,  4.64it/s, epoch=62, grad_norm=176.357, loss=295.4335, lr=6.84e-04]
Epoch 62 TRAIN:  71%|███████   | 96/136 [00:22<00:08,  4.64it/s, epoch=62, grad_norm=255.795, loss=317.0269, lr=6.84e-04]
Epoch 62 TRAIN:  71%|███████   | 96/136 [00:23<00:08,  4.64it/s, epoch=62, grad_norm=171.840, loss=206.8694, lr=6.84e-04]
Epoch 62 TRAIN:  73%|███████▎  | 99/136 [00:23<00:08,  4.59it/s, epoch=62, grad_norm=171.840, loss=206.8694, lr=6.84e-04]
Epoch 62 TRAIN:  73%|███████▎  | 99/136 [00:23<00:08,  4.59it/s, epoch=62, grad_norm=112.936, loss=284.4743, lr=6.85e-04]2026-04-16 20:23:54,162 DEBUG TRAIN | steps/sec 4.282| Batch 62/100 loss 284.474335 loss_att 241.052063 loss_ctc 385.792969 th_accuracy 0.324324 lr 6.8456e-04 grad_norm 112.935799 rank 0

Epoch 62 TRAIN:  73%|███████▎  | 99/136 [00:23<00:08,  4.59it/s, epoch=62, grad_norm=157.292, loss=215.9668, lr=6.85e-04]
Epoch 62 TRAIN:  73%|███████▎  | 99/136 [00:23<00:08,  4.59it/s, epoch=62, grad_norm=148.396, loss=281.5984, lr=6.85e-04]
Epoch 62 TRAIN:  75%|███████▌  | 102/136 [00:23<00:07,  4.62it/s, epoch=62, grad_norm=148.396, loss=281.5984, lr=6.85e-04]
Epoch 62 TRAIN:  75%|███████▌  | 102/136 [00:23<00:07,  4.62it/s, epoch=62, grad_norm=110.288, loss=220.1156, lr=6.85e-04]
Epoch 62 TRAIN:  75%|███████▌  | 102/136 [00:24<00:07,  4.62it/s, epoch=62, grad_norm=167.388, loss=245.3001, lr=6.85e-04]
Epoch 62 TRAIN:  75%|███████▌  | 102/136 [00:24<00:07,  4.62it/s, epoch=62, grad_norm=185.870, loss=285.8665, lr=6.85e-04]
Epoch 62 TRAIN:  77%|███████▋  | 105/136 [00:24<00:06,  4.67it/s, epoch=62, grad_norm=185.870, loss=285.8665, lr=6.85e-04]
Epoch 62 TRAIN:  77%|███████▋  | 105/136 [00:24<00:06,  4.67it/s, epoch=62, grad_norm=227.953, loss=349.0990, lr=6.85e-04]
Epoch 62 TRAIN:  77%|███████▋  | 105/136 [00:24<00:06,  4.67it/s, epoch=62, grad_norm=149.108, loss=333.3922, lr=6.85e-04]
Epoch 62 TRAIN:  77%|███████▋  | 105/136 [00:25<00:06,  4.67it/s, epoch=62, grad_norm=231.192, loss=267.1889, lr=6.85e-04]
Epoch 62 TRAIN:  79%|███████▉  | 108/136 [00:25<00:05,  4.73it/s, epoch=62, grad_norm=231.192, loss=267.1889, lr=6.85e-04]
Epoch 62 TRAIN:  79%|███████▉  | 108/136 [00:25<00:05,  4.73it/s, epoch=62, grad_norm=241.597, loss=230.6919, lr=6.85e-04]
Epoch 62 TRAIN:  79%|███████▉  | 108/136 [00:25<00:05,  4.73it/s, epoch=62, grad_norm=196.036, loss=256.4954, lr=6.85e-04]
Epoch 62 TRAIN:  79%|███████▉  | 108/136 [00:25<00:05,  4.73it/s, epoch=62, grad_norm=144.118, loss=265.9135, lr=6.85e-04]
Epoch 62 TRAIN:  82%|████████▏ | 111/136 [00:25<00:05,  4.40it/s, epoch=62, grad_norm=144.118, loss=265.9135, lr=6.85e-04]
Epoch 62 TRAIN:  82%|████████▏ | 111/136 [00:26<00:05,  4.40it/s, epoch=62, grad_norm=186.919, loss=266.2351, lr=6.86e-04]
Epoch 62 TRAIN:  82%|████████▏ | 111/136 [00:26<00:05,  4.40it/s, epoch=62, grad_norm=227.166, loss=344.9162, lr=6.86e-04]
Epoch 62 TRAIN:  82%|████████▏ | 111/136 [00:26<00:05,  4.40it/s, epoch=62, grad_norm=150.962, loss=263.3489, lr=6.86e-04]
Epoch 62 TRAIN:  84%|████████▍ | 114/136 [00:26<00:04,  4.47it/s, epoch=62, grad_norm=150.962, loss=263.3489, lr=6.86e-04]
Epoch 62 TRAIN:  84%|████████▍ | 114/136 [00:26<00:04,  4.47it/s, epoch=62, grad_norm=167.889, loss=299.3930, lr=6.86e-04]
Epoch 62 TRAIN:  84%|████████▍ | 114/136 [00:26<00:04,  4.47it/s, epoch=62, grad_norm=130.902, loss=297.5440, lr=6.86e-04]
Epoch 62 TRAIN:  84%|████████▍ | 114/136 [00:27<00:04,  4.47it/s, epoch=62, grad_norm=134.486, loss=251.5821, lr=6.86e-04]
Epoch 62 TRAIN:  86%|████████▌ | 117/136 [00:27<00:04,  4.58it/s, epoch=62, grad_norm=134.486, loss=251.5821, lr=6.86e-04]
Epoch 62 TRAIN:  86%|████████▌ | 117/136 [00:27<00:04,  4.58it/s, epoch=62, grad_norm=139.463, loss=251.6899, lr=6.86e-04]
Epoch 62 TRAIN:  86%|████████▌ | 117/136 [00:27<00:04,  4.58it/s, epoch=62, grad_norm=202.584, loss=299.2466, lr=6.86e-04]
Epoch 62 TRAIN:  86%|████████▌ | 117/136 [00:27<00:04,  4.58it/s, epoch=62, grad_norm=252.463, loss=284.5757, lr=6.86e-04]
Epoch 62 TRAIN:  88%|████████▊ | 120/136 [00:27<00:03,  4.37it/s, epoch=62, grad_norm=252.463, loss=284.5757, lr=6.86e-04]
Epoch 62 TRAIN:  88%|████████▊ | 120/136 [00:28<00:03,  4.37it/s, epoch=62, grad_norm=118.867, loss=320.9858, lr=6.86e-04]
Epoch 62 TRAIN:  88%|████████▊ | 120/136 [00:28<00:03,  4.37it/s, epoch=62, grad_norm=108.687, loss=256.0761, lr=6.86e-04]
Epoch 62 TRAIN:  88%|████████▊ | 120/136 [00:28<00:03,  4.37it/s, epoch=62, grad_norm=184.923, loss=273.9338, lr=6.86e-04]
Epoch 62 TRAIN:  90%|█████████ | 123/136 [00:28<00:02,  4.53it/s, epoch=62, grad_norm=184.923, loss=273.9338, lr=6.86e-04]
Epoch 62 TRAIN:  90%|█████████ | 123/136 [00:28<00:02,  4.53it/s, epoch=62, grad_norm=253.821, loss=265.9117, lr=6.86e-04]
Epoch 62 TRAIN:  90%|█████████ | 123/136 [00:28<00:02,  4.53it/s, epoch=62, grad_norm=285.722, loss=379.2697, lr=6.87e-04]
Epoch 62 TRAIN:  90%|█████████ | 123/136 [00:28<00:02,  4.53it/s, epoch=62, grad_norm=142.158, loss=322.1256, lr=6.87e-04]
Epoch 62 TRAIN:  90%|█████████ | 123/136 [00:29<00:02,  4.53it/s, epoch=62, grad_norm=185.113, loss=311.0516, lr=6.87e-04]
Epoch 62 TRAIN:  93%|█████████▎| 127/136 [00:29<00:01,  4.77it/s, epoch=62, grad_norm=185.113, loss=311.0516, lr=6.87e-04]
Epoch 62 TRAIN:  93%|█████████▎| 127/136 [00:29<00:01,  4.77it/s, epoch=62, grad_norm=224.862, loss=329.6561, lr=6.87e-04]
Epoch 62 TRAIN:  93%|█████████▎| 127/136 [00:29<00:01,  4.77it/s, epoch=62, grad_norm=165.298, loss=280.1258, lr=6.87e-04]
Epoch 62 TRAIN:  93%|█████████▎| 127/136 [00:29<00:01,  4.77it/s, epoch=62, grad_norm=200.452, loss=360.2571, lr=6.87e-04]
Epoch 62 TRAIN:  96%|█████████▌| 130/136 [00:29<00:01,  4.88it/s, epoch=62, grad_norm=200.452, loss=360.2571, lr=6.87e-04]
Epoch 62 TRAIN:  96%|█████████▌| 130/136 [00:29<00:01,  4.88it/s, epoch=62, grad_norm=344.200, loss=339.8083, lr=6.87e-04]
Epoch 62 TRAIN:  96%|█████████▌| 130/136 [00:30<00:01,  4.88it/s, epoch=62, grad_norm=192.730, loss=318.7599, lr=6.87e-04]
Epoch 62 TRAIN:  96%|█████████▌| 130/136 [00:30<00:01,  4.88it/s, epoch=62, grad_norm=258.516, loss=380.0332, lr=6.87e-04]
Epoch 62 TRAIN:  96%|█████████▌| 130/136 [00:30<00:01,  4.88it/s, epoch=62, grad_norm=321.009, loss=279.8394, lr=6.87e-04]
Epoch 62 TRAIN:  99%|█████████▊| 134/136 [00:30<00:00,  5.29it/s, epoch=62, grad_norm=321.009, loss=279.8394, lr=6.87e-04]
Epoch 62 TRAIN:  99%|█████████▊| 134/136 [00:30<00:00,  5.29it/s, epoch=62, grad_norm=236.910, loss=357.9689, lr=6.87e-04]
Epoch 62 TRAIN:  99%|█████████▉| 135/136 [00:30<00:00,  4.40it/s, epoch=62, grad_norm=236.910, loss=357.9689, lr=6.87e-04]

CV:   0%|          | 0/6 [00:00<?, ?it/s]/home/rookie/Documents/batuka/projects/wenet/wenet/wenet/utils/train_utils.py:689: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  amp_autocast(enabled=dtype is not None,

CV:  17%|█▋        | 1/6 [00:00<00:03,  1.33it/s]
                                                 
2026-04-16 20:24:02,257 INFO Epoch 62 Step 8592 CV info lr 6.8736e-04 cv_loss 450.1849087801847 rank 0 acc 0.36836276948451996
2026-04-16 20:24:02,259 INFO [Rank 0] Checkpoint: save to checkpoint exp/conformer_mongolian/epoch_62.pt
2026-04-16 20:24:02,369 INFO Epoch 63 Step 8592 TRAIN info lr 6.8736e-04 rank 0
2026-04-16 20:24:02,390 INFO using accumulate grad, new batch size is 1 times larger than before

Epoch 63 TRAIN:   0%|          | 0/135 [00:00<?, ?it/s]
Epoch 63 TRAIN:   0%|          | 0/135 [00:03<?, ?it/s, epoch=63, grad_norm=480.113, loss=364.2546, lr=6.87e-04]
Epoch 63 TRAIN:   1%|          | 1/135 [00:03<08:42,  3.90s/it, epoch=63, grad_norm=480.113, loss=364.2546, lr=6.87e-04]
Epoch 63 TRAIN:   1%|          | 1/135 [00:03<08:42,  3.90s/it, epoch=63, grad_norm=582.788, loss=456.7957, lr=6.88e-04]
Epoch 63 TRAIN:   1%|          | 1/135 [00:04<08:42,  3.90s/it, epoch=63, grad_norm=309.495, loss=255.7045, lr=6.88e-04]
Epoch 63 TRAIN:   1%|          | 1/135 [00:04<08:42,  3.90s/it, epoch=63, grad_norm=250.530, loss=243.6705, lr=6.88e-04]
Epoch 63 TRAIN:   1%|          | 1/135 [00:04<08:42,  3.90s/it, epoch=63, grad_norm=249.388, loss=261.4614, lr=6.88e-04]
Epoch 63 TRAIN:   1%|          | 1/135 [00:04<08:42,  3.90s/it, epoch=63, grad_norm=443.696, loss=346.9816, lr=6.88e-04]
Epoch 63 TRAIN:   4%|▍         | 6/135 [00:04<01:13,  1.76it/s, epoch=63, grad_norm=443.696, loss=346.9816, lr=6.88e-04]
Epoch 63 TRAIN:   4%|▍         | 6/135 [00:04<01:13,  1.76it/s, epoch=63, grad_norm=115.051, loss=204.1357, lr=6.88e-04]
Epoch 63 TRAIN:   4%|▍         | 6/135 [00:04<01:13,  1.76it/s, epoch=63, grad_norm=241.050, loss=283.4211, lr=6.88e-04]
Epoch 63 TRAIN:   4%|▍         | 6/135 [00:04<01:13,  1.76it/s, epoch=63, grad_norm=317.000, loss=373.7870, lr=6.88e-04]
Epoch 63 TRAIN:   4%|▍         | 6/135 [00:04<01:13,  1.76it/s, epoch=63, grad_norm=248.054, loss=350.4664, lr=6.88e-04]
Epoch 63 TRAIN:   4%|▍         | 6/135 [00:05<01:13,  1.76it/s, epoch=63, grad_norm=150.068, loss=324.3646, lr=6.88e-04]
Epoch 63 TRAIN:   8%|▊         | 11/135 [00:05<00:39,  3.14it/s, epoch=63, grad_norm=150.068, loss=324.3646, lr=6.88e-04]
Epoch 63 TRAIN:   8%|▊         | 11/135 [00:05<00:39,  3.14it/s, epoch=63, grad_norm=108.823, loss=302.5534, lr=6.88e-04]
Epoch 63 TRAIN:   8%|▊         | 11/135 [00:05<00:39,  3.14it/s, epoch=63, grad_norm=174.920, loss=256.8462, lr=6.88e-04]
Epoch 63 TRAIN:   8%|▊         | 11/135 [00:05<00:39,  3.14it/s, epoch=63, grad_norm=137.235, loss=292.5065, lr=6.88e-04]
Epoch 63 TRAIN:   8%|▊         | 11/135 [00:05<00:39,  3.14it/s, epoch=63, grad_norm=141.996, loss=289.3127, lr=6.89e-04]
Epoch 63 TRAIN:  11%|█         | 15/135 [00:05<00:30,  3.91it/s, epoch=63, grad_norm=141.996, loss=289.3127, lr=6.89e-04]
Epoch 63 TRAIN:  11%|█         | 15/135 [00:05<00:30,  3.91it/s, epoch=63, grad_norm=136.196, loss=310.6825, lr=6.89e-04]
Epoch 63 TRAIN:  11%|█         | 15/135 [00:06<00:30,  3.91it/s, epoch=63, grad_norm=229.199, loss=273.2951, lr=6.89e-04]
Epoch 63 TRAIN:  11%|█         | 15/135 [00:06<00:30,  3.91it/s, epoch=63, grad_norm=146.248, loss=306.8497, lr=6.89e-04]
Epoch 63 TRAIN:  11%|█         | 15/135 [00:06<00:30,  3.91it/s, epoch=63, grad_norm=144.113, loss=296.4760, lr=6.89e-04]
Epoch 63 TRAIN:  14%|█▍        | 19/135 [00:06<00:27,  4.19it/s, epoch=63, grad_norm=144.113, loss=296.4760, lr=6.89e-04]
Epoch 63 TRAIN:  14%|█▍        | 19/135 [00:06<00:27,  4.19it/s, epoch=63, grad_norm=125.990, loss=279.0193, lr=6.89e-04]
Epoch 63 TRAIN:  14%|█▍        | 19/135 [00:06<00:27,  4.19it/s, epoch=63, grad_norm=116.217, loss=312.3459, lr=6.89e-04]
Epoch 63 TRAIN:  14%|█▍        | 19/135 [00:06<00:27,  4.19it/s, epoch=63, grad_norm=88.284, loss=256.9552, lr=6.89e-04] 
Epoch 63 TRAIN:  14%|█▍        | 19/135 [00:07<00:27,  4.19it/s, epoch=63, grad_norm=138.343, loss=317.1803, lr=6.89e-04]
Epoch 63 TRAIN:  17%|█▋        | 23/135 [00:07<00:24,  4.59it/s, epoch=63, grad_norm=138.343, loss=317.1803, lr=6.89e-04]
Epoch 63 TRAIN:  17%|█▋        | 23/135 [00:07<00:24,  4.59it/s, epoch=63, grad_norm=93.998, loss=252.5686, lr=6.89e-04] 
Epoch 63 TRAIN:  17%|█▋        | 23/135 [00:07<00:24,  4.59it/s, epoch=63, grad_norm=74.632, loss=229.2296, lr=6.89e-04]
Epoch 63 TRAIN:  17%|█▋        | 23/135 [00:07<00:24,  4.59it/s, epoch=63, grad_norm=59.457, loss=235.7745, lr=6.89e-04]
Epoch 63 TRAIN:  19%|█▉        | 26/135 [00:07<00:23,  4.61it/s, epoch=63, grad_norm=59.457, loss=235.7745, lr=6.89e-04]
Epoch 63 TRAIN:  19%|█▉        | 26/135 [00:08<00:23,  4.61it/s, epoch=63, grad_norm=141.827, loss=296.2097, lr=6.90e-04]
Epoch 63 TRAIN:  19%|█▉        | 26/135 [00:08<00:23,  4.61it/s, epoch=63, grad_norm=107.155, loss=254.2706, lr=6.90e-04]
Epoch 63 TRAIN:  19%|█▉        | 26/135 [00:08<00:23,  4.61it/s, epoch=63, grad_norm=105.937, loss=290.1077, lr=6.90e-04]
Epoch 63 TRAIN:  21%|██▏       | 29/135 [00:08<00:22,  4.70it/s, epoch=63, grad_norm=105.937, loss=290.1077, lr=6.90e-04]
Epoch 63 TRAIN:  21%|██▏       | 29/135 [00:08<00:22,  4.70it/s, epoch=63, grad_norm=125.048, loss=291.8766, lr=6.90e-04]
Epoch 63 TRAIN:  21%|██▏       | 29/135 [00:08<00:22,  4.70it/s, epoch=63, grad_norm=131.728, loss=298.0346, lr=6.90e-04]
Epoch 63 TRAIN:  21%|██▏       | 29/135 [00:08<00:22,  4.70it/s, epoch=63, grad_norm=125.197, loss=319.3918, lr=6.90e-04]
Epoch 63 TRAIN:  24%|██▎       | 32/135 [00:08<00:21,  4.83it/s, epoch=63, grad_norm=125.197, loss=319.3918, lr=6.90e-04]
Epoch 63 TRAIN:  24%|██▎       | 32/135 [00:09<00:21,  4.83it/s, epoch=63, grad_norm=104.276, loss=269.5580, lr=6.90e-04]
Epoch 63 TRAIN:  24%|██▎       | 32/135 [00:09<00:21,  4.83it/s, epoch=63, grad_norm=135.744, loss=255.9396, lr=6.90e-04]
Epoch 63 TRAIN:  24%|██▎       | 32/135 [00:09<00:21,  4.83it/s, epoch=63, grad_norm=142.576, loss=346.5339, lr=6.90e-04]
Epoch 63 TRAIN:  26%|██▌       | 35/135 [00:09<00:20,  4.81it/s, epoch=63, grad_norm=142.576, loss=346.5339, lr=6.90e-04]
Epoch 63 TRAIN:  26%|██▌       | 35/135 [00:09<00:20,  4.81it/s, epoch=63, grad_norm=100.632, loss=276.6134, lr=6.90e-04]
Epoch 63 TRAIN:  26%|██▌       | 35/135 [00:10<00:20,  4.81it/s, epoch=63, grad_norm=155.587, loss=266.6785, lr=6.90e-04]
Epoch 63 TRAIN:  26%|██▌       | 35/135 [00:10<00:20,  4.81it/s, epoch=63, grad_norm=93.930, loss=325.6039, lr=6.90e-04] 
Epoch 63 TRAIN:  28%|██▊       | 38/135 [00:10<00:19,  4.88it/s, epoch=63, grad_norm=93.930, loss=325.6039, lr=6.90e-04]
Epoch 63 TRAIN:  28%|██▊       | 38/135 [00:10<00:19,  4.88it/s, epoch=63, grad_norm=121.401, loss=271.2174, lr=6.90e-04]
Epoch 63 TRAIN:  28%|██▊       | 38/135 [00:10<00:19,  4.88it/s, epoch=63, grad_norm=144.407, loss=262.4410, lr=6.91e-04]
Epoch 63 TRAIN:  28%|██▊       | 38/135 [00:10<00:19,  4.88it/s, epoch=63, grad_norm=132.844, loss=316.0267, lr=6.91e-04]
Epoch 63 TRAIN:  30%|███       | 41/135 [00:10<00:19,  4.79it/s, epoch=63, grad_norm=132.844, loss=316.0267, lr=6.91e-04]
Epoch 63 TRAIN:  30%|███       | 41/135 [00:11<00:19,  4.79it/s, epoch=63, grad_norm=115.793, loss=250.6057, lr=6.91e-04]
Epoch 63 TRAIN:  30%|███       | 41/135 [00:11<00:19,  4.79it/s, epoch=63, grad_norm=135.884, loss=316.0928, lr=6.91e-04]
Epoch 63 TRAIN:  30%|███       | 41/135 [00:11<00:19,  4.79it/s, epoch=63, grad_norm=86.063, loss=244.2905, lr=6.91e-04] 
Epoch 63 TRAIN:  33%|███▎      | 44/135 [00:11<00:19,  4.78it/s, epoch=63, grad_norm=86.063, loss=244.2905, lr=6.91e-04]
Epoch 63 TRAIN:  33%|███▎      | 44/135 [00:11<00:19,  4.78it/s, epoch=63, grad_norm=82.862, loss=255.2822, lr=6.91e-04]
Epoch 63 TRAIN:  33%|███▎      | 44/135 [00:11<00:19,  4.78it/s, epoch=63, grad_norm=101.209, loss=276.3458, lr=6.91e-04]
Epoch 63 TRAIN:  33%|███▎      | 44/135 [00:12<00:19,  4.78it/s, epoch=63, grad_norm=71.823, loss=255.5098, lr=6.91e-04] 
Epoch 63 TRAIN:  35%|███▍      | 47/135 [00:12<00:18,  4.82it/s, epoch=63, grad_norm=71.823, loss=255.5098, lr=6.91e-04]
Epoch 63 TRAIN:  35%|███▍      | 47/135 [00:12<00:18,  4.82it/s, epoch=63, grad_norm=108.722, loss=289.1021, lr=6.91e-04]
Epoch 63 TRAIN:  35%|███▍      | 47/135 [00:12<00:18,  4.82it/s, epoch=63, grad_norm=135.169, loss=321.5110, lr=6.91e-04]
Epoch 63 TRAIN:  35%|███▍      | 47/135 [00:12<00:18,  4.82it/s, epoch=63, grad_norm=182.682, loss=348.2707, lr=6.91e-04]
Epoch 63 TRAIN:  37%|███▋      | 50/135 [00:12<00:17,  4.89it/s, epoch=63, grad_norm=182.682, loss=348.2707, lr=6.91e-04]
Epoch 63 TRAIN:  37%|███▋      | 50/135 [00:12<00:17,  4.89it/s, epoch=63, grad_norm=179.193, loss=331.9556, lr=6.91e-04]
Epoch 63 TRAIN:  37%|███▋      | 50/135 [00:13<00:17,  4.89it/s, epoch=63, grad_norm=112.215, loss=245.9445, lr=6.92e-04]
Epoch 63 TRAIN:  37%|███▋      | 50/135 [00:13<00:17,  4.89it/s, epoch=63, grad_norm=145.741, loss=284.4247, lr=6.92e-04]
Epoch 63 TRAIN:  39%|███▉      | 53/135 [00:13<00:17,  4.80it/s, epoch=63, grad_norm=145.741, loss=284.4247, lr=6.92e-04]
Epoch 63 TRAIN:  39%|███▉      | 53/135 [00:13<00:17,  4.80it/s, epoch=63, grad_norm=136.168, loss=344.8549, lr=6.92e-04]
Epoch 63 TRAIN:  39%|███▉      | 53/135 [00:13<00:17,  4.80it/s, epoch=63, grad_norm=102.526, loss=282.0547, lr=6.92e-04]
Epoch 63 TRAIN:  39%|███▉      | 53/135 [00:13<00:17,  4.80it/s, epoch=63, grad_norm=181.141, loss=272.0831, lr=6.92e-04]
Epoch 63 TRAIN:  41%|████▏     | 56/135 [00:13<00:16,  4.86it/s, epoch=63, grad_norm=181.141, loss=272.0831, lr=6.92e-04]
Epoch 63 TRAIN:  41%|████▏     | 56/135 [00:14<00:16,  4.86it/s, epoch=63, grad_norm=106.197, loss=305.2083, lr=6.92e-04]
Epoch 63 TRAIN:  41%|████▏     | 56/135 [00:14<00:16,  4.86it/s, epoch=63, grad_norm=131.098, loss=293.4019, lr=6.92e-04]
Epoch 63 TRAIN:  41%|████▏     | 56/135 [00:14<00:16,  4.86it/s, epoch=63, grad_norm=108.651, loss=280.2438, lr=6.92e-04]
Epoch 63 TRAIN:  44%|████▎     | 59/135 [00:14<00:15,  4.84it/s, epoch=63, grad_norm=108.651, loss=280.2438, lr=6.92e-04]
Epoch 63 TRAIN:  44%|████▎     | 59/135 [00:14<00:15,  4.84it/s, epoch=63, grad_norm=138.789, loss=318.4759, lr=6.92e-04]
Epoch 63 TRAIN:  44%|████▎     | 59/135 [00:15<00:15,  4.84it/s, epoch=63, grad_norm=146.180, loss=271.1062, lr=6.92e-04]
Epoch 63 TRAIN:  44%|████▎     | 59/135 [00:15<00:15,  4.84it/s, epoch=63, grad_norm=138.182, loss=332.9504, lr=6.92e-04]
Epoch 63 TRAIN:  46%|████▌     | 62/135 [00:15<00:15,  4.74it/s, epoch=63, grad_norm=138.182, loss=332.9504, lr=6.92e-04]
Epoch 63 TRAIN:  46%|████▌     | 62/135 [00:15<00:15,  4.74it/s, epoch=63, grad_norm=102.872, loss=307.6160, lr=6.92e-04]
Epoch 63 TRAIN:  46%|████▌     | 62/135 [00:15<00:15,  4.74it/s, epoch=63, grad_norm=127.520, loss=304.6839, lr=6.92e-04]
Epoch 63 TRAIN:  46%|████▌     | 62/135 [00:16<00:15,  4.74it/s, epoch=63, grad_norm=142.526, loss=281.1407, lr=6.93e-04]
Epoch 63 TRAIN:  48%|████▊     | 65/135 [00:16<00:15,  4.40it/s, epoch=63, grad_norm=142.526, loss=281.1407, lr=6.93e-04]
Epoch 63 TRAIN:  48%|████▊     | 65/135 [00:16<00:15,  4.40it/s, epoch=63, grad_norm=150.517, loss=307.7225, lr=6.93e-04]
Epoch 63 TRAIN:  48%|████▊     | 65/135 [00:16<00:15,  4.40it/s, epoch=63, grad_norm=121.727, loss=285.4785, lr=6.93e-04]
Epoch 63 TRAIN:  48%|████▊     | 65/135 [00:16<00:15,  4.40it/s, epoch=63, grad_norm=135.946, loss=299.9205, lr=6.93e-04]
Epoch 63 TRAIN:  50%|█████     | 68/135 [00:16<00:14,  4.68it/s, epoch=63, grad_norm=135.946, loss=299.9205, lr=6.93e-04]
Epoch 63 TRAIN:  50%|█████     | 68/135 [00:16<00:14,  4.68it/s, epoch=63, grad_norm=234.427, loss=315.5112, lr=6.93e-04]
Epoch 63 TRAIN:  50%|█████     | 68/135 [00:17<00:14,  4.68it/s, epoch=63, grad_norm=172.706, loss=323.4451, lr=6.93e-04]
Epoch 63 TRAIN:  50%|█████     | 68/135 [00:17<00:14,  4.68it/s, epoch=63, grad_norm=128.322, loss=332.8283, lr=6.93e-04]
Epoch 63 TRAIN:  53%|█████▎    | 71/135 [00:17<00:14,  4.50it/s, epoch=63, grad_norm=128.322, loss=332.8283, lr=6.93e-04]
Epoch 63 TRAIN:  53%|█████▎    | 71/135 [00:17<00:14,  4.50it/s, epoch=63, grad_norm=197.364, loss=300.6808, lr=6.93e-04]
Epoch 63 TRAIN:  53%|█████▎    | 71/135 [00:17<00:14,  4.50it/s, epoch=63, grad_norm=161.742, loss=286.0279, lr=6.93e-04]
Epoch 63 TRAIN:  53%|█████▎    | 71/135 [00:17<00:14,  4.50it/s, epoch=63, grad_norm=216.763, loss=274.4051, lr=6.93e-04]
Epoch 63 TRAIN:  55%|█████▍    | 74/135 [00:17<00:13,  4.61it/s, epoch=63, grad_norm=216.763, loss=274.4051, lr=6.93e-04]
Epoch 63 TRAIN:  55%|█████▍    | 74/135 [00:18<00:13,  4.61it/s, epoch=63, grad_norm=130.781, loss=295.5399, lr=6.93e-04]
Epoch 63 TRAIN:  55%|█████▍    | 74/135 [00:18<00:13,  4.61it/s, epoch=63, grad_norm=80.719, loss=282.0081, lr=6.93e-04] 
Epoch 63 TRAIN:  55%|█████▍    | 74/135 [00:18<00:13,  4.61it/s, epoch=63, grad_norm=316.613, loss=228.5521, lr=6.94e-04]
Epoch 63 TRAIN:  57%|█████▋    | 77/135 [00:18<00:13,  4.28it/s, epoch=63, grad_norm=316.613, loss=228.5521, lr=6.94e-04]
Epoch 63 TRAIN:  57%|█████▋    | 77/135 [00:18<00:13,  4.28it/s, epoch=63, grad_norm=122.219, loss=295.1730, lr=6.94e-04]
Epoch 63 TRAIN:  57%|█████▋    | 77/135 [00:19<00:13,  4.28it/s, epoch=63, grad_norm=143.378, loss=252.4936, lr=6.94e-04]
Epoch 63 TRAIN:  57%|█████▋    | 77/135 [00:19<00:13,  4.28it/s, epoch=63, grad_norm=173.676, loss=300.7313, lr=6.94e-04]
Epoch 63 TRAIN:  59%|█████▉    | 80/135 [00:19<00:11,  4.67it/s, epoch=63, grad_norm=173.676, loss=300.7313, lr=6.94e-04]
Epoch 63 TRAIN:  59%|█████▉    | 80/135 [00:19<00:11,  4.67it/s, epoch=63, grad_norm=119.415, loss=276.7913, lr=6.94e-04]
Epoch 63 TRAIN:  59%|█████▉    | 80/135 [00:19<00:11,  4.67it/s, epoch=63, grad_norm=163.807, loss=355.3278, lr=6.94e-04]
Epoch 63 TRAIN:  59%|█████▉    | 80/135 [00:20<00:11,  4.67it/s, epoch=63, grad_norm=122.549, loss=217.3409, lr=6.94e-04]
Epoch 63 TRAIN:  61%|██████▏   | 83/135 [00:20<00:12,  4.20it/s, epoch=63, grad_norm=122.549, loss=217.3409, lr=6.94e-04]
Epoch 63 TRAIN:  61%|██████▏   | 83/135 [00:20<00:12,  4.20it/s, epoch=63, grad_norm=83.966, loss=232.5605, lr=6.94e-04] 
Epoch 63 TRAIN:  61%|██████▏   | 83/135 [00:20<00:12,  4.20it/s, epoch=63, grad_norm=158.412, loss=275.6552, lr=6.94e-04]
Epoch 63 TRAIN:  61%|██████▏   | 83/135 [00:20<00:12,  4.20it/s, epoch=63, grad_norm=175.528, loss=298.4704, lr=6.94e-04]
Epoch 63 TRAIN:  64%|██████▎   | 86/135 [00:20<00:10,  4.46it/s, epoch=63, grad_norm=175.528, loss=298.4704, lr=6.94e-04]
Epoch 63 TRAIN:  64%|██████▎   | 86/135 [00:20<00:10,  4.46it/s, epoch=63, grad_norm=158.926, loss=287.3384, lr=6.94e-04]
Epoch 63 TRAIN:  64%|██████▎   | 86/135 [00:21<00:10,  4.46it/s, epoch=63, grad_norm=116.072, loss=284.4374, lr=6.94e-04]
Epoch 63 TRAIN:  64%|██████▎   | 86/135 [00:21<00:10,  4.46it/s, epoch=63, grad_norm=160.019, loss=264.4077, lr=6.94e-04]
Epoch 63 TRAIN:  66%|██████▌   | 89/135 [00:21<00:10,  4.42it/s, epoch=63, grad_norm=160.019, loss=264.4077, lr=6.94e-04]
Epoch 63 TRAIN:  66%|██████▌   | 89/135 [00:21<00:10,  4.42it/s, epoch=63, grad_norm=122.640, loss=278.4418, lr=6.95e-04]
Epoch 63 TRAIN:  66%|██████▌   | 89/135 [00:21<00:10,  4.42it/s, epoch=63, grad_norm=215.810, loss=322.9474, lr=6.95e-04]
Epoch 63 TRAIN:  66%|██████▌   | 89/135 [00:22<00:10,  4.42it/s, epoch=63, grad_norm=140.357, loss=275.5815, lr=6.95e-04]
Epoch 63 TRAIN:  68%|██████▊   | 92/135 [00:22<00:09,  4.48it/s, epoch=63, grad_norm=140.357, loss=275.5815, lr=6.95e-04]
Epoch 63 TRAIN:  68%|██████▊   | 92/135 [00:22<00:09,  4.48it/s, epoch=63, grad_norm=311.806, loss=251.3876, lr=6.95e-04]
Epoch 63 TRAIN:  68%|██████▊   | 92/135 [00:22<00:09,  4.48it/s, epoch=63, grad_norm=149.339, loss=202.5774, lr=6.95e-04]
Epoch 63 TRAIN:  68%|██████▊   | 92/135 [00:22<00:09,  4.48it/s, epoch=63, grad_norm=209.638, loss=274.9857, lr=6.95e-04]
Epoch 63 TRAIN:  70%|███████   | 95/135 [00:22<00:08,  4.74it/s, epoch=63, grad_norm=209.638, loss=274.9857, lr=6.95e-04]
Epoch 63 TRAIN:  70%|███████   | 95/135 [00:22<00:08,  4.74it/s, epoch=63, grad_norm=173.726, loss=261.0029, lr=6.95e-04]
Epoch 63 TRAIN:  70%|███████   | 95/135 [00:23<00:08,  4.74it/s, epoch=63, grad_norm=145.601, loss=195.7804, lr=6.95e-04]
Epoch 63 TRAIN:  70%|███████   | 95/135 [00:23<00:08,  4.74it/s, epoch=63, grad_norm=128.564, loss=260.1289, lr=6.95e-04]
Epoch 63 TRAIN:  73%|███████▎  | 98/135 [00:23<00:07,  4.70it/s, epoch=63, grad_norm=128.564, loss=260.1289, lr=6.95e-04]
Epoch 63 TRAIN:  73%|███████▎  | 98/135 [00:23<00:07,  4.70it/s, epoch=63, grad_norm=317.587, loss=345.5988, lr=6.95e-04]
Epoch 63 TRAIN:  73%|███████▎  | 98/135 [00:23<00:07,  4.70it/s, epoch=63, grad_norm=98.613, loss=211.4256, lr=6.95e-04] 2026-04-16 20:24:26,159 DEBUG TRAIN | steps/sec 4.219| Batch 63/100 loss 211.425629 loss_att 173.533157 loss_ctc 299.841431 th_accuracy 0.374696 lr 6.9536e-04 grad_norm 98.612610 rank 0

Epoch 63 TRAIN:  73%|███████▎  | 98/135 [00:23<00:07,  4.70it/s, epoch=63, grad_norm=210.542, loss=324.4118, lr=6.95e-04]
Epoch 63 TRAIN:  75%|███████▍  | 101/135 [00:23<00:07,  4.48it/s, epoch=63, grad_norm=210.542, loss=324.4118, lr=6.95e-04]
Epoch 63 TRAIN:  75%|███████▍  | 101/135 [00:24<00:07,  4.48it/s, epoch=63, grad_norm=171.788, loss=256.4674, lr=6.96e-04]
Epoch 63 TRAIN:  75%|███████▍  | 101/135 [00:24<00:07,  4.48it/s, epoch=63, grad_norm=104.039, loss=260.2608, lr=6.96e-04]
Epoch 63 TRAIN:  75%|███████▍  | 101/135 [00:24<00:07,  4.48it/s, epoch=63, grad_norm=117.001, loss=241.3718, lr=6.96e-04]
Epoch 63 TRAIN:  77%|███████▋  | 104/135 [00:24<00:06,  4.48it/s, epoch=63, grad_norm=117.001, loss=241.3718, lr=6.96e-04]
Epoch 63 TRAIN:  77%|███████▋  | 104/135 [00:24<00:06,  4.48it/s, epoch=63, grad_norm=127.172, loss=262.1544, lr=6.96e-04]
Epoch 63 TRAIN:  77%|███████▋  | 104/135 [00:25<00:06,  4.48it/s, epoch=63, grad_norm=126.295, loss=296.0031, lr=6.96e-04]
Epoch 63 TRAIN:  77%|███████▋  | 104/135 [00:25<00:06,  4.48it/s, epoch=63, grad_norm=93.707, loss=257.1174, lr=6.96e-04] 
Epoch 63 TRAIN:  79%|███████▉  | 107/135 [00:25<00:06,  4.59it/s, epoch=63, grad_norm=93.707, loss=257.1174, lr=6.96e-04]
Epoch 63 TRAIN:  79%|███████▉  | 107/135 [00:25<00:06,  4.59it/s, epoch=63, grad_norm=221.776, loss=293.4687, lr=6.96e-04]
Epoch 63 TRAIN:  79%|███████▉  | 107/135 [00:25<00:06,  4.59it/s, epoch=63, grad_norm=235.882, loss=331.8693, lr=6.96e-04]
Epoch 63 TRAIN:  79%|███████▉  | 107/135 [00:25<00:06,  4.59it/s, epoch=63, grad_norm=147.200, loss=223.9972, lr=6.96e-04]
Epoch 63 TRAIN:  81%|████████▏ | 110/135 [00:25<00:05,  4.49it/s, epoch=63, grad_norm=147.200, loss=223.9972, lr=6.96e-04]
Epoch 63 TRAIN:  81%|████████▏ | 110/135 [00:26<00:05,  4.49it/s, epoch=63, grad_norm=212.711, loss=379.5597, lr=6.96e-04]
Epoch 63 TRAIN:  81%|████████▏ | 110/135 [00:26<00:05,  4.49it/s, epoch=63, grad_norm=177.590, loss=301.0035, lr=6.96e-04]
Epoch 63 TRAIN:  81%|████████▏ | 110/135 [00:26<00:05,  4.49it/s, epoch=63, grad_norm=161.343, loss=213.3245, lr=6.96e-04]
Epoch 63 TRAIN:  84%|████████▎ | 113/135 [00:26<00:04,  4.44it/s, epoch=63, grad_norm=161.343, loss=213.3245, lr=6.96e-04]
Epoch 63 TRAIN:  84%|████████▎ | 113/135 [00:26<00:04,  4.44it/s, epoch=63, grad_norm=318.297, loss=269.9505, lr=6.96e-04]
Epoch 63 TRAIN:  84%|████████▎ | 113/135 [00:27<00:04,  4.44it/s, epoch=63, grad_norm=151.093, loss=265.7583, lr=6.97e-04]
Epoch 63 TRAIN:  84%|████████▎ | 113/135 [00:27<00:04,  4.44it/s, epoch=63, grad_norm=154.358, loss=259.1897, lr=6.97e-04]
Epoch 63 TRAIN:  86%|████████▌ | 116/135 [00:27<00:03,  4.76it/s, epoch=63, grad_norm=154.358, loss=259.1897, lr=6.97e-04]
Epoch 63 TRAIN:  86%|████████▌ | 116/135 [00:27<00:03,  4.76it/s, epoch=63, grad_norm=115.889, loss=260.0239, lr=6.97e-04]
Epoch 63 TRAIN:  86%|████████▌ | 116/135 [00:27<00:03,  4.76it/s, epoch=63, grad_norm=143.359, loss=313.1754, lr=6.97e-04]
Epoch 63 TRAIN:  86%|████████▌ | 116/135 [00:27<00:03,  4.76it/s, epoch=63, grad_norm=229.633, loss=238.0772, lr=6.97e-04]
Epoch 63 TRAIN:  88%|████████▊ | 119/135 [00:27<00:03,  4.46it/s, epoch=63, grad_norm=229.633, loss=238.0772, lr=6.97e-04]
Epoch 63 TRAIN:  88%|████████▊ | 119/135 [00:28<00:03,  4.46it/s, epoch=63, grad_norm=230.861, loss=292.9500, lr=6.97e-04]
Epoch 63 TRAIN:  88%|████████▊ | 119/135 [00:28<00:03,  4.46it/s, epoch=63, grad_norm=161.334, loss=235.3585, lr=6.97e-04]
Epoch 63 TRAIN:  88%|████████▊ | 119/135 [00:28<00:03,  4.46it/s, epoch=63, grad_norm=183.062, loss=304.3010, lr=6.97e-04]
Epoch 63 TRAIN:  90%|█████████ | 122/135 [00:28<00:02,  4.51it/s, epoch=63, grad_norm=183.062, loss=304.3010, lr=6.97e-04]
Epoch 63 TRAIN:  90%|█████████ | 122/135 [00:28<00:02,  4.51it/s, epoch=63, grad_norm=174.067, loss=275.2683, lr=6.97e-04]
Epoch 63 TRAIN:  90%|█████████ | 122/135 [00:28<00:02,  4.51it/s, epoch=63, grad_norm=137.666, loss=340.5950, lr=6.97e-04]
Epoch 63 TRAIN:  90%|█████████ | 122/135 [00:29<00:02,  4.51it/s, epoch=63, grad_norm=226.656, loss=259.6023, lr=6.97e-04]
Epoch 63 TRAIN:  90%|█████████ | 122/135 [00:29<00:02,  4.51it/s, epoch=63, grad_norm=81.902, loss=215.0011, lr=6.97e-04] 
Epoch 63 TRAIN:  93%|█████████▎| 126/135 [00:29<00:01,  4.76it/s, epoch=63, grad_norm=81.902, loss=215.0011, lr=6.97e-04]
Epoch 63 TRAIN:  93%|█████████▎| 126/135 [00:29<00:01,  4.76it/s, epoch=63, grad_norm=299.932, loss=347.9261, lr=6.98e-04]
Epoch 63 TRAIN:  93%|█████████▎| 126/135 [00:29<00:01,  4.76it/s, epoch=63, grad_norm=250.563, loss=287.3083, lr=6.98e-04]
Epoch 63 TRAIN:  93%|█████████▎| 126/135 [00:29<00:01,  4.76it/s, epoch=63, grad_norm=203.059, loss=254.0288, lr=6.98e-04]
Epoch 63 TRAIN:  93%|█████████▎| 126/135 [00:30<00:01,  4.76it/s, epoch=63, grad_norm=316.975, loss=359.9249, lr=6.98e-04]
Epoch 63 TRAIN:  96%|█████████▋| 130/135 [00:30<00:00,  5.15it/s, epoch=63, grad_norm=316.975, loss=359.9249, lr=6.98e-04]
Epoch 63 TRAIN:  96%|█████████▋| 130/135 [00:30<00:00,  5.15it/s, epoch=63, grad_norm=176.595, loss=280.7218, lr=6.98e-04]
Epoch 63 TRAIN:  96%|█████████▋| 130/135 [00:30<00:00,  5.15it/s, epoch=63, grad_norm=347.706, loss=378.2570, lr=6.98e-04]
Epoch 63 TRAIN:  96%|█████████▋| 130/135 [00:30<00:00,  5.15it/s, epoch=63, grad_norm=152.198, loss=312.7219, lr=6.98e-04]
Epoch 63 TRAIN:  99%|█████████▊| 133/135 [00:30<00:00,  4.85it/s, epoch=63, grad_norm=152.198, loss=312.7219, lr=6.98e-04]
Epoch 63 TRAIN:  99%|█████████▊| 133/135 [00:30<00:00,  4.85it/s, epoch=63, grad_norm=413.775, loss=366.6724, lr=6.98e-04]
Epoch 63 TRAIN:  99%|█████████▊| 133/135 [00:31<00:00,  4.85it/s, epoch=63, grad_norm=296.270, loss=337.0300, lr=6.98e-04]
Epoch 63 TRAIN:  99%|█████████▊| 133/135 [00:31<00:00,  4.85it/s, epoch=63, grad_norm=520.366, loss=289.7769, lr=6.98e-04]
Epoch 63 TRAIN: 136it [00:31,  4.36it/s, epoch=63, grad_norm=520.366, loss=289.7769, lr=6.98e-04]                         

CV:   0%|          | 0/6 [00:00<?, ?it/s]/home/rookie/Documents/batuka/projects/wenet/wenet/wenet/utils/train_utils.py:689: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  amp_autocast(enabled=dtype is not None,

CV:  17%|█▋        | 1/6 [00:00<00:03,  1.32it/s]
                                                 
2026-04-16 20:24:34,432 INFO Epoch 63 Step 8728 CV info lr 6.9824e-04 cv_loss 475.5601307262074 rank 0 acc 0.36815160016218823
2026-04-16 20:24:34,433 INFO [Rank 0] Checkpoint: save to checkpoint exp/conformer_mongolian/epoch_63.pt
2026-04-16 20:24:34,544 INFO Epoch 64 Step 8728 TRAIN info lr 6.9824e-04 rank 0
2026-04-16 20:24:34,558 INFO using accumulate grad, new batch size is 1 times larger than before

Epoch 64 TRAIN:   0%|          | 0/136 [00:00<?, ?it/s]
Epoch 64 TRAIN:   0%|          | 0/136 [00:03<?, ?it/s, epoch=64, grad_norm=536.077, loss=426.6148, lr=6.98e-04]
Epoch 64 TRAIN:   1%|          | 1/136 [00:03<08:25,  3.74s/it, epoch=64, grad_norm=536.077, loss=426.6148, lr=6.98e-04]
Epoch 64 TRAIN:   1%|          | 1/136 [00:03<08:25,  3.74s/it, epoch=64, grad_norm=204.327, loss=224.1968, lr=6.98e-04]
Epoch 64 TRAIN:   1%|          | 1/136 [00:04<08:25,  3.74s/it, epoch=64, grad_norm=376.671, loss=356.8584, lr=6.98e-04]
Epoch 64 TRAIN:   1%|          | 1/136 [00:04<08:25,  3.74s/it, epoch=64, grad_norm=350.191, loss=356.1160, lr=6.99e-04]
Epoch 64 TRAIN:   1%|          | 1/136 [00:04<08:25,  3.74s/it, epoch=64, grad_norm=335.077, loss=353.6595, lr=6.99e-04]
Epoch 64 TRAIN:   1%|          | 1/136 [00:04<08:25,  3.74s/it, epoch=64, grad_norm=348.519, loss=393.7231, lr=6.99e-04]
Epoch 64 TRAIN:   4%|▍         | 6/136 [00:04<01:15,  1.72it/s, epoch=64, grad_norm=348.519, loss=393.7231, lr=6.99e-04]
Epoch 64 TRAIN:   4%|▍         | 6/136 [00:04<01:15,  1.72it/s, epoch=64, grad_norm=151.098, loss=259.2860, lr=6.99e-04]
Epoch 64 TRAIN:   4%|▍         | 6/136 [00:04<01:15,  1.72it/s, epoch=64, grad_norm=254.274, loss=303.1019, lr=6.99e-04]
Epoch 64 TRAIN:   4%|▍         | 6/136 [00:04<01:15,  1.72it/s, epoch=64, grad_norm=109.850, loss=308.8294, lr=6.99e-04]
Epoch 64 TRAIN:   4%|▍         | 6/136 [00:04<01:15,  1.72it/s, epoch=64, grad_norm=144.779, loss=331.9897, lr=6.99e-04]
Epoch 64 TRAIN:   4%|▍         | 6/136 [00:05<01:15,  1.72it/s, epoch=64, grad_norm=92.006, loss=316.5206, lr=6.99e-04] 
Epoch 64 TRAIN:   8%|▊         | 11/136 [00:05<00:40,  3.07it/s, epoch=64, grad_norm=92.006, loss=316.5206, lr=6.99e-04]
Epoch 64 TRAIN:   8%|▊         | 11/136 [00:05<00:40,  3.07it/s, epoch=64, grad_norm=200.430, loss=241.6356, lr=6.99e-04]
Epoch 64 TRAIN:   8%|▊         | 11/136 [00:05<00:40,  3.07it/s, epoch=64, grad_norm=169.731, loss=280.8748, lr=6.99e-04]
Epoch 64 TRAIN:   8%|▊         | 11/136 [00:05<00:40,  3.07it/s, epoch=64, grad_norm=128.651, loss=289.7376, lr=6.99e-04]
Epoch 64 TRAIN:   8%|▊         | 11/136 [00:05<00:40,  3.07it/s, epoch=64, grad_norm=137.037, loss=304.7509, lr=6.99e-04]
Epoch 64 TRAIN:  11%|█         | 15/136 [00:05<00:31,  3.81it/s, epoch=64, grad_norm=137.037, loss=304.7509, lr=6.99e-04]
Epoch 64 TRAIN:  11%|█         | 15/136 [00:05<00:31,  3.81it/s, epoch=64, grad_norm=103.666, loss=317.8137, lr=7.00e-04]
Epoch 64 TRAIN:  11%|█         | 15/136 [00:06<00:31,  3.81it/s, epoch=64, grad_norm=159.133, loss=290.2747, lr=7.00e-04]
Epoch 64 TRAIN:  11%|█         | 15/136 [00:06<00:31,  3.81it/s, epoch=64, grad_norm=141.468, loss=287.6439, lr=7.00e-04]
Epoch 64 TRAIN:  11%|█         | 15/136 [00:06<00:31,  3.81it/s, epoch=64, grad_norm=129.615, loss=304.2004, lr=7.00e-04]
Epoch 64 TRAIN:  14%|█▍        | 19/136 [00:06<00:27,  4.23it/s, epoch=64, grad_norm=129.615, loss=304.2004, lr=7.00e-04]
Epoch 64 TRAIN:  14%|█▍        | 19/136 [00:06<00:27,  4.23it/s, epoch=64, grad_norm=85.248, loss=299.0472, lr=7.00e-04] 
Epoch 64 TRAIN:  14%|█▍        | 19/136 [00:06<00:27,  4.23it/s, epoch=64, grad_norm=95.384, loss=310.1417, lr=7.00e-04]
Epoch 64 TRAIN:  14%|█▍        | 19/136 [00:07<00:27,  4.23it/s, epoch=64, grad_norm=88.773, loss=254.7862, lr=7.00e-04]
Epoch 64 TRAIN:  16%|█▌        | 22/136 [00:07<00:25,  4.50it/s, epoch=64, grad_norm=88.773, loss=254.7862, lr=7.00e-04]
Epoch 64 TRAIN:  16%|█▌        | 22/136 [00:07<00:25,  4.50it/s, epoch=64, grad_norm=93.793, loss=279.5136, lr=7.00e-04]
Epoch 64 TRAIN:  16%|█▌        | 22/136 [00:07<00:25,  4.50it/s, epoch=64, grad_norm=109.932, loss=296.1812, lr=7.00e-04]
Epoch 64 TRAIN:  16%|█▌        | 22/136 [00:07<00:25,  4.50it/s, epoch=64, grad_norm=104.906, loss=296.1626, lr=7.00e-04]
Epoch 64 TRAIN:  18%|█▊        | 25/136 [00:07<00:23,  4.77it/s, epoch=64, grad_norm=104.906, loss=296.1626, lr=7.00e-04]
Epoch 64 TRAIN:  18%|█▊        | 25/136 [00:07<00:23,  4.77it/s, epoch=64, grad_norm=109.683, loss=289.9949, lr=7.00e-04]
Epoch 64 TRAIN:  18%|█▊        | 25/136 [00:08<00:23,  4.77it/s, epoch=64, grad_norm=134.553, loss=309.4639, lr=7.00e-04]
Epoch 64 TRAIN:  18%|█▊        | 25/136 [00:08<00:23,  4.77it/s, epoch=64, grad_norm=160.643, loss=296.6999, lr=7.00e-04]
Epoch 64 TRAIN:  21%|██        | 28/136 [00:08<00:22,  4.71it/s, epoch=64, grad_norm=160.643, loss=296.6999, lr=7.00e-04]
Epoch 64 TRAIN:  21%|██        | 28/136 [00:08<00:22,  4.71it/s, epoch=64, grad_norm=117.214, loss=278.1197, lr=7.01e-04]
Epoch 64 TRAIN:  21%|██        | 28/136 [00:08<00:22,  4.71it/s, epoch=64, grad_norm=99.391, loss=277.1859, lr=7.01e-04] 
Epoch 64 TRAIN:  21%|██        | 28/136 [00:08<00:22,  4.71it/s, epoch=64, grad_norm=206.336, loss=322.8450, lr=7.01e-04]
Epoch 64 TRAIN:  23%|██▎       | 31/136 [00:08<00:21,  4.87it/s, epoch=64, grad_norm=206.336, loss=322.8450, lr=7.01e-04]
Epoch 64 TRAIN:  23%|██▎       | 31/136 [00:08<00:21,  4.87it/s, epoch=64, grad_norm=115.456, loss=284.1342, lr=7.01e-04]
Epoch 64 TRAIN:  23%|██▎       | 31/136 [00:09<00:21,  4.87it/s, epoch=64, grad_norm=152.342, loss=254.2376, lr=7.01e-04]
Epoch 64 TRAIN:  23%|██▎       | 31/136 [00:09<00:21,  4.87it/s, epoch=64, grad_norm=127.251, loss=305.7624, lr=7.01e-04]
Epoch 64 TRAIN:  25%|██▌       | 34/136 [00:09<00:20,  4.90it/s, epoch=64, grad_norm=127.251, loss=305.7624, lr=7.01e-04]
Epoch 64 TRAIN:  25%|██▌       | 34/136 [00:09<00:20,  4.90it/s, epoch=64, grad_norm=104.781, loss=290.9460, lr=7.01e-04]
Epoch 64 TRAIN:  25%|██▌       | 34/136 [00:09<00:20,  4.90it/s, epoch=64, grad_norm=111.712, loss=300.9090, lr=7.01e-04]
Epoch 64 TRAIN:  25%|██▌       | 34/136 [00:10<00:20,  4.90it/s, epoch=64, grad_norm=102.317, loss=261.2039, lr=7.01e-04]
Epoch 64 TRAIN:  27%|██▋       | 37/136 [00:10<00:20,  4.73it/s, epoch=64, grad_norm=102.317, loss=261.2039, lr=7.01e-04]
Epoch 64 TRAIN:  27%|██▋       | 37/136 [00:10<00:20,  4.73it/s, epoch=64, grad_norm=106.529, loss=279.4129, lr=7.01e-04]
Epoch 64 TRAIN:  27%|██▋       | 37/136 [00:10<00:20,  4.73it/s, epoch=64, grad_norm=89.783, loss=295.3156, lr=7.01e-04] 
Epoch 64 TRAIN:  27%|██▋       | 37/136 [00:10<00:20,  4.73it/s, epoch=64, grad_norm=139.566, loss=303.5672, lr=7.01e-04]
Epoch 64 TRAIN:  29%|██▉       | 40/136 [00:10<00:19,  4.85it/s, epoch=64, grad_norm=139.566, loss=303.5672, lr=7.01e-04]
Epoch 64 TRAIN:  29%|██▉       | 40/136 [00:10<00:19,  4.85it/s, epoch=64, grad_norm=85.370, loss=279.3872, lr=7.02e-04] 
Epoch 64 TRAIN:  29%|██▉       | 40/136 [00:11<00:19,  4.85it/s, epoch=64, grad_norm=90.836, loss=274.8700, lr=7.02e-04]
Epoch 64 TRAIN:  29%|██▉       | 40/136 [00:11<00:19,  4.85it/s, epoch=64, grad_norm=130.856, loss=257.7932, lr=7.02e-04]
Epoch 64 TRAIN:  32%|███▏      | 43/136 [00:11<00:19,  4.84it/s, epoch=64, grad_norm=130.856, loss=257.7932, lr=7.02e-04]
Epoch 64 TRAIN:  32%|███▏      | 43/136 [00:11<00:19,  4.84it/s, epoch=64, grad_norm=100.543, loss=311.7650, lr=7.02e-04]
Epoch 64 TRAIN:  32%|███▏      | 43/136 [00:11<00:19,  4.84it/s, epoch=64, grad_norm=106.604, loss=279.1270, lr=7.02e-04]
Epoch 64 TRAIN:  32%|███▏      | 43/136 [00:11<00:19,  4.84it/s, epoch=64, grad_norm=92.885, loss=309.2523, lr=7.02e-04] 
Epoch 64 TRAIN:  34%|███▍      | 46/136 [00:11<00:19,  4.74it/s, epoch=64, grad_norm=92.885, loss=309.2523, lr=7.02e-04]
Epoch 64 TRAIN:  34%|███▍      | 46/136 [00:12<00:19,  4.74it/s, epoch=64, grad_norm=103.611, loss=261.3108, lr=7.02e-04]
Epoch 64 TRAIN:  34%|███▍      | 46/136 [00:12<00:19,  4.74it/s, epoch=64, grad_norm=137.968, loss=208.0377, lr=7.02e-04]
Epoch 64 TRAIN:  34%|███▍      | 46/136 [00:12<00:19,  4.74it/s, epoch=64, grad_norm=120.090, loss=264.0475, lr=7.02e-04]
Epoch 64 TRAIN:  36%|███▌      | 49/136 [00:12<00:17,  4.84it/s, epoch=64, grad_norm=120.090, loss=264.0475, lr=7.02e-04]
Epoch 64 TRAIN:  36%|███▌      | 49/136 [00:12<00:17,  4.84it/s, epoch=64, grad_norm=147.360, loss=305.4219, lr=7.02e-04]
Epoch 64 TRAIN:  36%|███▌      | 49/136 [00:12<00:17,  4.84it/s, epoch=64, grad_norm=158.761, loss=320.9338, lr=7.02e-04]
Epoch 64 TRAIN:  36%|███▌      | 49/136 [00:13<00:17,  4.84it/s, epoch=64, grad_norm=144.646, loss=275.3242, lr=7.02e-04]
Epoch 64 TRAIN:  38%|███▊      | 52/136 [00:13<00:17,  4.73it/s, epoch=64, grad_norm=144.646, loss=275.3242, lr=7.02e-04]
Epoch 64 TRAIN:  38%|███▊      | 52/136 [00:13<00:17,  4.73it/s, epoch=64, grad_norm=126.629, loss=290.2145, lr=7.02e-04]
Epoch 64 TRAIN:  38%|███▊      | 52/136 [00:13<00:17,  4.73it/s, epoch=64, grad_norm=116.380, loss=304.0536, lr=7.03e-04]
Epoch 64 TRAIN:  38%|███▊      | 52/136 [00:13<00:17,  4.73it/s, epoch=64, grad_norm=118.747, loss=288.8182, lr=7.03e-04]
Epoch 64 TRAIN:  40%|████      | 55/136 [00:13<00:17,  4.65it/s, epoch=64, grad_norm=118.747, loss=288.8182, lr=7.03e-04]
Epoch 64 TRAIN:  40%|████      | 55/136 [00:14<00:17,  4.65it/s, epoch=64, grad_norm=105.115, loss=276.1957, lr=7.03e-04]
Epoch 64 TRAIN:  40%|████      | 55/136 [00:14<00:17,  4.65it/s, epoch=64, grad_norm=123.678, loss=278.8040, lr=7.03e-04]
Epoch 64 TRAIN:  40%|████      | 55/136 [00:14<00:17,  4.65it/s, epoch=64, grad_norm=110.472, loss=305.1139, lr=7.03e-04]
Epoch 64 TRAIN:  43%|████▎     | 58/136 [00:14<00:16,  4.68it/s, epoch=64, grad_norm=110.472, loss=305.1139, lr=7.03e-04]
Epoch 64 TRAIN:  43%|████▎     | 58/136 [00:14<00:16,  4.68it/s, epoch=64, grad_norm=99.823, loss=303.3255, lr=7.03e-04] 
Epoch 64 TRAIN:  43%|████▎     | 58/136 [00:14<00:16,  4.68it/s, epoch=64, grad_norm=138.318, loss=276.9981, lr=7.03e-04]
Epoch 64 TRAIN:  43%|████▎     | 58/136 [00:15<00:16,  4.68it/s, epoch=64, grad_norm=96.114, loss=249.9171, lr=7.03e-04] 
Epoch 64 TRAIN:  45%|████▍     | 61/136 [00:15<00:15,  4.74it/s, epoch=64, grad_norm=96.114, loss=249.9171, lr=7.03e-04]
Epoch 64 TRAIN:  45%|████▍     | 61/136 [00:15<00:15,  4.74it/s, epoch=64, grad_norm=149.074, loss=311.5768, lr=7.03e-04]
Epoch 64 TRAIN:  45%|████▍     | 61/136 [00:15<00:15,  4.74it/s, epoch=64, grad_norm=104.878, loss=304.2263, lr=7.03e-04]
Epoch 64 TRAIN:  45%|████▍     | 61/136 [00:15<00:15,  4.74it/s, epoch=64, grad_norm=109.069, loss=278.8972, lr=7.03e-04]
Epoch 64 TRAIN:  47%|████▋     | 64/136 [00:15<00:15,  4.60it/s, epoch=64, grad_norm=109.069, loss=278.8972, lr=7.03e-04]
Epoch 64 TRAIN:  47%|████▋     | 64/136 [00:15<00:15,  4.60it/s, epoch=64, grad_norm=132.572, loss=319.7671, lr=7.03e-04]
Epoch 64 TRAIN:  47%|████▋     | 64/136 [00:16<00:15,  4.60it/s, epoch=64, grad_norm=239.750, loss=263.6687, lr=7.04e-04]
Epoch 64 TRAIN:  47%|████▋     | 64/136 [00:16<00:15,  4.60it/s, epoch=64, grad_norm=158.987, loss=302.3503, lr=7.04e-04]
Epoch 64 TRAIN:  49%|████▉     | 67/136 [00:16<00:14,  4.75it/s, epoch=64, grad_norm=158.987, loss=302.3503, lr=7.04e-04]
Epoch 64 TRAIN:  49%|████▉     | 67/136 [00:16<00:14,  4.75it/s, epoch=64, grad_norm=150.790, loss=279.1642, lr=7.04e-04]
Epoch 64 TRAIN:  49%|████▉     | 67/136 [00:16<00:14,  4.75it/s, epoch=64, grad_norm=147.181, loss=237.7398, lr=7.04e-04]
Epoch 64 TRAIN:  49%|████▉     | 67/136 [00:17<00:14,  4.75it/s, epoch=64, grad_norm=131.126, loss=309.4480, lr=7.04e-04]
Epoch 64 TRAIN:  51%|█████▏    | 70/136 [00:17<00:14,  4.61it/s, epoch=64, grad_norm=131.126, loss=309.4480, lr=7.04e-04]
Epoch 64 TRAIN:  51%|█████▏    | 70/136 [00:17<00:14,  4.61it/s, epoch=64, grad_norm=97.039, loss=286.0522, lr=7.04e-04] 
Epoch 64 TRAIN:  51%|█████▏    | 70/136 [00:17<00:14,  4.61it/s, epoch=64, grad_norm=149.751, loss=301.7624, lr=7.04e-04]
Epoch 64 TRAIN:  51%|█████▏    | 70/136 [00:17<00:14,  4.61it/s, epoch=64, grad_norm=88.984, loss=254.2917, lr=7.04e-04] 
Epoch 64 TRAIN:  54%|█████▎    | 73/136 [00:17<00:13,  4.59it/s, epoch=64, grad_norm=88.984, loss=254.2917, lr=7.04e-04]
Epoch 64 TRAIN:  54%|█████▎    | 73/136 [00:17<00:13,  4.59it/s, epoch=64, grad_norm=233.139, loss=252.0526, lr=7.04e-04]
Epoch 64 TRAIN:  54%|█████▎    | 73/136 [00:18<00:13,  4.59it/s, epoch=64, grad_norm=176.845, loss=290.6951, lr=7.04e-04]
Epoch 64 TRAIN:  54%|█████▎    | 73/136 [00:18<00:13,  4.59it/s, epoch=64, grad_norm=137.419, loss=242.6444, lr=7.04e-04]
Epoch 64 TRAIN:  56%|█████▌    | 76/136 [00:18<00:12,  4.73it/s, epoch=64, grad_norm=137.419, loss=242.6444, lr=7.04e-04]
Epoch 64 TRAIN:  56%|█████▌    | 76/136 [00:18<00:12,  4.73it/s, epoch=64, grad_norm=137.371, loss=282.2635, lr=7.04e-04]
Epoch 64 TRAIN:  56%|█████▌    | 76/136 [00:18<00:12,  4.73it/s, epoch=64, grad_norm=129.899, loss=200.6352, lr=7.04e-04]
Epoch 64 TRAIN:  56%|█████▌    | 76/136 [00:19<00:12,  4.73it/s, epoch=64, grad_norm=228.066, loss=332.2823, lr=7.05e-04]
Epoch 64 TRAIN:  58%|█████▊    | 79/136 [00:19<00:12,  4.45it/s, epoch=64, grad_norm=228.066, loss=332.2823, lr=7.05e-04]
Epoch 64 TRAIN:  58%|█████▊    | 79/136 [00:19<00:12,  4.45it/s, epoch=64, grad_norm=176.530, loss=286.0215, lr=7.05e-04]
Epoch 64 TRAIN:  58%|█████▊    | 79/136 [00:19<00:12,  4.45it/s, epoch=64, grad_norm=300.996, loss=375.3986, lr=7.05e-04]
Epoch 64 TRAIN:  58%|█████▊    | 79/136 [00:19<00:12,  4.45it/s, epoch=64, grad_norm=104.551, loss=240.4181, lr=7.05e-04]
Epoch 64 TRAIN:  60%|██████    | 82/136 [00:19<00:11,  4.71it/s, epoch=64, grad_norm=104.551, loss=240.4181, lr=7.05e-04]
Epoch 64 TRAIN:  60%|██████    | 82/136 [00:19<00:11,  4.71it/s, epoch=64, grad_norm=146.561, loss=333.6945, lr=7.05e-04]
Epoch 64 TRAIN:  60%|██████    | 82/136 [00:20<00:11,  4.71it/s, epoch=64, grad_norm=149.471, loss=243.4925, lr=7.05e-04]
Epoch 64 TRAIN:  60%|██████    | 82/136 [00:20<00:11,  4.71it/s, epoch=64, grad_norm=227.320, loss=359.1215, lr=7.05e-04]
Epoch 64 TRAIN:  62%|██████▎   | 85/136 [00:20<00:10,  4.77it/s, epoch=64, grad_norm=227.320, loss=359.1215, lr=7.05e-04]
Epoch 64 TRAIN:  62%|██████▎   | 85/136 [00:20<00:10,  4.77it/s, epoch=64, grad_norm=93.474, loss=274.2607, lr=7.05e-04] 
Epoch 64 TRAIN:  62%|██████▎   | 85/136 [00:20<00:10,  4.77it/s, epoch=64, grad_norm=161.285, loss=293.3719, lr=7.05e-04]
Epoch 64 TRAIN:  62%|██████▎   | 85/136 [00:20<00:10,  4.77it/s, epoch=64, grad_norm=109.313, loss=247.2326, lr=7.05e-04]
Epoch 64 TRAIN:  65%|██████▍   | 88/136 [00:20<00:10,  4.61it/s, epoch=64, grad_norm=109.313, loss=247.2326, lr=7.05e-04]
Epoch 64 TRAIN:  65%|██████▍   | 88/136 [00:21<00:10,  4.61it/s, epoch=64, grad_norm=269.221, loss=373.6410, lr=7.05e-04]
Epoch 64 TRAIN:  65%|██████▍   | 88/136 [00:21<00:10,  4.61it/s, epoch=64, grad_norm=191.530, loss=273.5800, lr=7.05e-04]
Epoch 64 TRAIN:  65%|██████▍   | 88/136 [00:21<00:10,  4.61it/s, epoch=64, grad_norm=322.503, loss=266.6788, lr=7.06e-04]
Epoch 64 TRAIN:  67%|██████▋   | 91/136 [00:21<00:09,  4.75it/s, epoch=64, grad_norm=322.503, loss=266.6788, lr=7.06e-04]
Epoch 64 TRAIN:  67%|██████▋   | 91/136 [00:21<00:09,  4.75it/s, epoch=64, grad_norm=287.532, loss=265.6125, lr=7.06e-04]
Epoch 64 TRAIN:  67%|██████▋   | 91/136 [00:21<00:09,  4.75it/s, epoch=64, grad_norm=148.426, loss=332.6097, lr=7.06e-04]
Epoch 64 TRAIN:  67%|██████▋   | 91/136 [00:22<00:09,  4.75it/s, epoch=64, grad_norm=158.848, loss=272.7108, lr=7.06e-04]
Epoch 64 TRAIN:  69%|██████▉   | 94/136 [00:22<00:09,  4.67it/s, epoch=64, grad_norm=158.848, loss=272.7108, lr=7.06e-04]
Epoch 64 TRAIN:  69%|██████▉   | 94/136 [00:22<00:09,  4.67it/s, epoch=64, grad_norm=139.898, loss=208.4440, lr=7.06e-04]
Epoch 64 TRAIN:  69%|██████▉   | 94/136 [00:22<00:09,  4.67it/s, epoch=64, grad_norm=100.599, loss=239.4456, lr=7.06e-04]
Epoch 64 TRAIN:  69%|██████▉   | 94/136 [00:22<00:09,  4.67it/s, epoch=64, grad_norm=137.927, loss=240.1727, lr=7.06e-04]
Epoch 64 TRAIN:  71%|███████▏  | 97/136 [00:22<00:08,  4.70it/s, epoch=64, grad_norm=137.927, loss=240.1727, lr=7.06e-04]
Epoch 64 TRAIN:  71%|███████▏  | 97/136 [00:23<00:08,  4.70it/s, epoch=64, grad_norm=138.181, loss=174.7347, lr=7.06e-04]
Epoch 64 TRAIN:  71%|███████▏  | 97/136 [00:23<00:08,  4.70it/s, epoch=64, grad_norm=203.752, loss=282.7672, lr=7.06e-04]
Epoch 64 TRAIN:  71%|███████▏  | 97/136 [00:23<00:08,  4.70it/s, epoch=64, grad_norm=249.025, loss=275.9175, lr=7.06e-04]2026-04-16 20:24:58,014 DEBUG TRAIN | steps/sec 4.269| Batch 64/100 loss 275.917480 loss_att 223.919693 loss_ctc 397.245575 th_accuracy 0.373464 lr 7.0624e-04 grad_norm 249.025131 rank 0

Epoch 64 TRAIN:  74%|███████▎  | 100/136 [00:23<00:07,  4.74it/s, epoch=64, grad_norm=249.025, loss=275.9175, lr=7.06e-04]
Epoch 64 TRAIN:  74%|███████▎  | 100/136 [00:23<00:07,  4.74it/s, epoch=64, grad_norm=232.120, loss=287.7207, lr=7.06e-04]
Epoch 64 TRAIN:  74%|███████▎  | 100/136 [00:23<00:07,  4.74it/s, epoch=64, grad_norm=354.539, loss=340.1505, lr=7.06e-04]
Epoch 64 TRAIN:  74%|███████▎  | 100/136 [00:24<00:07,  4.74it/s, epoch=64, grad_norm=177.938, loss=246.6553, lr=7.06e-04]
Epoch 64 TRAIN:  76%|███████▌  | 103/136 [00:24<00:07,  4.69it/s, epoch=64, grad_norm=177.938, loss=246.6553, lr=7.06e-04]
Epoch 64 TRAIN:  76%|███████▌  | 103/136 [00:24<00:07,  4.69it/s, epoch=64, grad_norm=192.865, loss=226.9542, lr=7.07e-04]
Epoch 64 TRAIN:  76%|███████▌  | 103/136 [00:24<00:07,  4.69it/s, epoch=64, grad_norm=134.211, loss=207.2680, lr=7.07e-04]
Epoch 64 TRAIN:  76%|███████▌  | 103/136 [00:24<00:07,  4.69it/s, epoch=64, grad_norm=153.214, loss=193.9793, lr=7.07e-04]
Epoch 64 TRAIN:  78%|███████▊  | 106/136 [00:24<00:06,  4.67it/s, epoch=64, grad_norm=153.214, loss=193.9793, lr=7.07e-04]
Epoch 64 TRAIN:  78%|███████▊  | 106/136 [00:25<00:06,  4.67it/s, epoch=64, grad_norm=135.724, loss=288.0474, lr=7.07e-04]
Epoch 64 TRAIN:  78%|███████▊  | 106/136 [00:25<00:06,  4.67it/s, epoch=64, grad_norm=154.972, loss=262.4072, lr=7.07e-04]
Epoch 64 TRAIN:  78%|███████▊  | 106/136 [00:25<00:06,  4.67it/s, epoch=64, grad_norm=111.382, loss=230.5625, lr=7.07e-04]
Epoch 64 TRAIN:  80%|████████  | 109/136 [00:25<00:05,  4.58it/s, epoch=64, grad_norm=111.382, loss=230.5625, lr=7.07e-04]
Epoch 64 TRAIN:  80%|████████  | 109/136 [00:25<00:05,  4.58it/s, epoch=64, grad_norm=127.287, loss=287.5492, lr=7.07e-04]
Epoch 64 TRAIN:  80%|████████  | 109/136 [00:25<00:05,  4.58it/s, epoch=64, grad_norm=106.858, loss=227.0849, lr=7.07e-04]
Epoch 64 TRAIN:  80%|████████  | 109/136 [00:26<00:05,  4.58it/s, epoch=64, grad_norm=196.921, loss=302.3657, lr=7.07e-04]
Epoch 64 TRAIN:  82%|████████▏ | 112/136 [00:26<00:05,  4.73it/s, epoch=64, grad_norm=196.921, loss=302.3657, lr=7.07e-04]
Epoch 64 TRAIN:  82%|████████▏ | 112/136 [00:26<00:05,  4.73it/s, epoch=64, grad_norm=231.653, loss=295.0540, lr=7.07e-04]
Epoch 64 TRAIN:  82%|████████▏ | 112/136 [00:26<00:05,  4.73it/s, epoch=64, grad_norm=191.630, loss=278.8398, lr=7.07e-04]
Epoch 64 TRAIN:  82%|████████▏ | 112/136 [00:26<00:05,  4.73it/s, epoch=64, grad_norm=263.694, loss=246.1031, lr=7.07e-04]
Epoch 64 TRAIN:  82%|████████▏ | 112/136 [00:26<00:05,  4.73it/s, epoch=64, grad_norm=195.527, loss=310.6310, lr=7.08e-04]
Epoch 64 TRAIN:  85%|████████▌ | 116/136 [00:26<00:04,  4.99it/s, epoch=64, grad_norm=195.527, loss=310.6310, lr=7.08e-04]
Epoch 64 TRAIN:  85%|████████▌ | 116/136 [00:27<00:04,  4.99it/s, epoch=64, grad_norm=200.824, loss=312.2994, lr=7.08e-04]
Epoch 64 TRAIN:  85%|████████▌ | 116/136 [00:27<00:04,  4.99it/s, epoch=64, grad_norm=118.595, loss=259.5292, lr=7.08e-04]
Epoch 64 TRAIN:  85%|████████▌ | 116/136 [00:27<00:04,  4.99it/s, epoch=64, grad_norm=226.698, loss=256.0078, lr=7.08e-04]
Epoch 64 TRAIN:  88%|████████▊ | 119/136 [00:27<00:03,  4.75it/s, epoch=64, grad_norm=226.698, loss=256.0078, lr=7.08e-04]
Epoch 64 TRAIN:  88%|████████▊ | 119/136 [00:27<00:03,  4.75it/s, epoch=64, grad_norm=200.201, loss=327.8868, lr=7.08e-04]
Epoch 64 TRAIN:  88%|████████▊ | 119/136 [00:27<00:03,  4.75it/s, epoch=64, grad_norm=131.672, loss=260.7345, lr=7.08e-04]
Epoch 64 TRAIN:  88%|████████▊ | 119/136 [00:28<00:03,  4.75it/s, epoch=64, grad_norm=160.367, loss=296.3136, lr=7.08e-04]
Epoch 64 TRAIN:  90%|████████▉ | 122/136 [00:28<00:03,  4.48it/s, epoch=64, grad_norm=160.367, loss=296.3136, lr=7.08e-04]
Epoch 64 TRAIN:  90%|████████▉ | 122/136 [00:28<00:03,  4.48it/s, epoch=64, grad_norm=282.911, loss=320.3141, lr=7.08e-04]
Epoch 64 TRAIN:  90%|████████▉ | 122/136 [00:28<00:03,  4.48it/s, epoch=64, grad_norm=160.516, loss=253.3309, lr=7.08e-04]
Epoch 64 TRAIN:  90%|████████▉ | 122/136 [00:28<00:03,  4.48it/s, epoch=64, grad_norm=286.221, loss=337.3232, lr=7.08e-04]
Epoch 64 TRAIN:  92%|█████████▏| 125/136 [00:28<00:02,  4.61it/s, epoch=64, grad_norm=286.221, loss=337.3232, lr=7.08e-04]
Epoch 64 TRAIN:  92%|█████████▏| 125/136 [00:29<00:02,  4.61it/s, epoch=64, grad_norm=190.383, loss=275.0916, lr=7.08e-04]
Epoch 64 TRAIN:  92%|█████████▏| 125/136 [00:29<00:02,  4.61it/s, epoch=64, grad_norm=145.651, loss=303.7924, lr=7.08e-04]
Epoch 64 TRAIN:  92%|█████████▏| 125/136 [00:29<00:02,  4.61it/s, epoch=64, grad_norm=171.497, loss=331.5148, lr=7.08e-04]
Epoch 64 TRAIN:  94%|█████████▍| 128/136 [00:29<00:01,  4.73it/s, epoch=64, grad_norm=171.497, loss=331.5148, lr=7.08e-04]
Epoch 64 TRAIN:  94%|█████████▍| 128/136 [00:29<00:01,  4.73it/s, epoch=64, grad_norm=245.915, loss=311.8119, lr=7.09e-04]
Epoch 64 TRAIN:  94%|█████████▍| 128/136 [00:29<00:01,  4.73it/s, epoch=64, grad_norm=294.994, loss=355.5745, lr=7.09e-04]
Epoch 64 TRAIN:  94%|█████████▍| 128/136 [00:29<00:01,  4.73it/s, epoch=64, grad_norm=346.606, loss=274.5999, lr=7.09e-04]
Epoch 64 TRAIN:  96%|█████████▋| 131/136 [00:29<00:01,  4.98it/s, epoch=64, grad_norm=346.606, loss=274.5999, lr=7.09e-04]
Epoch 64 TRAIN:  96%|█████████▋| 131/136 [00:30<00:01,  4.98it/s, epoch=64, grad_norm=264.707, loss=339.6632, lr=7.09e-04]
Epoch 64 TRAIN:  96%|█████████▋| 131/136 [00:30<00:01,  4.98it/s, epoch=64, grad_norm=224.465, loss=280.6933, lr=7.09e-04]
Epoch 64 TRAIN:  96%|█████████▋| 131/136 [00:30<00:01,  4.98it/s, epoch=64, grad_norm=164.443, loss=365.8271, lr=7.09e-04]
Epoch 64 TRAIN:  99%|█████████▊| 134/136 [00:30<00:00,  5.05it/s, epoch=64, grad_norm=164.443, loss=365.8271, lr=7.09e-04]
Epoch 64 TRAIN:  99%|█████████▊| 134/136 [00:30<00:00,  5.05it/s, epoch=64, grad_norm=188.016, loss=310.6938, lr=7.09e-04]
Epoch 64 TRAIN:  99%|█████████▊| 134/136 [00:30<00:00,  5.05it/s, epoch=64, grad_norm=214.336, loss=295.6924, lr=7.09e-04]
Epoch 64 TRAIN: 100%|██████████| 136/136 [00:30<00:00,  4.41it/s, epoch=64, grad_norm=214.336, loss=295.6924, lr=7.09e-04]

CV:   0%|          | 0/6 [00:00<?, ?it/s]/home/rookie/Documents/batuka/projects/wenet/wenet/wenet/utils/train_utils.py:689: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  amp_autocast(enabled=dtype is not None,

CV:  17%|█▋        | 1/6 [00:00<00:03,  1.32it/s]
                                                 
2026-04-16 20:25:06,240 INFO Epoch 64 Step 8864 CV info lr 7.0912e-04 cv_loss 458.8609258478338 rank 0 acc 0.3624298920234044
2026-04-16 20:25:06,242 INFO [Rank 0] Checkpoint: save to checkpoint exp/conformer_mongolian/epoch_64.pt
2026-04-16 20:25:06,350 INFO Epoch 65 Step 8864 TRAIN info lr 7.0912e-04 rank 0
2026-04-16 20:25:06,367 INFO using accumulate grad, new batch size is 1 times larger than before

Epoch 65 TRAIN:   0%|          | 0/136 [00:00<?, ?it/s]
Epoch 65 TRAIN:   0%|          | 0/136 [00:03<?, ?it/s, epoch=65, grad_norm=311.861, loss=300.4564, lr=7.09e-04]
Epoch 65 TRAIN:   1%|          | 1/136 [00:03<08:20,  3.70s/it, epoch=65, grad_norm=311.861, loss=300.4564, lr=7.09e-04]
Epoch 65 TRAIN:   1%|          | 1/136 [00:03<08:20,  3.70s/it, epoch=65, grad_norm=721.911, loss=333.9309, lr=7.09e-04]
Epoch 65 TRAIN:   1%|          | 1/136 [00:04<08:20,  3.70s/it, epoch=65, grad_norm=570.868, loss=334.7992, lr=7.09e-04]
Epoch 65 TRAIN:   1%|          | 1/136 [00:04<08:20,  3.70s/it, epoch=65, grad_norm=273.521, loss=261.8964, lr=7.09e-04]
Epoch 65 TRAIN:   3%|▎         | 4/136 [00:04<01:55,  1.14it/s, epoch=65, grad_norm=273.521, loss=261.8964, lr=7.09e-04]
Epoch 65 TRAIN:   3%|▎         | 4/136 [00:04<01:55,  1.14it/s, epoch=65, grad_norm=363.098, loss=311.2962, lr=7.10e-04]
Epoch 65 TRAIN:   3%|▎         | 4/136 [00:04<01:55,  1.14it/s, epoch=65, grad_norm=475.778, loss=379.1617, lr=7.10e-04]
Epoch 65 TRAIN:   3%|▎         | 4/136 [00:04<01:55,  1.14it/s, epoch=65, grad_norm=337.199, loss=339.9608, lr=7.10e-04]
Epoch 65 TRAIN:   3%|▎         | 4/136 [00:04<01:55,  1.14it/s, epoch=65, grad_norm=220.200, loss=271.9183, lr=7.10e-04]
Epoch 65 TRAIN:   3%|▎         | 4/136 [00:04<01:55,  1.14it/s, epoch=65, grad_norm=167.598, loss=319.9172, lr=7.10e-04]
Epoch 65 TRAIN:   3%|▎         | 4/136 [00:04<01:55,  1.14it/s, epoch=65, grad_norm=133.960, loss=268.1169, lr=7.10e-04]
Epoch 65 TRAIN:   7%|▋         | 10/136 [00:04<00:41,  3.02it/s, epoch=65, grad_norm=133.960, loss=268.1169, lr=7.10e-04]
Epoch 65 TRAIN:   7%|▋         | 10/136 [00:05<00:41,  3.02it/s, epoch=65, grad_norm=172.494, loss=339.4016, lr=7.10e-04]
Epoch 65 TRAIN:   7%|▋         | 10/136 [00:05<00:41,  3.02it/s, epoch=65, grad_norm=137.170, loss=358.8528, lr=7.10e-04]
Epoch 65 TRAIN:   7%|▋         | 10/136 [00:05<00:41,  3.02it/s, epoch=65, grad_norm=123.000, loss=292.9257, lr=7.10e-04]
Epoch 65 TRAIN:   7%|▋         | 10/136 [00:05<00:41,  3.02it/s, epoch=65, grad_norm=124.380, loss=308.7711, lr=7.10e-04]
Epoch 65 TRAIN:  10%|█         | 14/136 [00:05<00:32,  3.79it/s, epoch=65, grad_norm=124.380, loss=308.7711, lr=7.10e-04]
Epoch 65 TRAIN:  10%|█         | 14/136 [00:05<00:32,  3.79it/s, epoch=65, grad_norm=165.365, loss=288.8994, lr=7.10e-04]
Epoch 65 TRAIN:  10%|█         | 14/136 [00:05<00:32,  3.79it/s, epoch=65, grad_norm=160.948, loss=310.8262, lr=7.10e-04]
Epoch 65 TRAIN:  10%|█         | 14/136 [00:06<00:32,  3.79it/s, epoch=65, grad_norm=256.253, loss=270.2581, lr=7.10e-04]
Epoch 65 TRAIN:  10%|█         | 14/136 [00:06<00:32,  3.79it/s, epoch=65, grad_norm=146.412, loss=316.9216, lr=7.11e-04]
Epoch 65 TRAIN:  13%|█▎        | 18/136 [00:06<00:26,  4.43it/s, epoch=65, grad_norm=146.412, loss=316.9216, lr=7.11e-04]
Epoch 65 TRAIN:  13%|█▎        | 18/136 [00:06<00:26,  4.43it/s, epoch=65, grad_norm=212.951, loss=266.9142, lr=7.11e-04]
Epoch 65 TRAIN:  13%|█▎        | 18/136 [00:06<00:26,  4.43it/s, epoch=65, grad_norm=154.290, loss=329.1527, lr=7.11e-04]
Epoch 65 TRAIN:  13%|█▎        | 18/136 [00:06<00:26,  4.43it/s, epoch=65, grad_norm=116.996, loss=294.1475, lr=7.11e-04]
Epoch 65 TRAIN:  13%|█▎        | 18/136 [00:07<00:26,  4.43it/s, epoch=65, grad_norm=92.732, loss=307.8560, lr=7.11e-04] 
Epoch 65 TRAIN:  16%|█▌        | 22/136 [00:07<00:24,  4.62it/s, epoch=65, grad_norm=92.732, loss=307.8560, lr=7.11e-04]
Epoch 65 TRAIN:  16%|█▌        | 22/136 [00:07<00:24,  4.62it/s, epoch=65, grad_norm=102.121, loss=268.7696, lr=7.11e-04]
Epoch 65 TRAIN:  16%|█▌        | 22/136 [00:07<00:24,  4.62it/s, epoch=65, grad_norm=81.048, loss=284.5846, lr=7.11e-04] 
Epoch 65 TRAIN:  16%|█▌        | 22/136 [00:07<00:24,  4.62it/s, epoch=65, grad_norm=161.567, loss=297.1840, lr=7.11e-04]
Epoch 65 TRAIN:  18%|█▊        | 25/136 [00:07<00:23,  4.79it/s, epoch=65, grad_norm=161.567, loss=297.1840, lr=7.11e-04]
Epoch 65 TRAIN:  18%|█▊        | 25/136 [00:07<00:23,  4.79it/s, epoch=65, grad_norm=106.574, loss=255.1041, lr=7.11e-04]
Epoch 65 TRAIN:  18%|█▊        | 25/136 [00:07<00:23,  4.79it/s, epoch=65, grad_norm=116.991, loss=262.2675, lr=7.11e-04]
Epoch 65 TRAIN:  18%|█▊        | 25/136 [00:08<00:23,  4.79it/s, epoch=65, grad_norm=199.071, loss=345.6303, lr=7.11e-04]
Epoch 65 TRAIN:  21%|██        | 28/136 [00:08<00:21,  5.07it/s, epoch=65, grad_norm=199.071, loss=345.6303, lr=7.11e-04]
Epoch 65 TRAIN:  21%|██        | 28/136 [00:08<00:21,  5.07it/s, epoch=65, grad_norm=91.160, loss=293.5048, lr=7.11e-04] 
Epoch 65 TRAIN:  21%|██        | 28/136 [00:08<00:21,  5.07it/s, epoch=65, grad_norm=191.591, loss=345.9648, lr=7.12e-04]
Epoch 65 TRAIN:  21%|██        | 28/136 [00:08<00:21,  5.07it/s, epoch=65, grad_norm=88.402, loss=259.6751, lr=7.12e-04] 
Epoch 65 TRAIN:  23%|██▎       | 31/136 [00:08<00:21,  4.85it/s, epoch=65, grad_norm=88.402, loss=259.6751, lr=7.12e-04]
Epoch 65 TRAIN:  23%|██▎       | 31/136 [00:08<00:21,  4.85it/s, epoch=65, grad_norm=104.377, loss=272.5401, lr=7.12e-04]
Epoch 65 TRAIN:  23%|██▎       | 31/136 [00:09<00:21,  4.85it/s, epoch=65, grad_norm=91.488, loss=285.5118, lr=7.12e-04] 
Epoch 65 TRAIN:  23%|██▎       | 31/136 [00:09<00:21,  4.85it/s, epoch=65, grad_norm=94.400, loss=278.9662, lr=7.12e-04]
Epoch 65 TRAIN:  25%|██▌       | 34/136 [00:09<00:21,  4.77it/s, epoch=65, grad_norm=94.400, loss=278.9662, lr=7.12e-04]
Epoch 65 TRAIN:  25%|██▌       | 34/136 [00:09<00:21,  4.77it/s, epoch=65, grad_norm=110.853, loss=285.2609, lr=7.12e-04]
Epoch 65 TRAIN:  25%|██▌       | 34/136 [00:09<00:21,  4.77it/s, epoch=65, grad_norm=93.573, loss=276.4098, lr=7.12e-04] 
Epoch 65 TRAIN:  25%|██▌       | 34/136 [00:09<00:21,  4.77it/s, epoch=65, grad_norm=110.730, loss=292.6943, lr=7.12e-04]
Epoch 65 TRAIN:  27%|██▋       | 37/136 [00:09<00:19,  5.03it/s, epoch=65, grad_norm=110.730, loss=292.6943, lr=7.12e-04]
Epoch 65 TRAIN:  27%|██▋       | 37/136 [00:10<00:19,  5.03it/s, epoch=65, grad_norm=101.966, loss=294.6529, lr=7.12e-04]
Epoch 65 TRAIN:  27%|██▋       | 37/136 [00:10<00:19,  5.03it/s, epoch=65, grad_norm=96.789, loss=313.2277, lr=7.12e-04] 
Epoch 65 TRAIN:  27%|██▋       | 37/136 [00:10<00:19,  5.03it/s, epoch=65, grad_norm=105.812, loss=276.1709, lr=7.12e-04]
Epoch 65 TRAIN:  29%|██▉       | 40/136 [00:10<00:20,  4.79it/s, epoch=65, grad_norm=105.812, loss=276.1709, lr=7.12e-04]
Epoch 65 TRAIN:  29%|██▉       | 40/136 [00:10<00:20,  4.79it/s, epoch=65, grad_norm=104.626, loss=285.4677, lr=7.12e-04]
Epoch 65 TRAIN:  29%|██▉       | 40/136 [00:11<00:20,  4.79it/s, epoch=65, grad_norm=116.131, loss=293.5023, lr=7.12e-04]
Epoch 65 TRAIN:  29%|██▉       | 40/136 [00:11<00:20,  4.79it/s, epoch=65, grad_norm=118.735, loss=264.6410, lr=7.13e-04]
Epoch 65 TRAIN:  32%|███▏      | 43/136 [00:11<00:19,  4.84it/s, epoch=65, grad_norm=118.735, loss=264.6410, lr=7.13e-04]
Epoch 65 TRAIN:  32%|███▏      | 43/136 [00:11<00:19,  4.84it/s, epoch=65, grad_norm=104.190, loss=290.7266, lr=7.13e-04]
Epoch 65 TRAIN:  32%|███▏      | 43/136 [00:11<00:19,  4.84it/s, epoch=65, grad_norm=116.058, loss=239.6562, lr=7.13e-04]
Epoch 65 TRAIN:  32%|███▏      | 43/136 [00:11<00:19,  4.84it/s, epoch=65, grad_norm=130.659, loss=298.1649, lr=7.13e-04]
Epoch 65 TRAIN:  34%|███▍      | 46/136 [00:11<00:18,  4.91it/s, epoch=65, grad_norm=130.659, loss=298.1649, lr=7.13e-04]
Epoch 65 TRAIN:  34%|███▍      | 46/136 [00:12<00:18,  4.91it/s, epoch=65, grad_norm=94.559, loss=278.9054, lr=7.13e-04] 
Epoch 65 TRAIN:  34%|███▍      | 46/136 [00:12<00:18,  4.91it/s, epoch=65, grad_norm=112.905, loss=309.0436, lr=7.13e-04]
Epoch 65 TRAIN:  34%|███▍      | 46/136 [00:12<00:18,  4.91it/s, epoch=65, grad_norm=141.618, loss=259.4976, lr=7.13e-04]
Epoch 65 TRAIN:  36%|███▌      | 49/136 [00:12<00:17,  4.84it/s, epoch=65, grad_norm=141.618, loss=259.4976, lr=7.13e-04]
Epoch 65 TRAIN:  36%|███▌      | 49/136 [00:12<00:17,  4.84it/s, epoch=65, grad_norm=127.584, loss=268.1834, lr=7.13e-04]
Epoch 65 TRAIN:  36%|███▌      | 49/136 [00:12<00:17,  4.84it/s, epoch=65, grad_norm=106.986, loss=311.0632, lr=7.13e-04]
Epoch 65 TRAIN:  36%|███▌      | 49/136 [00:13<00:17,  4.84it/s, epoch=65, grad_norm=92.212, loss=230.1768, lr=7.13e-04] 
Epoch 65 TRAIN:  38%|███▊      | 52/136 [00:13<00:17,  4.79it/s, epoch=65, grad_norm=92.212, loss=230.1768, lr=7.13e-04]
Epoch 65 TRAIN:  38%|███▊      | 52/136 [00:13<00:17,  4.79it/s, epoch=65, grad_norm=214.226, loss=324.0312, lr=7.13e-04]
Epoch 65 TRAIN:  38%|███▊      | 52/136 [00:13<00:17,  4.79it/s, epoch=65, grad_norm=142.484, loss=287.9090, lr=7.13e-04]
Epoch 65 TRAIN:  38%|███▊      | 52/136 [00:13<00:17,  4.79it/s, epoch=65, grad_norm=85.802, loss=257.2548, lr=7.14e-04] 
Epoch 65 TRAIN:  40%|████      | 55/136 [00:13<00:16,  4.83it/s, epoch=65, grad_norm=85.802, loss=257.2548, lr=7.14e-04]
Epoch 65 TRAIN:  40%|████      | 55/136 [00:13<00:16,  4.83it/s, epoch=65, grad_norm=126.234, loss=296.9308, lr=7.14e-04]
Epoch 65 TRAIN:  40%|████      | 55/136 [00:14<00:16,  4.83it/s, epoch=65, grad_norm=108.201, loss=273.6199, lr=7.14e-04]
Epoch 65 TRAIN:  40%|████      | 55/136 [00:14<00:16,  4.83it/s, epoch=65, grad_norm=83.953, loss=265.0799, lr=7.14e-04] 
Epoch 65 TRAIN:  43%|████▎     | 58/136 [00:14<00:15,  4.88it/s, epoch=65, grad_norm=83.953, loss=265.0799, lr=7.14e-04]
Epoch 65 TRAIN:  43%|████▎     | 58/136 [00:14<00:15,  4.88it/s, epoch=65, grad_norm=112.962, loss=300.8326, lr=7.14e-04]
Epoch 65 TRAIN:  43%|████▎     | 58/136 [00:14<00:15,  4.88it/s, epoch=65, grad_norm=142.530, loss=287.5441, lr=7.14e-04]
Epoch 65 TRAIN:  43%|████▎     | 58/136 [00:14<00:15,  4.88it/s, epoch=65, grad_norm=99.043, loss=268.5152, lr=7.14e-04] 
Epoch 65 TRAIN:  45%|████▍     | 61/136 [00:14<00:15,  4.85it/s, epoch=65, grad_norm=99.043, loss=268.5152, lr=7.14e-04]
Epoch 65 TRAIN:  45%|████▍     | 61/136 [00:15<00:15,  4.85it/s, epoch=65, grad_norm=106.484, loss=307.9313, lr=7.14e-04]
Epoch 65 TRAIN:  45%|████▍     | 61/136 [00:15<00:15,  4.85it/s, epoch=65, grad_norm=182.910, loss=319.1602, lr=7.14e-04]
Epoch 65 TRAIN:  45%|████▍     | 61/136 [00:15<00:15,  4.85it/s, epoch=65, grad_norm=100.647, loss=303.6165, lr=7.14e-04]
Epoch 65 TRAIN:  47%|████▋     | 64/136 [00:15<00:14,  4.86it/s, epoch=65, grad_norm=100.647, loss=303.6165, lr=7.14e-04]
Epoch 65 TRAIN:  47%|████▋     | 64/136 [00:15<00:14,  4.86it/s, epoch=65, grad_norm=106.728, loss=302.5570, lr=7.14e-04]
Epoch 65 TRAIN:  47%|████▋     | 64/136 [00:16<00:14,  4.86it/s, epoch=65, grad_norm=123.263, loss=296.0925, lr=7.14e-04]
Epoch 65 TRAIN:  47%|████▋     | 64/136 [00:16<00:14,  4.86it/s, epoch=65, grad_norm=115.185, loss=283.2210, lr=7.14e-04]
Epoch 65 TRAIN:  49%|████▉     | 67/136 [00:16<00:15,  4.53it/s, epoch=65, grad_norm=115.185, loss=283.2210, lr=7.14e-04]
Epoch 65 TRAIN:  49%|████▉     | 67/136 [00:16<00:15,  4.53it/s, epoch=65, grad_norm=116.123, loss=252.6931, lr=7.15e-04]
Epoch 65 TRAIN:  49%|████▉     | 67/136 [00:16<00:15,  4.53it/s, epoch=65, grad_norm=269.813, loss=213.5781, lr=7.15e-04]
Epoch 65 TRAIN:  49%|████▉     | 67/136 [00:16<00:15,  4.53it/s, epoch=65, grad_norm=179.424, loss=232.7254, lr=7.15e-04]
Epoch 65 TRAIN:  51%|█████▏    | 70/136 [00:16<00:13,  4.85it/s, epoch=65, grad_norm=179.424, loss=232.7254, lr=7.15e-04]
Epoch 65 TRAIN:  51%|█████▏    | 70/136 [00:17<00:13,  4.85it/s, epoch=65, grad_norm=141.467, loss=234.9440, lr=7.15e-04]
Epoch 65 TRAIN:  51%|█████▏    | 70/136 [00:17<00:13,  4.85it/s, epoch=65, grad_norm=110.613, loss=288.1011, lr=7.15e-04]
Epoch 65 TRAIN:  51%|█████▏    | 70/136 [00:17<00:13,  4.85it/s, epoch=65, grad_norm=98.169, loss=272.6685, lr=7.15e-04] 
Epoch 65 TRAIN:  54%|█████▎    | 73/136 [00:17<00:13,  4.67it/s, epoch=65, grad_norm=98.169, loss=272.6685, lr=7.15e-04]
Epoch 65 TRAIN:  54%|█████▎    | 73/136 [00:17<00:13,  4.67it/s, epoch=65, grad_norm=120.370, loss=285.2350, lr=7.15e-04]
Epoch 65 TRAIN:  54%|█████▎    | 73/136 [00:17<00:13,  4.67it/s, epoch=65, grad_norm=76.234, loss=255.1328, lr=7.15e-04] 
Epoch 65 TRAIN:  54%|█████▎    | 73/136 [00:18<00:13,  4.67it/s, epoch=65, grad_norm=141.012, loss=241.6435, lr=7.15e-04]
Epoch 65 TRAIN:  56%|█████▌    | 76/136 [00:18<00:12,  4.81it/s, epoch=65, grad_norm=141.012, loss=241.6435, lr=7.15e-04]
Epoch 65 TRAIN:  56%|█████▌    | 76/136 [00:18<00:12,  4.81it/s, epoch=65, grad_norm=138.436, loss=271.5196, lr=7.15e-04]
Epoch 65 TRAIN:  56%|█████▌    | 76/136 [00:18<00:12,  4.81it/s, epoch=65, grad_norm=101.803, loss=251.5798, lr=7.15e-04]
Epoch 65 TRAIN:  56%|█████▌    | 76/136 [00:18<00:12,  4.81it/s, epoch=65, grad_norm=211.394, loss=323.4915, lr=7.15e-04]
Epoch 65 TRAIN:  58%|█████▊    | 79/136 [00:18<00:12,  4.64it/s, epoch=65, grad_norm=211.394, loss=323.4915, lr=7.15e-04]
Epoch 65 TRAIN:  58%|█████▊    | 79/136 [00:19<00:12,  4.64it/s, epoch=65, grad_norm=134.759, loss=291.1292, lr=7.16e-04]
Epoch 65 TRAIN:  58%|█████▊    | 79/136 [00:19<00:12,  4.64it/s, epoch=65, grad_norm=125.173, loss=294.4424, lr=7.16e-04]
Epoch 65 TRAIN:  58%|█████▊    | 79/136 [00:19<00:12,  4.64it/s, epoch=65, grad_norm=130.995, loss=286.4708, lr=7.16e-04]
Epoch 65 TRAIN:  60%|██████    | 82/136 [00:19<00:11,  4.74it/s, epoch=65, grad_norm=130.995, loss=286.4708, lr=7.16e-04]
Epoch 65 TRAIN:  60%|██████    | 82/136 [00:19<00:11,  4.74it/s, epoch=65, grad_norm=159.769, loss=267.5807, lr=7.16e-04]
Epoch 65 TRAIN:  60%|██████    | 82/136 [00:19<00:11,  4.74it/s, epoch=65, grad_norm=164.031, loss=323.0127, lr=7.16e-04]
Epoch 65 TRAIN:  60%|██████    | 82/136 [00:20<00:11,  4.74it/s, epoch=65, grad_norm=380.799, loss=212.4099, lr=7.16e-04]
Epoch 65 TRAIN:  62%|██████▎   | 85/136 [00:20<00:10,  4.67it/s, epoch=65, grad_norm=380.799, loss=212.4099, lr=7.16e-04]
Epoch 65 TRAIN:  62%|██████▎   | 85/136 [00:20<00:10,  4.67it/s, epoch=65, grad_norm=106.266, loss=338.7497, lr=7.16e-04]
Epoch 65 TRAIN:  62%|██████▎   | 85/136 [00:20<00:10,  4.67it/s, epoch=65, grad_norm=107.979, loss=280.8255, lr=7.16e-04]
Epoch 65 TRAIN:  62%|██████▎   | 85/136 [00:20<00:10,  4.67it/s, epoch=65, grad_norm=105.088, loss=306.0341, lr=7.16e-04]
Epoch 65 TRAIN:  65%|██████▍   | 88/136 [00:20<00:10,  4.56it/s, epoch=65, grad_norm=105.088, loss=306.0341, lr=7.16e-04]
Epoch 65 TRAIN:  65%|██████▍   | 88/136 [00:21<00:10,  4.56it/s, epoch=65, grad_norm=251.111, loss=310.4184, lr=7.16e-04]
Epoch 65 TRAIN:  65%|██████▍   | 88/136 [00:21<00:10,  4.56it/s, epoch=65, grad_norm=209.468, loss=325.1798, lr=7.16e-04]
Epoch 65 TRAIN:  65%|██████▍   | 88/136 [00:21<00:10,  4.56it/s, epoch=65, grad_norm=124.601, loss=311.7241, lr=7.16e-04]
Epoch 65 TRAIN:  67%|██████▋   | 91/136 [00:21<00:09,  4.61it/s, epoch=65, grad_norm=124.601, loss=311.7241, lr=7.16e-04]
Epoch 65 TRAIN:  67%|██████▋   | 91/136 [00:21<00:09,  4.61it/s, epoch=65, grad_norm=108.739, loss=238.5682, lr=7.16e-04]
Epoch 65 TRAIN:  67%|██████▋   | 91/136 [00:21<00:09,  4.61it/s, epoch=65, grad_norm=157.626, loss=357.5404, lr=7.17e-04]
Epoch 65 TRAIN:  67%|██████▋   | 91/136 [00:22<00:09,  4.61it/s, epoch=65, grad_norm=117.857, loss=276.7714, lr=7.17e-04]
Epoch 65 TRAIN:  69%|██████▉   | 94/136 [00:22<00:09,  4.54it/s, epoch=65, grad_norm=117.857, loss=276.7714, lr=7.17e-04]
Epoch 65 TRAIN:  69%|██████▉   | 94/136 [00:22<00:09,  4.54it/s, epoch=65, grad_norm=140.007, loss=229.4920, lr=7.17e-04]
Epoch 65 TRAIN:  69%|██████▉   | 94/136 [00:22<00:09,  4.54it/s, epoch=65, grad_norm=121.111, loss=174.7957, lr=7.17e-04]
Epoch 65 TRAIN:  69%|██████▉   | 94/136 [00:22<00:09,  4.54it/s, epoch=65, grad_norm=132.336, loss=204.7442, lr=7.17e-04]
Epoch 65 TRAIN:  71%|███████▏  | 97/136 [00:22<00:08,  4.73it/s, epoch=65, grad_norm=132.336, loss=204.7442, lr=7.17e-04]
Epoch 65 TRAIN:  71%|███████▏  | 97/136 [00:22<00:08,  4.73it/s, epoch=65, grad_norm=149.378, loss=257.1483, lr=7.17e-04]
Epoch 65 TRAIN:  71%|███████▏  | 97/136 [00:23<00:08,  4.73it/s, epoch=65, grad_norm=184.486, loss=245.3271, lr=7.17e-04]
Epoch 65 TRAIN:  71%|███████▏  | 97/136 [00:23<00:08,  4.73it/s, epoch=65, grad_norm=227.242, loss=260.8356, lr=7.17e-04]2026-04-16 20:25:29,706 DEBUG TRAIN | steps/sec 4.291| Batch 65/100 loss 260.835602 loss_att 204.814880 loss_ctc 391.550629 th_accuracy 0.389241 lr 7.1712e-04 grad_norm 227.241516 rank 0

Epoch 65 TRAIN:  74%|███████▎  | 100/136 [00:23<00:07,  4.65it/s, epoch=65, grad_norm=227.242, loss=260.8356, lr=7.17e-04]
Epoch 65 TRAIN:  74%|███████▎  | 100/136 [00:23<00:07,  4.65it/s, epoch=65, grad_norm=205.817, loss=333.7656, lr=7.17e-04]
Epoch 65 TRAIN:  74%|███████▎  | 100/136 [00:23<00:07,  4.65it/s, epoch=65, grad_norm=196.722, loss=284.3856, lr=7.17e-04]
Epoch 65 TRAIN:  74%|███████▎  | 100/136 [00:23<00:07,  4.65it/s, epoch=65, grad_norm=104.949, loss=247.0706, lr=7.17e-04]
Epoch 65 TRAIN:  76%|███████▌  | 103/136 [00:23<00:06,  4.91it/s, epoch=65, grad_norm=104.949, loss=247.0706, lr=7.17e-04]
Epoch 65 TRAIN:  76%|███████▌  | 103/136 [00:24<00:06,  4.91it/s, epoch=65, grad_norm=89.503, loss=201.6572, lr=7.17e-04] 
Epoch 65 TRAIN:  76%|███████▌  | 103/136 [00:24<00:06,  4.91it/s, epoch=65, grad_norm=164.448, loss=292.0998, lr=7.18e-04]
Epoch 65 TRAIN:  76%|███████▌  | 103/136 [00:24<00:06,  4.91it/s, epoch=65, grad_norm=184.345, loss=239.1454, lr=7.18e-04]
Epoch 65 TRAIN:  78%|███████▊  | 106/136 [00:24<00:06,  4.70it/s, epoch=65, grad_norm=184.345, loss=239.1454, lr=7.18e-04]
Epoch 65 TRAIN:  78%|███████▊  | 106/136 [00:24<00:06,  4.70it/s, epoch=65, grad_norm=261.240, loss=355.5515, lr=7.18e-04]
Epoch 65 TRAIN:  78%|███████▊  | 106/136 [00:25<00:06,  4.70it/s, epoch=65, grad_norm=246.773, loss=314.9067, lr=7.18e-04]
Epoch 65 TRAIN:  78%|███████▊  | 106/136 [00:25<00:06,  4.70it/s, epoch=65, grad_norm=273.517, loss=366.4385, lr=7.18e-04]
Epoch 65 TRAIN:  80%|████████  | 109/136 [00:25<00:05,  4.65it/s, epoch=65, grad_norm=273.517, loss=366.4385, lr=7.18e-04]
Epoch 65 TRAIN:  80%|████████  | 109/136 [00:25<00:05,  4.65it/s, epoch=65, grad_norm=116.375, loss=256.8717, lr=7.18e-04]
Epoch 65 TRAIN:  80%|████████  | 109/136 [00:25<00:05,  4.65it/s, epoch=65, grad_norm=124.695, loss=289.3598, lr=7.18e-04]
Epoch 65 TRAIN:  80%|████████  | 109/136 [00:25<00:05,  4.65it/s, epoch=65, grad_norm=123.197, loss=212.7321, lr=7.18e-04]
Epoch 65 TRAIN:  82%|████████▏ | 112/136 [00:25<00:05,  4.67it/s, epoch=65, grad_norm=123.197, loss=212.7321, lr=7.18e-04]
Epoch 65 TRAIN:  82%|████████▏ | 112/136 [00:25<00:05,  4.67it/s, epoch=65, grad_norm=206.065, loss=361.4426, lr=7.18e-04]
Epoch 65 TRAIN:  82%|████████▏ | 112/136 [00:26<00:05,  4.67it/s, epoch=65, grad_norm=151.048, loss=250.6397, lr=7.18e-04]
Epoch 65 TRAIN:  82%|████████▏ | 112/136 [00:26<00:05,  4.67it/s, epoch=65, grad_norm=238.784, loss=341.7351, lr=7.18e-04]
Epoch 65 TRAIN:  85%|████████▍ | 115/136 [00:26<00:04,  4.88it/s, epoch=65, grad_norm=238.784, loss=341.7351, lr=7.18e-04]
Epoch 65 TRAIN:  85%|████████▍ | 115/136 [00:26<00:04,  4.88it/s, epoch=65, grad_norm=106.993, loss=228.0714, lr=7.18e-04]
Epoch 65 TRAIN:  85%|████████▍ | 115/136 [00:26<00:04,  4.88it/s, epoch=65, grad_norm=198.550, loss=295.2595, lr=7.18e-04]
Epoch 65 TRAIN:  85%|████████▍ | 115/136 [00:27<00:04,  4.88it/s, epoch=65, grad_norm=142.596, loss=244.1785, lr=7.19e-04]
Epoch 65 TRAIN:  87%|████████▋ | 118/136 [00:27<00:03,  4.66it/s, epoch=65, grad_norm=142.596, loss=244.1785, lr=7.19e-04]
Epoch 65 TRAIN:  87%|████████▋ | 118/136 [00:27<00:03,  4.66it/s, epoch=65, grad_norm=174.659, loss=290.7142, lr=7.19e-04]
Epoch 65 TRAIN:  87%|████████▋ | 118/136 [00:27<00:03,  4.66it/s, epoch=65, grad_norm=199.083, loss=308.1816, lr=7.19e-04]
Epoch 65 TRAIN:  87%|████████▋ | 118/136 [00:27<00:03,  4.66it/s, epoch=65, grad_norm=193.695, loss=304.6340, lr=7.19e-04]
Epoch 65 TRAIN:  89%|████████▉ | 121/136 [00:27<00:03,  4.94it/s, epoch=65, grad_norm=193.695, loss=304.6340, lr=7.19e-04]
Epoch 65 TRAIN:  89%|████████▉ | 121/136 [00:27<00:03,  4.94it/s, epoch=65, grad_norm=155.008, loss=271.3048, lr=7.19e-04]
Epoch 65 TRAIN:  89%|████████▉ | 121/136 [00:28<00:03,  4.94it/s, epoch=65, grad_norm=119.306, loss=219.8467, lr=7.19e-04]
Epoch 65 TRAIN:  89%|████████▉ | 121/136 [00:28<00:03,  4.94it/s, epoch=65, grad_norm=118.400, loss=288.8209, lr=7.19e-04]
Epoch 65 TRAIN:  91%|█████████ | 124/136 [00:28<00:02,  4.59it/s, epoch=65, grad_norm=118.400, loss=288.8209, lr=7.19e-04]
Epoch 65 TRAIN:  91%|█████████ | 124/136 [00:28<00:02,  4.59it/s, epoch=65, grad_norm=243.768, loss=270.5092, lr=7.19e-04]
Epoch 65 TRAIN:  91%|█████████ | 124/136 [00:28<00:02,  4.59it/s, epoch=65, grad_norm=320.566, loss=384.1920, lr=7.19e-04]
Epoch 65 TRAIN:  91%|█████████ | 124/136 [00:29<00:02,  4.59it/s, epoch=65, grad_norm=96.814, loss=273.2662, lr=7.19e-04] 
Epoch 65 TRAIN:  93%|█████████▎| 127/136 [00:29<00:02,  4.45it/s, epoch=65, grad_norm=96.814, loss=273.2662, lr=7.19e-04]
Epoch 65 TRAIN:  93%|█████████▎| 127/136 [00:29<00:02,  4.45it/s, epoch=65, grad_norm=136.779, loss=310.7245, lr=7.19e-04]
Epoch 65 TRAIN:  93%|█████████▎| 127/136 [00:29<00:02,  4.45it/s, epoch=65, grad_norm=132.342, loss=320.8091, lr=7.19e-04]
Epoch 65 TRAIN:  93%|█████████▎| 127/136 [00:29<00:02,  4.45it/s, epoch=65, grad_norm=256.733, loss=274.6649, lr=7.20e-04]
Epoch 65 TRAIN:  96%|█████████▌| 130/136 [00:29<00:01,  4.63it/s, epoch=65, grad_norm=256.733, loss=274.6649, lr=7.20e-04]
Epoch 65 TRAIN:  96%|█████████▌| 130/136 [00:29<00:01,  4.63it/s, epoch=65, grad_norm=155.738, loss=306.0128, lr=7.20e-04]
Epoch 65 TRAIN:  96%|█████████▌| 130/136 [00:30<00:01,  4.63it/s, epoch=65, grad_norm=163.627, loss=253.6381, lr=7.20e-04]
Epoch 65 TRAIN:  96%|█████████▌| 130/136 [00:30<00:01,  4.63it/s, epoch=65, grad_norm=194.358, loss=317.6945, lr=7.20e-04]
Epoch 65 TRAIN:  98%|█████████▊| 133/136 [00:30<00:00,  4.67it/s, epoch=65, grad_norm=194.358, loss=317.6945, lr=7.20e-04]
Epoch 65 TRAIN:  98%|█████████▊| 133/136 [00:30<00:00,  4.67it/s, epoch=65, grad_norm=186.173, loss=298.8382, lr=7.20e-04]
Epoch 65 TRAIN:  98%|█████████▊| 133/136 [00:30<00:00,  4.67it/s, epoch=65, grad_norm=199.493, loss=268.3113, lr=7.20e-04]
Epoch 65 TRAIN:  98%|█████████▊| 133/136 [00:30<00:00,  4.67it/s, epoch=65, grad_norm=281.555, loss=286.3541, lr=7.20e-04]
Epoch 65 TRAIN: 100%|██████████| 136/136 [00:30<00:00,  4.41it/s, epoch=65, grad_norm=281.555, loss=286.3541, lr=7.20e-04]

CV:   0%|          | 0/6 [00:00<?, ?it/s]/home/rookie/Documents/batuka/projects/wenet/wenet/wenet/utils/train_utils.py:689: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  amp_autocast(enabled=dtype is not None,

CV:  17%|█▋        | 1/6 [00:00<00:03,  1.32it/s]
                                                 
2026-04-16 20:25:38,059 INFO Epoch 65 Step 9000 CV info lr 7.2000e-04 cv_loss 481.4031594016335 rank 0 acc 0.3621234844128291
2026-04-16 20:25:38,060 INFO [Rank 0] Checkpoint: save to checkpoint exp/conformer_mongolian/epoch_65.pt
2026-04-16 20:25:38,179 INFO Epoch 66 Step 9000 TRAIN info lr 7.2000e-04 rank 0
2026-04-16 20:25:38,200 INFO using accumulate grad, new batch size is 1 times larger than before

Epoch 66 TRAIN:   0%|          | 0/136 [00:00<?, ?it/s]
Epoch 66 TRAIN:   0%|          | 0/136 [00:04<?, ?it/s, epoch=66, grad_norm=512.131, loss=331.2731, lr=7.20e-04]
Epoch 66 TRAIN:   1%|          | 1/136 [00:04<09:00,  4.00s/it, epoch=66, grad_norm=512.131, loss=331.2731, lr=7.20e-04]
Epoch 66 TRAIN:   1%|          | 1/136 [00:04<09:00,  4.00s/it, epoch=66, grad_norm=510.824, loss=402.0900, lr=7.20e-04]
Epoch 66 TRAIN:   1%|          | 1/136 [00:04<09:00,  4.00s/it, epoch=66, grad_norm=469.765, loss=372.2264, lr=7.20e-04]
Epoch 66 TRAIN:   1%|          | 1/136 [00:04<09:00,  4.00s/it, epoch=66, grad_norm=389.560, loss=346.4443, lr=7.20e-04]
Epoch 66 TRAIN:   1%|          | 1/136 [00:04<09:00,  4.00s/it, epoch=66, grad_norm=445.619, loss=359.6650, lr=7.20e-04]
Epoch 66 TRAIN:   1%|          | 1/136 [00:04<09:00,  4.00s/it, epoch=66, grad_norm=354.723, loss=345.0909, lr=7.20e-04]
Epoch 66 TRAIN:   1%|          | 1/136 [00:04<09:00,  4.00s/it, epoch=66, grad_norm=491.497, loss=410.4573, lr=7.21e-04]
Epoch 66 TRAIN:   1%|          | 1/136 [00:04<09:00,  4.00s/it, epoch=66, grad_norm=379.933, loss=326.8298, lr=7.21e-04]
Epoch 66 TRAIN:   1%|          | 1/136 [00:04<09:00,  4.00s/it, epoch=66, grad_norm=127.938, loss=260.5918, lr=7.21e-04]
Epoch 66 TRAIN:   7%|▋         | 9/136 [00:04<00:50,  2.53it/s, epoch=66, grad_norm=127.938, loss=260.5918, lr=7.21e-04]
Epoch 66 TRAIN:   7%|▋         | 9/136 [00:04<00:50,  2.53it/s, epoch=66, grad_norm=96.631, loss=247.9677, lr=7.21e-04] 
Epoch 66 TRAIN:   7%|▋         | 9/136 [00:04<00:50,  2.53it/s, epoch=66, grad_norm=97.193, loss=268.6917, lr=7.21e-04]
Epoch 66 TRAIN:   7%|▋         | 9/136 [00:05<00:50,  2.53it/s, epoch=66, grad_norm=93.015, loss=295.1568, lr=7.21e-04]
Epoch 66 TRAIN:   7%|▋         | 9/136 [00:05<00:50,  2.53it/s, epoch=66, grad_norm=100.328, loss=290.1130, lr=7.21e-04]
Epoch 66 TRAIN:  10%|▉         | 13/136 [00:05<00:37,  3.29it/s, epoch=66, grad_norm=100.328, loss=290.1130, lr=7.21e-04]
Epoch 66 TRAIN:  10%|▉         | 13/136 [00:05<00:37,  3.29it/s, epoch=66, grad_norm=82.292, loss=294.1562, lr=7.21e-04] 
Epoch 66 TRAIN:  10%|▉         | 13/136 [00:05<00:37,  3.29it/s, epoch=66, grad_norm=107.925, loss=293.6681, lr=7.21e-04]
Epoch 66 TRAIN:  10%|▉         | 13/136 [00:05<00:37,  3.29it/s, epoch=66, grad_norm=116.162, loss=267.3962, lr=7.21e-04]
Epoch 66 TRAIN:  10%|▉         | 13/136 [00:05<00:37,  3.29it/s, epoch=66, grad_norm=149.729, loss=277.1241, lr=7.21e-04]
Epoch 66 TRAIN:  12%|█▎        | 17/136 [00:05<00:29,  3.99it/s, epoch=66, grad_norm=149.729, loss=277.1241, lr=7.21e-04]
Epoch 66 TRAIN:  12%|█▎        | 17/136 [00:06<00:29,  3.99it/s, epoch=66, grad_norm=109.853, loss=270.4893, lr=7.21e-04]
Epoch 66 TRAIN:  12%|█▎        | 17/136 [00:06<00:29,  3.99it/s, epoch=66, grad_norm=114.942, loss=304.1346, lr=7.22e-04]
Epoch 66 TRAIN:  12%|█▎        | 17/136 [00:06<00:29,  3.99it/s, epoch=66, grad_norm=85.179, loss=317.4604, lr=7.22e-04] 
Epoch 66 TRAIN:  12%|█▎        | 17/136 [00:06<00:29,  3.99it/s, epoch=66, grad_norm=87.052, loss=293.4115, lr=7.22e-04]
Epoch 66 TRAIN:  15%|█▌        | 21/136 [00:06<00:27,  4.24it/s, epoch=66, grad_norm=87.052, loss=293.4115, lr=7.22e-04]
Epoch 66 TRAIN:  15%|█▌        | 21/136 [00:07<00:27,  4.24it/s, epoch=66, grad_norm=112.483, loss=300.5692, lr=7.22e-04]
Epoch 66 TRAIN:  15%|█▌        | 21/136 [00:07<00:27,  4.24it/s, epoch=66, grad_norm=97.166, loss=257.5567, lr=7.22e-04] 
Epoch 66 TRAIN:  15%|█▌        | 21/136 [00:07<00:27,  4.24it/s, epoch=66, grad_norm=132.288, loss=291.5187, lr=7.22e-04]
Epoch 66 TRAIN:  18%|█▊        | 24/136 [00:07<00:25,  4.35it/s, epoch=66, grad_norm=132.288, loss=291.5187, lr=7.22e-04]
Epoch 66 TRAIN:  18%|█▊        | 24/136 [00:07<00:25,  4.35it/s, epoch=66, grad_norm=113.097, loss=282.3997, lr=7.22e-04]
Epoch 66 TRAIN:  18%|█▊        | 24/136 [00:07<00:25,  4.35it/s, epoch=66, grad_norm=100.039, loss=266.8509, lr=7.22e-04]
Epoch 66 TRAIN:  18%|█▊        | 24/136 [00:07<00:25,  4.35it/s, epoch=66, grad_norm=81.888, loss=283.6302, lr=7.22e-04] 
Epoch 66 TRAIN:  18%|█▊        | 24/136 [00:08<00:25,  4.35it/s, epoch=66, grad_norm=104.467, loss=265.6201, lr=7.22e-04]
Epoch 66 TRAIN:  21%|██        | 28/136 [00:08<00:22,  4.75it/s, epoch=66, grad_norm=104.467, loss=265.6201, lr=7.22e-04]
Epoch 66 TRAIN:  21%|██        | 28/136 [00:08<00:22,  4.75it/s, epoch=66, grad_norm=130.337, loss=270.7289, lr=7.22e-04]
Epoch 66 TRAIN:  21%|██        | 28/136 [00:08<00:22,  4.75it/s, epoch=66, grad_norm=183.703, loss=342.2981, lr=7.22e-04]
Epoch 66 TRAIN:  21%|██        | 28/136 [00:08<00:22,  4.75it/s, epoch=66, grad_norm=178.486, loss=334.2823, lr=7.22e-04]
Epoch 66 TRAIN:  23%|██▎       | 31/136 [00:08<00:22,  4.73it/s, epoch=66, grad_norm=178.486, loss=334.2823, lr=7.22e-04]
Epoch 66 TRAIN:  23%|██▎       | 31/136 [00:08<00:22,  4.73it/s, epoch=66, grad_norm=87.396, loss=288.8097, lr=7.23e-04] 
Epoch 66 TRAIN:  23%|██▎       | 31/136 [00:09<00:22,  4.73it/s, epoch=66, grad_norm=89.510, loss=309.1810, lr=7.23e-04]
Epoch 66 TRAIN:  23%|██▎       | 31/136 [00:09<00:22,  4.73it/s, epoch=66, grad_norm=97.477, loss=297.4307, lr=7.23e-04]
Epoch 66 TRAIN:  25%|██▌       | 34/136 [00:09<00:21,  4.81it/s, epoch=66, grad_norm=97.477, loss=297.4307, lr=7.23e-04]
Epoch 66 TRAIN:  25%|██▌       | 34/136 [00:09<00:21,  4.81it/s, epoch=66, grad_norm=174.458, loss=252.6018, lr=7.23e-04]
Epoch 66 TRAIN:  25%|██▌       | 34/136 [00:09<00:21,  4.81it/s, epoch=66, grad_norm=97.981, loss=289.1075, lr=7.23e-04] 
Epoch 66 TRAIN:  25%|██▌       | 34/136 [00:09<00:21,  4.81it/s, epoch=66, grad_norm=106.003, loss=276.7230, lr=7.23e-04]
Epoch 66 TRAIN:  27%|██▋       | 37/136 [00:09<00:20,  4.88it/s, epoch=66, grad_norm=106.003, loss=276.7230, lr=7.23e-04]
Epoch 66 TRAIN:  27%|██▋       | 37/136 [00:10<00:20,  4.88it/s, epoch=66, grad_norm=89.766, loss=300.5624, lr=7.23e-04] 
Epoch 66 TRAIN:  27%|██▋       | 37/136 [00:10<00:20,  4.88it/s, epoch=66, grad_norm=93.196, loss=295.5843, lr=7.23e-04]
Epoch 66 TRAIN:  27%|██▋       | 37/136 [00:10<00:20,  4.88it/s, epoch=66, grad_norm=134.463, loss=301.8257, lr=7.23e-04]
Epoch 66 TRAIN:  29%|██▉       | 40/136 [00:10<00:19,  4.99it/s, epoch=66, grad_norm=134.463, loss=301.8257, lr=7.23e-04]
Epoch 66 TRAIN:  29%|██▉       | 40/136 [00:10<00:19,  4.99it/s, epoch=66, grad_norm=103.983, loss=261.8063, lr=7.23e-04]
Epoch 66 TRAIN:  29%|██▉       | 40/136 [00:10<00:19,  4.99it/s, epoch=66, grad_norm=146.150, loss=315.9197, lr=7.23e-04]
Epoch 66 TRAIN:  29%|██▉       | 40/136 [00:11<00:19,  4.99it/s, epoch=66, grad_norm=88.084, loss=273.7780, lr=7.23e-04] 
Epoch 66 TRAIN:  32%|███▏      | 43/136 [00:11<00:19,  4.84it/s, epoch=66, grad_norm=88.084, loss=273.7780, lr=7.23e-04]
Epoch 66 TRAIN:  32%|███▏      | 43/136 [00:11<00:19,  4.84it/s, epoch=66, grad_norm=101.898, loss=282.6995, lr=7.24e-04]
Epoch 66 TRAIN:  32%|███▏      | 43/136 [00:11<00:19,  4.84it/s, epoch=66, grad_norm=92.928, loss=261.9414, lr=7.24e-04] 
Epoch 66 TRAIN:  32%|███▏      | 43/136 [00:11<00:19,  4.84it/s, epoch=66, grad_norm=113.922, loss=315.2583, lr=7.24e-04]
Epoch 66 TRAIN:  34%|███▍      | 46/136 [00:11<00:18,  4.79it/s, epoch=66, grad_norm=113.922, loss=315.2583, lr=7.24e-04]
Epoch 66 TRAIN:  34%|███▍      | 46/136 [00:12<00:18,  4.79it/s, epoch=66, grad_norm=112.758, loss=305.2508, lr=7.24e-04]
Epoch 66 TRAIN:  34%|███▍      | 46/136 [00:12<00:18,  4.79it/s, epoch=66, grad_norm=210.861, loss=327.3110, lr=7.24e-04]
Epoch 66 TRAIN:  34%|███▍      | 46/136 [00:12<00:18,  4.79it/s, epoch=66, grad_norm=101.725, loss=258.6967, lr=7.24e-04]
Epoch 66 TRAIN:  36%|███▌      | 49/136 [00:12<00:19,  4.43it/s, epoch=66, grad_norm=101.725, loss=258.6967, lr=7.24e-04]
Epoch 66 TRAIN:  36%|███▌      | 49/136 [00:12<00:19,  4.43it/s, epoch=66, grad_norm=121.000, loss=248.8253, lr=7.24e-04]
Epoch 66 TRAIN:  36%|███▌      | 49/136 [00:12<00:19,  4.43it/s, epoch=66, grad_norm=110.106, loss=245.2984, lr=7.24e-04]
Epoch 66 TRAIN:  36%|███▌      | 49/136 [00:13<00:19,  4.43it/s, epoch=66, grad_norm=112.839, loss=291.2607, lr=7.24e-04]
Epoch 66 TRAIN:  38%|███▊      | 52/136 [00:13<00:17,  4.76it/s, epoch=66, grad_norm=112.839, loss=291.2607, lr=7.24e-04]
Epoch 66 TRAIN:  38%|███▊      | 52/136 [00:13<00:17,  4.76it/s, epoch=66, grad_norm=114.216, loss=305.5606, lr=7.24e-04]
Epoch 66 TRAIN:  38%|███▊      | 52/136 [00:13<00:17,  4.76it/s, epoch=66, grad_norm=96.719, loss=267.2531, lr=7.24e-04] 
Epoch 66 TRAIN:  38%|███▊      | 52/136 [00:13<00:17,  4.76it/s, epoch=66, grad_norm=141.665, loss=216.0031, lr=7.24e-04]
Epoch 66 TRAIN:  40%|████      | 55/136 [00:13<00:18,  4.46it/s, epoch=66, grad_norm=141.665, loss=216.0031, lr=7.24e-04]
Epoch 66 TRAIN:  40%|████      | 55/136 [00:14<00:18,  4.46it/s, epoch=66, grad_norm=171.474, loss=312.9489, lr=7.24e-04]
Epoch 66 TRAIN:  40%|████      | 55/136 [00:14<00:18,  4.46it/s, epoch=66, grad_norm=135.105, loss=313.5508, lr=7.25e-04]
Epoch 66 TRAIN:  40%|████      | 55/136 [00:14<00:18,  4.46it/s, epoch=66, grad_norm=127.797, loss=298.7954, lr=7.25e-04]
Epoch 66 TRAIN:  43%|████▎     | 58/136 [00:14<00:16,  4.61it/s, epoch=66, grad_norm=127.797, loss=298.7954, lr=7.25e-04]
Epoch 66 TRAIN:  43%|████▎     | 58/136 [00:14<00:16,  4.61it/s, epoch=66, grad_norm=108.837, loss=301.8083, lr=7.25e-04]
Epoch 66 TRAIN:  43%|████▎     | 58/136 [00:14<00:16,  4.61it/s, epoch=66, grad_norm=119.280, loss=271.9527, lr=7.25e-04]
Epoch 66 TRAIN:  43%|████▎     | 58/136 [00:15<00:16,  4.61it/s, epoch=66, grad_norm=109.560, loss=292.6222, lr=7.25e-04]
Epoch 66 TRAIN:  45%|████▍     | 61/136 [00:15<00:16,  4.66it/s, epoch=66, grad_norm=109.560, loss=292.6222, lr=7.25e-04]
Epoch 66 TRAIN:  45%|████▍     | 61/136 [00:15<00:16,  4.66it/s, epoch=66, grad_norm=104.770, loss=290.0386, lr=7.25e-04]
Epoch 66 TRAIN:  45%|████▍     | 61/136 [00:15<00:16,  4.66it/s, epoch=66, grad_norm=109.255, loss=303.8742, lr=7.25e-04]
Epoch 66 TRAIN:  45%|████▍     | 61/136 [00:15<00:16,  4.66it/s, epoch=66, grad_norm=115.892, loss=304.7967, lr=7.25e-04]
Epoch 66 TRAIN:  47%|████▋     | 64/136 [00:15<00:15,  4.59it/s, epoch=66, grad_norm=115.892, loss=304.7967, lr=7.25e-04]
Epoch 66 TRAIN:  47%|████▋     | 64/136 [00:16<00:15,  4.59it/s, epoch=66, grad_norm=120.567, loss=330.7881, lr=7.25e-04]
Epoch 66 TRAIN:  47%|████▋     | 64/136 [00:16<00:15,  4.59it/s, epoch=66, grad_norm=98.929, loss=309.9785, lr=7.25e-04] 
Epoch 66 TRAIN:  47%|████▋     | 64/136 [00:16<00:15,  4.59it/s, epoch=66, grad_norm=188.516, loss=252.3706, lr=7.25e-04]
Epoch 66 TRAIN:  49%|████▉     | 67/136 [00:16<00:15,  4.59it/s, epoch=66, grad_norm=188.516, loss=252.3706, lr=7.25e-04]
Epoch 66 TRAIN:  49%|████▉     | 67/136 [00:16<00:15,  4.59it/s, epoch=66, grad_norm=146.291, loss=305.9163, lr=7.25e-04]
Epoch 66 TRAIN:  49%|████▉     | 67/136 [00:16<00:15,  4.59it/s, epoch=66, grad_norm=125.883, loss=303.6427, lr=7.26e-04]
Epoch 66 TRAIN:  49%|████▉     | 67/136 [00:17<00:15,  4.59it/s, epoch=66, grad_norm=104.931, loss=287.9451, lr=7.26e-04]
Epoch 66 TRAIN:  51%|█████▏    | 70/136 [00:17<00:14,  4.66it/s, epoch=66, grad_norm=104.931, loss=287.9451, lr=7.26e-04]
Epoch 66 TRAIN:  51%|█████▏    | 70/136 [00:17<00:14,  4.66it/s, epoch=66, grad_norm=121.681, loss=268.4180, lr=7.26e-04]
Epoch 66 TRAIN:  51%|█████▏    | 70/136 [00:17<00:14,  4.66it/s, epoch=66, grad_norm=98.176, loss=254.7832, lr=7.26e-04] 
Epoch 66 TRAIN:  51%|█████▏    | 70/136 [00:17<00:14,  4.66it/s, epoch=66, grad_norm=120.007, loss=235.0045, lr=7.26e-04]
Epoch 66 TRAIN:  54%|█████▎    | 73/136 [00:17<00:13,  4.75it/s, epoch=66, grad_norm=120.007, loss=235.0045, lr=7.26e-04]
Epoch 66 TRAIN:  54%|█████▎    | 73/136 [00:17<00:13,  4.75it/s, epoch=66, grad_norm=111.240, loss=276.0607, lr=7.26e-04]
Epoch 66 TRAIN:  54%|█████▎    | 73/136 [00:18<00:13,  4.75it/s, epoch=66, grad_norm=143.366, loss=214.6976, lr=7.26e-04]
Epoch 66 TRAIN:  54%|█████▎    | 73/136 [00:18<00:13,  4.75it/s, epoch=66, grad_norm=140.770, loss=283.5355, lr=7.26e-04]
Epoch 66 TRAIN:  56%|█████▌    | 76/136 [00:18<00:12,  4.68it/s, epoch=66, grad_norm=140.770, loss=283.5355, lr=7.26e-04]
Epoch 66 TRAIN:  56%|█████▌    | 76/136 [00:18<00:12,  4.68it/s, epoch=66, grad_norm=185.159, loss=324.8098, lr=7.26e-04]
Epoch 66 TRAIN:  56%|█████▌    | 76/136 [00:18<00:12,  4.68it/s, epoch=66, grad_norm=89.103, loss=246.8958, lr=7.26e-04] 
Epoch 66 TRAIN:  56%|█████▌    | 76/136 [00:18<00:12,  4.68it/s, epoch=66, grad_norm=164.944, loss=297.6901, lr=7.26e-04]
Epoch 66 TRAIN:  58%|█████▊    | 79/136 [00:18<00:12,  4.70it/s, epoch=66, grad_norm=164.944, loss=297.6901, lr=7.26e-04]
Epoch 66 TRAIN:  58%|█████▊    | 79/136 [00:19<00:12,  4.70it/s, epoch=66, grad_norm=169.309, loss=300.1667, lr=7.26e-04]
Epoch 66 TRAIN:  58%|█████▊    | 79/136 [00:19<00:12,  4.70it/s, epoch=66, grad_norm=143.071, loss=285.0553, lr=7.26e-04]
Epoch 66 TRAIN:  58%|█████▊    | 79/136 [00:19<00:12,  4.70it/s, epoch=66, grad_norm=118.225, loss=279.9713, lr=7.27e-04]
Epoch 66 TRAIN:  60%|██████    | 82/136 [00:19<00:12,  4.37it/s, epoch=66, grad_norm=118.225, loss=279.9713, lr=7.27e-04]
Epoch 66 TRAIN:  60%|██████    | 82/136 [00:19<00:12,  4.37it/s, epoch=66, grad_norm=122.609, loss=292.8513, lr=7.27e-04]
Epoch 66 TRAIN:  60%|██████    | 82/136 [00:20<00:12,  4.37it/s, epoch=66, grad_norm=108.949, loss=276.6266, lr=7.27e-04]
Epoch 66 TRAIN:  60%|██████    | 82/136 [00:20<00:12,  4.37it/s, epoch=66, grad_norm=119.953, loss=292.6760, lr=7.27e-04]
Epoch 66 TRAIN:  62%|██████▎   | 85/136 [00:20<00:11,  4.52it/s, epoch=66, grad_norm=119.953, loss=292.6760, lr=7.27e-04]
Epoch 66 TRAIN:  62%|██████▎   | 85/136 [00:20<00:11,  4.52it/s, epoch=66, grad_norm=91.750, loss=297.4662, lr=7.27e-04] 
Epoch 66 TRAIN:  62%|██████▎   | 85/136 [00:20<00:11,  4.52it/s, epoch=66, grad_norm=262.222, loss=257.3231, lr=7.27e-04]
Epoch 66 TRAIN:  62%|██████▎   | 85/136 [00:20<00:11,  4.52it/s, epoch=66, grad_norm=121.025, loss=296.4213, lr=7.27e-04]
Epoch 66 TRAIN:  65%|██████▍   | 88/136 [00:20<00:09,  4.83it/s, epoch=66, grad_norm=121.025, loss=296.4213, lr=7.27e-04]
Epoch 66 TRAIN:  65%|██████▍   | 88/136 [00:21<00:09,  4.83it/s, epoch=66, grad_norm=123.265, loss=300.2303, lr=7.27e-04]
Epoch 66 TRAIN:  65%|██████▍   | 88/136 [00:21<00:09,  4.83it/s, epoch=66, grad_norm=158.445, loss=337.9225, lr=7.27e-04]
Epoch 66 TRAIN:  65%|██████▍   | 88/136 [00:21<00:09,  4.83it/s, epoch=66, grad_norm=134.046, loss=290.7237, lr=7.27e-04]
Epoch 66 TRAIN:  67%|██████▋   | 91/136 [00:21<00:09,  4.59it/s, epoch=66, grad_norm=134.046, loss=290.7237, lr=7.27e-04]
Epoch 66 TRAIN:  67%|██████▋   | 91/136 [00:21<00:09,  4.59it/s, epoch=66, grad_norm=165.124, loss=240.1558, lr=7.27e-04]
Epoch 66 TRAIN:  67%|██████▋   | 91/136 [00:22<00:09,  4.59it/s, epoch=66, grad_norm=206.822, loss=236.3483, lr=7.27e-04]
Epoch 66 TRAIN:  67%|██████▋   | 91/136 [00:22<00:09,  4.59it/s, epoch=66, grad_norm=90.385, loss=262.7128, lr=7.28e-04] 
Epoch 66 TRAIN:  69%|██████▉   | 94/136 [00:22<00:08,  4.67it/s, epoch=66, grad_norm=90.385, loss=262.7128, lr=7.28e-04]
Epoch 66 TRAIN:  69%|██████▉   | 94/136 [00:22<00:08,  4.67it/s, epoch=66, grad_norm=180.203, loss=267.6111, lr=7.28e-04]
Epoch 66 TRAIN:  69%|██████▉   | 94/136 [00:22<00:08,  4.67it/s, epoch=66, grad_norm=144.958, loss=162.9408, lr=7.28e-04]
Epoch 66 TRAIN:  69%|██████▉   | 94/136 [00:22<00:08,  4.67it/s, epoch=66, grad_norm=154.138, loss=265.3078, lr=7.28e-04]
Epoch 66 TRAIN:  71%|███████▏  | 97/136 [00:22<00:08,  4.53it/s, epoch=66, grad_norm=154.138, loss=265.3078, lr=7.28e-04]
Epoch 66 TRAIN:  71%|███████▏  | 97/136 [00:23<00:08,  4.53it/s, epoch=66, grad_norm=150.242, loss=254.3407, lr=7.28e-04]
Epoch 66 TRAIN:  71%|███████▏  | 97/136 [00:23<00:08,  4.53it/s, epoch=66, grad_norm=116.541, loss=232.5383, lr=7.28e-04]
Epoch 66 TRAIN:  71%|███████▏  | 97/136 [00:23<00:08,  4.53it/s, epoch=66, grad_norm=223.194, loss=284.7500, lr=7.28e-04]2026-04-16 20:26:01,775 DEBUG TRAIN | steps/sec 4.241| Batch 66/100 loss 284.749969 loss_att 236.102631 loss_ctc 398.260468 th_accuracy 0.357664 lr 7.2800e-04 grad_norm 223.194321 rank 0

Epoch 66 TRAIN:  74%|███████▎  | 100/136 [00:23<00:07,  4.59it/s, epoch=66, grad_norm=223.194, loss=284.7500, lr=7.28e-04]
Epoch 66 TRAIN:  74%|███████▎  | 100/136 [00:23<00:07,  4.59it/s, epoch=66, grad_norm=147.681, loss=230.9058, lr=7.28e-04]
Epoch 66 TRAIN:  74%|███████▎  | 100/136 [00:23<00:07,  4.59it/s, epoch=66, grad_norm=188.976, loss=261.5377, lr=7.28e-04]
Epoch 66 TRAIN:  74%|███████▎  | 100/136 [00:24<00:07,  4.59it/s, epoch=66, grad_norm=154.455, loss=308.5709, lr=7.28e-04]
Epoch 66 TRAIN:  76%|███████▌  | 103/136 [00:24<00:07,  4.59it/s, epoch=66, grad_norm=154.455, loss=308.5709, lr=7.28e-04]
Epoch 66 TRAIN:  76%|███████▌  | 103/136 [00:24<00:07,  4.59it/s, epoch=66, grad_norm=115.796, loss=242.3564, lr=7.28e-04]
Epoch 66 TRAIN:  76%|███████▌  | 103/136 [00:24<00:07,  4.59it/s, epoch=66, grad_norm=108.951, loss=241.8970, lr=7.28e-04]
Epoch 66 TRAIN:  76%|███████▌  | 103/136 [00:24<00:07,  4.59it/s, epoch=66, grad_norm=160.421, loss=189.3606, lr=7.28e-04]
Epoch 66 TRAIN:  78%|███████▊  | 106/136 [00:24<00:06,  4.65it/s, epoch=66, grad_norm=160.421, loss=189.3606, lr=7.28e-04]
Epoch 66 TRAIN:  78%|███████▊  | 106/136 [00:25<00:06,  4.65it/s, epoch=66, grad_norm=144.327, loss=258.5267, lr=7.29e-04]
Epoch 66 TRAIN:  78%|███████▊  | 106/136 [00:25<00:06,  4.65it/s, epoch=66, grad_norm=176.978, loss=230.5464, lr=7.29e-04]
Epoch 66 TRAIN:  78%|███████▊  | 106/136 [00:25<00:06,  4.65it/s, epoch=66, grad_norm=163.255, loss=261.8329, lr=7.29e-04]
Epoch 66 TRAIN:  80%|████████  | 109/136 [00:25<00:05,  4.58it/s, epoch=66, grad_norm=163.255, loss=261.8329, lr=7.29e-04]
Epoch 66 TRAIN:  80%|████████  | 109/136 [00:25<00:05,  4.58it/s, epoch=66, grad_norm=180.247, loss=291.6948, lr=7.29e-04]
Epoch 66 TRAIN:  80%|████████  | 109/136 [00:26<00:05,  4.58it/s, epoch=66, grad_norm=105.789, loss=292.3247, lr=7.29e-04]
Epoch 66 TRAIN:  80%|████████  | 109/136 [00:26<00:05,  4.58it/s, epoch=66, grad_norm=175.921, loss=278.5473, lr=7.29e-04]
Epoch 66 TRAIN:  82%|████████▏ | 112/136 [00:26<00:05,  4.51it/s, epoch=66, grad_norm=175.921, loss=278.5473, lr=7.29e-04]
Epoch 66 TRAIN:  82%|████████▏ | 112/136 [00:26<00:05,  4.51it/s, epoch=66, grad_norm=151.248, loss=278.5031, lr=7.29e-04]
Epoch 66 TRAIN:  82%|████████▏ | 112/136 [00:26<00:05,  4.51it/s, epoch=66, grad_norm=139.501, loss=289.2537, lr=7.29e-04]
Epoch 66 TRAIN:  82%|████████▏ | 112/136 [00:26<00:05,  4.51it/s, epoch=66, grad_norm=166.219, loss=272.8156, lr=7.29e-04]
Epoch 66 TRAIN:  85%|████████▍ | 115/136 [00:26<00:04,  4.56it/s, epoch=66, grad_norm=166.219, loss=272.8156, lr=7.29e-04]
Epoch 66 TRAIN:  85%|████████▍ | 115/136 [00:27<00:04,  4.56it/s, epoch=66, grad_norm=138.004, loss=294.2758, lr=7.29e-04]
Epoch 66 TRAIN:  85%|████████▍ | 115/136 [00:27<00:04,  4.56it/s, epoch=66, grad_norm=265.621, loss=375.9703, lr=7.29e-04]
Epoch 66 TRAIN:  85%|████████▍ | 115/136 [00:27<00:04,  4.56it/s, epoch=66, grad_norm=126.674, loss=267.9664, lr=7.29e-04]
Epoch 66 TRAIN:  87%|████████▋ | 118/136 [00:27<00:03,  4.59it/s, epoch=66, grad_norm=126.674, loss=267.9664, lr=7.29e-04]
Epoch 66 TRAIN:  87%|████████▋ | 118/136 [00:27<00:03,  4.59it/s, epoch=66, grad_norm=190.671, loss=282.5221, lr=7.30e-04]
Epoch 66 TRAIN:  87%|████████▋ | 118/136 [00:27<00:03,  4.59it/s, epoch=66, grad_norm=179.167, loss=246.3703, lr=7.30e-04]
Epoch 66 TRAIN:  87%|████████▋ | 118/136 [00:28<00:03,  4.59it/s, epoch=66, grad_norm=153.922, loss=330.6919, lr=7.30e-04]
Epoch 66 TRAIN:  89%|████████▉ | 121/136 [00:28<00:03,  4.65it/s, epoch=66, grad_norm=153.922, loss=330.6919, lr=7.30e-04]
Epoch 66 TRAIN:  89%|████████▉ | 121/136 [00:28<00:03,  4.65it/s, epoch=66, grad_norm=134.841, loss=271.6826, lr=7.30e-04]
Epoch 66 TRAIN:  89%|████████▉ | 121/136 [00:28<00:03,  4.65it/s, epoch=66, grad_norm=231.393, loss=297.5959, lr=7.30e-04]
Epoch 66 TRAIN:  89%|████████▉ | 121/136 [00:28<00:03,  4.65it/s, epoch=66, grad_norm=107.767, loss=256.6599, lr=7.30e-04]
Epoch 66 TRAIN:  91%|█████████ | 124/136 [00:28<00:02,  4.36it/s, epoch=66, grad_norm=107.767, loss=256.6599, lr=7.30e-04]
Epoch 66 TRAIN:  91%|█████████ | 124/136 [00:29<00:02,  4.36it/s, epoch=66, grad_norm=387.467, loss=366.4464, lr=7.30e-04]
Epoch 66 TRAIN:  91%|█████████ | 124/136 [00:29<00:02,  4.36it/s, epoch=66, grad_norm=132.693, loss=221.8160, lr=7.30e-04]
Epoch 66 TRAIN:  91%|█████████ | 124/136 [00:29<00:02,  4.36it/s, epoch=66, grad_norm=159.702, loss=300.2006, lr=7.30e-04]
Epoch 66 TRAIN:  93%|█████████▎| 127/136 [00:29<00:01,  4.63it/s, epoch=66, grad_norm=159.702, loss=300.2006, lr=7.30e-04]
Epoch 66 TRAIN:  93%|█████████▎| 127/136 [00:29<00:01,  4.63it/s, epoch=66, grad_norm=243.447, loss=331.0178, lr=7.30e-04]
Epoch 66 TRAIN:  93%|█████████▎| 127/136 [00:29<00:01,  4.63it/s, epoch=66, grad_norm=220.942, loss=252.1245, lr=7.30e-04]
Epoch 66 TRAIN:  93%|█████████▎| 127/136 [00:30<00:01,  4.63it/s, epoch=66, grad_norm=165.599, loss=283.4983, lr=7.30e-04]
Epoch 66 TRAIN:  96%|█████████▌| 130/136 [00:30<00:01,  4.59it/s, epoch=66, grad_norm=165.599, loss=283.4983, lr=7.30e-04]
Epoch 66 TRAIN:  96%|█████████▌| 130/136 [00:30<00:01,  4.59it/s, epoch=66, grad_norm=183.594, loss=303.0954, lr=7.30e-04]
Epoch 66 TRAIN:  96%|█████████▌| 130/136 [00:30<00:01,  4.59it/s, epoch=66, grad_norm=176.768, loss=327.9854, lr=7.31e-04]
Epoch 66 TRAIN:  96%|█████████▌| 130/136 [00:30<00:01,  4.59it/s, epoch=66, grad_norm=161.287, loss=262.6563, lr=7.31e-04]
Epoch 66 TRAIN:  98%|█████████▊| 133/136 [00:30<00:00,  4.69it/s, epoch=66, grad_norm=161.287, loss=262.6563, lr=7.31e-04]
Epoch 66 TRAIN:  98%|█████████▊| 133/136 [00:30<00:00,  4.69it/s, epoch=66, grad_norm=199.498, loss=347.2524, lr=7.31e-04]
Epoch 66 TRAIN:  98%|█████████▊| 133/136 [00:31<00:00,  4.69it/s, epoch=66, grad_norm=270.279, loss=363.0338, lr=7.31e-04]
Epoch 66 TRAIN:  99%|█████████▉| 135/136 [00:31<00:00,  4.34it/s, epoch=66, grad_norm=270.279, loss=363.0338, lr=7.31e-04]

CV:   0%|          | 0/6 [00:00<?, ?it/s]/home/rookie/Documents/batuka/projects/wenet/wenet/wenet/utils/train_utils.py:689: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  amp_autocast(enabled=dtype is not None,

CV:  17%|█▋        | 1/6 [00:00<00:03,  1.31it/s]
                                                 
2026-04-16 20:26:10,167 INFO Epoch 66 Step 9135 CV info lr 7.3080e-04 cv_loss 440.3524364124645 rank 0 acc 0.36835867166519165
2026-04-16 20:26:10,169 INFO [Rank 0] Checkpoint: save to checkpoint exp/conformer_mongolian/epoch_66.pt
2026-04-16 20:26:10,291 INFO Epoch 67 Step 9135 TRAIN info lr 7.3080e-04 rank 0
2026-04-16 20:26:10,304 INFO using accumulate grad, new batch size is 1 times larger than before

Epoch 67 TRAIN:   0%|          | 0/135 [00:00<?, ?it/s]
Epoch 67 TRAIN:   0%|          | 0/135 [00:05<?, ?it/s, epoch=67, grad_norm=248.773, loss=256.1070, lr=7.31e-04]
Epoch 67 TRAIN:   1%|          | 1/135 [00:05<11:15,  5.04s/it, epoch=67, grad_norm=248.773, loss=256.1070, lr=7.31e-04]
Epoch 67 TRAIN:   1%|          | 1/135 [00:05<11:15,  5.04s/it, epoch=67, grad_norm=344.873, loss=297.2695, lr=7.31e-04]
Epoch 67 TRAIN:   1%|          | 1/135 [00:05<11:15,  5.04s/it, epoch=67, grad_norm=390.625, loss=293.8961, lr=7.31e-04]
Epoch 67 TRAIN:   1%|          | 1/135 [00:05<11:15,  5.04s/it, epoch=67, grad_norm=307.702, loss=328.5096, lr=7.31e-04]
Epoch 67 TRAIN:   1%|          | 1/135 [00:05<11:15,  5.04s/it, epoch=67, grad_norm=413.312, loss=293.6743, lr=7.31e-04]
Epoch 67 TRAIN:   1%|          | 1/135 [00:05<11:15,  5.04s/it, epoch=67, grad_norm=163.166, loss=268.9632, lr=7.31e-04]
Epoch 67 TRAIN:   1%|          | 1/135 [00:05<11:15,  5.04s/it, epoch=67, grad_norm=307.824, loss=343.2438, lr=7.31e-04]
Epoch 67 TRAIN:   5%|▌         | 7/135 [00:05<01:18,  1.62it/s, epoch=67, grad_norm=307.824, loss=343.2438, lr=7.31e-04]
Epoch 67 TRAIN:   5%|▌         | 7/135 [00:05<01:18,  1.62it/s, epoch=67, grad_norm=172.835, loss=299.5512, lr=7.31e-04]
Epoch 67 TRAIN:   5%|▌         | 7/135 [00:05<01:18,  1.62it/s, epoch=67, grad_norm=115.876, loss=301.6433, lr=7.32e-04]
Epoch 67 TRAIN:   5%|▌         | 7/135 [00:06<01:18,  1.62it/s, epoch=67, grad_norm=118.653, loss=271.5771, lr=7.32e-04]
Epoch 67 TRAIN:   5%|▌         | 7/135 [00:06<01:18,  1.62it/s, epoch=67, grad_norm=151.542, loss=295.1955, lr=7.32e-04]
Epoch 67 TRAIN:   8%|▊         | 11/135 [00:06<00:50,  2.48it/s, epoch=67, grad_norm=151.542, loss=295.1955, lr=7.32e-04]
Epoch 67 TRAIN:   8%|▊         | 11/135 [00:06<00:50,  2.48it/s, epoch=67, grad_norm=153.237, loss=324.0298, lr=7.32e-04]
Epoch 67 TRAIN:   8%|▊         | 11/135 [00:06<00:50,  2.48it/s, epoch=67, grad_norm=206.263, loss=273.0144, lr=7.32e-04]
Epoch 67 TRAIN:   8%|▊         | 11/135 [00:06<00:50,  2.48it/s, epoch=67, grad_norm=131.901, loss=258.9048, lr=7.32e-04]
Epoch 67 TRAIN:   8%|▊         | 11/135 [00:07<00:50,  2.48it/s, epoch=67, grad_norm=106.034, loss=259.6888, lr=7.32e-04]
Epoch 67 TRAIN:  11%|█         | 15/135 [00:07<00:38,  3.12it/s, epoch=67, grad_norm=106.034, loss=259.6888, lr=7.32e-04]
Epoch 67 TRAIN:  11%|█         | 15/135 [00:07<00:38,  3.12it/s, epoch=67, grad_norm=102.225, loss=334.2830, lr=7.32e-04]
Epoch 67 TRAIN:  11%|█         | 15/135 [00:07<00:38,  3.12it/s, epoch=67, grad_norm=128.531, loss=270.1395, lr=7.32e-04]
Epoch 67 TRAIN:  11%|█         | 15/135 [00:07<00:38,  3.12it/s, epoch=67, grad_norm=115.735, loss=301.1487, lr=7.32e-04]
Epoch 67 TRAIN:  13%|█▎        | 18/135 [00:07<00:33,  3.51it/s, epoch=67, grad_norm=115.735, loss=301.1487, lr=7.32e-04]
Epoch 67 TRAIN:  13%|█▎        | 18/135 [00:07<00:33,  3.51it/s, epoch=67, grad_norm=96.888, loss=253.8329, lr=7.32e-04] 
Epoch 67 TRAIN:  13%|█▎        | 18/135 [00:08<00:33,  3.51it/s, epoch=67, grad_norm=95.482, loss=255.3473, lr=7.32e-04]
Epoch 67 TRAIN:  13%|█▎        | 18/135 [00:08<00:33,  3.51it/s, epoch=67, grad_norm=217.411, loss=325.0052, lr=7.32e-04]
Epoch 67 TRAIN:  16%|█▌        | 21/135 [00:08<00:28,  3.95it/s, epoch=67, grad_norm=217.411, loss=325.0052, lr=7.32e-04]
Epoch 67 TRAIN:  16%|█▌        | 21/135 [00:08<00:28,  3.95it/s, epoch=67, grad_norm=121.562, loss=291.2031, lr=7.33e-04]
Epoch 67 TRAIN:  16%|█▌        | 21/135 [00:08<00:28,  3.95it/s, epoch=67, grad_norm=202.414, loss=328.7287, lr=7.33e-04]
Epoch 67 TRAIN:  16%|█▌        | 21/135 [00:08<00:28,  3.95it/s, epoch=67, grad_norm=180.149, loss=332.3984, lr=7.33e-04]
Epoch 67 TRAIN:  18%|█▊        | 24/135 [00:08<00:27,  3.97it/s, epoch=67, grad_norm=180.149, loss=332.3984, lr=7.33e-04]
Epoch 67 TRAIN:  18%|█▊        | 24/135 [00:09<00:27,  3.97it/s, epoch=67, grad_norm=125.390, loss=244.1678, lr=7.33e-04]
Epoch 67 TRAIN:  18%|█▊        | 24/135 [00:09<00:27,  3.97it/s, epoch=67, grad_norm=143.105, loss=265.3817, lr=7.33e-04]
Epoch 67 TRAIN:  18%|█▊        | 24/135 [00:09<00:27,  3.97it/s, epoch=67, grad_norm=84.477, loss=279.4049, lr=7.33e-04] 
Epoch 67 TRAIN:  20%|██        | 27/135 [00:09<00:25,  4.23it/s, epoch=67, grad_norm=84.477, loss=279.4049, lr=7.33e-04]
Epoch 67 TRAIN:  20%|██        | 27/135 [00:09<00:25,  4.23it/s, epoch=67, grad_norm=77.639, loss=287.2079, lr=7.33e-04]
Epoch 67 TRAIN:  20%|██        | 27/135 [00:09<00:25,  4.23it/s, epoch=67, grad_norm=112.918, loss=313.7934, lr=7.33e-04]
Epoch 67 TRAIN:  20%|██        | 27/135 [00:10<00:25,  4.23it/s, epoch=67, grad_norm=97.246, loss=266.0288, lr=7.33e-04] 
Epoch 67 TRAIN:  22%|██▏       | 30/135 [00:10<00:23,  4.44it/s, epoch=67, grad_norm=97.246, loss=266.0288, lr=7.33e-04]
Epoch 67 TRAIN:  22%|██▏       | 30/135 [00:10<00:23,  4.44it/s, epoch=67, grad_norm=95.162, loss=275.0224, lr=7.33e-04]
Epoch 67 TRAIN:  22%|██▏       | 30/135 [00:10<00:23,  4.44it/s, epoch=67, grad_norm=114.863, loss=272.5742, lr=7.33e-04]
Epoch 67 TRAIN:  22%|██▏       | 30/135 [00:10<00:23,  4.44it/s, epoch=67, grad_norm=86.963, loss=266.2481, lr=7.33e-04] 
Epoch 67 TRAIN:  24%|██▍       | 33/135 [00:10<00:22,  4.55it/s, epoch=67, grad_norm=86.963, loss=266.2481, lr=7.33e-04]
Epoch 67 TRAIN:  24%|██▍       | 33/135 [00:10<00:22,  4.55it/s, epoch=67, grad_norm=102.662, loss=263.6239, lr=7.34e-04]
Epoch 67 TRAIN:  24%|██▍       | 33/135 [00:11<00:22,  4.55it/s, epoch=67, grad_norm=187.481, loss=344.5413, lr=7.34e-04]
Epoch 67 TRAIN:  24%|██▍       | 33/135 [00:11<00:22,  4.55it/s, epoch=67, grad_norm=75.926, loss=273.8218, lr=7.34e-04] 
Epoch 67 TRAIN:  27%|██▋       | 36/135 [00:11<00:21,  4.66it/s, epoch=67, grad_norm=75.926, loss=273.8218, lr=7.34e-04]
Epoch 67 TRAIN:  27%|██▋       | 36/135 [00:11<00:21,  4.66it/s, epoch=67, grad_norm=154.064, loss=312.5163, lr=7.34e-04]
Epoch 67 TRAIN:  27%|██▋       | 36/135 [00:11<00:21,  4.66it/s, epoch=67, grad_norm=104.902, loss=280.4484, lr=7.34e-04]
Epoch 67 TRAIN:  27%|██▋       | 36/135 [00:11<00:21,  4.66it/s, epoch=67, grad_norm=78.749, loss=241.8130, lr=7.34e-04] 
Epoch 67 TRAIN:  29%|██▉       | 39/135 [00:11<00:20,  4.71it/s, epoch=67, grad_norm=78.749, loss=241.8130, lr=7.34e-04]
Epoch 67 TRAIN:  29%|██▉       | 39/135 [00:12<00:20,  4.71it/s, epoch=67, grad_norm=74.267, loss=275.5020, lr=7.34e-04]
Epoch 67 TRAIN:  29%|██▉       | 39/135 [00:12<00:20,  4.71it/s, epoch=67, grad_norm=111.837, loss=249.5374, lr=7.34e-04]
Epoch 67 TRAIN:  29%|██▉       | 39/135 [00:12<00:20,  4.71it/s, epoch=67, grad_norm=214.898, loss=349.5992, lr=7.34e-04]
Epoch 67 TRAIN:  31%|███       | 42/135 [00:12<00:19,  4.72it/s, epoch=67, grad_norm=214.898, loss=349.5992, lr=7.34e-04]
Epoch 67 TRAIN:  31%|███       | 42/135 [00:12<00:19,  4.72it/s, epoch=67, grad_norm=119.630, loss=245.6812, lr=7.34e-04]
Epoch 67 TRAIN:  31%|███       | 42/135 [00:13<00:19,  4.72it/s, epoch=67, grad_norm=97.672, loss=277.8581, lr=7.34e-04] 
Epoch 67 TRAIN:  31%|███       | 42/135 [00:13<00:19,  4.72it/s, epoch=67, grad_norm=79.418, loss=269.8079, lr=7.34e-04]
Epoch 67 TRAIN:  33%|███▎      | 45/135 [00:13<00:19,  4.68it/s, epoch=67, grad_norm=79.418, loss=269.8079, lr=7.34e-04]
Epoch 67 TRAIN:  33%|███▎      | 45/135 [00:13<00:19,  4.68it/s, epoch=67, grad_norm=126.459, loss=304.7714, lr=7.34e-04]
Epoch 67 TRAIN:  33%|███▎      | 45/135 [00:13<00:19,  4.68it/s, epoch=67, grad_norm=142.089, loss=283.6994, lr=7.35e-04]
Epoch 67 TRAIN:  33%|███▎      | 45/135 [00:13<00:19,  4.68it/s, epoch=67, grad_norm=121.809, loss=312.3256, lr=7.35e-04]
Epoch 67 TRAIN:  36%|███▌      | 48/135 [00:13<00:18,  4.66it/s, epoch=67, grad_norm=121.809, loss=312.3256, lr=7.35e-04]
Epoch 67 TRAIN:  36%|███▌      | 48/135 [00:14<00:18,  4.66it/s, epoch=67, grad_norm=130.374, loss=325.4987, lr=7.35e-04]
Epoch 67 TRAIN:  36%|███▌      | 48/135 [00:14<00:18,  4.66it/s, epoch=67, grad_norm=164.367, loss=234.4267, lr=7.35e-04]
Epoch 67 TRAIN:  36%|███▌      | 48/135 [00:14<00:18,  4.66it/s, epoch=67, grad_norm=107.235, loss=315.9738, lr=7.35e-04]
Epoch 67 TRAIN:  38%|███▊      | 51/135 [00:14<00:18,  4.63it/s, epoch=67, grad_norm=107.235, loss=315.9738, lr=7.35e-04]
Epoch 67 TRAIN:  38%|███▊      | 51/135 [00:14<00:18,  4.63it/s, epoch=67, grad_norm=119.685, loss=280.9350, lr=7.35e-04]
Epoch 67 TRAIN:  38%|███▊      | 51/135 [00:14<00:18,  4.63it/s, epoch=67, grad_norm=120.022, loss=278.9960, lr=7.35e-04]
Epoch 67 TRAIN:  38%|███▊      | 51/135 [00:15<00:18,  4.63it/s, epoch=67, grad_norm=116.901, loss=288.6862, lr=7.35e-04]
Epoch 67 TRAIN:  40%|████      | 54/135 [00:15<00:17,  4.68it/s, epoch=67, grad_norm=116.901, loss=288.6862, lr=7.35e-04]
Epoch 67 TRAIN:  40%|████      | 54/135 [00:15<00:17,  4.68it/s, epoch=67, grad_norm=103.820, loss=281.8173, lr=7.35e-04]
Epoch 67 TRAIN:  40%|████      | 54/135 [00:15<00:17,  4.68it/s, epoch=67, grad_norm=127.692, loss=308.9522, lr=7.35e-04]
Epoch 67 TRAIN:  40%|████      | 54/135 [00:15<00:17,  4.68it/s, epoch=67, grad_norm=94.969, loss=283.1804, lr=7.35e-04] 
Epoch 67 TRAIN:  42%|████▏     | 57/135 [00:15<00:16,  4.70it/s, epoch=67, grad_norm=94.969, loss=283.1804, lr=7.35e-04]
Epoch 67 TRAIN:  42%|████▏     | 57/135 [00:16<00:16,  4.70it/s, epoch=67, grad_norm=103.953, loss=293.8306, lr=7.35e-04]
Epoch 67 TRAIN:  42%|████▏     | 57/135 [00:16<00:16,  4.70it/s, epoch=67, grad_norm=87.793, loss=296.2741, lr=7.36e-04] 
Epoch 67 TRAIN:  42%|████▏     | 57/135 [00:16<00:16,  4.70it/s, epoch=67, grad_norm=117.035, loss=275.1522, lr=7.36e-04]
Epoch 67 TRAIN:  44%|████▍     | 60/135 [00:16<00:15,  4.77it/s, epoch=67, grad_norm=117.035, loss=275.1522, lr=7.36e-04]
Epoch 67 TRAIN:  44%|████▍     | 60/135 [00:16<00:15,  4.77it/s, epoch=67, grad_norm=119.392, loss=247.5037, lr=7.36e-04]
Epoch 67 TRAIN:  44%|████▍     | 60/135 [00:16<00:15,  4.77it/s, epoch=67, grad_norm=84.085, loss=266.3807, lr=7.36e-04] 
Epoch 67 TRAIN:  44%|████▍     | 60/135 [00:17<00:15,  4.77it/s, epoch=67, grad_norm=89.436, loss=301.3483, lr=7.36e-04]
Epoch 67 TRAIN:  47%|████▋     | 63/135 [00:17<00:16,  4.49it/s, epoch=67, grad_norm=89.436, loss=301.3483, lr=7.36e-04]
Epoch 67 TRAIN:  47%|████▋     | 63/135 [00:17<00:16,  4.49it/s, epoch=67, grad_norm=130.831, loss=233.6915, lr=7.36e-04]
Epoch 67 TRAIN:  47%|████▋     | 63/135 [00:17<00:16,  4.49it/s, epoch=67, grad_norm=173.502, loss=314.0869, lr=7.36e-04]
Epoch 67 TRAIN:  47%|████▋     | 63/135 [00:17<00:16,  4.49it/s, epoch=67, grad_norm=85.479, loss=271.5333, lr=7.36e-04] 
Epoch 67 TRAIN:  49%|████▉     | 66/135 [00:17<00:15,  4.53it/s, epoch=67, grad_norm=85.479, loss=271.5333, lr=7.36e-04]
Epoch 67 TRAIN:  49%|████▉     | 66/135 [00:18<00:15,  4.53it/s, epoch=67, grad_norm=87.315, loss=267.6306, lr=7.36e-04]
Epoch 67 TRAIN:  49%|████▉     | 66/135 [00:18<00:15,  4.53it/s, epoch=67, grad_norm=172.142, loss=215.8106, lr=7.36e-04]
Epoch 67 TRAIN:  49%|████▉     | 66/135 [00:18<00:15,  4.53it/s, epoch=67, grad_norm=118.918, loss=256.6308, lr=7.36e-04]
Epoch 67 TRAIN:  51%|█████     | 69/135 [00:18<00:14,  4.60it/s, epoch=67, grad_norm=118.918, loss=256.6308, lr=7.36e-04]
Epoch 67 TRAIN:  51%|█████     | 69/135 [00:18<00:14,  4.60it/s, epoch=67, grad_norm=232.863, loss=348.2000, lr=7.36e-04]
Epoch 67 TRAIN:  51%|█████     | 69/135 [00:18<00:14,  4.60it/s, epoch=67, grad_norm=105.861, loss=271.4037, lr=7.36e-04]
Epoch 67 TRAIN:  51%|█████     | 69/135 [00:19<00:14,  4.60it/s, epoch=67, grad_norm=109.362, loss=246.3407, lr=7.37e-04]
Epoch 67 TRAIN:  53%|█████▎    | 72/135 [00:19<00:14,  4.47it/s, epoch=67, grad_norm=109.362, loss=246.3407, lr=7.37e-04]
Epoch 67 TRAIN:  53%|█████▎    | 72/135 [00:19<00:14,  4.47it/s, epoch=67, grad_norm=142.989, loss=288.2217, lr=7.37e-04]
Epoch 67 TRAIN:  53%|█████▎    | 72/135 [00:19<00:14,  4.47it/s, epoch=67, grad_norm=115.548, loss=219.6302, lr=7.37e-04]
Epoch 67 TRAIN:  53%|█████▎    | 72/135 [00:19<00:14,  4.47it/s, epoch=67, grad_norm=95.131, loss=260.2870, lr=7.37e-04] 
Epoch 67 TRAIN:  56%|█████▌    | 75/135 [00:19<00:12,  4.63it/s, epoch=67, grad_norm=95.131, loss=260.2870, lr=7.37e-04]
Epoch 67 TRAIN:  56%|█████▌    | 75/135 [00:19<00:12,  4.63it/s, epoch=67, grad_norm=234.271, loss=329.2535, lr=7.37e-04]
Epoch 67 TRAIN:  56%|█████▌    | 75/135 [00:20<00:12,  4.63it/s, epoch=67, grad_norm=147.323, loss=288.9019, lr=7.37e-04]
Epoch 67 TRAIN:  56%|█████▌    | 75/135 [00:20<00:12,  4.63it/s, epoch=67, grad_norm=184.057, loss=334.7761, lr=7.37e-04]
Epoch 67 TRAIN:  58%|█████▊    | 78/135 [00:20<00:12,  4.56it/s, epoch=67, grad_norm=184.057, loss=334.7761, lr=7.37e-04]
Epoch 67 TRAIN:  58%|█████▊    | 78/135 [00:20<00:12,  4.56it/s, epoch=67, grad_norm=95.098, loss=247.4004, lr=7.37e-04] 
Epoch 67 TRAIN:  58%|█████▊    | 78/135 [00:20<00:12,  4.56it/s, epoch=67, grad_norm=124.182, loss=287.0374, lr=7.37e-04]
Epoch 67 TRAIN:  58%|█████▊    | 78/135 [00:21<00:12,  4.56it/s, epoch=67, grad_norm=178.349, loss=329.5344, lr=7.37e-04]
Epoch 67 TRAIN:  60%|██████    | 81/135 [00:21<00:12,  4.48it/s, epoch=67, grad_norm=178.349, loss=329.5344, lr=7.37e-04]
Epoch 67 TRAIN:  60%|██████    | 81/135 [00:21<00:12,  4.48it/s, epoch=67, grad_norm=158.664, loss=261.2065, lr=7.37e-04]
Epoch 67 TRAIN:  60%|██████    | 81/135 [00:21<00:12,  4.48it/s, epoch=67, grad_norm=132.939, loss=321.9630, lr=7.37e-04]
Epoch 67 TRAIN:  60%|██████    | 81/135 [00:21<00:12,  4.48it/s, epoch=67, grad_norm=283.640, loss=225.4146, lr=7.38e-04]
Epoch 67 TRAIN:  62%|██████▏   | 84/135 [00:21<00:11,  4.35it/s, epoch=67, grad_norm=283.640, loss=225.4146, lr=7.38e-04]
Epoch 67 TRAIN:  62%|██████▏   | 84/135 [00:22<00:11,  4.35it/s, epoch=67, grad_norm=252.234, loss=315.0427, lr=7.38e-04]
Epoch 67 TRAIN:  62%|██████▏   | 84/135 [00:22<00:11,  4.35it/s, epoch=67, grad_norm=202.290, loss=267.9524, lr=7.38e-04]
Epoch 67 TRAIN:  62%|██████▏   | 84/135 [00:22<00:11,  4.35it/s, epoch=67, grad_norm=145.259, loss=309.7735, lr=7.38e-04]
Epoch 67 TRAIN:  64%|██████▍   | 87/135 [00:22<00:10,  4.44it/s, epoch=67, grad_norm=145.259, loss=309.7735, lr=7.38e-04]
Epoch 67 TRAIN:  64%|██████▍   | 87/135 [00:22<00:10,  4.44it/s, epoch=67, grad_norm=192.222, loss=318.3786, lr=7.38e-04]
Epoch 67 TRAIN:  64%|██████▍   | 87/135 [00:22<00:10,  4.44it/s, epoch=67, grad_norm=276.925, loss=333.7357, lr=7.38e-04]
Epoch 67 TRAIN:  64%|██████▍   | 87/135 [00:23<00:10,  4.44it/s, epoch=67, grad_norm=164.350, loss=293.1638, lr=7.38e-04]
Epoch 67 TRAIN:  67%|██████▋   | 90/135 [00:23<00:10,  4.33it/s, epoch=67, grad_norm=164.350, loss=293.1638, lr=7.38e-04]
Epoch 67 TRAIN:  67%|██████▋   | 90/135 [00:23<00:10,  4.33it/s, epoch=67, grad_norm=169.353, loss=336.7475, lr=7.38e-04]
Epoch 67 TRAIN:  67%|██████▋   | 90/135 [00:23<00:10,  4.33it/s, epoch=67, grad_norm=155.531, loss=320.5724, lr=7.38e-04]
Epoch 67 TRAIN:  67%|██████▋   | 90/135 [00:23<00:10,  4.33it/s, epoch=67, grad_norm=248.055, loss=289.0181, lr=7.38e-04]
Epoch 67 TRAIN:  69%|██████▉   | 93/135 [00:23<00:09,  4.40it/s, epoch=67, grad_norm=248.055, loss=289.0181, lr=7.38e-04]
Epoch 67 TRAIN:  69%|██████▉   | 93/135 [00:24<00:09,  4.40it/s, epoch=67, grad_norm=309.517, loss=197.9001, lr=7.38e-04]
Epoch 67 TRAIN:  69%|██████▉   | 93/135 [00:24<00:09,  4.40it/s, epoch=67, grad_norm=148.989, loss=238.2196, lr=7.38e-04]
Epoch 67 TRAIN:  69%|██████▉   | 93/135 [00:24<00:09,  4.40it/s, epoch=67, grad_norm=99.942, loss=241.6095, lr=7.38e-04] 
Epoch 67 TRAIN:  71%|███████   | 96/135 [00:24<00:08,  4.64it/s, epoch=67, grad_norm=99.942, loss=241.6095, lr=7.38e-04]
Epoch 67 TRAIN:  71%|███████   | 96/135 [00:24<00:08,  4.64it/s, epoch=67, grad_norm=133.316, loss=240.6938, lr=7.39e-04]
Epoch 67 TRAIN:  71%|███████   | 96/135 [00:24<00:08,  4.64it/s, epoch=67, grad_norm=109.471, loss=223.3534, lr=7.39e-04]
Epoch 67 TRAIN:  71%|███████   | 96/135 [00:25<00:08,  4.64it/s, epoch=67, grad_norm=119.530, loss=240.7362, lr=7.39e-04]
Epoch 67 TRAIN:  73%|███████▎  | 99/135 [00:25<00:08,  4.39it/s, epoch=67, grad_norm=119.530, loss=240.7362, lr=7.39e-04]
Epoch 67 TRAIN:  73%|███████▎  | 99/135 [00:25<00:08,  4.39it/s, epoch=67, grad_norm=83.778, loss=238.5501, lr=7.39e-04] 2026-04-16 20:26:35,840 DEBUG TRAIN | steps/sec 3.963| Batch 67/100 loss 238.550079 loss_att 197.947586 loss_ctc 333.289185 th_accuracy 0.413408 lr 7.3880e-04 grad_norm 83.777939 rank 0

Epoch 67 TRAIN:  73%|███████▎  | 99/135 [00:25<00:08,  4.39it/s, epoch=67, grad_norm=131.323, loss=289.8927, lr=7.39e-04]
Epoch 67 TRAIN:  73%|███████▎  | 99/135 [00:26<00:08,  4.39it/s, epoch=67, grad_norm=135.980, loss=171.6368, lr=7.39e-04]
Epoch 67 TRAIN:  76%|███████▌  | 102/135 [00:26<00:07,  4.19it/s, epoch=67, grad_norm=135.980, loss=171.6368, lr=7.39e-04]
Epoch 67 TRAIN:  76%|███████▌  | 102/135 [00:26<00:07,  4.19it/s, epoch=67, grad_norm=262.349, loss=326.5222, lr=7.39e-04]
Epoch 67 TRAIN:  76%|███████▌  | 102/135 [00:26<00:07,  4.19it/s, epoch=67, grad_norm=138.235, loss=247.8380, lr=7.39e-04]
Epoch 67 TRAIN:  76%|███████▌  | 102/135 [00:26<00:07,  4.19it/s, epoch=67, grad_norm=162.099, loss=247.9633, lr=7.39e-04]
Epoch 67 TRAIN:  78%|███████▊  | 105/135 [00:26<00:07,  4.12it/s, epoch=67, grad_norm=162.099, loss=247.9633, lr=7.39e-04]
Epoch 67 TRAIN:  78%|███████▊  | 105/135 [00:27<00:07,  4.12it/s, epoch=67, grad_norm=157.095, loss=261.7145, lr=7.39e-04]
Epoch 67 TRAIN:  78%|███████▊  | 105/135 [00:27<00:07,  4.12it/s, epoch=67, grad_norm=204.923, loss=307.0623, lr=7.39e-04]
Epoch 67 TRAIN:  78%|███████▊  | 105/135 [00:27<00:07,  4.12it/s, epoch=67, grad_norm=126.599, loss=274.6627, lr=7.39e-04]
Epoch 67 TRAIN:  80%|████████  | 108/135 [00:27<00:06,  4.29it/s, epoch=67, grad_norm=126.599, loss=274.6627, lr=7.39e-04]
Epoch 67 TRAIN:  80%|████████  | 108/135 [00:27<00:06,  4.29it/s, epoch=67, grad_norm=236.990, loss=304.3672, lr=7.40e-04]
Epoch 67 TRAIN:  80%|████████  | 108/135 [00:27<00:06,  4.29it/s, epoch=67, grad_norm=201.218, loss=213.7114, lr=7.40e-04]
Epoch 67 TRAIN:  80%|████████  | 108/135 [00:28<00:06,  4.29it/s, epoch=67, grad_norm=121.833, loss=302.3641, lr=7.40e-04]
Epoch 67 TRAIN:  82%|████████▏ | 111/135 [00:28<00:05,  4.24it/s, epoch=67, grad_norm=121.833, loss=302.3641, lr=7.40e-04]
Epoch 67 TRAIN:  82%|████████▏ | 111/135 [00:28<00:05,  4.24it/s, epoch=67, grad_norm=170.475, loss=231.0753, lr=7.40e-04]
Epoch 67 TRAIN:  82%|████████▏ | 111/135 [00:28<00:05,  4.24it/s, epoch=67, grad_norm=118.663, loss=223.9048, lr=7.40e-04]
Epoch 67 TRAIN:  82%|████████▏ | 111/135 [00:28<00:05,  4.24it/s, epoch=67, grad_norm=183.640, loss=254.6600, lr=7.40e-04]
Epoch 67 TRAIN:  84%|████████▍ | 114/135 [00:28<00:04,  4.40it/s, epoch=67, grad_norm=183.640, loss=254.6600, lr=7.40e-04]
Epoch 67 TRAIN:  84%|████████▍ | 114/135 [00:28<00:04,  4.40it/s, epoch=67, grad_norm=138.222, loss=260.5786, lr=7.40e-04]
Epoch 67 TRAIN:  84%|████████▍ | 114/135 [00:29<00:04,  4.40it/s, epoch=67, grad_norm=231.716, loss=203.1183, lr=7.40e-04]
Epoch 67 TRAIN:  84%|████████▍ | 114/135 [00:29<00:04,  4.40it/s, epoch=67, grad_norm=335.463, loss=332.6044, lr=7.40e-04]
Epoch 67 TRAIN:  84%|████████▍ | 114/135 [00:29<00:04,  4.40it/s, epoch=67, grad_norm=183.708, loss=292.6856, lr=7.40e-04]
Epoch 67 TRAIN:  87%|████████▋ | 118/135 [00:29<00:03,  5.00it/s, epoch=67, grad_norm=183.708, loss=292.6856, lr=7.40e-04]
Epoch 67 TRAIN:  87%|████████▋ | 118/135 [00:29<00:03,  5.00it/s, epoch=67, grad_norm=212.133, loss=271.1641, lr=7.40e-04]
Epoch 67 TRAIN:  87%|████████▋ | 118/135 [00:29<00:03,  5.00it/s, epoch=67, grad_norm=175.533, loss=249.5915, lr=7.40e-04]
Epoch 67 TRAIN:  87%|████████▋ | 118/135 [00:30<00:03,  5.00it/s, epoch=67, grad_norm=154.902, loss=252.3605, lr=7.40e-04]
Epoch 67 TRAIN:  90%|████████▉ | 121/135 [00:30<00:02,  4.76it/s, epoch=67, grad_norm=154.902, loss=252.3605, lr=7.40e-04]
Epoch 67 TRAIN:  90%|████████▉ | 121/135 [00:30<00:02,  4.76it/s, epoch=67, grad_norm=183.984, loss=289.5010, lr=7.41e-04]
Epoch 67 TRAIN:  90%|████████▉ | 121/135 [00:30<00:02,  4.76it/s, epoch=67, grad_norm=216.820, loss=277.0999, lr=7.41e-04]
Epoch 67 TRAIN:  90%|████████▉ | 121/135 [00:30<00:02,  4.76it/s, epoch=67, grad_norm=250.172, loss=293.0323, lr=7.41e-04]
Epoch 67 TRAIN:  90%|████████▉ | 121/135 [00:30<00:02,  4.76it/s, epoch=67, grad_norm=236.107, loss=307.7728, lr=7.41e-04]
Epoch 67 TRAIN:  93%|█████████▎| 125/135 [00:30<00:01,  5.12it/s, epoch=67, grad_norm=236.107, loss=307.7728, lr=7.41e-04]
Epoch 67 TRAIN:  93%|█████████▎| 125/135 [00:30<00:01,  5.12it/s, epoch=67, grad_norm=188.192, loss=304.9973, lr=7.41e-04]
Epoch 67 TRAIN:  93%|█████████▎| 125/135 [00:31<00:01,  5.12it/s, epoch=67, grad_norm=185.274, loss=322.1269, lr=7.41e-04]
Epoch 67 TRAIN:  93%|█████████▎| 125/135 [00:31<00:01,  5.12it/s, epoch=67, grad_norm=149.218, loss=299.7498, lr=7.41e-04]
Epoch 67 TRAIN:  95%|█████████▍| 128/135 [00:31<00:01,  4.86it/s, epoch=67, grad_norm=149.218, loss=299.7498, lr=7.41e-04]
Epoch 67 TRAIN:  95%|█████████▍| 128/135 [00:31<00:01,  4.86it/s, epoch=67, grad_norm=229.335, loss=326.6305, lr=7.41e-04]
Epoch 67 TRAIN:  95%|█████████▍| 128/135 [00:31<00:01,  4.86it/s, epoch=67, grad_norm=193.602, loss=342.2631, lr=7.41e-04]
Epoch 67 TRAIN:  95%|█████████▍| 128/135 [00:32<00:01,  4.86it/s, epoch=67, grad_norm=332.542, loss=356.7725, lr=7.41e-04]
Epoch 67 TRAIN:  97%|█████████▋| 131/135 [00:32<00:00,  4.94it/s, epoch=67, grad_norm=332.542, loss=356.7725, lr=7.41e-04]
Epoch 67 TRAIN:  97%|█████████▋| 131/135 [00:32<00:00,  4.94it/s, epoch=67, grad_norm=309.125, loss=318.6571, lr=7.41e-04]
Epoch 67 TRAIN:  97%|█████████▋| 131/135 [00:32<00:00,  4.94it/s, epoch=67, grad_norm=293.096, loss=326.5509, lr=7.41e-04]
Epoch 67 TRAIN:  97%|█████████▋| 131/135 [00:32<00:00,  4.94it/s, epoch=67, grad_norm=236.971, loss=368.8632, lr=7.42e-04]
Epoch 67 TRAIN:  97%|█████████▋| 131/135 [00:32<00:00,  4.94it/s, epoch=67, grad_norm=272.226, loss=267.4717, lr=7.42e-04]
Epoch 67 TRAIN: 100%|██████████| 135/135 [00:32<00:00,  5.33it/s, epoch=67, grad_norm=272.226, loss=267.4717, lr=7.42e-04]
Epoch 67 TRAIN: 100%|██████████| 135/135 [00:32<00:00,  5.33it/s, epoch=67, grad_norm=258.362, loss=268.3831, lr=7.42e-04]
Epoch 67 TRAIN: 100%|██████████| 135/135 [00:33<00:00,  5.33it/s, epoch=67, grad_norm=224.913, loss=275.6854, lr=7.42e-04]
Epoch 67 TRAIN: 100%|██████████| 135/135 [00:33<00:00,  5.33it/s, epoch=67, grad_norm=231.044, loss=353.6223, lr=7.42e-04]
Epoch 67 TRAIN: 138it [00:33,  4.16it/s, epoch=67, grad_norm=231.044, loss=353.6223, lr=7.42e-04]                         

CV:   0%|          | 0/6 [00:00<?, ?it/s]/home/rookie/Documents/batuka/projects/wenet/wenet/wenet/utils/train_utils.py:689: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  amp_autocast(enabled=dtype is not None,

CV:  17%|█▋        | 1/6 [00:00<00:03,  1.33it/s]
                                                 
2026-04-16 20:26:44,345 INFO Epoch 67 Step 9273 CV info lr 7.4184e-04 cv_loss 476.6256297718395 rank 0 acc 0.3614286035299301
2026-04-16 20:26:44,347 INFO [Rank 0] Checkpoint: save to checkpoint exp/conformer_mongolian/epoch_67.pt
2026-04-16 20:26:44,460 INFO Epoch 68 Step 9273 TRAIN info lr 7.4184e-04 rank 0
2026-04-16 20:26:44,475 INFO using accumulate grad, new batch size is 1 times larger than before

Epoch 68 TRAIN:   0%|          | 0/138 [00:00<?, ?it/s]
Epoch 68 TRAIN:   0%|          | 0/138 [00:03<?, ?it/s, epoch=68, grad_norm=487.010, loss=370.1484, lr=7.42e-04]
Epoch 68 TRAIN:   1%|          | 1/138 [00:03<08:54,  3.90s/it, epoch=68, grad_norm=487.010, loss=370.1484, lr=7.42e-04]
Epoch 68 TRAIN:   1%|          | 1/138 [00:03<08:54,  3.90s/it, epoch=68, grad_norm=315.877, loss=265.3575, lr=7.42e-04]
Epoch 68 TRAIN:   1%|          | 1/138 [00:04<08:54,  3.90s/it, epoch=68, grad_norm=367.994, loss=333.3288, lr=7.42e-04]
Epoch 68 TRAIN:   1%|          | 1/138 [00:04<08:54,  3.90s/it, epoch=68, grad_norm=370.533, loss=334.6813, lr=7.42e-04]
Epoch 68 TRAIN:   1%|          | 1/138 [00:04<08:54,  3.90s/it, epoch=68, grad_norm=223.585, loss=207.6435, lr=7.42e-04]
Epoch 68 TRAIN:   1%|          | 1/138 [00:04<08:54,  3.90s/it, epoch=68, grad_norm=268.179, loss=263.7005, lr=7.42e-04]
Epoch 68 TRAIN:   1%|          | 1/138 [00:04<08:54,  3.90s/it, epoch=68, grad_norm=296.017, loss=321.5448, lr=7.42e-04]
Epoch 68 TRAIN:   5%|▌         | 7/138 [00:04<01:04,  2.03it/s, epoch=68, grad_norm=296.017, loss=321.5448, lr=7.42e-04]
Epoch 68 TRAIN:   5%|▌         | 7/138 [00:04<01:04,  2.03it/s, epoch=68, grad_norm=211.900, loss=262.5428, lr=7.42e-04]
Epoch 68 TRAIN:   5%|▌         | 7/138 [00:04<01:04,  2.03it/s, epoch=68, grad_norm=195.779, loss=282.7728, lr=7.43e-04]
Epoch 68 TRAIN:   5%|▌         | 7/138 [00:04<01:04,  2.03it/s, epoch=68, grad_norm=207.473, loss=309.7234, lr=7.43e-04]
Epoch 68 TRAIN:   5%|▌         | 7/138 [00:05<01:04,  2.03it/s, epoch=68, grad_norm=115.422, loss=286.2009, lr=7.43e-04]
Epoch 68 TRAIN:   8%|▊         | 11/138 [00:05<00:41,  3.05it/s, epoch=68, grad_norm=115.422, loss=286.2009, lr=7.43e-04]
Epoch 68 TRAIN:   8%|▊         | 11/138 [00:05<00:41,  3.05it/s, epoch=68, grad_norm=86.566, loss=289.0470, lr=7.43e-04] 
Epoch 68 TRAIN:   8%|▊         | 11/138 [00:05<00:41,  3.05it/s, epoch=68, grad_norm=126.289, loss=294.1285, lr=7.43e-04]
Epoch 68 TRAIN:   8%|▊         | 11/138 [00:05<00:41,  3.05it/s, epoch=68, grad_norm=150.468, loss=317.2158, lr=7.43e-04]
Epoch 68 TRAIN:   8%|▊         | 11/138 [00:05<00:41,  3.05it/s, epoch=68, grad_norm=183.792, loss=264.6044, lr=7.43e-04]
Epoch 68 TRAIN:  11%|█         | 15/138 [00:05<00:33,  3.64it/s, epoch=68, grad_norm=183.792, loss=264.6044, lr=7.43e-04]
Epoch 68 TRAIN:  11%|█         | 15/138 [00:05<00:33,  3.64it/s, epoch=68, grad_norm=117.740, loss=285.7582, lr=7.43e-04]
Epoch 68 TRAIN:  11%|█         | 15/138 [00:06<00:33,  3.64it/s, epoch=68, grad_norm=130.472, loss=294.5162, lr=7.43e-04]
Epoch 68 TRAIN:  11%|█         | 15/138 [00:06<00:33,  3.64it/s, epoch=68, grad_norm=122.119, loss=312.6699, lr=7.43e-04]
Epoch 68 TRAIN:  11%|█         | 15/138 [00:06<00:33,  3.64it/s, epoch=68, grad_norm=178.420, loss=262.1097, lr=7.43e-04]
Epoch 68 TRAIN:  14%|█▍        | 19/138 [00:06<00:30,  3.96it/s, epoch=68, grad_norm=178.420, loss=262.1097, lr=7.43e-04]
Epoch 68 TRAIN:  14%|█▍        | 19/138 [00:06<00:30,  3.96it/s, epoch=68, grad_norm=148.984, loss=287.3271, lr=7.43e-04]
Epoch 68 TRAIN:  14%|█▍        | 19/138 [00:07<00:30,  3.96it/s, epoch=68, grad_norm=95.734, loss=298.6038, lr=7.44e-04] 
Epoch 68 TRAIN:  14%|█▍        | 19/138 [00:07<00:30,  3.96it/s, epoch=68, grad_norm=105.168, loss=279.7045, lr=7.44e-04]
Epoch 68 TRAIN:  16%|█▌        | 22/138 [00:07<00:27,  4.25it/s, epoch=68, grad_norm=105.168, loss=279.7045, lr=7.44e-04]
Epoch 68 TRAIN:  16%|█▌        | 22/138 [00:07<00:27,  4.25it/s, epoch=68, grad_norm=94.437, loss=272.8949, lr=7.44e-04] 
Epoch 68 TRAIN:  16%|█▌        | 22/138 [00:07<00:27,  4.25it/s, epoch=68, grad_norm=155.013, loss=279.1923, lr=7.44e-04]
Epoch 68 TRAIN:  16%|█▌        | 22/138 [00:07<00:27,  4.25it/s, epoch=68, grad_norm=203.197, loss=321.5237, lr=7.44e-04]
Epoch 68 TRAIN:  18%|█▊        | 25/138 [00:07<00:25,  4.48it/s, epoch=68, grad_norm=203.197, loss=321.5237, lr=7.44e-04]
Epoch 68 TRAIN:  18%|█▊        | 25/138 [00:07<00:25,  4.48it/s, epoch=68, grad_norm=138.765, loss=295.3402, lr=7.44e-04]
Epoch 68 TRAIN:  18%|█▊        | 25/138 [00:08<00:25,  4.48it/s, epoch=68, grad_norm=105.204, loss=251.6386, lr=7.44e-04]
Epoch 68 TRAIN:  18%|█▊        | 25/138 [00:08<00:25,  4.48it/s, epoch=68, grad_norm=117.975, loss=338.2880, lr=7.44e-04]
Epoch 68 TRAIN:  20%|██        | 28/138 [00:08<00:23,  4.65it/s, epoch=68, grad_norm=117.975, loss=338.2880, lr=7.44e-04]
Epoch 68 TRAIN:  20%|██        | 28/138 [00:08<00:23,  4.65it/s, epoch=68, grad_norm=89.563, loss=326.8975, lr=7.44e-04] 
Epoch 68 TRAIN:  20%|██        | 28/138 [00:08<00:23,  4.65it/s, epoch=68, grad_norm=104.420, loss=313.5378, lr=7.44e-04]
Epoch 68 TRAIN:  20%|██        | 28/138 [00:08<00:23,  4.65it/s, epoch=68, grad_norm=147.019, loss=266.2170, lr=7.44e-04]
Epoch 68 TRAIN:  22%|██▏       | 31/138 [00:08<00:22,  4.82it/s, epoch=68, grad_norm=147.019, loss=266.2170, lr=7.44e-04]
Epoch 68 TRAIN:  22%|██▏       | 31/138 [00:09<00:22,  4.82it/s, epoch=68, grad_norm=147.833, loss=287.2625, lr=7.44e-04]
Epoch 68 TRAIN:  22%|██▏       | 31/138 [00:09<00:22,  4.82it/s, epoch=68, grad_norm=157.972, loss=260.6180, lr=7.44e-04]
Epoch 68 TRAIN:  22%|██▏       | 31/138 [00:09<00:22,  4.82it/s, epoch=68, grad_norm=187.738, loss=245.6562, lr=7.45e-04]
Epoch 68 TRAIN:  25%|██▍       | 34/138 [00:09<00:21,  4.81it/s, epoch=68, grad_norm=187.738, loss=245.6562, lr=7.45e-04]
Epoch 68 TRAIN:  25%|██▍       | 34/138 [00:09<00:21,  4.81it/s, epoch=68, grad_norm=178.869, loss=250.1475, lr=7.45e-04]
Epoch 68 TRAIN:  25%|██▍       | 34/138 [00:09<00:21,  4.81it/s, epoch=68, grad_norm=147.517, loss=276.5529, lr=7.45e-04]
Epoch 68 TRAIN:  25%|██▍       | 34/138 [00:10<00:21,  4.81it/s, epoch=68, grad_norm=243.883, loss=200.1985, lr=7.45e-04]
Epoch 68 TRAIN:  27%|██▋       | 37/138 [00:10<00:21,  4.77it/s, epoch=68, grad_norm=243.883, loss=200.1985, lr=7.45e-04]
Epoch 68 TRAIN:  27%|██▋       | 37/138 [00:10<00:21,  4.77it/s, epoch=68, grad_norm=218.547, loss=340.8719, lr=7.45e-04]
Epoch 68 TRAIN:  27%|██▋       | 37/138 [00:10<00:21,  4.77it/s, epoch=68, grad_norm=115.256, loss=289.2348, lr=7.45e-04]
Epoch 68 TRAIN:  27%|██▋       | 37/138 [00:10<00:21,  4.77it/s, epoch=68, grad_norm=90.941, loss=291.9137, lr=7.45e-04] 
Epoch 68 TRAIN:  29%|██▉       | 40/138 [00:10<00:20,  4.86it/s, epoch=68, grad_norm=90.941, loss=291.9137, lr=7.45e-04]
Epoch 68 TRAIN:  29%|██▉       | 40/138 [00:11<00:20,  4.86it/s, epoch=68, grad_norm=121.787, loss=294.1972, lr=7.45e-04]
Epoch 68 TRAIN:  29%|██▉       | 40/138 [00:11<00:20,  4.86it/s, epoch=68, grad_norm=102.930, loss=279.0361, lr=7.45e-04]
Epoch 68 TRAIN:  29%|██▉       | 40/138 [00:11<00:20,  4.86it/s, epoch=68, grad_norm=103.625, loss=288.8652, lr=7.45e-04]
Epoch 68 TRAIN:  31%|███       | 43/138 [00:11<00:20,  4.65it/s, epoch=68, grad_norm=103.625, loss=288.8652, lr=7.45e-04]
Epoch 68 TRAIN:  31%|███       | 43/138 [00:11<00:20,  4.65it/s, epoch=68, grad_norm=131.455, loss=312.4418, lr=7.45e-04]
Epoch 68 TRAIN:  31%|███       | 43/138 [00:12<00:20,  4.65it/s, epoch=68, grad_norm=88.767, loss=286.2890, lr=7.45e-04] 
Epoch 68 TRAIN:  31%|███       | 43/138 [00:12<00:20,  4.65it/s, epoch=68, grad_norm=109.536, loss=301.7414, lr=7.46e-04]
Epoch 68 TRAIN:  33%|███▎      | 46/138 [00:12<00:20,  4.43it/s, epoch=68, grad_norm=109.536, loss=301.7414, lr=7.46e-04]
Epoch 68 TRAIN:  33%|███▎      | 46/138 [00:12<00:20,  4.43it/s, epoch=68, grad_norm=119.530, loss=297.7875, lr=7.46e-04]
Epoch 68 TRAIN:  33%|███▎      | 46/138 [00:12<00:20,  4.43it/s, epoch=68, grad_norm=102.488, loss=267.2747, lr=7.46e-04]
Epoch 68 TRAIN:  33%|███▎      | 46/138 [00:12<00:20,  4.43it/s, epoch=68, grad_norm=145.105, loss=263.0250, lr=7.46e-04]
Epoch 68 TRAIN:  36%|███▌      | 49/138 [00:12<00:19,  4.52it/s, epoch=68, grad_norm=145.105, loss=263.0250, lr=7.46e-04]
Epoch 68 TRAIN:  36%|███▌      | 49/138 [00:13<00:19,  4.52it/s, epoch=68, grad_norm=124.839, loss=278.2171, lr=7.46e-04]
Epoch 68 TRAIN:  36%|███▌      | 49/138 [00:13<00:19,  4.52it/s, epoch=68, grad_norm=118.484, loss=326.0911, lr=7.46e-04]
Epoch 68 TRAIN:  36%|███▌      | 49/138 [00:13<00:19,  4.52it/s, epoch=68, grad_norm=141.017, loss=290.5116, lr=7.46e-04]
Epoch 68 TRAIN:  38%|███▊      | 52/138 [00:13<00:19,  4.48it/s, epoch=68, grad_norm=141.017, loss=290.5116, lr=7.46e-04]
Epoch 68 TRAIN:  38%|███▊      | 52/138 [00:13<00:19,  4.48it/s, epoch=68, grad_norm=191.694, loss=275.3825, lr=7.46e-04]
Epoch 68 TRAIN:  38%|███▊      | 52/138 [00:14<00:19,  4.48it/s, epoch=68, grad_norm=139.084, loss=295.4058, lr=7.46e-04]
Epoch 68 TRAIN:  38%|███▊      | 52/138 [00:14<00:19,  4.48it/s, epoch=68, grad_norm=98.729, loss=290.7256, lr=7.46e-04] 
Epoch 68 TRAIN:  40%|███▉      | 55/138 [00:14<00:18,  4.55it/s, epoch=68, grad_norm=98.729, loss=290.7256, lr=7.46e-04]
Epoch 68 TRAIN:  40%|███▉      | 55/138 [00:14<00:18,  4.55it/s, epoch=68, grad_norm=78.779, loss=290.9604, lr=7.46e-04]
Epoch 68 TRAIN:  40%|███▉      | 55/138 [00:14<00:18,  4.55it/s, epoch=68, grad_norm=84.052, loss=304.7729, lr=7.46e-04]
Epoch 68 TRAIN:  40%|███▉      | 55/138 [00:14<00:18,  4.55it/s, epoch=68, grad_norm=102.125, loss=280.7200, lr=7.46e-04]
Epoch 68 TRAIN:  42%|████▏     | 58/138 [00:14<00:18,  4.41it/s, epoch=68, grad_norm=102.125, loss=280.7200, lr=7.46e-04]
Epoch 68 TRAIN:  42%|████▏     | 58/138 [00:15<00:18,  4.41it/s, epoch=68, grad_norm=113.771, loss=236.3850, lr=7.47e-04]
Epoch 68 TRAIN:  42%|████▏     | 58/138 [00:15<00:18,  4.41it/s, epoch=68, grad_norm=95.034, loss=268.6354, lr=7.47e-04] 
Epoch 68 TRAIN:  42%|████▏     | 58/138 [00:15<00:18,  4.41it/s, epoch=68, grad_norm=122.345, loss=253.4756, lr=7.47e-04]
Epoch 68 TRAIN:  44%|████▍     | 61/138 [00:15<00:17,  4.30it/s, epoch=68, grad_norm=122.345, loss=253.4756, lr=7.47e-04]
Epoch 68 TRAIN:  44%|████▍     | 61/138 [00:15<00:17,  4.30it/s, epoch=68, grad_norm=92.605, loss=277.7591, lr=7.47e-04] 
Epoch 68 TRAIN:  44%|████▍     | 61/138 [00:16<00:17,  4.30it/s, epoch=68, grad_norm=129.138, loss=293.2136, lr=7.47e-04]
Epoch 68 TRAIN:  44%|████▍     | 61/138 [00:16<00:17,  4.30it/s, epoch=68, grad_norm=184.885, loss=326.3616, lr=7.47e-04]
Epoch 68 TRAIN:  46%|████▋     | 64/138 [00:16<00:16,  4.41it/s, epoch=68, grad_norm=184.885, loss=326.3616, lr=7.47e-04]
Epoch 68 TRAIN:  46%|████▋     | 64/138 [00:16<00:16,  4.41it/s, epoch=68, grad_norm=103.068, loss=288.3417, lr=7.47e-04]
Epoch 68 TRAIN:  46%|████▋     | 64/138 [00:16<00:16,  4.41it/s, epoch=68, grad_norm=150.850, loss=319.5885, lr=7.47e-04]
Epoch 68 TRAIN:  46%|████▋     | 64/138 [00:17<00:16,  4.41it/s, epoch=68, grad_norm=131.217, loss=298.8504, lr=7.47e-04]
Epoch 68 TRAIN:  49%|████▊     | 67/138 [00:17<00:16,  4.32it/s, epoch=68, grad_norm=131.217, loss=298.8504, lr=7.47e-04]
Epoch 68 TRAIN:  49%|████▊     | 67/138 [00:17<00:16,  4.32it/s, epoch=68, grad_norm=134.389, loss=238.4707, lr=7.47e-04]
Epoch 68 TRAIN:  49%|████▊     | 67/138 [00:17<00:16,  4.32it/s, epoch=68, grad_norm=188.347, loss=255.1875, lr=7.47e-04]
Epoch 68 TRAIN:  49%|████▊     | 67/138 [00:17<00:16,  4.32it/s, epoch=68, grad_norm=128.921, loss=254.6653, lr=7.47e-04]
Epoch 68 TRAIN:  51%|█████     | 70/138 [00:17<00:15,  4.41it/s, epoch=68, grad_norm=128.921, loss=254.6653, lr=7.47e-04]
Epoch 68 TRAIN:  51%|█████     | 70/138 [00:17<00:15,  4.41it/s, epoch=68, grad_norm=89.722, loss=254.7124, lr=7.48e-04] 
Epoch 68 TRAIN:  51%|█████     | 70/138 [00:18<00:15,  4.41it/s, epoch=68, grad_norm=113.045, loss=286.0130, lr=7.48e-04]
Epoch 68 TRAIN:  51%|█████     | 70/138 [00:18<00:15,  4.41it/s, epoch=68, grad_norm=169.013, loss=283.7414, lr=7.48e-04]
Epoch 68 TRAIN:  53%|█████▎    | 73/138 [00:18<00:14,  4.34it/s, epoch=68, grad_norm=169.013, loss=283.7414, lr=7.48e-04]
Epoch 68 TRAIN:  53%|█████▎    | 73/138 [00:18<00:14,  4.34it/s, epoch=68, grad_norm=163.366, loss=311.9368, lr=7.48e-04]
Epoch 68 TRAIN:  53%|█████▎    | 73/138 [00:18<00:14,  4.34it/s, epoch=68, grad_norm=90.836, loss=268.6183, lr=7.48e-04] 
Epoch 68 TRAIN:  53%|█████▎    | 73/138 [00:18<00:14,  4.34it/s, epoch=68, grad_norm=136.594, loss=293.3777, lr=7.48e-04]
Epoch 68 TRAIN:  55%|█████▌    | 76/138 [00:18<00:13,  4.60it/s, epoch=68, grad_norm=136.594, loss=293.3777, lr=7.48e-04]
Epoch 68 TRAIN:  55%|█████▌    | 76/138 [00:19<00:13,  4.60it/s, epoch=68, grad_norm=145.651, loss=251.7850, lr=7.48e-04]
Epoch 68 TRAIN:  55%|█████▌    | 76/138 [00:19<00:13,  4.60it/s, epoch=68, grad_norm=100.256, loss=255.3226, lr=7.48e-04]
Epoch 68 TRAIN:  55%|█████▌    | 76/138 [00:19<00:13,  4.60it/s, epoch=68, grad_norm=92.185, loss=291.6882, lr=7.48e-04] 
Epoch 68 TRAIN:  57%|█████▋    | 79/138 [00:19<00:13,  4.46it/s, epoch=68, grad_norm=92.185, loss=291.6882, lr=7.48e-04]
Epoch 68 TRAIN:  57%|█████▋    | 79/138 [00:19<00:13,  4.46it/s, epoch=68, grad_norm=116.791, loss=306.9026, lr=7.48e-04]
Epoch 68 TRAIN:  57%|█████▋    | 79/138 [00:20<00:13,  4.46it/s, epoch=68, grad_norm=125.718, loss=297.4893, lr=7.48e-04]
Epoch 68 TRAIN:  57%|█████▋    | 79/138 [00:20<00:13,  4.46it/s, epoch=68, grad_norm=150.603, loss=296.6386, lr=7.48e-04]
Epoch 68 TRAIN:  59%|█████▉    | 82/138 [00:20<00:12,  4.42it/s, epoch=68, grad_norm=150.603, loss=296.6386, lr=7.48e-04]
Epoch 68 TRAIN:  59%|█████▉    | 82/138 [00:20<00:12,  4.42it/s, epoch=68, grad_norm=83.725, loss=265.2675, lr=7.48e-04] 
Epoch 68 TRAIN:  59%|█████▉    | 82/138 [00:20<00:12,  4.42it/s, epoch=68, grad_norm=144.828, loss=266.2020, lr=7.49e-04]
Epoch 68 TRAIN:  59%|█████▉    | 82/138 [00:21<00:12,  4.42it/s, epoch=68, grad_norm=113.863, loss=309.4819, lr=7.49e-04]
Epoch 68 TRAIN:  62%|██████▏   | 85/138 [00:21<00:12,  4.28it/s, epoch=68, grad_norm=113.863, loss=309.4819, lr=7.49e-04]
Epoch 68 TRAIN:  62%|██████▏   | 85/138 [00:21<00:12,  4.28it/s, epoch=68, grad_norm=144.730, loss=314.8802, lr=7.49e-04]
Epoch 68 TRAIN:  62%|██████▏   | 85/138 [00:21<00:12,  4.28it/s, epoch=68, grad_norm=132.405, loss=330.3241, lr=7.49e-04]
Epoch 68 TRAIN:  62%|██████▏   | 85/138 [00:21<00:12,  4.28it/s, epoch=68, grad_norm=106.607, loss=242.7356, lr=7.49e-04]
Epoch 68 TRAIN:  64%|██████▍   | 88/138 [00:21<00:11,  4.31it/s, epoch=68, grad_norm=106.607, loss=242.7356, lr=7.49e-04]
Epoch 68 TRAIN:  64%|██████▍   | 88/138 [00:22<00:11,  4.31it/s, epoch=68, grad_norm=247.708, loss=250.1082, lr=7.49e-04]
Epoch 68 TRAIN:  64%|██████▍   | 88/138 [00:22<00:11,  4.31it/s, epoch=68, grad_norm=186.047, loss=234.4558, lr=7.49e-04]
Epoch 68 TRAIN:  64%|██████▍   | 88/138 [00:22<00:11,  4.31it/s, epoch=68, grad_norm=186.406, loss=204.6677, lr=7.49e-04]
Epoch 68 TRAIN:  66%|██████▌   | 91/138 [00:22<00:10,  4.35it/s, epoch=68, grad_norm=186.406, loss=204.6677, lr=7.49e-04]
Epoch 68 TRAIN:  66%|██████▌   | 91/138 [00:22<00:10,  4.35it/s, epoch=68, grad_norm=302.156, loss=261.3820, lr=7.49e-04]
Epoch 68 TRAIN:  66%|██████▌   | 91/138 [00:22<00:10,  4.35it/s, epoch=68, grad_norm=178.263, loss=310.9979, lr=7.49e-04]
Epoch 68 TRAIN:  66%|██████▌   | 91/138 [00:23<00:10,  4.35it/s, epoch=68, grad_norm=152.091, loss=257.4240, lr=7.49e-04]
Epoch 68 TRAIN:  68%|██████▊   | 94/138 [00:23<00:09,  4.54it/s, epoch=68, grad_norm=152.091, loss=257.4240, lr=7.49e-04]
Epoch 68 TRAIN:  68%|██████▊   | 94/138 [00:23<00:09,  4.54it/s, epoch=68, grad_norm=213.492, loss=320.8484, lr=7.49e-04]
Epoch 68 TRAIN:  68%|██████▊   | 94/138 [00:23<00:09,  4.54it/s, epoch=68, grad_norm=158.270, loss=253.8652, lr=7.50e-04]
Epoch 68 TRAIN:  68%|██████▊   | 94/138 [00:23<00:09,  4.54it/s, epoch=68, grad_norm=105.037, loss=235.6377, lr=7.50e-04]
Epoch 68 TRAIN:  70%|███████   | 97/138 [00:23<00:08,  4.62it/s, epoch=68, grad_norm=105.037, loss=235.6377, lr=7.50e-04]
Epoch 68 TRAIN:  70%|███████   | 97/138 [00:23<00:08,  4.62it/s, epoch=68, grad_norm=151.019, loss=288.6823, lr=7.50e-04]
Epoch 68 TRAIN:  70%|███████   | 97/138 [00:24<00:08,  4.62it/s, epoch=68, grad_norm=150.543, loss=273.1228, lr=7.50e-04]
Epoch 68 TRAIN:  70%|███████   | 97/138 [00:24<00:08,  4.62it/s, epoch=68, grad_norm=155.411, loss=307.5002, lr=7.50e-04]2026-04-16 20:27:08,876 DEBUG TRAIN | steps/sec 4.177| Batch 68/100 loss 307.500183 loss_att 251.169540 loss_ctc 438.938385 th_accuracy 0.409375 lr 7.4984e-04 grad_norm 155.410522 rank 0

Epoch 68 TRAIN:  72%|███████▏  | 100/138 [00:24<00:08,  4.54it/s, epoch=68, grad_norm=155.411, loss=307.5002, lr=7.50e-04]
Epoch 68 TRAIN:  72%|███████▏  | 100/138 [00:24<00:08,  4.54it/s, epoch=68, grad_norm=179.649, loss=303.4049, lr=7.50e-04]
Epoch 68 TRAIN:  72%|███████▏  | 100/138 [00:24<00:08,  4.54it/s, epoch=68, grad_norm=237.762, loss=335.1142, lr=7.50e-04]
Epoch 68 TRAIN:  72%|███████▏  | 100/138 [00:25<00:08,  4.54it/s, epoch=68, grad_norm=197.086, loss=189.4610, lr=7.50e-04]
Epoch 68 TRAIN:  75%|███████▍  | 103/138 [00:25<00:07,  4.60it/s, epoch=68, grad_norm=197.086, loss=189.4610, lr=7.50e-04]
Epoch 68 TRAIN:  75%|███████▍  | 103/138 [00:25<00:07,  4.60it/s, epoch=68, grad_norm=131.419, loss=269.8771, lr=7.50e-04]
Epoch 68 TRAIN:  75%|███████▍  | 103/138 [00:25<00:07,  4.60it/s, epoch=68, grad_norm=131.197, loss=253.7350, lr=7.50e-04]
Epoch 68 TRAIN:  75%|███████▍  | 103/138 [00:25<00:07,  4.60it/s, epoch=68, grad_norm=181.459, loss=242.0742, lr=7.50e-04]
Epoch 68 TRAIN:  77%|███████▋  | 106/138 [00:25<00:07,  4.38it/s, epoch=68, grad_norm=181.459, loss=242.0742, lr=7.50e-04]
Epoch 68 TRAIN:  77%|███████▋  | 106/138 [00:25<00:07,  4.38it/s, epoch=68, grad_norm=134.632, loss=223.9193, lr=7.50e-04]
Epoch 68 TRAIN:  77%|███████▋  | 106/138 [00:26<00:07,  4.38it/s, epoch=68, grad_norm=245.257, loss=354.2275, lr=7.50e-04]
Epoch 68 TRAIN:  77%|███████▋  | 106/138 [00:26<00:07,  4.38it/s, epoch=68, grad_norm=196.235, loss=276.9515, lr=7.51e-04]
Epoch 68 TRAIN:  77%|███████▋  | 106/138 [00:26<00:07,  4.38it/s, epoch=68, grad_norm=103.734, loss=202.1867, lr=7.51e-04]
Epoch 68 TRAIN:  80%|███████▉  | 110/138 [00:26<00:06,  4.37it/s, epoch=68, grad_norm=103.734, loss=202.1867, lr=7.51e-04]
Epoch 68 TRAIN:  80%|███████▉  | 110/138 [00:26<00:06,  4.37it/s, epoch=68, grad_norm=218.264, loss=296.8893, lr=7.51e-04]
Epoch 68 TRAIN:  80%|███████▉  | 110/138 [00:27<00:06,  4.37it/s, epoch=68, grad_norm=138.026, loss=204.8345, lr=7.51e-04]
Epoch 68 TRAIN:  80%|███████▉  | 110/138 [00:27<00:06,  4.37it/s, epoch=68, grad_norm=152.836, loss=258.6358, lr=7.51e-04]
Epoch 68 TRAIN:  82%|████████▏ | 113/138 [00:27<00:05,  4.59it/s, epoch=68, grad_norm=152.836, loss=258.6358, lr=7.51e-04]
Epoch 68 TRAIN:  82%|████████▏ | 113/138 [00:27<00:05,  4.59it/s, epoch=68, grad_norm=234.490, loss=329.6949, lr=7.51e-04]
Epoch 68 TRAIN:  82%|████████▏ | 113/138 [00:27<00:05,  4.59it/s, epoch=68, grad_norm=322.171, loss=340.3040, lr=7.51e-04]
Epoch 68 TRAIN:  82%|████████▏ | 113/138 [00:27<00:05,  4.59it/s, epoch=68, grad_norm=162.385, loss=272.7995, lr=7.51e-04]
Epoch 68 TRAIN:  84%|████████▍ | 116/138 [00:27<00:04,  4.69it/s, epoch=68, grad_norm=162.385, loss=272.7995, lr=7.51e-04]
Epoch 68 TRAIN:  84%|████████▍ | 116/138 [00:28<00:04,  4.69it/s, epoch=68, grad_norm=143.681, loss=250.9333, lr=7.51e-04]
Epoch 68 TRAIN:  84%|████████▍ | 116/138 [00:28<00:04,  4.69it/s, epoch=68, grad_norm=105.009, loss=271.7737, lr=7.51e-04]
Epoch 68 TRAIN:  84%|████████▍ | 116/138 [00:28<00:04,  4.69it/s, epoch=68, grad_norm=100.656, loss=273.8311, lr=7.51e-04]
Epoch 68 TRAIN:  86%|████████▌ | 119/138 [00:28<00:04,  4.43it/s, epoch=68, grad_norm=100.656, loss=273.8311, lr=7.51e-04]
Epoch 68 TRAIN:  86%|████████▌ | 119/138 [00:28<00:04,  4.43it/s, epoch=68, grad_norm=180.699, loss=236.5743, lr=7.51e-04]
Epoch 68 TRAIN:  86%|████████▌ | 119/138 [00:29<00:04,  4.43it/s, epoch=68, grad_norm=159.862, loss=326.8229, lr=7.52e-04]
Epoch 68 TRAIN:  86%|████████▌ | 119/138 [00:29<00:04,  4.43it/s, epoch=68, grad_norm=130.414, loss=258.2318, lr=7.52e-04]
Epoch 68 TRAIN:  88%|████████▊ | 122/138 [00:29<00:03,  4.63it/s, epoch=68, grad_norm=130.414, loss=258.2318, lr=7.52e-04]
Epoch 68 TRAIN:  88%|████████▊ | 122/138 [00:29<00:03,  4.63it/s, epoch=68, grad_norm=124.876, loss=280.1365, lr=7.52e-04]
Epoch 68 TRAIN:  88%|████████▊ | 122/138 [00:29<00:03,  4.63it/s, epoch=68, grad_norm=170.355, loss=228.9936, lr=7.52e-04]
Epoch 68 TRAIN:  88%|████████▊ | 122/138 [00:29<00:03,  4.63it/s, epoch=68, grad_norm=190.660, loss=317.8658, lr=7.52e-04]
Epoch 68 TRAIN:  91%|█████████ | 125/138 [00:29<00:02,  4.65it/s, epoch=68, grad_norm=190.660, loss=317.8658, lr=7.52e-04]
Epoch 68 TRAIN:  91%|█████████ | 125/138 [00:30<00:02,  4.65it/s, epoch=68, grad_norm=187.590, loss=295.8295, lr=7.52e-04]
Epoch 68 TRAIN:  91%|█████████ | 125/138 [00:30<00:02,  4.65it/s, epoch=68, grad_norm=186.263, loss=354.4992, lr=7.52e-04]
Epoch 68 TRAIN:  91%|█████████ | 125/138 [00:30<00:02,  4.65it/s, epoch=68, grad_norm=90.105, loss=229.2416, lr=7.52e-04] 
Epoch 68 TRAIN:  93%|█████████▎| 128/138 [00:30<00:02,  4.92it/s, epoch=68, grad_norm=90.105, loss=229.2416, lr=7.52e-04]
Epoch 68 TRAIN:  93%|█████████▎| 128/138 [00:30<00:02,  4.92it/s, epoch=68, grad_norm=141.082, loss=252.5974, lr=7.52e-04]
Epoch 68 TRAIN:  93%|█████████▎| 128/138 [00:30<00:02,  4.92it/s, epoch=68, grad_norm=281.646, loss=319.2268, lr=7.52e-04]
Epoch 68 TRAIN:  93%|█████████▎| 128/138 [00:31<00:02,  4.92it/s, epoch=68, grad_norm=244.621, loss=334.4546, lr=7.52e-04]
Epoch 68 TRAIN:  95%|█████████▍| 131/138 [00:31<00:01,  4.74it/s, epoch=68, grad_norm=244.621, loss=334.4546, lr=7.52e-04]
Epoch 68 TRAIN:  95%|█████████▍| 131/138 [00:31<00:01,  4.74it/s, epoch=68, grad_norm=111.356, loss=281.0178, lr=7.52e-04]
Epoch 68 TRAIN:  95%|█████████▍| 131/138 [00:31<00:01,  4.74it/s, epoch=68, grad_norm=170.953, loss=371.7131, lr=7.52e-04]
Epoch 68 TRAIN:  95%|█████████▍| 131/138 [00:31<00:01,  4.74it/s, epoch=68, grad_norm=148.403, loss=336.0735, lr=7.53e-04]
Epoch 68 TRAIN:  97%|█████████▋| 134/138 [00:31<00:00,  4.70it/s, epoch=68, grad_norm=148.403, loss=336.0735, lr=7.53e-04]
Epoch 68 TRAIN:  97%|█████████▋| 134/138 [00:32<00:00,  4.70it/s, epoch=68, grad_norm=272.748, loss=319.4669, lr=7.53e-04]
Epoch 68 TRAIN:  97%|█████████▋| 134/138 [00:32<00:00,  4.70it/s, epoch=68, grad_norm=292.319, loss=308.2192, lr=7.53e-04]
Epoch 68 TRAIN:  97%|█████████▋| 134/138 [00:32<00:00,  4.70it/s, epoch=68, grad_norm=297.228, loss=243.5296, lr=7.53e-04]
Epoch 68 TRAIN:  99%|█████████▉| 137/138 [00:32<00:00,  4.40it/s, epoch=68, grad_norm=297.228, loss=243.5296, lr=7.53e-04]
Epoch 68 TRAIN:  99%|█████████▉| 137/138 [00:32<00:00,  4.21it/s, epoch=68, grad_norm=297.228, loss=243.5296, lr=7.53e-04]

CV:   0%|          | 0/6 [00:00<?, ?it/s]/home/rookie/Documents/batuka/projects/wenet/wenet/wenet/utils/train_utils.py:689: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  amp_autocast(enabled=dtype is not None,

CV:  17%|█▋        | 1/6 [00:00<00:03,  1.30it/s]
                                                 
2026-04-16 20:27:17,857 INFO Epoch 68 Step 9410 CV info lr 7.5280e-04 cv_loss 411.6837851784446 rank 0 acc 0.34645860890547436
2026-04-16 20:27:17,859 INFO [Rank 0] Checkpoint: save to checkpoint exp/conformer_mongolian/epoch_68.pt
2026-04-16 20:27:17,973 INFO Epoch 69 Step 9410 TRAIN info lr 7.5280e-04 rank 0
2026-04-16 20:27:17,987 INFO using accumulate grad, new batch size is 1 times larger than before

Epoch 69 TRAIN:   0%|          | 0/137 [00:00<?, ?it/s]
Epoch 69 TRAIN:   0%|          | 0/137 [00:04<?, ?it/s, epoch=69, grad_norm=519.553, loss=372.9177, lr=7.53e-04]
Epoch 69 TRAIN:   1%|          | 1/137 [00:04<09:24,  4.15s/it, epoch=69, grad_norm=519.553, loss=372.9177, lr=7.53e-04]
Epoch 69 TRAIN:   1%|          | 1/137 [00:04<09:24,  4.15s/it, epoch=69, grad_norm=275.003, loss=216.5866, lr=7.53e-04]
Epoch 69 TRAIN:   1%|          | 1/137 [00:04<09:24,  4.15s/it, epoch=69, grad_norm=492.517, loss=410.9168, lr=7.53e-04]
Epoch 69 TRAIN:   1%|          | 1/137 [00:04<09:24,  4.15s/it, epoch=69, grad_norm=512.945, loss=437.4786, lr=7.53e-04]
Epoch 69 TRAIN:   1%|          | 1/137 [00:04<09:24,  4.15s/it, epoch=69, grad_norm=311.633, loss=310.8419, lr=7.53e-04]
Epoch 69 TRAIN:   1%|          | 1/137 [00:04<09:24,  4.15s/it, epoch=69, grad_norm=391.132, loss=333.5956, lr=7.53e-04]
Epoch 69 TRAIN:   1%|          | 1/137 [00:04<09:24,  4.15s/it, epoch=69, grad_norm=295.270, loss=314.5251, lr=7.53e-04]
Epoch 69 TRAIN:   1%|          | 1/137 [00:04<09:24,  4.15s/it, epoch=69, grad_norm=133.967, loss=202.1151, lr=7.53e-04]
Epoch 69 TRAIN:   1%|          | 1/137 [00:04<09:24,  4.15s/it, epoch=69, grad_norm=205.146, loss=306.9075, lr=7.54e-04]
Epoch 69 TRAIN:   7%|▋         | 9/137 [00:04<00:52,  2.45it/s, epoch=69, grad_norm=205.146, loss=306.9075, lr=7.54e-04]
Epoch 69 TRAIN:   7%|▋         | 9/137 [00:04<00:52,  2.45it/s, epoch=69, grad_norm=157.886, loss=294.1189, lr=7.54e-04]
Epoch 69 TRAIN:   7%|▋         | 9/137 [00:05<00:52,  2.45it/s, epoch=69, grad_norm=143.702, loss=276.1476, lr=7.54e-04]
Epoch 69 TRAIN:   7%|▋         | 9/137 [00:05<00:52,  2.45it/s, epoch=69, grad_norm=132.702, loss=270.3671, lr=7.54e-04]
Epoch 69 TRAIN:   7%|▋         | 9/137 [00:05<00:52,  2.45it/s, epoch=69, grad_norm=115.938, loss=310.0804, lr=7.54e-04]
Epoch 69 TRAIN:   9%|▉         | 13/137 [00:05<00:38,  3.21it/s, epoch=69, grad_norm=115.938, loss=310.0804, lr=7.54e-04]
Epoch 69 TRAIN:   9%|▉         | 13/137 [00:05<00:38,  3.21it/s, epoch=69, grad_norm=134.362, loss=289.8770, lr=7.54e-04]
Epoch 69 TRAIN:   9%|▉         | 13/137 [00:05<00:38,  3.21it/s, epoch=69, grad_norm=91.717, loss=275.0339, lr=7.54e-04] 
Epoch 69 TRAIN:   9%|▉         | 13/137 [00:06<00:38,  3.21it/s, epoch=69, grad_norm=120.193, loss=303.7380, lr=7.54e-04]
Epoch 69 TRAIN:   9%|▉         | 13/137 [00:06<00:38,  3.21it/s, epoch=69, grad_norm=99.301, loss=259.7174, lr=7.54e-04] 
Epoch 69 TRAIN:  12%|█▏        | 17/137 [00:06<00:32,  3.66it/s, epoch=69, grad_norm=99.301, loss=259.7174, lr=7.54e-04]
Epoch 69 TRAIN:  12%|█▏        | 17/137 [00:06<00:32,  3.66it/s, epoch=69, grad_norm=114.864, loss=316.4450, lr=7.54e-04]
Epoch 69 TRAIN:  12%|█▏        | 17/137 [00:06<00:32,  3.66it/s, epoch=69, grad_norm=101.569, loss=290.8531, lr=7.54e-04]
Epoch 69 TRAIN:  12%|█▏        | 17/137 [00:06<00:32,  3.66it/s, epoch=69, grad_norm=68.489, loss=259.4706, lr=7.54e-04] 
Epoch 69 TRAIN:  12%|█▏        | 17/137 [00:06<00:32,  3.66it/s, epoch=69, grad_norm=105.616, loss=305.9760, lr=7.54e-04]
Epoch 69 TRAIN:  15%|█▌        | 21/137 [00:06<00:27,  4.14it/s, epoch=69, grad_norm=105.616, loss=305.9760, lr=7.54e-04]
Epoch 69 TRAIN:  15%|█▌        | 21/137 [00:07<00:27,  4.14it/s, epoch=69, grad_norm=96.178, loss=257.1837, lr=7.55e-04] 
Epoch 69 TRAIN:  15%|█▌        | 21/137 [00:07<00:27,  4.14it/s, epoch=69, grad_norm=94.442, loss=303.9364, lr=7.55e-04]
Epoch 69 TRAIN:  15%|█▌        | 21/137 [00:07<00:27,  4.14it/s, epoch=69, grad_norm=89.789, loss=287.5878, lr=7.55e-04]
Epoch 69 TRAIN:  18%|█▊        | 24/137 [00:07<00:26,  4.34it/s, epoch=69, grad_norm=89.789, loss=287.5878, lr=7.55e-04]
Epoch 69 TRAIN:  18%|█▊        | 24/137 [00:07<00:26,  4.34it/s, epoch=69, grad_norm=133.073, loss=239.2789, lr=7.55e-04]
Epoch 69 TRAIN:  18%|█▊        | 24/137 [00:07<00:26,  4.34it/s, epoch=69, grad_norm=117.598, loss=313.5851, lr=7.55e-04]
Epoch 69 TRAIN:  18%|█▊        | 24/137 [00:08<00:26,  4.34it/s, epoch=69, grad_norm=106.308, loss=273.2267, lr=7.55e-04]
Epoch 69 TRAIN:  20%|█▉        | 27/137 [00:08<00:24,  4.54it/s, epoch=69, grad_norm=106.308, loss=273.2267, lr=7.55e-04]
Epoch 69 TRAIN:  20%|█▉        | 27/137 [00:08<00:24,  4.54it/s, epoch=69, grad_norm=90.246, loss=281.5950, lr=7.55e-04] 
Epoch 69 TRAIN:  20%|█▉        | 27/137 [00:08<00:24,  4.54it/s, epoch=69, grad_norm=119.978, loss=258.2376, lr=7.55e-04]
Epoch 69 TRAIN:  20%|█▉        | 27/137 [00:08<00:24,  4.54it/s, epoch=69, grad_norm=92.716, loss=294.5592, lr=7.55e-04] 
Epoch 69 TRAIN:  22%|██▏       | 30/137 [00:08<00:22,  4.70it/s, epoch=69, grad_norm=92.716, loss=294.5592, lr=7.55e-04]
Epoch 69 TRAIN:  22%|██▏       | 30/137 [00:08<00:22,  4.70it/s, epoch=69, grad_norm=129.359, loss=318.6436, lr=7.55e-04]
Epoch 69 TRAIN:  22%|██▏       | 30/137 [00:09<00:22,  4.70it/s, epoch=69, grad_norm=99.564, loss=306.6492, lr=7.55e-04] 
Epoch 69 TRAIN:  22%|██▏       | 30/137 [00:09<00:22,  4.70it/s, epoch=69, grad_norm=92.583, loss=260.8022, lr=7.55e-04]
Epoch 69 TRAIN:  24%|██▍       | 33/137 [00:09<00:21,  4.81it/s, epoch=69, grad_norm=92.583, loss=260.8022, lr=7.55e-04]
Epoch 69 TRAIN:  24%|██▍       | 33/137 [00:09<00:21,  4.81it/s, epoch=69, grad_norm=82.655, loss=279.6210, lr=7.56e-04]
Epoch 69 TRAIN:  24%|██▍       | 33/137 [00:09<00:21,  4.81it/s, epoch=69, grad_norm=74.773, loss=281.0278, lr=7.56e-04]
Epoch 69 TRAIN:  24%|██▍       | 33/137 [00:09<00:21,  4.81it/s, epoch=69, grad_norm=141.718, loss=333.2468, lr=7.56e-04]
Epoch 69 TRAIN:  26%|██▋       | 36/137 [00:09<00:20,  4.86it/s, epoch=69, grad_norm=141.718, loss=333.2468, lr=7.56e-04]
Epoch 69 TRAIN:  26%|██▋       | 36/137 [00:10<00:20,  4.86it/s, epoch=69, grad_norm=108.297, loss=250.6284, lr=7.56e-04]
Epoch 69 TRAIN:  26%|██▋       | 36/137 [00:10<00:20,  4.86it/s, epoch=69, grad_norm=90.384, loss=253.0137, lr=7.56e-04] 
Epoch 69 TRAIN:  26%|██▋       | 36/137 [00:10<00:20,  4.86it/s, epoch=69, grad_norm=65.923, loss=275.8896, lr=7.56e-04]
Epoch 69 TRAIN:  28%|██▊       | 39/137 [00:10<00:20,  4.82it/s, epoch=69, grad_norm=65.923, loss=275.8896, lr=7.56e-04]
Epoch 69 TRAIN:  28%|██▊       | 39/137 [00:10<00:20,  4.82it/s, epoch=69, grad_norm=96.402, loss=259.7885, lr=7.56e-04]
Epoch 69 TRAIN:  28%|██▊       | 39/137 [00:11<00:20,  4.82it/s, epoch=69, grad_norm=90.520, loss=271.0587, lr=7.56e-04]
Epoch 69 TRAIN:  28%|██▊       | 39/137 [00:11<00:20,  4.82it/s, epoch=69, grad_norm=123.223, loss=271.2271, lr=7.56e-04]
Epoch 69 TRAIN:  31%|███       | 42/137 [00:11<00:20,  4.58it/s, epoch=69, grad_norm=123.223, loss=271.2271, lr=7.56e-04]
Epoch 69 TRAIN:  31%|███       | 42/137 [00:11<00:20,  4.58it/s, epoch=69, grad_norm=233.042, loss=346.0968, lr=7.56e-04]
Epoch 69 TRAIN:  31%|███       | 42/137 [00:11<00:20,  4.58it/s, epoch=69, grad_norm=179.611, loss=286.0318, lr=7.56e-04]
Epoch 69 TRAIN:  31%|███       | 42/137 [00:11<00:20,  4.58it/s, epoch=69, grad_norm=103.200, loss=284.1994, lr=7.56e-04]
Epoch 69 TRAIN:  33%|███▎      | 45/137 [00:11<00:19,  4.75it/s, epoch=69, grad_norm=103.200, loss=284.1994, lr=7.56e-04]
Epoch 69 TRAIN:  33%|███▎      | 45/137 [00:12<00:19,  4.75it/s, epoch=69, grad_norm=107.775, loss=309.4902, lr=7.56e-04]
Epoch 69 TRAIN:  33%|███▎      | 45/137 [00:12<00:19,  4.75it/s, epoch=69, grad_norm=81.989, loss=283.8230, lr=7.57e-04] 
Epoch 69 TRAIN:  33%|███▎      | 45/137 [00:12<00:19,  4.75it/s, epoch=69, grad_norm=105.714, loss=276.3213, lr=7.57e-04]
Epoch 69 TRAIN:  35%|███▌      | 48/137 [00:12<00:18,  4.75it/s, epoch=69, grad_norm=105.714, loss=276.3213, lr=7.57e-04]
Epoch 69 TRAIN:  35%|███▌      | 48/137 [00:12<00:18,  4.75it/s, epoch=69, grad_norm=100.799, loss=259.8969, lr=7.57e-04]
Epoch 69 TRAIN:  35%|███▌      | 48/137 [00:12<00:18,  4.75it/s, epoch=69, grad_norm=103.586, loss=330.4022, lr=7.57e-04]
Epoch 69 TRAIN:  35%|███▌      | 48/137 [00:13<00:18,  4.75it/s, epoch=69, grad_norm=127.158, loss=291.9344, lr=7.57e-04]
Epoch 69 TRAIN:  37%|███▋      | 51/137 [00:13<00:18,  4.65it/s, epoch=69, grad_norm=127.158, loss=291.9344, lr=7.57e-04]
Epoch 69 TRAIN:  37%|███▋      | 51/137 [00:13<00:18,  4.65it/s, epoch=69, grad_norm=114.963, loss=292.0532, lr=7.57e-04]
Epoch 69 TRAIN:  37%|███▋      | 51/137 [00:13<00:18,  4.65it/s, epoch=69, grad_norm=131.397, loss=261.1079, lr=7.57e-04]
Epoch 69 TRAIN:  37%|███▋      | 51/137 [00:13<00:18,  4.65it/s, epoch=69, grad_norm=94.651, loss=277.3258, lr=7.57e-04] 
Epoch 69 TRAIN:  39%|███▉      | 54/137 [00:13<00:17,  4.63it/s, epoch=69, grad_norm=94.651, loss=277.3258, lr=7.57e-04]
Epoch 69 TRAIN:  39%|███▉      | 54/137 [00:14<00:17,  4.63it/s, epoch=69, grad_norm=91.151, loss=276.6614, lr=7.57e-04]
Epoch 69 TRAIN:  39%|███▉      | 54/137 [00:14<00:17,  4.63it/s, epoch=69, grad_norm=98.316, loss=285.1173, lr=7.57e-04]
Epoch 69 TRAIN:  39%|███▉      | 54/137 [00:14<00:17,  4.63it/s, epoch=69, grad_norm=69.947, loss=246.6088, lr=7.57e-04]
Epoch 69 TRAIN:  42%|████▏     | 57/137 [00:14<00:17,  4.53it/s, epoch=69, grad_norm=69.947, loss=246.6088, lr=7.57e-04]
Epoch 69 TRAIN:  42%|████▏     | 57/137 [00:14<00:17,  4.53it/s, epoch=69, grad_norm=105.349, loss=267.7293, lr=7.57e-04]
Epoch 69 TRAIN:  42%|████▏     | 57/137 [00:15<00:17,  4.53it/s, epoch=69, grad_norm=90.419, loss=282.6945, lr=7.58e-04] 
Epoch 69 TRAIN:  42%|████▏     | 57/137 [00:15<00:17,  4.53it/s, epoch=69, grad_norm=151.768, loss=320.5071, lr=7.58e-04]
Epoch 69 TRAIN:  44%|████▍     | 60/137 [00:15<00:16,  4.62it/s, epoch=69, grad_norm=151.768, loss=320.5071, lr=7.58e-04]
Epoch 69 TRAIN:  44%|████▍     | 60/137 [00:15<00:16,  4.62it/s, epoch=69, grad_norm=118.935, loss=249.9571, lr=7.58e-04]
Epoch 69 TRAIN:  44%|████▍     | 60/137 [00:15<00:16,  4.62it/s, epoch=69, grad_norm=126.299, loss=333.4731, lr=7.58e-04]
Epoch 69 TRAIN:  44%|████▍     | 60/137 [00:15<00:16,  4.62it/s, epoch=69, grad_norm=178.063, loss=297.8050, lr=7.58e-04]
Epoch 69 TRAIN:  46%|████▌     | 63/137 [00:15<00:16,  4.56it/s, epoch=69, grad_norm=178.063, loss=297.8050, lr=7.58e-04]
Epoch 69 TRAIN:  46%|████▌     | 63/137 [00:16<00:16,  4.56it/s, epoch=69, grad_norm=137.403, loss=312.7021, lr=7.58e-04]
Epoch 69 TRAIN:  46%|████▌     | 63/137 [00:16<00:16,  4.56it/s, epoch=69, grad_norm=101.071, loss=278.8517, lr=7.58e-04]
Epoch 69 TRAIN:  46%|████▌     | 63/137 [00:16<00:16,  4.56it/s, epoch=69, grad_norm=116.522, loss=251.9577, lr=7.58e-04]
Epoch 69 TRAIN:  48%|████▊     | 66/137 [00:16<00:15,  4.54it/s, epoch=69, grad_norm=116.522, loss=251.9577, lr=7.58e-04]
Epoch 69 TRAIN:  48%|████▊     | 66/137 [00:16<00:15,  4.54it/s, epoch=69, grad_norm=125.916, loss=261.7732, lr=7.58e-04]
Epoch 69 TRAIN:  48%|████▊     | 66/137 [00:17<00:15,  4.54it/s, epoch=69, grad_norm=102.167, loss=295.2611, lr=7.58e-04]
Epoch 69 TRAIN:  48%|████▊     | 66/137 [00:17<00:15,  4.54it/s, epoch=69, grad_norm=112.800, loss=281.1166, lr=7.58e-04]
Epoch 69 TRAIN:  50%|█████     | 69/137 [00:17<00:15,  4.36it/s, epoch=69, grad_norm=112.800, loss=281.1166, lr=7.58e-04]
Epoch 69 TRAIN:  50%|█████     | 69/137 [00:17<00:15,  4.36it/s, epoch=69, grad_norm=127.825, loss=244.6742, lr=7.58e-04]
Epoch 69 TRAIN:  50%|█████     | 69/137 [00:17<00:15,  4.36it/s, epoch=69, grad_norm=151.952, loss=236.2598, lr=7.58e-04]
Epoch 69 TRAIN:  50%|█████     | 69/137 [00:17<00:15,  4.36it/s, epoch=69, grad_norm=125.534, loss=253.2363, lr=7.59e-04]
Epoch 69 TRAIN:  53%|█████▎    | 72/137 [00:17<00:14,  4.52it/s, epoch=69, grad_norm=125.534, loss=253.2363, lr=7.59e-04]
Epoch 69 TRAIN:  53%|█████▎    | 72/137 [00:18<00:14,  4.52it/s, epoch=69, grad_norm=278.695, loss=333.3223, lr=7.59e-04]
Epoch 69 TRAIN:  53%|█████▎    | 72/137 [00:18<00:14,  4.52it/s, epoch=69, grad_norm=150.575, loss=323.1105, lr=7.59e-04]
Epoch 69 TRAIN:  53%|█████▎    | 72/137 [00:18<00:14,  4.52it/s, epoch=69, grad_norm=129.367, loss=298.3431, lr=7.59e-04]
Epoch 69 TRAIN:  55%|█████▍    | 75/137 [00:18<00:13,  4.72it/s, epoch=69, grad_norm=129.367, loss=298.3431, lr=7.59e-04]
Epoch 69 TRAIN:  55%|█████▍    | 75/137 [00:18<00:13,  4.72it/s, epoch=69, grad_norm=121.521, loss=270.9296, lr=7.59e-04]
Epoch 69 TRAIN:  55%|█████▍    | 75/137 [00:18<00:13,  4.72it/s, epoch=69, grad_norm=176.378, loss=321.4843, lr=7.59e-04]
Epoch 69 TRAIN:  55%|█████▍    | 75/137 [00:19<00:13,  4.72it/s, epoch=69, grad_norm=208.388, loss=260.2785, lr=7.59e-04]
Epoch 69 TRAIN:  57%|█████▋    | 78/137 [00:19<00:12,  4.61it/s, epoch=69, grad_norm=208.388, loss=260.2785, lr=7.59e-04]
Epoch 69 TRAIN:  57%|█████▋    | 78/137 [00:19<00:12,  4.61it/s, epoch=69, grad_norm=104.525, loss=311.5850, lr=7.59e-04]
Epoch 69 TRAIN:  57%|█████▋    | 78/137 [00:19<00:12,  4.61it/s, epoch=69, grad_norm=188.442, loss=304.0075, lr=7.59e-04]
Epoch 69 TRAIN:  57%|█████▋    | 78/137 [00:19<00:12,  4.61it/s, epoch=69, grad_norm=99.638, loss=274.2673, lr=7.59e-04] 
Epoch 69 TRAIN:  59%|█████▉    | 81/137 [00:19<00:12,  4.58it/s, epoch=69, grad_norm=99.638, loss=274.2673, lr=7.59e-04]
Epoch 69 TRAIN:  59%|█████▉    | 81/137 [00:19<00:12,  4.58it/s, epoch=69, grad_norm=115.484, loss=305.1003, lr=7.59e-04]
Epoch 69 TRAIN:  59%|█████▉    | 81/137 [00:20<00:12,  4.58it/s, epoch=69, grad_norm=161.309, loss=254.4839, lr=7.59e-04]
Epoch 69 TRAIN:  59%|█████▉    | 81/137 [00:20<00:12,  4.58it/s, epoch=69, grad_norm=162.082, loss=284.8572, lr=7.60e-04]
Epoch 69 TRAIN:  61%|██████▏   | 84/137 [00:20<00:11,  4.51it/s, epoch=69, grad_norm=162.082, loss=284.8572, lr=7.60e-04]
Epoch 69 TRAIN:  61%|██████▏   | 84/137 [00:20<00:11,  4.51it/s, epoch=69, grad_norm=145.018, loss=276.1017, lr=7.60e-04]
Epoch 69 TRAIN:  61%|██████▏   | 84/137 [00:20<00:11,  4.51it/s, epoch=69, grad_norm=141.510, loss=271.8488, lr=7.60e-04]
Epoch 69 TRAIN:  61%|██████▏   | 84/137 [00:21<00:11,  4.51it/s, epoch=69, grad_norm=282.708, loss=156.1500, lr=7.60e-04]
Epoch 69 TRAIN:  64%|██████▎   | 87/137 [00:21<00:10,  4.57it/s, epoch=69, grad_norm=282.708, loss=156.1500, lr=7.60e-04]
Epoch 69 TRAIN:  64%|██████▎   | 87/137 [00:21<00:10,  4.57it/s, epoch=69, grad_norm=183.656, loss=266.0921, lr=7.60e-04]
Epoch 69 TRAIN:  64%|██████▎   | 87/137 [00:21<00:10,  4.57it/s, epoch=69, grad_norm=104.127, loss=254.4895, lr=7.60e-04]
Epoch 69 TRAIN:  64%|██████▎   | 87/137 [00:21<00:10,  4.57it/s, epoch=69, grad_norm=146.085, loss=169.6572, lr=7.60e-04]
Epoch 69 TRAIN:  66%|██████▌   | 90/137 [00:21<00:10,  4.40it/s, epoch=69, grad_norm=146.085, loss=169.6572, lr=7.60e-04]
Epoch 69 TRAIN:  66%|██████▌   | 90/137 [00:22<00:10,  4.40it/s, epoch=69, grad_norm=233.769, loss=315.2969, lr=7.60e-04]
Epoch 69 TRAIN:  66%|██████▌   | 90/137 [00:22<00:10,  4.40it/s, epoch=69, grad_norm=252.886, loss=290.0727, lr=7.60e-04]
Epoch 69 TRAIN:  66%|██████▌   | 90/137 [00:22<00:10,  4.40it/s, epoch=69, grad_norm=96.075, loss=191.2479, lr=7.60e-04] 
Epoch 69 TRAIN:  68%|██████▊   | 93/137 [00:22<00:09,  4.51it/s, epoch=69, grad_norm=96.075, loss=191.2479, lr=7.60e-04]
Epoch 69 TRAIN:  68%|██████▊   | 93/137 [00:22<00:09,  4.51it/s, epoch=69, grad_norm=111.444, loss=249.8481, lr=7.60e-04]
Epoch 69 TRAIN:  68%|██████▊   | 93/137 [00:22<00:09,  4.51it/s, epoch=69, grad_norm=157.233, loss=292.9592, lr=7.60e-04]
Epoch 69 TRAIN:  68%|██████▊   | 93/137 [00:23<00:09,  4.51it/s, epoch=69, grad_norm=301.830, loss=346.1978, lr=7.60e-04]
Epoch 69 TRAIN:  70%|███████   | 96/137 [00:23<00:09,  4.49it/s, epoch=69, grad_norm=301.830, loss=346.1978, lr=7.60e-04]
Epoch 69 TRAIN:  70%|███████   | 96/137 [00:23<00:09,  4.49it/s, epoch=69, grad_norm=212.033, loss=313.8312, lr=7.61e-04]
Epoch 69 TRAIN:  70%|███████   | 96/137 [00:23<00:09,  4.49it/s, epoch=69, grad_norm=170.912, loss=220.6490, lr=7.61e-04]
Epoch 69 TRAIN:  70%|███████   | 96/137 [00:23<00:09,  4.49it/s, epoch=69, grad_norm=165.574, loss=217.2012, lr=7.61e-04]
Epoch 69 TRAIN:  72%|███████▏  | 99/137 [00:23<00:08,  4.50it/s, epoch=69, grad_norm=165.574, loss=217.2012, lr=7.61e-04]
Epoch 69 TRAIN:  72%|███████▏  | 99/137 [00:24<00:08,  4.50it/s, epoch=69, grad_norm=118.629, loss=245.8071, lr=7.61e-04]2026-04-16 20:27:42,026 DEBUG TRAIN | steps/sec 4.133| Batch 69/100 loss 245.807053 loss_att 192.584961 loss_ctc 369.991943 th_accuracy 0.445312 lr 7.6080e-04 grad_norm 118.628677 rank 0

Epoch 69 TRAIN:  72%|███████▏  | 99/137 [00:24<00:08,  4.50it/s, epoch=69, grad_norm=146.664, loss=292.4090, lr=7.61e-04]
Epoch 69 TRAIN:  72%|███████▏  | 99/137 [00:24<00:08,  4.50it/s, epoch=69, grad_norm=161.112, loss=231.9603, lr=7.61e-04]
Epoch 69 TRAIN:  74%|███████▍  | 102/137 [00:24<00:07,  4.43it/s, epoch=69, grad_norm=161.112, loss=231.9603, lr=7.61e-04]
Epoch 69 TRAIN:  74%|███████▍  | 102/137 [00:24<00:07,  4.43it/s, epoch=69, grad_norm=125.243, loss=279.1902, lr=7.61e-04]
Epoch 69 TRAIN:  74%|███████▍  | 102/137 [00:24<00:07,  4.43it/s, epoch=69, grad_norm=114.069, loss=174.8219, lr=7.61e-04]
Epoch 69 TRAIN:  74%|███████▍  | 102/137 [00:25<00:07,  4.43it/s, epoch=69, grad_norm=290.878, loss=329.9785, lr=7.61e-04]
Epoch 69 TRAIN:  77%|███████▋  | 105/137 [00:25<00:07,  4.52it/s, epoch=69, grad_norm=290.878, loss=329.9785, lr=7.61e-04]
Epoch 69 TRAIN:  77%|███████▋  | 105/137 [00:25<00:07,  4.52it/s, epoch=69, grad_norm=122.485, loss=247.5247, lr=7.61e-04]
Epoch 69 TRAIN:  77%|███████▋  | 105/137 [00:25<00:07,  4.52it/s, epoch=69, grad_norm=93.887, loss=235.8739, lr=7.61e-04] 
Epoch 69 TRAIN:  77%|███████▋  | 105/137 [00:25<00:07,  4.52it/s, epoch=69, grad_norm=103.119, loss=203.7585, lr=7.61e-04]
Epoch 69 TRAIN:  79%|███████▉  | 108/137 [00:25<00:06,  4.36it/s, epoch=69, grad_norm=103.119, loss=203.7585, lr=7.61e-04]
Epoch 69 TRAIN:  79%|███████▉  | 108/137 [00:26<00:06,  4.36it/s, epoch=69, grad_norm=181.560, loss=273.0493, lr=7.62e-04]
Epoch 69 TRAIN:  79%|███████▉  | 108/137 [00:26<00:06,  4.36it/s, epoch=69, grad_norm=108.591, loss=243.5942, lr=7.62e-04]
Epoch 69 TRAIN:  79%|███████▉  | 108/137 [00:26<00:06,  4.36it/s, epoch=69, grad_norm=143.774, loss=303.5621, lr=7.62e-04]
Epoch 69 TRAIN:  81%|████████  | 111/137 [00:26<00:05,  4.59it/s, epoch=69, grad_norm=143.774, loss=303.5621, lr=7.62e-04]
Epoch 69 TRAIN:  81%|████████  | 111/137 [00:26<00:05,  4.59it/s, epoch=69, grad_norm=107.133, loss=192.6277, lr=7.62e-04]
Epoch 69 TRAIN:  81%|████████  | 111/137 [00:26<00:05,  4.59it/s, epoch=69, grad_norm=276.064, loss=323.6760, lr=7.62e-04]
Epoch 69 TRAIN:  81%|████████  | 111/137 [00:27<00:05,  4.59it/s, epoch=69, grad_norm=163.746, loss=255.2873, lr=7.62e-04]
Epoch 69 TRAIN:  83%|████████▎ | 114/137 [00:27<00:05,  4.44it/s, epoch=69, grad_norm=163.746, loss=255.2873, lr=7.62e-04]
Epoch 69 TRAIN:  83%|████████▎ | 114/137 [00:27<00:05,  4.44it/s, epoch=69, grad_norm=352.085, loss=338.0168, lr=7.62e-04]
Epoch 69 TRAIN:  83%|████████▎ | 114/137 [00:27<00:05,  4.44it/s, epoch=69, grad_norm=110.912, loss=270.2328, lr=7.62e-04]
Epoch 69 TRAIN:  83%|████████▎ | 114/137 [00:27<00:05,  4.44it/s, epoch=69, grad_norm=150.870, loss=293.8402, lr=7.62e-04]
Epoch 69 TRAIN:  85%|████████▌ | 117/137 [00:27<00:04,  4.37it/s, epoch=69, grad_norm=150.870, loss=293.8402, lr=7.62e-04]
Epoch 69 TRAIN:  85%|████████▌ | 117/137 [00:28<00:04,  4.37it/s, epoch=69, grad_norm=204.244, loss=256.7066, lr=7.62e-04]
Epoch 69 TRAIN:  85%|████████▌ | 117/137 [00:28<00:04,  4.37it/s, epoch=69, grad_norm=499.532, loss=306.5337, lr=7.62e-04]
Epoch 69 TRAIN:  85%|████████▌ | 117/137 [00:28<00:04,  4.37it/s, epoch=69, grad_norm=209.441, loss=272.6181, lr=7.62e-04]
Epoch 69 TRAIN:  88%|████████▊ | 120/137 [00:28<00:03,  4.46it/s, epoch=69, grad_norm=209.441, loss=272.6181, lr=7.62e-04]
Epoch 69 TRAIN:  88%|████████▊ | 120/137 [00:28<00:03,  4.46it/s, epoch=69, grad_norm=235.127, loss=319.1476, lr=7.62e-04]
Epoch 69 TRAIN:  88%|████████▊ | 120/137 [00:29<00:03,  4.46it/s, epoch=69, grad_norm=147.531, loss=271.5888, lr=7.63e-04]
Epoch 69 TRAIN:  88%|████████▊ | 120/137 [00:29<00:03,  4.46it/s, epoch=69, grad_norm=204.236, loss=329.8945, lr=7.63e-04]
Epoch 69 TRAIN:  90%|████████▉ | 123/137 [00:29<00:03,  4.56it/s, epoch=69, grad_norm=204.236, loss=329.8945, lr=7.63e-04]
Epoch 69 TRAIN:  90%|████████▉ | 123/137 [00:29<00:03,  4.56it/s, epoch=69, grad_norm=281.713, loss=264.8418, lr=7.63e-04]
Epoch 69 TRAIN:  90%|████████▉ | 123/137 [00:29<00:03,  4.56it/s, epoch=69, grad_norm=228.299, loss=297.3665, lr=7.63e-04]
Epoch 69 TRAIN:  90%|████████▉ | 123/137 [00:29<00:03,  4.56it/s, epoch=69, grad_norm=214.249, loss=245.5466, lr=7.63e-04]
Epoch 69 TRAIN:  92%|█████████▏| 126/137 [00:29<00:02,  4.32it/s, epoch=69, grad_norm=214.249, loss=245.5466, lr=7.63e-04]
Epoch 69 TRAIN:  92%|█████████▏| 126/137 [00:30<00:02,  4.32it/s, epoch=69, grad_norm=328.663, loss=361.7842, lr=7.63e-04]
Epoch 69 TRAIN:  92%|█████████▏| 126/137 [00:30<00:02,  4.32it/s, epoch=69, grad_norm=206.317, loss=308.0657, lr=7.63e-04]
Epoch 69 TRAIN:  92%|█████████▏| 126/137 [00:30<00:02,  4.32it/s, epoch=69, grad_norm=201.222, loss=272.9011, lr=7.63e-04]
Epoch 69 TRAIN:  94%|█████████▍| 129/137 [00:30<00:01,  4.48it/s, epoch=69, grad_norm=201.222, loss=272.9011, lr=7.63e-04]
Epoch 69 TRAIN:  94%|█████████▍| 129/137 [00:30<00:01,  4.48it/s, epoch=69, grad_norm=148.908, loss=293.9824, lr=7.63e-04]
Epoch 69 TRAIN:  94%|█████████▍| 129/137 [00:31<00:01,  4.48it/s, epoch=69, grad_norm=166.773, loss=318.5963, lr=7.63e-04]
Epoch 69 TRAIN:  94%|█████████▍| 129/137 [00:31<00:01,  4.48it/s, epoch=69, grad_norm=147.717, loss=284.2577, lr=7.63e-04]
Epoch 69 TRAIN:  96%|█████████▋| 132/137 [00:31<00:01,  4.42it/s, epoch=69, grad_norm=147.717, loss=284.2577, lr=7.63e-04]
Epoch 69 TRAIN:  96%|█████████▋| 132/137 [00:31<00:01,  4.42it/s, epoch=69, grad_norm=163.677, loss=329.4287, lr=7.63e-04]
Epoch 69 TRAIN:  96%|█████████▋| 132/137 [00:31<00:01,  4.42it/s, epoch=69, grad_norm=204.473, loss=341.7934, lr=7.64e-04]
Epoch 69 TRAIN:  96%|█████████▋| 132/137 [00:31<00:01,  4.42it/s, epoch=69, grad_norm=218.243, loss=356.4951, lr=7.64e-04]
Epoch 69 TRAIN:  99%|█████████▊| 135/137 [00:31<00:00,  4.39it/s, epoch=69, grad_norm=218.243, loss=356.4951, lr=7.64e-04]
Epoch 69 TRAIN:  99%|█████████▊| 135/137 [00:31<00:00,  4.22it/s, epoch=69, grad_norm=218.243, loss=356.4951, lr=7.64e-04]

CV:   0%|          | 0/6 [00:00<?, ?it/s]/home/rookie/Documents/batuka/projects/wenet/wenet/wenet/utils/train_utils.py:689: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  amp_autocast(enabled=dtype is not None,

CV:  17%|█▋        | 1/6 [00:00<00:03,  1.31it/s]
                                                 
2026-04-16 20:27:50,812 INFO Epoch 69 Step 9545 CV info lr 7.6360e-04 cv_loss 434.9723843661222 rank 0 acc 0.3567115068435669
2026-04-16 20:27:50,814 INFO [Rank 0] Checkpoint: save to checkpoint exp/conformer_mongolian/epoch_69.pt
2026-04-16 20:27:50,923 INFO Epoch 70 Step 9545 TRAIN info lr 7.6360e-04 rank 0
2026-04-16 20:27:50,939 INFO using accumulate grad, new batch size is 1 times larger than before

Epoch 70 TRAIN:   0%|          | 0/135 [00:00<?, ?it/s]
Epoch 70 TRAIN:   0%|          | 0/135 [00:03<?, ?it/s, epoch=70, grad_norm=364.729, loss=305.4032, lr=7.64e-04]
Epoch 70 TRAIN:   1%|          | 1/135 [00:03<08:48,  3.94s/it, epoch=70, grad_norm=364.729, loss=305.4032, lr=7.64e-04]
Epoch 70 TRAIN:   1%|          | 1/135 [00:04<08:48,  3.94s/it, epoch=70, grad_norm=470.120, loss=423.3651, lr=7.64e-04]
Epoch 70 TRAIN:   1%|          | 1/135 [00:04<08:48,  3.94s/it, epoch=70, grad_norm=264.531, loss=291.5968, lr=7.64e-04]
Epoch 70 TRAIN:   1%|          | 1/135 [00:04<08:48,  3.94s/it, epoch=70, grad_norm=306.840, loss=296.9860, lr=7.64e-04]
Epoch 70 TRAIN:   1%|          | 1/135 [00:04<08:48,  3.94s/it, epoch=70, grad_norm=251.009, loss=312.9612, lr=7.64e-04]
Epoch 70 TRAIN:   1%|          | 1/135 [00:04<08:48,  3.94s/it, epoch=70, grad_norm=237.237, loss=294.9822, lr=7.64e-04]
Epoch 70 TRAIN:   1%|          | 1/135 [00:04<08:48,  3.94s/it, epoch=70, grad_norm=179.358, loss=262.6339, lr=7.64e-04]
Epoch 70 TRAIN:   5%|▌         | 7/135 [00:04<01:03,  2.02it/s, epoch=70, grad_norm=179.358, loss=262.6339, lr=7.64e-04]
Epoch 70 TRAIN:   5%|▌         | 7/135 [00:04<01:03,  2.02it/s, epoch=70, grad_norm=160.713, loss=225.7333, lr=7.64e-04]
Epoch 70 TRAIN:   5%|▌         | 7/135 [00:04<01:03,  2.02it/s, epoch=70, grad_norm=112.444, loss=284.1964, lr=7.64e-04]
Epoch 70 TRAIN:   5%|▌         | 7/135 [00:04<01:03,  2.02it/s, epoch=70, grad_norm=156.198, loss=305.7893, lr=7.64e-04]
Epoch 70 TRAIN:   5%|▌         | 7/135 [00:05<01:03,  2.02it/s, epoch=70, grad_norm=127.142, loss=299.4453, lr=7.64e-04]
Epoch 70 TRAIN:   8%|▊         | 11/135 [00:05<00:41,  2.96it/s, epoch=70, grad_norm=127.142, loss=299.4453, lr=7.64e-04]
Epoch 70 TRAIN:   8%|▊         | 11/135 [00:05<00:41,  2.96it/s, epoch=70, grad_norm=122.032, loss=300.9760, lr=7.65e-04]
Epoch 70 TRAIN:   8%|▊         | 11/135 [00:05<00:41,  2.96it/s, epoch=70, grad_norm=104.518, loss=310.2712, lr=7.65e-04]
Epoch 70 TRAIN:   8%|▊         | 11/135 [00:05<00:41,  2.96it/s, epoch=70, grad_norm=102.346, loss=305.6461, lr=7.65e-04]
Epoch 70 TRAIN:   8%|▊         | 11/135 [00:05<00:41,  2.96it/s, epoch=70, grad_norm=112.097, loss=310.2764, lr=7.65e-04]
Epoch 70 TRAIN:  11%|█         | 15/135 [00:05<00:32,  3.74it/s, epoch=70, grad_norm=112.097, loss=310.2764, lr=7.65e-04]
Epoch 70 TRAIN:  11%|█         | 15/135 [00:05<00:32,  3.74it/s, epoch=70, grad_norm=134.504, loss=283.2779, lr=7.65e-04]
Epoch 70 TRAIN:  11%|█         | 15/135 [00:06<00:32,  3.74it/s, epoch=70, grad_norm=135.895, loss=275.7862, lr=7.65e-04]
Epoch 70 TRAIN:  11%|█         | 15/135 [00:06<00:32,  3.74it/s, epoch=70, grad_norm=76.478, loss=288.7357, lr=7.65e-04] 
Epoch 70 TRAIN:  11%|█         | 15/135 [00:06<00:32,  3.74it/s, epoch=70, grad_norm=111.699, loss=275.4095, lr=7.65e-04]
Epoch 70 TRAIN:  14%|█▍        | 19/135 [00:06<00:27,  4.27it/s, epoch=70, grad_norm=111.699, loss=275.4095, lr=7.65e-04]
Epoch 70 TRAIN:  14%|█▍        | 19/135 [00:06<00:27,  4.27it/s, epoch=70, grad_norm=73.941, loss=259.9599, lr=7.65e-04] 
Epoch 70 TRAIN:  14%|█▍        | 19/135 [00:06<00:27,  4.27it/s, epoch=70, grad_norm=80.466, loss=275.6513, lr=7.65e-04]
Epoch 70 TRAIN:  14%|█▍        | 19/135 [00:07<00:27,  4.27it/s, epoch=70, grad_norm=114.876, loss=285.1161, lr=7.65e-04]
Epoch 70 TRAIN:  14%|█▍        | 19/135 [00:07<00:27,  4.27it/s, epoch=70, grad_norm=115.104, loss=292.9633, lr=7.65e-04]
Epoch 70 TRAIN:  17%|█▋        | 23/135 [00:07<00:25,  4.43it/s, epoch=70, grad_norm=115.104, loss=292.9633, lr=7.65e-04]
Epoch 70 TRAIN:  17%|█▋        | 23/135 [00:07<00:25,  4.43it/s, epoch=70, grad_norm=116.682, loss=303.8986, lr=7.66e-04]
Epoch 70 TRAIN:  17%|█▋        | 23/135 [00:07<00:25,  4.43it/s, epoch=70, grad_norm=95.635, loss=293.5435, lr=7.66e-04] 
Epoch 70 TRAIN:  17%|█▋        | 23/135 [00:07<00:25,  4.43it/s, epoch=70, grad_norm=129.619, loss=317.5052, lr=7.66e-04]
Epoch 70 TRAIN:  19%|█▉        | 26/135 [00:07<00:24,  4.52it/s, epoch=70, grad_norm=129.619, loss=317.5052, lr=7.66e-04]
Epoch 70 TRAIN:  19%|█▉        | 26/135 [00:08<00:24,  4.52it/s, epoch=70, grad_norm=146.289, loss=294.1916, lr=7.66e-04]
Epoch 70 TRAIN:  19%|█▉        | 26/135 [00:08<00:24,  4.52it/s, epoch=70, grad_norm=127.636, loss=262.7390, lr=7.66e-04]
Epoch 70 TRAIN:  19%|█▉        | 26/135 [00:08<00:24,  4.52it/s, epoch=70, grad_norm=117.897, loss=325.7158, lr=7.66e-04]
Epoch 70 TRAIN:  21%|██▏       | 29/135 [00:08<00:22,  4.71it/s, epoch=70, grad_norm=117.897, loss=325.7158, lr=7.66e-04]
Epoch 70 TRAIN:  21%|██▏       | 29/135 [00:08<00:22,  4.71it/s, epoch=70, grad_norm=95.519, loss=299.0052, lr=7.66e-04] 
Epoch 70 TRAIN:  21%|██▏       | 29/135 [00:08<00:22,  4.71it/s, epoch=70, grad_norm=197.928, loss=238.2583, lr=7.66e-04]
Epoch 70 TRAIN:  21%|██▏       | 29/135 [00:09<00:22,  4.71it/s, epoch=70, grad_norm=157.965, loss=308.5744, lr=7.66e-04]
Epoch 70 TRAIN:  24%|██▎       | 32/135 [00:09<00:21,  4.86it/s, epoch=70, grad_norm=157.965, loss=308.5744, lr=7.66e-04]
Epoch 70 TRAIN:  24%|██▎       | 32/135 [00:09<00:21,  4.86it/s, epoch=70, grad_norm=131.549, loss=313.9937, lr=7.66e-04]
Epoch 70 TRAIN:  24%|██▎       | 32/135 [00:09<00:21,  4.86it/s, epoch=70, grad_norm=190.838, loss=227.1544, lr=7.66e-04]
Epoch 70 TRAIN:  24%|██▎       | 32/135 [00:09<00:21,  4.86it/s, epoch=70, grad_norm=162.998, loss=242.9592, lr=7.66e-04]
Epoch 70 TRAIN:  26%|██▌       | 35/135 [00:09<00:20,  4.87it/s, epoch=70, grad_norm=162.998, loss=242.9592, lr=7.66e-04]
Epoch 70 TRAIN:  26%|██▌       | 35/135 [00:09<00:20,  4.87it/s, epoch=70, grad_norm=107.063, loss=270.5563, lr=7.66e-04]
Epoch 70 TRAIN:  26%|██▌       | 35/135 [00:10<00:20,  4.87it/s, epoch=70, grad_norm=120.089, loss=307.0659, lr=7.67e-04]
Epoch 70 TRAIN:  26%|██▌       | 35/135 [00:10<00:20,  4.87it/s, epoch=70, grad_norm=121.212, loss=246.1938, lr=7.67e-04]
Epoch 70 TRAIN:  28%|██▊       | 38/135 [00:10<00:20,  4.81it/s, epoch=70, grad_norm=121.212, loss=246.1938, lr=7.67e-04]
Epoch 70 TRAIN:  28%|██▊       | 38/135 [00:10<00:20,  4.81it/s, epoch=70, grad_norm=109.258, loss=253.7755, lr=7.67e-04]
Epoch 70 TRAIN:  28%|██▊       | 38/135 [00:10<00:20,  4.81it/s, epoch=70, grad_norm=96.411, loss=264.7028, lr=7.67e-04] 
Epoch 70 TRAIN:  28%|██▊       | 38/135 [00:10<00:20,  4.81it/s, epoch=70, grad_norm=131.126, loss=299.7671, lr=7.67e-04]
Epoch 70 TRAIN:  30%|███       | 41/135 [00:10<00:18,  4.97it/s, epoch=70, grad_norm=131.126, loss=299.7671, lr=7.67e-04]
Epoch 70 TRAIN:  30%|███       | 41/135 [00:11<00:18,  4.97it/s, epoch=70, grad_norm=103.396, loss=273.0473, lr=7.67e-04]
Epoch 70 TRAIN:  30%|███       | 41/135 [00:11<00:18,  4.97it/s, epoch=70, grad_norm=106.479, loss=276.6591, lr=7.67e-04]
Epoch 70 TRAIN:  30%|███       | 41/135 [00:11<00:18,  4.97it/s, epoch=70, grad_norm=125.099, loss=286.0010, lr=7.67e-04]
Epoch 70 TRAIN:  33%|███▎      | 44/135 [00:11<00:19,  4.73it/s, epoch=70, grad_norm=125.099, loss=286.0010, lr=7.67e-04]
Epoch 70 TRAIN:  33%|███▎      | 44/135 [00:11<00:19,  4.73it/s, epoch=70, grad_norm=89.859, loss=276.1801, lr=7.67e-04] 
Epoch 70 TRAIN:  33%|███▎      | 44/135 [00:12<00:19,  4.73it/s, epoch=70, grad_norm=108.878, loss=305.4707, lr=7.67e-04]
Epoch 70 TRAIN:  33%|███▎      | 44/135 [00:12<00:19,  4.73it/s, epoch=70, grad_norm=107.571, loss=259.0942, lr=7.67e-04]
Epoch 70 TRAIN:  35%|███▍      | 47/135 [00:12<00:18,  4.67it/s, epoch=70, grad_norm=107.571, loss=259.0942, lr=7.67e-04]
Epoch 70 TRAIN:  35%|███▍      | 47/135 [00:12<00:18,  4.67it/s, epoch=70, grad_norm=183.581, loss=329.4918, lr=7.67e-04]
Epoch 70 TRAIN:  35%|███▍      | 47/135 [00:12<00:18,  4.67it/s, epoch=70, grad_norm=134.115, loss=273.5598, lr=7.68e-04]
Epoch 70 TRAIN:  35%|███▍      | 47/135 [00:12<00:18,  4.67it/s, epoch=70, grad_norm=131.419, loss=251.5999, lr=7.68e-04]
Epoch 70 TRAIN:  37%|███▋      | 50/135 [00:12<00:17,  4.77it/s, epoch=70, grad_norm=131.419, loss=251.5999, lr=7.68e-04]
Epoch 70 TRAIN:  37%|███▋      | 50/135 [00:13<00:17,  4.77it/s, epoch=70, grad_norm=91.472, loss=290.7126, lr=7.68e-04] 
Epoch 70 TRAIN:  37%|███▋      | 50/135 [00:13<00:17,  4.77it/s, epoch=70, grad_norm=135.486, loss=296.4441, lr=7.68e-04]
Epoch 70 TRAIN:  37%|███▋      | 50/135 [00:13<00:17,  4.77it/s, epoch=70, grad_norm=218.212, loss=209.3707, lr=7.68e-04]
Epoch 70 TRAIN:  39%|███▉      | 53/135 [00:13<00:16,  4.88it/s, epoch=70, grad_norm=218.212, loss=209.3707, lr=7.68e-04]
Epoch 70 TRAIN:  39%|███▉      | 53/135 [00:13<00:16,  4.88it/s, epoch=70, grad_norm=91.396, loss=277.0109, lr=7.68e-04] 
Epoch 70 TRAIN:  39%|███▉      | 53/135 [00:13<00:16,  4.88it/s, epoch=70, grad_norm=74.465, loss=280.1219, lr=7.68e-04]
Epoch 70 TRAIN:  39%|███▉      | 53/135 [00:14<00:16,  4.88it/s, epoch=70, grad_norm=87.581, loss=257.3022, lr=7.68e-04]
Epoch 70 TRAIN:  41%|████▏     | 56/135 [00:14<00:16,  4.74it/s, epoch=70, grad_norm=87.581, loss=257.3022, lr=7.68e-04]
Epoch 70 TRAIN:  41%|████▏     | 56/135 [00:14<00:16,  4.74it/s, epoch=70, grad_norm=108.520, loss=262.8665, lr=7.68e-04]
Epoch 70 TRAIN:  41%|████▏     | 56/135 [00:14<00:16,  4.74it/s, epoch=70, grad_norm=111.280, loss=292.8336, lr=7.68e-04]
Epoch 70 TRAIN:  41%|████▏     | 56/135 [00:14<00:16,  4.74it/s, epoch=70, grad_norm=128.456, loss=282.9811, lr=7.68e-04]
Epoch 70 TRAIN:  44%|████▎     | 59/135 [00:14<00:16,  4.48it/s, epoch=70, grad_norm=128.456, loss=282.9811, lr=7.68e-04]
Epoch 70 TRAIN:  44%|████▎     | 59/135 [00:15<00:16,  4.48it/s, epoch=70, grad_norm=97.161, loss=265.3333, lr=7.68e-04] 
Epoch 70 TRAIN:  44%|████▎     | 59/135 [00:15<00:16,  4.48it/s, epoch=70, grad_norm=165.088, loss=294.3933, lr=7.68e-04]
Epoch 70 TRAIN:  44%|████▎     | 59/135 [00:15<00:16,  4.48it/s, epoch=70, grad_norm=104.059, loss=282.1723, lr=7.69e-04]
Epoch 70 TRAIN:  46%|████▌     | 62/135 [00:15<00:15,  4.57it/s, epoch=70, grad_norm=104.059, loss=282.1723, lr=7.69e-04]
Epoch 70 TRAIN:  46%|████▌     | 62/135 [00:15<00:15,  4.57it/s, epoch=70, grad_norm=131.459, loss=326.2376, lr=7.69e-04]
Epoch 70 TRAIN:  46%|████▌     | 62/135 [00:15<00:15,  4.57it/s, epoch=70, grad_norm=111.275, loss=294.3292, lr=7.69e-04]
Epoch 70 TRAIN:  46%|████▌     | 62/135 [00:16<00:15,  4.57it/s, epoch=70, grad_norm=104.027, loss=294.7782, lr=7.69e-04]
Epoch 70 TRAIN:  48%|████▊     | 65/135 [00:16<00:15,  4.48it/s, epoch=70, grad_norm=104.027, loss=294.7782, lr=7.69e-04]
Epoch 70 TRAIN:  48%|████▊     | 65/135 [00:16<00:15,  4.48it/s, epoch=70, grad_norm=110.120, loss=313.0319, lr=7.69e-04]
Epoch 70 TRAIN:  48%|████▊     | 65/135 [00:16<00:15,  4.48it/s, epoch=70, grad_norm=106.426, loss=292.4142, lr=7.69e-04]
Epoch 70 TRAIN:  48%|████▊     | 65/135 [00:16<00:15,  4.48it/s, epoch=70, grad_norm=185.473, loss=233.7131, lr=7.69e-04]
Epoch 70 TRAIN:  50%|█████     | 68/135 [00:16<00:14,  4.67it/s, epoch=70, grad_norm=185.473, loss=233.7131, lr=7.69e-04]
Epoch 70 TRAIN:  50%|█████     | 68/135 [00:16<00:14,  4.67it/s, epoch=70, grad_norm=193.086, loss=228.9105, lr=7.69e-04]
Epoch 70 TRAIN:  50%|█████     | 68/135 [00:17<00:14,  4.67it/s, epoch=70, grad_norm=132.911, loss=277.0854, lr=7.69e-04]
Epoch 70 TRAIN:  50%|█████     | 68/135 [00:17<00:14,  4.67it/s, epoch=70, grad_norm=182.134, loss=225.3936, lr=7.69e-04]
Epoch 70 TRAIN:  53%|█████▎    | 71/135 [00:17<00:13,  4.71it/s, epoch=70, grad_norm=182.134, loss=225.3936, lr=7.69e-04]
Epoch 70 TRAIN:  53%|█████▎    | 71/135 [00:17<00:13,  4.71it/s, epoch=70, grad_norm=176.405, loss=307.6509, lr=7.69e-04]
Epoch 70 TRAIN:  53%|█████▎    | 71/135 [00:17<00:13,  4.71it/s, epoch=70, grad_norm=149.278, loss=262.1790, lr=7.69e-04]
Epoch 70 TRAIN:  53%|█████▎    | 71/135 [00:17<00:13,  4.71it/s, epoch=70, grad_norm=138.297, loss=301.8611, lr=7.70e-04]
Epoch 70 TRAIN:  55%|█████▍    | 74/135 [00:17<00:12,  4.74it/s, epoch=70, grad_norm=138.297, loss=301.8611, lr=7.70e-04]
Epoch 70 TRAIN:  55%|█████▍    | 74/135 [00:18<00:12,  4.74it/s, epoch=70, grad_norm=169.828, loss=290.6183, lr=7.70e-04]
Epoch 70 TRAIN:  55%|█████▍    | 74/135 [00:18<00:12,  4.74it/s, epoch=70, grad_norm=124.255, loss=297.2011, lr=7.70e-04]
Epoch 70 TRAIN:  55%|█████▍    | 74/135 [00:18<00:12,  4.74it/s, epoch=70, grad_norm=112.694, loss=267.0974, lr=7.70e-04]
Epoch 70 TRAIN:  57%|█████▋    | 77/135 [00:18<00:12,  4.64it/s, epoch=70, grad_norm=112.694, loss=267.0974, lr=7.70e-04]
Epoch 70 TRAIN:  57%|█████▋    | 77/135 [00:18<00:12,  4.64it/s, epoch=70, grad_norm=100.510, loss=274.4209, lr=7.70e-04]
Epoch 70 TRAIN:  57%|█████▋    | 77/135 [00:19<00:12,  4.64it/s, epoch=70, grad_norm=117.517, loss=253.6199, lr=7.70e-04]
Epoch 70 TRAIN:  57%|█████▋    | 77/135 [00:19<00:12,  4.64it/s, epoch=70, grad_norm=88.429, loss=267.5716, lr=7.70e-04] 
Epoch 70 TRAIN:  59%|█████▉    | 80/135 [00:19<00:11,  4.70it/s, epoch=70, grad_norm=88.429, loss=267.5716, lr=7.70e-04]
Epoch 70 TRAIN:  59%|█████▉    | 80/135 [00:19<00:11,  4.70it/s, epoch=70, grad_norm=143.079, loss=239.9374, lr=7.70e-04]
Epoch 70 TRAIN:  59%|█████▉    | 80/135 [00:19<00:11,  4.70it/s, epoch=70, grad_norm=226.603, loss=324.6759, lr=7.70e-04]
Epoch 70 TRAIN:  59%|█████▉    | 80/135 [00:19<00:11,  4.70it/s, epoch=70, grad_norm=186.759, loss=326.7790, lr=7.70e-04]
Epoch 70 TRAIN:  61%|██████▏   | 83/135 [00:19<00:11,  4.58it/s, epoch=70, grad_norm=186.759, loss=326.7790, lr=7.70e-04]
Epoch 70 TRAIN:  61%|██████▏   | 83/135 [00:20<00:11,  4.58it/s, epoch=70, grad_norm=104.887, loss=282.8409, lr=7.70e-04]
Epoch 70 TRAIN:  61%|██████▏   | 83/135 [00:20<00:11,  4.58it/s, epoch=70, grad_norm=168.656, loss=293.2627, lr=7.70e-04]
Epoch 70 TRAIN:  61%|██████▏   | 83/135 [00:20<00:11,  4.58it/s, epoch=70, grad_norm=101.417, loss=231.3951, lr=7.70e-04]
Epoch 70 TRAIN:  64%|██████▎   | 86/135 [00:20<00:10,  4.46it/s, epoch=70, grad_norm=101.417, loss=231.3951, lr=7.70e-04]
Epoch 70 TRAIN:  64%|██████▎   | 86/135 [00:20<00:10,  4.46it/s, epoch=70, grad_norm=98.820, loss=272.1576, lr=7.71e-04] 
Epoch 70 TRAIN:  64%|██████▎   | 86/135 [00:21<00:10,  4.46it/s, epoch=70, grad_norm=178.069, loss=328.2295, lr=7.71e-04]
Epoch 70 TRAIN:  64%|██████▎   | 86/135 [00:21<00:10,  4.46it/s, epoch=70, grad_norm=144.181, loss=220.7767, lr=7.71e-04]
Epoch 70 TRAIN:  66%|██████▌   | 89/135 [00:21<00:10,  4.45it/s, epoch=70, grad_norm=144.181, loss=220.7767, lr=7.71e-04]
Epoch 70 TRAIN:  66%|██████▌   | 89/135 [00:21<00:10,  4.45it/s, epoch=70, grad_norm=134.914, loss=288.4466, lr=7.71e-04]
Epoch 70 TRAIN:  66%|██████▌   | 89/135 [00:21<00:10,  4.45it/s, epoch=70, grad_norm=157.746, loss=299.8974, lr=7.71e-04]
Epoch 70 TRAIN:  66%|██████▌   | 89/135 [00:22<00:10,  4.45it/s, epoch=70, grad_norm=283.860, loss=168.9929, lr=7.71e-04]
Epoch 70 TRAIN:  68%|██████▊   | 92/135 [00:22<00:09,  4.54it/s, epoch=70, grad_norm=283.860, loss=168.9929, lr=7.71e-04]
Epoch 70 TRAIN:  68%|██████▊   | 92/135 [00:22<00:09,  4.54it/s, epoch=70, grad_norm=124.645, loss=314.2219, lr=7.71e-04]
Epoch 70 TRAIN:  68%|██████▊   | 92/135 [00:22<00:09,  4.54it/s, epoch=70, grad_norm=262.906, loss=179.4198, lr=7.71e-04]
Epoch 70 TRAIN:  68%|██████▊   | 92/135 [00:22<00:09,  4.54it/s, epoch=70, grad_norm=267.640, loss=269.4623, lr=7.71e-04]
Epoch 70 TRAIN:  70%|███████   | 95/135 [00:22<00:09,  4.37it/s, epoch=70, grad_norm=267.640, loss=269.4623, lr=7.71e-04]
Epoch 70 TRAIN:  70%|███████   | 95/135 [00:22<00:09,  4.37it/s, epoch=70, grad_norm=114.926, loss=317.1886, lr=7.71e-04]
Epoch 70 TRAIN:  70%|███████   | 95/135 [00:23<00:09,  4.37it/s, epoch=70, grad_norm=119.330, loss=282.2711, lr=7.71e-04]
Epoch 70 TRAIN:  70%|███████   | 95/135 [00:23<00:09,  4.37it/s, epoch=70, grad_norm=166.489, loss=187.8572, lr=7.71e-04]
Epoch 70 TRAIN:  73%|███████▎  | 98/135 [00:23<00:08,  4.41it/s, epoch=70, grad_norm=166.489, loss=187.8572, lr=7.71e-04]
Epoch 70 TRAIN:  73%|███████▎  | 98/135 [00:23<00:08,  4.41it/s, epoch=70, grad_norm=151.595, loss=268.6049, lr=7.72e-04]
Epoch 70 TRAIN:  73%|███████▎  | 98/135 [00:23<00:08,  4.41it/s, epoch=70, grad_norm=133.402, loss=230.9746, lr=7.72e-04]2026-04-16 20:28:14,797 DEBUG TRAIN | steps/sec 4.120| Batch 70/100 loss 230.974594 loss_att 183.656647 loss_ctc 341.383148 th_accuracy 0.452575 lr 7.7160e-04 grad_norm 133.402206 rank 0

Epoch 70 TRAIN:  73%|███████▎  | 98/135 [00:24<00:08,  4.41it/s, epoch=70, grad_norm=196.203, loss=341.6321, lr=7.72e-04]
Epoch 70 TRAIN:  75%|███████▍  | 101/135 [00:24<00:07,  4.51it/s, epoch=70, grad_norm=196.203, loss=341.6321, lr=7.72e-04]
Epoch 70 TRAIN:  75%|███████▍  | 101/135 [00:24<00:07,  4.51it/s, epoch=70, grad_norm=119.063, loss=225.3967, lr=7.72e-04]
Epoch 70 TRAIN:  75%|███████▍  | 101/135 [00:24<00:07,  4.51it/s, epoch=70, grad_norm=167.437, loss=234.4221, lr=7.72e-04]
Epoch 70 TRAIN:  75%|███████▍  | 101/135 [00:24<00:07,  4.51it/s, epoch=70, grad_norm=124.083, loss=253.8108, lr=7.72e-04]
Epoch 70 TRAIN:  77%|███████▋  | 104/135 [00:24<00:06,  4.51it/s, epoch=70, grad_norm=124.083, loss=253.8108, lr=7.72e-04]
Epoch 70 TRAIN:  77%|███████▋  | 104/135 [00:24<00:06,  4.51it/s, epoch=70, grad_norm=91.128, loss=232.6715, lr=7.72e-04] 
Epoch 70 TRAIN:  77%|███████▋  | 104/135 [00:25<00:06,  4.51it/s, epoch=70, grad_norm=151.832, loss=215.3640, lr=7.72e-04]
Epoch 70 TRAIN:  77%|███████▋  | 104/135 [00:25<00:06,  4.51it/s, epoch=70, grad_norm=121.781, loss=214.7243, lr=7.72e-04]
Epoch 70 TRAIN:  79%|███████▉  | 107/135 [00:25<00:06,  4.32it/s, epoch=70, grad_norm=121.781, loss=214.7243, lr=7.72e-04]
Epoch 70 TRAIN:  79%|███████▉  | 107/135 [00:25<00:06,  4.32it/s, epoch=70, grad_norm=243.842, loss=309.9022, lr=7.72e-04]
Epoch 70 TRAIN:  79%|███████▉  | 107/135 [00:25<00:06,  4.32it/s, epoch=70, grad_norm=143.193, loss=240.0433, lr=7.72e-04]
Epoch 70 TRAIN:  79%|███████▉  | 107/135 [00:26<00:06,  4.32it/s, epoch=70, grad_norm=202.003, loss=320.6357, lr=7.72e-04]
Epoch 70 TRAIN:  81%|████████▏ | 110/135 [00:26<00:05,  4.57it/s, epoch=70, grad_norm=202.003, loss=320.6357, lr=7.72e-04]
Epoch 70 TRAIN:  81%|████████▏ | 110/135 [00:26<00:05,  4.57it/s, epoch=70, grad_norm=179.039, loss=339.6522, lr=7.72e-04]
Epoch 70 TRAIN:  81%|████████▏ | 110/135 [00:26<00:05,  4.57it/s, epoch=70, grad_norm=122.249, loss=286.7078, lr=7.73e-04]
Epoch 70 TRAIN:  81%|████████▏ | 110/135 [00:26<00:05,  4.57it/s, epoch=70, grad_norm=202.227, loss=270.2406, lr=7.73e-04]
Epoch 70 TRAIN:  84%|████████▎ | 113/135 [00:26<00:05,  4.37it/s, epoch=70, grad_norm=202.227, loss=270.2406, lr=7.73e-04]
Epoch 70 TRAIN:  84%|████████▎ | 113/135 [00:27<00:05,  4.37it/s, epoch=70, grad_norm=134.731, loss=311.2292, lr=7.73e-04]
Epoch 70 TRAIN:  84%|████████▎ | 113/135 [00:27<00:05,  4.37it/s, epoch=70, grad_norm=160.114, loss=273.3724, lr=7.73e-04]
Epoch 70 TRAIN:  84%|████████▎ | 113/135 [00:27<00:05,  4.37it/s, epoch=70, grad_norm=428.284, loss=219.1638, lr=7.73e-04]
Epoch 70 TRAIN:  86%|████████▌ | 116/135 [00:27<00:04,  4.66it/s, epoch=70, grad_norm=428.284, loss=219.1638, lr=7.73e-04]
Epoch 70 TRAIN:  86%|████████▌ | 116/135 [00:27<00:04,  4.66it/s, epoch=70, grad_norm=154.643, loss=266.3250, lr=7.73e-04]
Epoch 70 TRAIN:  86%|████████▌ | 116/135 [00:27<00:04,  4.66it/s, epoch=70, grad_norm=176.244, loss=338.8997, lr=7.73e-04]
Epoch 70 TRAIN:  86%|████████▌ | 116/135 [00:27<00:04,  4.66it/s, epoch=70, grad_norm=280.392, loss=335.5589, lr=7.73e-04]
Epoch 70 TRAIN:  88%|████████▊ | 119/135 [00:27<00:03,  4.74it/s, epoch=70, grad_norm=280.392, loss=335.5589, lr=7.73e-04]
Epoch 70 TRAIN:  88%|████████▊ | 119/135 [00:28<00:03,  4.74it/s, epoch=70, grad_norm=134.089, loss=329.5899, lr=7.73e-04]
Epoch 70 TRAIN:  88%|████████▊ | 119/135 [00:28<00:03,  4.74it/s, epoch=70, grad_norm=215.202, loss=240.5913, lr=7.73e-04]
Epoch 70 TRAIN:  88%|████████▊ | 119/135 [00:28<00:03,  4.74it/s, epoch=70, grad_norm=142.304, loss=261.2980, lr=7.73e-04]
Epoch 70 TRAIN:  90%|█████████ | 122/135 [00:28<00:02,  4.67it/s, epoch=70, grad_norm=142.304, loss=261.2980, lr=7.73e-04]
Epoch 70 TRAIN:  90%|█████████ | 122/135 [00:28<00:02,  4.67it/s, epoch=70, grad_norm=142.873, loss=242.9001, lr=7.73e-04]
Epoch 70 TRAIN:  90%|█████████ | 122/135 [00:29<00:02,  4.67it/s, epoch=70, grad_norm=133.681, loss=248.0336, lr=7.74e-04]
Epoch 70 TRAIN:  90%|█████████ | 122/135 [00:29<00:02,  4.67it/s, epoch=70, grad_norm=198.706, loss=287.0928, lr=7.74e-04]
Epoch 70 TRAIN:  93%|█████████▎| 125/135 [00:29<00:02,  4.81it/s, epoch=70, grad_norm=198.706, loss=287.0928, lr=7.74e-04]
Epoch 70 TRAIN:  93%|█████████▎| 125/135 [00:29<00:02,  4.81it/s, epoch=70, grad_norm=342.808, loss=340.3561, lr=7.74e-04]
Epoch 70 TRAIN:  93%|█████████▎| 125/135 [00:29<00:02,  4.81it/s, epoch=70, grad_norm=368.604, loss=350.4476, lr=7.74e-04]
Epoch 70 TRAIN:  93%|█████████▎| 125/135 [00:29<00:02,  4.81it/s, epoch=70, grad_norm=141.292, loss=255.4142, lr=7.74e-04]
Epoch 70 TRAIN:  95%|█████████▍| 128/135 [00:29<00:01,  4.71it/s, epoch=70, grad_norm=141.292, loss=255.4142, lr=7.74e-04]
Epoch 70 TRAIN:  95%|█████████▍| 128/135 [00:30<00:01,  4.71it/s, epoch=70, grad_norm=161.601, loss=305.8600, lr=7.74e-04]
Epoch 70 TRAIN:  95%|█████████▍| 128/135 [00:30<00:01,  4.71it/s, epoch=70, grad_norm=190.497, loss=306.9518, lr=7.74e-04]
Epoch 70 TRAIN:  95%|█████████▍| 128/135 [00:30<00:01,  4.71it/s, epoch=70, grad_norm=175.597, loss=328.6868, lr=7.74e-04]
Epoch 70 TRAIN:  97%|█████████▋| 131/135 [00:30<00:00,  4.46it/s, epoch=70, grad_norm=175.597, loss=328.6868, lr=7.74e-04]
Epoch 70 TRAIN:  97%|█████████▋| 131/135 [00:30<00:00,  4.46it/s, epoch=70, grad_norm=304.309, loss=266.9635, lr=7.74e-04]
Epoch 70 TRAIN:  97%|█████████▋| 131/135 [00:30<00:00,  4.46it/s, epoch=70, grad_norm=204.253, loss=283.3910, lr=7.74e-04]
Epoch 70 TRAIN:  97%|█████████▋| 131/135 [00:31<00:00,  4.46it/s, epoch=70, grad_norm=313.408, loss=363.8167, lr=7.74e-04]
Epoch 70 TRAIN:  99%|█████████▉| 134/135 [00:31<00:00,  4.31it/s, epoch=70, grad_norm=313.408, loss=363.8167, lr=7.74e-04]

CV:   0%|          | 0/6 [00:00<?, ?it/s]/home/rookie/Documents/batuka/projects/wenet/wenet/wenet/utils/train_utils.py:689: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  amp_autocast(enabled=dtype is not None,

CV:  17%|█▋        | 1/6 [00:00<00:03,  1.33it/s]
                                                 
2026-04-16 20:28:22,865 INFO Epoch 70 Step 9679 CV info lr 7.7432e-04 cv_loss 465.6487926136364 rank 0 acc 0.3659451405207316
2026-04-16 20:28:22,866 INFO [Rank 0] Checkpoint: save to checkpoint exp/conformer_mongolian/epoch_70.pt
2026-04-16 20:28:22,979 INFO Epoch 71 Step 9679 TRAIN info lr 7.7432e-04 rank 0
2026-04-16 20:28:22,999 INFO using accumulate grad, new batch size is 1 times larger than before

Epoch 71 TRAIN:   0%|          | 0/134 [00:00<?, ?it/s]
Epoch 71 TRAIN:   0%|          | 0/134 [00:04<?, ?it/s, epoch=71, grad_norm=279.317, loss=241.7018, lr=7.74e-04]
Epoch 71 TRAIN:   1%|          | 1/134 [00:04<09:19,  4.21s/it, epoch=71, grad_norm=279.317, loss=241.7018, lr=7.74e-04]
Epoch 71 TRAIN:   1%|          | 1/134 [00:04<09:19,  4.21s/it, epoch=71, grad_norm=270.049, loss=248.2057, lr=7.74e-04]
Epoch 71 TRAIN:   1%|          | 1/134 [00:04<09:19,  4.21s/it, epoch=71, grad_norm=298.635, loss=274.0021, lr=7.75e-04]
Epoch 71 TRAIN:   1%|          | 1/134 [00:04<09:19,  4.21s/it, epoch=71, grad_norm=240.532, loss=264.3113, lr=7.75e-04]
Epoch 71 TRAIN:   1%|          | 1/134 [00:04<09:19,  4.21s/it, epoch=71, grad_norm=405.799, loss=316.5200, lr=7.75e-04]
Epoch 71 TRAIN:   1%|          | 1/134 [00:04<09:19,  4.21s/it, epoch=71, grad_norm=410.589, loss=355.6864, lr=7.75e-04]
Epoch 71 TRAIN:   1%|          | 1/134 [00:04<09:19,  4.21s/it, epoch=71, grad_norm=199.109, loss=275.5467, lr=7.75e-04]
Epoch 71 TRAIN:   1%|          | 1/134 [00:04<09:19,  4.21s/it, epoch=71, grad_norm=295.163, loss=328.0130, lr=7.75e-04]
Epoch 71 TRAIN:   1%|          | 1/134 [00:04<09:19,  4.21s/it, epoch=71, grad_norm=181.186, loss=316.5521, lr=7.75e-04]
Epoch 71 TRAIN:   7%|▋         | 9/134 [00:04<00:51,  2.43it/s, epoch=71, grad_norm=181.186, loss=316.5521, lr=7.75e-04]
Epoch 71 TRAIN:   7%|▋         | 9/134 [00:05<00:51,  2.43it/s, epoch=71, grad_norm=162.883, loss=313.1627, lr=7.75e-04]
Epoch 71 TRAIN:   7%|▋         | 9/134 [00:05<00:51,  2.43it/s, epoch=71, grad_norm=114.495, loss=305.0598, lr=7.75e-04]
Epoch 71 TRAIN:   7%|▋         | 9/134 [00:05<00:51,  2.43it/s, epoch=71, grad_norm=103.823, loss=290.8034, lr=7.75e-04]
Epoch 71 TRAIN:   7%|▋         | 9/134 [00:05<00:51,  2.43it/s, epoch=71, grad_norm=91.249, loss=287.2581, lr=7.75e-04] 
Epoch 71 TRAIN:  10%|▉         | 13/134 [00:05<00:37,  3.20it/s, epoch=71, grad_norm=91.249, loss=287.2581, lr=7.75e-04]
Epoch 71 TRAIN:  10%|▉         | 13/134 [00:05<00:37,  3.20it/s, epoch=71, grad_norm=109.004, loss=308.1259, lr=7.75e-04]
Epoch 71 TRAIN:  10%|▉         | 13/134 [00:05<00:37,  3.20it/s, epoch=71, grad_norm=136.400, loss=261.0077, lr=7.76e-04]
Epoch 71 TRAIN:  10%|▉         | 13/134 [00:06<00:37,  3.20it/s, epoch=71, grad_norm=103.327, loss=314.1570, lr=7.76e-04]
Epoch 71 TRAIN:  10%|▉         | 13/134 [00:06<00:37,  3.20it/s, epoch=71, grad_norm=144.565, loss=291.7996, lr=7.76e-04]
Epoch 71 TRAIN:  13%|█▎        | 17/134 [00:06<00:31,  3.75it/s, epoch=71, grad_norm=144.565, loss=291.7996, lr=7.76e-04]
Epoch 71 TRAIN:  13%|█▎        | 17/134 [00:06<00:31,  3.75it/s, epoch=71, grad_norm=118.270, loss=311.3095, lr=7.76e-04]
Epoch 71 TRAIN:  13%|█▎        | 17/134 [00:06<00:31,  3.75it/s, epoch=71, grad_norm=100.681, loss=277.9138, lr=7.76e-04]
Epoch 71 TRAIN:  13%|█▎        | 17/134 [00:06<00:31,  3.75it/s, epoch=71, grad_norm=86.127, loss=285.1228, lr=7.76e-04] 
Epoch 71 TRAIN:  13%|█▎        | 17/134 [00:06<00:31,  3.75it/s, epoch=71, grad_norm=91.917, loss=290.2971, lr=7.76e-04]
Epoch 71 TRAIN:  16%|█▌        | 21/134 [00:06<00:27,  4.16it/s, epoch=71, grad_norm=91.917, loss=290.2971, lr=7.76e-04]
Epoch 71 TRAIN:  16%|█▌        | 21/134 [00:07<00:27,  4.16it/s, epoch=71, grad_norm=115.068, loss=244.2094, lr=7.76e-04]
Epoch 71 TRAIN:  16%|█▌        | 21/134 [00:07<00:27,  4.16it/s, epoch=71, grad_norm=76.705, loss=281.9377, lr=7.76e-04] 
Epoch 71 TRAIN:  16%|█▌        | 21/134 [00:07<00:27,  4.16it/s, epoch=71, grad_norm=65.302, loss=249.5652, lr=7.76e-04]
Epoch 71 TRAIN:  18%|█▊        | 24/134 [00:07<00:25,  4.35it/s, epoch=71, grad_norm=65.302, loss=249.5652, lr=7.76e-04]
Epoch 71 TRAIN:  18%|█▊        | 24/134 [00:07<00:25,  4.35it/s, epoch=71, grad_norm=174.599, loss=319.4569, lr=7.76e-04]
Epoch 71 TRAIN:  18%|█▊        | 24/134 [00:07<00:25,  4.35it/s, epoch=71, grad_norm=132.807, loss=223.3596, lr=7.76e-04]
Epoch 71 TRAIN:  18%|█▊        | 24/134 [00:08<00:25,  4.35it/s, epoch=71, grad_norm=97.263, loss=257.4568, lr=7.76e-04] 
Epoch 71 TRAIN:  20%|██        | 27/134 [00:08<00:22,  4.68it/s, epoch=71, grad_norm=97.263, loss=257.4568, lr=7.76e-04]
Epoch 71 TRAIN:  20%|██        | 27/134 [00:08<00:22,  4.68it/s, epoch=71, grad_norm=194.235, loss=306.3217, lr=7.77e-04]
Epoch 71 TRAIN:  20%|██        | 27/134 [00:08<00:22,  4.68it/s, epoch=71, grad_norm=127.359, loss=255.7047, lr=7.77e-04]
Epoch 71 TRAIN:  20%|██        | 27/134 [00:08<00:22,  4.68it/s, epoch=71, grad_norm=82.183, loss=251.4721, lr=7.77e-04] 
Epoch 71 TRAIN:  22%|██▏       | 30/134 [00:08<00:22,  4.69it/s, epoch=71, grad_norm=82.183, loss=251.4721, lr=7.77e-04]
Epoch 71 TRAIN:  22%|██▏       | 30/134 [00:08<00:22,  4.69it/s, epoch=71, grad_norm=181.855, loss=315.3708, lr=7.77e-04]
Epoch 71 TRAIN:  22%|██▏       | 30/134 [00:09<00:22,  4.69it/s, epoch=71, grad_norm=137.765, loss=251.7444, lr=7.77e-04]
Epoch 71 TRAIN:  22%|██▏       | 30/134 [00:09<00:22,  4.69it/s, epoch=71, grad_norm=122.431, loss=277.7493, lr=7.77e-04]
Epoch 71 TRAIN:  25%|██▍       | 33/134 [00:09<00:20,  4.95it/s, epoch=71, grad_norm=122.431, loss=277.7493, lr=7.77e-04]
Epoch 71 TRAIN:  25%|██▍       | 33/134 [00:09<00:20,  4.95it/s, epoch=71, grad_norm=90.564, loss=277.7210, lr=7.77e-04] 
Epoch 71 TRAIN:  25%|██▍       | 33/134 [00:09<00:20,  4.95it/s, epoch=71, grad_norm=113.083, loss=299.9435, lr=7.77e-04]
Epoch 71 TRAIN:  25%|██▍       | 33/134 [00:09<00:20,  4.95it/s, epoch=71, grad_norm=191.364, loss=324.9190, lr=7.77e-04]
Epoch 71 TRAIN:  27%|██▋       | 36/134 [00:09<00:20,  4.83it/s, epoch=71, grad_norm=191.364, loss=324.9190, lr=7.77e-04]
Epoch 71 TRAIN:  27%|██▋       | 36/134 [00:10<00:20,  4.83it/s, epoch=71, grad_norm=124.803, loss=272.5201, lr=7.77e-04]
Epoch 71 TRAIN:  27%|██▋       | 36/134 [00:10<00:20,  4.83it/s, epoch=71, grad_norm=116.042, loss=263.7622, lr=7.77e-04]
Epoch 71 TRAIN:  27%|██▋       | 36/134 [00:10<00:20,  4.83it/s, epoch=71, grad_norm=149.365, loss=301.8001, lr=7.77e-04]
Epoch 71 TRAIN:  29%|██▉       | 39/134 [00:10<00:19,  4.80it/s, epoch=71, grad_norm=149.365, loss=301.8001, lr=7.77e-04]
Epoch 71 TRAIN:  29%|██▉       | 39/134 [00:10<00:19,  4.80it/s, epoch=71, grad_norm=129.871, loss=279.8186, lr=7.78e-04]
Epoch 71 TRAIN:  29%|██▉       | 39/134 [00:10<00:19,  4.80it/s, epoch=71, grad_norm=185.805, loss=276.1915, lr=7.78e-04]
Epoch 71 TRAIN:  29%|██▉       | 39/134 [00:11<00:19,  4.80it/s, epoch=71, grad_norm=147.340, loss=258.4294, lr=7.78e-04]
Epoch 71 TRAIN:  31%|███▏      | 42/134 [00:11<00:19,  4.80it/s, epoch=71, grad_norm=147.340, loss=258.4294, lr=7.78e-04]
Epoch 71 TRAIN:  31%|███▏      | 42/134 [00:11<00:19,  4.80it/s, epoch=71, grad_norm=184.241, loss=324.1679, lr=7.78e-04]
Epoch 71 TRAIN:  31%|███▏      | 42/134 [00:11<00:19,  4.80it/s, epoch=71, grad_norm=78.095, loss=292.7615, lr=7.78e-04] 
Epoch 71 TRAIN:  31%|███▏      | 42/134 [00:11<00:19,  4.80it/s, epoch=71, grad_norm=89.514, loss=291.2307, lr=7.78e-04]
Epoch 71 TRAIN:  34%|███▎      | 45/134 [00:11<00:18,  4.83it/s, epoch=71, grad_norm=89.514, loss=291.2307, lr=7.78e-04]
Epoch 71 TRAIN:  34%|███▎      | 45/134 [00:12<00:18,  4.83it/s, epoch=71, grad_norm=76.553, loss=252.3538, lr=7.78e-04]
Epoch 71 TRAIN:  34%|███▎      | 45/134 [00:12<00:18,  4.83it/s, epoch=71, grad_norm=95.984, loss=246.6872, lr=7.78e-04]
Epoch 71 TRAIN:  34%|███▎      | 45/134 [00:12<00:18,  4.83it/s, epoch=71, grad_norm=162.286, loss=320.3172, lr=7.78e-04]
Epoch 71 TRAIN:  36%|███▌      | 48/134 [00:12<00:17,  4.85it/s, epoch=71, grad_norm=162.286, loss=320.3172, lr=7.78e-04]
Epoch 71 TRAIN:  36%|███▌      | 48/134 [00:12<00:17,  4.85it/s, epoch=71, grad_norm=108.531, loss=279.2584, lr=7.78e-04]
Epoch 71 TRAIN:  36%|███▌      | 48/134 [00:12<00:17,  4.85it/s, epoch=71, grad_norm=110.139, loss=308.3091, lr=7.78e-04]
Epoch 71 TRAIN:  36%|███▌      | 48/134 [00:13<00:17,  4.85it/s, epoch=71, grad_norm=103.808, loss=271.8291, lr=7.78e-04]
Epoch 71 TRAIN:  38%|███▊      | 51/134 [00:13<00:17,  4.77it/s, epoch=71, grad_norm=103.808, loss=271.8291, lr=7.78e-04]
Epoch 71 TRAIN:  38%|███▊      | 51/134 [00:13<00:17,  4.77it/s, epoch=71, grad_norm=108.567, loss=268.1675, lr=7.78e-04]
Epoch 71 TRAIN:  38%|███▊      | 51/134 [00:13<00:17,  4.77it/s, epoch=71, grad_norm=174.700, loss=319.5204, lr=7.79e-04]
Epoch 71 TRAIN:  38%|███▊      | 51/134 [00:13<00:17,  4.77it/s, epoch=71, grad_norm=110.085, loss=292.2692, lr=7.79e-04]
Epoch 71 TRAIN:  40%|████      | 54/134 [00:13<00:16,  4.74it/s, epoch=71, grad_norm=110.085, loss=292.2692, lr=7.79e-04]
Epoch 71 TRAIN:  40%|████      | 54/134 [00:13<00:16,  4.74it/s, epoch=71, grad_norm=91.382, loss=288.2921, lr=7.79e-04] 
Epoch 71 TRAIN:  40%|████      | 54/134 [00:14<00:16,  4.74it/s, epoch=71, grad_norm=136.802, loss=239.0525, lr=7.79e-04]
Epoch 71 TRAIN:  40%|████      | 54/134 [00:14<00:16,  4.74it/s, epoch=71, grad_norm=189.328, loss=269.0254, lr=7.79e-04]
Epoch 71 TRAIN:  43%|████▎     | 57/134 [00:14<00:15,  4.87it/s, epoch=71, grad_norm=189.328, loss=269.0254, lr=7.79e-04]
Epoch 71 TRAIN:  43%|████▎     | 57/134 [00:14<00:15,  4.87it/s, epoch=71, grad_norm=168.682, loss=370.6189, lr=7.79e-04]
Epoch 71 TRAIN:  43%|████▎     | 57/134 [00:14<00:15,  4.87it/s, epoch=71, grad_norm=125.420, loss=319.7137, lr=7.79e-04]
Epoch 71 TRAIN:  43%|████▎     | 57/134 [00:14<00:15,  4.87it/s, epoch=71, grad_norm=121.769, loss=237.1720, lr=7.79e-04]
Epoch 71 TRAIN:  45%|████▍     | 60/134 [00:14<00:15,  4.81it/s, epoch=71, grad_norm=121.769, loss=237.1720, lr=7.79e-04]
Epoch 71 TRAIN:  45%|████▍     | 60/134 [00:15<00:15,  4.81it/s, epoch=71, grad_norm=172.867, loss=244.8868, lr=7.79e-04]
Epoch 71 TRAIN:  45%|████▍     | 60/134 [00:15<00:15,  4.81it/s, epoch=71, grad_norm=144.753, loss=327.9551, lr=7.79e-04]
Epoch 71 TRAIN:  45%|████▍     | 60/134 [00:15<00:15,  4.81it/s, epoch=71, grad_norm=97.978, loss=275.8082, lr=7.79e-04] 
Epoch 71 TRAIN:  47%|████▋     | 63/134 [00:15<00:15,  4.69it/s, epoch=71, grad_norm=97.978, loss=275.8082, lr=7.79e-04]
Epoch 71 TRAIN:  47%|████▋     | 63/134 [00:15<00:15,  4.69it/s, epoch=71, grad_norm=94.899, loss=270.8390, lr=7.79e-04]
Epoch 71 TRAIN:  47%|████▋     | 63/134 [00:15<00:15,  4.69it/s, epoch=71, grad_norm=92.106, loss=286.5992, lr=7.80e-04]
Epoch 71 TRAIN:  47%|████▋     | 63/134 [00:16<00:15,  4.69it/s, epoch=71, grad_norm=129.798, loss=339.2570, lr=7.80e-04]
Epoch 71 TRAIN:  49%|████▉     | 66/134 [00:16<00:14,  4.80it/s, epoch=71, grad_norm=129.798, loss=339.2570, lr=7.80e-04]
Epoch 71 TRAIN:  49%|████▉     | 66/134 [00:16<00:14,  4.80it/s, epoch=71, grad_norm=104.329, loss=267.4751, lr=7.80e-04]
Epoch 71 TRAIN:  49%|████▉     | 66/134 [00:16<00:14,  4.80it/s, epoch=71, grad_norm=77.948, loss=258.9192, lr=7.80e-04] 
Epoch 71 TRAIN:  49%|████▉     | 66/134 [00:16<00:14,  4.80it/s, epoch=71, grad_norm=109.815, loss=275.3672, lr=7.80e-04]
Epoch 71 TRAIN:  51%|█████▏    | 69/134 [00:16<00:13,  4.84it/s, epoch=71, grad_norm=109.815, loss=275.3672, lr=7.80e-04]
Epoch 71 TRAIN:  51%|█████▏    | 69/134 [00:17<00:13,  4.84it/s, epoch=71, grad_norm=89.163, loss=290.4633, lr=7.80e-04] 
Epoch 71 TRAIN:  51%|█████▏    | 69/134 [00:17<00:13,  4.84it/s, epoch=71, grad_norm=157.700, loss=261.2386, lr=7.80e-04]
Epoch 71 TRAIN:  51%|█████▏    | 69/134 [00:17<00:13,  4.84it/s, epoch=71, grad_norm=119.552, loss=258.9587, lr=7.80e-04]
Epoch 71 TRAIN:  54%|█████▎    | 72/134 [00:17<00:13,  4.47it/s, epoch=71, grad_norm=119.552, loss=258.9587, lr=7.80e-04]
Epoch 71 TRAIN:  54%|█████▎    | 72/134 [00:17<00:13,  4.47it/s, epoch=71, grad_norm=178.146, loss=260.7971, lr=7.80e-04]
Epoch 71 TRAIN:  54%|█████▎    | 72/134 [00:18<00:13,  4.47it/s, epoch=71, grad_norm=126.154, loss=318.9068, lr=7.80e-04]
Epoch 71 TRAIN:  54%|█████▎    | 72/134 [00:18<00:13,  4.47it/s, epoch=71, grad_norm=139.288, loss=306.1257, lr=7.80e-04]
Epoch 71 TRAIN:  56%|█████▌    | 75/134 [00:18<00:13,  4.53it/s, epoch=71, grad_norm=139.288, loss=306.1257, lr=7.80e-04]
Epoch 71 TRAIN:  56%|█████▌    | 75/134 [00:18<00:13,  4.53it/s, epoch=71, grad_norm=143.122, loss=295.1442, lr=7.80e-04]
Epoch 71 TRAIN:  56%|█████▌    | 75/134 [00:18<00:13,  4.53it/s, epoch=71, grad_norm=88.694, loss=289.5025, lr=7.80e-04] 
Epoch 71 TRAIN:  56%|█████▌    | 75/134 [00:18<00:13,  4.53it/s, epoch=71, grad_norm=102.367, loss=313.4123, lr=7.81e-04]
Epoch 71 TRAIN:  58%|█████▊    | 78/134 [00:18<00:12,  4.33it/s, epoch=71, grad_norm=102.367, loss=313.4123, lr=7.81e-04]
Epoch 71 TRAIN:  58%|█████▊    | 78/134 [00:19<00:12,  4.33it/s, epoch=71, grad_norm=112.973, loss=267.1252, lr=7.81e-04]
Epoch 71 TRAIN:  58%|█████▊    | 78/134 [00:19<00:12,  4.33it/s, epoch=71, grad_norm=162.376, loss=261.3434, lr=7.81e-04]
Epoch 71 TRAIN:  58%|█████▊    | 78/134 [00:19<00:12,  4.33it/s, epoch=71, grad_norm=136.207, loss=288.1858, lr=7.81e-04]
Epoch 71 TRAIN:  60%|██████    | 81/134 [00:19<00:11,  4.56it/s, epoch=71, grad_norm=136.207, loss=288.1858, lr=7.81e-04]
Epoch 71 TRAIN:  60%|██████    | 81/134 [00:19<00:11,  4.56it/s, epoch=71, grad_norm=203.515, loss=251.9925, lr=7.81e-04]
Epoch 71 TRAIN:  60%|██████    | 81/134 [00:20<00:11,  4.56it/s, epoch=71, grad_norm=143.337, loss=235.2195, lr=7.81e-04]
Epoch 71 TRAIN:  60%|██████    | 81/134 [00:20<00:11,  4.56it/s, epoch=71, grad_norm=113.476, loss=228.3964, lr=7.81e-04]
Epoch 71 TRAIN:  63%|██████▎   | 84/134 [00:20<00:11,  4.52it/s, epoch=71, grad_norm=113.476, loss=228.3964, lr=7.81e-04]
Epoch 71 TRAIN:  63%|██████▎   | 84/134 [00:20<00:11,  4.52it/s, epoch=71, grad_norm=171.104, loss=191.9910, lr=7.81e-04]
Epoch 71 TRAIN:  63%|██████▎   | 84/134 [00:20<00:11,  4.52it/s, epoch=71, grad_norm=212.254, loss=291.7748, lr=7.81e-04]
Epoch 71 TRAIN:  63%|██████▎   | 84/134 [00:20<00:11,  4.52it/s, epoch=71, grad_norm=232.739, loss=316.9376, lr=7.81e-04]
Epoch 71 TRAIN:  65%|██████▍   | 87/134 [00:20<00:10,  4.52it/s, epoch=71, grad_norm=232.739, loss=316.9376, lr=7.81e-04]
Epoch 71 TRAIN:  65%|██████▍   | 87/134 [00:21<00:10,  4.52it/s, epoch=71, grad_norm=98.501, loss=232.3659, lr=7.81e-04] 
Epoch 71 TRAIN:  65%|██████▍   | 87/134 [00:21<00:10,  4.52it/s, epoch=71, grad_norm=107.785, loss=255.7325, lr=7.81e-04]
Epoch 71 TRAIN:  65%|██████▍   | 87/134 [00:21<00:10,  4.52it/s, epoch=71, grad_norm=188.058, loss=283.4950, lr=7.82e-04]
Epoch 71 TRAIN:  67%|██████▋   | 90/134 [00:21<00:09,  4.47it/s, epoch=71, grad_norm=188.058, loss=283.4950, lr=7.82e-04]
Epoch 71 TRAIN:  67%|██████▋   | 90/134 [00:21<00:09,  4.47it/s, epoch=71, grad_norm=106.900, loss=224.7983, lr=7.82e-04]
Epoch 71 TRAIN:  67%|██████▋   | 90/134 [00:22<00:09,  4.47it/s, epoch=71, grad_norm=148.450, loss=257.3051, lr=7.82e-04]
Epoch 71 TRAIN:  67%|██████▋   | 90/134 [00:22<00:09,  4.47it/s, epoch=71, grad_norm=169.921, loss=186.5296, lr=7.82e-04]
Epoch 71 TRAIN:  69%|██████▉   | 93/134 [00:22<00:08,  4.63it/s, epoch=71, grad_norm=169.921, loss=186.5296, lr=7.82e-04]
Epoch 71 TRAIN:  69%|██████▉   | 93/134 [00:22<00:08,  4.63it/s, epoch=71, grad_norm=180.408, loss=258.9411, lr=7.82e-04]
Epoch 71 TRAIN:  69%|██████▉   | 93/134 [00:22<00:08,  4.63it/s, epoch=71, grad_norm=135.222, loss=195.3274, lr=7.82e-04]
Epoch 71 TRAIN:  69%|██████▉   | 93/134 [00:22<00:08,  4.63it/s, epoch=71, grad_norm=110.659, loss=252.1472, lr=7.82e-04]
Epoch 71 TRAIN:  72%|███████▏  | 96/134 [00:22<00:08,  4.64it/s, epoch=71, grad_norm=110.659, loss=252.1472, lr=7.82e-04]
Epoch 71 TRAIN:  72%|███████▏  | 96/134 [00:23<00:08,  4.64it/s, epoch=71, grad_norm=197.468, loss=215.2927, lr=7.82e-04]
Epoch 71 TRAIN:  72%|███████▏  | 96/134 [00:23<00:08,  4.64it/s, epoch=71, grad_norm=240.967, loss=325.9296, lr=7.82e-04]
Epoch 71 TRAIN:  72%|███████▏  | 96/134 [00:23<00:08,  4.64it/s, epoch=71, grad_norm=197.484, loss=310.3947, lr=7.82e-04]
Epoch 71 TRAIN:  74%|███████▍  | 99/134 [00:23<00:07,  4.57it/s, epoch=71, grad_norm=197.484, loss=310.3947, lr=7.82e-04]
Epoch 71 TRAIN:  74%|███████▍  | 99/134 [00:23<00:07,  4.57it/s, epoch=71, grad_norm=176.303, loss=269.5544, lr=7.82e-04]2026-04-16 20:28:46,728 DEBUG TRAIN | steps/sec 4.196| Batch 71/100 loss 269.554413 loss_att 218.501083 loss_ctc 388.678802 th_accuracy 0.389458 lr 7.8232e-04 grad_norm 176.302689 rank 0

Epoch 71 TRAIN:  74%|███████▍  | 99/134 [00:23<00:07,  4.57it/s, epoch=71, grad_norm=178.943, loss=299.7356, lr=7.82e-04]
Epoch 71 TRAIN:  74%|███████▍  | 99/134 [00:24<00:07,  4.57it/s, epoch=71, grad_norm=145.883, loss=240.5949, lr=7.82e-04]
Epoch 71 TRAIN:  76%|███████▌  | 102/134 [00:24<00:06,  4.76it/s, epoch=71, grad_norm=145.883, loss=240.5949, lr=7.82e-04]
Epoch 71 TRAIN:  76%|███████▌  | 102/134 [00:24<00:06,  4.76it/s, epoch=71, grad_norm=161.680, loss=262.0894, lr=7.83e-04]
Epoch 71 TRAIN:  76%|███████▌  | 102/134 [00:24<00:06,  4.76it/s, epoch=71, grad_norm=185.203, loss=211.2633, lr=7.83e-04]
Epoch 71 TRAIN:  76%|███████▌  | 102/134 [00:24<00:06,  4.76it/s, epoch=71, grad_norm=214.703, loss=285.7606, lr=7.83e-04]
Epoch 71 TRAIN:  78%|███████▊  | 105/134 [00:24<00:06,  4.41it/s, epoch=71, grad_norm=214.703, loss=285.7606, lr=7.83e-04]
Epoch 71 TRAIN:  78%|███████▊  | 105/134 [00:25<00:06,  4.41it/s, epoch=71, grad_norm=172.685, loss=293.0114, lr=7.83e-04]
Epoch 71 TRAIN:  78%|███████▊  | 105/134 [00:25<00:06,  4.41it/s, epoch=71, grad_norm=141.299, loss=206.0607, lr=7.83e-04]
Epoch 71 TRAIN:  78%|███████▊  | 105/134 [00:25<00:06,  4.41it/s, epoch=71, grad_norm=286.753, loss=337.7101, lr=7.83e-04]
Epoch 71 TRAIN:  81%|████████  | 108/134 [00:25<00:05,  4.70it/s, epoch=71, grad_norm=286.753, loss=337.7101, lr=7.83e-04]
Epoch 71 TRAIN:  81%|████████  | 108/134 [00:25<00:05,  4.70it/s, epoch=71, grad_norm=148.014, loss=267.3529, lr=7.83e-04]
Epoch 71 TRAIN:  81%|████████  | 108/134 [00:26<00:05,  4.70it/s, epoch=71, grad_norm=116.764, loss=252.0978, lr=7.83e-04]
Epoch 71 TRAIN:  81%|████████  | 108/134 [00:26<00:05,  4.70it/s, epoch=71, grad_norm=170.172, loss=318.8144, lr=7.83e-04]
Epoch 71 TRAIN:  83%|████████▎ | 111/134 [00:26<00:05,  4.34it/s, epoch=71, grad_norm=170.172, loss=318.8144, lr=7.83e-04]
Epoch 71 TRAIN:  83%|████████▎ | 111/134 [00:26<00:05,  4.34it/s, epoch=71, grad_norm=115.460, loss=256.7343, lr=7.83e-04]
Epoch 71 TRAIN:  83%|████████▎ | 111/134 [00:26<00:05,  4.34it/s, epoch=71, grad_norm=204.294, loss=301.1625, lr=7.83e-04]
Epoch 71 TRAIN:  83%|████████▎ | 111/134 [00:26<00:05,  4.34it/s, epoch=71, grad_norm=154.684, loss=306.8595, lr=7.83e-04]
Epoch 71 TRAIN:  83%|████████▎ | 111/134 [00:27<00:05,  4.34it/s, epoch=71, grad_norm=143.829, loss=280.7570, lr=7.84e-04]
Epoch 71 TRAIN:  86%|████████▌ | 115/134 [00:27<00:04,  4.62it/s, epoch=71, grad_norm=143.829, loss=280.7570, lr=7.84e-04]
Epoch 71 TRAIN:  86%|████████▌ | 115/134 [00:27<00:04,  4.62it/s, epoch=71, grad_norm=322.284, loss=350.9752, lr=7.84e-04]
Epoch 71 TRAIN:  86%|████████▌ | 115/134 [00:27<00:04,  4.62it/s, epoch=71, grad_norm=178.083, loss=241.4508, lr=7.84e-04]
Epoch 71 TRAIN:  86%|████████▌ | 115/134 [00:27<00:04,  4.62it/s, epoch=71, grad_norm=129.646, loss=266.1227, lr=7.84e-04]
Epoch 71 TRAIN:  88%|████████▊ | 118/134 [00:27<00:03,  4.78it/s, epoch=71, grad_norm=129.646, loss=266.1227, lr=7.84e-04]
Epoch 71 TRAIN:  88%|████████▊ | 118/134 [00:27<00:03,  4.78it/s, epoch=71, grad_norm=154.981, loss=281.0292, lr=7.84e-04]
Epoch 71 TRAIN:  88%|████████▊ | 118/134 [00:28<00:03,  4.78it/s, epoch=71, grad_norm=208.800, loss=376.5350, lr=7.84e-04]
Epoch 71 TRAIN:  88%|████████▊ | 118/134 [00:28<00:03,  4.78it/s, epoch=71, grad_norm=219.127, loss=307.3203, lr=7.84e-04]
Epoch 71 TRAIN:  90%|█████████ | 121/134 [00:28<00:02,  4.71it/s, epoch=71, grad_norm=219.127, loss=307.3203, lr=7.84e-04]
Epoch 71 TRAIN:  90%|█████████ | 121/134 [00:28<00:02,  4.71it/s, epoch=71, grad_norm=205.265, loss=343.8895, lr=7.84e-04]
Epoch 71 TRAIN:  90%|█████████ | 121/134 [00:28<00:02,  4.71it/s, epoch=71, grad_norm=204.672, loss=217.1840, lr=7.84e-04]
Epoch 71 TRAIN:  90%|█████████ | 121/134 [00:28<00:02,  4.71it/s, epoch=71, grad_norm=164.513, loss=259.2918, lr=7.84e-04]
Epoch 71 TRAIN:  90%|█████████ | 121/134 [00:28<00:02,  4.71it/s, epoch=71, grad_norm=177.085, loss=327.9422, lr=7.84e-04]
Epoch 71 TRAIN:  93%|█████████▎| 125/134 [00:28<00:01,  5.08it/s, epoch=71, grad_norm=177.085, loss=327.9422, lr=7.84e-04]
Epoch 71 TRAIN:  93%|█████████▎| 125/134 [00:29<00:01,  5.08it/s, epoch=71, grad_norm=228.804, loss=326.5707, lr=7.84e-04]
Epoch 71 TRAIN:  93%|█████████▎| 125/134 [00:29<00:01,  5.08it/s, epoch=71, grad_norm=182.271, loss=309.6569, lr=7.84e-04]
Epoch 71 TRAIN:  93%|█████████▎| 125/134 [00:29<00:01,  5.08it/s, epoch=71, grad_norm=285.464, loss=260.7357, lr=7.85e-04]
Epoch 71 TRAIN:  96%|█████████▌| 128/134 [00:29<00:01,  4.91it/s, epoch=71, grad_norm=285.464, loss=260.7357, lr=7.85e-04]
Epoch 71 TRAIN:  96%|█████████▌| 128/134 [00:29<00:01,  4.91it/s, epoch=71, grad_norm=167.567, loss=336.6373, lr=7.85e-04]
Epoch 71 TRAIN:  96%|█████████▌| 128/134 [00:29<00:01,  4.91it/s, epoch=71, grad_norm=94.891, loss=197.6653, lr=7.85e-04] 
Epoch 71 TRAIN:  96%|█████████▌| 128/134 [00:30<00:01,  4.91it/s, epoch=71, grad_norm=225.080, loss=314.9697, lr=7.85e-04]
Epoch 71 TRAIN:  98%|█████████▊| 131/134 [00:30<00:00,  5.08it/s, epoch=71, grad_norm=225.080, loss=314.9697, lr=7.85e-04]
Epoch 71 TRAIN:  98%|█████████▊| 131/134 [00:30<00:00,  5.08it/s, epoch=71, grad_norm=230.942, loss=349.8600, lr=7.85e-04]
Epoch 71 TRAIN:  98%|█████████▊| 131/134 [00:30<00:00,  5.08it/s, epoch=71, grad_norm=177.994, loss=297.6675, lr=7.85e-04]
Epoch 71 TRAIN:  98%|█████████▊| 131/134 [00:30<00:00,  5.08it/s, epoch=71, grad_norm=147.488, loss=288.0747, lr=7.85e-04]
Epoch 71 TRAIN: 100%|██████████| 134/134 [00:30<00:00,  5.07it/s, epoch=71, grad_norm=147.488, loss=288.0747, lr=7.85e-04]
Epoch 71 TRAIN: 100%|██████████| 134/134 [00:30<00:00,  5.07it/s, epoch=71, grad_norm=152.273, loss=216.2567, lr=7.85e-04]
Epoch 71 TRAIN: 100%|██████████| 134/134 [00:31<00:00,  5.07it/s, epoch=71, grad_norm=134.668, loss=200.0239, lr=7.85e-04]
Epoch 71 TRAIN: 100%|██████████| 134/134 [00:31<00:00,  5.07it/s, epoch=71, grad_norm=243.259, loss=294.7722, lr=7.85e-04]
Epoch 71 TRAIN: 137it [00:31,  4.85it/s, epoch=71, grad_norm=243.259, loss=294.7722, lr=7.85e-04]                         
Epoch 71 TRAIN: 137it [00:31,  4.36it/s, epoch=71, grad_norm=243.259, loss=294.7722, lr=7.85e-04]

CV:   0%|          | 0/6 [00:00<?, ?it/s]/home/rookie/Documents/batuka/projects/wenet/wenet/wenet/utils/train_utils.py:689: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  amp_autocast(enabled=dtype is not None,

CV:  17%|█▋        | 1/6 [00:00<00:03,  1.32it/s]
                                                 
2026-04-16 20:28:55,257 INFO Epoch 71 Step 9816 CV info lr 7.8528e-04 cv_loss 469.8798106800426 rank 0 acc 0.3807839552561442
2026-04-16 20:28:55,259 INFO [Rank 0] Checkpoint: save to checkpoint exp/conformer_mongolian/epoch_71.pt
2026-04-16 20:28:55,371 INFO Epoch 72 Step 9816 TRAIN info lr 7.8528e-04 rank 0
2026-04-16 20:28:55,387 INFO using accumulate grad, new batch size is 1 times larger than before

Epoch 72 TRAIN:   0%|          | 0/137 [00:00<?, ?it/s]
Epoch 72 TRAIN:   0%|          | 0/137 [00:04<?, ?it/s, epoch=72, grad_norm=343.235, loss=274.1599, lr=7.85e-04]
Epoch 72 TRAIN:   1%|          | 1/137 [00:04<09:17,  4.10s/it, epoch=72, grad_norm=343.235, loss=274.1599, lr=7.85e-04]
Epoch 72 TRAIN:   1%|          | 1/137 [00:04<09:17,  4.10s/it, epoch=72, grad_norm=496.289, loss=380.1706, lr=7.85e-04]
Epoch 72 TRAIN:   1%|          | 1/137 [00:04<09:17,  4.10s/it, epoch=72, grad_norm=258.468, loss=261.0714, lr=7.86e-04]
Epoch 72 TRAIN:   1%|          | 1/137 [00:04<09:17,  4.10s/it, epoch=72, grad_norm=328.057, loss=307.3863, lr=7.86e-04]
Epoch 72 TRAIN:   1%|          | 1/137 [00:04<09:17,  4.10s/it, epoch=72, grad_norm=367.863, loss=347.9099, lr=7.86e-04]
Epoch 72 TRAIN:   1%|          | 1/137 [00:04<09:17,  4.10s/it, epoch=72, grad_norm=351.444, loss=385.1136, lr=7.86e-04]
Epoch 72 TRAIN:   1%|          | 1/137 [00:04<09:17,  4.10s/it, epoch=72, grad_norm=470.484, loss=392.9756, lr=7.86e-04]
Epoch 72 TRAIN:   1%|          | 1/137 [00:04<09:17,  4.10s/it, epoch=72, grad_norm=195.099, loss=302.3221, lr=7.86e-04]
Epoch 72 TRAIN:   1%|          | 1/137 [00:04<09:17,  4.10s/it, epoch=72, grad_norm=161.475, loss=298.1984, lr=7.86e-04]
Epoch 72 TRAIN:   7%|▋         | 9/137 [00:04<00:51,  2.49it/s, epoch=72, grad_norm=161.475, loss=298.1984, lr=7.86e-04]
Epoch 72 TRAIN:   7%|▋         | 9/137 [00:04<00:51,  2.49it/s, epoch=72, grad_norm=127.220, loss=307.4009, lr=7.86e-04]
Epoch 72 TRAIN:   7%|▋         | 9/137 [00:05<00:51,  2.49it/s, epoch=72, grad_norm=126.298, loss=309.2903, lr=7.86e-04]
Epoch 72 TRAIN:   7%|▋         | 9/137 [00:05<00:51,  2.49it/s, epoch=72, grad_norm=182.515, loss=284.9998, lr=7.86e-04]
Epoch 72 TRAIN:   7%|▋         | 9/137 [00:05<00:51,  2.49it/s, epoch=72, grad_norm=157.333, loss=329.9318, lr=7.86e-04]
Epoch 72 TRAIN:   9%|▉         | 13/137 [00:05<00:38,  3.26it/s, epoch=72, grad_norm=157.333, loss=329.9318, lr=7.86e-04]
Epoch 72 TRAIN:   9%|▉         | 13/137 [00:05<00:38,  3.26it/s, epoch=72, grad_norm=218.311, loss=285.0641, lr=7.86e-04]
Epoch 72 TRAIN:   9%|▉         | 13/137 [00:05<00:38,  3.26it/s, epoch=72, grad_norm=208.633, loss=289.8417, lr=7.86e-04]
Epoch 72 TRAIN:   9%|▉         | 13/137 [00:05<00:38,  3.26it/s, epoch=72, grad_norm=112.906, loss=298.6652, lr=7.87e-04]
Epoch 72 TRAIN:   9%|▉         | 13/137 [00:06<00:38,  3.26it/s, epoch=72, grad_norm=159.947, loss=270.1292, lr=7.87e-04]
Epoch 72 TRAIN:  12%|█▏        | 17/137 [00:06<00:30,  3.94it/s, epoch=72, grad_norm=159.947, loss=270.1292, lr=7.87e-04]
Epoch 72 TRAIN:  12%|█▏        | 17/137 [00:06<00:30,  3.94it/s, epoch=72, grad_norm=128.532, loss=260.4382, lr=7.87e-04]
Epoch 72 TRAIN:  12%|█▏        | 17/137 [00:06<00:30,  3.94it/s, epoch=72, grad_norm=97.164, loss=301.0751, lr=7.87e-04] 
Epoch 72 TRAIN:  12%|█▏        | 17/137 [00:06<00:30,  3.94it/s, epoch=72, grad_norm=123.997, loss=295.7509, lr=7.87e-04]
Epoch 72 TRAIN:  12%|█▏        | 17/137 [00:06<00:30,  3.94it/s, epoch=72, grad_norm=102.631, loss=292.3413, lr=7.87e-04]
Epoch 72 TRAIN:  15%|█▌        | 21/137 [00:06<00:27,  4.28it/s, epoch=72, grad_norm=102.631, loss=292.3413, lr=7.87e-04]
Epoch 72 TRAIN:  15%|█▌        | 21/137 [00:06<00:27,  4.28it/s, epoch=72, grad_norm=109.083, loss=294.0325, lr=7.87e-04]
Epoch 72 TRAIN:  15%|█▌        | 21/137 [00:07<00:27,  4.28it/s, epoch=72, grad_norm=71.418, loss=258.5271, lr=7.87e-04] 
Epoch 72 TRAIN:  15%|█▌        | 21/137 [00:07<00:27,  4.28it/s, epoch=72, grad_norm=154.265, loss=306.6201, lr=7.87e-04]
Epoch 72 TRAIN:  15%|█▌        | 21/137 [00:07<00:27,  4.28it/s, epoch=72, grad_norm=83.371, loss=274.9584, lr=7.87e-04] 
Epoch 72 TRAIN:  18%|█▊        | 25/137 [00:07<00:24,  4.55it/s, epoch=72, grad_norm=83.371, loss=274.9584, lr=7.87e-04]
Epoch 72 TRAIN:  18%|█▊        | 25/137 [00:07<00:24,  4.55it/s, epoch=72, grad_norm=140.674, loss=311.8030, lr=7.87e-04]
Epoch 72 TRAIN:  18%|█▊        | 25/137 [00:07<00:24,  4.55it/s, epoch=72, grad_norm=145.350, loss=281.8196, lr=7.87e-04]
Epoch 72 TRAIN:  18%|█▊        | 25/137 [00:08<00:24,  4.55it/s, epoch=72, grad_norm=79.139, loss=306.7133, lr=7.88e-04] 
Epoch 72 TRAIN:  20%|██        | 28/137 [00:08<00:23,  4.69it/s, epoch=72, grad_norm=79.139, loss=306.7133, lr=7.88e-04]
Epoch 72 TRAIN:  20%|██        | 28/137 [00:08<00:23,  4.69it/s, epoch=72, grad_norm=149.917, loss=246.7212, lr=7.88e-04]
Epoch 72 TRAIN:  20%|██        | 28/137 [00:08<00:23,  4.69it/s, epoch=72, grad_norm=97.857, loss=265.1165, lr=7.88e-04] 
Epoch 72 TRAIN:  20%|██        | 28/137 [00:08<00:23,  4.69it/s, epoch=72, grad_norm=97.651, loss=242.3102, lr=7.88e-04]
Epoch 72 TRAIN:  23%|██▎       | 31/137 [00:08<00:22,  4.68it/s, epoch=72, grad_norm=97.651, loss=242.3102, lr=7.88e-04]
Epoch 72 TRAIN:  23%|██▎       | 31/137 [00:08<00:22,  4.68it/s, epoch=72, grad_norm=160.687, loss=284.7745, lr=7.88e-04]
Epoch 72 TRAIN:  23%|██▎       | 31/137 [00:09<00:22,  4.68it/s, epoch=72, grad_norm=125.782, loss=306.3244, lr=7.88e-04]
Epoch 72 TRAIN:  23%|██▎       | 31/137 [00:09<00:22,  4.68it/s, epoch=72, grad_norm=121.040, loss=329.9932, lr=7.88e-04]
Epoch 72 TRAIN:  25%|██▍       | 34/137 [00:09<00:21,  4.72it/s, epoch=72, grad_norm=121.040, loss=329.9932, lr=7.88e-04]
Epoch 72 TRAIN:  25%|██▍       | 34/137 [00:09<00:21,  4.72it/s, epoch=72, grad_norm=97.920, loss=276.2682, lr=7.88e-04] 
Epoch 72 TRAIN:  25%|██▍       | 34/137 [00:09<00:21,  4.72it/s, epoch=72, grad_norm=126.593, loss=228.2826, lr=7.88e-04]
Epoch 72 TRAIN:  25%|██▍       | 34/137 [00:10<00:21,  4.72it/s, epoch=72, grad_norm=80.803, loss=271.0276, lr=7.88e-04] 
Epoch 72 TRAIN:  27%|██▋       | 37/137 [00:10<00:20,  4.79it/s, epoch=72, grad_norm=80.803, loss=271.0276, lr=7.88e-04]
Epoch 72 TRAIN:  27%|██▋       | 37/137 [00:10<00:20,  4.79it/s, epoch=72, grad_norm=115.327, loss=291.2000, lr=7.88e-04]
Epoch 72 TRAIN:  27%|██▋       | 37/137 [00:10<00:20,  4.79it/s, epoch=72, grad_norm=98.105, loss=261.1351, lr=7.88e-04] 
Epoch 72 TRAIN:  27%|██▋       | 37/137 [00:10<00:20,  4.79it/s, epoch=72, grad_norm=81.953, loss=272.4094, lr=7.88e-04]
Epoch 72 TRAIN:  29%|██▉       | 40/137 [00:10<00:19,  4.90it/s, epoch=72, grad_norm=81.953, loss=272.4094, lr=7.88e-04]
Epoch 72 TRAIN:  29%|██▉       | 40/137 [00:10<00:19,  4.90it/s, epoch=72, grad_norm=116.633, loss=226.9013, lr=7.89e-04]
Epoch 72 TRAIN:  29%|██▉       | 40/137 [00:11<00:19,  4.90it/s, epoch=72, grad_norm=88.414, loss=264.6887, lr=7.89e-04] 
Epoch 72 TRAIN:  29%|██▉       | 40/137 [00:11<00:19,  4.90it/s, epoch=72, grad_norm=111.376, loss=225.3340, lr=7.89e-04]
Epoch 72 TRAIN:  31%|███▏      | 43/137 [00:11<00:19,  4.71it/s, epoch=72, grad_norm=111.376, loss=225.3340, lr=7.89e-04]
Epoch 72 TRAIN:  31%|███▏      | 43/137 [00:11<00:19,  4.71it/s, epoch=72, grad_norm=137.803, loss=273.6476, lr=7.89e-04]
Epoch 72 TRAIN:  31%|███▏      | 43/137 [00:11<00:19,  4.71it/s, epoch=72, grad_norm=113.358, loss=253.8011, lr=7.89e-04]
Epoch 72 TRAIN:  31%|███▏      | 43/137 [00:11<00:19,  4.71it/s, epoch=72, grad_norm=263.339, loss=366.1131, lr=7.89e-04]
Epoch 72 TRAIN:  34%|███▎      | 46/137 [00:11<00:19,  4.68it/s, epoch=72, grad_norm=263.339, loss=366.1131, lr=7.89e-04]
Epoch 72 TRAIN:  34%|███▎      | 46/137 [00:12<00:19,  4.68it/s, epoch=72, grad_norm=146.508, loss=291.6653, lr=7.89e-04]
Epoch 72 TRAIN:  34%|███▎      | 46/137 [00:12<00:19,  4.68it/s, epoch=72, grad_norm=150.494, loss=261.5856, lr=7.89e-04]
Epoch 72 TRAIN:  34%|███▎      | 46/137 [00:12<00:19,  4.68it/s, epoch=72, grad_norm=169.726, loss=327.6839, lr=7.89e-04]
Epoch 72 TRAIN:  36%|███▌      | 49/137 [00:12<00:18,  4.75it/s, epoch=72, grad_norm=169.726, loss=327.6839, lr=7.89e-04]
Epoch 72 TRAIN:  36%|███▌      | 49/137 [00:12<00:18,  4.75it/s, epoch=72, grad_norm=138.445, loss=254.1494, lr=7.89e-04]
Epoch 72 TRAIN:  36%|███▌      | 49/137 [00:12<00:18,  4.75it/s, epoch=72, grad_norm=166.052, loss=289.5531, lr=7.89e-04]
Epoch 72 TRAIN:  36%|███▌      | 49/137 [00:13<00:18,  4.75it/s, epoch=72, grad_norm=94.560, loss=265.7004, lr=7.89e-04] 
Epoch 72 TRAIN:  38%|███▊      | 52/137 [00:13<00:18,  4.69it/s, epoch=72, grad_norm=94.560, loss=265.7004, lr=7.89e-04]
Epoch 72 TRAIN:  38%|███▊      | 52/137 [00:13<00:18,  4.69it/s, epoch=72, grad_norm=112.045, loss=314.1893, lr=7.90e-04]
Epoch 72 TRAIN:  38%|███▊      | 52/137 [00:13<00:18,  4.69it/s, epoch=72, grad_norm=167.557, loss=248.2274, lr=7.90e-04]
Epoch 72 TRAIN:  38%|███▊      | 52/137 [00:13<00:18,  4.69it/s, epoch=72, grad_norm=122.836, loss=261.2737, lr=7.90e-04]
Epoch 72 TRAIN:  40%|████      | 55/137 [00:13<00:17,  4.77it/s, epoch=72, grad_norm=122.836, loss=261.2737, lr=7.90e-04]
Epoch 72 TRAIN:  40%|████      | 55/137 [00:14<00:17,  4.77it/s, epoch=72, grad_norm=146.074, loss=270.9323, lr=7.90e-04]
Epoch 72 TRAIN:  40%|████      | 55/137 [00:14<00:17,  4.77it/s, epoch=72, grad_norm=165.242, loss=282.8055, lr=7.90e-04]
Epoch 72 TRAIN:  40%|████      | 55/137 [00:14<00:17,  4.77it/s, epoch=72, grad_norm=137.841, loss=275.5451, lr=7.90e-04]
Epoch 72 TRAIN:  42%|████▏     | 58/137 [00:14<00:16,  4.74it/s, epoch=72, grad_norm=137.841, loss=275.5451, lr=7.90e-04]
Epoch 72 TRAIN:  42%|████▏     | 58/137 [00:14<00:16,  4.74it/s, epoch=72, grad_norm=101.541, loss=300.4774, lr=7.90e-04]
Epoch 72 TRAIN:  42%|████▏     | 58/137 [00:14<00:16,  4.74it/s, epoch=72, grad_norm=147.023, loss=312.3619, lr=7.90e-04]
Epoch 72 TRAIN:  42%|████▏     | 58/137 [00:15<00:16,  4.74it/s, epoch=72, grad_norm=98.995, loss=276.5960, lr=7.90e-04] 
Epoch 72 TRAIN:  45%|████▍     | 61/137 [00:15<00:15,  4.77it/s, epoch=72, grad_norm=98.995, loss=276.5960, lr=7.90e-04]
Epoch 72 TRAIN:  45%|████▍     | 61/137 [00:15<00:15,  4.77it/s, epoch=72, grad_norm=126.979, loss=246.8691, lr=7.90e-04]
Epoch 72 TRAIN:  45%|████▍     | 61/137 [00:15<00:15,  4.77it/s, epoch=72, grad_norm=98.404, loss=250.2273, lr=7.90e-04] 
Epoch 72 TRAIN:  45%|████▍     | 61/137 [00:15<00:15,  4.77it/s, epoch=72, grad_norm=80.310, loss=255.7809, lr=7.90e-04]
Epoch 72 TRAIN:  47%|████▋     | 64/137 [00:15<00:15,  4.72it/s, epoch=72, grad_norm=80.310, loss=255.7809, lr=7.90e-04]
Epoch 72 TRAIN:  47%|████▋     | 64/137 [00:15<00:15,  4.72it/s, epoch=72, grad_norm=136.348, loss=210.0096, lr=7.90e-04]
Epoch 72 TRAIN:  47%|████▋     | 64/137 [00:16<00:15,  4.72it/s, epoch=72, grad_norm=116.549, loss=317.9056, lr=7.91e-04]
Epoch 72 TRAIN:  47%|████▋     | 64/137 [00:16<00:15,  4.72it/s, epoch=72, grad_norm=88.223, loss=264.8746, lr=7.91e-04] 
Epoch 72 TRAIN:  49%|████▉     | 67/137 [00:16<00:14,  4.72it/s, epoch=72, grad_norm=88.223, loss=264.8746, lr=7.91e-04]
Epoch 72 TRAIN:  49%|████▉     | 67/137 [00:16<00:14,  4.72it/s, epoch=72, grad_norm=96.631, loss=229.1873, lr=7.91e-04]
Epoch 72 TRAIN:  49%|████▉     | 67/137 [00:16<00:14,  4.72it/s, epoch=72, grad_norm=118.863, loss=271.9933, lr=7.91e-04]
Epoch 72 TRAIN:  49%|████▉     | 67/137 [00:17<00:14,  4.72it/s, epoch=72, grad_norm=204.234, loss=303.4633, lr=7.91e-04]
Epoch 72 TRAIN:  51%|█████     | 70/137 [00:17<00:14,  4.57it/s, epoch=72, grad_norm=204.234, loss=303.4633, lr=7.91e-04]
Epoch 72 TRAIN:  51%|█████     | 70/137 [00:17<00:14,  4.57it/s, epoch=72, grad_norm=209.190, loss=304.2932, lr=7.91e-04]
Epoch 72 TRAIN:  51%|█████     | 70/137 [00:17<00:14,  4.57it/s, epoch=72, grad_norm=187.091, loss=268.4439, lr=7.91e-04]
Epoch 72 TRAIN:  51%|█████     | 70/137 [00:17<00:14,  4.57it/s, epoch=72, grad_norm=159.721, loss=266.5346, lr=7.91e-04]
Epoch 72 TRAIN:  53%|█████▎    | 73/137 [00:17<00:14,  4.49it/s, epoch=72, grad_norm=159.721, loss=266.5346, lr=7.91e-04]
Epoch 72 TRAIN:  53%|█████▎    | 73/137 [00:17<00:14,  4.49it/s, epoch=72, grad_norm=129.713, loss=271.6192, lr=7.91e-04]
Epoch 72 TRAIN:  53%|█████▎    | 73/137 [00:18<00:14,  4.49it/s, epoch=72, grad_norm=119.362, loss=337.5532, lr=7.91e-04]
Epoch 72 TRAIN:  53%|█████▎    | 73/137 [00:18<00:14,  4.49it/s, epoch=72, grad_norm=986.700, loss=240.0917, lr=7.91e-04]
Epoch 72 TRAIN:  55%|█████▌    | 76/137 [00:18<00:13,  4.58it/s, epoch=72, grad_norm=986.700, loss=240.0917, lr=7.91e-04]
Epoch 72 TRAIN:  55%|█████▌    | 76/137 [00:18<00:13,  4.58it/s, epoch=72, grad_norm=86.566, loss=302.5096, lr=7.91e-04] 
Epoch 72 TRAIN:  55%|█████▌    | 76/137 [00:18<00:13,  4.58it/s, epoch=72, grad_norm=200.151, loss=287.2678, lr=7.92e-04]
Epoch 72 TRAIN:  55%|█████▌    | 76/137 [00:18<00:13,  4.58it/s, epoch=72, grad_norm=82.690, loss=243.2609, lr=7.92e-04] 
Epoch 72 TRAIN:  58%|█████▊    | 79/137 [00:18<00:12,  4.76it/s, epoch=72, grad_norm=82.690, loss=243.2609, lr=7.92e-04]
Epoch 72 TRAIN:  58%|█████▊    | 79/137 [00:19<00:12,  4.76it/s, epoch=72, grad_norm=96.134, loss=283.5001, lr=7.92e-04]
Epoch 72 TRAIN:  58%|█████▊    | 79/137 [00:19<00:12,  4.76it/s, epoch=72, grad_norm=308.633, loss=224.3986, lr=7.92e-04]
Epoch 72 TRAIN:  58%|█████▊    | 79/137 [00:19<00:12,  4.76it/s, epoch=72, grad_norm=208.530, loss=245.1635, lr=7.92e-04]
Epoch 72 TRAIN:  60%|█████▉    | 82/137 [00:19<00:11,  4.81it/s, epoch=72, grad_norm=208.530, loss=245.1635, lr=7.92e-04]
Epoch 72 TRAIN:  60%|█████▉    | 82/137 [00:19<00:11,  4.81it/s, epoch=72, grad_norm=234.292, loss=218.8689, lr=7.92e-04]
Epoch 72 TRAIN:  60%|█████▉    | 82/137 [00:20<00:11,  4.81it/s, epoch=72, grad_norm=136.171, loss=317.2097, lr=7.92e-04]
Epoch 72 TRAIN:  60%|█████▉    | 82/137 [00:20<00:11,  4.81it/s, epoch=72, grad_norm=139.077, loss=234.6265, lr=7.92e-04]
Epoch 72 TRAIN:  62%|██████▏   | 85/137 [00:20<00:11,  4.63it/s, epoch=72, grad_norm=139.077, loss=234.6265, lr=7.92e-04]
Epoch 72 TRAIN:  62%|██████▏   | 85/137 [00:20<00:11,  4.63it/s, epoch=72, grad_norm=237.775, loss=325.1282, lr=7.92e-04]
Epoch 72 TRAIN:  62%|██████▏   | 85/137 [00:20<00:11,  4.63it/s, epoch=72, grad_norm=291.358, loss=359.0143, lr=7.92e-04]
Epoch 72 TRAIN:  62%|██████▏   | 85/137 [00:20<00:11,  4.63it/s, epoch=72, grad_norm=88.119, loss=233.6558, lr=7.92e-04] 
Epoch 72 TRAIN:  64%|██████▍   | 88/137 [00:20<00:10,  4.53it/s, epoch=72, grad_norm=88.119, loss=233.6558, lr=7.92e-04]
Epoch 72 TRAIN:  64%|██████▍   | 88/137 [00:21<00:10,  4.53it/s, epoch=72, grad_norm=170.159, loss=268.2672, lr=7.92e-04]
Epoch 72 TRAIN:  64%|██████▍   | 88/137 [00:21<00:10,  4.53it/s, epoch=72, grad_norm=107.773, loss=289.4680, lr=7.92e-04]
Epoch 72 TRAIN:  64%|██████▍   | 88/137 [00:21<00:10,  4.53it/s, epoch=72, grad_norm=110.428, loss=236.4110, lr=7.93e-04]
Epoch 72 TRAIN:  66%|██████▋   | 91/137 [00:21<00:10,  4.48it/s, epoch=72, grad_norm=110.428, loss=236.4110, lr=7.93e-04]
Epoch 72 TRAIN:  66%|██████▋   | 91/137 [00:21<00:10,  4.48it/s, epoch=72, grad_norm=91.053, loss=221.7771, lr=7.93e-04] 
Epoch 72 TRAIN:  66%|██████▋   | 91/137 [00:22<00:10,  4.48it/s, epoch=72, grad_norm=173.405, loss=258.9099, lr=7.93e-04]
Epoch 72 TRAIN:  66%|██████▋   | 91/137 [00:22<00:10,  4.48it/s, epoch=72, grad_norm=194.698, loss=262.3908, lr=7.93e-04]
Epoch 72 TRAIN:  69%|██████▊   | 94/137 [00:22<00:09,  4.70it/s, epoch=72, grad_norm=194.698, loss=262.3908, lr=7.93e-04]
Epoch 72 TRAIN:  69%|██████▊   | 94/137 [00:22<00:09,  4.70it/s, epoch=72, grad_norm=392.139, loss=367.1931, lr=7.93e-04]
Epoch 72 TRAIN:  69%|██████▊   | 94/137 [00:22<00:09,  4.70it/s, epoch=72, grad_norm=111.598, loss=212.5765, lr=7.93e-04]
Epoch 72 TRAIN:  69%|██████▊   | 94/137 [00:22<00:09,  4.70it/s, epoch=72, grad_norm=198.075, loss=221.2364, lr=7.93e-04]
Epoch 72 TRAIN:  71%|███████   | 97/137 [00:22<00:08,  4.65it/s, epoch=72, grad_norm=198.075, loss=221.2364, lr=7.93e-04]
Epoch 72 TRAIN:  71%|███████   | 97/137 [00:23<00:08,  4.65it/s, epoch=72, grad_norm=201.497, loss=286.3973, lr=7.93e-04]
Epoch 72 TRAIN:  71%|███████   | 97/137 [00:23<00:08,  4.65it/s, epoch=72, grad_norm=124.088, loss=248.2383, lr=7.93e-04]
Epoch 72 TRAIN:  71%|███████   | 97/137 [00:23<00:08,  4.65it/s, epoch=72, grad_norm=110.022, loss=257.2733, lr=7.93e-04]2026-04-16 20:29:19,162 DEBUG TRAIN | steps/sec 4.224| Batch 72/100 loss 257.273254 loss_att 207.313141 loss_ctc 373.846832 th_accuracy 0.421252 lr 7.9328e-04 grad_norm 110.022346 rank 0

Epoch 72 TRAIN:  73%|███████▎  | 100/137 [00:23<00:08,  4.17it/s, epoch=72, grad_norm=110.022, loss=257.2733, lr=7.93e-04]
Epoch 72 TRAIN:  73%|███████▎  | 100/137 [00:23<00:08,  4.17it/s, epoch=72, grad_norm=214.609, loss=294.5981, lr=7.93e-04]
Epoch 72 TRAIN:  73%|███████▎  | 100/137 [00:24<00:08,  4.17it/s, epoch=72, grad_norm=348.946, loss=307.9598, lr=7.93e-04]
Epoch 72 TRAIN:  73%|███████▎  | 100/137 [00:24<00:08,  4.17it/s, epoch=72, grad_norm=101.744, loss=188.4165, lr=7.94e-04]
Epoch 72 TRAIN:  75%|███████▌  | 103/137 [00:24<00:07,  4.29it/s, epoch=72, grad_norm=101.744, loss=188.4165, lr=7.94e-04]
Epoch 72 TRAIN:  75%|███████▌  | 103/137 [00:24<00:07,  4.29it/s, epoch=72, grad_norm=131.087, loss=269.9532, lr=7.94e-04]
Epoch 72 TRAIN:  75%|███████▌  | 103/137 [00:24<00:07,  4.29it/s, epoch=72, grad_norm=252.804, loss=290.7789, lr=7.94e-04]
Epoch 72 TRAIN:  75%|███████▌  | 103/137 [00:24<00:07,  4.29it/s, epoch=72, grad_norm=289.618, loss=345.3452, lr=7.94e-04]
Epoch 72 TRAIN:  77%|███████▋  | 106/137 [00:24<00:06,  4.63it/s, epoch=72, grad_norm=289.618, loss=345.3452, lr=7.94e-04]
Epoch 72 TRAIN:  77%|███████▋  | 106/137 [00:25<00:06,  4.63it/s, epoch=72, grad_norm=291.789, loss=236.9923, lr=7.94e-04]
Epoch 72 TRAIN:  77%|███████▋  | 106/137 [00:25<00:06,  4.63it/s, epoch=72, grad_norm=158.806, loss=298.8084, lr=7.94e-04]
Epoch 72 TRAIN:  77%|███████▋  | 106/137 [00:25<00:06,  4.63it/s, epoch=72, grad_norm=159.509, loss=248.5193, lr=7.94e-04]
Epoch 72 TRAIN:  80%|███████▉  | 109/137 [00:25<00:06,  4.58it/s, epoch=72, grad_norm=159.509, loss=248.5193, lr=7.94e-04]
Epoch 72 TRAIN:  80%|███████▉  | 109/137 [00:25<00:06,  4.58it/s, epoch=72, grad_norm=148.736, loss=279.9353, lr=7.94e-04]
Epoch 72 TRAIN:  80%|███████▉  | 109/137 [00:26<00:06,  4.58it/s, epoch=72, grad_norm=246.413, loss=322.0997, lr=7.94e-04]
Epoch 72 TRAIN:  80%|███████▉  | 109/137 [00:26<00:06,  4.58it/s, epoch=72, grad_norm=142.310, loss=269.2988, lr=7.94e-04]
Epoch 72 TRAIN:  82%|████████▏ | 112/137 [00:26<00:05,  4.47it/s, epoch=72, grad_norm=142.310, loss=269.2988, lr=7.94e-04]
Epoch 72 TRAIN:  82%|████████▏ | 112/137 [00:26<00:05,  4.47it/s, epoch=72, grad_norm=184.902, loss=308.2053, lr=7.94e-04]
Epoch 72 TRAIN:  82%|████████▏ | 112/137 [00:26<00:05,  4.47it/s, epoch=72, grad_norm=233.159, loss=382.0310, lr=7.94e-04]
Epoch 72 TRAIN:  82%|████████▏ | 112/137 [00:26<00:05,  4.47it/s, epoch=72, grad_norm=188.404, loss=181.0229, lr=7.94e-04]
Epoch 72 TRAIN:  84%|████████▍ | 115/137 [00:26<00:04,  4.66it/s, epoch=72, grad_norm=188.404, loss=181.0229, lr=7.94e-04]
Epoch 72 TRAIN:  84%|████████▍ | 115/137 [00:27<00:04,  4.66it/s, epoch=72, grad_norm=193.447, loss=228.1423, lr=7.95e-04]
Epoch 72 TRAIN:  84%|████████▍ | 115/137 [00:27<00:04,  4.66it/s, epoch=72, grad_norm=156.617, loss=296.2183, lr=7.95e-04]
Epoch 72 TRAIN:  84%|████████▍ | 115/137 [00:27<00:04,  4.66it/s, epoch=72, grad_norm=190.830, loss=276.5292, lr=7.95e-04]
Epoch 72 TRAIN:  86%|████████▌ | 118/137 [00:27<00:04,  4.64it/s, epoch=72, grad_norm=190.830, loss=276.5292, lr=7.95e-04]
Epoch 72 TRAIN:  86%|████████▌ | 118/137 [00:27<00:04,  4.64it/s, epoch=72, grad_norm=127.137, loss=269.1508, lr=7.95e-04]
Epoch 72 TRAIN:  86%|████████▌ | 118/137 [00:27<00:04,  4.64it/s, epoch=72, grad_norm=138.456, loss=313.3533, lr=7.95e-04]
Epoch 72 TRAIN:  86%|████████▌ | 118/137 [00:28<00:04,  4.64it/s, epoch=72, grad_norm=176.046, loss=262.8588, lr=7.95e-04]
Epoch 72 TRAIN:  88%|████████▊ | 121/137 [00:28<00:03,  4.84it/s, epoch=72, grad_norm=176.046, loss=262.8588, lr=7.95e-04]
Epoch 72 TRAIN:  88%|████████▊ | 121/137 [00:28<00:03,  4.84it/s, epoch=72, grad_norm=119.321, loss=217.5549, lr=7.95e-04]
Epoch 72 TRAIN:  88%|████████▊ | 121/137 [00:28<00:03,  4.84it/s, epoch=72, grad_norm=218.911, loss=300.4338, lr=7.95e-04]
Epoch 72 TRAIN:  88%|████████▊ | 121/137 [00:28<00:03,  4.84it/s, epoch=72, grad_norm=101.668, loss=200.6911, lr=7.95e-04]
Epoch 72 TRAIN:  91%|█████████ | 124/137 [00:28<00:02,  4.63it/s, epoch=72, grad_norm=101.668, loss=200.6911, lr=7.95e-04]
Epoch 72 TRAIN:  91%|█████████ | 124/137 [00:29<00:02,  4.63it/s, epoch=72, grad_norm=339.669, loss=330.6944, lr=7.95e-04]
Epoch 72 TRAIN:  91%|█████████ | 124/137 [00:29<00:02,  4.63it/s, epoch=72, grad_norm=234.749, loss=347.8717, lr=7.95e-04]
Epoch 72 TRAIN:  91%|█████████ | 124/137 [00:29<00:02,  4.63it/s, epoch=72, grad_norm=323.490, loss=259.9328, lr=7.95e-04]
Epoch 72 TRAIN:  93%|█████████▎| 127/137 [00:29<00:02,  4.86it/s, epoch=72, grad_norm=323.490, loss=259.9328, lr=7.95e-04]
Epoch 72 TRAIN:  93%|█████████▎| 127/137 [00:29<00:02,  4.86it/s, epoch=72, grad_norm=161.532, loss=325.0869, lr=7.96e-04]
Epoch 72 TRAIN:  93%|█████████▎| 127/137 [00:29<00:02,  4.86it/s, epoch=72, grad_norm=343.130, loss=258.1892, lr=7.96e-04]
Epoch 72 TRAIN:  93%|█████████▎| 127/137 [00:29<00:02,  4.86it/s, epoch=72, grad_norm=252.937, loss=367.8508, lr=7.96e-04]
Epoch 72 TRAIN:  95%|█████████▍| 130/137 [00:29<00:01,  4.90it/s, epoch=72, grad_norm=252.937, loss=367.8508, lr=7.96e-04]
Epoch 72 TRAIN:  95%|█████████▍| 130/137 [00:30<00:01,  4.90it/s, epoch=72, grad_norm=323.726, loss=269.7152, lr=7.96e-04]
Epoch 72 TRAIN:  95%|█████████▍| 130/137 [00:30<00:01,  4.90it/s, epoch=72, grad_norm=321.495, loss=261.9201, lr=7.96e-04]
Epoch 72 TRAIN:  95%|█████████▍| 130/137 [00:30<00:01,  4.90it/s, epoch=72, grad_norm=209.991, loss=304.6069, lr=7.96e-04]
Epoch 72 TRAIN:  97%|█████████▋| 133/137 [00:30<00:00,  5.15it/s, epoch=72, grad_norm=209.991, loss=304.6069, lr=7.96e-04]
Epoch 72 TRAIN:  97%|█████████▋| 133/137 [00:30<00:00,  5.15it/s, epoch=72, grad_norm=280.336, loss=342.7129, lr=7.96e-04]
Epoch 72 TRAIN:  97%|█████████▋| 133/137 [00:30<00:00,  5.15it/s, epoch=72, grad_norm=355.164, loss=279.8567, lr=7.96e-04]
Epoch 72 TRAIN:  97%|█████████▋| 133/137 [00:31<00:00,  5.15it/s, epoch=72, grad_norm=143.073, loss=325.1850, lr=7.96e-04]
Epoch 72 TRAIN:  99%|█████████▉| 136/137 [00:31<00:00,  5.19it/s, epoch=72, grad_norm=143.073, loss=325.1850, lr=7.96e-04]
Epoch 72 TRAIN:  99%|█████████▉| 136/137 [00:31<00:00,  5.19it/s, epoch=72, grad_norm=248.097, loss=326.7567, lr=7.96e-04]
Epoch 72 TRAIN: 100%|██████████| 137/137 [00:31<00:00,  4.39it/s, epoch=72, grad_norm=248.097, loss=326.7567, lr=7.96e-04]

CV:   0%|          | 0/6 [00:00<?, ?it/s]/home/rookie/Documents/batuka/projects/wenet/wenet/wenet/utils/train_utils.py:689: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  amp_autocast(enabled=dtype is not None,

CV:  17%|█▋        | 1/6 [00:00<00:03,  1.31it/s]
                                                 
2026-04-16 20:29:27,448 INFO Epoch 72 Step 9953 CV info lr 7.9624e-04 cv_loss 476.15188876065343 rank 0 acc 0.38345405956109363
2026-04-16 20:29:27,449 INFO [Rank 0] Checkpoint: save to checkpoint exp/conformer_mongolian/epoch_72.pt
2026-04-16 20:29:27,561 INFO Epoch 73 Step 9953 TRAIN info lr 7.9624e-04 rank 0
2026-04-16 20:29:27,577 INFO using accumulate grad, new batch size is 1 times larger than before

Epoch 73 TRAIN:   0%|          | 0/137 [00:00<?, ?it/s]
Epoch 73 TRAIN:   0%|          | 0/137 [00:03<?, ?it/s, epoch=73, grad_norm=350.339, loss=290.4225, lr=7.96e-04]
Epoch 73 TRAIN:   1%|          | 1/137 [00:03<08:42,  3.84s/it, epoch=73, grad_norm=350.339, loss=290.4225, lr=7.96e-04]
Epoch 73 TRAIN:   1%|          | 1/137 [00:04<08:42,  3.84s/it, epoch=73, grad_norm=462.938, loss=344.5528, lr=7.96e-04]
Epoch 73 TRAIN:   1%|          | 1/137 [00:04<08:42,  3.84s/it, epoch=73, grad_norm=462.027, loss=376.5001, lr=7.96e-04]
Epoch 73 TRAIN:   1%|          | 1/137 [00:04<08:42,  3.84s/it, epoch=73, grad_norm=277.194, loss=268.4064, lr=7.97e-04]
Epoch 73 TRAIN:   1%|          | 1/137 [00:04<08:42,  3.84s/it, epoch=73, grad_norm=451.915, loss=368.2750, lr=7.97e-04]
Epoch 73 TRAIN:   1%|          | 1/137 [00:04<08:42,  3.84s/it, epoch=73, grad_norm=155.869, loss=199.4545, lr=7.97e-04]
Epoch 73 TRAIN:   1%|          | 1/137 [00:04<08:42,  3.84s/it, epoch=73, grad_norm=373.722, loss=303.4835, lr=7.97e-04]
Epoch 73 TRAIN:   5%|▌         | 7/137 [00:04<01:02,  2.07it/s, epoch=73, grad_norm=373.722, loss=303.4835, lr=7.97e-04]
Epoch 73 TRAIN:   5%|▌         | 7/137 [00:04<01:02,  2.07it/s, epoch=73, grad_norm=344.890, loss=361.8402, lr=7.97e-04]
Epoch 73 TRAIN:   5%|▌         | 7/137 [00:04<01:02,  2.07it/s, epoch=73, grad_norm=137.320, loss=253.3671, lr=7.97e-04]
Epoch 73 TRAIN:   5%|▌         | 7/137 [00:04<01:02,  2.07it/s, epoch=73, grad_norm=120.988, loss=273.3922, lr=7.97e-04]
Epoch 73 TRAIN:   5%|▌         | 7/137 [00:04<01:02,  2.07it/s, epoch=73, grad_norm=62.900, loss=266.9913, lr=7.97e-04] 
Epoch 73 TRAIN:   8%|▊         | 11/137 [00:04<00:41,  3.07it/s, epoch=73, grad_norm=62.900, loss=266.9913, lr=7.97e-04]
Epoch 73 TRAIN:   8%|▊         | 11/137 [00:05<00:41,  3.07it/s, epoch=73, grad_norm=77.714, loss=276.3129, lr=7.97e-04]
Epoch 73 TRAIN:   8%|▊         | 11/137 [00:05<00:41,  3.07it/s, epoch=73, grad_norm=221.868, loss=331.4229, lr=7.97e-04]
Epoch 73 TRAIN:   8%|▊         | 11/137 [00:05<00:41,  3.07it/s, epoch=73, grad_norm=137.153, loss=287.1913, lr=7.97e-04]
Epoch 73 TRAIN:   8%|▊         | 11/137 [00:05<00:41,  3.07it/s, epoch=73, grad_norm=101.206, loss=288.0414, lr=7.97e-04]
Epoch 73 TRAIN:  11%|█         | 15/137 [00:05<00:32,  3.79it/s, epoch=73, grad_norm=101.206, loss=288.0414, lr=7.97e-04]
Epoch 73 TRAIN:  11%|█         | 15/137 [00:05<00:32,  3.79it/s, epoch=73, grad_norm=139.778, loss=276.0347, lr=7.98e-04]
Epoch 73 TRAIN:  11%|█         | 15/137 [00:06<00:32,  3.79it/s, epoch=73, grad_norm=163.082, loss=287.2499, lr=7.98e-04]
Epoch 73 TRAIN:  11%|█         | 15/137 [00:06<00:32,  3.79it/s, epoch=73, grad_norm=100.342, loss=308.9244, lr=7.98e-04]
Epoch 73 TRAIN:  11%|█         | 15/137 [00:06<00:32,  3.79it/s, epoch=73, grad_norm=159.816, loss=251.0443, lr=7.98e-04]
Epoch 73 TRAIN:  14%|█▍        | 19/137 [00:06<00:28,  4.20it/s, epoch=73, grad_norm=159.816, loss=251.0443, lr=7.98e-04]
Epoch 73 TRAIN:  14%|█▍        | 19/137 [00:06<00:28,  4.20it/s, epoch=73, grad_norm=72.529, loss=304.2744, lr=7.98e-04] 
Epoch 73 TRAIN:  14%|█▍        | 19/137 [00:06<00:28,  4.20it/s, epoch=73, grad_norm=115.040, loss=310.9923, lr=7.98e-04]
Epoch 73 TRAIN:  14%|█▍        | 19/137 [00:06<00:28,  4.20it/s, epoch=73, grad_norm=82.827, loss=284.9286, lr=7.98e-04] 
Epoch 73 TRAIN:  16%|█▌        | 22/137 [00:06<00:26,  4.41it/s, epoch=73, grad_norm=82.827, loss=284.9286, lr=7.98e-04]
Epoch 73 TRAIN:  16%|█▌        | 22/137 [00:07<00:26,  4.41it/s, epoch=73, grad_norm=154.162, loss=316.6494, lr=7.98e-04]
Epoch 73 TRAIN:  16%|█▌        | 22/137 [00:07<00:26,  4.41it/s, epoch=73, grad_norm=163.130, loss=329.0937, lr=7.98e-04]
Epoch 73 TRAIN:  16%|█▌        | 22/137 [00:07<00:26,  4.41it/s, epoch=73, grad_norm=99.729, loss=298.1861, lr=7.98e-04] 
Epoch 73 TRAIN:  18%|█▊        | 25/137 [00:07<00:24,  4.51it/s, epoch=73, grad_norm=99.729, loss=298.1861, lr=7.98e-04]
Epoch 73 TRAIN:  18%|█▊        | 25/137 [00:07<00:24,  4.51it/s, epoch=73, grad_norm=70.462, loss=244.0848, lr=7.98e-04]
Epoch 73 TRAIN:  18%|█▊        | 25/137 [00:08<00:24,  4.51it/s, epoch=73, grad_norm=116.928, loss=310.1202, lr=7.98e-04]
Epoch 73 TRAIN:  18%|█▊        | 25/137 [00:08<00:24,  4.51it/s, epoch=73, grad_norm=90.165, loss=311.2896, lr=7.98e-04] 
Epoch 73 TRAIN:  20%|██        | 28/137 [00:08<00:23,  4.61it/s, epoch=73, grad_norm=90.165, loss=311.2896, lr=7.98e-04]
Epoch 73 TRAIN:  20%|██        | 28/137 [00:08<00:23,  4.61it/s, epoch=73, grad_norm=91.259, loss=314.8125, lr=7.99e-04]
Epoch 73 TRAIN:  20%|██        | 28/137 [00:08<00:23,  4.61it/s, epoch=73, grad_norm=174.355, loss=216.2178, lr=7.99e-04]
Epoch 73 TRAIN:  20%|██        | 28/137 [00:08<00:23,  4.61it/s, epoch=73, grad_norm=106.956, loss=332.7964, lr=7.99e-04]
Epoch 73 TRAIN:  23%|██▎       | 31/137 [00:08<00:22,  4.70it/s, epoch=73, grad_norm=106.956, loss=332.7964, lr=7.99e-04]
Epoch 73 TRAIN:  23%|██▎       | 31/137 [00:09<00:22,  4.70it/s, epoch=73, grad_norm=138.271, loss=258.5682, lr=7.99e-04]
Epoch 73 TRAIN:  23%|██▎       | 31/137 [00:09<00:22,  4.70it/s, epoch=73, grad_norm=156.293, loss=299.5929, lr=7.99e-04]
Epoch 73 TRAIN:  23%|██▎       | 31/137 [00:09<00:22,  4.70it/s, epoch=73, grad_norm=99.556, loss=258.2197, lr=7.99e-04] 
Epoch 73 TRAIN:  25%|██▍       | 34/137 [00:09<00:21,  4.79it/s, epoch=73, grad_norm=99.556, loss=258.2197, lr=7.99e-04]
Epoch 73 TRAIN:  25%|██▍       | 34/137 [00:09<00:21,  4.79it/s, epoch=73, grad_norm=82.125, loss=287.9740, lr=7.99e-04]
Epoch 73 TRAIN:  25%|██▍       | 34/137 [00:09<00:21,  4.79it/s, epoch=73, grad_norm=76.918, loss=293.9600, lr=7.99e-04]
Epoch 73 TRAIN:  25%|██▍       | 34/137 [00:10<00:21,  4.79it/s, epoch=73, grad_norm=121.527, loss=254.7414, lr=7.99e-04]
Epoch 73 TRAIN:  27%|██▋       | 37/137 [00:10<00:20,  4.82it/s, epoch=73, grad_norm=121.527, loss=254.7414, lr=7.99e-04]
Epoch 73 TRAIN:  27%|██▋       | 37/137 [00:10<00:20,  4.82it/s, epoch=73, grad_norm=114.224, loss=242.1454, lr=7.99e-04]
Epoch 73 TRAIN:  27%|██▋       | 37/137 [00:10<00:20,  4.82it/s, epoch=73, grad_norm=84.045, loss=263.1956, lr=7.99e-04] 
Epoch 73 TRAIN:  27%|██▋       | 37/137 [00:10<00:20,  4.82it/s, epoch=73, grad_norm=105.504, loss=277.4398, lr=7.99e-04]
Epoch 73 TRAIN:  29%|██▉       | 40/137 [00:10<00:20,  4.75it/s, epoch=73, grad_norm=105.504, loss=277.4398, lr=7.99e-04]
Epoch 73 TRAIN:  29%|██▉       | 40/137 [00:10<00:20,  4.75it/s, epoch=73, grad_norm=137.001, loss=309.3732, lr=8.00e-04]
Epoch 73 TRAIN:  29%|██▉       | 40/137 [00:11<00:20,  4.75it/s, epoch=73, grad_norm=109.587, loss=283.6672, lr=8.00e-04]
Epoch 73 TRAIN:  29%|██▉       | 40/137 [00:11<00:20,  4.75it/s, epoch=73, grad_norm=129.061, loss=241.7396, lr=8.00e-04]
Epoch 73 TRAIN:  31%|███▏      | 43/137 [00:11<00:19,  4.78it/s, epoch=73, grad_norm=129.061, loss=241.7396, lr=8.00e-04]
Epoch 73 TRAIN:  31%|███▏      | 43/137 [00:11<00:19,  4.78it/s, epoch=73, grad_norm=79.057, loss=245.9373, lr=8.00e-04] 
Epoch 73 TRAIN:  31%|███▏      | 43/137 [00:11<00:19,  4.78it/s, epoch=73, grad_norm=164.662, loss=310.9731, lr=8.00e-04]
Epoch 73 TRAIN:  31%|███▏      | 43/137 [00:11<00:19,  4.78it/s, epoch=73, grad_norm=103.537, loss=283.0891, lr=8.00e-04]
Epoch 73 TRAIN:  34%|███▎      | 46/137 [00:11<00:19,  4.77it/s, epoch=73, grad_norm=103.537, loss=283.0891, lr=8.00e-04]
Epoch 73 TRAIN:  34%|███▎      | 46/137 [00:12<00:19,  4.77it/s, epoch=73, grad_norm=87.311, loss=239.7664, lr=8.00e-04] 
Epoch 73 TRAIN:  34%|███▎      | 46/137 [00:12<00:19,  4.77it/s, epoch=73, grad_norm=126.262, loss=293.1213, lr=8.00e-04]
Epoch 73 TRAIN:  34%|███▎      | 46/137 [00:12<00:19,  4.77it/s, epoch=73, grad_norm=99.661, loss=253.2583, lr=8.00e-04] 
Epoch 73 TRAIN:  36%|███▌      | 49/137 [00:12<00:18,  4.72it/s, epoch=73, grad_norm=99.661, loss=253.2583, lr=8.00e-04]
Epoch 73 TRAIN:  36%|███▌      | 49/137 [00:12<00:18,  4.72it/s, epoch=73, grad_norm=119.799, loss=291.6519, lr=8.00e-04]
Epoch 73 TRAIN:  36%|███▌      | 49/137 [00:13<00:18,  4.72it/s, epoch=73, grad_norm=108.973, loss=267.5687, lr=8.00e-04]
Epoch 73 TRAIN:  36%|███▌      | 49/137 [00:13<00:18,  4.72it/s, epoch=73, grad_norm=145.788, loss=315.6824, lr=8.00e-04]
Epoch 73 TRAIN:  38%|███▊      | 52/137 [00:13<00:17,  4.79it/s, epoch=73, grad_norm=145.788, loss=315.6824, lr=8.00e-04]
Epoch 73 TRAIN:  38%|███▊      | 52/137 [00:13<00:17,  4.79it/s, epoch=73, grad_norm=105.364, loss=279.3551, lr=8.00e-04]
Epoch 73 TRAIN:  38%|███▊      | 52/137 [00:13<00:17,  4.79it/s, epoch=73, grad_norm=155.294, loss=269.5327, lr=8.01e-04]
Epoch 73 TRAIN:  38%|███▊      | 52/137 [00:13<00:17,  4.79it/s, epoch=73, grad_norm=125.566, loss=297.1950, lr=8.01e-04]
Epoch 73 TRAIN:  40%|████      | 55/137 [00:13<00:17,  4.69it/s, epoch=73, grad_norm=125.566, loss=297.1950, lr=8.01e-04]
Epoch 73 TRAIN:  40%|████      | 55/137 [00:14<00:17,  4.69it/s, epoch=73, grad_norm=148.477, loss=319.1989, lr=8.01e-04]
Epoch 73 TRAIN:  40%|████      | 55/137 [00:14<00:17,  4.69it/s, epoch=73, grad_norm=109.379, loss=295.7165, lr=8.01e-04]
Epoch 73 TRAIN:  40%|████      | 55/137 [00:14<00:17,  4.69it/s, epoch=73, grad_norm=125.394, loss=275.2062, lr=8.01e-04]
Epoch 73 TRAIN:  42%|████▏     | 58/137 [00:14<00:17,  4.51it/s, epoch=73, grad_norm=125.394, loss=275.2062, lr=8.01e-04]
Epoch 73 TRAIN:  42%|████▏     | 58/137 [00:14<00:17,  4.51it/s, epoch=73, grad_norm=128.638, loss=298.1845, lr=8.01e-04]
Epoch 73 TRAIN:  42%|████▏     | 58/137 [00:15<00:17,  4.51it/s, epoch=73, grad_norm=214.667, loss=220.3892, lr=8.01e-04]
Epoch 73 TRAIN:  42%|████▏     | 58/137 [00:15<00:17,  4.51it/s, epoch=73, grad_norm=118.431, loss=268.9893, lr=8.01e-04]
Epoch 73 TRAIN:  45%|████▍     | 61/137 [00:15<00:16,  4.61it/s, epoch=73, grad_norm=118.431, loss=268.9893, lr=8.01e-04]
Epoch 73 TRAIN:  45%|████▍     | 61/137 [00:15<00:16,  4.61it/s, epoch=73, grad_norm=95.505, loss=307.1003, lr=8.01e-04] 
Epoch 73 TRAIN:  45%|████▍     | 61/137 [00:15<00:16,  4.61it/s, epoch=73, grad_norm=112.605, loss=299.1692, lr=8.01e-04]
Epoch 73 TRAIN:  45%|████▍     | 61/137 [00:15<00:16,  4.61it/s, epoch=73, grad_norm=98.374, loss=290.9367, lr=8.01e-04] 
Epoch 73 TRAIN:  47%|████▋     | 64/137 [00:15<00:15,  4.69it/s, epoch=73, grad_norm=98.374, loss=290.9367, lr=8.01e-04]
Epoch 73 TRAIN:  47%|████▋     | 64/137 [00:16<00:15,  4.69it/s, epoch=73, grad_norm=111.848, loss=237.3253, lr=8.01e-04]
Epoch 73 TRAIN:  47%|████▋     | 64/137 [00:16<00:15,  4.69it/s, epoch=73, grad_norm=126.915, loss=254.3997, lr=8.02e-04]
Epoch 73 TRAIN:  47%|████▋     | 64/137 [00:16<00:15,  4.69it/s, epoch=73, grad_norm=167.424, loss=309.3962, lr=8.02e-04]
Epoch 73 TRAIN:  49%|████▉     | 67/137 [00:16<00:14,  4.71it/s, epoch=73, grad_norm=167.424, loss=309.3962, lr=8.02e-04]
Epoch 73 TRAIN:  49%|████▉     | 67/137 [00:16<00:14,  4.71it/s, epoch=73, grad_norm=98.044, loss=243.3677, lr=8.02e-04] 
Epoch 73 TRAIN:  49%|████▉     | 67/137 [00:16<00:14,  4.71it/s, epoch=73, grad_norm=197.203, loss=329.4209, lr=8.02e-04]
Epoch 73 TRAIN:  49%|████▉     | 67/137 [00:17<00:14,  4.71it/s, epoch=73, grad_norm=126.911, loss=290.6918, lr=8.02e-04]
Epoch 73 TRAIN:  51%|█████     | 70/137 [00:17<00:14,  4.56it/s, epoch=73, grad_norm=126.911, loss=290.6918, lr=8.02e-04]
Epoch 73 TRAIN:  51%|█████     | 70/137 [00:17<00:14,  4.56it/s, epoch=73, grad_norm=96.734, loss=245.6413, lr=8.02e-04] 
Epoch 73 TRAIN:  51%|█████     | 70/137 [00:17<00:14,  4.56it/s, epoch=73, grad_norm=130.493, loss=278.2014, lr=8.02e-04]
Epoch 73 TRAIN:  51%|█████     | 70/137 [00:17<00:14,  4.56it/s, epoch=73, grad_norm=94.784, loss=314.9423, lr=8.02e-04] 
Epoch 73 TRAIN:  53%|█████▎    | 73/137 [00:17<00:14,  4.40it/s, epoch=73, grad_norm=94.784, loss=314.9423, lr=8.02e-04]
Epoch 73 TRAIN:  53%|█████▎    | 73/137 [00:18<00:14,  4.40it/s, epoch=73, grad_norm=107.054, loss=267.5070, lr=8.02e-04]
Epoch 73 TRAIN:  53%|█████▎    | 73/137 [00:18<00:14,  4.40it/s, epoch=73, grad_norm=139.390, loss=256.2396, lr=8.02e-04]
Epoch 73 TRAIN:  53%|█████▎    | 73/137 [00:18<00:14,  4.40it/s, epoch=73, grad_norm=96.717, loss=302.4059, lr=8.02e-04] 
Epoch 73 TRAIN:  55%|█████▌    | 76/137 [00:18<00:13,  4.50it/s, epoch=73, grad_norm=96.717, loss=302.4059, lr=8.02e-04]
Epoch 73 TRAIN:  55%|█████▌    | 76/137 [00:18<00:13,  4.50it/s, epoch=73, grad_norm=138.278, loss=257.3047, lr=8.02e-04]
Epoch 73 TRAIN:  55%|█████▌    | 76/137 [00:19<00:13,  4.50it/s, epoch=73, grad_norm=130.670, loss=290.2669, lr=8.02e-04]
Epoch 73 TRAIN:  55%|█████▌    | 76/137 [00:19<00:13,  4.50it/s, epoch=73, grad_norm=101.184, loss=262.3604, lr=8.03e-04]
Epoch 73 TRAIN:  58%|█████▊    | 79/137 [00:19<00:13,  4.33it/s, epoch=73, grad_norm=101.184, loss=262.3604, lr=8.03e-04]
Epoch 73 TRAIN:  58%|█████▊    | 79/137 [00:19<00:13,  4.33it/s, epoch=73, grad_norm=123.335, loss=275.3393, lr=8.03e-04]
Epoch 73 TRAIN:  58%|█████▊    | 79/137 [00:19<00:13,  4.33it/s, epoch=73, grad_norm=275.129, loss=173.9507, lr=8.03e-04]
Epoch 73 TRAIN:  58%|█████▊    | 79/137 [00:19<00:13,  4.33it/s, epoch=73, grad_norm=87.992, loss=272.8311, lr=8.03e-04] 
Epoch 73 TRAIN:  60%|█████▉    | 82/137 [00:19<00:12,  4.44it/s, epoch=73, grad_norm=87.992, loss=272.8311, lr=8.03e-04]
Epoch 73 TRAIN:  60%|█████▉    | 82/137 [00:20<00:12,  4.44it/s, epoch=73, grad_norm=132.525, loss=312.6619, lr=8.03e-04]
Epoch 73 TRAIN:  60%|█████▉    | 82/137 [00:20<00:12,  4.44it/s, epoch=73, grad_norm=99.880, loss=210.8971, lr=8.03e-04] 
Epoch 73 TRAIN:  60%|█████▉    | 82/137 [00:20<00:12,  4.44it/s, epoch=73, grad_norm=256.133, loss=297.2398, lr=8.03e-04]
Epoch 73 TRAIN:  62%|██████▏   | 85/137 [00:20<00:11,  4.57it/s, epoch=73, grad_norm=256.133, loss=297.2398, lr=8.03e-04]
Epoch 73 TRAIN:  62%|██████▏   | 85/137 [00:20<00:11,  4.57it/s, epoch=73, grad_norm=86.187, loss=234.6189, lr=8.03e-04] 
Epoch 73 TRAIN:  62%|██████▏   | 85/137 [00:20<00:11,  4.57it/s, epoch=73, grad_norm=187.527, loss=328.0907, lr=8.03e-04]
Epoch 73 TRAIN:  62%|██████▏   | 85/137 [00:21<00:11,  4.57it/s, epoch=73, grad_norm=149.227, loss=245.7736, lr=8.03e-04]
Epoch 73 TRAIN:  64%|██████▍   | 88/137 [00:21<00:10,  4.63it/s, epoch=73, grad_norm=149.227, loss=245.7736, lr=8.03e-04]
Epoch 73 TRAIN:  64%|██████▍   | 88/137 [00:21<00:10,  4.63it/s, epoch=73, grad_norm=220.626, loss=181.4797, lr=8.03e-04]
Epoch 73 TRAIN:  64%|██████▍   | 88/137 [00:21<00:10,  4.63it/s, epoch=73, grad_norm=159.279, loss=289.5535, lr=8.03e-04]
Epoch 73 TRAIN:  64%|██████▍   | 88/137 [00:21<00:10,  4.63it/s, epoch=73, grad_norm=145.146, loss=280.1031, lr=8.04e-04]
Epoch 73 TRAIN:  66%|██████▋   | 91/137 [00:21<00:10,  4.59it/s, epoch=73, grad_norm=145.146, loss=280.1031, lr=8.04e-04]
Epoch 73 TRAIN:  66%|██████▋   | 91/137 [00:22<00:10,  4.59it/s, epoch=73, grad_norm=125.773, loss=258.1903, lr=8.04e-04]
Epoch 73 TRAIN:  66%|██████▋   | 91/137 [00:22<00:10,  4.59it/s, epoch=73, grad_norm=116.813, loss=292.9979, lr=8.04e-04]
Epoch 73 TRAIN:  66%|██████▋   | 91/137 [00:22<00:10,  4.59it/s, epoch=73, grad_norm=179.141, loss=189.0150, lr=8.04e-04]
Epoch 73 TRAIN:  69%|██████▊   | 94/137 [00:22<00:09,  4.45it/s, epoch=73, grad_norm=179.141, loss=189.0150, lr=8.04e-04]
Epoch 73 TRAIN:  69%|██████▊   | 94/137 [00:22<00:09,  4.45it/s, epoch=73, grad_norm=142.839, loss=264.2039, lr=8.04e-04]
Epoch 73 TRAIN:  69%|██████▊   | 94/137 [00:23<00:09,  4.45it/s, epoch=73, grad_norm=195.781, loss=288.8408, lr=8.04e-04]
Epoch 73 TRAIN:  69%|██████▊   | 94/137 [00:23<00:09,  4.45it/s, epoch=73, grad_norm=117.433, loss=246.8397, lr=8.04e-04]
Epoch 73 TRAIN:  71%|███████   | 97/137 [00:23<00:08,  4.45it/s, epoch=73, grad_norm=117.433, loss=246.8397, lr=8.04e-04]
Epoch 73 TRAIN:  71%|███████   | 97/137 [00:23<00:08,  4.45it/s, epoch=73, grad_norm=135.781, loss=292.2861, lr=8.04e-04]
Epoch 73 TRAIN:  71%|███████   | 97/137 [00:23<00:08,  4.45it/s, epoch=73, grad_norm=191.675, loss=170.7164, lr=8.04e-04]
Epoch 73 TRAIN:  71%|███████   | 97/137 [00:24<00:08,  4.45it/s, epoch=73, grad_norm=170.622, loss=295.1420, lr=8.04e-04]2026-04-16 20:29:51,581 DEBUG TRAIN | steps/sec 4.226| Batch 73/100 loss 295.142029 loss_att 239.723419 loss_ctc 424.452118 th_accuracy 0.415385 lr 8.0424e-04 grad_norm 170.621811 rank 0

Epoch 73 TRAIN:  73%|███████▎  | 100/137 [00:24<00:08,  4.28it/s, epoch=73, grad_norm=170.622, loss=295.1420, lr=8.04e-04]
Epoch 73 TRAIN:  73%|███████▎  | 100/137 [00:24<00:08,  4.28it/s, epoch=73, grad_norm=176.766, loss=185.5228, lr=8.04e-04]
Epoch 73 TRAIN:  73%|███████▎  | 100/137 [00:24<00:08,  4.28it/s, epoch=73, grad_norm=157.602, loss=248.1375, lr=8.04e-04]
Epoch 73 TRAIN:  73%|███████▎  | 100/137 [00:24<00:08,  4.28it/s, epoch=73, grad_norm=112.464, loss=259.3581, lr=8.04e-04]
Epoch 73 TRAIN:  73%|███████▎  | 100/137 [00:24<00:08,  4.28it/s, epoch=73, grad_norm=246.933, loss=298.1771, lr=8.05e-04]
Epoch 73 TRAIN:  76%|███████▌  | 104/137 [00:24<00:07,  4.57it/s, epoch=73, grad_norm=246.933, loss=298.1771, lr=8.05e-04]
Epoch 73 TRAIN:  76%|███████▌  | 104/137 [00:25<00:07,  4.57it/s, epoch=73, grad_norm=103.952, loss=253.5416, lr=8.05e-04]
Epoch 73 TRAIN:  76%|███████▌  | 104/137 [00:25<00:07,  4.57it/s, epoch=73, grad_norm=194.733, loss=232.9098, lr=8.05e-04]
Epoch 73 TRAIN:  76%|███████▌  | 104/137 [00:25<00:07,  4.57it/s, epoch=73, grad_norm=120.650, loss=293.8408, lr=8.05e-04]
Epoch 73 TRAIN:  78%|███████▊  | 107/137 [00:25<00:06,  4.50it/s, epoch=73, grad_norm=120.650, loss=293.8408, lr=8.05e-04]
Epoch 73 TRAIN:  78%|███████▊  | 107/137 [00:25<00:06,  4.50it/s, epoch=73, grad_norm=131.088, loss=252.6857, lr=8.05e-04]
Epoch 73 TRAIN:  78%|███████▊  | 107/137 [00:25<00:06,  4.50it/s, epoch=73, grad_norm=173.584, loss=245.9211, lr=8.05e-04]
Epoch 73 TRAIN:  78%|███████▊  | 107/137 [00:26<00:06,  4.50it/s, epoch=73, grad_norm=189.520, loss=288.8275, lr=8.05e-04]
Epoch 73 TRAIN:  80%|████████  | 110/137 [00:26<00:05,  4.57it/s, epoch=73, grad_norm=189.520, loss=288.8275, lr=8.05e-04]
Epoch 73 TRAIN:  80%|████████  | 110/137 [00:26<00:05,  4.57it/s, epoch=73, grad_norm=115.208, loss=254.9755, lr=8.05e-04]
Epoch 73 TRAIN:  80%|████████  | 110/137 [00:26<00:05,  4.57it/s, epoch=73, grad_norm=149.788, loss=253.0470, lr=8.05e-04]
Epoch 73 TRAIN:  80%|████████  | 110/137 [00:26<00:05,  4.57it/s, epoch=73, grad_norm=159.236, loss=203.2362, lr=8.05e-04]
Epoch 73 TRAIN:  82%|████████▏ | 113/137 [00:26<00:05,  4.47it/s, epoch=73, grad_norm=159.236, loss=203.2362, lr=8.05e-04]
Epoch 73 TRAIN:  82%|████████▏ | 113/137 [00:27<00:05,  4.47it/s, epoch=73, grad_norm=197.434, loss=324.5164, lr=8.05e-04]
Epoch 73 TRAIN:  82%|████████▏ | 113/137 [00:27<00:05,  4.47it/s, epoch=73, grad_norm=148.362, loss=197.5076, lr=8.05e-04]
Epoch 73 TRAIN:  82%|████████▏ | 113/137 [00:27<00:05,  4.47it/s, epoch=73, grad_norm=173.816, loss=241.5692, lr=8.06e-04]
Epoch 73 TRAIN:  85%|████████▍ | 116/137 [00:27<00:04,  4.55it/s, epoch=73, grad_norm=173.816, loss=241.5692, lr=8.06e-04]
Epoch 73 TRAIN:  85%|████████▍ | 116/137 [00:27<00:04,  4.55it/s, epoch=73, grad_norm=208.031, loss=262.7959, lr=8.06e-04]
Epoch 73 TRAIN:  85%|████████▍ | 116/137 [00:27<00:04,  4.55it/s, epoch=73, grad_norm=247.124, loss=321.8738, lr=8.06e-04]
Epoch 73 TRAIN:  85%|████████▍ | 116/137 [00:28<00:04,  4.55it/s, epoch=73, grad_norm=238.719, loss=353.6011, lr=8.06e-04]
Epoch 73 TRAIN:  87%|████████▋ | 119/137 [00:28<00:04,  4.45it/s, epoch=73, grad_norm=238.719, loss=353.6011, lr=8.06e-04]
Epoch 73 TRAIN:  87%|████████▋ | 119/137 [00:28<00:04,  4.45it/s, epoch=73, grad_norm=163.967, loss=269.2937, lr=8.06e-04]
Epoch 73 TRAIN:  87%|████████▋ | 119/137 [00:28<00:04,  4.45it/s, epoch=73, grad_norm=159.751, loss=276.1727, lr=8.06e-04]
Epoch 73 TRAIN:  87%|████████▋ | 119/137 [00:28<00:04,  4.45it/s, epoch=73, grad_norm=139.530, loss=261.0646, lr=8.06e-04]
Epoch 73 TRAIN:  89%|████████▉ | 122/137 [00:28<00:03,  4.44it/s, epoch=73, grad_norm=139.530, loss=261.0646, lr=8.06e-04]
Epoch 73 TRAIN:  89%|████████▉ | 122/137 [00:29<00:03,  4.44it/s, epoch=73, grad_norm=276.757, loss=309.7822, lr=8.06e-04]
Epoch 73 TRAIN:  89%|████████▉ | 122/137 [00:29<00:03,  4.44it/s, epoch=73, grad_norm=178.140, loss=319.9510, lr=8.06e-04]
Epoch 73 TRAIN:  89%|████████▉ | 122/137 [00:29<00:03,  4.44it/s, epoch=73, grad_norm=150.268, loss=310.5251, lr=8.06e-04]
Epoch 73 TRAIN:  91%|█████████ | 125/137 [00:29<00:02,  4.47it/s, epoch=73, grad_norm=150.268, loss=310.5251, lr=8.06e-04]
Epoch 73 TRAIN:  91%|█████████ | 125/137 [00:29<00:02,  4.47it/s, epoch=73, grad_norm=369.771, loss=268.0247, lr=8.06e-04]
Epoch 73 TRAIN:  91%|█████████ | 125/137 [00:29<00:02,  4.47it/s, epoch=73, grad_norm=193.637, loss=276.8304, lr=8.06e-04]
Epoch 73 TRAIN:  91%|█████████ | 125/137 [00:30<00:02,  4.47it/s, epoch=73, grad_norm=221.125, loss=272.0687, lr=8.06e-04]
Epoch 73 TRAIN:  93%|█████████▎| 128/137 [00:30<00:01,  4.80it/s, epoch=73, grad_norm=221.125, loss=272.0687, lr=8.06e-04]
Epoch 73 TRAIN:  93%|█████████▎| 128/137 [00:30<00:01,  4.80it/s, epoch=73, grad_norm=148.493, loss=309.3405, lr=8.07e-04]
Epoch 73 TRAIN:  93%|█████████▎| 128/137 [00:30<00:01,  4.80it/s, epoch=73, grad_norm=284.767, loss=331.9339, lr=8.07e-04]
Epoch 73 TRAIN:  93%|█████████▎| 128/137 [00:30<00:01,  4.80it/s, epoch=73, grad_norm=199.744, loss=316.9398, lr=8.07e-04]
Epoch 73 TRAIN:  93%|█████████▎| 128/137 [00:30<00:01,  4.80it/s, epoch=73, grad_norm=201.623, loss=300.6926, lr=8.07e-04]
Epoch 73 TRAIN:  96%|█████████▋| 132/137 [00:30<00:00,  5.16it/s, epoch=73, grad_norm=201.623, loss=300.6926, lr=8.07e-04]
Epoch 73 TRAIN:  96%|█████████▋| 132/137 [00:30<00:00,  5.16it/s, epoch=73, grad_norm=315.705, loss=347.0433, lr=8.07e-04]
Epoch 73 TRAIN:  96%|█████████▋| 132/137 [00:31<00:00,  5.16it/s, epoch=73, grad_norm=149.968, loss=314.6868, lr=8.07e-04]
Epoch 73 TRAIN:  96%|█████████▋| 132/137 [00:31<00:00,  5.16it/s, epoch=73, grad_norm=223.884, loss=267.9635, lr=8.07e-04]
Epoch 73 TRAIN:  99%|█████████▊| 135/137 [00:31<00:00,  5.02it/s, epoch=73, grad_norm=223.884, loss=267.9635, lr=8.07e-04]
Epoch 73 TRAIN:  99%|█████████▊| 135/137 [00:31<00:00,  5.02it/s, epoch=73, grad_norm=184.494, loss=320.6873, lr=8.07e-04]
Epoch 73 TRAIN:  99%|█████████▉| 136/137 [00:31<00:00,  4.32it/s, epoch=73, grad_norm=184.494, loss=320.6873, lr=8.07e-04]

CV:   0%|          | 0/6 [00:00<?, ?it/s]/home/rookie/Documents/batuka/projects/wenet/wenet/wenet/utils/train_utils.py:689: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  amp_autocast(enabled=dtype is not None,

CV:  17%|█▋        | 1/6 [00:00<00:04,  1.05it/s]
                                                 
2026-04-16 20:30:00,096 INFO Epoch 73 Step 10089 CV info lr 8.0712e-04 cv_loss 458.1145130504261 rank 0 acc 0.38873768349488574
2026-04-16 20:30:00,098 INFO [Rank 0] Checkpoint: save to checkpoint exp/conformer_mongolian/epoch_73.pt
2026-04-16 20:30:00,209 INFO Epoch 74 Step 10089 TRAIN info lr 8.0712e-04 rank 0
2026-04-16 20:30:00,221 INFO using accumulate grad, new batch size is 1 times larger than before

Epoch 74 TRAIN:   0%|          | 0/136 [00:00<?, ?it/s]
Epoch 74 TRAIN:   0%|          | 0/136 [00:03<?, ?it/s, epoch=74, grad_norm=296.656, loss=283.0892, lr=8.07e-04]
Epoch 74 TRAIN:   1%|          | 1/136 [00:03<08:37,  3.84s/it, epoch=74, grad_norm=296.656, loss=283.0892, lr=8.07e-04]
Epoch 74 TRAIN:   1%|          | 1/136 [00:04<08:37,  3.84s/it, epoch=74, grad_norm=405.642, loss=389.7930, lr=8.07e-04]
Epoch 74 TRAIN:   1%|          | 1/136 [00:04<08:37,  3.84s/it, epoch=74, grad_norm=348.341, loss=325.6528, lr=8.07e-04]
Epoch 74 TRAIN:   1%|          | 1/136 [00:04<08:37,  3.84s/it, epoch=74, grad_norm=207.324, loss=249.5115, lr=8.07e-04]
Epoch 74 TRAIN:   1%|          | 1/136 [00:04<08:37,  3.84s/it, epoch=74, grad_norm=121.869, loss=207.5475, lr=8.08e-04]
Epoch 74 TRAIN:   1%|          | 1/136 [00:04<08:37,  3.84s/it, epoch=74, grad_norm=430.858, loss=442.5767, lr=8.08e-04]
Epoch 74 TRAIN:   4%|▍         | 6/136 [00:04<01:13,  1.77it/s, epoch=74, grad_norm=430.858, loss=442.5767, lr=8.08e-04]
Epoch 74 TRAIN:   4%|▍         | 6/136 [00:04<01:13,  1.77it/s, epoch=74, grad_norm=182.111, loss=268.6512, lr=8.08e-04]
Epoch 74 TRAIN:   4%|▍         | 6/136 [00:04<01:13,  1.77it/s, epoch=74, grad_norm=196.352, loss=220.4059, lr=8.08e-04]
Epoch 74 TRAIN:   4%|▍         | 6/136 [00:04<01:13,  1.77it/s, epoch=74, grad_norm=111.589, loss=272.0948, lr=8.08e-04]
Epoch 74 TRAIN:   4%|▍         | 6/136 [00:04<01:13,  1.77it/s, epoch=74, grad_norm=80.502, loss=275.2863, lr=8.08e-04] 
Epoch 74 TRAIN:   4%|▍         | 6/136 [00:05<01:13,  1.77it/s, epoch=74, grad_norm=140.867, loss=293.2445, lr=8.08e-04]
Epoch 74 TRAIN:   8%|▊         | 11/136 [00:05<00:40,  3.12it/s, epoch=74, grad_norm=140.867, loss=293.2445, lr=8.08e-04]
Epoch 74 TRAIN:   8%|▊         | 11/136 [00:05<00:40,  3.12it/s, epoch=74, grad_norm=88.430, loss=271.2796, lr=8.08e-04] 
Epoch 74 TRAIN:   8%|▊         | 11/136 [00:05<00:40,  3.12it/s, epoch=74, grad_norm=93.983, loss=319.9322, lr=8.08e-04]
Epoch 74 TRAIN:   8%|▊         | 11/136 [00:05<00:40,  3.12it/s, epoch=74, grad_norm=121.219, loss=255.5817, lr=8.08e-04]
Epoch 74 TRAIN:   8%|▊         | 11/136 [00:05<00:40,  3.12it/s, epoch=74, grad_norm=121.036, loss=276.6691, lr=8.08e-04]
Epoch 74 TRAIN:  11%|█         | 15/136 [00:05<00:31,  3.87it/s, epoch=74, grad_norm=121.036, loss=276.6691, lr=8.08e-04]
Epoch 74 TRAIN:  11%|█         | 15/136 [00:05<00:31,  3.87it/s, epoch=74, grad_norm=106.238, loss=328.2799, lr=8.08e-04]
Epoch 74 TRAIN:  11%|█         | 15/136 [00:05<00:31,  3.87it/s, epoch=74, grad_norm=148.487, loss=291.5193, lr=8.08e-04]
Epoch 74 TRAIN:  11%|█         | 15/136 [00:06<00:31,  3.87it/s, epoch=74, grad_norm=100.750, loss=334.7093, lr=8.09e-04]
Epoch 74 TRAIN:  11%|█         | 15/136 [00:06<00:31,  3.87it/s, epoch=74, grad_norm=92.246, loss=277.6756, lr=8.09e-04] 
Epoch 74 TRAIN:  14%|█▍        | 19/136 [00:06<00:27,  4.28it/s, epoch=74, grad_norm=92.246, loss=277.6756, lr=8.09e-04]
Epoch 74 TRAIN:  14%|█▍        | 19/136 [00:06<00:27,  4.28it/s, epoch=74, grad_norm=101.110, loss=275.4676, lr=8.09e-04]
Epoch 74 TRAIN:  14%|█▍        | 19/136 [00:06<00:27,  4.28it/s, epoch=74, grad_norm=105.200, loss=270.4949, lr=8.09e-04]
Epoch 74 TRAIN:  14%|█▍        | 19/136 [00:06<00:27,  4.28it/s, epoch=74, grad_norm=83.708, loss=285.9101, lr=8.09e-04] 
Epoch 74 TRAIN:  14%|█▍        | 19/136 [00:07<00:27,  4.28it/s, epoch=74, grad_norm=96.957, loss=282.6025, lr=8.09e-04]
Epoch 74 TRAIN:  17%|█▋        | 23/136 [00:07<00:24,  4.54it/s, epoch=74, grad_norm=96.957, loss=282.6025, lr=8.09e-04]
Epoch 74 TRAIN:  17%|█▋        | 23/136 [00:07<00:24,  4.54it/s, epoch=74, grad_norm=124.620, loss=243.1109, lr=8.09e-04]
Epoch 74 TRAIN:  17%|█▋        | 23/136 [00:07<00:24,  4.54it/s, epoch=74, grad_norm=100.195, loss=285.1311, lr=8.09e-04]
Epoch 74 TRAIN:  17%|█▋        | 23/136 [00:07<00:24,  4.54it/s, epoch=74, grad_norm=79.561, loss=263.5694, lr=8.09e-04] 
Epoch 74 TRAIN:  19%|█▉        | 26/136 [00:07<00:23,  4.78it/s, epoch=74, grad_norm=79.561, loss=263.5694, lr=8.09e-04]
Epoch 74 TRAIN:  19%|█▉        | 26/136 [00:07<00:23,  4.78it/s, epoch=74, grad_norm=180.076, loss=296.8795, lr=8.09e-04]
Epoch 74 TRAIN:  19%|█▉        | 26/136 [00:08<00:23,  4.78it/s, epoch=74, grad_norm=103.557, loss=291.1844, lr=8.09e-04]
Epoch 74 TRAIN:  19%|█▉        | 26/136 [00:08<00:23,  4.78it/s, epoch=74, grad_norm=113.831, loss=275.6782, lr=8.09e-04]
Epoch 74 TRAIN:  21%|██▏       | 29/136 [00:08<00:22,  4.74it/s, epoch=74, grad_norm=113.831, loss=275.6782, lr=8.09e-04]
Epoch 74 TRAIN:  21%|██▏       | 29/136 [00:08<00:22,  4.74it/s, epoch=74, grad_norm=78.211, loss=258.4004, lr=8.10e-04] 
Epoch 74 TRAIN:  21%|██▏       | 29/136 [00:08<00:22,  4.74it/s, epoch=74, grad_norm=85.969, loss=293.4193, lr=8.10e-04]
Epoch 74 TRAIN:  21%|██▏       | 29/136 [00:08<00:22,  4.74it/s, epoch=74, grad_norm=121.101, loss=287.4730, lr=8.10e-04]
Epoch 74 TRAIN:  24%|██▎       | 32/136 [00:08<00:21,  4.78it/s, epoch=74, grad_norm=121.101, loss=287.4730, lr=8.10e-04]
Epoch 74 TRAIN:  24%|██▎       | 32/136 [00:09<00:21,  4.78it/s, epoch=74, grad_norm=112.075, loss=300.1419, lr=8.10e-04]
Epoch 74 TRAIN:  24%|██▎       | 32/136 [00:09<00:21,  4.78it/s, epoch=74, grad_norm=123.854, loss=271.9722, lr=8.10e-04]
Epoch 74 TRAIN:  24%|██▎       | 32/136 [00:09<00:21,  4.78it/s, epoch=74, grad_norm=99.780, loss=299.2335, lr=8.10e-04] 
Epoch 74 TRAIN:  26%|██▌       | 35/136 [00:09<00:20,  4.85it/s, epoch=74, grad_norm=99.780, loss=299.2335, lr=8.10e-04]
Epoch 74 TRAIN:  26%|██▌       | 35/136 [00:09<00:20,  4.85it/s, epoch=74, grad_norm=115.403, loss=324.4606, lr=8.10e-04]
Epoch 74 TRAIN:  26%|██▌       | 35/136 [00:09<00:20,  4.85it/s, epoch=74, grad_norm=110.464, loss=260.5731, lr=8.10e-04]
Epoch 74 TRAIN:  26%|██▌       | 35/136 [00:10<00:20,  4.85it/s, epoch=74, grad_norm=134.427, loss=241.7302, lr=8.10e-04]
Epoch 74 TRAIN:  28%|██▊       | 38/136 [00:10<00:20,  4.88it/s, epoch=74, grad_norm=134.427, loss=241.7302, lr=8.10e-04]
Epoch 74 TRAIN:  28%|██▊       | 38/136 [00:10<00:20,  4.88it/s, epoch=74, grad_norm=155.205, loss=242.8170, lr=8.10e-04]
Epoch 74 TRAIN:  28%|██▊       | 38/136 [00:10<00:20,  4.88it/s, epoch=74, grad_norm=101.501, loss=289.9861, lr=8.10e-04]
Epoch 74 TRAIN:  28%|██▊       | 38/136 [00:10<00:20,  4.88it/s, epoch=74, grad_norm=106.639, loss=293.9784, lr=8.10e-04]
Epoch 74 TRAIN:  30%|███       | 41/136 [00:10<00:19,  4.76it/s, epoch=74, grad_norm=106.639, loss=293.9784, lr=8.10e-04]
Epoch 74 TRAIN:  30%|███       | 41/136 [00:11<00:19,  4.76it/s, epoch=74, grad_norm=73.752, loss=295.3424, lr=8.10e-04] 
Epoch 74 TRAIN:  30%|███       | 41/136 [00:11<00:19,  4.76it/s, epoch=74, grad_norm=105.475, loss=237.8252, lr=8.11e-04]
Epoch 74 TRAIN:  30%|███       | 41/136 [00:11<00:19,  4.76it/s, epoch=74, grad_norm=144.079, loss=305.3765, lr=8.11e-04]
Epoch 74 TRAIN:  32%|███▏      | 44/136 [00:11<00:19,  4.78it/s, epoch=74, grad_norm=144.079, loss=305.3765, lr=8.11e-04]
Epoch 74 TRAIN:  32%|███▏      | 44/136 [00:11<00:19,  4.78it/s, epoch=74, grad_norm=120.593, loss=290.3165, lr=8.11e-04]
Epoch 74 TRAIN:  32%|███▏      | 44/136 [00:11<00:19,  4.78it/s, epoch=74, grad_norm=185.212, loss=328.3381, lr=8.11e-04]
Epoch 74 TRAIN:  32%|███▏      | 44/136 [00:12<00:19,  4.78it/s, epoch=74, grad_norm=101.635, loss=265.5002, lr=8.11e-04]
Epoch 74 TRAIN:  35%|███▍      | 47/136 [00:12<00:18,  4.83it/s, epoch=74, grad_norm=101.635, loss=265.5002, lr=8.11e-04]
Epoch 74 TRAIN:  35%|███▍      | 47/136 [00:12<00:18,  4.83it/s, epoch=74, grad_norm=120.172, loss=288.0565, lr=8.11e-04]
Epoch 74 TRAIN:  35%|███▍      | 47/136 [00:12<00:18,  4.83it/s, epoch=74, grad_norm=101.161, loss=278.6809, lr=8.11e-04]
Epoch 74 TRAIN:  35%|███▍      | 47/136 [00:12<00:18,  4.83it/s, epoch=74, grad_norm=167.371, loss=314.2599, lr=8.11e-04]
Epoch 74 TRAIN:  37%|███▋      | 50/136 [00:12<00:18,  4.77it/s, epoch=74, grad_norm=167.371, loss=314.2599, lr=8.11e-04]
Epoch 74 TRAIN:  37%|███▋      | 50/136 [00:12<00:18,  4.77it/s, epoch=74, grad_norm=166.488, loss=267.0815, lr=8.11e-04]
Epoch 74 TRAIN:  37%|███▋      | 50/136 [00:13<00:18,  4.77it/s, epoch=74, grad_norm=182.106, loss=239.2449, lr=8.11e-04]
Epoch 74 TRAIN:  37%|███▋      | 50/136 [00:13<00:18,  4.77it/s, epoch=74, grad_norm=153.740, loss=261.3627, lr=8.11e-04]
Epoch 74 TRAIN:  39%|███▉      | 53/136 [00:13<00:17,  4.69it/s, epoch=74, grad_norm=153.740, loss=261.3627, lr=8.11e-04]
Epoch 74 TRAIN:  39%|███▉      | 53/136 [00:13<00:17,  4.69it/s, epoch=74, grad_norm=125.890, loss=271.8866, lr=8.11e-04]
Epoch 74 TRAIN:  39%|███▉      | 53/136 [00:13<00:17,  4.69it/s, epoch=74, grad_norm=99.460, loss=315.0846, lr=8.12e-04] 
Epoch 74 TRAIN:  39%|███▉      | 53/136 [00:14<00:17,  4.69it/s, epoch=74, grad_norm=112.305, loss=256.5522, lr=8.12e-04]
Epoch 74 TRAIN:  41%|████      | 56/136 [00:14<00:17,  4.61it/s, epoch=74, grad_norm=112.305, loss=256.5522, lr=8.12e-04]
Epoch 74 TRAIN:  41%|████      | 56/136 [00:14<00:17,  4.61it/s, epoch=74, grad_norm=149.602, loss=248.6620, lr=8.12e-04]
Epoch 74 TRAIN:  41%|████      | 56/136 [00:14<00:17,  4.61it/s, epoch=74, grad_norm=86.645, loss=248.3727, lr=8.12e-04] 
Epoch 74 TRAIN:  41%|████      | 56/136 [00:14<00:17,  4.61it/s, epoch=74, grad_norm=94.186, loss=274.7492, lr=8.12e-04]
Epoch 74 TRAIN:  43%|████▎     | 59/136 [00:14<00:17,  4.47it/s, epoch=74, grad_norm=94.186, loss=274.7492, lr=8.12e-04]
Epoch 74 TRAIN:  43%|████▎     | 59/136 [00:15<00:17,  4.47it/s, epoch=74, grad_norm=130.459, loss=293.7599, lr=8.12e-04]
Epoch 74 TRAIN:  43%|████▎     | 59/136 [00:15<00:17,  4.47it/s, epoch=74, grad_norm=189.441, loss=321.0662, lr=8.12e-04]
Epoch 74 TRAIN:  43%|████▎     | 59/136 [00:15<00:17,  4.47it/s, epoch=74, grad_norm=125.469, loss=291.0238, lr=8.12e-04]
Epoch 74 TRAIN:  46%|████▌     | 62/136 [00:15<00:16,  4.54it/s, epoch=74, grad_norm=125.469, loss=291.0238, lr=8.12e-04]
Epoch 74 TRAIN:  46%|████▌     | 62/136 [00:15<00:16,  4.54it/s, epoch=74, grad_norm=81.345, loss=260.7586, lr=8.12e-04] 
Epoch 74 TRAIN:  46%|████▌     | 62/136 [00:15<00:16,  4.54it/s, epoch=74, grad_norm=88.538, loss=263.8147, lr=8.12e-04]
Epoch 74 TRAIN:  46%|████▌     | 62/136 [00:16<00:16,  4.54it/s, epoch=74, grad_norm=104.228, loss=244.0687, lr=8.12e-04]
Epoch 74 TRAIN:  48%|████▊     | 65/136 [00:16<00:16,  4.31it/s, epoch=74, grad_norm=104.228, loss=244.0687, lr=8.12e-04]
Epoch 74 TRAIN:  48%|████▊     | 65/136 [00:16<00:16,  4.31it/s, epoch=74, grad_norm=97.988, loss=234.4970, lr=8.12e-04] 
Epoch 74 TRAIN:  48%|████▊     | 65/136 [00:16<00:16,  4.31it/s, epoch=74, grad_norm=85.587, loss=233.9654, lr=8.12e-04]
Epoch 74 TRAIN:  48%|████▊     | 65/136 [00:16<00:16,  4.31it/s, epoch=74, grad_norm=95.870, loss=265.6003, lr=8.13e-04]
Epoch 74 TRAIN:  50%|█████     | 68/136 [00:16<00:15,  4.42it/s, epoch=74, grad_norm=95.870, loss=265.6003, lr=8.13e-04]
Epoch 74 TRAIN:  50%|█████     | 68/136 [00:17<00:15,  4.42it/s, epoch=74, grad_norm=110.466, loss=275.1669, lr=8.13e-04]
Epoch 74 TRAIN:  50%|█████     | 68/136 [00:17<00:15,  4.42it/s, epoch=74, grad_norm=86.094, loss=254.0150, lr=8.13e-04] 
Epoch 74 TRAIN:  50%|█████     | 68/136 [00:17<00:15,  4.42it/s, epoch=74, grad_norm=113.679, loss=281.4299, lr=8.13e-04]
Epoch 74 TRAIN:  52%|█████▏    | 71/136 [00:17<00:14,  4.55it/s, epoch=74, grad_norm=113.679, loss=281.4299, lr=8.13e-04]
Epoch 74 TRAIN:  52%|█████▏    | 71/136 [00:17<00:14,  4.55it/s, epoch=74, grad_norm=172.805, loss=297.5551, lr=8.13e-04]
Epoch 74 TRAIN:  52%|█████▏    | 71/136 [00:17<00:14,  4.55it/s, epoch=74, grad_norm=156.297, loss=255.0531, lr=8.13e-04]
Epoch 74 TRAIN:  52%|█████▏    | 71/136 [00:18<00:14,  4.55it/s, epoch=74, grad_norm=116.958, loss=303.0024, lr=8.13e-04]
Epoch 74 TRAIN:  54%|█████▍    | 74/136 [00:18<00:13,  4.68it/s, epoch=74, grad_norm=116.958, loss=303.0024, lr=8.13e-04]
Epoch 74 TRAIN:  54%|█████▍    | 74/136 [00:18<00:13,  4.68it/s, epoch=74, grad_norm=102.126, loss=325.4595, lr=8.13e-04]
Epoch 74 TRAIN:  54%|█████▍    | 74/136 [00:18<00:13,  4.68it/s, epoch=74, grad_norm=190.658, loss=272.5072, lr=8.13e-04]
Epoch 74 TRAIN:  54%|█████▍    | 74/136 [00:18<00:13,  4.68it/s, epoch=74, grad_norm=197.665, loss=259.7636, lr=8.13e-04]
Epoch 74 TRAIN:  57%|█████▋    | 77/136 [00:18<00:12,  4.72it/s, epoch=74, grad_norm=197.665, loss=259.7636, lr=8.13e-04]
Epoch 74 TRAIN:  57%|█████▋    | 77/136 [00:18<00:12,  4.72it/s, epoch=74, grad_norm=202.259, loss=230.5076, lr=8.13e-04]
Epoch 74 TRAIN:  57%|█████▋    | 77/136 [00:19<00:12,  4.72it/s, epoch=74, grad_norm=193.253, loss=310.4095, lr=8.13e-04]
Epoch 74 TRAIN:  57%|█████▋    | 77/136 [00:19<00:12,  4.72it/s, epoch=74, grad_norm=148.531, loss=265.3323, lr=8.14e-04]
Epoch 74 TRAIN:  59%|█████▉    | 80/136 [00:19<00:12,  4.58it/s, epoch=74, grad_norm=148.531, loss=265.3323, lr=8.14e-04]
Epoch 74 TRAIN:  59%|█████▉    | 80/136 [00:19<00:12,  4.58it/s, epoch=74, grad_norm=140.969, loss=294.0561, lr=8.14e-04]
Epoch 74 TRAIN:  59%|█████▉    | 80/136 [00:19<00:12,  4.58it/s, epoch=74, grad_norm=155.495, loss=232.9563, lr=8.14e-04]
Epoch 74 TRAIN:  59%|█████▉    | 80/136 [00:20<00:12,  4.58it/s, epoch=74, grad_norm=142.203, loss=310.0504, lr=8.14e-04]
Epoch 74 TRAIN:  61%|██████    | 83/136 [00:20<00:11,  4.46it/s, epoch=74, grad_norm=142.203, loss=310.0504, lr=8.14e-04]
Epoch 74 TRAIN:  61%|██████    | 83/136 [00:20<00:11,  4.46it/s, epoch=74, grad_norm=111.221, loss=243.9185, lr=8.14e-04]
Epoch 74 TRAIN:  61%|██████    | 83/136 [00:20<00:11,  4.46it/s, epoch=74, grad_norm=202.511, loss=307.2639, lr=8.14e-04]
Epoch 74 TRAIN:  61%|██████    | 83/136 [00:20<00:11,  4.46it/s, epoch=74, grad_norm=135.980, loss=259.8522, lr=8.14e-04]
Epoch 74 TRAIN:  63%|██████▎   | 86/136 [00:20<00:10,  4.56it/s, epoch=74, grad_norm=135.980, loss=259.8522, lr=8.14e-04]
Epoch 74 TRAIN:  63%|██████▎   | 86/136 [00:20<00:10,  4.56it/s, epoch=74, grad_norm=100.344, loss=236.2132, lr=8.14e-04]
Epoch 74 TRAIN:  63%|██████▎   | 86/136 [00:21<00:10,  4.56it/s, epoch=74, grad_norm=194.093, loss=296.7996, lr=8.14e-04]
Epoch 74 TRAIN:  63%|██████▎   | 86/136 [00:21<00:10,  4.56it/s, epoch=74, grad_norm=132.881, loss=193.3107, lr=8.14e-04]
Epoch 74 TRAIN:  65%|██████▌   | 89/136 [00:21<00:10,  4.65it/s, epoch=74, grad_norm=132.881, loss=193.3107, lr=8.14e-04]
Epoch 74 TRAIN:  65%|██████▌   | 89/136 [00:21<00:10,  4.65it/s, epoch=74, grad_norm=152.718, loss=272.5157, lr=8.14e-04]
Epoch 74 TRAIN:  65%|██████▌   | 89/136 [00:21<00:10,  4.65it/s, epoch=74, grad_norm=99.798, loss=178.2255, lr=8.14e-04] 
Epoch 74 TRAIN:  65%|██████▌   | 89/136 [00:22<00:10,  4.65it/s, epoch=74, grad_norm=179.949, loss=289.7466, lr=8.14e-04]
Epoch 74 TRAIN:  68%|██████▊   | 92/136 [00:22<00:09,  4.45it/s, epoch=74, grad_norm=179.949, loss=289.7466, lr=8.14e-04]
Epoch 74 TRAIN:  68%|██████▊   | 92/136 [00:22<00:09,  4.45it/s, epoch=74, grad_norm=149.614, loss=240.0302, lr=8.15e-04]
Epoch 74 TRAIN:  68%|██████▊   | 92/136 [00:22<00:09,  4.45it/s, epoch=74, grad_norm=122.025, loss=279.0918, lr=8.15e-04]
Epoch 74 TRAIN:  68%|██████▊   | 92/136 [00:22<00:09,  4.45it/s, epoch=74, grad_norm=110.310, loss=230.8313, lr=8.15e-04]
Epoch 74 TRAIN:  70%|██████▉   | 95/136 [00:22<00:09,  4.46it/s, epoch=74, grad_norm=110.310, loss=230.8313, lr=8.15e-04]
Epoch 74 TRAIN:  70%|██████▉   | 95/136 [00:22<00:09,  4.46it/s, epoch=74, grad_norm=249.147, loss=291.6675, lr=8.15e-04]
Epoch 74 TRAIN:  70%|██████▉   | 95/136 [00:23<00:09,  4.46it/s, epoch=74, grad_norm=120.937, loss=218.0894, lr=8.15e-04]
Epoch 74 TRAIN:  70%|██████▉   | 95/136 [00:23<00:09,  4.46it/s, epoch=74, grad_norm=261.418, loss=319.7180, lr=8.15e-04]
Epoch 74 TRAIN:  72%|███████▏  | 98/136 [00:23<00:08,  4.45it/s, epoch=74, grad_norm=261.418, loss=319.7180, lr=8.15e-04]
Epoch 74 TRAIN:  72%|███████▏  | 98/136 [00:23<00:08,  4.45it/s, epoch=74, grad_norm=96.852, loss=234.0939, lr=8.15e-04] 
Epoch 74 TRAIN:  72%|███████▏  | 98/136 [00:23<00:08,  4.45it/s, epoch=74, grad_norm=179.128, loss=327.4713, lr=8.15e-04]2026-04-16 20:30:24,040 DEBUG TRAIN | steps/sec 4.190| Batch 74/100 loss 327.471252 loss_att 263.354553 loss_ctc 477.076904 th_accuracy 0.415323 lr 8.1512e-04 grad_norm 179.127945 rank 0

Epoch 74 TRAIN:  72%|███████▏  | 98/136 [00:24<00:08,  4.45it/s, epoch=74, grad_norm=119.218, loss=265.0407, lr=8.15e-04]
Epoch 74 TRAIN:  74%|███████▍  | 101/136 [00:24<00:08,  4.18it/s, epoch=74, grad_norm=119.218, loss=265.0407, lr=8.15e-04]
Epoch 74 TRAIN:  74%|███████▍  | 101/136 [00:24<00:08,  4.18it/s, epoch=74, grad_norm=153.555, loss=280.4245, lr=8.15e-04]
Epoch 74 TRAIN:  74%|███████▍  | 101/136 [00:24<00:08,  4.18it/s, epoch=74, grad_norm=159.863, loss=311.0728, lr=8.15e-04]
Epoch 74 TRAIN:  74%|███████▍  | 101/136 [00:24<00:08,  4.18it/s, epoch=74, grad_norm=169.294, loss=302.6136, lr=8.15e-04]
Epoch 74 TRAIN:  76%|███████▋  | 104/136 [00:24<00:07,  4.41it/s, epoch=74, grad_norm=169.294, loss=302.6136, lr=8.15e-04]
Epoch 74 TRAIN:  76%|███████▋  | 104/136 [00:25<00:07,  4.41it/s, epoch=74, grad_norm=201.810, loss=240.6973, lr=8.16e-04]
Epoch 74 TRAIN:  76%|███████▋  | 104/136 [00:25<00:07,  4.41it/s, epoch=74, grad_norm=126.166, loss=258.7561, lr=8.16e-04]
Epoch 74 TRAIN:  76%|███████▋  | 104/136 [00:25<00:07,  4.41it/s, epoch=74, grad_norm=186.335, loss=215.2054, lr=8.16e-04]
Epoch 74 TRAIN:  79%|███████▊  | 107/136 [00:25<00:06,  4.17it/s, epoch=74, grad_norm=186.335, loss=215.2054, lr=8.16e-04]
Epoch 74 TRAIN:  79%|███████▊  | 107/136 [00:25<00:06,  4.17it/s, epoch=74, grad_norm=134.270, loss=294.8307, lr=8.16e-04]
Epoch 74 TRAIN:  79%|███████▊  | 107/136 [00:25<00:06,  4.17it/s, epoch=74, grad_norm=196.229, loss=221.9468, lr=8.16e-04]
Epoch 74 TRAIN:  79%|███████▊  | 107/136 [00:26<00:06,  4.17it/s, epoch=74, grad_norm=163.106, loss=298.3355, lr=8.16e-04]
Epoch 74 TRAIN:  81%|████████  | 110/136 [00:26<00:05,  4.53it/s, epoch=74, grad_norm=163.106, loss=298.3355, lr=8.16e-04]
Epoch 74 TRAIN:  81%|████████  | 110/136 [00:26<00:05,  4.53it/s, epoch=74, grad_norm=141.623, loss=301.6656, lr=8.16e-04]
Epoch 74 TRAIN:  81%|████████  | 110/136 [00:26<00:05,  4.53it/s, epoch=74, grad_norm=150.354, loss=247.4480, lr=8.16e-04]
Epoch 74 TRAIN:  81%|████████  | 110/136 [00:26<00:05,  4.53it/s, epoch=74, grad_norm=128.701, loss=256.6610, lr=8.16e-04]
Epoch 74 TRAIN:  83%|████████▎ | 113/136 [00:26<00:05,  4.39it/s, epoch=74, grad_norm=128.701, loss=256.6610, lr=8.16e-04]
Epoch 74 TRAIN:  83%|████████▎ | 113/136 [00:27<00:05,  4.39it/s, epoch=74, grad_norm=149.631, loss=252.7109, lr=8.16e-04]
Epoch 74 TRAIN:  83%|████████▎ | 113/136 [00:27<00:05,  4.39it/s, epoch=74, grad_norm=140.736, loss=299.9849, lr=8.16e-04]
Epoch 74 TRAIN:  83%|████████▎ | 113/136 [00:27<00:05,  4.39it/s, epoch=74, grad_norm=115.511, loss=267.6484, lr=8.16e-04]
Epoch 74 TRAIN:  85%|████████▌ | 116/136 [00:27<00:04,  4.44it/s, epoch=74, grad_norm=115.511, loss=267.6484, lr=8.16e-04]
Epoch 74 TRAIN:  85%|████████▌ | 116/136 [00:27<00:04,  4.44it/s, epoch=74, grad_norm=242.723, loss=324.1944, lr=8.16e-04]
Epoch 74 TRAIN:  85%|████████▌ | 116/136 [00:27<00:04,  4.44it/s, epoch=74, grad_norm=209.540, loss=365.5578, lr=8.17e-04]
Epoch 74 TRAIN:  85%|████████▌ | 116/136 [00:28<00:04,  4.44it/s, epoch=74, grad_norm=131.762, loss=309.7369, lr=8.17e-04]
Epoch 74 TRAIN:  88%|████████▊ | 119/136 [00:28<00:03,  4.71it/s, epoch=74, grad_norm=131.762, loss=309.7369, lr=8.17e-04]
Epoch 74 TRAIN:  88%|████████▊ | 119/136 [00:28<00:03,  4.71it/s, epoch=74, grad_norm=166.082, loss=233.1430, lr=8.17e-04]
Epoch 74 TRAIN:  88%|████████▊ | 119/136 [00:28<00:03,  4.71it/s, epoch=74, grad_norm=139.605, loss=328.6180, lr=8.17e-04]
Epoch 74 TRAIN:  88%|████████▊ | 119/136 [00:28<00:03,  4.71it/s, epoch=74, grad_norm=138.117, loss=232.7750, lr=8.17e-04]
Epoch 74 TRAIN:  90%|████████▉ | 122/136 [00:28<00:02,  4.74it/s, epoch=74, grad_norm=138.117, loss=232.7750, lr=8.17e-04]
Epoch 74 TRAIN:  90%|████████▉ | 122/136 [00:28<00:02,  4.74it/s, epoch=74, grad_norm=172.115, loss=301.2971, lr=8.17e-04]
Epoch 74 TRAIN:  90%|████████▉ | 122/136 [00:29<00:02,  4.74it/s, epoch=74, grad_norm=220.703, loss=296.6152, lr=8.17e-04]
Epoch 74 TRAIN:  90%|████████▉ | 122/136 [00:29<00:02,  4.74it/s, epoch=74, grad_norm=270.013, loss=318.5156, lr=8.17e-04]
Epoch 74 TRAIN:  92%|█████████▏| 125/136 [00:29<00:02,  4.64it/s, epoch=74, grad_norm=270.013, loss=318.5156, lr=8.17e-04]
Epoch 74 TRAIN:  92%|█████████▏| 125/136 [00:29<00:02,  4.64it/s, epoch=74, grad_norm=444.321, loss=344.1702, lr=8.17e-04]
Epoch 74 TRAIN:  92%|█████████▏| 125/136 [00:29<00:02,  4.64it/s, epoch=74, grad_norm=274.250, loss=314.6684, lr=8.17e-04]
Epoch 74 TRAIN:  92%|█████████▏| 125/136 [00:29<00:02,  4.64it/s, epoch=74, grad_norm=113.532, loss=205.8836, lr=8.17e-04]
Epoch 74 TRAIN:  94%|█████████▍| 128/136 [00:29<00:01,  4.90it/s, epoch=74, grad_norm=113.532, loss=205.8836, lr=8.17e-04]
Epoch 74 TRAIN:  94%|█████████▍| 128/136 [00:30<00:01,  4.90it/s, epoch=74, grad_norm=323.701, loss=305.2880, lr=8.17e-04]
Epoch 74 TRAIN:  94%|█████████▍| 128/136 [00:30<00:01,  4.90it/s, epoch=74, grad_norm=444.173, loss=288.7594, lr=8.18e-04]
Epoch 74 TRAIN:  94%|█████████▍| 128/136 [00:30<00:01,  4.90it/s, epoch=74, grad_norm=131.859, loss=226.4385, lr=8.18e-04]
Epoch 74 TRAIN:  96%|█████████▋| 131/136 [00:30<00:00,  5.04it/s, epoch=74, grad_norm=131.859, loss=226.4385, lr=8.18e-04]
Epoch 74 TRAIN:  96%|█████████▋| 131/136 [00:30<00:00,  5.04it/s, epoch=74, grad_norm=184.363, loss=255.8655, lr=8.18e-04]
Epoch 74 TRAIN:  96%|█████████▋| 131/136 [00:30<00:00,  5.04it/s, epoch=74, grad_norm=226.200, loss=345.4942, lr=8.18e-04]
Epoch 74 TRAIN:  96%|█████████▋| 131/136 [00:31<00:00,  5.04it/s, epoch=74, grad_norm=205.808, loss=324.7585, lr=8.18e-04]
Epoch 74 TRAIN:  99%|█████████▊| 134/136 [00:31<00:00,  5.15it/s, epoch=74, grad_norm=205.808, loss=324.7585, lr=8.18e-04]
Epoch 74 TRAIN:  99%|█████████▊| 134/136 [00:31<00:00,  5.15it/s, epoch=74, grad_norm=185.361, loss=274.2148, lr=8.18e-04]
Epoch 74 TRAIN:  99%|█████████▊| 134/136 [00:31<00:00,  5.15it/s, epoch=74, grad_norm=168.896, loss=246.1819, lr=8.18e-04]
Epoch 74 TRAIN:  99%|█████████▊| 134/136 [00:31<00:00,  5.15it/s, epoch=74, grad_norm=196.423, loss=253.4264, lr=8.18e-04]
Epoch 74 TRAIN: 137it [00:31,  5.02it/s, epoch=74, grad_norm=196.423, loss=253.4264, lr=8.18e-04]                         
Epoch 74 TRAIN: 137it [00:31,  4.33it/s, epoch=74, grad_norm=196.423, loss=253.4264, lr=8.18e-04]

CV:   0%|          | 0/6 [00:00<?, ?it/s]/home/rookie/Documents/batuka/projects/wenet/wenet/wenet/utils/train_utils.py:689: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  amp_autocast(enabled=dtype is not None,

CV:  17%|█▋        | 1/6 [00:00<00:03,  1.32it/s]
                                                 
2026-04-16 20:30:32,746 INFO Epoch 74 Step 10226 CV info lr 8.1808e-04 cv_loss 471.9141290838068 rank 0 acc 0.3901726305484772
2026-04-16 20:30:32,747 INFO [Rank 0] Checkpoint: save to checkpoint exp/conformer_mongolian/epoch_74.pt
2026-04-16 20:30:32,855 INFO Epoch 75 Step 10226 TRAIN info lr 8.1808e-04 rank 0
2026-04-16 20:30:32,877 INFO using accumulate grad, new batch size is 1 times larger than before

Epoch 75 TRAIN:   0%|          | 0/137 [00:00<?, ?it/s]
Epoch 75 TRAIN:   0%|          | 0/137 [00:03<?, ?it/s, epoch=75, grad_norm=304.935, loss=246.2303, lr=8.18e-04]
Epoch 75 TRAIN:   1%|          | 1/137 [00:03<08:37,  3.80s/it, epoch=75, grad_norm=304.935, loss=246.2303, lr=8.18e-04]
Epoch 75 TRAIN:   1%|          | 1/137 [00:04<08:37,  3.80s/it, epoch=75, grad_norm=439.490, loss=382.9203, lr=8.18e-04]
Epoch 75 TRAIN:   1%|          | 1/137 [00:04<08:37,  3.80s/it, epoch=75, grad_norm=422.988, loss=328.5021, lr=8.18e-04]
Epoch 75 TRAIN:   1%|          | 1/137 [00:04<08:37,  3.80s/it, epoch=75, grad_norm=409.854, loss=333.5368, lr=8.18e-04]
Epoch 75 TRAIN:   1%|          | 1/137 [00:04<08:37,  3.80s/it, epoch=75, grad_norm=492.767, loss=338.1744, lr=8.18e-04]
Epoch 75 TRAIN:   1%|          | 1/137 [00:04<08:37,  3.80s/it, epoch=75, grad_norm=503.011, loss=403.7978, lr=8.19e-04]
Epoch 75 TRAIN:   1%|          | 1/137 [00:04<08:37,  3.80s/it, epoch=75, grad_norm=371.036, loss=342.0572, lr=8.19e-04]
Epoch 75 TRAIN:   5%|▌         | 7/137 [00:04<01:02,  2.08it/s, epoch=75, grad_norm=371.036, loss=342.0572, lr=8.19e-04]
Epoch 75 TRAIN:   5%|▌         | 7/137 [00:04<01:02,  2.08it/s, epoch=75, grad_norm=360.550, loss=314.9648, lr=8.19e-04]
Epoch 75 TRAIN:   5%|▌         | 7/137 [00:04<01:02,  2.08it/s, epoch=75, grad_norm=245.695, loss=316.5452, lr=8.19e-04]
Epoch 75 TRAIN:   5%|▌         | 7/137 [00:04<01:02,  2.08it/s, epoch=75, grad_norm=128.321, loss=261.4142, lr=8.19e-04]
Epoch 75 TRAIN:   5%|▌         | 7/137 [00:04<01:02,  2.08it/s, epoch=75, grad_norm=140.766, loss=303.1444, lr=8.19e-04]
Epoch 75 TRAIN:   8%|▊         | 11/137 [00:04<00:40,  3.10it/s, epoch=75, grad_norm=140.766, loss=303.1444, lr=8.19e-04]
Epoch 75 TRAIN:   8%|▊         | 11/137 [00:05<00:40,  3.10it/s, epoch=75, grad_norm=111.422, loss=279.0536, lr=8.19e-04]
Epoch 75 TRAIN:   8%|▊         | 11/137 [00:05<00:40,  3.10it/s, epoch=75, grad_norm=88.634, loss=298.9607, lr=8.19e-04] 
Epoch 75 TRAIN:   8%|▊         | 11/137 [00:05<00:40,  3.10it/s, epoch=75, grad_norm=158.425, loss=244.7831, lr=8.19e-04]
Epoch 75 TRAIN:   8%|▊         | 11/137 [00:05<00:40,  3.10it/s, epoch=75, grad_norm=200.617, loss=279.8630, lr=8.19e-04]
Epoch 75 TRAIN:  11%|█         | 15/137 [00:05<00:31,  3.87it/s, epoch=75, grad_norm=200.617, loss=279.8630, lr=8.19e-04]
Epoch 75 TRAIN:  11%|█         | 15/137 [00:05<00:31,  3.87it/s, epoch=75, grad_norm=101.527, loss=286.0569, lr=8.19e-04]
Epoch 75 TRAIN:  11%|█         | 15/137 [00:06<00:31,  3.87it/s, epoch=75, grad_norm=104.858, loss=304.1813, lr=8.19e-04]
Epoch 75 TRAIN:  11%|█         | 15/137 [00:06<00:31,  3.87it/s, epoch=75, grad_norm=186.945, loss=315.2855, lr=8.20e-04]
Epoch 75 TRAIN:  11%|█         | 15/137 [00:06<00:31,  3.87it/s, epoch=75, grad_norm=112.552, loss=284.9891, lr=8.20e-04]
Epoch 75 TRAIN:  14%|█▍        | 19/137 [00:06<00:29,  4.00it/s, epoch=75, grad_norm=112.552, loss=284.9891, lr=8.20e-04]
Epoch 75 TRAIN:  14%|█▍        | 19/137 [00:06<00:29,  4.00it/s, epoch=75, grad_norm=106.444, loss=278.2993, lr=8.20e-04]
Epoch 75 TRAIN:  14%|█▍        | 19/137 [00:06<00:29,  4.00it/s, epoch=75, grad_norm=100.383, loss=288.7095, lr=8.20e-04]
Epoch 75 TRAIN:  14%|█▍        | 19/137 [00:07<00:29,  4.00it/s, epoch=75, grad_norm=144.761, loss=330.5174, lr=8.20e-04]
Epoch 75 TRAIN:  16%|█▌        | 22/137 [00:07<00:26,  4.30it/s, epoch=75, grad_norm=144.761, loss=330.5174, lr=8.20e-04]
Epoch 75 TRAIN:  16%|█▌        | 22/137 [00:07<00:26,  4.30it/s, epoch=75, grad_norm=121.120, loss=290.6848, lr=8.20e-04]
Epoch 75 TRAIN:  16%|█▌        | 22/137 [00:07<00:26,  4.30it/s, epoch=75, grad_norm=73.025, loss=270.4196, lr=8.20e-04] 
Epoch 75 TRAIN:  16%|█▌        | 22/137 [00:07<00:26,  4.30it/s, epoch=75, grad_norm=144.939, loss=256.4405, lr=8.20e-04]
Epoch 75 TRAIN:  18%|█▊        | 25/137 [00:07<00:25,  4.45it/s, epoch=75, grad_norm=144.939, loss=256.4405, lr=8.20e-04]
Epoch 75 TRAIN:  18%|█▊        | 25/137 [00:07<00:25,  4.45it/s, epoch=75, grad_norm=181.768, loss=272.8705, lr=8.20e-04]
Epoch 75 TRAIN:  18%|█▊        | 25/137 [00:08<00:25,  4.45it/s, epoch=75, grad_norm=125.091, loss=254.1061, lr=8.20e-04]
Epoch 75 TRAIN:  18%|█▊        | 25/137 [00:08<00:25,  4.45it/s, epoch=75, grad_norm=200.862, loss=271.0188, lr=8.20e-04]
Epoch 75 TRAIN:  20%|██        | 28/137 [00:08<00:23,  4.56it/s, epoch=75, grad_norm=200.862, loss=271.0188, lr=8.20e-04]
Epoch 75 TRAIN:  20%|██        | 28/137 [00:08<00:23,  4.56it/s, epoch=75, grad_norm=87.594, loss=259.7395, lr=8.20e-04] 
Epoch 75 TRAIN:  20%|██        | 28/137 [00:08<00:23,  4.56it/s, epoch=75, grad_norm=90.553, loss=276.1290, lr=8.20e-04]
Epoch 75 TRAIN:  20%|██        | 28/137 [00:08<00:23,  4.56it/s, epoch=75, grad_norm=112.238, loss=297.4142, lr=8.21e-04]
Epoch 75 TRAIN:  23%|██▎       | 31/137 [00:08<00:22,  4.73it/s, epoch=75, grad_norm=112.238, loss=297.4142, lr=8.21e-04]
Epoch 75 TRAIN:  23%|██▎       | 31/137 [00:09<00:22,  4.73it/s, epoch=75, grad_norm=92.436, loss=265.6568, lr=8.21e-04] 
Epoch 75 TRAIN:  23%|██▎       | 31/137 [00:09<00:22,  4.73it/s, epoch=75, grad_norm=352.290, loss=317.2013, lr=8.21e-04]
Epoch 75 TRAIN:  23%|██▎       | 31/137 [00:09<00:22,  4.73it/s, epoch=75, grad_norm=148.301, loss=257.1211, lr=8.21e-04]
Epoch 75 TRAIN:  25%|██▍       | 34/137 [00:09<00:21,  4.81it/s, epoch=75, grad_norm=148.301, loss=257.1211, lr=8.21e-04]
Epoch 75 TRAIN:  25%|██▍       | 34/137 [00:09<00:21,  4.81it/s, epoch=75, grad_norm=119.730, loss=292.2799, lr=8.21e-04]
Epoch 75 TRAIN:  25%|██▍       | 34/137 [00:09<00:21,  4.81it/s, epoch=75, grad_norm=92.180, loss=269.8699, lr=8.21e-04] 
Epoch 75 TRAIN:  25%|██▍       | 34/137 [00:10<00:21,  4.81it/s, epoch=75, grad_norm=168.561, loss=294.6310, lr=8.21e-04]
Epoch 75 TRAIN:  27%|██▋       | 37/137 [00:10<00:21,  4.67it/s, epoch=75, grad_norm=168.561, loss=294.6310, lr=8.21e-04]
Epoch 75 TRAIN:  27%|██▋       | 37/137 [00:10<00:21,  4.67it/s, epoch=75, grad_norm=104.144, loss=242.5399, lr=8.21e-04]
Epoch 75 TRAIN:  27%|██▋       | 37/137 [00:10<00:21,  4.67it/s, epoch=75, grad_norm=99.930, loss=286.8249, lr=8.21e-04] 
Epoch 75 TRAIN:  27%|██▋       | 37/137 [00:10<00:21,  4.67it/s, epoch=75, grad_norm=148.839, loss=296.6509, lr=8.21e-04]
Epoch 75 TRAIN:  29%|██▉       | 40/137 [00:10<00:20,  4.77it/s, epoch=75, grad_norm=148.839, loss=296.6509, lr=8.21e-04]
Epoch 75 TRAIN:  29%|██▉       | 40/137 [00:10<00:20,  4.77it/s, epoch=75, grad_norm=97.749, loss=254.1043, lr=8.21e-04] 
Epoch 75 TRAIN:  29%|██▉       | 40/137 [00:11<00:20,  4.77it/s, epoch=75, grad_norm=95.580, loss=276.0078, lr=8.21e-04]
Epoch 75 TRAIN:  29%|██▉       | 40/137 [00:11<00:20,  4.77it/s, epoch=75, grad_norm=134.845, loss=202.1357, lr=8.22e-04]
Epoch 75 TRAIN:  31%|███▏      | 43/137 [00:11<00:19,  4.72it/s, epoch=75, grad_norm=134.845, loss=202.1357, lr=8.22e-04]
Epoch 75 TRAIN:  31%|███▏      | 43/137 [00:11<00:19,  4.72it/s, epoch=75, grad_norm=116.771, loss=269.4351, lr=8.22e-04]
Epoch 75 TRAIN:  31%|███▏      | 43/137 [00:11<00:19,  4.72it/s, epoch=75, grad_norm=76.188, loss=254.3525, lr=8.22e-04] 
Epoch 75 TRAIN:  31%|███▏      | 43/137 [00:12<00:19,  4.72it/s, epoch=75, grad_norm=132.430, loss=285.0736, lr=8.22e-04]
Epoch 75 TRAIN:  34%|███▎      | 46/137 [00:12<00:19,  4.66it/s, epoch=75, grad_norm=132.430, loss=285.0736, lr=8.22e-04]
Epoch 75 TRAIN:  34%|███▎      | 46/137 [00:12<00:19,  4.66it/s, epoch=75, grad_norm=76.772, loss=247.6040, lr=8.22e-04] 
Epoch 75 TRAIN:  34%|███▎      | 46/137 [00:12<00:19,  4.66it/s, epoch=75, grad_norm=87.572, loss=271.3457, lr=8.22e-04]
Epoch 75 TRAIN:  34%|███▎      | 46/137 [00:12<00:19,  4.66it/s, epoch=75, grad_norm=193.132, loss=318.3464, lr=8.22e-04]
Epoch 75 TRAIN:  36%|███▌      | 49/137 [00:12<00:18,  4.67it/s, epoch=75, grad_norm=193.132, loss=318.3464, lr=8.22e-04]
Epoch 75 TRAIN:  36%|███▌      | 49/137 [00:12<00:18,  4.67it/s, epoch=75, grad_norm=140.829, loss=306.0471, lr=8.22e-04]
Epoch 75 TRAIN:  36%|███▌      | 49/137 [00:13<00:18,  4.67it/s, epoch=75, grad_norm=120.103, loss=293.1613, lr=8.22e-04]
Epoch 75 TRAIN:  36%|███▌      | 49/137 [00:13<00:18,  4.67it/s, epoch=75, grad_norm=94.662, loss=266.7309, lr=8.22e-04] 
Epoch 75 TRAIN:  38%|███▊      | 52/137 [00:13<00:18,  4.67it/s, epoch=75, grad_norm=94.662, loss=266.7309, lr=8.22e-04]
Epoch 75 TRAIN:  38%|███▊      | 52/137 [00:13<00:18,  4.67it/s, epoch=75, grad_norm=132.537, loss=273.2418, lr=8.22e-04]
Epoch 75 TRAIN:  38%|███▊      | 52/137 [00:13<00:18,  4.67it/s, epoch=75, grad_norm=182.443, loss=246.2323, lr=8.22e-04]
Epoch 75 TRAIN:  38%|███▊      | 52/137 [00:13<00:18,  4.67it/s, epoch=75, grad_norm=211.649, loss=219.3694, lr=8.22e-04]
Epoch 75 TRAIN:  40%|████      | 55/137 [00:13<00:17,  4.70it/s, epoch=75, grad_norm=211.649, loss=219.3694, lr=8.22e-04]
Epoch 75 TRAIN:  40%|████      | 55/137 [00:14<00:17,  4.70it/s, epoch=75, grad_norm=119.804, loss=250.5569, lr=8.23e-04]
Epoch 75 TRAIN:  40%|████      | 55/137 [00:14<00:17,  4.70it/s, epoch=75, grad_norm=104.408, loss=267.1498, lr=8.23e-04]
Epoch 75 TRAIN:  40%|████      | 55/137 [00:14<00:17,  4.70it/s, epoch=75, grad_norm=205.153, loss=300.1772, lr=8.23e-04]
Epoch 75 TRAIN:  42%|████▏     | 58/137 [00:14<00:16,  4.67it/s, epoch=75, grad_norm=205.153, loss=300.1772, lr=8.23e-04]
Epoch 75 TRAIN:  42%|████▏     | 58/137 [00:14<00:16,  4.67it/s, epoch=75, grad_norm=158.633, loss=294.8995, lr=8.23e-04]
Epoch 75 TRAIN:  42%|████▏     | 58/137 [00:15<00:16,  4.67it/s, epoch=75, grad_norm=155.913, loss=286.6466, lr=8.23e-04]
Epoch 75 TRAIN:  42%|████▏     | 58/137 [00:15<00:16,  4.67it/s, epoch=75, grad_norm=137.753, loss=285.2435, lr=8.23e-04]
Epoch 75 TRAIN:  45%|████▍     | 61/137 [00:15<00:16,  4.59it/s, epoch=75, grad_norm=137.753, loss=285.2435, lr=8.23e-04]
Epoch 75 TRAIN:  45%|████▍     | 61/137 [00:15<00:16,  4.59it/s, epoch=75, grad_norm=166.987, loss=308.7884, lr=8.23e-04]
Epoch 75 TRAIN:  45%|████▍     | 61/137 [00:15<00:16,  4.59it/s, epoch=75, grad_norm=141.439, loss=333.9698, lr=8.23e-04]
Epoch 75 TRAIN:  45%|████▍     | 61/137 [00:15<00:16,  4.59it/s, epoch=75, grad_norm=119.603, loss=240.3055, lr=8.23e-04]
Epoch 75 TRAIN:  47%|████▋     | 64/137 [00:15<00:15,  4.66it/s, epoch=75, grad_norm=119.603, loss=240.3055, lr=8.23e-04]
Epoch 75 TRAIN:  47%|████▋     | 64/137 [00:16<00:15,  4.66it/s, epoch=75, grad_norm=165.805, loss=253.5502, lr=8.23e-04]
Epoch 75 TRAIN:  47%|████▋     | 64/137 [00:16<00:15,  4.66it/s, epoch=75, grad_norm=177.598, loss=276.2244, lr=8.23e-04]
Epoch 75 TRAIN:  47%|████▋     | 64/137 [00:16<00:15,  4.66it/s, epoch=75, grad_norm=109.154, loss=261.9927, lr=8.23e-04]
Epoch 75 TRAIN:  49%|████▉     | 67/137 [00:16<00:15,  4.65it/s, epoch=75, grad_norm=109.154, loss=261.9927, lr=8.23e-04]
Epoch 75 TRAIN:  49%|████▉     | 67/137 [00:16<00:15,  4.65it/s, epoch=75, grad_norm=104.037, loss=273.9846, lr=8.24e-04]
Epoch 75 TRAIN:  49%|████▉     | 67/137 [00:17<00:15,  4.65it/s, epoch=75, grad_norm=235.974, loss=210.7294, lr=8.24e-04]
Epoch 75 TRAIN:  49%|████▉     | 67/137 [00:17<00:15,  4.65it/s, epoch=75, grad_norm=384.759, loss=264.3757, lr=8.24e-04]
Epoch 75 TRAIN:  51%|█████     | 70/137 [00:17<00:15,  4.42it/s, epoch=75, grad_norm=384.759, loss=264.3757, lr=8.24e-04]
Epoch 75 TRAIN:  51%|█████     | 70/137 [00:17<00:15,  4.42it/s, epoch=75, grad_norm=113.939, loss=258.9209, lr=8.24e-04]
Epoch 75 TRAIN:  51%|█████     | 70/137 [00:17<00:15,  4.42it/s, epoch=75, grad_norm=141.301, loss=273.0068, lr=8.24e-04]
Epoch 75 TRAIN:  51%|█████     | 70/137 [00:17<00:15,  4.42it/s, epoch=75, grad_norm=116.191, loss=279.4962, lr=8.24e-04]
Epoch 75 TRAIN:  53%|█████▎    | 73/137 [00:17<00:14,  4.49it/s, epoch=75, grad_norm=116.191, loss=279.4962, lr=8.24e-04]
Epoch 75 TRAIN:  53%|█████▎    | 73/137 [00:18<00:14,  4.49it/s, epoch=75, grad_norm=98.381, loss=244.4742, lr=8.24e-04] 
Epoch 75 TRAIN:  53%|█████▎    | 73/137 [00:18<00:14,  4.49it/s, epoch=75, grad_norm=143.896, loss=276.5165, lr=8.24e-04]
Epoch 75 TRAIN:  53%|█████▎    | 73/137 [00:18<00:14,  4.49it/s, epoch=75, grad_norm=252.519, loss=281.6173, lr=8.24e-04]
Epoch 75 TRAIN:  55%|█████▌    | 76/137 [00:18<00:13,  4.62it/s, epoch=75, grad_norm=252.519, loss=281.6173, lr=8.24e-04]
Epoch 75 TRAIN:  55%|█████▌    | 76/137 [00:18<00:13,  4.62it/s, epoch=75, grad_norm=190.243, loss=287.8752, lr=8.24e-04]
Epoch 75 TRAIN:  55%|█████▌    | 76/137 [00:19<00:13,  4.62it/s, epoch=75, grad_norm=82.226, loss=225.9490, lr=8.24e-04] 
Epoch 75 TRAIN:  55%|█████▌    | 76/137 [00:19<00:13,  4.62it/s, epoch=75, grad_norm=118.367, loss=289.0245, lr=8.24e-04]
Epoch 75 TRAIN:  58%|█████▊    | 79/137 [00:19<00:13,  4.32it/s, epoch=75, grad_norm=118.367, loss=289.0245, lr=8.24e-04]
Epoch 75 TRAIN:  58%|█████▊    | 79/137 [00:19<00:13,  4.32it/s, epoch=75, grad_norm=165.817, loss=300.7180, lr=8.24e-04]
Epoch 75 TRAIN:  58%|█████▊    | 79/137 [00:19<00:13,  4.32it/s, epoch=75, grad_norm=234.565, loss=239.6634, lr=8.25e-04]
Epoch 75 TRAIN:  58%|█████▊    | 79/137 [00:20<00:13,  4.32it/s, epoch=75, grad_norm=115.400, loss=286.8508, lr=8.25e-04]
Epoch 75 TRAIN:  60%|█████▉    | 82/137 [00:20<00:12,  4.43it/s, epoch=75, grad_norm=115.400, loss=286.8508, lr=8.25e-04]
Epoch 75 TRAIN:  60%|█████▉    | 82/137 [00:20<00:12,  4.43it/s, epoch=75, grad_norm=185.226, loss=323.9146, lr=8.25e-04]
Epoch 75 TRAIN:  60%|█████▉    | 82/137 [00:20<00:12,  4.43it/s, epoch=75, grad_norm=159.576, loss=311.9872, lr=8.25e-04]
Epoch 75 TRAIN:  60%|█████▉    | 82/137 [00:20<00:12,  4.43it/s, epoch=75, grad_norm=147.560, loss=276.6924, lr=8.25e-04]
Epoch 75 TRAIN:  62%|██████▏   | 85/137 [00:20<00:12,  4.21it/s, epoch=75, grad_norm=147.560, loss=276.6924, lr=8.25e-04]
Epoch 75 TRAIN:  62%|██████▏   | 85/137 [00:20<00:12,  4.21it/s, epoch=75, grad_norm=178.983, loss=313.8007, lr=8.25e-04]
Epoch 75 TRAIN:  62%|██████▏   | 85/137 [00:21<00:12,  4.21it/s, epoch=75, grad_norm=154.692, loss=345.4655, lr=8.25e-04]
Epoch 75 TRAIN:  62%|██████▏   | 85/137 [00:21<00:12,  4.21it/s, epoch=75, grad_norm=116.798, loss=275.7612, lr=8.25e-04]
Epoch 75 TRAIN:  64%|██████▍   | 88/137 [00:21<00:11,  4.37it/s, epoch=75, grad_norm=116.798, loss=275.7612, lr=8.25e-04]
Epoch 75 TRAIN:  64%|██████▍   | 88/137 [00:21<00:11,  4.37it/s, epoch=75, grad_norm=324.636, loss=280.8416, lr=8.25e-04]
Epoch 75 TRAIN:  64%|██████▍   | 88/137 [00:21<00:11,  4.37it/s, epoch=75, grad_norm=400.104, loss=204.5528, lr=8.25e-04]
Epoch 75 TRAIN:  64%|██████▍   | 88/137 [00:21<00:11,  4.37it/s, epoch=75, grad_norm=142.171, loss=287.4832, lr=8.25e-04]
Epoch 75 TRAIN:  66%|██████▋   | 91/137 [00:21<00:09,  4.63it/s, epoch=75, grad_norm=142.171, loss=287.4832, lr=8.25e-04]
Epoch 75 TRAIN:  66%|██████▋   | 91/137 [00:22<00:09,  4.63it/s, epoch=75, grad_norm=190.743, loss=265.3525, lr=8.25e-04]
Epoch 75 TRAIN:  66%|██████▋   | 91/137 [00:22<00:09,  4.63it/s, epoch=75, grad_norm=296.089, loss=233.0852, lr=8.26e-04]
Epoch 75 TRAIN:  66%|██████▋   | 91/137 [00:22<00:09,  4.63it/s, epoch=75, grad_norm=119.795, loss=282.7914, lr=8.26e-04]
Epoch 75 TRAIN:  69%|██████▊   | 94/137 [00:22<00:09,  4.58it/s, epoch=75, grad_norm=119.795, loss=282.7914, lr=8.26e-04]
Epoch 75 TRAIN:  69%|██████▊   | 94/137 [00:22<00:09,  4.58it/s, epoch=75, grad_norm=121.377, loss=278.3251, lr=8.26e-04]
Epoch 75 TRAIN:  69%|██████▊   | 94/137 [00:23<00:09,  4.58it/s, epoch=75, grad_norm=305.123, loss=359.1562, lr=8.26e-04]
Epoch 75 TRAIN:  69%|██████▊   | 94/137 [00:23<00:09,  4.58it/s, epoch=75, grad_norm=172.938, loss=253.6810, lr=8.26e-04]
Epoch 75 TRAIN:  71%|███████   | 97/137 [00:23<00:09,  4.44it/s, epoch=75, grad_norm=172.938, loss=253.6810, lr=8.26e-04]
Epoch 75 TRAIN:  71%|███████   | 97/137 [00:23<00:09,  4.44it/s, epoch=75, grad_norm=211.187, loss=247.4619, lr=8.26e-04]
Epoch 75 TRAIN:  71%|███████   | 97/137 [00:23<00:09,  4.44it/s, epoch=75, grad_norm=121.621, loss=247.1585, lr=8.26e-04]
Epoch 75 TRAIN:  71%|███████   | 97/137 [00:23<00:09,  4.44it/s, epoch=75, grad_norm=97.611, loss=242.5409, lr=8.26e-04] 2026-04-16 20:30:56,860 DEBUG TRAIN | steps/sec 4.174| Batch 75/100 loss 242.540909 loss_att 190.996613 loss_ctc 362.810913 th_accuracy 0.440162 lr 8.2608e-04 grad_norm 97.610664 rank 0

Epoch 75 TRAIN:  73%|███████▎  | 100/137 [00:23<00:08,  4.61it/s, epoch=75, grad_norm=97.611, loss=242.5409, lr=8.26e-04]
Epoch 75 TRAIN:  73%|███████▎  | 100/137 [00:24<00:08,  4.61it/s, epoch=75, grad_norm=161.537, loss=332.0103, lr=8.26e-04]
Epoch 75 TRAIN:  73%|███████▎  | 100/137 [00:24<00:08,  4.61it/s, epoch=75, grad_norm=154.568, loss=205.1820, lr=8.26e-04]
Epoch 75 TRAIN:  73%|███████▎  | 100/137 [00:24<00:08,  4.61it/s, epoch=75, grad_norm=151.728, loss=294.4377, lr=8.26e-04]
Epoch 75 TRAIN:  75%|███████▌  | 103/137 [00:24<00:07,  4.54it/s, epoch=75, grad_norm=151.728, loss=294.4377, lr=8.26e-04]
Epoch 75 TRAIN:  75%|███████▌  | 103/137 [00:24<00:07,  4.54it/s, epoch=75, grad_norm=213.224, loss=334.9796, lr=8.26e-04]
Epoch 75 TRAIN:  75%|███████▌  | 103/137 [00:25<00:07,  4.54it/s, epoch=75, grad_norm=184.229, loss=320.2039, lr=8.26e-04]
Epoch 75 TRAIN:  75%|███████▌  | 103/137 [00:25<00:07,  4.54it/s, epoch=75, grad_norm=244.988, loss=226.4593, lr=8.27e-04]
Epoch 75 TRAIN:  77%|███████▋  | 106/137 [00:25<00:06,  4.53it/s, epoch=75, grad_norm=244.988, loss=226.4593, lr=8.27e-04]
Epoch 75 TRAIN:  77%|███████▋  | 106/137 [00:25<00:06,  4.53it/s, epoch=75, grad_norm=154.419, loss=294.6382, lr=8.27e-04]
Epoch 75 TRAIN:  77%|███████▋  | 106/137 [00:25<00:06,  4.53it/s, epoch=75, grad_norm=203.190, loss=168.4506, lr=8.27e-04]
Epoch 75 TRAIN:  77%|███████▋  | 106/137 [00:25<00:06,  4.53it/s, epoch=75, grad_norm=146.624, loss=263.1396, lr=8.27e-04]
Epoch 75 TRAIN:  80%|███████▉  | 109/137 [00:25<00:05,  4.73it/s, epoch=75, grad_norm=146.624, loss=263.1396, lr=8.27e-04]
Epoch 75 TRAIN:  80%|███████▉  | 109/137 [00:26<00:05,  4.73it/s, epoch=75, grad_norm=107.018, loss=206.2662, lr=8.27e-04]
Epoch 75 TRAIN:  80%|███████▉  | 109/137 [00:26<00:05,  4.73it/s, epoch=75, grad_norm=133.911, loss=235.3584, lr=8.27e-04]
Epoch 75 TRAIN:  80%|███████▉  | 109/137 [00:26<00:05,  4.73it/s, epoch=75, grad_norm=345.013, loss=307.3133, lr=8.27e-04]
Epoch 75 TRAIN:  82%|████████▏ | 112/137 [00:26<00:05,  4.64it/s, epoch=75, grad_norm=345.013, loss=307.3133, lr=8.27e-04]
Epoch 75 TRAIN:  82%|████████▏ | 112/137 [00:26<00:05,  4.64it/s, epoch=75, grad_norm=321.216, loss=334.5500, lr=8.27e-04]
Epoch 75 TRAIN:  82%|████████▏ | 112/137 [00:27<00:05,  4.64it/s, epoch=75, grad_norm=282.445, loss=299.9861, lr=8.27e-04]
Epoch 75 TRAIN:  82%|████████▏ | 112/137 [00:27<00:05,  4.64it/s, epoch=75, grad_norm=136.325, loss=251.6382, lr=8.27e-04]
Epoch 75 TRAIN:  84%|████████▍ | 115/137 [00:27<00:05,  4.32it/s, epoch=75, grad_norm=136.325, loss=251.6382, lr=8.27e-04]
Epoch 75 TRAIN:  84%|████████▍ | 115/137 [00:27<00:05,  4.32it/s, epoch=75, grad_norm=120.127, loss=185.9080, lr=8.27e-04]
Epoch 75 TRAIN:  84%|████████▍ | 115/137 [00:27<00:05,  4.32it/s, epoch=75, grad_norm=138.960, loss=266.1795, lr=8.27e-04]
Epoch 75 TRAIN:  84%|████████▍ | 115/137 [00:27<00:05,  4.32it/s, epoch=75, grad_norm=132.161, loss=232.0788, lr=8.28e-04]
Epoch 75 TRAIN:  86%|████████▌ | 118/137 [00:27<00:04,  4.68it/s, epoch=75, grad_norm=132.161, loss=232.0788, lr=8.28e-04]
Epoch 75 TRAIN:  86%|████████▌ | 118/137 [00:28<00:04,  4.68it/s, epoch=75, grad_norm=195.621, loss=279.3120, lr=8.28e-04]
Epoch 75 TRAIN:  86%|████████▌ | 118/137 [00:28<00:04,  4.68it/s, epoch=75, grad_norm=116.454, loss=241.9559, lr=8.28e-04]
Epoch 75 TRAIN:  86%|████████▌ | 118/137 [00:28<00:04,  4.68it/s, epoch=75, grad_norm=176.020, loss=335.7162, lr=8.28e-04]
Epoch 75 TRAIN:  88%|████████▊ | 121/137 [00:28<00:03,  4.65it/s, epoch=75, grad_norm=176.020, loss=335.7162, lr=8.28e-04]
Epoch 75 TRAIN:  88%|████████▊ | 121/137 [00:28<00:03,  4.65it/s, epoch=75, grad_norm=329.603, loss=213.5273, lr=8.28e-04]
Epoch 75 TRAIN:  88%|████████▊ | 121/137 [00:29<00:03,  4.65it/s, epoch=75, grad_norm=195.473, loss=272.0705, lr=8.28e-04]
Epoch 75 TRAIN:  88%|████████▊ | 121/137 [00:29<00:03,  4.65it/s, epoch=75, grad_norm=210.210, loss=306.3832, lr=8.28e-04]
Epoch 75 TRAIN:  91%|█████████ | 124/137 [00:29<00:02,  4.51it/s, epoch=75, grad_norm=210.210, loss=306.3832, lr=8.28e-04]
Epoch 75 TRAIN:  91%|█████████ | 124/137 [00:29<00:02,  4.51it/s, epoch=75, grad_norm=283.220, loss=297.6806, lr=8.28e-04]
Epoch 75 TRAIN:  91%|█████████ | 124/137 [00:29<00:02,  4.51it/s, epoch=75, grad_norm=154.828, loss=278.2814, lr=8.28e-04]
Epoch 75 TRAIN:  91%|█████████ | 124/137 [00:29<00:02,  4.51it/s, epoch=75, grad_norm=185.080, loss=281.9272, lr=8.28e-04]
Epoch 75 TRAIN:  93%|█████████▎| 127/137 [00:29<00:02,  4.78it/s, epoch=75, grad_norm=185.080, loss=281.9272, lr=8.28e-04]
Epoch 75 TRAIN:  93%|█████████▎| 127/137 [00:29<00:02,  4.78it/s, epoch=75, grad_norm=267.614, loss=265.5936, lr=8.28e-04]
Epoch 75 TRAIN:  93%|█████████▎| 127/137 [00:30<00:02,  4.78it/s, epoch=75, grad_norm=191.228, loss=324.2894, lr=8.28e-04]
Epoch 75 TRAIN:  93%|█████████▎| 127/137 [00:30<00:02,  4.78it/s, epoch=75, grad_norm=91.266, loss=234.3214, lr=8.28e-04] 
Epoch 75 TRAIN:  95%|█████████▍| 130/137 [00:30<00:01,  4.49it/s, epoch=75, grad_norm=91.266, loss=234.3214, lr=8.28e-04]
Epoch 75 TRAIN:  95%|█████████▍| 130/137 [00:30<00:01,  4.49it/s, epoch=75, grad_norm=209.674, loss=344.8550, lr=8.29e-04]
Epoch 75 TRAIN:  95%|█████████▍| 130/137 [00:30<00:01,  4.49it/s, epoch=75, grad_norm=176.537, loss=222.4970, lr=8.29e-04]
Epoch 75 TRAIN:  95%|█████████▍| 130/137 [00:31<00:01,  4.49it/s, epoch=75, grad_norm=249.443, loss=377.1823, lr=8.29e-04]
Epoch 75 TRAIN:  95%|█████████▍| 130/137 [00:31<00:01,  4.49it/s, epoch=75, grad_norm=210.791, loss=331.9170, lr=8.29e-04]
Epoch 75 TRAIN:  98%|█████████▊| 134/137 [00:31<00:00,  4.62it/s, epoch=75, grad_norm=210.791, loss=331.9170, lr=8.29e-04]
Epoch 75 TRAIN:  98%|█████████▊| 134/137 [00:31<00:00,  4.62it/s, epoch=75, grad_norm=313.270, loss=248.2886, lr=8.29e-04]
Epoch 75 TRAIN:  98%|█████████▊| 134/137 [00:31<00:00,  4.62it/s, epoch=75, grad_norm=271.214, loss=307.8275, lr=8.29e-04]
Epoch 75 TRAIN:  99%|█████████▉| 136/137 [00:31<00:00,  4.27it/s, epoch=75, grad_norm=271.214, loss=307.8275, lr=8.29e-04]

CV:   0%|          | 0/6 [00:00<?, ?it/s]/home/rookie/Documents/batuka/projects/wenet/wenet/wenet/utils/train_utils.py:689: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  amp_autocast(enabled=dtype is not None,

CV:  17%|█▋        | 1/6 [00:00<00:03,  1.31it/s]
                                                 
2026-04-16 20:31:05,606 INFO Epoch 75 Step 10362 CV info lr 8.2896e-04 cv_loss 440.71343716708094 rank 0 acc 0.3727468053499858
2026-04-16 20:31:05,608 INFO [Rank 0] Checkpoint: save to checkpoint exp/conformer_mongolian/epoch_75.pt
2026-04-16 20:31:05,721 INFO Epoch 76 Step 10362 TRAIN info lr 8.2896e-04 rank 0
2026-04-16 20:31:05,743 INFO using accumulate grad, new batch size is 1 times larger than before

Epoch 76 TRAIN:   0%|          | 0/136 [00:00<?, ?it/s]
Epoch 76 TRAIN:   0%|          | 0/136 [00:03<?, ?it/s, epoch=76, grad_norm=484.203, loss=393.6280, lr=8.29e-04]
Epoch 76 TRAIN:   1%|          | 1/136 [00:03<07:48,  3.47s/it, epoch=76, grad_norm=484.203, loss=393.6280, lr=8.29e-04]
Epoch 76 TRAIN:   1%|          | 1/136 [00:04<07:48,  3.47s/it, epoch=76, grad_norm=563.491, loss=387.5185, lr=8.29e-04]
Epoch 76 TRAIN:   1%|▏         | 2/136 [00:04<03:57,  1.77s/it, epoch=76, grad_norm=563.491, loss=387.5185, lr=8.29e-04]
Epoch 76 TRAIN:   1%|▏         | 2/136 [00:04<03:57,  1.77s/it, epoch=76, grad_norm=446.414, loss=388.3327, lr=8.29e-04]
Epoch 76 TRAIN:   1%|▏         | 2/136 [00:04<03:57,  1.77s/it, epoch=76, grad_norm=363.129, loss=332.5999, lr=8.29e-04]
Epoch 76 TRAIN:   1%|▏         | 2/136 [00:04<03:57,  1.77s/it, epoch=76, grad_norm=303.693, loss=282.0256, lr=8.29e-04]
Epoch 76 TRAIN:   1%|▏         | 2/136 [00:04<03:57,  1.77s/it, epoch=76, grad_norm=356.145, loss=317.3801, lr=8.29e-04]
Epoch 76 TRAIN:   1%|▏         | 2/136 [00:04<03:57,  1.77s/it, epoch=76, grad_norm=213.088, loss=225.3534, lr=8.30e-04]
Epoch 76 TRAIN:   1%|▏         | 2/136 [00:04<03:57,  1.77s/it, epoch=76, grad_norm=153.774, loss=212.7596, lr=8.30e-04]
Epoch 76 TRAIN:   1%|▏         | 2/136 [00:04<03:57,  1.77s/it, epoch=76, grad_norm=128.327, loss=294.9658, lr=8.30e-04]
Epoch 76 TRAIN:   7%|▋         | 9/136 [00:04<00:42,  3.02it/s, epoch=76, grad_norm=128.327, loss=294.9658, lr=8.30e-04]
Epoch 76 TRAIN:   7%|▋         | 9/136 [00:04<00:42,  3.02it/s, epoch=76, grad_norm=91.770, loss=280.4166, lr=8.30e-04] 
Epoch 76 TRAIN:   7%|▋         | 9/136 [00:04<00:42,  3.02it/s, epoch=76, grad_norm=110.436, loss=282.6836, lr=8.30e-04]
Epoch 76 TRAIN:   7%|▋         | 9/136 [00:05<00:42,  3.02it/s, epoch=76, grad_norm=99.784, loss=314.9449, lr=8.30e-04] 
Epoch 76 TRAIN:   7%|▋         | 9/136 [00:05<00:42,  3.02it/s, epoch=76, grad_norm=94.028, loss=294.6267, lr=8.30e-04]
Epoch 76 TRAIN:  10%|▉         | 13/136 [00:05<00:32,  3.82it/s, epoch=76, grad_norm=94.028, loss=294.6267, lr=8.30e-04]
Epoch 76 TRAIN:  10%|▉         | 13/136 [00:05<00:32,  3.82it/s, epoch=76, grad_norm=148.196, loss=275.8264, lr=8.30e-04]
Epoch 76 TRAIN:  10%|▉         | 13/136 [00:05<00:32,  3.82it/s, epoch=76, grad_norm=122.268, loss=292.4788, lr=8.30e-04]
Epoch 76 TRAIN:  10%|▉         | 13/136 [00:05<00:32,  3.82it/s, epoch=76, grad_norm=130.718, loss=270.0133, lr=8.30e-04]
Epoch 76 TRAIN:  10%|▉         | 13/136 [00:06<00:32,  3.82it/s, epoch=76, grad_norm=148.422, loss=282.4467, lr=8.30e-04]
Epoch 76 TRAIN:  12%|█▎        | 17/136 [00:06<00:27,  4.34it/s, epoch=76, grad_norm=148.422, loss=282.4467, lr=8.30e-04]
Epoch 76 TRAIN:  12%|█▎        | 17/136 [00:06<00:27,  4.34it/s, epoch=76, grad_norm=132.315, loss=284.5545, lr=8.30e-04]
Epoch 76 TRAIN:  12%|█▎        | 17/136 [00:06<00:27,  4.34it/s, epoch=76, grad_norm=92.513, loss=262.8655, lr=8.30e-04] 
Epoch 76 TRAIN:  12%|█▎        | 17/136 [00:06<00:27,  4.34it/s, epoch=76, grad_norm=72.698, loss=285.5053, lr=8.31e-04]
Epoch 76 TRAIN:  15%|█▍        | 20/136 [00:06<00:25,  4.51it/s, epoch=76, grad_norm=72.698, loss=285.5053, lr=8.31e-04]
Epoch 76 TRAIN:  15%|█▍        | 20/136 [00:06<00:25,  4.51it/s, epoch=76, grad_norm=139.506, loss=327.6840, lr=8.31e-04]
Epoch 76 TRAIN:  15%|█▍        | 20/136 [00:06<00:25,  4.51it/s, epoch=76, grad_norm=75.997, loss=273.6660, lr=8.31e-04] 
Epoch 76 TRAIN:  15%|█▍        | 20/136 [00:07<00:25,  4.51it/s, epoch=76, grad_norm=124.742, loss=283.6743, lr=8.31e-04]
Epoch 76 TRAIN:  17%|█▋        | 23/136 [00:07<00:24,  4.67it/s, epoch=76, grad_norm=124.742, loss=283.6743, lr=8.31e-04]
Epoch 76 TRAIN:  17%|█▋        | 23/136 [00:07<00:24,  4.67it/s, epoch=76, grad_norm=87.730, loss=236.8907, lr=8.31e-04] 
Epoch 76 TRAIN:  17%|█▋        | 23/136 [00:07<00:24,  4.67it/s, epoch=76, grad_norm=119.099, loss=314.3780, lr=8.31e-04]
Epoch 76 TRAIN:  17%|█▋        | 23/136 [00:07<00:24,  4.67it/s, epoch=76, grad_norm=91.558, loss=269.4589, lr=8.31e-04] 
Epoch 76 TRAIN:  19%|█▉        | 26/136 [00:07<00:22,  4.81it/s, epoch=76, grad_norm=91.558, loss=269.4589, lr=8.31e-04]
Epoch 76 TRAIN:  19%|█▉        | 26/136 [00:07<00:22,  4.81it/s, epoch=76, grad_norm=105.515, loss=289.8113, lr=8.31e-04]
Epoch 76 TRAIN:  19%|█▉        | 26/136 [00:08<00:22,  4.81it/s, epoch=76, grad_norm=130.392, loss=283.2328, lr=8.31e-04]
Epoch 76 TRAIN:  19%|█▉        | 26/136 [00:08<00:22,  4.81it/s, epoch=76, grad_norm=96.582, loss=314.0349, lr=8.31e-04] 
Epoch 76 TRAIN:  21%|██▏       | 29/136 [00:08<00:21,  4.91it/s, epoch=76, grad_norm=96.582, loss=314.0349, lr=8.31e-04]
Epoch 76 TRAIN:  21%|██▏       | 29/136 [00:08<00:21,  4.91it/s, epoch=76, grad_norm=184.109, loss=322.8599, lr=8.31e-04]
Epoch 76 TRAIN:  21%|██▏       | 29/136 [00:08<00:21,  4.91it/s, epoch=76, grad_norm=167.371, loss=260.1906, lr=8.31e-04]
Epoch 76 TRAIN:  21%|██▏       | 29/136 [00:09<00:21,  4.91it/s, epoch=76, grad_norm=121.043, loss=299.3864, lr=8.32e-04]
Epoch 76 TRAIN:  24%|██▎       | 32/136 [00:09<00:21,  4.83it/s, epoch=76, grad_norm=121.043, loss=299.3864, lr=8.32e-04]
Epoch 76 TRAIN:  24%|██▎       | 32/136 [00:09<00:21,  4.83it/s, epoch=76, grad_norm=114.104, loss=269.6650, lr=8.32e-04]
Epoch 76 TRAIN:  24%|██▎       | 32/136 [00:09<00:21,  4.83it/s, epoch=76, grad_norm=146.751, loss=301.8264, lr=8.32e-04]
Epoch 76 TRAIN:  24%|██▎       | 32/136 [00:09<00:21,  4.83it/s, epoch=76, grad_norm=133.326, loss=286.1079, lr=8.32e-04]
Epoch 76 TRAIN:  26%|██▌       | 35/136 [00:09<00:20,  4.92it/s, epoch=76, grad_norm=133.326, loss=286.1079, lr=8.32e-04]
Epoch 76 TRAIN:  26%|██▌       | 35/136 [00:09<00:20,  4.92it/s, epoch=76, grad_norm=117.687, loss=272.4564, lr=8.32e-04]
Epoch 76 TRAIN:  26%|██▌       | 35/136 [00:10<00:20,  4.92it/s, epoch=76, grad_norm=89.153, loss=275.5331, lr=8.32e-04] 
Epoch 76 TRAIN:  26%|██▌       | 35/136 [00:10<00:20,  4.92it/s, epoch=76, grad_norm=121.164, loss=282.8642, lr=8.32e-04]
Epoch 76 TRAIN:  28%|██▊       | 38/136 [00:10<00:20,  4.83it/s, epoch=76, grad_norm=121.164, loss=282.8642, lr=8.32e-04]
Epoch 76 TRAIN:  28%|██▊       | 38/136 [00:10<00:20,  4.83it/s, epoch=76, grad_norm=120.373, loss=264.6775, lr=8.32e-04]
Epoch 76 TRAIN:  28%|██▊       | 38/136 [00:10<00:20,  4.83it/s, epoch=76, grad_norm=90.841, loss=281.6220, lr=8.32e-04] 
Epoch 76 TRAIN:  28%|██▊       | 38/136 [00:10<00:20,  4.83it/s, epoch=76, grad_norm=102.934, loss=285.1395, lr=8.32e-04]
Epoch 76 TRAIN:  30%|███       | 41/136 [00:10<00:19,  4.84it/s, epoch=76, grad_norm=102.934, loss=285.1395, lr=8.32e-04]
Epoch 76 TRAIN:  30%|███       | 41/136 [00:11<00:19,  4.84it/s, epoch=76, grad_norm=96.616, loss=262.2115, lr=8.32e-04] 
Epoch 76 TRAIN:  30%|███       | 41/136 [00:11<00:19,  4.84it/s, epoch=76, grad_norm=248.932, loss=257.7352, lr=8.32e-04]
Epoch 76 TRAIN:  30%|███       | 41/136 [00:11<00:19,  4.84it/s, epoch=76, grad_norm=93.459, loss=265.4440, lr=8.32e-04] 
Epoch 76 TRAIN:  32%|███▏      | 44/136 [00:11<00:19,  4.78it/s, epoch=76, grad_norm=93.459, loss=265.4440, lr=8.32e-04]
Epoch 76 TRAIN:  32%|███▏      | 44/136 [00:11<00:19,  4.78it/s, epoch=76, grad_norm=83.677, loss=279.2056, lr=8.33e-04]
Epoch 76 TRAIN:  32%|███▏      | 44/136 [00:11<00:19,  4.78it/s, epoch=76, grad_norm=134.013, loss=304.3265, lr=8.33e-04]
Epoch 76 TRAIN:  32%|███▏      | 44/136 [00:12<00:19,  4.78it/s, epoch=76, grad_norm=90.683, loss=279.3878, lr=8.33e-04] 
Epoch 76 TRAIN:  35%|███▍      | 47/136 [00:12<00:18,  4.80it/s, epoch=76, grad_norm=90.683, loss=279.3878, lr=8.33e-04]
Epoch 76 TRAIN:  35%|███▍      | 47/136 [00:12<00:18,  4.80it/s, epoch=76, grad_norm=112.268, loss=284.8015, lr=8.33e-04]
Epoch 76 TRAIN:  35%|███▍      | 47/136 [00:12<00:18,  4.80it/s, epoch=76, grad_norm=129.809, loss=257.1248, lr=8.33e-04]
Epoch 76 TRAIN:  35%|███▍      | 47/136 [00:12<00:18,  4.80it/s, epoch=76, grad_norm=142.701, loss=272.8698, lr=8.33e-04]
Epoch 76 TRAIN:  37%|███▋      | 50/136 [00:12<00:17,  4.87it/s, epoch=76, grad_norm=142.701, loss=272.8698, lr=8.33e-04]
Epoch 76 TRAIN:  37%|███▋      | 50/136 [00:12<00:17,  4.87it/s, epoch=76, grad_norm=128.507, loss=313.9071, lr=8.33e-04]
Epoch 76 TRAIN:  37%|███▋      | 50/136 [00:13<00:17,  4.87it/s, epoch=76, grad_norm=104.795, loss=288.3190, lr=8.33e-04]
Epoch 76 TRAIN:  37%|███▋      | 50/136 [00:13<00:17,  4.87it/s, epoch=76, grad_norm=143.978, loss=219.1691, lr=8.33e-04]
Epoch 76 TRAIN:  39%|███▉      | 53/136 [00:13<00:17,  4.88it/s, epoch=76, grad_norm=143.978, loss=219.1691, lr=8.33e-04]
Epoch 76 TRAIN:  39%|███▉      | 53/136 [00:13<00:17,  4.88it/s, epoch=76, grad_norm=121.401, loss=270.6466, lr=8.33e-04]
Epoch 76 TRAIN:  39%|███▉      | 53/136 [00:13<00:17,  4.88it/s, epoch=76, grad_norm=94.278, loss=230.6746, lr=8.33e-04] 
Epoch 76 TRAIN:  39%|███▉      | 53/136 [00:13<00:17,  4.88it/s, epoch=76, grad_norm=145.516, loss=302.5155, lr=8.33e-04]
Epoch 76 TRAIN:  41%|████      | 56/136 [00:13<00:16,  4.88it/s, epoch=76, grad_norm=145.516, loss=302.5155, lr=8.33e-04]
Epoch 76 TRAIN:  41%|████      | 56/136 [00:14<00:16,  4.88it/s, epoch=76, grad_norm=92.853, loss=282.4077, lr=8.34e-04] 
Epoch 76 TRAIN:  41%|████      | 56/136 [00:14<00:16,  4.88it/s, epoch=76, grad_norm=123.499, loss=278.6821, lr=8.34e-04]
Epoch 76 TRAIN:  41%|████      | 56/136 [00:14<00:16,  4.88it/s, epoch=76, grad_norm=141.068, loss=284.2625, lr=8.34e-04]
Epoch 76 TRAIN:  43%|████▎     | 59/136 [00:14<00:16,  4.69it/s, epoch=76, grad_norm=141.068, loss=284.2625, lr=8.34e-04]
Epoch 76 TRAIN:  43%|████▎     | 59/136 [00:14<00:16,  4.69it/s, epoch=76, grad_norm=87.261, loss=245.4232, lr=8.34e-04] 
Epoch 76 TRAIN:  43%|████▎     | 59/136 [00:15<00:16,  4.69it/s, epoch=76, grad_norm=87.099, loss=248.3615, lr=8.34e-04]
Epoch 76 TRAIN:  43%|████▎     | 59/136 [00:15<00:16,  4.69it/s, epoch=76, grad_norm=173.070, loss=292.2541, lr=8.34e-04]
Epoch 76 TRAIN:  46%|████▌     | 62/136 [00:15<00:16,  4.57it/s, epoch=76, grad_norm=173.070, loss=292.2541, lr=8.34e-04]
Epoch 76 TRAIN:  46%|████▌     | 62/136 [00:15<00:16,  4.57it/s, epoch=76, grad_norm=103.787, loss=253.6941, lr=8.34e-04]
Epoch 76 TRAIN:  46%|████▌     | 62/136 [00:15<00:16,  4.57it/s, epoch=76, grad_norm=119.122, loss=287.7873, lr=8.34e-04]
Epoch 76 TRAIN:  46%|████▌     | 62/136 [00:15<00:16,  4.57it/s, epoch=76, grad_norm=144.884, loss=342.2907, lr=8.34e-04]
Epoch 76 TRAIN:  48%|████▊     | 65/136 [00:15<00:15,  4.61it/s, epoch=76, grad_norm=144.884, loss=342.2907, lr=8.34e-04]
Epoch 76 TRAIN:  48%|████▊     | 65/136 [00:16<00:15,  4.61it/s, epoch=76, grad_norm=83.228, loss=239.5686, lr=8.34e-04] 
Epoch 76 TRAIN:  48%|████▊     | 65/136 [00:16<00:15,  4.61it/s, epoch=76, grad_norm=116.786, loss=278.7677, lr=8.34e-04]
Epoch 76 TRAIN:  48%|████▊     | 65/136 [00:16<00:15,  4.61it/s, epoch=76, grad_norm=133.978, loss=259.0751, lr=8.34e-04]
Epoch 76 TRAIN:  50%|█████     | 68/136 [00:16<00:14,  4.59it/s, epoch=76, grad_norm=133.978, loss=259.0751, lr=8.34e-04]
Epoch 76 TRAIN:  50%|█████     | 68/136 [00:16<00:14,  4.59it/s, epoch=76, grad_norm=167.985, loss=307.9974, lr=8.34e-04]
Epoch 76 TRAIN:  50%|█████     | 68/136 [00:17<00:14,  4.59it/s, epoch=76, grad_norm=180.581, loss=244.0466, lr=8.35e-04]
Epoch 76 TRAIN:  50%|█████     | 68/136 [00:17<00:14,  4.59it/s, epoch=76, grad_norm=119.164, loss=291.4986, lr=8.35e-04]
Epoch 76 TRAIN:  52%|█████▏    | 71/136 [00:17<00:14,  4.51it/s, epoch=76, grad_norm=119.164, loss=291.4986, lr=8.35e-04]
Epoch 76 TRAIN:  52%|█████▏    | 71/136 [00:17<00:14,  4.51it/s, epoch=76, grad_norm=75.488, loss=234.2998, lr=8.35e-04] 
Epoch 76 TRAIN:  52%|█████▏    | 71/136 [00:17<00:14,  4.51it/s, epoch=76, grad_norm=207.619, loss=275.5952, lr=8.35e-04]
Epoch 76 TRAIN:  52%|█████▏    | 71/136 [00:18<00:14,  4.51it/s, epoch=76, grad_norm=228.890, loss=335.5034, lr=8.35e-04]
Epoch 76 TRAIN:  54%|█████▍    | 74/136 [00:18<00:13,  4.48it/s, epoch=76, grad_norm=228.890, loss=335.5034, lr=8.35e-04]
Epoch 76 TRAIN:  54%|█████▍    | 74/136 [00:18<00:13,  4.48it/s, epoch=76, grad_norm=120.947, loss=294.4914, lr=8.35e-04]
Epoch 76 TRAIN:  54%|█████▍    | 74/136 [00:18<00:13,  4.48it/s, epoch=76, grad_norm=171.232, loss=319.8987, lr=8.35e-04]
Epoch 76 TRAIN:  54%|█████▍    | 74/136 [00:18<00:13,  4.48it/s, epoch=76, grad_norm=153.387, loss=302.6239, lr=8.35e-04]
Epoch 76 TRAIN:  57%|█████▋    | 77/136 [00:18<00:13,  4.32it/s, epoch=76, grad_norm=153.387, loss=302.6239, lr=8.35e-04]
Epoch 76 TRAIN:  57%|█████▋    | 77/136 [00:18<00:13,  4.32it/s, epoch=76, grad_norm=224.422, loss=166.9448, lr=8.35e-04]
Epoch 76 TRAIN:  57%|█████▋    | 77/136 [00:19<00:13,  4.32it/s, epoch=76, grad_norm=123.678, loss=228.8257, lr=8.35e-04]
Epoch 76 TRAIN:  57%|█████▋    | 77/136 [00:19<00:13,  4.32it/s, epoch=76, grad_norm=119.686, loss=213.8963, lr=8.35e-04]
Epoch 76 TRAIN:  59%|█████▉    | 80/136 [00:19<00:12,  4.37it/s, epoch=76, grad_norm=119.686, loss=213.8963, lr=8.35e-04]
Epoch 76 TRAIN:  59%|█████▉    | 80/136 [00:19<00:12,  4.37it/s, epoch=76, grad_norm=246.768, loss=310.8401, lr=8.35e-04]
Epoch 76 TRAIN:  59%|█████▉    | 80/136 [00:19<00:12,  4.37it/s, epoch=76, grad_norm=196.791, loss=282.8249, lr=8.36e-04]
Epoch 76 TRAIN:  59%|█████▉    | 80/136 [00:20<00:12,  4.37it/s, epoch=76, grad_norm=96.137, loss=250.3768, lr=8.36e-04] 
Epoch 76 TRAIN:  61%|██████    | 83/136 [00:20<00:11,  4.49it/s, epoch=76, grad_norm=96.137, loss=250.3768, lr=8.36e-04]
Epoch 76 TRAIN:  61%|██████    | 83/136 [00:20<00:11,  4.49it/s, epoch=76, grad_norm=203.185, loss=299.2044, lr=8.36e-04]
Epoch 76 TRAIN:  61%|██████    | 83/136 [00:20<00:11,  4.49it/s, epoch=76, grad_norm=200.057, loss=298.0204, lr=8.36e-04]
Epoch 76 TRAIN:  61%|██████    | 83/136 [00:20<00:11,  4.49it/s, epoch=76, grad_norm=145.590, loss=317.0533, lr=8.36e-04]
Epoch 76 TRAIN:  63%|██████▎   | 86/136 [00:20<00:11,  4.38it/s, epoch=76, grad_norm=145.590, loss=317.0533, lr=8.36e-04]
Epoch 76 TRAIN:  63%|██████▎   | 86/136 [00:21<00:11,  4.38it/s, epoch=76, grad_norm=236.669, loss=309.2429, lr=8.36e-04]
Epoch 76 TRAIN:  63%|██████▎   | 86/136 [00:21<00:11,  4.38it/s, epoch=76, grad_norm=84.752, loss=229.7052, lr=8.36e-04] 
Epoch 76 TRAIN:  63%|██████▎   | 86/136 [00:21<00:11,  4.38it/s, epoch=76, grad_norm=384.691, loss=261.4511, lr=8.36e-04]
Epoch 76 TRAIN:  65%|██████▌   | 89/136 [00:21<00:09,  4.74it/s, epoch=76, grad_norm=384.691, loss=261.4511, lr=8.36e-04]
Epoch 76 TRAIN:  65%|██████▌   | 89/136 [00:21<00:09,  4.74it/s, epoch=76, grad_norm=130.542, loss=288.6926, lr=8.36e-04]
Epoch 76 TRAIN:  65%|██████▌   | 89/136 [00:21<00:09,  4.74it/s, epoch=76, grad_norm=125.310, loss=300.6483, lr=8.36e-04]
Epoch 76 TRAIN:  65%|██████▌   | 89/136 [00:22<00:09,  4.74it/s, epoch=76, grad_norm=113.645, loss=265.8260, lr=8.36e-04]
Epoch 76 TRAIN:  68%|██████▊   | 92/136 [00:22<00:09,  4.54it/s, epoch=76, grad_norm=113.645, loss=265.8260, lr=8.36e-04]
Epoch 76 TRAIN:  68%|██████▊   | 92/136 [00:22<00:09,  4.54it/s, epoch=76, grad_norm=201.298, loss=269.3627, lr=8.36e-04]
Epoch 76 TRAIN:  68%|██████▊   | 92/136 [00:22<00:09,  4.54it/s, epoch=76, grad_norm=167.262, loss=262.5437, lr=8.36e-04]
Epoch 76 TRAIN:  68%|██████▊   | 92/136 [00:22<00:09,  4.54it/s, epoch=76, grad_norm=157.047, loss=220.8913, lr=8.37e-04]
Epoch 76 TRAIN:  70%|██████▉   | 95/136 [00:22<00:09,  4.35it/s, epoch=76, grad_norm=157.047, loss=220.8913, lr=8.37e-04]
Epoch 76 TRAIN:  70%|██████▉   | 95/136 [00:22<00:09,  4.35it/s, epoch=76, grad_norm=185.592, loss=198.4968, lr=8.37e-04]
Epoch 76 TRAIN:  70%|██████▉   | 95/136 [00:23<00:09,  4.35it/s, epoch=76, grad_norm=184.233, loss=259.8059, lr=8.37e-04]
Epoch 76 TRAIN:  70%|██████▉   | 95/136 [00:23<00:09,  4.35it/s, epoch=76, grad_norm=158.403, loss=278.5742, lr=8.37e-04]
Epoch 76 TRAIN:  72%|███████▏  | 98/136 [00:23<00:08,  4.45it/s, epoch=76, grad_norm=158.403, loss=278.5742, lr=8.37e-04]
Epoch 76 TRAIN:  72%|███████▏  | 98/136 [00:23<00:08,  4.45it/s, epoch=76, grad_norm=107.212, loss=270.5469, lr=8.37e-04]
Epoch 76 TRAIN:  72%|███████▏  | 98/136 [00:23<00:08,  4.45it/s, epoch=76, grad_norm=105.231, loss=202.3560, lr=8.37e-04]2026-04-16 20:31:29,638 DEBUG TRAIN | steps/sec 4.149| Batch 76/100 loss 202.356003 loss_att 158.776291 loss_ctc 304.041992 th_accuracy 0.386861 lr 8.3696e-04 grad_norm 105.230614 rank 0

Epoch 76 TRAIN:  72%|███████▏  | 98/136 [00:24<00:08,  4.45it/s, epoch=76, grad_norm=221.745, loss=247.0441, lr=8.37e-04]
Epoch 76 TRAIN:  74%|███████▍  | 101/136 [00:24<00:08,  4.27it/s, epoch=76, grad_norm=221.745, loss=247.0441, lr=8.37e-04]
Epoch 76 TRAIN:  74%|███████▍  | 101/136 [00:24<00:08,  4.27it/s, epoch=76, grad_norm=138.269, loss=199.1204, lr=8.37e-04]
Epoch 76 TRAIN:  74%|███████▍  | 101/136 [00:24<00:08,  4.27it/s, epoch=76, grad_norm=148.351, loss=216.2543, lr=8.37e-04]
Epoch 76 TRAIN:  74%|███████▍  | 101/136 [00:24<00:08,  4.27it/s, epoch=76, grad_norm=319.503, loss=304.1783, lr=8.37e-04]
Epoch 76 TRAIN:  76%|███████▋  | 104/136 [00:24<00:07,  4.36it/s, epoch=76, grad_norm=319.503, loss=304.1783, lr=8.37e-04]
Epoch 76 TRAIN:  76%|███████▋  | 104/136 [00:24<00:07,  4.36it/s, epoch=76, grad_norm=127.113, loss=242.4711, lr=8.37e-04]
Epoch 76 TRAIN:  76%|███████▋  | 104/136 [00:25<00:07,  4.36it/s, epoch=76, grad_norm=209.585, loss=274.1297, lr=8.37e-04]
Epoch 76 TRAIN:  76%|███████▋  | 104/136 [00:25<00:07,  4.36it/s, epoch=76, grad_norm=123.713, loss=184.5521, lr=8.38e-04]
Epoch 76 TRAIN:  79%|███████▊  | 107/136 [00:25<00:06,  4.60it/s, epoch=76, grad_norm=123.713, loss=184.5521, lr=8.38e-04]
Epoch 76 TRAIN:  79%|███████▊  | 107/136 [00:25<00:06,  4.60it/s, epoch=76, grad_norm=174.355, loss=271.1754, lr=8.38e-04]
Epoch 76 TRAIN:  79%|███████▊  | 107/136 [00:25<00:06,  4.60it/s, epoch=76, grad_norm=83.781, loss=210.0110, lr=8.38e-04] 
Epoch 76 TRAIN:  79%|███████▊  | 107/136 [00:26<00:06,  4.60it/s, epoch=76, grad_norm=96.430, loss=215.6446, lr=8.38e-04]
Epoch 76 TRAIN:  81%|████████  | 110/136 [00:26<00:05,  4.69it/s, epoch=76, grad_norm=96.430, loss=215.6446, lr=8.38e-04]
Epoch 76 TRAIN:  81%|████████  | 110/136 [00:26<00:05,  4.69it/s, epoch=76, grad_norm=136.563, loss=252.3979, lr=8.38e-04]
Epoch 76 TRAIN:  81%|████████  | 110/136 [00:26<00:05,  4.69it/s, epoch=76, grad_norm=170.115, loss=295.8238, lr=8.38e-04]
Epoch 76 TRAIN:  81%|████████  | 110/136 [00:26<00:05,  4.69it/s, epoch=76, grad_norm=150.941, loss=175.3284, lr=8.38e-04]
Epoch 76 TRAIN:  83%|████████▎ | 113/136 [00:26<00:05,  4.55it/s, epoch=76, grad_norm=150.941, loss=175.3284, lr=8.38e-04]
Epoch 76 TRAIN:  83%|████████▎ | 113/136 [00:26<00:05,  4.55it/s, epoch=76, grad_norm=232.727, loss=311.5076, lr=8.38e-04]
Epoch 76 TRAIN:  83%|████████▎ | 113/136 [00:27<00:05,  4.55it/s, epoch=76, grad_norm=132.264, loss=226.4480, lr=8.38e-04]
Epoch 76 TRAIN:  83%|████████▎ | 113/136 [00:27<00:05,  4.55it/s, epoch=76, grad_norm=242.589, loss=299.1718, lr=8.38e-04]
Epoch 76 TRAIN:  85%|████████▌ | 116/136 [00:27<00:04,  4.41it/s, epoch=76, grad_norm=242.589, loss=299.1718, lr=8.38e-04]
Epoch 76 TRAIN:  85%|████████▌ | 116/136 [00:27<00:04,  4.41it/s, epoch=76, grad_norm=184.682, loss=302.8554, lr=8.38e-04]
Epoch 76 TRAIN:  85%|████████▌ | 116/136 [00:27<00:04,  4.41it/s, epoch=76, grad_norm=117.089, loss=257.7522, lr=8.38e-04]
Epoch 76 TRAIN:  85%|████████▌ | 116/136 [00:28<00:04,  4.41it/s, epoch=76, grad_norm=186.085, loss=291.5439, lr=8.38e-04]
Epoch 76 TRAIN:  88%|████████▊ | 119/136 [00:28<00:03,  4.47it/s, epoch=76, grad_norm=186.085, loss=291.5439, lr=8.38e-04]
Epoch 76 TRAIN:  88%|████████▊ | 119/136 [00:28<00:03,  4.47it/s, epoch=76, grad_norm=162.853, loss=304.3510, lr=8.39e-04]
Epoch 76 TRAIN:  88%|████████▊ | 119/136 [00:28<00:03,  4.47it/s, epoch=76, grad_norm=175.479, loss=260.9711, lr=8.39e-04]
Epoch 76 TRAIN:  88%|████████▊ | 119/136 [00:28<00:03,  4.47it/s, epoch=76, grad_norm=296.173, loss=285.1467, lr=8.39e-04]
Epoch 76 TRAIN:  90%|████████▉ | 122/136 [00:28<00:02,  4.72it/s, epoch=76, grad_norm=296.173, loss=285.1467, lr=8.39e-04]
Epoch 76 TRAIN:  90%|████████▉ | 122/136 [00:28<00:02,  4.72it/s, epoch=76, grad_norm=146.102, loss=308.5484, lr=8.39e-04]
Epoch 76 TRAIN:  90%|████████▉ | 122/136 [00:29<00:02,  4.72it/s, epoch=76, grad_norm=134.150, loss=327.1700, lr=8.39e-04]
Epoch 76 TRAIN:  90%|████████▉ | 122/136 [00:29<00:02,  4.72it/s, epoch=76, grad_norm=147.149, loss=303.3499, lr=8.39e-04]
Epoch 76 TRAIN:  92%|█████████▏| 125/136 [00:29<00:02,  4.64it/s, epoch=76, grad_norm=147.149, loss=303.3499, lr=8.39e-04]
Epoch 76 TRAIN:  92%|█████████▏| 125/136 [00:29<00:02,  4.64it/s, epoch=76, grad_norm=151.760, loss=329.9574, lr=8.39e-04]
Epoch 76 TRAIN:  92%|█████████▏| 125/136 [00:29<00:02,  4.64it/s, epoch=76, grad_norm=194.173, loss=339.4767, lr=8.39e-04]
Epoch 76 TRAIN:  92%|█████████▏| 125/136 [00:29<00:02,  4.64it/s, epoch=76, grad_norm=215.141, loss=335.2916, lr=8.39e-04]
Epoch 76 TRAIN:  94%|█████████▍| 128/136 [00:29<00:01,  4.76it/s, epoch=76, grad_norm=215.141, loss=335.2916, lr=8.39e-04]
Epoch 76 TRAIN:  94%|█████████▍| 128/136 [00:30<00:01,  4.76it/s, epoch=76, grad_norm=258.274, loss=267.6367, lr=8.39e-04]
Epoch 76 TRAIN:  94%|█████████▍| 128/136 [00:30<00:01,  4.76it/s, epoch=76, grad_norm=200.803, loss=242.4519, lr=8.39e-04]
Epoch 76 TRAIN:  94%|█████████▍| 128/136 [00:30<00:01,  4.76it/s, epoch=76, grad_norm=455.061, loss=323.1320, lr=8.39e-04]
Epoch 76 TRAIN:  96%|█████████▋| 131/136 [00:30<00:00,  5.01it/s, epoch=76, grad_norm=455.061, loss=323.1320, lr=8.39e-04]
Epoch 76 TRAIN:  96%|█████████▋| 131/136 [00:30<00:00,  5.01it/s, epoch=76, grad_norm=320.651, loss=251.2024, lr=8.40e-04]
Epoch 76 TRAIN:  96%|█████████▋| 131/136 [00:30<00:00,  5.01it/s, epoch=76, grad_norm=152.600, loss=263.6827, lr=8.40e-04]
Epoch 76 TRAIN:  96%|█████████▋| 131/136 [00:30<00:00,  5.01it/s, epoch=76, grad_norm=156.163, loss=260.6577, lr=8.40e-04]
Epoch 76 TRAIN:  96%|█████████▋| 131/136 [00:31<00:00,  5.01it/s, epoch=76, grad_norm=141.943, loss=266.8913, lr=8.40e-04]
Epoch 76 TRAIN:  99%|█████████▉| 135/136 [00:31<00:00,  5.29it/s, epoch=76, grad_norm=141.943, loss=266.8913, lr=8.40e-04]
Epoch 76 TRAIN:  99%|█████████▉| 135/136 [00:31<00:00,  5.29it/s, epoch=76, grad_norm=260.336, loss=345.1635, lr=8.40e-04]
Epoch 76 TRAIN: 100%|██████████| 136/136 [00:31<00:00,  4.33it/s, epoch=76, grad_norm=260.336, loss=345.1635, lr=8.40e-04]

CV:   0%|          | 0/6 [00:00<?, ?it/s]/home/rookie/Documents/batuka/projects/wenet/wenet/wenet/utils/train_utils.py:689: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  amp_autocast(enabled=dtype is not None,

CV:  17%|█▋        | 1/6 [00:00<00:03,  1.32it/s]
                                                 
2026-04-16 20:31:37,983 INFO Epoch 76 Step 10498 CV info lr 8.3984e-04 cv_loss 467.9282503995028 rank 0 acc 0.37873639663060504
2026-04-16 20:31:37,985 INFO [Rank 0] Checkpoint: save to checkpoint exp/conformer_mongolian/epoch_76.pt
2026-04-16 20:31:38,099 INFO Epoch 77 Step 10498 TRAIN info lr 8.3984e-04 rank 0
2026-04-16 20:31:38,113 INFO using accumulate grad, new batch size is 1 times larger than before

Epoch 77 TRAIN:   0%|          | 0/136 [00:00<?, ?it/s]
Epoch 77 TRAIN:   0%|          | 0/136 [00:03<?, ?it/s, epoch=77, grad_norm=607.143, loss=426.5454, lr=8.40e-04]
Epoch 77 TRAIN:   1%|          | 1/136 [00:03<08:55,  3.97s/it, epoch=77, grad_norm=607.143, loss=426.5454, lr=8.40e-04]
Epoch 77 TRAIN:   1%|          | 1/136 [00:04<08:55,  3.97s/it, epoch=77, grad_norm=505.300, loss=384.4270, lr=8.40e-04]
Epoch 77 TRAIN:   1%|          | 1/136 [00:04<08:55,  3.97s/it, epoch=77, grad_norm=526.347, loss=426.7430, lr=8.40e-04]
Epoch 77 TRAIN:   1%|          | 1/136 [00:04<08:55,  3.97s/it, epoch=77, grad_norm=356.477, loss=319.0999, lr=8.40e-04]
Epoch 77 TRAIN:   1%|          | 1/136 [00:04<08:55,  3.97s/it, epoch=77, grad_norm=435.505, loss=309.9646, lr=8.40e-04]
Epoch 77 TRAIN:   1%|          | 1/136 [00:04<08:55,  3.97s/it, epoch=77, grad_norm=438.799, loss=391.3914, lr=8.40e-04]
Epoch 77 TRAIN:   1%|          | 1/136 [00:04<08:55,  3.97s/it, epoch=77, grad_norm=234.321, loss=273.8200, lr=8.40e-04]
Epoch 77 TRAIN:   1%|          | 1/136 [00:04<08:55,  3.97s/it, epoch=77, grad_norm=185.954, loss=237.5237, lr=8.40e-04]
Epoch 77 TRAIN:   6%|▌         | 8/136 [00:04<00:54,  2.35it/s, epoch=77, grad_norm=185.954, loss=237.5237, lr=8.40e-04]
Epoch 77 TRAIN:   6%|▌         | 8/136 [00:04<00:54,  2.35it/s, epoch=77, grad_norm=109.500, loss=266.3685, lr=8.41e-04]
Epoch 77 TRAIN:   6%|▌         | 8/136 [00:04<00:54,  2.35it/s, epoch=77, grad_norm=109.993, loss=288.5591, lr=8.41e-04]
Epoch 77 TRAIN:   6%|▌         | 8/136 [00:04<00:54,  2.35it/s, epoch=77, grad_norm=152.251, loss=272.1657, lr=8.41e-04]
Epoch 77 TRAIN:   6%|▌         | 8/136 [00:05<00:54,  2.35it/s, epoch=77, grad_norm=124.780, loss=294.3091, lr=8.41e-04]
Epoch 77 TRAIN:   6%|▌         | 8/136 [00:05<00:54,  2.35it/s, epoch=77, grad_norm=145.681, loss=331.0906, lr=8.41e-04]
Epoch 77 TRAIN:  10%|▉         | 13/136 [00:05<00:36,  3.33it/s, epoch=77, grad_norm=145.681, loss=331.0906, lr=8.41e-04]
Epoch 77 TRAIN:  10%|▉         | 13/136 [00:05<00:36,  3.33it/s, epoch=77, grad_norm=128.667, loss=301.8608, lr=8.41e-04]
Epoch 77 TRAIN:  10%|▉         | 13/136 [00:05<00:36,  3.33it/s, epoch=77, grad_norm=183.646, loss=256.8443, lr=8.41e-04]
Epoch 77 TRAIN:  10%|▉         | 13/136 [00:05<00:36,  3.33it/s, epoch=77, grad_norm=216.052, loss=257.6225, lr=8.41e-04]
Epoch 77 TRAIN:  10%|▉         | 13/136 [00:06<00:36,  3.33it/s, epoch=77, grad_norm=165.709, loss=292.7368, lr=8.41e-04]
Epoch 77 TRAIN:  12%|█▎        | 17/136 [00:06<00:32,  3.71it/s, epoch=77, grad_norm=165.709, loss=292.7368, lr=8.41e-04]
Epoch 77 TRAIN:  12%|█▎        | 17/136 [00:06<00:32,  3.71it/s, epoch=77, grad_norm=120.995, loss=292.6574, lr=8.41e-04]
Epoch 77 TRAIN:  12%|█▎        | 17/136 [00:06<00:32,  3.71it/s, epoch=77, grad_norm=171.489, loss=271.4786, lr=8.41e-04]
Epoch 77 TRAIN:  12%|█▎        | 17/136 [00:06<00:32,  3.71it/s, epoch=77, grad_norm=176.179, loss=253.0226, lr=8.41e-04]
Epoch 77 TRAIN:  12%|█▎        | 17/136 [00:06<00:32,  3.71it/s, epoch=77, grad_norm=100.118, loss=279.8724, lr=8.42e-04]
Epoch 77 TRAIN:  15%|█▌        | 21/136 [00:06<00:27,  4.12it/s, epoch=77, grad_norm=100.118, loss=279.8724, lr=8.42e-04]
Epoch 77 TRAIN:  15%|█▌        | 21/136 [00:07<00:27,  4.12it/s, epoch=77, grad_norm=85.047, loss=271.5516, lr=8.42e-04] 
Epoch 77 TRAIN:  15%|█▌        | 21/136 [00:07<00:27,  4.12it/s, epoch=77, grad_norm=116.357, loss=277.0895, lr=8.42e-04]
Epoch 77 TRAIN:  15%|█▌        | 21/136 [00:07<00:27,  4.12it/s, epoch=77, grad_norm=107.876, loss=269.3494, lr=8.42e-04]
Epoch 77 TRAIN:  18%|█▊        | 24/136 [00:07<00:26,  4.19it/s, epoch=77, grad_norm=107.876, loss=269.3494, lr=8.42e-04]
Epoch 77 TRAIN:  18%|█▊        | 24/136 [00:07<00:26,  4.19it/s, epoch=77, grad_norm=105.254, loss=269.1298, lr=8.42e-04]
Epoch 77 TRAIN:  18%|█▊        | 24/136 [00:07<00:26,  4.19it/s, epoch=77, grad_norm=130.350, loss=299.1017, lr=8.42e-04]
Epoch 77 TRAIN:  18%|█▊        | 24/136 [00:08<00:26,  4.19it/s, epoch=77, grad_norm=102.657, loss=294.3692, lr=8.42e-04]
Epoch 77 TRAIN:  20%|█▉        | 27/136 [00:08<00:24,  4.46it/s, epoch=77, grad_norm=102.657, loss=294.3692, lr=8.42e-04]
Epoch 77 TRAIN:  20%|█▉        | 27/136 [00:08<00:24,  4.46it/s, epoch=77, grad_norm=106.755, loss=283.1107, lr=8.42e-04]
Epoch 77 TRAIN:  20%|█▉        | 27/136 [00:08<00:24,  4.46it/s, epoch=77, grad_norm=86.079, loss=285.0311, lr=8.42e-04] 
Epoch 77 TRAIN:  20%|█▉        | 27/136 [00:08<00:24,  4.46it/s, epoch=77, grad_norm=106.608, loss=258.6017, lr=8.42e-04]
Epoch 77 TRAIN:  22%|██▏       | 30/136 [00:08<00:22,  4.70it/s, epoch=77, grad_norm=106.608, loss=258.6017, lr=8.42e-04]
Epoch 77 TRAIN:  22%|██▏       | 30/136 [00:08<00:22,  4.70it/s, epoch=77, grad_norm=92.188, loss=254.8200, lr=8.42e-04] 
Epoch 77 TRAIN:  22%|██▏       | 30/136 [00:09<00:22,  4.70it/s, epoch=77, grad_norm=132.324, loss=327.0333, lr=8.42e-04]
Epoch 77 TRAIN:  22%|██▏       | 30/136 [00:09<00:22,  4.70it/s, epoch=77, grad_norm=145.245, loss=211.9401, lr=8.42e-04]
Epoch 77 TRAIN:  24%|██▍       | 33/136 [00:09<00:21,  4.80it/s, epoch=77, grad_norm=145.245, loss=211.9401, lr=8.42e-04]
Epoch 77 TRAIN:  24%|██▍       | 33/136 [00:09<00:21,  4.80it/s, epoch=77, grad_norm=152.599, loss=294.7497, lr=8.43e-04]
Epoch 77 TRAIN:  24%|██▍       | 33/136 [00:09<00:21,  4.80it/s, epoch=77, grad_norm=98.094, loss=291.6729, lr=8.43e-04] 
Epoch 77 TRAIN:  24%|██▍       | 33/136 [00:09<00:21,  4.80it/s, epoch=77, grad_norm=146.313, loss=224.1110, lr=8.43e-04]
Epoch 77 TRAIN:  26%|██▋       | 36/136 [00:09<00:21,  4.70it/s, epoch=77, grad_norm=146.313, loss=224.1110, lr=8.43e-04]
Epoch 77 TRAIN:  26%|██▋       | 36/136 [00:10<00:21,  4.70it/s, epoch=77, grad_norm=141.606, loss=264.1909, lr=8.43e-04]
Epoch 77 TRAIN:  26%|██▋       | 36/136 [00:10<00:21,  4.70it/s, epoch=77, grad_norm=84.685, loss=261.0021, lr=8.43e-04] 
Epoch 77 TRAIN:  26%|██▋       | 36/136 [00:10<00:21,  4.70it/s, epoch=77, grad_norm=83.211, loss=257.0249, lr=8.43e-04]
Epoch 77 TRAIN:  29%|██▊       | 39/136 [00:10<00:19,  4.85it/s, epoch=77, grad_norm=83.211, loss=257.0249, lr=8.43e-04]
Epoch 77 TRAIN:  29%|██▊       | 39/136 [00:10<00:19,  4.85it/s, epoch=77, grad_norm=85.132, loss=279.7470, lr=8.43e-04]
Epoch 77 TRAIN:  29%|██▊       | 39/136 [00:11<00:19,  4.85it/s, epoch=77, grad_norm=166.612, loss=316.7661, lr=8.43e-04]
Epoch 77 TRAIN:  29%|██▊       | 39/136 [00:11<00:19,  4.85it/s, epoch=77, grad_norm=82.446, loss=245.3575, lr=8.43e-04] 
Epoch 77 TRAIN:  31%|███       | 42/136 [00:11<00:19,  4.77it/s, epoch=77, grad_norm=82.446, loss=245.3575, lr=8.43e-04]
Epoch 77 TRAIN:  31%|███       | 42/136 [00:11<00:19,  4.77it/s, epoch=77, grad_norm=97.738, loss=287.5221, lr=8.43e-04]
Epoch 77 TRAIN:  31%|███       | 42/136 [00:11<00:19,  4.77it/s, epoch=77, grad_norm=84.211, loss=258.9474, lr=8.43e-04]
Epoch 77 TRAIN:  31%|███       | 42/136 [00:11<00:19,  4.77it/s, epoch=77, grad_norm=108.981, loss=287.5339, lr=8.43e-04]
Epoch 77 TRAIN:  33%|███▎      | 45/136 [00:11<00:19,  4.60it/s, epoch=77, grad_norm=108.981, loss=287.5339, lr=8.43e-04]
Epoch 77 TRAIN:  33%|███▎      | 45/136 [00:12<00:19,  4.60it/s, epoch=77, grad_norm=102.938, loss=225.6076, lr=8.44e-04]
Epoch 77 TRAIN:  33%|███▎      | 45/136 [00:12<00:19,  4.60it/s, epoch=77, grad_norm=106.134, loss=277.1163, lr=8.44e-04]
Epoch 77 TRAIN:  33%|███▎      | 45/136 [00:12<00:19,  4.60it/s, epoch=77, grad_norm=121.160, loss=284.4724, lr=8.44e-04]
Epoch 77 TRAIN:  35%|███▌      | 48/136 [00:12<00:18,  4.74it/s, epoch=77, grad_norm=121.160, loss=284.4724, lr=8.44e-04]
Epoch 77 TRAIN:  35%|███▌      | 48/136 [00:12<00:18,  4.74it/s, epoch=77, grad_norm=99.895, loss=284.5771, lr=8.44e-04] 
Epoch 77 TRAIN:  35%|███▌      | 48/136 [00:12<00:18,  4.74it/s, epoch=77, grad_norm=115.297, loss=275.3736, lr=8.44e-04]
Epoch 77 TRAIN:  35%|███▌      | 48/136 [00:13<00:18,  4.74it/s, epoch=77, grad_norm=94.239, loss=262.8846, lr=8.44e-04] 
Epoch 77 TRAIN:  38%|███▊      | 51/136 [00:13<00:18,  4.62it/s, epoch=77, grad_norm=94.239, loss=262.8846, lr=8.44e-04]
Epoch 77 TRAIN:  38%|███▊      | 51/136 [00:13<00:18,  4.62it/s, epoch=77, grad_norm=121.462, loss=299.7266, lr=8.44e-04]
Epoch 77 TRAIN:  38%|███▊      | 51/136 [00:13<00:18,  4.62it/s, epoch=77, grad_norm=116.130, loss=244.1486, lr=8.44e-04]
Epoch 77 TRAIN:  38%|███▊      | 51/136 [00:13<00:18,  4.62it/s, epoch=77, grad_norm=150.259, loss=326.0742, lr=8.44e-04]
Epoch 77 TRAIN:  40%|███▉      | 54/136 [00:13<00:17,  4.59it/s, epoch=77, grad_norm=150.259, loss=326.0742, lr=8.44e-04]
Epoch 77 TRAIN:  40%|███▉      | 54/136 [00:14<00:17,  4.59it/s, epoch=77, grad_norm=118.907, loss=269.4731, lr=8.44e-04]
Epoch 77 TRAIN:  40%|███▉      | 54/136 [00:14<00:17,  4.59it/s, epoch=77, grad_norm=93.365, loss=272.6340, lr=8.44e-04] 
Epoch 77 TRAIN:  40%|███▉      | 54/136 [00:14<00:17,  4.59it/s, epoch=77, grad_norm=92.755, loss=262.8342, lr=8.44e-04]
Epoch 77 TRAIN:  42%|████▏     | 57/136 [00:14<00:17,  4.52it/s, epoch=77, grad_norm=92.755, loss=262.8342, lr=8.44e-04]
Epoch 77 TRAIN:  42%|████▏     | 57/136 [00:14<00:17,  4.52it/s, epoch=77, grad_norm=100.306, loss=243.4629, lr=8.44e-04]
Epoch 77 TRAIN:  42%|████▏     | 57/136 [00:14<00:17,  4.52it/s, epoch=77, grad_norm=107.197, loss=278.4400, lr=8.45e-04]
Epoch 77 TRAIN:  42%|████▏     | 57/136 [00:15<00:17,  4.52it/s, epoch=77, grad_norm=96.187, loss=236.9433, lr=8.45e-04] 
Epoch 77 TRAIN:  44%|████▍     | 60/136 [00:15<00:16,  4.55it/s, epoch=77, grad_norm=96.187, loss=236.9433, lr=8.45e-04]
Epoch 77 TRAIN:  44%|████▍     | 60/136 [00:15<00:16,  4.55it/s, epoch=77, grad_norm=141.963, loss=291.4223, lr=8.45e-04]
Epoch 77 TRAIN:  44%|████▍     | 60/136 [00:15<00:16,  4.55it/s, epoch=77, grad_norm=95.829, loss=244.0837, lr=8.45e-04] 
Epoch 77 TRAIN:  44%|████▍     | 60/136 [00:15<00:16,  4.55it/s, epoch=77, grad_norm=252.405, loss=327.4009, lr=8.45e-04]
Epoch 77 TRAIN:  46%|████▋     | 63/136 [00:15<00:15,  4.59it/s, epoch=77, grad_norm=252.405, loss=327.4009, lr=8.45e-04]
Epoch 77 TRAIN:  46%|████▋     | 63/136 [00:16<00:15,  4.59it/s, epoch=77, grad_norm=312.080, loss=374.1810, lr=8.45e-04]
Epoch 77 TRAIN:  46%|████▋     | 63/136 [00:16<00:15,  4.59it/s, epoch=77, grad_norm=216.534, loss=305.8721, lr=8.45e-04]
Epoch 77 TRAIN:  46%|████▋     | 63/136 [00:16<00:15,  4.59it/s, epoch=77, grad_norm=123.131, loss=275.6150, lr=8.45e-04]
Epoch 77 TRAIN:  49%|████▊     | 66/136 [00:16<00:15,  4.62it/s, epoch=77, grad_norm=123.131, loss=275.6150, lr=8.45e-04]
Epoch 77 TRAIN:  49%|████▊     | 66/136 [00:16<00:15,  4.62it/s, epoch=77, grad_norm=112.797, loss=297.1549, lr=8.45e-04]
Epoch 77 TRAIN:  49%|████▊     | 66/136 [00:16<00:15,  4.62it/s, epoch=77, grad_norm=136.619, loss=304.3560, lr=8.45e-04]
Epoch 77 TRAIN:  49%|████▊     | 66/136 [00:17<00:15,  4.62it/s, epoch=77, grad_norm=109.797, loss=294.7405, lr=8.45e-04]
Epoch 77 TRAIN:  51%|█████     | 69/136 [00:17<00:14,  4.54it/s, epoch=77, grad_norm=109.797, loss=294.7405, lr=8.45e-04]
Epoch 77 TRAIN:  51%|█████     | 69/136 [00:17<00:14,  4.54it/s, epoch=77, grad_norm=220.121, loss=305.8215, lr=8.45e-04]
Epoch 77 TRAIN:  51%|█████     | 69/136 [00:17<00:14,  4.54it/s, epoch=77, grad_norm=116.069, loss=295.5757, lr=8.46e-04]
Epoch 77 TRAIN:  51%|█████     | 69/136 [00:17<00:14,  4.54it/s, epoch=77, grad_norm=124.131, loss=251.0120, lr=8.46e-04]
Epoch 77 TRAIN:  53%|█████▎    | 72/136 [00:17<00:14,  4.52it/s, epoch=77, grad_norm=124.131, loss=251.0120, lr=8.46e-04]
Epoch 77 TRAIN:  53%|█████▎    | 72/136 [00:18<00:14,  4.52it/s, epoch=77, grad_norm=124.682, loss=292.5354, lr=8.46e-04]
Epoch 77 TRAIN:  53%|█████▎    | 72/136 [00:18<00:14,  4.52it/s, epoch=77, grad_norm=126.969, loss=300.4166, lr=8.46e-04]
Epoch 77 TRAIN:  53%|█████▎    | 72/136 [00:18<00:14,  4.52it/s, epoch=77, grad_norm=181.424, loss=256.4533, lr=8.46e-04]
Epoch 77 TRAIN:  55%|█████▌    | 75/136 [00:18<00:13,  4.55it/s, epoch=77, grad_norm=181.424, loss=256.4533, lr=8.46e-04]
Epoch 77 TRAIN:  55%|█████▌    | 75/136 [00:18<00:13,  4.55it/s, epoch=77, grad_norm=159.402, loss=219.4428, lr=8.46e-04]
Epoch 77 TRAIN:  55%|█████▌    | 75/136 [00:18<00:13,  4.55it/s, epoch=77, grad_norm=387.516, loss=300.1254, lr=8.46e-04]
Epoch 77 TRAIN:  55%|█████▌    | 75/136 [00:19<00:13,  4.55it/s, epoch=77, grad_norm=173.198, loss=244.9906, lr=8.46e-04]
Epoch 77 TRAIN:  57%|█████▋    | 78/136 [00:19<00:13,  4.46it/s, epoch=77, grad_norm=173.198, loss=244.9906, lr=8.46e-04]
Epoch 77 TRAIN:  57%|█████▋    | 78/136 [00:19<00:13,  4.46it/s, epoch=77, grad_norm=228.169, loss=288.5381, lr=8.46e-04]
Epoch 77 TRAIN:  57%|█████▋    | 78/136 [00:19<00:13,  4.46it/s, epoch=77, grad_norm=217.108, loss=283.9421, lr=8.46e-04]
Epoch 77 TRAIN:  57%|█████▋    | 78/136 [00:19<00:13,  4.46it/s, epoch=77, grad_norm=131.641, loss=281.6925, lr=8.46e-04]
Epoch 77 TRAIN:  60%|█████▉    | 81/136 [00:19<00:12,  4.40it/s, epoch=77, grad_norm=131.641, loss=281.6925, lr=8.46e-04]
Epoch 77 TRAIN:  60%|█████▉    | 81/136 [00:20<00:12,  4.40it/s, epoch=77, grad_norm=162.552, loss=224.6112, lr=8.46e-04]
Epoch 77 TRAIN:  60%|█████▉    | 81/136 [00:20<00:12,  4.40it/s, epoch=77, grad_norm=268.898, loss=265.1408, lr=8.46e-04]
Epoch 77 TRAIN:  60%|█████▉    | 81/136 [00:20<00:12,  4.40it/s, epoch=77, grad_norm=181.108, loss=232.3207, lr=8.47e-04]
Epoch 77 TRAIN:  62%|██████▏   | 84/136 [00:20<00:11,  4.43it/s, epoch=77, grad_norm=181.108, loss=232.3207, lr=8.47e-04]
Epoch 77 TRAIN:  62%|██████▏   | 84/136 [00:20<00:11,  4.43it/s, epoch=77, grad_norm=146.821, loss=295.5763, lr=8.47e-04]
Epoch 77 TRAIN:  62%|██████▏   | 84/136 [00:21<00:11,  4.43it/s, epoch=77, grad_norm=208.287, loss=263.4089, lr=8.47e-04]
Epoch 77 TRAIN:  62%|██████▏   | 84/136 [00:21<00:11,  4.43it/s, epoch=77, grad_norm=137.837, loss=247.2244, lr=8.47e-04]
Epoch 77 TRAIN:  64%|██████▍   | 87/136 [00:21<00:11,  4.30it/s, epoch=77, grad_norm=137.837, loss=247.2244, lr=8.47e-04]
Epoch 77 TRAIN:  64%|██████▍   | 87/136 [00:21<00:11,  4.30it/s, epoch=77, grad_norm=259.808, loss=310.8732, lr=8.47e-04]
Epoch 77 TRAIN:  64%|██████▍   | 87/136 [00:21<00:11,  4.30it/s, epoch=77, grad_norm=164.332, loss=243.0011, lr=8.47e-04]
Epoch 77 TRAIN:  64%|██████▍   | 87/136 [00:21<00:11,  4.30it/s, epoch=77, grad_norm=139.483, loss=282.8697, lr=8.47e-04]
Epoch 77 TRAIN:  66%|██████▌   | 90/136 [00:21<00:10,  4.38it/s, epoch=77, grad_norm=139.483, loss=282.8697, lr=8.47e-04]
Epoch 77 TRAIN:  66%|██████▌   | 90/136 [00:22<00:10,  4.38it/s, epoch=77, grad_norm=220.276, loss=225.7661, lr=8.47e-04]
Epoch 77 TRAIN:  66%|██████▌   | 90/136 [00:22<00:10,  4.38it/s, epoch=77, grad_norm=112.390, loss=319.0585, lr=8.47e-04]
Epoch 77 TRAIN:  66%|██████▌   | 90/136 [00:22<00:10,  4.38it/s, epoch=77, grad_norm=155.923, loss=260.4741, lr=8.47e-04]
Epoch 77 TRAIN:  68%|██████▊   | 93/136 [00:22<00:09,  4.47it/s, epoch=77, grad_norm=155.923, loss=260.4741, lr=8.47e-04]
Epoch 77 TRAIN:  68%|██████▊   | 93/136 [00:22<00:09,  4.47it/s, epoch=77, grad_norm=203.258, loss=323.2488, lr=8.47e-04]
Epoch 77 TRAIN:  68%|██████▊   | 93/136 [00:22<00:09,  4.47it/s, epoch=77, grad_norm=125.900, loss=212.7438, lr=8.47e-04]
Epoch 77 TRAIN:  68%|██████▊   | 93/136 [00:23<00:09,  4.47it/s, epoch=77, grad_norm=198.243, loss=315.3847, lr=8.48e-04]
Epoch 77 TRAIN:  71%|███████   | 96/136 [00:23<00:08,  4.65it/s, epoch=77, grad_norm=198.243, loss=315.3847, lr=8.48e-04]
Epoch 77 TRAIN:  71%|███████   | 96/136 [00:23<00:08,  4.65it/s, epoch=77, grad_norm=113.916, loss=263.9686, lr=8.48e-04]
Epoch 77 TRAIN:  71%|███████   | 96/136 [00:23<00:08,  4.65it/s, epoch=77, grad_norm=94.955, loss=270.2365, lr=8.48e-04] 
Epoch 77 TRAIN:  71%|███████   | 96/136 [00:23<00:08,  4.65it/s, epoch=77, grad_norm=251.496, loss=244.9620, lr=8.48e-04]
Epoch 77 TRAIN:  73%|███████▎  | 99/136 [00:23<00:07,  4.77it/s, epoch=77, grad_norm=251.496, loss=244.9620, lr=8.48e-04]
Epoch 77 TRAIN:  73%|███████▎  | 99/136 [00:23<00:07,  4.77it/s, epoch=77, grad_norm=172.043, loss=333.3226, lr=8.48e-04]2026-04-16 20:32:02,102 DEBUG TRAIN | steps/sec 4.189| Batch 77/100 loss 333.322571 loss_att 265.849731 loss_ctc 490.759216 th_accuracy 0.391919 lr 8.4784e-04 grad_norm 172.042679 rank 0

Epoch 77 TRAIN:  73%|███████▎  | 99/136 [00:24<00:07,  4.77it/s, epoch=77, grad_norm=145.356, loss=266.3464, lr=8.48e-04]
Epoch 77 TRAIN:  73%|███████▎  | 99/136 [00:24<00:07,  4.77it/s, epoch=77, grad_norm=247.499, loss=260.7488, lr=8.48e-04]
Epoch 77 TRAIN:  75%|███████▌  | 102/136 [00:24<00:07,  4.65it/s, epoch=77, grad_norm=247.499, loss=260.7488, lr=8.48e-04]
Epoch 77 TRAIN:  75%|███████▌  | 102/136 [00:24<00:07,  4.65it/s, epoch=77, grad_norm=146.738, loss=275.9805, lr=8.48e-04]
Epoch 77 TRAIN:  75%|███████▌  | 102/136 [00:24<00:07,  4.65it/s, epoch=77, grad_norm=129.409, loss=219.2842, lr=8.48e-04]
Epoch 77 TRAIN:  75%|███████▌  | 102/136 [00:25<00:07,  4.65it/s, epoch=77, grad_norm=147.593, loss=236.7753, lr=8.48e-04]
Epoch 77 TRAIN:  77%|███████▋  | 105/136 [00:25<00:06,  4.56it/s, epoch=77, grad_norm=147.593, loss=236.7753, lr=8.48e-04]
Epoch 77 TRAIN:  77%|███████▋  | 105/136 [00:25<00:06,  4.56it/s, epoch=77, grad_norm=109.784, loss=233.6064, lr=8.48e-04]
Epoch 77 TRAIN:  77%|███████▋  | 105/136 [00:25<00:06,  4.56it/s, epoch=77, grad_norm=228.868, loss=248.1489, lr=8.48e-04]
Epoch 77 TRAIN:  77%|███████▋  | 105/136 [00:25<00:06,  4.56it/s, epoch=77, grad_norm=153.828, loss=216.9392, lr=8.48e-04]
Epoch 77 TRAIN:  79%|███████▉  | 108/136 [00:25<00:06,  4.39it/s, epoch=77, grad_norm=153.828, loss=216.9392, lr=8.48e-04]
Epoch 77 TRAIN:  79%|███████▉  | 108/136 [00:25<00:06,  4.39it/s, epoch=77, grad_norm=265.599, loss=340.1004, lr=8.49e-04]
Epoch 77 TRAIN:  79%|███████▉  | 108/136 [00:26<00:06,  4.39it/s, epoch=77, grad_norm=599.896, loss=301.6996, lr=8.49e-04]
Epoch 77 TRAIN:  79%|███████▉  | 108/136 [00:26<00:06,  4.39it/s, epoch=77, grad_norm=121.855, loss=213.7140, lr=8.49e-04]
Epoch 77 TRAIN:  79%|███████▉  | 108/136 [00:26<00:06,  4.39it/s, epoch=77, grad_norm=101.485, loss=237.7352, lr=8.49e-04]
Epoch 77 TRAIN:  82%|████████▏ | 112/136 [00:26<00:04,  4.83it/s, epoch=77, grad_norm=101.485, loss=237.7352, lr=8.49e-04]
Epoch 77 TRAIN:  82%|████████▏ | 112/136 [00:26<00:04,  4.83it/s, epoch=77, grad_norm=115.909, loss=278.8480, lr=8.49e-04]
Epoch 77 TRAIN:  82%|████████▏ | 112/136 [00:27<00:04,  4.83it/s, epoch=77, grad_norm=222.358, loss=270.2093, lr=8.49e-04]
Epoch 77 TRAIN:  82%|████████▏ | 112/136 [00:27<00:04,  4.83it/s, epoch=77, grad_norm=267.238, loss=318.7684, lr=8.49e-04]
Epoch 77 TRAIN:  85%|████████▍ | 115/136 [00:27<00:04,  4.67it/s, epoch=77, grad_norm=267.238, loss=318.7684, lr=8.49e-04]
Epoch 77 TRAIN:  85%|████████▍ | 115/136 [00:27<00:04,  4.67it/s, epoch=77, grad_norm=160.366, loss=293.2062, lr=8.49e-04]
Epoch 77 TRAIN:  85%|████████▍ | 115/136 [00:27<00:04,  4.67it/s, epoch=77, grad_norm=263.883, loss=301.7368, lr=8.49e-04]
Epoch 77 TRAIN:  85%|████████▍ | 115/136 [00:27<00:04,  4.67it/s, epoch=77, grad_norm=393.781, loss=258.0051, lr=8.49e-04]
Epoch 77 TRAIN:  87%|████████▋ | 118/136 [00:27<00:03,  4.51it/s, epoch=77, grad_norm=393.781, loss=258.0051, lr=8.49e-04]
Epoch 77 TRAIN:  87%|████████▋ | 118/136 [00:28<00:03,  4.51it/s, epoch=77, grad_norm=150.987, loss=209.1045, lr=8.49e-04]
Epoch 77 TRAIN:  87%|████████▋ | 118/136 [00:28<00:03,  4.51it/s, epoch=77, grad_norm=279.636, loss=260.8538, lr=8.49e-04]
Epoch 77 TRAIN:  87%|████████▋ | 118/136 [00:28<00:03,  4.51it/s, epoch=77, grad_norm=249.003, loss=322.7779, lr=8.50e-04]
Epoch 77 TRAIN:  89%|████████▉ | 121/136 [00:28<00:03,  4.45it/s, epoch=77, grad_norm=249.003, loss=322.7779, lr=8.50e-04]
Epoch 77 TRAIN:  89%|████████▉ | 121/136 [00:28<00:03,  4.45it/s, epoch=77, grad_norm=318.901, loss=348.8504, lr=8.50e-04]
Epoch 77 TRAIN:  89%|████████▉ | 121/136 [00:29<00:03,  4.45it/s, epoch=77, grad_norm=230.288, loss=239.3627, lr=8.50e-04]
Epoch 77 TRAIN:  89%|████████▉ | 121/136 [00:29<00:03,  4.45it/s, epoch=77, grad_norm=296.508, loss=364.6641, lr=8.50e-04]
Epoch 77 TRAIN:  91%|█████████ | 124/136 [00:29<00:02,  4.60it/s, epoch=77, grad_norm=296.508, loss=364.6641, lr=8.50e-04]
Epoch 77 TRAIN:  91%|█████████ | 124/136 [00:29<00:02,  4.60it/s, epoch=77, grad_norm=107.082, loss=254.6694, lr=8.50e-04]
Epoch 77 TRAIN:  91%|█████████ | 124/136 [00:29<00:02,  4.60it/s, epoch=77, grad_norm=108.629, loss=206.5151, lr=8.50e-04]
Epoch 77 TRAIN:  91%|█████████ | 124/136 [00:29<00:02,  4.60it/s, epoch=77, grad_norm=284.874, loss=378.8885, lr=8.50e-04]
Epoch 77 TRAIN:  93%|█████████▎| 127/136 [00:29<00:01,  4.92it/s, epoch=77, grad_norm=284.874, loss=378.8885, lr=8.50e-04]
Epoch 77 TRAIN:  93%|█████████▎| 127/136 [00:29<00:01,  4.92it/s, epoch=77, grad_norm=169.624, loss=266.9345, lr=8.50e-04]
Epoch 77 TRAIN:  93%|█████████▎| 127/136 [00:30<00:01,  4.92it/s, epoch=77, grad_norm=131.355, loss=259.5416, lr=8.50e-04]
Epoch 77 TRAIN:  93%|█████████▎| 127/136 [00:30<00:01,  4.92it/s, epoch=77, grad_norm=111.622, loss=238.2781, lr=8.50e-04]
Epoch 77 TRAIN:  96%|█████████▌| 130/136 [00:30<00:01,  4.77it/s, epoch=77, grad_norm=111.622, loss=238.2781, lr=8.50e-04]
Epoch 77 TRAIN:  96%|█████████▌| 130/136 [00:30<00:01,  4.77it/s, epoch=77, grad_norm=168.214, loss=303.4670, lr=8.50e-04]
Epoch 77 TRAIN:  96%|█████████▌| 130/136 [00:30<00:01,  4.77it/s, epoch=77, grad_norm=156.213, loss=268.5520, lr=8.50e-04]
Epoch 77 TRAIN:  96%|█████████▌| 130/136 [00:31<00:01,  4.77it/s, epoch=77, grad_norm=99.128, loss=229.9366, lr=8.50e-04] 
Epoch 77 TRAIN:  98%|█████████▊| 133/136 [00:31<00:00,  4.70it/s, epoch=77, grad_norm=99.128, loss=229.9366, lr=8.50e-04]
Epoch 77 TRAIN:  98%|█████████▊| 133/136 [00:31<00:00,  4.70it/s, epoch=77, grad_norm=266.223, loss=217.3499, lr=8.51e-04]
Epoch 77 TRAIN:  98%|█████████▊| 133/136 [00:31<00:00,  4.70it/s, epoch=77, grad_norm=313.740, loss=336.2639, lr=8.51e-04]
Epoch 77 TRAIN:  98%|█████████▊| 133/136 [00:31<00:00,  4.70it/s, epoch=77, grad_norm=333.179, loss=355.8607, lr=8.51e-04]
Epoch 77 TRAIN: 100%|██████████| 136/136 [00:31<00:00,  4.32it/s, epoch=77, grad_norm=333.179, loss=355.8607, lr=8.51e-04]

CV:   0%|          | 0/6 [00:00<?, ?it/s]/home/rookie/Documents/batuka/projects/wenet/wenet/wenet/utils/train_utils.py:689: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  amp_autocast(enabled=dtype is not None,

CV:  17%|█▋        | 1/6 [00:00<00:03,  1.33it/s]
                                                 
2026-04-16 20:32:10,474 INFO Epoch 77 Step 10634 CV info lr 8.5072e-04 cv_loss 460.8889437588778 rank 0 acc 0.35050228238105774
2026-04-16 20:32:10,475 INFO [Rank 0] Checkpoint: save to checkpoint exp/conformer_mongolian/epoch_77.pt
2026-04-16 20:32:10,586 INFO Epoch 78 Step 10634 TRAIN info lr 8.5072e-04 rank 0
2026-04-16 20:32:10,605 INFO using accumulate grad, new batch size is 1 times larger than before

Epoch 78 TRAIN:   0%|          | 0/136 [00:00<?, ?it/s]
Epoch 78 TRAIN:   0%|          | 0/136 [00:03<?, ?it/s, epoch=78, grad_norm=348.450, loss=306.3737, lr=8.51e-04]
Epoch 78 TRAIN:   1%|          | 1/136 [00:03<08:01,  3.57s/it, epoch=78, grad_norm=348.450, loss=306.3737, lr=8.51e-04]
Epoch 78 TRAIN:   1%|          | 1/136 [00:03<08:01,  3.57s/it, epoch=78, grad_norm=457.315, loss=402.0792, lr=8.51e-04]
Epoch 78 TRAIN:   1%|          | 1/136 [00:04<08:01,  3.57s/it, epoch=78, grad_norm=411.465, loss=302.0615, lr=8.51e-04]
Epoch 78 TRAIN:   1%|          | 1/136 [00:04<08:01,  3.57s/it, epoch=78, grad_norm=224.681, loss=218.4229, lr=8.51e-04]
Epoch 78 TRAIN:   3%|▎         | 4/136 [00:04<01:48,  1.21it/s, epoch=78, grad_norm=224.681, loss=218.4229, lr=8.51e-04]
Epoch 78 TRAIN:   3%|▎         | 4/136 [00:04<01:48,  1.21it/s, epoch=78, grad_norm=303.129, loss=311.3959, lr=8.51e-04]
Epoch 78 TRAIN:   3%|▎         | 4/136 [00:04<01:48,  1.21it/s, epoch=78, grad_norm=229.734, loss=259.0574, lr=8.51e-04]
Epoch 78 TRAIN:   3%|▎         | 4/136 [00:04<01:48,  1.21it/s, epoch=78, grad_norm=255.243, loss=287.3297, lr=8.51e-04]
Epoch 78 TRAIN:   3%|▎         | 4/136 [00:04<01:48,  1.21it/s, epoch=78, grad_norm=117.664, loss=191.7345, lr=8.51e-04]
Epoch 78 TRAIN:   3%|▎         | 4/136 [00:04<01:48,  1.21it/s, epoch=78, grad_norm=85.319, loss=246.2094, lr=8.51e-04] 
Epoch 78 TRAIN:   7%|▋         | 9/136 [00:04<00:45,  2.76it/s, epoch=78, grad_norm=85.319, loss=246.2094, lr=8.51e-04]
Epoch 78 TRAIN:   7%|▋         | 9/136 [00:04<00:45,  2.76it/s, epoch=78, grad_norm=194.779, loss=293.9614, lr=8.52e-04]
Epoch 78 TRAIN:   7%|▋         | 9/136 [00:05<00:45,  2.76it/s, epoch=78, grad_norm=114.561, loss=249.2351, lr=8.52e-04]
Epoch 78 TRAIN:   7%|▋         | 9/136 [00:05<00:45,  2.76it/s, epoch=78, grad_norm=152.015, loss=296.6456, lr=8.52e-04]
Epoch 78 TRAIN:   7%|▋         | 9/136 [00:05<00:45,  2.76it/s, epoch=78, grad_norm=99.977, loss=268.1929, lr=8.52e-04] 
Epoch 78 TRAIN:  10%|▉         | 13/136 [00:05<00:33,  3.64it/s, epoch=78, grad_norm=99.977, loss=268.1929, lr=8.52e-04]
Epoch 78 TRAIN:  10%|▉         | 13/136 [00:05<00:33,  3.64it/s, epoch=78, grad_norm=109.289, loss=291.4615, lr=8.52e-04]
Epoch 78 TRAIN:  10%|▉         | 13/136 [00:05<00:33,  3.64it/s, epoch=78, grad_norm=87.833, loss=299.7756, lr=8.52e-04] 
Epoch 78 TRAIN:  10%|▉         | 13/136 [00:05<00:33,  3.64it/s, epoch=78, grad_norm=115.594, loss=291.4371, lr=8.52e-04]
Epoch 78 TRAIN:  12%|█▏        | 16/136 [00:05<00:28,  4.14it/s, epoch=78, grad_norm=115.594, loss=291.4371, lr=8.52e-04]
Epoch 78 TRAIN:  12%|█▏        | 16/136 [00:06<00:28,  4.14it/s, epoch=78, grad_norm=145.855, loss=294.0356, lr=8.52e-04]
Epoch 78 TRAIN:  12%|█▏        | 16/136 [00:06<00:28,  4.14it/s, epoch=78, grad_norm=121.055, loss=273.7451, lr=8.52e-04]
Epoch 78 TRAIN:  12%|█▏        | 16/136 [00:06<00:28,  4.14it/s, epoch=78, grad_norm=117.039, loss=280.8416, lr=8.52e-04]
Epoch 78 TRAIN:  14%|█▍        | 19/136 [00:06<00:26,  4.39it/s, epoch=78, grad_norm=117.039, loss=280.8416, lr=8.52e-04]
Epoch 78 TRAIN:  14%|█▍        | 19/136 [00:06<00:26,  4.39it/s, epoch=78, grad_norm=146.064, loss=281.6330, lr=8.52e-04]
Epoch 78 TRAIN:  14%|█▍        | 19/136 [00:06<00:26,  4.39it/s, epoch=78, grad_norm=137.465, loss=308.3983, lr=8.52e-04]
Epoch 78 TRAIN:  14%|█▍        | 19/136 [00:07<00:26,  4.39it/s, epoch=78, grad_norm=118.020, loss=281.5584, lr=8.52e-04]
Epoch 78 TRAIN:  16%|█▌        | 22/136 [00:07<00:25,  4.55it/s, epoch=78, grad_norm=118.020, loss=281.5584, lr=8.52e-04]
Epoch 78 TRAIN:  16%|█▌        | 22/136 [00:07<00:25,  4.55it/s, epoch=78, grad_norm=106.640, loss=317.6833, lr=8.53e-04]
Epoch 78 TRAIN:  16%|█▌        | 22/136 [00:07<00:25,  4.55it/s, epoch=78, grad_norm=151.972, loss=302.6745, lr=8.53e-04]
Epoch 78 TRAIN:  16%|█▌        | 22/136 [00:07<00:25,  4.55it/s, epoch=78, grad_norm=144.898, loss=240.7831, lr=8.53e-04]
Epoch 78 TRAIN:  18%|█▊        | 25/136 [00:07<00:23,  4.64it/s, epoch=78, grad_norm=144.898, loss=240.7831, lr=8.53e-04]
Epoch 78 TRAIN:  18%|█▊        | 25/136 [00:07<00:23,  4.64it/s, epoch=78, grad_norm=80.465, loss=268.2938, lr=8.53e-04] 
Epoch 78 TRAIN:  18%|█▊        | 25/136 [00:08<00:23,  4.64it/s, epoch=78, grad_norm=137.540, loss=332.6620, lr=8.53e-04]
Epoch 78 TRAIN:  18%|█▊        | 25/136 [00:08<00:23,  4.64it/s, epoch=78, grad_norm=116.579, loss=244.4201, lr=8.53e-04]
Epoch 78 TRAIN:  21%|██        | 28/136 [00:08<00:23,  4.54it/s, epoch=78, grad_norm=116.579, loss=244.4201, lr=8.53e-04]
Epoch 78 TRAIN:  21%|██        | 28/136 [00:08<00:23,  4.54it/s, epoch=78, grad_norm=87.104, loss=273.5825, lr=8.53e-04] 
Epoch 78 TRAIN:  21%|██        | 28/136 [00:08<00:23,  4.54it/s, epoch=78, grad_norm=97.052, loss=270.0744, lr=8.53e-04]
Epoch 78 TRAIN:  21%|██        | 28/136 [00:08<00:23,  4.54it/s, epoch=78, grad_norm=123.671, loss=306.5297, lr=8.53e-04]
Epoch 78 TRAIN:  23%|██▎       | 31/136 [00:08<00:22,  4.70it/s, epoch=78, grad_norm=123.671, loss=306.5297, lr=8.53e-04]
Epoch 78 TRAIN:  23%|██▎       | 31/136 [00:09<00:22,  4.70it/s, epoch=78, grad_norm=122.759, loss=246.7862, lr=8.53e-04]
Epoch 78 TRAIN:  23%|██▎       | 31/136 [00:09<00:22,  4.70it/s, epoch=78, grad_norm=106.400, loss=293.6053, lr=8.53e-04]
Epoch 78 TRAIN:  23%|██▎       | 31/136 [00:09<00:22,  4.70it/s, epoch=78, grad_norm=88.128, loss=259.6403, lr=8.53e-04] 
Epoch 78 TRAIN:  25%|██▌       | 34/136 [00:09<00:21,  4.73it/s, epoch=78, grad_norm=88.128, loss=259.6403, lr=8.53e-04]
Epoch 78 TRAIN:  25%|██▌       | 34/136 [00:09<00:21,  4.73it/s, epoch=78, grad_norm=86.612, loss=248.6479, lr=8.54e-04]
Epoch 78 TRAIN:  25%|██▌       | 34/136 [00:10<00:21,  4.73it/s, epoch=78, grad_norm=118.239, loss=275.0457, lr=8.54e-04]
Epoch 78 TRAIN:  25%|██▌       | 34/136 [00:10<00:21,  4.73it/s, epoch=78, grad_norm=75.639, loss=258.4468, lr=8.54e-04] 
Epoch 78 TRAIN:  27%|██▋       | 37/136 [00:10<00:21,  4.61it/s, epoch=78, grad_norm=75.639, loss=258.4468, lr=8.54e-04]
Epoch 78 TRAIN:  27%|██▋       | 37/136 [00:10<00:21,  4.61it/s, epoch=78, grad_norm=81.756, loss=274.2473, lr=8.54e-04]
Epoch 78 TRAIN:  27%|██▋       | 37/136 [00:10<00:21,  4.61it/s, epoch=78, grad_norm=168.259, loss=283.8759, lr=8.54e-04]
Epoch 78 TRAIN:  27%|██▋       | 37/136 [00:10<00:21,  4.61it/s, epoch=78, grad_norm=195.462, loss=307.1262, lr=8.54e-04]
Epoch 78 TRAIN:  29%|██▉       | 40/136 [00:10<00:20,  4.80it/s, epoch=78, grad_norm=195.462, loss=307.1262, lr=8.54e-04]
Epoch 78 TRAIN:  29%|██▉       | 40/136 [00:11<00:20,  4.80it/s, epoch=78, grad_norm=83.325, loss=292.7729, lr=8.54e-04] 
Epoch 78 TRAIN:  29%|██▉       | 40/136 [00:11<00:20,  4.80it/s, epoch=78, grad_norm=105.268, loss=250.5045, lr=8.54e-04]
Epoch 78 TRAIN:  29%|██▉       | 40/136 [00:11<00:20,  4.80it/s, epoch=78, grad_norm=145.239, loss=288.8179, lr=8.54e-04]
Epoch 78 TRAIN:  32%|███▏      | 43/136 [00:11<00:19,  4.75it/s, epoch=78, grad_norm=145.239, loss=288.8179, lr=8.54e-04]
Epoch 78 TRAIN:  32%|███▏      | 43/136 [00:11<00:19,  4.75it/s, epoch=78, grad_norm=173.533, loss=306.1972, lr=8.54e-04]
Epoch 78 TRAIN:  32%|███▏      | 43/136 [00:11<00:19,  4.75it/s, epoch=78, grad_norm=94.825, loss=264.8229, lr=8.54e-04] 
Epoch 78 TRAIN:  32%|███▏      | 43/136 [00:12<00:19,  4.75it/s, epoch=78, grad_norm=84.994, loss=280.1862, lr=8.54e-04]
Epoch 78 TRAIN:  34%|███▍      | 46/136 [00:12<00:18,  4.76it/s, epoch=78, grad_norm=84.994, loss=280.1862, lr=8.54e-04]
Epoch 78 TRAIN:  34%|███▍      | 46/136 [00:12<00:18,  4.76it/s, epoch=78, grad_norm=117.565, loss=297.8088, lr=8.54e-04]
Epoch 78 TRAIN:  34%|███▍      | 46/136 [00:12<00:18,  4.76it/s, epoch=78, grad_norm=119.233, loss=251.8492, lr=8.55e-04]
Epoch 78 TRAIN:  34%|███▍      | 46/136 [00:12<00:18,  4.76it/s, epoch=78, grad_norm=144.715, loss=296.4526, lr=8.55e-04]
Epoch 78 TRAIN:  36%|███▌      | 49/136 [00:12<00:18,  4.65it/s, epoch=78, grad_norm=144.715, loss=296.4526, lr=8.55e-04]
Epoch 78 TRAIN:  36%|███▌      | 49/136 [00:13<00:18,  4.65it/s, epoch=78, grad_norm=101.706, loss=288.6681, lr=8.55e-04]
Epoch 78 TRAIN:  36%|███▌      | 49/136 [00:13<00:18,  4.65it/s, epoch=78, grad_norm=105.678, loss=298.3333, lr=8.55e-04]
Epoch 78 TRAIN:  36%|███▌      | 49/136 [00:13<00:18,  4.65it/s, epoch=78, grad_norm=114.598, loss=265.0962, lr=8.55e-04]
Epoch 78 TRAIN:  38%|███▊      | 52/136 [00:13<00:18,  4.64it/s, epoch=78, grad_norm=114.598, loss=265.0962, lr=8.55e-04]
Epoch 78 TRAIN:  38%|███▊      | 52/136 [00:13<00:18,  4.64it/s, epoch=78, grad_norm=118.348, loss=236.1357, lr=8.55e-04]
Epoch 78 TRAIN:  38%|███▊      | 52/136 [00:13<00:18,  4.64it/s, epoch=78, grad_norm=124.775, loss=234.6441, lr=8.55e-04]
Epoch 78 TRAIN:  38%|███▊      | 52/136 [00:14<00:18,  4.64it/s, epoch=78, grad_norm=97.724, loss=263.5993, lr=8.55e-04] 
Epoch 78 TRAIN:  40%|████      | 55/136 [00:14<00:17,  4.63it/s, epoch=78, grad_norm=97.724, loss=263.5993, lr=8.55e-04]
Epoch 78 TRAIN:  40%|████      | 55/136 [00:14<00:17,  4.63it/s, epoch=78, grad_norm=116.292, loss=264.9176, lr=8.55e-04]
Epoch 78 TRAIN:  40%|████      | 55/136 [00:14<00:17,  4.63it/s, epoch=78, grad_norm=161.998, loss=284.3879, lr=8.55e-04]
Epoch 78 TRAIN:  40%|████      | 55/136 [00:14<00:17,  4.63it/s, epoch=78, grad_norm=139.271, loss=289.2563, lr=8.55e-04]
Epoch 78 TRAIN:  43%|████▎     | 58/136 [00:14<00:16,  4.61it/s, epoch=78, grad_norm=139.271, loss=289.2563, lr=8.55e-04]
Epoch 78 TRAIN:  43%|████▎     | 58/136 [00:14<00:16,  4.61it/s, epoch=78, grad_norm=223.994, loss=327.2159, lr=8.55e-04]
Epoch 78 TRAIN:  43%|████▎     | 58/136 [00:15<00:16,  4.61it/s, epoch=78, grad_norm=119.715, loss=267.2681, lr=8.56e-04]
Epoch 78 TRAIN:  43%|████▎     | 58/136 [00:15<00:16,  4.61it/s, epoch=78, grad_norm=161.381, loss=318.1765, lr=8.56e-04]
Epoch 78 TRAIN:  45%|████▍     | 61/136 [00:15<00:16,  4.53it/s, epoch=78, grad_norm=161.381, loss=318.1765, lr=8.56e-04]
Epoch 78 TRAIN:  45%|████▍     | 61/136 [00:15<00:16,  4.53it/s, epoch=78, grad_norm=103.635, loss=233.2193, lr=8.56e-04]
Epoch 78 TRAIN:  45%|████▍     | 61/136 [00:15<00:16,  4.53it/s, epoch=78, grad_norm=99.526, loss=274.6344, lr=8.56e-04] 
Epoch 78 TRAIN:  45%|████▍     | 61/136 [00:16<00:16,  4.53it/s, epoch=78, grad_norm=115.206, loss=279.5459, lr=8.56e-04]
Epoch 78 TRAIN:  47%|████▋     | 64/136 [00:16<00:15,  4.68it/s, epoch=78, grad_norm=115.206, loss=279.5459, lr=8.56e-04]
Epoch 78 TRAIN:  47%|████▋     | 64/136 [00:16<00:15,  4.68it/s, epoch=78, grad_norm=367.498, loss=217.1870, lr=8.56e-04]
Epoch 78 TRAIN:  47%|████▋     | 64/136 [00:16<00:15,  4.68it/s, epoch=78, grad_norm=156.635, loss=297.3321, lr=8.56e-04]
Epoch 78 TRAIN:  47%|████▋     | 64/136 [00:16<00:15,  4.68it/s, epoch=78, grad_norm=135.277, loss=284.3909, lr=8.56e-04]
Epoch 78 TRAIN:  49%|████▉     | 67/136 [00:16<00:14,  4.80it/s, epoch=78, grad_norm=135.277, loss=284.3909, lr=8.56e-04]
Epoch 78 TRAIN:  49%|████▉     | 67/136 [00:16<00:14,  4.80it/s, epoch=78, grad_norm=151.242, loss=303.4692, lr=8.56e-04]
Epoch 78 TRAIN:  49%|████▉     | 67/136 [00:17<00:14,  4.80it/s, epoch=78, grad_norm=126.668, loss=256.3986, lr=8.56e-04]
Epoch 78 TRAIN:  49%|████▉     | 67/136 [00:17<00:14,  4.80it/s, epoch=78, grad_norm=97.947, loss=300.7963, lr=8.56e-04] 
Epoch 78 TRAIN:  51%|█████▏    | 70/136 [00:17<00:14,  4.63it/s, epoch=78, grad_norm=97.947, loss=300.7963, lr=8.56e-04]
Epoch 78 TRAIN:  51%|█████▏    | 70/136 [00:17<00:14,  4.63it/s, epoch=78, grad_norm=124.683, loss=256.0890, lr=8.56e-04]
Epoch 78 TRAIN:  51%|█████▏    | 70/136 [00:17<00:14,  4.63it/s, epoch=78, grad_norm=125.002, loss=278.7080, lr=8.56e-04]
Epoch 78 TRAIN:  51%|█████▏    | 70/136 [00:17<00:14,  4.63it/s, epoch=78, grad_norm=269.690, loss=343.1140, lr=8.57e-04]
Epoch 78 TRAIN:  54%|█████▎    | 73/136 [00:17<00:13,  4.70it/s, epoch=78, grad_norm=269.690, loss=343.1140, lr=8.57e-04]
Epoch 78 TRAIN:  54%|█████▎    | 73/136 [00:18<00:13,  4.70it/s, epoch=78, grad_norm=232.421, loss=192.3698, lr=8.57e-04]
Epoch 78 TRAIN:  54%|█████▎    | 73/136 [00:18<00:13,  4.70it/s, epoch=78, grad_norm=139.339, loss=224.1040, lr=8.57e-04]
Epoch 78 TRAIN:  54%|█████▎    | 73/136 [00:18<00:13,  4.70it/s, epoch=78, grad_norm=125.343, loss=247.0609, lr=8.57e-04]
Epoch 78 TRAIN:  56%|█████▌    | 76/136 [00:18<00:12,  4.80it/s, epoch=78, grad_norm=125.343, loss=247.0609, lr=8.57e-04]
Epoch 78 TRAIN:  56%|█████▌    | 76/136 [00:18<00:12,  4.80it/s, epoch=78, grad_norm=139.820, loss=275.3209, lr=8.57e-04]
Epoch 78 TRAIN:  56%|█████▌    | 76/136 [00:18<00:12,  4.80it/s, epoch=78, grad_norm=138.385, loss=271.9604, lr=8.57e-04]
Epoch 78 TRAIN:  56%|█████▌    | 76/136 [00:19<00:12,  4.80it/s, epoch=78, grad_norm=145.273, loss=274.9184, lr=8.57e-04]
Epoch 78 TRAIN:  58%|█████▊    | 79/136 [00:19<00:11,  4.77it/s, epoch=78, grad_norm=145.273, loss=274.9184, lr=8.57e-04]
Epoch 78 TRAIN:  58%|█████▊    | 79/136 [00:19<00:11,  4.77it/s, epoch=78, grad_norm=77.587, loss=221.3156, lr=8.57e-04] 
Epoch 78 TRAIN:  58%|█████▊    | 79/136 [00:19<00:11,  4.77it/s, epoch=78, grad_norm=233.915, loss=219.2379, lr=8.57e-04]
Epoch 78 TRAIN:  58%|█████▊    | 79/136 [00:19<00:11,  4.77it/s, epoch=78, grad_norm=130.310, loss=268.2767, lr=8.57e-04]
Epoch 78 TRAIN:  60%|██████    | 82/136 [00:19<00:11,  4.82it/s, epoch=78, grad_norm=130.310, loss=268.2767, lr=8.57e-04]
Epoch 78 TRAIN:  60%|██████    | 82/136 [00:19<00:11,  4.82it/s, epoch=78, grad_norm=100.698, loss=260.8401, lr=8.57e-04]
Epoch 78 TRAIN:  60%|██████    | 82/136 [00:20<00:11,  4.82it/s, epoch=78, grad_norm=236.744, loss=302.8381, lr=8.57e-04]
Epoch 78 TRAIN:  60%|██████    | 82/136 [00:20<00:11,  4.82it/s, epoch=78, grad_norm=205.874, loss=315.1518, lr=8.58e-04]
Epoch 78 TRAIN:  62%|██████▎   | 85/136 [00:20<00:10,  4.66it/s, epoch=78, grad_norm=205.874, loss=315.1518, lr=8.58e-04]
Epoch 78 TRAIN:  62%|██████▎   | 85/136 [00:20<00:10,  4.66it/s, epoch=78, grad_norm=109.177, loss=260.2637, lr=8.58e-04]
Epoch 78 TRAIN:  62%|██████▎   | 85/136 [00:20<00:10,  4.66it/s, epoch=78, grad_norm=116.767, loss=235.0677, lr=8.58e-04]
Epoch 78 TRAIN:  62%|██████▎   | 85/136 [00:21<00:10,  4.66it/s, epoch=78, grad_norm=75.341, loss=264.5587, lr=8.58e-04] 
Epoch 78 TRAIN:  65%|██████▍   | 88/136 [00:21<00:10,  4.68it/s, epoch=78, grad_norm=75.341, loss=264.5587, lr=8.58e-04]
Epoch 78 TRAIN:  65%|██████▍   | 88/136 [00:21<00:10,  4.68it/s, epoch=78, grad_norm=510.101, loss=252.4275, lr=8.58e-04]
Epoch 78 TRAIN:  65%|██████▍   | 88/136 [00:21<00:10,  4.68it/s, epoch=78, grad_norm=167.744, loss=254.3401, lr=8.58e-04]
Epoch 78 TRAIN:  65%|██████▍   | 88/136 [00:21<00:10,  4.68it/s, epoch=78, grad_norm=121.437, loss=315.0392, lr=8.58e-04]
Epoch 78 TRAIN:  67%|██████▋   | 91/136 [00:21<00:09,  4.61it/s, epoch=78, grad_norm=121.437, loss=315.0392, lr=8.58e-04]
Epoch 78 TRAIN:  67%|██████▋   | 91/136 [00:22<00:09,  4.61it/s, epoch=78, grad_norm=110.927, loss=223.1570, lr=8.58e-04]
Epoch 78 TRAIN:  67%|██████▋   | 91/136 [00:22<00:09,  4.61it/s, epoch=78, grad_norm=122.450, loss=266.0804, lr=8.58e-04]
Epoch 78 TRAIN:  67%|██████▋   | 91/136 [00:22<00:09,  4.61it/s, epoch=78, grad_norm=144.751, loss=306.7542, lr=8.58e-04]
Epoch 78 TRAIN:  69%|██████▉   | 94/136 [00:22<00:09,  4.66it/s, epoch=78, grad_norm=144.751, loss=306.7542, lr=8.58e-04]
Epoch 78 TRAIN:  69%|██████▉   | 94/136 [00:22<00:09,  4.66it/s, epoch=78, grad_norm=185.037, loss=274.2845, lr=8.58e-04]
Epoch 78 TRAIN:  69%|██████▉   | 94/136 [00:22<00:09,  4.66it/s, epoch=78, grad_norm=163.008, loss=294.8406, lr=8.58e-04]
Epoch 78 TRAIN:  69%|██████▉   | 94/136 [00:22<00:09,  4.66it/s, epoch=78, grad_norm=381.283, loss=261.6121, lr=8.58e-04]
Epoch 78 TRAIN:  69%|██████▉   | 94/136 [00:23<00:09,  4.66it/s, epoch=78, grad_norm=140.338, loss=347.4993, lr=8.59e-04]
Epoch 78 TRAIN:  72%|███████▏  | 98/136 [00:23<00:07,  5.04it/s, epoch=78, grad_norm=140.338, loss=347.4993, lr=8.59e-04]
Epoch 78 TRAIN:  72%|███████▏  | 98/136 [00:23<00:07,  5.04it/s, epoch=78, grad_norm=155.305, loss=244.0726, lr=8.59e-04]
Epoch 78 TRAIN:  72%|███████▏  | 98/136 [00:23<00:07,  5.04it/s, epoch=78, grad_norm=229.157, loss=319.3004, lr=8.59e-04]2026-04-16 20:32:34,174 DEBUG TRAIN | steps/sec 4.240| Batch 78/100 loss 319.300354 loss_att 251.082947 loss_ctc 478.474304 th_accuracy 0.438914 lr 8.5872e-04 grad_norm 229.156738 rank 0

Epoch 78 TRAIN:  72%|███████▏  | 98/136 [00:23<00:07,  5.04it/s, epoch=78, grad_norm=298.133, loss=208.3034, lr=8.59e-04]
Epoch 78 TRAIN:  74%|███████▍  | 101/136 [00:23<00:07,  4.70it/s, epoch=78, grad_norm=298.133, loss=208.3034, lr=8.59e-04]
Epoch 78 TRAIN:  74%|███████▍  | 101/136 [00:24<00:07,  4.70it/s, epoch=78, grad_norm=168.074, loss=296.8084, lr=8.59e-04]
Epoch 78 TRAIN:  74%|███████▍  | 101/136 [00:24<00:07,  4.70it/s, epoch=78, grad_norm=145.181, loss=287.5449, lr=8.59e-04]
Epoch 78 TRAIN:  74%|███████▍  | 101/136 [00:24<00:07,  4.70it/s, epoch=78, grad_norm=148.791, loss=227.7529, lr=8.59e-04]
Epoch 78 TRAIN:  76%|███████▋  | 104/136 [00:24<00:06,  4.83it/s, epoch=78, grad_norm=148.791, loss=227.7529, lr=8.59e-04]
Epoch 78 TRAIN:  76%|███████▋  | 104/136 [00:24<00:06,  4.83it/s, epoch=78, grad_norm=232.487, loss=295.5331, lr=8.59e-04]
Epoch 78 TRAIN:  76%|███████▋  | 104/136 [00:24<00:06,  4.83it/s, epoch=78, grad_norm=323.981, loss=205.1261, lr=8.59e-04]
Epoch 78 TRAIN:  76%|███████▋  | 104/136 [00:25<00:06,  4.83it/s, epoch=78, grad_norm=144.328, loss=268.9663, lr=8.59e-04]
Epoch 78 TRAIN:  79%|███████▊  | 107/136 [00:25<00:06,  4.58it/s, epoch=78, grad_norm=144.328, loss=268.9663, lr=8.59e-04]
Epoch 78 TRAIN:  79%|███████▊  | 107/136 [00:25<00:06,  4.58it/s, epoch=78, grad_norm=237.691, loss=190.1336, lr=8.59e-04]
Epoch 78 TRAIN:  79%|███████▊  | 107/136 [00:25<00:06,  4.58it/s, epoch=78, grad_norm=136.932, loss=307.8568, lr=8.59e-04]
Epoch 78 TRAIN:  79%|███████▊  | 107/136 [00:25<00:06,  4.58it/s, epoch=78, grad_norm=156.196, loss=181.8618, lr=8.60e-04]
Epoch 78 TRAIN:  81%|████████  | 110/136 [00:25<00:05,  4.58it/s, epoch=78, grad_norm=156.196, loss=181.8618, lr=8.60e-04]
Epoch 78 TRAIN:  81%|████████  | 110/136 [00:26<00:05,  4.58it/s, epoch=78, grad_norm=111.714, loss=209.1377, lr=8.60e-04]
Epoch 78 TRAIN:  81%|████████  | 110/136 [00:26<00:05,  4.58it/s, epoch=78, grad_norm=250.002, loss=297.7169, lr=8.60e-04]
Epoch 78 TRAIN:  81%|████████  | 110/136 [00:26<00:05,  4.58it/s, epoch=78, grad_norm=158.422, loss=254.3329, lr=8.60e-04]
Epoch 78 TRAIN:  83%|████████▎ | 113/136 [00:26<00:04,  4.69it/s, epoch=78, grad_norm=158.422, loss=254.3329, lr=8.60e-04]
Epoch 78 TRAIN:  83%|████████▎ | 113/136 [00:26<00:04,  4.69it/s, epoch=78, grad_norm=219.094, loss=291.3090, lr=8.60e-04]
Epoch 78 TRAIN:  83%|████████▎ | 113/136 [00:26<00:04,  4.69it/s, epoch=78, grad_norm=262.019, loss=311.0225, lr=8.60e-04]
Epoch 78 TRAIN:  83%|████████▎ | 113/136 [00:27<00:04,  4.69it/s, epoch=78, grad_norm=239.101, loss=308.8370, lr=8.60e-04]
Epoch 78 TRAIN:  85%|████████▌ | 116/136 [00:27<00:04,  4.34it/s, epoch=78, grad_norm=239.101, loss=308.8370, lr=8.60e-04]
Epoch 78 TRAIN:  85%|████████▌ | 116/136 [00:27<00:04,  4.34it/s, epoch=78, grad_norm=221.920, loss=294.0934, lr=8.60e-04]
Epoch 78 TRAIN:  85%|████████▌ | 116/136 [00:27<00:04,  4.34it/s, epoch=78, grad_norm=198.742, loss=256.2516, lr=8.60e-04]
Epoch 78 TRAIN:  85%|████████▌ | 116/136 [00:27<00:04,  4.34it/s, epoch=78, grad_norm=196.139, loss=287.6708, lr=8.60e-04]
Epoch 78 TRAIN:  88%|████████▊ | 119/136 [00:27<00:03,  4.56it/s, epoch=78, grad_norm=196.139, loss=287.6708, lr=8.60e-04]
Epoch 78 TRAIN:  88%|████████▊ | 119/136 [00:28<00:03,  4.56it/s, epoch=78, grad_norm=234.980, loss=262.5919, lr=8.60e-04]
Epoch 78 TRAIN:  88%|████████▊ | 119/136 [00:28<00:03,  4.56it/s, epoch=78, grad_norm=160.609, loss=292.8116, lr=8.60e-04]
Epoch 78 TRAIN:  88%|████████▊ | 119/136 [00:28<00:03,  4.56it/s, epoch=78, grad_norm=221.455, loss=348.1520, lr=8.60e-04]
Epoch 78 TRAIN:  90%|████████▉ | 122/136 [00:28<00:02,  4.79it/s, epoch=78, grad_norm=221.455, loss=348.1520, lr=8.60e-04]
Epoch 78 TRAIN:  90%|████████▉ | 122/136 [00:28<00:02,  4.79it/s, epoch=78, grad_norm=162.397, loss=304.1024, lr=8.61e-04]
Epoch 78 TRAIN:  90%|████████▉ | 122/136 [00:28<00:02,  4.79it/s, epoch=78, grad_norm=165.613, loss=218.9437, lr=8.61e-04]
Epoch 78 TRAIN:  90%|████████▉ | 122/136 [00:29<00:02,  4.79it/s, epoch=78, grad_norm=146.372, loss=221.7990, lr=8.61e-04]
Epoch 78 TRAIN:  92%|█████████▏| 125/136 [00:29<00:02,  4.47it/s, epoch=78, grad_norm=146.372, loss=221.7990, lr=8.61e-04]
Epoch 78 TRAIN:  92%|█████████▏| 125/136 [00:29<00:02,  4.47it/s, epoch=78, grad_norm=276.726, loss=312.0140, lr=8.61e-04]
Epoch 78 TRAIN:  92%|█████████▏| 125/136 [00:29<00:02,  4.47it/s, epoch=78, grad_norm=152.337, loss=224.6441, lr=8.61e-04]
Epoch 78 TRAIN:  92%|█████████▏| 125/136 [00:29<00:02,  4.47it/s, epoch=78, grad_norm=121.046, loss=232.2187, lr=8.61e-04]
Epoch 78 TRAIN:  92%|█████████▏| 125/136 [00:29<00:02,  4.47it/s, epoch=78, grad_norm=112.897, loss=217.1781, lr=8.61e-04]
Epoch 78 TRAIN:  95%|█████████▍| 129/136 [00:29<00:01,  4.93it/s, epoch=78, grad_norm=112.897, loss=217.1781, lr=8.61e-04]
Epoch 78 TRAIN:  95%|█████████▍| 129/136 [00:29<00:01,  4.93it/s, epoch=78, grad_norm=273.834, loss=277.5298, lr=8.61e-04]
Epoch 78 TRAIN:  95%|█████████▍| 129/136 [00:30<00:01,  4.93it/s, epoch=78, grad_norm=156.396, loss=251.8234, lr=8.61e-04]
Epoch 78 TRAIN:  95%|█████████▍| 129/136 [00:30<00:01,  4.93it/s, epoch=78, grad_norm=130.160, loss=205.2885, lr=8.61e-04]
Epoch 78 TRAIN:  97%|█████████▋| 132/136 [00:30<00:00,  5.04it/s, epoch=78, grad_norm=130.160, loss=205.2885, lr=8.61e-04]
Epoch 78 TRAIN:  97%|█████████▋| 132/136 [00:30<00:00,  5.04it/s, epoch=78, grad_norm=285.011, loss=306.3587, lr=8.61e-04]
Epoch 78 TRAIN:  97%|█████████▋| 132/136 [00:30<00:00,  5.04it/s, epoch=78, grad_norm=266.089, loss=328.5388, lr=8.61e-04]
Epoch 78 TRAIN:  97%|█████████▋| 132/136 [00:30<00:00,  5.04it/s, epoch=78, grad_norm=260.120, loss=322.5322, lr=8.62e-04]
Epoch 78 TRAIN:  99%|█████████▉| 135/136 [00:30<00:00,  5.11it/s, epoch=78, grad_norm=260.120, loss=322.5322, lr=8.62e-04]
Epoch 78 TRAIN:  99%|█████████▉| 135/136 [00:31<00:00,  5.11it/s, epoch=78, grad_norm=321.753, loss=361.5731, lr=8.62e-04]
Epoch 78 TRAIN:  99%|█████████▉| 135/136 [00:31<00:00,  5.11it/s, epoch=78, grad_norm=327.392, loss=335.2390, lr=8.62e-04]
Epoch 78 TRAIN: 137it [00:31,  4.38it/s, epoch=78, grad_norm=327.392, loss=335.2390, lr=8.62e-04]                         

CV:   0%|          | 0/6 [00:00<?, ?it/s]/home/rookie/Documents/batuka/projects/wenet/wenet/wenet/utils/train_utils.py:689: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  amp_autocast(enabled=dtype is not None,

CV:  17%|█▋        | 1/6 [00:00<00:03,  1.32it/s]
                                                 
2026-04-16 20:32:42,760 INFO Epoch 78 Step 10771 CV info lr 8.6168e-04 cv_loss 459.70166015625 rank 0 acc 0.3734610130389531
2026-04-16 20:32:42,762 INFO [Rank 0] Checkpoint: save to checkpoint exp/conformer_mongolian/epoch_78.pt
2026-04-16 20:32:42,874 INFO Epoch 79 Step 10771 TRAIN info lr 8.6168e-04 rank 0
2026-04-16 20:32:42,890 INFO using accumulate grad, new batch size is 1 times larger than before

Epoch 79 TRAIN:   0%|          | 0/137 [00:00<?, ?it/s]
Epoch 79 TRAIN:   0%|          | 0/137 [00:04<?, ?it/s, epoch=79, grad_norm=300.559, loss=259.6230, lr=8.62e-04]
Epoch 79 TRAIN:   1%|          | 1/137 [00:04<09:36,  4.24s/it, epoch=79, grad_norm=300.559, loss=259.6230, lr=8.62e-04]
Epoch 79 TRAIN:   1%|          | 1/137 [00:04<09:36,  4.24s/it, epoch=79, grad_norm=405.956, loss=300.4478, lr=8.62e-04]
Epoch 79 TRAIN:   1%|          | 1/137 [00:04<09:36,  4.24s/it, epoch=79, grad_norm=333.554, loss=323.0144, lr=8.62e-04]
Epoch 79 TRAIN:   1%|          | 1/137 [00:04<09:36,  4.24s/it, epoch=79, grad_norm=440.690, loss=387.9588, lr=8.62e-04]
Epoch 79 TRAIN:   1%|          | 1/137 [00:04<09:36,  4.24s/it, epoch=79, grad_norm=281.536, loss=349.0822, lr=8.62e-04]
Epoch 79 TRAIN:   1%|          | 1/137 [00:04<09:36,  4.24s/it, epoch=79, grad_norm=332.934, loss=309.2551, lr=8.62e-04]
Epoch 79 TRAIN:   1%|          | 1/137 [00:04<09:36,  4.24s/it, epoch=79, grad_norm=183.695, loss=298.0108, lr=8.62e-04]
Epoch 79 TRAIN:   1%|          | 1/137 [00:04<09:36,  4.24s/it, epoch=79, grad_norm=214.351, loss=302.4249, lr=8.62e-04]
Epoch 79 TRAIN:   1%|          | 1/137 [00:04<09:36,  4.24s/it, epoch=79, grad_norm=176.685, loss=310.8389, lr=8.62e-04]
Epoch 79 TRAIN:   7%|▋         | 9/137 [00:04<00:52,  2.42it/s, epoch=79, grad_norm=176.685, loss=310.8389, lr=8.62e-04]
Epoch 79 TRAIN:   7%|▋         | 9/137 [00:05<00:52,  2.42it/s, epoch=79, grad_norm=139.952, loss=290.6338, lr=8.62e-04]
Epoch 79 TRAIN:   7%|▋         | 9/137 [00:05<00:52,  2.42it/s, epoch=79, grad_norm=129.482, loss=279.0719, lr=8.63e-04]
Epoch 79 TRAIN:   7%|▋         | 9/137 [00:05<00:52,  2.42it/s, epoch=79, grad_norm=118.022, loss=304.1290, lr=8.63e-04]
Epoch 79 TRAIN:   7%|▋         | 9/137 [00:05<00:52,  2.42it/s, epoch=79, grad_norm=139.359, loss=276.4922, lr=8.63e-04]
Epoch 79 TRAIN:   9%|▉         | 13/137 [00:05<00:38,  3.20it/s, epoch=79, grad_norm=139.359, loss=276.4922, lr=8.63e-04]
Epoch 79 TRAIN:   9%|▉         | 13/137 [00:05<00:38,  3.20it/s, epoch=79, grad_norm=165.787, loss=267.8772, lr=8.63e-04]
Epoch 79 TRAIN:   9%|▉         | 13/137 [00:05<00:38,  3.20it/s, epoch=79, grad_norm=154.485, loss=264.7239, lr=8.63e-04]
Epoch 79 TRAIN:   9%|▉         | 13/137 [00:06<00:38,  3.20it/s, epoch=79, grad_norm=106.264, loss=277.3638, lr=8.63e-04]
Epoch 79 TRAIN:   9%|▉         | 13/137 [00:06<00:38,  3.20it/s, epoch=79, grad_norm=124.184, loss=258.5275, lr=8.63e-04]
Epoch 79 TRAIN:  12%|█▏        | 17/137 [00:06<00:32,  3.69it/s, epoch=79, grad_norm=124.184, loss=258.5275, lr=8.63e-04]
Epoch 79 TRAIN:  12%|█▏        | 17/137 [00:06<00:32,  3.69it/s, epoch=79, grad_norm=91.455, loss=284.8087, lr=8.63e-04] 
Epoch 79 TRAIN:  12%|█▏        | 17/137 [00:06<00:32,  3.69it/s, epoch=79, grad_norm=120.737, loss=311.7644, lr=8.63e-04]
Epoch 79 TRAIN:  12%|█▏        | 17/137 [00:06<00:32,  3.69it/s, epoch=79, grad_norm=107.003, loss=259.3023, lr=8.63e-04]
Epoch 79 TRAIN:  12%|█▏        | 17/137 [00:06<00:32,  3.69it/s, epoch=79, grad_norm=118.850, loss=285.8400, lr=8.63e-04]
Epoch 79 TRAIN:  15%|█▌        | 21/137 [00:06<00:27,  4.23it/s, epoch=79, grad_norm=118.850, loss=285.8400, lr=8.63e-04]
Epoch 79 TRAIN:  15%|█▌        | 21/137 [00:07<00:27,  4.23it/s, epoch=79, grad_norm=87.003, loss=275.5992, lr=8.63e-04] 
Epoch 79 TRAIN:  15%|█▌        | 21/137 [00:07<00:27,  4.23it/s, epoch=79, grad_norm=103.000, loss=260.6798, lr=8.64e-04]
Epoch 79 TRAIN:  15%|█▌        | 21/137 [00:07<00:27,  4.23it/s, epoch=79, grad_norm=121.788, loss=276.8454, lr=8.64e-04]
Epoch 79 TRAIN:  15%|█▌        | 21/137 [00:07<00:27,  4.23it/s, epoch=79, grad_norm=93.083, loss=243.1447, lr=8.64e-04] 
Epoch 79 TRAIN:  18%|█▊        | 25/137 [00:07<00:25,  4.42it/s, epoch=79, grad_norm=93.083, loss=243.1447, lr=8.64e-04]
Epoch 79 TRAIN:  18%|█▊        | 25/137 [00:08<00:25,  4.42it/s, epoch=79, grad_norm=105.567, loss=276.8609, lr=8.64e-04]
Epoch 79 TRAIN:  18%|█▊        | 25/137 [00:08<00:25,  4.42it/s, epoch=79, grad_norm=119.635, loss=244.0341, lr=8.64e-04]
Epoch 79 TRAIN:  18%|█▊        | 25/137 [00:08<00:25,  4.42it/s, epoch=79, grad_norm=125.050, loss=246.9997, lr=8.64e-04]
Epoch 79 TRAIN:  20%|██        | 28/137 [00:08<00:24,  4.40it/s, epoch=79, grad_norm=125.050, loss=246.9997, lr=8.64e-04]
Epoch 79 TRAIN:  20%|██        | 28/137 [00:08<00:24,  4.40it/s, epoch=79, grad_norm=226.733, loss=318.8977, lr=8.64e-04]
Epoch 79 TRAIN:  20%|██        | 28/137 [00:08<00:24,  4.40it/s, epoch=79, grad_norm=180.194, loss=294.1754, lr=8.64e-04]
Epoch 79 TRAIN:  20%|██        | 28/137 [00:09<00:24,  4.40it/s, epoch=79, grad_norm=100.605, loss=298.0839, lr=8.64e-04]
Epoch 79 TRAIN:  23%|██▎       | 31/137 [00:09<00:23,  4.60it/s, epoch=79, grad_norm=100.605, loss=298.0839, lr=8.64e-04]
Epoch 79 TRAIN:  23%|██▎       | 31/137 [00:09<00:23,  4.60it/s, epoch=79, grad_norm=123.599, loss=267.2085, lr=8.64e-04]
Epoch 79 TRAIN:  23%|██▎       | 31/137 [00:09<00:23,  4.60it/s, epoch=79, grad_norm=141.449, loss=254.7903, lr=8.64e-04]
Epoch 79 TRAIN:  23%|██▎       | 31/137 [00:09<00:23,  4.60it/s, epoch=79, grad_norm=101.469, loss=293.5590, lr=8.64e-04]
Epoch 79 TRAIN:  25%|██▍       | 34/137 [00:09<00:21,  4.77it/s, epoch=79, grad_norm=101.469, loss=293.5590, lr=8.64e-04]
Epoch 79 TRAIN:  25%|██▍       | 34/137 [00:09<00:21,  4.77it/s, epoch=79, grad_norm=87.384, loss=314.2336, lr=8.64e-04] 
Epoch 79 TRAIN:  25%|██▍       | 34/137 [00:10<00:21,  4.77it/s, epoch=79, grad_norm=149.245, loss=268.7855, lr=8.65e-04]
Epoch 79 TRAIN:  25%|██▍       | 34/137 [00:10<00:21,  4.77it/s, epoch=79, grad_norm=109.944, loss=266.5645, lr=8.65e-04]
Epoch 79 TRAIN:  27%|██▋       | 37/137 [00:10<00:21,  4.71it/s, epoch=79, grad_norm=109.944, loss=266.5645, lr=8.65e-04]
Epoch 79 TRAIN:  27%|██▋       | 37/137 [00:10<00:21,  4.71it/s, epoch=79, grad_norm=138.723, loss=321.1407, lr=8.65e-04]
Epoch 79 TRAIN:  27%|██▋       | 37/137 [00:10<00:21,  4.71it/s, epoch=79, grad_norm=105.784, loss=276.5247, lr=8.65e-04]
Epoch 79 TRAIN:  27%|██▋       | 37/137 [00:10<00:21,  4.71it/s, epoch=79, grad_norm=181.878, loss=241.3708, lr=8.65e-04]
Epoch 79 TRAIN:  29%|██▉       | 40/137 [00:10<00:20,  4.79it/s, epoch=79, grad_norm=181.878, loss=241.3708, lr=8.65e-04]
Epoch 79 TRAIN:  29%|██▉       | 40/137 [00:11<00:20,  4.79it/s, epoch=79, grad_norm=164.734, loss=264.7794, lr=8.65e-04]
Epoch 79 TRAIN:  29%|██▉       | 40/137 [00:11<00:20,  4.79it/s, epoch=79, grad_norm=134.419, loss=285.5822, lr=8.65e-04]
Epoch 79 TRAIN:  29%|██▉       | 40/137 [00:11<00:20,  4.79it/s, epoch=79, grad_norm=117.499, loss=259.0938, lr=8.65e-04]
Epoch 79 TRAIN:  31%|███▏      | 43/137 [00:11<00:20,  4.53it/s, epoch=79, grad_norm=117.499, loss=259.0938, lr=8.65e-04]
Epoch 79 TRAIN:  31%|███▏      | 43/137 [00:11<00:20,  4.53it/s, epoch=79, grad_norm=166.124, loss=286.9225, lr=8.65e-04]
Epoch 79 TRAIN:  31%|███▏      | 43/137 [00:11<00:20,  4.53it/s, epoch=79, grad_norm=129.823, loss=276.7933, lr=8.65e-04]
Epoch 79 TRAIN:  31%|███▏      | 43/137 [00:12<00:20,  4.53it/s, epoch=79, grad_norm=163.704, loss=330.8291, lr=8.65e-04]
Epoch 79 TRAIN:  34%|███▎      | 46/137 [00:12<00:19,  4.71it/s, epoch=79, grad_norm=163.704, loss=330.8291, lr=8.65e-04]
Epoch 79 TRAIN:  34%|███▎      | 46/137 [00:12<00:19,  4.71it/s, epoch=79, grad_norm=108.966, loss=253.2463, lr=8.65e-04]
Epoch 79 TRAIN:  34%|███▎      | 46/137 [00:12<00:19,  4.71it/s, epoch=79, grad_norm=137.777, loss=239.1903, lr=8.66e-04]
Epoch 79 TRAIN:  34%|███▎      | 46/137 [00:12<00:19,  4.71it/s, epoch=79, grad_norm=106.849, loss=256.7638, lr=8.66e-04]
Epoch 79 TRAIN:  36%|███▌      | 49/137 [00:12<00:19,  4.58it/s, epoch=79, grad_norm=106.849, loss=256.7638, lr=8.66e-04]
Epoch 79 TRAIN:  36%|███▌      | 49/137 [00:13<00:19,  4.58it/s, epoch=79, grad_norm=79.837, loss=223.1208, lr=8.66e-04] 
Epoch 79 TRAIN:  36%|███▌      | 49/137 [00:13<00:19,  4.58it/s, epoch=79, grad_norm=129.742, loss=280.3143, lr=8.66e-04]
Epoch 79 TRAIN:  36%|███▌      | 49/137 [00:13<00:19,  4.58it/s, epoch=79, grad_norm=102.696, loss=286.2513, lr=8.66e-04]
Epoch 79 TRAIN:  38%|███▊      | 52/137 [00:13<00:18,  4.62it/s, epoch=79, grad_norm=102.696, loss=286.2513, lr=8.66e-04]
Epoch 79 TRAIN:  38%|███▊      | 52/137 [00:13<00:18,  4.62it/s, epoch=79, grad_norm=172.657, loss=287.7013, lr=8.66e-04]
Epoch 79 TRAIN:  38%|███▊      | 52/137 [00:13<00:18,  4.62it/s, epoch=79, grad_norm=112.477, loss=220.0084, lr=8.66e-04]
Epoch 79 TRAIN:  38%|███▊      | 52/137 [00:14<00:18,  4.62it/s, epoch=79, grad_norm=99.052, loss=260.3805, lr=8.66e-04] 
Epoch 79 TRAIN:  40%|████      | 55/137 [00:14<00:17,  4.70it/s, epoch=79, grad_norm=99.052, loss=260.3805, lr=8.66e-04]
Epoch 79 TRAIN:  40%|████      | 55/137 [00:14<00:17,  4.70it/s, epoch=79, grad_norm=175.018, loss=318.9780, lr=8.66e-04]
Epoch 79 TRAIN:  40%|████      | 55/137 [00:14<00:17,  4.70it/s, epoch=79, grad_norm=140.372, loss=282.5906, lr=8.66e-04]
Epoch 79 TRAIN:  40%|████      | 55/137 [00:14<00:17,  4.70it/s, epoch=79, grad_norm=100.282, loss=287.3544, lr=8.66e-04]
Epoch 79 TRAIN:  42%|████▏     | 58/137 [00:14<00:16,  4.78it/s, epoch=79, grad_norm=100.282, loss=287.3544, lr=8.66e-04]
Epoch 79 TRAIN:  42%|████▏     | 58/137 [00:14<00:16,  4.78it/s, epoch=79, grad_norm=117.656, loss=283.5570, lr=8.66e-04]
Epoch 79 TRAIN:  42%|████▏     | 58/137 [00:15<00:16,  4.78it/s, epoch=79, grad_norm=143.558, loss=241.7757, lr=8.66e-04]
Epoch 79 TRAIN:  42%|████▏     | 58/137 [00:15<00:16,  4.78it/s, epoch=79, grad_norm=102.042, loss=302.1149, lr=8.67e-04]
Epoch 79 TRAIN:  45%|████▍     | 61/137 [00:15<00:16,  4.56it/s, epoch=79, grad_norm=102.042, loss=302.1149, lr=8.67e-04]
Epoch 79 TRAIN:  45%|████▍     | 61/137 [00:15<00:16,  4.56it/s, epoch=79, grad_norm=106.490, loss=244.6364, lr=8.67e-04]
Epoch 79 TRAIN:  45%|████▍     | 61/137 [00:15<00:16,  4.56it/s, epoch=79, grad_norm=143.071, loss=245.8134, lr=8.67e-04]
Epoch 79 TRAIN:  45%|████▍     | 61/137 [00:16<00:16,  4.56it/s, epoch=79, grad_norm=132.945, loss=241.9227, lr=8.67e-04]
Epoch 79 TRAIN:  47%|████▋     | 64/137 [00:16<00:15,  4.62it/s, epoch=79, grad_norm=132.945, loss=241.9227, lr=8.67e-04]
Epoch 79 TRAIN:  47%|████▋     | 64/137 [00:16<00:15,  4.62it/s, epoch=79, grad_norm=100.661, loss=269.0526, lr=8.67e-04]
Epoch 79 TRAIN:  47%|████▋     | 64/137 [00:16<00:15,  4.62it/s, epoch=79, grad_norm=107.186, loss=277.4125, lr=8.67e-04]
Epoch 79 TRAIN:  47%|████▋     | 64/137 [00:16<00:15,  4.62it/s, epoch=79, grad_norm=117.285, loss=281.1804, lr=8.67e-04]
Epoch 79 TRAIN:  49%|████▉     | 67/137 [00:16<00:15,  4.65it/s, epoch=79, grad_norm=117.285, loss=281.1804, lr=8.67e-04]
Epoch 79 TRAIN:  49%|████▉     | 67/137 [00:16<00:15,  4.65it/s, epoch=79, grad_norm=162.028, loss=325.7725, lr=8.67e-04]
Epoch 79 TRAIN:  49%|████▉     | 67/137 [00:17<00:15,  4.65it/s, epoch=79, grad_norm=116.580, loss=270.2282, lr=8.67e-04]
Epoch 79 TRAIN:  49%|████▉     | 67/137 [00:17<00:15,  4.65it/s, epoch=79, grad_norm=126.504, loss=287.9268, lr=8.67e-04]
Epoch 79 TRAIN:  51%|█████     | 70/137 [00:17<00:14,  4.58it/s, epoch=79, grad_norm=126.504, loss=287.9268, lr=8.67e-04]
Epoch 79 TRAIN:  51%|█████     | 70/137 [00:17<00:14,  4.58it/s, epoch=79, grad_norm=122.092, loss=280.8972, lr=8.67e-04]
Epoch 79 TRAIN:  51%|█████     | 70/137 [00:17<00:14,  4.58it/s, epoch=79, grad_norm=129.363, loss=335.1411, lr=8.67e-04]
Epoch 79 TRAIN:  51%|█████     | 70/137 [00:18<00:14,  4.58it/s, epoch=79, grad_norm=106.925, loss=293.0356, lr=8.68e-04]
Epoch 79 TRAIN:  53%|█████▎    | 73/137 [00:18<00:14,  4.52it/s, epoch=79, grad_norm=106.925, loss=293.0356, lr=8.68e-04]
Epoch 79 TRAIN:  53%|█████▎    | 73/137 [00:18<00:14,  4.52it/s, epoch=79, grad_norm=110.844, loss=267.2790, lr=8.68e-04]
Epoch 79 TRAIN:  53%|█████▎    | 73/137 [00:18<00:14,  4.52it/s, epoch=79, grad_norm=129.525, loss=293.3545, lr=8.68e-04]
Epoch 79 TRAIN:  53%|█████▎    | 73/137 [00:18<00:14,  4.52it/s, epoch=79, grad_norm=135.106, loss=269.2031, lr=8.68e-04]
Epoch 79 TRAIN:  55%|█████▌    | 76/137 [00:18<00:13,  4.52it/s, epoch=79, grad_norm=135.106, loss=269.2031, lr=8.68e-04]
Epoch 79 TRAIN:  55%|█████▌    | 76/137 [00:18<00:13,  4.52it/s, epoch=79, grad_norm=147.627, loss=337.2605, lr=8.68e-04]
Epoch 79 TRAIN:  55%|█████▌    | 76/137 [00:19<00:13,  4.52it/s, epoch=79, grad_norm=110.405, loss=284.0481, lr=8.68e-04]
Epoch 79 TRAIN:  55%|█████▌    | 76/137 [00:19<00:13,  4.52it/s, epoch=79, grad_norm=271.513, loss=253.3999, lr=8.68e-04]
Epoch 79 TRAIN:  58%|█████▊    | 79/137 [00:19<00:12,  4.63it/s, epoch=79, grad_norm=271.513, loss=253.3999, lr=8.68e-04]
Epoch 79 TRAIN:  58%|█████▊    | 79/137 [00:19<00:12,  4.63it/s, epoch=79, grad_norm=241.545, loss=256.6864, lr=8.68e-04]
Epoch 79 TRAIN:  58%|█████▊    | 79/137 [00:19<00:12,  4.63it/s, epoch=79, grad_norm=168.512, loss=220.1578, lr=8.68e-04]
Epoch 79 TRAIN:  58%|█████▊    | 79/137 [00:20<00:12,  4.63it/s, epoch=79, grad_norm=148.499, loss=231.7544, lr=8.68e-04]
Epoch 79 TRAIN:  60%|█████▉    | 82/137 [00:20<00:11,  4.70it/s, epoch=79, grad_norm=148.499, loss=231.7544, lr=8.68e-04]
Epoch 79 TRAIN:  60%|█████▉    | 82/137 [00:20<00:11,  4.70it/s, epoch=79, grad_norm=124.057, loss=246.9681, lr=8.68e-04]
Epoch 79 TRAIN:  60%|█████▉    | 82/137 [00:20<00:11,  4.70it/s, epoch=79, grad_norm=100.801, loss=227.2938, lr=8.68e-04]
Epoch 79 TRAIN:  60%|█████▉    | 82/137 [00:20<00:11,  4.70it/s, epoch=79, grad_norm=117.997, loss=217.6639, lr=8.68e-04]
Epoch 79 TRAIN:  62%|██████▏   | 85/137 [00:20<00:10,  4.83it/s, epoch=79, grad_norm=117.997, loss=217.6639, lr=8.68e-04]
Epoch 79 TRAIN:  62%|██████▏   | 85/137 [00:20<00:10,  4.83it/s, epoch=79, grad_norm=195.616, loss=271.1756, lr=8.69e-04]
Epoch 79 TRAIN:  62%|██████▏   | 85/137 [00:21<00:10,  4.83it/s, epoch=79, grad_norm=139.545, loss=221.1142, lr=8.69e-04]
Epoch 79 TRAIN:  62%|██████▏   | 85/137 [00:21<00:10,  4.83it/s, epoch=79, grad_norm=198.349, loss=194.9559, lr=8.69e-04]
Epoch 79 TRAIN:  64%|██████▍   | 88/137 [00:21<00:10,  4.65it/s, epoch=79, grad_norm=198.349, loss=194.9559, lr=8.69e-04]
Epoch 79 TRAIN:  64%|██████▍   | 88/137 [00:21<00:10,  4.65it/s, epoch=79, grad_norm=260.027, loss=293.9962, lr=8.69e-04]
Epoch 79 TRAIN:  64%|██████▍   | 88/137 [00:21<00:10,  4.65it/s, epoch=79, grad_norm=231.754, loss=256.3850, lr=8.69e-04]
Epoch 79 TRAIN:  64%|██████▍   | 88/137 [00:21<00:10,  4.65it/s, epoch=79, grad_norm=271.724, loss=300.4222, lr=8.69e-04]
Epoch 79 TRAIN:  66%|██████▋   | 91/137 [00:21<00:09,  4.67it/s, epoch=79, grad_norm=271.724, loss=300.4222, lr=8.69e-04]
Epoch 79 TRAIN:  66%|██████▋   | 91/137 [00:22<00:09,  4.67it/s, epoch=79, grad_norm=207.300, loss=284.7121, lr=8.69e-04]
Epoch 79 TRAIN:  66%|██████▋   | 91/137 [00:22<00:09,  4.67it/s, epoch=79, grad_norm=120.309, loss=259.4008, lr=8.69e-04]
Epoch 79 TRAIN:  66%|██████▋   | 91/137 [00:22<00:09,  4.67it/s, epoch=79, grad_norm=137.648, loss=280.0438, lr=8.69e-04]
Epoch 79 TRAIN:  69%|██████▊   | 94/137 [00:22<00:09,  4.45it/s, epoch=79, grad_norm=137.648, loss=280.0438, lr=8.69e-04]
Epoch 79 TRAIN:  69%|██████▊   | 94/137 [00:22<00:09,  4.45it/s, epoch=79, grad_norm=373.249, loss=296.5054, lr=8.69e-04]
Epoch 79 TRAIN:  69%|██████▊   | 94/137 [00:23<00:09,  4.45it/s, epoch=79, grad_norm=332.821, loss=305.9846, lr=8.69e-04]
Epoch 79 TRAIN:  69%|██████▊   | 94/137 [00:23<00:09,  4.45it/s, epoch=79, grad_norm=157.262, loss=276.7373, lr=8.69e-04]
Epoch 79 TRAIN:  71%|███████   | 97/137 [00:23<00:08,  4.58it/s, epoch=79, grad_norm=157.262, loss=276.7373, lr=8.69e-04]
Epoch 79 TRAIN:  71%|███████   | 97/137 [00:23<00:08,  4.58it/s, epoch=79, grad_norm=293.615, loss=190.2193, lr=8.70e-04]
Epoch 79 TRAIN:  71%|███████   | 97/137 [00:23<00:08,  4.58it/s, epoch=79, grad_norm=167.183, loss=268.9954, lr=8.70e-04]
Epoch 79 TRAIN:  71%|███████   | 97/137 [00:23<00:08,  4.58it/s, epoch=79, grad_norm=177.420, loss=216.3675, lr=8.70e-04]2026-04-16 20:33:06,812 DEBUG TRAIN | steps/sec 4.198| Batch 79/100 loss 216.367523 loss_att 160.154175 loss_ctc 347.531982 th_accuracy 0.450331 lr 8.6968e-04 grad_norm 177.419983 rank 0

Epoch 79 TRAIN:  73%|███████▎  | 100/137 [00:23<00:08,  4.61it/s, epoch=79, grad_norm=177.420, loss=216.3675, lr=8.70e-04]
Epoch 79 TRAIN:  73%|███████▎  | 100/137 [00:24<00:08,  4.61it/s, epoch=79, grad_norm=160.544, loss=201.5360, lr=8.70e-04]
Epoch 79 TRAIN:  73%|███████▎  | 100/137 [00:24<00:08,  4.61it/s, epoch=79, grad_norm=81.931, loss=198.3250, lr=8.70e-04] 
Epoch 79 TRAIN:  73%|███████▎  | 100/137 [00:24<00:08,  4.61it/s, epoch=79, grad_norm=125.406, loss=305.6862, lr=8.70e-04]
Epoch 79 TRAIN:  75%|███████▌  | 103/137 [00:24<00:07,  4.65it/s, epoch=79, grad_norm=125.406, loss=305.6862, lr=8.70e-04]
Epoch 79 TRAIN:  75%|███████▌  | 103/137 [00:24<00:07,  4.65it/s, epoch=79, grad_norm=161.039, loss=271.0551, lr=8.70e-04]
Epoch 79 TRAIN:  75%|███████▌  | 103/137 [00:25<00:07,  4.65it/s, epoch=79, grad_norm=178.799, loss=288.8156, lr=8.70e-04]
Epoch 79 TRAIN:  75%|███████▌  | 103/137 [00:25<00:07,  4.65it/s, epoch=79, grad_norm=101.833, loss=233.5429, lr=8.70e-04]
Epoch 79 TRAIN:  77%|███████▋  | 106/137 [00:25<00:06,  4.55it/s, epoch=79, grad_norm=101.833, loss=233.5429, lr=8.70e-04]
Epoch 79 TRAIN:  77%|███████▋  | 106/137 [00:25<00:06,  4.55it/s, epoch=79, grad_norm=125.950, loss=272.0274, lr=8.70e-04]
Epoch 79 TRAIN:  77%|███████▋  | 106/137 [00:25<00:06,  4.55it/s, epoch=79, grad_norm=116.671, loss=253.0384, lr=8.70e-04]
Epoch 79 TRAIN:  77%|███████▋  | 106/137 [00:25<00:06,  4.55it/s, epoch=79, grad_norm=91.183, loss=189.5305, lr=8.70e-04] 
Epoch 79 TRAIN:  80%|███████▉  | 109/137 [00:25<00:05,  4.77it/s, epoch=79, grad_norm=91.183, loss=189.5305, lr=8.70e-04]
Epoch 79 TRAIN:  80%|███████▉  | 109/137 [00:26<00:05,  4.77it/s, epoch=79, grad_norm=192.033, loss=270.7283, lr=8.70e-04]
Epoch 79 TRAIN:  80%|███████▉  | 109/137 [00:26<00:05,  4.77it/s, epoch=79, grad_norm=146.153, loss=267.4768, lr=8.71e-04]
Epoch 79 TRAIN:  80%|███████▉  | 109/137 [00:26<00:05,  4.77it/s, epoch=79, grad_norm=129.163, loss=261.9576, lr=8.71e-04]
Epoch 79 TRAIN:  82%|████████▏ | 112/137 [00:26<00:05,  4.75it/s, epoch=79, grad_norm=129.163, loss=261.9576, lr=8.71e-04]
Epoch 79 TRAIN:  82%|████████▏ | 112/137 [00:26<00:05,  4.75it/s, epoch=79, grad_norm=148.873, loss=232.4271, lr=8.71e-04]
Epoch 79 TRAIN:  82%|████████▏ | 112/137 [00:26<00:05,  4.75it/s, epoch=79, grad_norm=85.731, loss=242.1026, lr=8.71e-04] 
Epoch 79 TRAIN:  82%|████████▏ | 112/137 [00:27<00:05,  4.75it/s, epoch=79, grad_norm=164.702, loss=261.6413, lr=8.71e-04]
Epoch 79 TRAIN:  84%|████████▍ | 115/137 [00:27<00:04,  4.81it/s, epoch=79, grad_norm=164.702, loss=261.6413, lr=8.71e-04]
Epoch 79 TRAIN:  84%|████████▍ | 115/137 [00:27<00:04,  4.81it/s, epoch=79, grad_norm=255.919, loss=306.4274, lr=8.71e-04]
Epoch 79 TRAIN:  84%|████████▍ | 115/137 [00:27<00:04,  4.81it/s, epoch=79, grad_norm=298.526, loss=368.9008, lr=8.71e-04]
Epoch 79 TRAIN:  84%|████████▍ | 115/137 [00:27<00:04,  4.81it/s, epoch=79, grad_norm=202.687, loss=198.3307, lr=8.71e-04]
Epoch 79 TRAIN:  86%|████████▌ | 118/137 [00:27<00:04,  4.68it/s, epoch=79, grad_norm=202.687, loss=198.3307, lr=8.71e-04]
Epoch 79 TRAIN:  86%|████████▌ | 118/137 [00:27<00:04,  4.68it/s, epoch=79, grad_norm=213.573, loss=330.2499, lr=8.71e-04]
Epoch 79 TRAIN:  86%|████████▌ | 118/137 [00:28<00:04,  4.68it/s, epoch=79, grad_norm=157.374, loss=291.3194, lr=8.71e-04]
Epoch 79 TRAIN:  86%|████████▌ | 118/137 [00:28<00:04,  4.68it/s, epoch=79, grad_norm=300.456, loss=275.1367, lr=8.71e-04]
Epoch 79 TRAIN:  88%|████████▊ | 121/137 [00:28<00:03,  4.76it/s, epoch=79, grad_norm=300.456, loss=275.1367, lr=8.71e-04]
Epoch 79 TRAIN:  88%|████████▊ | 121/137 [00:28<00:03,  4.76it/s, epoch=79, grad_norm=132.329, loss=239.9577, lr=8.71e-04]
Epoch 79 TRAIN:  88%|████████▊ | 121/137 [00:28<00:03,  4.76it/s, epoch=79, grad_norm=112.910, loss=224.1633, lr=8.72e-04]
Epoch 79 TRAIN:  88%|████████▊ | 121/137 [00:28<00:03,  4.76it/s, epoch=79, grad_norm=259.426, loss=347.7794, lr=8.72e-04]
Epoch 79 TRAIN:  91%|█████████ | 124/137 [00:28<00:02,  4.70it/s, epoch=79, grad_norm=259.426, loss=347.7794, lr=8.72e-04]
Epoch 79 TRAIN:  91%|█████████ | 124/137 [00:29<00:02,  4.70it/s, epoch=79, grad_norm=155.418, loss=264.2598, lr=8.72e-04]
Epoch 79 TRAIN:  91%|█████████ | 124/137 [00:29<00:02,  4.70it/s, epoch=79, grad_norm=411.218, loss=316.1144, lr=8.72e-04]
Epoch 79 TRAIN:  91%|█████████ | 124/137 [00:29<00:02,  4.70it/s, epoch=79, grad_norm=173.340, loss=318.9862, lr=8.72e-04]
Epoch 79 TRAIN:  93%|█████████▎| 127/137 [00:29<00:02,  4.84it/s, epoch=79, grad_norm=173.340, loss=318.9862, lr=8.72e-04]
Epoch 79 TRAIN:  93%|█████████▎| 127/137 [00:29<00:02,  4.84it/s, epoch=79, grad_norm=140.561, loss=235.9434, lr=8.72e-04]
Epoch 79 TRAIN:  93%|█████████▎| 127/137 [00:30<00:02,  4.84it/s, epoch=79, grad_norm=138.828, loss=274.8345, lr=8.72e-04]
Epoch 79 TRAIN:  93%|█████████▎| 127/137 [00:30<00:02,  4.84it/s, epoch=79, grad_norm=158.351, loss=324.4964, lr=8.72e-04]
Epoch 79 TRAIN:  95%|█████████▍| 130/137 [00:30<00:01,  4.61it/s, epoch=79, grad_norm=158.351, loss=324.4964, lr=8.72e-04]
Epoch 79 TRAIN:  95%|█████████▍| 130/137 [00:30<00:01,  4.61it/s, epoch=79, grad_norm=143.937, loss=229.9480, lr=8.72e-04]
Epoch 79 TRAIN:  95%|█████████▍| 130/137 [00:30<00:01,  4.61it/s, epoch=79, grad_norm=158.327, loss=281.6421, lr=8.72e-04]
Epoch 79 TRAIN:  95%|█████████▍| 130/137 [00:30<00:01,  4.61it/s, epoch=79, grad_norm=163.507, loss=315.4657, lr=8.72e-04]
Epoch 79 TRAIN:  97%|█████████▋| 133/137 [00:30<00:00,  4.74it/s, epoch=79, grad_norm=163.507, loss=315.4657, lr=8.72e-04]
Epoch 79 TRAIN:  97%|█████████▋| 133/137 [00:31<00:00,  4.74it/s, epoch=79, grad_norm=152.120, loss=303.1897, lr=8.72e-04]
Epoch 79 TRAIN:  97%|█████████▋| 133/137 [00:31<00:00,  4.74it/s, epoch=79, grad_norm=184.376, loss=327.6329, lr=8.72e-04]
Epoch 79 TRAIN:  97%|█████████▋| 133/137 [00:31<00:00,  4.74it/s, epoch=79, grad_norm=217.700, loss=298.3490, lr=8.73e-04]
Epoch 79 TRAIN:  99%|█████████▉| 136/137 [00:31<00:00,  4.62it/s, epoch=79, grad_norm=217.700, loss=298.3490, lr=8.73e-04]
Epoch 79 TRAIN:  99%|█████████▉| 136/137 [00:31<00:00,  4.62it/s, epoch=79, grad_norm=267.182, loss=319.8832, lr=8.73e-04]
Epoch 79 TRAIN: 100%|██████████| 137/137 [00:31<00:00,  4.32it/s, epoch=79, grad_norm=267.182, loss=319.8832, lr=8.73e-04]

CV:   0%|          | 0/6 [00:00<?, ?it/s]/home/rookie/Documents/batuka/projects/wenet/wenet/wenet/utils/train_utils.py:689: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  amp_autocast(enabled=dtype is not None,

CV:  17%|█▋        | 1/6 [00:00<00:03,  1.33it/s]
                                                 
2026-04-16 20:33:15,461 INFO Epoch 79 Step 10908 CV info lr 8.7264e-04 cv_loss 457.5604248046875 rank 0 acc 0.37755684554576874
2026-04-16 20:33:15,463 INFO [Rank 0] Checkpoint: save to checkpoint exp/conformer_mongolian/epoch_79.pt
2026-04-16 20:33:15,574 INFO Epoch 80 Step 10908 TRAIN info lr 8.7264e-04 rank 0
2026-04-16 20:33:15,595 INFO using accumulate grad, new batch size is 1 times larger than before

Epoch 80 TRAIN:   0%|          | 0/137 [00:00<?, ?it/s]
Epoch 80 TRAIN:   0%|          | 0/137 [00:03<?, ?it/s, epoch=80, grad_norm=523.412, loss=398.6246, lr=8.73e-04]
Epoch 80 TRAIN:   1%|          | 1/137 [00:03<08:55,  3.94s/it, epoch=80, grad_norm=523.412, loss=398.6246, lr=8.73e-04]
Epoch 80 TRAIN:   1%|          | 1/137 [00:04<08:55,  3.94s/it, epoch=80, grad_norm=460.889, loss=368.6394, lr=8.73e-04]
Epoch 80 TRAIN:   1%|          | 1/137 [00:04<08:55,  3.94s/it, epoch=80, grad_norm=425.569, loss=329.4935, lr=8.73e-04]
Epoch 80 TRAIN:   1%|          | 1/137 [00:04<08:55,  3.94s/it, epoch=80, grad_norm=320.173, loss=216.9715, lr=8.73e-04]
Epoch 80 TRAIN:   1%|          | 1/137 [00:04<08:55,  3.94s/it, epoch=80, grad_norm=368.627, loss=288.5592, lr=8.73e-04]
Epoch 80 TRAIN:   1%|          | 1/137 [00:04<08:55,  3.94s/it, epoch=80, grad_norm=317.984, loss=339.8242, lr=8.73e-04]
Epoch 80 TRAIN:   1%|          | 1/137 [00:04<08:55,  3.94s/it, epoch=80, grad_norm=324.511, loss=357.0408, lr=8.73e-04]
Epoch 80 TRAIN:   1%|          | 1/137 [00:04<08:55,  3.94s/it, epoch=80, grad_norm=158.984, loss=217.8120, lr=8.73e-04]
Epoch 80 TRAIN:   6%|▌         | 8/137 [00:04<00:55,  2.33it/s, epoch=80, grad_norm=158.984, loss=217.8120, lr=8.73e-04]
Epoch 80 TRAIN:   6%|▌         | 8/137 [00:04<00:55,  2.33it/s, epoch=80, grad_norm=132.771, loss=284.2726, lr=8.73e-04]
Epoch 80 TRAIN:   6%|▌         | 8/137 [00:04<00:55,  2.33it/s, epoch=80, grad_norm=124.624, loss=329.9930, lr=8.73e-04]
Epoch 80 TRAIN:   6%|▌         | 8/137 [00:04<00:55,  2.33it/s, epoch=80, grad_norm=97.538, loss=289.2304, lr=8.74e-04] 
Epoch 80 TRAIN:   6%|▌         | 8/137 [00:05<00:55,  2.33it/s, epoch=80, grad_norm=187.411, loss=327.0182, lr=8.74e-04]
Epoch 80 TRAIN:   9%|▉         | 12/137 [00:05<00:39,  3.17it/s, epoch=80, grad_norm=187.411, loss=327.0182, lr=8.74e-04]
Epoch 80 TRAIN:   9%|▉         | 12/137 [00:05<00:39,  3.17it/s, epoch=80, grad_norm=115.772, loss=294.6829, lr=8.74e-04]
Epoch 80 TRAIN:   9%|▉         | 12/137 [00:05<00:39,  3.17it/s, epoch=80, grad_norm=130.880, loss=286.7810, lr=8.74e-04]
Epoch 80 TRAIN:   9%|▉         | 12/137 [00:05<00:39,  3.17it/s, epoch=80, grad_norm=186.976, loss=240.4168, lr=8.74e-04]
Epoch 80 TRAIN:   9%|▉         | 12/137 [00:05<00:39,  3.17it/s, epoch=80, grad_norm=107.074, loss=281.8375, lr=8.74e-04]
Epoch 80 TRAIN:  12%|█▏        | 16/137 [00:05<00:31,  3.87it/s, epoch=80, grad_norm=107.074, loss=281.8375, lr=8.74e-04]
Epoch 80 TRAIN:  12%|█▏        | 16/137 [00:06<00:31,  3.87it/s, epoch=80, grad_norm=100.300, loss=319.6859, lr=8.74e-04]
Epoch 80 TRAIN:  12%|█▏        | 16/137 [00:06<00:31,  3.87it/s, epoch=80, grad_norm=150.321, loss=281.2765, lr=8.74e-04]
Epoch 80 TRAIN:  12%|█▏        | 16/137 [00:06<00:31,  3.87it/s, epoch=80, grad_norm=102.114, loss=275.9712, lr=8.74e-04]
Epoch 80 TRAIN:  12%|█▏        | 16/137 [00:06<00:31,  3.87it/s, epoch=80, grad_norm=121.750, loss=252.4365, lr=8.74e-04]
Epoch 80 TRAIN:  15%|█▍        | 20/137 [00:06<00:27,  4.26it/s, epoch=80, grad_norm=121.750, loss=252.4365, lr=8.74e-04]
Epoch 80 TRAIN:  15%|█▍        | 20/137 [00:06<00:27,  4.26it/s, epoch=80, grad_norm=90.506, loss=292.7533, lr=8.74e-04] 
Epoch 80 TRAIN:  15%|█▍        | 20/137 [00:06<00:27,  4.26it/s, epoch=80, grad_norm=101.256, loss=276.2929, lr=8.74e-04]
Epoch 80 TRAIN:  15%|█▍        | 20/137 [00:07<00:27,  4.26it/s, epoch=80, grad_norm=124.868, loss=275.8327, lr=8.74e-04]
Epoch 80 TRAIN:  15%|█▍        | 20/137 [00:07<00:27,  4.26it/s, epoch=80, grad_norm=95.668, loss=265.3566, lr=8.75e-04] 
Epoch 80 TRAIN:  18%|█▊        | 24/137 [00:07<00:25,  4.43it/s, epoch=80, grad_norm=95.668, loss=265.3566, lr=8.75e-04]
Epoch 80 TRAIN:  18%|█▊        | 24/137 [00:07<00:25,  4.43it/s, epoch=80, grad_norm=92.372, loss=274.7999, lr=8.75e-04]
Epoch 80 TRAIN:  18%|█▊        | 24/137 [00:07<00:25,  4.43it/s, epoch=80, grad_norm=96.053, loss=284.3361, lr=8.75e-04]
Epoch 80 TRAIN:  18%|█▊        | 24/137 [00:07<00:25,  4.43it/s, epoch=80, grad_norm=102.265, loss=272.4337, lr=8.75e-04]
Epoch 80 TRAIN:  18%|█▊        | 24/137 [00:08<00:25,  4.43it/s, epoch=80, grad_norm=103.227, loss=280.8582, lr=8.75e-04]
Epoch 80 TRAIN:  20%|██        | 28/137 [00:08<00:22,  4.84it/s, epoch=80, grad_norm=103.227, loss=280.8582, lr=8.75e-04]
Epoch 80 TRAIN:  20%|██        | 28/137 [00:08<00:22,  4.84it/s, epoch=80, grad_norm=99.861, loss=271.3959, lr=8.75e-04] 
Epoch 80 TRAIN:  20%|██        | 28/137 [00:08<00:22,  4.84it/s, epoch=80, grad_norm=93.282, loss=264.6199, lr=8.75e-04]
Epoch 80 TRAIN:  20%|██        | 28/137 [00:08<00:22,  4.84it/s, epoch=80, grad_norm=89.862, loss=291.0805, lr=8.75e-04]
Epoch 80 TRAIN:  23%|██▎       | 31/137 [00:08<00:23,  4.61it/s, epoch=80, grad_norm=89.862, loss=291.0805, lr=8.75e-04]
Epoch 80 TRAIN:  23%|██▎       | 31/137 [00:08<00:23,  4.61it/s, epoch=80, grad_norm=108.945, loss=262.4594, lr=8.75e-04]
Epoch 80 TRAIN:  23%|██▎       | 31/137 [00:09<00:23,  4.61it/s, epoch=80, grad_norm=123.546, loss=230.2162, lr=8.75e-04]
Epoch 80 TRAIN:  23%|██▎       | 31/137 [00:09<00:23,  4.61it/s, epoch=80, grad_norm=91.523, loss=278.5110, lr=8.75e-04] 
Epoch 80 TRAIN:  25%|██▍       | 34/137 [00:09<00:21,  4.73it/s, epoch=80, grad_norm=91.523, loss=278.5110, lr=8.75e-04]
Epoch 80 TRAIN:  25%|██▍       | 34/137 [00:09<00:21,  4.73it/s, epoch=80, grad_norm=81.300, loss=258.3807, lr=8.75e-04]
Epoch 80 TRAIN:  25%|██▍       | 34/137 [00:09<00:21,  4.73it/s, epoch=80, grad_norm=96.308, loss=276.2043, lr=8.76e-04]
Epoch 80 TRAIN:  25%|██▍       | 34/137 [00:10<00:21,  4.73it/s, epoch=80, grad_norm=94.326, loss=268.7203, lr=8.76e-04]
Epoch 80 TRAIN:  27%|██▋       | 37/137 [00:10<00:21,  4.63it/s, epoch=80, grad_norm=94.326, loss=268.7203, lr=8.76e-04]
Epoch 80 TRAIN:  27%|██▋       | 37/137 [00:10<00:21,  4.63it/s, epoch=80, grad_norm=159.909, loss=293.0595, lr=8.76e-04]
Epoch 80 TRAIN:  27%|██▋       | 37/137 [00:10<00:21,  4.63it/s, epoch=80, grad_norm=167.920, loss=298.9401, lr=8.76e-04]
Epoch 80 TRAIN:  27%|██▋       | 37/137 [00:10<00:21,  4.63it/s, epoch=80, grad_norm=90.267, loss=227.4686, lr=8.76e-04] 
Epoch 80 TRAIN:  29%|██▉       | 40/137 [00:10<00:19,  4.90it/s, epoch=80, grad_norm=90.267, loss=227.4686, lr=8.76e-04]
Epoch 80 TRAIN:  29%|██▉       | 40/137 [00:10<00:19,  4.90it/s, epoch=80, grad_norm=142.662, loss=306.9098, lr=8.76e-04]
Epoch 80 TRAIN:  29%|██▉       | 40/137 [00:11<00:19,  4.90it/s, epoch=80, grad_norm=124.578, loss=260.5799, lr=8.76e-04]
Epoch 80 TRAIN:  29%|██▉       | 40/137 [00:11<00:19,  4.90it/s, epoch=80, grad_norm=75.656, loss=234.1588, lr=8.76e-04] 
Epoch 80 TRAIN:  31%|███▏      | 43/137 [00:11<00:19,  4.71it/s, epoch=80, grad_norm=75.656, loss=234.1588, lr=8.76e-04]
Epoch 80 TRAIN:  31%|███▏      | 43/137 [00:11<00:19,  4.71it/s, epoch=80, grad_norm=172.055, loss=301.1942, lr=8.76e-04]
Epoch 80 TRAIN:  31%|███▏      | 43/137 [00:11<00:19,  4.71it/s, epoch=80, grad_norm=149.859, loss=298.1986, lr=8.76e-04]
Epoch 80 TRAIN:  31%|███▏      | 43/137 [00:11<00:19,  4.71it/s, epoch=80, grad_norm=134.262, loss=246.9639, lr=8.76e-04]
Epoch 80 TRAIN:  34%|███▎      | 46/137 [00:11<00:18,  4.82it/s, epoch=80, grad_norm=134.262, loss=246.9639, lr=8.76e-04]
Epoch 80 TRAIN:  34%|███▎      | 46/137 [00:12<00:18,  4.82it/s, epoch=80, grad_norm=165.166, loss=254.3382, lr=8.76e-04]
Epoch 80 TRAIN:  34%|███▎      | 46/137 [00:12<00:18,  4.82it/s, epoch=80, grad_norm=108.219, loss=301.7516, lr=8.76e-04]
Epoch 80 TRAIN:  34%|███▎      | 46/137 [00:12<00:18,  4.82it/s, epoch=80, grad_norm=102.515, loss=274.6864, lr=8.77e-04]
Epoch 80 TRAIN:  36%|███▌      | 49/137 [00:12<00:18,  4.77it/s, epoch=80, grad_norm=102.515, loss=274.6864, lr=8.77e-04]
Epoch 80 TRAIN:  36%|███▌      | 49/137 [00:12<00:18,  4.77it/s, epoch=80, grad_norm=123.810, loss=296.6857, lr=8.77e-04]
Epoch 80 TRAIN:  36%|███▌      | 49/137 [00:12<00:18,  4.77it/s, epoch=80, grad_norm=118.025, loss=260.9560, lr=8.77e-04]
Epoch 80 TRAIN:  36%|███▌      | 49/137 [00:13<00:18,  4.77it/s, epoch=80, grad_norm=177.353, loss=213.4922, lr=8.77e-04]
Epoch 80 TRAIN:  38%|███▊      | 52/137 [00:13<00:18,  4.70it/s, epoch=80, grad_norm=177.353, loss=213.4922, lr=8.77e-04]
Epoch 80 TRAIN:  38%|███▊      | 52/137 [00:13<00:18,  4.70it/s, epoch=80, grad_norm=115.255, loss=300.7293, lr=8.77e-04]
Epoch 80 TRAIN:  38%|███▊      | 52/137 [00:13<00:18,  4.70it/s, epoch=80, grad_norm=74.471, loss=264.5695, lr=8.77e-04] 
Epoch 80 TRAIN:  38%|███▊      | 52/137 [00:13<00:18,  4.70it/s, epoch=80, grad_norm=222.271, loss=295.8210, lr=8.77e-04]
Epoch 80 TRAIN:  40%|████      | 55/137 [00:13<00:17,  4.76it/s, epoch=80, grad_norm=222.271, loss=295.8210, lr=8.77e-04]
Epoch 80 TRAIN:  40%|████      | 55/137 [00:14<00:17,  4.76it/s, epoch=80, grad_norm=114.742, loss=295.2581, lr=8.77e-04]
Epoch 80 TRAIN:  40%|████      | 55/137 [00:14<00:17,  4.76it/s, epoch=80, grad_norm=141.922, loss=307.8387, lr=8.77e-04]
Epoch 80 TRAIN:  40%|████      | 55/137 [00:14<00:17,  4.76it/s, epoch=80, grad_norm=95.016, loss=280.0332, lr=8.77e-04] 
Epoch 80 TRAIN:  42%|████▏     | 58/137 [00:14<00:16,  4.74it/s, epoch=80, grad_norm=95.016, loss=280.0332, lr=8.77e-04]
Epoch 80 TRAIN:  42%|████▏     | 58/137 [00:14<00:16,  4.74it/s, epoch=80, grad_norm=104.020, loss=281.5143, lr=8.77e-04]
Epoch 80 TRAIN:  42%|████▏     | 58/137 [00:14<00:16,  4.74it/s, epoch=80, grad_norm=171.255, loss=268.0649, lr=8.77e-04]
Epoch 80 TRAIN:  42%|████▏     | 58/137 [00:15<00:16,  4.74it/s, epoch=80, grad_norm=124.866, loss=251.9173, lr=8.78e-04]
Epoch 80 TRAIN:  45%|████▍     | 61/137 [00:15<00:16,  4.48it/s, epoch=80, grad_norm=124.866, loss=251.9173, lr=8.78e-04]
Epoch 80 TRAIN:  45%|████▍     | 61/137 [00:15<00:16,  4.48it/s, epoch=80, grad_norm=143.722, loss=261.4800, lr=8.78e-04]
Epoch 80 TRAIN:  45%|████▍     | 61/137 [00:15<00:16,  4.48it/s, epoch=80, grad_norm=112.436, loss=265.4742, lr=8.78e-04]
Epoch 80 TRAIN:  45%|████▍     | 61/137 [00:15<00:16,  4.48it/s, epoch=80, grad_norm=74.120, loss=262.3592, lr=8.78e-04] 
Epoch 80 TRAIN:  47%|████▋     | 64/137 [00:15<00:15,  4.69it/s, epoch=80, grad_norm=74.120, loss=262.3592, lr=8.78e-04]
Epoch 80 TRAIN:  47%|████▋     | 64/137 [00:16<00:15,  4.69it/s, epoch=80, grad_norm=98.161, loss=285.3792, lr=8.78e-04]
Epoch 80 TRAIN:  47%|████▋     | 64/137 [00:16<00:15,  4.69it/s, epoch=80, grad_norm=149.117, loss=318.1035, lr=8.78e-04]
Epoch 80 TRAIN:  47%|████▋     | 64/137 [00:16<00:15,  4.69it/s, epoch=80, grad_norm=89.568, loss=256.0101, lr=8.78e-04] 
Epoch 80 TRAIN:  49%|████▉     | 67/137 [00:16<00:15,  4.53it/s, epoch=80, grad_norm=89.568, loss=256.0101, lr=8.78e-04]
Epoch 80 TRAIN:  49%|████▉     | 67/137 [00:16<00:15,  4.53it/s, epoch=80, grad_norm=105.303, loss=305.7399, lr=8.78e-04]
Epoch 80 TRAIN:  49%|████▉     | 67/137 [00:16<00:15,  4.53it/s, epoch=80, grad_norm=111.071, loss=278.2020, lr=8.78e-04]
Epoch 80 TRAIN:  49%|████▉     | 67/137 [00:17<00:15,  4.53it/s, epoch=80, grad_norm=128.674, loss=244.6478, lr=8.78e-04]
Epoch 80 TRAIN:  51%|█████     | 70/137 [00:17<00:14,  4.58it/s, epoch=80, grad_norm=128.674, loss=244.6478, lr=8.78e-04]
Epoch 80 TRAIN:  51%|█████     | 70/137 [00:17<00:14,  4.58it/s, epoch=80, grad_norm=186.403, loss=315.9221, lr=8.78e-04]
Epoch 80 TRAIN:  51%|█████     | 70/137 [00:17<00:14,  4.58it/s, epoch=80, grad_norm=171.021, loss=294.2402, lr=8.78e-04]
Epoch 80 TRAIN:  51%|█████     | 70/137 [00:17<00:14,  4.58it/s, epoch=80, grad_norm=114.875, loss=295.6500, lr=8.78e-04]
Epoch 80 TRAIN:  53%|█████▎    | 73/137 [00:17<00:13,  4.64it/s, epoch=80, grad_norm=114.875, loss=295.6500, lr=8.78e-04]
Epoch 80 TRAIN:  53%|█████▎    | 73/137 [00:17<00:13,  4.64it/s, epoch=80, grad_norm=125.990, loss=239.4502, lr=8.79e-04]
Epoch 80 TRAIN:  53%|█████▎    | 73/137 [00:18<00:13,  4.64it/s, epoch=80, grad_norm=165.963, loss=239.7262, lr=8.79e-04]
Epoch 80 TRAIN:  53%|█████▎    | 73/137 [00:18<00:13,  4.64it/s, epoch=80, grad_norm=150.335, loss=299.0319, lr=8.79e-04]
Epoch 80 TRAIN:  55%|█████▌    | 76/137 [00:18<00:13,  4.51it/s, epoch=80, grad_norm=150.335, loss=299.0319, lr=8.79e-04]
Epoch 80 TRAIN:  55%|█████▌    | 76/137 [00:18<00:13,  4.51it/s, epoch=80, grad_norm=146.221, loss=295.7458, lr=8.79e-04]
Epoch 80 TRAIN:  55%|█████▌    | 76/137 [00:18<00:13,  4.51it/s, epoch=80, grad_norm=146.070, loss=225.0937, lr=8.79e-04]
Epoch 80 TRAIN:  55%|█████▌    | 76/137 [00:19<00:13,  4.51it/s, epoch=80, grad_norm=126.496, loss=297.6371, lr=8.79e-04]
Epoch 80 TRAIN:  58%|█████▊    | 79/137 [00:19<00:13,  4.26it/s, epoch=80, grad_norm=126.496, loss=297.6371, lr=8.79e-04]
Epoch 80 TRAIN:  58%|█████▊    | 79/137 [00:19<00:13,  4.26it/s, epoch=80, grad_norm=86.924, loss=251.6584, lr=8.79e-04] 
Epoch 80 TRAIN:  58%|█████▊    | 79/137 [00:19<00:13,  4.26it/s, epoch=80, grad_norm=112.300, loss=251.5265, lr=8.79e-04]
Epoch 80 TRAIN:  58%|█████▊    | 79/137 [00:19<00:13,  4.26it/s, epoch=80, grad_norm=118.453, loss=257.3756, lr=8.79e-04]
Epoch 80 TRAIN:  60%|█████▉    | 82/137 [00:19<00:13,  4.21it/s, epoch=80, grad_norm=118.453, loss=257.3756, lr=8.79e-04]
Epoch 80 TRAIN:  60%|█████▉    | 82/137 [00:20<00:13,  4.21it/s, epoch=80, grad_norm=119.117, loss=283.9064, lr=8.79e-04]
Epoch 80 TRAIN:  60%|█████▉    | 82/137 [00:20<00:13,  4.21it/s, epoch=80, grad_norm=149.320, loss=220.9042, lr=8.79e-04]
Epoch 80 TRAIN:  60%|█████▉    | 82/137 [00:20<00:13,  4.21it/s, epoch=80, grad_norm=109.931, loss=269.3470, lr=8.79e-04]
Epoch 80 TRAIN:  62%|██████▏   | 85/137 [00:20<00:11,  4.49it/s, epoch=80, grad_norm=109.931, loss=269.3470, lr=8.79e-04]
Epoch 80 TRAIN:  62%|██████▏   | 85/137 [00:20<00:11,  4.49it/s, epoch=80, grad_norm=111.162, loss=278.9267, lr=8.80e-04]
Epoch 80 TRAIN:  62%|██████▏   | 85/137 [00:21<00:11,  4.49it/s, epoch=80, grad_norm=124.622, loss=255.4116, lr=8.80e-04]
Epoch 80 TRAIN:  62%|██████▏   | 85/137 [00:21<00:11,  4.49it/s, epoch=80, grad_norm=162.107, loss=288.5396, lr=8.80e-04]
Epoch 80 TRAIN:  64%|██████▍   | 88/137 [00:21<00:11,  4.10it/s, epoch=80, grad_norm=162.107, loss=288.5396, lr=8.80e-04]
Epoch 80 TRAIN:  64%|██████▍   | 88/137 [00:21<00:11,  4.10it/s, epoch=80, grad_norm=118.886, loss=245.8412, lr=8.80e-04]
Epoch 80 TRAIN:  64%|██████▍   | 88/137 [00:21<00:11,  4.10it/s, epoch=80, grad_norm=135.725, loss=237.5291, lr=8.80e-04]
Epoch 80 TRAIN:  64%|██████▍   | 88/137 [00:21<00:11,  4.10it/s, epoch=80, grad_norm=116.622, loss=292.5833, lr=8.80e-04]
Epoch 80 TRAIN:  66%|██████▋   | 91/137 [00:21<00:10,  4.38it/s, epoch=80, grad_norm=116.622, loss=292.5833, lr=8.80e-04]
Epoch 80 TRAIN:  66%|██████▋   | 91/137 [00:22<00:10,  4.38it/s, epoch=80, grad_norm=224.389, loss=252.0408, lr=8.80e-04]
Epoch 80 TRAIN:  66%|██████▋   | 91/137 [00:22<00:10,  4.38it/s, epoch=80, grad_norm=176.339, loss=291.6544, lr=8.80e-04]
Epoch 80 TRAIN:  66%|██████▋   | 91/137 [00:22<00:10,  4.38it/s, epoch=80, grad_norm=194.440, loss=272.0757, lr=8.80e-04]
Epoch 80 TRAIN:  69%|██████▊   | 94/137 [00:22<00:09,  4.31it/s, epoch=80, grad_norm=194.440, loss=272.0757, lr=8.80e-04]
Epoch 80 TRAIN:  69%|██████▊   | 94/137 [00:22<00:09,  4.31it/s, epoch=80, grad_norm=134.400, loss=228.8147, lr=8.80e-04]
Epoch 80 TRAIN:  69%|██████▊   | 94/137 [00:23<00:09,  4.31it/s, epoch=80, grad_norm=156.787, loss=168.1141, lr=8.80e-04]
Epoch 80 TRAIN:  69%|██████▊   | 94/137 [00:23<00:09,  4.31it/s, epoch=80, grad_norm=106.535, loss=249.1707, lr=8.80e-04]
Epoch 80 TRAIN:  71%|███████   | 97/137 [00:23<00:08,  4.52it/s, epoch=80, grad_norm=106.535, loss=249.1707, lr=8.80e-04]
Epoch 80 TRAIN:  71%|███████   | 97/137 [00:23<00:08,  4.52it/s, epoch=80, grad_norm=122.066, loss=186.6653, lr=8.80e-04]
Epoch 80 TRAIN:  71%|███████   | 97/137 [00:23<00:08,  4.52it/s, epoch=80, grad_norm=118.304, loss=254.8321, lr=8.81e-04]
Epoch 80 TRAIN:  71%|███████   | 97/137 [00:24<00:08,  4.52it/s, epoch=80, grad_norm=266.805, loss=275.5989, lr=8.81e-04]2026-04-16 20:33:39,634 DEBUG TRAIN | steps/sec 4.174| Batch 80/100 loss 275.598938 loss_att 218.678711 loss_ctc 408.412842 th_accuracy 0.422472 lr 8.8064e-04 grad_norm 266.804840 rank 0

Epoch 80 TRAIN:  73%|███████▎  | 100/137 [00:24<00:08,  4.38it/s, epoch=80, grad_norm=266.805, loss=275.5989, lr=8.81e-04]
Epoch 80 TRAIN:  73%|███████▎  | 100/137 [00:24<00:08,  4.38it/s, epoch=80, grad_norm=85.009, loss=205.9740, lr=8.81e-04] 
Epoch 80 TRAIN:  73%|███████▎  | 100/137 [00:24<00:08,  4.38it/s, epoch=80, grad_norm=151.389, loss=233.6016, lr=8.81e-04]
Epoch 80 TRAIN:  73%|███████▎  | 100/137 [00:24<00:08,  4.38it/s, epoch=80, grad_norm=113.163, loss=221.6313, lr=8.81e-04]
Epoch 80 TRAIN:  75%|███████▌  | 103/137 [00:24<00:07,  4.47it/s, epoch=80, grad_norm=113.163, loss=221.6313, lr=8.81e-04]
Epoch 80 TRAIN:  75%|███████▌  | 103/137 [00:24<00:07,  4.47it/s, epoch=80, grad_norm=103.496, loss=241.8213, lr=8.81e-04]
Epoch 80 TRAIN:  75%|███████▌  | 103/137 [00:25<00:07,  4.47it/s, epoch=80, grad_norm=101.986, loss=268.7825, lr=8.81e-04]
Epoch 80 TRAIN:  75%|███████▌  | 103/137 [00:25<00:07,  4.47it/s, epoch=80, grad_norm=149.867, loss=290.8657, lr=8.81e-04]
Epoch 80 TRAIN:  77%|███████▋  | 106/137 [00:25<00:06,  4.48it/s, epoch=80, grad_norm=149.867, loss=290.8657, lr=8.81e-04]
Epoch 80 TRAIN:  77%|███████▋  | 106/137 [00:25<00:06,  4.48it/s, epoch=80, grad_norm=119.903, loss=284.7901, lr=8.81e-04]
Epoch 80 TRAIN:  77%|███████▋  | 106/137 [00:25<00:06,  4.48it/s, epoch=80, grad_norm=262.943, loss=270.9128, lr=8.81e-04]
Epoch 80 TRAIN:  77%|███████▋  | 106/137 [00:26<00:06,  4.48it/s, epoch=80, grad_norm=180.756, loss=245.3957, lr=8.81e-04]
Epoch 80 TRAIN:  80%|███████▉  | 109/137 [00:26<00:06,  4.45it/s, epoch=80, grad_norm=180.756, loss=245.3957, lr=8.81e-04]
Epoch 80 TRAIN:  80%|███████▉  | 109/137 [00:26<00:06,  4.45it/s, epoch=80, grad_norm=223.884, loss=347.4626, lr=8.81e-04]
Epoch 80 TRAIN:  80%|███████▉  | 109/137 [00:26<00:06,  4.45it/s, epoch=80, grad_norm=132.573, loss=281.0300, lr=8.82e-04]
Epoch 80 TRAIN:  80%|███████▉  | 109/137 [00:26<00:06,  4.45it/s, epoch=80, grad_norm=119.798, loss=216.9930, lr=8.82e-04]
Epoch 80 TRAIN:  82%|████████▏ | 112/137 [00:26<00:05,  4.62it/s, epoch=80, grad_norm=119.798, loss=216.9930, lr=8.82e-04]
Epoch 80 TRAIN:  82%|████████▏ | 112/137 [00:26<00:05,  4.62it/s, epoch=80, grad_norm=231.664, loss=349.0858, lr=8.82e-04]
Epoch 80 TRAIN:  82%|████████▏ | 112/137 [00:27<00:05,  4.62it/s, epoch=80, grad_norm=144.887, loss=279.5606, lr=8.82e-04]
Epoch 80 TRAIN:  82%|████████▏ | 112/137 [00:27<00:05,  4.62it/s, epoch=80, grad_norm=118.698, loss=209.5206, lr=8.82e-04]
Epoch 80 TRAIN:  84%|████████▍ | 115/137 [00:27<00:04,  4.52it/s, epoch=80, grad_norm=118.698, loss=209.5206, lr=8.82e-04]
Epoch 80 TRAIN:  84%|████████▍ | 115/137 [00:27<00:04,  4.52it/s, epoch=80, grad_norm=132.345, loss=283.9150, lr=8.82e-04]
Epoch 80 TRAIN:  84%|████████▍ | 115/137 [00:27<00:04,  4.52it/s, epoch=80, grad_norm=157.030, loss=296.9045, lr=8.82e-04]
Epoch 80 TRAIN:  84%|████████▍ | 115/137 [00:28<00:04,  4.52it/s, epoch=80, grad_norm=272.777, loss=363.5593, lr=8.82e-04]
Epoch 80 TRAIN:  86%|████████▌ | 118/137 [00:28<00:04,  4.44it/s, epoch=80, grad_norm=272.777, loss=363.5593, lr=8.82e-04]
Epoch 80 TRAIN:  86%|████████▌ | 118/137 [00:28<00:04,  4.44it/s, epoch=80, grad_norm=139.670, loss=278.0721, lr=8.82e-04]
Epoch 80 TRAIN:  86%|████████▌ | 118/137 [00:28<00:04,  4.44it/s, epoch=80, grad_norm=122.251, loss=209.8558, lr=8.82e-04]
Epoch 80 TRAIN:  86%|████████▌ | 118/137 [00:28<00:04,  4.44it/s, epoch=80, grad_norm=250.763, loss=335.7346, lr=8.82e-04]
Epoch 80 TRAIN:  88%|████████▊ | 121/137 [00:28<00:03,  4.65it/s, epoch=80, grad_norm=250.763, loss=335.7346, lr=8.82e-04]
Epoch 80 TRAIN:  88%|████████▊ | 121/137 [00:28<00:03,  4.65it/s, epoch=80, grad_norm=131.184, loss=339.2261, lr=8.82e-04]
Epoch 80 TRAIN:  88%|████████▊ | 121/137 [00:28<00:03,  4.65it/s, epoch=80, grad_norm=248.784, loss=206.9040, lr=8.82e-04]
Epoch 80 TRAIN:  88%|████████▊ | 121/137 [00:29<00:03,  4.65it/s, epoch=80, grad_norm=159.811, loss=264.7384, lr=8.83e-04]
Epoch 80 TRAIN:  91%|█████████ | 124/137 [00:29<00:02,  4.82it/s, epoch=80, grad_norm=159.811, loss=264.7384, lr=8.83e-04]
Epoch 80 TRAIN:  91%|█████████ | 124/137 [00:29<00:02,  4.82it/s, epoch=80, grad_norm=116.814, loss=280.1513, lr=8.83e-04]
Epoch 80 TRAIN:  91%|█████████ | 124/137 [00:29<00:02,  4.82it/s, epoch=80, grad_norm=276.873, loss=310.2603, lr=8.83e-04]
Epoch 80 TRAIN:  91%|█████████ | 124/137 [00:29<00:02,  4.82it/s, epoch=80, grad_norm=132.367, loss=200.9514, lr=8.83e-04]
Epoch 80 TRAIN:  93%|█████████▎| 127/137 [00:29<00:02,  4.81it/s, epoch=80, grad_norm=132.367, loss=200.9514, lr=8.83e-04]
Epoch 80 TRAIN:  93%|█████████▎| 127/137 [00:30<00:02,  4.81it/s, epoch=80, grad_norm=132.274, loss=292.2854, lr=8.83e-04]
Epoch 80 TRAIN:  93%|█████████▎| 127/137 [00:30<00:02,  4.81it/s, epoch=80, grad_norm=225.678, loss=349.4826, lr=8.83e-04]
Epoch 80 TRAIN:  93%|█████████▎| 127/137 [00:30<00:02,  4.81it/s, epoch=80, grad_norm=196.957, loss=278.1183, lr=8.83e-04]
Epoch 80 TRAIN:  95%|█████████▍| 130/137 [00:30<00:01,  4.95it/s, epoch=80, grad_norm=196.957, loss=278.1183, lr=8.83e-04]
Epoch 80 TRAIN:  95%|█████████▍| 130/137 [00:30<00:01,  4.95it/s, epoch=80, grad_norm=210.707, loss=325.3883, lr=8.83e-04]
Epoch 80 TRAIN:  95%|█████████▍| 130/137 [00:30<00:01,  4.95it/s, epoch=80, grad_norm=180.484, loss=276.9120, lr=8.83e-04]
Epoch 80 TRAIN:  95%|█████████▍| 130/137 [00:30<00:01,  4.95it/s, epoch=80, grad_norm=203.188, loss=314.6534, lr=8.83e-04]
Epoch 80 TRAIN:  97%|█████████▋| 133/137 [00:30<00:00,  5.02it/s, epoch=80, grad_norm=203.188, loss=314.6534, lr=8.83e-04]
Epoch 80 TRAIN:  97%|█████████▋| 133/137 [00:31<00:00,  5.02it/s, epoch=80, grad_norm=393.892, loss=269.0923, lr=8.83e-04]
Epoch 80 TRAIN:  97%|█████████▋| 133/137 [00:31<00:00,  5.02it/s, epoch=80, grad_norm=261.723, loss=300.9860, lr=8.83e-04]
Epoch 80 TRAIN:  97%|█████████▋| 133/137 [00:31<00:00,  5.02it/s, epoch=80, grad_norm=256.731, loss=250.5788, lr=8.84e-04]
Epoch 80 TRAIN:  99%|█████████▉| 136/137 [00:31<00:00,  4.98it/s, epoch=80, grad_norm=256.731, loss=250.5788, lr=8.84e-04]
Epoch 80 TRAIN:  99%|█████████▉| 136/137 [00:31<00:00,  4.31it/s, epoch=80, grad_norm=256.731, loss=250.5788, lr=8.84e-04]

CV:   0%|          | 0/6 [00:00<?, ?it/s]/home/rookie/Documents/batuka/projects/wenet/wenet/wenet/utils/train_utils.py:689: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  amp_autocast(enabled=dtype is not None,

CV:  17%|█▋        | 1/6 [00:00<00:03,  1.31it/s]
                                                 
2026-04-16 20:33:48,015 INFO Epoch 80 Step 11044 CV info lr 8.8352e-04 cv_loss 463.97711181640625 rank 0 acc 0.4058265785376231
2026-04-16 20:33:48,017 INFO [Rank 0] Checkpoint: save to checkpoint exp/conformer_mongolian/epoch_80.pt
2026-04-16 20:33:48,131 INFO Epoch 81 Step 11044 TRAIN info lr 8.8352e-04 rank 0
2026-04-16 20:33:48,152 INFO using accumulate grad, new batch size is 1 times larger than before

Epoch 81 TRAIN:   0%|          | 0/136 [00:00<?, ?it/s]
Epoch 81 TRAIN:   0%|          | 0/136 [00:03<?, ?it/s, epoch=81, grad_norm=476.826, loss=367.1534, lr=8.84e-04]
Epoch 81 TRAIN:   1%|          | 1/136 [00:03<08:32,  3.80s/it, epoch=81, grad_norm=476.826, loss=367.1534, lr=8.84e-04]
Epoch 81 TRAIN:   1%|          | 1/136 [00:03<08:32,  3.80s/it, epoch=81, grad_norm=518.241, loss=394.6595, lr=8.84e-04]
Epoch 81 TRAIN:   1%|          | 1/136 [00:04<08:32,  3.80s/it, epoch=81, grad_norm=535.325, loss=386.6800, lr=8.84e-04]
Epoch 81 TRAIN:   1%|          | 1/136 [00:04<08:32,  3.80s/it, epoch=81, grad_norm=368.895, loss=317.5228, lr=8.84e-04]
Epoch 81 TRAIN:   1%|          | 1/136 [00:04<08:32,  3.80s/it, epoch=81, grad_norm=458.443, loss=449.9227, lr=8.84e-04]
Epoch 81 TRAIN:   1%|          | 1/136 [00:04<08:32,  3.80s/it, epoch=81, grad_norm=237.980, loss=273.5538, lr=8.84e-04]
Epoch 81 TRAIN:   1%|          | 1/136 [00:04<08:32,  3.80s/it, epoch=81, grad_norm=152.098, loss=269.6981, lr=8.84e-04]
Epoch 81 TRAIN:   5%|▌         | 7/136 [00:04<01:01,  2.11it/s, epoch=81, grad_norm=152.098, loss=269.6981, lr=8.84e-04]
Epoch 81 TRAIN:   5%|▌         | 7/136 [00:04<01:01,  2.11it/s, epoch=81, grad_norm=210.520, loss=302.8658, lr=8.84e-04]
Epoch 81 TRAIN:   5%|▌         | 7/136 [00:04<01:01,  2.11it/s, epoch=81, grad_norm=171.078, loss=312.3110, lr=8.84e-04]
Epoch 81 TRAIN:   5%|▌         | 7/136 [00:04<01:01,  2.11it/s, epoch=81, grad_norm=113.602, loss=306.3080, lr=8.84e-04]
Epoch 81 TRAIN:   5%|▌         | 7/136 [00:04<01:01,  2.11it/s, epoch=81, grad_norm=190.789, loss=233.5885, lr=8.84e-04]
Epoch 81 TRAIN:   8%|▊         | 11/136 [00:04<00:40,  3.07it/s, epoch=81, grad_norm=190.789, loss=233.5885, lr=8.84e-04]
Epoch 81 TRAIN:   8%|▊         | 11/136 [00:05<00:40,  3.07it/s, epoch=81, grad_norm=244.027, loss=312.6415, lr=8.84e-04]
Epoch 81 TRAIN:   8%|▊         | 11/136 [00:05<00:40,  3.07it/s, epoch=81, grad_norm=125.733, loss=294.4109, lr=8.85e-04]
Epoch 81 TRAIN:   8%|▊         | 11/136 [00:05<00:40,  3.07it/s, epoch=81, grad_norm=105.637, loss=294.6947, lr=8.85e-04]
Epoch 81 TRAIN:   8%|▊         | 11/136 [00:05<00:40,  3.07it/s, epoch=81, grad_norm=206.904, loss=246.0917, lr=8.85e-04]
Epoch 81 TRAIN:  11%|█         | 15/136 [00:05<00:31,  3.83it/s, epoch=81, grad_norm=206.904, loss=246.0917, lr=8.85e-04]
Epoch 81 TRAIN:  11%|█         | 15/136 [00:05<00:31,  3.83it/s, epoch=81, grad_norm=170.251, loss=233.9362, lr=8.85e-04]
Epoch 81 TRAIN:  11%|█         | 15/136 [00:05<00:31,  3.83it/s, epoch=81, grad_norm=152.397, loss=263.7308, lr=8.85e-04]
Epoch 81 TRAIN:  11%|█         | 15/136 [00:06<00:31,  3.83it/s, epoch=81, grad_norm=97.680, loss=279.1250, lr=8.85e-04] 
Epoch 81 TRAIN:  11%|█         | 15/136 [00:06<00:31,  3.83it/s, epoch=81, grad_norm=100.538, loss=257.9206, lr=8.85e-04]
Epoch 81 TRAIN:  14%|█▍        | 19/136 [00:06<00:27,  4.29it/s, epoch=81, grad_norm=100.538, loss=257.9206, lr=8.85e-04]
Epoch 81 TRAIN:  14%|█▍        | 19/136 [00:06<00:27,  4.29it/s, epoch=81, grad_norm=130.266, loss=268.7516, lr=8.85e-04]
Epoch 81 TRAIN:  14%|█▍        | 19/136 [00:06<00:27,  4.29it/s, epoch=81, grad_norm=74.204, loss=260.5217, lr=8.85e-04] 
Epoch 81 TRAIN:  14%|█▍        | 19/136 [00:06<00:27,  4.29it/s, epoch=81, grad_norm=98.791, loss=259.7642, lr=8.85e-04]
Epoch 81 TRAIN:  14%|█▍        | 19/136 [00:07<00:27,  4.29it/s, epoch=81, grad_norm=128.694, loss=284.3464, lr=8.85e-04]
Epoch 81 TRAIN:  17%|█▋        | 23/136 [00:07<00:24,  4.68it/s, epoch=81, grad_norm=128.694, loss=284.3464, lr=8.85e-04]
Epoch 81 TRAIN:  17%|█▋        | 23/136 [00:07<00:24,  4.68it/s, epoch=81, grad_norm=203.964, loss=314.9127, lr=8.85e-04]
Epoch 81 TRAIN:  17%|█▋        | 23/136 [00:07<00:24,  4.68it/s, epoch=81, grad_norm=133.478, loss=254.0533, lr=8.86e-04]
Epoch 81 TRAIN:  17%|█▋        | 23/136 [00:07<00:24,  4.68it/s, epoch=81, grad_norm=155.791, loss=232.0596, lr=8.86e-04]
Epoch 81 TRAIN:  19%|█▉        | 26/136 [00:07<00:23,  4.72it/s, epoch=81, grad_norm=155.791, loss=232.0596, lr=8.86e-04]
Epoch 81 TRAIN:  19%|█▉        | 26/136 [00:07<00:23,  4.72it/s, epoch=81, grad_norm=91.312, loss=277.0225, lr=8.86e-04] 
Epoch 81 TRAIN:  19%|█▉        | 26/136 [00:08<00:23,  4.72it/s, epoch=81, grad_norm=68.866, loss=274.8632, lr=8.86e-04]
Epoch 81 TRAIN:  19%|█▉        | 26/136 [00:08<00:23,  4.72it/s, epoch=81, grad_norm=93.302, loss=267.2464, lr=8.86e-04]
Epoch 81 TRAIN:  21%|██▏       | 29/136 [00:08<00:21,  4.91it/s, epoch=81, grad_norm=93.302, loss=267.2464, lr=8.86e-04]
Epoch 81 TRAIN:  21%|██▏       | 29/136 [00:08<00:21,  4.91it/s, epoch=81, grad_norm=138.892, loss=302.1408, lr=8.86e-04]
Epoch 81 TRAIN:  21%|██▏       | 29/136 [00:08<00:21,  4.91it/s, epoch=81, grad_norm=141.535, loss=235.2111, lr=8.86e-04]
Epoch 81 TRAIN:  21%|██▏       | 29/136 [00:08<00:21,  4.91it/s, epoch=81, grad_norm=101.081, loss=263.6608, lr=8.86e-04]
Epoch 81 TRAIN:  24%|██▎       | 32/136 [00:08<00:21,  4.91it/s, epoch=81, grad_norm=101.081, loss=263.6608, lr=8.86e-04]
Epoch 81 TRAIN:  24%|██▎       | 32/136 [00:09<00:21,  4.91it/s, epoch=81, grad_norm=96.983, loss=278.9522, lr=8.86e-04] 
Epoch 81 TRAIN:  24%|██▎       | 32/136 [00:09<00:21,  4.91it/s, epoch=81, grad_norm=103.180, loss=291.5555, lr=8.86e-04]
Epoch 81 TRAIN:  24%|██▎       | 32/136 [00:09<00:21,  4.91it/s, epoch=81, grad_norm=153.077, loss=297.1756, lr=8.86e-04]
Epoch 81 TRAIN:  26%|██▌       | 35/136 [00:09<00:20,  4.83it/s, epoch=81, grad_norm=153.077, loss=297.1756, lr=8.86e-04]
Epoch 81 TRAIN:  26%|██▌       | 35/136 [00:09<00:20,  4.83it/s, epoch=81, grad_norm=67.888, loss=288.1441, lr=8.86e-04] 
Epoch 81 TRAIN:  26%|██▌       | 35/136 [00:09<00:20,  4.83it/s, epoch=81, grad_norm=67.147, loss=270.0365, lr=8.86e-04]
Epoch 81 TRAIN:  26%|██▌       | 35/136 [00:10<00:20,  4.83it/s, epoch=81, grad_norm=131.824, loss=256.6591, lr=8.87e-04]
Epoch 81 TRAIN:  28%|██▊       | 38/136 [00:10<00:19,  4.94it/s, epoch=81, grad_norm=131.824, loss=256.6591, lr=8.87e-04]
Epoch 81 TRAIN:  28%|██▊       | 38/136 [00:10<00:19,  4.94it/s, epoch=81, grad_norm=119.877, loss=247.1639, lr=8.87e-04]
Epoch 81 TRAIN:  28%|██▊       | 38/136 [00:10<00:19,  4.94it/s, epoch=81, grad_norm=88.019, loss=244.4064, lr=8.87e-04] 
Epoch 81 TRAIN:  28%|██▊       | 38/136 [00:10<00:19,  4.94it/s, epoch=81, grad_norm=84.220, loss=287.9151, lr=8.87e-04]
Epoch 81 TRAIN:  30%|███       | 41/136 [00:10<00:19,  4.88it/s, epoch=81, grad_norm=84.220, loss=287.9151, lr=8.87e-04]
Epoch 81 TRAIN:  30%|███       | 41/136 [00:10<00:19,  4.88it/s, epoch=81, grad_norm=85.745, loss=297.4510, lr=8.87e-04]
Epoch 81 TRAIN:  30%|███       | 41/136 [00:11<00:19,  4.88it/s, epoch=81, grad_norm=109.811, loss=280.9526, lr=8.87e-04]
Epoch 81 TRAIN:  30%|███       | 41/136 [00:11<00:19,  4.88it/s, epoch=81, grad_norm=74.507, loss=277.9714, lr=8.87e-04] 
Epoch 81 TRAIN:  32%|███▏      | 44/136 [00:11<00:19,  4.80it/s, epoch=81, grad_norm=74.507, loss=277.9714, lr=8.87e-04]
Epoch 81 TRAIN:  32%|███▏      | 44/136 [00:11<00:19,  4.80it/s, epoch=81, grad_norm=121.056, loss=316.2977, lr=8.87e-04]
Epoch 81 TRAIN:  32%|███▏      | 44/136 [00:11<00:19,  4.80it/s, epoch=81, grad_norm=105.157, loss=268.2114, lr=8.87e-04]
Epoch 81 TRAIN:  32%|███▏      | 44/136 [00:11<00:19,  4.80it/s, epoch=81, grad_norm=93.143, loss=285.4913, lr=8.87e-04] 
Epoch 81 TRAIN:  35%|███▍      | 47/136 [00:11<00:18,  4.86it/s, epoch=81, grad_norm=93.143, loss=285.4913, lr=8.87e-04]
Epoch 81 TRAIN:  35%|███▍      | 47/136 [00:12<00:18,  4.86it/s, epoch=81, grad_norm=97.393, loss=280.5767, lr=8.87e-04]
Epoch 81 TRAIN:  35%|███▍      | 47/136 [00:12<00:18,  4.86it/s, epoch=81, grad_norm=139.278, loss=265.0886, lr=8.87e-04]
Epoch 81 TRAIN:  35%|███▍      | 47/136 [00:12<00:18,  4.86it/s, epoch=81, grad_norm=170.046, loss=240.9444, lr=8.88e-04]
Epoch 81 TRAIN:  37%|███▋      | 50/136 [00:12<00:17,  4.91it/s, epoch=81, grad_norm=170.046, loss=240.9444, lr=8.88e-04]
Epoch 81 TRAIN:  37%|███▋      | 50/136 [00:12<00:17,  4.91it/s, epoch=81, grad_norm=120.714, loss=237.6771, lr=8.88e-04]
Epoch 81 TRAIN:  37%|███▋      | 50/136 [00:12<00:17,  4.91it/s, epoch=81, grad_norm=250.238, loss=308.1908, lr=8.88e-04]
Epoch 81 TRAIN:  37%|███▋      | 50/136 [00:13<00:17,  4.91it/s, epoch=81, grad_norm=163.168, loss=309.8337, lr=8.88e-04]
Epoch 81 TRAIN:  39%|███▉      | 53/136 [00:13<00:17,  4.77it/s, epoch=81, grad_norm=163.168, loss=309.8337, lr=8.88e-04]
Epoch 81 TRAIN:  39%|███▉      | 53/136 [00:13<00:17,  4.77it/s, epoch=81, grad_norm=164.475, loss=311.6582, lr=8.88e-04]
Epoch 81 TRAIN:  39%|███▉      | 53/136 [00:13<00:17,  4.77it/s, epoch=81, grad_norm=124.785, loss=248.1957, lr=8.88e-04]
Epoch 81 TRAIN:  39%|███▉      | 53/136 [00:13<00:17,  4.77it/s, epoch=81, grad_norm=158.159, loss=339.6046, lr=8.88e-04]
Epoch 81 TRAIN:  41%|████      | 56/136 [00:13<00:17,  4.52it/s, epoch=81, grad_norm=158.159, loss=339.6046, lr=8.88e-04]
Epoch 81 TRAIN:  41%|████      | 56/136 [00:14<00:17,  4.52it/s, epoch=81, grad_norm=139.759, loss=245.2672, lr=8.88e-04]
Epoch 81 TRAIN:  41%|████      | 56/136 [00:14<00:17,  4.52it/s, epoch=81, grad_norm=138.431, loss=255.4641, lr=8.88e-04]
Epoch 81 TRAIN:  41%|████      | 56/136 [00:14<00:17,  4.52it/s, epoch=81, grad_norm=115.593, loss=290.3678, lr=8.88e-04]
Epoch 81 TRAIN:  43%|████▎     | 59/136 [00:14<00:16,  4.61it/s, epoch=81, grad_norm=115.593, loss=290.3678, lr=8.88e-04]
Epoch 81 TRAIN:  43%|████▎     | 59/136 [00:14<00:16,  4.61it/s, epoch=81, grad_norm=98.523, loss=277.0169, lr=8.88e-04] 
Epoch 81 TRAIN:  43%|████▎     | 59/136 [00:14<00:16,  4.61it/s, epoch=81, grad_norm=93.726, loss=279.0929, lr=8.88e-04]
Epoch 81 TRAIN:  43%|████▎     | 59/136 [00:15<00:16,  4.61it/s, epoch=81, grad_norm=94.341, loss=272.9814, lr=8.88e-04]
Epoch 81 TRAIN:  46%|████▌     | 62/136 [00:15<00:15,  4.70it/s, epoch=81, grad_norm=94.341, loss=272.9814, lr=8.88e-04]
Epoch 81 TRAIN:  46%|████▌     | 62/136 [00:15<00:15,  4.70it/s, epoch=81, grad_norm=96.240, loss=256.0993, lr=8.89e-04]
Epoch 81 TRAIN:  46%|████▌     | 62/136 [00:15<00:15,  4.70it/s, epoch=81, grad_norm=171.161, loss=307.0197, lr=8.89e-04]
Epoch 81 TRAIN:  46%|████▌     | 62/136 [00:15<00:15,  4.70it/s, epoch=81, grad_norm=176.884, loss=295.7922, lr=8.89e-04]
Epoch 81 TRAIN:  48%|████▊     | 65/136 [00:15<00:15,  4.60it/s, epoch=81, grad_norm=176.884, loss=295.7922, lr=8.89e-04]
Epoch 81 TRAIN:  48%|████▊     | 65/136 [00:16<00:15,  4.60it/s, epoch=81, grad_norm=127.732, loss=236.4396, lr=8.89e-04]
Epoch 81 TRAIN:  48%|████▊     | 65/136 [00:16<00:15,  4.60it/s, epoch=81, grad_norm=110.420, loss=281.7108, lr=8.89e-04]
Epoch 81 TRAIN:  48%|████▊     | 65/136 [00:16<00:15,  4.60it/s, epoch=81, grad_norm=163.629, loss=280.7061, lr=8.89e-04]
Epoch 81 TRAIN:  50%|█████     | 68/136 [00:16<00:14,  4.56it/s, epoch=81, grad_norm=163.629, loss=280.7061, lr=8.89e-04]
Epoch 81 TRAIN:  50%|█████     | 68/136 [00:16<00:14,  4.56it/s, epoch=81, grad_norm=119.635, loss=275.2444, lr=8.89e-04]
Epoch 81 TRAIN:  50%|█████     | 68/136 [00:17<00:14,  4.56it/s, epoch=81, grad_norm=131.178, loss=215.7995, lr=8.89e-04]
Epoch 81 TRAIN:  50%|█████     | 68/136 [00:17<00:14,  4.56it/s, epoch=81, grad_norm=111.707, loss=233.0463, lr=8.89e-04]
Epoch 81 TRAIN:  52%|█████▏    | 71/136 [00:17<00:14,  4.44it/s, epoch=81, grad_norm=111.707, loss=233.0463, lr=8.89e-04]
Epoch 81 TRAIN:  52%|█████▏    | 71/136 [00:17<00:14,  4.44it/s, epoch=81, grad_norm=193.828, loss=299.4494, lr=8.89e-04]
Epoch 81 TRAIN:  52%|█████▏    | 71/136 [00:17<00:14,  4.44it/s, epoch=81, grad_norm=166.507, loss=249.5498, lr=8.89e-04]
Epoch 81 TRAIN:  52%|█████▏    | 71/136 [00:17<00:14,  4.44it/s, epoch=81, grad_norm=133.944, loss=280.1002, lr=8.89e-04]
Epoch 81 TRAIN:  54%|█████▍    | 74/136 [00:17<00:13,  4.65it/s, epoch=81, grad_norm=133.944, loss=280.1002, lr=8.89e-04]
Epoch 81 TRAIN:  54%|█████▍    | 74/136 [00:17<00:13,  4.65it/s, epoch=81, grad_norm=158.427, loss=214.7679, lr=8.90e-04]
Epoch 81 TRAIN:  54%|█████▍    | 74/136 [00:18<00:13,  4.65it/s, epoch=81, grad_norm=125.219, loss=295.5720, lr=8.90e-04]
Epoch 81 TRAIN:  54%|█████▍    | 74/136 [00:18<00:13,  4.65it/s, epoch=81, grad_norm=131.822, loss=256.9526, lr=8.90e-04]
Epoch 81 TRAIN:  57%|█████▋    | 77/136 [00:18<00:12,  4.70it/s, epoch=81, grad_norm=131.822, loss=256.9526, lr=8.90e-04]
Epoch 81 TRAIN:  57%|█████▋    | 77/136 [00:18<00:12,  4.70it/s, epoch=81, grad_norm=170.025, loss=283.8214, lr=8.90e-04]
Epoch 81 TRAIN:  57%|█████▋    | 77/136 [00:18<00:12,  4.70it/s, epoch=81, grad_norm=173.050, loss=297.5132, lr=8.90e-04]
Epoch 81 TRAIN:  57%|█████▋    | 77/136 [00:19<00:12,  4.70it/s, epoch=81, grad_norm=126.502, loss=274.7174, lr=8.90e-04]
Epoch 81 TRAIN:  59%|█████▉    | 80/136 [00:19<00:12,  4.64it/s, epoch=81, grad_norm=126.502, loss=274.7174, lr=8.90e-04]
Epoch 81 TRAIN:  59%|█████▉    | 80/136 [00:19<00:12,  4.64it/s, epoch=81, grad_norm=174.726, loss=275.6966, lr=8.90e-04]
Epoch 81 TRAIN:  59%|█████▉    | 80/136 [00:19<00:12,  4.64it/s, epoch=81, grad_norm=148.105, loss=309.2852, lr=8.90e-04]
Epoch 81 TRAIN:  59%|█████▉    | 80/136 [00:19<00:12,  4.64it/s, epoch=81, grad_norm=254.708, loss=204.0537, lr=8.90e-04]
Epoch 81 TRAIN:  61%|██████    | 83/136 [00:19<00:11,  4.75it/s, epoch=81, grad_norm=254.708, loss=204.0537, lr=8.90e-04]
Epoch 81 TRAIN:  61%|██████    | 83/136 [00:19<00:11,  4.75it/s, epoch=81, grad_norm=279.860, loss=276.4244, lr=8.90e-04]
Epoch 81 TRAIN:  61%|██████    | 83/136 [00:20<00:11,  4.75it/s, epoch=81, grad_norm=114.503, loss=258.8912, lr=8.90e-04]
Epoch 81 TRAIN:  61%|██████    | 83/136 [00:20<00:11,  4.75it/s, epoch=81, grad_norm=82.317, loss=260.4565, lr=8.90e-04] 
Epoch 81 TRAIN:  63%|██████▎   | 86/136 [00:20<00:10,  4.65it/s, epoch=81, grad_norm=82.317, loss=260.4565, lr=8.90e-04]
Epoch 81 TRAIN:  63%|██████▎   | 86/136 [00:20<00:10,  4.65it/s, epoch=81, grad_norm=100.632, loss=268.4290, lr=8.90e-04]
Epoch 81 TRAIN:  63%|██████▎   | 86/136 [00:20<00:10,  4.65it/s, epoch=81, grad_norm=99.813, loss=272.0882, lr=8.91e-04] 
Epoch 81 TRAIN:  63%|██████▎   | 86/136 [00:21<00:10,  4.65it/s, epoch=81, grad_norm=135.110, loss=280.4673, lr=8.91e-04]
Epoch 81 TRAIN:  65%|██████▌   | 89/136 [00:21<00:10,  4.44it/s, epoch=81, grad_norm=135.110, loss=280.4673, lr=8.91e-04]
Epoch 81 TRAIN:  65%|██████▌   | 89/136 [00:21<00:10,  4.44it/s, epoch=81, grad_norm=148.042, loss=236.3526, lr=8.91e-04]
Epoch 81 TRAIN:  65%|██████▌   | 89/136 [00:21<00:10,  4.44it/s, epoch=81, grad_norm=150.070, loss=224.0609, lr=8.91e-04]
Epoch 81 TRAIN:  65%|██████▌   | 89/136 [00:21<00:10,  4.44it/s, epoch=81, grad_norm=146.085, loss=309.2462, lr=8.91e-04]
Epoch 81 TRAIN:  68%|██████▊   | 92/136 [00:21<00:09,  4.62it/s, epoch=81, grad_norm=146.085, loss=309.2462, lr=8.91e-04]
Epoch 81 TRAIN:  68%|██████▊   | 92/136 [00:21<00:09,  4.62it/s, epoch=81, grad_norm=175.190, loss=286.1369, lr=8.91e-04]
Epoch 81 TRAIN:  68%|██████▊   | 92/136 [00:22<00:09,  4.62it/s, epoch=81, grad_norm=150.088, loss=260.3300, lr=8.91e-04]
Epoch 81 TRAIN:  68%|██████▊   | 92/136 [00:22<00:09,  4.62it/s, epoch=81, grad_norm=143.411, loss=254.6982, lr=8.91e-04]
Epoch 81 TRAIN:  70%|██████▉   | 95/136 [00:22<00:09,  4.50it/s, epoch=81, grad_norm=143.411, loss=254.6982, lr=8.91e-04]
Epoch 81 TRAIN:  70%|██████▉   | 95/136 [00:22<00:09,  4.50it/s, epoch=81, grad_norm=137.647, loss=269.5758, lr=8.91e-04]
Epoch 81 TRAIN:  70%|██████▉   | 95/136 [00:22<00:09,  4.50it/s, epoch=81, grad_norm=175.987, loss=246.7744, lr=8.91e-04]
Epoch 81 TRAIN:  70%|██████▉   | 95/136 [00:23<00:09,  4.50it/s, epoch=81, grad_norm=179.873, loss=272.6809, lr=8.91e-04]
Epoch 81 TRAIN:  72%|███████▏  | 98/136 [00:23<00:08,  4.61it/s, epoch=81, grad_norm=179.873, loss=272.6809, lr=8.91e-04]
Epoch 81 TRAIN:  72%|███████▏  | 98/136 [00:23<00:08,  4.61it/s, epoch=81, grad_norm=124.132, loss=245.4695, lr=8.91e-04]
Epoch 81 TRAIN:  72%|███████▏  | 98/136 [00:23<00:08,  4.61it/s, epoch=81, grad_norm=235.763, loss=206.2468, lr=8.92e-04]2026-04-16 20:34:11,659 DEBUG TRAIN | steps/sec 4.247| Batch 81/100 loss 206.246826 loss_att 150.062653 loss_ctc 337.343262 th_accuracy 0.522523 lr 8.9152e-04 grad_norm 235.763092 rank 0

Epoch 81 TRAIN:  72%|███████▏  | 98/136 [00:23<00:08,  4.61it/s, epoch=81, grad_norm=132.815, loss=200.6863, lr=8.92e-04]
Epoch 81 TRAIN:  74%|███████▍  | 101/136 [00:23<00:07,  4.50it/s, epoch=81, grad_norm=132.815, loss=200.6863, lr=8.92e-04]
Epoch 81 TRAIN:  74%|███████▍  | 101/136 [00:23<00:07,  4.50it/s, epoch=81, grad_norm=88.034, loss=234.0403, lr=8.92e-04] 
Epoch 81 TRAIN:  74%|███████▍  | 101/136 [00:24<00:07,  4.50it/s, epoch=81, grad_norm=172.930, loss=275.7951, lr=8.92e-04]
Epoch 81 TRAIN:  74%|███████▍  | 101/136 [00:24<00:07,  4.50it/s, epoch=81, grad_norm=185.596, loss=300.5353, lr=8.92e-04]
Epoch 81 TRAIN:  76%|███████▋  | 104/136 [00:24<00:06,  4.60it/s, epoch=81, grad_norm=185.596, loss=300.5353, lr=8.92e-04]
Epoch 81 TRAIN:  76%|███████▋  | 104/136 [00:24<00:06,  4.60it/s, epoch=81, grad_norm=193.718, loss=357.8564, lr=8.92e-04]
Epoch 81 TRAIN:  76%|███████▋  | 104/136 [00:24<00:06,  4.60it/s, epoch=81, grad_norm=151.606, loss=226.4067, lr=8.92e-04]
Epoch 81 TRAIN:  76%|███████▋  | 104/136 [00:24<00:06,  4.60it/s, epoch=81, grad_norm=193.478, loss=233.8549, lr=8.92e-04]
Epoch 81 TRAIN:  79%|███████▊  | 107/136 [00:24<00:06,  4.70it/s, epoch=81, grad_norm=193.478, loss=233.8549, lr=8.92e-04]
Epoch 81 TRAIN:  79%|███████▊  | 107/136 [00:25<00:06,  4.70it/s, epoch=81, grad_norm=178.639, loss=304.0785, lr=8.92e-04]
Epoch 81 TRAIN:  79%|███████▊  | 107/136 [00:25<00:06,  4.70it/s, epoch=81, grad_norm=98.739, loss=219.6818, lr=8.92e-04] 
Epoch 81 TRAIN:  79%|███████▊  | 107/136 [00:25<00:06,  4.70it/s, epoch=81, grad_norm=135.804, loss=260.7323, lr=8.92e-04]
Epoch 81 TRAIN:  81%|████████  | 110/136 [00:25<00:05,  4.67it/s, epoch=81, grad_norm=135.804, loss=260.7323, lr=8.92e-04]
Epoch 81 TRAIN:  81%|████████  | 110/136 [00:25<00:05,  4.67it/s, epoch=81, grad_norm=223.100, loss=297.9202, lr=8.92e-04]
Epoch 81 TRAIN:  81%|████████  | 110/136 [00:26<00:05,  4.67it/s, epoch=81, grad_norm=150.676, loss=260.7737, lr=8.92e-04]
Epoch 81 TRAIN:  81%|████████  | 110/136 [00:26<00:05,  4.67it/s, epoch=81, grad_norm=155.761, loss=306.7598, lr=8.93e-04]
Epoch 81 TRAIN:  83%|████████▎ | 113/136 [00:26<00:05,  4.44it/s, epoch=81, grad_norm=155.761, loss=306.7598, lr=8.93e-04]
Epoch 81 TRAIN:  83%|████████▎ | 113/136 [00:26<00:05,  4.44it/s, epoch=81, grad_norm=140.045, loss=288.5986, lr=8.93e-04]
Epoch 81 TRAIN:  83%|████████▎ | 113/136 [00:26<00:05,  4.44it/s, epoch=81, grad_norm=280.187, loss=253.1385, lr=8.93e-04]
Epoch 81 TRAIN:  83%|████████▎ | 113/136 [00:26<00:05,  4.44it/s, epoch=81, grad_norm=139.667, loss=288.9291, lr=8.93e-04]
Epoch 81 TRAIN:  85%|████████▌ | 116/136 [00:26<00:04,  4.61it/s, epoch=81, grad_norm=139.667, loss=288.9291, lr=8.93e-04]
Epoch 81 TRAIN:  85%|████████▌ | 116/136 [00:27<00:04,  4.61it/s, epoch=81, grad_norm=131.214, loss=232.4303, lr=8.93e-04]
Epoch 81 TRAIN:  85%|████████▌ | 116/136 [00:27<00:04,  4.61it/s, epoch=81, grad_norm=147.852, loss=218.0529, lr=8.93e-04]
Epoch 81 TRAIN:  85%|████████▌ | 116/136 [00:27<00:04,  4.61it/s, epoch=81, grad_norm=208.486, loss=270.6327, lr=8.93e-04]
Epoch 81 TRAIN:  88%|████████▊ | 119/136 [00:27<00:03,  4.61it/s, epoch=81, grad_norm=208.486, loss=270.6327, lr=8.93e-04]
Epoch 81 TRAIN:  88%|████████▊ | 119/136 [00:27<00:03,  4.61it/s, epoch=81, grad_norm=371.592, loss=291.0912, lr=8.93e-04]
Epoch 81 TRAIN:  88%|████████▊ | 119/136 [00:27<00:03,  4.61it/s, epoch=81, grad_norm=241.376, loss=306.5471, lr=8.93e-04]
Epoch 81 TRAIN:  88%|████████▊ | 119/136 [00:28<00:03,  4.61it/s, epoch=81, grad_norm=152.436, loss=288.8884, lr=8.93e-04]
Epoch 81 TRAIN:  90%|████████▉ | 122/136 [00:28<00:02,  4.83it/s, epoch=81, grad_norm=152.436, loss=288.8884, lr=8.93e-04]
Epoch 81 TRAIN:  90%|████████▉ | 122/136 [00:28<00:02,  4.83it/s, epoch=81, grad_norm=143.121, loss=274.2565, lr=8.93e-04]
Epoch 81 TRAIN:  90%|████████▉ | 122/136 [00:28<00:02,  4.83it/s, epoch=81, grad_norm=133.339, loss=269.5623, lr=8.93e-04]
Epoch 81 TRAIN:  90%|████████▉ | 122/136 [00:28<00:02,  4.83it/s, epoch=81, grad_norm=103.675, loss=243.4632, lr=8.94e-04]
Epoch 81 TRAIN:  92%|█████████▏| 125/136 [00:28<00:02,  4.57it/s, epoch=81, grad_norm=103.675, loss=243.4632, lr=8.94e-04]
Epoch 81 TRAIN:  92%|█████████▏| 125/136 [00:29<00:02,  4.57it/s, epoch=81, grad_norm=155.558, loss=280.5467, lr=8.94e-04]
Epoch 81 TRAIN:  92%|█████████▏| 125/136 [00:29<00:02,  4.57it/s, epoch=81, grad_norm=103.559, loss=237.5301, lr=8.94e-04]
Epoch 81 TRAIN:  92%|█████████▏| 125/136 [00:29<00:02,  4.57it/s, epoch=81, grad_norm=110.414, loss=226.3522, lr=8.94e-04]
Epoch 81 TRAIN:  94%|█████████▍| 128/136 [00:29<00:01,  4.57it/s, epoch=81, grad_norm=110.414, loss=226.3522, lr=8.94e-04]
Epoch 81 TRAIN:  94%|█████████▍| 128/136 [00:29<00:01,  4.57it/s, epoch=81, grad_norm=134.072, loss=273.9675, lr=8.94e-04]
Epoch 81 TRAIN:  94%|█████████▍| 128/136 [00:29<00:01,  4.57it/s, epoch=81, grad_norm=313.771, loss=348.8491, lr=8.94e-04]
Epoch 81 TRAIN:  94%|█████████▍| 128/136 [00:30<00:01,  4.57it/s, epoch=81, grad_norm=130.432, loss=272.1944, lr=8.94e-04]
Epoch 81 TRAIN:  96%|█████████▋| 131/136 [00:30<00:01,  4.85it/s, epoch=81, grad_norm=130.432, loss=272.1944, lr=8.94e-04]
Epoch 81 TRAIN:  96%|█████████▋| 131/136 [00:30<00:01,  4.85it/s, epoch=81, grad_norm=122.064, loss=275.7543, lr=8.94e-04]
Epoch 81 TRAIN:  96%|█████████▋| 131/136 [00:30<00:01,  4.85it/s, epoch=81, grad_norm=153.112, loss=281.7927, lr=8.94e-04]
Epoch 81 TRAIN:  96%|█████████▋| 131/136 [00:30<00:01,  4.85it/s, epoch=81, grad_norm=247.129, loss=199.4655, lr=8.94e-04]
Epoch 81 TRAIN:  99%|█████████▊| 134/136 [00:30<00:00,  4.90it/s, epoch=81, grad_norm=247.129, loss=199.4655, lr=8.94e-04]
Epoch 81 TRAIN:  99%|█████████▊| 134/136 [00:30<00:00,  4.90it/s, epoch=81, grad_norm=265.515, loss=295.7518, lr=8.94e-04]
Epoch 81 TRAIN:  99%|█████████▊| 134/136 [00:30<00:00,  4.90it/s, epoch=81, grad_norm=277.691, loss=341.5154, lr=8.94e-04]
Epoch 81 TRAIN:  99%|█████████▊| 134/136 [00:31<00:00,  4.90it/s, epoch=81, grad_norm=158.501, loss=329.3557, lr=8.94e-04]
Epoch 81 TRAIN:  99%|█████████▊| 134/136 [00:31<00:00,  4.90it/s, epoch=81, grad_norm=222.583, loss=314.7987, lr=8.95e-04]
Epoch 81 TRAIN: 138it [00:31,  5.41it/s, epoch=81, grad_norm=222.583, loss=314.7987, lr=8.95e-04]                         
Epoch 81 TRAIN: 138it [00:31,  4.41it/s, epoch=81, grad_norm=222.583, loss=314.7987, lr=8.95e-04]

CV:   0%|          | 0/6 [00:00<?, ?it/s]/home/rookie/Documents/batuka/projects/wenet/wenet/wenet/utils/train_utils.py:689: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  amp_autocast(enabled=dtype is not None,

CV:  17%|█▋        | 1/6 [00:00<00:03,  1.33it/s]
                                                 
2026-04-16 20:34:20,288 INFO Epoch 81 Step 11182 CV info lr 8.9456e-04 cv_loss 473.2965004660866 rank 0 acc 0.38775786260763806
2026-04-16 20:34:20,290 INFO [Rank 0] Checkpoint: save to checkpoint exp/conformer_mongolian/epoch_81.pt
2026-04-16 20:34:20,403 INFO Epoch 82 Step 11182 TRAIN info lr 8.9456e-04 rank 0
2026-04-16 20:34:20,424 INFO using accumulate grad, new batch size is 1 times larger than before

Epoch 82 TRAIN:   0%|          | 0/138 [00:00<?, ?it/s]
Epoch 82 TRAIN:   0%|          | 0/138 [00:03<?, ?it/s, epoch=82, grad_norm=392.610, loss=316.2430, lr=8.95e-04]
Epoch 82 TRAIN:   1%|          | 1/138 [00:03<08:50,  3.88s/it, epoch=82, grad_norm=392.610, loss=316.2430, lr=8.95e-04]
Epoch 82 TRAIN:   1%|          | 1/138 [00:04<08:50,  3.88s/it, epoch=82, grad_norm=351.442, loss=336.1647, lr=8.95e-04]
Epoch 82 TRAIN:   1%|          | 1/138 [00:04<08:50,  3.88s/it, epoch=82, grad_norm=332.979, loss=328.5840, lr=8.95e-04]
Epoch 82 TRAIN:   1%|          | 1/138 [00:04<08:50,  3.88s/it, epoch=82, grad_norm=281.883, loss=293.3904, lr=8.95e-04]
Epoch 82 TRAIN:   1%|          | 1/138 [00:04<08:50,  3.88s/it, epoch=82, grad_norm=121.649, loss=195.0055, lr=8.95e-04]
Epoch 82 TRAIN:   1%|          | 1/138 [00:04<08:50,  3.88s/it, epoch=82, grad_norm=256.605, loss=318.0774, lr=8.95e-04]
Epoch 82 TRAIN:   1%|          | 1/138 [00:04<08:50,  3.88s/it, epoch=82, grad_norm=223.238, loss=282.7313, lr=8.95e-04]
Epoch 82 TRAIN:   5%|▌         | 7/138 [00:04<01:03,  2.05it/s, epoch=82, grad_norm=223.238, loss=282.7313, lr=8.95e-04]
Epoch 82 TRAIN:   5%|▌         | 7/138 [00:04<01:03,  2.05it/s, epoch=82, grad_norm=212.941, loss=303.2522, lr=8.95e-04]
Epoch 82 TRAIN:   5%|▌         | 7/138 [00:04<01:03,  2.05it/s, epoch=82, grad_norm=111.055, loss=285.6301, lr=8.95e-04]
Epoch 82 TRAIN:   5%|▌         | 7/138 [00:04<01:03,  2.05it/s, epoch=82, grad_norm=85.782, loss=264.2521, lr=8.95e-04] 
Epoch 82 TRAIN:   5%|▌         | 7/138 [00:04<01:03,  2.05it/s, epoch=82, grad_norm=103.859, loss=264.4214, lr=8.95e-04]
Epoch 82 TRAIN:   8%|▊         | 11/138 [00:04<00:41,  3.09it/s, epoch=82, grad_norm=103.859, loss=264.4214, lr=8.95e-04]
Epoch 82 TRAIN:   8%|▊         | 11/138 [00:05<00:41,  3.09it/s, epoch=82, grad_norm=142.274, loss=249.0244, lr=8.96e-04]
Epoch 82 TRAIN:   8%|▊         | 11/138 [00:05<00:41,  3.09it/s, epoch=82, grad_norm=91.745, loss=285.5624, lr=8.96e-04] 
Epoch 82 TRAIN:   8%|▊         | 11/138 [00:05<00:41,  3.09it/s, epoch=82, grad_norm=94.417, loss=263.5242, lr=8.96e-04]
Epoch 82 TRAIN:   8%|▊         | 11/138 [00:05<00:41,  3.09it/s, epoch=82, grad_norm=94.365, loss=262.8407, lr=8.96e-04]
Epoch 82 TRAIN:  11%|█         | 15/138 [00:05<00:32,  3.77it/s, epoch=82, grad_norm=94.365, loss=262.8407, lr=8.96e-04]
Epoch 82 TRAIN:  11%|█         | 15/138 [00:05<00:32,  3.77it/s, epoch=82, grad_norm=91.019, loss=278.5568, lr=8.96e-04]
Epoch 82 TRAIN:  11%|█         | 15/138 [00:06<00:32,  3.77it/s, epoch=82, grad_norm=74.149, loss=264.0346, lr=8.96e-04]
Epoch 82 TRAIN:  11%|█         | 15/138 [00:06<00:32,  3.77it/s, epoch=82, grad_norm=133.729, loss=318.6075, lr=8.96e-04]
Epoch 82 TRAIN:  11%|█         | 15/138 [00:06<00:32,  3.77it/s, epoch=82, grad_norm=164.424, loss=309.1392, lr=8.96e-04]
Epoch 82 TRAIN:  14%|█▍        | 19/138 [00:06<00:28,  4.23it/s, epoch=82, grad_norm=164.424, loss=309.1392, lr=8.96e-04]
Epoch 82 TRAIN:  14%|█▍        | 19/138 [00:06<00:28,  4.23it/s, epoch=82, grad_norm=79.156, loss=263.6016, lr=8.96e-04] 
Epoch 82 TRAIN:  14%|█▍        | 19/138 [00:06<00:28,  4.23it/s, epoch=82, grad_norm=152.785, loss=281.4673, lr=8.96e-04]
Epoch 82 TRAIN:  14%|█▍        | 19/138 [00:07<00:28,  4.23it/s, epoch=82, grad_norm=106.826, loss=318.0157, lr=8.96e-04]
Epoch 82 TRAIN:  14%|█▍        | 19/138 [00:07<00:28,  4.23it/s, epoch=82, grad_norm=104.319, loss=307.2852, lr=8.96e-04]
Epoch 82 TRAIN:  17%|█▋        | 23/138 [00:07<00:26,  4.42it/s, epoch=82, grad_norm=104.319, loss=307.2852, lr=8.96e-04]
Epoch 82 TRAIN:  17%|█▋        | 23/138 [00:07<00:26,  4.42it/s, epoch=82, grad_norm=96.920, loss=279.0566, lr=8.96e-04] 
Epoch 82 TRAIN:  17%|█▋        | 23/138 [00:07<00:26,  4.42it/s, epoch=82, grad_norm=99.045, loss=326.7595, lr=8.97e-04]
Epoch 82 TRAIN:  17%|█▋        | 23/138 [00:07<00:26,  4.42it/s, epoch=82, grad_norm=97.174, loss=270.4190, lr=8.97e-04]
Epoch 82 TRAIN:  19%|█▉        | 26/138 [00:07<00:25,  4.48it/s, epoch=82, grad_norm=97.174, loss=270.4190, lr=8.97e-04]
Epoch 82 TRAIN:  19%|█▉        | 26/138 [00:08<00:25,  4.48it/s, epoch=82, grad_norm=238.118, loss=250.7120, lr=8.97e-04]
Epoch 82 TRAIN:  19%|█▉        | 26/138 [00:08<00:25,  4.48it/s, epoch=82, grad_norm=186.771, loss=222.6506, lr=8.97e-04]
Epoch 82 TRAIN:  19%|█▉        | 26/138 [00:08<00:25,  4.48it/s, epoch=82, grad_norm=147.190, loss=271.4997, lr=8.97e-04]
Epoch 82 TRAIN:  21%|██        | 29/138 [00:08<00:23,  4.63it/s, epoch=82, grad_norm=147.190, loss=271.4997, lr=8.97e-04]
Epoch 82 TRAIN:  21%|██        | 29/138 [00:08<00:23,  4.63it/s, epoch=82, grad_norm=95.248, loss=299.8937, lr=8.97e-04] 
Epoch 82 TRAIN:  21%|██        | 29/138 [00:08<00:23,  4.63it/s, epoch=82, grad_norm=106.862, loss=261.4585, lr=8.97e-04]
Epoch 82 TRAIN:  21%|██        | 29/138 [00:09<00:23,  4.63it/s, epoch=82, grad_norm=156.820, loss=298.8862, lr=8.97e-04]
Epoch 82 TRAIN:  23%|██▎       | 32/138 [00:09<00:22,  4.66it/s, epoch=82, grad_norm=156.820, loss=298.8862, lr=8.97e-04]
Epoch 82 TRAIN:  23%|██▎       | 32/138 [00:09<00:22,  4.66it/s, epoch=82, grad_norm=92.920, loss=299.0091, lr=8.97e-04] 
Epoch 82 TRAIN:  23%|██▎       | 32/138 [00:09<00:22,  4.66it/s, epoch=82, grad_norm=120.863, loss=280.3758, lr=8.97e-04]
Epoch 82 TRAIN:  23%|██▎       | 32/138 [00:09<00:22,  4.66it/s, epoch=82, grad_norm=156.633, loss=295.9962, lr=8.97e-04]
Epoch 82 TRAIN:  25%|██▌       | 35/138 [00:09<00:22,  4.68it/s, epoch=82, grad_norm=156.633, loss=295.9962, lr=8.97e-04]
Epoch 82 TRAIN:  25%|██▌       | 35/138 [00:09<00:22,  4.68it/s, epoch=82, grad_norm=79.167, loss=260.9514, lr=8.97e-04] 
Epoch 82 TRAIN:  25%|██▌       | 35/138 [00:10<00:22,  4.68it/s, epoch=82, grad_norm=91.284, loss=260.6169, lr=8.98e-04]
Epoch 82 TRAIN:  25%|██▌       | 35/138 [00:10<00:22,  4.68it/s, epoch=82, grad_norm=71.516, loss=262.7032, lr=8.98e-04]
Epoch 82 TRAIN:  28%|██▊       | 38/138 [00:10<00:21,  4.70it/s, epoch=82, grad_norm=71.516, loss=262.7032, lr=8.98e-04]
Epoch 82 TRAIN:  28%|██▊       | 38/138 [00:10<00:21,  4.70it/s, epoch=82, grad_norm=110.508, loss=251.3909, lr=8.98e-04]
Epoch 82 TRAIN:  28%|██▊       | 38/138 [00:10<00:21,  4.70it/s, epoch=82, grad_norm=126.649, loss=270.1340, lr=8.98e-04]
Epoch 82 TRAIN:  28%|██▊       | 38/138 [00:11<00:21,  4.70it/s, epoch=82, grad_norm=91.819, loss=252.6653, lr=8.98e-04] 
Epoch 82 TRAIN:  30%|██▉       | 41/138 [00:11<00:20,  4.66it/s, epoch=82, grad_norm=91.819, loss=252.6653, lr=8.98e-04]
Epoch 82 TRAIN:  30%|██▉       | 41/138 [00:11<00:20,  4.66it/s, epoch=82, grad_norm=113.288, loss=259.0555, lr=8.98e-04]
Epoch 82 TRAIN:  30%|██▉       | 41/138 [00:11<00:20,  4.66it/s, epoch=82, grad_norm=140.506, loss=271.0201, lr=8.98e-04]
Epoch 82 TRAIN:  30%|██▉       | 41/138 [00:11<00:20,  4.66it/s, epoch=82, grad_norm=142.986, loss=316.7352, lr=8.98e-04]
Epoch 82 TRAIN:  32%|███▏      | 44/138 [00:11<00:20,  4.66it/s, epoch=82, grad_norm=142.986, loss=316.7352, lr=8.98e-04]
Epoch 82 TRAIN:  32%|███▏      | 44/138 [00:11<00:20,  4.66it/s, epoch=82, grad_norm=149.786, loss=324.5953, lr=8.98e-04]
Epoch 82 TRAIN:  32%|███▏      | 44/138 [00:12<00:20,  4.66it/s, epoch=82, grad_norm=122.341, loss=254.6159, lr=8.98e-04]
Epoch 82 TRAIN:  32%|███▏      | 44/138 [00:12<00:20,  4.66it/s, epoch=82, grad_norm=130.281, loss=272.9069, lr=8.98e-04]
Epoch 82 TRAIN:  34%|███▍      | 47/138 [00:12<00:19,  4.70it/s, epoch=82, grad_norm=130.281, loss=272.9069, lr=8.98e-04]
Epoch 82 TRAIN:  34%|███▍      | 47/138 [00:12<00:19,  4.70it/s, epoch=82, grad_norm=108.482, loss=247.8409, lr=8.98e-04]
Epoch 82 TRAIN:  34%|███▍      | 47/138 [00:12<00:19,  4.70it/s, epoch=82, grad_norm=117.330, loss=292.9807, lr=8.98e-04]
Epoch 82 TRAIN:  34%|███▍      | 47/138 [00:12<00:19,  4.70it/s, epoch=82, grad_norm=88.810, loss=252.6824, lr=8.99e-04] 
Epoch 82 TRAIN:  36%|███▌      | 50/138 [00:12<00:19,  4.57it/s, epoch=82, grad_norm=88.810, loss=252.6824, lr=8.99e-04]
Epoch 82 TRAIN:  36%|███▌      | 50/138 [00:13<00:19,  4.57it/s, epoch=82, grad_norm=121.042, loss=255.9527, lr=8.99e-04]
Epoch 82 TRAIN:  36%|███▌      | 50/138 [00:13<00:19,  4.57it/s, epoch=82, grad_norm=116.527, loss=303.8623, lr=8.99e-04]
Epoch 82 TRAIN:  36%|███▌      | 50/138 [00:13<00:19,  4.57it/s, epoch=82, grad_norm=113.518, loss=295.0632, lr=8.99e-04]
Epoch 82 TRAIN:  38%|███▊      | 53/138 [00:13<00:19,  4.46it/s, epoch=82, grad_norm=113.518, loss=295.0632, lr=8.99e-04]
Epoch 82 TRAIN:  38%|███▊      | 53/138 [00:13<00:19,  4.46it/s, epoch=82, grad_norm=113.906, loss=292.6868, lr=8.99e-04]
Epoch 82 TRAIN:  38%|███▊      | 53/138 [00:14<00:19,  4.46it/s, epoch=82, grad_norm=154.920, loss=315.6821, lr=8.99e-04]
Epoch 82 TRAIN:  38%|███▊      | 53/138 [00:14<00:19,  4.46it/s, epoch=82, grad_norm=109.631, loss=262.4703, lr=8.99e-04]
Epoch 82 TRAIN:  41%|████      | 56/138 [00:14<00:18,  4.51it/s, epoch=82, grad_norm=109.631, loss=262.4703, lr=8.99e-04]
Epoch 82 TRAIN:  41%|████      | 56/138 [00:14<00:18,  4.51it/s, epoch=82, grad_norm=102.947, loss=286.7640, lr=8.99e-04]
Epoch 82 TRAIN:  41%|████      | 56/138 [00:14<00:18,  4.51it/s, epoch=82, grad_norm=94.043, loss=286.2138, lr=8.99e-04] 
Epoch 82 TRAIN:  41%|████      | 56/138 [00:15<00:18,  4.51it/s, epoch=82, grad_norm=100.522, loss=286.8717, lr=8.99e-04]
Epoch 82 TRAIN:  43%|████▎     | 59/138 [00:15<00:17,  4.52it/s, epoch=82, grad_norm=100.522, loss=286.8717, lr=8.99e-04]
Epoch 82 TRAIN:  43%|████▎     | 59/138 [00:15<00:17,  4.52it/s, epoch=82, grad_norm=123.287, loss=279.3324, lr=8.99e-04]
Epoch 82 TRAIN:  43%|████▎     | 59/138 [00:15<00:17,  4.52it/s, epoch=82, grad_norm=140.797, loss=256.4116, lr=8.99e-04]
Epoch 82 TRAIN:  43%|████▎     | 59/138 [00:15<00:17,  4.52it/s, epoch=82, grad_norm=78.354, loss=259.5811, lr=9.00e-04] 
Epoch 82 TRAIN:  45%|████▍     | 62/138 [00:15<00:16,  4.62it/s, epoch=82, grad_norm=78.354, loss=259.5811, lr=9.00e-04]
Epoch 82 TRAIN:  45%|████▍     | 62/138 [00:15<00:16,  4.62it/s, epoch=82, grad_norm=90.334, loss=270.7787, lr=9.00e-04]
Epoch 82 TRAIN:  45%|████▍     | 62/138 [00:16<00:16,  4.62it/s, epoch=82, grad_norm=169.138, loss=294.1207, lr=9.00e-04]
Epoch 82 TRAIN:  45%|████▍     | 62/138 [00:16<00:16,  4.62it/s, epoch=82, grad_norm=145.399, loss=268.7527, lr=9.00e-04]
Epoch 82 TRAIN:  47%|████▋     | 65/138 [00:16<00:17,  4.29it/s, epoch=82, grad_norm=145.399, loss=268.7527, lr=9.00e-04]
Epoch 82 TRAIN:  47%|████▋     | 65/138 [00:16<00:17,  4.29it/s, epoch=82, grad_norm=106.832, loss=257.7590, lr=9.00e-04]
Epoch 82 TRAIN:  47%|████▋     | 65/138 [00:16<00:17,  4.29it/s, epoch=82, grad_norm=138.605, loss=250.1776, lr=9.00e-04]
Epoch 82 TRAIN:  47%|████▋     | 65/138 [00:17<00:17,  4.29it/s, epoch=82, grad_norm=133.166, loss=285.6125, lr=9.00e-04]
Epoch 82 TRAIN:  49%|████▉     | 68/138 [00:17<00:15,  4.43it/s, epoch=82, grad_norm=133.166, loss=285.6125, lr=9.00e-04]
Epoch 82 TRAIN:  49%|████▉     | 68/138 [00:17<00:15,  4.43it/s, epoch=82, grad_norm=169.528, loss=308.1140, lr=9.00e-04]
Epoch 82 TRAIN:  49%|████▉     | 68/138 [00:17<00:15,  4.43it/s, epoch=82, grad_norm=120.399, loss=276.7192, lr=9.00e-04]
Epoch 82 TRAIN:  49%|████▉     | 68/138 [00:17<00:15,  4.43it/s, epoch=82, grad_norm=94.644, loss=217.1524, lr=9.00e-04] 
Epoch 82 TRAIN:  51%|█████▏    | 71/138 [00:17<00:15,  4.43it/s, epoch=82, grad_norm=94.644, loss=217.1524, lr=9.00e-04]
Epoch 82 TRAIN:  51%|█████▏    | 71/138 [00:17<00:15,  4.43it/s, epoch=82, grad_norm=83.121, loss=234.1796, lr=9.00e-04]
Epoch 82 TRAIN:  51%|█████▏    | 71/138 [00:18<00:15,  4.43it/s, epoch=82, grad_norm=173.280, loss=216.9250, lr=9.00e-04]
Epoch 82 TRAIN:  51%|█████▏    | 71/138 [00:18<00:15,  4.43it/s, epoch=82, grad_norm=151.847, loss=259.0781, lr=9.00e-04]
Epoch 82 TRAIN:  54%|█████▎    | 74/138 [00:18<00:14,  4.57it/s, epoch=82, grad_norm=151.847, loss=259.0781, lr=9.00e-04]
Epoch 82 TRAIN:  54%|█████▎    | 74/138 [00:18<00:14,  4.57it/s, epoch=82, grad_norm=162.542, loss=274.5858, lr=9.01e-04]
Epoch 82 TRAIN:  54%|█████▎    | 74/138 [00:18<00:14,  4.57it/s, epoch=82, grad_norm=115.035, loss=250.3629, lr=9.01e-04]
Epoch 82 TRAIN:  54%|█████▎    | 74/138 [00:19<00:14,  4.57it/s, epoch=82, grad_norm=174.474, loss=241.7926, lr=9.01e-04]
Epoch 82 TRAIN:  56%|█████▌    | 77/138 [00:19<00:14,  4.30it/s, epoch=82, grad_norm=174.474, loss=241.7926, lr=9.01e-04]
Epoch 82 TRAIN:  56%|█████▌    | 77/138 [00:19<00:14,  4.30it/s, epoch=82, grad_norm=95.524, loss=239.6707, lr=9.01e-04] 
Epoch 82 TRAIN:  56%|█████▌    | 77/138 [00:19<00:14,  4.30it/s, epoch=82, grad_norm=134.561, loss=258.4456, lr=9.01e-04]
Epoch 82 TRAIN:  56%|█████▌    | 77/138 [00:19<00:14,  4.30it/s, epoch=82, grad_norm=133.908, loss=284.6237, lr=9.01e-04]
Epoch 82 TRAIN:  58%|█████▊    | 80/138 [00:19<00:12,  4.55it/s, epoch=82, grad_norm=133.908, loss=284.6237, lr=9.01e-04]
Epoch 82 TRAIN:  58%|█████▊    | 80/138 [00:19<00:12,  4.55it/s, epoch=82, grad_norm=177.608, loss=242.0287, lr=9.01e-04]
Epoch 82 TRAIN:  58%|█████▊    | 80/138 [00:20<00:12,  4.55it/s, epoch=82, grad_norm=187.689, loss=308.0636, lr=9.01e-04]
Epoch 82 TRAIN:  58%|█████▊    | 80/138 [00:20<00:12,  4.55it/s, epoch=82, grad_norm=219.562, loss=253.4771, lr=9.01e-04]
Epoch 82 TRAIN:  60%|██████    | 83/138 [00:20<00:12,  4.27it/s, epoch=82, grad_norm=219.562, loss=253.4771, lr=9.01e-04]
Epoch 82 TRAIN:  60%|██████    | 83/138 [00:20<00:12,  4.27it/s, epoch=82, grad_norm=152.672, loss=194.8970, lr=9.01e-04]
Epoch 82 TRAIN:  60%|██████    | 83/138 [00:20<00:12,  4.27it/s, epoch=82, grad_norm=220.735, loss=257.0887, lr=9.01e-04]
Epoch 82 TRAIN:  60%|██████    | 83/138 [00:21<00:12,  4.27it/s, epoch=82, grad_norm=213.764, loss=289.9239, lr=9.01e-04]
Epoch 82 TRAIN:  62%|██████▏   | 86/138 [00:21<00:11,  4.35it/s, epoch=82, grad_norm=213.764, loss=289.9239, lr=9.01e-04]
Epoch 82 TRAIN:  62%|██████▏   | 86/138 [00:21<00:11,  4.35it/s, epoch=82, grad_norm=82.311, loss=236.0479, lr=9.02e-04] 
Epoch 82 TRAIN:  62%|██████▏   | 86/138 [00:21<00:11,  4.35it/s, epoch=82, grad_norm=292.288, loss=293.3085, lr=9.02e-04]
Epoch 82 TRAIN:  62%|██████▏   | 86/138 [00:21<00:11,  4.35it/s, epoch=82, grad_norm=120.615, loss=232.9123, lr=9.02e-04]
Epoch 82 TRAIN:  64%|██████▍   | 89/138 [00:21<00:10,  4.46it/s, epoch=82, grad_norm=120.615, loss=232.9123, lr=9.02e-04]
Epoch 82 TRAIN:  64%|██████▍   | 89/138 [00:22<00:10,  4.46it/s, epoch=82, grad_norm=197.942, loss=300.0545, lr=9.02e-04]
Epoch 82 TRAIN:  64%|██████▍   | 89/138 [00:22<00:10,  4.46it/s, epoch=82, grad_norm=225.030, loss=186.8412, lr=9.02e-04]
Epoch 82 TRAIN:  64%|██████▍   | 89/138 [00:22<00:10,  4.46it/s, epoch=82, grad_norm=204.132, loss=298.5513, lr=9.02e-04]
Epoch 82 TRAIN:  67%|██████▋   | 92/138 [00:22<00:10,  4.55it/s, epoch=82, grad_norm=204.132, loss=298.5513, lr=9.02e-04]
Epoch 82 TRAIN:  67%|██████▋   | 92/138 [00:22<00:10,  4.55it/s, epoch=82, grad_norm=151.263, loss=311.5026, lr=9.02e-04]
Epoch 82 TRAIN:  67%|██████▋   | 92/138 [00:22<00:10,  4.55it/s, epoch=82, grad_norm=80.040, loss=241.2076, lr=9.02e-04] 
Epoch 82 TRAIN:  67%|██████▋   | 92/138 [00:23<00:10,  4.55it/s, epoch=82, grad_norm=85.236, loss=248.4680, lr=9.02e-04]
Epoch 82 TRAIN:  69%|██████▉   | 95/138 [00:23<00:09,  4.42it/s, epoch=82, grad_norm=85.236, loss=248.4680, lr=9.02e-04]
Epoch 82 TRAIN:  69%|██████▉   | 95/138 [00:23<00:09,  4.42it/s, epoch=82, grad_norm=131.872, loss=264.4508, lr=9.02e-04]
Epoch 82 TRAIN:  69%|██████▉   | 95/138 [00:23<00:09,  4.42it/s, epoch=82, grad_norm=102.402, loss=212.0565, lr=9.02e-04]
Epoch 82 TRAIN:  69%|██████▉   | 95/138 [00:23<00:09,  4.42it/s, epoch=82, grad_norm=305.778, loss=313.0515, lr=9.02e-04]
Epoch 82 TRAIN:  71%|███████   | 98/138 [00:23<00:09,  4.44it/s, epoch=82, grad_norm=305.778, loss=313.0515, lr=9.02e-04]
Epoch 82 TRAIN:  71%|███████   | 98/138 [00:24<00:09,  4.44it/s, epoch=82, grad_norm=192.504, loss=225.6435, lr=9.02e-04]
Epoch 82 TRAIN:  71%|███████   | 98/138 [00:24<00:09,  4.44it/s, epoch=82, grad_norm=198.128, loss=295.6123, lr=9.03e-04]2026-04-16 20:34:44,646 DEBUG TRAIN | steps/sec 4.183| Batch 82/100 loss 295.612335 loss_att 234.009277 loss_ctc 439.352783 th_accuracy 0.409968 lr 9.0256e-04 grad_norm 198.127945 rank 0

Epoch 82 TRAIN:  71%|███████   | 98/138 [00:24<00:09,  4.44it/s, epoch=82, grad_norm=154.323, loss=316.6259, lr=9.03e-04]
Epoch 82 TRAIN:  73%|███████▎  | 101/138 [00:24<00:07,  4.63it/s, epoch=82, grad_norm=154.323, loss=316.6259, lr=9.03e-04]
Epoch 82 TRAIN:  73%|███████▎  | 101/138 [00:24<00:07,  4.63it/s, epoch=82, grad_norm=133.227, loss=205.5106, lr=9.03e-04]
Epoch 82 TRAIN:  73%|███████▎  | 101/138 [00:24<00:07,  4.63it/s, epoch=82, grad_norm=214.727, loss=306.3966, lr=9.03e-04]
Epoch 82 TRAIN:  73%|███████▎  | 101/138 [00:25<00:07,  4.63it/s, epoch=82, grad_norm=234.647, loss=301.1839, lr=9.03e-04]
Epoch 82 TRAIN:  75%|███████▌  | 104/138 [00:25<00:07,  4.49it/s, epoch=82, grad_norm=234.647, loss=301.1839, lr=9.03e-04]
Epoch 82 TRAIN:  75%|███████▌  | 104/138 [00:25<00:07,  4.49it/s, epoch=82, grad_norm=130.737, loss=217.5051, lr=9.03e-04]
Epoch 82 TRAIN:  75%|███████▌  | 104/138 [00:25<00:07,  4.49it/s, epoch=82, grad_norm=101.327, loss=251.7999, lr=9.03e-04]
Epoch 82 TRAIN:  75%|███████▌  | 104/138 [00:25<00:07,  4.49it/s, epoch=82, grad_norm=149.676, loss=231.1777, lr=9.03e-04]
Epoch 82 TRAIN:  78%|███████▊  | 107/138 [00:25<00:06,  4.45it/s, epoch=82, grad_norm=149.676, loss=231.1777, lr=9.03e-04]
Epoch 82 TRAIN:  78%|███████▊  | 107/138 [00:26<00:06,  4.45it/s, epoch=82, grad_norm=129.266, loss=261.0897, lr=9.03e-04]
Epoch 82 TRAIN:  78%|███████▊  | 107/138 [00:26<00:06,  4.45it/s, epoch=82, grad_norm=255.802, loss=268.6336, lr=9.03e-04]
Epoch 82 TRAIN:  78%|███████▊  | 107/138 [00:26<00:06,  4.45it/s, epoch=82, grad_norm=212.675, loss=186.4919, lr=9.03e-04]
Epoch 82 TRAIN:  80%|███████▉  | 110/138 [00:26<00:06,  4.37it/s, epoch=82, grad_norm=212.675, loss=186.4919, lr=9.03e-04]
Epoch 82 TRAIN:  80%|███████▉  | 110/138 [00:26<00:06,  4.37it/s, epoch=82, grad_norm=116.193, loss=264.0252, lr=9.03e-04]
Epoch 82 TRAIN:  80%|███████▉  | 110/138 [00:26<00:06,  4.37it/s, epoch=82, grad_norm=149.492, loss=260.6390, lr=9.04e-04]
Epoch 82 TRAIN:  80%|███████▉  | 110/138 [00:27<00:06,  4.37it/s, epoch=82, grad_norm=163.358, loss=321.8268, lr=9.04e-04]
Epoch 82 TRAIN:  82%|████████▏ | 113/138 [00:27<00:05,  4.58it/s, epoch=82, grad_norm=163.358, loss=321.8268, lr=9.04e-04]
Epoch 82 TRAIN:  82%|████████▏ | 113/138 [00:27<00:05,  4.58it/s, epoch=82, grad_norm=118.706, loss=258.2449, lr=9.04e-04]
Epoch 82 TRAIN:  82%|████████▏ | 113/138 [00:27<00:05,  4.58it/s, epoch=82, grad_norm=198.556, loss=220.8903, lr=9.04e-04]
Epoch 82 TRAIN:  82%|████████▏ | 113/138 [00:27<00:05,  4.58it/s, epoch=82, grad_norm=86.620, loss=222.6113, lr=9.04e-04] 
Epoch 82 TRAIN:  84%|████████▍ | 116/138 [00:27<00:04,  4.71it/s, epoch=82, grad_norm=86.620, loss=222.6113, lr=9.04e-04]
Epoch 82 TRAIN:  84%|████████▍ | 116/138 [00:27<00:04,  4.71it/s, epoch=82, grad_norm=115.836, loss=290.1628, lr=9.04e-04]
Epoch 82 TRAIN:  84%|████████▍ | 116/138 [00:28<00:04,  4.71it/s, epoch=82, grad_norm=134.252, loss=257.1437, lr=9.04e-04]
Epoch 82 TRAIN:  84%|████████▍ | 116/138 [00:28<00:04,  4.71it/s, epoch=82, grad_norm=169.845, loss=239.1985, lr=9.04e-04]
Epoch 82 TRAIN:  86%|████████▌ | 119/138 [00:28<00:04,  4.74it/s, epoch=82, grad_norm=169.845, loss=239.1985, lr=9.04e-04]
Epoch 82 TRAIN:  86%|████████▌ | 119/138 [00:28<00:04,  4.74it/s, epoch=82, grad_norm=203.786, loss=270.5339, lr=9.04e-04]
Epoch 82 TRAIN:  86%|████████▌ | 119/138 [00:28<00:04,  4.74it/s, epoch=82, grad_norm=246.043, loss=337.7344, lr=9.04e-04]
Epoch 82 TRAIN:  86%|████████▌ | 119/138 [00:29<00:04,  4.74it/s, epoch=82, grad_norm=130.802, loss=291.7305, lr=9.04e-04]
Epoch 82 TRAIN:  88%|████████▊ | 122/138 [00:29<00:03,  4.61it/s, epoch=82, grad_norm=130.802, loss=291.7305, lr=9.04e-04]
Epoch 82 TRAIN:  88%|████████▊ | 122/138 [00:29<00:03,  4.61it/s, epoch=82, grad_norm=221.535, loss=299.6212, lr=9.04e-04]
Epoch 82 TRAIN:  88%|████████▊ | 122/138 [00:29<00:03,  4.61it/s, epoch=82, grad_norm=169.448, loss=261.5249, lr=9.04e-04]
Epoch 82 TRAIN:  88%|████████▊ | 122/138 [00:29<00:03,  4.61it/s, epoch=82, grad_norm=356.830, loss=254.3235, lr=9.05e-04]
Epoch 82 TRAIN:  91%|█████████ | 125/138 [00:29<00:02,  4.69it/s, epoch=82, grad_norm=356.830, loss=254.3235, lr=9.05e-04]
Epoch 82 TRAIN:  91%|█████████ | 125/138 [00:29<00:02,  4.69it/s, epoch=82, grad_norm=175.326, loss=357.1499, lr=9.05e-04]
Epoch 82 TRAIN:  91%|█████████ | 125/138 [00:30<00:02,  4.69it/s, epoch=82, grad_norm=169.277, loss=273.7344, lr=9.05e-04]
Epoch 82 TRAIN:  91%|█████████ | 125/138 [00:30<00:02,  4.69it/s, epoch=82, grad_norm=183.372, loss=201.7491, lr=9.05e-04]
Epoch 82 TRAIN:  93%|█████████▎| 128/138 [00:30<00:02,  4.82it/s, epoch=82, grad_norm=183.372, loss=201.7491, lr=9.05e-04]
Epoch 82 TRAIN:  93%|█████████▎| 128/138 [00:30<00:02,  4.82it/s, epoch=82, grad_norm=216.854, loss=279.0598, lr=9.05e-04]
Epoch 82 TRAIN:  93%|█████████▎| 128/138 [00:30<00:02,  4.82it/s, epoch=82, grad_norm=169.873, loss=257.4152, lr=9.05e-04]
Epoch 82 TRAIN:  93%|█████████▎| 128/138 [00:30<00:02,  4.82it/s, epoch=82, grad_norm=140.523, loss=254.9097, lr=9.05e-04]
Epoch 82 TRAIN:  95%|█████████▍| 131/138 [00:30<00:01,  4.85it/s, epoch=82, grad_norm=140.523, loss=254.9097, lr=9.05e-04]
Epoch 82 TRAIN:  95%|█████████▍| 131/138 [00:31<00:01,  4.85it/s, epoch=82, grad_norm=231.845, loss=313.5493, lr=9.05e-04]
Epoch 82 TRAIN:  95%|█████████▍| 131/138 [00:31<00:01,  4.85it/s, epoch=82, grad_norm=279.601, loss=332.0170, lr=9.05e-04]
Epoch 82 TRAIN:  95%|█████████▍| 131/138 [00:31<00:01,  4.85it/s, epoch=82, grad_norm=166.534, loss=242.4004, lr=9.05e-04]
Epoch 82 TRAIN:  97%|█████████▋| 134/138 [00:31<00:00,  4.97it/s, epoch=82, grad_norm=166.534, loss=242.4004, lr=9.05e-04]
Epoch 82 TRAIN:  97%|█████████▋| 134/138 [00:31<00:00,  4.97it/s, epoch=82, grad_norm=195.873, loss=247.9032, lr=9.05e-04]
Epoch 82 TRAIN:  97%|█████████▋| 134/138 [00:31<00:00,  4.97it/s, epoch=82, grad_norm=201.399, loss=274.3050, lr=9.05e-04]
Epoch 82 TRAIN:  97%|█████████▋| 134/138 [00:31<00:00,  4.97it/s, epoch=82, grad_norm=166.532, loss=300.7400, lr=9.06e-04]
Epoch 82 TRAIN:  97%|█████████▋| 134/138 [00:32<00:00,  4.97it/s, epoch=82, grad_norm=206.979, loss=292.9882, lr=9.06e-04]
Epoch 82 TRAIN: 100%|██████████| 138/138 [00:32<00:00,  5.25it/s, epoch=82, grad_norm=206.979, loss=292.9882, lr=9.06e-04]
Epoch 82 TRAIN: 100%|██████████| 138/138 [00:32<00:00,  4.30it/s, epoch=82, grad_norm=206.979, loss=292.9882, lr=9.06e-04]

CV:   0%|          | 0/6 [00:00<?, ?it/s]/home/rookie/Documents/batuka/projects/wenet/wenet/wenet/utils/train_utils.py:689: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  amp_autocast(enabled=dtype is not None,

CV:  17%|█▋        | 1/6 [00:00<00:03,  1.33it/s]
                                                 
2026-04-16 20:34:53,546 INFO Epoch 82 Step 11320 CV info lr 9.0560e-04 cv_loss 461.017841685902 rank 0 acc 0.4059121459722519
2026-04-16 20:34:53,548 INFO [Rank 0] Checkpoint: save to checkpoint exp/conformer_mongolian/epoch_82.pt
2026-04-16 20:34:53,672 INFO Epoch 83 Step 11320 TRAIN info lr 9.0560e-04 rank 0
2026-04-16 20:34:53,694 INFO using accumulate grad, new batch size is 1 times larger than before

Epoch 83 TRAIN:   0%|          | 0/138 [00:00<?, ?it/s]
Epoch 83 TRAIN:   0%|          | 0/138 [00:04<?, ?it/s, epoch=83, grad_norm=241.728, loss=252.4581, lr=9.06e-04]
Epoch 83 TRAIN:   1%|          | 1/138 [00:04<10:28,  4.59s/it, epoch=83, grad_norm=241.728, loss=252.4581, lr=9.06e-04]
Epoch 83 TRAIN:   1%|          | 1/138 [00:04<10:28,  4.59s/it, epoch=83, grad_norm=491.671, loss=396.2969, lr=9.06e-04]
Epoch 83 TRAIN:   1%|          | 1/138 [00:04<10:28,  4.59s/it, epoch=83, grad_norm=453.475, loss=338.3961, lr=9.06e-04]
Epoch 83 TRAIN:   1%|          | 1/138 [00:04<10:28,  4.59s/it, epoch=83, grad_norm=147.600, loss=216.1227, lr=9.06e-04]
Epoch 83 TRAIN:   1%|          | 1/138 [00:05<10:28,  4.59s/it, epoch=83, grad_norm=172.973, loss=239.3376, lr=9.06e-04]
Epoch 83 TRAIN:   1%|          | 1/138 [00:05<10:28,  4.59s/it, epoch=83, grad_norm=372.577, loss=312.3557, lr=9.06e-04]
Epoch 83 TRAIN:   4%|▍         | 6/138 [00:05<01:26,  1.53it/s, epoch=83, grad_norm=372.577, loss=312.3557, lr=9.06e-04]
Epoch 83 TRAIN:   4%|▍         | 6/138 [00:05<01:26,  1.53it/s, epoch=83, grad_norm=358.809, loss=295.1474, lr=9.06e-04]
Epoch 83 TRAIN:   4%|▍         | 6/138 [00:05<01:26,  1.53it/s, epoch=83, grad_norm=209.650, loss=273.7275, lr=9.06e-04]
Epoch 83 TRAIN:   4%|▍         | 6/138 [00:05<01:26,  1.53it/s, epoch=83, grad_norm=103.553, loss=260.8859, lr=9.06e-04]
Epoch 83 TRAIN:   4%|▍         | 6/138 [00:05<01:26,  1.53it/s, epoch=83, grad_norm=157.060, loss=240.9260, lr=9.06e-04]
Epoch 83 TRAIN:   4%|▍         | 6/138 [00:05<01:26,  1.53it/s, epoch=83, grad_norm=124.312, loss=266.1284, lr=9.06e-04]
Epoch 83 TRAIN:   8%|▊         | 11/138 [00:05<00:45,  2.78it/s, epoch=83, grad_norm=124.312, loss=266.1284, lr=9.06e-04]
Epoch 83 TRAIN:   8%|▊         | 11/138 [00:05<00:45,  2.78it/s, epoch=83, grad_norm=131.467, loss=263.3958, lr=9.07e-04]
Epoch 83 TRAIN:   8%|▊         | 11/138 [00:06<00:45,  2.78it/s, epoch=83, grad_norm=105.160, loss=285.2001, lr=9.07e-04]
Epoch 83 TRAIN:   8%|▊         | 11/138 [00:06<00:45,  2.78it/s, epoch=83, grad_norm=92.086, loss=291.9498, lr=9.07e-04] 
Epoch 83 TRAIN:   8%|▊         | 11/138 [00:06<00:45,  2.78it/s, epoch=83, grad_norm=146.244, loss=262.4023, lr=9.07e-04]
Epoch 83 TRAIN:  11%|█         | 15/138 [00:06<00:34,  3.53it/s, epoch=83, grad_norm=146.244, loss=262.4023, lr=9.07e-04]
Epoch 83 TRAIN:  11%|█         | 15/138 [00:06<00:34,  3.53it/s, epoch=83, grad_norm=90.819, loss=274.9398, lr=9.07e-04] 
Epoch 83 TRAIN:  11%|█         | 15/138 [00:06<00:34,  3.53it/s, epoch=83, grad_norm=92.211, loss=267.6189, lr=9.07e-04]
Epoch 83 TRAIN:  11%|█         | 15/138 [00:06<00:34,  3.53it/s, epoch=83, grad_norm=91.176, loss=293.9284, lr=9.07e-04]
Epoch 83 TRAIN:  11%|█         | 15/138 [00:07<00:34,  3.53it/s, epoch=83, grad_norm=114.107, loss=289.7864, lr=9.07e-04]
Epoch 83 TRAIN:  14%|█▍        | 19/138 [00:07<00:29,  4.02it/s, epoch=83, grad_norm=114.107, loss=289.7864, lr=9.07e-04]
Epoch 83 TRAIN:  14%|█▍        | 19/138 [00:07<00:29,  4.02it/s, epoch=83, grad_norm=86.675, loss=298.6171, lr=9.07e-04] 
Epoch 83 TRAIN:  14%|█▍        | 19/138 [00:07<00:29,  4.02it/s, epoch=83, grad_norm=85.991, loss=289.1793, lr=9.07e-04]
Epoch 83 TRAIN:  14%|█▍        | 19/138 [00:07<00:29,  4.02it/s, epoch=83, grad_norm=108.909, loss=270.7902, lr=9.07e-04]
Epoch 83 TRAIN:  14%|█▍        | 19/138 [00:07<00:29,  4.02it/s, epoch=83, grad_norm=110.754, loss=272.9305, lr=9.07e-04]
Epoch 83 TRAIN:  17%|█▋        | 23/138 [00:07<00:26,  4.39it/s, epoch=83, grad_norm=110.754, loss=272.9305, lr=9.07e-04]
Epoch 83 TRAIN:  17%|█▋        | 23/138 [00:08<00:26,  4.39it/s, epoch=83, grad_norm=85.660, loss=293.2657, lr=9.08e-04] 
Epoch 83 TRAIN:  17%|█▋        | 23/138 [00:08<00:26,  4.39it/s, epoch=83, grad_norm=115.443, loss=260.9232, lr=9.08e-04]
Epoch 83 TRAIN:  17%|█▋        | 23/138 [00:08<00:26,  4.39it/s, epoch=83, grad_norm=125.242, loss=250.1888, lr=9.08e-04]
Epoch 83 TRAIN:  19%|█▉        | 26/138 [00:08<00:24,  4.67it/s, epoch=83, grad_norm=125.242, loss=250.1888, lr=9.08e-04]
Epoch 83 TRAIN:  19%|█▉        | 26/138 [00:08<00:24,  4.67it/s, epoch=83, grad_norm=196.228, loss=334.7864, lr=9.08e-04]
Epoch 83 TRAIN:  19%|█▉        | 26/138 [00:08<00:24,  4.67it/s, epoch=83, grad_norm=128.445, loss=261.4643, lr=9.08e-04]
Epoch 83 TRAIN:  19%|█▉        | 26/138 [00:08<00:24,  4.67it/s, epoch=83, grad_norm=122.204, loss=244.5847, lr=9.08e-04]
Epoch 83 TRAIN:  21%|██        | 29/138 [00:08<00:22,  4.79it/s, epoch=83, grad_norm=122.204, loss=244.5847, lr=9.08e-04]
Epoch 83 TRAIN:  21%|██        | 29/138 [00:09<00:22,  4.79it/s, epoch=83, grad_norm=75.943, loss=258.0221, lr=9.08e-04] 
Epoch 83 TRAIN:  21%|██        | 29/138 [00:09<00:22,  4.79it/s, epoch=83, grad_norm=93.262, loss=278.4792, lr=9.08e-04]
Epoch 83 TRAIN:  21%|██        | 29/138 [00:09<00:22,  4.79it/s, epoch=83, grad_norm=89.122, loss=293.9121, lr=9.08e-04]
Epoch 83 TRAIN:  23%|██▎       | 32/138 [00:09<00:22,  4.73it/s, epoch=83, grad_norm=89.122, loss=293.9121, lr=9.08e-04]
Epoch 83 TRAIN:  23%|██▎       | 32/138 [00:09<00:22,  4.73it/s, epoch=83, grad_norm=66.380, loss=244.3828, lr=9.08e-04]
Epoch 83 TRAIN:  23%|██▎       | 32/138 [00:10<00:22,  4.73it/s, epoch=83, grad_norm=79.966, loss=261.5343, lr=9.08e-04]
Epoch 83 TRAIN:  23%|██▎       | 32/138 [00:10<00:22,  4.73it/s, epoch=83, grad_norm=119.923, loss=254.8457, lr=9.08e-04]
Epoch 83 TRAIN:  25%|██▌       | 35/138 [00:10<00:21,  4.79it/s, epoch=83, grad_norm=119.923, loss=254.8457, lr=9.08e-04]
Epoch 83 TRAIN:  25%|██▌       | 35/138 [00:10<00:21,  4.79it/s, epoch=83, grad_norm=117.217, loss=232.4019, lr=9.08e-04]
Epoch 83 TRAIN:  25%|██▌       | 35/138 [00:10<00:21,  4.79it/s, epoch=83, grad_norm=79.189, loss=270.4437, lr=9.09e-04] 
Epoch 83 TRAIN:  25%|██▌       | 35/138 [00:10<00:21,  4.79it/s, epoch=83, grad_norm=101.543, loss=288.1385, lr=9.09e-04]
Epoch 83 TRAIN:  28%|██▊       | 38/138 [00:10<00:20,  4.89it/s, epoch=83, grad_norm=101.543, loss=288.1385, lr=9.09e-04]
Epoch 83 TRAIN:  28%|██▊       | 38/138 [00:11<00:20,  4.89it/s, epoch=83, grad_norm=106.566, loss=283.4265, lr=9.09e-04]
Epoch 83 TRAIN:  28%|██▊       | 38/138 [00:11<00:20,  4.89it/s, epoch=83, grad_norm=128.954, loss=253.7006, lr=9.09e-04]
Epoch 83 TRAIN:  28%|██▊       | 38/138 [00:11<00:20,  4.89it/s, epoch=83, grad_norm=91.669, loss=248.5101, lr=9.09e-04] 
Epoch 83 TRAIN:  30%|██▉       | 41/138 [00:11<00:20,  4.80it/s, epoch=83, grad_norm=91.669, loss=248.5101, lr=9.09e-04]
Epoch 83 TRAIN:  30%|██▉       | 41/138 [00:11<00:20,  4.80it/s, epoch=83, grad_norm=138.715, loss=314.4626, lr=9.09e-04]
Epoch 83 TRAIN:  30%|██▉       | 41/138 [00:11<00:20,  4.80it/s, epoch=83, grad_norm=107.798, loss=299.8038, lr=9.09e-04]
Epoch 83 TRAIN:  30%|██▉       | 41/138 [00:12<00:20,  4.80it/s, epoch=83, grad_norm=85.649, loss=274.8054, lr=9.09e-04] 
Epoch 83 TRAIN:  32%|███▏      | 44/138 [00:12<00:19,  4.83it/s, epoch=83, grad_norm=85.649, loss=274.8054, lr=9.09e-04]
Epoch 83 TRAIN:  32%|███▏      | 44/138 [00:12<00:19,  4.83it/s, epoch=83, grad_norm=98.563, loss=282.1944, lr=9.09e-04]
Epoch 83 TRAIN:  32%|███▏      | 44/138 [00:12<00:19,  4.83it/s, epoch=83, grad_norm=103.865, loss=276.7108, lr=9.09e-04]
Epoch 83 TRAIN:  32%|███▏      | 44/138 [00:12<00:19,  4.83it/s, epoch=83, grad_norm=100.057, loss=259.2950, lr=9.09e-04]
Epoch 83 TRAIN:  34%|███▍      | 47/138 [00:12<00:18,  4.83it/s, epoch=83, grad_norm=100.057, loss=259.2950, lr=9.09e-04]
Epoch 83 TRAIN:  34%|███▍      | 47/138 [00:12<00:18,  4.83it/s, epoch=83, grad_norm=120.669, loss=313.6161, lr=9.09e-04]
Epoch 83 TRAIN:  34%|███▍      | 47/138 [00:13<00:18,  4.83it/s, epoch=83, grad_norm=96.026, loss=316.0524, lr=9.10e-04] 
Epoch 83 TRAIN:  34%|███▍      | 47/138 [00:13<00:18,  4.83it/s, epoch=83, grad_norm=137.758, loss=278.9855, lr=9.10e-04]
Epoch 83 TRAIN:  36%|███▌      | 50/138 [00:13<00:18,  4.66it/s, epoch=83, grad_norm=137.758, loss=278.9855, lr=9.10e-04]
Epoch 83 TRAIN:  36%|███▌      | 50/138 [00:13<00:18,  4.66it/s, epoch=83, grad_norm=154.009, loss=231.4191, lr=9.10e-04]
Epoch 83 TRAIN:  36%|███▌      | 50/138 [00:13<00:18,  4.66it/s, epoch=83, grad_norm=95.060, loss=254.2782, lr=9.10e-04] 
Epoch 83 TRAIN:  36%|███▌      | 50/138 [00:14<00:18,  4.66it/s, epoch=83, grad_norm=90.514, loss=263.1631, lr=9.10e-04]
Epoch 83 TRAIN:  38%|███▊      | 53/138 [00:14<00:18,  4.48it/s, epoch=83, grad_norm=90.514, loss=263.1631, lr=9.10e-04]
Epoch 83 TRAIN:  38%|███▊      | 53/138 [00:14<00:18,  4.48it/s, epoch=83, grad_norm=149.890, loss=266.0526, lr=9.10e-04]
Epoch 83 TRAIN:  38%|███▊      | 53/138 [00:14<00:18,  4.48it/s, epoch=83, grad_norm=123.742, loss=293.6525, lr=9.10e-04]
Epoch 83 TRAIN:  38%|███▊      | 53/138 [00:14<00:18,  4.48it/s, epoch=83, grad_norm=105.597, loss=260.3018, lr=9.10e-04]
Epoch 83 TRAIN:  41%|████      | 56/138 [00:14<00:17,  4.63it/s, epoch=83, grad_norm=105.597, loss=260.3018, lr=9.10e-04]
Epoch 83 TRAIN:  41%|████      | 56/138 [00:14<00:17,  4.63it/s, epoch=83, grad_norm=110.272, loss=208.4734, lr=9.10e-04]
Epoch 83 TRAIN:  41%|████      | 56/138 [00:15<00:17,  4.63it/s, epoch=83, grad_norm=85.852, loss=276.1198, lr=9.10e-04] 
Epoch 83 TRAIN:  41%|████      | 56/138 [00:15<00:17,  4.63it/s, epoch=83, grad_norm=99.947, loss=271.6405, lr=9.10e-04]
Epoch 83 TRAIN:  43%|████▎     | 59/138 [00:15<00:16,  4.75it/s, epoch=83, grad_norm=99.947, loss=271.6405, lr=9.10e-04]
Epoch 83 TRAIN:  43%|████▎     | 59/138 [00:15<00:16,  4.75it/s, epoch=83, grad_norm=177.084, loss=330.9480, lr=9.10e-04]
Epoch 83 TRAIN:  43%|████▎     | 59/138 [00:15<00:16,  4.75it/s, epoch=83, grad_norm=199.680, loss=330.0931, lr=9.10e-04]
Epoch 83 TRAIN:  43%|████▎     | 59/138 [00:16<00:16,  4.75it/s, epoch=83, grad_norm=131.043, loss=233.5158, lr=9.11e-04]
Epoch 83 TRAIN:  45%|████▍     | 62/138 [00:16<00:17,  4.41it/s, epoch=83, grad_norm=131.043, loss=233.5158, lr=9.11e-04]
Epoch 83 TRAIN:  45%|████▍     | 62/138 [00:16<00:17,  4.41it/s, epoch=83, grad_norm=103.968, loss=299.5687, lr=9.11e-04]
Epoch 83 TRAIN:  45%|████▍     | 62/138 [00:16<00:17,  4.41it/s, epoch=83, grad_norm=111.508, loss=284.3706, lr=9.11e-04]
Epoch 83 TRAIN:  45%|████▍     | 62/138 [00:16<00:17,  4.41it/s, epoch=83, grad_norm=115.875, loss=260.1819, lr=9.11e-04]
Epoch 83 TRAIN:  47%|████▋     | 65/138 [00:16<00:16,  4.42it/s, epoch=83, grad_norm=115.875, loss=260.1819, lr=9.11e-04]
Epoch 83 TRAIN:  47%|████▋     | 65/138 [00:17<00:16,  4.42it/s, epoch=83, grad_norm=132.117, loss=260.8171, lr=9.11e-04]
Epoch 83 TRAIN:  47%|████▋     | 65/138 [00:17<00:16,  4.42it/s, epoch=83, grad_norm=107.931, loss=299.6768, lr=9.11e-04]
Epoch 83 TRAIN:  47%|████▋     | 65/138 [00:17<00:16,  4.42it/s, epoch=83, grad_norm=124.558, loss=290.5999, lr=9.11e-04]
Epoch 83 TRAIN:  49%|████▉     | 68/138 [00:17<00:16,  4.36it/s, epoch=83, grad_norm=124.558, loss=290.5999, lr=9.11e-04]
Epoch 83 TRAIN:  49%|████▉     | 68/138 [00:17<00:16,  4.36it/s, epoch=83, grad_norm=170.136, loss=222.9658, lr=9.11e-04]
Epoch 83 TRAIN:  49%|████▉     | 68/138 [00:17<00:16,  4.36it/s, epoch=83, grad_norm=140.156, loss=322.6464, lr=9.11e-04]
Epoch 83 TRAIN:  49%|████▉     | 68/138 [00:18<00:16,  4.36it/s, epoch=83, grad_norm=127.470, loss=291.1661, lr=9.11e-04]
Epoch 83 TRAIN:  51%|█████▏    | 71/138 [00:18<00:14,  4.58it/s, epoch=83, grad_norm=127.470, loss=291.1661, lr=9.11e-04]
Epoch 83 TRAIN:  51%|█████▏    | 71/138 [00:18<00:14,  4.58it/s, epoch=83, grad_norm=120.806, loss=294.8278, lr=9.11e-04]
Epoch 83 TRAIN:  51%|█████▏    | 71/138 [00:18<00:14,  4.58it/s, epoch=83, grad_norm=103.010, loss=234.6518, lr=9.11e-04]
Epoch 83 TRAIN:  51%|█████▏    | 71/138 [00:18<00:14,  4.58it/s, epoch=83, grad_norm=106.235, loss=297.3126, lr=9.12e-04]
Epoch 83 TRAIN:  54%|█████▎    | 74/138 [00:18<00:14,  4.55it/s, epoch=83, grad_norm=106.235, loss=297.3126, lr=9.12e-04]
Epoch 83 TRAIN:  54%|█████▎    | 74/138 [00:18<00:14,  4.55it/s, epoch=83, grad_norm=181.826, loss=249.4504, lr=9.12e-04]
Epoch 83 TRAIN:  54%|█████▎    | 74/138 [00:19<00:14,  4.55it/s, epoch=83, grad_norm=141.347, loss=297.8922, lr=9.12e-04]
Epoch 83 TRAIN:  54%|█████▎    | 74/138 [00:19<00:14,  4.55it/s, epoch=83, grad_norm=167.020, loss=307.9703, lr=9.12e-04]
Epoch 83 TRAIN:  56%|█████▌    | 77/138 [00:19<00:13,  4.59it/s, epoch=83, grad_norm=167.020, loss=307.9703, lr=9.12e-04]
Epoch 83 TRAIN:  56%|█████▌    | 77/138 [00:19<00:13,  4.59it/s, epoch=83, grad_norm=127.830, loss=271.4141, lr=9.12e-04]
Epoch 83 TRAIN:  56%|█████▌    | 77/138 [00:19<00:13,  4.59it/s, epoch=83, grad_norm=153.816, loss=230.3245, lr=9.12e-04]
Epoch 83 TRAIN:  56%|█████▌    | 77/138 [00:20<00:13,  4.59it/s, epoch=83, grad_norm=90.800, loss=239.8672, lr=9.12e-04] 
Epoch 83 TRAIN:  58%|█████▊    | 80/138 [00:20<00:12,  4.57it/s, epoch=83, grad_norm=90.800, loss=239.8672, lr=9.12e-04]
Epoch 83 TRAIN:  58%|█████▊    | 80/138 [00:20<00:12,  4.57it/s, epoch=83, grad_norm=155.828, loss=216.2274, lr=9.12e-04]
Epoch 83 TRAIN:  58%|█████▊    | 80/138 [00:20<00:12,  4.57it/s, epoch=83, grad_norm=98.972, loss=249.8946, lr=9.12e-04] 
Epoch 83 TRAIN:  58%|█████▊    | 80/138 [00:20<00:12,  4.57it/s, epoch=83, grad_norm=123.936, loss=267.3526, lr=9.12e-04]
Epoch 83 TRAIN:  60%|██████    | 83/138 [00:20<00:12,  4.52it/s, epoch=83, grad_norm=123.936, loss=267.3526, lr=9.12e-04]
Epoch 83 TRAIN:  60%|██████    | 83/138 [00:20<00:12,  4.52it/s, epoch=83, grad_norm=304.773, loss=349.2659, lr=9.12e-04]
Epoch 83 TRAIN:  60%|██████    | 83/138 [00:21<00:12,  4.52it/s, epoch=83, grad_norm=101.857, loss=237.7307, lr=9.12e-04]
Epoch 83 TRAIN:  60%|██████    | 83/138 [00:21<00:12,  4.52it/s, epoch=83, grad_norm=99.608, loss=219.7381, lr=9.12e-04] 
Epoch 83 TRAIN:  62%|██████▏   | 86/138 [00:21<00:12,  4.31it/s, epoch=83, grad_norm=99.608, loss=219.7381, lr=9.12e-04]
Epoch 83 TRAIN:  62%|██████▏   | 86/138 [00:21<00:12,  4.31it/s, epoch=83, grad_norm=114.233, loss=269.6725, lr=9.13e-04]
Epoch 83 TRAIN:  62%|██████▏   | 86/138 [00:21<00:12,  4.31it/s, epoch=83, grad_norm=100.402, loss=231.8752, lr=9.13e-04]
Epoch 83 TRAIN:  62%|██████▏   | 86/138 [00:22<00:12,  4.31it/s, epoch=83, grad_norm=116.932, loss=280.3217, lr=9.13e-04]
Epoch 83 TRAIN:  64%|██████▍   | 89/138 [00:22<00:10,  4.65it/s, epoch=83, grad_norm=116.932, loss=280.3217, lr=9.13e-04]
Epoch 83 TRAIN:  64%|██████▍   | 89/138 [00:22<00:10,  4.65it/s, epoch=83, grad_norm=105.840, loss=275.5255, lr=9.13e-04]
Epoch 83 TRAIN:  64%|██████▍   | 89/138 [00:22<00:10,  4.65it/s, epoch=83, grad_norm=202.316, loss=250.6041, lr=9.13e-04]
Epoch 83 TRAIN:  64%|██████▍   | 89/138 [00:22<00:10,  4.65it/s, epoch=83, grad_norm=154.307, loss=307.0833, lr=9.13e-04]
Epoch 83 TRAIN:  67%|██████▋   | 92/138 [00:22<00:10,  4.58it/s, epoch=83, grad_norm=154.307, loss=307.0833, lr=9.13e-04]
Epoch 83 TRAIN:  67%|██████▋   | 92/138 [00:22<00:10,  4.58it/s, epoch=83, grad_norm=183.108, loss=187.4630, lr=9.13e-04]
Epoch 83 TRAIN:  67%|██████▋   | 92/138 [00:23<00:10,  4.58it/s, epoch=83, grad_norm=112.128, loss=291.2927, lr=9.13e-04]
Epoch 83 TRAIN:  67%|██████▋   | 92/138 [00:23<00:10,  4.58it/s, epoch=83, grad_norm=165.339, loss=298.0129, lr=9.13e-04]
Epoch 83 TRAIN:  69%|██████▉   | 95/138 [00:23<00:09,  4.54it/s, epoch=83, grad_norm=165.339, loss=298.0129, lr=9.13e-04]
Epoch 83 TRAIN:  69%|██████▉   | 95/138 [00:23<00:09,  4.54it/s, epoch=83, grad_norm=120.250, loss=242.2283, lr=9.13e-04]
Epoch 83 TRAIN:  69%|██████▉   | 95/138 [00:23<00:09,  4.54it/s, epoch=83, grad_norm=110.494, loss=246.7163, lr=9.13e-04]
Epoch 83 TRAIN:  69%|██████▉   | 95/138 [00:23<00:09,  4.54it/s, epoch=83, grad_norm=142.477, loss=220.7421, lr=9.13e-04]
Epoch 83 TRAIN:  71%|███████   | 98/138 [00:23<00:08,  4.74it/s, epoch=83, grad_norm=142.477, loss=220.7421, lr=9.13e-04]
Epoch 83 TRAIN:  71%|███████   | 98/138 [00:24<00:08,  4.74it/s, epoch=83, grad_norm=145.232, loss=230.2265, lr=9.14e-04]
Epoch 83 TRAIN:  71%|███████   | 98/138 [00:24<00:08,  4.74it/s, epoch=83, grad_norm=80.878, loss=214.2165, lr=9.14e-04] 2026-04-16 20:35:18,046 DEBUG TRAIN | steps/sec 4.132| Batch 83/100 loss 214.216537 loss_att 165.163513 loss_ctc 328.673584 th_accuracy 0.458065 lr 9.1360e-04 grad_norm 80.878441 rank 0

Epoch 83 TRAIN:  71%|███████   | 98/138 [00:24<00:08,  4.74it/s, epoch=83, grad_norm=135.188, loss=147.3728, lr=9.14e-04]
Epoch 83 TRAIN:  73%|███████▎  | 101/138 [00:24<00:07,  4.76it/s, epoch=83, grad_norm=135.188, loss=147.3728, lr=9.14e-04]
Epoch 83 TRAIN:  73%|███████▎  | 101/138 [00:24<00:07,  4.76it/s, epoch=83, grad_norm=239.329, loss=288.5625, lr=9.14e-04]
Epoch 83 TRAIN:  73%|███████▎  | 101/138 [00:25<00:07,  4.76it/s, epoch=83, grad_norm=96.379, loss=206.6379, lr=9.14e-04] 
Epoch 83 TRAIN:  73%|███████▎  | 101/138 [00:25<00:07,  4.76it/s, epoch=83, grad_norm=154.640, loss=251.6230, lr=9.14e-04]
Epoch 83 TRAIN:  75%|███████▌  | 104/138 [00:25<00:07,  4.56it/s, epoch=83, grad_norm=154.640, loss=251.6230, lr=9.14e-04]
Epoch 83 TRAIN:  75%|███████▌  | 104/138 [00:25<00:07,  4.56it/s, epoch=83, grad_norm=105.405, loss=217.8859, lr=9.14e-04]
Epoch 83 TRAIN:  75%|███████▌  | 104/138 [00:25<00:07,  4.56it/s, epoch=83, grad_norm=291.992, loss=336.2825, lr=9.14e-04]
Epoch 83 TRAIN:  75%|███████▌  | 104/138 [00:26<00:07,  4.56it/s, epoch=83, grad_norm=170.152, loss=304.2295, lr=9.14e-04]
Epoch 83 TRAIN:  78%|███████▊  | 107/138 [00:26<00:06,  4.46it/s, epoch=83, grad_norm=170.152, loss=304.2295, lr=9.14e-04]
Epoch 83 TRAIN:  78%|███████▊  | 107/138 [00:26<00:06,  4.46it/s, epoch=83, grad_norm=133.733, loss=272.3099, lr=9.14e-04]
Epoch 83 TRAIN:  78%|███████▊  | 107/138 [00:26<00:06,  4.46it/s, epoch=83, grad_norm=133.667, loss=241.5551, lr=9.14e-04]
Epoch 83 TRAIN:  78%|███████▊  | 107/138 [00:26<00:06,  4.46it/s, epoch=83, grad_norm=134.286, loss=235.1862, lr=9.14e-04]
Epoch 83 TRAIN:  80%|███████▉  | 110/138 [00:26<00:06,  4.49it/s, epoch=83, grad_norm=134.286, loss=235.1862, lr=9.14e-04]
Epoch 83 TRAIN:  80%|███████▉  | 110/138 [00:26<00:06,  4.49it/s, epoch=83, grad_norm=181.139, loss=271.4851, lr=9.14e-04]
Epoch 83 TRAIN:  80%|███████▉  | 110/138 [00:27<00:06,  4.49it/s, epoch=83, grad_norm=206.614, loss=202.9831, lr=9.15e-04]
Epoch 83 TRAIN:  80%|███████▉  | 110/138 [00:27<00:06,  4.49it/s, epoch=83, grad_norm=172.318, loss=213.1804, lr=9.15e-04]
Epoch 83 TRAIN:  82%|████████▏ | 113/138 [00:27<00:05,  4.46it/s, epoch=83, grad_norm=172.318, loss=213.1804, lr=9.15e-04]
Epoch 83 TRAIN:  82%|████████▏ | 113/138 [00:27<00:05,  4.46it/s, epoch=83, grad_norm=196.756, loss=305.1835, lr=9.15e-04]
Epoch 83 TRAIN:  82%|████████▏ | 113/138 [00:27<00:05,  4.46it/s, epoch=83, grad_norm=125.751, loss=272.7652, lr=9.15e-04]
Epoch 83 TRAIN:  82%|████████▏ | 113/138 [00:27<00:05,  4.46it/s, epoch=83, grad_norm=146.881, loss=249.3871, lr=9.15e-04]
Epoch 83 TRAIN:  84%|████████▍ | 116/138 [00:27<00:04,  4.55it/s, epoch=83, grad_norm=146.881, loss=249.3871, lr=9.15e-04]
Epoch 83 TRAIN:  84%|████████▍ | 116/138 [00:28<00:04,  4.55it/s, epoch=83, grad_norm=120.051, loss=226.8101, lr=9.15e-04]
Epoch 83 TRAIN:  84%|████████▍ | 116/138 [00:28<00:04,  4.55it/s, epoch=83, grad_norm=208.000, loss=283.2603, lr=9.15e-04]
Epoch 83 TRAIN:  84%|████████▍ | 116/138 [00:28<00:04,  4.55it/s, epoch=83, grad_norm=282.537, loss=315.2982, lr=9.15e-04]
Epoch 83 TRAIN:  86%|████████▌ | 119/138 [00:28<00:04,  4.64it/s, epoch=83, grad_norm=282.537, loss=315.2982, lr=9.15e-04]
Epoch 83 TRAIN:  86%|████████▌ | 119/138 [00:28<00:04,  4.64it/s, epoch=83, grad_norm=196.419, loss=314.8619, lr=9.15e-04]
Epoch 83 TRAIN:  86%|████████▌ | 119/138 [00:29<00:04,  4.64it/s, epoch=83, grad_norm=179.973, loss=240.4366, lr=9.15e-04]
Epoch 83 TRAIN:  86%|████████▌ | 119/138 [00:29<00:04,  4.64it/s, epoch=83, grad_norm=174.609, loss=285.5692, lr=9.15e-04]
Epoch 83 TRAIN:  88%|████████▊ | 122/138 [00:29<00:03,  4.43it/s, epoch=83, grad_norm=174.609, loss=285.5692, lr=9.15e-04]
Epoch 83 TRAIN:  88%|████████▊ | 122/138 [00:29<00:03,  4.43it/s, epoch=83, grad_norm=317.757, loss=339.6705, lr=9.15e-04]
Epoch 83 TRAIN:  88%|████████▊ | 122/138 [00:29<00:03,  4.43it/s, epoch=83, grad_norm=172.341, loss=273.0364, lr=9.16e-04]
Epoch 83 TRAIN:  88%|████████▊ | 122/138 [00:29<00:03,  4.43it/s, epoch=83, grad_norm=115.391, loss=270.9262, lr=9.16e-04]
Epoch 83 TRAIN:  91%|█████████ | 125/138 [00:29<00:02,  4.58it/s, epoch=83, grad_norm=115.391, loss=270.9262, lr=9.16e-04]
Epoch 83 TRAIN:  91%|█████████ | 125/138 [00:30<00:02,  4.58it/s, epoch=83, grad_norm=209.644, loss=333.4702, lr=9.16e-04]
Epoch 83 TRAIN:  91%|█████████ | 125/138 [00:30<00:02,  4.58it/s, epoch=83, grad_norm=168.030, loss=261.0500, lr=9.16e-04]
Epoch 83 TRAIN:  91%|█████████ | 125/138 [00:30<00:02,  4.58it/s, epoch=83, grad_norm=262.233, loss=227.9744, lr=9.16e-04]
Epoch 83 TRAIN:  93%|█████████▎| 128/138 [00:30<00:02,  4.80it/s, epoch=83, grad_norm=262.233, loss=227.9744, lr=9.16e-04]
Epoch 83 TRAIN:  93%|█████████▎| 128/138 [00:30<00:02,  4.80it/s, epoch=83, grad_norm=199.132, loss=320.4290, lr=9.16e-04]
Epoch 83 TRAIN:  93%|█████████▎| 128/138 [00:30<00:02,  4.80it/s, epoch=83, grad_norm=155.898, loss=242.3434, lr=9.16e-04]
Epoch 83 TRAIN:  93%|█████████▎| 128/138 [00:31<00:02,  4.80it/s, epoch=83, grad_norm=307.586, loss=304.9448, lr=9.16e-04]
Epoch 83 TRAIN:  95%|█████████▍| 131/138 [00:31<00:01,  5.10it/s, epoch=83, grad_norm=307.586, loss=304.9448, lr=9.16e-04]
Epoch 83 TRAIN:  95%|█████████▍| 131/138 [00:31<00:01,  5.10it/s, epoch=83, grad_norm=214.675, loss=338.7181, lr=9.16e-04]
Epoch 83 TRAIN:  95%|█████████▍| 131/138 [00:31<00:01,  5.10it/s, epoch=83, grad_norm=182.934, loss=239.7083, lr=9.16e-04]
Epoch 83 TRAIN:  95%|█████████▍| 131/138 [00:31<00:01,  5.10it/s, epoch=83, grad_norm=273.829, loss=351.3267, lr=9.16e-04]
Epoch 83 TRAIN:  97%|█████████▋| 134/138 [00:31<00:00,  5.02it/s, epoch=83, grad_norm=273.829, loss=351.3267, lr=9.16e-04]
Epoch 83 TRAIN:  97%|█████████▋| 134/138 [00:31<00:00,  5.02it/s, epoch=83, grad_norm=309.682, loss=382.1247, lr=9.16e-04]
Epoch 83 TRAIN:  97%|█████████▋| 134/138 [00:31<00:00,  5.02it/s, epoch=83, grad_norm=170.516, loss=310.9333, lr=9.16e-04]
Epoch 83 TRAIN:  97%|█████████▋| 134/138 [00:32<00:00,  5.02it/s, epoch=83, grad_norm=259.092, loss=250.7176, lr=9.17e-04]
Epoch 83 TRAIN:  99%|█████████▉| 137/138 [00:32<00:00,  4.27it/s, epoch=83, grad_norm=259.092, loss=250.7176, lr=9.17e-04]

CV:   0%|          | 0/6 [00:00<?, ?it/s]/home/rookie/Documents/batuka/projects/wenet/wenet/wenet/utils/train_utils.py:689: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  amp_autocast(enabled=dtype is not None,

CV:  17%|█▋        | 1/6 [00:00<00:03,  1.32it/s]
                                                 
2026-04-16 20:35:26,665 INFO Epoch 83 Step 11457 CV info lr 9.1656e-04 cv_loss 453.9970814098011 rank 0 acc 0.37615201870600384
2026-04-16 20:35:26,667 INFO [Rank 0] Checkpoint: save to checkpoint exp/conformer_mongolian/epoch_83.pt
2026-04-16 20:35:26,796 INFO Epoch 84 Step 11457 TRAIN info lr 9.1656e-04 rank 0
2026-04-16 20:35:26,813 INFO using accumulate grad, new batch size is 1 times larger than before

Epoch 84 TRAIN:   0%|          | 0/137 [00:00<?, ?it/s]
Epoch 84 TRAIN:   0%|          | 0/137 [00:03<?, ?it/s, epoch=84, grad_norm=283.227, loss=237.7202, lr=9.17e-04]
Epoch 84 TRAIN:   1%|          | 1/137 [00:03<08:51,  3.91s/it, epoch=84, grad_norm=283.227, loss=237.7202, lr=9.17e-04]
Epoch 84 TRAIN:   1%|          | 1/137 [00:04<08:51,  3.91s/it, epoch=84, grad_norm=303.671, loss=224.7403, lr=9.17e-04]
Epoch 84 TRAIN:   1%|          | 1/137 [00:04<08:51,  3.91s/it, epoch=84, grad_norm=352.611, loss=347.3782, lr=9.17e-04]
Epoch 84 TRAIN:   1%|          | 1/137 [00:04<08:51,  3.91s/it, epoch=84, grad_norm=544.412, loss=399.7102, lr=9.17e-04]
Epoch 84 TRAIN:   1%|          | 1/137 [00:04<08:51,  3.91s/it, epoch=84, grad_norm=245.096, loss=226.7754, lr=9.17e-04]
Epoch 84 TRAIN:   1%|          | 1/137 [00:04<08:51,  3.91s/it, epoch=84, grad_norm=238.666, loss=277.4350, lr=9.17e-04]
Epoch 84 TRAIN:   4%|▍         | 6/137 [00:04<01:15,  1.74it/s, epoch=84, grad_norm=238.666, loss=277.4350, lr=9.17e-04]
Epoch 84 TRAIN:   4%|▍         | 6/137 [00:04<01:15,  1.74it/s, epoch=84, grad_norm=322.361, loss=307.7255, lr=9.17e-04]
Epoch 84 TRAIN:   4%|▍         | 6/137 [00:04<01:15,  1.74it/s, epoch=84, grad_norm=261.842, loss=315.5015, lr=9.17e-04]
Epoch 84 TRAIN:   4%|▍         | 6/137 [00:04<01:15,  1.74it/s, epoch=84, grad_norm=144.521, loss=291.1098, lr=9.17e-04]
Epoch 84 TRAIN:   4%|▍         | 6/137 [00:04<01:15,  1.74it/s, epoch=84, grad_norm=131.121, loss=275.8629, lr=9.17e-04]
Epoch 84 TRAIN:   4%|▍         | 6/137 [00:05<01:15,  1.74it/s, epoch=84, grad_norm=86.859, loss=244.9986, lr=9.17e-04] 
Epoch 84 TRAIN:   8%|▊         | 11/137 [00:05<00:40,  3.10it/s, epoch=84, grad_norm=86.859, loss=244.9986, lr=9.17e-04]
Epoch 84 TRAIN:   8%|▊         | 11/137 [00:05<00:40,  3.10it/s, epoch=84, grad_norm=116.535, loss=264.1306, lr=9.18e-04]
Epoch 84 TRAIN:   8%|▊         | 11/137 [00:05<00:40,  3.10it/s, epoch=84, grad_norm=128.301, loss=308.9720, lr=9.18e-04]
Epoch 84 TRAIN:   8%|▊         | 11/137 [00:05<00:40,  3.10it/s, epoch=84, grad_norm=151.722, loss=274.2196, lr=9.18e-04]
Epoch 84 TRAIN:   8%|▊         | 11/137 [00:05<00:40,  3.10it/s, epoch=84, grad_norm=100.676, loss=288.8297, lr=9.18e-04]
Epoch 84 TRAIN:  11%|█         | 15/137 [00:05<00:32,  3.80it/s, epoch=84, grad_norm=100.676, loss=288.8297, lr=9.18e-04]
Epoch 84 TRAIN:  11%|█         | 15/137 [00:05<00:32,  3.80it/s, epoch=84, grad_norm=97.961, loss=272.1654, lr=9.18e-04] 
Epoch 84 TRAIN:  11%|█         | 15/137 [00:06<00:32,  3.80it/s, epoch=84, grad_norm=84.445, loss=277.2061, lr=9.18e-04]
Epoch 84 TRAIN:  11%|█         | 15/137 [00:06<00:32,  3.80it/s, epoch=84, grad_norm=103.502, loss=291.2214, lr=9.18e-04]
Epoch 84 TRAIN:  11%|█         | 15/137 [00:06<00:32,  3.80it/s, epoch=84, grad_norm=89.442, loss=266.2158, lr=9.18e-04] 
Epoch 84 TRAIN:  14%|█▍        | 19/137 [00:06<00:29,  4.06it/s, epoch=84, grad_norm=89.442, loss=266.2158, lr=9.18e-04]
Epoch 84 TRAIN:  14%|█▍        | 19/137 [00:06<00:29,  4.06it/s, epoch=84, grad_norm=133.018, loss=276.9827, lr=9.18e-04]
Epoch 84 TRAIN:  14%|█▍        | 19/137 [00:07<00:29,  4.06it/s, epoch=84, grad_norm=118.724, loss=285.5056, lr=9.18e-04]
Epoch 84 TRAIN:  14%|█▍        | 19/137 [00:07<00:29,  4.06it/s, epoch=84, grad_norm=126.845, loss=305.3393, lr=9.18e-04]
Epoch 84 TRAIN:  16%|█▌        | 22/137 [00:07<00:26,  4.27it/s, epoch=84, grad_norm=126.845, loss=305.3393, lr=9.18e-04]
Epoch 84 TRAIN:  16%|█▌        | 22/137 [00:07<00:26,  4.27it/s, epoch=84, grad_norm=108.083, loss=269.4642, lr=9.18e-04]
Epoch 84 TRAIN:  16%|█▌        | 22/137 [00:07<00:26,  4.27it/s, epoch=84, grad_norm=88.173, loss=291.5218, lr=9.18e-04] 
Epoch 84 TRAIN:  16%|█▌        | 22/137 [00:07<00:26,  4.27it/s, epoch=84, grad_norm=118.774, loss=295.4345, lr=9.19e-04]
Epoch 84 TRAIN:  18%|█▊        | 25/137 [00:07<00:25,  4.40it/s, epoch=84, grad_norm=118.774, loss=295.4345, lr=9.19e-04]
Epoch 84 TRAIN:  18%|█▊        | 25/137 [00:08<00:25,  4.40it/s, epoch=84, grad_norm=93.789, loss=308.4125, lr=9.19e-04] 
Epoch 84 TRAIN:  18%|█▊        | 25/137 [00:08<00:25,  4.40it/s, epoch=84, grad_norm=191.221, loss=240.2472, lr=9.19e-04]
Epoch 84 TRAIN:  18%|█▊        | 25/137 [00:08<00:25,  4.40it/s, epoch=84, grad_norm=174.639, loss=303.4223, lr=9.19e-04]
Epoch 84 TRAIN:  20%|██        | 28/137 [00:08<00:23,  4.56it/s, epoch=84, grad_norm=174.639, loss=303.4223, lr=9.19e-04]
Epoch 84 TRAIN:  20%|██        | 28/137 [00:08<00:23,  4.56it/s, epoch=84, grad_norm=165.511, loss=251.7769, lr=9.19e-04]
Epoch 84 TRAIN:  20%|██        | 28/137 [00:08<00:23,  4.56it/s, epoch=84, grad_norm=101.979, loss=330.6883, lr=9.19e-04]
Epoch 84 TRAIN:  20%|██        | 28/137 [00:09<00:23,  4.56it/s, epoch=84, grad_norm=97.066, loss=291.2160, lr=9.19e-04] 
Epoch 84 TRAIN:  23%|██▎       | 31/137 [00:09<00:22,  4.61it/s, epoch=84, grad_norm=97.066, loss=291.2160, lr=9.19e-04]
Epoch 84 TRAIN:  23%|██▎       | 31/137 [00:09<00:22,  4.61it/s, epoch=84, grad_norm=97.797, loss=279.2833, lr=9.19e-04]
Epoch 84 TRAIN:  23%|██▎       | 31/137 [00:09<00:22,  4.61it/s, epoch=84, grad_norm=139.166, loss=241.4809, lr=9.19e-04]
Epoch 84 TRAIN:  23%|██▎       | 31/137 [00:09<00:22,  4.61it/s, epoch=84, grad_norm=80.788, loss=256.5699, lr=9.19e-04] 
Epoch 84 TRAIN:  25%|██▍       | 34/137 [00:09<00:22,  4.64it/s, epoch=84, grad_norm=80.788, loss=256.5699, lr=9.19e-04]
Epoch 84 TRAIN:  25%|██▍       | 34/137 [00:09<00:22,  4.64it/s, epoch=84, grad_norm=104.145, loss=296.4819, lr=9.19e-04]
Epoch 84 TRAIN:  25%|██▍       | 34/137 [00:10<00:22,  4.64it/s, epoch=84, grad_norm=133.090, loss=290.2141, lr=9.19e-04]
Epoch 84 TRAIN:  25%|██▍       | 34/137 [00:10<00:22,  4.64it/s, epoch=84, grad_norm=110.627, loss=301.9743, lr=9.20e-04]
Epoch 84 TRAIN:  27%|██▋       | 37/137 [00:10<00:21,  4.69it/s, epoch=84, grad_norm=110.627, loss=301.9743, lr=9.20e-04]
Epoch 84 TRAIN:  27%|██▋       | 37/137 [00:10<00:21,  4.69it/s, epoch=84, grad_norm=107.313, loss=278.7277, lr=9.20e-04]
Epoch 84 TRAIN:  27%|██▋       | 37/137 [00:10<00:21,  4.69it/s, epoch=84, grad_norm=97.034, loss=263.0725, lr=9.20e-04] 
Epoch 84 TRAIN:  27%|██▋       | 37/137 [00:11<00:21,  4.69it/s, epoch=84, grad_norm=112.212, loss=273.4385, lr=9.20e-04]
Epoch 84 TRAIN:  29%|██▉       | 40/137 [00:11<00:21,  4.58it/s, epoch=84, grad_norm=112.212, loss=273.4385, lr=9.20e-04]
Epoch 84 TRAIN:  29%|██▉       | 40/137 [00:11<00:21,  4.58it/s, epoch=84, grad_norm=105.920, loss=290.7866, lr=9.20e-04]
Epoch 84 TRAIN:  29%|██▉       | 40/137 [00:11<00:21,  4.58it/s, epoch=84, grad_norm=84.484, loss=253.7053, lr=9.20e-04] 
Epoch 84 TRAIN:  29%|██▉       | 40/137 [00:11<00:21,  4.58it/s, epoch=84, grad_norm=103.490, loss=297.6935, lr=9.20e-04]
Epoch 84 TRAIN:  31%|███▏      | 43/137 [00:11<00:20,  4.61it/s, epoch=84, grad_norm=103.490, loss=297.6935, lr=9.20e-04]
Epoch 84 TRAIN:  31%|███▏      | 43/137 [00:11<00:20,  4.61it/s, epoch=84, grad_norm=81.580, loss=269.2198, lr=9.20e-04] 
Epoch 84 TRAIN:  31%|███▏      | 43/137 [00:12<00:20,  4.61it/s, epoch=84, grad_norm=151.150, loss=225.9121, lr=9.20e-04]
Epoch 84 TRAIN:  31%|███▏      | 43/137 [00:12<00:20,  4.61it/s, epoch=84, grad_norm=133.781, loss=282.0680, lr=9.20e-04]
Epoch 84 TRAIN:  34%|███▎      | 46/137 [00:12<00:19,  4.66it/s, epoch=84, grad_norm=133.781, loss=282.0680, lr=9.20e-04]
Epoch 84 TRAIN:  34%|███▎      | 46/137 [00:12<00:19,  4.66it/s, epoch=84, grad_norm=147.399, loss=276.3656, lr=9.20e-04]
Epoch 84 TRAIN:  34%|███▎      | 46/137 [00:12<00:19,  4.66it/s, epoch=84, grad_norm=144.733, loss=272.5871, lr=9.20e-04]
Epoch 84 TRAIN:  34%|███▎      | 46/137 [00:13<00:19,  4.66it/s, epoch=84, grad_norm=133.717, loss=223.8925, lr=9.20e-04]
Epoch 84 TRAIN:  36%|███▌      | 49/137 [00:13<00:19,  4.49it/s, epoch=84, grad_norm=133.717, loss=223.8925, lr=9.20e-04]
Epoch 84 TRAIN:  36%|███▌      | 49/137 [00:13<00:19,  4.49it/s, epoch=84, grad_norm=157.464, loss=252.9925, lr=9.21e-04]
Epoch 84 TRAIN:  36%|███▌      | 49/137 [00:13<00:19,  4.49it/s, epoch=84, grad_norm=104.137, loss=259.2074, lr=9.21e-04]
Epoch 84 TRAIN:  36%|███▌      | 49/137 [00:13<00:19,  4.49it/s, epoch=84, grad_norm=90.651, loss=210.5450, lr=9.21e-04] 
Epoch 84 TRAIN:  38%|███▊      | 52/137 [00:13<00:18,  4.57it/s, epoch=84, grad_norm=90.651, loss=210.5450, lr=9.21e-04]
Epoch 84 TRAIN:  38%|███▊      | 52/137 [00:13<00:18,  4.57it/s, epoch=84, grad_norm=95.590, loss=292.7898, lr=9.21e-04]
Epoch 84 TRAIN:  38%|███▊      | 52/137 [00:14<00:18,  4.57it/s, epoch=84, grad_norm=300.069, loss=271.8862, lr=9.21e-04]
Epoch 84 TRAIN:  38%|███▊      | 52/137 [00:14<00:18,  4.57it/s, epoch=84, grad_norm=167.565, loss=294.6984, lr=9.21e-04]
Epoch 84 TRAIN:  40%|████      | 55/137 [00:14<00:18,  4.39it/s, epoch=84, grad_norm=167.565, loss=294.6984, lr=9.21e-04]
Epoch 84 TRAIN:  40%|████      | 55/137 [00:14<00:18,  4.39it/s, epoch=84, grad_norm=74.981, loss=264.5194, lr=9.21e-04] 
Epoch 84 TRAIN:  40%|████      | 55/137 [00:14<00:18,  4.39it/s, epoch=84, grad_norm=166.396, loss=280.2554, lr=9.21e-04]
Epoch 84 TRAIN:  40%|████      | 55/137 [00:15<00:18,  4.39it/s, epoch=84, grad_norm=139.148, loss=279.1398, lr=9.21e-04]
Epoch 84 TRAIN:  42%|████▏     | 58/137 [00:15<00:17,  4.42it/s, epoch=84, grad_norm=139.148, loss=279.1398, lr=9.21e-04]
Epoch 84 TRAIN:  42%|████▏     | 58/137 [00:15<00:17,  4.42it/s, epoch=84, grad_norm=87.198, loss=250.7521, lr=9.21e-04] 
Epoch 84 TRAIN:  42%|████▏     | 58/137 [00:15<00:17,  4.42it/s, epoch=84, grad_norm=107.693, loss=292.0438, lr=9.21e-04]
Epoch 84 TRAIN:  42%|████▏     | 58/137 [00:15<00:17,  4.42it/s, epoch=84, grad_norm=171.740, loss=241.9564, lr=9.21e-04]
Epoch 84 TRAIN:  45%|████▍     | 61/137 [00:15<00:16,  4.54it/s, epoch=84, grad_norm=171.740, loss=241.9564, lr=9.21e-04]
Epoch 84 TRAIN:  45%|████▍     | 61/137 [00:15<00:16,  4.54it/s, epoch=84, grad_norm=130.285, loss=234.0529, lr=9.22e-04]
Epoch 84 TRAIN:  45%|████▍     | 61/137 [00:16<00:16,  4.54it/s, epoch=84, grad_norm=173.129, loss=222.8901, lr=9.22e-04]
Epoch 84 TRAIN:  45%|████▍     | 61/137 [00:16<00:16,  4.54it/s, epoch=84, grad_norm=158.719, loss=273.6086, lr=9.22e-04]
Epoch 84 TRAIN:  47%|████▋     | 64/137 [00:16<00:15,  4.67it/s, epoch=84, grad_norm=158.719, loss=273.6086, lr=9.22e-04]
Epoch 84 TRAIN:  47%|████▋     | 64/137 [00:16<00:15,  4.67it/s, epoch=84, grad_norm=176.640, loss=253.3972, lr=9.22e-04]
Epoch 84 TRAIN:  47%|████▋     | 64/137 [00:16<00:15,  4.67it/s, epoch=84, grad_norm=73.489, loss=245.8372, lr=9.22e-04] 
Epoch 84 TRAIN:  47%|████▋     | 64/137 [00:16<00:15,  4.67it/s, epoch=84, grad_norm=247.085, loss=298.4670, lr=9.22e-04]
Epoch 84 TRAIN:  49%|████▉     | 67/137 [00:16<00:15,  4.56it/s, epoch=84, grad_norm=247.085, loss=298.4670, lr=9.22e-04]
Epoch 84 TRAIN:  49%|████▉     | 67/137 [00:17<00:15,  4.56it/s, epoch=84, grad_norm=181.934, loss=268.9284, lr=9.22e-04]
Epoch 84 TRAIN:  49%|████▉     | 67/137 [00:17<00:15,  4.56it/s, epoch=84, grad_norm=153.215, loss=230.2374, lr=9.22e-04]
Epoch 84 TRAIN:  49%|████▉     | 67/137 [00:17<00:15,  4.56it/s, epoch=84, grad_norm=125.814, loss=210.3246, lr=9.22e-04]
Epoch 84 TRAIN:  51%|█████     | 70/137 [00:17<00:14,  4.69it/s, epoch=84, grad_norm=125.814, loss=210.3246, lr=9.22e-04]
Epoch 84 TRAIN:  51%|█████     | 70/137 [00:17<00:14,  4.69it/s, epoch=84, grad_norm=100.953, loss=279.8329, lr=9.22e-04]
Epoch 84 TRAIN:  51%|█████     | 70/137 [00:17<00:14,  4.69it/s, epoch=84, grad_norm=87.715, loss=242.2306, lr=9.22e-04] 
Epoch 84 TRAIN:  51%|█████     | 70/137 [00:18<00:14,  4.69it/s, epoch=84, grad_norm=119.042, loss=263.4465, lr=9.22e-04]
Epoch 84 TRAIN:  53%|█████▎    | 73/137 [00:18<00:14,  4.57it/s, epoch=84, grad_norm=119.042, loss=263.4465, lr=9.22e-04]
Epoch 84 TRAIN:  53%|█████▎    | 73/137 [00:18<00:14,  4.57it/s, epoch=84, grad_norm=113.388, loss=255.7764, lr=9.22e-04]
Epoch 84 TRAIN:  53%|█████▎    | 73/137 [00:18<00:14,  4.57it/s, epoch=84, grad_norm=206.889, loss=313.0786, lr=9.23e-04]
Epoch 84 TRAIN:  53%|█████▎    | 73/137 [00:18<00:14,  4.57it/s, epoch=84, grad_norm=174.756, loss=305.6312, lr=9.23e-04]
Epoch 84 TRAIN:  55%|█████▌    | 76/137 [00:18<00:13,  4.57it/s, epoch=84, grad_norm=174.756, loss=305.6312, lr=9.23e-04]
Epoch 84 TRAIN:  55%|█████▌    | 76/137 [00:19<00:13,  4.57it/s, epoch=84, grad_norm=161.844, loss=257.5241, lr=9.23e-04]
Epoch 84 TRAIN:  55%|█████▌    | 76/137 [00:19<00:13,  4.57it/s, epoch=84, grad_norm=137.581, loss=279.2115, lr=9.23e-04]
Epoch 84 TRAIN:  55%|█████▌    | 76/137 [00:19<00:13,  4.57it/s, epoch=84, grad_norm=142.629, loss=229.9945, lr=9.23e-04]
Epoch 84 TRAIN:  58%|█████▊    | 79/137 [00:19<00:13,  4.42it/s, epoch=84, grad_norm=142.629, loss=229.9945, lr=9.23e-04]
Epoch 84 TRAIN:  58%|█████▊    | 79/137 [00:19<00:13,  4.42it/s, epoch=84, grad_norm=181.308, loss=283.7780, lr=9.23e-04]
Epoch 84 TRAIN:  58%|█████▊    | 79/137 [00:20<00:13,  4.42it/s, epoch=84, grad_norm=138.132, loss=244.6969, lr=9.23e-04]
Epoch 84 TRAIN:  58%|█████▊    | 79/137 [00:20<00:13,  4.42it/s, epoch=84, grad_norm=218.148, loss=328.4515, lr=9.23e-04]
Epoch 84 TRAIN:  60%|█████▉    | 82/137 [00:20<00:11,  4.63it/s, epoch=84, grad_norm=218.148, loss=328.4515, lr=9.23e-04]
Epoch 84 TRAIN:  60%|█████▉    | 82/137 [00:20<00:11,  4.63it/s, epoch=84, grad_norm=120.427, loss=258.3015, lr=9.23e-04]
Epoch 84 TRAIN:  60%|█████▉    | 82/137 [00:20<00:11,  4.63it/s, epoch=84, grad_norm=177.226, loss=325.2300, lr=9.23e-04]
Epoch 84 TRAIN:  60%|█████▉    | 82/137 [00:20<00:11,  4.63it/s, epoch=84, grad_norm=218.585, loss=204.5680, lr=9.23e-04]
Epoch 84 TRAIN:  62%|██████▏   | 85/137 [00:20<00:11,  4.64it/s, epoch=84, grad_norm=218.585, loss=204.5680, lr=9.23e-04]
Epoch 84 TRAIN:  62%|██████▏   | 85/137 [00:21<00:11,  4.64it/s, epoch=84, grad_norm=102.057, loss=307.4384, lr=9.23e-04]
Epoch 84 TRAIN:  62%|██████▏   | 85/137 [00:21<00:11,  4.64it/s, epoch=84, grad_norm=165.471, loss=248.9935, lr=9.24e-04]
Epoch 84 TRAIN:  62%|██████▏   | 85/137 [00:21<00:11,  4.64it/s, epoch=84, grad_norm=115.406, loss=293.4051, lr=9.24e-04]
Epoch 84 TRAIN:  64%|██████▍   | 88/137 [00:21<00:11,  4.44it/s, epoch=84, grad_norm=115.406, loss=293.4051, lr=9.24e-04]
Epoch 84 TRAIN:  64%|██████▍   | 88/137 [00:21<00:11,  4.44it/s, epoch=84, grad_norm=231.025, loss=238.7807, lr=9.24e-04]
Epoch 84 TRAIN:  64%|██████▍   | 88/137 [00:22<00:11,  4.44it/s, epoch=84, grad_norm=116.710, loss=264.0198, lr=9.24e-04]
Epoch 84 TRAIN:  64%|██████▍   | 88/137 [00:22<00:11,  4.44it/s, epoch=84, grad_norm=135.293, loss=209.4505, lr=9.24e-04]
Epoch 84 TRAIN:  66%|██████▋   | 91/137 [00:22<00:10,  4.36it/s, epoch=84, grad_norm=135.293, loss=209.4505, lr=9.24e-04]
Epoch 84 TRAIN:  66%|██████▋   | 91/137 [00:22<00:10,  4.36it/s, epoch=84, grad_norm=110.061, loss=270.5157, lr=9.24e-04]
Epoch 84 TRAIN:  66%|██████▋   | 91/137 [00:22<00:10,  4.36it/s, epoch=84, grad_norm=219.656, loss=212.1406, lr=9.24e-04]
Epoch 84 TRAIN:  66%|██████▋   | 91/137 [00:22<00:10,  4.36it/s, epoch=84, grad_norm=168.186, loss=215.7060, lr=9.24e-04]
Epoch 84 TRAIN:  69%|██████▊   | 94/137 [00:22<00:09,  4.46it/s, epoch=84, grad_norm=168.186, loss=215.7060, lr=9.24e-04]
Epoch 84 TRAIN:  69%|██████▊   | 94/137 [00:23<00:09,  4.46it/s, epoch=84, grad_norm=159.552, loss=214.7770, lr=9.24e-04]
Epoch 84 TRAIN:  69%|██████▊   | 94/137 [00:23<00:09,  4.46it/s, epoch=84, grad_norm=225.019, loss=276.1472, lr=9.24e-04]
Epoch 84 TRAIN:  69%|██████▊   | 94/137 [00:23<00:09,  4.46it/s, epoch=84, grad_norm=111.737, loss=187.9292, lr=9.24e-04]
Epoch 84 TRAIN:  71%|███████   | 97/137 [00:23<00:08,  4.46it/s, epoch=84, grad_norm=111.737, loss=187.9292, lr=9.24e-04]
Epoch 84 TRAIN:  71%|███████   | 97/137 [00:23<00:08,  4.46it/s, epoch=84, grad_norm=281.569, loss=312.2412, lr=9.24e-04]
Epoch 84 TRAIN:  71%|███████   | 97/137 [00:24<00:08,  4.46it/s, epoch=84, grad_norm=183.895, loss=254.0829, lr=9.24e-04]
Epoch 84 TRAIN:  71%|███████   | 97/137 [00:24<00:08,  4.46it/s, epoch=84, grad_norm=230.936, loss=255.0522, lr=9.25e-04]2026-04-16 20:35:51,047 DEBUG TRAIN | steps/sec 4.151| Batch 84/100 loss 255.052155 loss_att 199.915375 loss_ctc 383.704590 th_accuracy 0.410609 lr 9.2456e-04 grad_norm 230.935928 rank 0

Epoch 84 TRAIN:  73%|███████▎  | 100/137 [00:24<00:08,  4.62it/s, epoch=84, grad_norm=230.936, loss=255.0522, lr=9.25e-04]
Epoch 84 TRAIN:  73%|███████▎  | 100/137 [00:24<00:08,  4.62it/s, epoch=84, grad_norm=157.322, loss=226.6444, lr=9.25e-04]
Epoch 84 TRAIN:  73%|███████▎  | 100/137 [00:24<00:08,  4.62it/s, epoch=84, grad_norm=103.465, loss=218.1150, lr=9.25e-04]
Epoch 84 TRAIN:  73%|███████▎  | 100/137 [00:24<00:08,  4.62it/s, epoch=84, grad_norm=152.567, loss=283.0487, lr=9.25e-04]
Epoch 84 TRAIN:  75%|███████▌  | 103/137 [00:24<00:07,  4.56it/s, epoch=84, grad_norm=152.567, loss=283.0487, lr=9.25e-04]
Epoch 84 TRAIN:  75%|███████▌  | 103/137 [00:25<00:07,  4.56it/s, epoch=84, grad_norm=269.287, loss=311.2468, lr=9.25e-04]
Epoch 84 TRAIN:  75%|███████▌  | 103/137 [00:25<00:07,  4.56it/s, epoch=84, grad_norm=212.738, loss=325.0948, lr=9.25e-04]
Epoch 84 TRAIN:  75%|███████▌  | 103/137 [00:25<00:07,  4.56it/s, epoch=84, grad_norm=100.569, loss=213.6098, lr=9.25e-04]
Epoch 84 TRAIN:  77%|███████▋  | 106/137 [00:25<00:07,  4.42it/s, epoch=84, grad_norm=100.569, loss=213.6098, lr=9.25e-04]
Epoch 84 TRAIN:  77%|███████▋  | 106/137 [00:25<00:07,  4.42it/s, epoch=84, grad_norm=142.960, loss=250.7493, lr=9.25e-04]
Epoch 84 TRAIN:  77%|███████▋  | 106/137 [00:26<00:07,  4.42it/s, epoch=84, grad_norm=192.679, loss=197.9603, lr=9.25e-04]
Epoch 84 TRAIN:  77%|███████▋  | 106/137 [00:26<00:07,  4.42it/s, epoch=84, grad_norm=123.434, loss=304.7460, lr=9.25e-04]
Epoch 84 TRAIN:  80%|███████▉  | 109/137 [00:26<00:06,  4.35it/s, epoch=84, grad_norm=123.434, loss=304.7460, lr=9.25e-04]
Epoch 84 TRAIN:  80%|███████▉  | 109/137 [00:26<00:06,  4.35it/s, epoch=84, grad_norm=125.472, loss=253.9200, lr=9.25e-04]
Epoch 84 TRAIN:  80%|███████▉  | 109/137 [00:26<00:06,  4.35it/s, epoch=84, grad_norm=229.050, loss=271.0557, lr=9.25e-04]
Epoch 84 TRAIN:  80%|███████▉  | 109/137 [00:27<00:06,  4.35it/s, epoch=84, grad_norm=248.899, loss=328.7062, lr=9.26e-04]
Epoch 84 TRAIN:  82%|████████▏ | 112/137 [00:27<00:05,  4.39it/s, epoch=84, grad_norm=248.899, loss=328.7062, lr=9.26e-04]
Epoch 84 TRAIN:  82%|████████▏ | 112/137 [00:27<00:05,  4.39it/s, epoch=84, grad_norm=154.907, loss=310.9645, lr=9.26e-04]
Epoch 84 TRAIN:  82%|████████▏ | 112/137 [00:27<00:05,  4.39it/s, epoch=84, grad_norm=111.037, loss=206.2801, lr=9.26e-04]
Epoch 84 TRAIN:  82%|████████▏ | 112/137 [00:27<00:05,  4.39it/s, epoch=84, grad_norm=113.556, loss=255.3842, lr=9.26e-04]
Epoch 84 TRAIN:  84%|████████▍ | 115/137 [00:27<00:05,  4.30it/s, epoch=84, grad_norm=113.556, loss=255.3842, lr=9.26e-04]
Epoch 84 TRAIN:  84%|████████▍ | 115/137 [00:27<00:05,  4.30it/s, epoch=84, grad_norm=131.524, loss=253.3643, lr=9.26e-04]
Epoch 84 TRAIN:  84%|████████▍ | 115/137 [00:28<00:05,  4.30it/s, epoch=84, grad_norm=149.637, loss=240.0417, lr=9.26e-04]
Epoch 84 TRAIN:  84%|████████▍ | 115/137 [00:28<00:05,  4.30it/s, epoch=84, grad_norm=120.583, loss=254.4931, lr=9.26e-04]
Epoch 84 TRAIN:  86%|████████▌ | 118/137 [00:28<00:04,  4.39it/s, epoch=84, grad_norm=120.583, loss=254.4931, lr=9.26e-04]
Epoch 84 TRAIN:  86%|████████▌ | 118/137 [00:28<00:04,  4.39it/s, epoch=84, grad_norm=178.234, loss=299.6696, lr=9.26e-04]
Epoch 84 TRAIN:  86%|████████▌ | 118/137 [00:28<00:04,  4.39it/s, epoch=84, grad_norm=116.062, loss=253.6039, lr=9.26e-04]
Epoch 84 TRAIN:  86%|████████▌ | 118/137 [00:28<00:04,  4.39it/s, epoch=84, grad_norm=150.892, loss=246.1485, lr=9.26e-04]
Epoch 84 TRAIN:  88%|████████▊ | 121/137 [00:28<00:03,  4.61it/s, epoch=84, grad_norm=150.892, loss=246.1485, lr=9.26e-04]
Epoch 84 TRAIN:  88%|████████▊ | 121/137 [00:29<00:03,  4.61it/s, epoch=84, grad_norm=102.885, loss=267.1534, lr=9.26e-04]
Epoch 84 TRAIN:  88%|████████▊ | 121/137 [00:29<00:03,  4.61it/s, epoch=84, grad_norm=154.655, loss=290.5025, lr=9.26e-04]
Epoch 84 TRAIN:  88%|████████▊ | 121/137 [00:29<00:03,  4.61it/s, epoch=84, grad_norm=91.134, loss=233.0653, lr=9.26e-04] 
Epoch 84 TRAIN:  91%|█████████ | 124/137 [00:29<00:02,  4.49it/s, epoch=84, grad_norm=91.134, loss=233.0653, lr=9.26e-04]
Epoch 84 TRAIN:  91%|█████████ | 124/137 [00:29<00:02,  4.49it/s, epoch=84, grad_norm=215.138, loss=340.6310, lr=9.27e-04]
Epoch 84 TRAIN:  91%|█████████ | 124/137 [00:30<00:02,  4.49it/s, epoch=84, grad_norm=219.638, loss=310.2996, lr=9.27e-04]
Epoch 84 TRAIN:  91%|█████████ | 124/137 [00:30<00:02,  4.49it/s, epoch=84, grad_norm=214.770, loss=316.4701, lr=9.27e-04]
Epoch 84 TRAIN:  93%|█████████▎| 127/137 [00:30<00:02,  4.55it/s, epoch=84, grad_norm=214.770, loss=316.4701, lr=9.27e-04]
Epoch 84 TRAIN:  93%|█████████▎| 127/137 [00:30<00:02,  4.55it/s, epoch=84, grad_norm=138.690, loss=305.5962, lr=9.27e-04]
Epoch 84 TRAIN:  93%|█████████▎| 127/137 [00:30<00:02,  4.55it/s, epoch=84, grad_norm=222.246, loss=262.2527, lr=9.27e-04]
Epoch 84 TRAIN:  93%|█████████▎| 127/137 [00:31<00:02,  4.55it/s, epoch=84, grad_norm=110.032, loss=265.2074, lr=9.27e-04]
Epoch 84 TRAIN:  95%|█████████▍| 130/137 [00:31<00:01,  4.31it/s, epoch=84, grad_norm=110.032, loss=265.2074, lr=9.27e-04]
Epoch 84 TRAIN:  95%|█████████▍| 130/137 [00:31<00:01,  4.31it/s, epoch=84, grad_norm=202.386, loss=268.7957, lr=9.27e-04]
Epoch 84 TRAIN:  95%|█████████▍| 130/137 [00:31<00:01,  4.31it/s, epoch=84, grad_norm=278.351, loss=317.7842, lr=9.27e-04]
Epoch 84 TRAIN:  95%|█████████▍| 130/137 [00:31<00:01,  4.31it/s, epoch=84, grad_norm=290.562, loss=312.2809, lr=9.27e-04]
Epoch 84 TRAIN:  95%|█████████▍| 130/137 [00:31<00:01,  4.31it/s, epoch=84, grad_norm=156.994, loss=298.8287, lr=9.27e-04]
Epoch 84 TRAIN:  98%|█████████▊| 134/137 [00:31<00:00,  4.65it/s, epoch=84, grad_norm=156.994, loss=298.8287, lr=9.27e-04]
Epoch 84 TRAIN:  98%|█████████▊| 134/137 [00:32<00:00,  4.65it/s, epoch=84, grad_norm=159.083, loss=244.1469, lr=9.27e-04]
Epoch 84 TRAIN:  99%|█████████▊| 135/137 [00:32<00:00,  4.21it/s, epoch=84, grad_norm=159.083, loss=244.1469, lr=9.27e-04]

CV:   0%|          | 0/6 [00:00<?, ?it/s]/home/rookie/Documents/batuka/projects/wenet/wenet/wenet/utils/train_utils.py:689: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  amp_autocast(enabled=dtype is not None,

CV:  17%|█▋        | 1/6 [00:00<00:03,  1.32it/s]
                                                 
2026-04-16 20:35:59,765 INFO Epoch 84 Step 11592 CV info lr 9.2736e-04 cv_loss 469.28500088778407 rank 0 acc 0.37786488234996796
2026-04-16 20:35:59,766 INFO [Rank 0] Checkpoint: save to checkpoint exp/conformer_mongolian/epoch_84.pt
2026-04-16 20:35:59,882 INFO Epoch 85 Step 11592 TRAIN info lr 9.2736e-04 rank 0
2026-04-16 20:35:59,897 INFO using accumulate grad, new batch size is 1 times larger than before

Epoch 85 TRAIN:   0%|          | 0/135 [00:00<?, ?it/s]
Epoch 85 TRAIN:   0%|          | 0/135 [00:04<?, ?it/s, epoch=85, grad_norm=298.138, loss=258.9455, lr=9.27e-04]
Epoch 85 TRAIN:   1%|          | 1/135 [00:04<09:30,  4.26s/it, epoch=85, grad_norm=298.138, loss=258.9455, lr=9.27e-04]
Epoch 85 TRAIN:   1%|          | 1/135 [00:04<09:30,  4.26s/it, epoch=85, grad_norm=364.777, loss=263.5161, lr=9.28e-04]
Epoch 85 TRAIN:   1%|          | 1/135 [00:04<09:30,  4.26s/it, epoch=85, grad_norm=461.558, loss=317.3268, lr=9.28e-04]
Epoch 85 TRAIN:   1%|          | 1/135 [00:04<09:30,  4.26s/it, epoch=85, grad_norm=421.184, loss=335.1459, lr=9.28e-04]
Epoch 85 TRAIN:   1%|          | 1/135 [00:04<09:30,  4.26s/it, epoch=85, grad_norm=342.580, loss=290.8560, lr=9.28e-04]
Epoch 85 TRAIN:   1%|          | 1/135 [00:04<09:30,  4.26s/it, epoch=85, grad_norm=291.404, loss=252.8963, lr=9.28e-04]
Epoch 85 TRAIN:   1%|          | 1/135 [00:04<09:30,  4.26s/it, epoch=85, grad_norm=308.976, loss=294.6716, lr=9.28e-04]
Epoch 85 TRAIN:   1%|          | 1/135 [00:04<09:30,  4.26s/it, epoch=85, grad_norm=253.209, loss=270.5124, lr=9.28e-04]
Epoch 85 TRAIN:   1%|          | 1/135 [00:04<09:30,  4.26s/it, epoch=85, grad_norm=223.941, loss=296.2590, lr=9.28e-04]
Epoch 85 TRAIN:   7%|▋         | 9/135 [00:04<00:52,  2.41it/s, epoch=85, grad_norm=223.941, loss=296.2590, lr=9.28e-04]
Epoch 85 TRAIN:   7%|▋         | 9/135 [00:05<00:52,  2.41it/s, epoch=85, grad_norm=199.221, loss=315.9404, lr=9.28e-04]
Epoch 85 TRAIN:   7%|▋         | 9/135 [00:05<00:52,  2.41it/s, epoch=85, grad_norm=168.924, loss=297.4308, lr=9.28e-04]
Epoch 85 TRAIN:   7%|▋         | 9/135 [00:05<00:52,  2.41it/s, epoch=85, grad_norm=84.645, loss=260.2611, lr=9.28e-04] 
Epoch 85 TRAIN:   7%|▋         | 9/135 [00:05<00:52,  2.41it/s, epoch=85, grad_norm=109.183, loss=281.5939, lr=9.28e-04]
Epoch 85 TRAIN:  10%|▉         | 13/135 [00:05<00:38,  3.16it/s, epoch=85, grad_norm=109.183, loss=281.5939, lr=9.28e-04]
Epoch 85 TRAIN:  10%|▉         | 13/135 [00:05<00:38,  3.16it/s, epoch=85, grad_norm=137.628, loss=256.2919, lr=9.28e-04]
Epoch 85 TRAIN:  10%|▉         | 13/135 [00:05<00:38,  3.16it/s, epoch=85, grad_norm=145.606, loss=265.1380, lr=9.29e-04]
Epoch 85 TRAIN:  10%|▉         | 13/135 [00:06<00:38,  3.16it/s, epoch=85, grad_norm=170.828, loss=249.3216, lr=9.29e-04]
Epoch 85 TRAIN:  10%|▉         | 13/135 [00:06<00:38,  3.16it/s, epoch=85, grad_norm=223.666, loss=242.4783, lr=9.29e-04]
Epoch 85 TRAIN:  13%|█▎        | 17/135 [00:06<00:31,  3.75it/s, epoch=85, grad_norm=223.666, loss=242.4783, lr=9.29e-04]
Epoch 85 TRAIN:  13%|█▎        | 17/135 [00:06<00:31,  3.75it/s, epoch=85, grad_norm=111.353, loss=268.9246, lr=9.29e-04]
Epoch 85 TRAIN:  13%|█▎        | 17/135 [00:06<00:31,  3.75it/s, epoch=85, grad_norm=105.969, loss=285.2859, lr=9.29e-04]
Epoch 85 TRAIN:  13%|█▎        | 17/135 [00:06<00:31,  3.75it/s, epoch=85, grad_norm=121.952, loss=305.5532, lr=9.29e-04]
Epoch 85 TRAIN:  13%|█▎        | 17/135 [00:07<00:31,  3.75it/s, epoch=85, grad_norm=170.163, loss=271.4329, lr=9.29e-04]
Epoch 85 TRAIN:  16%|█▌        | 21/135 [00:07<00:28,  4.04it/s, epoch=85, grad_norm=170.163, loss=271.4329, lr=9.29e-04]
Epoch 85 TRAIN:  16%|█▌        | 21/135 [00:07<00:28,  4.04it/s, epoch=85, grad_norm=83.293, loss=256.9288, lr=9.29e-04] 
Epoch 85 TRAIN:  16%|█▌        | 21/135 [00:07<00:28,  4.04it/s, epoch=85, grad_norm=86.139, loss=273.7848, lr=9.29e-04]
Epoch 85 TRAIN:  16%|█▌        | 21/135 [00:07<00:28,  4.04it/s, epoch=85, grad_norm=158.675, loss=298.5600, lr=9.29e-04]
Epoch 85 TRAIN:  18%|█▊        | 24/135 [00:07<00:26,  4.24it/s, epoch=85, grad_norm=158.675, loss=298.5600, lr=9.29e-04]
Epoch 85 TRAIN:  18%|█▊        | 24/135 [00:07<00:26,  4.24it/s, epoch=85, grad_norm=76.195, loss=242.6316, lr=9.29e-04] 
Epoch 85 TRAIN:  18%|█▊        | 24/135 [00:08<00:26,  4.24it/s, epoch=85, grad_norm=66.867, loss=285.7458, lr=9.29e-04]
Epoch 85 TRAIN:  18%|█▊        | 24/135 [00:08<00:26,  4.24it/s, epoch=85, grad_norm=88.831, loss=277.8831, lr=9.30e-04]
Epoch 85 TRAIN:  20%|██        | 27/135 [00:08<00:24,  4.43it/s, epoch=85, grad_norm=88.831, loss=277.8831, lr=9.30e-04]
Epoch 85 TRAIN:  20%|██        | 27/135 [00:08<00:24,  4.43it/s, epoch=85, grad_norm=98.224, loss=232.6295, lr=9.30e-04]
Epoch 85 TRAIN:  20%|██        | 27/135 [00:08<00:24,  4.43it/s, epoch=85, grad_norm=102.151, loss=277.2298, lr=9.30e-04]
Epoch 85 TRAIN:  20%|██        | 27/135 [00:08<00:24,  4.43it/s, epoch=85, grad_norm=84.553, loss=280.0346, lr=9.30e-04] 
Epoch 85 TRAIN:  22%|██▏       | 30/135 [00:08<00:23,  4.55it/s, epoch=85, grad_norm=84.553, loss=280.0346, lr=9.30e-04]
Epoch 85 TRAIN:  22%|██▏       | 30/135 [00:09<00:23,  4.55it/s, epoch=85, grad_norm=98.422, loss=269.9960, lr=9.30e-04]
Epoch 85 TRAIN:  22%|██▏       | 30/135 [00:09<00:23,  4.55it/s, epoch=85, grad_norm=136.198, loss=274.3380, lr=9.30e-04]
Epoch 85 TRAIN:  22%|██▏       | 30/135 [00:09<00:23,  4.55it/s, epoch=85, grad_norm=107.100, loss=264.4937, lr=9.30e-04]
Epoch 85 TRAIN:  24%|██▍       | 33/135 [00:09<00:22,  4.54it/s, epoch=85, grad_norm=107.100, loss=264.4937, lr=9.30e-04]
Epoch 85 TRAIN:  24%|██▍       | 33/135 [00:09<00:22,  4.54it/s, epoch=85, grad_norm=108.372, loss=243.9458, lr=9.30e-04]
Epoch 85 TRAIN:  24%|██▍       | 33/135 [00:10<00:22,  4.54it/s, epoch=85, grad_norm=137.904, loss=286.3120, lr=9.30e-04]
Epoch 85 TRAIN:  24%|██▍       | 33/135 [00:10<00:22,  4.54it/s, epoch=85, grad_norm=99.699, loss=265.2227, lr=9.30e-04] 
Epoch 85 TRAIN:  27%|██▋       | 36/135 [00:10<00:21,  4.60it/s, epoch=85, grad_norm=99.699, loss=265.2227, lr=9.30e-04]
Epoch 85 TRAIN:  27%|██▋       | 36/135 [00:10<00:21,  4.60it/s, epoch=85, grad_norm=130.020, loss=276.8094, lr=9.30e-04]
Epoch 85 TRAIN:  27%|██▋       | 36/135 [00:10<00:21,  4.60it/s, epoch=85, grad_norm=83.065, loss=252.2157, lr=9.30e-04] 
Epoch 85 TRAIN:  27%|██▋       | 36/135 [00:10<00:21,  4.60it/s, epoch=85, grad_norm=149.977, loss=281.8182, lr=9.30e-04]
Epoch 85 TRAIN:  29%|██▉       | 39/135 [00:10<00:20,  4.58it/s, epoch=85, grad_norm=149.977, loss=281.8182, lr=9.30e-04]
Epoch 85 TRAIN:  29%|██▉       | 39/135 [00:11<00:20,  4.58it/s, epoch=85, grad_norm=143.033, loss=302.4773, lr=9.31e-04]
Epoch 85 TRAIN:  29%|██▉       | 39/135 [00:11<00:20,  4.58it/s, epoch=85, grad_norm=109.628, loss=275.9681, lr=9.31e-04]
Epoch 85 TRAIN:  29%|██▉       | 39/135 [00:11<00:20,  4.58it/s, epoch=85, grad_norm=144.795, loss=329.4301, lr=9.31e-04]
Epoch 85 TRAIN:  31%|███       | 42/135 [00:11<00:20,  4.57it/s, epoch=85, grad_norm=144.795, loss=329.4301, lr=9.31e-04]
Epoch 85 TRAIN:  31%|███       | 42/135 [00:11<00:20,  4.57it/s, epoch=85, grad_norm=125.076, loss=289.8716, lr=9.31e-04]
Epoch 85 TRAIN:  31%|███       | 42/135 [00:12<00:20,  4.57it/s, epoch=85, grad_norm=83.785, loss=279.8595, lr=9.31e-04] 
Epoch 85 TRAIN:  31%|███       | 42/135 [00:12<00:20,  4.57it/s, epoch=85, grad_norm=136.764, loss=250.9531, lr=9.31e-04]
Epoch 85 TRAIN:  33%|███▎      | 45/135 [00:12<00:20,  4.30it/s, epoch=85, grad_norm=136.764, loss=250.9531, lr=9.31e-04]
Epoch 85 TRAIN:  33%|███▎      | 45/135 [00:12<00:20,  4.30it/s, epoch=85, grad_norm=161.693, loss=275.9265, lr=9.31e-04]
Epoch 85 TRAIN:  33%|███▎      | 45/135 [00:12<00:20,  4.30it/s, epoch=85, grad_norm=120.291, loss=235.5988, lr=9.31e-04]
Epoch 85 TRAIN:  33%|███▎      | 45/135 [00:12<00:20,  4.30it/s, epoch=85, grad_norm=110.682, loss=276.2191, lr=9.31e-04]
Epoch 85 TRAIN:  36%|███▌      | 48/135 [00:12<00:19,  4.46it/s, epoch=85, grad_norm=110.682, loss=276.2191, lr=9.31e-04]
Epoch 85 TRAIN:  36%|███▌      | 48/135 [00:13<00:19,  4.46it/s, epoch=85, grad_norm=112.348, loss=299.0058, lr=9.31e-04]
Epoch 85 TRAIN:  36%|███▌      | 48/135 [00:13<00:19,  4.46it/s, epoch=85, grad_norm=76.658, loss=273.7828, lr=9.31e-04] 
Epoch 85 TRAIN:  36%|███▌      | 48/135 [00:13<00:19,  4.46it/s, epoch=85, grad_norm=132.396, loss=304.3569, lr=9.31e-04]
Epoch 85 TRAIN:  38%|███▊      | 51/135 [00:13<00:17,  4.73it/s, epoch=85, grad_norm=132.396, loss=304.3569, lr=9.31e-04]
Epoch 85 TRAIN:  38%|███▊      | 51/135 [00:13<00:17,  4.73it/s, epoch=85, grad_norm=149.170, loss=258.4352, lr=9.32e-04]
Epoch 85 TRAIN:  38%|███▊      | 51/135 [00:14<00:17,  4.73it/s, epoch=85, grad_norm=146.708, loss=240.9692, lr=9.32e-04]
Epoch 85 TRAIN:  38%|███▊      | 51/135 [00:14<00:17,  4.73it/s, epoch=85, grad_norm=107.003, loss=260.1651, lr=9.32e-04]
Epoch 85 TRAIN:  40%|████      | 54/135 [00:14<00:18,  4.46it/s, epoch=85, grad_norm=107.003, loss=260.1651, lr=9.32e-04]
Epoch 85 TRAIN:  40%|████      | 54/135 [00:14<00:18,  4.46it/s, epoch=85, grad_norm=162.917, loss=315.9749, lr=9.32e-04]
Epoch 85 TRAIN:  40%|████      | 54/135 [00:14<00:18,  4.46it/s, epoch=85, grad_norm=113.939, loss=268.6815, lr=9.32e-04]
Epoch 85 TRAIN:  40%|████      | 54/135 [00:14<00:18,  4.46it/s, epoch=85, grad_norm=95.680, loss=284.0097, lr=9.32e-04] 
Epoch 85 TRAIN:  42%|████▏     | 57/135 [00:14<00:17,  4.47it/s, epoch=85, grad_norm=95.680, loss=284.0097, lr=9.32e-04]
Epoch 85 TRAIN:  42%|████▏     | 57/135 [00:15<00:17,  4.47it/s, epoch=85, grad_norm=167.663, loss=294.5016, lr=9.32e-04]
Epoch 85 TRAIN:  42%|████▏     | 57/135 [00:15<00:17,  4.47it/s, epoch=85, grad_norm=79.841, loss=247.6008, lr=9.32e-04] 
Epoch 85 TRAIN:  42%|████▏     | 57/135 [00:15<00:17,  4.47it/s, epoch=85, grad_norm=139.571, loss=246.5829, lr=9.32e-04]
Epoch 85 TRAIN:  44%|████▍     | 60/135 [00:15<00:16,  4.51it/s, epoch=85, grad_norm=139.571, loss=246.5829, lr=9.32e-04]
Epoch 85 TRAIN:  44%|████▍     | 60/135 [00:15<00:16,  4.51it/s, epoch=85, grad_norm=108.514, loss=282.1303, lr=9.32e-04]
Epoch 85 TRAIN:  44%|████▍     | 60/135 [00:15<00:16,  4.51it/s, epoch=85, grad_norm=137.827, loss=258.6806, lr=9.32e-04]
Epoch 85 TRAIN:  44%|████▍     | 60/135 [00:16<00:16,  4.51it/s, epoch=85, grad_norm=102.251, loss=311.1165, lr=9.32e-04]
Epoch 85 TRAIN:  47%|████▋     | 63/135 [00:16<00:16,  4.44it/s, epoch=85, grad_norm=102.251, loss=311.1165, lr=9.32e-04]
Epoch 85 TRAIN:  47%|████▋     | 63/135 [00:16<00:16,  4.44it/s, epoch=85, grad_norm=150.281, loss=223.2186, lr=9.32e-04]
Epoch 85 TRAIN:  47%|████▋     | 63/135 [00:16<00:16,  4.44it/s, epoch=85, grad_norm=101.565, loss=292.1009, lr=9.33e-04]
Epoch 85 TRAIN:  47%|████▋     | 63/135 [00:16<00:16,  4.44it/s, epoch=85, grad_norm=140.146, loss=269.9355, lr=9.33e-04]
Epoch 85 TRAIN:  49%|████▉     | 66/135 [00:16<00:15,  4.55it/s, epoch=85, grad_norm=140.146, loss=269.9355, lr=9.33e-04]
Epoch 85 TRAIN:  49%|████▉     | 66/135 [00:17<00:15,  4.55it/s, epoch=85, grad_norm=108.145, loss=293.2477, lr=9.33e-04]
Epoch 85 TRAIN:  49%|████▉     | 66/135 [00:17<00:15,  4.55it/s, epoch=85, grad_norm=127.810, loss=215.2810, lr=9.33e-04]
Epoch 85 TRAIN:  49%|████▉     | 66/135 [00:17<00:15,  4.55it/s, epoch=85, grad_norm=95.707, loss=278.7913, lr=9.33e-04] 
Epoch 85 TRAIN:  51%|█████     | 69/135 [00:17<00:15,  4.27it/s, epoch=85, grad_norm=95.707, loss=278.7913, lr=9.33e-04]
Epoch 85 TRAIN:  51%|█████     | 69/135 [00:17<00:15,  4.27it/s, epoch=85, grad_norm=72.743, loss=260.2823, lr=9.33e-04]
Epoch 85 TRAIN:  51%|█████     | 69/135 [00:18<00:15,  4.27it/s, epoch=85, grad_norm=107.295, loss=282.4162, lr=9.33e-04]
Epoch 85 TRAIN:  51%|█████     | 69/135 [00:18<00:15,  4.27it/s, epoch=85, grad_norm=93.992, loss=267.2419, lr=9.33e-04] 
Epoch 85 TRAIN:  53%|█████▎    | 72/135 [00:18<00:14,  4.36it/s, epoch=85, grad_norm=93.992, loss=267.2419, lr=9.33e-04]
Epoch 85 TRAIN:  53%|█████▎    | 72/135 [00:18<00:14,  4.36it/s, epoch=85, grad_norm=103.274, loss=293.8362, lr=9.33e-04]
Epoch 85 TRAIN:  53%|█████▎    | 72/135 [00:18<00:14,  4.36it/s, epoch=85, grad_norm=93.166, loss=328.2458, lr=9.33e-04] 
Epoch 85 TRAIN:  53%|█████▎    | 72/135 [00:19<00:14,  4.36it/s, epoch=85, grad_norm=112.946, loss=246.7130, lr=9.33e-04]
Epoch 85 TRAIN:  56%|█████▌    | 75/135 [00:19<00:13,  4.45it/s, epoch=85, grad_norm=112.946, loss=246.7130, lr=9.33e-04]
Epoch 85 TRAIN:  56%|█████▌    | 75/135 [00:19<00:13,  4.45it/s, epoch=85, grad_norm=220.854, loss=303.0213, lr=9.33e-04]
Epoch 85 TRAIN:  56%|█████▌    | 75/135 [00:19<00:13,  4.45it/s, epoch=85, grad_norm=142.135, loss=240.6389, lr=9.34e-04]
Epoch 85 TRAIN:  56%|█████▌    | 75/135 [00:19<00:13,  4.45it/s, epoch=85, grad_norm=121.836, loss=256.7584, lr=9.34e-04]
Epoch 85 TRAIN:  58%|█████▊    | 78/135 [00:19<00:12,  4.53it/s, epoch=85, grad_norm=121.836, loss=256.7584, lr=9.34e-04]
Epoch 85 TRAIN:  58%|█████▊    | 78/135 [00:19<00:12,  4.53it/s, epoch=85, grad_norm=87.611, loss=295.4682, lr=9.34e-04] 
Epoch 85 TRAIN:  58%|█████▊    | 78/135 [00:20<00:12,  4.53it/s, epoch=85, grad_norm=100.209, loss=235.5246, lr=9.34e-04]
Epoch 85 TRAIN:  58%|█████▊    | 78/135 [00:20<00:12,  4.53it/s, epoch=85, grad_norm=164.471, loss=289.9537, lr=9.34e-04]
Epoch 85 TRAIN:  60%|██████    | 81/135 [00:20<00:12,  4.29it/s, epoch=85, grad_norm=164.471, loss=289.9537, lr=9.34e-04]
Epoch 85 TRAIN:  60%|██████    | 81/135 [00:20<00:12,  4.29it/s, epoch=85, grad_norm=230.185, loss=209.0988, lr=9.34e-04]
Epoch 85 TRAIN:  60%|██████    | 81/135 [00:20<00:12,  4.29it/s, epoch=85, grad_norm=159.303, loss=272.5621, lr=9.34e-04]
Epoch 85 TRAIN:  60%|██████    | 81/135 [00:21<00:12,  4.29it/s, epoch=85, grad_norm=122.931, loss=262.6898, lr=9.34e-04]
Epoch 85 TRAIN:  62%|██████▏   | 84/135 [00:21<00:11,  4.44it/s, epoch=85, grad_norm=122.931, loss=262.6898, lr=9.34e-04]
Epoch 85 TRAIN:  62%|██████▏   | 84/135 [00:21<00:11,  4.44it/s, epoch=85, grad_norm=131.412, loss=290.2626, lr=9.34e-04]
Epoch 85 TRAIN:  62%|██████▏   | 84/135 [00:21<00:11,  4.44it/s, epoch=85, grad_norm=183.395, loss=199.5229, lr=9.34e-04]
Epoch 85 TRAIN:  62%|██████▏   | 84/135 [00:21<00:11,  4.44it/s, epoch=85, grad_norm=120.621, loss=256.5954, lr=9.34e-04]
Epoch 85 TRAIN:  64%|██████▍   | 87/135 [00:21<00:10,  4.58it/s, epoch=85, grad_norm=120.621, loss=256.5954, lr=9.34e-04]
Epoch 85 TRAIN:  64%|██████▍   | 87/135 [00:21<00:10,  4.58it/s, epoch=85, grad_norm=143.698, loss=270.1105, lr=9.34e-04]
Epoch 85 TRAIN:  64%|██████▍   | 87/135 [00:22<00:10,  4.58it/s, epoch=85, grad_norm=141.157, loss=252.0589, lr=9.34e-04]
Epoch 85 TRAIN:  64%|██████▍   | 87/135 [00:22<00:10,  4.58it/s, epoch=85, grad_norm=130.183, loss=284.2138, lr=9.35e-04]
Epoch 85 TRAIN:  67%|██████▋   | 90/135 [00:22<00:09,  4.52it/s, epoch=85, grad_norm=130.183, loss=284.2138, lr=9.35e-04]
Epoch 85 TRAIN:  67%|██████▋   | 90/135 [00:22<00:09,  4.52it/s, epoch=85, grad_norm=190.009, loss=297.8003, lr=9.35e-04]
Epoch 85 TRAIN:  67%|██████▋   | 90/135 [00:22<00:09,  4.52it/s, epoch=85, grad_norm=132.846, loss=223.8222, lr=9.35e-04]
Epoch 85 TRAIN:  67%|██████▋   | 90/135 [00:23<00:09,  4.52it/s, epoch=85, grad_norm=102.633, loss=300.4692, lr=9.35e-04]
Epoch 85 TRAIN:  69%|██████▉   | 93/135 [00:23<00:09,  4.43it/s, epoch=85, grad_norm=102.633, loss=300.4692, lr=9.35e-04]
Epoch 85 TRAIN:  69%|██████▉   | 93/135 [00:23<00:09,  4.43it/s, epoch=85, grad_norm=191.945, loss=275.0155, lr=9.35e-04]
Epoch 85 TRAIN:  69%|██████▉   | 93/135 [00:23<00:09,  4.43it/s, epoch=85, grad_norm=102.488, loss=269.8395, lr=9.35e-04]
Epoch 85 TRAIN:  69%|██████▉   | 93/135 [00:23<00:09,  4.43it/s, epoch=85, grad_norm=105.242, loss=291.6730, lr=9.35e-04]
Epoch 85 TRAIN:  71%|███████   | 96/135 [00:23<00:08,  4.38it/s, epoch=85, grad_norm=105.242, loss=291.6730, lr=9.35e-04]
Epoch 85 TRAIN:  71%|███████   | 96/135 [00:23<00:08,  4.38it/s, epoch=85, grad_norm=188.042, loss=283.5138, lr=9.35e-04]
Epoch 85 TRAIN:  71%|███████   | 96/135 [00:24<00:08,  4.38it/s, epoch=85, grad_norm=130.443, loss=291.8721, lr=9.35e-04]
Epoch 85 TRAIN:  71%|███████   | 96/135 [00:24<00:08,  4.38it/s, epoch=85, grad_norm=146.593, loss=208.9734, lr=9.35e-04]
Epoch 85 TRAIN:  73%|███████▎  | 99/135 [00:24<00:07,  4.51it/s, epoch=85, grad_norm=146.593, loss=208.9734, lr=9.35e-04]
Epoch 85 TRAIN:  73%|███████▎  | 99/135 [00:24<00:07,  4.51it/s, epoch=85, grad_norm=188.689, loss=271.8407, lr=9.35e-04]2026-04-16 20:36:24,580 DEBUG TRAIN | steps/sec 4.026| Batch 85/100 loss 271.840698 loss_att 214.052582 loss_ctc 406.679657 th_accuracy 0.430657 lr 9.3536e-04 grad_norm 188.689316 rank 0

Epoch 85 TRAIN:  73%|███████▎  | 99/135 [00:24<00:07,  4.51it/s, epoch=85, grad_norm=105.332, loss=263.4479, lr=9.35e-04]
Epoch 85 TRAIN:  73%|███████▎  | 99/135 [00:25<00:07,  4.51it/s, epoch=85, grad_norm=186.816, loss=205.9498, lr=9.36e-04]
Epoch 85 TRAIN:  76%|███████▌  | 102/135 [00:25<00:07,  4.49it/s, epoch=85, grad_norm=186.816, loss=205.9498, lr=9.36e-04]
Epoch 85 TRAIN:  76%|███████▌  | 102/135 [00:25<00:07,  4.49it/s, epoch=85, grad_norm=93.580, loss=188.2087, lr=9.36e-04] 
Epoch 85 TRAIN:  76%|███████▌  | 102/135 [00:25<00:07,  4.49it/s, epoch=85, grad_norm=152.076, loss=267.6998, lr=9.36e-04]
Epoch 85 TRAIN:  76%|███████▌  | 102/135 [00:25<00:07,  4.49it/s, epoch=85, grad_norm=185.900, loss=298.8352, lr=9.36e-04]
Epoch 85 TRAIN:  78%|███████▊  | 105/135 [00:25<00:06,  4.30it/s, epoch=85, grad_norm=185.900, loss=298.8352, lr=9.36e-04]
Epoch 85 TRAIN:  78%|███████▊  | 105/135 [00:25<00:06,  4.30it/s, epoch=85, grad_norm=172.034, loss=272.0963, lr=9.36e-04]
Epoch 85 TRAIN:  78%|███████▊  | 105/135 [00:26<00:06,  4.30it/s, epoch=85, grad_norm=151.396, loss=230.7631, lr=9.36e-04]
Epoch 85 TRAIN:  78%|███████▊  | 105/135 [00:26<00:06,  4.30it/s, epoch=85, grad_norm=126.779, loss=238.0352, lr=9.36e-04]
Epoch 85 TRAIN:  80%|████████  | 108/135 [00:26<00:05,  4.59it/s, epoch=85, grad_norm=126.779, loss=238.0352, lr=9.36e-04]
Epoch 85 TRAIN:  80%|████████  | 108/135 [00:26<00:05,  4.59it/s, epoch=85, grad_norm=89.483, loss=227.1901, lr=9.36e-04] 
Epoch 85 TRAIN:  80%|████████  | 108/135 [00:26<00:05,  4.59it/s, epoch=85, grad_norm=138.993, loss=232.3326, lr=9.36e-04]
Epoch 85 TRAIN:  80%|████████  | 108/135 [00:27<00:05,  4.59it/s, epoch=85, grad_norm=107.547, loss=222.2371, lr=9.36e-04]
Epoch 85 TRAIN:  82%|████████▏ | 111/135 [00:27<00:05,  4.47it/s, epoch=85, grad_norm=107.547, loss=222.2371, lr=9.36e-04]
Epoch 85 TRAIN:  82%|████████▏ | 111/135 [00:27<00:05,  4.47it/s, epoch=85, grad_norm=125.687, loss=219.5853, lr=9.36e-04]
Epoch 85 TRAIN:  82%|████████▏ | 111/135 [00:27<00:05,  4.47it/s, epoch=85, grad_norm=157.105, loss=275.0543, lr=9.36e-04]
Epoch 85 TRAIN:  82%|████████▏ | 111/135 [00:27<00:05,  4.47it/s, epoch=85, grad_norm=122.412, loss=269.7833, lr=9.36e-04]
Epoch 85 TRAIN:  84%|████████▍ | 114/135 [00:27<00:04,  4.45it/s, epoch=85, grad_norm=122.412, loss=269.7833, lr=9.36e-04]
Epoch 85 TRAIN:  84%|████████▍ | 114/135 [00:28<00:04,  4.45it/s, epoch=85, grad_norm=101.478, loss=181.0645, lr=9.37e-04]
Epoch 85 TRAIN:  84%|████████▍ | 114/135 [00:28<00:04,  4.45it/s, epoch=85, grad_norm=269.754, loss=321.6798, lr=9.37e-04]
Epoch 85 TRAIN:  84%|████████▍ | 114/135 [00:28<00:04,  4.45it/s, epoch=85, grad_norm=180.445, loss=255.6684, lr=9.37e-04]
Epoch 85 TRAIN:  87%|████████▋ | 117/135 [00:28<00:04,  4.28it/s, epoch=85, grad_norm=180.445, loss=255.6684, lr=9.37e-04]
Epoch 85 TRAIN:  87%|████████▋ | 117/135 [00:28<00:04,  4.28it/s, epoch=85, grad_norm=169.151, loss=305.3151, lr=9.37e-04]
Epoch 85 TRAIN:  87%|████████▋ | 117/135 [00:28<00:04,  4.28it/s, epoch=85, grad_norm=157.722, loss=264.9315, lr=9.37e-04]
Epoch 85 TRAIN:  87%|████████▋ | 117/135 [00:28<00:04,  4.28it/s, epoch=85, grad_norm=232.843, loss=333.0443, lr=9.37e-04]
Epoch 85 TRAIN:  87%|████████▋ | 117/135 [00:29<00:04,  4.28it/s, epoch=85, grad_norm=174.471, loss=248.3634, lr=9.37e-04]
Epoch 85 TRAIN:  90%|████████▉ | 121/135 [00:29<00:02,  4.76it/s, epoch=85, grad_norm=174.471, loss=248.3634, lr=9.37e-04]
Epoch 85 TRAIN:  90%|████████▉ | 121/135 [00:29<00:02,  4.76it/s, epoch=85, grad_norm=132.247, loss=256.5643, lr=9.37e-04]
Epoch 85 TRAIN:  90%|████████▉ | 121/135 [00:29<00:02,  4.76it/s, epoch=85, grad_norm=211.738, loss=208.2653, lr=9.37e-04]
Epoch 85 TRAIN:  90%|████████▉ | 121/135 [00:29<00:02,  4.76it/s, epoch=85, grad_norm=127.242, loss=213.5320, lr=9.37e-04]
Epoch 85 TRAIN:  92%|█████████▏| 124/135 [00:29<00:02,  4.74it/s, epoch=85, grad_norm=127.242, loss=213.5320, lr=9.37e-04]
Epoch 85 TRAIN:  92%|█████████▏| 124/135 [00:30<00:02,  4.74it/s, epoch=85, grad_norm=160.671, loss=245.1897, lr=9.37e-04]
Epoch 85 TRAIN:  92%|█████████▏| 124/135 [00:30<00:02,  4.74it/s, epoch=85, grad_norm=214.260, loss=287.6190, lr=9.37e-04]
Epoch 85 TRAIN:  92%|█████████▏| 124/135 [00:30<00:02,  4.74it/s, epoch=85, grad_norm=408.456, loss=368.6966, lr=9.38e-04]
Epoch 85 TRAIN:  94%|█████████▍| 127/135 [00:30<00:01,  4.78it/s, epoch=85, grad_norm=408.456, loss=368.6966, lr=9.38e-04]
Epoch 85 TRAIN:  94%|█████████▍| 127/135 [00:30<00:01,  4.78it/s, epoch=85, grad_norm=198.054, loss=244.5167, lr=9.38e-04]
Epoch 85 TRAIN:  94%|█████████▍| 127/135 [00:30<00:01,  4.78it/s, epoch=85, grad_norm=155.586, loss=246.3215, lr=9.38e-04]
Epoch 85 TRAIN:  94%|█████████▍| 127/135 [00:30<00:01,  4.78it/s, epoch=85, grad_norm=328.065, loss=307.2160, lr=9.38e-04]
Epoch 85 TRAIN:  96%|█████████▋| 130/135 [00:30<00:01,  4.99it/s, epoch=85, grad_norm=328.065, loss=307.2160, lr=9.38e-04]
Epoch 85 TRAIN:  96%|█████████▋| 130/135 [00:31<00:01,  4.99it/s, epoch=85, grad_norm=183.957, loss=272.9381, lr=9.38e-04]
Epoch 85 TRAIN:  96%|█████████▋| 130/135 [00:31<00:01,  4.99it/s, epoch=85, grad_norm=186.158, loss=289.8421, lr=9.38e-04]
Epoch 85 TRAIN:  96%|█████████▋| 130/135 [00:31<00:01,  4.99it/s, epoch=85, grad_norm=322.702, loss=352.7821, lr=9.38e-04]
Epoch 85 TRAIN:  99%|█████████▊| 133/135 [00:31<00:00,  4.92it/s, epoch=85, grad_norm=322.702, loss=352.7821, lr=9.38e-04]
Epoch 85 TRAIN:  99%|█████████▊| 133/135 [00:31<00:00,  4.92it/s, epoch=85, grad_norm=186.053, loss=309.1658, lr=9.38e-04]
Epoch 85 TRAIN:  99%|█████████▊| 133/135 [00:32<00:00,  4.92it/s, epoch=85, grad_norm=143.324, loss=319.8001, lr=9.38e-04]
Epoch 85 TRAIN:  99%|█████████▊| 133/135 [00:32<00:00,  4.92it/s, epoch=85, grad_norm=273.339, loss=252.8048, lr=9.38e-04]
Epoch 85 TRAIN: 136it [00:32,  4.90it/s, epoch=85, grad_norm=273.339, loss=252.8048, lr=9.38e-04]                         
Epoch 85 TRAIN: 136it [00:32,  4.22it/s, epoch=85, grad_norm=273.339, loss=252.8048, lr=9.38e-04]

CV:   0%|          | 0/6 [00:00<?, ?it/s]/home/rookie/Documents/batuka/projects/wenet/wenet/wenet/utils/train_utils.py:689: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  amp_autocast(enabled=dtype is not None,

CV:  17%|█▋        | 1/6 [00:00<00:03,  1.33it/s]
                                                 
2026-04-16 20:36:32,979 INFO Epoch 85 Step 11728 CV info lr 9.3824e-04 cv_loss 459.19052678888494 rank 0 acc 0.3641466200351715
2026-04-16 20:36:32,980 INFO [Rank 0] Checkpoint: save to checkpoint exp/conformer_mongolian/epoch_85.pt
2026-04-16 20:36:33,092 INFO Epoch 86 Step 11728 TRAIN info lr 9.3824e-04 rank 0
2026-04-16 20:36:33,105 INFO using accumulate grad, new batch size is 1 times larger than before

Epoch 86 TRAIN:   0%|          | 0/136 [00:00<?, ?it/s]
Epoch 86 TRAIN:   0%|          | 0/136 [00:03<?, ?it/s, epoch=86, grad_norm=355.102, loss=296.8096, lr=9.38e-04]
Epoch 86 TRAIN:   1%|          | 1/136 [00:03<08:38,  3.84s/it, epoch=86, grad_norm=355.102, loss=296.8096, lr=9.38e-04]
Epoch 86 TRAIN:   1%|          | 1/136 [00:03<08:38,  3.84s/it, epoch=86, grad_norm=219.501, loss=207.0410, lr=9.38e-04]
Epoch 86 TRAIN:   1%|          | 1/136 [00:03<08:38,  3.84s/it, epoch=86, grad_norm=375.212, loss=388.8384, lr=9.38e-04]
Epoch 86 TRAIN:   1%|          | 1/136 [00:04<08:38,  3.84s/it, epoch=86, grad_norm=282.427, loss=286.7870, lr=9.39e-04]
Epoch 86 TRAIN:   1%|          | 1/136 [00:04<08:38,  3.84s/it, epoch=86, grad_norm=332.445, loss=310.2806, lr=9.39e-04]
Epoch 86 TRAIN:   1%|          | 1/136 [00:04<08:38,  3.84s/it, epoch=86, grad_norm=263.029, loss=341.0836, lr=9.39e-04]
Epoch 86 TRAIN:   4%|▍         | 6/136 [00:04<01:13,  1.77it/s, epoch=86, grad_norm=263.029, loss=341.0836, lr=9.39e-04]
Epoch 86 TRAIN:   4%|▍         | 6/136 [00:04<01:13,  1.77it/s, epoch=86, grad_norm=199.537, loss=278.2217, lr=9.39e-04]
Epoch 86 TRAIN:   4%|▍         | 6/136 [00:04<01:13,  1.77it/s, epoch=86, grad_norm=164.252, loss=240.7597, lr=9.39e-04]
Epoch 86 TRAIN:   4%|▍         | 6/136 [00:04<01:13,  1.77it/s, epoch=86, grad_norm=117.597, loss=287.2363, lr=9.39e-04]
Epoch 86 TRAIN:   4%|▍         | 6/136 [00:04<01:13,  1.77it/s, epoch=86, grad_norm=90.084, loss=281.5157, lr=9.39e-04] 
Epoch 86 TRAIN:   4%|▍         | 6/136 [00:05<01:13,  1.77it/s, epoch=86, grad_norm=118.491, loss=277.3549, lr=9.39e-04]
Epoch 86 TRAIN:   8%|▊         | 11/136 [00:05<00:40,  3.11it/s, epoch=86, grad_norm=118.491, loss=277.3549, lr=9.39e-04]
Epoch 86 TRAIN:   8%|▊         | 11/136 [00:05<00:40,  3.11it/s, epoch=86, grad_norm=160.130, loss=254.2204, lr=9.39e-04]
Epoch 86 TRAIN:   8%|▊         | 11/136 [00:05<00:40,  3.11it/s, epoch=86, grad_norm=125.018, loss=280.7867, lr=9.39e-04]
Epoch 86 TRAIN:   8%|▊         | 11/136 [00:05<00:40,  3.11it/s, epoch=86, grad_norm=95.505, loss=298.7204, lr=9.39e-04] 
Epoch 86 TRAIN:   8%|▊         | 11/136 [00:05<00:40,  3.11it/s, epoch=86, grad_norm=104.771, loss=276.5073, lr=9.39e-04]
Epoch 86 TRAIN:  11%|█         | 15/136 [00:05<00:31,  3.80it/s, epoch=86, grad_norm=104.771, loss=276.5073, lr=9.39e-04]
Epoch 86 TRAIN:  11%|█         | 15/136 [00:05<00:31,  3.80it/s, epoch=86, grad_norm=129.777, loss=261.5680, lr=9.40e-04]
Epoch 86 TRAIN:  11%|█         | 15/136 [00:06<00:31,  3.80it/s, epoch=86, grad_norm=113.933, loss=292.8252, lr=9.40e-04]
Epoch 86 TRAIN:  11%|█         | 15/136 [00:06<00:31,  3.80it/s, epoch=86, grad_norm=100.080, loss=265.1831, lr=9.40e-04]
Epoch 86 TRAIN:  11%|█         | 15/136 [00:06<00:31,  3.80it/s, epoch=86, grad_norm=76.537, loss=281.9316, lr=9.40e-04] 
Epoch 86 TRAIN:  14%|█▍        | 19/136 [00:06<00:27,  4.25it/s, epoch=86, grad_norm=76.537, loss=281.9316, lr=9.40e-04]
Epoch 86 TRAIN:  14%|█▍        | 19/136 [00:06<00:27,  4.25it/s, epoch=86, grad_norm=84.988, loss=273.7259, lr=9.40e-04]
Epoch 86 TRAIN:  14%|█▍        | 19/136 [00:06<00:27,  4.25it/s, epoch=86, grad_norm=114.287, loss=289.5909, lr=9.40e-04]
Epoch 86 TRAIN:  14%|█▍        | 19/136 [00:07<00:27,  4.25it/s, epoch=86, grad_norm=97.075, loss=258.1895, lr=9.40e-04] 
Epoch 86 TRAIN:  14%|█▍        | 19/136 [00:07<00:27,  4.25it/s, epoch=86, grad_norm=104.733, loss=258.5934, lr=9.40e-04]
Epoch 86 TRAIN:  17%|█▋        | 23/136 [00:07<00:25,  4.42it/s, epoch=86, grad_norm=104.733, loss=258.5934, lr=9.40e-04]
Epoch 86 TRAIN:  17%|█▋        | 23/136 [00:07<00:25,  4.42it/s, epoch=86, grad_norm=124.052, loss=275.4855, lr=9.40e-04]
Epoch 86 TRAIN:  17%|█▋        | 23/136 [00:07<00:25,  4.42it/s, epoch=86, grad_norm=139.981, loss=297.3073, lr=9.40e-04]
Epoch 86 TRAIN:  17%|█▋        | 23/136 [00:07<00:25,  4.42it/s, epoch=86, grad_norm=86.260, loss=245.0886, lr=9.40e-04] 
Epoch 86 TRAIN:  19%|█▉        | 26/136 [00:07<00:24,  4.48it/s, epoch=86, grad_norm=86.260, loss=245.0886, lr=9.40e-04]
Epoch 86 TRAIN:  19%|█▉        | 26/136 [00:08<00:24,  4.48it/s, epoch=86, grad_norm=74.333, loss=260.7596, lr=9.40e-04]
Epoch 86 TRAIN:  19%|█▉        | 26/136 [00:08<00:24,  4.48it/s, epoch=86, grad_norm=84.824, loss=255.4021, lr=9.40e-04]
Epoch 86 TRAIN:  19%|█▉        | 26/136 [00:08<00:24,  4.48it/s, epoch=86, grad_norm=84.364, loss=239.4268, lr=9.41e-04]
Epoch 86 TRAIN:  21%|██▏       | 29/136 [00:08<00:22,  4.67it/s, epoch=86, grad_norm=84.364, loss=239.4268, lr=9.41e-04]
Epoch 86 TRAIN:  21%|██▏       | 29/136 [00:08<00:22,  4.67it/s, epoch=86, grad_norm=75.895, loss=258.3416, lr=9.41e-04]
Epoch 86 TRAIN:  21%|██▏       | 29/136 [00:08<00:22,  4.67it/s, epoch=86, grad_norm=126.376, loss=276.0473, lr=9.41e-04]
Epoch 86 TRAIN:  21%|██▏       | 29/136 [00:09<00:22,  4.67it/s, epoch=86, grad_norm=123.803, loss=260.7973, lr=9.41e-04]
Epoch 86 TRAIN:  24%|██▎       | 32/136 [00:09<00:22,  4.72it/s, epoch=86, grad_norm=123.803, loss=260.7973, lr=9.41e-04]
Epoch 86 TRAIN:  24%|██▎       | 32/136 [00:09<00:22,  4.72it/s, epoch=86, grad_norm=181.957, loss=316.1417, lr=9.41e-04]
Epoch 86 TRAIN:  24%|██▎       | 32/136 [00:09<00:22,  4.72it/s, epoch=86, grad_norm=73.716, loss=243.9284, lr=9.41e-04] 
Epoch 86 TRAIN:  24%|██▎       | 32/136 [00:09<00:22,  4.72it/s, epoch=86, grad_norm=94.694, loss=303.8318, lr=9.41e-04]
Epoch 86 TRAIN:  26%|██▌       | 35/136 [00:09<00:21,  4.72it/s, epoch=86, grad_norm=94.694, loss=303.8318, lr=9.41e-04]
Epoch 86 TRAIN:  26%|██▌       | 35/136 [00:09<00:21,  4.72it/s, epoch=86, grad_norm=73.156, loss=293.1744, lr=9.41e-04]
Epoch 86 TRAIN:  26%|██▌       | 35/136 [00:10<00:21,  4.72it/s, epoch=86, grad_norm=134.331, loss=265.8954, lr=9.41e-04]
Epoch 86 TRAIN:  26%|██▌       | 35/136 [00:10<00:21,  4.72it/s, epoch=86, grad_norm=99.753, loss=300.1635, lr=9.41e-04] 
Epoch 86 TRAIN:  28%|██▊       | 38/136 [00:10<00:20,  4.71it/s, epoch=86, grad_norm=99.753, loss=300.1635, lr=9.41e-04]
Epoch 86 TRAIN:  28%|██▊       | 38/136 [00:10<00:20,  4.71it/s, epoch=86, grad_norm=92.156, loss=274.4995, lr=9.41e-04]
Epoch 86 TRAIN:  28%|██▊       | 38/136 [00:10<00:20,  4.71it/s, epoch=86, grad_norm=141.240, loss=264.1385, lr=9.41e-04]
Epoch 86 TRAIN:  28%|██▊       | 38/136 [00:11<00:20,  4.71it/s, epoch=86, grad_norm=163.730, loss=277.7083, lr=9.42e-04]
Epoch 86 TRAIN:  30%|███       | 41/136 [00:11<00:20,  4.69it/s, epoch=86, grad_norm=163.730, loss=277.7083, lr=9.42e-04]
Epoch 86 TRAIN:  30%|███       | 41/136 [00:11<00:20,  4.69it/s, epoch=86, grad_norm=159.839, loss=256.1149, lr=9.42e-04]
Epoch 86 TRAIN:  30%|███       | 41/136 [00:11<00:20,  4.69it/s, epoch=86, grad_norm=139.768, loss=286.3636, lr=9.42e-04]
Epoch 86 TRAIN:  30%|███       | 41/136 [00:11<00:20,  4.69it/s, epoch=86, grad_norm=114.167, loss=281.0939, lr=9.42e-04]
Epoch 86 TRAIN:  32%|███▏      | 44/136 [00:11<00:19,  4.66it/s, epoch=86, grad_norm=114.167, loss=281.0939, lr=9.42e-04]
Epoch 86 TRAIN:  32%|███▏      | 44/136 [00:11<00:19,  4.66it/s, epoch=86, grad_norm=81.945, loss=279.7068, lr=9.42e-04] 
Epoch 86 TRAIN:  32%|███▏      | 44/136 [00:12<00:19,  4.66it/s, epoch=86, grad_norm=124.268, loss=264.8521, lr=9.42e-04]
Epoch 86 TRAIN:  32%|███▏      | 44/136 [00:12<00:19,  4.66it/s, epoch=86, grad_norm=105.753, loss=291.9517, lr=9.42e-04]
Epoch 86 TRAIN:  35%|███▍      | 47/136 [00:12<00:18,  4.74it/s, epoch=86, grad_norm=105.753, loss=291.9517, lr=9.42e-04]
Epoch 86 TRAIN:  35%|███▍      | 47/136 [00:12<00:18,  4.74it/s, epoch=86, grad_norm=81.617, loss=251.3612, lr=9.42e-04] 
Epoch 86 TRAIN:  35%|███▍      | 47/136 [00:12<00:18,  4.74it/s, epoch=86, grad_norm=144.808, loss=295.7948, lr=9.42e-04]
Epoch 86 TRAIN:  35%|███▍      | 47/136 [00:12<00:18,  4.74it/s, epoch=86, grad_norm=92.351, loss=272.2487, lr=9.42e-04] 
Epoch 86 TRAIN:  37%|███▋      | 50/136 [00:12<00:17,  4.81it/s, epoch=86, grad_norm=92.351, loss=272.2487, lr=9.42e-04]
Epoch 86 TRAIN:  37%|███▋      | 50/136 [00:13<00:17,  4.81it/s, epoch=86, grad_norm=178.464, loss=322.6019, lr=9.42e-04]
Epoch 86 TRAIN:  37%|███▋      | 50/136 [00:13<00:17,  4.81it/s, epoch=86, grad_norm=96.330, loss=231.4457, lr=9.42e-04] 
Epoch 86 TRAIN:  37%|███▋      | 50/136 [00:13<00:17,  4.81it/s, epoch=86, grad_norm=86.897, loss=245.0591, lr=9.42e-04]
Epoch 86 TRAIN:  39%|███▉      | 53/136 [00:13<00:18,  4.42it/s, epoch=86, grad_norm=86.897, loss=245.0591, lr=9.42e-04]
Epoch 86 TRAIN:  39%|███▉      | 53/136 [00:13<00:18,  4.42it/s, epoch=86, grad_norm=94.431, loss=268.3206, lr=9.43e-04]
Epoch 86 TRAIN:  39%|███▉      | 53/136 [00:14<00:18,  4.42it/s, epoch=86, grad_norm=68.069, loss=229.8645, lr=9.43e-04]
Epoch 86 TRAIN:  39%|███▉      | 53/136 [00:14<00:18,  4.42it/s, epoch=86, grad_norm=134.116, loss=297.9095, lr=9.43e-04]
Epoch 86 TRAIN:  41%|████      | 56/136 [00:14<00:17,  4.46it/s, epoch=86, grad_norm=134.116, loss=297.9095, lr=9.43e-04]
Epoch 86 TRAIN:  41%|████      | 56/136 [00:14<00:17,  4.46it/s, epoch=86, grad_norm=80.040, loss=290.6613, lr=9.43e-04] 
Epoch 86 TRAIN:  41%|████      | 56/136 [00:14<00:17,  4.46it/s, epoch=86, grad_norm=139.400, loss=307.5338, lr=9.43e-04]
Epoch 86 TRAIN:  41%|████      | 56/136 [00:14<00:17,  4.46it/s, epoch=86, grad_norm=129.727, loss=343.3111, lr=9.43e-04]
Epoch 86 TRAIN:  43%|████▎     | 59/136 [00:14<00:16,  4.69it/s, epoch=86, grad_norm=129.727, loss=343.3111, lr=9.43e-04]
Epoch 86 TRAIN:  43%|████▎     | 59/136 [00:15<00:16,  4.69it/s, epoch=86, grad_norm=140.516, loss=311.2571, lr=9.43e-04]
Epoch 86 TRAIN:  43%|████▎     | 59/136 [00:15<00:16,  4.69it/s, epoch=86, grad_norm=99.719, loss=294.2190, lr=9.43e-04] 
Epoch 86 TRAIN:  43%|████▎     | 59/136 [00:15<00:16,  4.69it/s, epoch=86, grad_norm=92.033, loss=287.5964, lr=9.43e-04]
Epoch 86 TRAIN:  46%|████▌     | 62/136 [00:15<00:16,  4.50it/s, epoch=86, grad_norm=92.033, loss=287.5964, lr=9.43e-04]
Epoch 86 TRAIN:  46%|████▌     | 62/136 [00:15<00:16,  4.50it/s, epoch=86, grad_norm=113.623, loss=255.4695, lr=9.43e-04]
Epoch 86 TRAIN:  46%|████▌     | 62/136 [00:16<00:16,  4.50it/s, epoch=86, grad_norm=191.861, loss=222.9028, lr=9.43e-04]
Epoch 86 TRAIN:  46%|████▌     | 62/136 [00:16<00:16,  4.50it/s, epoch=86, grad_norm=206.477, loss=239.9849, lr=9.43e-04]
Epoch 86 TRAIN:  48%|████▊     | 65/136 [00:16<00:15,  4.46it/s, epoch=86, grad_norm=206.477, loss=239.9849, lr=9.43e-04]
Epoch 86 TRAIN:  48%|████▊     | 65/136 [00:16<00:15,  4.46it/s, epoch=86, grad_norm=176.853, loss=275.9254, lr=9.44e-04]
Epoch 86 TRAIN:  48%|████▊     | 65/136 [00:16<00:15,  4.46it/s, epoch=86, grad_norm=136.703, loss=277.2298, lr=9.44e-04]
Epoch 86 TRAIN:  48%|████▊     | 65/136 [00:17<00:15,  4.46it/s, epoch=86, grad_norm=78.715, loss=268.0154, lr=9.44e-04] 
Epoch 86 TRAIN:  50%|█████     | 68/136 [00:17<00:15,  4.35it/s, epoch=86, grad_norm=78.715, loss=268.0154, lr=9.44e-04]
Epoch 86 TRAIN:  50%|█████     | 68/136 [00:17<00:15,  4.35it/s, epoch=86, grad_norm=88.819, loss=271.4112, lr=9.44e-04]
Epoch 86 TRAIN:  50%|█████     | 68/136 [00:17<00:15,  4.35it/s, epoch=86, grad_norm=152.440, loss=243.2886, lr=9.44e-04]
Epoch 86 TRAIN:  50%|█████     | 68/136 [00:17<00:15,  4.35it/s, epoch=86, grad_norm=107.479, loss=261.9773, lr=9.44e-04]
Epoch 86 TRAIN:  52%|█████▏    | 71/136 [00:17<00:14,  4.48it/s, epoch=86, grad_norm=107.479, loss=261.9773, lr=9.44e-04]
Epoch 86 TRAIN:  52%|█████▏    | 71/136 [00:17<00:14,  4.48it/s, epoch=86, grad_norm=123.767, loss=242.6560, lr=9.44e-04]
Epoch 86 TRAIN:  52%|█████▏    | 71/136 [00:18<00:14,  4.48it/s, epoch=86, grad_norm=120.036, loss=241.5989, lr=9.44e-04]
Epoch 86 TRAIN:  52%|█████▏    | 71/136 [00:18<00:14,  4.48it/s, epoch=86, grad_norm=184.007, loss=221.0668, lr=9.44e-04]
Epoch 86 TRAIN:  54%|█████▍    | 74/136 [00:18<00:13,  4.69it/s, epoch=86, grad_norm=184.007, loss=221.0668, lr=9.44e-04]
Epoch 86 TRAIN:  54%|█████▍    | 74/136 [00:18<00:13,  4.69it/s, epoch=86, grad_norm=137.967, loss=251.8195, lr=9.44e-04]
Epoch 86 TRAIN:  54%|█████▍    | 74/136 [00:18<00:13,  4.69it/s, epoch=86, grad_norm=221.423, loss=292.1382, lr=9.44e-04]
Epoch 86 TRAIN:  54%|█████▍    | 74/136 [00:18<00:13,  4.69it/s, epoch=86, grad_norm=70.603, loss=228.7067, lr=9.44e-04] 
Epoch 86 TRAIN:  57%|█████▋    | 77/136 [00:18<00:12,  4.70it/s, epoch=86, grad_norm=70.603, loss=228.7067, lr=9.44e-04]
Epoch 86 TRAIN:  57%|█████▋    | 77/136 [00:19<00:12,  4.70it/s, epoch=86, grad_norm=153.466, loss=209.2883, lr=9.44e-04]
Epoch 86 TRAIN:  57%|█████▋    | 77/136 [00:19<00:12,  4.70it/s, epoch=86, grad_norm=146.380, loss=251.6378, lr=9.45e-04]
Epoch 86 TRAIN:  57%|█████▋    | 77/136 [00:19<00:12,  4.70it/s, epoch=86, grad_norm=107.612, loss=246.1901, lr=9.45e-04]
Epoch 86 TRAIN:  59%|█████▉    | 80/136 [00:19<00:12,  4.51it/s, epoch=86, grad_norm=107.612, loss=246.1901, lr=9.45e-04]
Epoch 86 TRAIN:  59%|█████▉    | 80/136 [00:19<00:12,  4.51it/s, epoch=86, grad_norm=106.908, loss=211.1890, lr=9.45e-04]
Epoch 86 TRAIN:  59%|█████▉    | 80/136 [00:20<00:12,  4.51it/s, epoch=86, grad_norm=163.242, loss=306.2609, lr=9.45e-04]
Epoch 86 TRAIN:  59%|█████▉    | 80/136 [00:20<00:12,  4.51it/s, epoch=86, grad_norm=140.206, loss=305.4969, lr=9.45e-04]
Epoch 86 TRAIN:  61%|██████    | 83/136 [00:20<00:11,  4.51it/s, epoch=86, grad_norm=140.206, loss=305.4969, lr=9.45e-04]
Epoch 86 TRAIN:  61%|██████    | 83/136 [00:20<00:11,  4.51it/s, epoch=86, grad_norm=117.141, loss=258.5940, lr=9.45e-04]
Epoch 86 TRAIN:  61%|██████    | 83/136 [00:20<00:11,  4.51it/s, epoch=86, grad_norm=179.893, loss=279.3103, lr=9.45e-04]
Epoch 86 TRAIN:  61%|██████    | 83/136 [00:20<00:11,  4.51it/s, epoch=86, grad_norm=160.444, loss=314.1037, lr=9.45e-04]
Epoch 86 TRAIN:  63%|██████▎   | 86/136 [00:20<00:10,  4.66it/s, epoch=86, grad_norm=160.444, loss=314.1037, lr=9.45e-04]
Epoch 86 TRAIN:  63%|██████▎   | 86/136 [00:21<00:10,  4.66it/s, epoch=86, grad_norm=109.696, loss=264.3987, lr=9.45e-04]
Epoch 86 TRAIN:  63%|██████▎   | 86/136 [00:21<00:10,  4.66it/s, epoch=86, grad_norm=116.927, loss=272.0437, lr=9.45e-04]
Epoch 86 TRAIN:  63%|██████▎   | 86/136 [00:21<00:10,  4.66it/s, epoch=86, grad_norm=178.805, loss=251.8300, lr=9.45e-04]
Epoch 86 TRAIN:  65%|██████▌   | 89/136 [00:21<00:10,  4.31it/s, epoch=86, grad_norm=178.805, loss=251.8300, lr=9.45e-04]
Epoch 86 TRAIN:  65%|██████▌   | 89/136 [00:21<00:10,  4.31it/s, epoch=86, grad_norm=362.916, loss=276.7564, lr=9.45e-04]
Epoch 86 TRAIN:  65%|██████▌   | 89/136 [00:22<00:10,  4.31it/s, epoch=86, grad_norm=294.262, loss=242.8496, lr=9.46e-04]
Epoch 86 TRAIN:  65%|██████▌   | 89/136 [00:22<00:10,  4.31it/s, epoch=86, grad_norm=115.486, loss=251.2834, lr=9.46e-04]
Epoch 86 TRAIN:  68%|██████▊   | 92/136 [00:22<00:10,  4.33it/s, epoch=86, grad_norm=115.486, loss=251.2834, lr=9.46e-04]
Epoch 86 TRAIN:  68%|██████▊   | 92/136 [00:22<00:10,  4.33it/s, epoch=86, grad_norm=180.234, loss=214.7619, lr=9.46e-04]
Epoch 86 TRAIN:  68%|██████▊   | 92/136 [00:22<00:10,  4.33it/s, epoch=86, grad_norm=100.359, loss=296.4984, lr=9.46e-04]
Epoch 86 TRAIN:  68%|██████▊   | 92/136 [00:23<00:10,  4.33it/s, epoch=86, grad_norm=142.892, loss=198.4979, lr=9.46e-04]
Epoch 86 TRAIN:  70%|██████▉   | 95/136 [00:23<00:09,  4.40it/s, epoch=86, grad_norm=142.892, loss=198.4979, lr=9.46e-04]
Epoch 86 TRAIN:  70%|██████▉   | 95/136 [00:23<00:09,  4.40it/s, epoch=86, grad_norm=239.888, loss=320.2433, lr=9.46e-04]
Epoch 86 TRAIN:  70%|██████▉   | 95/136 [00:23<00:09,  4.40it/s, epoch=86, grad_norm=212.039, loss=288.0580, lr=9.46e-04]
Epoch 86 TRAIN:  70%|██████▉   | 95/136 [00:23<00:09,  4.40it/s, epoch=86, grad_norm=216.244, loss=258.7153, lr=9.46e-04]
Epoch 86 TRAIN:  72%|███████▏  | 98/136 [00:23<00:08,  4.54it/s, epoch=86, grad_norm=216.244, loss=258.7153, lr=9.46e-04]
Epoch 86 TRAIN:  72%|███████▏  | 98/136 [00:23<00:08,  4.54it/s, epoch=86, grad_norm=117.193, loss=220.6374, lr=9.46e-04]
Epoch 86 TRAIN:  72%|███████▏  | 98/136 [00:24<00:08,  4.54it/s, epoch=86, grad_norm=231.866, loss=292.5394, lr=9.46e-04]2026-04-16 20:36:57,241 DEBUG TRAIN | steps/sec 4.164| Batch 86/100 loss 292.539429 loss_att 218.451538 loss_ctc 465.411133 th_accuracy 0.463492 lr 9.4624e-04 grad_norm 231.865570 rank 0

Epoch 86 TRAIN:  72%|███████▏  | 98/136 [00:24<00:08,  4.54it/s, epoch=86, grad_norm=171.618, loss=244.1158, lr=9.46e-04]
Epoch 86 TRAIN:  74%|███████▍  | 101/136 [00:24<00:07,  4.52it/s, epoch=86, grad_norm=171.618, loss=244.1158, lr=9.46e-04]
Epoch 86 TRAIN:  74%|███████▍  | 101/136 [00:24<00:07,  4.52it/s, epoch=86, grad_norm=108.217, loss=254.8310, lr=9.46e-04]
Epoch 86 TRAIN:  74%|███████▍  | 101/136 [00:24<00:07,  4.52it/s, epoch=86, grad_norm=90.121, loss=209.5280, lr=9.46e-04] 
Epoch 86 TRAIN:  74%|███████▍  | 101/136 [00:24<00:07,  4.52it/s, epoch=86, grad_norm=88.564, loss=216.9099, lr=9.47e-04]
Epoch 86 TRAIN:  76%|███████▋  | 104/136 [00:24<00:06,  4.63it/s, epoch=86, grad_norm=88.564, loss=216.9099, lr=9.47e-04]
Epoch 86 TRAIN:  76%|███████▋  | 104/136 [00:25<00:06,  4.63it/s, epoch=86, grad_norm=109.615, loss=289.2437, lr=9.47e-04]
Epoch 86 TRAIN:  76%|███████▋  | 104/136 [00:25<00:06,  4.63it/s, epoch=86, grad_norm=151.202, loss=240.6938, lr=9.47e-04]
Epoch 86 TRAIN:  76%|███████▋  | 104/136 [00:25<00:06,  4.63it/s, epoch=86, grad_norm=169.732, loss=162.0880, lr=9.47e-04]
Epoch 86 TRAIN:  79%|███████▊  | 107/136 [00:25<00:06,  4.63it/s, epoch=86, grad_norm=169.732, loss=162.0880, lr=9.47e-04]
Epoch 86 TRAIN:  79%|███████▊  | 107/136 [00:25<00:06,  4.63it/s, epoch=86, grad_norm=152.635, loss=278.3743, lr=9.47e-04]
Epoch 86 TRAIN:  79%|███████▊  | 107/136 [00:26<00:06,  4.63it/s, epoch=86, grad_norm=104.636, loss=270.6425, lr=9.47e-04]
Epoch 86 TRAIN:  79%|███████▊  | 107/136 [00:26<00:06,  4.63it/s, epoch=86, grad_norm=131.245, loss=238.5110, lr=9.47e-04]
Epoch 86 TRAIN:  81%|████████  | 110/136 [00:26<00:05,  4.50it/s, epoch=86, grad_norm=131.245, loss=238.5110, lr=9.47e-04]
Epoch 86 TRAIN:  81%|████████  | 110/136 [00:26<00:05,  4.50it/s, epoch=86, grad_norm=92.987, loss=253.2067, lr=9.47e-04] 
Epoch 86 TRAIN:  81%|████████  | 110/136 [00:26<00:05,  4.50it/s, epoch=86, grad_norm=180.211, loss=270.4018, lr=9.47e-04]
Epoch 86 TRAIN:  81%|████████  | 110/136 [00:26<00:05,  4.50it/s, epoch=86, grad_norm=143.982, loss=244.6319, lr=9.47e-04]
Epoch 86 TRAIN:  83%|████████▎ | 113/136 [00:26<00:05,  4.56it/s, epoch=86, grad_norm=143.982, loss=244.6319, lr=9.47e-04]
Epoch 86 TRAIN:  83%|████████▎ | 113/136 [00:27<00:05,  4.56it/s, epoch=86, grad_norm=97.030, loss=233.3979, lr=9.47e-04] 
Epoch 86 TRAIN:  83%|████████▎ | 113/136 [00:27<00:05,  4.56it/s, epoch=86, grad_norm=143.822, loss=220.4051, lr=9.47e-04]
Epoch 86 TRAIN:  83%|████████▎ | 113/136 [00:27<00:05,  4.56it/s, epoch=86, grad_norm=144.387, loss=268.4052, lr=9.48e-04]
Epoch 86 TRAIN:  85%|████████▌ | 116/136 [00:27<00:04,  4.59it/s, epoch=86, grad_norm=144.387, loss=268.4052, lr=9.48e-04]
Epoch 86 TRAIN:  85%|████████▌ | 116/136 [00:27<00:04,  4.59it/s, epoch=86, grad_norm=112.465, loss=201.6402, lr=9.48e-04]
Epoch 86 TRAIN:  85%|████████▌ | 116/136 [00:28<00:04,  4.59it/s, epoch=86, grad_norm=207.358, loss=258.3993, lr=9.48e-04]
Epoch 86 TRAIN:  85%|████████▌ | 116/136 [00:28<00:04,  4.59it/s, epoch=86, grad_norm=172.097, loss=295.6918, lr=9.48e-04]
Epoch 86 TRAIN:  88%|████████▊ | 119/136 [00:28<00:03,  4.45it/s, epoch=86, grad_norm=172.097, loss=295.6918, lr=9.48e-04]
Epoch 86 TRAIN:  88%|████████▊ | 119/136 [00:28<00:03,  4.45it/s, epoch=86, grad_norm=296.137, loss=312.3435, lr=9.48e-04]
Epoch 86 TRAIN:  88%|████████▊ | 119/136 [00:28<00:03,  4.45it/s, epoch=86, grad_norm=132.906, loss=250.6590, lr=9.48e-04]
Epoch 86 TRAIN:  88%|████████▊ | 119/136 [00:28<00:03,  4.45it/s, epoch=86, grad_norm=215.136, loss=229.4800, lr=9.48e-04]
Epoch 86 TRAIN:  88%|████████▊ | 119/136 [00:28<00:03,  4.45it/s, epoch=86, grad_norm=165.799, loss=284.3484, lr=9.48e-04]
Epoch 86 TRAIN:  90%|█████████ | 123/136 [00:28<00:02,  4.93it/s, epoch=86, grad_norm=165.799, loss=284.3484, lr=9.48e-04]
Epoch 86 TRAIN:  90%|█████████ | 123/136 [00:29<00:02,  4.93it/s, epoch=86, grad_norm=220.798, loss=274.9167, lr=9.48e-04]
Epoch 86 TRAIN:  90%|█████████ | 123/136 [00:29<00:02,  4.93it/s, epoch=86, grad_norm=188.025, loss=286.1172, lr=9.48e-04]
Epoch 86 TRAIN:  90%|█████████ | 123/136 [00:29<00:02,  4.93it/s, epoch=86, grad_norm=125.972, loss=252.5742, lr=9.48e-04]
Epoch 86 TRAIN:  93%|█████████▎| 126/136 [00:29<00:02,  4.68it/s, epoch=86, grad_norm=125.972, loss=252.5742, lr=9.48e-04]
Epoch 86 TRAIN:  93%|█████████▎| 126/136 [00:29<00:02,  4.68it/s, epoch=86, grad_norm=170.977, loss=287.9242, lr=9.48e-04]
Epoch 86 TRAIN:  93%|█████████▎| 126/136 [00:30<00:02,  4.68it/s, epoch=86, grad_norm=185.723, loss=272.8418, lr=9.48e-04]
Epoch 86 TRAIN:  93%|█████████▎| 126/136 [00:30<00:02,  4.68it/s, epoch=86, grad_norm=160.187, loss=316.5027, lr=9.49e-04]
Epoch 86 TRAIN:  95%|█████████▍| 129/136 [00:30<00:01,  4.72it/s, epoch=86, grad_norm=160.187, loss=316.5027, lr=9.49e-04]
Epoch 86 TRAIN:  95%|█████████▍| 129/136 [00:30<00:01,  4.72it/s, epoch=86, grad_norm=185.703, loss=248.1669, lr=9.49e-04]
Epoch 86 TRAIN:  95%|█████████▍| 129/136 [00:30<00:01,  4.72it/s, epoch=86, grad_norm=174.645, loss=272.8232, lr=9.49e-04]
Epoch 86 TRAIN:  95%|█████████▍| 129/136 [00:30<00:01,  4.72it/s, epoch=86, grad_norm=268.895, loss=333.4693, lr=9.49e-04]
Epoch 86 TRAIN:  97%|█████████▋| 132/136 [00:30<00:00,  4.66it/s, epoch=86, grad_norm=268.895, loss=333.4693, lr=9.49e-04]
Epoch 86 TRAIN:  97%|█████████▋| 132/136 [00:31<00:00,  4.66it/s, epoch=86, grad_norm=169.251, loss=351.7829, lr=9.49e-04]
Epoch 86 TRAIN:  97%|█████████▋| 132/136 [00:31<00:00,  4.66it/s, epoch=86, grad_norm=240.584, loss=205.0374, lr=9.49e-04]
Epoch 86 TRAIN:  97%|█████████▋| 132/136 [00:31<00:00,  4.66it/s, epoch=86, grad_norm=296.201, loss=333.0464, lr=9.49e-04]
Epoch 86 TRAIN:  99%|█████████▉| 135/136 [00:31<00:00,  4.91it/s, epoch=86, grad_norm=296.201, loss=333.0464, lr=9.49e-04]
Epoch 86 TRAIN:  99%|█████████▉| 135/136 [00:31<00:00,  4.91it/s, epoch=86, grad_norm=201.699, loss=295.6630, lr=9.49e-04]
Epoch 86 TRAIN: 100%|██████████| 136/136 [00:31<00:00,  4.26it/s, epoch=86, grad_norm=201.699, loss=295.6630, lr=9.49e-04]

CV:   0%|          | 0/6 [00:00<?, ?it/s]/home/rookie/Documents/batuka/projects/wenet/wenet/wenet/utils/train_utils.py:689: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  amp_autocast(enabled=dtype is not None,

CV:  17%|█▋        | 1/6 [00:00<00:03,  1.30it/s]
                                                 
2026-04-16 20:37:05,880 INFO Epoch 86 Step 11864 CV info lr 9.4912e-04 cv_loss 449.5114635120739 rank 0 acc 0.3774115393559138
2026-04-16 20:37:05,881 INFO [Rank 0] Checkpoint: save to checkpoint exp/conformer_mongolian/epoch_86.pt
2026-04-16 20:37:05,995 INFO Epoch 87 Step 11864 TRAIN info lr 9.4912e-04 rank 0
2026-04-16 20:37:06,015 INFO using accumulate grad, new batch size is 1 times larger than before

Epoch 87 TRAIN:   0%|          | 0/136 [00:00<?, ?it/s]
Epoch 87 TRAIN:   0%|          | 0/136 [00:04<?, ?it/s, epoch=87, grad_norm=336.093, loss=285.0691, lr=9.49e-04]
Epoch 87 TRAIN:   1%|          | 1/136 [00:04<09:04,  4.03s/it, epoch=87, grad_norm=336.093, loss=285.0691, lr=9.49e-04]
Epoch 87 TRAIN:   1%|          | 1/136 [00:04<09:04,  4.03s/it, epoch=87, grad_norm=351.214, loss=319.2592, lr=9.49e-04]
Epoch 87 TRAIN:   1%|          | 1/136 [00:04<09:04,  4.03s/it, epoch=87, grad_norm=468.113, loss=340.3305, lr=9.49e-04]
Epoch 87 TRAIN:   1%|          | 1/136 [00:04<09:04,  4.03s/it, epoch=87, grad_norm=465.436, loss=383.1259, lr=9.49e-04]
Epoch 87 TRAIN:   1%|          | 1/136 [00:04<09:04,  4.03s/it, epoch=87, grad_norm=380.798, loss=272.4113, lr=9.50e-04]
Epoch 87 TRAIN:   1%|          | 1/136 [00:04<09:04,  4.03s/it, epoch=87, grad_norm=444.177, loss=395.4731, lr=9.50e-04]
Epoch 87 TRAIN:   1%|          | 1/136 [00:04<09:04,  4.03s/it, epoch=87, grad_norm=345.788, loss=352.1974, lr=9.50e-04]
Epoch 87 TRAIN:   1%|          | 1/136 [00:04<09:04,  4.03s/it, epoch=87, grad_norm=211.927, loss=265.6781, lr=9.50e-04]
Epoch 87 TRAIN:   6%|▌         | 8/136 [00:04<00:55,  2.31it/s, epoch=87, grad_norm=211.927, loss=265.6781, lr=9.50e-04]
Epoch 87 TRAIN:   6%|▌         | 8/136 [00:04<00:55,  2.31it/s, epoch=87, grad_norm=233.040, loss=310.7421, lr=9.50e-04]
Epoch 87 TRAIN:   6%|▌         | 8/136 [00:04<00:55,  2.31it/s, epoch=87, grad_norm=124.577, loss=272.2673, lr=9.50e-04]
Epoch 87 TRAIN:   6%|▌         | 8/136 [00:05<00:55,  2.31it/s, epoch=87, grad_norm=179.535, loss=315.9657, lr=9.50e-04]
Epoch 87 TRAIN:   6%|▌         | 8/136 [00:05<00:55,  2.31it/s, epoch=87, grad_norm=74.821, loss=231.1663, lr=9.50e-04] 
Epoch 87 TRAIN:   6%|▌         | 8/136 [00:05<00:55,  2.31it/s, epoch=87, grad_norm=85.185, loss=270.3553, lr=9.50e-04]
Epoch 87 TRAIN:  10%|▉         | 13/136 [00:05<00:37,  3.28it/s, epoch=87, grad_norm=85.185, loss=270.3553, lr=9.50e-04]
Epoch 87 TRAIN:  10%|▉         | 13/136 [00:05<00:37,  3.28it/s, epoch=87, grad_norm=216.337, loss=284.6975, lr=9.50e-04]
Epoch 87 TRAIN:  10%|▉         | 13/136 [00:05<00:37,  3.28it/s, epoch=87, grad_norm=120.627, loss=270.4970, lr=9.50e-04]
Epoch 87 TRAIN:  10%|▉         | 13/136 [00:05<00:37,  3.28it/s, epoch=87, grad_norm=151.198, loss=250.8986, lr=9.50e-04]
Epoch 87 TRAIN:  10%|▉         | 13/136 [00:06<00:37,  3.28it/s, epoch=87, grad_norm=119.329, loss=256.6605, lr=9.50e-04]
Epoch 87 TRAIN:  12%|█▎        | 17/136 [00:06<00:30,  3.88it/s, epoch=87, grad_norm=119.329, loss=256.6605, lr=9.50e-04]
Epoch 87 TRAIN:  12%|█▎        | 17/136 [00:06<00:30,  3.88it/s, epoch=87, grad_norm=141.613, loss=257.2407, lr=9.51e-04]
Epoch 87 TRAIN:  12%|█▎        | 17/136 [00:06<00:30,  3.88it/s, epoch=87, grad_norm=91.079, loss=290.9135, lr=9.51e-04] 
Epoch 87 TRAIN:  12%|█▎        | 17/136 [00:06<00:30,  3.88it/s, epoch=87, grad_norm=97.656, loss=304.8035, lr=9.51e-04]
Epoch 87 TRAIN:  12%|█▎        | 17/136 [00:06<00:30,  3.88it/s, epoch=87, grad_norm=90.866, loss=274.8481, lr=9.51e-04]
Epoch 87 TRAIN:  15%|█▌        | 21/136 [00:06<00:27,  4.13it/s, epoch=87, grad_norm=90.866, loss=274.8481, lr=9.51e-04]
Epoch 87 TRAIN:  15%|█▌        | 21/136 [00:07<00:27,  4.13it/s, epoch=87, grad_norm=116.561, loss=238.4456, lr=9.51e-04]
Epoch 87 TRAIN:  15%|█▌        | 21/136 [00:07<00:27,  4.13it/s, epoch=87, grad_norm=79.251, loss=275.0447, lr=9.51e-04] 
Epoch 87 TRAIN:  15%|█▌        | 21/136 [00:07<00:27,  4.13it/s, epoch=87, grad_norm=130.471, loss=308.6856, lr=9.51e-04]
Epoch 87 TRAIN:  18%|█▊        | 24/136 [00:07<00:25,  4.33it/s, epoch=87, grad_norm=130.471, loss=308.6856, lr=9.51e-04]
Epoch 87 TRAIN:  18%|█▊        | 24/136 [00:07<00:25,  4.33it/s, epoch=87, grad_norm=78.972, loss=285.2594, lr=9.51e-04] 
Epoch 87 TRAIN:  18%|█▊        | 24/136 [00:07<00:25,  4.33it/s, epoch=87, grad_norm=74.030, loss=273.9091, lr=9.51e-04]
Epoch 87 TRAIN:  18%|█▊        | 24/136 [00:08<00:25,  4.33it/s, epoch=87, grad_norm=79.820, loss=244.5642, lr=9.51e-04]
Epoch 87 TRAIN:  20%|█▉        | 27/136 [00:08<00:23,  4.55it/s, epoch=87, grad_norm=79.820, loss=244.5642, lr=9.51e-04]
Epoch 87 TRAIN:  20%|█▉        | 27/136 [00:08<00:23,  4.55it/s, epoch=87, grad_norm=81.348, loss=249.2417, lr=9.51e-04]
Epoch 87 TRAIN:  20%|█▉        | 27/136 [00:08<00:23,  4.55it/s, epoch=87, grad_norm=63.849, loss=236.8938, lr=9.51e-04]
Epoch 87 TRAIN:  20%|█▉        | 27/136 [00:08<00:23,  4.55it/s, epoch=87, grad_norm=97.997, loss=275.6578, lr=9.52e-04]
Epoch 87 TRAIN:  22%|██▏       | 30/136 [00:08<00:22,  4.78it/s, epoch=87, grad_norm=97.997, loss=275.6578, lr=9.52e-04]
Epoch 87 TRAIN:  22%|██▏       | 30/136 [00:08<00:22,  4.78it/s, epoch=87, grad_norm=73.567, loss=236.2056, lr=9.52e-04]
Epoch 87 TRAIN:  22%|██▏       | 30/136 [00:09<00:22,  4.78it/s, epoch=87, grad_norm=96.575, loss=274.5893, lr=9.52e-04]
Epoch 87 TRAIN:  22%|██▏       | 30/136 [00:09<00:22,  4.78it/s, epoch=87, grad_norm=86.522, loss=270.2922, lr=9.52e-04]
Epoch 87 TRAIN:  24%|██▍       | 33/136 [00:09<00:20,  4.94it/s, epoch=87, grad_norm=86.522, loss=270.2922, lr=9.52e-04]
Epoch 87 TRAIN:  24%|██▍       | 33/136 [00:09<00:20,  4.94it/s, epoch=87, grad_norm=63.189, loss=243.2216, lr=9.52e-04]
Epoch 87 TRAIN:  24%|██▍       | 33/136 [00:09<00:20,  4.94it/s, epoch=87, grad_norm=93.473, loss=280.0727, lr=9.52e-04]
Epoch 87 TRAIN:  24%|██▍       | 33/136 [00:09<00:20,  4.94it/s, epoch=87, grad_norm=126.115, loss=253.0163, lr=9.52e-04]
Epoch 87 TRAIN:  26%|██▋       | 36/136 [00:09<00:20,  4.87it/s, epoch=87, grad_norm=126.115, loss=253.0163, lr=9.52e-04]
Epoch 87 TRAIN:  26%|██▋       | 36/136 [00:10<00:20,  4.87it/s, epoch=87, grad_norm=101.366, loss=257.2686, lr=9.52e-04]
Epoch 87 TRAIN:  26%|██▋       | 36/136 [00:10<00:20,  4.87it/s, epoch=87, grad_norm=95.603, loss=274.9372, lr=9.52e-04] 
Epoch 87 TRAIN:  26%|██▋       | 36/136 [00:10<00:20,  4.87it/s, epoch=87, grad_norm=142.730, loss=277.5109, lr=9.52e-04]
Epoch 87 TRAIN:  29%|██▊       | 39/136 [00:10<00:20,  4.73it/s, epoch=87, grad_norm=142.730, loss=277.5109, lr=9.52e-04]
Epoch 87 TRAIN:  29%|██▊       | 39/136 [00:10<00:20,  4.73it/s, epoch=87, grad_norm=100.710, loss=261.9760, lr=9.52e-04]
Epoch 87 TRAIN:  29%|██▊       | 39/136 [00:10<00:20,  4.73it/s, epoch=87, grad_norm=92.705, loss=275.9036, lr=9.52e-04] 
Epoch 87 TRAIN:  29%|██▊       | 39/136 [00:11<00:20,  4.73it/s, epoch=87, grad_norm=168.484, loss=294.0910, lr=9.52e-04]
Epoch 87 TRAIN:  31%|███       | 42/136 [00:11<00:19,  4.75it/s, epoch=87, grad_norm=168.484, loss=294.0910, lr=9.52e-04]
Epoch 87 TRAIN:  31%|███       | 42/136 [00:11<00:19,  4.75it/s, epoch=87, grad_norm=97.151, loss=293.1141, lr=9.53e-04] 
Epoch 87 TRAIN:  31%|███       | 42/136 [00:11<00:19,  4.75it/s, epoch=87, grad_norm=100.885, loss=271.2196, lr=9.53e-04]
Epoch 87 TRAIN:  31%|███       | 42/136 [00:11<00:19,  4.75it/s, epoch=87, grad_norm=114.520, loss=311.3776, lr=9.53e-04]
Epoch 87 TRAIN:  33%|███▎      | 45/136 [00:11<00:19,  4.66it/s, epoch=87, grad_norm=114.520, loss=311.3776, lr=9.53e-04]
Epoch 87 TRAIN:  33%|███▎      | 45/136 [00:12<00:19,  4.66it/s, epoch=87, grad_norm=103.351, loss=259.3037, lr=9.53e-04]
Epoch 87 TRAIN:  33%|███▎      | 45/136 [00:12<00:19,  4.66it/s, epoch=87, grad_norm=107.113, loss=285.5661, lr=9.53e-04]
Epoch 87 TRAIN:  33%|███▎      | 45/136 [00:12<00:19,  4.66it/s, epoch=87, grad_norm=148.385, loss=292.0073, lr=9.53e-04]
Epoch 87 TRAIN:  35%|███▌      | 48/136 [00:12<00:18,  4.63it/s, epoch=87, grad_norm=148.385, loss=292.0073, lr=9.53e-04]
Epoch 87 TRAIN:  35%|███▌      | 48/136 [00:12<00:18,  4.63it/s, epoch=87, grad_norm=134.053, loss=281.9152, lr=9.53e-04]
Epoch 87 TRAIN:  35%|███▌      | 48/136 [00:12<00:18,  4.63it/s, epoch=87, grad_norm=161.125, loss=249.4571, lr=9.53e-04]
Epoch 87 TRAIN:  35%|███▌      | 48/136 [00:13<00:18,  4.63it/s, epoch=87, grad_norm=153.043, loss=281.2385, lr=9.53e-04]
Epoch 87 TRAIN:  38%|███▊      | 51/136 [00:13<00:18,  4.61it/s, epoch=87, grad_norm=153.043, loss=281.2385, lr=9.53e-04]
Epoch 87 TRAIN:  38%|███▊      | 51/136 [00:13<00:18,  4.61it/s, epoch=87, grad_norm=132.234, loss=250.9865, lr=9.53e-04]
Epoch 87 TRAIN:  38%|███▊      | 51/136 [00:13<00:18,  4.61it/s, epoch=87, grad_norm=93.147, loss=266.1648, lr=9.53e-04] 
Epoch 87 TRAIN:  38%|███▊      | 51/136 [00:13<00:18,  4.61it/s, epoch=87, grad_norm=119.214, loss=283.9831, lr=9.53e-04]
Epoch 87 TRAIN:  40%|███▉      | 54/136 [00:13<00:17,  4.56it/s, epoch=87, grad_norm=119.214, loss=283.9831, lr=9.53e-04]
Epoch 87 TRAIN:  40%|███▉      | 54/136 [00:14<00:17,  4.56it/s, epoch=87, grad_norm=152.458, loss=298.2995, lr=9.54e-04]
Epoch 87 TRAIN:  40%|███▉      | 54/136 [00:14<00:17,  4.56it/s, epoch=87, grad_norm=121.998, loss=260.7883, lr=9.54e-04]
Epoch 87 TRAIN:  40%|███▉      | 54/136 [00:14<00:17,  4.56it/s, epoch=87, grad_norm=116.743, loss=292.9569, lr=9.54e-04]
Epoch 87 TRAIN:  42%|████▏     | 57/136 [00:14<00:17,  4.60it/s, epoch=87, grad_norm=116.743, loss=292.9569, lr=9.54e-04]
Epoch 87 TRAIN:  42%|████▏     | 57/136 [00:14<00:17,  4.60it/s, epoch=87, grad_norm=79.238, loss=251.3010, lr=9.54e-04] 
Epoch 87 TRAIN:  42%|████▏     | 57/136 [00:14<00:17,  4.60it/s, epoch=87, grad_norm=139.582, loss=308.9694, lr=9.54e-04]
Epoch 87 TRAIN:  42%|████▏     | 57/136 [00:15<00:17,  4.60it/s, epoch=87, grad_norm=90.718, loss=270.5650, lr=9.54e-04] 
Epoch 87 TRAIN:  44%|████▍     | 60/136 [00:15<00:17,  4.43it/s, epoch=87, grad_norm=90.718, loss=270.5650, lr=9.54e-04]
Epoch 87 TRAIN:  44%|████▍     | 60/136 [00:15<00:17,  4.43it/s, epoch=87, grad_norm=139.528, loss=320.4991, lr=9.54e-04]
Epoch 87 TRAIN:  44%|████▍     | 60/136 [00:15<00:17,  4.43it/s, epoch=87, grad_norm=123.241, loss=204.3248, lr=9.54e-04]
Epoch 87 TRAIN:  44%|████▍     | 60/136 [00:15<00:17,  4.43it/s, epoch=87, grad_norm=126.210, loss=269.1628, lr=9.54e-04]
Epoch 87 TRAIN:  46%|████▋     | 63/136 [00:15<00:15,  4.73it/s, epoch=87, grad_norm=126.210, loss=269.1628, lr=9.54e-04]
Epoch 87 TRAIN:  46%|████▋     | 63/136 [00:16<00:15,  4.73it/s, epoch=87, grad_norm=156.913, loss=252.0240, lr=9.54e-04]
Epoch 87 TRAIN:  46%|████▋     | 63/136 [00:16<00:15,  4.73it/s, epoch=87, grad_norm=77.132, loss=243.2356, lr=9.54e-04] 
Epoch 87 TRAIN:  46%|████▋     | 63/136 [00:16<00:15,  4.73it/s, epoch=87, grad_norm=160.280, loss=261.7894, lr=9.54e-04]
Epoch 87 TRAIN:  49%|████▊     | 66/136 [00:16<00:14,  4.71it/s, epoch=87, grad_norm=160.280, loss=261.7894, lr=9.54e-04]
Epoch 87 TRAIN:  49%|████▊     | 66/136 [00:16<00:14,  4.71it/s, epoch=87, grad_norm=171.847, loss=312.0422, lr=9.54e-04]
Epoch 87 TRAIN:  49%|████▊     | 66/136 [00:16<00:14,  4.71it/s, epoch=87, grad_norm=68.351, loss=250.7830, lr=9.55e-04] 
Epoch 87 TRAIN:  49%|████▊     | 66/136 [00:16<00:14,  4.71it/s, epoch=87, grad_norm=128.303, loss=216.2654, lr=9.55e-04]
Epoch 87 TRAIN:  51%|█████     | 69/136 [00:16<00:14,  4.72it/s, epoch=87, grad_norm=128.303, loss=216.2654, lr=9.55e-04]
Epoch 87 TRAIN:  51%|█████     | 69/136 [00:17<00:14,  4.72it/s, epoch=87, grad_norm=89.759, loss=270.3951, lr=9.55e-04] 
Epoch 87 TRAIN:  51%|█████     | 69/136 [00:17<00:14,  4.72it/s, epoch=87, grad_norm=112.553, loss=272.3946, lr=9.55e-04]
Epoch 87 TRAIN:  51%|█████     | 69/136 [00:17<00:14,  4.72it/s, epoch=87, grad_norm=134.436, loss=278.8777, lr=9.55e-04]
Epoch 87 TRAIN:  53%|█████▎    | 72/136 [00:17<00:14,  4.48it/s, epoch=87, grad_norm=134.436, loss=278.8777, lr=9.55e-04]
Epoch 87 TRAIN:  53%|█████▎    | 72/136 [00:17<00:14,  4.48it/s, epoch=87, grad_norm=82.919, loss=243.6309, lr=9.55e-04] 
Epoch 87 TRAIN:  53%|█████▎    | 72/136 [00:18<00:14,  4.48it/s, epoch=87, grad_norm=190.484, loss=295.9685, lr=9.55e-04]
Epoch 87 TRAIN:  53%|█████▎    | 72/136 [00:18<00:14,  4.48it/s, epoch=87, grad_norm=94.108, loss=240.4379, lr=9.55e-04] 
Epoch 87 TRAIN:  55%|█████▌    | 75/136 [00:18<00:13,  4.53it/s, epoch=87, grad_norm=94.108, loss=240.4379, lr=9.55e-04]
Epoch 87 TRAIN:  55%|█████▌    | 75/136 [00:18<00:13,  4.53it/s, epoch=87, grad_norm=84.489, loss=230.1350, lr=9.55e-04]
Epoch 87 TRAIN:  55%|█████▌    | 75/136 [00:18<00:13,  4.53it/s, epoch=87, grad_norm=98.900, loss=256.0435, lr=9.55e-04]
Epoch 87 TRAIN:  55%|█████▌    | 75/136 [00:19<00:13,  4.53it/s, epoch=87, grad_norm=103.770, loss=259.9189, lr=9.55e-04]
Epoch 87 TRAIN:  57%|█████▋    | 78/136 [00:19<00:12,  4.52it/s, epoch=87, grad_norm=103.770, loss=259.9189, lr=9.55e-04]
Epoch 87 TRAIN:  57%|█████▋    | 78/136 [00:19<00:12,  4.52it/s, epoch=87, grad_norm=124.309, loss=293.3896, lr=9.55e-04]
Epoch 87 TRAIN:  57%|█████▋    | 78/136 [00:19<00:12,  4.52it/s, epoch=87, grad_norm=280.324, loss=206.7550, lr=9.56e-04]
Epoch 87 TRAIN:  57%|█████▋    | 78/136 [00:19<00:12,  4.52it/s, epoch=87, grad_norm=95.899, loss=230.2258, lr=9.56e-04] 
Epoch 87 TRAIN:  60%|█████▉    | 81/136 [00:19<00:11,  4.60it/s, epoch=87, grad_norm=95.899, loss=230.2258, lr=9.56e-04]
Epoch 87 TRAIN:  60%|█████▉    | 81/136 [00:19<00:11,  4.60it/s, epoch=87, grad_norm=86.672, loss=220.7166, lr=9.56e-04]
Epoch 87 TRAIN:  60%|█████▉    | 81/136 [00:20<00:11,  4.60it/s, epoch=87, grad_norm=109.772, loss=263.4019, lr=9.56e-04]
Epoch 87 TRAIN:  60%|█████▉    | 81/136 [00:20<00:11,  4.60it/s, epoch=87, grad_norm=176.924, loss=190.2398, lr=9.56e-04]
Epoch 87 TRAIN:  62%|██████▏   | 84/136 [00:20<00:11,  4.52it/s, epoch=87, grad_norm=176.924, loss=190.2398, lr=9.56e-04]
Epoch 87 TRAIN:  62%|██████▏   | 84/136 [00:20<00:11,  4.52it/s, epoch=87, grad_norm=175.082, loss=199.5360, lr=9.56e-04]
Epoch 87 TRAIN:  62%|██████▏   | 84/136 [00:20<00:11,  4.52it/s, epoch=87, grad_norm=281.351, loss=309.6865, lr=9.56e-04]
Epoch 87 TRAIN:  62%|██████▏   | 84/136 [00:20<00:11,  4.52it/s, epoch=87, grad_norm=108.782, loss=220.5655, lr=9.56e-04]
Epoch 87 TRAIN:  64%|██████▍   | 87/136 [00:20<00:10,  4.58it/s, epoch=87, grad_norm=108.782, loss=220.5655, lr=9.56e-04]
Epoch 87 TRAIN:  64%|██████▍   | 87/136 [00:21<00:10,  4.58it/s, epoch=87, grad_norm=102.239, loss=246.8564, lr=9.56e-04]
Epoch 87 TRAIN:  64%|██████▍   | 87/136 [00:21<00:10,  4.58it/s, epoch=87, grad_norm=152.930, loss=288.4613, lr=9.56e-04]
Epoch 87 TRAIN:  64%|██████▍   | 87/136 [00:21<00:10,  4.58it/s, epoch=87, grad_norm=228.022, loss=286.2523, lr=9.56e-04]
Epoch 87 TRAIN:  66%|██████▌   | 90/136 [00:21<00:10,  4.47it/s, epoch=87, grad_norm=228.022, loss=286.2523, lr=9.56e-04]
Epoch 87 TRAIN:  66%|██████▌   | 90/136 [00:21<00:10,  4.47it/s, epoch=87, grad_norm=224.599, loss=323.6732, lr=9.56e-04]
Epoch 87 TRAIN:  66%|██████▌   | 90/136 [00:22<00:10,  4.47it/s, epoch=87, grad_norm=125.800, loss=215.2630, lr=9.56e-04]
Epoch 87 TRAIN:  66%|██████▌   | 90/136 [00:22<00:10,  4.47it/s, epoch=87, grad_norm=138.392, loss=291.4613, lr=9.57e-04]
Epoch 87 TRAIN:  68%|██████▊   | 93/136 [00:22<00:09,  4.55it/s, epoch=87, grad_norm=138.392, loss=291.4613, lr=9.57e-04]
Epoch 87 TRAIN:  68%|██████▊   | 93/136 [00:22<00:09,  4.55it/s, epoch=87, grad_norm=119.416, loss=252.3299, lr=9.57e-04]
Epoch 87 TRAIN:  68%|██████▊   | 93/136 [00:22<00:09,  4.55it/s, epoch=87, grad_norm=106.574, loss=256.1440, lr=9.57e-04]
Epoch 87 TRAIN:  68%|██████▊   | 93/136 [00:22<00:09,  4.55it/s, epoch=87, grad_norm=163.273, loss=253.4378, lr=9.57e-04]
Epoch 87 TRAIN:  71%|███████   | 96/136 [00:22<00:08,  4.58it/s, epoch=87, grad_norm=163.273, loss=253.4378, lr=9.57e-04]
Epoch 87 TRAIN:  71%|███████   | 96/136 [00:23<00:08,  4.58it/s, epoch=87, grad_norm=157.254, loss=239.7336, lr=9.57e-04]
Epoch 87 TRAIN:  71%|███████   | 96/136 [00:23<00:08,  4.58it/s, epoch=87, grad_norm=86.301, loss=231.9497, lr=9.57e-04] 
Epoch 87 TRAIN:  71%|███████   | 96/136 [00:23<00:08,  4.58it/s, epoch=87, grad_norm=137.232, loss=213.9541, lr=9.57e-04]
Epoch 87 TRAIN:  73%|███████▎  | 99/136 [00:23<00:08,  4.39it/s, epoch=87, grad_norm=137.232, loss=213.9541, lr=9.57e-04]
Epoch 87 TRAIN:  73%|███████▎  | 99/136 [00:23<00:08,  4.39it/s, epoch=87, grad_norm=95.324, loss=288.2704, lr=9.57e-04] 2026-04-16 20:37:29,954 DEBUG TRAIN | steps/sec 4.157| Batch 87/100 loss 288.270447 loss_att 228.647156 loss_ctc 427.391449 th_accuracy 0.478992 lr 9.5712e-04 grad_norm 95.324326 rank 0

Epoch 87 TRAIN:  73%|███████▎  | 99/136 [00:24<00:08,  4.39it/s, epoch=87, grad_norm=152.747, loss=211.0752, lr=9.57e-04]
Epoch 87 TRAIN:  73%|███████▎  | 99/136 [00:24<00:08,  4.39it/s, epoch=87, grad_norm=183.162, loss=252.9158, lr=9.57e-04]
Epoch 87 TRAIN:  75%|███████▌  | 102/136 [00:24<00:07,  4.46it/s, epoch=87, grad_norm=183.162, loss=252.9158, lr=9.57e-04]
Epoch 87 TRAIN:  75%|███████▌  | 102/136 [00:24<00:07,  4.46it/s, epoch=87, grad_norm=148.216, loss=214.9731, lr=9.57e-04]
Epoch 87 TRAIN:  75%|███████▌  | 102/136 [00:24<00:07,  4.46it/s, epoch=87, grad_norm=166.861, loss=294.3250, lr=9.57e-04]
Epoch 87 TRAIN:  75%|███████▌  | 102/136 [00:24<00:07,  4.46it/s, epoch=87, grad_norm=85.612, loss=168.6361, lr=9.58e-04] 
Epoch 87 TRAIN:  77%|███████▋  | 105/136 [00:24<00:06,  4.67it/s, epoch=87, grad_norm=85.612, loss=168.6361, lr=9.58e-04]
Epoch 87 TRAIN:  77%|███████▋  | 105/136 [00:25<00:06,  4.67it/s, epoch=87, grad_norm=325.404, loss=267.0014, lr=9.58e-04]
Epoch 87 TRAIN:  77%|███████▋  | 105/136 [00:25<00:06,  4.67it/s, epoch=87, grad_norm=182.139, loss=221.3873, lr=9.58e-04]
Epoch 87 TRAIN:  77%|███████▋  | 105/136 [00:25<00:06,  4.67it/s, epoch=87, grad_norm=145.426, loss=229.4547, lr=9.58e-04]
Epoch 87 TRAIN:  79%|███████▉  | 108/136 [00:25<00:06,  4.65it/s, epoch=87, grad_norm=145.426, loss=229.4547, lr=9.58e-04]
Epoch 87 TRAIN:  79%|███████▉  | 108/136 [00:25<00:06,  4.65it/s, epoch=87, grad_norm=202.964, loss=243.0637, lr=9.58e-04]
Epoch 87 TRAIN:  79%|███████▉  | 108/136 [00:26<00:06,  4.65it/s, epoch=87, grad_norm=373.232, loss=347.0712, lr=9.58e-04]
Epoch 87 TRAIN:  79%|███████▉  | 108/136 [00:26<00:06,  4.65it/s, epoch=87, grad_norm=220.161, loss=252.2747, lr=9.58e-04]
Epoch 87 TRAIN:  82%|████████▏ | 111/136 [00:26<00:05,  4.68it/s, epoch=87, grad_norm=220.161, loss=252.2747, lr=9.58e-04]
Epoch 87 TRAIN:  82%|████████▏ | 111/136 [00:26<00:05,  4.68it/s, epoch=87, grad_norm=232.686, loss=328.8564, lr=9.58e-04]
Epoch 87 TRAIN:  82%|████████▏ | 111/136 [00:26<00:05,  4.68it/s, epoch=87, grad_norm=169.889, loss=250.0971, lr=9.58e-04]
Epoch 87 TRAIN:  82%|████████▏ | 111/136 [00:26<00:05,  4.68it/s, epoch=87, grad_norm=213.522, loss=321.2151, lr=9.58e-04]
Epoch 87 TRAIN:  84%|████████▍ | 114/136 [00:26<00:04,  4.59it/s, epoch=87, grad_norm=213.522, loss=321.2151, lr=9.58e-04]
Epoch 87 TRAIN:  84%|████████▍ | 114/136 [00:27<00:04,  4.59it/s, epoch=87, grad_norm=155.190, loss=286.2461, lr=9.58e-04]
Epoch 87 TRAIN:  84%|████████▍ | 114/136 [00:27<00:04,  4.59it/s, epoch=87, grad_norm=298.595, loss=279.0230, lr=9.58e-04]
Epoch 87 TRAIN:  84%|████████▍ | 114/136 [00:27<00:04,  4.59it/s, epoch=87, grad_norm=274.257, loss=243.9387, lr=9.58e-04]
Epoch 87 TRAIN:  86%|████████▌ | 117/136 [00:27<00:04,  4.36it/s, epoch=87, grad_norm=274.257, loss=243.9387, lr=9.58e-04]
Epoch 87 TRAIN:  86%|████████▌ | 117/136 [00:27<00:04,  4.36it/s, epoch=87, grad_norm=111.330, loss=240.9475, lr=9.59e-04]
Epoch 87 TRAIN:  86%|████████▌ | 117/136 [00:28<00:04,  4.36it/s, epoch=87, grad_norm=354.224, loss=287.4971, lr=9.59e-04]
Epoch 87 TRAIN:  86%|████████▌ | 117/136 [00:28<00:04,  4.36it/s, epoch=87, grad_norm=193.380, loss=342.4162, lr=9.59e-04]
Epoch 87 TRAIN:  88%|████████▊ | 120/136 [00:28<00:03,  4.42it/s, epoch=87, grad_norm=193.380, loss=342.4162, lr=9.59e-04]
Epoch 87 TRAIN:  88%|████████▊ | 120/136 [00:28<00:03,  4.42it/s, epoch=87, grad_norm=219.088, loss=221.7662, lr=9.59e-04]
Epoch 87 TRAIN:  88%|████████▊ | 120/136 [00:28<00:03,  4.42it/s, epoch=87, grad_norm=98.766, loss=241.1165, lr=9.59e-04] 
Epoch 87 TRAIN:  88%|████████▊ | 120/136 [00:28<00:03,  4.42it/s, epoch=87, grad_norm=183.083, loss=265.9474, lr=9.59e-04]
Epoch 87 TRAIN:  90%|█████████ | 123/136 [00:28<00:02,  4.60it/s, epoch=87, grad_norm=183.083, loss=265.9474, lr=9.59e-04]
Epoch 87 TRAIN:  90%|█████████ | 123/136 [00:29<00:02,  4.60it/s, epoch=87, grad_norm=174.598, loss=281.8796, lr=9.59e-04]
Epoch 87 TRAIN:  90%|█████████ | 123/136 [00:29<00:02,  4.60it/s, epoch=87, grad_norm=151.131, loss=256.0311, lr=9.59e-04]
Epoch 87 TRAIN:  90%|█████████ | 123/136 [00:29<00:02,  4.60it/s, epoch=87, grad_norm=223.376, loss=295.4587, lr=9.59e-04]
Epoch 87 TRAIN:  93%|█████████▎| 126/136 [00:29<00:02,  4.70it/s, epoch=87, grad_norm=223.376, loss=295.4587, lr=9.59e-04]
Epoch 87 TRAIN:  93%|█████████▎| 126/136 [00:29<00:02,  4.70it/s, epoch=87, grad_norm=156.704, loss=280.7125, lr=9.59e-04]
Epoch 87 TRAIN:  93%|█████████▎| 126/136 [00:29<00:02,  4.70it/s, epoch=87, grad_norm=260.322, loss=341.1370, lr=9.59e-04]
Epoch 87 TRAIN:  93%|█████████▎| 126/136 [00:30<00:02,  4.70it/s, epoch=87, grad_norm=160.477, loss=269.3403, lr=9.59e-04]
Epoch 87 TRAIN:  95%|█████████▍| 129/136 [00:30<00:01,  4.55it/s, epoch=87, grad_norm=160.477, loss=269.3403, lr=9.59e-04]
Epoch 87 TRAIN:  95%|█████████▍| 129/136 [00:30<00:01,  4.55it/s, epoch=87, grad_norm=321.434, loss=320.3219, lr=9.60e-04]
Epoch 87 TRAIN:  95%|█████████▍| 129/136 [00:30<00:01,  4.55it/s, epoch=87, grad_norm=200.820, loss=278.0151, lr=9.60e-04]
Epoch 87 TRAIN:  95%|█████████▍| 129/136 [00:30<00:01,  4.55it/s, epoch=87, grad_norm=221.360, loss=277.8024, lr=9.60e-04]
Epoch 87 TRAIN:  95%|█████████▍| 129/136 [00:30<00:01,  4.55it/s, epoch=87, grad_norm=156.867, loss=217.2744, lr=9.60e-04]
Epoch 87 TRAIN:  98%|█████████▊| 133/136 [00:30<00:00,  5.03it/s, epoch=87, grad_norm=156.867, loss=217.2744, lr=9.60e-04]
Epoch 87 TRAIN:  98%|█████████▊| 133/136 [00:31<00:00,  5.03it/s, epoch=87, grad_norm=246.177, loss=242.3894, lr=9.60e-04]
Epoch 87 TRAIN:  98%|█████████▊| 133/136 [00:31<00:00,  5.03it/s, epoch=87, grad_norm=152.812, loss=251.6875, lr=9.60e-04]
Epoch 87 TRAIN:  98%|█████████▊| 133/136 [00:31<00:00,  5.03it/s, epoch=87, grad_norm=217.088, loss=240.8592, lr=9.60e-04]
Epoch 87 TRAIN: 100%|██████████| 136/136 [00:31<00:00,  5.15it/s, epoch=87, grad_norm=217.088, loss=240.8592, lr=9.60e-04]
Epoch 87 TRAIN: 100%|██████████| 136/136 [00:31<00:00,  5.15it/s, epoch=87, grad_norm=227.509, loss=309.6034, lr=9.60e-04]
Epoch 87 TRAIN: 137it [00:31,  4.33it/s, epoch=87, grad_norm=227.509, loss=309.6034, lr=9.60e-04]                         

CV:   0%|          | 0/6 [00:00<?, ?it/s]/home/rookie/Documents/batuka/projects/wenet/wenet/wenet/utils/train_utils.py:689: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  amp_autocast(enabled=dtype is not None,

CV:  17%|█▋        | 1/6 [00:00<00:03,  1.33it/s]
                                                 
2026-04-16 20:37:38,509 INFO Epoch 87 Step 12001 CV info lr 9.6008e-04 cv_loss 442.2331431995739 rank 0 acc 0.40943661828835803
2026-04-16 20:37:38,510 INFO [Rank 0] Checkpoint: save to checkpoint exp/conformer_mongolian/epoch_87.pt
2026-04-16 20:37:38,623 INFO Epoch 88 Step 12001 TRAIN info lr 9.6008e-04 rank 0
2026-04-16 20:37:38,644 INFO using accumulate grad, new batch size is 1 times larger than before

Epoch 88 TRAIN:   0%|          | 0/137 [00:00<?, ?it/s]
Epoch 88 TRAIN:   0%|          | 0/137 [00:04<?, ?it/s, epoch=88, grad_norm=311.012, loss=270.6783, lr=9.60e-04]
Epoch 88 TRAIN:   1%|          | 1/137 [00:04<09:15,  4.08s/it, epoch=88, grad_norm=311.012, loss=270.6783, lr=9.60e-04]
Epoch 88 TRAIN:   1%|          | 1/137 [00:04<09:15,  4.08s/it, epoch=88, grad_norm=401.265, loss=320.6761, lr=9.60e-04]
Epoch 88 TRAIN:   1%|          | 1/137 [00:04<09:15,  4.08s/it, epoch=88, grad_norm=356.677, loss=297.1334, lr=9.60e-04]
Epoch 88 TRAIN:   1%|          | 1/137 [00:04<09:15,  4.08s/it, epoch=88, grad_norm=412.743, loss=341.5834, lr=9.60e-04]
Epoch 88 TRAIN:   1%|          | 1/137 [00:04<09:15,  4.08s/it, epoch=88, grad_norm=266.713, loss=245.7924, lr=9.60e-04]
Epoch 88 TRAIN:   1%|          | 1/137 [00:04<09:15,  4.08s/it, epoch=88, grad_norm=384.992, loss=343.2625, lr=9.61e-04]
Epoch 88 TRAIN:   1%|          | 1/137 [00:04<09:15,  4.08s/it, epoch=88, grad_norm=297.627, loss=294.0496, lr=9.61e-04]
Epoch 88 TRAIN:   1%|          | 1/137 [00:04<09:15,  4.08s/it, epoch=88, grad_norm=255.858, loss=227.0685, lr=9.61e-04]
Epoch 88 TRAIN:   6%|▌         | 8/137 [00:04<00:57,  2.26it/s, epoch=88, grad_norm=255.858, loss=227.0685, lr=9.61e-04]
Epoch 88 TRAIN:   6%|▌         | 8/137 [00:04<00:57,  2.26it/s, epoch=88, grad_norm=282.619, loss=297.7803, lr=9.61e-04]
Epoch 88 TRAIN:   6%|▌         | 8/137 [00:04<00:57,  2.26it/s, epoch=88, grad_norm=174.782, loss=270.4782, lr=9.61e-04]
Epoch 88 TRAIN:   6%|▌         | 8/137 [00:05<00:57,  2.26it/s, epoch=88, grad_norm=303.226, loss=340.0986, lr=9.61e-04]
Epoch 88 TRAIN:   6%|▌         | 8/137 [00:05<00:57,  2.26it/s, epoch=88, grad_norm=207.679, loss=302.7405, lr=9.61e-04]
Epoch 88 TRAIN:   9%|▉         | 12/137 [00:05<00:40,  3.08it/s, epoch=88, grad_norm=207.679, loss=302.7405, lr=9.61e-04]
Epoch 88 TRAIN:   9%|▉         | 12/137 [00:05<00:40,  3.08it/s, epoch=88, grad_norm=102.053, loss=229.1913, lr=9.61e-04]
Epoch 88 TRAIN:   9%|▉         | 12/137 [00:05<00:40,  3.08it/s, epoch=88, grad_norm=80.502, loss=273.9624, lr=9.61e-04] 
Epoch 88 TRAIN:   9%|▉         | 12/137 [00:05<00:40,  3.08it/s, epoch=88, grad_norm=104.053, loss=307.0715, lr=9.61e-04]
Epoch 88 TRAIN:   9%|▉         | 12/137 [00:06<00:40,  3.08it/s, epoch=88, grad_norm=91.084, loss=286.4673, lr=9.61e-04] 
Epoch 88 TRAIN:  12%|█▏        | 16/137 [00:06<00:33,  3.64it/s, epoch=88, grad_norm=91.084, loss=286.4673, lr=9.61e-04]
Epoch 88 TRAIN:  12%|█▏        | 16/137 [00:06<00:33,  3.64it/s, epoch=88, grad_norm=193.687, loss=241.0387, lr=9.61e-04]
Epoch 88 TRAIN:  12%|█▏        | 16/137 [00:06<00:33,  3.64it/s, epoch=88, grad_norm=175.601, loss=260.7288, lr=9.62e-04]
Epoch 88 TRAIN:  12%|█▏        | 16/137 [00:06<00:33,  3.64it/s, epoch=88, grad_norm=195.449, loss=266.0324, lr=9.62e-04]
Epoch 88 TRAIN:  12%|█▏        | 16/137 [00:06<00:33,  3.64it/s, epoch=88, grad_norm=178.076, loss=255.7254, lr=9.62e-04]
Epoch 88 TRAIN:  15%|█▍        | 20/137 [00:06<00:28,  4.06it/s, epoch=88, grad_norm=178.076, loss=255.7254, lr=9.62e-04]
Epoch 88 TRAIN:  15%|█▍        | 20/137 [00:07<00:28,  4.06it/s, epoch=88, grad_norm=134.828, loss=283.8158, lr=9.62e-04]
Epoch 88 TRAIN:  15%|█▍        | 20/137 [00:07<00:28,  4.06it/s, epoch=88, grad_norm=152.505, loss=274.3907, lr=9.62e-04]
Epoch 88 TRAIN:  15%|█▍        | 20/137 [00:07<00:28,  4.06it/s, epoch=88, grad_norm=97.519, loss=284.0081, lr=9.62e-04] 
Epoch 88 TRAIN:  17%|█▋        | 23/137 [00:07<00:26,  4.25it/s, epoch=88, grad_norm=97.519, loss=284.0081, lr=9.62e-04]
Epoch 88 TRAIN:  17%|█▋        | 23/137 [00:07<00:26,  4.25it/s, epoch=88, grad_norm=99.924, loss=306.0247, lr=9.62e-04]
Epoch 88 TRAIN:  17%|█▋        | 23/137 [00:07<00:26,  4.25it/s, epoch=88, grad_norm=118.537, loss=292.8665, lr=9.62e-04]
Epoch 88 TRAIN:  17%|█▋        | 23/137 [00:08<00:26,  4.25it/s, epoch=88, grad_norm=82.965, loss=248.8300, lr=9.62e-04] 
Epoch 88 TRAIN:  19%|█▉        | 26/137 [00:08<00:25,  4.31it/s, epoch=88, grad_norm=82.965, loss=248.8300, lr=9.62e-04]
Epoch 88 TRAIN:  19%|█▉        | 26/137 [00:08<00:25,  4.31it/s, epoch=88, grad_norm=69.046, loss=290.3041, lr=9.62e-04]
Epoch 88 TRAIN:  19%|█▉        | 26/137 [00:08<00:25,  4.31it/s, epoch=88, grad_norm=109.676, loss=236.5587, lr=9.62e-04]
Epoch 88 TRAIN:  19%|█▉        | 26/137 [00:08<00:25,  4.31it/s, epoch=88, grad_norm=62.827, loss=240.4399, lr=9.62e-04] 
Epoch 88 TRAIN:  21%|██        | 29/137 [00:08<00:23,  4.53it/s, epoch=88, grad_norm=62.827, loss=240.4399, lr=9.62e-04]
Epoch 88 TRAIN:  21%|██        | 29/137 [00:08<00:23,  4.53it/s, epoch=88, grad_norm=90.533, loss=269.6244, lr=9.62e-04]
Epoch 88 TRAIN:  21%|██        | 29/137 [00:09<00:23,  4.53it/s, epoch=88, grad_norm=88.538, loss=222.2367, lr=9.63e-04]
Epoch 88 TRAIN:  21%|██        | 29/137 [00:09<00:23,  4.53it/s, epoch=88, grad_norm=73.396, loss=248.2793, lr=9.63e-04]
Epoch 88 TRAIN:  23%|██▎       | 32/137 [00:09<00:22,  4.72it/s, epoch=88, grad_norm=73.396, loss=248.2793, lr=9.63e-04]
Epoch 88 TRAIN:  23%|██▎       | 32/137 [00:09<00:22,  4.72it/s, epoch=88, grad_norm=77.574, loss=253.8549, lr=9.63e-04]
Epoch 88 TRAIN:  23%|██▎       | 32/137 [00:09<00:22,  4.72it/s, epoch=88, grad_norm=198.012, loss=315.3733, lr=9.63e-04]
Epoch 88 TRAIN:  23%|██▎       | 32/137 [00:09<00:22,  4.72it/s, epoch=88, grad_norm=122.647, loss=270.6055, lr=9.63e-04]
Epoch 88 TRAIN:  26%|██▌       | 35/137 [00:09<00:21,  4.75it/s, epoch=88, grad_norm=122.647, loss=270.6055, lr=9.63e-04]
Epoch 88 TRAIN:  26%|██▌       | 35/137 [00:10<00:21,  4.75it/s, epoch=88, grad_norm=70.856, loss=232.6140, lr=9.63e-04] 
Epoch 88 TRAIN:  26%|██▌       | 35/137 [00:10<00:21,  4.75it/s, epoch=88, grad_norm=101.569, loss=285.2791, lr=9.63e-04]
Epoch 88 TRAIN:  26%|██▌       | 35/137 [00:10<00:21,  4.75it/s, epoch=88, grad_norm=115.200, loss=251.8451, lr=9.63e-04]
Epoch 88 TRAIN:  28%|██▊       | 38/137 [00:10<00:21,  4.66it/s, epoch=88, grad_norm=115.200, loss=251.8451, lr=9.63e-04]
Epoch 88 TRAIN:  28%|██▊       | 38/137 [00:10<00:21,  4.66it/s, epoch=88, grad_norm=78.664, loss=272.7423, lr=9.63e-04] 
Epoch 88 TRAIN:  28%|██▊       | 38/137 [00:10<00:21,  4.66it/s, epoch=88, grad_norm=139.592, loss=319.7186, lr=9.63e-04]
Epoch 88 TRAIN:  28%|██▊       | 38/137 [00:11<00:21,  4.66it/s, epoch=88, grad_norm=90.891, loss=263.3652, lr=9.63e-04] 
Epoch 88 TRAIN:  30%|██▉       | 41/137 [00:11<00:20,  4.71it/s, epoch=88, grad_norm=90.891, loss=263.3652, lr=9.63e-04]
Epoch 88 TRAIN:  30%|██▉       | 41/137 [00:11<00:20,  4.71it/s, epoch=88, grad_norm=83.915, loss=255.4921, lr=9.63e-04]
Epoch 88 TRAIN:  30%|██▉       | 41/137 [00:11<00:20,  4.71it/s, epoch=88, grad_norm=90.653, loss=290.1201, lr=9.64e-04]
Epoch 88 TRAIN:  30%|██▉       | 41/137 [00:11<00:20,  4.71it/s, epoch=88, grad_norm=97.790, loss=278.6410, lr=9.64e-04]
Epoch 88 TRAIN:  32%|███▏      | 44/137 [00:11<00:19,  4.81it/s, epoch=88, grad_norm=97.790, loss=278.6410, lr=9.64e-04]
Epoch 88 TRAIN:  32%|███▏      | 44/137 [00:12<00:19,  4.81it/s, epoch=88, grad_norm=102.653, loss=251.6247, lr=9.64e-04]
Epoch 88 TRAIN:  32%|███▏      | 44/137 [00:12<00:19,  4.81it/s, epoch=88, grad_norm=95.913, loss=262.4071, lr=9.64e-04] 
Epoch 88 TRAIN:  32%|███▏      | 44/137 [00:12<00:19,  4.81it/s, epoch=88, grad_norm=119.566, loss=250.8499, lr=9.64e-04]
Epoch 88 TRAIN:  34%|███▍      | 47/137 [00:12<00:19,  4.72it/s, epoch=88, grad_norm=119.566, loss=250.8499, lr=9.64e-04]
Epoch 88 TRAIN:  34%|███▍      | 47/137 [00:12<00:19,  4.72it/s, epoch=88, grad_norm=116.996, loss=239.1384, lr=9.64e-04]
Epoch 88 TRAIN:  34%|███▍      | 47/137 [00:12<00:19,  4.72it/s, epoch=88, grad_norm=88.750, loss=278.9196, lr=9.64e-04] 
Epoch 88 TRAIN:  34%|███▍      | 47/137 [00:13<00:19,  4.72it/s, epoch=88, grad_norm=79.251, loss=242.6362, lr=9.64e-04]
Epoch 88 TRAIN:  36%|███▋      | 50/137 [00:13<00:19,  4.48it/s, epoch=88, grad_norm=79.251, loss=242.6362, lr=9.64e-04]
Epoch 88 TRAIN:  36%|███▋      | 50/137 [00:13<00:19,  4.48it/s, epoch=88, grad_norm=80.896, loss=238.2803, lr=9.64e-04]
Epoch 88 TRAIN:  36%|███▋      | 50/137 [00:13<00:19,  4.48it/s, epoch=88, grad_norm=94.739, loss=224.9860, lr=9.64e-04]
Epoch 88 TRAIN:  36%|███▋      | 50/137 [00:13<00:19,  4.48it/s, epoch=88, grad_norm=133.553, loss=265.6545, lr=9.64e-04]
Epoch 88 TRAIN:  39%|███▊      | 53/137 [00:13<00:18,  4.43it/s, epoch=88, grad_norm=133.553, loss=265.6545, lr=9.64e-04]
Epoch 88 TRAIN:  39%|███▊      | 53/137 [00:14<00:18,  4.43it/s, epoch=88, grad_norm=70.944, loss=246.6109, lr=9.64e-04] 
Epoch 88 TRAIN:  39%|███▊      | 53/137 [00:14<00:18,  4.43it/s, epoch=88, grad_norm=89.842, loss=237.5157, lr=9.64e-04]
Epoch 88 TRAIN:  39%|███▊      | 53/137 [00:14<00:18,  4.43it/s, epoch=88, grad_norm=177.897, loss=293.1843, lr=9.65e-04]
Epoch 88 TRAIN:  41%|████      | 56/137 [00:14<00:17,  4.64it/s, epoch=88, grad_norm=177.897, loss=293.1843, lr=9.65e-04]
Epoch 88 TRAIN:  41%|████      | 56/137 [00:14<00:17,  4.64it/s, epoch=88, grad_norm=179.982, loss=307.5428, lr=9.65e-04]
Epoch 88 TRAIN:  41%|████      | 56/137 [00:14<00:17,  4.64it/s, epoch=88, grad_norm=188.639, loss=301.7256, lr=9.65e-04]
Epoch 88 TRAIN:  41%|████      | 56/137 [00:15<00:17,  4.64it/s, epoch=88, grad_norm=135.096, loss=302.4912, lr=9.65e-04]
Epoch 88 TRAIN:  43%|████▎     | 59/137 [00:15<00:16,  4.76it/s, epoch=88, grad_norm=135.096, loss=302.4912, lr=9.65e-04]
Epoch 88 TRAIN:  43%|████▎     | 59/137 [00:15<00:16,  4.76it/s, epoch=88, grad_norm=89.401, loss=251.8050, lr=9.65e-04] 
Epoch 88 TRAIN:  43%|████▎     | 59/137 [00:15<00:16,  4.76it/s, epoch=88, grad_norm=87.338, loss=254.9518, lr=9.65e-04]
Epoch 88 TRAIN:  43%|████▎     | 59/137 [00:15<00:16,  4.76it/s, epoch=88, grad_norm=112.802, loss=241.7575, lr=9.65e-04]
Epoch 88 TRAIN:  45%|████▌     | 62/137 [00:15<00:15,  4.71it/s, epoch=88, grad_norm=112.802, loss=241.7575, lr=9.65e-04]
Epoch 88 TRAIN:  45%|████▌     | 62/137 [00:15<00:15,  4.71it/s, epoch=88, grad_norm=116.170, loss=269.1868, lr=9.65e-04]
Epoch 88 TRAIN:  45%|████▌     | 62/137 [00:16<00:15,  4.71it/s, epoch=88, grad_norm=122.342, loss=266.7496, lr=9.65e-04]
Epoch 88 TRAIN:  45%|████▌     | 62/137 [00:16<00:15,  4.71it/s, epoch=88, grad_norm=143.799, loss=265.7995, lr=9.65e-04]
Epoch 88 TRAIN:  47%|████▋     | 65/137 [00:16<00:15,  4.63it/s, epoch=88, grad_norm=143.799, loss=265.7995, lr=9.65e-04]
Epoch 88 TRAIN:  47%|████▋     | 65/137 [00:16<00:15,  4.63it/s, epoch=88, grad_norm=178.334, loss=303.0316, lr=9.65e-04]
Epoch 88 TRAIN:  47%|████▋     | 65/137 [00:16<00:15,  4.63it/s, epoch=88, grad_norm=106.590, loss=279.4148, lr=9.65e-04]
Epoch 88 TRAIN:  47%|████▋     | 65/137 [00:17<00:15,  4.63it/s, epoch=88, grad_norm=113.446, loss=307.4389, lr=9.66e-04]
Epoch 88 TRAIN:  50%|████▉     | 68/137 [00:17<00:15,  4.45it/s, epoch=88, grad_norm=113.446, loss=307.4389, lr=9.66e-04]
Epoch 88 TRAIN:  50%|████▉     | 68/137 [00:17<00:15,  4.45it/s, epoch=88, grad_norm=158.126, loss=289.5419, lr=9.66e-04]
Epoch 88 TRAIN:  50%|████▉     | 68/137 [00:17<00:15,  4.45it/s, epoch=88, grad_norm=102.278, loss=280.2656, lr=9.66e-04]
Epoch 88 TRAIN:  50%|████▉     | 68/137 [00:17<00:15,  4.45it/s, epoch=88, grad_norm=130.186, loss=290.1757, lr=9.66e-04]
Epoch 88 TRAIN:  52%|█████▏    | 71/137 [00:17<00:15,  4.40it/s, epoch=88, grad_norm=130.186, loss=290.1757, lr=9.66e-04]
Epoch 88 TRAIN:  52%|█████▏    | 71/137 [00:18<00:15,  4.40it/s, epoch=88, grad_norm=187.454, loss=223.1867, lr=9.66e-04]
Epoch 88 TRAIN:  52%|█████▏    | 71/137 [00:18<00:15,  4.40it/s, epoch=88, grad_norm=76.382, loss=286.0996, lr=9.66e-04] 
Epoch 88 TRAIN:  52%|█████▏    | 71/137 [00:18<00:15,  4.40it/s, epoch=88, grad_norm=119.575, loss=260.4745, lr=9.66e-04]
Epoch 88 TRAIN:  54%|█████▍    | 74/137 [00:18<00:14,  4.49it/s, epoch=88, grad_norm=119.575, loss=260.4745, lr=9.66e-04]
Epoch 88 TRAIN:  54%|█████▍    | 74/137 [00:18<00:14,  4.49it/s, epoch=88, grad_norm=182.489, loss=273.2219, lr=9.66e-04]
Epoch 88 TRAIN:  54%|█████▍    | 74/137 [00:18<00:14,  4.49it/s, epoch=88, grad_norm=140.006, loss=270.3753, lr=9.66e-04]
Epoch 88 TRAIN:  54%|█████▍    | 74/137 [00:19<00:14,  4.49it/s, epoch=88, grad_norm=143.135, loss=275.1360, lr=9.66e-04]
Epoch 88 TRAIN:  56%|█████▌    | 77/137 [00:19<00:14,  4.20it/s, epoch=88, grad_norm=143.135, loss=275.1360, lr=9.66e-04]
Epoch 88 TRAIN:  56%|█████▌    | 77/137 [00:19<00:14,  4.20it/s, epoch=88, grad_norm=141.896, loss=243.2057, lr=9.66e-04]
Epoch 88 TRAIN:  56%|█████▌    | 77/137 [00:19<00:14,  4.20it/s, epoch=88, grad_norm=99.061, loss=247.4767, lr=9.66e-04] 
Epoch 88 TRAIN:  56%|█████▌    | 77/137 [00:19<00:14,  4.20it/s, epoch=88, grad_norm=126.435, loss=262.3938, lr=9.66e-04]
Epoch 88 TRAIN:  58%|█████▊    | 80/137 [00:19<00:13,  4.36it/s, epoch=88, grad_norm=126.435, loss=262.3938, lr=9.66e-04]
Epoch 88 TRAIN:  58%|█████▊    | 80/137 [00:20<00:13,  4.36it/s, epoch=88, grad_norm=140.163, loss=279.0741, lr=9.67e-04]
Epoch 88 TRAIN:  58%|█████▊    | 80/137 [00:20<00:13,  4.36it/s, epoch=88, grad_norm=99.332, loss=243.1399, lr=9.67e-04] 
Epoch 88 TRAIN:  58%|█████▊    | 80/137 [00:20<00:13,  4.36it/s, epoch=88, grad_norm=139.514, loss=285.2440, lr=9.67e-04]
Epoch 88 TRAIN:  61%|██████    | 83/137 [00:20<00:12,  4.49it/s, epoch=88, grad_norm=139.514, loss=285.2440, lr=9.67e-04]
Epoch 88 TRAIN:  61%|██████    | 83/137 [00:20<00:12,  4.49it/s, epoch=88, grad_norm=135.110, loss=276.1884, lr=9.67e-04]
Epoch 88 TRAIN:  61%|██████    | 83/137 [00:21<00:12,  4.49it/s, epoch=88, grad_norm=167.701, loss=276.7032, lr=9.67e-04]
Epoch 88 TRAIN:  61%|██████    | 83/137 [00:21<00:12,  4.49it/s, epoch=88, grad_norm=111.806, loss=216.8264, lr=9.67e-04]
Epoch 88 TRAIN:  63%|██████▎   | 86/137 [00:21<00:11,  4.50it/s, epoch=88, grad_norm=111.806, loss=216.8264, lr=9.67e-04]
Epoch 88 TRAIN:  63%|██████▎   | 86/137 [00:21<00:11,  4.50it/s, epoch=88, grad_norm=114.407, loss=240.2314, lr=9.67e-04]
Epoch 88 TRAIN:  63%|██████▎   | 86/137 [00:21<00:11,  4.50it/s, epoch=88, grad_norm=230.031, loss=182.9091, lr=9.67e-04]
Epoch 88 TRAIN:  63%|██████▎   | 86/137 [00:21<00:11,  4.50it/s, epoch=88, grad_norm=108.497, loss=256.6390, lr=9.67e-04]
Epoch 88 TRAIN:  65%|██████▍   | 89/137 [00:21<00:11,  4.33it/s, epoch=88, grad_norm=108.497, loss=256.6390, lr=9.67e-04]
Epoch 88 TRAIN:  65%|██████▍   | 89/137 [00:22<00:11,  4.33it/s, epoch=88, grad_norm=90.140, loss=254.4682, lr=9.67e-04] 
Epoch 88 TRAIN:  65%|██████▍   | 89/137 [00:22<00:11,  4.33it/s, epoch=88, grad_norm=138.645, loss=273.1663, lr=9.67e-04]
Epoch 88 TRAIN:  65%|██████▍   | 89/137 [00:22<00:11,  4.33it/s, epoch=88, grad_norm=114.666, loss=218.1240, lr=9.67e-04]
Epoch 88 TRAIN:  67%|██████▋   | 92/137 [00:22<00:10,  4.25it/s, epoch=88, grad_norm=114.666, loss=218.1240, lr=9.67e-04]
Epoch 88 TRAIN:  67%|██████▋   | 92/137 [00:22<00:10,  4.25it/s, epoch=88, grad_norm=106.701, loss=276.5034, lr=9.68e-04]
Epoch 88 TRAIN:  67%|██████▋   | 92/137 [00:23<00:10,  4.25it/s, epoch=88, grad_norm=126.133, loss=287.8750, lr=9.68e-04]
Epoch 88 TRAIN:  67%|██████▋   | 92/137 [00:23<00:10,  4.25it/s, epoch=88, grad_norm=181.541, loss=179.3001, lr=9.68e-04]
Epoch 88 TRAIN:  69%|██████▉   | 95/137 [00:23<00:09,  4.43it/s, epoch=88, grad_norm=181.541, loss=179.3001, lr=9.68e-04]
Epoch 88 TRAIN:  69%|██████▉   | 95/137 [00:23<00:09,  4.43it/s, epoch=88, grad_norm=113.547, loss=239.5229, lr=9.68e-04]
Epoch 88 TRAIN:  69%|██████▉   | 95/137 [00:23<00:09,  4.43it/s, epoch=88, grad_norm=212.994, loss=315.9143, lr=9.68e-04]
Epoch 88 TRAIN:  69%|██████▉   | 95/137 [00:24<00:09,  4.43it/s, epoch=88, grad_norm=98.538, loss=250.1383, lr=9.68e-04] 
Epoch 88 TRAIN:  72%|███████▏  | 98/137 [00:24<00:09,  4.29it/s, epoch=88, grad_norm=98.538, loss=250.1383, lr=9.68e-04]
Epoch 88 TRAIN:  72%|███████▏  | 98/137 [00:24<00:09,  4.29it/s, epoch=88, grad_norm=111.824, loss=209.9583, lr=9.68e-04]
Epoch 88 TRAIN:  72%|███████▏  | 98/137 [00:24<00:09,  4.29it/s, epoch=88, grad_norm=119.625, loss=232.3850, lr=9.68e-04]2026-04-16 20:38:03,122 DEBUG TRAIN | steps/sec 4.131| Batch 88/100 loss 232.385025 loss_att 180.655045 loss_ctc 353.088318 th_accuracy 0.501279 lr 9.6808e-04 grad_norm 119.624527 rank 0

Epoch 88 TRAIN:  72%|███████▏  | 98/137 [00:24<00:09,  4.29it/s, epoch=88, grad_norm=84.379, loss=258.8647, lr=9.68e-04] 
Epoch 88 TRAIN:  74%|███████▎  | 101/137 [00:24<00:08,  4.43it/s, epoch=88, grad_norm=84.379, loss=258.8647, lr=9.68e-04]
Epoch 88 TRAIN:  74%|███████▎  | 101/137 [00:24<00:08,  4.43it/s, epoch=88, grad_norm=85.528, loss=195.5800, lr=9.68e-04]
Epoch 88 TRAIN:  74%|███████▎  | 101/137 [00:25<00:08,  4.43it/s, epoch=88, grad_norm=154.961, loss=252.2034, lr=9.68e-04]
Epoch 88 TRAIN:  74%|███████▎  | 101/137 [00:25<00:08,  4.43it/s, epoch=88, grad_norm=107.336, loss=267.7078, lr=9.68e-04]
Epoch 88 TRAIN:  76%|███████▌  | 104/137 [00:25<00:07,  4.41it/s, epoch=88, grad_norm=107.336, loss=267.7078, lr=9.68e-04]
Epoch 88 TRAIN:  76%|███████▌  | 104/137 [00:25<00:07,  4.41it/s, epoch=88, grad_norm=208.162, loss=285.6986, lr=9.68e-04]
Epoch 88 TRAIN:  76%|███████▌  | 104/137 [00:25<00:07,  4.41it/s, epoch=88, grad_norm=164.093, loss=248.2453, lr=9.69e-04]
Epoch 88 TRAIN:  76%|███████▌  | 104/137 [00:25<00:07,  4.41it/s, epoch=88, grad_norm=108.694, loss=248.7837, lr=9.69e-04]
Epoch 88 TRAIN:  78%|███████▊  | 107/137 [00:25<00:06,  4.64it/s, epoch=88, grad_norm=108.694, loss=248.7837, lr=9.69e-04]
Epoch 88 TRAIN:  78%|███████▊  | 107/137 [00:26<00:06,  4.64it/s, epoch=88, grad_norm=126.259, loss=206.0745, lr=9.69e-04]
Epoch 88 TRAIN:  78%|███████▊  | 107/137 [00:26<00:06,  4.64it/s, epoch=88, grad_norm=115.026, loss=249.7326, lr=9.69e-04]
Epoch 88 TRAIN:  78%|███████▊  | 107/137 [00:26<00:06,  4.64it/s, epoch=88, grad_norm=182.752, loss=280.7887, lr=9.69e-04]
Epoch 88 TRAIN:  80%|████████  | 110/137 [00:26<00:05,  4.62it/s, epoch=88, grad_norm=182.752, loss=280.7887, lr=9.69e-04]
Epoch 88 TRAIN:  80%|████████  | 110/137 [00:26<00:05,  4.62it/s, epoch=88, grad_norm=154.230, loss=211.0044, lr=9.69e-04]
Epoch 88 TRAIN:  80%|████████  | 110/137 [00:27<00:05,  4.62it/s, epoch=88, grad_norm=136.056, loss=236.0068, lr=9.69e-04]
Epoch 88 TRAIN:  80%|████████  | 110/137 [00:27<00:05,  4.62it/s, epoch=88, grad_norm=122.791, loss=244.6666, lr=9.69e-04]
Epoch 88 TRAIN:  82%|████████▏ | 113/137 [00:27<00:05,  4.62it/s, epoch=88, grad_norm=122.791, loss=244.6666, lr=9.69e-04]
Epoch 88 TRAIN:  82%|████████▏ | 113/137 [00:27<00:05,  4.62it/s, epoch=88, grad_norm=121.396, loss=277.9601, lr=9.69e-04]
Epoch 88 TRAIN:  82%|████████▏ | 113/137 [00:27<00:05,  4.62it/s, epoch=88, grad_norm=190.029, loss=226.3277, lr=9.69e-04]
Epoch 88 TRAIN:  82%|████████▏ | 113/137 [00:27<00:05,  4.62it/s, epoch=88, grad_norm=256.027, loss=298.6943, lr=9.69e-04]
Epoch 88 TRAIN:  85%|████████▍ | 116/137 [00:27<00:04,  4.44it/s, epoch=88, grad_norm=256.027, loss=298.6943, lr=9.69e-04]
Epoch 88 TRAIN:  85%|████████▍ | 116/137 [00:28<00:04,  4.44it/s, epoch=88, grad_norm=139.090, loss=249.6662, lr=9.69e-04]
Epoch 88 TRAIN:  85%|████████▍ | 116/137 [00:28<00:04,  4.44it/s, epoch=88, grad_norm=199.923, loss=265.9270, lr=9.70e-04]
Epoch 88 TRAIN:  85%|████████▍ | 116/137 [00:28<00:04,  4.44it/s, epoch=88, grad_norm=190.978, loss=288.3368, lr=9.70e-04]
Epoch 88 TRAIN:  87%|████████▋ | 119/137 [00:28<00:03,  4.58it/s, epoch=88, grad_norm=190.978, loss=288.3368, lr=9.70e-04]
Epoch 88 TRAIN:  87%|████████▋ | 119/137 [00:28<00:03,  4.58it/s, epoch=88, grad_norm=178.632, loss=277.8657, lr=9.70e-04]
Epoch 88 TRAIN:  87%|████████▋ | 119/137 [00:29<00:03,  4.58it/s, epoch=88, grad_norm=122.611, loss=274.2778, lr=9.70e-04]
Epoch 88 TRAIN:  87%|████████▋ | 119/137 [00:29<00:03,  4.58it/s, epoch=88, grad_norm=118.199, loss=247.2237, lr=9.70e-04]
Epoch 88 TRAIN:  89%|████████▉ | 122/137 [00:29<00:03,  4.36it/s, epoch=88, grad_norm=118.199, loss=247.2237, lr=9.70e-04]
Epoch 88 TRAIN:  89%|████████▉ | 122/137 [00:29<00:03,  4.36it/s, epoch=88, grad_norm=165.596, loss=260.4764, lr=9.70e-04]
Epoch 88 TRAIN:  89%|████████▉ | 122/137 [00:29<00:03,  4.36it/s, epoch=88, grad_norm=120.415, loss=291.8331, lr=9.70e-04]
Epoch 88 TRAIN:  89%|████████▉ | 122/137 [00:29<00:03,  4.36it/s, epoch=88, grad_norm=162.473, loss=226.9463, lr=9.70e-04]
Epoch 88 TRAIN:  91%|█████████ | 125/137 [00:29<00:02,  4.46it/s, epoch=88, grad_norm=162.473, loss=226.9463, lr=9.70e-04]
Epoch 88 TRAIN:  91%|█████████ | 125/137 [00:30<00:02,  4.46it/s, epoch=88, grad_norm=179.828, loss=248.8987, lr=9.70e-04]
Epoch 88 TRAIN:  91%|█████████ | 125/137 [00:30<00:02,  4.46it/s, epoch=88, grad_norm=152.595, loss=273.0069, lr=9.70e-04]
Epoch 88 TRAIN:  91%|█████████ | 125/137 [00:30<00:02,  4.46it/s, epoch=88, grad_norm=251.262, loss=317.2000, lr=9.70e-04]
Epoch 88 TRAIN:  93%|█████████▎| 128/137 [00:30<00:01,  4.72it/s, epoch=88, grad_norm=251.262, loss=317.2000, lr=9.70e-04]
Epoch 88 TRAIN:  93%|█████████▎| 128/137 [00:30<00:01,  4.72it/s, epoch=88, grad_norm=262.554, loss=364.8878, lr=9.70e-04]
Epoch 88 TRAIN:  93%|█████████▎| 128/137 [00:30<00:01,  4.72it/s, epoch=88, grad_norm=111.530, loss=224.2181, lr=9.70e-04]
Epoch 88 TRAIN:  93%|█████████▎| 128/137 [00:31<00:01,  4.72it/s, epoch=88, grad_norm=186.106, loss=311.5980, lr=9.71e-04]
Epoch 88 TRAIN:  96%|█████████▌| 131/137 [00:31<00:01,  4.72it/s, epoch=88, grad_norm=186.106, loss=311.5980, lr=9.71e-04]
Epoch 88 TRAIN:  96%|█████████▌| 131/137 [00:31<00:01,  4.72it/s, epoch=88, grad_norm=256.441, loss=332.8908, lr=9.71e-04]
Epoch 88 TRAIN:  96%|█████████▌| 131/137 [00:31<00:01,  4.72it/s, epoch=88, grad_norm=194.879, loss=256.8901, lr=9.71e-04]
Epoch 88 TRAIN:  96%|█████████▌| 131/137 [00:31<00:01,  4.72it/s, epoch=88, grad_norm=178.698, loss=262.9546, lr=9.71e-04]
Epoch 88 TRAIN:  98%|█████████▊| 134/137 [00:31<00:00,  4.79it/s, epoch=88, grad_norm=178.698, loss=262.9546, lr=9.71e-04]
Epoch 88 TRAIN:  98%|█████████▊| 134/137 [00:32<00:00,  4.79it/s, epoch=88, grad_norm=186.450, loss=305.7628, lr=9.71e-04]
Epoch 88 TRAIN:  98%|█████████▊| 134/137 [00:32<00:00,  4.79it/s, epoch=88, grad_norm=158.885, loss=284.4774, lr=9.71e-04]
Epoch 88 TRAIN:  98%|█████████▊| 134/137 [00:32<00:00,  4.79it/s, epoch=88, grad_norm=200.038, loss=332.6567, lr=9.71e-04]
Epoch 88 TRAIN: 100%|██████████| 137/137 [00:32<00:00,  4.57it/s, epoch=88, grad_norm=200.038, loss=332.6567, lr=9.71e-04]
Epoch 88 TRAIN: 100%|██████████| 137/137 [00:32<00:00,  4.57it/s, epoch=88, grad_norm=308.476, loss=304.2356, lr=9.71e-04]
Epoch 88 TRAIN: 138it [00:32,  4.23it/s, epoch=88, grad_norm=308.476, loss=304.2356, lr=9.71e-04]                         

CV:   0%|          | 0/6 [00:00<?, ?it/s]/home/rookie/Documents/batuka/projects/wenet/wenet/wenet/utils/train_utils.py:689: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  amp_autocast(enabled=dtype is not None,

CV:  17%|█▋        | 1/6 [00:00<00:03,  1.30it/s]
                                                 
2026-04-16 20:38:12,128 INFO Epoch 88 Step 12139 CV info lr 9.7112e-04 cv_loss 481.3840054598722 rank 0 acc 0.3935262809197108
2026-04-16 20:38:12,130 INFO [Rank 0] Checkpoint: save to checkpoint exp/conformer_mongolian/epoch_88.pt
2026-04-16 20:38:12,244 INFO Epoch 89 Step 12139 TRAIN info lr 9.7112e-04 rank 0
2026-04-16 20:38:12,259 INFO using accumulate grad, new batch size is 1 times larger than before

Epoch 89 TRAIN:   0%|          | 0/138 [00:00<?, ?it/s]
Epoch 89 TRAIN:   0%|          | 0/138 [00:04<?, ?it/s, epoch=89, grad_norm=600.459, loss=454.1286, lr=9.71e-04]
Epoch 89 TRAIN:   1%|          | 1/138 [00:04<10:19,  4.52s/it, epoch=89, grad_norm=600.459, loss=454.1286, lr=9.71e-04]
Epoch 89 TRAIN:   1%|          | 1/138 [00:04<10:19,  4.52s/it, epoch=89, grad_norm=479.351, loss=358.9163, lr=9.71e-04]
Epoch 89 TRAIN:   1%|          | 1/138 [00:04<10:19,  4.52s/it, epoch=89, grad_norm=289.170, loss=259.1963, lr=9.71e-04]
Epoch 89 TRAIN:   1%|          | 1/138 [00:04<10:19,  4.52s/it, epoch=89, grad_norm=202.118, loss=183.6551, lr=9.71e-04]
Epoch 89 TRAIN:   1%|          | 1/138 [00:04<10:19,  4.52s/it, epoch=89, grad_norm=174.533, loss=207.9102, lr=9.72e-04]
Epoch 89 TRAIN:   1%|          | 1/138 [00:04<10:19,  4.52s/it, epoch=89, grad_norm=269.279, loss=299.2027, lr=9.72e-04]
Epoch 89 TRAIN:   1%|          | 1/138 [00:04<10:19,  4.52s/it, epoch=89, grad_norm=263.815, loss=307.3747, lr=9.72e-04]
Epoch 89 TRAIN:   1%|          | 1/138 [00:05<10:19,  4.52s/it, epoch=89, grad_norm=390.244, loss=314.7066, lr=9.72e-04]
Epoch 89 TRAIN:   1%|          | 1/138 [00:05<10:19,  4.52s/it, epoch=89, grad_norm=226.989, loss=308.5720, lr=9.72e-04]
Epoch 89 TRAIN:   7%|▋         | 9/138 [00:05<00:56,  2.29it/s, epoch=89, grad_norm=226.989, loss=308.5720, lr=9.72e-04]
Epoch 89 TRAIN:   7%|▋         | 9/138 [00:05<00:56,  2.29it/s, epoch=89, grad_norm=175.870, loss=258.9709, lr=9.72e-04]
Epoch 89 TRAIN:   7%|▋         | 9/138 [00:05<00:56,  2.29it/s, epoch=89, grad_norm=124.803, loss=284.5947, lr=9.72e-04]
Epoch 89 TRAIN:   7%|▋         | 9/138 [00:05<00:56,  2.29it/s, epoch=89, grad_norm=92.836, loss=245.9565, lr=9.72e-04] 
Epoch 89 TRAIN:   7%|▋         | 9/138 [00:05<00:56,  2.29it/s, epoch=89, grad_norm=133.851, loss=276.9705, lr=9.72e-04]
Epoch 89 TRAIN:   9%|▉         | 13/138 [00:05<00:41,  3.05it/s, epoch=89, grad_norm=133.851, loss=276.9705, lr=9.72e-04]
Epoch 89 TRAIN:   9%|▉         | 13/138 [00:05<00:41,  3.05it/s, epoch=89, grad_norm=115.495, loss=269.6299, lr=9.72e-04]
Epoch 89 TRAIN:   9%|▉         | 13/138 [00:06<00:41,  3.05it/s, epoch=89, grad_norm=91.617, loss=295.8552, lr=9.72e-04] 
Epoch 89 TRAIN:   9%|▉         | 13/138 [00:06<00:41,  3.05it/s, epoch=89, grad_norm=115.352, loss=293.4769, lr=9.72e-04]
Epoch 89 TRAIN:   9%|▉         | 13/138 [00:06<00:41,  3.05it/s, epoch=89, grad_norm=110.247, loss=293.1704, lr=9.72e-04]
Epoch 89 TRAIN:  12%|█▏        | 17/138 [00:06<00:32,  3.68it/s, epoch=89, grad_norm=110.247, loss=293.1704, lr=9.72e-04]
Epoch 89 TRAIN:  12%|█▏        | 17/138 [00:06<00:32,  3.68it/s, epoch=89, grad_norm=119.103, loss=308.5505, lr=9.73e-04]
Epoch 89 TRAIN:  12%|█▏        | 17/138 [00:06<00:32,  3.68it/s, epoch=89, grad_norm=98.541, loss=258.0673, lr=9.73e-04] 
Epoch 89 TRAIN:  12%|█▏        | 17/138 [00:07<00:32,  3.68it/s, epoch=89, grad_norm=153.097, loss=241.9532, lr=9.73e-04]
Epoch 89 TRAIN:  12%|█▏        | 17/138 [00:07<00:32,  3.68it/s, epoch=89, grad_norm=107.080, loss=266.5430, lr=9.73e-04]
Epoch 89 TRAIN:  15%|█▌        | 21/138 [00:07<00:29,  3.91it/s, epoch=89, grad_norm=107.080, loss=266.5430, lr=9.73e-04]
Epoch 89 TRAIN:  15%|█▌        | 21/138 [00:07<00:29,  3.91it/s, epoch=89, grad_norm=108.744, loss=252.8956, lr=9.73e-04]
Epoch 89 TRAIN:  15%|█▌        | 21/138 [00:07<00:29,  3.91it/s, epoch=89, grad_norm=74.080, loss=245.7499, lr=9.73e-04] 
Epoch 89 TRAIN:  15%|█▌        | 21/138 [00:07<00:29,  3.91it/s, epoch=89, grad_norm=98.360, loss=280.3782, lr=9.73e-04]
Epoch 89 TRAIN:  17%|█▋        | 24/138 [00:07<00:27,  4.17it/s, epoch=89, grad_norm=98.360, loss=280.3782, lr=9.73e-04]
Epoch 89 TRAIN:  17%|█▋        | 24/138 [00:08<00:27,  4.17it/s, epoch=89, grad_norm=101.236, loss=260.2391, lr=9.73e-04]
Epoch 89 TRAIN:  17%|█▋        | 24/138 [00:08<00:27,  4.17it/s, epoch=89, grad_norm=87.441, loss=204.4085, lr=9.73e-04] 
Epoch 89 TRAIN:  17%|█▋        | 24/138 [00:08<00:27,  4.17it/s, epoch=89, grad_norm=121.644, loss=266.1526, lr=9.73e-04]
Epoch 89 TRAIN:  20%|█▉        | 27/138 [00:08<00:25,  4.41it/s, epoch=89, grad_norm=121.644, loss=266.1526, lr=9.73e-04]
Epoch 89 TRAIN:  20%|█▉        | 27/138 [00:08<00:25,  4.41it/s, epoch=89, grad_norm=77.657, loss=235.5177, lr=9.73e-04] 
Epoch 89 TRAIN:  20%|█▉        | 27/138 [00:09<00:25,  4.41it/s, epoch=89, grad_norm=115.617, loss=279.1037, lr=9.73e-04]
Epoch 89 TRAIN:  20%|█▉        | 27/138 [00:09<00:25,  4.41it/s, epoch=89, grad_norm=149.018, loss=288.6705, lr=9.74e-04]
Epoch 89 TRAIN:  22%|██▏       | 30/138 [00:09<00:24,  4.42it/s, epoch=89, grad_norm=149.018, loss=288.6705, lr=9.74e-04]
Epoch 89 TRAIN:  22%|██▏       | 30/138 [00:09<00:24,  4.42it/s, epoch=89, grad_norm=171.128, loss=287.8961, lr=9.74e-04]
Epoch 89 TRAIN:  22%|██▏       | 30/138 [00:09<00:24,  4.42it/s, epoch=89, grad_norm=78.053, loss=263.9124, lr=9.74e-04] 
Epoch 89 TRAIN:  22%|██▏       | 30/138 [00:09<00:24,  4.42it/s, epoch=89, grad_norm=95.866, loss=270.0815, lr=9.74e-04]
Epoch 89 TRAIN:  24%|██▍       | 33/138 [00:09<00:23,  4.54it/s, epoch=89, grad_norm=95.866, loss=270.0815, lr=9.74e-04]
Epoch 89 TRAIN:  24%|██▍       | 33/138 [00:10<00:23,  4.54it/s, epoch=89, grad_norm=108.224, loss=246.7910, lr=9.74e-04]
Epoch 89 TRAIN:  24%|██▍       | 33/138 [00:10<00:23,  4.54it/s, epoch=89, grad_norm=100.586, loss=300.9285, lr=9.74e-04]
Epoch 89 TRAIN:  24%|██▍       | 33/138 [00:10<00:23,  4.54it/s, epoch=89, grad_norm=85.972, loss=281.1050, lr=9.74e-04] 
Epoch 89 TRAIN:  26%|██▌       | 36/138 [00:10<00:22,  4.48it/s, epoch=89, grad_norm=85.972, loss=281.1050, lr=9.74e-04]
Epoch 89 TRAIN:  26%|██▌       | 36/138 [00:10<00:22,  4.48it/s, epoch=89, grad_norm=116.235, loss=254.2311, lr=9.74e-04]
Epoch 89 TRAIN:  26%|██▌       | 36/138 [00:10<00:22,  4.48it/s, epoch=89, grad_norm=98.743, loss=264.4350, lr=9.74e-04] 
Epoch 89 TRAIN:  26%|██▌       | 36/138 [00:11<00:22,  4.48it/s, epoch=89, grad_norm=160.493, loss=298.4191, lr=9.74e-04]
Epoch 89 TRAIN:  28%|██▊       | 39/138 [00:11<00:21,  4.57it/s, epoch=89, grad_norm=160.493, loss=298.4191, lr=9.74e-04]
Epoch 89 TRAIN:  28%|██▊       | 39/138 [00:11<00:21,  4.57it/s, epoch=89, grad_norm=100.998, loss=300.8162, lr=9.74e-04]
Epoch 89 TRAIN:  28%|██▊       | 39/138 [00:11<00:21,  4.57it/s, epoch=89, grad_norm=93.470, loss=298.4429, lr=9.74e-04] 
Epoch 89 TRAIN:  28%|██▊       | 39/138 [00:11<00:21,  4.57it/s, epoch=89, grad_norm=92.528, loss=297.2376, lr=9.74e-04]
Epoch 89 TRAIN:  30%|███       | 42/138 [00:11<00:20,  4.67it/s, epoch=89, grad_norm=92.528, loss=297.2376, lr=9.74e-04]
Epoch 89 TRAIN:  30%|███       | 42/138 [00:12<00:20,  4.67it/s, epoch=89, grad_norm=97.880, loss=265.7560, lr=9.75e-04]
Epoch 89 TRAIN:  30%|███       | 42/138 [00:12<00:20,  4.67it/s, epoch=89, grad_norm=102.181, loss=255.6848, lr=9.75e-04]
Epoch 89 TRAIN:  30%|███       | 42/138 [00:12<00:20,  4.67it/s, epoch=89, grad_norm=146.987, loss=274.6089, lr=9.75e-04]
Epoch 89 TRAIN:  33%|███▎      | 45/138 [00:12<00:20,  4.54it/s, epoch=89, grad_norm=146.987, loss=274.6089, lr=9.75e-04]
Epoch 89 TRAIN:  33%|███▎      | 45/138 [00:12<00:20,  4.54it/s, epoch=89, grad_norm=167.756, loss=274.8300, lr=9.75e-04]
Epoch 89 TRAIN:  33%|███▎      | 45/138 [00:12<00:20,  4.54it/s, epoch=89, grad_norm=95.361, loss=288.1124, lr=9.75e-04] 
Epoch 89 TRAIN:  33%|███▎      | 45/138 [00:13<00:20,  4.54it/s, epoch=89, grad_norm=105.331, loss=256.4653, lr=9.75e-04]
Epoch 89 TRAIN:  35%|███▍      | 48/138 [00:13<00:19,  4.69it/s, epoch=89, grad_norm=105.331, loss=256.4653, lr=9.75e-04]
Epoch 89 TRAIN:  35%|███▍      | 48/138 [00:13<00:19,  4.69it/s, epoch=89, grad_norm=159.286, loss=266.2224, lr=9.75e-04]
Epoch 89 TRAIN:  35%|███▍      | 48/138 [00:13<00:19,  4.69it/s, epoch=89, grad_norm=101.072, loss=299.8822, lr=9.75e-04]
Epoch 89 TRAIN:  35%|███▍      | 48/138 [00:13<00:19,  4.69it/s, epoch=89, grad_norm=171.639, loss=307.2460, lr=9.75e-04]
Epoch 89 TRAIN:  37%|███▋      | 51/138 [00:13<00:19,  4.48it/s, epoch=89, grad_norm=171.639, loss=307.2460, lr=9.75e-04]
Epoch 89 TRAIN:  37%|███▋      | 51/138 [00:14<00:19,  4.48it/s, epoch=89, grad_norm=139.552, loss=239.4148, lr=9.75e-04]
Epoch 89 TRAIN:  37%|███▋      | 51/138 [00:14<00:19,  4.48it/s, epoch=89, grad_norm=81.419, loss=267.6855, lr=9.75e-04] 
Epoch 89 TRAIN:  37%|███▋      | 51/138 [00:14<00:19,  4.48it/s, epoch=89, grad_norm=87.681, loss=309.7287, lr=9.75e-04]
Epoch 89 TRAIN:  39%|███▉      | 54/138 [00:14<00:18,  4.62it/s, epoch=89, grad_norm=87.681, loss=309.7287, lr=9.75e-04]
Epoch 89 TRAIN:  39%|███▉      | 54/138 [00:14<00:18,  4.62it/s, epoch=89, grad_norm=130.678, loss=306.3486, lr=9.76e-04]
Epoch 89 TRAIN:  39%|███▉      | 54/138 [00:14<00:18,  4.62it/s, epoch=89, grad_norm=124.321, loss=257.4344, lr=9.76e-04]
Epoch 89 TRAIN:  39%|███▉      | 54/138 [00:15<00:18,  4.62it/s, epoch=89, grad_norm=92.724, loss=288.7947, lr=9.76e-04] 
Epoch 89 TRAIN:  41%|████▏     | 57/138 [00:15<00:17,  4.63it/s, epoch=89, grad_norm=92.724, loss=288.7947, lr=9.76e-04]
Epoch 89 TRAIN:  41%|████▏     | 57/138 [00:15<00:17,  4.63it/s, epoch=89, grad_norm=115.429, loss=274.3694, lr=9.76e-04]
Epoch 89 TRAIN:  41%|████▏     | 57/138 [00:15<00:17,  4.63it/s, epoch=89, grad_norm=133.900, loss=236.6108, lr=9.76e-04]
Epoch 89 TRAIN:  41%|████▏     | 57/138 [00:15<00:17,  4.63it/s, epoch=89, grad_norm=140.002, loss=287.1446, lr=9.76e-04]
Epoch 89 TRAIN:  43%|████▎     | 60/138 [00:15<00:17,  4.59it/s, epoch=89, grad_norm=140.002, loss=287.1446, lr=9.76e-04]
Epoch 89 TRAIN:  43%|████▎     | 60/138 [00:15<00:17,  4.59it/s, epoch=89, grad_norm=129.339, loss=243.1909, lr=9.76e-04]
Epoch 89 TRAIN:  43%|████▎     | 60/138 [00:16<00:17,  4.59it/s, epoch=89, grad_norm=84.774, loss=292.1794, lr=9.76e-04] 
Epoch 89 TRAIN:  43%|████▎     | 60/138 [00:16<00:17,  4.59it/s, epoch=89, grad_norm=121.854, loss=253.8409, lr=9.76e-04]
Epoch 89 TRAIN:  46%|████▌     | 63/138 [00:16<00:17,  4.41it/s, epoch=89, grad_norm=121.854, loss=253.8409, lr=9.76e-04]
Epoch 89 TRAIN:  46%|████▌     | 63/138 [00:16<00:17,  4.41it/s, epoch=89, grad_norm=101.196, loss=220.9849, lr=9.76e-04]
Epoch 89 TRAIN:  46%|████▌     | 63/138 [00:16<00:17,  4.41it/s, epoch=89, grad_norm=136.903, loss=281.5300, lr=9.76e-04]
Epoch 89 TRAIN:  46%|████▌     | 63/138 [00:17<00:17,  4.41it/s, epoch=89, grad_norm=126.419, loss=221.1543, lr=9.76e-04]
Epoch 89 TRAIN:  48%|████▊     | 66/138 [00:17<00:16,  4.44it/s, epoch=89, grad_norm=126.419, loss=221.1543, lr=9.76e-04]
Epoch 89 TRAIN:  48%|████▊     | 66/138 [00:17<00:16,  4.44it/s, epoch=89, grad_norm=92.154, loss=257.6544, lr=9.76e-04] 
Epoch 89 TRAIN:  48%|████▊     | 66/138 [00:17<00:16,  4.44it/s, epoch=89, grad_norm=188.179, loss=311.4058, lr=9.77e-04]
Epoch 89 TRAIN:  48%|████▊     | 66/138 [00:17<00:16,  4.44it/s, epoch=89, grad_norm=82.259, loss=254.3746, lr=9.77e-04] 
Epoch 89 TRAIN:  50%|█████     | 69/138 [00:17<00:14,  4.65it/s, epoch=89, grad_norm=82.259, loss=254.3746, lr=9.77e-04]
Epoch 89 TRAIN:  50%|█████     | 69/138 [00:17<00:14,  4.65it/s, epoch=89, grad_norm=90.747, loss=267.8871, lr=9.77e-04]
Epoch 89 TRAIN:  50%|█████     | 69/138 [00:18<00:14,  4.65it/s, epoch=89, grad_norm=174.203, loss=272.3410, lr=9.77e-04]
Epoch 89 TRAIN:  50%|█████     | 69/138 [00:18<00:14,  4.65it/s, epoch=89, grad_norm=99.384, loss=202.7040, lr=9.77e-04] 
Epoch 89 TRAIN:  52%|█████▏    | 72/138 [00:18<00:14,  4.65it/s, epoch=89, grad_norm=99.384, loss=202.7040, lr=9.77e-04]
Epoch 89 TRAIN:  52%|█████▏    | 72/138 [00:18<00:14,  4.65it/s, epoch=89, grad_norm=94.000, loss=248.4094, lr=9.77e-04]
Epoch 89 TRAIN:  52%|█████▏    | 72/138 [00:18<00:14,  4.65it/s, epoch=89, grad_norm=120.778, loss=243.8418, lr=9.77e-04]
Epoch 89 TRAIN:  52%|█████▏    | 72/138 [00:19<00:14,  4.65it/s, epoch=89, grad_norm=131.729, loss=296.8608, lr=9.77e-04]
Epoch 89 TRAIN:  54%|█████▍    | 75/138 [00:19<00:14,  4.47it/s, epoch=89, grad_norm=131.729, loss=296.8608, lr=9.77e-04]
Epoch 89 TRAIN:  54%|█████▍    | 75/138 [00:19<00:14,  4.47it/s, epoch=89, grad_norm=84.753, loss=250.9575, lr=9.77e-04] 
Epoch 89 TRAIN:  54%|█████▍    | 75/138 [00:19<00:14,  4.47it/s, epoch=89, grad_norm=132.352, loss=301.5059, lr=9.77e-04]
Epoch 89 TRAIN:  54%|█████▍    | 75/138 [00:19<00:14,  4.47it/s, epoch=89, grad_norm=158.546, loss=312.8846, lr=9.77e-04]
Epoch 89 TRAIN:  57%|█████▋    | 78/138 [00:19<00:13,  4.46it/s, epoch=89, grad_norm=158.546, loss=312.8846, lr=9.77e-04]
Epoch 89 TRAIN:  57%|█████▋    | 78/138 [00:19<00:13,  4.46it/s, epoch=89, grad_norm=110.086, loss=267.0172, lr=9.77e-04]
Epoch 89 TRAIN:  57%|█████▋    | 78/138 [00:20<00:13,  4.46it/s, epoch=89, grad_norm=99.401, loss=262.7399, lr=9.78e-04] 
Epoch 89 TRAIN:  57%|█████▋    | 78/138 [00:20<00:13,  4.46it/s, epoch=89, grad_norm=83.129, loss=249.2162, lr=9.78e-04]
Epoch 89 TRAIN:  59%|█████▊    | 81/138 [00:20<00:12,  4.53it/s, epoch=89, grad_norm=83.129, loss=249.2162, lr=9.78e-04]
Epoch 89 TRAIN:  59%|█████▊    | 81/138 [00:20<00:12,  4.53it/s, epoch=89, grad_norm=204.646, loss=234.1588, lr=9.78e-04]
Epoch 89 TRAIN:  59%|█████▊    | 81/138 [00:20<00:12,  4.53it/s, epoch=89, grad_norm=119.683, loss=285.5236, lr=9.78e-04]
Epoch 89 TRAIN:  59%|█████▊    | 81/138 [00:21<00:12,  4.53it/s, epoch=89, grad_norm=100.482, loss=270.1187, lr=9.78e-04]
Epoch 89 TRAIN:  61%|██████    | 84/138 [00:21<00:12,  4.49it/s, epoch=89, grad_norm=100.482, loss=270.1187, lr=9.78e-04]
Epoch 89 TRAIN:  61%|██████    | 84/138 [00:21<00:12,  4.49it/s, epoch=89, grad_norm=118.335, loss=290.3309, lr=9.78e-04]
Epoch 89 TRAIN:  61%|██████    | 84/138 [00:21<00:12,  4.49it/s, epoch=89, grad_norm=118.338, loss=259.0378, lr=9.78e-04]
Epoch 89 TRAIN:  61%|██████    | 84/138 [00:21<00:12,  4.49it/s, epoch=89, grad_norm=119.492, loss=251.8775, lr=9.78e-04]
Epoch 89 TRAIN:  63%|██████▎   | 87/138 [00:21<00:11,  4.39it/s, epoch=89, grad_norm=119.492, loss=251.8775, lr=9.78e-04]
Epoch 89 TRAIN:  63%|██████▎   | 87/138 [00:22<00:11,  4.39it/s, epoch=89, grad_norm=152.779, loss=297.7253, lr=9.78e-04]
Epoch 89 TRAIN:  63%|██████▎   | 87/138 [00:22<00:11,  4.39it/s, epoch=89, grad_norm=158.846, loss=249.2903, lr=9.78e-04]
Epoch 89 TRAIN:  63%|██████▎   | 87/138 [00:22<00:11,  4.39it/s, epoch=89, grad_norm=217.619, loss=235.3392, lr=9.78e-04]
Epoch 89 TRAIN:  65%|██████▌   | 90/138 [00:22<00:10,  4.50it/s, epoch=89, grad_norm=217.619, loss=235.3392, lr=9.78e-04]
Epoch 89 TRAIN:  65%|██████▌   | 90/138 [00:22<00:10,  4.50it/s, epoch=89, grad_norm=152.977, loss=180.7026, lr=9.78e-04]
Epoch 89 TRAIN:  65%|██████▌   | 90/138 [00:22<00:10,  4.50it/s, epoch=89, grad_norm=122.109, loss=207.7011, lr=9.78e-04]
Epoch 89 TRAIN:  65%|██████▌   | 90/138 [00:23<00:10,  4.50it/s, epoch=89, grad_norm=142.356, loss=294.5838, lr=9.79e-04]
Epoch 89 TRAIN:  67%|██████▋   | 93/138 [00:23<00:09,  4.62it/s, epoch=89, grad_norm=142.356, loss=294.5838, lr=9.79e-04]
Epoch 89 TRAIN:  67%|██████▋   | 93/138 [00:23<00:09,  4.62it/s, epoch=89, grad_norm=148.452, loss=293.3728, lr=9.79e-04]
Epoch 89 TRAIN:  67%|██████▋   | 93/138 [00:23<00:09,  4.62it/s, epoch=89, grad_norm=110.883, loss=212.8519, lr=9.79e-04]
Epoch 89 TRAIN:  67%|██████▋   | 93/138 [00:23<00:09,  4.62it/s, epoch=89, grad_norm=107.134, loss=233.2033, lr=9.79e-04]
Epoch 89 TRAIN:  70%|██████▉   | 96/138 [00:23<00:09,  4.26it/s, epoch=89, grad_norm=107.134, loss=233.2033, lr=9.79e-04]
Epoch 89 TRAIN:  70%|██████▉   | 96/138 [00:24<00:09,  4.26it/s, epoch=89, grad_norm=233.565, loss=198.7304, lr=9.79e-04]
Epoch 89 TRAIN:  70%|██████▉   | 96/138 [00:24<00:09,  4.26it/s, epoch=89, grad_norm=150.684, loss=286.7747, lr=9.79e-04]
Epoch 89 TRAIN:  70%|██████▉   | 96/138 [00:24<00:09,  4.26it/s, epoch=89, grad_norm=202.683, loss=275.0324, lr=9.79e-04]
Epoch 89 TRAIN:  72%|███████▏  | 99/138 [00:24<00:08,  4.60it/s, epoch=89, grad_norm=202.683, loss=275.0324, lr=9.79e-04]
Epoch 89 TRAIN:  72%|███████▏  | 99/138 [00:24<00:08,  4.60it/s, epoch=89, grad_norm=129.988, loss=212.4022, lr=9.79e-04]2026-04-16 20:38:36,977 DEBUG TRAIN | steps/sec 4.076| Batch 89/100 loss 212.402161 loss_att 144.825562 loss_ctc 370.080872 th_accuracy 0.595331 lr 9.7912e-04 grad_norm 129.987686 rank 0

Epoch 89 TRAIN:  72%|███████▏  | 99/138 [00:24<00:08,  4.60it/s, epoch=89, grad_norm=227.785, loss=174.9469, lr=9.79e-04]
Epoch 89 TRAIN:  72%|███████▏  | 99/138 [00:25<00:08,  4.60it/s, epoch=89, grad_norm=138.677, loss=228.9236, lr=9.79e-04]
Epoch 89 TRAIN:  74%|███████▍  | 102/138 [00:25<00:08,  4.38it/s, epoch=89, grad_norm=138.677, loss=228.9236, lr=9.79e-04]
Epoch 89 TRAIN:  74%|███████▍  | 102/138 [00:25<00:08,  4.38it/s, epoch=89, grad_norm=176.574, loss=247.5974, lr=9.79e-04]
Epoch 89 TRAIN:  74%|███████▍  | 102/138 [00:25<00:08,  4.38it/s, epoch=89, grad_norm=237.335, loss=304.6451, lr=9.79e-04]
Epoch 89 TRAIN:  74%|███████▍  | 102/138 [00:25<00:08,  4.38it/s, epoch=89, grad_norm=149.324, loss=241.6854, lr=9.80e-04]
Epoch 89 TRAIN:  76%|███████▌  | 105/138 [00:25<00:07,  4.46it/s, epoch=89, grad_norm=149.324, loss=241.6854, lr=9.80e-04]
Epoch 89 TRAIN:  76%|███████▌  | 105/138 [00:25<00:07,  4.46it/s, epoch=89, grad_norm=151.085, loss=278.6208, lr=9.80e-04]
Epoch 89 TRAIN:  76%|███████▌  | 105/138 [00:26<00:07,  4.46it/s, epoch=89, grad_norm=140.164, loss=280.6833, lr=9.80e-04]
Epoch 89 TRAIN:  76%|███████▌  | 105/138 [00:26<00:07,  4.46it/s, epoch=89, grad_norm=120.272, loss=223.1516, lr=9.80e-04]
Epoch 89 TRAIN:  78%|███████▊  | 108/138 [00:26<00:06,  4.63it/s, epoch=89, grad_norm=120.272, loss=223.1516, lr=9.80e-04]
Epoch 89 TRAIN:  78%|███████▊  | 108/138 [00:26<00:06,  4.63it/s, epoch=89, grad_norm=153.475, loss=304.5956, lr=9.80e-04]
Epoch 89 TRAIN:  78%|███████▊  | 108/138 [00:26<00:06,  4.63it/s, epoch=89, grad_norm=115.181, loss=258.5400, lr=9.80e-04]
Epoch 89 TRAIN:  78%|███████▊  | 108/138 [00:27<00:06,  4.63it/s, epoch=89, grad_norm=172.447, loss=223.9788, lr=9.80e-04]
Epoch 89 TRAIN:  80%|████████  | 111/138 [00:27<00:05,  4.63it/s, epoch=89, grad_norm=172.447, loss=223.9788, lr=9.80e-04]
Epoch 89 TRAIN:  80%|████████  | 111/138 [00:27<00:05,  4.63it/s, epoch=89, grad_norm=292.646, loss=296.4041, lr=9.80e-04]
Epoch 89 TRAIN:  80%|████████  | 111/138 [00:27<00:05,  4.63it/s, epoch=89, grad_norm=108.631, loss=194.5036, lr=9.80e-04]
Epoch 89 TRAIN:  80%|████████  | 111/138 [00:27<00:05,  4.63it/s, epoch=89, grad_norm=227.636, loss=287.3552, lr=9.80e-04]
Epoch 89 TRAIN:  83%|████████▎ | 114/138 [00:27<00:05,  4.44it/s, epoch=89, grad_norm=227.636, loss=287.3552, lr=9.80e-04]
Epoch 89 TRAIN:  83%|████████▎ | 114/138 [00:28<00:05,  4.44it/s, epoch=89, grad_norm=151.909, loss=244.4655, lr=9.80e-04]
Epoch 89 TRAIN:  83%|████████▎ | 114/138 [00:28<00:05,  4.44it/s, epoch=89, grad_norm=199.568, loss=199.3611, lr=9.80e-04]
Epoch 89 TRAIN:  83%|████████▎ | 114/138 [00:28<00:05,  4.44it/s, epoch=89, grad_norm=111.109, loss=228.4400, lr=9.80e-04]
Epoch 89 TRAIN:  85%|████████▍ | 117/138 [00:28<00:04,  4.22it/s, epoch=89, grad_norm=111.109, loss=228.4400, lr=9.80e-04]
Epoch 89 TRAIN:  85%|████████▍ | 117/138 [00:28<00:04,  4.22it/s, epoch=89, grad_norm=183.855, loss=295.1860, lr=9.81e-04]
Epoch 89 TRAIN:  85%|████████▍ | 117/138 [00:29<00:04,  4.22it/s, epoch=89, grad_norm=95.753, loss=207.1790, lr=9.81e-04] 
Epoch 89 TRAIN:  85%|████████▍ | 117/138 [00:29<00:04,  4.22it/s, epoch=89, grad_norm=103.433, loss=204.1792, lr=9.81e-04]
Epoch 89 TRAIN:  87%|████████▋ | 120/138 [00:29<00:04,  4.37it/s, epoch=89, grad_norm=103.433, loss=204.1792, lr=9.81e-04]
Epoch 89 TRAIN:  87%|████████▋ | 120/138 [00:29<00:04,  4.37it/s, epoch=89, grad_norm=192.064, loss=306.9403, lr=9.81e-04]
Epoch 89 TRAIN:  87%|████████▋ | 120/138 [00:29<00:04,  4.37it/s, epoch=89, grad_norm=98.065, loss=203.4713, lr=9.81e-04] 
Epoch 89 TRAIN:  87%|████████▋ | 120/138 [00:29<00:04,  4.37it/s, epoch=89, grad_norm=224.126, loss=304.7718, lr=9.81e-04]
Epoch 89 TRAIN:  87%|████████▋ | 120/138 [00:29<00:04,  4.37it/s, epoch=89, grad_norm=275.903, loss=343.9488, lr=9.81e-04]
Epoch 89 TRAIN:  90%|████████▉ | 124/138 [00:29<00:02,  4.69it/s, epoch=89, grad_norm=275.903, loss=343.9488, lr=9.81e-04]
Epoch 89 TRAIN:  90%|████████▉ | 124/138 [00:30<00:02,  4.69it/s, epoch=89, grad_norm=228.375, loss=240.3922, lr=9.81e-04]
Epoch 89 TRAIN:  90%|████████▉ | 124/138 [00:30<00:02,  4.69it/s, epoch=89, grad_norm=122.282, loss=255.2370, lr=9.81e-04]
Epoch 89 TRAIN:  90%|████████▉ | 124/138 [00:30<00:02,  4.69it/s, epoch=89, grad_norm=140.881, loss=217.4427, lr=9.81e-04]
Epoch 89 TRAIN:  92%|█████████▏| 127/138 [00:30<00:02,  4.65it/s, epoch=89, grad_norm=140.881, loss=217.4427, lr=9.81e-04]
Epoch 89 TRAIN:  92%|█████████▏| 127/138 [00:30<00:02,  4.65it/s, epoch=89, grad_norm=146.577, loss=261.1472, lr=9.81e-04]
Epoch 89 TRAIN:  92%|█████████▏| 127/138 [00:31<00:02,  4.65it/s, epoch=89, grad_norm=147.736, loss=281.6385, lr=9.81e-04]
Epoch 89 TRAIN:  92%|█████████▏| 127/138 [00:31<00:02,  4.65it/s, epoch=89, grad_norm=296.326, loss=320.1106, lr=9.82e-04]
Epoch 89 TRAIN:  94%|█████████▍| 130/138 [00:31<00:01,  4.67it/s, epoch=89, grad_norm=296.326, loss=320.1106, lr=9.82e-04]
Epoch 89 TRAIN:  94%|█████████▍| 130/138 [00:31<00:01,  4.67it/s, epoch=89, grad_norm=126.287, loss=247.5133, lr=9.82e-04]
Epoch 89 TRAIN:  94%|█████████▍| 130/138 [00:31<00:01,  4.67it/s, epoch=89, grad_norm=251.254, loss=276.2262, lr=9.82e-04]
Epoch 89 TRAIN:  94%|█████████▍| 130/138 [00:31<00:01,  4.67it/s, epoch=89, grad_norm=175.128, loss=282.3249, lr=9.82e-04]
Epoch 89 TRAIN:  96%|█████████▋| 133/138 [00:31<00:01,  4.56it/s, epoch=89, grad_norm=175.128, loss=282.3249, lr=9.82e-04]
Epoch 89 TRAIN:  96%|█████████▋| 133/138 [00:32<00:01,  4.56it/s, epoch=89, grad_norm=144.774, loss=291.1510, lr=9.82e-04]
Epoch 89 TRAIN:  96%|█████████▋| 133/138 [00:32<00:01,  4.56it/s, epoch=89, grad_norm=162.525, loss=294.0570, lr=9.82e-04]
Epoch 89 TRAIN:  96%|█████████▋| 133/138 [00:32<00:01,  4.56it/s, epoch=89, grad_norm=154.207, loss=316.0826, lr=9.82e-04]
Epoch 89 TRAIN:  99%|█████████▊| 136/138 [00:32<00:00,  4.46it/s, epoch=89, grad_norm=154.207, loss=316.0826, lr=9.82e-04]
Epoch 89 TRAIN:  99%|█████████▊| 136/138 [00:32<00:00,  4.17it/s, epoch=89, grad_norm=154.207, loss=316.0826, lr=9.82e-04]

CV:   0%|          | 0/6 [00:00<?, ?it/s]/home/rookie/Documents/batuka/projects/wenet/wenet/wenet/utils/train_utils.py:689: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  amp_autocast(enabled=dtype is not None,

CV:  17%|█▋        | 1/6 [00:00<00:03,  1.29it/s]
                                                 
2026-04-16 20:38:45,805 INFO Epoch 89 Step 12275 CV info lr 9.8200e-04 cv_loss 458.485068581321 rank 0 acc 0.39538312951723736
2026-04-16 20:38:45,807 INFO [Rank 0] Checkpoint: save to checkpoint exp/conformer_mongolian/epoch_89.pt
2026-04-16 20:38:45,930 INFO Epoch 90 Step 12275 TRAIN info lr 9.8200e-04 rank 0
2026-04-16 20:38:45,948 INFO using accumulate grad, new batch size is 1 times larger than before

Epoch 90 TRAIN:   0%|          | 0/136 [00:00<?, ?it/s]
Epoch 90 TRAIN:   0%|          | 0/136 [00:04<?, ?it/s, epoch=90, grad_norm=237.178, loss=230.4944, lr=9.82e-04]
Epoch 90 TRAIN:   1%|          | 1/136 [00:04<09:01,  4.01s/it, epoch=90, grad_norm=237.178, loss=230.4944, lr=9.82e-04]
Epoch 90 TRAIN:   1%|          | 1/136 [00:04<09:01,  4.01s/it, epoch=90, grad_norm=489.077, loss=400.1973, lr=9.82e-04]
Epoch 90 TRAIN:   1%|          | 1/136 [00:04<09:01,  4.01s/it, epoch=90, grad_norm=509.144, loss=442.0211, lr=9.82e-04]
Epoch 90 TRAIN:   1%|          | 1/136 [00:04<09:01,  4.01s/it, epoch=90, grad_norm=352.195, loss=331.0940, lr=9.82e-04]
Epoch 90 TRAIN:   1%|          | 1/136 [00:04<09:01,  4.01s/it, epoch=90, grad_norm=506.130, loss=435.0965, lr=9.82e-04]
Epoch 90 TRAIN:   1%|          | 1/136 [00:04<09:01,  4.01s/it, epoch=90, grad_norm=286.693, loss=272.8700, lr=9.82e-04]
Epoch 90 TRAIN:   4%|▍         | 6/136 [00:04<01:16,  1.71it/s, epoch=90, grad_norm=286.693, loss=272.8700, lr=9.82e-04]
Epoch 90 TRAIN:   4%|▍         | 6/136 [00:04<01:16,  1.71it/s, epoch=90, grad_norm=491.312, loss=377.5541, lr=9.83e-04]
Epoch 90 TRAIN:   4%|▍         | 6/136 [00:04<01:16,  1.71it/s, epoch=90, grad_norm=277.665, loss=324.6406, lr=9.83e-04]
Epoch 90 TRAIN:   4%|▍         | 6/136 [00:04<01:16,  1.71it/s, epoch=90, grad_norm=239.862, loss=243.4569, lr=9.83e-04]
Epoch 90 TRAIN:   4%|▍         | 6/136 [00:05<01:16,  1.71it/s, epoch=90, grad_norm=162.037, loss=286.2445, lr=9.83e-04]
Epoch 90 TRAIN:   4%|▍         | 6/136 [00:05<01:16,  1.71it/s, epoch=90, grad_norm=97.081, loss=247.7767, lr=9.83e-04] 
Epoch 90 TRAIN:   8%|▊         | 11/136 [00:05<00:41,  3.01it/s, epoch=90, grad_norm=97.081, loss=247.7767, lr=9.83e-04]
Epoch 90 TRAIN:   8%|▊         | 11/136 [00:05<00:41,  3.01it/s, epoch=90, grad_norm=88.576, loss=301.0235, lr=9.83e-04]
Epoch 90 TRAIN:   8%|▊         | 11/136 [00:05<00:41,  3.01it/s, epoch=90, grad_norm=83.127, loss=264.9297, lr=9.83e-04]
Epoch 90 TRAIN:   8%|▊         | 11/136 [00:05<00:41,  3.01it/s, epoch=90, grad_norm=112.463, loss=247.8170, lr=9.83e-04]
Epoch 90 TRAIN:   8%|▊         | 11/136 [00:05<00:41,  3.01it/s, epoch=90, grad_norm=90.813, loss=258.3212, lr=9.83e-04] 
Epoch 90 TRAIN:  11%|█         | 15/136 [00:05<00:32,  3.72it/s, epoch=90, grad_norm=90.813, loss=258.3212, lr=9.83e-04]
Epoch 90 TRAIN:  11%|█         | 15/136 [00:06<00:32,  3.72it/s, epoch=90, grad_norm=82.006, loss=281.5704, lr=9.83e-04]
Epoch 90 TRAIN:  11%|█         | 15/136 [00:06<00:32,  3.72it/s, epoch=90, grad_norm=120.556, loss=287.6189, lr=9.83e-04]
Epoch 90 TRAIN:  11%|█         | 15/136 [00:06<00:32,  3.72it/s, epoch=90, grad_norm=77.581, loss=278.5983, lr=9.83e-04] 
Epoch 90 TRAIN:  11%|█         | 15/136 [00:06<00:32,  3.72it/s, epoch=90, grad_norm=125.018, loss=252.4581, lr=9.84e-04]
Epoch 90 TRAIN:  14%|█▍        | 19/136 [00:06<00:29,  3.96it/s, epoch=90, grad_norm=125.018, loss=252.4581, lr=9.84e-04]
Epoch 90 TRAIN:  14%|█▍        | 19/136 [00:06<00:29,  3.96it/s, epoch=90, grad_norm=107.452, loss=302.6624, lr=9.84e-04]
Epoch 90 TRAIN:  14%|█▍        | 19/136 [00:07<00:29,  3.96it/s, epoch=90, grad_norm=84.814, loss=304.0504, lr=9.84e-04] 
Epoch 90 TRAIN:  14%|█▍        | 19/136 [00:07<00:29,  3.96it/s, epoch=90, grad_norm=92.018, loss=283.6977, lr=9.84e-04]
Epoch 90 TRAIN:  16%|█▌        | 22/136 [00:07<00:26,  4.27it/s, epoch=90, grad_norm=92.018, loss=283.6977, lr=9.84e-04]
Epoch 90 TRAIN:  16%|█▌        | 22/136 [00:07<00:26,  4.27it/s, epoch=90, grad_norm=96.739, loss=239.1847, lr=9.84e-04]
Epoch 90 TRAIN:  16%|█▌        | 22/136 [00:07<00:26,  4.27it/s, epoch=90, grad_norm=122.469, loss=249.0239, lr=9.84e-04]
Epoch 90 TRAIN:  16%|█▌        | 22/136 [00:07<00:26,  4.27it/s, epoch=90, grad_norm=105.337, loss=271.1775, lr=9.84e-04]
Epoch 90 TRAIN:  18%|█▊        | 25/136 [00:07<00:25,  4.36it/s, epoch=90, grad_norm=105.337, loss=271.1775, lr=9.84e-04]
Epoch 90 TRAIN:  18%|█▊        | 25/136 [00:08<00:25,  4.36it/s, epoch=90, grad_norm=102.556, loss=272.6158, lr=9.84e-04]
Epoch 90 TRAIN:  18%|█▊        | 25/136 [00:08<00:25,  4.36it/s, epoch=90, grad_norm=88.635, loss=293.0345, lr=9.84e-04] 
Epoch 90 TRAIN:  18%|█▊        | 25/136 [00:08<00:25,  4.36it/s, epoch=90, grad_norm=80.053, loss=268.3256, lr=9.84e-04]
Epoch 90 TRAIN:  21%|██        | 28/136 [00:08<00:24,  4.50it/s, epoch=90, grad_norm=80.053, loss=268.3256, lr=9.84e-04]
Epoch 90 TRAIN:  21%|██        | 28/136 [00:08<00:24,  4.50it/s, epoch=90, grad_norm=93.980, loss=277.9522, lr=9.84e-04]
Epoch 90 TRAIN:  21%|██        | 28/136 [00:09<00:24,  4.50it/s, epoch=90, grad_norm=97.955, loss=265.9435, lr=9.84e-04]
Epoch 90 TRAIN:  21%|██        | 28/136 [00:09<00:24,  4.50it/s, epoch=90, grad_norm=62.374, loss=248.5660, lr=9.84e-04]
Epoch 90 TRAIN:  23%|██▎       | 31/136 [00:09<00:23,  4.49it/s, epoch=90, grad_norm=62.374, loss=248.5660, lr=9.84e-04]
Epoch 90 TRAIN:  23%|██▎       | 31/136 [00:09<00:23,  4.49it/s, epoch=90, grad_norm=96.641, loss=298.1286, lr=9.85e-04]
Epoch 90 TRAIN:  23%|██▎       | 31/136 [00:09<00:23,  4.49it/s, epoch=90, grad_norm=78.333, loss=259.0713, lr=9.85e-04]
Epoch 90 TRAIN:  23%|██▎       | 31/136 [00:09<00:23,  4.49it/s, epoch=90, grad_norm=79.132, loss=274.2709, lr=9.85e-04]
Epoch 90 TRAIN:  25%|██▌       | 34/136 [00:09<00:22,  4.60it/s, epoch=90, grad_norm=79.132, loss=274.2709, lr=9.85e-04]
Epoch 90 TRAIN:  25%|██▌       | 34/136 [00:10<00:22,  4.60it/s, epoch=90, grad_norm=151.392, loss=252.7360, lr=9.85e-04]
Epoch 90 TRAIN:  25%|██▌       | 34/136 [00:10<00:22,  4.60it/s, epoch=90, grad_norm=120.984, loss=288.6230, lr=9.85e-04]
Epoch 90 TRAIN:  25%|██▌       | 34/136 [00:10<00:22,  4.60it/s, epoch=90, grad_norm=99.141, loss=260.8108, lr=9.85e-04] 
Epoch 90 TRAIN:  27%|██▋       | 37/136 [00:10<00:21,  4.67it/s, epoch=90, grad_norm=99.141, loss=260.8108, lr=9.85e-04]
Epoch 90 TRAIN:  27%|██▋       | 37/136 [00:10<00:21,  4.67it/s, epoch=90, grad_norm=171.800, loss=256.6776, lr=9.85e-04]
Epoch 90 TRAIN:  27%|██▋       | 37/136 [00:10<00:21,  4.67it/s, epoch=90, grad_norm=152.718, loss=266.5096, lr=9.85e-04]
Epoch 90 TRAIN:  27%|██▋       | 37/136 [00:11<00:21,  4.67it/s, epoch=90, grad_norm=112.716, loss=260.5194, lr=9.85e-04]
Epoch 90 TRAIN:  29%|██▉       | 40/136 [00:11<00:19,  4.81it/s, epoch=90, grad_norm=112.716, loss=260.5194, lr=9.85e-04]
Epoch 90 TRAIN:  29%|██▉       | 40/136 [00:11<00:19,  4.81it/s, epoch=90, grad_norm=162.149, loss=303.1716, lr=9.85e-04]
Epoch 90 TRAIN:  29%|██▉       | 40/136 [00:11<00:19,  4.81it/s, epoch=90, grad_norm=158.291, loss=307.0620, lr=9.85e-04]
Epoch 90 TRAIN:  29%|██▉       | 40/136 [00:11<00:19,  4.81it/s, epoch=90, grad_norm=99.596, loss=219.8521, lr=9.85e-04] 
Epoch 90 TRAIN:  32%|███▏      | 43/136 [00:11<00:19,  4.71it/s, epoch=90, grad_norm=99.596, loss=219.8521, lr=9.85e-04]
Epoch 90 TRAIN:  32%|███▏      | 43/136 [00:11<00:19,  4.71it/s, epoch=90, grad_norm=113.503, loss=254.0677, lr=9.86e-04]
Epoch 90 TRAIN:  32%|███▏      | 43/136 [00:12<00:19,  4.71it/s, epoch=90, grad_norm=98.318, loss=227.9511, lr=9.86e-04] 
Epoch 90 TRAIN:  32%|███▏      | 43/136 [00:12<00:19,  4.71it/s, epoch=90, grad_norm=89.955, loss=246.5531, lr=9.86e-04]
Epoch 90 TRAIN:  34%|███▍      | 46/136 [00:12<00:19,  4.62it/s, epoch=90, grad_norm=89.955, loss=246.5531, lr=9.86e-04]
Epoch 90 TRAIN:  34%|███▍      | 46/136 [00:12<00:19,  4.62it/s, epoch=90, grad_norm=106.640, loss=275.0195, lr=9.86e-04]
Epoch 90 TRAIN:  34%|███▍      | 46/136 [00:12<00:19,  4.62it/s, epoch=90, grad_norm=68.970, loss=249.4393, lr=9.86e-04] 
Epoch 90 TRAIN:  34%|███▍      | 46/136 [00:13<00:19,  4.62it/s, epoch=90, grad_norm=157.020, loss=284.0672, lr=9.86e-04]
Epoch 90 TRAIN:  36%|███▌      | 49/136 [00:13<00:18,  4.61it/s, epoch=90, grad_norm=157.020, loss=284.0672, lr=9.86e-04]
Epoch 90 TRAIN:  36%|███▌      | 49/136 [00:13<00:18,  4.61it/s, epoch=90, grad_norm=80.631, loss=234.8165, lr=9.86e-04] 
Epoch 90 TRAIN:  36%|███▌      | 49/136 [00:13<00:18,  4.61it/s, epoch=90, grad_norm=108.146, loss=251.9076, lr=9.86e-04]
Epoch 90 TRAIN:  36%|███▌      | 49/136 [00:13<00:18,  4.61it/s, epoch=90, grad_norm=96.938, loss=211.8233, lr=9.86e-04] 
Epoch 90 TRAIN:  38%|███▊      | 52/136 [00:13<00:18,  4.56it/s, epoch=90, grad_norm=96.938, loss=211.8233, lr=9.86e-04]
Epoch 90 TRAIN:  38%|███▊      | 52/136 [00:13<00:18,  4.56it/s, epoch=90, grad_norm=106.044, loss=258.8602, lr=9.86e-04]
Epoch 90 TRAIN:  38%|███▊      | 52/136 [00:14<00:18,  4.56it/s, epoch=90, grad_norm=133.145, loss=258.0782, lr=9.86e-04]
Epoch 90 TRAIN:  38%|███▊      | 52/136 [00:14<00:18,  4.56it/s, epoch=90, grad_norm=88.040, loss=237.0113, lr=9.86e-04] 
Epoch 90 TRAIN:  40%|████      | 55/136 [00:14<00:17,  4.61it/s, epoch=90, grad_norm=88.040, loss=237.0113, lr=9.86e-04]
Epoch 90 TRAIN:  40%|████      | 55/136 [00:14<00:17,  4.61it/s, epoch=90, grad_norm=103.977, loss=278.7108, lr=9.86e-04]
Epoch 90 TRAIN:  40%|████      | 55/136 [00:14<00:17,  4.61it/s, epoch=90, grad_norm=111.275, loss=257.8328, lr=9.87e-04]
Epoch 90 TRAIN:  40%|████      | 55/136 [00:15<00:17,  4.61it/s, epoch=90, grad_norm=99.916, loss=299.0624, lr=9.87e-04] 
Epoch 90 TRAIN:  43%|████▎     | 58/136 [00:15<00:17,  4.47it/s, epoch=90, grad_norm=99.916, loss=299.0624, lr=9.87e-04]
Epoch 90 TRAIN:  43%|████▎     | 58/136 [00:15<00:17,  4.47it/s, epoch=90, grad_norm=121.405, loss=293.5103, lr=9.87e-04]
Epoch 90 TRAIN:  43%|████▎     | 58/136 [00:15<00:17,  4.47it/s, epoch=90, grad_norm=127.463, loss=261.7863, lr=9.87e-04]
Epoch 90 TRAIN:  43%|████▎     | 58/136 [00:15<00:17,  4.47it/s, epoch=90, grad_norm=73.324, loss=277.1746, lr=9.87e-04] 
Epoch 90 TRAIN:  45%|████▍     | 61/136 [00:15<00:17,  4.38it/s, epoch=90, grad_norm=73.324, loss=277.1746, lr=9.87e-04]
Epoch 90 TRAIN:  45%|████▍     | 61/136 [00:16<00:17,  4.38it/s, epoch=90, grad_norm=147.176, loss=258.8468, lr=9.87e-04]
Epoch 90 TRAIN:  45%|████▍     | 61/136 [00:16<00:17,  4.38it/s, epoch=90, grad_norm=79.578, loss=244.9623, lr=9.87e-04] 
Epoch 90 TRAIN:  45%|████▍     | 61/136 [00:16<00:17,  4.38it/s, epoch=90, grad_norm=74.558, loss=293.1151, lr=9.87e-04]
Epoch 90 TRAIN:  47%|████▋     | 64/136 [00:16<00:16,  4.38it/s, epoch=90, grad_norm=74.558, loss=293.1151, lr=9.87e-04]
Epoch 90 TRAIN:  47%|████▋     | 64/136 [00:16<00:16,  4.38it/s, epoch=90, grad_norm=111.956, loss=281.8792, lr=9.87e-04]
Epoch 90 TRAIN:  47%|████▋     | 64/136 [00:16<00:16,  4.38it/s, epoch=90, grad_norm=93.178, loss=256.3722, lr=9.87e-04] 
Epoch 90 TRAIN:  47%|████▋     | 64/136 [00:17<00:16,  4.38it/s, epoch=90, grad_norm=96.974, loss=253.3680, lr=9.87e-04]
Epoch 90 TRAIN:  49%|████▉     | 67/136 [00:17<00:15,  4.58it/s, epoch=90, grad_norm=96.974, loss=253.3680, lr=9.87e-04]
Epoch 90 TRAIN:  49%|████▉     | 67/136 [00:17<00:15,  4.58it/s, epoch=90, grad_norm=97.294, loss=261.2933, lr=9.87e-04]
Epoch 90 TRAIN:  49%|████▉     | 67/136 [00:17<00:15,  4.58it/s, epoch=90, grad_norm=161.620, loss=286.9145, lr=9.88e-04]
Epoch 90 TRAIN:  49%|████▉     | 67/136 [00:17<00:15,  4.58it/s, epoch=90, grad_norm=121.353, loss=242.3773, lr=9.88e-04]
Epoch 90 TRAIN:  51%|█████▏    | 70/136 [00:17<00:13,  4.74it/s, epoch=90, grad_norm=121.353, loss=242.3773, lr=9.88e-04]
Epoch 90 TRAIN:  51%|█████▏    | 70/136 [00:17<00:13,  4.74it/s, epoch=90, grad_norm=123.797, loss=267.9110, lr=9.88e-04]
Epoch 90 TRAIN:  51%|█████▏    | 70/136 [00:18<00:13,  4.74it/s, epoch=90, grad_norm=99.807, loss=275.9359, lr=9.88e-04] 
Epoch 90 TRAIN:  51%|█████▏    | 70/136 [00:18<00:13,  4.74it/s, epoch=90, grad_norm=128.250, loss=265.1169, lr=9.88e-04]
Epoch 90 TRAIN:  54%|█████▎    | 73/136 [00:18<00:14,  4.42it/s, epoch=90, grad_norm=128.250, loss=265.1169, lr=9.88e-04]
Epoch 90 TRAIN:  54%|█████▎    | 73/136 [00:18<00:14,  4.42it/s, epoch=90, grad_norm=102.102, loss=236.4932, lr=9.88e-04]
Epoch 90 TRAIN:  54%|█████▎    | 73/136 [00:18<00:14,  4.42it/s, epoch=90, grad_norm=101.295, loss=281.4836, lr=9.88e-04]
Epoch 90 TRAIN:  54%|█████▎    | 73/136 [00:19<00:14,  4.42it/s, epoch=90, grad_norm=106.642, loss=273.7454, lr=9.88e-04]
Epoch 90 TRAIN:  56%|█████▌    | 76/136 [00:19<00:13,  4.37it/s, epoch=90, grad_norm=106.642, loss=273.7454, lr=9.88e-04]
Epoch 90 TRAIN:  56%|█████▌    | 76/136 [00:19<00:13,  4.37it/s, epoch=90, grad_norm=89.603, loss=280.9355, lr=9.88e-04] 
Epoch 90 TRAIN:  56%|█████▌    | 76/136 [00:19<00:13,  4.37it/s, epoch=90, grad_norm=240.024, loss=190.3509, lr=9.88e-04]
Epoch 90 TRAIN:  56%|█████▌    | 76/136 [00:19<00:13,  4.37it/s, epoch=90, grad_norm=131.650, loss=287.2317, lr=9.88e-04]
Epoch 90 TRAIN:  58%|█████▊    | 79/136 [00:19<00:12,  4.51it/s, epoch=90, grad_norm=131.650, loss=287.2317, lr=9.88e-04]
Epoch 90 TRAIN:  58%|█████▊    | 79/136 [00:20<00:12,  4.51it/s, epoch=90, grad_norm=106.948, loss=286.7440, lr=9.88e-04]
Epoch 90 TRAIN:  58%|█████▊    | 79/136 [00:20<00:12,  4.51it/s, epoch=90, grad_norm=131.301, loss=267.7944, lr=9.88e-04]
Epoch 90 TRAIN:  58%|█████▊    | 79/136 [00:20<00:12,  4.51it/s, epoch=90, grad_norm=152.624, loss=272.1727, lr=9.89e-04]
Epoch 90 TRAIN:  60%|██████    | 82/136 [00:20<00:12,  4.44it/s, epoch=90, grad_norm=152.624, loss=272.1727, lr=9.89e-04]
Epoch 90 TRAIN:  60%|██████    | 82/136 [00:20<00:12,  4.44it/s, epoch=90, grad_norm=142.421, loss=264.4145, lr=9.89e-04]
Epoch 90 TRAIN:  60%|██████    | 82/136 [00:20<00:12,  4.44it/s, epoch=90, grad_norm=155.426, loss=214.9158, lr=9.89e-04]
Epoch 90 TRAIN:  60%|██████    | 82/136 [00:21<00:12,  4.44it/s, epoch=90, grad_norm=258.627, loss=263.9625, lr=9.89e-04]
Epoch 90 TRAIN:  62%|██████▎   | 85/136 [00:21<00:11,  4.55it/s, epoch=90, grad_norm=258.627, loss=263.9625, lr=9.89e-04]
Epoch 90 TRAIN:  62%|██████▎   | 85/136 [00:21<00:11,  4.55it/s, epoch=90, grad_norm=197.710, loss=221.3886, lr=9.89e-04]
Epoch 90 TRAIN:  62%|██████▎   | 85/136 [00:21<00:11,  4.55it/s, epoch=90, grad_norm=112.418, loss=265.6968, lr=9.89e-04]
Epoch 90 TRAIN:  62%|██████▎   | 85/136 [00:21<00:11,  4.55it/s, epoch=90, grad_norm=95.518, loss=199.3188, lr=9.89e-04] 
Epoch 90 TRAIN:  65%|██████▍   | 88/136 [00:21<00:10,  4.38it/s, epoch=90, grad_norm=95.518, loss=199.3188, lr=9.89e-04]
Epoch 90 TRAIN:  65%|██████▍   | 88/136 [00:22<00:10,  4.38it/s, epoch=90, grad_norm=177.642, loss=262.6809, lr=9.89e-04]
Epoch 90 TRAIN:  65%|██████▍   | 88/136 [00:22<00:10,  4.38it/s, epoch=90, grad_norm=125.428, loss=251.1714, lr=9.89e-04]
Epoch 90 TRAIN:  65%|██████▍   | 88/136 [00:22<00:10,  4.38it/s, epoch=90, grad_norm=170.694, loss=291.7820, lr=9.89e-04]
Epoch 90 TRAIN:  67%|██████▋   | 91/136 [00:22<00:10,  4.39it/s, epoch=90, grad_norm=170.694, loss=291.7820, lr=9.89e-04]
Epoch 90 TRAIN:  67%|██████▋   | 91/136 [00:22<00:10,  4.39it/s, epoch=90, grad_norm=138.738, loss=196.5851, lr=9.89e-04]
Epoch 90 TRAIN:  67%|██████▋   | 91/136 [00:22<00:10,  4.39it/s, epoch=90, grad_norm=93.428, loss=230.5465, lr=9.89e-04] 
Epoch 90 TRAIN:  67%|██████▋   | 91/136 [00:23<00:10,  4.39it/s, epoch=90, grad_norm=157.264, loss=236.6005, lr=9.90e-04]
Epoch 90 TRAIN:  69%|██████▉   | 94/136 [00:23<00:09,  4.41it/s, epoch=90, grad_norm=157.264, loss=236.6005, lr=9.90e-04]
Epoch 90 TRAIN:  69%|██████▉   | 94/136 [00:23<00:09,  4.41it/s, epoch=90, grad_norm=194.646, loss=272.6208, lr=9.90e-04]
Epoch 90 TRAIN:  69%|██████▉   | 94/136 [00:23<00:09,  4.41it/s, epoch=90, grad_norm=140.754, loss=289.9603, lr=9.90e-04]
Epoch 90 TRAIN:  69%|██████▉   | 94/136 [00:23<00:09,  4.41it/s, epoch=90, grad_norm=112.702, loss=273.4915, lr=9.90e-04]
Epoch 90 TRAIN:  71%|███████▏  | 97/136 [00:23<00:08,  4.41it/s, epoch=90, grad_norm=112.702, loss=273.4915, lr=9.90e-04]
Epoch 90 TRAIN:  71%|███████▏  | 97/136 [00:24<00:08,  4.41it/s, epoch=90, grad_norm=140.794, loss=226.2937, lr=9.90e-04]
Epoch 90 TRAIN:  71%|███████▏  | 97/136 [00:24<00:08,  4.41it/s, epoch=90, grad_norm=105.769, loss=276.4120, lr=9.90e-04]
Epoch 90 TRAIN:  71%|███████▏  | 97/136 [00:24<00:08,  4.41it/s, epoch=90, grad_norm=149.656, loss=203.5695, lr=9.90e-04]2026-04-16 20:39:10,500 DEBUG TRAIN | steps/sec 4.057| Batch 90/100 loss 203.569473 loss_att 151.252731 loss_ctc 325.641876 th_accuracy 0.511494 lr 9.9000e-04 grad_norm 149.655823 rank 0

Epoch 90 TRAIN:  74%|███████▎  | 100/136 [00:24<00:08,  4.42it/s, epoch=90, grad_norm=149.656, loss=203.5695, lr=9.90e-04]
Epoch 90 TRAIN:  74%|███████▎  | 100/136 [00:24<00:08,  4.42it/s, epoch=90, grad_norm=139.172, loss=277.6397, lr=9.90e-04]
Epoch 90 TRAIN:  74%|███████▎  | 100/136 [00:24<00:08,  4.42it/s, epoch=90, grad_norm=136.815, loss=311.8621, lr=9.90e-04]
Epoch 90 TRAIN:  74%|███████▎  | 100/136 [00:25<00:08,  4.42it/s, epoch=90, grad_norm=103.395, loss=256.2051, lr=9.90e-04]
Epoch 90 TRAIN:  76%|███████▌  | 103/136 [00:25<00:07,  4.54it/s, epoch=90, grad_norm=103.395, loss=256.2051, lr=9.90e-04]
Epoch 90 TRAIN:  76%|███████▌  | 103/136 [00:25<00:07,  4.54it/s, epoch=90, grad_norm=161.697, loss=278.7818, lr=9.90e-04]
Epoch 90 TRAIN:  76%|███████▌  | 103/136 [00:25<00:07,  4.54it/s, epoch=90, grad_norm=150.461, loss=231.0394, lr=9.90e-04]
Epoch 90 TRAIN:  76%|███████▌  | 103/136 [00:25<00:07,  4.54it/s, epoch=90, grad_norm=174.763, loss=283.0579, lr=9.90e-04]
Epoch 90 TRAIN:  78%|███████▊  | 106/136 [00:25<00:06,  4.44it/s, epoch=90, grad_norm=174.763, loss=283.0579, lr=9.90e-04]
Epoch 90 TRAIN:  78%|███████▊  | 106/136 [00:26<00:06,  4.44it/s, epoch=90, grad_norm=211.184, loss=168.2629, lr=9.91e-04]
Epoch 90 TRAIN:  78%|███████▊  | 106/136 [00:26<00:06,  4.44it/s, epoch=90, grad_norm=166.183, loss=237.9119, lr=9.91e-04]
Epoch 90 TRAIN:  78%|███████▊  | 106/136 [00:26<00:06,  4.44it/s, epoch=90, grad_norm=92.621, loss=243.3451, lr=9.91e-04] 
Epoch 90 TRAIN:  80%|████████  | 109/136 [00:26<00:06,  4.44it/s, epoch=90, grad_norm=92.621, loss=243.3451, lr=9.91e-04]
Epoch 90 TRAIN:  80%|████████  | 109/136 [00:26<00:06,  4.44it/s, epoch=90, grad_norm=124.173, loss=254.9123, lr=9.91e-04]
Epoch 90 TRAIN:  80%|████████  | 109/136 [00:26<00:06,  4.44it/s, epoch=90, grad_norm=160.556, loss=266.5234, lr=9.91e-04]
Epoch 90 TRAIN:  80%|████████  | 109/136 [00:27<00:06,  4.44it/s, epoch=90, grad_norm=103.777, loss=200.9956, lr=9.91e-04]
Epoch 90 TRAIN:  82%|████████▏ | 112/136 [00:27<00:05,  4.25it/s, epoch=90, grad_norm=103.777, loss=200.9956, lr=9.91e-04]
Epoch 90 TRAIN:  82%|████████▏ | 112/136 [00:27<00:05,  4.25it/s, epoch=90, grad_norm=83.174, loss=186.1801, lr=9.91e-04] 
Epoch 90 TRAIN:  82%|████████▏ | 112/136 [00:27<00:05,  4.25it/s, epoch=90, grad_norm=148.357, loss=238.9004, lr=9.91e-04]
Epoch 90 TRAIN:  82%|████████▏ | 112/136 [00:27<00:05,  4.25it/s, epoch=90, grad_norm=105.922, loss=216.9375, lr=9.91e-04]
Epoch 90 TRAIN:  85%|████████▍ | 115/136 [00:27<00:04,  4.33it/s, epoch=90, grad_norm=105.922, loss=216.9375, lr=9.91e-04]
Epoch 90 TRAIN:  85%|████████▍ | 115/136 [00:28<00:04,  4.33it/s, epoch=90, grad_norm=83.754, loss=212.6498, lr=9.91e-04] 
Epoch 90 TRAIN:  85%|████████▍ | 115/136 [00:28<00:04,  4.33it/s, epoch=90, grad_norm=180.556, loss=266.2629, lr=9.91e-04]
Epoch 90 TRAIN:  85%|████████▍ | 115/136 [00:28<00:04,  4.33it/s, epoch=90, grad_norm=310.256, loss=329.4623, lr=9.91e-04]
Epoch 90 TRAIN:  87%|████████▋ | 118/136 [00:28<00:04,  4.46it/s, epoch=90, grad_norm=310.256, loss=329.4623, lr=9.91e-04]
Epoch 90 TRAIN:  87%|████████▋ | 118/136 [00:28<00:04,  4.46it/s, epoch=90, grad_norm=197.707, loss=248.6420, lr=9.92e-04]
Epoch 90 TRAIN:  87%|████████▋ | 118/136 [00:29<00:04,  4.46it/s, epoch=90, grad_norm=136.618, loss=255.8015, lr=9.92e-04]
Epoch 90 TRAIN:  87%|████████▋ | 118/136 [00:29<00:04,  4.46it/s, epoch=90, grad_norm=121.535, loss=219.1090, lr=9.92e-04]
Epoch 90 TRAIN:  89%|████████▉ | 121/136 [00:29<00:03,  4.29it/s, epoch=90, grad_norm=121.535, loss=219.1090, lr=9.92e-04]
Epoch 90 TRAIN:  89%|████████▉ | 121/136 [00:29<00:03,  4.29it/s, epoch=90, grad_norm=200.679, loss=294.5387, lr=9.92e-04]
Epoch 90 TRAIN:  89%|████████▉ | 121/136 [00:29<00:03,  4.29it/s, epoch=90, grad_norm=149.440, loss=248.8757, lr=9.92e-04]
Epoch 90 TRAIN:  89%|████████▉ | 121/136 [00:30<00:03,  4.29it/s, epoch=90, grad_norm=140.035, loss=325.0444, lr=9.92e-04]
Epoch 90 TRAIN:  91%|█████████ | 124/136 [00:30<00:02,  4.16it/s, epoch=90, grad_norm=140.035, loss=325.0444, lr=9.92e-04]
Epoch 90 TRAIN:  91%|█████████ | 124/136 [00:30<00:02,  4.16it/s, epoch=90, grad_norm=164.692, loss=297.4438, lr=9.92e-04]
Epoch 90 TRAIN:  91%|█████████ | 124/136 [00:30<00:02,  4.16it/s, epoch=90, grad_norm=129.282, loss=231.4422, lr=9.92e-04]
Epoch 90 TRAIN:  91%|█████████ | 124/136 [00:30<00:02,  4.16it/s, epoch=90, grad_norm=196.317, loss=273.0416, lr=9.92e-04]
Epoch 90 TRAIN:  93%|█████████▎| 127/136 [00:30<00:02,  4.04it/s, epoch=90, grad_norm=196.317, loss=273.0416, lr=9.92e-04]
Epoch 90 TRAIN:  93%|█████████▎| 127/136 [00:31<00:02,  4.04it/s, epoch=90, grad_norm=372.808, loss=317.5578, lr=9.92e-04]
Epoch 90 TRAIN:  93%|█████████▎| 127/136 [00:31<00:02,  4.04it/s, epoch=90, grad_norm=304.317, loss=357.1354, lr=9.92e-04]
Epoch 90 TRAIN:  93%|█████████▎| 127/136 [00:31<00:02,  4.04it/s, epoch=90, grad_norm=257.769, loss=339.0644, lr=9.92e-04]
Epoch 90 TRAIN:  93%|█████████▎| 127/136 [00:31<00:02,  4.04it/s, epoch=90, grad_norm=181.298, loss=324.8231, lr=9.92e-04]
Epoch 90 TRAIN:  96%|█████████▋| 131/136 [00:31<00:01,  4.58it/s, epoch=90, grad_norm=181.298, loss=324.8231, lr=9.92e-04]
Epoch 90 TRAIN:  96%|█████████▋| 131/136 [00:31<00:01,  4.58it/s, epoch=90, grad_norm=172.327, loss=269.8218, lr=9.93e-04]
Epoch 90 TRAIN:  96%|█████████▋| 131/136 [00:31<00:01,  4.58it/s, epoch=90, grad_norm=221.581, loss=322.8607, lr=9.93e-04]
Epoch 90 TRAIN:  96%|█████████▋| 131/136 [00:32<00:01,  4.58it/s, epoch=90, grad_norm=142.738, loss=287.1274, lr=9.93e-04]
Epoch 90 TRAIN:  96%|█████████▋| 131/136 [00:32<00:01,  4.58it/s, epoch=90, grad_norm=168.518, loss=293.0836, lr=9.93e-04]
Epoch 90 TRAIN:  99%|█████████▉| 135/136 [00:32<00:00,  4.73it/s, epoch=90, grad_norm=168.518, loss=293.0836, lr=9.93e-04]
Epoch 90 TRAIN:  99%|█████████▉| 135/136 [00:32<00:00,  4.17it/s, epoch=90, grad_norm=168.518, loss=293.0836, lr=9.93e-04]

CV:   0%|          | 0/6 [00:00<?, ?it/s]/home/rookie/Documents/batuka/projects/wenet/wenet/wenet/utils/train_utils.py:689: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  amp_autocast(enabled=dtype is not None,

CV:  17%|█▋        | 1/6 [00:00<00:03,  1.31it/s]
                                                 
2026-04-16 20:39:19,222 INFO Epoch 90 Step 12410 CV info lr 9.9280e-04 cv_loss 431.8510464754972 rank 0 acc 0.3976708600918452
2026-04-16 20:39:19,224 INFO [Rank 0] Checkpoint: save to checkpoint exp/conformer_mongolian/epoch_90.pt
2026-04-16 20:39:19,335 INFO Epoch 91 Step 12410 TRAIN info lr 9.9280e-04 rank 0
2026-04-16 20:39:19,356 INFO using accumulate grad, new batch size is 1 times larger than before

Epoch 91 TRAIN:   0%|          | 0/135 [00:00<?, ?it/s]
Epoch 91 TRAIN:   0%|          | 0/135 [00:04<?, ?it/s, epoch=91, grad_norm=193.207, loss=250.6829, lr=9.93e-04]
Epoch 91 TRAIN:   1%|          | 1/135 [00:04<09:15,  4.14s/it, epoch=91, grad_norm=193.207, loss=250.6829, lr=9.93e-04]
Epoch 91 TRAIN:   1%|          | 1/135 [00:04<09:15,  4.14s/it, epoch=91, grad_norm=289.177, loss=318.3889, lr=9.93e-04]
Epoch 91 TRAIN:   1%|          | 1/135 [00:04<09:15,  4.14s/it, epoch=91, grad_norm=337.093, loss=306.8730, lr=9.93e-04]
Epoch 91 TRAIN:   1%|          | 1/135 [00:04<09:15,  4.14s/it, epoch=91, grad_norm=253.246, loss=335.6925, lr=9.93e-04]
Epoch 91 TRAIN:   1%|          | 1/135 [00:04<09:15,  4.14s/it, epoch=91, grad_norm=289.663, loss=314.5098, lr=9.93e-04]
Epoch 91 TRAIN:   1%|          | 1/135 [00:04<09:15,  4.14s/it, epoch=91, grad_norm=286.572, loss=348.4179, lr=9.93e-04]
Epoch 91 TRAIN:   1%|          | 1/135 [00:04<09:15,  4.14s/it, epoch=91, grad_norm=247.132, loss=296.7112, lr=9.93e-04]
Epoch 91 TRAIN:   1%|          | 1/135 [00:04<09:15,  4.14s/it, epoch=91, grad_norm=259.202, loss=211.8936, lr=9.93e-04]
Epoch 91 TRAIN:   6%|▌         | 8/135 [00:04<00:56,  2.26it/s, epoch=91, grad_norm=259.202, loss=211.8936, lr=9.93e-04]
Epoch 91 TRAIN:   6%|▌         | 8/135 [00:04<00:56,  2.26it/s, epoch=91, grad_norm=218.371, loss=294.9768, lr=9.94e-04]
Epoch 91 TRAIN:   6%|▌         | 8/135 [00:04<00:56,  2.26it/s, epoch=91, grad_norm=236.610, loss=303.4218, lr=9.94e-04]
Epoch 91 TRAIN:   6%|▌         | 8/135 [00:05<00:56,  2.26it/s, epoch=91, grad_norm=109.777, loss=288.5569, lr=9.94e-04]
Epoch 91 TRAIN:   6%|▌         | 8/135 [00:05<00:56,  2.26it/s, epoch=91, grad_norm=126.019, loss=301.6754, lr=9.94e-04]
Epoch 91 TRAIN:   6%|▌         | 8/135 [00:05<00:56,  2.26it/s, epoch=91, grad_norm=81.375, loss=250.6936, lr=9.94e-04] 
Epoch 91 TRAIN:  10%|▉         | 13/135 [00:05<00:37,  3.22it/s, epoch=91, grad_norm=81.375, loss=250.6936, lr=9.94e-04]
Epoch 91 TRAIN:  10%|▉         | 13/135 [00:05<00:37,  3.22it/s, epoch=91, grad_norm=87.523, loss=286.9846, lr=9.94e-04]
Epoch 91 TRAIN:  10%|▉         | 13/135 [00:05<00:37,  3.22it/s, epoch=91, grad_norm=128.185, loss=294.0734, lr=9.94e-04]
Epoch 91 TRAIN:  10%|▉         | 13/135 [00:06<00:37,  3.22it/s, epoch=91, grad_norm=72.965, loss=255.2994, lr=9.94e-04] 
Epoch 91 TRAIN:  10%|▉         | 13/135 [00:06<00:37,  3.22it/s, epoch=91, grad_norm=127.709, loss=258.4680, lr=9.94e-04]
Epoch 91 TRAIN:  13%|█▎        | 17/135 [00:06<00:32,  3.68it/s, epoch=91, grad_norm=127.709, loss=258.4680, lr=9.94e-04]
Epoch 91 TRAIN:  13%|█▎        | 17/135 [00:06<00:32,  3.68it/s, epoch=91, grad_norm=85.486, loss=268.2672, lr=9.94e-04] 
Epoch 91 TRAIN:  13%|█▎        | 17/135 [00:06<00:32,  3.68it/s, epoch=91, grad_norm=104.329, loss=263.6825, lr=9.94e-04]
Epoch 91 TRAIN:  13%|█▎        | 17/135 [00:06<00:32,  3.68it/s, epoch=91, grad_norm=120.576, loss=294.9155, lr=9.94e-04]
Epoch 91 TRAIN:  13%|█▎        | 17/135 [00:07<00:32,  3.68it/s, epoch=91, grad_norm=89.438, loss=278.3128, lr=9.94e-04] 
Epoch 91 TRAIN:  16%|█▌        | 21/135 [00:07<00:28,  4.02it/s, epoch=91, grad_norm=89.438, loss=278.3128, lr=9.94e-04]
Epoch 91 TRAIN:  16%|█▌        | 21/135 [00:07<00:28,  4.02it/s, epoch=91, grad_norm=94.627, loss=277.9050, lr=9.95e-04]
Epoch 91 TRAIN:  16%|█▌        | 21/135 [00:07<00:28,  4.02it/s, epoch=91, grad_norm=102.648, loss=247.8045, lr=9.95e-04]
Epoch 91 TRAIN:  16%|█▌        | 21/135 [00:07<00:28,  4.02it/s, epoch=91, grad_norm=79.176, loss=275.3673, lr=9.95e-04] 
Epoch 91 TRAIN:  18%|█▊        | 24/135 [00:07<00:25,  4.28it/s, epoch=91, grad_norm=79.176, loss=275.3673, lr=9.95e-04]
Epoch 91 TRAIN:  18%|█▊        | 24/135 [00:07<00:25,  4.28it/s, epoch=91, grad_norm=128.171, loss=240.4097, lr=9.95e-04]
Epoch 91 TRAIN:  18%|█▊        | 24/135 [00:08<00:25,  4.28it/s, epoch=91, grad_norm=80.532, loss=283.0076, lr=9.95e-04] 
Epoch 91 TRAIN:  18%|█▊        | 24/135 [00:08<00:25,  4.28it/s, epoch=91, grad_norm=78.135, loss=232.4280, lr=9.95e-04]
Epoch 91 TRAIN:  20%|██        | 27/135 [00:08<00:23,  4.54it/s, epoch=91, grad_norm=78.135, loss=232.4280, lr=9.95e-04]
Epoch 91 TRAIN:  20%|██        | 27/135 [00:08<00:23,  4.54it/s, epoch=91, grad_norm=107.954, loss=251.6026, lr=9.95e-04]
Epoch 91 TRAIN:  20%|██        | 27/135 [00:08<00:23,  4.54it/s, epoch=91, grad_norm=101.118, loss=255.8115, lr=9.95e-04]
Epoch 91 TRAIN:  20%|██        | 27/135 [00:08<00:23,  4.54it/s, epoch=91, grad_norm=133.311, loss=287.8611, lr=9.95e-04]
Epoch 91 TRAIN:  22%|██▏       | 30/135 [00:08<00:23,  4.55it/s, epoch=91, grad_norm=133.311, loss=287.8611, lr=9.95e-04]
Epoch 91 TRAIN:  22%|██▏       | 30/135 [00:09<00:23,  4.55it/s, epoch=91, grad_norm=101.646, loss=267.1846, lr=9.95e-04]
Epoch 91 TRAIN:  22%|██▏       | 30/135 [00:09<00:23,  4.55it/s, epoch=91, grad_norm=146.555, loss=250.4029, lr=9.95e-04]
Epoch 91 TRAIN:  22%|██▏       | 30/135 [00:09<00:23,  4.55it/s, epoch=91, grad_norm=118.154, loss=253.6740, lr=9.95e-04]
Epoch 91 TRAIN:  24%|██▍       | 33/135 [00:09<00:22,  4.51it/s, epoch=91, grad_norm=118.154, loss=253.6740, lr=9.95e-04]
Epoch 91 TRAIN:  24%|██▍       | 33/135 [00:09<00:22,  4.51it/s, epoch=91, grad_norm=96.816, loss=244.3409, lr=9.96e-04] 
Epoch 91 TRAIN:  24%|██▍       | 33/135 [00:10<00:22,  4.51it/s, epoch=91, grad_norm=122.363, loss=279.3919, lr=9.96e-04]
Epoch 91 TRAIN:  24%|██▍       | 33/135 [00:10<00:22,  4.51it/s, epoch=91, grad_norm=104.392, loss=274.8063, lr=9.96e-04]
Epoch 91 TRAIN:  27%|██▋       | 36/135 [00:10<00:21,  4.56it/s, epoch=91, grad_norm=104.392, loss=274.8063, lr=9.96e-04]
Epoch 91 TRAIN:  27%|██▋       | 36/135 [00:10<00:21,  4.56it/s, epoch=91, grad_norm=132.742, loss=294.2881, lr=9.96e-04]
Epoch 91 TRAIN:  27%|██▋       | 36/135 [00:10<00:21,  4.56it/s, epoch=91, grad_norm=119.520, loss=249.0378, lr=9.96e-04]
Epoch 91 TRAIN:  27%|██▋       | 36/135 [00:10<00:21,  4.56it/s, epoch=91, grad_norm=97.383, loss=233.8444, lr=9.96e-04] 
Epoch 91 TRAIN:  29%|██▉       | 39/135 [00:10<00:20,  4.73it/s, epoch=91, grad_norm=97.383, loss=233.8444, lr=9.96e-04]
Epoch 91 TRAIN:  29%|██▉       | 39/135 [00:10<00:20,  4.73it/s, epoch=91, grad_norm=91.711, loss=266.0885, lr=9.96e-04]
Epoch 91 TRAIN:  29%|██▉       | 39/135 [00:11<00:20,  4.73it/s, epoch=91, grad_norm=121.623, loss=300.1489, lr=9.96e-04]
Epoch 91 TRAIN:  29%|██▉       | 39/135 [00:11<00:20,  4.73it/s, epoch=91, grad_norm=106.536, loss=301.1505, lr=9.96e-04]
Epoch 91 TRAIN:  31%|███       | 42/135 [00:11<00:19,  4.77it/s, epoch=91, grad_norm=106.536, loss=301.1505, lr=9.96e-04]
Epoch 91 TRAIN:  31%|███       | 42/135 [00:11<00:19,  4.77it/s, epoch=91, grad_norm=74.500, loss=268.3063, lr=9.96e-04] 
Epoch 91 TRAIN:  31%|███       | 42/135 [00:11<00:19,  4.77it/s, epoch=91, grad_norm=100.777, loss=269.9484, lr=9.96e-04]
Epoch 91 TRAIN:  31%|███       | 42/135 [00:12<00:19,  4.77it/s, epoch=91, grad_norm=128.378, loss=295.4120, lr=9.96e-04]
Epoch 91 TRAIN:  33%|███▎      | 45/135 [00:12<00:19,  4.69it/s, epoch=91, grad_norm=128.378, loss=295.4120, lr=9.96e-04]
Epoch 91 TRAIN:  33%|███▎      | 45/135 [00:12<00:19,  4.69it/s, epoch=91, grad_norm=98.278, loss=267.4691, lr=9.96e-04] 
Epoch 91 TRAIN:  33%|███▎      | 45/135 [00:12<00:19,  4.69it/s, epoch=91, grad_norm=93.609, loss=286.5681, lr=9.97e-04]
Epoch 91 TRAIN:  33%|███▎      | 45/135 [00:12<00:19,  4.69it/s, epoch=91, grad_norm=94.143, loss=256.5277, lr=9.97e-04]
Epoch 91 TRAIN:  36%|███▌      | 48/135 [00:12<00:18,  4.72it/s, epoch=91, grad_norm=94.143, loss=256.5277, lr=9.97e-04]
Epoch 91 TRAIN:  36%|███▌      | 48/135 [00:12<00:18,  4.72it/s, epoch=91, grad_norm=119.887, loss=281.5656, lr=9.97e-04]
Epoch 91 TRAIN:  36%|███▌      | 48/135 [00:13<00:18,  4.72it/s, epoch=91, grad_norm=77.026, loss=298.8002, lr=9.97e-04] 
Epoch 91 TRAIN:  36%|███▌      | 48/135 [00:13<00:18,  4.72it/s, epoch=91, grad_norm=85.100, loss=259.6479, lr=9.97e-04]
Epoch 91 TRAIN:  38%|███▊      | 51/135 [00:13<00:18,  4.65it/s, epoch=91, grad_norm=85.100, loss=259.6479, lr=9.97e-04]
Epoch 91 TRAIN:  38%|███▊      | 51/135 [00:13<00:18,  4.65it/s, epoch=91, grad_norm=91.743, loss=253.0182, lr=9.97e-04]
Epoch 91 TRAIN:  38%|███▊      | 51/135 [00:13<00:18,  4.65it/s, epoch=91, grad_norm=86.393, loss=252.2337, lr=9.97e-04]
Epoch 91 TRAIN:  38%|███▊      | 51/135 [00:14<00:18,  4.65it/s, epoch=91, grad_norm=143.549, loss=248.8322, lr=9.97e-04]
Epoch 91 TRAIN:  40%|████      | 54/135 [00:14<00:17,  4.64it/s, epoch=91, grad_norm=143.549, loss=248.8322, lr=9.97e-04]
Epoch 91 TRAIN:  40%|████      | 54/135 [00:14<00:17,  4.64it/s, epoch=91, grad_norm=91.368, loss=266.6485, lr=9.97e-04] 
Epoch 91 TRAIN:  40%|████      | 54/135 [00:14<00:17,  4.64it/s, epoch=91, grad_norm=69.348, loss=247.4028, lr=9.97e-04]
Epoch 91 TRAIN:  40%|████      | 54/135 [00:14<00:17,  4.64it/s, epoch=91, grad_norm=114.053, loss=285.6145, lr=9.97e-04]
Epoch 91 TRAIN:  42%|████▏     | 57/135 [00:14<00:16,  4.70it/s, epoch=91, grad_norm=114.053, loss=285.6145, lr=9.97e-04]
Epoch 91 TRAIN:  42%|████▏     | 57/135 [00:14<00:16,  4.70it/s, epoch=91, grad_norm=86.545, loss=252.8938, lr=9.97e-04] 
Epoch 91 TRAIN:  42%|████▏     | 57/135 [00:15<00:16,  4.70it/s, epoch=91, grad_norm=64.202, loss=248.4671, lr=9.98e-04]
Epoch 91 TRAIN:  42%|████▏     | 57/135 [00:15<00:16,  4.70it/s, epoch=91, grad_norm=105.112, loss=278.7797, lr=9.98e-04]
Epoch 91 TRAIN:  44%|████▍     | 60/135 [00:15<00:16,  4.64it/s, epoch=91, grad_norm=105.112, loss=278.7797, lr=9.98e-04]
Epoch 91 TRAIN:  44%|████▍     | 60/135 [00:15<00:16,  4.64it/s, epoch=91, grad_norm=131.879, loss=284.4775, lr=9.98e-04]
Epoch 91 TRAIN:  44%|████▍     | 60/135 [00:15<00:16,  4.64it/s, epoch=91, grad_norm=105.865, loss=228.4805, lr=9.98e-04]
Epoch 91 TRAIN:  44%|████▍     | 60/135 [00:15<00:16,  4.64it/s, epoch=91, grad_norm=96.028, loss=279.6564, lr=9.98e-04] 
Epoch 91 TRAIN:  47%|████▋     | 63/135 [00:15<00:15,  4.65it/s, epoch=91, grad_norm=96.028, loss=279.6564, lr=9.98e-04]
Epoch 91 TRAIN:  47%|████▋     | 63/135 [00:16<00:15,  4.65it/s, epoch=91, grad_norm=127.686, loss=300.1769, lr=9.98e-04]
Epoch 91 TRAIN:  47%|████▋     | 63/135 [00:16<00:15,  4.65it/s, epoch=91, grad_norm=90.708, loss=254.3556, lr=9.98e-04] 
Epoch 91 TRAIN:  47%|████▋     | 63/135 [00:16<00:15,  4.65it/s, epoch=91, grad_norm=107.659, loss=257.5889, lr=9.98e-04]
Epoch 91 TRAIN:  49%|████▉     | 66/135 [00:16<00:14,  4.63it/s, epoch=91, grad_norm=107.659, loss=257.5889, lr=9.98e-04]
Epoch 91 TRAIN:  49%|████▉     | 66/135 [00:16<00:14,  4.63it/s, epoch=91, grad_norm=75.478, loss=258.4273, lr=9.98e-04] 
Epoch 91 TRAIN:  49%|████▉     | 66/135 [00:17<00:14,  4.63it/s, epoch=91, grad_norm=98.471, loss=241.3867, lr=9.98e-04]
Epoch 91 TRAIN:  49%|████▉     | 66/135 [00:17<00:14,  4.63it/s, epoch=91, grad_norm=186.881, loss=308.4954, lr=9.98e-04]
Epoch 91 TRAIN:  51%|█████     | 69/135 [00:17<00:14,  4.64it/s, epoch=91, grad_norm=186.881, loss=308.4954, lr=9.98e-04]
Epoch 91 TRAIN:  51%|█████     | 69/135 [00:17<00:14,  4.64it/s, epoch=91, grad_norm=123.155, loss=297.0419, lr=9.98e-04]
Epoch 91 TRAIN:  51%|█████     | 69/135 [00:17<00:14,  4.64it/s, epoch=91, grad_norm=92.582, loss=248.4694, lr=9.98e-04] 
Epoch 91 TRAIN:  51%|█████     | 69/135 [00:17<00:14,  4.64it/s, epoch=91, grad_norm=86.983, loss=251.3816, lr=9.99e-04]
Epoch 91 TRAIN:  53%|█████▎    | 72/135 [00:17<00:13,  4.63it/s, epoch=91, grad_norm=86.983, loss=251.3816, lr=9.99e-04]
Epoch 91 TRAIN:  53%|█████▎    | 72/135 [00:18<00:13,  4.63it/s, epoch=91, grad_norm=117.197, loss=207.3957, lr=9.99e-04]
Epoch 91 TRAIN:  53%|█████▎    | 72/135 [00:18<00:13,  4.63it/s, epoch=91, grad_norm=77.116, loss=240.5167, lr=9.99e-04] 
Epoch 91 TRAIN:  53%|█████▎    | 72/135 [00:18<00:13,  4.63it/s, epoch=91, grad_norm=96.849, loss=257.3585, lr=9.99e-04]
Epoch 91 TRAIN:  56%|█████▌    | 75/135 [00:18<00:13,  4.50it/s, epoch=91, grad_norm=96.849, loss=257.3585, lr=9.99e-04]
Epoch 91 TRAIN:  56%|█████▌    | 75/135 [00:18<00:13,  4.50it/s, epoch=91, grad_norm=115.274, loss=260.6832, lr=9.99e-04]
Epoch 91 TRAIN:  56%|█████▌    | 75/135 [00:19<00:13,  4.50it/s, epoch=91, grad_norm=130.081, loss=290.9675, lr=9.99e-04]
Epoch 91 TRAIN:  56%|█████▌    | 75/135 [00:19<00:13,  4.50it/s, epoch=91, grad_norm=143.418, loss=285.0041, lr=9.99e-04]
Epoch 91 TRAIN:  58%|█████▊    | 78/135 [00:19<00:12,  4.51it/s, epoch=91, grad_norm=143.418, loss=285.0041, lr=9.99e-04]
Epoch 91 TRAIN:  58%|█████▊    | 78/135 [00:19<00:12,  4.51it/s, epoch=91, grad_norm=170.038, loss=264.7566, lr=9.99e-04]
Epoch 91 TRAIN:  58%|█████▊    | 78/135 [00:19<00:12,  4.51it/s, epoch=91, grad_norm=85.943, loss=282.8489, lr=9.99e-04] 
Epoch 91 TRAIN:  58%|█████▊    | 78/135 [00:19<00:12,  4.51it/s, epoch=91, grad_norm=135.567, loss=230.8078, lr=9.99e-04]
Epoch 91 TRAIN:  60%|██████    | 81/135 [00:19<00:11,  4.72it/s, epoch=91, grad_norm=135.567, loss=230.8078, lr=9.99e-04]
Epoch 91 TRAIN:  60%|██████    | 81/135 [00:20<00:11,  4.72it/s, epoch=91, grad_norm=155.122, loss=215.3851, lr=9.99e-04]
Epoch 91 TRAIN:  60%|██████    | 81/135 [00:20<00:11,  4.72it/s, epoch=91, grad_norm=115.958, loss=307.5604, lr=9.99e-04]
Epoch 91 TRAIN:  60%|██████    | 81/135 [00:20<00:11,  4.72it/s, epoch=91, grad_norm=132.291, loss=240.7642, lr=1.00e-03]
Epoch 91 TRAIN:  62%|██████▏   | 84/135 [00:20<00:10,  4.70it/s, epoch=91, grad_norm=132.291, loss=240.7642, lr=1.00e-03]
Epoch 91 TRAIN:  62%|██████▏   | 84/135 [00:20<00:10,  4.70it/s, epoch=91, grad_norm=150.803, loss=264.1349, lr=1.00e-03]
Epoch 91 TRAIN:  62%|██████▏   | 84/135 [00:20<00:10,  4.70it/s, epoch=91, grad_norm=113.315, loss=280.9143, lr=1.00e-03]
Epoch 91 TRAIN:  62%|██████▏   | 84/135 [00:21<00:10,  4.70it/s, epoch=91, grad_norm=133.341, loss=278.4706, lr=1.00e-03]
Epoch 91 TRAIN:  64%|██████▍   | 87/135 [00:21<00:10,  4.52it/s, epoch=91, grad_norm=133.341, loss=278.4706, lr=1.00e-03]
Epoch 91 TRAIN:  64%|██████▍   | 87/135 [00:21<00:10,  4.52it/s, epoch=91, grad_norm=192.205, loss=220.0613, lr=1.00e-03]
Epoch 91 TRAIN:  64%|██████▍   | 87/135 [00:21<00:10,  4.52it/s, epoch=91, grad_norm=281.832, loss=146.0612, lr=1.00e-03]
Epoch 91 TRAIN:  64%|██████▍   | 87/135 [00:21<00:10,  4.52it/s, epoch=91, grad_norm=200.876, loss=306.4770, lr=1.00e-03]
Epoch 91 TRAIN:  67%|██████▋   | 90/135 [00:21<00:09,  4.66it/s, epoch=91, grad_norm=200.876, loss=306.4770, lr=1.00e-03]
Epoch 91 TRAIN:  67%|██████▋   | 90/135 [00:22<00:09,  4.66it/s, epoch=91, grad_norm=156.571, loss=277.2676, lr=1.00e-03]
Epoch 91 TRAIN:  67%|██████▋   | 90/135 [00:22<00:09,  4.66it/s, epoch=91, grad_norm=188.240, loss=190.4590, lr=1.00e-03]
Epoch 91 TRAIN:  67%|██████▋   | 90/135 [00:22<00:09,  4.66it/s, epoch=91, grad_norm=163.092, loss=245.0089, lr=1.00e-03]
Epoch 91 TRAIN:  69%|██████▉   | 93/135 [00:22<00:09,  4.55it/s, epoch=91, grad_norm=163.092, loss=245.0089, lr=1.00e-03]
Epoch 91 TRAIN:  69%|██████▉   | 93/135 [00:22<00:09,  4.55it/s, epoch=91, grad_norm=173.488, loss=292.2709, lr=1.00e-03]
Epoch 91 TRAIN:  69%|██████▉   | 93/135 [00:22<00:09,  4.55it/s, epoch=91, grad_norm=115.948, loss=235.7373, lr=1.00e-03]
Epoch 91 TRAIN:  69%|██████▉   | 93/135 [00:23<00:09,  4.55it/s, epoch=91, grad_norm=146.648, loss=259.5407, lr=1.00e-03]
Epoch 91 TRAIN:  71%|███████   | 96/135 [00:23<00:08,  4.50it/s, epoch=91, grad_norm=146.648, loss=259.5407, lr=1.00e-03]
Epoch 91 TRAIN:  71%|███████   | 96/135 [00:23<00:08,  4.50it/s, epoch=91, grad_norm=83.933, loss=225.5054, lr=1.00e-03] 
Epoch 91 TRAIN:  71%|███████   | 96/135 [00:23<00:08,  4.50it/s, epoch=91, grad_norm=102.216, loss=231.9883, lr=1.00e-03]
Epoch 91 TRAIN:  71%|███████   | 96/135 [00:23<00:08,  4.50it/s, epoch=91, grad_norm=109.920, loss=251.0853, lr=1.00e-03]
Epoch 91 TRAIN:  73%|███████▎  | 99/135 [00:23<00:07,  4.55it/s, epoch=91, grad_norm=109.920, loss=251.0853, lr=1.00e-03]
Epoch 91 TRAIN:  73%|███████▎  | 99/135 [00:24<00:07,  4.55it/s, epoch=91, grad_norm=91.859, loss=240.8723, lr=1.00e-03] 2026-04-16 20:39:43,374 DEBUG TRAIN | steps/sec 4.107| Batch 91/100 loss 240.872314 loss_att 185.041794 loss_ctc 371.143555 th_accuracy 0.472062 lr 1.0008e-03 grad_norm 91.859200 rank 0

Epoch 91 TRAIN:  73%|███████▎  | 99/135 [00:24<00:07,  4.55it/s, epoch=91, grad_norm=150.692, loss=292.4629, lr=1.00e-03]
Epoch 91 TRAIN:  73%|███████▎  | 99/135 [00:24<00:07,  4.55it/s, epoch=91, grad_norm=178.562, loss=158.1174, lr=1.00e-03]
Epoch 91 TRAIN:  76%|███████▌  | 102/135 [00:24<00:07,  4.66it/s, epoch=91, grad_norm=178.562, loss=158.1174, lr=1.00e-03]
Epoch 91 TRAIN:  76%|███████▌  | 102/135 [00:24<00:07,  4.66it/s, epoch=91, grad_norm=112.889, loss=252.8865, lr=1.00e-03]
Epoch 91 TRAIN:  76%|███████▌  | 102/135 [00:24<00:07,  4.66it/s, epoch=91, grad_norm=122.329, loss=221.8711, lr=1.00e-03]
Epoch 91 TRAIN:  76%|███████▌  | 102/135 [00:25<00:07,  4.66it/s, epoch=91, grad_norm=114.567, loss=173.2834, lr=1.00e-03]
Epoch 91 TRAIN:  78%|███████▊  | 105/135 [00:25<00:06,  4.48it/s, epoch=91, grad_norm=114.567, loss=173.2834, lr=1.00e-03]
Epoch 91 TRAIN:  78%|███████▊  | 105/135 [00:25<00:06,  4.48it/s, epoch=91, grad_norm=139.164, loss=248.2050, lr=1.00e-03]
Epoch 91 TRAIN:  78%|███████▊  | 105/135 [00:25<00:06,  4.48it/s, epoch=91, grad_norm=118.818, loss=198.6789, lr=1.00e-03]
Epoch 91 TRAIN:  78%|███████▊  | 105/135 [00:25<00:06,  4.48it/s, epoch=91, grad_norm=215.747, loss=298.0187, lr=1.00e-03]
Epoch 91 TRAIN:  80%|████████  | 108/135 [00:25<00:05,  4.62it/s, epoch=91, grad_norm=215.747, loss=298.0187, lr=1.00e-03]
Epoch 91 TRAIN:  80%|████████  | 108/135 [00:25<00:05,  4.62it/s, epoch=91, grad_norm=195.953, loss=297.0104, lr=1.00e-03]
Epoch 91 TRAIN:  80%|████████  | 108/135 [00:26<00:05,  4.62it/s, epoch=91, grad_norm=189.990, loss=267.8774, lr=1.00e-03]
Epoch 91 TRAIN:  80%|████████  | 108/135 [00:26<00:05,  4.62it/s, epoch=91, grad_norm=124.160, loss=231.1258, lr=1.00e-03]
Epoch 91 TRAIN:  82%|████████▏ | 111/135 [00:26<00:05,  4.61it/s, epoch=91, grad_norm=124.160, loss=231.1258, lr=1.00e-03]
Epoch 91 TRAIN:  82%|████████▏ | 111/135 [00:26<00:05,  4.61it/s, epoch=91, grad_norm=92.648, loss=280.6288, lr=1.00e-03] 
Epoch 91 TRAIN:  82%|████████▏ | 111/135 [00:26<00:05,  4.61it/s, epoch=91, grad_norm=99.201, loss=246.4934, lr=1.00e-03]
Epoch 91 TRAIN:  82%|████████▏ | 111/135 [00:27<00:05,  4.61it/s, epoch=91, grad_norm=148.988, loss=216.4125, lr=1.00e-03]
Epoch 91 TRAIN:  84%|████████▍ | 114/135 [00:27<00:04,  4.55it/s, epoch=91, grad_norm=148.988, loss=216.4125, lr=1.00e-03]
Epoch 91 TRAIN:  84%|████████▍ | 114/135 [00:27<00:04,  4.55it/s, epoch=91, grad_norm=156.558, loss=212.0372, lr=1.00e-03]
Epoch 91 TRAIN:  84%|████████▍ | 114/135 [00:27<00:04,  4.55it/s, epoch=91, grad_norm=104.822, loss=256.5156, lr=1.00e-03]
Epoch 91 TRAIN:  84%|████████▍ | 114/135 [00:27<00:04,  4.55it/s, epoch=91, grad_norm=187.806, loss=271.0430, lr=1.00e-03]
Epoch 91 TRAIN:  87%|████████▋ | 117/135 [00:27<00:04,  4.37it/s, epoch=91, grad_norm=187.806, loss=271.0430, lr=1.00e-03]
Epoch 91 TRAIN:  87%|████████▋ | 117/135 [00:28<00:04,  4.37it/s, epoch=91, grad_norm=117.241, loss=212.0632, lr=1.00e-03]
Epoch 91 TRAIN:  87%|████████▋ | 117/135 [00:28<00:04,  4.37it/s, epoch=91, grad_norm=104.715, loss=286.0989, lr=1.00e-03]
Epoch 91 TRAIN:  87%|████████▋ | 117/135 [00:28<00:04,  4.37it/s, epoch=91, grad_norm=150.054, loss=273.8179, lr=1.00e-03]
Epoch 91 TRAIN:  89%|████████▉ | 120/135 [00:28<00:03,  4.25it/s, epoch=91, grad_norm=150.054, loss=273.8179, lr=1.00e-03]
Epoch 91 TRAIN:  89%|████████▉ | 120/135 [00:28<00:03,  4.25it/s, epoch=91, grad_norm=206.049, loss=303.2306, lr=1.00e-03]
Epoch 91 TRAIN:  89%|████████▉ | 120/135 [00:28<00:03,  4.25it/s, epoch=91, grad_norm=188.912, loss=289.9629, lr=1.00e-03]
Epoch 91 TRAIN:  89%|████████▉ | 120/135 [00:29<00:03,  4.25it/s, epoch=91, grad_norm=298.933, loss=312.7337, lr=1.00e-03]
Epoch 91 TRAIN:  91%|█████████ | 123/135 [00:29<00:02,  4.38it/s, epoch=91, grad_norm=298.933, loss=312.7337, lr=1.00e-03]
Epoch 91 TRAIN:  91%|█████████ | 123/135 [00:29<00:02,  4.38it/s, epoch=91, grad_norm=143.363, loss=263.7583, lr=1.00e-03]
Epoch 91 TRAIN:  91%|█████████ | 123/135 [00:29<00:02,  4.38it/s, epoch=91, grad_norm=127.327, loss=255.1561, lr=1.00e-03]
Epoch 91 TRAIN:  91%|█████████ | 123/135 [00:29<00:02,  4.38it/s, epoch=91, grad_norm=239.008, loss=269.1423, lr=1.00e-03]
Epoch 91 TRAIN:  93%|█████████▎| 126/135 [00:29<00:01,  4.66it/s, epoch=91, grad_norm=239.008, loss=269.1423, lr=1.00e-03]
Epoch 91 TRAIN:  93%|█████████▎| 126/135 [00:29<00:01,  4.66it/s, epoch=91, grad_norm=208.929, loss=208.9177, lr=1.00e-03]
Epoch 91 TRAIN:  93%|█████████▎| 126/135 [00:30<00:01,  4.66it/s, epoch=91, grad_norm=179.928, loss=282.7094, lr=1.00e-03]
Epoch 91 TRAIN:  93%|█████████▎| 126/135 [00:30<00:01,  4.66it/s, epoch=91, grad_norm=156.986, loss=291.8159, lr=1.00e-03]
Epoch 91 TRAIN:  93%|█████████▎| 126/135 [00:30<00:01,  4.66it/s, epoch=91, grad_norm=306.637, loss=285.7800, lr=1.00e-03]
Epoch 91 TRAIN:  96%|█████████▋| 130/135 [00:30<00:01,  4.94it/s, epoch=91, grad_norm=306.637, loss=285.7800, lr=1.00e-03]
Epoch 91 TRAIN:  96%|█████████▋| 130/135 [00:30<00:01,  4.94it/s, epoch=91, grad_norm=313.009, loss=302.2206, lr=1.00e-03]
Epoch 91 TRAIN:  96%|█████████▋| 130/135 [00:30<00:01,  4.94it/s, epoch=91, grad_norm=204.278, loss=273.5682, lr=1.00e-03]
Epoch 91 TRAIN:  96%|█████████▋| 130/135 [00:30<00:01,  4.94it/s, epoch=91, grad_norm=182.657, loss=257.4362, lr=1.00e-03]
Epoch 91 TRAIN:  96%|█████████▋| 130/135 [00:31<00:01,  4.94it/s, epoch=91, grad_norm=299.678, loss=270.2061, lr=1.00e-03]
Epoch 91 TRAIN:  99%|█████████▉| 134/135 [00:31<00:00,  5.21it/s, epoch=91, grad_norm=299.678, loss=270.2061, lr=1.00e-03]
Epoch 91 TRAIN:  99%|█████████▉| 134/135 [00:31<00:00,  5.21it/s, epoch=91, grad_norm=450.033, loss=317.6859, lr=1.00e-03]
Epoch 91 TRAIN:  99%|█████████▉| 134/135 [00:31<00:00,  5.21it/s, epoch=91, grad_norm=228.283, loss=284.4314, lr=1.00e-03]
Epoch 91 TRAIN:  99%|█████████▉| 134/135 [00:31<00:00,  5.21it/s, epoch=91, grad_norm=200.095, loss=332.9535, lr=1.00e-03]
Epoch 91 TRAIN: 137it [00:31,  5.27it/s, epoch=91, grad_norm=200.095, loss=332.9535, lr=1.00e-03]                         
Epoch 91 TRAIN: 137it [00:31,  4.32it/s, epoch=91, grad_norm=200.095, loss=332.9535, lr=1.00e-03]

CV:   0%|          | 0/6 [00:00<?, ?it/s]/home/rookie/Documents/batuka/projects/wenet/wenet/wenet/utils/train_utils.py:689: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  amp_autocast(enabled=dtype is not None,

CV:  17%|█▋        | 1/6 [00:00<00:03,  1.33it/s]
                                                 
2026-04-16 20:39:51,940 INFO Epoch 91 Step 12547 CV info lr 1.0038e-03 cv_loss 450.2561534534801 rank 0 acc 0.4021059274673462
2026-04-16 20:39:51,941 INFO [Rank 0] Checkpoint: save to checkpoint exp/conformer_mongolian/epoch_91.pt
2026-04-16 20:39:52,051 INFO Epoch 92 Step 12547 TRAIN info lr 1.0038e-03 rank 0
2026-04-16 20:39:52,073 INFO using accumulate grad, new batch size is 1 times larger than before

Epoch 92 TRAIN:   0%|          | 0/137 [00:00<?, ?it/s]
Epoch 92 TRAIN:   0%|          | 0/137 [00:04<?, ?it/s, epoch=92, grad_norm=444.419, loss=345.9343, lr=1.00e-03]
Epoch 92 TRAIN:   1%|          | 1/137 [00:04<09:14,  4.08s/it, epoch=92, grad_norm=444.419, loss=345.9343, lr=1.00e-03]
Epoch 92 TRAIN:   1%|          | 1/137 [00:04<09:14,  4.08s/it, epoch=92, grad_norm=321.558, loss=284.2207, lr=1.00e-03]
Epoch 92 TRAIN:   1%|          | 1/137 [00:04<09:14,  4.08s/it, epoch=92, grad_norm=399.986, loss=343.5813, lr=1.00e-03]
Epoch 92 TRAIN:   1%|          | 1/137 [00:04<09:14,  4.08s/it, epoch=92, grad_norm=515.697, loss=402.6563, lr=1.00e-03]
Epoch 92 TRAIN:   1%|          | 1/137 [00:04<09:14,  4.08s/it, epoch=92, grad_norm=331.763, loss=320.1133, lr=1.00e-03]
Epoch 92 TRAIN:   1%|          | 1/137 [00:04<09:14,  4.08s/it, epoch=92, grad_norm=257.601, loss=297.3696, lr=1.00e-03]
Epoch 92 TRAIN:   1%|          | 1/137 [00:04<09:14,  4.08s/it, epoch=92, grad_norm=124.524, loss=205.2282, lr=1.00e-03]
Epoch 92 TRAIN:   1%|          | 1/137 [00:04<09:14,  4.08s/it, epoch=92, grad_norm=146.030, loss=199.7144, lr=1.00e-03]
Epoch 92 TRAIN:   1%|          | 1/137 [00:04<09:14,  4.08s/it, epoch=92, grad_norm=170.520, loss=273.3385, lr=1.00e-03]
Epoch 92 TRAIN:   7%|▋         | 9/137 [00:04<00:51,  2.48it/s, epoch=92, grad_norm=170.520, loss=273.3385, lr=1.00e-03]
Epoch 92 TRAIN:   7%|▋         | 9/137 [00:04<00:51,  2.48it/s, epoch=92, grad_norm=83.354, loss=268.5302, lr=1.00e-03] 
Epoch 92 TRAIN:   7%|▋         | 9/137 [00:05<00:51,  2.48it/s, epoch=92, grad_norm=103.685, loss=248.0152, lr=1.00e-03]
Epoch 92 TRAIN:   7%|▋         | 9/137 [00:05<00:51,  2.48it/s, epoch=92, grad_norm=139.442, loss=299.4195, lr=1.00e-03]
Epoch 92 TRAIN:   7%|▋         | 9/137 [00:05<00:51,  2.48it/s, epoch=92, grad_norm=92.931, loss=275.3806, lr=1.00e-03] 
Epoch 92 TRAIN:   9%|▉         | 13/137 [00:05<00:38,  3.24it/s, epoch=92, grad_norm=92.931, loss=275.3806, lr=1.00e-03]
Epoch 92 TRAIN:   9%|▉         | 13/137 [00:05<00:38,  3.24it/s, epoch=92, grad_norm=87.020, loss=289.5585, lr=1.00e-03]
Epoch 92 TRAIN:   9%|▉         | 13/137 [00:05<00:38,  3.24it/s, epoch=92, grad_norm=109.726, loss=283.7758, lr=1.00e-03]
Epoch 92 TRAIN:   9%|▉         | 13/137 [00:05<00:38,  3.24it/s, epoch=92, grad_norm=201.152, loss=258.5150, lr=1.01e-03]
Epoch 92 TRAIN:   9%|▉         | 13/137 [00:06<00:38,  3.24it/s, epoch=92, grad_norm=119.011, loss=289.4959, lr=1.01e-03]
Epoch 92 TRAIN:  12%|█▏        | 17/137 [00:06<00:30,  3.90it/s, epoch=92, grad_norm=119.011, loss=289.4959, lr=1.01e-03]
Epoch 92 TRAIN:  12%|█▏        | 17/137 [00:06<00:30,  3.90it/s, epoch=92, grad_norm=133.147, loss=295.4091, lr=1.01e-03]
Epoch 92 TRAIN:  12%|█▏        | 17/137 [00:06<00:30,  3.90it/s, epoch=92, grad_norm=142.294, loss=292.6946, lr=1.01e-03]
Epoch 92 TRAIN:  12%|█▏        | 17/137 [00:06<00:30,  3.90it/s, epoch=92, grad_norm=65.101, loss=295.3013, lr=1.01e-03] 
Epoch 92 TRAIN:  12%|█▏        | 17/137 [00:06<00:30,  3.90it/s, epoch=92, grad_norm=65.075, loss=271.9509, lr=1.01e-03]
Epoch 92 TRAIN:  15%|█▌        | 21/137 [00:06<00:27,  4.20it/s, epoch=92, grad_norm=65.075, loss=271.9509, lr=1.01e-03]
Epoch 92 TRAIN:  15%|█▌        | 21/137 [00:07<00:27,  4.20it/s, epoch=92, grad_norm=73.319, loss=246.0027, lr=1.01e-03]
Epoch 92 TRAIN:  15%|█▌        | 21/137 [00:07<00:27,  4.20it/s, epoch=92, grad_norm=98.995, loss=269.6266, lr=1.01e-03]
Epoch 92 TRAIN:  15%|█▌        | 21/137 [00:07<00:27,  4.20it/s, epoch=92, grad_norm=105.012, loss=300.1898, lr=1.01e-03]
Epoch 92 TRAIN:  18%|█▊        | 24/137 [00:07<00:25,  4.39it/s, epoch=92, grad_norm=105.012, loss=300.1898, lr=1.01e-03]
Epoch 92 TRAIN:  18%|█▊        | 24/137 [00:07<00:25,  4.39it/s, epoch=92, grad_norm=96.738, loss=289.7608, lr=1.01e-03] 
Epoch 92 TRAIN:  18%|█▊        | 24/137 [00:07<00:25,  4.39it/s, epoch=92, grad_norm=97.033, loss=297.8714, lr=1.01e-03]
Epoch 92 TRAIN:  18%|█▊        | 24/137 [00:08<00:25,  4.39it/s, epoch=92, grad_norm=130.371, loss=253.9701, lr=1.01e-03]
Epoch 92 TRAIN:  20%|█▉        | 27/137 [00:08<00:24,  4.41it/s, epoch=92, grad_norm=130.371, loss=253.9701, lr=1.01e-03]
Epoch 92 TRAIN:  20%|█▉        | 27/137 [00:08<00:24,  4.41it/s, epoch=92, grad_norm=76.053, loss=259.8201, lr=1.01e-03] 
Epoch 92 TRAIN:  20%|█▉        | 27/137 [00:08<00:24,  4.41it/s, epoch=92, grad_norm=212.629, loss=315.7198, lr=1.01e-03]
Epoch 92 TRAIN:  20%|█▉        | 27/137 [00:08<00:24,  4.41it/s, epoch=92, grad_norm=94.512, loss=235.9948, lr=1.01e-03] 
Epoch 92 TRAIN:  22%|██▏       | 30/137 [00:08<00:23,  4.58it/s, epoch=92, grad_norm=94.512, loss=235.9948, lr=1.01e-03]
Epoch 92 TRAIN:  22%|██▏       | 30/137 [00:08<00:23,  4.58it/s, epoch=92, grad_norm=91.348, loss=262.1285, lr=1.01e-03]
Epoch 92 TRAIN:  22%|██▏       | 30/137 [00:09<00:23,  4.58it/s, epoch=92, grad_norm=96.276, loss=275.4832, lr=1.01e-03]
Epoch 92 TRAIN:  22%|██▏       | 30/137 [00:09<00:23,  4.58it/s, epoch=92, grad_norm=97.689, loss=242.8925, lr=1.01e-03]
Epoch 92 TRAIN:  24%|██▍       | 33/137 [00:09<00:22,  4.67it/s, epoch=92, grad_norm=97.689, loss=242.8925, lr=1.01e-03]
Epoch 92 TRAIN:  24%|██▍       | 33/137 [00:09<00:22,  4.67it/s, epoch=92, grad_norm=102.445, loss=219.9366, lr=1.01e-03]
Epoch 92 TRAIN:  24%|██▍       | 33/137 [00:09<00:22,  4.67it/s, epoch=92, grad_norm=75.270, loss=266.9272, lr=1.01e-03] 
Epoch 92 TRAIN:  24%|██▍       | 33/137 [00:09<00:22,  4.67it/s, epoch=92, grad_norm=144.338, loss=271.8449, lr=1.01e-03]
Epoch 92 TRAIN:  26%|██▋       | 36/137 [00:09<00:21,  4.76it/s, epoch=92, grad_norm=144.338, loss=271.8449, lr=1.01e-03]
Epoch 92 TRAIN:  26%|██▋       | 36/137 [00:10<00:21,  4.76it/s, epoch=92, grad_norm=117.924, loss=278.4025, lr=1.01e-03]
Epoch 92 TRAIN:  26%|██▋       | 36/137 [00:10<00:21,  4.76it/s, epoch=92, grad_norm=100.619, loss=274.2765, lr=1.01e-03]
Epoch 92 TRAIN:  26%|██▋       | 36/137 [00:10<00:21,  4.76it/s, epoch=92, grad_norm=88.716, loss=236.3652, lr=1.01e-03] 
Epoch 92 TRAIN:  28%|██▊       | 39/137 [00:10<00:20,  4.84it/s, epoch=92, grad_norm=88.716, loss=236.3652, lr=1.01e-03]
Epoch 92 TRAIN:  28%|██▊       | 39/137 [00:10<00:20,  4.84it/s, epoch=92, grad_norm=90.933, loss=196.2614, lr=1.01e-03]
Epoch 92 TRAIN:  28%|██▊       | 39/137 [00:10<00:20,  4.84it/s, epoch=92, grad_norm=99.346, loss=279.3458, lr=1.01e-03]
Epoch 92 TRAIN:  28%|██▊       | 39/137 [00:11<00:20,  4.84it/s, epoch=92, grad_norm=111.546, loss=263.0918, lr=1.01e-03]
Epoch 92 TRAIN:  31%|███       | 42/137 [00:11<00:19,  4.77it/s, epoch=92, grad_norm=111.546, loss=263.0918, lr=1.01e-03]
Epoch 92 TRAIN:  31%|███       | 42/137 [00:11<00:19,  4.77it/s, epoch=92, grad_norm=89.786, loss=251.2406, lr=1.01e-03] 
Epoch 92 TRAIN:  31%|███       | 42/137 [00:11<00:19,  4.77it/s, epoch=92, grad_norm=97.619, loss=263.3269, lr=1.01e-03]
Epoch 92 TRAIN:  31%|███       | 42/137 [00:11<00:19,  4.77it/s, epoch=92, grad_norm=109.909, loss=272.1137, lr=1.01e-03]
Epoch 92 TRAIN:  33%|███▎      | 45/137 [00:11<00:18,  4.85it/s, epoch=92, grad_norm=109.909, loss=272.1137, lr=1.01e-03]
Epoch 92 TRAIN:  33%|███▎      | 45/137 [00:11<00:18,  4.85it/s, epoch=92, grad_norm=92.792, loss=248.1445, lr=1.01e-03] 
Epoch 92 TRAIN:  33%|███▎      | 45/137 [00:12<00:18,  4.85it/s, epoch=92, grad_norm=124.387, loss=272.7285, lr=1.01e-03]
Epoch 92 TRAIN:  33%|███▎      | 45/137 [00:12<00:18,  4.85it/s, epoch=92, grad_norm=94.664, loss=257.7865, lr=1.01e-03] 
Epoch 92 TRAIN:  35%|███▌      | 48/137 [00:12<00:18,  4.73it/s, epoch=92, grad_norm=94.664, loss=257.7865, lr=1.01e-03]
Epoch 92 TRAIN:  35%|███▌      | 48/137 [00:12<00:18,  4.73it/s, epoch=92, grad_norm=98.776, loss=267.5733, lr=1.01e-03]
Epoch 92 TRAIN:  35%|███▌      | 48/137 [00:12<00:18,  4.73it/s, epoch=92, grad_norm=115.542, loss=270.5959, lr=1.01e-03]
Epoch 92 TRAIN:  35%|███▌      | 48/137 [00:13<00:18,  4.73it/s, epoch=92, grad_norm=148.408, loss=279.0746, lr=1.01e-03]
Epoch 92 TRAIN:  37%|███▋      | 51/137 [00:13<00:18,  4.66it/s, epoch=92, grad_norm=148.408, loss=279.0746, lr=1.01e-03]
Epoch 92 TRAIN:  37%|███▋      | 51/137 [00:13<00:18,  4.66it/s, epoch=92, grad_norm=88.400, loss=271.5690, lr=1.01e-03] 
Epoch 92 TRAIN:  37%|███▋      | 51/137 [00:13<00:18,  4.66it/s, epoch=92, grad_norm=109.244, loss=305.3086, lr=1.01e-03]
Epoch 92 TRAIN:  37%|███▋      | 51/137 [00:13<00:18,  4.66it/s, epoch=92, grad_norm=109.048, loss=238.1584, lr=1.01e-03]
Epoch 92 TRAIN:  39%|███▉      | 54/137 [00:13<00:17,  4.70it/s, epoch=92, grad_norm=109.048, loss=238.1584, lr=1.01e-03]
Epoch 92 TRAIN:  39%|███▉      | 54/137 [00:13<00:17,  4.70it/s, epoch=92, grad_norm=125.592, loss=259.8798, lr=1.01e-03]
Epoch 92 TRAIN:  39%|███▉      | 54/137 [00:14<00:17,  4.70it/s, epoch=92, grad_norm=103.571, loss=237.4371, lr=1.01e-03]
Epoch 92 TRAIN:  39%|███▉      | 54/137 [00:14<00:17,  4.70it/s, epoch=92, grad_norm=101.765, loss=306.3600, lr=1.01e-03]
Epoch 92 TRAIN:  42%|████▏     | 57/137 [00:14<00:16,  4.73it/s, epoch=92, grad_norm=101.765, loss=306.3600, lr=1.01e-03]
Epoch 92 TRAIN:  42%|████▏     | 57/137 [00:14<00:16,  4.73it/s, epoch=92, grad_norm=86.461, loss=252.8291, lr=1.01e-03] 
Epoch 92 TRAIN:  42%|████▏     | 57/137 [00:14<00:16,  4.73it/s, epoch=92, grad_norm=84.088, loss=273.5596, lr=1.01e-03]
Epoch 92 TRAIN:  42%|████▏     | 57/137 [00:15<00:16,  4.73it/s, epoch=92, grad_norm=90.747, loss=252.9374, lr=1.01e-03]
Epoch 92 TRAIN:  44%|████▍     | 60/137 [00:15<00:16,  4.67it/s, epoch=92, grad_norm=90.747, loss=252.9374, lr=1.01e-03]
Epoch 92 TRAIN:  44%|████▍     | 60/137 [00:15<00:16,  4.67it/s, epoch=92, grad_norm=100.062, loss=275.3247, lr=1.01e-03]
Epoch 92 TRAIN:  44%|████▍     | 60/137 [00:15<00:16,  4.67it/s, epoch=92, grad_norm=99.894, loss=247.3820, lr=1.01e-03] 
Epoch 92 TRAIN:  44%|████▍     | 60/137 [00:15<00:16,  4.67it/s, epoch=92, grad_norm=88.694, loss=243.8441, lr=1.01e-03]
Epoch 92 TRAIN:  46%|████▌     | 63/137 [00:15<00:15,  4.67it/s, epoch=92, grad_norm=88.694, loss=243.8441, lr=1.01e-03]
Epoch 92 TRAIN:  46%|████▌     | 63/137 [00:15<00:15,  4.67it/s, epoch=92, grad_norm=134.436, loss=301.8628, lr=1.01e-03]
Epoch 92 TRAIN:  46%|████▌     | 63/137 [00:16<00:15,  4.67it/s, epoch=92, grad_norm=109.980, loss=279.3676, lr=1.01e-03]
Epoch 92 TRAIN:  46%|████▌     | 63/137 [00:16<00:15,  4.67it/s, epoch=92, grad_norm=108.474, loss=288.9915, lr=1.01e-03]
Epoch 92 TRAIN:  48%|████▊     | 66/137 [00:16<00:15,  4.66it/s, epoch=92, grad_norm=108.474, loss=288.9915, lr=1.01e-03]
Epoch 92 TRAIN:  48%|████▊     | 66/137 [00:16<00:15,  4.66it/s, epoch=92, grad_norm=179.426, loss=284.5980, lr=1.01e-03]
Epoch 92 TRAIN:  48%|████▊     | 66/137 [00:16<00:15,  4.66it/s, epoch=92, grad_norm=113.804, loss=268.1136, lr=1.01e-03]
Epoch 92 TRAIN:  48%|████▊     | 66/137 [00:17<00:15,  4.66it/s, epoch=92, grad_norm=143.782, loss=277.6253, lr=1.01e-03]
Epoch 92 TRAIN:  50%|█████     | 69/137 [00:17<00:15,  4.51it/s, epoch=92, grad_norm=143.782, loss=277.6253, lr=1.01e-03]
Epoch 92 TRAIN:  50%|█████     | 69/137 [00:17<00:15,  4.51it/s, epoch=92, grad_norm=87.543, loss=259.1407, lr=1.01e-03] 
Epoch 92 TRAIN:  50%|█████     | 69/137 [00:17<00:15,  4.51it/s, epoch=92, grad_norm=102.338, loss=257.4786, lr=1.01e-03]
Epoch 92 TRAIN:  50%|█████     | 69/137 [00:17<00:15,  4.51it/s, epoch=92, grad_norm=114.019, loss=251.3232, lr=1.01e-03]
Epoch 92 TRAIN:  53%|█████▎    | 72/137 [00:17<00:13,  4.69it/s, epoch=92, grad_norm=114.019, loss=251.3232, lr=1.01e-03]
Epoch 92 TRAIN:  53%|█████▎    | 72/137 [00:17<00:13,  4.69it/s, epoch=92, grad_norm=113.735, loss=211.7094, lr=1.01e-03]
Epoch 92 TRAIN:  53%|█████▎    | 72/137 [00:18<00:13,  4.69it/s, epoch=92, grad_norm=176.973, loss=241.1904, lr=1.01e-03]
Epoch 92 TRAIN:  53%|█████▎    | 72/137 [00:18<00:13,  4.69it/s, epoch=92, grad_norm=176.643, loss=256.0907, lr=1.01e-03]
Epoch 92 TRAIN:  55%|█████▍    | 75/137 [00:18<00:13,  4.55it/s, epoch=92, grad_norm=176.643, loss=256.0907, lr=1.01e-03]
Epoch 92 TRAIN:  55%|█████▍    | 75/137 [00:18<00:13,  4.55it/s, epoch=92, grad_norm=94.244, loss=257.2501, lr=1.01e-03] 
Epoch 92 TRAIN:  55%|█████▍    | 75/137 [00:18<00:13,  4.55it/s, epoch=92, grad_norm=129.616, loss=265.9977, lr=1.01e-03]
Epoch 92 TRAIN:  55%|█████▍    | 75/137 [00:18<00:13,  4.55it/s, epoch=92, grad_norm=123.127, loss=265.5537, lr=1.01e-03]
Epoch 92 TRAIN:  57%|█████▋    | 78/137 [00:18<00:13,  4.49it/s, epoch=92, grad_norm=123.127, loss=265.5537, lr=1.01e-03]
Epoch 92 TRAIN:  57%|█████▋    | 78/137 [00:19<00:13,  4.49it/s, epoch=92, grad_norm=109.026, loss=261.0648, lr=1.01e-03]
Epoch 92 TRAIN:  57%|█████▋    | 78/137 [00:19<00:13,  4.49it/s, epoch=92, grad_norm=187.486, loss=297.6205, lr=1.01e-03]
Epoch 92 TRAIN:  57%|█████▋    | 78/137 [00:19<00:13,  4.49it/s, epoch=92, grad_norm=96.963, loss=275.3980, lr=1.01e-03] 
Epoch 92 TRAIN:  59%|█████▉    | 81/137 [00:19<00:12,  4.59it/s, epoch=92, grad_norm=96.963, loss=275.3980, lr=1.01e-03]
Epoch 92 TRAIN:  59%|█████▉    | 81/137 [00:19<00:12,  4.59it/s, epoch=92, grad_norm=234.234, loss=180.2160, lr=1.01e-03]
Epoch 92 TRAIN:  59%|█████▉    | 81/137 [00:19<00:12,  4.59it/s, epoch=92, grad_norm=135.752, loss=272.5284, lr=1.01e-03]
Epoch 92 TRAIN:  59%|█████▉    | 81/137 [00:20<00:12,  4.59it/s, epoch=92, grad_norm=89.293, loss=239.6307, lr=1.01e-03] 
Epoch 92 TRAIN:  61%|██████▏   | 84/137 [00:20<00:11,  4.68it/s, epoch=92, grad_norm=89.293, loss=239.6307, lr=1.01e-03]
Epoch 92 TRAIN:  61%|██████▏   | 84/137 [00:20<00:11,  4.68it/s, epoch=92, grad_norm=77.631, loss=235.5916, lr=1.01e-03]
Epoch 92 TRAIN:  61%|██████▏   | 84/137 [00:20<00:11,  4.68it/s, epoch=92, grad_norm=173.199, loss=244.9502, lr=1.01e-03]
Epoch 92 TRAIN:  61%|██████▏   | 84/137 [00:20<00:11,  4.68it/s, epoch=92, grad_norm=103.838, loss=291.7990, lr=1.01e-03]
Epoch 92 TRAIN:  64%|██████▎   | 87/137 [00:20<00:10,  4.68it/s, epoch=92, grad_norm=103.838, loss=291.7990, lr=1.01e-03]
Epoch 92 TRAIN:  64%|██████▎   | 87/137 [00:21<00:10,  4.68it/s, epoch=92, grad_norm=96.663, loss=254.1107, lr=1.01e-03] 
Epoch 92 TRAIN:  64%|██████▎   | 87/137 [00:21<00:10,  4.68it/s, epoch=92, grad_norm=135.178, loss=193.9762, lr=1.01e-03]
Epoch 92 TRAIN:  64%|██████▎   | 87/137 [00:21<00:10,  4.68it/s, epoch=92, grad_norm=113.858, loss=273.3658, lr=1.01e-03]
Epoch 92 TRAIN:  66%|██████▌   | 90/137 [00:21<00:10,  4.57it/s, epoch=92, grad_norm=113.858, loss=273.3658, lr=1.01e-03]
Epoch 92 TRAIN:  66%|██████▌   | 90/137 [00:21<00:10,  4.57it/s, epoch=92, grad_norm=158.619, loss=246.0218, lr=1.01e-03]
Epoch 92 TRAIN:  66%|██████▌   | 90/137 [00:21<00:10,  4.57it/s, epoch=92, grad_norm=129.890, loss=189.7084, lr=1.01e-03]
Epoch 92 TRAIN:  66%|██████▌   | 90/137 [00:22<00:10,  4.57it/s, epoch=92, grad_norm=169.453, loss=262.5219, lr=1.01e-03]
Epoch 92 TRAIN:  68%|██████▊   | 93/137 [00:22<00:09,  4.69it/s, epoch=92, grad_norm=169.453, loss=262.5219, lr=1.01e-03]
Epoch 92 TRAIN:  68%|██████▊   | 93/137 [00:22<00:09,  4.69it/s, epoch=92, grad_norm=158.266, loss=290.0882, lr=1.01e-03]
Epoch 92 TRAIN:  68%|██████▊   | 93/137 [00:22<00:09,  4.69it/s, epoch=92, grad_norm=83.518, loss=249.2368, lr=1.01e-03] 
Epoch 92 TRAIN:  68%|██████▊   | 93/137 [00:22<00:09,  4.69it/s, epoch=92, grad_norm=124.044, loss=280.0587, lr=1.01e-03]
Epoch 92 TRAIN:  70%|███████   | 96/137 [00:22<00:08,  4.63it/s, epoch=92, grad_norm=124.044, loss=280.0587, lr=1.01e-03]
Epoch 92 TRAIN:  70%|███████   | 96/137 [00:23<00:08,  4.63it/s, epoch=92, grad_norm=107.414, loss=276.7407, lr=1.01e-03]
Epoch 92 TRAIN:  70%|███████   | 96/137 [00:23<00:08,  4.63it/s, epoch=92, grad_norm=166.271, loss=196.4950, lr=1.01e-03]
Epoch 92 TRAIN:  70%|███████   | 96/137 [00:23<00:08,  4.63it/s, epoch=92, grad_norm=186.440, loss=269.8475, lr=1.01e-03]
Epoch 92 TRAIN:  72%|███████▏  | 99/137 [00:23<00:08,  4.44it/s, epoch=92, grad_norm=186.440, loss=269.8475, lr=1.01e-03]
Epoch 92 TRAIN:  72%|███████▏  | 99/137 [00:23<00:08,  4.44it/s, epoch=92, grad_norm=115.022, loss=221.7180, lr=1.01e-03]2026-04-16 20:40:15,883 DEBUG TRAIN | steps/sec 4.214| Batch 92/100 loss 221.717987 loss_att 169.325867 loss_ctc 343.966248 th_accuracy 0.481894 lr 1.0118e-03 grad_norm 115.022057 rank 0

Epoch 92 TRAIN:  72%|███████▏  | 99/137 [00:24<00:08,  4.44it/s, epoch=92, grad_norm=173.474, loss=311.3905, lr=1.01e-03]
Epoch 92 TRAIN:  72%|███████▏  | 99/137 [00:24<00:08,  4.44it/s, epoch=92, grad_norm=88.257, loss=216.7889, lr=1.01e-03] 
Epoch 92 TRAIN:  74%|███████▍  | 102/137 [00:24<00:07,  4.38it/s, epoch=92, grad_norm=88.257, loss=216.7889, lr=1.01e-03]
Epoch 92 TRAIN:  74%|███████▍  | 102/137 [00:24<00:07,  4.38it/s, epoch=92, grad_norm=174.388, loss=210.3082, lr=1.01e-03]
Epoch 92 TRAIN:  74%|███████▍  | 102/137 [00:24<00:07,  4.38it/s, epoch=92, grad_norm=140.357, loss=184.5250, lr=1.01e-03]
Epoch 92 TRAIN:  74%|███████▍  | 102/137 [00:24<00:07,  4.38it/s, epoch=92, grad_norm=115.129, loss=252.7737, lr=1.01e-03]
Epoch 92 TRAIN:  77%|███████▋  | 105/137 [00:24<00:06,  4.63it/s, epoch=92, grad_norm=115.129, loss=252.7737, lr=1.01e-03]
Epoch 92 TRAIN:  77%|███████▋  | 105/137 [00:25<00:06,  4.63it/s, epoch=92, grad_norm=213.984, loss=345.7960, lr=1.01e-03]
Epoch 92 TRAIN:  77%|███████▋  | 105/137 [00:25<00:06,  4.63it/s, epoch=92, grad_norm=169.736, loss=245.2315, lr=1.01e-03]
Epoch 92 TRAIN:  77%|███████▋  | 105/137 [00:25<00:06,  4.63it/s, epoch=92, grad_norm=177.759, loss=253.6210, lr=1.01e-03]
Epoch 92 TRAIN:  79%|███████▉  | 108/137 [00:25<00:06,  4.42it/s, epoch=92, grad_norm=177.759, loss=253.6210, lr=1.01e-03]
Epoch 92 TRAIN:  79%|███████▉  | 108/137 [00:25<00:06,  4.42it/s, epoch=92, grad_norm=84.801, loss=171.1542, lr=1.01e-03] 
Epoch 92 TRAIN:  79%|███████▉  | 108/137 [00:25<00:06,  4.42it/s, epoch=92, grad_norm=109.030, loss=213.7382, lr=1.01e-03]
Epoch 92 TRAIN:  79%|███████▉  | 108/137 [00:26<00:06,  4.42it/s, epoch=92, grad_norm=146.846, loss=233.6113, lr=1.01e-03]
Epoch 92 TRAIN:  81%|████████  | 111/137 [00:26<00:05,  4.67it/s, epoch=92, grad_norm=146.846, loss=233.6113, lr=1.01e-03]
Epoch 92 TRAIN:  81%|████████  | 111/137 [00:26<00:05,  4.67it/s, epoch=92, grad_norm=258.090, loss=274.6747, lr=1.01e-03]
Epoch 92 TRAIN:  81%|████████  | 111/137 [00:26<00:05,  4.67it/s, epoch=92, grad_norm=86.005, loss=212.6958, lr=1.01e-03] 
Epoch 92 TRAIN:  81%|████████  | 111/137 [00:26<00:05,  4.67it/s, epoch=92, grad_norm=98.812, loss=267.7083, lr=1.01e-03]
Epoch 92 TRAIN:  83%|████████▎ | 114/137 [00:26<00:05,  4.49it/s, epoch=92, grad_norm=98.812, loss=267.7083, lr=1.01e-03]
Epoch 92 TRAIN:  83%|████████▎ | 114/137 [00:27<00:05,  4.49it/s, epoch=92, grad_norm=252.435, loss=323.5110, lr=1.01e-03]
Epoch 92 TRAIN:  83%|████████▎ | 114/137 [00:27<00:05,  4.49it/s, epoch=92, grad_norm=169.340, loss=280.8354, lr=1.01e-03]
Epoch 92 TRAIN:  83%|████████▎ | 114/137 [00:27<00:05,  4.49it/s, epoch=92, grad_norm=146.542, loss=267.6263, lr=1.01e-03]
Epoch 92 TRAIN:  85%|████████▌ | 117/137 [00:27<00:04,  4.66it/s, epoch=92, grad_norm=146.542, loss=267.6263, lr=1.01e-03]
Epoch 92 TRAIN:  85%|████████▌ | 117/137 [00:27<00:04,  4.66it/s, epoch=92, grad_norm=129.206, loss=259.0152, lr=1.01e-03]
Epoch 92 TRAIN:  85%|████████▌ | 117/137 [00:27<00:04,  4.66it/s, epoch=92, grad_norm=173.787, loss=279.1714, lr=1.01e-03]
Epoch 92 TRAIN:  85%|████████▌ | 117/137 [00:28<00:04,  4.66it/s, epoch=92, grad_norm=230.864, loss=268.8220, lr=1.01e-03]
Epoch 92 TRAIN:  88%|████████▊ | 120/137 [00:28<00:03,  4.78it/s, epoch=92, grad_norm=230.864, loss=268.8220, lr=1.01e-03]
Epoch 92 TRAIN:  88%|████████▊ | 120/137 [00:28<00:03,  4.78it/s, epoch=92, grad_norm=159.047, loss=218.5903, lr=1.01e-03]
Epoch 92 TRAIN:  88%|████████▊ | 120/137 [00:28<00:03,  4.78it/s, epoch=92, grad_norm=198.749, loss=293.7317, lr=1.01e-03]
Epoch 92 TRAIN:  88%|████████▊ | 120/137 [00:28<00:03,  4.78it/s, epoch=92, grad_norm=195.225, loss=273.0863, lr=1.01e-03]
Epoch 92 TRAIN:  90%|████████▉ | 123/137 [00:28<00:02,  4.71it/s, epoch=92, grad_norm=195.225, loss=273.0863, lr=1.01e-03]
Epoch 92 TRAIN:  90%|████████▉ | 123/137 [00:28<00:02,  4.71it/s, epoch=92, grad_norm=176.531, loss=278.9371, lr=1.01e-03]
Epoch 92 TRAIN:  90%|████████▉ | 123/137 [00:29<00:02,  4.71it/s, epoch=92, grad_norm=152.987, loss=200.8594, lr=1.01e-03]
Epoch 92 TRAIN:  90%|████████▉ | 123/137 [00:29<00:02,  4.71it/s, epoch=92, grad_norm=165.435, loss=239.5760, lr=1.01e-03]
Epoch 92 TRAIN:  92%|█████████▏| 126/137 [00:29<00:02,  4.73it/s, epoch=92, grad_norm=165.435, loss=239.5760, lr=1.01e-03]
Epoch 92 TRAIN:  92%|█████████▏| 126/137 [00:29<00:02,  4.73it/s, epoch=92, grad_norm=164.608, loss=315.8175, lr=1.01e-03]
Epoch 92 TRAIN:  92%|█████████▏| 126/137 [00:29<00:02,  4.73it/s, epoch=92, grad_norm=193.677, loss=313.6104, lr=1.01e-03]
Epoch 92 TRAIN:  92%|█████████▏| 126/137 [00:30<00:02,  4.73it/s, epoch=92, grad_norm=115.806, loss=220.4441, lr=1.01e-03]
Epoch 92 TRAIN:  94%|█████████▍| 129/137 [00:30<00:01,  4.59it/s, epoch=92, grad_norm=115.806, loss=220.4441, lr=1.01e-03]
Epoch 92 TRAIN:  94%|█████████▍| 129/137 [00:30<00:01,  4.59it/s, epoch=92, grad_norm=103.970, loss=256.8342, lr=1.01e-03]
Epoch 92 TRAIN:  94%|█████████▍| 129/137 [00:30<00:01,  4.59it/s, epoch=92, grad_norm=165.501, loss=270.3558, lr=1.01e-03]
Epoch 92 TRAIN:  94%|█████████▍| 129/137 [00:30<00:01,  4.59it/s, epoch=92, grad_norm=127.451, loss=254.6675, lr=1.01e-03]
Epoch 92 TRAIN:  96%|█████████▋| 132/137 [00:30<00:01,  4.72it/s, epoch=92, grad_norm=127.451, loss=254.6675, lr=1.01e-03]
Epoch 92 TRAIN:  96%|█████████▋| 132/137 [00:30<00:01,  4.72it/s, epoch=92, grad_norm=158.593, loss=268.2290, lr=1.01e-03]
Epoch 92 TRAIN:  96%|█████████▋| 132/137 [00:30<00:01,  4.72it/s, epoch=92, grad_norm=309.860, loss=372.2430, lr=1.01e-03]
Epoch 92 TRAIN:  96%|█████████▋| 132/137 [00:31<00:01,  4.72it/s, epoch=92, grad_norm=246.680, loss=185.0173, lr=1.01e-03]
Epoch 92 TRAIN:  96%|█████████▋| 132/137 [00:31<00:01,  4.72it/s, epoch=92, grad_norm=256.091, loss=321.1163, lr=1.01e-03]
Epoch 92 TRAIN:  99%|█████████▉| 136/137 [00:31<00:00,  5.28it/s, epoch=92, grad_norm=256.091, loss=321.1163, lr=1.01e-03]
Epoch 92 TRAIN:  99%|█████████▉| 136/137 [00:31<00:00,  4.35it/s, epoch=92, grad_norm=256.091, loss=321.1163, lr=1.01e-03]

CV:   0%|          | 0/6 [00:00<?, ?it/s]/home/rookie/Documents/batuka/projects/wenet/wenet/wenet/utils/train_utils.py:689: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  amp_autocast(enabled=dtype is not None,

CV:  17%|█▋        | 1/6 [00:00<00:03,  1.33it/s]
                                                 
2026-04-16 20:40:24,156 INFO Epoch 92 Step 12683 CV info lr 1.0146e-03 cv_loss 469.12682550603694 rank 0 acc 0.40067459642887115
2026-04-16 20:40:24,158 INFO [Rank 0] Checkpoint: save to checkpoint exp/conformer_mongolian/epoch_92.pt
2026-04-16 20:40:24,277 INFO Epoch 93 Step 12683 TRAIN info lr 1.0146e-03 rank 0
2026-04-16 20:40:24,297 INFO using accumulate grad, new batch size is 1 times larger than before

Epoch 93 TRAIN:   0%|          | 0/136 [00:00<?, ?it/s]
Epoch 93 TRAIN:   0%|          | 0/136 [00:04<?, ?it/s, epoch=93, grad_norm=411.067, loss=351.5755, lr=1.01e-03]
Epoch 93 TRAIN:   1%|          | 1/136 [00:04<09:11,  4.08s/it, epoch=93, grad_norm=411.067, loss=351.5755, lr=1.01e-03]
Epoch 93 TRAIN:   1%|          | 1/136 [00:04<09:11,  4.08s/it, epoch=93, grad_norm=334.659, loss=301.7258, lr=1.01e-03]
Epoch 93 TRAIN:   1%|          | 1/136 [00:04<09:11,  4.08s/it, epoch=93, grad_norm=326.335, loss=334.9031, lr=1.01e-03]
Epoch 93 TRAIN:   1%|          | 1/136 [00:04<09:11,  4.08s/it, epoch=93, grad_norm=246.804, loss=229.1712, lr=1.01e-03]
Epoch 93 TRAIN:   1%|          | 1/136 [00:04<09:11,  4.08s/it, epoch=93, grad_norm=366.112, loss=330.2781, lr=1.02e-03]
Epoch 93 TRAIN:   1%|          | 1/136 [00:04<09:11,  4.08s/it, epoch=93, grad_norm=268.042, loss=278.6164, lr=1.02e-03]
Epoch 93 TRAIN:   1%|          | 1/136 [00:04<09:11,  4.08s/it, epoch=93, grad_norm=149.064, loss=268.2557, lr=1.02e-03]
Epoch 93 TRAIN:   1%|          | 1/136 [00:04<09:11,  4.08s/it, epoch=93, grad_norm=161.208, loss=231.7924, lr=1.02e-03]
Epoch 93 TRAIN:   6%|▌         | 8/136 [00:04<00:56,  2.25it/s, epoch=93, grad_norm=161.208, loss=231.7924, lr=1.02e-03]
Epoch 93 TRAIN:   6%|▌         | 8/136 [00:04<00:56,  2.25it/s, epoch=93, grad_norm=181.614, loss=295.7899, lr=1.02e-03]
Epoch 93 TRAIN:   6%|▌         | 8/136 [00:04<00:56,  2.25it/s, epoch=93, grad_norm=76.859, loss=266.0735, lr=1.02e-03] 
Epoch 93 TRAIN:   6%|▌         | 8/136 [00:05<00:56,  2.25it/s, epoch=93, grad_norm=124.150, loss=255.0020, lr=1.02e-03]
Epoch 93 TRAIN:   6%|▌         | 8/136 [00:05<00:56,  2.25it/s, epoch=93, grad_norm=81.600, loss=285.0761, lr=1.02e-03] 
Epoch 93 TRAIN:   9%|▉         | 12/136 [00:05<00:39,  3.10it/s, epoch=93, grad_norm=81.600, loss=285.0761, lr=1.02e-03]
Epoch 93 TRAIN:   9%|▉         | 12/136 [00:05<00:39,  3.10it/s, epoch=93, grad_norm=96.157, loss=316.5569, lr=1.02e-03]
Epoch 93 TRAIN:   9%|▉         | 12/136 [00:05<00:39,  3.10it/s, epoch=93, grad_norm=106.876, loss=281.6901, lr=1.02e-03]
Epoch 93 TRAIN:   9%|▉         | 12/136 [00:05<00:39,  3.10it/s, epoch=93, grad_norm=88.742, loss=262.4709, lr=1.02e-03] 
Epoch 93 TRAIN:   9%|▉         | 12/136 [00:05<00:39,  3.10it/s, epoch=93, grad_norm=89.677, loss=284.6198, lr=1.02e-03]
Epoch 93 TRAIN:  12%|█▏        | 16/136 [00:05<00:31,  3.82it/s, epoch=93, grad_norm=89.677, loss=284.6198, lr=1.02e-03]
Epoch 93 TRAIN:  12%|█▏        | 16/136 [00:06<00:31,  3.82it/s, epoch=93, grad_norm=75.826, loss=300.3558, lr=1.02e-03]
Epoch 93 TRAIN:  12%|█▏        | 16/136 [00:06<00:31,  3.82it/s, epoch=93, grad_norm=101.789, loss=267.3438, lr=1.02e-03]
Epoch 93 TRAIN:  12%|█▏        | 16/136 [00:06<00:31,  3.82it/s, epoch=93, grad_norm=78.601, loss=293.8674, lr=1.02e-03] 
Epoch 93 TRAIN:  12%|█▏        | 16/136 [00:06<00:31,  3.82it/s, epoch=93, grad_norm=78.209, loss=280.3693, lr=1.02e-03]
Epoch 93 TRAIN:  15%|█▍        | 20/136 [00:06<00:27,  4.26it/s, epoch=93, grad_norm=78.209, loss=280.3693, lr=1.02e-03]
Epoch 93 TRAIN:  15%|█▍        | 20/136 [00:06<00:27,  4.26it/s, epoch=93, grad_norm=148.517, loss=275.2308, lr=1.02e-03]
Epoch 93 TRAIN:  15%|█▍        | 20/136 [00:07<00:27,  4.26it/s, epoch=93, grad_norm=97.779, loss=248.0652, lr=1.02e-03] 
Epoch 93 TRAIN:  15%|█▍        | 20/136 [00:07<00:27,  4.26it/s, epoch=93, grad_norm=71.041, loss=269.7763, lr=1.02e-03]
Epoch 93 TRAIN:  15%|█▍        | 20/136 [00:07<00:27,  4.26it/s, epoch=93, grad_norm=126.423, loss=284.0591, lr=1.02e-03]
Epoch 93 TRAIN:  18%|█▊        | 24/136 [00:07<00:25,  4.43it/s, epoch=93, grad_norm=126.423, loss=284.0591, lr=1.02e-03]
Epoch 93 TRAIN:  18%|█▊        | 24/136 [00:07<00:25,  4.43it/s, epoch=93, grad_norm=77.594, loss=273.2346, lr=1.02e-03] 
Epoch 93 TRAIN:  18%|█▊        | 24/136 [00:07<00:25,  4.43it/s, epoch=93, grad_norm=106.112, loss=267.1687, lr=1.02e-03]
Epoch 93 TRAIN:  18%|█▊        | 24/136 [00:08<00:25,  4.43it/s, epoch=93, grad_norm=79.516, loss=249.1548, lr=1.02e-03] 
Epoch 93 TRAIN:  20%|█▉        | 27/136 [00:08<00:24,  4.52it/s, epoch=93, grad_norm=79.516, loss=249.1548, lr=1.02e-03]
Epoch 93 TRAIN:  20%|█▉        | 27/136 [00:08<00:24,  4.52it/s, epoch=93, grad_norm=137.070, loss=270.8768, lr=1.02e-03]
Epoch 93 TRAIN:  20%|█▉        | 27/136 [00:08<00:24,  4.52it/s, epoch=93, grad_norm=129.284, loss=280.2244, lr=1.02e-03]
Epoch 93 TRAIN:  20%|█▉        | 27/136 [00:08<00:24,  4.52it/s, epoch=93, grad_norm=99.749, loss=247.7913, lr=1.02e-03] 
Epoch 93 TRAIN:  22%|██▏       | 30/136 [00:08<00:22,  4.69it/s, epoch=93, grad_norm=99.749, loss=247.7913, lr=1.02e-03]
Epoch 93 TRAIN:  22%|██▏       | 30/136 [00:08<00:22,  4.69it/s, epoch=93, grad_norm=146.499, loss=267.6775, lr=1.02e-03]
Epoch 93 TRAIN:  22%|██▏       | 30/136 [00:09<00:22,  4.69it/s, epoch=93, grad_norm=112.999, loss=263.1433, lr=1.02e-03]
Epoch 93 TRAIN:  22%|██▏       | 30/136 [00:09<00:22,  4.69it/s, epoch=93, grad_norm=74.027, loss=251.7493, lr=1.02e-03] 
Epoch 93 TRAIN:  24%|██▍       | 33/136 [00:09<00:21,  4.79it/s, epoch=93, grad_norm=74.027, loss=251.7493, lr=1.02e-03]
Epoch 93 TRAIN:  24%|██▍       | 33/136 [00:09<00:21,  4.79it/s, epoch=93, grad_norm=98.992, loss=273.3804, lr=1.02e-03]
Epoch 93 TRAIN:  24%|██▍       | 33/136 [00:09<00:21,  4.79it/s, epoch=93, grad_norm=96.456, loss=267.5171, lr=1.02e-03]
Epoch 93 TRAIN:  24%|██▍       | 33/136 [00:09<00:21,  4.79it/s, epoch=93, grad_norm=133.781, loss=297.5183, lr=1.02e-03]
Epoch 93 TRAIN:  26%|██▋       | 36/136 [00:09<00:20,  4.84it/s, epoch=93, grad_norm=133.781, loss=297.5183, lr=1.02e-03]
Epoch 93 TRAIN:  26%|██▋       | 36/136 [00:10<00:20,  4.84it/s, epoch=93, grad_norm=144.323, loss=226.2350, lr=1.02e-03]
Epoch 93 TRAIN:  26%|██▋       | 36/136 [00:10<00:20,  4.84it/s, epoch=93, grad_norm=110.120, loss=296.6530, lr=1.02e-03]
Epoch 93 TRAIN:  26%|██▋       | 36/136 [00:10<00:20,  4.84it/s, epoch=93, grad_norm=128.116, loss=264.2226, lr=1.02e-03]
Epoch 93 TRAIN:  29%|██▊       | 39/136 [00:10<00:19,  4.98it/s, epoch=93, grad_norm=128.116, loss=264.2226, lr=1.02e-03]
Epoch 93 TRAIN:  29%|██▊       | 39/136 [00:10<00:19,  4.98it/s, epoch=93, grad_norm=94.989, loss=228.5689, lr=1.02e-03] 
Epoch 93 TRAIN:  29%|██▊       | 39/136 [00:10<00:19,  4.98it/s, epoch=93, grad_norm=125.719, loss=237.1905, lr=1.02e-03]
Epoch 93 TRAIN:  29%|██▊       | 39/136 [00:11<00:19,  4.98it/s, epoch=93, grad_norm=131.984, loss=238.9406, lr=1.02e-03]
Epoch 93 TRAIN:  31%|███       | 42/136 [00:11<00:20,  4.63it/s, epoch=93, grad_norm=131.984, loss=238.9406, lr=1.02e-03]
Epoch 93 TRAIN:  31%|███       | 42/136 [00:11<00:20,  4.63it/s, epoch=93, grad_norm=125.517, loss=267.1702, lr=1.02e-03]
Epoch 93 TRAIN:  31%|███       | 42/136 [00:11<00:20,  4.63it/s, epoch=93, grad_norm=95.750, loss=267.7684, lr=1.02e-03] 
Epoch 93 TRAIN:  31%|███       | 42/136 [00:11<00:20,  4.63it/s, epoch=93, grad_norm=113.704, loss=273.5625, lr=1.02e-03]
Epoch 93 TRAIN:  33%|███▎      | 45/136 [00:11<00:18,  4.87it/s, epoch=93, grad_norm=113.704, loss=273.5625, lr=1.02e-03]
Epoch 93 TRAIN:  33%|███▎      | 45/136 [00:12<00:18,  4.87it/s, epoch=93, grad_norm=118.453, loss=218.2721, lr=1.02e-03]
Epoch 93 TRAIN:  33%|███▎      | 45/136 [00:12<00:18,  4.87it/s, epoch=93, grad_norm=96.992, loss=247.7777, lr=1.02e-03] 
Epoch 93 TRAIN:  33%|███▎      | 45/136 [00:12<00:18,  4.87it/s, epoch=93, grad_norm=111.837, loss=268.1517, lr=1.02e-03]
Epoch 93 TRAIN:  35%|███▌      | 48/136 [00:12<00:18,  4.67it/s, epoch=93, grad_norm=111.837, loss=268.1517, lr=1.02e-03]
Epoch 93 TRAIN:  35%|███▌      | 48/136 [00:12<00:18,  4.67it/s, epoch=93, grad_norm=81.727, loss=268.4916, lr=1.02e-03] 
Epoch 93 TRAIN:  35%|███▌      | 48/136 [00:12<00:18,  4.67it/s, epoch=93, grad_norm=107.808, loss=289.2355, lr=1.02e-03]
Epoch 93 TRAIN:  35%|███▌      | 48/136 [00:13<00:18,  4.67it/s, epoch=93, grad_norm=107.872, loss=276.8473, lr=1.02e-03]
Epoch 93 TRAIN:  38%|███▊      | 51/136 [00:13<00:18,  4.64it/s, epoch=93, grad_norm=107.872, loss=276.8473, lr=1.02e-03]
Epoch 93 TRAIN:  38%|███▊      | 51/136 [00:13<00:18,  4.64it/s, epoch=93, grad_norm=86.641, loss=246.4092, lr=1.02e-03] 
Epoch 93 TRAIN:  38%|███▊      | 51/136 [00:13<00:18,  4.64it/s, epoch=93, grad_norm=100.561, loss=287.7442, lr=1.02e-03]
Epoch 93 TRAIN:  38%|███▊      | 51/136 [00:13<00:18,  4.64it/s, epoch=93, grad_norm=98.844, loss=216.6656, lr=1.02e-03] 
Epoch 93 TRAIN:  40%|███▉      | 54/136 [00:13<00:17,  4.57it/s, epoch=93, grad_norm=98.844, loss=216.6656, lr=1.02e-03]
Epoch 93 TRAIN:  40%|███▉      | 54/136 [00:13<00:17,  4.57it/s, epoch=93, grad_norm=145.249, loss=251.5749, lr=1.02e-03]
Epoch 93 TRAIN:  40%|███▉      | 54/136 [00:14<00:17,  4.57it/s, epoch=93, grad_norm=173.563, loss=292.5215, lr=1.02e-03]
Epoch 93 TRAIN:  40%|███▉      | 54/136 [00:14<00:17,  4.57it/s, epoch=93, grad_norm=106.637, loss=268.9139, lr=1.02e-03]
Epoch 93 TRAIN:  42%|████▏     | 57/136 [00:14<00:16,  4.78it/s, epoch=93, grad_norm=106.637, loss=268.9139, lr=1.02e-03]
Epoch 93 TRAIN:  42%|████▏     | 57/136 [00:14<00:16,  4.78it/s, epoch=93, grad_norm=99.062, loss=238.2553, lr=1.02e-03] 
Epoch 93 TRAIN:  42%|████▏     | 57/136 [00:14<00:16,  4.78it/s, epoch=93, grad_norm=96.431, loss=262.2582, lr=1.02e-03]
Epoch 93 TRAIN:  42%|████▏     | 57/136 [00:15<00:16,  4.78it/s, epoch=93, grad_norm=114.395, loss=281.6406, lr=1.02e-03]
Epoch 93 TRAIN:  44%|████▍     | 60/136 [00:15<00:16,  4.51it/s, epoch=93, grad_norm=114.395, loss=281.6406, lr=1.02e-03]
Epoch 93 TRAIN:  44%|████▍     | 60/136 [00:15<00:16,  4.51it/s, epoch=93, grad_norm=96.955, loss=249.7100, lr=1.02e-03] 
Epoch 93 TRAIN:  44%|████▍     | 60/136 [00:15<00:16,  4.51it/s, epoch=93, grad_norm=141.116, loss=269.5364, lr=1.02e-03]
Epoch 93 TRAIN:  44%|████▍     | 60/136 [00:15<00:16,  4.51it/s, epoch=93, grad_norm=128.001, loss=300.1597, lr=1.02e-03]
Epoch 93 TRAIN:  46%|████▋     | 63/136 [00:15<00:15,  4.64it/s, epoch=93, grad_norm=128.001, loss=300.1597, lr=1.02e-03]
Epoch 93 TRAIN:  46%|████▋     | 63/136 [00:15<00:15,  4.64it/s, epoch=93, grad_norm=99.471, loss=277.0172, lr=1.02e-03] 
Epoch 93 TRAIN:  46%|████▋     | 63/136 [00:16<00:15,  4.64it/s, epoch=93, grad_norm=109.434, loss=247.0888, lr=1.02e-03]
Epoch 93 TRAIN:  46%|████▋     | 63/136 [00:16<00:15,  4.64it/s, epoch=93, grad_norm=107.557, loss=274.5704, lr=1.02e-03]
Epoch 93 TRAIN:  49%|████▊     | 66/136 [00:16<00:14,  4.68it/s, epoch=93, grad_norm=107.557, loss=274.5704, lr=1.02e-03]
Epoch 93 TRAIN:  49%|████▊     | 66/136 [00:16<00:14,  4.68it/s, epoch=93, grad_norm=128.109, loss=250.1902, lr=1.02e-03]
Epoch 93 TRAIN:  49%|████▊     | 66/136 [00:16<00:14,  4.68it/s, epoch=93, grad_norm=137.324, loss=263.1169, lr=1.02e-03]
Epoch 93 TRAIN:  49%|████▊     | 66/136 [00:16<00:14,  4.68it/s, epoch=93, grad_norm=151.032, loss=249.4037, lr=1.02e-03]
Epoch 93 TRAIN:  51%|█████     | 69/136 [00:16<00:14,  4.66it/s, epoch=93, grad_norm=151.032, loss=249.4037, lr=1.02e-03]
Epoch 93 TRAIN:  51%|█████     | 69/136 [00:17<00:14,  4.66it/s, epoch=93, grad_norm=99.082, loss=266.5669, lr=1.02e-03] 
Epoch 93 TRAIN:  51%|█████     | 69/136 [00:17<00:14,  4.66it/s, epoch=93, grad_norm=98.980, loss=283.7847, lr=1.02e-03]
Epoch 93 TRAIN:  51%|█████     | 69/136 [00:17<00:14,  4.66it/s, epoch=93, grad_norm=146.513, loss=238.5360, lr=1.02e-03]
Epoch 93 TRAIN:  53%|█████▎    | 72/136 [00:17<00:14,  4.44it/s, epoch=93, grad_norm=146.513, loss=238.5360, lr=1.02e-03]
Epoch 93 TRAIN:  53%|█████▎    | 72/136 [00:17<00:14,  4.44it/s, epoch=93, grad_norm=197.966, loss=293.1840, lr=1.02e-03]
Epoch 93 TRAIN:  53%|█████▎    | 72/136 [00:18<00:14,  4.44it/s, epoch=93, grad_norm=102.181, loss=237.3291, lr=1.02e-03]
Epoch 93 TRAIN:  53%|█████▎    | 72/136 [00:18<00:14,  4.44it/s, epoch=93, grad_norm=145.965, loss=294.3938, lr=1.02e-03]
Epoch 93 TRAIN:  55%|█████▌    | 75/136 [00:18<00:13,  4.44it/s, epoch=93, grad_norm=145.965, loss=294.3938, lr=1.02e-03]
Epoch 93 TRAIN:  55%|█████▌    | 75/136 [00:18<00:13,  4.44it/s, epoch=93, grad_norm=97.708, loss=260.0443, lr=1.02e-03] 
Epoch 93 TRAIN:  55%|█████▌    | 75/136 [00:18<00:13,  4.44it/s, epoch=93, grad_norm=114.240, loss=259.6249, lr=1.02e-03]
Epoch 93 TRAIN:  55%|█████▌    | 75/136 [00:19<00:13,  4.44it/s, epoch=93, grad_norm=146.618, loss=242.8339, lr=1.02e-03]
Epoch 93 TRAIN:  57%|█████▋    | 78/136 [00:19<00:13,  4.27it/s, epoch=93, grad_norm=146.618, loss=242.8339, lr=1.02e-03]
Epoch 93 TRAIN:  57%|█████▋    | 78/136 [00:19<00:13,  4.27it/s, epoch=93, grad_norm=111.871, loss=258.2083, lr=1.02e-03]
Epoch 93 TRAIN:  57%|█████▋    | 78/136 [00:19<00:13,  4.27it/s, epoch=93, grad_norm=84.105, loss=232.1879, lr=1.02e-03] 
Epoch 93 TRAIN:  57%|█████▋    | 78/136 [00:19<00:13,  4.27it/s, epoch=93, grad_norm=154.350, loss=249.9109, lr=1.02e-03]
Epoch 93 TRAIN:  60%|█████▉    | 81/136 [00:19<00:12,  4.52it/s, epoch=93, grad_norm=154.350, loss=249.9109, lr=1.02e-03]
Epoch 93 TRAIN:  60%|█████▉    | 81/136 [00:20<00:12,  4.52it/s, epoch=93, grad_norm=106.182, loss=250.5869, lr=1.02e-03]
Epoch 93 TRAIN:  60%|█████▉    | 81/136 [00:20<00:12,  4.52it/s, epoch=93, grad_norm=105.442, loss=297.8292, lr=1.02e-03]
Epoch 93 TRAIN:  60%|█████▉    | 81/136 [00:20<00:12,  4.52it/s, epoch=93, grad_norm=208.349, loss=173.2226, lr=1.02e-03]
Epoch 93 TRAIN:  62%|██████▏   | 84/136 [00:20<00:12,  4.24it/s, epoch=93, grad_norm=208.349, loss=173.2226, lr=1.02e-03]
Epoch 93 TRAIN:  62%|██████▏   | 84/136 [00:20<00:12,  4.24it/s, epoch=93, grad_norm=182.598, loss=215.8003, lr=1.02e-03]
Epoch 93 TRAIN:  62%|██████▏   | 84/136 [00:20<00:12,  4.24it/s, epoch=93, grad_norm=138.543, loss=244.2340, lr=1.02e-03]
Epoch 93 TRAIN:  62%|██████▏   | 84/136 [00:21<00:12,  4.24it/s, epoch=93, grad_norm=134.712, loss=188.3996, lr=1.02e-03]
Epoch 93 TRAIN:  64%|██████▍   | 87/136 [00:21<00:10,  4.49it/s, epoch=93, grad_norm=134.712, loss=188.3996, lr=1.02e-03]
Epoch 93 TRAIN:  64%|██████▍   | 87/136 [00:21<00:10,  4.49it/s, epoch=93, grad_norm=175.861, loss=293.2222, lr=1.02e-03]
Epoch 93 TRAIN:  64%|██████▍   | 87/136 [00:21<00:10,  4.49it/s, epoch=93, grad_norm=156.567, loss=259.0402, lr=1.02e-03]
Epoch 93 TRAIN:  64%|██████▍   | 87/136 [00:21<00:10,  4.49it/s, epoch=93, grad_norm=156.020, loss=258.9439, lr=1.02e-03]
Epoch 93 TRAIN:  66%|██████▌   | 90/136 [00:21<00:10,  4.49it/s, epoch=93, grad_norm=156.020, loss=258.9439, lr=1.02e-03]
Epoch 93 TRAIN:  66%|██████▌   | 90/136 [00:22<00:10,  4.49it/s, epoch=93, grad_norm=106.293, loss=278.8741, lr=1.02e-03]
Epoch 93 TRAIN:  66%|██████▌   | 90/136 [00:22<00:10,  4.49it/s, epoch=93, grad_norm=144.401, loss=298.9525, lr=1.02e-03]
Epoch 93 TRAIN:  66%|██████▌   | 90/136 [00:22<00:10,  4.49it/s, epoch=93, grad_norm=124.277, loss=230.5469, lr=1.02e-03]
Epoch 93 TRAIN:  68%|██████▊   | 93/136 [00:22<00:09,  4.52it/s, epoch=93, grad_norm=124.277, loss=230.5469, lr=1.02e-03]
Epoch 93 TRAIN:  68%|██████▊   | 93/136 [00:22<00:09,  4.52it/s, epoch=93, grad_norm=130.833, loss=274.1516, lr=1.02e-03]
Epoch 93 TRAIN:  68%|██████▊   | 93/136 [00:22<00:09,  4.52it/s, epoch=93, grad_norm=109.946, loss=198.7243, lr=1.02e-03]
Epoch 93 TRAIN:  68%|██████▊   | 93/136 [00:23<00:09,  4.52it/s, epoch=93, grad_norm=213.666, loss=309.5441, lr=1.02e-03]
Epoch 93 TRAIN:  71%|███████   | 96/136 [00:23<00:08,  4.59it/s, epoch=93, grad_norm=213.666, loss=309.5441, lr=1.02e-03]
Epoch 93 TRAIN:  71%|███████   | 96/136 [00:23<00:08,  4.59it/s, epoch=93, grad_norm=125.271, loss=210.2329, lr=1.02e-03]
Epoch 93 TRAIN:  71%|███████   | 96/136 [00:23<00:08,  4.59it/s, epoch=93, grad_norm=102.632, loss=237.6824, lr=1.02e-03]
Epoch 93 TRAIN:  71%|███████   | 96/136 [00:23<00:08,  4.59it/s, epoch=93, grad_norm=168.212, loss=270.5090, lr=1.02e-03]
Epoch 93 TRAIN:  73%|███████▎  | 99/136 [00:23<00:08,  4.58it/s, epoch=93, grad_norm=168.212, loss=270.5090, lr=1.02e-03]
Epoch 93 TRAIN:  73%|███████▎  | 99/136 [00:23<00:08,  4.58it/s, epoch=93, grad_norm=180.722, loss=227.8615, lr=1.02e-03]2026-04-16 20:40:48,252 DEBUG TRAIN | steps/sec 4.202| Batch 93/100 loss 227.861511 loss_att 169.219040 loss_ctc 364.693939 th_accuracy 0.486702 lr 1.0226e-03 grad_norm 180.721985 rank 0

Epoch 93 TRAIN:  73%|███████▎  | 99/136 [00:24<00:08,  4.58it/s, epoch=93, grad_norm=116.837, loss=265.3530, lr=1.02e-03]
Epoch 93 TRAIN:  73%|███████▎  | 99/136 [00:24<00:08,  4.58it/s, epoch=93, grad_norm=144.877, loss=207.8956, lr=1.02e-03]
Epoch 93 TRAIN:  75%|███████▌  | 102/136 [00:24<00:07,  4.70it/s, epoch=93, grad_norm=144.877, loss=207.8956, lr=1.02e-03]
Epoch 93 TRAIN:  75%|███████▌  | 102/136 [00:24<00:07,  4.70it/s, epoch=93, grad_norm=112.752, loss=251.0789, lr=1.02e-03]
Epoch 93 TRAIN:  75%|███████▌  | 102/136 [00:24<00:07,  4.70it/s, epoch=93, grad_norm=176.172, loss=264.6446, lr=1.02e-03]
Epoch 93 TRAIN:  75%|███████▌  | 102/136 [00:24<00:07,  4.70it/s, epoch=93, grad_norm=94.793, loss=255.8458, lr=1.02e-03] 
Epoch 93 TRAIN:  77%|███████▋  | 105/136 [00:24<00:06,  4.77it/s, epoch=93, grad_norm=94.793, loss=255.8458, lr=1.02e-03]
Epoch 93 TRAIN:  77%|███████▋  | 105/136 [00:25<00:06,  4.77it/s, epoch=93, grad_norm=95.237, loss=220.9597, lr=1.02e-03]
Epoch 93 TRAIN:  77%|███████▋  | 105/136 [00:25<00:06,  4.77it/s, epoch=93, grad_norm=132.692, loss=238.4515, lr=1.02e-03]
Epoch 93 TRAIN:  77%|███████▋  | 105/136 [00:25<00:06,  4.77it/s, epoch=93, grad_norm=94.800, loss=251.5672, lr=1.02e-03] 
Epoch 93 TRAIN:  79%|███████▉  | 108/136 [00:25<00:05,  4.83it/s, epoch=93, grad_norm=94.800, loss=251.5672, lr=1.02e-03]
Epoch 93 TRAIN:  79%|███████▉  | 108/136 [00:25<00:05,  4.83it/s, epoch=93, grad_norm=91.262, loss=262.8941, lr=1.02e-03]
Epoch 93 TRAIN:  79%|███████▉  | 108/136 [00:25<00:05,  4.83it/s, epoch=93, grad_norm=143.706, loss=179.3544, lr=1.02e-03]
Epoch 93 TRAIN:  79%|███████▉  | 108/136 [00:26<00:05,  4.83it/s, epoch=93, grad_norm=137.657, loss=270.2379, lr=1.02e-03]
Epoch 93 TRAIN:  82%|████████▏ | 111/136 [00:26<00:05,  4.49it/s, epoch=93, grad_norm=137.657, loss=270.2379, lr=1.02e-03]
Epoch 93 TRAIN:  82%|████████▏ | 111/136 [00:26<00:05,  4.49it/s, epoch=93, grad_norm=189.713, loss=262.4533, lr=1.02e-03]
Epoch 93 TRAIN:  82%|████████▏ | 111/136 [00:26<00:05,  4.49it/s, epoch=93, grad_norm=128.739, loss=213.8436, lr=1.02e-03]
Epoch 93 TRAIN:  82%|████████▏ | 111/136 [00:26<00:05,  4.49it/s, epoch=93, grad_norm=139.613, loss=258.3679, lr=1.02e-03]
Epoch 93 TRAIN:  84%|████████▍ | 114/136 [00:26<00:04,  4.55it/s, epoch=93, grad_norm=139.613, loss=258.3679, lr=1.02e-03]
Epoch 93 TRAIN:  84%|████████▍ | 114/136 [00:27<00:04,  4.55it/s, epoch=93, grad_norm=173.663, loss=318.8799, lr=1.02e-03]
Epoch 93 TRAIN:  84%|████████▍ | 114/136 [00:27<00:04,  4.55it/s, epoch=93, grad_norm=213.378, loss=309.1933, lr=1.02e-03]
Epoch 93 TRAIN:  84%|████████▍ | 114/136 [00:27<00:04,  4.55it/s, epoch=93, grad_norm=165.371, loss=277.3937, lr=1.02e-03]
Epoch 93 TRAIN:  86%|████████▌ | 117/136 [00:27<00:04,  4.39it/s, epoch=93, grad_norm=165.371, loss=277.3937, lr=1.02e-03]
Epoch 93 TRAIN:  86%|████████▌ | 117/136 [00:27<00:04,  4.39it/s, epoch=93, grad_norm=111.143, loss=264.5527, lr=1.02e-03]
Epoch 93 TRAIN:  86%|████████▌ | 117/136 [00:28<00:04,  4.39it/s, epoch=93, grad_norm=203.039, loss=230.3473, lr=1.02e-03]
Epoch 93 TRAIN:  86%|████████▌ | 117/136 [00:28<00:04,  4.39it/s, epoch=93, grad_norm=246.333, loss=297.0133, lr=1.02e-03]
Epoch 93 TRAIN:  88%|████████▊ | 120/136 [00:28<00:03,  4.63it/s, epoch=93, grad_norm=246.333, loss=297.0133, lr=1.02e-03]
Epoch 93 TRAIN:  88%|████████▊ | 120/136 [00:28<00:03,  4.63it/s, epoch=93, grad_norm=264.198, loss=243.5314, lr=1.02e-03]
Epoch 93 TRAIN:  88%|████████▊ | 120/136 [00:28<00:03,  4.63it/s, epoch=93, grad_norm=196.883, loss=214.3786, lr=1.02e-03]
Epoch 93 TRAIN:  88%|████████▊ | 120/136 [00:28<00:03,  4.63it/s, epoch=93, grad_norm=511.296, loss=305.7424, lr=1.02e-03]
Epoch 93 TRAIN:  90%|█████████ | 123/136 [00:28<00:02,  4.61it/s, epoch=93, grad_norm=511.296, loss=305.7424, lr=1.02e-03]
Epoch 93 TRAIN:  90%|█████████ | 123/136 [00:29<00:02,  4.61it/s, epoch=93, grad_norm=134.854, loss=259.2362, lr=1.02e-03]
Epoch 93 TRAIN:  90%|█████████ | 123/136 [00:29<00:02,  4.61it/s, epoch=93, grad_norm=182.346, loss=281.0246, lr=1.02e-03]
Epoch 93 TRAIN:  90%|█████████ | 123/136 [00:29<00:02,  4.61it/s, epoch=93, grad_norm=213.348, loss=299.6103, lr=1.02e-03]
Epoch 93 TRAIN:  93%|█████████▎| 126/136 [00:29<00:02,  4.95it/s, epoch=93, grad_norm=213.348, loss=299.6103, lr=1.02e-03]
Epoch 93 TRAIN:  93%|█████████▎| 126/136 [00:29<00:02,  4.95it/s, epoch=93, grad_norm=237.684, loss=303.8124, lr=1.02e-03]
Epoch 93 TRAIN:  93%|█████████▎| 126/136 [00:29<00:02,  4.95it/s, epoch=93, grad_norm=90.549, loss=209.0161, lr=1.02e-03] 
Epoch 93 TRAIN:  93%|█████████▎| 126/136 [00:30<00:02,  4.95it/s, epoch=93, grad_norm=162.604, loss=245.7841, lr=1.02e-03]
Epoch 93 TRAIN:  95%|█████████▍| 129/136 [00:30<00:01,  4.98it/s, epoch=93, grad_norm=162.604, loss=245.7841, lr=1.02e-03]
Epoch 93 TRAIN:  95%|█████████▍| 129/136 [00:30<00:01,  4.98it/s, epoch=93, grad_norm=211.440, loss=315.7888, lr=1.03e-03]
Epoch 93 TRAIN:  95%|█████████▍| 129/136 [00:30<00:01,  4.98it/s, epoch=93, grad_norm=179.858, loss=236.7401, lr=1.03e-03]
Epoch 93 TRAIN:  95%|█████████▍| 129/136 [00:30<00:01,  4.98it/s, epoch=93, grad_norm=430.633, loss=304.8199, lr=1.03e-03]
Epoch 93 TRAIN:  95%|█████████▍| 129/136 [00:30<00:01,  4.98it/s, epoch=93, grad_norm=153.359, loss=227.7683, lr=1.03e-03]
Epoch 93 TRAIN:  98%|█████████▊| 133/136 [00:30<00:00,  5.19it/s, epoch=93, grad_norm=153.359, loss=227.7683, lr=1.03e-03]
Epoch 93 TRAIN:  98%|█████████▊| 133/136 [00:31<00:00,  5.19it/s, epoch=93, grad_norm=102.179, loss=214.3127, lr=1.03e-03]
Epoch 93 TRAIN:  98%|█████████▊| 133/136 [00:31<00:00,  5.19it/s, epoch=93, grad_norm=171.186, loss=272.2722, lr=1.03e-03]
Epoch 93 TRAIN:  98%|█████████▊| 133/136 [00:31<00:00,  5.19it/s, epoch=93, grad_norm=222.288, loss=274.8923, lr=1.03e-03]
Epoch 93 TRAIN: 100%|██████████| 136/136 [00:31<00:00,  4.94it/s, epoch=93, grad_norm=222.288, loss=274.8923, lr=1.03e-03]
Epoch 93 TRAIN: 100%|██████████| 136/136 [00:31<00:00,  4.33it/s, epoch=93, grad_norm=222.288, loss=274.8923, lr=1.03e-03]

CV:   0%|          | 0/6 [00:00<?, ?it/s]/home/rookie/Documents/batuka/projects/wenet/wenet/wenet/utils/train_utils.py:689: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  amp_autocast(enabled=dtype is not None,

CV:  17%|█▋        | 1/6 [00:00<00:03,  1.33it/s]
                                                 
2026-04-16 20:40:56,573 INFO Epoch 93 Step 12819 CV info lr 1.0255e-03 cv_loss 458.2207142223011 rank 0 acc 0.41755159695943195
2026-04-16 20:40:56,574 INFO [Rank 0] Checkpoint: save to checkpoint exp/conformer_mongolian/epoch_93.pt
2026-04-16 20:40:56,684 INFO Epoch 94 Step 12819 TRAIN info lr 1.0255e-03 rank 0
2026-04-16 20:40:56,705 INFO using accumulate grad, new batch size is 1 times larger than before

Epoch 94 TRAIN:   0%|          | 0/136 [00:00<?, ?it/s]
Epoch 94 TRAIN:   0%|          | 0/136 [00:03<?, ?it/s, epoch=94, grad_norm=433.335, loss=352.7417, lr=1.03e-03]
Epoch 94 TRAIN:   1%|          | 1/136 [00:03<08:27,  3.76s/it, epoch=94, grad_norm=433.335, loss=352.7417, lr=1.03e-03]
Epoch 94 TRAIN:   1%|          | 1/136 [00:04<08:27,  3.76s/it, epoch=94, grad_norm=366.121, loss=304.7285, lr=1.03e-03]
Epoch 94 TRAIN:   1%|          | 1/136 [00:04<08:27,  3.76s/it, epoch=94, grad_norm=460.474, loss=374.2618, lr=1.03e-03]
Epoch 94 TRAIN:   1%|          | 1/136 [00:04<08:27,  3.76s/it, epoch=94, grad_norm=163.561, loss=201.8779, lr=1.03e-03]
Epoch 94 TRAIN:   1%|          | 1/136 [00:04<08:27,  3.76s/it, epoch=94, grad_norm=297.189, loss=296.2647, lr=1.03e-03]
Epoch 94 TRAIN:   1%|          | 1/136 [00:04<08:27,  3.76s/it, epoch=94, grad_norm=248.656, loss=265.5735, lr=1.03e-03]
Epoch 94 TRAIN:   4%|▍         | 6/136 [00:04<01:12,  1.79it/s, epoch=94, grad_norm=248.656, loss=265.5735, lr=1.03e-03]
Epoch 94 TRAIN:   4%|▍         | 6/136 [00:04<01:12,  1.79it/s, epoch=94, grad_norm=220.551, loss=283.5865, lr=1.03e-03]
Epoch 94 TRAIN:   4%|▍         | 6/136 [00:04<01:12,  1.79it/s, epoch=94, grad_norm=179.601, loss=262.4957, lr=1.03e-03]
Epoch 94 TRAIN:   4%|▍         | 6/136 [00:04<01:12,  1.79it/s, epoch=94, grad_norm=208.075, loss=276.1903, lr=1.03e-03]
Epoch 94 TRAIN:   4%|▍         | 6/136 [00:04<01:12,  1.79it/s, epoch=94, grad_norm=93.889, loss=235.2535, lr=1.03e-03] 
Epoch 94 TRAIN:   4%|▍         | 6/136 [00:04<01:12,  1.79it/s, epoch=94, grad_norm=102.885, loss=229.4866, lr=1.03e-03]
Epoch 94 TRAIN:   8%|▊         | 11/136 [00:04<00:39,  3.14it/s, epoch=94, grad_norm=102.885, loss=229.4866, lr=1.03e-03]
Epoch 94 TRAIN:   8%|▊         | 11/136 [00:05<00:39,  3.14it/s, epoch=94, grad_norm=157.758, loss=298.9713, lr=1.03e-03]
Epoch 94 TRAIN:   8%|▊         | 11/136 [00:05<00:39,  3.14it/s, epoch=94, grad_norm=109.934, loss=268.9689, lr=1.03e-03]
Epoch 94 TRAIN:   8%|▊         | 11/136 [00:05<00:39,  3.14it/s, epoch=94, grad_norm=167.207, loss=270.9788, lr=1.03e-03]
Epoch 94 TRAIN:   8%|▊         | 11/136 [00:05<00:39,  3.14it/s, epoch=94, grad_norm=147.793, loss=339.7089, lr=1.03e-03]
Epoch 94 TRAIN:  11%|█         | 15/136 [00:05<00:31,  3.80it/s, epoch=94, grad_norm=147.793, loss=339.7089, lr=1.03e-03]
Epoch 94 TRAIN:  11%|█         | 15/136 [00:05<00:31,  3.80it/s, epoch=94, grad_norm=155.229, loss=235.0942, lr=1.03e-03]
Epoch 94 TRAIN:  11%|█         | 15/136 [00:06<00:31,  3.80it/s, epoch=94, grad_norm=129.932, loss=279.6162, lr=1.03e-03]
Epoch 94 TRAIN:  11%|█         | 15/136 [00:06<00:31,  3.80it/s, epoch=94, grad_norm=166.995, loss=261.8206, lr=1.03e-03]
Epoch 94 TRAIN:  11%|█         | 15/136 [00:06<00:31,  3.80it/s, epoch=94, grad_norm=108.960, loss=269.0894, lr=1.03e-03]
Epoch 94 TRAIN:  14%|█▍        | 19/136 [00:06<00:27,  4.19it/s, epoch=94, grad_norm=108.960, loss=269.0894, lr=1.03e-03]
Epoch 94 TRAIN:  14%|█▍        | 19/136 [00:06<00:27,  4.19it/s, epoch=94, grad_norm=112.027, loss=273.2045, lr=1.03e-03]
Epoch 94 TRAIN:  14%|█▍        | 19/136 [00:06<00:27,  4.19it/s, epoch=94, grad_norm=69.430, loss=272.7013, lr=1.03e-03] 
Epoch 94 TRAIN:  14%|█▍        | 19/136 [00:07<00:27,  4.19it/s, epoch=94, grad_norm=138.007, loss=222.9145, lr=1.03e-03]
Epoch 94 TRAIN:  16%|█▌        | 22/136 [00:07<00:25,  4.43it/s, epoch=94, grad_norm=138.007, loss=222.9145, lr=1.03e-03]
Epoch 94 TRAIN:  16%|█▌        | 22/136 [00:07<00:25,  4.43it/s, epoch=94, grad_norm=95.413, loss=265.5181, lr=1.03e-03] 
Epoch 94 TRAIN:  16%|█▌        | 22/136 [00:07<00:25,  4.43it/s, epoch=94, grad_norm=65.174, loss=255.3761, lr=1.03e-03]
Epoch 94 TRAIN:  16%|█▌        | 22/136 [00:07<00:25,  4.43it/s, epoch=94, grad_norm=131.873, loss=292.2816, lr=1.03e-03]
Epoch 94 TRAIN:  18%|█▊        | 25/136 [00:07<00:23,  4.64it/s, epoch=94, grad_norm=131.873, loss=292.2816, lr=1.03e-03]
Epoch 94 TRAIN:  18%|█▊        | 25/136 [00:07<00:23,  4.64it/s, epoch=94, grad_norm=119.926, loss=294.4722, lr=1.03e-03]
Epoch 94 TRAIN:  18%|█▊        | 25/136 [00:07<00:23,  4.64it/s, epoch=94, grad_norm=107.036, loss=290.0609, lr=1.03e-03]
Epoch 94 TRAIN:  18%|█▊        | 25/136 [00:08<00:23,  4.64it/s, epoch=94, grad_norm=112.328, loss=265.5856, lr=1.03e-03]
Epoch 94 TRAIN:  21%|██        | 28/136 [00:08<00:22,  4.86it/s, epoch=94, grad_norm=112.328, loss=265.5856, lr=1.03e-03]
Epoch 94 TRAIN:  21%|██        | 28/136 [00:08<00:22,  4.86it/s, epoch=94, grad_norm=84.984, loss=248.0136, lr=1.03e-03] 
Epoch 94 TRAIN:  21%|██        | 28/136 [00:08<00:22,  4.86it/s, epoch=94, grad_norm=92.845, loss=274.7549, lr=1.03e-03]
Epoch 94 TRAIN:  21%|██        | 28/136 [00:08<00:22,  4.86it/s, epoch=94, grad_norm=69.870, loss=245.6200, lr=1.03e-03]
Epoch 94 TRAIN:  23%|██▎       | 31/136 [00:08<00:22,  4.77it/s, epoch=94, grad_norm=69.870, loss=245.6200, lr=1.03e-03]
Epoch 94 TRAIN:  23%|██▎       | 31/136 [00:08<00:22,  4.77it/s, epoch=94, grad_norm=164.920, loss=329.9353, lr=1.03e-03]
Epoch 94 TRAIN:  23%|██▎       | 31/136 [00:09<00:22,  4.77it/s, epoch=94, grad_norm=156.251, loss=277.0897, lr=1.03e-03]
Epoch 94 TRAIN:  23%|██▎       | 31/136 [00:09<00:22,  4.77it/s, epoch=94, grad_norm=152.203, loss=252.6513, lr=1.03e-03]
Epoch 94 TRAIN:  25%|██▌       | 34/136 [00:09<00:21,  4.77it/s, epoch=94, grad_norm=152.203, loss=252.6513, lr=1.03e-03]
Epoch 94 TRAIN:  25%|██▌       | 34/136 [00:09<00:21,  4.77it/s, epoch=94, grad_norm=113.241, loss=302.3350, lr=1.03e-03]
Epoch 94 TRAIN:  25%|██▌       | 34/136 [00:09<00:21,  4.77it/s, epoch=94, grad_norm=84.428, loss=292.5106, lr=1.03e-03] 
Epoch 94 TRAIN:  25%|██▌       | 34/136 [00:10<00:21,  4.77it/s, epoch=94, grad_norm=161.150, loss=265.3582, lr=1.03e-03]
Epoch 94 TRAIN:  27%|██▋       | 37/136 [00:10<00:20,  4.81it/s, epoch=94, grad_norm=161.150, loss=265.3582, lr=1.03e-03]
Epoch 94 TRAIN:  27%|██▋       | 37/136 [00:10<00:20,  4.81it/s, epoch=94, grad_norm=142.679, loss=257.7599, lr=1.03e-03]
Epoch 94 TRAIN:  27%|██▋       | 37/136 [00:10<00:20,  4.81it/s, epoch=94, grad_norm=147.716, loss=236.6880, lr=1.03e-03]
Epoch 94 TRAIN:  27%|██▋       | 37/136 [00:10<00:20,  4.81it/s, epoch=94, grad_norm=132.411, loss=245.5517, lr=1.03e-03]
Epoch 94 TRAIN:  29%|██▉       | 40/136 [00:10<00:20,  4.78it/s, epoch=94, grad_norm=132.411, loss=245.5517, lr=1.03e-03]
Epoch 94 TRAIN:  29%|██▉       | 40/136 [00:10<00:20,  4.78it/s, epoch=94, grad_norm=124.440, loss=267.1895, lr=1.03e-03]
Epoch 94 TRAIN:  29%|██▉       | 40/136 [00:11<00:20,  4.78it/s, epoch=94, grad_norm=176.888, loss=275.4128, lr=1.03e-03]
Epoch 94 TRAIN:  29%|██▉       | 40/136 [00:11<00:20,  4.78it/s, epoch=94, grad_norm=89.940, loss=240.7432, lr=1.03e-03] 
Epoch 94 TRAIN:  32%|███▏      | 43/136 [00:11<00:19,  4.69it/s, epoch=94, grad_norm=89.940, loss=240.7432, lr=1.03e-03]
Epoch 94 TRAIN:  32%|███▏      | 43/136 [00:11<00:19,  4.69it/s, epoch=94, grad_norm=93.934, loss=283.2136, lr=1.03e-03]
Epoch 94 TRAIN:  32%|███▏      | 43/136 [00:11<00:19,  4.69it/s, epoch=94, grad_norm=89.659, loss=264.8768, lr=1.03e-03]
Epoch 94 TRAIN:  32%|███▏      | 43/136 [00:12<00:19,  4.69it/s, epoch=94, grad_norm=118.859, loss=291.6641, lr=1.03e-03]
Epoch 94 TRAIN:  34%|███▍      | 46/136 [00:12<00:19,  4.53it/s, epoch=94, grad_norm=118.859, loss=291.6641, lr=1.03e-03]
Epoch 94 TRAIN:  34%|███▍      | 46/136 [00:12<00:19,  4.53it/s, epoch=94, grad_norm=99.654, loss=264.3595, lr=1.03e-03] 
Epoch 94 TRAIN:  34%|███▍      | 46/136 [00:12<00:19,  4.53it/s, epoch=94, grad_norm=67.371, loss=198.7217, lr=1.03e-03]
Epoch 94 TRAIN:  34%|███▍      | 46/136 [00:12<00:19,  4.53it/s, epoch=94, grad_norm=83.489, loss=253.1009, lr=1.03e-03]
Epoch 94 TRAIN:  36%|███▌      | 49/136 [00:12<00:17,  4.84it/s, epoch=94, grad_norm=83.489, loss=253.1009, lr=1.03e-03]
Epoch 94 TRAIN:  36%|███▌      | 49/136 [00:12<00:17,  4.84it/s, epoch=94, grad_norm=67.098, loss=230.2482, lr=1.03e-03]
Epoch 94 TRAIN:  36%|███▌      | 49/136 [00:13<00:17,  4.84it/s, epoch=94, grad_norm=107.638, loss=319.4529, lr=1.03e-03]
Epoch 94 TRAIN:  36%|███▌      | 49/136 [00:13<00:17,  4.84it/s, epoch=94, grad_norm=110.860, loss=270.6307, lr=1.03e-03]
Epoch 94 TRAIN:  38%|███▊      | 52/136 [00:13<00:18,  4.56it/s, epoch=94, grad_norm=110.860, loss=270.6307, lr=1.03e-03]
Epoch 94 TRAIN:  38%|███▊      | 52/136 [00:13<00:18,  4.56it/s, epoch=94, grad_norm=91.342, loss=270.8504, lr=1.03e-03] 
Epoch 94 TRAIN:  38%|███▊      | 52/136 [00:13<00:18,  4.56it/s, epoch=94, grad_norm=128.056, loss=266.9905, lr=1.03e-03]
Epoch 94 TRAIN:  38%|███▊      | 52/136 [00:13<00:18,  4.56it/s, epoch=94, grad_norm=91.309, loss=263.0216, lr=1.03e-03] 
Epoch 94 TRAIN:  40%|████      | 55/136 [00:13<00:17,  4.54it/s, epoch=94, grad_norm=91.309, loss=263.0216, lr=1.03e-03]
Epoch 94 TRAIN:  40%|████      | 55/136 [00:14<00:17,  4.54it/s, epoch=94, grad_norm=106.765, loss=227.1534, lr=1.03e-03]
Epoch 94 TRAIN:  40%|████      | 55/136 [00:14<00:17,  4.54it/s, epoch=94, grad_norm=135.368, loss=258.2560, lr=1.03e-03]
Epoch 94 TRAIN:  40%|████      | 55/136 [00:14<00:17,  4.54it/s, epoch=94, grad_norm=96.788, loss=249.7497, lr=1.03e-03] 
Epoch 94 TRAIN:  43%|████▎     | 58/136 [00:14<00:16,  4.61it/s, epoch=94, grad_norm=96.788, loss=249.7497, lr=1.03e-03]
Epoch 94 TRAIN:  43%|████▎     | 58/136 [00:14<00:16,  4.61it/s, epoch=94, grad_norm=132.087, loss=303.6965, lr=1.03e-03]
Epoch 94 TRAIN:  43%|████▎     | 58/136 [00:15<00:16,  4.61it/s, epoch=94, grad_norm=115.327, loss=291.6952, lr=1.03e-03]
Epoch 94 TRAIN:  43%|████▎     | 58/136 [00:15<00:16,  4.61it/s, epoch=94, grad_norm=163.719, loss=293.9653, lr=1.03e-03]
Epoch 94 TRAIN:  45%|████▍     | 61/136 [00:15<00:16,  4.64it/s, epoch=94, grad_norm=163.719, loss=293.9653, lr=1.03e-03]
Epoch 94 TRAIN:  45%|████▍     | 61/136 [00:15<00:16,  4.64it/s, epoch=94, grad_norm=107.916, loss=288.8989, lr=1.03e-03]
Epoch 94 TRAIN:  45%|████▍     | 61/136 [00:15<00:16,  4.64it/s, epoch=94, grad_norm=102.527, loss=248.2543, lr=1.03e-03]
Epoch 94 TRAIN:  45%|████▍     | 61/136 [00:15<00:16,  4.64it/s, epoch=94, grad_norm=95.403, loss=262.5836, lr=1.03e-03] 
Epoch 94 TRAIN:  47%|████▋     | 64/136 [00:15<00:15,  4.58it/s, epoch=94, grad_norm=95.403, loss=262.5836, lr=1.03e-03]
Epoch 94 TRAIN:  47%|████▋     | 64/136 [00:16<00:15,  4.58it/s, epoch=94, grad_norm=214.112, loss=241.8625, lr=1.03e-03]
Epoch 94 TRAIN:  47%|████▋     | 64/136 [00:16<00:15,  4.58it/s, epoch=94, grad_norm=107.318, loss=235.2198, lr=1.03e-03]
Epoch 94 TRAIN:  47%|████▋     | 64/136 [00:16<00:15,  4.58it/s, epoch=94, grad_norm=178.054, loss=300.9048, lr=1.03e-03]
Epoch 94 TRAIN:  49%|████▉     | 67/136 [00:16<00:14,  4.72it/s, epoch=94, grad_norm=178.054, loss=300.9048, lr=1.03e-03]
Epoch 94 TRAIN:  49%|████▉     | 67/136 [00:16<00:14,  4.72it/s, epoch=94, grad_norm=146.802, loss=280.3405, lr=1.03e-03]
Epoch 94 TRAIN:  49%|████▉     | 67/136 [00:16<00:14,  4.72it/s, epoch=94, grad_norm=107.240, loss=236.6523, lr=1.03e-03]
Epoch 94 TRAIN:  49%|████▉     | 67/136 [00:17<00:14,  4.72it/s, epoch=94, grad_norm=261.903, loss=322.9041, lr=1.03e-03]
Epoch 94 TRAIN:  51%|█████▏    | 70/136 [00:17<00:13,  4.80it/s, epoch=94, grad_norm=261.903, loss=322.9041, lr=1.03e-03]
Epoch 94 TRAIN:  51%|█████▏    | 70/136 [00:17<00:13,  4.80it/s, epoch=94, grad_norm=94.625, loss=265.6831, lr=1.03e-03] 
Epoch 94 TRAIN:  51%|█████▏    | 70/136 [00:17<00:13,  4.80it/s, epoch=94, grad_norm=118.190, loss=284.5513, lr=1.03e-03]
Epoch 94 TRAIN:  51%|█████▏    | 70/136 [00:17<00:13,  4.80it/s, epoch=94, grad_norm=136.977, loss=244.3057, lr=1.03e-03]
Epoch 94 TRAIN:  54%|█████▎    | 73/136 [00:17<00:13,  4.81it/s, epoch=94, grad_norm=136.977, loss=244.3057, lr=1.03e-03]
Epoch 94 TRAIN:  54%|█████▎    | 73/136 [00:17<00:13,  4.81it/s, epoch=94, grad_norm=121.227, loss=218.2970, lr=1.03e-03]
Epoch 94 TRAIN:  54%|█████▎    | 73/136 [00:18<00:13,  4.81it/s, epoch=94, grad_norm=190.243, loss=229.1712, lr=1.03e-03]
Epoch 94 TRAIN:  54%|█████▎    | 73/136 [00:18<00:13,  4.81it/s, epoch=94, grad_norm=129.467, loss=264.6473, lr=1.03e-03]
Epoch 94 TRAIN:  56%|█████▌    | 76/136 [00:18<00:13,  4.42it/s, epoch=94, grad_norm=129.467, loss=264.6473, lr=1.03e-03]
Epoch 94 TRAIN:  56%|█████▌    | 76/136 [00:18<00:13,  4.42it/s, epoch=94, grad_norm=142.492, loss=272.2100, lr=1.03e-03]
Epoch 94 TRAIN:  56%|█████▌    | 76/136 [00:18<00:13,  4.42it/s, epoch=94, grad_norm=144.691, loss=278.0203, lr=1.03e-03]
Epoch 94 TRAIN:  56%|█████▌    | 76/136 [00:19<00:13,  4.42it/s, epoch=94, grad_norm=109.920, loss=216.2589, lr=1.03e-03]
Epoch 94 TRAIN:  58%|█████▊    | 79/136 [00:19<00:12,  4.66it/s, epoch=94, grad_norm=109.920, loss=216.2589, lr=1.03e-03]
Epoch 94 TRAIN:  58%|█████▊    | 79/136 [00:19<00:12,  4.66it/s, epoch=94, grad_norm=103.588, loss=251.7136, lr=1.03e-03]
Epoch 94 TRAIN:  58%|█████▊    | 79/136 [00:19<00:12,  4.66it/s, epoch=94, grad_norm=116.071, loss=296.6624, lr=1.03e-03]
Epoch 94 TRAIN:  58%|█████▊    | 79/136 [00:19<00:12,  4.66it/s, epoch=94, grad_norm=142.084, loss=303.7831, lr=1.03e-03]
Epoch 94 TRAIN:  60%|██████    | 82/136 [00:19<00:11,  4.60it/s, epoch=94, grad_norm=142.084, loss=303.7831, lr=1.03e-03]
Epoch 94 TRAIN:  60%|██████    | 82/136 [00:19<00:11,  4.60it/s, epoch=94, grad_norm=215.051, loss=218.1772, lr=1.03e-03]
Epoch 94 TRAIN:  60%|██████    | 82/136 [00:20<00:11,  4.60it/s, epoch=94, grad_norm=111.089, loss=260.5555, lr=1.03e-03]
Epoch 94 TRAIN:  60%|██████    | 82/136 [00:20<00:11,  4.60it/s, epoch=94, grad_norm=106.849, loss=269.8547, lr=1.03e-03]
Epoch 94 TRAIN:  62%|██████▎   | 85/136 [00:20<00:10,  4.67it/s, epoch=94, grad_norm=106.849, loss=269.8547, lr=1.03e-03]
Epoch 94 TRAIN:  62%|██████▎   | 85/136 [00:20<00:10,  4.67it/s, epoch=94, grad_norm=88.267, loss=240.0527, lr=1.03e-03] 
Epoch 94 TRAIN:  62%|██████▎   | 85/136 [00:20<00:10,  4.67it/s, epoch=94, grad_norm=89.062, loss=234.9320, lr=1.03e-03]
Epoch 94 TRAIN:  62%|██████▎   | 85/136 [00:21<00:10,  4.67it/s, epoch=94, grad_norm=153.657, loss=251.4849, lr=1.03e-03]
Epoch 94 TRAIN:  65%|██████▍   | 88/136 [00:21<00:10,  4.61it/s, epoch=94, grad_norm=153.657, loss=251.4849, lr=1.03e-03]
Epoch 94 TRAIN:  65%|██████▍   | 88/136 [00:21<00:10,  4.61it/s, epoch=94, grad_norm=198.153, loss=204.1233, lr=1.03e-03]
Epoch 94 TRAIN:  65%|██████▍   | 88/136 [00:21<00:10,  4.61it/s, epoch=94, grad_norm=125.448, loss=272.5350, lr=1.03e-03]
Epoch 94 TRAIN:  65%|██████▍   | 88/136 [00:21<00:10,  4.61it/s, epoch=94, grad_norm=187.045, loss=216.2652, lr=1.03e-03]
Epoch 94 TRAIN:  67%|██████▋   | 91/136 [00:21<00:09,  4.53it/s, epoch=94, grad_norm=187.045, loss=216.2652, lr=1.03e-03]
Epoch 94 TRAIN:  67%|██████▋   | 91/136 [00:21<00:09,  4.53it/s, epoch=94, grad_norm=91.577, loss=245.8240, lr=1.03e-03] 
Epoch 94 TRAIN:  67%|██████▋   | 91/136 [00:22<00:09,  4.53it/s, epoch=94, grad_norm=122.594, loss=263.8302, lr=1.03e-03]
Epoch 94 TRAIN:  67%|██████▋   | 91/136 [00:22<00:09,  4.53it/s, epoch=94, grad_norm=153.236, loss=281.3090, lr=1.03e-03]
Epoch 94 TRAIN:  69%|██████▉   | 94/136 [00:22<00:09,  4.50it/s, epoch=94, grad_norm=153.236, loss=281.3090, lr=1.03e-03]
Epoch 94 TRAIN:  69%|██████▉   | 94/136 [00:22<00:09,  4.50it/s, epoch=94, grad_norm=79.216, loss=193.0213, lr=1.03e-03] 
Epoch 94 TRAIN:  69%|██████▉   | 94/136 [00:22<00:09,  4.50it/s, epoch=94, grad_norm=130.217, loss=286.5633, lr=1.03e-03]
Epoch 94 TRAIN:  69%|██████▉   | 94/136 [00:23<00:09,  4.50it/s, epoch=94, grad_norm=145.663, loss=197.9588, lr=1.03e-03]
Epoch 94 TRAIN:  71%|███████▏  | 97/136 [00:23<00:08,  4.52it/s, epoch=94, grad_norm=145.663, loss=197.9588, lr=1.03e-03]
Epoch 94 TRAIN:  71%|███████▏  | 97/136 [00:23<00:08,  4.52it/s, epoch=94, grad_norm=96.628, loss=263.1322, lr=1.03e-03] 
Epoch 94 TRAIN:  71%|███████▏  | 97/136 [00:23<00:08,  4.52it/s, epoch=94, grad_norm=209.611, loss=237.9394, lr=1.03e-03]
Epoch 94 TRAIN:  71%|███████▏  | 97/136 [00:23<00:08,  4.52it/s, epoch=94, grad_norm=135.983, loss=235.7821, lr=1.03e-03]2026-04-16 20:41:20,453 DEBUG TRAIN | steps/sec 4.223| Batch 94/100 loss 235.782135 loss_att 168.151642 loss_ctc 393.586578 th_accuracy 0.514507 lr 1.0335e-03 grad_norm 135.983185 rank 0

Epoch 94 TRAIN:  74%|███████▎  | 100/136 [00:23<00:08,  4.49it/s, epoch=94, grad_norm=135.983, loss=235.7821, lr=1.03e-03]
Epoch 94 TRAIN:  74%|███████▎  | 100/136 [00:23<00:08,  4.49it/s, epoch=94, grad_norm=119.882, loss=235.9362, lr=1.03e-03]
Epoch 94 TRAIN:  74%|███████▎  | 100/136 [00:24<00:08,  4.49it/s, epoch=94, grad_norm=126.021, loss=174.4772, lr=1.03e-03]
Epoch 94 TRAIN:  74%|███████▎  | 100/136 [00:24<00:08,  4.49it/s, epoch=94, grad_norm=116.081, loss=221.9676, lr=1.03e-03]
Epoch 94 TRAIN:  76%|███████▌  | 103/136 [00:24<00:07,  4.46it/s, epoch=94, grad_norm=116.081, loss=221.9676, lr=1.03e-03]
Epoch 94 TRAIN:  76%|███████▌  | 103/136 [00:24<00:07,  4.46it/s, epoch=94, grad_norm=109.334, loss=239.9476, lr=1.03e-03]
Epoch 94 TRAIN:  76%|███████▌  | 103/136 [00:24<00:07,  4.46it/s, epoch=94, grad_norm=231.668, loss=278.1822, lr=1.03e-03]
Epoch 94 TRAIN:  76%|███████▌  | 103/136 [00:25<00:07,  4.46it/s, epoch=94, grad_norm=121.466, loss=202.1431, lr=1.03e-03]
Epoch 94 TRAIN:  78%|███████▊  | 106/136 [00:25<00:06,  4.39it/s, epoch=94, grad_norm=121.466, loss=202.1431, lr=1.03e-03]
Epoch 94 TRAIN:  78%|███████▊  | 106/136 [00:25<00:06,  4.39it/s, epoch=94, grad_norm=136.029, loss=264.9469, lr=1.03e-03]
Epoch 94 TRAIN:  78%|███████▊  | 106/136 [00:25<00:06,  4.39it/s, epoch=94, grad_norm=96.169, loss=202.2285, lr=1.03e-03] 
Epoch 94 TRAIN:  78%|███████▊  | 106/136 [00:25<00:06,  4.39it/s, epoch=94, grad_norm=117.082, loss=273.0834, lr=1.03e-03]
Epoch 94 TRAIN:  80%|████████  | 109/136 [00:25<00:06,  4.46it/s, epoch=94, grad_norm=117.082, loss=273.0834, lr=1.03e-03]
Epoch 94 TRAIN:  80%|████████  | 109/136 [00:26<00:06,  4.46it/s, epoch=94, grad_norm=93.505, loss=183.9061, lr=1.03e-03] 
Epoch 94 TRAIN:  80%|████████  | 109/136 [00:26<00:06,  4.46it/s, epoch=94, grad_norm=144.237, loss=215.8992, lr=1.03e-03]
Epoch 94 TRAIN:  80%|████████  | 109/136 [00:26<00:06,  4.46it/s, epoch=94, grad_norm=105.100, loss=215.5089, lr=1.03e-03]
Epoch 94 TRAIN:  82%|████████▏ | 112/136 [00:26<00:05,  4.48it/s, epoch=94, grad_norm=105.100, loss=215.5089, lr=1.03e-03]
Epoch 94 TRAIN:  82%|████████▏ | 112/136 [00:26<00:05,  4.48it/s, epoch=94, grad_norm=255.655, loss=260.3434, lr=1.03e-03]
Epoch 94 TRAIN:  82%|████████▏ | 112/136 [00:26<00:05,  4.48it/s, epoch=94, grad_norm=295.486, loss=318.5084, lr=1.03e-03]
Epoch 94 TRAIN:  82%|████████▏ | 112/136 [00:26<00:05,  4.48it/s, epoch=94, grad_norm=312.221, loss=253.3751, lr=1.03e-03]
Epoch 94 TRAIN:  85%|████████▍ | 115/136 [00:26<00:04,  4.79it/s, epoch=94, grad_norm=312.221, loss=253.3751, lr=1.03e-03]
Epoch 94 TRAIN:  85%|████████▍ | 115/136 [00:27<00:04,  4.79it/s, epoch=94, grad_norm=132.953, loss=190.5795, lr=1.03e-03]
Epoch 94 TRAIN:  85%|████████▍ | 115/136 [00:27<00:04,  4.79it/s, epoch=94, grad_norm=200.221, loss=306.4832, lr=1.03e-03]
Epoch 94 TRAIN:  85%|████████▍ | 115/136 [00:27<00:04,  4.79it/s, epoch=94, grad_norm=182.995, loss=241.0275, lr=1.03e-03]
Epoch 94 TRAIN:  87%|████████▋ | 118/136 [00:27<00:03,  4.88it/s, epoch=94, grad_norm=182.995, loss=241.0275, lr=1.03e-03]
Epoch 94 TRAIN:  87%|████████▋ | 118/136 [00:27<00:03,  4.88it/s, epoch=94, grad_norm=198.440, loss=250.7549, lr=1.04e-03]
Epoch 94 TRAIN:  87%|████████▋ | 118/136 [00:28<00:03,  4.88it/s, epoch=94, grad_norm=160.979, loss=280.1392, lr=1.04e-03]
Epoch 94 TRAIN:  87%|████████▋ | 118/136 [00:28<00:03,  4.88it/s, epoch=94, grad_norm=268.156, loss=298.3677, lr=1.04e-03]
Epoch 94 TRAIN:  89%|████████▉ | 121/136 [00:28<00:03,  4.78it/s, epoch=94, grad_norm=268.156, loss=298.3677, lr=1.04e-03]
Epoch 94 TRAIN:  89%|████████▉ | 121/136 [00:28<00:03,  4.78it/s, epoch=94, grad_norm=203.149, loss=264.6051, lr=1.04e-03]
Epoch 94 TRAIN:  89%|████████▉ | 121/136 [00:28<00:03,  4.78it/s, epoch=94, grad_norm=183.338, loss=343.3199, lr=1.04e-03]
Epoch 94 TRAIN:  89%|████████▉ | 121/136 [00:28<00:03,  4.78it/s, epoch=94, grad_norm=182.393, loss=230.3253, lr=1.04e-03]
Epoch 94 TRAIN:  91%|█████████ | 124/136 [00:28<00:02,  4.90it/s, epoch=94, grad_norm=182.393, loss=230.3253, lr=1.04e-03]
Epoch 94 TRAIN:  91%|█████████ | 124/136 [00:28<00:02,  4.90it/s, epoch=94, grad_norm=176.561, loss=299.6824, lr=1.04e-03]
Epoch 94 TRAIN:  91%|█████████ | 124/136 [00:29<00:02,  4.90it/s, epoch=94, grad_norm=188.834, loss=262.4621, lr=1.04e-03]
Epoch 94 TRAIN:  91%|█████████ | 124/136 [00:29<00:02,  4.90it/s, epoch=94, grad_norm=188.438, loss=188.1582, lr=1.04e-03]
Epoch 94 TRAIN:  93%|█████████▎| 127/136 [00:29<00:01,  4.99it/s, epoch=94, grad_norm=188.438, loss=188.1582, lr=1.04e-03]
Epoch 94 TRAIN:  93%|█████████▎| 127/136 [00:29<00:01,  4.99it/s, epoch=94, grad_norm=91.453, loss=231.8011, lr=1.04e-03] 
Epoch 94 TRAIN:  93%|█████████▎| 127/136 [00:29<00:01,  4.99it/s, epoch=94, grad_norm=255.094, loss=320.1041, lr=1.04e-03]
Epoch 94 TRAIN:  93%|█████████▎| 127/136 [00:29<00:01,  4.99it/s, epoch=94, grad_norm=268.941, loss=243.5708, lr=1.04e-03]
Epoch 94 TRAIN:  96%|█████████▌| 130/136 [00:29<00:01,  5.11it/s, epoch=94, grad_norm=268.941, loss=243.5708, lr=1.04e-03]
Epoch 94 TRAIN:  96%|█████████▌| 130/136 [00:30<00:01,  5.11it/s, epoch=94, grad_norm=149.681, loss=269.6738, lr=1.04e-03]
Epoch 94 TRAIN:  96%|█████████▌| 130/136 [00:30<00:01,  5.11it/s, epoch=94, grad_norm=124.129, loss=256.5986, lr=1.04e-03]
Epoch 94 TRAIN:  96%|█████████▌| 130/136 [00:30<00:01,  5.11it/s, epoch=94, grad_norm=337.715, loss=311.1314, lr=1.04e-03]
Epoch 94 TRAIN:  98%|█████████▊| 133/136 [00:30<00:00,  4.99it/s, epoch=94, grad_norm=337.715, loss=311.1314, lr=1.04e-03]
Epoch 94 TRAIN:  98%|█████████▊| 133/136 [00:30<00:00,  4.99it/s, epoch=94, grad_norm=310.589, loss=295.4080, lr=1.04e-03]
Epoch 94 TRAIN:  98%|█████████▊| 133/136 [00:30<00:00,  4.99it/s, epoch=94, grad_norm=148.826, loss=244.2620, lr=1.04e-03]
Epoch 94 TRAIN:  98%|█████████▊| 133/136 [00:31<00:00,  4.99it/s, epoch=94, grad_norm=157.607, loss=281.8702, lr=1.04e-03]
Epoch 94 TRAIN: 100%|██████████| 136/136 [00:31<00:00,  5.01it/s, epoch=94, grad_norm=157.607, loss=281.8702, lr=1.04e-03]
Epoch 94 TRAIN: 100%|██████████| 136/136 [00:31<00:00,  5.01it/s, epoch=94, grad_norm=189.733, loss=309.8047, lr=1.04e-03]
Epoch 94 TRAIN: 137it [00:31,  4.38it/s, epoch=94, grad_norm=189.733, loss=309.8047, lr=1.04e-03]                         

CV:   0%|          | 0/6 [00:00<?, ?it/s]/home/rookie/Documents/batuka/projects/wenet/wenet/wenet/utils/train_utils.py:689: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  amp_autocast(enabled=dtype is not None,

CV:  17%|█▋        | 1/6 [00:00<00:03,  1.32it/s]
                                                 
2026-04-16 20:41:28,939 INFO Epoch 94 Step 12956 CV info lr 1.0365e-03 cv_loss 465.5605579723011 rank 0 acc 0.4072280675172806
2026-04-16 20:41:28,941 INFO [Rank 0] Checkpoint: save to checkpoint exp/conformer_mongolian/epoch_94.pt
2026-04-16 20:41:29,055 INFO Epoch 95 Step 12956 TRAIN info lr 1.0365e-03 rank 0
2026-04-16 20:41:29,069 INFO using accumulate grad, new batch size is 1 times larger than before

Epoch 95 TRAIN:   0%|          | 0/137 [00:00<?, ?it/s]
Epoch 95 TRAIN:   0%|          | 0/137 [00:03<?, ?it/s, epoch=95, grad_norm=318.425, loss=285.5379, lr=1.04e-03]
Epoch 95 TRAIN:   1%|          | 1/137 [00:03<08:34,  3.79s/it, epoch=95, grad_norm=318.425, loss=285.5379, lr=1.04e-03]
Epoch 95 TRAIN:   1%|          | 1/137 [00:04<08:34,  3.79s/it, epoch=95, grad_norm=328.331, loss=273.1359, lr=1.04e-03]
Epoch 95 TRAIN:   1%|          | 1/137 [00:04<08:34,  3.79s/it, epoch=95, grad_norm=395.309, loss=352.0164, lr=1.04e-03]
Epoch 95 TRAIN:   2%|▏         | 3/137 [00:04<02:38,  1.18s/it, epoch=95, grad_norm=395.309, loss=352.0164, lr=1.04e-03]
Epoch 95 TRAIN:   2%|▏         | 3/137 [00:04<02:38,  1.18s/it, epoch=95, grad_norm=371.052, loss=340.6358, lr=1.04e-03]
Epoch 95 TRAIN:   2%|▏         | 3/137 [00:04<02:38,  1.18s/it, epoch=95, grad_norm=302.931, loss=314.9679, lr=1.04e-03]
Epoch 95 TRAIN:   2%|▏         | 3/137 [00:04<02:38,  1.18s/it, epoch=95, grad_norm=283.696, loss=308.2065, lr=1.04e-03]
Epoch 95 TRAIN:   2%|▏         | 3/137 [00:04<02:38,  1.18s/it, epoch=95, grad_norm=250.330, loss=317.9467, lr=1.04e-03]
Epoch 95 TRAIN:   2%|▏         | 3/137 [00:04<02:38,  1.18s/it, epoch=95, grad_norm=133.385, loss=214.1089, lr=1.04e-03]
Epoch 95 TRAIN:   2%|▏         | 3/137 [00:04<02:38,  1.18s/it, epoch=95, grad_norm=143.552, loss=304.2861, lr=1.04e-03]
Epoch 95 TRAIN:   7%|▋         | 9/137 [00:04<00:44,  2.89it/s, epoch=95, grad_norm=143.552, loss=304.2861, lr=1.04e-03]
Epoch 95 TRAIN:   7%|▋         | 9/137 [00:04<00:44,  2.89it/s, epoch=95, grad_norm=152.658, loss=304.9533, lr=1.04e-03]
Epoch 95 TRAIN:   7%|▋         | 9/137 [00:05<00:44,  2.89it/s, epoch=95, grad_norm=108.927, loss=244.0288, lr=1.04e-03]
Epoch 95 TRAIN:   7%|▋         | 9/137 [00:05<00:44,  2.89it/s, epoch=95, grad_norm=113.360, loss=291.7044, lr=1.04e-03]
Epoch 95 TRAIN:   7%|▋         | 9/137 [00:05<00:44,  2.89it/s, epoch=95, grad_norm=127.834, loss=309.4428, lr=1.04e-03]
Epoch 95 TRAIN:   9%|▉         | 13/137 [00:05<00:33,  3.70it/s, epoch=95, grad_norm=127.834, loss=309.4428, lr=1.04e-03]
Epoch 95 TRAIN:   9%|▉         | 13/137 [00:05<00:33,  3.70it/s, epoch=95, grad_norm=76.685, loss=256.5795, lr=1.04e-03] 
Epoch 95 TRAIN:   9%|▉         | 13/137 [00:05<00:33,  3.70it/s, epoch=95, grad_norm=106.902, loss=260.7162, lr=1.04e-03]
Epoch 95 TRAIN:   9%|▉         | 13/137 [00:05<00:33,  3.70it/s, epoch=95, grad_norm=90.115, loss=310.7074, lr=1.04e-03] 
Epoch 95 TRAIN:   9%|▉         | 13/137 [00:06<00:33,  3.70it/s, epoch=95, grad_norm=98.261, loss=258.7983, lr=1.04e-03]
Epoch 95 TRAIN:  12%|█▏        | 17/137 [00:06<00:28,  4.16it/s, epoch=95, grad_norm=98.261, loss=258.7983, lr=1.04e-03]
Epoch 95 TRAIN:  12%|█▏        | 17/137 [00:06<00:28,  4.16it/s, epoch=95, grad_norm=101.073, loss=286.8867, lr=1.04e-03]
Epoch 95 TRAIN:  12%|█▏        | 17/137 [00:06<00:28,  4.16it/s, epoch=95, grad_norm=111.404, loss=244.7259, lr=1.04e-03]
Epoch 95 TRAIN:  12%|█▏        | 17/137 [00:06<00:28,  4.16it/s, epoch=95, grad_norm=122.117, loss=298.9204, lr=1.04e-03]
Epoch 95 TRAIN:  12%|█▏        | 17/137 [00:06<00:28,  4.16it/s, epoch=95, grad_norm=81.939, loss=247.8708, lr=1.04e-03] 
Epoch 95 TRAIN:  15%|█▌        | 21/137 [00:06<00:24,  4.68it/s, epoch=95, grad_norm=81.939, loss=247.8708, lr=1.04e-03]
Epoch 95 TRAIN:  15%|█▌        | 21/137 [00:07<00:24,  4.68it/s, epoch=95, grad_norm=107.778, loss=253.1046, lr=1.04e-03]
Epoch 95 TRAIN:  15%|█▌        | 21/137 [00:07<00:24,  4.68it/s, epoch=95, grad_norm=85.199, loss=245.8253, lr=1.04e-03] 
Epoch 95 TRAIN:  15%|█▌        | 21/137 [00:07<00:24,  4.68it/s, epoch=95, grad_norm=116.572, loss=292.4243, lr=1.04e-03]
Epoch 95 TRAIN:  15%|█▌        | 21/137 [00:07<00:24,  4.68it/s, epoch=95, grad_norm=90.954, loss=242.1955, lr=1.04e-03] 
Epoch 95 TRAIN:  18%|█▊        | 25/137 [00:07<00:23,  4.75it/s, epoch=95, grad_norm=90.954, loss=242.1955, lr=1.04e-03]
Epoch 95 TRAIN:  18%|█▊        | 25/137 [00:07<00:23,  4.75it/s, epoch=95, grad_norm=123.499, loss=279.4786, lr=1.04e-03]
Epoch 95 TRAIN:  18%|█▊        | 25/137 [00:08<00:23,  4.75it/s, epoch=95, grad_norm=179.923, loss=294.9603, lr=1.04e-03]
Epoch 95 TRAIN:  18%|█▊        | 25/137 [00:08<00:23,  4.75it/s, epoch=95, grad_norm=86.424, loss=259.1873, lr=1.04e-03] 
Epoch 95 TRAIN:  20%|██        | 28/137 [00:08<00:22,  4.93it/s, epoch=95, grad_norm=86.424, loss=259.1873, lr=1.04e-03]
Epoch 95 TRAIN:  20%|██        | 28/137 [00:08<00:22,  4.93it/s, epoch=95, grad_norm=76.840, loss=251.5113, lr=1.04e-03]
Epoch 95 TRAIN:  20%|██        | 28/137 [00:08<00:22,  4.93it/s, epoch=95, grad_norm=95.013, loss=255.6702, lr=1.04e-03]
Epoch 95 TRAIN:  20%|██        | 28/137 [00:08<00:22,  4.93it/s, epoch=95, grad_norm=98.814, loss=288.6603, lr=1.04e-03]
Epoch 95 TRAIN:  23%|██▎       | 31/137 [00:08<00:21,  4.83it/s, epoch=95, grad_norm=98.814, loss=288.6603, lr=1.04e-03]
Epoch 95 TRAIN:  23%|██▎       | 31/137 [00:09<00:21,  4.83it/s, epoch=95, grad_norm=122.422, loss=234.8087, lr=1.04e-03]
Epoch 95 TRAIN:  23%|██▎       | 31/137 [00:09<00:21,  4.83it/s, epoch=95, grad_norm=98.611, loss=258.9688, lr=1.04e-03] 
Epoch 95 TRAIN:  23%|██▎       | 31/137 [00:09<00:21,  4.83it/s, epoch=95, grad_norm=116.512, loss=293.3013, lr=1.04e-03]
Epoch 95 TRAIN:  25%|██▍       | 34/137 [00:09<00:20,  5.04it/s, epoch=95, grad_norm=116.512, loss=293.3013, lr=1.04e-03]
Epoch 95 TRAIN:  25%|██▍       | 34/137 [00:09<00:20,  5.04it/s, epoch=95, grad_norm=84.302, loss=259.2543, lr=1.04e-03] 
Epoch 95 TRAIN:  25%|██▍       | 34/137 [00:09<00:20,  5.04it/s, epoch=95, grad_norm=128.601, loss=284.4180, lr=1.04e-03]
Epoch 95 TRAIN:  25%|██▍       | 34/137 [00:10<00:20,  5.04it/s, epoch=95, grad_norm=115.704, loss=283.0804, lr=1.04e-03]
Epoch 95 TRAIN:  27%|██▋       | 37/137 [00:10<00:20,  4.92it/s, epoch=95, grad_norm=115.704, loss=283.0804, lr=1.04e-03]
Epoch 95 TRAIN:  27%|██▋       | 37/137 [00:10<00:20,  4.92it/s, epoch=95, grad_norm=120.231, loss=286.6905, lr=1.04e-03]
Epoch 95 TRAIN:  27%|██▋       | 37/137 [00:10<00:20,  4.92it/s, epoch=95, grad_norm=131.152, loss=225.1512, lr=1.04e-03]
Epoch 95 TRAIN:  27%|██▋       | 37/137 [00:10<00:20,  4.92it/s, epoch=95, grad_norm=73.613, loss=260.4836, lr=1.04e-03] 
Epoch 95 TRAIN:  29%|██▉       | 40/137 [00:10<00:19,  4.88it/s, epoch=95, grad_norm=73.613, loss=260.4836, lr=1.04e-03]
Epoch 95 TRAIN:  29%|██▉       | 40/137 [00:10<00:19,  4.88it/s, epoch=95, grad_norm=140.841, loss=247.2773, lr=1.04e-03]
Epoch 95 TRAIN:  29%|██▉       | 40/137 [00:11<00:19,  4.88it/s, epoch=95, grad_norm=208.861, loss=225.0343, lr=1.04e-03]
Epoch 95 TRAIN:  29%|██▉       | 40/137 [00:11<00:19,  4.88it/s, epoch=95, grad_norm=121.075, loss=315.4476, lr=1.04e-03]
Epoch 95 TRAIN:  31%|███▏      | 43/137 [00:11<00:19,  4.77it/s, epoch=95, grad_norm=121.075, loss=315.4476, lr=1.04e-03]
Epoch 95 TRAIN:  31%|███▏      | 43/137 [00:11<00:19,  4.77it/s, epoch=95, grad_norm=84.428, loss=269.7532, lr=1.04e-03] 
Epoch 95 TRAIN:  31%|███▏      | 43/137 [00:11<00:19,  4.77it/s, epoch=95, grad_norm=94.641, loss=256.2807, lr=1.04e-03]
Epoch 95 TRAIN:  31%|███▏      | 43/137 [00:12<00:19,  4.77it/s, epoch=95, grad_norm=93.940, loss=243.9848, lr=1.04e-03]
Epoch 95 TRAIN:  34%|███▎      | 46/137 [00:12<00:19,  4.69it/s, epoch=95, grad_norm=93.940, loss=243.9848, lr=1.04e-03]
Epoch 95 TRAIN:  34%|███▎      | 46/137 [00:12<00:19,  4.69it/s, epoch=95, grad_norm=75.709, loss=254.9385, lr=1.04e-03]
Epoch 95 TRAIN:  34%|███▎      | 46/137 [00:12<00:19,  4.69it/s, epoch=95, grad_norm=103.663, loss=264.7041, lr=1.04e-03]
Epoch 95 TRAIN:  34%|███▎      | 46/137 [00:12<00:19,  4.69it/s, epoch=95, grad_norm=100.737, loss=232.4526, lr=1.04e-03]
Epoch 95 TRAIN:  36%|███▌      | 49/137 [00:12<00:18,  4.79it/s, epoch=95, grad_norm=100.737, loss=232.4526, lr=1.04e-03]
Epoch 95 TRAIN:  36%|███▌      | 49/137 [00:12<00:18,  4.79it/s, epoch=95, grad_norm=83.236, loss=261.4163, lr=1.04e-03] 
Epoch 95 TRAIN:  36%|███▌      | 49/137 [00:13<00:18,  4.79it/s, epoch=95, grad_norm=95.282, loss=281.0002, lr=1.04e-03]
Epoch 95 TRAIN:  36%|███▌      | 49/137 [00:13<00:18,  4.79it/s, epoch=95, grad_norm=87.552, loss=288.9631, lr=1.04e-03]
Epoch 95 TRAIN:  38%|███▊      | 52/137 [00:13<00:18,  4.51it/s, epoch=95, grad_norm=87.552, loss=288.9631, lr=1.04e-03]
Epoch 95 TRAIN:  38%|███▊      | 52/137 [00:13<00:18,  4.51it/s, epoch=95, grad_norm=77.068, loss=240.4417, lr=1.04e-03]
Epoch 95 TRAIN:  38%|███▊      | 52/137 [00:13<00:18,  4.51it/s, epoch=95, grad_norm=113.946, loss=254.1084, lr=1.04e-03]
Epoch 95 TRAIN:  38%|███▊      | 52/137 [00:14<00:18,  4.51it/s, epoch=95, grad_norm=124.402, loss=251.3883, lr=1.04e-03]
Epoch 95 TRAIN:  40%|████      | 55/137 [00:14<00:18,  4.48it/s, epoch=95, grad_norm=124.402, loss=251.3883, lr=1.04e-03]
Epoch 95 TRAIN:  40%|████      | 55/137 [00:14<00:18,  4.48it/s, epoch=95, grad_norm=123.391, loss=242.7563, lr=1.04e-03]
Epoch 95 TRAIN:  40%|████      | 55/137 [00:14<00:18,  4.48it/s, epoch=95, grad_norm=112.232, loss=303.9358, lr=1.04e-03]
Epoch 95 TRAIN:  40%|████      | 55/137 [00:14<00:18,  4.48it/s, epoch=95, grad_norm=129.040, loss=254.1196, lr=1.04e-03]
Epoch 95 TRAIN:  42%|████▏     | 58/137 [00:14<00:16,  4.73it/s, epoch=95, grad_norm=129.040, loss=254.1196, lr=1.04e-03]
Epoch 95 TRAIN:  42%|████▏     | 58/137 [00:14<00:16,  4.73it/s, epoch=95, grad_norm=89.759, loss=309.7570, lr=1.04e-03] 
Epoch 95 TRAIN:  42%|████▏     | 58/137 [00:15<00:16,  4.73it/s, epoch=95, grad_norm=110.923, loss=242.9215, lr=1.04e-03]
Epoch 95 TRAIN:  42%|████▏     | 58/137 [00:15<00:16,  4.73it/s, epoch=95, grad_norm=106.416, loss=220.0383, lr=1.04e-03]
Epoch 95 TRAIN:  45%|████▍     | 61/137 [00:15<00:15,  4.77it/s, epoch=95, grad_norm=106.416, loss=220.0383, lr=1.04e-03]
Epoch 95 TRAIN:  45%|████▍     | 61/137 [00:15<00:15,  4.77it/s, epoch=95, grad_norm=92.349, loss=260.5163, lr=1.04e-03] 
Epoch 95 TRAIN:  45%|████▍     | 61/137 [00:15<00:15,  4.77it/s, epoch=95, grad_norm=93.361, loss=237.1388, lr=1.04e-03]
Epoch 95 TRAIN:  45%|████▍     | 61/137 [00:15<00:15,  4.77it/s, epoch=95, grad_norm=149.396, loss=271.8246, lr=1.04e-03]
Epoch 95 TRAIN:  47%|████▋     | 64/137 [00:15<00:15,  4.70it/s, epoch=95, grad_norm=149.396, loss=271.8246, lr=1.04e-03]
Epoch 95 TRAIN:  47%|████▋     | 64/137 [00:16<00:15,  4.70it/s, epoch=95, grad_norm=161.973, loss=262.0450, lr=1.04e-03]
Epoch 95 TRAIN:  47%|████▋     | 64/137 [00:16<00:15,  4.70it/s, epoch=95, grad_norm=93.070, loss=243.6132, lr=1.04e-03] 
Epoch 95 TRAIN:  47%|████▋     | 64/137 [00:16<00:15,  4.70it/s, epoch=95, grad_norm=196.061, loss=286.0580, lr=1.04e-03]
Epoch 95 TRAIN:  49%|████▉     | 67/137 [00:16<00:14,  4.72it/s, epoch=95, grad_norm=196.061, loss=286.0580, lr=1.04e-03]
Epoch 95 TRAIN:  49%|████▉     | 67/137 [00:16<00:14,  4.72it/s, epoch=95, grad_norm=107.973, loss=244.5614, lr=1.04e-03]
Epoch 95 TRAIN:  49%|████▉     | 67/137 [00:17<00:14,  4.72it/s, epoch=95, grad_norm=94.646, loss=272.2519, lr=1.04e-03] 
Epoch 95 TRAIN:  49%|████▉     | 67/137 [00:17<00:14,  4.72it/s, epoch=95, grad_norm=157.443, loss=284.2513, lr=1.04e-03]
Epoch 95 TRAIN:  51%|█████     | 70/137 [00:17<00:14,  4.61it/s, epoch=95, grad_norm=157.443, loss=284.2513, lr=1.04e-03]
Epoch 95 TRAIN:  51%|█████     | 70/137 [00:17<00:14,  4.61it/s, epoch=95, grad_norm=151.112, loss=237.7337, lr=1.04e-03]
Epoch 95 TRAIN:  51%|█████     | 70/137 [00:17<00:14,  4.61it/s, epoch=95, grad_norm=115.798, loss=245.9795, lr=1.04e-03]
Epoch 95 TRAIN:  51%|█████     | 70/137 [00:17<00:14,  4.61it/s, epoch=95, grad_norm=119.535, loss=277.6502, lr=1.04e-03]
Epoch 95 TRAIN:  53%|█████▎    | 73/137 [00:17<00:13,  4.77it/s, epoch=95, grad_norm=119.535, loss=277.6502, lr=1.04e-03]
Epoch 95 TRAIN:  53%|█████▎    | 73/137 [00:17<00:13,  4.77it/s, epoch=95, grad_norm=147.914, loss=262.7125, lr=1.04e-03]
Epoch 95 TRAIN:  53%|█████▎    | 73/137 [00:18<00:13,  4.77it/s, epoch=95, grad_norm=86.977, loss=255.0307, lr=1.04e-03] 
Epoch 95 TRAIN:  53%|█████▎    | 73/137 [00:18<00:13,  4.77it/s, epoch=95, grad_norm=182.322, loss=215.8384, lr=1.04e-03]
Epoch 95 TRAIN:  55%|█████▌    | 76/137 [00:18<00:12,  4.89it/s, epoch=95, grad_norm=182.322, loss=215.8384, lr=1.04e-03]
Epoch 95 TRAIN:  55%|█████▌    | 76/137 [00:18<00:12,  4.89it/s, epoch=95, grad_norm=104.067, loss=287.5469, lr=1.04e-03]
Epoch 95 TRAIN:  55%|█████▌    | 76/137 [00:18<00:12,  4.89it/s, epoch=95, grad_norm=75.745, loss=240.0030, lr=1.04e-03] 
Epoch 95 TRAIN:  55%|█████▌    | 76/137 [00:19<00:12,  4.89it/s, epoch=95, grad_norm=93.068, loss=274.4756, lr=1.04e-03]
Epoch 95 TRAIN:  58%|█████▊    | 79/137 [00:19<00:12,  4.53it/s, epoch=95, grad_norm=93.068, loss=274.4756, lr=1.04e-03]
Epoch 95 TRAIN:  58%|█████▊    | 79/137 [00:19<00:12,  4.53it/s, epoch=95, grad_norm=81.157, loss=258.0321, lr=1.04e-03]
Epoch 95 TRAIN:  58%|█████▊    | 79/137 [00:19<00:12,  4.53it/s, epoch=95, grad_norm=106.702, loss=221.3717, lr=1.04e-03]
Epoch 95 TRAIN:  58%|█████▊    | 79/137 [00:19<00:12,  4.53it/s, epoch=95, grad_norm=123.346, loss=258.2722, lr=1.04e-03]
Epoch 95 TRAIN:  60%|█████▉    | 82/137 [00:19<00:11,  4.62it/s, epoch=95, grad_norm=123.346, loss=258.2722, lr=1.04e-03]
Epoch 95 TRAIN:  60%|█████▉    | 82/137 [00:19<00:11,  4.62it/s, epoch=95, grad_norm=145.010, loss=297.0088, lr=1.04e-03]
Epoch 95 TRAIN:  60%|█████▉    | 82/137 [00:20<00:11,  4.62it/s, epoch=95, grad_norm=92.651, loss=274.9000, lr=1.04e-03] 
Epoch 95 TRAIN:  60%|█████▉    | 82/137 [00:20<00:11,  4.62it/s, epoch=95, grad_norm=146.604, loss=198.3178, lr=1.04e-03]
Epoch 95 TRAIN:  62%|██████▏   | 85/137 [00:20<00:11,  4.71it/s, epoch=95, grad_norm=146.604, loss=198.3178, lr=1.04e-03]
Epoch 95 TRAIN:  62%|██████▏   | 85/137 [00:20<00:11,  4.71it/s, epoch=95, grad_norm=98.553, loss=252.5617, lr=1.04e-03] 
Epoch 95 TRAIN:  62%|██████▏   | 85/137 [00:20<00:11,  4.71it/s, epoch=95, grad_norm=115.588, loss=244.0132, lr=1.04e-03]
Epoch 95 TRAIN:  62%|██████▏   | 85/137 [00:20<00:11,  4.71it/s, epoch=95, grad_norm=112.271, loss=262.5139, lr=1.04e-03]
Epoch 95 TRAIN:  64%|██████▍   | 88/137 [00:20<00:10,  4.76it/s, epoch=95, grad_norm=112.271, loss=262.5139, lr=1.04e-03]
Epoch 95 TRAIN:  64%|██████▍   | 88/137 [00:21<00:10,  4.76it/s, epoch=95, grad_norm=84.990, loss=263.3093, lr=1.04e-03] 
Epoch 95 TRAIN:  64%|██████▍   | 88/137 [00:21<00:10,  4.76it/s, epoch=95, grad_norm=162.095, loss=204.8414, lr=1.04e-03]
Epoch 95 TRAIN:  64%|██████▍   | 88/137 [00:21<00:10,  4.76it/s, epoch=95, grad_norm=287.846, loss=279.1506, lr=1.04e-03]
Epoch 95 TRAIN:  66%|██████▋   | 91/137 [00:21<00:09,  4.70it/s, epoch=95, grad_norm=287.846, loss=279.1506, lr=1.04e-03]
Epoch 95 TRAIN:  66%|██████▋   | 91/137 [00:21<00:09,  4.70it/s, epoch=95, grad_norm=274.602, loss=157.6665, lr=1.04e-03]
Epoch 95 TRAIN:  66%|██████▋   | 91/137 [00:22<00:09,  4.70it/s, epoch=95, grad_norm=153.137, loss=206.3376, lr=1.04e-03]
Epoch 95 TRAIN:  66%|██████▋   | 91/137 [00:22<00:09,  4.70it/s, epoch=95, grad_norm=267.852, loss=274.2728, lr=1.04e-03]
Epoch 95 TRAIN:  69%|██████▊   | 94/137 [00:22<00:09,  4.60it/s, epoch=95, grad_norm=267.852, loss=274.2728, lr=1.04e-03]
Epoch 95 TRAIN:  69%|██████▊   | 94/137 [00:22<00:09,  4.60it/s, epoch=95, grad_norm=187.891, loss=248.9698, lr=1.04e-03]
Epoch 95 TRAIN:  69%|██████▊   | 94/137 [00:22<00:09,  4.60it/s, epoch=95, grad_norm=330.739, loss=328.5175, lr=1.04e-03]
Epoch 95 TRAIN:  69%|██████▊   | 94/137 [00:22<00:09,  4.60it/s, epoch=95, grad_norm=95.210, loss=197.9552, lr=1.04e-03] 
Epoch 95 TRAIN:  71%|███████   | 97/137 [00:22<00:08,  4.85it/s, epoch=95, grad_norm=95.210, loss=197.9552, lr=1.04e-03]
Epoch 95 TRAIN:  71%|███████   | 97/137 [00:23<00:08,  4.85it/s, epoch=95, grad_norm=148.890, loss=228.3346, lr=1.04e-03]
Epoch 95 TRAIN:  71%|███████   | 97/137 [00:23<00:08,  4.85it/s, epoch=95, grad_norm=314.604, loss=302.2986, lr=1.04e-03]
Epoch 95 TRAIN:  71%|███████   | 97/137 [00:23<00:08,  4.85it/s, epoch=95, grad_norm=105.431, loss=252.6555, lr=1.04e-03]2026-04-16 20:41:52,709 DEBUG TRAIN | steps/sec 4.247| Batch 95/100 loss 252.655457 loss_att 191.800385 loss_ctc 394.650574 th_accuracy 0.501193 lr 1.0445e-03 grad_norm 105.431175 rank 0

Epoch 95 TRAIN:  73%|███████▎  | 100/137 [00:23<00:08,  4.54it/s, epoch=95, grad_norm=105.431, loss=252.6555, lr=1.04e-03]
Epoch 95 TRAIN:  73%|███████▎  | 100/137 [00:23<00:08,  4.54it/s, epoch=95, grad_norm=147.897, loss=263.0829, lr=1.04e-03]
Epoch 95 TRAIN:  73%|███████▎  | 100/137 [00:24<00:08,  4.54it/s, epoch=95, grad_norm=114.569, loss=182.1666, lr=1.04e-03]
Epoch 95 TRAIN:  73%|███████▎  | 100/137 [00:24<00:08,  4.54it/s, epoch=95, grad_norm=197.945, loss=261.7779, lr=1.04e-03]
Epoch 95 TRAIN:  75%|███████▌  | 103/137 [00:24<00:07,  4.44it/s, epoch=95, grad_norm=197.945, loss=261.7779, lr=1.04e-03]
Epoch 95 TRAIN:  75%|███████▌  | 103/137 [00:24<00:07,  4.44it/s, epoch=95, grad_norm=90.169, loss=230.4713, lr=1.04e-03] 
Epoch 95 TRAIN:  75%|███████▌  | 103/137 [00:24<00:07,  4.44it/s, epoch=95, grad_norm=211.442, loss=347.3461, lr=1.04e-03]
Epoch 95 TRAIN:  75%|███████▌  | 103/137 [00:24<00:07,  4.44it/s, epoch=95, grad_norm=91.105, loss=245.2110, lr=1.04e-03] 
Epoch 95 TRAIN:  75%|███████▌  | 103/137 [00:25<00:07,  4.44it/s, epoch=95, grad_norm=115.698, loss=290.0551, lr=1.05e-03]
Epoch 95 TRAIN:  78%|███████▊  | 107/137 [00:25<00:06,  4.50it/s, epoch=95, grad_norm=115.698, loss=290.0551, lr=1.05e-03]
Epoch 95 TRAIN:  78%|███████▊  | 107/137 [00:25<00:06,  4.50it/s, epoch=95, grad_norm=225.441, loss=240.9005, lr=1.05e-03]
Epoch 95 TRAIN:  78%|███████▊  | 107/137 [00:25<00:06,  4.50it/s, epoch=95, grad_norm=182.446, loss=260.4876, lr=1.05e-03]
Epoch 95 TRAIN:  78%|███████▊  | 107/137 [00:25<00:06,  4.50it/s, epoch=95, grad_norm=125.270, loss=206.4864, lr=1.05e-03]
Epoch 95 TRAIN:  78%|███████▊  | 107/137 [00:25<00:06,  4.50it/s, epoch=95, grad_norm=88.229, loss=232.4910, lr=1.05e-03] 
Epoch 95 TRAIN:  81%|████████  | 111/137 [00:25<00:05,  4.92it/s, epoch=95, grad_norm=88.229, loss=232.4910, lr=1.05e-03]
Epoch 95 TRAIN:  81%|████████  | 111/137 [00:25<00:05,  4.92it/s, epoch=95, grad_norm=191.021, loss=302.8685, lr=1.05e-03]
Epoch 95 TRAIN:  81%|████████  | 111/137 [00:26<00:05,  4.92it/s, epoch=95, grad_norm=83.156, loss=236.2781, lr=1.05e-03] 
Epoch 95 TRAIN:  81%|████████  | 111/137 [00:26<00:05,  4.92it/s, epoch=95, grad_norm=358.235, loss=271.6303, lr=1.05e-03]
Epoch 95 TRAIN:  83%|████████▎ | 114/137 [00:26<00:04,  5.01it/s, epoch=95, grad_norm=358.235, loss=271.6303, lr=1.05e-03]
Epoch 95 TRAIN:  83%|████████▎ | 114/137 [00:26<00:04,  5.01it/s, epoch=95, grad_norm=385.532, loss=323.9351, lr=1.05e-03]
Epoch 95 TRAIN:  83%|████████▎ | 114/137 [00:26<00:04,  5.01it/s, epoch=95, grad_norm=161.592, loss=281.1683, lr=1.05e-03]
Epoch 95 TRAIN:  83%|████████▎ | 114/137 [00:27<00:04,  5.01it/s, epoch=95, grad_norm=116.203, loss=193.4098, lr=1.05e-03]
Epoch 95 TRAIN:  85%|████████▌ | 117/137 [00:27<00:04,  4.88it/s, epoch=95, grad_norm=116.203, loss=193.4098, lr=1.05e-03]
Epoch 95 TRAIN:  85%|████████▌ | 117/137 [00:27<00:04,  4.88it/s, epoch=95, grad_norm=158.065, loss=242.9662, lr=1.05e-03]
Epoch 95 TRAIN:  85%|████████▌ | 117/137 [00:27<00:04,  4.88it/s, epoch=95, grad_norm=163.688, loss=252.7151, lr=1.05e-03]
Epoch 95 TRAIN:  85%|████████▌ | 117/137 [00:27<00:04,  4.88it/s, epoch=95, grad_norm=143.792, loss=248.7307, lr=1.05e-03]
Epoch 95 TRAIN:  88%|████████▊ | 120/137 [00:27<00:03,  4.61it/s, epoch=95, grad_norm=143.792, loss=248.7307, lr=1.05e-03]
Epoch 95 TRAIN:  88%|████████▊ | 120/137 [00:28<00:03,  4.61it/s, epoch=95, grad_norm=149.523, loss=308.1394, lr=1.05e-03]
Epoch 95 TRAIN:  88%|████████▊ | 120/137 [00:28<00:03,  4.61it/s, epoch=95, grad_norm=226.724, loss=324.3567, lr=1.05e-03]
Epoch 95 TRAIN:  88%|████████▊ | 120/137 [00:28<00:03,  4.61it/s, epoch=95, grad_norm=116.714, loss=239.0072, lr=1.05e-03]
Epoch 95 TRAIN:  90%|████████▉ | 123/137 [00:28<00:02,  4.90it/s, epoch=95, grad_norm=116.714, loss=239.0072, lr=1.05e-03]
Epoch 95 TRAIN:  90%|████████▉ | 123/137 [00:28<00:02,  4.90it/s, epoch=95, grad_norm=133.935, loss=225.4515, lr=1.05e-03]
Epoch 95 TRAIN:  90%|████████▉ | 123/137 [00:28<00:02,  4.90it/s, epoch=95, grad_norm=106.635, loss=249.0665, lr=1.05e-03]
Epoch 95 TRAIN:  90%|████████▉ | 123/137 [00:29<00:02,  4.90it/s, epoch=95, grad_norm=217.173, loss=284.4564, lr=1.05e-03]
Epoch 95 TRAIN:  92%|█████████▏| 126/137 [00:29<00:02,  4.72it/s, epoch=95, grad_norm=217.173, loss=284.4564, lr=1.05e-03]
Epoch 95 TRAIN:  92%|█████████▏| 126/137 [00:29<00:02,  4.72it/s, epoch=95, grad_norm=234.835, loss=271.1597, lr=1.05e-03]
Epoch 95 TRAIN:  92%|█████████▏| 126/137 [00:29<00:02,  4.72it/s, epoch=95, grad_norm=118.478, loss=234.9747, lr=1.05e-03]
Epoch 95 TRAIN:  92%|█████████▏| 126/137 [00:29<00:02,  4.72it/s, epoch=95, grad_norm=117.379, loss=229.5934, lr=1.05e-03]
Epoch 95 TRAIN:  94%|█████████▍| 129/137 [00:29<00:01,  4.70it/s, epoch=95, grad_norm=117.379, loss=229.5934, lr=1.05e-03]
Epoch 95 TRAIN:  94%|█████████▍| 129/137 [00:29<00:01,  4.70it/s, epoch=95, grad_norm=165.696, loss=276.1472, lr=1.05e-03]
Epoch 95 TRAIN:  94%|█████████▍| 129/137 [00:30<00:01,  4.70it/s, epoch=95, grad_norm=214.546, loss=294.1028, lr=1.05e-03]
Epoch 95 TRAIN:  94%|█████████▍| 129/137 [00:30<00:01,  4.70it/s, epoch=95, grad_norm=172.932, loss=244.4095, lr=1.05e-03]
Epoch 95 TRAIN:  94%|█████████▍| 129/137 [00:30<00:01,  4.70it/s, epoch=95, grad_norm=165.891, loss=249.5393, lr=1.05e-03]
Epoch 95 TRAIN:  97%|█████████▋| 133/137 [00:30<00:00,  5.20it/s, epoch=95, grad_norm=165.891, loss=249.5393, lr=1.05e-03]
Epoch 95 TRAIN:  97%|█████████▋| 133/137 [00:30<00:00,  5.20it/s, epoch=95, grad_norm=150.086, loss=274.1396, lr=1.05e-03]
Epoch 95 TRAIN:  97%|█████████▋| 133/137 [00:30<00:00,  5.20it/s, epoch=95, grad_norm=160.277, loss=226.0581, lr=1.05e-03]
Epoch 95 TRAIN:  97%|█████████▋| 133/137 [00:31<00:00,  5.20it/s, epoch=95, grad_norm=146.528, loss=245.3170, lr=1.05e-03]
Epoch 95 TRAIN:  99%|█████████▉| 136/137 [00:31<00:00,  4.81it/s, epoch=95, grad_norm=146.528, loss=245.3170, lr=1.05e-03]
Epoch 95 TRAIN:  99%|█████████▉| 136/137 [00:31<00:00,  4.81it/s, epoch=95, grad_norm=271.414, loss=319.8500, lr=1.05e-03]
Epoch 95 TRAIN:  99%|█████████▉| 136/137 [00:31<00:00,  4.81it/s, epoch=95, grad_norm=190.460, loss=298.0015, lr=1.05e-03]
Epoch 95 TRAIN:  99%|█████████▉| 136/137 [00:31<00:00,  4.81it/s, epoch=95, grad_norm=213.240, loss=338.5220, lr=1.05e-03]
Epoch 95 TRAIN: 139it [00:31,  4.42it/s, epoch=95, grad_norm=213.240, loss=338.5220, lr=1.05e-03]                         

CV:   0%|          | 0/6 [00:00<?, ?it/s]/home/rookie/Documents/batuka/projects/wenet/wenet/wenet/utils/train_utils.py:689: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  amp_autocast(enabled=dtype is not None,

CV:  17%|█▋        | 1/6 [00:00<00:03,  1.33it/s]
                                                 
2026-04-16 20:42:01,381 INFO Epoch 95 Step 13095 CV info lr 1.0476e-03 cv_loss 450.41251997514206 rank 0 acc 0.41493453582127887
2026-04-16 20:42:01,383 INFO [Rank 0] Checkpoint: save to checkpoint exp/conformer_mongolian/epoch_95.pt
2026-04-16 20:42:01,495 INFO Epoch 96 Step 13095 TRAIN info lr 1.0476e-03 rank 0
2026-04-16 20:42:01,515 INFO using accumulate grad, new batch size is 1 times larger than before

Epoch 96 TRAIN:   0%|          | 0/139 [00:00<?, ?it/s]
Epoch 96 TRAIN:   0%|          | 0/139 [00:03<?, ?it/s, epoch=96, grad_norm=358.169, loss=277.8801, lr=1.05e-03]
Epoch 96 TRAIN:   1%|          | 1/139 [00:03<09:07,  3.97s/it, epoch=96, grad_norm=358.169, loss=277.8801, lr=1.05e-03]
Epoch 96 TRAIN:   1%|          | 1/139 [00:04<09:07,  3.97s/it, epoch=96, grad_norm=411.862, loss=342.1287, lr=1.05e-03]
Epoch 96 TRAIN:   1%|          | 1/139 [00:04<09:07,  3.97s/it, epoch=96, grad_norm=247.661, loss=249.6132, lr=1.05e-03]
Epoch 96 TRAIN:   1%|          | 1/139 [00:04<09:07,  3.97s/it, epoch=96, grad_norm=233.146, loss=264.8498, lr=1.05e-03]
Epoch 96 TRAIN:   1%|          | 1/139 [00:04<09:07,  3.97s/it, epoch=96, grad_norm=212.501, loss=262.4190, lr=1.05e-03]
Epoch 96 TRAIN:   1%|          | 1/139 [00:04<09:07,  3.97s/it, epoch=96, grad_norm=368.958, loss=362.6684, lr=1.05e-03]
Epoch 96 TRAIN:   1%|          | 1/139 [00:04<09:07,  3.97s/it, epoch=96, grad_norm=183.816, loss=265.3704, lr=1.05e-03]
Epoch 96 TRAIN:   1%|          | 1/139 [00:04<09:07,  3.97s/it, epoch=96, grad_norm=178.835, loss=287.0976, lr=1.05e-03]
Epoch 96 TRAIN:   6%|▌         | 8/139 [00:04<00:55,  2.35it/s, epoch=96, grad_norm=178.835, loss=287.0976, lr=1.05e-03]
Epoch 96 TRAIN:   6%|▌         | 8/139 [00:04<00:55,  2.35it/s, epoch=96, grad_norm=118.228, loss=277.0994, lr=1.05e-03]
Epoch 96 TRAIN:   6%|▌         | 8/139 [00:04<00:55,  2.35it/s, epoch=96, grad_norm=154.908, loss=300.1428, lr=1.05e-03]
Epoch 96 TRAIN:   6%|▌         | 8/139 [00:04<00:55,  2.35it/s, epoch=96, grad_norm=103.596, loss=263.6595, lr=1.05e-03]
Epoch 96 TRAIN:   6%|▌         | 8/139 [00:05<00:55,  2.35it/s, epoch=96, grad_norm=80.469, loss=317.4379, lr=1.05e-03] 
Epoch 96 TRAIN:   6%|▌         | 8/139 [00:05<00:55,  2.35it/s, epoch=96, grad_norm=107.972, loss=278.9854, lr=1.05e-03]
Epoch 96 TRAIN:   9%|▉         | 13/139 [00:05<00:37,  3.33it/s, epoch=96, grad_norm=107.972, loss=278.9854, lr=1.05e-03]
Epoch 96 TRAIN:   9%|▉         | 13/139 [00:05<00:37,  3.33it/s, epoch=96, grad_norm=141.332, loss=287.1197, lr=1.05e-03]
Epoch 96 TRAIN:   9%|▉         | 13/139 [00:05<00:37,  3.33it/s, epoch=96, grad_norm=97.920, loss=285.5930, lr=1.05e-03] 
Epoch 96 TRAIN:   9%|▉         | 13/139 [00:05<00:37,  3.33it/s, epoch=96, grad_norm=98.409, loss=270.6872, lr=1.05e-03]
Epoch 96 TRAIN:   9%|▉         | 13/139 [00:05<00:37,  3.33it/s, epoch=96, grad_norm=143.160, loss=293.8928, lr=1.05e-03]
Epoch 96 TRAIN:  12%|█▏        | 17/139 [00:05<00:30,  3.97it/s, epoch=96, grad_norm=143.160, loss=293.8928, lr=1.05e-03]
Epoch 96 TRAIN:  12%|█▏        | 17/139 [00:06<00:30,  3.97it/s, epoch=96, grad_norm=105.354, loss=277.4470, lr=1.05e-03]
Epoch 96 TRAIN:  12%|█▏        | 17/139 [00:06<00:30,  3.97it/s, epoch=96, grad_norm=106.918, loss=272.0199, lr=1.05e-03]
Epoch 96 TRAIN:  12%|█▏        | 17/139 [00:06<00:30,  3.97it/s, epoch=96, grad_norm=125.891, loss=237.1867, lr=1.05e-03]
Epoch 96 TRAIN:  12%|█▏        | 17/139 [00:06<00:30,  3.97it/s, epoch=96, grad_norm=92.979, loss=256.9305, lr=1.05e-03] 
Epoch 96 TRAIN:  15%|█▌        | 21/139 [00:06<00:27,  4.27it/s, epoch=96, grad_norm=92.979, loss=256.9305, lr=1.05e-03]
Epoch 96 TRAIN:  15%|█▌        | 21/139 [00:06<00:27,  4.27it/s, epoch=96, grad_norm=181.225, loss=316.6167, lr=1.05e-03]
Epoch 96 TRAIN:  15%|█▌        | 21/139 [00:07<00:27,  4.27it/s, epoch=96, grad_norm=175.764, loss=281.6980, lr=1.05e-03]
Epoch 96 TRAIN:  15%|█▌        | 21/139 [00:07<00:27,  4.27it/s, epoch=96, grad_norm=100.722, loss=254.2366, lr=1.05e-03]
Epoch 96 TRAIN:  15%|█▌        | 21/139 [00:07<00:27,  4.27it/s, epoch=96, grad_norm=98.750, loss=244.1005, lr=1.05e-03] 
Epoch 96 TRAIN:  18%|█▊        | 25/139 [00:07<00:25,  4.50it/s, epoch=96, grad_norm=98.750, loss=244.1005, lr=1.05e-03]
Epoch 96 TRAIN:  18%|█▊        | 25/139 [00:07<00:25,  4.50it/s, epoch=96, grad_norm=166.136, loss=264.3319, lr=1.05e-03]
Epoch 96 TRAIN:  18%|█▊        | 25/139 [00:07<00:25,  4.50it/s, epoch=96, grad_norm=89.264, loss=261.2039, lr=1.05e-03] 
Epoch 96 TRAIN:  18%|█▊        | 25/139 [00:08<00:25,  4.50it/s, epoch=96, grad_norm=59.835, loss=226.7568, lr=1.05e-03]
Epoch 96 TRAIN:  20%|██        | 28/139 [00:08<00:24,  4.54it/s, epoch=96, grad_norm=59.835, loss=226.7568, lr=1.05e-03]
Epoch 96 TRAIN:  20%|██        | 28/139 [00:08<00:24,  4.54it/s, epoch=96, grad_norm=153.573, loss=294.4555, lr=1.05e-03]
Epoch 96 TRAIN:  20%|██        | 28/139 [00:08<00:24,  4.54it/s, epoch=96, grad_norm=123.762, loss=287.9268, lr=1.05e-03]
Epoch 96 TRAIN:  20%|██        | 28/139 [00:08<00:24,  4.54it/s, epoch=96, grad_norm=62.500, loss=244.9729, lr=1.05e-03] 
Epoch 96 TRAIN:  22%|██▏       | 31/139 [00:08<00:22,  4.70it/s, epoch=96, grad_norm=62.500, loss=244.9729, lr=1.05e-03]
Epoch 96 TRAIN:  22%|██▏       | 31/139 [00:08<00:22,  4.70it/s, epoch=96, grad_norm=140.744, loss=285.2577, lr=1.05e-03]
Epoch 96 TRAIN:  22%|██▏       | 31/139 [00:09<00:22,  4.70it/s, epoch=96, grad_norm=92.109, loss=256.9681, lr=1.05e-03] 
Epoch 96 TRAIN:  22%|██▏       | 31/139 [00:09<00:22,  4.70it/s, epoch=96, grad_norm=132.106, loss=281.7576, lr=1.05e-03]
Epoch 96 TRAIN:  24%|██▍       | 34/139 [00:09<00:21,  4.94it/s, epoch=96, grad_norm=132.106, loss=281.7576, lr=1.05e-03]
Epoch 96 TRAIN:  24%|██▍       | 34/139 [00:09<00:21,  4.94it/s, epoch=96, grad_norm=105.923, loss=245.9811, lr=1.05e-03]
Epoch 96 TRAIN:  24%|██▍       | 34/139 [00:09<00:21,  4.94it/s, epoch=96, grad_norm=112.444, loss=305.2956, lr=1.05e-03]
Epoch 96 TRAIN:  24%|██▍       | 34/139 [00:09<00:21,  4.94it/s, epoch=96, grad_norm=96.705, loss=260.0468, lr=1.05e-03] 
Epoch 96 TRAIN:  27%|██▋       | 37/139 [00:09<00:21,  4.70it/s, epoch=96, grad_norm=96.705, loss=260.0468, lr=1.05e-03]
Epoch 96 TRAIN:  27%|██▋       | 37/139 [00:10<00:21,  4.70it/s, epoch=96, grad_norm=111.209, loss=250.0126, lr=1.05e-03]
Epoch 96 TRAIN:  27%|██▋       | 37/139 [00:10<00:21,  4.70it/s, epoch=96, grad_norm=67.126, loss=273.2122, lr=1.05e-03] 
Epoch 96 TRAIN:  27%|██▋       | 37/139 [00:10<00:21,  4.70it/s, epoch=96, grad_norm=103.301, loss=225.3678, lr=1.05e-03]
Epoch 96 TRAIN:  29%|██▉       | 40/139 [00:10<00:20,  4.82it/s, epoch=96, grad_norm=103.301, loss=225.3678, lr=1.05e-03]
Epoch 96 TRAIN:  29%|██▉       | 40/139 [00:10<00:20,  4.82it/s, epoch=96, grad_norm=98.846, loss=242.6472, lr=1.05e-03] 
Epoch 96 TRAIN:  29%|██▉       | 40/139 [00:10<00:20,  4.82it/s, epoch=96, grad_norm=67.195, loss=277.3393, lr=1.05e-03]
Epoch 96 TRAIN:  29%|██▉       | 40/139 [00:11<00:20,  4.82it/s, epoch=96, grad_norm=137.179, loss=273.8503, lr=1.05e-03]
Epoch 96 TRAIN:  31%|███       | 43/139 [00:11<00:19,  4.93it/s, epoch=96, grad_norm=137.179, loss=273.8503, lr=1.05e-03]
Epoch 96 TRAIN:  31%|███       | 43/139 [00:11<00:19,  4.93it/s, epoch=96, grad_norm=103.188, loss=227.4544, lr=1.05e-03]
Epoch 96 TRAIN:  31%|███       | 43/139 [00:11<00:19,  4.93it/s, epoch=96, grad_norm=145.132, loss=314.3314, lr=1.05e-03]
Epoch 96 TRAIN:  31%|███       | 43/139 [00:11<00:19,  4.93it/s, epoch=96, grad_norm=83.749, loss=273.5834, lr=1.05e-03] 
Epoch 96 TRAIN:  33%|███▎      | 46/139 [00:11<00:20,  4.54it/s, epoch=96, grad_norm=83.749, loss=273.5834, lr=1.05e-03]
Epoch 96 TRAIN:  33%|███▎      | 46/139 [00:12<00:20,  4.54it/s, epoch=96, grad_norm=102.279, loss=250.5513, lr=1.05e-03]
Epoch 96 TRAIN:  33%|███▎      | 46/139 [00:12<00:20,  4.54it/s, epoch=96, grad_norm=106.984, loss=289.8659, lr=1.05e-03]
Epoch 96 TRAIN:  33%|███▎      | 46/139 [00:12<00:20,  4.54it/s, epoch=96, grad_norm=99.248, loss=268.4830, lr=1.05e-03] 
Epoch 96 TRAIN:  35%|███▌      | 49/139 [00:12<00:18,  4.79it/s, epoch=96, grad_norm=99.248, loss=268.4830, lr=1.05e-03]
Epoch 96 TRAIN:  35%|███▌      | 49/139 [00:12<00:18,  4.79it/s, epoch=96, grad_norm=95.307, loss=259.0144, lr=1.05e-03]
Epoch 96 TRAIN:  35%|███▌      | 49/139 [00:12<00:18,  4.79it/s, epoch=96, grad_norm=92.139, loss=242.9816, lr=1.05e-03]
Epoch 96 TRAIN:  35%|███▌      | 49/139 [00:13<00:18,  4.79it/s, epoch=96, grad_norm=99.260, loss=255.3303, lr=1.05e-03]
Epoch 96 TRAIN:  37%|███▋      | 52/139 [00:13<00:18,  4.80it/s, epoch=96, grad_norm=99.260, loss=255.3303, lr=1.05e-03]
Epoch 96 TRAIN:  37%|███▋      | 52/139 [00:13<00:18,  4.80it/s, epoch=96, grad_norm=131.359, loss=229.4652, lr=1.05e-03]
Epoch 96 TRAIN:  37%|███▋      | 52/139 [00:13<00:18,  4.80it/s, epoch=96, grad_norm=127.410, loss=279.5920, lr=1.05e-03]
Epoch 96 TRAIN:  37%|███▋      | 52/139 [00:13<00:18,  4.80it/s, epoch=96, grad_norm=69.608, loss=246.5987, lr=1.05e-03] 
Epoch 96 TRAIN:  40%|███▉      | 55/139 [00:13<00:17,  4.75it/s, epoch=96, grad_norm=69.608, loss=246.5987, lr=1.05e-03]
Epoch 96 TRAIN:  40%|███▉      | 55/139 [00:13<00:17,  4.75it/s, epoch=96, grad_norm=116.703, loss=244.5699, lr=1.05e-03]
Epoch 96 TRAIN:  40%|███▉      | 55/139 [00:14<00:17,  4.75it/s, epoch=96, grad_norm=87.742, loss=229.5117, lr=1.05e-03] 
Epoch 96 TRAIN:  40%|███▉      | 55/139 [00:14<00:17,  4.75it/s, epoch=96, grad_norm=96.750, loss=246.9887, lr=1.05e-03]
Epoch 96 TRAIN:  42%|████▏     | 58/139 [00:14<00:17,  4.75it/s, epoch=96, grad_norm=96.750, loss=246.9887, lr=1.05e-03]
Epoch 96 TRAIN:  42%|████▏     | 58/139 [00:14<00:17,  4.75it/s, epoch=96, grad_norm=78.020, loss=221.1551, lr=1.05e-03]
Epoch 96 TRAIN:  42%|████▏     | 58/139 [00:14<00:17,  4.75it/s, epoch=96, grad_norm=157.372, loss=298.7804, lr=1.05e-03]
Epoch 96 TRAIN:  42%|████▏     | 58/139 [00:15<00:17,  4.75it/s, epoch=96, grad_norm=81.391, loss=240.1900, lr=1.05e-03] 
Epoch 96 TRAIN:  44%|████▍     | 61/139 [00:15<00:16,  4.65it/s, epoch=96, grad_norm=81.391, loss=240.1900, lr=1.05e-03]
Epoch 96 TRAIN:  44%|████▍     | 61/139 [00:15<00:16,  4.65it/s, epoch=96, grad_norm=128.430, loss=277.3818, lr=1.05e-03]
Epoch 96 TRAIN:  44%|████▍     | 61/139 [00:15<00:16,  4.65it/s, epoch=96, grad_norm=124.283, loss=255.0050, lr=1.05e-03]
Epoch 96 TRAIN:  44%|████▍     | 61/139 [00:15<00:16,  4.65it/s, epoch=96, grad_norm=206.519, loss=327.7904, lr=1.05e-03]
Epoch 96 TRAIN:  46%|████▌     | 64/139 [00:15<00:15,  4.75it/s, epoch=96, grad_norm=206.519, loss=327.7904, lr=1.05e-03]
Epoch 96 TRAIN:  46%|████▌     | 64/139 [00:15<00:15,  4.75it/s, epoch=96, grad_norm=87.061, loss=252.6848, lr=1.05e-03] 
Epoch 96 TRAIN:  46%|████▌     | 64/139 [00:16<00:15,  4.75it/s, epoch=96, grad_norm=146.341, loss=223.3578, lr=1.05e-03]
Epoch 96 TRAIN:  46%|████▌     | 64/139 [00:16<00:15,  4.75it/s, epoch=96, grad_norm=152.646, loss=245.3531, lr=1.05e-03]
Epoch 96 TRAIN:  48%|████▊     | 67/139 [00:16<00:15,  4.66it/s, epoch=96, grad_norm=152.646, loss=245.3531, lr=1.05e-03]
Epoch 96 TRAIN:  48%|████▊     | 67/139 [00:16<00:15,  4.66it/s, epoch=96, grad_norm=117.280, loss=265.1346, lr=1.05e-03]
Epoch 96 TRAIN:  48%|████▊     | 67/139 [00:16<00:15,  4.66it/s, epoch=96, grad_norm=270.737, loss=193.6597, lr=1.05e-03]
Epoch 96 TRAIN:  48%|████▊     | 67/139 [00:16<00:15,  4.66it/s, epoch=96, grad_norm=210.215, loss=245.7724, lr=1.05e-03]
Epoch 96 TRAIN:  50%|█████     | 70/139 [00:16<00:14,  4.65it/s, epoch=96, grad_norm=210.215, loss=245.7724, lr=1.05e-03]
Epoch 96 TRAIN:  50%|█████     | 70/139 [00:17<00:14,  4.65it/s, epoch=96, grad_norm=93.471, loss=271.9320, lr=1.05e-03] 
Epoch 96 TRAIN:  50%|█████     | 70/139 [00:17<00:14,  4.65it/s, epoch=96, grad_norm=98.635, loss=222.3615, lr=1.05e-03]
Epoch 96 TRAIN:  50%|█████     | 70/139 [00:17<00:14,  4.65it/s, epoch=96, grad_norm=103.615, loss=265.0852, lr=1.05e-03]
Epoch 96 TRAIN:  53%|█████▎    | 73/139 [00:17<00:14,  4.58it/s, epoch=96, grad_norm=103.615, loss=265.0852, lr=1.05e-03]
Epoch 96 TRAIN:  53%|█████▎    | 73/139 [00:17<00:14,  4.58it/s, epoch=96, grad_norm=205.230, loss=283.3350, lr=1.05e-03]
Epoch 96 TRAIN:  53%|█████▎    | 73/139 [00:18<00:14,  4.58it/s, epoch=96, grad_norm=73.882, loss=220.6416, lr=1.05e-03] 
Epoch 96 TRAIN:  53%|█████▎    | 73/139 [00:18<00:14,  4.58it/s, epoch=96, grad_norm=134.147, loss=234.8063, lr=1.05e-03]
Epoch 96 TRAIN:  55%|█████▍    | 76/139 [00:18<00:13,  4.79it/s, epoch=96, grad_norm=134.147, loss=234.8063, lr=1.05e-03]
Epoch 96 TRAIN:  55%|█████▍    | 76/139 [00:18<00:13,  4.79it/s, epoch=96, grad_norm=103.939, loss=269.8109, lr=1.05e-03]
Epoch 96 TRAIN:  55%|█████▍    | 76/139 [00:18<00:13,  4.79it/s, epoch=96, grad_norm=136.169, loss=260.6548, lr=1.05e-03]
Epoch 96 TRAIN:  55%|█████▍    | 76/139 [00:19<00:13,  4.79it/s, epoch=96, grad_norm=178.172, loss=275.9735, lr=1.05e-03]
Epoch 96 TRAIN:  57%|█████▋    | 79/139 [00:19<00:13,  4.45it/s, epoch=96, grad_norm=178.172, loss=275.9735, lr=1.05e-03]
Epoch 96 TRAIN:  57%|█████▋    | 79/139 [00:19<00:13,  4.45it/s, epoch=96, grad_norm=160.133, loss=299.3043, lr=1.05e-03]
Epoch 96 TRAIN:  57%|█████▋    | 79/139 [00:19<00:13,  4.45it/s, epoch=96, grad_norm=103.082, loss=261.8412, lr=1.05e-03]
Epoch 96 TRAIN:  57%|█████▋    | 79/139 [00:19<00:13,  4.45it/s, epoch=96, grad_norm=154.411, loss=269.2750, lr=1.05e-03]
Epoch 96 TRAIN:  59%|█████▉    | 82/139 [00:19<00:12,  4.45it/s, epoch=96, grad_norm=154.411, loss=269.2750, lr=1.05e-03]
Epoch 96 TRAIN:  59%|█████▉    | 82/139 [00:19<00:12,  4.45it/s, epoch=96, grad_norm=148.734, loss=277.5186, lr=1.05e-03]
Epoch 96 TRAIN:  59%|█████▉    | 82/139 [00:20<00:12,  4.45it/s, epoch=96, grad_norm=210.467, loss=222.3590, lr=1.05e-03]
Epoch 96 TRAIN:  59%|█████▉    | 82/139 [00:20<00:12,  4.45it/s, epoch=96, grad_norm=84.380, loss=263.2685, lr=1.05e-03] 
Epoch 96 TRAIN:  61%|██████    | 85/139 [00:20<00:12,  4.47it/s, epoch=96, grad_norm=84.380, loss=263.2685, lr=1.05e-03]
Epoch 96 TRAIN:  61%|██████    | 85/139 [00:20<00:12,  4.47it/s, epoch=96, grad_norm=169.221, loss=199.2784, lr=1.05e-03]
Epoch 96 TRAIN:  61%|██████    | 85/139 [00:20<00:12,  4.47it/s, epoch=96, grad_norm=125.660, loss=207.0604, lr=1.05e-03]
Epoch 96 TRAIN:  61%|██████    | 85/139 [00:21<00:12,  4.47it/s, epoch=96, grad_norm=103.199, loss=260.9278, lr=1.05e-03]
Epoch 96 TRAIN:  63%|██████▎   | 88/139 [00:21<00:11,  4.41it/s, epoch=96, grad_norm=103.199, loss=260.9278, lr=1.05e-03]
Epoch 96 TRAIN:  63%|██████▎   | 88/139 [00:21<00:11,  4.41it/s, epoch=96, grad_norm=107.447, loss=217.8314, lr=1.05e-03]
Epoch 96 TRAIN:  63%|██████▎   | 88/139 [00:21<00:11,  4.41it/s, epoch=96, grad_norm=124.358, loss=305.0322, lr=1.05e-03]
Epoch 96 TRAIN:  63%|██████▎   | 88/139 [00:21<00:11,  4.41it/s, epoch=96, grad_norm=162.681, loss=284.0170, lr=1.05e-03]
Epoch 96 TRAIN:  65%|██████▌   | 91/139 [00:21<00:11,  4.28it/s, epoch=96, grad_norm=162.681, loss=284.0170, lr=1.05e-03]
Epoch 96 TRAIN:  65%|██████▌   | 91/139 [00:22<00:11,  4.28it/s, epoch=96, grad_norm=305.087, loss=202.9856, lr=1.05e-03]
Epoch 96 TRAIN:  65%|██████▌   | 91/139 [00:22<00:11,  4.28it/s, epoch=96, grad_norm=84.064, loss=221.4309, lr=1.06e-03] 
Epoch 96 TRAIN:  65%|██████▌   | 91/139 [00:22<00:11,  4.28it/s, epoch=96, grad_norm=98.191, loss=242.7519, lr=1.06e-03]
Epoch 96 TRAIN:  68%|██████▊   | 94/139 [00:22<00:10,  4.44it/s, epoch=96, grad_norm=98.191, loss=242.7519, lr=1.06e-03]
Epoch 96 TRAIN:  68%|██████▊   | 94/139 [00:22<00:10,  4.44it/s, epoch=96, grad_norm=108.249, loss=281.7867, lr=1.06e-03]
Epoch 96 TRAIN:  68%|██████▊   | 94/139 [00:22<00:10,  4.44it/s, epoch=96, grad_norm=72.930, loss=243.4402, lr=1.06e-03] 
Epoch 96 TRAIN:  68%|██████▊   | 94/139 [00:23<00:10,  4.44it/s, epoch=96, grad_norm=171.575, loss=202.7109, lr=1.06e-03]
Epoch 96 TRAIN:  70%|██████▉   | 97/139 [00:23<00:09,  4.37it/s, epoch=96, grad_norm=171.575, loss=202.7109, lr=1.06e-03]
Epoch 96 TRAIN:  70%|██████▉   | 97/139 [00:23<00:09,  4.37it/s, epoch=96, grad_norm=104.293, loss=249.7690, lr=1.06e-03]
Epoch 96 TRAIN:  70%|██████▉   | 97/139 [00:23<00:09,  4.37it/s, epoch=96, grad_norm=76.049, loss=196.3615, lr=1.06e-03] 
Epoch 96 TRAIN:  70%|██████▉   | 97/139 [00:23<00:09,  4.37it/s, epoch=96, grad_norm=190.537, loss=225.6180, lr=1.06e-03]2026-04-16 20:42:25,231 DEBUG TRAIN | steps/sec 4.274| Batch 96/100 loss 225.617981 loss_att 168.255463 loss_ctc 359.463806 th_accuracy 0.494652 lr 1.0556e-03 grad_norm 190.537415 rank 0

Epoch 96 TRAIN:  72%|███████▏  | 100/139 [00:23<00:08,  4.57it/s, epoch=96, grad_norm=190.537, loss=225.6180, lr=1.06e-03]
Epoch 96 TRAIN:  72%|███████▏  | 100/139 [00:23<00:08,  4.57it/s, epoch=96, grad_norm=201.765, loss=254.2669, lr=1.06e-03]
Epoch 96 TRAIN:  72%|███████▏  | 100/139 [00:24<00:08,  4.57it/s, epoch=96, grad_norm=213.756, loss=289.5624, lr=1.06e-03]
Epoch 96 TRAIN:  72%|███████▏  | 100/139 [00:24<00:08,  4.57it/s, epoch=96, grad_norm=156.568, loss=221.3388, lr=1.06e-03]
Epoch 96 TRAIN:  74%|███████▍  | 103/139 [00:24<00:08,  4.47it/s, epoch=96, grad_norm=156.568, loss=221.3388, lr=1.06e-03]
Epoch 96 TRAIN:  74%|███████▍  | 103/139 [00:24<00:08,  4.47it/s, epoch=96, grad_norm=150.855, loss=205.4211, lr=1.06e-03]
Epoch 96 TRAIN:  74%|███████▍  | 103/139 [00:24<00:08,  4.47it/s, epoch=96, grad_norm=157.613, loss=268.8192, lr=1.06e-03]
Epoch 96 TRAIN:  74%|███████▍  | 103/139 [00:25<00:08,  4.47it/s, epoch=96, grad_norm=280.054, loss=299.4852, lr=1.06e-03]
Epoch 96 TRAIN:  76%|███████▋  | 106/139 [00:25<00:07,  4.60it/s, epoch=96, grad_norm=280.054, loss=299.4852, lr=1.06e-03]
Epoch 96 TRAIN:  76%|███████▋  | 106/139 [00:25<00:07,  4.60it/s, epoch=96, grad_norm=152.739, loss=239.5644, lr=1.06e-03]
Epoch 96 TRAIN:  76%|███████▋  | 106/139 [00:25<00:07,  4.60it/s, epoch=96, grad_norm=145.107, loss=289.4143, lr=1.06e-03]
Epoch 96 TRAIN:  76%|███████▋  | 106/139 [00:25<00:07,  4.60it/s, epoch=96, grad_norm=87.662, loss=220.5880, lr=1.06e-03] 
Epoch 96 TRAIN:  78%|███████▊  | 109/139 [00:25<00:06,  4.68it/s, epoch=96, grad_norm=87.662, loss=220.5880, lr=1.06e-03]
Epoch 96 TRAIN:  78%|███████▊  | 109/139 [00:25<00:06,  4.68it/s, epoch=96, grad_norm=118.203, loss=231.7250, lr=1.06e-03]
Epoch 96 TRAIN:  78%|███████▊  | 109/139 [00:26<00:06,  4.68it/s, epoch=96, grad_norm=100.405, loss=245.7630, lr=1.06e-03]
Epoch 96 TRAIN:  78%|███████▊  | 109/139 [00:26<00:06,  4.68it/s, epoch=96, grad_norm=226.876, loss=280.3032, lr=1.06e-03]
Epoch 96 TRAIN:  81%|████████  | 112/139 [00:26<00:06,  4.37it/s, epoch=96, grad_norm=226.876, loss=280.3032, lr=1.06e-03]
Epoch 96 TRAIN:  81%|████████  | 112/139 [00:26<00:06,  4.37it/s, epoch=96, grad_norm=114.497, loss=265.8918, lr=1.06e-03]
Epoch 96 TRAIN:  81%|████████  | 112/139 [00:26<00:06,  4.37it/s, epoch=96, grad_norm=86.167, loss=208.4076, lr=1.06e-03] 
Epoch 96 TRAIN:  81%|████████  | 112/139 [00:26<00:06,  4.37it/s, epoch=96, grad_norm=172.914, loss=285.2556, lr=1.06e-03]
Epoch 96 TRAIN:  83%|████████▎ | 115/139 [00:26<00:05,  4.62it/s, epoch=96, grad_norm=172.914, loss=285.2556, lr=1.06e-03]
Epoch 96 TRAIN:  83%|████████▎ | 115/139 [00:27<00:05,  4.62it/s, epoch=96, grad_norm=201.905, loss=292.4746, lr=1.06e-03]
Epoch 96 TRAIN:  83%|████████▎ | 115/139 [00:27<00:05,  4.62it/s, epoch=96, grad_norm=214.413, loss=244.0321, lr=1.06e-03]
Epoch 96 TRAIN:  83%|████████▎ | 115/139 [00:27<00:05,  4.62it/s, epoch=96, grad_norm=280.784, loss=330.3411, lr=1.06e-03]
Epoch 96 TRAIN:  85%|████████▍ | 118/139 [00:27<00:04,  4.76it/s, epoch=96, grad_norm=280.784, loss=330.3411, lr=1.06e-03]
Epoch 96 TRAIN:  85%|████████▍ | 118/139 [00:27<00:04,  4.76it/s, epoch=96, grad_norm=203.860, loss=278.8846, lr=1.06e-03]
Epoch 96 TRAIN:  85%|████████▍ | 118/139 [00:27<00:04,  4.76it/s, epoch=96, grad_norm=103.792, loss=186.4969, lr=1.06e-03]
Epoch 96 TRAIN:  85%|████████▍ | 118/139 [00:28<00:04,  4.76it/s, epoch=96, grad_norm=442.930, loss=325.9630, lr=1.06e-03]
Epoch 96 TRAIN:  87%|████████▋ | 121/139 [00:28<00:03,  5.02it/s, epoch=96, grad_norm=442.930, loss=325.9630, lr=1.06e-03]
Epoch 96 TRAIN:  87%|████████▋ | 121/139 [00:28<00:03,  5.02it/s, epoch=96, grad_norm=168.179, loss=239.1696, lr=1.06e-03]
Epoch 96 TRAIN:  87%|████████▋ | 121/139 [00:28<00:03,  5.02it/s, epoch=96, grad_norm=157.580, loss=210.9574, lr=1.06e-03]
Epoch 96 TRAIN:  87%|████████▋ | 121/139 [00:28<00:03,  5.02it/s, epoch=96, grad_norm=165.962, loss=277.1978, lr=1.06e-03]
Epoch 96 TRAIN:  89%|████████▉ | 124/139 [00:28<00:02,  5.11it/s, epoch=96, grad_norm=165.962, loss=277.1978, lr=1.06e-03]
Epoch 96 TRAIN:  89%|████████▉ | 124/139 [00:28<00:02,  5.11it/s, epoch=96, grad_norm=247.528, loss=300.8766, lr=1.06e-03]
Epoch 96 TRAIN:  89%|████████▉ | 124/139 [00:29<00:02,  5.11it/s, epoch=96, grad_norm=154.771, loss=271.5475, lr=1.06e-03]
Epoch 96 TRAIN:  89%|████████▉ | 124/139 [00:29<00:02,  5.11it/s, epoch=96, grad_norm=117.476, loss=261.7878, lr=1.06e-03]
Epoch 96 TRAIN:  91%|█████████▏| 127/139 [00:29<00:02,  5.07it/s, epoch=96, grad_norm=117.476, loss=261.7878, lr=1.06e-03]
Epoch 96 TRAIN:  91%|█████████▏| 127/139 [00:29<00:02,  5.07it/s, epoch=96, grad_norm=259.762, loss=301.7699, lr=1.06e-03]
Epoch 96 TRAIN:  91%|█████████▏| 127/139 [00:29<00:02,  5.07it/s, epoch=96, grad_norm=166.736, loss=214.5418, lr=1.06e-03]
Epoch 96 TRAIN:  91%|█████████▏| 127/139 [00:29<00:02,  5.07it/s, epoch=96, grad_norm=217.260, loss=232.9641, lr=1.06e-03]
Epoch 96 TRAIN:  94%|█████████▎| 130/139 [00:29<00:01,  5.07it/s, epoch=96, grad_norm=217.260, loss=232.9641, lr=1.06e-03]
Epoch 96 TRAIN:  94%|█████████▎| 130/139 [00:30<00:01,  5.07it/s, epoch=96, grad_norm=106.462, loss=193.6532, lr=1.06e-03]
Epoch 96 TRAIN:  94%|█████████▎| 130/139 [00:30<00:01,  5.07it/s, epoch=96, grad_norm=281.691, loss=305.4668, lr=1.06e-03]
Epoch 96 TRAIN:  94%|█████████▎| 130/139 [00:30<00:01,  5.07it/s, epoch=96, grad_norm=151.847, loss=208.5528, lr=1.06e-03]
Epoch 96 TRAIN:  96%|█████████▌| 133/139 [00:30<00:01,  5.18it/s, epoch=96, grad_norm=151.847, loss=208.5528, lr=1.06e-03]
Epoch 96 TRAIN:  96%|█████████▌| 133/139 [00:30<00:01,  5.18it/s, epoch=96, grad_norm=286.054, loss=337.5117, lr=1.06e-03]
Epoch 96 TRAIN:  96%|█████████▌| 133/139 [00:30<00:01,  5.18it/s, epoch=96, grad_norm=129.213, loss=243.1829, lr=1.06e-03]
Epoch 96 TRAIN:  96%|█████████▌| 133/139 [00:31<00:01,  5.18it/s, epoch=96, grad_norm=161.290, loss=278.8184, lr=1.06e-03]
Epoch 96 TRAIN:  98%|█████████▊| 136/139 [00:31<00:00,  5.00it/s, epoch=96, grad_norm=161.290, loss=278.8184, lr=1.06e-03]
Epoch 96 TRAIN:  98%|█████████▊| 136/139 [00:31<00:00,  4.38it/s, epoch=96, grad_norm=161.290, loss=278.8184, lr=1.06e-03]

CV:   0%|          | 0/6 [00:00<?, ?it/s]/home/rookie/Documents/batuka/projects/wenet/wenet/wenet/utils/train_utils.py:689: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  amp_autocast(enabled=dtype is not None,

CV:  17%|█▋        | 1/6 [00:00<00:03,  1.33it/s]
                                                 
2026-04-16 20:42:33,429 INFO Epoch 96 Step 13231 CV info lr 1.0585e-03 cv_loss 453.26940363103694 rank 0 acc 0.3896709481875102
2026-04-16 20:42:33,430 INFO [Rank 0] Checkpoint: save to checkpoint exp/conformer_mongolian/epoch_96.pt
2026-04-16 20:42:33,540 INFO Epoch 97 Step 13231 TRAIN info lr 1.0585e-03 rank 0
2026-04-16 20:42:33,558 INFO using accumulate grad, new batch size is 1 times larger than before

Epoch 97 TRAIN:   0%|          | 0/136 [00:00<?, ?it/s]
Epoch 97 TRAIN:   0%|          | 0/136 [00:03<?, ?it/s, epoch=97, grad_norm=247.578, loss=219.1582, lr=1.06e-03]
Epoch 97 TRAIN:   1%|          | 1/136 [00:03<08:54,  3.96s/it, epoch=97, grad_norm=247.578, loss=219.1582, lr=1.06e-03]
Epoch 97 TRAIN:   1%|          | 1/136 [00:04<08:54,  3.96s/it, epoch=97, grad_norm=337.084, loss=268.3253, lr=1.06e-03]
Epoch 97 TRAIN:   1%|          | 1/136 [00:04<08:54,  3.96s/it, epoch=97, grad_norm=168.247, loss=201.6873, lr=1.06e-03]
Epoch 97 TRAIN:   1%|          | 1/136 [00:04<08:54,  3.96s/it, epoch=97, grad_norm=433.201, loss=368.4347, lr=1.06e-03]
Epoch 97 TRAIN:   1%|          | 1/136 [00:04<08:54,  3.96s/it, epoch=97, grad_norm=263.421, loss=267.6841, lr=1.06e-03]
Epoch 97 TRAIN:   1%|          | 1/136 [00:04<08:54,  3.96s/it, epoch=97, grad_norm=346.117, loss=355.1235, lr=1.06e-03]
Epoch 97 TRAIN:   1%|          | 1/136 [00:04<08:54,  3.96s/it, epoch=97, grad_norm=256.132, loss=276.4471, lr=1.06e-03]
Epoch 97 TRAIN:   1%|          | 1/136 [00:04<08:54,  3.96s/it, epoch=97, grad_norm=167.057, loss=248.0152, lr=1.06e-03]
Epoch 97 TRAIN:   6%|▌         | 8/136 [00:04<00:55,  2.32it/s, epoch=97, grad_norm=167.057, loss=248.0152, lr=1.06e-03]
Epoch 97 TRAIN:   6%|▌         | 8/136 [00:04<00:55,  2.32it/s, epoch=97, grad_norm=113.102, loss=267.2838, lr=1.06e-03]
Epoch 97 TRAIN:   6%|▌         | 8/136 [00:04<00:55,  2.32it/s, epoch=97, grad_norm=86.117, loss=258.5921, lr=1.06e-03] 
Epoch 97 TRAIN:   6%|▌         | 8/136 [00:04<00:55,  2.32it/s, epoch=97, grad_norm=116.408, loss=295.9237, lr=1.06e-03]
Epoch 97 TRAIN:   6%|▌         | 8/136 [00:05<00:55,  2.32it/s, epoch=97, grad_norm=92.401, loss=288.2665, lr=1.06e-03] 
Epoch 97 TRAIN:   9%|▉         | 12/136 [00:05<00:39,  3.15it/s, epoch=97, grad_norm=92.401, loss=288.2665, lr=1.06e-03]
Epoch 97 TRAIN:   9%|▉         | 12/136 [00:05<00:39,  3.15it/s, epoch=97, grad_norm=108.128, loss=273.4654, lr=1.06e-03]
Epoch 97 TRAIN:   9%|▉         | 12/136 [00:05<00:39,  3.15it/s, epoch=97, grad_norm=152.050, loss=243.4250, lr=1.06e-03]
Epoch 97 TRAIN:   9%|▉         | 12/136 [00:05<00:39,  3.15it/s, epoch=97, grad_norm=103.646, loss=303.6292, lr=1.06e-03]
Epoch 97 TRAIN:   9%|▉         | 12/136 [00:05<00:39,  3.15it/s, epoch=97, grad_norm=88.327, loss=292.7642, lr=1.06e-03] 
Epoch 97 TRAIN:  12%|█▏        | 16/136 [00:05<00:31,  3.85it/s, epoch=97, grad_norm=88.327, loss=292.7642, lr=1.06e-03]
Epoch 97 TRAIN:  12%|█▏        | 16/136 [00:06<00:31,  3.85it/s, epoch=97, grad_norm=176.544, loss=264.2069, lr=1.06e-03]
Epoch 97 TRAIN:  12%|█▏        | 16/136 [00:06<00:31,  3.85it/s, epoch=97, grad_norm=161.261, loss=246.7893, lr=1.06e-03]
Epoch 97 TRAIN:  12%|█▏        | 16/136 [00:06<00:31,  3.85it/s, epoch=97, grad_norm=131.818, loss=264.1128, lr=1.06e-03]
Epoch 97 TRAIN:  12%|█▏        | 16/136 [00:06<00:31,  3.85it/s, epoch=97, grad_norm=90.456, loss=254.8713, lr=1.06e-03] 
Epoch 97 TRAIN:  15%|█▍        | 20/136 [00:06<00:28,  4.09it/s, epoch=97, grad_norm=90.456, loss=254.8713, lr=1.06e-03]
Epoch 97 TRAIN:  15%|█▍        | 20/136 [00:06<00:28,  4.09it/s, epoch=97, grad_norm=121.676, loss=221.9307, lr=1.06e-03]
Epoch 97 TRAIN:  15%|█▍        | 20/136 [00:07<00:28,  4.09it/s, epoch=97, grad_norm=99.824, loss=279.7769, lr=1.06e-03] 
Epoch 97 TRAIN:  15%|█▍        | 20/136 [00:07<00:28,  4.09it/s, epoch=97, grad_norm=117.816, loss=285.9102, lr=1.06e-03]
Epoch 97 TRAIN:  17%|█▋        | 23/136 [00:07<00:25,  4.36it/s, epoch=97, grad_norm=117.816, loss=285.9102, lr=1.06e-03]
Epoch 97 TRAIN:  17%|█▋        | 23/136 [00:07<00:25,  4.36it/s, epoch=97, grad_norm=129.148, loss=283.9910, lr=1.06e-03]
Epoch 97 TRAIN:  17%|█▋        | 23/136 [00:07<00:25,  4.36it/s, epoch=97, grad_norm=143.434, loss=262.0091, lr=1.06e-03]
Epoch 97 TRAIN:  17%|█▋        | 23/136 [00:07<00:25,  4.36it/s, epoch=97, grad_norm=85.672, loss=249.2737, lr=1.06e-03] 
Epoch 97 TRAIN:  19%|█▉        | 26/136 [00:07<00:24,  4.55it/s, epoch=97, grad_norm=85.672, loss=249.2737, lr=1.06e-03]
Epoch 97 TRAIN:  19%|█▉        | 26/136 [00:08<00:24,  4.55it/s, epoch=97, grad_norm=123.418, loss=246.3573, lr=1.06e-03]
Epoch 97 TRAIN:  19%|█▉        | 26/136 [00:08<00:24,  4.55it/s, epoch=97, grad_norm=158.943, loss=307.1180, lr=1.06e-03]
Epoch 97 TRAIN:  19%|█▉        | 26/136 [00:08<00:24,  4.55it/s, epoch=97, grad_norm=127.931, loss=268.5239, lr=1.06e-03]
Epoch 97 TRAIN:  21%|██▏       | 29/136 [00:08<00:23,  4.64it/s, epoch=97, grad_norm=127.931, loss=268.5239, lr=1.06e-03]
Epoch 97 TRAIN:  21%|██▏       | 29/136 [00:08<00:23,  4.64it/s, epoch=97, grad_norm=110.688, loss=267.9404, lr=1.06e-03]
Epoch 97 TRAIN:  21%|██▏       | 29/136 [00:08<00:23,  4.64it/s, epoch=97, grad_norm=75.529, loss=273.0938, lr=1.06e-03] 
Epoch 97 TRAIN:  21%|██▏       | 29/136 [00:09<00:23,  4.64it/s, epoch=97, grad_norm=101.757, loss=228.6087, lr=1.06e-03]
Epoch 97 TRAIN:  24%|██▎       | 32/136 [00:09<00:21,  4.75it/s, epoch=97, grad_norm=101.757, loss=228.6087, lr=1.06e-03]
Epoch 97 TRAIN:  24%|██▎       | 32/136 [00:09<00:21,  4.75it/s, epoch=97, grad_norm=109.508, loss=271.3783, lr=1.06e-03]
Epoch 97 TRAIN:  24%|██▎       | 32/136 [00:09<00:21,  4.75it/s, epoch=97, grad_norm=127.726, loss=230.5536, lr=1.06e-03]
Epoch 97 TRAIN:  24%|██▎       | 32/136 [00:09<00:21,  4.75it/s, epoch=97, grad_norm=107.395, loss=249.4781, lr=1.06e-03]
Epoch 97 TRAIN:  26%|██▌       | 35/136 [00:09<00:20,  4.82it/s, epoch=97, grad_norm=107.395, loss=249.4781, lr=1.06e-03]
Epoch 97 TRAIN:  26%|██▌       | 35/136 [00:09<00:20,  4.82it/s, epoch=97, grad_norm=73.953, loss=270.6144, lr=1.06e-03] 
Epoch 97 TRAIN:  26%|██▌       | 35/136 [00:10<00:20,  4.82it/s, epoch=97, grad_norm=90.570, loss=285.1378, lr=1.06e-03]
Epoch 97 TRAIN:  26%|██▌       | 35/136 [00:10<00:20,  4.82it/s, epoch=97, grad_norm=75.272, loss=273.5724, lr=1.06e-03]
Epoch 97 TRAIN:  28%|██▊       | 38/136 [00:10<00:19,  4.95it/s, epoch=97, grad_norm=75.272, loss=273.5724, lr=1.06e-03]
Epoch 97 TRAIN:  28%|██▊       | 38/136 [00:10<00:19,  4.95it/s, epoch=97, grad_norm=103.928, loss=247.3322, lr=1.06e-03]
Epoch 97 TRAIN:  28%|██▊       | 38/136 [00:10<00:19,  4.95it/s, epoch=97, grad_norm=88.376, loss=215.3663, lr=1.06e-03] 
Epoch 97 TRAIN:  28%|██▊       | 38/136 [00:10<00:19,  4.95it/s, epoch=97, grad_norm=73.718, loss=268.5717, lr=1.06e-03]
Epoch 97 TRAIN:  30%|███       | 41/136 [00:10<00:20,  4.66it/s, epoch=97, grad_norm=73.718, loss=268.5717, lr=1.06e-03]
Epoch 97 TRAIN:  30%|███       | 41/136 [00:11<00:20,  4.66it/s, epoch=97, grad_norm=73.913, loss=252.2504, lr=1.06e-03]
Epoch 97 TRAIN:  30%|███       | 41/136 [00:11<00:20,  4.66it/s, epoch=97, grad_norm=110.469, loss=222.7698, lr=1.06e-03]
Epoch 97 TRAIN:  30%|███       | 41/136 [00:11<00:20,  4.66it/s, epoch=97, grad_norm=128.110, loss=291.1857, lr=1.06e-03]
Epoch 97 TRAIN:  32%|███▏      | 44/136 [00:11<00:19,  4.71it/s, epoch=97, grad_norm=128.110, loss=291.1857, lr=1.06e-03]
Epoch 97 TRAIN:  32%|███▏      | 44/136 [00:11<00:19,  4.71it/s, epoch=97, grad_norm=77.845, loss=238.0889, lr=1.06e-03] 
Epoch 97 TRAIN:  32%|███▏      | 44/136 [00:11<00:19,  4.71it/s, epoch=97, grad_norm=135.603, loss=284.1978, lr=1.06e-03]
Epoch 97 TRAIN:  32%|███▏      | 44/136 [00:12<00:19,  4.71it/s, epoch=97, grad_norm=145.479, loss=289.2704, lr=1.06e-03]
Epoch 97 TRAIN:  35%|███▍      | 47/136 [00:12<00:18,  4.73it/s, epoch=97, grad_norm=145.479, loss=289.2704, lr=1.06e-03]
Epoch 97 TRAIN:  35%|███▍      | 47/136 [00:12<00:18,  4.73it/s, epoch=97, grad_norm=122.535, loss=275.8361, lr=1.06e-03]
Epoch 97 TRAIN:  35%|███▍      | 47/136 [00:12<00:18,  4.73it/s, epoch=97, grad_norm=84.480, loss=272.5626, lr=1.06e-03] 
Epoch 97 TRAIN:  35%|███▍      | 47/136 [00:12<00:18,  4.73it/s, epoch=97, grad_norm=82.836, loss=248.7202, lr=1.06e-03]
Epoch 97 TRAIN:  37%|███▋      | 50/136 [00:12<00:18,  4.75it/s, epoch=97, grad_norm=82.836, loss=248.7202, lr=1.06e-03]
Epoch 97 TRAIN:  37%|███▋      | 50/136 [00:13<00:18,  4.75it/s, epoch=97, grad_norm=107.574, loss=232.4697, lr=1.06e-03]
Epoch 97 TRAIN:  37%|███▋      | 50/136 [00:13<00:18,  4.75it/s, epoch=97, grad_norm=137.930, loss=291.6826, lr=1.06e-03]
Epoch 97 TRAIN:  37%|███▋      | 50/136 [00:13<00:18,  4.75it/s, epoch=97, grad_norm=98.923, loss=241.6184, lr=1.06e-03] 
Epoch 97 TRAIN:  39%|███▉      | 53/136 [00:13<00:17,  4.79it/s, epoch=97, grad_norm=98.923, loss=241.6184, lr=1.06e-03]
Epoch 97 TRAIN:  39%|███▉      | 53/136 [00:13<00:17,  4.79it/s, epoch=97, grad_norm=163.950, loss=318.1338, lr=1.06e-03]
Epoch 97 TRAIN:  39%|███▉      | 53/136 [00:13<00:17,  4.79it/s, epoch=97, grad_norm=101.423, loss=281.5270, lr=1.06e-03]
Epoch 97 TRAIN:  39%|███▉      | 53/136 [00:14<00:17,  4.79it/s, epoch=97, grad_norm=110.099, loss=238.2172, lr=1.06e-03]
Epoch 97 TRAIN:  41%|████      | 56/136 [00:14<00:17,  4.56it/s, epoch=97, grad_norm=110.099, loss=238.2172, lr=1.06e-03]
Epoch 97 TRAIN:  41%|████      | 56/136 [00:14<00:17,  4.56it/s, epoch=97, grad_norm=119.401, loss=252.8407, lr=1.06e-03]
Epoch 97 TRAIN:  41%|████      | 56/136 [00:14<00:17,  4.56it/s, epoch=97, grad_norm=139.931, loss=294.1475, lr=1.06e-03]
Epoch 97 TRAIN:  41%|████      | 56/136 [00:14<00:17,  4.56it/s, epoch=97, grad_norm=98.821, loss=254.5927, lr=1.06e-03] 
Epoch 97 TRAIN:  43%|████▎     | 59/136 [00:14<00:17,  4.48it/s, epoch=97, grad_norm=98.821, loss=254.5927, lr=1.06e-03]
Epoch 97 TRAIN:  43%|████▎     | 59/136 [00:15<00:17,  4.48it/s, epoch=97, grad_norm=107.837, loss=283.9124, lr=1.06e-03]
Epoch 97 TRAIN:  43%|████▎     | 59/136 [00:15<00:17,  4.48it/s, epoch=97, grad_norm=89.504, loss=238.1205, lr=1.06e-03] 
Epoch 97 TRAIN:  43%|████▎     | 59/136 [00:15<00:17,  4.48it/s, epoch=97, grad_norm=100.928, loss=283.6671, lr=1.06e-03]
Epoch 97 TRAIN:  46%|████▌     | 62/136 [00:15<00:15,  4.72it/s, epoch=97, grad_norm=100.928, loss=283.6671, lr=1.06e-03]
Epoch 97 TRAIN:  46%|████▌     | 62/136 [00:15<00:15,  4.72it/s, epoch=97, grad_norm=99.195, loss=283.0962, lr=1.06e-03] 
Epoch 97 TRAIN:  46%|████▌     | 62/136 [00:15<00:15,  4.72it/s, epoch=97, grad_norm=64.987, loss=245.8926, lr=1.06e-03]
Epoch 97 TRAIN:  46%|████▌     | 62/136 [00:16<00:15,  4.72it/s, epoch=97, grad_norm=77.940, loss=267.5216, lr=1.06e-03]
Epoch 97 TRAIN:  48%|████▊     | 65/136 [00:16<00:15,  4.48it/s, epoch=97, grad_norm=77.940, loss=267.5216, lr=1.06e-03]
Epoch 97 TRAIN:  48%|████▊     | 65/136 [00:16<00:15,  4.48it/s, epoch=97, grad_norm=83.441, loss=262.7050, lr=1.06e-03]
Epoch 97 TRAIN:  48%|████▊     | 65/136 [00:16<00:15,  4.48it/s, epoch=97, grad_norm=137.580, loss=290.1405, lr=1.06e-03]
Epoch 97 TRAIN:  48%|████▊     | 65/136 [00:16<00:15,  4.48it/s, epoch=97, grad_norm=92.098, loss=246.8029, lr=1.06e-03] 
Epoch 97 TRAIN:  50%|█████     | 68/136 [00:16<00:14,  4.58it/s, epoch=97, grad_norm=92.098, loss=246.8029, lr=1.06e-03]
Epoch 97 TRAIN:  50%|█████     | 68/136 [00:16<00:14,  4.58it/s, epoch=97, grad_norm=115.872, loss=260.5294, lr=1.06e-03]
Epoch 97 TRAIN:  50%|█████     | 68/136 [00:17<00:14,  4.58it/s, epoch=97, grad_norm=124.966, loss=249.5520, lr=1.06e-03]
Epoch 97 TRAIN:  50%|█████     | 68/136 [00:17<00:14,  4.58it/s, epoch=97, grad_norm=80.281, loss=250.6759, lr=1.06e-03] 
Epoch 97 TRAIN:  52%|█████▏    | 71/136 [00:17<00:14,  4.38it/s, epoch=97, grad_norm=80.281, loss=250.6759, lr=1.06e-03]
Epoch 97 TRAIN:  52%|█████▏    | 71/136 [00:17<00:14,  4.38it/s, epoch=97, grad_norm=132.788, loss=296.9821, lr=1.06e-03]
Epoch 97 TRAIN:  52%|█████▏    | 71/136 [00:17<00:14,  4.38it/s, epoch=97, grad_norm=106.535, loss=225.9021, lr=1.06e-03]
Epoch 97 TRAIN:  52%|█████▏    | 71/136 [00:18<00:14,  4.38it/s, epoch=97, grad_norm=100.415, loss=257.9730, lr=1.06e-03]
Epoch 97 TRAIN:  54%|█████▍    | 74/136 [00:18<00:13,  4.67it/s, epoch=97, grad_norm=100.415, loss=257.9730, lr=1.06e-03]
Epoch 97 TRAIN:  54%|█████▍    | 74/136 [00:18<00:13,  4.67it/s, epoch=97, grad_norm=136.521, loss=328.0774, lr=1.06e-03]
Epoch 97 TRAIN:  54%|█████▍    | 74/136 [00:18<00:13,  4.67it/s, epoch=97, grad_norm=175.884, loss=179.2720, lr=1.06e-03]
Epoch 97 TRAIN:  54%|█████▍    | 74/136 [00:18<00:13,  4.67it/s, epoch=97, grad_norm=88.551, loss=236.2352, lr=1.06e-03] 
Epoch 97 TRAIN:  57%|█████▋    | 77/136 [00:18<00:13,  4.34it/s, epoch=97, grad_norm=88.551, loss=236.2352, lr=1.06e-03]
Epoch 97 TRAIN:  57%|█████▋    | 77/136 [00:19<00:13,  4.34it/s, epoch=97, grad_norm=132.373, loss=312.9277, lr=1.06e-03]
Epoch 97 TRAIN:  57%|█████▋    | 77/136 [00:19<00:13,  4.34it/s, epoch=97, grad_norm=139.019, loss=221.5486, lr=1.06e-03]
Epoch 97 TRAIN:  57%|█████▋    | 77/136 [00:19<00:13,  4.34it/s, epoch=97, grad_norm=129.447, loss=224.4711, lr=1.06e-03]
Epoch 97 TRAIN:  59%|█████▉    | 80/136 [00:19<00:12,  4.47it/s, epoch=97, grad_norm=129.447, loss=224.4711, lr=1.06e-03]
Epoch 97 TRAIN:  59%|█████▉    | 80/136 [00:19<00:12,  4.47it/s, epoch=97, grad_norm=106.583, loss=280.4061, lr=1.06e-03]
Epoch 97 TRAIN:  59%|█████▉    | 80/136 [00:20<00:12,  4.47it/s, epoch=97, grad_norm=156.845, loss=221.5071, lr=1.07e-03]
Epoch 97 TRAIN:  59%|█████▉    | 80/136 [00:20<00:12,  4.47it/s, epoch=97, grad_norm=83.830, loss=241.2679, lr=1.07e-03] 
Epoch 97 TRAIN:  61%|██████    | 83/136 [00:20<00:12,  4.41it/s, epoch=97, grad_norm=83.830, loss=241.2679, lr=1.07e-03]
Epoch 97 TRAIN:  61%|██████    | 83/136 [00:20<00:12,  4.41it/s, epoch=97, grad_norm=130.195, loss=239.3466, lr=1.07e-03]
Epoch 97 TRAIN:  61%|██████    | 83/136 [00:20<00:12,  4.41it/s, epoch=97, grad_norm=121.748, loss=180.8679, lr=1.07e-03]
Epoch 97 TRAIN:  61%|██████    | 83/136 [00:20<00:12,  4.41it/s, epoch=97, grad_norm=116.379, loss=197.3973, lr=1.07e-03]
Epoch 97 TRAIN:  63%|██████▎   | 86/136 [00:20<00:11,  4.54it/s, epoch=97, grad_norm=116.379, loss=197.3973, lr=1.07e-03]
Epoch 97 TRAIN:  63%|██████▎   | 86/136 [00:21<00:11,  4.54it/s, epoch=97, grad_norm=111.334, loss=190.2955, lr=1.07e-03]
Epoch 97 TRAIN:  63%|██████▎   | 86/136 [00:21<00:11,  4.54it/s, epoch=97, grad_norm=169.705, loss=247.5832, lr=1.07e-03]
Epoch 97 TRAIN:  63%|██████▎   | 86/136 [00:21<00:11,  4.54it/s, epoch=97, grad_norm=141.650, loss=260.2669, lr=1.07e-03]
Epoch 97 TRAIN:  65%|██████▌   | 89/136 [00:21<00:10,  4.69it/s, epoch=97, grad_norm=141.650, loss=260.2669, lr=1.07e-03]
Epoch 97 TRAIN:  65%|██████▌   | 89/136 [00:21<00:10,  4.69it/s, epoch=97, grad_norm=111.970, loss=250.2787, lr=1.07e-03]
Epoch 97 TRAIN:  65%|██████▌   | 89/136 [00:21<00:10,  4.69it/s, epoch=97, grad_norm=116.976, loss=240.1839, lr=1.07e-03]
Epoch 97 TRAIN:  65%|██████▌   | 89/136 [00:22<00:10,  4.69it/s, epoch=97, grad_norm=156.252, loss=274.2692, lr=1.07e-03]
Epoch 97 TRAIN:  68%|██████▊   | 92/136 [00:22<00:10,  4.31it/s, epoch=97, grad_norm=156.252, loss=274.2692, lr=1.07e-03]
Epoch 97 TRAIN:  68%|██████▊   | 92/136 [00:22<00:10,  4.31it/s, epoch=97, grad_norm=136.236, loss=268.3874, lr=1.07e-03]
Epoch 97 TRAIN:  68%|██████▊   | 92/136 [00:22<00:10,  4.31it/s, epoch=97, grad_norm=272.546, loss=362.0633, lr=1.07e-03]
Epoch 97 TRAIN:  68%|██████▊   | 92/136 [00:22<00:10,  4.31it/s, epoch=97, grad_norm=152.275, loss=280.4281, lr=1.07e-03]
Epoch 97 TRAIN:  70%|██████▉   | 95/136 [00:22<00:09,  4.37it/s, epoch=97, grad_norm=152.275, loss=280.4281, lr=1.07e-03]
Epoch 97 TRAIN:  70%|██████▉   | 95/136 [00:23<00:09,  4.37it/s, epoch=97, grad_norm=95.717, loss=220.2720, lr=1.07e-03] 
Epoch 97 TRAIN:  70%|██████▉   | 95/136 [00:23<00:09,  4.37it/s, epoch=97, grad_norm=173.345, loss=219.4756, lr=1.07e-03]
Epoch 97 TRAIN:  70%|██████▉   | 95/136 [00:23<00:09,  4.37it/s, epoch=97, grad_norm=232.118, loss=183.8291, lr=1.07e-03]
Epoch 97 TRAIN:  72%|███████▏  | 98/136 [00:23<00:08,  4.28it/s, epoch=97, grad_norm=232.118, loss=183.8291, lr=1.07e-03]
Epoch 97 TRAIN:  72%|███████▏  | 98/136 [00:23<00:08,  4.28it/s, epoch=97, grad_norm=113.049, loss=260.4883, lr=1.07e-03]
Epoch 97 TRAIN:  72%|███████▏  | 98/136 [00:24<00:08,  4.28it/s, epoch=97, grad_norm=102.379, loss=290.0812, lr=1.07e-03]2026-04-16 20:42:57,645 DEBUG TRAIN | steps/sec 4.196| Batch 97/100 loss 290.081177 loss_att 216.766586 loss_ctc 461.148529 th_accuracy 0.525562 lr 1.0665e-03 grad_norm 102.378586 rank 0

Epoch 97 TRAIN:  72%|███████▏  | 98/136 [00:24<00:08,  4.28it/s, epoch=97, grad_norm=189.019, loss=185.1248, lr=1.07e-03]
Epoch 97 TRAIN:  74%|███████▍  | 101/136 [00:24<00:07,  4.47it/s, epoch=97, grad_norm=189.019, loss=185.1248, lr=1.07e-03]
Epoch 97 TRAIN:  74%|███████▍  | 101/136 [00:24<00:07,  4.47it/s, epoch=97, grad_norm=169.750, loss=260.3815, lr=1.07e-03]
Epoch 97 TRAIN:  74%|███████▍  | 101/136 [00:24<00:07,  4.47it/s, epoch=97, grad_norm=154.275, loss=223.2873, lr=1.07e-03]
Epoch 97 TRAIN:  74%|███████▍  | 101/136 [00:24<00:07,  4.47it/s, epoch=97, grad_norm=258.810, loss=285.1807, lr=1.07e-03]
Epoch 97 TRAIN:  76%|███████▋  | 104/136 [00:24<00:06,  4.60it/s, epoch=97, grad_norm=258.810, loss=285.1807, lr=1.07e-03]
Epoch 97 TRAIN:  76%|███████▋  | 104/136 [00:25<00:06,  4.60it/s, epoch=97, grad_norm=96.375, loss=242.8554, lr=1.07e-03] 
Epoch 97 TRAIN:  76%|███████▋  | 104/136 [00:25<00:06,  4.60it/s, epoch=97, grad_norm=153.275, loss=282.6967, lr=1.07e-03]
Epoch 97 TRAIN:  76%|███████▋  | 104/136 [00:25<00:06,  4.60it/s, epoch=97, grad_norm=152.273, loss=235.4274, lr=1.07e-03]
Epoch 97 TRAIN:  79%|███████▊  | 107/136 [00:25<00:06,  4.59it/s, epoch=97, grad_norm=152.273, loss=235.4274, lr=1.07e-03]
Epoch 97 TRAIN:  79%|███████▊  | 107/136 [00:25<00:06,  4.59it/s, epoch=97, grad_norm=156.165, loss=175.4021, lr=1.07e-03]
Epoch 97 TRAIN:  79%|███████▊  | 107/136 [00:25<00:06,  4.59it/s, epoch=97, grad_norm=101.516, loss=213.4214, lr=1.07e-03]
Epoch 97 TRAIN:  79%|███████▊  | 107/136 [00:26<00:06,  4.59it/s, epoch=97, grad_norm=93.316, loss=226.5940, lr=1.07e-03] 
Epoch 97 TRAIN:  81%|████████  | 110/136 [00:26<00:05,  4.67it/s, epoch=97, grad_norm=93.316, loss=226.5940, lr=1.07e-03]
Epoch 97 TRAIN:  81%|████████  | 110/136 [00:26<00:05,  4.67it/s, epoch=97, grad_norm=104.204, loss=219.6396, lr=1.07e-03]
Epoch 97 TRAIN:  81%|████████  | 110/136 [00:26<00:05,  4.67it/s, epoch=97, grad_norm=115.130, loss=195.2002, lr=1.07e-03]
Epoch 97 TRAIN:  81%|████████  | 110/136 [00:26<00:05,  4.67it/s, epoch=97, grad_norm=121.778, loss=241.5114, lr=1.07e-03]
Epoch 97 TRAIN:  83%|████████▎ | 113/136 [00:26<00:04,  4.69it/s, epoch=97, grad_norm=121.778, loss=241.5114, lr=1.07e-03]
Epoch 97 TRAIN:  83%|████████▎ | 113/136 [00:26<00:04,  4.69it/s, epoch=97, grad_norm=186.547, loss=271.2965, lr=1.07e-03]
Epoch 97 TRAIN:  83%|████████▎ | 113/136 [00:27<00:04,  4.69it/s, epoch=97, grad_norm=122.501, loss=211.6357, lr=1.07e-03]
Epoch 97 TRAIN:  83%|████████▎ | 113/136 [00:27<00:04,  4.69it/s, epoch=97, grad_norm=207.395, loss=219.6916, lr=1.07e-03]
Epoch 97 TRAIN:  85%|████████▌ | 116/136 [00:27<00:04,  4.71it/s, epoch=97, grad_norm=207.395, loss=219.6916, lr=1.07e-03]
Epoch 97 TRAIN:  85%|████████▌ | 116/136 [00:27<00:04,  4.71it/s, epoch=97, grad_norm=159.131, loss=263.1058, lr=1.07e-03]
Epoch 97 TRAIN:  85%|████████▌ | 116/136 [00:27<00:04,  4.71it/s, epoch=97, grad_norm=124.244, loss=229.3889, lr=1.07e-03]
Epoch 97 TRAIN:  85%|████████▌ | 116/136 [00:28<00:04,  4.71it/s, epoch=97, grad_norm=176.864, loss=250.4899, lr=1.07e-03]
Epoch 97 TRAIN:  88%|████████▊ | 119/136 [00:28<00:03,  4.45it/s, epoch=97, grad_norm=176.864, loss=250.4899, lr=1.07e-03]
Epoch 97 TRAIN:  88%|████████▊ | 119/136 [00:28<00:03,  4.45it/s, epoch=97, grad_norm=218.880, loss=314.4807, lr=1.07e-03]
Epoch 97 TRAIN:  88%|████████▊ | 119/136 [00:28<00:03,  4.45it/s, epoch=97, grad_norm=115.739, loss=252.9176, lr=1.07e-03]
Epoch 97 TRAIN:  88%|████████▊ | 119/136 [00:28<00:03,  4.45it/s, epoch=97, grad_norm=343.819, loss=285.7122, lr=1.07e-03]
Epoch 97 TRAIN:  90%|████████▉ | 122/136 [00:28<00:03,  4.49it/s, epoch=97, grad_norm=343.819, loss=285.7122, lr=1.07e-03]
Epoch 97 TRAIN:  90%|████████▉ | 122/136 [00:29<00:03,  4.49it/s, epoch=97, grad_norm=273.799, loss=289.7828, lr=1.07e-03]
Epoch 97 TRAIN:  90%|████████▉ | 122/136 [00:29<00:03,  4.49it/s, epoch=97, grad_norm=172.424, loss=244.7847, lr=1.07e-03]
Epoch 97 TRAIN:  90%|████████▉ | 122/136 [00:29<00:03,  4.49it/s, epoch=97, grad_norm=223.779, loss=289.0081, lr=1.07e-03]
Epoch 97 TRAIN:  92%|█████████▏| 125/136 [00:29<00:02,  4.61it/s, epoch=97, grad_norm=223.779, loss=289.0081, lr=1.07e-03]
Epoch 97 TRAIN:  92%|█████████▏| 125/136 [00:29<00:02,  4.61it/s, epoch=97, grad_norm=271.248, loss=251.6886, lr=1.07e-03]
Epoch 97 TRAIN:  92%|█████████▏| 125/136 [00:29<00:02,  4.61it/s, epoch=97, grad_norm=255.673, loss=248.7224, lr=1.07e-03]
Epoch 97 TRAIN:  92%|█████████▏| 125/136 [00:30<00:02,  4.61it/s, epoch=97, grad_norm=157.588, loss=273.3726, lr=1.07e-03]
Epoch 97 TRAIN:  94%|█████████▍| 128/136 [00:30<00:01,  4.51it/s, epoch=97, grad_norm=157.588, loss=273.3726, lr=1.07e-03]
Epoch 97 TRAIN:  94%|█████████▍| 128/136 [00:30<00:01,  4.51it/s, epoch=97, grad_norm=157.647, loss=293.2182, lr=1.07e-03]
Epoch 97 TRAIN:  94%|█████████▍| 128/136 [00:30<00:01,  4.51it/s, epoch=97, grad_norm=179.255, loss=281.1591, lr=1.07e-03]
Epoch 97 TRAIN:  94%|█████████▍| 128/136 [00:30<00:01,  4.51it/s, epoch=97, grad_norm=225.249, loss=298.2025, lr=1.07e-03]
Epoch 97 TRAIN:  96%|█████████▋| 131/136 [00:30<00:01,  4.69it/s, epoch=97, grad_norm=225.249, loss=298.2025, lr=1.07e-03]
Epoch 97 TRAIN:  96%|█████████▋| 131/136 [00:30<00:01,  4.69it/s, epoch=97, grad_norm=169.617, loss=244.5284, lr=1.07e-03]
Epoch 97 TRAIN:  96%|█████████▋| 131/136 [00:31<00:01,  4.69it/s, epoch=97, grad_norm=223.213, loss=296.2973, lr=1.07e-03]
Epoch 97 TRAIN:  96%|█████████▋| 131/136 [00:31<00:01,  4.69it/s, epoch=97, grad_norm=336.429, loss=356.1902, lr=1.07e-03]
Epoch 97 TRAIN:  99%|█████████▊| 134/136 [00:31<00:00,  4.53it/s, epoch=97, grad_norm=336.429, loss=356.1902, lr=1.07e-03]
Epoch 97 TRAIN:  99%|█████████▊| 134/136 [00:31<00:00,  4.53it/s, epoch=97, grad_norm=300.583, loss=323.9587, lr=1.07e-03]
Epoch 97 TRAIN:  99%|█████████▉| 135/136 [00:31<00:00,  4.28it/s, epoch=97, grad_norm=300.583, loss=323.9587, lr=1.07e-03]

CV:   0%|          | 0/6 [00:00<?, ?it/s]/home/rookie/Documents/batuka/projects/wenet/wenet/wenet/utils/train_utils.py:689: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  amp_autocast(enabled=dtype is not None,

CV:  17%|█▋        | 1/6 [00:00<00:03,  1.33it/s]
                                                 
2026-04-16 20:43:05,932 INFO Epoch 97 Step 13366 CV info lr 1.0693e-03 cv_loss 457.7505937056108 rank 0 acc 0.41581375400225323
2026-04-16 20:43:05,934 INFO [Rank 0] Checkpoint: save to checkpoint exp/conformer_mongolian/epoch_97.pt
2026-04-16 20:43:06,047 INFO Epoch 98 Step 13366 TRAIN info lr 1.0693e-03 rank 0
2026-04-16 20:43:06,070 INFO using accumulate grad, new batch size is 1 times larger than before

Epoch 98 TRAIN:   0%|          | 0/135 [00:00<?, ?it/s]
Epoch 98 TRAIN:   0%|          | 0/135 [00:04<?, ?it/s, epoch=98, grad_norm=283.191, loss=266.9973, lr=1.07e-03]
Epoch 98 TRAIN:   1%|          | 1/135 [00:04<09:09,  4.10s/it, epoch=98, grad_norm=283.191, loss=266.9973, lr=1.07e-03]
Epoch 98 TRAIN:   1%|          | 1/135 [00:04<09:09,  4.10s/it, epoch=98, grad_norm=384.521, loss=331.4077, lr=1.07e-03]
Epoch 98 TRAIN:   1%|          | 1/135 [00:04<09:09,  4.10s/it, epoch=98, grad_norm=512.172, loss=411.1221, lr=1.07e-03]
Epoch 98 TRAIN:   1%|          | 1/135 [00:04<09:09,  4.10s/it, epoch=98, grad_norm=255.836, loss=238.5031, lr=1.07e-03]
Epoch 98 TRAIN:   1%|          | 1/135 [00:04<09:09,  4.10s/it, epoch=98, grad_norm=306.237, loss=286.6332, lr=1.07e-03]
Epoch 98 TRAIN:   1%|          | 1/135 [00:04<09:09,  4.10s/it, epoch=98, grad_norm=320.677, loss=280.2118, lr=1.07e-03]
Epoch 98 TRAIN:   1%|          | 1/135 [00:04<09:09,  4.10s/it, epoch=98, grad_norm=224.471, loss=255.6421, lr=1.07e-03]
Epoch 98 TRAIN:   1%|          | 1/135 [00:04<09:09,  4.10s/it, epoch=98, grad_norm=210.820, loss=241.8410, lr=1.07e-03]
Epoch 98 TRAIN:   1%|          | 1/135 [00:04<09:09,  4.10s/it, epoch=98, grad_norm=244.323, loss=290.6510, lr=1.07e-03]
Epoch 98 TRAIN:   7%|▋         | 9/135 [00:04<00:50,  2.47it/s, epoch=98, grad_norm=244.323, loss=290.6510, lr=1.07e-03]
Epoch 98 TRAIN:   7%|▋         | 9/135 [00:04<00:50,  2.47it/s, epoch=98, grad_norm=152.136, loss=252.6436, lr=1.07e-03]
Epoch 98 TRAIN:   7%|▋         | 9/135 [00:05<00:50,  2.47it/s, epoch=98, grad_norm=152.724, loss=289.5994, lr=1.07e-03]
Epoch 98 TRAIN:   7%|▋         | 9/135 [00:05<00:50,  2.47it/s, epoch=98, grad_norm=135.887, loss=298.2959, lr=1.07e-03]
Epoch 98 TRAIN:   7%|▋         | 9/135 [00:05<00:50,  2.47it/s, epoch=98, grad_norm=114.466, loss=288.7222, lr=1.07e-03]
Epoch 98 TRAIN:  10%|▉         | 13/135 [00:05<00:38,  3.20it/s, epoch=98, grad_norm=114.466, loss=288.7222, lr=1.07e-03]
Epoch 98 TRAIN:  10%|▉         | 13/135 [00:05<00:38,  3.20it/s, epoch=98, grad_norm=100.633, loss=275.0683, lr=1.07e-03]
Epoch 98 TRAIN:  10%|▉         | 13/135 [00:05<00:38,  3.20it/s, epoch=98, grad_norm=128.601, loss=265.9476, lr=1.07e-03]
Epoch 98 TRAIN:  10%|▉         | 13/135 [00:06<00:38,  3.20it/s, epoch=98, grad_norm=134.801, loss=241.8338, lr=1.07e-03]
Epoch 98 TRAIN:  10%|▉         | 13/135 [00:06<00:38,  3.20it/s, epoch=98, grad_norm=200.888, loss=225.0801, lr=1.07e-03]
Epoch 98 TRAIN:  13%|█▎        | 17/135 [00:06<00:32,  3.65it/s, epoch=98, grad_norm=200.888, loss=225.0801, lr=1.07e-03]
Epoch 98 TRAIN:  13%|█▎        | 17/135 [00:06<00:32,  3.65it/s, epoch=98, grad_norm=157.362, loss=318.8423, lr=1.07e-03]
Epoch 98 TRAIN:  13%|█▎        | 17/135 [00:06<00:32,  3.65it/s, epoch=98, grad_norm=136.229, loss=265.4612, lr=1.07e-03]
Epoch 98 TRAIN:  13%|█▎        | 17/135 [00:06<00:32,  3.65it/s, epoch=98, grad_norm=154.332, loss=241.2973, lr=1.07e-03]
Epoch 98 TRAIN:  15%|█▍        | 20/135 [00:06<00:28,  4.07it/s, epoch=98, grad_norm=154.332, loss=241.2973, lr=1.07e-03]
Epoch 98 TRAIN:  15%|█▍        | 20/135 [00:07<00:28,  4.07it/s, epoch=98, grad_norm=87.855, loss=259.9198, lr=1.07e-03] 
Epoch 98 TRAIN:  15%|█▍        | 20/135 [00:07<00:28,  4.07it/s, epoch=98, grad_norm=117.015, loss=270.5635, lr=1.07e-03]
Epoch 98 TRAIN:  15%|█▍        | 20/135 [00:07<00:28,  4.07it/s, epoch=98, grad_norm=88.258, loss=243.7390, lr=1.07e-03] 
Epoch 98 TRAIN:  17%|█▋        | 23/135 [00:07<00:26,  4.19it/s, epoch=98, grad_norm=88.258, loss=243.7390, lr=1.07e-03]
Epoch 98 TRAIN:  17%|█▋        | 23/135 [00:07<00:26,  4.19it/s, epoch=98, grad_norm=97.421, loss=255.9630, lr=1.07e-03]
Epoch 98 TRAIN:  17%|█▋        | 23/135 [00:07<00:26,  4.19it/s, epoch=98, grad_norm=86.823, loss=286.9198, lr=1.07e-03]
Epoch 98 TRAIN:  17%|█▋        | 23/135 [00:08<00:26,  4.19it/s, epoch=98, grad_norm=99.793, loss=264.5497, lr=1.07e-03]
Epoch 98 TRAIN:  19%|█▉        | 26/135 [00:08<00:24,  4.42it/s, epoch=98, grad_norm=99.793, loss=264.5497, lr=1.07e-03]
Epoch 98 TRAIN:  19%|█▉        | 26/135 [00:08<00:24,  4.42it/s, epoch=98, grad_norm=130.639, loss=226.3762, lr=1.07e-03]
Epoch 98 TRAIN:  19%|█▉        | 26/135 [00:08<00:24,  4.42it/s, epoch=98, grad_norm=72.824, loss=269.7022, lr=1.07e-03] 
Epoch 98 TRAIN:  19%|█▉        | 26/135 [00:08<00:24,  4.42it/s, epoch=98, grad_norm=124.727, loss=211.8190, lr=1.07e-03]
Epoch 98 TRAIN:  21%|██▏       | 29/135 [00:08<00:22,  4.68it/s, epoch=98, grad_norm=124.727, loss=211.8190, lr=1.07e-03]
Epoch 98 TRAIN:  21%|██▏       | 29/135 [00:08<00:22,  4.68it/s, epoch=98, grad_norm=126.565, loss=300.0429, lr=1.07e-03]
Epoch 98 TRAIN:  21%|██▏       | 29/135 [00:08<00:22,  4.68it/s, epoch=98, grad_norm=95.860, loss=263.6758, lr=1.07e-03] 
Epoch 98 TRAIN:  21%|██▏       | 29/135 [00:09<00:22,  4.68it/s, epoch=98, grad_norm=94.110, loss=281.4113, lr=1.07e-03]
Epoch 98 TRAIN:  24%|██▎       | 32/135 [00:09<00:21,  4.78it/s, epoch=98, grad_norm=94.110, loss=281.4113, lr=1.07e-03]
Epoch 98 TRAIN:  24%|██▎       | 32/135 [00:09<00:21,  4.78it/s, epoch=98, grad_norm=73.386, loss=265.7904, lr=1.07e-03]
Epoch 98 TRAIN:  24%|██▎       | 32/135 [00:09<00:21,  4.78it/s, epoch=98, grad_norm=87.922, loss=261.9769, lr=1.07e-03]
Epoch 98 TRAIN:  24%|██▎       | 32/135 [00:09<00:21,  4.78it/s, epoch=98, grad_norm=88.458, loss=237.3891, lr=1.07e-03]
Epoch 98 TRAIN:  26%|██▌       | 35/135 [00:09<00:20,  4.77it/s, epoch=98, grad_norm=88.458, loss=237.3891, lr=1.07e-03]
Epoch 98 TRAIN:  26%|██▌       | 35/135 [00:10<00:20,  4.77it/s, epoch=98, grad_norm=99.558, loss=248.4940, lr=1.07e-03]
Epoch 98 TRAIN:  26%|██▌       | 35/135 [00:10<00:20,  4.77it/s, epoch=98, grad_norm=88.170, loss=284.6034, lr=1.07e-03]
Epoch 98 TRAIN:  26%|██▌       | 35/135 [00:10<00:20,  4.77it/s, epoch=98, grad_norm=83.006, loss=249.5305, lr=1.07e-03]
Epoch 98 TRAIN:  28%|██▊       | 38/135 [00:10<00:19,  4.86it/s, epoch=98, grad_norm=83.006, loss=249.5305, lr=1.07e-03]
Epoch 98 TRAIN:  28%|██▊       | 38/135 [00:10<00:19,  4.86it/s, epoch=98, grad_norm=85.528, loss=255.3020, lr=1.07e-03]
Epoch 98 TRAIN:  28%|██▊       | 38/135 [00:10<00:19,  4.86it/s, epoch=98, grad_norm=81.268, loss=268.1339, lr=1.07e-03]
Epoch 98 TRAIN:  28%|██▊       | 38/135 [00:11<00:19,  4.86it/s, epoch=98, grad_norm=94.213, loss=261.9790, lr=1.07e-03]
Epoch 98 TRAIN:  30%|███       | 41/135 [00:11<00:19,  4.76it/s, epoch=98, grad_norm=94.213, loss=261.9790, lr=1.07e-03]
Epoch 98 TRAIN:  30%|███       | 41/135 [00:11<00:19,  4.76it/s, epoch=98, grad_norm=87.223, loss=255.2438, lr=1.07e-03]
Epoch 98 TRAIN:  30%|███       | 41/135 [00:11<00:19,  4.76it/s, epoch=98, grad_norm=150.232, loss=299.7534, lr=1.07e-03]
Epoch 98 TRAIN:  30%|███       | 41/135 [00:11<00:19,  4.76it/s, epoch=98, grad_norm=136.160, loss=258.8439, lr=1.07e-03]
Epoch 98 TRAIN:  33%|███▎      | 44/135 [00:11<00:19,  4.69it/s, epoch=98, grad_norm=136.160, loss=258.8439, lr=1.07e-03]
Epoch 98 TRAIN:  33%|███▎      | 44/135 [00:11<00:19,  4.69it/s, epoch=98, grad_norm=68.778, loss=254.7514, lr=1.07e-03] 
Epoch 98 TRAIN:  33%|███▎      | 44/135 [00:12<00:19,  4.69it/s, epoch=98, grad_norm=92.653, loss=287.5908, lr=1.07e-03]
Epoch 98 TRAIN:  33%|███▎      | 44/135 [00:12<00:19,  4.69it/s, epoch=98, grad_norm=91.619, loss=249.1782, lr=1.07e-03]
Epoch 98 TRAIN:  35%|███▍      | 47/135 [00:12<00:18,  4.67it/s, epoch=98, grad_norm=91.619, loss=249.1782, lr=1.07e-03]
Epoch 98 TRAIN:  35%|███▍      | 47/135 [00:12<00:18,  4.67it/s, epoch=98, grad_norm=108.445, loss=271.6196, lr=1.07e-03]
Epoch 98 TRAIN:  35%|███▍      | 47/135 [00:12<00:18,  4.67it/s, epoch=98, grad_norm=99.462, loss=249.4512, lr=1.07e-03] 
Epoch 98 TRAIN:  35%|███▍      | 47/135 [00:13<00:18,  4.67it/s, epoch=98, grad_norm=86.266, loss=242.3075, lr=1.07e-03]
Epoch 98 TRAIN:  37%|███▋      | 50/135 [00:13<00:18,  4.60it/s, epoch=98, grad_norm=86.266, loss=242.3075, lr=1.07e-03]
Epoch 98 TRAIN:  37%|███▋      | 50/135 [00:13<00:18,  4.60it/s, epoch=98, grad_norm=129.498, loss=263.8352, lr=1.07e-03]
Epoch 98 TRAIN:  37%|███▋      | 50/135 [00:13<00:18,  4.60it/s, epoch=98, grad_norm=104.553, loss=225.4678, lr=1.07e-03]
Epoch 98 TRAIN:  37%|███▋      | 50/135 [00:13<00:18,  4.60it/s, epoch=98, grad_norm=119.032, loss=296.2855, lr=1.07e-03]
Epoch 98 TRAIN:  39%|███▉      | 53/135 [00:13<00:18,  4.44it/s, epoch=98, grad_norm=119.032, loss=296.2855, lr=1.07e-03]
Epoch 98 TRAIN:  39%|███▉      | 53/135 [00:13<00:18,  4.44it/s, epoch=98, grad_norm=107.082, loss=247.1648, lr=1.07e-03]
Epoch 98 TRAIN:  39%|███▉      | 53/135 [00:14<00:18,  4.44it/s, epoch=98, grad_norm=105.535, loss=253.7568, lr=1.07e-03]
Epoch 98 TRAIN:  39%|███▉      | 53/135 [00:14<00:18,  4.44it/s, epoch=98, grad_norm=108.151, loss=278.0805, lr=1.07e-03]
Epoch 98 TRAIN:  41%|████▏     | 56/135 [00:14<00:17,  4.52it/s, epoch=98, grad_norm=108.151, loss=278.0805, lr=1.07e-03]
Epoch 98 TRAIN:  41%|████▏     | 56/135 [00:14<00:17,  4.52it/s, epoch=98, grad_norm=99.676, loss=224.4922, lr=1.07e-03] 
Epoch 98 TRAIN:  41%|████▏     | 56/135 [00:14<00:17,  4.52it/s, epoch=98, grad_norm=242.107, loss=313.3990, lr=1.07e-03]
Epoch 98 TRAIN:  41%|████▏     | 56/135 [00:15<00:17,  4.52it/s, epoch=98, grad_norm=87.570, loss=259.2348, lr=1.07e-03] 
Epoch 98 TRAIN:  44%|████▎     | 59/135 [00:15<00:16,  4.65it/s, epoch=98, grad_norm=87.570, loss=259.2348, lr=1.07e-03]
Epoch 98 TRAIN:  44%|████▎     | 59/135 [00:15<00:16,  4.65it/s, epoch=98, grad_norm=74.559, loss=245.3968, lr=1.07e-03]
Epoch 98 TRAIN:  44%|████▎     | 59/135 [00:15<00:16,  4.65it/s, epoch=98, grad_norm=142.513, loss=295.8349, lr=1.07e-03]
Epoch 98 TRAIN:  44%|████▎     | 59/135 [00:15<00:16,  4.65it/s, epoch=98, grad_norm=125.943, loss=280.4955, lr=1.07e-03]
Epoch 98 TRAIN:  46%|████▌     | 62/135 [00:15<00:16,  4.53it/s, epoch=98, grad_norm=125.943, loss=280.4955, lr=1.07e-03]
Epoch 98 TRAIN:  46%|████▌     | 62/135 [00:15<00:16,  4.53it/s, epoch=98, grad_norm=108.298, loss=250.6982, lr=1.07e-03]
Epoch 98 TRAIN:  46%|████▌     | 62/135 [00:16<00:16,  4.53it/s, epoch=98, grad_norm=145.897, loss=249.5950, lr=1.07e-03]
Epoch 98 TRAIN:  46%|████▌     | 62/135 [00:16<00:16,  4.53it/s, epoch=98, grad_norm=143.822, loss=226.8721, lr=1.07e-03]
Epoch 98 TRAIN:  48%|████▊     | 65/135 [00:16<00:15,  4.62it/s, epoch=98, grad_norm=143.822, loss=226.8721, lr=1.07e-03]
Epoch 98 TRAIN:  48%|████▊     | 65/135 [00:16<00:15,  4.62it/s, epoch=98, grad_norm=117.931, loss=230.2709, lr=1.07e-03]
Epoch 98 TRAIN:  48%|████▊     | 65/135 [00:16<00:15,  4.62it/s, epoch=98, grad_norm=91.229, loss=261.9273, lr=1.07e-03] 
Epoch 98 TRAIN:  48%|████▊     | 65/135 [00:17<00:15,  4.62it/s, epoch=98, grad_norm=128.074, loss=290.1235, lr=1.07e-03]
Epoch 98 TRAIN:  50%|█████     | 68/135 [00:17<00:14,  4.56it/s, epoch=98, grad_norm=128.074, loss=290.1235, lr=1.07e-03]
Epoch 98 TRAIN:  50%|█████     | 68/135 [00:17<00:14,  4.56it/s, epoch=98, grad_norm=107.567, loss=258.8291, lr=1.07e-03]
Epoch 98 TRAIN:  50%|█████     | 68/135 [00:17<00:14,  4.56it/s, epoch=98, grad_norm=79.299, loss=261.4471, lr=1.07e-03] 
Epoch 98 TRAIN:  50%|█████     | 68/135 [00:17<00:14,  4.56it/s, epoch=98, grad_norm=88.115, loss=261.7997, lr=1.07e-03]
Epoch 98 TRAIN:  53%|█████▎    | 71/135 [00:17<00:14,  4.56it/s, epoch=98, grad_norm=88.115, loss=261.7997, lr=1.07e-03]
Epoch 98 TRAIN:  53%|█████▎    | 71/135 [00:17<00:14,  4.56it/s, epoch=98, grad_norm=94.079, loss=267.7330, lr=1.08e-03]
Epoch 98 TRAIN:  53%|█████▎    | 71/135 [00:18<00:14,  4.56it/s, epoch=98, grad_norm=71.151, loss=260.5426, lr=1.08e-03]
Epoch 98 TRAIN:  53%|█████▎    | 71/135 [00:18<00:14,  4.56it/s, epoch=98, grad_norm=96.366, loss=293.0142, lr=1.08e-03]
Epoch 98 TRAIN:  55%|█████▍    | 74/135 [00:18<00:13,  4.46it/s, epoch=98, grad_norm=96.366, loss=293.0142, lr=1.08e-03]
Epoch 98 TRAIN:  55%|█████▍    | 74/135 [00:18<00:13,  4.46it/s, epoch=98, grad_norm=73.627, loss=251.5708, lr=1.08e-03]
Epoch 98 TRAIN:  55%|█████▍    | 74/135 [00:18<00:13,  4.46it/s, epoch=98, grad_norm=121.314, loss=238.9134, lr=1.08e-03]
Epoch 98 TRAIN:  55%|█████▍    | 74/135 [00:19<00:13,  4.46it/s, epoch=98, grad_norm=87.037, loss=255.8339, lr=1.08e-03] 
Epoch 98 TRAIN:  57%|█████▋    | 77/135 [00:19<00:13,  4.44it/s, epoch=98, grad_norm=87.037, loss=255.8339, lr=1.08e-03]
Epoch 98 TRAIN:  57%|█████▋    | 77/135 [00:19<00:13,  4.44it/s, epoch=98, grad_norm=117.572, loss=291.0125, lr=1.08e-03]
Epoch 98 TRAIN:  57%|█████▋    | 77/135 [00:19<00:13,  4.44it/s, epoch=98, grad_norm=200.696, loss=272.7876, lr=1.08e-03]
Epoch 98 TRAIN:  57%|█████▋    | 77/135 [00:19<00:13,  4.44it/s, epoch=98, grad_norm=250.518, loss=176.6481, lr=1.08e-03]
Epoch 98 TRAIN:  59%|█████▉    | 80/135 [00:19<00:12,  4.58it/s, epoch=98, grad_norm=250.518, loss=176.6481, lr=1.08e-03]
Epoch 98 TRAIN:  59%|█████▉    | 80/135 [00:19<00:12,  4.58it/s, epoch=98, grad_norm=173.823, loss=275.3009, lr=1.08e-03]
Epoch 98 TRAIN:  59%|█████▉    | 80/135 [00:20<00:12,  4.58it/s, epoch=98, grad_norm=124.706, loss=244.5172, lr=1.08e-03]
Epoch 98 TRAIN:  59%|█████▉    | 80/135 [00:20<00:12,  4.58it/s, epoch=98, grad_norm=93.059, loss=240.8575, lr=1.08e-03] 
Epoch 98 TRAIN:  61%|██████▏   | 83/135 [00:20<00:11,  4.47it/s, epoch=98, grad_norm=93.059, loss=240.8575, lr=1.08e-03]
Epoch 98 TRAIN:  61%|██████▏   | 83/135 [00:20<00:11,  4.47it/s, epoch=98, grad_norm=123.520, loss=279.0920, lr=1.08e-03]
Epoch 98 TRAIN:  61%|██████▏   | 83/135 [00:20<00:11,  4.47it/s, epoch=98, grad_norm=125.898, loss=206.5939, lr=1.08e-03]
Epoch 98 TRAIN:  61%|██████▏   | 83/135 [00:21<00:11,  4.47it/s, epoch=98, grad_norm=93.858, loss=240.7590, lr=1.08e-03] 
Epoch 98 TRAIN:  64%|██████▎   | 86/135 [00:21<00:11,  4.37it/s, epoch=98, grad_norm=93.858, loss=240.7590, lr=1.08e-03]
Epoch 98 TRAIN:  64%|██████▎   | 86/135 [00:21<00:11,  4.37it/s, epoch=98, grad_norm=93.118, loss=249.5576, lr=1.08e-03]
Epoch 98 TRAIN:  64%|██████▎   | 86/135 [00:21<00:11,  4.37it/s, epoch=98, grad_norm=108.270, loss=243.7459, lr=1.08e-03]
Epoch 98 TRAIN:  64%|██████▎   | 86/135 [00:21<00:11,  4.37it/s, epoch=98, grad_norm=76.934, loss=233.7573, lr=1.08e-03] 
Epoch 98 TRAIN:  66%|██████▌   | 89/135 [00:21<00:10,  4.43it/s, epoch=98, grad_norm=76.934, loss=233.7573, lr=1.08e-03]
Epoch 98 TRAIN:  66%|██████▌   | 89/135 [00:22<00:10,  4.43it/s, epoch=98, grad_norm=111.382, loss=193.1657, lr=1.08e-03]
Epoch 98 TRAIN:  66%|██████▌   | 89/135 [00:22<00:10,  4.43it/s, epoch=98, grad_norm=123.948, loss=253.6367, lr=1.08e-03]
Epoch 98 TRAIN:  66%|██████▌   | 89/135 [00:22<00:10,  4.43it/s, epoch=98, grad_norm=156.040, loss=242.4156, lr=1.08e-03]
Epoch 98 TRAIN:  68%|██████▊   | 92/135 [00:22<00:09,  4.59it/s, epoch=98, grad_norm=156.040, loss=242.4156, lr=1.08e-03]
Epoch 98 TRAIN:  68%|██████▊   | 92/135 [00:22<00:09,  4.59it/s, epoch=98, grad_norm=98.236, loss=230.4324, lr=1.08e-03] 
Epoch 98 TRAIN:  68%|██████▊   | 92/135 [00:22<00:09,  4.59it/s, epoch=98, grad_norm=127.811, loss=250.6926, lr=1.08e-03]
Epoch 98 TRAIN:  68%|██████▊   | 92/135 [00:23<00:09,  4.59it/s, epoch=98, grad_norm=252.151, loss=278.8152, lr=1.08e-03]
Epoch 98 TRAIN:  70%|███████   | 95/135 [00:23<00:08,  4.47it/s, epoch=98, grad_norm=252.151, loss=278.8152, lr=1.08e-03]
Epoch 98 TRAIN:  70%|███████   | 95/135 [00:23<00:08,  4.47it/s, epoch=98, grad_norm=133.611, loss=272.3431, lr=1.08e-03]
Epoch 98 TRAIN:  70%|███████   | 95/135 [00:23<00:08,  4.47it/s, epoch=98, grad_norm=173.955, loss=204.0656, lr=1.08e-03]
Epoch 98 TRAIN:  70%|███████   | 95/135 [00:23<00:08,  4.47it/s, epoch=98, grad_norm=285.628, loss=303.4479, lr=1.08e-03]
Epoch 98 TRAIN:  73%|███████▎  | 98/135 [00:23<00:08,  4.46it/s, epoch=98, grad_norm=285.628, loss=303.4479, lr=1.08e-03]
Epoch 98 TRAIN:  73%|███████▎  | 98/135 [00:23<00:08,  4.46it/s, epoch=98, grad_norm=156.854, loss=257.8944, lr=1.08e-03]
Epoch 98 TRAIN:  73%|███████▎  | 98/135 [00:24<00:08,  4.46it/s, epoch=98, grad_norm=179.689, loss=267.6559, lr=1.08e-03]2026-04-16 20:43:30,228 DEBUG TRAIN | steps/sec 4.143| Batch 98/100 loss 267.655914 loss_att 189.976242 loss_ctc 448.908447 th_accuracy 0.553600 lr 1.0773e-03 grad_norm 179.688751 rank 0

Epoch 98 TRAIN:  73%|███████▎  | 98/135 [00:24<00:08,  4.46it/s, epoch=98, grad_norm=162.657, loss=193.6107, lr=1.08e-03]
Epoch 98 TRAIN:  75%|███████▍  | 101/135 [00:24<00:07,  4.61it/s, epoch=98, grad_norm=162.657, loss=193.6107, lr=1.08e-03]
Epoch 98 TRAIN:  75%|███████▍  | 101/135 [00:24<00:07,  4.61it/s, epoch=98, grad_norm=337.172, loss=288.7894, lr=1.08e-03]
Epoch 98 TRAIN:  75%|███████▍  | 101/135 [00:24<00:07,  4.61it/s, epoch=98, grad_norm=224.258, loss=274.8717, lr=1.08e-03]
Epoch 98 TRAIN:  75%|███████▍  | 101/135 [00:24<00:07,  4.61it/s, epoch=98, grad_norm=145.603, loss=271.4914, lr=1.08e-03]
Epoch 98 TRAIN:  77%|███████▋  | 104/135 [00:24<00:06,  4.65it/s, epoch=98, grad_norm=145.603, loss=271.4914, lr=1.08e-03]
Epoch 98 TRAIN:  77%|███████▋  | 104/135 [00:25<00:06,  4.65it/s, epoch=98, grad_norm=140.953, loss=184.2481, lr=1.08e-03]
Epoch 98 TRAIN:  77%|███████▋  | 104/135 [00:25<00:06,  4.65it/s, epoch=98, grad_norm=209.747, loss=196.5985, lr=1.08e-03]
Epoch 98 TRAIN:  77%|███████▋  | 104/135 [00:25<00:06,  4.65it/s, epoch=98, grad_norm=112.838, loss=194.9805, lr=1.08e-03]
Epoch 98 TRAIN:  79%|███████▉  | 107/135 [00:25<00:06,  4.57it/s, epoch=98, grad_norm=112.838, loss=194.9805, lr=1.08e-03]
Epoch 98 TRAIN:  79%|███████▉  | 107/135 [00:25<00:06,  4.57it/s, epoch=98, grad_norm=274.999, loss=284.2646, lr=1.08e-03]
Epoch 98 TRAIN:  79%|███████▉  | 107/135 [00:26<00:06,  4.57it/s, epoch=98, grad_norm=226.020, loss=256.3975, lr=1.08e-03]
Epoch 98 TRAIN:  79%|███████▉  | 107/135 [00:26<00:06,  4.57it/s, epoch=98, grad_norm=135.940, loss=217.9496, lr=1.08e-03]
Epoch 98 TRAIN:  81%|████████▏ | 110/135 [00:26<00:05,  4.43it/s, epoch=98, grad_norm=135.940, loss=217.9496, lr=1.08e-03]
Epoch 98 TRAIN:  81%|████████▏ | 110/135 [00:26<00:05,  4.43it/s, epoch=98, grad_norm=101.132, loss=225.1602, lr=1.08e-03]
Epoch 98 TRAIN:  81%|████████▏ | 110/135 [00:26<00:05,  4.43it/s, epoch=98, grad_norm=132.410, loss=221.5357, lr=1.08e-03]
Epoch 98 TRAIN:  81%|████████▏ | 110/135 [00:27<00:05,  4.43it/s, epoch=98, grad_norm=117.362, loss=209.9738, lr=1.08e-03]
Epoch 98 TRAIN:  84%|████████▎ | 113/135 [00:27<00:04,  4.50it/s, epoch=98, grad_norm=117.362, loss=209.9738, lr=1.08e-03]
Epoch 98 TRAIN:  84%|████████▎ | 113/135 [00:27<00:04,  4.50it/s, epoch=98, grad_norm=126.791, loss=233.1538, lr=1.08e-03]
Epoch 98 TRAIN:  84%|████████▎ | 113/135 [00:27<00:04,  4.50it/s, epoch=98, grad_norm=115.656, loss=251.0911, lr=1.08e-03]
Epoch 98 TRAIN:  84%|████████▎ | 113/135 [00:27<00:04,  4.50it/s, epoch=98, grad_norm=186.090, loss=298.3300, lr=1.08e-03]
Epoch 98 TRAIN:  86%|████████▌ | 116/135 [00:27<00:04,  4.37it/s, epoch=98, grad_norm=186.090, loss=298.3300, lr=1.08e-03]
Epoch 98 TRAIN:  86%|████████▌ | 116/135 [00:28<00:04,  4.37it/s, epoch=98, grad_norm=138.665, loss=253.8514, lr=1.08e-03]
Epoch 98 TRAIN:  86%|████████▌ | 116/135 [00:28<00:04,  4.37it/s, epoch=98, grad_norm=148.572, loss=285.8515, lr=1.08e-03]
Epoch 98 TRAIN:  86%|████████▌ | 116/135 [00:28<00:04,  4.37it/s, epoch=98, grad_norm=191.225, loss=303.9583, lr=1.08e-03]
Epoch 98 TRAIN:  88%|████████▊ | 119/135 [00:28<00:03,  4.28it/s, epoch=98, grad_norm=191.225, loss=303.9583, lr=1.08e-03]
Epoch 98 TRAIN:  88%|████████▊ | 119/135 [00:28<00:03,  4.28it/s, epoch=98, grad_norm=211.733, loss=329.6946, lr=1.08e-03]
Epoch 98 TRAIN:  88%|████████▊ | 119/135 [00:28<00:03,  4.28it/s, epoch=98, grad_norm=146.654, loss=249.2212, lr=1.08e-03]
Epoch 98 TRAIN:  88%|████████▊ | 119/135 [00:28<00:03,  4.28it/s, epoch=98, grad_norm=178.158, loss=261.5292, lr=1.08e-03]
Epoch 98 TRAIN:  88%|████████▊ | 119/135 [00:29<00:03,  4.28it/s, epoch=98, grad_norm=108.649, loss=174.3905, lr=1.08e-03]
Epoch 98 TRAIN:  91%|█████████ | 123/135 [00:29<00:02,  4.59it/s, epoch=98, grad_norm=108.649, loss=174.3905, lr=1.08e-03]
Epoch 98 TRAIN:  91%|█████████ | 123/135 [00:29<00:02,  4.59it/s, epoch=98, grad_norm=165.998, loss=287.4069, lr=1.08e-03]
Epoch 98 TRAIN:  91%|█████████ | 123/135 [00:29<00:02,  4.59it/s, epoch=98, grad_norm=176.471, loss=247.4398, lr=1.08e-03]
Epoch 98 TRAIN:  91%|█████████ | 123/135 [00:29<00:02,  4.59it/s, epoch=98, grad_norm=180.288, loss=293.8123, lr=1.08e-03]
Epoch 98 TRAIN:  91%|█████████ | 123/135 [00:29<00:02,  4.59it/s, epoch=98, grad_norm=136.489, loss=237.5174, lr=1.08e-03]
Epoch 98 TRAIN:  94%|█████████▍| 127/135 [00:29<00:01,  5.00it/s, epoch=98, grad_norm=136.489, loss=237.5174, lr=1.08e-03]
Epoch 98 TRAIN:  94%|█████████▍| 127/135 [00:30<00:01,  5.00it/s, epoch=98, grad_norm=170.897, loss=274.7509, lr=1.08e-03]
Epoch 98 TRAIN:  94%|█████████▍| 127/135 [00:30<00:01,  5.00it/s, epoch=98, grad_norm=118.559, loss=188.9435, lr=1.08e-03]
Epoch 98 TRAIN:  94%|█████████▍| 127/135 [00:30<00:01,  5.00it/s, epoch=98, grad_norm=147.648, loss=237.1294, lr=1.08e-03]
Epoch 98 TRAIN:  96%|█████████▋| 130/135 [00:30<00:01,  4.90it/s, epoch=98, grad_norm=147.648, loss=237.1294, lr=1.08e-03]
Epoch 98 TRAIN:  96%|█████████▋| 130/135 [00:30<00:01,  4.90it/s, epoch=98, grad_norm=109.588, loss=232.9992, lr=1.08e-03]
Epoch 98 TRAIN:  96%|█████████▋| 130/135 [00:30<00:01,  4.90it/s, epoch=98, grad_norm=192.928, loss=223.1778, lr=1.08e-03]
Epoch 98 TRAIN:  96%|█████████▋| 130/135 [00:31<00:01,  4.90it/s, epoch=98, grad_norm=236.630, loss=298.4222, lr=1.08e-03]
Epoch 98 TRAIN:  99%|█████████▊| 133/135 [00:31<00:00,  4.99it/s, epoch=98, grad_norm=236.630, loss=298.4222, lr=1.08e-03]
Epoch 98 TRAIN:  99%|█████████▊| 133/135 [00:31<00:00,  4.99it/s, epoch=98, grad_norm=293.349, loss=262.7981, lr=1.08e-03]
Epoch 98 TRAIN:  99%|█████████▊| 133/135 [00:31<00:00,  4.99it/s, epoch=98, grad_norm=277.467, loss=341.5551, lr=1.08e-03]
Epoch 98 TRAIN:  99%|█████████▊| 133/135 [00:31<00:00,  4.99it/s, epoch=98, grad_norm=256.407, loss=328.8914, lr=1.08e-03]
Epoch 98 TRAIN: 136it [00:31,  5.20it/s, epoch=98, grad_norm=256.407, loss=328.8914, lr=1.08e-03]                         
Epoch 98 TRAIN: 136it [00:31,  5.20it/s, epoch=98, grad_norm=152.699, loss=272.8133, lr=1.08e-03]
Epoch 98 TRAIN: 136it [00:31,  5.20it/s, epoch=98, grad_norm=217.860, loss=309.6180, lr=1.08e-03]
Epoch 98 TRAIN: 138it [00:31,  4.32it/s, epoch=98, grad_norm=217.860, loss=309.6180, lr=1.08e-03]

CV:   0%|          | 0/6 [00:00<?, ?it/s]/home/rookie/Documents/batuka/projects/wenet/wenet/wenet/utils/train_utils.py:689: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  amp_autocast(enabled=dtype is not None,

CV:  17%|█▋        | 1/6 [00:00<00:03,  1.32it/s]
                                                 
2026-04-16 20:43:38,899 INFO Epoch 98 Step 13504 CV info lr 1.0803e-03 cv_loss 476.0256430886009 rank 0 acc 0.3812611401081085
2026-04-16 20:43:38,900 INFO [Rank 0] Checkpoint: save to checkpoint exp/conformer_mongolian/epoch_98.pt
2026-04-16 20:43:39,015 INFO Epoch 99 Step 13504 TRAIN info lr 1.0803e-03 rank 0
2026-04-16 20:43:39,030 INFO using accumulate grad, new batch size is 1 times larger than before

Epoch 99 TRAIN:   0%|          | 0/138 [00:00<?, ?it/s]
Epoch 99 TRAIN:   0%|          | 0/138 [00:04<?, ?it/s, epoch=99, grad_norm=273.364, loss=286.7863, lr=1.08e-03]
Epoch 99 TRAIN:   1%|          | 1/138 [00:04<09:30,  4.17s/it, epoch=99, grad_norm=273.364, loss=286.7863, lr=1.08e-03]
Epoch 99 TRAIN:   1%|          | 1/138 [00:04<09:30,  4.17s/it, epoch=99, grad_norm=256.462, loss=254.1764, lr=1.08e-03]
Epoch 99 TRAIN:   1%|          | 1/138 [00:04<09:30,  4.17s/it, epoch=99, grad_norm=300.005, loss=299.3250, lr=1.08e-03]
Epoch 99 TRAIN:   1%|          | 1/138 [00:04<09:30,  4.17s/it, epoch=99, grad_norm=419.327, loss=393.1651, lr=1.08e-03]
Epoch 99 TRAIN:   1%|          | 1/138 [00:04<09:30,  4.17s/it, epoch=99, grad_norm=207.165, loss=236.4981, lr=1.08e-03]
Epoch 99 TRAIN:   1%|          | 1/138 [00:04<09:30,  4.17s/it, epoch=99, grad_norm=250.975, loss=295.3382, lr=1.08e-03]
Epoch 99 TRAIN:   1%|          | 1/138 [00:04<09:30,  4.17s/it, epoch=99, grad_norm=347.506, loss=388.7313, lr=1.08e-03]
Epoch 99 TRAIN:   1%|          | 1/138 [00:04<09:30,  4.17s/it, epoch=99, grad_norm=103.029, loss=196.2815, lr=1.08e-03]
Epoch 99 TRAIN:   6%|▌         | 8/138 [00:04<00:58,  2.23it/s, epoch=99, grad_norm=103.029, loss=196.2815, lr=1.08e-03]
Epoch 99 TRAIN:   6%|▌         | 8/138 [00:04<00:58,  2.23it/s, epoch=99, grad_norm=150.363, loss=263.8275, lr=1.08e-03]
Epoch 99 TRAIN:   6%|▌         | 8/138 [00:05<00:58,  2.23it/s, epoch=99, grad_norm=97.485, loss=282.6530, lr=1.08e-03] 
Epoch 99 TRAIN:   6%|▌         | 8/138 [00:05<00:58,  2.23it/s, epoch=99, grad_norm=109.931, loss=284.7551, lr=1.08e-03]
Epoch 99 TRAIN:   6%|▌         | 8/138 [00:05<00:58,  2.23it/s, epoch=99, grad_norm=93.546, loss=268.5682, lr=1.08e-03] 
Epoch 99 TRAIN:   9%|▊         | 12/138 [00:05<00:41,  3.03it/s, epoch=99, grad_norm=93.546, loss=268.5682, lr=1.08e-03]
Epoch 99 TRAIN:   9%|▊         | 12/138 [00:05<00:41,  3.03it/s, epoch=99, grad_norm=125.238, loss=312.3061, lr=1.08e-03]
Epoch 99 TRAIN:   9%|▊         | 12/138 [00:05<00:41,  3.03it/s, epoch=99, grad_norm=163.412, loss=226.4460, lr=1.08e-03]
Epoch 99 TRAIN:   9%|▊         | 12/138 [00:05<00:41,  3.03it/s, epoch=99, grad_norm=121.127, loss=255.7244, lr=1.08e-03]
Epoch 99 TRAIN:   9%|▊         | 12/138 [00:06<00:41,  3.03it/s, epoch=99, grad_norm=131.473, loss=267.2797, lr=1.08e-03]
Epoch 99 TRAIN:  12%|█▏        | 16/138 [00:06<00:33,  3.68it/s, epoch=99, grad_norm=131.473, loss=267.2797, lr=1.08e-03]
Epoch 99 TRAIN:  12%|█▏        | 16/138 [00:06<00:33,  3.68it/s, epoch=99, grad_norm=97.594, loss=297.8438, lr=1.08e-03] 
Epoch 99 TRAIN:  12%|█▏        | 16/138 [00:06<00:33,  3.68it/s, epoch=99, grad_norm=140.446, loss=274.3562, lr=1.08e-03]
Epoch 99 TRAIN:  12%|█▏        | 16/138 [00:06<00:33,  3.68it/s, epoch=99, grad_norm=129.028, loss=269.5801, lr=1.08e-03]
Epoch 99 TRAIN:  12%|█▏        | 16/138 [00:06<00:33,  3.68it/s, epoch=99, grad_norm=74.833, loss=280.0716, lr=1.08e-03] 
Epoch 99 TRAIN:  14%|█▍        | 20/138 [00:06<00:29,  3.94it/s, epoch=99, grad_norm=74.833, loss=280.0716, lr=1.08e-03]
Epoch 99 TRAIN:  14%|█▍        | 20/138 [00:07<00:29,  3.94it/s, epoch=99, grad_norm=121.307, loss=214.8389, lr=1.08e-03]
Epoch 99 TRAIN:  14%|█▍        | 20/138 [00:07<00:29,  3.94it/s, epoch=99, grad_norm=97.371, loss=309.7549, lr=1.08e-03] 
Epoch 99 TRAIN:  14%|█▍        | 20/138 [00:07<00:29,  3.94it/s, epoch=99, grad_norm=101.789, loss=227.9066, lr=1.08e-03]
Epoch 99 TRAIN:  17%|█▋        | 23/138 [00:07<00:27,  4.14it/s, epoch=99, grad_norm=101.789, loss=227.9066, lr=1.08e-03]
Epoch 99 TRAIN:  17%|█▋        | 23/138 [00:07<00:27,  4.14it/s, epoch=99, grad_norm=73.365, loss=260.1340, lr=1.08e-03] 
Epoch 99 TRAIN:  17%|█▋        | 23/138 [00:07<00:27,  4.14it/s, epoch=99, grad_norm=114.864, loss=293.5701, lr=1.08e-03]
Epoch 99 TRAIN:  17%|█▋        | 23/138 [00:08<00:27,  4.14it/s, epoch=99, grad_norm=103.267, loss=266.4685, lr=1.08e-03]
Epoch 99 TRAIN:  19%|█▉        | 26/138 [00:08<00:24,  4.48it/s, epoch=99, grad_norm=103.267, loss=266.4685, lr=1.08e-03]
Epoch 99 TRAIN:  19%|█▉        | 26/138 [00:08<00:24,  4.48it/s, epoch=99, grad_norm=98.776, loss=275.0855, lr=1.08e-03] 
Epoch 99 TRAIN:  19%|█▉        | 26/138 [00:08<00:24,  4.48it/s, epoch=99, grad_norm=65.191, loss=236.2544, lr=1.08e-03]
Epoch 99 TRAIN:  19%|█▉        | 26/138 [00:08<00:24,  4.48it/s, epoch=99, grad_norm=105.562, loss=278.9426, lr=1.08e-03]
Epoch 99 TRAIN:  21%|██        | 29/138 [00:08<00:25,  4.30it/s, epoch=99, grad_norm=105.562, loss=278.9426, lr=1.08e-03]
Epoch 99 TRAIN:  21%|██        | 29/138 [00:09<00:25,  4.30it/s, epoch=99, grad_norm=136.133, loss=289.2292, lr=1.08e-03]
Epoch 99 TRAIN:  21%|██        | 29/138 [00:09<00:25,  4.30it/s, epoch=99, grad_norm=76.482, loss=244.8108, lr=1.08e-03] 
Epoch 99 TRAIN:  21%|██        | 29/138 [00:09<00:25,  4.30it/s, epoch=99, grad_norm=79.629, loss=255.1963, lr=1.08e-03]
Epoch 99 TRAIN:  23%|██▎       | 32/138 [00:09<00:23,  4.45it/s, epoch=99, grad_norm=79.629, loss=255.1963, lr=1.08e-03]
Epoch 99 TRAIN:  23%|██▎       | 32/138 [00:09<00:23,  4.45it/s, epoch=99, grad_norm=95.576, loss=250.2510, lr=1.08e-03]
Epoch 99 TRAIN:  23%|██▎       | 32/138 [00:09<00:23,  4.45it/s, epoch=99, grad_norm=113.776, loss=233.9107, lr=1.08e-03]
Epoch 99 TRAIN:  23%|██▎       | 32/138 [00:10<00:23,  4.45it/s, epoch=99, grad_norm=114.132, loss=238.8900, lr=1.08e-03]
Epoch 99 TRAIN:  25%|██▌       | 35/138 [00:10<00:22,  4.66it/s, epoch=99, grad_norm=114.132, loss=238.8900, lr=1.08e-03]
Epoch 99 TRAIN:  25%|██▌       | 35/138 [00:10<00:22,  4.66it/s, epoch=99, grad_norm=133.574, loss=254.0454, lr=1.08e-03]
Epoch 99 TRAIN:  25%|██▌       | 35/138 [00:10<00:22,  4.66it/s, epoch=99, grad_norm=74.891, loss=253.7224, lr=1.08e-03] 
Epoch 99 TRAIN:  25%|██▌       | 35/138 [00:10<00:22,  4.66it/s, epoch=99, grad_norm=81.700, loss=248.6716, lr=1.08e-03]
Epoch 99 TRAIN:  28%|██▊       | 38/138 [00:10<00:21,  4.58it/s, epoch=99, grad_norm=81.700, loss=248.6716, lr=1.08e-03]
Epoch 99 TRAIN:  28%|██▊       | 38/138 [00:10<00:21,  4.58it/s, epoch=99, grad_norm=122.291, loss=262.9692, lr=1.08e-03]
Epoch 99 TRAIN:  28%|██▊       | 38/138 [00:11<00:21,  4.58it/s, epoch=99, grad_norm=173.910, loss=283.9570, lr=1.08e-03]
Epoch 99 TRAIN:  28%|██▊       | 38/138 [00:11<00:21,  4.58it/s, epoch=99, grad_norm=107.700, loss=271.0088, lr=1.08e-03]
Epoch 99 TRAIN:  30%|██▉       | 41/138 [00:11<00:21,  4.61it/s, epoch=99, grad_norm=107.700, loss=271.0088, lr=1.08e-03]
Epoch 99 TRAIN:  30%|██▉       | 41/138 [00:11<00:21,  4.61it/s, epoch=99, grad_norm=79.983, loss=237.4100, lr=1.08e-03] 
Epoch 99 TRAIN:  30%|██▉       | 41/138 [00:11<00:21,  4.61it/s, epoch=99, grad_norm=78.112, loss=271.6766, lr=1.08e-03]
Epoch 99 TRAIN:  30%|██▉       | 41/138 [00:12<00:21,  4.61it/s, epoch=99, grad_norm=164.148, loss=286.0704, lr=1.08e-03]
Epoch 99 TRAIN:  32%|███▏      | 44/138 [00:12<00:20,  4.57it/s, epoch=99, grad_norm=164.148, loss=286.0704, lr=1.08e-03]
Epoch 99 TRAIN:  32%|███▏      | 44/138 [00:12<00:20,  4.57it/s, epoch=99, grad_norm=92.741, loss=267.7037, lr=1.08e-03] 
Epoch 99 TRAIN:  32%|███▏      | 44/138 [00:12<00:20,  4.57it/s, epoch=99, grad_norm=102.649, loss=251.7808, lr=1.08e-03]
Epoch 99 TRAIN:  32%|███▏      | 44/138 [00:12<00:20,  4.57it/s, epoch=99, grad_norm=112.854, loss=233.2041, lr=1.08e-03]
Epoch 99 TRAIN:  34%|███▍      | 47/138 [00:12<00:19,  4.61it/s, epoch=99, grad_norm=112.854, loss=233.2041, lr=1.08e-03]
Epoch 99 TRAIN:  34%|███▍      | 47/138 [00:12<00:19,  4.61it/s, epoch=99, grad_norm=113.368, loss=258.1302, lr=1.08e-03]
Epoch 99 TRAIN:  34%|███▍      | 47/138 [00:13<00:19,  4.61it/s, epoch=99, grad_norm=159.511, loss=259.3662, lr=1.08e-03]
Epoch 99 TRAIN:  34%|███▍      | 47/138 [00:13<00:19,  4.61it/s, epoch=99, grad_norm=115.028, loss=235.2058, lr=1.08e-03]
Epoch 99 TRAIN:  36%|███▌      | 50/138 [00:13<00:18,  4.66it/s, epoch=99, grad_norm=115.028, loss=235.2058, lr=1.08e-03]
Epoch 99 TRAIN:  36%|███▌      | 50/138 [00:13<00:18,  4.66it/s, epoch=99, grad_norm=108.344, loss=253.1316, lr=1.08e-03]
Epoch 99 TRAIN:  36%|███▌      | 50/138 [00:13<00:18,  4.66it/s, epoch=99, grad_norm=125.795, loss=277.9730, lr=1.08e-03]
Epoch 99 TRAIN:  36%|███▌      | 50/138 [00:14<00:18,  4.66it/s, epoch=99, grad_norm=101.395, loss=269.0348, lr=1.08e-03]
Epoch 99 TRAIN:  38%|███▊      | 53/138 [00:14<00:19,  4.30it/s, epoch=99, grad_norm=101.395, loss=269.0348, lr=1.08e-03]
Epoch 99 TRAIN:  38%|███▊      | 53/138 [00:14<00:19,  4.30it/s, epoch=99, grad_norm=120.107, loss=288.2025, lr=1.08e-03]
Epoch 99 TRAIN:  38%|███▊      | 53/138 [00:14<00:19,  4.30it/s, epoch=99, grad_norm=106.877, loss=259.2955, lr=1.08e-03]
Epoch 99 TRAIN:  38%|███▊      | 53/138 [00:14<00:19,  4.30it/s, epoch=99, grad_norm=132.380, loss=314.2830, lr=1.08e-03]
Epoch 99 TRAIN:  41%|████      | 56/138 [00:14<00:18,  4.37it/s, epoch=99, grad_norm=132.380, loss=314.2830, lr=1.08e-03]
Epoch 99 TRAIN:  41%|████      | 56/138 [00:15<00:18,  4.37it/s, epoch=99, grad_norm=96.647, loss=291.3547, lr=1.08e-03] 
Epoch 99 TRAIN:  41%|████      | 56/138 [00:15<00:18,  4.37it/s, epoch=99, grad_norm=98.428, loss=260.8345, lr=1.08e-03]
Epoch 99 TRAIN:  41%|████      | 56/138 [00:15<00:18,  4.37it/s, epoch=99, grad_norm=136.046, loss=202.5688, lr=1.09e-03]
Epoch 99 TRAIN:  43%|████▎     | 59/138 [00:15<00:17,  4.51it/s, epoch=99, grad_norm=136.046, loss=202.5688, lr=1.09e-03]
Epoch 99 TRAIN:  43%|████▎     | 59/138 [00:15<00:17,  4.51it/s, epoch=99, grad_norm=90.707, loss=270.3888, lr=1.09e-03] 
Epoch 99 TRAIN:  43%|████▎     | 59/138 [00:15<00:17,  4.51it/s, epoch=99, grad_norm=105.513, loss=241.2960, lr=1.09e-03]
Epoch 99 TRAIN:  43%|████▎     | 59/138 [00:16<00:17,  4.51it/s, epoch=99, grad_norm=101.781, loss=237.6749, lr=1.09e-03]
Epoch 99 TRAIN:  45%|████▍     | 62/138 [00:16<00:16,  4.54it/s, epoch=99, grad_norm=101.781, loss=237.6749, lr=1.09e-03]
Epoch 99 TRAIN:  45%|████▍     | 62/138 [00:16<00:16,  4.54it/s, epoch=99, grad_norm=82.161, loss=227.7064, lr=1.09e-03] 
Epoch 99 TRAIN:  45%|████▍     | 62/138 [00:16<00:16,  4.54it/s, epoch=99, grad_norm=105.733, loss=257.0823, lr=1.09e-03]
Epoch 99 TRAIN:  45%|████▍     | 62/138 [00:16<00:16,  4.54it/s, epoch=99, grad_norm=80.069, loss=272.5159, lr=1.09e-03] 
Epoch 99 TRAIN:  47%|████▋     | 65/138 [00:16<00:16,  4.46it/s, epoch=99, grad_norm=80.069, loss=272.5159, lr=1.09e-03]
Epoch 99 TRAIN:  47%|████▋     | 65/138 [00:16<00:16,  4.46it/s, epoch=99, grad_norm=104.373, loss=278.3023, lr=1.09e-03]
Epoch 99 TRAIN:  47%|████▋     | 65/138 [00:17<00:16,  4.46it/s, epoch=99, grad_norm=118.364, loss=268.1917, lr=1.09e-03]
Epoch 99 TRAIN:  47%|████▋     | 65/138 [00:17<00:16,  4.46it/s, epoch=99, grad_norm=88.669, loss=235.1233, lr=1.09e-03] 
Epoch 99 TRAIN:  49%|████▉     | 68/138 [00:17<00:15,  4.65it/s, epoch=99, grad_norm=88.669, loss=235.1233, lr=1.09e-03]
Epoch 99 TRAIN:  49%|████▉     | 68/138 [00:17<00:15,  4.65it/s, epoch=99, grad_norm=142.211, loss=261.9371, lr=1.09e-03]
Epoch 99 TRAIN:  49%|████▉     | 68/138 [00:17<00:15,  4.65it/s, epoch=99, grad_norm=147.073, loss=266.2541, lr=1.09e-03]
Epoch 99 TRAIN:  49%|████▉     | 68/138 [00:18<00:15,  4.65it/s, epoch=99, grad_norm=102.837, loss=264.1942, lr=1.09e-03]
Epoch 99 TRAIN:  51%|█████▏    | 71/138 [00:18<00:14,  4.54it/s, epoch=99, grad_norm=102.837, loss=264.1942, lr=1.09e-03]
Epoch 99 TRAIN:  51%|█████▏    | 71/138 [00:18<00:14,  4.54it/s, epoch=99, grad_norm=83.295, loss=258.7186, lr=1.09e-03] 
Epoch 99 TRAIN:  51%|█████▏    | 71/138 [00:18<00:14,  4.54it/s, epoch=99, grad_norm=81.282, loss=265.1841, lr=1.09e-03]
Epoch 99 TRAIN:  51%|█████▏    | 71/138 [00:18<00:14,  4.54it/s, epoch=99, grad_norm=207.967, loss=212.3764, lr=1.09e-03]
Epoch 99 TRAIN:  54%|█████▎    | 74/138 [00:18<00:14,  4.46it/s, epoch=99, grad_norm=207.967, loss=212.3764, lr=1.09e-03]
Epoch 99 TRAIN:  54%|█████▎    | 74/138 [00:18<00:14,  4.46it/s, epoch=99, grad_norm=146.953, loss=278.3158, lr=1.09e-03]
Epoch 99 TRAIN:  54%|█████▎    | 74/138 [00:19<00:14,  4.46it/s, epoch=99, grad_norm=108.631, loss=248.4519, lr=1.09e-03]
Epoch 99 TRAIN:  54%|█████▎    | 74/138 [00:19<00:14,  4.46it/s, epoch=99, grad_norm=130.026, loss=257.8191, lr=1.09e-03]
Epoch 99 TRAIN:  56%|█████▌    | 77/138 [00:19<00:13,  4.42it/s, epoch=99, grad_norm=130.026, loss=257.8191, lr=1.09e-03]
Epoch 99 TRAIN:  56%|█████▌    | 77/138 [00:19<00:13,  4.42it/s, epoch=99, grad_norm=197.752, loss=186.1527, lr=1.09e-03]
Epoch 99 TRAIN:  56%|█████▌    | 77/138 [00:19<00:13,  4.42it/s, epoch=99, grad_norm=158.986, loss=213.2177, lr=1.09e-03]
Epoch 99 TRAIN:  56%|█████▌    | 77/138 [00:20<00:13,  4.42it/s, epoch=99, grad_norm=92.655, loss=263.8416, lr=1.09e-03] 
Epoch 99 TRAIN:  58%|█████▊    | 80/138 [00:20<00:12,  4.56it/s, epoch=99, grad_norm=92.655, loss=263.8416, lr=1.09e-03]
Epoch 99 TRAIN:  58%|█████▊    | 80/138 [00:20<00:12,  4.56it/s, epoch=99, grad_norm=93.008, loss=242.0762, lr=1.09e-03]
Epoch 99 TRAIN:  58%|█████▊    | 80/138 [00:20<00:12,  4.56it/s, epoch=99, grad_norm=117.027, loss=304.6608, lr=1.09e-03]
Epoch 99 TRAIN:  58%|█████▊    | 80/138 [00:20<00:12,  4.56it/s, epoch=99, grad_norm=118.334, loss=266.0234, lr=1.09e-03]
Epoch 99 TRAIN:  60%|██████    | 83/138 [00:20<00:11,  4.60it/s, epoch=99, grad_norm=118.334, loss=266.0234, lr=1.09e-03]
Epoch 99 TRAIN:  60%|██████    | 83/138 [00:20<00:11,  4.60it/s, epoch=99, grad_norm=144.758, loss=180.3301, lr=1.09e-03]
Epoch 99 TRAIN:  60%|██████    | 83/138 [00:21<00:11,  4.60it/s, epoch=99, grad_norm=84.362, loss=234.9140, lr=1.09e-03] 
Epoch 99 TRAIN:  60%|██████    | 83/138 [00:21<00:11,  4.60it/s, epoch=99, grad_norm=86.461, loss=258.5953, lr=1.09e-03]
Epoch 99 TRAIN:  62%|██████▏   | 86/138 [00:21<00:11,  4.47it/s, epoch=99, grad_norm=86.461, loss=258.5953, lr=1.09e-03]
Epoch 99 TRAIN:  62%|██████▏   | 86/138 [00:21<00:11,  4.47it/s, epoch=99, grad_norm=180.379, loss=296.5476, lr=1.09e-03]
Epoch 99 TRAIN:  62%|██████▏   | 86/138 [00:21<00:11,  4.47it/s, epoch=99, grad_norm=225.764, loss=311.5204, lr=1.09e-03]
Epoch 99 TRAIN:  62%|██████▏   | 86/138 [00:22<00:11,  4.47it/s, epoch=99, grad_norm=113.128, loss=284.6614, lr=1.09e-03]
Epoch 99 TRAIN:  64%|██████▍   | 89/138 [00:22<00:10,  4.48it/s, epoch=99, grad_norm=113.128, loss=284.6614, lr=1.09e-03]
Epoch 99 TRAIN:  64%|██████▍   | 89/138 [00:22<00:10,  4.48it/s, epoch=99, grad_norm=227.299, loss=216.3211, lr=1.09e-03]
Epoch 99 TRAIN:  64%|██████▍   | 89/138 [00:22<00:10,  4.48it/s, epoch=99, grad_norm=271.650, loss=153.3001, lr=1.09e-03]
Epoch 99 TRAIN:  64%|██████▍   | 89/138 [00:22<00:10,  4.48it/s, epoch=99, grad_norm=115.004, loss=241.0515, lr=1.09e-03]
Epoch 99 TRAIN:  67%|██████▋   | 92/138 [00:22<00:09,  4.61it/s, epoch=99, grad_norm=115.004, loss=241.0515, lr=1.09e-03]
Epoch 99 TRAIN:  67%|██████▋   | 92/138 [00:22<00:09,  4.61it/s, epoch=99, grad_norm=168.618, loss=303.9810, lr=1.09e-03]
Epoch 99 TRAIN:  67%|██████▋   | 92/138 [00:23<00:09,  4.61it/s, epoch=99, grad_norm=110.329, loss=320.3686, lr=1.09e-03]
Epoch 99 TRAIN:  67%|██████▋   | 92/138 [00:23<00:09,  4.61it/s, epoch=99, grad_norm=138.300, loss=225.3250, lr=1.09e-03]
Epoch 99 TRAIN:  69%|██████▉   | 95/138 [00:23<00:09,  4.78it/s, epoch=99, grad_norm=138.300, loss=225.3250, lr=1.09e-03]
Epoch 99 TRAIN:  69%|██████▉   | 95/138 [00:23<00:09,  4.78it/s, epoch=99, grad_norm=194.984, loss=165.5437, lr=1.09e-03]
Epoch 99 TRAIN:  69%|██████▉   | 95/138 [00:23<00:09,  4.78it/s, epoch=99, grad_norm=205.899, loss=207.9223, lr=1.09e-03]
Epoch 99 TRAIN:  69%|██████▉   | 95/138 [00:23<00:09,  4.78it/s, epoch=99, grad_norm=269.431, loss=274.9769, lr=1.09e-03]
Epoch 99 TRAIN:  71%|███████   | 98/138 [00:23<00:08,  4.60it/s, epoch=99, grad_norm=269.431, loss=274.9769, lr=1.09e-03]
Epoch 99 TRAIN:  71%|███████   | 98/138 [00:24<00:08,  4.60it/s, epoch=99, grad_norm=121.261, loss=258.4756, lr=1.09e-03]
Epoch 99 TRAIN:  71%|███████   | 98/138 [00:24<00:08,  4.60it/s, epoch=99, grad_norm=147.648, loss=171.4482, lr=1.09e-03]2026-04-16 20:44:03,397 DEBUG TRAIN | steps/sec 4.161| Batch 99/100 loss 171.448151 loss_att 125.581390 loss_ctc 278.470581 th_accuracy 0.501742 lr 1.0883e-03 grad_norm 147.647842 rank 0

Epoch 99 TRAIN:  71%|███████   | 98/138 [00:24<00:08,  4.60it/s, epoch=99, grad_norm=157.529, loss=240.1615, lr=1.09e-03]
Epoch 99 TRAIN:  73%|███████▎  | 101/138 [00:24<00:07,  4.72it/s, epoch=99, grad_norm=157.529, loss=240.1615, lr=1.09e-03]
Epoch 99 TRAIN:  73%|███████▎  | 101/138 [00:24<00:07,  4.72it/s, epoch=99, grad_norm=213.420, loss=266.7475, lr=1.09e-03]
Epoch 99 TRAIN:  73%|███████▎  | 101/138 [00:24<00:07,  4.72it/s, epoch=99, grad_norm=99.433, loss=194.4125, lr=1.09e-03] 
Epoch 99 TRAIN:  73%|███████▎  | 101/138 [00:25<00:07,  4.72it/s, epoch=99, grad_norm=234.758, loss=264.6603, lr=1.09e-03]
Epoch 99 TRAIN:  75%|███████▌  | 104/138 [00:25<00:07,  4.84it/s, epoch=99, grad_norm=234.758, loss=264.6603, lr=1.09e-03]
Epoch 99 TRAIN:  75%|███████▌  | 104/138 [00:25<00:07,  4.84it/s, epoch=99, grad_norm=234.099, loss=229.2732, lr=1.09e-03]
Epoch 99 TRAIN:  75%|███████▌  | 104/138 [00:25<00:07,  4.84it/s, epoch=99, grad_norm=216.230, loss=285.7552, lr=1.09e-03]
Epoch 99 TRAIN:  75%|███████▌  | 104/138 [00:25<00:07,  4.84it/s, epoch=99, grad_norm=101.107, loss=236.2880, lr=1.09e-03]
Epoch 99 TRAIN:  78%|███████▊  | 107/138 [00:25<00:06,  4.64it/s, epoch=99, grad_norm=101.107, loss=236.2880, lr=1.09e-03]
Epoch 99 TRAIN:  78%|███████▊  | 107/138 [00:26<00:06,  4.64it/s, epoch=99, grad_norm=113.141, loss=237.8690, lr=1.09e-03]
Epoch 99 TRAIN:  78%|███████▊  | 107/138 [00:26<00:06,  4.64it/s, epoch=99, grad_norm=135.385, loss=247.6272, lr=1.09e-03]
Epoch 99 TRAIN:  78%|███████▊  | 107/138 [00:26<00:06,  4.64it/s, epoch=99, grad_norm=92.911, loss=227.6981, lr=1.09e-03] 
Epoch 99 TRAIN:  80%|███████▉  | 110/138 [00:26<00:06,  4.59it/s, epoch=99, grad_norm=92.911, loss=227.6981, lr=1.09e-03]
Epoch 99 TRAIN:  80%|███████▉  | 110/138 [00:26<00:06,  4.59it/s, epoch=99, grad_norm=135.022, loss=279.8610, lr=1.09e-03]
Epoch 99 TRAIN:  80%|███████▉  | 110/138 [00:27<00:06,  4.59it/s, epoch=99, grad_norm=104.550, loss=267.8998, lr=1.09e-03]
Epoch 99 TRAIN:  80%|███████▉  | 110/138 [00:27<00:06,  4.59it/s, epoch=99, grad_norm=143.086, loss=228.8042, lr=1.09e-03]
Epoch 99 TRAIN:  82%|████████▏ | 113/138 [00:27<00:05,  4.45it/s, epoch=99, grad_norm=143.086, loss=228.8042, lr=1.09e-03]
Epoch 99 TRAIN:  82%|████████▏ | 113/138 [00:27<00:05,  4.45it/s, epoch=99, grad_norm=149.071, loss=293.7548, lr=1.09e-03]
Epoch 99 TRAIN:  82%|████████▏ | 113/138 [00:27<00:05,  4.45it/s, epoch=99, grad_norm=186.700, loss=253.9382, lr=1.09e-03]
Epoch 99 TRAIN:  82%|████████▏ | 113/138 [00:27<00:05,  4.45it/s, epoch=99, grad_norm=194.607, loss=227.2350, lr=1.09e-03]
Epoch 99 TRAIN:  84%|████████▍ | 116/138 [00:27<00:04,  4.57it/s, epoch=99, grad_norm=194.607, loss=227.2350, lr=1.09e-03]
Epoch 99 TRAIN:  84%|████████▍ | 116/138 [00:28<00:04,  4.57it/s, epoch=99, grad_norm=140.035, loss=215.4419, lr=1.09e-03]
Epoch 99 TRAIN:  84%|████████▍ | 116/138 [00:28<00:04,  4.57it/s, epoch=99, grad_norm=133.202, loss=272.2627, lr=1.09e-03]
Epoch 99 TRAIN:  84%|████████▍ | 116/138 [00:28<00:04,  4.57it/s, epoch=99, grad_norm=112.564, loss=243.8390, lr=1.09e-03]
Epoch 99 TRAIN:  86%|████████▌ | 119/138 [00:28<00:04,  4.37it/s, epoch=99, grad_norm=112.564, loss=243.8390, lr=1.09e-03]
Epoch 99 TRAIN:  86%|████████▌ | 119/138 [00:28<00:04,  4.37it/s, epoch=99, grad_norm=102.256, loss=232.0357, lr=1.09e-03]
Epoch 99 TRAIN:  86%|████████▌ | 119/138 [00:29<00:04,  4.37it/s, epoch=99, grad_norm=116.325, loss=234.0361, lr=1.09e-03]
Epoch 99 TRAIN:  86%|████████▌ | 119/138 [00:29<00:04,  4.37it/s, epoch=99, grad_norm=244.829, loss=304.0687, lr=1.09e-03]
Epoch 99 TRAIN:  88%|████████▊ | 122/138 [00:29<00:03,  4.61it/s, epoch=99, grad_norm=244.829, loss=304.0687, lr=1.09e-03]
Epoch 99 TRAIN:  88%|████████▊ | 122/138 [00:29<00:03,  4.61it/s, epoch=99, grad_norm=187.966, loss=277.6824, lr=1.09e-03]
Epoch 99 TRAIN:  88%|████████▊ | 122/138 [00:29<00:03,  4.61it/s, epoch=99, grad_norm=176.575, loss=265.6263, lr=1.09e-03]
Epoch 99 TRAIN:  88%|████████▊ | 122/138 [00:29<00:03,  4.61it/s, epoch=99, grad_norm=106.231, loss=220.0889, lr=1.09e-03]
Epoch 99 TRAIN:  91%|█████████ | 125/138 [00:29<00:02,  4.42it/s, epoch=99, grad_norm=106.231, loss=220.0889, lr=1.09e-03]
Epoch 99 TRAIN:  91%|█████████ | 125/138 [00:30<00:02,  4.42it/s, epoch=99, grad_norm=164.819, loss=245.0143, lr=1.09e-03]
Epoch 99 TRAIN:  91%|█████████ | 125/138 [00:30<00:02,  4.42it/s, epoch=99, grad_norm=124.093, loss=274.2996, lr=1.09e-03]
Epoch 99 TRAIN:  91%|█████████ | 125/138 [00:30<00:02,  4.42it/s, epoch=99, grad_norm=130.429, loss=199.4019, lr=1.09e-03]
Epoch 99 TRAIN:  93%|█████████▎| 128/138 [00:30<00:02,  4.52it/s, epoch=99, grad_norm=130.429, loss=199.4019, lr=1.09e-03]
Epoch 99 TRAIN:  93%|█████████▎| 128/138 [00:30<00:02,  4.52it/s, epoch=99, grad_norm=209.711, loss=255.9571, lr=1.09e-03]
Epoch 99 TRAIN:  93%|█████████▎| 128/138 [00:30<00:02,  4.52it/s, epoch=99, grad_norm=142.736, loss=268.4276, lr=1.09e-03]
Epoch 99 TRAIN:  93%|█████████▎| 128/138 [00:31<00:02,  4.52it/s, epoch=99, grad_norm=243.421, loss=314.3876, lr=1.09e-03]
Epoch 99 TRAIN:  95%|█████████▍| 131/138 [00:31<00:01,  4.41it/s, epoch=99, grad_norm=243.421, loss=314.3876, lr=1.09e-03]
Epoch 99 TRAIN:  95%|█████████▍| 131/138 [00:31<00:01,  4.41it/s, epoch=99, grad_norm=178.835, loss=262.1275, lr=1.09e-03]
Epoch 99 TRAIN:  95%|█████████▍| 131/138 [00:31<00:01,  4.41it/s, epoch=99, grad_norm=143.279, loss=236.0921, lr=1.09e-03]
Epoch 99 TRAIN:  95%|█████████▍| 131/138 [00:31<00:01,  4.41it/s, epoch=99, grad_norm=201.667, loss=296.3379, lr=1.09e-03]
Epoch 99 TRAIN:  95%|█████████▍| 131/138 [00:31<00:01,  4.41it/s, epoch=99, grad_norm=294.341, loss=320.8150, lr=1.09e-03]
Epoch 99 TRAIN:  98%|█████████▊| 135/138 [00:31<00:00,  5.08it/s, epoch=99, grad_norm=294.341, loss=320.8150, lr=1.09e-03]
Epoch 99 TRAIN:  98%|█████████▊| 135/138 [00:32<00:00,  5.08it/s, epoch=99, grad_norm=204.380, loss=271.8969, lr=1.09e-03]
Epoch 99 TRAIN:  99%|█████████▊| 136/138 [00:32<00:00,  4.24it/s, epoch=99, grad_norm=204.380, loss=271.8969, lr=1.09e-03]

CV:   0%|          | 0/6 [00:00<?, ?it/s]/home/rookie/Documents/batuka/projects/wenet/wenet/wenet/utils/train_utils.py:689: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  amp_autocast(enabled=dtype is not None,

CV:  17%|█▋        | 1/6 [00:00<00:03,  1.31it/s]
                                                 
2026-04-16 20:44:12,008 INFO Epoch 99 Step 13640 CV info lr 1.0912e-03 cv_loss 459.20543046431106 rank 0 acc 0.40657492478688556
2026-04-16 20:44:12,010 INFO [Rank 0] Checkpoint: save to checkpoint exp/conformer_mongolian/epoch_99.pt
2026-04-16 20:44:12,774 DEBUG Attempting to acquire lock 137082720490192 on /home/rookie/.triton/autotune/Fp16Matmul_2d_kernel.pickle.lock
2026-04-16 20:44:12,774 DEBUG Lock 137082720490192 acquired on /home/rookie/.triton/autotune/Fp16Matmul_2d_kernel.pickle.lock
2026-04-16 20:44:12,775 DEBUG Attempting to release lock 137082720490192 on /home/rookie/.triton/autotune/Fp16Matmul_2d_kernel.pickle.lock
2026-04-16 20:44:12,775 DEBUG Lock 137082720490192 released on /home/rookie/.triton/autotune/Fp16Matmul_2d_kernel.pickle.lock
2026-04-16 20:44:12,777 DEBUG Attempting to acquire lock 137082720487072 on /home/rookie/.triton/autotune/Fp16Matmul_4d_kernel.pickle.lock
2026-04-16 20:44:12,777 DEBUG Lock 137082720487072 acquired on /home/rookie/.triton/autotune/Fp16Matmul_4d_kernel.pickle.lock
2026-04-16 20:44:12,777 DEBUG Attempting to release lock 137082720487072 on /home/rookie/.triton/autotune/Fp16Matmul_4d_kernel.pickle.lock
2026-04-16 20:44:12,777 DEBUG Lock 137082720487072 released on /home/rookie/.triton/autotune/Fp16Matmul_4d_kernel.pickle.lock
/home/rookie/miniconda3/envs/wenet/lib/python3.10/tempfile.py:869: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpo0ov1d4n'>
  _warnings.warn(warn_message, ResourceWarning)