eval_custom_0.25.log

2026-03-27 15:14:23.038334: I tensorflow/core/util/util.cc:169] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2026-03-27 15:14:23.061742: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/lib/python3.10/dist-packages/torch/lib:/usr/local/lib/python3.10/dist-packages/torch_tensorrt/lib:/usr/local/cuda/compat/lib:/usr/local/nvidia/lib:/usr/local/nvidia/lib64
2026-03-27 15:14:23.061828: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
2026-03-27 15:14:30.816713: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F AVX512_VNNI AVX512_BF16 AVX_VNNI AMX_TILE AMX_INT8 AMX_BF16 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2026-03-27 15:14:30.882778: E tensorflow/stream_executor/cuda/cuda_driver.cc:271] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected
2026-03-27 15:14:30.882918: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:169] retrieving CUDA diagnostic information for host: 58da3b71d0d102c97ed081cf3f42d037-taskrole1-0
2026-03-27 15:14:30.882962: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:176] hostname: 58da3b71d0d102c97ed081cf3f42d037-taskrole1-0
2026-03-27 15:14:30.883135: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:200] libcuda reported version is: 580.82.7
2026-03-27 15:14:30.883186: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:204] kernel reported version is: 580.82.7
2026-03-27 15:14:30.883219: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:310] kernel version seems to match DSO: 580.82.7
2026-03-27 15:14:31.395883: W tensorflow/core/framework/op_def_util.cc:371] Op BatchNormWithGlobalNormalization is deprecated. It will cease to work in GraphDef version 9. Use tf.nn.batch_normalization().
warming up TensorFlow...

  0%|          | 0/1 [00:00<?, ?it/s]2026-03-27 15:14:32.318003: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:354] MLIR V1 optimization pass is not enabled

100%|██████████| 1/1 [00:01<00:00,  1.48s/it]
100%|██████████| 1/1 [00:01<00:00,  1.48s/it]
computing reference batch activations...

  0%|          | 0/157 [00:00<?, ?it/s]
  1%|          | 1/157 [00:01<03:13,  1.24s/it]
  1%|▏         | 2/157 [00:02<02:59,  1.16s/it]
  2%|▏         | 3/157 [00:03<02:54,  1.13s/it]
  3%|▎         | 4/157 [00:04<02:51,  1.12s/it]
  3%|▎         | 5/157 [00:05<02:55,  1.16s/it]
  4%|▍         | 6/157 [00:07<03:26,  1.37s/it]
  4%|▍         | 7/157 [00:08<03:13,  1.29s/it]
  5%|▌         | 8/157 [00:09<03:02,  1.22s/it]
  6%|▌         | 9/157 [00:10<02:56,  1.19s/it]
  6%|▋         | 10/157 [00:11<02:50,  1.16s/it]
  7%|▋         | 11/157 [00:13<02:45,  1.14s/it]
  8%|▊         | 12/157 [00:14<02:42,  1.12s/it]
  8%|▊         | 13/157 [00:15<02:41,  1.12s/it]
  9%|▉         | 14/157 [00:16<02:38,  1.11s/it]
 10%|▉         | 15/157 [00:17<02:37,  1.11s/it]
 10%|█         | 16/157 [00:18<02:40,  1.14s/it]
 11%|█         | 17/157 [00:19<02:45,  1.18s/it]
 11%|█▏        | 18/157 [00:21<02:48,  1.22s/it]
 12%|█▏        | 19/157 [00:22<02:43,  1.18s/it]
 13%|█▎        | 20/157 [00:23<02:38,  1.16s/it]
 13%|█▎        | 21/157 [00:24<02:35,  1.14s/it]
 14%|█▍        | 22/157 [00:25<02:33,  1.14s/it]
 15%|█▍        | 23/157 [00:26<02:30,  1.12s/it]
 15%|█▌        | 24/157 [00:27<02:27,  1.11s/it]
 16%|█▌        | 25/157 [00:28<02:26,  1.11s/it]
 17%|█▋        | 26/157 [00:30<02:24,  1.11s/it]
 17%|█▋        | 27/157 [00:31<02:24,  1.11s/it]
 18%|█▊        | 28/157 [00:32<02:22,  1.11s/it]
 18%|█▊        | 29/157 [00:33<02:20,  1.10s/it]
 19%|█▉        | 30/157 [00:34<02:19,  1.10s/it]
 20%|█▉        | 31/157 [00:35<02:18,  1.10s/it]
 20%|██        | 32/157 [00:36<02:17,  1.10s/it]
 21%|██        | 33/157 [00:37<02:16,  1.10s/it]
 22%|██▏       | 34/157 [00:38<02:15,  1.10s/it]
 22%|██▏       | 35/157 [00:39<02:14,  1.10s/it]
 23%|██▎       | 36/157 [00:41<02:13,  1.10s/it]
 24%|██▎       | 37/157 [00:42<02:11,  1.10s/it]
 24%|██▍       | 38/157 [00:43<02:11,  1.10s/it]
 25%|██▍       | 39/157 [00:44<02:09,  1.10s/it]
 25%|██▌       | 40/157 [00:45<02:08,  1.10s/it]
 26%|██▌       | 41/157 [00:46<02:07,  1.10s/it]
 27%|██▋       | 42/157 [00:47<02:09,  1.13s/it]
 27%|██▋       | 43/157 [00:48<02:08,  1.13s/it]
 28%|██▊       | 44/157 [00:49<02:06,  1.12s/it]
 29%|██▊       | 45/157 [00:51<02:04,  1.11s/it]
 29%|██▉       | 46/157 [00:52<02:02,  1.11s/it]
 30%|██▉       | 47/157 [00:53<02:01,  1.10s/it]
 31%|███       | 48/157 [00:54<02:00,  1.10s/it]
 31%|███       | 49/157 [00:55<01:58,  1.10s/it]
 32%|███▏      | 50/157 [00:56<01:57,  1.10s/it]
 32%|███▏      | 51/157 [00:57<01:56,  1.10s/it]
 33%|███▎      | 52/157 [00:58<01:55,  1.10s/it]
 34%|███▍      | 53/157 [00:59<01:54,  1.10s/it]
 34%|███▍      | 54/157 [01:00<01:53,  1.10s/it]
 35%|███▌      | 55/157 [01:02<01:52,  1.10s/it]
 36%|███▌      | 56/157 [01:03<01:50,  1.10s/it]
 36%|███▋      | 57/157 [01:04<01:49,  1.10s/it]
 37%|███▋      | 58/157 [01:05<01:48,  1.10s/it]
 38%|███▊      | 59/157 [01:06<01:47,  1.10s/it]
 38%|███▊      | 60/157 [01:07<01:46,  1.10s/it]
 39%|███▉      | 61/157 [01:08<01:45,  1.10s/it]
 39%|███▉      | 62/157 [01:09<01:45,  1.11s/it]
 40%|████      | 63/157 [01:10<01:45,  1.13s/it]
 41%|████      | 64/157 [01:12<01:44,  1.12s/it]
 41%|████▏     | 65/157 [01:13<01:42,  1.11s/it]
 42%|████▏     | 66/157 [01:14<01:41,  1.11s/it]
 43%|████▎     | 67/157 [01:15<01:39,  1.11s/it]
 43%|████▎     | 68/157 [01:16<01:38,  1.10s/it]
 44%|████▍     | 69/157 [01:17<01:37,  1.11s/it]
 45%|████▍     | 70/157 [01:18<01:35,  1.10s/it]
 45%|████▌     | 71/157 [01:19<01:34,  1.10s/it]
 46%|████▌     | 72/157 [01:20<01:33,  1.10s/it]
 46%|████▋     | 73/157 [01:21<01:32,  1.10s/it]
 47%|████▋     | 74/157 [01:23<01:31,  1.10s/it]
 48%|████▊     | 75/157 [01:24<01:30,  1.10s/it]
 48%|████▊     | 76/157 [01:25<01:29,  1.10s/it]
 49%|████▉     | 77/157 [01:26<01:27,  1.10s/it]
 50%|████▉     | 78/157 [01:27<01:27,  1.10s/it]
 50%|█████     | 79/157 [01:28<01:30,  1.16s/it]
 51%|█████     | 80/157 [01:29<01:29,  1.17s/it]
 52%|█████▏    | 81/157 [01:31<01:27,  1.15s/it]
 52%|█████▏    | 82/157 [01:32<01:25,  1.13s/it]
 53%|█████▎    | 83/157 [01:33<01:23,  1.12s/it]
 54%|█████▎    | 84/157 [01:34<01:21,  1.12s/it]
 54%|█████▍    | 85/157 [01:35<01:22,  1.15s/it]
 55%|█████▍    | 86/157 [01:36<01:20,  1.13s/it]
 55%|█████▌    | 87/157 [01:37<01:18,  1.12s/it]
 56%|█████▌    | 88/157 [01:38<01:16,  1.11s/it]
 57%|█████▋    | 89/157 [01:39<01:15,  1.11s/it]
 57%|█████▋    | 90/157 [01:41<01:15,  1.12s/it]
 58%|█████▊    | 91/157 [01:42<01:13,  1.11s/it]
 59%|█████▊    | 92/157 [01:43<01:11,  1.10s/it]
 59%|█████▉    | 93/157 [01:44<01:10,  1.10s/it]
 60%|█████▉    | 94/157 [01:45<01:09,  1.10s/it]
 61%|██████    | 95/157 [01:46<01:08,  1.10s/it]
 61%|██████    | 96/157 [01:47<01:07,  1.10s/it]
 62%|██████▏   | 97/157 [01:48<01:05,  1.10s/it]
 62%|██████▏   | 98/157 [01:49<01:04,  1.10s/it]
 63%|██████▎   | 99/157 [01:50<01:04,  1.11s/it]
 64%|██████▎   | 100/157 [01:52<01:02,  1.10s/it]
 64%|██████▍   | 101/157 [01:53<01:01,  1.10s/it]
 65%|██████▍   | 102/157 [01:54<01:00,  1.10s/it]
 66%|██████▌   | 103/157 [01:55<00:59,  1.10s/it]
 66%|██████▌   | 104/157 [01:56<00:58,  1.10s/it]
 67%|██████▋   | 105/157 [01:57<00:57,  1.10s/it]
 68%|██████▊   | 106/157 [01:58<00:56,  1.10s/it]
 68%|██████▊   | 107/157 [01:59<00:55,  1.10s/it]
 69%|██████▉   | 108/157 [02:00<00:53,  1.10s/it]
 69%|██████▉   | 109/157 [02:01<00:52,  1.10s/it]
 70%|███████   | 110/157 [02:03<00:51,  1.10s/it]
 71%|███████   | 111/157 [02:04<00:50,  1.10s/it]
 71%|███████▏  | 112/157 [02:05<00:49,  1.10s/it]
 72%|███████▏  | 113/157 [02:06<00:48,  1.10s/it]
 73%|███████▎  | 114/157 [02:07<00:47,  1.10s/it]
 73%|███████▎  | 115/157 [02:08<00:46,  1.11s/it]
 74%|███████▍  | 116/157 [02:09<00:45,  1.10s/it]
 75%|███████▍  | 117/157 [02:10<00:43,  1.10s/it]
 75%|███████▌  | 118/157 [02:11<00:42,  1.10s/it]
 76%|███████▌  | 119/157 [02:12<00:42,  1.11s/it]
 76%|███████▋  | 120/157 [02:14<00:40,  1.11s/it]
 77%|███████▋  | 121/157 [02:15<00:39,  1.10s/it]
 78%|███████▊  | 122/157 [02:16<00:38,  1.10s/it]
 78%|███████▊  | 123/157 [02:17<00:37,  1.10s/it]
 79%|███████▉  | 124/157 [02:18<00:36,  1.10s/it]
 80%|███████▉  | 125/157 [02:19<00:35,  1.11s/it]
 80%|████████  | 126/157 [02:20<00:34,  1.10s/it]
 81%|████████  | 127/157 [02:21<00:32,  1.10s/it]
 82%|████████▏ | 128/157 [02:22<00:31,  1.10s/it]
 82%|████████▏ | 129/157 [02:23<00:30,  1.10s/it]
 83%|████████▎ | 130/157 [02:25<00:29,  1.10s/it]
 83%|████████▎ | 131/157 [02:26<00:28,  1.11s/it]
 84%|████████▍ | 132/157 [02:27<00:27,  1.10s/it]
 85%|████████▍ | 133/157 [02:28<00:27,  1.14s/it]
 85%|████████▌ | 134/157 [02:29<00:25,  1.13s/it]
 86%|████████▌ | 135/157 [02:30<00:24,  1.12s/it]
 87%|████████▋ | 136/157 [02:31<00:23,  1.11s/it]
 87%|████████▋ | 137/157 [02:32<00:22,  1.10s/it]
 88%|████████▊ | 138/157 [02:33<00:20,  1.10s/it]
 89%|████████▊ | 139/157 [02:35<00:19,  1.10s/it]
 89%|████████▉ | 140/157 [02:36<00:18,  1.10s/it]
 90%|████████▉ | 141/157 [02:37<00:17,  1.10s/it]
 90%|█████████ | 142/157 [02:38<00:16,  1.10s/it]
 91%|█████████ | 143/157 [02:39<00:15,  1.10s/it]
 92%|█████████▏| 144/157 [02:40<00:14,  1.10s/it]
 92%|█████████▏| 145/157 [02:41<00:13,  1.11s/it]
 93%|█████████▎| 146/157 [02:43<00:13,  1.19s/it]
 94%|█████████▎| 147/157 [02:44<00:11,  1.17s/it]
 94%|█████████▍| 148/157 [02:45<00:10,  1.14s/it]
 95%|█████████▍| 149/157 [02:46<00:09,  1.13s/it]
 96%|█████████▌| 150/157 [02:47<00:07,  1.12s/it]
 96%|█████████▌| 151/157 [02:48<00:06,  1.11s/it]
 97%|█████████▋| 152/157 [02:49<00:05,  1.12s/it]
 97%|█████████▋| 153/157 [02:50<00:04,  1.11s/it]
 98%|█████████▊| 154/157 [02:51<00:03,  1.10s/it]
 99%|█████████▊| 155/157 [02:52<00:02,  1.10s/it]
 99%|█████████▉| 156/157 [02:54<00:01,  1.10s/it]
100%|██████████| 157/157 [02:54<00:00,  1.16it/s]
100%|██████████| 157/157 [02:54<00:00,  1.11s/it]
computing/reading reference batch statistics...
computing sample batch activations...

  0%|          | 0/2 [00:00<?, ?it/s]
 50%|█████     | 1/2 [00:01<00:01,  1.10s/it]
100%|██████████| 2/2 [00:01<00:00,  1.23it/s]
100%|██████████| 2/2 [00:01<00:00,  1.17it/s]
fid_custom.py:31: UserWarning: mid matrix contains NaN/Inf; returning NaN for Tr(sqrt(M))
  warnings.warn("mid matrix contains NaN/Inf; returning NaN for Tr(sqrt(M))")
computing/reading sample batch statistics...
Computing custom FID...
FID(custom): nan
Saved report to /gemini/space/zhaozy/guzhenyu/UAVFlow/UAV_Flow_base/exps/jsflow-experiment/samples/fid_custom_0.5.txt