| # Split downstream outputs |
|
|
| Several downstream profiling caches are larger than Hugging Face Hub's 50 GB |
| single-file limit. Those files are uploaded as ordered 10 GiB chunks with names |
| ending in `.pt.part-000`, `.pt.part-001`, and so on. |
|
|
| The whitening-only `.pt` files are uploaded directly because they are below the |
| Hub hard limit. |
|
|
| ## Reassembly |
|
|
| Download the relevant directory and concatenate the parts in lexical order. |
| Shell glob ordering is sufficient because the suffixes are zero-padded. |
|
|
| ```bash |
| cat artifacts/downstream/outputs/llama2_r0.2_baseline/meta-llama_Llama-2-7b-chat-hf_profiling___calib_mix_jsonl_128_0.pt.part-* \ |
| > artifacts/downstream/outputs/llama2_r0.2_baseline/meta-llama_Llama-2-7b-chat-hf_profiling___calib_mix_jsonl_128_0.pt |
| |
| cat artifacts/downstream/outputs/llama2_r0.2_decodeshare_a2/meta-llama_Llama-2-7b-chat-hf_profiling___calib_mix_jsonl_128_0.pt.part-* \ |
| > artifacts/downstream/outputs/llama2_r0.2_decodeshare_a2/meta-llama_Llama-2-7b-chat-hf_profiling___calib_mix_jsonl_128_0.pt |
| |
| cat artifacts/downstream/outputs/svdllm_whiten_r0.2/meta_llama_Llama_2_7b_chat_hf_profiling_wikitext2_128_0.pt.part-* \ |
| > artifacts/downstream/outputs/svdllm_whiten_r0.2/meta_llama_Llama_2_7b_chat_hf_profiling_wikitext2_128_0.pt |
| ``` |
|
|
| ## Uploaded downstream files |
|
|
| - `llama2_r0.2_baseline/`: baseline whitening output and split profiling cache. |
| - `llama2_r0.2_decodeshare_a2/`: DecodeShare A2 whitening output and split |
| profiling cache. |
| - `svdllm_whiten_r0.2/`: SVD-LLM whitening output and split profiling cache. |
|
|
