card: add llama-server speculative decoding example
Browse files
README.md
CHANGED
|
@@ -65,7 +65,7 @@ Carbon-500M shares the HybridDNA vocab, so it works as a drop-in draft model. Gr
|
|
| 65 |
hf download HuggingFaceBio/Carbon-500M-GGUF carbon-500m-bf16.gguf --local-dir .
|
| 66 |
```
|
| 67 |
|
| 68 |
-
Then run with `--model-draft`:
|
| 69 |
|
| 70 |
```bash
|
| 71 |
./build/bin/llama-speculative \
|
|
@@ -75,6 +75,24 @@ Then run with `--model-draft`:
|
|
| 75 |
-n 256 --temp 0
|
| 76 |
```
|
| 77 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 78 |
### Likelihood scoring
|
| 79 |
|
| 80 |
The source card's Python `score()` function computes mean log-prob per DNA token. In llama.cpp the closest tools are `llama-perplexity` for corpus-level perplexity (`perplexity = exp(-mean_logprob)`):
|
|
|
|
| 65 |
hf download HuggingFaceBio/Carbon-500M-GGUF carbon-500m-bf16.gguf --local-dir .
|
| 66 |
```
|
| 67 |
|
| 68 |
+
Then run the standalone tool with `--model-draft`:
|
| 69 |
|
| 70 |
```bash
|
| 71 |
./build/bin/llama-speculative \
|
|
|
|
| 75 |
-n 256 --temp 0
|
| 76 |
```
|
| 77 |
|
| 78 |
+
Or serve the 3B with the 500M draft (`llama-server` accepts the same `-md` flag):
|
| 79 |
+
|
| 80 |
+
```bash
|
| 81 |
+
./build/bin/llama-server \
|
| 82 |
+
-m carbon-3b-bf16.gguf \
|
| 83 |
+
-md carbon-500m-bf16.gguf \
|
| 84 |
+
--draft-max 16 --draft-min 1 \
|
| 85 |
+
--port 8080
|
| 86 |
+
```
|
| 87 |
+
|
| 88 |
+
```bash
|
| 89 |
+
curl -s http://localhost:8080/completion -d '{
|
| 90 |
+
"prompt": "<dna>ATGCGCTAGCTACGATCGATCGTAGCTAGCTAGCTAGCTACG",
|
| 91 |
+
"n_predict": 256,
|
| 92 |
+
"temperature": 0
|
| 93 |
+
}' | jq -r .content
|
| 94 |
+
```
|
| 95 |
+
|
| 96 |
### Likelihood scoring
|
| 97 |
|
| 98 |
The source card's Python `score()` function computes mean log-prob per DNA token. In llama.cpp the closest tools are `llama-perplexity` for corpus-level perplexity (`perplexity = exp(-mean_logprob)`):
|