Loadingā¦
23
A guy on r/LocalLLaMA told me my 3B model's tokenizer vocab was wasting 40% of my context, and he was right
I switched from a 32k BPE vocab to 16k and my little summarizer went from 12 seconds per page to 7 on the same laptop, so now I want to know if anyone else here has measured vocab size against actual inference speed or if I just got lucky with my data.
1 comments
Log in to join the discussion
Log In1 Comment
holly_craig1mo ago
Reminds me of when I cut my old guitar pedal board down and somehow played faster.
4