Run Deepseek locally on a 24g GPU: Quantizing on our Giga Computing 6980P Xeon

Category

With the help of some forum friends, Wendell now has the smallest model of Deepseek so far! Thank you Giga Computing, Kioxia, and Intel, for lending us all the goods to use for our testing!

Hugging Face Link https://huggingface.co/ubergarm/DeepSeek-R1-0528-GGUF

Forum Link https://forum.level1techs.com/t/deepseek-deep-dive-r1-at-home/225826/13…

The Chassis https://www.gigabyte.com/Enterprise/Rack-Server/R284-A91-AAL1

Kioxia https://americas.kioxia.com/en-us/business/ssd/data-center-ssd/cd8-r.ht…

0:00 Intro
2:17 Online vs Local
3:40 Hardware
9:36 Using the Machine and Results
18:00 Conclusion

https://www.youtube.com/@Level1LinksWithFriends
Go Subscribe to our New Links with Friends Channel!
Starting in July, we're going to post links with friends on its own channel. This is still Level1Techs, we’re just separating the content types so they can take their own algorithm journeys. We also plan on posting more videos with the trio on that page!