LocalAI LLM Testing: Distributed Inference on a network? Llama 3.1 70B on Multi GPUs/Multiple Nodes

This week in the RoboTF lab: Blown power supply Saying goodbye to some of the 4060's Most importantly hitting the topic of Distributed Inference! It's a long video... This week we are taking Llama 3.1 70B at a Q5 quant running 56k of context through the gauntlet with several GPUs and across Nodes in a distributed swarm of llama.cpp workers! The whole lab is getting involved in this one to run a single model. Both GPU Kubernetes nodes (These are affiliate-based links that help the channel if you purchase from them!) 3x 4060Ti 16GB https://amzn.to/3NeSEGT 6x A4500 20GB https://amzn.to/3TXtAYR 1x 3090 24GB https://amzn.to/3Yf57AI LocalAI docs on distributed inference: https://localai.io/features/distribute/ Llama.cpp docs: https://github.com/ggerganov/llama.cpp/blob/master/examples/rpc/README.md Link to blog on Llama 3.1 and memory requirements https://huggingface.co/blog/llama31 Just a fun day in the lab, grab your favorite relaxation method and join in. Our website: https://robotf.ai Machine specs here: https://robotf.ai/Machine_Lab_Specs GPU Node: (These are affiliate-based links that help the channel if you purchase from them!) 30cm Gen 4 PCIe Extender https://amzn.to/3Unhclh 20cm Gen 4 PCIe Extender https://amzn.to/4eEiosA 2 Tb NVME https://amzn.to/3XYSokg EVGA SuperNova 1600 G+ Power Supply https://amzn.to/3XWorBB 128GB Lexar SSD https://amzn.to/3TZYYGh Nocuta NH-U12DX i4 CPU Cooler: https://amzn.to/3TZ7O6R G.SKILL Ripjaws V Series DDR 128GB Kit https://amzn.to/4ev174M Asus WS X299 SAGE/10G Logic Board https://amzn.to/4eOskz2 Core I9 7960x https://amzn.to/3NhMaHy Open Air Case https://amzn.to/3U08Y27 Remote Power Switch https://amzn.to/3BubQOg GPU Bench Node Open Air Case https://amzn.to/3U08Y27 30cm Gen 4 PCIe Extender https://amzn.to/3Unhclh 20cm Gen 4 PCIe Extender https://amzn.to/4eEiosA 1 TB NVME https://amzn.to/4gWFcFb Corsair RM850x https://amzn.to/3NkITa4 128GB Lexar SSD https://amzn.to/3TZYYGh G.SKILL Ripjaws V Series DDR 64GB Kit https://amzn.to/4dAZrWm Core I9 9820x https://amzn.to/47UuIST Nocuta NH-U12DX i4 CPU Cooler: https://amzn.to/3TZ7O6R Supermicro CX299-PGF Logic Board https://amzn.to/3BxbWVr Remote Power Switch https://amzn.to/3BubQOg Recorded and best viewed in 4K Your results may vary due to hardware, software, model used, context size, weather, wallet, and more!