firstly , i appreciate the effort and work put to do this setup.
due to budget concerns , im in the same boat as well
Im in India, the price of 4090's are nearing 5090's price.
and 4090's are low in stock too..

i watched the whole stream of geohot archive where he built the 2 x amd xtx 24gb pc build for tinygrad.
and thought of buying 2 xtx cards..

for my setup.. its just ubuntu,c++ and raw gpu compute code(no pytorch or any external libs) (it can be hipc or cuda . i really dont care tbh)
are the drivers stable ? to setup multi-gpu distributed training !
any challenges that you have gone so far.. ?
could you please elaborate on those issues that you have faced so far..
as long as the system doesnt crash or low level bugs in the rocm env..
im willing to buy 2 xtx cards ..
it would really help me to clear my doubts

It was a very interesting article, I would like to know how much is the total cost for having the right Mother Board, Cooling system and the two AMD cards. Can you share that specs?

Thanks for the kind word! Well, there's no right motherboard. It depends on your needs.

Take my retired work pc as an example, here's the list of components and the current price

Motherboard - ASUS AMD AM4 ROG X570 Crosshair VIII Hero (Wi-Fi), $287.58
CPU - AMD AMD Ryzen 9 3950X, $289.99
Memory - G.SKILL Ripjaws V Series (XMP) DDR4 RAM 128GB (4x32GB), $319.99
Powersupply - be quiet! Dark Power Pro 13 1600W, $289.99
Case - Obsidian Series 500D, $200 (probably not available anywhere)
GPU - AMD 7900XTX * 2, $2,200

So that would be $3,587.55 in total. As you mentioned cooling, it's indeed a problem. The two GPUs are too close, and the fan of the first card has no space to blow hot air out. As a result, the first GPU would be way hotter than the second one. The temperature can come to around 110 C, its limit. I usually open the case and blow air with a fan into it directly.

Ideally, if budget and time is not a problem, I would do water cooling. Also, this AM4 platform motherboard with 2 GPU slots used will making the second slot slower than the first one, due to the limited bandwidth of PCIe. If you really want to fully utilize the PCIe lanes of the two GPUs, i.e, memory bandwidth, I would recommend go with AMD Ryzen Threadripper. But it ain't cheapt tho, just the motherboard could set you back $1K. If money is not a problem, that's what I would do.

It depends on your workload, if you need beefy CPU for pre-processing data before feeding them into GPUs, you may need to consider buying higher end Threadripper. But again, they are very expensive, haha 😅

Thank you a lot, this is a very useful information. You rock!

The problem with AMD is prevalence of preexisting CUDA code. People should learn OpenCL to save everyone's money.

Yeah, if they wrote code with CUDA previously that would indeed be a problem. But other than that, I think AMD GPU is underrated.

People should also check Tinygrad out. The frontend API is very close to PyTorch, basically followed PyTorch's interface. It's AI era, I think there should be tool for migrating from PyTorch to Tinygrad automatically.

Found this article via ChatGPT while looking for suggestions to build a 2x&7900xtx build. I love tinygrad as well and my GPU choice was inspired by tinybox red as well. I working on the fast.ai course and would like to use Pytorch on this system. I tried to rent this gpu but no cloud providers literally none have it. My faith in AMD comes from the using the MI300x gpu on cloud for some work and facing literally no issues with it. Plus AMD was kind enough to give me ~150$ of credits which is an absolute plus.

I'm intrigued by the inter gpu communication bit in tinygrad. Also, I'd love if you could run some experiments on your setup for me. I want to run the code described in https://www.answer.ai/posts/2024-04-26-fsdp-qdora-llama3.html on the 2x7900 xtx setup.