Yeah, this thing is unusual. It resembles a consumer GPU but using normal DRAM with more channels, so then you get more RAM but less memory bandwidth than GDDR/HBM, which is the same thing Apple does. And the TDP is pretty attractive for what it is.
But what is that amount of SRAM for? It simultaneously seems like a lot, because SRAM is so expensive, but still not very much, because what can you fit in 0.5GB? Is there a specific workload that needs something that looks like this?
Ok so it’s significantly worse than Apple M6 and we don’t know if it’s cheaper or not
Edit: Ah sorry you’re right, the bandwidth is lower but the TOPS is a lot higher. I’m not sure how much effect you get from higher TOPS with such low bandwidth though.
Aren't these specs already outdated? Seems meh. I would think players would like to bring something new to the table instead of bringing what was okay two years ago or so.
Give me 1TB RAM and 5TB/s or something that exciting.
For most purposes, this seems reasonable? This looks like a single-height card, so fitting four of these in a workstation or server would give 512 GB RAM and 2 TB/s total bandwidth. That's pretty attractive for a lot of current LLMs, even if one wishes for 1 TB capacity at 5 TB/s bandwidth.
It may be a semantic thing but I don't think bandwidth pooling works like that? Unless your blocks are distributed perfectly between cards there's going to be data passed between each card and from VRAM to compute. Correct me if I'm wrong though
But what is that amount of SRAM for? It simultaneously seems like a lot, because SRAM is so expensive, but still not very much, because what can you fit in 0.5GB? Is there a specific workload that needs something that looks like this?
Edit: Ah sorry you’re right, the bandwidth is lower but the TOPS is a lot higher. I’m not sure how much effect you get from higher TOPS with such low bandwidth though.
Qualcomm Cloud AI 100 Ultra has 800 TOPS, almost 10 times more.
Give me 1TB RAM and 5TB/s or something that exciting.
Plenty of businesses want in local inference and are priced out.