Samsung's Processing-in-Memory (PIM)

reliabilityguy

Interesting that Samsung still pursues PIM. IIRC they had a paper in ISCA21 or 22 where they showed HBM2 module with PIM, which back then impressed me quite a lot. That being said, I am not sure what’s the killer application for this technology, and without such application adoption is unlikely.

consp

So you basically dispose of cache for the memory region used? I wonder what the offsets of the cache misses is going to be in practice (the article addresses it but there is no solution/impact given by samsung).

userbinator

In-memory computation was already possible with regular DRAM: https://news.ycombinator.com/item?id=22712811 Add a new set of CPU instructions like “rep macb” ...and it's been long enough now, that I can say there was an effort to implement this on standard x86 memory controllers and have the existing string instructions do so, back in the days of SDR SDRAM, but the tradeoffs weren't (yet) in favour.

saejox

If we could buy a 64gb stick and run a 32b model with 30tps on it. This could sell

bhouston

I wrote up a theoretical post here about LMM performance of a MacBook Pro with PIM memory: https://ben3d.ca/blog/m5-max-samsung-lpddr5-pim-650-tokens-p...

pragma_x

What I find amusing about moving compute to a RAM bank is it _almost_ resembles where we were with ISA-based extended RAM back in the 1980's. Some cards featured a CPU that took over the whole system and/or functioned like an upgrade. Others were a "computer on a card" that provided other features. I think this goes to show how cyclic tech can be. So, something like Samsung's invention here might have gained traction, as overcoming the slow PC ISA bus would have been a huge accelerator, kind of like where we are now.

mr_toad

“ Each PIM block only has fast access to its locally attached DRAM bank. All other input data has to be brought in through the DRAM chip’s comparatively constrained external interface. PIM blocks can’t directly exchange data with each other, so the host has to move data using regular DRAM reads and writes if one PIM block needs to use results generated by another.” So how big are these banks? If you can’t fit the weights of a layer into one bank then presumably you lose a lot of the speed gains.

londons_explore

Whilst processing in memory is clearly the future, I am unconvinced by this implementation. Matrix multiplication involves getting every entry of the input and output matrices to be at the same multiplier at the same time. (Ie. N^2). To do that, a lot of data movement needs to happen. Movement is the main thing - the multiplication and addition is a sideshow as far as energy and silicon space is concerned. You need a 'around the chip' ring shift register to pass every element of one matrix past every element of the other.

ginko

Feels like the most realistic/short-term way to make use of this would be to set up some barebones RTOS to run from CPU cache with the PIM memory being used for compute only and use the device as a network attached accelerator.

plywoodShadow

What about energy consumption? Wouldn't active cooling be needed for RAM as well as for CPU and GPU?

krater23

Self changing RAM and a complex way to interact with it in software. A new security nightmare is emerging.

OptionX

So instead of putting more cache on the cpu you just put the cpu on the cache.

harshaw

This is somewhat orthogonal to the article, but the whole bubble on AI data centers seems to presume that the need for compute is so massive that it far exceeds the expected optimizations we would expect with at scale inference (PIM, ASICs, etc). I would expect that there is a set of optimizations like this one (or variations) that would someone negate the buildout. But it's not really discussed.

hham

memory plus memory bandwidth > GPU, this is the gist of it.

samuelknight

I saw them present a similar concept at Hot Chips in 2020 or 2021. It's still a cool idea, however people should remember that there are like 20 of these exotic accelerators designs pitched at trade shows every year that go nowhere.

tesnorindian

In memory compute has been in talks since LLMs took up. I remember few flocks were trying to get RISC V cores in the memory like these papers https://arxiv.org/abs/2602.01827

bob1029

The tradeoff with putting the compute in the memory is that you have to know exactly where the dependent information will be at all times. Most problems do not fit this pattern very well. AI, gaming and crypto being the most obvious exceptions. It is incredibly constraining to develop applications using specialized hardware like this. You might as well spin out an ASIC for whatever it is you are doing. All 3 applications noted above eventually got their own flavors. I think the Von Neumann bottleneck is mostly a feature. The fact that communication of information across distances is expensive should not be immediately assumed to mean that it is universally flawed to do this. You are paying for something when you use all those joules. I'd argue we are usually wasting our energy with regard to information communication (e.g., lighting up a network interface & copper because we couldn't be bothered to use SQLite), but other times this stuff is fundamentally required for practical solutions to exist.

howdyhowdy

I thought DMR and Venice were supporting memory encryption by default. With the keys living on the CPU side and no standards for key sharing, I wonder how this will gain traction.

rivetfasten

The discussion about cpu caching challenges etc seems like it's missing the point. Wouldn't it be more likely to DMA the results to a GPU anyway?

HarHarVeryFunny

I remember taking VLSI design as part of my Comp. Sci. degree at Bristol, UK c.1980, using the Conway & Mead book, and "Commingling of Processing and Memory" was mentioned even back then. Obviously you (eventually) need your data where the compute is, especially in a non-von-Neumann architecture where moving data around isn't an option even if you were OK with the performance drop. It seems kinda obvious that eventually AI will be implemented as low power dataflow custom chips integrating memory/state & compute, but who knows!