What is malloc doing, anyway?

amelius

If the DRAM scarcity continues, not much.

assumed_throwaw

Tangential, but the complexity and over-head of the malloc abstraction is precisely what made me very intrigued by "stack-only" languages, such a recently published Goose. There is something very elegant and exciting about the idea of (almost) completely getting rid of it. Just as modern programming languages have developed good ergonomics that replaces a garbage collector, the same could hopefully be done to replace heap-allocated memory.

wpmaven

I found this one very well explained time ago, for those looking for more details: https://sploitfun.wordpress.com/2015/02/10/understanding-gli...

TZubiri

like many C functions you must remember that C is not low level, you won’t learn how memory is allocated by inspecting malloc() anymore than you would learn how characters are printed to the screen by reading printf() or how a file is written by reading write(). you will only learn about the thin or heavy wrapper that C is responsible for, the real magic is happening in the kernel and the device firmware

OhMeadhbh

One of my pet peeves is when people, trying to make a point, use code that doesn't compile. I mean, it's fine when you're sketching out code on a white board. But on an article you've had time to prepare, it's a little annoying. Take for instance, this code fragment: .. static void *ptrs[1000]; .. .. int main(void) { .. for (int i = 0; i < 1000; i++) .. ptrs[i] = malloc(16); .. } .. } I'm pretty sure what they //meant// to write was: .. #include <stdlib.h> .. .. static void *ptrs[1000]; .. .. int main(void) { .. for (int i = 0; i < 1000; i++) .. ptrs[i] = malloc(16); .. } .. return( 0 ); .. } I understand why the author produced the code on top instead of the code at the bottom. The code on the bottom includes features extraneous to the conversation. The include directive and return statement only serve to distract from the point the author is making. If you read far enough down, you'll see he has links to a broken GitHub repo presumably with code fragments. (or maybe GitHub itself is broken today or it decided it didn't like me trying to access the repo from an AT&T fiber line.) I'm sure when GitHub decides to behave and serve bits like it's supposed to, there will be compilable code so readers can understand the author's intent within the context it was intended to be viewed. Which is to say: 1. It's probably okay to have only code fragments in the main part of your discussion. Please provide a note that there is code which shows your snippet in the context of a compilable program. And put that note somewhere near the code fragment. Or... consider putting a note at the beginning of the article saying something like "Compilable versions of code fragments in this article are available at a GitHub repository referenced at the bottom of the page." 2. Do not depend on GitHub to hold your code fragments. You do not know when it will be down or which readers it will randomly decide to block. 3. I really do think you should include more of the "layout" code. When reading "The Address Space" section, it was unclear where the initialized, uninitialized and local came from. I initially thought the author was using a version of C I had not yet encountered that defined these symbols. 4. Since the code uses ``sbrk()``, the author should probably have cautioned readers the code isn't POSIX compliant, but should compile and run on Linux systems using gcc. There's a comment about demonstrating it with glibc, but I think that's not a strong enough warning since you can compile programs with clang that are intended to link against glibc. I doubt there are many QNX users out there who expect random code from the internet to run on their systems without alteration, but it seems good form to describe the environment you are targeting. Now I'm wondering if this is expected to work on Darwin as well as Linux. 5. It might be a decent idea to order the SHOW() statements in the code in "The Address Space" section to follow the order of the diagram at the bottom of that section: .. SHOW( "local (stack)", &local ); .. SHOW( "printf (libc)", printf ); .. SHOW( "program break", sbrk(0) ); .. SHOW( "malloc(16) (heap)", malloc(16) ); .. SHOW( "uninitialised (bss)", &uninitialised); .. SHOW( "initialised (data)", &initialised ); .. SHOW( "main(text)", main ); 6. In the "Chunks and Headers" section, it may be interesting to mention what the flags in the allocation headers actually mean. 7. glibc does late fill, right? I can't remember. If it does, it might be useful to mention in the Demand Paging section since it affects the program logic. Now that I've yelled at the cloud a bit, allow me to say, I enjoyed the article. Things I liked: 1. The author used the exact strace command they used to produce the output discussed in the text of the article. 2. The graphics are quite nice. 3. Despite grousing about fragments being fragments, they do a good job of illustrating points made in the text. EDIT: After complaining about GitHub not being available, it's finally come back. The web interface is still borked for me, but I can clone the repo mentioned at the bottom of the page and view the local copy of the source.

0xbadcafebee

This shows you how you could "secure" memory code in C by using more robust implementations of these functions. But it doesn't show you that you don't even need them anyway. I think in my entire career I've only used dynamic memory in C twice. The rest of the time I use functions with static allocations, buffers, unrolled loops, etc. Maybe it's paranoia, or habits from embedded programming, or being a control freak... But my C programs don't get memory leaks or crash from memory issues.

K0IN

When getting into wasm when it was new and had clang support, I wrote my own libc and malloc, which was so a great learning all these low level functions and tricks (free lists for example), I also like that I can read wasm quite well, and tooling is good to just return a .wat and read it.

bajard

Malloc() is sloooow. Even more if you also use free(). Consider allocating a single array and scratching there if possible.

projectileboy

Good learning resource for this (and related) topics: https://cs61.seas.harvard.edu/site/2026/

senderista

If you want to study how modern allocators work I'd recommend something like mimalloc, which has a reasonably approachable codebase. No brk/sbrk in sight, btw (or boundary tags, or splitting/merging...).

incognito124

mental reminder to finish my history of malloc writeup. It's almost 20 llm-less pages with context and examples. goes through general purpose malloc like dlmalloc, mimalloc, tcmalloc, jemalloc and others, while also mentioning some purpose-specific ones like scudo.

giannischatz

A thing worth adding for anyone reading: in long-running PHP processes (workers, queue runners, Laravel Octane), the overhead of malloc/free patterns compounds in ways the request/response model hides. A page handler that leaks 2KB per request is invisible under php-fpm (the process dies on the next request anyway) but becomes a slow memory creep in a queue worker that runs for hours. The fix usually isn't "call unset more" — it's recognizing which objects Zend's refcount can't collect without a full GC cycle (circular references, mostly), and forcing `gc_collect_cycles()` at known-safe points. Took me a while to realize that in production.

virtyaluk

I highly recommend reading Professional Linux Kernel Architecture by Wolfgang Mauerer, which describes the Linux memory management system in great detail.

loeg

This is mostly describing a kind of obsolete allocator that uses the brk to adjust a single contiguous heap range. This is basically irrelevant with virtual memory and mmap. The good malloc impls (jemalloc, mimalloc) don't do this.

gebt

This phrack article on malloc is highly recommended: Vudo malloc tricks by Michel "MaXX" Kaempf (2001) https://phrack.org/issues/57/vudo-malloc-tricks

BatchJob

allocating memory, usually.

syngrog66

As a rough rule of thumb: if you want smallest latency for small allocs use malloc not mmap, and for largest allocs prefer an ANON mmap. The blog post goes into more fractal detail, in general.

puffybuf

I tried this on OpenBSD using ktrace / kdump. I added some utrace calls and had to increase the amount allocated significantly to get the mmap calls to show up. obsd just calls mmap with prot_read|prot_write meaning pages may be read/written.

Panzerschrek

malloc has terrible interface. It has no way to specify custom alignment. So, if more than 8 or 16-bytes alignment is needed, custom user-code tricks are necessary (allocate more than necessary, round up). It has no way to report how many bytes are actually allocated. Sometimes it may be useful, like if I reserve 9 bytes for a string and underlying allocation granularity is 16 bytes, I want to know that I can increase string size up to 16 bytes without calling malloc again. It's a single entry point allowing no allocation size customizations. So, if I allocate objects of fixed size, I still need to pass a dynamic size parameter to malloc and it needs to perform branching in case if it does have some optimizations for particular sizes. free has no size parameter, which is a mistake. This forces the allocator to maintain size values somewhere inside, which is often an unnecessary overhead, since user code already knows size (like for fixed-size allocations or containers storing explicit capacity). With interface fixes described above a malloc implementation can be much more performant and (sometimes) even easier to use.