> Nested data stays inline. A string, an array of strings, a record with variable-size fields and an array of those records are each one contiguous block with no pointer in it. A record that is 160 bytes and an allocation in C++ is 29 bytes and none in Goose.
It means that to do variable sized things like vectors and strings, C++ often ends up having to allocate their contents dynamically. Which results in data structures with lots of 64-bit pointers over multiple allocations.
Even Rust's "enum" feature, which can store different sized things all in the same space, does that at the cost of the smallest variant being just as big as the biggest.
Goose can do all of that not only inline (no pointers), but with variable size, meaning small items take less space. It's like a serialized data representation, but iterable. It makes for big space savings vs these languages.
As far as I can tell, you're essentially creating a new stack for every dynamically-sized variable (because otherwise nothing would be able to grow by bump allocation without moving). But this means you're creating a separate virtual memory allocation for each of these objects, which (I presume?) then rounds the size up to the OS's minimum page size (4KiB or similar).
Doesn't that waste a lot of space? Or is there something I'm missing?
Also it would seem to force every allocation to go through the OS kernel to allocate unless there's some trick to avoid that.
I think what you might be missing is that you can also do a sort of livenes analysis to determine how many dynamically-sized variables are "awake" at the same time at most. Then you only need to create as many stacks as that number, because you can reuse the stacks of "sleeping" variables (since the "awake" variables are guaranteed to be freed once their block goes out of scope, and threads don't share any memory so no data race worries regarding those stacks).
For example, suppose we have a function with a growable array that calls another function (without passing said array) that also uses a growable array as some temporary scratch space. Then the first function's array isn't being used when the second is sleeping, so the inner function's array could safely share the stack because it will all be freed with a single pointer restore once the second function exits.
Now, I actually don't know how the compiler works but as I understand it does things similar to my example to optimize the number of stacks needed (but let's wait for WvO to confirm or deny whether my understanding matches reality).
Only top level growable arrays get their own "stack", if you create an array of arrays it only uses one stack.
How you deal with the inner stack growing depends, ideally they don't grow (construction only), or you can make them fixed size, or with a capacity, or in the worst case, as a pool. The language has facilities for all of these.
> Threads that share nothing. A worker is compiled as a separate program with its own memory, and flat values cross typed queues as a memcpy. Data races, locks, atomics and memory orderings do not exist in the language.
how to perform this copy without atomics?
do the workers run as threads or processes?
also you claim it's faster than rust and c++. but for certain kinds of software it's certainly (and substantially) slower, right?
The copy is made by the runtime, which obviously does use atomics underneath, but you don't ever use these yourself.
The workers are threads. The separate memory is the concurrency model, but it's still single process, multiple threads.
The "faster than" is on average, and both in my own benchmarks and independent benchmarks you can find examples of algorithms where it is slower, but they tend to be the exception. "substantially" slower I haven't found yet.
You noted AI creating a large amount of this code base with yourself directing it. Do you have a process that keeps you sane while doing this over such a large codebase?(especially for a brand new concept and implementation) [Obviously congrats and can't wait to read through to have deeper questions!]
A big part of this making it base the code of this new compiler on my previous compiler for the Lobster Programming Language. I had it go thru that codebase, distill an elaborate code style guide from it, and used that to make it write the new code pass by pass in that style. That made me feel "at home" as if it is my code, easier to review etc, and easier to know we're going in the right direction by seeing similar structure to my past compiler work.
Once the initial compiler was there it became easier, as Fable is rather good of working within the context of what is already there.
Thanks much appreciated!! Feeling silly for not noticing my best experience with AI in larger/complex cases has been languages/frameworks that are opinionated in usage/design(zig/go/elixir/rust)
Yes, the compiler guarantees references to data structures can't escape. I believe it does that "perfectly" :)
It certainly requires the programmer to rethink how they structure their data, as you have to think of where it is going to be stored, and how to store it in a way such that it will have similar lifetimes to what it is stored with, to get the most out of the language.
What happens if the memory is freed in a different order to allocation? Does the language disallow that? Or does it get allowed but mean the stack can't shrink?
The compiler automatically optimally allocates all arrays currently in scope that are able to grow to "stacks" (address spaced reserved areas).
The programmer choose what to store where, so you typically store things in arrays by lifetime. Goose gives you many more tools to do that, it can store entire variable sized trees of an enum type in a single array.
You try to avoid having to deallocate in mixed/random order, but if you absolutely must, you can turn arrays into pools, basically.
I'm a bit confused about how you could decide the number of stacks at compile time. I can see you could see how many arrays are lexically in scope. However if the code puts one dynamic array inside another dynamic array I can get to an unbounded number of total stacks without ever needing more than two in lexical scope at once. Do you also add stacks at runtime? Like a stack of stacks?
I'm sorry if I'm being dense here but I can't see how that would work?
The spec on your GitHub says: "Two simultaneously-live resizable values never share a stack."
So I surely can't have a dynamic array of dynamic arrays share a single stack? That's an unbounded number of resizeable values sharing a stack?
You could make it work on a single stack but that would involve having non-contiguous memory for each inner array since you can add to any array in any order.
The spec implies that we need one stack per element of the outer dynamic array which would let you have contiguous memory for each inner array, but you wouldn't know how many stacks you needed at compile time.
I think they have slightly different places on the "speed vs ease of use" pareto frontier.
With Goose, the core design favors speed over everything else. Though not only was I able to make that safe, it is fairly pleasant to use. But it remains a low level systems language that requires you to think hard about data organization.
Lobster is a waaay more high level language made to be pleasant to use first, fast second. Surprisingly though, it can get pretty close in speed to C++/Rust nowadays too, but still a lot slower than Goose.
Do you have plans to continue using AI to improve Lobster too, e.g. working on implementation speed, or perhaps self-hosting?
How have you found AI at writing Goose or Lobster code, if you give them a skill or something, are they relatively competent at actually writing code in the language?
Both are really interesting projects, BTW - thank you! I've been meaning to try Lobster out but haven't found time, I think this might be the push I needed.
Self-hosting would be cool too, but for both Lobster and Goose I've tried to keep it simple for people to build the language in any environment, which is why its still all in C++.
I don't use any "skills". From what I can see, the #1 way an AI produces "good code" is to simply work in the context of existing good code. I don't need to hold it by a style guide, it seems to infer that from context just fine. At least Fable does, other models less so.
Part of this is that 64-bit address spaces make this design practical, and almost all languages in use today were designed with the limitations of 32-bit in mind. Once you can assign arrays huge amounts of address space, you don't have to realloc, which gets you stable pointers, and a lot of the other Goose features.
Most of my current Goose projects are games, not surprisingly, but I have some language/tooling ideas as well.
As for Amiga E, I am very proud of having made it way back when obviously, but it doesn't stack up against my more modern languages (Goose and Lobster), so I think I'll leave it be :)
Goose is pretty cool. The idea is that you don’t need to make representable all programs: a language for a restricted purpose is ok. As long as you can statically specify the number of growable memory regions, you can use Goose. However, since there’s no garbage collection, you’ll probably need to be careful about memory use while a program is running. Ie: the stacks can get too big if you keep growing them, even as old data is not used any more.
Still, for a large number of programs - like data-oriented applications - it’s probably a good fit. It might work ok with entity-component game systems as long as you reuse old memory…
"That looks really neat! The automatic analysis and use of multiple data stacks and resizable fields takes it from "lots of work can be done this way" to "basically everything can be".
The relative references and built in binary structure validation also enable direct memory mapping of data structures and immediate use of network data. "Parsing" file and network data is often 100x slower than it needs to be." - John Carmack
I seem to remember the ML guys went down this route. I believe they called it regions.
From what little I read in the literature I seem to recall that there were enough programs with quite a bit more memory usage that they decided this approach was a dead end.
Yes, ML Kit region inference is probably the closest prior art to Goose.
Back then though, they couldn't lean on large 64-bit address spaces like Goose can. They implemented all of this with a chain of blocks of memory instead, and tracked most of it at runtime that Goose now does at compile time.
Such a cool idea! I've been toying around making a few libraries to force languages to do that direct io zero-read mapping to data structures. Ofcourse this looks way better and infinity less hacky, but it is a fun and trippy experience to find someone working in the same direction as you independently.
The Twitter/X link doesn't show the full thread without a login. It also doesn't tell you that something is missing, so I guess it's normal for people to think they're seeing everything.
> Nested data stays inline. A string, an array of strings, a record with variable-size fields and an array of those records are each one contiguous block with no pointer in it. A record that is 160 bytes and an allocation in C++ is 29 bytes and none in Goose.
I’m having trouble parsing that last sentence.
Even Rust's "enum" feature, which can store different sized things all in the same space, does that at the cost of the smallest variant being just as big as the biggest.
Goose can do all of that not only inline (no pointers), but with variable size, meaning small items take less space. It's like a serialized data representation, but iterable. It makes for big space savings vs these languages.
Doesn't that waste a lot of space? Or is there something I'm missing?
Also it would seem to force every allocation to go through the OS kernel to allocate unless there's some trick to avoid that.
For example, suppose we have a function with a growable array that calls another function (without passing said array) that also uses a growable array as some temporary scratch space. Then the first function's array isn't being used when the second is sleeping, so the inner function's array could safely share the stack because it will all be freed with a single pointer restore once the second function exits.
Now, I actually don't know how the compiler works but as I understand it does things similar to my example to optimize the number of stacks needed (but let's wait for WvO to confirm or deny whether my understanding matches reality).
How you deal with the inner stack growing depends, ideally they don't grow (construction only), or you can make them fixed size, or with a capacity, or in the worst case, as a pool. The language has facilities for all of these.
So, I read it as, "C++ uses 160 bytes and an allocation, when Goose requires 29 bytes and no allocation".
how to perform this copy without atomics?
do the workers run as threads or processes?
also you claim it's faster than rust and c++. but for certain kinds of software it's certainly (and substantially) slower, right?
The workers are threads. The separate memory is the concurrency model, but it's still single process, multiple threads.
The "faster than" is on average, and both in my own benchmarks and independent benchmarks you can find examples of algorithms where it is slower, but they tend to be the exception. "substantially" slower I haven't found yet.
https://kostya.github.io/LangArena/#/top_tab
Once the initial compiler was there it became easier, as Fable is rather good of working within the context of what is already there.
I'm wondering what kind of architectural corners this forces the programmer into ... have you found any so far?
It certainly requires the programmer to rethink how they structure their data, as you have to think of where it is going to be stored, and how to store it in a way such that it will have similar lifetimes to what it is stored with, to get the most out of the language.
How are you deciding how many stacks to use?
The programmer choose what to store where, so you typically store things in arrays by lifetime. Goose gives you many more tools to do that, it can store entire variable sized trees of an enum type in a single array.
You try to avoid having to deallocate in mixed/random order, but if you absolutely must, you can turn arrays into pools, basically.
I'm a bit confused about how you could decide the number of stacks at compile time. I can see you could see how many arrays are lexically in scope. However if the code puts one dynamic array inside another dynamic array I can get to an unbounded number of total stacks without ever needing more than two in lexical scope at once. Do you also add stacks at runtime? Like a stack of stacks?
The spec on your GitHub says: "Two simultaneously-live resizable values never share a stack."
So I surely can't have a dynamic array of dynamic arrays share a single stack? That's an unbounded number of resizeable values sharing a stack?
You could make it work on a single stack but that would involve having non-contiguous memory for each inner array since you can add to any array in any order.
The spec implies that we need one stack per element of the outer dynamic array which would let you have contiguous memory for each inner array, but you wouldn't know how many stacks you needed at compile time.
With Goose, the core design favors speed over everything else. Though not only was I able to make that safe, it is fairly pleasant to use. But it remains a low level systems language that requires you to think hard about data organization.
Lobster is a waaay more high level language made to be pleasant to use first, fast second. Surprisingly though, it can get pretty close in speed to C++/Rust nowadays too, but still a lot slower than Goose.
How have you found AI at writing Goose or Lobster code, if you give them a skill or something, are they relatively competent at actually writing code in the language?
Both are really interesting projects, BTW - thank you! I've been meaning to try Lobster out but haven't found time, I think this might be the push I needed.
Self-hosting would be cool too, but for both Lobster and Goose I've tried to keep it simple for people to build the language in any environment, which is why its still all in C++.
I don't use any "skills". From what I can see, the #1 way an AI produces "good code" is to simply work in the context of existing good code. I don't need to hold it by a style guide, it seems to infer that from context just fine. At least Fable does, other models less so.
"Just Git Gud First" - Wouter van Oortmerssen
Do you know if Goose is the first time something like this has been done?
There is lots of prior art on the individual parts of the design, e.g. "regions" and various stack languages etc, see: https://github.com/aardappel/goose/blob/master/docs/design/p...
Part of this is that 64-bit address spaces make this design practical, and almost all languages in use today were designed with the limitations of 32-bit in mind. Once you can assign arrays huge amounts of address space, you don't have to realloc, which gets you stable pointers, and a lot of the other Goose features.
(And why isn't it Amiga E?)
As for Amiga E, I am very proud of having made it way back when obviously, but it doesn't stack up against my more modern languages (Goose and Lobster), so I think I'll leave it be :)
Still, for a large number of programs - like data-oriented applications - it’s probably a good fit. It might work ok with entity-component game systems as long as you reuse old memory…
The relative references and built in binary structure validation also enable direct memory mapping of data structures and immediate use of network data. "Parsing" file and network data is often 100x slower than it needs to be." - John Carmack
From what little I read in the literature I seem to recall that there were enough programs with quite a bit more memory usage that they decided this approach was a dead end.
Does anyone know anymore details?
Back then though, they couldn't lean on large 64-bit address spaces like Goose can. They implemented all of this with a chain of blocks of memory instead, and tracked most of it at runtime that Goose now does at compile time.
https://github.com/aardappel/goose/blob/master/docs/design/p...
I could follow the link and read the thread and I don't have an X account.
You can also check out the language's website by googling "goose programming language", which should lead you to: https://aardappel.github.io/goose/index.html
What X shows me: https://celsoazevedo.com/files/2026/x.webp
With a login/nitter: https://celsoazevedo.com/files/2026/nitter.webp