diff --git a/src/content/docs/blog/2020/06/30/first-post/index.md b/src/content/docs/blog/2020/06/30/first-post/index.md new file mode 100644 index 00000000..a96ee802 --- /dev/null +++ b/src/content/docs/blog/2020/06/30/first-post/index.md @@ -0,0 +1,28 @@ +--- +title: "First Post" +date: 2020-06-30T16:24:32+01:00 +draft: false +authors: [ikey] +tags: [news] +license: "CC-BY-ND-4.0" +copyright: "Copyright © 2020 aerynOS Developers" +--- + +In a nutshell, this post wasn't meant to be coming out for a few weeks. As it turns out, we had to get ahead of the cat coming out of the bag, and ensure clarity of goals and ambition. + +We've got an awful lot of work ahead of us across the next few months and weeks, and we strongly suggest checking out the [About](/about) page which should answer most questions. + +TL;DR This is not your typical user-facing Linux distribution. + +So go ahead, check out the [About](/about) page. + +Posts will follow in the coming weeks, introducing the core team. It should be pointed out, internally, we're in a stage1 bootstrap phase, with stage2 ready. Tooling is next on the agenda. + +If you're looking for a modern, lightweight, user-friendly, privacy-focused Linux desktop distribution, then you're in the wrong place. + +Otherwise, stay tuned for the updates. We know our website is insanely wbasic, but it will be improved when it becomes a priority. You're more than welcome to join us on `#serpentOS` on freenode[^1] to become part of the community discussion in the meantime. + +Til next time. + +--- +[^1]: aerynOS now uses a [Zulip server](https://aerynos.zulipchat.com/) for community discussion. diff --git a/src/content/docs/blog/2020/06/30/first-post/index.mdx b/src/content/docs/blog/2020/06/30/first-post/index.mdx deleted file mode 100644 index 39c3cfac..00000000 --- a/src/content/docs/blog/2020/06/30/first-post/index.mdx +++ /dev/null @@ -1,37 +0,0 @@ ---- -title: "First Post" -date: 2020-06-30T16:24:32+01:00 -draft: false -authors: [ikey] -tags: [news] -license: "CC-BY-ND-4.0" -copyright: "Copyright © 2020 aerynOS Developers" ---- - -In a nutshell, this post wasn't meant to be coming out for a few weeks. -As it turns out, we had to get ahead of the cat coming out of the bag, -and ensure clarity of goals and ambition. - -We've got an awful lot of work ahead of us across the next few months and -weeks, and we strongly suggest checking out the [About](/about) page which -should answer most questions. - -TL;DR This is not your typical user-facing Linux distribution. - -{/*more*/} - -So go ahead, check out the [About](/about) page. - -Posts will follow in the coming weeks, introducing the core team. -It should be pointed out, internally, we're in a stage1 bootstrap phase, -with stage2 ready. Tooling is next on the agenda. - -If you're looking for a modern, lightweight, user-friendly, privacy-focused -Linux desktop distribution, then you're in the wrong place. - -Otherwise, stay tuned for the updates. We know our website is insanely -basic, but it will be improved when it becomes a priority. You're more -than welcome to join us on `#serpentOS` on freenode to become part of the -community discussion in the mean time. - -Til next time. diff --git a/src/content/docs/blog/2020/07/01/the-great-experiment/index.md b/src/content/docs/blog/2020/07/01/the-great-experiment/index.md index bb26ea6e..ec4b7d15 100644 --- a/src/content/docs/blog/2020/07/01/the-great-experiment/index.md +++ b/src/content/docs/blog/2020/07/01/the-great-experiment/index.md @@ -8,34 +8,25 @@ license: "CC-BY-ND-4.0" copyright: "Copyright © 2020 aerynOS Developers" --- -So as many have come to realise, we had to rush out a website super quick yesterday as the cat -was already out of the bag. One thing that should also be clarified, is our approach to development. - - +So as many have come to realise, we had to rush out a website super quick yesterday as the cat was already out of the bag. One thing that should also be clarified, is our approach to development. ## We're A Lab -Primarily (but not exclusively) Serpent OS (future: Serpent Linux) is an endeavour worked on by the -Lispy Snake, Ltd crew. So we're all about creating awesome technology and -trying to further the playing field. +Primarily (but not exclusively) Serpent OS (future: Serpent Linux) is an endeavour worked on by the _Lispy Snake, Ltd_[^1] crew. So we're all about creating awesome technology and trying to further the playing field. -Serpent OS is not, however, **owned** by Lispy Snake. It is a contributor-based open source project, -with specific roles. Those will be clarified over the coming week, but there is a dedicated **Project Manager**. -Hint: It's not me. +Serpent OS is not, however, **owned** by Lispy Snake. It is a contributor-based open source project, with specific roles. Those will be clarified over the coming week, but there is a dedicated **Project Manager**. Hint: It's not me. ## Measuring Success -So, how does an experimental project.. measure success? Quite simply - if others adopt our methodologies or -technology, we've succeeded. We're really **not** looking at "number of installs" or "size of userbase" as -a metric, as our chief aim is to _build technology_. +So, how does an experimental project... measure success? Quite simply - if others adopt our methodologies or technology, we've succeeded. We're really **not** looking at "number of installs" or "size of userbase" as a metric, as our chief aim is to _build technology_. ## Options for Longevity -If the experiment is a success, which of course means having tight controls on scope and timescale, then -one would assume the primary way to **use** Serpent OS would be through some downstream repackaging or -reconfiguration, i.e. basing upon the project, to meet specific user demand. +If the experiment is a success, which of course means having tight controls on scope and timescale, then one would assume the primary way to **use** Serpent OS would be through some downstream repackaging or reconfiguration, i.e. basing upon the project, to meet specific user demand. ## It Should Be Fun -This is the heart and soul of all Linux development. Developers should _enjoy_ the involvement and seeing -the end result. If we build something more in the end, then that's a journey we can create together. +This is the heart and soul of all Linux development. Developers should _enjoy_ the involvement and seeing the end result. If we build something more in the end, then that's a journey we can create together. + +--- +[^1]: Since this blog post, the Lipsy Snake website no longer exists. diff --git a/src/content/docs/blog/2020/07/06/welcome-peter/index.md b/src/content/docs/blog/2020/07/06/welcome-peter/index.md index 772b054a..92b18409 100644 --- a/src/content/docs/blog/2020/07/06/welcome-peter/index.md +++ b/src/content/docs/blog/2020/07/06/welcome-peter/index.md @@ -8,20 +8,16 @@ license: "CC-BY-ND-4.0" copyright: "Copyright © 2020 aerynOS Developers" --- -Recently we did reveal on Twitter that long time friend and colleague-of-many-projects, Peter O'Connor, has formally joined the core team for -Serpent OS. Many of you know him as `sunnyflunk`. He will be working on core toolchain optimisations, benchmarking, and eventually Plasma inclusion. +Recently we did reveal on Twitter that long time friend and colleague-of-many-projects, Peter O'Connor, has formally joined the core team for Serpent OS. Many of you know him as `sunnyflunk`. He will be working on core toolchain optimisations, benchmarking, and eventually Plasma inclusion. - - -Peter has long wanted to get the most out of his computing experience, starting from Gentoo back in the 2000's to ensure there was no wasted resources on -features he didn't use. The desire to over-tinker and use unstable software often led to poor outcomes. While the itch laid dormant for over a decade, -it has always been there. +Peter has long wanted to get the most out of his computing experience, starting from Gentoo back in the 2000's to ensure there was no wasted resources on features he didn't use. The desire to over-tinker and use unstable software often led to poor outcomes. While the itch laid dormant for over a decade, it has always been there. **Peter O'Connor (sunnyflunk)** > There's not much about Linux that currently excites me, but I've always wanted to see what's possible with a full LLVM/Clang stack. There's never been a better time to find out, but it can be difficult to push the envelope when limited with backwards compatibility and supporting older CPU architectures. After experimenting with some tool-chain customisations the chance to test them out more broadly and with a new tool-chain was too hard to pass up. This is the research project I've always wanted to undertake and looking forward to using it as my daily driver -We'd like to officially extend our welcome to Peter, as the ramp up gets in place. It should be noted, he is partly to blame for Serpent OS -becoming manifest, as the core team got together after discussions on IRC. He has some fantastic ideas, and is a brilliant distribution -engineer with proven success. +We'd like to officially extend our welcome to Peter, as the ramp up gets in place. It should be noted, he is partly to blame for Serpent OS becoming manifest, as the core team got together after discussions on IRC[^1]. He has some fantastic ideas, and is a brilliant distribution engineer with proven success. As an aside, we're working on improving the website to incorporate a team page, ready for the third team member reveal. + +--- +[^1]: aerynOS now uses a [Zulip server](https://aerynos.zulipchat.com/) for community discussion. diff --git a/src/content/docs/blog/2020/07/07/welcome-project-manager/index.md b/src/content/docs/blog/2020/07/07/welcome-project-manager/index.md new file mode 100644 index 00000000..f0ca961f --- /dev/null +++ b/src/content/docs/blog/2020/07/07/welcome-project-manager/index.md @@ -0,0 +1,18 @@ +--- +title: "Welcome: Project Manager" +date: 2020-07-07T11:35:50+01:00 +draft: false +authors: [ikey] +tags: [news] +license: "CC-BY-ND-4.0" +copyright: "Copyright © 2020 aerynOS Developers" +--- + +The Serpent OS team wishes to formally welcome Aydemir Ulaş Şahin to the core team in the primary capacity of Project Manager. Aydemir will help us to coordinate and manage the project goals and infrastructure, being responsible for certain "keys to the kingdom". + +Aydemir has extensive experience with Linux, both as a user and developer, having worked in the media industry for many years. Additionally he will be contributing to the core system, code and packaging. + +Our initial trio has been announced, and you can always access information on the current team at the new [About page](/about). Now that much of our groundwork has been laid, we can get on with building the project. Check progress at the new Roadmap[^1] page. + +--- +[^1]: Since this blog post, we have removed our Roadmap page. diff --git a/src/content/docs/blog/2020/07/07/welcome-project-manager/index.mdx b/src/content/docs/blog/2020/07/07/welcome-project-manager/index.mdx deleted file mode 100644 index ddd23d7a..00000000 --- a/src/content/docs/blog/2020/07/07/welcome-project-manager/index.mdx +++ /dev/null @@ -1,23 +0,0 @@ ---- -title: "Welcome: Project Manager" -date: 2020-07-07T11:35:50+01:00 -draft: false -authors: [ikey] -tags: [news] -license: "CC-BY-ND-4.0" -copyright: "Copyright © 2020 aerynOS Developers" ---- - -The Serpent OS team wishes to formally welcome Aydemir Ulaş Şahin to the core team in the primary -capacity of Project Manager. Aydemir will help us to coordinate and manage the project goals and -infrastructure, being responsible for certain "keys to the kingdom". - - - -Aydemir has extensive experience with Linux, both as a user and developer, having worked in the -media industry for many years. Additionally he will be contributing to the core system, code -and packaging. - -Our initial trio has been announced, and you can always access information on the current team -at the new [Team page](/team). Now that much of our groundwork has been laid, we can get on -with building the project. Check progress at the new [Roadmap](/roadmap) page. diff --git a/src/content/docs/blog/2020/08/08/stage1-complete/index.md b/src/content/docs/blog/2020/08/08/stage1-complete/index.md index 4c625027..0e809de6 100644 --- a/src/content/docs/blog/2020/08/08/stage1-complete/index.md +++ b/src/content/docs/blog/2020/08/08/stage1-complete/index.md @@ -8,21 +8,18 @@ license: "CC-BY-ND-4.0" copyright: "Copyright © 2020 aerynOS Developers" --- -Short and sweet, stage1 of the bootstrap is complete. As I indicated on the [Lispy Snake](https://lispysnake.com/blog/2020/08/03/status-update/) blog, -I'm still in the process of settling into new accommodation. This is going well, but still awaiting proper -broadband connectivity. Work has begun, however, and we're now moving onto stage2 of the bootstrap. +Short and sweet, stage1 of the bootstrap is complete. As I indicated on the _Lispy Snake_[^1] blog, I'm still in the process of settling into new accommodation. This is going well, but still awaiting proper broadband connectivity. Work has begun, however, and we're now moving onto stage2 of the bootstrap. - +This is handled via our [bootstrap-scripts](https://github.com/serpent-os/)[^2] project and can be run by anyone on a relatively modern Linux distribution. -This is handled via our [bootstrap-scripts](https://github.com/serpent-linux/bootstrap-scripts) project -and can be run by anyone on a relatively modern Linux distribution. +![Validating stage1 cross-compiler](./progress.webp "Validating stage1 cross-compiler") -![Validating stage1 cross-compiler](./progress.webp) +Next on the list is completing stage2, which we've already started on. This is simply a cross-compiled chroot environment with all the basic bits in place to build stage3, sanitizing and cleansing the toolchain. -Next on the list is completing stage2, which we've already started on. This is simply a cross-compiled -chroot environment with all the basic bits in place to build stage3, sanitizing and cleansing the toolchain. - -Even though it is early days, we can already, automatically produce a working cross-compiler that targets -LLVM's `libc++`, the musl `libc.so`, and `x86_64-serpent-linux-musl` host triplet. +Even though it is early days, we can already, automatically produce a working cross-compiler that targets LLVM's `libc++`, the musl `libc.so`, and `x86_64-serpent-linux-musl` host triplet. Probably a dull update for most, but an update it is. + +--- +[^1]: Since this blog post, the Lipsy Snake website no longer exists. +[^2]: Since this blog post, the bootstrap-scripts project no longer exists. diff --git a/src/content/docs/blog/2020/08/16/stage2-complete/index.md b/src/content/docs/blog/2020/08/16/stage2-complete/index.md index 79751376..106df98b 100644 --- a/src/content/docs/blog/2020/08/16/stage2-complete/index.md +++ b/src/content/docs/blog/2020/08/16/stage2-complete/index.md @@ -8,24 +8,15 @@ license: "CC-BY-ND-4.0" copyright: "Copyright © 2020 aerynOS Developers" --- -Just in case you thought we were sleeping behind the wheel, we've got -another blogpost for your viewing pleasure. In a nutshell, we completed -stage2 bootstrap. +Just in case you thought we were sleeping behind the wheel, we've got another blogpost for your viewing pleasure. In a nutshell, we completed stage2 bootstrap. - +![Complete build-target for ARMv8](./ARMv8.webp "Complete build-target for ARMv8") -![Complete build-target for ARMv8](./ARMv8.webp) +In order to simplify life, we greatly reduced the size of the `stage2` build component. This decision was taken to better support cross-compilation in the face of software that is distinctly cross-compilation unfriendly. -In order to simplify life, we greatly reduced the size of the `stage2` build component. -This decision was taken to better support cross-compilation in the face of software that -is distinctly cross-compilation unfriendly. +A support stage, `stage2.5` will be added which will `chroot` into a copy of `stage2`, and natively compile a small handful of packages required to complete `stage3`, also within the chroot environment. -A support stage, `stage2.5` will be added which will `chroot` into a copy of `stage2`, and -natively compile a small handful of packages required to complete `stage3`, also within the -chroot environment. - -For cross-compilation, we'll be relying on `qemu-static` to complete `2.5` and `3`. -However, at this point in time, we have the following: +For cross-compilation, we'll be relying on `qemu-static` to complete `2.5` and `3`. However, at this point in time, we have the following: - Working cross-compilation of the entire bootstrap - Complete LLVM based toolchain: `clang`, `llvm`, `libc++`, `lib++abi`, `libunwind` @@ -38,37 +29,23 @@ This is a major milestone for the project, as it is an early indication that we' ## Multiple Architecture Support -At this point in time, we now have build support for two targets: `x86_64` and `ARMV8a`. -Our intent is to support `haswell` and above, or `zen` and above, for the x86_64 target. +At this point in time, we now have build support for two targets: `x86_64` and `ARMV8a`. Our intent is to support `haswell` and above, or `zen` and above, for the x86_64 target. -With our `ARMv8` target, we're currently looking to support the Pine Book Pro, if we can -manage to get hold of some testing hardware. It will likely be some time after full -x86_64 support that we'd officially support more hardware, however it is very important -that our bootstrap-scripts can trivially target multiple platforms. +With our `ARMv8` target, we're currently looking to support the Pine Book Pro, if we can manage to get hold of some testing hardware. It will likely be some time after full x86_64 support that we'd officially support more hardware, however it is very important that our bootstrap-scripts can trivially target multiple platforms. ![ARMv8](./ARMv8.webp "ARMv8 validation") -An interesting change when cross-compiling for other architectures, is the chicken & egg -situation with `compiler-rt` and other LLVM libraries. When we detect cross-compilation, -we'll automatically bootstrap `compiler-rt` before building `musl`, and then cross-compile -`libc++`, `libc++abi` and `libunwind` to ensure `stage1` can produce native binaries for -the target with correct linkage. +An interesting change when cross-compiling for other architectures, is the chicken & egg situation with `compiler-rt` and other LLVM libraries. When we detect cross-compilation, we'll automatically bootstrap `compiler-rt` before building `musl`, and then cross-compile `libc++`, `libc++abi` and `libunwind` to ensure `stage1` can produce native binaries for the target with correct linkage. ## Next Steps -As we've mentioned, we'll push ahead with `2.5` and `3`, which will complete the initial -Serpent OS bootstrap, producing a self-hosting, self-reliant rootfs. This is the point -at which we can begin to bolt-on package management, boot management, stateless configuration -utilities, etc. +As we've mentioned, we'll push ahead with `2.5` and `3`, which will complete the initial Serpent OS bootstrap, producing a self-hosting, self-reliant rootfs. This is the point at which we can begin to bolt-on package management, boot management, stateless configuration utilities, etc. + +Our initial focus is `x86_64` hardware with UEFI, and as we gain access to more hardware we can enable support for more targets, such as `ARMv8a`. Our `bootstrap-scripts` will always remain open source, as will all processes and tooling within Serpent OS, or anything used to build and deploy Serpent OS. -Our initial focus is `x86_64` hardware with UEFI, and as we gain access to more hardware we -can enable support for more targets, such as `ARMv8a`. Our `bootstrap-scripts` will always -remain open source, as will all processes and tooling within Serpent OS, or anything used -to build and deploy Serpent OS. +This will make it much easier in future to create custom spins of Serpent OS for different configurations or targets, without derailing the core project. It should therefore be the simplest thing in the world to fork Serpent OS to one's liking or needs. -This will make it much easier in future to create custom spins of Serpent OS for different -configurations or targets, without derailing the core project. It should therefore be the -simplest thing in the world to fork Serpent OS to one's liking or needs. +If you want to support our work, you can jump onto our IRC channel (`#serpentOS` on freenode)[^1] or support us via the [Funding page](/funding). -If you want to support our work, you can jump onto our IRC channel (`#serpentOS` on freenode) -or support us via the [Team page](/team). +--- +[^1]: aerynOS now uses a [Zulip server](https://aerynos.zulipchat.com/) for community discussion. diff --git a/src/content/docs/blog/2020/08/20/stage3-progress/index.md b/src/content/docs/blog/2020/08/20/stage3-progress/index.md new file mode 100644 index 00000000..67b8892e --- /dev/null +++ b/src/content/docs/blog/2020/08/20/stage3-progress/index.md @@ -0,0 +1,68 @@ +--- +title: "Stage 3 Progress" +date: 2020-08-20T14:14:49+01:00 +draft: false +authors: [ikey] +tags: [news] +license: "CC-BY-ND-4.0" +copyright: "Copyright © 2020 aerynOS Developers" +--- + +Well, it's been a few days since we last spoke, so now it's time for a quick roundup. Long story short, we're approaching the end of the `stage3` bootstrap. + +![Fully functional chroot](./screenshot.webp "Fully functional chroot") + +In an effort to simplify our bootstrap process, we dropped the newly-introduced `stage2.5` and came up with a new strategy for `stage3`. In order to make it all work nicely, we bind-mount the `stage2` resulting runtime at `/serpent` within the `stage3` chroot environment, executing the `/serpent/usr/bin/bash` shell. + +In order to make this work, we build an intermediate `musl` package for `libc.so` in the very start of `stage3`, with all subsequent builds being performed in chroots. Part of the build is done on the host, i.e. extraction and patching, minimising the tool requirements for the chroot environment. The configuration, build and install is performed from within the initially empty chroot environment, replacing all the `/serpent/usr/bin` tools and `/serpent/usr/lib` libraries. + +## Mild Blockers + +As we move further through `stage3`, towards a fully usable chroot environment, we've encountered a small number of blockers. Now, we **could** solve them by using existing patchwork and workarounds, but most have not and will not be accepted upstream. Additionally it is incredibly hard to track the origin and history of most of these, making security rather more painful. + +## libc-support + +We're going to start working on a project to flesh out the `musl` runtime with some missing utilities, written with a clean-room approach. These will initially include the `getconf` and `getent` tools, which will be written only with Linux in mind, and no legacy/BSD support. + +These will be maintained over at our [GitHub](https://github.com/serpent-os/libc-support) + +## libwildebeest + +As a project we strive for correctness in the most pragmatic way. Some software, such as systemd, is heavily reliant on GNU GCC/GLibc extensions. In some software there are feasible alternatives when using `musl`, however in a good number of cases, functionality required to build certain +software is missing and has no alternative. + +Over time we'll try to work with upstreams to resolve those issues, but we're working on an interim solution called `libwildebeest`. This will provide **source compatibility** for a limited number of software packages relying on so-called `GNUisms`. Binary compatibility is not an aim whatsoever, and will not be implemented. This convenience library will centralise all patchwork on packages that need more work to integrate with `musl`, until such time as upstreams have resolved the remaining issues. + +Additionally it will help us track those packages needing attention in the distribution, as they will have a build-time dependency on `libwildebeest`. We do not intend to use this project extensively. + +This will be maintained over at our [GitHub](https://github.com/ikeycode/libwildebeest) + +## A Word On Init Systems + +Recently we've had many queries regarding the init system, as there is an expectation that due to our use of musl/llvm we also dislike systemd or wish to be a small OS, etc. There is a place in the world for those projects already, and we wish them much success. However from our own stance and goals, `systemd` has already "won the battle" and actually fits in with our design. + +**If** it is possible in future with package manager considerations and packaging design, then we may make it possible to swap `systemd` for a similar set of packages. However, we only intend at this time to support `systemd/udev/dbus` directly in Serpent OS and leave alternatives to the community. + +## Other News + +Just a quick heads up, we've been talking to the cool folks over at fosshost.org[^1] and they've agreed to kindly provide us with additional hosting and mirroring. This will allow us to build scale in from the very start, ensuring updates and images are always available. Once the new server is up and running we'll push another blogpost with the details and links. + +While initially we intended to avoid public bug trackers, the rate of growth within the project and community have made it highly apparent that proper communication channels need establishing. Therefore we will be setting up a public Phabricator instance for reporting issues, security flaws, and contributing packaging. + +Much of our website is in much need of update, but our current priority is with building the OS. Please be patient with us, we'll have it all sorted out in no time. + +## Where We At? + +Well, `stage3` completes fully, builds the final compiler, which has also been verified. A usable chroot system is produced, built using `musl`, `libc++`, `libunwind`, `clang`, etc. Some might say that `stage3` is complete, however we wish to avoid circular dependency situations. We'll mark `stage3` as complete once we've integrated an initial slimmed down build of `systemd` and appropriately relinked the build. + +As soon as this stage is done, we'll proceed with `stage4`. This is the final stage where we'll add package management and define the OS itself, with global flags, policies, etc. + +With the speed we're moving at, that really isn't too far away. + +## And finally.. + +I personally wish to thank the Serpent OS team as a whole for the commitment and work undertaken of late. Additionally I want to thank the growing community around Serpent OS, primarily residing in our IRC channel (`#serpentOS` on freenode)[^2] and our Twitter account. Your input has been amazing, and it's so refreshing to have so many people on the same page. Stay awesome. + +--- +[^1]: Since this blog post, fosshost no longer exists +[^2]: AerynOS now uses a [Zulip server](https://aerynos.zulipchat.com/) for community discussion. diff --git a/src/content/docs/blog/2020/08/20/stage3-progress/index.mdx b/src/content/docs/blog/2020/08/20/stage3-progress/index.mdx deleted file mode 100644 index b038caa3..00000000 --- a/src/content/docs/blog/2020/08/20/stage3-progress/index.mdx +++ /dev/null @@ -1,106 +0,0 @@ ---- -title: "Stage 3 Progress" -date: 2020-08-20T14:14:49+01:00 -draft: false -authors: [ikey] -tags: [news] -license: "CC-BY-ND-4.0" -copyright: "Copyright © 2020 aerynOS Developers" ---- - -Well, it's been a few days since we last spoke, so now it's time for a quick -roundup. Long story short, we're approaching the end of the stage3 bootstrap. - - - -![Fully functional chroot](./screenshot.webp) - - -In an effort to simplify our bootstrap process, we dropped the newly-introduced `stage2.5` and -came up with a new strategy for `stage3`. In order to make it all work nicely, we bind-mount the -`stage2` resulting runtime at `/serpent` within the `stage3` chroot environment, executing the -`/serpent/usr/bin/bash` shell. - -In order to make this work, we build an intermediate `musl` package for `libc.so` in the very -start of `stage3`, with all subsequent builds being performed in chroots. Part of the build -is done on the host, i.e. extraction and patching, minimising the tool requirements for the chroot -environment. The configuration, build and install is performed from within the initially empty -chroot environment, replacing all the `/serpent/usr/bin` tools and `/serpent/usr/lib` libraries. - -## Mild Blockers - -As we move further through stage3, towards a fully usable chroot environment, we've encountered -a small number of blockers. Now, we **could** solve them by using existing patchwork and workarounds, -but most have not and will not be accepted upstream. Additionally it is incredibly hard to track the -origin and history of most of these, making security rather more painful. - -## libc-support - -We're going to start working on a project to flesh out the `musl` runtime with some missing utilities, -written with a clean-room approach. These will initially include the `getconf` and `getent` tools, -which will be written only with Linux in mind, and no legacy/BSD support. - -These will be maintained over at our [GitHub](https://github.com/serpent-linux/libc-support) - -## libwildebeest - -As a project we strive for correctness in the most pragmatic way. Some software, such as systemd, -is heavily reliant on GNU GCC/GLibc extensions. In some software there are feasible alternatives -when using musl, however in a good number of cases, functionality required to build certain -software is missing and has no alternative. - -Over time we'll try to work with upstreams to resolve those issues, but we're working on an interim -solution called 'libwildebeest'. This will provide **source compatibility** for a limited number -of software packages relying on so-called 'GNUisms'. Binary compatibility is not an aim whatsoever, -and will not be implemented. This convenience library will centralise all patchwork on packages -that need more work to integrate with musl, until such time as upstreams have resolved the remaining -issues. - -Additionally it will help us track those packages needing attention in the distribution, as they -will have a build-time dependency on `libwildebeest`. We do not intend to use this project extensively. - -This will be maintained over at our [GitHub](https://github.com/serpent-linux/libwildebeest) - -## A Word On Init Systems - -Recently we've had many queries regarding the init system, as there is an expectation that due to our -use of musl/llvm we also dislike systemd or wish to be a small OS, etc. There is a place in the world -for those projects already, and we wish them much success. However from our own stance and goals, -`systemd` has already "won the battle" and actually fits in with our design. - -**If** it is possible in future with package manager considerations and packaging design, then we may -make it possible to swap `systemd` for a similar set of packages. However, we only intend at this time -to support `systemd/udev/dbus` directly in Serpent OS and leave alternatives to the community. - -## Other News - -Just a quick heads up, we've been talking to the cool folks over at [fosshost.org](https://fosshost.org/) -and they've agreed to kindly provide us with additional hosting and mirroring. This will allow us to -build scale in from the very start, ensuring updates and images are always available. Once the new server -is up and running we'll push another blogpost with the details and links. - -While initially we intended to avoid public bug trackers, the rate of growth within the project and community -have made it highly apparent that proper communication channels need establishing. Therefore we will be -setting up a public Phabricator instance for reporting issues, security flaws, and contributing packaging. - -Much of our website is in much need of update, but our current priority is with building the OS. Please -be patient with us, we'll have it all sorted out in no time. - -## Where We At? - -Well, stage3 completes fully, builds the final compiler, which has also been verified. A usable chroot -system is produced, built using `musl`, `libc++`, `libunwind`, `clang`, etc. Some might say that stage3 -is complete, however we wish to avoid circular dependency situations. We'll mark stage3 as complete once -we've integrated an initial slimmed down build of `systemd` and appropriately relinked the build. - -As soon as this stage is done, we'll proceed with `stage4`. This is the final stage where we'll add -package management and define the OS itself, with global flags, policies, etc. - -With the speed we're moving at, that really isn't too far away. - -## And finally.. - -I personally wish to thank the Serpent OS team as a whole for the commitment and work undertaken of late. -Additionally I want to thank the growing community around Serpent OS, primarily residing in our IRC -channel (`#serpentOS` on freenode) and our Twitter account. Your input has been amazing, and it's -so refreshing to have so many people on the same page. Stay awesome. diff --git a/src/content/docs/blog/2020/08/28/looking-stage4-in-the-eye/index.md b/src/content/docs/blog/2020/08/28/looking-stage4-in-the-eye/index.md index 77454686..d42a33b2 100644 --- a/src/content/docs/blog/2020/08/28/looking-stage4-in-the-eye/index.md +++ b/src/content/docs/blog/2020/08/28/looking-stage4-in-the-eye/index.md @@ -8,49 +8,29 @@ license: "CC-BY-ND-4.0" copyright: "Copyright © 2020 aerynOS Developers" --- -Well, we've made an awful lot of progress in these last few days. It wasn't -that long ago that we introduced some of the new projects required to get -stage3 off the ground. +Well, we've made an awful lot of progress in these last few days. It wasn't that long ago that we introduced some of the new projects required to get `stage3` off the ground. - - -![Building systemd the easy way](./FeaturedSD.webp) +![Building systemd the easy way](./FeaturedSD.webp "Building systemd the easy way") ## libc-support -Our [libc-support](https://dev.serpentos.com/source/libc-support/) project has been growing, thanks primarily to the contributions -of [Jouni Roivas](https://dev.serpentos.com/p/joroi/). We now have initial working versions -of `getent` and `getconf`. The `getconf` program is considered feature-complete -for our current requirements, and the focus is now on cleaning up `getent` making -it more modular and easy to maintain in the long run. +Our [libc-support](https://github.com/ikeycode/libc-support) project has been growing, thanks primarily to the contributions of [Jouni Roivas](https://github.com/jroivas). We now have initial working versions of `getent` and `getconf`. The `getconf` program is considered feature-complete for our current requirements, and the focus is now on cleaning up `getent` making it more modular and easy to maintain in the long run. ## libwildebeest -We began work on [libwildebeest](https://dev.serpentos.com/source/libwildebeest/) to quickly unlock building `systemd`. -Remember, our ambition with this project is to provide a sane, centralised way of -maintaining source compatibility with various projects that rely on features currently -only available in the GNU toolchain + runtime. This is our alternative vision to -patching every single package that fails to build against our LLVM+musl toolchain, -ensuring our work scales. +We began work on [libwildebeest](https://github.com/ikeycode/libwildebeest) to quickly unlock building `systemd`. Remember, our ambition with this project is to provide a sane, centralised way of maintaining source compatibility with various projects that rely on features currently only available in the GNU toolchain + runtime. This is our alternative vision to patching every single package that fails to build against our LLVM+musl toolchain, ensuring our work scales. -Right now, `libwildebeest` implements some missing APIs, and indeed, some **replacement** APIs, -for when GNU behaviours are expected. It does so in a way that doesn't impact the -resulting binary's license, or any of the system ABI. We provide some `pkg-config` files -with explicit `cflags` and `libs` fields set, such as: +Right now, `libwildebeest` implements some missing APIs, and indeed, some **replacement** APIs, for when GNU behaviours are expected. It does so in a way that doesn't impact the resulting binary's license, or any of the system ABI. We provide some `pkg-config` files with explicit `cflags` and `libs` fields set, such as: ```bash - -L${libdir} -lwildebeest-ftw -Wl,--wrap=ftw -Wl,--wrap=nftw -I${includedir}/libwildebeest --include=lwb_ftw.h ``` -Our headers take special care to **mask** the headers provided by musl to avoid redefinitions, -and instruct the linker to replace calls to these functions with our own versions, i.e.: +Our headers take special care to **mask** the headers provided by musl to avoid redefinitions, and instruct the linker to replace calls to these functions with our own versions, i.e.: ```c - int __wrap_ftw(const char *dir, int (*funcc)(const char *, const struct stat *, int), int descriptors) - ``` It then becomes trivial to enable wildebeest for a package build, i.e.: @@ -59,52 +39,36 @@ It then becomes trivial to enable wildebeest for a package build, i.e.: export CFLAGS="${CFLAGS} $(pkg-config --cflags --libs libwildebeest)" ``` -Right now - we've only provided stubs in `libwildebeest` to ensure we can build our packages. -Our next focus is to actually implement those stubs using MIT licensed code so that applications -and libraries can rely on `libwildebeest` to provide a basic level of GNU compatibility in a -reliable fashion. +Right now - we've only provided stubs in `libwildebeest` to ensure we can build our packages. Our next focus is to actually implement those stubs using MIT licensed code so that applications and libraries can rely on `libwildebeest` to provide a basic level of GNU compatibility in a reliable fashion. -Until such point as all the APIs are fully and safely implemented, it would be highly ill-advised -to **use** libwildebeest in any project. We'll announce stability in the coming weeks. +Until such point as all the APIs are fully and safely implemented, it would be highly ill-advised to **use** libwildebeest in any project. We'll announce stability in the coming weeks. ## systemd -We've made great progress in enabling systemd in Serpent OS. Where `libwildebeest` is in -place, it now enables our currently required level of source compatibility to a point where -systemd is building with networkd, resolved and a number of other significant targets enabled. +We've made great progress in enabling systemd in Serpent OS. Where `libwildebeest` is in place, it now enables our currently required level of source compatibility to a point where systemd is building with networkd, resolved and a number of other significant targets enabled. -In a small number of cases, we've had to patch systemd, but not in the traditional sense -expected to make it work with musl. +In a small number of cases, we've had to patch systemd, but not in the traditional sense expected to make it work with musl. -The only non-upstreamable patching we've done (in combination with libwildebeest enabling) -was to the UAPI headers, as the musl provided headers clash with the upstream kernel headers -in certain places (`if_ether.h`, `if_arp.h`) - but this is a tiny cost to bear. +The only non-upstreamable patching we've done (in combination with libwildebeest enabling) was to the UAPI headers, as the musl provided headers clash with the upstream kernel headers in certain places (`if_ether.h`, `if_arp.h`) - but this is a tiny cost to bear. The other patches, were simply portability fixes, ensuring all headers were included: - [partition/makefs: Include missing sys/file.h header #16876](https://github.com/systemd/systemd/pull/16876) - [login/logind: Include sys/stat.h for struct stat usage #16887](https://github.com/systemd/systemd/pull/16887) -It should be noted both of these (very trivial) pull requests were accepted and merged upstream, -and will be part of the next systemd release. +It should be noted both of these (very trivial) pull requests were accepted and merged upstream, and will be part of the next systemd release. ## Next On The Agenda -Our major ticket items involve fleshing out stage3 with some missing libraries to further -enable systemd, rebuilds of util-linux to be systemd-aware, and continue fleshing out -`dbus`, `systemd` and `libwildebeest` support to the point we have a bootable disk image. +Our major ticket items involve fleshing out stage3 with some missing libraries to further enable systemd, rebuilds of util-linux to be systemd-aware, and continue fleshing out `dbus`, `systemd` and `libwildebeest` support to the point we have a bootable disk image. + +At that point we'll move into stage4 with package management, boot management, and a whole host of other goodies. And, if there is enough interest, perhaps some early access ISOs! -At that point we'll move into stage4 with package management, boot management, and -a whole host of other goodies. And, if there is enough interest, perhaps some early -access ISOs! +After having engaged in discussions with a variety of developers using musl as their primary libc, we've catalogued common pain points. We therefore encourage developers to contribute to our `libwildebeest` and `libc-support` projects to complete the tooling and experience around musl-based distributions. -After having engaged in discussions with a variety of developers using musl as their -primary libc, we've catalogued common pain points. We therefore encourage developers -to contribute to our `libwildebeest` and `libc-support` projects to complete the -tooling and experience around musl-based distributions. +Our aim for Serpent OS is a **full fat experience**, which means we have large ticket items on our horizon, such as NSS/IDN integration, performance improvements, increasing the default stack size, along with source compatibility for major upstream projects. -Our aim for Serpent OS is a **full fat experience**, which means we have large ticket -items on our horizon, such as NSS/IDN integration, performance improvements, increasing -the default stack size, along with source compatibility for major upstream projects. +Until the next blog post, you can keep up to date on our IRC channel. Join `#serpentOS` on freenode![^1] -Until the next blog post, you can keep up to date on our IRC channel. Join `#serpentOS` on freenode! +--- +[^1]: aerynOS now uses a [Zulip server](https://aerynos.zulipchat.com/) for community discussion. diff --git a/src/content/docs/blog/2020/09/07/stage3-complete/index.md b/src/content/docs/blog/2020/09/07/stage3-complete/index.md index 1588a84e..8c3329e1 100644 --- a/src/content/docs/blog/2020/09/07/stage3-complete/index.md +++ b/src/content/docs/blog/2020/09/07/stage3-complete/index.md @@ -8,58 +8,37 @@ license: "CC-BY-ND-4.0" copyright: "Copyright © 2020 aerynOS Developers" --- -Another week, another milestone completed. We're pleased to announce that -we've completed the initial Stage3 bootstrap. While we have some parallel -works to complete for Stage 4, we can now begin to work on the exciting -pieces! We're now bootstrapped on ARMv8(-a) and x86_64 - - +Another week, another milestone completed. We're pleased to announce that we've completed the initial `stage3` bootstrap. While we have some parallel works to complete for `stage4`, we can now begin to work on the exciting pieces! We're now bootstrapped on ARMv8(-a) and x86_64 ![Build](./Build.webp "Complete build-target for x86_64") ## Announcing Moss -Our immediate focus is now to implement our package manager: [moss](https://github.com/serpent-linux/moss). -Currently it only has a trivial CLI and does absolutely nothing. We will now shift our attention -to implement this as a core part of the Stage 4 bootstrap. Moss is so-called as we're a rolling -release, _a rolling stone gathers no moss_. The package manager is being implemented in the D Programming Language. +Our immediate focus is now to implement our package manager: [moss](https://github.com/AerynOS/os-tools/). Currently it only has a trivial CLI and does absolutely nothing. We will now shift our attention to implement this as a core part of the `stage4` bootstrap. Moss is so-called as we're a rolling release, _a rolling stone gathers no moss_. The package manager is being implemented in the D Programming Language[^1]:. -Moss does not aim to be a next generation package management solution, it instead inspired by -eopkg, RPM and swupd, aiming to provide a reliable modern package management and update -solution with stateless design at the core. Core features and automatic dependencies will be managed -through capability subscriptions. The binary format will be versioned and deduplicated, with multiple -internal lookup tables and checksums. Our chief focus with moss is a reliable system -management tool that is accessible even in situations where internet access is limited. +Moss does not aim to be a next generation package management solution, it instead inspired by [eopkg](https://github.com/getsolus/eopkg), [RPM](https://github.com/rpm-software-management/rpm) and [swupd](https://github.com/clearlinux/swupd-server), aiming to provide a reliable modern package management and update solution with stateless design at the core. Core features and automatic dependencies will be managed through capability subscriptions. The binary format will be versioned and deduplicated, with multiple internal lookup tables and checksums. Our chief focus with moss is a reliable system management tool that is accessible even in situations where internet access is limited. ## Ongoing Works -In order to complete stage3, we provided linker stubs in `libwildebeest` to ensure -we can build. Obviously this introduces runtime errors in systemd and our stage3 isn't -bootable, just chrootable. This will be resolved when systemd is properly packaged by -moss in stage4. +In order to complete `stage3`, we provided linker stubs in `libwildebeest` to ensure we can build. Obviously this introduces runtime errors in systemd and our `stage3` isn't bootable, just chrootable. This will be resolved when systemd is properly packaged by moss in `stage4`. ## Docker Image -We now have a test container available on Docker Hub. You can install and run the -simple bash environment by executing the following command with a Docker-enabled user: +We now have a test container available on Docker Hub. You can install and run the simple bash environment by executing the following command with a Docker-enabled user: ```bash docker run -it --rm serpentos/staging:latest ``` -Currently we only have an x86_64 image, but may experiment with multiarch builds later -in stage4. +Currently we only have an x86_64 image, but may experiment with multiarch builds later in `stage4`. -**IMPORTANT**: The `staging` image is currently only a dump of the stage3 tree with -minor cleaning + tweakups. It is not, in any way, shape or form, representative of -the final quality of Serpent OS. Additionally zero performance work or security -patching has been done, so do **not** use in a production environment. +**IMPORTANT**: The `staging` image is currently only a dump of the `stage3` tree with minor cleaning + tweakups. It is not, in any way, shape or form, representative of the final quality of Serpent OS. Additionally zero performance work or security patching has been done, so do **not** use in a production environment. The image is provided currently as a means to validate the LLVM/musl toolchain. ## Future ISO -We cannot currently say when we'll **definitely** have an ISO, however we do -know that some VM-specific images will arrive first. After that we'll focus -on an installer (package selection based) and a default developer experience. -All we can say is strap in, and enjoy the ride. +We cannot currently say when we'll **definitely** have an ISO, however we do know that some VM-specific images will arrive first. After that we'll focus on an installer (package selection based) and a default developer experience. All we can say is strap in, and enjoy the ride. + +--- +[^1]: Since this blog post, `moss` and all other Serpent OS components have been re-written in Rust. diff --git a/src/content/docs/blog/2020/09/10/defining-moss/index.md b/src/content/docs/blog/2020/09/10/defining-moss/index.md index a680ac84..d208649e 100644 --- a/src/content/docs/blog/2020/09/10/defining-moss/index.md +++ b/src/content/docs/blog/2020/09/10/defining-moss/index.md @@ -9,98 +9,54 @@ license: "CC-BY-ND-4.0" copyright: "Copyright © 2020 aerynOS Developers" --- -Over the past few weeks, throughout the entire bootstrap process, we've -been deliberating on what our package manager is going to look like. We -now have a very solid idea on what that'll be, so it's time for a blog -post to share with the community. +Over the past few weeks, throughout the entire bootstrap process, we've been deliberating on what our package manager is going to look like. We now have a very solid idea on what that'll be, so it's time for a blog post to share with the community. - - -![Initial moss prototype CLI](./Featured.webp) +![Initial moss prototype CLI](./Featured.webp "Initial moss prototype CLI") ## Old, but new -The team has been very clear in wanting a traditional package management solution, -whereby repositories and packages compose the OS as a whole. However, we also -want modern features, such as being stateless by default. A common theme -is wanting to reduce the complex iterations of an OS into something that -is sanely versioned, but also flexible, to ensure a consistent, tested -experience with multiple end targets. +The team has been very clear in wanting a traditional package management solution, whereby repositories and packages compose the OS as a whole. However, we also want modern features, such as being stateless by default. A common theme is wanting to reduce the complex iterations of an OS into something that is sanely versioned, but also flexible, to ensure a consistent, tested experience with multiple end targets. -Additionally, the OS must be incredibly easy for contributors and team members -to maintain, with intelligent tooling and simple, but powerful formats. +Additionally, the OS must be incredibly easy for contributors and team members to maintain, with intelligent tooling and simple, but powerful formats. ## Atomic updates -One of the most recent software update trends of recent years is atomic updates. -In essence this allows applying an update in a single operation, and importantly, -reversing an update in a single operation, without impacting the running system. +One of the most recent software update trends of recent years is atomic updates. In essence this allows applying an update in a single operation, and importantly, reversing an update in a single operation, without impacting the running system. -This is typically achieved using a so-called `A/B switch`, which is what we will -also do with moss. We won't rely on any specific filesystem for this implementation, -instead relying on a smart layout, `pivot_root` and a few other tricks. +This is typically achieved using a so-called `A/B switch`, which is what we will also do with moss. We won't rely on any specific filesystem for this implementation, instead relying on a smart layout, `pivot_root` and a few other tricks. -Primarily we'll update a single `/usr` symlink to point to the current OS image, -with `/` being a read-only faux rootfs, populated with established mountpoints -and symlinks. Mutation will be possible only via `moss` transactions, or in -world-writable locations (`/opt`, `/usr/local`, `/etc` ...) +Primarily we'll update a single `/usr` symlink to point to the current OS image, with `/` being a read-only faux rootfs, populated with established mountpoints and symlinks. Mutation will be possible only via `moss` transactions, or in world-writable locations (`/opt`, `/usr/local`, `/etc` ...) ## Deduplication -The moss binary package format will be deduplicated in nature, containing hash-keyed -blobs in an `zstd` compressed payload. Unique blobs will be stored in a global cache, -and hard-linked into their final location (i.e. `/moss/root/$number/usr/...`) to -deduplicate the installed system too. This allows multiple system snapshots with -minimal overhead, and the ability to perform an offline rollback. +The moss binary package format will be deduplicated in nature, containing hash-keyed blobs in an `zstd` compressed payload. Unique blobs will be stored in a global cache, and hard-linked into their final location (i.e. `/moss/root/$number/usr/...`) to deduplicate the installed system too. This allows multiple system snapshots with minimal overhead, and the ability to perform an offline rollback. ## Implications -We'll need to lock kernels to their relevant transactions (or repo versions) to prevent -untested configurations. Additionally the boot menu will need to know about older versions -of the OS that are installed, so they can be activated in the `initrd` at boot. This -will require us doing some work with `clr-boot-manager` to achieve our goals. +We'll need to lock kernels to their relevant transactions (or repo versions) to prevent untested configurations. Additionally the boot menu will need to know about older versions of the OS that are installed, so they can be activated in the `initrd` at boot. This will require us doing some work with `clr-boot-manager` to achieve our goals. ## Difference from existing solutions -We're trying to minimise the iterations of the OS to what is available in a given version -of the package repositories. Additionally we wish to avoid extensive "symlink farms" -as we're not a profile-oriented distribution. Instead we focus on deduplication, atomic -updates and resolution performance. +We're trying to minimise the iterations of the OS to what is available in a given version of the package repositories. Additionally we wish to avoid extensive "symlink farms" as we're not a profile-oriented distribution. Instead, we focus on deduplication, atomic updates and resolution performance. ## Subscriptions -Keeping a system slim is often a very difficult thing to achieve, without extensive -package splitting and effort on the user's part. An example might be enabling SELinux -support on a system, or culling the locales to only include the used ones. +Keeping a system slim is often a very difficult thing to achieve, without extensive package splitting and effort on the user's part. An example might be enabling SELinux support on a system, or culling the locales to only include the used ones. -In Serpent OS (and moss, more specifically) we intend to address this through "subscriptions". -Well defined names will be used by moss to filter (or enable) certain configurations -in packages + dependencies. In some instances this will toggle subpackages/dependencies -and in others it will control which paths of a package are actually installed to disk. +In Serpent OS (and moss, more specifically) we intend to address this through "subscriptions". Well defined names will be used by moss to filter (or enable) certain configurations in packages + dependencies. In some instances this will toggle subpackages/dependencies and in others it will control which paths of a package are actually installed to disk. -Going further with this concept, we will eventually introduce modalias based capabilities -to automatically subscribe users to required kernel modules, allowing slim or fullfat -installations as the user chooses. This in turn takes the burden of maintenance away -from developers + users, and enables an incredibly flexible, approachable system. +Going further with this concept, we will eventually introduce modalias based capabilities to automatically subscribe users to required kernel modules, allowing slim or full fat installations as the user chooses. This in turn takes the burden of maintenance away from developers + users, and enables an incredibly flexible, approachable system. ## Mandatory reboots -Where possible we will limit mandatory reboots, preferring an in-place atomic update -to facilitate high uptime. However, there are situations where a reboot is absolutely -unavoidable, and the system administrator should plan some downtime to handle this -case. +Where possible we will limit mandatory reboots, preferring an in-place atomic update to facilitate high uptime. However, there are situations where a reboot is absolutely unavoidable, and the system administrator should plan some downtime to handle this case. -Certain situations like a kernel update, or security fix to a core library, would -require a reboot. In these instances, the atomic update will be deferred until the -next boot. In most situations, however, reboots will not be mandatory. +Certain situations like a kernel update, or security fix to a core library, would require a reboot. In these instances, the atomic update will be deferred until the next boot. In most situations, however, reboots will not be mandatory. ## Until the next time -Well, we've given a brief introduction to our aims with `moss` and associated tooling, -and you can get more information by checking the moss [README.md](https://github.com/serpent-linux/moss/blob/main/README.md). +Well, we've given a brief introduction to our aims with `moss` and associated tooling, and you can get more information by checking the moss [README.md](https://github.com/AerynOS/os-tools/blob/main/README.md). -The takeaway is we want a package-based software update mechanism that is -reliable and trusted, and custom-built to handle Serpent OS intricities, -with a simple approach to building and maintaining the distribution. +The takeaway is we want a package-based software update mechanism that is reliable and trusted, and custom-built to handle Serpent OS intricacies, with a simple approach to building and maintaining the distribution. For now, we're gonna stop talking, and start coding. diff --git a/src/content/docs/blog/2020/09/20/moss-format-defined/index.mdx b/src/content/docs/blog/2020/09/20/moss-format-defined/index.md similarity index 51% rename from src/content/docs/blog/2020/09/20/moss-format-defined/index.mdx rename to src/content/docs/blog/2020/09/20/moss-format-defined/index.md index b2423870..f427732c 100644 --- a/src/content/docs/blog/2020/09/20/moss-format-defined/index.mdx +++ b/src/content/docs/blog/2020/09/20/moss-format-defined/index.md @@ -8,13 +8,9 @@ license: "CC-BY-ND-4.0" copyright: "Copyright © 2020 aerynOS Developers" --- -The core team have been hard at work lately implementing the [Moss package manager](https://github.com/serpent-linux/moss). -We now have an initial version of the binary format that we're happy with, so we thought we'd share a progress update -with you. +The core team have been hard at work lately implementing the [Moss package manager](https://github.com/AerynOS/os-tools/). We now have an initial version of the binary format that we're happy with, so we thought we'd share a progress update with you. - - -![Development work on moss](./Featured.webp) +![Development work on moss](./Featured.webp "Development work on moss") ## Explaining the format @@ -25,9 +21,7 @@ Briefly, the binary container format consists of 4 payloads: - Index (indices to files within the binary blob) - Layout (How to apply the files to disk) -Each payload is verified internally using a CRC64-ISO, and contains basic information such as the length of the payload -both compressed and uncompressed, the compression algorithm used (`zstd` and `zlib` supported) as well as the type and -version of the payload. All multiple-byte values are stored in Big Endian order (i.e. Network Byte Order). +Each payload is verified internally using a CRC64-ISO, and contains basic information such as the length of the payload both compressed and uncompressed, the compression algorithm used (`zstd` and `zlib` supported) as well as the type and version of the payload. All multiple-byte values are stored in Big Endian order (i.e. Network Byte Order). ![Payloads](./Payloads.webp "All relevant payloads") @@ -43,18 +37,11 @@ Internally the representation of a Payload is defined as a 32-byte struct: PayloadCompression compression = PayloadCompression.Unknown; /* 1 byte */ ``` -We merge all unique files in a package rootfs into the `Content` payload, and compress that using zstd. The offsets to -each unique file (i.e. the sha256sum) are stored within the `Index` payload, allowing us to extract relevant portions -from the "megablob" using `copy_file_range()`. +We merge all unique files in a package rootfs into the `Content` payload, and compress that using zstd. The offsets to each unique file (i.e. the sha256sum) are stored within the `Index` payload, allowing us to extract relevant portions from the "megablob" using `copy_file_range()`. -These files will become part of the system hash store, allowing another level of deduplication between all system -packages. Finally, we use the `Layout` payload to **apply** the layout of the package into a transactional rootfs. -This will define paths, such as `/usr/bin/nano`, along with permissions, types, etc. All regular files will actually -be created as hard links from the hash store, allowing deduplication and snapshots. +These files will become part of the system hash store, allowing another level of deduplication between all system packages. Finally, we use the `Layout` payload to **apply** the layout of the package into a transactional rootfs. This will define paths, such as `/usr/bin/nano`, along with permissions, types, etc. All regular files will actually be created as hard links from the hash store, allowing deduplication and snapshots. -The `Meta` payload consists of a number of records, each with strongly defined types (such as `String` or `Int64`) along -with the tag, i.e. `Name` or `Summary`. The entire format is binary to ensure greater resilience and a more compact -representation. For example, each metadata key is only 8 bytes. +The `Meta` payload consists of a number of records, each with strongly defined types (such as `String` or `Int64`) along with the tag, i.e. `Name` or `Summary`. The entire format is binary to ensure greater resilience and a more compact representation. For example, each metadata key is only 8 bytes. ```d @autoEndian uint32_t length; /** 4 bytes per record length*/ @@ -63,16 +50,12 @@ representation. For example, each metadata key is only 8 bytes. ubyte[1] padding = 0; ``` - ## TLDR that for me... Binary format that is self deduplicating at several layers, permitting fast transactional operations. ## Up Next -Before we work any more on the binary format, we now need to pivot to the source format. Our immediate goal is to now -have moss actually **build** packages from source, with resulting `.stone` packages. Once this step is complete we can -work on installation, upgrades, repositories, etc, and race to becoming a self hosting distribution. +Before we work any more on the binary format, we now need to pivot to the source format. Our immediate goal is to now have moss actually **build** packages from source, with resulting `.stone` packages. Once this step is complete, we can work on installation, upgrades, repositories etc. and race to becoming a self hosting distribution. -Note, the format may still change before it goes into production, as we encounter more cases for optimisation or -improvement. +Note, the format may still change before it goes into production, as we encounter more cases for optimisation or improvement. diff --git a/src/content/docs/blog/2020/09/21/source-format-defined/index.md b/src/content/docs/blog/2020/09/21/source-format-defined/index.md index a7ced992..85ec10a7 100644 --- a/src/content/docs/blog/2020/09/21/source-format-defined/index.md +++ b/src/content/docs/blog/2020/09/21/source-format-defined/index.md @@ -8,29 +8,19 @@ license: "CC-BY-ND-4.0" copyright: "Copyright © 2020 aerynOS Developers" --- -Following quickly on the heels of yesterday's announcement that the binary format has been defined, we've -now implemented the initial version of our source format. The source format provides metadata on a package -along with instructions on how to build the package. +Following quickly on the heels of yesterday's announcement that the binary format has been defined, we've now implemented the initial version of our source format. The source format provides metadata on a package along with instructions on how to build the package. - +The next step of course, is to implement the build tool, converting the source specification into a binary package that the end user can install. With our 2 formats defined, we can now go ahead and implement the build routines. -The next step of course, is to implement the build tool, converting the source specification into a binary package -that the end user can install. With our 2 formats defined, we can now go ahead and implement the build routines. - -![Very trivial package recipe](./Featured.webp) +![Very trivial package recipe](./Featured.webp "Very trivial package recipe") ## A YAML based format -The eagle-eyed among you will already see this is a derivation of the `package.yml` format I originally created -while at [Solus](https://getsol.us). Minor adaptations to the format have been made to support multiple architectures -via the `profiles` key, and package splitting behaviour has now been grouped under a `packages` key to make -the structure more readable. +The eagle-eyed among you will already see this is a derivation of the `package.yml` format I originally created while at [Solus](https://getsol.us). Minor adaptations to the format have been made to support multiple architectures via the `profiles` key, and package splitting behaviour has now been grouped under a `packages` key to make the structure more readable. -In `package.yml`, one would have to redefine subpackage summaries as a key in a list of the primary `summary` key, -such as: +In `package.yml`, one would have to redefine subpackage summaries as a key in a list of the primary `summary` key, such as: ```yaml - rundeps: - primary-run-dep - dev: secondary-run-dep @@ -39,11 +29,9 @@ such as: - dev: Some different summary ``` -We've opted to group "Package Definition" behaviour into core structs, which are allowed to appear in the root-level -package and subpackages: +We've opted to group "Package Definition" behaviour into core structs, which are allowed to appear in the root-level package and subpackages: ```yaml - summary: Top-level summary packages: - dev: @@ -53,11 +41,9 @@ package and subpackages: ## Multiple architecture support -In keeping with the grouping behaviour, we're baking multiple architecture configurations into the YML file. A common -issue encountered with the older format was how to handle `emul32`: +In keeping with the grouping behaviour, we're baking multiple architecture configurations into the YML file. A common issue encountered with the older format was how to handle `emul32`: ```yaml - setup: | if [[ -z "${EMUL32BUILD}" ]]; then %configure --some-emul32-option @@ -69,11 +55,9 @@ issue encountered with the older format was how to handle `emul32`: %make ``` -Our new approach is to group Build Definitions into the root level struct, which may then individually be overridden for -each architecture. For example: +Our new approach is to group Build Definitions into the root level struct, which may then individually be overridden for each architecture. For example: ```yaml - profiles: - ia32: setup: | @@ -86,13 +70,8 @@ each architecture. For example: ## Differences -As you can see it is highly similar to package.yml - which is a great format. However, with our tooling and aims being -slightly different, it was time to reevaluate the spec and bolster it where appropriate. We're happy to share our -changes, but in the interest of not causing a conflict between the 2 variants, we'll be calling ours "stone.yml". +As you can see it is highly similar to package.yml - which is a great format. However, with our tooling and aims being slightly different, it was time to reevaluate the spec and bolster it where appropriate. We're happy to share our changes, but in the interest of not causing a conflict between the 2 variants, we'll be calling ours "stone.yml". -Our main motivation came from the tooling, which is written in the D language. With D we were able to create a strongly -typed parser and explicit schema, and with a struct-based approach it made it more trivial to group similar -definitions. +Our main motivation came from the tooling, which is written in the D language. With D we were able to create a strongly typed parser and explicit schema, and with a struct-based approach it made it more trivial to group similar definitions. -Other than that, we have the same notions with the format, intelligent automatic package splitting, ease of developer -experience, etc. +Other than that, we have the same notions with the format, intelligent automatic package splitting, ease of developer experience, etc. diff --git a/src/content/docs/blog/2020/09/22/results-of-the-experiment/index.md b/src/content/docs/blog/2020/09/22/results-of-the-experiment/index.md new file mode 100644 index 00000000..4de494b5 --- /dev/null +++ b/src/content/docs/blog/2020/09/22/results-of-the-experiment/index.md @@ -0,0 +1,60 @@ +--- +title: "Results Of The Experiment" +date: 2020-09-22T18:05:39+01:00 +draft: false +authors: [ikey] +tags: [news] +featuredImage: "/img/blog/results-of-the-experiment/Featured.webp" +license: "CC-BY-ND-4.0" +copyright: "Copyright © 2020 aerynOS Developers" +--- + +It seems like only yesterday we announced to the world a [Great Experiment](https://serpentos.com/blog/2020/07/01/the-great-experiment/). It was in fact 2 months ago, and a whole lot of work has happened since that point. A few take-homes are immediately clear, the primary one being the need to be a community-oriented Linux distribution. + +To quote ourselves 2 months ago: + +> If the experiment is a success, which of course means having tight controls on scope and timescale, then one would assume the primary way to use Serpent OS would be through some downstream repackaging or reconfiguration, i.e. basing upon the project, to meet specific user demand. + +It turns out that so far the experiment has been successful, and being forkable is still at the very heart of our goals. Others have joined us on our journey, and expressed the same passion in our goals as we have. A community has formed around the project, with individuals sharing the same ambitions for a reliable, rolling operating system with a powerful package manager. + +![Toolchain bootstrap](./Featured.webp "Toolchain bootstrap") + +## An open community + +Over the past 2 months we've transformed from set of ideas into a transparent, community-first organisation with clear leadership and open goals. I've stepped into the Benevolent Dictator For Life position, and Peter has taken on daily responsibilities for the project running. Aydemir is now our treasurer on Open Collective[^1], and many individuals contribute to our project. + +## Transactional package management + +No need to rehash this, but the defining feature of Serpent OS has clearly become moss, something initially not anticipated when we started. A read-only root filesystem, transactional operations, rollbacks, deduplicating throughout and atomic updates. Combine that with a rolling release model, stateless policy and ease-of-use, the core feature-set is already powerful. + +## Deferral of musl integration + +In recent weeks we've been working on `libwildebeest` and `libc-support`, primarily as a stop-gap to provide glibc compatibility without using glibc. While musl has many advantages, it is clear to us now that writing another libc through our support projects isn't what we originally planned. With that in mind we're adopting glibc and putting our musl works under community ownership, until such time as reevaluation shows that musl is what is needed in Serpent OS. Note the primary motivator here is investing our efforts where it makes sense, and obtaining the best results in the most manageable fashion for our users. + +Our new toolchain configuration will be as follows: + + - LLVM/clang as primary toolchain + - libc++ as C++ library + - glibc as C library + - gcc+binutils (`ld.bfd`) provided to build glibc, elfutils, etc. + - Permitting per-package builds using GCC, i.e. kernel to alleviate Clang IAS issues. + +Thus, our toolchain will in fact be a hybrid GNU/LLVM one. This will allow both source and binary compatibility with the majority of desktop + server Linux distributions, facilitating choice of function for our users. + +It should be noted this decision has been made after much discussion internally, on our IRC[^1], on our OpenCollective[^2], etc. Our bootstrap-scripts is being improved to support both `glibc` and `musl`, so that the decision can continuously be reviewed. If we reach a position whereby musl inclusion once again makes sense, thanks to atomic updates from moss, it will be possible to switch. + +## Final thoughts + +Initially Serpent OS emerged as a collective agreement on IRC[^1] as a set of notions as opinions. Over the past few months those opinions have solidified into tangible ideas, and a sense of community. In keeping with what is right for the community, our messaging has been reworked. + +It is fair to say our initial stance appeared quite hostile, as a bullet-point list of exhaustion with past experiences. As we've pivoted to being a fully community-oriented distribution, we've established our goals of being a reliable, flexible, open, rolling Linux distribution with powerful features imbued by the package manager, and an upstream-first approach. + +As such we've agreed to not let our own pet-peeves interfere with the direction of the project, and instead enable users to do what they wish on Serpent OS, be it devops, engineering, browsing, gaming, you name it. We're a general purpose OS with resilience at the core. + +Our focus is on the usability and reliability of the OS - thus our efforts will be invested in areas such as the package manager, hardware enabling, the default experience, etc. + +So, strap yourself in, as we're fond of saying. Development of Serpent OS is about to accelerate rapidly. + +--- +[^1]: aerynOS now uses a [Zulip server](https://aerynos.zulipchat.com/) for community discussion. +[^2]: Since this blog post, we no longer use OpenCollective. For funding, please see our [Funding Page](/sponsor) diff --git a/src/content/docs/blog/2020/09/22/results-of-the-experiment/index.mdx b/src/content/docs/blog/2020/09/22/results-of-the-experiment/index.mdx deleted file mode 100644 index 05b9be0a..00000000 --- a/src/content/docs/blog/2020/09/22/results-of-the-experiment/index.mdx +++ /dev/null @@ -1,93 +0,0 @@ ---- -title: "Results Of The Experiment" -date: 2020-09-22T18:05:39+01:00 -draft: false -authors: [ikey] -tags: [news] -featuredImage: "/img/blog/results-of-the-experiment/Featured.webp" -license: "CC-BY-ND-4.0" -copyright: "Copyright © 2020 aerynOS Developers" ---- -import FigureScreenshotOne from "@/components/ui/FigureScreenshotOne.astro" -import featuredImage from "./Featured.webp" - -It seems like only yesterday we announced to the world a [Great Experiment](https://serpentos.com/blog/2020/07/01/the-great-experiment/). -It was in fact 2 months ago, and a whole lot of work has happened since that point. A few take-homes are immediately clear, the primary -one being the need to be a community-oriented Linux distribution. - - - -To quote ourselves 2 months ago: - - -``` - If the experiment is a success, which of course means having tight controls on scope and timescale, - then one would assume the primary way to use Serpent OS would be through some downstream repackaging - or reconfiguration, i.e. basing upon the project, to meet specific user demand. -``` - -It turns out that so far the experiment has been successful, and being forkable is still at the very -heart of our goals. Others have joined us on our journey, and expressed the same passion in our goals -as we have. A community has formed around the project, with individuals sharing the same ambitions -for a reliable, rolling operating system with a powerful package manager. - - - -## An open community - -Over the past 2 months we've transformed from set of ideas into a transparent, community-first organisation -with clear leadership and open goals. I've stepped into the Benevolent Dictator For Life position, and Peter -has taken on daily responsibilities for the project running. Aydemir is now our treasurer on Open Collective, -and many individuals contribute to our project. - -## Transactional package management - -No need to rehash this, but the defining feature of Serpent OS has clearly become moss, something initially -not anticipated when we started. A read-only root filesystem, transactional operations, rollbacks, deduplicating -throughout and atomic updates. Combine that with a rolling release model, stateless policy and ease-of-use, -the core feature-set is already powerful. - -## Deferral of musl integration - -In recent weeks we've been working on `libwildebeest` and `libc-support`, primarily as a stop-gap to provide -glibc compatibility without using glibc. While musl has many advantages, it is clear to us now that writing -another libc through our support projects isn't what we originally planned. With that in mind we're adopting -glibc and putting our musl works under community ownership, until such time as reevaluation shows that musl is -what is needed in Serpent OS. Note the primary motivator here is investing our efforts where it makes sense, -and obtaining the best results in the most manageable fashion for our users. - -Our new toolchain configuration will be as follows: - - - LLVM/clang as primary toolchain - - libc++ as C++ library - - glibc as C library - - gcc+binutils (`ld.bfd`) provided to build glibc, elfutils, etc. - - Permitting per-package builds using GCC, i.e. kernel to alleviate Clang IAS issues. - -Thus our toolchain will in fact be a hybrid GNU/LLVM one. This will allow both source and binary compatibility -with the majority of desktop + server Linux distributions, facilitating choice of function for our users. - -It should be noted this decision has been made after much discussion internally, on our IRC, on our OpenCollective, -etc. Our bootstrap-scripts is being improved to support both `glibc` and `musl`, so that the decision can continuously -be reviewed. If we reach a position whereby musl inclusion once again makes sense, thanks to atomic updates -from moss, it will be possible to switch. - -## Final thoughts - -Initially Serpent OS emerged as a collective agreement on IRC as a set of notions as opinions. Over the past few -months those opinions have solidified into tangible ideas, and a sense of community. In keeping with what is -right for the community, our messaging has been reworked. - -It is fair to say our initial stance appeared quite hostile, as a bullet-point list of exhaustion with past -experiences. As we've pivoted to being a fully community-oriented distribution, we've established our goals -of being a reliable, flexible, open, rolling Linux distribution with powerful features imbued by the -package manager, and an upstream-first approach. - -As such we've agreed to not let our own pet-peeves interfere with the direction of the project, and instead -enable users to do what they wish on Serpent OS, be it devops, engineering, browsing, gaming, you name it. -We're a general purpose OS with resilience at the core. - -Our focus is on the usability and reliability of the OS - thus our efforts will be invested in areas such -as the package manager, hardware enabling, the default experience, etc. - -So, strap yourself in, as we're fond of saying. Development of Serpent OS is about to accelerate rapidly. diff --git a/src/content/docs/blog/2020/10/11/build-system-functional/index.md b/src/content/docs/blog/2020/10/11/build-system-functional/index.md new file mode 100644 index 00000000..1f15454c --- /dev/null +++ b/src/content/docs/blog/2020/10/11/build-system-functional/index.md @@ -0,0 +1,44 @@ +--- +title: "Build System Functional" +date: 2020-10-11T10:17:25+01:00 +draft: false +authors: [ikey] +tags: [news] +## featuredImage: "/img/blog/build-system-functional/Featured.webp" +license: "CC-BY-ND-4.0" +copyright: "Copyright © 2020 aerynOS Developers" +--- + +Wow, has it been a hectic few weeks, and it definitely shows: last time we blogged it was about re-bootstrapping with glibc. Feels like ancient news already! So, what's new in the world of Serpent OS? Apart from yours truly now being proud parent to a beautiful baby girl, work has resumed on the development of Moss, our package manager. And it builds stuff. Awesomely. Let's quickly catch up with those updates and see where we're headed next. + +![Look at all the buildiness](./Featured.webp "Look at all the buildiness") + + +## It lives! + +We're now able to build **the majority** of a moss package. Notice I've made a distinction there. So, we're able to run the build instructions, and use all the metadata, configuration and macros available. The only thing we're not actually doing is dumping the built files into a binary package. + +:::note[Missing YouTube video] +There used to be a YouTube video at this part of the blog, however our YouTube channel was unfortunately deleted outside of our control and we no longer have access to the video. +::: + +We're able to do the build-system part _rather well_, now. Right now we support the following features in the build system: + + - Multiple, profile-based architecture support + - Automatic `-m32` profile based cross compilation support for 32-bit libraries on 64-bit OS + - Basic Profile Guided Optimisation via the `workload` key. Set a workload, optimise accordingly. + - LLVM Context Sensitive Profile Guided Optimisation. This is default for LLVM with the `workload` key, and comes for free, with multiple build stages. + - Profile based tuning options, such as `optimize` for speed, size, disabling hardening, etc. + - Trivially switch between `gnu` and `llvm` toolchain in `stone.yml`, with profiles knowing the right flags to use with tuning options. + - Recursive macro support in build scripts, defined on per-profile level + - Architecture specific profiles support in stone.yml, i.e. `profiles -> aarch64 -> setup:` to avoid if/else spaghetti. + + +## Next on the agenda + +Now that the huge amount of scaffolding work has been done, we can actually turn the results of builds into installable binary packages using our `moss.format.binary` module. We'll add some magic sauce to have automatic subpackages + inter-package dependencies, along with the expected automatic runtime dependencies + such. Cue some linting, et voila, build tool that's a pleasure to work with. + +Once we have all those packages building, we'll need a way to install them. Luckily some scaffolding is in place for this already, and it won't take much effort to support `moss install somepkg.stone`. Then we throw a few dozen packages in the mix, add some dependency handling, repo support, Bob's your uncle, and your aunt is downloading exclusive early-access images from our Open Collective[^1] as soon as they're available. 😮 + +--- +[^1]: Since this blog post, we no longer use OpenCollective. For funding, please see our [Funding Page](/sponsor) and for ISOs, please see our [Download Page](/download). diff --git a/src/content/docs/blog/2020/10/11/build-system-functional/index.mdx b/src/content/docs/blog/2020/10/11/build-system-functional/index.mdx deleted file mode 100644 index 75b19c11..00000000 --- a/src/content/docs/blog/2020/10/11/build-system-functional/index.mdx +++ /dev/null @@ -1,56 +0,0 @@ ---- -title: "Build System Functional" -date: 2020-10-11T10:17:25+01:00 -draft: false -authors: [ikey] -tags: [news] -## featuredImage: "/img/blog/build-system-functional/Featured.webp" -license: "CC-BY-ND-4.0" -copyright: "Copyright © 2020 aerynOS Developers" ---- -import YouTube from '@/components/ui/YouTube.astro'; - -Wow, has it been a hectic few weeks, and it definitely shows: last time -we blogged it was about re-bootstrapping with glibc. Feels like ancient news -already! So, what's new in the world of Serpent OS? Apart from yours truly -now being proud parent to a beautiful baby girl, work has resumed on -the development of Moss, our package manager. And it builds stuff. Awesomely. -Let's quickly catch up with those updates and see where we're headed next. - - - -![Look at all the buildiness](./Featured.webp) - -## It lives! - -We're now able to build **the majority** of a moss package. Notice I've made -a distinction there. So, we're able to run the build instructions, and use -all of the metadata, configuration and macros available. The only thing -we're not actually doing is dumping the built files into a binary package. - - - -We're able to do the build-system part _rather well_, now. Right now -we support the following features in the build system: - - - Multiple, profile-based architecture support - - Automatic `-m32` profile based cross compilation support for 32-bit libraries on 64-bit OS - - Basic Profile Guided Optimisation via the `workload` key. Set a workload, optimise accordingly. - - LLVM Context Sensitive Profile Guided Optimisation. This is default for LLVM with the `workload` key, and comes for free, with multiple build stages. - - Profile based tuning options, such as `optimize` for speed, size, disabling hardening, etc. - - Trivially switch between `gnu` and `llvm` toolchain in `stone.yml`, with profiles knowing the right flags to use with tuning options. - - Recursive macro support in build scripts, defined on per-profile level - - Architecture specific profiles support in stone.yml, i.e. `profiles -> aarch64 -> setup:` to avoid if/else spaghetti. - -## Next on the agenda - -Now that the huge amount of scaffolding work has been done, we can actually turn the results of builds -into installable binary packages using our `moss.format.binary` module. We'll add some magic sauce to -have automatic subpackages + inter-package dependencies, along with the expected automatic runtime -dependencies + such. Cue some linting, et voila, build tool that's a pleasure to work with. - -Once we have all those packages building, we'll need a way to install them. Luckily some scaffolding -is in place for this already, and it won't take much effort to support `moss install somepkg.stone`. -Then we throw a few dozen packages in the mix, add some dependency handling, repo support, Bob's your -uncle, and your aunt is downloading exclusive early-access images from our [Open Collective](https://opencollective.com/serpent-os) as soon -as they're available. :O diff --git a/src/content/docs/blog/2021/02/08/cleanups-complete/dr-evil.gif b/src/content/docs/blog/2021/02/08/cleanups-complete/dr-evil.gif new file mode 100644 index 00000000..5777a2f5 Binary files /dev/null and b/src/content/docs/blog/2021/02/08/cleanups-complete/dr-evil.gif differ diff --git a/src/content/docs/blog/2021/02/08/cleanups-complete/index.md b/src/content/docs/blog/2021/02/08/cleanups-complete/index.md new file mode 100644 index 00000000..2e5c06aa --- /dev/null +++ b/src/content/docs/blog/2021/02/08/cleanups-complete/index.md @@ -0,0 +1,44 @@ +--- +title: "Cleanups Complete" +date: 2021-02-08T11:50:46Z +draft: false +authors: [ikey] +tags: [news] +license: "CC-BY-ND-4.0" +copyright: "Copyright © 2021 aerynOS Developers" +--- + +Well, we're officially back to working around the clock. After spending some time optimising my workflow, I've been busy getting the entire codebase cleaned up, so we can begin working towards an MVP. + +![Lots and lots of work](./Featured.webp "Lots and lots of work") + +## Codebase hygiene + +Since Friday, I've been working extensively on cleaning up the codebase for the following projects: + + - boulder + - moss + - moss-core + - moss-format + +As it now stands, 1 lint issue (a non-issue, really) exists across all 4 projects. A plethora of issues have been resolved, ranging from endian correctness in the format to correct idiomatic D-lang integration and module naming. + +## Bored + +Granted, cleanups aren't all that sexy. Peter has been updating many parts of the bootstrap project, introducing systemd 247.3, LLVM 11.0.1, etc. We now have all 3 stages building correctly again for the x86_64 target. + +## And then.... + +Yikes, tough audience! So we've formed a new working TODO which will make its way online as a public document at some point. The first stage, cleanups, is done. Next is to restore feature development, working specifically on the extraction and install routines. From there, much work and cadence will be unlocked, allowing us to work towards an MVP. + +![TODO](./TODO.webp "See, we organised stuff") + +## You keep saying MVP.. + +I know, it makes me feel all ... cute and professional. At the moment we're cooking up a high level description of how an MVP demonstration could be deployed. While most of the features are ambiguous, our current ambition is to deploy a preinstalled QCOW2 Serpent OS image as a prealpha to help validate moss/boulder/etc. + +It'll be ugly. Likely slow. Probably insecure as all hell. But it'll be **something**. It'll allow super basic interaction with moss, and a small collection of utilities that can be used in a terminal environment. No display whatsoever. That can come in a future iteration :) + +ETA? ~~Definitely not by the 28th of February~~. When it's done. + +![Dr-Evil making air quotes](./dr-evil.gif) diff --git a/src/content/docs/blog/2021/02/08/cleanups-complete/index.mdx b/src/content/docs/blog/2021/02/08/cleanups-complete/index.mdx deleted file mode 100644 index 8188e98d..00000000 --- a/src/content/docs/blog/2021/02/08/cleanups-complete/index.mdx +++ /dev/null @@ -1,64 +0,0 @@ ---- -title: "Cleanups Complete" -date: 2021-02-08T11:50:46Z -draft: false -authors: [ikey] -tags: [news] -license: "CC-BY-ND-4.0" -copyright: "Copyright © 2021 aerynOS Developers" ---- -import Giphy from '@/components/ui/Giphy.astro'; - -Well, we're officially back to working around the clock. After spending -some time optimising my workflow, I've been busy getting the entire codebase -cleaned up, so we can begin working towards an MVP. - - - -![Lots and lots of work](./Featured.webp) - -## Codebase hygiene - -Since Friday, I've been working extensively on cleaning up the codebase for the -following projects: - - - boulder - - moss - - moss-core - - moss-format - -As it now stands, 1 lint issue (a non-issue, really) exists across all 4 -projects. A plethora of issues have been resolved, ranging from endian correctness -in the format to correct idiomatic D-lang integration and module naming. - -## Bored - -Granted, cleanups aren't all that sexy. Peter has been updating many parts of -the bootstrap project, introducing systemd 247.3, LLVM 11.0.1, etc. We now have -all 3 stages building correctly again for the x86_64 target. - -## And then.... - -Yikes, tough audience! So we've formed a new working TODO which will make its -way online as a public document at some point. The first stage, cleanups, is -done. Next is to restore feature development, working specifically on the -extraction and install routines. From there, much work and cadence will be -unlocked, allowing us to work towards an MVP. - -![TODO](./TODO.webp "See, we organised stuff") - -## You keep saying MVP.. - -I know, it makes me feel all ... cute and professional. At the moment we're cooking up -a high level description of how an MVP demonstration could be deployed. While most of -the features are ambiguous, our current ambition is to deploy a preinstalled QCOW2 -Serpent OS image as a prealpha to help validate moss/boulder/etc. - -It'll be ugly. Likely slow. Probably insecure as all hell. But it'll be **something**. -It'll allow super basic interaction with moss, and a small collection of utilities that -can be used in a terminal environment. No display whatsoever. That can come in a future -iteration :) - -ETA? ~~Definitely not by the 28th of February~~. When it's done. - - diff --git a/src/content/docs/blog/2021/02/09/unlocking-moss/index.md b/src/content/docs/blog/2021/02/09/unlocking-moss/index.md new file mode 100644 index 00000000..e321b81e --- /dev/null +++ b/src/content/docs/blog/2021/02/09/unlocking-moss/index.md @@ -0,0 +1,38 @@ +--- +title: "Unlocking Moss" +date: 2021-02-09T12:45:34Z +draft: false +authors: [ikey] +tags: [news] +featuredImage: "/img/blog/unlocking-moss/Featured.webp" +license: "CC-BY-ND-4.0" +copyright: "Copyright © 2021 aerynOS Developers" +--- + +Wait, what? Another blog post? In the same WEEK? Yeah totally doing that now. So, this is just another devlog update but there have been some interesting updates that we'd like to share. + +## LDC present in the bootstrap + +Thanks to some awesome work from Peter, we now have LDC (The LLVM D Lang Compiler) present in the `stage3` bootstrap. To simplify the process we use the official binary release of LDC to bootstrap LDC for Serpent OS. + +In turn, this has allowed us to get to a point where we can now build `moss` and `boulder` within `stage3`. This is super important, as we'll use the `stage3` chroot and `boulder` to produce the binary packages that create `stage4`. + +Some patching has taken place to prevent using `ld.gold` and instead use `lld` to integrate with our toolchain decisions. + +![A wild LDC appears](./Featured.webp "A wild LDC appears") + +## Reworking the Payload implementation + +Originally our prototype moss format only contained a `ContentPayload` for files, and a `MetaPayload` for metadata, package information, etc. As a result, we opted for simple structs, basic case handling and an iterable `Reader` implementation. + +As the format expanded, we bolted deduplication in as a core feature. To achieve this, we converted the `ContentPayload` into a "megablob", i.e. every **unique** file in the package, one after the other, all compressed in one operation. We then store the offsets and IDs of these files within an `IndexPayload` to allow splitting the "megablob" into separate, unique assets. Consequently, we added a `LayoutPayload` which describes the final file system layout of the package, referencing the unique ID (hash) of the asset to install. + +So, while the format grew into something we liked, the code supporting it became very limiting. After many internal debates and discussions, we're going to approach the format from a different angle on the code front. + +It will no longer be necessary/possible to iterate payloads in _sequence_ within a moss archive, instead we'll preload the data (unparsed) and stick it aside when reading the file, winding it to the `ContentPayload` segment if found. After initial loading of the file is complete, the `Reader` API will support retrieval (and lazy unpacking) of a data segment. In turn this will allow code to "grab" the content, index and layout payloads and use advanced APIs to cache assets, and apply them to disk in a single blit operation. + +In short, we've unlocked the path to installing moss packages while preserving the advanced features of the format. The same new APIs will permit introspection of the archives metadata, and of course, storing these records in a stateful system database. + +## Thank you for the unnecessary update + +Oh you're quite welcome 😛 Hopefully now you can see our plan, and that we're on track to meet our not-28th target. Sure, some code needs throwing away, but all codebases are evolutionary. Our major hurdle has been solved (mentally) - now it's just time to implement it and let the good times roll. diff --git a/src/content/docs/blog/2021/02/09/unlocking-moss/index.mdx b/src/content/docs/blog/2021/02/09/unlocking-moss/index.mdx deleted file mode 100644 index d795665b..00000000 --- a/src/content/docs/blog/2021/02/09/unlocking-moss/index.mdx +++ /dev/null @@ -1,71 +0,0 @@ ---- -title: "Unlocking Moss" -date: 2021-02-09T12:45:34Z -draft: false -authors: [ikey] -tags: [news] -featuredImage: "/img/blog/unlocking-moss/Featured.webp" -license: "CC-BY-ND-4.0" -copyright: "Copyright © 2021 aerynOS Developers" ---- -import FigureScreenshotOne from "@/components/ui/FigureScreenshotOne.astro" -import featuredImage from "./Featured.webp" - -Wait, what? Another blog post? In the same WEEK? Yeah totally doing that -now. So, this is just another devlog update but there have been some interesting -updates that we'd like to share. - - - - -## LDC present in the bootstrap - -Thanks to some awesome work from Peter, we now have LDC (The LLVM D Lang Compiler) -present in the stage3 bootstrap. To simplify the process we use the official -binary release of LDC to bootstrap LDC for Serpent OS. - -In turn, this has allowed us to get to a point where we can now build `moss` and -`boulder` within stage3. This is super important, as we'll use the stage3 chroot -and `boulder` to produce the binary packages that create stage4. - -Some patching has taken place to prevent using `ld.gold` and instead use `lld` -to integrate with our toolchain decisions. - - - -## Reworking the Payload implementation - -Originally our prototype moss format only contained a `ContentPayload` for files, and -a `MetaPayload` for metadata, package information, etc. As a result, we opted for simple -structs, basic case handling and an iterable `Reader` implementation. - -As the format expanded, we bolted deduplication in as a core feature. To achieve this, -we converted the `ContentPayload` into a "megablob", i.e. every **unique** file in the -package, one after the other, all compressed in one operation. We then store the offsets -and IDs of these files within an `IndexPayload` to allow splitting the "megablob" into -separate, unique assets. Consequently, we added a `LayoutPayload` which describes the -final file system layout of the package, referencing the unique ID (hash) of the asset -to install. - -So, while the format grew into something we liked, the code supporting it became very -limiting. After many internal debates and discussions, we're going to approach the -format from a different angle on the code front. - -It will no longer be necessary/possible to iterate payloads in _sequence_ within a -moss archive, instead we'll preload the data (unparsed) and stick it aside when reading -the file, winding it to the `ContentPayload` segment if found. After initial loading of -the file is complete, the `Reader` API will support retrieval (and lazy unpacking ) of -a data segment. In turn this will allow code to "grab" the content, index and layout -payloads and use advanced APIs to cache assets, and apply them to disk in a single -blit operation. - -In short, we've unlocked the path to installing moss packages while preserving the -advanced features of the format. The same new APIs will permit introspection of the -archives metadata, and of course, storing these records in a stateful system database. - -## Thank you for the unnecessary update - -Oh you're quite welcome :P Hopefully now you can see our plan, and that we're on track -to meet our not-28th target. Sure, some code needs throwing away, but all codebases -are evolutionary. Our major hurdle has been solved (mentally) - now it's just time -to implement it and let the good times roll. diff --git a/src/content/docs/blog/2021/02/17/moss-format-read-write-support/index.md b/src/content/docs/blog/2021/02/17/moss-format-read-write-support/index.md index c4183762..55c72552 100644 --- a/src/content/docs/blog/2021/02/17/moss-format-read-write-support/index.md +++ b/src/content/docs/blog/2021/02/17/moss-format-read-write-support/index.md @@ -8,23 +8,13 @@ license: "CC-BY-ND-4.0" copyright: "Copyright © 2021 aerynOS Developers" --- -It's been 8 days since our last blogpost and a lot of development work has happened -in that time. Specifically, we've completely reworked the internals of the `moss-format` -module to support read/write operations.. Which means installation is coming soon (TM) - - +It's been 8 days since our last blogpost and a lot of development work has happened in that time. Specifically, we've completely reworked the internals of the `moss-format` module to support read/write operations... Which means installation is coming soon (TM) ![Development work on moss-format](./Featured.webp) -So, many commits have been made to the core repositories, however the most -important project to focus on right now is `moss-format`, which we used to -define and implement our binary and source formats. This module is shared -between `boulder`, our build tool, and `moss`, our package manager. +So, many commits have been made to the core repositories, however the most important project to focus on right now is `moss-format`, which we used to define and implement our binary and source formats. This module is shared between `boulder`, our build tool, and `moss`, our package manager. -We've removed the old enumeration approach from the `Reader` class, instead -requiring that it processes **data** payloads in-place, deferring reading the -**content** payload streams. We've also enforced strong typing to allow -safe and powerful APIs: +We've removed the old enumeration approach from the `Reader` class, instead requiring that it processes **data** payloads in-place, deferring reading the **content** payload streams. We've also enforced strong typing to allow safe and powerful APIs: ```d import moss.format.binary.payload.meta : MetaPayload; @@ -32,50 +22,24 @@ safe and powerful APIs: auto reader = new Reader(File(argv[0], "rb")); auto metadata = reader.payload!MetaPayload(); ``` -# -Right now we can read and write our `MetaPayload` from and to the stream, -allowing us to encode & decode the metadata associated with the package, -with strong type information (i.e. `Uint64`, `String`, etc.) +Right now we can read and write our `MetaPayload` from and to the stream, allowing us to encode & decode the metadata associated with the package, with strong type information (i.e. `Uint64`, `String`, etc.) ## Next Steps -We need to restore the `IndexPayload`, `LayoutPayload` and `ContentPayload` -definitions. The first two are simply **data** payloads and will largely -follow the design of the newly reimplemented `MetaPayload`. Then we restore -`ContentPayload` support, and this will allow the next steps: unpack, install. - -Many of the babysteps required are done now, which power our binary format. -The design of the API is done in a way which will allow powerful manipulation -via the `std.algorithm` and `std.range` APIs, enabling extremely simple and -reliable installation routines. - -It might seem odd that we've spent so much time on the core **format**, -however I should point out that the design of the format is central to the -OS design. Our installation routine is **not** unpacking of an archive. - -With our binary format, the stream contains multiple PayloadHeaders, -with fixed lengths, type, tag, version, and compression information. -It is up to each Payload implementation to then parse the binary -data contained within. Currently our Payloads are compressed using ZSTD, though -other compression algorithms are supported. - -So, we have the `MetaPayload` for metadata. Additionally, we encode all **unique** -files in the package in a single compressed payload, the `ContentPayload`. The -offset to each unique file (by hash) is stored within the `IndexPayload`, and -the filesystem layout is encoded as data within the `LayoutPayload`. - -In order to unpack a moss package, the `ContentPayload` blob will be decompressed -to a temporary location. Then, each struct within the `IndexPayload` can be used -to copy portions (`copy_file_range`) of the blob to the cache store for permanence. We skip each Index -if its already present within the cache. Finally, the target filesystem is populated -with a view of all installed packages using the `LayoutPayload` structs, creating -a shallow installation using hardlinks and directories. - -The net result is deep deduplication, atomic updates, and flexibility for the user. -Once we add transactions it'll be possible to boot an older version of the OS using -the deduplication capabilities for offline recovery. Additionally there is no requirement -for file deletion, rename or modification for an update to succeed. +We need to restore the `IndexPayload`, `LayoutPayload` and `ContentPayload` definitions. The first two are simply **data** payloads and will largely follow the design of the newly reimplemented `MetaPayload`. Then we restore `ContentPayload` support, and this will allow the next steps: unpack, install. + +Many of the baby steps required are done now, which power our binary format. The design of the API is done in a way which will allow powerful manipulation via the `std.algorithm` and `std.range` APIs, enabling extremely simple and reliable installation routines. + +It might seem odd that we've spent so much time on the core **format**, however I should point out that the design of the format is central to the OS design. Our installation routine is **not** unpacking of an archive. + +With our binary format, the stream contains multiple PayloadHeaders, with fixed lengths, type, tag, version, and compression information. It is up to each Payload implementation to then parse the binary data contained within. Currently our Payloads are compressed using ZSTD, though other compression algorithms are supported. + +So, we have the `MetaPayload` for metadata. Additionally, we encode all **unique** files in the package in a single compressed payload, the `ContentPayload`. The offset to each unique file (by hash) is stored within the `IndexPayload`, and the filesystem layout is encoded as data within the `LayoutPayload`. + +In order to unpack a moss package, the `ContentPayload` blob will be decompressed to a temporary location. Then, each struct within the `IndexPayload` can be used to copy portions (`copy_file_range`) of the blob to the cache store for permanence. We skip each Index if its already present within the cache. Finally, the target filesystem is populated with a view of all installed packages using the `LayoutPayload` structs, creating a shallow installation using hardlinks and directories. + +The net result is deep deduplication, atomic updates, and flexibility for the user. Once we add transactions it'll be possible to boot an older version of the OS using the deduplication capabilities for offline recovery. Additionally, there is no requirement for file deletion, rename or modification for an update to succeed. ## Nutshell diff --git a/src/content/docs/blog/2021/03/16/optimising-package-distribution/index.md b/src/content/docs/blog/2021/03/16/optimising-package-distribution/index.md index 1f03d313..db4d8cd1 100644 --- a/src/content/docs/blog/2021/03/16/optimising-package-distribution/index.md +++ b/src/content/docs/blog/2021/03/16/optimising-package-distribution/index.md @@ -8,42 +8,26 @@ license: "CC-BY-ND-4.0" copyright: "Copyright © 2021 aerynOS Developers" --- -Getting updates as fast as possible to users has made deltas a popular and sought after feature for distributing -packages. Over the last couple of days, I've been investigating various techniques we can look at to support deltas in -`moss`. - - +Getting updates as fast as possible to users has made deltas a popular and sought after feature for distributing packages. Over the last couple of days, I've been investigating various techniques we can look at to support deltas in `moss`. ## Trade-offs Between Packages and Deltas -Minimising the size of updates is particularly valuable where files are downloaded many times and even better if they're -updated infrequently. With a rolling release, packages will be updated frequently, so creating deltas can become -resource intensive, especially if supporting updates over a longer period of time. Therefore it's important to get the -right balance between compression speed, decompression memory and minimising file size. +Minimising the size of updates is particularly valuable where files are downloaded many times and even better if they're updated infrequently. With a rolling release, packages will be updated frequently, so creating deltas can become resource intensive, especially if supporting updates over a longer period of time. Therefore, it's important to get the right balance between compression speed, decompression memory and minimising file size. | | Package priorities | How best to meet these needs | |--------------|------------------------------|------------------------------| | Developers | Creation speed | Quickly created packages | | Users | File size and update speed | Size minimised deltas | -From the users point of view, minimising file size and upgrade time are important priorities, but for a developer, the -speed at which packages are created and indexed is vital to progression. Deltas are different to packages in that they -aren't required immediately, so there's minimal impact in taking longer to minimise their size. By getting deltas right, -we can trade-off the size of packages to speed up development, while users will not be affected and fetch only size -optimised deltas. +From the users point of view, minimising file size and upgrade time are important priorities, but for a developer, the speed at which packages are created and indexed is vital to progression. Deltas are different to packages in that they aren't required immediately, so there's minimal impact in taking longer to minimise their size. By getting deltas right, we can trade-off the size of packages to speed up development, while users will not be affected and fetch only size optimised deltas. ## Test Case - QtWebEngine -QtWebEngine provides a reasonable test case where the package is a mix of binaries, resources and translations, but -above average in size (157.3MB uncompressed). The first trade-off for speed over size has already been made by -incorporating `zstd` in moss over `xz`, where even with max compression `zstd` is already 5.6% larger than using `xz`. -This is of course due to the amazing decompression speeds where `zstd` is magnitudes faster. +QtWebEngine provides a reasonable test case where the package is a mix of binaries, resources and translations, but above average in size (157.3MB uncompressed). The first trade-off for speed over size has already been made by incorporating `zstd` in moss over `xz`, where even with max compression `zstd` is already 5.6% larger than using `xz`. This is of course due to the amazing decompression speeds where `zstd` is magnitudes faster. -![Compression levels with zstd](./Featured.webp) +![Compression levels with zstd](./Featured.webp "Compression levels with zstd") -With maximum compression, large packages can take over a minute to compress. With a moderate increase in size, one can -reduce compression time by 2-10x. While making me happier as a developer, it does create extra network load during -updates. +With maximum compression, large packages can take over a minute to compress. With a moderate increase in size, one can reduce compression time by 2-10x. While making me happier as a developer, it does create extra network load during updates. | Full Package | zstd -16 -T0 | zstd -16 | zstd -19 -T0 | zstd -19 | zstd -22 | xz -9 | |-----------------|--------------|----------|--------------|----------|----------|--------| @@ -52,30 +36,15 @@ updates. ## Deltas to the Rescue! -There are two basic methods for deltas. One simple method is to include only files that have been changed since the -earlier release. With reproducible builds, it is typical to create the same file from the same inputs. However, with a -rolling release model, files will frequently have a small change from dependency changes and new versions of the package -itself. In these circumstances the delta size starts to get closer to the full package anyway. As a comparison to other -delta techniques, this approach resulted in a 38.2MB delta as it was a rebuild of the same version at a different time -so the resources and translations were unchanged (and therefore omitted from the delta). +There are two basic methods for deltas. One simple method is to include only files that have been changed since the earlier release. With reproducible builds, it is typical to create the same file from the same inputs. However, with a rolling release model, files will frequently have a small change from dependency changes and new versions of the package itself. In these circumstances the delta size starts to get closer to the full package anyway. As a comparison to other delta techniques, this approach resulted in a 38.2MB delta as it was a rebuild of the same version at a different time so the resources and translations were unchanged (and therefore omitted from the delta). -An alternative is a binary diff, which is a significant improvement when files have small changes between releases. -`bsdiff` has long been used for this purpose and trying it out (without knowing much about it) I managed to create a -delta of 33.2MB, not a bad start at all. +An alternative is a binary diff, which is a significant improvement when files have small changes between releases. `bsdiff` has long been used for this purpose and trying it out (without knowing much about it) I managed to create a delta of 33.2MB, not a bad start at all. -To highlight the weakness of the simple method, when you compare the delta across a version change, the simple delta was -only a 7% reduction of the full package (as most files have changed), while using an optimal binary diff, it still -managed to achieve a respectable 31% size reduction. +To highlight the weakness of the simple method, when you compare the delta across a version change, the simple delta was only a 7% reduction of the full package (as most files have changed). Whilst using an optimal binary diff, it still managed to achieve a respectable 31% size reduction. ## A New Contender -While looking into alternatives, I happened to [stumble across](https://github.com/facebook/zstd/releases/tag/v1.4.5) -a new feature in `zstd` which can be used to create deltas. As we already use `zstd` heavily it should make integration -easier. `--patch-from` allows `zstd` to use the old uncompressed package as a dictionary to create the newer package. In -this way common parts between the releases will be reused in order to reduce the file size. Playing around I quickly -achieved the same size as `bsdiff`, and with a few tweaks was able to further reduce the delta by a **further 23.5%!** -The best part is that it has the same speedy decompression as `zstd`, so it will recreate most packages from deltas in -the blink of an eye! +While looking into alternatives, I happened to [stumble across](https://github.com/facebook/zstd/releases/tag/v1.4.5) a new feature in `zstd` which can be used to create deltas. As we already use `zstd` heavily it should make integration easier. `--patch-from` allows `zstd` to use the old uncompressed package as a dictionary to create the newer package. In this way common parts between the releases will be reused in order to reduce the file size. Playing around I quickly achieved the same size as `bsdiff`, and with a few tweaks was able to further reduce the delta by a **further 23.5%!** The best part is that it has the same speedy decompression as `zstd`, so it will recreate most packages from deltas in the blink of an eye! | Delta | only changed files | bsdiff | zstd -19 | zstd -22 | zstd -22 --zstd=chainLog=30 | |-----------------|--------------------|--------|----------|----------|-----------------------------| @@ -85,24 +54,12 @@ the blink of an eye! ## Next Steps -There's certainly a lot of information to digest, but the next step is to integrate a robust delta solution into the -`moss` format. I really like the `zstd` approach, where you can tune for speed with an increase in size if desired. With -minimising on delta size, users can benefit from smaller updates while developers can benefit from faster package -creation times. +There's certainly a lot of information to digest, but the next step is to integrate a robust delta solution into the `moss` format. I really like the `zstd` approach, where you can tune for speed with an increase in size if desired. With minimising on delta size, users can benefit from smaller updates while developers can benefit from faster package creation times. Some final thoughts for future consideration: -- `zstd` has seen many improvements over the years, so I believe that ratios and performance will see incremental -improvements over time. Release [1.4.7](https://github.com/facebook/zstd/releases/tag/v1.4.7) already brought -significant improvements to deltas (which are reflected in this testing). -- The highest compression levels (`--ultra`) are single threaded in `zstd`, so delta creation can be done in parallel to -maximise utilisation. -- Over optimising the tunables can have a negative impact on both speed and size. As an example, -`--zstd=targetLength=4096` did result in a 2KB reduction in size at the same speed, but when applied to different inputs -(kernel source tree), it not only made it larger by a few hundred bytes, but added **4 minutes** to delta creation! -- Memory usage of applying deltas can be high for large packages (1.74GB for the kernel source tree) as it has to ingest -the full size of the original uncompressed package. It is certainly possible to split up payloads with some delta users -even creating patches on a per file basis. It is a bit more complicated when library names change version numbers each -release with only the SONAME symlink remaining unchanged. -- There's always the option to repack packages at higher compression levels later (when deltas are created). This solves -getting the package 'live' ASAP and minimises the size (eventually), but adds some complication. +- `zstd` has seen many improvements over the years, so I believe that ratios and performance will see incremental improvements over time. Release [1.4.7](https://github.com/facebook/zstd/releases/tag/v1.4.7) already brought significant improvements to deltas (which are reflected in this testing). +- The highest compression levels (`--ultra`) are single threaded in `zstd`, so delta creation can be done in parallel to maximise utilisation. +- Over optimising the tunables can have a negative impact on both speed and size. As an example, `--zstd=targetLength=4096` did result in a 2KB reduction in size at the same speed, but when applied to different inputs (kernel source tree), it not only made it larger by a few hundred bytes, but added **4 minutes** to delta creation! +- Memory usage of applying deltas can be high for large packages (1.74GB for the kernel source tree) as it has to ingest the full size of the original uncompressed package. It is certainly possible to split up payloads with some delta users even creating patches on a per file basis. It is a bit more complicated when library names change version numbers each release with only the SONAME symlink remaining unchanged. +- There's always the option to repack packages at higher compression levels later (when deltas are created). This solves getting the package 'live' ASAP and minimises the size (eventually), but adds some complication. diff --git a/src/content/docs/blog/2021/03/20/moss-unlocked/index.md b/src/content/docs/blog/2021/03/20/moss-unlocked/index.md index 7232e7bb..bb62d7f3 100644 --- a/src/content/docs/blog/2021/03/20/moss-unlocked/index.md +++ b/src/content/docs/blog/2021/03/20/moss-unlocked/index.md @@ -8,30 +8,19 @@ license: "CC-BY-ND-4.0" copyright: "Copyright © 2021 aerynOS Developers" --- -Well, it's not all doom and gloom these days. We've actually made some -significant progress in the last few days, so it seems a good time to -share a progress update. - - - -![Extracting content from moss archives](./Featured.webp) +Well, it's not all doom and gloom these days. We've actually made some significant progress in the last few days, so it seems a good time to share a progress update. +![Extracting content from moss archives](./Featured.webp "Extracting content from moss archives") ## moss can now extract -Oh yeah, that totally happened. So, we can now successfully build moss -packages from `boulder` and then extract them to disk once again with -`moss`. This might sound totally uninteresting, but it demonstrates -that our format is actually working as intended. +Oh yeah, that totally happened. So, we can now successfully build moss packages from `boulder` and then extract them to disk once again with `moss`. This might sound totally uninteresting, but it demonstrates that our format is actually working as intended. -Admittedly the code is super rough within `moss` and somewhat proof -of concept, however we're able to extract the contents of the moss -archive and rebuild the layout on disk. +Admittedly the code is super rough within `moss` and somewhat proof of concept, however we're able to extract the contents of the moss archive and rebuild the layout on disk. ## What makes extraction difficult? -Well, quirky new format for one. A moss archive currently consists of -4 "payloads", or major sections: +Well, quirky new format for one. A moss archive currently consists of 4 "payloads", or major sections: - `MetaPayload` @@ -39,62 +28,35 @@ Well, quirky new format for one. A moss archive currently consists of - `IndexPayload` - Contains the IDs of all unique files (hash) and their offsets within - the ContentPayload + Contains the IDs of all unique files (hash) and their offsets within the ContentPayload - `LayoutPayload` - A sequence of specialised structs describing the final "layout" of the - package on disk, with attributes, paths, and for regular files, the ID - of the file in the ContentPayload to build this file **from**. + A sequence of specialised structs describing the final "layout" of the package on disk, with attributes, paths, and for regular files, the ID of the file in the ContentPayload to build this file **from**. - `ContentPayload` - A binary blob containing every unique file from the package, in an order - described by the `IndexPayload`. The files are stored sequentially with - no gaps. + A binary blob containing every unique file from the package, in an order described by the `IndexPayload`. The files are stored sequentially with no gaps. -Additionally, each payload is independently compressed using `zstd`. In order -to extract files to disk, we must first decompress `ContentPayload` to a -temporary file. Next, we blit each file from the "megablob" to the cache store, -using the `IndexPayload` to understand the offsets. Finally, we apply the -instructions in `LayoutPayload` to construct the final layout on disk, hardlinking -the cache assets into their final locations, setting attributes, etc. +Additionally, each payload is independently compressed using `zstd`. In order to extract files to disk, we must first decompress `ContentPayload` to a temporary file. Next, we blit each file from the "megablob" to the cache store, using the `IndexPayload` to understand the offsets. Finally, we apply the instructions in `LayoutPayload` to construct the final layout on disk, hardlinking the cache assets into their final locations, setting attributes, etc. -Net result? Deduplication on a per package basis, and a system-wide deduplication -policy allowing sharing of identical assets on disk between multiple packages. -This will also power our core update mechanism, whereby each update is atomic, -and is simply the difference on disk from the previous version, permitting a -powerful rollback mechanism. +Net result? Deduplication on a per package basis, and a system-wide deduplication policy allowing sharing of identical assets on disk between multiple packages. This will also power our core update mechanism, whereby each update is atomic, and is simply the difference on disk from the previous version, permitting a powerful rollback mechanism. ## Room for improvement ![Multiple](./Multiple.webp "Extracting multiple moss archives") -There are areas where we're doing things inefficiently, and we'll certainly improve -that in future revisions of the important. For example, `IndexPayload` actually -wastes some bytes by storing redundant information that can be calculated at -runtime. Additionally, we want to use the zstd C APIs directly to gain the level -of control we **actually need**. We're also going to wrap the `copy_file_range` -syscall to make extraction of the content payload more efficient and not rely on -userspace copies. +There are areas where we're doing things inefficiently, and we'll certainly improve that in future revisions of the important. For example, `IndexPayload` actually wastes some bytes by storing redundant information that can be calculated at runtime. Additionally, we want to use the zstd C APIs directly to gain the level of control we **actually need**. We're also going to wrap the `copy_file_range` syscall to make extraction of the content payload more efficient and not rely on userspace copies. -However, we're working towards a prealpha, and some inefficiencies are OK. Our -first port of call will be a prealpha image constructed from `.stone` files, produced -by `boulder`, installed by `moss`. This will validate our toolchain and tooling -and serve as a jumping off point for the project. +However, we're working towards a prealpha, and some inefficiencies are OK. Our first port of call will be a prealpha image constructed from `.stone` files, produced by `boulder`, installed by `moss`. This will validate our toolchain and tooling and serve as a jumping off point for the project. -Stay tuned, there is a whole bunch of awesome coming now that moss is officially -unlocked and progressing. +Stay tuned, there is a whole bunch of awesome coming now that moss is officially unlocked and progressing. ## And finally -I want to thank everyone who is currently supporting the project. I also want to personally -thank you for your understanding of the setbacks of real life, given the difficult times myself -and my family have been going through. I hope it is clear that I remain committed to the -project and it's future, which is why we're transparently run and funded via [OpenCollective](https://opencollective.com/serpent-os). +I want to thank everyone who is currently supporting the project. I also want to personally thank you for your understanding of the setbacks of real life, given the difficult times myself and my family have been going through. I hope it is clear that I remain committed to the project and it's future, which is why we're transparently run and funded via OpenCollective[^1]. -Despite the rough times, work continues, and awesome people join our ranks on a regular basis. -Stability is on the immediate horizon and work with Serpent OS grows exponentially. You can -be part of our journey, and help us build an amazing community and project that outlives us -all. +Despite the rough times, work continues, and awesome people join our ranks on a regular basis. Stability is on the immediate horizon and work with Serpent OS grows exponentially. You can be part of our journey, and help us build an amazing community and project that outlives us all. + +--- +[^1]: Since this blog post, we no longer use OpenCollective. For funding, please see our [Funding Page](/sponsor) diff --git a/src/content/docs/blog/2021/05/18/let-there-be-databases/index.mdx b/src/content/docs/blog/2021/05/18/let-there-be-databases/index.md similarity index 50% rename from src/content/docs/blog/2021/05/18/let-there-be-databases/index.mdx rename to src/content/docs/blog/2021/05/18/let-there-be-databases/index.md index bad27097..b553f544 100644 --- a/src/content/docs/blog/2021/05/18/let-there-be-databases/index.mdx +++ b/src/content/docs/blog/2021/05/18/let-there-be-databases/index.md @@ -8,39 +8,23 @@ license: "CC-BY-ND-4.0" copyright: "Copyright © 2021 aerynOS Developers" --- -We haven't been too great on sharing progress lately, so welcome to an overdue update on -timelines, progress, and database related shmexiness. +We haven't been too great on sharing progress lately, so welcome to an overdue update on timelines, progress, and database related shmexiness. - +![Emerging DB design](./Featured.webp "Emerging DB design") -![Emerging DB design](./Featured.webp) +OK, so you may remember `moss-format`, our module for reading and writing moss binary archives. It naturally contains much in the way of binary serialisation support, so we've extended the format to support "database" files. In reality, they are more like tables binary encoded into a single file, identified by a filepath. +The DB archives are currently stored **without** compression to ensure 0-copy mmap() access when loading from disk, as a premature optimisation. This may change in future if we find the DB files taking up too much disk space. -OK, so you may remember `moss-format`, our module for reading and writing moss binary archives. -It naturally contains much in the way of binary serialisation support, so we've extended the -format to support "database" files. In reality, they are more like tables binary encoded into -a single file, identified by a filepath. +So far we've implemented a "StateMetaDB", which stores metadata on every recorded State on the system, and right now I'm in the progress of implementing the "StateEntriesDB", which is something akin to a binary encoded dpkg selections file with candidate specification reasons. -The DB archives are currently stored **without** compression to ensure 0-copy mmap() access -when loading from disk, as a premature optimisation. This may change in future if we find the -DB files taking up too much disk space. - -So far we've implemented a "StateMetaDB", which stores metadata on every recorded State on -the system, and right now I'm in the progress of implementing the "StateEntriesDB", which is -something akin to a binary encoded dpkg selections file with candidate specification reasons. - -Next on the list is the LayoutsDB (file manifests) and the CacheDB, for recording refcounts -of every cached file in the OS pool. +Next on the list is the LayoutsDB (file manifests) and the CacheDB, for recording refcounts of every cached file in the OS pool. ## Integration with Serpent ECS -An interesting trial we're currently implementing is to hook the DB implementation up to -our Entity Component system from the Serpent Engine, in order to provide fast, cache coherent, -in memory storage for the DB. It's implemented using many nice DLang idioms, allowing the full -use of `std.algorithm` APIs: +An interesting trial we're currently implementing is to hook the DB implementation up to our Entity Component system from the Serpent Engine, in order to provide fast, cache coherent, in memory storage for the DB. It's implemented using many nice DLang idioms, allowing the full use of `std.algorithm` APIs: ```d - auto states() { import std.algorithm : map; @@ -61,14 +45,9 @@ use of `std.algorithm` APIs: ## Tying it all together -Ok, so you can see we need basic DB types for storing the files for each moss archive, plus each -cache and state entry. If you look at the ECS code above, it becomes quite easy to imagine how this -will impact installation of archives. Our new install code will simply modify the existing state, -cache the incoming package, and apply the layout from the DB to disk, before committing the new -DB state. +Ok, so you can see we need basic DB types for storing the files for each moss archive, plus each cache and state entry. If you look at the ECS code above, it becomes quite easy to imagine how this will impact installation of archives. Our new install code will simply modify the existing state, cache the incoming package, and apply the layout from the DB to disk, before committing the new DB state. -In essence, our DB work is the current complex target, and installation is a <50 line trick -tying it all together. +In essence, our DB work is the current complex target, and installation is a <50 line trick tying it all together. ```d /* Pseudocode */ @@ -87,4 +66,4 @@ tying it all together. Til next time - - Ikey +Ikey diff --git a/src/content/docs/blog/2021/07/27/boulder-keeps-on-rolling/index.mdx b/src/content/docs/blog/2021/07/27/boulder-keeps-on-rolling/index.md similarity index 72% rename from src/content/docs/blog/2021/07/27/boulder-keeps-on-rolling/index.mdx rename to src/content/docs/blog/2021/07/27/boulder-keeps-on-rolling/index.md index e5fc0238..8c13fe2d 100644 --- a/src/content/docs/blog/2021/07/27/boulder-keeps-on-rolling/index.mdx +++ b/src/content/docs/blog/2021/07/27/boulder-keeps-on-rolling/index.md @@ -9,12 +9,9 @@ license: "CC-BY-ND-4.0" copyright: "Copyright © 2021 aerynOS Developers" --- -Squirrelling away in the background has been some great changes to bring `boulder` closer to its full potential. Here's -a quick recap of some of the more important ones. +We have been squirreling away in the background to bring some great changes to bring `boulder` closer to its full potential. Here's a quick recap of some of the more important ones. - - -![Boulder hard at work](./Featured.webp) +![Boulder hard at work](./Featured.webp "Boulder hard at work") ## Key Changes to Boulder @@ -31,6 +28,4 @@ a quick recap of some of the more important ones. - Added the llvm profile merge steps after running the workload - Recreate a clean working directory at the start of each PGO phase -With all this now in place, the build stages of `boulder` are close to completion. But don't worry, there's plenty more -great features to come to make building packages for Serpent OS simple, flexible and performant. Next steps will be testing -out these new features to see how much they can add to the overall `stage4` performance. +With all this now in place, the build stages of `boulder` are close to completion. But don't worry, there's plenty more great features to come to make building packages for Serpent OS simple, flexible and performant. Next steps will be testing out these new features to see how much they can add to the overall `stage4` performance. diff --git a/src/content/docs/blog/2021/08/02/initial-performance-testing/index.md b/src/content/docs/blog/2021/08/02/initial-performance-testing/index.md new file mode 100644 index 00000000..62b8c224 --- /dev/null +++ b/src/content/docs/blog/2021/08/02/initial-performance-testing/index.md @@ -0,0 +1,71 @@ +--- +title: "Initial Performance Testing" +date: 2021-08-02T11:28:24+10:00 +draft: false +authors: [sunnyflunk] +tags: [news] +license: "CC-BY-ND-4.0" +copyright: "Copyright © 2021 aerynOS Developers" +--- + +With further progress on `boulder`, we can now build native stone packages with some easy tweaks such as profile guided optimizations (PGO) and link time optimizations (LTO). That means we can take a first look at what the performance of the first cut of Serpent OS shows for the future. The tests have been conducted using [`benchmarking-tools`](https://github.com/sunnyflunk/benchmarking-tools) with Serpent OS measured in a `chroot` on the same host with the same kernel and config. + +One of the key focuses for early in the project is on reducing build time. Every feature can either add or subtract from the time it takes to produce a package. With a source/binary hybrid model, users will greatly benefit from the faster builds as well. In terms of what I've targeted in these tests is the performance of `clang` and testing some compiler flag options on `cmake`. + +## Clang Shows its Promise + +`clang` has always been a compiler with a big future. The performance credentials have also been improving each release and are now starting to see it perform strongly against its `GNU` counterpart. It is common to hear that `clang` is slow and produces less optimized code. I will admit that most distros provide a slow build of `clang`, but that will not be the case in Serpent OS. + +It is important to note that in this comparison the Host distro has pulled in some patches from `LLVM-13` that greatly improve the performance of `clang`. Prior to this, their tests actually took `50% longer` for `cmake` and `configure` but only `10% longer` for compiling. `boulder` does not yet support patching in builds so the packages are completely vanilla. + +| Test using clang | Serpent | Host | Difference | +|-----------------------|--------------|-------------|------------| +| `cmake` LLVM | 5.89s | 10.58s | 79.7% | +| Compile -j4 `llvm-ar` | 126.16s | 155.32s | 23.1% | +| `configure` gettext | 36.64s | 63.55s | 73.4% | + +Based on the results during testing, the performance of `clang` in Serpent OS still has room to improve and was just a quick tuning pass. At stages where I would have expected to be ahead already, the compile performance was only equal (but `cmake` and `configure` were still well ahead). + +## GCC Matters Too! + +While `clang` is the default compiler in Serpent OS, there may be instances where the performance is not quite where it could be. It is common to see software have more optimized code paths where they are not tested with `clang` upstream. As an example, here's a couple of patches in flac ([1](https://github.com/xiph/flac/commit/67ea8badadd3e63b8e8af5fe837d075104569330), [2](https://github.com/xiph/flac/commit/d4a1b345dd16591ff6f17c67ee519afebe2f9792)) that demonstrate this being improved. Using `benchmarking-tools`, it is easy to see where `gcc` and `clang` builds are running different functions via `perf` results. + +In circumstances where the slowdown is due to hitting poor optimization paths in `clang`, we always have the option to build packages using `gcc`, where the `GNU` toolchain is essential for building `glibc`. Therefore, having a solid `GNU` toolchain is important but small compile time improvements won't be noticed by users or developers as much. + +| Test using gcc | Serpent | Host | Difference | +|---------------------|--------------|-------------|------------| +| `cmake` LLVM | 7.00s | 7.95s | 13.6% | +| Compile `llvm-ar` | 168.11s | 199.07s | 18.4% | +| `configure` gettext | 45.45s | 51.93s | 14.3% | + +## An OS is More Than Just a Compiler + +While the current bootstrap exists only as a starting point for building the rest of Serpent OS, there are some other packages we can easily test and compare. Here's a summary of those results. + +| Test | Serpent | Host | Difference | +|-----------------------------------|--------------|-------------|------------| +| Pybench | 1199.67ms | 1024.33ms | -14.6% | +| `xz` Compress Kernel (-3 -T1) | 42.67s | 46.57s | 9.1% | +| `xz` Compress Kernel (-9 -T4) | 71.25s | 76.12s | 6.8% | +| `xz` Decompress Kernel | 8.03s | 8.18s | 1.9% | +| `zlib` Compress Kernel | 12.60s | 13.17s | 4.5% | +| `zlib` Decompress Kernel | 5.14s | 5.21s | 1.4% | +| `zstd` Compress Kernel (-8 -T1) | 5.77s | 7.06s | 22.3% | +| `zstd` Compress Kernel (-19 -T4) | 51.87s | 66.52s | 28.3% | +| `zstd` Decompress Kernel | 2.90s | 3.08s | 6.3% | + +## State of the Bootstrap + +From my experiences with testing the bootstrap, it is clear there's some cobwebs in there that require some more iterations of the toolchain. There also seems to be some slowdowns in not including all the dependencies of some packages. Once more packages are included, naturally all the testing will be redone and help influence the default compiler flags of the project. + +It's not yet clear the experience of using `libc++` vs `libstdc++` with the `clang` compiler. Once the cobwebs are out and Serpent OS further developed, the impact (if any) should become more obvious. There are also some parts not yet included in `boulder` such as stripping files, LTO and other flags by default that will speed up loading libraries. At this stage this is deliberate until integrating outputs from builds (such as symbol information). + +But this provides an excellent platform to build out the rest of the OS. The raw speed of the `clang` compiler will make iterating and expanding the package set a real joy! + +## Hang On, What's Going on With Python? + +Very astute of you to notice! `python` in its current state is an absolute minimal build of `python` in order to run `meson`. However, I did an `analyze` run in `benchmarking-tools` where it became obvious that they were doing completely different things. + +![Apples and oranges comparison](./Featured.webp "An apples and oranges comparison") + +For now I'll simply be assuming this will sort itself out when `python` is built complete with all its functionality. And before anyone wants to point the finger at `clang`, you get the same result with `gcc`. diff --git a/src/content/docs/blog/2021/08/02/initial-performance-testing/index.mdx b/src/content/docs/blog/2021/08/02/initial-performance-testing/index.mdx deleted file mode 100644 index 3c7707f9..00000000 --- a/src/content/docs/blog/2021/08/02/initial-performance-testing/index.mdx +++ /dev/null @@ -1,106 +0,0 @@ ---- -title: "Initial Performance Testing" -date: 2021-08-02T11:28:24+10:00 -draft: false -authors: [sunnyflunk] -tags: [news] -license: "CC-BY-ND-4.0" -copyright: "Copyright © 2021 aerynOS Developers" ---- - -With further progress on `boulder`, we can now build native stone packages with some easy tweaks such as profile -guided optimizations (PGO) and link time optimizations (LTO). That means we can take a first look at what the -performance of the first cut of Serpent OS shows for the future. The tests have been conducted using -[`benchmarking-tools`](https://github.com/sunnyflunk/benchmarking-tools) with Serpent OS measured in a `chroot` on -the same host with the same kernel and config. - - - -One of the key focuses for early in the project is on reducing build time. Every feature can either add or subtract -from the time it takes to produce a package. With a source/binary hybrid model, users will greatly benefit from the -faster builds as well. In terms of what I've targeted in these tests is the performance of `clang` and testing some -compiler flag options on `cmake`. - -## Clang Shows its Promise - -`clang` has always been a compiler with a big future. The performance credentials have also been improving each release -and are now starting to see it perform strongly against its `GNU` counterpart. It is common to hear that `clang` is slow -and produces less optimized code. I will admit that most distros provide a slow build of `clang`, but that will not be -the case in Serpent OS. - -It is important to note that in this comparison the Host distro has pulled in some patches from `LLVM-13` that greatly -improve the performance of `clang`. Prior to this, their tests actually took `50% longer` for `cmake` and `configure` -but only `10% longer` for compiling. `boulder` does not yet support patching in builds so the packages are completely -vanilla. - -| Test using clang | Serpent | Host | Difference | -|-----------------------|--------------|-------------|------------| -| `cmake` LLVM | 5.89s | 10.58s | 79.7% | -| Compile -j4 `llvm-ar` | 126.16s | 155.32s | 23.1% | -| `configure` gettext | 36.64s | 63.55s | 73.4% | - -Based on the results during testing, the performance of `clang` in Serpent OS still has room to improve and was just a -quick tuning pass. At stages where I would have expected to be ahead already, the compile performance was only equal -(but `cmake` and `configure` were still well ahead). - -## GCC Matters Too! - -While `clang` is the default compiler in Serpent OS, there may be instances where the performance is not quite where it -could be. It is common to see software have more optimized code paths where they are not tested with `clang` upstream. As -an example, here's a couple of patches in flac ([1](https://github.com/xiph/flac/commit/67ea8badadd3e63b8e8af5fe837d075104569330), -[2](https://github.com/xiph/flac/commit/d4a1b345dd16591ff6f17c67ee519afebe2f9792)) that demonstrate this being improved. -Using `benchmarking-tools`, it is easy to see where `gcc` and `clang` builds are running different functions via `perf` -results. - -In circumstances where the slowdown is due to hitting poor optimization paths in `clang`, we always have the option to -build packages using `gcc`, where the `GNU` toolchain is essential for building `glibc`. Therefore having a solid `GNU` -toolchain is important but small compile time improvements won't be noticed by users or developers as much. - -| Test using gcc | Serpent | Host | Difference | -|---------------------|--------------|-------------|------------| -| `cmake` LLVM | 7.00s | 7.95s | 13.6% | -| Compile `llvm-ar` | 168.11s | 199.07s | 18.4% | -| `configure` gettext | 45.45s | 51.93s | 14.3% | - -## An OS is More Than Just a Compiler - -While the current bootstrap exists only as a starting point for building the rest of Serpent OS, there are some other -packages we can easily test and compare. Here's a summary of those results. - -| Test | Serpent | Host | Difference | -|-----------------------------------|--------------|-------------|------------| -| Pybench | 1199.67ms | 1024.33ms | -14.6% | -| `xz` Compress Kernel (-3 -T1) | 42.67s | 46.57s | 9.1% | -| `xz` Compress Kernel (-9 -T4) | 71.25s | 76.12s | 6.8% | -| `xz` Decompress Kernel | 8.03s | 8.18s | 1.9% | -| `zlib` Compress Kernel | 12.60s | 13.17s | 4.5% | -| `zlib` Decompress Kernel | 5.14s | 5.21s | 1.4% | -| `zstd` Compress Kernel (-8 -T1) | 5.77s | 7.06s | 22.3% | -| `zstd` Compress Kernel (-19 -T4) | 51.87s | 66.52s | 28.3% | -| `zstd` Decompress Kernel | 2.90s | 3.08s | 6.3% | - -## State of the Bootstrap - -From my experiences with testing the bootstrap, it is clear there's some cobwebs in there that require some more iterations -of the toolchain. -There also seems to be some slowdowns in not including all the dependencies of some packages. Once more packages are included, -naturally all the testing will be redone and help influence the default compiler flags of the project. - -It's not yet clear the experience of using `libc++` vs `libstdc++` with the `clang` compiler. Once the cobwebs are out and -Serpent OS further developed, the impact (if any) should become more obvious. There are also some parts not yet included in -`boulder` such as stripping files, LTO and other flags by default that will speed up loading libraries. At this stage this is -deliberate until integrating outputs from builds (such as symbol information). - -But this provides an excellent platform to build out the rest of the OS. The raw speed of the `clang` compiler will make -iterating and expanding the package set a real joy! - -## Hang On, What's Going on With Python? - -Very astute of you to notice! `python` in its current state is an absolute minimal build of `python` in order to run `meson`. -However, I did an `analyze` run in `benchmarking-tools` where it became obvious that they were doing completely different -things. - -![Apples and oranges comparison](./Featured.webp) - -For now I'll simply be assuming this will sort itself out when `python` is built complete with all its functionality. And -before anyone wants to point the finger at `clang`, you get the same result with `gcc`. diff --git a/src/content/docs/blog/2021/08/03/moss-db-progress/index.md b/src/content/docs/blog/2021/08/03/moss-db-progress/index.md new file mode 100644 index 00000000..8b989142 --- /dev/null +++ b/src/content/docs/blog/2021/08/03/moss-db-progress/index.md @@ -0,0 +1,65 @@ +--- +title: "Moss DB Progress" +date: 2021-08-03T14:16:16+01:00 +draft: false +authors: [ikey] +tags: [news] +featuredImage: "/img/blog/moss-db-progress/Featured.webp" +license: "CC-BY-ND-4.0" +copyright: "Copyright © 2021 aerynOS Developers" +--- + +I'll try to make this update as brief as I can but it's certainly an important one, so let's dive right into it. The last few weeks have been rough but work on our package manager has still been happening. Today we're happy to reveal another element of the equation: moss-db. + +![Putting moss-db to the test](./Featured.webp "Putting moss-db to the test") + +`moss-db` is an abstract API providing access to simplistic "Key Value" stores. We had initially used some payload based files as databases but that introduced various hurdles, so we decided to take a more abstract approach to not tie ourselves to any specific implementation of a database. + +Our main goal with `moss-db` is to encapsulate the [RocksDB](https://rocksdb.org/) library, providing sane, idiomatic access to a key value store. + +## High level requirements + +At the highest level, we needed something that could store arbitrary keys and values, grouped by some kind of common key (commonly known as "buckets"). We've succeeded in that abstraction, which also required us to fork a `rocksdb-binding` to add the `Transform` APIs required. + +Additionally, we required idiomatic range behaviours for iteration, as well as generic access patterns. To that affect we can now `foreach` a bucket, pipe it through the awesomely powerful `std.algorithm` APIs, and automatically encode/decode keys and values through our generic APIs when implementing the `mossdbEncode()` and `mossdbDecode()` functions for a specific **type**. + +In a nutshell, this was the old, ugly, hard way: + +```d + /* old, hard way */ + auto nameZ = name.toStringz(); + int age = 100; + ubyte[int.sizeof] ageEncoded = nativeToBigEndian(ageEncoded); + db.setDatum(cast(ubyte[]) (nameZ[0 .. strlen(nameZ)]), ageEncoded); +``` + +And this is the new, shmexy way: + +```d + db.set("john", 100); + db.set("user 100", "bobby is my name"); + + auto result = db.get!int("john"); + if (result.found) + { + writeln(result.value); + } + + auto result2 = db.get!string("user 100"); + if (result2.found) + { + writeln(result2.value); + } +``` + +It's quite easy to see the new API lends itself robustly to our needs, so that we may implement stateful, strongly typed databases for moss. + +## Next Steps + +Even though some APIs in `moss-db` may still be lacking (remove, for example) we're happy that it can provide the foundation for our next steps. We now need to roll out the new `StateDB`, `MetaDB` and `LayoutDB`, to record system states, package metadata, and filesystem layout information, respectively. + +With those 3 basic requirements in place we can combine the respective works into installation routines. Which, clearly, warrants another blog post ... 😀 + +For now you can see the relevant projects on our GitLab[^1] project. + +[^1]: We no longer use GitLab and keep our codebase on [GitHub](https://github.com/aerynos) diff --git a/src/content/docs/blog/2021/08/03/moss-db-progress/index.mdx b/src/content/docs/blog/2021/08/03/moss-db-progress/index.mdx deleted file mode 100644 index 333246da..00000000 --- a/src/content/docs/blog/2021/08/03/moss-db-progress/index.mdx +++ /dev/null @@ -1,83 +0,0 @@ ---- -title: "Moss DB Progress" -date: 2021-08-03T14:16:16+01:00 -draft: false -authors: [ikey] -tags: [news] -featuredImage: "/img/blog/moss-db-progress/Featured.webp" -license: "CC-BY-ND-4.0" -copyright: "Copyright © 2021 aerynOS Developers" ---- -import FigureScreenshotOne from "@/components/ui/FigureScreenshotOne.astro" -import featuredImage from "./Featured.webp" - -I'll try to make this update as brief as I can but it's certainly an important one, so -let's dive right into it. The last few weeks have been rough but work on our package manager -has still been happening. Today we're happy to reveal another element of the equation: moss-db. - - - - - -`moss-db` is an abstract API providing access to simplistic "Key Value" stores. We had initially used -some payload based files as databases but that introduced various hurdles, so we decided to take -a more abstract approach to not tie ourselves to any specific implementation of a database. - -Our main goal with `moss-db` is to encapsulate the [RocksDB](https://rocksdb.org/) library, providing sane, idiomatic access -to a key value store. - -## High level requirements - -At the highest level, we needed something that could store arbitrary keys and values, grouped -by some kind of common key (commonly known as "buckets"). We've succeeded in that abstraction, -which also required us to fork a `rocksdb-binding` to add the `Transform` APIs required. - -Additionally we required idiomatic range behaviours for iteration, as well as generic access -patterns. To that affect we can now `foreach` a bucket, pipe it through the awesomely powerful -`std.algorithm` APIs, and automatically encode/decode keys and values through our generic APIs -when implementing the `mossdbEncode()` and `mossdbDecode()` functions for a specific **type**. - -In a nutshell, this was the old, ugly, hard way: - - -```d - /* old, hard way */ - auto nameZ = name.toStringz(); - int age = 100; - ubyte[int.sizeof] ageEncoded = nativeToBigEndian(ageEncoded); - db.setDatum(cast(ubyte[]) (nameZ[0 .. strlen(nameZ)]), ageEncoded); -``` - -And this is the new, shmexy way: - -```d - db.set("john", 100); - db.set("user 100", "bobby is my name"); - - auto result = db.get!int("john"); - if (result.found) - { - writeln(result.value); - } - - auto result2 = db.get!string("user 100"); - if (result2.found) - { - writeln(result2.value); - } -``` - -It's quite easy to see the new API lends itself robustly to our needs, so that -we may implement stateful, strongly typed databases for moss. - -## Next Steps - -Even though some APIs in `moss-db` may still be lacking (remove, for example) -we're happy that it can provide the foundation for our next steps. We now need -to roll out the new `StateDB`, `MetaDB` and `LayoutDB`, to record system states, -package metadata, and filesystem layout information, respectively. - -With those 3 basic requirements in place we can combine the respective works -into installation routines. Which, clearly, warrants another blog post ... :) - -For now you can see the relevant projects on our [GitLab](https://gitlab.com/serpent-os/core) project. diff --git a/src/content/docs/blog/2021/08/10/a-rolling-boulder-gathers-no-moss/index.md b/src/content/docs/blog/2021/08/10/a-rolling-boulder-gathers-no-moss/index.md new file mode 100644 index 00000000..1183fc46 --- /dev/null +++ b/src/content/docs/blog/2021/08/10/a-rolling-boulder-gathers-no-moss/index.md @@ -0,0 +1,60 @@ +--- +title: "A Rolling Boulder Gathers No Moss" +date: 2021-08-10T12:02:37+01:00 +draft: false +authors: [ikey] +tags: [news] +license: "CC-BY-ND-4.0" +copyright: "Copyright © 2021 aerynOS Developers" +--- + +We actually did it. Super pleased to announce that `moss` is now capable of installing and removing packages. Granted, super rough, but gotta start somewhere right? + +![Transactional system roots + installation in moss](./Featured.webp "Transactional system roots + installation in moss") + +OK let's recap. A moss archive is super weird, and consists of multiple containers, or payloads. We use a strongly typed binary format, per-payload compression (Currently `zstd`), and don't store files in a typical archive fashion. + +Instead a `.stone` file (moss archive) has a Content Payload, which is a compressed "megablob" of all the unique files in a given package. The various files contained within that "megablob" are described in an IndexPayload, which simply contains some IDs and offsets, acting much like a lookup table. + +That data alone doesn't actually tell us **where files go** on the filesystem when installed. For that, we have a specialist Layout Payload, encoding the final layout of the package on disk. + +As can be imagined, the weirdness made it quite difficult to install in a trivial fashion. + +## Databases + +Well, persistence really. Thanks to `RocksDB` and our new `moss-db` project, we can trivially store information we need from each package we "precache". Primarily, we store full system states within our new StateDB, which at present is simply a series of package ID selections grouped by a unique 64-bit integer. + +Additionally, we remember the layouts within the LayoutDB so that we can eventually recreate said layout on disk. + +## Precaching + +Before we actually commit to an install, we try to precache all of the stone files in our pool. So we unpack the content payload ("megablob"), split it into various unique files in the pool ready for use. At this point we also record the Layouts, but do not "install" the package to a system root. + +## Blitting + +This is our favourite step. When our cache is populated, we gather all relevant layouts for the current selections, and then begin applying them in a **new** system root. All directories and symlinks are created as normal, whereas any regular file is hardlinked from the pool. This process takes a fraction of a second and gives us completely clean, deduplicated system roots. + +Currently these live in `/.moss/store/root/$ID/usr`. To complete the transaction, we update `/usr` to point to the new `usr` tree **atomically** assuming that a reboot isn't needed. In future, boot switch logic will update the tree for us. + +## Removal + +Removal is quite the same as installation. We simply remove the package IDs from the new state selections (copied from the last state) and blit a new system root, finally updating the atomic `/usr` pointer. + +![Removal](./Removal.webp "Removal of packages with moss. Everything is a transaction") + +## Tying it all together + +We retain classic package management traits such as having granular selections, multiple repositories, etc, whilst sporting advanced features like full system deduplication and transactions/rollbacks. + +When we're far enough along, it'll be possible to boot back to the last working transaction without requiring an internet connection. Due to the use of pooling and hardlinks, each transaction tree is only a few KiB, with files shared between each transaction/install. + +## On the list.. + +We need some major cleanups, better error handling, logging, timed functions, and an eventloop driven process to allow parallel fetching/precaching prior to final system rootfs construction. + +It's taken us a very long time to get to this point, and there is still more work to be done. However, this is a major milestone and we can now start adding features and polish. + +Once the required features are in place, we'll work on the much needed pre alpha ISO 😀 If you fancy helping us get to that stage quicker, do check out our OpenCollective![^1] (We won't limit prealpha availability, don't worry 😀) + +--- +[^1]: Since this blog post, we no longer use OpenCollective. For funding, please see our [Funding Page](/sponsor) diff --git a/src/content/docs/blog/2021/08/10/a-rolling-boulder-gathers-no-moss/index.mdx b/src/content/docs/blog/2021/08/10/a-rolling-boulder-gathers-no-moss/index.mdx deleted file mode 100644 index 451814a8..00000000 --- a/src/content/docs/blog/2021/08/10/a-rolling-boulder-gathers-no-moss/index.mdx +++ /dev/null @@ -1,98 +0,0 @@ ---- -title: "A Rolling Boulder Gathers No Moss" -date: 2021-08-10T12:02:37+01:00 -draft: false -authors: [ikey] -tags: [news] -license: "CC-BY-ND-4.0" -copyright: "Copyright © 2021 aerynOS Developers" ---- - -We actually did it. Super pleased to announce that `moss` is now capable -of installing and removing packages. Granted, super rough, but gotta start -somewhere right? - - - -![Transactional system roots + installation in moss](./Featured.webp) - -OK let's recap. A moss archive is super weird, and consists of multiple -containers, or payloads. We use a strongly typed binary format, per-payload -compression (Currently `zstd`), and don't store files in a typical archive -fashion. - -Instead a `.stone` file (moss archive) has a Content Payload, which is -a compressed "megablob" of all the unique files in a given package. The -various files contained within that "megablob" are described in an IndexPayload, -which simply contains some IDs and offsets, acting much like a lookup table. - -That data alone doesn't actually tell us **where files go** on the filesystem -when installed. For that, we have a specialist Layout Payload, encoding the -final layout of the package on disk. - -As can be imagined, the weirdness made it quite difficult to install in -a trivial fashion. - -## Databases - -Well, persistence really. Thanks to `RocksDB` and our new `moss-db` project, -we can trivially store information we need from each package we "precache". -Primarily, we store full system states within our new StateDB, which at -present is simply a series of package ID selections grouped by a unique -64-bit integer. - -Additionally we remember the layouts within the LayoutDB so that we can -eventually recreate said layout on disk. - -## Precaching - -Before we actually commit to an install, we try to precache all of the stone -files in our pool. So we unpack the content payload ("megablob"), split it -into various unique files in the pool ready for use. At this point we also -record the Layouts, but do not "install" the package to a system root. - -## Blitting - -This is our favourite step. When our cache is populated, we gather all -relevant layouts for the current selections, and then begin applying them -in a **new** system root. All directories and symlinks are created as normal, -whereas any regular file is hardlinked from the pool. This process takes a -fraction of a second and gives us completely clean, deduplicated system roots. - -Currently these live in `/.moss/store/root/$ID/usr`. To complete the transaction, -we update `/usr` to point to the new `usr` tree **atomically** assuming that -a reboot isn't needed. In future, boot switch logic will update the tree for us. - -## Removal - -Removal is quite the same as installation. We simply remove the package IDs -from the new state selections (copied from the last state) and blit a new -system root, finally updating the atomic `/usr` pointer. - -![Removal](./Removal.webp "Removal of packages with moss. Everything is a transaction") - -## Tying it all together - -We retain classic package management traits such as having granular selections, -multiple repositories, etc, whilst sporting advanced features like full system -deduplication and transactions/rollbacks. - -When we're far enough along, it'll be possible to boot back to the last working -transaction without requiring an internet connection. Due to the use of pooling -and hardlinks, each transaction tree is only a few KiB, with files shared between -each transaction/install. - -## On the list.. - -We need some major cleanups, better error handling, logging, timed functions, -and an eventloop driven process to allow parallel fetching/precaching prior -to final system rootfs construction. - -It's taken us a very long time to get to this point, and there is still more -work to be done. However this is a major milestone and we can now start -adding features and polish. - -Once the required features are in place, we'll work on the much needed -pre alpha ISO :) If you fancy helping us get to that stage quicker, do -check out our OpenCollective! (We won't limit prealpha availability, -don't worry :)) diff --git a/src/content/docs/blog/2021/08/25/unpacking-the-build-process-1/index.md b/src/content/docs/blog/2021/08/25/unpacking-the-build-process-1/index.md new file mode 100644 index 00000000..2fe923eb --- /dev/null +++ b/src/content/docs/blog/2021/08/25/unpacking-the-build-process-1/index.md @@ -0,0 +1,49 @@ +--- +title: "Unpacking the Build Process: Part 1" +date: 2021-08-25T19:04:12+10:00 +draft: false +authors: [sunnyflunk] +tags: [news] +featuredImage: "/img/blog/unpacking-the-build-process-1/Featured.webp" +license: "CC-BY-ND-4.0" +copyright: "Copyright © 2021 aerynOS Developers" +--- + +While the build process (or packaging as it's commonly referred to) is largely hidden to most users, it forms a fundamental and important aspect to the efficiency of development. In Serpent OS this efficiency also extends to users via source based builds for packages you may want to try/use that aren't available as binaries upstream. + +The build process can be thought of in three distinct parts, setting up the build environment, compiling the source and post build analysis plus package creation. Please note that this process hasn't been finalized in Serpent OS so we will be making further changes to the process where possible. + +## Setting up the Build Environment + +Some key parts to setting up the build environment: + +- Downloading packages needed as dependencies for the build +- Downloading upstream source files used in the build +- Fetching and analyzing the latest repository index +- Creating a reproducible environment for the build (chroot, container or VM for example) +- Extracting and installing packages into the environment +- Extracting tarballs for the build (this is frequently incorporated as part of the build process instead) + +While the focus of early optimization work has been on build time performance, there's more overhead to creating packages time than simply compiling code. Now the compiler is in a good place, we can explore the rest of the build process. + +## Packaging is More than just Compile Time + +There's been plenty of progress in speeding up the creation of the build environment such as parallel downloads to reduce connection overhead and using `zstd` for the fast decompression of packages. But there's more that we can do to provide an optimal experience to our packagers. + +Some parts of the process are challenging to optimize as while you can download multiple files at once to ensure maximum throughput, you are still ultimately limited by your internet speed. When packaging regularly (or building a single package multiple times), downloaded files are cached so become a one off cost. One part we have taken particular interest in speeding up is extracting and installing packages into the environment. + +![Eight seconds (for a small number of dependencies) that don't need to be endlessly repeated](./Featured.webp "Eight seconds (for a small number of dependencies) that don't need to be endlessly repeated") + +## The Dynamic Duo: boulder and moss + +Installing packages to a clean environment can be the most time consuming part of setting up the build (excluding fetching files which is highly variable). Serpent OS has a massive advantage with the design of `moss` where packages are cached (extracted on disk) and ready to be used by multiple roots, including the creation of clean build environments for `boulder`. Having experienced a few build systems in action, setting up the root could take quite some time with a large number of dependencies (even getting over a minute). `moss` avoids the cost of extracting packages entirely every build by utilizing its cache! + +There are also secondary benefits to how `moss` handles packages via its caches where disk writes are reduced by only needing to extract packages a single time. But hang on, won't you be using `tmpfs` for builds? Of course we will have RAM builds as an option and there are benefits there too! When extracting packages to the RAM disk, it consumes memory which can add up to more than a GB before the build even begins. `moss` allows for us to start with an empty `tmpfs` so we can perform larger builds before exhausting the memory available on our system. + +Another great benefit is due to the atomic nature of `moss`. This means that we can add packages to be cached as soon as they're fetched while waiting for the remaining files to finish downloading (both for `boulder` and system updates). Scheduling jobs becomes much more efficient and we can have the build environment available in moments after the last file is downloaded! + +`moss` allows us to eliminate one of the bigger time sinks in setting up builds, enabling developers and contributors alike to be more efficient in getting work done for Serpent OS. With greater efficiency it may become possible to provide a second architecture for older machines (if the demand arises). + +## Part 1? + +Yes, there's plenty more to discuss. There will be more follow up posts showing the cool features Serpent OS is doing to both reduce the time taken to build packages and in making packages easier to create so stay tuned! diff --git a/src/content/docs/blog/2021/08/25/unpacking-the-build-process-1/index.mdx b/src/content/docs/blog/2021/08/25/unpacking-the-build-process-1/index.mdx deleted file mode 100644 index 7c740310..00000000 --- a/src/content/docs/blog/2021/08/25/unpacking-the-build-process-1/index.mdx +++ /dev/null @@ -1,79 +0,0 @@ ---- -title: "Unpacking the Build Process: Part 1" -date: 2021-08-25T19:04:12+10:00 -draft: false -authors: [sunnyflunk] -tags: [news] -featuredImage: "/img/blog/unpacking-the-build-process-1/Featured.webp" -license: "CC-BY-ND-4.0" -copyright: "Copyright © 2021 aerynOS Developers" ---- -import FigureScreenshotOne from "@/components/ui/FigureScreenshotOne.astro" -import featuredImage from "./Featured.webp" - -While the build process (or packaging as it's commonly referred to) is largely hidden to most users, it forms a -fundamental and important aspect to the efficiency of development. In Serpent OS this efficiency also extends to -users via source based builds for packages you may want to try/use that aren't available as binaries upstream. - - - -The build process can be thought of in three distinct parts, setting up the build environment, compiling the source -and post build analysis plus package creation. Please note that this process hasn't been finalized in Serpent OS so -we will be making further changes to the process where possible. - -## Setting up the Build Environment - -Some key parts to setting up the build environment: - -- Downloading packages needed as dependencies for the build -- Downloading upstream source files used in the build -- Fetching and analyzing the latest repository index -- Creating a reproducible environment for the build (chroot, container or VM for example) -- Extracting and installing packages into the environment -- Extracting tarballs for the build (this is frequently incorporated as part of the build process instead) - -While the focus of early optimization work has been on build time performance, there's more overhead to creating -packages time than simply compiling code. Now the compiler is in a good place, we can explore the rest of the -build process. - -## Packaging is More than just Compile Time - -There's been plenty of progress in speeding up the creation of the build environment such as parallel downloads to -reduce connection overhead and using `zstd` for the fast decompression of packages. But there's more that we can -do to provide an optimal experience to our packagers. - -Some parts of the process are challenging to optimize as while you can download multiple files at once to ensure -maximum throughput, you are still ultimately limited by your internet speed. When packaging regularly (or building -a single package multiple times), downloaded files are cached so become a one off cost. One part we have taken -particular interest in speeding up is extracting and installing packages into the environment. - - - -## The Dynamic Duo: boulder and moss - -Installing packages to a clean environment can be the most time consuming part of setting up the build (excluding -fetching files which is highly variable). Serpent OS has a massive advantage with the design of `moss` where -packages are cached (extracted on disk) and ready to be used by multiple roots, including the creation of clean -build environments for `boulder`. Having experienced a few build systems in action, setting up the root could take -quite some time with a large number of dependencies (even getting over a minute). `moss` avoids the cost of extracting -packages entirely every build by utilizing its cache! - -There are also secondary benefits to how `moss` handles packages via its caches where disk writes are reduced by only -needing to extract packages a single time. But hang on, won't you be using `tmpfs` for builds? Of course we will have -RAM builds as an option and there are benefits there too! When extracting packages to the RAM disk, it consumes memory -which can add up to more than a GB before the build even begins. `moss` allows for us to start with an empty `tmpfs` so -we can perform larger builds before exhausting the memory available on our system. - -Another great benefit is due to the atomic nature of `moss`. This means that we can add packages to be cached as -soon as they're fetched while waiting for the remaining files to finish downloading (both for `boulder` and system -updates). Scheduling jobs becomes much more efficient and we can have the build environment available in moments after -the last file is downloaded! - -`moss` allows us to eliminate one of the bigger time sinks in setting up builds, enabling developers and -contributors alike to be more efficient in getting work done for Serpent OS. With greater efficiency it may become -possible to provide a second architecture for older machines (if the demand arises). - -## Part 1? - -Yes, there's plenty more to discuss so there will be more follow up posts showing the cool features Serpent OS is doing -to both reduce the time taken to build packages and in making packages easier to create so stay tuned! diff --git a/src/content/docs/blog/2021/09/20/unpacking-the-build-process-2/index.md b/src/content/docs/blog/2021/09/20/unpacking-the-build-process-2/index.md new file mode 100644 index 00000000..05a9effe --- /dev/null +++ b/src/content/docs/blog/2021/09/20/unpacking-the-build-process-2/index.md @@ -0,0 +1,65 @@ +--- +title: "Unpacking the Build Process: Part 2" +date: 2021-09-20T16:04:12+10:00 +draft: false +authors: [sunnyflunk] +tags: [news] +featuredImage: "/img/blog/unpacking-the-build-process-2/Featured.webp" +license: "CC-BY-ND-4.0" +copyright: "Copyright © 2021 aerynOS Developers" +--- + +Part 2 looks at the core of the build process, turning the source into compiled code. In Serpent OS this is handled by our build tool `boulder`. It is usually the part of the build that takes the longest, so where speed ups have the most impact. How long it takes is largely down to the performance of your compiler and what compile flags you are building with. + +This post follows on from [Unpacking the Build Process: Part 1](/blog/2021/08/25/unpacking-the-build-process-1). + +## Turning Source into Compiled Code + +The steps for compiling code are generally quite straight-forward: +- Setting up the build (cmake, configure, meson) +- Compiling the source (in parallel threads) +- Installing the build into a package directory + +This will build software compiled against packages installed on your system. It's a bit more complicated when packaging as we first set up an environment to compile in (Part 1). But even then you have many choices to make and each can have an impact on how long it takes to compile the code. Do you build with Link Time Optimizations (LTO) or Profile Guided Optimizations (PGO), do you build the package for performance or for the smallest size? Then there's packages that benefit considerably from individual tuning flags (like `-fno-semantic-interposition` with `python`). With so many possibilities, `boulder` helps us utilize them through convenient configuration options. + +## What Makes boulder so Special? + +As I do a lot of packaging and performance tuning, `boulder` is where I spend most of my time. Here are some key features that `boulder` brings to make my life easier. + + - Ultimate control over build C/CXX/LDFLAGS via the tuning key + - Integrated 2 stage context sensitive PGO builds with a single line workload + - Able to switch between `gnu` and `llvm` toolchains easily + - Rules based package creation + - Control the extraction locations for multiple upstream tarballs + +`boulder` will also be used to generate and amend our `stone.yml` files to take care of as much as possible automatically. This is only the beginning for `boulder` as it will continue to be expanded to learn new tricks to make packaging more automated and able to bring more information to help packagers know when they can improve their `stone.yml`, or alert them that something might be missing. + +Serpent OS is focused on the performance of produced packages, even if that means that builds will take longer to complete. This is why we have put in significant efforts to speed up the compiler and setup tools in order to offset and minimize the time needed to enable greater performance. + +## Why do You Care so Much About Setup Time? + +My initial testing focused on the performance of `clang` as well as the time taken to run `cmake` and `configure`. This lays the foundation for all future work in expanding the Serpent OS package archives at a much faster pace. On the surface, running `cmake` can be a small part of the overall build. However, it is important in that it utilizes a single thread, so is not sped up by adding more CPU cores like the compile time is. With a more compile heavy build, our highly tuned compiler can build the source in around 75s. So tuning the setup step to run in 5s rather than 10s actually reduces the overall build time by an additional 6%! + +There are many smaller packages where the setup time is an even higher proportion of the overall build and becomes more relevant as you increase the numbers of threads on the builder. For example, when building `nano` on the host, the configure step takes 13.5s, while the build itself takes only 2.3s, so there's significant gains to be had from speeding up the setup stage of a build (which we will absolutely be taking advantage of!). + +## A Closer Look at the clang Compiler's Performance + +A first cut of the compiler results were shared earlier in [Initial Performance Testing](/blog/2021/08/02/initial-performance-testing), and given the importance to overall build time, I've been taking a closer look. In that post, I said that + +> "At stages where I would have expected to be ahead already, the compile performance was only equal" + +and now I have identified the discrepancy. + +I've tested multiple configurations for the `clang` compiler and noticed that changing the default standard C++ library makes a difference to the time of this particular build. The difference in the two runs is compiling `llvm-ar` with the LLVM libraries of `compiler-rt/libc++/libunwind` or the GNU libraries of `libgcc/libstdc++`. And just to be clear, this is increasing the time of compiling `llvm-ar` with `libc++` vs `libstdc++` and not to do with the performance of either library. The `clang` compiler itself is built with `libc++` in both cases as it produces a faster compiler. + +![Comparison of compile times](./Featured.webp "Comparison of compile times") + +| Test using clang | Serpent LLVM libs | Serpent GNU libs | Host | +|-----------------------|-------------------|------------------|--------------| +| `cmake` LLVM | 5.89s | 5.67s | 10.58s | +| Compile -j4 `llvm-ar` | 126.16s | 112.51s | 155.32s | +| `configure` gettext | 36.64s | 36.98s | 63.55s | + +The host now takes `38% longer` than the Serpent OS clang when building with the same GNU libraries and is much more in line with my expectations. Next steps will be getting `bolt` and `perf` integrated into Serpent OS to see if we can shave even more time off the build. + +What remains unclear is whether this difference is due to something specifically in the `LLVM` build or whether it would translate to other C++ packages. I haven't noticed a 10% increase in build time when performing the full compiler build with `libc++` vs `libstdc++`. diff --git a/src/content/docs/blog/2021/09/20/unpacking-the-build-process-2/index.mdx b/src/content/docs/blog/2021/09/20/unpacking-the-build-process-2/index.mdx deleted file mode 100644 index 2f055b53..00000000 --- a/src/content/docs/blog/2021/09/20/unpacking-the-build-process-2/index.mdx +++ /dev/null @@ -1,98 +0,0 @@ ---- -title: "Unpacking the Build Process: Part 2" -date: 2021-09-20T16:04:12+10:00 -draft: false -authors: [sunnyflunk] -tags: [news] -featuredImage: "/img/blog/unpacking-the-build-process-2/Featured.webp" -license: "CC-BY-ND-4.0" -copyright: "Copyright © 2021 aerynOS Developers" ---- -import FigureScreenshotOne from "@/components/ui/FigureScreenshotOne.astro" -import featuredImage from "./Featured.webp" - -Part 2 looks at the core of the build process, turning the source into compiled code. In Serpent OS this is handled by -our build tool `boulder`. It is usually the part of the build that takes the longest, so where speed ups have the most -impact. How long it takes is largely down to the performance of your compiler and what compile flags you are building -with. - - - -This post follows on from [Part 1](/blog/2021/08/25/unpacking-the-build-process-1). - -## Turning Source into Compiled Code - -The steps for compiling code are generally quite straight-forward: -- Setting up the build (cmake, configure, meson) -- Compiling the source (in parallel threads) -- Installing the build into a package directory - -This will build software compiled against packages installed on your system. It's a bit more complicated when packaging -as we first set up an environment to compile in (Part 1). But even then you have many choices to make and each can have -an impact on how long it takes to compile the code. Do you build with Link Time Optimizations (LTO) or Profile Guided -Optimizations (PGO), do you build the package for performance or for the smallest size? Then there's packages that -benefit considerably from individual tuning flags (like `-fno-semantic-interposition` with `python`). With so many -possibilities, `boulder` helps us utilize them through convenient configuration options. - -## What Makes boulder so Special? - -As I do a lot of packaging and performance tuning, `boulder` is where I spend most of my time. Here are some key -features that `boulder` brings to make my life easier. - - - Ultimate control over build C/CXX/LDFLAGS via the tuning key - - Integrated 2 stage context sensitive PGO builds with a single line workload - - Able to switch between `gnu` and `llvm` toolchains easily - - Rules based package creation - - Control the extraction locations for multiple upstream tarballs - -`boulder` will also be used to generate and amend our `stone.yml` files to take care of as much as possible -automatically. This is only the beginning for `boulder` as it will continue to be expanded to learn new tricks to make -packaging more automated and able to bring more information to help packagers know when they can improve their -`stone.yml`, or alert them that something might be missing. - -Serpent OS is focused on the performance of produced packages, even if that means that builds will take longer to -complete. This is why we have put in significant efforts to speed up the compiler and setup tools in order to offset and -minimize the time needed to enable greater performance. - -## Why do You Care so Much About Setup Time? - -My initial testing focused on the performance of `clang` as well as the time taken to run `cmake` and `configure`. This -lays the foundation for all future work in expanding the Serpent OS package archives at a much faster pace. On the -surface, running `cmake` can be a small part of the overall build. However, it is important in that it utilizes a single -thread, so is not sped up by adding more CPU cores like the compile time is. With a more compile heavy build, our -highly tuned compiler can build the source in around 75s. So tuning the setup step to run in 5s rather than 10s actually -reduces the overall build time by an additional 6%! - -There are many smaller packages where the setup time is an even higher proportion of the overall build and becomes more -relevant as you increase the numbers of threads on the builder. For example, when building `nano` on the host, the -configure step takes 13.5s, while the build itself takes only 2.3s, so there's significant gains to be had from speeding -up the setup stage of a build (which we will absolutely be taking advantage of!). - -## A Closer Look at the clang Compiler's Performance - -A first cut of the compiler results were shared earlier in [Initial Performance Testing](/blog/2021/08/02/initial-performance-testing), -and given the importance to overall build time, I've been taking a closer look. In the post I said that `"At stages -where I would have expected to be ahead already, the compile performance was only equal"` and now I have identified the -discrepancy. - -I've tested multiple configurations for the `clang` compiler and noticed that changing the default standard C++ library -makes a difference to the time of this particular build. The difference in the two runs is compiling `llvm-ar` with the -LLVM libraries of `compiler-rt/libc++/libunwind` or the GNU libraries of `libgcc/libstdc++`. And just to be clear, this -is increasing the time of compiling `llvm-ar` with `libc++` vs `libstdc++` and not to do with the performance of either -library. The `clang` compiler itself is built with `libc++` in both cases as it produces a faster compiler. - - - -| Test using clang | Serpent LLVM libs | Serpent GNU libs | Host | -|-----------------------|-------------------|------------------|--------------| -| `cmake` LLVM | 5.89s | 5.67s | 10.58s | -| Compile -j4 `llvm-ar` | 126.16s | 112.51s | 155.32s | -| `configure` gettext | 36.64s | 36.98s | 63.55s | - -The host now takes `38% longer` than the Serpent OS clang when building with the same GNU libraries and is much more in -line with my expectations. Next steps will be getting `bolt` and `perf` integrated into Serpent OS to see if we can -shave even more time off the build. - -What remains unclear is whether this difference is due to something specifically in the `LLVM` build or whether it would -translate to other C++ packages. I haven't noticed a 10% increase in build time when performing the full compiler -build with `libc++` vs `libstdc++`. diff --git a/src/content/docs/blog/2021/10/04/optimal-file-locality/index.md b/src/content/docs/blog/2021/10/04/optimal-file-locality/index.md new file mode 100644 index 00000000..df1bcbce --- /dev/null +++ b/src/content/docs/blog/2021/10/04/optimal-file-locality/index.md @@ -0,0 +1,76 @@ +--- +title: "Optimal File Locality" +date: 2021-10-04T16:49:12+11:00 +draft: false +authors: [sunnyflunk] +tags: [news] +license: "CC-BY-ND-4.0" +copyright: "Copyright © 2021 aerynOS Developers" +--- + +File locality in this post refers to the order of files in our content payload. Yes that's right, we're focused on the small details and incremental improvements that combined add up to significant benefits! All of this came about from testing the efficiency of content payload in `moss-format` and how well it compared against a plain tarball. One day `boulder` was looking extremely inefficient and then retesting the following day was proving to be extremely efficient without any changes made to `boulder` or `moss-format`. What on Earth was going on? + +## Making Sure you Aren't Going Crazy! + +To test the efficiency our content payload, the natural choice was to compare it to a tarball containing the same files. When first running the test the results were quite frankly awful! Our payload was 10% larger than the equivalent tarball! It was almost unbelievable in a way, so the following day I repeated the test again only this time the content payload was smaller than the tarball. This didn't actually make sense, I made the tarball with the same files, but only changed the directory it was created from. Does it really matter? + +## File Locality Really Matters! + +Of course it does (otherwise it would be a pretty crappy blog post!). When extracting a `.stone` package it creates two directories, `mossExtract` where the sha256sum named files are stored and `mossInstall` where those files are hardlinked to their full path name. The first day I created the tarball from `mossInstall` and the second day I realised that creating the tarball from `mossExtract` would provide the closest match to the content payload since it was a direct comparison. When compressing the tarballs to match the `.stone` compression level, the tarball compressed from `mossInstall` was 10% smaller, despite the uncompressed tarball being slightly larger. + +## Compression Wants to Separate Apples and Oranges + +In simplistic terms, the way compression works is comparing data that it's currently reading versus data that it's read earlier in the file. `zstd` has some great options like `--long` that increases the distance in which these matches can be made at the cost of increased memory use. To limit memory use while making compression and decompression fast, it takes shortcuts that reduce the compression ratio. For optimal compression, you want files that are most similar to each other to be as close as possible. You won't get as many matches from a text file to an ELF file as you would from a similar looking text file. + +## Spot the Difference + +Files in `mossExtract` are listed in their sha256sum order, which is basically random, where files in `mossInstall` are ordered by their path. Sorting files by path actually does some semblance of sorting where binaries are in `/usr/bin` and libraries are in `/usr/lib` bringing them closer together. This is in no way a perfect order, but is a large improvement on a random order (up to 10% in our case!). + +Our `glibc` package has been an interesting test case for `boulder`, where an uncompressed tarball of the install directory was just under 1GB. As `boulder` stores files by their sha256sum, it is able to deduplicate files that are the same even when the build hasn't used symlinks or hardlinks to prevent the wasted space. In this case, the deduplication reduced the uncompressed size of the payload by 750MB alone (that's a lot of duplicate locale data!). In the `python` package, it removes 1,870 duplicate cache files to reduce the installation size. + +As part of the deduplication process `boulder` would sort files by sha256sum to remove duplicate hashes. If two files have the same sha256sum, then only one copy needs to be stored. It also felt clean with the output of `moss info` looking nice where the hashes are listed in alphabetical order. But it was having a significant negative impact on package sizes so that needed to be addressed by resorting the files by path order (a simple one-liner), making the content payload more efficient than a tarball once again. + +| Compression Level | sha256sum Order | File path Order | +|:-----------------:|------------------|--------------------| +| 1 | 72,724,389 | 70,924,858 | +| 6 | 65,544,322 | 63,372,056 | +| 12 | 49,066,505 | 44,039,782 | +| 16 | 45,365,415 | 40,785,385 | +| 19 | 26,643,334 | 24,134,820 | +| 22 | 16,013,048 | 15,504,806 | + +Testing has shown that higher compression levels (and enabling `--long`) is more forgiving of a suboptimal file order (3-11% smaller vs only 2-5% smaller when using `--long`). The table above is without `--long` so the difference is larger. + +## Hang On, Why Don't You... + +There's certainly something to this and sorting by file order is a first step. In future we can consider creating an efficient order for files to improve locality. Putting all the ELF, image or text files together in the payload will help to shave a bit off our package sizes at only the cost to sort the files. However, we don't want to go crazy here, the biggest impact on reducing package sizes will be using deltas as the optimal package delivery system (and there will be a followup on this approach shortly). The `moss-format` content payload is quite simple and contains no filenames or paths in it. Therefore it's effectively costless to switch around the order of files, so we can try out a few things and see what happens. + + +## An Academic Experiment + +To prove the value of `moss-format` and the content payload, I tried out some crude sorting methods and their impact on compression for the package. As you want similar files chunked together, it divided the files into 4 groups, still sorted by their path order in their corresponding chunk: + +- **gz:** gzipped files +- **data:** non-text files that weren't ELF +- **elf:** ELF files +- **text:** text files (bash scripts, perl etc) + +![Path order vs optimal order](./Featured.webp "Path order vs optimal order") + +As the chart shows, you can get some decent improvements from reordering files within the tarball when grouping files in logical chunks. At the highest compression level, the package is reduced by 0.83% without any impact on compression or decompression time. In the compression world, such a change would be greatly celebrated! + +Also important to note was that just moving the gzipped files to the front of the payload was able to capture 40% of the size improvement at high compression levels, but had slightly worse compression at levels 1-4. So simple changes to the order (in this case moving non-compressible files to the edge of the payload) can provide a reduction in size at the higher levels that we care about. We don't want to spend a long time analyzing files for a small reduction in package size, so we can start off with some basic concepts like this. Moving files that don't compress a lot such as already compressed files, images and video to the start of payload meaning that the remaining files are closer together. We also need to test out a broader range of packages and the impact any changes would have on them. + +## Food For Thought? + +So ultimately the answer to the original question (was `moss-format` efficient?), the answer is yes! While there are some things that we still want to change to make it even better, in its current state package creation time was faster and overheads were lower than with compressing an equivalent tarball. The compressed tarball at `zstd -16` was 700KB larger than the full `.stone` file (which contains a bit more data than the tarball). + +The unique format also proves its worth in that we can make further adjustments to increase performance, reduce memory requirements and reduce package sizes. What this experiment shows is that file order really does matter, but using the basic sorting method of filepath gets you most of the way there and is likely good enough for most cases. + +Here are some questions we can explore in future to see whether there's greater value in tweaking the file order: + +- Do we sort ELF files by path order, file name or by size? +- Does it matter the order of chunks in the file? (i.e. ELF-Images-Text vs Images-Text-ELF) +- How many categories do we need to segregate and order? +- Can we sort by extension? (i.e. for images, all the png files will be together and the jpegs together) +- Do we simply make a couple of obvious changes to order and leave `zstd` to do the heavy lifting? diff --git a/src/content/docs/blog/2021/10/04/optimal-file-locality/index.mdx b/src/content/docs/blog/2021/10/04/optimal-file-locality/index.mdx deleted file mode 100644 index d7240723..00000000 --- a/src/content/docs/blog/2021/10/04/optimal-file-locality/index.mdx +++ /dev/null @@ -1,129 +0,0 @@ ---- -title: "Optimal File Locality" -date: 2021-10-04T16:49:12+11:00 -draft: false -authors: [sunnyflunk] -tags: [news] -license: "CC-BY-ND-4.0" -copyright: "Copyright © 2021 aerynOS Developers" ---- - -File locality in this post refers to the order of files in our content payload. Yes that's right, we're focused on the -small details and incremental improvements that combined add up to significant benefits! All of this came about from -testing the efficiency of content payload in `moss-format` and how well it compared against a plain tarball. One day -`boulder` was looking extremely inefficient and then retesting the following day was proving to be extremely efficient -without any changes made to `boulder` or `moss-format`. What on Earth was going on? - - - -## Making Sure you Aren't Going Crazy! - -To test the efficiency our content payload, the natural choice was to compare it to a tarball containing the same files. -When first running the test the results were quite frankly awful! Our payload was 10% larger than the equivalent -tarball! It was almost unbelievable in a way, so the following day I repeated the test again only this time the content -payload was smaller than the tarball. This didn't actually make sense, I made the tarball with the same files, but -only changed the directory it was created from. Does it really matter? - -## File Locality Really Matters! - -Of course it does (otherwise it would be a pretty crappy blog post!). When extracting a `.stone` package it creates two -directories, `mossExtract` where the sha256sum named files are stored and `mossInstall` where those files are -hardlinked to their full path name. The first day I created the tarball from `mossInstall` and the second day I -realised that creating the tarball from `mossExtract` would provide the closest match to the content payload since it -was a direct comparison. When compressing the tarballs to match the `.stone` compression level, the tarball compressed -from `mossInstall` was 10% smaller, despite the uncompressed tarball being slightly larger. - -## Compression Wants to Separate Apples and Oranges - -In simplistic terms, the way compression works is comparing data that it's currently reading versus data that it's read -earlier in the file. `zstd` has some great options like `--long` that increases the distance in which these matches can -be made at the cost of increased memory use. To limit memory use while making compression and decompression fast, it -takes shortcuts that reduce the compression ratio. For optimal compression, you want files that are most similar to -each other to be as close as possible. You won't get as many matches from a text file to an ELF file as you would from a -similar looking text file. - -## Spot the Difference - -Files in `mossExtract` are listed in their sha256sum order, which is basically random, where files in `mossInstall` are -ordered by their path. Sorting files by path actually does some semblance of sorting where binaries are in `/usr/bin` -and libraries are in `/usr/lib` bringing them closer together. This is in no way a perfect order, but is a large -improvement on a random order (up to 10% in our case!). - -Our `glibc` package has been an interesting test case for `boulder`, where an uncompressed tarball of the install -directory was just under 1GB. As `boulder` stores files by their sha256sum, it is able to deduplicate files that -are the same even when the build hasn't used symlinks or hardlinks to prevent the wasted space. In this case, the -deduplication reduced the uncompressed size of the payload by 750MB alone (that's a lot of duplicate locale data!). In -the `python` package, it removes 1,870 duplicate cache files to reduce the installation size. - -As part of the deduplication process `boulder` would sort files by sha256sum to remove duplicate hashes. If two files -have the same sha256sum, then only one copy needs to be stored. It also felt clean with the output of `moss info` -looking nice where the hashes are listed in alphabetical order. But it was having a significant negative impact on -package sizes so that needed to be addressed by resorting the files by path order (a simple one-liner), making the -content payload more efficient than a tarball once again. - -| Compression Level | sha256sum Order | File path Order | -|:-----------------:|------------------|--------------------| -| 1 | 72,724,389 | 70,924,858 | -| 6 | 65,544,322 | 63,372,056 | -| 12 | 49,066,505 | 44,039,782 | -| 16 | 45,365,415 | 40,785,385 | -| 19 | 26,643,334 | 24,134,820 | -| 22 | 16,013,048 | 15,504,806 | - -Testing has shown that higher compression levels (and enabling `--long`) is more forgiving of a suboptimal file order -(3-11% smaller vs only 2-5% smaller when using `--long`). The table above is without `--long` so the difference is -larger. - -## Hang On, Why Don't You... - -There's certainly something to this and sorting by file order is a first step. In future we can consider creating an -efficient order for files to improve locality. Putting all the ELF, image or text files together in the payload will -help to shave a bit off our package sizes at only the cost to sort the files. However, we don't want to go crazy here, -the biggest impact on reducing package sizes will be using deltas as the optimal package delivery system (and there will -be a followup on this approach shortly). The `moss-format` content payload is quite simple and contains no filenames or -paths in it. Therefore it's effectively costless to switch around the order of files, so we can try out a few things and -see what happens. - -## An Academic Experiment - -To prove the value of `moss-format` and the content payload, I tried out some crude sorting methods and their impact on -compression for the package. As you want similar files chunked together, it divided the files into 4 groups, still -sorted by their path order in their corresponding chunk: - -- **gz:** gzipped files -- **data:** non-text files that weren't ELF -- **elf:** ELF files -- **text:** text files (bash scripts, perl etc) - -![Path order vs optimal order](./Featured.webp) - -As the chart shows, you can get some decent improvements from reordering files within the tarball when grouping files -in logical chunks. At the highest compression level, the package is reduced by 0.83% without any impact on compression -or decompression time. In the compression world, such a change would be greatly celebrated! - -Also important to note was that just moving the gzipped files to the front of the payload was able to capture 40% of the -size improvement at high compression levels, but had slightly worse compression at levels 1-4. So simple changes to the -order (in this case moving non-compressible files to the edge of the payload) can provide a reduction in size at the -higher levels that we care about. We don't want to spend a long time analyzing files for a small reduction in package -size, so we can start off with some basic concepts like this. Moving files that don't compress a lot such as already -compressed files, images and video to the start of payload meaning that the remaining files are closer together. We also -need to test out a broader range of packages and the impact any changes would have on them. - -## Food For Thought? - -So ultimately the answer to the original question (was `moss-format` efficient?), the answer is yes! While there are -some things that we still want to change to make it even better, in its current state package creation time was faster -and overheads were lower than with compressing an equivalent tarball. The compressed tarball at `zstd -16` was 700KB -larger than the full `.stone` file (which contains a bit more data than the tarball). - -The unique format also proves its worth in that we can make further adjustments to increase performance, reduce memory -requirements and reduce package sizes. What this experiment shows is that file order really does matter, but using the -basic sorting method of filepath gets you most of the way there and is likely good enough for most cases. - -Here are some questions we can explore in future to see whether there's greater value in tweaking the file order: - -- Do we sort ELF files by path order, file name or by size? -- Does it matter the order of chunks in the file? (i.e. ELF-Images-Text vs Images-Text-ELF) -- How many categories do we need to segregate and order? -- Can we sort by extension? (i.e. for images, all the png files will be together and the jpegs together) -- Do we simply make a couple of obvious changes to order and leave `zstd` to do the heavy lifting? diff --git a/src/content/docs/blog/2021/11/05/performance-corner-faster-builds-smaller-packages/index.md b/src/content/docs/blog/2021/11/05/performance-corner-faster-builds-smaller-packages/index.md new file mode 100644 index 00000000..af6b04d5 --- /dev/null +++ b/src/content/docs/blog/2021/11/05/performance-corner-faster-builds-smaller-packages/index.md @@ -0,0 +1,82 @@ +--- +title: "Performance Corner: Faster Builds, Smaller Packages" +date: 2021-11-05T19:23:32+11:00 +draft: false +authors: [sunnyflunk] +tags: [news] +license: "CC-BY-ND-4.0" +copyright: "Copyright © 2021 aerynOS Developers" +--- + +`Performance Corner` is a new series where we highlight to you some changes in Serpent OS that may not be obvious, but show a real improvement. Performance is a broad term that also covers efficiency, so things like making files smaller, making things faster or reducing power consumption. In general things that are unquestionably improvements with little or no downside. While the technical details may be of interest to some, the main purpose is to highlight the real benefit to users and/or developers that will make using Serpent OS a more enjoyable experience. Show me the numbers! + +Here we focus on a few performance changes `Ikey` has been working on to the build process that are showing some pretty awesome results! If you end up doing any source builds, you'll be thankful for these improvements. Special thanks to `ermo` for the research into hash algorithms and enabling our ELF processing. + +## Start From the Beginning + +When measuring changes, it's always important to know where you're starting from. Here are some results from a recent `glibc` build, but before these latest changes were incorporated. + +``` +Payload: Layout [Records: 5441 Compression: Zstd, Savings: 83.13%, Size: 673.46 KB] +Payload: Index [Records: 2550 Compression: Zstd, Savings: 55.08%, Size: 247.35 KB] +Payload: Content [Records: 2550 Compression: Zstd, Savings: 81.46%, Size: 236.72 MB] + ==> 'BuildState.Build' finished [4 minutes, 6 secs, 136 ms, 464 μs, and 7 hnsecs] + ==> 'BuildState.Analyse' finished [21 secs, 235 ms, 300 μs, and 2 hnsecs] + ==> 'BuildState.ProducePackages' finished [25 secs, 624 ms, 996 μs, and 8 hnsecs] +``` + +The build time is a little high, but a lot of that is due to a slow compiler on the host machine. But analysing and producing packages was also taking a lot longer than it needed to. + +## The Death of moss-jobs in boulder + +In testing an equivalent build outside of `boulder`, the build stages were about 5% faster. Testing under `perf`, the jobs system was a bit excessive for the needs of `boulder`, polling for work when we already know the times when parallel jobs would be useful. Removing `moss-jobs` allowed for simpler codepaths using multiprocessing techniques from the core language. This work is integrated in `moss-deps` and the excess overhead of the build has now been eliminated. + +``` +Before: + ==> 'BuildState.Build' finished [4 minutes, 6 secs, 136 ms, 464 μs, and 7 hnsecs] + ==> 'BuildState.Analyse' finished [21 secs, 235 ms, 300 μs, and 2 hnsecs] + +After: +[Build] Finished: 3 minutes, 53 secs, 386 ms, 306 μs, and 4 hnsecs +[Analyse] Finished: 8 secs, 136 ms, 22 μs, and 8 hnsecs +``` + +The new results reflect a 26s reduction in the overall build time. But only 13s of this relates to the `moss-jobs` removal. The other major change is making the analyse stage parallel in `moss-deps` (a key part of why we wanted parallelism to begin with). Decreasing the time from 21.2s to 8.1s is a great achievement despite it doing more work as we've also added ELF scanning for dependency information in-between these results. + +## New Hashing Algorithm + +One of the unique features in `moss` is using hashes for file names which allows full deduplication within packages, the running system, previous system roots and for source builds with `boulder`. Initially this was hooked up using `sha256`, but it was proving to be a bit of a slowdown. + +Enter `xxhash`, the hash algorithm by Yann Collet for use in fast decompression software such as `lz4` and `zstd` (and now in many places!). This is seriously fast, with the potential to produce hashes faster than RAM can feed the CPU. The hash is merely used as a unique identifier in the context of deduplication, not a cryptographic verification of origin. `XXH3_128bit` has been chosen due to it having an almost zero probability of a collision across 10s of millions of files. + +The benefit is actually two-fold. First of all, the hash length is halved from `sha256`, so there's savings in the package metadata. This shouldn't be understated as hash data is generally not as compressible as typical text and there are packages with a lot of files! Here the metadata for the Layout and Index payloads has reduced by 232KB! That's about a 25% reduction with no other changes. + +``` +Before: +Payload: Layout [Records: 5441 Compression: Zstd, Savings: 83.13%, Size: 673.46 KB] +Payload: Index [Records: 2550 Compression: Zstd, Savings: 55.08%, Size: 247.35 KB] + +After: +Payload: Layout [Records: 5441 Compression: Zstd, Savings: 86.66%, Size: 522.97 KB] +Payload: Index [Records: 2550 Compression: Zstd, Savings: 60.02%, Size: 165.75 KB] +``` + +Compressed this turns out to be about a 89KB reduction in the package size. For larger packages, this probably doesn't mean much but could help a lot more with delta packages. For deltas, we will be including the full metadata of the Layout and Index payloads, so the difference will be more significant there. + +The other benefit of course is the speed and the numbers speak for themselves! A further 6.4s reduction in build time removing most of the delay at the end of the build for the final package. This will also improve speeds for caching or validating a package. + +``` +Before: + ==> 'BuildState.Analyse' finished [21 secs, 235 ms, 300 μs, and 2 hnsecs] + +After: +[Analyse] Finished: 1 sec, 688 ms, 681 μs, and 8 hnsecs +``` + +With these changes combined, building packages can take 12x less time in the analyse stage, while reducing the size of the metadata and the overall package. We do expect the analyse time to increase in future as we add more dependency types, debug handling and stripping, but with the integrated parallel model, we can minimize the increase in time. + +![Building](./Featured.webp "Building a package") + +## We're Not Done Yet + +The first installment of `Performance Corner` shows some great wins to the Serpent OS tools and architecture. This is just the beginning and there will likely be a follow up soon (you may have also noticed that it takes too long to make the packages), and there's a couple more tweaks to further decrease the size of the metadata. Kudos to `Ikey` for getting these implemented! diff --git a/src/content/docs/blog/2021/11/05/performance-corner-faster-builds-smaller-packages/index.mdx b/src/content/docs/blog/2021/11/05/performance-corner-faster-builds-smaller-packages/index.mdx deleted file mode 100644 index c84c9622..00000000 --- a/src/content/docs/blog/2021/11/05/performance-corner-faster-builds-smaller-packages/index.mdx +++ /dev/null @@ -1,113 +0,0 @@ ---- -title: "Performance Corner: Faster Builds, Smaller Packages" -date: 2021-11-05T19:23:32+11:00 -draft: false -authors: [sunnyflunk] -tags: [news] -license: "CC-BY-ND-4.0" -copyright: "Copyright © 2021 aerynOS Developers" ---- - -`Performance Corner` is a new series where we highlight to you some changes in Serpent OS that may not be obvious, but -show a real improvement. Performance is a broad term that also covers efficiency, so things like making files smaller, -making things faster or reducing power consumption. In general things that are unquestionably improvements with little -or no downside. While the technical details may be of interest to some, the main purpose is to highlight the real -benefit to users and/or developers that will make using Serpent OS a more enjoyable experience. Show me the numbers! - - - -Here we focus on a few performance changes `Ikey` has been working on to the build process that are showing some pretty -awesome results! If you end up doing any source builds, you'll be thankful for these improvements. Special thanks to -`ermo` for the research into hash algorithms and enabling our ELF processing. - -## Start From the Beginning - -When measuring changes, it's always important to know where you're starting from. Here are some results from a recent -`glibc` build, but before these latest changes were incorporated. - -``` -Payload: Layout [Records: 5441 Compression: Zstd, Savings: 83.13%, Size: 673.46 KB] -Payload: Index [Records: 2550 Compression: Zstd, Savings: 55.08%, Size: 247.35 KB] -Payload: Content [Records: 2550 Compression: Zstd, Savings: 81.46%, Size: 236.72 MB] - ==> 'BuildState.Build' finished [4 minutes, 6 secs, 136 ms, 464 μs, and 7 hnsecs] - ==> 'BuildState.Analyse' finished [21 secs, 235 ms, 300 μs, and 2 hnsecs] - ==> 'BuildState.ProducePackages' finished [25 secs, 624 ms, 996 μs, and 8 hnsecs] -``` - -The build time is a little high, but a lot of that is due to a slow compiler on the host machine. But analysing and -producing packages was also taking a lot longer than it needed to. - -## The Death of moss-jobs in boulder - -In testing an equivalent build outside of `boulder`, the build stages were about 5% faster. Testing under `perf`, the -jobs system was a bit excessive for the needs of `boulder`, polling for work when we already know the times when -parallel jobs would be useful. Removing `moss-jobs` allowed for simpler codepaths using multiprocessing techniques from -the core language. This work is integrated in `moss-deps` and the excess overhead of the build has now been eliminated. - -``` -Before: - ==> 'BuildState.Build' finished [4 minutes, 6 secs, 136 ms, 464 μs, and 7 hnsecs] - ==> 'BuildState.Analyse' finished [21 secs, 235 ms, 300 μs, and 2 hnsecs] -After: -[Build] Finished: 3 minutes, 53 secs, 386 ms, 306 μs, and 4 hnsecs -[Analyse] Finished: 8 secs, 136 ms, 22 μs, and 8 hnsecs -``` - -The new results reflect a 26s reduction in the overall build time. But only 13s of this relates to the `moss-jobs` -removal. The other major change is making the analyse stage parallel in `moss-deps` (a key part of why we wanted -parallelism to begin with). Decreasing the time from 21.2s to 8.1s is a great achievement despite it doing more work as -we've also added ELF scanning for dependency information in-between these results. - -## New Hashing Algorithm - -One of the unique features in `moss` is using hashes for file names which allows full deduplication within packages, -the running system, previous system roots and for source builds with `boulder`. Initially this was hooked up using -`sha256`, but it was proving to be a bit of a slowdown. - -Enter `xxhash`, the hash algorithm by Yann Collet for use in fast decompression software such as `lz4` and `zstd` (and -now in many places!). This is seriously fast, with the potential to produce hashes faster than RAM can feed the CPU. The -hash is merely used as a unique identifier in the context of deduplication, not a cryptographic verification of origin. -`XXH3_128bit` has been chosen due to it having an almost zero probability of a collision across 10s of millions of -files. - -The benefit is actually two-fold. First of all, the hash length is halved from `sha256`, so there's savings in the -package metadata. This shouldn't be understated as hash data is generally not as compressible as typical text and there -are packages with a lot of files! Here the metadata for the Layout and Index payloads has reduced by 232KB! That's about -a 25% reduction with no other changes. - -``` -Before: -Payload: Layout [Records: 5441 Compression: Zstd, Savings: 83.13%, Size: 673.46 KB] -Payload: Index [Records: 2550 Compression: Zstd, Savings: 55.08%, Size: 247.35 KB] -After: -Payload: Layout [Records: 5441 Compression: Zstd, Savings: 86.66%, Size: 522.97 KB] -Payload: Index [Records: 2550 Compression: Zstd, Savings: 60.02%, Size: 165.75 KB] -``` - -Compressed this turns out to be about a 89KB reduction in the package size. For larger packages, this probably doesn't -mean much but could help a lot more with delta packages. For deltas, we will be including the full metadata of the -Layout and Index payloads, so the difference will be more significant there. - -The other benefit of course is the speed and the numbers speak for themselves! A further 6.4s reduction in build time -removing most of the delay at the end of the build for the final package. This will also improve speeds for caching or -validating a package. - -``` -Before: - ==> 'BuildState.Analyse' finished [21 secs, 235 ms, 300 μs, and 2 hnsecs] -After: -[Analyse] Finished: 1 sec, 688 ms, 681 μs, and 8 hnsecs -``` - -With these changes combined, building packages can take 12x less time in the analyse stage, while reducing the size of -the metadata and the overall package. We do expect the analyse time to increase in future as we add more dependency -types, debug handling and stripping, but with the integrated parallel model, we can minimize the increase in time. - -![Building](./Featured.webp) - -## We're Not Done Yet - -The first installment of `Performance Corner` shows some great wins to the Serpent OS tools and architecture. This is -just the beginning and there will likely be a follow up soon (you may have also noticed that it takes too long to make -the packages), and there's a couple more tweaks to further decrease the size of the metadata. Kudos to `Ikey` for -getting these implemented! diff --git a/src/content/docs/blog/2021/11/23/it-all-depends/index.md b/src/content/docs/blog/2021/11/23/it-all-depends/index.md index 50a96ad9..d8adf30d 100644 --- a/src/content/docs/blog/2021/11/23/it-all-depends/index.md +++ b/src/content/docs/blog/2021/11/23/it-all-depends/index.md @@ -8,77 +8,45 @@ license: "CC-BY-ND-4.0" copyright: "Copyright © 2021 aerynOS Developers" --- -It all depends.. it really does. On shared libraries.. interpreters.. `pkg-config` providers -and packages. It's the same story for all "package managers", how do we ensure that the -installed software has everything it needs at runtime? +It all depends.. it really does. On shared libraries.. interpreters.. `pkg-config` providers and packages. It's the same story for all "package managers", how do we ensure that the installed software has everything it needs at runtime? -Our merry crew has been involved in designing and building Linux distributions for a very, very -long time, so we didn't want to simply repeat history. +Our merry crew has been involved in designing and building Linux distributions for a very, very long time, so we didn't want to simply repeat history. - +![Using updated moss](./Featured.webp "Using an updated moss") -![Using updated moss](./Featured.webp) +Thanks to many improvements across our codebase, including `moss-deps`, we automatically analyse the assets in a package (rapidly too) to determine any dependencies we can add without requiring the maintainer to list them. This is similar in nature to RPM's behaviour. +As such we encode dependencies into our (endian-aware, binary) format which is then stored in the local installation database. Global providers are keyed for quick access, and the vast majority of packages will not explicitly depend on another package's *name*, rather, they'll depend on a capability or provider. For subpackage dependencies that usually depend on "NEVRA" equality (i.e. matching against a name with a minimum version, release, matching epoch and architecture), we'll introduce a lockstep dependency that can only be resolved from its origin source (repo). -Thanks to many improvements across our codebase, including `moss-deps`, we automatically analyse -the assets in a package (rapidly too) to determine any dependencies we can add without requiring -the maintainer to list them. This is similar in nature to RPM's behaviour. - -As such we encode dependencies into our (endian-aware, binary) format which is then stored in the -local installation database. Global providers are keyed for quick access, and the vast majority -of packages will not explicitly depend on another package's *name*, rather, they'll depend on -a capability or provider. For subpackage dependencies that usually depend on "NEVRA" equality -(i.e. matching against a name with a minimum version, release, matching epoch and architecture), -we'll introduce a lockstep dependency that can only be resolved from its origin source (repo). - -Lastly, we'll always ensure there is no possibility for "partial update" woes. With these -considerations, we have no need to support `>=` style dependencies, and instead rely on -strict design goals and maintainer responsibility. +Lastly, we'll always ensure there is no possibility for "partial update" woes. With these considerations, we have no need to support `>=` style dependencies, and instead rely on strict design goals and maintainer responsibility. ## First and foremost! -The rapid move we're enjoying from concept, to prototype, and soon to be fully fledged Linux distribution, -is only possible with the amazing community support. The last few months have seen us pull off some amazing -feats, and we're now executing on our first public milestones. With your help, more and more hours can be -spent getting us ready for release, and would probably help to insulate my shed office! (Spoiler: its -plastic and electric heaters are expensive =)) +The rapid move we're enjoying from concept, to prototype, and soon to be fully fledged Linux distribution, is only possible with the amazing community support. The last few months have seen us pull off some amazing feats, and we're now executing on our first public milestones. With your help, more and more hours can be spent getting us ready for release, and would probably help to insulate my shed office! (Spoiler: its plastic and electric heaters are expensive 😀) ## The Milestones -We have created our initial milestones that our quite literally our escape trajectory from -bootstrap to distro. We're considerably closer now, hence this open announcement. +We have created our initial milestones that our quite literally our escape trajectory from bootstrap to distro. We're considerably closer now, hence this open announcement. -## [v0.0](https://gitlab.com/groups/serpent-os/-/milestones/1]): Container (`systemd-nspawn`) +### [v0.0](https://gitlab.com/groups/serpent-os/-/milestones/1]): Container (`systemd-nspawn`) -Our first release medium will be a `systemd-nspawn` compatible container image. Our primary -driver for this is to allow us to add encapsulation for our build tool, `boulder`, permitting -us to create distributable builder images to seed our infrastructure and first public binary -repository. +Our first release medium will be a `systemd-nspawn` compatible container image. Our primary driver for this is to allow us to add encapsulation for our build tool, `boulder`, permitting us to create distributable builder images to seed our infrastructure and first public binary repository. -## [v0.1](https://gitlab.com/groups/serpent-os/-/milestones/2): Bootable "image" +### [v0.1](https://gitlab.com/groups/serpent-os/-/milestones/2): Bootable "image" -Once our build infra is up and running (honestly a lot of work has been completed for this in -advance) we'll work towards our first 0.1 image. This will initially target VM usage, with -a basic console environment and tooling (`moss`, `boulder`, etc). +Once our build infra is up and running (honestly a lot of work has been completed for this in advance) we'll work towards our first 0.1 image. This will initially target VM usage, with a basic console environment and tooling (`moss`, `boulder`, etc). -## And then.. +### And then.. -We have a clear linear path ahead of us, with each stage unlocking the next. During the development -of `v0.0` and `v0.1` we'll establish our build and test infrastructure, and begin hosting our -package sources and binaries. At this point we can enter a rapid development cycle with -incremental, and considerable improvements. Such as a usable desktop experience and installer.. :) +We have a clear linear path ahead of us, with each stage unlocking the next. During the development of `v0.0` and `v0.1` we'll establish our build and test infrastructure, and begin hosting our package sources and binaries. At this point we can enter a rapid development cycle with incremental, and considerable improvements. Such as a usable desktop experience and installer.. :) ## Recent changes -I haven't blogged in quite a while, as I've been deep in the trenches working on our core features. -As we've expressed before, we tend to work on the more complex systems *first* and then glue them -together after to form a cohesive hole. The last few days have involved plenty of glue, and we now -have distinct package management features. +I haven't blogged in quite a while, as I've been deep in the trenches working on our core features. As we've expressed before, we tend to work on the more complex systems *first* and then glue them together after to form a cohesive hole. The last few days have involved plenty of glue, and we now have distinct package management features. ## Refactor - - Replaced defunct InstallDB with reusable MetaDB for local installation of archives as well as - forming the backbone of repository support. + - Replaced defunct InstallDB with reusable MetaDB for local installation of archives as well as forming the backbone of repository support. - Added `ActivePackagesPlugin` to identify installed packages - Swapped non cryptographic hash usage with `xxhash` @@ -86,37 +54,27 @@ have distinct package management features. - Introduced new Transaction type to utilise a directed acyclical graph for dependency solving. - Reworked moss-deps into plugins + registry core for all resolution operations. - - Locally provided `.stone` files handled by `CobblePlugin` to ensure we depsolve from this - set too. - - New Transaction set management for identifying state changes and ensuring full resolution - of target system state. - - Shared library and interpreter (DT_INTERP) dependencies and producers automatically encoded - into packages and resolved by depsolver. - + - Locally provided `.stone` files handled by `CobblePlugin` to ensure we depsolve from this set too. + - New Transaction set management for identifying state changes and ensuring full resolution of target system state. + - Shared library and interpreter (DT_INTERP) dependencies and producers automatically encoded into packages and resolved by depsolver. ## Package Installation -We handle locally provided `.stone` packages passed to the `install` command identically to -those found in a repository. This eliminates a lot of special casing for local archives and -allows us to find dependencies within the provided set, before looking to the system and the -repositories. +We handle locally provided `.stone` packages passed to the `install` command identically to those found in a repository. This eliminates a lot of special casing for local archives and allows us to find dependencies within the provided set, before looking to the system and the repositories. ![Install](./Install.webp "Installing packages") -Dependency resolution is performed now for our package installation and is validated at -multiple points, allowing a package like nano to depend on compact automatic dependencies: +Dependency resolution is performed now for our package installation and is validated at multiple points, allowing a package like nano to depend on compact automatic dependencies: ```d Dependency(DependencyType.SharedLibraryName, "libc.so.6(x86_64)"); ``` -Note our format and database are binary and endian aware. The dependency type only requires -1 byte of storage and no string comparisons. +Note our format and database are binary and endian aware. The dependency type only requires 1 byte of storage and no string comparisons. ## List packages -Thanks to the huge refactor, we can now trivially access the installed packages as a list. -This code will be reused for a `list available` command in future. +Thanks to the huge refactor, we can now trivially access the installed packages as a list. This code will be reused for a `list available` command in future. Example `list installed` output: @@ -132,30 +90,20 @@ Example `list installed` output: ## Inspect archives -For debugging and development purposes, we've moved our old "info" command to a new -"inspect" command to work directly on local `.stone` files. This displays extended -information on the various payloads and their compression stats. +For debugging and development purposes, we've moved our old "info" command to a new "inspect" command to work directly on local `.stone` files. This displays extended information on the various payloads and their compression stats. -For general users - the new `info` command displays basic metadata and package -dependencies. +For general users - the new `info` command displays basic metadata and package dependencies. ![Info](./Info.webp "Display package information") ## Package Removal -Upon generating a new system state, "removed" packages are simply no longer installed. As such -no live mutation is performed. As of today we can now request the removal of packages from the -current state, which generates a new filtered state. Additionally we remove all reverse dependencies, -direct and transitive. This is accomplished by utilising a transposed copy of the directed acyclical -graph, identifying the relevant subgraph and occluding the set from the newly generated state. +Upon generating a new system state, "removed" packages are simply no longer installed. As such no live mutation is performed. As of today we can now request the removal of packages from the current state, which generates a new filtered state. Additionally we remove all reverse dependencies, direct and transitive. This is accomplished by utilising a transposed copy of the directed acyclical graph, identifying the relevant subgraph and occluding the set from the newly generated state. ![Remove](./Remove.webp "Remove packages") ## Lastly -The past few weeks have been especially enjoyable. I've truly had a fantastic time working on the project -and cannot wait for the team and I to start offering our first downloads, and iterate as a truly new -Linux distribution that borrows some ideas from a lot of great places, and fuses them into something -awesome. +The past few weeks have been especially enjoyable. I've truly had a fantastic time working on the project and cannot wait for the team and I to start offering our first downloads, and iterate as a truly new Linux distribution that borrows some ideas from a lot of great places, and fuses them into something awesome. Keep toasty - this train isn't slowing down. diff --git a/src/content/docs/blog/2021/12/02/out-of-the-bootstrap-towards-serpent-os/index.md b/src/content/docs/blog/2021/12/02/out-of-the-bootstrap-towards-serpent-os/index.md new file mode 100644 index 00000000..df301549 --- /dev/null +++ b/src/content/docs/blog/2021/12/02/out-of-the-bootstrap-towards-serpent-os/index.md @@ -0,0 +1,45 @@ +--- +title: "Out of the Bootstrap - Towards Serpent OS" +date: 2021-12-02T18:07:12+11:00 +draft: false +authors: [sunnyflunk] +tags: [news] +license: "CC-BY-ND-4.0" +copyright: "Copyright © 2021 aerynOS Developers" +--- + +The initial `stone` packages that will seed the first Serpent OS repo have now been finalized! This means that work towards setting up the infrastructure for live package updates begins now. We plan on taking time to streamline the processes with a focus of fixing issues at the source. In this way we can make packaging fast and efficient so we can spend time working on features rather than package updates. + +## The End of Bootstrap + +Bootstrapping a distribution involves building a new toolchain and many packages needed to support it. For us bootstrap was getting us to the point where we have built `stone` packages that we can use to start an initial repository with full working dependencies. This has been enabled by integrating dependencies into `moss`, creating the first repo index. Of note is that it is already enabled for 32bit support, so we have you covered there. While this is the end of bootstrap, the fun has only just begun! + +![The first install from the bootstrap](./Featured.webp "The first install from the bootstrap") + +## Where to Next? + +The next goal is to make Serpent OS self-hosting, where we can build packages in a container and update the repo index with the newly built packages. It is essentially a live repository accessible from the internet. There's still plenty of improvements to be made with the tooling, but will soon enable more users to participate in bringing Serpent OS into fruition. + +## Becoming More Inclusive + +While there's a strong focus in Serpent OS on performance, the decision has been made to lower the initial requirements for users. Despite AVX2 being an older technology, there are still computers sold today that don't support it. Because of this (and already having interested users who don't have newer machines), the baseline requirement for Serpent OS will be `x86_64-v2`, which only requires SSE4.2. + +It was always the plan to add support for these machines, just later down the track. In reality, this makes a lot more sense, as there will be many cases where building 2 versions of a package provides little value. This is where a package takes a long time to build and doesn't result in a notable performance improvement. We will always need the `x86_64-v2` version of a package to be compatible with the older machines. With this approach we can reduce the build server requirements without a noticeable impact to users as only a few packages you use will be without extra optimizations (and probably don't benefit from them anyway). + +I want to make it clear that this will be temporary, with impactful `x86_64-v3+` packages rolling out as soon as possible. This change paves the way to integrate our technology stack taking care of your system for you and increases its priority. Users meeting the requirements of the `x86_64-v3+` instruction set (this includes additional instructions beyond `x86_64-v3`) will automatically start installing these faster packages as they become available. Our `subscriptions` model will seamlessly take care of everything for you behind the scenes so you don't need to read a wiki or forum to learn how to get these faster packages. We can utilize the same approach in future for our `ARM` stack, offering more optimized packages where it provides the most benefit. + +Note that from the bootstrap, most packages built in under 15s and only three took longer than 2 minutes. + +## Trying Out Some Tweaks From the Get Go + +While the project is young is a great time to test out new technologies. The use of `clang` and `lld` open up new possibilities to reduce file sizes, shorten load times and decrease memory usage. Some of these choices may have impacts on compatibility, so testing them out will be the best way to grasp that. Making sure that you can run apps like Steam is vital to the experience, so whatever happens we will make sure it works. The good news is that due to the unique architecture of Serpent OS, we can push updates that break compatibility with just a reboot, so if we ever feel the need to change the `libc`, we can make the change without you having to reinstall! More importantly, we can test major stack updates by rebooting into a staged update and go straight back to the old system, regardless of your file system. + +## Speed Packaging! + +In the early days of the repository, tooling to make creating new packages as simple as possible is vital for efficiency. Spending some time automating as much of the process as possible will take weeks off bringing up Serpent OS. By making packaging as easy as possible will also help users when creating their own packages. While it would be faster to work around issues, the build tooling upgrades will benefit everyone. + +The other way we'll be speeding up the process is by holding back some of the tuning options by default. `LTO` for instance can result in much longer build times so will not initially be the default. The same is true for debug packages, where it slows down progress without any tangible benefit. + +## Things Are Happening! + +We hope you are as excited as we are! diff --git a/src/content/docs/blog/2021/12/02/out-of-the-bootstrap-towards-serpent-os/index.mdx b/src/content/docs/blog/2021/12/02/out-of-the-bootstrap-towards-serpent-os/index.mdx deleted file mode 100644 index 2e0f2cfa..00000000 --- a/src/content/docs/blog/2021/12/02/out-of-the-bootstrap-towards-serpent-os/index.mdx +++ /dev/null @@ -1,83 +0,0 @@ ---- -title: "Out of the Bootstrap - Towards Serpent OS" -date: 2021-12-02T18:07:12+11:00 -draft: false -authors: [sunnyflunk] -tags: [news] -license: "CC-BY-ND-4.0" -copyright: "Copyright © 2021 aerynOS Developers" ---- - -The initial `stone` packages that will seed the first Serpent OS repo have now been finalized! This means that work -towards setting up the infrastructure for live package updates begins now. We plan on taking time to streamline the -processes with a focus of fixing issues at the source. In this way we can make packaging fast and efficient so we can -spend time working on features rather than package updates. - - - -## The End of Bootstrap - -Bootstrapping a distribution involves building a new toolchain and many packages needed to support it. For us bootstrap -was getting us to the point where we have built `stone` packages that we can use to start an initial repository with -full working dependencies. This has been enabled by integrating dependencies into `moss`, creating the first repo index. -Of note is that it is already enabled for 32bit support, so we have you covered there. While this is the end of -bootstrap, the fun has only just begun! - -![The first install from the bootstrap](./Featured.webp) - -## Where to Next? - -The next goal is to make Serpent OS self-hosting, where we can build packages in a container and update the repo index -with the newly built packages. It is essentially a live repository accessible from the internet. There's still plenty of -improvements to be made with the tooling, but will soon enable more users to participate in bringing Serpent OS into -fruition. - -## Becoming More Inclusive - -While there's a strong focus in Serpent OS on performance, the decision has been made to lower the initial requirements -for users. Despite AVX2 being an older technology, there are still computers sold today that don't support it. Because -of this (and already having interested users who don't have newer machines), the baseline requirement for Serpent OS -will be `x86_64-v2`, which only requires SSE4.2. - -It was always the plan to add support for these machines, just later down the track. In reality, this makes a lot more -sense, as there will be many cases where building 2 versions of a package provides little value. This is where a package -takes a long time to build and doesn't result in a notable performance improvement. We will always need the `x86_64-v2` -version of a package to be compatible with the older machines. With this approach we can reduce the build server -requirements without a noticeable impact to users as only a few packages you use will be without extra optimizations -(and probably don't benefit from them anyway). - -I want to make it clear that this will be temporary, with impactful `x86_64-v3+` packages rolling out as soon as -possible. This change paves the way to integrate our technology stack taking care of your system for you and increases -its priority. Users meeting the requirements of the `x86_64-v3+` instruction set (this includes additional instructions -beyond `x86_64-v3`) will automatically start installing these faster packages as they become available. Our -`subscriptions` model will seamlessly take care of everything for you behind the scenes so you don't need to read a -wiki or forum to learn how to get these faster packages. We can utilize the same approach in future for our `ARM` stack, -offering more optimized packages where it provides the most benefit. - -Note that from the bootstrap, most packages built in under 15s and only three took longer than 2 minutes. - -## Trying Out Some Tweaks From the Get Go - -While the project is young is a great time to test out new technologies. The use of `clang` and `lld` open up new -possibilities to reduce file sizes, shorten load times and decrease memory usage. Some of these choices may have -impacts on compatibility, so testing them out will be the best way to grasp that. Making sure that you can run apps like -Steam is vital to the experience, so whatever happens we will make sure it works. The good news is that due to the -unique architecture of Serpent OS, we can push updates that break compatibility with just a reboot, so if we ever feel -the need to change the `libc`, we can make the change without you having to reinstall! More importantly, we can test -major stack updates by rebooting into a staged update and go straight back to the old system, regardless of your file -system. - -## Speed Packaging! - -In the early days of the repository, tooling to make creating new packages as simple as possible is vital for -efficiency. Spending some time automating as much of the process as possible will take weeks off bringing up Serpent OS. -By making packaging as easy as possible will also help users when creating their own packages. While it would be faster -to work around issues, the build tooling upgrades will benefit everyone. - -The other way we'll be speeding up the process is by holding back some of the tuning options by default. `LTO` for -instance can result in much longer build times so will not initially be the default. The same is true for debug -packages, where it slows down progress without any tangible benefit. - -## Things Are Happening! - -We hope you are as excited as we are! diff --git a/src/content/docs/blog/2021/12/14/performance-corner-small-changes-pack-a-punch/index.md b/src/content/docs/blog/2021/12/14/performance-corner-small-changes-pack-a-punch/index.md new file mode 100644 index 00000000..54d16d72 --- /dev/null +++ b/src/content/docs/blog/2021/12/14/performance-corner-small-changes-pack-a-punch/index.md @@ -0,0 +1,134 @@ +--- +title: "Performance Corner: Small Changes Pack a Punch" +date: 2021-12-14T18:14:23+11:00 +draft: false +authors: [sunnyflunk] +tags: [news] +license: "CC-BY-ND-4.0" +copyright: "Copyright © 2021 aerynOS Developers" +--- + +Here we have another round of changes to make packages smaller and show just how much we care about performance and efficiency! Today we are focusing mainly on `moss-format` changes to reduce the size of its payloads. The purpose of these changes is to reduce the size of packages, DB storage for transactions and memory usage for `moss`. These changes were made just before we moved out of bootstrap, so we wouldn't have to rebuild the world after changing the format. Lets get started! + + +## Squeezing Out the Last Blit of Performance + +`Blitting` in Serpent OS is the process of setting up a root, using hardlinks of files installed in the `moss` store to construct the system `/usr` directory. Some initial testing showed `buildPath` using about 3% of the total events when installing a package with many files. As a system is made up of over 100,000 files, that's a lot of paths that need to be calculated! Performance is therefore important to minimize time taken and power consumed. + +After running a few benchmarks of `buildPath`, there were a few tweaks which could improve the performance. Rather than calculating the full path of the file, we can reuse the constant root path for each file instead, reducing `buildPath` events by 15%. Here we see some callgrind data from this small change (as it was difficult to pick up in the noise). + +``` +Before: + 48,766,347 ( 1.50%) /usr/include/dlang/ldc/std/path.d:pure nothrow @safe immutable(char)[] std.path.buildPath + 34,241,805 ( 1.05%) /usr/include/dlang/ldc/std/utf.d:pure nothrow @safe immutable(char)[] std.path.buildPath + 14,363,684 ( 0.44%) /usr/include/dlang/ldc/std/range/package.d:pure nothrow @safe immutable(char)[] std.path.buildPath + +After: + 40,357,800 ( 1.25%) /usr/include/dlang/ldc/std/path.d:pure nothrow @safe immutable(char)[] std.path.buildPath + 29,523,966 ( 0.91%) /usr/include/dlang/ldc/std/utf.d:pure nothrow @safe immutable(char)[] std.path.buildPath + 12,814,283 ( 0.40%) /usr/include/dlang/ldc/std/range/package.d:pure nothrow @safe immutable(char)[] std.path.buildPath +``` + +## Only Record What's Needed! + +Our `Index Payload` is used for extracting the `Content Payload` into its hashed file names. As it has one job (and then discarded), we could hone in on including only the information we needed. Previously an entry was a 32 byte key and storing a 33 byte hash. We have now integrated the hash as a `ubyte[16]` field, cut other unneeded fields so that we can fit the whole entry into 32 bytes. That's a 51% reduction in the size of the `Index Payload` and about 25% smaller when compressed. + +``` +Before: Index [Records: 3688 Compression: Zstd, Savings: 60.61%, Size: 239.72 KB] +After: Index [Records: 3688 Compression: Zstd, Savings: 40.03%, Size: 118.02 KB] +``` + +## Too Many Entries Makes for a Large Payload + +One of the bugbears about the Layout Payload, was the inclusion of `Directory` paths for every single directory. This is handy in that directories can be created easily before any files are included (to ensure they exist), but it comes at a price. The issue was twofold, the extra entries made inspecting the file list take longer and also made the `Layout Payload` larger than it needed to be. Therefore directories are no longer included as Layout entries with the exceptions of empty directories and directories with different permissions. Lets compare `nano` and `glibc` builds before and after the change: + +``` +nano +Before: Layout [Records: 174 Compression: Zstd, Savings: 80.58%, Size: 13.78 KB] +After: Layout [Records: 93 Compression: Zstd, Savings: 73.49%, Size: 9.15 KB] + +glibc +Before: Layout [Records: 7879 Compression: Zstd, Savings: 86.88%, Size: 755.00 KB] +After: Layout [Records: 6813 Compression: Zstd, Savings: 86.24%, Size: 689.20 KB] +``` + +A surprisingly large impact from a small change, with 1,000 fewer entries for `glibc` and cutting the Layout Payload size of `nano` by a third. What it shows is that it's hugely beneficial where locales are involved and % size reduction increases where you have fewer files. To give an example of how bad it could be, the KDE Frameworks package `ktexteditor` would have produced 300 entries in the Layout Payload, where 200 of those would have been for directories! I'd estimate a 50% reduction in the Layout Payload size for this package! Here's an example of how a locale file used to be stored (where only the last line is added now). + +``` + - /usr/share/locale [Directory] + - /usr/share/locale/bg [Directory] + - /usr/share/locale/bg/LC_MESSAGES [Directory] + - /usr/share/locale/bg/LC_MESSAGES/nano.mo -> ec5b82819ec2355d4e7bbccc7d78ce60 [Regular] +``` + +This will be exceptionally useful for keeping the `LayoutDB` slimmer and faster as the number of packages grows. + +## Cutting Back 1 Byte at a Time + +Next we removed recording the timestamps of files, which for reproducible builds, is often a number of little relevance as you have to force them all to a fixed number. As `moss` is de-duplicating for files, there's a second issue where two packages could have different timestamps for the same hash. Therefore it was considered an improvement to simply exclude timestamps altogether. This improved install time as we no longer overwrite the timestamps and made the payload more compressible due to replacing it with 8 bytes of padding. Unfortunately we weren't quite able to free up 16 bytes to reduce the size of each entry, but will be something to pursue in future. + +Another quick improvement was reducing the lengths of paths for each entry. `moss` creates system roots and switches between them by changing the `/usr` symlink. Therefore, all system files need to be installed into `/usr` or they will not make up your base system. Therefore we have no need to store `/usr` in the Payload so we strip `/usr/` from the paths (the extra / gives us another byte off!) which we recreate on install. This improved the uncompressed size of the Payload, with only a minor reduction when compressed. + +Combined these result in about a 5% decrease in the compressed and uncompressed size of the `Layout Payload`. + +``` +Before: Layout [Records: 6812 Compression: Zstd, Savings: 86.24%, Size: 689.10 KB] +After: Layout [Records: 6812 Compression: Zstd, Savings: 86.20%, Size: 655.00 KB] +``` + +## Storing the Hash Efficiently + +Using the same method for the `Index Payload`, we are now storing the hash as `ubyte[16]`, but not directly in the `Payload Entry`. This gives us a sizeable reduction of 17 bytes per entry which is the most significant of all the `Layout Payload` changes. As the extra space was unneeded, it compressed well so only resulted in a small reduction in the compressed Payload size. + +``` +Before: Layout [Records: 6812 Compression: Zstd, Savings: 86.20%, Size: 655.00 KB] +After: Layout [Records: 6812 Compression: Zstd, Savings: 83.92%, Size: 539.26 KB] +``` + +![Planning payload changes](./Featured.webp "Planning payload changes") + +## Hang On, Why am I Getting Faster Installation? + +As a side-effect of small code rewrites to implement these changes, we've seen a nice decrease in time to install packages. There are fewer entries to iterate over with the removal of directories and `buildPath` is now only called once for each path. It goes to show that focusing on the small details leads to less, more efficient and faster code. Here we find that we have now essentially halved the number of events related to `buildPath` with all the changes resulting in about a 5% reduction in install time. Note that for this test, over 80% of time is spent in `zstd` decompressing the package which we haven't optimized (yet!). Here's another look the `buildPath` numbers factoring in all the changes: + +``` +Before: + 48,766,347 ( 1.50%) /usr/include/dlang/ldc/std/path.d:pure nothrow @safe immutable(char)[] std.path.buildPath + 34,241,805 ( 1.05%) /usr/include/dlang/ldc/std/utf.d:pure nothrow @safe immutable(char)[] std.path.buildPath + 14,363,684 ( 0.44%) /usr/include/dlang/ldc/std/range/package.d:pure nothrow @safe immutable(char)[] std.path.buildPath + +After: + 25,145,625 ( 0.79%) /usr/include/dlang/ldc/std/path.d:pure nothrow @safe immutable(char)[] std.path.buildPath + 17,967,216 ( 0.57%) /usr/include/dlang/ldc/std/utf.d:pure nothrow @safe immutable(char)[] std.path.buildPath + 7,761,417 ( 0.24%) /usr/include/dlang/ldc/std/range/package.d:pure nothrow @safe immutable(char)[] std.path.buildPath +``` + +## Summing it All Up + +It was a pretty awesome weekend of work (a few weeks ago now), making some quick changes that will improve `Serpent OS` a great deal for a long time to come. This also means we have integrated all the quick format changes so we won't have to rebuild packages while bringing up the repos. + +Here's a quick summary of the results of all these small changes: + +- 51% reduction in the uncompressed size of the Index Payload +- 25% reduction in the compressed size of the Index Payload +- 29-50% reduction in the uncompressed size of the Layout Payload (much more with fewer files and more locales) +- 12-15% reduction in the compressed size of the Layout Payload +- 5% faster package installation for our benchmark (800ms to 760ms) + +These are some pretty huge numbers and even excluded the massive improvements we made in the previous blog! + +## What if We Include the Previous Changes? + +I'm glad you asked, cause I was curious too! Here we see a before and after with all the changes included. For the `Layout Payload` we see a ~45% reduction in the compressed and uncompressed size. For the `Index Payload` we have reduced the uncompressed size by 67% and the compressed size by 56%. Together resulting in halving the compressed and uncompressed size of the metadata of our stone packages! + +``` +Before: +Payload: Layout [Records: 5441 Compression: Zstd, Savings: 83.13%, Size: 673.46 KB] (113.61 KB compressed) +Payload: Index [Records: 2550 Compression: Zstd, Savings: 55.08%, Size: 247.35 KB] (111.11 KB compressed) + +After: +Payload: Layout [Records: 4718 Compression: Zstd, Savings: 83.62%, Size: 374.42 KB] (61.33 KB compressed) +Payload: Index [Records: 2550 Compression: Zstd, Savings: 39.85%, Size: 81.60 KB] (49.01 KB compressed) +``` + +Now we proceed towards bringing up repos to enjoy our very efficient packages! diff --git a/src/content/docs/blog/2021/12/14/performance-corner-small-changes-pack-a-punch/index.mdx b/src/content/docs/blog/2021/12/14/performance-corner-small-changes-pack-a-punch/index.mdx deleted file mode 100644 index f73bf0c6..00000000 --- a/src/content/docs/blog/2021/12/14/performance-corner-small-changes-pack-a-punch/index.mdx +++ /dev/null @@ -1,181 +0,0 @@ ---- -title: "Performance Corner: Small Changes Pack a Punch" -date: 2021-12-14T18:14:23+11:00 -draft: false -authors: [sunnyflunk] -tags: [news] -license: "CC-BY-ND-4.0" -copyright: "Copyright © 2021 aerynOS Developers" ---- - -Here we have another round of changes to make packages smaller and show just how much we care about performance and -efficiency! Today we are focusing mainly on `moss-format` changes to reduce the size of its payloads. The purpose of -these changes is to reduce the size of packages, DB storage for transactions and memory usage for `moss`. These changes -were made just before we moved out of bootstrap, so we wouldn't have to rebuild the world after changing the format. -Lets get started! - - - -## Squeezing Out the Last Blit of Performance - -`Blitting` in Serpent OS is the process of setting up a root, using hardlinks of files installed in the `moss` store to -construct the system `/usr` directory. Some initial testing showed `buildPath` using about 3% of the total events -when installing a package with many files. As a system is made up of over 100,000 files, that's a lot of paths that need -to be calculated! Performance is therefore important to minimize time taken and power consumed. - -After running a few benchmarks of `buildPath`, there were a few tweaks which could improve the performance. Rather than -calculating the full path of the file, we can reuse the constant root path for each file instead, reducing `buildPath` -events by 15%. Here we see some callgrind data from this small change (as it was difficult to pickup in the noise). - -``` -Before: - 48,766,347 ( 1.50%) /usr/include/dlang/ldc/std/path.d:pure nothrow @safe immutable(char)[] std.path.buildPath - 34,241,805 ( 1.05%) /usr/include/dlang/ldc/std/utf.d:pure nothrow @safe immutable(char)[] std.path.buildPath - 14,363,684 ( 0.44%) /usr/include/dlang/ldc/std/range/package.d:pure nothrow @safe immutable(char)[] std.path.buildPath - -After: - 40,357,800 ( 1.25%) /usr/include/dlang/ldc/std/path.d:pure nothrow @safe immutable(char)[] std.path.buildPath - 29,523,966 ( 0.91%) /usr/include/dlang/ldc/std/utf.d:pure nothrow @safe immutable(char)[] std.path.buildPath - 12,814,283 ( 0.40%) /usr/include/dlang/ldc/std/range/package.d:pure nothrow @safe immutable(char)[] std.path.buildPath -``` - -## Only Record What's Needed! - -Our `Index Payload` is used for extracting the `Content Payload` into its hashed file names. As it has one job (and then -discarded), we could hone in on including only the information we needed. Previously an entry was a 32 byte key and -storing a 33 byte hash. We have now integrated the hash as a `ubyte[16]` field, cut other unneeded fields so that we can -fit the whole entry into 32 bytes. That's a 51% reduction in the size of the `Index Payload` and about 25% smaller when -compressed. - -``` -Before: Index [Records: 3688 Compression: Zstd, Savings: 60.61%, Size: 239.72 KB] -After: Index [Records: 3688 Compression: Zstd, Savings: 40.03%, Size: 118.02 KB] -``` - -## Too Many Entries Makes for a Large Payload - -One of the bugbears about the Layout Payload, was the inclusion of `Directory` paths for every single directory. This is -handy in that directories can be created easily before any files are included (to ensure they exist), but it comes at a -price. The issue was twofold, the extra entries made inspecting the file list take longer and also made the `Layout -Payload` larger than it needed to be. Therefore directories are no longer included as Layout entries with the exceptions -of empty directories and directories with different permissions. Lets compare `nano` and `glibc` builds before and -after the change: - -``` -nano -Before: Layout [Records: 174 Compression: Zstd, Savings: 80.58%, Size: 13.78 KB] -After: Layout [Records: 93 Compression: Zstd, Savings: 73.49%, Size: 9.15 KB] - -glibc -Before: Layout [Records: 7879 Compression: Zstd, Savings: 86.88%, Size: 755.00 KB] -After: Layout [Records: 6813 Compression: Zstd, Savings: 86.24%, Size: 689.20 KB] -``` - -A surprisingly large impact from a small change, with 1,000 fewer entries for `glibc` and cutting the Layout Payload -size of `nano` by a third. What it shows is that it's hugely beneficial where locales are involved and % size reduction -increases where you have fewer files. To give an example of how bad it could be, the KDE Frameworks package -`ktexteditor` would have produced 300 entries in the Layout Payload, where 200 of those would have been for directories! -I'd estimate a 50% reduction in the Layout Payload size for this package! Here's an example of how a locale file used -to be stored (where only the last line is added now). - -``` - - /usr/share/locale [Directory] - - /usr/share/locale/bg [Directory] - - /usr/share/locale/bg/LC_MESSAGES [Directory] - - /usr/share/locale/bg/LC_MESSAGES/nano.mo -> ec5b82819ec2355d4e7bbccc7d78ce60 [Regular] -``` - -This will be exceptionally useful for keeping the `LayoutDB` slimmer and faster as the number of packages grows. - -## Cutting Back 1 Byte at a Time - -Next we removed recording the timestamps of files, which for reproducible builds, is often a number of little relevance -as you have to force them all to a fixed number. As `moss` is de-duplicating for files, there's a second issue where two -packages could have different timestamps for the same hash. Therefore it was considered an improvement to simply exclude -timestamps altogether. This improved install time as we no longer overwrite the timestamps and made the payload more -compressible due to replacing it with 8 bytes of padding. Unfortunately we weren't quite able to free up 16 bytes to -reduce the size of each entry, but will be something to pursue in future. - -Another quick improvement was reducing the lengths of paths for each entry. `moss` creates system roots and switches -between them by changing the `/usr` symlink. Therefore, all system files need to be installed into `/usr` or they will -not make up your base system. Therefore we have no need to store `/usr` in the Payload so we strip `/usr/` from the -paths (the extra / gives us another byte off!) which we recreate on install. This improved the uncompressed size of the -Payload, with only a minor reduction when compressed. - -Combined these result in about a 5% decrease in the compressed and uncompressed size of the `Layout Payload`. - -``` -Before: Layout [Records: 6812 Compression: Zstd, Savings: 86.24%, Size: 689.10 KB] -After: Layout [Records: 6812 Compression: Zstd, Savings: 86.20%, Size: 655.00 KB] -``` - -## Storing the Hash Efficiently - -Using the same method for the `Index Payload`, we are now storing the hash as `ubyte[16]`, but not directly in the -`Payload Entry`. This gives us a sizeable reduction of 17 bytes per entry which is the most significant of all the -`Layout Payload` changes. As the extra space was unneeded, it compressed well so only resulted in a small reduction in -the compressed Payload size. - -``` -Before: Layout [Records: 6812 Compression: Zstd, Savings: 86.20%, Size: 655.00 KB] -After: Layout [Records: 6812 Compression: Zstd, Savings: 83.92%, Size: 539.26 KB] -``` - -![Planning payload changes](./Featured.webp) - -## Hang On, Why am I Getting Faster Installation? - -As a side-effect of small code rewrites to implement these changes, we've seen a nice decrease in time to install -packages. There are fewer entries to iterate over with the removal of directories and `buildPath` is now only called -once for each path. It goes to show that focusing on the small details leads to less, more efficient and faster code. -Here we find that we have now essentially halved the number of events related to `buildPath` with all the changes -resulting in about a 5% reduction in install time. Note that for this test, over 80% of time is spent in `zstd` -decompressing the package which we haven't optimized (yet!). Here's another look the `buildPath` numbers factoring in -all the changes: - -``` -Before: - 48,766,347 ( 1.50%) /usr/include/dlang/ldc/std/path.d:pure nothrow @safe immutable(char)[] std.path.buildPath - 34,241,805 ( 1.05%) /usr/include/dlang/ldc/std/utf.d:pure nothrow @safe immutable(char)[] std.path.buildPath - 14,363,684 ( 0.44%) /usr/include/dlang/ldc/std/range/package.d:pure nothrow @safe immutable(char)[] std.path.buildPath - -After: - 25,145,625 ( 0.79%) /usr/include/dlang/ldc/std/path.d:pure nothrow @safe immutable(char)[] std.path.buildPath - 17,967,216 ( 0.57%) /usr/include/dlang/ldc/std/utf.d:pure nothrow @safe immutable(char)[] std.path.buildPath - 7,761,417 ( 0.24%) /usr/include/dlang/ldc/std/range/package.d:pure nothrow @safe immutable(char)[] std.path.buildPath -``` - -## Summing it All Up - -It was a pretty awesome weekend of work (a few weeks ago now), making some quick changes that will improve `Serpent OS` -a great deal for a long time to come. This also means we have integrated all the quick format changes so we won't have -to rebuild packages while bringing up the repos. - -Here's a quick summary of the results of all these small changes: - -- 51% reduction in the uncompressed size of the Index Payload -- 25% reduction in the compressed size of the Index Payload -- 29-50% reduction in the uncompressed size of the Layout Payload (much more with fewer files and more locales) -- 12-15% reduction in the compressed size of the Layout Payload -- 5% faster package installation for our benchmark (800ms to 760ms) - -These are some pretty huge numbers and even excluded the massive improvements we made in the previous blog! - -## What if We Include the Previous Changes? - -I'm glad you asked, cause I was curious too! Here we see a before and after with all the changes included. For the -`Layout Payload` we see a ~45% reduction in the compressed and uncompressed size. For the `Index Payload` we have -reduced the uncompressed size by 67% and the compressed size by 56%. Together resulting in halving the compressed and -uncompressed size of the metadata of our stone packages! - -``` -Before: -Payload: Layout [Records: 5441 Compression: Zstd, Savings: 83.13%, Size: 673.46 KB] (113.61 KB compressed) -Payload: Index [Records: 2550 Compression: Zstd, Savings: 55.08%, Size: 247.35 KB] (111.11 KB compressed) - -After: -Payload: Layout [Records: 4718 Compression: Zstd, Savings: 83.62%, Size: 374.42 KB] (61.33 KB compressed) -Payload: Index [Records: 2550 Compression: Zstd, Savings: 39.85%, Size: 81.60 KB] (49.01 KB compressed) -``` - -Now we proceed towards bringing up repos to enjoy our very efficient packages! diff --git a/src/content/docs/blog/2022/01/20/can-hardly-contain-myself-plus-a-bonus/index.md b/src/content/docs/blog/2022/01/20/can-hardly-contain-myself-plus-a-bonus/index.md new file mode 100644 index 00000000..34ad76d5 --- /dev/null +++ b/src/content/docs/blog/2022/01/20/can-hardly-contain-myself-plus-a-bonus/index.md @@ -0,0 +1,48 @@ +--- +title: "Can Hardly Contain Myself, Plus a Bonus" +date: 2022-01-20T18:08:07+11:00 +draft: false +authors: [sunnyflunk] +tags: [news] +license: "CC-BY-ND-4.0" +copyright: "Copyright © 2022 aerynOS Developers" +--- + +One of the core steps for building a package is setting up a minimal environment with only the required (and stated) dependencies. Currently we have been building our stones in an `systemd-nspawn` container, where the root contains every package that's been built so far. This makes the environment extremely difficult to reproduce! + +Today we announce `moss-container`, a simple but flexible container creator that we can integrate for proper containerized builds. + +## Versatile For Many Use Cases + +Containers have a multitude of uses for a Linux distro, but our immediate use case is for reproducible container builds for `boulder`. However, we have plans to use `moss-container` for testing, validation and benchmarking purposes as well. Therefore it's important to consider all workloads, so features like `fakeroot` and `networking` can be toggled on or off depending on what features are needed. + +`moss-container` takes care of everything, the device nodes in the `/dev` tree, mounting directories as `tmpfs` so the environment is left in a clean state, and mounting the `/sys` and `/proc` special file-systems. These are essential for a fully functioning container where programs like `python` and even `clang` won't work without them. And best of all, it's very fast so fits in well with the rest of our tooling! + +The next step is integrating `moss-container` into `boulder`, so that builds become reproducible across machines, and makes it much easier for users to run builds on their host machines. + +## moss Now Understands Repositories + +Previously (but not covered in the blogs) work was also done on `moss` so that it can understand and fetch `stone` packages from an online repo. This ties in nicely with the `moss-container` work and is a requirement for finishing up a proper build process for Serpent OS. We are now one step closer to having a full distribution cycle from building packages and pushing those packages as system updates! + +![Container with functioning device nodes](./Featured.webp "Container with functioning device nodes") + +## Check Out The Development + +In case you've missed it, `ikey` has been streaming some of the development of the tooling on his [Twitch channel](https://www.twitch.tv/ikeydoherty). DLang is not as commonly used as other languages, so check it out to see the advantages it brings. Streams are typically announced on twitter, or give him a follow to see when he next goes live! + +## Bonus Content Refresh + +This year we've had a considerable number of new visitors and interest in Serpent OS. Unfortunately the content on the website had been a bit stale and untouched for some time. There was some confusion and misunderstanding over some of the terms and content. Some of the common issues were: + +- Subscriptions is a loaded term relating to software +- Subscriptions only referred to a fraction of the smart features +- Seemed targeted at advanced users with too many technical terms +- Lack of understanding around what moss and boulder do +- That features would add complexity when the tools were actually removing the complexity + +The good news is that a good chunk of it has been redone, including two new pages for our core tools `boulder` and `moss`. `Subscriptions` has been renamed to `Smart System Management` to reflect its broader nature (which you can read about [here](/docs)). + +Much of the content has also had a refresh or a rewrite, so if you've seen it before, it will likely be a lot easier to digest now. But this isn't the final state of the content, as more features will need to be added and there's still a few rough edges (and I like to rewrite things every once in awhile). Many ideas have been raised by our community in the matrix channel[^1], so a shout-out to the good folks we have hanging out there. + +--- +[^1]: aerynOS now uses a [Zulip server](https://aerynos.zulipchat.com/) for community discussion. diff --git a/src/content/docs/blog/2022/01/20/can-hardly-contain-myself-plus-a-bonus/index.mdx b/src/content/docs/blog/2022/01/20/can-hardly-contain-myself-plus-a-bonus/index.mdx deleted file mode 100644 index 2909e9b6..00000000 --- a/src/content/docs/blog/2022/01/20/can-hardly-contain-myself-plus-a-bonus/index.mdx +++ /dev/null @@ -1,70 +0,0 @@ ---- -title: "Can Hardly Contain Myself, Plus a Bonus" -date: 2022-01-20T18:08:07+11:00 -draft: false -authors: [sunnyflunk] -tags: [news] -license: "CC-BY-ND-4.0" -copyright: "Copyright © 2022 aerynOS Developers" ---- - -One of the core steps for building a package is setting up a minimal environment with only the required (and stated) -dependencies. Currently we have been building our stones in an `systemd-nspawn` container, where the root contains every -package that's been built so far. This makes the environment extremely difficult to reproduce! - - - -Today we announce `moss-container`, a simple but flexible container creator that we can integrate for proper -containerized builds. - -## Versatile For Many Use Cases - -Containers have a multitude of uses for a Linux distro, but our immediate use case is for reproducible container builds -for `boulder`. However, we have plans to use `moss-container` for testing, validation and benchmarking purposes as well. -Therefore it's important to consider all workloads, so features like `fakeroot` and `networking` can be toggled on or -off depending on what features are needed. - -`moss-container` takes care of everything, the device nodes in the `/dev` tree, mounting directories as `tmpfs` so the -environment is left in a clean state, and mounting the `/sys` and `/proc` special file-systems. These are essential for -a fully functioning container where programs like `python` and even `clang` won't work without them. And best of all, -it's very fast so fits in well with the rest of our tooling! - -The next step is integrating `moss-container` into `boulder`, so that builds become reproducible across machines, and -makes it much easier for users to run builds on their host machines. - -## moss Now Understands Repositories - -Previously (but not covered in the blogs) work was also done on `moss` so that it can understand and fetch `stone` -packages from an online repo. This ties in nicely with the `moss-container` work and is a requirement for finishing up a -proper build process for Serpent OS. We are now one step closer to having a full distribution cycle from building -packages and pushing those packages as system updates! - -![Container with functioning device nodes](./Featured.webp) - -## Check Out The Development - -In case you've missed it, `ikey` has been streaming some of the development of the tooling on his -[Twitch channel](https://www.twitch.tv/ikeydoherty). DLang is not as commonly used as other languages, so check it out -to see the advantages it brings. Streams are typically announced on twitter, or give him a follow to see when he next -goes live! - -## Bonus Content Refresh - -This year we've had a considerable number of new visitors and interest in Serpent OS. Unfortunately the content on the -website had been a bit stale and untouched for some time. There was some confusion and misunderstanding over some of the -terms and content. Some of the common issues were: - -- Subscriptions is a loaded term relating to software -- Subscriptions only referred to a fraction of the smart features -- Seemed targeted at advanced users with too many technical terms -- Lack of understanding around what moss and boulder do -- That features would add complexity when the tools were actually removing the complexity - -The good news is that a good chunk of it has been redone, including two new pages for our core tools `boulder` and -`moss`. `Subscriptions` has been renamed to `Smart System Management` to reflect its broader nature (which you can read -about [here](/tooling)). - -Much of the content has also had a refresh or a rewrite, so if you've seen it before, it will likely be a lot easier to -digest now. But this isn't the final state of the content, as more features will need to be added and there's still a -few rough edges (and I like to rewrite things every once in awhile). Many ideas have been raised by our community in the -[matrix channel](https://matrix.to/#/#serpentos:matrix.org), so a shout-out to the good folks we have hanging out there. diff --git a/src/content/docs/blog/2022/02/11/making-deltas-great-again-1/index.md b/src/content/docs/blog/2022/02/11/making-deltas-great-again-1/index.md new file mode 100644 index 00000000..d32ce5c2 --- /dev/null +++ b/src/content/docs/blog/2022/02/11/making-deltas-great-again-1/index.md @@ -0,0 +1,140 @@ +--- +title: "Making Deltas Great Again! (Part 1)" +date: 2022-02-11T20:45:12+11:00 +draft: false +authors: [sunnyflunk] +tags: [news] +license: "CC-BY-ND-4.0" +copyright: "Copyright © 2022 aerynOS Developers" +--- + +In [Optimising Package Distribution](/blog/2021/03/16/optimising-package-distribution) we discussed some early findings for implementing binary deltas in Serpent OS. While discussing the implementation we have found the requirements to be suboptimal for what we were after. We provide a fresh look at the issue and what we can do to make it useful in almost all situations without the drawbacks. + +## Deltas Have a Bad Reputation + +I remember back in the early 2000s on Gentoo when someone set up a server to produce delta patches from source tarballs to distribute to users with small data caps such as myself. When requesting the delta, the job would be added to the queue (which occasionally could be minutes) and then created a small patch to download. This was so important at the time to reduce the download size that the extra time was well worth it! + +Today things are quite different. The case for deltas has reduced for users as internet speeds have increased. Shrinking 20MB off a package size may be a second reduction for some, but 10 seconds for others. The largest issue is that deltas have typically pushed extra computational effort onto the users computer in compensation for the smaller size. With a fast internet connection that cost is a real burden where deltas take longer to install than simply downloading the full package. + +## What's so Bad About the Old Method? + +The previous idea of using the full payload for deltas was very efficient in terms of distribution, but required changes in how `moss` handles packages to implement it. Having the previous payload available (and being fast) means storing the old payloads on disk. This increases the storage requirements for the base system, although that can be reduced by compressing the payloads to reduce disk usage (but increasing CPU usage at install time). + +To make it work well, we needed the following requirements: +- Install all the files from a package and recreate the payload from the individual files. However, Smart System Management allows users to avoid installing unneeded locale files and we would not be able to recreate the full payload without them. +- Alternatively we can store the full payloads on disk. Then there becomes a tradeoff from doubling storage space or additional overhead from compressing the payloads to reduce it. +- Significant memory requirements to use a delta when a package is large. + +In short we weren't happy with having to increase the disk storage requirements (potentially more than 2x) or the increase in CPU time to create compressed payloads to minimize it. This was akin to the old delta model, of smaller downloads but significantly slower updates. + +## Exploring an Alternative Approach + +Optimal compression generally benefits from combining multiple files into one archive than to compress each file individually. With `zstd` deltas, since you read in a dictionary (the old file), you already have good candidates for compression matching. The real question was simply whether creating a delta for each file was a viable method for delta distribution (packaged into a single payload of course). + +As the real benefit of deltas is reducing download size, the first thing to consider is the size impact. Using the same package as the previous blog post (but with newer versions of `QtWebEngine` and `zstd`) we look at what happens when you delta each file individually. Note that the package is quite unique in that the largest file is 76% of the total package size and that delta creation time increased a lot for files larger than 15-20MB at maximum compression. + +| | Full Tarball | Individual Files | Full Tarball --zstd=chainLog=30 | Individual Files --zstd=chainLog=30 | +|-----------------|--------------|------------------|---------------------------------|-------------------------------------| +| Time to create | 134.6s | 137.6s | 157.9s | 150.9s | +| Size of delta | 30.8MB | 29.8MB | 28.3MB | 28.6MB | +| Peak Memory | 1.77GB | 1.64GB | 4.77GB | 2.64GB | + +Quite surprisingly, the delta sizes were very close! Most surprising was that without increasing the size of the chainLog in `zstd`, the individual file approach actually resulted in smaller deltas! We can also see how much lower the memory requirements were (and they would be much smaller when there isn't one large file in the package!). Our locale and doc trimming of files will still work nicely, as we don't need to recreate the locale files that aren't installed (as we still don't want them!). + +The architecture of `moss` allows us to cache packages, where all cached packages are available for generating multiple system roots including with our build tool `boulder` without any need for the original package file. Therefore any need to retain old payloads or packages is no longer required or useful, eliminating the drawbacks of the previous delta approach. The memory requirements are also reduced as the maximum memory requirement scales with the size of the largest file, rather than the entire package (which is generally a lot bigger). There are many packages containing hundreds of MBs of uncompressed data and a few into the GBs. But the largest file I could find installed locally was only 140MB, and only a handful over 100MB. This smaller increase in memory requirements is a huge improvement and the small increase in memory use to extract the deltas is likely to go unnoticed by users. + +Well everything sounds pretty positive so far, there must be some drawback? + +![CLI terminal](./Featured.webp "CLI showing progress") + +## The Impact on Package Cache Time + +As the testing method for this article is quite simplistic (`bash` loops and calls to `zstd` directly), the additional overhead from creating deltas for individual files I estimated to be about 20ms compared to a proper solution. The main difference from the old delta method is how we extract the payloads and recreate the files of the new package. Using the full package you simply extract the content payload and split it into its corresponding files. The new approach requires two steps, extracting the payload (we could in theory not compress it) and then applying patches to the original files to recreate the new ones. Note that times differ slightly from the previous table due to minor variations between test runs. + +| | Normal Package | Individual Delta File Package | +|--------------------------------------|------------------|-------------------------------| +| Time to Delta Files | - | 148.0s (137 files) | +| Time to Compress Payload | 78.6s | 4.0s | +| Size of Uncompressed Payload | 165.8MB | 28.9MB | +| Size of Compressed Payload | 51.3MB | 28.6MB | +| Instructions to Extract Payload | 2,876.9m (349ms) | 33.1m (34ms) | +| Instructions to Recreate Files | - | 1,785.6m (452ms) | +| | | | +| Total instructions to Extract | 2,876.9m (349ms) | 1,818.7m (486ms) | + +What's important to note is that is this reflects a worst case scenario for the delta approach, where all 137 files were different between the old and new version of the package. Package updates where files remain unchanged allows us to omit them from the delta package altogether! So the delta approach not only saves time downloading files, but also requires fewer CPU instructions to apply the update. It's not quite a slam dunk though as reading the original file as a dictionary results in an increase in elapsed time of extraction (though the extra time is likely much less than the time saved downloading 20MB less!). + +In Part 2 we will look at some ways we can tweak the approach to balance the needed resources for creating delta packages and to reduce the time taken to apply them. + +:::note +This was intended to be a 2 part series as it contains a lot of information to digest. However, Part 2 was committed and follows below. +::: + +There's more than one way to create a delta! This post follows on from the earlier analysis of creating some basic delta packages. Where this approach, and the use of `zstd`, really thrives is that it gives us options in how to manage the overhead, from creating the deltas to getting them installed locally. Next we explore some ideas of how we can minimize the caching time of delta files. + +## Improving Delta Cache Time + +To get the best bang for your buck with deltas, it is essential to reduce the size of the larger files. My experience in testing was that there wasn't a significant benefit from creating deltas for small files. In this example, we only create delta files when they are larger than a specific size while including the full version of files that are under the cutoff. This reduces the number of delta files created without impacting the overall package size by much. + +| Only Delta Files Greater Than | Greater than 10KB | Greater than 100KB | Greater than 500KB | +|--------------------------------------|-------------------|--------------------|--------------------| +| Time to Delta Files | 146.1s (72 files) | 146.3s (64 files) | 139.4s (17 files) | +| Time to Compress Payload | 3.9s | 4.0s | 8.3s | +| Size of Uncompressed Payload | 28.9MB | 29.3MB | 42.4MB | +| Size of Compressed Payload | 28.6MB | 28.7MB | 30.5MB | +| Instructions to Extract Payload | 37.8m (36ms) | 34.7m (29ms) | 299.1m (66ms) | +| Instructions to Recreate Files | 1,787.7m (411ms) | 1,815.0m (406ms) | 1,721.4m (368ms) | +| | | | | +| Total instructions to Extract | 1,825.5m (447ms) | 1,849.7m (435ms) | 2,020.5m (434ms) | + +Here we see that by not creating deltas for files under 100KB, it barely impacts the size of the delta at all, while reducing caching time by 50ms compared to creating a delta for every file (486ms from the previous blog post). It even leads to up to a 36% reduction in CPU instructions to undertake caching through the delta than using the full package. In terms of showing how effective this delta technique really is, I chose one of the worst examples and I would expect that many deltas would be faster to cache when there's files that are exact matches between the old and new package. The largest file alone took 300ms to apply the delta, where overheads tend to scale a lot when you start getting to larger files. + +There are also some steps we can take to make sure that caching a delta is almost always faster than the full package (solving the only real drawback to users), only requiring Serpent OS resources to create these delta packages. + +## Creating More Efficient Deltas + +For this article, all the tests have been run with `zstd --ultra -22 --zstd=chainLog=30`...until now! The individual file delta approach is more robust at lower compression levels to keep package size small while reducing how long they take to create. Lets take a look at the difference while also ensuring `--long` is enabled. This testing combined with the results above for only creating deltas for files larger than 10KB. + +| Individual Delta File Package | zstd -12 | zstd -16 | zstd -19 | zstd -22 | +|--------------------------------------|------------------|------------------|------------------|------------------| +| Time to Delta Files | 6.7s | 113.9s | 142.3s | 148.3s | +| Time to Compress Payload | 0.5s | 3.2s | 5.3s | 4.0s | +| Size of Uncompressed Payload | 41.1MB | 30.6MB | 28.9MB | 28.9MB | +| Size of Compressed Payload | 40.9MB | 30.3MB | 28.6MB | 28.6MB | +| Instructions to Extract Payload | 46.5m (35ms) | 51.2m (28ms) | 42.6m (33ms) | 37.8m (36ms) | +| Instructions to Recreate Files | 1,773.7m (382ms) | 1,641.5m (385ms) | 1,804.2m (416ms) | 1,810.9m (430ms) | +| | | | | | +| Total instructions to Extract | 1,820.2m (417ms) | 1,692.7m (413ms) | 1,846.8m (449ms) | 1,848.7m (466ms) | + +Compression levels 16-19 look quite interesting where you start to reduce the time taken to apply the delta as well and only seeing a small increase in size. For comparison, at `-19` it only took 9s to delta the remaining 39.8MB of files when excluding the largest file (it was 15s at `-22`). While the time taken between 19 and 22 was almost the same, at `-19` it took 27% fewer instructions to create the deltas than at `-22` (`-16` uses 64% fewer instructions than `-22`). It will need testing across a broader range of packages to see the overall impact and to evaluate a sensible default. + +As a side effect of reducing the compression level, you also get another small decrease in the time to cache a package. The best news of all is that these numbers are already out of date. Testing was performed last year with `zstd` 1.5.0, where there have been notable speed improvements to both compression and decompression that have been included in newer releases. Great news given how fast it is already! Here's a quick summary of how it all ties together. + +## Delta's Are Back! + +This blog series has put forward a lot of data that might be difficult to digest...but what does it mean for users of Serpent OS? Here's a quick summary of the advantages of using this delta method on individual files when compared to fetching the full packages: + +- Package update sizes are greatly reduced to speed up fetching of package updates. +- `moss` architecture means that we have no need to leave packages on disk for later use, reducing the storage footprint. In fact, we could probably avoid writes (except for the extracted package of course!) by downloading packages to `tmpfs` where you have sufficient free memory. +- Delta's can be used for updating packages for packaging **and** your installed system. There's no need for a full copy of the full original package for packaging. A great benefit when combined with the source repository. +- Delta's are often overlooked due to being CPU intensive while most people have pretty decent internet speeds. This has a lot to do with how they are implemented. +- With the current vision for Serpent OS deltas they will require fewer CPU instructions to use than full packages, but may slightly increase the time to cache some packages (but we are talking ms). But we haven't even considered the reduction in time taken to download the delta vs the full package which more than makes up the difference! +- The memory requirements are reduced compared to the prior delta approach, especially if you factor in extracting the payload in memory (possibly using `tmpfs`) as part of installation. + +## Getting More Bang for Your Buck + +There's still plenty more work to be done for implementing delta's in Serpent OS and they likely aren't that helpful early on. To make delta packages sustainable and efficient over the long run, we can make them even better and reduce some wastage. Here are some more ideas in how to make deltas less resource intensive and better for users: + +- As we delta each file individually, we could easily use two or more threads to speed up caching time. Using this package as an example, two threads would reduce the caching time to 334ms, the time the largest file took to recreate plus the time to extract the payload. Now the delta takes less time and CPU to cache than the full package! +- `zstd` gives us options to tradeoff some increase in delta size to reduce the resources needed to create delta packages. This testing was performed with `--ultra -22 --zstd=chainLog=30` which is quite slow, but produces the smallest files. Even reducing the compression level to `-16 --long` didn't result in a large increase in delta size. +- We always have the option to not create deltas for small packages to ease the burden, but in reality the biggest overhead is created from large files. +- When creating deltas, you typically generate them for multiple prior releases. We can use smart criteria when to stop delta generation from earlier releases for instance if they save less than 10% total size or less than 100KB. A delta against an earlier release will almost always be larger than versus a more recent release. + +## Even Better than These Numbers Suggest + +While the numbers included have been nothing short of remarkable, they don't quite reflect how good this approach will be. The results shown lack some of the key advantages of our delta approach such as excluding files that are unchanged between the two packages. Other things that will show better results are: + +- When package updates include minimal changes between versions (and smaller files), we would expect the average package to be much closer in elapsed time than indicated in these results. +- A quick test using a delta of two package builds of the same source version resulted in a 13MB delta (rather than the 28.6MB delta we had here). On top of that it took 62% fewer CPU instructions and less time (295ms) than the full package to extract (349ms) without resorting to multi-threading. +- Delta creation of the average package will be quite fast where the largest files are < 30MB. In our example, one file is 76% of the package size (126MB) but can take nearly 95% of the total creation time! +- We will be applying features to our packages that will reduce file sizes (such as the much smaller RELR relocations and identical code folding), making the approach even better, but that will be for a future blog post. diff --git a/src/content/docs/blog/2022/02/11/making-deltas-great-again-1/index.mdx b/src/content/docs/blog/2022/02/11/making-deltas-great-again-1/index.mdx deleted file mode 100644 index d46d58d2..00000000 --- a/src/content/docs/blog/2022/02/11/making-deltas-great-again-1/index.mdx +++ /dev/null @@ -1,243 +0,0 @@ ---- -title: "Making Deltas Great Again! (Part 1)" -date: 2022-02-11T20:45:12+11:00 -draft: false -authors: [sunnyflunk] -tags: [news] -license: "CC-BY-ND-4.0" -copyright: "Copyright © 2022 aerynOS Developers" ---- -import FigureScreenshotOne from "@/components/ui/FigureScreenshotOne.astro" -import featuredImage from "./Featured.webp" - -In [Optimising Package Distribution](/blog/2021/03/16/optimising-package-distribution) we discussed some early -findings for implementing binary deltas in Serpent OS. While discussing the implementation we have found the -requirements to be suboptimal for what we were after. We provide a fresh look at the issue and what we can do to make it -useful in almost all situations without the drawbacks. - - - -## Deltas Have a Bad Reputation - -I remember back in the early 2000s on Gentoo when someone set up a server to produce delta patches from source tarballs -to distribute to users with small data caps such as myself. When requesting the delta, the job would be added to the -queue (which occasionally could be minutes) and then created a small patch to download. This was so important at the -time to reduce the download size that the extra time was well worth it! - -Today things are quite different. The case for deltas has reduced for users as internet speeds have increased. Shrinking -20MB off a package size may be a second reduction for some, but 10 seconds for others. The largest issue is that deltas -have typically pushed extra computational effort onto the users computer in compensation for the smaller size. With a -fast internet connection that cost is a real burden where deltas take longer to install than simply downloading the full -package. - -## What's so Bad About the Old Method? - -The previous idea of using the full payload for deltas was very efficient in terms of distribution, but required changes -in how `moss` handles packages to implement it. Having the previous payload available (and being fast) means storing the -old payloads on disk. This increases the storage requirements for the base system, although that can be reduced by -compressing the payloads to reduce disk usage (but increasing CPU usage at install time). - -To make it work well, we needed the following requirements: -- Install all the files from a package and recreate the payload from the individual files. However, Smart System - Management allows users to avoid installing unneeded locale files and we would not be able to recreate the full - payload without them. -- Alternatively we can store the full payloads on disk. Then there becomes a tradeoff from doubling storage space or - additional overhead from compressing the payloads to reduce it. -- Significant memory requirements to use a delta when a package is large. - -In short we weren't happy with having to increase the disk storage requirements (potentially more than 2x) or the -increase in CPU time to create compressed payloads to minimize it. This was akin to the old delta model, of smaller -downloads but significantly slower updates. - -## Exploring an Alternative Approach - -Optimal compression generally benefits from combining multiple files into one archive than to compress each file -individually. With `zstd` deltas, since you read in a dictionary (the old file), you already have good candidates for -compression matching. The real question was simply whether creating a delta for each file was a viable method for delta -distribution (packaged into a single payload of course). - -As the real benefit of deltas is reducing download size, the first thing to consider is the size impact. Using the same -package as the previous blog post (but with newer versions of `QtWebEngine` and `zstd`) we look at what happens when you -delta each file individually. Note that the package is quite unique in that the largest file is 76% of the total package -size and that delta creation time increased a lot for files larger than 15-20MB at maximum compression. - -| | Full Tarball | Individual Files | Full Tarball --zstd=chainLog=30 | Individual Files --zstd=chainLog=30 | -|-----------------|--------------|------------------|---------------------------------|-------------------------------------| -| Time to create | 134.6s | 137.6s | 157.9s | 150.9s | -| Size of delta | 30.8MB | 29.8MB | 28.3MB | 28.6MB | -| Peak Memory | 1.77GB | 1.64GB | 4.77GB | 2.64GB | - -Quite surprisingly, the delta sizes were very close! Most surprising was that without increasing the size of the -chainLog in `zstd`, the individual file approach actually resulted in smaller deltas! We can also see how much lower the -memory requirements were (and they would be much smaller when there isn't one large file in the package!). Our locale -and doc trimming of files will still work nicely, as we don't need to recreate the locale files that aren't installed -(as we still don't want them!). - -The architecture of `moss` allows us to cache packages, where all cached packages are available for generating multiple -system roots including with our build tool `boulder` without any need for the original package file. Therefore any need -to retain old payloads or packages is no longer required or useful, eliminating the drawbacks of the previous delta -approach. The memory requirements are also reduced as the maximum memory requirement scales with the size of the largest -file, rather than the entire package (which is generally a lot bigger). There are many packages containing hundreds of -MBs of uncompressed data and a few into the GBs. But the largest file I could find installed locally was only 140MB, and -only a handful over 100MB. This smaller increase in memory requirements is a huge improvement and the small increase in -memory use to extract the deltas is likely to go unnoticed by users. - -Well everything sounds pretty positive so far, there must be some drawback? - - - -## The Impact on Package Cache Time - -As the testing method for this article is quite simplistic (`bash` loops and calls to `zstd` directly), the additional -overhead from creating deltas for individual files I estimated to be about 20ms compared to a proper solution. The main -difference from the old delta method is how we extract the payloads and recreate the files of the new package. Using the -full package you simply extract the content payload and split it into its corresponding files. The new approach requires -two steps, extracting the payload (we could in theory not compress it) and then applying patches to the original files -to recreate the new ones. Note that times differ slightly from the previous table due to minor variations between test -runs. - -| | Normal Package | Individual Delta File Package | -|--------------------------------------|------------------|-------------------------------| -| Time to Delta Files | - | 148.0s (137 files) | -| Time to Compress Payload | 78.6s | 4.0s | -| Size of Uncompressed Payload | 165.8MB | 28.9MB | -| Size of Compressed Payload | 51.3MB | 28.6MB | -| Instructions to Extract Payload | 2,876.9m (349ms) | 33.1m (34ms) | -| Instructions to Recreate Files | - | 1,785.6m (452ms) | -| | | | -| Total instructions to Extract | 2,876.9m (349ms) | 1,818.7m (486ms) | - -What's important to note is that is this reflects a worst case scenario for the delta approach, where all 137 files were -different between the old and new version of the package. Package updates where files remain unchanged allows us to omit -them from the delta package altogether! So the delta approach not only saves time downloading files, but also requires -fewer CPU instructions to apply the update. It's not quite a slam dunk though as reading the original file as a -dictionary results in an increase in elapsed time of extraction (though the extra time is likely much less than the time -saved downloading 20MB less!). - -In Part 2 we will look at some ways we can tweak the approach to balance the needed resources for creating delta -packages and to reduce the time taken to apply them. - -``` - -Note: This was intended to be a 2 part series as it contains a lot of information to digest. -However, Part 2 was committed and follows below. - -``` - -There's more than one way to create a delta! This post follows on from the earlier analysis of creating some basic -delta packages. Where this approach, and the use of `zstd`, really thrives is that it gives us options in how to manage -the overhead, from creating the deltas to getting them installed locally. Next we explore some ideas of how we can -minimize the caching time of delta files. - -## Improving Delta Cache Time - -To get the best bang for your buck with deltas, it is essential to reduce the size of the larger files. My experience in -testing was that there wasn't a significant benefit from creating deltas for small files. In this example, we only -create delta files when they are larger than a specific size while including the full version of files that are under -the cutoff. This reduces the number of delta files created without impacting the overall package size by much. - -| Only Delta Files Greater Than | Greater than 10KB | Greater than 100KB | Greater than 500KB | -|--------------------------------------|-------------------|--------------------|--------------------| -| Time to Delta Files | 146.1s (72 files) | 146.3s (64 files) | 139.4s (17 files) | -| Time to Compress Payload | 3.9s | 4.0s | 8.3s | -| Size of Uncompressed Payload | 28.9MB | 29.3MB | 42.4MB | -| Size of Compressed Payload | 28.6MB | 28.7MB | 30.5MB | -| Instructions to Extract Payload | 37.8m (36ms) | 34.7m (29ms) | 299.1m (66ms) | -| Instructions to Recreate Files | 1,787.7m (411ms) | 1,815.0m (406ms) | 1,721.4m (368ms) | -| | | | | -| Total instructions to Extract | 1,825.5m (447ms) | 1,849.7m (435ms) | 2,020.5m (434ms) | - -Here we see that by not creating deltas for files under 100KB, it barely impacts the size of the delta at all, while -reducing caching time by 50ms compared to creating a delta for every file (486ms from the previous blog post). It even -leads to up to a 36% reduction in CPU instructions to undertake caching through the delta than using the full package. -In terms of showing how effective this delta technique really is, I chose one of the worst examples and I would expect -that many deltas would be faster to cache when there's files that are exact matches between the old and new package. The -largest file alone took 300ms to apply the delta, where overheads tend to scale a lot when you start getting to larger -files. - -There are also some steps we can take to make sure that caching a delta is almost always faster than the full package -(solving the only real drawback to users), only requiring Serpent OS resources to create these delta packages. - -## Creating More Efficient Deltas - -For this article, all the tests have been run with `zstd --ultra -22 --zstd=chainLog=30`...until now! The individual -file delta approach is more robust at lower compression levels to keep package size small while reducing how long they -take to create. Lets take a look at the difference while also ensuring `--long` is enabled. This testing combined with -the results above for only creating deltas for files larger than 10KB. - -| Individual Delta File Package | zstd -12 | zstd -16 | zstd -19 | zstd -22 | -|--------------------------------------|------------------|------------------|------------------|------------------| -| Time to Delta Files | 6.7s | 113.9s | 142.3s | 148.3s | -| Time to Compress Payload | 0.5s | 3.2s | 5.3s | 4.0s | -| Size of Uncompressed Payload | 41.1MB | 30.6MB | 28.9MB | 28.9MB | -| Size of Compressed Payload | 40.9MB | 30.3MB | 28.6MB | 28.6MB | -| Instructions to Extract Payload | 46.5m (35ms) | 51.2m (28ms) | 42.6m (33ms) | 37.8m (36ms) | -| Instructions to Recreate Files | 1,773.7m (382ms) | 1,641.5m (385ms) | 1,804.2m (416ms) | 1,810.9m (430ms) | -| | | | | | -| Total instructions to Extract | 1,820.2m (417ms) | 1,692.7m (413ms) | 1,846.8m (449ms) | 1,848.7m (466ms) | - -Compression levels 16-19 look quite interesting where you start to reduce the time taken to apply the delta as well -and only seeing a small increase in size. For comparison, at `-19` it only took 9s to delta the remaining 39.8MB of -files when excluding the largest file (it was 15s at `-22`). While the time taken between 19 and 22 was almost the same, -at `-19` it took 27% fewer instructions to create the deltas than at `-22` (`-16` uses 64% fewer instructions than -`-22`). It will need testing across a broader range of packages to see the overall impact and to evaluate a sensible -default. - -As a side effect of reducing the compression level, you also get another small decrease in the time to cache a package. -The best news of all is that these numbers are already out of date. Testing was performed last year with `zstd` 1.5.0, -where there have been notable speed improvements to both compression and decompression that have been included in newer -releases. Great news given how fast it is already! Here's a quick summary of how it all ties together. - -## Delta's Are Back! - -This blog series has put forward a lot of data that might be difficult to digest...but what does it mean for users of -Serpent OS? Here's a quick summary of the advantages of using this delta method on individual files when compared to -fetching the full packages: - -- Package update sizes are greatly reduced to speed up fetching of package updates. -- `moss` architecture means that we have no need to leave packages on disk for later use, reducing the storage - footprint. In fact, we could probably avoid writes (except for the extracted package of course!) by downloading - packages to `tmpfs` where you have sufficient free memory. -- Delta's can be used for updating packages for packaging **and** your installed system. There's no need for a full - copy of the full original package for packaging. A great benefit when combined with the source repository. -- Delta's are often overlooked due to being CPU intensive while most people have pretty decent internet speeds. This has - a lot to do with how they are implemented. -- With the current vision for Serpent OS deltas they will require fewer CPU instructions to use than full packages, but - may slightly increase the time to cache some packages (but we are talking ms). But we haven't even considered the - reduction in time taken to download the delta vs the full package which more than makes up the difference! -- The memory requirements are reduced compared to the prior delta approach, especially if you factor in extracting the - payload in memory (possibly using `tmpfs`) as part of installation. - -## Getting More Bang for Your Buck - -There's still plenty more work to be done for implementing delta's in Serpent OS and they likely aren't that helpful -early on. To make delta packages sustainable and efficient over the long run, we can make them even better and reduce -some wastage. Here are some more ideas in how to make deltas less resource intensive and better for users: - -- As we delta each file individually, we could easily use two or more threads to speed up caching time. Using this - package as an example, two threads would reduce the caching time to 334ms, the time the largest file took to recreate - plus the time to extract the payload. Now the delta takes less time and CPU to cache than the full package! -- `zstd` gives us options to tradeoff some increase in delta size to reduce the resources needed to create delta - packages. This testing was performed with `--ultra -22 --zstd=chainLog=30` which is quite slow, but produces the - smallest files. Even reducing the compression level to `-16 --long` didn't result in a large increase in delta size. -- We always have the option to not create deltas for small packages to ease the burden, but in reality the biggest - overhead is created from large files. -- When creating deltas, you typically generate them for multiple prior releases. We can use smart criteria when to stop - delta generation from earlier releases for instance if they save less than 10% total size or less than 100KB. A delta - against an earlier release will almost always be larger than versus a more recent release. - -## Even Better than These Numbers Suggest - -While the numbers included have been nothing short of remarkable, they don't quite reflect how good this approach will -be. The results shown lack some of the key advantages of our delta approach such as excluding files that are unchanged -between the two packages. Other things that will show better results are: - -- When package updates include minimal changes between versions (and smaller files), we would expect the average package - to be much closer in elapsed time than indicated in these results. -- A quick test using a delta of two package builds of the same source version resulted in a 13MB delta (rather than the - 28.6MB delta we had here). On top of that it took 62% fewer CPU instructions and less time (295ms) than the full - package to extract (349ms) without resorting to multi-threading. -- Delta creation of the average package will be quite fast where the largest files are < 30MB. In our example, one file - is 76% of the package size (126MB) but can take nearly 95% of the total creation time! -- We will be applying features to our packages that will reduce file sizes (such as the much smaller RELR relocations - and identical code folding), making the approach even better, but that will be for a future blog post. diff --git a/src/content/docs/blog/2022/04/05/relr-brings-smaller-files-more-performance/index.md b/src/content/docs/blog/2022/04/05/relr-brings-smaller-files-more-performance/index.md new file mode 100644 index 00000000..5e1a948b --- /dev/null +++ b/src/content/docs/blog/2022/04/05/relr-brings-smaller-files-more-performance/index.md @@ -0,0 +1,81 @@ +--- +title: "RELR Brings Smaller Files, More Performance?" +date: 2022-04-05T11:12:54+10:00 +draft: false +authors: [sunnyflunk] +tags: [news] +featuredImage: "/img/blog/relr-brings-smaller-files-more-performance/Featured.webp" +license: "CC-BY-ND-4.0" +copyright: "Copyright © 2022 aerynOS Developers" +--- + +`RELR` is an efficient method of storing relative relocations (but is not yet available in `glibc` upstream). This has a significant reduction on file size often in the vicinity of 5% for libraries and even higher for PIE binaries. We also take a look at the performance impact on enabling `RELR` and that looks really good too! Smaller files with more performance - you **can** have your cake and eat it too! + +## Delicious Size Savings + +Everyone enjoys smaller files, especially when it's for free! `RELR` provides a very efficient method of storing relative relocations where it only requires a few percent compared to storing them in the `.rela.dyn` section which is what currently happens. However, it can't store everything so the `.rela.dyn` section remains (though much smaller). + +Here's an example of the sections of `libLLVM-13.so` with and without `RELR`. Here we see the `.rela.dyn` section taking up a whole 2MB! When enabling `RELR`, `.rela.dyn` shrinks down to just under 100KB while adding a new section `.relr.dyn` which is just over 20KB! That's nearly a 1.9MB file size reduction, so you'll get smaller packages, smaller updates and it will be even faster to create delta packages from our servers. For reference, some of the biggest files have a `.rela.dyn` section over 10MB! + +| Section | Without RELR | With RELR | +|:---------:|--------------|------------| +| .dynstr | 2,285,738 | 2,285,723 | +| .rela.dyn | 2,006,472 | 97,464 | +| .relr.dyn | - | 21,688 | +| .text | 28,708,290 | 28,708,386 | +| .dynamic | 592 | 608 | +| | | | +| Total | 44,853,987 | 42,966,764 | + +## Smaller, But at What Cost? + +While most of the discussion about `RELR` is around the size savings, there's been very little in terms of the performance numbers of enabling `RELR`. For most things, it's not going to make a noticeable difference, as it should only really impact loading of binaries. There's one rule we have and that's to measure everything! We care about every little detail where many 1-2% improvements can add up to significant gains. + +First, we require a test to determine if we could detect changes between an `LLVM` built with `RELR` and one without. The speed of the compiler is vital to a distro, where lackluster performance of the build system hurts all contributors and anyone performing source based builds. `clang` in this example was built using a shared `libLLVM` so that it would load the `RELR` section and it's large enough to be able to measure a difference in load times (if one exists). Building `gettext` was the chosen test (total time includes tarball extraction, configure, make and make install stages), rather than a synthetic binary loading test to reflect real world usage. The configure stage is very long when building `gettext` so `clang` is called many times for short compiles. Lets take a look at the results: + +``` +no RELR: +[Build] Finished: 1 minute, 57 secs, 80 ms, 741 μs, and 2 hnsecs +[Build] Finished: 1 minute, 57 secs, 691 ms, 586 μs, and 4 hnsecs +[Build] Finished: 1 minute, 56 secs, 861 ms, 31 μs, and 8 hnsecs + +RELR: +[Build] Finished: 1 minute, 55 secs, 244 ms, 213 μs, and 8 hnsecs +[Build] Finished: 1 minute, 55 secs, 400 ms, 158 μs, and 8 hnsecs +[Build] Finished: 1 minute, 55 secs, 775 ms, 40 μs, and 8 hnsecs + +RELR+startup patch: +[Build] Finished: 1 minute, 54 secs, 979 ms, 166 μs, and 8 hnsecs +[Build] Finished: 1 minute, 54 secs, 820 ms, and 675 μs +[Build] Finished: 1 minute, 54 secs, 713 ms, 440 μs, and 3 hnsecs +``` + +Here we see the base configuration was able to build `gettext` in 117.21s on average. When we enabled `RELR` in our `LLVM` build (all other packages were without `RELR` still), the average build time decreased by 1.74s! That does not sound like a lot, but the time spent loading `clang` would only be a portion of the total, yet still gives a 1-2% performance lift over the whole build. While we were reducing start up time, I ran another test, but this time adding a patch to reduce paths searched on startup as well as enabling `RELR`. This patch reduced the average build time by a further 0.63s! + +That's a 2.37s reduction in the build just from improving the `clang` binary's load time. + +## What This Means - RELR by Default + +So what actually is `RELR`? I can't really do the topic justice, so will point you to a great blog post about RELR, [Relative Relocations and RELR](https://maskray.me/blog/2021-10-31-relative-relocations-and-relr). It's quite technical for the average reader, but definitely worth a read if you like getting into the details. To no surprise the author (Fangrui Song) started the initial push for getting `RELR` support upstream in `glibc` (at the time of this post the patch series has not yet been committed to `glibc` git). + +What I can tell you, is that we've applied the requisite patches for `RELR` support and enabled `RELR` by default in `boulder` for builds. Our container has been rebuilt and all is working well with `RELR` enabled. More measurements will be done in future in the same controlled manner, particularly around PIE load times. + +## Caveats - The Hidden Details + +The performance benchmark was quite limited in terms of being an optimal case for `RELR` as `clang` is called thousands of times in the build so on average improved load time by about 0.6-0.7ms. We can presume that using `RELR` on smaller files is unlikely to regress load times. It definitely gives us confidence that it would be about the same or better in most situations, but not noticeable or measurable in most use cases. Minimizing build times is a pretty significant target for us, so even these small gains are appreciated. + +The size savings can vary between packages and not everything can be converted into the `.relr.dyn` section. The current default use of `RELR` is not without cost as it adds a version dependency on `glibc`. We will ensure we ship a sane implementation that minimizes or removes such overhead. + +It was also not straight forward to utilize `RELR` in Serpent. The pending upstream `glibc` [patch series](https://sourceware.org/pipermail/libc-alpha/2022-February/136290.html) included a patch which caused issues when enabling `RELR` in Serpent OS (patch 3/5). As we utilize two toolchains, `gcc/bfd` and `clang/lld`, both need to function independently to create outputs of a functional OS. However the part "Issue an error if there is a DT_RELR entry without GLIBC_ABI_DT_RELR dependency nor GLIBC_PRIVATE definition." meant that `glibc` would **refuse to load files linked by `lld` despite having the ability to load them**. `lld` has supported `RELR` for some time already, but does not create the `GLIBC_ABI_DT_RELR` dependency that is required by `glibc`. I have added my [feedback](https://sourceware.org/pipermail/libc-alpha/2022-March/136773.html) to the patch set upstream. `lld` now has support for this version dependency upstream if we ever decide to use it in future. + +After dropping the patch and patching `bfd` to no longer generate the `GLIBC_ABI_DT_RELR` dependency either, I was finally able to build both `glibc` and `LLVM` with the same patches. With overcoming that hurdle, rebuilding the rest of the repository went without a hitch, so we are now enjoying `RELR` in all of our builds and is enabled by default. + +There is even further scope for more size savings, by switching the `rela.dyn` section for the `rel.dyn` section (this is what is used for 32-bit builds and one of the reasons files are smaller!). `lld` supports switching the section type, but I don't believe `glibc` will read the output as it expects the `psABI` specified section (something `musl` can handle though). + +![34 wasted bytes with GLIBC_ABI_DT_RELR](./Featured.webp "34 wasted bytes with GLIBC_ABI_DT_RELR") + +## The Cost of Adding GLIBC_ABI_DT_RELR + +A quick check of two equivalent builds (one adding the `GLIBC_ABI_DT_RELR` version dependency and one not), there was an increase of 34 bytes to the file's sections (18 bytes to `.dynstr` and 16 bytes to `.gnu.version_r`). It also means having to validate that the `GLIBC_ABI_DT_RELR` version is present in the `libc` and that the file using `RELR` includes this version dependency. This may not sound like much but it is completely unnecessary! Note that the testing provided in this blog post is without `GLIBC_ABI_DT_RELR`. + +Regardless of what eventuates, these negatives won't ship in Serpent OS. This will allow for us to support files that include the version dependency (when appimage and other distros catch up) as it will still exist in `libc`, but we won't have the version check in files, nor will `glibc` check that the version exists before loading for packages built by `boulder`. diff --git a/src/content/docs/blog/2022/04/05/relr-brings-smaller-files-more-performance/index.mdx b/src/content/docs/blog/2022/04/05/relr-brings-smaller-files-more-performance/index.mdx deleted file mode 100644 index 8f786562..00000000 --- a/src/content/docs/blog/2022/04/05/relr-brings-smaller-files-more-performance/index.mdx +++ /dev/null @@ -1,140 +0,0 @@ ---- -title: "RELR Brings Smaller Files, More Performance?" -date: 2022-04-05T11:12:54+10:00 -draft: false -authors: [sunnyflunk] -tags: [news] -featuredImage: "/img/blog/relr-brings-smaller-files-more-performance/Featured.webp" -license: "CC-BY-ND-4.0" -copyright: "Copyright © 2022 aerynOS Developers" ---- -import FigureScreenshotOne from "@/components/ui/FigureScreenshotOne.astro" -import featuredImage from "./Featured.webp" - -`RELR` is an efficient method of storing relative relocations (but is not yet available in `glibc` upstream). This has a -significant reduction on file size often in the vicinity of 5% for libraries and even higher for PIE binaries. We also -take a look at the performance impact on enabling `RELR` and that looks really good too! Smaller files with more -performance - you **can** have your cake and eat it too! - - - -## Delicious Size Savings - -Everyone enjoys smaller files, especially when it's for free! `RELR` provides a very efficient method of storing -relative relocations where it only requires a few percent compared to storing them in the `.rela.dyn` section which is -what currently happens. However, it can't store everything so the `.rela.dyn` section remains (though much smaller). - -Here's an example of the sections of `libLLVM-13.so` with and without `RELR`. Here we see the `.rela.dyn` section taking -up a whole 2MB! When enabling `RELR`, `.rela.dyn` shrinks down to just under 100KB while adding a new section -`.relr.dyn` which is just over 20KB! That's nearly a 1.9MB file size reduction, so you'll get smaller packages, smaller -updates and it will be even faster to create delta packages from our servers. For reference, some of the biggest files -have a `.rela.dyn` section over 10MB! - -| Section | Without RELR | With RELR | -|:---------:|--------------|------------| -| .dynstr | 2,285,738 | 2,285,723 | -| .rela.dyn | 2,006,472 | 97,464 | -| .relr.dyn | - | 21,688 | -| .text | 28,708,290 | 28,708,386 | -| .dynamic | 592 | 608 | -| | | | -| Total | 44,853,987 | 42,966,764 | - -## Smaller, But at What Cost? - -While most of the discussion about `RELR` is around the size savings, there's been very little in terms of the -performance numbers of enabling `RELR`. For most things, it's not going to make a noticeable difference, as it should -only really impact loading of binaries. There's one rule we have and that's to measure everything! We care about every -little detail where many 1-2% improvements can add up to significant gains. - -First, we require a test to determine if we could detect changes between an `LLVM` built with `RELR` and one without. -The speed of the compiler is vital to a distro, where lackluster performance of the build system hurts all contributors -and anyone performing source based builds. `clang` in this example was built using a shared `libLLVM` so that it would -load the `RELR` section and it's large enough to be able to measure a difference in load times (if one exists). Building -`gettext` was the chosen test (total time includes tarball extraction, configure, make and make install stages), rather -than a synthetic binary loading test to reflect real world usage. The configure stage is very long when building -`gettext` so `clang` is called many times for short compiles. Lets take a look at the results: - -``` -no RELR: -[Build] Finished: 1 minute, 57 secs, 80 ms, 741 μs, and 2 hnsecs -[Build] Finished: 1 minute, 57 secs, 691 ms, 586 μs, and 4 hnsecs -[Build] Finished: 1 minute, 56 secs, 861 ms, 31 μs, and 8 hnsecs - -RELR: -[Build] Finished: 1 minute, 55 secs, 244 ms, 213 μs, and 8 hnsecs -[Build] Finished: 1 minute, 55 secs, 400 ms, 158 μs, and 8 hnsecs -[Build] Finished: 1 minute, 55 secs, 775 ms, 40 μs, and 8 hnsecs - -RELR+startup patch: -[Build] Finished: 1 minute, 54 secs, 979 ms, 166 μs, and 8 hnsecs -[Build] Finished: 1 minute, 54 secs, 820 ms, and 675 μs -[Build] Finished: 1 minute, 54 secs, 713 ms, 440 μs, and 3 hnsecs -``` - -Here we see the base configuration was able to build `gettext` in 117.21s on average. When we enabled `RELR` in our -`LLVM` build (all other packages were without `RELR` still), the average build time decreased by 1.74s! That does not -sound like a lot, but the time spent loading `clang` would only be a portion of the total, yet still gives a 1-2% -performance lift over the whole build. While we were reducing start up time, I ran another test, but this time adding a -patch to reduce paths searched on startup as well as enabling `RELR`. This patch reduced the average build time by a -further 0.63s! - -That's a 2.37s reduction in the build just from improving the `clang` binary's load time. - -## What This Means - RELR by Default - -So what actually is `RELR`? I can't really do the topic justice, so will point you to a great blog post about RELR, -[Relative Relocations and RELR](https://maskray.me/blog/2021-10-31-relative-relocations-and-relr). It's quite technical -for the average reader, but definitely worth a read if you like getting into the details. To no surprise the author -(Fangrui Song) started the initial push for getting `RELR` support upstream in `glibc` (at the time of this post the -patch series has not yet been committed to `glibc` git). - -What I can tell you, is that we've applied the requisite patches for `RELR` support and enabled `RELR` by default in -`boulder` for builds. Our container has been rebuilt and all is working well with `RELR` enabled. More measurements will -be done in future in the same controlled manner, particularly around PIE load times. - -## Caveats - The Hidden Details - -The performance benchmark was quite limited in terms of being an optimal case for `RELR` as `clang` is called thousands -of times in the build so on average improved load time by about 0.6-0.7ms. We can presume that using `RELR` on smaller -files is unlikely to regress load times. It definitely gives us confidence that it would be about the same or better in -most situations, but not noticeable or measurable in most use cases. Minimizing build times is a pretty significant -target for us, so even these small gains are appreciated. - -The size savings can vary between packages and not everything can be converted into the `.relr.dyn` section. The current -default use of `RELR` is not without cost as it adds a version dependency on `glibc`. We will ensure we ship a sane -implementation that minimizes or removes such overhead. - -It was also not straight forward to utilize `RELR` in Serpent. The pending upstream `glibc` -[patch series](https://sourceware.org/pipermail/libc-alpha/2022-February/136290.html) included a patch which caused -issues when enabling `RELR` in Serpent OS (patch 3/5). As we utilize two toolchains, `gcc/bfd` and `clang/lld`, both -need to function independently to create outputs of a functional OS. However the part "Issue an error if there is a -DT_RELR entry without GLIBC_ABI_DT_RELR dependency nor GLIBC_PRIVATE definition." meant that `glibc` would **refuse to -load files linked by `lld` despite having the ability to load them**. `lld` has supported `RELR` for some time already, -but does not create the `GLIBC_ABI_DT_RELR` dependency that is required by `glibc`. I have added my -[feedback](https://sourceware.org/pipermail/libc-alpha/2022-March/136773.html) to the patch set upstream. `lld` now has -support for this version dependency upstream if we ever decide to use it in future. - -After dropping the patch and patching `bfd` to no longer generate the `GLIBC_ABI_DT_RELR` dependency either, I was -finally able to build both `glibc` and `LLVM` with the same patches. With overcoming that hurdle, rebuilding the rest of -the repository went without a hitch, so we are now enjoying `RELR` in all of our builds and is enabled by default. - -There is even further scope for more size savings, by switching the `rela.dyn` section for the `rel.dyn` section (this -is what is used for 32-bit builds and one of the reasons files are smaller!). `lld` supports switching the section type, -but I don't believe `glibc` will read the output as it expects the `psABI` specified section (something `musl` can -handle though). - - - -## The Cost of Adding GLIBC_ABI_DT_RELR - -A quick check of two equivalent builds (one adding the `GLIBC_ABI_DT_RELR` version dependency and one not), there was an -increase of 34 bytes to the file's sections (18 bytes to `.dynstr` and 16 bytes to `.gnu.version_r`). It also means -having to validate that the `GLIBC_ABI_DT_RELR` version is present in the `libc` and that the file using `RELR` includes -this version dependency. This may not sound like much but it is completely unnecessary! Note that the testing provided -in this blog post is without `GLIBC_ABI_DT_RELR`. - -Regardless of what eventuates, these negatives won't ship in Serpent OS. This will allow for us to support files that -include the version dependency (when appimage and other distros catch up) as it will still exist in `libc`, but we won't -have the version check in files, nor will `glibc` check that the version exists before loading for packages built by -`boulder`. diff --git a/src/content/docs/blog/2022/06/06/a-word-from-the-founder/index.md b/src/content/docs/blog/2022/06/06/a-word-from-the-founder/index.md new file mode 100644 index 00000000..5e1f6c4b --- /dev/null +++ b/src/content/docs/blog/2022/06/06/a-word-from-the-founder/index.md @@ -0,0 +1,23 @@ +--- +title: "A Word From The Founder" +date: 2022-06-06T20:36:51+01:00 +draft: false +authors: [ikey] +tags: [news] +license: "CC-BY-ND-4.0" +copyright: "Copyright © 2022 aerynOS Developers" +--- + +Well well, it's been a long time since I personally wrote a post.. 😀 So let's keep this short and sweet, shall we? I'm returning to full time work on Serpent OS. + +The 6th of July will be my last day at my current employment having tendered my 30 day notice today. Despite having enjoyment at my current position, the reality is that my passion and focus is Serpent OS. + +I'm now in a transition process and will ramp up my efforts with Serpent OS. Realistically I need to reduce the outgoing costs of the project and with your help I can gain some level of financial support as we move through the next stages of development. Worst case, I will only take on any part-time or contractual gigs, allowing my primary focus to be Serpent OS. + +I'll begin accelerating works and enabling community contribution so we can get the derailed-alpha train back on the tracks. + +I have absolute faith in this project, the community and our shared ability to deliver the OS and tooling. To achieve it will require far more of my time and I'm perfectly willing to give it. + +Thank you all to everyone who has been supporting the project, it is now time to deliver. Not just another run of the mill distribution but a technically competent and usable distribution that is not only different but *better*. + +Let's do this in the most grassroots and enjoyable way possible 😊 diff --git a/src/content/docs/blog/2022/06/06/a-word-from-the-founder/index.mdx b/src/content/docs/blog/2022/06/06/a-word-from-the-founder/index.mdx deleted file mode 100644 index 65d04fa3..00000000 --- a/src/content/docs/blog/2022/06/06/a-word-from-the-founder/index.mdx +++ /dev/null @@ -1,37 +0,0 @@ ---- -title: "A Word From The Founder" -date: 2022-06-06T20:36:51+01:00 -draft: false -authors: [ikey] -tags: [news] -license: "CC-BY-ND-4.0" -copyright: "Copyright © 2022 aerynOS Developers" ---- - -Well well, it's been a long time since I personally wrote a post.. :) So let's keep -this short and sweet, shall we? I'm returning to full time work on Serpent OS. - - - -The 6th of July will be my last day at my current employment having tendered my -30 day notice today. Despite having enjoyment at my current position, the reality -is that my passion and focus is Serpent OS. - -I'm now in a transition process and will ramp up my efforts with Serpent OS. -Realistically I need to reduce the outgoing costs of the project and with -your help I can gain some level of financial support as we move through the -next stages of development. Worst case, I will only take on any part-time or -contractual gigs, allowing my primary focus to be Serpent OS. - -I'll begin accelerating works and enabling community contribution so we can -get the derailed-alpha train back on the tracks. - -I have absolute faith in this project, the community and our shared ability -to deliver the OS and tooling. To achieve it will require far more of my time -and I'm perfectly willing to give it. - -Thank you all to everyone who has been supporting the project, it is now -time to deliver. Not just another run of the mill distribution but a technically -competent and usable distribution that is not only different but *better*. - -Let's do this in the most grassroots and enjoyable way possible =) diff --git a/src/content/docs/blog/2022/06/22/packaging-automation-next-steps/index.md b/src/content/docs/blog/2022/06/22/packaging-automation-next-steps/index.md index 0facc5a4..43d9b371 100644 --- a/src/content/docs/blog/2022/06/22/packaging-automation-next-steps/index.md +++ b/src/content/docs/blog/2022/06/22/packaging-automation-next-steps/index.md @@ -8,72 +8,43 @@ license: "CC-BY-ND-4.0" copyright: "Copyright © 2022 aerynOS Developers" --- -Hot damn we've been busy lately. No, [really](https://gitlab.com/groups/serpent-os/-/activity). -The latest development cycle saw us focus exclusively on `boulder`, our build tooling. As of -today it features a proof of concept `boulder new` subcommand for the automatic generation of -packaging templates from an upstream source (i.e. tarball). +Hot damn we've been busy lately. No, [really](https://gitlab.com/groups/serpent-os/-/activity). The latest development cycle saw us focus exclusively on `boulder`, our build tooling. As of today it features a proof of concept `boulder new` subcommand for the automatic generation of packaging templates from an upstream source (i.e. tarball). - +Before we really start this blog post off, I'd like to thank everyone who is supporting the project! All of the OpenCollective[^1] contributions will make it easier for me to work full time on Serpent OS 😀 Much love ❤️ -Before we really start this blog post off, I'd like to thank everyone who is supporting the -project! All of the [OpenCollective](https://opencollective.com/serpent-os) contributions will make it easier for me to work full -time on Serpent OS =) Much love <3 - -![Look at all the buildiness](./Featured.webp) +![Look at all the buildiness](./Featured.webp "Look at all the buildiness") ## But, but build _flavours_ ... -Alright you got me there, certain projects prefer to abstract the configuration, build and -installation of packages and be provided with some kind of hint to the build system, i.e. -manually setting `autotools`, etc. +Alright you got me there, certain projects prefer to abstract the configuration, build and installation of packages and be provided with some kind of hint to the build system, i.e. manually setting `autotools`, etc. -Serpent OS packaging is declarative and well structured, and relies on the use of RPM-style -"macros" for distribution integration and common tasks to ensure consistent packaging. +Serpent OS packaging is declarative and well structured, and relies on the use of RPM-style "macros" for distribution integration and common tasks to ensure consistent packaging. -We prefer a self documenting approach that can be machine validated rather than depending -on introspection at the time of build. Our `stone.yml` format is very flexible and powerful, -with automatic runtime dependencies and package splitting as standard. +We prefer a self documenting approach that can be machine validated rather than depending on introspection at the time of build. Our `stone.yml` format is very flexible and powerful, with automatic runtime dependencies and package splitting as standard. ..Doesn't mean we can't make packaging **even easier** though. ## Build discovery -Pointing boulder at an upstream source will perform a deep analysis of the sources to determine -the build system type, build dependencies, metadata, licensing etc. Right now it's just getting -ready to leave POC stage so it has a few warts, however it does have support for generating -package skeletons for the following build systems: +Pointing boulder at an upstream source will perform a deep analysis of the sources to determine the build system type, build dependencies, metadata, licensing etc. Right now it's just getting ready to leave POC stage so it has a few warts, however it does have support for generating package skeletons for the following build systems: - `cmake` - `meson` - `autotools` -We're adding automation for Perl and Python packaging (and Golang, Rust, etc) so we can enforce consistency, -integration and ease without being a burden on developers. This will greatly reduce the friction -of contribution - allowing anyone to package for Serpent OS. +We're adding automation for Perl and Python packaging (and Golang, Rust, etc) so we can enforce consistency, integration and ease without being a burden on developers. This will greatly reduce the friction of contribution - allowing anyone to package for Serpent OS. -We're also able to automatically discover build time dependencies during analysis and add those -to the skeleton `stone.yml` file. We'll enhance support for other build systems as we go, ensuring -that each new package is as close to done on creation as possible, with review and iteration left -to the developer. +We're also able to automatically discover build time dependencies during analysis and add those to the skeleton `stone.yml` file. We'll enhance support for other build systems as we go, ensuring that each new package is as close to done on creation as possible, with review and iteration left to the developer. ## License compliance -A common pain in the arse when packaging for *any* Linux distribution is ensuring the package -information is *compliant* in terms of licensing. As such we must know all of the licensing -information, as well as FSF and OSI compliance for our continuous integration testing. +A common pain in the arse when packaging for *any* Linux distribution is ensuring the package information is *compliant* in terms of licensing. As such we must know all of the licensing information, as well as FSF and OSI compliance for our continuous integration testing. -...Finding all of that information is truly a painful process when conducted manually. -Thankfully `boulder` can perform analysis of all licensing files within the project to -greatly improve compliance and packaging. +...Finding all of that information is truly a painful process when conducted manually. Thankfully `boulder` can perform analysis of all licensing files within the project to greatly improve compliance and packaging. -Every license listed in a `stone.yml` file must use a valid [SPDX](https://spdx.dev/) identifier, -and be accurate. boulder now scans all license files, looking for matches with both SPDX IDs -as well as fuzzy-matching the text of input licenses to make a best-guess at the license ID. +Every license listed in a `stone.yml` file must use a valid [SPDX](https://spdx.dev/) identifier, and be accurate. boulder now scans all license files, looking for matches with both SPDX IDs as well as fuzzy-matching the text of input licenses to make a best-guess at the license ID. -This has so far been highly accurate and correctly identifies many hundreds of licenses, -ensuring a compliant packaging process with less pain for the developers. Over time we'll -optimise and improve this process to ensure compliance *for* our developers rather than -blocking them. +This has so far been highly accurate and correctly identifies many hundreds of licenses, ensuring a compliant packaging process with less pain for the developers. Over time we'll optimise and improve this process to ensure compliance *for* our developers rather than blocking them. As of today we support the [REUSE](https://reuse.software/) specification for expressing software licenses too! @@ -83,22 +54,16 @@ The next steps are honest-to-goodness exciting for us. Or should I say.. exiting ## bill -Work formally begins now on Bootstrap Bill (Turner). Whilst we did successfully bootstrap Serpent OS -and construct the `Protosnek` repository, the process for that is **not** reproducible as `boulder` -has gone through massive changes in this time. +Work formally begins now on Bootstrap Bill (Turner). Whilst we did successfully bootstrap Serpent OS and construct the `Protosnek` repository, the process for that is **not** reproducible as `boulder` has gone through massive changes in this time. -The new [project](https://gitlab.com/serpent-os/core/bill) will leverage `boulder` and a newly -designed bootstrap process to eliminate all host contamination and bootstrap Serpent OS from -`stone.yml` files, emitting an _immutable_ bootstrap repository. +The new [project](https://gitlab.com/serpent-os/core/bill) will leverage `boulder` and a newly designed bootstrap process to eliminate all host contamination and bootstrap Serpent OS from `stone.yml` files, emitting an _immutable_ bootstrap repository. Layering support will land in `moss` and `boulder` to begin the infrastructure projects. ## Build Submission ("Let's Roll") -The aim is to complete `bill` in a very short time so we can bring some initial infrastructure -online to facilitate the automatic build of submitted build jobs. We'll use this process -to create our live repository, replacing the initial bootstrap repository from bill. +The aim is to complete `bill` in a very short time so we can bring some initial infrastructure online to facilitate the automatic build of submitted build jobs. We'll use this process to create our live repository, replacing the initial bootstrap repository from bill. + +At this point all of the tooling we have will come together to allow us all to very quickly iterate on packaging, polish up `moss` and race towards installed systems with online updates. -At this point all of the tooling we have will come together to allow us all to very -quickly iterate on packaging, polish up `moss` and race towards installed systems with -online updates. +[^1]: Since this blog post, we no longer use OpenCollective. For funding, please see our [Funding Page](/sponsor) and for ISOs, please see our [Download Page](/download). diff --git a/src/content/docs/blog/2022/07/10/infrastructure-update/index.md b/src/content/docs/blog/2022/07/10/infrastructure-update/index.md new file mode 100644 index 00000000..51643207 --- /dev/null +++ b/src/content/docs/blog/2022/07/10/infrastructure-update/index.md @@ -0,0 +1,82 @@ +--- +title: "Infrastructure Update" +date: 2022-07-10T11:26:05+01:00 +draft: false +authors: [ikey] +tags: [news] +license: "CC-BY-ND-4.0" +copyright: "Copyright © 2022 aerynOS Developers" +--- + +Since the last post, I've pivoted to full time work on Serpent OS, which is made all the more possible thanks to everyone supporting us via OpenCollective[^1] ❤️. + +We've been working towards establishing an online infrastructure to support the automation of package **builds**, while revisiting some core components. + +## moss-db + +During the development of the Serpent OS tooling we've been exploring the possibilities of D Lang, picking up new practices and refining our approach as we go. Naturally, some of our older modules are somewhat ... _smelly_. Most noticeable is our `moss-db` module, which was initially intended as a lightweight wrapper around [RocksDB](http://rocksdb.org/). + +In practice that required an encapsulation API written in D around the C API, and our own wrapping on top of that. Naturally, it resulted in a very allocation-heavy implementation that just didn't sit right with us, and due to the absurd complexity of RocksDB was still missing quite a few features. + +## Enter LMDB + +We're now using the [Lightning Memory-Mapped Database](https://www.symas.com/lmdb) as the driver implementation for moss-db. In short, we get rapid reads, ACID transactions, bulk inserts, you name it. Our implementation takes advantage of multiple database indexes (`MDB_dbi`) in LMDB to partition the database into internal components, so that we can provide "buckets", or collections. These internal DBs are used for bucket mapping to permit a key-compaction strategy - iteration of top level buckets and key-value pairs within a bucket. + +## Hat tip, boltdb + +The majority of the API was designed with the [boltdb](https://github.com/boltdb/bolt) API in mind. Additionally it was built with `-preview=dip1000` and `-preview=in` enabled, ensuring safely scoped memory use and no room for memory lifetime issues. While we prefer the use of generics, the API is built with `immutable(ubyte[]`) as the internal key and value type. + +Custom types can simply implement `mossEncode` or `mossDecode` to be instantly serialisable into the database as keys, values or bucket identifiers. + +Example API usage: + +```d +Database db; +/* setup the DB with lmdb:// URI */ + +/* Write transaction */ +auto err = db.update((scope tx) @safe +{ + auto bucket = tx.bucket("letters"); + return tx.set(bucket, "a", 1); +}); + +/* do something with the error */ + +err = db.view((in tx) @safe +{ + foreach (name, bucket ; tx.buckets!int) + { + foreach (key, value ; tx.iterator!(string,string)(bucket)) + { + /* do something with the key value pairs, decoded as strings */ + } + } + + /* WILL NOT COMPILE. tx is const scope ref :) */ + tx.removeBucket("numbers"); + + return NoDatabaseError; +} +``` + +## Next for moss + +Moss will be ported to the new DB API and we'll gather some performance metrics, while implementing features like expired state garbage collection (disk cleanup), searching for names/descriptions, etc. + +## Avalanche + +![Early version of avalanche, in development](./Featured.webp "Early version of avalanche, in development") + +Avalanche is a core component of our upcoming infrastructure, providing the service for running builds on a local node, and a controller to coordinate a group of builders. + +Summit will be the publicly accessible project dashboard, and will be responsible for coordinating incoming builds to Avalanche controllers and repositories. Developers will submit builds to Summit and have them dispatched correctly. + +So far we have the core service process in place for the Controller + Node, and now we're working on persistence and handshake. TLDR; fancy use of moss-db and JSON Web tokens over mandated SSL. This means our build infra will be scalable from day 1 allowing multiple builders to be online very early on. + +## Timescale + +We're planning to get an early version of our infrastructure up and running within the next 2 weeks, and get builds flowing =) + +--- +[^1]: Since this blog post, we no longer use OpenCollective. For funding, please see our [Funding Page](/sponsor) and for ISOs, please see our [Download Page](/download). diff --git a/src/content/docs/blog/2022/07/10/infrastructure-update/index.mdx b/src/content/docs/blog/2022/07/10/infrastructure-update/index.mdx deleted file mode 100644 index 96c96f0e..00000000 --- a/src/content/docs/blog/2022/07/10/infrastructure-update/index.mdx +++ /dev/null @@ -1,106 +0,0 @@ ---- -title: "Infrastructure Update" -date: 2022-07-10T11:26:05+01:00 -draft: false -authors: [ikey] -tags: [news] -license: "CC-BY-ND-4.0" -copyright: "Copyright © 2022 aerynOS Developers" ---- - -Since the last post, I've pivoted to full time work on Serpent OS, which is -made all the more possible thanks to everyone supporting us via [OpenCollective](https://opencollective.com/serpent-os) <3. - -We've been working towards establishing an online infrastructure to support the automation of package **builds**, while -revisiting some core components. - - - -## moss-db - -During the development of the Serpent OS tooling we've been exploring the possibilities of D Lang, picking up new -practices and refining our approach as we go. Naturally, some of our older modules are somewhat ... _smelly_. -Most noticeable is our `moss-db` module, which was initially intended as a lightweight wrapper around [RocksDB](http://rocksdb.org/). - -In practice that required an encapsulation API written in D around the C API, and our own wrapping on top of that. Naturally, -it resulted in a very allocation-heavy implementation that just didn't sit right with us, and due to the absurd complexity -of RocksDB was still missing quite a few features. - -## Enter LMDB - -We're now using the [Lightning Memory-Mapped Database](https://www.symas.com/lmdb) as the driver implementation -for moss-db. In short, we get rapid reads, ACID transactions, bulk inserts, you name it. Our implementation takes -advantage of multiple database indexes (`MDB_dbi`) in LMDB to partition the database into internal components, -so that we can provide "buckets", or collections. These internal DBs are used for bucket mapping to permit a -key-compaction strategy - iteration of top level buckets and key-value pairs within a bucket. - -## Hat tip, boltdb - -The majority of the API was designed with the [boltdb](https://github.com/boltdb/bolt) API in mind. Additionally -it was built with `-preview=dip1000` and `-preview=in` enabled, ensuring safely scoped memory use and no -room for memory lifetime issues. While we prefer the use of generics, the API is built with `immutable(ubyte[]`) -as the internal key and value type. - -Custom types can simply implement `mossEncode` or `mossDecode` to be instantly serialisable into the database -as keys, values or bucket identifiers. - -Example API usage: - -```d -Database db; -/* setup the DB with lmdb:// URI */ - -/* Write transaction */ -auto err = db.update((scope tx) @safe -{ - auto bucket = tx.bucket("letters"); - return tx.set(bucket, "a", 1); -}); - -/* do something with the error */ - -err = db.view((in tx) @safe -{ - foreach (name, bucket ; tx.buckets!int) - { - foreach (key, value ; tx.iterator!(string,string)(bucket)) - { - /* do something with the key value pairs, decoded as strings */ - } - } - - /* WILL NOT COMPILE. tx is const scope ref :) */ - tx.removeBucket("numbers"); - - return NoDatabaseError; -} -``` - -## Next for moss - -Moss will be ported to the new DB API and we'll gather some performance metrics, -while implementing features like expired state garbage collection (disk cleanup), -searching for names/descriptions, etc. - -## Avalanche - -![Early version of avalanche, in development](./Featured.webp) - -Avalanche is a core component of our upcoming infrastructure, providing the -service for running builds on a local node, and a controller to coordinate -a group of builders. - -Summit will be the publicly accessible project dashboard, and will be responsible -for coordinating incoming builds to Avalanche controllers and repositories. -Developers will submit builds to Summit and have them dispatched correctly. - -So far we have the core service process in place for the Controller + Node, -and now we're working on persistence and handshake. TLDR; fancy use of -moss-db and JSON Web tokens over mandated SSL. This means our build infra -will be scalable from day 1 allowing multiple builders to be online very -early on. - -## Timescale - -We're planning to get an early version of our infrastructure up and running -within the next 2 weeks, and get builds flowing =) diff --git a/src/content/docs/blog/2022/09/14/the-big-update/index.md b/src/content/docs/blog/2022/09/14/the-big-update/index.md new file mode 100644 index 00000000..41e9d3f7 --- /dev/null +++ b/src/content/docs/blog/2022/09/14/the-big-update/index.md @@ -0,0 +1,117 @@ +--- +title: "The Big Update" +date: 2022-09-14T12:55:04+01:00 +draft: false +authors: [ikey] +tags: [news] +license: "CC-BY-ND-4.0" +copyright: "Copyright © 2022 aerynOS Developers" +--- + +Well - we've got some big news! The past few weeks have been an incredibly busy time for us, and we've hit some major milestones. + +## Funding + +After much deliberation - we've decided to pull out of Open Collective[^1]. Among other reasons, the fees are simply too high and severely impact the funds available to us. In our early stages, the team consensus is that funds generated are used to compensate my time working on Serpent OS. + +As such I'm now moving funding to my own GitHub Sponsors page[^1] - please do migrate! It ensures your entire donation makes it and keeps the lights on for longer =) Please remember I'm working **full time** on Serpent OS exclusively - I need your help to keep working. + +## Moved to GitHub + +We've pretty much completed our transition to GitHub. We've now got the following organisations: + + - [snekpit](https://github.com/snekpit) - Packaging work + - [serpent-os](https://github.com/serpent-os) - Main code development (moss/boulder/etc) + +## Forums + +Don't forget - our forums are **live** over at forums.serpentos.com[^2] - please feel free to drop in and join in with the community 😀 + +## Rehash on the tooling + +OK so what exactly *are* `moss` and `boulder`? In short - they're the absolute core pieces of our distribution model. + +## "What is moss?" + +On the surface, moss looks and feels roughly the same as just about any other traditional package manager out there. Internally, however, its far more modern and has a few tricks up its sleeve. For instance, every time you initiate an operation in moss, be it installation, removal, upgrade, etc, a new filesystem transaction is generated. In short, if something is wrong with the new transaction - you can just boot to an older transaction when things worked fine. + +Now, it's not implemented using bundles or filesystem specific functionality, internally its just intelligent use of hardlinks and deduplication policies, and we have our own container format with zstd based payload compression. Our strongly typed, deduplicating binary format is what powers moss. + +Behind the scenes we also use some other cool technology, such as LMDB for super reliable and speedy database access. The net result is a next generation package management solution that offers all the benefits of traditional package managers (i.e. granularity and composition) with new world features, like atomic updates, deduplication, and repository snapshots. + +## boulder + +It's one thing to manage and install packages, it's another entirely to **build them**. `boulder` builds conceptually on prior art such as `pisi` and the `package.yml` format used in `ypkg`. It is designed with automation and ease of integration in mind, i.e. less time spent focusing on **packaging** and more time on actually getting the thing building and installing correctly. + +Boulder supports "macros" as seen in the RPM and ypkg world, to support consistent builds and integration. Additionally it automatically splits up packages into the appropriate subpackages, and automatically scans for binary, pkgconfig, perl and other dependencies during source analysis and build time. The end result is some `.stone` binary packages and a build manifest, which we use to flesh out our **source package index**. + +## Major moss improvements + +We've spent considerable time reworking `moss`, our package manager. It now features a fresher (terminal) user interface, progress bars, and is rewritten to use the [moss-db](https://github.com/serpent-os/moss-db) module encapsulating LMDB. + +![nspawn](./nspawn.webp "nspawn of moss") + +It's also possible to manipulate the binary collections (software repositories) used by moss now. Note we're going to rename "remote" to "collection" for consistency. + +At the time of writing: + +```bash +$ mkdir destdir +$ sudo moss remote add -D destdir protosnek https://dev.serpentos.com/protosnek/x86_64/stone.index +$ sudo moss install -D destdir bash dash dbus dbus-broker systemd coreutils util-linux which moss nano +$ sudo systemd-nspawn -b -D destdir +``` + +This will be simplified once we introduce `virtual` packages (Coming Soon ™) + +## Local Collections + +Boulder can now be instructed to utilise a local collection of stone packages, simplifying the development of large stack items. + +```bash +sudo boulder bi stone.yml -p local-x86_64 +``` + +Packages should be moved to `/var/cache/boulder/collections/local-x86_64` and the index can be updated by running: + +```bash +sudo moss idx /var/cache/boulder/collections/local-x86_64 +``` + +## Self Hosting + +Serpent OS is now officially self hosting. Using our own packages, we're able to construct a root filesystem, then within that rootfs container we can use our own build tooling (`boulder`) to construct **new builds** of our packages in a nested container. + +The `protosnek` collection has been updated to include the newest versions of moss and boulder. + +![self hosting](./self-hosting.webp "Serpent OS is now self hosting") + +## Booting + +As a fun experiment, we wanted to see how far along things are. Using a throwaway kernel + initrd build, we were able to get Serpent OS booting using virtualisation (`qemu`) + +![booting](./booting.webp "Serpent OS now booting via qemu") + +## ISO When? + +Right now everyone is working in the [snekpit](https://github.com/snekpit) organisation to get our base packaging in order. I'm looking to freeze `protosnek`, our bootstrap collection, at the latest of tomorrow evening. + +We now support layered, priority based collections (repositories) and dependency solving across collection boundaries, allowing us to build our new `main` collection with `protosnek` acting as a bootstrap seed. + +Throughout this week, I'll focus on getting Avalanche, Summit and Vessel into shape for PoC so that we can enjoy automated builds of packages landing in the yet-to-be-launched `volatile` collection. + +From there, we're going to iterate + improve packaging, fixing bugs and landing features as we discover the issues. Initially we'll look to integrate **triggers** in a stateless-friendly fashion (our packages can only ship `/usr` by design) - after that will come boot management. + +An early target will be Qemu support via a stripped `linux-kvm` package to accelerate the bring up, and we encourage everyone to join in the testing. We're self hosting, we know how to boot, and now we're able to bring the awesome. + +I cannot stress how important the support to the project is. Without it - I'm unable to work full time on the project. Please consider supporting my development work via GitHub Sponsors[^1]. + +> I'm so broke they've started naming black holes after my IBAN. + +Thank you! + +You can discuss this blog post over on forums[^2]. + +--- +[^1]: Since this blog post, we no longer use OpenCollective or GitHub Sponsors. For funding, please see our [Funding Page](/sponsor) and for ISOs, please see our [Download Page](/download). +[^2]: Since this blog post, we now use [Discourse](https://aerynos.discourse.group) as our forum. diff --git a/src/content/docs/blog/2022/09/14/the-big-update/index.mdx b/src/content/docs/blog/2022/09/14/the-big-update/index.mdx deleted file mode 100644 index 5920c2b5..00000000 --- a/src/content/docs/blog/2022/09/14/the-big-update/index.mdx +++ /dev/null @@ -1,156 +0,0 @@ ---- -title: "The Big Update" -date: 2022-09-14T12:55:04+01:00 -draft: false -authors: [ikey] -tags: [news] -license: "CC-BY-ND-4.0" -copyright: "Copyright © 2022 aerynOS Developers" ---- - -Well - we've got some big news! The past few weeks have been an incredibly busy -time for us, and we've hit some major milestones. - - - -## Funding - -After much deliberation - we've decided to pull out of Open Collective. Among other reasons, the fees -are simply too high and severely impact the funds available to us. In our early stages, the team consensus -is that funds generated are used to compensate my time working on Serpent OS. - -As such I'm now moving funding to my own [GitHub Sponsors](https://github.com/sponsors/ikeycode?o=sd&sc=t) page - please do migrate! It ensures your entire -donation makes it and keeps the lights on for longer =) Please remember I'm working **full time** on Serpent OS -exclusively - I need your help to keep working. - -## Moved to GitHub - -We've pretty much completed our transition to GitHub. We've now got the following organisations: - - - [snekpit](https://github.com/snekpit) - Packaging work - - [serpent-os](https://github.com/serpent-os) - Main code development (moss/boulder/etc) - -## Forums - -Don't forget - our forums are **live** over at [forums.serpentos.com](https://forums.serpentos.com) - please feel -free to drop in and join in with the community =) - -## Rehash on the tooling - -OK so what exactly *are* `moss` and `boulder`? In short - they're the absolute core pieces of our distribution model. - -## "What is moss?" - -On the surface, moss looks and feels roughly the same as just about any other traditional package manager out there. -Internally, however, its far more modern and has a few tricks up its sleeve. For instance, every time you initiate -an operation in moss, be it installation, removal, upgrade, etc, a new filesystem transaction is generated. In short, -if something is wrong with the new transaction - you can just boot to an older transaction when things worked fine. - -Now, it's not implemented using bundles or filesystem specific functionality, internally its just intelligent use of -hardlinks and deduplication policies, and we have our own container format with zstd based payload compression. Our -strongly typed, deduplicating binary format is what powers moss. - -Behind the scenes we also use some other cool technology, such as LMDB for super reliable and speedy database access. -The net result is a next generation package management solution that offers all the benefits of traditional package -managers (i.e. granularity and composition) with new world features, like atomic updates, deduplication, and repository -snapshots. - -## boulder - -It's one thing to manage and install packages, it's another entirely to **build them**. `boulder` builds conceptually -on prior art such as `pisi` and the `package.yml` format used in `ypkg`. It is designed with automation -and ease of integration in mind, i.e. less time spent focusing on **packaging** and more time on actually -getting the thing building and installing correctly. - -Boulder supports "macros" as seen in the RPM and ypkg world, to support consistent builds and integration. -Additionally it automatically splits up packages into the appropriate subpackages, and automatically scans -for binary, pkgconfig, perl and other dependencies during source analysis and build time. The end result -is some `.stone` binary packages and a build manifest, which we use to flesh out our **source package index**. - -## Major moss improvements - -We've spent considerable time reworking `moss`, our package manager. It now features -a fresher (terminal) user interface, progress bars, and is rewritten to use the -[moss-db](https://github.com/serpent-os/moss-db) module encapsulating LMDB. - -![nspawn](./nspawn.webp) - -It's also possible to manipulate the binary collections (software repositories) -used by moss now. Note we're going to rename "remote" to "collection" for consistency. - -At the time of writing: - -```bash -$ mkdir destdir -$ sudo moss remote add -D destdir protosnek https://dev.serpentos.com/protosnek/x86_64/stone.index -$ sudo moss install -D destdir bash dash dbus dbus-broker systemd coreutils util-linux which moss nano -$ sudo systemd-nspawn -b -D destdir -``` - -This will be simplified once we introduce `virtual` packages (Coming Soon ™) - -## Local Collections - -Boulder can now be instructed to utilise a local collection of stone packages, simplifying the development of large stack items. - -```bash -sudo boulder bi stone.yml -p local-x86_64 -``` - -Packages should be moved to `/var/cache/boulder/collections/local-x86_64` and the index -can be updated by running: - -```bash -sudo moss idx /var/cache/boulder/collections/local-x86_64 -``` - - -## Self Hosting - -Serpent OS is now officially self hosting. Using our own packages, we're able to -construct a root filesystem, then within that rootfs container we can use our own -build tooling (`boulder`) to construct **new builds** of our packages in a nested -container. - -The `protosnek` collection has been updated to include the newest versions of moss -and boulder. - -![self hosting](./self-hosting.webp) - - -## Booting - -As a fun experiment, we wanted to see how far along things are. Using a throwaway -kernel + initrd build, we were able to get Serpent OS booting using virtualisation (`qemu`) - -![booting](./booting.webp) - -## ISO When? - -Right now everyone is working in the [snekpit](https://github.com/snekpit) organisation to -get our base packaging in order. I'm looking to freeze `protosnek`, our bootstrap collection, -at the latest of tomorrow evening. - -We now support layered, priority based collections (repositories) and dependency solving across -collection boundaries, allowing us to build our new `main` collection with `protosnek` acting as -a bootstrap seed. - -Throughout this week, I'll focus on getting Avalanche, Summit and Vessel into shape for PoC so -that we can enjoy automated builds of packages landing in the yet-to-be-launched `volatile` collection. - -From there, we're going to iterate + improve packaging, fixing bugs and landing features as we -discover the issues. Initially we'll look to integrate **triggers** in a stateless-friendly -fashion (our packages can only ship `/usr` by design) - after that will come boot management. - -An early target will be Qemu support via a stripped `linux-kvm` package to accelerate the bring up, -and we encourage everyone to join in the testing. We're self hosting, we know how to boot, and -now we're able to bring the awesome. - -I cannot stress how important the support to the project is. Without it - I'm unable to work full -time on the project. Please consider supporting my development work via [GitHub Sponsors](https://github.com/sponsors/ikeycode?o=sd&sc=t). - -> I'm so broke they've started naming black holes after my IBAN. - -Thank you! - -You can discuss this blog post over on [our forums](https://forums.serpentos.com/d/20-the-big-update) diff --git a/src/content/docs/blog/2022/12/24/lift-off/index.md b/src/content/docs/blog/2022/12/24/lift-off/index.md new file mode 100644 index 00000000..249a8137 --- /dev/null +++ b/src/content/docs/blog/2022/12/24/lift-off/index.md @@ -0,0 +1,59 @@ +--- +title: "Lift Off" +date: 2022-12-24T01:32:04Z +draft: false +authors: [ikey] +tags: [news] +license: "CC-BY-ND-4.0" +copyright: "Copyright © 2022 aerynOS Developers" +--- + +Enough of this "2 years" nonsense. We're finally ready for lift off. It is with *immense pleasure* we can finally announce that Serpent OS has transitioned from a promise to a deliverable. Bye bye, phantomware! + +![A booted Serpent OS](./Featured.webp "A booted Serpent OS in qemu") + +### We exist + +As mentioned, we spent 2 years working on tooling and process. That's .. well. Kinda dull, honestly. You're not here for the tooling, you're here for the OS. To that end I made a decision to accelerate development of the actual *Linux distro* - and shift development of tooling into a parallel effort. + +### Infrastructure .. intelligently deferred + +I deferred final enabling of the infrastructure until January to rectify the chicken/egg scenario whilst allowing us to grow a base of contributors and an actual distro to work with. We're in a good position with minimal blockers so no concern there. + +### A real software collection + +This is our term for the classical "package repository". We're using a temporary collection right now to store all of the builds we produce. In keeping with the `Avalanche` requirements, this is the **volatile** software collection. Changes a lot, hasn't got a release policy. + +### A community. + +It goes without saying, really, that our project isn't remotely possible without a **community**. I want to take the time to personally thank everyone that stepped up to the plate lately and contributed to Serpent OS. Without the work of the team, in which I include the contributors to our `venom` recipe repository, an ISO was never possible. Additionally contributions to tooling has helped us make significant strides. + +It should be noted we've practically folded our old "team" concept and ensured we operate across the board as a singular community, with some members having additional responsibilities. Our belief is all in the community have equal share and say. With that said, to the original "team", members both past and present, I thank for their (long) support and contributions to the project. + +### An ISO. + +We actually went ahead and created our first ISO. OK that's a lie, this is probably the 20th revision by now. And let's be brutally honest here: + +**It sucks.** + +We expected no less. However, the time is definitely here for us to begin our public iteration, transitioning from suckness to a project worth using. In order to do that, we need to get ourselves to a point whereby we can dogfood our work and build a daily driver. Our focus right now is building out the core technology and packaging to achieve those aims. + +So if you want to try our uninstallable, buggy ISO, chiefly created as a brief introduction to our package manager and toolchain, head to our newly minted [Download](/download) page. Set your expectations low, ignore your dreams, and you will not be disappointed! + +All jokes aside, it took a long time to get to point where we could even construct our first, KVM-focused, UEFI-only `snekvalidator.iso`. We now have a baseline to improve on, a working contribution process, and a booting, self-hosting system. + +The ISO is built using 2 layered collections, the `protosnek` collection containing our toolchain, and the new `volatile` collection. Much of the packaging work has been submitted by `venom` contributors and the core team. Note you can install `neofetch` which our very own Rune Morling (`ermo`) patched to support the Serpent OS logo. + +Boot it in Qemu (or certain Intel laptops) and play with moss now! Note, this ISO is not installable, and no upgrade path exists. It is simply the beginnings of a public iteration process. + +### Next steps + +In January we'll launch our infrastructure to scale out contributions as well as to permit the mass-rebuilds that need to happen. We have to enable our `-dbginfo` packages and stripping, which were disabled due to a parallelism issue. We need to introduce our boot management based around `systemd-boot`, provide more kernels, do hardware enabling, introduce `moss-triggers`, and much more. However, this is a **pivotal moment** for our project as we've finally become a **real**, if not __sucky__, distro. The future is incredibly bright, and we intend to deliver on every one of our promises. + +As always, if you want to support our development, please consider [sponsoring](/sponsor) the work, or engaging with the community on Matrix[^1] or indeed our [forums](https://aerynos.discourse.group)[^2]. + +You can discuss this blog post, or leave feedback on the ISO, over at our [forums](https://aerynos.discourse.group)[^2]. + +--- +[^1]: aerynOS now uses a [Zulip server](https://aerynos.zulipchat.com/) for community discussion. +[^2]: aerynOS now uses a [Discourse](https://aerynos.discourse.group) for community discussion. diff --git a/src/content/docs/blog/2022/12/24/lift-off/index.mdx b/src/content/docs/blog/2022/12/24/lift-off/index.mdx deleted file mode 100644 index 40074594..00000000 --- a/src/content/docs/blog/2022/12/24/lift-off/index.mdx +++ /dev/null @@ -1,82 +0,0 @@ ---- -title: "Lift Off" -date: 2022-12-24T01:32:04Z -draft: false -authors: [ikey] -tags: [news] -license: "CC-BY-ND-4.0" -copyright: "Copyright © 2022 aerynOS Developers" ---- - -Enough of this "2 years" nonsense. We're finally ready for lift off. It is with *immense pleasure* we can -finally announce that Serpent OS has transitioned from a promise to a deliverable. Bye bye, phantomware! - - - -![featured image](Featured.webp) - -### We exist - -As mentioned, we spent 2 years working on tooling and process. That's .. well. Kinda dull, honestly. You're -not here for the tooling, you're here for the OS. To that end I made a decision to accelerate development of -the actual *Linux distro* - and shift development of tooling into a parallel effort. - -### Infrastructure .. intelligently deferred - -I deferred final enabling of the infrastructure until January to rectify the chicken/egg scenario whilst allowing -us to grow a base of contributors and an actual distro to work with. We're in a good position with minimal blockers -so no concern there. - -### A real software collection - -This is our term for the classical "package repository". We're using a temporary collection right now to store all -of the builds we produce. In keeping with the `Avalanche` requirements, this is the **volatile** software collection. Changes -a lot, hasn't got a release policy. - -### A community. - -It goes without saying, really, that our project isn't remotely possible without a **community**. I want to take the time -to personally thank everyone that stepped up to the plate lately and contributed to Serpent OS. Without the work of the -team, in which I include the contributors to our `venom` recipe repository, an ISO was never possible. Additionally contributions -to tooling has helped us make significant strides. - -It should be noted we've practically folded our old "team" concept and ensured we operate across the board as a singular community, -with some members having additional responsibilities. Our belief is all in the community have equal share and say. With that said, -to the original "team", members both past and present, I thank for their (long) support and contributions to the project. - -### An ISO. - -We actually went ahead and created our first ISO. OK that's a lie, this is probably the 20th revision by now. And let's be brutally -honest here: - -**It sucks.** - -We expected no less. However, the time is definitely here for us to begin our public iteration, transitioning from suckness to a project -worth using. In order to do that, we need to get ourselves to a point whereby we can dogfood our work and build a daily driver. Our focus -right now is building out the core technology and packaging to achieve those aims. - -So if you want to try our uninstallable, buggy ISO, chiefly created as a brief introduction to our package manager and toolchain, head to our -newly minted [Download](/download) page. Set your expectations low, ignore your dreams, and you will not be disappointed! - -All jokes aside, it took a long time to get to point where we could even construct our first, KVM-focused, UEFI-only `snekvalidator.iso`. We now -have a baseline to improve on, a working contribution process, and a booting, self-hosting system. - -The ISO is built using 2 layered collections, the `protosnek` collection containing our toolchain, and the new `volatile` collection. Much of the -packaging work has been submitted by `venom` contributors and the core team. Note you can install `neofetch` which our very own Rune Morling (`ermo`) -patched to support the Serpent OS logo. - -Boot it in Qemu (or certain Intel laptops) and play with moss now! Note, this ISO is not installable, and no upgrade path exists. It is simply -the beginnings of a public iteration process. - -### Next steps - -In January we'll launch our infrastructure to scale out contributions as well as to permit the mass-rebuilds that need to happen. We have to -enable our `-dbginfo` packages and stripping, which were disabled due to a parallelism issue. We need to introduce our boot management based around -`systemd-boot`, provide more kernels, do hardware enabling, introduce `moss-triggers`, and much more. However, this is a **pivotal moment** for our -project as we've finally become a **real**, if not __sucky__, distro. The future is incredibly bright, and we intend to deliver on every one of our -promises. - -As always, if you want to support our development, please consider [sponsoring](https://github.com/sponsors/ikeycode) the work, or engaging with the community on Matrix or indeed -our [forums](https://forums.serpentos.com). - -You can discuss this blog post, or leave feedback on the ISO, over at our [forums](https://forums.serpentos.com/d/40-lift-off). diff --git a/src/content/docs/blog/2023/03/18/infrastructure-launched/Featured.png b/src/content/docs/blog/2023/03/18/infrastructure-launched/Featured.png deleted file mode 100644 index a7853387..00000000 Binary files a/src/content/docs/blog/2023/03/18/infrastructure-launched/Featured.png and /dev/null differ diff --git a/src/content/docs/blog/2023/03/18/infrastructure-launched/index.md b/src/content/docs/blog/2023/03/18/infrastructure-launched/index.md index 3ae18282..8c8b7f30 100644 --- a/src/content/docs/blog/2023/03/18/infrastructure-launched/index.md +++ b/src/content/docs/blog/2023/03/18/infrastructure-launched/index.md @@ -8,15 +8,9 @@ license: "CC-BY-ND-4.0" copyright: "Copyright © 2023 aerynOS Developers" --- -After many months and much work, our infrastructure is [finally online](https://dash.serpentos.com). -We've had a few restarts, but it's now running fully online with 2 builders, ready to serve builds -around the clock. +After many months and much work, our infrastructure is [finally online](https://dash.aerynos.dev)[^1]. We've had a few restarts, but it's now running fully online with 2 builders, ready to serve builds around the clock. - - -Firstly, I'd like to apologise for the delay since our last blog post. We made the decision to move -this website to static content, which took longer than expected. We're still using our own D codebase, -but prebuilding the site (and fake "API") so we can lower the load on our web server. +Firstly, I'd like to apologise for the delay since our last blog post. We made the decision to move this website to static content, which took longer than expected. We're still using our own D codebase, but prebuilding the site (and fake "API") so we can lower the load on our web server. ### Introducing the Infra @@ -24,27 +18,26 @@ Our infrastructure is composed of 3 main components. #### Summit -This is the page you can see over at [dash.serpentos.com](https://dash.serpentos.com). It contains -the build scheduler. It monitors our git repositories, and as soon as it discovers any missing builds -it creates build tasks for them. It uses a graph to ensure parallel builds happen as much as possible, -and correctly orders (and blocks) builds based on build dependencies. +![First look at summit](./Featured.webp "The first look at Summit") + +This is the page you can see over at [dash.aerynos.dev](https://dash.aerynos.dev)[^1]. It contains the build scheduler. It monitors our git repositories, and as soon as it discovers any missing builds it creates build tasks for them. It uses a graph to ensure parallel builds happen as much as possible, and correctly orders (and blocks) builds based on build dependencies. #### Avalanche -We have 2 instances of Avalanche, our builder tool, running. This accepts configuration + build requests -from Summit, reporting status and available builds. +We have 2 instances of Avalanche, our builder tool, running. This accepts configuration + build requests from Summit, reporting status and available builds. #### Vessel -This is an internal repository manager, which accepts completed packages from an Avalanche instance. -Published packages land at [dev.serpentos.com](https://dev.serpentos.com) +This is an internal repository manager, which accepts completed packages from an Avalanche instance. Published packages land at [build.aerynos.dev](https://build.aerynos.dev)[^2] ### Next steps -Super early days with the infra but we now have builds flowing as part of our continuous delivery solution. -Keeping this blog post short and sweet... we're about to package GNOME and start racing towards our first -**real** ISOs! +Super early days with the infra but we now have builds flowing as part of our continuous delivery solution. Keeping this blog post short and sweet... we're about to package GNOME and start racing towards our first **real** ISOs! ### Don't forget -If you like what the project is doing, don't forget to [sponsor](https://github.com/sponsors/ikeycode?o=sd&sc=t)! +If you like what the project is doing, don't forget to [sponsor](/sponsor)! + +--- +[^1]: The infrastructure used to be hosted at dash.serpentos.com but has since moved to our aerynOS domain. +[^2]: Vessel used to be hosted at dev.serpentos.com but has since moved to our aerynOS domain. diff --git a/src/content/docs/blog/2023/04/21/snakes-on-a-boat/index.md b/src/content/docs/blog/2023/04/21/snakes-on-a-boat/index.md index b3afcee4..e5cb8ab0 100644 --- a/src/content/docs/blog/2023/04/21/snakes-on-a-boat/index.md +++ b/src/content/docs/blog/2023/04/21/snakes-on-a-boat/index.md @@ -8,13 +8,9 @@ license: "CC-BY-ND-4.0" copyright: "Copyright © 2023 aerynOS Developers" --- -We had intended to get a blog post out a little bit quicker, but the last month has been extremely -action packed. However, it has paid off immensely. As our friends at Solus [recently announced](https://getsol.us/2023/04/18/a-new-voyage/) -it is time to embark on a new voyage. +We had intended to get a blog post out a little bit quicker, but the last month has been extremely action packed. However, it has paid off immensely. As our friends at Solus [recently announced](https://getsol.us/2023/04/18/a-new-voyage/) it is time to embark on a new voyage. -![dashboard preview](featured.webp) - - +![dashboard preview](./featured.webp "Preview of the Summit Dashboard") ### General gist of it @@ -26,35 +22,23 @@ There is very little point in rehashing the post, so I'll cover the essentials ### Is Serpent OS over? -Oh no, no. The most practical angle to view this from is that Serpent OS is free to build and innovate to create the -basis of Solus 5. Outside of the distribution we're still keen to continue development on our tooling -and strategies. +Oh no, no. The most practical angle to view this from is that Serpent OS is free to build and innovate to create the basis of Solus 5. Outside of the distribution we're still keen to continue development on our tooling and strategies. -It is therefore critical that we continue development, and find the best approach for both projects, to make -the transition to Solus 5 as seamless as possible. To that end we will still need to produce ISOs and have an -active community of users and testers. During this transition period, despite being two separate projects, we're both -heading to a common goal and interest. +It is therefore critical that we continue development, and find the best approach for both projects, to make the transition to Solus 5 as seamless as possible. To that end we will still need to produce ISOs and have an active community of users and testers. During this transition period, despite being two separate projects, we're both heading to a common goal and interest. ### Finances -We have an exciting journey ahead for all of us, and there are many moving parts involved. Until we're at the point -of mutual merge, we will keep the entities and billing separate. Thus, funds to the Solus [OpenCollective](https://opencollective.com/getsolus) -are intended for use within Solus, whereas we currently use [GitHub Sponsors](https://github.com/sponsors/ikeycode?o=sd&sc=t) -for our own project needs. +We have an exciting journey ahead for all of us, and there are many moving parts involved. Until we're at the point of mutual merge, we will keep the entities and billing separate. Thus, funds to the Solus [OpenCollective](https://opencollective.com/getsolus) are intended for use within Solus, whereas we currently use GitHub Sponsors[^1] for our own project needs. -Currently Solus and Serpent OS share one server, which was essential for quick turnaround on infrastructure enabling. -At the end of this month Serpent OS will migrate from that server to a new, separate system, ensuring the projects -are billed separately. +Currently Solus and Serpent OS share one server, which was essential for quick turnaround on infrastructure enabling. At the end of this month Serpent OS will migrate from that server to a new, separate system, ensuring the projects are billed separately. -Long story short, if you wish to sponsor Serpent OS **specifically**, please do so via [our GitHub sponsors account](https://github.com/sponsors/ikeycode?o=sd&sc=t) as our monthly running costs -will immediately rise at the end of this month. If you're supporting Solus development, please do visit them and sponsor them =) +Long story short, if you wish to sponsor Serpent OS **specifically**, please do so via [funding](/sponsor) as our monthly running costs will immediately rise at the end of this month. If you're supporting Solus development, please do visit them and sponsor them 😀 ### Next steps? -Glad you asked! Now that the dust is settling, we're focusing on Serpent OS requirements, and helping Solus where we can. -Our immediate goals are to build a dogfooding system for a small collection of developers to run as an unsupported prealpha -configuration, allowing us to flesh out the tooling and processes. +Glad you asked! Now that the dust is settling, we're focusing on Serpent OS requirements, and helping Solus where we can. Our immediate goals are to build a dogfooding system for a small collection of developers to run as an unsupported prealpha configuration, allowing us to flesh out the tooling and processes. + +This will include a live booting GNOME ISO, and sufficient base packages to freely iterate on `moss`, `boulder`, etc as well as our own infrastructure. Once we've attained a basic quality and have an installer option, those ISOs will be made available to you guys! -This will include a live booting GNOME ISO, and sufficient base packages to freely iterate on `moss`, `boulder`, etc as well -as our own infrastructure. Once we've attained a basic quality and have an installer option, those ISOs will be made available -to you guys! +--- +[^1]: Since this blog post, we no longer use GitHub Sponsors. For funding, please see our [Funding Page](/sponsor). diff --git a/src/content/docs/blog/2023/09/06/oxidised-moss/Featured.png b/src/content/docs/blog/2023/09/06/oxidised-moss/Featured.png deleted file mode 100644 index b49c49b2..00000000 Binary files a/src/content/docs/blog/2023/09/06/oxidised-moss/Featured.png and /dev/null differ diff --git a/src/content/docs/blog/2023/09/06/oxidised-moss/index.md b/src/content/docs/blog/2023/09/06/oxidised-moss/index.md index 1b2af38c..44afc2f3 100644 --- a/src/content/docs/blog/2023/09/06/oxidised-moss/index.md +++ b/src/content/docs/blog/2023/09/06/oxidised-moss/index.md @@ -8,47 +8,29 @@ license: "CC-BY-ND-4.0" copyright: "Copyright © 2023 aerynOS Developers" --- -Allow me to start this post by stating something very important to me: I absolutely love the -D programming language, along with the expressivity and creative freedom it brings. Therefore -please do not interpret this post as an assassination piece. +Allow me to start this post by stating something very important to me: I absolutely love the D programming language, along with the expressivity and creative freedom it brings. Therefore please do not interpret this post as an assassination piece. -For a number of months now the Serpent OS project has stood rather still. While this could naively -be attributed to our shared plans with [Solus](https://getsol.us) - a deeper, technical issue is -to be acredited. +For a number of months now the Serpent OS project has stood rather still. While this could naively be attributed to our shared plans with [Solus](https://getsol.us) - a deeper, technical issue is to be acredited. -![img](Featured.png) - - +![img](./Featured.webp "Screenshot of moss built in Rust") ## D isn't quite there *yet*. But it will be, some day. -Again, allow me to say I have thoroughly enjoyed my experience with D over the last 3 or so years, -it has been truly illuminating for me as an engineer. With that said, we have also become responsible -for an awful lot of code. As an engineering-led distribution + tooling project, our focus is that of -secure and auditable code paths. To that extent we pursued [DIP1000](https://github.com/dlang/DIPs/blob/master/DIPs/other/DIP1000.md) as far as practical and admit it has a way to go before addressing our immediate needs of memory safety. +Again, allow me to say I have thoroughly enjoyed my experience with D over the last 3 or so years, it has been truly illuminating for me as an engineer. With that said, we have also become responsible for an awful lot of code. As an engineering-led distribution + tooling project, our focus is that of secure and auditable code paths. To that extent we pursued [DIP1000](https://github.com/dlang/DIPs/blob/master/DIPs/other/DIP1000.md) as far as practical and admit it has a way to go before addressing our immediate needs of memory safety. -While we're quite happy to be an upstream for Linux distributions by way of release channels and tooling -releases, we don't quite have the resources to also be an upstream for the numerous D packages we'd need to -create and maintain to get our works over the finish line. +While we're quite happy to be an upstream for Linux distributions by way of release channels and tooling releases, we don't quite have the resources to also be an upstream for the numerous D packages we'd need to create and maintain to get our works over the finish line. -With that said, I will still continue to use D in my own *personal* projects, and firmly believe that one day -D will realise its true potential. +With that said, I will still continue to use D in my own *personal* projects, and firmly believe that one day D will realise its true potential. ## Rewarding contributors -Our priorities have shifted somewhat since the announcement of our shared venture with Solus, and we must make -architectural decisions based on the needs of all stakeholders involved, including the existing contributor pool. -Additionally, care should be taken to be somewhat populist in our choice of stacks in order to give contributors -industry-relevant experience to add to their résumé (CV). +Our priorities have shifted somewhat since the announcement of our shared venture with Solus, and we must make architectural decisions based on the needs of all stakeholders involved, including the existing contributor pool. Additionally, care should be taken to be somewhat populist in our choice of stacks in order to give contributors industry-relevant experience to add to their résumé (CV). ## Playing to strengths -Typically Solus has been a Golang-oriented project, and has a number of experienced developers. With the addition -of the Serpent developers, the total cross-over development team has a skill pool featuring Rust and Go, as well as -various web stack technologies. +Typically Solus has been a Golang-oriented project, and has a number of experienced developers. With the addition of the Serpent developers, the total cross-over development team has a skill pool featuring Rust and Go, as well as various web stack technologies. -Reconsidering the total project architecture including our automated builds, the following decisions have been made -that incorporate the requirements of being widely adopted/supported, robust ecosystems and established tooling: +Reconsidering the total project architecture including our automated builds, the following decisions have been made that incorporate the requirements of being widely adopted/supported, robust ecosystems and established tooling: - Rust, for low level tooling and components. Chiefly: `moss`, `boulder`, `libstone` - ReactJS/TypeScript for our frontend work (Summit Web) @@ -56,36 +38,26 @@ that incorporate the requirements of being widely adopted/supported, robust ecos ## Lets start... 2 days ago! -The new infrastructure will be brought up using widely available modules, and designed to be scalable from the outset -as part of a Kubernetes deployment, with as minimal user interaction as needed. Our eventual plans include rebuilding -the entire distribution from source with heavy caching once some part of the dependency graph changes. +The new infrastructure will be brought up using widely available modules, and designed to be scalable from the outset as part of a Kubernetes deployment, with as minimal user interaction as needed. Our eventual plans include rebuilding the entire distribution from source with heavy caching once some part of the dependency graph changes. -This infrastructure will then be extended to support the Solus 4 series for quality of life improvements to Solus developers, -enabling a more streamlined dev workflow: TL;DR less time babysitting builds = more Serpent development focus. +This infrastructure will then be extended to support the Solus 4 series for quality of life improvements to Solus developers, enabling a more streamlined dev workflow: TL;DR less time babysitting builds = more Serpent development focus. -Our priority these past few days has been on the new [moss-rs](https://github.com/serpent-os/moss-rs) repository where we -have begun to reimplement `moss` in Rust. So far we have a skeleton CLI powered by `clap` with an in-progress library for reading -`.stone` archives, our custom package format. +Our priority these past few days has been on the new [moss-rs](https://github.com/serpent-os/moss-rs) repository where we have begun to reimplement `moss` in Rust. So far we have a skeleton CLI powered by `clap` with an in-progress library for reading `.stone` archives, our custom package format. -The project is organised as a Rust workspace, with the view that `stone`, `moss` and `boulder` will all live in the same tree. -Our hope is this vastly improves the onboarding experience and opens the doors (finally) to contributors. +The project is organised as a Rust workspace, with the view that `stone`, `moss` and `boulder` will all live in the same tree. Our hope is this vastly improves the onboarding experience and opens the doors (finally) to contributors. ### Licensing -It should also be noted that the new tooling is made available under the terms of the Mozilla Public License ([MPL-2.0](https://spdx.org/licenses/MPL-2.0.html)). -After internal discussion, we felt the MPL offered the greatest level of defence against patent trolls while still ensuring our code -was widely free for all to respectfully use and adapt. +It should also be noted that the new tooling is made available under the terms of the Mozilla Public License ([MPL-2.0](https://spdx.org/licenses/MPL-2.0.html)). After internal discussion, we felt the MPL offered the greatest level of defence against patent trolls while still ensuring our code was widely free for all to respectfully use and adapt. Please also note that we have always, and continue to deliberately credit copyright as: Copyright © 2020-2023 Serpent OS Developers -This is a virtual collective consisting of all whom have contributed to Serpent OS (per git logs) and is designed to prevent us from -being able to change the license down the line, i.e. a community protective measure. +This is a virtual collective consisting of all whom have contributed to Serpent OS (per git logs) and is designed to prevent us from being able to change the license down the line, i.e. a community protective measure. ## Glad to be back -Despite some sadness in the change of circumstances, we must make decisions that benefit us collectively as a community. -Please join us in raising a virtual glass to new pastures, and a brave new `blazing fast 🚀` (TM) future for Serpent OS and Solus 5. +Despite some sadness in the change of circumstances, we must make decisions that benefit us collectively as a community. Please join us in raising a virtual glass to new pastures, and a brave new `blazing fast 🚀` (TM) future for Serpent OS and Solus 5. Cheers! diff --git a/src/content/docs/blog/2023/12/19/end-of-year-summary/index.md b/src/content/docs/blog/2023/12/19/end-of-year-summary/index.md new file mode 100644 index 00000000..9f0ddd9d --- /dev/null +++ b/src/content/docs/blog/2023/12/19/end-of-year-summary/index.md @@ -0,0 +1,223 @@ +--- +title: "End Of Year Summary" +date: 2023-12-19T13:07:16+00:00 +draft: false +authors: [ikey, ermo] +tags: [news] +license: "CC-BY-ND-4.0" +copyright: "Copyright © 2023 aerynOS Developers" +--- + +Burning question - how long before we can use Serpent OS on our development systems? It's a fair question - and one that requires context to answer. Let's start our new monthly update cycle with a recap of progress so far. + +![screenshot of moss](./featured.webp "Screenshot of moss mid installation") + +## Introducing tarkah + +We'd like to extend a huge official welcome, and thank you, to newest team member Cory Forsstrom (aka `tarkah`)! + +Firstly, a quote from Cory: + +> I've been a long time Solus user and was very excited about what serpent os was doing. Really got invested once I started diving into the D code and seeing how powerful the tools and ideas were. The Rust rewrite was just a perfect storm for me with my experience there and my desire to make contributions. Getting to know you and ermo has just been icing on the cake, you've both been so welcoming and friendly. So lots of fun times. +> +> On the personal side, I'm on the west coast in the States, have a lovely wife, just had a baby girl and am enjoying my time with fatherhood and coding + +Chances are you know tarkah for his contributions to [iced-rs](https://github.com/iced-rs/iced), and to [Solus](https://getsol.us). His contributions to the [moss-rs](https://github.com/serpent-os/moss-rs) project have been absolutely critical, and his patience essential as we got ourselves quickly up to speed with Rust. It is entirely fair to say that our Rust efforts would not have been possible without him! + +## Rust Progress + +I think it is fair to say that people collectively facepalmed when we announced our plans to [adopt Rust](https://serpentos.com/blog/2023/09/06/oxidised-moss/) - assuming this would be a huge set back. We're really happy to report this is **not** the case, and we've made tremendous progress in this area. + +### moss + +Our Rust based `moss` system package tool is now leaps and bounds ahead of its predecessor, supporting the original features and more! + + - State management + - Garbage collection of old transactions (via `state prune`) + - Optimised write order of transactions (see more below) + - File conflict detection + - Parallel fetching and decompression of packages + - Update support (as `sync`) + - Local and remote repositories with priority-based layering + - Support for building with `musl` for fully self-contained, static builds of moss with zero system dependencies. + +We now have a version-agnostic "stone" crate that is more efficient at reading packages than our original D code. In addition, it benefits from the memory safety promises of Rust. + +:::note[What is moss?] +moss aims to **feel** like a traditional package manager while offering features typically only found in tools like [ostree](https://github.com/ostreedev/ostree). It is still *package* based and offers the same level of granularity and composition as classic package managers while still managing to offer full OS deduplication, offline rollbacks and atomic updates. + +At a *technical level*, we have a custom binary package format that focuses on read performance, deduplication, strong types and error reduction. Every operation in moss creates an entirely new filesystem tree composed of **hard links** to content addressable storage (Hash-based) which is then "promoted" by swapping the host `/usr` with the staging tree using the `renameat2` system call. Phew! +::: + +#### Virtual filesystem management + +OK, lets look at the problem: Every transaction in moss requires us to generate a new staging tree containing all of the directories and files for the `/usr` tree, using hard links, from content addressable storage. As one might expect, creating tens of thousands of new nodes is really, really slow! + +To alleviate this issue we created a [vfs](https://github.com/serpent-os/moss-rs/tree/main/crates/vfs) crate in moss to optimise *how* we construct each transaction tree: + + - Organise all incoming registered paths into a specific order + - Inject all missing implicitly created directories for accounting + - Re-parent nodes in tree which live under redirected symlinks + - Compile into a recursive data-structure using relative names. + +There is a multipart "build" for the tree that detects any filesystem conflicts before any files have been written to disk, preventing broken transactions. With the fully built data structure we can recurse a *virtual filesystem*, making use of the `at` family of functions like `linkat`, `mkdirat`, to "blit" a filesystem tree without requiring any path resolution, in the most efficient order possible. + +The net result? New transaction roots are created in milliseconds, not seconds. + +#### Upgrade support - *kind of* + +moss now offers a `sync` command in place of the conventional `upgrade` one might expect. Rather than upgrading to the latest *version* of a package, we rely on moss and our infrastructure tooling work in conjunction to ensure we can simply *synchronize* the package selection with the the "tip" candidates in the configured repositories, as ordered by priority. From a distro-builder PoV, we can now revert and pull builds. For a user PoV, you can remove a test repository and `moss sync` to go back to the official versions of packages. + +Generally speaking, when you `moss sync` you will be seeing new packages, however. :smile: This feature is being built to enable a future feature: exported states. Want to try a new edition? Sync to the lockfile. Need to quickly and reproducibly deploy containers from a locked down configuration? You get the idea. + +![moss sync](./sync.webp "terminal showing a moss sync command") + +#### Fully `async` + +The moss tool is now completely asynchronous, making efficient use of both coroutines and OS threads to perform as much work possible in the shortest space of time. This allows us to download, fetch, check and unpack many packages at the same time without being blocked on network or disk, greatly speeding up the fetch part of a transaction. + +This is achieved by building on the famous [tokio](https://tokio.rs/) runtime and [:material-github: reqwest](https://github.com/seanmonstar/reqwest) crates, among others. + +### boulder + +At the time of writing our port of boulder hasn't quite *yet* caught up with the original implementation, given all of the ground we covered with moss. With that said, we decided to step back and evaluate where we could improve upon the first version of boulder. + +:::note[What is boulder?] +Boulder is the tool we use to generate installable `.stone` packages from a YAML recipe, `stone.yml`. The recipe contains machine-readable metadata in a declarative format, along with "steps" (instructions) used to build the package. + +Leveraging a series of intelligent patterns and helpers, boulder can analyse and collect the build artefacts into the correct subpackages and automatically discover most runtime dependencies, thus vastly decreasing the workload for the developer. + + + Learn More + +::: + +![boulder in action](./boulder.webp "terminal showing boulder in action") + +#### User namespaces (rootless) + +Arguably one of the most important changes: `boulder` no longer requires running as root. Even better, we make use of user namespaces and execute using `clone` - meaning we no longer need a separate binary (`mason`) within the container environment! + +#### Superior error handling + +Thanks to everything being a self-contained binary, we can tell you up front that you're using unknown action or definition macros (i.e. `%(datadir)`) before attempting any builds, saving you valuable time. + +#### Breakpoints + +When this was first proposed, I did a double take. Breakpoints.. in recipes... ? No, seriously. Not only was it proposed, but between ermo and tarkah, they only went and *implemented it*. You can now use special macros within your recipe to add debugging breakpoints in your recipe to drop to a shell within the container and debug the more difficult builds! + +Our documentation will be updated when the new boulder ships a stable build. + +#### Unified codebase + +We now build both `moss` and `boulder` from the `moss-rs` repository, allowing them to share the same code and mechanisms for dealing with dependencies, system roots, etc., allowing for far higher levels of integration than were previously possible. + +A feature we're currently working on allows you to use a single directory for the system "root", where all the unpacking, fetching and databases happen, while almost *instantly* creating new build roots! + +## What We Already Have + +I get it. "All this rewriting - how will you ever make progress" ? It's a fair question. We didn't throw anything away, quite the opposite. Our repos, tooling, are all still in use enabling us to build the distribution in parallel to the ongoing rewrites. + +### Build infrastructure + +For a long time, we've had our [:material-view-dashboard: dashboard](https://dash.serpentos.com) up and running for our build infrastructure. Sure, we're going to rewrite it beyond the "Proof of concept" stage when required - but for now it still serves us well. Admittedly there is a deep issue within the `druntime` causing thread leaks over time, but we have it restarting via `systemd` every 24 hours as a bandaid. Needless to say, we're big fans of memory safety now. + +Our current deployment watches git repos in our [:material-github: snekpit](https://github.com/snekpit) collection, and automatically forms build queues from the pending builds, ordering them by dependency. Using two of ermo's beefy builders, the packages are automatically built and deployed to our repository. Our workflow means that maintainers only have to merge a pull request for builds to appear in the `volatile` repository. + +### Legacy tooling + +Despite having some warts, the build infrastructure is still able to use legacy `moss` and `boulder` to make our new builds available, and we've been minimally patching these old tools to support our transition to the Rust based tooling. As of this moment, `moss-rs` officially supersedes the D version, and our new `boulder` is quickly catching up. + +Long story short, we're still building on all fronts and aren't blocked anywhere. + +### Live repository + +Builds are published to our `volatile` repository, and we do have a fully functioning software repository despite being quite small. Once we're happy with the new `moss` we'll speed up in our packaging efforts. With that said, we do have two kernels (`kvm` and `desktop`) and even `gtk-4` at this early stage! + +## The immediate future + +OK let's tldr this: + + - Rust ports are coming along fantastically + - moss has been replaced + - still making use of existing tools and infrastructure to build the OS + - actually building the fully bootstrapped, self-contained OS, built on libc++, clang, glibc, etc. + - actively pushing updates to binary repository with fully automated pipeline + - new stuff is :rocket: blazing fast + +### System triggers + +We're experimenting with system triggers right now, execution units that run to fully bake the OS state after each transaction completes. In keeping with our stateless philosophy (and pending adoption of `hermetic /usr`), we need triggers that understand the union state of the operating system, not individual per-package triggers. + +A YAML-based trigger recipe has been baked up, which deliberately avoids the use of shell helpers and execution security weaknesses by using an extended variant of globs: + +```rust + let pattern = "/usr/lib/modules/(version:*)/*".parse::()?; +``` + +Our new [:material-github: fnmatch](https://github.com/serpent-os/moss-rs/tree/main/crates/fnmatch) crate, heavily inspired by Python fnmatch, compiles specially format glob strings to Regex, extending with capture groups for named variables. In YAML, it looks a bit like this: + +```yaml +handlers: + depmod: + run: /sbin/depmod + args: ["-a", "$(version)"] + +## Link paths to handlers and filter on the type +paths: + "/usr/lib/modules/(version:*)/kernel" : + handlers: + - depmod + type: directory +``` + +Once we roll out the trigger support, we unblock the packaging of critical items like `gtk-3` for gdm, as well as enable usable installations and ISOs. Note that they differ from classical triggers due to the architecture of moss: they need to run in an isolated namespace (container) so we can ensure the staged transaction won't break. Additionally we cache and collect the trigger artefacts to speed up each filesystem operation, giving us a first: versioned state artefacts. + +### Boot management + +Our plan is to build on my prior work in the Solus boot management design and [:material-github: clr-boot-manager](https://github.com/clearlinux/clr-boot-manager), addressing some long standing shortcomings. Most importantly, our new module will not need to inspect or manage the OS filesystem as `moss` will be able to provide all of the relevant information (full accounting for all used paths in **all** transactions). + +Initially we will focus on UEFI and supporting the [:material-github: Discoverable Partitions Specification](https://github.com/uapi-group/specifications/blob/main/specs/discoverable_partitions_specification.md) by way of `XBOOTLDR` partitions, the `ESP` and the [Boot Loader Interface](https://systemd.io/BOOT_LOADER_INTERFACE/). Currently we have no plans to support `Unified Kernel Images` as the approach taken by CBM (and soon, moss) alleviates the data concerns of dealing with `vfat`. However, as and when UKIs gain configurable, standardised behaviour for `cmdline` control we will investigate their use. Until that point please note we prebuild our `initrd` images and ship them directly in our packages, as Solus has done so for years already. + +The net gain for taking control of boot management will be the deduplication and garbage collection of assets installed to either the `ESP` or `XBOOTLDR` partitions, along with generation of boot entries relevant to Serpent OS: Transaction specific entries allowing us to directly boot back to older installs in case upgrades go wrong. + +Last, but certainly not least, this approach will make dual-booting with another OS less irritating by no longer being bound by a fixed ESP size. + +### Desktop + +Our plan is to produce a GNOME Shell based live ISO containing the bare basics to get everything else validated, and open the path to installations. In short, it will include: + + - Desktop. + - moss tooling + - web browser (firefox) + - terminal (potentially `gnome-console`) + - flatpak + - Eventually: GNOME Software (integrated with flatpak+moss) + +It should be obvious we're not building a general purpose home-use distribution. However, we also won't *stop* people using Serpent OS for their needs, and we plan to beef up the repository, fully support flatpak and eventually custom moss source recipe repos. (ala AUR) + +### Safety is important + +Our journey over the last few years through D, DIP1000, Rust and memory safety has been incredibly eye opening for us. To our own surprise, memory safety has become a huge interest and concern for the team. Not only have we embarked on a full transition of our tooling to Rust, we've started looking at the OS itself. + +As an early indication of our intent, our [curl](https://curl.se/) package is now [built with the `rustls` backend](https://github.com/snekpit/main/commit/7bccc31388a68b79ed42962501d27f97187b232c) and *not* OpenSSL. Needless to say, we're very keen to reduce the surface area for attacks by adopting alternative, safer implementations where possible. Other projects we're keeping a very close eye on include: + + - [sudo-rs](https://github.com/memorysafety/sudo-rs) + - [uutils/coreutils](https://github.com/uutils/coreutils) + +## Looking to next year + +With most of us entering our holiday leave, and the new year practically around the corner, the Serpent OS team wishes you many happy returns and a *wonderful* new year. With one blocker being worked on (triggers), and the last on the horizon (boot management), your dream installation of Serpent OS is firmly within reach. + +In the spirit of the season..... if you feel like supporting our project, you can help with the rising cost of living and our expanding hosting requirements via GitHub Sponsors[^1]! + +❤️ + + + +--- +[^1]: Since this blog post, we no longer use GitHub Sponsors. For funding, please see our [Funding Page](/sponsor). diff --git a/src/content/docs/blog/2023/12/19/end-of-year-summary/index.mdx b/src/content/docs/blog/2023/12/19/end-of-year-summary/index.mdx deleted file mode 100644 index d93554bd..00000000 --- a/src/content/docs/blog/2023/12/19/end-of-year-summary/index.mdx +++ /dev/null @@ -1,287 +0,0 @@ ---- -title: "End Of Year Summary" -date: 2023-12-19T13:07:16+00:00 -draft: false -authors: [ikey, ermo] -tags: [news] -license: "CC-BY-ND-4.0" -copyright: "Copyright © 2023 aerynOS Developers" ---- -import Icon from "@/components/ui/Icon.astro"; -import { Aside } from '@astrojs/starlight/components'; - -Burning question - how long before we can use Serpent OS on our development systems? -It's a fair question - and one that requires context to answer. Let's start our -new monthly update cycle with a recap of progress so far. - -![screenshot of moss](featured.webp) - -## Introducing tarkah - -We'd like to extend a huge official welcome, and thank you, to newest team member Cory Forsstrom (aka `tarkah`)! - - - -Firstly, a quote from Cory: - -> I've been a long time Solus user and was very excited about what serpent os was doing. Really got invested once I started diving into the D code and seeing how powerful the tools and ideas were. The Rust rewrite was just a perfect storm for me with my experience there and my desire to make contributions. Getting to know you and ermo has just been icing on the cake, you've both been so welcoming and friendly. So lots of fun times. - -> On the personal side, I'm on the west coast in the States, have a lovely wife, just had a baby girl and am enjoying my time with fatherhood and coding - -Chances are you know tarkah for his contributions to [iced-rs](https://github.com/iced-rs/iced), and to [Solus](https://getsol.us). His contributions to -the [moss-rs](https://github.com/serpent-os/moss-rs) project have been absolutely critical, and his patience essential as we got ourselves quickly up to -speed with Rust. It is entirely fair to say that our Rust efforts would not have been possible without him! - -## Rust Progress - -I think it is fair to say that people collectively facepalmed when we announced our plans -to [adopt Rust](https://serpentos.com/blog/2023/09/06/oxidised-moss/) - assuming this would be a huge set back. We're really -happy to report this is **not** the case, and we've made tremendous progress in this area. - -### moss - -Our Rust based `moss` system package tool is now leaps and bounds ahead of its predecessor, -supporting the original features and more! - - - State management - - Garbage collection of old transactions (via `state prune`) - - Optimised write order of transactions (see more below) - - File conflict detection - - Parallel fetching and decompression of packages - - Update support (as `sync`) - - Local and remote repositories with priority-based layering - - Support for building with `musl` for fully self-contained, static builds of moss with zero system dependencies. - -We now have a version-agnostic "stone" crate that is more efficient at reading packages than our original D code. -In addition, it benefits from the memory safety promises of Rust. - - - -#### Virtual filesystem management - -OK, lets look at the problem: Every transaction in moss requires us to generate a new staging tree containing all of the directories and files for the -`/usr` tree, using hard links, from content addressable storage. As one might expect, creating tens of thousands of new nodes is really, really slow! - -To alleviate this issue we created a [vfs](https://github.com/serpent-os/moss-rs/tree/main/crates/vfs) crate in moss to optimise *how* we construct each -transaction tree: - - - Organise all incoming registered paths into a specific order - - Inject all missing implicitly created directories for accounting - - Re-parent nodes in tree which live under redirected symlinks - - Compile into a recursive data-structure using relative names. - -There is a multipart "build" for the tree that detects any filesystem conflicts before any files have been written to disk, preventing broken transactions. -With the fully built data structure we can recurse a *virtual filesystem*, making use of the `at` family of functions like `linkat`, `mkdirat`, to "blit" -a filesystem tree without requiring any path resolution, in the most efficient order possible. - -The net result? New transaction roots are created in milliseconds, not seconds. - -#### Upgrade support - *kind of* - -moss now offers a `sync` command in place of the conventional `upgrade` one might expect. Rather than upgrading to the latest *version* of a package, we -rely on moss and our infrastructure tooling work in conjunction to ensure we can simply *synchronize* the package selection with the the "tip" candidates -in the configured repositories, as ordered by priority. From a distro-builder PoV, we can now revert and pull builds. For a user PoV, you can remove a test -repository and `moss sync` to go back to the official versions of packages. - -Generally speaking, when you `moss sync` you will be seeing new packages, however. :smile: This feature is being built to enable a future feature: exported states. Want to try a new edition? Sync to the lockfile. Need to quickly and reproducibly deploy containers from a locked down configuration? You get the idea. - -![moss sync](sync.webp) - -#### Fully `async` - -The moss tool is now completely asynchronous, making efficient use of both coroutines and OS threads to perform as much work possible in the shortest space -of time. This allows us to download, fetch, check and unpack many packages at the same time without being blocked on network or disk, greatly speeding up -the fetch part of a transaction. - -This is achieved by building on the famous [tokio](https://tokio.rs/) runtime and [:material-github: reqwest](https://github.com/seanmonstar/reqwest) crates, among others. - -### boulder - -At the time of writing our port of boulder hasn't quite *yet* caught up with the original implementation, given all -of the ground we covered with moss. With that said, we decided to step back and evaluate where we could improve upon -the first version of boulder. - - - -![boulder in action](boulder.webp) - -#### User namespaces (rootless) - -Arguably one of the most important changes: `boulder` no longer requires running as root. Even better, we make use of -user namespaces and execute using `clone` - meaning we no longer need a separate binary (`mason`) within the container -environment! - -#### Superior error handling - -Thanks to everything being a self-contained binary, we can tell you up front that you're using unknown action or definition -macros (i.e. `%(datadir)`) before attempting any builds, saving you valuable time. - -#### Breakpoints - -When this was first proposed, I did a double take. Breakpoints.. in recipes... ? No, seriously. Not only was it proposed, -but between ermo and tarkah, they only went and *implemented it*. You can now use special macros within your recipe to add -debugging breakpoints in your recipe to drop to a shell within the container and debug the more difficult builds! - -Our documentation will be updated when the new boulder ships a stable build. - -#### Unified codebase - -We now build both `moss` and `boulder` from the `moss-rs` repository, allowing them to share the same code and mechanisms -for dealing with dependencies, system roots, etc., allowing for far higher levels of integration than were previously possible. - -A feature we're currently working on allows you to use a single directory for the system "root", where all the unpacking, fetching -and databases happen, while almost *instantly* creating new build roots! - -## What We Already Have - -I get it. "All this rewriting - how will you ever make progress" ? It's a fair question. We didn't throw anything -away, quite the opposite. Our repos, tooling, are all still in use enabling us to build the distribution in parallel -to the ongoing rewrites. - -### Build infrastructure - -For a long time, we've had our [:material-view-dashboard: dashboard](https://dash.serpentos.com) up and running for our build infrastructure. -Sure, we're going to rewrite it beyond the "Proof of concept" stage when required - but for now it still serves us well. Admittedly there is -a deep issue within the `druntime` causing thread leaks over time, but we have it restarting via `systemd` every 24 hours as a bandaid. Needless -to say, we're big fans of memory safety now. - -Our current deployment watches git repos in our [:material-github: snekpit](https://github.com/snekpit) collection, and automatically forms build -queues from the pending builds, ordering them by dependency. Using two of ermo's beefy builders, the packages are automatically built and deployed -to our repository. Our workflow means that maintainers only have to merge a pull request for builds to appear in the `volatile` repository. - -### Legacy tooling - -Despite having some warts, the build infrastructure is still able to use legacy `moss` and `boulder` to make our new builds available, and we've -been minimally patching these old tools to support our transition to the Rust based tooling. As of this moment, `moss-rs` officially supersedes -the D version, and our new `boulder` is quickly catching up. - -Long story short, we're still building on all fronts and aren't blocked anywhere. - -### Live repository - -Builds are published to our `volatile` repository, and we do have a fully functioning software repository despite being quite small. Once we're -happy with the new `moss` we'll speed up in our packaging efforts. With that said, we do have two kernels (`kvm` and `desktop`) and even `gtk-4` -at this early stage! - -## The immediate future - -OK let's tldr this: - - - Rust ports are coming along fantastically - - moss has been replaced - - still making use of existing tools and infrastructure to build the OS - - actually building the fully bootstrapped, self-contained OS, built on libc++, clang, glibc, etc. - - actively pushing updates to binary repository with fully automated pipeline - - new stuff is :rocket: blazing fast - -### System triggers - -We're experimenting with system triggers right now, execution units that run to fully bake the OS state after each transaction completes. In keeping -with our stateless philosophy (and pending adoption of `hermetic /usr`), we need triggers that understand the union state of the operating system, not -individual per-package triggers. - -A YAML-based trigger recipe has been baked up, which deliberately avoids the use of shell helpers and execution security weaknesses by using an extended -variant of globs: - -```rust - let pattern = "/usr/lib/modules/(version:*)/*".parse::()?; -``` - -Our new [:material-github: fnmatch](https://github.com/serpent-os/moss-rs/tree/main/crates/fnmatch) crate, heavily inspired by Python fnmatch, compiles specially format glob strings to Regex, extending with capture -groups for named variables. In YAML, it looks a bit like this: - -```yaml -handlers: - depmod: - run: /sbin/depmod - args: ["-a", "$(version)"] - -## Link paths to handlers and filter on the type -paths: - "/usr/lib/modules/(version:*)/kernel" : - handlers: - - depmod - type: directory -``` - -Once we roll out the trigger support, we unblock the packaging of critical items like `gtk-3` for gdm, as well as enable usable installations and ISOs. -Note that they differ from classical triggers due to the architecture of moss: they need to run in an isolated namespace (container) so we can ensure -the staged transaction won't break. Additionally we cache and collect the trigger artefacts to speed up each filesystem operation, giving us a first: -versioned state artefacts. - -### Boot management - -Our plan is to build on my prior work in the Solus boot management design and [:material-github: clr-boot-manager](https://github.com/clearlinux/clr-boot-manager), addressing some long standing shortcomings. Most importantly, our new module will not need to inspect or manage the OS filesystem -as `moss` will be able to provide all of the relevant information (full accounting for all used paths in **all** transactions). - -Initially we will focus on UEFI and supporting the [:material-github: Discoverable Partitions Specification](https://github.com/uapi-group/specifications/blob/main/specs/discoverable_partitions_specification.md) by way of `XBOOTLDR` partitions, the `ESP` and the [Boot Loader Interface](https://systemd.io/BOOT_LOADER_INTERFACE/). Currently we have no plans to support `Unified Kernel Images` as the approach taken by CBM (and soon, moss) alleviates the -data concerns of dealing with `vfat`. However, as and when UKIs gain configurable, standardised behaviour for `cmdline` control we will investigate their -use. Until that point please note we prebuild our `initrd` images and ship them directly in our packages, as Solus has done so for years already. - -The net gain for taking control of boot management will be the deduplication and garbage collection of assets installed to either the `ESP` or `XBOOTLDR` -partitions, along with generation of boot entries relevant to Serpent OS: Transaction specific entries allowing us to directly boot back to older installs -in case upgrades go wrong. - -Last, but certainly not least, this approach will make dual-booting with another OS less irritating by no longer being bound by a fixed ESP size. - -### Desktop - -Our plan is to produce a GNOME Shell based live ISO containing the bare basics to get everything else validated, and open the path to installations. -In short, it will include: - - - Desktop. - - moss tooling - - web browser (firefox) - - terminal (potentially `gnome-console`) - - flatpak - - Eventually: GNOME Software (integrated with flatpak+moss) - -It should be obvious we're not building a general purpose home-use distribution. However, we also won't *stop* people using Serpent OS for their needs, -and we plan to beef up the repository, fully support flatpak and eventually custom moss source recipe repos. (ala AUR) - -### Safety is important - -Our journey over the last few years through D, DIP1000, Rust and memory safety has been incredibly eye opening for us. To our own surprise, -memory safety has become a huge interest and concern for the team. Not only have we embarked on a full transition of our tooling to Rust, -we've started looking at the OS itself. - -As an early indication of our intent, our [curl](https://curl.se/) package is now [built with the `rustls` backend](https://github.com/snekpit/main/commit/7bccc31388a68b79ed42962501d27f97187b232c) and *not* OpenSSL. Needless to say, we're very keen to reduce the surface area for attacks by -adopting alternative, safer implementations where possible. Other projects we're keeping a very close eye on include: - - - [:material-github: sudo-rs](https://github.com/memorysafety/sudo-rs) - - [:material-github: uutils/coreutils](https://github.com/uutils/coreutils) - -## Looking to next year - -With most of us entering our holiday leave, and the new year practically around the corner, the Serpent OS team wishes you many happy returns and -a *wonderful* new year. With one blocker being worked on (triggers), and the last on the horizon (boot management), your dream installation of -Serpent OS is firmly within reach. - -In the spirit of the season..... if you feel like supporting our project, you can help with the rising cost of living and our expanding hosting -requirements via GitHub Sponsors! - - - -[Sponsor us!](https://github.com/sponsors/ikeycode) diff --git a/src/content/docs/blog/2024/01/19/january-updates/index.md b/src/content/docs/blog/2024/01/19/january-updates/index.md index f8610d3d..0c91bcad 100644 --- a/src/content/docs/blog/2024/01/19/january-updates/index.md +++ b/src/content/docs/blog/2024/01/19/january-updates/index.md @@ -8,100 +8,65 @@ license: "CC-BY-ND-4.0" copyright: "Copyright © 2024 aerynOS Developers" --- -Precisely one month since our end of year summary, so, what have we been up to? -With a new year, a new start was long overdue. We're pleased to announce that we've -finally shifted our butts over to our own server (again). +Precisely one month since our end of year summary, so, what have we been up to? With a new year, a new start was long overdue. We're pleased to announce that we've finally shifted our butts over to our own server (again). -While this isn't quite as chunky a news update as the last post, we rather hope you appreciate -the regularity of the news. +While this isn't quite as chunky a news update as the last post, we rather hope you appreciate the regularity of the news. -![moss in action](Featured.webp) - - +![moss in action](Featured.webp "Terminal showing moss in action installing packages") ## Infrastructure, revisited -'tarkah' has been tirelessly porting `moss-service`, the core foundation of our automated build system, -to a shiny new Rust codebase. While we don't have an *immediately* pressing need for the port to -come online, it will certainly address some "v1 issues" and ensure the best-in-class development -experience for our contributors. +'tarkah' has been tirelessly porting `moss-service`, the core foundation of our automated build system, to a shiny new Rust codebase. While we don't have an *immediately* pressing need for the port to come online, it will certainly address some "v1 issues" and ensure the best-in-class development experience for our contributors. ## Licensing -In our pursuit for a sustainable, community oriented project, we've been doing a lot of in house organisation. -Very importantly, we've now [relicensed](https://github.com/serpent-os/recipes/issues/2) our packaging recipes to [MPL-2.0](https://spdx.org/licenses/MPL-2.0.html), bringing them in line with the license chosen for our current engineering efforts (`moss`, etc). +In our pursuit for a sustainable, community oriented project, we've been doing a lot of in house organisation. Very importantly, we've now [relicensed](https://github.com/serpent-os/recipes/issues/2) our packaging recipes to [MPL-2.0](https://spdx.org/licenses/MPL-2.0.html), bringing them in line with the license chosen for our current engineering efforts (`moss`, etc). -To reiterate a critical point, we require that our contributions are attributed to the virtual collective "Serpent OS Developers" -to ensure we're unable to relicense any contributions *without express consent of the contributors involved*. +To reiterate a critical point, we require that our contributions are attributed to the virtual collective "Serpent OS Developers" to ensure we're unable to relicense any contributions *without express consent of the contributors involved*. -Additionally we picked MPL-2.0 over Zlib due to some distinct advantages in relation to patent trolls. In combination -we believe both the project and our users will have the greatest protection from patent trolls and fiscally-motivated -relicensing. +Additionally we picked MPL-2.0 over Zlib due to some distinct advantages in relation to patent trolls. In combination we believe both the project and our users will have the greatest protection from patent trolls and fiscally-motivated relicensing. ## As I Raise My Trigger Finger -As you may recall from the last post, we announced our intention to add system triggers to moss. -We also stated that our triggers must run in **isolation** thanks to the unique architecture of our -package management. After much consideration, we've landed on a strategy that will work for us. +As you may recall from the last post, we announced our intention to add system triggers to moss. We also stated that our triggers must run in **isolation** thanks to the unique architecture of our package management. After much consideration, we've landed on a strategy that will work for us. ### Post-blit triggers -Every *mutable operation* in moss results in a fresh internal transaction, which is present as a full -`/usr` tree inside a staging tree. The majority of triggers can be executed here, just before we "activate" -the new rootfs. This allows us to detect potential trigger failures, and abort the activation. +Every *mutable operation* in moss results in a fresh internal transaction, which is present as a full `/usr` tree inside a staging tree. The majority of triggers can be executed here, just before we "activate" the new rootfs. This allows us to detect potential trigger failures, and abort the activation. -These YAML-defined triggers will be run in an isolated environment (clone-based container) with the new `/usr` -and read-only access to `/etc`. +These YAML-defined triggers will be run in an isolated environment (clone-based container) with the new `/usr` and read-only access to `/etc`. ### Post-activation triggers -Once we have a freshly activated `/usr` (ie new system view) we can run a very limited set of post-activation -triggers. An obvious example may be `systemctl daemon-reexec`. Note that by splitting post-blit and post-activation -triggers, we leave the door open to multiple activation strategies for non-simple updates, including soft reboots -and kernel patching. +Once we have a freshly activated `/usr` (ie new system view) we can run a very limited set of post-activation triggers. An obvious example may be `systemctl daemon-reexec`. Note that by splitting post-blit and post-activation triggers, we leave the door open to multiple activation strategies for non-simple updates, including soft reboots and kernel patching. ## Packaging updates! -With all of the migration work, our builds have also resumed. In the space of a few short days we've had 32 package -builds. Ok, not a huge number, but we also have 15 outstanding pull requests to our new recipes repo! +With all of the migration work, our builds have also resumed. In the space of a few short days we've had 32 package builds. Ok, not a huge number, but we also have 15 outstanding pull requests to our new recipes repo! -In order to make life not only simpler for us, but for Solus in its future role as a source-derivative of Serpent, we've -unified all of our git recipe repos into a new single repo, [recipes](https://github.com/serpent-os/recipes). +In order to make life not only simpler for us, but for Solus in its future role as a source-derivative of Serpent, we've unified all of our git recipe repos into a new single repo, [recipes](https://github.com/serpent-os/recipes). ## Improved hosting -Our devops team has been working hard to mitigate bus factor with our transition to the new infrastructure, Bitwarden account, -shared DNS, and opentofu configuration. +Our devops team has been working hard to mitigate bus factor with our transition to the new infrastructure, Bitwarden account, shared DNS, and opentofu configuration. ### Self Hosting. Again -As part of the recovery process for Solus, the two projects shared resources, including -a large Hetzner node. This has actually worked remarkably well for a long time now, however -with Serpent growing we do not wish to impact the *availability* and *stability* of the Solus -update experience. +As part of the recovery process for Solus, the two projects shared resources, including a large Hetzner node. This has actually worked remarkably well for a long time now, however with Serpent growing we do not wish to impact the *availability* and *stability* of the Solus update experience. -Long story short, money from [GitHub Sponsors](https://github.com/sponsors/ikeycode) (You guys! < 3) is now paying for an AX52 server -from Hetzner. It's sufficiently powerful that we're running a backup *builder* there as well as -repo management and build controller. +Long story short, money from GitHub Sponsors[^1] (You guys! ❤️) is now paying for an AX52 server from Hetzner. It's sufficiently powerful that we're running a backup *builder* there as well as repo management and build controller. ### El Cheapo Web Hosting -Looking at our project's use of static generators - we decided to review our hosting for web content. -Long story short, we deemed it entirely unnecessary. We're now using GitHub pages to deploy this -website and our [documentation site](https://docs.serpentos.com) via GitHub actions. Don't worry, -we use git, we have backups, and we manage the infrastructure using `IaC` practices. +Looking at our project's use of static generators - we decided to review our hosting for web content. Long story short, we deemed it entirely unnecessary. We're now using GitHub pages to deploy this +website and our [documentation site](https://docs.serpentos.com) via GitHub actions. Don't worry, we use git, we have backups, and we manage the infrastructure using `IaC` practices. ## Clarifying our baseline target -Lately there have been some questions in both Serpent and Solus as to what our baseline will be. -In Serpent OS, this will be at minimum `x86_64-v2`, with `x86_64-v3x` packages automatically -offered if the system base criteria are met. +Lately there have been some questions in both Serpent and Solus as to what our baseline will be. In Serpent OS, this will be at minimum `x86_64-v2`, with `x86_64-v3x` packages automatically offered if the system base criteria are met. -This decision has been made to offer the widest baseline compatibility without compromising heavily -on system performance (Such that `x86_64-generic` is unsupported in Serpent OS). We will continue -offering both `v2` and `v3x` until it is no longer feasible from a storage/financial perspective. -Source derivatives (Including Solus) are free to chart their own course here, if necessary, -and still benefit from tight upstream integration. +This decision has been made to offer the widest baseline compatibility without compromising heavily on system performance (Such that `x86_64-generic` is unsupported in Serpent OS). We will continue +offering both `v2` and `v3x` until it is no longer feasible from a storage/financial perspective. Source derivatives (Including Solus) are free to chart their own course here, if necessary, and still benefit from tight upstream integration. ### Towards Next Month @@ -111,3 +76,6 @@ In our next update, we'll have quite a bit to cover, including: - Progress with GNOME packaging/readiness - Boulder / Infrastructure porting progress - Actually useful Serpent images + + --- + [^1]: Since this blog post, we no longer use GitHub Sponsors. For funding, please see our [Funding Page](/sponsor). diff --git a/src/content/docs/blog/2024/02/29/end-of-feb-update/index.md b/src/content/docs/blog/2024/02/29/end-of-feb-update/index.md new file mode 100644 index 00000000..bc356efc --- /dev/null +++ b/src/content/docs/blog/2024/02/29/end-of-feb-update/index.md @@ -0,0 +1,90 @@ +--- +title: "End of February Update" +date: 2024-02-29T18:32:35+00:00 +draft: false +authors: [ikey] +tags: [news] +license: "CC-BY-ND-4.0" +copyright: "Copyright © 2024 aerynOS Developers" +--- + +This update came a little later in the month, as we've got a lot of exciting news to share. Everything from `boulder` in Rust, to the GNOME 45 Desktop complete with `moss triggers` built atop a rebootstrapped toolchain. + +## Rusty Boulder + +We're pleased to announce that over the course of this weekend, once testing has completed we'll deploy the latest version of `boulder`, our packaging build tool. This has been given the Rust treatment, directly sharing the codebase with `moss`. + +![Boulder in action](featured.webp "Boulder in action") + +Unlike our PoC implementation, the new `boulder` makes use of the `clone` syscall to execute portions of itself under user namespaces, eliminating a longstanding deployment irritation of having the main build binary inside packaging. + +Just like `moss` it makes use of `tokio` for async right where we need it, and is rootless to make life easier for our contributors. Far from being just a direct port of the proof of concept, it adds new tools to empower developers, such as **breakpoints in packaging recipes**. + +![Breakpoints, in recipes](BoulderBreakpoints.webp "Breakpoints in recipes") + +## Triggers, integrated! + +After much discussion we finally integrated support for *triggers* in `moss`. These are actions that are executed at different stages of package management operations to finalize or "bake" some state based on the unified components of the installation. + +:::note[Find out more about triggers] +In order for triggers to be *useful*, they need to be shipped in packages installed on client machines. These are easy-to-write `yaml` files that live in either `/usr/share/moss/triggers/sys.d` or within +`/usr/share/moss/triggers/tx.d`, depending on their scope. + + + Learn More + +::: + +Note that right now triggers are re-executed for each transaction, and no caching support is yet in place. With that said, they're still very fast and we plan to add a store based cache to prevent unnecessary execution. One of the greatest benefits of our triggers is ensuring these actions run within the right context under a `namespace`, or container. + +## In pursuit of stateless + +We've iterated many times over the years our belief in the stateless philosophy, and as a design decision we mandated that packages can **only** contain `/usr` files. This has meant we've had to get a bit inventive in shipping working packages out of the box. + +One such strategy has been to abolish the reliance on package triggers for system users and groups. In the past we've relied on `systemd-sysusers` to construct these accounts, such as `gdm`. A downside is there is no simple way to handle the cleanup of these accounts. + +We're now happily using systemd `userdb` user and group drop-in records via `nss-systemd` to provide the default users + groups (including `users`) group with centrally managed, fixed UIDs and GIDs in our git recipes. + +:::note[Find out more about system users] +In order to make management of users easier, we employ a specific systemd mechanism that makes drop-in user and group records automatically "appear" to applications that use the proper libc APIs. These are JSON format files that live in `/usr/lib/userdb` and are exposed to applications via `nss`. + + + Learn More + +::: + +## GNOME Desktop + +Thanks to triggers now being fully integrated in Serpent OS via `moss`, we've actually managed to package up most of the basic parts of GNOME 45. This includes a functioning GDM, GNOME Shell and a handful of applications. + +![GNOME on Serpent OS](./KeepOnRunning.webp "GNOME running on Serpent OS") + +## Packaging work + +Outside of the heavy work on the tooling and GNOME, we've also been tidying up our recipes repo and recently performed a fresh rebootstrap using: + + - glibc 2.39 + - binutils 2.42 + - GCC 13.2.0 + - LLVM 17.4.0 + - Rust 1.78.0 + +Note that our primary toolchain is `glibc`, `lld`, `clang` / `clang++`, `libc++`, `libunwind`, etc. The majority of packages are built with this LLVM + glibc toolchain, and a subset of packages currently build with the GNU toolchain due to various wrong assumptions or use of GCC extensions. + +Our kernel has always been built with clang. + +## Images + +Our shortterm plan is to deliver a prealpha image for the general public to test, and to get a "feel" for Serpent OS and associated tooling. As soon as this is out of the door we will immediately pivot to getting Serpent OS fully functioning on baremetal, with explicit targets in mind: the hardware that core developers are currently using. + +These images will be available but not promoted for general use - given their intended use as dogfooding systems. This is part of our shift to the `oxide-alpha-1` target milestone, which will be discussed further in our next post +announcing the prealpha image. + +## Next time + +Our next monthly blog post should return to its regular slot, with updates on the following topics: + + - Limiting `async` in moss to targeted areas (PR ready and pending review) + - Life on Serpent: How baremetal testing + enabling is going. + + Until then, please, stay tuned! diff --git a/src/content/docs/blog/2024/02/29/end-of-feb-update/index.mdx b/src/content/docs/blog/2024/02/29/end-of-feb-update/index.mdx deleted file mode 100644 index 2ccef941..00000000 --- a/src/content/docs/blog/2024/02/29/end-of-feb-update/index.mdx +++ /dev/null @@ -1,126 +0,0 @@ ---- -title: "End of February Update" -date: 2024-02-29T18:32:35+00:00 -draft: false -authors: [ikey] -tags: [news] -license: "CC-BY-ND-4.0" -copyright: "Copyright © 2024 aerynOS Developers" ---- -import { Aside } from '@astrojs/starlight/components'; - -This update came a little later in the month, as we've got a lot of exciting news to share. -Everything from `boulder` in Rust, to the GNOME 45 Desktop complete with `moss triggers` built -atop a rebootstrapped toolchain. - -## Rusty Boulder - -We're pleased to announce that over the course of this weekend, once testing has completed we'll -deploy the latest version of `boulder`, our packaging build tool. This has been given the Rust treatment, -directly sharing the codebase with `moss`. - -![Boulder in action](featured.webp) - - - -Unlike our PoC implementation, the new `boulder` makes use of the `clone` syscall to execute portions of -itself under user namespaces, eliminating a longstanding deployment irritation of having the main build binary -inside packaging. - -Just like `moss` it makes use of `tokio` for async right where we need it, and is rootless to -make life easier for our contributors. Far from being just a direct port of the proof of concept, it adds -new tools to empower developers, such as **breakpoints in packaging recipes**. - -![Breakpoints, in recipes](BoulderBreakpoints.webp) - -## Triggers, integrated! - -After much discussion we finally integrated support for *triggers* in `moss`. These are actions -that are executed at different stages of package management operations to finalize or "bake" -some state based on the unified components of the installation. - - - - -Note that right now triggers are re-executed for each transaction, and no caching support is yet in place. -With that said, they're still very fast and we plan to add a store based cache to prevent unnecessary -execution. One of the greatest benefits of our triggers is ensuring these actions run within the right -context under a `namespace`, or container. - -## In pursuit of stateless - -We've iterated many times over the years our belief in the stateless philosophy, and as a design -decision we mandated that packages can **only** contain `/usr` files. This has meant we've had to -get a bit inventive in shipping working packages out of the box. - -One such strategy has been to abolish the reliance on package triggers for system users and groups. -In the past we've relied on `systemd-sysusers` to construct these accounts, such as `gdm`. A downside -is there is no simple way to handle the cleanup of these accounts. - -We're now happily using systemd `userdb` user and group drop-in records via `nss-systemd` to provide -the default users + groups (including `users`) group with centrally managed, fixed UIDs and GIDs in our -git recipes. - - - -## GNOME Desktop - -Thanks to triggers now being fully integrated in Serpent OS via `moss`, we've actually managed to -package up most of the basic parts of GNOME 45. This includes a functioning GDM, GNOME Shell and -a handful of applications. - -![GNOME on Serpent OS](./KeepOnRunning.webp) - -## Packaging work - -Outside of the heavy work on the tooling and GNOME, we've also been tidying up our recipes repo and -recently performed a fresh rebootstrap using: - - - glibc 2.39 - - binutils 2.42 - - GCC 13.2.0 - - LLVM 17.4.0 - - Rust 1.78.0 - -Note that our primary toolchain is `glibc`, `lld`, `clang` / `clang++`, `libc++`, `libunwind`, etc. -The majority of packages are built with this LLVM + glibc toolchain, and a subset of packages currently -build with the GNU toolchain due to various wrong assumptions or use of GCC extensions. - -Our kernel has always been built with clang. - -## Images - -Our shortterm plan is to deliver a prealpha image for the general public to test, and to get a "feel" for -Serpent OS and associated tooling. As soon as this is out of the door we will immediately pivot to getting -Serpent OS fully functioning on baremetal, with explicit targets in mind: the hardware that core developers -are currently using. - -These images will be available but not promoted for general use - given their intended use as dogfooding systems. -This is part of our shift to the `oxide-alpha-1` target milestone, which will be discussed further in our next post -announcing the prealpha image. - -## Next time - -Our next monthly blog post should return to its regular slot, with updates on the following topics: - - - Limiting `async` in moss to targeted areas (PR ready and pending review) - - Life on Serpent: How baremetal testing + enabling is going. - - Until then, please, stay tuned! diff --git a/src/content/docs/blog/2024/03/31/the-march-of-progress/index.md b/src/content/docs/blog/2024/03/31/the-march-of-progress/index.md index e9bcb2b9..0b3dfba2 100644 --- a/src/content/docs/blog/2024/03/31/the-march-of-progress/index.md +++ b/src/content/docs/blog/2024/03/31/the-march-of-progress/index.md @@ -8,12 +8,9 @@ license: "CC-BY-ND-4.0" copyright: "Copyright © 2024 aerynOS Developers" --- -Despite a brief excursion out of the country for a first-in-a-lifetime holiday, I'm happily back at the desk to bring you up to date with the -latest goings in Serpent OS. TLDR: Loads of awesome, baremetal is enabled, ISO cycle in next couple of weeks. +Despite a brief excursion out of the country for a first-in-a-lifetime holiday, I'm happily back at the desk to bring you up to date with the latest goings in Serpent OS. TLDR: Loads of awesome, baremetal is enabled, ISO cycle in next couple of weeks. -![GNOME on Serpent OS](Gnome.webp) - - +![GNOME on Serpent OS](Gnome.webp "GNOME running on Serpent OS") ## Tooling Improvements @@ -21,23 +18,17 @@ OK, we all know you want to know about the baremetal stuff, but c'mon, check out ### Boulder -We're officially at a point where the Rust boulder implementation has become our "blessed" build tool. This is actually a crate within the `moss` -repository, allowing the two tools to share huge chunks of code. Our focus over the past few weeks has been to ensure we can drop the tool into -our existing build network and "just work", while making it significantly faster. - -One of the most important high level changes is *performance*, in some cases virtually 50% faster packaging times. Our package payloads are now -compressed using multi-threaded zstd compression, and a bunch of internal components were reworked to use faster code paths leading to a significant -speed up in the raw generation of each `.stone` package. +We're officially at a point where the Rust boulder implementation has become our "blessed" build tool. This is actually a crate within the `moss` repository, allowing the two tools to share huge chunks of code. Our focus over the past few weeks has been to ensure we can drop the tool into our existing build network and "just work", while making it significantly faster. -!!! warning "Legacy boulder tool has been retired" +One of the most important high level changes is *performance*, in some cases virtually 50% faster packaging times. Our package payloads are now compressed using multi-threaded zstd compression, and a bunch of internal components were reworked to use faster code paths leading to a significant speed up in the raw generation of each `.stone` package. - As of March 31st, 2024, the legacy D implementation of `boulder` is no longer supported. The runtime component, `mason`, has been removed from the - volatile repository. Our new tool is written in Rust and contains the `moss` code, enabling a self-contained all-in-one approach to buildroot management. +:::danger[Legacy boulder tool has been retired] +As of March 31st, 2024, the legacy D implementation of `boulder` is no longer supported. The runtime component, `mason`, has been removed from the volatile repository. Our new tool is written in Rust and contains the `moss` code, enabling a self-contained all-in-one approach to buildroot management. - It relies on clone-based user namespaces, so is fully "rootless". Please migrate immediately to the new Rust based tool. Not only is it more advanced, - it is significantly faster at generating the raw `.stone` packages. +It relies on clone-based user namespaces, so is fully "rootless". Please migrate immediately to the new Rust based tool. Not only is it more advanced, it is significantly faster at generating the raw `.stone` packages. +::: -![Comparison](Featured.webp) +![Comparison](Featured.webp "Comparison of legacy vs new boulder tools") Other, quite awesome changes: @@ -50,9 +41,7 @@ Other, quite awesome changes: ### Moss -Moss also got some love these past few weeks, including a full migration over to [diesel.rs](https://diesel.rs/). Combined with a limiting of `async` -to largely network bound operations, and some clever performance optimisations under the hood, moss is now faster than ever. In fact, we're able to -generate desktop ISOs in under 20 seconds, and install the full GNOME desktop to a `virtiofs` VM in almost no time. +Moss also got some love these past few weeks, including a full migration over to [diesel.rs](https://diesel.rs/). Combined with a limiting of `async` to largely network bound operations, and some clever performance optimisations under the hood, moss is now faster than ever. In fact, we're able to generate desktop ISOs in under 20 seconds, and install the full GNOME desktop to a `virtiofs` VM in almost no time. - Dropped some unnecessary dependencies - Significant code cleanups @@ -65,23 +54,17 @@ generate desktop ISOs in under 20 seconds, and install the full GNOME desktop to ## Hardware/Desktop Enablings. -Rather than extensively widen our repository at this point we've opted for further low-level enabling of components required for the developer -experience desktop. This includes Network Manager, the desktop kernel package, Vulkan + mesa integration, etc. +Rather than extensively widen our repository at this point we've opted for further low-level enabling of components required for the developer experience desktop. This includes Network Manager, the desktop kernel package, Vulkan + mesa integration, etc. -![Gnome Software](GnomeSW.webp) +![Gnome Software](GnomeSW.webp "Gnome Software running on Serpent OS") -As well as enabling core applications and features such as GNOME Software and Flatpak, we've also enabled preliminary usage on baremetal booting -hardware. Note, this isn't yet installable but we do plan to offer early access installable images to our [sponsors](https://github.com/sponsors/ikeycode)! +As well as enabling core applications and features such as GNOME Software and Flatpak, we've also enabled preliminary usage on baremetal booting hardware. Note, this isn't yet installable but we do plan to offer early access installable images to our sponsors! -Also worth noting.. we've begun to package the Cosmic Epoch desktop from the cool guys over at [System76](https://system76.com/). We're highly interested -in this project and plan to track progress closely, offering it within Serpent OS as a fully featured development focused desktop experience. +Also worth noting.. we've begun to package the Cosmic Epoch desktop from the cool guys over at [System76](https://system76.com/). We're highly interested in this project and plan to track progress closely, offering it within Serpent OS as a fully featured development focused desktop experience. ## The Agenda -This post was finished moments before the March deadline, so needless to say we've been super busy! Thankfully a lot of pivotal pieces have landed now and -we can focus on some big ticket items. Our primary concern is to ship one last "pre-alpha" image for the public, and a test installable pre-alpha ISO for -our sponsors to help gain early feedback. The hope is this will help fund a widening of our infrastructure to support more concurrent repository users and -our own `debuginfod` compatible extensions to our repository. +This post was finished moments before the March deadline, so needless to say we've been super busy! Thankfully a lot of pivotal pieces have landed now and we can focus on some big ticket items. Our primary concern is to ship one last "pre-alpha" image for the public, and a test installable pre-alpha ISO for our sponsors to help gain early feedback. The hope is this will help fund a widening of our infrastructure to support more concurrent repository users and our own `debuginfod` compatible extensions to our repository. The April (and beyond) agenda: @@ -97,5 +80,4 @@ The April (and beyond) agenda: - Get core team running Serpent OS daily. WiFi and GPU support are in place already. - Build an amazing developer experience -On behalf of the Serpent OS team, I look forward to sharing some amazing updates with you towards the end of April! Keep your eyes peeled for the ISO drops -in the coming weeks. +On behalf of the Serpent OS team, I look forward to sharing some amazing updates with you towards the end of April! Keep your eyes peeled for the ISO drops in the coming weeks. diff --git a/src/content/docs/blog/2024/04/30/calm-before-the-storm/index.md b/src/content/docs/blog/2024/04/30/calm-before-the-storm/index.md new file mode 100644 index 00000000..977175a1 --- /dev/null +++ b/src/content/docs/blog/2024/04/30/calm-before-the-storm/index.md @@ -0,0 +1,61 @@ +--- +title: "Calm Before The Storm" +date: 2024-04-30T22:46:17+01:00 +draft: false +authors: [ikey] +tags: [news] +license: "CC-BY-ND-4.0" +copyright: "Copyright © 2024 aerynOS Developers" +--- + +It's that time of month again, and we have some details to share with you on boot management, as well as plans for real installs landing in May. Additionally, plans for "deferred updates" via mandatory reboots have been dropped! + +![wip boot code](Featured.webp "Work in progress boot code") + +## Project priorities + +To better understand the technical deep-dive on boot management further below, we should clarify a slight change in priorities for the project as a whole. For some time now we've served as a technical beacon for Solus to guide them towards better strategy and technical solutions whilst awaiting a situation where a rebase is feasible. + +While we still look forwards to that future, the present day is the most important to us. Serpent OS must now transition from a collection of tools and promises into a daily driver with actual users. We still aim to cater to the more developer/technical oriented end of the spectrum rather than a "desktop distro", and want to start this journey yesterday. + +The boot management code is currently landing over the coming days, and will be immediately followed by an incredibly rudimentary system installer to facilitate an initial adoption of Serpent OS for the general public. Our intent is to ship some early test ISOs for supporters, and by the end of May have a feasible option for people to use and contribute to Serpent OS on real hardware. + +We acknowledge this is an exchange of the pursuit of perfection for a pragmatic rapid iteration cycle and believe the best way forward is to open the doors wide, fixing issues on the job. The initial system will likely be rough and buggy, but it is an official starting point from which we can all build Serpent OS together. + +## Moss + Boulder + +Our Rust tooling has now been in "production" use in our development environment and build servers for the last month without any major hiccup. We have recently discovered some minor issues in our `emul32` (32-bit) package generation which are being resolved, chiefly our move from the improperly named ISA `x86`, now updated to `386` affecting 32-bit dependency chains. + +## Boot + +Our main topic of conversation this month is the enabling of _boot management_ for Serpent OS installations. To be crystal clear, we are not talking about a boot loader. In fact, Serpent OS uses `systemd-boot` for UEFI. Instead, we are talking about the automated management of boot loader *entries*, their corresponding kernel/initrd assets, cleanup from old states, etc. + +In a former life I authored the `clr-boot-manager` tool to manage this in an agnostic fashion and this served well for a number of years. However, existing as an externally invoked binary meant that the tool had to become responsible for all accountancy and asset lifetimes. + +### Enter ... moss-boot + +Ok - technically it's `crates/boot` in the moss repo. This is a Rust crate that will be integrated into the main `moss` project to provide the aforementioned boot management facilities. A huge advantage for us is that moss can only allow a single version of a package to exist within a given state, and all states are numerically identified due to the transaction nature of our package manager. + +In simple terms, one "state" == "one kernel candidate" (per type..) so all we *really* need to do for now is identify the `ESP` in use by way of the [Boot Loader Interface](https://systemd.io/BOOT_LOADER_INTERFACE/) and any `XBOOTLDR` partition. Then, map our "boot environment" into something `moss-boot` can convert into boot entries, and voila.. + +At a more technical level, moss-boot will handle the following: + + - form candidate `cmdline` from filesystem snippets + - auto-generate the `root=` parameter by probing the `/` partition (LVM2, etc) + - record the `initrd` needed + - compare and atomically write "missing" files in FAT-respecting fashion (copy/unlink/rename) + - garbage collect unused boot entries + +### Live atomic updates and kernels... + +An interesting, perhaps not widely considered result of upgrading a kernel, is filesystem path changes. If for any reason the kernel's module tree becomes inaccessible (say, due to updating to a new version) then it becomes impossible to load the old kernel modules. To remedy this, distributions will tend to leave multiple versions of the kernel on disk, invariably within mounted `/boot`. + +The approach Serpent OS is taking will mean that like Solus, `/boot` will not need to be premounted nor will any package ship files in that directory. Unlike the existing design in Solus 4, we will not leave kernels behind on upgrade. Instead, they become locked to the system state (transaction) in which they are used. + +To solve the "missing version" issue, we will have `moss` record the qualified snapshot path (in `/.moss`) within the `/run` tree so that once the `/usr` tree has been swapped with a new transaction atomically, a patched `kmod` package will fall back to probing the suddenly orphaned kernel tree in the cached `/run` location. + +This will mean that the `/lib/modules` tree may not have the current kernel version, but the OS will still be usable while having had a live atomic update. Of course, to *use* the new kernel you must reboot. Unlike other atomic OS implementations, it will be up to *you* when you do so: no more deferred updates! + +## Next time + +We are _aiming_ to get back to monthly blogs on the 19th, but the cycle for the coming 6 weeks may be quite interesting. Expect out of schedule posts looking for testers and volunteers while we use Serpent OS to build Serpent OS, enabling a community of pioneers benefiting from the unique design of our architecture: a full-featured OS featuring hermetic `/usr`, offline rollbacks, and live atomic upgrades with containerised system triggers. diff --git a/src/content/docs/blog/2024/04/30/calm-before-the-storm/index.mdx b/src/content/docs/blog/2024/04/30/calm-before-the-storm/index.mdx deleted file mode 100644 index 45f5a24d..00000000 --- a/src/content/docs/blog/2024/04/30/calm-before-the-storm/index.mdx +++ /dev/null @@ -1,92 +0,0 @@ ---- -title: "Calm Before The Storm" -date: 2024-04-30T22:46:17+01:00 -draft: false -authors: [ikey] -tags: [news] -license: "CC-BY-ND-4.0" -copyright: "Copyright © 2024 aerynOS Developers" ---- - -It's that time of month again, and we have some details to share with you on boot management, as well as plans for real installs -landing in May. Additionally, plans for "deferred updates" via mandatory reboots have been dropped! - -![wip boot code](Featured.webp) - - - -## Project priorities - -To better understand the technical deep-dive on boot management further below, we should clarify a slight change in priorities -for the project as a whole. For some time now we've served as a technical beacon for Solus to guide them towards better strategy -and technical solutions whilst awaiting a situation where a rebase is feasible. - -While we still look forwards to that future, the present day is the most important to us. Serpent OS must now transition from -a collection of tools and promises into a daily driver with actual users. We still aim to cater to the more developer/technical oriented -end of the spectrum rather than a "desktop distro", and want to start this journey yesterday. - -The boot management code is currently landing over the coming days, and will be immediately followed by an incredibly rudimentary -system installer to facilitate an initial adoption of Serpent OS for the general public. Our intent is to ship some early test ISOs -for supporters, and by the end of May have a feasible option for people to use and contribute to Serpent OS on real hardware. - -We acknowledge this is an exchange of the pursuit of perfection for a pragmatic rapid iteration cycle and believe the best way -forward is to open the doors wide, fixing issues on the job. The initial system will likely be rough and buggy, but it is an official -starting point from which we can all build Serpent OS together. - -## Moss + Boulder - -Our Rust tooling has now been in "production" use in our development environment and build servers for the last month -without any major hiccup. We have recently discovered some minor issues in our `emul32` (32-bit) package generation which -are being resolved, chiefly our move from the improperly named ISA `x86`, now updated to `386` affecting 32-bit dependency -chains. - -## Boot - -Our main topic of conversation this month is the enabling of _boot management_ for Serpent OS installations. To be crystal -clear, we are not talking about a boot loader. In fact, Serpent OS uses `systemd-boot` for UEFI. Instead, we are talking -about the automated management of boot loader *entries*, their corresponding kernel/initrd assets, cleanup from old states, etc. - -In a former life I authored the `clr-boot-manager` tool to manage this in an agnostic fashion and this served well for a number -of years. However, existing as an externally invoked binary meant that the tool had to become responsible for all accountancy and -asset lifetimes. - -### Enter ... moss-boot - -Ok - technically it's `crates/boot` in the moss repo. This is a Rust crate that will be integrated into the main `moss` project to -provide the aforementioned boot management facilities. A huge advantage for us is that moss can only allow a single version of a package -to exist within a given state, and all states are numerically identified due to the transaction nature of our package manager. - -In simple terms, one "state" == "one kernel candidate" (per type..) so all we *really* need to do for now is identify the `ESP` in use by -way of the [Boot Loader Interface](https://systemd.io/BOOT_LOADER_INTERFACE/) and any `XBOOTLDR` partition. Then, map our "boot environment" -into something `moss-boot` can convert into boot entries, and voila.. - -At a more technical level, moss-boot will handle the following: - - - form candidate `cmdline` from filesystem snippets - - auto-generate the `root=` parameter by probing the `/` partition (LVM2, etc) - - record the `initrd` needed - - compare and atomically write "missing" files in FAT-respecting fashion (copy/unlink/rename) - - garbage collect unused boot entries - -### Live atomic updates and kernels... - -An interesting, perhaps not widely considered result of upgrading a kernel, is filesystem path changes. If for any reason the kernel's module -tree becomes inaccessible (say, due to updating to a new version) then it becomes impossible to load the old kernel modules. To remedy this, distributions will -tend to leave multiple versions of the kernel on disk, invariably within mounted `/boot`. - -The approach Serpent OS is taking will mean that like Solus, `/boot` will not need to be premounted nor will any package ship files in that directory. -Unlike the existing design in Solus 4, we will not leave kernels behind on upgrade. Instead, they become locked to the system state (transaction) in which -they are used. - -To solve the "missing version" issue, we will have `moss` record the qualified snapshot path (in `/.moss`) within the `/run` tree so that once the `/usr` -tree has been swapped with a new transaction atomically, a patched `kmod` package will fall back to probing the suddenly orphaned kernel tree in the cached -`/run` location. - -This will mean that the `/lib/modules` tree may not have the current kernel version, but the OS will still be usable while having had a live atomic update. -Of course, to *use* the new kernel you must reboot. Unlike other atomic OS implementations, it will be up to *you* when you do so: no more deferred updates! - -## Next time - -We are _aiming_ to get back to monthly blogs on the 19th, but the cycle for the coming 6 weeks may be quite interesting. Expect out of schedule posts -looking for testers and volunteers while we use Serpent OS to build Serpent OS, enabling a community of pioneers benefiting from the unique design of -our architecture: a full-featured OS featuring hermetic `/usr`, offline rollbacks, and live atomic upgrades with containerised system triggers. diff --git a/src/content/docs/blog/2024/08/01/prealpha0-released/featured-background.png b/src/content/docs/blog/2024/08/01/prealpha0-released/featured-background.png deleted file mode 100644 index 01497c58..00000000 Binary files a/src/content/docs/blog/2024/08/01/prealpha0-released/featured-background.png and /dev/null differ diff --git a/src/content/docs/blog/2024/08/01/prealpha0-released/index.md b/src/content/docs/blog/2024/08/01/prealpha0-released/index.md new file mode 100644 index 00000000..304d1a25 --- /dev/null +++ b/src/content/docs/blog/2024/08/01/prealpha0-released/index.md @@ -0,0 +1,69 @@ +--- +title: "Serpent OS Prealpha0 Released" +date: 2024-08-01T01:18:07+00:00 +draft: false +authors: [ikey] +tags: [news] +license: "CC-BY-ND-4.0" +copyright: "Copyright © 2024 aerynOS Developers" +--- + +Well, it didn't take us that long, **really** ... Our technical preview, *prealpha0*, is now available for testing! + +![A live GNOME environment running on prealpha0 of SerpentOS](prealpha0.webp "A live GNOME environment running on prealpha0 of SerpentOS") + +Head to our [download](/download) page to grab `serpent-os-prealpha-0.iso` now! + +This is a super rough version of Serpent OS that is *capable* of being installed on baremetal hardware and VMs that support UEFI and OpenGL acceleration. It is however not *recommended* for use daily due to the early nature and a bunch of packages being wildly out of date. It features a minimal GNOME desktop with the Firefox web browser and a terminal. + +Right now it contains a CLI installer that can be accessed from the terminal by typing: + +```bash +sudo lichen +``` + +To detract from casual use it is necessary to manually partition the disk before running the (net) installer. You can use `fdisk` to create a GPT disk with a (mandatory) `EFI System Partition` and an optional `XBOOTLDR` partition. This currently also needs to be FAT32 until we integrate systemd-boot driver support. Lastly of course you will need a large enough root partition. + +You will need a working network connection, so make sure you're connected before starting the installer! + +## Technologies involved + +This will of course appear to be a very rough (crap) prealpha ISO. Underneath the surface it is using the [moss](https://github.com/serpent-os/moss) package manager, our very own package management solution written in Rust. Quite simply, every single transaction in moss generates a new filesystem tree (`/usr`) in a staging area as a full, stateless, OS transaction. + +### moss + +When the package installs succeed, any *transaction triggers* are run in a private namespace (container) before finally activating the new `/usr` tree. Through our enforced stateless design, `usr-merge`, etc, we can atomically update the running OS with a single `renameat2` call. + +As a neat aside, all OS content is deduplicated, meaning your last system transaction is still available on disk allowing offline rollbacks. + +Translated: Like A/B swaps? Don't like rebooting? ... Ok that explained it. + +### blsforme + +A few crates deep we find [blsforme](https://github.com/serpent-os/blsforme) - a library for automatically managing the ESP and XBOOTLDR partitions. Whenever kernels and associated files are present in the OS filesystem, they are synchronised to the boot partitions along with automatically generated boot entries and cmdlines. Specifically this means moss can discover and mount necessary partitions according to the disk topology, GPT entries, and Boot Loader Specification, generating configurations by scanning your local rootfs to build the proper `root=` parameters for you. + +Translated: Magic boot management makes way for offline rollbacks. + +## Warning + +Super **pre**-alpha. Will 100% break! We just wanted you for the journey <3 + +## Start of public iterations + +Ok you have this super rough ISO, what next? We now have an actual *startpoint* and will continue to iterate on the ISOs, delivering new installer updates/improvements and removing the unnecessary live mode from the ISO completely. + +The next release will also feature more installer options so you can fresh-install the Cosmic Desktop (in repos now) For the brave, go forth and `sudo moss sync -u` ! (ok you want `moss help` first.) + +## Closing word + +As a distro, it's kinda crap right now. The tooling has been our focus for years and now we can actually build something with it. With only a handful of packages, `flatpak` is your best friend. Or you could swing a PR into our [recipes](https://github.com/serpent-os/recipes) repo! + +The project is only possible with your support. Something tells me that putting out this ISO is going to somewhat increase our hosting costs and stress the datacenter hamsters. + +Feel free to sponsor the project to support our work and increase our capacity! + + diff --git a/src/content/docs/blog/2024/08/01/prealpha0-released/index.mdx b/src/content/docs/blog/2024/08/01/prealpha0-released/index.mdx deleted file mode 100644 index c3004c62..00000000 --- a/src/content/docs/blog/2024/08/01/prealpha0-released/index.mdx +++ /dev/null @@ -1,83 +0,0 @@ ---- -title: "Serpent OS Prealpha0 Released" -date: 2024-08-01T01:18:07+00:00 -draft: false -authors: [ikey] -tags: [news] -license: "CC-BY-ND-4.0" -copyright: "Copyright © 2024 aerynOS Developers" ---- - -Well, it didn't take us that long, **really** ... Our technical preview, *prealpha0*, is now available for testing! - -![wip boot code](prealpha0.png) - -Head to our [download](/download) page to grab `serpent-os-prealpha-0.iso` now! - - - -This is a super rough version of Serpent OS that is *capable* of being installed on baremetal hardware and VMs that -support UEFI and OpenGL acceleration. It is however not *recommended* for use daily due to the early nature and a bunch -of packages being wildly out of date. It features a minimal GNOME desktop with the Firefox web browser and a terminal. - -Right now it contains a CLI installer that can be accessed from the terminal by typing: - -```bash -sudo lichen -``` - -To detract from casual use it is necessary to manually partition the disk before running the (net) installer. You can use -`fdisk` to create a GPT disk with a (mandatory) `EFI System Partition` and an optional `XBOOTLDR` partition. This currently -also needs to be FAT32 until we integrate systemd-boot driver support. Lastly of course you will need a large enough root -partition. - -You will need a working network connection, so make sure you're connected before starting the installer! - - -## Technologies involved - -This will of course appear to be a very rough (crap) prealpha ISO. Underneath the surface it is using the [moss](https://github.com/serpent-os/moss) -package manager, our very own package management solution written in Rust. Quite simply, every single transaction in moss generates -a new filesystem tree (`/usr`) in a staging area as a full, stateless, OS transaction. - -### moss - -When the package installs succeed, any *transaction triggers* are run in a private namespace (container) before finally activating -the new `/usr` tree. Through our enforced stateless design, `usr-merge`, etc, we can atomically update the running OS with a single `renameat2` -call. - -As a neat aside, all OS content is deduplicated, meaning your last system transaction is still available on disk allowing offline rollbacks. - -Translated: Like A/B swaps? Don't like rebooting? ... Ok that explained it. - -### blsforme - -A few crates deep we find [blsforme](https://github.com/serpent-os/blsforme) - a library for automatically managing the ESP and XBOOTLDR -partitions. Whenever kernels and associated files are present in the OS filesystem, they are synchronised to the boot partitions along -with automatically generated boot entries and cmdlines. Specifically this means moss can discover and mount necessary partitions according to -the disk topology, GPT entries, and Boot Loader Specification, generating configurations by scanning your local rootfs to build the proper -`root=` parameters for you. - -Translated: Magic boot management makes way for offline rollbacks. - -## Warning - -Super **pre**-alpha. Will 100% break! We just wanted you for the journey <3 - -## Start of public iterations - -Ok you have this super rough ISO, what next? We now have an actual *startpoint* and will continue to iterate on the ISOs, -delivering new installer updates/improvements and removing the unnecessary live mode from the ISO completely. - -The next release will also feature more installer options so you can fresh-install the Cosmic Desktop (in repos now) -For the brave, go forth and `sudo moss sync -u` ! (ok you want `moss help` first.) - -## Closing word - -As a distro, it's kinda crap right now. The tooling has been our focus for years and now we can actually build something -with it. With only a handful of packages, `flatpak` is your best friend. Or you could swing a PR into our [recipes](https://github.com/serpent-os/recipes) repo! - -The project is only possible with your support. Something tells me that putting out this ISO is going to somewhat increase -our hosting costs and stress the datacenter hamsters. - -Feel free to [sponsor](https://github.com/sponsors/ikeycode) to support our work and increase our capacity! diff --git a/src/content/docs/blog/2024/08/01/prealpha0-released/prealpha0.png b/src/content/docs/blog/2024/08/01/prealpha0-released/prealpha0.png deleted file mode 100644 index 01497c58..00000000 Binary files a/src/content/docs/blog/2024/08/01/prealpha0-released/prealpha0.png and /dev/null differ diff --git a/src/content/docs/blog/2024/08/01/prealpha0-released/prealpha0.webp b/src/content/docs/blog/2024/08/01/prealpha0-released/prealpha0.webp new file mode 100644 index 00000000..8ccde814 Binary files /dev/null and b/src/content/docs/blog/2024/08/01/prealpha0-released/prealpha0.webp differ diff --git a/src/content/docs/blog/2024/12/23/serpent-os-enters-alpha/featured.png b/src/content/docs/blog/2024/12/23/serpent-os-enters-alpha/featured.png deleted file mode 100644 index db2f03e9..00000000 Binary files a/src/content/docs/blog/2024/12/23/serpent-os-enters-alpha/featured.png and /dev/null differ diff --git a/src/content/docs/blog/2024/12/23/serpent-os-enters-alpha/featured.webp b/src/content/docs/blog/2024/12/23/serpent-os-enters-alpha/featured.webp new file mode 100644 index 00000000..c7c9649d Binary files /dev/null and b/src/content/docs/blog/2024/12/23/serpent-os-enters-alpha/featured.webp differ diff --git a/src/content/docs/blog/2024/12/23/serpent-os-enters-alpha/index.mdx b/src/content/docs/blog/2024/12/23/serpent-os-enters-alpha/index.md similarity index 75% rename from src/content/docs/blog/2024/12/23/serpent-os-enters-alpha/index.mdx rename to src/content/docs/blog/2024/12/23/serpent-os-enters-alpha/index.md index e78b4b11..883e10ce 100644 --- a/src/content/docs/blog/2024/12/23/serpent-os-enters-alpha/index.mdx +++ b/src/content/docs/blog/2024/12/23/serpent-os-enters-alpha/index.md @@ -7,12 +7,9 @@ license: "CC-BY-ND-4.0" copyright: "Copyright © 2024 aerynOS Developers" --- -Following a successful prealpha phase, we're excited to announce that Serpent OS has now entered the alpha stage of development. This milestone marks a significant step forward in the project's journey. -While the usual disclaimers apply, as a certain level of project *fluidity is to be expected*, we invite you to explore the latest alpha release and experience the cutting-edge features and improvements we've introduced. +Following a successful prealpha phase, we're excited to announce that Serpent OS has now entered the alpha stage of development. This milestone marks a significant step forward in the project's journey. While the usual disclaimers apply, as a certain level of project *fluidity is to be expected*, we invite you to explore the latest alpha release and experience the cutting-edge features and improvements we've introduced. - - -![Serpent OS](featured.png) +![Screenshot of a Gnome desktop showing files open](featured.webp "A Serpent OS install showing a GNOME environment") ## Supporting Serpent OS @@ -20,14 +17,15 @@ Serpent OS is a community-driven project, and we welcome contributions from deve Lastly, you can support the project by becoming a sponsor. Your support helps cover our hosting costs and allows us to dedicate more time to making Serpent OS exceptional. We're grateful for the support we've received so far and look forward to continuing to build something meaningful in the Linux ecosystem. - - Sponsor Serpent OS - + ## What is Serpent OS? -Serpent OS is a heavily engineering-led Linux distribution seeking to redefine *how* we distribute Linux. -It is a stateless OS that leverages atomic updates, cutting-edge tooling, and rock-solid reliability to deliver a safe and efficient system. Built by industry veterans with decades of experience, Serpent OS represents the next evolution in Linux distributions. +Serpent OS is a heavily engineering-led Linux distribution seeking to redefine *how* we distribute Linux. It is a stateless OS that leverages atomic updates, cutting-edge tooling, and rock-solid reliability to deliver a safe and efficient system. Built by industry veterans with decades of experience, Serpent OS represents the next evolution in Linux distributions. We currently offer x86_64-v2 desktop builds, for both GNOME and COSMIC desktop environments. We only support UEFI systems, and plan secure boot support via `shim` in the near future. The majority of the distribution (including the kernel) is built against the LLVM toolchain, with libc++ as the default C++ standard library. @@ -39,9 +37,7 @@ The alpha release introduces several key features and improvements that enhance ### Hardware support -We've rounded out our hardware support, now including patches for ASUS devices and Surface devices. Additionally, we now have the NVIDIA driver available in the repositories. -Note we only offer [open-gpu-kernel-modules](https://github.com/NVIDIA/open-gpu-kernel-modules), which we prebuild against our `linux-desktop` package to streamline -installation. +We've rounded out our hardware support, now including patches for ASUS devices and Surface devices. Additionally, we now have the NVIDIA driver available in the repositories. Note we only offer [open-gpu-kernel-modules](https://github.com/NVIDIA/open-gpu-kernel-modules), which we prebuild against our `linux-desktop` package to streamline installation. ### Steam @@ -50,7 +46,7 @@ You can now install Steam from our repos, making use of our newly enabled multil - mesa-32bit - nvidia-graphics-driver-32bit -![Steam on Serpent OS](steam.png) +![Steam on Serpent OS](steam.webp "Steam running on Serpent OS") ### More Rust @@ -79,8 +75,7 @@ We've made significant improvements to our tooling, including updates to `moss`, ### moss - Package Management -Our atomic package manager swaps the entire `/usr` directory during system updates, ensuring a stateless, bulletproof upgrade process. Updates either succeed completely or not at all. -Recently we made improvements to `moss` to handle certain special cases: +Our atomic package manager swaps the entire `/usr` directory during system updates, ensuring a stateless, bulletproof upgrade process. Updates either succeed completely or not at all. Recently we made improvements to `moss` to handle certain special cases: - `EMLINK` - `ext4` is limited to 65k hardlinks per inode, and we were hitting this limit on some systems due to empty files. We now handle this case gracefully by creating new inodes for empty files. - `ENOSPC` - We now mitigate the risk of `ENOSPC` on the boot partition by performing automatic cleanup of old kernels and initrds. This paves the way for us to land transaction-specific entries for the offline rollback feature. @@ -89,32 +84,27 @@ Recently we made improvements to `moss` to handle certain special cases: Automated boot management that just works. Seamlessly handles your EFI System Partition and boot entries without manual intervention. -We've improved blsforme to support drop-in command line snippets for kernel parameters, allowing for easy customisation of boot entries. -These can be supplied by packages, or by the user directly. This has allowed us to make `plymouth` work out of the box, in tandem with our -prebuilt reproducible initrd images. +We've improved blsforme to support drop-in command line snippets for kernel parameters, allowing for easy customisation of boot entries. These can be supplied by packages, or by the user directly. This has allowed us to make `plymouth` work out of the box, in tandem with our prebuilt reproducible initrd images. We also now support multiple initrd images, which in turn allowed us to land support for early KMS in the initrd for NVIDIA users. ### lichen - Installer -We've revamped the UI for Lichen, and now allow picking between xfs, ext4, and f2fs for the root filesystem. The team advises against ext4 for systems where more than a few hundred -transactions will need to be retained, due to ext4 limitations with hardlink counts. +We've revamped the UI for Lichen, and now allow picking between xfs, ext4, and f2fs for the root filesystem. The team advises against ext4 for systems where more than a few hundred transactions will need to be retained, due to ext4 limitations with hardlink counts. ### Sync protections -In preparation for "versioned repositories", we've added a gate to prevent the automatic builds on our infrastructure from directly reaching users. Once we're happy with our validation, we -then "sync" the repositories to the public mirrors. This ensures that users are not exposed to any potential issues with the latest builds. +In preparation for "versioned repositories", we've added a gate to prevent the automatic builds on our infrastructure from directly reaching users. Once we're happy with our validation, we then "sync" the repositories to the public mirrors. This ensures that users are not exposed to any potential issues with the latest builds. ## What's next? -With us now entering the alpha phase, we're looking to grow our contributor base and community. Our immediate focus is improving/creating documentation to enable more people to use Serpent OS -as a daily driver in the near future. We're also going to keep iterating on our tooling, ensuring a reliable and efficient system for all users. +With us now entering the alpha phase, we're looking to grow our contributor base and community. Our immediate focus is improving/creating documentation to enable more people to use Serpent OS as a daily driver in the near future. We're also going to keep iterating on our tooling, ensuring a reliable and efficient system for all users. We'll begin deeper integration of the tooling, such as `moss` with `packagekitd` for GNOME Software and COSMIC Store use. ## Issues and feedback -If you encounter any issues or have feedback to share, please don't hesitate to reach out to us. You can report bugs on our [GitHub issue tracker](https://github.com/serpent-os/recipes/issues) or join our [Matrix space](https://matrix.to/#/!trFJOzhpDUejJKnPYg:matrix.org) to chat with other users and developers. +If you encounter any issues or have feedback to share, please don't hesitate to reach out to us. You can report bugs on our [GitHub issue tracker](https://github.com/serpent-os/recipes/issues) or join our Matrix[^1] space to chat with other users and developers. We are aware of some known issues in the alpha release, and we're actively working to address them in future updates. Your feedback is invaluable to us, and we appreciate your help in making Serpent OS the best it can be. @@ -130,3 +120,6 @@ We are aware of some known issues in the alpha release, and we're actively worki ## Finally We're excited to share the alpha release with you and look forward to your feedback and contributions. Thank you for your continued support and interest in Serpent OS. We're committed to building a safe, efficient, and reliable system that empowers users to get the most out of their computing experience. + +--- +[^1]: AerynOS now uses a [Zulip server](https://aerynos.zulipchat.com/) for community discussion. diff --git a/src/content/docs/blog/2024/12/23/serpent-os-enters-alpha/steam.png b/src/content/docs/blog/2024/12/23/serpent-os-enters-alpha/steam.png deleted file mode 100644 index 64268e0c..00000000 Binary files a/src/content/docs/blog/2024/12/23/serpent-os-enters-alpha/steam.png and /dev/null differ diff --git a/src/content/docs/blog/2024/12/23/serpent-os-enters-alpha/steam.webp b/src/content/docs/blog/2024/12/23/serpent-os-enters-alpha/steam.webp new file mode 100644 index 00000000..545e5322 Binary files /dev/null and b/src/content/docs/blog/2024/12/23/serpent-os-enters-alpha/steam.webp differ diff --git a/src/content/docs/blog/2024/12/29/alpha-refresh-available/featured-background.png b/src/content/docs/blog/2024/12/29/alpha-refresh-available/featured-background.png deleted file mode 100644 index b1388512..00000000 Binary files a/src/content/docs/blog/2024/12/29/alpha-refresh-available/featured-background.png and /dev/null differ diff --git a/src/content/docs/blog/2024/12/29/alpha-refresh-available/featured-background.webp b/src/content/docs/blog/2024/12/29/alpha-refresh-available/featured-background.webp new file mode 100644 index 00000000..eb1d0dc8 Binary files /dev/null and b/src/content/docs/blog/2024/12/29/alpha-refresh-available/featured-background.webp differ diff --git a/src/content/docs/blog/2024/12/29/alpha-refresh-available/index.md b/src/content/docs/blog/2024/12/29/alpha-refresh-available/index.md new file mode 100644 index 00000000..183d1fac --- /dev/null +++ b/src/content/docs/blog/2024/12/29/alpha-refresh-available/index.md @@ -0,0 +1,81 @@ +--- +title: Alpha Refresh Available +date: 2024-12-29T16:45:48Z +authors: [ikey] +tags: [news] +license: "CC-BY-ND-4.0" +copyright: "Copyright © 2024 aerynOS Developers" +--- + +In response to feedback from the community, we're issuing a refresh of the Alpha 1 release, `0.24.6`. This refresh includes a number of key updates and improvements, including: + + - Now works with etcher + - Fractional scaling enabled by default + - Prebuild icon theme caches to speed up trigger execution (except for `hicolor`) + - Add new `moss boot status` command for introspection into [blsforme](https://github.com/serpent-os/blsforme) functionality + - Fix amdgpu initialisation for older devices, previously leading to black screen on boot + +![Fractional scaling, much loved](./featured-background.webp "Fractional scalling now enabled by default") + +## Clarification on moss boot management + +Frequently in Team Serpent we joke that `moss` is the "systemd of package managers". It actually does a whole lot more than is easily visible.. for example: + + - Content addressable storage for the entire OS + - Atomic updates of `/usr` by `renameat2()` w/ `RENAME_EXCHANGE` + - Executes **transactional triggers** in a namespace (container) with RW `/usr` and RO binds of `/etc` and `/var` ... + - Executes **system triggers** postblit, with dependency ordering.. + - Via blsforme, manages the ESP, XBOOTLDR, boot entries, sync and promotion of bootloader, kernel and assets, initrds.. + - Entirely zero configuration. No `/etc/default/grub`, no remembering your `root={}`, it's determined automatically. + - Basically does all the work of OS provisioning, barring disk mounting and partitioning. Our installer, lichen, is basically a thin wrapper of moss. + +We've had no debugging capabilities around the boot management, so we've landed `moss boot status` command. It's super primitive but does offer an insight into the introspection capabilities of blsforme. In this simple screenshot, moss/blsforme are supporting the Boot Loader Protocol, querying the `systemd-boot` entries in efivars. When found, `LoaderDevicePartUUID` is used to determine the boot ESP, but moss/blsforme will quite happily read the GPT directly (relative to `/` mount) and find the ESP and XBOOTLDR... + +Remember - we happily use `systemd-boot`. We do not use `bootctl` or `kernel-install`. Entries are automatically managed and we're extending this to support offline rollbacks via boot menu (feel free to extend with `/etc/kernel/cmdline.d/*.cmdline` drop-in snippets!). Also note we do not allow systemd to automount the ESP or XBOOTLDR via the GPT autogenerator, instead moss/blsforme will automatically discover and if necessary, mount these partitions, when dealing with bootloader or kernel updates. + +With this explicit control over the boot management, we'll happily be landing secure boot and fwupd support in the near future, without the downsides of trying to provide EFI assets to `/boot` in a package itself. 😬 + +![moss boot status](./moss_boot_status.webp "Output of moss boot status command") + +## A note on virtual machine usage + +It must be clearly stated that Serpent OS does not support anything other than UEFI booting on x86_64-v2+ capable hardware. Please note that GNOME Boxes will default to BIOS, not UEFI, and will need hand-mangling of the XML file. Also note in other solutions such as VirtualBox you will need to specifically select `EFI` in the VM settings. + +In order to maximise compatibility with other tooling and firmware, we've modified the generation of our ISO to include the isolinux MBR/GPT hybrid components, **however** we deliberately did not include a functioning bootloader for BIOS boot. Instead, you will see a boot failure indicating that `ldlinux.c32` could not be found, which is a sure-fire way to know your VM or hardware is incorrectly configured. + +![Invalid VM configuration](./le_no_uefi.webp "Screenshot showing what happens when trying to boot a VM without UEFI only mode enabled") + +## Installer notes + +Lichen is due to receive some love in the next few cycles, so please be aware of the following limitations: + + - A **pre-configured** GPT disk is required before `lichen` is launched + - A working internet connection is also required, this is a **network installer** + - A FAT32 formatted EFI System Partition is required (marked as `esp` in gparted) + - For optimal usage, or when your ESP is tiny, please create an `XBOOTLDR` partition: + - Create an FAT32 partition in gparted of 2GB in size + - Mark it as `bls_boot` in gparted + - Further filesystem support for XBOOTLDR will be added to blsforme when we've finished packaging `efifs`. + - XFS or F2FS are strongly recommended for the rootfs, due to the limitations of ext4 with hardlink counts. + +## COSMIC Desktop + +We've made the decision to pause our offering of COSMIC Desktop ISOs. We're huge fans of the project, and we strive to keep it up to date and functional. However, it is very early in alpha, and so are we. It doesn't make a lot of sense for us to extend our workload to two alpha codebases! + +We still offer COSMIC as an option in the installer for those who wish to try it out, and do encourage testing and engaging upstream with the developers. For now, we will offer just the GNOME ISO. + +![Lichen + COSMIC](./lichen_dev.webp "Option to install COSMIC Desktop directly from our lichen installer") + +## Next up + +In the new year we'll be focusing heavily on our tooling to facilitate a scale up of our packaging efforts. We're quite proud of our tooling, but we're not oblivious to the warts and issues. The next couple of weeks will be spent improving our documentation (or making it exist!) and making sure that we have the ideal workflow in place to permit large stack updates along with the ABI consistency verification we need. This will also pool work for `ent`, `moss`, `boulder`, by streamlining packaging, and making it trivial to update an entire stack of packages in one go. Eventually this will be an automatic thing, with successful PRs being verified and then merged into the volatile repository. + +Don't forget, versioned repositories are also coming your way, and very soon we're landing offline rollbacks in the boot menu! `blsforme` has been significantly restructured for `cmdline` handling, and understands entries in the context of a distinct system root (ie moss filesystem transaction) - which is the bulk of the work needed to facilitate offline rollbacks. The last piece will be running `moss` at an early stage in the `initrd` to swap the `/usr` pointer to the older transaction, which is instantaneous and atomic. + +## Final thoughts + +5 years and no release, and now two alpha ISOs before the end of the year. We're pretty happy with that, and we're going to keep surprising you next year with more releases, more features, and more stability. Happy new year from all of us at Team Serpent! 🐍 + +If you like what we're doing, please consider [sponsoring us](/sponsor) - this will allow us to grow our infrastructure and deliver builds quicker. Due to health issues in the last few months, I was forced into a position whereby I had to resign from my job. Any and all contributions to the project are greatly appreciated, and will help us to continue delivering what folks are frequently now calling "the cleanest Linux implementation they've ever seen". + +Try it. You might like it. 🚀 diff --git a/src/content/docs/blog/2024/12/29/alpha-refresh-available/index.mdx b/src/content/docs/blog/2024/12/29/alpha-refresh-available/index.mdx deleted file mode 100644 index 8dae99c8..00000000 --- a/src/content/docs/blog/2024/12/29/alpha-refresh-available/index.mdx +++ /dev/null @@ -1,114 +0,0 @@ ---- -title: Alpha Refresh Available -date: 2024-12-29T16:45:48Z -authors: [ikey] -tags: [news] -license: "CC-BY-ND-4.0" -copyright: "Copyright © 2024 aerynOS Developers" ---- - -In response to feedback from the community, we're issuing a refresh of the Alpha 1 release, `0.24.6`. This refresh includes a number of key updates and improvements, including: - - - Now works with etcher - - Fractional scaling enabled by default - - Prebuild icon theme caches to speed up trigger execution (except for `hicolor`) - - Add new `moss boot status` command for introspection into [blsforme](https://github.com/serpent-os/blsforme) functionality - - Fix amdgpu initialisation for older devices, previously leading to black screen on boot - - - -![Fractional scaling, much loved](featured-background.png) - -## Clarification on moss boot management - -Frequently in Team Serpent we joke that `moss` is the "systemd of package managers". It actually does a whole lot more than is easily visible.. for example: - - - Content addressable storage for the entire OS - - Atomic updates of `/usr` by `renameat2()` w/ `RENAME_EXCHANGE` - - Executes **transactional triggers** in a namespace (container) with RW `/usr` and RO binds of `/etc` and `/var` ... - - Executes **system triggers** postblit, with dependency ordering.. - - Via blsforme, manages the ESP, XBOOTLDR, boot entries, sync and promotion of bootloader, kernel and assets, initrds.. - - Entirely zero configuration. No `/etc/default/grub`, no remembering your `root={}`, it's determined automatically. - - Basically does all the work of OS provisioning, barring disk mounting and partitioning. Our installer, lichen, is basically a thin wrapper of moss. - -We've had no debugging capabilities around the boot management, so we've landed `moss boot status` command. It's super primitive -but does offer an insight into the introspection capabilities of blsforme. In this simple screenshot, moss/blsforme are supporting the -Boot Loader Protocol, querying the `systemd-boot` entries in efivars. When found, `LoaderDevicePartUUID` is used to determine the -boot ESP, but moss/blsforme will quite happily read the GPT directly (relative to `/` mount) and find the ESP and XBOOTLDR... - -Remember - we happily use `systemd-boot`. We do not use `bootctl` or `kernel-install`. Entries are automatically managed and -we're extending this to support offline rollbacks via boot menu (feel free to extend with `/etc/kernel/cmdline.d/*.cmdline` drop-in snippets!) -Also note we do not allow systemd to automount the ESP or XBOOTLDR via the GPT autogenerator, instead moss/blsforme will automatically -discover and if necessary, mount these partitions, when dealing with bootloader or kernel updates. - -With this explicit control over the boot management, we'll happily be landing secure boot and fwupd support in the near future, -without the downsides of trying to provide EFI assets to `/boot` in a package itself. 😬 - -![moss boot status](moss_boot_status.png) - -## A note on virtual machine usage - -It must be clearly stated that Serpent OS does not support anything other than UEFI booting on x86_64-v2+ capable hardware. -Please note that GNOME Boxes will default to BIOS, not UEFI, and will need hand-mangling of the XML file. Also note in other -solutions such as VirtualBox you will need to specifically select `EFI` in the VM settings. - -In order to maximise compatibility with other tooling and firmware, we've modified the generation of our ISO to include -the isolinux MBR/GPT hybrid components, **however** we deliberately did not include a functioning bootloader for BIOS boot. -Instead, you will see a boot failure indicating that `ldlinux.c32` could not be found, which is a sure-fire way to know your -VM or hardware is incorrectly configured. - -![Invalid VM configuration](le_no_uefi.png) - -## Installer notes - -Lichen is due to receive some love in the next few cycles, so please be aware of the following limitations: - - - A **pre-configured** GPT disk is required before `lichen` is launched - - A working internet connection is also required, this is a **network installer** - - A FAT32 formatted EFI System Partition is required (marked as `esp` in gparted) - - For optimal usage, or when your ESP is tiny, please create an `XBOOTLDR` partition: - - Create an FAT32 partition in gparted of 2GB in size - - Mark it as `bls_boot` in gparted - - Further filesystem support for XBOOTLDR will be added to blsforme when we've finished packaging `efifs`. - - XFS or F2FS are strongly recommended for the rootfs, due to the limitations of ext4 with hardlink counts. - -## COSMIC Desktop - -We've made the decision to pause our offering of COSMIC Desktop ISOs. We're huge fans of the project, and we -strive to keep it up to date and functional. However, it is very early in alpha, and so are we. It doesn't make -a lot of sense for us to extend our workload to two alpha codebases! - -We still offer COSMIC as an option in the installer for those who wish to try it out, and do encourage testing -and engaging upstream with the developers. For now, we will offer just the GNOME ISO. - -![Lichen + COSMIC](lichen_dev.png) - -## Next up - -In the new year we'll be focusing heavily on our tooling to facilitate a scale up of our packaging efforts. -We're quite proud of our tooling, but we're not oblivious to the warts and issues. The next couple of weeks -will be spent improving our documentation (or making it exist!) and making sure that we have the ideal workflow -in place to permit large stack updates along with the ABI consistency verification we need. This will also -pool work for `ent`, `moss`, `boulder`, by streamlining packaging, and making it trivial to update an entire -stack of packages in one go. Eventually this will be an automatic thing, with successful PRs being verified -and then merged into the volatile repository. - -Don't forget, versioned repositories are also coming your way, and very soon we're landing offline rollbacks in -the boot menu! `blsforme` has been significantly restructured for `cmdline` handling, and understands entries in -the context of a distinct system root (ie moss filesystem transaction) - which is the bulk of the work needed -to facilitate offline rollbacks. The last piece will be running `moss` at an early stage in the `initrd` to -swap the `/usr` pointer to the older transaction, which is instantaneous and atomic. - - -## Final thoughts - -5 years and no release, and now two alpha ISOs before the end of the year. We're pretty happy with that, and we're -going to keep surprising you next year with more releases, more features, and more stability. Happy new year from -all of us at Team Serpent! 🐍 - -If you like what we're doing, please consider [sponsoring us](/sponsor) - this will allow us to grow our infrastructure -and deliver builds quicker. Due to health issues in the last few months, I was forced into a position whereby I had to -resign from my job. Any and all contributions to the project are greatly appreciated, and will help us to continue delivering -what folks are frequently now calling "the cleanest Linux implementation they've ever seen". - -Try it. You might like it. 🚀 diff --git a/src/content/docs/blog/2024/12/29/alpha-refresh-available/le_no_uefi.png b/src/content/docs/blog/2024/12/29/alpha-refresh-available/le_no_uefi.png deleted file mode 100644 index 9d669d13..00000000 Binary files a/src/content/docs/blog/2024/12/29/alpha-refresh-available/le_no_uefi.png and /dev/null differ diff --git a/src/content/docs/blog/2024/12/29/alpha-refresh-available/le_no_uefi.webp b/src/content/docs/blog/2024/12/29/alpha-refresh-available/le_no_uefi.webp new file mode 100644 index 00000000..d4c49b52 Binary files /dev/null and b/src/content/docs/blog/2024/12/29/alpha-refresh-available/le_no_uefi.webp differ diff --git a/src/content/docs/blog/2024/12/29/alpha-refresh-available/lichen_dev.png b/src/content/docs/blog/2024/12/29/alpha-refresh-available/lichen_dev.png deleted file mode 100644 index 914799e6..00000000 Binary files a/src/content/docs/blog/2024/12/29/alpha-refresh-available/lichen_dev.png and /dev/null differ diff --git a/src/content/docs/blog/2024/12/29/alpha-refresh-available/lichen_dev.webp b/src/content/docs/blog/2024/12/29/alpha-refresh-available/lichen_dev.webp new file mode 100644 index 00000000..1277a55a Binary files /dev/null and b/src/content/docs/blog/2024/12/29/alpha-refresh-available/lichen_dev.webp differ diff --git a/src/content/docs/blog/2024/12/29/alpha-refresh-available/moss_boot_status.png b/src/content/docs/blog/2024/12/29/alpha-refresh-available/moss_boot_status.png deleted file mode 100644 index 027095eb..00000000 Binary files a/src/content/docs/blog/2024/12/29/alpha-refresh-available/moss_boot_status.png and /dev/null differ diff --git a/src/content/docs/blog/2024/12/29/alpha-refresh-available/moss_boot_status.webp b/src/content/docs/blog/2024/12/29/alpha-refresh-available/moss_boot_status.webp new file mode 100644 index 00000000..09f65916 Binary files /dev/null and b/src/content/docs/blog/2024/12/29/alpha-refresh-available/moss_boot_status.webp differ diff --git a/src/content/docs/blog/2024/12/31/investing-in-the-future/index.md b/src/content/docs/blog/2024/12/31/investing-in-the-future/index.md index a82eb660..a51ef735 100644 --- a/src/content/docs/blog/2024/12/31/investing-in-the-future/index.md +++ b/src/content/docs/blog/2024/12/31/investing-in-the-future/index.md @@ -9,82 +9,49 @@ copyright: "Copyright © 2024 aerynOS Developers" Here we are, at the end of 2024, and what a year it's been! We've made significant progress on Serpent OS, and we're excited to share some of the highlights with you. - - ## Attained Alpha -Virtually 5 years in the making, we recently attained alpha status. Our tooling and concepts have aligned, allowing us to now rapidly iterate -on the core deliverable itself: Serpent OS. We've seen an explosion in cadence, with our tooling enabling us to quickly and -easily deliver updates, new packages and enabling new features. +Virtually 5 years in the making, we recently attained alpha status. Our tooling and concepts have aligned, allowing us to now rapidly iterate on the core deliverable itself: Serpent OS. We've seen an explosion in cadence, with our tooling enabling us to quickly and easily deliver updates, new packages and enabling new features. -Since the Alpha 1 ISO (`0.24.5`), we've fixed a number of issues in relation to hardware and booting. We've also added -new features to moss (refined boot management) and boulder (our build system). Even now we have multiple PRs open to -tidy up moss, add automatic generation of `monitoring.yaml` files, and more. +Since the Alpha 1 ISO (`0.24.5`), we've fixed a number of issues in relation to hardware and booting. We've also added new features to moss (refined boot management) and boulder (our build system). Even now we have multiple PRs open to tidy up moss, add automatic generation of `monitoring.yaml` files, and more. ## Keeping the cadence -We've been working hard to keep the cadence up, and we're now at a point where we can deliver new ISOs on a regular basis. -Our main focus now is the tooling, in enabling a far more automatic delivery pipeline. The vision is having automated pull -requests generated for package updates, automatically handling changes in dependencies and ABI, and ordered by build tier. -This will significantly reduce the maintainer burden, in conjunction with planned security monitoring. +We've been working hard to keep the cadence up, and we're now at a point where we can deliver new ISOs on a regular basis. Our main focus now is the tooling, in enabling a far more automatic delivery pipeline. The vision is having automated pull requests generated for package updates, automatically handling changes in dependencies and ABI, and ordered by build tier. This will significantly reduce the maintainer burden, in conjunction with planned security monitoring. ## Short term deliverables In the short term we're going to land the following features: -- **Offline Rollbacks** - We're going to land offline rollbacks in the boot menu, allowing you to easily revert to a previous - system state. -- **Versioned Repositories** - We're going to introduce versioned repositories, allowing us to gracefully handle format changes - in moss and boulder, permitting an "update forever" epoch-staggering approach. Additionally, this will allow users to stay on an - older repo version if they're awaiting a fix to a regression in a newer version. +- **Offline Rollbacks** - We're going to land offline rollbacks in the boot menu, allowing you to easily revert to a previous system state. +- **Versioned Repositories** - We're going to introduce versioned repositories, allowing us to gracefully handle format changes in moss and boulder, permitting an "update forever" epoch-staggering approach. Additionally, this will allow users to stay on an older repo version if they're awaiting a fix to a regression in a newer version. - **Improved Documentation** - We're going to improve our documentation, making it easier for new users to get started with Serpent OS. -- **Tools as a service** - We're adding IPC layers to our tooling, allowing them to be launched as helper services for integration in - other applications. This will allow us to integrate moss with GNOME Software and COSMIC Store, for example. Additionally it will ensure - the latest version of the `moss` binary is always used by the system, gracefully handling version repo changes. - Chiefly, this will allow `moss` to cheaply integrate anywhere we need it, including for `lichen`. -- **Lichen Improvements** - Lichen will also be converted to an IPC based backend, allowing it to be executed via `pkexec` for integration into - a GUI (or even CLI) frontend. This is a necessary step to allow the frontend to operate without privileges, facilitating live - updates to the NetworkManager configuration, or indeed keyboard layouts. Long term we're planning a `cage` based kiosk with a full screen - installer mode, on a far smaller ISO. +- **Tools as a service** - We're adding IPC layers to our tooling, allowing them to be launched as helper services for integration in other applications. This will allow us to integrate moss with GNOME Software and COSMIC Store, for example. Additionally it will ensure the latest version of the `moss` binary is always used by the system, gracefully handling version repo changes. Chiefly, this will allow `moss` to cheaply integrate anywhere we need it, including for `lichen`. +- **Lichen Improvements** - Lichen will also be converted to an IPC based backend, allowing it to be executed via `pkexec` for integration into a GUI (or even CLI) frontend. This is a necessary step to allow the frontend to operate without privileges, facilitating live updates to the NetworkManager configuration, or indeed keyboard layouts. Long term we're planning a `cage` based kiosk with a full screen installer mode, on a far smaller ISO. ## Long term vision -Our long term vision is to deliver a system that is easy to use, reliable, and secure. We're building Serpent OS to be a system that -updates forever, confidently. We're also shaking things up by being first-adopter of new technologies as default solutions, with a view -to enhancing the robustness, origin-diversity and security of the system through Rust based replacements and alternatives to key components. -You can already see this with our adoption of `sudo-rs`, `coreutils` (`uutils`), `ntpd-rs`, and not to forget `rustls` by default for our curl build. +Our long term vision is to deliver a system that is easy to use, reliable, and secure. We're building Serpent OS to be a system that updates forever, confidently. We're also shaking things up by being first-adopter of new technologies as default solutions, with a view to enhancing the robustness, origin-diversity and security of the system through Rust based replacements and alternatives to key components. You can already see this with our adoption of `sudo-rs`, `coreutils` (`uutils`), `ntpd-rs`, and not to forget `rustls` by default for our curl build. Outside of that, `moss`, `boulder` and the supporting crates are also written in Rust, giving us a strong foundation for the future. -Serpent OS is a project that is built to last, and challenge the status quo. Deliver a Linux distribution for use on multiple verticals, -with the feeling of a conventional Linux distribution, but with key management facilities and architecture designed to make a rolling release -distribution that can be trusted for long term use, extensive updates, and the ability to entirely re-engineer the entire OS, delivering in a safe -atomic update without breaking client installations. +Serpent OS is a project that is built to last, and challenge the status quo. Deliver a Linux distribution for use on multiple verticals, with the feeling of a conventional Linux distribution, but with key management facilities and architecture designed to make a rolling release distribution that can be trusted for long term use, extensive updates, and the ability to entirely re-engineer the entire OS, delivering in a safe atomic update without breaking client installations. ## How you can help -In order to give Serpent OS the development time it needs, we're looking for sponsors to help fund development. If you're interested in becoming a sponsor -or investor, please do not hesitate to get in touch. We want to secure funding for development and infrastructure for the next 2 years. +In order to give Serpent OS the development time it needs, we're looking for sponsors to help fund development. If you're interested in becoming a sponsor or investor, please do not hesitate to get in touch. We want to secure funding for development and infrastructure for the next 2 years. - [Sponsors](/sponsor) - [Discuss sponsorship/investment](mailto:ikey@serpentos.com) -While of course we need to iterate on our own tooling, we also need to be able to work more upstream and with kindred projects to drive -the ecosystem forward. In time, we wish to become the incubator for new technologies and solutions, and we need your help to do that. +While of course we need to iterate on our own tooling, we also need to be able to work more upstream and with kindred projects to drive the ecosystem forward. In time, we wish to become the incubator for new technologies and solutions, and we need your help to do that. -With your support, we can expand our builder network and enable more hosting, bandwidth, and compute resources. This will in turn enable -our automated rebootstraps, which will of course allow downstreams to extend upon Serpent OS as a kit-distro. Indeed, it'll allow us to expand -beyond `x86_64` when the time is right. +With your support, we can expand our builder network and enable more hosting, bandwidth, and compute resources. This will in turn enable our automated rebootstraps, which will of course allow downstreams to extend upon Serpent OS as a kit-distro. Indeed, it'll allow us to expand beyond `x86_64` when the time is right. -While Serpent OS currently delivers a desktop ISO, this is not the limit of our potential. As indicated, our priorities in the short term -are designed to massively aimplify project maintainer bandwidth and free up more time for feature iteration and development. Once we're quite -happy with the base (and we're not far from that point) - we'll quite happily extend to other desktops, and even server/cloud offerings. +While Serpent OS currently delivers a desktop ISO, this is not the limit of our potential. As indicated, our priorities in the short term are designed to massively aimplify project maintainer bandwidth and free up more time for feature iteration and development. Once we're quite happy with the base (and we're not far from that point) - we'll quite happily extend to other desktops, and even server/cloud offerings. -Nothing is bullet proof, but with the architecture and tooling of Serpent OS we can get pretty close. Gotta admit, having a transactional, atomic OS -with rollbacks, and the full granularity of a conventional package manager, is pretty cool. 😎 Especially when things "Just work" out of the box -thanks to the stateless ("hermetic usr") requirements of the system. +Nothing is bullet proof, but with the architecture and tooling of Serpent OS we can get pretty close. Gotta admit, having a transactional, atomic OS with rollbacks, and the full granularity of a conventional package manager, is pretty cool. 😎 Especially when things "Just work" out of the box thanks to the stateless ("hermetic usr") requirements of the system. -Know what's even cooler? When we land our system-model implementation, you'll be able to completely duplicate the setup between local development -and production. When we also land our export/compose functionality in moss, you'll be able to reap these benefits in containers too. 🐍 +Know what's even cooler? When we land our system-model implementation, you'll be able to completely duplicate the setup between local development and production. When we also land our export/compose functionality in moss, you'll be able to reap these benefits in containers too. 🐍 Here's to a great 2025, and we hope you'll join us on this journey. 🚀 diff --git a/src/content/docs/blog/2025/01/04/offline-rollbacks-enabled/featured-background.png b/src/content/docs/blog/2025/01/04/offline-rollbacks-enabled/featured-background.png deleted file mode 100644 index b2a402ea..00000000 Binary files a/src/content/docs/blog/2025/01/04/offline-rollbacks-enabled/featured-background.png and /dev/null differ diff --git a/src/content/docs/blog/2025/01/04/offline-rollbacks-enabled/featured-background.webp b/src/content/docs/blog/2025/01/04/offline-rollbacks-enabled/featured-background.webp new file mode 100644 index 00000000..cb83823c Binary files /dev/null and b/src/content/docs/blog/2025/01/04/offline-rollbacks-enabled/featured-background.webp differ diff --git a/src/content/docs/blog/2025/01/04/offline-rollbacks-enabled/index.md b/src/content/docs/blog/2025/01/04/offline-rollbacks-enabled/index.md new file mode 100644 index 00000000..fd6538e7 --- /dev/null +++ b/src/content/docs/blog/2025/01/04/offline-rollbacks-enabled/index.md @@ -0,0 +1,55 @@ +--- +title: Offline Rollbacks Enabled +date: 2025-01-04T22:19:37Z +authors: [ikey] +cover: + alt: AerynOS terminal removing packages that would brick most other distros + image: ./featured-background.webp +tags: [news] +license: "CC-BY-ND-4.0" +copyright: "Copyright © 2025 aerynOS Developers" +--- + +If a picture tells a thousand words... well then this video .. Yknow what, sod it. Here's a video of the offline rollback support in action, as tested in a newly installed VM using `xfs` for the root filesystem. We jokingly refer to this as the "LTT" test due to LinusTechTips' "interesting encounters" with package management systems in the past. + +This has already landed in our volatile repository, so simply `sudo moss sync -u` and grab the latest updates. Any subsequent transactions will automatically generate the boot entries for you, no intervention required. Or mounting. Or anything. Just update and go. Srsly. + + + +:::note[Missing YouTube video] +There used to be a YouTube video at this part of the blog, however our YouTube channel was unfortunately deleted outside of our control and we no longer have access to the video. +::: + +We actually saw this as a golden standard for the feature, as it's a real-world scenario where you'd want to roll back to a known good state. The video shows the system booting and recovering in multiple ways, from complete system nuke (via glibc) and simpler scenarios like removal of the GNOME desktop environment. + +The intent is to ensure a user can quickly revert to the last transaction that worked in the instance an update goes awry, without needing to boot into a rescue environment or live CD. + +### Like what we're doing? + +Developing a project with the scope of Serpent OS is no small feat. In order to maintain our current cadence, and to land huge features like the offline rollback, or impending system model work, we need your support. I'm as happy as the next person to return to the job market once my medical conditions ease up, but the simple matter is this will greatly slow down our recent progress. + +Our aim is to entirely redefine the **distribution** of Linux, and the years of prior effort are finally coming together rapidly to make this a reality. If you like what we're doing, consider supporting us! + + + +### Technical Details + +This is all achieved using moss's internal content addressable storage for deduplication, where each old transaction is swapped with the live `/usr` at the time into the archive. + +The behaviour is implemented in the initramfs (`dracut`) by invoking `moss state activate $transaction_id -D /sysroot`, ensuring that the activation is performed *before* the new root is mounted and the system is booted. We actually shrunk our `moss` and `uutils-coreutils` builds to half their usual size so that we can fit them comfortably into the initramfs. + +For now, we'll automatically generate entries on the ESP (or XBOOTLDR) for the last 5 transactions, and you can select them during early boot to perform the rollback. It's not slow, as it's just doing yet another `renameat2()` to atomically exchange the live `/usr` with the archived transaction. + +TLDR: Boot time rollback, no network required, no live CD required, no rescue environment required. Just a working system. + +### What's next? + +The system model is coming next, which will expose the non-imperative core of the architecture to users. The usual commands will continue to emulate an imperative OS, by mutating the system model and applying the internal `sync` operation. This ensures dependencies are resolved only according to the layered repository configuration, without special exceptions for "installed" (which.. we spent a lot of effort faking this behaviour). + +It's good to remember that moss is effectively a huge cache, and that each state has no relation to the other. We simulate upgrades by building a new state with the same selections as the old one, then begin to upgrade those candidates. + +The new model (somewhat Gentoo inspired ❤️) will allow moss to build the dependency graph for the finished system state. As a sneak preview of what we're enabling, imagine the newly landed transactional reboot code linked with **multiple** system models... such that you might decide to `moss sync` from your existing model to an entirely different one, rebooting from your GNOME desktop to a KDE desktop, for example. Cleanly, atomically, and without any manual intervention. diff --git a/src/content/docs/blog/2025/01/04/offline-rollbacks-enabled/index.mdx b/src/content/docs/blog/2025/01/04/offline-rollbacks-enabled/index.mdx deleted file mode 100644 index 2d623e77..00000000 --- a/src/content/docs/blog/2025/01/04/offline-rollbacks-enabled/index.mdx +++ /dev/null @@ -1,79 +0,0 @@ ---- -title: Offline Rollbacks Enabled -date: 2025-01-04T22:19:37Z -authors: [ikey] -tags: [news] -license: "CC-BY-ND-4.0" -copyright: "Copyright © 2025 aerynOS Developers" ---- -import YouTube from '@/components/ui/YouTube.astro'; -import Kofi from '@/components/ui/Kofi.astro'; - -If a picture tells a thousand words... well then this video .. Yknow what, sod it. -Here's a video of the offline rollback support in action, as tested in a newly installed -VM using `xfs` for the root filesystem. We jokingly refer to this as the "LTT" test due -to LinusTechTips' "interesting encounters" with package management systems in the past. - -This has already landed in our volatile repository, so simply `sudo moss sync -u` and grab -the latest updates. Any subsequent transactions will automatically generate the boot entries -for you, no intervention required. Or mounting. Or anything. Just update and go. Srsly. - - - -We actually saw this as a golden standard for the feature, as it's a real-world scenario -where you'd want to roll back to a known good state. The video shows the system booting -and recovering in multiple ways, from complete system nuke (via glibc) and simpler scenarios -like removal of the GNOME desktop environment. - -The intent is to ensure a user can quickly revert to the last transaction that worked in the -instance an update goes awry, without needing to boot into a rescue environment or live CD. - - - -### Like what we're doing? - -Developing a project with the scope of Serpent OS is no small feat. In order to maintain our -current cadence, and to land huge features like the offline rollback, or impending system -model work, we need your support. I'm as happy as the next person to return to the job market -once my medical conditions ease up, but the simple matter is this will greatly slow down our -recent progress. - -Our aim is to entirely redefine the **distribution** of Linux, and the years of prior effort -are finally coming together rapidly to make this a reality. If you like what we're doing, -consider supporting us! - - - -Visit our [sponsor](/sponsor) page for more information on how you can help us grow and deliver -the future of Linux distributions. - -### Technical Details - -This is all achieved using moss's internal content addressable storage for deduplication, -where each old transaction is swapped with the live `/usr` at the time into the archive. - -The behaviour is implemented in the initramfs (`dracut`) by invoking `moss state activate $transaction_id -D /sysroot`, -ensuring that the activation is performed *before* the new root is mounted and the system is booted. -We actually shrunk our `moss` and `uutils-coreutils` builds to half their usual size so that we can -fit them comfortably into the initramfs. - -For now, we'll automatically generate entries on the ESP (or XBOOTLDR) for the last 5 transactions, and you -can select them during early boot to perform the rollback. It's not slow, as it's just doing yet another -`renameat2()` to atomically exchange the live `/usr` with the archived transaction. - -TLDR: Boot time rollback, no network required, no live CD required, no rescue environment required. Just a working system. - -### What's next? - -The system model is coming next, which will expose the non-imperative core of the architecture to users. -The usual commands will continue to emulate an imperative OS, by mutating the system model and applying -the internal `sync` operation. This ensures dependencies are resolved only according to the layered repository -configuration, without special exceptions for "installed" (which.. we spent a lot of effort faking this behaviour). - -It's good to remember that moss is effectively a huge cache, and that each state has no relation to the other. We -simulate upgrades by building a new state with the same selections as the old one, then begin to upgrade those candidates. - -The new model (somewhat Gentoo inspired ❤️) will allow moss to build the dependency graph for the finished system state. -As a sneak preview of what we're enabling, imagine the newly landed transactional reboot code linked with **multiple** -system models... such that you might decide to `moss sync` from your existing model to an entirely different one, rebooting -from your GNOME desktop to a KDE desktop, for example. Cleanly, atomically, and without any manual intervention. diff --git a/src/content/docs/blog/2025/02/06/hello-2025/Featured.png b/src/content/docs/blog/2025/02/06/hello-2025/Featured.png deleted file mode 100644 index 8e7dc3b5..00000000 Binary files a/src/content/docs/blog/2025/02/06/hello-2025/Featured.png and /dev/null differ diff --git a/src/content/docs/blog/2025/02/06/hello-2025/Featured.webp b/src/content/docs/blog/2025/02/06/hello-2025/Featured.webp new file mode 100644 index 00000000..41d953ca Binary files /dev/null and b/src/content/docs/blog/2025/02/06/hello-2025/Featured.webp differ diff --git a/src/content/docs/blog/2025/02/06/hello-2025/index.md b/src/content/docs/blog/2025/02/06/hello-2025/index.md new file mode 100644 index 00000000..9bd1e311 --- /dev/null +++ b/src/content/docs/blog/2025/02/06/hello-2025/index.md @@ -0,0 +1,214 @@ +--- +title: Hello 2025 +date: 2025-02-06T22:05:55Z +cover: + alt: Preview of new AerynOS installer + image: ./Featured.webp +authors: [ikey] +tags: [news] +license: "CC-BY-ND-4.0" +copyright: "Copyright © 2025 aerynOS Developers" +--- + +**TL;DR**: Serpent OS is facing funding challenges but development continues. Alpha2 is coming soon with an improved installer. We're seeking community support through donations and volunteers for key roles. Our technical roadmap includes versioned repositories, immutable OS features, and improved package management workflows. + +--- + +We had a flurry of activity around the Christmas period, including our first alpha release as well as enabling offline rollbacks early in January. We're actively working on alpha2, but we also need to talk about the elephant in the room. + +## 🐘 The elephant in the room + +Recently I posted a [tweet](https://x.com/Serpent_OS/status/1886715364744007935) that has been covered by a few news outlets. The long and short is quite simple: distinct lack of funds. A *very long* story +short: a change in my contract led to a loss of income protection, combined with a herniated disc (C6-C7), culminated in having to resign. Naturally I've been working exclusively on Serpent OS since then, but I've also got to keep the lights on at home. + +![Broke](./pockets_empty.gif) + +This doesn't mean the project is closing or anything like that, but it does mean that I actively need to keep costs down whilst using daytime hours for financing. This is what's led to the significant slowdown in the last few weeks, and I'm just trying to be as transparent as possible and manage expectations. + + + +### 💼 But couldn't you just get a job? + +Sure. The reality is that with a job, and kids, there's very little time left for Serpent OS. I personally believe very strongly in Serpent OS, and I want to see it succeed. As a truly independent project, we're in a unique position to do things that other projects can't. We haven't got to answer to shareholders, or concern ourselves with market share. We can focus on making the best possible OS for our users. + +But it's not *just* that. We're building the tools so that others can build *their* vision of the best OS. Or provide appliances. Or <insert your idea here>. We're building a set of tools and principles to facilitate innovation and disruption, for projects of any size. We're not just building an OS, we're building a platform. Bring your own distro, if you will. + +My sincere hope is that others also believe in this vision, and can help us get there. It's make or break time, and you have the power to help us make it. + +### 💰 Funding + +OK, so how can folks help? The most obvious one is funding, so we'll get that out of the way first. Right now it's possible to support us via Ko-fi or GitHub Sponsors[^1]. For the sake of transparency, as noted on the Funding page, we haven't got an established company or business bank account. In time, we're looking towards something more cooperative. For now, the reality is that the daily running and development largely falls on me. + + + +### 🏗️ Infrastructure + +One way that we could certainly cut some costs, and improve scaling, is through additional infrastructure. We're currently hosted on a single Hetzner server, and we're looking to expand that to a more distributed setup. Rune Morling currently hosts 2 of the builders in his home at his own expense, and the main server is also the primary builder, which is less than ideal. + +We need to split the repository hosting as its grown enormously, and could do with a few more builders to facilitate "try builds" of pull requests. + +Lastly we're looking at CDN solutions to improve the speed of package downloads (efforts currently ongoing). + +### 👥 Roles and responsibilities + +We're growing quickly and that comes with some pain points. We're looking for immediate help in the following key areas: + +- **Community management**: Someone to manage social media, forums, and other community platforms. +- **Documentation**: Once we pivot to Astro + Starship, we'll need to focus on adding high quality documentation. +- **Translation**: We're enabling gettext + fluent where appropriate, and will need extensive translation work. +- **Web development**: We need to improve our web presence, management of web properties, transition to Astro, etc. + +While of course we'd also love more developers and packagers, most of the time is spent in these areas already, and the other roles desperately need attention. + +## 🚀 Latest developments + +Before we get into the future, let's talk about what we've been working on lately. + +### 💾 Disk Management (disks-rs) + +A couple of weeks back we broke the news that we were working on a new disk management library, `disks-rs`. The scope wasn't made quite clear at the time, but it's another building block to facilitate smarter applications. Eventually we want to support the manipulation of various filesystems and disk structures, for use in CI pipelines, etc. + +Another requirement for disk-rs is to not only expose sane partitioning logic, but also to provide a way to declaratively define disk layouts. This will be used in the installer, but also for reproducing entire installations. It forms the basis for our future provisioning system. + +Right now we're working on a very cool [`.kdl`](https://kdl.dev) based format for defining disk layouts. It takes advantage of `kdl-rs` streaming parser to implement a procedural syntax complete with miette error handling for a very nice developer experience: + +```kdl +// Create a partition table. Defaults to GPT +create-partition-table type="gpt" disk="root_disk" + +// Create the ESP +create-partition disk="root_disk" id="esp" { + constraints { + min (GB)1 + max (GB)2 + } + type ... +} +``` + +### 🛠️ Installer + +Our installer has quite a nice backend, but honestly the frontend experience leaves a lot to be desired. Given that our current focus is being able to not only onboard developers but build tooling against an installed system, we need to make the core of this effectively scriptable. To do so will build upon ongoing IPC work for moss and disks-rs, and the introduction of a new GUI frontend based on [Slint](https://slint.dev/). While early days we're trying to think 10 miles down the road, where we can utilise the LinuxKMS backend for minimal installer ISOs. + +Early preview: + +![Installer preview](./Featured.webp "Preview of new AerynOS installer") + +## 🔮 The road to stability + +So we have some exploratory works in progress, and it's time to discuss how they all relate to the future of Serpent OS. You'll immediately notice that these are highly circular in nature, meaning we need to switch between different areas routinely. In order to design an effective architecture, whilst minimising disruption, we need to have a clear vision of the future that we're referring to as "10 miles down the road". + +### 📚 Versioned repositories + +As previously explained, one of the built-in mechanisms to support a rolling release with breaking changes to the format is to introduce explicitly versioned repositories. Long story short, a local `moss` client will only "see" the latest version of the repository that it can actually use. Breaking changes will mean format bumps, and newer repository versions will be unavailable until the client has updated to a client/Serpent OS version that actually supports it. + +This work requires changes to the repository format, infrastructure, and a migration path from the current repository to a new archive. However, it is also dependent on having a new format version being available for the versioned-indices. + +### 🔒 Immutable OS + +Frequently Serpent OS is described as an immutable OS, but in actuality it is an **atomic** OS. That means we've focused entirely on providing a reliable update mechanism, rollbacks, etc. We have not yet enforced immutability, aka a read-only root filesystem. + +Our plan is heavily inspired by [composefs](https://github.com/containers/composefs). Currently `moss` produces new filesystem trees using a hardlink farm approach, and relies on `renameat2()` to atomically switch the root filesystem. This is a very fast operation, but does not provide immutability. + +Leveraging the ability of moss to dynamically construct filesystems from packages, it will be abstracted into a driver mechanism. This will allow the classic hardlink approach for buildroots, but employ a new `erofs` strategy for immutable installs. The newly generated `erofs` image will essentially rely on extended attributes to "point" to the underlying content addressable storage, and utilise `overlayfs` to present a read-only view of the filesystem. Further building on this we'll use so-called "mount tucking" to provide the mechanism for atomic updates without the need for a reboot. + +#### 🤔 Why not use composefs? + +We're actually going to make moss support exports to composefs format, making it an ideal match for `podman` use cases, deduplicating the image content. Whereas `composefs` produces content from an existing tree, we've already got content addressable storage, independent transactions and databases to facilitate this. This means we'll be able to retain our composition features and very quickly produce the new `erofs` images. Using `composefs` directly would significantly slow down the time to produce each transactions, whereas we can instead build a similar internal functionality and integrate at higher levels. + +We also plan to integrate the file digests in a new revision of the moss `stone` format, which will open the door to supporting SELinux xattrs, as well as `fsverity` for the immutable images. + +Note that this goal will inform the requirements for the next stone format, which will in turn require versioned repositories. Of course, this also impacts the build pipeline infrastructure. + +### 👷 Maintainer burden + +Truly the most time consuming task in most distributions is the *cost* of maintaining packages. This is being approached, as with all of the planned goals, in an incremental fashion. However, it often pays to have a view 10 miles down the road to determine exactly how to get there. + +#### 🧬 ABI Tracking + +In order to provide sanity for developers, and make guarantees about the stability of the system, we need to track the binary ABI for all packages. Furthering on this, we can of course immediately detect any breakage and schedule rebuilds as appropriate. This will also add a higher degree of confidence for pull requests as we'll know if its consistent or introducing problems. + +#### 🔄 Pull Request workflows + +The workflow for packaging right now is for someone to open a pull request, which contains their recipes, the automatic "build records" (manifest) and our assumption that they built it using a local repo based on top of the latest repository. What is actually needed here is for the infrastructure to perform automatic (low priority) builds of each pull request in a *transient* repository, making those builds available for verification purposes. + +#### ⚛️ Atomic repo merges + +Another limitation with the current infrastructure is the reliance on a global build queue. Simply put, once a PR is merged, all of those builds are scheduled and individually published to the target repository upon completion. This is very naive, and leads to a lot of "nannying" in the event of a failed merge. + +In future, those builds will be scheduled in a transient repository, and upon successful completion the resulting artefacts would be published to the target repository in a single atomic operation. This will also allow for the introduction of "try builds" for pull requests. + +#### 🩸 The bleed-through problem + +Another issue which partially relates to bootstraps, is the bleed through problem. This is an artefact of using layered repositories, such as a local repository, to build updates that would change the providers available in the underlying repository upon merge. Take for example, LLVM. + +``` +main repo: + ├── rust [runtime-depends: libLLVM-18.so] + └── build-dep: binary(rust) [runtime-depends: libLLVM-18.so] + +local repo: + ├── rust [runtime-depends: libLLVM-19.so] + └── build-dep: binary(rust) [runtime-depends: libLLVM-18.so] +``` + +After completion of libLLVM-19.so, the local repository would contain the new version, but the main repository would still contain the old version. Locally this would allow the circular build dependencies to work as `rust` builddep for `rust` could still resolve `libLLVM-18.so`. However, upon, merging to the repository, we have a new problem. The main repository would now contain `libLLVM-19.so`, but the `rust` package would still be built against `libLLVM-18.so`. This then breaks the dependency chain for `rust`, requiring manual intervention. + +To solve this, we'll ensure that **filtered** repository views are introduced both for local builds and our build infrastructure. This will mean that the index of the local repository will *occlude* the main repository, and eliminate bleed-through. + +#### ⭕ The bootstrap problem + +Whilst ABI tracking is very important, it doesn't actually solve the bootstrap problem of circular dependencies. This is a consequence of using an incremental development model for a binary-first distribution. In reality the only real way to solve this is by eventually altering the tooling to support a "world view", with intermediate recipes that are not published to the final repository. In effect, combined with ABI tracking for dependency chain invalidation, we can determine which recipes need to run again as part of an invalidation-solver loop that relies on "cached artifacts", i.e. the final packages. + +This isn't an immediate issue to solve, but it will inform our next steps to ensure we get to that point without requiring extensive overhauls or "stop the world" rewrites. Our eventual vision is that building any new package will feel much like it does now, but changing a reverse dependency will evaluate the dependent chain for rebuilds. To make this streamlined, the infrastructure will by this time have grown to a point where we can share the compute power by way of so-called "try builds", and a shared cache of artifacts. + +#### 📦 Keeping the packages up to date + +A large portion of the burden is in reality chasing updates. Currently we have a very simplistic utility that cross-references [release-monitoring.org](https://release-monitoring.org) to spit out a list of recipes that have available updates. Truthfully we could **easily** generate diffs for these updates, and even automate the process of updating the recipes. This is dependent however on having confidence in the pull request, which in turn is dependent on ABI tracking being in place. + +### 🗂️ Organising it all + +We have a complex series of interdependent goals, which require research and frequently shifting between focus areas. Not all of them will be immediately realised, and some are simply informing the direction for our future. Additionally, we need to keep moving without any "rewrite the world" pauses. + +It's important to highlight that none of this is a reinvention of the project, but a series of steps to "1.0" and beyond to ensure that the project is sustainable and reliable. This means that delivery continues, and the delivery itself becomes simpler and more trustworthy. + +Despite the interdependencies, once we've finished scoping the end requirements we can work towards the immediate concerns: + + - Pull request workflow + - Repository *consistency* (initially ABI story) + - Update automation + - Immutability + +Note: These timelines are estimates and may adjust based on available resources and community support. + +## Scale-up considerations + +We're at the point where we need a functional OS for contributors to be able to work on the project. It also presents a paradox because we wish to keep the project lean whilst we work on extending the scale-out capabilities. In reality this means that we're going to permit a **limited** amount of repository growth: + + - Addition of the Plasma desktop for daily use + - Container-management tooling (podman, docker, etc) + - Development tools (IDEs, etc) + - Onboarding requirements (such as Matrix clients, documentation, etc) + +This in turn also requires a more reliable installation experience, which is why we've been spending cycles on the installer project. Internally we want the installer to support a variety of scriptable configurations in order to provide better test coverage for our updates and boot management, whilst we also intend to **incrementally** improve the user experience. + +## 📅 Conclusion + +Alpha2 is coming, and will feature a preliminary version of the updated installer. The plan initially is for a simple "use whole disk" strategy and will be followed up with supported for encrypted installations (post alpha2). Quite simply, some employers require their developers have encrypted installs, and not supporting this configuration will restrict our ability to onboard developers. + +We're doing the Alpha2 as a baseline, from which all update testing, infrastructure adjustments, etc, will be measured against. This is the point at which the project will formally "open the doors" for more developers and testers, with the explicit view that growth is sustainable and manageable. Per the requirements set above, we're not going to "explode", but enable specific workflows that in turn will permit us to scale out the project and inform direction. + +It's important to me that we're explicit in how we take our next steps. As stated earlier, pausing the world is not an option. Continued delivery whilst enabling an incremental path to the future is the only way to ensure that we can deliver on our promises. Contrary to a reboot, this is a solidification of our vision and a commitment to the future. Sure, new features will also appear in time, such as easier packaging recipe formats, but no decision is being made that will require a rewrite of the project. However, these goals require stating and explaining up front so that the high level vision is clear and provides the context for any short-term deliverables. The last thing we need is to be in a position where we're "stuck" and unable to deliver, due to a lack of foresight. + +## 🙏 Thank you + +On behalf of the project, I'd like to thank everyone who has supported us so far. Whether that's through donations, code contributions, or even having the time of day to try out the ISO, it's deeply appreciated. Being the underdog comes with its own set of challenges, and of course the "indie cost", but it's also an opportunity to challenge the status quo and find new, more optimal ways, to do things. + +We're not just building "Serpent **OS**", we're building a set of tools to solve the problem of distributing an OS. Way down the line this will be powerful enough to support a number of use cases, from appliances to desktops, and even servers. In short, we want to bring high quality tooling to the masses, and we're doing it in a way that's sustainable and reliable. + +--- +[^1]: Since this blog post, we no longer use GitHub Sponsors. For funding, please see our [Funding Page](/sponsor). diff --git a/src/content/docs/blog/2025/02/06/hello-2025/index.mdx b/src/content/docs/blog/2025/02/06/hello-2025/index.mdx deleted file mode 100644 index 65607a25..00000000 --- a/src/content/docs/blog/2025/02/06/hello-2025/index.mdx +++ /dev/null @@ -1,299 +0,0 @@ ---- -title: Hello 2025 -date: 2025-02-06T22:05:55Z -authors: [ikey] -tags: [news] ---- -import Giphy from '@/components/ui/Giphy.astro'; - -**TL;DR**: Serpent OS is facing funding challenges but development continues. Alpha2 is coming soon with an improved installer. We're seeking community support through donations and volunteers for key roles. Our technical roadmap includes versioned repositories, immutable OS features, and improved package management workflows. - - - ---- - -We had a flurry of activity around the Christmas period, including our first alpha release as well as -enabling offline rollbacks early in January. We're actively working on alpha2, but we also need to -talk about the elephant in the room. - -## 🐘 The elephant in the room - -Recently I posted a [tweet](https://x.com/Serpent_OS/status/1886715364744007935) that has been covered -by a few news outlets. The long and short is quite simple: distinct lack of funds. A *very long* story -short: a change in my contract led to a loss of income protection, combined with a herniated disc (C6-C7), -culminated in having to resign. Naturally I've been working exclusively on Serpent OS since then, but -I've also got to keep the lights on at home. - - - -This doesn't mean the project is closing or anything like that, but it does mean that I actively need to keep -costs down whilst using daytime hours for financing. This is what's led to the significant slowdown in the -last few weeks, and I'm just trying to be as transparent as possible and manage expectations. - -### 💼 But couldn't you just get a job? - -Sure. The reality is that with a job, and kids, there's very little time left for Serpent OS. I personally -believe very strongly in Serpent OS, and I want to see it succeed. As a truly independent project, we're -in a unique position to do things that other projects can't. We haven't got to answer to shareholders, or -concern ourselves with market share. We can focus on making the best possible OS for our users. - -But it's not *just* that. We're building the tools so that others can build *their* vision of the best OS. -Or provide appliances. Or <insert your idea here>. We're building a set of tools and principles to facilitate -innovation and disruption, for projects of any size. We're not just building an OS, we're building a platform. -Bring your own distro, if you will. - -My sincere hope is that others also believe in this vision, and can help us get there. It's make or break time, -and you have the power to help us make it. - -### 💰 Funding - -OK, so how can folks help? The most obvious one is funding, so we'll get that out of the way first. Right now -it's possible to support us via [Ko-fi](https://ko-fi.com/serpentos) or [GitHub Sponsors](https://github.com/sponsors/ikeycode). -For the sake of transparency, as noted on the [Sponsor](/sponsor) page, we haven't got an established company -or business bank account. In time, we're looking towards something more cooperative. For now, the reality is -the daily running and development largely falls on me. - -### 🏗️ Infrastructure - -One way that we could certainly cut some costs, and improve scaling, is through additional infrastructure. -We're currently hosted on a single Hetzner server, and we're looking to expand that to a more distributed -setup. Rune Morling currently hosts 2 of the builders in his home at his own expense, and the main server -is also the primary builder, which is less than ideal. - -We need to split the repository hosting as its grown enormously, and could do with a few more builders to -facilitate "try builds" of pull requests. - -Lastly we're looking at CDN solutions to improve the speed of package downloads (efforts currently ongoing). - -### 👥 Roles and responsibilities - -We're growing quickly and that comes with some pain points. We're looking for immediate help in the following key areas: - -- **Community management**: Someone to manage social media, forums, and other community platforms. -- **Documentation**: Once we pivot to Astro + Starship, we'll need to focus on adding high quality documentation. -- **Translation**: We're enabling gettext + fluent where appropriate, and will need extensive translation work. -- **Web development**: We need to improve our web presence, management of web properties, transition to Astro, etc. - -While of course we'd also love more developers and packagers, most of the time is spent in these areas already, and the other -roles desperately need attention. - -## 🚀 Latest developments - -Before we get into the future, let's talk about what we've been working on lately. - -### 💾 Disk Management (disks-rs) - -A couple of weeks back we broke the news that we were working on a new disk management library, `disks-rs`. -The scope wasn't made quite clear at the time, but it's another building block to facilitate smarter applications. -Eventually we want to support the manipulation of various filesystems and disk structures, for use in CI pipelines, etc. - -Another requirement for disk-rs is to not only expose sane partitioning logic, but also to provide a way to declaratively -define disk layouts. This will be used in the installer, but also for reproducing entire installations. It forms the basis -for our future provisioning system. - -Right now we're working on a very cool [`.kdl`](https://kdl.dev) based format for defining disk layouts. It takes advantage -of `kdl-rs` streaming parser to implement a procedural syntax complete with miette error handling for a very nice developer -experience: - -```kdl -// Create a partition table. Defaults to GPT -create-partition-table type="gpt" disk="root_disk" - -// Create the ESP -create-partition disk="root_disk" id="esp" { - constraints { - min (GB)1 - max (GB)2 - } - type ... -} -``` - -### 🛠️ Installer - -Our installer has quite a nice backend, but honestly the frontend experience leaves a lot to be desired. Given that our current -focus is being able to not only onboard developers but build tooling against an installed system, we need to make the core of -this effectively scriptable. To do so will build upon ongoing IPC work for moss and disks-rs, and the introduction of a new GUI -frontend based on [Slint](https://slint.dev/). While early days we're trying to think 10 miles down the road, where we can utilise -the LinuxKMS backend for minimal installer ISOs. - -Early preview: - -![Installer preview](Featured.png) - - -## 🔮 The road to stability - -So we have some exploratory works in progress, and it's time to discuss how they all relate to the future of Serpent OS. -You'll immediately notice that these are highly circular in nature, meaning we need to switch between different areas -routinely. In order to design an effective architecture, whilst minimising disruption, we need to have a clear vision -of the future that we're referring to as "10 miles down the road". - -### 📚 Versioned repositories - -As previously explained, one of the built-in mechanisms to support a rolling release with breaking changes to the format is to introduce -explicitly versioned repositories. Long story short, a local `moss` client will only "see" the latest version of the repository that it can -actually use. Breaking changes will mean format bumps, and newer repository versions will be unavailable until the client has updated to -a client/Serpent OS version that actually supports it. - -This work requires changes to the repository format, infrastructure, and a migration path from the current repository to a new -archive. However, it is also dependent on having a new format version being available for the versioned-indices. - -### 🔒 Immutable OS - -Frequently Serpent OS is described as an immutable OS, but in actuality it is an **atomic** OS. That means we've focused entirely -on providing a reliable update mechanism, rollbacks, etc. We have not yet enforced immutability, aka a read-only root filesystem. - -Our plan is heavily inspired by [composefs](https://github.com/containers/composefs). Currently `moss` produces new filesystem trees -using a hardlink farm approach, and relies on `renameat2()` to atomically switch the root filesystem. This is a very fast operation, -but does not provide immutability. - -Leveraging the ability of moss to dynamically construct filesystems from packages, it will be abstracted into a driver mechanism. This will -allow the classic hardlink approach for buildroots, but employ a new `erofs` strategy for immutable installs. The newly generated -`erofs` image will essentially rely on extended attributes to "point" to the underlying content addressable storage, and utilise -`overlayfs` to present a read-only view of the filesystem. Further building on this we'll use so-called "mount tucking" to provide -the mechanism for atomic updates without the need for a reboot. - -#### 🤔 Why not use composefs? - -We're actually going to make moss support exports to composefs format, making it an ideal match for `podman` use cases, deduplicating -the image content. Whereas `composefs` produces content from an existing tree, we've already got content addressable storage, independent -transactions and databases to facilitate this. This means we'll be able to retain our composition features and very quickly produce the -new `erofs` images. Using `composefs` directly would significantly slow down the time to produce each transactions, whereas we can instead -build a similar internal functionality and integrate at higher levels. - -We also plan to integrate the file digests in a new revision of the moss `stone` format, which will open the door to supporting -SELinux xattrs, as well as `fsverity` for the immutable images. - -Note that this goal will inform the requirements for the next stone format, which will in turn require versioned repositories. -Of course, this also impacts the build pipeline infrastructure. - -### 👷 Maintainer burden - -Truly the most time consuming task in most distributions is the *cost* of maintaining packages. This is being approached, as with all -of the planned goals, in an incremental fashion. However, it often pays to have a view 10 miles down the road to determine exactly how -to get there. - -#### 🧬 ABI Tracking - -In order to provide sanity for developers, and make guarantees about the stability of the system, we need to track the binary ABI for -all packages. Furthering on this, we can of course immediately detect any breakage and schedule rebuilds as appropriate. This will also -add a higher degree of confidence for pull requests as we'll know if its consistent or introducing problems. - -#### 🔄 Pull Request workflows - -The workflow for packaging right now is for someone to open a pull request, which contains their recipes, the automatic "build records" (manifest) -and our assumption that they built it using a local repo based on top of the latest repository. What is actually needed here is for the infrastructure -to perform automatic (low priority) builds of each pull request in a *transient* repository, making those builds available for verification -purposes. - -#### ⚛️ Atomic repo merges - -Another limitation with the current infrastructure is the reliance on a global build queue. Simply put, once a PR is merged, all of those builds -are scheduled and individually published to the target repository upon completion. This is very naive, and leads to a lot of "nannying" in the -event of a failed merge. - -In future, those builds will be scheduled in a transient repository, and upon successful completion the resulting artefacts would be published -to the target repository in a single atomic operation. This will also allow for the introduction of "try builds" for pull requests. - -#### 🩸 The bleed-through problem - -Another issue which partially relates to bootstraps, is the bleed through problem. This is an artefact of using layered repositories, such as a local -repository, to build updates that would change the providers available in the underlying repository upon merge. Take for example, LLVM. - -``` -main repo: - ├── rust [runtime-depends: libLLVM-18.so] - └── build-dep: binary(rust) [runtime-depends: libLLVM-18.so] - -local repo: - ├── rust [runtime-depends: libLLVM-19.so] - └── build-dep: binary(rust) [runtime-depends: libLLVM-18.so] -``` - -After completion of libLLVM-19.so, the local repository would contain the new version, but the main repository would still contain the old version. -Locally this would allow the circular build dependencies to work as `rust` builddep for `rust` could still resolve `libLLVM-18.so`. However, upon, -merging to the repository, we have a new problem. The main repository would now contain `libLLVM-19.so`, but the `rust` package would still be -built against `libLLVM-18.so`. This then breaks the dependency chain for `rust`, requiring manual intervention. - -To solve this, we'll ensure that **filtered** repository views are introduced both for local builds and our build infrastructure. This will mean -that the index of the local repository will *occlude* the main repository, and eliminate bleed-through. - -#### ⭕ The bootstrap problem - -Whilst ABI tracking is very important, it doesn't actually solve the bootstrap problem of circular dependencies. This is a consequence -of using an incremental development model for a binary-first distribution. In reality the only real way to solve this is by eventually -altering the tooling to support a "world view", with intermediate recipes that are not published to the final repository. In effect, combined -with ABI tracking for dependency chain invalidation, we can determine which recipes need to run again as part of an invalidation-solver loop -that relies on "cached artifacts", i.e. the final packages. - -This isn't an immediate issue to solve, but it will inform our next steps to ensure we get to that point without requiring extensive -overhauls or "stop the world" rewrites. Our eventual vision is that building any new package will feel much like it does now, but changing a -reverse dependency will evaluate the dependent chain for rebuilds. To make this streamlined, the infrastructure will by this time have -grown to a point where we can share the compute power by way of so-called "try builds", and a shared cache of artifacts. - -#### 📦 Keeping the packages up to date - -A large portion of the burden is in reality chasing updates. Currently we have a very simplistic utility that cross-references -[release-monitoring.org](https://release-monitoring.org) to spit out a list of recipes that have available updates. Truthfully -we could **easily** generate diffs for these updates, and even automate the process of updating the recipes. This is dependent -however on having confidence in the pull request, which in turn is dependent on ABI tracking being in place. - -### 🗂️ Organising it all - -We have a complex series of interdependent goals, which require research and frequently shifting between focus areas. Not all of them will be immediately realised, -and some are simply informing the direction for our future. Additionally, we need to keep moving without any "rewrite the world" pauses. - -It's important to highlight that none of this is a reinvention of the project, but a series of steps to "1.0" and beyond to ensure that -the project is sustainable and reliable. This means that delivery continues, and the delivery itself becomes simpler and more trustworthy. - -Despite the interdependencies, once we've finished scoping the end requirements we can work towards the immediate concerns: - - - Pull request workflow - - Repository *consistency* (initially ABI story) - - Update automation - - Immutability - -Note: These timelines are estimates and may adjust based on available resources and community support. -## Scale-up considerations - -We're at the point where we need a functional OS for contributors to be able to work on the project. It also presents a paradox because -we wish to keep the project lean whilst we work on extending the scale-out capabilities. In reality this means that we're going to permit -a **limited** amount of repository growth: - - - Addition of the Plasma desktop for daily use - - Container-management tooling (podman, docker, etc) - - Development tools (IDEs, etc) - - Onboarding requirements (such as Matrix clients, documentation, etc) - -This in turn also requires a more reliable installation experience, which is why we've been spending cycles on the installer project. -Internally we want the installer to support a variety of scriptable configurations in order to provide better test coverage for our updates -and boot management, whilst we also intend to **incrementally** improve the user experience. - -## 📅 Conclusion - -Alpha2 is coming, and will feature a preliminary version of the updated installer. The plan initially is for a simple "use whole disk" strategy -and will be followed up with supported for encrypted installations (post alpha2). Quite simply, some employers require their developers have encrypted installs, -and not supporting this configuration will restrict our ability to onboard developers. - -We're doing the Alpha2 as a baseline, from which all update testing, infrastructure adjustments, etc, will be measured against. This is the -point at which the project will formally "open the doors" for more developers and testers, with the explicit view that growth is -sustainable and manageable. Per the requirements set above, we're not going to "explode", but enable specific workflows that in turn will permit -us to scale out the project and inform direction. - -It's important to me that we're explicit in how we take our next steps. As stated earlier, pausing the world is not an option. -Continued delivery whilst enabling an incremental path to the future is the only way to ensure that we can deliver on our promises. -Contrary to a reboot, this is a solidification of our vision and a commitment to the future. Sure, new features will also appear in time, -such as easier packaging recipe formats, but no decision is being made that will require a rewrite of the project. However, these goals -require stating and explaining up front so that the high level vision is clear and provides the context for any short-term deliverables. -The last thing we need is to be in a position where we're "stuck" and unable to deliver, due to a lack of foresight. - -## 🙏 Thank you - -On behalf of the project, I'd like to thank everyone who has supported us so far. Whether that's through donations, code contributions, -or even having the time of day to try out the ISO, it's deeply appreciated. Being the underdog comes with its own set of challenges, -and of course the "indie cost", but it's also an opportunity to challenge the status quo and find new, more optimal ways, to do things. - -We're not just building "Serpent **OS**", we're building a set of tools to solve the problem of distributing an OS. Way down the line this -will be powerful enough to support a number of use cases, from appliances to desktops, and even servers. In short, we want to bring high -quality tooling to the masses, and we're doing it in a way that's sustainable and reliable. diff --git a/src/content/docs/blog/2025/02/06/hello-2025/pockets_empty.gif b/src/content/docs/blog/2025/02/06/hello-2025/pockets_empty.gif new file mode 100644 index 00000000..82cc4bb4 Binary files /dev/null and b/src/content/docs/blog/2025/02/06/hello-2025/pockets_empty.gif differ diff --git a/src/content/docs/blog/2025/02/14/evolve-this-os/index.md b/src/content/docs/blog/2025/02/14/evolve-this-os/index.md index 6e1db647..0c6351b8 100644 --- a/src/content/docs/blog/2025/02/14/evolve-this-os/index.md +++ b/src/content/docs/blog/2025/02/14/evolve-this-os/index.md @@ -9,52 +9,32 @@ copyright: "Copyright © 2025 aerynOS Developers" A long overdue, and highly requested update today. We're finally rebranding the project! - ## 🤔 Why the change? -The "Serpent OS" name was a quickly chosen name that stuck. Unfortunately "serpents" are often associated with -negative connotations, and we've had a lot of feedback over the years that the name was off-putting. Let's be completely -honest, it's not the most inviting name for a project. Who wants to trust a serpent? Generally speaking they're considered -dangerous at best. - - +The "Serpent OS" name was a quickly chosen name that stuck. Unfortunately "serpents" are often associated with negative connotations, and we've had a lot of feedback over the years that the name was off-putting. Let's be completely honest, it's not the most inviting name for a project. Who wants to trust a serpent? Generally speaking they're considered dangerous at best. -It's fair to say we've spent a long time in prototype and alpha phases. In order to move forward, our identity needs -to be more befitting of the project we're building. A move into the real world. This isn't a hobby project, it's a full -blown Linux distribution with serious technical underpinnings, achievements and goals. Getting the tone right from day dot -is critical. +It's fair to say we've spent a long time in prototype and alpha phases. In order to move forward, our identity needs to be more befitting of the project we're building. A move into the real world. This isn't a hobby project, it's a full blown Linux distribution with serious technical underpinnings, achievements and goals. Getting the tone right from day dot is critical. -Do note, there is no change to the internal core team, so we're aiming for full continuity and a seamless transition. -Had enough of these snakes on this plane? We have too. +Do note, there is no change to the internal core team, so we're aiming for full continuity and a seamless transition. Had enough of these snakes on this plane? We have too. ## ✨ What's the new name? Hah, it's not "Evolve OS", despite the clickbait title. Our new name is... drum roll please... + + ## 🌟 **AerynOS** -Pronounced like "Erin", it's a name that we feel is more befitting of the project. Pulling from multiple etymologies, it's a name that better describes the project -**now** versus the project that started as Serpent OS. +Pronounced like "Erin", it's a name that we feel is more befitting of the project. Pulling from multiple etymologies, it's a name that better describes the project **now** versus the project that started as Serpent OS. -"Aer" is rather obvious, Latin in origin. The phonetic "Erin" is a nod to the Irish roots of the project, and of course a home. -There are a number of reasons for the name, which will form part of the initial documentation on the new website. +"Aer" is rather obvious, Latin in origin. The phonetic "Erin" is a nod to the Irish roots of the project, and of course a home. There are a number of reasons for the name, which will form part of the initial documentation on the new website. -Our intent is to have a name that is more inviting, and more descriptive of the project's goals and aspirations. We're not -anti-establishment or anti-corporation - if anything, we're a statement that without the fiscal handcuffs, we can produce a -technically sound *and* user-friendly operating system. +Our intent is to have a name that is more inviting, and more descriptive of the project's goals and aspirations. We're not anti-establishment or anti-corporation - if anything, we're a statement that without the fiscal handcuffs, we can produce a technically sound *and* user-friendly operating system. ## ⏰ Timeline -Having already secured `AerynOS.com`, `AerynOS.dev`, plus the associated social media accounts, we're now in the process of -rebranding the project. For sheer irony, we'll make the final transition day the 17th of March, 2025. Yes, St. Patrick's Day. -The Matrix space has already been updated, and any areas where a new name isn't feasible will see almost immediate renames. +Having already secured `AerynOS.com`, `AerynOS.dev`, plus the associated social media accounts, we're now in the process of rebranding the project. For sheer irony, we'll make the final transition day the 17th of March, 2025. Yes, St. Patrick's Day. The Matrix space has already been updated, and any areas where a new name isn't feasible will see almost immediate renames. -A new GitHub organisation has been created, with our first priority there to create a proper organisation structure for once. -Over the coming weeks everything will be migrated over, and we'll mark the old organisation read-only/deprecated. Can't have folks -thinking Serpent OS was abandoned, after all. +A new GitHub organisation has been created, with our first priority there to create a proper organisation structure for once. Over the coming weeks everything will be migrated over, and we'll mark the old organisation read-only/deprecated. Can't have folks thinking Serpent OS was abandoned, after all. -Stick with us, we've got plenty of exciting news to come over the next days and weeks, as we embrace new transparency and -shed our old skin. Plus, we'll be starting to onboard for web properties, translations, artwork, and more. Cherry on the cake? -Admitting that we're actually building a *full* distribution will allow us to put relevant priorities in place to expose -Serpent OS.. I mean AerynOS .. functionality to the wider world. Get ready for a reboot of the grassroots, independent Linux -scene. We're here to stay. +Stick with us, we've got plenty of exciting news to come over the next days and weeks, as we embrace new transparency and shed our old skin. Plus, we'll be starting to onboard for web properties, translations, artwork, and more. Cherry on the cake? Admitting that we're actually building a *full* distribution will allow us to put relevant priorities in place to expose Serpent OS.. I mean AerynOS .. functionality to the wider world. Get ready for a reboot of the grassroots, independent Linux scene. We're here to stay. diff --git a/src/content/docs/blog/2025/03/25/hello-aerynos/Featured.png b/src/content/docs/blog/2025/03/25/hello-aerynos/Featured.png deleted file mode 100644 index e2613d4d..00000000 Binary files a/src/content/docs/blog/2025/03/25/hello-aerynos/Featured.png and /dev/null differ diff --git a/src/content/docs/blog/2025/03/25/hello-aerynos/Featured.webp b/src/content/docs/blog/2025/03/25/hello-aerynos/Featured.webp new file mode 100644 index 00000000..fb5c4fbd Binary files /dev/null and b/src/content/docs/blog/2025/03/25/hello-aerynos/Featured.webp differ diff --git a/src/content/docs/blog/2025/03/25/hello-aerynos/index.mdx b/src/content/docs/blog/2025/03/25/hello-aerynos/index.md similarity index 52% rename from src/content/docs/blog/2025/03/25/hello-aerynos/index.mdx rename to src/content/docs/blog/2025/03/25/hello-aerynos/index.md index cda1deca..1abac495 100644 --- a/src/content/docs/blog/2025/03/25/hello-aerynos/index.mdx +++ b/src/content/docs/blog/2025/03/25/hello-aerynos/index.md @@ -2,20 +2,17 @@ title: 🚀 Hello AerynOS date: 2025-03-25T15:35:21Z authors: [ikey] +cover: + alt: AerynOS 2025.03 installed with GNOME Desktop + image: ./Featured.webp tags: [news] license: "CC-BY-ND-4.0" copyright: "Copyright © 2025 aerynOS Developers" --- -import YouTube from "@/components/ui/YouTube.astro"; -import Kofi from "@/components/ui/Kofi.astro"; -Hello from AerynOS! ✨ As you recall from February, we set about rebranding Serpent OS into what you now see, -AerynOS. It's not been without challenges, and where possible we've ensured continuity. Additionally we silently -dropped a few ISOs, and can now take our first formal steps under our new identity. TLDR: the transition has been entirely -fluid and you only need to keep updating, no manual intervention is necessary! 🎉 +Hello from AerynOS! ✨ As you recall from February, we set about rebranding Serpent OS into what you now see, AerynOS. It's not been without challenges, and where possible we've ensured continuity. Additionally we silently dropped a few ISOs, and can now take our first formal steps under our new identity. TLDR: the transition has been entirely fluid and you only need to keep updating, no manual intervention is necessary! 🎉 -Progress has been steady but quiet lately, as due to unfortunate circumstances I've been working in bursts from my wifes -hospital bedside. +Progress has been steady but quiet lately, as due to unfortunate circumstances I've been working in bursts from my wifes hospital bedside. ## 💿 But first, an ISO drop! @@ -28,29 +25,29 @@ OK, lets get right to it. We've released AerynOS 2025.03, with the following shi - 🚀 Vulkan SDK 1.4.309.0 - 🛠️ LLVM 19.1.7 -We've also decoupled the internal tooling version from the ISO versions to make them a little bit more.. well.. *readable*, -for humans. +We've also decoupled the internal tooling version from the ISO versions to make them a little bit more.. well.. *readable*, for humans. Grab it now from the [download](/download) page. -![Installer preview](Featured.png) + +![Gnome Install](./Featured.webp "AerynOS 2025.03 installed with GNOME Desktop") ## 💰 Funding -I want to thank everyone for their support of the project since we got more transparent around goals. It's been a huge help and has -facilitated massive progress on the project! 🙏 AerynOS no longer feels like a hacked together PoC, but a very solid daily runner. Yes, -it's certainly alpha, and the installer leaves a lot to be desired. For daily use and updates? It's really quite something. +I want to thank everyone for their support of the project since we got more transparent around goals. It's been a huge help and has facilitated massive progress on the project! 🙏 AerynOS no longer feels like a hacked together PoC, but a very solid daily runner. Yes, it's certainly alpha, and the installer leaves a lot to be desired. For daily use and updates? It's really quite something. -Please note that I never realised the Ko-fi goals don't automatically reset, so the currently listed goal has been running for way more than -a month.. xD However, over a period of time, we did manage to achieve it! 🎉 I'll be resetting it shortly on a fixed date for each month ahead. +Please note that I never realised the Ko-fi goals don't automatically reset, so the currently listed goal has been running for way more than a month.. xD However, over a period of time, we did manage to achieve it! 🎉 I'll be resetting it shortly on a fixed date for each month ahead. - + ## 🔄 Rebranding, the sane way. -Rebranding can be more challenging than one expects. The easy stuff is out of the way, HTTP redirects and DNS trickery -to ensure old URLs continue to work without manual intervention. In the case of the repos, we've updated the scheme: +Rebranding can be more challenging than one expects. The easy stuff is out of the way, HTTP redirects and DNS trickery to ensure old URLs continue to work without manual intervention. In the case of the repos, we've updated the scheme: ### 🧪 Unstable presync repos @@ -64,16 +61,11 @@ to ensure old URLs continue to work without manual intervention. In the case of `docs.serpentos.com` -> `aerynos.dev` -Note the former scheme made *no sense* at all whereas now we take advantage of the `dotdev` for unstable work. A future release -of `moss` will automatically handle the transition on installs for the sake of consistency. +Note the former scheme made *no sense* at all whereas now we take advantage of the `dotdev` for unstable work. A future release of `moss` will automatically handle the transition on installs for the sake of consistency. ### 📊 Introducing `os-info` -Previously we had `moss` generate the `/usr/lib/os-release` file on demand using compiled in defaults, which was somewhat inflexible. -Now, we ship a JSON file (`/usr/lib/os-info.json`) containing a description of the OS, the composition of technologies, and the capabilities. -While `os-release` and `lsb_release` exist, they provide very primitive identification and metadata. `os-info` is designed to provide compatibility -with those formats while being far more expressive. Importantly, it also contains a mechanism for identifying the **former identities** of -an operating system: +Previously we had `moss` generate the `/usr/lib/os-release` file on demand using compiled in defaults, which was somewhat inflexible. Now, we ship a JSON file (`/usr/lib/os-info.json`) containing a description of the OS, the composition of technologies, and the capabilities. While `os-release` and `lsb_release` exist, they provide very primitive identification and metadata. `os-info` is designed to provide compatibility with those formats while being far more expressive. Importantly, it also contains a mechanism for identifying the **former identities** of an operating system: ```json ... @@ -90,31 +82,22 @@ an operating system: ... ``` -`moss` now utilises `os-info` to generate the `os-release` file, as well as to provide identification and history for our [blsforme](https://github.com/AerynOS/blsforme) crate -to manage each entry on the boot partition. This has allowed us to sync the branding on a per transaction basis, while still "owning" the -legacy branded transactions on the ESP too (i.e `/EFI/serpentos`) and ensure they are correctly garbage collected. Interestingly this means that for -a short period of time the boot menu will still show some of the older Serpent OS entries, allowing you to roll back to it. +`moss` now utilises `os-info` to generate the `os-release` file, as well as to provide identification and history for our [blsforme](https://github.com/AerynOS/blsforme) crate to manage each entry on the boot partition. This has allowed us to sync the branding on a per transaction basis, while still "owning" the legacy branded transactions on the ESP too (i.e `/EFI/serpentos`) and ensure they are correctly garbage collected. Interestingly this means that for a short period of time the boot menu will still show some of the older Serpent OS entries, allowing you to roll back to it. -Please visit the [os-info](https://github.com/AerynOS/os-info) repository for more details, the schema, etc. We're very keen to open collaboration and adoption of this project, envisaging -use cases in installers, welcome apps, and indeed even container introspection tooling. +Please visit the [os-info](https://github.com/AerynOS/os-info) repository for more details, the schema, etc. We're very keen to open collaboration and adoption of this project, envisaging use cases in installers, welcome apps, and indeed even container introspection tooling. ## 🔧 Installer progress -We've officially begun the revamp of [lichen-installer](https://github.com/AerynOS/lichen-installer) using a proper split between -the privileged backend and frontend. In order to permit a number of frontends and use cases in future, we're using `tonic` gRPC messages -to communicate, currently just on a UNIX domain socket. In future, we'll have use cases for TCP using WASM frontend. +We've officially begun the revamp of [lichen-installer](https://github.com/AerynOS/lichen-installer) using a proper split between the privileged backend and frontend. In order to permit a number of frontends and use cases in future, we're using `tonic` gRPC messages to communicate, currently just on a UNIX domain socket. In future, we'll have use cases for TCP using WASM frontend. -More importantly than that, we've now implemented the foundational aspects for [disks-rs](https://github.com/AerynOS/disks-rs). -Long story short - we're using provisioning strategies defined in custom `.kdl` files that are dynamically tested/validated against -an input storage pool to produce a working set of changes. Such as, create a new partition table and add some partitions to it, using -some constraints logic. +More importantly than that, we've now implemented the foundational aspects for [disks-rs](https://github.com/AerynOS/disks-rs). Long story short - we're using provisioning strategies defined in custom `.kdl` files that are dynamically tested/validated against +an input storage pool to produce a working set of changes. Such as, create a new partition table and add some partitions to it, using some constraints logic. - +:::note[Missing YouTube video] +There used to be a YouTube video at this part of the blog, however our YouTube channel was unfortunately deleted outside of our control and we no longer have access to the video. +::: -It is early days, but its quite awesome that we're able to automatically partition disks based on these strategy files! ✨ These will form the -backbone of `lichen`, allowing us to offer rich automatic storage strategies as well as manual partitioning options. Notable is the ability to -set the `volid` or `uuid` of a specific filesystem, and `partuuid`, etc. This in essence allows shallow reproduction of an installation/disk topology -using a configuration file. +It is early days, but its quite awesome that we're able to automatically partition disks based on these strategy files! ✨ These will form the backbone of `lichen`, allowing us to offer rich automatic storage strategies as well as manual partitioning options. Notable is the ability to set the `volid` or `uuid` of a specific filesystem, and `partuuid`, etc. This in essence allows shallow reproduction of an installation/disk topology using a configuration file. ## 🙏 Acknowledgments @@ -124,18 +107,20 @@ Special thanks to Cameron for his instrumental work in porting our websites to a Whether you've contributed code, reported bugs, tested releases, spread the word, or supported us financially - thank you. AerynOS is truly a community effort, and we're excited to continue this journey together. -As always, you can join our community on [Matrix](https://matrix.to/#/#aerynos:matrix.org) or contribute on [GitHub](https://github.com/AerynOS). +As always, you can join our community on Matrix[^1] or contribute on [GitHub](https://github.com/AerynOS). ## 🔄 Migration Guide For existing Serpent OS users, migrating to AerynOS is straightforward: 1. Simply run the following command in your terminal: + ```bash sudo moss sync -u ``` 2. Some branding changes may require a second `moss` operation to fully "take" effect (as they'll be using the new moss binary). After the initial sync, run one of these commands to complete the transition: + ```bash sudo moss sync # or @@ -143,16 +128,16 @@ For existing Serpent OS users, migrating to AerynOS is straightforward: ``` That's it! Your system will seamlessly transition to AerynOS while maintaining all your existing configurations and installed packages. + ## 🔮 Next up -We'll continue on the path we set at the start of the year. By and large we'll improve the core tooling to continue delivering a -better experience for users, developers, gamers, etc. +We'll continue on the path we set at the start of the year. By and large we'll improve the core tooling to continue delivering a better experience for users, developers, gamers, etc. -By the way, we *do* have `nvidia-open-gpu-kernel-modules` / `nvidia-graphics-driver` for the `linux-desktop` users wanting to game on their -shiny AerynOS installs. Our Steam package works great, and any feedback is appreciated on improving it (ie udev/controller stuff, 6.14 kernel is planned). +By the way, we *do* have `nvidia-open-gpu-kernel-modules` / `nvidia-graphics-driver` for the `linux-desktop` users wanting to game on their shiny AerynOS installs. Our Steam package works great, and any feedback is appreciated on improving it (ie udev/controller stuff, 6.14 kernel is planned). -- Shifting more towards the installer images, using [Slint](https://slint.dev) for the installer frontend, and dropping unnecessary -packages from the installer image. Live ISOs will of course be available, but our primary target is installer images. -- Integration of [upstreams-rs](https://github.com/AerynOS/upstreams-rs) and ABI tracking into the tooling in order to vastly lighten the load - for maintainers. +- Shifting more towards the installer images, using [Slint](https://slint.dev) for the installer frontend, and dropping unnecessary packages from the installer image. Live ISOs will of course be available, but our primary target is installer images. +- Integration of [upstreams-rs](https://github.com/AerynOS/upstreams-rs) and ABI tracking into the tooling in order to vastly lighten the load for maintainers. - Accelerated delivery of milestone ISOs. + +--- +[^1]: aerynOS now uses a [Zulip server](https://aerynos.zulipchat.com/) for community discussion. diff --git a/src/content/docs/blog/2025/03/29/aerynos-the-os-as-infrastructure/index.md b/src/content/docs/blog/2025/03/29/aerynos-the-os-as-infrastructure/index.md new file mode 100644 index 00000000..83c08da8 --- /dev/null +++ b/src/content/docs/blog/2025/03/29/aerynos-the-os-as-infrastructure/index.md @@ -0,0 +1,150 @@ +--- +title: "🏗️ AerynOS: The OS As Infrastructure" +date: 2025-03-30T02:35:00Z +authors: [ikey] +tags: [news] +license: "CC-BY-ND-4.0" +copyright: "Copyright © 2025 aerynOS Developers" +--- + +Taking a break from our usual release-oriented updates, I thought it was high time to dive into what AerynOS actually *is* and what sets it apart from other distros. Fair warning, this is a meaty, in depth post, and still only scratches the surface. If you're interested in the future of Linux distributions, read on. It may also help to visit our work-in-progress [documentation site](https://aerynos.dev) for more information. + +## 🧱 The OS As Infrastructure + +Firstly, AerynOS isn't Yet Another Linux Distribution. It's a platform, a foundation, and a set of tools engineered in accordance with a vision and *design* that just so happens to produce a Linux distribution. Were we to go back in time and build a design brief for AerynOS, the initial question might be: + +> What if the operating system itself behaved like modern infrastructure? + +AerynOS is the answer to that question. What if, instead of doing things the way they've always been done, we started from the ground up and produced a *well designed* system, rather than the traditional model of in-place mutation internal to a distribution? + +## 🏗️ The AerynOS Foundation + +Leveraging years of experience across the industry, including some notable pioneering projects such as Solus, Clear Linux, and others, we set out to build things the hard way, in order to make them easy.. eventually. + +### 🔧 Toolchain + +At the very core, is the decision to not be another GNU/Linux distribution. We default to the LLVM toolchain, using `libc++` and `compiler-rt` by default. This isn't just a case of "we like LLVM", but rather a strategic decision to leverage the superior diagnostics, and ensure correctness / portability of packages. Whilst in our very early stages a few years ago we did *trial* `musl`, we do default to `glibc` for compatibility reasons and its superior performance characteristics. The performance advantage of glibc over musl is well-documented, particularly for compute-intensive workloads and applications requiring optimal threading performance. We're here to build a working, usable system, for a multitude of use cases and verticals, and glibc provides the best balance of compatibility and performance. + +Note, we do still package `gcc` and it's trivial to enforce its use in a package recipe by setting the `toolchain` field in `stone.yaml` to `gnu`. + +Lastly, it's worth noting we build all packages for `x86_64-v2` as our minimum baseline, and perform targeted optimisation in our package recipes using the extensive tuning options configurable in `boulder`. + + + +### 📄 Statelessness + +Our packages are forbidden from containing any files outside of `/usr`. In order to enable this, packages and/or configurations are altered in AerynOS to ensure they can operate *in the absence of a user provided configuration*. This forces us to ensure sane-defaults are baked in at all levels, and eliminates dreaded 3-way merge conflicts on package updates. There are no conflicts, because everything in `/etc` and `/var` is yours. Likewise, `/usr` belongs exclusively to the system. + +This approach was coined and developed during Clear Linux and Solus days, and we're refining it further in AerynOS. Other projects have adopted *similar* approaches, termed "hermetic `/usr`". + +You might be wondering how files end up in `/etc` automatically. To this end, we support two forms of package triggers: + +#### 🔄 Transaction Triggers + +These triggers are run at the end of a transaction in an ephemeral container (linux namespace) and may affect the contents of the transaction-specific `/usr` tree. This is useful for interdependent packages that need to dynamically produce plugin registries, for example. + +#### 🖥️ System Triggers + +These triggers do not run in an isolated container, but instead are run in the context of the host system **after** the transaction has been successfully built and applied. It is these (minimally used) triggers that invoke `systemd-tmpfiles`, `systemd-sysusers`, etc. Even for these cases we take special care to ensure that our default configs are sane and that a rebuild is always possible. + +#### 👤 System Accounts + +Another important aspect is the handling of local system accounts. Traditionally these are snippets/shell scripts that invoke `useradd`, `groupadd`, etc. In AerynOS we default to systemd `userdb` for any users/groups that do not explicitly need groups that users would join. Thus, using drop-ins in `/usr/lib/userdb`, most accounts are defined and made available via NSS. We use this already in `gdm`, `polkit`, `colord`, and many others. + +For the other cases, we utilise `systemd-sysusers` using snippets in `/usr/lib/sysusers.d` to create the necessary system accounts. In time, when GNOME and other desktops (as well as shadow et al.) are more adapted to `userdb` and `systemd-homed`, we won't need to rely so much on `sysusers`. The goal of course is elimination of `/etc/passwd` and `/etc/group` entirely, facilitating quicker recovery, provisioning, etc. For now some packages will ship sysusers-based groups, ie `docker` group, etc. + +### ⚛️ Atomic Updates + +Every `moss` transaction is atomic. Very high level: we produce a new `usr` tree (rootfs essentially due to mandated `usr-merge`) very quickly using hardlinks from a deduplicated cache. Once successfully built and primed, we atomically swap the new tree into place. Effectively, the staged transaction is swapped with the real `/usr` using `renameat2` with `RENAME_EXCHANGE`. It either works or it doesn't. Nothing in between. + +### 🥾 Boot Management + +Leaning heavily on our [blsforme](https://github.com/AerynOS/blsforme) and [disks-rs](https://github.com/AerynOS/disks-rs) projects, we've taken a very different approach to boot management. As part of applying a transaction, we produce a Boot Loader Specification (BLS) entry for the new transaction. The tooling will discover the EFI System Partition (ESP), and mount it if necessary. It then synchronises the bootloader itself (`systemd-boot`), the relevant BLS entries, and the kernel/initramfs. Additionally it will automatically garbage collect older entries, currently retaining 5 of the last transactions. + +It's not just a case of copy/paste and hope for the best. We dynamically produce the root parameters for the kernel command line by directly reading the superblocks (natively, in Rust) of the devices all the way up the chain for the root filesystem. It means there's no configuration file anywhere in AerynOS containing your `root=` parameter, and we're even able to read LUKS2 encrypted devices to add the `rd.luks.uuid` parameter. + +If that wasn't cool enough, we *also* encode the moss transaction ID into the kernel command line. This is picked up during early boot in our initramfs, *before* `/sysroot` is pivoted to. Long story short it means that every kernel is correctly synchronised with the right rootfs, and that rollback is cheap, easy, and accessible directly from the boot menu. + +In addition, we also automatically support `XBOOTLDR` partitions. In the absence of `LoaderDevicePartUUID` EFI variable, we'll scan the GPT table itself relative to the rootfs to find the ESP, and we'll always scan GPT relative to ESP to find the `XBOOTLDR` partition. + +Long story short? No `/etc/default/grub`. In fact, if you wipe your ESP, moss can rebuild it from scratch. + +### 📦 The Stone Format + +At the heart of moss lies our `.stone` format. At a basic level, it's our binary package format. Using a **version-agnostic header**, we've ensured that we design for the future. We also version every payload within the stone, allowing us to refine and evolve the format over time. Currently we support 4 payloads in a single stone: + + - 📄 Content payload: A sequential blob of deduplicated data + - 🔍 Index payload: Contains offsets for the content payload, keyed by the XXH128 hash of the content + - 🗂️ Layout payload: Describes the intended filesystem layout when the stone is applied + - 📋 Metadata payload: Sequence of strongly typed, tagged metadata entries such as name, providers, etc. + +Right now we default to XXH128 for hashing, but Blake3 is on the horizon (and used in blsforme). We compress all payloads using Zstd, offering great decompression performance whilst still providing a decent compression ratio. + +The process for "installing" a `.stone` is quite different to other systems. + + - 📥 The stone is fetched according to the repository data + - 🧠 Index payload is unpacked in memory + - 🗜️ Content payload is decompressed in a single run to a temporary location + - 🔗 Using the index payload, the content payload is spliced into the content addressable storage (CAS) + - 📂 The layout payload is loaded and merged into the LayoutDB, keyed by the unique package ID (SHA256 for the entire `.stone`, per the repo) + - 📊 Using the same key, the metadata payload is loaded into the "InstallDB". + + Notice that at no point are we actually "installing" anything. We cache into the CAS, and store metadata and layout details. This is used when producing a transaction. Also note there are various safe guards, integrity checks and such in place. For example every payload contains a "CRC" in the payload header, verified on read. We actually use XXH64 for this. + + ### 🔁 What is a transaction? + +In AerynOS, a transaction is an entirely self-contained rootfs produced by `moss`. Right now we have to emulate imperative package management, by producing a new dependency graph seeded from the previous system state as recorded in the StateDB. Alterations are made and validated, and then the new DAG selects the packages to be included in the transaction. + +At a high level, the transaction is produced by: + + - 📊 Producing a valid dependency graph, seeded from the previous system state + - 🧮 Determine cache availability for each package + - 📥 Fetch and cache every missing `.stone` to produce the transaction + - 📂 Load all layouts for the stones by ID + - 🌐 Using our multi-layered vfs approach, we build an arena graph of the target filesystem, detecting conflicts ahead of time, whilst also precomputing reparented nodes (i.e symlink redirection) and produce an optimal iteration order for transaction application + - 🚶 Walking the vfs iterator to produce the new rootfs in a staging location, using `linkat`, `mkdirat`, etc, to optimally produce the new filesystem, linked from the deduplicated cache + - 📦 Binding an ephemeral container to the new rootfs, and running transaction triggers + - 💾 Recording the transaction in the StateDB. + +As mentioned above, once a successful transaction is produced, it is atomically swapped into place. If the transaction fails, no ill effects are observed on the system, and your work day continues as normal. In the event of a successful transaction, the system is updated and ready to go, along with system trigger execution and boot management updates. + +## 🔮 The Future + +All of this is just the start. It's taken a few years, and perhaps now it's clear **why**. We've not even covered our package build tool, `boulder`, or indeed our build infrastructure! With that said, lets skip forward a short while and see what's coming down the pipe. + +### 🌍 Beyond the distro model + +To reiterate, we **emulate** imperative package management. And to be honest, it's totally pointless. It actually introduces more bugs than it solves. Given that we produce a new rootfs for every transaction, we could just as easily produce an entirely new graph each time too. + +That's exactly what we're planning to do. In a similar vein to Gentoo or Nix, the intent is a global file to explicitly state the *desired* state of the system, whilst the tooling will simply fulfil it based on the moss plugin cascade. Whilst we have no intention of conflating state and configuration, we will in time extend this system model to support **variants** of packages in order to provide a kind of "slots" system, and to implement a richer, saner alternative to the infamous `update-alternatives`. + +Obvious candidates include mutually exclusive packages like `coreutils` and `uutils-coreutils` (a `gnu` variant weight, perhaps?) or coinstallables such as `clang`, ensuring a default vendor version whilst allowing overrides at the local install or indeed package recipe level. + +### 🔒 Immutability + +Often AerynOS is described as an immutable OS, and that's not strictly true. Granted, every transaction results in a new `/usr` tree, so local changes won't persist and recovery is immediately available. However, we're not immutable in the sense of read-only. + +Having produced a composition-first, developer&user friendly atomic update implementation, we want to take this further in order to also support immutability without compromise: no unnecessary reboots. To do so we'll implement something *similar* to [composefs](https://github.com/composefs/composefs), without the drawbacks. Using the same transaction driven approach we have now, we'll simply produce an `erofs` metadata image dynamically instead of the exploded filesystem tree. Utilising `trusted.overlay.redirect` xattr, and an `overlayfs` mount, we'll expose our own CAS through the layouts in `erofs`, and also support `fsverity` hashes. Icing on the cake? We'll leverage mount stacks to retain our composition-first, no unnecessary reboot ethos. + +### 📚 Versioned repositories + +We've touched on this a few times in our blog posts. Essentially the binary repository will be produced as a result of available build artefacts correlating to the state of our git repo manifest files (boulder proofs of build). The main repository index will link releases to moss/stone format versions, such that moss would only update to the latest release that it supports. This allows us to stagger breaking changes in a way that nothing breaks at all. + +## 📊 Current Status + + - 🚀 Actively shipping GNOME ISOs + - 🎮 Quite usable for gaming with NVIDIA drivers, Steam, Flatpak, etc + - 👥 Real users are already praising stability and innovation + - 🛠️ Focused heavily on `disks-rs` and `lichen-installer` to leverage disk strategy files (automatic provisioning described in KDL) + +## 🎉 Conclusion + +This isn't just another distro. It's us redefining distributing Linux. We've achieved a tremendous amount, having successfully integrated all of the above into a cohesive, singular whole. The amusing part of course being that the resulting system is "boring" - it just works. + +Now, we are alpha. We're not done, we're not without issues. That said, we're building the future and with your support, we'll get there even faster. This post has only scratched the surface of AerynOS, but if you like what we're doing, please do get involved or support us. + + diff --git a/src/content/docs/blog/2025/03/29/aerynos-the-os-as-infrastructure/index.mdx b/src/content/docs/blog/2025/03/29/aerynos-the-os-as-infrastructure/index.mdx deleted file mode 100644 index 62acc030..00000000 --- a/src/content/docs/blog/2025/03/29/aerynos-the-os-as-infrastructure/index.mdx +++ /dev/null @@ -1,218 +0,0 @@ ---- -title: "🏗️ AerynOS: The OS As Infrastructure" -date: 2025-03-30T02:35:00Z -authors: [ikey] -tags: [news] -license: "CC-BY-ND-4.0" -copyright: "Copyright © 2025 aerynOS Developers" ---- -import Kofi from "@/components/ui/Kofi.astro"; - -Taking a break from our usual release-oriented updates, I thought it was high time to dive into what AerynOS actually -*is* and what sets it apart from other distros. Fair warning, this is a meaty, in depth post, and still only scratches -the surface. If you're interested in the future of Linux distributions, read on. It may also help to visit our work-in-progress -[documentation site](https://aerynos.dev) for more information. - -## 🧱 The OS As Infrastructure - -Firstly, AerynOS isn't Yet Another Linux Distribution. It's a platform, a foundation, and a set of tools engineered in -accordance with a vision and *design* that just so happens to produce a Linux distribution. Were we to go back in time -and build a design brief for AerynOS, the initial question might be: - -> What if the operating system itself behaved like modern infrastructure? - -AerynOS is the answer to that question. What if, instead of doing things the way they've always been done, we started -from the ground up and produced a *well designed* system, rather than the traditional model of in-place mutation internal -to a distribution? - -## 🏗️ The AerynOS Foundation - -Leveraging years of experience across the industry, including some notable pioneering projects such as Solus, Clear Linux, -and others, we set out to build things the hard way, in order to make them easy.. eventually. - -### 🔧 Toolchain - -At the very core, is the decision to not be another GNU/Linux distribution. We default to the LLVM toolchain, using `libc++` -and `compiler-rt` by default. This isn't just a case of "we like LLVM", but rather a strategic decision to leverage the -superior diagnostics, and ensure correctness / portability of packages. Whilst in our very early stages a few years ago -we did *trial* `musl`, we do default to `glibc` for compatibility reasons and its superior performance characteristics. -The performance advantage of glibc over musl is well-documented, particularly for compute-intensive workloads and applications -requiring optimal threading performance. We're here to build a working, usable system, for a multitude of use cases and verticals, -and glibc provides the best balance of compatibility and performance. - -Note, we do still package `gcc` and it's trivial to enforce its use in a package recipe by setting the `toolchain` field -in `stone.yaml` to `gnu`. - -Lastly, it's worth noting we build all packages for `x86_64-v2` as our minimum baseline, and perform targeted optimisation -in our package recipes using the extensive tuning options configurable in `boulder`. - -### 📄 Statelessness - -Our packages are forbidden from containing any files outside of `/usr`. In order to enable this, packages and/or configurations -are altered in AerynOS to ensure they can operate *in the absence of a user provided configuration*. This forces us to ensure -sane-defaults are baked in at all levels, and eliminates dreaded 3-way merge conflicts on package updates. There are no conflicts, -because everything in `/etc` and `/var` is yours. Likewise, `/usr` belongs exclusively to the system. - -This approach was coined and developed during Clear Linux and Solus days, and we're refining it further in AerynOS. Other projects -have adopted *similar* approaches, termed "hermetic `/usr`". - -You might be wondering how files end up in `/etc` automatically. To this end, we support two forms of package triggers: - -#### 🔄 Transaction Triggers - -These triggers are run at the end of a transaction in an ephemeral container (linux namespace) and may affect the contents of the -transaction-specific `/usr` tree. This is useful for interdependent packages that need to dynamically produce plugin registries, for -example. - -#### 🖥️ System Triggers - -These triggers do not run in an isolated container, but instead are run in the context of the host system **after** the transaction -has been successfully built and applied. It is these (minimally used) triggers that invoke `systemd-tmpfiles`, `systemd-sysusers`, etc. -Even for these cases we take special care to ensure that our default configs are sane and that a rebuild is always possible. - -#### 👤 System Accounts - -Another important aspect is the handling of local system accounts. Traditionally these are snippets/shell scripts that invoke `useradd`, -`groupadd`, etc. In AerynOS we default to systemd `userdb` for any users/groups that do not explicitly need groups that users would -join. Thus, using drop-ins in `/usr/lib/userdb`, most accounts are defined and made available via NSS. We use this already in -`gdm`, `polkit`, `colord`, and many others. - -For the other cases, we utilise `systemd-sysusers` using snippets in `/usr/lib/sysusers.d` to create the necessary system accounts. -In time, when GNOME and other desktops (as well as shadow et al.) are more adapted to `userdb` and `systemd-homed`, we won't need to rely -so much on `sysusers`. The goal of course is elimination of `/etc/passwd` and `/etc/group` entirely, facilitating quicker recovery, -provisioning, etc. For now some packages will ship sysusers-based groups, ie `docker` group, etc. - -### ⚛️ Atomic Updates - -Every `moss` transaction is atomic. Very high level: we produce a new `usr` tree (rootfs essentially due to mandated `usr-merge`) very quickly -using hardlinks from a deduplicated cache. Once successfully built and primed, we atomically swap the new tree into place. Effectively, the -staged transaction is swapped with the real `/usr` using `renameat2` with `RENAME_EXCHANGE`. It either works or it doesn't. Nothing in between. - -### 🥾 Boot Management - -Leaning heavily on our [blsforme](https://github.com/AerynOS/blsforme) and [disks-rs](https://github.com/AerynOS/disks-rs) projects, we've taken -a very different approach to boot management. As part of applying a transaction, we produce a Boot Loader Specification (BLS) entry for the new -transaction. The tooling will discover the EFI System Partition (ESP), and mount it if necessary. It then synchronises the bootloader itself (`systemd-boot`), -the relevant BLS entries, and the kernel/initramfs. Additionally it will automatically garbage collect older entries, currently retaining 5 of the last -transactions. - -It's not just a case of copy/paste and hope for the best. We dynamically produce the root parameters for the kernel command line by directly reading the -superblocks (natively, in Rust) of the devices all the way up the chain for the root filesystem. It means there's no configuration file anywhere in AerynOS -containing your `root=` parameter, and we're even able to read LUKS2 encrypted devices to add the `rd.luks.uuid` parameter. - -If that wasn't cool enough, we *also* encode the moss transaction ID into the kernel command line. This is picked up during early boot -in our initramfs, *before* `/sysroot` is pivoted to. Long story short it means that every kernel is correctly synchronised with the right rootfs, -and that rollback is cheap, easy, and accessible directly from the boot menu. - -In addition, we also automatically support `XBOOTLDR` partitions. In the absence of `LoaderDevicePartUUID` EFI variable, we'll scan the -GPT table itself relative to the rootfs to find the ESP, and we'll always scan GPT relative to ESP to find the `XBOOTLDR` partition. - -Long story short? No `/etc/default/grub`. In fact, if you wipe your ESP, moss can rebuild it from scratch. - -### 📦 The Stone Format - -At the heart of moss lies our `.stone` format. At a basic level, it's our binary package format. Using a **version-agnostic header**, we've ensured -that we design for the future. We also version every payload within the stone, allowing us to refine and evolve the format over time. Currently we support -4 payloads in a single stone: - - - 📄 Content payload: A sequential blob of deduplicated data - - 🔍 Index payload: Contains offsets for the content payload, keyed by the XXH128 hash of the content - - 🗂️ Layout payload: Describes the intended filesystem layout when the stone is applied - - 📋 Metadata payload: Sequence of strongly typed, tagged metadata entries such as name, providers, etc. - -Right now we default to XXH128 for hashing, but Blake3 is on the horizon (and used in blsforme). We compress all -payloads using Zstd, offering great decompression performance whilst still providing a decent compression ratio. - -The process for "installing" a `.stone` is quite different to other systems. - - - 📥 The stone is fetched according to the repository data - - 🧠 Index payload is unpacked in memory - - 🗜️ Content payload is decompressed in a single run to a temporary location - - 🔗 Using the index payload, the content payload is spliced into the content addressable storage (CAS) - - 📂 The layout payload is loaded and merged into the LayoutDB, keyed by the unique package ID (SHA256 for the entire `.stone`, per the repo) - - 📊 Using the same key, the metadata payload is loaded into the "InstallDB". - - Notice that at no point are we actually "installing" anything. We cache into the CAS, and store metadata and layout details. This is used - when producing a transaction. Also note there are various safe guards, integrity checks and such in place. For example every payload contains - a "CRC" in the payload header, verified on read. We actually use XXH64 for this. - - ### 🔁 What is a transaction? - - In AerynOS, a transaction is an entirely self-contained rootfs produced by `moss`. Right now we have to emulate imperative package management, - by producing a new dependency graph seeded from the previous system state as recorded in the StateDB. Alterations are made and validated, and then - the new DAG selects the packages to be included in the transaction. - - At a high level, the transaction is produced by: - - - 📊 Producing a valid dependency graph, seeded from the previous system state - - 🧮 Determine cache availability for each package - - 📥 Fetch and cache every missing `.stone` to produce the transaction - - 📂 Load all layouts for the stones by ID - - 🌐 Using our multi-layered vfs approach, we build an arena graph of the target filesystem, detecting conflicts ahead of time, whilst also precomputing - reparented nodes (i.e symlink redirection) and produce an optimal iteration order for transaction application - - 🚶 Walking the vfs iterator to produce the new rootfs in a staging location, using `linkat`, `mkdirat`, etc, to optimally produce the - new filesystem, linked from the deduplicated cache - - 📦 Binding an ephemeral container to the new rootfs, and running transaction triggers - - 💾 Recording the transaction in the StateDB. - -As mentioned above, once a successful transaction is produced, it is atomically swapped into place. If the transaction fails, no ill effects are -observed on the system, and your work day continues as normal. In the event of a successful transaction, the system is updated and ready to go, -along with system trigger execution and boot management updates. - -## 🔮 The Future - -All of this is just the start. It's taken a few years, and perhaps now it's clear **why**. We've not even covered our package build tool, `boulder`, -or indeed our build infrastructure! With that said, lets skip forward a short while and see what's coming down the pipe. - -### 🌍 Beyond the distro model - -To reiterate, we **emulate** imperative package management. And to be honest, it's totally pointless. It actually introduces -more bugs than it solves. Given that we produce a new rootfs for every transaction, we could just as easily produce an entirely -new graph each time too. - -That's exactly what we're planning to do. In a similar vein to Gentoo or Nix, the intent is a global file to explicitly state -the *desired* state of the system, whilst the tooling will simply fulfil it based on the moss plugin cascade. Whilst we have -no intention of conflating state and configuration, we will in time extend this system model to support **variants** of packages -in order to provide a kind of "slots" system, and to implement a richer, saner alternative to the infamous `update-alternatives`. - -Obvious candidates include mutually exclusive packages like `coreutils` and `uutils-coreutils` (a `gnu` variant weight, perhaps?) or -coinstallables such as `clang`, ensuring a default vendor version whilst allowing overrides at the local install or indeed package recipe -level. - -### 🔒 Immutability - -Often AerynOS is described as an immutable OS, and that's not strictly true. Granted, every transaction results in a new `/usr` tree, -so local changes won't persist and recovery is immediately available. However, we're not immutable in the sense of read-only. - -Having produced a composition-first, developer&user friendly atomic update implementation, we want to take this further in order to also -support immutability without compromise: no unnecessary reboots. To do so we'll implement something *similar* to [composefs](https://github.com/composefs/composefs), -without the drawbacks. Using the same transaction driven approach we have now, we'll simply produce an `erofs` metadata image dynamically instead -of the exploded filesystem tree. Utilising `trusted.overlay.redirect` xattr, and an `overlayfs` mount, we'll expose our own CAS through the -layouts in `erofs`, and also support `fsverity` hashes. Icing on the cake? We'll leverage mount stacks to retain our composition-first, no -unnecessary reboot ethos. - -### 📚 Versioned repositories - -We've touched on this a few times in our blog posts. Essentially the binary repository will be produced as a result of available build artefacts -correlating to the state of our git repo manifest files (boulder proofs of build). The main repository index will link releases to moss/stone -format versions, such that moss would only update to the latest release that it supports. This allows us to stagger breaking changes in a way -that nothing breaks at all. - -## 📊 Current Status - - - 🚀 Actively shipping GNOME ISOs - - 🎮 Quite usable for gaming with NVIDIA drivers, Steam, Flatpak, etc - - 👥 Real users are already praising stability and innovation - - 🛠️ Focused heavily on `disks-rs` and `lichen-installer` to leverage disk strategy files (automatic provisioning described in KDL) - -## 🎉 Conclusion - -This isn't just another distro. It's us redefining distributing Linux. We've achieved a tremendous amount, having successfully integrated -all of the above into a cohesive, singular whole. The amusing part of course being that the resulting system is "boring" - it just works. - -Now, we are alpha. We're not done, we're not without issues. That said, we're building the future and with your support, we'll get there -even faster. This post has only scratched the surface of AerynOS, but if you like what we're doing, please do get involved or support us. - - - -Or see [other ways](/sponsor) in which you can support the project financially. diff --git a/src/content/docs/blog/2025/06/30/mid-year-update/index.md b/src/content/docs/blog/2025/06/30/mid-year-update/index.md index 76ce5d86..8b7bbfd4 100644 --- a/src/content/docs/blog/2025/06/30/mid-year-update/index.md +++ b/src/content/docs/blog/2025/06/30/mid-year-update/index.md @@ -27,6 +27,8 @@ For us, this cycle has been particularly demanding, as we have completed an MVP We have covered the reasons for this transition [previously](https://aerynos.com/blog/2023/09/06/oxidised-moss/), and it's fair to say that we are already feeling the benefits of easy and native reuse of code in our tooling repositories and welcoming more Rust contributors into our community. + + ### Why now? Earlier this year, our existing DLang build infrastructure started showing signs of instability and required more and more manual intervention to successfully land packages. @@ -48,7 +50,6 @@ Our [infra](https://github.com/AerynOS/infra) is comprised of the Summit, Avalan - We have some cool features planned in AerynOS that we envision will make package maintenance a lot easier to manage through smart use of automation. Until these features are implemented, however, maintaining the AerynOS repository will remain somewhat human resource intensive. This is the main reason why the repository is consciously being kept "small" for now, with us deliberately focusing on having packages that will help developers and contributors improve AerynOS, while still delivering a nice Daily Driver experience. @@ -113,7 +114,7 @@ Once this script has been sufficiently productized, the next time existing users The exact number will vary from system to system depending on how many other packages are installed from the repository but for context, on a base AerynOS GNOME install, this is around 500 packages. -In the meantime, we have created a manual guide on how to transition existing installs to the new repository in our GitHub Discussions forum [here](https://github.com/orgs/AerynOS/discussions/53). The process is fairly simple, but if you do have any issues transitioning manually, do get in touch via a comment under the GitHub Discussions [post](https://github.com/orgs/AerynOS/discussions/53) or via [Matrix](https://matrix.to/#/#aerynos:matrix.org). +In the meantime, we have created a manual guide on how to transition existing installs to the new repository in our GitHub Discussions forum [here](https://github.com/orgs/AerynOS/discussions/53). The process is fairly simple, but if you do have any issues transitioning manually, do get in touch via a comment under the GitHub Discussions [post](https://github.com/orgs/AerynOS/discussions/53) or via Matrix[^1]. ### Content Delivery Network for Packages and ISOs @@ -172,7 +173,7 @@ Alternatively, you can wait until the automatic transition script is functional If you want to engage with the team, feel free to drop by our GitHub [Discussions](https://github.com/orgs/AerynOS/discussions), raise issues across our various repositories or if you're interested in contributing, feel free to raise PRs where you think our code can be improved or where you want to submit recipes for our repo. -We also have our matrix space that you can access via this [link](https://matrix.to/#/#aerynos:matrix.org): +We also have our matrix[^1] space that you can access: - The Development room in particular is a great place for discussions around our code. - The General room is a great place to drop by and get to know the team. - The Packaging room is where you want to be if you're interested in building packages for yourself and/or submitting them to the repository. @@ -184,3 +185,6 @@ Concurrently to our work around the infrastructure re-write and repository rebui The team has been refactoring our existing Rust code, mainly focused on our os-tools (Moss and Boulder) and we are working on several additional improvements that we want to get over the finish line before our next ISO release. We will be sharing details of this work in upcoming blog posts over the next few weeks. + +--- +[^1]: aerynOS now uses a [Zulip server](https://aerynos.zulipchat.com/) for community discussion. diff --git a/src/content/docs/blog/2025/07/11/Development update: os-tools/index.md b/src/content/docs/blog/2025/07/11/Development update: os-tools/index.md index a89809e5..1f523e0e 100644 --- a/src/content/docs/blog/2025/07/11/Development update: os-tools/index.md +++ b/src/content/docs/blog/2025/07/11/Development update: os-tools/index.md @@ -25,17 +25,17 @@ Overall, this work will help users when they come across unexpected errors and a The JSON output feature, in contrast, is largely targeted at convenient machine parsing of structured output for automation and integration purposes, which we are banking on will come in handy for future development work currently in the planning stages. + ## Introducing NomadicCore -Before going any further, you may have noticed my name as one of the authors of our mid-year update blog post. If you hang around our Matrix rooms, you will likely already know me but I thought it prudent to provide a formal introduction. +Before going any further, you may have noticed my name as one of the authors of our mid-year update blog post. If you hang around our Matrix[^1] rooms, you will likely already know me but I thought it prudent to provide a formal introduction. -I first became aware of SerpentOS about three years ago but only joined the Matrix chat rooms in September 2023. I'm not actually a developer or have any coding experience, however, I am interested in open source projects and Linux distributions that can help me get the most out of my hardware. I liked what I saw with SerpentOS and over the course of 2024, started getting involved, trying to help out where I could. +I first became aware of SerpentOS about three years ago but only joined the Matrix[^1] chat rooms in September 2023. I'm not actually a developer or have any coding experience, however, I am interested in open source projects and Linux distributions that can help me get the most out of my hardware. I liked what I saw with SerpentOS and over the course of 2024, started getting involved, trying to help out where I could. Earlier this year, I ended up formally joining the team, around the time of the AerynOS rebrand, taking more of a support/communications role and providing feedback from an "average user" perspective of what I think might be important. -My focus will mainly be around working on our documentation, writing blog posts and engaging on our various social media platforms and Matrix rooms. I'm looking forward to getting stuck in and helping support a Linux distribution I want to use on my various devices. - +My focus will mainly be around working on our documentation, writing blog posts and engaging on our various social media platforms and Matrix[^1] rooms. I'm looking forward to getting stuck in and helping support a Linux distribution I want to use on my various devices. ## A deeper dive into our os-tools work @@ -43,7 +43,6 @@ Since their port to Rust, our os-tools have been working well enough. However, w The following subsections outline some of the os-tools work we have been doing throughout Q2. - ### Refactor work Both [tarkah](https://aerynos.com/blog/2023/12/19/end-of-year-summary/#introducing-tarkah) and new contributor, [Jonas Platte](https://github.com/jplatte), have been working on refactoring our existing codebase. To increase the available insight and diagnostic information, we have decided to align around the use of the `tracing` crate given that important parts of our code base are asynchronous, for which `tracing` is particularly suited. @@ -58,7 +57,6 @@ One aspect of managing our tooling is ensuring that our codebase remains up to d Part of this upgrade work also involved being able to [lock dependencies](https://github.com/AerynOS/os-tools/pull/504) for Rust packages as a way to ensure robustness of the Moss and Boulder builds we use in production. - ### Moss: Parallel blitting Long-time contributor Joey Riches identified a parallelization improvement in Moss's [blitting](https://aerynos.com/blog/2021/08/10/a-rolling-boulder-gathers-no-moss/#blitting) process which was merged after several months of local testing. @@ -79,7 +77,6 @@ While tarkah's [fix](https://github.com/AerynOS/os-tools/pull/514) to Moss was r Consequently, we synced this bug-fix to the Moss version in the old repository to ensure that users will be able to seamlessly upgrade to the new rolling `unstable` repository. - ### Moss: Add index output directory option As we were preparing the process for syncing the packages in our `volatile` build-server repository to our new downstream rolling `unstable` repository on our public-facing server, we ran into an issue with the existing `moss index` code path. @@ -96,14 +93,12 @@ This in turns made it possible to make sure that the new rolling `unstable` repo - volatile = https://build.aerynos.dev/volatile/x86_64/stone.index [10] - local = file:///home/ermo/.cache/local_repo/x86_64/stone.index [100] - ### Boulder: Fix up phase timing in end-of-build report When Boulder successfully completes a package build, it emits a report detailing how long each phase of the build process took. ermo noticed that the output was wrong when the time exceeded an hour. For example `136m` would be formatted as a rather silly-looking `1h76m` instead of `2h16m`. He fixed this in the following [commit](https://github.com/AerynOS/os-tools/pull/502). - ### Boulder: Improve cache hit rates when updating packages Boulder is designed to cache files and hash them as part of its build process. By hashing files, it uniquely identifies each file and stores this for later utilization. When a package is updated from one version to the next, where the package has files that have not changed, we have the opportunity to reuse the cache from a prior build (as long as the cache has not been purged for space saving). @@ -112,7 +107,6 @@ Given the way boulder previously cached files, in named directories based on the Reilly [implemented](https://github.com/AerynOS/os-tools/pull/483) a change to boulder so that our ccache entries will persist over version number updates improving the hit rate and therefore performance on boulder package builds. - ### Boulder: Tweak how we use sh-compatible shells For user-facing recipes, our recipe snippets are now always interpreted by bash. @@ -125,31 +119,26 @@ Having said that, there are certain packages that just expect and/or are hardcod This gives our tooling the "dash as /bin/sh" `./configure` speed improvements by default, yet allows packagers to still successfully invoke `./configure` _et al._ with bash, where doing so is necessary for the build to complete. - ### Boulder: Update build macros On reviewing Boulder's build macros, we found some low hanging fruit improvements to make to our cmake, ninja, and meson macros, which we [landed](https://github.com/AerynOS/os-tools/pull/451) back in April. Implementing and improving the various build macros available in AerynOS makes it easier and more convenient for packagers to package up applications with Boulder; either for their own personal use or for submission into the official repositories. - ### Boulder: Use bsdtar-static for unpacking At Reilly's initiative, we moved our decompression solution away from GNU tar to bsdtar-static. This [change](https://github.com/AerynOS/os-tools/pull/500) reduces the likelihood of compatibility issues and ensures Boulder package creation doesn't rely on a dynamically linked system library, thus making it more of a reliable solution. With this move, we have also [added](https://github.com/AerynOS/os-tools/pull/535) the ability to decompress tgz based source packages as part of the boulder build process. - ## Other notable work Some of the work we have done has been aimed more at how we use our os-tools rather than the os-tools themselves. - ### Update build triple and fix up ARM AArch targets With our recent transition from SerpentOS to AerynOS, we needed to update our build triple accordingly. This step has been [completed](https://github.com/AerynOS/os-tools/pull/450) in the background whilst allowing for seamless updates from older SerpentOS systems onto AerynOS based systems. This is part of a wider rebranding effort that is still on-going through our documentation site, repo READMEs and anywhere else we have an official presence. In this same area, whilst AerynOS currently only supports `x86_64` based devices, there is a desire to be able to target other system types longer term. One of our contributors has been experimenting with RISC-V so we have added [preliminary support](https://github.com/AerynOS/os-tools/pull/448) for this to aid their testing. Don't expect to see AerynOS on RISC-V any time soon, but it's great to see our distro becoming a sandbox for fun and experimenting on alternative systems. - ### Emul32 ELF Machine type migration During the port of boulder from DLang over to Rust, there was a change in how we expressed the ISA for packages built for the x86 emul32 architecture target. The old DLang version of Boulder expressed it as `x86` where the Rust `elf` crate expresses it as `386` (`EM_386` for those of you familiar with ELF parsing internals). @@ -160,7 +149,6 @@ The end goal was to flush out the packages containing references to the `x86` Em In the end, this worked out nicely for us. - ## Current and near-future os-tools focus We reviewed the open issues in our various GitHub repositories and the plethora of ideas we have for our tooling, and developed a high level set of milestones for our os-tools. For this milestone (os-tools alpha2), we want to focus on: @@ -181,6 +169,9 @@ As already covered in this blog, we have been reviewing all open issues and PRs We are open to and actively looking for contributors who might be interested in looking through our code and providing feedback. -If you would like to try your hand at contributing, look out for issues marked "good first issue" and get in touch on [Matrix](https://matrix.to/#/#aerynos:matrix.org). +If you would like to try your hand at contributing, look out for issues marked "good first issue" and get in touch on Matrix[^1]. We hope to see you there! + +--- +[^1]: aerynOS now uses a [Zulip server](https://aerynos.zulipchat.com/) for community discussion. diff --git a/src/content/docs/blog/2025/08/31/August 2025 project update/index.mdx b/src/content/docs/blog/2025/08/31/August 2025 project update/index.md similarity index 98% rename from src/content/docs/blog/2025/08/31/August 2025 project update/index.mdx rename to src/content/docs/blog/2025/08/31/August 2025 project update/index.md index c93051c3..5e142da1 100644 --- a/src/content/docs/blog/2025/08/31/August 2025 project update/index.mdx +++ b/src/content/docs/blog/2025/08/31/August 2025 project update/index.md @@ -33,6 +33,8 @@ Whilst not an exhaustive list, some of the top line repository updates include: In addition, we fixed a subtle issue with our PATH configuration that mostly affected our console logins. With this fix, we have made our login experience fully stateless. We have also enabled sulogin for a single-user root shell to diagnose and repair boot failures. + + ### Package sets AerynOS is transitioning to a package set model for core packages installed on a user's system. @@ -155,7 +157,7 @@ We will continue fleshing out our documentation in the coming months, with a spe Some of the feedback we have received is that documentation is fragmented and/or not yet created. This is a frustration we can remove through our documentation efforts and we have new contributors helping out in this aspect too. -We are always looking for more support so if you have any interest in getting involved with our documentation efforts, please feel free to reach out on Matrix and specifically engage with NomadicCore. +We are always looking for more support so if you have any interest in getting involved with our documentation efforts, please feel free to reach out on Matrix[^1] and specifically engage with NomadicCore. ### Next steps @@ -168,3 +170,6 @@ For the os-tooling, we are adding structured logging for better insight and repo ## Download AerynOS The link for our latest iso can be found at our [download](https://aerynos.com/download/) page. + +--- +[^1]: aerynOS now uses a [Zulip server](https://aerynos.zulipchat.com/) for community discussion. diff --git a/src/content/docs/blog/2025/09/30/September 2025 project update/index.mdx b/src/content/docs/blog/2025/09/30/September 2025 project update/index.md similarity index 90% rename from src/content/docs/blog/2025/09/30/September 2025 project update/index.mdx rename to src/content/docs/blog/2025/09/30/September 2025 project update/index.md index de8c3a60..3e9f082a 100644 --- a/src/content/docs/blog/2025/09/30/September 2025 project update/index.mdx +++ b/src/content/docs/blog/2025/09/30/September 2025 project update/index.md @@ -7,7 +7,6 @@ license: "CC-BY-ND-4.0" copyright: "Copyright © 2025 aerynOS Developers" --- - As we reach the end of September, it's time for another monthly update. For this month, we are deferring our technical blog post into October but wanted to provide wider project updates. ## Kernels @@ -32,25 +31,27 @@ We are now at the point where we are confident in calling our new infrastructure It cannot be overstated how much better the state of our infra is compared to the start of the year. In turn, this now allows us to focus our efforts in other areas of our technology stack to similarly work through bugs, implement features and generally refine our code base. + + ## Cosmic maintainership A couple of months ago we had a new contributor, [Bryan Hyland](https://github.com/bhh32), join us and begin working on the Cosmic stack. Bryan's contributions have been invaluable to the team and we thank him for his engagement. Unfortunately, due to personal reasons, he is having to take a temporary step back from the project. He has shared his methodology for providing updates to the Cosmic packages and we already have willing contributors from our community picking up the in-flight tasks. -We know he will be back once his circumstances improve, but in the meantime, if you are curious about supporting Cosmic on AerynOS, please do get in touch in our [Matrix packaging channel](https://matrix.to/#/#AerynOS-Packaging:matrix.org) as it's always good to have several people capable of maintaining each of our DE stacks. +We know he will be back once his circumstances improve, but in the meantime, if you are curious about supporting Cosmic on AerynOS, please do get in touch in our Matrix[^1] packaging channel as it's always good to have several people capable of maintaining each of our DE stacks. ## Web development One area in which we often receive feedback is our website. Given we are in an alpha state, it has not been our highest priority to improve it. We do have a couple of interested contributors wanting to work on our website to help improve the overall look and feel of the project, though. -Hence, if you are interested in website design and want to lend a hand, please feel free to join our [Matrix webdev channel](https://matrix.to/#/#AerynOS-web:matrix.org) as we will likely begin work on this area in parallel to our core development work. +Hence, if you are interested in website design and want to lend a hand, please feel free to join our Matrix[^1] webdev channel as we will likely begin work on this area in parallel to our core development work. ## Documentation Similarly to our web development work, we are seeing increasing numbers of contributors willing and able to help with our documentation. Outside of our core tooling development, this is arguably our second most important workstream and another area in which we often receive feedback. -If you are interested in supporting our documentation drive, feel free to join us in our [Matrix documentation channel](https://matrix.to/#/#aerynos-documentation:matrix.org). +If you are interested in supporting our documentation drive, feel free to join us in our Matrix[^1] documentation channel. ## Holidays @@ -69,3 +70,6 @@ We hope to release out next ISO in October, once: We are also aware of the planned release of KDE Plasma 6.5 on the 21st of October 2025. Similarly to Linux 6.17.x, we are not planning / trying to be first out of the gate with a large stack update. This will likely land at a later point after a few weeks / a point release or two. That's it for this monthly roundup. Thanks for reading, and we hope to see you around in our community. + +--- +[^1]: aerynOS now uses a [Zulip server](https://aerynos.zulipchat.com/) for community discussion. diff --git a/src/content/docs/blog/2025/10/31/Featured.png b/src/content/docs/blog/2025/10/31/Featured.png deleted file mode 100644 index 3461addf..00000000 Binary files a/src/content/docs/blog/2025/10/31/Featured.png and /dev/null differ diff --git a/src/content/docs/blog/2025/10/31/October 2025 project update/Featured.webp b/src/content/docs/blog/2025/10/31/October 2025 project update/Featured.webp new file mode 100644 index 00000000..28387c77 Binary files /dev/null and b/src/content/docs/blog/2025/10/31/October 2025 project update/Featured.webp differ diff --git a/src/content/docs/blog/2025/10/31/index.mdx b/src/content/docs/blog/2025/10/31/October 2025 project update/index.md similarity index 94% rename from src/content/docs/blog/2025/10/31/index.mdx rename to src/content/docs/blog/2025/10/31/October 2025 project update/index.md index 175f243f..784b2b11 100644 --- a/src/content/docs/blog/2025/10/31/index.mdx +++ b/src/content/docs/blog/2025/10/31/October 2025 project update/index.md @@ -2,13 +2,14 @@ title: "October 2025 project update" date: 2025-10-31T09:00:00Z authors: [nomadiccore, ermo] +cover: + alt: AerynOS 2025.10 installed with GNOME Desktop + image: ./Featured.webp tags: [news] license: "CC-BY-ND-4.0" copyright: "Copyright © 2025 aerynOS Developers" --- -import Kofi from "@/components/ui/Kofi.astro"; - We are firmly in the final quarter of 2025 and what a year it's been so far. In our last blog post at the end of September, we mentioned that we were delaying the release of our next ISO into October to give our Gnome 49 stack (and the wider extensions ecosystem) more time to mature. Coming into the final week of October, the team made the decision to transition from clang's libc++ to GNU libstdc++ and work through the associated rebuild requirements across our repository. @@ -21,6 +22,8 @@ In addition to the build infrastructure related boulder fixes, the team has also With that all said and done, we are ready with our 2025.10 GNOME Live ISO refresh for you to enjoy along with this blog post. + + ## Whats new in the distro Package / stack updates for this month include: @@ -136,7 +139,11 @@ Due to the cancellations mentioned above, we are currently down to €25 of new/ Rest assured however, that this will not impact the project's long term viability as we have the ability as a team to cover these costs. However, if you are able to provide any sort of donation, it would be greatly appreciated. - + As a side note, we have also moved away from taking donations in USD by default and switched to EUR as this is the currency we are primarily spending in for our project costs. This does not preclude us from accepting USD based donations however. @@ -144,7 +151,7 @@ As a side note, we have also moved away from taking donations in USD by default Whilst we provide a bootable ISO image with a live GNOME environment, its main purpose is to serve as a vehicle for our 'lichen' installer which is able to install all of our 'GNOME', 'KDE Plasma', 'Cosmic' and 'Terminal Only' versions of AerynOS. Given that 'lichen' is a net installer, it will always pull the latest packages from our repository at the time of installation and as such, it requires an active internet connection. This means there is less of a need for regular ISO updates, as we have carefully designed our installation routine and our pkgsets to not be tightly coupled to the packages that happened to be available at the time of the live ISO creation. -![Installer preview](Featured.png) +![Gnome Install](./Featured.webp "AerynOS 2025.10 installed with GNOME Desktop") That said, with the addition of some significant updates in the Gnome stack, we felt it was a good opportunity to land a new live ISO for you all to try out. The link for our 2025.10 ISO can be found at our [download](https://aerynos.com/download/) page. @@ -154,4 +161,7 @@ Going into November, the team will continue to execute on our development plans On a final note, we are progressing on the documentation front, and work is steadily trickling into our documentation site. Thank you to everyone who has contributed! -If you are interested in our work or generally want to hang out, the best place to join us is in our [Matrix space](https://matrix.to/#/#aerynos:matrix.org). Alternatively, our Forums can be found on [GitHub](https://github.com/orgs/AerynOS/discussions) +If you are interested in our work or generally want to hang out, the best place to join us is in our Matrix[^1] space. Alternatively, our Forums can be found on [GitHub](https://github.com/orgs/AerynOS/discussions) + +--- +[^1]: aerynOS now uses a [Zulip server](https://aerynos.zulipchat.com/) for community discussion. diff --git a/src/content/docs/blog/2025/11/04/Musings on Inode Watchers and Atomic Live Upgrades/index.mdx b/src/content/docs/blog/2025/11/04/Musings on Inode Watchers and Atomic Live Upgrades/index.md similarity index 90% rename from src/content/docs/blog/2025/11/04/Musings on Inode Watchers and Atomic Live Upgrades/index.mdx rename to src/content/docs/blog/2025/11/04/Musings on Inode Watchers and Atomic Live Upgrades/index.md index d9e0e8f6..7caf5b30 100644 --- a/src/content/docs/blog/2025/11/04/Musings on Inode Watchers and Atomic Live Upgrades/index.mdx +++ b/src/content/docs/blog/2025/11/04/Musings on Inode Watchers and Atomic Live Upgrades/index.md @@ -7,8 +7,6 @@ license: "CC-BY-ND-4.0" copyright: "Copyright © 2025 aerynOS Developers" --- -import { Aside } from '@astrojs/starlight/components'; - This blog post will focus on inode watcher applications as well as the difficulties they present for live atomic updates for our `moss` package manager. ### What is Moss? @@ -19,7 +17,9 @@ Although `moss` presents itself as a traditional package manager, under the hood When you install a package with moss, it actually downloads and extracts the package to a Content Addressable Store (CAS) in `/.moss/assets`. It then constructs a new virtual filesystem in memory comparing the current installed state to the new desired installed state containing the additional package. From the CAS it then constructs a `/usr` tree in `/.moss/root/staging` containing the additional package using hardlinks into the CAS. Then, using the `renameat2` Linux kernel syscall with the `RENAME_EXCHANGE` flag set, moss atomically promotes the current `/.moss/root/staging` tree to be the new `/usr` tree, and simultaneously demotes the current `/usr` tree back to being an inactive, numbered filesystem transaction (fstx) tree in `/.moss/root/`. Finally, moss then updates the bootloader configuration to reference the five newest numbered filesystem transactions for rollback purposes. - +:::note +This explanation leaves out a couple of details such post-install triggers and boot management. +::: Compared to other atomic distributions on the market, the ability to live-update without needing to reboot is an important usability requirement for us, such that the user experience remains friendly to downstream users. @@ -36,7 +36,9 @@ $ inotifywait -m /usr/bin/le-foo & In this case, if you hold an inode to a path that is deleted after the the `/usr` tree is atomically swapped as part of a moss transaction, you will continue to hold the inode to the file in the archived state e.g. `/.moss/root//bin/le-foo`. This is due to the fact that the underlying file in the CAS that was referenced from the previous `/usr` tree was not removed from the system; it still exists in the now archived previous `/usr` tree. - +:::note +Note that moss encodes all packaged files to start with an implicit `/usr` prefix. Hence the `/usr/bin/le-foo` file becomes `/.moss/root//bin/le-foo` in the archived fstx id directory. +::: For any running applications whose functionality depends on these inode watchers, it can leave the system in a weird state as the application has no real way to know that the "real path" has now changed from `/usr/` to `/.moss/root//`. @@ -61,7 +63,9 @@ In this example, once `/mnt` is unmounted it will unmount `my-image.img` and lea When combined with EROFS (extended read-only filesystem), we can construct a lightweight, metadata-only EROFS `/usr` tree image which then links into the underlying CAS to form the new `/usr` tree instead, then mount it beneath the currently running `/usr` tree lightweight EROFS image. Lastly, we can then lazy unmount the current running `/usr` image so the new image will apply in its place. As an additional benefit, this approach ensures that the `/usr` tree is also immutable whilst still remaining atomic. - +:::note +This does not protect the underlying files in `/.moss`, which are referenced from the EROFS metadata-image, from mutation. Protecting `/.moss` will require additional changes that are outside the scope of this blog post. +::: However, this approach has now been explored, and the fundamental problem remains that any running applications holding an inode to a changed file will simply not see any change. diff --git a/src/content/docs/blog/2025/12/09/November and early December Blog Post/Cosmic_Screenshot.png b/src/content/docs/blog/2025/12/09/November and early December Blog Post/Cosmic_Screenshot.png deleted file mode 100644 index d6ebc110..00000000 Binary files a/src/content/docs/blog/2025/12/09/November and early December Blog Post/Cosmic_Screenshot.png and /dev/null differ diff --git a/src/content/docs/blog/2025/12/09/November and early December Blog Post/Cosmic_Screenshot.webp b/src/content/docs/blog/2025/12/09/November and early December Blog Post/Cosmic_Screenshot.webp new file mode 100644 index 00000000..2c478885 Binary files /dev/null and b/src/content/docs/blog/2025/12/09/November and early December Blog Post/Cosmic_Screenshot.webp differ diff --git a/src/content/docs/blog/2025/12/09/November and early December Blog Post/Gnome_Screenshot.png b/src/content/docs/blog/2025/12/09/November and early December Blog Post/Gnome_Screenshot.png deleted file mode 100644 index a69e3732..00000000 Binary files a/src/content/docs/blog/2025/12/09/November and early December Blog Post/Gnome_Screenshot.png and /dev/null differ diff --git a/src/content/docs/blog/2025/12/09/November and early December Blog Post/Gnome_Screenshot.webp b/src/content/docs/blog/2025/12/09/November and early December Blog Post/Gnome_Screenshot.webp new file mode 100644 index 00000000..ffe5445a Binary files /dev/null and b/src/content/docs/blog/2025/12/09/November and early December Blog Post/Gnome_Screenshot.webp differ diff --git a/src/content/docs/blog/2025/12/09/November and early December Blog Post/KDE_Screenshot.png b/src/content/docs/blog/2025/12/09/November and early December Blog Post/KDE_Screenshot.png deleted file mode 100644 index 5184e8ff..00000000 Binary files a/src/content/docs/blog/2025/12/09/November and early December Blog Post/KDE_Screenshot.png and /dev/null differ diff --git a/src/content/docs/blog/2025/12/09/November and early December Blog Post/KDE_Screenshot.webp b/src/content/docs/blog/2025/12/09/November and early December Blog Post/KDE_Screenshot.webp new file mode 100644 index 00000000..d579918f Binary files /dev/null and b/src/content/docs/blog/2025/12/09/November and early December Blog Post/KDE_Screenshot.webp differ diff --git a/src/content/docs/blog/2025/12/09/November and early December Blog Post/index.mdx b/src/content/docs/blog/2025/12/09/November and early December Blog Post/index.md similarity index 84% rename from src/content/docs/blog/2025/12/09/November and early December Blog Post/index.mdx rename to src/content/docs/blog/2025/12/09/November and early December Blog Post/index.md index f7298543..600ee212 100644 --- a/src/content/docs/blog/2025/12/09/November and early December Blog Post/index.mdx +++ b/src/content/docs/blog/2025/12/09/November and early December Blog Post/index.md @@ -2,16 +2,14 @@ title: "November + early December 2025 project update" date: 2025-12-09T12:00:00Z authors: [nomadiccore, ermo] +cover: + alt: AerynOS 2025.12 installed with KDE Plasma Desktop + image: ./KDE_Screenshot.webp tags: [news] license: "CC-BY-ND-4.0" copyright: "Copyright © 2025 aerynOS Developers" --- -import Kofi from "@/components/ui/Kofi.astro"; -import { Aside } from "@astrojs/starlight/components"; - -# AerynOS: November + early December 2025 project update - As we near the end of the year, the team has been reflecting on how best to position itself going into 2026. Over the course of November and early December, the team has made a number of changes, some of which have been public facing which you may have already seen if you've been following our socials. On the public side, the two most visible changes has been our transition away from Matrix to Zulip for our instant messaging chat platform, and our transition to netcup for our server requirements. In the background, the team has also changed email hosting providers, however no one would really have noticed any difference there. @@ -22,6 +20,8 @@ On the infrastructure side of things, we have landed simple auto-pruning logic i Things are progressing nicely in terms of getting [phase 1 of our Versioned Repositories infra feature](https://github.com/orgs/AerynOS/discussions/117) ready for production use. It was designed to from the outset to mesh nicely with our Moss system-model feature. Internal testing on a private infra instance has been running smoothly with these new features so far. + + ## What’s new in the distro Package / stack updates for this iteration include: @@ -58,7 +58,7 @@ Package / stack updates for this iteration include: ### Cosmic -![Cosmic Install](Cosmic_Screenshot.png) +![Cosmic Install](./Cosmic_Screenshot.webp "AerynOS 2025.12 installed with Cosmic Desktop") Our Cosmic packaging team have automated much of the update process based around the more frequent git tags that the System76 team are now publishing for the Cosmic Beta phase. As such, we are more quickly able to push updates out to our volatile repository for eventual availability in our unstable repository. @@ -72,13 +72,13 @@ Additional details about Cosmic can be found on System76's [website](https://sys ### Gnome -![Gnome Install](Gnome_Screenshot.png) +![Gnome Install](./Gnome_Screenshot.webp "AerynOS 2025.12 installed with GNOME Desktop") As usual, there isn't really much to say wrt. GNOME. The team has updated it to 49.2 and it is running nicely as expected. ### KDE Plasma -![KDE Install](KDE_Screenshot.png) +![KDE Plasma Install](./KDE_Screenshot.webp "AerynOS 2025.12 installed with KDE Plasma Desktop") KDE Plasma has been updated to 6.5.4, KDE Frameworks to 6.20.0. @@ -172,24 +172,22 @@ The rest of december is dedicated to bringing phase 1 of the Versioned Repositor As mentioned in the introduction, these PRs are already in testing, and merely need some final bits and bobs wrapped up before they can be landed and put in production. - +- Automated try-builds denotes the process whereby the infrastructure discovers an update to the upstream source repository of a package, attempts to auto-update the recipe and then attempts to build the updated package recipe in question. +- We think this will be a useful tool for contributors as it will automate some of the packaging tedium related to simple package version updates. It will also help enable automated regression testing and build flag optimisation in a future workstream. +- Included under the multi-arch umbrella is our ability to target ARM, RISC-V, and different x86 architecture levels such as x86-64-v3 and v4. +::: ## Supporting the project @@ -199,4 +197,8 @@ Following the October blog post, we had a flurry of new donors whom we want to t If you are following along with our project and are in a position to support us, please consider donating via our Ko-fi page. - + diff --git a/src/content/docs/blog/2026/01/02/2025 in retrospect/Camp-Fire.jpg b/src/content/docs/blog/2026/01/02/2025 in retrospect/Camp-Fire.jpg deleted file mode 100644 index f18b192f..00000000 Binary files a/src/content/docs/blog/2026/01/02/2025 in retrospect/Camp-Fire.jpg and /dev/null differ diff --git a/src/content/docs/blog/2026/01/02/2025 in retrospect/Camp-Fire.webp b/src/content/docs/blog/2026/01/02/2025 in retrospect/Camp-Fire.webp new file mode 100644 index 00000000..caf59d67 Binary files /dev/null and b/src/content/docs/blog/2026/01/02/2025 in retrospect/Camp-Fire.webp differ diff --git a/src/content/docs/blog/2026/01/02/2025 in retrospect/index.md b/src/content/docs/blog/2026/01/02/2025 in retrospect/index.md index 9c398f67..63843ca8 100644 --- a/src/content/docs/blog/2026/01/02/2025 in retrospect/index.md +++ b/src/content/docs/blog/2026/01/02/2025 in retrospect/index.md @@ -2,6 +2,9 @@ title: "2025 in retrospect" date: 2026-01-02T23:00:00Z authors: [nomadiccore, ermo] +cover: + alt: People sitting around a camp fire in the snow. Signifies the team coming together to reflect on the year! + image: ./Camp-Fire.webp tags: [news] license: "CC-BY-ND-4.0" copyright: "Copyright © 2026 aerynOS Developers" @@ -11,8 +14,6 @@ copyright: "Copyright © 2026 aerynOS Developers" We know many people will be following along, waiting for our beta and/or stable releases but for now we want to take a look back at 2025 and summarize what we have delivered and how that is positioning us strongly for 2026. -![Camp Fire](Camp-Fire.jpg) - **The TL;DR summary:** - We changed our name from Serpent OS to AerynOS. Aeryn is a different form of Erin, which means Ireland in Irish Gaelic. @@ -27,6 +28,7 @@ We know many people will be following along, waiting for our beta and/or stable - We reworked our cloud hosting setup to deliver more build capabilities at lower cost base - We renewed our media strategy through increased blog posts and engagement across social media platforms + ## What's in a name @@ -66,7 +68,6 @@ We would like to stress that, as a project, we hold no ill will towards Ikey, an At the same time, we think the present blog post will serve to outline why, based on our ability to deliver up until now, we are confident in our ability to continue developing AerynOS and its tooling into the future. - ## Infrastructure & Tooling development Late March and April also saw the start of the porting effort of our infrastructure from DLang to Rust. The approach for this porting effort was to develop a so-called "Minimum Viable Product" (MVP) code base that could iteratively be improved. @@ -121,7 +122,6 @@ All of these features, particularly where users could potentially face issues wi } - ## Team organisation We kicked off the year with looking for support in 4 key areas: @@ -155,7 +155,6 @@ For now, we don't have any concrete plans to package up additional DEs or WMs fo In terms of future potential, it's worth mentioning that AerynOS is first and foremost a forward-looking, Wayland-aligned project, which means that Wayland session support is a minimum requirement for any future DE or WM additions to the repository. - ### Installer At the start of this year, Ikey shared initial screenshots of a GUI based installer that would eventually replace our text based installer as the primary installation method. With Ikey stepping back from the project, the development of this GUI installer took a back seat to allow the team to focus on higher-priority items such as delivering working infra. @@ -164,7 +163,6 @@ Recently, one of our new staff members, Bryan Hyland, began working in the backg To put it simply, the idea is to eventually use the newly developed system-model approach to define the packages that would be installed per DE / WM and also allow users to use their own custom system-model.kdl files as the basis of an installation. - ## Sponsorship With Ikey stepping away earlier this year, for a period of time, it was unclear whether he would be returning or not. As the months passed by, the prospect of his return grew less and less likely. Therefore, as covered earlier in this blog post, we are now in the latter stages of formally retiring Ikey from the project. @@ -187,18 +185,10 @@ Similarly, we are interested in and open to CDN sponsorship as a way of strength If you wish to discuss sponsorship details, please reach out to us at contact@aerynos.com. -
-Sponsor AerynOS + ## Going into 2026 diff --git a/src/content/docs/blog/2026/01/31/January 2026 project update/Cosmic_Screenshot.jpg b/src/content/docs/blog/2026/01/31/January 2026 project update/Cosmic_Screenshot.jpg deleted file mode 100644 index 597bf92d..00000000 Binary files a/src/content/docs/blog/2026/01/31/January 2026 project update/Cosmic_Screenshot.jpg and /dev/null differ diff --git a/src/content/docs/blog/2026/01/31/January 2026 project update/Cosmic_Screenshot.webp b/src/content/docs/blog/2026/01/31/January 2026 project update/Cosmic_Screenshot.webp new file mode 100644 index 00000000..d1c17b93 Binary files /dev/null and b/src/content/docs/blog/2026/01/31/January 2026 project update/Cosmic_Screenshot.webp differ diff --git a/src/content/docs/blog/2026/01/31/January 2026 project update/Fresh-start.jpg b/src/content/docs/blog/2026/01/31/January 2026 project update/Fresh-start.jpg deleted file mode 100644 index 8a5f5fdd..00000000 Binary files a/src/content/docs/blog/2026/01/31/January 2026 project update/Fresh-start.jpg and /dev/null differ diff --git a/src/content/docs/blog/2026/01/31/January 2026 project update/Fresh-start.webp b/src/content/docs/blog/2026/01/31/January 2026 project update/Fresh-start.webp new file mode 100644 index 00000000..313e0ef9 Binary files /dev/null and b/src/content/docs/blog/2026/01/31/January 2026 project update/Fresh-start.webp differ diff --git a/src/content/docs/blog/2026/01/31/January 2026 project update/Gnome_Screenshot.jpg b/src/content/docs/blog/2026/01/31/January 2026 project update/Gnome_Screenshot.jpg deleted file mode 100644 index 36ef17e0..00000000 Binary files a/src/content/docs/blog/2026/01/31/January 2026 project update/Gnome_Screenshot.jpg and /dev/null differ diff --git a/src/content/docs/blog/2026/01/31/January 2026 project update/Gnome_Screenshot.webp b/src/content/docs/blog/2026/01/31/January 2026 project update/Gnome_Screenshot.webp new file mode 100644 index 00000000..a209e461 Binary files /dev/null and b/src/content/docs/blog/2026/01/31/January 2026 project update/Gnome_Screenshot.webp differ diff --git a/src/content/docs/blog/2026/01/31/January 2026 project update/KDE_Screenshot.jpg b/src/content/docs/blog/2026/01/31/January 2026 project update/KDE_Screenshot.jpg deleted file mode 100644 index f1f6767e..00000000 Binary files a/src/content/docs/blog/2026/01/31/January 2026 project update/KDE_Screenshot.jpg and /dev/null differ diff --git a/src/content/docs/blog/2026/01/31/January 2026 project update/KDE_Screenshot.webp b/src/content/docs/blog/2026/01/31/January 2026 project update/KDE_Screenshot.webp new file mode 100644 index 00000000..cfda8358 Binary files /dev/null and b/src/content/docs/blog/2026/01/31/January 2026 project update/KDE_Screenshot.webp differ diff --git a/src/content/docs/blog/2026/01/31/January 2026 project update/index.md b/src/content/docs/blog/2026/01/31/January 2026 project update/index.md index ac8f4d97..622db181 100644 --- a/src/content/docs/blog/2026/01/31/January 2026 project update/index.md +++ b/src/content/docs/blog/2026/01/31/January 2026 project update/index.md @@ -2,16 +2,14 @@ title: "January 2026 project update" date: 2026-01-30T12:00:00Z authors: [nomadiccore, ermo] +cover: + alt: The sun shining through clouds over a mountainous valley. Signifies a fresh start for the new year! + image: ./Fresh-start.webp tags: [news] license: "CC-BY-ND-4.0" copyright: "Copyright © 2026 aerynOS Developers" --- - -# AerynOS: January 2026 project update - -![Fresh Start](Fresh-start.jpg) - Kicking off 2026, activity across the AerynOS project has been moving at pace. As covered in our 2025 retrospective, last year the team laid the groundwork for the project's future growth and development. Starting off 2026, the team is building upon this foundation to keep delivering progress across the project. From a core tooling perspective, we are preparing for the next phase of our Versioned Repository feature set. During January, we have been refining our existing codebase to make future improvements easier to implement. Once delivered, moss will be able to update itself seamlessly through what would otherwise have been breaking changes to its codebase. This functionality will be very important for our long term goals as a rolling release distro where you never have to reinstall to receive the latest updates. @@ -24,6 +22,8 @@ Additional progress has been made project-wide on our documentation and we have Our efforts seem to be received well, as we are seeing an influx of new members joining our [Zulip server](https://aerynos.zulipchat.com/) and getting involved with AerynOS, whether through general discussion or through packaging efforts. We have also seen a jump in our followers across our various social media accounts and generally a more positive reception of what AerynOS is trying to achieve in the Linux space. + + ## What’s new in the distro Package / stack updates for this iteration include: @@ -58,24 +58,23 @@ Package / stack updates for this iteration include: ... along with sundry additions and updates. - ## Desktop Updates ### Cosmic -![Cosmic Install](Cosmic_Screenshot.jpg) +![Cosmic Install](./Cosmic_Screenshot.webp "AerynOS 2026.01 installed with Cosmic Desktop") Given System76's move to a more regular release cycle for [Cosmic DE](https://system76.com/cosmic), we are able to land updates to our repository more frequently. This month, System76 landed Cosmic 1.0.3 with some key updates including support for rounded corners and window shadows across all applications and additional appearance settings being made available. -> #### Cosmic context -> AerynOS is following System76's "Epoch" git branch for Cosmic DE. This is where System76 stages new updates to the Cosmic ecosystem before publishing new point releases. As such, our Cosmic package is actually somewhere between 1.0.3 and 1.0.4. -> -> Given that Cosmic 1.0.4 has been released, it will be included in our repository shortly. It may already be there by the time you read this blog post. +:::note[Cosmic context] +AerynOS is following System76's "Epoch" git branch for Cosmic DE. This is where System76 stages new updates to the Cosmic ecosystem before publishing new point releases. As such, our Cosmic package is actually somewhere between 1.0.3 and 1.0.4. +Given that Cosmic 1.0.4 has been released, it will be included in our repository shortly. It may already be there by the time you read this blog post. +::: ### Gnome -![Gnome Install](Gnome_Screenshot.jpg) +![Gnome Install](./Gnome_Screenshot.webp "AerynOS 2026.01 installed with GNOME Desktop") This month sees the inclusion of [Gnome 49.3](https://discourse.gnome.org/t/gnome-49-3-released/33609) which is a stable bug-fix release with updates across the Gnome stack. @@ -89,7 +88,7 @@ Plus many more fixes. See the upstream [changelog](https://download.gnome.org/te ### KDE Plasma -![KDE Install](KDE_Screenshot.jpg) +![KDE Install](./KDE_Screenshot.webp "AerynOS 2026.01 installed with KDE Plamsa Desktop") KDE Plasma has been updated to [6.5.5](https://kde.org/announcements/plasma/6/6.5.5/), KDE Frameworks to 6.22.0 and KDE Gear to 25.12.1. @@ -128,12 +127,11 @@ As a reminder, we do not include MangoWC, Niri or Sway as options to install dir } +:::note[Why not use our system-model?] +Our new system-model approach can be used as a way of significantly simplifying the process of setting up a new AerynOS install by declaratively stating which packages you want installed on your system from a given repository (currently either Unstable or Volatile) and even being able to lock your environment to any given fixed-in-time stream update tag. -> #### Why not use our system-model? -> Our new system-model approach can be used as a way of significantly simplifying the process of setting up a new AerynOS install by declaratively stating which packages you want installed on your system from a given repository (currently either Unstable or Volatile) and even being able to lock your environment to any given fixed-in-time stream update tag. -> -> For more information about our system-model approach, please refer to the our previous [2025 in retrospect blog post](https://aerynos.com/blog/2026/01/2025-in-retrospect/#system-model) - +For more information about our system-model approach, please refer to the our previous [2025 in retrospect blog post](https://aerynos.com/blog/2026/01/2025-in-retrospect/#system-model) +::: ## Infrastructure and Tooling Updates @@ -157,7 +155,6 @@ This is useful when troubleshooting, as it allows us to diagnose whether there i This error handling workstream will continue in the background to further improve the quality of human readable error output in our tooling. - ## Wider Project Updates ### Website redesign @@ -206,7 +203,6 @@ As and when we do make the transition to Codeberg, we would also like to sponsor Sponsorship, where we are able, is another way we feel we can support the open source community and as our own sponsorship / income grows, we will be in a better position to contribute to the upstream projects on which we rely. - ## ISO refresh We are releasing our newest Alpha ISO, AerynOS 2026.01, which includes the updates we've worked on since the start of December, and which features the 6.18.7 kernel. @@ -215,7 +211,6 @@ As usual, this is a Live GNOME ISO that merely serves as a delivery vehicle for The link for our 2026.01 ISO can be found on our [download](/download/) page. - ## Next Steps We have spent the month of January taking stock of our future development direction, including short and medium term goals. @@ -232,7 +227,6 @@ Due to the work done during 2025 on the infrastructure code-base, enabling the s We're very, very excited about what this will enable us to do in the future! - ## Supporting the project As mentioned in our previous blog post, our sponsorship goals in 2026 will be to continue growing our recurring sponsorships to help recover the backlog of historic project costs, and also to build up towards future hardware investments such as an x86_64-v4 capable builder. @@ -243,18 +237,10 @@ Similarly, we are interested in and open to EU-based CDN sponsorship as a way of If you are following along with our project and are in a position to support us, please consider donating via our Ko-fi page. If you wish to discuss other sponsorship details, please reach out to us at contact@aerynos.com. -
-Sponsor AerynOS + ### Thank You! diff --git a/src/content/docs/blog/2026/02/28/February 2026 project update/Being-forged.jpg b/src/content/docs/blog/2026/02/28/February 2026 project update/Being-forged.jpg deleted file mode 100644 index 40e54e5c..00000000 Binary files a/src/content/docs/blog/2026/02/28/February 2026 project update/Being-forged.jpg and /dev/null differ diff --git a/src/content/docs/blog/2026/02/28/February 2026 project update/Being-forged.webp b/src/content/docs/blog/2026/02/28/February 2026 project update/Being-forged.webp new file mode 100644 index 00000000..4bf7d3ec Binary files /dev/null and b/src/content/docs/blog/2026/02/28/February 2026 project update/Being-forged.webp differ diff --git a/src/content/docs/blog/2026/02/28/February 2026 project update/Cosmic_Screenshot.png b/src/content/docs/blog/2026/02/28/February 2026 project update/Cosmic_Screenshot.png deleted file mode 100644 index e67837c7..00000000 Binary files a/src/content/docs/blog/2026/02/28/February 2026 project update/Cosmic_Screenshot.png and /dev/null differ diff --git a/src/content/docs/blog/2026/02/28/February 2026 project update/Cosmic_Screenshot.webp b/src/content/docs/blog/2026/02/28/February 2026 project update/Cosmic_Screenshot.webp new file mode 100644 index 00000000..62a5da10 Binary files /dev/null and b/src/content/docs/blog/2026/02/28/February 2026 project update/Cosmic_Screenshot.webp differ diff --git a/src/content/docs/blog/2026/02/28/February 2026 project update/Gnome_Screenshot.png b/src/content/docs/blog/2026/02/28/February 2026 project update/Gnome_Screenshot.png deleted file mode 100644 index 56fa3c0d..00000000 Binary files a/src/content/docs/blog/2026/02/28/February 2026 project update/Gnome_Screenshot.png and /dev/null differ diff --git a/src/content/docs/blog/2026/02/28/February 2026 project update/Gnome_Screenshot.webp b/src/content/docs/blog/2026/02/28/February 2026 project update/Gnome_Screenshot.webp new file mode 100644 index 00000000..8f8a2700 Binary files /dev/null and b/src/content/docs/blog/2026/02/28/February 2026 project update/Gnome_Screenshot.webp differ diff --git a/src/content/docs/blog/2026/02/28/February 2026 project update/KDE_Screenshot.png b/src/content/docs/blog/2026/02/28/February 2026 project update/KDE_Screenshot.png deleted file mode 100644 index 5eecbbf9..00000000 Binary files a/src/content/docs/blog/2026/02/28/February 2026 project update/KDE_Screenshot.png and /dev/null differ diff --git a/src/content/docs/blog/2026/02/28/February 2026 project update/KDE_Screenshot.webp b/src/content/docs/blog/2026/02/28/February 2026 project update/KDE_Screenshot.webp new file mode 100644 index 00000000..94eeae37 Binary files /dev/null and b/src/content/docs/blog/2026/02/28/February 2026 project update/KDE_Screenshot.webp differ diff --git a/src/content/docs/blog/2026/02/28/February 2026 project update/index.md b/src/content/docs/blog/2026/02/28/February 2026 project update/index.md index efd5ecb1..3f12282d 100644 --- a/src/content/docs/blog/2026/02/28/February 2026 project update/index.md +++ b/src/content/docs/blog/2026/02/28/February 2026 project update/index.md @@ -2,15 +2,14 @@ title: "February 2026 project update" date: 2026-02-28T12:00:00Z authors: [nomadiccore, ermo] +cover: + alt: A man forging metal, signifying aerynOS being forged on month at a time + image: ./Being-forged.webp tags: [news] license: "CC-BY-ND-4.0" copyright: "Copyright © 2026 aerynOS Developers" --- -# AerynOS: February 2026 project update - -![Being Forged](Being-forged.jpg) - February has been a busy month for the project with a lot of activity around our tooling and infrastructure. We have merged a number of smaller improvements, which have both led to an increase in useful features, correctness and maintainability. Moss has become significantly quicker in usage, boulder has seen improvements to help automate recipe creation & updates to a set standard, and our summit dashboard has seen improvements to better represent build queues live and dynamically in graphical form. The website re-design is progressing nicely. Last month we shared that we are using the [Hextra theme](https://github.com/imfing/hextra). Over the last month, we have coordinated with the Hextra developer around conformance to [WCAG 2.2 Level AA](https://www.w3.org/TR/WCAG22/) (Web Content Accessibility Guidelines) and have been really impressed with their rate of development in this area. @@ -21,12 +20,12 @@ We have also seen packagers becoming more active in helping flesh out the recipe During the last week of February, we have shared with our packaging community that we will likely be more focused on maintaining our existing set of packages and becoming more selective on adding new packages in the coming months, whilst our tooling capabilities mature to where they need to be for efficiently managing long term growth. + + ## What’s new in the distro Package / stack updates for this iteration include: - - - COSMIC 1.0.8 - GNOME 49.4 - KDE Frameworks 6.23.0 @@ -62,7 +61,7 @@ Package / stack updates for this iteration include: ### Cosmic -![Cosmic Install](Cosmic_Screenshot.png) +![Cosmic Install](./Cosmic_Screenshot.webp "AerynOS 2026.02 installed with Cosmic Desktop") With Cosmic Desktop landing frequent updates, the AerynOS team have landed on a nice rhythm for packaging and landing these point releases into our own repository shortly thereafter. @@ -78,7 +77,7 @@ If you do become aware of any issues, as with any other DE, you can report these ### Gnome -![Gnome Install](Gnome_Screenshot.png) +![Gnome Install](./Gnome_Screenshot.webp "AerynOS 2026.02 installed with GNOME") Similarly with the GNOME stack, the team are in a nice rhythm of packaging updates as upstream updates come through. @@ -96,7 +95,7 @@ Plus many more fixes. ### KDE Plasma -![KDE Install](KDE_Screenshot.png) +![KDE Install](./KDE_Screenshot.webp "AerynOS 2026.02 installed with KDE Plasma") KDE Plasma has been updated to [6.6.1](https://kde.org/announcements/plasma/6/6.6.1/), KDE Frameworks to 6.23.0 and KDE Gear to 25.12.2. With this update, Plasma Login Manager has been promoted to become the default KDE install option in lichen, with SDDM being the backup alternative. @@ -155,7 +154,6 @@ This dependency graph is live and dynamic so will update itself as it works thro } - ### Boulder `--verify` flag for manifest checks Boulder has gained the ability to verify newly rebuilt manifests against existing repo manifests. This enables us to fail builds where the rebuilt manifest does not match the existing manifest, indicating the recipe and its associated .stone package artefacts are out of date with respect to the current infrastructure or tooling. @@ -223,7 +221,6 @@ We see this as the benefit of open source projects and as part of our commitment If you are interested or experienced in website design and want to help us out, join our [Zulip](https://aerynos.zulipchat.com) server and get to know the team. - ### Documentation Work is continuing on the [documentation](https://aerynos.zulipchat.com/) site with a focus this month on our FAQ section. @@ -263,7 +260,6 @@ Additionally, one area Ko-fi is justifying their 5% fee is the engagement aspect At this stage, we believe our donors would prefer as much of their donation as possible to come through directly to the project, but will continue offering both options so as to enable supporters to make their own choice in this regard. - ## ISO refresh We are releasing our newest Alpha ISO, AerynOS 2026.02, which includes the updates we've worked on since the start of February, and which features the 6.18.15 kernel. @@ -308,7 +304,6 @@ This is the final step towards realising the core tenet of tooling-based "Instal When completed, this will in turn enable us to begin to tackle some long-standing issues in our tooling and our recipes repository, that we have until now been unable to rectify without significant disruption to installed systems. - ## Supporting the project Outside of financial donations through Stripe and Ko-fi mentioned above, we are always looking for people to get involved with development and packaging efforts and welcome anyone curious about AerynOS to join us in our Zulip server! @@ -317,18 +312,10 @@ If any hardware vendors are interested in sponsoring the project either financia If you wish to discuss other sponsorship details, please reach out to us at contact@aerynos.com. -
-Sponsor AerynOS + ## Thank You! diff --git a/src/content/docs/blog/2026/03/31/March 2026 project update/Gnome.png b/src/content/docs/blog/2026/03/31/March 2026 project update/Gnome.png deleted file mode 100644 index 3e93ef37..00000000 Binary files a/src/content/docs/blog/2026/03/31/March 2026 project update/Gnome.png and /dev/null differ diff --git a/src/content/docs/blog/2026/03/31/March 2026 project update/Gnome.webp b/src/content/docs/blog/2026/03/31/March 2026 project update/Gnome.webp new file mode 100644 index 00000000..644dc063 Binary files /dev/null and b/src/content/docs/blog/2026/03/31/March 2026 project update/Gnome.webp differ diff --git a/src/content/docs/blog/2026/03/31/March 2026 project update/KDE_Plasma.jpg b/src/content/docs/blog/2026/03/31/March 2026 project update/KDE_Plasma.jpg deleted file mode 100644 index a455d2f4..00000000 Binary files a/src/content/docs/blog/2026/03/31/March 2026 project update/KDE_Plasma.jpg and /dev/null differ diff --git a/src/content/docs/blog/2026/03/31/March 2026 project update/KDE_Plasma.webp b/src/content/docs/blog/2026/03/31/March 2026 project update/KDE_Plasma.webp new file mode 100644 index 00000000..bcc04de7 Binary files /dev/null and b/src/content/docs/blog/2026/03/31/March 2026 project update/KDE_Plasma.webp differ diff --git a/src/content/docs/blog/2026/03/31/March 2026 project update/climbing.jpg b/src/content/docs/blog/2026/03/31/March 2026 project update/climbing.jpg deleted file mode 100644 index d57acbd4..00000000 Binary files a/src/content/docs/blog/2026/03/31/March 2026 project update/climbing.jpg and /dev/null differ diff --git a/src/content/docs/blog/2026/03/31/March 2026 project update/climbing.webp b/src/content/docs/blog/2026/03/31/March 2026 project update/climbing.webp new file mode 100644 index 00000000..0be50581 Binary files /dev/null and b/src/content/docs/blog/2026/03/31/March 2026 project update/climbing.webp differ diff --git a/src/content/docs/blog/2026/03/31/March 2026 project update/framework.jpg b/src/content/docs/blog/2026/03/31/March 2026 project update/framework.jpg deleted file mode 100644 index 4eba1121..00000000 Binary files a/src/content/docs/blog/2026/03/31/March 2026 project update/framework.jpg and /dev/null differ diff --git a/src/content/docs/blog/2026/03/31/March 2026 project update/framework.webp b/src/content/docs/blog/2026/03/31/March 2026 project update/framework.webp new file mode 100644 index 00000000..c41a5d99 Binary files /dev/null and b/src/content/docs/blog/2026/03/31/March 2026 project update/framework.webp differ diff --git a/src/content/docs/blog/2026/03/31/March 2026 project update/index.md b/src/content/docs/blog/2026/03/31/March 2026 project update/index.md index 3e39a275..2b7728c3 100644 --- a/src/content/docs/blog/2026/03/31/March 2026 project update/index.md +++ b/src/content/docs/blog/2026/03/31/March 2026 project update/index.md @@ -2,15 +2,14 @@ title: "March 2026 project update" date: 2026-03-31T12:00:00Z authors: [nomadiccore, ermo] +cover: + alt: A man climbing up a mountain to signify AerynOS overcoming it's challenges + image: ./climbing.webp tags: [news] license: "CC-BY-ND-4.0" copyright: "Copyright © 2026 aerynOS Developers" --- -# AerynOS: March 2026 project update - -![Climbing](climbing.jpg) - Another month brings another project update. In some respects, March has felt somewhat quieter than usual for the project. However, in the background, things have been fairly busy on the development front, as we prepare for larger and more visible tooling updates to land in the months to come. On the packaging front, notable package & stack updates include GNOME 50, KDE Plasma 6.6.3, LLVM 22.1.1, Wayland 1.25.0, QT6.11.0, FFmpeg 8.1 and mesa 26.0.3 which have kept our builders busy with a significant number of package builds and rebuilds. The team mentioned last month that there is a soft-freeze in place for our repository, in terms of accepting new package recipes. This is not a full freeze as packages with little or no reverse dependency rebuild chains are still being accepted where we believe they will prove useful. @@ -19,6 +18,7 @@ The development work this month has centered around the features we offer in ter Lastly, we want to take a moment to thank Framework for their on-going hardware sponsorship of our project. This month, they have provided the project with an AMD AI 300 based Framework 16 that Joey Riches is now using as a dedicated device for AerynOS development. + ## What’s new in the distro @@ -54,14 +54,13 @@ Package / stack updates for this iteration include: - llvm 22.1.1 - Qt6 6.11.0 - ... along with sundry additions and updates. ## Desktop Updates ### GNOME -![Gnome Install](Gnome.png) +![A screenshot of the GNOME desktop environment on the latest AerynOS 2026.03 release](./Gnome.webp "AerynOS 2026.03 GNOME install") This month sees a significant update to the GNOME stack with the delivery of [GNOME 50](https://release.gnome.org/50/). We have not received any reports of any issues but if you do find any bugs, please report them back to us via [GitHub Issues](https://github.com/AerynOS/recipes/issues) or through our [Zulip](https://aerynos.zulipchat.com/) community. @@ -75,7 +74,7 @@ Plus many more features and fixes. ### KDE Plasma -![KDE Install](KDE_Plasma.jpg) +![A screenshot of the KDE Plasma desktop environment on the latest AerynOS 2026.03 release](./KDE_Plasma.webp "AerynOS 2026.03 KDE install") KDE Plasma has been updated to [6.6.3](https://kde.org/announcements/plasma/6/6.6.3/), KDE Frameworks to 6.24.0 and KDE Gear to 25.12.3. @@ -101,7 +100,6 @@ A few key updates this month include: On top of this, several packages have been rebuilt for better performance, support or defaults. The default terminal in our Sway package set has been swapped from alacritty to foot as this is the default terminal proposed by Sway itself. Elephant has had upstream support for moss included and this version has now been added to our own repository. Niri has seen a few upstream improvements backported into our repository also. - ## Infrastructure and Tooling Updates ### Use CDN for the default volatile stream @@ -155,7 +153,6 @@ We have built the control file using the KDL format as part of our wider transit To those wondering why we didn't just add a boulder build flag for this, the reason is that we have a few other future use cases in mind for this feature that are also projected to benefit from a control file, hence settling on the current design. - ## Wider Project Updates ### Recipe repository REUSE compliance @@ -168,10 +165,9 @@ For anyone doing local packaging work, you will need to ensure you update your l In addition, a new CI process has been created to check for this REUSE compliance for package recipes. This will ensure compliance with our licensing standards into the future rather than letting the files deviate over time again. - ### Framework hardware sponsorship -![Framework](framework.jpg) +![Framework](./framework.webp "AerynOS' sponsored Framework 16 laptop out and about in Wales") Back in August 2024, Framework provided the project (then SerpentOS) with an AMD Ryzen 7840u based Framework 13. At the time, this was helpful for Ikey (our project founder and former project lead) to have a separate and dedicated laptop for working on the project following the appropriate hardware enablement. @@ -183,7 +179,6 @@ Unfortunately, when Ikey stepped away from the project, we lost access to the Fr As a team, we like the repairable nature of Framework hardware and appreciate them supporting an up-and-coming distro such as ours! - ## ISO refresh We are releasing our newest Alpha ISO, AerynOS 2026.03, which includes the updates we've worked on since the start of March, and which features the 6.18.20 kernel. @@ -194,7 +189,6 @@ We did notice an issue in our 2026.02 ISO whereby it would not boot from Ventoy The link for our 2026.03 ISO can be found on our [download](/download/) page. - ## Next Steps ### Python upgrade and repository rebuild @@ -224,7 +218,6 @@ This month we have been steadily building towards phase2 of our Versioned Repos This will set us up nicely from an "install once, update forever" perspective, as we will always be able to add new features to our tooling without requiring fresh installs or manual upgrade interventions on user systems. - ## Supporting the project Outside of financial donations through Stripe and Ko-fi mentioned above, we are always looking for people to get involved with development and packaging efforts and welcome anyone curious about AerynOS to join us in our Zulip server! @@ -233,18 +226,10 @@ If any hardware vendors are interested in sponsoring the project either financia If you wish to discuss other sponsorship details, please reach out to us at contact@aerynos.com. -
-Sponsor AerynOS + ## Thank You! diff --git a/src/content/docs/blog/2026/04/30/Rebranding-Upgrading-and-Wallpapering/Cosmic.png b/src/content/docs/blog/2026/04/30/Rebranding-Upgrading-and-Wallpapering/Cosmic.png deleted file mode 100644 index d6e360d1..00000000 Binary files a/src/content/docs/blog/2026/04/30/Rebranding-Upgrading-and-Wallpapering/Cosmic.png and /dev/null differ diff --git a/src/content/docs/blog/2026/04/30/Rebranding-Upgrading-and-Wallpapering/Cosmic.webp b/src/content/docs/blog/2026/04/30/Rebranding-Upgrading-and-Wallpapering/Cosmic.webp new file mode 100644 index 00000000..4ed587ea Binary files /dev/null and b/src/content/docs/blog/2026/04/30/Rebranding-Upgrading-and-Wallpapering/Cosmic.webp differ diff --git a/src/content/docs/blog/2026/04/30/Rebranding-Upgrading-and-Wallpapering/Gnome.png b/src/content/docs/blog/2026/04/30/Rebranding-Upgrading-and-Wallpapering/Gnome.png deleted file mode 100644 index dc8bfa00..00000000 Binary files a/src/content/docs/blog/2026/04/30/Rebranding-Upgrading-and-Wallpapering/Gnome.png and /dev/null differ diff --git a/src/content/docs/blog/2026/04/30/Rebranding-Upgrading-and-Wallpapering/Gnome.webp b/src/content/docs/blog/2026/04/30/Rebranding-Upgrading-and-Wallpapering/Gnome.webp new file mode 100644 index 00000000..c3ca1667 Binary files /dev/null and b/src/content/docs/blog/2026/04/30/Rebranding-Upgrading-and-Wallpapering/Gnome.webp differ diff --git a/src/content/docs/blog/2026/04/30/Rebranding-Upgrading-and-Wallpapering/Logo.png b/src/content/docs/blog/2026/04/30/Rebranding-Upgrading-and-Wallpapering/Logo.png deleted file mode 100644 index 5b5b2f09..00000000 Binary files a/src/content/docs/blog/2026/04/30/Rebranding-Upgrading-and-Wallpapering/Logo.png and /dev/null differ diff --git a/src/content/docs/blog/2026/04/30/Rebranding-Upgrading-and-Wallpapering/boulder_cache_size.png b/src/content/docs/blog/2026/04/30/Rebranding-Upgrading-and-Wallpapering/boulder_cache_size.png deleted file mode 100644 index 10ed120e..00000000 Binary files a/src/content/docs/blog/2026/04/30/Rebranding-Upgrading-and-Wallpapering/boulder_cache_size.png and /dev/null differ diff --git a/src/content/docs/blog/2026/04/30/Rebranding-Upgrading-and-Wallpapering/boulder_cache_size.webp b/src/content/docs/blog/2026/04/30/Rebranding-Upgrading-and-Wallpapering/boulder_cache_size.webp new file mode 100644 index 00000000..f4f4730f Binary files /dev/null and b/src/content/docs/blog/2026/04/30/Rebranding-Upgrading-and-Wallpapering/boulder_cache_size.webp differ diff --git a/src/content/docs/blog/2026/04/30/Rebranding-Upgrading-and-Wallpapering/boulder_up.png b/src/content/docs/blog/2026/04/30/Rebranding-Upgrading-and-Wallpapering/boulder_up.png deleted file mode 100644 index c44e1e55..00000000 Binary files a/src/content/docs/blog/2026/04/30/Rebranding-Upgrading-and-Wallpapering/boulder_up.png and /dev/null differ diff --git a/src/content/docs/blog/2026/04/30/Rebranding-Upgrading-and-Wallpapering/boulder_up.webp b/src/content/docs/blog/2026/04/30/Rebranding-Upgrading-and-Wallpapering/boulder_up.webp new file mode 100644 index 00000000..89dcfdaa Binary files /dev/null and b/src/content/docs/blog/2026/04/30/Rebranding-Upgrading-and-Wallpapering/boulder_up.webp differ diff --git a/src/content/docs/blog/2026/04/30/Rebranding-Upgrading-and-Wallpapering/index.md b/src/content/docs/blog/2026/04/30/Rebranding-Upgrading-and-Wallpapering/index.md index 36fc3f05..75abfcaf 100644 --- a/src/content/docs/blog/2026/04/30/Rebranding-Upgrading-and-Wallpapering/index.md +++ b/src/content/docs/blog/2026/04/30/Rebranding-Upgrading-and-Wallpapering/index.md @@ -2,13 +2,14 @@ title: "Rebranding, Upgrading, and Wallpapering: AerynOS’ April glow-up!" date: 2026-04-30T12:00:00Z authors: [nomadiccore, ermo] +cover: + alt: Image captured by a drone camera of wintery snowy fields with a river meandering through the middle + image: ./winter.webp tags: [news] license: "CC-BY-ND-4.0" copyright: "Copyright © 2026 aerynOS Developers" --- -![Image captured by a drone camera of wintery snowy fields with a river meandering through the middle](winter.jpg) - No, this isn’t a suspicious phishing attempt! AerynOS has officially gone through a rebrand 🎉 Over the course of April, we rolled out a new logo and refreshed colour palette across the entire project. It took a bit of time to thread this through every corner of the OS and our web presence, but we’re happy to say it’s now fully in place. @@ -29,6 +30,7 @@ In addition, we’ve continued our [reuse compliance work](https://github.com/Ae Lastly, we have expanded our kernel configurations and now offer an LTS, stable and gaming kernel, though switching away from the stable kernel isn't yet a simple process. + ## What’s new in the distro @@ -71,21 +73,19 @@ Package / stack updates for this iteration include: ... along with sundry additions and updates. - ## Infrastructure and Tooling Updates ### boulder cache subcommands with clean and size options -![Screenshot showing boulder cache size command with output](boulder_cache_size.png) +![Screenshot showing boulder cache size command with output](./boulder_cache_size.webp "Terminal output of boulder cache size command") This month, Joey has added additional subcommands for boulder to calculate cache sizes (both for boulder and moss). Additionally, a `boulder cache clean` command was added that will delete the cache to help free up space on a user's system. This will be particularly helpful for our packagers who after building packages will have increasing amounts of space taken up by the boulder cache. The team itself frequently moves between our `unstable` and `volatile` repositories so having the ability to delete the moss cache will also be helpful to ensure predictable outcomes. - ### boulder recipe automation -![Screenshot of a terminal session after running `boulder recipe update` on the yq package recipe, resulting in the automatic update of the corresponding `stone.yaml` recipe file](boulder_up.png) +![Screenshot of a terminal session after running `boulder recipe update` on the yq package recipe, resulting in the automatic update of the corresponding `stone.yaml` recipe file](./boulder_up.webp "Terminal output of boulder recipe update command") Using our self built [`ent`](https://github.com/AerynOS/ent) tool (which integrates with [Anitya](https://release-monitoring.org/) for release monitoring), we’ve taken another step toward automating package maintenance by combining it with our `boulder recipe update` command. When invoking this command, boulder will: @@ -100,16 +100,14 @@ There’s also JSON output support, which will be useful as we expand our automa For simple package updates, this helps consolidate some of our mundane workflow steps and help make packaging a more streamlined affair on AerynOS, letting packagers focus on logic and problem solving, not administrative minutiae. - ### Faster `moss state remove` -![Screenshot of a terminal session after running `sudo moss state remove 56-77`, resulting in a confirmation prompt and log of the deleted state directories](moss_state_remove.png) +![Screenshot of a terminal session after running `sudo moss state remove 56-77`, resulting in a confirmation prompt and log of the deleted state directories](./moss_state_remove.webp "Terminal output of moss state remove command") State removal is now faster thanks to parallelization, and it finally provides real-time feedback via a progress bar. Behind the scenes, moss intelligently handles our deduplicated CAS storage, which is why removal order might look a little unusual, it’s working out which files are safe to delete across shared states. - ### Improved `moss search` A new contributor, *otherJL0*, has made some great improvements to our `moss search` command: @@ -120,7 +118,6 @@ A new contributor, *otherJL0*, has made some great improvements to our `moss sea This is especially useful for packaging workflows as our packagers can get a better understanding of which packages provide a given binary or library. - ### Continuation of our Versioned Repository feature set We’ve continued work on phase 2 of our Versioned Repositories feature, and a [draft PR](https://github.com/AerynOS/os-tools/pull/765) with the basic workflow and new configuration format has been opened for moss. The vessel repository manager companion work has been mapped out, but no PR has yet been opened for this. @@ -131,7 +128,6 @@ In practical terms, this moves us closer to a true install-once, update-forever We hope to land the phase 2 related feature PRs in the near future. - ## Wider Project Updates ### New Branding: A Fresh Look for AerynOS @@ -146,10 +142,9 @@ Design-wise, we’ve divided the three points of the triquetra into two colours: We’re really excited about this fresh new look, and the fact that it’s something the whole community helped us shape. We’ve spent the month rolling it out across the operating system and our web presence, and we’re happy with how it all ties together. - ### New Wallpapers: Bringing Nature to Your Desktop -![Screenshot of a newly installed virtual machine instance of AerynOS, Gnome edition showing the about this system page with the new default AerynOS wallpaper](Gnome.png) +![Screenshot of a newly installed virtual machine instance of AerynOS, Gnome edition showing the about this system page with the new default AerynOS wallpaper](./Gnome.webp "AerynOS 2026.05 installed on GNOME with our new custom wallpaper") Next to the new logomark, we’ve also worked with community member Gabriel Janich (aka *ziegenmelker5*) to bring some stunning new wallpapers to AerynOS. Gabriel has an amazing collection of photos, and choosing just a handful to feature was no easy task! @@ -157,8 +152,7 @@ In the end, we selected seven that we felt best represented the spirit of AerynO We hope these new wallpapers, paired with the fresh logo, help make your AerynOS experience even more enjoyable as you dive into the project in the coming days and months. -![Screenshot of a newly installed virtual machine instance of AerynOS, Cosmic edition showing the about this system page with the new default AerynOS wallpaper](Cosmic.png) - +![Screenshot of a newly installed virtual machine instance of AerynOS, Cosmic edition showing the about this system page with the new default AerynOS wallpaper](./Cosmic.webp "AerynOS 2026.05 installed on Cosmic with our new custom wallpaper") ### Python stack upgrade @@ -166,7 +160,6 @@ We mentioned last month that we planned a Python stack upgrade. Due to diligent Our python stack isn't currently very large (only around 200 packages) which also played a role in the fairly seamless nature of this update. From what we can tell, the updated has enabled our early adopters to continue using Python packages without any regressions. - ### Kernel updates We have packaged two alternative kernel options, though we have yet to finish the supporting work required to let users easily switch between them: @@ -177,7 +170,6 @@ We have packaged two alternative kernel options, though we have yet to finish th Switching away from our linux-stable kernel to either of the other two alternative options isn’t a smooth process just yet. Improving that experience is on our roadmap. - ## ISO refresh We are releasing our newest Alpha ISO, AerynOS 2026.05, which includes the updates we've worked on since the start of April, and which features the 7.0.2 linux-stable kernel. @@ -198,7 +190,6 @@ That said -- and assuming we succeed in landing the Versioned Repos, phase2 feat In parallel to that, we hope to spend some time getting our systemd-preset story straight from a packaging perspective, which will give us the ability to enable services as a packaging operation. This will be especially useful when leveraged via our declarative system-model capabilities. - ## Supporting the project Over the last year, the project has been through a significant period of change. As detailed in our [October 2025 blog post](https://aerynos.com/blog/2025/10/31/#donations), we had to update our sponsorship accounts to receive future sponsorship funds once it became clear our previous project leader had permanently stepped away from the project. @@ -214,23 +205,14 @@ Ideally we would like to grow our monthly income (and therefore surplus). Doing 3. Consider purchasing hardware for compatibility testing 4. Fund future initiatives for the betterment of the project -
-Sponsor AerynOS + If you wish to discuss other sponsorship opportunities, such as hosting or hardware sponsorship, please reach out to us at [contact@aerynos.com](mailto:contact@aerynos.com). - ## Thank You! We are very grateful for your support, be it financial or via project contributions in the form of carefully written bug reports, code contributions, design contributions, documentation updates, general feedback, package updates and overall enthusiasm around the project. diff --git a/src/content/docs/blog/2026/04/30/Rebranding-Upgrading-and-Wallpapering/moss_state_remove.png b/src/content/docs/blog/2026/04/30/Rebranding-Upgrading-and-Wallpapering/moss_state_remove.png deleted file mode 100644 index a2805566..00000000 Binary files a/src/content/docs/blog/2026/04/30/Rebranding-Upgrading-and-Wallpapering/moss_state_remove.png and /dev/null differ diff --git a/src/content/docs/blog/2026/04/30/Rebranding-Upgrading-and-Wallpapering/moss_state_remove.webp b/src/content/docs/blog/2026/04/30/Rebranding-Upgrading-and-Wallpapering/moss_state_remove.webp new file mode 100644 index 00000000..f4ade0a4 Binary files /dev/null and b/src/content/docs/blog/2026/04/30/Rebranding-Upgrading-and-Wallpapering/moss_state_remove.webp differ diff --git a/src/content/docs/blog/2026/04/30/Rebranding-Upgrading-and-Wallpapering/winter.jpg b/src/content/docs/blog/2026/04/30/Rebranding-Upgrading-and-Wallpapering/winter.jpg deleted file mode 100644 index 0aa3970a..00000000 Binary files a/src/content/docs/blog/2026/04/30/Rebranding-Upgrading-and-Wallpapering/winter.jpg and /dev/null differ diff --git a/src/content/docs/blog/2026/04/30/Rebranding-Upgrading-and-Wallpapering/winter.webp b/src/content/docs/blog/2026/04/30/Rebranding-Upgrading-and-Wallpapering/winter.webp new file mode 100644 index 00000000..13ec4f73 Binary files /dev/null and b/src/content/docs/blog/2026/04/30/Rebranding-Upgrading-and-Wallpapering/winter.webp differ diff --git a/src/content/docs/blog/2026/05/03/A post we never want to have to make!/index.md b/src/content/docs/blog/2026/05/03/A post we never want to have to make!/index.md index ede1db87..c2a96325 100644 --- a/src/content/docs/blog/2026/05/03/A post we never want to have to make!/index.md +++ b/src/content/docs/blog/2026/05/03/A post we never want to have to make!/index.md @@ -2,13 +2,14 @@ title: "A post we never want to have to make!" date: 2026-05-03T23:00:00Z authors: [nomadiccore, ermo] +cover: + alt: An image showing the word "Sorry" with three exclamation marks + image: ./sorry.jpg tags: [news] license: "CC-BY-ND-4.0" copyright: "Copyright © 2026 aerynOS Developers" --- -![An image showing the word "Sorry" with three exclamation marks](sorry.jpg) - This is the post we never want to make, where we made errors with our previous ISO release and are having to take steps to fix the issue, including releasing an updated ISO. Over the course of April, the team worked on systemd-presets as a workstream we didn't actually talk about in our April blog post. We thought that all the work had been completed in such a fashion that users wouldn't see any discernible difference whilst allowing us to move closer to "best-practice" approaches. @@ -24,6 +25,7 @@ Thanks to user feedback, we were able to narrow down, identify and resolve the i Following additional testing with a wider audience on our Zulip server, we have updated the links on our download page to our new ISO so any users downloading the latest ISO won't see these issues. + ## How we view the project and our roles in creating it @@ -35,7 +37,6 @@ There is an ethos within the team that we take seriously our craft, that being t In retrospect, we're glad that in the year that we have collectively had stewardship of the project, this is the first and only time we have had to rush out a new ISO to fix an issue. We hope to not have this occur again in the future. - ## NVIDIA driver issues for certain GPUs In the background, we have been offering a "best effort" approach to supporting NVIDIA GPUs. This is primarily because nobody on the core team are actually using NVIDIA GPUs, and because NVIDIA's approach to open source leaves a lot to be desired from a package- and distro-maintenance point of view. @@ -44,7 +45,6 @@ For an alpha tag distribution that is primarily focused on dogfooding itself, NV That said, Reilly identified an issue with our build ordering that caused the NVIDIA module to fail to work for GPUs that require GSP firmware. With this knowledge, we have implemented a manual fix for now. The underlying issue was already known to the team, we just hadn't caught that it presented an issue for this particular case. Fixing that issue in our infrastructure tooling is therefore moving up on our list of priorities. - ## Why have monthly ISOs anyway?! The issue with new installs only presented because of our new ISO release. Had users installed AerynOS from any of our previous ISOs, the distribution would have installed without issue. Last year, we made a decision to move to a monthly ISO cadence as part of a wider "hearts and minds" effort, to demonstrate that AerynOS is in good hands, and that we are able to consistently deliver progress at a time when there was uncertainty of whether the project would be able to survive during Ikey's initial (and, as it turned out, eventually permanent) absence from the project. @@ -91,7 +91,6 @@ Added: - oxidize: A tool for atomically changing themes in supported WMs - yt-dlp 2026.03.17 - ## oxidize Window Manager theming
@@ -123,7 +122,6 @@ This work builds on top of the great work Christian and a number of other dedica Of course, for those wishing to configure their Window Manager experience from scratch, this option is of course available to you as well. - ## ISO refresh With this blog post, we already have a new 2026.05.2 ISO available on our [download](/download/) page. @@ -140,30 +138,20 @@ That said -- and assuming we succeed in landing the Versioned Repos, phase2 feat In parallel to that, we hope to spend some time getting our systemd-preset story straight from a packaging perspective, which will give us the ability to enable services as a packaging operation. This will be especially useful when leveraged via our declarative system-model capabilities. - ## Supporting the project Outside of financial donations through Stripe and Ko-fi mentioned above, we are always looking for people to get involved with development and packaging efforts and welcome anyone curious about AerynOS to join us in our Zulip server! If any hardware vendors are interested in sponsoring the project either financially or through hardware sponsorship, this would be warmly received. -
-Sponsor AerynOS + If you wish to discuss other sponsorship opportunities, such as hosting or hardware sponsorship, please reach out to us at [contact@aerynos.com](mailto:contact@aerynos.com). - ## Thank You! We are very grateful for your support, be it financial or via project contributions in the form of carefully written bug reports, code contributions, design contributions, documentation updates, general feedback, package updates and overall enthusiasm around the project. diff --git a/src/content/docs/blog/2026/08/02/Solid foundations are leading to expanding horizons/index.md b/src/content/docs/blog/2026/08/02/Solid foundations are leading to expanding horizons/index.md index caae2adc..6fc2a694 100644 --- a/src/content/docs/blog/2026/08/02/Solid foundations are leading to expanding horizons/index.md +++ b/src/content/docs/blog/2026/08/02/Solid foundations are leading to expanding horizons/index.md @@ -2,13 +2,14 @@ title: "Solid foundations are leading to expanding horizons" date: 2026-08-02T21:00:00Z authors: [nomadiccore, ermo] +cover: + alt: A car light trail signifying moving at pace + image: ./unsplash.webp tags: [news] license: "CC-BY-ND-4.0" copyright: "Copyright © 2026 aerynOS Developers" --- -![A car light trail signifying moving at pace](unsplash.webp) - It's been several months since our last update in May, and a lot has happened around the AerynOS project in the meantime! First and foremost, we have tested, landed and productised our Versioned Repositories phase 2 feature, which will serve to firm up AerynOS' foundations and help us further develop our vision of how to build a Linux distribution. @@ -29,6 +30,8 @@ From a development story arc perspective, AerynOS has been deliberately kept und This shift in approach has already been evident over the last couple of months, as engagement around the project continues to grow through all of our social media platforms and particularly in our Zulip server. + + ## What's new in the distro ### Experimental inclusion of OpenZFS @@ -105,7 +108,7 @@ As ever, our intrepid contributors and staff have landed a bunch of updates. Ple ### Versioned Repositories phase 2 -![A terminal window showing the transition through Versioned Repositories phase 2](mossvrp2.webp) +![A terminal window showing the transition through Versioned Repositories phase 2](./mossvrp2.webp "Terminal window showing the transition through Versioned Repositories phase 2") The team, spearheaded by efforts from tarkah and ermo, have landed our Versioned Repositories phase 2 development sprint. This feature was included in the Unstable Stream update released on 5 July with users having reported a positive transition experience ever since. @@ -137,7 +140,7 @@ We will talk more about this in a future blog post when the currently in-develop ### Systemd packaging and user-facing changes -![A moss sync command in terminal showing the changes to systemd](systemd.webp) +![A moss sync command in terminal showing the changes to systemd](./systemd.webp "moss sync command in terminal showing changes to systemd") Reilly reworked our systemd package to ensure that manually enabling or disabling systemd units will no longer be overridden on moss package transactions. @@ -177,7 +180,7 @@ We are taking a deliberate approach of focusing on what we have delivered rather If you have experience developing websites with Hugo and want to contribute, we welcome PRs on the [Codeberg repository](https://codeberg.org/AerynOS/dotcom-hugo). -![A screenshot of the homepage of the new in progress AerynOS website](hugo-dotcom.webp) +![A screenshot of the homepage of the new in progress aerynOS website](./hugo-dotcom.webp "A screenshot of the in development aerynOS website") Please note that this is just a placeholder as the current focus is on the design rather than the content. @@ -207,7 +210,7 @@ The new ISO incorporates all the latest updates to date including the latest 7.1 ## CDN77 sponsorship -![The cdn77 logo](cdn77.png) +![The cdn77 logo](./cdn77.png "The CDN77 logo") The latest project sponsor for AerynOS is [CDN77](https://www.cdn77.com/), who is providing the project with content delivery and storage hosting services. @@ -263,18 +266,10 @@ Ideally we would like to grow our monthly income (and therefore surplus). Doing 3. Consider purchasing hardware for compatibility testing 4. Fund future initiatives for the betterment of the project -
-Sponsor AerynOS + If you wish to discuss other sponsorship opportunities, such as hosting or hardware sponsorship, please reach out to us at contact@aerynos.com. diff --git a/src/content/docs/blog/2026/08/31/A community that works together, builds together/index.mdx b/src/content/docs/blog/2026/08/31/A community that works together, builds together/index.md similarity index 98% rename from src/content/docs/blog/2026/08/31/A community that works together, builds together/index.mdx rename to src/content/docs/blog/2026/08/31/A community that works together, builds together/index.md index 5935b4d1..bd2c2e57 100644 --- a/src/content/docs/blog/2026/08/31/A community that works together, builds together/index.mdx +++ b/src/content/docs/blog/2026/08/31/A community that works together, builds together/index.md @@ -2,15 +2,14 @@ title: "A community that works together, builds together" date: 2026-08-31T23:00:00Z authors: [nomadiccore, ermo] +cover: + alt: An image of a welder building something in his workshop + image: ./unsplash-welding.webp tags: [news] license: "CC-BY-ND-4.0" copyright: "Copyright © 2026 aerynOS Developers" --- -import { Aside } from "@astrojs/starlight/components"; - -![An image of a welder building something in his workshop](unsplash-welding.webp) - Whilst August is a holiday month around Europe, it's safe to say it's been a very busy month around the project, with a lot of progress made across various core tooling repositories. Following on from our Versioned Repositories phase 2 work that we landed last month, we are developing `moss` on multiple fronts. [tarkah](https://github.com/tarkah) has supplemented our current hardlink driver approach with a reflink strategy for filesystems that support it, along with working on an in-progress development sprint to eventually deliver an EROFS metadata-image based approach that will be fully filesystem agnostic. [Fabio](https://github.com/livingsilver94) is working on moss' command tree, reorganising the commands we already have but also looking forward to the command functionality we will eventually want to have available. Lastly, [Jonathan](https://github.com/otherJL0) is also working towards improved search and shell completion functionality that will make `moss` easier to interact with for both users and packagers alike. @@ -25,6 +24,8 @@ Staff members [Alice](https://github.com/CookieSource/) and [NomadicCore](https: Finally, we're continuing to refine the project's branding with a slight tweak to the project name from AerynOS to aerynOS along with tweaks to logo colours and fonts. + + ## What's new in the distro ### Packaging and stack updates @@ -71,12 +72,11 @@ Package highlights for this month include: We mentioned EROFS metadata-only images that would operate in a read-only capacity. Work is still ongoing in this area, however in the meantime, we have delivered a more general filesystem tree abstraction within `moss`. This has then been supplemented with a [reflink](https://linuxjunkies.org/glossary/reflink) approach for the filesystems that support it. - - +::: The new `fstree` driver API separates the concept of a filesystem tree from the mechanism used to create and manage it. Our existing native implementation now works through this abstraction, while an overlay-image driver is also being developed and tested. @@ -166,11 +166,11 @@ One area of feedback has been a concern that having a more traditional forum on You can find the Discourse server [here](https://aerynos.discourse.group/). Sign up, look around and get involved! -![A screenshot of the aerynOS Discourse site](discourse.webp) +![A screenshot of the aerynOS Discourse site](./discourse.webp "A screenshot of the new aerynOS discourse forum") ### Branding and visual identity -![Comparison image showing our new logo brand colours on both a light and dark background](rebrand.webp) +![Comparison image showing our new logo brand colours on both a light and dark background](./rebrand.webp "Comparison of the new logo on light and dark backgrounds") During our [April blog post](https://aerynos.com/blog/2026/04/30/rebranding-upgrading-and-wallpapering-aerynos-april-glow-up/) we launched our new logomark, moving away from our older LLM-created A symbol. As the project continues to evolve, and as new members with design expertise join our community, we are continuing to refine our branding.