New Post: Embedded Linux Logging Article - #644
Open
grace-nordic wants to merge 4 commits into
Open
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a new draft Interrupt post describing logging strategies for resource-constrained embedded Linux devices, focusing on flash wear, RAM pressure, and practical daemon/configuration tradeoffs.
Changes:
- Introduces a new draft article covering kernel logging, syslog vs journald, and storage/rate-limiting considerations.
- Adds example configurations (journald.conf options and a Fluent Bit pipeline) plus references.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+2
to
+13
| # This is a template for Interrupt posts. Use previous, more recent posts from the _posts/ | ||
| # directory for more inspiration and patterns. | ||
| # | ||
| # When submitting a post for publishing, submit a PR with the post in the _drafts/ directory | ||
| # and it will be moved to _posts/ on the date of publish. | ||
| # | ||
| # e.g. | ||
| # $ cp _drafts/_template.md _drafts/my_post.md | ||
| # | ||
| # It will now show up in the front page when running Jekyll locally. | ||
|
|
||
| title: Embedded Linux Logging |
Deploying interrupt with
|
| Latest commit: |
df91487
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://3085330e.interrupt.pages.dev |
| Branch Preview URL: | https://grace-linux-log.interrupt.pages.dev |
|
|
||
| **How much should you actually log?** Severity levels let you dial verbosity up for active debugging and down for production, but the real lever for device longevity isn't the volume of logs you generate, it is how much of that volume actually gets written to flash, how often, and in what size chunks. Concretely, that means filtering relevant logs, rate-limiting logs as early as possible, and moving anything you only need to *count* out of the log stream and into a metric. | ||
|
|
||
| None of these are settings you get right once and forget. Rather, they are tradeoffs worth revisiting as a device's fleet, firmware, and failure modes evolve. Getting them right is what separates a logging architecture that helps you debug a fleet in the field from one that quietly wears out the flash underneath it. |
Member
There was a problem hiding this comment.
I really like this closing, nice!
|
Good topic, and good axes (tools and tradeoffs) to look at it with - excited to get this one added to the library! |
jakegwood
reviewed
Jul 29, 2026
grace-nordic
marked this pull request as ready for review
August 4, 2026 22:21
|
I know you can technically hit merge with my approval, but would appreciate one last pass from our benevolent interrupt leaders (@bahildebrand / @gminn others?) before I start giving full "ship it" approval, in case there's any best practices/preferences I missed :) |
gminn
requested changes
Aug 6, 2026
gminn
left a comment
gminn
left a comment
Member
There was a problem hiding this comment.
This is an awesome article @grace-nordic :star-struck: have a few comments + recommendations!
Member
There was a problem hiding this comment.
If you rename the file without a _ prefix, we will add it to the build, and we can test for any issues rendering
| title: Embedded Linux Logging | ||
| description: | ||
| "A look at embedded Linux logging challenges around collecting relevant kernel and user space information for device debugging while balancing flash wear and RAM pressure on resource-constrained devices." | ||
| author: grace |
Member
There was a problem hiding this comment.
We'll need your author bio under _data/authors.yml so we can render attribution (looks like grace is a free handle :) )
|
|
||
| <!-- excerpt start --> | ||
|
|
||
| Embedded Linux offers developers a flexible platform to develop intricate and complex systems. By nature, embedded systems are memory and resource constrained and require more consideration around how to optimize usage of the resources that are available. When it comes to device observability in the field, there are lots of questions around how to balance gathering enough critical information to debug systems without consuming too many resources and harming your overall system performance. This article will walk through various types of embedded Linux logging platforms and consider various tradeoffs between logging storage media, compression, and verbosity to enhance device debugging without eating into your limited resource budget. |
Member
There was a problem hiding this comment.
This excerpt (encapsulated with <!-- excerpt start/end -->) will render on the landing page of the blog, so we typically want 1-2 sentences max. I think this last sentence is great for an excerpt, so my suggestion is we move the start there
|
|
||
| ## Introduction to Embedded Linux Logging | ||
|
|
||
| Log messages are a valuable tool that an engineer has when debugging device issues, and they're typically a good first place to look when triaging a problem. As the complexity and number of devices enter the field, log files start to become cumbersome and hard to filter through to capture meaningful insights. Furthermore, as embedded systems are becoming more sophisticated, the logging architecture and design are playing a more critical role in overall system performance. Simply put, as compute is added more at the edge, log verbosity climbs, and memory writes climb with it. If this is not properly managed, it can lead to overall degradation of your system's performance. |
Member
There was a problem hiding this comment.
Suggested change
| Log messages are a valuable tool that an engineer has when debugging device issues, and they're typically a good first place to look when triaging a problem. As the complexity and number of devices enter the field, log files start to become cumbersome and hard to filter through to capture meaningful insights. Furthermore, as embedded systems are becoming more sophisticated, the logging architecture and design are playing a more critical role in overall system performance. Simply put, as compute is added more at the edge, log verbosity climbs, and memory writes climb with it. If this is not properly managed, it can lead to overall degradation of your system's performance. | |
| Log messages are a valuable tool that an engineer has when debugging device issues, and they're typically a good first place to look when triaging a problem. As the complexity and number of devices entering the field increases, log files start to become cumbersome and hard to filter through to capture meaningful insights. Furthermore, as embedded systems are becoming more sophisticated, the logging architecture and design are playing a more critical role in overall system performance. Simply put, as compute is added more at the edge, log verbosity climbs, and memory writes climb with it. If this is not properly managed, it can lead to overall degradation of your system's performance. |
|
|
||
| Since syslogd writes continuously to one file, syslogd should always be paired with some sort of log rotation utility to keep `/var/log` from growing without bounds. Busybox syslogd does include some rotation ability, but generally speaking, syslogd relies on logrotate for rotation. Either way, it is critical to set strict caps on maximum log size and age, compress and rotate the logs into persistent storage for debugging purposes. | ||
|
|
||
| syslogd is a good fit for low-to-moderate verbosity systems with tight resource budgets; however, due to syslogd's limited configuration capabilities, its performance is not as desirable as some other logging systems that have finer filtering and rotation capabilities configured out of the box. If `/var/log` is mounted on flash, a noisy system will wear the flash. Conversely, with `/var/log` on tmpfs, high-volume logs can put pressure on RAM. Either way, if you need finer control over what gets written, you'll want to consider heavier log filtering layers or logs to metrics, which can help alleviate some logging storage pressure. [^2] |
Member
There was a problem hiding this comment.
nit: I think moving the footnote to the first reference of busybox is better?
|
|
||
| Enforcing size limits on your log files will directly impact how much you are storing. Things like **`Compress=`** will reduce the physical bytes written for the same logical log volume. Utilizing configurations like **`SystemMaxUse=` / `RuntimeMaxUse=`** provides hard caps on the overall size impact in Flash + RAM. Other configurations for tweaking the overall storage strategy of these logs can be found in journald.conf. | ||
|
|
||
| journald also includes rate limiting and write amplification optimizations. The **`SyncIntervalSec=`** batches writes and doesn't `fsync()` on every message, except for the CRIT/ALERT/EMERG messages. A longer sync interval typically means, on average, fewer, larger physical writes for the same log volume, which will correlate with less flash wear impact, at the cost of losing more of the tail of the log if power is lost between syncs. The **`RateLimitIntervalSec=` / `RateLimitBurst=`** cap how many messages per interval will get accepted by journald at all, so a single noisy service can't flood your logs with useless information [^3]. |
|
|
||
| BusyBox does offer the ability to configure syslogd to only log messages into a small shared-memory ring buffer in RAM that you can leverage to read back with logread. The required configurations at compile time are shown below: | ||
|
|
||
| ``` |
Member
There was a problem hiding this comment.
we can format these as bash I think:
Suggested change
| ``` | |
| ```bash |
|
|
||
| The last tool you can leverage to optimize information captured from your devices in the field is to convert Logs to Metrics. What if, instead of storing hundreds of lines of logs for thousands of deployed devices in the field, you counted occurrences of the critical states the device enters? Instead of persisting every Ethernet TX failure as a log line that has to be searched and aggregated on the backend, you could just increment a counter each time this event occurs. Now you can answer questions like "How often is this issue happening?" and gauge issue severity without needing to store all these log lines. | ||
|
|
||
| Converting log events to metrics can allow you to filter out textual noise while still gaining meaningful insights. Most log-to-metrics tools are simple to implement and require you to write regular expressions to match against incoming log lines. There are applications that will hook into your logging daemon and increment local metrics of critical events instead of (or in addition to) passing the log through. Both Fluent Bit and memfaultd have easy integrations that let you configure which regex patterns in your logs should increment metrics. |
Member
There was a problem hiding this comment.
A small intro to what memfaultd is would help orient the reader (something simple like "at Memfault, our linux daemon for collecting observaibility data, memfaultd, ..."). We try not to assume too much about what the reader knows
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a new draft article covering Embedded Linux logging strategies for resource-constrained devices. It walks through various different logging daemons available and ideas to help reduce overall log volume while still preserving critical information for debugging devices.