DevOps on cache cascade

Introducing grlx

tai@taigrr.com (Tai Groot) — Tue, 07 Nov 2023 19:17:25 +0600

Background

In a recent blog post, I discussed a few of the reasons why in 2021, I began working on a new Fleet Configuration Management tool to replace SaltStack. However, I did not go into much detail about what the replacement is, how it works, or the roadmap into the future.

As mentioned in the prior post, the first inspiration to build a new tool arose from requirements regarding constrained memory environments. In our typical install, salt-minion at idle required between 140mb and 300mb of memory. (This wide variance in metrics results from several factors including the method of measurement, the use case, the system uptime, the service uptime, and the versions of salt-minion and python being run. Regardless, this is simply too much memory for salt to be usable in many situations.)

For widely-deployed, 1GB IoT systems, sacrificing between 10-30% of all available memory for a background task is not ideal. But in a world where even wrist watches ship with 2GB of RAM, and IoT components are getting cheaper by the month, maybe 300mb is not that big of a deal?

For the virtual machine use case, I’d argue it’s still very important. Linode (now Akamai) offers VPS systems starting at 1GB/mo. Amazon Lightsail’s smallest offering comes with only half a gigabyte of memory. Imagine trying to scale horizontally across many nodes only for half of the memory you paid for to be eaten up by a system management tool running in the background.

Without repeating too much from the prior blogpost, I challenge the reader to pause for a moment and consider the hurdles with their own configuration management software, memory requirements aside: how easy was it to get started? What kind of maintenance is required to keep things running and secured? Does it break? How often? What is the DX (Developer eXperience) like? Is the current solution easy (and safe!) to use at 2am for an emergency rollback from your parents’ house on holiday?¹

When I set out to build this tool from scratch, I saw it as a golden opportunity to not only solve existing problems but also to shape the tool according to my vision. The goal was clear: to develop a tool that functions seamlessly, aligning to the needs of the landscape, including the creature comforts missing from other options. It should be low overhead, offer familiarity with existing tools, be secure by default, easily scalable, and resilient to OS upgrades (read: runtime dependency-free).

Introducing grlx - Effective Fleet Configuration Management

Without any further introduction, here are some of the major issues we are looking to address in grlx compared to other tools:

Low Overhead

I’ve mentioned this one a lot already, but it bears repeating. The primary reason that drove us to pursue building grlx was the unavoidable memory footprint of the interpreted language ecosystems. While our first choice was to simple take an existing tool and fork it, paring down the featureset and optimizing memory, we quickly realized this would be an exercise in futility–we might see slight improvement, but nothing on the order of magnitude we were hoping for. The CPU usage of grlx should also be lower for the same reason: no JIT compilation or JVM (looking at you, Closure) taking up intermediate cycles.

Dependency-free by default

A tool written in Python or Ruby depends on the environment’s Python and Ruby packages for its own stability. When a primary goal of a system and configuration management tool is to keep the system up to date, and updating the system involves updating system libraries such as Python or Ruby libraries, there is always a small but non-zero risk that a breakage in the ecosystem APIs will crash the management tool and prevent it from restarting, leaving the node (or worse, the whole fleet) stranded and unresponsive. Python 3.7 for example, introduced new reserved words to the language (await and async) which subsequently broke many packages that hadn’t been updated in time. Typically, these ecosystems move slowly enough for maintainers to update in time, but it’s never a good idea to be stuck between waiting for a new software release and keeping your ecosystem out-of-date and vulnerable to prevent breakages.

Scalable and Fault-tolerant

Built using NATS.io, grlx can take advantage of NATS’s clustering and fault-tolerance features. Using only a single farmer system, you can join several more NATS Server instances to the cluster and have them all share the load of incoming messages. Read more about NATS clustering here.

Future additions to grlx will allow for a single sprout to connect to multiple independent farmers, to allow for failover across control servers.

Secure by default

Did you know that it’s a violation of SaltStack best practices (specifically the linked Hardening Rules page) to use Salt over the internet without taking precautionary encryption measures? It commonly recommended to use a tool like stunnel or WireGuard® to add an extra layer between minions and the master.

With grlx, there’s no need for a third-party encryption suite to stay secure. All communications from cli to farmer, and from farmer to sprout, are encrypted using self-signed TLS certificates and NATS.io NKey encryption. These certificates are pinned to the clients on first connection as an extra security precaution.

Easy to set up

Easy to automate. Need it be said again? Only one binary (and one configuration file) is required to provision a sprout (node).

We also offer a bootstrap script hosted at bootstrap.grlx.dev that will always get you the latest and greatest release in a simple one-liner, if you so choose.

Easy to Understand

Go is often touted as simple language. All grlx code is written in Go (except for the Web UI), and we explain how both the overall system and the individual components work in the documentation. Auditing the grlx codebase is easy, and most of the source code can be read through in a day or so.

Our non-stdlib build dependencies are selective–kept to as short a list as reasonably possible, so there’s little happening “elsewhere.” We follow best practices, strive for high test coverage, and keep an eye on our Go report card.

Open Source

It would be crazy to install a tool on a fleet of systems and give it root access without the ability to do some introspection on what’s happening under the hood. All the source is available on GitHub, where I encourage you to give it a star!

Not only is grlx open and free, it’s also permissively licensed as 0BSD and is therefore compatible with nearly any enterprise company’s policy on OSS software.

Keep in mind, the logos, the mascot (also referred to as “Clove”), the name “grlx” itself, and the overall brand are copyrighted, all rights reserved, etc. Do not try to pass off grlx as your own–but forking and renaming is allowed!

Supported with Corporate Backing and Buy-in

grlx has entered into an agreement with ADAtomic, Inc., to offer official, commercial support. If this is something that would be valuable to your company or organization, please contact ADAtomic here.

Additionally, grlx is already seeing production use from several companies, listed in the README. As mentioned above, grlx development is being driven by a real need at our organization, and is seeing heavy usage in our own lineup of products.

Extensible Plugin System

To state the obvious, not every possible feature is required by every customer. To keep grlx light and fast, all endpoints and features take advantage of dependency injection and there is a well-defined interface already in place for loading plugins at runtime. These plugins can offer features such as supporting obscure package managers or downloading files from uncommon endpoints. In short order, two example file endpoint provider plugins will be released, one to download files from IPFS, and another to download from a BitTorrent magnet link. Hopefully, these examples indicate both the reason for and capability of the plugin system. Using runtime-loaded plugins is also a boon for development and testing, as mainline code that should be added to grlx can start as a plugin for development, and get merged over time.

At first, the plugin interface will only support Go plugins, but support for WASM plugins is on the roadmap, allowing for plugin development in nearly any compiled language.

To be clear, grlx is dependency-free by default, but we recognize there are some use cases where optional plugins may offer increased functionality without adding significant hooks into environmental state. Both Go and WASM plugins can be distributed as single files, dropped into a directory and hot-reloaded automatically at runtime.

Configurable Message Delivery Rules

Some tools only allow for deploys to go out the door to nodes that are online at the time of the push–if it’s not online right now then you have to either watch and wait for it to come online, or write some scripts to programmatically check for the next time it does come online and push the release then. In a datacenter, this is not a big deal as you should have extremely high (near 100%) uptime and connectivity between the CNC Server and the nodes. For an IoT deployment on the other hand, connectivity can be tricky, and the aforementioned scripts must come into use.

However, once you start writing them, you might realize that there’s a lot more involved regarding error checking, reporting, logging, etc., that the simplest while ! online; do sleep 60; done && push script cannot handle. You end up writing a whole deployment framework around trying to catch your node when it’s online and push to it at that exact moment, or maybe you add a cron job to the node itself to trigger an agentless or standalone mode periodically.

This whole mess could have been avoided if the configuration management tool provided a utility for you, which stores the deploy jobs into persistent queues to ensure delivery on next reconnection. NATS.io’s concept of Durable Queues allows for exactly this behavior.

RBAC

grlx takes a Role-Based Access Control (RBAC) approach for designating who is allowed to deploy what–and where. Secured service worker accounts or admins might be allowed root access to all machines. Your coworker John, who manages some nginx configurations, can cook (push out) the nginx recipe to all sprouts, and Celine has access to cook all recipes but only on select development/canary nodes. Future work will allow these roles to come from an LDAP endpoint or other external service provider plugin.

More information about the RBAC system can be found in the documentation.

Usability Improvement Features

Below, find some of the small tweaks and differences to make using grlx easier than competing tools:

Advanced Developer Tooling

Our development team is hard at work creating an LSP server and Linter for grlx. The roadmap includes support for jumping to import definitions, full syntax highlighting (even in recipe files containing templates!), and property checking. Our plugin interface also includes definitions to generate custom LSP inputs so that the tools you might need to build in-house are treated as first-class citizens in your editor.

Official GUI and TUI built-in

Many home-grown WebUIs have been built to support existing solutions which don’t come with official WebUIs. The grlx command line utility will soon support spinning up a local webUI, server over localhost, which can be used to access most functionality. Communications to the farmer are proxied through the CLI, so there’s no need to open up ports on the farmer to access the WebUI.

A TUI (Terminal UI), built with BubbleTea is on the roadmap.

Optional DropShell Support

While grlx already supports arbitrary command execution, sometimes it happens where you really just need a shell to do some in-depth troubleshooting. For SaltStack, I developed TunnelBunny, a process for setting up port forwarding and reverse shells to minions, which you can read about here. grlx will support a similar feature, called DropShell, which will allow you to drop into a shell on a single sprout, from your local machine, without having to open up any ports on the minion, configure SSH, or drop firewall rules. This feature is still in development, but will be available once RBAC is fully implemented, as it might be a security concern for some organizations.

High test coverage

We strive for 100% test coverage on that parts that matter. Specifically, all ingredients are thoroughly unit tested, and the core functionality of the platform runs through a suite of integration tests inside of a docker-compose environment.

No Server Access Required

The command line utility is separate from the farmer, and can be run from any machine with network access to the farmer. All communication is encrypted using TLS and NKey encryption, so there’s no need to provide a VPN or SSH tunnel to the farmer. Depending on company policy or developer preference, the CLI can be run directly on developer machines, or from a dedicated bastion server. Authentication and authorization is handled on a per-user basis (to support RBAC), so there’s no need to share credentials, SSH keys, or configure sudo access to the main CNC server.

Supports GitOps

The command line utility cook command will soon support a --git flag, which will tell the farmer to check out a particular branch, tag, or commit of a git repository, and use the files in that branch as the source of truth for the recipes. Additionally, the currently checked-out commit, tag, and branch are made available to the recipe files as property variables, so you can use them in your templates should the need arise. There is also a command to update the recipe’s git repository to the latest commit, tag, or branch, and a command to list all available branches, tags, and commits.

Synchronous Job Status reporting

One of the most frustrating things about using a configuration management tool is the lack of visibility into what’s happening, as it’s happening. When submitting a job, you might get a job ID back for polling, or wait synchronously for the job to finish and see the summary of results but most tools won’t let you watch the states finish as they happen. Most of the time, this is fine, but the few times you need live introspection into why a deploy is stuck, you’re out of luck.

grlx will soon support a --watch flag on the cook command, which will stream the job status to the terminal as it happens, with a live progress bar and summary of results at the end.

Restful API

The farmer exposes a RESTful API for all of its functionality, so you can build your own tools around it. If you don’t like the built-in WebUI, you have the option to build your own! If you need to build automations around your deployment from a third-party tool, you can do that too! Everything from cooking to command execution is available via the API.

More information about the API can be found in the documentation.

Lifecycle Hooks for Outgoing Webhooks

Do you need to track one particular step of a deploy for a particular reason? Maybe you need to know whenever a specific file is created or modified? Recipe files have a field for providing an upstream webhook URL and JSON body you need sent. Hooks can also be configured to fire off WebHooks when a deploy starts, fails, or finishes. Additionally, you can configure a hook to fire when a new sprout is added to the fleet, or when a sprout’s online status has changed.

Introspection and visibility has never been this easy!

Friendly Discord

Our Discord invite is open to one and all, where you can chat with the developers of grlx and get involved in the grlx community.

Meet like-minded DevOps professionals and swap ideas about the best way to get something deployed–or embroil yourself into a flame war over Cloud or On-Prem–as long as you keep things respectful it’s up to you!

Cute Mascot!

His name is Clove, isn’t he great?

Summary

In summary, I hope this article has presented the litany of reasons for a new tool in the configuration and system management space. Our roadmap is clear, and we are excited to announce that version 1.0.x is available for download now. Please see ADAtomic.com for more information!

Updating or rolling back production from your parents’ house is not a recommended DevSecOps or GitOps practice, nor am I endorsing it in any way. Ideally, you’ve got canaries and auto-rollbacks, split traffic, staging environments… This statement is meant to be humorous, don’t take it literally. ↩︎

So Long, Salt Project

tai@taigrr.com (Tai Groot) — Mon, 09 Oct 2023 18:19:25 +0600

This post has been a long time in the making. Anyone who has followed me and my work for any period of time probably knows by know that I’ve been working on grlx (pronounced ‘garlic’) for some time now (since mid-2021). But why? What’s wrong with SaltStack? More specifically, what problem am I trying to solve that doesn’t already have a solution solved many times over?

A Note to the Salt Project

First of all, I want to make something clear: this post is not an attack on Salt, Thomas Hatch, or any of the people involved in the Salt Project. I’ve been a long-time user of SaltStack, going back to the Boron release in 2016. SaltStack has gotten me really far, and I owe much of my career’s success to Salt, which I have carried with me from job to job.

I also want to make a call-out here: my use case for Salt is not necessarily the target use case, and it’s really unfair to criticize a screwdriver for being a bad hammer, especially when the screwdriver is free.

Real-world Requirements

At my first role as a Salt user, I was not the decision maker. I was a junior developer, working with the tools selected by peers and higher-ups, and Salt was chosen to be part of our stack. Without going into too much detail, here was our problem statement:

We are building IoT, embedded Linux boards
We need a way to update the software and packages remotely
We are using field-prototypes, which don’t have fixed or public IPs and therefore need to be the Client in a Client-Server model (agentless solutions won’t work)
We are not just updating application code, our environments are going through rapid updates (emphasis on field prototypes)
Our distribution is a custom Yocto build, without a package manager, so we need a way to declaratively replace system-level packages transactionally
The ability to remotely run shell commands and return the output is a plus (we might need to trigger fswebcam and snap a photo as a one-off)
Rolling out instant updates to targeted devices (where a ’target’ could be a small sampling, the whole fleet, or a single device) is a huge boon
Some devices will connect over 3g cellular as not all of our field installations have WiFi available

As this platform was a product of its time, here’s some additional information to keep in mind:

The year is 2016, and the hardware development went back to 2012 (so pre-Raspberry Pi GA)
Google IoT Core doesn’t exist (yet, and also, RIP)
Amazon GreenGrass is not generally available (until 2017)
Ubuntu Core IoT was prohibitively expensive and didn’t support our custom hardware

SaltStack, in theory, checked all of our boxes, and allowed us to ship units to the field for testing, with remote, OTA updates.

Full Linux support? Check. [x]
Remote updates using a client / server model to get around the IP contstraints? Check. [x]
Ability to upgrade system files in addition to application files? Check. [x]
Remote command execution? Check. [x]
Specific device targeting? Check. [x]

And so, our small team set out to create a configuration management system built on top of SaltStack as the center pillar. Yes, we even managed the SaltStack installation with SaltStack, by replacing the Python libraries and .egg/wheel files with Salt states. We were careful in how we did this, and actually added a cron-based backdoor ‘rescue’ service in case the snake swallowed its own tail, but it was a calculated risk. In the end, the deployment worked, but it wasn’t without headaches.

Issues and Complaints

Dependencies

There were several runtime and compile-time bugs related to the dependencies. Anybody who’s been around SaltStack long enough will shudder at the mention of Tornado or PyCrypto. We were especially subject to the issues regarding pycrypto compilation, as our Yocto project build ran on ARMv5 (oh whoops, did I not mention that requirement earlier? That sucks, but we’re all engineers here and you just have to learn to roll with it…) and there weren’t precompiled wheels for our architecture–we had to build them ourselves. Memory was always running low, as we had less than 1GB of physical memory, and most of that needed to go to the application for some on-device image processing.

Crashing

There was a memory leak, somewhere, in either salt-minion itself, or in a dependency. It was difficult to isolate, and I’m not sure my team actually ever found it. It was likely patched in a later version of salt-minion (again, or in a bumped dependency). Our solution was to limit the memory used by systemd and tell the unit to crash and restart if it went over. This was not ideal. Systemd had no visibilty into how we were using salt-minion, whether we were killing the process due to a real memory leak or due to a spike in usage associated with an active software update. It’s reasonable to allocate more memory to salt-minion and away from our application during a software update, but not at the expense of allowing run-away memory usage during regular runtime. Allowing systemd to indiscriminately OOM salt-minion becomes even more dangerous if we are currently applying a state to update the system or python libs themselves–a crash mid-update for core and critical python libraries might prevent salt-minion itself from ever coming back up. In the end, we had to isolate the states for the system by themselves and run those states very selectively, not in the top/high state, which somewhat defeats the purpose of the ‘statefulness’ Salt brings you–but that’s not Salt’s fault, as it certainly wasn’t designed for that. Hammer, screwdriver, all that. Really the problem here was the memory leak itself, as that’s the cascading issue.

Connectivity issues

Memory leaks and crashing aside, we had a lot of trouble keeping the zeroMQ connection active. Now, we did have many systems connected over 3G cellular, but the problems weren’t isolated to those units. Our WiFi-connected units also had issues maintaining connectivity. The dreaded Minion did not return. [Not connected] message was a commonplace error. More disturbing, early versions of Salt didn’t allow you to easily configure the retry logic. Sometimes, the connection would break, and the client wouldn’t ever try to reconnect, so it would stay disconnected until the memory leak would trigger an OOM, which would in-turn restart the salt-minion daemon. ( This has since been addressed.)

Access control

Whoever needed to send the updates out to the units in the field needed to be root (or have sudo access to run salt) on the Salt master, which means someone needed shell access to the CNC server. From a security perspective, this is an ick.

I loved SaltStack

Yes, I’ve had my fair share of problems with SaltStack, but my teams and I made things work. When working with software of this scale, some tips and tricks tend to reveal themselves over time, (like using get_template to pre-render salt states before applying). Sometimes, you just need the tribal knowledge gained through experience. Several issues were fixed by the Salt Project team, others have gone away simply through a revision to requirements on my end, and still others I’ve learned to live with or work around.

I’ve brought SaltStack with me to 4 other career roles since my introduction to it, like some kind of DevOps Evangelist (I’ll likely write about these projects at a later date). I didn’t have enough good things to say about Salt. I became very involved in the community Slack channels. I answered a question or two on Stack Overflow regarding SaltStack. I’ve had a minor PR merged into salt-bootstrap. I’ve been featured on the Salt Community Open Hour podcast. The project called me out in an advisory bulletin to thank me for my help on CVE-2020-11651 and CVE-2020-11652.

Acquisition Heartache

In October, 2020, VMware acquired SaltStack, and the branding was changed to Salt Project. Large portions of the team were made redundant (read: laid off) and the focus of the project seems to have shifted to cloud-native and VMware-related projects. The name change alone causes a bit of a headache. The Salt Project is already a(n entirely unrelated) thing. Changing from “SaltStack” to “Salt” makes finding security issues difficult when SALT exists. The overall “Googleability” of the project has gone downhill, which really isn’t great when some of the page redirects are broken for all pre-hydrogen releases.

The number of unresolved issues in the project have skyrocketed (even despite the stale bot doing numbers on these issues). It seems the reduced team size and increasing other priorities has prevented development from keeping up with security and bugfixes in some dependencies, and the ‘solution’ is to freeze the updates and vendor the dependencies. That’s right, until recently, SaltStack has been vendoring a copy of the Tornado library, despite publicly-known CVEs.

Salt has also recently changed their packaging format to Onedir, effectively vendoring all dependencies inside of a single folder (including python itself). While this sounds great on the surface, keep in mind it will prevent non-breaking, API-compliant hotfix changes from making it down to the user through their system package managers, so users are now completely at the mercy of the Salt Project’s timelines and priorities, and their responsiveness record there is…not perfect in my opinion.

Where To Go From Here

Perhaps now you see why I’ve started grlx. I need to fill a SaltStack-sized hole in my heart; build a tool that solves all the same problems Salt did for me, but also address my own issues with the tool. grlx will never have 1:1 feature parity with SaltStack, especially as the Salt Project focus shifts to VMware first-party products and integrations, but for people like me, that’s a pro, not a con. In grlx, our goal is to support all the things that make SaltStack great–the extensibility, human-readable configuration as code, lightning fast remote execution–and do it with a small footprint, secure by default.

It’s my feeling that SaltStack probably isn’t to blame for its memory usage either–it is written in Python, and that’s just the price paid for an interpeted language. For this reason, among others, grlx is written in Go, a compiled language. Similarly, the connectivity issues are likely not part of the Salt codebase, and are probably part of the zeroMQ dependency. For grlx, we’re using NATS.io instead, and we’ve even submitted code upstream in an attempt to be good Open Source stewards.

If you’d like to follow the development of grlx, there are several ways you can do so.

We have a Twitter/X account here.
We have a community discord!
You can also star our project on GitHub using the button below:

Star

I’ll also be releasing several posts in the coming weeks going into more detail about the features and roadmap for grlx, so hit that RSS button at the top of the page to make sure you don’t miss anything!