cache cascade

FreeCodeCamp Podcast Announcement

Fri, 27 Jun 2025 13:22:25 +0600

Hello, everyone! This is not a typical blog post, but I wanted to share my recent guest appearance on the FreeCodeCamp podcast, where I had the pleasure of discussing grlx, my development journey, some more details on how to choose a programming language, and more.

We also discussed career advice, the importance of upskilling, and the mindset needed to succeed in the tech industry.

Enjoy!

If the embedded video doesn’t work, you can watch it directly on YouTube: FreeCodeCamp Podcast.

Why I Chose Go for grlx Instead of Rust

Wed, 11 Jun 2025 22:31:25 +0600

Why Not Rust?

There’s been a lot of debate lately about language choices, especially after Microsoft announced that the new TypeScript compiler is being rewritten from TypeScript to Go. This sparked a fresh round of questions on social media: why Go, and why not Rust? It’s a fair question, and one that applies to grlx as well. The answer, in both cases, is rooted in pragmatism.

I’ve written extensively in the past on why I chose Go over Python, but many of those arguments for Go over Python also apply to Rust over Python! Between Go and Rust, Go continues to be the language of choice for teams that prioritize development speed, ecosystem maturity, and operational simplicity. Let’s take a look at why.

Familiarity and Developer Productivity

Let’s be honest: I’m a Go developer by trade. That familiarity translates directly into productivity. When building grlx, being able to quickly write, test, and maintain code is more important than chasing theoretical speed or memory guarantees. My experience with Go allows me to implement features quickly and pivot when needed. I’ve written production Rust, and I respect it, but Go is where I’m fastest.

Performance in Practice

Rust shines when you need tight control over memory and raw performance. But, grlx isn’t compute-bound. It shells out to system commands, modifies configuration files, and reacts to asynchronous events. The bottlenecks are I/O and process orchestration, not CPU or memory allocation. While Rust’s performance profile is technically superior, those benefits wouldn’t be realized in a tool like grlx. In practice, Go’s performance is more than sufficient.

Type Systems and Safety Guarantees

Rust’s type system is more advanced. That’s a fact. But for grlx, it’s not a clear win. Much of grlx’s core functionality involves invoking and managing external processes, tasks that don’t gain much from advanced type guarantees. In many cases, to maintain portability across distributions, the safest way to perform a task is to shell out and handle the output explicitly. Trying to enforce strict safety via Rust’s type system often leads to awkward wrappers and, ironically, more unsafe code. If you’re already bypassing the compiler’s guarantees, the rest of the system doesn’t benefit much from theoretical safety.

Iteration and Flexibility

The grlx project is still actively evolving. In this phase, fast iteration is key. Go’s lightweight syntax and forgiving compiler make it easy to prototype, refactor, and shift architectural decisions as the product matures. Rust’s compiler enforces rigor early on, which can be helpful in large, stable systems—but it often gets in the way during the creative stage. Go lets me move faster.

Ecosystem and Tooling

Go’s ecosystem is mature and focused, especially in the realm of system tooling. From HTTP servers to CLI libraries, the tools I need already exist and are well-maintained. For missing pieces, writing custom packages is straightforward. I’ve built wrappers for systemd, SysV Init, and OpenRC support. In Go, this feels natural. Rust’s ecosystem for system-level orchestration is improving but still lags behind in simplicity and breadth.

Readability and Auditing

The grlx sprout runs with root permissions, so transparency is critical. Go’s straightforward syntax and lack of metaprogramming make it easy to understand and audit. Other developers, ops engineers, and security reviewers can all look at the code and follow what’s going on. Rust, while expressive, often introduces cognitive overhead via macros, lifetimes, and traits. This makes security reviews more complex. With Go, what you see is what you get.

Security via Simplicity

Using Go’s standard library reduces reliance on external dependencies. That matters. Every third-party package increases the surface area for supply chain attacks. With Rust, even basic functionality often comes from external crates, which need to be audited, version-locked, and maintained. Go includes batteries for most common tasks, making it easier to write secure, auditable, and maintainable code from day one.

Conclusion

Rust is an excellent language, and I wouldn’t hesitate to use it in the right context. But grlx isn’t a performance-critical game engine or a systems kernel. It’s a fleet orchestration tool. For that purpose, Go’s fast iteration cycle, strong ecosystem, simple deployment model, and security-minded standard library make it the better choice. This isn’t about hype or language tribalism. It’s about building the right tool with the right technology—and for grlx, Go is it.

If you want to learn more about the tradeoffs between Go and Rust (or TypeScript!), you can check out my course on LinkedIn Learning. It’s free!

Comparing Golang vs Rust from Choosing the Right Back-End Language: TypeScript, Go, or Rust for Your Greenfield Project by Tai Groot

Using PAM for SSH Login Telephony

Sun, 18 May 2025 22:31:25 +0600

Pluggable Authentication Modules ( PAM) provide a flexible mechanism for authenticating users on Linux systems. By integrating PAM with Gotify, you can receive real-time notifications for SSH login attempts, enhancing your system’s security and awareness.

I’ve been using this approach for over 5 years across many systems — it’s lightweight, reliable, and highly effective. Let’s walk through how to set it up.

Prerequisites

Before you begin, make sure you have the following:

A Linux server with SSH access.
Root or sudo privileges.
curl installed.

Step 1: Install and Configure Gotify

Gotify is a simple self-hosted server for sending notifications. You can also use a hosted instance if desired.

Install Gotify or use an existing instance.
Create an application in Gotify and note down the application token.

Test that your setup works by sending a test message:

curl -X POST "http://<gotify-server>/message" \
 -H "X-Gotify-Key: <application-token>" \
 -F "title=Test Notification" \
 -F "message=This is a test message"

Step 2: Create the PAM Notification Script

We’ll configure PAM to trigger this script every time someone logs in via SSH.

Create the script:

sudo vim /usr/local/bin/pam-ssh-notify.sh

Paste the following:

#!/usr/bin/env bash

GOTIFY_URL="http://<gotify-server>/message"
GOTIFY_TOKEN="<gotify-token>"

/usr/bin/curl -s -X POST "$GOTIFY_URL" \
 -H "X-Gotify-Key: $GOTIFY_TOKEN" \
 -F "title=SSH Login Alert" \
 -F "message=User \"$PAM_USER\" logged in from \"$PAM_RHOST\""

Make it executable:

sudo chmod +x /usr/local/bin/pam-ssh-notify.sh

Step 3: Update PAM Configuration

Edit the PAM configuration for SSH:
```
sudo vim /etc/pam.d/sshd
```

Add this line at the end of the file:

session optional pam_exec.so /usr/local/bin/pam-ssh-notify.sh

Why optional? If Gotify is unreachable and this script fails, PAM will ignore the failure. If you use required, login failures could result from simple network errors — not ideal for a monitoring hook.

Step 4: Enable and Test It

Ensure the UsePAM setting is set to ‘yes’ inside of /etc/ssh/sshd_config (or wherever your configuration is).
Restart SSH:
```
sudo systemctl restart sshd
```
Log in from another machine and confirm that your Gotify app receives a notification.

Appendix: PAM Environment Variables

When PAM executes your custom script, it passes in useful environment variables that can be used to enhance logic, filtering, and alerting.

Variable	Description	Example Use Case
`PAM_USER`	User being authenticated	Notify only on root logins
`PAM_RHOST`	Remote IP or hostname	GeoIP filter or blacklist
`PAM_SERVICE`	Service name (e.g. `sshd`, `sudo`)	Alert only on `sshd` logins
`PAM_TTY`	Terminal device (e.g. `pts/0`)	Log which TTY was used
`PAM_TYPE`	Event type (`auth`, `session`, etc.)	Differentiate login vs session start
`PAM_RUSER`	Originating user (e.g. for `sudo`)	Log who used `sudo`

Considerations

While this PAM-to-Gotify setup is simple and effective, it’s best suited for single-node systems like personal VPS setups or small self-hosted boxes, systems you’re willing to give your full attention to. It is not scalable for large fleets or production environments with dozens or hundreds of machines. For those cases, you’d want something more robust and asynchronous — like a logging agent with webhook triggers or a centralized security event bus. For a midrange use case with an audit trail, I’ve also swapped out Gotify for a custom Slack App, using the channel webhooks in the past.

Also, a critical caution: setting your PAM module to required instead of optional can be dangerous. If Gotify (or DNS) is down, a required script failure will prevent user logins entirely. This is not hypothetical — in 2021 Facebook famously locked themselves out of their own data center because DNS broke, and they couldn’t log in to fix the DNS server. By marking this hook as optional, you avoid cascading failures. Learn from their mistakes.

PAM is capable of much more–adding 2-factor authentication, YubiKey support, LDAP auth, encrypted filesystem mounting, etc. If you want to learn more, the ArchLinux Wiki is a great source, even if you’re not running Arch yourself.

Conclusion

With just a few lines of shell and PAM configuration, you can build powerful real-time login alerts and security automation using Gotify. The $PAM_* variables give you rich context for decision-making and logging, and the system is lightweight and reliable.

Feel free to extend this setup to logins via sudo, su, or even failed login attempts — and adapt it to your environment as needed.

Stay secure!

Temporal Server: Self-hosting a Production-Ready Instance

Tue, 29 Apr 2025 18:19:25 +0600

Temporal is an open-source workflow orchestration engine designed to manage, execute, and monitor complex workflows and activities. In plain terms, Temporal allows you to write crash-proof code: it ensures that your long-running processes (workflows) continue execution reliably even if your services (or external services!) crash or restart.

Temporal’s walkthroughs and CLI provide extremely easy adoption and low-friction for getting started: the temporal CLI contains an embedded development server (temporal server start-dev), which binds to localhost and allows for easy, unauthenticated workflows. Connecting to Temporal’s managed service is similarly easy, but at writing, has a minimum cost of $100/month, which is not always affordable for smaller orgs, especially when running a production-grade self-managed server can cost less than a quarter of that.

In this article, we’ll set up a single-node Temporal server backed by PostgreSQL, and we’ll secure it with TLS using NGINX as a reverse proxy. The end result will be Temporal’s services running in Docker containers, accessible at https://temporal.example.com (our example domain) with encrypted connections.

Note: This setup is suitable for small-scale, development and production deployments. Temporal is designed to scale horizontally and run multiple nodes for high availability and heavy workloads. For production, you should consider a multi-node cluster and follow Temporal’s production best practices (e.g. running each Temporal service separately, managing database schemas manually, etc.) community.temporal.io. There is significant effort required to deploy and manage multiple horizontal Temporal servers, between certificate rotation and configuration management, especially as Temporal often changes configuration requirements. Once you get to the stage where you need the reliability of multiple Temporal servers, my recommendation is that you pay for their managed service.

Overview of the Deployment

Our deployment will consist of the following components:

Temporal Server – runs Temporal’s core services (History, Matching) in one container. We will use the official temporalio/auto-setup image for convenience, which launches all services in one process and auto-initializes the database schema docs.temporal.io.
Temporal Web UI – runs in a separate container (temporalio/ui image) to provide a web interface for viewing workflows, tasks, and Temporal namespaces
PostgreSQL – a single Postgres instance as Temporal’s persistence store (keeping workflow state, history, etc.). I recommend using a managed database for this, like DigitalOcean (Disclaimer: this is my personal RefLink to DigitalOcean) to ensure uptime and reliability.
NGINX – runs on the host as a reverse proxy. It will terminate TLS (HTTPS), forwarding requests to the Temporal WebUI container, and provide Basic Auth support.

All these services can run on a single machine via Docker Compose (again, I do recommend using a managed Postgres instance for stability). The domain temporal.example.com will point to this machine (use an A or CNAME record!), and we’ll obtain an SSL certificate for it using Let’s Encrypt.

Prerequisites

Before we begin, make sure you have the following on your VPS:

Docker and Docker Compose: This is how we’ll be standing up our temporal services and ensuring they automatically restart
A domain name: for accessing Temporal (e.g., temporal.example.com), with DNS records pointing to your server’s IP. Don’t forget to reserve a public IP! Some cloud platforms will auto-release and reassign a new IP for you if you reboot, and this will break your DNS!
Firewall Rules: Block all incoming traffic to your server, especially port 8080, and only allow through TCP/7233, TCP/80, and TCP/443. Later, you can whitelist your worker nodes for 7233, or just allow all (0.0.0.0) to access this port.
Certbot: Let’s Encrypt’s CertBot utility can be helpful for provisioning and rotating SSL certs in nginx
GRPCurl: A grpc CLI for doing quick spot-checks on the installation
temporal (cli): A handy CLI utility for interacting with the Temporal server

If you’re running an Ubuntu/Debian/Debian Derivative, this shell snippet may help:

apt install docker-compose-v2 nginx python3-certbot-nginx apache2-utils golang
go install github.com/fullstorydev/grpcurl/cmd/grpcurl@latest
curl -L 'https://temporal.download/cli/archive/latest?platform=linux&arch=amd64' > x.tar.gz
mv temporal /usr/bin
rm LICENSE x.tar.gz

Now, with that out of the way, let’s get started!

Setting Up Our Docker Compose

First, create a directory for your Temporal deployment and navigate into it. Then create a file named docker-compose.yml with the following content:

version: "3.5"
services:
 temporal:
 container_name: temporal
 image: temporalio/auto-setup:${TEMPORAL_VERSION:-1.24.2.1} # You can substitute for the latest Temporal version
 env_file: .env
 restart: always
 networks:
 - temporal-network
 ports:
 - 7233:7233 # Expose gRPC frontend port
 volumes:
 - ./certs:/etc/temporal/certs
 - ./base.yaml:/etc/temporal/config/base.yaml

 temporal-ui:
 container_name: temporal-ui
 image: temporalio/ui:${TEMPORAL_VERSION:-1.24.2.1} # Use the latest Temporal Web UI version
 env_file: .env.ui
 restart: always
 depends_on:
 - temporal
 networks:
 - temporal-network
 ports:
 - 8080:8080 # Expose Web UI port
 volumes:
 - ./ui.yaml:/etc/temporal/config/development.yaml
 - ./certs:/etc/temporal/certs
networks:
 temporal-network:
 driver: bridge
 name: temporal-network

Next, let’s set up Postgres.

Configuring Postgres

There are many ways to configure your Postgres instance, and plenty of ways to restrict permissions. Here are the minimum requirements to make it work:

Create a database named temporal
Create a database named temporal_visibility
Create a user for the Temporal server to use (here we’ll assume tuser)
Using psql on your personal/work machine, you’ll need to connect to the Postgres instance as an admin and grant full access to the user on these two databases:

Note: Don’t forget to whitelist the IP address of your temporal server to access the database! Similarly, you may need to whitelist whichever machine you’re using to connect to the database with psql.

GRANT CONNECT ON DATABASE temporal TO tuser;
GRANT USAGE ON SCHEMA public TO tuser;
GRANT CREATE ON SCHEMA public TO tuser;
ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON TABLES TO tuser;

Do the same for temporal_visibility:

GRANT CONNECT ON DATABASE temporal_visibility TO tuser;
GRANT USAGE ON SCHEMA public TO tuser;
GRANT CREATE ON SCHEMA public TO tuser;
ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON TABLES TO tuser;

Now, let’s create some TLS certificates.

TLS certificates

Make a directory called certs, and cd into it. Inside this directory, we’ll generate a RootCA file:

openssl genrsa -out rootCA.key 2048
openssl req -x509 -new -nodes -key rootCA.key -sha256 -days 3650 -out rootCA.pem

You can adjust the expiration time or keysize as you wish.

Create a client and server key:

openssl genrsa -out client.key 2048
openssl genrsa -out server.key 2048

Now, using our RootCA, we’ll mint some certificates. Let’s start by creating some Certificicate Signing Request (.csr) files:

openssl req -new -key server.key -out server.csr -config server.cnf

You’ll be prompted to answer some questions about your organization.

Be sure the commonName and field matches your DNS record, i.e. temporal.example.com
For DNS Alt-names, provide the following:
- DNS.1: temporal
- DNS.2: temporal.example.com
- IP.1: 127.0.0.1
- IP.2: <YOUR RESERVED IP HERE>

And similarly, for the client:

openssl req -new -key client.key -out client.csr -config client.cnf

Here, the values don’t really matter. Configure as you see fit.

Now that we have our .csrs, let’s generate the certificates:

openssl x509 -req -in client.csr -CA rootCA.pem -CAkey rootCA.key -CAcreateserial -out client.crt -days 3650 -sha256 -extfile client.cnf -extensions v3_req
openssl x509 -req -in server.csr -CA rootCA.pem -CAkey rootCA.key -CAcreateserial -out server.crt -days 3650 -sha256 -extfile server.cnf -extensions v3_req

Now, you should have a directory that looks like this:

temporal@temporal-server:~$ tree
.
├── certs
│   ├── client.cnf
│   ├── client.crt
│   ├── client.csr
│   ├── client.key
│   ├── rootCA.key
│   ├── rootCA.pem
│   ├── rootCA.srl
│   ├── server.cnf
│   ├── server.crt
│   ├── server.csr
│   └── server.key
└── docker-compose.yml

Next, we need to create our .env and .env.ui files.

Configuring Env Vars

Create a .env file next to your docker-compose.yml, replacing the <SUBSTITUTIONS>:

POSTGRES_USER=<YOUR USER HERE>
POSTGRES_PWD=<PASSWORD HERE>
DB=postgres12_pgx
DBNAME=temporal
VISIBILITY_DBNAME=temporal_visibility
DB_PORT=<YOUR DB PORT>
POSTGRES_TLS_ENABLED=true
POSTGRES_SEEDS=<YOU DB HOSTNAME/IP HERE>
POSTGRES_TLS_DISABLE_HOST_VERIFICATION=true
SQL_TLS=true
SQL_TLS_DISABLE_HOST_VERIFICATION=true
SQL_TLS_ENABLED=true
SQL_HOST_VERIFICATION=false
SKIP_DB_CREATE=true
TEMPORAL_AUTH_ENABLED=true
TEMPORAL_ADDRESS=temporal:7233
TEMPORAL_TLS_REQUIRE_CLIENT_AUTH=true
TEMPORAL_TLS_CERTS_DIR=/etc/temporal/certs
SKIP_DEFAULT_NAMESPACE_CREATION=false
DEFAULT_NAMESPACE=default
TEMPORAL_CLI_TLS=true
TEMPORAL_CLI_TLS_CERT=/etc/temporal/certs/client.crt
TEMPORAL_CLI_TLS_KEY=/etc/temporal/certs/client.key
TEMPORAL_CLI_TLS_CA_CERT=/etc/temporal/certs/rootCA.pem
TEMPORAL_ADDRESS=temporal:7233

Then, create a .env.ui file to configure the webui:

TEMPORAL_TLS_CA_DATA=<REPLACE WITH BASE64 CERT rootCA.pem>
TEMPORAL_TLS_CERT_DATA=<REPLACE WITH BASE64 CERT client.crt>
TEMPORAL_TLS_KEY_DATA=<REPLACE WITH BASE64 CERT client.key>

You can create these base64-encoded values as follows:

echo TEMPORAL_TLS_CA_DATA=$(cat certs/rootCA.pem | base64 -w0) > .env.ui
echo TEMPORAL_TLS_CERT_DATA=$(cat certs/client.crt | base64 -w0) >> .env.ui
echo TEMPORAL_TLS_KEY_DATA=$(cat certs/client.key | base64 -w0) >> .env.ui

Temporal Configuration

Next to our docker-compose.yml, you need to create two configuration files, one for temporal-server and one for the webui.

Create base.yml:

global:
 tls:
 internode:
 server:
 certFile: /etc/temporal/certs/server.crt
 keyFile: /etc/temporal/certs/server.key
 requireClientAuth: true
 clientCaFiles:
 - /etc/temporal/certs/rootCA.pem
 client:
 serverName: temporal
 rootCaFiles:
 - /etc/temporal/certs/rootCA.pem
 frontend:
 server:
 certFile: /etc/temporal/certs/server.crt
 keyFile: /etc/temporal/certs/server.key
 requireClientAuth: true
 clientCaFiles:
 - /etc/temporal/certs/rootCA.pem
 rootCaFiles:
 - /etc/temporal/certs/rootCA.pem
 client:
 serverName: temporal
 rootCaFiles:
 - /etc/temporal/certs/rootCA.pem

services:
 history:
 matching:
 worker:

log:
 level: debug

And ui.yaml:

tls:
 caFile: /etc/temporal/certs/rootCA.pem
 certFile: /etc/temporal/certs/client.crt
 keyFile: /etc/temporal/certs/client.key
 enableHostVerification: false
 serverName: temporal

Spinning Up

Next, run docker compose up and see your services come to life!

You should see a whole mess of tables populate in your Postgres instance, as the auto-setup image does its thing.

Once the logs start to slow down and your server starts into a retry loop, go ahead and kill the process with Ctrl+c. Wait for the containers to spin down, and we have one change to make to our docker-compose.yml: swap out the word autosetup for server to change our specified container image. autosetup is meant to help you easily provision database tables and schema, but isn’t suitable for production deployments, so once our tables are set up, we switch it to the production-ready temporalio/server image.

Configuring NGINX as a TLS Reverse Proxy

We’re so close to complete! We only have one step left: setting up our nginx reverse-proxy with password auth.

Here’s a config you can drop in on top of /etc/nginx/nginx.conf (optionally: instead, add a sites file and link it to sites-enabled):

user www-data;
worker_processes auto;
pid /run/nginx.pid;
error_log /var/log/nginx/error.log;
include /etc/nginx/modules-enabled/*.conf;

events {
 worker_connections 768;
}
http {
 sendfile on;
 tcp_nopush on;
 types_hash_max_size 2048;
 include /etc/nginx/mime.types;
 default_type application/octet-stream;
 ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3; # Dropping SSLv3, ref: POODLE
 ssl_prefer_server_ciphers on;
 access_log /var/log/nginx/access.log;
 gzip on;
 upstream temporal {
 server 127.0.0.1:8080;
 }

 server {
 auth_basic "Temporal Admins Only!";
 auth_basic_user_file /etc/nginx/.htpasswd;
 server_name temporal.example.com;
 location / {
 proxy_pass http://temporal;
 proxy_http_version 1.1;
 # Ensuring it can use websockets
 proxy_set_header Upgrade $http_upgrade;
 proxy_set_header Connection "upgrade";
 proxy_set_header X-Real-IP $remote_addr;
 proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
 proxy_set_header X-Forwarded-Proto http;
 proxy_redirect http:// $scheme://;
 proxy_set_header Host $http_host;
 }
 }
 include /etc/nginx/conf.d/*.conf;
 include /etc/nginx/sites-enabled/*;
}

Add the following to your /etc/hosts:

cat << EOF >> /etc/hosts
127.0.0.1 temporal
EOF

Generate a .htpasswd file to control user authentication (you will be prompted to enter a password):

htpasswd -c /etc/nginx/.htpasswd <YOUR USERNAME>

Enable and reload nginx:

systemctl enable --now nginx && systemctl reload nginx

Setup Certbot (this is interactive and may prompt you for your email):

certbot -d temporal.example.com

Note: don’t forget to add certbot renew to a cron job! LetsEncrypt certs typically expire every 3 months, so check for renewals every 1-2 months.

Testing the Setup

Moment of truth: open your browser and navigate to https://temporal.example.com. You should be greeted with a Basic Auth prompt. Enter your username and password you generated earlier. Then, you should see Temporal’s Web UI homepage.

If you see something similar to the screenshot above, congratulations! 🎉 You now have a single-node Temporal server running with TLS. The connection is secure (check that browser lock icon), and you can start building or testing workflows. The UI will show zero workflows initially (since none have been run); it’s expected.

If you see errors regarding “no default namespace”, you can manually create one using the following command:

temporal operator namespace create --namespace default --tls-ca-path certs/rootCA.pem --tls-cert-path certs/client.crt --tls-key-path certs/client.key

Troubleshooting

Health Check

To further verify that gRPC traffic is working through TLS, you can use grpcurl to check the health probe:

~/go/bin/grpcurl --cacert certs/rootCA.pem --cert certs/client.crt --key certs/client.key -d '{"service": "temporal.api.workflowservice.v1.WorkflowService"}' temporal:7233 grpc.health.v1.Health/Check

If you see the following, temporal-server is properly configured:

{
 "status": "SERVING"
}

Creating a Test Workflow

To verify your task queues are set up, you can create a HELLO WORLD workflow (note the nested quotes, which are necessary):

temporal workflow start --address temporal:7233 --type HelloWorldWorkflow --task-queue=HELLO_WORLD_TASK_QUEUE --tls-ca-path certs/rootCA.pem --tls-cert-path certs/client.crt --tls-key-path certs/client.key --input '"Hello, World!"'

Cleaning up Tests

If you’ve created too many test workflows, you can Terminate them from within the Web UI or run the following to delete all of them:

for x in $(temporal workflow list --address temporal:7233 --tls-ca-path certs/rootCA.pem --tls-cert-path certs/client.crt --tls-key-path certs/client.key | awk '{ print $2 }'); do
 temporal workflow delete --address temporal:7233 --tls-ca-path certs/rootCA.pem --tls-cert-path certs/client.crt --tls-key-path certs/client.key -w $x
done

Connecting a Client

Because we have self-signed certificates, we need some way to get the certificates into our client.

Here’s how I do it in go (it will be similar in other languages):

package tclient

import (
 "context"
 "crypto/tls"
 "crypto/x509"
 "encoding/base64"
 "fmt"
 "math/rand"
 "os"
 "sync"

 "github.com/taigrr/temporal-worker/vars"
 "go.temporal.io/sdk/client"
)

var (
 tClient client.Client
 clientTex sync.Mutex
)

func New() (client.Client, string, error) {
 clientTex.Lock()
 defer clientTex.Unlock()
 if tClient != nil {
 return tClient, "", nil
 }
 // base64-encoded TLS certs created the same way we did for .env.ui
 clientCertData := os.Getenv(vars.TemporalTLSCertData)
 clientCertKeyData := os.Getenv(vars.TemporalTLSKeyData)
 serverRootCAData := os.Getenv(vars.TemporalTLSCAData)

 // the FQDN:PORT i.e. temporal.example.com:7233
 hostPort := os.Getenv(vars.TemporalHostPort)

 // default
 namespace := os.Getenv(vars.TemporalNamespace)

 // Decode the base64 encoded keys and certs
 clientCert, err := base64.StdEncoding.DecodeString(clientCertData)
 if err != nil {
 return nil, "", fmt.Errorf("unable to decode client cert data: %w", err)
 }
 clientCertKey, err := base64.StdEncoding.DecodeString(clientCertKeyData)
 if err != nil {
 return nil, "", fmt.Errorf("unable to decode client cert key data: %w", err)
 }
 serverRootCA, err := base64.StdEncoding.DecodeString(serverRootCAData)
 if err != nil {
 return nil, "", fmt.Errorf("unable to decode server root CA data: %w", err)
 }
 cert, err := tls.X509KeyPair(clientCert, clientCertKey)
 if err != nil {
 return nil, "", fmt.Errorf("unable to load cert and key pair: %w", err)
 }

 // Create Certpool to add the server root CA to the client pool
 certPool := x509.NewCertPool()
 certPool.AppendCertsFromPEM(serverRootCA)

 // Ensure no two workers have the same ID
 randString := fmt.Sprintf("%04x", rand.Intn(0x10000))
 // Embed the git commit of the currently running code into the worker ID
 commitHash := os.Getenv(vars.WorkerCommit)
 if len(commitHash) > 7 {
 commitHash = commitHash[:7]
 }
 workerID := fmt.Sprintf("%s-%s-%s", os.Getenv(vars.WorkerID), commitHash, randString)

 // Add the cert to the tls certificates in the ConnectionOptions of the Client
 clientOptions := client.Options{
 HostPort: hostPort,
 Namespace: namespace,
 ConnectionOptions: client.ConnectionOptions{
 TLS: &tls.Config{
 Certificates: []tls.Certificate{cert},
 RootCAs: certPool,
 },
 },
 Identity: workerID,
 }
 temporalClient, err := client.Dial(clientOptions)
 if err != nil {
 return nil, "", fmt.Errorf("unable to connect to Temporal: %w", err)
 }
 tClient = temporalClient
 return tClient, workerID, nil
}

Wrap Up

We’ve successfully deployed Temporal on a single node with Docker Compose and added TLS termination with NGINX.

In summary, we:

Introduced Temporal and set up a PostgreSQL-backed Temporal server using Docker Compose
Used Temporal’s auto-setup image to provision our databases
Switched to the Temporal production server
Deployed the Temporal Web UI and passed it through NGINX via a friendly HTTPS URL
Configured NGINX to handle TLS, forwarding web requests to the UI, thereby securing the traffic

This setup provides a starting point for working with Temporal’s powerful workflow capabilities. You can now write workflows in your language of choice (Go, Java, Python, etc.), point the SDK client to temporal.example.com:7233, and begin executing workflows on your server–just make sure your self-signed certificates are imported (see my go example above)!

Before we conclude, a few final notes and best practices:

Security: Your NGINX config can also restrict access by IP. If you intend to expose the Temporal UI or gRPC port more broadly, consider additional security layers, i.e. cloud infra-level firewalls
Persistence: Your temporal workflows are only as stable as your database. Again, I highly recommend using a stable, managed solution
Certificates: A time will come when your certificates expire. Be sure to have a rotation plan in place for scheduled maintenance!
Postgres: We initially granted sweeping access to the tuser in Postgres for our temporal and temporal_visibility databases. Now that the schema is set up, consider locking that down.

Temporal brings reliability superpowers to your applications – even in this modest single-node setup, you can develop and test workflows that will never give up in the face of failures. Now that you have Temporal running at temporal.example.com, it’s time to start orchestrating some workflows. Have fun exploring Temporal, and may your workflows always complete (eventually)! 🚀

Introducing grlx

Tue, 07 Nov 2023 19:17:25 +0600

Background

In a recent blog post, I discussed a few of the reasons why in 2021, I began working on a new Fleet Configuration Management tool to replace SaltStack. However, I did not go into much detail about what the replacement is, how it works, or the roadmap into the future.

As mentioned in the prior post, the first inspiration to build a new tool arose from requirements regarding constrained memory environments. In our typical install, salt-minion at idle required between 140mb and 300mb of memory. (This wide variance in metrics results from several factors including the method of measurement, the use case, the system uptime, the service uptime, and the versions of salt-minion and python being run. Regardless, this is simply too much memory for salt to be usable in many situations.)

For widely-deployed, 1GB IoT systems, sacrificing between 10-30% of all available memory for a background task is not ideal. But in a world where even wrist watches ship with 2GB of RAM, and IoT components are getting cheaper by the month, maybe 300mb is not that big of a deal?

For the virtual machine use case, I’d argue it’s still very important. Linode (now Akamai) offers VPS systems starting at 1GB/mo. Amazon Lightsail’s smallest offering comes with only half a gigabyte of memory. Imagine trying to scale horizontally across many nodes only for half of the memory you paid for to be eaten up by a system management tool running in the background.

Without repeating too much from the prior blogpost, I challenge the reader to pause for a moment and consider the hurdles with their own configuration management software, memory requirements aside: how easy was it to get started? What kind of maintenance is required to keep things running and secured? Does it break? How often? What is the DX (Developer eXperience) like? Is the current solution easy (and safe!) to use at 2am for an emergency rollback from your parents’ house on holiday?¹

When I set out to build this tool from scratch, I saw it as a golden opportunity to not only solve existing problems but also to shape the tool according to my vision. The goal was clear: to develop a tool that functions seamlessly, aligning to the needs of the landscape, including the creature comforts missing from other options. It should be low overhead, offer familiarity with existing tools, be secure by default, easily scalable, and resilient to OS upgrades (read: runtime dependency-free).

Introducing grlx - Effective Fleet Configuration Management

Without any further introduction, here are some of the major issues we are looking to address in grlx compared to other tools:

Low Overhead

I’ve mentioned this one a lot already, but it bears repeating. The primary reason that drove us to pursue building grlx was the unavoidable memory footprint of the interpreted language ecosystems. While our first choice was to simple take an existing tool and fork it, paring down the featureset and optimizing memory, we quickly realized this would be an exercise in futility–we might see slight improvement, but nothing on the order of magnitude we were hoping for. The CPU usage of grlx should also be lower for the same reason: no JIT compilation or JVM (looking at you, Closure) taking up intermediate cycles.

Dependency-free by default

A tool written in Python or Ruby depends on the environment’s Python and Ruby packages for its own stability. When a primary goal of a system and configuration management tool is to keep the system up to date, and updating the system involves updating system libraries such as Python or Ruby libraries, there is always a small but non-zero risk that a breakage in the ecosystem APIs will crash the management tool and prevent it from restarting, leaving the node (or worse, the whole fleet) stranded and unresponsive. Python 3.7 for example, introduced new reserved words to the language (await and async) which subsequently broke many packages that hadn’t been updated in time. Typically, these ecosystems move slowly enough for maintainers to update in time, but it’s never a good idea to be stuck between waiting for a new software release and keeping your ecosystem out-of-date and vulnerable to prevent breakages.

Scalable and Fault-tolerant

Built using NATS.io, grlx can take advantage of NATS’s clustering and fault-tolerance features. Using only a single farmer system, you can join several more NATS Server instances to the cluster and have them all share the load of incoming messages. Read more about NATS clustering here.

Future additions to grlx will allow for a single sprout to connect to multiple independent farmers, to allow for failover across control servers.

Secure by default

Did you know that it’s a violation of SaltStack best practices (specifically the linked Hardening Rules page) to use Salt over the internet without taking precautionary encryption measures? It commonly recommended to use a tool like stunnel or WireGuard® to add an extra layer between minions and the master.

With grlx, there’s no need for a third-party encryption suite to stay secure. All communications from cli to farmer, and from farmer to sprout, are encrypted using self-signed TLS certificates and NATS.io NKey encryption. These certificates are pinned to the clients on first connection as an extra security precaution.

Easy to set up

Easy to automate. Need it be said again? Only one binary (and one configuration file) is required to provision a sprout (node).

We also offer a bootstrap script hosted at bootstrap.grlx.dev that will always get you the latest and greatest release in a simple one-liner, if you so choose.

Easy to Understand

Go is often touted as simple language. All grlx code is written in Go (except for the Web UI), and we explain how both the overall system and the individual components work in the documentation. Auditing the grlx codebase is easy, and most of the source code can be read through in a day or so.

Our non-stdlib build dependencies are selective–kept to as short a list as reasonably possible, so there’s little happening “elsewhere.” We follow best practices, strive for high test coverage, and keep an eye on our Go report card.

Open Source

It would be crazy to install a tool on a fleet of systems and give it root access without the ability to do some introspection on what’s happening under the hood. All the source is available on GitHub, where I encourage you to give it a star!

Not only is grlx open and free, it’s also permissively licensed as 0BSD and is therefore compatible with nearly any enterprise company’s policy on OSS software.

Keep in mind, the logos, the mascot (also referred to as “Clove”), the name “grlx” itself, and the overall brand are copyrighted, all rights reserved, etc. Do not try to pass off grlx as your own–but forking and renaming is allowed!

Supported with Corporate Backing and Buy-in

grlx has entered into an agreement with ADAtomic, Inc., to offer official, commercial support. If this is something that would be valuable to your company or organization, please contact ADAtomic here.

Additionally, grlx is already seeing production use from several companies, listed in the README. As mentioned above, grlx development is being driven by a real need at our organization, and is seeing heavy usage in our own lineup of products.

Extensible Plugin System

To state the obvious, not every possible feature is required by every customer. To keep grlx light and fast, all endpoints and features take advantage of dependency injection and there is a well-defined interface already in place for loading plugins at runtime. These plugins can offer features such as supporting obscure package managers or downloading files from uncommon endpoints. In short order, two example file endpoint provider plugins will be released, one to download files from IPFS, and another to download from a BitTorrent magnet link. Hopefully, these examples indicate both the reason for and capability of the plugin system. Using runtime-loaded plugins is also a boon for development and testing, as mainline code that should be added to grlx can start as a plugin for development, and get merged over time.

At first, the plugin interface will only support Go plugins, but support for WASM plugins is on the roadmap, allowing for plugin development in nearly any compiled language.

To be clear, grlx is dependency-free by default, but we recognize there are some use cases where optional plugins may offer increased functionality without adding significant hooks into environmental state. Both Go and WASM plugins can be distributed as single files, dropped into a directory and hot-reloaded automatically at runtime.

Configurable Message Delivery Rules

Some tools only allow for deploys to go out the door to nodes that are online at the time of the push–if it’s not online right now then you have to either watch and wait for it to come online, or write some scripts to programmatically check for the next time it does come online and push the release then. In a datacenter, this is not a big deal as you should have extremely high (near 100%) uptime and connectivity between the CNC Server and the nodes. For an IoT deployment on the other hand, connectivity can be tricky, and the aforementioned scripts must come into use.

However, once you start writing them, you might realize that there’s a lot more involved regarding error checking, reporting, logging, etc., that the simplest while ! online; do sleep 60; done && push script cannot handle. You end up writing a whole deployment framework around trying to catch your node when it’s online and push to it at that exact moment, or maybe you add a cron job to the node itself to trigger an agentless or standalone mode periodically.

This whole mess could have been avoided if the configuration management tool provided a utility for you, which stores the deploy jobs into persistent queues to ensure delivery on next reconnection. NATS.io’s concept of Durable Queues allows for exactly this behavior.

RBAC

grlx takes a Role-Based Access Control (RBAC) approach for designating who is allowed to deploy what–and where. Secured service worker accounts or admins might be allowed root access to all machines. Your coworker John, who manages some nginx configurations, can cook (push out) the nginx recipe to all sprouts, and Celine has access to cook all recipes but only on select development/canary nodes. Future work will allow these roles to come from an LDAP endpoint or other external service provider plugin.

More information about the RBAC system can be found in the documentation.

Usability Improvement Features

Below, find some of the small tweaks and differences to make using grlx easier than competing tools:

Advanced Developer Tooling

Our development team is hard at work creating an LSP server and Linter for grlx. The roadmap includes support for jumping to import definitions, full syntax highlighting (even in recipe files containing templates!), and property checking. Our plugin interface also includes definitions to generate custom LSP inputs so that the tools you might need to build in-house are treated as first-class citizens in your editor.

Official GUI and TUI built-in

Many home-grown WebUIs have been built to support existing solutions which don’t come with official WebUIs. The grlx command line utility will soon support spinning up a local webUI, server over localhost, which can be used to access most functionality. Communications to the farmer are proxied through the CLI, so there’s no need to open up ports on the farmer to access the WebUI.

A TUI (Terminal UI), built with BubbleTea is on the roadmap.

Optional DropShell Support

While grlx already supports arbitrary command execution, sometimes it happens where you really just need a shell to do some in-depth troubleshooting. For SaltStack, I developed TunnelBunny, a process for setting up port forwarding and reverse shells to minions, which you can read about here. grlx will support a similar feature, called DropShell, which will allow you to drop into a shell on a single sprout, from your local machine, without having to open up any ports on the minion, configure SSH, or drop firewall rules. This feature is still in development, but will be available once RBAC is fully implemented, as it might be a security concern for some organizations.

High test coverage

We strive for 100% test coverage on that parts that matter. Specifically, all ingredients are thoroughly unit tested, and the core functionality of the platform runs through a suite of integration tests inside of a docker-compose environment.

No Server Access Required

The command line utility is separate from the farmer, and can be run from any machine with network access to the farmer. All communication is encrypted using TLS and NKey encryption, so there’s no need to provide a VPN or SSH tunnel to the farmer. Depending on company policy or developer preference, the CLI can be run directly on developer machines, or from a dedicated bastion server. Authentication and authorization is handled on a per-user basis (to support RBAC), so there’s no need to share credentials, SSH keys, or configure sudo access to the main CNC server.

Supports GitOps

The command line utility cook command will soon support a --git flag, which will tell the farmer to check out a particular branch, tag, or commit of a git repository, and use the files in that branch as the source of truth for the recipes. Additionally, the currently checked-out commit, tag, and branch are made available to the recipe files as property variables, so you can use them in your templates should the need arise. There is also a command to update the recipe’s git repository to the latest commit, tag, or branch, and a command to list all available branches, tags, and commits.

Synchronous Job Status reporting

One of the most frustrating things about using a configuration management tool is the lack of visibility into what’s happening, as it’s happening. When submitting a job, you might get a job ID back for polling, or wait synchronously for the job to finish and see the summary of results but most tools won’t let you watch the states finish as they happen. Most of the time, this is fine, but the few times you need live introspection into why a deploy is stuck, you’re out of luck.

grlx will soon support a --watch flag on the cook command, which will stream the job status to the terminal as it happens, with a live progress bar and summary of results at the end.

Restful API

The farmer exposes a RESTful API for all of its functionality, so you can build your own tools around it. If you don’t like the built-in WebUI, you have the option to build your own! If you need to build automations around your deployment from a third-party tool, you can do that too! Everything from cooking to command execution is available via the API.

More information about the API can be found in the documentation.

Lifecycle Hooks for Outgoing Webhooks

Do you need to track one particular step of a deploy for a particular reason? Maybe you need to know whenever a specific file is created or modified? Recipe files have a field for providing an upstream webhook URL and JSON body you need sent. Hooks can also be configured to fire off WebHooks when a deploy starts, fails, or finishes. Additionally, you can configure a hook to fire when a new sprout is added to the fleet, or when a sprout’s online status has changed.

Introspection and visibility has never been this easy!

Friendly Discord

Our Discord invite is open to one and all, where you can chat with the developers of grlx and get involved in the grlx community.

Meet like-minded DevOps professionals and swap ideas about the best way to get something deployed–or embroil yourself into a flame war over Cloud or On-Prem–as long as you keep things respectful it’s up to you!

Cute Mascot!

His name is Clove, isn’t he great?

Summary

In summary, I hope this article has presented the litany of reasons for a new tool in the configuration and system management space. Our roadmap is clear, and we are excited to announce that version 1.0.x is available for download now. Please see ADAtomic.com for more information!

Updating or rolling back production from your parents’ house is not a recommended DevSecOps or GitOps practice, nor am I endorsing it in any way. Ideally, you’ve got canaries and auto-rollbacks, split traffic, staging environments… This statement is meant to be humorous, don’t take it literally. ↩︎

So Long, Salt Project

Mon, 09 Oct 2023 18:19:25 +0600

This post has been a long time in the making. Anyone who has followed me and my work for any period of time probably knows by know that I’ve been working on grlx (pronounced ‘garlic’) for some time now (since mid-2021). But why? What’s wrong with SaltStack? More specifically, what problem am I trying to solve that doesn’t already have a solution solved many times over?

A Note to the Salt Project

First of all, I want to make something clear: this post is not an attack on Salt, Thomas Hatch, or any of the people involved in the Salt Project. I’ve been a long-time user of SaltStack, going back to the Boron release in 2016. SaltStack has gotten me really far, and I owe much of my career’s success to Salt, which I have carried with me from job to job.

I also want to make a call-out here: my use case for Salt is not necessarily the target use case, and it’s really unfair to criticize a screwdriver for being a bad hammer, especially when the screwdriver is free.

Real-world Requirements

At my first role as a Salt user, I was not the decision maker. I was a junior developer, working with the tools selected by peers and higher-ups, and Salt was chosen to be part of our stack. Without going into too much detail, here was our problem statement:

We are building IoT, embedded Linux boards
We need a way to update the software and packages remotely
We are using field-prototypes, which don’t have fixed or public IPs and therefore need to be the Client in a Client-Server model (agentless solutions won’t work)
We are not just updating application code, our environments are going through rapid updates (emphasis on field prototypes)
Our distribution is a custom Yocto build, without a package manager, so we need a way to declaratively replace system-level packages transactionally
The ability to remotely run shell commands and return the output is a plus (we might need to trigger fswebcam and snap a photo as a one-off)
Rolling out instant updates to targeted devices (where a ’target’ could be a small sampling, the whole fleet, or a single device) is a huge boon
Some devices will connect over 3g cellular as not all of our field installations have WiFi available

As this platform was a product of its time, here’s some additional information to keep in mind:

The year is 2016, and the hardware development went back to 2012 (so pre-Raspberry Pi GA)
Google IoT Core doesn’t exist (yet, and also, RIP)
Amazon GreenGrass is not generally available (until 2017)
Ubuntu Core IoT was prohibitively expensive and didn’t support our custom hardware

SaltStack, in theory, checked all of our boxes, and allowed us to ship units to the field for testing, with remote, OTA updates.

Full Linux support? Check. [x]
Remote updates using a client / server model to get around the IP contstraints? Check. [x]
Ability to upgrade system files in addition to application files? Check. [x]
Remote command execution? Check. [x]
Specific device targeting? Check. [x]

And so, our small team set out to create a configuration management system built on top of SaltStack as the center pillar. Yes, we even managed the SaltStack installation with SaltStack, by replacing the Python libraries and .egg/wheel files with Salt states. We were careful in how we did this, and actually added a cron-based backdoor ‘rescue’ service in case the snake swallowed its own tail, but it was a calculated risk. In the end, the deployment worked, but it wasn’t without headaches.

Issues and Complaints

Dependencies

There were several runtime and compile-time bugs related to the dependencies. Anybody who’s been around SaltStack long enough will shudder at the mention of Tornado or PyCrypto. We were especially subject to the issues regarding pycrypto compilation, as our Yocto project build ran on ARMv5 (oh whoops, did I not mention that requirement earlier? That sucks, but we’re all engineers here and you just have to learn to roll with it…) and there weren’t precompiled wheels for our architecture–we had to build them ourselves. Memory was always running low, as we had less than 1GB of physical memory, and most of that needed to go to the application for some on-device image processing.

Crashing

There was a memory leak, somewhere, in either salt-minion itself, or in a dependency. It was difficult to isolate, and I’m not sure my team actually ever found it. It was likely patched in a later version of salt-minion (again, or in a bumped dependency). Our solution was to limit the memory used by systemd and tell the unit to crash and restart if it went over. This was not ideal. Systemd had no visibilty into how we were using salt-minion, whether we were killing the process due to a real memory leak or due to a spike in usage associated with an active software update. It’s reasonable to allocate more memory to salt-minion and away from our application during a software update, but not at the expense of allowing run-away memory usage during regular runtime. Allowing systemd to indiscriminately OOM salt-minion becomes even more dangerous if we are currently applying a state to update the system or python libs themselves–a crash mid-update for core and critical python libraries might prevent salt-minion itself from ever coming back up. In the end, we had to isolate the states for the system by themselves and run those states very selectively, not in the top/high state, which somewhat defeats the purpose of the ‘statefulness’ Salt brings you–but that’s not Salt’s fault, as it certainly wasn’t designed for that. Hammer, screwdriver, all that. Really the problem here was the memory leak itself, as that’s the cascading issue.

Connectivity issues

Memory leaks and crashing aside, we had a lot of trouble keeping the zeroMQ connection active. Now, we did have many systems connected over 3G cellular, but the problems weren’t isolated to those units. Our WiFi-connected units also had issues maintaining connectivity. The dreaded Minion did not return. [Not connected] message was a commonplace error. More disturbing, early versions of Salt didn’t allow you to easily configure the retry logic. Sometimes, the connection would break, and the client wouldn’t ever try to reconnect, so it would stay disconnected until the memory leak would trigger an OOM, which would in-turn restart the salt-minion daemon. ( This has since been addressed.)

Access control

Whoever needed to send the updates out to the units in the field needed to be root (or have sudo access to run salt) on the Salt master, which means someone needed shell access to the CNC server. From a security perspective, this is an ick.

I loved SaltStack

Yes, I’ve had my fair share of problems with SaltStack, but my teams and I made things work. When working with software of this scale, some tips and tricks tend to reveal themselves over time, (like using get_template to pre-render salt states before applying). Sometimes, you just need the tribal knowledge gained through experience. Several issues were fixed by the Salt Project team, others have gone away simply through a revision to requirements on my end, and still others I’ve learned to live with or work around.

I’ve brought SaltStack with me to 4 other career roles since my introduction to it, like some kind of DevOps Evangelist (I’ll likely write about these projects at a later date). I didn’t have enough good things to say about Salt. I became very involved in the community Slack channels. I answered a question or two on Stack Overflow regarding SaltStack. I’ve had a minor PR merged into salt-bootstrap. I’ve been featured on the Salt Community Open Hour podcast. The project called me out in an advisory bulletin to thank me for my help on CVE-2020-11651 and CVE-2020-11652.

Acquisition Heartache

In October, 2020, VMware acquired SaltStack, and the branding was changed to Salt Project. Large portions of the team were made redundant (read: laid off) and the focus of the project seems to have shifted to cloud-native and VMware-related projects. The name change alone causes a bit of a headache. The Salt Project is already a(n entirely unrelated) thing. Changing from “SaltStack” to “Salt” makes finding security issues difficult when SALT exists. The overall “Googleability” of the project has gone downhill, which really isn’t great when some of the page redirects are broken for all pre-hydrogen releases.

The number of unresolved issues in the project have skyrocketed (even despite the stale bot doing numbers on these issues). It seems the reduced team size and increasing other priorities has prevented development from keeping up with security and bugfixes in some dependencies, and the ‘solution’ is to freeze the updates and vendor the dependencies. That’s right, until recently, SaltStack has been vendoring a copy of the Tornado library, despite publicly-known CVEs.

Salt has also recently changed their packaging format to Onedir, effectively vendoring all dependencies inside of a single folder (including python itself). While this sounds great on the surface, keep in mind it will prevent non-breaking, API-compliant hotfix changes from making it down to the user through their system package managers, so users are now completely at the mercy of the Salt Project’s timelines and priorities, and their responsiveness record there is…not perfect in my opinion.

Where To Go From Here

Perhaps now you see why I’ve started grlx. I need to fill a SaltStack-sized hole in my heart; build a tool that solves all the same problems Salt did for me, but also address my own issues with the tool. grlx will never have 1:1 feature parity with SaltStack, especially as the Salt Project focus shifts to VMware first-party products and integrations, but for people like me, that’s a pro, not a con. In grlx, our goal is to support all the things that make SaltStack great–the extensibility, human-readable configuration as code, lightning fast remote execution–and do it with a small footprint, secure by default.

It’s my feeling that SaltStack probably isn’t to blame for its memory usage either–it is written in Python, and that’s just the price paid for an interpeted language. For this reason, among others, grlx is written in Go, a compiled language. Similarly, the connectivity issues are likely not part of the Salt codebase, and are probably part of the zeroMQ dependency. For grlx, we’re using NATS.io instead, and we’ve even submitted code upstream in an attempt to be good Open Source stewards.

If you’d like to follow the development of grlx, there are several ways you can do so.

We have a Twitter/X account here.
We have a community discord!
You can also star our project on GitHub using the button below:

Star

I’ll also be releasing several posts in the coming weeks going into more detail about the features and roadmap for grlx, so hit that RSS button at the top of the page to make sure you don’t miss anything!

Frontend FOMO: A Hype-Driven Ecosystem

Tue, 12 Sep 2023 18:19:25 +0600

Ah, JavaScript development, the ever-evolving frontier of web development. New runtimes, package managers, libraries, frameworks, module systems, and packages pop up like mushrooms after a rainstorm. Every week, it feels like a tech carnival, with flashy new tools and configurations tempting you to hop aboard the latest hype train.

Let’s start by acknowledging that innovation is a positive force for good. The JavaScript ecosystem is a hotbed of creativity, where developers are constantly pushing the boundaries of what’s possible. New libraries and frameworks emerge to solve real-world problems, making developers’ lives easier and applications more powerful.

Innovation itself is not the problem. The problem is the collective behavior of developers, often resembling a herd of lemmings, that can lead to chaos. Front-end (and consequently Full-Stack) developers seem to be particularly susceptible to FOMO and analysis paralysis. Once (or even before) a developer starts to feel comfortable with a tech stack, a new kid on the block arrives, and some begin suddenly questioning every decision they’ve ever made.

Imagine this scenario: you’ve been working with React for a while, building beautiful (let’s be honest, janky but functional) user interfaces, and you’re finally getting the hang of it. But wait, Svelte comes along and promises to revolutionize the way you think about front-end development. You start to wonder if you’ve been living in the Stone Age and if Svelte is the key to enlightenment.

Perhaps you’re knee-deep in Node.js, happily coding away server-side, when Deno makes its grand entrance, boasting improved security and a new approach to runtime environments. Suddenly, your Node.js project feels like last year’s news, and you can’t help but question if Deno is the future. Nota bene: Bun was released right before this article was published, and its release only makes my point stronger.

FOMO leads to a never-ending cycle of tool evaluation, configuration tweaking, and second-guessing. It’s like being stuck in a revolving door that never stops spinning. You find yourself constantly switching between libraries and frameworks, never truly mastering any of them, and your deliverables suffer from analysis paralysis.

I’ve been picking on Front-end developers, but let’s be fair: backend developers face choices too; I’m all too familiar with the Go / Rust / Zig argument (and I’ll save Ruby on Rails and PHP for a future date). There are so many general-purpose languages (GPLs, as opposed to domain-specific languages, DSLs) that, of course, there is bound to be arguing, evangelizing, and a general lack of consensus.

However, I would posit the arguments that arise over which backend language to use are of an entirely different class than the arguments that arise within the JavaScript ecosystem.

Your average Go developer and Python developer get into an argument over whose language is Better. What are their talking points? Typically, the Python developer will criticize Go for having a smaller package ecosystem, and no significant support for Machine Learning, praising Python for ease of development – it’s often faster to get a prototype out the door for Python than for Go.

The Go developer will then come back complaining about Python’s slower speed, dynamic typing, and runtime package management. If both parties are reasonable, they may agree that while both languages are “General Purpose,” there are still “domain preferences” for each language. In other words, there are problems and use cases where one language is the better tool for the job, even if both could work. (I’m aware the comparison between a compiled and interpreted language may be more extreme than between Go and Rust or Python and Julia, but the point still carries.)

Now look at an argument between a Vue user and a React user. The Vue user might complain about JSX syntax, the corporate backing (Meta), and the relaxed project scaffolding (the strict 1:1 component to file rule means all Vue projects end up organized somewhat the same, without requiring developers to search across files to find where a component is defined.) Conversely, React developers insist ‘the future is React’, more developers know React (hiring talent is easier), and complain the strictness of Vue’s scaffolding leads to verbose, sprawling directories and ReallyLongAndSpecificComponentNames.vue (getting Java flashbacks, anybody?)

Of course, I won’t pretend all arguments about these libraries are the same. React does mobile better (thanks to React Native). Vue does incremental adoption better. (React and Vue have gone back and forth on which has better performance.)

So, while there are Domain arguments to be had across JavaScript frameworks, the point I am making here is that the loudest and oft repeated disagreements over JavaScript stacks are about Developer Experience (DX) and preference, rather than which stack can do the job better in production.

I still don’t think these arguments over preference are The Problem With The JavaScript Ecosystem™. Different developers will make judgment calls for one choice over another. But if the tools can all (mostly) get you to the same, functional result, and it’s really a DX and preferential choice question, FOMO is the problem. The most important factor for seamless DX is the stability of the ecosystem: building components and shipping changes to production should become muscle memory over time. This muscle memory can never develop when the developer is constantly switching between tools and libraries.

Backend developers listen to the arguments over which JS stack does what and how. Inevitably overwhelmed, they retreat back into their caves to write some of the finest, purest Haskell ever conceived, then sit back with a glass of brandy admiring their monads (or something like that, I don’t know, I don’t write Haskell) leaving the Scary Frontend World alone.

It’s generally acknowledged that Linux “distro hopping” is an unhinged practice fit for masochists and the clinically deranged. It’s high time we collectively agree stack churn in the JavaScript Ecosystem is a comparable vice. What is the solution then? Where’s a good tradeoff between stifling innovation and maintaining stability?

Be pragmatic. If you’re a library maintainer, keep building awesome tools, but don’t expect everyone to use them right away. There are already so many moving pieces. If what you’re building is truly revolutionary, it will stand the test of time and eventually become a standard (whether you like it or not!).

If you’re building a product, you need to ship. Leave the bleeding edge, the framework wars, and library hopping to the maintainers of the tools. Chances are you don’t need to be an expert in every new library that comes out.

Stop, settle, and pick something to move forward. If you can’t decide where to start, then vox populi vox dei: start with Next.js. Get really good at it. Only switch up your stack if there’s a feature your platform can’t function without. It’s time to focus on shipping, not deliberating. I regretfully inform you, that even if you can’t take advantage of the latest shiny, you’ll be ok!

Setting up a WireGuard VPN for Remote Access

Thu, 24 Mar 2022 18:19:25 +0600

What is WireGuard?

WireGuard is a VPN, which tunnels traffic over encrypted, secure UDP packet streams. Once a tunnel is established, the endpoints are allowed to roam (change IP addresses, such as might happen switching between WiFi networks or when connected to a cell tower). It is special when compared to other VPN technologies for many technical reasons, which led to its rapid adoption into the Linux Kernel, but three features in particular make WireGuard stand out: the CryptoKey Routing Table, the way public/private keypairs are handled, and the “silent by default” behavior.

According to the wireguard website,

WireGuard® is an extremely simple yet fast and modern VPN that utilizes state-of-the-art cryptography. It aims to be faster, simpler, leaner, and more useful than IPsec, while avoiding the massive headache. It intends to be considerably more performant than OpenVPN. WireGuard is designed as a general purpose VPN for running on embedded interfaces and super computers alike, fit for many different circumstances. Initially released for the Linux kernel, it is now cross-platform (Windows, macOS, BSD, iOS, Android) and widely deployable. It is currently under heavy development, but already it might be regarded as the most secure, easiest to use, and simplest VPN solution in the industry.

WireGuard Basics

Unlike traditional VPN technologies, WireGuard has no concept of Client or Server, but instead all nodes on the network are simply Peers, connecting Point-to-Point. This allows for some really interesting scenarios where packets sent through the VPN can be sent through very specific routes by setting IP masks appropriately, and if all Peers on the VPN have public IP addresses (or else they can ‘see’ each other in some other way, perhaps by sharing an insecure local network), the single point of failure built into the Hub-and-Spoke model becomes optional.

However, even if WireGuard doesn’t specifically recognize the Hub and Spoke model, it’s still a valid configuration,and we’ll be using it for this tutorial. Let’s see how easy it is to configure a Hub and Spoke-style, simple WireGuard VPN ready for personal or commercial use.

Server Instructions

Choose a private IP block to reserve. In order to use WireGuard inside of a Spoke and Hub paradigm, it’s necessary to reserve a block of IP addresses for all the nodes. (When running Peer to Peer, it’s not necessary for these addresses to share a subnet, as a CIDR of /32 can be used for each peer.) Private IP ranges include:
- 10.0.0.0 to 10.255.255.255
- 172.16.0.0 to 172.31.255.255
- 192.168.0.0 to 192.168.255.255
An example range for WireGuard might look like: 192.168.2.1-192.168.2.255 (be sure this range doesn’t conflict with that of your router). In CIDR notation, this range would be: 192.168.2.0/24. You can use a CIDR Calculator to determine the proper format for a given range.
Choose a port for your tunneled traffic to use. WireGuard traffic flows over UDP, and by default uses port 51820. For this tutorial, we will assume you are using the default port, but if you are running multiple WireGuard VPNs on the same server, you will need to choose a different port for each one.
Choose a cloud provider that provides you with a public IP address. My personal preference is Linode/Akamai (by using my signup link you’ll get free credits, as will I.)
Once you are ready to spin up a server, be sure to choose a distribution/flavor which ships Linux 5.6 or above. (Some versions of Ubuntu with older kernels also received backports, but if you have the option, go with the latest stable release available.) It is possible to use WireGuard in Userspace (if you don’t know what this means, don’t worry about it, just use a newer installation image) but it is much slower than the version running in the kernel. Going forward, we will assume you have the public IP of 93.184.216.34 (used for example.com).
Install WireGuard and its helper utilities. The name of the helper utilities package is usually wireguard-tools, and installing that package should bring in wireguard as well. If you don’t already have iptables installed, you will need to install that as well. (On Debian-based systems, run apt install wireguard-tools iptables.)
Enable IP forwarding. This is a necessary step for the server to act as a router. On Linux, this is done by running sysctl -w net.ipv4.ip_forward=1. To make this change permanent, edit or create /etc/sysctl.conf and add the line net.ipv4.ip_forward=1.
Create a keypair for your server. Once WireGuard is installed, you can generate a private key by running wg genkey. To get the public key, run wg pubkey and paste (or pipe) in the private key. This keypair is just like an SSH public/private keypair, so keep it safe!
Use the keypairs to create a configuration file. The next step is to create a configuration file for your server. There are three main pieces of information required:
- the interface and port to listen on
- the WireGuard private IP and netmask (CIDR Notation) you are committing to use (such as 192.168.2.1/24)
- the Private Key of your server
Create a file at /etc/wireguard/my-vpn.conf and add in the following:
```
[Interface]
ListenPort = 51820
PrivateKey = <SERVER PRIVATE KEY HERE> # <SERVER PUBLIC KEY HERE>
Address = 192.168.2.1/24
PostUp = iptables -A FORWARD -i %i -j ACCEPT; iptables -A FORWARD -o %i -j ACCEPT; iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
PostDown = iptables -D FORWARD -i %i -j ACCEPT; iptables -D FORWARD -o %i -j ACCEPT; iptables -t nat -D POSTROUTING -o eth0 -j MASQUERADE
```
Many systems use eth0 as the name of the primary network interface, but if yours is different, you will need to change it in the PostUp and PostDown commands. If you are unsure of your default gateway’s interfacve name, run ip a and look for the interface with the public IP address you are using to connect to the server.

Additionally, you will add a block for each Peer, but we’ll need to generate those keypairs first. More on that later.
Be sure the connection port is exposed through your hosting provider (for UDP traffic). Instructions for Linode can be found here. You may need to consult your provider’s documentation for more information.
Before we start up our server, we need some more information from our clients. We’ll come back to the server later.

Client Instructions

Linux instructions:

Ensure your kernel has support for WireGuard, or install the userspace version. On Ubuntu/Debian-based systems, run apt install wireguard-tools
Create the folder /etc/wireguard/
Generate a new keypair the same way you did on the server: wg genkey
Place the keypair into a config file named my-vpn.conf and save it, along with the details you got from the server. The format is as follows:

[Interface]
PrivateKey = <CLIENT PRIVATE KEY HERE> # <CLIENT PUBLIC KEY HERE>
Address = 192.168.2.4/24 # Or whatever IP you want your client to use on the VPN
[Peer]
PublicKey = <SERVER PUBLIC KEY HERE>
AllowedIPs = 192.168.2.0/24 # This is the subnet you chose for your VPN, and should match the server's
Endpoint = 93.184.216.34:51820 # This is the public IP of your server, and the port you chose
# Note you can also use a domain name, if you want:
# Endpoint = example.com:51820
PersistentKeepalive = 25 # allow your client to roam without dropping the connection

chmod the config file and the /etc/wireguard directory to 400 and chown it to root:root.
On systemd-based systems, run systemctl enable wg-quick@my-vpn.
Move on to the server section, when complete move on to the next step.

Mac instructions:

Install Wireguard for Mac
Follow prompts to fill in the info from your server.
Choose an IP from the subnet you chose for your VPN.
Save your Public Key for later.
Enable start on demand.
Complete the server steps next, before hitting Save.

Final Server Instructions

Inside of your my-vpn.conf file, create a Peer block following the pattern of the previous entries:

[Peer] # name of client
PublicKey = <CLIENT PUBLIC KEY HERE>
AllowedIPs = 192.168.2.<n>/32 # replace `<n>` with the last octet of the Client's WireGuard private IP

Save and close the config file.
Ensure the config file is owned by root and has permissions of 400.
Run systemctl enable --now wg-quick@my-vpn to start the WireGuard tunnel and set it to start automatically on boot.
Continue setting up from the client side.

Linux

Same as for the server, run systemctl enable --now wg-quick@my-vpn

Validate the VPN is running with wg. You should see something like this:

interface: my-vpn
public key: <CLIENT PUBLIC KEY HERE>
private key: (hidden)
listening port: 58341
peer: <SERVER PUBLIC KEY HERE>
endpoint: 93.184.216.34:51820
allowed ips: 192.168.2.0/24
latest handshake: 13 seconds ago
transfer: 5.37 MiB received, 6.40 MiB sent
persistent keepalive: every 25 seconds

If you see an error, check the logs with journalctl -u wg-quick@my-vpn.

MacOS

Hit Save.
Optionally, in your Mac’s System Preferences, go to Network and check the box for Enable VPN on Demand.

Next Steps

You should now have a working WireGuard VPN, ready for roaming and personal use. There’s only one client for now, but you can add as many as you like by repeating the client steps on each new device. (Be sure to pick a unique private IP address inside of the subnet you chose for your VPN for each client.) Then, you can jump directly from one client to another using the WireGuard private IP addresses. This can be particularly useful when pairing the WireGuard app for Android with Termux or JuiceSSH to access your laptop or desktop on the go.

Once the client side setup is done, simply add a Peer block for each client in the server’s config file, using a unique private IP address, then run systemctl restart wg-quick@my-vpn on the server to restart the tunnel with the new configuration. It is normal for the tunnel to drop when you run the restart command, but it should come back up within a few seconds.

If you have any issues, check the logs with journalctl -u wg-quick@my-vpn. If you’re stumped, feel free to reach out to the community in my Discord, and someone may be able to help you out.

Going Further

If you want to take your WireGuard VPN to the next level, you can add a DNS server to your server’s config file, and then add a DNS entry to each client’s Interface block. You can read more about that here.

It’s possible to use WireGuard to route all of your traffic through the VPN, pair WireGuard with netns or Docker containers to force applications to use the VPN, or even use WireGuard to create a mesh network. I’ll likely be writing more about these topics in the future. For now, there are great resources available on the WireGuard website and on YouTube.

“WireGuard” and the “WireGuard” logo are registered trademarks of Jason A. Donenfeld.

cache cascade

FreeCodeCamp Podcast Announcement

Why I Chose Go for grlx Instead of Rust

Why Not Rust?

Familiarity and Developer Productivity

Performance in Practice

Type Systems and Safety Guarantees

Iteration and Flexibility

Ecosystem and Tooling

Readability and Auditing

Security via Simplicity

Conclusion

Using PAM for SSH Login Telephony

Real-Time SSH Login Alerts Using PAM and Gotify

Prerequisites

Step 1: Install and Configure Gotify

Step 2: Create the PAM Notification Script

Step 3: Update PAM Configuration

Step 4: Enable and Test It

Appendix: PAM Environment Variables

Considerations

Conclusion

Temporal Server: Self-hosting a Production-Ready Instance

Overview of the Deployment

Prerequisites

Setting Up Our Docker Compose

Configuring Postgres

TLS certificates

Configuring Env Vars

Temporal Configuration

Spinning Up

Configuring NGINX as a TLS Reverse Proxy

Testing the Setup

Troubleshooting

Health Check

Creating a Test Workflow

Cleaning up Tests

Connecting a Client

Wrap Up

Introducing grlx

Background

Introducing grlx - Effective Fleet Configuration Management

Low Overhead

Dependency-free by default

Scalable and Fault-tolerant

Secure by default

Easy to set up

Easy to Understand

Open Source

Supported with Corporate Backing and Buy-in

Extensible Plugin System

Configurable Message Delivery Rules

RBAC

Usability Improvement Features

Advanced Developer Tooling

Official GUI and TUI built-in

Optional DropShell Support

High test coverage

No Server Access Required

Supports GitOps

Synchronous Job Status reporting

Restful API

Lifecycle Hooks for Outgoing Webhooks

Friendly Discord

Cute Mascot!

Summary

So Long, Salt Project

A Note to the Salt Project

Real-world Requirements

Issues and Complaints

Dependencies

Crashing

Connectivity issues

Access control

I loved SaltStack

Acquisition Heartache

Where To Go From Here

Frontend FOMO: A Hype-Driven Ecosystem

Setting up a WireGuard VPN for Remote Access

What is WireGuard?