Setting up a WireGuard VPN for Remote Access
What is WireGuard? WireGuard is a VPN, which tunnels traffic over encrypted, secure UDP packet streams. Once a tunnel is established, the endpoints …
Read ArticleTemporal is an open-source workflow orchestration engine designed to manage, execute, and monitor complex workflows and activities. In plain terms, Temporal allows you to write crash-proof code: it ensures that your long-running processes (workflows) continue execution reliably even if your services (or external services!) crash or restart.
Temporal’s
walkthroughs and CLI provide extremely easy adoption and low-friction for getting started: the temporal CLI contains an embedded development server (temporal server start-dev
), which binds to localhost and allows for easy, unauthenticated workflows.
Connecting to Temporal’s managed service is similarly easy, but at writing, has a minimum cost of
$100/month, which is not always affordable for smaller orgs, especially when running a production-grade self-managed server can cost less than a quarter of that.
In this article, we’ll set up a single-node Temporal server backed by PostgreSQL, and we’ll secure it with TLS using NGINX as a reverse proxy.
The end result will be Temporal’s services running in Docker containers, accessible at https://temporal.example.com
(our example domain) with encrypted connections.
Note: This setup is suitable for small-scale, development and production deployments. Temporal is designed to scale horizontally and run multiple nodes for high availability and heavy workloads. For production, you should consider a multi-node cluster and follow Temporal’s production best practices (e.g. running each Temporal service separately, managing database schemas manually, etc.) community.temporal.io. There is significant effort required to deploy and manage multiple horizontal Temporal servers, between certificate rotation and configuration management, especially as Temporal often changes configuration requirements. Once you get to the stage where you need the reliability of multiple Temporal servers, my recommendation is that you pay for their managed service.
Our deployment will consist of the following components:
temporalio/auto-setup
image for convenience, which launches all services in one process and auto-initializes the database schema
docs.temporal.io.temporalio/ui
image) to provide a web interface for viewing workflows, tasks, and Temporal namespacesAll these services can run on a single machine via Docker Compose (again, I do recommend using a managed Postgres instance for stability).
The domain temporal.example.com
will point to this machine (use an A or CNAME record!), and we’ll obtain an SSL certificate for it using Let’s Encrypt.
Before we begin, make sure you have the following on your VPS:
temporal.example.com
), with DNS records pointing to your server’s IP. Don’t forget to reserve a public IP! Some cloud platforms will auto-release and reassign a new IP for you if you reboot, and this will break your DNS!TCP/7233
, TCP/80
, and TCP/443
. Later, you can whitelist your worker nodes for 7233, or just allow all (0.0.0.0
) to access this port.If you’re running an Ubuntu/Debian/Debian Derivative, this shell snippet may help:
apt install docker-compose-v2 nginx python3-certbot-nginx apache2-utils golang
go install github.com/fullstorydev/grpcurl/cmd/grpcurl@latest
curl -L 'https://temporal.download/cli/archive/latest?platform=linux&arch=amd64' > x.tar.gz
mv temporal /usr/bin
rm LICENSE x.tar.gz
Now, with that out of the way, let’s get started!
First, create a directory for your Temporal deployment and navigate into it.
Then create a file named docker-compose.yml
with the following content:
version: "3.5"
services:
temporal:
container_name: temporal
image: temporalio/auto-setup:${TEMPORAL_VERSION:-1.24.2.1} # You can substitute for the latest Temporal version
env_file: .env
restart: always
networks:
- temporal-network
ports:
- 7233:7233 # Expose gRPC frontend port
volumes:
- ./certs:/etc/temporal/certs
- ./base.yaml:/etc/temporal/config/base.yaml
temporal-ui:
container_name: temporal-ui
image: temporalio/ui:${TEMPORAL_VERSION:-1.24.2.1} # Use the latest Temporal Web UI version
env_file: .env.ui
restart: always
depends_on:
- temporal
networks:
- temporal-network
ports:
- 8080:8080 # Expose Web UI port
volumes:
- ./ui.yaml:/etc/temporal/config/development.yaml
- ./certs:/etc/temporal/certs
networks:
temporal-network:
driver: bridge
name: temporal-network
Next, let’s set up Postgres.
There are many ways to configure your Postgres instance, and plenty of ways to restrict permissions. Here are the minimum requirements to make it work:
temporal
temporal_visibility
tuser
)psql
on your personal/work machine, you’ll need to connect to the Postgres instance as an admin and grant full access to the user on these two databases:Note: Don’t forget to whitelist the IP address of your temporal server to access the database! Similarly, you may need to whitelist whichever machine you’re using to connect to the database with
psql
.
GRANT CONNECT ON DATABASE temporal TO tuser;
GRANT USAGE ON SCHEMA public TO tuser;
GRANT CREATE ON SCHEMA public TO tuser;
ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON TABLES TO tuser;
Do the same for temporal_visibility
:
GRANT CONNECT ON DATABASE temporal_visibility TO tuser;
GRANT USAGE ON SCHEMA public TO tuser;
GRANT CREATE ON SCHEMA public TO tuser;
ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON TABLES TO tuser;
Now, let’s create some TLS certificates.
Make a directory called certs, and cd
into it.
Inside this directory, we’ll generate a RootCA file:
openssl genrsa -out rootCA.key 2048
openssl req -x509 -new -nodes -key rootCA.key -sha256 -days 3650 -out rootCA.pem
You can adjust the expiration time or keysize as you wish.
Create a client and server key:
openssl genrsa -out client.key 2048
openssl genrsa -out server.key 2048
Now, using our RootCA, we’ll mint some certificates.
Let’s start by creating some Certificicate Signing Request (.csr
) files:
openssl req -new -key server.key -out server.csr -config server.cnf
You’ll be prompted to answer some questions about your organization.
temporal.example.com
<YOUR RESERVED IP HERE>
And similarly, for the client:
openssl req -new -key client.key -out client.csr -config client.cnf
Here, the values don’t really matter. Configure as you see fit.
Now that we have our .csr
s, let’s generate the certificates:
openssl x509 -req -in client.csr -CA rootCA.pem -CAkey rootCA.key -CAcreateserial -out client.crt -days 3650 -sha256 -extfile client.cnf -extensions v3_req
openssl x509 -req -in server.csr -CA rootCA.pem -CAkey rootCA.key -CAcreateserial -out server.crt -days 3650 -sha256 -extfile server.cnf -extensions v3_req
Now, you should have a directory that looks like this:
temporal@temporal-server:~$ tree
.
├── certs
│ ├── client.cnf
│ ├── client.crt
│ ├── client.csr
│ ├── client.key
│ ├── rootCA.key
│ ├── rootCA.pem
│ ├── rootCA.srl
│ ├── server.cnf
│ ├── server.crt
│ ├── server.csr
│ └── server.key
└── docker-compose.yml
Next, we need to create our .env
and .env.ui
files.
Create a .env
file next to your docker-compose.yml
, replacing the <SUBSTITUTIONS>
:
POSTGRES_USER=<YOUR USER HERE>
POSTGRES_PWD=<PASSWORD HERE>
DB=postgres12_pgx
DBNAME=temporal
VISIBILITY_DBNAME=temporal_visibility
DB_PORT=<YOUR DB PORT>
POSTGRES_TLS_ENABLED=true
POSTGRES_SEEDS=<YOU DB HOSTNAME/IP HERE>
POSTGRES_TLS_DISABLE_HOST_VERIFICATION=true
SQL_TLS=true
SQL_TLS_DISABLE_HOST_VERIFICATION=true
SQL_TLS_ENABLED=true
SQL_HOST_VERIFICATION=false
SKIP_DB_CREATE=true
TEMPORAL_AUTH_ENABLED=true
TEMPORAL_ADDRESS=temporal:7233
TEMPORAL_TLS_REQUIRE_CLIENT_AUTH=true
TEMPORAL_TLS_CERTS_DIR=/etc/temporal/certs
SKIP_DEFAULT_NAMESPACE_CREATION=false
DEFAULT_NAMESPACE=default
TEMPORAL_CLI_TLS=true
TEMPORAL_CLI_TLS_CERT=/etc/temporal/certs/client.crt
TEMPORAL_CLI_TLS_KEY=/etc/temporal/certs/client.key
TEMPORAL_CLI_TLS_CA_CERT=/etc/temporal/certs/rootCA.pem
TEMPORAL_ADDRESS=temporal:7233
Then, create a .env.ui
file to configure the webui:
TEMPORAL_TLS_CA_DATA=<REPLACE WITH BASE64 CERT rootCA.pem>
TEMPORAL_TLS_CERT_DATA=<REPLACE WITH BASE64 CERT client.crt>
TEMPORAL_TLS_KEY_DATA=<REPLACE WITH BASE64 CERT client.key>
You can create these base64-encoded values as follows:
echo TEMPORAL_TLS_CA_DATA=$(cat certs/rootCA.pem | base64 -w0) > .env.ui
echo TEMPORAL_TLS_CERT_DATA=$(cat certs/client.crt | base64 -w0) >> .env.ui
echo TEMPORAL_TLS_KEY_DATA=$(cat certs/client.key | base64 -w0) >> .env.ui
Next to our docker-compose.yml
, you need to create two configuration files, one for temporal-server
and one for the webui
.
Create base.yml
:
global:
tls:
internode:
server:
certFile: /etc/temporal/certs/server.crt
keyFile: /etc/temporal/certs/server.key
requireClientAuth: true
clientCaFiles:
- /etc/temporal/certs/rootCA.pem
client:
serverName: temporal
rootCaFiles:
- /etc/temporal/certs/rootCA.pem
frontend:
server:
certFile: /etc/temporal/certs/server.crt
keyFile: /etc/temporal/certs/server.key
requireClientAuth: true
clientCaFiles:
- /etc/temporal/certs/rootCA.pem
rootCaFiles:
- /etc/temporal/certs/rootCA.pem
client:
serverName: temporal
rootCaFiles:
- /etc/temporal/certs/rootCA.pem
services:
history:
matching:
worker:
log:
level: debug
And ui.yaml
:
tls:
caFile: /etc/temporal/certs/rootCA.pem
certFile: /etc/temporal/certs/client.crt
keyFile: /etc/temporal/certs/client.key
enableHostVerification: false
serverName: temporal
Next, run docker compose up
and see your services come to life!
You should see a whole mess of tables populate in your Postgres instance, as the auto-setup image does its thing.
Once the logs start to slow down and your server starts into a retry loop, go ahead and kill the process with Ctrl+c
.
Wait for the containers to spin down, and we have one change to make to our docker-compose.yml
: swap out the word autosetup
for server
to change our specified container image.
autosetup
is meant to help you easily provision database tables and schema, but
isn’t suitable for production deployments, so once our tables are set up, we switch it to the production-ready temporalio/server
image.
We’re so close to complete! We only have one step left: setting up our nginx
reverse-proxy with password auth.
Here’s a config you can drop in on top of /etc/nginx/nginx.conf
(optionally: instead, add a sites
file and link it to sites-enabled
):
user www-data;
worker_processes auto;
pid /run/nginx.pid;
error_log /var/log/nginx/error.log;
include /etc/nginx/modules-enabled/*.conf;
events {
worker_connections 768;
}
http {
sendfile on;
tcp_nopush on;
types_hash_max_size 2048;
include /etc/nginx/mime.types;
default_type application/octet-stream;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3; # Dropping SSLv3, ref: POODLE
ssl_prefer_server_ciphers on;
access_log /var/log/nginx/access.log;
gzip on;
upstream temporal {
server 127.0.0.1:8080;
}
server {
auth_basic "Temporal Admins Only!";
auth_basic_user_file /etc/nginx/.htpasswd;
server_name temporal.example.com;
location / {
proxy_pass http://temporal;
proxy_http_version 1.1;
# Ensuring it can use websockets
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto http;
proxy_redirect http:// $scheme://;
proxy_set_header Host $http_host;
}
}
include /etc/nginx/conf.d/*.conf;
include /etc/nginx/sites-enabled/*;
}
Add the following to your /etc/hosts
:
cat << EOF >> /etc/hosts
127.0.0.1 temporal
EOF
Generate a .htpasswd
file to control user authentication (you will be prompted to enter a password):
htpasswd -c /etc/nginx/.htpasswd <YOUR USERNAME>
Enable and reload nginx
:
systemctl enable --now nginx && systemctl reload nginx
Setup Certbot (this is interactive and may prompt you for your email):
certbot -d temporal.example.com
Note: don’t forget to add
certbot renew
to a cron job! LetsEncrypt certs typically expire every 3 months, so check for renewals every 1-2 months.
Moment of truth: open your browser and navigate to https://temporal.example.com
.
You should be greeted with a Basic Auth prompt.
Enter your username and password you generated earlier.
Then, you should see Temporal’s Web UI homepage.
If you see something similar to the screenshot above, congratulations! 🎉 You now have a single-node Temporal server running with TLS. The connection is secure (check that browser lock icon), and you can start building or testing workflows. The UI will show zero workflows initially (since none have been run); it’s expected.
If you see errors regarding “no default namespace”, you can manually create one using the following command:
temporal operator namespace create --namespace default --tls-ca-path certs/rootCA.pem --tls-cert-path certs/client.crt --tls-key-path certs/client.key
To further verify that gRPC traffic is working through TLS, you can use grpcurl
to check the health probe:
~/go/bin/grpcurl --cacert certs/rootCA.pem --cert certs/client.crt --key certs/client.key -d '{"service": "temporal.api.workflowservice.v1.WorkflowService"}' temporal:7233 grpc.health.v1.Health/Check
If you see the following, temporal-server
is properly configured:
{
"status": "SERVING"
}
To verify your task queues are set up, you can create a HELLO WORLD
workflow (note the nested quotes, which are necessary):
temporal workflow start --address temporal:7233 --type HelloWorldWorkflow --task-queue=HELLO_WORLD_TASK_QUEUE --tls-ca-path certs/rootCA.pem --tls-cert-path certs/client.crt --tls-key-path certs/client.key --input '"Hello, World!"'
If you’ve created too many test workflows, you can Terminate them from within the Web UI or run the following to delete all of them:
for x in $(temporal workflow list --address temporal:7233 --tls-ca-path certs/rootCA.pem --tls-cert-path certs/client.crt --tls-key-path certs/client.key | awk '{ print $2 }'); do
temporal workflow delete --address temporal:7233 --tls-ca-path certs/rootCA.pem --tls-cert-path certs/client.crt --tls-key-path certs/client.key -w $x
done
Because we have self-signed certificates, we need some way to get the certificates into our client.
Here’s how I do it in go (it will be similar in other languages):
package tclient
import (
"context"
"crypto/tls"
"crypto/x509"
"encoding/base64"
"fmt"
"math/rand"
"os"
"sync"
"github.com/taigrr/temporal-worker/vars"
"go.temporal.io/sdk/client"
)
var (
tClient client.Client
clientTex sync.Mutex
)
func New() (client.Client, string, error) {
clientTex.Lock()
defer clientTex.Unlock()
if tClient != nil {
return tClient, "", nil
}
// base64-encoded TLS certs created the same way we did for .env.ui
clientCertData := os.Getenv(vars.TemporalTLSCertData)
clientCertKeyData := os.Getenv(vars.TemporalTLSKeyData)
serverRootCAData := os.Getenv(vars.TemporalTLSCAData)
// the FQDN:PORT i.e. temporal.example.com:7233
hostPort := os.Getenv(vars.TemporalHostPort)
// default
namespace := os.Getenv(vars.TemporalNamespace)
// Decode the base64 encoded keys and certs
clientCert, err := base64.StdEncoding.DecodeString(clientCertData)
if err != nil {
return nil, "", fmt.Errorf("unable to decode client cert data: %w", err)
}
clientCertKey, err := base64.StdEncoding.DecodeString(clientCertKeyData)
if err != nil {
return nil, "", fmt.Errorf("unable to decode client cert key data: %w", err)
}
serverRootCA, err := base64.StdEncoding.DecodeString(serverRootCAData)
if err != nil {
return nil, "", fmt.Errorf("unable to decode server root CA data: %w", err)
}
cert, err := tls.X509KeyPair(clientCert, clientCertKey)
if err != nil {
return nil, "", fmt.Errorf("unable to load cert and key pair: %w", err)
}
// Create Certpool to add the server root CA to the client pool
certPool := x509.NewCertPool()
certPool.AppendCertsFromPEM(serverRootCA)
// Ensure no two workers have the same ID
randString := fmt.Sprintf("%04x", rand.Intn(0x10000))
// Embed the git commit of the currently running code into the worker ID
commitHash := os.Getenv(vars.WorkerCommit)
if len(commitHash) > 7 {
commitHash = commitHash[:7]
}
workerID := fmt.Sprintf("%s-%s-%s", os.Getenv(vars.WorkerID), commitHash, randString)
// Add the cert to the tls certificates in the ConnectionOptions of the Client
clientOptions := client.Options{
HostPort: hostPort,
Namespace: namespace,
ConnectionOptions: client.ConnectionOptions{
TLS: &tls.Config{
Certificates: []tls.Certificate{cert},
RootCAs: certPool,
},
},
Identity: workerID,
}
temporalClient, err := client.Dial(clientOptions)
if err != nil {
return nil, "", fmt.Errorf("unable to connect to Temporal: %w", err)
}
tClient = temporalClient
return tClient, workerID, nil
}
We’ve successfully deployed Temporal on a single node with Docker Compose and added TLS termination with NGINX.
In summary, we:
This setup provides a starting point for working with Temporal’s powerful workflow capabilities.
You can now write workflows in your language of choice (Go, Java, Python, etc.), point the SDK client to temporal.example.com:7233
, and begin executing workflows on your server–just make sure your self-signed certificates are imported (see my go example above)!
Before we conclude, a few final notes and best practices:
tuser
in Postgres for our temporal
and temporal_visibility
databases. Now that the schema is set up, consider locking that down.Temporal brings reliability superpowers to your applications – even in this modest single-node setup, you can develop and test workflows that will never give up in the face of failures.
Now that you have Temporal running at temporal.example.com
, it’s time to start orchestrating some workflows.
Have fun exploring Temporal, and may your workflows always complete (eventually)! 🚀
What is WireGuard? WireGuard is a VPN, which tunnels traffic over encrypted, secure UDP packet streams. Once a tunnel is established, the endpoints …
Read ArticleBackground In a recent blog post, I discussed a few of the reasons why in 2021, I began working on a new Fleet Configuration Management tool to …
Read Article