Temporal Server: Self-hosting a Production-Ready Instance

post-thumb

Temporal is an open-source workflow orchestration engine designed to manage, execute, and monitor complex workflows and activities. In plain terms, Temporal allows you to write crash-proof code: it ensures that your long-running processes (workflows) continue execution reliably even if your services (or external services!) crash or restart.

Temporal’s walkthroughs and CLI provide extremely easy adoption and low-friction for getting started: the temporal CLI contains an embedded development server (temporal server start-dev), which binds to localhost and allows for easy, unauthenticated workflows. Connecting to Temporal’s managed service is similarly easy, but at writing, has a minimum cost of $100/month, which is not always affordable for smaller orgs, especially when running a production-grade self-managed server can cost less than a quarter of that.

In this article, we’ll set up a single-node Temporal server backed by PostgreSQL, and we’ll secure it with TLS using NGINX as a reverse proxy. The end result will be Temporal’s services running in Docker containers, accessible at https://temporal.example.com (our example domain) with encrypted connections.

Note: This setup is suitable for small-scale, development and production deployments. Temporal is designed to scale horizontally and run multiple nodes for high availability and heavy workloads. For production, you should consider a multi-node cluster and follow Temporal’s production best practices (e.g. running each Temporal service separately, managing database schemas manually, etc.) ​ community.temporal.io. There is significant effort required to deploy and manage multiple horizontal Temporal servers, between certificate rotation and configuration management, especially as Temporal often changes configuration requirements. Once you get to the stage where you need the reliability of multiple Temporal servers, my recommendation is that you pay for their managed service.

Overview of the Deployment

Our deployment will consist of the following components:

  • Temporal Server – runs Temporal’s core services (History, Matching) in one container. We will use the official temporalio/auto-setup image for convenience, which launches all services in one process and auto-initializes the database schema docs.temporal.io.
  • Temporal Web UI – runs in a separate container (temporalio/ui image) to provide a web interface for viewing workflows, tasks, and Temporal namespaces
  • PostgreSQL – a single Postgres instance as Temporal’s persistence store (keeping workflow state, history, etc.). I recommend using a managed database for this, like DigitalOcean (Disclaimer: this is my personal RefLink to DigitalOcean) to ensure uptime and reliability.
  • NGINX – runs on the host as a reverse proxy. It will terminate TLS (HTTPS), forwarding requests to the Temporal WebUI container, and provide Basic Auth support.

All these services can run on a single machine via Docker Compose (again, I do recommend using a managed Postgres instance for stability). The domain temporal.example.com will point to this machine (use an A or CNAME record!), and we’ll obtain an SSL certificate for it using Let’s Encrypt.

Prerequisites

Before we begin, make sure you have the following on your VPS:

  • Docker and Docker Compose: This is how we’ll be standing up our temporal services and ensuring they automatically restart
  • A domain name: for accessing Temporal (e.g., temporal.example.com), with DNS records pointing to your server’s IP. Don’t forget to reserve a public IP! Some cloud platforms will auto-release and reassign a new IP for you if you reboot, and this will break your DNS!
  • Firewall Rules: Block all incoming traffic to your server, especially port 8080, and only allow through TCP/7233, TCP/80, and TCP/443. Later, you can whitelist your worker nodes for 7233, or just allow all (0.0.0.0) to access this port.
  • Certbot: Let’s Encrypt’s CertBot utility can be helpful for provisioning and rotating SSL certs in nginx
  • GRPCurl: A grpc CLI for doing quick spot-checks on the installation
  • temporal (cli): A handy CLI utility for interacting with the Temporal server

If you’re running an Ubuntu/Debian/Debian Derivative, this shell snippet may help:

apt install docker-compose-v2 nginx python3-certbot-nginx apache2-utils golang
go install github.com/fullstorydev/grpcurl/cmd/grpcurl@latest
curl -L 'https://temporal.download/cli/archive/latest?platform=linux&arch=amd64' > x.tar.gz
mv temporal /usr/bin
rm LICENSE x.tar.gz

Now, with that out of the way, let’s get started!

Setting Up Our Docker Compose

First, create a directory for your Temporal deployment and navigate into it. Then create a file named docker-compose.yml with the following content:

version: "3.5"
services:
  temporal:
    container_name: temporal
    image: temporalio/auto-setup:${TEMPORAL_VERSION:-1.24.2.1} # You can substitute for the latest Temporal version
    env_file: .env
    restart: always
    networks:
      - temporal-network
    ports:
      - 7233:7233 # Expose gRPC frontend port
    volumes:
      - ./certs:/etc/temporal/certs
      - ./base.yaml:/etc/temporal/config/base.yaml

  temporal-ui:
    container_name: temporal-ui
    image: temporalio/ui:${TEMPORAL_VERSION:-1.24.2.1} # Use the latest Temporal Web UI version
    env_file: .env.ui
    restart: always
    depends_on:
      - temporal
    networks:
      - temporal-network
    ports:
      - 8080:8080 # Expose Web UI port
    volumes:
      - ./ui.yaml:/etc/temporal/config/development.yaml
      - ./certs:/etc/temporal/certs
networks:
  temporal-network:
    driver: bridge
    name: temporal-network

Next, let’s set up Postgres.

Configuring Postgres

There are many ways to configure your Postgres instance, and plenty of ways to restrict permissions. Here are the minimum requirements to make it work:

  1. Create a database named temporal
  2. Create a database named temporal_visibility
  3. Create a user for the Temporal server to use (here we’ll assume tuser)
  4. Using psql on your personal/work machine, you’ll need to connect to the Postgres instance as an admin and grant full access to the user on these two databases:

Note: Don’t forget to whitelist the IP address of your temporal server to access the database! Similarly, you may need to whitelist whichever machine you’re using to connect to the database with psql.

GRANT CONNECT ON DATABASE temporal TO tuser;
GRANT USAGE ON SCHEMA public TO tuser;
GRANT CREATE ON SCHEMA public TO tuser;
ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON TABLES TO tuser;

Do the same for temporal_visibility:

GRANT CONNECT ON DATABASE temporal_visibility TO tuser;
GRANT USAGE ON SCHEMA public TO tuser;
GRANT CREATE ON SCHEMA public TO tuser;
ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON TABLES TO tuser;

Now, let’s create some TLS certificates.

TLS certificates

Make a directory called certs, and cd into it. Inside this directory, we’ll generate a RootCA file:

openssl genrsa -out rootCA.key 2048
openssl req -x509 -new -nodes -key rootCA.key -sha256 -days 3650 -out rootCA.pem

You can adjust the expiration time or keysize as you wish.

Create a client and server key:

openssl genrsa -out client.key 2048
openssl genrsa -out server.key 2048

Now, using our RootCA, we’ll mint some certificates. Let’s start by creating some Certificicate Signing Request (.csr) files:

openssl req -new -key server.key -out server.csr -config server.cnf

You’ll be prompted to answer some questions about your organization.

  • Be sure the commonName and field matches your DNS record, i.e. temporal.example.com
  • For DNS Alt-names, provide the following:
    • DNS.1: temporal
    • DNS.2: temporal.example.com
    • IP.1: 127.0.0.1
    • IP.2: <YOUR RESERVED IP HERE>

And similarly, for the client:

openssl req -new -key client.key -out client.csr -config client.cnf

Here, the values don’t really matter. Configure as you see fit.

Now that we have our .csrs, let’s generate the certificates:

openssl x509 -req -in client.csr -CA rootCA.pem -CAkey rootCA.key -CAcreateserial -out client.crt -days 3650 -sha256 -extfile client.cnf -extensions v3_req
openssl x509 -req -in server.csr -CA rootCA.pem -CAkey rootCA.key -CAcreateserial -out server.crt -days 3650 -sha256 -extfile server.cnf -extensions v3_req

Now, you should have a directory that looks like this:

temporal@temporal-server:~$ tree
.
├── certs
│   ├── client.cnf
│   ├── client.crt
│   ├── client.csr
│   ├── client.key
│   ├── rootCA.key
│   ├── rootCA.pem
│   ├── rootCA.srl
│   ├── server.cnf
│   ├── server.crt
│   ├── server.csr
│   └── server.key
└── docker-compose.yml

Next, we need to create our .env and .env.ui files.

Configuring Env Vars

Create a .env file next to your docker-compose.yml, replacing the <SUBSTITUTIONS>:

POSTGRES_USER=<YOUR USER HERE>
POSTGRES_PWD=<PASSWORD HERE>
DB=postgres12_pgx
DBNAME=temporal
VISIBILITY_DBNAME=temporal_visibility
DB_PORT=<YOUR DB PORT>
POSTGRES_TLS_ENABLED=true
POSTGRES_SEEDS=<YOU DB HOSTNAME/IP HERE>
POSTGRES_TLS_DISABLE_HOST_VERIFICATION=true
SQL_TLS=true
SQL_TLS_DISABLE_HOST_VERIFICATION=true
SQL_TLS_ENABLED=true
SQL_HOST_VERIFICATION=false
SKIP_DB_CREATE=true
TEMPORAL_AUTH_ENABLED=true
TEMPORAL_ADDRESS=temporal:7233
TEMPORAL_TLS_REQUIRE_CLIENT_AUTH=true
TEMPORAL_TLS_CERTS_DIR=/etc/temporal/certs
SKIP_DEFAULT_NAMESPACE_CREATION=false
DEFAULT_NAMESPACE=default
TEMPORAL_CLI_TLS=true
TEMPORAL_CLI_TLS_CERT=/etc/temporal/certs/client.crt
TEMPORAL_CLI_TLS_KEY=/etc/temporal/certs/client.key
TEMPORAL_CLI_TLS_CA_CERT=/etc/temporal/certs/rootCA.pem
TEMPORAL_ADDRESS=temporal:7233

Then, create a .env.ui file to configure the webui:

TEMPORAL_TLS_CA_DATA=<REPLACE WITH BASE64 CERT rootCA.pem>
TEMPORAL_TLS_CERT_DATA=<REPLACE WITH BASE64 CERT client.crt>
TEMPORAL_TLS_KEY_DATA=<REPLACE WITH BASE64 CERT client.key>

You can create these base64-encoded values as follows:

echo TEMPORAL_TLS_CA_DATA=$(cat certs/rootCA.pem | base64 -w0) > .env.ui
echo TEMPORAL_TLS_CERT_DATA=$(cat certs/client.crt | base64 -w0) >> .env.ui
echo TEMPORAL_TLS_KEY_DATA=$(cat certs/client.key | base64 -w0) >> .env.ui

Temporal Configuration

Next to our docker-compose.yml, you need to create two configuration files, one for temporal-server and one for the webui.

Create base.yml:

global:
  tls:
    internode:
      server:
        certFile: /etc/temporal/certs/server.crt
        keyFile: /etc/temporal/certs/server.key
        requireClientAuth: true
        clientCaFiles:
          - /etc/temporal/certs/rootCA.pem
      client:
        serverName: temporal
        rootCaFiles:
          - /etc/temporal/certs/rootCA.pem
    frontend:
      server:
        certFile: /etc/temporal/certs/server.crt
        keyFile: /etc/temporal/certs/server.key
        requireClientAuth: true
        clientCaFiles:
          - /etc/temporal/certs/rootCA.pem
        rootCaFiles:
          - /etc/temporal/certs/rootCA.pem
      client:
        serverName: temporal
        rootCaFiles:
          - /etc/temporal/certs/rootCA.pem

services:
  history:
  matching:
  worker:

log:
  level: debug

And ui.yaml:

tls:
  caFile: /etc/temporal/certs/rootCA.pem
  certFile: /etc/temporal/certs/client.crt
  keyFile: /etc/temporal/certs/client.key
  enableHostVerification: false
  serverName: temporal

Spinning Up

Next, run docker compose up and see your services come to life!

You should see a whole mess of tables populate in your Postgres instance, as the auto-setup image does its thing.

Once the logs start to slow down and your server starts into a retry loop, go ahead and kill the process with Ctrl+c. Wait for the containers to spin down, and we have one change to make to our docker-compose.yml: swap out the word autosetup for server to change our specified container image. autosetup is meant to help you easily provision database tables and schema, but isn’t suitable for production deployments, so once our tables are set up, we switch it to the production-ready temporalio/server image.

Configuring NGINX as a TLS Reverse Proxy

We’re so close to complete! We only have one step left: setting up our nginx reverse-proxy with password auth.

Here’s a config you can drop in on top of /etc/nginx/nginx.conf (optionally: instead, add a sites file and link it to sites-enabled):

user www-data;
worker_processes auto;
pid /run/nginx.pid;
error_log /var/log/nginx/error.log;
include /etc/nginx/modules-enabled/*.conf;

events {
        worker_connections 768;
}
http {
        sendfile on;
        tcp_nopush on;
        types_hash_max_size 2048;
        include /etc/nginx/mime.types;
        default_type application/octet-stream;
        ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3; # Dropping SSLv3, ref: POODLE
        ssl_prefer_server_ciphers on;
        access_log /var/log/nginx/access.log;
        gzip on;
        upstream temporal {
                server 127.0.0.1:8080;
        }

        server {
                auth_basic           "Temporal Admins Only!";
                auth_basic_user_file /etc/nginx/.htpasswd;
                server_name temporal.example.com;
                location / {
                        proxy_pass         http://temporal;
                        proxy_http_version 1.1;
                        # Ensuring it can use websockets
                        proxy_set_header   Upgrade $http_upgrade;
                        proxy_set_header   Connection "upgrade";
                        proxy_set_header   X-Real-IP $remote_addr;
                        proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;
                        proxy_set_header   X-Forwarded-Proto http;
                        proxy_redirect     http:// $scheme://;
                        proxy_set_header   Host $http_host;
                }
        }
        include /etc/nginx/conf.d/*.conf;
        include /etc/nginx/sites-enabled/*;
}

Add the following to your /etc/hosts:

cat << EOF >> /etc/hosts
127.0.0.1 temporal
EOF

Generate a .htpasswd file to control user authentication (you will be prompted to enter a password):

htpasswd -c /etc/nginx/.htpasswd <YOUR USERNAME>

Enable and reload nginx:

systemctl enable --now nginx && systemctl reload nginx

Setup Certbot (this is interactive and may prompt you for your email):

certbot -d temporal.example.com

Note: don’t forget to add certbot renew to a cron job! LetsEncrypt certs typically expire every 3 months, so check for renewals every 1-2 months.

Testing the Setup

Moment of truth: open your browser and navigate to https://temporal.example.com. You should be greeted with a Basic Auth prompt. Enter your username and password you generated earlier. Then, you should see Temporal’s Web UI homepage.

Default temporal web ui view without any workflows

If you see something similar to the screenshot above, congratulations! 🎉 You now have a single-node Temporal server running with TLS. The connection is secure (check that browser lock icon), and you can start building or testing workflows. The UI will show zero workflows initially (since none have been run); it’s expected.

If you see errors regarding “no default namespace”, you can manually create one using the following command:

temporal operator namespace create --namespace default --tls-ca-path certs/rootCA.pem --tls-cert-path certs/client.crt --tls-key-path certs/client.key

Troubleshooting

Health Check

To further verify that gRPC traffic is working through TLS, you can use grpcurl to check the health probe:

~/go/bin/grpcurl  --cacert certs/rootCA.pem --cert certs/client.crt --key certs/client.key  -d '{"service": "temporal.api.workflowservice.v1.WorkflowService"}'  temporal:7233 grpc.health.v1.Health/Check

If you see the following, temporal-server is properly configured:

{
  "status": "SERVING"
}

Creating a Test Workflow

To verify your task queues are set up, you can create a HELLO WORLD workflow (note the nested quotes, which are necessary):

temporal workflow start --address temporal:7233 --type HelloWorldWorkflow --task-queue=HELLO_WORLD_TASK_QUEUE --tls-ca-path certs/rootCA.pem --tls-cert-path certs/client.crt --tls-key-path certs/client.key --input '"Hello, World!"'

Cleaning up Tests

If you’ve created too many test workflows, you can Terminate them from within the Web UI or run the following to delete all of them:

for x in $(temporal  workflow list   --address temporal:7233  --tls-ca-path certs/rootCA.pem --tls-cert-path certs/client.crt --tls-key-path certs/client.key | awk '{ print $2 }'); do
    temporal workflow delete   --address temporal:7233  --tls-ca-path certs/rootCA.pem --tls-cert-path certs/client.crt --tls-key-path certs/client.key -w $x
done

Connecting a Client

Because we have self-signed certificates, we need some way to get the certificates into our client.

Here’s how I do it in go (it will be similar in other languages):

package tclient

import (
        "context"
        "crypto/tls"
        "crypto/x509"
        "encoding/base64"
        "fmt"
        "math/rand"
        "os"
        "sync"

        "github.com/taigrr/temporal-worker/vars"
        "go.temporal.io/sdk/client"
)

var (
        tClient   client.Client
        clientTex sync.Mutex
)

func New() (client.Client, string, error) {
        clientTex.Lock()
        defer clientTex.Unlock()
        if tClient != nil {
                return tClient, "", nil
        }
        // base64-encoded TLS certs created the same way we did for .env.ui
        clientCertData := os.Getenv(vars.TemporalTLSCertData)
        clientCertKeyData := os.Getenv(vars.TemporalTLSKeyData)
        serverRootCAData := os.Getenv(vars.TemporalTLSCAData)

        // the FQDN:PORT i.e. temporal.example.com:7233
        hostPort := os.Getenv(vars.TemporalHostPort)

        // default
        namespace := os.Getenv(vars.TemporalNamespace)

        // Decode the base64 encoded keys and certs
        clientCert, err := base64.StdEncoding.DecodeString(clientCertData)
        if err != nil {
                return nil, "", fmt.Errorf("unable to decode client cert data: %w", err)
        }
        clientCertKey, err := base64.StdEncoding.DecodeString(clientCertKeyData)
        if err != nil {
                return nil, "", fmt.Errorf("unable to decode client cert key data: %w", err)
        }
        serverRootCA, err := base64.StdEncoding.DecodeString(serverRootCAData)
        if err != nil {
                return nil, "", fmt.Errorf("unable to decode server root CA data: %w", err)
        }
        cert, err := tls.X509KeyPair(clientCert, clientCertKey)
        if err != nil {
                return nil, "", fmt.Errorf("unable to load cert and key pair: %w", err)
        }

        // Create Certpool to add the server root CA to the client pool
        certPool := x509.NewCertPool()
        certPool.AppendCertsFromPEM(serverRootCA)

        // Ensure no two workers have the same ID
        randString := fmt.Sprintf("%04x", rand.Intn(0x10000))
        // Embed the git commit of the currently running code into the worker ID
        commitHash := os.Getenv(vars.WorkerCommit)
        if len(commitHash) > 7 {
                commitHash = commitHash[:7]
        }
        workerID := fmt.Sprintf("%s-%s-%s", os.Getenv(vars.WorkerID), commitHash, randString)

        // Add the cert to the tls certificates in the ConnectionOptions of the Client
        clientOptions := client.Options{
                HostPort:  hostPort,
                Namespace: namespace,
                ConnectionOptions: client.ConnectionOptions{
                        TLS: &tls.Config{
                                Certificates: []tls.Certificate{cert},
                                RootCAs:      certPool,
                        },
                },
                Identity: workerID,
        }
        temporalClient, err := client.Dial(clientOptions)
        if err != nil {
                return nil, "", fmt.Errorf("unable to connect to Temporal: %w", err)
        }
        tClient = temporalClient
        return tClient, workerID, nil
}

Wrap Up

We’ve successfully deployed Temporal on a single node with Docker Compose and added TLS termination with NGINX.

In summary, we:

  • Introduced Temporal and set up a PostgreSQL-backed Temporal server using Docker Compose
  • Used Temporal’s auto-setup image to provision our databases
  • Switched to the Temporal production server
  • Deployed the Temporal Web UI and passed it through NGINX via a friendly HTTPS URL
  • Configured NGINX to handle TLS, forwarding web requests to the UI, thereby securing the traffic

This setup provides a starting point for working with Temporal’s powerful workflow capabilities. You can now write workflows in your language of choice (Go, Java, Python, etc.), point the SDK client to temporal.example.com:7233, and begin executing workflows on your server–just make sure your self-signed certificates are imported (see my go example above)!

Before we conclude, a few final notes and best practices:

  • Security: Your NGINX config can also restrict access by IP. If you intend to expose the Temporal UI or gRPC port more broadly, consider additional security layers, i.e. cloud infra-level firewalls
  • Persistence: Your temporal workflows are only as stable as your database. Again, I highly recommend using a stable, managed solution
  • Certificates: A time will come when your certificates expire. Be sure to have a rotation plan in place for scheduled maintenance!
  • Postgres: We initially granted sweeping access to the tuser in Postgres for our temporal and temporal_visibility databases. Now that the schema is set up, consider locking that down.

Temporal brings reliability superpowers to your applications – even in this modest single-node setup, you can develop and test workflows that will never give up in the face of failures. Now that you have Temporal running at temporal.example.com, it’s time to start orchestrating some workflows. Have fun exploring Temporal, and may your workflows always complete (eventually)! 🚀

Default temporal web ui view without any workflows

You May Also Like