Restate Cluster Playground
A three-node Restate cluster for inspecting distributed Restate server behavior.
📖 Overview
This playground runs a three-node Restate cluster and a separate dev-machine for cluster inspection. Use dev-machine as your workstation, then inspect the Restate servers over the private local network.
This is a learning cluster, not a production high-availability recipe. Production Restate clusters should configure durable volumes, snapshots, and object storage according to the Restate cluster docs.
🔧 System Components
| Machine | Address | Purpose |
|---|---|---|
dev-machine | 172.16.0.2 | CLI workstation with restate and restatectl |
restate-1 | 172.16.0.10 | Restate node that provisions the cluster |
restate-2 | 172.16.0.11 | Restate node |
restate-3 | 172.16.0.12 | Restate node |
| Component | Port | Purpose |
|---|---|---|
| Restate ingress | 8080 | Invoke registered services |
| Restate Admin API and UI | 9070 | Manage deployments, inspect services, query state |
| Restate fabric | 5122 | Node-to-node communication and metrics |
🛠️ Tools
restate: Restate CLI for service and deployment management.restatectl: Restate control CLI for node and cluster inspection.
🎯 Getting Started
Run these commands from dev-machine.
Check the cluster status:
restatectl --addresses http://restate-1:5122 status
Check the discovered nodes:
restatectl --addresses http://restate-1:5122 nodes list
This proves the three Restate servers discovered each other and joined the same cluster.
Open the UI
Open the ui tab. It points at the Restate Admin UI on restate-1:9070.
No need to remember the port number while you are in the playground, but the direct endpoint is useful when debugging:
curl -sf http://restate-1:9070/health && echo OK
🔎 Inspect and Customize
Restate is installed as a system service on each node. The most useful paths and commands are:
| Item | Where to look |
|---|---|
| Restate config | /etc/restate/restate.toml |
| Restate environment overrides | /etc/default/restate |
| Restate data directory | /var/lib/restate |
| systemd unit | /etc/systemd/system/restate.service |
| Restate logs | sudo journalctl -u restate -f |
Inspect restate-1 from dev-machine:
ssh restate-1 "systemctl status restate --no-pager"
ssh restate-1 "journalctl -u restate -n 100 --no-pager"
ssh restate-1 "cat /etc/restate/restate.toml"
Swap restate-1 for restate-2 or restate-3 when inspecting the other nodes.
After changing Restate config on a node, restart the service:
ssh restate-1 "sudo systemctl restart restate"
🧯 Troubleshooting
| Symptom | What to check | Fix |
|---|---|---|
A node is missing from nodes list | Run ssh restate-2 "systemctl status restate --no-pager" and check journalctl -u restate on the missing node. | Restart the failed node service after fixing /etc/restate/restate.toml or /etc/default/restate. |
| Admin health is failing | Check curl -sf http://restate-1:9070/health and confirm /etc/restate/restate.toml contains the expected metadata client addresses. | Restart restate on the affected node and inspect logs. |
| No services are registered | That is expected in a fresh cluster. | Deploy or register a Restate service, then refresh the UI. |
📚 Learn More
🧩 Related Content
🧪 Playgrounds
- Restate Playground - a single-node Restate server for local experiments
Happy learning! 🚀
Actions
Restate Cluster Playground
A three-node Restate cluster for inspecting distributed Restate server behavior.
📖 Overview
This playground runs a three-node Restate cluster and a separate dev-machine for cluster inspection. Use dev-machine as your workstation, then inspect the Restate servers over the private local network.
This is a learning cluster, not a production high-availability recipe. Production Restate clusters should configure durable volumes, snapshots, and object storage according to the Restate cluster docs.
🔧 System Components
| Machine | Address | Purpose |
|---|---|---|
dev-machine | 172.16.0.2 | CLI workstation with restate and restatectl |
restate-1 | 172.16.0.10 | Restate node that provisions the cluster |
restate-2 | 172.16.0.11 | Restate node |
restate-3 | 172.16.0.12 | Restate node |
| Component | Port | Purpose |
|---|---|---|
| Restate ingress | 8080 | Invoke registered services |
| Restate Admin API and UI | 9070 | Manage deployments, inspect services, query state |
| Restate fabric | 5122 | Node-to-node communication and metrics |
🛠️ Tools
restate: Restate CLI for service and deployment management.restatectl: Restate control CLI for node and cluster inspection.
🎯 Getting Started
Run these commands from dev-machine.
Check the cluster status:
restatectl --addresses http://restate-1:5122 status
Check the discovered nodes:
restatectl --addresses http://restate-1:5122 nodes list
This proves the three Restate servers discovered each other and joined the same cluster.
Open the UI
Open the ui tab. It points at the Restate Admin UI on restate-1:9070.
No need to remember the port number while you are in the playground, but the direct endpoint is useful when debugging:
curl -sf http://restate-1:9070/health && echo OK
🔎 Inspect and Customize
Restate is installed as a system service on each node. The most useful paths and commands are:
| Item | Where to look |
|---|---|
| Restate config | /etc/restate/restate.toml |
| Restate environment overrides | /etc/default/restate |
| Restate data directory | /var/lib/restate |
| systemd unit | /etc/systemd/system/restate.service |
| Restate logs | sudo journalctl -u restate -f |
Inspect restate-1 from dev-machine:
ssh restate-1 "systemctl status restate --no-pager"
ssh restate-1 "journalctl -u restate -n 100 --no-pager"
ssh restate-1 "cat /etc/restate/restate.toml"
Swap restate-1 for restate-2 or restate-3 when inspecting the other nodes.
After changing Restate config on a node, restart the service:
ssh restate-1 "sudo systemctl restart restate"
🧯 Troubleshooting
| Symptom | What to check | Fix |
|---|---|---|
A node is missing from nodes list | Run ssh restate-2 "systemctl status restate --no-pager" and check journalctl -u restate on the missing node. | Restart the failed node service after fixing /etc/restate/restate.toml or /etc/default/restate. |
| Admin health is failing | Check curl -sf http://restate-1:9070/health and confirm /etc/restate/restate.toml contains the expected metadata client addresses. | Restart restate on the affected node and inspect logs. |
| No services are registered | That is expected in a fresh cluster. | Deploy or register a Restate service, then refresh the UI. |
📚 Learn More
🧩 Related Content
🧪 Playgrounds
- Restate Playground - a single-node Restate server for local experiments
Happy learning! 🚀