Use a Kubernetes Job to Test NATS Pub/Sub
Scenario
NATS is a messaging system. Applications use it to send messages to each other without connecting to each other directly. It works with publish/subscribe:
- A publisher sends a message to a subject, a name such as
payments.requests. - A subscriber listens on a subject and receives every message sent to it.
- The NATS server sits between them. Clients connect to it on port
4222, and it delivers each message to every subscriber of the subject.
A NATS server already runs in this cluster.
More about NATS:
- What is NATS (NATS)
- NATS CLI (GitHub)
NATS is not NAT. NAT (Network Address Translation) is a networking feature that rewrites IP addresses. NATS is a message broker, in the same family as Apache Kafka or RabbitMQ.
In the payment-service namespace, the payment-processor subscribes to payments.requests and prints every message it receives in its logs. The checkout namespace is empty. This is where you create the test Job.
The checkout team wants to prove that a Job in checkout can publish a payment request that the payment service receives. The Job is the publisher, and the payment-processor is the subscriber. The team wrote a test script for the Job.
Core NATS does not store messages. Only subscribers that are connected when a message is published receive it, so a successful publish alone does not prove delivery. The payment service logs do.
Task
Create a Job in checkout that publishes a test payment request, so you can check that it reaches the payment service. The Job must have these settings:
| Setting | Value |
|---|---|
| Name | payment-pubsub-test |
| Namespace | checkout |
| Image | natsio/nats-box:0.20.1 |
| Container command and args | The script below |
| Successful Pods needed | 1, one Pod at a time |
| Retries of a failed Pod | At most 1 |
| Time limit for the whole Job | 15 seconds |
| A failed container | Is not restarted inside the same Pod |
| Clean up | The Job is deleted automatically 1 hour after it finishes |
A member of the checkout team prepared this command and args for the Job container.
Use the supplied command and args unchanged for the first run:
command: ["/bin/sh", "-c"]
args:
- |
echo "sending payment request ORD-1001"
nats pub -s nats://nats:4222 payments.requests '{"order_id":"ORD-1001","amount":49.99}'
Check the Job's status, events, and Pod logs to find out whether the payment request was published. If the Job fails, find the cause, delete the Job, and create it again with the necessary correction.
Keep the subject payments.requests and the message unchanged.
The task is complete when the Job succeeds and the payment-processor logs show that the ORD-1001 payment request was received.
Do not change anything outside the checkout namespace. If you need to change the Job, delete it and create it again.
Hint 1: How the script talks to NATS
The script is a NATS client. nats pub connects to the NATS server, publishes one message to a subject, and exits:
nats pub -s <server-url> <subject> <message>
-sis the address of the NATS server, in the formnats://<host>:<port>.<subject>is the name that subscribers listen on.<message>is the body of the message.
If nats pub cannot reach the server, it prints an error and exits with a non-zero code, so the Pod fails.
Docs:
- NATS CLI (GitHub)
- Publish-Subscribe (NATS)
Hint 2: Building the Job
The script has two lines, so a shell must run it. A container can override the start command of its image. Start from a generated Job skeleton:
kubectl create job <name> -n <namespace> --image=<image> --dry-run=client -o yaml
kubectl explain job.spec.template.spec.containers.command
kubectl explain job.spec.template.spec.containers.args
Every row of the settings table is one field. Most of them are in the Job spec. Whether a failed container restarts is set in the Pod template:
kubectl explain job.spec
kubectl explain job.spec.template.spec.restartPolicy
Docs:
- Define a Command and Arguments for a Container (Kubernetes)
- Jobs (Kubernetes)
- Automatic Cleanup for Finished Jobs (Kubernetes)
Hint 3: Checking the Job
Check whether the Job created its Pods, and whether it succeeded or failed:
kubectl get job,pods -n <namespace>
kubectl describe job <job-name> -n <namespace>
The events at the end of describe show what the Job controller did. A Job that ran out of retries or time shows BackoffLimitExceeded or DeadlineExceeded. These reasons only say that the Pods failed. The Pods themselves show why.
Docs:
- Jobs: Pod backoff failure policy (Kubernetes)
Hint 4: Reading both sides
The Pods of the Job are the publisher. Their logs show what the script printed, including any error from nats pub:
kubectl logs -n <namespace> -l job-name=<job-name>
The subscriber logs show whether a message arrived. Keep them open in a second terminal while you test:
kubectl logs -n <namespace> deployment/<name> -f
Docs:
- kubectl logs (Kubernetes)
- Publish-Subscribe (NATS)
Hint 5: Reading the error
An error such as dial tcp: lookup <host> on <ip>:53 means the client never reached the server. Before it opens a TCP connection, it asks the DNS server at <ip>, port 53, for the IP address of <host>. That lookup failed.
Repeat the lookup from a test Pod in the same namespace as the Job:
kubectl run <pod-name> -n <namespace> --image=<image> --restart=Never --command -- sleep 3600
kubectl exec <pod-name> -n <namespace> -- nslookup <host>
Docs:
- Debugging DNS Resolution (Kubernetes)
Hint 6: How Services get names
Every Service gets DNS names that include its namespace:
<service>.<namespace>
<service>.<namespace>.svc
<service>.<namespace>.svc.cluster.local
Find the Service that the script connects to, and its namespace:
kubectl get service -A
Docs:
- DNS for Services and Pods (Kubernetes)
Hint 7: How a short name is completed
A short name such as <service> is completed with the search domains in the Pod's /etc/resolv.conf. The first search domain is the Pod's own namespace:
kubectl exec <pod-name> -n <namespace> -- cat /etc/resolv.conf
Compare that namespace with the namespace of the Service. A Job's Pod template cannot be changed, so delete the Job and create it again with your correction.
Docs:
- DNS for Services and Pods: Namespaces of Services (Kubernetes)
âš’ Test Cases
Understanding the challenge
Three namespaces take part in this message flow:

A message published in checkout reaches payment-service through the NATS server.
The Job runs in checkout, and the NATS server runs in platform-nats-server. Two things decide the result:
- The address of the NATS server. The script uses
nats://nats:4222. A short Service name is resolved relative to the namespace of the Pod that looks it up. Fromcheckout,natsdoes not lead to the Service inplatform-nats-server, so the Job needs a name that includes the target namespace. - The Job settings. One successful Pod, at most one retry, 15 seconds in total, no container restarts, and automatic clean up after 1 hour.
The payment service is already running. When the Job sends ORD-1001, the payment service prints it in its logs. That log line proves that the message went from one namespace, through NATS, to another namespace.
Publish/subscribe with NATS
NATS is a message broker. Programs connect to it and exchange messages by subject, a name such as payments.requests. There are three roles:
- Subscriber: tells NATS "send me every message on this subject" and waits. Here,
payment-processorlistens onpayments.requests. - Publisher: sends a message to a subject and moves on. Here, your Job runs
nats pub ... payments.requests .... - Server: delivers each message to every subscriber of that subject. Here, the
natsDeployment inplatform-nats-server.
The publisher and the subscriber never talk to each other directly. They only need to reach the same NATS server.
Reaching a Service in another namespace
Every Service gets a DNS name in this format:
<service-name>.<namespace>.svc.cluster.local
For the nats Service in platform-nats-server:
natsis a short name. It is resolved relative to the namespace of the Pod that looks it up, so from a Pod inplatform-nats-serverit leads to this Service.nats.platform-nats-serverincludes the namespace and leads to this Service from any namespace.nats.platform-nats-server.svcworks the same way.nats.platform-nats-server.svc.cluster.localis the full name and works from any namespace.
With the scheme and the client port, the full NATS address is:
nats://nats.platform-nats-server.svc.cluster.local:4222
The search domains in the Pod's /etc/resolv.conf complete a short name, and the first one is the Pod's own namespace. A Pod in checkout gets:
search checkout.svc.cluster.local svc.cluster.local cluster.local
So nats is tried as nats.checkout.svc.cluster.local, then nats.svc.cluster.local, then nats.cluster.local, and last as nats on its own. There is no nats Service in checkout, and the other names do not exist either, so the lookup fails. If checkout had its own nats Service, the short name would lead to that one instead.
Job settings
Each setting from the task maps to one field:
- Successful Pods needed
1:spec.completions: 1 - One Pod at a time:
spec.parallelism: 1 - Retries of a failed Pod, at most
1:spec.backoffLimit: 1 - Time limit for the whole Job,
15seconds:spec.activeDeadlineSeconds: 15 - A failed container is not restarted inside the same Pod:
spec.template.spec.restartPolicy: Never - Deleted automatically 1 hour after it finishes:
spec.ttlSecondsAfterFinished: 3600 - Container command and args: the script from the task
How the limits work together:
- With
restartPolicy: Never, a failed container is not restarted. Instead, the Job creates a new Pod. backoffLimit: 1allows one such retry. When the number of failed Pods goes over the limit, the Job fails withBackoffLimitExceeded.- Kubernetes waits before each retry. The first wait is about 10 seconds, and later waits are longer.
activeDeadlineSeconds: 15stops the whole Job after 15 seconds, retries included. The Job then fails withDeadlineExceeded.
Whichever limit is reached first ends the Job. A failed Job stays failed, and its Pod template cannot be changed, so a fixed Job has to be created again.
Step 1: Look at what is running
The output in these steps comes from a real run. In your cluster, the Pod names, the Pod IP addresses, the Service IP addresses and the times will be different. Use the values from your own output.
Start with every Pod in the cluster:
kubectl get po -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-577d995dff-zqdwg 1/1 Running 0 42s
kube-system helm-install-gateway-api-crd-csxss 0/1 Completed 0 43s
kube-system helm-install-traefik-br75n 0/1 Completed 1 (29s ago) 43s
kube-system helm-install-traefik-crd-fwflp 0/1 Completed 0 43s
kube-system local-path-provisioner-6858d854cf-zs675 1/1 Running 0 42s
kube-system metrics-server-778c4649b4-pdtd5 1/1 Running 0 42s
kube-system svclb-traefik-17a6ffb5-fj455 2/2 Running 0 27s
kube-system svclb-traefik-17a6ffb5-s5n7d 2/2 Running 0 27s
kube-system svclb-traefik-17a6ffb5-s9rc2 2/2 Running 0 27s
kube-system traefik-77b898c5c8-88g8g 1/1 Running 0 27s
payment-service payment-processor-b5649d7d8-vwnvp 1/1 Running 0 22s
platform-nats-server nats-78c869f6b4-6n9fk 1/1 Running 0 23s
payment-processor runs in payment-service, and a Pod named nats runs in platform-nats-server. No Pod runs in checkout.
The task does not say where the NATS server runs. The nats Pod is a good guess. Clients connect to the NATS server on port 4222, so confirm it with the Services of all namespaces:
kubectl get service -A
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default kubernetes ClusterIP 10.43.0.1 <none> 443/TCP 49s
kube-system kube-dns ClusterIP 10.43.0.10 <none> 53/UDP,53/TCP,9153/TCP 47s
kube-system metrics-server ClusterIP 10.43.111.122 <none> 443/TCP 47s
kube-system traefik LoadBalancer 10.43.53.242 172.16.0.2,172.16.0.3,172.16.0.4 80:32634/TCP,443:32431/TCP 27s
platform-nats-server nats ClusterIP 10.43.17.224 <none> 4222/TCP,8222/TCP 23s
The nats Service in the platform-nats-server namespace has port 4222. See where it sends the traffic for that port:
kubectl describe service nats -n platform-nats-server
Name: nats
Namespace: platform-nats-server
Labels: <none>
Annotations: <none>
Selector: app=nats
Type: ClusterIP
IP Family Policy: SingleStack
IP Families: IPv4
IP: 10.43.17.224
IPs: 10.43.17.224
Port: client 4222/TCP
TargetPort: 4222/TCP
Endpoints: 10.42.2.3:4222
Port: monitor 8222/TCP
TargetPort: 8222/TCP
Endpoints: 10.42.2.3:8222
Session Affinity: None
Internal Traffic Policy: Cluster
Events: <none>
Port: client 4222/TCPis the port of the Service.TargetPort: 4222/TCPis the port on the Pod that receives the traffic.Endpoints: 10.42.2.3:4222is the IP address and port of that Pod.
Find the Pod with that IP address:
kubectl get pods -n platform-nats-server -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nats-78c869f6b4-6n9fk 1/1 Running 0 23s 10.42.2.3 node-02 <none> <none>
Its logs show that the NATS server listens for clients on port 4222:
kubectl logs -n platform-nats-server deployment/nats
[1] 2026/11/01 14:12:10.644377 [INF] Starting nats-server
[1] 2026/11/01 14:12:10.644478 [INF] Version: 2.15.2
[1] 2026/11/01 14:12:10.644500 [INF] Git: [5a815f1]
[1] 2026/11/01 14:12:10.644556 [INF] Name: NDGVSTHD5UC7LO4GZV622HXBULU6ZOBW63RITT5QSAZ5BLVPDRTCL7VC
[1] 2026/11/01 14:12:10.644576 [INF] ID: NDGVSTHD5UC7LO4GZV622HXBULU6ZOBW63RITT5QSAZ5BLVPDRTCL7VC
[1] 2026/11/01 14:12:10.644923 [INF] Starting http monitor on 0.0.0.0:8222
[1] 2026/11/01 14:12:10.645154 [INF] Listening for client connections on 0.0.0.0:4222
[1] 2026/11/01 14:12:10.645318 [INF] Server is ready
The NATS server runs in platform-nats-server, and the nats Service sends port 4222 to it. Step 4 comes back to this namespace.
Then look at the subscriber in payment-service:
kubectl get deployment,pods -n payment-service
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/payment-processor 1/1 1 1 24s
NAME READY STATUS RESTARTS AGE
pod/payment-processor-b5649d7d8-vwnvp 1/1 Running 0 23s
kubectl logs -n payment-service deployment/payment-processor
14:12:13 payment-processor is listening for payment requests
Last, look at checkout, the namespace where the Job will run:
kubectl get all -n checkout
No resources found in checkout namespace.
The payment service is listening. checkout is empty: no Pods, no Jobs and no Services. So there is no nats Service in checkout, and the NATS server is in another namespace.
Step 2: Write the Job manifest with the team's script
The command and args from the task, part by part:
command: ["/bin/sh", "-c"]:/bin/shstarts a shell, and-ctells it to run the next argument as a script.argsholds that script.echo "sending payment request ORD-1001"prints a line to the Job log, so you can see the script started.nats pubpublishes one message, then exits.-s nats://nats:4222is the NATS server to connect to: the Service namenatsand the client port4222.payments.requestsis the subject the payment service subscribes to.'{"order_id":"ORD-1001","amount":49.99}'is the message body.
If nats pub cannot reach the broker, it exits with an error code, so the Pod fails. If it succeeds, the Pod completes.
Write the Job manifest:
vi /home/laborant/payment-pubsub-test.yaml
apiVersion: batch/v1
kind: Job
metadata:
name: payment-pubsub-test
namespace: checkout
spec:
completions: 1 # 1 successful Pod needed
parallelism: 1 # one Pod at a time
backoffLimit: 1 # at most 1 retry of a failed Pod
activeDeadlineSeconds: 15 # whole Job, retries included
ttlSecondsAfterFinished: 3600 # delete the Job 1 hour after it finishes
template:
spec:
restartPolicy: Never # failed container is not restarted in the same Pod
containers:
- name: pubsub-test
image: natsio/nats-box:0.20.1 # image with the nats CLI
command: ["/bin/sh", "-c"] # run args as a shell script
args: # the two lines below are the script
- |
echo "sending payment request ORD-1001"
nats pub -s nats://nats:4222 payments.requests '{"order_id":"ORD-1001","amount":49.99}'
You can start from a generated skeleton and add the missing fields:
kubectl create job payment-pubsub-test -n checkout --image=natsio/nats-box:0.20.1 --dry-run=client -o yaml
Step 3: Create the Job and watch it fail
kubectl apply -f /home/laborant/payment-pubsub-test.yaml
job.batch/payment-pubsub-test created
After about 25 seconds:
kubectl get job,pods -n checkout
NAME STATUS COMPLETIONS DURATION AGE
job.batch/payment-pubsub-test Failed 0/1 26s 26s
NAME READY STATUS RESTARTS AGE
pod/payment-pubsub-test-7grtk 0/1 Error 0 26s
pod/payment-pubsub-test-swjls 0/1 Error 0 15s
The Job failed. It ran two Pods: the first one and its one retry. Both ended in Error. Read their logs:
kubectl logs -n checkout -l job-name=payment-pubsub-test
sending payment request ORD-1001
nats: error: dial tcp: lookup nats on 10.43.0.10:53: no such host
sending payment request ORD-1001
nats: error: dial tcp: lookup nats on 10.43.0.10:53: server misbehaving
The script started, so the echo line is there. Then nats pub failed. Read the error from left to right:
nats: error:thenatsCLI stopped with an error. The script exited with a non-zero code, so the Pod ended inError.dial tcp:it was opening a TCP connection to the server set with-s nats://nats:4222.lookup nats: to open that connection, it first needs the IP address of the hostnats, so it asks DNS.on 10.43.0.10:53: the DNS server it asked, on port53. This is the cluster DNS from the Pod's/etc/resolv.conf.no such hostorserver misbehaving: the lookup failed. The connection was never opened, so no message was sent.
Both Pods failed in the same lookup, but their errors end differently. Step 4 shows where the two endings come from.
The address 10.43.0.10 belongs to the kube-dns Service, the cluster DNS:
kubectl get service kube-dns -n kube-system
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kube-dns ClusterIP 10.43.0.10 <none> 53/UDP,53/TCP,9153/TCP 74s
kubectl describe job payment-pubsub-test -n checkout ends with these events:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal SuccessfulCreate 26s job-controller Created pod: payment-pubsub-test-7grtk
Normal SuccessfulCreate 15s job-controller Created pod: payment-pubsub-test-swjls
Warning BackoffLimitExceeded 12s job-controller Job has reached the specified backoff limit
The retry started 11 seconds after the first Pod and failed too. That was the second failed Pod, more than backoffLimit: 1 allows.
The DURATION of 26s above does not mean that the Job ran past its 15 second limit. For a Job that failed, kubectl get job keeps counting from the start time. The status part of kubectl get job payment-pubsub-test -n checkout -o yaml shows when the Job really failed:
status:
conditions:
- lastProbeTime: "2026-11-01T14:12:48Z"
lastTransitionTime: "2026-11-01T14:12:48Z"
message: Job has reached the specified backoff limit
reason: BackoffLimitExceeded
status: "True"
type: FailureTarget
- lastProbeTime: "2026-11-01T14:12:48Z"
lastTransitionTime: "2026-11-01T14:12:48Z"
message: Job has reached the specified backoff limit
reason: BackoffLimitExceeded
status: "True"
type: Failed
failed: 2
ready: 0
startTime: "2026-11-01T14:12:34Z"
terminating: 0
uncountedTerminatedPods: {}
The Job started at 14:12:34 and failed at 14:12:48, 14 seconds later and inside the 15 second deadline. Here the retry failed quickly, so backoffLimit ended the Job first. If the retry is slower and still running at 15 seconds, activeDeadlineSeconds ends the Job with DeadlineExceeded instead. Either reason can show up, depending on timing, and both come from your own settings.
Step 4: Find an address that works from checkout
Start a test Pod with DNS tools in checkout. This image is the one from the Kubernetes guide Debugging DNS Resolution, and it has nslookup and dig:
kubectl run dns-test -n checkout --image=registry.k8s.io/e2e-test-images/jessie-dnsutils:1.3 --restart=Never --command -- sleep 3600
pod/dns-test created
Look at its DNS search domains:
kubectl exec dns-test -n checkout -- cat /etc/resolv.conf
search checkout.svc.cluster.local svc.cluster.local cluster.local
nameserver 10.43.0.10
options ndots:5
options ndots:5 means that a name with fewer than 5 dots is first tried with each search domain, and then on its own. nats has no dots, so it is tried as nats.checkout.svc.cluster.local, nats.svc.cluster.local, nats.cluster.local, and last as nats.
Look up the short name, the same way the Job did:
kubectl exec dns-test -n checkout -- nslookup nats
Server: 10.43.0.10
Address: 10.43.0.10#53
** server can't find nats: SERVFAIL
command terminated with exit code 1
nslookup prints only the answer for the last name it tried, nats on its own. That name is outside the cluster domain, so the cluster DNS forwards it to the DNS server outside the cluster. In this run that server answered with an error, SERVFAIL. An answer like this ends the Job log with server misbehaving. When the outside DNS server answers NXDOMAIN, "this name does not exist", the Job log ends with no such host. The answer can change from one lookup to the next, which is why the two Pods in Step 3 got different endings.
The name that matters is the first one, built with the checkout search domain:
kubectl exec dns-test -n checkout -- nslookup nats.checkout.svc.cluster.local
Server: 10.43.0.10
Address: 10.43.0.10#53
** server can't find nats.checkout.svc.cluster.local: NXDOMAIN
command terminated with exit code 1
NXDOMAIN means that the name does not exist: there is no nats Service in checkout. In Step 1 you found the nats Service in platform-nats-server. Only in a Pod in that namespace would the same short name become nats.platform-nats-server.svc.cluster.local.
Look up the full name of the Service from Step 1:
kubectl exec dns-test -n checkout -- nslookup nats.platform-nats-server.svc.cluster.local
Server: 10.43.0.10
Address: 10.43.0.10#53
Name: nats.platform-nats-server.svc.cluster.local
Address: 10.43.17.224
The full name resolves to 10.43.17.224, the cluster IP of the nats Service from Step 1. Delete the test Pod:
kubectl delete pod dns-test -n checkout --now
pod "dns-test" deleted from checkout namespace
Step 5: Fix the address and create the Job again
A Job's Pod template cannot be changed, so delete the failed Job first:
kubectl delete job payment-pubsub-test -n checkout
job.batch "payment-pubsub-test" deleted from checkout namespace
In /home/laborant/payment-pubsub-test.yaml, change only the server address in the nats pub line. The subject and the message stay as they are:
echo "sending payment request ORD-1001"
nats pub -s nats://nats.platform-nats-server.svc.cluster.local:4222 payments.requests '{"order_id":"ORD-1001","amount":49.99}'
nats.platform-nats-server or nats.platform-nats-server.svc work as well.
Changing the address is the simplest fix. Two other fixes also make the unchanged script work: an ExternalName Service named nats in checkout that points to nats.platform-nats-server.svc.cluster.local, or a dnsConfig.searches entry with platform-nats-server.svc.cluster.local in the Pod template of the Job. The checks accept all three, because they look at the result.
kubectl apply -f /home/laborant/payment-pubsub-test.yaml
job.batch/payment-pubsub-test created
kubectl wait job/payment-pubsub-test -n checkout --for=condition=Complete --timeout=30s
job.batch/payment-pubsub-test condition met
kubectl get job,pods -n checkout
NAME STATUS COMPLETIONS DURATION AGE
job.batch/payment-pubsub-test Complete 1/1 3s 3s
NAME READY STATUS RESTARTS AGE
pod/payment-pubsub-test-knzx5 0/1 Completed 0 3s
In this run the Job completed in 3 seconds, well inside the 15 second limit.
Step 6: Check both sides of the message
The sender, your Job:
kubectl logs -n checkout job/payment-pubsub-test
sending payment request ORD-1001
14:13:36 Published 38 bytes to "payments.requests"
The listener, the payment service:
kubectl logs -n payment-service deployment/payment-processor
14:12:13 payment-processor is listening for payment requests
14:13:36 received payment request: {"order_id":"ORD-1001","amount":49.99}
The message sent from checkout at 14:13:36 reached the payment service in the same second. The failed Job from Step 3 never reached the broker, so the log has no request from it.
Published and received are two different things. Published 38 bytes in the Job log only means that the NATS server accepted the message. Core NATS does not confirm delivery and does not keep messages for subscribers that connect later. Only the received payment request line in the payment service log proves that the message arrived.
Step 7: Confirm the Job settings
kubectl describe job payment-pubsub-test -n checkout
Name: payment-pubsub-test
Namespace: checkout
Selector: batch.kubernetes.io/controller-uid=bf554134-718e-4f81-ac62-e7d36eff6f0c
Labels: batch.kubernetes.io/controller-uid=bf554134-718e-4f81-ac62-e7d36eff6f0c
batch.kubernetes.io/job-name=payment-pubsub-test
controller-uid=bf554134-718e-4f81-ac62-e7d36eff6f0c
job-name=payment-pubsub-test
Annotations: <none>
Parallelism: 1
Completions: 1
Completion Mode: NonIndexed
Suspend: false
Backoff Limit: 1
TTL Seconds After Finished: 3600
Start Time: Sun, 01 Nov 2026 14:13:35 +0000
Completed At: Sun, 01 Nov 2026 14:13:38 +0000
Duration: 3s
Active Deadline Seconds: 15s
Pods Statuses: 0 Active (0 Ready) / 1 Succeeded / 0 Failed
Pod Template:
Labels: batch.kubernetes.io/controller-uid=bf554134-718e-4f81-ac62-e7d36eff6f0c
batch.kubernetes.io/job-name=payment-pubsub-test
controller-uid=bf554134-718e-4f81-ac62-e7d36eff6f0c
job-name=payment-pubsub-test
Containers:
pubsub-test:
Image: natsio/nats-box:0.20.1
Port: <none>
Host Port: <none>
Command:
/bin/sh
-c
Args:
echo "sending payment request ORD-1001"
nats pub -s nats://nats.platform-nats-server.svc.cluster.local:4222 payments.requests '{"order_id":"ORD-1001","amount":49.99}'
Environment: <none>
Mounts: <none>
Volumes: <none>
Node-Selectors: <none>
Tolerations: <none>
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal SuccessfulCreate 3s job-controller Created pod: payment-pubsub-test-knzx5
Normal Completed 0s job-controller Job completed
The highlighted lines show every setting from the task except the restart policy. describe does not list it, so read it from the Pod template:
kubectl get job payment-pubsub-test -n checkout -o jsonpath='{.spec.template.spec.restartPolicy}'
Never
ttlSecondsAfterFinished: 3600 deletes the Job and its Pod 1 hour after it finishes. Read the Job logs before that, or they are gone with the Pod.
Summary
- Find the cause of the failure: the Pod logs show
lookup nats on 10.43.0.10:53: no such host. A short Service name only resolves from Pods in the same namespace as the Service. - Reach NATS from
checkout:-s nats://nats.platform-nats-server.svc.cluster.local:4222, with the subject and the message unchanged. - Recreate the Job: a Job's Pod template cannot be changed, so delete it and apply the fixed manifest.
- One Pod, one at a time:
completions: 1,parallelism: 1. - At most one retry:
backoffLimit: 1withrestartPolicy: Never. - 15 seconds in total:
activeDeadlineSeconds: 15. - Clean up after 1 hour:
ttlSecondsAfterFinished: 3600. - Proof that pub/sub works: the payment log shows
received payment request: {"order_id":"ORD-1001","amount":49.99}. The Job log alone only proves that the message was published.
Understanding the challenge
Three namespaces take part in this message flow:

A message published in checkout reaches payment-service through the NATS server.
The Job runs in checkout, and the NATS server runs in platform-nats-server. Two things decide the result:
- The address of the NATS server. The script uses
nats://nats:4222. A short Service name is resolved relative to the namespace of the Pod that looks it up. Fromcheckout,natsdoes not lead to the Service inplatform-nats-server, so the Job needs a name that includes the target namespace. - The Job settings. One successful Pod, at most one retry, 15 seconds in total, no container restarts, and automatic clean up after 1 hour.
The payment service is already running. When the Job sends ORD-1001, the payment service prints it in its logs. That log line proves that the message went from one namespace, through NATS, to another namespace.
Publish/subscribe with NATS
NATS is a message broker. Programs connect to it and exchange messages by subject, a name such as payments.requests. There are three roles:
- Subscriber: tells NATS "send me every message on this subject" and waits. Here,
payment-processorlistens onpayments.requests. - Publisher: sends a message to a subject and moves on. Here, your Job runs
nats pub ... payments.requests .... - Server: delivers each message to every subscriber of that subject. Here, the
natsDeployment inplatform-nats-server.
The publisher and the subscriber never talk to each other directly. They only need to reach the same NATS server.
Reaching a Service in another namespace
Every Service gets a DNS name in this format:
<service-name>.<namespace>.svc.cluster.local
For the nats Service in platform-nats-server:
natsis a short name. It is resolved relative to the namespace of the Pod that looks it up, so from a Pod inplatform-nats-serverit leads to this Service.nats.platform-nats-serverincludes the namespace and leads to this Service from any namespace.nats.platform-nats-server.svcworks the same way.nats.platform-nats-server.svc.cluster.localis the full name and works from any namespace.
With the scheme and the client port, the full NATS address is:
nats://nats.platform-nats-server.svc.cluster.local:4222
The search domains in the Pod's /etc/resolv.conf complete a short name, and the first one is the Pod's own namespace. A Pod in checkout gets:
search checkout.svc.cluster.local svc.cluster.local cluster.local
So nats is tried as nats.checkout.svc.cluster.local, then nats.svc.cluster.local, then nats.cluster.local, and last as nats on its own. There is no nats Service in checkout, and the other names do not exist either, so the lookup fails. If checkout had its own nats Service, the short name would lead to that one instead.
Job settings
Each setting from the task maps to one field:
- Successful Pods needed
1:spec.completions: 1 - One Pod at a time:
spec.parallelism: 1 - Retries of a failed Pod, at most
1:spec.backoffLimit: 1 - Time limit for the whole Job,
15seconds:spec.activeDeadlineSeconds: 15 - A failed container is not restarted inside the same Pod:
spec.template.spec.restartPolicy: Never - Deleted automatically 1 hour after it finishes:
spec.ttlSecondsAfterFinished: 3600 - Container command and args: the script from the task
How the limits work together:
- With
restartPolicy: Never, a failed container is not restarted. Instead, the Job creates a new Pod. backoffLimit: 1allows one such retry. When the number of failed Pods goes over the limit, the Job fails withBackoffLimitExceeded.- Kubernetes waits before each retry. The first wait is about 10 seconds, and later waits are longer.
activeDeadlineSeconds: 15stops the whole Job after 15 seconds, retries included. The Job then fails withDeadlineExceeded.
Whichever limit is reached first ends the Job. A failed Job stays failed, and its Pod template cannot be changed, so a fixed Job has to be created again.
Step 1: Look at what is running
The output in these steps comes from a real run. In your cluster, the Pod names, the Pod IP addresses, the Service IP addresses and the times will be different. Use the values from your own output.
Start with every Pod in the cluster:
kubectl get po -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-577d995dff-zqdwg 1/1 Running 0 42s
kube-system helm-install-gateway-api-crd-csxss 0/1 Completed 0 43s
kube-system helm-install-traefik-br75n 0/1 Completed 1 (29s ago) 43s
kube-system helm-install-traefik-crd-fwflp 0/1 Completed 0 43s
kube-system local-path-provisioner-6858d854cf-zs675 1/1 Running 0 42s
kube-system metrics-server-778c4649b4-pdtd5 1/1 Running 0 42s
kube-system svclb-traefik-17a6ffb5-fj455 2/2 Running 0 27s
kube-system svclb-traefik-17a6ffb5-s5n7d 2/2 Running 0 27s
kube-system svclb-traefik-17a6ffb5-s9rc2 2/2 Running 0 27s
kube-system traefik-77b898c5c8-88g8g 1/1 Running 0 27s
payment-service payment-processor-b5649d7d8-vwnvp 1/1 Running 0 22s
platform-nats-server nats-78c869f6b4-6n9fk 1/1 Running 0 23s
payment-processor runs in payment-service, and a Pod named nats runs in platform-nats-server. No Pod runs in checkout.
The task does not say where the NATS server runs. The nats Pod is a good guess. Clients connect to the NATS server on port 4222, so confirm it with the Services of all namespaces:
kubectl get service -A
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default kubernetes ClusterIP 10.43.0.1 <none> 443/TCP 49s
kube-system kube-dns ClusterIP 10.43.0.10 <none> 53/UDP,53/TCP,9153/TCP 47s
kube-system metrics-server ClusterIP 10.43.111.122 <none> 443/TCP 47s
kube-system traefik LoadBalancer 10.43.53.242 172.16.0.2,172.16.0.3,172.16.0.4 80:32634/TCP,443:32431/TCP 27s
platform-nats-server nats ClusterIP 10.43.17.224 <none> 4222/TCP,8222/TCP 23s
The nats Service in the platform-nats-server namespace has port 4222. See where it sends the traffic for that port:
kubectl describe service nats -n platform-nats-server
Name: nats
Namespace: platform-nats-server
Labels: <none>
Annotations: <none>
Selector: app=nats
Type: ClusterIP
IP Family Policy: SingleStack
IP Families: IPv4
IP: 10.43.17.224
IPs: 10.43.17.224
Port: client 4222/TCP
TargetPort: 4222/TCP
Endpoints: 10.42.2.3:4222
Port: monitor 8222/TCP
TargetPort: 8222/TCP
Endpoints: 10.42.2.3:8222
Session Affinity: None
Internal Traffic Policy: Cluster
Events: <none>
Port: client 4222/TCPis the port of the Service.TargetPort: 4222/TCPis the port on the Pod that receives the traffic.Endpoints: 10.42.2.3:4222is the IP address and port of that Pod.
Find the Pod with that IP address:
kubectl get pods -n platform-nats-server -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nats-78c869f6b4-6n9fk 1/1 Running 0 23s 10.42.2.3 node-02 <none> <none>
Its logs show that the NATS server listens for clients on port 4222:
kubectl logs -n platform-nats-server deployment/nats
[1] 2026/11/01 14:12:10.644377 [INF] Starting nats-server
[1] 2026/11/01 14:12:10.644478 [INF] Version: 2.15.2
[1] 2026/11/01 14:12:10.644500 [INF] Git: [5a815f1]
[1] 2026/11/01 14:12:10.644556 [INF] Name: NDGVSTHD5UC7LO4GZV622HXBULU6ZOBW63RITT5QSAZ5BLVPDRTCL7VC
[1] 2026/11/01 14:12:10.644576 [INF] ID: NDGVSTHD5UC7LO4GZV622HXBULU6ZOBW63RITT5QSAZ5BLVPDRTCL7VC
[1] 2026/11/01 14:12:10.644923 [INF] Starting http monitor on 0.0.0.0:8222
[1] 2026/11/01 14:12:10.645154 [INF] Listening for client connections on 0.0.0.0:4222
[1] 2026/11/01 14:12:10.645318 [INF] Server is ready
The NATS server runs in platform-nats-server, and the nats Service sends port 4222 to it. Step 4 comes back to this namespace.
Then look at the subscriber in payment-service:
kubectl get deployment,pods -n payment-service
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/payment-processor 1/1 1 1 24s
NAME READY STATUS RESTARTS AGE
pod/payment-processor-b5649d7d8-vwnvp 1/1 Running 0 23s
kubectl logs -n payment-service deployment/payment-processor
14:12:13 payment-processor is listening for payment requests
Last, look at checkout, the namespace where the Job will run:
kubectl get all -n checkout
No resources found in checkout namespace.
The payment service is listening. checkout is empty: no Pods, no Jobs and no Services. So there is no nats Service in checkout, and the NATS server is in another namespace.
Step 2: Write the Job manifest with the team's script
The command and args from the task, part by part:
command: ["/bin/sh", "-c"]:/bin/shstarts a shell, and-ctells it to run the next argument as a script.argsholds that script.echo "sending payment request ORD-1001"prints a line to the Job log, so you can see the script started.nats pubpublishes one message, then exits.-s nats://nats:4222is the NATS server to connect to: the Service namenatsand the client port4222.payments.requestsis the subject the payment service subscribes to.'{"order_id":"ORD-1001","amount":49.99}'is the message body.
If nats pub cannot reach the broker, it exits with an error code, so the Pod fails. If it succeeds, the Pod completes.
Write the Job manifest:
vi /home/laborant/payment-pubsub-test.yaml
apiVersion: batch/v1
kind: Job
metadata:
name: payment-pubsub-test
namespace: checkout
spec:
completions: 1 # 1 successful Pod needed
parallelism: 1 # one Pod at a time
backoffLimit: 1 # at most 1 retry of a failed Pod
activeDeadlineSeconds: 15 # whole Job, retries included
ttlSecondsAfterFinished: 3600 # delete the Job 1 hour after it finishes
template:
spec:
restartPolicy: Never # failed container is not restarted in the same Pod
containers:
- name: pubsub-test
image: natsio/nats-box:0.20.1 # image with the nats CLI
command: ["/bin/sh", "-c"] # run args as a shell script
args: # the two lines below are the script
- |
echo "sending payment request ORD-1001"
nats pub -s nats://nats:4222 payments.requests '{"order_id":"ORD-1001","amount":49.99}'
You can start from a generated skeleton and add the missing fields:
kubectl create job payment-pubsub-test -n checkout --image=natsio/nats-box:0.20.1 --dry-run=client -o yaml
Step 3: Create the Job and watch it fail
kubectl apply -f /home/laborant/payment-pubsub-test.yaml
job.batch/payment-pubsub-test created
After about 25 seconds:
kubectl get job,pods -n checkout
NAME STATUS COMPLETIONS DURATION AGE
job.batch/payment-pubsub-test Failed 0/1 26s 26s
NAME READY STATUS RESTARTS AGE
pod/payment-pubsub-test-7grtk 0/1 Error 0 26s
pod/payment-pubsub-test-swjls 0/1 Error 0 15s
The Job failed. It ran two Pods: the first one and its one retry. Both ended in Error. Read their logs:
kubectl logs -n checkout -l job-name=payment-pubsub-test
sending payment request ORD-1001
nats: error: dial tcp: lookup nats on 10.43.0.10:53: no such host
sending payment request ORD-1001
nats: error: dial tcp: lookup nats on 10.43.0.10:53: server misbehaving
The script started, so the echo line is there. Then nats pub failed. Read the error from left to right:
nats: error:thenatsCLI stopped with an error. The script exited with a non-zero code, so the Pod ended inError.dial tcp:it was opening a TCP connection to the server set with-s nats://nats:4222.lookup nats: to open that connection, it first needs the IP address of the hostnats, so it asks DNS.on 10.43.0.10:53: the DNS server it asked, on port53. This is the cluster DNS from the Pod's/etc/resolv.conf.no such hostorserver misbehaving: the lookup failed. The connection was never opened, so no message was sent.
Both Pods failed in the same lookup, but their errors end differently. Step 4 shows where the two endings come from.
The address 10.43.0.10 belongs to the kube-dns Service, the cluster DNS:
kubectl get service kube-dns -n kube-system
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kube-dns ClusterIP 10.43.0.10 <none> 53/UDP,53/TCP,9153/TCP 74s
kubectl describe job payment-pubsub-test -n checkout ends with these events:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal SuccessfulCreate 26s job-controller Created pod: payment-pubsub-test-7grtk
Normal SuccessfulCreate 15s job-controller Created pod: payment-pubsub-test-swjls
Warning BackoffLimitExceeded 12s job-controller Job has reached the specified backoff limit
The retry started 11 seconds after the first Pod and failed too. That was the second failed Pod, more than backoffLimit: 1 allows.
The DURATION of 26s above does not mean that the Job ran past its 15 second limit. For a Job that failed, kubectl get job keeps counting from the start time. The status part of kubectl get job payment-pubsub-test -n checkout -o yaml shows when the Job really failed:
status:
conditions:
- lastProbeTime: "2026-11-01T14:12:48Z"
lastTransitionTime: "2026-11-01T14:12:48Z"
message: Job has reached the specified backoff limit
reason: BackoffLimitExceeded
status: "True"
type: FailureTarget
- lastProbeTime: "2026-11-01T14:12:48Z"
lastTransitionTime: "2026-11-01T14:12:48Z"
message: Job has reached the specified backoff limit
reason: BackoffLimitExceeded
status: "True"
type: Failed
failed: 2
ready: 0
startTime: "2026-11-01T14:12:34Z"
terminating: 0
uncountedTerminatedPods: {}
The Job started at 14:12:34 and failed at 14:12:48, 14 seconds later and inside the 15 second deadline. Here the retry failed quickly, so backoffLimit ended the Job first. If the retry is slower and still running at 15 seconds, activeDeadlineSeconds ends the Job with DeadlineExceeded instead. Either reason can show up, depending on timing, and both come from your own settings.
Step 4: Find an address that works from checkout
Start a test Pod with DNS tools in checkout. This image is the one from the Kubernetes guide Debugging DNS Resolution, and it has nslookup and dig:
kubectl run dns-test -n checkout --image=registry.k8s.io/e2e-test-images/jessie-dnsutils:1.3 --restart=Never --command -- sleep 3600
pod/dns-test created
Look at its DNS search domains:
kubectl exec dns-test -n checkout -- cat /etc/resolv.conf
search checkout.svc.cluster.local svc.cluster.local cluster.local
nameserver 10.43.0.10
options ndots:5
options ndots:5 means that a name with fewer than 5 dots is first tried with each search domain, and then on its own. nats has no dots, so it is tried as nats.checkout.svc.cluster.local, nats.svc.cluster.local, nats.cluster.local, and last as nats.
Look up the short name, the same way the Job did:
kubectl exec dns-test -n checkout -- nslookup nats
Server: 10.43.0.10
Address: 10.43.0.10#53
** server can't find nats: SERVFAIL
command terminated with exit code 1
nslookup prints only the answer for the last name it tried, nats on its own. That name is outside the cluster domain, so the cluster DNS forwards it to the DNS server outside the cluster. In this run that server answered with an error, SERVFAIL. An answer like this ends the Job log with server misbehaving. When the outside DNS server answers NXDOMAIN, "this name does not exist", the Job log ends with no such host. The answer can change from one lookup to the next, which is why the two Pods in Step 3 got different endings.
The name that matters is the first one, built with the checkout search domain:
kubectl exec dns-test -n checkout -- nslookup nats.checkout.svc.cluster.local
Server: 10.43.0.10
Address: 10.43.0.10#53
** server can't find nats.checkout.svc.cluster.local: NXDOMAIN
command terminated with exit code 1
NXDOMAIN means that the name does not exist: there is no nats Service in checkout. In Step 1 you found the nats Service in platform-nats-server. Only in a Pod in that namespace would the same short name become nats.platform-nats-server.svc.cluster.local.
Look up the full name of the Service from Step 1:
kubectl exec dns-test -n checkout -- nslookup nats.platform-nats-server.svc.cluster.local
Server: 10.43.0.10
Address: 10.43.0.10#53
Name: nats.platform-nats-server.svc.cluster.local
Address: 10.43.17.224
The full name resolves to 10.43.17.224, the cluster IP of the nats Service from Step 1. Delete the test Pod:
kubectl delete pod dns-test -n checkout --now
pod "dns-test" deleted from checkout namespace
Step 5: Fix the address and create the Job again
A Job's Pod template cannot be changed, so delete the failed Job first:
kubectl delete job payment-pubsub-test -n checkout
job.batch "payment-pubsub-test" deleted from checkout namespace
In /home/laborant/payment-pubsub-test.yaml, change only the server address in the nats pub line. The subject and the message stay as they are:
echo "sending payment request ORD-1001"
nats pub -s nats://nats.platform-nats-server.svc.cluster.local:4222 payments.requests '{"order_id":"ORD-1001","amount":49.99}'
nats.platform-nats-server or nats.platform-nats-server.svc work as well.
Changing the address is the simplest fix. Two other fixes also make the unchanged script work: an ExternalName Service named nats in checkout that points to nats.platform-nats-server.svc.cluster.local, or a dnsConfig.searches entry with platform-nats-server.svc.cluster.local in the Pod template of the Job. The checks accept all three, because they look at the result.
kubectl apply -f /home/laborant/payment-pubsub-test.yaml
job.batch/payment-pubsub-test created
kubectl wait job/payment-pubsub-test -n checkout --for=condition=Complete --timeout=30s
job.batch/payment-pubsub-test condition met
kubectl get job,pods -n checkout
NAME STATUS COMPLETIONS DURATION AGE
job.batch/payment-pubsub-test Complete 1/1 3s 3s
NAME READY STATUS RESTARTS AGE
pod/payment-pubsub-test-knzx5 0/1 Completed 0 3s
In this run the Job completed in 3 seconds, well inside the 15 second limit.
Step 6: Check both sides of the message
The sender, your Job:
kubectl logs -n checkout job/payment-pubsub-test
sending payment request ORD-1001
14:13:36 Published 38 bytes to "payments.requests"
The listener, the payment service:
kubectl logs -n payment-service deployment/payment-processor
14:12:13 payment-processor is listening for payment requests
14:13:36 received payment request: {"order_id":"ORD-1001","amount":49.99}
The message sent from checkout at 14:13:36 reached the payment service in the same second. The failed Job from Step 3 never reached the broker, so the log has no request from it.
Published and received are two different things. Published 38 bytes in the Job log only means that the NATS server accepted the message. Core NATS does not confirm delivery and does not keep messages for subscribers that connect later. Only the received payment request line in the payment service log proves that the message arrived.
Step 7: Confirm the Job settings
kubectl describe job payment-pubsub-test -n checkout
Name: payment-pubsub-test
Namespace: checkout
Selector: batch.kubernetes.io/controller-uid=bf554134-718e-4f81-ac62-e7d36eff6f0c
Labels: batch.kubernetes.io/controller-uid=bf554134-718e-4f81-ac62-e7d36eff6f0c
batch.kubernetes.io/job-name=payment-pubsub-test
controller-uid=bf554134-718e-4f81-ac62-e7d36eff6f0c
job-name=payment-pubsub-test
Annotations: <none>
Parallelism: 1
Completions: 1
Completion Mode: NonIndexed
Suspend: false
Backoff Limit: 1
TTL Seconds After Finished: 3600
Start Time: Sun, 01 Nov 2026 14:13:35 +0000
Completed At: Sun, 01 Nov 2026 14:13:38 +0000
Duration: 3s
Active Deadline Seconds: 15s
Pods Statuses: 0 Active (0 Ready) / 1 Succeeded / 0 Failed
Pod Template:
Labels: batch.kubernetes.io/controller-uid=bf554134-718e-4f81-ac62-e7d36eff6f0c
batch.kubernetes.io/job-name=payment-pubsub-test
controller-uid=bf554134-718e-4f81-ac62-e7d36eff6f0c
job-name=payment-pubsub-test
Containers:
pubsub-test:
Image: natsio/nats-box:0.20.1
Port: <none>
Host Port: <none>
Command:
/bin/sh
-c
Args:
echo "sending payment request ORD-1001"
nats pub -s nats://nats.platform-nats-server.svc.cluster.local:4222 payments.requests '{"order_id":"ORD-1001","amount":49.99}'
Environment: <none>
Mounts: <none>
Volumes: <none>
Node-Selectors: <none>
Tolerations: <none>
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal SuccessfulCreate 3s job-controller Created pod: payment-pubsub-test-knzx5
Normal Completed 0s job-controller Job completed
The highlighted lines show every setting from the task except the restart policy. describe does not list it, so read it from the Pod template:
kubectl get job payment-pubsub-test -n checkout -o jsonpath='{.spec.template.spec.restartPolicy}'
Never
ttlSecondsAfterFinished: 3600 deletes the Job and its Pod 1 hour after it finishes. Read the Job logs before that, or they are gone with the Pod.
Summary
- Find the cause of the failure: the Pod logs show
lookup nats on 10.43.0.10:53: no such host. A short Service name only resolves from Pods in the same namespace as the Service. - Reach NATS from
checkout:-s nats://nats.platform-nats-server.svc.cluster.local:4222, with the subject and the message unchanged. - Recreate the Job: a Job's Pod template cannot be changed, so delete it and apply the fixed manifest.
- One Pod, one at a time:
completions: 1,parallelism: 1. - At most one retry:
backoffLimit: 1withrestartPolicy: Never. - 15 seconds in total:
activeDeadlineSeconds: 15. - Clean up after 1 hour:
ttlSecondsAfterFinished: 3600. - Proof that pub/sub works: the payment log shows
received payment request: {"order_id":"ORD-1001","amount":49.99}. The Job log alone only proves that the message was published.