Challenge
DifficultyMedium
Learner activity27 solved of 43 attempts

Use a Kubernetes Job to Test NATS Pub/Sub

A payment service subscribes to payment requests through NATS. Run the checkout team's test script as a Job in the checkout namespace, with the required settings, and make sure the payment request arrives.

Scenario

NATS is a messaging system. Applications use it to send messages to each other without connecting to each other directly. It works with publish/subscribe:

  • A publisher sends a message to a subject, a name such as payments.requests.
  • A subscriber listens on a subject and receives every message sent to it.
  • The NATS server sits between them. Clients connect to it on port 4222, and it delivers each message to every subscriber of the subject.

A NATS server already runs in this cluster.

More about NATS:

Note

NATS is not NAT. NAT (Network Address Translation) is a networking feature that rewrites IP addresses. NATS is a message broker, in the same family as Apache Kafka or RabbitMQ.

In the payment-service namespace, the payment-processor subscribes to payments.requests and prints every message it receives in its logs. The checkout namespace is empty. This is where you create the test Job.

The checkout team wants to prove that a Job in checkout can publish a payment request that the payment service receives. The Job is the publisher, and the payment-processor is the subscriber. The team wrote a test script for the Job.

Note

Core NATS does not store messages. Only subscribers that are connected when a message is published receive it, so a successful publish alone does not prove delivery. The payment service logs do.


Task

Create a Job in checkout that publishes a test payment request, so you can check that it reaches the payment service. The Job must have these settings:

SettingValue
Namepayment-pubsub-test
Namespacecheckout
Imagenatsio/nats-box:0.20.1
Container command and argsThe script below
Successful Pods needed1, one Pod at a time
Retries of a failed PodAt most 1
Time limit for the whole Job15 seconds
A failed containerIs not restarted inside the same Pod
Clean upThe Job is deleted automatically 1 hour after it finishes

A member of the checkout team prepared this command and args for the Job container.

Use the supplied command and args unchanged for the first run:

command: ["/bin/sh", "-c"]
args:
- |
  echo "sending payment request ORD-1001"
  nats pub -s nats://nats:4222 payments.requests '{"order_id":"ORD-1001","amount":49.99}'

Check the Job's status, events, and Pod logs to find out whether the payment request was published. If the Job fails, find the cause, delete the Job, and create it again with the necessary correction.

Keep the subject payments.requests and the message unchanged.

The task is complete when the Job succeeds and the payment-processor logs show that the ORD-1001 payment request was received.

Important

Do not change anything outside the checkout namespace. If you need to change the Job, delete it and create it again.


Hint 1: How the script talks to NATS

The script is a NATS client. nats pub connects to the NATS server, publishes one message to a subject, and exits:

nats pub -s <server-url> <subject> <message>
  • -s is the address of the NATS server, in the form nats://<host>:<port>.
  • <subject> is the name that subscribers listen on.
  • <message> is the body of the message.

If nats pub cannot reach the server, it prints an error and exits with a non-zero code, so the Pod fails.

Docs:

Hint 2: Building the Job

The script has two lines, so a shell must run it. A container can override the start command of its image. Start from a generated Job skeleton:

kubectl create job <name> -n <namespace> --image=<image> --dry-run=client -o yaml
kubectl explain job.spec.template.spec.containers.command
kubectl explain job.spec.template.spec.containers.args

Every row of the settings table is one field. Most of them are in the Job spec. Whether a failed container restarts is set in the Pod template:

kubectl explain job.spec
kubectl explain job.spec.template.spec.restartPolicy

Docs:

Hint 3: Checking the Job

Check whether the Job created its Pods, and whether it succeeded or failed:

kubectl get job,pods -n <namespace>
kubectl describe job <job-name> -n <namespace>

The events at the end of describe show what the Job controller did. A Job that ran out of retries or time shows BackoffLimitExceeded or DeadlineExceeded. These reasons only say that the Pods failed. The Pods themselves show why.

Docs:

Hint 4: Reading both sides

The Pods of the Job are the publisher. Their logs show what the script printed, including any error from nats pub:

kubectl logs -n <namespace> -l job-name=<job-name>

The subscriber logs show whether a message arrived. Keep them open in a second terminal while you test:

kubectl logs -n <namespace> deployment/<name> -f

Docs:

Hint 5: Reading the error

An error such as dial tcp: lookup <host> on <ip>:53 means the client never reached the server. Before it opens a TCP connection, it asks the DNS server at <ip>, port 53, for the IP address of <host>. That lookup failed.

Repeat the lookup from a test Pod in the same namespace as the Job:

kubectl run <pod-name> -n <namespace> --image=<image> --restart=Never --command -- sleep 3600
kubectl exec <pod-name> -n <namespace> -- nslookup <host>

Docs:

Hint 6: How Services get names

Every Service gets DNS names that include its namespace:

<service>.<namespace>
<service>.<namespace>.svc
<service>.<namespace>.svc.cluster.local

Find the Service that the script connects to, and its namespace:

kubectl get service -A

Docs:

Hint 7: How a short name is completed

A short name such as <service> is completed with the search domains in the Pod's /etc/resolv.conf. The first search domain is the Pod's own namespace:

kubectl exec <pod-name> -n <namespace> -- cat /etc/resolv.conf

Compare that namespace with the namespace of the Service. A Job's Pod template cannot be changed, so delete the Job and create it again with your correction.

Docs:


âš’ Test Cases

Problem
Solution
Start
Black Friday 2026