Debugging Scenarios

The toolbox is the same, but where to start depends on what exactly is broken. Find your situation in the units of this lesson.

The play is up, the terminals work, but a service isn't listening, a command fails, a container won't start. Nothing playground-specific here - it's a Linux box, and all the usual debugging techniques apply:

systemctl status <unit>; journalctl -u <unit> -n 100    # is the service running? what did it say?
ps aux | grep <name>; top                                # is the process alive? is something eating the CPU?
ss -ltnp                                                 # who is listening on which port (and on which address)?
curl -v http://localhost:<port>/                         # does it answer locally? (before blaming port exposure or networking)
dmesg | tail; df -h; free -m                             # kernel complaints, full disks, memory pressure

A few playground-flavored hints:

• Make sure you're on the right machine: labctl ssh <play-id> -m <machine>, or check the machine name in the terminal tab / shell prompt.
• The machines are sized by the manifest (resources), and the defaults are modest - a service that works on a beefy laptop may be OOM-killed in a 2-4 GB VM.
• If the service was supposed to be set up by an init task, verify the task actually did what you think - examinerctl task get <name> or the Tasks tab (see the next scenario).
• To watch a unit's logs without occupying a terminal, stream them from outside: labctl playground machine journal <play-id> <machine> -u <unit>.

Symptoms: the "Warming up playground..." screen shows Init tasks completed: N/M and never reaches M/M, a task badge turns red, or the environment lacks something the task was supposed to install.

1. Look at the tasks' statuses and output. Open the Tasks tab of the Play Debug Console and click the failing (or the never-completing) task - its exit code and stdout/stderr almost always explain the problem. From the command line: labctl playground tasks <play-id> -o yaml (or --wait --fail-fast to follow the tasks live), and from inside the VM: examinerctl task list / examinerctl task get <name>.
2. To get a shell while the tasks are still running, close the loading screen with the X in its top-right corner - the machines are already booted. With labctl, start the play with --skip-wait-init (or just labctl ssh it from another terminal).
3. Reproduce the failing step interactively: copy the run script from the task details, run it as the same user (tasks run as root unless the manifest says otherwise), and iterate.
4. Fix the manifest, labctl playground update, and start a new play - init tasks execute once per play instance and are not re-run on restarts or reboots.

Common culprits: the default timeout_seconds of 60 (too short for apt-get, pip install, docker pull); a missing needs dependency (the task runs before another task has provided a file or started a service); running as root while preparing files for laborant (or the other way around); relying on shell profiles that only load in login shells.

Symptoms: the "Booting Playground" screen ("Waiting for playground to become ready...") never goes away, machines linger in STARTING, or the play ends up on the "It's gone..." page with "The play failed to start".

1. Open the Play Debug Console (from the context menu of the starting play, or right on the failed page) and check the Play Spec. Look at status.stateEvents - the sequence of play states with timestamps (an event flagged with error marks where the platform gave up) - and at the conditions: status.conditions for the play as a whole and status.machines[].conditions for every machine. Conditions flip to True as the boot progresses (drive sources pulled, volumes created, rootfs baked, startup files written, users resolved, guest networking configured, sandbox started, ...), so the first condition that is still False/Unknown - together with its message - tells you at which stage the boot is stuck or failed. For example, a MachineUsersResolved: False points at a users entry that doesn't exist in the rootfs image, and a DriveSourcePulled that stays False for a long time means a big OCI image is still being pulled.
2. If the platform-side conditions are all True but the machine still doesn't become ready, the problem is inside the guest - look at the Boot Logs tab (or run labctl playground machine console <play-id> <machine>). The serial console shows the kernel boot, the init system's unit start-up, and where it stops: a kernel panic, a missing /sbin/init, a failed root mount, systemd dropping into emergency mode, a unit hanging with "A start job is running for ...", or an sshd that never starts. This is the typical case for custom rootfs images built from a plain distro base - no init system, no sshd, missing system files.
3. From the terminal, labctl playground status <play-id> and labctl playground machines <play-id> show the machine states and readiness at a glance (labctl playground start itself doesn't return until the machines are RUNNING and ready - unless told otherwise with --skip-wait-running/--skip-wait-ready).

Symptoms: the terminal tab shows "Connecting..." forever, the IDE tab doesn't load, labctl ssh hangs.

1. First things first - exclude local connectivity issues. If the web terminal isn't connecting, try labctl ssh <play-id> from your machine; if labctl ssh fails, open the play page and try the web terminal. If one of them works, the play is fine, and the problem is on the path: a corporate proxy or firewall dropping WebSocket connections, a browser extension, a VPN, a flaky network. The connection status dot in the play page header helps too: green means the browser holds a live connection to the play, orange - it's still connecting, red - the play is gone.
2. Then assess the play's state in the Play Debug Console (Play Spec): is running true, are all machines in the RUNNING state, are the boot conditions True, has the play perhaps already expired (expiresIn)? The same from the command line: labctl playground status <play-id> (machines are listed as name=STATE (ready)) and labctl playground machines <play-id>.
3. Finally, resort to the per-machine commands. labctl playground machine console <play-id> <machine> shows whether the guest booted and got to the login prompt; labctl playground machine journal <play-id> <machine> -u ssh (-u sshd on non-Debian distros) shows what the SSH daemon is doing; a machine in the STOPPED state can be brought back with labctl playground machine restart, and a wedged one bounced with labctl playground machine reboot.

Also keep in mind that a machine with noSSH: true in the manifest has no shell access by design - neither terminals nor labctl ssh will work for it (the Play Spec shows the flag).

Symptoms: everything worked for a while, then the terminals froze, new SSH sessions hang, the services stopped answering.

1. Confirm the play hasn't simply run out of time. Every play has a lifetime (the countdown in the header; labctl playground lifetime <play-id>) and an idle timeout, and an expired play is stopped (if persistent) or destroyed. labctl playground list and labctl playground status <play-id> show the current state; the Play Spec has expiresIn and maxIdleTime. If it's still running, read on.
2. Check what the machines are up to: labctl playground machines <play-id> for the states, then labctl playground machine journal <play-id> <machine> (or --since -10m) for the systemd journal around the time things went south - the OOM killer ("Out of memory: Killed process ..."), a full disk ("No space left on device"), a crash-looping unit, and runaway processes all leave traces there. labctl playground machine console <play-id> <machine> catches what the journal can't - a kernel panic or oops is printed to the serial console. If the machine still accepts commands, labctl ssh <play-id> -m <machine> -- uptime and free -m/df -h give the quick picture.
3. Recover: labctl playground machine reboot <play-id> <machine> restarts a wedged guest with its disks intact (init tasks are not re-run); a machine that shut itself down (STOPPED) comes back with labctl playground machine restart. If the play is persistent, labctl playground stop + labctl playground restart is the heavier hammer that recreates the whole play from its saved state.

Resource exhaustion is the usual reason for this scenario - the machines have as much RAM and disk as the manifest gives them, and a docker build, a database import, or a memory leak eventually hits the ceiling. Bump resources in the manifest or bake heavy artifacts into a custom rootfs image.

Symptoms: the play page says "It's gone...", or the play simply isn't in labctl playground list anymore.

1. labctl playground list -a lists all your recent plays including the stopped and the recently terminated ones - find the play by its ID or title, and look at the CREATED and STATUS columns. labctl playground status <play-id> works for a gone play too and adds the page URL.
2. Compare the play's age with its limits. Ephemeral plays are destroyed when their lifetime runs out or after being idle for too long (both limits are in the Play Spec - maxPlayTime/expiresIn and maxIdleTime - and depend on the playground and your tier); persistent plays are stopped instead. A play that is much younger than its lifetime didn't expire - it failed.
3. Open the play page (/playgrounds/<playground-name>/<play-id>, or the Page URL from labctl playground status): the "It's gone..." page shows the Play Debug Console, and the stateEvents and the Boot Logs are the post-mortem - "The play failed while running" means the platform lost the machines mid-flight (the serial console of the machine usually shows why), "The play failed to start" sends you back to the boot scenario. The boot logs stay readable for a while after the play is gone, from the UI and via labctl playground machine console.

To not lose work in the future, make the play persistent (labctl playground persist <play-id>, or the persistence toggle in the play page header) - a persistent play is stopped rather than destroyed when its time is up, and can be restarted later.