Network, Web, and Firewall Inspection

🎯 Learning Objective

By the end of this unit, you'll understand how to use curl to monitor API health, authenticate with different services, and debug connectivity issues - essential skills for maintaining production services and troubleshooting when things go wrong.

📚 Concept Introduction

You already know basic curl <url> for fetching web pages, but production API troubleshooting requires much more. You need to test authentication, measure response times, check different HTTP methods, and debug exactly what's happening at the network level.

Every DevOps engineer works with APIs daily: they're the nervous system of modern applications. When they break, you need advanced curl techniques to diagnose problems quickly and understand what's actually happening during failures.

📁 Pre-created:

• api-tests/ - Your workspace for API testing experiments
• endpoints.txt - Sample endpoints for health checking practice
• auth-tokens.txt - Authentication examples for different methods

🚀 Beyond Simple GET Requests

Most people think curl means "download a webpage." But APIs speak multiple languages, and GET is just one of them.

◆ Testing All the Methods That Matter

Let's look at what endpoints we'll be working with:

cd api-tests && cat endpoints.txt

Real APIs handle different operations through different HTTP methods. Here's how to test them properly:

curl -X GET http://httpbin.org/get

This explicitly tests GET requests - the "read" operation. You'll get back JSON showing exactly what the server received, perfect for confirming your request worked.

curl -X POST http://httpbin.org/post

POST requests create new resources. Even without sending data, this tests whether the endpoint accepts creation requests. In production, you'd use this to test user registration, order creation, or data submission endpoints.

curl -X PUT http://httpbin.org/put

PUT handles updates - modifying existing users, changing configurations, updating records. Testing this ensures your API can handle the full lifecycle of data management.

curl -X DELETE http://httpbin.org/delete

DELETE removes resources. Critical for testing data cleanup, user deletion, or content removal workflows.

Why test all methods? Because a GET might work fine while POST is completely broken. Different methods often hit different code paths, different databases, different security checks.

🔒 The Authentication Challenge

Here's a harsh reality: almost every production API requires authentication. Your basic curl commands will hit authentication errors constantly, making them useless for real troubleshooting.

◆ Modern API Authentication

Most modern APIs use bearer tokens - think JWT tokens, API keys, or OAuth tokens:

curl -H "Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.sample-token" http://httpbin.org/bearer

The -H flag adds custom headers. This specific pattern (Authorization: Bearer <token>) is everywhere - Slack APIs, GitHub APIs, cloud services, microservices. Master this and you can test most modern systems.

◆ API Key Headers

Many services use custom header names for API keys:

curl -H "X-API-Key: abc123def456ghi789" http://httpbin.org/headers

Different services use different header names: X-API-Key, Api-Key, X-Auth-Token, X-RapidAPI-Key. The pattern is the same, but you need to know what each service expects.

◆ Legacy Basic Authentication

Older systems and internal tools often use basic authentication:

curl -u test:test http://httpbin.org/basic-auth/test/test

The -u username:password flag handles the Base64 encoding automatically. Much cleaner than manually constructing authentication headers.

🩺 Diagnosing API Health

When APIs are slow, broken, or behaving strangely, you need diagnostic information beyond just "it worked" or "it failed."

◆ Status Code Reality Check

APIs communicate through status codes, but you need to capture and understand them:

curl http://httpbin.org/status/200

This always returns success. But in production, you need to capture those status codes:

curl -w "%{http_code}" http://httpbin.org/status/200

The -w "%{http_code}" extraction gives you just the status code - perfect for automated health checks and monitoring scripts.

◆ Testing Failure Scenarios

Don't just test the happy path. Production breaks in predictable ways:

curl -w "%{http_code}" http://httpbin.org/status/404

This simulates "not found" errors. How does your monitoring handle missing resources?

curl -w "%{http_code}" http://httpbin.org/status/500

This simulates server errors. When databases are overloaded or services crash, they return 500. Your monitoring needs to detect and alert on these.

◆ Performance Monitoring

Sometimes APIs work but are painfully slow. Users complain, but you need data:

curl -w "%{time_total}" http://httpbin.org/delay/2

The -w "%{time_total}" shows response time in seconds. This endpoint artificially delays for 2 seconds - perfect for testing your performance monitoring.

You can combine multiple metrics:

curl -w "Status: %{http_code}, Time: %{time_total}s\n" http://httpbin.org/get

Now you get both status and timing in one request. Essential for comprehensive API monitoring.

🔍 When Things Go Wrong

APIs fail in mysterious ways. Networks are flaky. SSL certificates expire. Firewalls block requests. When debugging, you need visibility into what's actually happening.

◆ Verbose Debugging

When a request fails, -v is your best friend:

curl -v http://httpbin.org/get

This shows everything: DNS lookups, SSL handshakes, request headers, response headers, connection details. It's like having X-ray vision into the HTTP transaction.

◆ Quick Health Checks

Sometimes you just need to know "is this service responding?" without downloading content:

curl -I http://httpbin.org/get

The -I flag sends a HEAD request - you get headers but no body content. Faster and lighter for quick health checks.

◆ Timeout Protection

Networks are unreliable. Services hang. You need protection from waiting forever:

curl --connect-timeout 5 --max-time 10 http://httpbin.org/delay/12

--connect-timeout 5 means "give up connecting after 5 seconds." --max-time 10 means "give up entirely after 10 seconds total." This request will timeout because the endpoint delays for 12 seconds.

curl -f http://httpbin.org/status/404

The -f flag makes curl exit with an error code for HTTP errors. Perfect for scripts where you want to handle failures programmatically.

🛡️ Authentication in the Real World

Let's test authentication with a protected endpoint that actually requires a token:

curl http://httpbin.org/bearer

This endpoint requires authentication and will return an error without proper credentials. Now let's fix that:

💡 Real-World Applications

These techniques solve actual production problems:

• Incident response: Use -w "%{http_code}" to quickly check if services are returning errors
• Performance debugging: Use -w "%{time_total}" to identify slow endpoints during outages
• Authentication testing: Use -H "Authorization: Bearer <token>" to validate API access after deployments
• Network debugging: Use -v to diagnose SSL issues, DNS problems, or connection failures
• Health monitoring: Use -I for lightweight service health checks in monitoring scripts

The difference between knowing basic curl and these advanced techniques is the difference between being helpless during outages and being the person who fixes them quickly.

In the next unit, we'll explore advanced wget techniques for robust file downloads and mirroring - complementing your API testing skills with solid file management capabilities.

🎯 Learning Objective

By the end of this unit, you'll understand how to download files reliably when networks are unstable, handle authentication for protected resources, and automate bulk downloads - essential skills for managing software deployments and maintaining local mirrors in production.

📚 Concept Introduction

You already know basic wget <url> for downloading files, but production environments require much more reliable techniques. You need to handle network interruptions, authenticate with protected resources, download multiple files efficiently, and control bandwidth usage.

In production environments, reliable downloads aren't optional - they're critical. Whether you're fetching large database backups, synchronizing documentation, or downloading software packages, wget has powerful features that go far beyond basic downloading.

📁 Pre-created:

• downloads/ - Your workspace for testing download techniques
• download-urls.txt - A list of sample URLs for batch processing experiments

📥 When Downloads Go Wrong

Network interruptions happen. Servers get overloaded. Connections timeout. Basic wget usage leaves you vulnerable to all of these problems.

◆ Resume Interrupted Downloads

The most frustrating thing? Losing hours of download progress because your connection hiccupped. The -c flag solves this:

wget -c http://httpbin.org/delay/20

If this download gets interrupted (try pressing Ctrl+C), running the exact same command again will pick up where it left off. No wasted time, no duplicate data transfer.

◆ Handling Unreliable Networks

Some networks are just flaky. Maybe you're downloading over a mobile connection, or the server is under heavy load. wget can retry automatically:

wget --tries=5 --timeout=30 http://httpbin.org/delay/3

This says "try up to 5 times, and don't wait more than 30 seconds for any single response." It's much more robust than hoping your connection stays perfect.

For really problematic connections, you can even make wget wait between retries:

wget --retry-connrefused --waitretry=10 http://httpbin.org/delay/2

This waits 10 seconds between retry attempts, giving overwhelmed servers time to recover.

🌐 Smart Downloading

Sometimes you need more than just one file. Maybe you want to mirror documentation, download multiple packages, or grab only specific file types from a repository.

◆ Recursive Downloads with Limits

The -r flag downloads not just the file you specify, but also any files it links to. This can quickly get out of hand on modern websites, so you typically want to limit it:

wget -r -l 2 http://httpbin.org/

The -l 2 limits recursion to 2 levels deep. This prevents wget from downloading the entire internet when you just want a small section of a site.

◆ Filtering What You Download

Real repositories contain lots of files you don't need. The -A (accept) and -R (reject) flags let you be selective:

wget -r -A "*.pdf,*.doc" https://example.com/documents/

This downloads only PDF and DOC files, ignoring HTML pages, images, and everything else.

Conversely, you might want everything except certain file types:

wget -r -R "*.html,*.css" http://httpbin.org/

This grabs everything except HTML and CSS files - useful when you want the data but not the web presentation.

🔐 Protected Resources

Many production resources require authentication. Basic wget can't handle this, but the advanced features can.

◆ Basic Authentication

For resources protected with HTTP basic auth:

wget --user=demo --password=demo http://httpbin.org/basic-auth/demo/demo

This handles the authentication header automatically. Much cleaner than trying to construct the header yourself.

◆ Custom Headers

For modern APIs that use bearer tokens or custom auth headers:

wget --header="Authorization: Bearer your-token-here" http://httpbin.org/bearer

This adds any custom header you need. Essential for accessing protected APIs or services that require specific authentication.

🤖 Automating Downloads

Manual downloads don't scale. When you need to download dozens of files, or run downloads as part of automated scripts, wget has features specifically designed for this.

◆ Batch Processing

Rather than running wget dozens of times, you can give it a list of URLs:

cat download-urls.txt

This shows you the URLs we'll process. Then:

wget -i url-list.txt --wait=5

The -i flag reads URLs from a file, while --wait=5 adds a 5-second delay between downloads. This prevents you from overwhelming the server with rapid-fire requests.

◆ Background and Quiet Operations

For long-running downloads that you don't need to babysit:

wget -b http://httpbin.org/delay/10

The -b flag runs the download in the background, freeing up your terminal for other work.

For script automation where you only want to see errors:

wget -q -O result.json http://httpbin.org/json

The -q flag suppresses normal output, making your scripts cleaner.

💡 Real-World Applications

These techniques solve actual production problems:

• Database backups: Use -c to resume large backup downloads that get interrupted
• Software deployment: Use --tries and --timeout to handle unreliable networks during package downloads
• Documentation mirrors: Use -r with -A filtering to create local copies of documentation sites
• API integration: Use --header to download configuration files from authenticated endpoints
• Bulk processing: Use -i to download hundreds of files from a manifest list

The difference between basic wget and these advanced techniques is the difference between crossing your fingers and having confidence your downloads will complete successfully.

In the next unit, we'll explore advanced dig techniques for DNS troubleshooting, giving you tools to debug connectivity issues that might be affecting your downloads.

🎯 Learning Objective

By the end of this unit, you'll be able to use dig for systematic DNS troubleshooting, understand DNS record types, and investigate DNS-related connectivity issues that commonly break DevOps deployments.

📚 Concept Introduction

You know basic network troubleshooting with curl and wget, but when connections fail, the problem often lies deeper: DNS resolution issues that prevent your tools from even reaching the target servers.

Every DevOps engineer encounters this reality: applications fail with "connection timeout" while your monitoring shows everything is running. DNS issues hide behind generic error messages, and random troubleshooting wastes hours. You need techniques that methodically isolate DNS problems from application problems.

📁 Pre-created:

• domains.txt - List of test domains for DNS investigation
• expected_ips.txt - Expected IP addresses for domain verification

🔍 Understanding DNS Record Types

DNS failures manifest differently depending on the record type affected. Knowing what each record does helps target your investigation efficiently.

◆ Core Record Types

A records map domain names to IPv4 addresses - when these fail, nothing connects. Quick dig yourapp.com reveals if the problem is DNS resolution or application code.

MX records control email delivery. When alerting systems stop sending notifications, MX record investigation often reveals misconfigured mail routing priorities.

CNAME records create domain aliases used heavily in CDN integration and load balancing. When performance suddenly degrades, CNAME chains might be misconfigured.

NS records define authoritative nameservers. When entire domains become unreachable, NS record problems control which servers have authoritative DNS information.

🚀 Basic dig Usage

The most common DNS investigation starts with basic domain resolution:

dig google.com

This produces a detailed output that looks intimidating at first, but let's break it down:

; <<>> DiG 9.18.30-0ubuntu0.24.04.2-Ubuntu <<>> google.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 9969
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;google.com.                    IN      A

;; ANSWER SECTION:
google.com.             300     IN      A       142.250.67.174

;; Query time: 3 msec
;; SERVER: 148.113.10.111#53(148.113.10.111) (UDP)
;; MSG SIZE  rcvd: 55

Key sections explained:

Header Information: Shows the dig version and what you queried

• status: NOERROR means the query succeeded
• flags: qr rd ra shows query type and recursive resolution

QUESTION SECTION: Shows exactly what you asked for

• google.com. IN A means "What's the A record for google.com?"

ANSWER SECTION: The actual DNS data you need

• google.com. 300 IN A 142.250.67.174 means:
  • Domain: google.com
  • TTL: 300 seconds (how long to cache this)
  • Record type: A (IPv4 address)
  • Value: 142.250.67.174 (the actual IP address)

Statistics: Query performance and server info

• Query time: 3 msec - how fast the lookup was
• SERVER: 148.113.10.111#53 - which DNS server answered

For troubleshooting, you mainly care about the ANSWER SECTION and the status in the header.

◆ Getting Clean Output

Production troubleshooting requires rapid verification:

dig +short google.com

The +short option returns just the IP address - perfect for scripts, monitoring, and rapid problem verification during incidents.

◆ Querying MX Records

Email delivery problems require different investigation than web connectivity:

dig google.com MX

MX record output shows mail server information. Here's what you'll see:

; <<>> DiG 9.18.30-0ubuntu0.24.04.2-Ubuntu <<>> google.com MX
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 1382

;; QUESTION SECTION:
;google.com.                    IN      MX

;; ANSWER SECTION:
google.com.             300     IN      MX      10 smtp.google.com.

Key parts of MX records in the ANSWER SECTION:

• Domain: google.com.
• TTL: 300 seconds
• Record type: MX (mail exchange)
• Priority: 10 (lower numbers = higher priority)
• Mail server: smtp.google.com.

For domains with multiple mail servers, you'll see multiple MX records with different priorities.

◆ Querying NS Records

When you need to find which nameservers are authoritative for a domain:

dig google.com NS

NS record output shows nameserver information:

;; ANSWER SECTION:
google.com.             21600   IN      NS      ns1.google.com.
google.com.             21600   IN      NS      ns2.google.com.
google.com.             21600   IN      NS      ns3.google.com.

Key parts of NS records:

• Domain: google.com.
• TTL: 21600 seconds
• Record type: NS (nameserver)
• Nameserver: ns1.google.com., ns2.google.com., etc.

These are the authoritative servers that have the official DNS records for the domain.

🔧 DNS Troubleshooting Techniques

Systematic DNS investigation prevents fixing symptoms instead of root causes.

◆ Testing Different Nameservers

Is the problem local DNS configuration or the domain itself?

dig @8.8.8.8 example.com

This queries Google's public DNS directly, bypassing local DNS resolver. If this works but normal dig fails, the problem is local DNS configuration.

dig @1.1.1.1 example.com

Cloudflare's DNS provides comparison. Consistent results across multiple nameservers confirm DNS records are correct - investigate elsewhere.

◆ Investigating Domain Authority

When DNS resolution fails completely, check nameserver configuration:

dig example.com NS

This reveals which servers should have authoritative answers. If NS records point to non-existent or misconfigured servers, that's your root cause.

📋 Practical DNS Investigation

Real-world DNS problems require methodical investigation. Random fixes waste time and create new problems.

📧 Email Troubleshooting with MX Records

Email delivery failures are particularly problematic because they're often silent - messages disappear without obvious errors.

◆ MX Record Priority System

Email systems use priority numbers to determine mail server precedence. Lower numbers mean higher priority:

dig gmail.com MX

This shows all mail servers and their priorities. You'll see output like:

gmail.com.      3600    IN      MX      5 gmail-smtp-in.l.google.com.
gmail.com.      3600    IN      MX      10 alt1.gmail-smtp-in.l.google.com.
gmail.com.      3600    IN      MX      20 alt2.gmail-smtp-in.l.google.com.

The server with priority 5 (gmail-smtp-in.l.google.com) is the primary mail server since it has the lowest number. If it's unreachable, mail flows to the backup servers with higher numbers.

💡 Key Takeaways

• Systematic DNS investigation with dig transforms guesswork into methodical problem-solving
• The +short option provides clean, scriptable output essential for automation and rapid troubleshooting
• Different record types (A, MX, NS, CNAME) serve specific purposes - knowing which to check saves investigation time
• Testing multiple nameservers with @server quickly isolates local vs. upstream DNS issues
• MX record priorities determine email routing - understanding this prevents email delivery failures
• Professional DNS investigation prevents fixing symptoms while missing root causes

In the next unit, we'll explore Linux firewall inspection with iptables and nftables - completing your network troubleshooting toolkit for production environments.

🎯 Learning Objective

By the end of this unit, you'll be able to use iptables to view and interpret firewall rules, identify blocked ports, and systematically troubleshoot connectivity issues caused by firewall configurations.

📚 Concept Introduction

You know network troubleshooting with curl, wget, and dig, but when all DNS resolution works perfectly and services are running, connections still fail. Often the culprit is firewall rules silently blocking traffic.

Firewall rules protect servers but can also block legitimate traffic when misconfigured. You need techniques to quickly identify whether connection failures are application problems or firewall restrictions.

In this environment, we've set up real iptables rules that you can inspect directly using the iptables command. This gives you practical experience with live firewall configurations.

🔍 Understanding iptables Basics

Firewalls operate by examining network packets and making decisions: allow or block. The iptables command shows you these decision rules, helping you understand why connections succeed or fail.

◆ Core iptables Concepts

Chains control different traffic directions:

• INPUT: Traffic coming into the server (most common for troubleshooting)
• OUTPUT: Traffic leaving the server
• FORWARD: Traffic passing through the server (routing scenarios)

Targets define what happens to matching packets:

• ACCEPT: Allow the traffic through
• DROP: Silently ignore the traffic (connection timeout)
• REJECT: Block traffic and send error response

Default Policy determines what happens to traffic that doesn't match any specific rules - either ACCEPT (permissive) or DROP (restrictive).

🚀 Essential iptables Commands

The most common firewall investigation starts with viewing all rules:

sudo iptables -L

This produces detailed output showing all firewall chains and rules. Let's understand what you'll see:

Chain INPUT (policy DROP)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:ssh
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:http
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:https
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:8080
ACCEPT     icmp --  anywhere             anywhere             icmp type 8

Chain FORWARD (policy DROP)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

Key parts explained:

Chain Header: Shows chain name and default policy

• Chain INPUT (policy DROP) means if no rules match, traffic is blocked

Rule Structure: Each rule has specific components

• target: What to do (ACCEPT, DROP, REJECT)
• prot: Protocol (tcp, udp, icmp, all)
• source: Where traffic comes from (IP addresses or "anywhere")
• destination: Where traffic goes to (IP addresses or "anywhere")
• Additional info: Port numbers (tcp dpt:ssh means TCP port 22, tcp dpt:8080 means TCP port 8080)

◆ Getting Numeric Output

sudo iptables -L -n

The -n flag shows IP addresses and port numbers instead of resolving hostnames:

Chain INPUT (policy DROP)
target     prot opt source               destination         
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:22
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:80
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:443
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:8080
ACCEPT     icmp --  0.0.0.0/0            0.0.0.0/0            icmptype 8

◆ Verbose Output for Troubleshooting

When you need to see how much traffic hits each rule:

sudo iptables -L -v

This shows packet and byte counters for each rule, helping identify which rules are actually being used:

Chain INPUT (policy DROP 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
  15K 1140K ACCEPT     all  --  any    any     anywhere             anywhere             ctstate RELATED,ESTABLISHED
   45  2700 ACCEPT     tcp  --  any    any     anywhere             anywhere             tcp dpt:ssh
   12   720 ACCEPT     tcp  --  any    any     anywhere             anywhere             tcp dpt:http
    0     0 ACCEPT     tcp  --  any    any     anywhere             anywhere             tcp dpt:https
    0     0 ACCEPT     tcp  --  any    any     anywhere             anywhere             tcp dpt:8080
    5   300 ACCEPT     icmp --  any    any     anywhere             anywhere             icmptype 8

The pkts column shows how many packets matched each rule. Zero packets on some ACCEPT rules means those services haven't received connections yet.

🔧 Reading Firewall Rules

Understanding rule order is critical - iptables processes rules from top to bottom and stops at the first match.

◆ Rule Order Matters

Look at the current firewall rules on this server:

sudo iptables -L -n

In this configuration, you'll see SSH (port 22) has an ACCEPT rule before the default DROP policy. This is why SSH connections work. If there were no explicit ACCEPT rule for SSH, connections would fail due to the default DROP policy.

◆ Understanding Default Policies

Check the policy at the top of each chain:

sudo iptables -L | grep "policy"

• policy ACCEPT: If no rules match, allow traffic (permissive)
• policy DROP: If no rules match, block traffic (restrictive)

This server uses a restrictive policy DROP for INPUT and FORWARD chains, which means you need explicit ACCEPT rules for every service that should work. This is a security best practice - block everything by default, then explicitly allow only what's needed.

📋 Practical Firewall Troubleshooting

Real-world connectivity problems require systematic firewall analysis combined with service verification.

🔍 Service Port Investigation

When specific services fail to receive connections, targeted port analysis reveals whether firewall rules are the cause.

◆ Combining Service and Firewall Analysis

First, verify what ports your services expect by checking the service configuration:

cat /home/laborant/firewall-investigation/service-config.txt

Then check if those ports are allowed through the firewall:

sudo iptables -L -n | grep "tcp dpt:8080"

If you see no ACCEPT rules for port 8080, but your application expects traffic on that port, you've found the problem.

💡 Key Takeaways

• iptables shows firewall rules that can silently block network connections even when services are running
• The -n flag provides numeric output essential for fast production troubleshooting without DNS delays
• Rule order matters - iptables processes rules top to bottom and stops at first match
• Default policies (ACCEPT vs DROP) determine behavior for traffic that doesn't match any specific rules
• Systematic port analysis combining service configuration and firewall rules quickly identifies connectivity issues
• Verbose output with -v shows packet counters revealing which rules are actively blocking traffic

In this lesson, you've mastered the complete network troubleshooting toolkit: advanced curl and wget for testing connections, dig for DNS investigation, and iptables for firewall analysis - essential skills for diagnosing any connectivity problem in production environments.