Downloading Files from the Web

🎯 Learning Objective

By the end of this unit, you'll master downloading files and web content using curl and wget, essential skills for automating deployments, retrieving configuration files, and managing software installations in DevOps environments.

📚 Concept Introduction

File downloading is fundamental to DevOps operations, enabling automated software installations, configuration retrieval, and deployment artifact management. Production environments rely on command-line downloading for CI/CD pipelines, infrastructure provisioning, and automated system updates.

Understanding download tools enables you to automate software installations, retrieve remote configurations, download deployment artifacts, and build reliable automation scripts that work in headless server environments without graphical interfaces.

📁 Pre-created:

• Ubuntu 24.04 system with internet connectivity
• Command-line access for testing download operations
• Network access to external resources for practice

🌐 Understanding curl for Data Transfer

◆ What is curl?

curl (Client URL Request Library) transfers data using various protocols including HTTP, HTTPS, and FTP. It's designed for flexibility and can handle complex web interactions beyond simple file downloads.

Basic curl syntax:

curl [options] URL

◆ Basic curl Usage

Display content in terminal:

curl http://info.cern.ch/

By default, curl displays the downloaded content directly in your terminal. This is useful for viewing web page content or API responses.

Understanding the output: When you run this command, you'll see the HTML source code of the first website ever created, displayed in your terminal.

◆ Saving Content with curl

The basic curl command shows content on screen, but for automation you need to save files.

Save to a specific filename:

curl -o example_content.html http://info.cern.ch/

The -o option lets you choose any filename for the downloaded content.

Save with original filename:

curl -O https://www.gnu.org/licenses/gpl-3.0.txt

The -O option (capital O) saves the file using the same name as in the URL.

◆ Handling Redirects

Many websites use redirects to send you to the actual content location. Basic curl won't follow these automatically.

Following redirects:

curl -L -o example.html https://www.example.com

The -L option tells curl to follow redirects automatically, essential for reliable automation scripts.

📥 Understanding wget for File Downloads

◆ What is wget?

wget (World Wide Web Get) is purpose-built for downloading files. Unlike curl, it automatically saves content to files and shows download progress by default.

Basic wget syntax:

wget [options] URL

◆ Basic wget Usage

Download with automatic naming:

wget https://dlcdn.apache.org/tomcat/tomcat-10/v10.1.41/bin/apache-tomcat-10.1.41.tar.gz

wget automatically:

• Determines the filename from the URL
• Saves the file in your current directory
• Shows download progress with a progress bar

◆ Custom File Naming

Save with custom filename:

wget -O apache-tomcat.tar.gz https://dlcdn.apache.org/tomcat/tomcat-10/v10.1.41/bin/apache-tomcat-10.1.41.tar.gz

The -O option (capital O) lets you rename the downloaded file, useful for simplifying long or complex filenames.

◆ wget Advantages for Automation

Built-in features:

• Automatic retries - Restarts failed downloads
• Resume capability - Continues interrupted downloads
• Progress display - Shows download status
• Redirect following - Follows redirects by default

These features make wget reliable for automated deployment scripts.

◆ Task: To download using wget

use wget to download the file from https://releases.hashicorp.com/terraform/1.8.4/terraform_1.8.4_linux_amd64.zip

⚖️ Choosing Between curl and wget

◆ When to Use curl

Use curl for:

• API interactions - Testing REST APIs and web services
• Custom headers - Adding authentication or custom headers
• Multiple protocols - FTP, SMTP, and other protocols beyond HTTP
• Scripting flexibility - Complex data transfer operations

Example DevOps use case:

curl -H "Authorization: Bearer $TOKEN" https://api.github.com/repos/owner/repo/releases/latest

◆ When to Use wget

Use wget for:

• Simple file downloads - Straightforward file retrieval
• Large files - Built-in resume and retry capabilities
• Batch downloads - Multiple files or recursive downloads
• Reliability - Automatic error handling and retries

Example DevOps use case:

wget https://releases.hashicorp.com/terraform/1.8.4/terraform_1.8.4_linux_amd64.zip

◆ Key Differences Summary

📋 Essential Command Reference

💡 Key Takeaways

• curl displays content in terminal by default, while wget saves files automatically
• Use -o with curl to save files with custom names, -O to use the original filename
• curl requires -L to follow redirects, while wget follows them automatically
• wget is purpose-built for downloading and includes automatic retries and progress display
• curl is more versatile for API interactions and complex data transfer scenarios
• Both tools are essential for DevOps automation and can be used in deployment scripts
• Choose curl for flexibility and API work, wget for reliable file downloading
• Understanding both tools enables you to select the right tool for specific automation needs

Mastering command-line downloading enables reliable automation for software installations, configuration management, and deployment workflows in production environments.