How to Set a Timer for Downloading with wget
wget is a powerful command-line tool for downloading files from the internet. It offers various features to control the download process, including the ability to set a timer for automatic downloads. This blog post will guide you through the steps to set a timer for downloading with wget.
Combining wget with Other Shell Commands
To achieve timed downloads with wget, we will combine it with other shell commands and utilize a cronjob to schedule the script's execution. Below is a sample shell script (download_and_unique.sh) that we will create:
#!/bin/bash
# Set the timer in minutes
TIMER=5
# Function to download URLs from urls.txt and remove duplicates
download_and_unique() {
wget -i urls.txt -O - | sort -u >> lines.txt
}
# Loop to download and remove duplicates until script is stopped
while :
do
download_and_unique
sleep $(($TIMER * 60)) # Convert minutes to seconds for sleep
done
Saving and Executing the Script
Once you have created the script, save it and make it executable by running the following command:
chmod +x download_and_unique.shYou can then execute the script by running:
./download_and_unique.shScheduling the Script with cron
To schedule the script to run every 5 minutes using cron, edit your crontab file with the command:
crontab -eAdd the following line to your crontab file:
*/5 * * * * /path/to/download_and_unique.shReplace /path/to/download_and_unique.sh with the actual path where you saved the script.
Functionality of the Script
The script will continuously download URLs from a file named urls.txt, remove duplicate lines, and append the unique lines to a file named lines.txt. This process will continue every 5 minutes until the script is manually stopped.
Conclusion
By combining wget with other shell commands and utilizing cron, you can set a timer for automated downloads, allowing you to download files at specific intervals without manual intervention.