From 383cd7e975f7c606be564c03047bc0bab22f00ea Mon Sep 17 00:00:00 2001 From: Radek Date: Fri, 21 Feb 2025 14:54:03 +0000 Subject: [PATCH] new README --- README.md | 101 ++++++++++----------------------------- run_azcopy-single-dir.sh | 0 2 files changed, 26 insertions(+), 75 deletions(-) mode change 100755 => 100644 run_azcopy-single-dir.sh diff --git a/README.md b/README.md index 07e96d4..76cb113 100644 --- a/README.md +++ b/README.md @@ -1,90 +1,41 @@ -# **Azure Blob Sync Script - How To Use** +### **📌 How to Use** -This script syncs a directory to **Azure Blob Storage** using `azcopy` while: -✅ **Avoiding business hours** (default: 9 AM - 5 PM, configurable). -✅ **Preventing duplicate instances** (via a lock file). -✅ **Automatically resuming** unfinished jobs. -✅ **Logging progress & generating reports**. +#### **1️⃣ Create the Directory List File** ---- +Example: `/opt/AZURE/dir2sync_list` -## **📌 1. Script Usage** -Run the script manually: -```bash -./run_azcopy.sh [log=true|false] [bandwidth_mbps] -``` -### **Example Commands** -- **Basic sync with no bandwidth limit:** - ```bash - ./run_azcopy.sh /opt/AZURE/ false - ``` -- **Enable logging & limit bandwidth to 10 Mbps:** - ```bash - ./run_azcopy.sh /opt/AZURE/ true 10 - ``` -- **Run in the background & detach from terminal:** - ```bash - nohup ./run_azcopy.sh /opt/AZURE/ true 10 & disown - ``` - ---- - -## **⏲️ 2. Automating with Cron** -Schedule the script to **run every hour**: -```bash -crontab -e -``` -Add this line: -```bash -0 * * * * /path/to/run_azcopy.sh /opt/AZURE/ true 10 -``` -### **How It Works** -- Runs at **00 minutes past every hour** (e.g., 1:00, 2:00, 3:00, etc.). -- If already running, the **next cron execution exits** to prevent duplicates. -- If interrupted (e.g., business hours), the **next run resumes**. - ---- - -## **🔍 3. Checking If the Script Is Running** -Check if `azcopy` or the script is running: -```bash -pgrep -fl run_azcopy.sh -pgrep -fl azcopy -``` -To **stop it manually**: -```bash -pkill -f azcopy -pkill -f run_azcopy.sh +```plaintext +/opt/AZURE/BLOB > testcontainer115 +/opt/AZURE/OTHERBLOB > testcontainer115b +/opt/AZURE/EXTRABLOB > testcontainer_extra ``` --- -## **📄 4. Checking Logs & Reports** -- **Sync Log (if enabled)**: - ```bash - tail -f azcopy_log_*.txt - ``` -- **Completion Report**: - ```bash - cat completion_report_*.txt - ``` +#### **2️⃣ Run the Script** + +```bash +./run_azcopy.sh /opt/AZURE/dir2sync_list true 20 +``` + +- `true` → Enables logging. +- `20` → Limits bandwidth to **20 Mbps**. --- -## **⚙️ 5. Customizing Business Hours** -Modify the script to change business hours: +#### **3️⃣ Cron Job (Runs Every Hour)** ```bash -BUSINESS_HOURS_START=9 -BUSINESS_HOURS_END=17 +0 * * * * /opt/AZURE/run_azcopy.sh /opt/AZURE/dir2sync_list true 20 ``` --- -## **✅ 6. Expected Behavior** -| Scenario | What Happens? | -|----------|--------------| -| **Cron runs script inside business hours** | Script exits immediately. | -| **Script is running when cron starts again** | Second instance exits to prevent duplicates. | -| **Sync job interrupted by business hours** | Next run resumes automatically. | -| **Sync completes normally** | Report logs all transferred files. | - +### **✅ Key Features** +✅ **Prevents multiple instances** using a lock file (`/tmp/run_azcopy.lock`). +✅ **Reads a file with directory-to-container mappings.** +✅ **Validates each source directory before syncing.** +✅ **Checks if the destination container exists** (test write). +✅ **Stops immediately on any error** (clean runs only). +✅ **Monitors `azcopy` every 30 seconds** and stops if business hours begin. +✅ **Processes all directories sequentially using an array (fixing loop issue).** +✅ **Uses one log file for the entire run.** diff --git a/run_azcopy-single-dir.sh b/run_azcopy-single-dir.sh old mode 100755 new mode 100644