new README
This commit is contained in:
97
README.md
97
README.md
@@ -1,90 +1,41 @@
|
||||
# **Azure Blob Sync Script - How To Use**
|
||||
### **📌 How to Use**
|
||||
|
||||
This script syncs a directory to **Azure Blob Storage** using `azcopy` while:
|
||||
✅ **Avoiding business hours** (default: 9 AM - 5 PM, configurable).
|
||||
✅ **Preventing duplicate instances** (via a lock file).
|
||||
✅ **Automatically resuming** unfinished jobs.
|
||||
✅ **Logging progress & generating reports**.
|
||||
#### **1️⃣ Create the Directory List File**
|
||||
|
||||
---
|
||||
Example: `/opt/AZURE/dir2sync_list`
|
||||
|
||||
## **📌 1. Script Usage**
|
||||
Run the script manually:
|
||||
```bash
|
||||
./run_azcopy.sh <directory> [log=true|false] [bandwidth_mbps]
|
||||
```
|
||||
### **Example Commands**
|
||||
- **Basic sync with no bandwidth limit:**
|
||||
```bash
|
||||
./run_azcopy.sh /opt/AZURE/ false
|
||||
```
|
||||
- **Enable logging & limit bandwidth to 10 Mbps:**
|
||||
```bash
|
||||
./run_azcopy.sh /opt/AZURE/ true 10
|
||||
```
|
||||
- **Run in the background & detach from terminal:**
|
||||
```bash
|
||||
nohup ./run_azcopy.sh /opt/AZURE/ true 10 & disown
|
||||
```plaintext
|
||||
/opt/AZURE/BLOB > testcontainer115
|
||||
/opt/AZURE/OTHERBLOB > testcontainer115b
|
||||
/opt/AZURE/EXTRABLOB > testcontainer_extra
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## **⏲️ 2. Automating with Cron**
|
||||
Schedule the script to **run every hour**:
|
||||
#### **2️⃣ Run the Script**
|
||||
|
||||
```bash
|
||||
crontab -e
|
||||
./run_azcopy.sh /opt/AZURE/dir2sync_list true 20
|
||||
```
|
||||
Add this line:
|
||||
```bash
|
||||
0 * * * * /path/to/run_azcopy.sh /opt/AZURE/ true 10
|
||||
```
|
||||
### **How It Works**
|
||||
- Runs at **00 minutes past every hour** (e.g., 1:00, 2:00, 3:00, etc.).
|
||||
- If already running, the **next cron execution exits** to prevent duplicates.
|
||||
- If interrupted (e.g., business hours), the **next run resumes**.
|
||||
|
||||
- `true` → Enables logging.
|
||||
- `20` → Limits bandwidth to **20 Mbps**.
|
||||
|
||||
---
|
||||
|
||||
## **🔍 3. Checking If the Script Is Running**
|
||||
Check if `azcopy` or the script is running:
|
||||
#### **3️⃣ Cron Job (Runs Every Hour)**
|
||||
```bash
|
||||
pgrep -fl run_azcopy.sh
|
||||
pgrep -fl azcopy
|
||||
```
|
||||
To **stop it manually**:
|
||||
```bash
|
||||
pkill -f azcopy
|
||||
pkill -f run_azcopy.sh
|
||||
0 * * * * /opt/AZURE/run_azcopy.sh /opt/AZURE/dir2sync_list true 20
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## **📄 4. Checking Logs & Reports**
|
||||
- **Sync Log (if enabled)**:
|
||||
```bash
|
||||
tail -f azcopy_log_*.txt
|
||||
```
|
||||
- **Completion Report**:
|
||||
```bash
|
||||
cat completion_report_*.txt
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## **⚙️ 5. Customizing Business Hours**
|
||||
Modify the script to change business hours:
|
||||
```bash
|
||||
BUSINESS_HOURS_START=9
|
||||
BUSINESS_HOURS_END=17
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## **✅ 6. Expected Behavior**
|
||||
| Scenario | What Happens? |
|
||||
|----------|--------------|
|
||||
| **Cron runs script inside business hours** | Script exits immediately. |
|
||||
| **Script is running when cron starts again** | Second instance exits to prevent duplicates. |
|
||||
| **Sync job interrupted by business hours** | Next run resumes automatically. |
|
||||
| **Sync completes normally** | Report logs all transferred files. |
|
||||
|
||||
### **✅ Key Features**
|
||||
✅ **Prevents multiple instances** using a lock file (`/tmp/run_azcopy.lock`).
|
||||
✅ **Reads a file with directory-to-container mappings.**
|
||||
✅ **Validates each source directory before syncing.**
|
||||
✅ **Checks if the destination container exists** (test write).
|
||||
✅ **Stops immediately on any error** (clean runs only).
|
||||
✅ **Monitors `azcopy` every 30 seconds** and stops if business hours begin.
|
||||
✅ **Processes all directories sequentially using an array (fixing loop issue).**
|
||||
✅ **Uses one log file for the entire run.**
|
||||
|
||||
0
run_azcopy-single-dir.sh
Executable file → Normal file
0
run_azcopy-single-dir.sh
Executable file → Normal file
Reference in New Issue
Block a user