replaced everything with ws
This commit is contained in:
Mykhailo Svishchov
492
BACKUP_TROUBLESHOOTING.md
Normal file
492
BACKUP_TROUBLESHOOTING.md
Normal file
@@ -0,0 +1,492 @@
|
||||
# Backup Script Permission Denied - Troubleshooting Guide
|
||||
|
||||
## Error Message
|
||||
```
|
||||
/bin/sh: 1: /opt/app/backend/lottery-be/scripts/backup-database.sh: Permission denied
|
||||
```
|
||||
|
||||
This error occurs when the system cannot execute the script, even if you've already run `chmod +x`. Here's a systematic approach to find the root cause.
|
||||
|
||||
---
|
||||
|
||||
## Step 1: Verify File Permissions
|
||||
|
||||
### Check Current Permissions
|
||||
```bash
|
||||
ls -la /opt/app/backend/lottery-be/scripts/backup-database.sh
|
||||
```
|
||||
|
||||
**Expected output:**
|
||||
```
|
||||
-rwxr-xr-x 1 root root 5678 Jan 15 10:00 /opt/app/backend/lottery-be/scripts/backup-database.sh
|
||||
```
|
||||
|
||||
**What to look for:**
|
||||
- The `x` (execute) permission should be present for owner, group, or others
|
||||
- If you see `-rw-r--r--` (no `x`), the file is not executable
|
||||
|
||||
**Fix if needed:**
|
||||
```bash
|
||||
chmod +x /opt/app/backend/lottery-be/scripts/backup-database.sh
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step 2: Check File System Mount Options
|
||||
|
||||
The file system might be mounted with `noexec` flag, which prevents executing scripts.
|
||||
|
||||
### Check Mount Options
|
||||
```bash
|
||||
mount | grep -E "(/opt|/app|/backend)"
|
||||
```
|
||||
|
||||
**What to look for:**
|
||||
- If you see `noexec` in the mount options, that's the problem
|
||||
- Example of problematic mount: `/dev/sda1 on /opt type ext4 (rw,noexec,relatime)`
|
||||
|
||||
**Fix:**
|
||||
1. Check `/etc/fstab`:
|
||||
```bash
|
||||
cat /etc/fstab | grep -E "(/opt|/app)"
|
||||
```
|
||||
2. If `noexec` is present, remove it and remount:
|
||||
```bash
|
||||
# Edit fstab (remove noexec)
|
||||
sudo nano /etc/fstab
|
||||
|
||||
# Remount (if /opt is a separate partition)
|
||||
sudo mount -o remount /opt
|
||||
```
|
||||
3. **Note:** If `/opt` is part of the root filesystem, you may need to reboot
|
||||
|
||||
---
|
||||
|
||||
## Step 3: Check Line Endings (CRLF vs LF)
|
||||
|
||||
Windows line endings (CRLF) can cause "Permission denied" errors on Linux.
|
||||
|
||||
### Check Line Endings
|
||||
```bash
|
||||
file /opt/app/backend/lottery-be/scripts/backup-database.sh
|
||||
```
|
||||
|
||||
**Expected output:**
|
||||
```
|
||||
/opt/app/backend/lottery-be/scripts/backup-database.sh: Bourne-Again shell script, ASCII text executable
|
||||
```
|
||||
|
||||
**If you see:**
|
||||
```
|
||||
/opt/app/backend/lottery-be/scripts/backup-database.sh: ASCII text, with CRLF line terminators
|
||||
```
|
||||
|
||||
**Fix:**
|
||||
```bash
|
||||
# Convert CRLF to LF
|
||||
dos2unix /opt/app/backend/lottery-be/scripts/backup-database.sh
|
||||
|
||||
# Or using sed
|
||||
sed -i 's/\r$//' /opt/app/backend/lottery-be/scripts/backup-database.sh
|
||||
|
||||
# Or using tr
|
||||
tr -d '\r' < /opt/app/backend/lottery-be/scripts/backup-database.sh > /tmp/backup-database.sh
|
||||
mv /tmp/backup-database.sh /opt/app/backend/lottery-be/scripts/backup-database.sh
|
||||
chmod +x /opt/app/backend/lottery-be/scripts/backup-database.sh
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step 4: Verify Shebang Line
|
||||
|
||||
The shebang line must point to a valid interpreter.
|
||||
|
||||
### Check Shebang
|
||||
```bash
|
||||
head -1 /opt/app/backend/lottery-be/scripts/backup-database.sh
|
||||
```
|
||||
|
||||
**Expected:**
|
||||
```bash
|
||||
#!/bin/bash
|
||||
```
|
||||
|
||||
**Verify bash exists:**
|
||||
```bash
|
||||
which bash
|
||||
ls -la /bin/bash
|
||||
```
|
||||
|
||||
**If bash doesn't exist or path is wrong:**
|
||||
```bash
|
||||
# Find bash location
|
||||
which bash
|
||||
# or
|
||||
whereis bash
|
||||
|
||||
# Update shebang if needed (bash is usually at /bin/bash or /usr/bin/bash)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step 5: Check SELinux (if enabled)
|
||||
|
||||
SELinux can block script execution even with correct permissions.
|
||||
|
||||
### Check if SELinux is Enabled
|
||||
```bash
|
||||
getenforce
|
||||
```
|
||||
|
||||
**Outputs:**
|
||||
- `Enforcing` - SELinux is active and blocking
|
||||
- `Permissive` - SELinux is active but only logging
|
||||
- `Disabled` - SELinux is off
|
||||
|
||||
### Check SELinux Context
|
||||
```bash
|
||||
ls -Z /opt/app/backend/lottery-be/scripts/backup-database.sh
|
||||
```
|
||||
|
||||
**Fix if SELinux is blocking:**
|
||||
```bash
|
||||
# Set correct context for shell scripts
|
||||
chcon -t bin_t /opt/app/backend/lottery-be/scripts/backup-database.sh
|
||||
|
||||
# Or restore default context
|
||||
restorecon -v /opt/app/backend/lottery-be/scripts/backup-database.sh
|
||||
|
||||
# Or temporarily set to permissive (for testing only)
|
||||
setenforce 0
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step 6: Check AppArmor (if enabled)
|
||||
|
||||
AppArmor can also block script execution.
|
||||
|
||||
### Check AppArmor Status
|
||||
```bash
|
||||
aa-status
|
||||
```
|
||||
|
||||
**If AppArmor is active and blocking:**
|
||||
```bash
|
||||
# Check AppArmor logs
|
||||
sudo dmesg | grep -i apparmor
|
||||
sudo journalctl -u apparmor | tail -20
|
||||
|
||||
# Temporarily disable for testing (not recommended for production)
|
||||
sudo systemctl stop apparmor
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step 7: Verify Cron Job User
|
||||
|
||||
The cron job might be running as a different user than expected.
|
||||
|
||||
### Check Cron Job
|
||||
```bash
|
||||
# Check root's crontab
|
||||
sudo crontab -l
|
||||
|
||||
# Check if cron job specifies a user
|
||||
# Example: 0 2 * * * root /opt/app/backend/lottery-be/scripts/backup-database.sh
|
||||
```
|
||||
|
||||
### Check Which User Runs Cron
|
||||
```bash
|
||||
# Check cron service logs
|
||||
sudo journalctl -u cron | tail -20
|
||||
|
||||
# Or check syslog
|
||||
sudo grep CRON /var/log/syslog | tail -10
|
||||
```
|
||||
|
||||
**Important:** The script requires root access (line 71-74 checks for EUID=0). Make sure cron runs as root:
|
||||
|
||||
```bash
|
||||
# Edit root's crontab (correct way)
|
||||
sudo crontab -e
|
||||
|
||||
# NOT user's crontab
|
||||
# crontab -e # This runs as current user, not root
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step 8: Test Script Execution Manually
|
||||
|
||||
Test the script with the same user that cron uses.
|
||||
|
||||
### Test as Root
|
||||
```bash
|
||||
# Test directly
|
||||
sudo /opt/app/backend/lottery-be/scripts/backup-database.sh
|
||||
|
||||
# Test with bash explicitly
|
||||
sudo bash /opt/app/backend/lottery-be/scripts/backup-database.sh
|
||||
|
||||
# Test with sh (if bash is not available)
|
||||
sudo sh /opt/app/backend/lottery-be/scripts/backup-database.sh
|
||||
```
|
||||
|
||||
**If manual execution works but cron doesn't:**
|
||||
- The issue is likely with cron's environment or user context
|
||||
- See Step 9 for cron environment issues
|
||||
|
||||
---
|
||||
|
||||
## Step 9: Check Cron Environment
|
||||
|
||||
Cron has a minimal environment. The script might need specific environment variables or paths.
|
||||
|
||||
### Check Script Dependencies
|
||||
The script uses:
|
||||
- `docker` command
|
||||
- `ssh` command
|
||||
- `gzip` command
|
||||
- `/run/secrets/lottery-config.properties` file
|
||||
|
||||
### Verify Commands are in PATH
|
||||
```bash
|
||||
# Check if commands are accessible
|
||||
which docker
|
||||
which ssh
|
||||
which gzip
|
||||
which bash
|
||||
|
||||
# If commands are not in standard PATH, update cron job:
|
||||
# Add PATH to cron job:
|
||||
0 2 * * * PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin /opt/app/backend/lottery-be/scripts/backup-database.sh >> /opt/app/logs/backup.log 2>&1
|
||||
```
|
||||
|
||||
### Test Cron Environment
|
||||
Create a test cron job to see the environment:
|
||||
|
||||
```bash
|
||||
# Add to crontab
|
||||
* * * * * env > /tmp/cron-env.txt
|
||||
|
||||
# Wait 1 minute, then check
|
||||
cat /tmp/cron-env.txt
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step 10: Check Directory Permissions
|
||||
|
||||
The directory containing the script must be executable.
|
||||
|
||||
### Check Directory Permissions
|
||||
```bash
|
||||
ls -ld /opt/app/backend/lottery-be/scripts/
|
||||
```
|
||||
|
||||
**Expected:**
|
||||
```
|
||||
drwxr-xr-x 2 root root 4096 Jan 15 10:00 /opt/app/backend/lottery-be/scripts/
|
||||
```
|
||||
|
||||
**If directory is not executable:**
|
||||
```bash
|
||||
chmod +x /opt/app/backend/lottery-be/scripts/
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step 11: Check for Hidden Characters
|
||||
|
||||
Hidden characters or encoding issues can break the shebang.
|
||||
|
||||
### View File in Hex
|
||||
```bash
|
||||
head -c 20 /opt/app/backend/lottery-be/scripts/backup-database.sh | od -c
|
||||
```
|
||||
|
||||
**Expected:**
|
||||
```
|
||||
0000000 # ! / b i n / b a s h \n
|
||||
```
|
||||
|
||||
**If you see strange characters:**
|
||||
```bash
|
||||
# Recreate the shebang line
|
||||
sed -i '1s/.*/#!\/bin\/bash/' /opt/app/backend/lottery-be/scripts/backup-database.sh
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step 12: Comprehensive Diagnostic Script
|
||||
|
||||
Run this diagnostic script to check all common issues:
|
||||
|
||||
```bash
|
||||
cat > /tmp/check-backup-script.sh << 'EOF'
|
||||
#!/bin/bash
|
||||
echo "=== Backup Script Diagnostic ==="
|
||||
echo ""
|
||||
|
||||
SCRIPT="/opt/app/backend/lottery-be/scripts/backup-database.sh"
|
||||
|
||||
echo "1. File exists?"
|
||||
[ -f "$SCRIPT" ] && echo " ✅ Yes" || echo " ❌ No"
|
||||
|
||||
echo "2. File permissions:"
|
||||
ls -la "$SCRIPT"
|
||||
|
||||
echo "3. File is executable?"
|
||||
[ -x "$SCRIPT" ] && echo " ✅ Yes" || echo " ❌ No"
|
||||
|
||||
echo "4. Shebang line:"
|
||||
head -1 "$SCRIPT"
|
||||
|
||||
echo "5. Bash exists?"
|
||||
[ -f /bin/bash ] && echo " ✅ Yes: /bin/bash" || [ -f /usr/bin/bash ] && echo " ✅ Yes: /usr/bin/bash" || echo " ❌ No"
|
||||
|
||||
echo "6. Line endings:"
|
||||
file "$SCRIPT"
|
||||
|
||||
echo "7. Mount options for /opt:"
|
||||
mount | grep -E "(/opt|/app)" || echo " (Not a separate mount)"
|
||||
|
||||
echo "8. SELinux status:"
|
||||
getenforce 2>/dev/null || echo " (Not installed)"
|
||||
|
||||
echo "9. Directory permissions:"
|
||||
ls -ld "$(dirname "$SCRIPT")"
|
||||
|
||||
echo "10. Test execution:"
|
||||
bash -n "$SCRIPT" && echo " ✅ Syntax OK" || echo " ❌ Syntax error"
|
||||
|
||||
echo ""
|
||||
echo "=== End Diagnostic ==="
|
||||
EOF
|
||||
|
||||
chmod +x /tmp/check-backup-script.sh
|
||||
/tmp/check-backup-script.sh
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step 13: Alternative Solutions
|
||||
|
||||
If the issue persists, try these workarounds:
|
||||
|
||||
### Solution A: Use bash Explicitly in Cron
|
||||
```bash
|
||||
# Instead of:
|
||||
0 2 * * * /opt/app/backend/lottery-be/scripts/backup-database.sh
|
||||
|
||||
# Use:
|
||||
0 2 * * * /bin/bash /opt/app/backend/lottery-be/scripts/backup-database.sh
|
||||
```
|
||||
|
||||
### Solution B: Create Wrapper Script
|
||||
```bash
|
||||
cat > /opt/app/backend/lottery-be/scripts/run-backup-wrapper.sh << 'EOF'
|
||||
#!/bin/bash
|
||||
cd /opt/app/backend/lottery-be
|
||||
exec /opt/app/backend/lottery-be/scripts/backup-database.sh "$@"
|
||||
EOF
|
||||
|
||||
chmod +x /opt/app/backend/lottery-be/scripts/run-backup-wrapper.sh
|
||||
|
||||
# Update cron to use wrapper
|
||||
0 2 * * * /opt/app/backend/lottery-be/scripts/run-backup-wrapper.sh >> /opt/app/logs/backup.log 2>&1
|
||||
```
|
||||
|
||||
### Solution C: Use systemd Timer Instead of Cron
|
||||
```bash
|
||||
# Create systemd service
|
||||
cat > /etc/systemd/system/lottery-backup.service << 'EOF'
|
||||
[Unit]
|
||||
Description=Lottery Database Backup
|
||||
After=network.target
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
ExecStart=/opt/app/backend/lottery-be/scripts/backup-database.sh
|
||||
User=root
|
||||
StandardOutput=append:/opt/app/logs/backup.log
|
||||
StandardError=append:/opt/app/logs/backup.log
|
||||
EOF
|
||||
|
||||
# Create systemd timer
|
||||
cat > /etc/systemd/system/lottery-backup.timer << 'EOF'
|
||||
[Unit]
|
||||
Description=Run Lottery Database Backup Daily
|
||||
Requires=lottery-backup.service
|
||||
|
||||
[Timer]
|
||||
OnCalendar=02:00
|
||||
Persistent=true
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
EOF
|
||||
|
||||
# Enable and start
|
||||
systemctl daemon-reload
|
||||
systemctl enable lottery-backup.timer
|
||||
systemctl start lottery-backup.timer
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Most Common Causes (Quick Reference)
|
||||
|
||||
1. **Line endings (CRLF)** - Most common if file was edited on Windows
|
||||
2. **File system mounted with `noexec`** - Check mount options
|
||||
3. **Cron running as wrong user** - Must run as root (use `sudo crontab -e`)
|
||||
4. **SELinux/AppArmor blocking** - Check security contexts
|
||||
5. **Missing execute permission** - Run `chmod +x` again
|
||||
6. **Directory not executable** - Check parent directory permissions
|
||||
|
||||
---
|
||||
|
||||
## Quick Fix Checklist
|
||||
|
||||
Run these commands in order:
|
||||
|
||||
```bash
|
||||
# 1. Fix line endings
|
||||
dos2unix /opt/app/backend/lottery-be/scripts/backup-database.sh
|
||||
# OR if dos2unix not available:
|
||||
sed -i 's/\r$//' /opt/app/backend/lottery-be/scripts/backup-database.sh
|
||||
|
||||
# 2. Ensure execute permission
|
||||
chmod +x /opt/app/backend/lottery-be/scripts/backup-database.sh
|
||||
|
||||
# 3. Ensure directory is executable
|
||||
chmod +x /opt/app/backend/lottery-be/scripts/
|
||||
|
||||
# 4. Test execution
|
||||
sudo /opt/app/backend/lottery-be/scripts/backup-database.sh --keep-local
|
||||
|
||||
# 5. Verify cron job uses bash explicitly
|
||||
sudo crontab -e
|
||||
# Change to: 0 2 * * * /bin/bash /opt/app/backend/lottery-be/scripts/backup-database.sh >> /opt/app/logs/backup.log 2>&1
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Still Not Working?
|
||||
|
||||
If none of the above fixes work, provide the output of:
|
||||
|
||||
```bash
|
||||
# Run diagnostic
|
||||
/tmp/check-backup-script.sh
|
||||
|
||||
# Check cron logs
|
||||
sudo journalctl -u cron | tail -50
|
||||
|
||||
# Check system logs
|
||||
sudo dmesg | tail -20
|
||||
```
|
||||
|
||||
This will help identify the exact issue.
|
||||
|
||||
Reference in New Issue
Block a user