493 lines
11 KiB
Markdown
493 lines
11 KiB
Markdown
# Backup Script Permission Denied - Troubleshooting Guide
|
|
|
|
## Error Message
|
|
```
|
|
/bin/sh: 1: /opt/app/backend/lottery-be/scripts/backup-database.sh: Permission denied
|
|
```
|
|
|
|
This error occurs when the system cannot execute the script, even if you've already run `chmod +x`. Here's a systematic approach to find the root cause.
|
|
|
|
---
|
|
|
|
## Step 1: Verify File Permissions
|
|
|
|
### Check Current Permissions
|
|
```bash
|
|
ls -la /opt/app/backend/lottery-be/scripts/backup-database.sh
|
|
```
|
|
|
|
**Expected output:**
|
|
```
|
|
-rwxr-xr-x 1 root root 5678 Jan 15 10:00 /opt/app/backend/lottery-be/scripts/backup-database.sh
|
|
```
|
|
|
|
**What to look for:**
|
|
- The `x` (execute) permission should be present for owner, group, or others
|
|
- If you see `-rw-r--r--` (no `x`), the file is not executable
|
|
|
|
**Fix if needed:**
|
|
```bash
|
|
chmod +x /opt/app/backend/lottery-be/scripts/backup-database.sh
|
|
```
|
|
|
|
---
|
|
|
|
## Step 2: Check File System Mount Options
|
|
|
|
The file system might be mounted with `noexec` flag, which prevents executing scripts.
|
|
|
|
### Check Mount Options
|
|
```bash
|
|
mount | grep -E "(/opt|/app|/backend)"
|
|
```
|
|
|
|
**What to look for:**
|
|
- If you see `noexec` in the mount options, that's the problem
|
|
- Example of problematic mount: `/dev/sda1 on /opt type ext4 (rw,noexec,relatime)`
|
|
|
|
**Fix:**
|
|
1. Check `/etc/fstab`:
|
|
```bash
|
|
cat /etc/fstab | grep -E "(/opt|/app)"
|
|
```
|
|
2. If `noexec` is present, remove it and remount:
|
|
```bash
|
|
# Edit fstab (remove noexec)
|
|
sudo nano /etc/fstab
|
|
|
|
# Remount (if /opt is a separate partition)
|
|
sudo mount -o remount /opt
|
|
```
|
|
3. **Note:** If `/opt` is part of the root filesystem, you may need to reboot
|
|
|
|
---
|
|
|
|
## Step 3: Check Line Endings (CRLF vs LF)
|
|
|
|
Windows line endings (CRLF) can cause "Permission denied" errors on Linux.
|
|
|
|
### Check Line Endings
|
|
```bash
|
|
file /opt/app/backend/lottery-be/scripts/backup-database.sh
|
|
```
|
|
|
|
**Expected output:**
|
|
```
|
|
/opt/app/backend/lottery-be/scripts/backup-database.sh: Bourne-Again shell script, ASCII text executable
|
|
```
|
|
|
|
**If you see:**
|
|
```
|
|
/opt/app/backend/lottery-be/scripts/backup-database.sh: ASCII text, with CRLF line terminators
|
|
```
|
|
|
|
**Fix:**
|
|
```bash
|
|
# Convert CRLF to LF
|
|
dos2unix /opt/app/backend/lottery-be/scripts/backup-database.sh
|
|
|
|
# Or using sed
|
|
sed -i 's/\r$//' /opt/app/backend/lottery-be/scripts/backup-database.sh
|
|
|
|
# Or using tr
|
|
tr -d '\r' < /opt/app/backend/lottery-be/scripts/backup-database.sh > /tmp/backup-database.sh
|
|
mv /tmp/backup-database.sh /opt/app/backend/lottery-be/scripts/backup-database.sh
|
|
chmod +x /opt/app/backend/lottery-be/scripts/backup-database.sh
|
|
```
|
|
|
|
---
|
|
|
|
## Step 4: Verify Shebang Line
|
|
|
|
The shebang line must point to a valid interpreter.
|
|
|
|
### Check Shebang
|
|
```bash
|
|
head -1 /opt/app/backend/lottery-be/scripts/backup-database.sh
|
|
```
|
|
|
|
**Expected:**
|
|
```bash
|
|
#!/bin/bash
|
|
```
|
|
|
|
**Verify bash exists:**
|
|
```bash
|
|
which bash
|
|
ls -la /bin/bash
|
|
```
|
|
|
|
**If bash doesn't exist or path is wrong:**
|
|
```bash
|
|
# Find bash location
|
|
which bash
|
|
# or
|
|
whereis bash
|
|
|
|
# Update shebang if needed (bash is usually at /bin/bash or /usr/bin/bash)
|
|
```
|
|
|
|
---
|
|
|
|
## Step 5: Check SELinux (if enabled)
|
|
|
|
SELinux can block script execution even with correct permissions.
|
|
|
|
### Check if SELinux is Enabled
|
|
```bash
|
|
getenforce
|
|
```
|
|
|
|
**Outputs:**
|
|
- `Enforcing` - SELinux is active and blocking
|
|
- `Permissive` - SELinux is active but only logging
|
|
- `Disabled` - SELinux is off
|
|
|
|
### Check SELinux Context
|
|
```bash
|
|
ls -Z /opt/app/backend/lottery-be/scripts/backup-database.sh
|
|
```
|
|
|
|
**Fix if SELinux is blocking:**
|
|
```bash
|
|
# Set correct context for shell scripts
|
|
chcon -t bin_t /opt/app/backend/lottery-be/scripts/backup-database.sh
|
|
|
|
# Or restore default context
|
|
restorecon -v /opt/app/backend/lottery-be/scripts/backup-database.sh
|
|
|
|
# Or temporarily set to permissive (for testing only)
|
|
setenforce 0
|
|
```
|
|
|
|
---
|
|
|
|
## Step 6: Check AppArmor (if enabled)
|
|
|
|
AppArmor can also block script execution.
|
|
|
|
### Check AppArmor Status
|
|
```bash
|
|
aa-status
|
|
```
|
|
|
|
**If AppArmor is active and blocking:**
|
|
```bash
|
|
# Check AppArmor logs
|
|
sudo dmesg | grep -i apparmor
|
|
sudo journalctl -u apparmor | tail -20
|
|
|
|
# Temporarily disable for testing (not recommended for production)
|
|
sudo systemctl stop apparmor
|
|
```
|
|
|
|
---
|
|
|
|
## Step 7: Verify Cron Job User
|
|
|
|
The cron job might be running as a different user than expected.
|
|
|
|
### Check Cron Job
|
|
```bash
|
|
# Check root's crontab
|
|
sudo crontab -l
|
|
|
|
# Check if cron job specifies a user
|
|
# Example: 0 2 * * * root /opt/app/backend/lottery-be/scripts/backup-database.sh
|
|
```
|
|
|
|
### Check Which User Runs Cron
|
|
```bash
|
|
# Check cron service logs
|
|
sudo journalctl -u cron | tail -20
|
|
|
|
# Or check syslog
|
|
sudo grep CRON /var/log/syslog | tail -10
|
|
```
|
|
|
|
**Important:** The script requires root access (line 71-74 checks for EUID=0). Make sure cron runs as root:
|
|
|
|
```bash
|
|
# Edit root's crontab (correct way)
|
|
sudo crontab -e
|
|
|
|
# NOT user's crontab
|
|
# crontab -e # This runs as current user, not root
|
|
```
|
|
|
|
---
|
|
|
|
## Step 8: Test Script Execution Manually
|
|
|
|
Test the script with the same user that cron uses.
|
|
|
|
### Test as Root
|
|
```bash
|
|
# Test directly
|
|
sudo /opt/app/backend/lottery-be/scripts/backup-database.sh
|
|
|
|
# Test with bash explicitly
|
|
sudo bash /opt/app/backend/lottery-be/scripts/backup-database.sh
|
|
|
|
# Test with sh (if bash is not available)
|
|
sudo sh /opt/app/backend/lottery-be/scripts/backup-database.sh
|
|
```
|
|
|
|
**If manual execution works but cron doesn't:**
|
|
- The issue is likely with cron's environment or user context
|
|
- See Step 9 for cron environment issues
|
|
|
|
---
|
|
|
|
## Step 9: Check Cron Environment
|
|
|
|
Cron has a minimal environment. The script might need specific environment variables or paths.
|
|
|
|
### Check Script Dependencies
|
|
The script uses:
|
|
- `docker` command
|
|
- `ssh` command
|
|
- `gzip` command
|
|
- `/run/secrets/lottery-config.properties` file
|
|
|
|
### Verify Commands are in PATH
|
|
```bash
|
|
# Check if commands are accessible
|
|
which docker
|
|
which ssh
|
|
which gzip
|
|
which bash
|
|
|
|
# If commands are not in standard PATH, update cron job:
|
|
# Add PATH to cron job:
|
|
0 2 * * * PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin /opt/app/backend/lottery-be/scripts/backup-database.sh >> /opt/app/logs/backup.log 2>&1
|
|
```
|
|
|
|
### Test Cron Environment
|
|
Create a test cron job to see the environment:
|
|
|
|
```bash
|
|
# Add to crontab
|
|
* * * * * env > /tmp/cron-env.txt
|
|
|
|
# Wait 1 minute, then check
|
|
cat /tmp/cron-env.txt
|
|
```
|
|
|
|
---
|
|
|
|
## Step 10: Check Directory Permissions
|
|
|
|
The directory containing the script must be executable.
|
|
|
|
### Check Directory Permissions
|
|
```bash
|
|
ls -ld /opt/app/backend/lottery-be/scripts/
|
|
```
|
|
|
|
**Expected:**
|
|
```
|
|
drwxr-xr-x 2 root root 4096 Jan 15 10:00 /opt/app/backend/lottery-be/scripts/
|
|
```
|
|
|
|
**If directory is not executable:**
|
|
```bash
|
|
chmod +x /opt/app/backend/lottery-be/scripts/
|
|
```
|
|
|
|
---
|
|
|
|
## Step 11: Check for Hidden Characters
|
|
|
|
Hidden characters or encoding issues can break the shebang.
|
|
|
|
### View File in Hex
|
|
```bash
|
|
head -c 20 /opt/app/backend/lottery-be/scripts/backup-database.sh | od -c
|
|
```
|
|
|
|
**Expected:**
|
|
```
|
|
0000000 # ! / b i n / b a s h \n
|
|
```
|
|
|
|
**If you see strange characters:**
|
|
```bash
|
|
# Recreate the shebang line
|
|
sed -i '1s/.*/#!\/bin\/bash/' /opt/app/backend/lottery-be/scripts/backup-database.sh
|
|
```
|
|
|
|
---
|
|
|
|
## Step 12: Comprehensive Diagnostic Script
|
|
|
|
Run this diagnostic script to check all common issues:
|
|
|
|
```bash
|
|
cat > /tmp/check-backup-script.sh << 'EOF'
|
|
#!/bin/bash
|
|
echo "=== Backup Script Diagnostic ==="
|
|
echo ""
|
|
|
|
SCRIPT="/opt/app/backend/lottery-be/scripts/backup-database.sh"
|
|
|
|
echo "1. File exists?"
|
|
[ -f "$SCRIPT" ] && echo " ✅ Yes" || echo " ❌ No"
|
|
|
|
echo "2. File permissions:"
|
|
ls -la "$SCRIPT"
|
|
|
|
echo "3. File is executable?"
|
|
[ -x "$SCRIPT" ] && echo " ✅ Yes" || echo " ❌ No"
|
|
|
|
echo "4. Shebang line:"
|
|
head -1 "$SCRIPT"
|
|
|
|
echo "5. Bash exists?"
|
|
[ -f /bin/bash ] && echo " ✅ Yes: /bin/bash" || [ -f /usr/bin/bash ] && echo " ✅ Yes: /usr/bin/bash" || echo " ❌ No"
|
|
|
|
echo "6. Line endings:"
|
|
file "$SCRIPT"
|
|
|
|
echo "7. Mount options for /opt:"
|
|
mount | grep -E "(/opt|/app)" || echo " (Not a separate mount)"
|
|
|
|
echo "8. SELinux status:"
|
|
getenforce 2>/dev/null || echo " (Not installed)"
|
|
|
|
echo "9. Directory permissions:"
|
|
ls -ld "$(dirname "$SCRIPT")"
|
|
|
|
echo "10. Test execution:"
|
|
bash -n "$SCRIPT" && echo " ✅ Syntax OK" || echo " ❌ Syntax error"
|
|
|
|
echo ""
|
|
echo "=== End Diagnostic ==="
|
|
EOF
|
|
|
|
chmod +x /tmp/check-backup-script.sh
|
|
/tmp/check-backup-script.sh
|
|
```
|
|
|
|
---
|
|
|
|
## Step 13: Alternative Solutions
|
|
|
|
If the issue persists, try these workarounds:
|
|
|
|
### Solution A: Use bash Explicitly in Cron
|
|
```bash
|
|
# Instead of:
|
|
0 2 * * * /opt/app/backend/lottery-be/scripts/backup-database.sh
|
|
|
|
# Use:
|
|
0 2 * * * /bin/bash /opt/app/backend/lottery-be/scripts/backup-database.sh
|
|
```
|
|
|
|
### Solution B: Create Wrapper Script
|
|
```bash
|
|
cat > /opt/app/backend/lottery-be/scripts/run-backup-wrapper.sh << 'EOF'
|
|
#!/bin/bash
|
|
cd /opt/app/backend/lottery-be
|
|
exec /opt/app/backend/lottery-be/scripts/backup-database.sh "$@"
|
|
EOF
|
|
|
|
chmod +x /opt/app/backend/lottery-be/scripts/run-backup-wrapper.sh
|
|
|
|
# Update cron to use wrapper
|
|
0 2 * * * /opt/app/backend/lottery-be/scripts/run-backup-wrapper.sh >> /opt/app/logs/backup.log 2>&1
|
|
```
|
|
|
|
### Solution C: Use systemd Timer Instead of Cron
|
|
```bash
|
|
# Create systemd service
|
|
cat > /etc/systemd/system/lottery-backup.service << 'EOF'
|
|
[Unit]
|
|
Description=Lottery Database Backup
|
|
After=network.target
|
|
|
|
[Service]
|
|
Type=oneshot
|
|
ExecStart=/opt/app/backend/lottery-be/scripts/backup-database.sh
|
|
User=root
|
|
StandardOutput=append:/opt/app/logs/backup.log
|
|
StandardError=append:/opt/app/logs/backup.log
|
|
EOF
|
|
|
|
# Create systemd timer
|
|
cat > /etc/systemd/system/lottery-backup.timer << 'EOF'
|
|
[Unit]
|
|
Description=Run Lottery Database Backup Daily
|
|
Requires=lottery-backup.service
|
|
|
|
[Timer]
|
|
OnCalendar=02:00
|
|
Persistent=true
|
|
|
|
[Install]
|
|
WantedBy=timers.target
|
|
EOF
|
|
|
|
# Enable and start
|
|
systemctl daemon-reload
|
|
systemctl enable lottery-backup.timer
|
|
systemctl start lottery-backup.timer
|
|
```
|
|
|
|
---
|
|
|
|
## Most Common Causes (Quick Reference)
|
|
|
|
1. **Line endings (CRLF)** - Most common if file was edited on Windows
|
|
2. **File system mounted with `noexec`** - Check mount options
|
|
3. **Cron running as wrong user** - Must run as root (use `sudo crontab -e`)
|
|
4. **SELinux/AppArmor blocking** - Check security contexts
|
|
5. **Missing execute permission** - Run `chmod +x` again
|
|
6. **Directory not executable** - Check parent directory permissions
|
|
|
|
---
|
|
|
|
## Quick Fix Checklist
|
|
|
|
Run these commands in order:
|
|
|
|
```bash
|
|
# 1. Fix line endings
|
|
dos2unix /opt/app/backend/lottery-be/scripts/backup-database.sh
|
|
# OR if dos2unix not available:
|
|
sed -i 's/\r$//' /opt/app/backend/lottery-be/scripts/backup-database.sh
|
|
|
|
# 2. Ensure execute permission
|
|
chmod +x /opt/app/backend/lottery-be/scripts/backup-database.sh
|
|
|
|
# 3. Ensure directory is executable
|
|
chmod +x /opt/app/backend/lottery-be/scripts/
|
|
|
|
# 4. Test execution
|
|
sudo /opt/app/backend/lottery-be/scripts/backup-database.sh --keep-local
|
|
|
|
# 5. Verify cron job uses bash explicitly
|
|
sudo crontab -e
|
|
# Change to: 0 2 * * * /bin/bash /opt/app/backend/lottery-be/scripts/backup-database.sh >> /opt/app/logs/backup.log 2>&1
|
|
```
|
|
|
|
---
|
|
|
|
## Still Not Working?
|
|
|
|
If none of the above fixes work, provide the output of:
|
|
|
|
```bash
|
|
# Run diagnostic
|
|
/tmp/check-backup-script.sh
|
|
|
|
# Check cron logs
|
|
sudo journalctl -u cron | tail -50
|
|
|
|
# Check system logs
|
|
sudo dmesg | tail -20
|
|
```
|
|
|
|
This will help identify the exact issue.
|
|
|