honey-be/BACKUP_TROUBLESHOOTING.md

# Backup Script Permission Denied - Troubleshooting Guide

## Error Message
```
/bin/sh: 1: /opt/app/backend/lottery-be/scripts/backup-database.sh: Permission denied
```

This error occurs when the system cannot execute the script, even if you've already run `chmod +x`. Here's a systematic approach to find the root cause.

---

## Step 1: Verify File Permissions

### Check Current Permissions
```bash
ls -la /opt/app/backend/lottery-be/scripts/backup-database.sh
```

**Expected output:**
```
-rwxr-xr-x 1 root root 5678 Jan 15 10:00 /opt/app/backend/lottery-be/scripts/backup-database.sh
```

**What to look for:**
- The `x` (execute) permission should be present for owner, group, or others
- If you see `-rw-r--r--` (no `x`), the file is not executable

**Fix if needed:**
```bash
chmod +x /opt/app/backend/lottery-be/scripts/backup-database.sh
```

---

## Step 2: Check File System Mount Options

The file system might be mounted with `noexec` flag, which prevents executing scripts.

### Check Mount Options
```bash
mount | grep -E "(/opt|/app|/backend)"
```

**What to look for:**
- If you see `noexec` in the mount options, that's the problem
- Example of problematic mount: `/dev/sda1 on /opt type ext4 (rw,noexec,relatime)`

**Fix:**
1. Check `/etc/fstab`:
   ```bash
   cat /etc/fstab | grep -E "(/opt|/app)"
   ```
2. If `noexec` is present, remove it and remount:
   ```bash
   # Edit fstab (remove noexec)
   sudo nano /etc/fstab
   
   # Remount (if /opt is a separate partition)
   sudo mount -o remount /opt
   ```
3. **Note:** If `/opt` is part of the root filesystem, you may need to reboot

---

## Step 3: Check Line Endings (CRLF vs LF)

Windows line endings (CRLF) can cause "Permission denied" errors on Linux.

### Check Line Endings
```bash
file /opt/app/backend/lottery-be/scripts/backup-database.sh
```

**Expected output:**
```
/opt/app/backend/lottery-be/scripts/backup-database.sh: Bourne-Again shell script, ASCII text executable
```

**If you see:**
```
/opt/app/backend/lottery-be/scripts/backup-database.sh: ASCII text, with CRLF line terminators
```

**Fix:**
```bash
# Convert CRLF to LF
dos2unix /opt/app/backend/lottery-be/scripts/backup-database.sh

# Or using sed
sed -i 's/\r$//' /opt/app/backend/lottery-be/scripts/backup-database.sh

# Or using tr
tr -d '\r' < /opt/app/backend/lottery-be/scripts/backup-database.sh > /tmp/backup-database.sh
mv /tmp/backup-database.sh /opt/app/backend/lottery-be/scripts/backup-database.sh
chmod +x /opt/app/backend/lottery-be/scripts/backup-database.sh
```

---

## Step 4: Verify Shebang Line

The shebang line must point to a valid interpreter.

### Check Shebang
```bash
head -1 /opt/app/backend/lottery-be/scripts/backup-database.sh
```

**Expected:**
```bash
#!/bin/bash
```

**Verify bash exists:**
```bash
which bash
ls -la /bin/bash
```

**If bash doesn't exist or path is wrong:**
```bash
# Find bash location
which bash
# or
whereis bash

# Update shebang if needed (bash is usually at /bin/bash or /usr/bin/bash)
```

---

## Step 5: Check SELinux (if enabled)

SELinux can block script execution even with correct permissions.

### Check if SELinux is Enabled
```bash
getenforce
```

**Outputs:**
- `Enforcing` - SELinux is active and blocking
- `Permissive` - SELinux is active but only logging
- `Disabled` - SELinux is off

### Check SELinux Context
```bash
ls -Z /opt/app/backend/lottery-be/scripts/backup-database.sh
```

**Fix if SELinux is blocking:**
```bash
# Set correct context for shell scripts
chcon -t bin_t /opt/app/backend/lottery-be/scripts/backup-database.sh

# Or restore default context
restorecon -v /opt/app/backend/lottery-be/scripts/backup-database.sh

# Or temporarily set to permissive (for testing only)
setenforce 0
```

---

## Step 6: Check AppArmor (if enabled)

AppArmor can also block script execution.

### Check AppArmor Status
```bash
aa-status
```

**If AppArmor is active and blocking:**
```bash
# Check AppArmor logs
sudo dmesg | grep -i apparmor
sudo journalctl -u apparmor | tail -20

# Temporarily disable for testing (not recommended for production)
sudo systemctl stop apparmor
```

---

## Step 7: Verify Cron Job User

The cron job might be running as a different user than expected.

### Check Cron Job
```bash
# Check root's crontab
sudo crontab -l

# Check if cron job specifies a user
# Example: 0 2 * * * root /opt/app/backend/lottery-be/scripts/backup-database.sh
```

### Check Which User Runs Cron
```bash
# Check cron service logs
sudo journalctl -u cron | tail -20

# Or check syslog
sudo grep CRON /var/log/syslog | tail -10
```

**Important:** The script requires root access (line 71-74 checks for EUID=0). Make sure cron runs as root:

```bash
# Edit root's crontab (correct way)
sudo crontab -e

# NOT user's crontab
# crontab -e  # This runs as current user, not root
```

---

## Step 8: Test Script Execution Manually

Test the script with the same user that cron uses.

### Test as Root
```bash
# Test directly
sudo /opt/app/backend/lottery-be/scripts/backup-database.sh

# Test with bash explicitly
sudo bash /opt/app/backend/lottery-be/scripts/backup-database.sh

# Test with sh (if bash is not available)
sudo sh /opt/app/backend/lottery-be/scripts/backup-database.sh
```

**If manual execution works but cron doesn't:**
- The issue is likely with cron's environment or user context
- See Step 9 for cron environment issues

---

## Step 9: Check Cron Environment

Cron has a minimal environment. The script might need specific environment variables or paths.

### Check Script Dependencies
The script uses:
- `docker` command
- `ssh` command
- `gzip` command
- `/run/secrets/lottery-config.properties` file

### Verify Commands are in PATH
```bash
# Check if commands are accessible
which docker
which ssh
which gzip
which bash

# If commands are not in standard PATH, update cron job:
# Add PATH to cron job:
0 2 * * * PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin /opt/app/backend/lottery-be/scripts/backup-database.sh >> /opt/app/logs/backup.log 2>&1
```

### Test Cron Environment
Create a test cron job to see the environment:

```bash
# Add to crontab
* * * * * env > /tmp/cron-env.txt

# Wait 1 minute, then check
cat /tmp/cron-env.txt
```

---

## Step 10: Check Directory Permissions

The directory containing the script must be executable.

### Check Directory Permissions
```bash
ls -ld /opt/app/backend/lottery-be/scripts/
```

**Expected:**
```
drwxr-xr-x 2 root root 4096 Jan 15 10:00 /opt/app/backend/lottery-be/scripts/
```

**If directory is not executable:**
```bash
chmod +x /opt/app/backend/lottery-be/scripts/
```

---

## Step 11: Check for Hidden Characters

Hidden characters or encoding issues can break the shebang.

### View File in Hex
```bash
head -c 20 /opt/app/backend/lottery-be/scripts/backup-database.sh | od -c
```

**Expected:**
```
0000000   #   !   /   b   i   n   /   b   a   s   h  \n
```

**If you see strange characters:**
```bash
# Recreate the shebang line
sed -i '1s/.*/#!\/bin\/bash/' /opt/app/backend/lottery-be/scripts/backup-database.sh
```

---

## Step 12: Comprehensive Diagnostic Script

Run this diagnostic script to check all common issues:

```bash
cat > /tmp/check-backup-script.sh << 'EOF'
#!/bin/bash
echo "=== Backup Script Diagnostic ==="
echo ""

SCRIPT="/opt/app/backend/lottery-be/scripts/backup-database.sh"

echo "1. File exists?"
[ -f "$SCRIPT" ] && echo "   ✅ Yes" || echo "   ❌ No"

echo "2. File permissions:"
ls -la "$SCRIPT"

echo "3. File is executable?"
[ -x "$SCRIPT" ] && echo "   ✅ Yes" || echo "   ❌ No"

echo "4. Shebang line:"
head -1 "$SCRIPT"

echo "5. Bash exists?"
[ -f /bin/bash ] && echo "   ✅ Yes: /bin/bash" || [ -f /usr/bin/bash ] && echo "   ✅ Yes: /usr/bin/bash" || echo "   ❌ No"

echo "6. Line endings:"
file "$SCRIPT"

echo "7. Mount options for /opt:"
mount | grep -E "(/opt|/app)" || echo "   (Not a separate mount)"

echo "8. SELinux status:"
getenforce 2>/dev/null || echo "   (Not installed)"

echo "9. Directory permissions:"
ls -ld "$(dirname "$SCRIPT")"

echo "10. Test execution:"
bash -n "$SCRIPT" && echo "   ✅ Syntax OK" || echo "   ❌ Syntax error"

echo ""
echo "=== End Diagnostic ==="
EOF

chmod +x /tmp/check-backup-script.sh
/tmp/check-backup-script.sh
```

---

## Step 13: Alternative Solutions

If the issue persists, try these workarounds:

### Solution A: Use bash Explicitly in Cron
```bash
# Instead of:
0 2 * * * /opt/app/backend/lottery-be/scripts/backup-database.sh

# Use:
0 2 * * * /bin/bash /opt/app/backend/lottery-be/scripts/backup-database.sh
```

### Solution B: Create Wrapper Script
```bash
cat > /opt/app/backend/lottery-be/scripts/run-backup-wrapper.sh << 'EOF'
#!/bin/bash
cd /opt/app/backend/lottery-be
exec /opt/app/backend/lottery-be/scripts/backup-database.sh "$@"
EOF

chmod +x /opt/app/backend/lottery-be/scripts/run-backup-wrapper.sh

# Update cron to use wrapper
0 2 * * * /opt/app/backend/lottery-be/scripts/run-backup-wrapper.sh >> /opt/app/logs/backup.log 2>&1
```

### Solution C: Use systemd Timer Instead of Cron
```bash
# Create systemd service
cat > /etc/systemd/system/lottery-backup.service << 'EOF'
[Unit]
Description=Lottery Database Backup
After=network.target

[Service]
Type=oneshot
ExecStart=/opt/app/backend/lottery-be/scripts/backup-database.sh
User=root
StandardOutput=append:/opt/app/logs/backup.log
StandardError=append:/opt/app/logs/backup.log
EOF

# Create systemd timer
cat > /etc/systemd/system/lottery-backup.timer << 'EOF'
[Unit]
Description=Run Lottery Database Backup Daily
Requires=lottery-backup.service

[Timer]
OnCalendar=02:00
Persistent=true

[Install]
WantedBy=timers.target
EOF

# Enable and start
systemctl daemon-reload
systemctl enable lottery-backup.timer
systemctl start lottery-backup.timer
```

---

## Most Common Causes (Quick Reference)

1. **Line endings (CRLF)** - Most common if file was edited on Windows
2. **File system mounted with `noexec`** - Check mount options
3. **Cron running as wrong user** - Must run as root (use `sudo crontab -e`)
4. **SELinux/AppArmor blocking** - Check security contexts
5. **Missing execute permission** - Run `chmod +x` again
6. **Directory not executable** - Check parent directory permissions

---

## Quick Fix Checklist

Run these commands in order:

```bash
# 1. Fix line endings
dos2unix /opt/app/backend/lottery-be/scripts/backup-database.sh
# OR if dos2unix not available:
sed -i 's/\r$//' /opt/app/backend/lottery-be/scripts/backup-database.sh

# 2. Ensure execute permission
chmod +x /opt/app/backend/lottery-be/scripts/backup-database.sh

# 3. Ensure directory is executable
chmod +x /opt/app/backend/lottery-be/scripts/

# 4. Test execution
sudo /opt/app/backend/lottery-be/scripts/backup-database.sh --keep-local

# 5. Verify cron job uses bash explicitly
sudo crontab -e
# Change to: 0 2 * * * /bin/bash /opt/app/backend/lottery-be/scripts/backup-database.sh >> /opt/app/logs/backup.log 2>&1
```

---

## Still Not Working?

If none of the above fixes work, provide the output of:

```bash
# Run diagnostic
/tmp/check-backup-script.sh

# Check cron logs
sudo journalctl -u cron | tail -50

# Check system logs
sudo dmesg | tail -20
```

This will help identify the exact issue.