honey-be/BACKUP_TROUBLESHOOTING.md

Backup Script Permission Denied - Troubleshooting Guide

Error Message

/bin/sh: 1: /opt/app/backend/lottery-be/scripts/backup-database.sh: Permission denied

This error occurs when the system cannot execute the script, even if you've already run chmod +x. Here's a systematic approach to find the root cause.

---

Step 1: Verify File Permissions

Check Current Permissions

ls -la /opt/app/backend/lottery-be/scripts/backup-database.sh

Expected output:

-rwxr-xr-x 1 root root 5678 Jan 15 10:00 /opt/app/backend/lottery-be/scripts/backup-database.sh

What to look for:

• The x (execute) permission should be present for owner, group, or others
• If you see -rw-r--r-- (no x), the file is not executable

Fix if needed:

chmod +x /opt/app/backend/lottery-be/scripts/backup-database.sh

---

Step 2: Check File System Mount Options

The file system might be mounted with noexec flag, which prevents executing scripts.

Check Mount Options

mount | grep -E "(/opt|/app|/backend)"

What to look for:

• If you see noexec in the mount options, that's the problem
• Example of problematic mount: /dev/sda1 on /opt type ext4 (rw,noexec,relatime)

Fix:

1. Check /etc/fstab:

   cat /etc/fstab | grep -E "(/opt|/app)"
   

2. If noexec is present, remove it and remount:

   # Edit fstab (remove noexec)
   sudo nano /etc/fstab
   
   # Remount (if /opt is a separate partition)
   sudo mount -o remount /opt
   

3. Note: If /opt is part of the root filesystem, you may need to reboot

---

Step 3: Check Line Endings (CRLF vs LF)

Windows line endings (CRLF) can cause "Permission denied" errors on Linux.

Check Line Endings

file /opt/app/backend/lottery-be/scripts/backup-database.sh

Expected output:

/opt/app/backend/lottery-be/scripts/backup-database.sh: Bourne-Again shell script, ASCII text executable

If you see:

/opt/app/backend/lottery-be/scripts/backup-database.sh: ASCII text, with CRLF line terminators

Fix:

# Convert CRLF to LF
dos2unix /opt/app/backend/lottery-be/scripts/backup-database.sh

# Or using sed
sed -i 's/\r$//' /opt/app/backend/lottery-be/scripts/backup-database.sh

# Or using tr
tr -d '\r' < /opt/app/backend/lottery-be/scripts/backup-database.sh > /tmp/backup-database.sh
mv /tmp/backup-database.sh /opt/app/backend/lottery-be/scripts/backup-database.sh
chmod +x /opt/app/backend/lottery-be/scripts/backup-database.sh

---

Step 4: Verify Shebang Line

The shebang line must point to a valid interpreter.

Check Shebang

head -1 /opt/app/backend/lottery-be/scripts/backup-database.sh

Expected:

#!/bin/bash

Verify bash exists:

which bash
ls -la /bin/bash

If bash doesn't exist or path is wrong:

# Find bash location
which bash
# or
whereis bash

# Update shebang if needed (bash is usually at /bin/bash or /usr/bin/bash)

---

Step 5: Check SELinux (if enabled)

SELinux can block script execution even with correct permissions.

Check if SELinux is Enabled

getenforce

Outputs:

• Enforcing - SELinux is active and blocking
• Permissive - SELinux is active but only logging
• Disabled - SELinux is off

Check SELinux Context

ls -Z /opt/app/backend/lottery-be/scripts/backup-database.sh

Fix if SELinux is blocking:

# Set correct context for shell scripts
chcon -t bin_t /opt/app/backend/lottery-be/scripts/backup-database.sh

# Or restore default context
restorecon -v /opt/app/backend/lottery-be/scripts/backup-database.sh

# Or temporarily set to permissive (for testing only)
setenforce 0

---

Step 6: Check AppArmor (if enabled)

AppArmor can also block script execution.

Check AppArmor Status

aa-status

If AppArmor is active and blocking:

# Check AppArmor logs
sudo dmesg | grep -i apparmor
sudo journalctl -u apparmor | tail -20

# Temporarily disable for testing (not recommended for production)
sudo systemctl stop apparmor

---

Step 7: Verify Cron Job User

The cron job might be running as a different user than expected.

Check Cron Job

# Check root's crontab
sudo crontab -l

# Check if cron job specifies a user
# Example: 0 2 * * * root /opt/app/backend/lottery-be/scripts/backup-database.sh

Check Which User Runs Cron

# Check cron service logs
sudo journalctl -u cron | tail -20

# Or check syslog
sudo grep CRON /var/log/syslog | tail -10

Important: The script requires root access (line 71-74 checks for EUID=0). Make sure cron runs as root:

# Edit root's crontab (correct way)
sudo crontab -e

# NOT user's crontab
# crontab -e  # This runs as current user, not root

---

Step 8: Test Script Execution Manually

Test the script with the same user that cron uses.

Test as Root

# Test directly
sudo /opt/app/backend/lottery-be/scripts/backup-database.sh

# Test with bash explicitly
sudo bash /opt/app/backend/lottery-be/scripts/backup-database.sh

# Test with sh (if bash is not available)
sudo sh /opt/app/backend/lottery-be/scripts/backup-database.sh

If manual execution works but cron doesn't:

• The issue is likely with cron's environment or user context
• See Step 9 for cron environment issues

---

Step 9: Check Cron Environment

Cron has a minimal environment. The script might need specific environment variables or paths.

Check Script Dependencies

The script uses:

• docker command
• ssh command
• gzip command
• /run/secrets/lottery-config.properties file

Verify Commands are in PATH

# Check if commands are accessible
which docker
which ssh
which gzip
which bash

# If commands are not in standard PATH, update cron job:
# Add PATH to cron job:
0 2 * * * PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin /opt/app/backend/lottery-be/scripts/backup-database.sh >> /opt/app/logs/backup.log 2>&1

Test Cron Environment

Create a test cron job to see the environment:

# Add to crontab
* * * * * env > /tmp/cron-env.txt

# Wait 1 minute, then check
cat /tmp/cron-env.txt

---

Step 10: Check Directory Permissions

The directory containing the script must be executable.

Check Directory Permissions

ls -ld /opt/app/backend/lottery-be/scripts/

Expected:

drwxr-xr-x 2 root root 4096 Jan 15 10:00 /opt/app/backend/lottery-be/scripts/

If directory is not executable:

chmod +x /opt/app/backend/lottery-be/scripts/

---

Step 11: Check for Hidden Characters

Hidden characters or encoding issues can break the shebang.

View File in Hex

head -c 20 /opt/app/backend/lottery-be/scripts/backup-database.sh | od -c

Expected:

0000000   #   !   /   b   i   n   /   b   a   s   h  \n

If you see strange characters:

# Recreate the shebang line
sed -i '1s/.*/#!\/bin\/bash/' /opt/app/backend/lottery-be/scripts/backup-database.sh

---

Step 12: Comprehensive Diagnostic Script

Run this diagnostic script to check all common issues:

cat > /tmp/check-backup-script.sh << 'EOF'
#!/bin/bash
echo "=== Backup Script Diagnostic ==="
echo ""

SCRIPT="/opt/app/backend/lottery-be/scripts/backup-database.sh"

echo "1. File exists?"
[ -f "$SCRIPT" ] && echo "   ✅ Yes" || echo "   ❌ No"

echo "2. File permissions:"
ls -la "$SCRIPT"

echo "3. File is executable?"
[ -x "$SCRIPT" ] && echo "   ✅ Yes" || echo "   ❌ No"

echo "4. Shebang line:"
head -1 "$SCRIPT"

echo "5. Bash exists?"
[ -f /bin/bash ] && echo "   ✅ Yes: /bin/bash" || [ -f /usr/bin/bash ] && echo "   ✅ Yes: /usr/bin/bash" || echo "   ❌ No"

echo "6. Line endings:"
file "$SCRIPT"

echo "7. Mount options for /opt:"
mount | grep -E "(/opt|/app)" || echo "   (Not a separate mount)"

echo "8. SELinux status:"
getenforce 2>/dev/null || echo "   (Not installed)"

echo "9. Directory permissions:"
ls -ld "$(dirname "$SCRIPT")"

echo "10. Test execution:"
bash -n "$SCRIPT" && echo "   ✅ Syntax OK" || echo "   ❌ Syntax error"

echo ""
echo "=== End Diagnostic ==="
EOF

chmod +x /tmp/check-backup-script.sh
/tmp/check-backup-script.sh

---

Step 13: Alternative Solutions

If the issue persists, try these workarounds:

Solution A: Use bash Explicitly in Cron

# Instead of:
0 2 * * * /opt/app/backend/lottery-be/scripts/backup-database.sh

# Use:
0 2 * * * /bin/bash /opt/app/backend/lottery-be/scripts/backup-database.sh

Solution B: Create Wrapper Script

cat > /opt/app/backend/lottery-be/scripts/run-backup-wrapper.sh << 'EOF'
#!/bin/bash
cd /opt/app/backend/lottery-be
exec /opt/app/backend/lottery-be/scripts/backup-database.sh "$@"
EOF

chmod +x /opt/app/backend/lottery-be/scripts/run-backup-wrapper.sh

# Update cron to use wrapper
0 2 * * * /opt/app/backend/lottery-be/scripts/run-backup-wrapper.sh >> /opt/app/logs/backup.log 2>&1

Solution C: Use systemd Timer Instead of Cron

# Create systemd service
cat > /etc/systemd/system/lottery-backup.service << 'EOF'
[Unit]
Description=Lottery Database Backup
After=network.target

[Service]
Type=oneshot
ExecStart=/opt/app/backend/lottery-be/scripts/backup-database.sh
User=root
StandardOutput=append:/opt/app/logs/backup.log
StandardError=append:/opt/app/logs/backup.log
EOF

# Create systemd timer
cat > /etc/systemd/system/lottery-backup.timer << 'EOF'
[Unit]
Description=Run Lottery Database Backup Daily
Requires=lottery-backup.service

[Timer]
OnCalendar=02:00
Persistent=true

[Install]
WantedBy=timers.target
EOF

# Enable and start
systemctl daemon-reload
systemctl enable lottery-backup.timer
systemctl start lottery-backup.timer

---

Most Common Causes (Quick Reference)

1. Line endings (CRLF) - Most common if file was edited on Windows
2. File system mounted with noexec - Check mount options
3. Cron running as wrong user - Must run as root (use sudo crontab -e)
4. SELinux/AppArmor blocking - Check security contexts
5. Missing execute permission - Run chmod +x again
6. Directory not executable - Check parent directory permissions

---

Quick Fix Checklist

Run these commands in order:

# 1. Fix line endings
dos2unix /opt/app/backend/lottery-be/scripts/backup-database.sh
# OR if dos2unix not available:
sed -i 's/\r$//' /opt/app/backend/lottery-be/scripts/backup-database.sh

# 2. Ensure execute permission
chmod +x /opt/app/backend/lottery-be/scripts/backup-database.sh

# 3. Ensure directory is executable
chmod +x /opt/app/backend/lottery-be/scripts/

# 4. Test execution
sudo /opt/app/backend/lottery-be/scripts/backup-database.sh --keep-local

# 5. Verify cron job uses bash explicitly
sudo crontab -e
# Change to: 0 2 * * * /bin/bash /opt/app/backend/lottery-be/scripts/backup-database.sh >> /opt/app/logs/backup.log 2>&1

---

Still Not Working?

If none of the above fixes work, provide the output of:

# Run diagnostic
/tmp/check-backup-script.sh

# Check cron logs
sudo journalctl -u cron | tail -50

# Check system logs
sudo dmesg | tail -20

This will help identify the exact issue.