Server Health Monitoring System

A robust, lightweight Python-based server health monitoring solution with HTTP and ICMP ping checks, smart email alerting, and state persistence. Designed for reliable execution as a scheduled task (cron job).

Features

• Multiple Check Types: HTTP/HTTPS requests and ICMP ping support
• Smart Alerting:
  • Sends alerts only on status changes (server down → server recovered)
  • Prevents duplicate alerts for continuous failures
  • Configurable retry logic before alerting
• State Persistence: Tracks server status between runs with file locking
• Comprehensive Logging: Rotating logs with configurable levels
• Security Hardened:
  • Command injection prevention for ping checks
  • Credentials stored in environment variables
  • Input validation for all user-provided data
• Modular Design: Clean separation of concerns (Strategy pattern for health checks)
• Email Notifications: HTML and plain text email alerts via SMTP

Requirements

• Python 3.8 or higher
• Linux (for ICMP ping functionality with fcntl file locking)
• Email account with SMTP access (Gmail or Outlook/Microsoft 365 supported)

Installation

1. Clone or Download the Project

cd /home/eo37/PycharmProjects/Server_Live_Test

2. Create Virtual Environment

python3 -m venv .venv
source .venv/bin/activate

3. Install Dependencies

pip install -r requirements.txt

4. Configure Environment

# Copy example configuration
cp .env.example .env

# Edit configuration with your settings
nano .env  # or your preferred editor

# Protect credentials
chmod 600 .env

Configuration

Edit the .env file with your settings:

Application Settings

LOG_LEVEL=INFO                    # DEBUG, INFO, WARNING, ERROR, CRITICAL
RETRY_COUNT=3                     # Failures before alerting

# Optional: Override default paths (defaults to ./logs and ./state in project directory)
# LOG_FILE=/custom/path/to/logs/monitor.log
# STATE_FILE=/custom/path/to/state/server_status.json

Timeout Settings

HTTP_TIMEOUT=30                   # HTTP request timeout (seconds)
PING_TIMEOUT=5                    # Ping timeout (seconds)
PING_COUNT=3                      # Number of ping packets

Email Settings

The system supports Gmail and Outlook/Microsoft 365 for sending alerts.

Option 1: Gmail

SMTP_HOST=smtp.gmail.com
SMTP_PORT=587
SMTP_USER=your-email@gmail.com
SMTP_PASSWORD=xxxx-xxxx-xxxx-xxxx   # 16-character app-specific password
FROM_EMAIL=your-email@gmail.com
TO_EMAILS=recipient@example.com
USE_TLS=true

Gmail Setup Requirements:

1. Enable 2-Step Verification on your Google Account
2. Generate app-specific password at myaccount.google.com/apppasswords
3. Use the 16-character password (not your regular Gmail password)
4. Limit: 500 emails/day

Option 2: Outlook/Microsoft 365

SMTP_HOST=smtp-mail.outlook.com
SMTP_PORT=587
SMTP_USER=your-email@outlook.com
SMTP_PASSWORD=your-regular-password   # Use your normal account password
FROM_EMAIL=your-email@outlook.com
TO_EMAILS=recipient@example.com
USE_TLS=true

Outlook Setup Requirements:

1. Works with @outlook.com, @hotmail.com, @live.com
2. Also works with Office 365 business accounts
3. Use your regular account password (no app password needed)
4. Limit: 300 emails/day (free), 10,000/day (Office 365)

Server Configuration

Format: url|check_type|timeout|name

SERVERS=http://example.com|http|30|Example Site,8.8.8.8|ping|5|Google DNS,https://api.mysite.com/health|http|15|API Health

Parameters:

• url: URL to check (HTTP) or hostname/IP (ping)
• check_type: http or ping
• timeout: Timeout in seconds for this server
• name: Friendly name for alerts (optional but recommended)

Usage

Manual Execution

Test the monitoring script:

# Activate virtual environment
source .venv/bin/activate

# Run the monitor
python main.py

Check logs:

tail -f logs/monitor.log

Automated Execution (Cron)

Set up cron to run every 15 minutes:

# Edit crontab
crontab -e

# Add this line (adjust path as needed):
*/15 * * * * cd /home/eo37/PycharmProjects/Server_Live_Test && .venv/bin/python main.py >> logs/cron.log 2>&1

Cron Schedule Options:

• */15 * * * * - Every 15 minutes
• */5 * * * * - Every 5 minutes
• 0 * * * * - Every hour
• */30 * * * * - Every 30 minutes

Verify Cron Setup:

# List current cron jobs
crontab -l

# Check cron logs
tail -f logs/cron.log

How It Works

Alert Logic (Duplicate Prevention)

The monitoring system uses smart state tracking to prevent alert spam:

1. Initial State: All servers start as "healthy" (prevents false alerts on first run)
2. First Failure: Counter increments to 1, no alert sent (might be transient network issue)
3. Second Failure: Counter increments to 2, no alert sent (default RETRY_COUNT=3)
4. Third Failure (Threshold): Counter reaches 3, sends DOWN alert via email
5. Continued Failures: Counter increments but no duplicate alerts sent
6. Server Recovery: When check succeeds, sends RECOVERY alert and resets counter to 0

Example Timeline (with RETRY_COUNT=3):

10:00 - Check succeeds → is_healthy=True, failures=0
10:15 - Check fails    → is_healthy=True, failures=1 (no alert)
10:30 - Check fails    → is_healthy=True, failures=2 (no alert)
10:45 - Check fails    → is_healthy=False, failures=3 (DOWN ALERT SENT)
11:00 - Check fails    → is_healthy=False, failures=4 (no alert - already down)
11:15 - Check succeeds → is_healthy=True, failures=0 (RECOVERY ALERT SENT)

This prevents hundreds of duplicate alerts if a server is down for hours or days.

State Persistence

Server state is stored in state/server_status.json with file locking to prevent corruption:

{
  "http://example.com": {
    "server_url": "http://example.com",
    "server_name": "Example Site",
    "is_healthy": false,
    "last_check": "2026-01-16T10:30:00",
    "consecutive_failures": 3,
    "last_alert_sent": "2026-01-16T10:15:00",
    "last_status": "timeout"
  }
}

Key State Fields:

• is_healthy: Boolean flag - prevents duplicate alerts when False
• consecutive_failures: Counter for retry threshold logic
• last_alert_sent: Timestamp of last alert (for debugging)
• last_status: Most recent check result (success/failure/timeout/error)

State persists between cron runs, enabling smart alerting across monitoring cycles.

Logging

Logs are stored in logs/monitor.log with daily rotation:

• Current log: logs/monitor.log
• Archived logs: logs/monitor.log.2026-01-16, logs/monitor.log.2026-01-15, etc.
• Retention: 30 days of logs kept automatically
• Rotation time: Midnight each day
• Format: [2026-01-16 10:30:00] INFO [server_monitor.main] - Server recovered: Example Site

Security Features

1. Ping Command Injection Prevention:

   • Whitelist validation (IP/hostname format)
   • Rejects shell metacharacters (;, |, &, etc.)
   • Uses subprocess without shell=True
   • Additional shlex.quote() escaping

2. Credential Security:

   • Stored in .env file (gitignored)
   • File permissions: chmod 600
   • Never logged or exposed in error messages

3. Input Validation:

   • Email addresses validated with regex
   • URLs validated with regex
   • IP addresses validated with regex

Testing

Run Test Suite

# Install test dependencies (already in requirements.txt)
source .venv/bin/activate

# Run all tests with coverage
pytest tests/ --cov=. --cov-report=html --cov-report=term

# Run specific test file
pytest tests/test_monitor.py -v

# Run tests matching pattern
pytest tests/ -k "test_http" -v

View Coverage Report

# Open HTML coverage report
open htmlcov/index.html  # macOS
xdg-open htmlcov/index.html  # Linux

Manual Testing

Test individual components:

# Test configuration loading
from config import ConfigLoader
loader = ConfigLoader()
app_config, email_config, servers = loader.load()
print(f"Loaded {len(servers)} servers")

# Test SMTP connection
from notifier import EmailNotifier
notifier = EmailNotifier(email_config)
notifier.test_connection()

# Test HTTP check
from monitor import HttpCheckStrategy
strategy = HttpCheckStrategy(timeout=30)
result = strategy.check("http://example.com")
print(f"Status: {result.status.value}")

Troubleshooting

No Emails Received

1. Check SMTP credentials:

   python -c "from config import ConfigLoader; from notifier import EmailNotifier; loader = ConfigLoader(); _, email_config, _ = loader.load(); EmailNotifier(email_config).test_connection()"

2. Verify SMTP settings: Ensure SMTP_HOST, SMTP_PORT are correct for your provider

3. Check spam folder: Initial alerts might be marked as spam

4. Review logs: Check logs/monitor.log for SMTP errors

Ping Not Working

1. Check permissions: ICMP ping may require elevated privileges on some systems

   # Grant ping capability (Linux)
   sudo setcap cap_net_raw+ep /usr/bin/ping

2. Test ping manually:

   ping -c 3 8.8.8.8

3. Check host format: Ensure hostname/IP is valid in SERVERS config

State File Corruption

If state file becomes corrupted:

# Backup corrupted state
mv state/server_status.json state/server_status.json.bak

# System will create new state on next run
python main.py

Configuration Errors

Common issues:

• Missing .env: Copy from .env.example
• Invalid format: Check SERVERS format (pipe-separated: url|type|timeout|name)
• Type errors: Ensure numeric values (timeouts, ports) are valid integers
• Email validation: Check email addresses match required format

Cron Not Running

1. Check cron service:

   sudo systemctl status cron  # or crond

2. Verify cron entry:

   crontab -l

3. Check cron logs:

   grep CRON /var/log/syslog  # Ubuntu/Debian
   grep CRON /var/log/cron    # CentOS/RHEL

4. Test script path:

   cd /home/eo37/PycharmProjects/Server_Live_Test && .venv/bin/python main.py

High False Positive Rate

Adjust RETRY_COUNT in .env:

• Increase value (e.g., 5) for more tolerance to transient failures
• Decrease value (e.g., 2) for faster alerting

Architecture

Module Overview

• main.py: Orchestrator with state management and alert logic
• monitor.py: Health check strategies (HTTP and ICMP ping)
• notifier.py: Email composition and SMTP sending
• config.py: Configuration loading and validation
• logger.py: Centralized logging with file rotation

Design Patterns

• Strategy Pattern: Pluggable health check strategies (easy to add new types)
• State Pattern: Server state transitions drive alert logic
• Dependency Injection: Components receive dependencies via constructor

Data Flow

1. Load Configuration (.env) → ConfigLoader
2. Setup Logging → MonitorLogger
3. Load State → StateManager
4. For each server:
   a. Create Strategy → HealthCheckFactory
   b. Perform Check → ServerMonitor
   c. Update State → StateManager
   d. Send Alert if needed → EmailNotifier
5. Save State → StateManager

Project Structure

Server_Live_Test/
├── main.py               # Entry point and orchestration
├── monitor.py            # Health check strategies
├── notifier.py           # Email alerting
├── config.py             # Configuration management
├── logger.py             # Logging setup
├── requirements.txt      # Python dependencies
├── .env.example          # Configuration template
├── .env                  # Actual configuration (gitignored)
├── .gitignore           # Git ignore rules
├── README.md            # This file
├── logs/                # Log files (created automatically)
│   └── monitor.log
├── state/               # State persistence (created automatically)
│   └── server_status.json
└── tests/               # Test suite
    ├── test_config.py
    ├── test_logger.py
    ├── test_monitor.py
    ├── test_notifier.py
    ├── test_main.py
    └── test_integration.py

Performance

• Execution Time: ~5s for 10 healthy servers, up to 250s worst case (all timeouts)
• Memory Usage: <50MB
• Disk Usage: Minimal (small log and state files)
• Network: One request per server per check

Contributing

This is a self-contained operational tool. For modifications:

1. Follow PEP 8 style guidelines
2. Add tests for new functionality
3. Update documentation
4. Ensure >80% test coverage

License

MIT License - feel free to use and modify for your needs.

Support

For issues or questions:

• Check logs: logs/monitor.log
• Review configuration: .env
• Verify server accessibility manually
• Test SMTP connection
• Check cron execution logs

Changelog

v1.0.0 (2026-01-16)

• Initial release
• HTTP and ICMP ping health checks
• Smart alerting with duplicate prevention
• State persistence with file locking
• Comprehensive test suite
• Full documentation