Systemd Services

This document describes the background services that are managed by systemd in the Tracker GraphQL application.

Overview

The application uses several background services that run continuously to perform various tasks:

Location History Refresher: Refreshes the location_history materialized view when new data is available
Geofence Processing: Processes location history to update tracker statuses based on geofences
Status Change Notifier: Notifies clients about tracker status changes via Socket.IO

These services are managed by systemd, which ensures they start automatically on system boot and restart if they fail.

Location History Refresher

The Location History Refresher service performs two key functions:

Listens for PostgreSQL notifications and refreshes the location_history materialized view when new data is available
Periodically refreshes the continuous aggregates (at most once per hour) using Redis-based distributed locking

Service File

[Unit]
Description=Location History Refresher Service
After=network.target postgresql.service redis.service

[Service]
User=paulb
Group=paulb
WorkingDirectory=/home/paulb/scripts/tracker-graphql-2
ExecStart=/home/paulb/scripts/tracker-graphql-2/.venv/bin/python /home/paulb/scripts/tracker-graphql-2/app/src/location_history_refresher.py
Restart=on-failure
RestartSec=5s
StandardOutput=journal
StandardError=journal
SyslogIdentifier=location-history-refresher
Environment="REDIS_HOST=localhost"
Environment="REDIS_PORT=6379"
Environment="REDIS_USERNAME="
Environment="REDIS_PASSWORD="
Environment="REDIS_SSL=false"
Environment="REDIS_DB=0"

# Hardening
ProtectSystem=full
PrivateTmp=true
NoNewPrivileges=true

[Install]
WantedBy=multi-user.target

Docker Compose

The service can also be run using Docker Compose:

services:
  location-history-refresher:
    build:
      context: ./build
      dockerfile: Dockerfile
    image: tracker-services:latest
    command: refresher
    restart: unless-stopped
    environment:
      - DB_HOST=${DB_HOST:-postgres}
      - DB_PORT=${DB_PORT:-5432}
      - DB_NAME=${DB_NAME:-postgres}
      - DB_USER=${DB_USER:-postgres}
      - DB_PASSWORD=${DB_PASSWORD:-postgres}
      - REDIS_HOST=${REDIS_HOST:-redis}
      - REDIS_PORT=${REDIS_PORT:-6379}
      - REDIS_USERNAME=${REDIS_USERNAME:-}
      - REDIS_PASSWORD=${REDIS_PASSWORD:-}
      - REDIS_SSL=${REDIS_SSL:-}
      - REDIS_DB=${REDIS_DB:-0}
    deploy:
      restart_policy:
        condition: on-failure
        delay: 5s
        max_attempts: 3
        window: 120s

Installation

Systemd Installation

# Copy the systemd service file to the systemd directory
sudo cp systemd/location-history-refresher.service /etc/systemd/system/

# Reload systemd to recognize the new service
sudo systemctl daemon-reload

# Enable and start the service
sudo systemctl enable location-history-refresher
sudo systemctl start location-history-refresher

Docker Compose Installation

# Navigate to the fetcher directory
cd fetcher

# Start the services
docker-compose up -d

Geofence Processing

The Geofence Processing service periodically processes location history to update tracker statuses based on geofences.

Service File

[Unit]
Description=Geofence Processing Service
After=network.target

[Service]
User=paulb
WorkingDirectory=/home/paulb/scripts/tracker-graphql-2
ExecStart=/home/paulb/scripts/tracker-graphql-2/.venv/bin/python /home/paulb/scripts/tracker-graphql-2/app/src/process_geofences.py
Restart=on-failure
RestartSec=5s

[Install]
WantedBy=multi-user.target

Installation

# Copy the systemd service file to the systemd directory
sudo cp systemd/geofence-processing.service /etc/systemd/system/

# Reload systemd to recognize the new service
sudo systemctl daemon-reload

# Enable and start the service
sudo systemctl enable geofence-processing
sudo systemctl start geofence-processing

Status Change Notifier

The Status Change Notifier service listens for PostgreSQL notifications about tracker status changes and forwards them to connected clients via Socket.IO.

Service File

[Unit]
Description=Tracker Status Change Notifier
After=network.target

[Service]
User=tracker
WorkingDirectory=/home/paulb/scripts/tracker-graphql-2
ExecStart=/usr/bin/python3 app/src/status_change_notifier.py
Restart=always
RestartSec=5
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=status-change-notifier

[Install]
WantedBy=multi-user.target

Installation

# Copy the systemd service file to the systemd directory
sudo cp systemd/status-change-notifier.service /etc/systemd/system/

# Reload systemd to recognize the new service
sudo systemctl daemon-reload

# Enable and start the service
sudo systemctl enable status-change-notifier
sudo systemctl start status-change-notifier

Monitoring Services

Systemd Monitoring

You can monitor the status of these services using standard systemd commands:

# Check the status of a service
sudo systemctl status location-history-refresher

# View the logs for a service
sudo journalctl -u location-history-refresher -f

# Restart a service
sudo systemctl restart geofence-processing

# Stop a service
sudo systemctl stop status-change-notifier

Docker Compose Monitoring

For services running in Docker Compose:

# Check the status of all services
docker-compose -f fetcher/compose.yml ps

# View the logs for a specific service
docker-compose -f fetcher/compose.yml logs -f location-history-refresher

# Restart a service
docker-compose -f fetcher/compose.yml restart location-history-refresher

# Stop a service
docker-compose -f fetcher/compose.yml stop location-history-refresher

Redis Monitoring

For monitoring the Redis-based distributed locking:

# Connect to Redis CLI (standard)
redis-cli

# Connect to Redis CLI with authentication
redis-cli -h localhost -p 6379 -a your_password

# Connect to Redis CLI with username and password
redis-cli -h localhost -p 6379 -u your_username -a your_password

# Connect to Redis CLI with TLS (for AWS MemoryDB)
redis-cli -h your-cluster-endpoint.amazonaws.com --tls -u your_username -a your_password

# Check if Redis is running
ping

# View all keys related to location history refresher
keys location_history:*

# Check when the last aggregate refresh happened
get location_history:last_aggregate_refresh

# Check if a lock is currently held
exists location_history:aggregate_refresh_lock

Redis Cluster Support

When using AWS MemoryDB or other Redis Cluster deployments:

The system automatically detects Redis Cluster endpoints (containing "clustercfg" in the hostname)
It uses the appropriate Redis Cluster client
All keys use hash tags (e.g., {tracker}:pending) to ensure proper slot allocation

For detailed information about Redis Cluster support, including how we solved the CROSSSLOT error and ensured backward compatibility with standalone Redis, see Redis Cluster Support.

Troubleshooting

Service Won't Start

If a service won't start, check the logs:

# For systemd services
sudo journalctl -u service-name -f

# For Docker Compose services
docker-compose -f fetcher/compose.yml logs -f service-name

Common issues include:

Python path incorrect in the service file
Missing dependencies
Permission problems
Configuration errors
Redis connection issues (for the location-history-refresher)

Service Crashes Repeatedly

If a service crashes repeatedly:

Check the logs for error messages
Verify that the database is accessible
Check that the required tables and functions exist
Ensure that the service has the necessary permissions

Manual Testing

You can test the services manually by running the Python scripts directly:

# Activate the virtual environment
source .venv/bin/activate

# Run the location history refresher
python app/src/location_history_refresher.py

# Run the geofence processing
python app/src/process_geofences.py

# Run the status change notifier
python app/src/status_change_notifier.py

For Docker-based services:

# Run the location history refresher
docker-compose -f fetcher/compose.yml run --rm location-history-refresher

# Run the tracker report fetcher with options
docker-compose -f fetcher/compose.yml run --rm tracker-report-fetcher fetcher --once

This can help identify issues that might be related to the configuration rather than the scripts themselves.

Continuous Aggregate Refresh Issues

If the continuous aggregates aren't being refreshed:

Check if Redis is running and accessible:
```
redis-cli ping
```
Check the Redis keys related to the refresh lock:
```
redis-cli keys "location_history:*"
```

Manually refresh the continuous aggregates:

-- In PostgreSQL
CALL refresh_continuous_aggregate('location_history_hourly', NOW() - INTERVAL '48 hours', NOW());
CALL refresh_continuous_aggregate('location_history_daily', NOW() - INTERVAL '7 days', NOW());

Check the logs for any errors related to Redis:

# For systemd
sudo journalctl -u location-history-refresher -f | grep -i redis

# For Docker
docker-compose -f fetcher/compose.yml logs -f location-history-refresher | grep -i redis

Overview​

Location History Refresher​

Service File​

Docker Compose​

Installation​

Systemd Installation​

Docker Compose Installation​

Geofence Processing​

Service File​

Installation​

Status Change Notifier​

Service File​

Installation​

Monitoring Services​

Systemd Monitoring​

Docker Compose Monitoring​

Redis Monitoring​

Redis Cluster Support​

Troubleshooting​

Service Won't Start​

Service Crashes Repeatedly​

Manual Testing​

Continuous Aggregate Refresh Issues​

Overview

Location History Refresher

Service File

Docker Compose

Installation

Systemd Installation

Docker Compose Installation

Geofence Processing

Service File

Installation

Status Change Notifier

Service File

Installation

Monitoring Services

Systemd Monitoring

Docker Compose Monitoring

Redis Monitoring

Redis Cluster Support

Troubleshooting

Service Won't Start

Service Crashes Repeatedly

Manual Testing

Continuous Aggregate Refresh Issues