Fetch All Tracker Reports Details

This utility script fetches location reports for all trackers for the past 7 days (or a configurable number of days). It's designed to be run as a one-time operation to fill in missing data for trackers.

Purpose

The main purpose of this script is to address situations where tracker data might be missing. It:

1. Fetches all trackers from the database
2. For each tracker, retrieves location reports for the past 7 days (configurable)
3. Stores new reports in the database
4. Refreshes the location_history materialized view

Usage

Using Docker Compose

The script is included in the tracker-report-fetcher Docker container. You can run it using docker compose exec:

# Basic usage (uses default settings from .env)
docker compose -f fetcher/compose.yml exec tracker-report-fetcher python fetch_all_tracker_reports.py

# Specify number of days to fetch (default is 7)
docker compose -f fetcher/compose.yml exec tracker-report-fetcher python fetch_all_tracker_reports.py --days 14

# Specify batch size (number of trackers to process in each batch, default is 10)
docker compose -f fetcher/compose.yml exec tracker-report-fetcher python fetch_all_tracker_reports.py --batch-size 20

# Specify batch delay (seconds to wait between batches, default is 5)
docker compose -f fetcher/compose.yml exec tracker-report-fetcher python fetch_all_tracker_reports.py --batch-delay 10

# Combine multiple options
docker compose -f fetcher/compose.yml exec tracker-report-fetcher python fetch_all_tracker_reports.py --days 10 --batch-size 15 --batch-delay 8

The script will use the environment variables defined in the Docker container, which are set from your .env file or the defaults in the compose.yml file.

Using the Python Script Directly

If you're not using Docker, you can also run the Python script directly:

# Basic usage (uses default settings)
python fetcher/build/fetch_all_tracker_reports.py

# Specify number of days to fetch (default is 7)
python fetcher/build/fetch_all_tracker_reports.py --days 14

# Specify anisette server URL (if not set in environment)
python fetcher/build/fetch_all_tracker_reports.py --anisette-server https://your-anisette-server.com

# Specify batch size (number of trackers to process in each batch, default is 10)
python fetcher/build/fetch_all_tracker_reports.py --batch-size 20

# Specify batch delay (seconds to wait between batches, default is 5)
python fetcher/build/fetch_all_tracker_reports.py --batch-delay 10

# Specify account store path
python fetcher/build/fetch_all_tracker_reports.py --account-store /path/to/account.json

# Combine multiple options
python fetcher/build/fetch_all_tracker_reports.py --days 10 --batch-size 15 --batch-delay 8

Note that when running the Python script directly, you'll need to ensure that the necessary environment variables are set. The script will look for environment variables in the following order:

1. Command line arguments
2. Environment variables
3. Default values

Requirements

The script requires:

1. Access to the Apple FindMy API via an anisette server
2. Database connection credentials (from environment variables or .env files)
3. The findmy Python package and its dependencies

Environment Variables

The script uses the following environment variables:

Variable	Description	Default
ANISETTE_SERVER	URL of the Anisette server	(required)
DB_HOST	Database host	localhost
DB_PORT	Database port	5432
DB_NAME	Database name	postgres
DB_USER	Database user	postgres
DB_PASSWORD	Database password	postgres
REDIS_HOST	Redis host	localhost
REDIS_PORT	Redis port	6379
REDIS_PASSWORD	Redis password	(empty)
REDIS_DB	Redis database number	0
MAX_REPORT_AGE_DAYS	Default number of days to fetch	7

These can be set in .env, fetcher/.env, or fetcher/.env.example files, or passed directly as environment variables.

How It Works

1. Authentication: The script authenticates with Apple's FindMy service using the provided anisette server.
2. Tracker Processing: It processes trackers in batches to avoid overwhelming the API.
3. Report Fetching: For each tracker, it fetches reports for the specified time period.
4. Data Storage: New reports are stored in the database, avoiding duplicates.
5. View Refresh: After all reports are processed, the location_history materialized view is refreshed.

Batch Processing

The script processes trackers in batches to:

• Reduce memory usage
• Provide progress updates
• Allow for graceful error handling
• Avoid API rate limiting

You can adjust the batch size and delay between batches using the command-line arguments.

Error Handling

The script includes robust error handling:

• Individual tracker failures don't stop the entire process
• Database connection issues are logged
• API authentication problems are reported clearly

Logging

Detailed logs are written to help track the progress and troubleshoot any issues.