========== Monitoring ========== ELZA provides health checks and metrics in the Prometheus format (`Spring Boot Actuator `_). They can be connected to common monitoring tools such as Prometheus, Grafana or Zabbix. Management port =============== The monitoring endpoints are enabled by default. They are published on a separate management port, ``8081``, bound to ``127.0.0.1``, so they are reachable only from the server itself. The application port is not affected. The built-in defaults are: .. code-block:: yaml management: server: port: 8081 address: 127.0.0.1 endpoints: web: exposure: include: health,prometheus endpoint: health: probes: enabled: true show-details: always Only the ``health`` and ``prometheus`` endpoints are published. Any value can be overridden in :file:`elza.yaml` or with an environment variable (for example ``MANAGEMENT_SERVER_PORT``). .. warning:: The monitoring endpoints do not require authentication; they are protected by being reachable only locally. To let a remote Prometheus server scrape the metrics, change ``management.server.address`` and restrict access to the port by other means (firewall, reverse proxy with authentication). Never publish the port to a public network. Endpoints ========= ``http://127.0.0.1:8081/actuator/health`` Overall state (``UP`` / ``DOWN``) with the state of the components (database, disk space, LDAP when configured). ``http://127.0.0.1:8081/actuator/health/liveness`` and ``.../readiness`` Liveness and readiness probes, for load balancers or container orchestrators. ``http://127.0.0.1:8081/actuator/prometheus`` Metrics in the Prometheus format. Example: .. code-block:: console $ curl http://127.0.0.1:8081/actuator/health {"status":"UP","components":{"db":{"status":"UP"},"diskSpace":{"status":"UP"}, "livenessState":{"status":"UP"},"ping":{"status":"UP"},"readinessState":{"status":"UP"}}} The LDAP health check is included only when LDAP authentication is configured (``elza.security.ldap.ad-domain``); it then checks the configured LDAP server. It can be switched off with ``management.health.ldap.enabled: false``. .. _monitoring-metrics: ELZA metrics ============ Besides the standard metrics of the JVM, the HTTP server (``http_server_requests_*``), the database connection pool (HikariCP) and the system, ELZA publishes metrics of its synchronisation queues with external systems (CAM and digital archives) and of its task scheduler. The metrics have no labels; queue values are aggregated over all configured external systems. Age metrics return ``-1`` when there is nothing to measure. .. list-table:: :header-rows: 1 :widths: 34 46 20 * - Metric - Meaning - Suggested alert * - ``elza_cam_last_success_age_seconds`` - Seconds since the last successful poll of a CAM external system (the oldest across all systems). A poll that finds no changes counts as successful. The main signal of CAM availability. - above 7200 * - ``elza_cam_error_count`` - Number of synchronisation items in the error state (for example a failed upload to CAM). - above 0 * - ``elza_cam_oldest_error_age_seconds`` - Seconds since the oldest item in the error state last changed state. Gives severity to the error count and avoids alerts on transient failures. - above 3600 * - ``elza_cam_deferred_oldest_age_seconds`` - Seconds since the oldest deferred item last changed state (for example a replacement whose replacing entity is not available yet). Deferred items normally resolve themselves. - above 86400 * - ``elza_cam_pending_oldest_age_seconds`` - Seconds since the oldest item waiting to be downloaded or uploaded entered its state. Grows when the queue is not processed, which the last-success metric cannot see. - above 3600 * - ``elza_da_failing_age_seconds`` - Seconds the longest failing waiting request of a digital archive queue has been failing. Such a request is retried without end, for example a package the digital archive cannot prepare until it is fixed there; the administrator either has it fixed or marks the request as failed in the queue (see :ref:`da-queue`). - above 14400 * - ``elza_da_failing_count`` - Number of waiting requests of digital archive queues whose exchanges with the digital archive keep failing. - none * - ``elza_da_error_count`` - Number of digital archive queue requests that ended in error (download, processing or export) and were not repeated. - none * - ``elza_scheduler_heartbeat_age_seconds`` - Seconds since the task scheduler last ticked. Grows without limit when the scheduler is frozen, which stops CAM synchronisation and all other scheduled tasks until a restart. - above 300 * - ``elza_scheduler_longest_running_task_seconds`` - Seconds the longest running scheduled task has been running; ``-1`` when idle. For diagnostics; a task running longer than ``elza.monitoring.scheduler.stuckThresholdSeconds`` is also logged with its stack trace. - none * - ``elza_users_connected`` - Number of connected users (WebSocket sessions). Informational. - none The machine-readable list of these metrics with the suggested thresholds is part of the distribution: :file:`server/config/csc-metrics.json`. The scheduler can be tuned with ``elza.monitoring.scheduler.poolSize`` (default 4 threads), ``elza.monitoring.scheduler.heartbeatMs`` (default 30000) and ``elza.monitoring.scheduler.stuckThresholdSeconds`` (default 1800). .. _monitoring-reporting: Reporting to a supervision service ================================== The ``elza-health`` command of the install scripts can report the state of the instance to a remote supervision service. LightComp v.o.s. offers such a service as part of its support (Customer Service Center, CSC); another system implementing the same interface can be used as well. - Reporting is active only when the access credentials (``CUSTOMER_ID`` and ``CUSTOMER_SERVICE_SECRET`` in :file:`elza-env`) are set. Without them, the scripts only check the state locally and send nothing. - Communication goes only from the instance to the supervision service (push) over HTTPS; each report is signed with the customer's key (HMAC). The supervision service never connects to the customer's infrastructure. Each report (every 10 minutes by default, ``HEARTBEAT_INTERVAL``) contains: - the overall state of the instance (``up`` / ``degraded`` / ``down``), - the results of the individual checks (systemd service state, HTTP check), - the application version and its start time, - the host name, - the values of the metrics listed in :file:`csc-metrics.json`. Only numeric values of the listed metrics are sent (an allow-list); no archival content, personal data or complete metrics output. To stop sending metrics while keeping the state reports, set ``CSC_METRICS_URL`` to an empty value; to stop reporting completely, remove the credentials. The other settings of the scripts are described at https://get.lightcomp.com/elza/.