============================================ Digital Repositories and the Digital Archive ============================================ Digital objects - scans, born-digital files, archival information packages (AIP) - are not stored in ELZA. They stay in a repository, and ELZA links them to the units of description. Each repository is set up in *Administration* > *External systems* as an external system of the class *Digital object repository*. Its settings are stored in the database and apply without a restart; nothing about repositories is configured in :file:`elza.yaml`. Creating and changing repositories requires the administrator permission. Repository types ================ The type is chosen when the repository is created and cannot be changed later. ================ =========================================================== Type Use ================ =========================================================== Digital archive A digital archive (DA) storing AIPs. ELZA reads the packages over the REST API of the archive and sends changes back to it (see :ref:`da-repository`). File system A directory on the ELZA server or a mounted share. ELZA serves the files itself (see :ref:`fs-repository`). WSDL A repository connected over the older SOAP interface, which sends digital objects to ELZA (see :ref:`wsdl-repository`). ================ =========================================================== Common settings =============== Code Unique identifier of the repository. It cannot be changed later. For a digital archive it is part of the entry URL of an AIP (``/da//aip/``, see :doc:`07-integrations`); a WSDL repository identifies itself by it. Name Name shown to the users. URL Address of the repository; its meaning depends on the type. ELZA code Identifier of this ELZA instance towards the repository. It is sent to a WSDL repository and can be used in paths and URL templates as ``{repoCode}``. Multiple links Whether one digital object may be linked to more than one unit of description at the same time. With *No*, a second link is refused. In a digital archive the object is the whole AIP or a part of it. .. _da-repository: Digital archive =============== ELZA communicates with a digital archive over its REST API and over the File Transfer protocol, which is suitable for large packages: - the REST API reports changed AIPs, prepares packages for download and reports the result of a received export, - File Transfer carries every export and, when set, the downloads. Connection ---------- URL Base address of the REST API of the digital archive, for example ``https://da.archive.example/api``. A trailing slash is ignored. File Transfer URL Address of the File Transfer endpoint. When empty, it is derived from the URL as ``/ws/ft``; the form shows the derived address as a hint, and the detail of the repository shows the address in use. Fill it in only when the archive runs File Transfer elsewhere. REST API authentication How ELZA authenticates its calls to the REST API: - *None* - the requests carry no authentication. - *API key (NDA-HMAC-SHA256)* - ELZA signs every request with the key (headers ``Authorization: NDA-HMAC-SHA256`` and ``X-NDA-Date``); the key value itself is not sent. Requires *ApiKey - ID* and *ApiKey - value*. - *HTTP Basic (username and password)* - every request carries *Username* and *Password*, the same ones File Transfer uses. Requires *Username*. Use it only over HTTPS: Basic sends the password with every request, merely encoded. The setting does not affect File Transfer. ApiKey - ID, ApiKey - value API key for the *API key* authentication. Offered and stored only with that method. Username, Password Credentials for File Transfer (HTTP Basic), and for the REST API with the *HTTP Basic* authentication. Without a username, File Transfer connects without authentication. Verify TLS certificate *Yes* (the default) - the certificate of the archive is verified for the REST API and File Transfer alike. *No* - any certificate is trusted and its host name is not checked, for internal servers with a self-signed certificate. The connection is then not protected against a man-in-the-middle; prefer adding the certificate of the archive to the Java truststore of ELZA. The result of *Test configuration* says when the certificate is not verified. FileURL Address of the component viewer of the archive. ELZA opens it in a frame when a user views a linked digital object. The other URL fields (*DaoURL*, *ThumbnailURL*) are not used by a digital archive. The button *Test configuration* in the detail of the repository asks the REST API for the first changed AIP and reports whether the archive responds. It also shows the File Transfer address, but it does not test the File Transfer connection. Synchronisation --------------- ELZA asks the digital archive periodically which AIPs were added, changed or invalidated since the last question, and queues a download for each of them. An invalidated AIP is withdrawn: its links are closed and its local copy is deleted. Synchronization interval (s) Seconds between two questions; the default is 300 (five minutes). ``0`` stops the synchronisation. Saving the repository starts a synchronisation at once. Downloading packages -------------------- The digital archive prepares the requested packages as a batch; ELZA asks for the state of the batch until it is ready and then downloads it. AIP download method *Standard (HTTP)* downloads the batch over the REST API. *File Transfer* uses the File Transfer endpoint and suits large packages. An archive may refuse a standard download of a large batch with HTTP 413; the download then fails with a message asking to switch the repository to File Transfer. File Transfer download request type Offered only with the File Transfer method. Value sent with every download request as the type of the request (``type`` of ``GenericDataType``, next to the batch identifier in ``id``). When empty, ``download`` is sent, which is the type the DA communication standard defines. Fill it in only for an archive that expects another value; agree on it with the operator of the archive. The archive delivers the batch as a directory tree, one directory per package, as the standard prescribes; a batch delivered as a single zip file is accepted as well. Batch status check interval (s) Pause between two questions about the state of a batch, for downloads and exports alike; the default is 2 seconds, the minimum 1 second. On AIP received *Do nothing* - ELZA downloads only the basic information about the package (PACKAGE-INFO); users link the AIP themselves. *Download metadata* - ELZA also downloads the metadata of every received AIP whose fund it knows, builds its structure and links it to the description automatically: by the identifiers (UUID) of the package or its parts, and when none matches, by the placement script of the rules of the fund, if the rules have one. An AIP that cannot be placed stays unlinked, and a problem is recorded on it. Exports ------- A user sends the changes of linked AIPs to the digital archive from the AIP list (*Generate change AIPs*). ELZA builds a change package for each AIP and uploads it over File Transfer; the archive reports whether it accepted the package. AIPs not linked to a unit of description are not exported. Nothing is exported automatically. The package carries the contextual archival description of the AIP (``metadata/descriptive/EAD-CONTEXT.xml``, EAD3 by the Czech profile): the fund root, the path to every unit linked to the AIP and the subtrees of the linked units, generated by the EAD template of the ZP2015 rules. The units of one AIP must belong to one fund. The packages being downloaded and exported are kept in the :file:`aip` directory under ``elza.workingDir``. .. _da-queue: Request queue ------------- Every download and export is a request in the queue of the digital archive, *Administration* > *Digital archive queue* (also linked from the detail of the repository). The queue survives a restart: a batch the archive is preparing is taken up again. ========================= ================================================== State Meaning ========================= ================================================== To download, To update Waiting to be requested from the archive. DA is preparing the The archive is preparing the batch; ELZA checks download its state. Updated/Downloaded Done. Download error The package arrived but could not be processed, or the user ended a download that kept failing; the problem is also recorded on the AIP. To export Waiting to be uploaded. Sent, DA is processing Uploaded; the archive is processing it. Exported Accepted by the archive. Export error Refused or not confirmed by the archive. ========================= ================================================== A request replaced by a newer one for the same AIP, or cancelled, is marked *inactive*. Requests are not deleted. When the archive cannot be reached or answers with a server error, the request is retried after a growing pause: the batch status check interval, doubled after each failure, at most five minutes. The number of attempts, the time of the next one and what the archive answered are shown in the queue; the detail of a request also shows since when it has been failing. ELZA never gives up a download by itself. Some failures do not pass by retrying: the archive cannot prepare a package until the package is fixed there (for example metadata it cannot read). Report such a package to the administrator of the digital archive. A request that keeps failing is reported by the monitoring metric ``elza_da_failing_age_seconds`` (see :ref:`monitoring-metrics`). Actions on the selected requests: Retry now Takes a waiting request at once, without waiting for the next attempt. Cancel request Cancels a request the archive has not received yet. Repeat Requests a failed download again; for a failed export, builds a new change package from the current description and sends it. In the detail of a download that keeps failing (it has a *Failing since* time): Mark as failed After a confirmation, stops retrying the download; the request ends as *Download error* with the user's name and the last failure, which is also recorded on the AIP. Once the package is fixed in the archive, use *Repeat*. .. _fs-repository: File-system repository ====================== The URL is a path on the ELZA server, for example ``/mnt/digi/{fundCode}``. It may contain variables, so that each fund has its own directory: ===================== ========================================== Variable Value ===================== ========================================== ``{repoId}`` ID of the repository ``{repoCode}`` ELZA code of the repository ``{fundId}`` ID of the fund ``{fundNumber}`` number of the fund ``{fundCode}`` internal code of the fund ``{fundMark}`` mark of the fund ``{institutionId}`` ID of the institution of the fund ``{institutionCode}`` internal code of the institution ===================== ========================================== The ELZA service account must be able to read the directory. *Test configuration* checks that the path exists and can be read and lists its first entries; for a path with variables only the fixed part is checked. .. _wsdl-repository: WSDL repository =============== A WSDL repository sends digital objects to ELZA through the SOAP service ``DaoCoreService`` (see :doc:`07-integrations`) and identifies itself by the code of the repository. ELZA calls back the services ``DaoRequests`` and ``DaoNotifications`` of the repository at the URL followed directly by the service name, so the URL must end with a slash, for example ``https://dms.archive.example/ws/``. Username and password are used for these calls. Send notifications *Yes* - ELZA notifies the repository when a user links or unlinks one of its objects. DaoURL, FileURL, ThumbnailURL Templates of links to a digital object, to a file and to a thumbnail in the viewer of the repository. Variables: ``{code}`` (code of the object or file), ``{repoId}``, ``{repoCode}``, ``{id}``, ``{label}``, ``{nodeId}`` and ``{nodeUuid}`` (the linked unit of description) for an object, ``{fileName}`` for a file. New installations connect a digital archive instead of a WSDL repository.