The technology stack behind the Authoring Platform consists of the following components:

• The Snow Owl Terminology Server application

• Elasticsearch as the data layer

• An LDAP-compliant authentication and authorization service

• Bugzilla Issue Tracker to back collaborative workflows

• A MySQL database to store Bugzilla's data

• The Snow Owl thick client

• Optional: A reverse-proxy handling the requests towards the REST API or Bugzilla

Terminology Server

Outgoing communication from the Terminology Server goes via:

• HTTP(s) towards Elasticsearch and Bugzilla

• LDAP(s) towards the A&A service

Incoming communication is handled through:

• the HTTP port of 8080 (from the REST API)

• the TCP port of 2036 (from the thick client)

A selected reverse proxy solution is responsible for channeling all incoming HTTP traffic through to the Terminology Server.

Elasticsearch

The currently supported version of Elasticsearch is which is upward compatible with any patch releases coming on the 7.x version stream. Elasticsearch v8 is not supported yet.

The Elasticsearch cluster can either be:

• a co-located, single-node, self-hosted cluster

• a managed Elasticsearch cluster hosted by

LDAP-compliant A&A service

For authorization and authentication, the application supports any traditional LDAP Directory Servers. We recommend starting with and evolving to other solutions later because it is easy to set up and maintain while keeping Snow Owl's user data isolated from any other A&A services.

Bugzilla & MySQL

To support the Authoring Platform's collaborative authoring features certain workflow items are stored in a Bugzilla Issue Tracker. However, it is not used as an Issue Tracker in our context but as a work item tracker, so-called Tasks.

Bugzilla stores its internal data in a MySQL database, hence why the technology stack contains one.

Snow Owl thick client

The Snow Owl thick client is the desktop application that is used for doing the actual authoring work. It is distributed together with the Authoring Platform release files but as a separate downloadable zip archive. Currently, only 64 bit Windows operating systems are supported.

Reverse proxy

A reverse proxy, such as is recommended to be utilized between the Terminology Server and either the intranet or the internet. This will increase security and help with channeling REST API requests appropriately.

With a preconfigured domain name and DNS record, the default installation package can take care of requesting and maintaining the necessary certificates for secure HTTP. See the details of this in the Configuration section.

Introduction

Welcome to the official documentation of the Snow Owl Authoring Platform. If you want to learn how to install and provision Snow Owl AP, you've come to the right place. This guide shows you how to:

• Select the appropriate hardware and software environment to host the service

• Download, install and configure the entire technology stack necessary for operating the server

• Handle release packages to upgrade to a newer version

• Perform a data backup or a restore

• Manage intermittent tasks, e.g. adding/revoking user access

In case you would like to skip ahead, here is a set of quick links leading to different sections of the guide.

To configure the Authoring Platform to work with a managed Elasticsearch cluster two settings require attention.

Configure Terminology Server

First, the local Elasticsearch container and all its configurations should be removed from the docker-compose.yml file. Once that is done, we have to tell the Terminology Server where to find the cluster. This can be set in the file ./snow-owl/docker/configs/snowowl/snowowl.yml:

Configure Elastic Cloud

Snow Owl TS leverages Elasticssearch's synonym filters. To have this feature work properly with a managed Elasticsearch cluster our custom dictionary has to be uploaded and configured. The synonym file can be found in the release package under ./snow-owl/docker/configs/elasticsearch/synonym.txt. This file needs to be compressed as an zip archive by following this structure:

For the managed Elasticsearch instance this zip file needs to be configured as a bundle extension. The steps required are covered in this guide in great detail:

Once the bundle is configured and the cluster is up we can (re)start the docker stack. In case there are any troubles the Terminology Server will refuse to initialize and let you know what the problem is in its log files.

Authoring Platform releases are shared with customers through custom download URLs. The downloaded server artifact is a Linux (tar.gz) archive that contains:

• an initial folder structure

• the configuration files for all services

• a docker-compose.yml file that brings together the entire technology stack to run and manage the service

• the credentials required to pull our proprietary docker images

As a best practice, it is advised to extract the content of the archive under /opt. So the deployment folder will be /opt/snow-owl. The docker-compose setup will rely on this path, however, if required it can be changed by editing the ./snow-owl/docker/.env file later on (see DEPLOYMENT_FOLDER environment variable).

When decompressing the archive it is important to use the --same-owner and --preserve-permissions options so the docker containers can access the files and folders appropriately.

The next page will describe the content of the release package in more detail.

Operating System

The server-side components of the Authoring Platform are recommended to be installed on x86_64 / amd64 Linux operating systems where Docker Engine is available. See the list of supported distributions by Docker:

Here is the list of distributions that we suggest in the order of recommendation:

The Authoring Platform release package contains a download URL for the Snow Owl thick client.

There are no special requirements to install, it is as easy as extracting the contents of the archive to a folder.

To avoid having issues the following should be considered:

• make sure to extract the files to a folder that has sufficient permissions for the logged-in user

The Snow Owl Authoring Platform has two different ways to manage users. The primary authentication and authorization service is the LDAP Directory Server. The secondary option is a file-based database, used only for administrative purposes. Whenever user access has to be granted or revoked the following methods could be applied.

LDAP based identity provider

There are several ways to access and manage an OpenLDAP server, hereby we will only describe one of them, through the Apache Directory Studio.

Apache Directory Studio is an open-source, free application. It is available to download for different platforms (Windows, macOS, and Linux).

Before accessing the LDAP database there is one technical prerequisite to satisfy. The OpenLDAP server has to be accessible from the machine Apache Directory Studio is installed. The best and most secure way to achieve that is to set up an SSH tunnel. Follow to an article that describes how to configure an SSH tunnel using PuTTY and Windows.

The OpenLDAP server uses port 389 for communication. This is the port that needs to be tunneled through the SSH connection. Here is what the final configuration looks like in PuTTY:

Once the SSH tunnel works, it's time to set up our connection in Apache DS. Go to File -> New -> LDAP Connection and set the following:

Hit the "Check Network Parameter" button to verify the network connection.

Go to the next page of the wizard and provide your credentials. The default Bind DN and Bind password can be found in the Authoring Platform release package under ./snow-owl/docker/.env.

Hit the "Check Authentication" button to verify your credentials. Hit Finish to complete the setup procedure.

All users and groups should be browseable now through the LDAP Browser view:

Grant user access

To grant access to a new user an LDAP entry has to be created. Go to the LDAP Browse view and right-click on the organization node, then New -> New Entry:

It is the easiest to use an existing entry as a template:

Leave everything as is on the Object Classes page, then hit Next. Fill in the new user's credentials:

On the final page, double click on the userPassword row and provide the user's password:

Hit Finish to add the user to the database.

Now we need to assign a role for the user. Before going forward, get ahold of the user's DN using the LDAP Browser view:

Select the desired role group in the Browser view and add a new attribute:

Select the attribute type uniqueMember and hit Finish:

Paste the user's DN as the value of the attribute and hit Enter to make your changes permanent:

Revoke user access

To revoke access the user has to be deleted from the list of users:

And also has to be removed from the role group:

Change credentials

To change either the first or last name, or the password of a user, just edit any of the attributes in the user editor:

File-based identity provider

There is a configuration file ./snow-owl/docker/configs/snowowl/users that contains the list of users with their credentials encrypted. The passwords are encrypted using the hash algorithm (variant $2a$). This method of authentication though should be used only for administration purposes (e.g. bot users) because any of the users added here will have admin privileges.

Grant user access

To grant access the users file has to be amended with the new user and its credentials. There are several ways to encrypt a password using the bcrypt algorithm but here is one that is easy and available on most of the Linux variants. The package called htpasswd has to be installed:

It will prompt for the password and will amend the file with the new user at the end.

Revoke user access

Simply remove the user's line from the file and restart the service.

Change credentials

Remove the user's line from the file and regenerate the credentials according to the section.

Here is the list of files and folders extracted from the release package and their role described down below.

/docker

Contains every configuration file used for the docker stack, including docker-compose.yml.

This folder is considered to be the context by docker, which means that upon executing commands one must either address the config file explicitly or execute docker-compose commands directly inside here.

E.g. to verify the status of the stack there are two approaches:

Execute the command inside ./snow-owl/docker:

Execute the command from somewhere else then ./snow-owl/docker:

/docker/configs/cert

This folder contains the files necessary to acquire an SSL certificate. None of the files should be changed here ideally.

/docker/configs/elasticsearch

There is one important file here, elasticsearch.yml which can be used for fine-tuning the Elasticsearch cluster. However, this is not necessary by default, only if an advanced configuration is required.

/docker/configs/ldap-boostrap

This folder contains the files used upon the first start of the OpenLDAP server. The files within describe a set of groups and users to set up an initial user access model. User credentials for the test users can be found in the file called 200_users.ldif.

/docker/configs/mysql

This folder contains the configuration file of the MySQL server.

/docker/configs/nginx

Location of all configuration files for NGINX. By default, a non-secure HTTP configuration is assumed. If there is no need for an SSL certificate, then the files here will be used. If an SSL certificate was acquired, then the main configuration file of NGINX (nginx.conf) will be overwritten with the one under /docker/cert/nginx.conf.

/docker/configs/snowowl

snowowl.yml: this file is the default configuration file of the Terminology Server. It does not need any changes by default either.

users: list of users for file-based authentication. There is one default user called snowowl for which the credentials can be found under ./docker/.env.

/docker/docker-compose.yml

The main configuration file for the docker stack. This file is replaced in case an SSL certificate was acquired (with file /docker/cert/docker-compose.yml). This is where volumes, ports, or environment variables can be configured.

/docker/docker_login.txt

The credentials to use for authenticating with the B2i private docker registry.

/docker/.env

The collection of environment variables for the docker-compose.yml file.

This is the file to configure most of the settings of the Terminology Server. Including java heap size, Snow Owl or Elasticsearch version, passwords, or folder structure.

/ldap

The location where the OpenLDAP server stores its data.

/logs

Log files of the Terminology Server

/resources

Location of Elasticsearch and Snow Owl resources.

/resources/indexes

This is the data folder of Elasticsearch. Datasets must be extracted to this directory.

/resources/attachments

Snow Owl's local file storage. Import and export artifacts are stored here.

/cert (optional)

In case an SSL certificate is acquired, all the files used by certbot and NGINX are stored here. This folder is automatically created by the certificate retrieval script.

/backup (optional)

This is the initial folder of all backup artifacts. This should be configured as a network mount to achieve data redundancy.

Having secure HTTP in case the Authoring Platform is a public-facing instance is definitely a must. For such cases, we are providing a pre-configured environment and a convenience script to acquire the necessary SSL certificate.

SSL certificate retrieval and renewal are managed by , the official ACME client recommended by .

To be able to obtain an SSL certificate the following requirements must be met:

• docker and docker-compose are installed

In certain cases, a pre-built dataset is also shipped together with the Authoring Platform. This is to ease the initial setup procedure and get going fast.

Snow Owl AP and co-located Elasticsearch cluster

For installations where Snow Owl AP and Elasticsearch is co-located we recommend the following hardware specification:

Snow Owl AP and managed Elasticsearch cluster

For installations where Snow Owl AP connects to a managed Elasticsearch cluster at we recommend the following hardware specification:

Cloud VMs

Here are a few examples of which Virtual Machine types could be used for hosting the Terminology Server at the three most popular Cloud providers (including but not limited to):

The Authoring Platform release package contains a built-in solution to perform rolling and permanent data backups. The docker stack has a specialized container (called snow-owl-backup) that is responsible for creating scheduled backups of:

• the Elasticsearch indices

• the OpenLDAP database (if present)

• Bugzilla's configuration files and SQL database

For the Elasticsearch indices, the backup container uses the . Snapshots are labeled in a predefined format with timestamps. E.g. snowowl-daily-20220324030001

The OpenLDAP database is backed up by compressing the contents of the folder under ./snow-owl/ldap. Filenames are generated using the name of the corresponding Elasticsearch snapshot. E.g. snowowl-daily-20220324030001.tar.gz.

Bugzilla is backed up by compressing the configuration files and the MySQL database dump. Filenames are generated using the name of the corresponding Elasticsearch snapshot. E.g. snowowl-daily-20220324030001.tar.gz.

Daily backups

Daily backups are rolling backups, scheduled, and cleaned up based on the settings specified in the ./snow-owl/docker/.env file. Here is a summary of the important settings that could be changed.

BACKUP_FOLDER

To store backups redundantly it is advised to mount a remote file share to a local path on the host. By default, this folder is configured to be at ./snow-owl/backup. It contains:

• the snapshot files of the Elasticsearch cluster

• the backup files of the OpenLDAP database

• the backup files of Bugzilla

CRON_DAYS, CRON_HOURS, CRON_MINUTES

Backup jobs are scheduled by crond, so cron-expressions can be defined here to specify the time a daily backup should happen.

NUMBER_OF_DAILY_BACKUPS_TO_KEEP

This is used to tell the backup container how many daily backups must be kept.

Example daily backup config

Let's say we have an external file share mounted to /mnt/external_folder. There is a need to create daily backups after each working day, during the night at 2:00 am. Only the last two-weeks-worth of data should be kept (assuming 5 working days each week).

One-off backups

It is also possible to perform backups occasionally, e.g. before versioning an important SNOMED CT release or before a Terminology Server version upgrade. These backups are kept until manually removed.

To create such backups the following command needs to be executed using the backup container's terminal:

The script will create a snapshot backup of the Elasticsearch data with a label snowowl-my-backup-label-20220405030002,an archive that contains the database of the OpenLDAP server and an archive that contains the configuration and database of Bugzilla with the name snowowl-my-backup-label-20220405030002.tar.gz.

Full list of steps to perform before spinning up the service:

1. Extract the Authoring Platform release archive to a folder. E.g. /opt/snow-owl

2. (Optional) Obtain an SSL certificate

   1. Make sure a DNS A record is routed to the host's public IP address

   2. Go into the folder ./snow-owl/docker/cert

   3. Execute the ./init-certificate.sh script:

3. (Optional) Configure access for managed Elasticsearch Cluster (elastic.co)

4. (Optional) Extract dataset to ./snow-owl/resources where folder structure should look like ./snow-owl/resources/indexes/nodes/0 at the end.

5. Verify file ownership to be UID=1000 and GID=0:

6. Check any credentials or settings that need to be changed in ./snow-owl/docker/.env

7. Make sure the hostname is configured properly in the environment file. This is important for being able to reach Bugzilla through a web browser.

8. Authenticate with our private docker registry while in the folder ./snow-owl/docker:

9. Issue a pull (in folder ./snow-owl/docker)

10. Spin up the service (in the folder ./snow-owl/docker)

11. Verify that the REST API of the Terminology Server is available at:

    1. With SSL: https://snow-owl.example.com/snowowl

    2. Without SSL: http://hostname:8080/snowowl

12. Verify that the server and cluster status is GREEN by querying the following REST API endpoint:

    1. With SSL:

    2. Without SSL:

13. Verify that Bugzilla's web UI is available at:

    1. With SSL:

    2. Without SSL:

14. Enjoy using the Snow Owl Authoring Platform 🎉

Using the custom backup container it is possible to restore:

• the Elasticsearch indices

• the Bugzilla database

• the OpenLDAP database (if present)

To restore any of the data the following steps have to be performed:

• stop Snow Owl, Elasticsearch, and the OpenLDAP containers (in the folder ./snow-owl/docker):

• (re)move the contents of the old / corrupted Elasticsearch data folder:

• restart the Elasticsearch container only (keep Snow Owl stopped):

• use the backup container's terminal and execute the restore script:

  • without any parameters, if only the Elasticsearch indices have to be restored

  • with parameter -l

• enter the numerical identifier of the backup to restore and wait until the process finishes

• exit the backup container and restart all containers:

When a new Snow Owl Authoring Platform release is available we recommend performing the following steps.

New releases are going to be distributed the same way: a docker stack and its configuration within an archive.

It is advised to decompress the new release files to a temporary folder and compare the contents of ./snow-owl/docker .

[root@host]# diff /opt/snow-owl/docker/ /opt/new-snow-owl-release/snow-owl/docker/
Common subdirectories: /opt/snow-owl/docker/configs and /opt/new-snow-owl-release/snow-owl/docker/configs
diff /opt/snow-owl/docker/.env /opt/new-snow-owl-release/snow-owl/docker/.env
10c10
< ELASTICSEARCH_VERSION=7.12.0
---
> ELASTICSEARCH_VERSION=7.16.3
24c24
< SNOWOWL_VERSION=7.18.0
---
> SNOWOWL_VERSION=7.19.0

The changes usually are restricted to version numbers in the .env file. In such cases, it is equally acceptable to overwrite the contents of the ./snow-owl/docker folder as is or cherry-pick the necessary modifications by hand.

Once the new version of the files is in place it is sufficient to just issue the following commands, an explicit stop of the service is not even required (in the folder ./snow-owl/docker):