Introduction

Welcome to the official documentation of the Snow Owl Authoring Platform. If you want to learn how to install and provision Snow Owl AP, you've come to the right place. This guide shows you how to:

• Select the appropriate hardware and software environment to host the service

• Download, install and configure the entire technology stack necessary for operating the server

• Handle release packages to upgrade to a newer version

• Perform a data backup or a restore

• Manage intermittent tasks, e.g. adding/revoking user access

In case you would like to skip ahead, here is a set of quick links leading to different sections of the guide.

Authoring Platform releases are shared with customers through custom download URLs. The downloaded server artifact is a Linux (tar.gz) archive that contains:

• an initial folder structure

• the configuration files for all services

• a docker-compose.yml file that brings together the entire technology stack to run and manage the service

• the credentials required to pull our proprietary docker images

As a best practice, it is advised to extract the content of the archive under /opt. So the deployment folder will be /opt/snow-owl. The docker-compose setup will rely on this path, however, if required it can be changed by editing the ./snow-owl/docker/.env file later on (see DEPLOYMENT_FOLDER environment variable).

When decompressing the archive it is important to use the --same-owner and --preserve-permissions options so the docker containers can access the files and folders appropriately.

The next page will describe the content of the release package in more detail.

Having secure HTTP in case the Authoring Platform is a public-facing instance is definitely a must. For such cases, we are providing a pre-configured environment and a convenience script to acquire the necessary SSL certificate.

SSL certificate retrieval and renewal are managed by certbot, the official ACME client recommended by Let's Encrypt.

To be able to obtain an SSL certificate the following requirements must be met:

• docker and docker-compose are installed

• the server instance has a public IP address

• a DNS A record is configured for the desired domain name routing to the server's IP address

For the sake of example let's say the target domain name is snow-owl.b2ihealthcare.com .

Go to the sub-folder called ./snow-owl/docker/configs/cert. Make sure the init-certificate.sh script has permissions to be executable and get some details about its parameters:

[root@host]# pwd
/opt/snow-owl/docker/configs/cert
[root@host]# chmod +x init-certificate.sh
[root@host]# ./init-certificate.sh -h
  DESCRIPTION:

     Get certificate for the specified domain name using Let's Encrypt and certbot

  OPTIONS:
     -h
        Show this help
     -d domain
        Define the domain name to get the certificate for
     -e email (optional)
        The email address to use for the certificate registration

  EXAMPLES:

     ./init-certificate.sh -d mywebsite.com -e example@mail.com

     ./init-certificate.sh -d example.com

As you can see -d is used for specifying the domain name, and -e is used for specifying a contact email address (optional). Now execute the script with our example parameters:

./init-certificate.sh -d snow-owl.b2ihealthcare.com -e domain@b2ihealthcare.com

After successful execution, a new folder is created ./snow-owl/cert which contains all the certificate files required by NGINX. The docker-compose.yml file is also amended with a piece of code that guarantees automatic renewal of the certificate:

  nginx:
    image: nginx:stable
    container_name: nginx
    volumes:
      - ./configs/nginx/conf.d/:/etc/nginx/conf.d/
      - ./configs/nginx/nginx.conf:/etc/nginx/nginx.conf
      - ${CERT_FOLDER}/conf:/etc/letsencrypt
      - ${CERT_FOLDER}/www:/var/www/certbot
    depends_on:
      - snowowl
    ports:
      - "80:80"
      - "443:443"
    # Reload nginx config every 6 hours and restart
    command: "/bin/sh -c 'while :; do sleep 6h & wait $${!}; nginx -s reload; done & nginx -g \"daemon off;\"'"
    restart: unless-stopped
  certbot:
    image: certbot/certbot:latest
    container_name: certbot
    volumes:
      - ${CERT_FOLDER}/conf:/etc/letsencrypt
      - ${CERT_FOLDER}/www:/var/www/certbot
    # Check for SSL cert renewal every 12 hours
    entrypoint: "/bin/sh -c 'trap exit TERM; while :; do certbot renew; sleep 12h & wait $${!}; done;'"
    restart: unless-stopped

At this point everything is prepared for having secure HTTP, let's see what else needs to be configured before spinning up the service.

Snow Owl AP and co-located Elasticsearch cluster

For installations where Snow Owl AP and Elasticsearch is co-located we recommend the following hardware specification:

Snow Owl AP and managed Elasticsearch cluster

For installations where Snow Owl AP connects to a managed Elasticsearch cluster at elastic.co we recommend the following hardware specification:

Cloud VMs

Here are a few examples of which Virtual Machine types could be used for hosting the Terminology Server at the three most popular Cloud providers (including but not limited to):

The technology stack behind the Authoring Platform consists of the following components:

• The Snow Owl Terminology Server application

• Elasticsearch as the data layer

• An LDAP-compliant authentication and authorization service

• Bugzilla Issue Tracker to back collaborative workflows

• A MySQL database to store Bugzilla's data

• The Snow Owl thick client

• Optional: A reverse-proxy handling the requests towards the REST API or Bugzilla

Terminology Server

Outgoing communication from the Terminology Server goes via:

• HTTP(s) towards Elasticsearch and Bugzilla

• LDAP(s) towards the A&A service

Incoming communication is handled through:

• the HTTP port of 8080 (from the REST API)

• the TCP port of 2036 (from the thick client)

A selected reverse proxy solution is responsible for channeling all incoming HTTP traffic through to the Terminology Server.

Elasticsearch

The Elasticsearch cluster can either be:

• a co-located, single-node, self-hosted cluster

LDAP-compliant A&A service

Bugzilla & MySQL

To support the Authoring Platform's collaborative authoring features certain workflow items are stored in a Bugzilla Issue Tracker. However, it is not used as an Issue Tracker in our context but as a work item tracker, so-called Tasks.

Bugzilla stores its internal data in a MySQL database, hence why the technology stack contains one.

Snow Owl thick client

The Snow Owl thick client is the desktop application that is used for doing the actual authoring work. It is distributed together with the Authoring Platform release files but as a separate downloadable zip archive. Currently, only 64 bit Windows operating systems are supported.

Reverse proxy

With a preconfigured domain name and DNS record, the default installation package can take care of requesting and maintaining the necessary certificates for secure HTTP. See the details of this in the Configuration section.

Here is the list of files and folders extracted from the release package and their role described down below.

/docker

Contains every configuration file used for the docker stack, including docker-compose.yml.

This folder is considered to be the context by docker, which means that upon executing commands one must either address the config file explicitly or execute docker-compose commands directly inside here.

E.g. to verify the status of the stack there are two approaches:

Execute the command inside ./snow-owl/docker:

Execute the command from somewhere else then ./snow-owl/docker:

/docker/configs/cert

This folder contains the files necessary to acquire an SSL certificate. None of the files should be changed here ideally.

/docker/configs/elasticsearch

There is one important file here, elasticsearch.yml which can be used for fine-tuning the Elasticsearch cluster. However, this is not necessary by default, only if an advanced configuration is required.

/docker/configs/ldap-boostrap

This folder contains the files used upon the first start of the OpenLDAP server. The files within describe a set of groups and users to set up an initial user access model. User credentials for the test users can be found in the file called 200_users.ldif.

/docker/configs/mysql

This folder contains the configuration file of the MySQL server.

/docker/configs/nginx

Location of all configuration files for NGINX. By default, a non-secure HTTP configuration is assumed. If there is no need for an SSL certificate, then the files here will be used. If an SSL certificate was acquired, then the main configuration file of NGINX (nginx.conf) will be overwritten with the one under /docker/cert/nginx.conf.

/docker/configs/snowowl

snowowl.yml: this file is the default configuration file of the Terminology Server. It does not need any changes by default either.

users: list of users for file-based authentication. There is one default user called snowowl for which the credentials can be found under ./docker/.env.

/docker/docker-compose.yml

The main configuration file for the docker stack. This file is replaced in case an SSL certificate was acquired (with file /docker/cert/docker-compose.yml). This is where volumes, ports, or environment variables can be configured.

/docker/docker_login.txt

The credentials to use for authenticating with the B2i private docker registry.

/docker/.env

The collection of environment variables for the docker-compose.yml file.

This is the file to configure most of the settings of the Terminology Server. Including java heap size, Snow Owl or Elasticsearch version, passwords, or folder structure.

/ldap

The location where the OpenLDAP server stores its data.

/logs

Log files of the Terminology Server

/resources

Location of Elasticsearch and Snow Owl resources.

/resources/indexes

This is the data folder of Elasticsearch. Datasets must be extracted to this directory.

/resources/attachments

Snow Owl's local file storage. Import and export artifacts are stored here.

/cert (optional)

In case an SSL certificate is acquired, all the files used by certbot and NGINX are stored here. This folder is automatically created by the certificate retrieval script.

/backup (optional)

This is the initial folder of all backup artifacts. This should be configured as a network mount to achieve data redundancy.

To configure the Authoring Platform to work with a managed Elasticsearch cluster two settings require attention.

Configure Terminology Server

First, the local Elasticsearch container and all its configurations should be removed from the docker-compose.yml file. Once that is done, we have to tell the Terminology Server where to find the cluster. This can be set in the file ./snow-owl/docker/configs/snowowl/snowowl.yml:

Configure Elastic Cloud

Snow Owl TS leverages Elasticssearch's synonym filters. To have this feature work properly with a managed Elasticsearch cluster our custom dictionary has to be uploaded and configured. The synonym file can be found in the release package under ./snow-owl/docker/configs/elasticsearch/synonym.txt. This file needs to be compressed as an zip archive by following this structure:

For the managed Elasticsearch instance this zip file needs to be configured as a bundle extension. The steps required are covered in this guide in great detail:

Once the bundle is configured and the cluster is up we can (re)start the docker stack. In case there are any troubles the Terminology Server will refuse to initialize and let you know what the problem is in its log files.

In certain cases, a pre-built dataset is also shipped together with the Authoring Platform. This is to ease the initial setup procedure and get going fast.

These datasets are the compressed form of the Elasticsearch data folder which follows the same structure. Except for having a top folder called indexes . This is the same folder as in ./snow-owl/resources/indexes . So to be able to load the dataset one should just extract the contents of the dataset archive to this path.

Using the custom backup container it is possible to restore:

• the Elasticsearch indices

• the Bugzilla database

• the OpenLDAP database (if present)

To restore any of the data the following steps have to be performed:

• stop Snow Owl, Elasticsearch, and the OpenLDAP containers (in the folder ./snow-owl/docker):

docker-compose stop snowowl elasticsearch ldap

• (re)move the contents of the old / corrupted Elasticsearch data folder:

mv -t /tmp ./snow-owl/resources/indexes/nodes

• restart the Elasticsearch container only (keep Snow Owl stopped):

docker-compose start elasticsearch

• use the backup container's terminal and execute the restore script:

  • without any parameters, if only the Elasticsearch indices have to be restored

  Copy

  root@host:# docker exec -it backup bash
  root@ad36cfb0448c:# /backup/restore.sh

  • with parameter -l in case the Elasticsearch indices and the OpenLDAP database have to be restored at the same time

  Copy

  root@host:# docker exec -it backup bash
  root@ad36cfb0448c:# /backup/restore.sh -l

  • with parameter -b in case the Elasticsearch indices and the Bugzilla database have to be restored at the same time

  Copy

  root@host:# docker exec -it backup bash
  root@ad36cfb0448c:# /backup/restore.sh -b

  • with parameter -b and -l if a full restore has to be performed

  Copy

  root@host:# docker exec -it backup bash
  root@ad36cfb0448c:# /backup/restore.sh -b -l

• the script will list all available backups and prompts for selection:

root@ad36cfb0448c:# /backup/restore.sh

################################
Snow Owl restore script STARTED.

#### Verify Elasticsearch snapshot repository ####

Checking existence of repository 'snowowl-snapshots' ...
Repository with name 'snowowl-snapshots' is present, verifying repository state ...
Repository 'snowowl-snapshots' is functional

#### Select backup to restore ####

Found 10 available backups under '/backup'
Please select the backup to restore by choosing the right number in the menu below (hit Enter when the selection was made)

 1) snowowl-daily-20220323030001
 2) snowowl-daily-20220324030001
 3) snowowl-daily-20220325030002
 4) snowowl-daily-20220326030002
 5) snowowl-daily-20220329030001
 6) snowowl-daily-20220330030001
 7) snowowl-daily-20220331030002
 8) snowowl-daily-20220401030002
 9) snowowl-daily-20220402030001
10) snowowl-daily-20220405030002

#?

• enter the numerical identifier of the backup to restore and wait until the process finishes

• exit the backup container and restart all containers:

root@ad36cfb0448c:# exit
root@host:# docker-compose stop snowowl elasticsearch ldap bugzilla
root@host:# docker-compose up -d

When a new Snow Owl Authoring Platform release is available we recommend performing the following steps.

New releases are going to be distributed the same way: a docker stack and its configuration within an archive.

It is advised to decompress the new release files to a temporary folder and compare the contents of ./snow-owl/docker .

[root@host]# diff /opt/snow-owl/docker/ /opt/new-snow-owl-release/snow-owl/docker/
Common subdirectories: /opt/snow-owl/docker/configs and /opt/new-snow-owl-release/snow-owl/docker/configs
diff /opt/snow-owl/docker/.env /opt/new-snow-owl-release/snow-owl/docker/.env
10c10
< ELASTICSEARCH_VERSION=7.12.0
---
> ELASTICSEARCH_VERSION=7.16.3
24c24
< SNOWOWL_VERSION=7.18.0
---
> SNOWOWL_VERSION=7.19.0

The changes usually are restricted to version numbers in the .env file. In such cases, it is equally acceptable to overwrite the contents of the ./snow-owl/docker folder as is or cherry-pick the necessary modifications by hand.

Once the new version of the files is in place it is sufficient to just issue the following commands, an explicit stop of the service is not even required (in the folder ./snow-owl/docker):

docker-compose pull
docker-compose up -d

The Authoring Platform release package contains a built-in solution to perform rolling and permanent data backups. The docker stack has a specialized container (called snow-owl-backup) that is responsible for creating scheduled backups of:

• the Elasticsearch indices

• the OpenLDAP database (if present)

• Bugzilla's configuration files and SQL database

For the Elasticsearch indices, the backup container uses the Snapshot API. Snapshots are labeled in a predefined format with timestamps. E.g. snowowl-daily-20220324030001

The OpenLDAP database is backed up by compressing the contents of the folder under ./snow-owl/ldap. Filenames are generated using the name of the corresponding Elasticsearch snapshot. E.g. snowowl-daily-20220324030001.tar.gz.

Bugzilla is backed up by compressing the configuration files and the MySQL database dump. Filenames are generated using the name of the corresponding Elasticsearch snapshot. E.g. snowowl-daily-20220324030001.tar.gz.

Daily backups

Daily backups are rolling backups, scheduled, and cleaned up based on the settings specified in the ./snow-owl/docker/.env file. Here is a summary of the important settings that could be changed.

BACKUP_FOLDER

To store backups redundantly it is advised to mount a remote file share to a local path on the host. By default, this folder is configured to be at ./snow-owl/backup. It contains:

• the snapshot files of the Elasticsearch cluster

• the backup files of the OpenLDAP database

• the backup files of Bugzilla

• extra configuration files

CRON_DAYS, CRON_HOURS, CRON_MINUTES

Backup jobs are scheduled by crond, so cron-expressions can be defined here to specify the time a daily backup should happen.

NUMBER_OF_DAILY_BACKUPS_TO_KEEP

This is used to tell the backup container how many daily backups must be kept.

Example daily backup config

Let's say we have an external file share mounted to /mnt/external_folder. There is a need to create daily backups after each working day, during the night at 2:00 am. Only the last two-weeks-worth of data should be kept (assuming 5 working days each week).

BACKUP_FOLDER=/mnt/external_folder
NUMBER_OF_DAILY_BACKUPS_TO_KEEP=10

CRON_DAYS=Tue-Sat
CRON_HOURS=2
CRON_MINUTES=0

One-off backups

It is also possible to perform backups occasionally, e.g. before versioning an important SNOMED CT release or before a Terminology Server version upgrade. These backups are kept until manually removed.

To create such backups the following command needs to be executed using the backup container's terminal:

root@host:/# docker exec -it backup bash
root@ad36cfb0448c:/# /backup/backup.sh -l my-backup-label

The script will create a snapshot backup of the Elasticsearch data with a label snowowl-my-backup-label-20220405030002,an archive that contains the database of the OpenLDAP server and an archive that contains the configuration and database of Bugzilla with the name snowowl-my-backup-label-20220405030002.tar.gz.

Operating System

The server-side components of the Authoring Platform are recommended to be installed on x86_64 / amd64 Linux operating systems where Docker Engine is available. See the list of supported distributions by Docker:

Here is the list of distributions that we suggest in the order of recommendation:

• CentOS 7

• Ubuntu 20.04 (or 18.04)

• Debian 10 - Buster

Software packages

Before starting the actual deployment of the Authoring Platform make sure that the following packages are installed and configured properly:

• Docker Engine

• docker-compose

• ability to execute bash scripts

Firewall

Regardless of having a reverse proxy, the following port must be opened to let the Snow Owl thick client communicate with the Terminology Server:

• tcp:2036

In case a reverse proxy is used, the Terminology Server requires two ports to be opened either towards the intranet or the internet (depends on usage):

• http:80

• https:443

In case there is no reverse proxy installed, the following port must be opened to be able to access the server's REST API:

• http:8080

The Snow Owl Authoring Platform has two different ways to manage users. The primary authentication and authorization service is the LDAP Directory Server. The secondary option is a file-based database, used only for administrative purposes. Whenever user access has to be granted or revoked the following methods could be applied.

LDAP based identity provider

There are several ways to access and manage an OpenLDAP server, hereby we will only describe one of them, through the Apache Directory Studio.

Apache Directory Studio is an open-source, free application. It is available to download for different platforms (Windows, macOS, and Linux).

The OpenLDAP server uses port 389 for communication. This is the port that needs to be tunneled through the SSH connection. Here is what the final configuration looks like in PuTTY:

Once the SSH tunnel works, it's time to set up our connection in Apache DS. Go to File -> New -> LDAP Connection and set the following:

Hit the "Check Network Parameter" button to verify the network connection.

Go to the next page of the wizard and provide your credentials. The default Bind DN and Bind password can be found in the Authoring Platform release package under ./snow-owl/docker/.env.

Hit the "Check Authentication" button to verify your credentials. Hit Finish to complete the setup procedure.

All users and groups should be browseable now through the LDAP Browser view:

Grant user access

To grant access to a new user an LDAP entry has to be created. Go to the LDAP Browse view and right-click on the organization node, then New -> New Entry:

It is the easiest to use an existing entry as a template:

Leave everything as is on the Object Classes page, then hit Next. Fill in the new user's credentials:

On the final page, double click on the userPassword row and provide the user's password:

Hit Finish to add the user to the database.

Now we need to assign a role for the user. Before going forward, get ahold of the user's DN using the LDAP Browser view:

Select the desired role group in the Browser view and add a new attribute:

Select the attribute type uniqueMember and hit Finish:

Paste the user's DN as the value of the attribute and hit Enter to make your changes permanent:

Revoke user access

To revoke access the user has to be deleted from the list of users:

And also has to be removed from the role group:

Change credentials

To change either the first or last name, or the password of a user, just edit any of the attributes in the user editor:

File-based identity provider

Grant user access

To grant access the users file has to be amended with the new user and its credentials. There are several ways to encrypt a password using the bcrypt algorithm but here is one that is easy and available on most of the Linux variants. The package called htpasswd has to be installed:

It will prompt for the password and will amend the file with the new user at the end.

Revoke user access

Simply remove the user's line from the file and restart the service.

Change credentials

Full list of steps to perform before spinning up the service:

1. Extract the Authoring Platform release archive to a folder. E.g. /opt/snow-owl

2. (Optional) Obtain an SSL certificate

   1. Make sure a DNS A record is routed to the host's public IP address

   2. Go into the folder ./snow-owl/docker/cert

   3. Execute the ./init-certificate.sh script:

   Copy

   ./init-certificate.sh -d snow-owl.example.com

3. (Optional) Configure access for managed Elasticsearch Cluster (elastic.co)

4. (Optional) Extract dataset to ./snow-owl/resources where folder structure should look like ./snow-owl/resources/indexes/nodes/0 at the end.

5. Verify file ownership to be UID=1000 and GID=0:

   Copy

   chmod -R 1000:0 ./snow-owl/docker ./snow-owl/logs ./snow-owl/resources

6. Check any credentials or settings that need to be changed in ./snow-owl/docker/.env

7. Make sure the hostname is configured properly in the environment file. This is important for being able to reach Bugzilla through a web browser.

8. Authenticate with our private docker registry while in the folder ./snow-owl/docker:

   Copy

   cat docker_login.txt | docker login -u <username> --password-stdin https://docker.b2i.sg

9. Issue a pull (in folder ./snow-owl/docker)

   Copy

   docker-compose pull

10. Spin up the service (in the folder ./snow-owl/docker)

    Copy

    docker-compose up -d

11. Verify that the REST API of the Terminology Server is available at:

    1. With SSL: https://snow-owl.example.com/snowowl

    2. Without SSL: http://hostname:8080/snowowl

12. Verify that the server and cluster status is GREEN by querying the following REST API endpoint:

    1. With SSL:

       Copy

       curl https://snow-owl.example.com/snowowl/info

    2. Without SSL:

       Copy

       curl http://hostname:8080/snowowl/info

13. Verify that Bugzilla's web UI is available at:

    1. With SSL:

       Copy

       curl https://snow-owl.example.com/bugzilla

    2. Without SSL:

       Copy

       curl http://hostname:8080/bugzilla

14. Enjoy using the Snow Owl Authoring Platform

The Authoring Platform release package contains a download URL for the Snow Owl thick client.

There are no special requirements to install, it is as easy as extracting the contents of the archive to a folder.

To avoid having issues the following should be considered:

• make sure to extract the files to a folder that has sufficient permissions for the logged-in user

• try to avoid folders that are "too deep" in the file system tree. Certain Windows versions have trouble dealing with long file paths. Do not use "Desktop" or "Program Files" preferably.

Upon the first start of the client make sure the correct remote repository address is configured for the given user: