Skip to content

Service infrastructure

This section describes what is in the SODa SCS Manager deployment and how the services are wired.

For a C4-style Mermaid map (Compose → containers → modules → integrations → configs/scripts), see Architecture overview. Naming scheme: Naming vocabulary. Migration (aliases, Manager settings, env, code): Naming migration plan.

Overview

The deployment is built from multiple Docker Compose files merged via the COMPOSE_FILE environment variable (see example-env). It includes:

  • A main stack (docker-compose.yml): Traefik reverse proxy, shared MariaDB, Portainer, phpMyAdmin, and the Nextcloud FUSE mounter sidecar (nextcloud-mounter).
  • Submodule stacks: SCS Manager (Drupal), Nextcloud, JupyterHub, Keycloak, OpenGDB, WebProtégé, the project website stack, and the SCS Health dashboard.

All services that serve HTTP or need to be reached by Traefik attach to the same Docker network, reverse-proxy. The shared MariaDB instance is used by Keycloak, SCS Manager, Nextcloud, and the project website; each has its own database and user created by pre-install scripts.

Keycloak is the central identity provider (IdP): SCS Manager, Nextcloud, JupyterHub, and WebProtégé use OpenID Connect to authenticate users against Keycloak.

Architecture diagram

flowchart LR
  subgraph internet [Internet]
    Users[Users]
  end

  subgraph traefik [Traefik]
    Web[web :80]
    WebSecure[websecure :443]
  end

  subgraph services [Services]
    Keycloak[Keycloak]
    SCSManager[SCS Manager]
    Nextcloud[Nextcloud]
    JupyterHub[JupyterHub]
    OpenGDB[OpenGDB]
    WebProtege[WebProtégé]
    ProjectPage[Project Website]
  end

  subgraph shared [Shared]
    DB[(MariaDB)]
  end

  Users --> Web
  Users --> WebSecure
  Web --> WebSecure
  WebSecure --> Keycloak
  WebSecure --> SCSManager
  WebSecure --> Nextcloud
  WebSecure --> JupyterHub
  WebSecure --> OpenGDB
  WebSecure --> WebProtege
  WebSecure --> ProjectPage

  Keycloak --> DB
  SCSManager --> DB
  Nextcloud --> DB
  ProjectPage --> DB

  SCSManager -.->|OIDC| Keycloak
  Nextcloud -.->|OIDC| Keycloak
  JupyterHub -.->|OIDC| Keycloak
  WebProtege -.->|OIDC| Keycloak

Main stack (docker-compose.yml)

  • core--traefik--edge (alias scs--reverse-proxy) — Traefik: HTTP/HTTPS entrypoints, TLS (e.g. Let’s Encrypt), middlewares (HTTPS redirect, Nextcloud headers, rate limiting). Routes by Host(...) using Docker labels. The Docker network remains reverse-proxy.
  • core--mariadb--db (alias scs--database) — MariaDB. Single instance; databases and users for Keycloak, SCS Manager, Nextcloud, and project website are created by pre-install scripts.
  • core--portainer--app (alias scs--portainer) — Portainer CE for container management; used by SCS Manager for WissKI stack operations.
  • dbms--phpmyadmin--app (alias scs--phpmyadmin) — phpMyAdmin for database access (optional).
  • dbms--forwardauth--proxy (alias scs--forward-auth) — Keycloak forward-auth for phpMyAdmin SSO.
  • nextcloud-mounter — rclone rcd sidecar for per-user Nextcloud WebDAV/FUSE mounts. See Nextcloud mount sidecar.

Compose file aggregation

The COMPOSE_FILE in example-env lists, in order:

  • Main: docker-compose.yml
  • Stacks: scs-manager-stack/docker-compose.yml, scs-nextcloud-stack/docker-compose.yml, jupyterhub/docker-compose.yml, keycloak/docker-compose.yml, open_gdb/docker-compose.yml, webprotege/docker-compose.yml, scs-project-website-stack/docker-compose.yml, scs-health/docker-compose.yml
  • Overrides: the corresponding docker-compose.override.yml files for each stack.

Override files are copied from 00_custom_configs/<stack>/docker/ to each stack directory by start.sh (see Pre-start steps).

Stacks and wiring

Traefik

  • Listens on ports 80 (HTTP) and 443 (HTTPS).
  • Uses Let’s Encrypt for TLS (certificate resolver le).
  • Middlewares: rate-limit, rate-limit-high, rate-limit-strict, and redirects (e.g. SCS Manager second domain).
  • Drupal-facing services (SCS Manager, WissKI instances via Portainer, project website) use rate-limit-high@docker on external routers with an internal bypass router for Docker CIDRs. See WissKI stack — bot protection.
  • Each service that should be reachable gets Traefik labels for router rule, entrypoints, TLS, and service port.

Database

  • core--mariadb--db (MariaDB, alias scs--database) is started first by start.sh. Pre-install scripts create:
  • Keycloak DB and user
  • SCS Manager DB and user
  • Nextcloud DB and user
  • Project website DB and user

Keycloak

  • Realm is generated from 00_custom_configs/keycloak/templates/realm/scs-realm.json.tpl (pre-install) and imported at startup.
  • Clients: JupyterHub (by JUPYTERHUB_DOMAIN), Nextcloud (by NEXTCLOUD_NEXTCLOUD_DOMAIN), SCS Manager (by SCS_MANAGER_DOMAIN), phpMyAdmin/DBMS (by SCS_DBMS_DOMAIN), WebProtégé, DIDMOS. Each has a client secret from .env.
  • Uses the shared MariaDB for persistence.
  • Exposed via Traefik on the Keycloak domain (e.g. auth.scs.localhost).

SCS Manager

  • Drupal application (and optionally Varnish in front). Image and mode from env (e.g. scs-manager-image-development:latest).
  • OpenID Connect client config is generated from 00_custom_configs/scs-manager-stack/openid/openid_connect.client.scs_sso.yml.tpl into scs-manager-stack/custom_configs/openid_connect.client.scs_sso.yml during pre-install. Drupal uses this for Keycloak SSO.
  • Drupal settings (e.g. OpenID behaviour, reverse proxy) come from 00_custom_configs/scs-manager-stack/drupal/ (e.g. openid-connect.settings.php, reverse-proxy.settings.php).
  • Nginx / clean URLs: scs-manager-stack/configs/nginx/drupal.conf is mounted into scs-manager--drupal so Drupal generates paths without /index.php. See SCS Manager clean URLs if links still show index.php.
  • PHP / OPcache: Production must not mount scs-manager-stack/configs/php/zz-xdebug-debug.ini (that overlay disables OPcache and enables Xdebug). For local debugging, add the volume mount back in a dev-only compose override; see PHP debugging (Xdebug).
  • User lifecycle (create/delete): SCS Manager user create and delete routines — Keycloak registration, OIDC linking, Nextcloud credential attributes, and hook_user_delete cleanup.
  • Nextcloud mounts: On nextcloud-mounter-net; owns mount lifecycle via NextcloudMountManager / Drush scs:nextcloud-*. See Nextcloud mount sidecar.
  • WissKI stacks are deployed via Portainer from wisski-base-stack. Drupal reverse proxy addresses in SCS Manager settings (auto by default) become DRUPAL_PROXY_ADDRESSES on the stack; the wisski-base-image syncs trusted proxy CIDRs into each instance settings.php on boot.
  • Varnish VCL is generated from 00_custom_configs/scs-manager-stack/varnish/default.vcl.tpl during pre-install.
  • Connects to shared MariaDB (own database).

Nextcloud

  • Nextcloud FPM, reverse proxy (nginx), Redis, OnlyOffice. Database and user created in pre-install.
  • Post-installation hooks in scs-nextcloud-stack/hooks/post-installation/ install and configure apps (e.g. Social Login, OnlyOffice, Draw.io) and trusted domains. These must be run when the stack is up (see Post-configuration).

JupyterHub

  • Generic OAuthenticator pointing at Keycloak (client ID, secret, authorize/token/userinfo URLs from env).
  • Group and GID handling in jupyterhub/jupyterhub/jupyterhub_config.py: auth_state_groups_key reads groups and gids from the OAuth userinfo; spawner uses group_map.json and gids for Linux group membership in the notebook container.
  • Spawner binds Keycloak project Team Folders only (${NEXTCLOUD_MOUNTS_ROOT}/<user>/<project-label> → /home/jovyan/nextcloud/<project-label>, rslave, GID 33). See Nextcloud mount sidecar.
  • Spawner image can include tools (e.g. wisski_py, OpenRefine). JupyterHub needs the Keycloak “groups” (and optionally “gids”) scope on its client for group-based access and spawner gids to work (see Post-configuration checklist).

WebProtégé

  • Shared instance (webprotege + webprotege-mongodb), not a container per SCS project. Image webprotege:scs is built from the webprotege/ submodule (00_custom_configs/webprotege/docker/docker-compose.override.yml).
  • Deltas vs protegeproject/webprotege (not a fork) live in 00_custom_configs/webprotege/patches/README.md: OIDC SSO, project REST (owner permissions, collaborators, trash), and the Docker build.
  • Users sign in with Keycloak OIDC (preferred_username, sanitised to [a-z0-9_.-]). SCS Manager stores the ontology-project UUID on the Drupal project field webprotegeProjectId.
  • Creating an SCS project calls POST /data/projects with Authorization: apikey <key> and then grants EDIT to members (PUT /data/projects/{id}/collaborators/{user}). The project-page WebProtégé card deep-links to #projects/{uuid}/perspectives/69df8fa8-4f84-499e-9341-28eb5085c40b. The dashboard card stays on #projects/list.
  • API key (once per environment): rebuild/recreate webprotege after Java changes, then generate a key for a user with CREATE_EMPTY_PROJECT:
docker compose up -d --force-recreate --build webprotege--webprotege--app
docker exec -it webprotege--webprotege--app java -jar /webprotege-cli.jar generate-api-key

Store the key in SCS Manager settings (WebProtégé → API key), not in git. Optional API base URL http://webprotege:8080 for Docker-internal REST (Today alias until soak; Target http://webprotege--webprotege--app:8080). Run drush updb for field webprotegeProjectId (update 11028) and rebuild caches. Then backfill existing SCS projects (owners + member EDIT):

scs-drush soda_scs_manager:backfill-webprotege-projects --dry-run --all
scs-drush soda_scs_manager:backfill-webprotege-projects --all

The same Drush command is the migration on scs-prime once that environment has the API key and update 11028.

OpenGDB

  • RDF4J (opengdb--rdf4j--db, alias scs--rdf4j), AuthProxy (opengdb--authproxy--proxy, alias scs--authproxy), OutProxy, nginx (opengdb--nginx--edge). Internal SPARQL from WissKI still uses the alias until cutover.

Project website (scs-project-website-stack)

  • Drupal plus Varnish and Redis. Database and user created by pre-install; Varnish VCL is generated from a template.
  • Note: The pre-install script lives in 01_scripts/scs-project-page/ and references 00_custom_configs/scs-project-website/varnish/default.vcl.tpl in the script, but the template file is under 00_custom_configs/scs-project-page/varnish/default.vcl.tpl. If the script fails, use the path under scs-project-page. start.sh lists the script as 01_scripts/scs-project-website/pre-install.bash; the actual directory is scs-project-page. These path/naming inconsistencies may need to be fixed in a separate change.

SCS Health (scs-health)

  • Next.js health dashboard for operators: HTTPS checks against stack service domains and optional Docker container status via the host socket.
  • Container scs--health, submodule at scs-health/, Traefik host SCS_HEALTH_DOMAIN.
  • Auth: local credentials (SCS_HEALTH_AUTH_*) and optional Keycloak OIDC (SCS_HEALTH_OIDC_*). UI follows SODa corporate design (Inter, orange #e68500, menu-bar divider) aligned with keycloak/keycloak/themes/scs and soda_scs_manager_theme. See SCS Health dashboard.

WissKI stacks (per-instance, via Portainer)

WissKI is not merged into the root COMPOSE_FILE. Each catalogue instance is a separate stack (wisski-base-stack) managed by SCS Manager through Portainer.

  • Infrastructure: WissKI stack — Traefik → Varnish → Nginx + PHP-FPM, networks, headers, raw.* bypass
  • Nextcloud: Project Team Folder via sidecar (NEXTCLOUD_MOUNT_MODE=external) — WissKI Nextcloud mount and sidecar
  • Drupal settings.php snippets: configs/ — trusted hosts, private files, Redis, reverse proxy