Contribute a service

This page describes how to contribute a language technology service to run on the cloud platform of the European Language Grid.

Currently, ELG supports the integration of tools/services that fall into one of the following broad categories:

  • Information Extraction (IE) : Services that take text and annotate it with metadata on specific segments, e.g. Named Entity Recognition (NER), the task of extracting persons, locations, and organisations from a given text.

  • Text Classification (TC) : Services that take text and return a classification for the given text from a finite set of classes, e.g. Text Categorization which is the task of categorizing text into (usually labelled) organized categories.

  • Machine Translation (MT) : Services that take text in one language and translate it into text in another language, possibly with additional metadata associated with each segment (sentence, phrase, etc.).

  • Automatic Speech Recognition (ASR) : Services that take audio as input and produce text (e.g., a transcription) as output, possibly with metadata associated with each segment.

  • Text-to-Speech Generation (TTS) : Services that take text as input and produce audio as output.

Overview: how an LT Service is integrated to ELG

An overview of the ELG platform is depicted below.

Platform overview

The following bullets summarize how LT services are deployed and invoked in ELG.

  • All LT Services (as well as all the other ELG components) are deployed (run as containers) on a Kubernetes (k8s) cluster; k8s is a system for automating deployment, scaling, and management of containerised applications.

  • All LT Services are integrated into ELG via the LT Service Execution Orchestrator/Server. This server exposes a common public REST API (Representational state transfer) used for invoking any of the deployed backend LT Services. The public API is used from ELG’s Test/Trial UIs that are embedded in the ELT Catalogue; however, it can also be invoked from the command line or any programming language; see Use an LT service section for more information. Some of the HTTP endpoints that are offered in the API are given below; for more information see Public LT API specification.

Endpoint

Type

Consumes

Produces

https://{domain}/execution/processText/{ltServiceID}

POST

‘application/json’

‘application/json’

https://{domain}/execution/processText/{ltServiceID}

POST

‘text/plain’ or ‘text/html’

‘application/json’

https://{domain}/execution/processAudio/{ltServiceID}

POST

‘audio/x-wav’ or ‘audio/wav’

‘application/json’

https://{domain}/execution/processAudio/{ltServiceID}

POST

‘audio/mpeg’

‘application/json’

{domain} is ‘live.european-language-grid.eu’ and {ltServiceID} is the ID of the backend LT service. This ID is assigned/configured during registration; see section Step 2: register the service with the platform (‘LT Service is deployed to ELG and configured’ step).

Note

The REST API that is exposed from an LT Service X (see previous section) is for the communication between LT Service Execution Orchestrator Server and X (ELG Internal LT API).

  • When LT Service Execution Orchestrator receives a processing request for service X, it retrieves from the database X’s k8s REST endpoint and sends a request to it. This endpoint is configured/specified during the registration process; see section Step 2: register the service with the platform (‘LT Service is deployed to ELG and configured’ step). When the Orchestrator gets the response from the LT Service, it returns it to the application/client that sent the initial call.

Before you start

  • Please make sure that the service you want to contribute complies with our terms of use.

  • Please make sure you have registered and been assigned the provider role.

  • Please make sure that your service meets the technical requirements below, and choose one of the three integration options.

Technical requirements and integration options

The requirements for integrating an LT tool/service are the following:

Expose an ELG compatible endpoint: You MUST create an application that exposes an HTTP endpoint for the provided LT tool(s). The application MUST consume (via the aforementioned HTTP endpoint) requests that follow the ELG JSON format, call the underlying LT tool and produce responses again in the ELG JSON format. For a detailed description of the JSON-based HTTP protocol (ELG Internal LT API) that you have to implement, see the Internal LT API specification annex.

Dockerisation: You MUST dockerise the application and upload the respective image(s) in a Docker Registry, such as GitLab, DockerHub, Azure Container Registry etc. You MAY select out of the three following options, the one that best fits your needs:

  • LT tools packaged in one standalone image: One docker image is created that contains the application that exposes the ELG-compatible endpoint and the actual LT tool.

  • LT tools running remotely outside the ELG infrastructure: For these tools, one proxy image is created that exposes one (or more) ELG-compatible endpoints; the proxy container communicates with the actual LT service that runs outside the ELG infrastructure.

  • LT tools requiring an adapter: For tools that already offer an image that exposes a non-ELG compatible endpoint (HTTP-based or other), a second adapter image SHOULD be created that exposes an ELG-compatible endpoint and acts as proxy to the container that hosts the actual LT tool.

In the following diagram the three different options for integrating a LT tool are shown:

Integration options

Step 0: Dockerize your service

Build/Store Docker images

Ideally, the source code of your LT tool/service already resides on GitLab where a built-in Continuous Integration (CI) Runner can take care of building the image; GitLab also offers a container registry that can be used for storing the built image. For this, you need to add on the root level of your GitLab repository a .gitlab-ci.yml file as well as a Dockerfile; i.e, the recipe for building the image. Here you can find an example. After each new commit, the CI Runner is automatically triggered and runs the CI pipeline that is defined in .gitlab-ci.yml. You can see the progress of the pipeline on the respective page in GitLab UI (“CI / CD -> Jobs”); also when it completes successfully, you can find the image at “Packages -> Container Registry”.

Your image can also be built and tagged in your machine by running the docker build command. Then it can be uploaded (with docker push) to GitLab registry, in DockerHub (which is a public Docker registry) or any other Docker registry.

E.g for this GitLab hosted project the commands would be:

docker login registry.gitlab.com

For logging in and be allowed to push an image.

docker build -t registry.gitlab.com/european-language-grid/dfki/elg-jtok

For building an image (locally) for the project. Before running docker build you have to download (clone) a copy of the project and be in the top-level directory (elg-jtok).

docker push registry.gitlab.com/european-language-grid/dfki/elg-jtok

For pushing the image to GitLab.

In the following links you can find some more inforrmation on docker commands plus some examples:

Dockerization of a Python-based LT service/tool

From Python script to Service

First you need to ensure that your python script provides either a std i/o messaging or a RESTful API.

A ) create a REST API using Flask

Example of a Shakespeare Bot

from flask import Flask
from flask import render_template
from flask import request

# creates a Flask application, named app
app = Flask(__name__)

from chatterbot import ChatBot

# a route where we will display a welcome message via an HTML template
@app.route("/", methods=['POST','GET'])
def hello():
    if request.method == 'POST':  # this block is only entered when the form is submitted
        user_message = request.form['user_message']
        chatbot = ChatBot("Frank")
        bot_message = chatbot.get_response(user_message).text

        data = {
            'bot_message': bot_message,
            'user_message': user_message,
            'user_message_visibility': '',
        }
        return render_template('index.html', **data)

    data = {
        'bot_message': "Speak. I am bound to hear.",
        'user_message': '',
        'user_message_visibility': 'style=visibility:hidden;',
    }
    return render_template('index.html', **data)

# run the application
if __name__ == "__main__":
    app.run(debug=True)

B ) create a std i/o messaging

In Python, this can easily be done via print.

Note

Example to be added soon.

From Python Service to Docker image

Then you can make your service a Docker image by taking the following steps:

  • choose python environment, e.g. python:3.6.4-slim-jessie (see also: Official Docker images)

  • add/copy python scripts

  • add/copy other resources

  • install missing modules

  • define entrypoint

We provide you with some dockerfile examples to see its simplicity:

Example 1: Shakespeare Bot

from python:3.6.4-slim-jessie

COPY shakespearebot.py .
COPY corpora/hamlet.csv corpora/

RUN pip install pandas
RUN pip install chatterbot
RUN pip install chatterbot-corpus

ENTRYPOINT ["python","shakespearebot.py"]

Example 2: Legal Entity Recognition, install all requirements from .txt-file

FROM python:3.7

COPY . .

RUN pip install -r requirements.txt

EXPOSE 8080
ENTRYPOINT ["python", "ler-ws.py"]

Dockerization of a Java-based tool

A Spring Boot starter to make it as easy as possible to create ELG-compliant tools in Java using is provided at: ELG Spring Boot Starter

Step 1: create metadata

The first step is to describe your software using ELG’s metadata format, ELG-SHARE. Future releases of ELG will include an interactive editor for this. However, for now, you must create an XML file. Refer to the examples below for how to do this.

The elements you need are documented on the following pages:

For more information about ELG-SHARE, see:

At the ELG GitLab, you will find templates (that you can use to create new metadata records) and examples in XML format.

Example 1: information extraction service

ANNIE’s Named Entity Recognizer published at: https://live.european-language-grid.eu/catalogue/#/resource/service/tool/512

<?xml version="1.0" encoding="UTF-8"?>
<ms:MetadataRecord xsi:schemaLocation="http://w3id.org/meta-share/meta-share/ ../../../Schema/ELG-SHARE.xsd" xmlns:ms="http://w3id.org/meta-share/meta-share/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
        <ms:MetadataRecordIdentifier ms:MetadataRecordIdentifierScheme="http://w3id.org/meta-share/meta-share/elg">default id</ms:MetadataRecordIdentifier>
        <ms:metadataCreationDate>2020-02-25</ms:metadataCreationDate>
        <ms:metadataLastDateUpdated>2020-02-25</ms:metadataLastDateUpdated>
        <ms:metadataCurator>
                <ms:actorType>Person</ms:actorType>
                <ms:surname xml:lang="en">Roberts</ms:surname>
                <ms:givenName xml:lang="en">Ian</ms:givenName>
                <ms:email>username1@somedomain.com</ms:email>
        </ms:metadataCurator>
        <ms:compliesWith>http://w3id.org/meta-share/meta-share/ELG-SHARE</ms:compliesWith>
        <ms:metadataCreator>
                <ms:actorType>Person</ms:actorType>
                <ms:surname xml:lang="en">Roberts</ms:surname>
                <ms:givenName xml:lang="en">Ian</ms:givenName>
                <ms:email>username2@somedomain.com</ms:email>
        </ms:metadataCreator>
        <ms:DescribedEntity>
                <ms:LanguageResource>
                        <ms:entityType>LanguageResource</ms:entityType>
                        <ms:resourceName xml:lang="en">GATE: English Named Entity Recognizer</ms:resourceName>
                        <ms:resourceShortName xml:lang="en">annie-named-entity-recognizer</ms:resourceShortName>
                        <ms:description xml:lang="en">Identify names of &lt;em&gt;persons&lt;/em&gt;, &lt;em&gt;locations&lt;/em&gt;, &lt;em&gt;organizations&lt;/em&gt;, as well as &lt;em&gt;money amounts&lt;/em&gt;, &lt;em&gt;time and date expressions&lt;/em&gt; in English texts automatically. </ms:description>
                        <ms:LRIdentifier ms:LRIdentifierScheme="http://w3id.org/meta-share/meta-share/elg">ELG id automatically assigned</ms:LRIdentifier>
                        <ms:version>v8.6</ms:version>
                        <ms:additionalInfo>
                                <ms:landingPage>https://cloud.gate.ac.uk/shopfront/displayItem/annie-named-entity-recognizer</ms:landingPage>
                        </ms:additionalInfo>
                        <ms:keyword xml:lang="en">Named Entity Recognition</ms:keyword>
                        <ms:keyword xml:lang="en">English</ms:keyword>
                        <ms:resourceProvider>
                                <ms:Group>
                                        <ms:actorType>Group</ms:actorType>
                                        <ms:organizationName xml:lang="en">GATE Team, University of Sheffield</ms:organizationName>
                                        <ms:website>https://gate.ac.uk/</ms:website>
                                </ms:Group>
                        </ms:resourceProvider>
                        <ms:publicationDate>2020-02-25</ms:publicationDate>
                        <ms:resourceCreator>
                                <ms:Person>
                                        <ms:actorType>Person</ms:actorType>
                                        <ms:surname xml:lang="en">Roberts</ms:surname>
                                        <ms:givenName xml:lang="en">Ian</ms:givenName>
                                        <ms:email>username3@somedomain.com</ms:email>
                                </ms:Person>
                        </ms:resourceCreator>
                        <ms:intendedApplication>
                                <ms:LTClassRecommended>http://w3id.org/meta-share/omtd-share/NamedEntityRecognition</ms:LTClassRecommended>
                        </ms:intendedApplication>
                        <ms:LRSubclass>
                                <ms:ToolService>
                                        <ms:lrType>ToolService</ms:lrType>
                                        <ms:function>
                                                <ms:LTClassRecommended>http://w3id.org/meta-share/omtd-share/NamedEntityRecognition</ms:LTClassRecommended>
                                        </ms:function>
                                        <ms:SoftwareDistribution>
                                                <ms:SoftwareDistributionForm>http://w3id.org/meta-share/meta-share/dockerImage</ms:SoftwareDistributionForm>
                                                <ms:executionLocation>http://localhost:8080/process</ms:executionLocation>
                                                <ms:dockerDownloadLocation>registry.gitlab.com/european-language-grid/usfd/gate-ie-tools/annie:8.6-0.0.3</ms:dockerDownloadLocation>
                                                <ms:licenceTerms>
                                                        <ms:licenceTermsName xml:lang="en">GNU Lesser General Public License v3.0 only</ms:licenceTermsName>
                                                        <ms:licenceTermsURL>https://spdx.org/licenses/LGPL-3.0-only.html</ms:licenceTermsURL>
                                                        <ms:LicenceIdentifier ms:LicenceIdentifierScheme="http://w3id.org/meta-share/meta-share/SPDX">LGPL-3.0-only</ms:LicenceIdentifier>
                                                        <ms:LicenceIdentifier ms:LicenceIdentifierScheme="http://w3id.org/meta-share/meta-share/elg">ELG-ENT-LIC-270220-00000199</ms:LicenceIdentifier>
                                                </ms:licenceTerms>
                                        </ms:SoftwareDistribution>
                                        <ms:languageDependent>true</ms:languageDependent>
                                        <ms:inputContentResource>
                                                <ms:processingResourceType>http://w3id.org/meta-share/meta-share/file1</ms:processingResourceType>
                                                <ms:language>
                                                        <ms:languageTag>en</ms:languageTag> <ms:languageId>en</ms:languageId>
                                                </ms:language>
                                                <ms:mediaType>http://w3id.org/meta-share/meta-share/text</ms:mediaType>
                                                <ms:dataFormat>http://w3id.org/meta-share/omtd-share/Json</ms:dataFormat>
                                                <ms:characterEncoding>http://w3id.org/meta-share/meta-share/UTF-8</ms:characterEncoding>
                                        </ms:inputContentResource>
                                        <ms:outputResource>
                                                <ms:processingResourceType>http://w3id.org/meta-share/meta-share/file1</ms:processingResourceType>
                                                <ms:language>
                                                        <ms:languageTag>en</ms:languageTag> <ms:languageId>en</ms:languageId>
                                                </ms:language>
                                                <ms:mediaType>http://w3id.org/meta-share/meta-share/text</ms:mediaType>
                                                <ms:dataFormat>http://w3id.org/meta-share/omtd-share/Json</ms:dataFormat>
                                                <ms:characterEncoding>http://w3id.org/meta-share/meta-share/UTF-8</ms:characterEncoding>
                                                <!-- annotations: :Address, :Date, :Location, :Organization, :Person, :Money, :Percent, :Token, :SpaceToken, :Sentence -->
                                                <ms:annotationType>http://w3id.org/meta-share/omtd-share/Person</ms:annotationType>
                                                <ms:annotationType>http://w3id.org/meta-share/omtd-share/Location</ms:annotationType>
                                                <ms:annotationType>http://w3id.org/meta-share/omtd-share/Organization</ms:annotationType>
                                                <ms:annotationType>http://w3id.org/meta-share/omtd-share/Date</ms:annotationType>
                                        </ms:outputResource>
                                        <ms:trl>http://w3id.org/meta-share/meta-share/trl4</ms:trl>
                                        <ms:evaluated>false</ms:evaluated>
                                </ms:ToolService>
                        </ms:LRSubclass>
                </ms:LanguageResource>
        </ms:DescribedEntity>
</ms:MetadataRecord>

The Docker image for this service is stored at the GitLab registry.

Example 2: machine translation service

Edinburgh’s German to English engine published at: https://live.european-language-grid.eu/catalogue/#/resource/service/tool/623

<?xml version="1.0" encoding="UTF-8"?>
<ms:MetadataRecord xmlns:ms="http://w3id.org/meta-share/meta-share/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://w3id.org/meta-share/meta-share/ ../../Schema/ELG-SHARE.xsd">
        <ms:MetadataRecordIdentifier ms:MetadataRecordIdentifierScheme="http://w3id.org/meta-share/meta-share/elg">default id</ms:MetadataRecordIdentifier>
        <ms:metadataCreationDate>2020-02-28</ms:metadataCreationDate>
        <ms:metadataLastDateUpdated>2020-02-28</ms:metadataLastDateUpdated>
        <ms:metadataCurator>
                <ms:actorType>Person</ms:actorType>
                <ms:surname xml:lang="en">Germann</ms:surname>
                <ms:givenName xml:lang="en">Ulrich</ms:givenName>
                <ms:PersonalIdentifier ms:PersonalIdentifierScheme="http://w3id.org/meta-share/meta-share/elg">ELG-ENT-PER-050320-00000787</ms:PersonalIdentifier>
                <ms:email>user@somedomain.uk</ms:email>
        </ms:metadataCurator>
        <ms:compliesWith>http://w3id.org/meta-share/meta-share/ELG-SHARE</ms:compliesWith>
        <ms:metadataCreator>
                <ms:actorType>Person</ms:actorType>
                <ms:surname xml:lang="en">Germann</ms:surname>
                <ms:givenName xml:lang="en">Ulrich</ms:givenName>
                <ms:PersonalIdentifier ms:PersonalIdentifierScheme="http://w3id.org/meta-share/meta-share/elg">ELG-ENT-PER-050320-00000787</ms:PersonalIdentifier>
                <ms:email>user@somedomain.uk</ms:email>
        </ms:metadataCreator>
        <ms:DescribedEntity>
                <ms:LanguageResource>
                        <ms:entityType>LanguageResource</ms:entityType>
                        <ms:resourceName xml:lang="en">UEDIN Machine Translation Service for German to English</ms:resourceName>
                        <ms:resourceShortName xml:lang="en">UEDIN-MT-DeEn</ms:resourceShortName>
                        <ms:description xml:lang="en">A machine translation (MT) service for German-to-English translation based on the Marian machine translation framework. The translation model is a basic transformer model trained on ca 13.3M sentence pairs using Marian NMT</ms:description>
                        <ms:LRIdentifier ms:LRIdentifierScheme="http://w3id.org/meta-share/meta-share/elg">ELG id automatically assigned</ms:LRIdentifier>
                        <ms:version>v1.0.0</ms:version>
                        <ms:additionalInfo>
                                <ms:email>user@somedomain.uk</ms:email>
                        </ms:additionalInfo>
                        <ms:keyword xml:lang="en">Machine Translation</ms:keyword>
                        <ms:keyword xml:lang="en">German</ms:keyword>
                        <ms:keyword xml:lang="en">English</ms:keyword>
                        <ms:keyword xml:lang="en">Neural machine translation</ms:keyword>
                        <ms:keyword xml:lang="en">Marian framework</ms:keyword>
                        <ms:resourceProvider>
                                <ms:Organization>
                                        <ms:actorType>Organization</ms:actorType>
                                        <ms:organizationName xml:lang="en">UEDIN</ms:organizationName>
                                        <ms:OrganizationIdentifier ms:OrganizationIdentifierScheme="http://w3id.org/meta-share/meta-share/elg">ELG-ENT-ORG-280220-00000397</ms:OrganizationIdentifier>
                                        <ms:website>https://www.ed.ac.uk/informatics/</ms:website>
                                </ms:Organization>
                        </ms:resourceProvider>
                        <ms:publicationDate>2020-02-28</ms:publicationDate>
                        <ms:resourceCreator>
                                <ms:Person>
                                        <ms:actorType>Person</ms:actorType>
                                        <ms:surname xml:lang="en">Germann</ms:surname>
                                        <ms:givenName xml:lang="en">Ulrich</ms:givenName>
                                        <ms:PersonalIdentifier ms:PersonalIdentifierScheme="http://w3id.org/meta-share/meta-share/elg">ELG-ENT-PER-050320-00000787</ms:PersonalIdentifier>
                                        <ms:email>user@somedomain.uk</ms:email>
                                </ms:Person>
                        </ms:resourceCreator>
                        <ms:intendedApplication>
                                <ms:LTClassRecommended>http://w3id.org/meta-share/omtd-share/MachineTranslation</ms:LTClassRecommended>
                        </ms:intendedApplication>
                        <ms:LRSubclass>
                                <ms:ToolService>
                                        <ms:lrType>ToolService</ms:lrType>
                                        <ms:function>
                                                <ms:LTClassRecommended>http://w3id.org/meta-share/omtd-share/MachineTranslation</ms:LTClassRecommended>
                                        </ms:function>
                                        <ms:SoftwareDistribution>
                                                <ms:SoftwareDistributionForm>http://w3id.org/meta-share/meta-share/dockerImage</ms:SoftwareDistributionForm>
                                                <ms:executionLocation>http://localhost:18080/api/elg/v1</ms:executionLocation>
                                                <ms:dockerDownloadLocation>mt4elg-de-en</ms:dockerDownloadLocation>
                                                <ms:additionalHWRequirements>limits_memory: 2048Mi limits_cpu: 1.5</ms:additionalHWRequirements>
                                                <ms:licenceTerms>
                                                        <ms:licenceTermsName xml:lang="en">CC BY-SA 4.0</ms:licenceTermsName>
                                                        <ms:licenceTermsURL>https://creativecommons.org/licenses/by-sa/4.0/</ms:licenceTermsURL>
                                                        <ms:LicenceIdentifier ms:LicenceIdentifierScheme="http://w3id.org/meta-share/meta-share/elg">ELG-ENT-LIC-270220-00000097</ms:LicenceIdentifier>
                                                </ms:licenceTerms>
                                        </ms:SoftwareDistribution>
                                        <ms:languageDependent>true</ms:languageDependent>
                                        <ms:inputContentResource>
                                                <ms:processingResourceType>http://w3id.org/meta-share/meta-share/file1</ms:processingResourceType>
                                                <ms:language>
                                                        <ms:languageTag>de</ms:languageTag>
                                                        <ms:languageId>de</ms:languageId>
                                                </ms:language>
                                                <ms:mediaType>http://w3id.org/meta-share/meta-share/text</ms:mediaType>
                                                <ms:dataFormat>http://w3id.org/meta-share/omtd-share/Json</ms:dataFormat>
                                                <ms:characterEncoding>http://w3id.org/meta-share/meta-share/UTF-8</ms:characterEncoding>
                                        </ms:inputContentResource>
                                        <ms:outputResource>
                                                <ms:processingResourceType>http://w3id.org/meta-share/meta-share/file1</ms:processingResourceType>
                                                <ms:language>
                                                        <ms:languageTag>en</ms:languageTag>
                                                        <ms:languageId>en</ms:languageId>
                                                </ms:language>
                                                <ms:mediaType>http://w3id.org/meta-share/meta-share/text</ms:mediaType>
                                                <ms:dataFormat>http://w3id.org/meta-share/omtd-share/Json</ms:dataFormat>
                                                <ms:characterEncoding>http://w3id.org/meta-share/meta-share/UTF-8</ms:characterEncoding>
                                        </ms:outputResource>
                                        <ms:trl>http://w3id.org/meta-share/meta-share/trl4</ms:trl>
                                        <ms:evaluated>false</ms:evaluated>
                                </ms:ToolService>
                        </ms:LRSubclass>
                </ms:LanguageResource>
        </ms:DescribedEntity>
</ms:MetadataRecord>

The Docker image for this service is stored at DockerHub.

Step 2: register the service with the platform

Sign in to ELG, click on the Dashboard and select the “Create item” option as below:

Create item

On the item type selection page, click on “Upload” as below:

Upload icon

Now upload the file you created in Step 1:

Upload metadata XML

If there are any errors in your XML file, these will be shown to you. Fix them and try the upload again. Eventually, a success message will be shown to you and the metadata will be imported into the database.

Step 3: submit for publication

Your metadata record is now accessible via the “my items” page. We will soon make available an interactive editor where you will be able to view and edit the uploaded metadata. For now, on this page, you can select to view it and submit for publication

Actions on my items

Step 4: wait for approval

  • LT Service is assigned to a validator: The service is assigned to a validator (from the ELG team) that will check mainly the metadata and technical compliance of the service; during the validation process, the metadata record is visible only to you (LT provider) and the validator.

LT Service under validation.
  • LT Service is deployed to ELG and configured: The LT service is deployed (by the validator) to the k8s cluster by creating the appropriate configuration YAML file and uploading to the respective GitLab repository. The CI/CD pipeline that is responsible for deployments will automatically install the new service at the k8s cluster. If you request it, a separate dedicated k8s namespace can be created for the LT service before creating the YAML file. The validator of the service assigns to it:

    • the k8s REST endpoint that will be used for invoking it. The endpoint follows this template: http://{k8s service name for the registered LT tool}.{k8s namespace for the registered LT tool}.svc.cluster.local{the path where the REST service is running at}. The {the path where the REST service is running at} part can be found in the executionLocation field in the metadata. For instance, for the Edinburgh’s MT tool above it is ‘/api/elg/v1’.

    • An ID that will be used to call it.

    • Which “try out” UI will be used for testing it and visualizing the returned results.

  • LT Service is tested: On the LT landing page, there is a “Try out” tab and a “Code samples” tab; both can be used to test the service with some input; see Use an LT service section. The validator can help you identify integration issues and resolve them. This process is continued until the LT service is correctly integrated to the platform. The procedure may require access to the k8s cluster for the validator (e.g., to check containers start-up/failures, logs, etc.).

Try out UI
  • LT Service is published: When the LT service works as expected, the validator will approve it; the metadata record is then published and visible to all ELG users through the catalogue.

Frequently asked questions

Question: What is a k8s namespace and when should an LT Provider ask for one?
Answer: A k8s namespace is a virtual sub-cluster, which can be used to restrict access to the respective containers that run within it. You should ask for a dedicated namespace (in ELG k8s cluster) when you need to ensure isolation and security; i.e, limit access to your container, logs etc.
Question: The image that I have created is not publicly available. Is it possible to register it to the ELG platform?
Answer: Yes, it can be registered. A k8s secret containing the required credentials will be created for the namespace in which your image is going to be deployed. k8s will then be able to pull the image and deploy it.
Question: Are there any requirements for executionLocation? For example, an IE tool has to expose a specific path or use a specific port?
Answer: No, you can use any valid port or path. This holds for any kind of LT tool (IE, MT, ASR, etc.). The internal container port will be mapped (via port mapping) to port 80. Remember that the endpoint of the LT service follows this pattern: http://{k8s service name for the registered LT tool}.{k8s namespace for the registered LT tool}.svc.cluster.local{the path where the REST service is running at}, which assumes that the service is exposed to port 80.
Question: I have n different versions of the same IE LT tool; e.g., one version per language. How should I register them to the platform? I have to create one Docker image with all the different versions or one image per version?
Answer: Both are possible. In both cases you will have to provide a separate metadata record for each LT tool. However, in the case where the tools are packaged together, all metadata records must point to the same image location (dockerDownloadLocation) and each of them has to listen in a different HTTP endpoint (executionLocation) but on the same port (for simplicity). E.g, “http://localhost:8080/NamedEntityRecognitionEN”, “http://localhost:8080/NamedEntityRecognitionDE”.
Question: Should the Docker image that I will provide have a specific tag?
Answer: The images that are stored in GitLab or DockerHub are not immutable, even when they have been assigned a specific/custom tag; thus, it is possible that they are overwritten (by their creators). ELG (currently) does not have a private Docker registry that caches images. Therefore, when ELG will try (at some point) to spawn a new instance of an LT service, it might download (pull) and use an image that is not (any more) ELG compatible, because it has been overwritten (e.g. by accident). So, yes, it is recommended (but not enforced) to put a custom tag (dedicated for ELG) to the image that you will register, since it is usually more common to override the :latest one.
Question: How many resources will be allocated for my LT container in the k8s cluster?
Answer: By default, 512MB of RAM and half a CPU core. If your LT service requires more resources you have to specify it by using the additionalHWRequirements metadata element (see the MT example above) or by communicating with the ELG administrators.
Question: What is a YAML file and what does it contain?
Answer: Each service has a YAML file which contains information about the allocated resources in the k8s cluster (see question above) and the scaling parameters (whether it is readily available at all times or started on demand).