Validate an ELG compatible LT service (at technical/metadata level)¶

See here how to access the Validation Tasks, which is the list of items assigned to you for validation. You can, then, apply the filters on the left to help you reduce the number of items presented or search for a specific item using the search box.

Deployment of the service at ELG and service registration¶

Before you perform the technical/metadata validation, you must first deploy the service at ELG and register it at the ELG catalogue. First you should verify that the metadata record follows the standard ELG conventions, in particular:

the docker download location (and service adapter download location if relevant) must be an image reference suitable for use with docker pull, not a link to a Docker Hub or GitLab container registry web page
the execution location must be of the form http://localhost:<port>/<path>
for non-public services, you will need to ensure that you have created a suitable namespace (if one does not already exist for this provider) and any necessary secrets for the image pull credentials. You may need to contact the curator directly to obtain this information.

If this is not the case, reject the record and ask the provider to correct these items.

You can now deploy the LT service into the ELG kubernetes cluster: For this you have to create the required yaml file, as in the example below, in the respective GitLab repository and branch.

image : "registry.gitlab.com/qurator-platform/dfki/srv-ler:1.0.569798760"
limits_memory: 2048Mi
scalability: "dynamic"
minScale: "1"

To create the yaml file, you will need the docker_download_location, service_adapter_download_location, execution_location metadata element values, that you will find on the service registration form, and any additional_hw_requirements from the “download/run” tab. By selecting one of the metadata records on the My validations tab, you will be directed to the its view page; click on Actions and select Perform service registration.

When the yaml file is commited/pushed, the automatic CI/CD deployment pipeline of ELG will be notified and install the LT service to the respective kubernetes cluster; for instance, the master branch is used for the production ELG cluster, while the develop branch is used for the development cluster. You will also need access to the cluster so that you can check which containers/pods are running, inpect the logs and statuses of the containers, etc.; for this, you will receive the required information and credentials from the ELG technical team.

On the service registration form, you must also provide the following necessary information

select the Tool type value (IE, MT, ASR, etc.), depending on the type of service you validate. The metadata of the service will help you decide which is the appropriate value;
fill in the ELG execution location; the value of this field depends on how the LT service is deployed into the cluster; e.g. knative vs. non-knative (kubernetes) service, which namespace was used, etc. As described here, this field follows this template: http://{k8s service name for the registered LT tool}.{k8s namespace for the registered LT tool}.svc.cluster.local{the path where the REST service is running at}. The {the path where the REST service is running at} part can be filled based on the executionLocation element value of the metadata record. For this reason, the ELG execution location field id pre-filled with the executionLocation value, so that you can change only the part that is required.
set the Elg gui url; the standard set of GUIs available in the ELG platform by default are:
- /dev/gui-ie/ for IE tools that take text as input and return a response that is either annotations (relative to the original text), or texts that should be shown instead of the original input text,
- /dev/gui-ie/index-mt.html for MT and similar services that take text as input and return a response of type texts that should be shown along with the original input text,
- /dev/gui-ie/index-asr.html for ASR services that take audio as input and return texts,
- /dev/gui-ie/index-audio-annotation.html for “audio annotation” services that take audio as input and return an annotations response where the start/end of each annotation is a possibly-fractional number of seconds from the start of the audio stream,
- /dev/gui-ie/index-text-classification.html for Text Classification services that take text input and return a classification response,
- /dev/gui-ie/index-image.html for services such as OCR that take images as input and return texts as output,
- /dev/gui-udpipe/ for dependency parsers, or
- /dev/gui-tts/ for TTS services.
To specify the text direction of the results in the try out UIs you should use the appropriate parameters; e.g. for ASRs use /dev/gui-ie/index-asr.html?dir=ltr or /dev/gui-ie/index-asr.html?dir=rtl for left-to-right and right-to-left directions, respectively. Similarly, /dev/gui-ie/?dir=ltr or /dev/gui-ie/?dir=rtl for IE services. For MT services use srcdir and targetdir parameters (e.g. index-mt.html?srcdir=ltr&targetdir=rtl) to specify the text direction of input/output. Additionally, if the service returns a “texts” response consisting of a list of tokens, rather than a flat text with standoff annotations, you can specify ?roles=token (for a flat list of tokens) or ?roles=sentence-token (for a list of sentences where each sentence is a list of tokens), in order to display the token list as if it were a string with standoff token annotations.

Depending on the value that you have set, the appropriate try out UI will be displayed in the respective tab of the landing page of the LT service. If there is no available/appropriate try out UI for the specific service or for any other reason you can disable/hide the try out UI tab by setting ``none`` to the ``Elg gui url`` field.
set the Accessor id of the service. This id is unique and it is used for calling the service via the ELG public LT REST API. For a brief overview of the LT REST API, see this section.
the execution location is pre-filled with the respective metadata value;
the docker download location is pre-filled with the respective metadata value;
set the value of status to completed and click on Submit in order to activate the service.

Τest the deployed service: You have to use the Try out tab of the service view page or with any other available clients/tools. You must ensure that the service follows the ELG specifications and works as expected. For any issues that arise, you can use the specified slack channel to communicate with the provider of the service.

Now that the service is tested and integrated into ELG, you can continue with the technical/metadata validation.

Technical/Metadata validation¶

Go to the item view page and click on action to proceed.

A form will open in which you must say whether you Approve or Reject the item after the technical and metadata validation. You must provide both answers before you submit the form.

For the Metadata validation you are asked to check whether the values of the following elements are included in the metadata record and whether their values match the description of the service:

function: important for findability purposes
input & output language(s): for MT services, the output language(s) must be included; for services of other types, the output language is not recommended (redundant information)
input & output data type(s): important for findability and interoperability purposes
output annotation type(s): if the tool is of the IE type, it’s recommended that this element has values for the types of information annotated/extracted
resource creator(s) and publication date: although not mandatory, they are useful for citation purposes;
domain(s): recommended for findability purposes; if possible, recommend the use of an existing value.
documentation: user and installation manuals for services are recommended; publications describing the use of the resource are also welcome
distribution(s): if a resource is available in multiple forms (e.g. as a functional service, but also as source code or downloadable form), it’s recommended to describe them as different distributions
software distribution form: check the values at https://european-language-grid.readthedocs.io/en/stable/Documentation/ELG-SHAREschema.html#SoftwareDistributionForm; depending on the form, a different element (access, download, execution or docker download location) is recommended.

If you are satisfied, approve both types of validation and click on Submit.

If not, set the value of the Technical validation and/or the Metadata validation (depending on the source of the issue) to Reject. This will generate a new field where you can write the recommendations you would like to share with the curator. You can also add comments in the Validator notes field which will be visible only to other validators. When you have finished, click on Submit.

Technical/metadata validation form with review comments

The provider will be notified by email (containing the review comments) in order to update the record. Once finished, the provider will re-submit the record for publication and you will be notified to perform the validation.

Note

Please, keep in mind that an item is published only when it has been approved at all validation levels (technical, metadata and legal).