Pipeline of services

The pipeline class allows to run services one after the other. The output of the first service is used as the input of the next one and so one. It is a basic approach but allows users to create complex services.

[1]:
from elg import Pipeline

A pipeline is simply a list of services and therefore can be initialized with a list of service ids or a list of entities.

[2]:
german_sentiment_analysis_pipeline = Pipeline.from_ids([607, 510])

To map the output of a service to the next one, we use the ouput_funcs parameter to extract the needed information from the service response. In this example, we need to extract only the translated text to use it as input of the sentiment analyser service. The first item of the output_funcs list corresponds to the callable object that extracts the translated text from the first service output, and the second item of the list corresponds to the function apply to the result of the second service. Here it is set the auto which means that the automatic content extraction will be used.

[3]:
results = german_sentiment_analysis_pipeline(
    "Ich habe diesen Film geliebt. Die Schauspieler, das Drehbuch: alles von einem Meisterwerk.",
    output_funcs=[
        lambda x: x.texts[0].content,
        "auto"
    ]
)
Calling:
        [607] Tilde MT Machine Translation engine, German - English
with request:
        type: text - content: Ich habe diesen Film geliebt. Die Schauspieler, das Drehbuch: alles von einem Meisterwerk. - mimeType: text/plain

Result:
        I loved this movie. The actors, the script, all from a masterpiece.

Calling:
        [510] GATE: Generic Opinion Mining (English)
with request:
        type: text - content: I loved this movie. The actors, the script, all from a masterpiece. - mimeType: text/plain

Result:
        {'SentenceSentiment': [{'start': 0, 'end': 19, 'source_start': None, 'source_end': None, 'features': {'polarity': 'positive', 'score': 0.5, 'sarcasm': 'no', 'emotion': 'happy', 'rule': 'NPSentimentNP', 'sentiment_string': 'loved', 'holder': 'I', 'entity_target': 'this movie', 'rule2': 'SentenceSentimentTarget'}}], 'Sentence': [{'start': 0, 'end': 19, 'source_start': None, 'source_end': None, 'features': {'firstPerson': 'yes', 'sentiment': 'positive', 'emotion': 'happy'}}, {'start': 20, 'end': 67, 'source_start': None, 'source_end': None, 'features': None}], 'SentenceSet': [{'start': 0, 'end': 67, 'source_start': None, 'source_end': None, 'features': {'score': 0.5, 'score_std_dev': 0.0, 'polarity': 'positive'}}]}

The returned results object contains the result of each service of the pipeline.

[4]:
print("Result of the translation service: ", results[0])
print("\nResult of the sentiment analyser service: ", results[1])
Result of the translation service:  I loved this movie. The actors, the script, all from a masterpiece.

Result of the sentiment analyser service:  {'SentenceSentiment': [{'start': 0, 'end': 19, 'source_start': None, 'source_end': None, 'features': {'polarity': 'positive', 'score': 0.5, 'sarcasm': 'no', 'emotion': 'happy', 'rule': 'NPSentimentNP', 'sentiment_string': 'loved', 'holder': 'I', 'entity_target': 'this movie', 'rule2': 'SentenceSentimentTarget'}}], 'Sentence': [{'start': 0, 'end': 19, 'source_start': None, 'source_end': None, 'features': {'firstPerson': 'yes', 'sentiment': 'positive', 'emotion': 'happy'}}, {'start': 20, 'end': 67, 'source_start': None, 'source_end': None, 'features': None}], 'SentenceSet': [{'start': 0, 'end': 67, 'source_start': None, 'source_end': None, 'features': {'score': 0.5, 'score_std_dev': 0.0, 'polarity': 'positive'}}]}

With the pipeline class we can create a lot of different tools. For example, let’s create a pipeline that generate an German summary from an English news article. To do that, we will use a summarization service (478) and an English to German Machine Translation service (610).

[5]:
pipeline = Pipeline.from_ids([478, 610])
TEXT_TO_SUMMARIZE = """In his new book, Fulfillment, Alec MacGillis writes of an Amazon distribution center in Sparrows Point, Maryland that sits on land once occupied by a Bethlehem Steel plant. The story underscores how dramatically the U.S. economy has transformed in recent years. Instead of making things, many of our biggest companies now distribute things made elsewhere. We've moved from an economy of production to one of dispersion. The shift from factory to fulfillment work is core to the American story right now. For the American worker, a factory job like one at Bethlehem Steel was dangerous, but it paid $30 to $40 per hour, and many stuck with it for life. At an Amazon Fulfillment Center, pay starts at $15 per hour, algorithms monitor your performance, and many workers leave soon after joining. "There's 100% turnover in the warehouses," MacGillis told me this week. "100% \every single year." Some blame the move to fulfillment work entirely on Amazon, but it didn't happen in a vacuum. American politicians helped it along by signing trade deals like NAFTA and enthusiastically welcoming China into the World Trade Organization - and doing so without sufficient safeguards. American industry then suffered the consequences, and Amazon reaped the benefits. Listen to Alec MacGillis on this week's Big Technology Podcast on Apple, Spotify, or your app of choice. The U.S. embrace of globalization flooded the country's markets with inexpensive products, and plants stateside couldn't keep up. As American factories went under or moved overseas, there was glaring a need for a company that could get the new, affordable products to Americans' doorsteps. Amazon eagerly stepped in. And its timing couldn't have been better. In January 1994, NAFTA opened up trade between U.S., Canada, and Mexico. Seven months later, Jeff Bezos founded Amazon. In 2000, Amazon launched its third-party marketplace. One year later, China joined the WTO.For workers, going from $40 to $15 per hour meant moving from an annual salary of $80,000 to one closer to $30,000. And as U.S. industry struggled, Amazon's hired 1,400 workers per day. Many American workers are now removed from the production process - employed by middlemen taking a cut - so the drop in wage is natural. MacGillis tells the story of some workers who, on the same land, worked for Bethlehem Steel and then Amazon for one-third the wage. American workers have thus taken a hard look at our political system and found it wanting. In the aftermath of the trade deals, they've been left too often to decide between unemployment and jobs at fulfillment centers, ride-hailing services, and app-based food delivery. In 2016, many workers remembered Bill Clinton signing NAFTA and backing China's entry to the WTO, and voted for Donald Trump. In 2020, as Covid raged, enough moved to Joe Biden's camp that he won the election with an amalgamation of workers and college-educated liberals, what MacGillis calls the "Amazon Coalition." In some cases, Amazon workers drawn to the Democratic party for its alliance with labor were peeing in bottles as they delivered packages to urban-dwellers drawn to the party for its social values. The underlying tension flared up recently as workers at an Amazon fulfillment center in Bessemer, Alabama pushed to unionize. Bernie Sanders and Elizabeth Warren came out strongly in support of the union, fully aware of how vital fulfillment workers are to their future. Amazon then mocked the government for failing the working class. Its executive overseeing the fulfillment centers, Dave Clark, said "We actually deliver a progressive workplace," while hailing Amazon's $15 minimum wage compared to the federal government's $7.25. The union effort in Bessemer seems en route to defeat. But even in victory, it would not have replaced the fulfillment center work with factory work. The reality facing U.S. workers today is that low-paying fulfillment jobs are often their best option. And we're likely to see more political instability if we don't find some way to generate jobs that give workers a shot at the middle class. """

results = pipeline(
    TEXT_TO_SUMMARIZE,
    output_funcs=[
        lambda x: " ".join([data.features["prefLabel"] for data in x.annotations["Main sentence"]]),
        lambda x: x.texts[0].content,
    ]
)
print("German summary: ", results[-1])
Calling:
        [478] Cogito Discover Summarizer
with request:
        type: text - content: In his new book, Fulfillment, Alec MacGillis writes of an Amazon distribution center in Sparrows Point, Maryland that sits on land once occupied by a Bethlehem Steel plant. The story underscores how dramatically the U.S. economy has transformed in recent years. Instead of making things, many of our biggest companies now distribute things made elsewhere. We've moved from an economy of production to one of dispersion. The shift from factory to fulfillment work is core to the American story right now. For the American worker, a factory job like one at Bethlehem Steel was dangerous, but it paid $30 to $40 per hour, and many stuck with it for life. At an Amazon Fulfillment Center, pay starts at $15 per hour, algorithms monitor your performance, and many workers leave soon after joining. "There's 100% turnover in the warehouses," MacGillis told me this week. "100% \every single year." Some blame the move to fulfillment work entirely on Amazon, but it didn't happen in a vacuum. American politicians helped it along by signing trade deals like NAFTA and enthusiastically welcoming China into the World Trade Organization - and doing so without sufficient safeguards. American industry then suffered the consequences, and Amazon reaped the benefits. Listen to Alec MacGillis on this week's Big Technology Podcast on Apple, Spotify, or your app of choice. The U.S. embrace of globalization flooded the country's markets with inexpensive products, and plants stateside couldn't keep up. As American factories went under or moved overseas, there was glaring a need for a company that could get the new, affordable products to Americans' doorsteps. Amazon eagerly stepped in. And its timing couldn't have been better. In January 1994, NAFTA opened up trade between U.S., Canada, and Mexico. Seven months later, Jeff Bezos founded Amazon. In 2000, Amazon launched its third-party marketplace. One year later, China joined the WTO.For workers, going from $40 to $15 per hour meant moving from an annual salary of $80,000 to one closer to $30,000. And as U.S. industry struggled, Amazon's hired 1,400 workers per day. Many American workers are now removed from the production process - employed by middlemen taking a cut - so the drop in wage is natural. MacGillis tells the story of some workers who, on the same land, worked for Bethlehem Steel and then Amazon for one-third the wage. American workers have thus taken a hard look at our political system and found it wanting. In the aftermath of the trade deals, they've been left too often to decide between unemployment and jobs at fulfillment centers, ride-hailing services, and app-based food delivery. In 2016, many workers remembered Bill Clinton signing NAFTA and backing China's entry to the WTO, and voted for Donald Trump. In 2020, as Covid raged, enough moved to Joe Biden's camp that he won the election with an amalgamation of workers and college-educated liberals, what MacGillis calls the "Amazon Coalition." In some cases, Amazon workers drawn to the Democratic party for its alliance with labor were peeing in bottles as they delivered packages to urban-dwellers drawn to the party for its social values. The underlying tension flared up recently as workers at an Amazon fulfillment center in Bessemer, Alabama pushed to unionize. Bernie Sanders and Elizabeth Warren came out strongly in support of the union, fully aware of how vital fulfillment workers are to their future. Amazon then mocked the government for failing the working class. Its executive overseeing the fulfillment centers, Dave Clark, said "We actually deliver a progressive workplace," while hailing Amazon's $15 minimum wage compared to the federal government's $7.25. The union effort in Bessemer seems en route to defeat. But even in victory, it would not have replaced the fulfillment center work with factory work. The reality facing U.S. workers today is that low-paying fulfillment jobs are often their best option. And we're likely to see more political instability if we don't find some way to generate jobs that give workers a shot at the middle class.  - mimeType: text/plain

Result:
        In his new book, Fulfillment, Alec MacGillis writes of an Amazon distribution center in Sparrows Point, Maryland that sits on land once occupied by a Bethlehem Steel plant. MacGillis tells the story of some workers who, on the same land, worked for Bethlehem Steel and then Amazon for one-third the wage. In some cases, Amazon workers drawn to the Democratic party for its alliance with labor were peeing in bottles as they delivered packages to urban-dwellers drawn to the party for its social values.

Calling:
        [610] Tilde MT Machine Translation engine, English - German
with request:
        type: text - content: In his new book, Fulfillment, Alec MacGillis writes of an Amazon distribution center in Sparrows Point, Maryland that sits on land once occupied by a Bethlehem Steel plant. MacGillis tells the story of some workers who, on the same land, worked for Bethlehem Steel and then Amazon for one-third the wage. In some cases, Amazon workers drawn to the Democratic party for its alliance with labor were peeing in bottles as they delivered packages to urban-dwellers drawn to the party for its social values. - mimeType: text/plain

Result:
        In seinem neuen Buch „Erfüllung“ schreibt Alec MacGillis von einem Amazonas-Vertriebszentrum in Splitterpunkten, Maryland, das auf einmal von einer Stahlfabrik Bethlehem besetzten Flächen sitzt. Macgillis erzählt die Geschichte einiger Arbeiter, die auf demselben Land für Bethlehem Steel und Amazon für ein Drittel des Lohns gearbeitet haben. In einigen Fällen haben Amazon-Arbeiter, die wegen ihrer Arbeitsgemeinschaft an die Demokratische Partei verwiesen wurden, in Flaschen gepinkelt, als sie Pakete für ihre sozialen Werte an Stadtbewohner geliefert haben.

German summary:  In seinem neuen Buch „Erfüllung“ schreibt Alec MacGillis von einem Amazonas-Vertriebszentrum in Splitterpunkten, Maryland, das auf einmal von einer Stahlfabrik Bethlehem besetzten Flächen sitzt. Macgillis erzählt die Geschichte einiger Arbeiter, die auf demselben Land für Bethlehem Steel und Amazon für ein Drittel des Lohns gearbeitet haben. In einigen Fällen haben Amazon-Arbeiter, die wegen ihrer Arbeitsgemeinschaft an die Demokratische Partei verwiesen wurden, in Flaschen gepinkelt, als sie Pakete für ihre sozialen Werte an Stadtbewohner geliefert haben.