Benchmark¶
-
class
elg.benchmark.
Benchmark
(services: List[elg.service.Service])¶ Class to execute multiple services in parallel and allows for easy comparison of their outputs using the same input.
Examples:
from elg import Benchmark # A benchmark can be initialized with a list of services ids (from_ids method) or with a list of entities (from_entities method). # Here we compare English to German Machine Translation services. ben = Benchmark.from_ids([610, 624]) # The benchmark can be run on multiple inputs and can be run multiple times to guarantee the result (the first run is also # usually longer than the next ones due to the service pods initialization). result = ben(["Bush is the president of the USA and lives in Washington.", "ELG is an amazing project."], number_of_runs=2) # The benchmark call returns a benchmark result object that can be used to compare the result. # You can have an overview of the result, df = result.compare() print("General comparison:\n", df) # compare only the results, df = result.compare_results() print("Comparison of the results:\n", df) # or only the response time. df = result.compare_response_times() print("Comparison of the response time:\n", df) # The compare methods return a DataFrame object that can be exported to csv, excel and many other formats to have a # better visualization result.compare().to_csv("/tmp/result.csv") # We can take another example and compare sentiment analysis services. ben = Benchmark.from_ids([477, 510]) inputs = [ "This movie is not good at all.", "This movie is not good but it was a good moment at the cinema.", "This movie is not so bad.", "I liked the movie but it was not must seen.", "It was the best movie I have ever seen." ] result = ben( inputs, output_funcs=[ lambda x: x.features["OVERALL"], lambda x: x.annotations["SentenceSet"][0].features["score"] * 100 ] ) print("Result:\n") result.compare()
-
__call__
(request_inputs: Optional[Union[str, List[str], elg.model.base.Request.Request, List[elg.model.base.Request.Request]]] = None, request_type: str = 'text', sync_mode: bool = False, timeout: Optional[int] = None, check_file: bool = True, output_funcs: Union[str, Callable, List[Union[Callable, str]]] = 'auto', number_of_runs: int = 2)¶ Method to run the comparison of the services with the given inputs.
- Parameters
request_inputs (Union[str, List[str], Request, List[Request]], optional) – list of inputs on which to compare the services. Each input must correspond to the request_input parameter of the Service __call__ method. Defaults to None.
request_type (str, optional) – precise the type of the request. Can be “text”, “structuredText”, or “audio”. It is only used if request_input is not a Request object. Defaults to “text”.
sync_mode (bool, optional) – sync_mode parameter to give to the Service __call__ method. Defaults to False.
timeout (int, optional) – timeout parameter to give to the Service __call__ method. Defaults to None.
check_file (bool, optional) – check_file parameter to give to the Service __call__ method. Defaults to True.
output_funcs (Union[str, Callable, List[Union[str, Callable]]], optional) – output_func parameters to give to the Services __call__ method. Defaults to “auto”.
number_of_runs (int, optional) – number of times to run the services on each input. It is recommended to run the services at least 2 times because on the first time the services usually need to be loaded in the ELG cluster, which will increase the response time. The response time of the second pass is, therefore, more precise. Defaults to 2.
- Returns
- result of the Benchmark call. To obtain the pandas DataFrame containing all the results, run the compare
method on the obtained BenchmarkResult object.
- Return type
-
classmethod
from_entities
(entities: List[elg.entity.Entity], auth_object: Optional[str] = None, auth_file: Optional[str] = None, scope: Optional[str] = None, use_cache: bool = True, cache_dir='~/.cache/elg')¶ Class method to init a Benchmark using a list of entities which will be convert into services using the from_entity class method of the Service class. Refer to this method for further explanation.
-
classmethod
from_ids
(ids: List[int], auth_object: Optional[elg.authentication.Authentication] = None, auth_file: Optional[str] = None, scope: Optional[str] = None, domain: Optional[str] = None, use_cache: bool = True, cache_dir: str = '~/.cache/elg')¶ Class method to init a Benchmark using a list of ids which will be convert into services using the from_id class method of the Service class. Refer to this method for further explanation.
-
-
class
elg.benchmark.
BenchmarkResult
(services: List[elg.service.Service], request_inputs: List[str])¶ Class the represent the result of a Benchmark call
-
compare
(columns: List[str] = ['result', 'response_time'], func: Union[str, list, dict] = 'last', level: str = 'run', colwidth: int = 0, **agg_kwargs)¶ Method to compare the obtained results. It returns a pandas DataFrame object containing the comparison
- Parameters
columns (List[str], optional) – colums of the DataFrame to returned. Defaults to [“result”, “response_time”].
func (Union[str, list, dict], optional) – function to use for the comparison. To see all the possible function, please see https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.core.groupby.DataFrameGroupBy.aggregate.html?. Defaults to “last”.
level (str, optional) – level of the comparison. The level value can be: ‘service’, ‘request_input
' –
'run'. Defaults to "run". (or) –
colwidth (int, optional) – if set, will change the colwidth parameter of pandas to better vizualize the DataFrame. Defaults to 0.
- Raises
ValueError – error if the level parameters is not set to a correct value.
- Returns
pandas DataFrame object containing the comparison
- Return type
pd.DataFrame
-
compare_response_times
(func: Union[str, list, dict] = 'describe', level: str = 'service', colwidth: int = 0, **agg_kwargs)¶ Method similar to the compare method with default parameters optimized to compare the response_times
-
compare_results
(func: Union[str, list, dict] = 'last', level: str = 'request_input', colwidth: int = 0, **agg_kwargs)¶ Method similar to the compare method with default parameters optimized to compare the results
-
set_colwidth
(value: Optional[int] = None)¶ Method to easily change the colwidth value of pandas to better vizualize the DataFrame
- Parameters
value (int, optional) – value of the colwidth. Defaults to None.
-