Valid Efficiency Score

#evaluation #VES #matching

Quick-start

This metric is a modification of on which accurate predictions are also weighted by relative execution time.

Formula

Notations:

It is based on the formalisation of EX:

Let be the number of samples
Let be a dataset containing samples of questions, their database schema with their respective queries.
Let be a prediction of the queries using
Let be a database against which the queries will be executed.
Let be the execution function, which executes a query on a database and return a response and its execution time

The valid efficiency score of between and over is defined as:

Where:

is the execution of the ground truth query over
is the execution of the predicted query over
is the execution time of the ground truth query
is the execution time of the predicted query

Advantages

The key benefit lies in how the places greater emphasis on efficient and accurate queries over slower yet correct ones.

Short-comings

This metric does not have a solid theoretical justification.
A prediction can be mistakenly rewarded/penalised because of external factors such as the load of the underlying system.

Interactive Graph

Table Of Contents

Valid Efficiency Score