The execution accuracy is defined as the fraction of predictions that match the ground truth as measured by an execution on some database.
Formula
Notations:
Let be the number of samples
Let be a dataset containing samples of questions, their database schema with their respective queries.
Let be a prediction of the queries using
Let be a database against which the queries will be executed.
Let be the execution function, which executes a query on a database and return a response
The execution accuracy between and and over is defined as:
Where:
Advantages
This metric does never penalise between semantically equivalent queries. In other words, it is guaranteed that implies
Since the databases are fixed. they can be manually optimised by indexing all columns, which can improve execution times drastically.
Short-comings
This metric is dependent on the database system, and thus it may take large running times if the database is sufficiently large or the queries are not efficient.
Execution accuracy suffers from false positives. That is, two semantically different queries can return the same responses and thus