Example Selection

#prompting #few-shots #LM #LLM #representation

Introduction

Example selection methods are a set of strategies designed to improve the "Few-shot" process.

Few-Shot

Few-Shot prompting is a method that enables in-context learning where demonstrations are provided in the prompt to steer the model to better performance.

The demonstrations serve as conditioning for subsequent examples where one would like the model to generate a response.

It is a challenging task especially in Text2SQL since it is not trivial to find similar questions to the proposed one.

For that reason, many methods were proposed.

Objective

Overall, the objective is to find examples close enough to the target question.

Assumption

The implicit assumption in this method is relies on the fact that in-context learning from close examples improve positively the performance of the language model on the task.

Same Domain

Notations

The objective of example selection is to find examples such that are the closest^[1] examples to

Cross Domain

Notations

Here each example is represented by:

A natural language question
A database schema corresponding to that example
Its SQL query

The objective is to find examples such that are the closest^[2] examples to

Random

This strategy randomly samples examples from the available candidates. It is used as a baseline for evaluating other strategies.

Question Similarity Selection

It is denoted by It chooses examples with the most similar questions. Specifically, it:

Embeds both example questions in and the target question with a pre-trained language model.
Then it applies a pre-defined distance measure^[3] to each example-target pair.
Finally, k-NN algorithm is leveraged to select examples from Q that closely match the target question .

Masked Question Selection

It is denoted by It eliminates the negative influence of domain-specific information by replacing table names, column names, and values in all questions with a mask token, and then compute the similarities of their embedding with k-NN algorithm, similarly to

Query Similarity Selection

It is denoted by It aims to select examples that are similar to target SQL query . Specifically, it:

Employs a preliminary model to generate SQL query using target question and database .
Then it encodes queries from examples into binary discrete syntax vectors according to their keywords.
After that, it chooses examples by considering both similarity to the approximated query and diversity among selected examples.

In is considered as a first approximation of And is a refinement of that approximation.

DAIL Selection

Dail Selection, denoted by is an attempt to combine both and It was introduced in DAIL-SQL, and works as follows:

Both masked example questions in and the masked target question are embedded with a pre-trained language model
For each example question , a pre-defined distance measure^[4] between the embedding of and the embedding of the target question are calculated.
A preliminary model is used to generate a preliminary SQL query using target question and database .
For each example query , a similarity measure^[5] between and the the preliminary prediction are calculated.
Elements of are partitioned into two groups via the predicate In other words, elements with similarity are placed in and the others in
Each group is sorted increasingly via
Once sorted, the first elements of are selected.

Notes & References

The meaning of close is defined within the selection method.↩︎
For simplicity, one can ignore schemas in the definition of closeness.↩︎
Such as the Euclidean distance or negative cosine similarity.↩︎
In the official Spider submission, Cosine similarity was used.↩︎
In the official Spider submission, Jacard similarity was used.↩︎