In-Context Learning
503 papers with code • 0 benchmarks • 0 datasets
Benchmarks
These leaderboards are used to track progress in In-Context Learning
Libraries
Use these libraries to find In-Context Learning models and implementationsMost implemented papers
What needs to go right for an induction head? A mechanistic study of in-context learning circuits and their formation
By clamping subsets of activations throughout training, we then identify three underlying subcircuits that interact to drive IH formation, yielding the phase change.
What Changes Can Large-scale Language Models Bring? Intensive Study on HyperCLOVA: Billions-scale Korean Generative Pretrained Transformers
kakaobrain/kogpt
•
•
EMNLP 2021
GPT-3 shows remarkable in-context learning ability of large-scale language models (LMs) trained on hundreds of billion scale data.
MetaICL: Learning to Learn In Context
facebookresearch/metaicl
•
•
NAACL 2022
We introduce MetaICL (Meta-training for In-Context Learning), a new meta-training framework for few-shot learning where a pretrained language model is tuned to do in-context learning on a large set of training tasks.
Learning To Retrieve Prompts for In-Context Learning
In-context learning is a recent paradigm in natural language understanding, where a large pre-trained language model (LM) observes a test instance and a few training examples as its input, and directly decodes the output without any update to its parameters.
Black-Box Tuning for Language-Model-as-a-Service
txsun1997/black-box-tuning
•
•
In such a scenario, which we call Language-Model-as-a-Service (LMaaS), the gradients of PTMs are usually unavailable.
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
alrope123/rethinking-demonstrations
•
•
Large language models (LMs) are able to in-context learn -- perform a new task via inference alone by conditioning on a few input-label pairs (demonstrations) and making predictions for new inputs.
UL2: Unifying Language Learning Paradigms
google-research/google-research
•
•
Our model also achieve strong results at in-context learning, outperforming 175B GPT-3 on zero-shot SuperGLUE and tripling the performance of T5-XXL on one-shot summarization.
Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning
r-three/t-few
•
•
ICL incurs substantial computational, memory, and storage costs because it involves processing all of the training examples every time a prediction is made.
Don't Generate, Discriminate: A Proposal for Grounding Language Models to Real-World Environments
dki-lab/pangu
•
•
Most existing work for grounded language understanding uses LMs to directly generate plans that can be executed in the environment to achieve the desired effects.
Z-ICL: Zero-Shot In-Context Learning with Pseudo-Demonstrations
alrope123/z-icl
•
•
Although large language models can be prompted for both zero- and few-shot learning, performance drops significantly when no demonstrations are available.
