diff --git a/_posts/2022-08-01-understanding-incontext.md b/_posts/2022-08-01-understanding-incontext.md index d21de7e..9533c97 100644 --- a/_posts/2022-08-01-understanding-incontext.md +++ b/_posts/2022-08-01-understanding-incontext.md @@ -13,6 +13,26 @@ draft: True + *In this post, we provide a Bayesian inference framework for in-context learning in large language models like GPT-3 and show empirical evidence for our framework, highlighting the differences from traditional supervised learning. This blog post primarily draws from the theoretical framework for in-context learning from [An Explanation of In-context Learning as Implicit Bayesian Inference](https://arxiv.org/abs/2111.02080) [^BI] and experiments from [Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?](https://arxiv.org/abs/2202.12837) [^RRD].* @@ -47,11 +67,11 @@ Two examples of in-context learning, where a language model (LM) is given a list **What can in-context learning do?** On many benchmark NLP benchmarks, in-context learning is competitive with models trained with much more labeled data and is state-of-the-art on LAMBADA (commonsense sentence completion) and TriviaQA (question answering). Perhaps even more exciting is the array of applications that in-context learning has enabled people to spin up in just a few hours, including writing code from natural language descriptions, helping with app design mockups, and generalizing spreadsheet functions: {% figure %} -

Here's a sentence describing what Google's home page should look and here's GPT-3 generating the code for it nearly perfectly. pic.twitter.com/m49hoKiEpR

— Sharif Shameem (@sharifshameem) July 15, 2020
+

Here's a sentence describing what Google's home page should look and here's GPT-3 generating the code for it nearly perfectly. pic.twitter.com/m49hoKiEpR

— Sharif Shameem (@sharifshameem) July 15, 2020
-

This changes everything. 🤯

With GPT-3, I built a Figma plugin to design for you.

I call it "Designer" pic.twitter.com/OzW1sKNLEC

— jordan singer (@jsngr) July 18, 2020
+

This changes everything. 🤯

With GPT-3, I built a Figma plugin to design for you.

I call it "Designer" pic.twitter.com/OzW1sKNLEC

— jordan singer (@jsngr) July 18, 2020
-

=GPT3()... the spreadsheet function to rule them all.

Impressed with how well it pattern matches from a few examples.

The same function looked up state populations, peoples' twitter usernames and employers, and did some math. pic.twitter.com/W8FgVAov2f

— Paul Katsen (@pavtalk) July 21, 2020
+

=GPT3()... the spreadsheet function to rule them all.

Impressed with how well it pattern matches from a few examples.

The same function looked up state populations, peoples' twitter usernames and employers, and did some math. pic.twitter.com/W8FgVAov2f

— Paul Katsen (@pavtalk) July 21, 2020
{% endfigure %} In-context learning allows users to quickly build models for a new use case without worrying about fine-tuning and storing new parameters for each task. It typically requires very few training examples to get a prototype working, and the natural language interface is intuitive even for non-experts.