Skip to content

Commit

Permalink
Update 2024-05-30-counting.md
Browse files Browse the repository at this point in the history
  • Loading branch information
golkar authored Jun 16, 2024
1 parent 622024c commit 4bbd1e9
Showing 1 changed file with 2 additions and 2 deletions.
4 changes: 2 additions & 2 deletions _posts/2024-05-30-counting.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,8 +50,8 @@ This information helps disambiguate the different regions based on context.

#### Key Propositions

1. **Proposition 1:** If the regional contextual position information is available in the latent representation of the tokens at some layer of a Transformer, the contextual counting task can be solved with a single additional layer.
2. **Proposition 2:** A causal Transformer with a single layer and no position encoding (NoPE) can infer the regional contextual position.
- **Proposition 1:** If the regional contextual position information is available in the latent representation of the tokens at some layer of a Transformer, the contextual counting task can be solved with a single additional layer.
- **Proposition 2:** A causal Transformer with a single layer and no position encoding (NoPE) can infer the regional contextual position.

These propositions imply that a two-layer causal Transformer with NoPE can solve the contextual counting task.

Expand Down

0 comments on commit 4bbd1e9

Please sign in to comment.