0
1
2
3
4
5
6
7
8
9
0
1
2
3
4
5
6
7
8
9
0
1
2
3
4
5
6
7
8
9
0
1
2
3
4
5
6
7
8
9
0
1
2
3
4
5
6
7
8
9
0
1
2
3
4
5
6
7
8
9
0
1
2
3
4
5
6
7
8
9
0
1
2
3
4
5
6
7
8
9
0
1
2
3
4
5
6
7
8
9
0
1
2
3
4
5
6
7
8
9
0
1
2
3
4
5
6
7
8
9
0
1
2
3
4
5
6
7
8
9
/
0
1
2
3
4
5
6
7
8
9
0
1
2
3
4
5
6
7
8
9
/
0
1
2
3
4
5
6
7
8
9
0
1
2
3
4
5
6
7
8
9
ANNO·​TRICESIMO·​DIE·​DVCENTESIMO·​TRICESIMO·​VITÆ·​POVYA
LLMRemarksOnThePaper"Ahierarchyoflinguisticpredictionsduringnaturallanguagecomprehension"
LLM Remarks On The Paper "A hierarchy of linguistic predictions during natural language comprehension"
PaperAnnotationsOfReading"In-ContextLearningCreatesTaskVectors"
Paper Annotations Of Reading "In-Context Learning Creates Task Vectors"
Quotes & Excerpts

In-context learning (ICL) in Large Language Models has emerged as a powerful new learning paradigm. However, its underlying mechanism is still not well understood. In particular, it is challenging to map it to the “standard” machine learning framework where the model can learn from a training set S to find a best-fitting function f(x) in some hypothesis class. Here, we make progress on this problem by showing that the functions learned by ICL often have a very simple structure: they correspond to the transformer LLM whose only inputs are the query x and a single “task vector” θ(S) that can be calculated from the training set S. Thus, ICL can be seen as compressing S into a single task vector θ(S) and then using this task vector to modulate the transformer to produce the output.

ROEE HENDEL & MOR GEVA & AMIR GLOBERSON

Symbolic placeholders that can take on specific values. Templates represent the underlying structure of patterns, allowing us to explore whether models can learn these abstract patterns and generalize to new instances that the model has never seen, such as ABACABAC, that follow similar structural rules. These templates can be instantiated into specific sequences, such as 12312312 or 45645645, by assigning values to the placeholders. For example, consider the abstract pattern ABCABCAB. By assigning specific values to A, B, and C, we generate different instantiations of the same template such as 12312312, 45645645, 78978978, and 15915915.

ROEE HENDEL & MOR GEVA & AMIR GLOBERSON

In-context learning (ICL) in Large Language Models has emerged as a powerful new learning paradigm. However, its underlying mechanism is still not well understood. In particular, it is challenging to map it to the “standard” machine learning framework where the model can learn from a training set S to find a best-fitting function f(x) in some hypothesis class. Here, we make progress on this problem by showing that the functions learned by ICL often have a very simple structure: they correspond to the transformer LLM whose only inputs are the query x and a single “task vector” θ(S) that can be calculated from the training set S. Thus, ICL can be seen as compressing S into a single task vector θ(S) and then using this task vector to modulate the transformer to produce the output. Furthermore, we aim to determine whether a model can recognize such instantiations in-context, that is, whether it can identify the symbolic mappings during inference without retraining and by using this understanding to solve new instances of the same abstract template. This ability would indicate that the model is not only learning patterns from its training data but also reasoning dynamically based on the input it encounters during inference.

ROEE HENDEL & MOR GEVA & AMIR GLOBERSON

Our experiments demonstrate that small transformer models with two or three layers can successfully solve the task of abstract pattern matching, whereas single-layer models fail to solve the task. We identify the mechanism that enables the induction head (Sanford et al., 2024) to perform pattern matching. We define an abstraction head as an attention mechanism in transformer models that attends to previous instantiations of a pattern in an abstract manner, identifying the structural relationships between symbolic placeholders, and using this information to perform pattern matching on unseen instances with a similar abstract structure.

ROEE HENDEL & MOR GEVA & AMIR GLOBERSON

The identification of the mechanisms of symbolic reasoning emerging in the training of neural networks can help to build new types of neuro-symbolic processing paradigms. Moreover, it may also help to train neural networks with an internal visible symbolic reasoning mechanism. The recent survey by Bhuayan et al. (2024) gives a taxonomy of different neurosymbolic systems in their taxonomy.

ROEE HENDEL & MOR GEVA & AMIR GLOBERSON

[…] Bump occurs at approximately 70,000 steps. We can also see a correspondence between the number of steps where the bumps occur, and the number of steps before the sudden rise in the last digit accuracy shown in Figure 3 for both models appears in green and pink. Figure 3 also shows that not all models achieve perfect accuracy, with two models reaching an accuracy of approximately 0.98. While Figure 2 shows that all models successfully achieve an accuracy of approximately 1.0 across all runs.

ROEE HENDEL & MOR GEVA & AMIR GLOBERSON

To understand how the model solves the problem, we visualized the attention patterns of solving a test instance during inference time. Among the six attention heads, two heads provided us with useful insight into how both tasks are solved (Figures 4 and 5). Figure 4 visualizes the abstraction of patterns, where the attention mechanism focuses on the relationship between the first instantiation and the last three instantiations. Specifically, the tokens in the last three instantiations, which are attending back to the first instantiation to predict the next token. For example, in the second instantiation the bolded 6 in 66227767 is attending to the bolded 3 in 33117737 in the first instantiation. This behavior is consistent across all three later instantiations (second, third, and fourth). Moreover, this pattern of attention is not only specific to the first token, with nearly all tokens in the instantiations attending back to their corresponding ‘abstract’ next token in the first instantiation, with the exception of the fifth position.

ROEE HENDEL & MOR GEVA & AMIR GLOBERSON

Understanding spoken language requires transforming ambiguous acoustic streams into a hierarchy of representations, from phonemes to meaning. It is often argued that during this process, the brain relies on prediction to guide the interpretation of incoming input. […] we disentangle model-based predictions into distinct dimensions, revealing discrete neural signatures of predictions about syntactic category (parts of speech), phonemes, and semantics.

[…] Finally, we show that high-level (word) predictions inform low-level (phoneme) predictions, supporting hierarchical predictive processing. Together, these results underscore the ubiquity of prediction in language processing, showing that the brain spontaneously predicts upcoming language at multiple levels of abstraction

We find that brain responses are continuously modulated by linguistic predictions. We observe predictions at the level of meaning, grammar, words, and sounds, and find that high-level predictions can inform low-level predictions. Our results establish the predictive nature of language processing, demonstrating that the brain spontaneously predicts upcoming language at multiple levels of abstraction.

Day's Context
Open Books