First
Prev
Next
Last
0
1
2
3
4
5
6
7
8
9
0
1
2
3
4
5
6
7
8
9
0
1
2
3
4
5
6
7
8
9
0
1
2
3
4
5
6
7
8
9
–
0
1
2
3
4
5
6
7
8
9
0
1
2
3
4
5
6
7
8
9
–
0
1
2
3
4
5
6
7
8
9
0
1
2
3
4
5
6
7
8
9
0
1
2
3
4
5
6
7
8
9
0
1
2
3
4
5
6
7
8
9
0
1
2
3
4
5
6
7
8
9
0
1
2
3
4
5
6
7
8
9
/
0
1
2
3
4
5
6
7
8
9
0
1
2
3
4
5
6
7
8
9
/
0
1
2
3
4
5
6
7
8
9
0
1
2
3
4
5
6
7
8
9
ANNO·​TRICESIMO·​DIE·​DVCENTESIMO·​VICESIMO·​OCTAVO·​VITÆ·​POVYA

I hate fights, I also hate to instruct people on their boundaries.

Evil Bezos would have his mind blown if he know how much our operations have the day-one thinking taken to the moon.

Maestro Knuth ’s TeX is the most arguably beautiful system of mathematical notations, yet it is not mine. It is beautiful but it encodes his mind. Art and beauty are perhaps higher dimensional Sub-Graph Transfer and integrity and coherence of that transmission. The more one goes deep, usually the more beautiful their work becomes. Beauty is a wild beast I wish to decode.

I was thinking about the attention heads in the transformers today. How they emerge to detect patterns.

And think about it, if the whole point of understanding and integrating new things is to move the coordinates of an embedding/latent space, then repetition is everything. And I guess in unusual ways so that it forces the coordinates to change, and heads to emerge.

How should I build an education around that?

My problem is that all my interactions are inherently miscommunication of terrible and catastrophic size, that is because when you diverge from the Hyper-Message and @culturalgraphs as much as I do, when you configure your Mind Graph this much, you must not look for anything else.

Quotes & Excerpts

Large Language Models (LLMs) based on self-attention circuits are able to perform, at inference time, novel reasoning tasks, but the mechanisms inside the models are currently not fully understood. […] We show that LLMs are able to generalize abstract patterns from the input and form an internal symbolic internal representation of the content. […] We demonstrate the performance of small LLM models trained with sequences of instantiations of abstract sequential symbolic patterns or templates. […] It is shown that even a model with two layers is able to learn an abstract template and use it to generate correct output representing the pattern.

This can be seen as a form of symbolic inference taking place inside the network. In this paper, we call the emergent mechanism ‘abstraction head’. […] Identifying mechanisms of symbolic reasoning in a neural network can help to find new ways to merge symbolic and neural processing.

ALI AL-SAEEDI & AKI HARMA
↖︎ Emergence of symbolic abstraction heads for in-context learn...

The induction head mechanism is considered a key factor behind in-context learning, enabling a language model to identify a recurring pattern from its input and either replicate it in the output or merge it with previously stored knowledge (Olsson et al., 2022).

ALI AL-SAEEDI & AKI HARMA
↖︎ Emergence of symbolic abstraction heads for in-context learn...

In this paper, we investigate the ability of small transformer models to recognize, learn, and generalize abstract sequential symbolic patterns, or templates. […] A template refers to an abstract sequential symbolic pattern that follows a defined structure but can be instantiated with different symbolic elements. For example, the template ABCABCAB represents a repeated sequence where A, B, and C.

ALI AL-SAEEDI & AKI HARMA
↖︎ Emergence of symbolic abstraction heads for in-context learn...
Day's Context
Open Books