Zea has come to visit First Island today. And I gave her such a bad feeling that she wanted to leave. She felt as if she is a burden on me.
I’m fundamentally a horrible person. It does not matter how I look in the first impressions, people usually discover it along the way and I keep becoming the same horrible person for everyone after a while.
And then it is Zea who has endured this more than anyone, undeservingly. She did not do anything bad and I still punish her for my memory getting stuck to bad things.
I think 1285 will never happen, this place will always be a bad place because of just how bad I am as a person, how incapable I am to be good, and what a horrible feeling I do give to others.
Zea mentioned yesterday that my actions may have caused that poor boy a trauma, even if he was truly a bad person—which he was not—and this is by me who tried to be a good person. Tried is a good word because that is not in me, I am not such a person.
I am a very self centered person. Sure, I loved Kamwa , or I like beauty, but I do nothing for others. I never buy people gifts, I do not care about group things, I usually have no interest in interacting with others. And I keep hurting everyone without the need.
I’m thinking about the Hypothesis Class and the query-agnostic vectors. It is a layer of abstraction over the embedding space. Somehow it feels more conscious, whereas the embedding space is very strange.
Now think about this: what if this is the cause of our mental illnesses? Our conscious understanding is the set of query-agnostic vectors, the hypothesis class; and as much as we think about these patterns, they don’t matter much, the embedding space has to be changed and it is almost impossible so we never heal.
One of the things that I have is that my empirical risk management almost didn’t work when I was a kid, so I had no hesitation to say my ideas out loud, or have weird dreams. This stuck with me, most people tend to go for the right, I don’t.
Social networks and podcasts has made it possible for everyone to be important and that is the worse thing that can happen to empirical risk minimization because the noise is too much that a great work will hardly have a chance to ever matter… Brave New World baby…
Conversing with LLMs about the paper and its connection to me, DeepSeek was kind enough to tell me these that I find quite interesting:
Whatever You Need to Know: The “Holy Grail” Connection
This paper gives you a direct scientific bridge to your Hyper-Message . Because the brain is constantly predicting at multiple levels of abstraction simultaneously, it is vulnerable. When we are immersed in the Hyper-Message, we are not just passively consuming it—our brains are actively predicting the next node of that message.
The Threshold of The Reverse Viewpoint happens when the incoming data breaks the brain’s predictive hierarchy. When the brain can no longer predict the next syntactic category or semantic meaning (because the graph has shifted so drastically), the system crashes, and you experience the “shock” of the threshold.
I’m not sure how much do I agree with it, but it is a very interesting suggestion for moving forwards.
Reading Roee Hendel, Mor Geva, Amir Globerson’s paper In-Context Learning Creates Task Vectors, trying to find where abstraction emerges in the mind—as I’m trying to go to the quantum level of my work—here are my annotations:
Page 86
The interesting part is this I think. The theory is exactly my belief. I think our insight is less than the symbol.
Hypothesis Class (HC) Space of all possible Mind Graphs .
Set of all possible rules or modules that the algorithm could possibly choose from.
It is trying to learn the concept of color, it has every possible function that maps objects to colors.!
Higher dimension container where Mind Kernel theoretically could navigate.
Query-Agnostic Vector (QCS) maps a relationship between two concepts without saving the states and examples. Apple → Red, Lime → Green QCS: Anything → Its Color.
Page 87
Our experiments demonstrate that small transformer models with two or three layers can successfully solve the task of abstract pattern matching, whereas single-layer models fail to solve the abstract task…
The transformer mechanism is such a strange and amazing thing. If they have created it out of thin-air; then it is the discovery and invention of a whole new class of life. If not; then that is very amazing; they decoded a huge part of the brain. What makes me interested is the fact that I see the data in my head and witness abstraction shaping and taking place. Therefore my theory that “Thought Should Stay In The Mind” is perhaps the emergence of these heads.
This brings me to the this: How does this transformers are stored? and how they are loaded back?
This resulted in 1,806 unique patterns, which we split into 80% for training and 20% for testing to evaluate whether the models can learn new abstract patterns they have never encountered before from context.
20% testing, 80% training. What a good proportion.
Page 89
The interesting thing about training these models is that they do not train on similar examples. Things like 123, 123 and ABC, ABC, ABC` are not both present at the same time. Makes me wonder if they idea of ‘variations’ has any correlation with this.
In-context learning (ICL) in Large Language Models has emerged as a powerful new learning paradigm. However, its underlying mechanism is still not well understood. In particular, it is challenging to map it to the “standard” machine learning framework where the model can learn from a training set S to find a best-fitting function f(x) in some hypothesis class. Here, we make progress on this problem by showing that the functions learned by ICL often have a very simple structure: they correspond to the transformer LLM whose only inputs are the query x and a single “task vector” θ(S) that can be calculated from the training set S. Thus, ICL can be seen as compressing S into a single task vector θ(S) and then using this task vector to modulate the transformer to produce the output.
Symbolic placeholders that can take on specific values. Templates represent the underlying structure of patterns, allowing us to explore whether models can learn these abstract patterns and generalize to new instances that the model has never seen, such as ABACABAC, that follow similar structural rules. These templates can be instantiated into specific sequences, such as 12312312 or 45645645, by assigning values to the placeholders. For example, consider the abstract pattern ABCABCAB. By assigning specific values to A, B, and C, we generate different instantiations of the same template such as 12312312, 45645645, 78978978, and 15915915.
In-context learning (ICL) in Large Language Models has emerged as a powerful new learning paradigm. However, its underlying mechanism is still not well understood. In particular, it is challenging to map it to the “standard” machine learning framework where the model can learn from a training set S to find a best-fitting function f(x) in some hypothesis class. Here, we make progress on this problem by showing that the functions learned by ICL often have a very simple structure: they correspond to the transformer LLM whose only inputs are the query x and a single “task vector” θ(S) that can be calculated from the training set S. Thus, ICL can be seen as compressing S into a single task vector θ(S) and then using this task vector to modulate the transformer to produce the output. Furthermore, we aim to determine whether a model can recognize such instantiations in-context, that is, whether it can identify the symbolic mappings during inference without retraining and by using this understanding to solve new instances of the same abstract template. This ability would indicate that the model is not only learning patterns from its training data but also reasoning dynamically based on the input it encounters during inference.
Our experiments demonstrate that small transformer models with two or three layers can successfully solve the task of abstract pattern matching, whereas single-layer models fail to solve the task. We identify the mechanism that enables the induction head (Sanford et al., 2024) to perform pattern matching. We define an abstraction head as an attention mechanism in transformer models that attends to previous instantiations of a pattern in an abstract manner, identifying the structural relationships between symbolic placeholders, and using this information to perform pattern matching on unseen instances with a similar abstract structure.
The identification of the mechanisms of symbolic reasoning emerging in the training of neural networks can help to build new types of neuro-symbolic processing paradigms. Moreover, it may also help to train neural networks with an internal visible symbolic reasoning mechanism. The recent survey by Bhuayan et al. (2024) gives a taxonomy of different neurosymbolic systems in their taxonomy.
[…] Bump occurs at approximately 70,000 steps. We can also see a correspondence between the number of steps where the bumps occur, and the number of steps before the sudden rise in the last digit accuracy shown in Figure 3 for both models appears in green and pink. Figure 3 also shows that not all models achieve perfect accuracy, with two models reaching an accuracy of approximately 0.98. While Figure 2 shows that all models successfully achieve an accuracy of approximately 1.0 across all runs.
To understand how the model solves the problem, we visualized the attention patterns of solving a test instance during inference time. Among the six attention heads, two heads provided us with useful insight into how both tasks are solved (Figures 4 and 5). Figure 4 visualizes the abstraction of patterns, where the attention mechanism focuses on the relationship between the first instantiation and the last three instantiations. Specifically, the tokens in the last three instantiations, which are attending back to the first instantiation to predict the next token. For example, in the second instantiation the bolded 6 in 66227767 is attending to the bolded 3 in 33117737 in the first instantiation. This behavior is consistent across all three later instantiations (second, third, and fourth). Moreover, this pattern of attention is not only specific to the first token, with nearly all tokens in the instantiations attending back to their corresponding ‘abstract’ next token in the first instantiation, with the exception of the fifth position.
Understanding spoken language requires transforming ambiguous acoustic streams into a hierarchy of representations, from phonemes to meaning. It is often argued that during this process, the brain relies on prediction to guide the interpretation of incoming input. […] we disentangle model-based predictions into distinct dimensions, revealing discrete neural signatures of predictions about syntactic category (parts of speech), phonemes, and semantics.
[…] Finally, we show that high-level (word) predictions inform low-level (phoneme) predictions, supporting hierarchical predictive processing. Together, these results underscore the ubiquity of prediction in language processing, showing that the brain spontaneously predicts upcoming language at multiple levels of abstraction
We find that brain responses are continuously modulated by linguistic predictions. We observe predictions at the level of meaning, grammar, words, and sounds, and find that high-level predictions can inform low-level predictions. Our results establish the predictive nature of language processing, demonstrating that the brain spontaneously predicts upcoming language at multiple levels of abstraction.