Large Language Models (LLMs) based on self-attention circuits are able to perform, at inference time, novel reasoning tasks, but the mechanisms inside the models are currently not fully understood. […] We show that LLMs are able to generalize abstract patterns from the input and form an internal symbolic internal representation of the content. […] We demonstrate the performance of small LLM models trained with sequences of instantiations of abstract sequential symbolic patterns or templates. […] It is shown that even a model with two layers is able to learn an abstract template and use it to generate correct output representing the pattern.

This can be seen as a form of symbolic inference taking place inside the network. In this paper, we call the emergent mechanism ‘abstraction head’. […] Identifying mechanisms of symbolic reasoning in a neural network can help to find new ways to merge symbolic and neural processing.

The induction head mechanism is considered a key factor behind in-context learning, enabling a language model to identify a recurring pattern from its input and either replicate it in the output or merge it with previously stored knowledge (Olsson et al., 2022).

In this paper, we investigate the ability of small transformer models to recognize, learn, and generalize abstract sequential symbolic patterns, or templates. […] A template refers to an abstract sequential symbolic pattern that follows a defined structure but can be instantiated with different symbolic elements. For example, the template ABCABCAB represents a repeated sequence where A, B, and C.

You’ve seen for yourselves that there are only two numbers in the calculation, the growth rate and how long it continues. If it’s impossible to make a billion dollars without cheating, which of those two numbers is impossible? It’s certainly not impossible to grow at 15% a month without cheating. Startups do that all the time. And how long you can continue to grow at that rate depends on the size of the market. Obviously for you to grow 4000x, there has to be at least 4000x more demand. But that’s all you need. And how could you possibly cheat to increase the market size?

Even unpleasant experiences can be inspiring.

How do you get an idea like that? The answer is one of the most counterintuitive things about startups — which is saying something, because there are a lot of counterintuitive things about startups. But the way to get the very best startup ideas is not to look for startup ideas. If you’re consciously looking for startup ideas, it will make you too conservative. You’ll lop off the outliers. Because the very best startup ideas tend to sound so lame, at first, that you’d reject them if you were consciously looking for startup ideas. That’s what has prevented them from being discovered.

Don’t worry about the second number, the market size. Since you predict future demand, the market will grow. And it’s always possible to expand into adjacent markets. All you need is a beachhead in the territory of unsatisfied need that you can expand from.

So how, exactly, do you make something people like so much that they tell their friends about it? The problem with market economies, and also the great thing about market economies, is that it’s hard to make something customers want that they don’t already have. As soon as a new, satisfiable need is discovered, people rush to satisfy it. So you’re going to have to discover a need that no one else knows about yet.

How do you do that? By feeling the need yourself.

You’re young, and usually young founders should make something that they themselves want. You don’t have enough experience yet to know what other people need. But at the same time your own needs are uniquely valuable, because your needs predict future demand. You’re the age when people start using new things. Whatever you and your friends start using now, everyone is going to be using in ten years. Since your intuitions about other people’s needs are usually a crap signal, and your own needs are an especially valuable one, you should usually listen to the second signal; you should make something you and your friends want.

Exponential growth is like magic. It generates outcomes that seem impossible. And that’s why some politicians distrust it. They don’t understand the math of exponential growth, so when they see people becoming what seems to them impossibly rich, they assume they must have cheated.

There are other ways to get rich than by starting startups. Some of those do require you to exploit people. But startups are the most common way to become really rich, and if you want to start a successful startup, the key is not exploitation but empathy. What do users really want? What could you do for them that would make their lives dramatically better? That kind of empathy is what we look for in founders, and what we cultivate in the ones we accept.

If you just build random stuff you think would be cool, the things you build will actually be far from random.

The FDA has cleared a blood test developed from technology invented at WashU Medicine to aid in diagnosing Alzheimer’s disease. The test, PrecivityAD2, can detect amyloid plaque biomarkers with over 90% accuracy, offering a less invasive alternative to spinal taps and brain scans.