Taste is like physics, porn, and grammar

Predictive processing explains objective beauty, aesthetic grammar and hedonic treadmills of beauty. Contra Scott Alexander, and elaborating on Zvi.

What it taste? This is the question Scott Alexander asks in a recent controversial post. He then proposes a list of different hypotheses that he finds more or less plausible. I believe we can clarify his question by grounding our explanation in what beauty fundamentally is, at the mechanistic and evolutionary level. At heart we will see that taste is determined by our brain’s capability for prediction of future sensory signals. This capability is shaped by genetics, the art that we have already consumed, and cultural background knowledge. By unraveling how this works we can explain most observed facts, I claim. As a bonus, we will find satisfying mechanistic explanations for why techniques such as symmetry, ornament, rhyme, and repetition are so frequently used.

First, let us take a look at the different explanations for taste proposed by Scott.

1. Taste is like physics - universal with a correct answer

   But why? Experience seems to contradict this.

2. Taste Is Like A Priesthood

   Taste is a set of self-consistent but arbitrary conventions constructed incrementally by a priesthood of high status art consumers.

3. Taste Is Like A Priesthood, But With A Fig Leaf Of Semi-Fake Justifications

   At some point some person is aesthetically repulsed or pleased by something. To justify this after the fact, they come up with some reasonable-sounding explanation. Because of the seeming reasonableness of this explanation, it is internalized by others. Now, when the rule is broken, people genuinely think it is ugly, because it breaks the rule that they think has good reasons to exist.

4. As Above, Except The Justifications Are Good And Important

   […] if the facts were obvious to everyone, then taste would be universal, not limited to a few sophisticates.

   Still, there are ways to rescue this. You could say they’re obvious “once you pay attention”, but that paying attention to them is itself a trained skill.

5. Taste Is Like BDSM Porn

   People say that if you watch too much regular porn, you get desensitized to it and need weirder stuff. Eventually you get desensitized to the weirder stuff too, until finally you’re watching horrible taboo BDSM snuff porn or whatever.

   Maybe taste is also like this. You look at all the nice pretty houses on your block until you’re bored of nice pretty houses and want something new and exciting.

6. Taste is like fashion

   Cool people want to show everyone else they’re cool. But uncool people want to pretend to be cool. We assume that cool people are mostly friends with other cool people, and that coolness radiates outward along the social graph. So the cool people pick a signifier, at first only other cool people know it, and over a few months it gradually radiates to the less cool people, the uncool people, the very uncool people, and finally, years later, me. Then the cool people pick a different signifier and the process begins again.

7. Taste is like grammar

   Grammar is a set of rules for speaking a language. Some of these rules are sensible and necessary, but others are arbitrary or even actively anti-rational. For example, it would make more sense to say “he goed” than “he went”, but only the latter is correct.

   People feel on a deep level that poor grammar is wrong - misplaced apostrophes can send pedant’s into a rage. But descriptivists helpfully tell us that this is mostly arbitrary, and that some minority groups have alternate grammars which are just as good and consistent as ours, despite sounding atrocious (eg “I ain’t be going”).

   […]

   
   So the sophisticates do have reasons behind what they do - but the reasons are arbitrary and kind of stupid. Still, if you do it the wrong way, they’ll laugh at you. Most people don’t want to be laughed at by sophisticated people, and we summarize this situation as “it’s bad grammar, but only grammarians are sophisticated enough to realize this.”

Scott thinks all these have merit, but concludes that Priesthood With Semi-Fake Justifications is most compelling. I think Priesthood works in certain academic and elite circles of art obsessed with conceptual art. But outside these circles, I claim it is usually more objective than the priesthood explanation allows.

Many people have written interesting responses to Scott. Zvi Mowshowitz recently wrote one where he says

If we are going with one answer from Scott’s list, it is obviously grammar. The real answer is it is all of them at different times and places.

I largely agree with this, except I think it is more of a tie between grammar, physics, and BDSM porn. Zvi also believes that quality in many ways is like physics:

Within a given grammar and context, I will stand up for taste Platonism and physics. I believe that, for all practical purposes, yes, there is a right answer to the Quality level of a given work, to whether liking it reflects good taste.

But some cans have been kicked down the road here. Why is taste like grammar? Why does our creation of beautiful objects need to adhere to grammatical rules? What logic shapes them? Within a grammar, how do we determine Quality? Here I aim to answer these.

So let us turn to an important question that will unlock everything. What is beauty? Why and when do we feel it? Are there good evolutionary reasons to be equipped with an aesthetic sense?

What beauty is

In Beauty as entropic fine-tuning, I wrote:

The ability to recognize patterns is clearly useful for survival. Registering a subtle sound cloaked by the hiss of the wind can save you from a predator. Noticing a disturbance in the soil from animal tracks can lead you to your next meal. Intuiting weather patterns can ensure that you seek shelter in time for a storm. Discriminating subtle differences in the color of berries might prevent you from getting poisoned. Detecting the ire in the voice of a fellow caveman might defuse a situation that might otherwise have turned violent.

A generalized pattern recognition ability is clearly evolutionarily beneficial. However, which patterns are important to recognize depends on the particular ancestral environment. In one area your survival depends on recognizing the sound of the local snake species. In another, there are no snakes, but detecting subtleties in the weather patterns is required for you to not freeze to death. Thus, it is not efficient for evolution to hard code an ability to recognize a predetermined set of patterns. It is better to give us an adaptive mechanism that improves our ability to recognize the particular classes of patterns that appear frequently in our environment. However, for such a mechanism to function, there needs to be a reward signal to encourage us to practice and improve. This reward, I believe, is the experience of beauty.

How do you get better at recognizing patterns? Let’s think about a slightly different case: how to get better at an instrument. Any musician worth anything knows that to become a better player, you need to practice music that lies just barely outside your comfort zone. If it is too hard or too easy, you are wasting your time. So we expect something similar for pattern recognition. Patterns in sensory signals that are too easy or hard to recognize will not improve our abilities (the hard ones, by definition, can’t even be detected). Thus, just from this observation, we conclude that only certain fine tuned sensory signals induce a sense of beauty. This is of course corroborated by the fact that making good art takes lots of work. It takes fine tuning.

The next question is: how does the brain know if it gets pattern recognition right or wrong? How does the brain know that it is productively improving, and thus beauty should be dispensed? With food-pleasure, the brain knows to dispense it when food contains high levels proteins, carbs and fats. But sensory signals do not come equipped with an answer sheet. What are the carbs, proteins, and fats of sensory signals?

The nutrient of a sensory signal is predictable pattern. Successful prediction is the only way the brain can verify whether it has extracted a true pattern rather than an imagined one. So, to check its work, the brain makes a prediction of what the sensory signal will be a short moment in the future, and then checks if it is correct. Or rather, then it grades how correct it was, probably with some weighting depending on how probable the brain considered the outcome. If the brain assigned a relatively high probability to the observed outcome, it scores some points. Now, let me dispel a common confusion here. When I talk about this process of prediction and verification, all of this happens subconsciously. That obviously has to be the case, since you just need a split second to determine whether you think a painting is beautiful or not.

What are some examples of predictions? For a piece of music, we might make a prediction of what chords are likely to appear short moments later. Or that a kick drum will land the moment you expect it to. For a landscape, as you look in one direction and take in the scene, the brain can extract certain regularities in the landscape and predict that these regularities will reappear as you move your gaze across the horizon. What about a painting, which you can behold in totality all at once? Not a problem. The brain could easily solve this problem by masking: making a prediction using only information from subset of the painting and then check on other subsets. It can artificially introduce “time” if it wants to. Either way, our attention and focus is usually placed on localized subregions of the painting, rather than diffusely spread over the whole, so as we shift attention in time, the brain can naturally test its predictions.

If you doubt that your brain is making predictions, listen to this snippet of music:

If you are like most people, you have a very strong expectation about the note to follow. Then listen to this one:

Yuck. And now finally this one:

Isn’t that small satisfaction you felt now, upon your successful prediction, a plausible atom of beauty?

Now let us return to the fine-tuning aspect. If the brain almost perfectly predicts the sensory signal, why pay attention? It has learned all there is to learn. On the other hand, what if you are just listening to white noise? There literally is no structure in the signal. No predictions succeed, except if you get astronomically lucky. Thus, in these cases, it has not been evolutionarily advantageous to dispense beauty. Dispense arousal or hunger instead. Time and energy is valuable. But if the sensory signal lies in that magical zone where surprise and predictability are in balance? Or, even better, a signal where you uncover further structure as you engage, and where your predictions get better and better with time? Then you should keep paying attention. And so evolution has made sure that you feel beauty.

This gives us good sense of how beauty is like physics. An object cannot be beautiful if it does not have a delicately tuned moderate level of pattern-complexity. A piece of art can be objectively bad because it violates this.

But subjectivity quickly comes right back into the picture. Because what does moderate mean? A musician or composer spending their whole life listening to, analyzing, and crafting music will have trained their ear and brain to a whole different level than someone who occasionally listens to music. Things that are predictable for the composer might not be for the casual listener. Furthermore, there are surely genetic influences here too. We have different capabilities for pattern recognition. Complicating things further, we might have different spectra of skills. There is probably no single convenient number determining how good we are at pattern recognition. Probably some of us are good at pattern matching certain types of patterns over others, perhaps due to small differences in our sensory hardware. So beauty is determined by both the objective level of complexity emitted by the artwork, but also the capability for decoding this complexity by the beholder’s brain. When there is a mismatch, there is no beauty.

It is not random that it was Arnold Schoenberg, a master of tonal music, who wanted to break free of the chains of tonal music. It probably all had become too predictable for him, due to his mastery of tonal music. Since Schoenberg lived before the age of electronic music, he could not easily go wild in the space of musical texture and timbre. So he had to produce a sufficient level of novelty by breaking the rules in dimensions of pattern he had easy access to: harmony and melody. We see the porn aspect of beauty in play here. Medium complexity sensory signals effectively become low complexity when the brain has trained enough on this class of signals. Everything becomes a cliché. Novelty wears off. But beauty is quite literally optimal levels of decodable novelty, so that’s a problem. This effect is illustrated at a smaller scale when you get sick of a song that you used to love because you’ve listened to it too many times.

Now, all of this makes perfect sense in light of predictive processing theory. I’ll let Wikipedia do the summary:

In neuroscience, predictive coding (also known as predictive processing) is a theory of brain function which postulates that the brain is constantly generating and updating a "mental model" of the environment. According to the theory, such a mental model is used to predict input signals from the senses that are then compared with the actual input signals from those senses. Predictive coding is member of a wider set of theories that follow the Bayesian brain hypothesis.

So in this language, beauty is the reward for successful predictive processing at the edge of our capability. While I had this idea on my own while making art and thinking about the process, I am most definitely late to the party, as I suspected but did not know for sure when I wrote Beauty as entropic fine-tuning. Researchers are increasingly interested in the connection between aesthetics and predictive coding. As Frascaroli et al. write in the 2023 paper Aesthetics and predictive processing: grounds and prospects of a fruitful encounter

Several psychologists and neuroscientists are in fact pointing out that the pleasurable experience that we associate with beauty or successful aesthetic encounters might be related to the discovery of patterns in our sensorium (see e.g. [10–17]). In fact, in one of the pioneering works in neuroaesthetics, Ramachandran & Hirstein [18] were already putting forward the hypothesis that (as Armstrong & Detweiler-Bedell [15, p. 311] aptly put it) ‘the brain rewards progress toward organizing the perceptual field into a meaningful configuration'. Artworks, then, according to them too, would afford this experience to an enhanced degree: they would highlight our ability to arrange our sensorium into meaningful configurations.

[…]

What we call aesthetic pleasure, so the PP story suggests, is the positive affective feedback that we get when we are more successful than usual in making sense of our environment (or, in PP terms, in reducing prediction error; see [45] for a more detailed exposition). Aesthetic pleasure is, in other words, the mark of a cognitive and existential conquest.

To try to further convince ourselves that this theory is right, let us quickly understand why some common artistic techniques work.

Why do we like rhymes? Because it provides a natural structure that allows frequent successful, but still not too easy, predictions.

the man was in luck
he was sure they would ….

With this structure, the brain can easily narrow down to a few options that fill in the blank. Yet it is not completely trivial. Maybe the man is watching his rival sports team, and the next word is suck? Or just maybe…

What about ornaments? As you look at a couple of individual units of ornament on a building, you can form your prediction that this shape will appear again with certain intervals. And as you move your gaze around the building, your prediction is repeatedly verified. Symmetry serves an obviously similar function. It lets you make global predictions from local data. And then of course, the same with repetition. In music, chorus-verse structure allows a balance of surprise and predictability. Surprise in transition between chorus and verse, prediction in return to chorus or verse.

If you got this far and is get value out of what you are reading, perhaps you would consider clicking the like button? It would greatly help my little Substack, and it would be much appreciated.

Why art is like grammar

Why do genres of art develop grammar? The answer is quite clear at this point. The possible number of outcomes in the space of sensory signals is very large. If you freely roam this space, the beholder will never get predictions right. By developing grammar, you restrict the outcome space. You develop protocols so that the art can push complexity higher without becoming too unpredictable. It is a focusing of complexity into a smaller number of dimensions. This lets the artist push up the complexity in these dimensions, because now the beholder knows the complexity will be located there, so their attention and prediction making apparatus is focused on the right place. An exercise to hammer this home for a musician would be to try to make a piece of music where you frequently shift the complexity between melody, texture, and rhythm. It doesn’t work unless you build new grammars to warn the listeners of these shifts.

Artistic grammar also encodes a whole slew of useful heuristics. The grammar is there to make it easier to make good artistic decisions. Certain patterns stack better together — they do not interfere with each other in adverse ways. Type A patterns can be added without making type B patterns harder to digest, etc.

What is poor taste

If I say that Jim has poor taste, what do I mean? Zvi already gave a very good list, if you assume that objective Quality exists and you accept the grammar theory. Here I’ve tried to argue at a more basic level why grammar and physics are in fact good models for beauty, solidifying his list. But also let me describe some ways to look at poor taste that rely directly on our insights here, and that furthermore emphasizes how taste is analogous to porn.

If I say Jim has poor taste, I might really be deriding Jim for liking art that is too predictable, i.e. too low on complex pattern. Maybe his sensory prediction capabilities are very low, so that he needs this level of predictability for pleasure. Or I might be scorning Jim for consuming art with too much constant gratification — where no patience is paid in order to cash out a larger amount of complex prediction later on, with a larger payoff in beauty. It might also be that Jim has a fairly low aesthetic sensitivity, even with further training, and that he quickly saturates at the level of aesthetic satisfaction he is capable of having. So he has no ability to discriminate well from the perspective of someone more sensitive. These different levels of sensitivity were nicely discussed by sympathetic opposition, so you can go there for a discussion of this.

But, there is one really important thing to remember. I might really just be mistaken about Jim’s taste. Jim might be much more sensitive to some dimension of pattern that I am not sensitive too. When there is a large group of people who clearly care a lot about some art and you have little reason to suspect status games, then there probably is beauty there. The problem is either that you cannot access it yet, that you have graduated past it, or that your genetics leaves it locked down forever.