Joining the Threads

In this section, I will discuss a compositional approach that has the potential to expand on some of the ideas presented earlier, particularly the single-domain approach (described as a characteristic of connectionist models) and some limitations of symbolic AI systems in the same regard as well. I am referring to abstraction. As will be further discussed above, abstraction mainly relies on the successful translation *Translation here is understood as the process of converting information from one domain to one or other domains through symbolic relationships and mappings. translation is intrinsically tied to hermeneutics –the theory and methodology of interpretation –which originated in the analysis of sacred and philosophical texts. Hermeneutics reflects the dual capacity of language to both reveal and obscure truth. For example, in religious contexts, interpretive principles were developed to resolve textual inconsistencies and uncover deeper meanings within sacred texts. Ultimately, translation, hermeneutics, and aesthetics can be seen as processes of uncovering deeper realities through interpretive acts. For example, Hans-Georg Gadamer’s hermeneutical aesthetics builds on these theses. Gadamer argues that art’s immediate impact prompts reflection, by fostering an understanding that transcends mere aesthetic pleasure. However, Gadamer sees art as resistant to reduction into conceptual knowledge or translation into other mediums. This possibility of multiple interpretations grants the artwork an ideality of possible meanings (Hans-Georg Gadamer, The relevance of the beautiful and other essays, Cambridge UP, 1986, p.146), making it inherently more than any single reading. of information across diverse domains. Therefore, I view these two concepts as closely connected. In what comes next, I will explore how abstraction and translation can act as human cognitive capacities that have the potential to spark musical narratives.

I will outline some processes I have primarily used as vehicles for cross-domain translation of musical information and representation. As I see it, this process potentially serves to connect the threads I have so far presented independently –or rather, to intertwine them. The techniques I will discuss are all fundamentally connected and derived from processes of sonification. From there, I identify two main types: one that produces symbolic musical information and another that generates raw digital audio signals. In addition, I will discuss some applications of principles of remapping and transformation of musical profiles, as well as their utilization as structures that span across different musical domains.

Abstraction lies at the core of what potentially defines us as humans and what defines art. Therefore, it can be approached from multiple perspectives, exceeding the somewhat limited views from psychology and artificial intelligence and moving towards more philosophical, cultural, religious, and even aesthetical views. However, I will maintain a relatively narrow scope –as I have throughout this text– focusing on the topic mainly from a psychological approach and its potential emulation within computational frameworks. To maintain consistency with the structure of the previous chapters, I will begin the discussion of abstraction from a cognitive perspective.

Abstraction

The human mind is capable of identifying patterns through our sensory systems. abstraction involves isolating a common feature or relationship observed across multiple elements and often focuses on essential qualities by removing specific, individual, or recognizable details.*Kim, Jaegwon. "abstraction". Encyclopedia Britannica, 12 Jul. 2024, https://www.britannica.com/science/abstraction Accessed 24 August 2024.

Human experience often involves sensory and motor dimensions that are closely interconnected. When operating in the real world, humans must deal with sensory uncertainty and efficiently process continuous streams of multisensory information. The capacity for abstraction in humans is likely supported by the integration of sensory-motor associations and multimodal processing areas in the brain,*Multimodal processing areas in the brain are regions that integrate information from multiple sensory modalities. One example of a multimodal processing area is thesuperior temporal sulcus superior temporal sulcus (STS), which integrates auditory and visual information, like speech and facial expressions which together enable the representation of concepts.*L. Fernandino et al., "Concept Representation Reflects Multimodal Abstraction: A Framework for Embodied Semantics," Cereb Cortex 26, no. 5 (May 2016), https://doi.org/10.1093/cercor/bhv020. In humans, multisensory integration develops over time, beginning with basic processing abilities and gradually specializing in more complex sensorimotor functions.*David J. Lewkowicz, "Early experience and multisensory perceptual narrowing," Developmental Psychobiology 56, no. 2 (2014), https://doi.org/https://doi.org/10.1002/dev.21197.

Sometimes, the neural pathways responsible for processing different types of sensory information in the brain form unusual cross-connections. One rare manifestation of this is synesthesia, a perceptual phenomenon where stimulation of one sensory pathway involuntarily triggers experiences in another, for example, seeing colors when hearing music. A notable case of synesthesia is the composer Olivier Messiaen, who suggested that his synesthetic experiences influenced his compositional work:

“And I realized that (…) I also associate colors with sounds, but intellectually, not with my eyes. In fact, for as long as I can remember, whenever I hear or read music, I see in my mind complexes of colors that move and shift with the sound complexes.”

For example, he writes about the piece Couleurs de la Cité Céleste (1963):

“The form of this work depends entirely on colors. The melodic or rhythmic themes, the combinations of sounds and timbres, evolve in the manner of colors. In their perpetually renewed variations, one can find (by analogy) warm and cool colors, complementary colors influencing their neighbors, colors fading toward white, or subdued by black. These transformations can also be compared to characters acting on several overlapping stages, simultaneously unfolding multiple different stories.”

Fragment of Messiaen’s Couleurs de la Cité Céleste. Notice the texts describing colors for different chords.* O. Messiaen, "Couleurs de la Cité Céleste: Orchestra Score," 15.

As discussed previously, connectionist models, particularly neural networks, are very good at identifying patterns and generalizing them. However, they cannot directly apply this knowledge across different domains. They are only effective within their specific domain. Recent research carried out in the field of AI has addressed this problem, and some results are starting to show that new, more advanced systems can effectively transfer and operate on multimodal information. These systems, however, typically rely on separate and independent mechanisms for processing unimodal representations, where each modality operates without influencing the others.*Pablo Barros et al., "Expectation learning for adaptive crossmodal stimuli association," arXiv preprint arXiv:1801.07654 (2018). They normally consist of separate neural networks trained on parallel datasets –each from a different modality– and establish some mapping between their intermediate layers and posterior training of a new model based on these abstract mappings. *Seungwhan Moon, Suyoun Kim, and Haohan Wang, "Multimodal transfer deep learning with applications in audio-visual recognition," arXiv preprint arXiv:1412.3121(2014). The field of multimodal learning in AI and deep learning is advancing rapidly. Achieving strong multimodal learning capabilities could bring AI models one step closer to replicating human intelligence and reasoning. For now, however, this goal remains largely unrealized.

In contrast, humans can efficiently abstract information from one domain and reinterpret it in another. In some cases, abstraction simplifies complex forms into simpler, often non-representational elements. In other cases, it goes beyond a mere reduction of detail and becomes a re-interpretation of meaning and essence: it becomes a form of reconstruction and reshaping of meaning across different sensory modalities, mediums, languages, or modes of expression. It allows engagement with forms and concepts on a more general level without directly referencing reality. At the core of abstraction lies the notion of interpretive engagement, as perceivers mentally reconstruct and interpret what they see or hear through their own experiences and understanding. This ability to perform cross-modal translations has been a powerful driving force in artistic production. Abstraction, therefore, lies at the core of what we understand as art.

“While art enacts its separation from nature through its artificial character, it simultaneously conceals an inherent dependence on natural beauty. Art appears as the opposite of nature—what is not made—yet ‘as pure antithesis… each refers to the other: nature to experience of a mediated and objectified world, the artwork to nature as the mediated plenipotentiary of immediacy.’”

The use of technology has the potential to enhance our creative capacity for abstraction and enable explorations of diverse forms of translation, particularly in domains involving large volumes of information. In the following sections, I will discuss how technology is utilized in this project to explore processes of abstraction and the translation of information across domains, mainly resulting in sound and musical representation.