A / V /K - AudioVisual Kinesthetic
notes and writings on things
Written by Daria Kaufman, Nov 2020
Glitch Crush
“Primeiro estranha-se, depois entranha-se”(At first it’s strange, then it gets into your veins)
- Fernando Pessoa
I am part of the Human/ID team, a collaboration with StratoFyzika and Ian Heisters that probes how identity is rendered legible (or illegible) through movement and technology.
Tagging onto the notion of digital flaws and their rich potential (which my colleague Ian wrote about in his last post), and coming from a movement/choreographic lens, here are my queries for the moment:
What happens when I map digital errors – these flaws that actually belie a kind of humanity - onto my body? What happens when I then feed that body back into the machine, and continue this cycle in a kind of reiterative loop? Do ‘I’ eventually disappear or amplify? When? Where?
- - - - -
Courtesy of video calls, I now routinely witness my friends and loved ones’ countenances glitch, freeze, and lapse. Sometimes it catches them in a really awkward moment – face and body askew, totally unrecognizable - and I feel like I'm violating them by continuing to look, like some slightly sadistic voyeur. Then I wonder if my screen’s doing the same on their end, and if we’re both simultaneously gazing into each other’s interrupted self.
I was on a video call with a close friend the other day when this happened. She had been sitting very still as we spoke, gazing upward into the camera with a kind of wide-eyed blank stare. And then I suddenly got the sense that she was frozen. Or was she?
For about 7 seconds, I didn’t know. I could have said something, like, are you frozen? or I think you just froze. But I didn’t. I just sat there. Staring. She didn’t say anything either, and we had reached a natural lull in the conversation, so the silence wasn’t out of place. But why didn’t I say anything? I guess I really liked looking at her like that. Not sure... if she was really there. Something about it let me see her better.
After so many months of this (and so many more to come), it’s starting to creep in, or more accurately, I’m feeling the desire to let it creep in - to glitch my body in an approximation of the errors I’m seeing onscreen. To freeze. To tremor. To lapse. Some might pooh-pooh this as illustration or representation, but I prefer to think of it as honoring a new liminality, one that I (and so many of us) am now confronted with nearly every day.
These errors that serve to reinforce the digital barrier, to remind me of the medium (as if I could forget) - what if, instead of regarding them as nuisances and mistakes, or problems to be fixed… what if instead, I become them?
- - - - -
So that’s where I’m at - trying to move like her (my screen body/face) when she’s in error. What can be culled from these interstitial moments? And what happens when I feed them back into the machine that created them?
Written by Ian Heisters, Sept 2020
IDENTITY AND MOVEMENT: MOTION TRANSFER WITH NEURAL NETWORKS
Deep fake dancing and breaking technology.I’ve been talking with the StratoFyzika team about how identity resides in the body for their residency in CounterPulse’s Combustible program. We’re researching surveillance technology, machine learning, and dance for a performance in spring 2020. The research is conceptual as well as practical, and the following comprises my notes in building a first prototype of an AI for the performance.
A common conception of identity might include one’s values, likes, dislikes, various personality traits. “The things that make me who I am.” How does your body make you who you are? Conversely, how does who you are make your body? What are the indelible traces of your identity in your body? Our bodies are our interfaces with reality. There is nothing we do or experience that is not mediated by the body, even in virtual or online reality.
These are not entirely philosophical or abstract questions. Surveillance technologies track us based on our gait, facial geometry, or voices. They are recognizing the traces of our identities in our bodies. Meanwhile, every aspect of our online presences can be faked using AI, as Jordan Peele notably demonstrated in what is now a quaint seeming deep fake. These technologies are leveraged as instruments of control, whether for influencing consumers to buy products, or for enforcing fascist hegemony.
One of my dance teachers was fond of quoting Martha Graham, saying “the body never lies” as an invocation of the inherent authenticity of dance and its ability to find and speak truth. To a degree, surveillance and deep fake technologies are exploiting our notion that the body never lies. Peele’s deep fake plays on our trust that if we saw Obama’s mouth moving, speaking those words in a voice that sounds like his, then he said those words. But these technologies also easily fall down when confronted with physical, bodily reality. Surveillance is easily defeated by makeup, or taping a picture of people holding umbrellas to your shirt. Indeed, early explorations for this project quickly discarded an idea for building a vocabulary of anti-surveillance movements, simply because it was so easy to confound gait detection that it became uninteresting.
The conception of the never-lying body seems simultaneously naive and like a potent vector for intervening in these systems of control. At the very least, dance can provide a perspective for complicating our assumptions about the body, truth, reality, and technology; for finding the edges and spaces between.
As a first step in investigating this space for our project, I worked with Daria and Hen to create a deep fake dance. Here’s how it works: Hen video records herself moving around so that a computer can build a visual model of her body. Meanwhile, Daria video records herself performing a dance so that a computer can build a structural model of her movement. Then, the computer applies the structure and movement from Daria to Hen’s visual representation. The result is a video of what appears to be Hen performing a dance she never performed.
This approach is based on Chan, Ginosar, Zhou, and Efros’ paper Everybody Dance Now (2019). Being an initial test, there’s a lot of room for improvement. Notably, the shadows in Hen’s target video are enough to throw off the pose estimation. My slap-dash removal of the shadows caused the fuzzy black artifacts in the synthesized output.
This video demonstrates an alarming potential for not only showing a person saying something they did not say, but doing something they did not do. But it also finds the edges of the technology, the errors that describe an open space of questions and possibility. Svetlana Boym describes technology like this as nostalgic or “off-modern”:
To err is human, says a Roman proverb. In the advanced technological lingo the space of humanity itself is relegated to the margin of error. Technology, we are told, is wholly trustworthy, were it not for the human factor. We seem to have gone full circle: to be human means to err. Yet, this margin of error is our margin of freedom. It’s a choice beyond the multiple choices programmed for us, an interaction excluded from computerized interactivity. The error is a chance encounter between us and the machines in which we surprise each other.
The art of computer erring is neither high tech nor low tech. Rather it’s broken-tech. It cheats both on technological progress and on technological obsolescence. And any amateur artist can afford it. Art’s new technology is a broken technology.
– Svetlana Boym, Nostalgic Technology: Notes for an Off-modern Manifesto, 2006
This broken technology provides a paradoxical perspective on deep fakes and motion tracking, at once butting up against the limits of the technology, problematizing the technology, and opening up a space for creative exploration. We find Boym’s chance encounter with the machine, in which we are surprised simultaneously by its fragility, its consequences, and the potentials for repurposing it in unintended ways.
To further explore this space, we’re stepping forward with another set of questions and investigations. Can Daria and Hen perform live deep fakes, coercing new performances from each others’ images? Can the source structures be algorithmically modified to produce choreographies that are collaborations between the machine and the dancers? Can the technology be developed to the point where it could fool a viewer, like a Turing test for dance?
︎︎︎ Original post link
ON PHI
Berlin,
24.10.18
Further developments of Phi may took in the future the path towards 360 video and VR experimentation, aiming to bring the liveness of the piece very close to the audience: basically around it .
Daria Kaufman:
“This work strives to move beyond a linear storyline to a more organic, evolutive and personalized experience. We always try to provide an immersive environment for the audience to get lost in, and to mine their own meaning. Sometimes in the case of a very set (almost mathematical) choreography, the meaning can only be derived through interpretation. In the case of a VR environment, we would hope to amplify the audience engagement and outreach, transporting them to the center of the action, allowing them to become a part of the piece itself based on their viewing direction. In this way, they have a more visceral, immediate experience of the performance, collapsing the usual barrier that exists between artist and viewer.”
Alessandra Leone
“After doing some test filming, the tight sync of the dancers and the mathematical reconfiguration of movement vocabulary into equal time slices became visually obvious at a glance by applying an echo effect to the footage. This type of video post-effect combines frames from different times at constant distance. The result is a beautiful construction of bodies in motion, organically drawing mathematical structures. “
Berlin, 20.10.18
From the Choreographic point of view, there were three main influences: Minimalism in sound design, particularly the tape work of Steve Reich and Terry Riley's "Persian Surgery Dervishes." This sound has a continuousness with slow evolution over time. Anne Teresa De Keersmaeker's choreography was naturally a source of inspiration since she made work to the music of Steve Reich.Similarly, Lunar Phases can represent this same idea, visually. The moon's trajectory through constellations began to present a working map for choreography.
In previous StratoFyzika works (e.g. Thaeta and AikiA), we used animated, interactive visuals. In Phi, we took what we did with video projection and translated it into an orbital lighting design to achieve a similar effect, but using only light and shadow to enhance the visual experience.