Experimenting with webcam overlay. Video recorded using gstreamer, source for screencaster here (screensave.c).
UPDATE, here’s another from a different angle to appease douglas.
There is something about artist-programmers, the way they’re caught using general purpose languages and tools in specific, unusual circumstances. Many of the basic assumptions underlying the development of these general purpose systems, such as errors are bad, the passing of time need not be structured only minimised, standards and pre-defined plans are good, etc, often just don’t apply. It’s not that artist-programmers can get away with being bad programmers. Far from it, in my opinion they should be fluent with their language, it’s no good being baffled by syntax errors and spaghetti code while you’re trying to work out some weird idea. However if you are following your imagination as part of a creative process, then established and fashionable software development methods often look arbitrary and inhibiting.
The last few days I’ve been thinking about revision control. Revision control systems are really excellent and have a great deal to offer artist-programmers, particularly those working in groups. What I’ve been wondering though is whether they assume a particular conception of time that doesn’t always apply in the arts.
Consider a live coder, writing software to generate a music performance. In terms of revision control they are in an unusual situation. Normally we think of programmers making revisions towards a final result or milestone, at which point they ship. For live coders, every revision they make is part of the final result, and nothing gets shipped, they are already the end users. We might somewhat crassly think about shipping a product to an audience, but what we’re `shipping’ them isn’t software, but a software development process, as musical development.
Another unusual thing about live coding revisions is that whereas software development conventionally begins with nothing and finishes with a complete, complex structure, a live coder begins and ends with nothing. Rather than aim for a linear path towards a predefined goal, musicians instead are concerned with how to return to nothing in a satisfying manner. Indeed perhaps the biggest problem for Live Algorithms is the problem of how to stop playing. The musician’s challenge is both how to build and how to deconstruct.
There are two ways of thinking about time, either as a linear progression and as a recurrent cycle or oscillation. Here’s a figure from the excellent book Rhythms of the Brain by György Buzsáki:
“Oscillations illustrate the orthogonal relationship between frequency and time and space and time. An event can repeat over and over, giving the impression of no change (e.g., circle of life). Alternatively, the event evolves over time (pantha rei). The forward order of succession is a main argument for causality. One period (right) corresponds to the perimeter of the circle (left).” (pg. 7)
This illustrates nicely that these approaches aren’t mutually exclusive, they’re just different ways of looking at the same thing. Indeed it’s normal to think of conventional design processes as cycles of development, with repeating patterns between milestones. It’s not conventional to think of the code itself ending up back where it started however, but this can happen several times during a music performance, we are all familiar with chorus and verse structure for example, and performances necessarily begin and end at silence.
So where am I going with this? I’m not sure, but I think there’s plenty of mileage in rethinking revision control for artist-programmers. There’s already active, radical work in this area, for example the code timeline scrubbing in field looks awesome, and Julian Rohrhuber et al have some great research on time and programming, and have worked on non-linear scheduling of code changes in SuperCollider.
As far as I can see though, the revision control timeline has so far been treated as a linear structure with occasional parts branching and remeeting the main flow later on. You do sometimes get instances of timelines feeding back on themselves, a process called backporting, but this is generally avoided, only done in urgent circumstances such as for applying security fixes to old code.
What if instead, timelines were of cycles within cycles, with revision control designed not to aid progression towards future features, but help the programmer wrestle their code back towards the state it was in ten minutes ago, and ten minutes before that? Just questions for now, but I think there is something to be done here. After all, there is something about artist-programmers, the way they’re caught using general purpose languages and tools in specific, unusual circumstances.
The Text live coding workshop went really well, surprisingly well considering it was the first time anyone apart from me had used it and (so I found out after) most of the participants didn’t have any programming experience. The six participants took to the various combinators surprisingly quickly, the main stumbling block being getting the functions to connect in the right way… Some UI work to do there, and I got some valuable feedback on it.
Once the participants had got the hang of things on headphones, we all switched to speakers and the seven of us played acid techno for an hour or so together, in perfect time sync thanks to netclock. Here’s a mobile phone snippet:
The sound quality doesn’t capture it there, but for me things got really interesting musically, and it was fun walking around the room panning between the seven players…
I’ve updated Text a bit to improve the visual representation of higher order types (you’d probably need to full screen to view):
I won’t be touching this until after the workshop on Saturday.
I’ve also made the source for the visual interface available here under the GPLv3 free license. To get it actually working as above you’d also need to install my tidal library, Jamie Forth’s network sync, my sampler, the nekobee synth, and somehow get it all working together. In short, it’s a bit tricky, I’ll be working on packaging soonish though.
I’ve been rather busy writing lately, my PhD funding runs out in April, and I hope by then I’ll have finished and will be looking for things to do next.
I have had a bit of time to make Text, a visual language I mentioned earlier, a bit more stable, here’s a test run:
A bit of a struggle, partly due to the small screen area I gave myself for the grab, but also due to some UI design issues I need to sort out before my workshop at Access Space in Sheffield next week, on the 5th February. Access Space is a really nice free media lab, but will turn nasty unless I free the workshop software, so expect a release soon.
In case someone is interested, here’s the linux commandline I use to record a screencast with audio from jackd:
gst-launch-0.10 avimux name=mux \
! filesink location=cast.avi \
ximagesrc name=videosource use-damage=false endx=640 endy=480 \
! video/x-raw-rgb,framerate=10/1 \
! videorate \
! ffmpegcolorspace \
! videoscale method=1 \
! video/x-raw-yuv,width=640,height=480,framerate=10/1 \
! queue \
! mux. \
jackaudiosrc connect=0 name=audiosource \
! audio/x-raw-float,rate=44100,channels=2,depth=16 \
! audioconvert \
! queue \
Text is a experimental visual language under development. Code and docs will appear here at some point, but all I have for now is this video of a proof of concept.
It’s basically Haskell but with syntax based on proximity in 2D space, rather than adjacency. Type compatible things connect automatically, made possible though Haskell’s strong types and currying. I implemented the interface in C, using clutter, and ended up implementing a lot of Haskell’s type system. Whenever something changes it compiles the graph into Haskell code, which gets piped to ghci. The different colours are the different types. Stripes are curried function parameters. Lots more to do, but I think this could be a really useful system for live performance.
I wrote a paper with Dave Griffiths and Nick Collins on the visualisation of live code, exploring ideas around live coding interfaces, accepted for the EVA London 2010 conference in July. A HTML version is below, or see the PDF Preprint.
Alex McLean (Goldsmiths), Dave Griffiths (FoAM), Nick Collins (University of Sussex) and Geraint Wiggins (Goldsmiths)
In this paper we outline the issues surrounding live coding which is projected for an audience, and in this context, approaches to code visualisation. This includes natural language parsing techniques, using geometrical properties of space in language semantics, representation of execution flow in live coding environments, code as visual data and computer games as live coding environments. We will also touch on the unifying perceptual basis behind symbols, graphics, movement and sound.
I’ve been through a few linux distros over the years, neatly getting progressively easier to install and configure as I get less willing to spend time recompiling kernels, culminating in ubuntu, enjoying the attention to detail and simplicity of use. Recently though, I’ve had to give ubuntu up and go back upstream to the rather higher maintenance Debian again. Linux suffers from creeping featurism in its layers of audio APIs, it started with OSS, a straightforward API based on files, then came ALSA, a wildly complex API with broken documentation in a wiki you can’t edit, and an architecture that somehow means only one OSS application can write sound at a time. It seems to me that it’s a failing of ALSA that further layers of abstraction are piled on top of it, creating a rather complex landscape for sound hackers to navigate.
Ubuntu has joined in the fun by shipping with PulseAudio, which is probably great for general users but a pain for those needing to work with audio on a low level without using loads of CPU. Pulse is not straightforward to remove, and when I removed it had problems with volume controls not working, and the likelihood that future system upgrades wouldn’t work so well. That’s why I switched to debian sidux, but then I couldn’t get laptop hibernation, or my firewire sound card working, and had the stress of maintaining an unstable distribution.
However this week Puredyne carrot and coriander came out, and it’s really great. The kernel is optimised for realtime sound, and jack audio runs solidly without any drop outs, something I haven’t seen before. My firewire sound works reliably, better than I managed under ubuntu. It has a really nice logo and clean look, with no plump penguins in sight. It comes with all the best a/v software beautifully packaged, including all the live coding languages. The people behind it are super friendly and helpful. It’s downstream from ubuntu, so all the software is available. It’s a dream!
They make a big deal out of it being good for booting off a USB key, and I think have worked out some nice practicalities of working that way. This makes it great for doing workshops and running linux in a non-linux lab etc. It installs and works just as nicely on a permanent hard drive though, and that’s what I’ve done.
Anyway, heartily recommended, a dream come true, congratulations to all those involved.
I’m giving a paper at the CHArt conference in Birkbeck tomorrow. I’ll edit it a little after the conference for publication, but here’s a draft of the paper, here’s the presentation (which I’m currently editing) and here’s the abstract:
Programmers do their work by writing — a piece of software is a structure made from words. These structures are generally too big to comprehend in their entirety, so programmers instead focus on small detail and overall plans; zooming in to find parts to combine and simplify and zooming out to find places to build. But this is not architecture: these structures are more like machines than static buildings. A programmer’s work is set in motion by a program interpreter, with information flowing in and around processing units before being directed outward in response.
Usually a programmer will write some text, and then step back to start it up, watch it work and decide upon the next edit. Live coding programmers however work on their software while it is running, as if they were modifying a machine without switching it off first. Because software is built from words, this is done by editing it as text, adding new routines or changing the character of an existing one. Such a change takes immediate effect, allowing fast creative feedback.
Where a written novel exists to describe human activity, written software exists to simulate it. Therefore the live coder can take the role of an artist, constructing simulators in order to generate patterns of movement, either as music, video animation or both. This can be done in front of a live audience, so that the process of writing software becomes the process of improvising music or video in performance art.
Programmers are finally taking to the stage. Introspecting and encoding their musical thoughts before an audience. A tradition of live coding has quickly formed where computer screens are projected, making the programmer’s reactions to their work visible. Questions of authorship disappear; the performance is live, the programmer improvising through the medium of written language.
I’m still working on my upgrade, hoping to be a proper PhD student by the end of this month…
I’m excited to be organising an evening called transfer at Goldsmiths, London on Friday October 16th. It’ll be fantastic with all my favourite electronic music superheroes — Leafcutter John, Finn Peters, Yeeking and friends, Sarah Angliss as well as slub. Tickets now on sale!
Congratulations to those who made it to the end of the hackpact, sorry for breaking my part of the bargain but I really had to get this draft done, and couldn’t justify taking time away from my family without also taking time away from my nightly practice. Looking forward to spending time on other parts of my life this month, including a rather looser aim of recording and editing one track every week. If I manage it there’ll be an EP at the end of it.