Category: visualisation
Text
Text is a experimental visual language under development. Code and docs will appear here at some point, but all I have for now is this video of a proof of concept.
It’s basically Haskell but with syntax based on proximity in 2D space, rather than adjacency. Type compatible things connect automatically, made possible though Haskell’s strong types and currying. I implemented the interface in C, using clutter, and ended up implementing a lot of Haskell’s type system. Whenever something changes it compiles the graph into Haskell code, which gets piped to ghci. The different colours are the different types. Stripes are curried function parameters. Lots more to do, but I think this could be a really useful system for live performance.
Visualisation of Live Code
I wrote a paper with Dave Griffiths and Nick Collins on the visualisation of live code, exploring ideas around live coding interfaces, accepted for the EVA London 2010 conference in July. A HTML version is below, or see the PDF Preprint.
Alex McLean (Goldsmiths), Dave Griffiths (FoAM), Nick Collins (University of Sussex) and Geraint Wiggins (Goldsmiths)
Abstract
In this paper we outline the issues surrounding live coding which is projected for an audience, and in this context, approaches to code visualisation. This includes natural language parsing techniques, using geometrical properties of space in language semantics, representation of execution flow in live coding environments, code as visual data and computer games as live coding environments. We will also touch on the unifying perceptual basis behind symbols, graphics, movement and sound.
Acid sketching
I’ve been thinking about visual languages and the morphology of symbols (as opposed to words) for a while. I had the opportunity to start putting some of these ideas into code at a really excellent openframeworks workshop this week, run by Joel Gethin Lewis and Arturo Castro.
Here’s what it does:
Makes the point nicely that symbols and spaces can intertwine.
Using opencv blob detection, the regularity, direction and area of the shapes map to envelope modulation, resonance and pitch. The drawing is then sequenced into a melody using the minimum spanning tree (from the boost library) of the shape centroids, where distance maps to inter-onset interval.
It also has a mode for projecting the red circles and highlights back on the drawing surface which worked well.
This is only the second thing I’ve made with openframeworks, and while I don’t really get on with the codeblocks editor recommended for linux, I’m impressed with how accessible it makes opencv and all that.
Update: open frameworks sourcecode
Another update (1st August 2013): I ported this to Python, get the source here
Mary Hallock-Greenewalt
“Broadly, it is my desire to express emotions by means of timed variations of light and color in a manner analogous to that employed in the art of music. Such expression may either be for its own sake, or … as an accompaniment.”
In 1906, about 40 years after the invention of the commercial light bulb, Mary Hallock-Greenwalt (1871-1950) began work on her colour organ, the sarabet. She was an accomplished musician, but wanted to create an equivalent artform for colour which she called nourathar (derived from the Arabic for the essense/flavour/influence of light). Interestingly though, she came to the conclusion that colour wasn’t as important as brightness:
“In this art they, the darknesses and brightnesses, constitute the woof of the play. They carry a chief burden of the transmitted feeling. They also tend to make a oneness out of more than one colour, or colours, simultaneously produced.”
She was also quick to dismiss the idea of direct mappings between music and colour.
“… there is no octave to color. color has no harmonics. … Its pristine strength is such that no two colors can fit together as identical.”
While she composed nourathar pieces to accompany music, she was against cross-domain mappings in general.
“To seek to fasten the form of one art on the form of another art, is, on the face of it, a mistake, if not an impossibility. They are organically different things. They will speak in different ways.”
I’m not sure if I agree with this strong claim, the human senses are integrated after all. But I still think it insightful to reject the naive “colour scales” which others came up with — while synaesthetics can experience pitch as colour (or vice versa), I understand that no two synaesthetics experience the same scale.
All the quotes in this post came from Hallock-Greenewalt’s book “Nourathar: The Fine Art of Light-Color Playing”, which is a joy to read. She was not the first colour organist, but from what I’ve seen she was the most insightful and interesting of the bunch. Sadly however no video recordings can exist from back then, so we can only imagine what her performances could have been like, with only her notation to guide us.
I gave a quick dorkbot presentation about Mary Hallock-Greenewalt a couple of years ago, and one audience member jokingly accused me of inventing her to justify VJ culture with false history. Well her work is well documented in her writing and patent applications, but she should certainly be better known — I recently saw a talk about colour organists which didn’t mention her, despite her huge contribution to the field.
I’ll finish this post with one last quote from the woman herself.
“Is there no expressing of fervor in the deepening of a rose to red? Can quality of ardor not be suggested in the quickness or slowness with which this transition is done? Can zeal or eagerness not be expressed in the manner of change from blue to purple? Are colors not “warm” or “cold”? Is there not the fervid, the burning of intensity of feeling in the ray’s glowing into or embering back? So much there is to choose from.”
Posted on the occasion of Ada Lovelace day 2009.
Rhythm space
I’m working with Jamie Forth on ideas around spaces of rhythm. Here’s a demo (which might not work in feed readers):
The space has two quality dimensions, “intensity” (X) and “disorder” (Y). Drum patterns are arranged along these dimensions, so more intense ones are towards the left and more ordered ones towards the top.
Draw a line from a high hat to a kick drum. If you draw a short line the rhythms will be more homogenous. Certain angles have certain feels to them. Maybe. It seems a nice way of playing with polymetric rhythms as vectors anyway.
The demo by the way is coded in haxe and compiled to flash. The source code is here.
Textual patching
I wrote a perl script that allows you to compose puredata patches in a text editor. You define the patch using ASCII art like this:
*------------------------*
| .--------. \
.-x--------. | osc~ 5 | *
| osc~ 500 | `-x------' |
`-x--------' | |
| .-x------. |
| | *~ 300 | |
| `-x------' |
*---* | |
| *------------*
.-x------.
| *~ 0.2 |
`-x------'
|
*
|\
| *
| |
.-x-x--.
| dac~ |
`------'Then run the Perl script over it to produce a .pd file, that you can then load into puredata to get this:
The ASCII syntax basically allows you to define pd objects and connect them together. Layout is preserved. Much like in ghostbusters, you can’t cross the lines, and there isn’t syntax for different box types (messages and numbers). Fixing this would be short work, but I ran out of train journey 🙂
There is a particular syntax for drawing the lines. You use – for going left and right, | for going up and down and \ and / for going diagonally. To change direction or fork a wire you have to place a * . Mark inlets and outlets with x .
I think this shows nicely how there is no real difference between patching and coding. Shades of pixels are an alphabet, anything can be a program if you define a suitable interpreter to go with it.
Sadly you can’t do live patching with this, but perhaps this could be a starting point for thinking about more interesting ways of programming with text.
If you are curious you can download the script (and patch) here.
Thank you Graphviz
I love graphviz. You feed in data in a simple, easy to generate format and it creates the most beautifully laid out visualisations from it.
I’m trying to make a triangular waveguide mesh, and wasn’t sure if my code was doing the right thing, so ran neato over the data and got this:
I didn’t tell it to lay them out in a hexagon, it just did because it was the simplest way of doing so. I then tried manually adding extra connections
SoundVis
Frederic Leymarie and I have created a blog called SoundVis to document our research into the visualisation of sound and music. We’ll be adding our findings to it as time allows…
20010203 (translated)
Peano weave applied to a slub classic for Ade‘s birthday..
Higher quality AVI available at slub.org
Peano curve weaves of whole songs
Some nine months ago I played with weaving images from music, including using a peano curve as a mapping.
I’ve returned to this subject, having many good ideas to explore from recent discussions with Tim Blackwell. We thought rendering some whole songs would work nicely. I didn’t fancy playing with my Java code again so wrote some Haskell, which I’m rather pleased with. The source is available (feedback welcome!). It does the the mapping using seeks on the output file, allowing impressive memory efficiency via Haskell’s lazy evaluation.
Some examples of some indie synth pop, disco, minimal techno (*3) and industrial gabba below, click on the images for the full versions but beware, they are rather large, around 5M each. Mouseover for the original track names.
