Category: haskell

Haskell patterns ad nauseam

TL;DR I’m now describing algorave music as functions from time ranges to lists of events, with arbitrary time precision, where you can query continuously varying patterns for more detail by specifying narrower time ranges.

For more practical demo-based description of my current system see this post.

I’ve been restructuring and rewriting my Haskell pattern library for quite some time now. I’ve just done it again, and thought it would be a useful point to compare the different approaches I’ve taken. In all of the following my underlying aim has been to get people to dance to my code, while I edit it live (see this video for an example). So the aim has been to make an expressive language for describing periodic, musical structures quickly.

First some pre-history – I started by describing patterns with Perl. I wrote about this about ten years ago, and here’s a short video showing it in action. This was quite frustrating, particularly when working with live instrumentalists — imperative language is just too slow to work with for a number of reasons.

When I first picked up Haskell, I tried describing musical patterns in terms of a tree structure:

data Event = Sound String
           | Silence
data Structure = Atom Event
               | Cycle [Structure]
               | Polymetry [Structure]

(For brevity, I will just concentrate on the types — in each case there was a fair amount of code to allow the types to be composed together and used).

Cycles structure events into a sequence, and polymetries overlay several structures, which as the name suggests, may have different metres.

The problem with this structure is that it doesn’t really lend itself to live improvisation. It represents musical patterns as lists embedded within lists, with no random access — to get at the 100th metric cycle (or musical loop) you have to generate the 99 cycles before it. This is fine for off-line batch generation, but not so good for live coding, and is restrictive in other ways — for example transforming events based on future or past events is awkward.

So then I moved on to representing patterns as functions, starting with this:

data Pattern a = Pattern {at :: Int -> [a], period :: Int}

So here a pattern is a function, from integers to lists. This was quite a revelation for me, and might have been brought on by reading Conal Eliot’s work on functional reactive programming, I don’t clearly remember. I still find it strange and wonderful that it’s possible to manipulate this kind of pattern, as a trivial example reversing it, without turning it into a list of first order values first. Because these patterns are functions from time to values, you can manipulate time without having to touch the values. You can still generate music from recursive tree structures, but with functions within functions instead of in the datatypes. Great!

In the above representation, the pattern kept note of its “period”. This was to keep track of the duration of the cycle, useful when combining patterns of different lengths. This made things fiddly though, and was a code smell for an underlying problem — I was representing time with an integer. This meant I always had to work to a predefined “temporal atom” or “tatum”, the lowest possible subdivision.

Having a fixed tatum is fine for acid house and other grid-based musics, but at the time I wanted to make structures more expressive on the temporal level. So in response, I came up with this rather complex structure:

data Pattern a = Atom {event :: a}
                 | Arc {pattern :: Pattern a,
                        onset :: Double,
                        duration :: Maybe Double
                 | Cycle {patterns :: [Pattern a]}
                 | Signal {at :: Double -> Pattern a}

So lists are back in the form of Cycles. However, time is represented with floating point (Double) values, where a Cycle is given a floating point onset and duration as part of an Arc.

Patterns may also be constructed as a Signal, which represents constantly varying patterns, such as sinewaves. I found this a really big deal – representing discrete and continuous patterns in a single datatype, and allowing them to be composed together into rich structures.

As with all the other representations, this did kind of work, and was tested and developed through live performance and audience/collaborator feedback. But clearly this representation had got complex again, so had the supporting code, and the use of doubles presented the ugly problem of floating point precision.

So simplifying again, I arrived at this:

  data Pattern a = Sequence {arc :: Range -> [Event a]}
                 | Signal {at :: Rational -> [a]}
  type Event a = (Range, a)
  type Range = (Rational, Rational)

This is back to a wholly higher-order representation and is much more straightforward. Now we have Sequences of discrete events (where each event is a value which has a start and end time), and Signals of continuously varying values. Time is now represented as fractions, with arbitrary precision. An underlying assumption is that metric cycles have a duration of 1, so that all time values with a denominator of 1 represent the end of one cycle and the beginning of the next.

A key insight behind the above was that we can represent patterns of discrete events with arbitrary temporal precision, by representing them as functions from time ranges to events. This is important, because if we can only ask for discrete events occurring at particular points in time, we’ll never know if we’ve missed some short-lived events which begin and end in between our “samples” of the structure. When it comes to rendering the music (e.g. sending the events to a synthesiser), we can render the pattern in chunks, and know that we haven’t missed any events.

At this point, things really started to get quite beautiful, and I could delete a lot of housekeeping code. However, I still wasn’t out of the woods..

Having both Sequence and Signal part of the same type meant that it was somehow not possible to specify patterns as a clean instance of Applicative Functor. It meant the patterns could “change shape” when they are combined in various ways, causing problems. So I split them out into their own types, and defined them as instances of a type class with lots of housekeeping functions so that they could be treated the same way:

data Sequence a = Sequence {range :: Range -> [Event a]}
data Signal a = Signal {at :: Time -> [a]}

class Pattern p where
  pt :: (p a) -> Time -> [a]
  atom :: a -> p a
  silence :: p a
  toSignal :: p a -> Signal a
  toSignal p = Signal $ \t -> pt p t
  squash :: Int -> (Int, p a) -> p a
  combine' :: p a -> p a -> p a
  mapOnset :: (Time -> Time) -> p a -> p a
  mapTime :: (Time -> Time) -> p a -> p a
  mapTime = mapOnset
  mapTimeOut :: (Time -> Time) -> p a -> p a

I’ll save you the instance declarations, but things got messy. But! Yesterday I had the insight that a continuous signal can be represented as a discrete pattern, which just gets more detailed the closer you look. So both discrete and continuous patterns can be represented with the same datatype:

type Time = Rational
type Arc = (Time, Time)
data Pattern a = Pattern {arc :: Arc -> [Event a]}

Much simpler! And I could delete about half of the supporting code. Here’s an example of what a “continuous” pattern looks like:

sig :: (Time -> a) -> Pattern a
sig f = Pattern f'
  where f' (s,e) | s > e = []
                 | otherwise = [((s,e), f s)]

sinewave :: Pattern Double
sinewave = sig $ \t -> sin $ pi * 2 * (fromRational t)

It just gives you a single value for the range you ask for (the start value in the range, although on reflection perhaps the middle one or an average value would be better), and if you want more precision you just ask for a smaller range. If you want a value at a particular point, you just give a zero-length range.

I’ve found that this representation actually makes sense as a monad. This has unlocked some exciting expressive possibilities, for example taking one pattern, and using it to manipulate a second pattern, in this case changing the density of the pattern over time:

listToPat [1%1, 2%1, 1%2] >>= (flip density) (listToPat ["a", "b"])

Well this isn’t fully working yet, but I’ll work up some clearer examples soon.

So I hope that’s it for now, it’s taken me a ridiculous amount of effort to get to this point, and I’ve ended up with less code than I begun with. I’ve found programming with Haskell a remarkably humbling experience, but an enjoyable one. I really hope that this representation will stick though, so I can concentrate more on making interesting functions for transforming patterns.

In case you’re wondering what the mysterious “a” type is in the above definitions of “Pattern a“, well of course it could be anything. In practice what I end up with is a pattern of hashes, which represent synthesiser control messages. I can represent all the different synthesiser parameters as their own patterns (which are of different types depending on their function), and combine them into a pattern of synthesiser event, and manipulate that further until they eventually end up with a scheduler which sends the messages to the synth. For a close up look at an earlier version of my system in use, here’s a video.

The current state of the sourcecode is here if you fancy a look, I’ve gone back to calling it “tidal”. It’s not really in a state that other people could use it, but hopefully one day soon.. Otherwise, it’s coming to an algorave near you soon.

As ever, thanks to those who have given me advice along the way.

Patterns again

Back to patterns in Haskell, an unruly puzzle that’s run through the last few years of my life, trying to work out how I want to represent my music.  Here’s the current state of my types:

  data Pattern a = Sequence {arc :: Range -> [Event a]}
                 | Signal {at :: Rational -> [a]}
  type Event a = (Range, a)
  type Range = (Rational, Rational)

A Range is a time range, with a start (onset) and duration.  An Event is of some type a, that occurs over a Range.  A Pattern can be instantiated either as a Sequence or Signal.  These are directly equivalent to the distinction between digital and analogue, or discrete and continuous.  A Sequence is a set of discrete events (with start and duration) occurring within a given range, and a Signal is a set of values for a given position in time.  In other words, both are represented as functions from time to values, but Sequence is for representing a set of events which have beginnings and ends, and Range is for a continuously varying set of values.

This is a major improvement on my previous version, simply because the types are significantly simpler, which makes the code significantly easier to work with.  This simplicity is due to the structure of patterns being represented entirely with functional composition, so is closer to my (loose) understanding of functional reactive programming..

The Functor definition is straightforward enough:

  mapSnd f (x,y) = (x,f y)
  instance Functor Pattern where
    fmap f (Sequence a) = Sequence $ fmap (fmap (mapSnd f)) a
    fmap f (Signal a) = Signal $ fmap (fmap f) a

The Applicative definition allows signals and patterns to be combined in in a fairly reasonable manner too, although I imagine this could be tidied up a fair bit:

  instance Applicative Pattern where
    pure x = Signal $ const [x]
    (Sequence fs) <*> (Sequence xs) = 
      Sequence $ \r -> concatMap
                       (\((o,d),x) -> map
                                      (\(r', f) -> (r', f x))
                                        (\((o',d'),_) -> (o' >= o) && (o' < (o+d)))
                                        (fs r)
                       (xs r)
  (Signal fs) <*> (Signal xs) = Signal $ \t -> (fs t) <*> (xs t)
  (Signal fs) <*> px@(Sequence _) = 
    Signal $ \t -> concatMap (\(_, x) -> map (\f -> f x) (fs t)) (at' px t)
  (Sequence fs) <*> (Signal xs) = 
    Sequence $ \r -> concatMap (\((o,d), f) -> 
                                map (\x -> ((o,d), f x)) (xs o)) (fs r)

In the Pattern datatype, time values are represented using Rational numbers, where each whole number represents the start of a metrical cycle, i.e. something like a bar.  Therefore, concatenating patterns involves ‘playing’ one cycle from each pattern within every cycle:

  cat :: [Pattern a] -> Pattern a
  cat ps = combine $ map (squash l) (zip [0..] ps)
    where l = length ps
  squash :: Int -> (Int, Pattern a) -> Pattern a
  squash n (i, p) = Sequence $ \r -> concatMap doBit (bits r)
    where o' = (fromIntegral i)%(fromIntegral n)
          d' = 1%(fromIntegral n)
          cycle o = (fromIntegral $ floor o)
          subR o = ((cycle o) + o', d')
          doBit (o,d) = mapFsts scaleOut $ maybe [] ((arc p) . scaleIn) (subRange (o,d) (subR o))
          scaleIn (o,d) = (o-o',d* (fromIntegral n))
          scaleOut (o,d) = ((cycle o)+o'+ ((o-(cycle o))/(fromIntegral n)), d/ (fromIntegral n))
  subRange :: Range -> Range -> Maybe Range
  subRange (o,d) (o',d') | d'' > 0 = Just (o'', d'')
                       | otherwise = Nothing
    where o'' = max o (o')
          d'' = (min (o+d) (o'+d')) - o''
  -- chop range into ranges of unit cycles
  bits :: Range -> [Range]
  bits (_, 0) = []
  bits (o, d) = (o,d'):bits (o+d',d-d')
    where d' = min ((fromIntegral $ (floor o) + 1) - o) d

Well this code could definitely be improved..

If anyone is interested the code is on github, but is not really ready for public consumption yet.  Now I can get back to making music with it though, more on that elsewhere, soon, maybe under a new pseudonym..


I’ve got some sounds out of my new live coding system, codenamed “smoothdirt”.  Here’s an mp3 for you.  The sounds are triggered with some C and structured and scheduled with some Haskell.  Plenty more to do, but already really happy hearing embedded juxtoposition of timescales, smooth multichannel panning (2 channels in this test, but I’m playing on a quadrophonic cinema soundsystem at lovebytes) and sample accuracy, which I test at the end by playing a kick drum sample a lot.

My new representation also allows me to treat musical structure as both a discrete pattern and a continuous signal, which I’m very happy about, but haven’t explored the depths of yet..

Anyway with a few tweaks and effects it’ll be ready for the algorave in London this weekend.

Patterns in Haskell revisited

A while back I came up with this way of representing musical patterns as pure functions in Haskell:

data Pattern a = Pattern {at :: Int -> [a], period :: Int}
These patterns can be composed nicely with pattern combinators, creating strange polyrhythmic structures, see my earlier post for info.
This turned out just great for representing acid techno, see for example this video of people dancing to Dave and I.  I was using Tidal which uses a representation similar to the above (and Dave was using his lovely SchemeBricks software).
However lately I’ve been wanting to make music other than acid techno, in particular in preparation for a performance with Hester Reeve, a Live Artist.

After a lot of fiddling about, I seem to be settling on this:

Read more

PhD Thesis: Artist-Programmers and Programming Languages for the Arts

With some minor corrections done, my thesis is finally off to the printers.  I’ve made a PDF available, and here’s the abstract:

We consider the artist-programmer, who creates work through its description as source code. The artist-programmer grandstands computer language, giving unique vantage over human-computer interaction in a creative context. We focus on the human in this relationship, noting that humans use an amalgam of language and gesture to express themselves. Accordingly we expose the deep relationship between computer languages and continuous expression, examining how these realms may support one another, and how the artist-programmer may fully engage with both.

Our argument takes us up through layers of representation, starting with symbols, then words, language and notation, to consider the role that these representations may play in human creativity. We form a cross-disciplinary perspective from psychology, computer science, linguistics, human-computer interaction, computational creativity, music technology and the arts.

We develop and demonstrate the potential of this view to inform arts practice, through the practical introduction of software prototypes, artworks, programming languages and improvised performances. In particular, we introduce works which demonstrate the role of perception in symbolic semantics, embed the representation of time in programming language, include visuospatial arrangement in syntax, and embed the activity of programming in the improvisation and experience of art.

Feedback is very welcome!

BibTeX record:

    title = {{Artist-Programmers} and Programming Languages for the Arts},
    author = {McLean, Alex},
    month = {October},
    year = {2011},
    school = {Department of Computing, Goldsmiths, University of London}

RIS record:

ID  - McLean2011
TI  - Artist-Programmers and Programming Languages for the Arts
PB  - Department of Computing, Goldsmiths, University of London
AU  - McLean, Alex
PY  - 2011/10/01

Pitter patter

Experimenting with webcam overlay. Video recorded using gstreamer, source for screencaster here (screensave.c).

UPDATE, here’s another from a different angle to appease douglas.

Workshop output

The Text live coding workshop went really well, surprisingly well considering it was the first time anyone apart from me had used it and (so I found out after) most of the participants didn’t have any programming experience. The six participants took to the various combinators surprisingly quickly, the main stumbling block being getting the functions to connect in the right way… Some UI work to do there, and I got some valuable feedback on it.

Once the participants had got the hang of things on headphones, we all switched to speakers and the seven of us played acid techno for an hour or so together, in perfect time sync thanks to netclock. Here’s a mobile phone snippet:

The sound quality doesn’t capture it there, but for me things got really interesting musically, and it was fun walking around the room panning between the seven players…

Text update and source

I’ve updated Text a bit to improve the visual representation of higher order types (you’d probably need to full screen to view):

I won’t be touching this until after the workshop on Saturday.

I’ve also made the source for the visual interface available here under the GPLv3 free license. To get it actually working as above you’d also need to install my tidal library, Jamie Forth’s network sync, my sampler, the nekobee synth, and somehow get it all working together. In short, it’s a bit tricky, I’ll be working on packaging soonish though.

Test run of Text

I’ve been rather busy writing lately, my PhD funding runs out in April, and I hope by then I’ll have finished and will be looking for things to do next.

I have had a bit of time to make Text, a visual language I mentioned earlier, a bit more stable, here’s a test run:

A bit of a struggle, partly due to the small screen area I gave myself for the grab, but also due to some UI design issues I need to sort out before my workshop at Access Space in Sheffield next week, on the 5th February. Access Space is a really nice free media lab, but will turn nasty unless I free the workshop software, so expect a release soon.

In case someone is interested, here’s the linux commandline I use to record a screencast with audio from jackd:

gst-launch-0.10 avimux name=mux \
! filesink location=cast.avi \
ximagesrc name=videosource use-damage=false endx=640 endy=480 \
! video/x-raw-rgb,framerate=10/1 \
! videorate \
! ffmpegcolorspace \
! videoscale method=1 \
! video/x-raw-yuv,width=640,height=480,framerate=10/1 \
! queue \
! mux. \
jackaudiosrc connect=0 name=audiosource \
! audio/x-raw-float,rate=44100,channels=2,depth=16 \
! audioconvert \
! queue \
! mux.


Text is a experimental visual language under development.  Code and docs will appear here at some point, but all I have for now is this video of a proof of concept.

It’s basically Haskell but with syntax based on proximity in 2D space, rather than adjacency.  Type compatible things connect automatically, made possible though Haskell’s strong types and currying.  I implemented the interface in C, using clutter, and ended up implementing a lot of Haskell’s type system.  Whenever something changes it compiles the graph into Haskell code, which gets piped to ghci.  The different colours are the different types.  Stripes are curried function parameters.  Lots more to do, but I think this could be a really useful system for live performance.