netzstaub

beatz & funkz

Friday, August 15, 2008

Writing Ableton Control Surface Scripts

I’ve been dabbling with ableton control surface scripts about one year ago (some people may have seen the FCB1010 control surface script: FCB1010 Mappings, which I don’t really have the time to support at the moment). Btw I have a new version of the mappings, but they’re not really documented, but if you’re a bit into programming you may be able to get the mappings from the consts.py file: new undocumented mappings. In the following, I assume you know how to use remote control surface mappings inside ableton (the whole Midi Devices Input Output thingie).

But let’s get down to the business of writing ableton remote control surface scripts. The scripts are written in Python and are stored inside the Ableton Live application in the Resources folder under Windows, or inside the application bundle under MacOSX. The directory containing the scripts is called “MIDI Remote Scripts”. As you can see in this folder, there is an additional folder for each supported device. These folders containing the actual scripts. I’m writing the scripts for my midi controller called MidiKontrol, so I created a directory called “MidiKontrol” inside this folder. The MidiKontrol folder will contain my script. You can download my files at midi-kontrol.zip.

The main problem with developing Ableton scripts is that you have no direct interface to debugging the scripts. Poking around the ableton application, you can find a hidden menu containing options to open a python console and reload scripts, but I haven’t tried to hook into these functions (I suppose that’s what people at Ableton use). Instead, I wrote my own kind of “debugging” functions. This is not perfect though. If your python code contains semantical errors or import errors, Ableton will not load your script and you are left on your own. One way to check for typos is to compile the python files before copying them into the Ableton folder. You have to use python2.2 for this, which I sadly didn’t get to compile on Macosx Leopard. I used this script when I was on Macosx 10.4. py-compileall.py, which compiles all the files in the directories given in the command line.

#!/usr/bin/env python22

import os, sys
import py_compile
import compileall

def main():
    if len(sys.argv) < 2:
	print "usage: compileall tree ..."
	return 0
    else:
	for dir in sys.argv[1:]:
	    print "compiling %s" % dir
	    compileall.compile_dir(dir)
	return 1

if __name__ == '__main__':
    exit_status = not main()
    sys.exit(exit_status)

If you just copy the python files into the folder, Ableton will compile them for you.

The structure of a script is as follows: Ableton looks for a file called __init__.py and loads it. This file returns an object that will be used as the script "controller". My __init__.py file for the MidiKontrol control surface looks like this:

import Live
from MidiKontrol import MidiKontrol

def create_instance(c_instance):
    return MidiKontrol(c_instance)

As you can see, I import the Live module (which doesn't exist in Ableton Live 5, so I wrote my own version for Live 5) which is provide by Ableton Live, and also import the main object of the script, which is MidiKontrol. All the real work is done inside this module. If you want to get used to scripts, try to write a simple __init__.py that will just create a file in /tmp and write something into it. This way, you can see if your script setup works fine. Another problem is reloading script when you have changed it. I just quit Ableton and restart it, which is a bit annoying, but once Ableton is cached it loads quickly.

Let's have a look at the MidiKontrol.py file:

import Live
from MidiKontrolScript import MidiKontrolScript
from MidiKontrolMixerController import MidiKontrolMixerController
from MidiKontrolDeviceController import MidiKontrolDeviceController
from consts import *

class MidiKontrol(MidiKontrolScript):
    __module__ = __name__
    __doc__ = 'Automap script for MidiKontrol controllers'
    __name__ = "MidiKontrol Remote Script"
    
    def __init__(self, c_instance):
        self.suffix = ""
	MidiKontrol.realinit(self, c_instance)

    def realinit(self, c_instance):
	MidiKontrolScript.realinit(self, c_instance)
	self.mixer_controller = MidiKontrolMixerController(self)
	self.device_controller = MidiKontrolDeviceController(self)
        self.components = [ self.mixer_controller, self.device_controller ]

    def suggest_map_mode(self, cc_no):
        return Live.MidiMap.MapMode.absolute

I have separated the __init__ method in a second part "realinit", which is because I use a weird debugging mechanism to trace calls (I think that's the reason, to be honest I don't remember it quite clearly). The MidiKontrol inherits from a class MidiKontrolScript which is a quite "general" wrapper for script functions. The c_instance parameter is a Boost Python object that is used to communicate with the C++ core of Ableton Live. The script then instantiates two further objects, the Mixer Controller and the Device Controller. Real work gets delegated to these 2 objects. Mixer Controller is responsible for handling mappings relating to tracks, while the Device Controller handles device specific mappings. I usually have a third controller for the transport, but my MidiKontrol doesn't support these functions (yet). As you can see, this class doesn't contain much real code either, all the work is actually handled by MidiKontrolScript, which delegates most function calls to the two controllers. Let's have a look at MidiKontrolScript (just excerpts, as the file is way more complicated):

from Tracing import Traced
...

class MidiKontrolScript(Traced):
...
    __filter_funcs__ = ["update_display", "exec_commands", "log", "song"]

This is actually the biggest piece of voodoo I use to debug my scripts. The Traced class, defined in Tracing, is a metaclass that will log every method call of classes inheriting from it in a special file. This way, I can trace my function calls and see when a call goes wrong (casts an Exception). The __filter_funcs__ field is there to filter out functions that are not really interesting or that are called very often. I took most of the code for the Tracing class from this python essay: Trace.py. It will log all method calls in the file /tmp/fftrace.log under Macosx and C:/tmp/fftrace.log under Windows. If you are interested in the workings of Tracing.py, feel free to read the essay I linked, I don't really remember the semantics. I have to say though that this kind of meta-hackery was quite a pain compared to how easy that stuff is in Lisp (yes, managed to sneer at python and mention Lisp, my good deed of the day is done).

Continuing our investigation of MidiKontrolScript.py, you can see in the realinit method that when __myDebug__ is true, a logfile is created. This is used as an additional way to log data out of Ableton for me to read while my script runs. To write things into the logfile, you can use the methods log and logfmt. You can also see that I reference a file called "MidiKontrol-cmd". Commands written into this file are polled by my script in the exec_commands method. This method reads in the file, executes the statements, and then clears it. These 3 tools (fftrace.log, /tmp/midi-kontrol log, and commands in /tmp/midi-kontrol-cmd) are my main helps when debugging scripts. I found out about most of the API by looking at the other scripts in the Midi Remote Scripts folder, and looking at the sourcecode by using the "decompyle" utility to transform the python bytecode into something readable (mostly).

But let's get down to actual Ableton Live API stuff. All the lower methods in MidiKontrolScript are part of the Ableton Live API. disconnect is called when removing the script or closing ableton. I just call disconnect on all the child components (the Mixer Controller and the Device Controller mentionned above). application returns the Python object representing the whole application, while song returns the Python object representing the current liveset. This object is used a lot, because it contains links to the tracks, devices, clips, scenes, etc... suggest_input_port and suggest_output_port can be used to automagically choose the right MIDI device. can_lock_to_devices tells Ableton if the script can "hold on" to a specific device. This is used by the Device Controller, which will then remember the locked device and not react to selecting new devices until the locked device is unlocked. For more information locking devices, read the Ableton Live Remote Control Surface documentation. The actual locking is done using the lock_to_device, unlock_to_device (sic) and toggle_lock methods. These calls are also just delegated to the child components. suggest_map_mode is used to tell Ableton which kind of CC data is sent by a specific control (relative, absolute, etc...). show_message is a helper to display an informational message in the status bar of ableton. request_rebuild_midi_map is a helper to tell Ableton that the script would like to reinitialize its midi mappings. This is closely related to build_midi_map, which is called by Ableton and allows the script to actually provide midi mappings. The handle passed as an argument is used by the script to register mappings. As you can see, this call is also delegated to the child components. The Mixer Controller will register mappings pertaining to tracks (like volume control, sends, and also channel EQs). update_display is called periodically by ableton to allow the script to do some regular tasks, update status displays etc... It calls the logging function and is then passed on to child components. send_midi is used to send a byte tuple as midi (useful to send sysex, for example). receive_midi is called by ableton when midi data is received on the script interface which is not directly handled by registered mappings. This way, you can receive sysex data, but also ask for ableton to forward note data and ccs from the device without having to register them to a specific parameter. The method does a bit of parsing by recognizing notes and ccs and calling the appropriate methods in the child components.

The actual MIDI mapping work likes this: Ableton calls build_midi_map, which allows the script to register mappings using the passed midi_map_handle. A registered mapping is a link from a midi parameter (Note or CC) to an element in the GUI of ableton. For example mixer parameters (volume, send, cue level, etc...), device parameters, or clip parameters. The script can also use the midi map handle to ask Ableton to forward specific ccs or notes to the script. This allows to do more specific interpretation of some data. For example, the MidiKontrol script asks ableton to forward the relative CCs used to scroll through tracks and scenes, and interprets it in its own custom way.

Another interesting class used is DumpXML, which I use to dump the documentation of ableton as XML, and also ableton tracks. I used this a while ago by dumping my whole liveset structure, and using that to organize my drum machine patterns. I had an audio clip for each track and pattern in my drum machine, featuring a recorded a loop of the drummachine, and used this to reorganize my loops. Then, I would generate a sysex file for the drummachine containing the whole liveset structure organized as rows, and used that in the Song Mode of the drum machine. THis way, I didn't have to spend days in the editing mode of the drum machine to organize my hundreds of loops 🙂 Sadly, I have found no way to access the arrangement view information out of a script.

Finally, the MidiKontrolScript instantiates a helper class, MidiKontrolHelper, which contains a bunch of useful methods for writing Ableton scripts and communicating with the MidiKontrol. They for example contain the sysex packing I documented yesterday, and helper methods to initialize parameters on the MidiKontrol. THere are also methods to select scenes, tracks, find the last EQ on a track channel, etc... All the methods should be pretty self explanatory.

After this tour of all the utility I have at my disposal to write scripts, let's look at the two files doing the actual work: MidiKontrolDeviceController.py and MidiKontrolMixerController.py.
As written above, the Device Controller handles mapping pertaining to devices in Ableton (Compressor, Simpler, etc...). My MidiKontrol has 4 pages with 4 encoders, and a small display able to display 3 characters per encoder (actually 4 characters, but one is used as whitespace to separate the fields). I map the 8 encoders on the last 2 pages to the Best-Of Bank parameters of the currently selected device. Also, when a new device is selected, I flash the name of the device on the MidiKontrol. The key to the magic can be found in the build_midi_map method, which is called by Ableton when a new device is inserted, or a new device is selected. Actually the midi map building is triggered by the lock_to_device and unlock_from_device methods (which are called by the parent object MidiKontrolScript, if you remember from a few paragraphs back). You can also add a listener to when a new device is selected, but Ableton already automatically calls build_midi_map when this happens. build_midi_map forwards the work to the method map_device_params, which checks if a device has been selected. If this is the case, it looks for the best-of parameters in dictionaries (which you can find in the file Devices.py). It then calls either map_params_by_name for Ableton internal devices, or map_params_by_number for VST or AU instruments. map_params_by_number just maps the first 8 parameters of a VST or AU.

	def map_params_by_number(device):
	    ccs = PAGE3_CCS + PAGE4_CCS
	    channel = CHANNEL_MIDI_KONTROL
	    mode = Live.MidiMap.MapMode.absolute
	    for encoder in range(8):
		# +1 to skip "Device on"
		if (len(device.parameters) >= encoder + 1):
		    ParamMap.map_with_feedback(midi_map_handle, channel, \
					       ccs[encoder], \
					       device.parameters[encoder + 1], \
					       mode)

The CC numbers used by MidiKontrol are stored in PAGEx_CCS in consts.py. All I do is iterate over the device parameters, and call the helper method "map_with_feedback" defined in ParamMap.py, which defines a standard feedback rule and stores the mapping in the midi map handle. And that's basically it, all the further work will be done by Ableton itself. Sending a CC will update the parameter, and changing the parameter with the mouse will send out feedback CCs to the midi kontrol. map_params_by_name does basically the same, but looks up parameters using the names stored in the best-of bank tuple defined in Devices.py.

At the end of the build_midi_map method, I send the SYSEX to update the page on the MidiKontrol (mostly to display the update parameters). To do this I use the methods I defined in MidiKontrolHelper.py . Finally I send a flash message if a new device has been selected. Et voila, an automagic device mapping script for Ableton, that will display the name of the mapped parameters on the LCD of my MidiKontrol.

The Mixer Controller also handles the navigation inside Ableton. I use two relative CCs to scroll scenes and scroll tracks. I can't map these directly through the script interface, so I have to interpret the MIDI data myself. I ask Ableton to forward the CCs to my script:

	def forward_cc(chan, cc):
	    Live.MidiMap.forward_midi_cc(script_handle, midi_map_handle, chan, cc)
	idx = 0

        forward_cc(CHANNEL_MIDI_KONTROL, SCENE_SCROLL_CC)
        forward_cc(CHANNEL_MIDI_KONTROL, TRACK_SCROLL_CC)

and handle received CCs in the method receive_midi_cc (which is called by the parent MidiKontrolScript, look a few paragraphs back):

    def receive_midi_cc(self, channel, cc_no, cc_value):
        def rel_to_val(rel):
	    val = 0
	    if (cc_value >= 64):
		val = cc_value - 128
	    else:
		val = cc_value
            return val
            
        if (channel != CHANNEL_MIDI_KONTROL):
            return
        
        if (cc_no == SCENE_SCROLL_CC):
	    val = rel_to_val(cc_value)
	    idx = self.helper.selected_scene_idx() + val
	    new_scene_idx = min(len(self.parent.song().scenes) - 1, max(0, idx))
	    self.parent.song().view.selected_scene = self.parent.song().scenes[new_scene_idx]
        elif (cc_no == TRACK_SCROLL_CC):
            val = rel_to_val(cc_value)
            current_idx = self.helper.selected_track_idx()
            idx = current_idx + val
            tracks = self.parent.song().tracks + self.parent.song().return_tracks + (self.parent.song().master_track,)
            new_track_idx = min(len(tracks) - 1, max(0, idx))
            track = tracks[new_track_idx]
            if (current_idx != idx):
                self.parent.song().view.selected_track = track

This method is a bit more complicated. First it checks if the CC received was on the correct channel, then converts the absolute CC value to a relative value (using the rel_to_val helper). It then selects the new scenes out of the self.parent.song().scenes tuple. Assigning the variable self.parent.song().view.selected_scene will change the selected scene in Ableton. The tracks updating is similar, except that we have to append normal tracks, return tracks and the master_track ourselves.

The mapping magic of the mixer controller is very similar to the mapping magic of the device controller. The first page of the MidiKontrol contorls the volume of the first 4 tracks in Ableton, while the second page controls the volume, the first send and low and high-pass levels if an EQ is present on the channel. I go through the tracks, map to the volume (for the first page), and then send the update description of the first page. Then I go through the selected track, map the send if its present, and use helper methods to get hold off the last EQ present on the track (either a Filter8 or an EQ3), and map the high and low parameters to the last 2 encoders of the second page. Then I send the updated names to the MidiKontrol, et voila!

Writing ableton control scripts isn't very difficult in theory, but as there is no documentation or development tools whatsoever, it can be a bit of a challenge sometimes. I hope this information will help you to create awesome mappings for all of us to enjoy 🙂

posted by manuel at 7:38 pm  

6 Comments »

  1. […] Original post by manuel […]

    Pingback by Writing Ableton Control Surface Scripts : blog lowerautoinsurance — August 15, 2008 @ 7:58 pm

  2. Hi Manuel,

    To make Ableton recompile the updated python code, just select another control surface, and then get back to yours in the control surface selector.

    I keep the preferences dialog box open all the time to cmd-tab to it and press up, then down arrows. It reloads the code immediately.

    Also, I wanted to point that it’s a pity that your techno experiments (mp3’s) are no longer available. I would very much like to check them. I’m a techno geek myself — playing, composing and building hardware with my dad and always interested in what others do.

    Our latest development is the monome-like device. The schematics is the same, but the buttons / leds, casing, rubber etc all different. I’m still programming it (updating firmware to support native blinking) and writing the Ableton control surface driver for my needs. Just to let you know… 🙂

    Good luck!

    Comment by Aleksey — August 18, 2008 @ 7:05 pm

  3. Hi Manuel,

    Where do I download the Live API from? I thought I’d downloaded it from http://www.liveapi.org/, however this appears to be code which uses the Live API rather than the API itself. Within the code that I downloaded there is no reference to the MIDI mapping functionality (such as build_midi_map) which is exactly what I’m after for an application that I’m writing.

    Also near the start of your article you mention “a hidden menu containing options to open a python console and reload scripts”. How do I access this menu?

    Is there documentation for the Live API?

    Cheers, JOe

    Comment by Joe — November 2, 2008 @ 1:24 am

  4. The Live API seemingly lives here now:

    http://www.assembla.com/wiki/show/live-api

    I got the code from that site to somewhat work with Live 7.0.12, after having changed the library paths in the __init__.py script according to what my Windows box wants to see.

    Comment by Hans — December 15, 2008 @ 1:44 pm

  5. Great!
    and WHERE is the freaking “MIDI Remote Scripts” folder!?

    There’s a user pref folder in Preferences with mapping examples that have nothing to do with python. What are you writing about here?

    Why is all this crap so freaking hard???

    Comment by AndyC — January 21, 2009 @ 6:17 am

  6. […] saved by ddavey 21 days ago1 votesWe need Staff Members!!>> saved by raghaille 34 days ago5 votesWriting Ableton Control Surface Scripts>> saved by VideosenFamilia 40 days ago1 votesPresenters Needed for Video Format/Encoders and […]

    Pingback by User links about "encoders" on iLinkShare — February 11, 2009 @ 8:33 am

RSS feed for comments on this post. TrackBack URI

Leave a comment

Powered by WordPress