netzstaub

beatz & funkz

Tuesday, September 6, 2005

Algorithmic movie editing in Lisp

Algorithmic art

Alt Tag Added during WordPress Conversion

While composing music for a movie by a friend, I took a closer look at
how he was editing the raw movie material. He was working with what
I’ll be calling “effects”, for example movie fluttering, where a small
segment of the movie gets cut in short frame sequences and reorganized
in a seemingly random manner. These segments would then be synced onto
the underlying soundtrack (for example music, or words). Reorganizing
the frames by hand, even in a quite advanced software like Adobe
Premiere, was quite painstaking, and it took a few hours to arrange
something like 30 seconds of movie material.

I was feeling quite hacky this day, and writing a software that can
rearrange movie frames in a random manner can’t be quite complicated,
can it? I first dwelved into DirectShow, the Microsoft API used to
handle movies and similar things, but it turned out to be quite
overengineered and a bit to complicated for what I was trying to do
(no realtime manipulation, for example). After searching a bit on
freshmeat, I stumbled upon avisplice, a tool in C that can glue
different avi movies together. Upon inspection of the code, I noticed
that all avisplice was doing was taking the avi header out of the
first movie, and then sticking the frames together, assuming that the
compression format of the different avi movies was the same. I
modified the C program to do the permutation effect my friend was
working with, and gave him the executable version to work with. This
way, he could try out different parameters (the number of frames to be
permutated as a block). Instead of working for hours with Premiere,
and then concluding that the permutation was not as good as he wanted
it to be, he could generate 20 permutations, try them out, and keep
the best or try a different thing altogether.

I was playing around with Common Music at this time, and found the
pattern composition approach quite interesting. What I would do was to
formulate some kind of ideas in my mind as to how a specific kind of
music was working, and then write down the code in Lisp, and play
around with certain parameters, and maybe insert some kind of
randomness. For example, in a piece of dub music I was writing, I had
a pretty terse ground beat, and added some snare hits with big reverb
on offbeats. To me, this is the musical idea, which sometimes sounds
good, sometimes not so good, but using Common Music, I could generate
a great number of loops, and then pick the grooviest ones and
incorporate them in my music. In a certain way, this was the same
approach that my friend was having with his video editing. His idea
was “permutate the frames”, and then he would pick the good
permutations. Of course, this is only “technical” stuff. I think it is
not enough to produce “interesting” pieces of art, but in a way, it
enables the artist to go much further in what he is trying to do.

Actually this is a quite interesting debate. Is doing the editing by
hand the better way to produce art? In a way, it is, because while
doing the repetitive work, you get a much more organical feel of what
is going on, what may be working, what may be not. Doing errors while
working can produce interesting insights. I also think that in the
end, you are prouder of your work because a lot of efforts went into
it. On the other hand, once you “mastered” the technique, it becomes
uninteresting, and gets in the way when formulating your ideas, it
stops the artistic flow. I think this is especially true of digital
techniques, because working in front of a screen is frustrating, while
working with your hands is quite “zen”. I don’t mind sitting in front
of a stencil for 8 hours cutting paper, but I get very irritated when
I’m rearranging HTML documents in my editor for example. The great
thing about programming is that as soon as you notice that something
is boring, you can try to write down the process behind it, and let
the computer do the work. Then you can work on the metalevel. This
also compels you to analyse what you are doing in a very precise and
methodic way, because you have to write a program that does the work
for you.

Code, show us some code!

Well, enough blabla about artsy and brainy stuff. I quickly stumbled
upon the limits of the C software when trying to write down new
algorithms for my friend (for example, reversing small fragments of
the movie, and glueing them back together), so I took a few hours to
rewrite the software in Lisp. In order to get things done, I set
myself a goal and a deadline, which was to produce a music video for
an art event on the following day. The code I’ll show now is really ugly,
but remember it was hacked together in a few hours in a highly
coffeinated state ๐Ÿ™‚

Alt Tag Added during WordPress Conversion

The AVI file format

The first step in writing the software was to be able to read and
write AVI files. AVI is actually a RIFF container format, and is quite
similar to for example WAVE. It starts with a RIFF header, and then
contains a succession of RIFF blocks, which consist of a RIFF code
(four characters), a length, and the content. A RIFF block can be a
RIFF list, containing a variable number of RIFF blocks. An AVI file
consists of an avi header, one or several streams (for example, a
video stream and an audio stream), and an optional index. The streams
are described by stream headers, stored in a RIFF list at the
beginning of the file. The first version of the software could only
work with video streams, and all the video clips I manipulated
contained only uncompressed DV-video streams. I’m doing a rewrite
right now, but it is far from finished. Here is the class definition
for an AVI movie. AVIH is the avi header, STRH is the stream header
for the video stream, and STRF is the stream format for the video
stream. FRAMES is a list of frames. Like avisplice, a frame is only a
filename, an offset, and a length. The frames themselves are not read
into memory, but are only copied from file to file when rendering a
movie.

(defclass avi ()
  ((type :initform nil
	 :accessor avi-type
	 :initarg :type)

   (avih :initform nil :accessor avi-avih
	 :initarg :avih)
   (strh :initform nil :accessor avi-strh
	 :initarg :strh)
   (strf :initform nil :accessor avi-strf
	 :initarg :strf)


   (size :initform 0
	 :accessor avi-size
	 :initarg :size)
   
   (frames :initform nil
	   :initarg :frames
	   :accessor avi-frames)
   
   (pathname :initform nil
	     :accessor avi-pathname
	     :initarg :pathname)))

(defmethod print-object ((avi avi) s)
  (print-unreadable-object (avi s :type t :identity (unless (avi-pathname avi) t))
    (format s "\"~A\" (~A frames)" (or (avi-pathname avi)
				       "virtual avi file")
	    (avi-num-frames avi))))

Reading the AVI file is quite straight forward. The file is opened,
the RIFF header is read, and then a list of RIFF blocks is
read. These blocks are parsed according to the RIFF block type. The
basic binary types present in the RIFF file are FOURCC codes (four
characters), and little endian short and long values. These are
extracted from arrays of type (UNSIGNED-BYTE 8) using the following
functions:

(defun make-fourcc (seq)
  (coerce (mapcar #'code-char (coerce seq 'list)) 'string))

(defun make-long (seq)
  (loop for i from 0 below 4
	sum (* (elt seq i) (expt 256 i))))

(defun make-short (seq)
  (loop for i from 0 below 2
	sum (* (elt seq i) (expt 256 i))))

The functions to parse the avi file. Notice that READ-CHUNK recognizes
the different block types. “00dc” is the code for a compressed AVI
frame. In this case, READ-CHUNK adds the frame (the filename, the
offset and the size) to the frame list of the avi file.

(defun read-avi-file (filename)
  (with-open-file (s filename :element-type '(unsigned-byte 8)
		     :direction :input)
    (let ((avi (make-instance 'avi
			      :pathname filename)))
      (read-riff-hdr avi s)
      (read-list avi s (avi-size avi))
      avi)))

(defun read-riff-hdr (avi s)
  (let ((hdr-arr (make-array 12 :initial-element 0 :element-type '(unsigned-byte 8))))
    (read-sequence hdr-arr s)
    (unless (string-equal (make-fourcc (subseq hdr-arr 0 4)) "RIFF")
      (error "~A is not an AVI file" (avi-pathname avi)))
    (setf (avi-type avi) (make-long (subseq hdr-arr 8 12))
	  (avi-size avi) (make-long (subseq hdr-arr 4 8)))))

(defun read-list (avi s list-size)
  (let ((cur-size 0))
    (loop while ( size max-size))
	(error "Chunk size is incorrect"))
      (cond ((string-equal type "LIST")
	     (let ((list-arr (make-array 4 :initial-element 0 :element-type '(unsigned-byte 8))))
	       (read-sequence list-arr s)
	       (let ((list-type (make-fourcc list-arr)))
		 (cond ((string-equal list-type "movi")
			(read-movi avi s (- size 4)))
		       (t (read-list avi s (- size 4))))
		 (incf size 4))))
	    ((string-equal type "strh")
	     (read-strh avi s))
	    ((string-equal type "avih")
	     (read-avih avi s))
	    ((string-equal type "strf")
	     (read-strf avi s))
	    ((string-equal type "00dc")
	     (let ((avi-offset (file-position s)))
	       (push (list (avi-pathname avi) avi-offset size) (avi-frames avi))
	       (file-position s (+ avi-offset size))))
	    (t ; (warn "Unknown chunk ~A at offset ~A" type (file-position s))
	       (file-position s (+ (file-position s) size))))
      (+ size 8))))

(defun read-movi (avi s size)
  (read-list avi s size)
  (setf (avi-frames avi)
	(nreverse (avi-frames avi))))

(defun read-strf (avi s)
  (let ((strf (make-array 40 :initial-element 0 :element-type '(unsigned-byte 8))))
    (read-sequence strf s)
    (setf (avi-strf avi) strf)
    40))

(defun read-avih (avi s)
  (let ((avih (make-array 56 :initial-element 0 :element-type '(unsigned-byte 8))))
    (read-sequence avih s)
    (setf (avi-avih avi) avih)
    56))

(defun read-strh (avi s)
  (let ((strh (make-array 56 :initial-element 0 :element-type '(unsigned-byte 8))))
    (read-sequence strh s)
    (setf (avi-strh avi) strh)
    56))

AVI files are written in quite the same way. A few functions convert
the basic binary types:

(defun write-fourcc (fourcc s)
  (write-sequence (mapcar #'char-code (coerce fourcc 'list)) s))


(defun long->seq (long)
  (loop for i from 0 below 4
	collect (mod (floor (/ long (expt 256 i))) 256)))

(defun write-long (long s)
  (write-sequence (long->seq long) s))

(defun write-short (short s)
  (write-sequence (loop for i from 0 below 2
			collect (mod (floor (/ short (expt 256 i))) 256)) s))

Writing a RIFF block is quite straight forward too. While writing a
RIFF list, we need to recognized embedded lists, and AVI frames have
to be written in a special way too. This is quite hacked together, but
it works ๐Ÿ™‚ When writing AVI frames, I open the original file (which
is stored in the frame list), seek to the offset of the frame, read
the binary data into a temporary buffer of 64k bytes, and write it
into the output file. This is a bit of overhead, as writing a sequence
of frames involves reopening the same file over and over again, but
hopefully the operating system will cache everything ๐Ÿ™‚

(defun write-chunk (s type data)
  (write-fourcc type s)
  (write-long (length data) s)
  (write-sequence data s)
  (+ 8 (length data)))

(defun write-frame (s filename offset length)
  (with-open-file (in filename :direction :input
		      :element-type '(unsigned-byte 8))
    (write-fourcc "00dc" s)
    (write-long length s)
    (file-position in offset)
    (let* ((buffer-size 65536)
	   (c (make-array buffer-size :element-type '(unsigned-byte 8))))
      (loop for i from 0 below length by buffer-size
	    for l = (min buffer-size (- length i))
	    do (read-sequence c in :end l)
	    (write-sequence c s :end l)))
    (+ length 8)))

When writing a list, we don’t know how much bytes the list will take
up in the AVI file, so we remember the start offset of the list,
subtract this from the end offset, rewind to the list header, and
write the value there (4 additional bytes for the list type).

(defun write-list (s type chunks)
  (write-fourcc "LIST" s)
  (let ((pos (file-position s)))
    (write-long 0 s)
    (write-fourcc type s)
    (let ((length (loop for chunk in chunks
			sum (cond ((string-equal (first chunk) "LIST")
				   (write-list s (second chunk) (third chunk)))
				  ((string-equal (first chunk) "00dc")
				   (write-frame s (second chunk) (third chunk)
						(fourth chunk)))
				  (t (write-chunk s (first chunk) (second chunk)))))))
      (let ((end-pos (file-position s)))
	(file-position s pos)
;	(warn "list length is ~A" (+ 4 length))
	(write-long (+ 4 length) s)
	(file-position s end-pos))
      (+ 12 length))))

Writing the AVI file is a bit more involved. We first have to write
the RIFF header, then the avi header and the stream headers, then the
frames, and finally the index, because Premiere won’t import AVI files
without a working index. The index is a special RIFF block containing
the offsets of the frames in the “movi” list relative to the beginning
of the list. The flags are fixed, because we don’t have keyframes in
uncompressed DV streams. We also remember the start position to write
the correct file size in the RIFF header.

(defmethod write-avi-file ((avi avi) filename)
  (let ((len 12))
    (with-open-file (s filename :element-type '(unsigned-byte 8)
		       :if-does-not-exist :create
		       :if-exists :supersede
		       :direction :output)
      (write-fourcc "RIFF" s)
      (let ((pos (file-position s))
	    frames-pos)
	(write-long 0 s)
	(write-long (avi-type avi) s)
	(loop for i in (long->seq (avi-num-frames avi))
	      for j from 0
	      do (setf (aref (avi-avih avi) (+ 16 j)) i
		       (aref (avi-strh avi) (+ 32 j)) i))
	(incf len
	      (write-list s "hdrl" `(("avih" ,(avi-avih avi))
				     ("LIST" "strl"
				      (("strh" ,(avi-strh avi))
				       ("strf" ,(avi-strf avi)))))))
	(incf len
	      (write-list s "movi" (let ((cur-pos 4))
				     (mapcar #'(lambda (data)
						 (let ((len (third data)))
						   (push (list cur-pos len) frames-pos)
						   (incf cur-pos (+ 8 len)))
						 `("00dc" ,@data))
					     (avi-frames avi)))))
	(setf frames-pos (nreverse frames-pos))

	(incf len
	      (write-chunk s "idx1"
			   (coerce (mapcan #'(lambda (pos-len)
					       (append '(#x30 #x30 #x64 #x63) ; 00dc RIFF header
						       '(#x10 #x00 #x00 #x00) ; FLAGS
						       (long->seq (first pos-len))
						       (long->seq (second pos-len))))
					   frames-pos) 'array)))
						     
	
	(file-position s pos)
	(write-long len s))
      )))

Manipulating movies

Alt Tag Added during WordPress Conversion

Now that the basic functionality has been implemented, we can start to
work on movies, and reorganize frames. This is done by working on the
frame list stored in AVI objects. In order to work nondestructively on
avi movies, we define a function COPY-AVI which copies the avi header
and the stream headers out of an AVI object, but allows the programmer
to supply a new frame list. This function will be the basic building
block for all the editing algorithms we will implement later on.

(defmethod copy-avi ((avi avi) &key (frames :noframes) strh type strf avih (class 'avi))
  (make-instance class
	 :type (or type (avi-type avi))
	 :strh (or strh (avi-strh avi))
	 :strf (or strf (avi-strf avi))
	 :avih (or avih (avi-avih avi))
	 :frames (if (eq frames :noframes)
		     (avi-frames avi)
		     frames)))

(defmethod copy-frames ((avi avi))
  (copy-list (avi-frames avi)))

We are now ready for the first algorithm, which is just pasting avi
movies together (the original functionality of avisplice).

(defmethod concatenate-avis (avis)
  (copy-avi (first avis) :frames (mapcan #'copy-frames avis)))

Now that was quite elegant, now we want to extract segments out of
avis, maybe using a random start and a random length.

(defmethod subsegment ((avi avi) &key start length random)
  (let ((len (length (avi-frames avi))))
    (if random
	(progn
	  (unless start
	    (setf start (random len)))
	  (unless length
	    (setf length (random (- len  start)))))
	(progn (unless start (setf start 0))))
    (copy-avi avi :frames (subseq (avi-frames avi) start (when length (+ start length))))))

Let’s split avi files into groups of N frames.

(defun group-by (list n)
  (do ((i 0 (1+ i))
       (l list (cdr l))
       group res)
      ((null l)
       (when group (push (nreverse group) res))
       (nreverse res))
    (push (first l) group)
    (when (= (length group) n)
      (push (nreverse group) res)
      (setf group nil))))

(defmethod split-avi ((avi avi) n)
  (let ((groups (group-by (avi-frames avi) n)))
    (mapcar #'(lambda (frames)
		(copy-avi avi :frames frames)) groups)))

We can also write a few manipulation functions to permutate avis,
reverse them. Notice that there is a version of the function producing
a new AVI object, and another version producing a list of frames, for
further manipulation by list functions.

(defun permutate-list (list)
  (let* ((arr (coerce list 'array))
	 (length (length arr)))
    (dotimes (i length)
      (let* ((i1 (random length))
	     (i2 (random length))
	     (tmp (aref arr i1)))
	(setf (aref arr i1) (aref arr i2)
	      (aref arr i2) tmp)))
    (coerce arr 'list)))

(defmethod reverse-avi ((avi avi))
  (copy-avi avi :frames (reverse (avi-frames avi))))

(defmethod permutate-avi ((avi avi) n)
  (concatenate-avis (permutate-list (split-avi avi n))))

(defmethod permutate-avi-to-segments ((avi avi) n)
  (permutate-list (split-avi avi n)))

(defmethod permutate-avis (avis n)
  (concatenate-avis (permutate-list (mapcan #'(lambda (avi) (split-avi avi n)) avis))))

(defmethod permutate-avis-to-segments (avis n)
  (permutate-list (mapcan #'(lambda (avi) (split-avi avi n)) avis)))

(defmethod crop-avi ((avi avi) &key (start 0) length)
  (subsegment avi :start start :length length))

Other interesting editing algorithms are mixing different clips
together, taking N frames of the first, then N frames of the next,
producing some kind of flickering effect (FLIMMERN is german for
FLICKERING).

(defun flatten1 (list)
  (let (res)
    (dolist (l list)
      (if (listp l)
	  (dolist (l2 l)
	    (push l2 res))
	  (push l res)))
    (nreverse res)))

(defun repeat (list n)
  (loop repeat n
	appending list))

(defun flimmern (groups n)
  (loop for group1-2 on groups by #'cddr
	for group1 = (car group1-2)
	for group2 = (cadr group1-2)
	appending (if (and group1 group2)
		      (list group1
			    (repeat (list (first group2)
					  (car (last group1))) n)
			    group2)
		      (list group1))))

(defmethod flimmer-avis (avis n)
  (copy-avi (first avis) :frames (flatten1  (flimmern (mapcar #'avi-frames avis) n))))

Actually, there are no boundaries to what effects you can achieve, as
long as you can write them down as a Lisp function ๐Ÿ™‚ You can get the
complete code to the movie editing software here, but remember it
is a one-shot, ugly version that is being rewritten. The code can only
work on the ordering of frames at the moment, but a very interesting
step would be to manipulate the frames directly. I have some code to
load Quart Composer compositions and control them programmatically in
objective C, and it would be quite interesting to build a Lisp bridge
to it. This way, a Lisp program could control the outlets of a
composition, feeding it with images, numerical values, music, and
produce very nice video effects without having to control everything
per hand. I’ll have a look at it as soon as my powerbook is working
again.

A music video written in Lisp

Alt Tag Added during WordPress Conversion

I mentioned above that my goal was to produce a music video using the
editing software. I fell in love with “Ongyilkos Vasarnap” by Venetian
Snares, a dark remix of “Gloomy Sunday”, with a hypnotic 7/4
drum’n’bass beat. I first imported the audio track into live, warped
it and exported it to have a stable 144 bpm timing on the whole
track. I then took apart the track on paper, writing down specific
scenes, and also a few drum breaks. Sadly, I didn’t have enough time
to do this for the whole track, so most effects are not precisely beat
matched (I’ll render a new version when I’ll have enough time). I had
some very nice video clips of a friend hanging herself in a dark
corridor. Actually all the video clips come from an art installation
by the friend for whom I wrote the software. The art installation,
called the Angstapparat (the Fear Machine) consisted of a long dark
corridor built in the Pirateria in
Cologne. Inside the corridor were a few TV-screens showing clips
recorded in the installation, and a camera recorded the visitors, also
projecting the images on the outside of the installation. These images
were later on manipulated by the artist, and reintroduced into the
installation.

So I wrote down the story I imagined to the song: a girl enters the
machine, there are flashbacks of happier times when she was laughing,
but then she hangs herself, influenced by demons and ghosts, and
finally she herself becomes a ghost and leaves the machine. Actually,
I’m not happy at all with the video, because it is overloaded with
video effects, there is no real basic rhythm, which shows that I have
no clue about movie editing ๐Ÿ™‚ . As mentioned above, I think that’s
exactly the danger of algorithmic art, doing something without
understanding its implications, something which I think comes only
with a lot of work and effort.

Anyway, I imagined the movie to consist of 9 “scenes”. The first two
scenes show the girl entering the machine, interrupted by flashes of
an eye and flashes of herself matching the music. The basic movie of
the girl entering the machine is itself slowed down and sped up
according to the rhythm of the song. The speed manipulation is done
using the function BESCHLEUNIG-AVI (SPEED-AVI in german), which takes
a movie and a list of frame offsets. An offset of 1 is the “standard”
speed, and offset of 2 means skip the next frame, actually speeding up
the movie by a factor of 2. A factor of 1/3 means repeat each frame 3
times, actually slowing down the movie by a factor of 3.

(defmethod beschleunig-avi ((avi avi) frame-speeds)
  (copy-avi avi :frames
	    (loop with cur = 0
		  for speed in frame-speeds
		  collect (elt (avi-frames avi) (round cur))
		  do (incf cur speed))))

The speeds themselves are generated using the functions
MAKE-CONSTANT-SPEED and ACCELERATE2:

(defun accelerate2 (from acc2 steps)
  (loop for acc = 0 then (+ acc acc2)
	for speed = from then (+ speed acc)
	repeat steps
	collect speed))

(defun make-constant-speed (speed length)
  (make-list (round length) :initial-element speed))

Generating accelerations matching the music was done “by hand”. A list
of 4 different acceleration patterns was used. The different pattern
were then chosen randomly.

(defparameter *erster-abschnitt-speeds*
  (list
   (append (make-constant-speed 0.3 10)
	   (accelerate2 0.3 0.04 11)
	   (make-constant-speed 0.3 10)
	   (accelerate2 0.3 0.04 11)
	   (make-constant-speed 0.3 10)
	   (accelerate2 0.3 0.04 10)
	   (accelerate2 0.3 0.04 11))
   (append (make-constant-speed 0.5 21)
	   (accelerate2 0.3 0.01 21)
	   (make-constant-speed 0.5 10)
	   (accelerate2 0.5 0.03 10)
	   (accelerate2 0.5 0.03 11))
   (append (make-constant-speed 0.5 21)
	   (make-constant-speed 0.3 10)
	   (accelerate2 0.3 0.04 11)
	   (make-constant-speed 0.3 10)
	   (accelerate2 0.5 0.03 10)
	   (accelerate2 0.5 0.03 11))
   (append (make-constant-speed 0.3 10)
	   (accelerate2 0.3 0.04 11)
	   (accelerate2 0.3 0.01 21)
	   (make-constant-speed 0.3 10)
	   (accelerate2 0.3 0.04 10)
	   (accelerate2 0.3 0.04 11))))

(defun random-erster-abschnitt-speed ()
  (random-elt *erster-abschnitt-speeds*))

The flashes for the first segment are chosen randomly. 20% of the time
there will be the flash of an eye, the rest of the time the flash will
be chosen from the clip of the girl entering the machine.

(defun reinrenn-flash (n length)
  (subsegment (permutate-avi *reinrenn* n) :length length))

(defun augen-flash (n length)
  (subsegment (permutate-avi *auge* n) :length length))

(defmacro random-choice (&rest choices)
  (let ((rand (gensym)))
  `(let ((,rand (random 1.0)))
    (cond ,@(loop for choice in choices
		  for p = (first choice) then (if (eql (first choice) t)
						  1.0
						  (+ p (first choice)))
		  when (> p 1.0)
		  do (error "p can not be > 1.0")
		  collect `((

These flashes are inserted on the beats of the music, with a varying length and probability. The base pattern of the music is stored into a function called ERSTER-ABSCHNITT-SIMPLE-MUSTER (FIRST-SEGMENT-BASIC-PATTERN in german). On beat 1, 3, 5 and 7, with a default probability of 60%, a flash will be inserted.

(defun erster-abschnitt-simple-muster (&optional (start 0) (func #'erster-abschnitt-flash)
				       &key (p 0.6))
  (remove nil
	  `(,(when (

The first scene is then rendered using the function ERSTER-ABSCHNITT, which takes the sped up base movie, a list of patterns and flashes, inserting the flashes on top of the base movie.

(defun reinrenn-grundfilm ()
  (let* ((reinrenn (repeat-loop-avi *reinrenn* 1))
	 (film
	  (schedule-avis (loop for i from 0 below 16
			       with start = 0
			       for speed = (random-erster-abschnitt-speed)
			       for segment = (beschleunig-avi (subsegment reinrenn :start start) speed)
			       collect (list segment (beat-length (1+ (* 7.0 i))))
			       do (incf start (count-speed-frames speed))))))
    (setf *erster-abschnitt-grundfilm* (subsegment film :length (round (* 8 *takt-frames*)))
	  *zweiter-abschnitt-grundfilm* (subsegment film :start (round (* 8 *takt-frames*))
						    :length (round (* 8 *takt-frames*))))))

(defun erster-abschnitt ()
  (insert-avis *erster-abschnitt-grundfilm*
	       (beat-offsets->avi-offsets
		(append `((,(erster-abschnitt-flash 2) 3)
			  (,(erster-abschnitt-flash 2) 5)
			  (,(erster-abschnitt-flash 4) 7))
			(erster-abschnitt-simple-muster 7)
			(erster-abschnitt-simple-muster 14)
			`((,(erster-abschnitt-flash 2) 22)
			  (,(erster-abschnitt-flash 2) 24)
			  (,(erster-abschnitt-flash 2) 28)
			  (,(erster-abschnitt-flash 10) 28))
			(erster-abschnitt-simple-muster 28)
			(erster-abschnitt-simple-muster 35)
			(erster-abschnitt-simple-muster 42)
			`((,(erster-abschnitt-flash 2) 50)
			  (,(erster-abschnitt-flash 2) 52)
			  (,(erster-abschnitt-flash 2) 54)
			  (,(erster-abschnitt-flash 10) 56))))
	       :insert nil))

The other eight scenes are generated in a similar way, using different
flash parameters, other effects for the base movies. The resulting
movie is available here (thank you xach), meanwhile the code can be
downloaded here. The movie is
actually quite bad, for the reasons I mentioned above. Because of the
deadline, I didn't take enough time to keep good scenes, understand
the effects, and so on. As a result, the movie is overloaded with
flashes and flickering effects, and has no "real rhythm". However, I
think it is a quite interesting way to go, a way which is pretty
conventional in music for example. I didn't find much on algorithmic
movie editing though, and would be quite interested if someone could
give me some pointers.

Future ideas

The next step is going to be the complete rewrite, and adding a GUI
to the software so that non-hackers can use it. The challenge here is
to preserve the flexibility to integrate new algorithms, combine them,
store them and apply them on different movie snippets. Another
important feature will be to store the process that produced a certain
clip, including the random values produced. It should be possible for
each frame to trace where it came from, which algorithms it went
through, and which random values or external values influenced the
algorithms.

posted by manuel at 3:28 pm  

3 Comments »

  1. You might find ISIS (http://web.media.mit.edu/~stefan/isis/) quite interesting.

    Comment by Marcin Tustin — September 6, 2005 @ 4:58 pm

  2. Nach wie vor. Begeisterung. GrรผรŸe aus Kรถln

    Comment by micha — September 8, 2005 @ 7:17 pm

  3. you might want to look at avisynth

    Comment by mikeytown2 — March 18, 2009 @ 11:53 pm

RSS feed for comments on this post.

Leave a comment

Powered by WordPress