The Closure Problem in Python

August 4, 2012 § Leave a comment

I’ve seen several posts now with people complaining about the following python code:

>>> funcs = []
>>> for i in range(11):
...     def func(x):
...         return x + i
...     funcs.append(func)
...
>>> funcs[1](5)
15

Most people first looking at the code would expect the value of funcs[1](5) to be 6, which it is clearly not, but I've found many confusing and sometimes just wrong explanations for why this is so. I wanted to clarify my understanding of the issue and hopefully provide a simple and clear reasoning for others. Hopefully this is not one of those wrong explanations, and if someone who didn't just learn about lexical scoping today wants to correct me please do. I also may be playing a little fast and loose with some terminology, so let me know if anything is confusing.

The surprise is the result of two non-obvious python features:

for loops do not create their own isolated scope

This is clearly demonstrated with:

>>> for i in range(10):
...     j = 5
... 
>>> j
5
>>> i
9

Where you can see that not only is the loop variable i maintained after the for loop, but j as well. This seems like a somewhat questionable design decision to me, but perhaps that's because I'm primarily a C developer.

Expressions within the body of a function are evaluated when the function is called, not when it is defined

This is also clearly demonstrated with a simple bit of code:

>>> def func(x):
...     return x + i
... 
>>> func(5)
Traceback (most recent call last):
  File "", line 1, in 
  File "", line 2, in func
NameError: global name 'i' is not defined
>>> i = 3
>>> func(5)
8
>>> i = 4
>>> func(5)
9

So you can see, even though i didn't exist with the function was defined, the interpreter happily allowed us to use it in the body of the function, and then only attempted to evaluate it when it was actually called.

Putting these two things together it's clear why python evaluates our first code example the way it does. The variable i inside the body of the function has nothing to do with the loop variable i until the moment that the function in the array is actually called, at which point python looks up the name i in the symbol table and finds the one that happens to have 'leaked out' of the for loop.

Solutions to this issue usually involve forcing evaluation of the variable so that the function references the value instead of the name. My favorite version involves adding the variable as a default value to the function, forcing evaluation at definition time. e.g.

>>> funcs = []
>>> for i in range(11):
...     def func(x, inc=i):
...         return x + inc
...     funcs.append(func)
... 
>>> funcs[1](5)
6

In this version i is evaluated and the value stored in inc when each copy of func is defined.

Now we're cooking with Closures

OK, so things get a little more complicated when we need to specify where exactly the interpreter looks when it finds a variable in your defined function that's not local to the function (a 'free' variable). As mentioned above, the interpreter looks up a value at the time of the call, but it looks in the scope within which the function was defined. More code!

>>> def inc(x):
...     return x + y
... 
>>> def outer():
...     y = 3
...     print inc(4)
... 
>>> outer()
Traceback (most recent call last):
  File "", line 1, in 
  File "", line 3, in outer
  File "", line 2, in inc
NameError: global name 'y' is not defined
>>> y = 1
>>> outer()
5

So here we see that when inc is called within the outer scope, the interpreter looks up y within the original scope where inc was defined, not the scope from which it was called. After we define y where it is in reach of inc, all is well and happy. This is where closures come into play, because what if the scope in which the function was defined no longer exists? In a language without closure support we'd be out of luck, but with closures those references stick around after the function that created the scope finishes executing.

Haskell Spectrograms 2: Array Slicing and Dicing

May 10, 2012 § Leave a comment

Spectrogram of a short string loop

So we have a vector

Given a vector of audio, the first step in making a spectrogram is to slice up the audio into frames. This slicing process is defined by the frame size and the hop. The frame size is the number of samples in each frame, the hop is the number of samples between the beginning of adjacent frames. For instance, if slicing the vector [1,2,3,4,5,6,7,8] with a frame size of 4 and a hop size of 2, the resulting frames would be [[1,2,3,4],[3,4,5,6],[5,6,7,8],[7,8,0,0]]. Note that the last frame was padded out with zeros because it extended beyond the original vector.

Slicing an Array in Haskell

This is where the variety of vector/array libraries in Haskell started to become a bit more of a drag. hsndfile supports Data.StorableVector or Data.Vector as an output type. The haskell-dsp package (available through cabal) uses the standard Haskell Array type. So my first order of business was to convert my freshly-minted StorableVector into an Array.

import qualified Data.StorableVector as V
import Array as A

arrayFromVector :: V.Vector Double -> A.Array Int Double
arrayFromVector vect =
   let l = V.length vect - 1 in
      A.array (0, l) (zip [0..l] (V.unpack vect))

The above code converts the Vector to a list with V.unpack, then uses the array constructor to create an Array of Double (with Int indexes). Not the most elegant or fast, but effective.

Next we need to take this array and slice it into frames. Lets call that function "getFrames", which will take a frames size and hop size and give back a list of subarrays of the original array.

getFrames :: A.Array Int Double -> Int -> Int -> [A.Array Int Double]
getFrames inArr frameSize hop =
   [getFrame inArr start frameSize | start  Int -> Int -> A.Array Int Double]
getFrame inVect start length = pad slice length
   where
      slice = A.ixmap (0, l - 1) (+ start) inVect
      l = min length (end - start)
      (_,end) = A.bounds inVect

Getting a subarray in Haskell is a little bit tricky. The Array library provides an "ixmap" function that takes what you want the bounds of the new array to be, as well as a transformation function to get an index into the old array given an index into the new array. getFrames uses a list comprehension to create a list of slices, each of which is created with getFrame using ixmap. The bounds of the new array are 0 and the length-1, the transformation function to get an index into the original array from the new array is just an offset operation. the pad function comes from the DSP library, available through cabal

The Furrier[sic] Transform

Once we have ourselves a list of Arrays, the DSP library provides an fft implementation for real signals called rfft, which returns the complex spectrum of an input array. Applying the FFT to each frame in our list is a simple map operation. Here we compose a getFrameMagnitude function with rfft, which takes the complex signal and gives us something that we can plot as a spectrogram.

getFrameMagnitude :: A.Array Int (Complex Double) -> A.Array Int Double
getFrameMagnitude frame = A.array (0,(l-1)`div`2) \
      [(i,log (magnitude (frame!(i+(l-1)`div`2)) + 1)) | i <- [0..((l-1)`div`2)]]
   where (_,l) = A.bounds frame
main :: IO ()
main = do
   audioVect <- readWavFile sndFileName
   drawSpec (map (getFrameMagnitude . rfft) \
      (getFrames (arrayFromVector audioVect) 1024 512)) "spec.png"

getFrameMagnitude looks a little hairy, but the gist is that we're using list comprehension to create a new list that's the second half of the input array, where at each sample we take the magnitude, add one to it, then take the log. We add 1 before taking the log so that the log-scaled output will start at 0, for ease of plotting

Join us next time to see how drawSpec works and actually make some spectrograms!

Haskell Spectrograms 1: Loading a Sound File

May 10, 2012 § Leave a comment

Spectrogram of a short string loop

Intro

So I took it up as a challenge to generate a spectrogram in Haskell. How hard could it be? Turns out that what’s a few lines of python code using numpy and matplotlib took me down quite a rabbit hole.

For the impatient, you can find the finished code at https://github.com/ssfrr/scimitar in the “haskell” directory.

Disclaimer: I am not an experienced Haskell programmer, and I’m likely doing things all wrong. Please correct me in the comments

hsndfile

I chose to use the hsndfile wrapper around Erik de Castro Lopo’s immensely useful libsndfile to load the file. This immediately forced me into the abundance of vector libraries for Haskell. This was a huge sticking point for me. There are literally about a dozen different libraries that one can use in Haskell to represent a vector of floats (or vectors of anything storable, but I digress).

Luckily hsndfile only has bindings to 2 of them, which cut down my options. I ended up using hsndfile-storablevector, which is available for installation through cabal. Loading a sound using hsndfile looks a little like:

import qualified Sound.File.Sndfile as Snd
import qualified Sound.File.Sndfile.Buffer.StorableVector as BV
import qualified Data.StorableVector as V

readWavFile :: String -> IO (V.Vector Double)
readWavFile fileName = do
   handle <- Snd.openFile fileName Snd.ReadMode Snd.defaultInfo
   (info, Just buf) <- Snd.hGetContents handle :: \
      IO (Snd.Info, Maybe (BV.Buffer Double))
   return (BV.fromBuffer buf)

To unpack this a little Snd.openFile takes in a filename, a mode, and an Info instance (check this out for more details on the individual datatypes), and gives you a handle to the file (wrapped up in an IO monad, of course).

Snd.hGetContents takes a handle and gives you a IO-wrapped pair. The first item is another Info instance that describes the soundfile you just read, and the second might be a buffer (or Nothing). The tricky bit here is that Haskell doesn't know what type that it's supposed to be, so you need to make the type explicit with the annotation shown above to make it a buffer of Doubles. The cool thing here is that if the datatype you request does not match the datatype of the original sound, hsndfile will do the conversion for you.

Once you have the buffer of doubles, you use the fromBuffer function to get back a normal StorableVector.Vector that you can do other fun things with. Come back next time for some of those fun things!

libmanta released!

March 24, 2012 § Leave a comment

Snyderphonics Manta

(photo nabbed from MusicRadar)

After more than a year(sporadically), several re-writes, and much learning about USB, cross-platform threading, concurrency, OSX device driver loading, and many more, the libmanta library and the included PD and Max/MSP objects are finally ready for release.

Get The Source
Read The Documentation

libmanta is a cross-platform API for the Snyderphonics Manta control surface. The low-level USB communication uses hidapi to communicate with the Manta, and provides a way to subscribe to the different events the Manta can generate, as well as methods to set the LEDs.

There are currently binaries available for Pure Data on linux and OSX. Max/MSP binaries should be available shortly, as soon as Jeff (the inventor of the Manta) finishes the help patch and a wrapper that gives the new libmanta-based object the same interface as the old one.

gendy~ PD + Max/MSP object and libgendy 0.6.0 released!

May 17, 2010 § Leave a comment

I’ve been working off and on for a bit on a C++ library to implement a variant of Dynamic Stochastic Synthesis, a technique developed by Iannis Xenakis. (For a great history and more details check out Sergio Luque’s thesis.

The library is still pretty rough around the edges and more or less completely undocumented, but the PD object that uses it should be pretty functional. Included with the code is a helpfile (gendy~-help.pd), as well as a gui (gendy-gui.pd) that should make it pretty easy to jump right in.

The basic idea of DSS is to describe a waveform with a set of “breakpoints” and interpolate between them to generate the actual audio signal. The set of breakpoints forms a single cycle of a waveform, and each time one cycle gets played, the breakpoints each move for the next cycle. So each breakpoint ends up on a two-dimensional random walk.

One improvement I’ve made to some traditional DSS implementations is the use of cubic spline interpolation instead of linear, which improves the aliasing considerably. It’s not a truly bandlimited signal, but is differentiable.

The other modification is the introduction of selectable center waveforms that the breakpoints will gravitate towards. Right now I only have flat, square, and sine implemented, but I’m planning on including basic triangle and sawtooth as well. The “h_pull” and “v_pull” controls effect how much the breakpoints are pulled towards the center waveform.

The external requires Thomas Grill’s Flext library. Once you have it installed simply run

FLEXTDIR/build.sh pd gcc

From the base gendy directory (Where FLEXTDIR is the location of your flext install) and you should have a shiny new binary in the pd-linux directory. The arguments to build.sh will vary depending on your system.

Downloads:
Source Tarball
x86_64 PD Binary

LTSpice Tutorial #1 – Basic Schematic Capture and Simulation

April 20, 2010 § Leave a comment

This is the first in a series of tutorials intended to introduce LTSpice to the uninitiated. LTSpice has some rather unusual interface conventions that take some getting used to, but it’s an incredibly powerful tool that’s available for free!

The first step is to download it from Linear Technology’s website.

Unfortunately it’s a windows-only application, but the main developer, Mike Engelhardt has gone out of his way to make it work well under WINE, so Linux users aren’t left out of the fun.

In these tutorials I’m going to spend more time talking about using LTspice specifically rather than about the electronics themselves, so I will sometimes assume a certain level of circuit design knowledge.

I should also note that I wrote the bulk of these tutorials about 3 years ago, and am now in the process of migrating them over to this site and improving them. You can see the originals here until the material is fully migrated.
« Read the rest of this entry »

A cute python scheduler

March 2, 2010 § Leave a comment

The Python Logo

I spent a couple of hours on Sunday whipping up this little scheduler in python.

It was mostly to have something fun to do in Python, and doesn’t do much that’s useful, but it does demonstrate some general process scheduling concepts.

It provides base classes for “Processes” and “Events”. Subclass a process to do something when it’s “run()” function is called. A process can also tell the scheduler that it wants to wait on an an “Event”. Any event that has a process waiting on it gets its “occured()” function polled, which returns true when the event has occurred, and the process that was waiting on it wakes up(it’s run() function starts being called again).

I’ve got some examples of a timer event that goes off after a time interval. There’s also a filewatcher event that goes off when a given file is created.

This is meant to simulate a scheduler that would run at the root of an OS, so it loops through as fast as it can and will take up %100 CPU while it’s running.

git clone git://github.com/ssfrr/pysched.git

or

Download the tarball

Follow

Get every new post delivered to your Inbox.