However, all of this convenience comes at a cost. Often, that cost is literal, if an API provider decides to charge for access. This is the entire business model of many companies, and there are now even companies that provide API-monetization-as-a-service. This has created a kind of purely digital marketplace, by literally allowing people to buy access to data and functions. (This is a Good Thing™, as it encourages competition and variety in the API market, and reduces time-to-ship for many developers.)

Many of these monetized APIs are providing access to something inherently proprietary - an enormous dataset, neural network, or advanced algorithm. A problem arises when these APIs provide access to something open. Imagine an API that provides access to datasets that are completely public and free, or an API that performs simple operations on provided data. For an extreme example, imagine an API that implements strlen() - the simple, common task of finding the length of a string.

Imagine a group of young, new programmers building the next cool app. They need to find the lengths of their strings, as you do. As C is hard to learn, and reading documentation would take too much time, they instead outsource their character-counting to api.strlen.com. (The venue of the hackathon has great wi-fi, so the additional network overhead is not a big deal.) They launch the app to much fanfare, pitch it, then win first prize at their Startup Weekend.

Months later, thousands of people are using their hot new app. They’re soaking in the TechCrunch coverage and brainstorming monetization ideas when, suddenly, their app stops working. They trace the error down to one call - to api.strlen.com - and finally see:

>HTTP/1.1 429 Too Many Requests

It’s 11pm on a Saturday night. The folks that run strlen.com are nowhere to be found.

Monday morning rolls around, and our favourite team of programmers has barely slept. Their star app has been down all weekend. Finally, they get an email back from support@strlen.com:

Congratulations on the TechCrunch coverage! Unfortunately, you’ve way exceeded our rate limit (in fact, we had to put a rate limit in place just because of your app) and we need to chat. We’re now charging $0.0001 per character counted with our string length API. Let me know if you’re interested in upgrading your free account and I can get you set up!

-Bjørn, CEO, strlen.com

The team runs the numbers to find that, with the new rates, every additional user of the app would lose them a ridiculous amount of money every day. But, hey - they just had a great chat with an angel - and they might be getting some financing soon. They send Bjørn their details and get set up with a paid account. The app starts working again. Their users are happy, TechCrunch comes by for another interview, and the team’s reason for sleeplessness goes from “anxiety” back to “coding.”

It’s been a month since our team - now incorporated as Blue Blanket, Inc. - signed up for their paid strlen.com account. Software engineers are expensive, and while they’ve considered hiring somebody to write their own version of the strlen.com API, they’re really not sure where to start. Their fancy new analytics dashboard shows increasing numbers, minute over minute, until - all at once - the graphs go dead. The app is down again, and once again, it’s due to strlen.com. The team points their web browsers angrily at api.strlen.com, only to find:

>HTTP/1.1 410 Gone

The homepage of strlen.com has an even bleaker message:

Dear friends,

We at strlen.com are very proud to announce that we’ve been acquired by Standard Library Incorporated. It’s been a wild ride counting characters for you over the past six months, but we’re excited to move on and solve hard new problems with the great people at stdlib. All API endpoints will be disabled, effective immediately.

-Bjørn, VP String at stdlib.com

Our trusty team’s app stays down for the better part of a month while they scrounge up a handful of competent engineers to recreate the missing functionality. Once back online, their app is all but forgotten. TechCrunch runs an article a year later - “What ever happened to Blue Blanket?” - that places the blame on a power struggle between the co-founders.

Obviously, implementing strlen as a paid API is an absurd example, but there are real APIs out there that are not much different. If you depend on an external service for your app’s core functionality, that’s okay. But if you can feasibly replicate the API yourself, then relying on the external service is a source of extremely risky technical debt. Your debtors (in this case, the API providers) could demand immediate repayment at any time by rate limiting or shutting down.

Don’t let your app be crippled by someone else’s acquisition.

Thanks to Zameer Manji for proofreading this post.

Over the past year, a number of Waterloo interns have had the pleasure of interning at Khan Academy, the groundbreaking non-profit dedicated to “accelerate learning for students of all ages.” They’ve made such an impression on Sal Khan, its founder, that he’s gone on to speak extremely highly of Waterloo - even suggesting using its model as a base for furthering education - in an extremely well-written article in Communications of the ACM:

Waterloo has already proven that the division between the intellectual and the useful is artificial; I challenge anyone to argue that Waterloo co-op students are in any way less intellectual or broad thinking than the political science or history majors from other elite universities. If anything, based on my experience with Waterloo students, they tend to have a more expansive worldview and are more mature than typical new college graduates—arguably due to their broad and deep experience base.

While every student gains valuable experience and ends up hugely enriched by their time in co-op, those in technical disciplines arguably have the opportunity to make a more lasting impact. In particular, the nature of the software industry allows co-ops to contribute to a company on an extremely meaningful level. Classmates of mine interning at companies like Facebook, Google, Square and Twitter have made contributions that are on par with - if not exceeding - those of senior full-time employees. It’s hugely exciting, and in my experience, it makes us interns forget that we’re only interns.

The Abysmal State of Higher Education

While I can’t extoll the virtues of the co-op system enough, it does set co-op students apart from the general student body in unnerving ways. Co-op students, demanding competitive wages during their internships, often do graduate with little-to-no student debt, while traditional university programs might give no workplace experience and leave a student with loans of tens-of-thousands of dollars. It’s immensely depressing to realize that outside of the little bubble of co-op, that’s the norm for higher education. Almost everybody that goes to college experiences student debt and difficulty finding employment.

It boggles the mind that for many of today’s students, it’s normal to graduate with zero industry experience before immediately searching for a job. Companies shouldn’t have to guess if a candidate, fresh out of school, can apply their theoretical skills to a workplace that is foreign to them. The question:

should we even consider hiring a fresh college grad?

should never have to be asked. Regrettably, with the current state of higher education, being a “fresh college grad” doesn’t give any confidence to an employer, at least for graduates of most schools. (And yet it leaves each student in considerable amounts of debt. What an amazingly flawed system.)

The class of Software Engineering students that I belong to will graduate in just over a year, after having spent five years at Waterloo and abroad. Most will graduate with no debt and with multiple job offers in hand, obtained from real world experience that complements their theoretical knowledge. Why must this be a unique situation? We should not be the outliers - this is how higher education should be.

Rather than being decisive and choosing on a single colour to define the site, I decided to let the colour of the background dictate the main colour of the site at any given time. As my old design featured hundreds of random thumbnails on its homepage, I opted to use those exact same thumbnails - just blown up, blurred, and very colourful, as the randomly-chosen backdrop for each page.

To accomplish the text knockout effect in the site’s header, I simply used a transparent PNG, as it’s lighter to bake in the font to a PNG than embed Proxima Nova. However, I also wanted to knock out each h1 and h2 in the body of each blog post. To do so, I turned to HTML5 canvas.

Titles that look like this, punching out to the background!

Canvas has a number of different compositing modes that allow Photoshop-style blending tricks, although somewhat less complex.

To reproduce Photoshop’s knockout effect, I simply turned to the destination-out compositing mode:

ctx.globalCompositeOperation = 'destination-out'

However, it wasn’t quite that simple. To allow smooth fallback for older browsers (or even browsers like Safari that don’t fully support compositing modes) I wanted to dynamically swap each tag with a canvas tag after each page load. To replicate the style of each tag, I turned to a very useful function I hadn’t used before: getComputedStyle. To get every single style on any tag, I had to simply run:

style = document.defaultView.getComputedStyle(this.element, "")
//  style.paddingTop, style.paddingBottom, etc...

Thus, on document.ready, all I had to do was:

• Find the computed styles of each tag to be replaced
• Create a new <canvas> element with the same outer width and height
• Apply the original styles to a new <canvas> element
• Move the padding of the element to inside its ‘width’ (breaking the box model)
• Set the compositing mode to destination-out
• Draw the text on the canvas
• Swap the original element for the new canvas

Et voilà, it works and looks awesome. (…in some browsers.)

As always, code is on Github:

That’s 25% of my little 512MB Linode’s monthly bandwidth allotment being used up in 6 hours. With Linode (as of this writing) charging $0.10/GB for bandwidth (allotted or through overages), that huge server load could get very expensive, very fast. (At that rate, each listener would cost me roughly $0.25 per day of constant listening. Not viable for a free service!)

So, this afternoon, I was faced with a dilemma. How do I quickly and easily make it cheaper for me to host the site at peak times? A tried and true CDN would be a good solution, but even simple CDNs like Amazon CloudFront would cost more than my existing Linode. (Such systems are generally made for scaling to multiple petabytes, while I’m looking at maybe 1TB tops.)

Instead of going for a large, expensive CDN, I decided to make my own small one. Currently, it contains exactly two nodes: the original forever.fm streaming server, and the hugely overpowered VPS I use to serve the Wub Machine, my other major music hack.

This “CDN” is simple: I’ve added a single Python script to forever.fm that acts as a basic “relay” server. Each relay has a copy of the repo, although it runs python -m forever.relay start rather than python -m forever.server start. Each relay listens to the stream from the “root” url and re-broadcasts it to n users. Then, each time a user requests a new stream from the root, the logic is simple:

if len(self.listeners) > config.relay_limit:
    self.redirect(random.choice(config.relays))

I’ve got more features to add to the relay system - namely, each relay should be able to send back statistics about number of listeners, user agents, and more back to the central server for logging and live status monitoring. Relays could also be smart and stop listening to the “source” stream if nobody is listening to them - preventing additional bandwidth usage. However, with this very simple star pattern, the single stream can be efficiently broadcast to hundreds (if not thousands) of listeners.

As always, the code is available on github.

WARNING: Past this point, you’ll find only gory technical details of how forever.fm was made.

Overview

Forever is powered by a large number of technologies, some of which I stole from my previous music hack, the Wub Machine:

• For this project, I chose to use a 512MB Linode VPS, which has been running spectacularly.
• The entire site runs on Python and uses Facebook’s Tornado evented server.
• To stream track metadata, waveforms, and other live updates, I’ve used Socket.IO and tornadio2, its Tornado wrapper.
• the SoundCloud API provides the songs, metadata and audio streams that you hear.
• the Echo Nest Remix API analyzes each song to find the best beats for beatmatching.
• The Echo Nest released some cool beatmatching examples back in 2010, and my beatmatching code is heavily based off of their capsule example. (Although heavily patched to fix memory leaks, allow infinite execution and lighter CPU usage.)
• LAME and FFMPEG for efficient encoding and decoding of the MP3 stream.
• I found some great code for approximating the Travelling Salesman Problem by John Montgomery. This is used for ordering tracks - more on that later.
• Scott Schiller’s spectacular SoundManager2 JavaScript library and 360º player UI play back the audio stream in-browser and provide very neat visualizations.
• the Python Imaging Library is used to colour, stylize, and fade the waveforms of each song.
• Charles Leifer’s algorithm for using k-means to find the dominant colours in images is used to colour each track based on its album artwork.

Streaming MP3 in Python

The toughest problem to solve when creating Forever was that of live streaming. The core beatmatching algorithm at its heart (”Capsule”) has existed for a couple years now. However, making this run infinitely required some different approaches.

Python’s built-in generators provide a great way to implement an iterative beatmatching algorithm, as each generator can carry its own internal state. In this case, that state is the last-played song. Here’s some pseudocode:

def forever(track_queue):
    t1 = track_queue.get()
    while not track_queue.empty():
        t2 = track_queue.get()
        yield make_transition(t1, t2)
        t1 = t2

This code is obviously oversimplified, but basically how the core of Forever works. Assuming that forever constantly yields raw audio data (i.e.: WAV, AIFF, PCM), this needs to be encoded to MP3 and streamed out to the user.

To tackle this MP3 problem, I created a LAME MP3 encoder interface in Python that allows real-time, buffered and synchronized MP3 encoding.

With this simple interface, Forever hands off chunks of large, raw audio data to LAME for MP3 encoding. Python then reads single MP3 frames from LAME as they become available and puts those frames into a queue. (If the queue is full, this blocks and prevents more raw audio from being generated, essentially throttling the entire process via negative queue pressure.)

Finally, someone has to read from this MP3 queue. After trying custom thread-based solutions for throttling this properly, I instead settled on a much more stable solution - asking the Tornado web server itself to hand out each MP3 frame as it becomes available. Surprisingly, this is extremely stable, and results in a perfectly real-time stream with no lead or lag:

SECONDS_PER_FRAME = 1152.0 / 44100  # As defined by the spec
seconds_to_buffer = 60
mp3_queue = Queue(int(seconds_to_buffer / SECONDS_PER_FRAME))
. . .
def send():
    frame = mp3_queue.get_nowait()  
    if not frame:
        print "OH NOES, MP3 queue is empty!"
        return
    for listener in listeners:
        listener.send(frame)
. . .
PeriodicCallback(send, SECONDS_PER_FRAME * 1000).start()

This strategy has not (yet) been load tested, but is nevertheless the system I’m using in production at the moment. For all I know, this could fail completely while serving a large number of listeners. Testing locally by spinning up 200 instances of CURL, this performs admirably and causes Python to use only 5% of my aging Macbook’s CPU.

Interestingly, placing a limit on this final MP3 buffer propagates this queue pressure backwards all the way to the audio generator. If the output MP3 buffer is full, the LAME wrapper will be blocked until it can write to the queue. If LAME’s output is blocked, then an internal semaphore will block in the LAME wrapper’s input function, which will delay the audio generator. (Internally, the LAME wrapper writes all of the PCM to the LAME process at once, to prevent encoding delay and decrease memory usage by only storing lightweight MP3 instead of heavy PCM. This blocking behaviour is artifically implemented to save memory.)

Memory Leaks and Python C Extensions

In the process of implementing Forever using the Echo Nest’s action and cAction libraries, I ran across an absurdly annoying bug that took me into the depths of Python C extensions. Each time I executed a Crossfade.render or pydirac.timeStretch call, I lost memory. (A lot of memory - often between 25 and 100 MB.) As Forever is built around these methods, I couldn’t have used them as-is - but I couldn’t find a suitable replacement, as they were written specifically for this purpose by Tristan Jehan, a co-founder of the Echo Nest.

So I busted out Apple’s Instruments, which include a memory profiler. Upon initial search, I found no actual memory leaks. (i.e.: There were no calls to malloc without a corresponding free.) However, memory was increasing linearly over time on the smallest possible test case - something was definitely being leaked!

As Python is a garbage-collected language, I then turned to the internal heap in an attempt to find objects without references. After trying guppy and heapy to find the size of Python’s heap, I realized it wouldn’t help. PCM audio in the Echo Nest API is stored within numpy arrays - which are garbage collected, but whose memory is allocated outside of Python. Searching for numpy arrays with heapy, guppy, or the wonderful objgraph ends up being relatively futile, as the large chunks of memory that you’re searching for won’t be in the scope of Python.

This left one possibility - the C extensions being used were leaving a reference to the large Numpy array somewhere. As it turns out, this was the case. (Discovered by manually reading the code, finding every PyObject*, and tracing it to ensure that its reference count was handled properly.) The solution?

. .  . . 
     215    +    Py_DECREF(inSound1);
     216    +    Py_DECREF(inSound2);
215  217         return PyArray_Return(outSound);
. .  . .

As it turns out, a Numpy array allocation function (NA_InputArray) was being used incorrectly. The docs state that the return value of this function should should always be DECREF‘d, but they weren’t. It’s that simple, but with such a huge impact.

The Dreaded GIL

After fixing the memory leaks, optimizing the beatmaching algorithms and ensuring that the system runs indefinitely, I ran into another problem - Python’s global interpreter lock. As Wikipedia explains succintly, a GIL is:

a mutual exclusion lock held by a programming language interpreter thread to avoid sharing code that is not thread-safe with other threads. In languages with a GIL, there is always one GIL for each interpreter process. CPython and CRuby use GILs.

This poses a big problem for Forever. The Tornado server has to deal out MP3 frames in real time to each listener, so any execution delays will cause noticeable audio dropouts. Worse still, the important audio operations needed to beatmatch songs (including calls to the aforementioned C extensions) are very computiationally expensive, often holding the GIL for seconds at a time. (Note: there exists a simple way to release the GIL from extension code that I haven’t yet tried, which might mitigate the issue.)

To work around this and ensure total isolation between heavy, blocking audio operations and efficient, real-time MP3 streaming, Forever makes use of Python’s multiprocessing module to split itself into server and worker processes. By opening a queue (well, multiple queues) between the server and the worker, the server can consistently stream MP3 packets in real time, while the worker thread can block and hold the GIL for any amount of time.

In theory.

Unfortunately, the synchronized Queue class provided in the multiprocessing module buffers its data in the sending process, not the receiving process. This means that even if the Queue is full of audio, the GIL of the worker process could be acquired by one thread, preventing the server process from reading any data from the queue until it is released. This doesn’t help at all.

To work around this restriction yet again, I created a simple BufferedReadQueue class that eagerly fetches all of the data from the child process and simply buffers it in the parent, allowing the parent to read data even when the child’s GIL is blocked, and providing an isolated buffer of audio data to safeguard against dropouts.

Cycles of Similar Songs

A significant part of what makes Forever sound good is its choice of tracks. One of the hardest problems that a DJ faces is choosing which tracks to play in their set, and Forever is no different. To solve this problem programatically, I created a module called the Brain and turned to graph theory.

Forever starts by grabbing a list of the top n tracks from SoundCloud, ordered by “hotness.” It then culls this list by removing songs that are too short, too long, or duplicates of other songs in the list. Forever then arranges the remaining tracks in a complete graph, where each song is a vertex, and each edge describes a measure of “distance” between tracks - an inverse of similarity, if you will. For example, if two tracks have similar tempo, tags and genre, their similarity will be high, making the edge weight between them very low.

Here’s an example of the edge weights in a simple three-song graph:

To play these songs forever, the problem reduces to finding the lowest-weight cycle in the entire undirected graph… which requires a solution to the Travelling Salesman Problem. As TSP is one of the great NP-complete problems in computer science, I resort to using approximation algorithms. Forever uses a great bit of Python code for a hill-climbing approximation, written by John Montgomery to “solve” this problem. After running the approximation for something like 10,000 iterations, I accept the solution and use that to order the resulting tracks.

After the track order is determined, the worker process receives these tracks and begins to remix them in order when necessary. At this step, the worker fetches the Echo Nest’s analysis (a very lengthy call) to find the metadata required to beatmatch each track. With this data comes a great “summary” that includes many factors that would be useful for determining track order - including energy, danceability and loudness. Hence, this summary data is cached in a SQLite database to allow the Brain to make better decisions.

Live Brain Transplants (Reloading Modules)

Once Forever was working without dropouts or stalls, I faced another issue. If I wanted to make a code change to any of the core algorithms, I’d usually just restart the server. However, if there are people listening to the radio stream, this would cut them off, as the underlying socket connection to the server would be broken as soon as the Python server process is killed. Hence, to ensure a truly endless stream of audio, I had to find a way to hot-swap portions of code that I might want to update frequently.

To test this, I started with Forever’s Brain module. To wrap the Brain in a container that could easily hot-swap its internal logic, I created another module - aptly-named “skull.” This module performs the infinite loop around the Brain’s logic, calling the Brain as a generator and adding its results directly to a limited queue. In short:

class Skull(threading.Thread):
    def __init__(self, track_queue):
        self.track_queue = track_queue

        import brain
        self.brain = brain
        self.loaded = self.modtime

        threading.Thread.__init__(self)
        self.daemon = True

    @property
    def modtime(self):
        return os.path.getmtime(self.brain.__file__)

    def run(self):
        g = self.brain.add_tracks()
        while True:
            if self.modtime != self.loaded:
                log.info("Hot-swapping brain!")
                self.brain = reload(self.brain)
                self.loaded = self.modtime
                g = self.brain.add_tracks()

            track = g.next()
            log.info("Adding new track to queue.")
            self.track_queue.put(track)

I’ve since extended this concept to a number of other core modules - namely, the beatmatching generator, as I’m currently trying to increase its efficiency every day, and the MP3 decoding/encoding classes, as they’re still a bit too memory-hungry for my liking.

Conclusions

Try out Forever.fm. If you like the kind of music that’s popular on SoundCloud (currently lots of EDM) then you’ll enjoy it. I have big plans for it from here on out, but it was super fun to build, and I learned a ton. As always, feel free to email me or tweet at me if you have any questions.

On large JPEGs from my own photo library, like the one above, this Ruby script takes roughly 2 seconds to perform a smart 124px square crop on the most interesting part of the image:

Matched 9 images.
Originals/2012/NYC/IMG_7054.JPG => ./18.jpg in 1801.717ms
Originals/2012/NYC/IMG_7055.JPG => ./19.jpg in 1856.692ms
Originals/2012/NYC/IMG_7052.JPG => ./20.jpg in 1787.717ms
Originals/2012/NYC/IMG_7059.JPG => ./21.jpg in 1727.487ms
Originals/2012/NYC/IMG_7057.JPG => ./22.jpg in 1716.977ms
Originals/2012/NYC/IMG_7056.JPG => ./23.jpg in 1692.648ms
Originals/2012/NYC/IMG_7058.JPG => ./24.jpg in 1887.043ms
Originals/2012/NYC/IMG_7051.JPG => ./25.jpg in 1977.311ms

As I often run this algorithm on entire folders of images at once, I decided to experiment and reimplement the entire program in C++. As a developer that works primarily with Python and Ruby, I’ve always felt a small amount of guilt for incurring the crazy performance overhead of interpreted and heavily dynamic languages. (A friend of mine working in the hardware industry recently got angry over the fact that he was working to make chips faster, while us developers then “throw away” the speed gains by running interpreted languages!)

Trying libjpeg

While the original Ruby script took maybe 2 hours to write, my straight port to C++ took more than 10! Most of this time was spent navigating the API of libjpeg, fumbling with pointers and buffer arithmetic, and hunting down type casting errors that subtly caused inaccurate results. However, check out the speed gains:

Thumbnailing 9 images...
Processing Originals/2012/NYC/IMG_7051.JPG... 63ms.
Processing Originals/2012/NYC/IMG_7052.JPG... 41ms.
Processing Originals/2012/NYC/IMG_7053.JPG... 5ms.
Processing Originals/2012/NYC/IMG_7054.JPG... 33ms.
Processing Originals/2012/NYC/IMG_7055.JPG... 36ms.
Processing Originals/2012/NYC/IMG_7056.JPG... 31ms.
Processing Originals/2012/NYC/IMG_7057.JPG... 35ms.
Processing Originals/2012/NYC/IMG_7058.JPG... 30ms.
Processing Originals/2012/NYC/IMG_7059.JPG... 34ms.

The same sample images that took ~2 seconds to process in Ruby take, on average, 35ms to process in C++. That’s a speed up of more than 50x. This happens to line up roughly with the popular language benchmarks that put Ruby at ~45x slower than gcc-compiled C++, despite the fact that a lot of the work is done by RMagick. (I should point out that these programs are not exactly identical - the libjpeg version makes some small feature concessions in the name of speed. Their output is nearly identical, however.)

What’s also interesting is the cost in developer time and code quantity. The original Ruby script was ~80 lines, give or take comments - while my C++ port is ~350 lines. In this one isolated, little-optimized, amateur test, C++ took 4x the code and 5x the development time to deliver 50x the performance.

Trying Magick++

However, this joyous speed boost was short-lived. I soon discovered that my libjpeg-based solution was quite buggy. Most cameras nowadays don’t rotate the raw image from the sensor before encoding to JPEG, preferring a lossless “orientation” flag in the EXIF data instead, forcing the decoding library to parse this to display the image upright. Unfortunately, libjpeg doesn’t contain any built-in facilities to “right” an image with such a tag, and doing so manually is extremely difficult. In addition, I hadn’t written any custom image scaling code, so I depended on a libjpeg flag to scale down the input image by a power of two before decoding.

Faced with this insurmountable rotation bug, and after spending another 8 hours trying to fix it, I decided to yet again rewrite the solution using Magick++, ImageMagick’s C++ client library. Without further ado, the benchmarks:

Thumbnailing 9 images...
Processing Originals/2012/NYC/IMG_7051.JPG... 340.717ms.
Processing Originals/2012/NYC/IMG_7052.JPG... 287.965ms.
Processing Originals/2012/NYC/IMG_7053.JPG... 94.133ms.
Processing Originals/2012/NYC/IMG_7054.JPG... 279.776ms.
Processing Originals/2012/NYC/IMG_7055.JPG... 286.434ms.
Processing Originals/2012/NYC/IMG_7056.JPG... 281.245ms.
Processing Originals/2012/NYC/IMG_7057.JPG... 289.052ms.
Processing Originals/2012/NYC/IMG_7058.JPG... 280.193ms.
Processing Originals/2012/NYC/IMG_7059.JPG... 283.7ms.

For numerous reasons here (Magick++ overhead, image pre-scaling, orientation correction) the Magick++-using code falls right in the middle in terms of performance. It’s ~8 times slower than using libjpeg directly, but much more correct and still more than 5 times as fast as the equivalent Ruby code.

This final version of the program took about 2 hours to put together, with most of that time spent searching for auto-orientation code (eventually pulling it out of Magick’s Mogrify command-line tool) and optimizing for speed.

The Value of Abstractions

When given the right amount of abstraction - in this case, a fast C++ library - writing the code to be adequately fast was trivial. Using old-school C-style library integration, on the other hand, ended with me wasting hours making little to no progress. The resulting program was indeed much faster, but questionably worth the time and frustration. (My head still hurts from getting improper call to jpeg library in state xyz errors repeatedly, only to find zero helpful documentation on each error state.) Using a low-level library simply requires more knowledge and more mental state than any commonly used high-level language. (Of course, this sounds obvious.)

This brings up an important point on the state of popular (and slow) languages today. When acceptable speeds are measured in seconds rather than in milliseconds, it makes perfect sense to write slow and inefficient code quickly. Every time a Rubyist runs gem install, they’re abstracting away the low level implementation details in favour of a simple interface that helps them solve their problem faster. Amortizing the cost of running the program over its runtime, rather than its development time, is logical considering the absurdly high price of a professional software developer.

It’s not that I don’t want a good camera, it’s that I already have a camera good enough. Most of us have one on us at all times.

It’s my smartphone, and it can capture images like this:

This image isn’t pristine. It’s vibrant, although it could be moreso. It’s lacking in detail, a bit noisy, and somewhat compressed. (Compressing it for web didn’t help with the presentation, either.) However, none of that is important. It’s a beautiful reminder of that moment - a great lunch with a great person, on a bright and sunny day, in a crowded restaurant. Having that moment captured is vastly more valuable than the quality of the image.

It’s true that my smartphone doesn’t take beautiful 18MP stills, nor does it have an immaculate 10x optical zoom. The lens isn’t removable, nor is the sensor very large. Low-light performance is horrible. The built in HDR mode, while better than nothing, often produces horrible artifacts and delays my next shot for seconds. I have no control over aperture, ISO, shutter speed or white balance.

Most of this doesn’t matter though, as my phone is always in my pocket. What I lose in image quality and configurability, I gain in ubiquity. I might forget to grab my camera before I leave the house - and I’ll definitely have to think twice about bringing a huge DSLR along. But my phone? The only places I might not bring it are the shower and the swimming pool. I certainly use my camera enough, now that I’ve found an interesting use for lots of low-quality smartphone photos.

Apple’s iPhones 4 and 4S are now the two most popular cameras on Flickr.

Interestingly, people often embrace the lack of quality in smartphone photos. Photo filters are all the rage. When beautiful photo quality can’t be achieved, people delight in reducing quality even further by adding artistic emotion with a filter. (Ostensibly, most Instagram users don’t think that deeply about the filter they choose.) Instead of creating art via careful manipulation of a fancy camera, people do it with a one-touch filter.

Extending the idea to another medium, smartphones often have adequate microphones. As a musician with a penchant for experiments in audio, I do a lot of recording. I nearly bought an expensive Zoom field recorder recently, only to stop myself again in favour of my phone’s more-than-adequate microphones. To capture musical ideas or field recordings, it’s perfect for the same reason - my phone is always with me, and is good enough.

Most of my recent songs are built around samples taken with my iPhone. ”Train In The Sky” takes its snare sample from the closing of a Vancouver SkyTrain’s doors. ”Mace and Anvil” takes part of its melody from the same train’s public alert sounds. ”Somnambulist” (below) starts with samples (albeit, altered samples) of myself walking, and is filled with the enthusiastic cries of the chefs at a Japanese restaurant. All of these samples, while relatively low-quality, were taken with my phone.

As Chase Jarvis puts it, “the best camera is the one that’s with you.” The ability to capture any moment at any time, no matter the quality level, is key. For all I care, my pictures and sounds could come out as a grainy, hazy mess. As long as I can extract meaning and value from them, I’ve captured a moment and strengthened a memory.

In theory, it’s simple enough, but I did run into a few very nasty problems when dealing with multiprocessing in Python:

• The multiprocessing module, at least on *nixes, forks the current process and communicates with the child with a pipe. This works wonderfully if the data you’re transferring can be easily pickled, and if the child process doesn’t need to modify any global state in the parent. Unfortunately, certain useful constructs in Python can’t be pickled, including functions and lambdas (or pretty much anything callable).

  In my app, I had a peculiar use case - I would callback pass lambdas into the constructor of a class, then spawn another process on one of that class’s methods like so:

      class MyClass(object):
          def __init__(self, my_callback):
              self.my_callback = my_callback
  
          def start_work():
              p = multiprocessing.Process(target=self.do_work)
              p.start()
              p.join()
  
          def do_work():
              # Calculate fibonacci or something, iunno
              self.my_callback("hey look, some data!")
  

  This lead to a baffling bug - while the callbacks were being run, their side-effects weren’t persistent. I inserted logging in the callback function to verify, and noted that not only was it running, but at the end of its execution, the global state had been set properly. However, from the perspective of the parent process, nothing had changed.

  The reason was simple: the callback had been run in the child process and had modified the global state of the child process, not the parent. A simple fix would be to have eliminated these callbacks, but I instead used some of Armin Ronacher’s bad ideas in Python to create an experimental module that allows pseudo-function-calls between processes. Use (or even just read) at your own risk - it’s a hack.

• Logging, the wonderful built-in Python module for meticulously logging everything, is thread-safe. Sadly, it doesn’t seem to be multiprocessing-safe. Logging makes use of its own internal I/O thread, to ensure that all log messages are properly queued and written without clobbering eachother. This thread is locked for every write.

  After forking another process, the first call to the logger often hangs while waiting for the logging thread to become unlocked. If the logging thread was in use (i.e.: locked) at the exact instant the process was forked, then the locked thread will be copied to the new process. However, whatever log operation was in progress will then unlock the original thread, not the copied thread, leaving the new process to wait forever on a lock that will never be unlocked.

  The solution, in my case, was to replace the logger in use with the one provided by multiprocessing if logging from a new process:

  def initlog():
      if multiprocessing.current_process().name == "MainProcess":
          _log = logging.getLogger(config.log_name)
      else:
          _log = multiprocessing.get_logger()
      ...
  

To find and fix these bugs took a lot of time, and a good debugging strategy. The most valuable tool turned out, surprisingly, to be GDB. GDB 7 has support for debugging Python runtimes, complete with pseudo-stack traces. Take a look at the following backtrace of a Python process provided by GDB and formatted for clarity:

    [Thread debugging using libthread_db enabled]
    [New Thread 0xb0c2fb70 (LWP 12895)]
    0x006da405 in __kernel_vsyscall ()

    Thread 1 (Thread 0xaf23ab70 (LWP 12894)):
    #0  0x006da405 in __kernel_vsyscall ()
    #1  0x003a27d5 in sem_wait@@GLIBC_2.1 ()
                  from /lib/i386-linux-gnu/libpthread.so.0
    #2  0x080f2139 in PyThread_acquire_lock (...)
                  at ../Python/thread_pthread.h:309
    #3  0x080f2fd8 in lock_PyThread_acquire_lock (...)
                  at ../Modules/threadmodule.c:52
    #4  0x080da7d5 in call_function
                    (f=Frame 0x937b47c,
                      for file /usr/lib/python2.7/threading.py,
                      line 128,
                      in acquire
                      (self=<_RLock(...) at remote 0x9caabec>,
                        blocking=1,
                        me=-1356616848),
                      throwflag=0) at ../Python/ceval.c:4013
    ...
    (goes down 79 frames)

Obviously, this looks much more complicated than a normal Python stack trace, but it’s a huge step up from zero debugability. If I proceed down a couple more frames, I find:

    #7  0x080dac2a in fast_function
                      (f=Frame 0x9ca278c,
                      for file /usr/lib/python2.7/logging/__init__.py,
                      line 693,
                      in acquire (self=<FileHandler(stream=...

…which is the first piece of familiar code. Line 693 of logging/__init__.py is surrounded by a short function, and has a comment that brings the first bit of understanding:

    def acquire(self):
        """
        Acquire the I/O thread lock.
        """
        if self.lock:
            self.lock.acquire()

Well, there you go. After fixing these race conditions and deadlocks, the Wub Machine’s success rate immediately jumped from horrible to 95% under load.

All it took was GDB and an understanding of fork() to solve these bugs. My only advice: be very, very, very careful when using multiprocessing.

Enter A Meal for Me. Roughly 200 lines of code for a fun site that now helps me be more adventurous in the kitchen. (Grab the source on GitHub!)

Development took a couple hours, and was simple enough:

1. Have dinner.
2. Google for “Recipe API.”
3. Get an API key.
4. Layout a simple page in HAML.
5. Style with SASS.
6. Wire it up to the API, and use some basic jQuery to munge the data.
7. Apply a Google Web Font and subtle background pattern to make things look good.
8. Apply API caching in Nginx.
9. Sleep.

To save time (and lines of code), the site is nearly 100% in-browser. It makes use of the wonderful Punchfork recipe API to grab data, then simply formats the resulting recipe cleanly and simply, providing an image of the meal and a link to instructions.

The site also makes use of HAML, SASS and CoffeeScript, rather than HTML, CSS and JavaScript. This saved a significant amount of development time, and allowed me to use third-party style mixins like Bourbon. Images are sparse - the only .png files are the favicon, Punchfork reference, and the background, which was graciously taken from SubtlePatterns. Google Web Fonts also came in handy here, providing a well-suited font after roughly 60 seconds of searching.

To tie it all together are two non-browser components - a Rakefile and an nginx config. The Rakefile allows me to easily compile the HAML, SASS and CoffeeScript before deployment, and also fetches the required mixins (Bourbon). It could very easily be extended to watch the files during development, making the feedback loop much quicker.

The nginx config, on the other hand, serves two purposes: to cache queries to the Punchfork API, and to hide my private API key. As their API is rate-limited, I cache every query for 24 hours to make best use of the data I get. I also hard-code my API key in the nginx config, to prevent others from reading it from the client-side code and using it. All of this is quite simple to do with the proxy_pass and proxy_cache directives:

Although I haven’t load-tested or browser-tested the site, I’m done. Its mission was to provide an evening’s worth of learning and challenge. Now, it can hopefully help some others learn how to as well - and if nothing else, it’ll help me learn how to cook more things.

Note: I’ve definitely made some errors in the code. If you find any, or even just have any suggestions or comments, please do get in touch.

Years later, as I started university and was accepted into VeloCity, Waterloo’s startup incubator, I noticed a lot of familiar dreams. Although their domains are vastly different, startups are just bands for hackers.

Like many bands, many startups begin in parents’ garages - serious business in a non-serious atmosphere. Founders (or band members) spend every waking moment together, working hard to perfect their craft and their endeavour. While bands work to perfect their lyrics, melodies and rhythms, startups work to perfect their pitch, product and business plan. Instead of fans, startups get users; instead of trying to impress the A&R rep that may show up to a performance, startups try to impress the investors in the audience on demo day. For bands, this demo is a rough CD - for startups, it’s their first product. Both are MVPs.

Then comes recording. Record labels often finance their artists, allowing them to spend months in studio making a record, toiling day and night behind a recording console. This is where the product comes from, the hard work that is initially the band’s raison-d’être. (Their private IP, if you will.) Startups are often financed by VCs, or by themselves, to make their initial product. For months, they toil in their studio apartments, toiling day and night behind a Linux console. By now, both band and startup have something nobody else does - their work. While they both have funding, neither of them have revenue.

Then comes touring. It’s no secret that this is how many musical artists make most of their revenue. While the original music may be great, one purchase of their album makes them pitiful amounts of money. Many SaaS startups are in the exact same position. Selling their software directly would be equivalent to selling the copyright on a band’s music - a nice one-time sale, but the loss of all of the private IP. Startups tour just like musicians do, and attempt to generate revenue. They acquire customers, attempt to gain virality, live off user (fan) counts and try to bring in whatever cash they can. They could take the advertising approach, and play a show sponsored by a large brand, or they could make money from their users directly by charging for tickets.

Then comes the pivot. A startup can quickly realize that their idea isn’t profitable, or that nobody’s interested. Their team obviously has the skill, so they choose to make a different product. A band can quickly realize that their music isn’t liked, or that nobody’s interested. They obviously have the skill as musicians, so they broaden their artistic horizons and make new music in a different genre. Neither type of pivot is bad for the organization, although they will both lose fans or customers. In either case, the funding party (VC or Record Label) will definitely be involved in the decision.

Bands and startups are also identical in one other important area - motivation. There exist bands that are motivated by money, just as there exist startups whose sole goal is to generate income. These bands still make art, although it is rarely well accepted (or noticed) by critics, and often written for hire. (The Backstreet Boys and Justin Bieber are two examples that come to mind.) Startups with the one goal of profitability may achieve success, but are often similarly panned, and often stumble frequently on the way. (Groupon.)

On the other hand, musicians who set out to make great music are, at the very least, taken seriously. Music reviewers devote their attention to those who care about their work, and it’s rare to see a “classic” or universally-adored piece of music that was written for hire*. Startups that set out to make a great product are, on the whole, adored for their work, and often enjoy success as a side-effect. Google was originally a PhD research project, motivated by passion and interest rather than profit. Dollar signs were not the focus of Zuck’s attention when he frantically wrote the first incarnation of Facebook, nor as Jack Dorsey brainstormed how Twitter would start.

The best bands are made of those who care about their music, not their profits. The best startups are made by those who care about their work, not their profits.

*EDIT: A few good commenters on Hacker News have reminded me that European classical music (i.e.: Mozart and the like) was most certainly for-hire. My point here was merely to highlight the commodity of modern, hired pop music, and I neglected to think about any time period other than our own.

These photos aren’t good enough, or meaningful enough to anyone else, to post on Flickr. 500px would scoff at them. The few people on Facebook that would recognize the people, places and events in the photos wouldn’t see the point. They’re tiny fragments of my life, and that’s about it.

Instead of forcing these thousands of photos to stay hidden in my iPhoto library, I found an outlet for them - my homepage. Crudely modelled after the stellar TED.com landing page, it’s supplied by a random set of hundreds of images, all of which I’ve taken, and until now, hand-cropped and hand-selected.

Michael Macias, in a submission to a Codebrawl last November, came up with a brilliantly simple method of content-aware image cropping. By measuring the greyscale entropy of a window as it slides over an image, the highest-interest thumbnail can be determined automatically. I took this solution, modified it (faster, uses ImageMagick, etc.), and hacked together a quick Ruby script.

This script automatically chooses 50 random images from a given path (or shell glob) and crops them to their most “interesting” thumbnails. The thumbnails are scaled to size, and saved in incrementing order in the destination folder. It’s highly optimized for my personal workflow, but it does seem to work quite well. For example, take the following shot of Zameer Manji:

The original photo was poorly exposed, had no clear subject, and was, well, weird. After automatically cropping it down to a tiny thumbnail, it fits in nicely on my homepage as an artsy shot of a bike rack in the daylight.

Only one thing left to do: take more photos.

Few people correlate software engineering with art.

There are those that will argue that “software itself is a form of art,” or that “this code is beautiful.” There are certainly pieces of software, written in different languages, that could be considered their own distinct forms of “poetry.” (And no, I’m not just talking about Lisp poetry.) Elegance, cleverness, and the functionality of the code all contribute to this sense of inherent artistry.

I prefer to write code that is outwardly visible as art. Code that you need to run, not read, to appreciate.

This is the schedule view of a radio station’s website. (CFRU 93.3fm at the University of Guelph, Ontario, to be exact.) I did not design this site - that was done by the wonderful folks at Studio Function. However, I did have the pleasure of implementing the design and creating the website itself during my last work term at The Working Group.

Although there is a fair amount of complexity behind this site, the part that was most enjoyable to implement was this schedule view. It helps that it’s beautiful and eye-catching, but writing code to make this design functional was extremely satisfying. Even more satisfying was the ability to see someone use the site, enjoy it, and being able to say “Yes, I helped make that.”

This site, via its design and partially through its functionality, is a form of art. I’ve spent my first three work terms (one year in total) working at web development shops on client projects, implementing (and sometimes designing) beautiful software that can be appreciated by almost anybody. I had a great time doing that, and enjoyed nearly every minute.

This is the Wub Machine, my online music remixer. If you know me, or if you read this blog, I’m sure you’ve heard enough about it so far. One thing I haven’t talked about yet is the art behind it.

I initially created the Wub Machine as an experiment in computer-generated music. If I were an arts student (or even a grad student in some software programs), it would have made a great thesis project to explore computer-generated art. While the technology used to power it is stunningly awesome, and the site itself is somewhat complex, that’s not the purpose of it. (Although, I did learn a lot.)

The average user of the site is not a technophile. They could care less about the software. However, the average user can definitely appreciate the product - a piece of music (ahem, mostly) that is not only listenable, but danceable and entertaining. Many would call it art.

(Hopefully.)

This is a screenshot from Dead Rising 2, an awesome action-adventure game released a couple years ago by Capcom. Yes, those are zombies, and that’s the main character (Chuck Greene) using a modified yard tool to mow them down. It’s a great game, with a great story, great gameplay, visuals, music, and the like. Save for some vocal critics, most people would consider this art.

Visual artists modeled Chuck Greene’s character. Writers crafted the brilliant story. Software engineers put it all together, and made the entirely immersive experience possible. Their work, while technical and complex, is just as much art as the models, textures, sounds and words in the game. It doesn’t just allow users to interact with art; it forms the fundamental experience that is enjoyed and appreciated.

Using software to make immersive, beautiful, artistic experiences that can be appreciated by anybody is awesome.

TL;DR: Software can be art, in many ways. That’s what I like to make.

Earlier this month, I put together a quick site called lndr.me, which tracks the usage of laundry machines at VeloCity, my student residence at the University of Waterloo. It’s simple and email-driven. Residents can email washer@lndr.me to say that they’re using a washer, and they’ll get an email back in ~30 minutes to remind them that their clothes are done. Other residents can also check the site and see if the machines are occupied.

It’s an exceedingly simple idea, with very little code required on the backend. (It’s a Rails app with ~300 lines of ruby.) I’ve even made an API to allow other residents to make apps out of it, or link in hardware sensors with Arduinos and ethernet shields.

However, before I even had the idea fleshed out, or the implementation decided on, I did a mockup. I opened Photoshop, drew some icons, found a simple colour scheme, searched for a viable domain name, and scribbled a UX flow into my Moleskine before ever typing rails new app.

This simple (some call it cute) design was my starting point. I added some things along the way - animation on the waves in the washing machine to show it’s running, or a slightly-shaking dryer icon to show the same - but most of the product was finished before I started writing code. I essentially started from the user’s perspective and then built inwards.

Now, some people will surely think this is obvious. “Of course you wait for designs first before starting implementation, that’s just obvious!” you yell. In the client-and-project-driven world of software contracting, that’s absolutely true. Specs must be finalized, and designs (or at least mockups) finished before the product is built.

A lot of other people, though, are confused by this. “It’s only a side project, who cares how it looks?” you might say. Or “I’m not a designer, I’m a coder.” I’ve heard both of those far too often to dismiss.

Your product’s user experience is just as important as what it does. Most apps do things that are marginally useful - track laundry, wake you up in the morning, play music, or give you directions. Would you use a music player that required a screwdriver to change songs? What about a map that gave directions in a series of JSON-encoded latitude and longitude coordinates, to then be decoded by the user? Of course not.

Products are successful, useful, and a joy to use if they have great user experience. A lot of hackers and coders nowadays don’t realize how important this is.

Let me give another example:

Ninjaquote is a site created by Scott Greenlay, Jinny Kim and myself in 24 hours (21:15, to be exact) during the recent Facebook hackathon at the University of Waterloo. Its goal is simple: it takes two of your Facebook friends, and finds something one of them said in the past, and quizzes you on it. The game is exceedingly simple, and has another dead-simple user experience.

1. Click to authorize the app to view your Facebook account.
2. Receive quote.
3. Click answer.
4. See if you were correct.
5. Goto step 2.

This simple UX, coupled with a good domain name and great mascot, makes the site a pleasure to use. So simple to use, in fact, that it won the hackathon.

This confused me at first. Other entries were far more technically complex - Hachi was an in-browser collaborative code editor built in Node.js and Socket.IO. FriendMozaic did some image processing to make your profile picture a mosaic of friends’ pictures. PrivacyVeil used some crazy OpenCV processing to detect faces behind you while you work, and pop up an Excel spreadsheet to cover your Reddit browsing.

Our winning entry was effectively ~1000 lines of Javascript, CSS3 and HTML5. Nothing fancy, nothing new - just a working, effective, and addictive user experience. Having the minimum number of features wasn’t a hinderance, as we had design to make the site appealing anyways.

tl;dr: Find the middle ground between form and function. It’s much more valuable than either extreme.

That intersection, which some call Toronto’s equivalent of Times Square, has a large number of street preachers. Loud, startling, obnoxious people that yell warnings of doom or urge repentance. Silly people.

I decided to use Twitter’s real-time streaming API to make an extremely specific location-based Twitter bot. The purpose? To respond to you if you tweet near the street preachers at Yonge & Dundas, with similar messages. Call it art, or a statement about society, or making fun of those preachers, whatever - I call it a fun technical and social experiment.

Using an excellent ArsTechnica article as a guide, I created a quick Python script that watches the Twitter stream for a given area, and replies to tweets in a very specific location. (±10 meters or so, by my guess.) If you’re one of the lucky few to tweet within those bounds, you’ll get a reply from @yonge_dundas:

A day later, I decided to clean up the script (rewrite it in Ruby, too) and open-source it. Well, here it is, in a quick Github gist:

Feel free to fork it, repurpose it, and do whatever! (Just keep my name at the top, if you please.)

Since then, I’ve polished up a completely new framework for the Wub Machine - nearly everything about the site has been rewritten since its first release. The first version was held together with duct tape, PHP and prayers, which resulted in some catastrophic failures when the site was initially launched. I’ve sinced rebuilt it in 100% Python, load tested, and added features.

Instead of talking about the code (which I do over on GitHub), I have a better story - being featured on the immensely popular VSauce channel on YouTube. I got a seven-second mention (and the thumbnail of the video!) and on Tuesday night, when the video first went up, all hell broke loose. My little Prgmr server couldn’t keep up with the 15,000 visits in 3 hours, and the load has kept up steadily ever since.more

A day later, I’ve flirted with Amazon EC2 and other hosting solutions, migrated databases back and forth, jumped up and down and watched live Google Analytics for far too long. I moved the site to a new host (not Linode, although they’re awesome) and finally, it can handle the load.

So, what lessons have I learned from this surge in Wub Machine usage?

• Be prepared to scale on-demand. I should have built the site on some sort of scalable, EC2 or Heroku-based architecture that I can instantly increase capacity with. As it is now, I had to scramble to find a faster web host that I could deploy onto, wait for DNS to switch over, clone the database, etc. Ideally, I could have integrated the site with EC2 to detect surges in popularity and spin up a new worker instance. Alas, that costs money and I’m cheap.

  • Know your options for scaling, too. I didn’t realize I could have spun up instances running the same code, and just proxied with a cookie via one master Nginx machine. Either way, there’s a number of ways I could have scaled up, and I didn’t consider some of the most popular options, only because I hadn’t heard of them.

• You never know who your next audience will be. I had previously prepared the site for Reddit, HN, Evolver.fm, etc: sites with a more tech-savvy audience. YouTube, on the other hand, is one of the most accessible sites on the internet. My latest surge in users has come from what seems to be teenagers and your average, casual YouTube user. (the kind with a four-digit number on the end of their username.)

  • Why is that important? Well… Reddit, Hacker News and other similarly technical audiences respond badly to advertising. I had left off ads from the site for a number of reasons, one being they wouldn’t be very effective. As soon as I saw YouTube flocking to the site… I figured they’d be slightly more interested. And so far, they have been. Very much so. I think I’ll leave the ads in place. (The site can finally pay for itself!)

• Know how to jump ship. In the past couple hours, as I’ve been trying to make the site faster, I’ve literally set up the site on 3 different servers. It saved me a ton of time to have an installer script that (mostly) worked. All I need to do now is make that automatic, and I can do something like a Capistrano deploy to add capacity.

  • …and know how to take your data with you! I forgot about my DB at one point, which caused a couple conflicts, and now my statistics are missing about 100 remixes. Not a big deal, but still - have some plan in place to flip databases over instead of copying them. Or something.

• People will get fed up and leave. I have a handy statistics page now that shows me (in real-time!) the activity on the site. I can see as people upload a track, how it progresses, if it fails, why it failed, etc. I can also see clearly when people upload a track, don’t want to wait, and close their browser. Although that can sometimes happen due to an overloaded server, more than half of the people who visited during this spike turned away after seeing that they’d have to wait. Perhaps something to try to avoid, but that ties in to my next point…

• Care about your users… just enough. That sounds incredibly callous, so I need to clarify myself here. Any website owner should absolutely care about their users. User experience is, in my books, the most important thing to work on. However, if your app goes viral and/or reaches a very wide, very diverse audience, you have to balance that care for UX with your own needs and sanity. I’ve recieved a handful of emails so far from ordinary people, demanding that I get the site running faster, or that I add feature X and feature Y. Not suggestions, demands!

  • If you can provide a working app, and do your best to keep it working and user-friendly, then you’ll still never hit 100% in all of your metrics. There will always be people who have a bad experience on the site, if only because of their own personal situation. (i.e.: browser, connection, or internet literacy) I usually try to go the extra mile and make what I’m working on close to perfect, but at some point, it becomes futile.

So, what’s next for the Wub Machine? I’m not sure. I still consider the algorithm a bit of a hack, although it sounds distinctive and kinda cool now. I’m planning a YouTube remixer, but that’s technically challenging. There are some small technical problems I can try to fix as well, and I can continue to improve the code base and open source the changes. And of course, I need to keep it running. Otherwise… statistics!

Wub Machine Statistics (as of November 10, 2011):

• 637,568 Facebook story impressions
• 208,792 total site pageviews
• 64,673 total unique visitors
• 29,801 total songs uploaded
• 1,956 hours of music remixed (82 days worth!)
• 85% of tracks remixed successfully
• 1,143 remixes shared to SoundCloud
• Most commonly-remixed artists:
  1. The Beatles
  2. Daft Punk
  3. Adele
  4. Deadmau5
  5. Skrillex
  6. Gorillaz
  7. Blink-182
  8. Led Zeppelin
  9. Radiohead
  10. Deerhunter (oddly, only the song ”Helicopter”)

1. Uploads (whenever a song was uploaded for remixing)
2. Processing (started/finished/failed) (the analysis & rendering of the remix)
3. Sharing (sharing of a remix to SoundCloud)
4. Downloads (when a user explicitly downloads their remix)

All of these actions are graphed separately, to provide a detailed look at what happened over the first week of running the Wub Machine.

So, what did I learn?

1. Don’t pretend to have capacity.

   • I didn’t expect such massive server load, and as such, I assumed there would be no harm in allowing people to upload tracks when other tracks were being remixed. This ended up creating a never-ending queue of songs, and prevented anybody from effectively hearing a remix for what would have been a 16 hour wait. Those who did upload songs for remixing had to wait exorbitant amounts of time for their remixes to finish, and the service effectively ground to a halt. Instead, I should have implemented a system that forces users to wait until capacity is available. (And I did, afterwards. Currently, you can’t upload a track if the site is currently working on one.)
   • I did make an emergency fix, though: the giant red spike you see on the graph above is the queue of 900 songs, all from eager Redditors, being cleared at once. The queue would have taken nearly 16 hours to process, and effectively caused the site to grind to a halt while still on the front page of /r/Music. A significant portion of the songs uploaded had been abandoned, and wouldn’t have been heard, but were still stuck in the processing queue. Needless to say, the entire system could have used more load testing.

2. Test, test, and load test before pushing to production.

   • I had been testing on staging for about a week, on a low-powered server, with at most 5 songs processing at once. Had I tested the site in staging with heavier load, and accounted for very unexpected amounts of traffic, I would have been much better prepared for the initial spike.
   • I had prototyped the site using SQLite and with client-side polling to deliver progress updates, as it was simplest to develop. As soon as the front page of the Wub Machine was getting 40 pageviews/sec, SQLite crumbled in production, and PHP started using a ridiculous amount of resources due to the constant polling. It wasn’t until many days later that I had time to migrate the database to MySQL and patch the progress indicator to use long polling.
   • I had planned to spend the Monday testing the site with load from Twitter, before posting it to Reddit. Someone decided to post to Reddit before I had anticipated, and I wasn’t prepared to make the required changes. A lot of songs failed processing, a lot of bandwidth was used needlessly, and a lot of exposure was wasted with a site too busy to remix songs.

3. Expect users to abuse features.

   • I built the site with the ability to link to individual remixes. I provided a disclaimer letting people know the links would die after an hour, figuring that anybody who wanted to share a remix would post it to SoundCloud, or at least download it. Not so. Hundreds of people started sending around links, essentially turning the Wub Machine into a very bandwidth-heavy temporary remix-sharing site. Had I removed this feature from the start, I would have caused a minor inconvenience to approximately 5% of users, while saving me a ton of bandwidth and hassle.
   • Later in the week, when 4chan’s /mu/ discovered the site, I counted a couple hundred links to individual Wub Machine remixes in each threads. Obviously, 4chan is a more anonymous site, and very few people wanted to share via SoundCloud. I tried to lessen the impact of the links by forcing each remix to expire within 15 minutes, instead of 1 hour, which helped slightly. As soon as the traffic died down and the site dropped off of /mu/, I immediately disabled the link feature to save me bandwidth. (Had I disabled the feature while it was popular, I risked a backlash from 4chan… not something anybody wants. Ever.)

4. Buy more capacity than necessary.

   • I expected the Wub Machine to use a significant amount of bandwidth and tons of server load, but I could have still used more to deal with the unexpected spikes. I started by buying a Linode 512 to host the site, which worked for a while, then started to choke once it hit Reddit. That was quickly upgraded to a Linode 1024, which worked wonders for a while. 4chan drove a lot more traffic than expected, and after a week, I had used up 220GB of bandwidth.
   • After the initial traffic spike and exposure, I shut down my Linode, moved the site back to my Prgmr, and put an hourly limit on the number of songs that can be remixed. I’ll probably still need to upgrade my server or increase its bandwidth next month. Who would have guessed that creating a cool web app costs money to keep running?
   • I also realized before even launching that making any money was a very slippery slope. I could have added Adsense to the site, and probably would have been able to recoup my server costs. However, I expected that if the site was ever seen by any record companies, making even one cent off of having people upload their music could be a massive legal liability. (Even though I delete uploads as soon as they’re done remixing and don’t keep remixes for more than 15 minutes, that wouldn’t stop people from taking issue.) Besides, as long as I limit the capacity and keep the site on the VPS I already own, it costs me very little to run.

So, there you have it. Final first-week stats: 15,793 uniques, 46,062 pageviews, 1,950 Facebook likes, 273,640 story impressions on Facebook, 139 tweets and 8,808 songs remixed.

The Wub Machine’s not dead, but it’s slower, and during the first week, I learned a lot more than I’d ever expected to learn about scaling web services. Having a site go viral was quite an unexpected, thrilling, and crazy experience.

moreI opened up my trusty Photoshop, cranked out some multicoloured waves and set “The Wub Machine” in beautiful Proxima Nova. Then I set about the immense task of actually implementing the remixer on the web.

I’d go into the technical impressiveness of the system, and how it’s brilliant and took me months to come up with… but it’s really not. It’s one big hack.

I ended up using:

• PHP to serve the front-end, as well as serve the AJAX progress updates and interface with SoundCloud
• Python to power and tie together all of the processing on the back-end
• the Echo Nest Remix API to do the heavy lifting, audio analysis and beat detection
• FFMPEG to decode & encode the MP3s
• Mutagen and PIL to rewrite the MP3’s metadata, extract artwork, overlay a graphic and put it back in to the final MP3
• Beanstalkd to queue processing jobs and connect PHP to Python
• SQLite3 for logging and some queue intelligence
• HTML5 Audio, used for a beautiful HTML5 player (taken from the extremely impressive Neutron Creations blog)
• Flash for the fallback player on older browsers
• Javascript and jQuery to hold together the very rickety frontend
• CSS3 animations, for the moving waves at the top of the page
• the SoundCloud API for sharing tracks (without putting me at risk of nasty legal issues or pushing storage constraints)

I did have to make a couple changes to the original algorithm, though:

• I realized that audio volume is a nonlinear curve, so I had to account for that and create a new mixing algorithm. The volume of the original track vs. the wubwubs is now almost always about 50%.
• I went back into my dubstep template in Logic Pro and added different types of TransitionFX samples to the intro and the wubs - booms, splashes and such. Although I’m still not happy with certain parts of the template, it’ll have to do for now. I’m not a dubstep producer - I’m a rock/metal/electronica/jazz guy. (for now!)
• I made the algorithm as deterministic as possible. The remixer is essentially a function (depending on the analysis I get back from the Echo Nest) so if you put in the same song, you should get the exact same remix.
• I improved the loudness calculation algorithms, fixed some stupidly-inefficient bugs, killed off a statistically-improbable-but-still-possible infinite loop, added logging, error handling, and progress indicators.

Since my blog post about the initial hack, it’s taken me 3 weeks to assemble this web front end. That said, there’s probably still tons of bugs - it only accepts MP3s at the moment, and it’s probably somewhat unstable. If I push it too hard, or post it to Reddit or Hacker News, my shiny new Linode will probably spontaneously combust. Be gentle!

Go try out the Wub Machine, share your tracks on SoundCloud, and enjoy!

I like dubstep.

There, I said it!

That massive bassline, two-step beat and killer rhythm has some odd allure that I can’t resist - and I’m typically a fan of rock, metal and prog!

I’m also a huge fan of the Echo Nest and their brilliant Remix API. In their words, the Remix API is an “internet synthesizer” - quite true. I can send off an mp3, and get back extremely detailed beat, timbre and pitch information within seconds. Some people have already used this to make any song swing, put a donk on any song, and much, much more.

For the first SE Hack Day, I decided to use the Remix API to automagically add dubstep to any song.

Now, as you may be able to tell from the samples above, this isn’t quite ready yet. In fact, it’s extremely rough. Even the code looks horrifically ugly. (And yes, you can download it, fork it, and edit it freely - it’s open sourced on GitHub.) I have a lot of work left to make the results sound passable.

Technically, I’m not really doing anything too complicated:

• I used Logic Pro and Native Instruments’ new Reaktor synth to make some dirty, dirty wub basslines at the proper dubstep tempo (140 bpm) and mangled my own kick and snare samples. I then rendered 8 bars of this pattern, in two different variants, in every key of one octave.
• Using Python and the Echo Nest Remix API, I get an analysis of each track’s bars, beats, pitches, timbres and more. I still need to make some better use of this information, as right now, a lot of songs end up being detected and used improperly. (an 8th or 16th note off, ruining the beat)
• Using the Echo Nest Remix API’s Dirac time-stretching abilities, I take the input song, bar-by-bar, and alter the tempo to be exactly 140bpm. Then, for each “section” of the song (as defined by, again, the API’s analysis) I do 16 bars of dubstep with a repeating 8-bar pattern of either beats, bars or tatums (notes) from the original song.
• For the dubstep backing, I take whatever key the API tells me the song is in, and just choose the corresponding backing file from the ones I’ve prerendered.
• For the intro, I have a pre-rendered intro with some noise sweeps. I then use a bit of brute-force audio manipulation to play a build-up pattern before the initial “drop” after 8 bars.

I’m taking a ton of suggestions on how to improve the script - adding variables for time till drop, allowing overrides if the API mis-identifies the key or tempo, and allowing a different choice of beat, bar or tatum for sampling the original song. There’s obviously tons of work left to do, and I plan to improve it whenever I get the chance.

Let me know what you think below in the comments, leave some suggestions, and check out the code on GitHub if you’re interested!

UPDATE: I built a web interface and a very nice-looking site around the basic algorithm. Go and remix your own tracks!

P.S: When I said I like dubstep, I meant I really do like dubstep. When I’m not making my own music, I also enjoy drumming to dubstep:

I don’t necessarily release a lot of it (or finish all of it) but I have at least 100 songs I consider developed enough to listen to, and about 300 other song files that are just riffs, beats and vocal ideas floating alone.

With so many files, projects, songs, sounds, and work in one place, I’ve developed musical workflow that borrows a lot from software development patterns. My music tends to be produced in stages and cycles.

I’ll usually start with an idea from noodling around on some instruments, then do two things:

1. Record (track) instruments with the main riffs of the song or basic chord structure.
2. Record a basic drum loop and bass loop, then arrange to fit song structure.

Then, I don’t do anything.

For any length of time between a day and a month, I usually don’t touch the song-in-progress. The song needs a break, while I can forget about whatever ideas I had while writing the initial sections of the song.

Then, I’ll come back to it, add some more riffs, ideas, melody, direction, possibly a scratch vocal track. Then the real fun begins.

1. Re-record any parts with mistakes or even rhythm that’s slightly off.
2. Add whatever new sections, parts, instruments, riffs, or notes that need to be added.
3. Make updates, edits and changes written down from previous listens.
4. Bounce to disk.
5. Place .flac in Dropbox for remote listening and burn to CD for in-car auditioning.
6. Make comments on SoundCloud/Evernote on each track.
7. See step 1.

This process continues until I find it extremely hard to find changes to make to a track, or until I’ve reached the self-imposed deadline and am happy with the track.

In an ideal world, I wouldn’t need to burn CDs for the car - I could plug in my phone and have it stream the latest version of the song from SoundCloud with no user intervention. (very similar to “nightlies” in the open-source programming world.) I could listen at work, make comments (that would get synced back into Logic), listen on my phone, make comments, send the links to others, get their feedback, and have this all happen with one utility.

(just to round out the post with some audio, here’s one of my SoundCloud tracks.)

Unfortunately, if I continue writing this script at the moment, I won’t have any time to finish the album the tool would be created for. Once I’ve released my next album, (ETA: May 2011) I’ll be open-sourcing some cool tools to make such a workflow easy.

A lot of people wouldn’t expect that posting a YouTube comment like this:

…would cause the device in my pocket (or on my nightstand) to vibrate and alert me instantly. This wasn’t even something I had to go out of my way to set up - YouTube’s default email notification settings accomplished this.

Responding to a tweet, commenting on Facebook, or any other number of nearly-instinctive online actions people do nowadays all cause unexpected side-effects: vibrating phones. It’s not a bad thing, nor is it even that annoying. (yet) Just somewhat mind-blowing that the click of a mouse on one side of the world will (near-)instantly cause a device in my pocket to vibrate and alert me.

And I still think the world could and should be more connected.