Tuesday, October 31, 2006



The Guardian recently published an article by Steven Levy about the non randomness of the ipod shuffle function. This article concludes that,
  1. true randomness is hard to achieve and
  2. humans perceive patterns where there are none.
I wonder if a paradoxical interpretation of these facts may better explain what is happening. That is to say
  1. randomness is difficult to achieve because the universe is not random and
  2. as part of the universe, human beings are attuned to these patterns. This is in fact why there is such a thing as meaningful coincidence. Wasn't this what Jung was talking about when he advanced the concept of synchronicity?

Like levy and everyone else apparently, my ipod tends to favour certain songs over others. The trick is to delete those songs when they get too boring and thus disrupt the pattern of meaningful coincidence. This got me wondering if the ipod analogy could be extended to apply to life. In life, certain pattern clusters predominate, for example a person might have a run of good or bad luck. The wisdom of ipod suggests to me that, if it is the latter, a person should deliberately mess with the patterns of their lives (i.e. change their habits) and this would produce a different pattern configuration. Changing a habit would be the life-equivalent of deleting the oft recurring tune on the ipod. Thus, a whole self help philosophy emerges from the latest must-have consumer durable.

Here is an edit of the Levy article.


My first iPod loved Steely Dan. So do I. But not as much as my iPod did. By 2003, among the 3,000 or so songs in my iTunes library, I had about 50 Steely Dan tracks. Yet every time I shuffled my music collection "randomly" to mix the tunes, it seemed that the Dan was weirdly over-represented. Meanwhile, it began to dawn on me that there were songs, and even artists, that my iPod had taken a dislike to, if not a formal boycott. Where was Van Morrison? His work was in abundance in my iTunes library, but in my iPod's marathon rock fest, the Belfast Cowboy was perpetually waiting in the wings.

I made it a point to ask iPod owners if their beloved little units were judicious in distributing the songs among various artists or whether they played favourites. People would generally respond with a sigh of relief. Yes! Someone else has noticed! From the results of this admittedly nonscientific survey, it appeared that nearly everybody's iPod seemed to have a favourite artist, or two, or three. After I wrote about [this] in Newsweek, my inbox was flooded with emails - iPod owners were taking serious note of what happens in shuffle, and virtually all of them seemed to think that something funny was happening.

Apple insists that there is no computational flaw in its execution. "It is completely random. It is absolutely, unequivocally random," says Jeff Robbin, one of the original authors of iTunes and later head of the iTunes development team.

Robbin is talking randomness in terms that software can reasonably produce, which is not perfect randomness. Let's look at the way shuffle works. First of all, note what it doesn't do - it's not like mixing all the songs in the equivalent of a big bucket of lottery balls and picking out the next one. Instead…it shuffles the entire library so as to reorder them, just as a blackjack dealer shuffles a deck of cards. If you listen to the entire library all through, you will hear every song once and once only.

True randomness, it turns out, is very difficult to produce. This subject was most famously examined by Claude Shannon, arguably the Father of Randomness. Basically, he defined randomness as a question of unpredictability. If a series of numbers is truly random, you have no possible way of guessing what comes next. If something isn't random (as in the case of what letter might follow another in a message written in English), you have a better chance of figuring out what comes next. That's why it's so crucial to remove the natural redundancy of language from an encoded message and make the coded text look random.

But perfect randomness is an elusive ideal. For instance, if you're flipping a coin, a minuscule weight imbalance might, over the course of millions of tosses, make heads come up slightly more than tails. And if you're randomising on a computer, you have to introduce a "seed", which is a starting point for the algorithm that mixes up the selections. The seed must draw on some unpredictable input of time that begins outside the computer. Otherwise, the results would be the same over and over again.

Paul Kocher, CEO of Cryptography Research concludes that Apple's claims of a high degree of randomness are almost certainly valid. Another expert I consulted, John Allen Paulos, a Temple University mathematician, agreed. He wasn't surprised, though, that iPod users were questioning whether the shuffle was random. "We often interpret and impose patterns on events that are random," he says. "Especially with something like songs. Songs evoke emotion, and some stick in our minds more than others."

Steven D Levitt, the self-described "rogue economist" who co-wrote the bestselling Freakonomics, also fell into the trap. Writing on his blog, he professed constant surprise at how often his iPod shuffle "plays two, three or even four songs by the same artist, even though I have songs by dozens of different artists on it". But as a statistics maven, Levitt understood that the bottom line is that "the human mind does badly with randomness".

Indeed, says Kocher, "Our brains aren't wired to understand randomness - there's even a huge industry that takes advantage of people's inability to deal with random distributions. It's called gambling."

So why does Autofill produce nine Springsteen songs out of 188? Because that is what almost always happens in normal distributions of items from databases. Clusters of something are to be expected. Here's a classic maths trick: gather 40 people in a room and have everyone write down the day he or she was born. What are the odds that two people will have the same birthday? Nearly 100%. It sounds like a coincidence, but mathematicians will tell you it's much more unusual for there to be no such clusters.

We perceive trends when there are none. Poker players invariably believe they can lock into streaks. Backgammon champions swear that dice can go hot or cold. Likewise, people think they can cosmically predict what song will come next on their shuffle.

he seemingly magical effects of the shuffle function - a spooky just-rightness, even brilliance, that comes from great song juxtapositions - were also consequences of randomness. And, in its own way, that was much more disturbing.

Sunday, October 01, 2006

[[ZEITGEIST]] Cosmos
I have just discovered that the entire series of Carl Sagan's seminal Cosmos programme are available on google video. Sagan co-wrote and presented the series in 1979, billed as a "personal vision," it covers an ecletic range of subjects from tours of the known and speculative universe, to a brilliantly staged explication of the 'theory' of evolution, which Sagan confidently proclaims is, "not a theory, but fact."

I remember seeing some of the episodes of Cosmos on the BBC around the time when it was first produced. Then its thirteen episodes were cut down and broadcast as 50 minute segments and shown late in the evening. Why the BBC did this I have no idea. The series, although inspired apparently by home-grown documentaries like David Attenborough's Life on Earth, surpasses even these impressive achievements in my opinion. I am astonished revisiting the series again how much of my own speculative day dreams and musings were in fact inspired by the speculations of Sagan. His lamentation of the loss of the Library of Alexandra in the first episode, for example, seared into my consciousness and later manifested in all sorts of ideas which I have speculated upon and been obsessed with over the years. Later in my life, I came to admire Sagan's writing, both his non-fiction works and the novel Contact, which was later made into a cowardly movie by Robert Zemeckis, starring Jodi Foster. Cowardly because Zemeckis did not attempt the books jaw droppingly astonishing ending.

Sagan is of course a superlative teacher, his explanations of even the most gnomic and difficult theories have a clarity and an entertainment value which has not been surpassed. This series has several some heart stopping moments of astonishment, the majority of these are generated, not by the special effects, groundbreaking for the time, but which now appear a little ropey, but by Sagan's conviction and the strenght of the writing . I stongly recommend if you have a broadband connection and thirteen plus hours to kill to tune into google video and watch the show.