Mappiness development

In the previous post I wrote about the chart for analysing Mappiness app data I made. In this post I’ll write about the process of making it. It’s like a director’s commentary!

As I wrote, I started this as another way to get better at using D3.js but it turned out that the bulk of the work was nothing to do with D3. I don’t know how long I spent on the whole thing, from initial idea to launch, but I’m guessing it would add up to maybe six full-time weeks? Eight? Wild guesses. But, looking at the thing it seems like way too long.


Near the beginning I remember thinking, “Ah, I see: when doing some kind of data visualisation, a large chunk of the work is in preparing the data, rather than working on the display of it.” That thought made sense early on because, even though the data was in decent shape I did spent what seemed like a long time working out what to do with it and how best to use it with D3.

But from this end of the project, with it finished, that data-wrangling seems like a small proportion of the overall project. Even displaying the data on a chart doesn’t seem like a major part of the work. Designing and developing everything around the chart — the descriptions of the lines beneath, the controls to show/delete/edit/duplicate them, and the editing process — must have taken at least half of the time. Which is annoying. Given I was aiming to improve my D3 skills, I’ve spent an awful lot of time laying out forms, processing submitted data, updating HTML, tweaking the interface design, etc.


On the plus side, I am definitely more comfortable with D3 than I was last time around. Admittedly, another line chart isn’t stretching my skills hugely after the previous one, but there were a number of tricky (for me) issues I managed to figure out without resorting to asking on Stack Overflow. This is definitely progress.

One of the nicest things about the chart is the small “brush” chart that lets you select a focus area for the large chart. I can’t claim much credit for this because it’s a built-in D3 component, but it does highlight one of the things I find odd about D3. If I’d wanted to build something like that from scratch I’d barely know where to begin. But because the component existed I could follow the example, read a couple of blog posts, tweak a few things, and there we go! There’s a big gap between being able to adapt an existing example and creating an element, or an entire form of visualisation, from scratch.

Similarly, because examples and tutorials are usually simple in order to explain a single idea they rarely take account of issues that crop up in real world usage. Even if you’re able to adapt an existing example, working out how to handle edge cases is another matter.

For example, although I got the brush to work fine initially, changing the data displayed (i.e., if the user adds or removes a line, or edits the criteria used to generate one) caused problems: the top chart would ignore the selected focus and redraw the whole chart; and if the new data changed the range of the x-axis, the selected area would no longer be in the right place or the selected area might move off the end of the small chart. All of this was solvable but there’s a huge difference between being able to adapt a D3 example and understanding enough to handle your real world usage. I’m getting better at the latter — I don’t feel entirely lost — but I have a long way to go.


It seems crazy to me that I spent so long on this thing that is going to be of use to a very small number of people. If any. If it was purely for my own use I could have made something that was Good Enough much, much quicker. But there’s something satisfying, to me, about trying to make something of a higher quality, that’s more presentable and useful. (This is the same reason I spent way, way too long on the unused Twelescreen.)

I’m always frustrated with how long graphic/UI design (as opposed to technical/code design) takes, and it’s no different if you’re doing the design with code rather than, say, Photoshop. For example, figuring out exactly how the show/delete/edit/duplicate controls should function and present themselves, and managing that nicely in the code, took many iterations and corrections beyond the point at which I had some rough, uglier controls “working”. But that polish, working out the UI logic (both visible and behind-the-scenes), is enormously satisfying, even while it’s frustratingly time-consuming.

And then there are entirely unplanned chunks of work that only seem necessary as the project goes on. It was only when I’d nearly finished that I realised only Mappiness users would be able to even see this thing I’d spent weeks on. So I decided to provide an option to have a go with random data. This ended up being a couple of days fiddling with code to generate data that, while random, was mostly believable. It’s a persona, a fake individual who is less happy at work than at home, who enjoys being with their partner and kids, who is a little more tired early and late in the day, who eats at meal times… this unexpected bit of work was good fun, a nice thing to crop up near the end of the project.


But there were many points where I wondered, “Why on Earth am I plodding on with this very work-like project when I could be sitting outside in the sun?” But I concluded that this is what many hobbies are like. A man in his shed making a cathedral out of matchsticks isn’t solving any important problems, and only a few people will ever see the thing, but while the process is sometimes tedious, overall it’s satisfying.

Similarly, I enjoy the doing of programming, improving skills, doing a good job, making a thing, even if it’s of little actual use. If this had been a client project it would have been finished more quickly, with more compromises. But with no client, no budget, and no deadlines, I could take as long as I wanted and get pleasure from the process of deciding how best to do everything, to keep trying things until I was satisfied, and to simply enjoy the process.

Having said that, this was very much a “work” project as opposed to a “fun” one. The initial aim was to improve my D3 skills and I only worked on it during conventional working hours. There are other projects I’d prefer to be working on which somehow seem “less worky” and so I feel guilty spending time on them unless it’s evenings and weekends.

I’m not sure I’ve got this right. I don’t subscribe to the “Just do what you love!” mantra as a recipe for life or career success but I sometimes wonder if I’ve got the balance wrong. Sometimes there’s too much time-consuming frustration compared to satisfaction. Maybe I’m trying too much? Maybe I need to know when to stop something sooner (or at all)? Maybe next time.

In Projects, Web Development on 24 July 2014. Permalink

Mappiness chart

Mappiness is a free iPhone app that helps you manually track your happiness, relaxedness and awakeness in different situations over time. I’ve made a tool that helps users examine their data more closely (or you can try it out with random data).

The Mappiness app itself shows a series of graphs of your data but these don’t allow for detailed analysis of the data. And, once you’ve been using the app for a long time, there’s too much data to see clearly on phone-sized charts. However, the app gives each user a code that allows them to download their data in a variety of formats from the web, so I decided, as another exercise in using D3.js, to make my own chart.

If you use Mappiness you can enter your personal code and, if you’re patient, your data will be downloaded and displayed. Alternatively, if you just want to see what the chart looks like, it can generate some semi-random data to play with. (The data isn’t entirely random, but tries to emulate a possibly-believable person.)

Screenshot of chart

You can duplicate lines and edit them, to change either what’s displayed (happiness, relaxedness or awakeness) or the criteria used to choose which points are shown. This is based on the data entered with each response in the app, and lets you filter by the people you were with, the kind of location (e.g., indoors, at work, etc.), the activities you were doing at the time, and any text notes added:

Screenshot of chart with two lines

To see more detail in the large chart, you can use the small “brush” or “scrubber” chart below to select a region to zoom into. Very useful when you’ve accumulated a lot of data; this chart shows a year’s worth of data, but I have nearly four.

Screenshot of zoomed in

The screen to edit a line’s criteria hopefully makes some sense to people who are familiar with Mappiness — all the terms used are identical to those in the app. But even so, as simple and clear as I tried to make it, I’m sure it’s a little baffling:

Screenshot of the edit screen

But still, this does provide for much, much more fine-grained analysis of one’s data, which is surely the point of “life-logging” tools like Mappiness. There’s no point collecting all this data without reflecting on it in aggregate. Am I more happy at work or home or elsewhere? When listening to music or not? Do I actually enjoy watching TV?

Of course, we have to be careful not to confuse cause and effect — maybe I only watch TV when I’m fed up, rather than TV making me fed up. And the fact that one of my most happy-making activities is apparently drinking alcohol is more a factor of the occasions in which I do it: when chatting with friends.

So, if you use Mappiness, or are just intrigued, give it a whirl.

I’ve also written more about the process of building this.

In Projects on 24 July 2014. Permalink

Famous exes updated

Thanks to some suggestions, I’ve added a few more songs to yesterday’s post about songs written from the point of view of someone whose once-girl/boyfriend is now famous. You can jump straight to the new songs here.

In Music on 2 July 2014. Permalink

Mini music genres: Famous exes

I like finding tiny musical genres: a few tracks that have something really specific in common. Not a broad theme like “Songs about London” but something more focused. So I thought I’d share them here, because you might have more suggestions for some of them. The first one is…

Famous exes

UPDATED with new songs, 2 July 2014

I’ve come across a few songs from the point of view of someone whose once-girl/boyfriend is now a famous star. It’s a nice twist on longing for a past love.

Fountains of Wayne — Hackensack (2003)

I used to know you when we were young
You were in all my dreams
We sat together in period one
Fridays at 8:15
Now I see your face in the strangest places
Movies and magazines
I saw you talkin’ to Christopher Walken
On my TV screen

I find this one a little mawkish, and some of those rhymes (talkin’ / Walken !) are way too neat. But so long as I can get past that, this is like a straight-up exemplar of the mini genre.

Thomas Dolby — Europa and the Pirate Twins (1981)

Nine years after, who’d I see
On the cover of a magazine?
Buy her singles and see all her films
Paste her pictures on my windowsill
But that’s not quite the same — It isn’t, is it?
Europa my old friend…

The video’s endearingly daft, but I love this one so much. British internationalist synth-pop retro-future romanticism. Oh, the longing.

Eleven years later, Thomas Dolby recorded a sequel to Europa, with an update to her story after the fall of the Iron Curtain…

Thomas Dolby — Eastern Bloc (1992)

And last night I swear I saw her face
As they stormed the gates of satellite TV (“Europa”)
Too bad I don’t get News At Ten
‘Cause the CNN would tell a different story

I find post-80s Thomas Dolby a bit odd, and sometimes can’t even decide if I like it or not. (Mostly: yes.) But I adore the idea of a song’s narrative being picked up a decade later, Before Sunrise/Sunset/Midnight style. Extra doomed-romanticism points for Mr Dolby.

Emmy the Great — Two Steps Forward (2009)

…the time’s already gone
When people were just people not the jobs that they perform,
Our songs were just a thing we did with melodies and chords,
Now you’re available in all good record stores.
But I knew you best, back when love was just a feeling
That ran out between my legs onto the back of my dress
Onto the clothes that I was wearing.

This is vaguer, murkier, and it’s not entirely clear what relationship these two had back then. I don’t know what to say; sometimes Emmy the Great stops me in my tracks.

The Human League — Don’t You Want Me (1981)

You were working as a waitress in a cocktail bar
When I met you
I picked you out, I shook you up
And turned you around
Turned you into someone new
Now five years later on you’ve got the world at your feet
Success has been so easy for you
But don’t forget it’s me who put you where you are now
And I can put you back down too.

I hesitated about whether this quite fits the mini genre. The song’s not specific about exactly how much, or even what kind, of success Phil Oakey’s ex has achieved. Maybe she simply runs her own cocktail bar now. Who knows? For the purposes of this exercise we’ll assume she’s now an internationally famous superstar. Either way, it’s clear she’s moved on, has left Phil behind, and he’s not too happy about it.

That was all I came up with myself, but within a day I had a bunch of new examples…

Reel Big Fish — She’s Famous Now (2013)

You won’t believe it but don’t you want to know
I let her go but I can’t let her go
I don’t wanna hear it

I’m singin’ too high tonight, I’m gonna lose my voice
I heard her on the radio don’t want to sing along
But I’ve got no choice

She used to be my girl but now she’s famous

He’s obviously not entirely pleased to have let this salad-eating, mumps-suffering woman go. Thanks to Alice Bartlett for this one.

The Beatles — Honey Pie (1968)

You became a legend
Of the silver screen
And now the thought of meeting you
Makes me weak in the knee

Oh, Honey Pie
You are driving me frantic
Sail across the Atlantic
To be where you belong
Honey Pie, come back to me

The oldest track here and a fine example of the mini genre. All rather cheerfully music hall, rather than letting silly un-British emotions get in the way. Thanks to me ol’ pal Ted Mills for that.

Peter Sarstedt — Where Do You Go To (My Lovely)? (1969)

I remember the back streets of Naples
Two children begging in rags
Both touched with a burning ambition
To shake off their lowly-borne tags, they try

So look into my face Marie-Claire
And remember just who you are
Then go and forget me forever
But I know you still bear
the scar, deep inside, yes you do

For most of this song it sounds, to me, like a stalker who knows way too much about a woman and wants “to look inside your head”. It’s only when it gets to the final verses, above, that the background becomes clear. Oddly, the Right Said Fred cover version from 2006, which is otherwise surprisingly faithful, omits these final, revealing, verses. Hmm.

Apparently the song’s not about anyone specific, or else Sarstedt either wrote it about a girl he fell in love with in Vienna who died in a hotel fire. Thanks to TV’s Dave Green for this sad number.

J. Geils Band — Centerfold (1981)

Years go by
I’m lookin’ through a girly magazine
And there’s my homeroom Angel
On the pages in between

My blood runs cold
My memory has just been sold
My Angel is the centerfold

“The real micro genre classic” as Dave Green (again) put it. This time, the singer, Peter Wolf, is both surprised, dismayed and turned on when he discovers the sweet girl he once dreamed of talking to at school is now famous.

There are a few other songs which, for various reasons, don’t quite fit into this mini genre.

The one that sprang to mind for me is The Model by Kraftwerk, although it’s only the final line of which (“Now she’s a big success, I want to meet her again”) that suggests maybe the singer once knew, or at least met, the woman.

Dave was dismissive about Sk8er Boi by Avril Lavigne, and rightly so… but mainly because it’s written from the wrong point of view — by the woman who ends up with the guy who becomes famous, singing about the girl who once dismissed him (“Sorry girl, but you missed out, well tough luck, that boy’s mine now.”)

And finally, another suggestion from Dave, You’re So Vain by Carly Simon, which would be a perfect fit except while the mystery man she’s singing about is successful (“You flew your Learjet up to Nova Scotia to see the total eclipse of the sun”) it’s not at all clear that he’s famous. And that’s what this mini genre is all about.

In Music on 1 July 2014. Permalink

A good morning on Twitter

Over breakfast I usually catch up on the overnight Twitter and read the (online) paper. This often isn’t a cheery way to start the day but it was a good Twitter morning today, so I thought I’d share.

At the end of one conversation Timo linked to this article about Richard Florida’s theory of the creative class, which is very interesting (as are Florida’s and Bures’ follow-ups).

Then some lighter, thoughtful, relief from Danny in fine form. (“this dates back from a socialist workers party festival when i was 10, seeing a cop and organizer jeering together at my clown facepaint / clown facepaint i had been obliged to adopt due to social pressure. I swore then enmity to the organized radical left and sulked rest of day”)

Jonty linked to this “semi-fictional” film from 1973 set in London’s Thamesmead estate (“featuring a young Veruca Salt no less”), which I’ve only watched the first few minutes of but love so far. (I find it interesting how there’s a particularly 1970s way of speaking in these things, much as people in, say, 1940s films have their own manner. The early 21st century is going to sound as equally quaint in not very long.)

A private correspondent linked to a Motor Trend video in a Citroen DS and a Tesla Model S are compared. The cars have very little in common but the number of cars that get me excited is small and here’s two in one 12 minute video. (Honestly, I wish the Tesla looked more distinctive but I guess if you’ve got to persuade petrol-car-fans that electric is worthwhile you don’t want to turn them off with something outlandish.)

And finally, a brief conversation about writing and how to indicate bits you’ll do later. (“TK” is the professional way, as Tom L says, but I usually go for the “XXX” option.)

With all that excitement I didn’t have time to read much news, so that makes the morning even cheerier!

In Misc on 23 June 2014. Permalink

Clumsy-footed idiot

When I finished my last job I decided that unless the perfect role magically appeared I’d spend some time getting to grips with, to be vague, “data visualisation and mappy stuff”. I decided I could treat it like studying: set myself projects, learn new things, read related material, watch talks, and post frequent updates here. But there have been no updates here.

I thought a good way to get up to speed with designing and making “data viz stuff” would be to take another stab at D3.js. I’ve used it a handful of times before but only to the extent of trying to cram different bits of example code together and being confused for days at a time.

From talking to friends, this seems a common pattern with D3. How is this code doing something? Why is it now doing nothing? What is enter(), what am I entering? Where’s the data gone, how do I get at it, can I please get a single sodding line to appear, show me something anything please oh God please.

But this time, this time, I’d methodically make progress until it clicked, until I was, to some degree, fluent. One step at a time. I wanted to be able to sketch an idea and know how to make a live, data-driven version of it appear on the screen. D3 can do so many things and before long I would be able to as well!

To warm up, and remember what little I’d learned before, I set myself a quick project. Re-reading The Visual Display of Quantitative Information I came across William Playfair’s 18th century charts showing the balance of imports and exports for a country:

A time series chart

Making a D3 version of this, with modern data, letting the user compare different countries, seemed feasible. I’d drawn simple line charts before with D3 so this would be a manageable next step. “Only two or three days,” I thought.

I just… I don’t know. Is it my brain? Does my brain work at a substantially slower speed than it should? Am I an imbecile and all my friends are too polite to tell me? Even with so much example code floating around this was difficult. IT’S JUST A LINE CHART! What’s wrong with me?

After stumbling around for a while, making little progress, I thought I should go back to basics. I read a little book that in retrospect only confused me further. I tried another, bigger book that did begin to help. I could almost see understanding on the horizon, only a day or two’s journey away. I could almost grasp the basics, but not quite, not so as the understanding lasted.

“This is fine,” I told myself, “learning new things is always like this.” I remembered back a few years when I started learning Python and Django and how I’d spend days at a time making no progress at all, bashing my head against mysterious and un-Googlable error messages. But eventually, although there are always new problems, I felt gradually less and less of an idiot.

Learning new coding stuff is, for me, like this scene from Intacto:

“I’m doing new things! This is so exhilarating!” I think as I run blindfold through a forest of new terms and concepts. “Look what I can do!” Then, BAM! I come to a complete and bloody halt, flat on my back, and I feel more stupid than before I started.

And trying to draw this one simple chart was the densest, most unforgiving of forests for this clumsy-footed idiot of a runner. Every. Single. Sodding. Step. I’d end up broken-nosed on the ground for a day or two, often, eventually, having to reluctantly ask for a hand up (usually the hand of Lars Kotthoff to whom I owe many thanks).

Why is it so hard to work out what format this data needs to be in? Why does my x-axis only have two points on it? Why do the shaded areas between the lines break when new countries are added to the chart? How do I display two lines each for multiple countries? Why does it keep re-drawing the axes over and over? Why can’t I update the chart with new data? Why can I only draw one circle, rather than one for every data point? Why can’t I give a unique name to each point so that they update correctly? Have I really got the data into the correct format?

The weeks went by. It was much less exhilarating that running through a forest blindfold. But now, I’ve finished. You can try it out. Here’s a static picture of it:

A chart

Ignoring this sorry tale, it’s OK, and I’m reasonably pleased. It could be improved, but it’s not embarrassing. Comparing too many lines will get confusing and the country names labelling lines will become useless. It’s not ideal if you’re colourblind. I haven’t done any browser testing.

I don’t know how long this “two or three day” quick little project took me. We could count it in weeks. Imagine if I was being paid a day rate for this. Pathetic. Really. I don’t even have any interest in the imports or exports of different countries.

But, still… even though it was painful right up until the end, and even though, even after all that, I still don’t feel I “get” D3 yet, something has stuck. Over the final few days I managed to figure out a few things for myself, things that would have left me flat on the forest floor a few weeks earlier. It’s not much. I couldn’t even create this really quite simple chart, very, very slowly, without getting completely and utterly stuck several times. But I’m clinging on to those tiny, possibly imagined, moments of understanding as embarrassingly tiny signs that maybe I’m not entirely an idiot.

In Projects on 20 May 2014. Permalink

Black and white

It’s the same joke, from Frasier and Parks and Recreation, but still a good one.

From Frasier, season 9, episode 7, the shades of white:

Niles: What colour is the new carpet?

Frasier: I’m going up a shade… to Harvest Wheat.

Niles: I thought the next shade up was Buff?

Frasier: It used to be. But they’ve discovered a whole new colour in between.

Niles: So now it’s Tofu, Putty, Oatmeal, [Frasier joins in] Almond, Harvest Wheat, and Buff.

Niles: That’s going to be hard to get used to.

And from Parks and Recreation, season 3, episode 16, the shades of black:

Jean-Ralphio holds up swatches of material.

Tom: The memorial ribbons?

Jean-Ralphio: No doubt. Which one floats your penis?

Jerry [confused]: They’re all black.

Tom [laughing]: Maybe to the layman Jerry! Obsidian, Onyx, Midnight, Lost Soul, Rolling Blackout, Sleeping Panther, and Void by Armani. [Holding up red tie] Hey, Jerry, does this look black to you too?!

Any others?

(Also, I love how increasingly easy it is to find exactly the right, brief, clips on YouTube.)

In Television on 28 April 2014. Permalink

Recent comments on writing

Writing archives by category