On the trickiness of crowdsourcing competitions: some lessons from Sydney Design

I generally maintain a diplomatic silence about crowdsourcing competitions when I'm talking about crowdsourcing in cultural heritage as I believe spec work (or asking people to invest time in creating designs then paying just one 'winner') is unethical, and it's really tricky for design competitions to avoid looking like 'spec work'. I discovered this for myself when I ran the 'Cosmic Collections' mashup competition, so I have a lot of sympathy for museums who unknowingly get it wrong when experimenting with crowdsourcing. I also tend not to talk about poorly conceived or executed crowdsourcing projects as it doesn't seem fair to single out cultural heritage institutions that were trying to do the right thing against odds that ended up beating them, but I think the lessons to be drawn from the Sydney Design festival's competition are important enough to discuss here.

'Is it a free poster yet?'
'Is it a free poster yet?'

A crowdsourcing competition model that the museum had previously applied successfully (the Lace Award and Trainspotting, with prizes up to $AUD20,000 and display in the exhibition for winning designs) had a very different reception when the context and rewards changed. When the Powerhouse Museum's design competition to produce the visual identity for the Sydney Design festival was launched with a $US1000 prize, the design community's sensitivity to spec work and 'free pitching' was triggered, and they started throwing in some sarcastic responses.  The public feedback loop created as people could see previous designs and realised their own would also be featured on the site had a 4Chan-ish feel of a fun new meme about it, and once the norm of satirical responses was set, it was only going to escalate.

More importantly, there was a sense that Sydney Design was pulling a swifty. As Kate Sweetapple puts it in How the Sydney Design festival poster competition went horribly wrong:

'The fundamental difference [to the previous competitions], however, is that by running the competition, the Museum pulled a substantial job – worth tens of thousands of dollars – out of the professional marketplace. The submissions to Love Lace and Trainspotting did not have a commercial context one year, and none the next.'

If the previous reward was mostly monetary, offering a lesser intrinsic reward in exchange for a previously extrinsic reward is unlikely to work. If there's a bigger reward than than the competition brief itself would suggest, one important lesson is to make it unavoidably obvious. In this case, the Sydney Design Team's response said 'the Museum would have engaged the winning designer for further work and remuneration required to roll out the winning design into a more comprehensive marketing campaign', but this wasn't clear in the original brief. Many museum competitions display highly-ranked entries in their gallery spaces, and being exhibited in the museum or festival spaces might have been another form of valid reward, but only if it worked as an aspiration for the competition's audience, who in this case might well have a breadth of experience and exposure that rendered it less valuable.

Finally, in working with museums online, I've noticed the harshness of criticism is often proportionate to how deeply people care about you or identify you with certain values they hold dear.  When you're a beloved institution, people who care deeply about you feel betrayed when you get things wrong. As one commentator said in With friends like these, who needs enemies?, 'Sydney Design are meant to be in our corner'. If you regard critics as 'critical friends' you can turn the relationship around (as Merel van der Vaart discusses in the 'Opening up' section of her post on lessons from the Science Museum's Oramics exhibition) and build an even stronger relationship with them. Maybe Sydney Design can still turn this around…

Slides and talk from 'Cosmic Collections' paper

This is a lazy post, a straight copy and paste of my presentation notes (my excuse is that I'm eight days behind on everything at work and uni after being grounded in the US by volcanic ash). Anyway, I hope you enjoy it or that it's useful in some way.

Cosmic Collections: creating a big bang?

View more presentations from Mia .

Slide 1 (solar rays – Cosmic Collections):

The Cosmic Collections project was based on a simple idea – what if we gave people the ability to make their own collection website? The Science Museum was planning an exhibition on astronomy and culture, to be called ‘Cosmos & Culture’. We had limited time and resources to produce a site to support the exhibition and we risked creating ‘just another exhibition microsite’. So what if we provided access to the machine-readable exhibition content that was already being gathered internally, and threw it open to the public to make websites with it?  And what if we motivated them to enter by offering competition prizes?  Competition participants could win a prize and kudos, and museum audiences might get a much more interesting, innovative site.
The idea was a good match for museum mission, exhibition content, technical context, hopefully audience – but was that enough?
Slide 2 (satellite dish):
Questions…
If we built an API, would anyone use it?
Can you really crowdsource the creation of collections interfaces?
The project gave me a chance to investigate some specific questions.  At the time, there were lots of calls from some quarters for museums to produce APIs for each project, but would anyone actually use a museum API?  The competition might help us understand whether or how we should invest in APIs and machine-readable data.
We can never build interfaces to meet the needs of every type of audience.  One of the promises of machine-readable data is that anyone can make something with your data, allowing people with particular needs to create something that supports their own requirements or combines their data with ours – but would anyone actually do it?
Slide 3 (map mashup):
Mashups combine data from one or more sources and/or data and visualisation tools such as maps or timelines.
I'm going to get the geek stuff out of the way and quickly define mashups and APIs…
Mashups are computer applications that take existing information from known sources and present it to the viewer in a new way. Here’s a mashup of content edits from Wikipedia with a map showing the location of the edit.
Slide 4 (APIs)
APIs (Application Programming Interfaces) are a way for one machine to talk to another: ‘Hi Bob, I’d like a list of objects from you, and hey, Alice, could you draw me a timeline to put the objects on?’
APIs tell a computer, 'if you go here, you will get that information, presented like this, and you can do that with it'.
A way of providing re-usable content to the public, other museums and other departments within our museum – we created a shared backend for web and gallery interactives.
I think of APIs as user interfaces for developers and wanted to design a good experience for developers with the same care you would for end users*.  I hoped that feedback from the competition could be used to improve the beta API
* we didn’t succeed in the first go but it’s something to aim for post-beta
Slide 5: (what if nobody came?)
AKA 'the fears and how to deal with them'
Acknowledge those fears
Plan for the worst case scenario
Take a deep breath and do it anyway
And on the next slides, the results.  If I was replicating the real experience, you’d have several nerve-biting months while you waited for the museum to lumber into gear, planned the launch event, publicised the project in the participant communities… Then waited for results to come in. But let’s skip that bit…
Slide 6: (Ryan Ludwig's http://www.serostar.com/cosmic/)
The results – our judges declared a winner and a runner-up, these are screenshots – this is the second prize winning entry.
People came to the party. Yay! I'd like to thank all the participants, whether they submitted a final entry or not. It wouldn't have worked without them.
Slide 7: (Natalie and Simon's http://cosmos.natimon.com/)
This is a screenshot from the winning site – it made the best use of the API and was designed to lure the visitor in and keep drawing them through the site.
(We didn’t get subject specialists scratching their own itch – maybe they don’t need to share their work, maybe we didn’t reach them. Would like to reach researchers, let them know we have resources to be used, also that they can help us/our audiences by sharing their work)
Slide 8: (astrolabe – what did we learn?)
People need (more) help to participate in a geektastic project like this
The dynamics of a competition are tricky
Mashups are shaped by the data provided – you get out what you put in
Can we help people bring their own content to a future mashup?
Slide 9: (evaluation)
I did a small survey to evaluate the project… Turns out the project was excellent outreach into the developer community. People were really excited about being invited to play with our data.  My favourite quote: "The very idea of the competition was awesome"
Slide 10: (paper sheet)
Also positive coverage in technical press. So in conclusion?
Slide 11: (Tim Berners-Lee):
“The thing people are amazed about with the web is that, when you put something online, you don’t know who is going to use it—but it does get used.”
There are a lot of opportunities and excitement around putting machine-readable data online…
Slide 12: Tim Berners-Lee 2:
But:  It doesn’t happen automatically; It’s not a magic bullet
But people won't find and use your APIs without some encouragement. You need to support your API users. People outside the museum bring new ideas but there's still a big role for people who really understand the data and audiences to help make it a quality experience…
Slide 13 (space):
What next?
Using the feedback to focus and improve collection-wide API
Adding other forms of machine-readable data
Connecting with data from your collections?
I've been thinking about how to improve APIs – offer subject authorities with links to collections, embed markup in the collections pages to help search engines understand our data…
I want more! The more of us with machine-readable data available for re-use, the better the cross-collections searches, the region or specialism-wide mashups… I'd love to be able to put together a mashup showing all the cultural heritage content about my suburb; all the Boucher self-portraits; all the inventions that helped make the Space Shuttle work…
Slide 14: (thank you)
If you're interested in possibilities of machine-readable data and access to your collections, join in the conversation on the museum API wiki or follow along on twitter or on blogs.  Join in at http://museum-api.pbworks.com/
More at https://openobjects.org.uk/ or @mia_out

Image credits include:
http://antwrp.gsfc.nasa.gov/apod/ap100415.html
http://antwrp.gsfc.nasa.gov/apod/ap100414.html
http://antwrp.gsfc.nasa.gov/apod/ap100409.html
http://antwrp.gsfc.nasa.gov/apod/ap100209.html
http://antwrp.gsfc.nasa.gov/apod/ap100315.html
http://www.sciencemuseum.org.uk/Centenary/Home/Icons/Pilot_ACE_Computer.aspx

Mash the state

Cosmic Collections – the results are in. And can you help us ask the right questions?

For various reasons, the announcement of the winners of our mashup competition has been a bit low key – but we're working on a site that combines the best bits of the winners, and we'll make a bit more of a song and dance about it when that's ready.

I'd like to take the opportunity to personally thank the winners – Simon Willison and Natalie Down in first place, and Ryan Ludwig as runner-up – and equally importantly, those who took part but didn't win; those who had a play and gave us some feedback; those who helped spread the word, and those who cheered along the way.

I have a cheeky final request for your time.  I would normally do a few interviews to get an idea of useful questions for a survey, but it's not been possible lately. I particularly want to get a sense of the right questions to ask in an evaluation because it's been such a tricky project to explain and 'market', and I'm far too close to it to have any perspective.  So if you'd like to help us understand what questions to ask in evaluation, please take our short survey http://www.surveymonkey.com/s/5ZNSCQ6 – or leave a comment here or on the Cosmic Collections wiki.  I'm writing a paper on it at the moment, so hopefully other museums (and also the Science Museum itself) will get to learn from our experiences.

And again – my thanks to those who've already taken the survey – it's been immensely useful, and I really appreciate your honesty and time.

Nine days to go! And entering Cosmic Collections just got easier

Quoting myself over on the museum developers blog, Cosmic Collections – do one thing and do it well:

I’ve realised that there may be some mismatch between the way mashups tend to work, and the scope we’ve suggested for entries to our competition. The types of interfaces someone might produce with the API may lend themselves more to exploring one particular idea in depth than produce something suitable for the broadest range of our audiences.

So I’m proposing to change the scope for entries to the competition, to make it more realistic and a better experience for entrants: I’d like to ask you to build a section of a site, rather than a whole site. The scope for entrants would then be: “create something that does one thing, and does it well”. Our criteria – use of collections data, creativity, accessibility, user experience and ease of deployment and maintenance – are still important but we’ll consider them alongside the type of mashup you submit.

I've updated the Cosmic Collections competition page to reflect this change. This page also features a new 'how to take part' section, including a direct link to the API and to a discussion group.

I'd love to hear your thoughts on this change – there's an email address lurking on the competition page, and I'm on twitter @mia_out and @coscultcom.

In other news, programmableweb published a blog post about the competition today: Science Museum Opens API and Challenges Developers to Mashup the Cosmos. Woo!

And I don't know if it's any kind of consolation if you're entering, but I'll be working right alongside you up until Friday 28th, on an assignment for my MSc.

'Cosmic Collections' launches at the Science Museum this weekend

I think I've already said pretty much everything I can about the museum website mashup competition we're launching around the 'Cosmos and Culture' exhibition, but it'd be a bit silly of me not to mention it here since the existence and design of the project reflects a lot of the issues I've written about here.

If you make it along to the launch at the Science Museum on Saturday, make sure you say hello – I should be easy to find cos I'm giving a quick talk at some point.
Right now the laziest thing I could do is to give you a list of places where you can find out more:
Finally, you can talk to us @coscultcom on twitter, or tag content with #coscultcom.
Btw – if you want an idea of how slowly museums move, I think I first came up with the idea in January (certainly before dev8D because it was one of the reasons I wanted to go) and first blogged about it (I think) on the museum developers blog in March. The timing was affected by other issues, but still – it's a different pace of life!

Woohoo!

The results of JISC's dev8D 'developer happiness' prize have been announced – congratulations to List8D and their "web 2.0-friendly reading lists" – it's something I'd love to see in my own uni course.

And yay the Three Lazy Geeks, because we came second! That was a lovely surprise, and really the glace cherry on the icing of the cake because the whole event was a great experience, and I really enjoyed working with Ian and Pete, as rushed as the whole thing was.

Yay! Three Lazy Geeks shortlisted for dev8D prize

We're in the top five, whoop!

The reviewers said, "Really comprehensive treatment of the problem and associated issues. Worth pursuing I think… As a solution this is a good idea and was produced by genuine collaboration at the Dev8D event."

So a short but happy developer post from me. The whole experience was lots of fun, and it would never have worked without Ian Ibbotson and Pete Sefton. I think the thing that I like most about it is that it not only re-uses existing tools, it fits with how people already work. It's not "this application will change your life, but first you have to change your life". I know that the (mostly junior) academics I've mentioned it to have loved the idea, so it might have real users if it was developed, which would be lovely.

Our dev8D 'Lazy Lecturer' prototype

In the interests of transparency, I thought I'd put the submission for the 'Lazy Lecturer' prototype I worked on with Ian Ibbotson and Pete Sefton for the JISC dev8D 'developer decathlon' online.

I really should blog more about the event – both lessons I learnt from the content and event structure, but also the experience of being surrounded by actual (higher education, mostly open source/LAMP) geeks. But hey, this will do in the meantime.