Tag: intelligent failure

Thinking aloud: does a museum's obsession with polish hinder innovation?

I'm blogging several conversations on twitter around the subject of innovation and experimentation that I thought were worth saving, not least because I'm still thinking about their implications.

To start with, Lynda Kelly (@lyndakelly61) quoted @sebchan at the Hot Science conference on climate change and museums:

'Museums want everything to be slick and polished for mass audience, we lose capacity to be experimental and rapid'

 which lead me to tweet:

'does big museum obsession with polish hinder innovation? ('innovation' = keeping up with digital world outside)'.

which lead to a really interesting series of conversations.  Erin Blasco responded (over several tweets):

We can't pilot if it's not perfect. … Need to pilot 15 quick/dirty QR codes but we can't put ANY up unless there are 50 & perfectly, expensively designed & impressive. … So basically not allowed to fail and learn = not allowed to pilot = we spend a bunch of $ and fail anyway? … To clarify: it's a cross-dept project. One dept ok with post-it notes & golf pencils. Two others are not. Kinda deadlock.

I think this perfectly illustrates the point and it neatly defines the kind of 'polish' that slows things down – the quality of the user experience with the QR codes would rest with the explanatory text, call to action and the content the user finds at the other end, not the weight and texture of the paper or vinyl they're printed on.  Suddenly you've got extra rounds of emails and meetings for those extra layers of sign-off, a work request or contract for design time, plus all the stakeholder engagement that you already, but does that extra investment of time and resources result in a better experiment in audience research?

But kudos to Erin for gettings things this far!  (An interesting discussion followed with Erin and @artlust about possible solutions, including holding stakeholder evaluations of the prototypes so they could see how the process worked, and 'making the pilot-ness of it a selling point in the design, letting audiences feel they're part of something special', which made me realise that turning challenges into positives is one of my core design techniques.)

For Linda Spurdle, the barriers are more basic:

Innovation costs, even my plans to try things cheap/free get scuppered by lack of time. For me less about risk more about resources

Which also rings perfectly true – many potential museum innovators were in this position before the museum funding cuts took hold, so innovating your way out of funding-related crises must be even more difficult now.

On the topic of innovation, Lindsey Green said the 'definite reluctance to pilot and fail impacts innovation'. Rachel Coldicutt had just blogged about 'digital innovation in the arts' in Making Things New, pointing out that the question 'privileges the means of delivery over the thing that’s being delivered', and tweeting that 'innovating a system and innovating art aren't the same thing and perhaps there's more impact from innovating the system'.

If the quest is to, as Rachel problematises in her post, 'use digital technologies to remake the Arts Establishment', then (IMO) it's doomed to failure. You can't introduce new technologies and expect that the people and processes within a cultural organisation will magically upgrade themselves to match. More realistically, people will work around any technology that doesn't suit them (for entirely understandable reasons), and even the best user experience design will fail if it doesn't take account of its context of use. If you want to change the behaviour of people in an organisation, change the metrics they work to. Or, as Rachel says, '[r]ather than change for change’s sake, perhaps we should be identifying required outcomes'.  Handily, Bridget McKenzie pointed out that 'The Museums for the Future toolkit includes new eval framework (GEOs = Generic Environmental Outcomes)', so there's hope on the horizon.

The caveats: it's not that I'm against polish, and I think high production values really help our audiences value museum content. But – I think investing in a high level of polish is a waste of resources during prototyping or pilot stages, and a focus on high production values is incompatible with rapid prototyping – 'fail faster' becomes impossible. Usability researchers would also say polished prototypes get less useful feedback because people think the design is set (see also debates around the appearance of wireframes).

It's also worth pointing out my 'scare quotes' around the term 'innovation' above – sadly, things that are regarded as amazing innovations in the museum world are often delayed enough that they're regarded as pretty normal, even expected, by our more digitally-savvy audiences. But that's a whole other conversation…

So, what do you think: does a museum's obsession with polish hinder innovation?

Update, January 2013: Rob Stein has written 'Museum Innovation: Risk, Experimentation and New Ideas', which resonated strongly:

A common pitfall for museums is an unhealthy addiction to monumental undertakings. When massive projects loom with ties to outside support and countless staff hours invested in a single deliverable, it becomes very difficult to admit the possibility of failure. As a result, we shy away from risk, mitigate the probability of embarrassment, and crush innovation in the process.

Museums and iterative agility: do your ideas get oxygen?

Re-visiting the results of the survey I ran about issues facing museum technologists has inspired me to gather together some great pieces I've read on museum projects moving away from detailed up-front briefs and specifications toward iterative and/or agile development.

In 'WaterWorx – our first in-gallery iPad interactive at the Powerhouse Museum', Seb Chan writes:

"the process by which this game was developed was in itself very different for us. … Rather than an explicit and ‘completed’ brief be given to Digital Eskimo, the game developed using an iterative and agile methodology, begun by a process that they call ‘considered design‘. This brought together stakeholders and potential users all the way through the development process with ‘real working prototypes’ being delivered along the way – something which is pretty common for how websites and web applications are made, but is still unfortunately not common practice for exhibition development."

I'd also recommend the presentation 'Play at Work: Applying Agile Methods to Museum Website Development' given at the 2010 Museum Computer Network Conference by Dana Mitroff Silvers and Alon Salant for examples of how user stories were used to identify requirements and prioritise development, and for an insight into how games can be used to get everyone working in an agile way.  If their presentation inspires you, you can find games you can play with people to help everyone understand various agile, scrum and other project management techniques and approaches at tastycupcakes.com.

I'm really excited by these examples, as I'm probably not alone in worrying about the mis-match between industry-standard technology project management methods and museum processes. In a 'lunchtime manifesto' written in early 2009, I hoped the sector would be able to 'figure out agile project structures that funders and bid writers can also understand and buy into' – maybe we're finally at that point.

And from outside the museum sector, a view on why up-front briefs don't work for projects that where user experience design is important.  Peter Merholz of Adaptive Path writes:

"1. The nature of the user experience problems are typically too complex and nuanced to be articulated explicitly in a brief. Because of that, good user experience work requires ongoing collaboration with the client. Ideally, client and agency basically work as one big team.

2. Unlike the marketing communications that ad agencies develop, user experience solutions will need to live on, and evolve, within the clients’ business. If you haven’t deeply involved the client throughout your process, there is a high likelihood that the client will be unable to maintain whatever you produce."

Finally, a challenge to the perfectionism of museums.  Matt Mullenweg (of WordPress fame), writes in '1.0 Is the Loneliest Number': 'if you’re not embarrassed when you ship your first version you waited too long'.  Ok, so that might be a bit difficult for museums to cope with, but what if it was ok to release your beta websites to the public?  Mullenweg makes a strong case for iterating in public:

"Usage is like oxygen for ideas. You can never fully anticipate how an audience is going to react to something you’ve created until it’s out there. That means every moment you’re working on something without it being in the public it’s actually dying, deprived of the oxygen of the real world.

…

By shipping early and often you have the unique competitive advantage of hearing from real people what they think of your work, which in best case helps you anticipate market direction, and in worst case gives you a few people rooting for you that you can email when your team pivots to a new idea. Nothing can recreate the crucible of real usage.

You think your business is different, that you’re only going to have one shot at press and everything needs to be perfect for when Techcrunch brings the world to your door. But if you only have one shot at getting an audience, you’re doing it wrong."

* The Merholz article above is great because you can play a fun game with the paragraph below – in your museum, what job titles would you put in place of 'art director' and 'copywriter'?  Answers in a comment, if you dare!  I think it feels particularly relevant because of the number of survey responses that suggested museums still aren't very good at applying the expertise of their museum technologists.

"One thing I haven’t yet touched on is the legacy ad agency practice where the art director and copywriter are the voices that matter, and the rest of the team exists to serve their bidding. This might be fine in communications work, but in user experience, where utility is king, this means that the people who best understand user engagement are often the least empowered to do anything about it, while those who have little true understanding of the medium are put in charge. In user experience, design teams need to recognize that great ideas can come from anywhere, and are not just the purview of a creative director."

—
If you liked this post, you may also be interested in Confluence on digital channels; technologists and organisational change? (29 September 2012) and A call for agile museum projects (a lunchtime manifesto) (10 March 2009).

Oh noes, a FAIL! Notes from the unconference session on 'failure' at MW2009

These are my really rough notes from the unconference session at Museums and the Web, written up quickly in order to capture the essence of the discussion and open it up for comment.

Susan Chun, Dana Mitroff Silvers, Bruce Wyman and I began and were later joined by Seb Chan and Jennifer Trant.

I explained my motivation in suggesting the session – intelligent, constructive failure is important. Finding ways to create a space for that conversation isn't something we do well at the moment.

Susan started the conversation by pointing out that there were different definitions or types of failure. Defining 'failure' more precisely is useful.

Types of failures include: over budget, badly implemented, badly specified, future failures.

Dana pointed out that we needed to define success as well as defining failure. A more nuanced understanding of failure is important, especially when hoping to encourage more people to talk about failure. Discussion about choosing the right metrics for success – the right metrics may vary depending on whether you're a funder or a department or whoever.

Funding models can set you up for failure.

Bruce pointed out that it's not the failure that matters, it's what you do with the failure.

Some apparent failures may not really be failures.

Are you funding the process or the product?

Not having the mechanism for exposing the knowledge is a failure.

The definitions of failure and success need to include the net gain for an organisation or in new/improved processes as well as the product.

What kind of environment is needed so that people can publish judgements of their own success or failure?

Susan suggested the MCN project registry would be a good place for this information.

What if it was routine to talk about what failed or succeeded in each project? Funding should reward people who talk about failures. Discussion about space for reflection on 'lessons learned' in project summation.

Agency is important – you talk about the failures of your own projects, other people don't dob you in.

Dana – talking about failures in a project should be a normal part of MW papers.

Label it 'lessons learned', not 'failure'.

Susan – [Remove roadblocks about what happens if funders hear you think your project failed in some way -] Talk to funders about requiring an examination or reflection of each project for failure in the same way the issue of open source development was tackled. Pro-active approach!

Me: when you're putting in for funding, you should have to show that you've talked to people with similar projects about the lessons they learned.

Susan – put ILMS (Institute of Museum and Library Services) reports online. [A small but practical thing to do]. Change the culture of secrecy.

Funding can be a carrot and a stick. Without that, institutional change is hard.

Points of resistance (some summing up):
understanding how to define failure/success
culture of secrecy
fear of exposure to funders
lacking the jargon to describe failure (which would also help normalise the process of discussing it openly)

Jennifer – if there aren't any negative consequences, why can't you talk about it?

General discussion about the need for early, continual dialogue about projects. It's difficult to talk about failures if you're not already talking about the project. Paraphrasing Seb -talking about it already in an informal context, like a blog, may help here.

Iterative, transparent reporting is important. It also helps other people talk about failures.

Susan – other causes of failures are project that never happened. Whether they missed their time, didn't get funding, whatever. Consider those as failures too, and talk about them. Everyone benefits, whether that's the person with the great idea that never got to see it happen, or people who've built on it later.

Talk about nascent projects. Exposing them to comment early can help prevent failure. Like the old crack about voting, public discussion about projects should happen early and often!

Hoarding ideas is pointless.

We need a template for talking about failure. Prompts or questions for consideration.

It's not just overall project failures, it can include institutional, departmental or structural failures.

Dana suggested confessional sessions, perhaps at the next Museums and the Web conference. Jennifer and Seb took it up, suggesting YouTube captures with disguised voices and silhouettes to make it easier, and encouraging discussion of failures by type or theme.

Discussion about the role of commentators, respondents in sessions. The voice of the one that didn't work.

Find an acceptable form of critical questions so that people can help prevent other projects failing, make the most of the experience out there.

Putting my money where my mouth is, one final comment from Seb was about a possible failure of the unconference sessions in not getting people together again at the end to report back. This was received constructively, and might happen during the final plenary.

A call for agile museum projects (a lunchtime manifesto)

Yet another conversation on twitter about the NMOLP/Creative Spaces project lead to a discussion of the long lead times for digital projects in the cultural heritage sector. I've worked on projects that were specced and goals agreed with funders five years before delivery, and two years before any technical or user-focussed specification or development began, and I wouldn't be surprised if something similar happened with NMOLP.

Five years is obviously a *very* long time in internet time, though it's a blink of an eye for a museum. So how do we work with that institutional reality? We need to figure out agile, adaptable project structures that funders and bid writers can also understand and buy into…

The initial project bid must be written to allow for implementation decisions that take into account the current context, and ideally a major goal of the bid writing process should be finding points where existing infrastructure could be re-used. The first step for any new project should be a proper study of the needs of current and potential users in the context of the stated goals of the project. All schema, infrastructure and interface design decisions should have a link to one or more of those goals. Projects should built around usability goals, not object counts or interface milestones set in stone three years earlier.

Taking institutional parameters into account is of course necessary, but letting them drive the decision making process leads to sub-optimal projects, so projects should have the ability to point out where institutional constraints are a risk for the project. Constraints might be cultural, technical, political or collections-related – we're good at talking about the technical and resourcing constraints, but while we all acknowledge the cultural and political constraints it often happens behind closed doors and usually not in a way that explicitly helps the project succeed.

And since this is my lunchtime dream world, I would like plain old digitisation to be considered sexy without the need to promise funders more infrastructure they can show their grandkids.

We also need to work out project models that will get buy-in from contractors and 3rd party suppliers. As Dan Zambonini said, "Usability goals' sounds like an incredibly difficult thing to quantify' so existing models like Agile/sprint-esque 'user stories' might be easier to manage.

We, as developers, need to create a culture in which 'failing intelligently' is rewarded. I think most of us believe in 'failing faster to succeed sooner', at least to some extent, but we need to think carefully about the discourse around public discussions of project weaknesses or failures if we want this to be a reality. My notes from Clay Shirky's ICA talk earlier this year say that the members of the Invisible College (a society which aimed to 'acquire knowledge through experimental investigation') "went after alchemists for failing to be informative when they were wrong" – " it was ok to be wrong but they wanted them to think about and share what went wrong". They had ideas about how results should be written up and shared for maximum benefit. I think we should too.

I think the MCG and Collections Trust could both have a role to play in advocating more agile models to those who write and fund project bids. Each museum also has a responsibility to make sure projects it puts forward (whether singly or in a partnership) have been reality checked by its own web or digital specialists as well as other consultants, but we should also look to projects and developers (regardless of sector) that have managed to figure out agile project structures that funders and bid writers can also understand and buy into.

So – a blatant call for self-promotion – if you've worked on projects that could provide a good example, written about your dream project structures, know people or projects that'd make a good case study – let me know in the comments.

Thanks also to Mike, Giv and Mike, Daniel Evans (and the MCG plus people who listened to me rant at dev8D in general) for the conversations that triggered this post.

—
If you liked this post, you may also be interested in Confluence on digital channels; technologists and organisational change? (29 September 2012) and Museums and iterative agility: do your ideas get oxygen? (21 November 2010).

'The strikethrough is the canonical symbol of the Web'

Below is a quote from Wired's Chris Anderson on museum, curatorial authority and the long tail, from a Washington Post report, 'Smithsonian Click-n-Drags Itself Forward' on Smithsonian 2.0 ('A Gathering to Re-Imagine the Smithsonian in the Digital Age').

The quote really covers two issues – making failures and mistakes in public and leaving them there, and training external volunteers and experts to curate parts of collections, because no one curator can be authoritative on everything in their remit: "in exchange for a slight diminution of the credentialed voice for a small number of things, you would get far more for a lot of things".

I suspect this is a false dichotomy – there's a place for both internal and external expertise. The Science Museum object wiki doesn't mean the rest of the collection catalogue and interpretation has no value or relevance. The challenge lies in presenting organisation and user-contributed content in the same interface – can those boundaries be removed? Is it wise to try? And what about taking external content back into the catalogue?

This isn't a new conversation for museum technologists, but it's a conversation I'd love to have with curators. I've never been sure how the technologists who get really excited by the possibilities of sharing content online in various ways can go about working with curators to find the best way of managing it so that the public, the collections and the curators benefit.

Anyway, onto Chris Anderson:

The discovery of the "long tail" principle has implications for museums because it means there is vast room at the bottom for everything. Which means, Anderson said, that curators need to get over themselves. Their influence will never be the same.

"The Web is messy, and in that messiness comes something new and interesting and really rich," he said. "The strikethrough is the canonical symbol of the Web. It says, 'We blew it, but we are leaving that mistake out there. We're not perfect, but we get better over time.' "

If you think that notion gives indigestion to an organization like the Smithsonian — full of people who have devoted much of their lifetimes to bringing near-perfect luster to some tiny pearl of truth — you would be correct.

The problem is, "the best curators of any given artifact do not work here, and you do not know them," Anderson told the Smithsonian thought leaders. "Not only that, but you can't find them. They can find you, but you can't find them. The only way to find them is to put stuff out there and let them reveal themselves as being an expert."

Take something like, oh, everything the Smithsonian's got on 1950s Cold War aircraft. Put it out there, Anderson suggested, and say, "If you know something about this, tell us." Focus on the those who sound like they have phenomenal expertise, and invest your time and effort into training these volunteers how to curate. "I'll bet that they would be thrilled, and that they would pay their own money to be given the privilege of seeing this stuff up close. It would be their responsibility to do a good job" in authenticating it and explaining it. "It would be the best free labor that you can imagine."

It didn't go down easily among the thought leaders, who have staked their lives' work on authoritativeness, on avoiding strikethroughs. What about the quality and strength of the knowledge we offer? asked one Smithsonian attendee.

You don't get it, Anderson suggested. "There aren't enough of you. Your skills cannot be invested in enough areas to give that quality."

It's like Wikipedia and the Encyclopedia Britannica, Anderson said. Some Wikipedia entries certainly are not as perfectly polished as the Britannica. But "most of the things I'm interested in are not in the Britannica. In exchange for a slight diminution of the credentialed voice for a small number of things, you would get far more for a lot of things. Something is better than nothing." And right now at the Smithsonian, what you get, he said, is "great" or "nothing."

"Is it our job to be smart and be the best? Or is it our job to share knowledge?" Anderson asked.