The history of a failure (Where's the value in that?)

Yes, this post represents a quest (nay an odyssey!) for that silver lining, because even though as I hope to show; failure has value, it still hurts!

It all began on the 16^th of June of this past year 2011 when an ex-colleague of mine contacted me regarding project management at her new place of work. Introductions ensued and the first contacts were so bright that by the 23^rd I was led to tweet:

One hot afternoon another two weeks down the line, we “pitched” by delivering a Scrum Basics workshop, on the framework of a retrospective, to a collection of interested individuals. One month and a few meetings later, my colleague and I were given the opportunity to propose a model for helping the development organisation of some 7 teams move to scrum. It seemed logical to all concerned to grow scrum in the organisation, team by team. This naturally generated the possibility of a pilot to start.

We reasoned as follows:

We had 3 workshops (Basics, Backlog Management and Planning & Estimating) and reckoned we needed 2 to 4 sprints to get a team off to a solid start. Where possible we would graft our workshops onto actual scrum ceremonies(planning, grooming, review & retrospective), using 'live' input, so that on top of having imparted whatever theory was necessary for the activity in question, the workshops would produce actionable output, in the form of the familiar scrum artefacts (realease & sprint backlogs etc). In this way we hoped to do more showing, though doing, than telling and to be one step ahead of anyone inclined to feel that our learning was getting in the way of the 'real' work.

We also wanted to just hang out a bit, on the one hand to explain and discuss and help out where possible with new ideas like swarming, pairing, WIP etc, should the need arise, but also to get to know the team and experience the physical environment in which the work took place, attend stand-ups, help with setting up other ceremonies etc., etc but we did not know how much time this would represent (we had no data from previous efforts). Wouter and I were also keen to pair throughout the pilot transition; we knew we had an awful lot to learn too

So, to remain honest and open in the relationship, and further sweeten the deal – we offered all our consulting hours for the pilot at zero euro. In this way all parties would have at least some basis to predict costs/effort before committing to anything truly substantial and Wouter & I could both hang around as we saw fit during the pilot.

It was a good deal. Our return, besides 2 paid workshops (we agreed that we would only charge for workshops if they had been 'tested' at least once before), was the opportunity to learn, structured feedback and some word of mouth advertisement. Of course, with a bit of luck, the whole thing would be the start of a mutually fruitful, and profitable, collaboration.

We got the go-ahead, the only decision that remained to be taken then, in early August, was with which team we would cooperate on that pilot. We were invited back in early September to discuss particulars and possible start dates. Little had evolved in the holiday month in between but the establishing of certain communication patterns. I was champing at the bit at this meeting, and it was here, is my own analysis, that the seeds of future failure were sown - if it had been chess this would have been my fatal error:

The team we were to work with were busy with the UAT phase of the current release of their product and their next development cycle was to start sometime in November. For various reasons, it was difficult to say when we could get the whole team together in the meantime such that we could discover of them what it was they did and how approximately (we had decided this should always be our first step when getting involved with a new team).

In the face of all this uncertainty about dates and states and next moves, I boldly suggested that we “just start!”. Wouter and I would hang our Basics workshop on a planning meeting and we could get learning in this UAT period, which for everybody but Test, wasn't actually that busy a period. We could do this at the drop of a hat, so why not? The team was unstable they warned us! My riposte was that we could get to know the team in (informal, relaxed) bilateral meetings after the first session, as the opportunities presented themselves. If we acted now, I asserted, the basic scrum cadence could be well established by the time the next development cycle started! (Privately, I also congratulated myself on how easy it was going to be in the UAT period to get the whole team thinking about testing as a whole-team activity)

How unstable could the team configuration be? All agreed.

As in chess, as in life - in chess too my downfall is often impatience, or hubris, depending on your perspective.

When we met (most of) the team on the morning of the Scrum Basics workshop, some of them for the first time, it was immediately obvious that the decision described above had caused some damage. The team felt crept up upon, and in some cases, put upon, the PO-elect among them.

And yet everybody pitched in positively, we had made no secret of the fact that we were new to the coaching side of things and the team generously allowed us some early fumbling, the workshop was a success. A sized sprint backlog (comprising outstanding UAT work as the team knew it at that moment) was produced and the scrum board setup and prepared. At the same time we had delivered most of our Scrum Basics slides. The perfection game feedback for the session averaged 8...

Lengthy sessions with the PO, the testers and the developers respectively took place during that first sprint. From our point of view it was essential to gain a better understanding of the domain, and the general technical and organisational parameters. But we also used these sessions to follow up on any questions or other input the team might have regarding specifically the adoption of scrum. After the fact it transpired that we had in some cases achieved the exact opposite of our aims, coming across at best, as hard of hearing, and at worst, as cerebrally challenged.

We had agreed to try and keep the number of open stories to lower than half the number of team members. This forced the team to search for new ways to cooperate during this first sprint and things seemed to go quite well initially, but eventually a lot of worked piled up in test anyway. All the skills necessary to deliver a story were available within the team but historically roles had been clearly demarcated so the team needed time to get to know one another's work. The sprint was successful though, twice as many story points were eventually delivered than that the team had committed to in the planning meeting!

On day n-2 of the sprint however, there had in fact only been one single story point actually delivered, DONE. Essentially the team had 'flatlined' all the way through the sprint and on the last day or two managed to get everything done and dusted. Some points were basically delivered through negotiation.

I had introduced the scrum master to Derby & Larsen's “Agile Retrospectives” and we discussed setting up the review and especially the retro along the lines as laid out by the “retrospective goddesses”. The stated goal of the retrospective, having dwelt briefly on the successful delivery, would be the production of one “improvement story”. In our case, a story that would start the team on the road to finishing with starting and starting to finish and give us the opportunity to discuss WIP limits and all that good stuff.

Surprisingly (or not) there was some resistance from the developers when it came to the demo. There wasn't much GUI work involved and everybody had already worked so closely together the whole sprint, telling each other, every day, what they were doing, on a mature project, etc. We insisted anyway and were quietly delighted when almost every single story demonstrated resulted in lively discussion.

The planning meeting for the second sprint under our guidance was less of a success. There was confusion concerning who should have done what to be prepared for the meeting and some of the team were in London while the meeting was being held. It was a difficult session but it clearly surfaced the necessity of excellent communication infrastructure when not co-located and more importantly; latent problems in understanding the scrum roles, and so was very useful. Again we produced a sized sprint backlog, including the improvement story that the retro of the previous day had yielded. But it was also becoming apparent just how badly I had underestimated what people had meant when they talked of team instability in the project phase (UAT) the team found itself in - during that meeting one of the team was called away such that he could put his development skills to good use on another project. And then there was our impending absence...

The week this 2^nd sprint was to end, Wouter and I would be away, firstly at LKBE11, and then CSD training with Ron Jeffries & Chet Hendrickson, so we tried to schedule dates with the PO and the Scrum Master for the two remaining workshops we wanted to give such that we wouldn't exceed the 4 sprint maximum we had agreed with management. We were also convinced of the value that they would deliver. That planning meeting would not have been so problematic if we had had the chance to cover backlog management.

It proved difficult to set a date for this (or any other) workshop over time. Not only was it tricky to get everyone together (as we had been warned it would be) there was also a difference of opinion as to what material could be used as input for these workshops, the team was suspicious of any information that had not been signed off and the PO was wary of re-work, it seemed our hands were tied. Wouter and I had long and involved, but ultimately unproductive, discussions about how to get which management to give what signal. The last day at the client's just prior to our planned absence, having failed again to set a date for either workshop, I looked back for the first time at my insistence that we start immediately with serious trepidation. The team clearly felt that there was no room for mistakes.

On the upside though, at that point in time there were three days to go in the second sprint (sprints ran from Thursday to Wednesday) and the scrum board had been looking okay. The following Wednesday evening I mailed the scrum master and asked how the sprint had run down, how she had got on with the review & retro and if she was ready for the planning meeting the next day, or if there were any problems, questions etc... I received an evasive answer explaining how the sprint had been completed successfully, and that the team had delivered many more stories than anticipated.

I surmised that the improvement story had not been done. That there had been neither a review nor a retrospective and that there would probably be no planning meeting the next day.

When I (Wouter was at the Scrum Gathering in London, the lucky lucky...) turned up at the office the following Monday my worst fears were confirmed. The scrum board was closed and abandoned. When I opened it I saw the remains of the previous sprint (with indeed many many post-its in DONE). The improvement story however had not been touched...

I asked the member of the team sitting closest by, “What's going on, how come we don't have a sprint backlog, weren't there any ceremonies run?” They had collectively decided that there was no point for the moment, the workload was too variable, more people had been moved off the team (that was also why the improvement story wasn't picked up in the end, the guy who was to pick it up had been moved...) and so on.

I sought and found our sponsor. She explained that there had been some delay in signing off the next release with their client and that they had therefore re-assigned as many people from the team as they could. (It was clearly not the right moment to re-open the discussion on the advantages of stable teams.) There was however no doubt that the release/project would go ahead. It was just going to start a little later, probably the end of November (it was the middle of October at that point). We decided to leave things for a while until there was more clarity. Then, we could come back and finish off our transition with the hopefully largely re-grouped team. Precisely how we would start up again at a later date, we would leave up to the circumstances and how much material we still needed to cover.

In the mean time, the next day, we (our sponsor and I) ran an impromptu retrospective of the pilot transition. We wanted to gather feedback both on scrum, and the coaching. I was beating myself up about how badly we must have done our job - the first chance that had presented itself, everything had been ditched, by everybody!

Upward and onward, the retrospective itself went very well. Using the timeline activity we gathered data and emotions. We then dot-voted to bring some structure to our reflections. Some difficult discussions ensued but we all stuck to the task and reviewed events and each other honestly. The upshot of it all was that we coaches needed to be much more aware of our customer's narrative, where they were coming from if you will, and that the team needed to be more open to change, and dare to challenge their managers to change. I left that day much happier than when I had arrived, as far as I could tell, despite the problems, the team were still onside:

I was all full of the value of failure, I tweeted as follows:

And then fate struck. It's not just bad decisions that are required for failure, you must also have some bad luck along the way. A herniated disc in my lower back laid me low as of late October.

Wouter returned to our client's on his own in early November and they agreed that we would get a call the week before the re-scheduled project was due to launch (as of that moment, the week of the 21^st).

In so far as I was able (treatment for herniated disk is mind-bending painkillers & rest) I helped Wouter to turn our Backlog Management and Planning & Estimating workshops into a sort of project kick-off built on the framework of Rasmusson's Inception Deck so that we were ready for any eventuality.

Meanwhile, time ticked on, and on. And the promised phone call never came.

When contact eventually was established, we heard that “everything was a go. Somebody just needed to check back with somebody else...” Now I know my project manager's instinct would have had me knocking down the door at the our customer's long before their deadlines for calling us had passed, but would it have made any difference? Ultimately? I don't think so, as I said, I think a little patience on my behalf, way back in September, would have served us all well!

Anyway, another 4 days went by, and then we were told that our services were no longer required. The project was up and running, what was the point? The rest of the teams had also decided to transition under their own steam. We had missed the boat.

And so, the balance.

On the negative side:

• We did not successfully sell agile and I am concerned for our customer's adoption
• We did not complete the pilot transition
• We ran only one billable workshop
• We did not test our new workshops
• Direct return on investment was negative

And on the positive side, the value:

• We now have a thoroughly tested Scrum Basics workshops, which we can execute on various frameworks (retrospective, planning meeting)
• We have working experience as (external) coaches and have sent an invoice
• We completed our Planning & Estimating workshop
• We added the bones of a Project Kick-off workshop (framework inception deck) to our portfolio
• Directly or indirectly, our work there was the stuff of a number of  blog posts!
• Learning, learning, learning...

It is with regret that I share with you

that Jack is momentarily 

a little less nimble than is his wont.

In fact,

Jack slipped a disk

while jumping that damned candlestick.

I had been working on a post concerning feedback and chaos and strange loops and... Until I am able to work at length again, please do enjoy the Mandelbrot set:

Lean and Kanban 2011 BE - A narrative

The day before the Lean & Kanban 2011 Benelux conference started I was still undecided as to whether I should head for Antwerp that evening as planned or take an early train the next morning (my Sunday was proving to be busier than I had envisaged). I eventually plumbed for the riskier option of the early train (departure 6 am), which was possible because my colleague had gone to Antwerp as planned and secured the room in our hotel. I had unwittingly set a marker for myself; early rises and mental gymnastics would define my week. I had also exercised real options.

I arrived in Antwerp as scheduled. Having had just enough time to catch up with my colleague, register and marvel at the industrial aesthetics of the revamped hangar where the conference was taking place, I settled in for the first keynote: “Rethinking Deming” by Don Reinertsen. Given the context, it seemed that someone was spoiling for a fight. Those of us who enjoy (intellectual) confrontation, especially when peppered with rhetorical flourish, were not disappointed. Not knowing Deming (or statistics) as well as I might I took the whole thing at face value; food for thought. I did however enjoy his trashing of the red bead game (“an entertaining con”) and identified with some of his reservations about the deeper (“terrifying”) implications of some elements of systems thinking (the well-designed system aims to totally emasculate the individual). Reinertsen also introduced the idea of random drift – an accumulation of (random) events affects events in the future (by changing the system through incremental feedback) – I'm sure if this idea was developed, chaos, tipping points and complexity theory generally would quickly come into play.

For the rest of the morning I followed a case study. Three different speakers (or speaker pairs) took us through their experiences with specific transitions which they had guided or were busy guiding. This session delivered plenty of practical know-how and it was followed by a lively discussion. Although I had hoped to pick up some pointers on dealing with that most difficult element of change processes – the sceptic - we didn't get much farther than a version of “build it and they will come”. Which is fine, but it introduces a chicken-an-egg problem; how to demonstrate without investing?

After lunch on day one I followed the Real Options and BDD/Testing track. The notion of Real Options appeals to me, as do its cousins set based design (which Michael Kennedy talked on on day 2) and BDD. Here again however I was left wondering how to convince the sceptical that set based design/real options was worth a try as an economically sound way to realise value (when not given the opportunity to actually demonstrate).

As if in tune with my concerns generally on aids to persuasion, Mark Robinson brought the afternoon to a close with a refreshingly old-style presentation in which he used an ingeniously simple spreadsheet to demonstrate the effects of various WIP limits on throughput. Limiting WIP is anathema to many managers as they think it will reduce their efficiency, which it might, but I feel its efficacy we should be after and not total(ly illusory) efficiency.

Dave Snowden delivered the afternoon keynote. The gist of his talk (and I do his erudition, learning and sense of humour no justice by summarising it thus) was that context should be instrumental in determining appropriate action and that only well understood theory could help us decide what to do when faced with varying contexts in practice. Snowden continued in the combative spirit displayed by Reinertsen in the morning keynote by energetically stepping on some absolutist toes, claiming for instance that systems thinking was Talyorism by another name. His message was that kanban is not Truth and neither is Lean, nor is it Scrum, or anything else for that matter; context determines truth. Whereby Snowden of course cleverly laid claim to the Truth (There is no Truth, only truths). I was enjoying myself thoroughly!

On day 2 Alan Shalloway's opening keynote was also informed by the tension between the individual and the system. For the sake of argument he had chosen to state that it was “all about the people” but the point that he was actually making was that it is all about the interplay of individuals and the system which they inhabit/cause to emerge. Complex systems and their constituents co-evolve. Matter is both particle and wave. Man is both nature and nurture. It's all about the feedback.

After lunch I attended Bob Marshall & Grant Rule's double session on understanding effectiveness and the possibilities that Rightshifting delivers for managing same. This was all so completely new to me that I won't trouble you with half formulated thoughts on it. Suffice it to say that I am busy studying the Marshall model and hungrily consuming anything I can find on chaordic organisations.

And then there was Jim Benson, what a delight this guy was! He's allergic to powerpoint and delivered his utterly engaging and frequently funny talk strolling back and forth between two flip-charts; one was his kanban and the other was his whiteboard so to speak. As he spoke and wrote and drew, he moved sticky notes appropriately on his planning chart thereby breezily guiding his impressive delivery while keeping his process & progress transparent to all. It was a virtuoso performance in the application of kanban. The subject matter was substantively different to that which had come before too; we were talking the psychology of kanban.

I liked Benson's idea of existential overload. As I mentioned, my week was to be one of short nights and heavy cerebral exercise - my 2 days at LKBE were followed up by three days of CSD training with Ron Jeffries and Chet Hendrickson. At one point during that training Ron, in explaining one of the positive side effects of test driven development, told a story of how TDD had effected his wife's (!) life for the better. Ron, as he was sure many of the programmers in the room were too, was smart enough to keep many things in his head at the one time, and be dealing with many issues simultaneously, and investigating a few options in different areas from various perspectives all at the same time while also wondering how he might approach that other problem but that if his wife came to ask him what he wanted to eat for dinner that evening at that precise moment, that he would suddenly yell at her at the top of his voice that he was damn well THINKING, HOW COULD he know what he wanted to have for dinner?!?!?!? TDD allowed Ron to deal with things; start something, develop it, finish it. And so on. It helped him keep a clean, clear mind, with plenty of room for dealing with unexpected pleasantries like an invitation to dinner from his loving spouse.

On the other hand, I was troubled somewhat by another phenomenon that Benson mentioned; the Zeigarnik-effect. According to studies conducted by Bluma Zeigarnik we retain memories of unfinished work better than we retain memories of that which we have brought to completion. That strikes me in a way as a slightly depressing finding. If completion is equivalent to success and that which is unfinished represents failure, and we remember failure at the expense of success, then we're doomed to be pretty miserable animals aren't we? But if that's what the science says... Luckily there is somereason to doubt the veracity of Zeigarnik's findings but that doesn't take away from the general guideline; getting stuff done feels good.

Most interestingly, in the light of the undercurrent of (largely good natured?) tribal nettle throughout the conference, Benson spoke about methodologies as liberation theologies: Our life sucks because we're oppressed by some methodology-or-other. But, then some-NEW-methodology-or-other comes along and liberates us, we feel good because we are free and we decide some-methodology-or-other is okay. But as time goes on, we get all caught up in the rules, aren't going anywhere and are feeling all oppressed again. He called it the cycle of Liberation, Redemption & Addiction, which he laid on top of the story arc; Birth, Life & Resolution, which he laid on top of the kanban lanes; Ready, In-Progress & Done. It was all getting very universal.

John Seddon - “It's the system stupid!” - delivered the conference's closing keynote. He came to wipe the floor with everybody who had gone before and did that with an abrasive elan. Reinertsen had upset him, Snowden was full of it, other systems thinkers hated him. He really didn't care because he knew nothing about software development except this one thing; if you did as he said you should, you would do it better! What he said was, "Study, study, study!"

Surely safe-to-fail probes, set-based design and real options, OODA loops, PDCA cycles, and/or study, study, study are all variations on the theme of “inspect & adapt”. In order to inspect meaningfully and adapt successfully, systems must be extended incrementally via short iterations - it's all about the feedback stupid!

There will always be a better way to do things.

The trouble with success

This week the first sprint of the pilot transition at our new client was completed successfully. Kudos to the team, if they learned as much about scrum as I learned about coaching then things will turn out alright!

Before embarking on this transition I had imagined (in my wisdom!) that teams transitioning to scrum (or any flavour of agile) would generally be coming from either chaos or toxic and petrified waterfall, so that anybody in an executive role (as opposed to management or admin) would be only too happy to try something new. 

But what about a start situation wherein success is the problem, so to speak? The team we are working with is part of a young company which has grown so rapidly in the last twelve months that it's quite literally bursting at the seams; space in the office is at an absolute premium.

Initially I thought that this would make the job of coaching a transition to agile that bit easier – especially when it became apparent that the existing processes had evolved such that they included some obviously agile elements already. For example, scope was derived from goals via a series of workshops involving the client, this scoping phase was followed by a definition phase, which included (high level) requirements gathering, and subsequent to that, a cycle of design/development iterations was entered into. Finally there was an acceptance & delivery phase whereby a portion of the team would be on-site at the client. Releases tended to fall into the three to six month range and teams were organised by client/project and not by function.

Sure it would be just be a case of crossing the t's and dotting the i's and they would be equipped to deal with phenomenal growth for years to come. If nothing else, in the past two weeks I've learned that naiveté is a good learning stance!

To start with, there is the 80/20 rule: The last 20% of any job will cost 80% of the effort. To take one example from the list above – a feature team is not yet a feature team simply by its comprising all the skills required to deliver a feature; if responsibilities are still strictly demarcated by role then you retain silos, queues and handoff, and it remains exceedingly difficult to genuinely work as a team according to priority (even with the best will in the world).

So, the first sprint has been completed and all the stories have been delivered, in fact, stories were added during the sprint and still the entire scope was delivered. The burndown flat-lined all the way to the second last day of the sprint however and on that last day there was a sheer drop all the way to zero – a typical indicator that the sprint was in fact a mini-waterfall. This posed a challenge for me in setting up the retrospective. I wanted to ensure that the team enjoyed its success and that the people took the time to recognise their achievement, yet I felt it was imperative to discuss the risks (getting a lot of work done without delivering anything) inherent in the approach that had led to the burndown profile described.

I started by focussing explicitly on the successful delivery and then defined the retrospective goal as generating at least one improvement story (for inclusion in the next sprint backlog) which would help the team along the road to delivering sprint scope story by completed story. As you might expect, resistance to the suggestion that there had been anything 'wrong' was strong. Discussion was energetic and I was repeatedly faced with 'the way things are done here'.

“The way things are done here” is a notoriously difficult notion to challenge at the best of times - scrum is incomplete and adaptive, every implementation of scrum is unique and it is most certainly so that some things are best done in a particular way in a particular situation. That said, much of “the way things are done here” during the early stages of a transition is just general resistance to change.

If most of the people are unhappy and the projects are generally disastrous then “the way things are done here” can be rejected relatively easily as obviously the wrong way to do things. If however your team is motivated and successful such that projects are generally delivered on time (even if stress levels do sky-rocket as delivery nears) dealing with the “way things are done here” requires careful thought, concentrated listening and skilled discussion.

I'm working on it!

A commuter's rant

This morning was one of those mornings; when the alarm clock went off, I felt like throwing it against the wall. Instead I turned it off and lay back down so that I could ease into the day. I promptly fell back to sleep and woke with an even bigger start some twenty minutes later. I let my colleagues know that I would be late.

As we know random events cluster – today they clustered at a railway near me!

I caught a later train and as it rolled to a halt at its first port of call, the conductor announced that we would go no further as there had been a crash further up the line. The advice proffered, which I duly took, was to return to the station from which we had just come and follow an alternate route. Half an hour after I had left my local train station I was back there waiting on an overcrowded platform for the next train to Den Haag. As the scheduled departure time came and went without any sign of our train, and the crowd of unhappy commuters continued to swell, it was announced that there was a problem with a switch near one of the main stations on our route. I rang the office and let them know that I would be working from home...

My commute normally consists of a ten to twelve minute walk, a train journey of some 46 minutes and a tram ride of not quite 15 minutes. Most days I can get to work in about 90 minutes. The journey home is less predictable because it starts with a tram journey – tram schedules are not strict – and can take up to, and on the odd occasion more than, two hours (the difference usually made up almost entirely of waiting).

Why don't I drive? Well, I don't have (nor have I ever had) a driver's licence. Besides that pertinent impediment, there are the issues of gridlock, congestion, pollution and carbon emissions. But I will readily admit that if the only problem were gridlock I would without doubt have converted to the car long ago. Travelling by car is generally quicker and much more fault tolerant - if you step into your car 5 minutes late at the end of the day, chances are you will be home 5 minutes later than normal. If you leave the office 5 minutes late on the way to catch a train, your journey home may end up being extended by a half an hour or more.

If even well organised and largely dependable public transport (as the Dutch system surely is) results in significant delays at least a few times a month and the freedom that travel by car offers all too often equates to the freedom to sit idle in a traffic jam, what can we do to keep things moving?

There are a few obvious stop gap measures; teleworking may help somewhat in combating congestion and should be considered a useful tool but its efficacy is limited as teleworking is detrimental to team work. Flexible working hours could reduce pressure on transport systems during peak hours but any potential tends to be nullified by the demands of the rest of our lives (opening hours for crèche, school etc. may not be flexible) and here again, the issues that flexible hours raise when working in a team are not trivial. Carpooling is another option but it shares many of the drawbacks of public transport and can be socially taxing to boot.

Traffic needs to flow, maybe there are answers to be found in queueing theory. Take my journey as sketched above; I tend to leave myself  13 or 14 minutes to get to the station in the morning even though I know it generally takes around 10 minutes to walk the short distance required. I have to build in slack – if I aim at arriving on the platform as the train pulls in, an extra long wait at a traffic light might be the difference between happily catching the train and being left on the platform panting and cursing under my breath as I fumble for my phone, my connection riding into the distance. The greater the potential for variation in my journey to the station the more slack I'm likely to build in to allow for the unforeseen. Hand-offs are expensive.

More frequent trains would alleviate my problem somewhat – I wouldn't be as worried about missing a train if I could catch another less than 5 minutes later. The rail-road has limited capacity however and a large increase in the number of trains is not easily realised – unless the trains were significantly shorter on average (smaller batches). Thereafter cycle cost needs to be considered; the ratio of manpower & material to customer-kilometres would limit the extent to which batch sizes could be reduced.

If we were to remove hand-off and radically minimise batch size, say to one traveller, would we not simply arrive at travel by car? Yes. But. We still need to consider WIP - Work In Progress; allowing your WIP to rise above certain levels will be detrimental to throughput, alternatively; maximising the use of capacity (efficiency) will, after a certain threshold, reduce flow. If you get out on to the motorway on time in the morning you can whiz on through to your destination. If you're that little bit later, the road is chock-a-block and although capacity is being used efficiently (no. of cars per m² of road), you will not be going anywhere in a hurry.

In short, if we view travel by public transport as the waterfall option (silos, hand-off, long queues, large batches) and travel by car (as we know it) as the chaos or cowboy option, does that offer up any agile solutions to the traffic problems of the 21^st century? Possibly. If there were a system put in place to ensure that the capacity of the motorway was never utilised beyond around 70%, the issue of gridlock could be so atomised as to no longer exist. Cars would be allowed onto the motorway via many slipways, managed centrally by a  control system of traffic lights which would monitor and manipulate (motorway) utilisation, queue length (on the slipways) and batch size (entering traffic). In that way the motorway could actively pull traffic as opposed to playing the proverbial gullet, packed so tight that it can only choke...

To complete the agile/lean allegory; there is even evidence that self-organising traffic is the safest, although I must admit that I'm not sure I'd like to be the one testing that hypothesis on a high-speed motorway.

The power of play

These days, more a coach than a player, I've been thinking not only about scrum, agile and all the rest but how best to communicate on these topics and how best to generate and/or impart knowledge generally. The power of scrum lies in self-organisation and that complicates the coaching role; even if it were possible, there is no value in telling your team from moment to moment how they should tackle the challenges they will face along the way – you need to get people on the right track and help them develop whatever skills or practices are necessary to stay on that path.

In this regard I am beginning to understand the potential power of play. For instance, Innovation Games can be a powerful tool in getting those mental cogs a-grinding. The sail boat (variation on the speed boat) exercise can be used both for envisioning risks & leverage at the start of a project and as a tool to facilitate retrospectives. Formulating an elevator pitch or designing a product box are also great ways to generate collective focus and get the creative juices flowing.

In developing my coaching skills (and our service generally) I am lucky to be able to interface with a product development group here at our own company; the Qafé team. If I want to try out any ideas that I come across, and the Qafé team and I can identify some reciprocal advantage, then we collaborate accordingly. For instance, having read Derby & Larsen's excellent Agile Retrospectives, I asked the team if I could facilitate their sprint retrospectives for a set period. I gained through practical experience and the team gained through the generally refreshing approach to retrospectives.

We (Wouter and I) also 'tested' our user story (splitting) workshop on the Qafé team. Apart from all the excellent feedback that we got on the workshop itself we also learned that the Qafé team was having a problem relating to potential customers. Strange as it may seem, it is not at all unusual that developers do not explicitly think about who their users might be and what they might expect from the software being built. I decided to run a pared down version of Jonathan Rasmusson's Inception Deck exercise – even though Qafé was long past inception, in fact it is already in production and about to take off in a big way, more of that in a moment – to focus the development team's mind on the why of their daily what.

The inception deck is a technique for envisioning projects which comprises, among other things, several popular innovation games. Some elements of the inception deck were not applicable given the timing of our exercise but in a little under 2 and a half hours we went through Why are we here?, Create an elevator pitch, Design a product box, Show solution and What keeps us up at night? It proved to be a very valuable exercise and thereafter the team was generally much more focussed on who they were building for and why.

Some weeks later the company CEO arrived in the team room with good news and a serious challenge. Several large integrators had voiced an interest in Qafé and had undertaken to invest in some training and, assuming the product matched the potential of the idea behind it, to add Qafé to their list of supported/preferred development frameworks for enterprise systems. “But,” warned the CEO, “realise that this is make or break, if these guys don't like the system, that's it, you won't get another chance and the bad publicity will kill the product before it takes off. How confident are you that the product is ready, that it's stable enough?”.

The answer was a less than resounding “Quite”.

Don't get me wrong; this team is confident that they have a good product, they know that it's well conceived, designed and executed. The problem lies in the limited feedback they have been able to generate up until now – several enterprise applications have been built (by a sister organisation) using Qafé, each one resulting in invaluable feedback in the form of bug reports, feature requests etc. but the team might have been more comfortable with one or two more implementations under their belt before being faced with such a make or break opportunity.

The PO jokingly recalled the inception deck exercise, “Do you still wonder what keeps us up at night?” he asked of me. Suddenly inspired, he instructed the team to think on what would keep them up at night in the specific context of the news they had just heard. At the following planning meeting, with a focus on the release to the big vendors, the team detailed what their greatest concerns were. These were then collated and prioritised. Starting at the top of the list the team translated their concerns into actions and so determined, in cooperation with the PO, what their product backlog should be for the coming period. Sprints were reduced from 2 weeks to 1 and the team has gone about methodically addressing and eliminating the risks identified. 

Confidence has soared.

The virtues (or otherwise) of feedback

Agile methodologies are successful because they are empirical, incremental and iterative; every action is examined and feedback is generated such that improvement is attained by taking small refining steps. Running scrum with XP generates feedback (TDD, pairing) as often as every few minutes!

It is abundantly clear to me why this results in great software better aligned with the customer's needs. However, in a recent conversation on the merits of frequent (structured) feedback I decided to play the devil's advocate and asked if it weren't possible that frequent feedback might sometimes in fact stymie some of the very best minds. This suggestion was met with appropriate derision (yes, it was a conversation among 'the converted').

I wasn't to be put off that easily however, what about Van Gogh I asked? He never sold a single a painting while he was alive, and when he paired with Gauguin it ended in tears. If Van Gogh had heeded the feedback he received on the fruits of his artistic endeavours he would have given up long before he got to Arles and the world would have been a poorer place for it. Art is not science was the rejoinder. There is a science to art though, and an art to science.

What about music then, it's still art but art with an explicitly mathematical character, how do great musicians deal with feedback? Beethoven was stone deaf by the time he wrote his majestic 9^th symphony, a symphony which to this day sets the standard for all other symphonies. When Stravinsky's Sacre du Printemps was first performed some ninety years later there was a riot! The composer (and the choreographer, who was also a touch too modern for the tastes of Paris' beau monde) were nearly run out of town. The press was almost unanimous in its condemnation of this newfangled noise and the weird dancing that went with it. These days, Sacre du Printemps is regarded as one of the masterpieces of twentieth century music. As for Schoenberg...

I wasn't convincing however, art was far too subjective (we hadn't even touched on literature yet), and that's exactly why creative geeks get into software; a bit is either on or off. Feedback is always good. And Schoenberg really is unadulterated noise!

However, new research would seem to indicate that I may not have been entirely misguided in my playful challenging of the virtues of feedback. Creative ideas tend to generate resistance, even among people explicitly seeking innovative breakthroughs, and regardless of objective evidence supporting the new idea!

So, when is negative feedback a sign that you need to change your ways and when is it a sign that you are most definitely on the right track? There is no simple answer to that question. As a rule I would err on the side of responding to feedback at face value but try and allow for the occasions when feedback says one thing but means another. In fact, this is a specific instance of a generic problem with all process guidelines, and a topic of hot debate in the scrum community;

• When does a product owner challenging a team become a manager pressuring the team?
• When does guarding the process become procedural nit-picking?
• When does positive conflict become damaging mudslinging?
• When does the avant garde represent the emperor’s new clothes?
• Etc.

In all cases the answer is (a sometimes seriously unsatisfying), “it depends”. And that is the art to this science of software development.

Fixing to get flexible

When adopting scrum, one of the more subtly difficult issues that can arise, derives from the varying expectations generated by the word agile. Customers who are used to long lead times and cumbersome procedures are delighted at the prospect of 'going agile' – they will get to call the shots and there will be no penalty for change. This lightweight interpretation of agile, inspired by a craving for flexibility, can cause problems when customers decide mid-sprint that they need something new, tomorrow.

When faced with the situation outlined above, the scrum master will of course state that the customer needs to talk to the PO and that, in all likelihood, (some portion of) the new functionality will be delivered no earlier than at the end of the next sprint (which could be up to 4 weeks hence!). The customer's response will often range from disappointment to disillusion; what's so agile about that?

Or, what to do as freshly minted scrum master when the sprint ends and the last story or two (!) on the sprint backlog are not done? The freshly minted PO, who is also still learning, and his customer base are pushing hard for an extension to the sprint – surely adding a day or two to the sprint is better than delaying delivery of the story in question for (at least) another whole sprint; where is the agility in (such) strict time-boxing?

Management might also react negatively when it transpires that agility does not involve people hopping nimbly from one team to the other as the (perceived) need arises. How to ensure efficient use of resources; what's so agile about all this rigidity?

So, how should a scrum master explain to his or her environment that a measure of intransigence is in fact conducive to flexibility?

They might point out that four weeks is still significantly shorter than 12 or 26 weeks but I wouldn't recommend it.

They could delve into the theory of complex adaptive systems and explain how scrum is an astutely chosen set of boundaries (constraints; time-boxes, rules etc) imposed on the behaviour of a complex adaptive system (team) catalysed by the probe of certain types of information (user stories, feedback).

Or they could point out that scrum's basic measure of progress and unit of prediction, velocity, requires fixed sprint length and fixed team composition to have any meaning.

Otherwise they might pontificate on the magic of cadence and appeal to their listeners' sense of (universal) rhythm. Or they could point out that if the team does not have to constantly busy itself with a fluid agenda that they can concentrate on what really matters; delivering working software that the customer actually wants with astonishing regularity.

For the PO, whose time is also usually at a premium, stable sprint configuration means that he or she knows months in advance what possible delivery dates are, but also when they need to be in the office as planning meetings, grooming sessions and reviews take place in the same locale, at the same time & for the same duration and on the same day of the week, sprint in, sprint out.

If all else fails, a story might help; it is often said that Einstein's wardrobe was filled with identical suits because he felt that (mental) energy expended on deciding what to wear was energy wasted. As with much of the myth of Einstein it is difficult to establish the veracity of this anecdote but recent research indicates that decision fatigue is indeed a force to be reckoned with. Developers, scrum masters, POs, line managers and customers will surely all agree that a team's (decision) energy quotient is far better spent on coding questions than on deliberations as to the best time for this week's grooming session.

What scrum is (not) – a meme

Although it's difficult to test or otherwise corroborate in any way, I have the feeling that 'the agile community' has been trending negatively over the last couple of weeks. At the very least there has been much to-do in my personal infosphere on the evils of agile (aficionados). The debate as to what agile, and specifically scrum, is seems to have been revived via a couple of slightly bad tempered posts by Ken Schwaber. Even well-loved fixtures of scrum lore like the chicken and the pig are coming under fire.

A framework that has inspect&adapt as one of its central tenets should obviously itself be under constant review, but what surprises me about some of the negative utterances I have come across recently is the vehemence with which they are delivered. Onediatribe in particular caught my attention, not only because (in a discussion of scrum) it ranged – in a series of three articles - over politics, (social) history and religion (subjects close to my heart), but most especially because of its toxicity. This guy really has a bee in his bonnet. His central metaphor, of priests and followers, is not without its worth though, and the unthinking uncomprehending dogmatists that seemed to have riled him so are definitely not a figment of his imagination – I have met them too. As for the parasitic consultants; I don't doubt they're out there either.

However.

Lambasting scrum as a religion, when it is explicitly constructed on empiricism, is a bit like creationists claiming that the theory of evolution is just one more belief system.

One of the issues that came up in discussions engendered by said article (also re-posted to LinkedIn) was the Scrum Alliance and its certification (specifically the Certified Scrum Master). I do not want to get into a discussion on the Scrum Alliance but I will defend the CSM, at least as delivered by Jeff Sutherland; not that everybody who attends the course comes away a good scrum practitioner but then again not everybody with a university degree in computer science is a good programmer.

Two days is enough time to get through the basics of scrum, scrum is simple and incomplete; that's the beauty of it. More importantly, Sutherland starts by introducing scrum as, above all else, a frame of mind; the way. He then goes on to explain shu ha ri and states that although it is good to follow the rules to the letter initially, gaining understanding is the goal and that once that goal has been attained the rules can be left for what they are; useful guidelines. He also explicitly includes XP and Lean as means to maximising the efficacy of scrum. Scrum as taught by Sutherland is therefore anything but an exclusive collection of dogma and rites. 

Further; the belief that agile development equates to no documentation, no planning and no commitments abounds. I have met teams that claimed to be running scrum when sprints were not fixed-length, team composition was wildly variable, random managers could change the 'sprint backlog' whenever it suited them, and most importantly, delivery of working software was haphazard at best. I have known team leads and scrum masters who 'itched' at the idea that 'their' charges were not all fully allocated – that their team was thus inefficient – and therefore assumed responsibility for sprint planning. With so much confusion out there some attempt at certification is absolutely necessary. If someone is a CSM then you know at least that they have been exposed to a version of scrum that will work if applied as learned.

But the CSM is not enough, Sutherland & Schwaber on the topic: certifications do not guarantee excellence

That is to say; scrum is a meme. And just as with the its physical cousin the gene, it is as good as impossible to make a perfect copy of a meme. Everybody who has ever learned anything about scrum has their very own unique scrum meme, some of which will be instantly recognisable as incomplete or otherwise faulty, some of which will be more difficult to assess, unless you test them. In that regard, it could be a good idea to view certification as a necessary, but not a sufficient, condition for recognising the owner of a good scrum meme. An agile hiring process could be even more useful – new scrum masters are hired initially on a one or two month contract, their performance (basic understanding, facilitation & communication skills, removal of impediments) can then be tested over a small number of sprints whereupon the team can decide whether the new scrum master is the scrum master they have been looking for.

I sincerely hope that the agile community succeeds in keeping its conflicts positive.

Agile contracting (testing, testing, testing)

As you may have noticed dear reader, I have been less regular of late. Holidays can do that to a body. Holidays or writer's block!

Recently I have embarked on a journey of discovery in the world of (automated) testing. A wonderful world it is too. I've subscribed to magazines, and read my Crispin & Gregory, I've discovered James Bach and Cem Kaner and Ward Cunningham, I've learned about Dan North's BDD and Liz Keogh, and that people can be test-obsessed. I've even joined a tester's social network. My appetite has been whetted and I'm keen to find out about specification by example.

Along the way, I've also been getting down and dirty with Java JDKs, Eclipse this and Eclipse that, Subversion and Maven and Sonar and Hudson, Java script, GWT, Flex, and Selenium. Hopefully I can add Fitnesse to that list shortly too. And then there's test theory; model testing, domain testing, permutations & combinations, set theory and so on... I, most recently and for many years now a project manager, am decidedly uncomfortable!

I thought all this would at least yield a couple of good blog posts (continuous improvement?). Fat chance. To-date at any rate.

In other news - our first assignment to coach a transition was confirmed today. Wouter and I are completely new to the acquisition side of things but we thought agile and guess what; it works for contracting too!

Agile contracting

We sat down together to explore possibilities	–	we didn't send Sales to see Purchasing
We ran one of our workshops on-site	–	we didn't pitch
We investigated opportunities for mutual benefit	–	we didn't negotiate terms up-front
We agreed on a framework for co-operation	–	we didn't write statements of work

    -o-

The actual contract to come out of all of this will incorporate change-for-free & money-for-nothing type clauses. 

I'm sure I will be back to my usual prolix self the next time we meet!

Introduction to scrum (A retrospective)

A few weeks ago I wrote about how our first workshop came about and what we had learned from it. As luck would have it, the same day that we ran that first workshop, I received a message from an ex-colleague of mine, Leonie, who was working as a business analyst at Second Floor. This small company is dealing with the difficulties of success – growth is so strong that their existing processes, which have worked well up to now, are beginning to hinder progress. One of the things they were considering in an effort to manage their success, was a transition to scrum (which some teams had already partially implemented) and they were looking for project managers versed in same.

Although I had to disappoint Leonie in as much as I am not for hire as a project manager we quickly agreed that there was potential for mutual benefit in our respective situations; Qualogy could assist Second Floor in a transition to scrum, while Second Floor could help Qualogy through structured feedback on our expanding workshop portfolio.

A meeting was arranged to discuss possibilities. We agreed, with the delivery manager and technical lead at Second Floor, to kick-start possible further cooperation with an afternoon workshop as a general introduction to scrum. If that delivered enough learning and was generally well received we would talk about collaborating for the entire transition. At that point we had already agreed in broad terms on what a transition would look like; it would progress team per team (teams already comprised all the skills required to deliver working software) and would comprise a series of workshops and some targeted coaching.

First things first though; we needed to create a “Scrum basics” workshop that would leave a bunch of very smart, semi-initiated (in terms of scrum and agile) and fully dedicated professionals with an appetite for more. From our previous experience we were very aware of the need to run any workshop as a series of broadcast-interaction cycles, but we only had a half a day for the session and we wanted to ensure that the basics were thoroughly covered. The urge to take the easy way out and create a presentation (broadcast) with a high wow-factor (assuming we could do such a thing) was strong!

Our discussion with the delivery manager and the technical lead at Second Floor had indicated that there were some obvious (and relatively inexpensive) opportunities for process improvement, quick wins if you like. We were considering the possibility of creating hooks in our presentation-to-be for these basic improvement experiments (e.g. keeping sprint length stable) when Wouter made the inspired suggestion of setting the workshop up along the lines of a retrospective. The fact remained that it would not be trivial to involve our audience actively enough while ensuring that we covered the basics, but a retrospective framework (we used Derby & Larsen's Set the Scene-Data Gathering-Generating Insights-Decide What to Do-Wrap) provided an excellent starting point for attempting just that.

We set the scene in a half an hour by introducing ourselves, relating how the workshop had come about and how things might develop thereafter. We explained how the agenda had been set up, how we intended to make the material covered as relevant as possible to Second Floor, and how we wanted to garner feedback throughout the afternoon. This was followed by a short broadcast including definitions and an extremely brief history of both scrum and agile.

To gather data, we used a (familiar) brainstorm; we asked everybody to think about the process at Second Floor and to organise their thoughts by writing down (on sticky notes) what they felt was good about their process, what was bad and what could be improved. Notes were added to a prepared flip chart in the relevant category. We then asked the group to collate and de-duplicate the notes in the various sections. Using dot voting we generated a shared view of what was most important to keep doing, and what needed changing most urgently.

Then followed three Generating Insights/Decide What To Do cycles. Each cycle was built around one of the scrum roles whereby a short broadcast was followed by some Q&A and finally an explicit effort to connect some of the material just covered with specific points raised via the brainstorm. Envisioning how exactly we would make that explicit connection beforehand was difficult; we obviously we did not know what would come out of the brainstorm. On the other hand we knew that there were some quick wins to be had. At the same time, we wanted to avoid obviously guiding people to pre-defined conclusions using leading questions.

Eventually we decided to formulate a list of quick wins per role/section as a list of Try.../Avoid... pairs (à la Larman & Vodde) and play it by ear. If during discussion following a given broadcast, an obvious opportunity to make a link to the Try.../Avoid... pairs presented itself we would capitalise on it. Also, and alternatively, if discussion stalled we would introduce the appropriate Try.../Avoid... pairs as a catalyst for conversation. Organising the information according to the scrum roles worked well; it provided a natural framework for covering required ground on artefacts, ceremonies and rules while simultaneously detailing varying perspectives. However, in spite of (or maybe precisely because of) energetic discussion with the group, we struggled to connect to the results of the brainstorm in the fashion that we had hoped.

We wound the workshop up with a retrospective. We had chosen a combination of the Happiness Metric and the Perfection Game as feedback mechanisms. Feedback from the Perfection Game was again very useful, if challenging. Positive feedback regarding the brainstorm and the general structure of the workshop was tempered by the fact that we had struggled to explicitly incorporate the results of the brainstorm into further proceedings. This was not lost on our audience who complained via the Perfection Game of occasionally missing the relevance to their own situation.

Another interesting aspect of the feedback from this session was that it seemed to contradict itself at times. While some participants felt that we had spent too much time on certain basics, others complained that we had raced through concepts that were not yet familiar. The Dreyfus Model (a modern variant of Shu Ha Ri if you will) provides insight here, although not necessarily a neatly packaged solution! The Dreyfus Model works on the premise that a person's level of training or expertise on a given subject will dictate the best way for them to gain further proficiency in that field. In the extreme that means that the kind of instruction best suited to beginners (context-free instruction to be followed to the letter) is useless, even potentially damaging, to experts honing their intuition. And vice versa. As yet, we are not decided on how to deal with this insight.

This workshop was important for us on several levels. Particularly satisfying is that fact that we have added another increment (Scrum Basics workshop) to our product (Agile Coaching service) while refining (that is; iteratively improving on) already existing elements of same on the basis of (potential) customer feedback. We are hoping that our extremely collaborative approach will serve the double function of setting us apart in an increasingly competitive market while also ensuring that our customers get what they want, much as that may change along the way.