A few years ago, I worked as a service manager at Basefarm, a European managed services provider. I was part of a team supporting customers with infrastructure and managed services.
One of our customers was TV4, the largest commercial TV company in Sweden. As part of our agreement, the four engineers in our team would dedicate 400 hours per month to TV4. The client expressed a simple but irritating problem: They always seemed waiting for us to implement the changes they wanted.
Their development team felt we were the bottleneck of their delivery system. Our team, on the other hand, was feeling stressed. We felt we could never finish anything, but had to jump between customers and projects and constant firefighting. There was no growth, as everyone was working on incoming tickets all the time. To say the least, no one was happy.
So we decided to establish a more collaborative relationship with the customer's development team. Let me explain how we did it—and what we learned as a result.
The problem
The customer developed most of its products in house. When TV4 wanted to release something new, the process for them to get the new code into the staging and production environment was slow. The lead times were too long for them (especially in their competitive market). They wanted to get updates out quickly, but after they shipped the code to us and created a ticket requesting we push it into staging or production, nothing happened.
Well, at least in their view.
On our side, the team was stressed and constantly interrupted with new prioritization, requests, or expedites. The engineers were mostly firefighting, which meant they had no time to properly fix or improve things (or learn new things themselves). Also, having several projects and tasks ongoing at the same time, they were doing much task switching—hence, nothing ever got finished.
Creating a true team
It's common for team members to more or less only sit together but not really work together. They each have their own to-do lists and tickets, and they each participate in separate projects. We aimed to create a team on which everyone could jump in on any task at any time. The team should work as a unit, not just as four people working on their own stuff in the same room.
To do that we needed to pair up on tasks to achieve instant knowledge sharing. We needed to stop separating customers across individuals on the team. It's a typical practice in development—what some will call "pair" or "mob" programming—and it works just as well for Operations teams.
Here's how we did it.
Our process for visualizing work
1. Write down all tasks
From what we'd learned from books and through recommendations from TV4, we decided to use kanban to help us with this new way of working. The first step was to write down all ongoing and planned work. Everyone transferred their tickets and to-do lists into sticky notes to our physical board. (When you're doing this, it's important not to hide or downplay anything; put everything up there when you begin. You can sort the rest out later.)
2. Categorize the work
To make differentiating between different kinds of tasks easier, we decided to categorize the cards. We chose the following categories:
- Problems
- Improvements
- Tasks (which would cover Change Requests and Service Requests)
We determined that we wouldn't create cards for "Incidents," as we decided we should be handling those immediately. In those cases, the team worked directly in the ticketing system. We also determined that all work should correspond to a registered ticket in the system, and we started putting ticket numbers on top of each card.
Next, we put the title of the ticket on the card, so we could grasp what it was about. Additional detail was available in the ticketing system.
We started out setting due dates in the bottom left corner of every card; later, though, we changed that to the date the ticket was created. As cards begun to flow through the board, it was more important to flag cards that had stalled than it was to monitor due dates.
In the bottom right corner of every kanban card, we placed customer abbreviations that allowed us to quickly differentiate between the customers our team was handling. We marked blocked cards with a red sticker. Every team member also received two colored magnets to indicate the cards they were currently working on.
3. Create a board
Creating the first iteration of the board was much easier after we'd categorized the work. We didn't plan that far ahead, as we were aware that the board layout will change a lot in the beginning.
Our initial setup was:
- Backlog, for all identified upcoming work
- Next, the prioritized cards to work on
- WIP (Work In Progress), what we are actively working on right now
- Accept, work that is to be verified, tested, and accepted by the customer
- Done, work that is completed
Reality hits
The reality of our new system really hit us once we got all the sticky notes up on the board. There were way too many cards in the WIP column. No wonder we and the customers felt we never got anything done!
We decided to stop beginning new items until the WIP had decreased to a more reasonable number. That was easier said than done. Customers and stakeholders wondered what happened with their tickets and their new, incoming (important) tickets. Saying "no" was hard work. We had to explain that everyone would benefit in the end if we could just finish what we'd already started.
Holding daily sync meetings
The scariest part for us (and the management) was the level of transparency we introduced.
We began to run daily sync meetings with the customer, so we could look back at what was done yesterday, what the focus would be for the day, and whether anything was standing in our way. To make this work as we'd hoped, we had to be honest with TV4 on whether something was stuck in another department; whether the team was short on people due to illness, vacation, or something similar; or whether we would be tied up with other customers' work for a while.
These were things you typically never tell customers outright, as it's not really their problem if we are short on time or people. But by starting to be open about it, we could discuss alternatives or prioritize differently until we could catch up. Of course, we would not always agree, but mostly we could handle each situation and focus on the most important tasks at hand.
Handling interruptions
A big time thief—and a significant cause of stress—was interruptions. They usually came from several sources: monitoring systems, incidents, customers contacting us directly, account managers rushing in to expedite something, line managers wanting to re-prioritize, or other team members asking for help. We knew we'd need to do something about these interruptions if we were going to be able to focus, minimize task switching, and keep our WIP tickets at a reasonable limit.
To protect the team, we decided to appoint someone to a "gatekeeper" role. The gatekeeper would handle all incoming incidents including people coming by to ask questions, try to expedite tasks, and discuss prioritization. If the gatekeeper could not solve an incident by himself, he would check with someone in the team who could help out. The gatekeeper would not transfer the ticket to the other technician; they would solve the incident together, so the gatekeeper could learn and solve similar incidents on his own the next time. The gatekeeper role was a rolling schedule, four hours at a time.
If a major incident occurred, though, we would "pull the andon cord" and everyone teamed up to solve the ongoing incident together.
Automation
Developers at TV4 wished to be able to push new releases to the production environment as often and as quickly as they wanted, which was several times a day. At the time, the standard process (for all customers) was time consuming and required involvement from several departments. The customer had to create a "Change Request" at our service desk; the desk in turn passed the ticket to the responsible tech team who implemented the change. The ticket went back to the service desk, who then contacted the customer asking to verify the release. If everything seemed good, the customer would then open a new "Change Request" for release in the production environment.
The whole release could, in some cases, take weeks.
We saw many opportunities for improvements. We started by allowing the customer to do releases in the stage environment themselves. This helped us shorten lead time while working on automating as much as possible in the final solution. The result was a mostly automated process with checkpoints, one entirely triggered by the customer. In this way, we managed to go from one release every week or two weeks to several releases per day—just as the customer wanted.
Feedback and retrospectives
To evaluate how we were doing and what to improve, we ran weekly retrospective meetings. These were a good way to discuss what had went well, what could be improved, and what we should stop doing.
We typically ran the meeting only with the team. Sometimes, though, a representative from the customer participated, too. One of our first insights from a retrospective occurred when we discovered we had neglected some customers. As each customer card was both color coded and marked, it was easy to spot trends when looking at all the completed cards after a week or two and see that almost everything was for only one customer. The next week we could then focus on other customers a bit more.
So how did it go?
The entire processes was not "all upwards and onwards," but we did get a great start. Due to outside circumstances, the team frequently had to work with other projects and customers during our first week, so we could only put in 50 hours of work (instead of the agreed 100 hours per week).
But the next week, I received a call from one of the lead developers at TV4. I feared the customer would be unhappy with our results.
I could not have been more wrong.
Instead, he told me how fantastic these new improvements were. In fact, he said, it was the best week ever since they'd become a customer.
That was saying a lot, as they were one of Basefarm's first customers. It showed us that by putting focus on what we were currently doing, prioritizing together with the customer, and not starting new things before we'd finish ongoing tasks, we could work half the time and make the customer much happier!
We also had our fair share of setbacks. The most common was "WIP creep." Several times, when we noticed cards getting stuck on the board, people began feeling stressed. But it looked like we should have a normal amount of work! Then we huddled up by the board, asking each member of the team to be honest and transparent: Are we working on anything that is not up on the board? Usually, we were—small things that were "only" going to take an hour or so, but that eventually grew bigger and bigger. In one of our board resets, we were back to counting 31 cards as ongoing (way over the agreed WIP limit!).
Eventually, the new way of working showed the whole company that both we and the customers benefited from the better collaboration, and it started to spread to other teams, each one making their own twist to fit their needs and their specific customers.
This article is part of the Open Organization Workbook project.
2 Comments