Refactor your Team 1

This serie of article is on my experience as a coach in a recruiting company I joined 2 years ago, from the struggle to the solution implemented, and who knows, maybe it would give you ideas to solve yours !

At that time the team was about 6 people in development (5 developers and 1 designer). The development department was in charge of several recruiting websites as well as a set of internal tools to help employees (sales, marketing). The main service has been developed externally and the newly created development team was in charge to take over this legacy/horrible piece of software.

In total we counted 5 projects in the portfolio and the ambition was to recruit 4 engineers to go up to 9 projects. One of my mission was to be an agile coach amongst this team and smoothen the practices as well as increase the productivity. At the time of the interview the team leader was confessing me their struggles, even if in his sense, they followed all the good practices : continuous deployment,  multi-disciplinary team… as well as retrospective method which he was quite proud about: KPT (Keep Problem Try).

State of play when I joined

The workflow of the tasks was handled as follows :

1. Adding issues in an “Agile” issue tracker, without any date or milestone, only a business priority decided by the development team.(little involvement from the business teams)
2. Creating a “feature” or “bugfix” branch from master on the GIT repository
3. Implementing the change on this branch
4. Deploy on a local environment (vagrant) supposedly alike the production and manual testing of the feature/bugfix
5. If the developer is satisfied with himself, he commits the code and merge onto the master branch.
6. If there is any doubt, they deploy a second time on a test server (AWS EC2 instance) which supposedly looks like more the production but non identical (no firewall, no load balancing, different size, different web server config).
7. If the developer is satisfied he can push on the unique production server using a homemade bash script (basically does git pull, composer install and copy in /var/www/html)
8. Mark the issue as resolved
9. The project leader validates the feature/bugfix and close the issue.

Furthermore the team gathered once a week to discuss problems, challenges and possible improvements using a method called KPT that I described in another post.

Observing and asking around about the process and the problems encountered, I could infer the following pros and cons of the current practices:

The cons :

• Developers complain about multiple bugs at each release
• No automated tests for feature or regression (The border effects are rarely noticed prior to release).
• No sprint or milestone handling from the team which makes the synchronisation of teams and management quite painful. It moved to a micro management pattern where the project leader checks in with each member individually to know what’s going on.
• The weekly meeting(KPT) turns into a fight where people tell problems without bringing solution at a deeper level (Treat only the symptoms) and problems just grow infinitely.
• The motivation of the team decreases as they can’t see the bigger picture.
• Absolutely zero metrics or “feedback” in the development process which prevents any incremental improvement inside the team or real meaningful features for the user. (The only metric is the Page View metric from “Google Analytics”)
• Absolutely no policy on security or system management (Server updates, password management, code security… )
• Developers complain about the lack of occasion to improve their skills. As they work on a legacy application that they try to move slowly toward modern patterns, the change is too slow and nobody is excited about it anymore.

The pros :

• A real will from the whole team to do better
• A really low resistance to change both from developers and management team
• Developers are really skilful (PHP code, framework knowledge)
• Most of tools are already there : vagrant, chef, git, issue tracking, deployment script, agile infrastructure handling via AWS but not mastered at all.
• A retrospective method in place which is a good start to improve things

Back off to better blow up

Even if the team thought so, I do not call this method Continuous Delivery. There was way too much missing parts to deliver software peacefully 10 times a day and make it a “non-event”. I then decided to rewind a little in a less agile process in order to train the team and create new habits. In a 2 months interval the following practices have been set up:

• Setup a one week sprint : developers cannot deploy on their own and must wait and synchronise with others until deployment day (Thursday).
• Prior to the deploy day, we set up a review day of the features (Wednesday). All features will be reviewed by the all team functionally and in the code. Since there was no automated tests, that was the only way to somehow ensure some higher quality at the time being.
• Thanks to the one week sprints, it became possible to compile a weekly “Release Note” aimed for business stakeholders in the project to show them progress and give them visibility.
• Nominated a “deploy master” entitled for each project whom is in charge of merging the features and give a go/no-go to production deployment.
• Creation of acceptance tests, at first, alike smoke tests which allows to train the team, and check regressions. Since the code inherited was mostly legacy (no object) it was difficult to begin with unit tests.
• Establishment of a Definition Of Done. Developers agreed to define when a feature is considered completed:
  • Code the feature
  • Implement acceptance tests
  • Local testing on vagrant
  • Fill in or update the information on the internal wiki
  • Fill in the issue tracking properly
  • Push the code on the right branch (naming rules)
  • If necessary add new dependencies to the deployment recipe
• Improvement of the deployment scripts by adding dependencies check (PHP packages, running process) via Ansible
• Compliance of the test environment to finally be the copy of the production environment and use it as staging and QA testing.
• Organization of “code retreat”  once a month. In the facts, it’s mainly presentations (1-2 hours) on new subjects (Advanced GIT, Security, Unit Testing, DDD)
• Creation of a common “GIT workflow” to all projects. Mainly based on the Gitflow Workflow but slightly modified to fit the internal legacy and practices.

A lot of things happened in two months and even if the team willpower is to praise, some practices demand some experience and to get used to it to make it an habit. However after some iterations, developers were unanimous, the website quality did go up steadily. Deployment errors almost entirely faded away as well as regression and border effects. The establishment of a “deploy master” allows to control what is being deployed as well as validate the whole system through acceptance testing. He also ensures the respect of the GIT workflow and the quality in the issue tracker during weekly reviews.

Next steps

It is a continuous improvement system and despite the important changes, it is only the first step to bring agility, quality and continuous deployment into the organization. As a coach I do not believe there is a single correct solution to improve productivity inside an organization and the objective should always be to empower the team rather than applying rules blindly.

In the next post I’ll present you the next step to answer the following challenges :

• Integrate metrics & feedback in the loop for either developers and business stakeholders
• Automate the build, tests and deployments
• Upgrade the whole security (process, tools, infrastructure and code)
• Introduce the concept of DevOps to expand the current stakeholders and force the business teams to step in. To do that the main theme would be “Time To Market” and Feedback.
• Create a training policy internally to enable the skill up of developers and create new opportunity inside the organization.

Do not hesitate to leave me some comments to give your advice or tell me how you would have done it. Even better, if you have experience coaching and creating new process for organization I’d love to hear your story!

You can see the Part 2 of this series over here !