Success in software development triggers an insatiable demand for more complex software to be developed even faster and preferably cheaper. To achieve more ambitious software goals, software development needs to be stepped up.
At the recent Collision from Home virtual conference, Stephen Deasy, Engineering Manager at Atlassian, explored the issues that affect scalable software development. Atlassian is a well-known provider of software development tools, including Jira and Trello. He said, “As you develop your software, you will come across various dysfunctional patterns such as too many meetings, a push to add another person to the development team, or creating a design that reflects your. organizational chart. Position your projects for productive success by keeping teams small, developing juniors, and adding entirely new teams as needed.
Scaling up software development isn’t easy. Most non-IT managers intuitively believe that just adding more capacity, i.e. people, will produce more software without negatively impacting unit costs or quality. After all, this is a long-standing experience with other functions like production, transportation or warehousing. Unfortunately, this experience is irrelevant and even dangerous when applied to software development.
What should a CIO think about when it comes to building on successful software development? What makes software development at scale different? What are the failed approaches to avoid? What approaches can lead to success?
Stephen Deasy reminded his audience of some historical perspectives related to the problem of scaling software development that remain valid and relevant in today’s software world.
In 1967, Melvin E. Conway, a computer programmer, stated that “any organization that designs a system will produce a design whose structure is a copy of the organization’s communication structure.”
Common failures in project organization and software architecture that derive from Conway’s Law include:
- Separately design the project organization and the software architecture. The split will lead to poor design choices and integration disconnects. Instead, design the project organization and software architecture at the same time and plan to revise them as the project evolves.
- Design a multi-level and hierarchical project organization. This approach will limit the software design to a small set of solutions that will most likely exclude the optimal solution. Instead, design the project organization as flat as possible with no more than 3 levels. A flat project organization is more likely to arrive at the optimal solution.
- Plan for success. Intuitively, rigid or detailed planning will produce inflexible software architecture and software design. Instead, we’ll get better performing software when we expect to encounter failures, assume learning, and recognize the need for repeated adaptation.
- Design the organization of the project to involve large teams because the planned system is large. Large teams will unfortunately produce little software due to the enormous communication and coordination effort. Instead, organize multiple small teams that may each have some of the functionality or architecture that the team can develop independently of the others.
- Design the project organization as a set of functional teams such as software developers, business analysts, database administrators or quality assurance personnel. Functional teams will create silos that work on their own priorities and don’t coordinate well. Instead, organize each team to include all of the skills needed to accomplish the functionality or architecture assigned to the team.
In the 1990s, British anthropologist Robin Dunbar proposed that there is a “cognitive limit to the number of people with whom one can have stable social relationships, relationships in which an individual knows who each person is and how each person is. relates to every other person ”.
Dunbar proposed that humans could comfortably maintain up to 150 stable relationships. Therefore, a typical person can have up to:
- 4 to 5 intimate relationships.
- 12 to 15 relationships of trust.
- 25 – 35 close relationships.
- 75 to 100 casual relationships.
This observation on social relations limits the effective size of the team for many activities, including software development. Experience suggests that 6-8 software developers per project team is the upper limit. The project team may be larger in total as it will include other roles such as project manager, business analyst, database administrator, data modeler, and infrastructure administrator. However, the increase in the number of software developers will produce less and less software per software developer due to more effort put into communication and maintaining relationships.
Concepts like pair programming can improve quality, reduce rework, speed up orientation for new developers, and reduce the negative impact on the project if a valued software developer leaves the project. Although this idea increases the number of software developers, pair programming renames the limit to 6-8 pairs of software developers and increases costs.
This observation on human cognitive limit also limits the number of concepts or variables that even a talented software developer can juggle their mind at a time. This limit should consciously influence the complexity of the proposed system architecture. The project team should favor designs that consist of a greater number of smaller, well-defined modules over designs that consist of a smaller number of larger, more complex modules.
The mythical man-month
Fred Brooks, the world’s first computer architect and software engineer, first published The Mythical Man-Month in 1975. The central theme of the book is that “adding manpower to a late software project on makes later ”.
This observation is true because software development projects cannot be neatly divided into truly discrete tasks that software developers can work on independently. On the other hand, the manufacture of gadgets or the transport of materials or finished products by several trucks or wagons are independent tasks.
All software projects require fairly constant communication between software developers to create software modules and database schemas that can successfully interact with each other. Therefore, assigning more programmers to a project that is behind schedule will make it even later. In fact, the time needed to:
- Orienting a new software developer to the project decreases the effort available to the software developer performing the orientation.
- Intra-team communication will consume an ever increasing percentage of available calendar time.
This table of sample team sizes illustrates how the number of communication channels and therefore communication effort increases exponentially with team size. In this example, each team member only spends one hour communicating with each channel or team member during a 40-hour work week.
As the number of project team members increases, their communication effort increases and their performance decreases. In a large team, the communication effort will exceed the total capacity of the team. In this situation, no time is available to complete tasks on the project plan and the project stops progressing.
What strategies would you recommend to scale software development successfully? Let us know in the comments below.