Each profession has its state of nirvana (eternal bliss). A baseball pitcher’s nirvana is throwing a perfect game, not allowing a single opposing batter to get on base. A cricket bowler’s is getting a hat trick – three outs in three consecutive balls. These feats are rare, so when they’re attained, they are celebrated.
Software developers also have a state of nirvana. That is producing zero-defect code. As in baseball and cricket, this feat is seldom achieved. Even the titans of our industry – Microsoft, Google, Apple, IBM – talk about zero-defect code as a holy grail they only dream of.
Vertex recently achieved this holy grail by developing a zero-defect software system for a custom application we developed for a Fortune 50 US-based enterprise. Our feat is even more satisfying because we were able to produce high-quality software for a highly complex business problem for which the customer had only a strawman of what they wanted to do and had no specific requirements.
This is a story of how the talented Vertex team achieved a feat that few have done in our industry.
The Customer and the Problem Statement
For confidentiality reasons, the customer cannot be named, but to set the context, let me just say that they are a Fortune 50 CPG company with over 80 brands operating in most of the global regions. Most products and brands are sold worldwide, with some variance for regional demographics and preferences. Product launches (called initiatives) are a big deal with the customer; it takes bringing together globally distributed teams to plan, design, and execute product launches.
When a new initiative is launched, there are usually around 1,000 collaborators spread across several world regions, and functional roles and technical expertise that come together to gate the decisions that need to be made to keep steady and timely progress. The customer had no effective tool to capture, communicate, track, and keep up with such a humongous undertaking.
This is where we entered and built a custom solution.
Building the Vision and Evolving Through Discovery
All engineering projects are guided by a few basic principles. When you build a ship, you start by designing the hull of the ship. When you build a house, you start with building the foundation. Software development projects, like other engineering disciplines, should start with defining the requirements of the solution. In our situation, the customer only knew about the problem that they wanted to solve. They did not have a vision of what a solution would look like. We used the requirements gathering stage to define the key problem statements and build a vision of what the solution should do. We spent about 30% of the entire duration on the requirements definition. This helped us to align with the customer to build a holistic scope of the system. The requirements document thoroughly defined the project boundaries and were built iteratively with customer’s inputs.
Design First
In a trade school, carpenters are taught to measure twice and cut once. In software engineering, it should be first design, then build. Our approach was thorough and detailed. We were launching the project for the Initiatives team of just one product category. Our customer had told us that if the solution was successful, they were going to open it up for all categories. We knew that our design had to accommodate changes in the future.
Data Model Design: The primary goal of the data model was to keep the database lean with lesser objects, denormalized to reduce network latency and the number of calls. We avoided direct relations in tables, limited applying constraints, and eliminated all kinds of cascading operations to reduce over-dependency on the database. All kinds of database programming were eliminated. Constraints and on-demand relations were controlled from the program’s entity framework model to allow more flexibility to the overall design.
Application/API Design: The application design went through phased stages (refer to diagram), something we did not try before. The principle was to establish a working prototype or template solution using few modules (as API or services). Once the prototype was tested and benchmarked the team was ready for continuous programming for the rest of the application building and focus on the functional aspects. The primary concept of API architecture was to identify services from business contexts, define micro-modules adhering to the core principles of microservices, build communication patterns between modules and yet develop it for a single Cloud Platform-As-A-Service (PAAS) solution. Each module represented a business functionality in the product and was designed to work independently. 90% of the system was designed to be Administrator controlled, hence a loosely coupled service-based architecture was the need.
Our Approach
We saved time by reducing a large upfront effort on mockup or wireframe by sharing an outline of the UX using PowerPoint and engaged with the customer in designing the system flow. This provided more touchpoints with the customer during the design stage, helping them visualize their system before it was built.
Agile Development
Before development started, the development team was fully immersed in the requirements document, which gave them the roadmap for the entire system. We adopted an agile development approach, with sprint cycles defined well in advance with tasks to be achieved. Each sprint consisted of 10 days. The building principle was to develop fast, fail fast, and recover fast. For each sprint cycle, we dedicated 75% of the time for development, 15% for integration testing, and 10% for issue fixing before turning it over to the users as a beta release. The development team was engaged in daily scrum calls, beyond other technical brainstorming sessions to reduce understanding gaps. Communication helped in reducing friction between modules built by multiple developers to ensure smoother integration. This ensured that each sprint cycle was defect-free and the users were asked to just validate the system. We presumed that about 20% of the requirements would change as we were building the system. Our agile method and our modular design approach gave us the flexibility to accommodate these changes without introducing defects.
Our Approach
Customer Insights
We kept engaging with the customer closely during the construction stage to exchange thoughts and demonstrate the continuous progress of the product. Wherever certain things were either technically not feasible or contradicted good design principles, we logically explained the reasons and adjusted our design accordingly. The customer was assured of the best design principles being adopted at every stage of the build. Using agile development principles helped in including our customer’s thoughts and adopting late changes in the product lifecycle easily. Including customer insights at key stages of the development helped us reach a high level of customer satisfaction at the end.
Analyze and Remediate
A key component towards a zero-defect journey was to predict defects before they arose. Our vast experience in custom development helped us to conceive a checklist of potential defects from a typical web development process. Developers treated the checkpoints as part of their development process, thus strengthening the unit testing practice. Quality software checks included performance checks and negative validation. Fixed issues which impacted core design were repeatedly checked after every sprint to ensure the particular functional area was not impacted due to other changes.
It is Not UAT Anymore, but UVA
Most custom software falls into the trap of a User Acceptance Test (UAT) stage. We did not want the user to test; we wanted them to validate the function. We started by calling the stage UVA – User Validation and Acceptance. This name change shifted the mindset of the developers and users. The users came into this stage with the mindset that they were being provided a functioning system and they just had to validate that the functions existed.
Yes, Zero-Defect Code is Possible
Custom software development often scares organizations. They are unsure of what needs to be built, how it should be built, and whether they will get what they want. Our journey proved that with the right approach, we can build software that will deliver with zero defects. Our journey is repeatable. It is just a matter of discipline.
