Zero Defects Code: Vertex Launches Custom
Project Management Solution for a Fortune 50 Customer

Each profession has its state of nirvana (eternal bliss). A baseball pitcher’s nirvana is throwing a perfect game, not allowing a single opposing batter to get on base. A cricket bowler’s is getting a hat trick – three outs in three consecutive balls. These feats are rare, so when they’re attained, they are celebrated.

Software developers also have a state of nirvana. That is producing zero-defect code. As in baseball and cricket, this feat is seldom achieved. Even the titans of our industry – Microsoft, Google, Apple, IBM – talk about zero-defect code as a holy grail they only dream of.

Vertex recently achieved this holy grail by developing a zero-defect software system for a custom application we developed for a Fortune 50 US-based enterprise. Our feat is even more satisfying because we were able to produce high-quality software for a highly complex business problem for which the customer had only a strawman of what they wanted to do and had no specific requirements.

This is a story of how the talented Vertex team achieved a feat that few have done in our industry.

The Customer and the Problem Statement

For confidentiality reasons, the customer cannot be named, but to set the context, let me just say that they are a Fortune 50 CPG company with over 80 brands operating in most of the global regions. Most products and brands are sold worldwide, with some variance for regional demographics and preferences. Product launches (called initiatives) are a big deal with the customer; it takes bringing together globally distributed teams to plan, design, and execute product launches.

When a new initiative is launched, there are usually around 1,000 collaborators spread across several world regions, and functional roles and technical expertise that come together to gate the decisions that need to be made to keep steady and timely progress. The customer had no effective tool to capture, communicate, track, and keep up with such a humongous undertaking.

This is where we entered and built a custom solution.



Building the Vision and Evolving Through Discovery

All engineering projects are guided by a few basic principles. When you build a ship, you start by designing the hull of the ship. When you build a house, you start with building the foundation. Software development projects, like other engineering disciplines, should start with defining the requirements of the solution. In our situation, the customer only knew about the problem that they wanted to solve. They did not have a vision of what a solution would look like. We used the requirements gathering stage to define the key problem statements and build a vision of what the solution should do. We spent about 30% of the entire duration on the requirements definition. This helped us to align with the customer to build a holistic scope of the system. The requirements document thoroughly defined the project boundaries and were built iteratively with customer’s inputs.

Design First

In a trade school, carpenters are taught to measure twice and cut once. In software engineering, it should be first design, then build. Our approach was thorough and detailed. We were launching the project for the Initiatives team of just one product category. Our customer had told us that if the solution was successful, they were going to open it up for all categories. We knew that our design had to accommodate changes in the future.

Data Model Design: The primary goal of the data model was to keep the database lean with lesser objects, denormalized to reduce network latency and the number of calls. We avoided direct relations in tables, limited applying constraints, and eliminated all kinds of cascading operations to reduce over-dependency on the database. All kinds of database programming were eliminated. Constraints and on-demand relations were controlled from the program’s entity framework model to allow more flexibility to the overall design.


Application/API Design: The application design went through phased stages (refer to diagram), something we did not try before. The principle was to establish a working prototype or template solution using few modules (as API or services). Once the prototype was tested and benchmarked the team was ready for continuous programming for the rest of the application building and focus on the functional aspects. The primary concept of API architecture was to identify services from business contexts, define micro-modules adhering to the core principles of microservices, build communication patterns between modules and yet develop it for a single Cloud Platform-As-A-Service (PAAS) solution. Each module represented a business functionality in the product and was designed to work independently. 90% of the system was designed to be Administrator controlled, hence a loosely coupled service-based architecture was the need.

Our Approach

Our Approach

We saved time by reducing a large upfront effort on mockup or wireframe by sharing an outline of the UX using PowerPoint and engaged with the customer in designing the system flow. This provided more touchpoints with the customer during the design stage, helping them visualize their system before it was built.

Agile Development

Before development started, the development team was fully immersed in the requirements document, which gave them the roadmap for the entire system. We adopted an agile development approach, with sprint cycles defined well in advance with tasks to be achieved. Each sprint consisted of 10 days. The building principle was to develop fast, fail fast, and recover fast. For each sprint cycle, we dedicated 75% of the time for development, 15% for integration testing, and 10% for issue fixing before turning it over to the users as a beta release. The development team was engaged in daily scrum calls, beyond other technical brainstorming sessions to reduce understanding gaps. Communication helped in reducing friction between modules built by multiple developers to ensure smoother integration. This ensured that each sprint cycle was defect-free and the users were asked to just validate the system. We presumed that about 20% of the requirements would change as we were building the system. Our agile method and our modular design approach gave us the flexibility to accommodate these changes without introducing defects.

Our Approach


Customer Insights

We kept engaging with the customer closely during the construction stage to exchange thoughts and demonstrate the continuous progress of the product. Wherever certain things were either technically not feasible or contradicted good design principles, we logically explained the reasons and adjusted our design accordingly. The customer was assured of the best design principles being adopted at every stage of the build. Using agile development principles helped in including our customer’s thoughts and adopting late changes in the product lifecycle easily. Including customer insights at key stages of the development helped us reach a high level of customer satisfaction at the end.

Analyze and Remediate

A key component towards a zero-defect journey was to predict defects before they arose. Our vast experience in custom development helped us to conceive a checklist of potential defects from a typical web development process. Developers treated the checkpoints as part of their development process, thus strengthening the unit testing practice. Quality software checks included performance checks and negative validation. Fixed issues which impacted core design were repeatedly checked after every sprint to ensure the particular functional area was not impacted due to other changes.

It is Not UAT Anymore, but UVA

Most custom software falls into the trap of a User Acceptance Test (UAT) stage. We did not want the user to test; we wanted them to validate the function. We started by calling the stage UVA – User Validation and Acceptance. This name change shifted the mindset of the developers and users. The users came into this stage with the mindset that they were being provided a functioning system and they just had to validate that the functions existed.

Yes, Zero-Defect Code is Possible

Custom software development often scares organizations. They are unsure of what needs to be built, how it should be built, and whether they will get what they want.  Our journey proved that with the right approach, we can build software that will deliver with zero defects. Our journey is repeatable. It is just a matter of discipline.

Building an Organizational Culture That Supports DevOps 

DevOps is at its core is a set of best practices for developers and IT operations that aims to shorten release cycles and increase software quality. As such, it is closely related to other development methodologies such as Agile Software Development, CI/CD (Continuous Integration/Continuous Deployment), and Kaizen.

As DevOps has matured, several toolsets have emerged to support the application development lifecycle, providing version control, reporting, requirements management, project management, automated builds, testing, and release management capabilities. The best tools in the world, however, will not ensure success if the development and IT organizations are not committed to instilling the culture necessary to support it.

Click here to read another article about how a strong DevOps culture at Vertex achieved the development paragon of zero defects code.

Our top five practices with the strongest correlation to organizational culture are:

  1. Organizational investment
  2. Team leaders’ experience and effectiveness
  3. Continuous delivery
  4. Organizational performance
  5. Lean management practices

Organizational Investment

DevOps success begins at the top. IT and development managers need to break down the silos between development, testing, deployment, and operations to create a cross-functional, unified team capable of working across the entire development lifecycle. While technology stacks and tooling aid in the quick and reliable development of applications, it is really the shared responsibility and ownership that increases the team’s velocity.

Experience and Effectiveness

Leaders of DevOps teams need to be experienced across the spectrum of IT disciplines in order to break down potential barriers and ensure that their teams are performing at the highest potential.

As a DevOps engineer at IBM said in an article published by InfoWorld, “We needed to answer some basic questions and determine the problems we were trying to solve…If you don’t know how the work is actually done, you don’t know which problems are worth solving.”

Continuous Delivery

Continuous delivery is the DevOps approach that aims to produce software in short cycles. By developing in short cycles, teams can reliably release their software at any time. With CD, development teams can build, test, and release software faster and more frequently. As a result, they can reduce the cost, time, and risk of delivering each change. A repeatable deployment process is important for continuous delivery. Rather than throwing code ‘over the wall’ to QA and Operations, integrated teams can achieve heretofore unexpected release cycles.

Organizational Performance

The transformation to a DevOps organization takes a large investment in time, money, and resources. The ability to clearly and accurately measure the performance of your DevOps team is critical to improving efficiency and tracking success.

The following metrics are a great start for measuring DevOps performance and progress:

  • Lead Time – The time it takes to implement, test, and deliver code
  • Deployment Frequency – The number of software deployments over a period of time.
  • Change Failure Rate – While shortening and increasing the frequency of release cycles is the goal of DevOps, management should also look at failure rates.
  • Mean Time to Recovery (MTTR) – How long it takes the organization to recover from a failure and release a fix.
  • Customer Ticket Volume – This is ultimately a measure of end-user satisfaction. Bugs and errors that are missed in QA are often detected by the end-user. The number of customer tickets marked as problems or bugs is an important indicator of application reliability.

Lean Management Practices

A recent article in Communications of the ACM stated that “DevOps is about good development practices that continually deliver product features (Agile) effectively with minimal wasted efforts (Lean) which are overseen by good governance controls (Information Technology Service Management, or ITSM). To that end, a growing consensus within the information technology community is that DevOps = Agile + Lean + ITSM.”

Contrary to popular belief, while DevOps may eliminate the traditional ‘waterfall’ method of project management, it does not brush aside Agile/Scrum either. Nor does it do away with traditional IT organizations rather than force a transformational shift.

Vertex Computer Systems Has The Practices AND The Tools

Look to the DevOps experts at Vertex to guide your way to the efficiencies of DevOps and to help you choose the right tools to create repeatable, clean releases. We Got IT.

Vertex Computer Systems is Hiring!Join the Team »