agile-requirements-framework

By MIKE VOGEL globallyunique@gmail.com. This article borrows extensively from those who’ve established how we can work agile. It is my attempt to introduce teams to how I think we should work.1

Real-Agile with a Comprehensive Requirements Framework

Products2 need comprehensive requirements specifications. Not an up-front waterfall spec, rather a continuously evolving framework of requirements specifications that organize the path to building the product. User stories as used in current agile processes cannot accomplish this because they need to drive short term work-management rather than be a fully connected requirements story; In this article I detail my reasoning by describing a way-of-working compatible with agile and based on incrementally built comprehensive requirements. The following is an overview of the way-of-working, each part is explored in detail.

Overview

First let me define what I mean by the words in the title. My definition of Real-Agile is based on the ideas from those I consider the master craftspeople of agile ways-of-working3 :

My definition of Comprehensive Requirements is a document(s) that cover the full scope of the product as a connected journey6, spanning all the releases.

Why Do We Need Comprehensive Requirements

Formally, many domains mandate a requirements documents, e.g., I primarily work in the regulated / validated Life Science domain where, among other mandates, there must be a requirements document. Informally, we need requirements because humans forget things and these forgetful humans routinely leave and join our product team. Without something like a requirements document to provide the overarching goals and enough of the details, we lose track of what we’ve done, and we struggle to see how new work should fit in.7 In this article I’ll go beyond just meeting the mandates and make the case that a comprehensive requirements document makes every step of an agile way-of-working better.

I see little discussion or innovation on the need for comprehensive requirements when working agile. What I see way too often are two extremes:

Let’s examine the issues with each of these requirements approaches.

User Stories as Requirements

What I see at many companies is a process that mashes together the user stories from the tracking tool into a ‘requirements document’. I’d go so far as to say that this is the current standard practice for most companies that need requirements documents and are trying to maintain an agile way-of-working. While it’s the easy approach, it doesn’t work because user stories are NOT requirements! Stories don’t work as the requirements because:

For more details see Stories Are Not Requirements.

Not having user stories as the requirements should be an easy topic. Since so many companies that are trying to work agile, need requirements documents, but are making a mess of it, a more detailed discussion is needed.

Waterfall Style Requirements Documents

At the other end of the spectrum is creation of traditional up-front requirements documents, a.k.a. waterfall. These take the form of Business Requirements Documents (BRD), Functional Requirements Documents (FRD), or Software Requirements Specification (SRS). This kind of document doesn’t fit an agile way-of-working. They are used because of a combination of:

Agile came into existence because we learned that requirements cannot accurately be specified up front. I see little hope for combining these types of requirements documents with an agile way-of-working.

Approaches Between the Extremes

If we’re not going to take the easy but wrong approach of user stories as the requirements or cripple the agile way-of-working with a waterfall style, we need to find something between these extremes. We need to capture the requirements in a way that fits with an agile way-of-working but doesn’t create massive overhead.

The purpose and value of capturing requirements changes as we work on a product. As we are getting started in each unit of work, be it as small as a sprint or as large as multiple releases, writing things down is a powerful thinking tool in the iterative learning process. We can’t get all the details right before we implement so the requirements framework needs to strongly support working iteratively and incrementally. After we’re done it needs to naturally serve as the documentation of what we’ve built. With that in mind we can list the ‘ilities’ for a great requirements framework:

To meet all the above we need a framework rather than just a requirements document. The following are practices I’ve tried and that I consider essential to get to such a framework:

As noted above, these practices have limits that keep us from getting to the desired framework. My requirements framework meets all the ‘ilities’ by combining the Story Mapping, Domain Analysis Documents, and BDD centered around an updated approach to use-cases. I assume you are doing something like the above practices. If not then you need some combination of understanding them and actively starting to do them as part of moving to this more comprehensive framework. For example, if you use the framework you don’t need to do story mapping because you’ll get the equivalent from the framework vs. you must be doing BDD to use the framework.

The following sections drill into the details of each part of the framework.

Overview

Use Cases

Since deep integration of use cases is the novel part of the framework we’ll explore them first. See why Use Cases are Essential for material from the ultimate use case experts. The following picture highlights the use cases in the framework. Use cases may seem like an outdated, big-requirements-up-front approach that can’t align with real-agile. This isn’t so in general and is especially wrong when you’re working agile in a validated or regulated situation.

use cases

Writing good use cases requires doing good analysis and modeling of the domain. There is a lot of information on use cases available in books9 and on the web, so I’ll focus on the aspects related to how they fit into the framework. The essential idea is to build a structure like:

Use Case Name (a user goal)

The main structure of a use-case-based requirements document are:

The following is a simplified example of some use cases for a product to design configuration files. The idea is that a configuration is made up of many parts, each with a unique structure that the user needs to populate. I present the example first and then review how its structure provides the ideal requirements framework.

Access Configuration Designer

An example of a ‘top-level’ use case that calls other use cases.

Base Flow - Access Designer
  1. User accesses Configuration Designer by browsing to designer.com and enters valid credentials.
  2. User Views or Edits Configuration.
  3. User closes the Configuration.
  4. User logs-out of the Designer
Alternatives
1a - Setup Credentials
  1. …
Detailed Requirement Statements

This section will list the requirements statement / business rule for each BDD scenario associated with the use case. It is automatically generated from the text at the beginning of each BDD scenarios.

View and Edit Configuration

An example of a sub-use case. It is called by the Access Configuration Details use case.

Base Flow - View and Edit Configuration
  1. User does any of the following by navigating to the appropriate part of >the configuration.
  2. Other steps…
Data
Configuration (the name of a thing in the business domain)

This is an example of what goes in a ‘Data’ sub-section. Splitting out the data from the steps isn’t a standard practice when writing use cases. Such a section would contain things like the definition of the parts of a ‘Configuration’, navigation to and operations on those parts, e.g.,

Detailed Requirement Statements

BDD scenario associated with the use case….

For details on how the detailed requirements are presented, see Organizing and Managing BDD Scenarios As Your Detailed Requirements

See Working With Use Cases for additional guidance on how I write use cases.

Ideally, you’ll have a top-level picture of the use cases. Such a picture gives context of how the main use cases fit together. The picture need not, probably should not, show alternative flows or a lot of other details. It should not be a UML style use case diagram. It should be a flow with arrows connecting use cases.

Use Case Overview

A picture is not testable so it doesn’t need to be correct. Its job is simply to give the context and act as a graphical table-of-contents. Keeping it high-level and with minimal detail eliminates the need to constantly re-draw it as the use cases are refactored. You should not be doing a separate process-flow style picture for every use case. It is not worth the significant extra work of keeping a lot of pictures up to date and having them be correct enough to be valuable vs. just working with the text versions of the use cases.

Benefits of The Framework

Use Cases as a Better Backlog

The flat product backlog does not capture the customer’s journey. This problem and using story mapping as the solution has been written about extensively10.

Flat Backlog vs. Story Map

The flat backlog provides no context or big-picture for a product. Arranging user stories in the order they’ll be delivered doesn’t help a product manager explain to others what the product does or determine if they’ve identified the relevant user stories.

The work on the use cases should start as a top-level list of user goals11 similar to a regular flat backlog of epics and user stories. The result can be a backlog of use cases. They are incrementally expanded to be a connected set of more detailed use cases. This provides an even better organization of the product backlog than a story map, while setting us up to have comprehensive requirements.

Ordering the stories based on things like priority happens after the initial set of use cases is identified. That ordering is the beginning of work-management, e.g., put the user stories in priority order into a work-management tool like Jira or use a story mapping tool instead.

When you have the some or all of the use cases before starting implementation of a feature it’s easy to create user stories from them. The first cut at user stories can be as simple as one story for the base flow and one for each alternative. The following picture shows the idea of such paths through the flows. My experience is that typically a flow gets broken into smaller user stories. There are not hard-and-fast rules. A story can be a path through a flow or multiple flows. How you create user stories largely depends on the effort to implement and how splitting or grouping helps with work-management.

Use Case Schematic

Factoring out the data and rules from the use case steps pays dividends here. Varying the data is an ideal way to find BDD scenarios that then become the automated tests. If, as I recommend, you are considering every BDD scenario a ‘story’ then walking paths through the use case flows with different data is a great way to find user stories and generate BDD scenarios. A powerful benefit is that you’ll be walking through use case flows and this is likely to be a longer path than a single story thus yielding better, or at least more comprehensive, tests.

Also important is that we aren’t working on all the use cases at the same time. We’re working on one goal at a time with minimal requirements on things to be built in the future. So the many parts of the requirements documentation could be just names of use cases that may get expanded in the future. This also makes review of the use cases easier. At any point in time we’re only working on a small subset. The full set is there as the context but not changing much in a sprint(s).

Finally, we want to write the absolute minimum that provides a framework of the requirements as a way to control the overhead from constant refactoring of the use cases. The details should instead be migrated into the BDD scenarios that we link to the use cases. We might start with a lot of detail in the use case doc but look to move that detail out into BDD scenarios as much as possible as the implementation proceeds.

Work In All Directions

While your goal might be to have decent use cases before starting implementation, this will frequently not be appropriate or possible. We may start by just writing a list of use cases and then evolve from all directions to update and refactor the use cases as we learn. Working-in-all-directions happens because every part of the framework is iterative and incremental. Up to a point, the more we have the opportunity to discuss and think deeply about what to build the better the foundation of understanding we start with. There is no magic answer to finding the balance between the waterfall big-requirements-up-front trap and the code-and-fail-fast that can lead to lots of wasted implementation work. This framework strives to help find that balance while setting up for rapid cycles of implementation and learning. It’s also typical that features have wildly difference cycles depending on the amount of unknowns. Where the unknowns are limited, more upfront thinking can speed delivery. The following picture highlights all the other parts of the framework feeding into the use cases.

All Directions Highlighted

Working in all directions means:

Store Everything in Git

Git unites developers, designers, managers, and QA engineers in industry, academic, and personal settings because of its broad utility. As a precise and flexible system to track and control the various versions of your code and documents, it is incredibly useful for collaborative projects being developed over time (so, most projects).Having the analysts and users working collaboratively in the same place as the devs and testers with a rapid feedback cycle is a massive benefit. Using different tools that aren’t directly linked to the product substantially slow such a cycle. The simplest and I believe only way this is possible is to have everyone storing everything in the git repository.

All In Git

The changes needed to have everyone working in git is worth it. The devs and testers are already putting everything in git. The devs are putting their code there12, the detailed requirements, as BDD scenarios, are also either BBD scenario text files or actual code13. Everyone collaborating in git becomes immediately possible when the requirements documents are text files written in markdown14 and managed in git.

I can hear the response. Git is scary. Markdown is only for techies. My experience is that I can teach anyone how to use git and edit markdown quickly. The users don’t need to be git or markdown experts, just doing the basics gets us all actively collaborating in a much better way than separate tools. Anyone you trust to be writing requirements needs to have the skills to work in markdown. In the Life Science domain I primarily work in, most of the users are really smart, e.g., scientists. If they see how working closely in tight feedback cycles gets them rapid results, they will learn something new like git. It’s a big but important change. I expect the situation is similar in most other domains.

As a concrete example of what a tight feedback cycle looks like, consider the following:

If the change is just a scenario addition or detail change, the use cases need not be updated. The change will be reflected by the scenarios presence under the appropriate use case. If the change impacts the use case, then they are easy enough to edit directly as part of the discussion about the change.

There is a multitude of separate specialized tools to capture the requirements and the supporting details as BDD test scenarios. They don’t store their artifacts in a text or code format in git. This results in disconnects between the product and its supporting artifacts. I expect every reader has seen requirements that are out of sync with the current implementation, especially for mature products. Having the detailed requirements be living, constantly executed, BDD scenarios is a major way this disconnect is avoided. Having these living BDD artifacts organized in a light-weight use case framework enables keeping the rest of the specifications in sync with the product. Another factor is the cost of separate tools, teams to support them, and user training. Working all in git using markdown eliminates all these costs.

work-management

A tool(s) is needed to manage the work on user stories, e.g., a work-management system like Jira15. As mentioned above, the user stories in the work-management system should not be kept after the user stories are done, hence the garbage can in the overview picture.

work-management

It is important to note that the work-management system is not storing its artifacts in git. The work-management is thrown away after a sprint, so it doesn’t matter where it is stored. While it might be nice to have this stored in git, it is not important to the requirements framework that this is done. Some might argue for keeping the estimates, burn-downs, etc. Use of those kinds of estimation and tracking artifacts is not part of the agile way-of-working advocated for in this article. The assumptions are:

It’s good to put details in work-management system while the work is happening, e.g., copy in parts of the use case flow related to a story to start the work, add comments about how it needs to change, add detailed notes about requirements, etc. Ultimately move this into the use cases or the BDD scenarios as their permanent and organized form. See the work-management article for additional thoughts on use of the work-management system.

Automate All Artifact Creation

Since everything is in well structured files, ideally markdown, simple programs can produce all the artifacts, e.g.,

These are ‘simple programs’ because all they need to do is scan the markdown files for simple patterns and assemble the pieces into new documents. Frequently no program even needs to be written because there are numerous open source tools that produce other document formats or organizations from markdown documents, e.g., Pandoc to produce PDF, HTML, Word, and many other formats.

Shift Requirement Details to the Automated Tests

The idea of making the detailed requirements be the BDD Scenarios has been discussed throughout this article. This gets you to having executable requirements. Keep the use cases to a minimum and let them be the framework to organize the BDD tests. What hasn’t been called out is that to accomplish this the team must make on-going investments in the BDD scenarios including the documentation that is part of them so they become good detailed requirements statements. This means the whole team needs to be refactoring and updating scenarios as the requirements evolve and adding good descriptions that become the requirements statements under the use cases to them.

Closing Thoughts

Why is this a Great Requirements Framework

Relationship to Agile Frameworks

Comparisons can be made to the official agile frameworks, e.g., Scrum, SAFe, etc. Since Scrum is mostly just process, this framework can be used with it except for some conflicts around allowing more incremental learning and evolution and making the stories so small they are the BDD scenarios. I’m not a fan or an expert on SAFe. This framework is a team level process that should fit any of these frameworks, however what I mostly see is the combination of ALM tools and SAFe where the details of the ways-of-working are very constrained. In that kind of environment it won’t fit. For teams using a very lite agile way-of-working this framework potentially requires adding work to maintain and organize around the the use cases.

Challenges to Adoption

Beyond just the time and work of switching to doing what is proposed in this article, be prepared for the following challenges and consider the suggested mitigations:

Footnotes

  1. This article does not get into details on roles and team structures. The assumption is that there are Subject Matter Experts (SMEs) for the domain as a permanent part of the team and that someone is acting in a Business Analyst role to write the requirements. No approach to product building is going to work if you don’t have this in place. ↩

  2. Rather than repeat ‘product / project’, I’m going to say ‘product’. For discussion of why ‘projects’ are a bad idea, see sources like: Product rather than Project Teams, Product Mindset ↩

  3. I primarily consider the signers of the agile manifesto to be the master craftsmen of agile. Few of them are Scrum advocates and even fewer like SAFe. I share those views. The people I have taken most of my ideas from are: Ron Jeffries and GeePaw Hill. ↩

  4. It’s summed up by Ron’s article: Developers Should Abandon Agile. ↩

  5. Don’t fall into the RAT trap, see GeePaws article: Rework Avoidance Theory ↩

  6. It cannot be a long list of ‘the system must…’ statements. ↩

  7. It’s possible for the user guide to provide the info on what a product does, I’ve never seen one written as the system was built. Further, the organization provided for things like traceability don’t map well to a user guide. I’m interested to hear about the user guide as the substitute for requirements. ↩

  8. ‘Work Manageability’ isn’t an ideal name. Forgive me for trying to force everything into ‘ilities’. ↩

  9. See the definitive book for Writing Effective Use Cases by Alistair Cockburn. ↩

  10. See information from any of the story mapping products, e.g., Easy Agile, or the work by the creator of Story Mapping ↩

  11. While there are techniques for capturing higher level business capabilities for an organization, that isn’t necessary to deliver a product. It’s a good higher level context but is outside the scope of this framework. ↩

  12. The design documents are also be markdown documents stored in git. ↩

  13. The typical BDD tools, e.g., Cucumber or Behave, have you write the scenarios in text based feature files. On a recent project all the BDD scenarios were written in a DSL running in the same language engineering tool as the product code. In this case the BDD scenarios generated HTML reports that were included in the ‘Detailed Requirement Statements’ section. See Nodeworld: Adventures from a world with turtles all the way down for an details of the approach. This is also an example of the power of a custom DSL for testing vs. limiting yourself to BDDs Given-When-Then. ↩

  14. See the many resources on the web on markdown, e.g., The benefits and challenges of using markdown software, Why you should and should not use markdown ↩

  15. Depending on the work-management tool selected, it maybe necessary to supplement it by using a separate issue tracking system like the one available in github or gitlab. This is a team specific decision. It doesn’t change the fact that whatever is put in the work-management system isn’t going to be good as the comprehensive requirements. ↩

  16. There is a lot available from google about #NoEstimates, e.g., A nice overview of the #NoEstimates ideas,The first chapter of the book by one of the creators of the #NoEstimates movement, Ron Jeffries on open questions about #NoEstimates ↩