Topic: Agile Testing -- A Statement of Concern

Message

I apologize for the length of this post but it's one of those things
that requires elaboration. I'm referencing here the article "Agile Methods and Agile Testing"
by Brian Marick and I'm doing so with a skeptical intent. I don't have
an axe to grind with Brian. I don't even know the guy. His article just
happened to be presented to me as being the "right way" by a fellow
manager. What I'm really speaking to here is the notion of "agile" that
I seen get thrown about so much. Quotes from the article will be in
bold italics.

My focus here is more on how agility is used to filter into testing.

"Programmers
are often in the grip of the Hands-On Imperative; that's why they don't
like to write documentation. Agile Methods accept the Hands-On
Imperative, but extend it to the users."

That's what I
find "agile methods" do when they start to filter to testing: accept
all the time. Instead of making a stand and advocating that something
should be different, it just says: "Okay, I'll adapt and go with
whatever." To some people that sounds good. After all, we should be
flexible, right? Well, to me, it sounds much too accomodationist. They
don't like to write documentation. Therefore it must be a bad thing.
Therefore we should accept the "Hands-On Imperative."

"And
a user asked to evaluate an abstraction like a requirements document or
a design model thinks profoundly differently than one presented with
working software."

This is a contention that is
debatable. First of all, "working software" can be an abstraction of a
problem that the software helps to deal with. But let's not quibble at
that level of detail. Instead let's look at what the "abstraction" of a
requirements or design model is. It's a story. People, over the course
centuries of cognitive developement (and over the course of millions of
years of evolutionary pattern development), have come up with workable,
usable, successful techniques for telling stories to each other.
Requirements and design models are stories. People can use the same
techniques today because, at their root, they've changed very little
over the centuries. Why? Because the human brain has not changed all
that much. We still receive information and assimilate it in our minds
in the way that our ancestors did. Our basic neural wiring has not
changed, so the techniques of storytelling, of putting information into
that human neural wiring, are basically unchanged. Yes, believe it or
not, the software world did not alter thousands of years of cognitive
wiring, even though some people like to think so.

Are
requirements different from working software? Of course. There I would
not disagree. But where I would disagree is regarding the conclusion
that the agile community seems to draw from this.

The further contention is: "Development is organized into a rapid series of functionally complete releases, each one made available for the user to try."
Okay. That sounds good. However, nothing says that documentation
efforts can't be organized into a "rapid series of functionally
complete releases", where such documentation is made available either
before working development builds or along with it. The two could
evolve together, for example, so that at the end of those rapid
developments, you have rapid documentation that matches what was
developed. You never see this advocated. Why not? Now, would this mean
putting the brakes on some development speed? Maybe. But that's not
necessarily a bad thing!!! Moving as quickly as possible means just
that: as possible. Getting some documentation down can be part of what
factors into the notion of "as quickly as possible."

"One of Scrum's practices is carefully crafted daily standup meetings that create and preserve group understanding."

That's
great for institutional knowledge. However, is this how you want to
preserve group understanding? By standup meetings? I would much rather
preserve knowledge in documentation. The idea of oral tradition went
out even before the printing press. I'm not sure I want to go back to
it. I prefer practices that allow us to deinstitutionalize knowledge so
that knowledge is not just captured in a group of individuals that are
attending the same meetings.

"With a suitable customer
representative at hand, you don't need a detailed requirements
document. When you have a question, you turn around and ask. Worried
that you won't know to ask the right question? Implement something and
show it to the customer for a quick reaction - you'll quickly learn if
you're going off track."

What if the "suitable customer
representation" isn't always at hand? I know Agile methods want this,
but what if they can't be? What if you can't just turn around and ask?
What if the customer can't always give a quick reaction. Even if they
can, don't you want to document what was decided and why so that later
on people can refer to that, such as when (as often happens) the
customer "forgets" what they said they wanted. Beyond this, agile
approaches often indicate that customers don't know what they want even
when they have time to write it down. So do they all the sudden know
what they want on the spur of the moment? Yes, they can see the
development happening so you could argue this lets them immediately see
if they want it. But there's such a thing as changing their mind.
There's such a thing as only seeing a limited part of the development
as opposed to the whole thing. There's also the idea that adaptable
documentation could go along with this "spur of the moment" customer
focus.

"But there'll be a background of tacit
understanding of the fundamentals, so that different people will
reflexively make consistent choices when confronted with similar
problems."

And where is that "tacit understanding"
coming from? What's the "background"? Does everyone accept that
background? How do we know? Do we ever check our "tacit understanding"?
If so, against what? How do we know that what's "fundamental" is truly
fundamental? Is all of this literally just based on past conversations
or standup meetings? Is it truly based just on a shared understanding
that isn't written down? How we can tell the choices are consistent? Do
we just keep everything about our shared understanding in memory? In
simple situations, maybe that's easy enough (although I doubt it). But
situations have a way of becoming complex. Also, what's our basis for
recognizing "similar problems"?

"We've always realized
that the documents we base our tests on are flawed - incomplete,
incorrect, and ambiguous - but our reaction has been to insist, in our
usually powerless way, that the document producers do better. But now
we can see that 'better' will never be good enough."

Regarding
that last statement: According to who? The "Agile" crowd? To say
something is "never" the case seems like a weak statement to me. (Plus
in the context-driven world, you could only say this relative to a
given context, right? It can't be a universal, by definition, since
everything is contextual.) Seems to me that here's another case of
accomodationist thinking. "Well, we're powerless anyway, so let's just
assume that the practice is bad and come up with another one." Maybe
the documents they've seen have been flawed. But the solution is what?
To scrap all documentation as a bad thing? Or do we work at coming up
with better ways to document?

"Documents can't be an adequate representation of working code."

And
why should they be? And who said they were? This is a straw man that I
often see established to argue against a focus on documentation.

All this said, the article ends with this: "Agile
Testing is not the answer for all projects. Neither is any of the Agile
Methods named in the first paragraph. No single approach can be."
I would agree, even though a lot of times the Agile crowd talks as if
you would have to be an idiot to not be utilizing "agile testing" or
working in an "agile" environment.

I should point out here that
I agree with some aspects of agile development. However, part of a good
testing effort, to me, is making sure that there are reasonable breaks
applied to some "agility" when it's in the best interests not just of
the current project, but of the culture in which projects are done. My
contention is that I ultimately see "agility" (as a concept for
development and then filtering into testing) leading us, as an
industry, down a dangerous path.

Author	Message
tanushree Senior Member Joined: 04Apr2007 Online Status: Offline Posts: 2160	Topic: Agile Testing -- A Statement of Concern Posted: 03Oct2007 at 12:05am
	I apologize for the length of this post but it's one of those things that requires elaboration. I'm referencing here the article "Agile Methods and Agile Testing" by Brian Marick and I'm doing so with a skeptical intent. I don't have an axe to grind with Brian. I don't even know the guy. His article just happened to be presented to me as being the "right way" by a fellow manager. What I'm really speaking to here is the notion of "agile" that I seen get thrown about so much. Quotes from the article will be in bold italics. My focus here is more on how agility is used to filter into testing. *"Programmers are often in the grip of the Hands-On Imperative; that's why they don't like to write documentation. Agile Methods accept the Hands-On Imperative, but extend it to the users."* That's what I find "agile methods" do when they start to filter to testing: accept all the time. Instead of making a stand and advocating that something should be different, it just says: "Okay, I'll adapt and go with whatever." To some people that sounds good. After all, we should be flexible, right? Well, to me, it sounds much too accomodationist. They don't like to write documentation. Therefore it must be a bad thing. Therefore we should accept the "Hands-On Imperative." *"And a user asked to evaluate an abstraction like a requirements document or a design model thinks profoundly differently than one presented with working software."* This is a contention that is debatable. First of all, "working software" can be an abstraction of a problem that the software helps to deal with. But let's not quibble at that level of detail. Instead let's look at what the "abstraction" of a requirements or design model is. It's a story. People, over the course centuries of cognitive developement (and over the course of millions of years of evolutionary pattern development), have come up with workable, usable, successful techniques for telling stories to each other. Requirements and design models are stories. People can use the same techniques today because, at their root, they've changed very little over the centuries. Why? Because the human brain has not changed all that much. We still receive information and assimilate it in our minds in the way that our ancestors did. Our basic neural wiring has not changed, so the techniques of storytelling, of putting information into that human neural wiring, are basically unchanged. Yes, believe it or not, the software world did not alter thousands of years of cognitive wiring, even though some people like to think so. Are requirements different from working software? Of course. There I would not disagree. But where I would disagree is regarding the conclusion that the agile community seems to draw from this. The further contention is: *"Development is organized into a rapid series of functionally complete releases, each one made available for the user to try."* Okay. That sounds good. However, nothing says that documentation efforts can't be organized into a "rapid series of functionally complete releases", where such documentation is made available either before working development builds or along with it. The two could evolve together, for example, so that at the end of those rapid developments, you have rapid documentation that matches what was developed. You never see this advocated. Why not? Now, would this mean putting the brakes on some development speed? Maybe. But that's not necessarily a bad thing!!! Moving as quickly as possible means just that: as possible. Getting some documentation down can be part of what factors into the notion of "as quickly as possible." *"One of Scrum's practices is carefully crafted daily standup meetings that create and preserve group understanding."* That's great for institutional knowledge. However, is this how you want to preserve group understanding? By standup meetings? I would much rather preserve knowledge in documentation. The idea of oral tradition went out even before the printing press. I'm not sure I want to go back to it. I prefer practices that allow us to deinstitutionalize knowledge so that knowledge is not just captured in a group of individuals that are attending the same meetings. "With a suitable customer representative at hand, you don't need a detailed requirements document. When you have a question, you turn around and ask. Worried that you won't know to ask the right question? Implement something and show it to the customer for a quick reaction - you'll quickly learn if you're going off track." What if the "suitable customer representation" isn't always at hand? I know Agile methods want this, but what if they can't be? What if you can't just turn around and ask? What if the customer can't always give a quick reaction. Even if they can, don't you want to document what was decided and why so that later on people can refer to that, such as when (as often happens) the customer "forgets" what they said they wanted. Beyond this, agile approaches often indicate that customers don't know what they want even when they have time to write it down. So do they all the sudden know what they want on the spur of the moment? Yes, they can see the development happening so you could argue this lets them immediately see if they want it. But there's such a thing as changing their mind. There's such a thing as only seeing a limited part of the development as opposed to the whole thing. There's also the idea that adaptable documentation could go along with this "spur of the moment" customer focus. *"But there'll be a background of tacit understanding of the fundamentals, so that different people will reflexively make consistent choices when confronted with similar problems."* And where is that "tacit understanding" coming from? What's the "background"? Does everyone accept that background? How do we know? Do we ever check our "tacit understanding"? If so, against what? How do we know that what's "fundamental" is truly fundamental? Is all of this literally just based on past conversations or standup meetings? Is it truly based just on a shared understanding that isn't written down? How we can tell the choices are consistent? Do we just keep everything about our shared understanding in memory? In simple situations, maybe that's easy enough (although I doubt it). But situations have a way of becoming complex. Also, what's our basis for recognizing "similar problems"? "We've always realized that the documents we base our tests on are flawed - incomplete, incorrect, and ambiguous - but our reaction has been to insist, in our usually powerless way, that the document producers do better. But now we can see that 'better' will never be good enough." Regarding that last statement: According to who? The "Agile" crowd? To say something is "never" the case seems like a weak statement to me. (Plus in the context-driven world, you could only say this relative to a given context, right? It can't be a universal, by definition, since everything is contextual.) Seems to me that here's another case of accomodationist thinking. "Well, we're powerless anyway, so let's just assume that the practice is bad and come up with another one." Maybe the documents they've seen have been flawed. But the solution is what? To scrap all documentation as a bad thing? Or do we work at coming up with better ways to document? *"Documents can't be an adequate representation of working code."* And why should they be? And who said they were? This is a straw man that I often see established to argue against a focus on documentation. All this said, the article ends with this: *"Agile Testing is not the answer for all projects. Neither is any of the Agile Methods named in the first paragraph. No single approach can be."* I would agree, even though a lot of times the Agile crowd talks as if you would have to be an idiot to not be utilizing "agile testing" or working in an "agile" environment. I should point out here that I agree with some aspects of agile development. However, part of a good testing effort, to me, is making sure that there are reasonable breaks applied to some "agility" when it's in the best interests not just of the current project, but of the culture in which projects are done. My contention is that I ultimately see "agility" (as a concept for development and then filtering into testing) leading us, as an industry, down a dangerous path. Post Resume: Click here to Upload your Resume & Apply for Jobs

	IP Logged


One Stop Testing Forum : Quality Assurance @ OneStopTesting : Quality Methodologies / Streams @ OneStopTesting