- Quick Overview
- Freeware Tools on the Net for ET and SBTM
- SBTM Critical Success Factor – Senior Required for Test Lead
- SBTM Critical Success Factor – Debrief Every Day
- Writing Good Test Notes Takes Time and Practice
- ET.XLS Tips
- Testing History at Your Fingertips
- Mine Your Data – Create Test Guides
1. Quick Overview
What is SBTM? It is a way of managing your Testing effort that is different from how you probably learned to do it. Here are some links to provide you with some background information:
- The SBTM home page at Satisfice
- Jon Bach’s presentation on ET and SBTM
- James Lindsay’s “Adventures in Session-Based Testing”
- Wikipedia entry on Session-Based Testing
- Satisfice: More articles about Exploratory Testing
- Article: Applying Session-Based Testing to Medical Software
According to the Satisfice SBTM home page, Session-Based Test Management is “a method for measuring and managing exploratory testing.” In a nutshell, it is a Test Management Framework. It is one of many that you can choose to use and apply in your particular situation. I happen to like it.
Note that some people refer to it as SBT (Session-Based Testing), and some include this as part of “xBTM” which also includes “Thread-Based Test Management”. TBTM is an alternate way to manage Exploratory Testing (ET) that doesn’t focus on time-boxing the test sessions.
2. Freeware Tools on the Net for ET and SBTM
This list was last updated in September 2011.
|Session Scan Tool||Perl, DOS Batch files||The original command-line scan tool. (No changes since 2000.)|
|+ SBTM Ruby Tools||Ruby (installs on top of the original Session Scan Tool)||Add-on for the original Sessions framework. Scan Tools are in Ruby – includes bug fixes and enhancements. (last updated 2009)|
|CLR-Sessions||Ruby, batch files and shell scripts
(Windows, OS X, Linux)
|Version 2.0 of the Ruby command-line “Sessions” framework.
Includes customisable options like: template sections and time-box length.
|Rapid Reporter||Windows .Net 3.5||Exploratory note-taking tool. Very slick with some nice integrated features.|
|SessionCreator||Java||Next step up from the original SBTM framework. Nice integrated interface. I like the built-in Review/Debrief workflow.|
|Session Tester||Java||Working prototype. Cool features. Looking for programmers to continue development.|
|Session Based Tester||Web app: Apache, MySQL, Perl||(I haven’t tried it yet – no comments)|
|SBTExecute||Java, MySQL||It’s Swedish – I translated the page to read it but haven’t tried it yet.|
TestExplorer is another tool that warrants special mention but it is not freeware. It is a Windows desktop application with a rich feature set. You may download it and “try before you buy” to see if it fits your needs.
3. SBTM Critical Success Factor – Senior Required for Test Lead
About six months after I started using SBTM at one company, I gave an overview presentation to the Development Team on how we do our testing. I wanted the developers and leads to be familiar with the new terminology that we used – for example, get used to hearing things like ‘sessions’ and ‘debriefs’ instead of words like ‘test plans’ and ‘test cases’. Most importantly, I wanted to get across the importance of an ‘uninterrupted session’. Distraction levels were generally high but after the presentation the situation improved as people started asking us if we were “in a session” before interrupting us to do or discuss something else.
Sometime afterwards, I talked with the VP of R&D to get some feedback about our testing approach. He was generally very pleased and said something that I hadn’t thought much about until that moment. He said that the success of our approach definitely depended upon having a good Senior person in the Test Lead role managing the testing effort.
Hmm. This made me think. It just so happened that I had many years experience as a Team Lead, and I was definitely Senior in terms of my testing skills and knowledge, so was it a coincidence that we were successful? Or were we successful because I had both the passion and experience required to succeed?
What are some of the things I do in the Test Lead role in Session-Based Test Management?
- I’m persistent and meticulous when debriefing the reports. The scan utility can scan the session sheets for missing or unexpected elements, but it can’t tell you that a tester has forgotten (for the umpteenth time) to select the correct Areas for their session. How do you handle that?
- For that matter, when I look at a session report, I always view it with an audit/review in mind. Would this session report provide enough information to stand up to an independent audit at some point in the future when we’ve moved onto some other project and no longer remember what we did on that particular day?
- Constant vigilance! (Note that it may not always be your requirement for high accountability, but if it is then this point is relevant.)
- Bridge the terminology gap between the Testing effort (estimation and progress) and the Project Manager (PM). We use a Risk-Based Test Strategy for a given release that generally slices the work in a different way from how the PM’s Work Breakdown Structure (WBS) describes it. How do you bridge that gap to ensure that everyone knows where we are? This is about more than just a Traceability Matrix, but yes, the Test Lead is the living Traceability Matrix between the tests performed and the Release Requirements. We also estimate effort in terms of ‘sessions’ while the PM talks in terms of ‘hours’ and ‘days’. Some math may be required.
- Help decide when ‘enough testing’ has been performed on an area and move onto a new area. A lot of factors and experience goes into helping you develop a ‘good feeling’ about when it’s time to move on and/or decide that a particular charter has been adequately covered to sufficiently reduce the risk you started with.
- The ‘risk’ I speak of here is the risk of ‘not knowing’. We test a particular area because we feel there are certain risks about it. As we test we learn about the features and the system, report any bugs or issues, and become confident in our coverage of that area.
- When you feel that the ROI of your time and resources will be better spent exploring a different area of the application or system, it’s time to move on. How do you make this decision?
- Mentor and Coach testers on test techniques, tools, and other ideas to help them improve the efficiency by which they can get their job done. I love this part. This is where I have the most fun and see the biggest improvements in the testing effort as a result.
- Review, audit and maintain the data to support the SBTM metrics. If you don’t stay on top of these, you can’t put much faith in the numbers produced. I like to regularly review these metrics throughout a development project to give me a general feeling of where we are and how we’re doing. I only share these metrics with my team members. No one sees them outside of the team. They’re for our use only, and that’s something else I do – protect the team by not leaking any metrics that might be misused against us.
There are more things that I could probably include, but the list above is a reasonable start. Is there anything there that I wouldn’t expect of a Test or Team Lead managing their testing effort using a different approach? No, not really. But that’s the point isn’t it? The Tools are a bit different and the steps or processes are not spelled out for you. Thinking is required.
I think it’s a valid requirement to say that someone in the Test Lead position needs to have some prior experience managing testing efforts. They should have a pretty good idea of how to adapt to a new way of working, how to adequately support the team as they work, and how to communicate with managers outside of the test team.
4. SBTM Critical Success Factor – Debrief Every Day
Don’t let these slip! When we first started with SBTM, there were times when we were extremely busy and putting in overtime hours that we would get behind on our session debriefs. It was easy to say that we could make better use of our time by doing testing rather than reviewing the testing already done. That makes sense, right?
Well actually, no, it doesn’t work that way.
Here’s the thing: session debriefs are like Code Reviews for programmers. Programmers don’t have to do them either, but when you stop doing reviews you often notice the quality of work start to decline. When we finally got around to reviewing the backlog of session sheets, we would often discover additional tests and risks that would have been worth exploring. You don’t really want to get these kinds of epiphanies after a release has gone out the door. It’s too late then.
The ‘debrief’ aspect of SBTM is more than just a quality check. It’s a complement to your Exploratory Testing effort by putting your respective heads together and revisiting the test strategy described in the Test Notes for elements of coverage, completeness, risk and repeatability. “Two heads are better than one” is the force at work here to increase your individual testing powers. Each team member benefits as the testing notes are shared and the knowledge helps build newer and better tests as you go along.
We never skip these debriefs anymore. Even when the going gets tough and deadlines are tight, we always make time every day to stop and debrief the sessions from the previous day. Personally, I like to do these first thing in the morning. It gives me a good idea of the problems we encountered from the previous day and lets me set or adjust the plan for the current day’s work.
There was a question I remember asking people during the time when I was a Quality Auditor years ago: “How do you know what you’re working on when you come in on any given workday?” It was a question to get someone talking about how they set priorities, where they get their tasks from, lines of communication, and so on. By doing the session debriefs in the morning, I know that I only have to worry about my session sheets being complete at the end of any given day. If I have to stay late, I stay late and finish up what I need to. But when I go home I don’t generally worry about my work priorities for the next day. I know that as soon as I come in, my session sheets will be reviewed and I will review other people’s sessions, and within the hour we will all know what we’re working on for the rest of the day. Risks discussed, test strategies confirmed or adjusted, priorities clarified, we’ve got the game plan… it’s time to Rock and Roll!
Don’t let these slip. Do the session debriefs every day.
5. Writing Good Test Notes Takes Time and Practice
The session sheet template has many important elements, none of which I would want to give up completely managing or tracking in some way or another. The single most important element for me has to be the ‘Test Notes’ section though.
When I review a session sheet, I know that I can ‘approve’ it when I can say to myself that I know more or less exactly what someone did for the ‘Duration’ listed in the sheet. Whether it is one hour or four, if I can’t say to myself that I know what this person has spent that whole time doing, then the session report is incomplete.
The analogy that I often give at times like these is that a session sheet is like a Science Report that you probably had to do at some point in elementary or high school. A typical Science Report has elements like the following: Objective/Purpose, Materials and Methods, Data and Observations, Conclusions or Inferences.
A good session report has all the same elements, although they may be laid out in a slightly different way.
The ‘Charter’ is the ‘Objective’. It sets the scope for the session and should help you know when testing is complete and it’s time to move on to something else. A Charter can be as specific or general as you like to help you get the information you need in the time allotted. Testing is for a specific purpose – what’s that purpose?
The ‘Materials and Methods’, ‘Data and Observations’, ‘Conclusions or Inferences’ all seem to fall into the “Test Notes” section. That makes this section pretty important. So how do you get good at writing Test Notes to give you all this information? The same way you get to Carnegie Hall – practice, practice, practice!
A good note-taker is like a good Sports Commentator with a twist. A Sports Commentator gives you the play-by-play of the particular game or sporting event that you are watching. A really good commentator also commentates. That is, provide you with background information on the players, the team, or some opinions about how or why certain things might have happened.
A good Test Note-taker does the same thing, and also needs to provide sufficient information to make their test session repeatable. Sounds simple enough, right? Ha! Try it.
You need to start by being clear about your ‘Materials and Methods’ (a.k.a. “The Setup”).
- What data did you use? Where did you get it from, generate it, transform it or store it?
- How is the system or application configured? What operating system, browser or hardware are you using? Any particular tools or automated scripts? And so on.
- What steps did you follow? What areas did you explore? What inputs did you use? What Test Techniques did you apply? Why? How are you keeping track of all of the possible test factors influencing the testing of a particular feature? (i.e. using a list or test matrix, etc.)
- Are there multiple scenarios that you need to cover to complete a particular charter? How are you keeping track of these? (e.g. Test Coverage Outlines) Were there any particular tools that you used to gather information during this session? And so on.
Next there are the ‘Data and Observations’.
- How are you capturing the information and data that you observe? Are you doing it manually or using a tool? If you are doing it manually, can you trust the consistency and repeatability of your measurements?
- If you are using a tool, are there any known bugs, problems or calibration issues that might produce faulty data? Do you even know how to use the tool properly to get the data you want? (Training may be an issue sometimes. Session debriefs should give you insights into these opportunities. Tool ‘calibration’ may also be an important consideration.)
- Can you chart the data to develop a new understanding or insight into what the system might be doing? Are you familiar with analysis tools and charts to help you work with the data?
- How do you know that what you are observing is correct? If you start a test with certain expectations or assumptions, what are those expectations based upon? Are they based upon some reference like a specification or requirement? Is that information current and up-to-date?
- Are your expectations based upon a conversation you had with someone about how they think a particular feature should work? Does the application match those expectations? Even if they match, does that mean what you see is correct? How can you avoid Type I and Type II errors?
- Are there any biases or influencing factors in your observations that lead you to a certain ‘expected result’? Do you know how you might be able to identify these or work around them?
- Did you really see what you thought you saw? Did you get a screen capture? Can a log file or database audit-type table confirm certain events happened in the order you thought they happened? Where is this information stored so that it can be called upon during a later review if required?
- Was an observed result repeatable? Can you make it repeatable? If not, it’s still worth mentioning. Highlight the fact that it wasn’t repeatable so that you might share your observations with others.
Finally there are the ‘Conclusions and Inferences’.
- Summarise what you have seen. There will always be the moments when you need to quickly skim a report to see if it contains the information you are looking for. Having a brief Test Summary or Synopsis either at the top or bottom of the test notes helps focus the reader to the important highlights required to complete that session’s charter.
- I’ve had teachers who have said that “nothing is ever conclusive, so don’t use conclusions – use inferences instead” and teachers who swear by “conclusions”. I think it’s a matter of timing really. I think you can draw conclusions based on the information you have at a given time to help you make a particular decision. How much you know about something changes over time, but at any given time you should be pretty certain about the information you have and what it means right now. I like to see comments like these scattered throughout the Test Notes. It tells me that the Tester is thinking about what they’re seeing and what it means at any given moment.
- A good tester can make their test notes better to read than a good book. I look forward to reading these good session notes and following along on their journey of discovery and exploration. That’s really cool.
- How do you feel about the session? If you have nothing ‘conclusive’ to say, is there something you can infer?
- Some opinion is always better than no opinion. At the end of the day, Test Leads, Managers, and staff in other departments don’t have the time to rethink everything that you’ve already gone through. (If we did, we wouldn’t need you then would we?) Tell us what you think it means to you in plain & simple language, how you feel about it, and what recommendations you might make based on the information you have so far. Now that’s a good way to end a session report!
The above elements may seem like a lot to put into your Test Notes. To the novice tester, it probably is. There’s a learning curve required as novices start off by putting too much information into the Test Notes section. To the more experienced tester, there are many shortcuts and assumptions that can be clarified in the notes to help you get to and focus on the important stuff. For example, I’ve read many good session reports that addressed all the elements and questions above and were only a few sentences long.
How do you know what’s worth writing and what’s worth skipping or implying? You need practice and a good Test Lead who cares about the quality and repeatability of the session notes you produce. It isn’t easy, but it’s a good habit to develop. Remember that Session-Based Test Management is also referred to as “High Accountability Exploratory Testing”. So if you’re not prepared to provide the ‘accountability’ then you might want to think about changing teams.
In the end, when you get good at writing test notes, I think you’ll have a lot in common with those good scientists who keep journals as records of the experiments they perform and the discoveries they make. This is applied Creative Writing. Put your ‘Thinking Cap” on because not only do you have to think to do your job, but you need to be your own personal commentator as you go along and do it!
6. ET.XLS Tips
The original ‘sessions.exe‘ archive includes a sample spreadsheet called ET.XLS that can be used to generate some interesting metrics based on the scanned session reports. It took me a few projects over the course of several months to figure out all the little details of how this spreadsheet works. The spreadsheet that I use now generates considerably more charts and information based upon the submitted reports than the original/default spreadsheet. I highly recommend that you play with this spreadsheet, get to know what the numbers and charts mean, and customise it to your own needs.
Getting Started with ToDos
To begin with I tried to understand what each of the sheets/tabs in the Excel Workbook did so that I could understand what data would be useful to me in my current context/situation. Not all of the details are documented and one item in particular took me a while to figure out was the “TODO” sheet titled “TODO Items in the Hopper”.
I didn’t know how to get data into this report. The sequence of steps seemed a bit odd, so here’s what I found out:
- Use the supplied ‘todo.xls‘ to come up with the list of topics and areas that you feel are important to cover during your testing project.
- Export or save the data to the ‘todos.txt‘ file. (My Ruby script above skips this step and just reads the Excel spreadsheet directly.)
- Use the ‘todo-maker.bat‘ file to generate empty session sheets that can be used as starting points for your charter sessions.
- The batch file creates these ‘et-todo-…ses‘ files in ‘c:\sessions\todos‘ folder by default
- When we use one of these ‘todo’ template files, we move them into the ‘c:\sessions\submitted‘ folder and work on them until they are complete and ready to be reviewed and eventually moved into the ‘approved‘ folder.
The ET.XLS file just never seems to detect any of the files in the ‘c:\sessions\todos‘ folder/hopper, so what gives?
It turns out that while step 3 puts the empty session reports into the ‘todos‘ folder, if you want these files to show up in the ‘hopper’ spreadsheet page, you need to move them into the ‘approved‘ folder. Then when you run the ‘scan-approved-then-run-report.bat‘ tool, it will detect the ToDo sheets and generate the required report that can be imported into the ET.XLS spreadsheet.
Once you know that you can make adjustments accordingly. In the end, I don’t use this spreadsheet to manage the ToDo hopper, I use another approach. It just took me a while to figure this tidbit out so I thought I would share it here.
Charts – Just the Beginning
The nice thing about having these numbers in an Excel spreadsheet is that you can then generate all sorts of charts and metrics to help you manage the testing effort.
The “Exploratory Testing: Test Sessions” chart on the first “Summary” worksheet/tab took me a while to really grasp. Once I figured out that it is essentially a productivity chart that shows you the cumulative number of sessions performed over time, I was able to use that information in a meaningful way.
A new chart that I include in my spreadsheet is the cumulative number of BUGS reported (according to the submitted session reports) over time. I then superimpose that data series on top of the Summary sheet chart so that I can see both the number of sessions and the cumulative number of bugs reported over time on the same chart. I now find this chart to be even more useful to me on the day-to-day management of our test projects.
I have other charts for coverage, but I keep those charts separate. I think I have more work to do when it comes to chart and metric analysis. The good thing is that I have the data and numbers to work with. I just need the time to sit down and sift through it.
7. Testing History at Your Fingertips
The session reports are like gold. We keep every session report we’ve ever created both on the network and stored safely away in a document control system. We haveinstant access to the testing notes going back to the initial release of our flagship software from several years ago. That’s amazing!
The first thing that really blew me away about SBTM was that all my test records are stored electronically as simple text files on the network. I’ve been doing QA and testing software since the early 90’s and this is the first time that my test results have ever, really, truly been completely paperless. Being something of an environmentalist that makes me somewhat pleased. The sheer efficiency of it is also marvelous for anyone who has ever done process improvement work before.
Talking about it is one thing, but I think an example is in order.
There was this one time when a Lead developer came up to me to talk about a particular feature in the application that we were testing. He asked if I knew how or when I tested it last because he couldn’t find any bug reports in the system relating to that feature. I used my ruby ‘search’ script to scan through all the archived session reports on the network. (BTW, a colleague of mine uses a visual grep utility to do the same thing – same result, different tool.) Within a minute, I had called up several session reports from a year before (3 releases back) that described the full extent of what we had tested, what we didn’t, and more importantly what bugs we had reported during those sessions. With that knowledge we were able to devise an appropriate test strategy to complete the testing of the updated feature in the current release we were working on.
It was surreal. Simply fantastic! I don’t ever recall having had the ability to search through my complete testing records for the last 3 years so quickly to bring up the exact moment and details of when I had reported specific bugs down to the minute. I challenge anyone using more ‘traditional’ test documentation approaches to live up to that kind of standard.
This is the information age, and “time is money.” If you can’t tell me what you’ve tested at any given moment over the last 3-5 years, how you tested it, who worked on it, what bugs you reported and any issues or problems that you encountered along the way, within minutes, then you need to take a serious look at the usefulness of your test management approach.
Text files aren’t glorious. They aren’t even particularly fancy. They are easily readable on every major operating system that I’ve worked with, though, they also compress really well in archive (zip) files, and they are easily searchable with a myriad of tools readily available on the internet. As a result, I can find out the Who, What, Where, When, How and Why of any testing activity our test team has ever performed since we first started keeping records several years and over a dozen releases back – all within Google time. That’s progress!
8. Mine Your Data – Create Test Guides
This isn’t a part of Session-Based Test Management, but is such an important complement to it that I feel I need to mention it here.
All these session reports you keep add up over time. So what do you do when the next major release comes around and you find yourself having to perform regression testing on the major features from the last release?
Well for a start, I wouldn’t recommend that you start exploring those features from scratch all over again. You already went through that exercise once in great detail and it probably took you anywhere from several days to several weeks to cover a particularly interesting or complex feature. Remember that SBTM is the framework to support your Exploratory Testing effort, and that ET is the simultaneous Learning, Test Design, and Test Execution of your application.
You already learned something when you tested a feature the last time, the first time you saw it. Testing the same feature over again will have a greatly diminished learning curve this time around. So how can you efficiently regression test a feature that took several weeks to cover the first time, in a timely manner this time?
Start by asking yourself what did you actually learn from your last testing experience? You need to compile the session reports that cover a particular feature or area of the application in order to see the complete picture. When you bring them all together, you can identify the facts that you need to develop a good test strategy starting point. You should be able to extract things like:
- How many testers did it take? How many sessions over how many days?
- Were there any particular system or environment configurations that you needed to test this feature or area properly? Any specific test data files?
- Additional equipment or hardware? Specific tools, scripts or software?
- What specific Risks were identified or did the Charters try to cover?
- What test techniques were used? Which ones were effective at finding bugs? Which were not?
- Did you have any checklists or tables to help you cover all the required elements or test factors affecting this feature? Do you have any reusable Test Notes?
- Were there any tests that you skipped last time because you ran out of time?
- What bugs did you discover and report? What kinds of bugs were they? (i.e. simple UI spelling errors, complex mathematical/computational bugs, security or performance bugs, etc.)
- Were there any issues or roadblocks that appeared while you tested this area that could possibly arise again? i.e. did focussing on the particular area put an excessive load or stress on the system that prevented you from continuing to test for any length of time?
Once you have these questions and answers in mind, it’s time to bring them all together into a new document – a document I call a “Test Guide”. This document summarizes the best part of the testing strategy employed and what you learned the last time you tested a particular feature or area of the application. We use MS Word documents for our Test Guides, we keep them simple & useful and anywhere from two to ten pages in length. Don’t use cover pages or Table of Contents or any of that fluff.
I should be clear that a Test Guide is just that – a guide – and contains no specific test cases. We sometimes describe interesting scenarios, but no test cases. I think I need to mention it again – NO TEST CASES! Individual test cases are useless unless used to illustrate an example of an application of a particular test technique.
Could a Test Guide contain Use Cases? Maybe. Are these use cases documented anywhere else? If so, then go and hit yourself over the head with your keyboard for asking such a question. A Test Guide wouldn’t be easily maintainable if you duplicated information that is better maintained elsewhere. Just identify the link or location of where to get the other document(s) and let someone else worry about maintaining the use cases separately from your Test Strategy documents. If the use cases that you discovered during your testing are not documented anywhere else, I would recommend you keep them documented separately anyway. If you have nowhere better to store them, put them at the end of the document in an “Appendix” and just refer to them in the right place(s) in your strategy guide.
When is a good time to work on creating these Test Guides? Good question. It depends. If you have to test a feature right away and you haven’t already got a guide, then just review the past session reports as a prelude to complete the testing you need to do now. Perhaps you can create a Test Guide draft while you go through the material the second time around.
Generally, we work on these Test Guides in that down-time period after a release has shipped and before the next development project is ready for testing. Some people might call this test planning. I would probably more accurately call it “reviewing what we have learned to do it better and more efficiently the next time.”
Now when we test an area that has been tested before, we just refer to the specific Test Guide in our session report Test Notes section. It’s clear and efficient and helps us focus on what’s important: learning something new and not relearning what we have already forgotten.
Have a question? Think something is missing? Drop me a line to ask any questions you may have about SBTM or my notes above.
COPYRIGHT NOTE: All content on this page is (c) Paul Carvalho. If you wish to reference anything on this page, please (1) indicate me as the author and (2) provide the link back to this page. You may NOT reprint any of this material in any format to be sold.
Great full & informative post – I feel its great place to start looking into exploratory testing.
Thanks for posting!
This was amazing. I am looking at moving our team’s process over to sBTM and this was a great guide.
Right on the money. The notes section is where the most ambiguity lies and you traced it to something familiar to a majority of testers. We are Mad Scientist in a lab making sure and experiment in coding is up to snuff.
Very good insights into how to be organized, building your test repository and connect the dots in Testing and Test Reporting.
Awesome… this article is very helpful for all who want to implement session based in their work.
Really like the way lessons learned placed by Paul. Cheers 🙂
I tried the Ruby Tools, but unfortunately they don’t seem to work properly. Are you still maintaining those (if not: ignore rest of this reply)?
Running Ruby193, I started by running the ruby-scan-submitted-only.bat (from console) and I got:
scan.rb:734: formal argument cannot be an instance variable
@sheets.sort.each do |@file|
Am I doing something wrong here, or is it because the code wasn’t maintained?
The Ruby Tools haven’t been maintained or updated in a few years, so they don’t work with ruby versions 1.9.3 and above. I am scheduling time to update the scripts this Fall (2014). Please use the GitHub repository for reporting issues and tracking progress. Development help is appreciated. Email me if you’re interested.
I appreciate the feedback. Cheers! Paul.
Awesome. Very nice article about building test repositories and mining our data