We just had Abila support install the latest Service Pack in a customer environment and the client asked us a question that we get asked quite frequently – “What should we test?” “How long should we take to test it?” “We just finished a whole bunch of testing for the last major release and now we have to go back to our end-users and ask them to test some more?” (OK, they didn’t ask ALL those questions, but those are all questions we have heard repeatedly in the past when it comes to testing upgrades, service packs and hot fixes for netForum as they are released by Abila)
Like most things in IT, while there unfortunately isn’t a “right” answer for every customer and every situation, there are certainly some best practices and rules of thumb to help you understand what level of testing effort is right for your Association. Let’s start on the small end of the spectrum…
Abila will release a “hot fix” more frequently than Service Packs or major releases when they determine that a “bug” is critical enough for a short-term fix. In other words, hot fixes usually contain solutions to very unfortunate and troublesome bugs. Traditionally these have been fairly straight-forward to test. You would first view the release notes for the hot fix that explains all the updates and changes included (http://kb.abila.com/knowledgebase?f=field_kb_product_line%3A16951) and answer the following questions:
- Does the hot fix apply to your environment or business processes? Do you need any of the fixes that were outlined in the release notes for the hot fix? Note that you do NOT HAVE TO install a hot fix! Abila might recommend that you stay on the latest version – and generally that is a good idea – but hot fixes are released frequently and could result in quite a bit of testing for you and your team. Note that hot fixes are cumulative; so if you decide to skip one now, the next one will include all the hot fixes from your current version to the version you are having installed. You can always “catch up”.
- If you are going to apply the hot fix, what specific business processes or test cases do you need to test based on the release notes? And keep in mind here that you might need to do a level of “regression testing” (which is defined as “the process of testing changes to computer programs to make sure that the older programming still works with the new changes”*). For example, if the hot fix contains a fix for how payments are applied to invoices with discounts, you may not only want to test apply a payment to an invoice with a discount line item, but also test things like: percentage-based discounts as well as flat discount amounts; discounts on events, products and memberships (depending on what your business sells and discounts); test a mock month-end close to verify the impact of the discounts all the way to your general ledger, etc.
You can generally get a hot fix tested in a few days to a week depending on how thorough you want to be. Hot fixes will be applied to your TEST or DEVELOPMENT environments by Abila support (so it’s always a good idea to have a recent copy of your LIVE database applied to your testing environment BEFORE having Abila apply/install the hot fix), and after your team has tested everything required for the hot fix, Abila will install the hot fix in your LIVE environment.
In the 10 years that we’ve been working with netForum, only once did a netForum Service Pack contain primarily new features. Service Packs are typically a larger number of bug fixes that have all been packaged, tested and QA’d together. Whereas hot fixes might come out once per quarter (or more frequently), there might only be 2 service packs in a given calendar year for each major release. As a result, service packs should be tested much more thoroughly and detailed by your team…
In the past, we’d be comfortable with having Abila apply a service pack to a test environment and then just having power-users bang away on the new version for anywhere from 3-6 weeks. We would ask them to test “critical” business processes in addition to any fixes specifically identified in the release notes that we were looking for. This approach has worked well for our clients (and is still a better approach than not doing any testing at all on a service pack) up until the 2013 release. Unfortunately, over the past few years Abila has been experiencing quite a few “QA-related” issues (e.g. something that was fixed in a previous release being broke with the latest service pack), and these QA issues mean that, until Abila can give us a comfort level that they have resolved their QA issues, service packs should be tested almost like a major release (outlined below). This is a lot to ask for our clients and can result in frustration and burnout of power-users and/or testers – so, a compromise or happy-medium is usually reached and understood by all involved. Obviously, having your test cases documented and having testing tools in place will help manage these testing efforts and make it easier on all involved (and The Square Pegs can certainly help with developing, documenting and/or making your testing efforts more efficient and effective – just drop us a line!).
Major releases of netForum come out almost every year – e.g. 2013.01 and 2014.01 were “major” releases. Major releases may contain fundamental changes to the software (e.g. the 2016 version will introduce a new iWeb User Interface and a new backend database server, SQL Server 2016, among many other large changes to the system). As a result of the scope of changes involved with a major release, clients should basically test these releases until their fingers fall off and their heads explode! Ok, not really…but, you get the point. Major releases need test cases defined, meta-data reviewed prior to the installation of the release in your test environment, testers defined and their calendars cleared (as much as can be) for the testing effort, executive leadership on board with the level of effort from the team, integrated systems lined up and ready to regression test, etc. Expect to take 2 to as much as 6 months (maybe even more depending on if “show-stopping issues” are identified in testing that need to be addressed by Abila before you can install the major release in your live environment) testing a major release. You can almost think of this as a “mini” implementation, and if you need help, reach out to the Square Pegs as we help customers with major release upgrades all the time (even if you just need to re-compile some customizations that you, Abila or an partner built for you)!
Things to keep in mind for all upgrades – whether hot fix or service pack or major release:
- Always try and update your TEST or DEVELOPMENT environment from your LIVE environment PRIOR TO installing and upgrade (both your back-end database and web server file system – this will help make sure you won’t be surprised when the upgrade is applied from your test system to your live system.
- If you have the time, it’s always a good idea to document your test results – using a tool such as Microsoft Test Manager, or even just Microsoft Excel can help manage your testing efforts both large and small.
- Don’t forget to regression test! While Abila makes every effort to not break something that was previously working, this does happen…so, don’t just test the single scenario outlined in the release notes – think about your specific business processes and test the critical RELATED processes at a minimum.
- Keep your metadata and customizations in mind. netForum does a fantastic job of keeping your customizations intact with baseline upgrades and new features – but if something is more likely to break during an upgrade, it’s usually something you have customized.
As a side-note – Abila has been working on AUTOMATED TESTS for netForum to help with their QA-related processes. While these automated tests would be a valuable resource for Abila customers to help with these testing efforts, Abila has decided to NOT make these automated tests available to their customers due to the level of investment required to build these tests as well as the variety of each clients’ netForum environment and customizations. Square Peg Services has launched an informal effort for the netForum community to try and build our own Automated Tests based on the forth-coming 2016 release (with the new iWeb user interface). If you are interested in this effort, or if you would like for one of the Square Pegs to help provide some specific automated tests for your environment, please reach out to us at email@example.com for more information. Happy Testing!
Jeff M. – SPS Owner/Consultant