Testing, test strategy, test planning, and a good paper-trail help Admins support their employer when entertaining updates, upgrades, new apps from the AppExchange, accepting developer work-product, or migrating to Salesforce. This article is for anyone who would answer yes to some of the following:
- Do you work in a large environment with developers periodically introducing new custom code?
- Do you use App-Exchange or third-party products?
- Do you have a highly customized environment such that migrations to new Salesforce releases present concerns?
- Are there new applications in your org that bring custom objects with parent-child relationships to standard objects?
- Are there applications that add custom fields to key standard objects, or make updates to standard and custom fields in critical standard objects?
- Are you planning a migration from Classic to Lightning, with custom built, App-Exchange, or vendor products in your org?
- Is your company migrating to Salesforce? Will there be large amounts of data to import? How clean is the data? How long will imports take? What will the error and failure rates look like?
- Has anything ever broken in your production instance? And if not, does this possibility worry you?
If you are saddled with the challenge (opportunity!) of testing a significant change to the Salesforce environment you are responsible for managing, the question then becomes, “How do I test it?” Strategies outlined here are intended to help Admins answer this question.
Test Strategy
When developing a strategy for testing a change to your Salesforce environment there are key elements of test strategy to be considered:
- How do I plan this?
- How do I make the change work?
- How do I make it fail? Does it fail easily? Is it durable?
- Does the change conform to a specification? Is there a specification?
- What are the customer and end-user goals and expectations?
- What business problems does this solve?
- How does this change behave when started and stopped?
- How does this change perform under load?
- Performance measurement and resource utilization?
- What tools and resources do I need to test this effectively?
- What tools does Salesforce have to help with testing?
Following are key considerations when developing a strategy to test changes to your Salesforce environment.
Make it work
This is an ad-hoc form of testing intended to understand what it takes to make the software work. Most software products are not entirely perfect. There are points of failure. It is important to know and understand the normal behavior of a solution, how it behaves when used as intended, how easy or difficult it is to use.
Make it fail
This is the self-evident opposite of “try and make it work”. As stated, very few software products are perfect and the questions to answer are how and where is it vulnerable? How can it be made to fail? How easily does it fail? What are the kinds of things users may normally do that create a risk of failure. Good testing will expose all the risks and leave no surprises or unexpected consequences.
Conformance to specification
This form of testing requires an understanding of what the software is intended to do, and to write and execute test cases that effectively prove whether the software does what it was intended to.
Specification may exist in formal documents, or less formal means such as emails or meeting notes. If specification documents do not exist, it may be necessary for the person doing the tests to write a memo of understanding of what the software is intended to do, and how test cases will measure whether the specification objectives are met. If contractors and consultants developed the software there are likely specification documents, contractual agreements, timesheets, or invoices that may contain useful information about specifications and requirements.
Customer goals, expectations, complaints, prior issues
In a Salesforce environment, information about goals and expectations may be easier to obtain. The Admin and end-users probably work at or for the same company. In a perfect world, where none of us live, there would be strict consistency between customer goals, expectations, prior issues; and the software designs and specifications. This is often not the case.
See the picture below. For this reason a good test strategy will consider formal specifications as well as what the customers, or internal users, expect from the application, and whether the software will address prior issues and complaints.
Startup and shutdown
While Admins do not directly control the initialization and shutdown of a Salesforce instance, there are a great many things in a Salesforce environment that start, get started, terminate, end, or shutdown.
Software products behave differently during startup and shutdown, compared to normal operation. Initialization and shutdown often exercise elements of software that are not normally strained during regular operation. The Admin, or accidental test engineer, must examine the solution under test for instances and actions that cause something to start or stop. This would include actions such as bringing up and ending an application, invoking and exiting a user interface, refreshing dashboards, generating reports, or logging in and out of an org.
Quantify performance and resource utilization
Performance measurement is an essential element of software quality assurance. The objective is to quantify and define baseline behaviors such as wall clock times, file sizes, overall performance, how the application may affect what other users are doing, does it slow other parts of Salesforce down significantly or in a way that may trigger users’ complaints?
A frequent result of new implementations is unexpected increases in the amount of disk storage space an application uses. Admins know that Salesforce charges customers based in part on storage space utilization.
Salesforce Optimizer reports can be consulted to help understand changes in the number of API calls, validation rules, flows, etc.
Consult the following Salesforce Setup sources when constructing a baseline of system performance and behavior:
- Salesforce Company Information
- Release Updates
- Duplicate Error Logs
- Salesforce Optimizer
- System Overview
- Health Check
- Network Access
- Storage Usage
- Critical Updates
- Login History
Stress testing is a key element of performance measurement. Most of the Salesforce Apps I have tested can be made to fail under some form of load and stress test. This is an expected result. The question to answer is how much load and stress does it take to make an application fail and is the scenario likely in a user’s environment.
The work product from this effort is a statement defining the baseline performance and resource utilization of a Salesforce instance, how introduction of a new application changes that performance profile, and how the system tolerates loads and stresses.
These named test strategies are intended to help the Admin test significant changes and additions to a Salesforce instance. It would be important to understand the scope and scale of changes that are introduced to the environment so an appropriately sized and defined test strategy can be developed.
Tools of the Trade for Salesforce Testing
You will find multiple tools that can support you when testing, including several indigenous tools and features in Salesforce:
- Salesforce Setup information displays
- Salesforce Case management
- IEEE Software Engineering Standard 829
- Test data generators
- Your own wits
Salesforce Case Management is a great tool for logging issues during tests.
IEEE Software Engineering standard 829 has document templates and processes for test plans and test case procedures. Everyone I know in this business has their own opinion on how to best write a test plan, and everyone thinks their idea is the best. It can be difficult to reach consensus on how to structure a good test plan.
IEEE represents decades of accumulated software engineering expertise. Their standards and procedures documentation carries heft and credibility, making it easier to reach consensus on how test plans and test cases should be designed, written, and executed. These templates are a great model for writing up test plans and test case procedures. Using a universally recognized authoritative source for developing software test plans and test cases helps guide test strategy and lends credibility to the Administrator’s efforts.
If test data is needed, such as a large volume of accounts or contacts, there are many test generators available such as this one.
Key Considerations for Testing
Write up a plan. Tell people what you are doing and planning. Using IEEE standard format test plans and cases can be advantageous.
Think about what information is needed to develop a good plan for testing and where it can be found. Think about what does and does not need to be tested.
Size and scale the scope of the plan to the size, scale, and scope of the problem. Don’t shoot a fly with a blunderbuss, and don’t perform lightweight testing against a heavy or complex application.
Capitalize on the many Salesforce tools and resources to help you do the job.
When developing plans to test changes to your software instance, think about how to make it work and make it fail. Measure performance before and after. Verify it works as intended, and that it solves the problems end users expect it to.
Summary
This article is intended as a guide to help the Admin and “Accidental Software Test Engineer”. When greeted with the need to test a new application, migration plan, or consultant’s development work, I hope this article will help provide a roadmap and strategy to help you meet that challenge!
Comments: