6 Steps for Successful LMS Selection

LMS selection and purchase decisions are high-stakes because choosing the wrong vendor will haunt your organization for years to come. A successful selection process requires a selection team that has a shared vision of the business requirements and is wary of product marketing material. Unsuccessful teams take vendor demos and marketing materials at face value, failing to perform rigorous testing and validation against their organization’s business requirements. Here are six simple steps to sort LMS product fact from fiction, and use this information to make the best choice for your organization.

1. Verify vendor marketing materials and demos

Let’s face it. Software vendors have a strong incentive to inflate product features in marketing materials and demonstrations. They’ll throw buzzwords at you, like “cross-listing,” “gamification,” and “eCommerce,” but you can’t assume that features listed in marketing are designed and implemented to meet your specific business needs. For example, does “eCommerce” mean the LMS supports your approved payment processor? Will your existing eLearning content display correctly in the LMS’s “Mobile Learning” app? There is no way to know unless you see the feature demoed and tested with your content and business case.

Keep in mind that software capabilities are not always black and white. The current version of the system may not include a feature demonstrated during a marketing presentation. Vendors may add features in their marketing that are either “on the roadmap,” “in beta testing,” or created for the narrow requirements of one client that are unlikely to overlap with your own. The bottom line is that vendors write feature lists so that they check every box in buyer RFIs or RFPs. It’s rare that a vendor will blatantly misrepresent capabilities. They will, however, take advantage of gray areas to present the most favorable picture of their product.

2. Build a selection team with the necessary experience

LMS vendors have a natural advantage over LMS buyers because they sell every day. LMS buyers, on the other hand, typically evaluate and choose an LMS system once or twice in a career. The danger is that the selection team will not have the experience to ask the hard questions that will lead to the best decision. Moreover, an inexperienced selection team may be tempted to use the vendor salesperson as a consultant or advisor.

To address this potential imbalance, you need to staff your team with as much experience as possible. Learning Department or HR leaders should lead the team. The current LMS Administrator should support these leaders. In larger organizations this role may be shared, so choose the person with the most hands-on experience. Consider including a representative from Procurement. Procurement often has the most experience dealing with vendors, and they understand the organization’s procurement process. Finally, include a representative from IT. Ultimately, the IT organization will support the LMS after implementation. They will provide critical knowledge to assess technical requirements related to the API, single sign-on and many other points of integration with other enterprise systems. If you don’t have the necessary internal expertise, you may need to look outside your organization for a consultant.

Outside consultants often bring a level of experience and objectivity that helps keep the decision focused on requirements. A common pitfall for selection committees is getting too focused on the “cool” features at the expense of fundamentals. Gamification and virtual reality have exciting potential, but they shouldn’t distract from ensuring that the vendor has fully developed more mundane features like reporting and user management and that these features support the business model. An outside consultant can help cut through the hype, identify the core features needed to meet your business goals, and set up a process that validates essential capabilities.

3. Partner with IT, but clarify roles

In larger organizations, the selection committee includes an IT representative. IT is a valuable partner in the budgetary, selection, and implementation process. Leaning too heavily on the IT department to develop detailed business requirements, however, is a common mistake. The Learning organization should take the lead on requirements and generate a comprehensive list of issues with the current system, that the new system will correct.

To be sure, there are areas where IT needs to take the lead. For example, vetting the API documentation, choosing between different Single Sign-On methods, and integration with existing systems are areas where it is best practice to defer to your IT partner.

When selection is complete, and implementation begins, IT’s role will further expand. Make sure you have a well-developed agreement with IT about implementation roles and responsibilities. A common understanding will be necessary to establish budgets and schedules for the implementation phase. During the selection process, it is also wise to discuss and plan who will take the lead in project management. Vendor communication, implementation meetings, and trouble ticket follow-up will likely require a full-time commitment for many months.

There will be hurdles during implementation no matter which LMS you choose, so spend the time during the selection process to build relationships with IT that will ensure a smooth implementation and successful rollout.

4. Create and document detailed business requirements

A detailed business requirements document will go a long way towards ensuring your new LMS will support your business needs. Failure to adequately document requirements, or writing requirements that are not sufficiently detailed, will increase the risk of overlooking important gaps in system capabilities and features.

Be specific. A high-level requirement such as “eCommerce” is so broad that the selection team may interpret underdeveloped capabilities as meeting the requirement. Instead, lay out exactly what needs to happen in words used by the business. For example, “User must be able to purchase course access with a credit card via PayPal and Authorize.NET.” Adding this additional detail allows you to test actual functionality instead of just checking to see if the feature is listed.

Be thorough. Describing use cases for each requirement will help you identify the needed capabilities. For example, “A new hire is added to the system via API. They are assigned the onboarding curriculum automatically, based on the placement of the new hire in an organizational unit during onboarding.” Analyzing this use case allows you to extract specific needs you might otherwise overlook. For example, it suggests the need for API integration, curriculum support, organizational units, and automatic assignments. As you flesh out each use case, you will uncover additional business requirements that are critical to your decision-making process.

Without specific, detailed requirements, you won’t be able to vet the LMS during testing adequately. A little extra documentation effort will speed up the testing process and ensure that it reveals important issues before a contract is signed.

5. Get your hands dirty

We started this list with the need to “Verify vendor marketing materials.” We’ll assume you have experienced the vendor demo and verified that the features important to your organization are in the current version of the product. Now it’s time to use your detailed requirements and use cases to perform hands-on testing.

You will start by establishing an internal testing and validation protocol that ensures that the product you buy has features that are fully implemented and are designed to support your specific needs. First, prioritize your business requirements so that you test the most critical capabilities first. If you are choosing with time pressure, you may not have time to check all requirements. Next, create a test script based on use-cases identified earlier. (If you are not familiar with creating test scripts, check out our earlier post on designing test scripts for eLearning QA. Many of the concepts will be similar.)

Structure test items to express results with a standardized scale (Yes or No, 1-10, etc.). Subjective responses can lead to confusion when comparing results between systems. Finally, test each requirement from the perspective of user experience, administrative functions, and instructor functions. Evaluate workflow and usability for each role based on how the experience impacts time and ultimately the total cost of ownership.

When it comes time to conduct testing, do your best to carry out the test under real-world circumstances, or as close to it as possible. Realism may mean populating the LMS with dummy user data, using actual courses, and asking testers to use a variety of browsers and network configurations.

6. Resolve issues before you buy

Your negotiation power and influence with the vendor is at its peak during the selection process. Try to resolve outstanding issues before you sign on the dotted line.

The testing process should narrow your list to 2 vendors. Use the issues identified during testing to help you make your final selection. Press hard to resolve the critical problems to your satisfaction before the purchase. Don’t trust that the vendor will release a future version or patch that fixes a critical flaw. You may choose to let low-priority features slide if they aren’t quite “there” yet, but never assume that a patch will correct features that are not currently working after you complete the purchase. If the vendor promises to add a capability after the date of purchase, consider adding a clause to the contract that obligates the vendor to deliver and permits your organization to exit the contract or receive a discount on license fees if promises are unmet.

Hopefully, this article will help you avoid some of the pitfalls of LMS selection. There is no perfect LMS system. Every system has shortcomings and deficiencies. The trick is to identify these issues before you make a purchase decision and take action to ensure you get the most value for your investment. There are no shortcuts. Only a rigorous selection process based on business requirements and validated through testing will avoid unpleasant surprises discovered after the purchase contract is signed.