User Import Planning

The "User import planning" function is where .csv batch user file import jobs are configured and import log files are accessed directly in the GUI. Multiple client-specific import jobs can be configured that are either scheduled or manually processed and possibly restricted to user creation and/or updating.

The "User import planning" function is accessed via the "Admin > People > User administration" menu path where there are two tabs "User imports" and "Log file".

This function is intended for use by imc Consultants, but many clients that are familiar with batch file interface processes successfully utilise the function. Once in the function, it is possible to create new import jobs, edit or delete existing jobs, and access user import log files. This page will explain the options in detail.

Functions of the User Import Planning

How to Create a new User Import Job?

Creating a new user import job is quite simple if the required Personal attributes are already configured. To commence creating an import click the "Create" icon and a new browser tab will open. The configuration of user import jobs requires the completion of nine fields including an .xml mapping file upload.

Each field and setting in the "Description" tab is described below:

Identifier: This is the unique name of the import job, but not the name of the .csv file.
Client: A Client must be set which will become the imported user's "Master client" unless business rules are written otherwise. In a multi-client system the user import will also only be seen by users within the client is performing a manual user import in the "Users" function.
Attribute mapping: This upload field requires the creation of a .xml that basically maps .csv file column names to Personal attribute database names. Discussed in detail below.
Import data: This is the name of the .csv file to be used if the user import job will be scheduled as either "Daily", "Weekly" or "Monthly". If the scheduling is "Manual" then the file name is not enforced in the manual import process.
Separator for import data: Here the column separator/delimiter can be defined as either a TAB, semi-colon, comma, or pipe, and the file must match when importing. A Pipe is generally recommended if the source system is able to generate a .csv file with this separator.
Checkboxes:
1. Synchronisation of the import: When checked this setting will automatically deactivate any previously imported user that is no longer in the file on subsequent imports. Synchronisation is primarily used in cases where a source system is unable to send a termination status in an import as only "Active" records can be provided. Many clients use this setting, but it must be used with caution as poorly generated partial files can deactivate unintended users if they are not in the file.
2. Use encryption: Use encryption: For added security, user import files can be protected with PGP encryption. If enabled an encrypted file must be delivered that matches the encryption settings defined in the "Configuration > Key management" function.
3. Deactivate unclear users: Defines that all users that are mentioned twice in the .csv file will be deactivated. This is only true for users that have been created with the same import (verifiable via user attribute import id).
Import restrictions: Import jobs can be restricted to only "Create" users or "Update" users. The default value of "None" means there is no restriction active so the import job can both create new user records and update existing ones.
Implement as user: Allows selecting a user in the LMS on who's behalf the import will process and import log files will be sent.
Scheduling: A single selection setting that determines how and when the import job will be processed.
1. Inactive: The import job will not be available for both Manual and Automatic jobs.
2. Manual: Can be selected as an available import when importing in the GUI via the "Users" menu.
3. Daily: The import job will process daily at the time set in the CRON and the .csv file will require being loaded to an SFTP prior.
4. Weekly: The import job will process weekly at the time set in the CRON and the .csv file will require being loaded to an SFTP prior.
5. Monthly: The import job will process monthly at the time set in the CRON and the .csv file will require being loaded to an SFTP prior.

How to Create a .xml Import Mapping File?

The .xml mapping file is used to define the structure of the .csv file including how column names map to Personal Attribute database names. At the very least the user import file must include the mandatory Personal attributes in LMS as shown below, but the amount of mapped columns to Personal attributes is normally much higher. To create a new user the following are the standard mandatory attributes in the system to be included:

LOGIN (Can use generator)
FIRSTNAME
LASTNAME
EMAIL
EXT_ID_CSV (mandatory unique identifier for .csv imports)

There are many tools available to create .xml files including the freely available Notepad++. The format of a basic .xml mapping file covering the above mandatory attributes would look like below:

<CSV>

<mappingsourceField="Username" clixField="LOGIN"/>

</attributeMapping>

</CSV>

Note: The <CSV> tag is only needed if the .xml file is uploaded directly during an import in the "Users" function as custom mapping can be uploaded at the time of manual import. If the mapping file is uploaded in the User Import Planning as per this example then the CSV tag is actually automatically generated by the specified settings.

Advanced Mapping Settings

The .xml mapping can support some advanced such as:

ignoreUnmappedFields - Ignores any columns in the .csv file that are not mapped.
ignoreEmptyField - Configured per mapping line where "false" can overwrite a populated value with a blank and "true" will not overwrite a populated value with a blank.
hashident - Relates to a hash table identifier which is a conversion table in the registration business rules (described below). Commonly used for importing against selection lists to use they instead of imc ID or with date field format conversions.

An example of a more complex table is below:

<mappingsourceField="Username" clixField="LOGIN"/>

</attributeMapping>

How is the SFTP Setup?

Where import jobs are scheduled to automatically process daily, weekly, or monthly an imc consultant will be required to organise access to an imc supplied SFTP server. The customer is then responsible for loading the agreed .csv file name by the agreed time and a transfer job will transfer the file from the SFTP server to a specified location on the LMS content server (E.g. \data\person\personimport\src\staff). The SFTP server is managed by imc's Hosting team (ECMT) as no direct access is available to imc-hosted LMS servers for security reasons. The imc Hosting team will provide unique credentials for customers to access their SFTP account and access can be restricted by IP whitelisting if required.

What is a Cron Job and how is it Configured?

A cron job is an automatic scheduled job that can be configured to run at desired times and there are many cron jobs available in the LMS including "UserImport". The crons were traditionally coded by imc consultants and in 2022 there is the possibility to configure directly in the GUI via a new "Cron jobs" function. With cron configuration the sequence timing of jobs is important to ensure data is up to date as possible; e.g. the "UserImport" cron must run after the SFTP delivers the file and is recommended to run before "TargetGroupPersonAssignment" and "ParticipantAutomaticRegistration" jobs. With crons imc uses the quartz cron standard and the date/time formats are described here https://www.freeformatter.com/cron-expression-generator-quartz.html. In the example below the import would execute daily at 11 pm.

Can .csv Files be Encrypted?

For .csv user import files imc supports the use of PGP encryption where customers can load encrypted files to the SFTP server for extra security. The customer is responsible for the Private Key and must share a Public Key with imc that allows files to be decrypted at the time of import. These jobs can be configured by imc consultants from ILS14.13 and can be client-specific so that not every user import job is affected.

What are Registration/Business Rules?

When users are created or updated either locally (in GUI) or by import a series of business rules are triggered. The rules are written in .xml code by imc consultants to do many things based on Personal attribute values; this includes assigning users to groups (roles or business units) or clients, assigning job profiles, granting clearance on users to admin groups, populating settings such as language preferences or time zones, determining authentication status and other more advanced rules such as conversions. Typically business rules are written by consultants during implementation projects but can be updated at any time directly in the GUI either globally for all clients (Configuration) or client-specific (Client).

How to Access and Interpret User Import Log Files?

Log files are normally emailed automatically to the implemented user when an import is completed. However, they can also be accessed directly in the "User import planning" function in the "Log file" tab. Here a convenient summary is displayed in date order for each import that displays important overview information such as status, number of records, and importantly the number of incorrect records to address.

A detailed .csv log file can be downloaded by highlighting the desired import date and clicking the "Download" button to download a .zip file containing the import log file. When opening the .csv file contained in the .zip there is once again a summary overview provided and further down is a row for each user record that is contained in the user data file with an import status.

If there are errors use "Ctrl + F" and search for the word "Error" to locate all records that did not import. Then in each affected row click on column be to view the description of each error including the error code. The most common errors are normally related to login clashes, missing mandatory attributes such as Email, incorrect email formats, incorrect date formats (if there are date attributes), and duplicate entries in the EXT_ID_CSV column. A sample error message for a duplicate login is "Error, Unfortunately, the login name you entered is already assigned to another user. Please choose another login name. [ErrorCode:PRS0126]".

Import Error Troubleshooting Tips

Check if the required attributes are included in the mapping and .csv file.
Check coding and fieldSeparator defined in mapping.xml and the applied setting in the .csv file.
Check column names and attribute names defined in the mapping.xml, the .csv file, and the imc Learning Suite LMS.
Check registration rules and correct spelling of attribute names and if given hashtable references.

Hints & Tips

For configuration of complex imports including those requiring business rules and automatic scheduling please contact the imc Consulting team.
If using a scheduled import that is not synchronised it can be valuable to have a manual import that uses the same mapping and settings. This is ideal for testing and as a contingency in case the scheduled import is missed for some reason.
If using a Comma separator/delimiter the risk of there being a comma within a value is often quite high. In this case, the comma can be enclosed in quotations to avoid incorrect column processing.
Where logic can be conceptualized to complete a task based on attribute data it could be possible to write a registration/business rule. In such cases, it is best to speak with an experienced imc consultant by lodging a 2nd Level Support ticket.