Data Connector Configuration
Data Connectors serve as bridges between various data sources and target systems, automating the extraction and transfer of data through the use of plugins. They can either push or pull data from a source system to a target system. Their core functions include fetching, aggregating, transforming, and transferring data between systems.
The “Data Connector Configuration” allows you to manage which content areas (data sources) are included for indexing in the platform-wide search, as well as define when the data should be indexed.
To enable this, the data connector configuration provides three tabs for managing plugins, configurations and jobs.
Data Connector Plugin
Tab “Plugins”
A Data Connector plugin connect to its designated source system, fetch data, transform it if necessary and pushes that to its defined sink system - similar to pure ETL processes.
Here admins can manage the content areas considered for indexing for AI-enhanced content search. The content from only these content areas are available for searching by the learner.
Currently catalogue and channels (only if channel license is active) are provided as two content areas as standard offering and configured by default. Additional content areas / data sources can be added by requesting imc.
Data Connector Configuration
This tab allows users to
install plugins from the plugin repository
update an already installed plugin if a new version is available
Plugin repository
Plugin repository shows the plugins available for installation.
Following information related to a plugin is displayed:
Name of the plugin
Version of the plugin
Vendor of the plugin
Description of the plugin
Editing a plugin configuration
Currently the configurations for catalogue and channels (only if channel license is active) are provided by default.
This function enables the administrator to edit a plugin configuration. A plugin configuration represents properties that must or can be configured.
A Data Connector plugin is executed as asynchronous job based on its schedule and plugin configuration.
Configurations
Plugin Configuration helps to define configuration parameters required for the source and sink system.
By default, jobs have already been scheduled for the catalogue and channel (only if license is available) to run daily in the night. The job for channel plugin has been scheduled to run at 9 p.m. server time and for catalogue plugin has been scheduled to run at 10 p.m. server time. Please note the user interface shows the regional time zone. Hence it might show a different time than 9 p.m. or 10 p.m. depending on your location. If you want to ensure that the jobs run according to your regional time zone please update the jobs accordingly.
Configurations tab allows an admin to do the following functions:
add a scheduled job. This function is only available if a job has not already been added for the selected configuration. The job can be scheduled either one-time, daily or with a cron job expression.
edit a plugin configuration. In case a job has been already scheduled for an existing plugin configuration, the job will be updated with the recent changes. For jobs currently executed while updating a configuration, the modification takes effect only with the next execution.
delete a plugin configuration. Scheduled jobs will be deleted as well. In case a job is currently executing, that job will nevertheless run to completion.
Plugin and configuration is automatically created for channels and catalogues. Hence it is recommended not to edit/delete them.
For channels, by default, the service account user in the channels plugin configuration has been set to noreplysuper@im-c.de. If automatic scheduling of channel plugin job fails, it is important to enter a valid super user’s email address as the service account user so that the channels plugin has access to the channels microservice.
Full-indexing
After the switch of the semantic to lexical search or vice-versa, full-indexing is required for all the plugin configurations to take effect for the user. Please note the full-indexing might take a lot of time depending on the size of the data to be indexed. So it is recommended to run the job in the night.
To do full-indexing of the catalogue plugin, edit the plugin configuration and set the property “Number of days to sync back” to 0 (already the default).
To do full-indexing of the channel plugin, edit the plugin configuration and enable the property “Full Synchronisation”.
Jobs
Jobs tab allows an admin to do the following functions:
edit a scheduled job
pause a scheduled job
resume a scheduled job
delete a scheduled job