Data connector configuration
Data Connectors serve as bridges between various data sources and target systems, automating the extraction and transfer of data through the use of plugins. They can either push or pull data from a source system to a target system. Their core functions include fetching, aggregating, transforming, and transferring data between systems.
The “Data Connector Configuration” allows you to manage which content areas (data sources) are included for indexing in the platform-wide search, as well as define when the data should be indexed.
To enable this, the data connector configuration provides three tabs for managing plugins, configurations and jobs.
Data connector plugin
Tab “Plugins”
Here admins can manage the content areas considered for indexing for platform-wide search. The content from only these content areas are available for searching by the learner.
Currently catalogue and channels are provided as two content areas as standard offering. Additional content areas / data sources can be added by requesting imc.
This tab allows users to
install plugins from the plugin repository
add a plugin configuration
delete an already installed plugin. A plugin can only be deleted, if there are no jobs available for this plugin configuration.
update an already installed plugin if a new version is available
A Data Connector plugin connect to its designated source system, fetch data, transform it if necessary and pushes that to its defined sink system - similar to pure ETL processes.
Plugin repository
Plugin repository shows the plugins available for installation.
Following information related to a plugin is displayed:
Name of the plugin
Version of the plugin
Vendor of the plugin
Description of the plugin
Adding/Editing a plugin configuration
This function enables the administrator to add/edit a plugin configuration. A plugin configuration represents properties that must or can be configured.
A Data Connector plugin is executed as asynchronous job based on its schedule and plugin configuration.
Configurations
Plugin Configuration helps to define configuration parameters required for the source and sink system.
Configurations tab allows an admin to do the following functions:
add a scheduled job. This function is only available if a job has not already been added for the selected configuration. The job can be scheduled either one-time, daily or with a cron job expression.
edit a plugin configuration. In case a job has been already scheduled for an existing plugin configuration, the job will be updated with the recent changes. For jobs currently executed while updating a configuration, the modification takes effect only with the next execution.
delete a plugin configuration. Scheduled jobs will be deleted as well. In case a job is currently executing, that job will nevertheless run to completion.
Plugin and configuration is automatically created for channels and catalogues. Hence it is recommended not to edit/delete them.
For channels, it is important to configure the service account user (super user’s email address)in the channels plugin configuration so that the channels plugin has access to the channels microservice.
Plugin and configuration is automatically created for channels and catalogues. Hence it is recommended not to edit/delete them.
Full-indexing
After the switch of the semantic to lexical search or vice-versa, full-indexing is required for all the plugin configurations to take effect for the user. Please note the full-indexing might take a lot of time depending on the size of the data to be indexed. So it is recommended to run the job in the night.
To do full-indexing of the catalogue plugin, edit the plugin configuration and set the property “Number of days to sync back” to 0 (already the default).
To do full-indexing of the channel plugin, edit the plugin configuration and enable the property “Full Synchronisation”.
Jobs
Jobs tab allows an admin to do the following functions:
edit a scheduled job
pause a scheduled job
delete a scheduled job