Data Loading

Data Loaders are used to bring data into an xpna workspace from direct external sources in various formats. Data Loaders provide a flexible way to import data that may not be available through standard connections, such as data from SQL Server databases, FTP servers, Public API uploads, or CSV/JSON files.

What is a Data Loader?

A Data Loader is a configuration that defines:

• One or more sources where data originates from (e.g., SQL Server, FTP, Public API, local files)
• One or more source items that specify what data to retrieve from each source (e.g., specific tables, queries, files, or payload definitions)
• Targets that define how the retrieved data maps to xpna entities and dimensions (each target maps 1:1 to a source item)

Key Features

Data Loaders offer several capabilities:

• Multiple Sources: A single Data Loader can handle different data sources and types under one execution
• Flexible Data Formats: Support for direct SQL queries, CSV files, JSON files, and JSON payload uploads
• On-Demand Execution: Run Data Loaders manually via the connections panel when needed
• Scheduled Execution: Configure Data Loaders to run daily, ensuring data is always up-to-date
• Sample Data Testing: Test configurations with sample data before running a full load

Data Loader Components

To successfully configure a Data Loader, the following components need to be set up:

• Sources: Configure where data comes from (SQL Server, FTP, etc.)
• Source Items: Define what data to retrieve from each source
• Targets: Map the retrieved data to xpna dimensions and entities

Common Use Cases

Data Loaders are particularly useful for:

• Loading transactional data from legacy systems not supported by standard connectors
• Importing data from custom databases or data warehouses
• Loading data from CSV files exported from other systems
• Loading data via our Public API
• Combining data from multiple sources into a single xpna workspace
• Loading data that requires custom SQL queries or transformations

Note:

Data Loaders require workspace administrator permissions to create and manage.