Microsoft Dynamics 365 Business Central and Databricks integration

What moves between them

The flow runs one direction, from Microsoft Dynamics 365 Business Central into Databricks. ml-connector reads vendors, customers, purchase and sales invoices, general ledger entries, items, and dimensions from Business Central and writes them into mapped Delta Lake tables in a target catalog and schema. Each Business Central entity maps to its own table, and the BC number or id becomes the business key for upserts. Master data such as vendors, customers, and the chart of accounts refreshes on a daily cadence, while transactional entries can run more often. Databricks holds no records Business Central needs, so nothing is written back to the ERP.

How ml-connector handles it

ml-connector stores both credential sets encrypted. On the Business Central side it requests an Entra ID client-credentials token for the api.businesscentral.dynamics.com scope and builds the base URL from the stored environment name and company id. On the Databricks side it requests an OAuth machine-to-machine token for the service principal and refreshes it before the one-hour expiry rather than waiting for a 401. Reads are incremental: it filters on lastModifiedDateTime greater than the last successful run and pages through odata.nextLink, and where a BC subscription is registered it takes the change notification as a trigger, then fetches the record because the notification carries no data. Subscriptions expire after three days, so a renewal job runs before expiry, and purchaseOrders is polled because it has no webhook. Rows are loaded into Delta Lake through the SQL Statement Execution API against a running SQL warehouse, with a MERGE on the BC business key so a re-read record updates in place instead of duplicating. Where a Databricks Job performs the load, the idempotency token guarantees a retried submission reuses the existing run. Both sides return HTTP 429 under throttling, so ml-connector backs off with jitter and retries, and every record carries an audit trail and can be replayed if a load fails.

A real-world example

A mid-sized distribution company with roughly two hundred staff runs Microsoft Dynamics 365 Business Central for purchasing, sales, and finance, and its analytics team uses Databricks for margin and spend reporting. Before the integration, an analyst exported invoice and GL extracts from Business Central to CSV every week and uploaded them by hand, so dashboards lagged the ledger by days and a missed export broke the numbers. With Business Central and Databricks connected, vendors, invoices, and general ledger entries land in Delta Lake tables on a schedule, keyed on the BC document number, so the warehouse reflects posted finance data without manual exports. Reporting runs on current figures and the weekly upload step is gone.

What you can do

• •Land Microsoft Dynamics 365 Business Central vendors, customers, invoices, and GL entries in Delta Lake tables.
• •Read only changed records using an OData lastModifiedDateTime filter and Business Central change subscriptions.
• •Bridge Entra ID client-credentials auth on the ERP with Databricks OAuth machine-to-machine tokens.
• •Upsert each entity on its BC business key with a Delta MERGE so re-read records never duplicate.
• •Refresh master and transaction tables on a schedule you set, with retries and a full audit trail.

Questions

Which direction does data move between Microsoft Dynamics 365 Business Central and Databricks?

It moves one direction, from Business Central into Databricks. Vendors, customers, invoices, general ledger entries, items, and dimensions are read from Business Central and written into Delta Lake tables. Databricks is a data platform with no invoice, vendor, or GL object of its own, so it is treated as a destination and nothing is written back to the ERP.

How does ml-connector write data into Databricks if the REST API does not load rows?

The Databricks REST API manages table metadata, not row data, so ml-connector loads records through the SQL Statement Execution API against a running SQL warehouse, or through a Databricks Job. It runs a MERGE keyed on the Business Central business key so a re-read record updates in place. Where a Job performs the load, the idempotency token ensures a retried submission reuses the existing run instead of creating a duplicate.

How does the integration keep the warehouse current given Business Central webhooks carry no data?

Business Central notifications signal that a record changed but include no payload, so ml-connector fetches the changed record after each notification. It also runs an incremental poll filtered on lastModifiedDateTime to catch anything missed, since purchaseOrders has no webhook and subscriptions expire after three days. A renewal job re-registers subscriptions before they expire so the change feed does not go silent.