SYSPRO and Databricks integration

What moves between them

The main data flow runs from SYSPRO into Databricks. ml-connector polls SYSPRO's OData endpoint on a schedule (recommended every 5-15 minutes for financial data) and reads new and updated invoices, purchase orders, supplier records, GL accounts, and GL balances, filtering by timestamp fields such as PostDate and InvoiceDate. Each batch is written into Databricks tables in the appropriate Unity Catalog schema, partitioned by company and date for easy analytics. Because both systems are pull-only from SYSPRO's perspective, ml-connector does not write back to SYSPRO.

How ml-connector handles it

ml-connector stores SYSPRO and Databricks credentials encrypted. For SYSPRO, it obtains and refreshes the session token by calling Utilities/Logon with the operator credentials, then polls OData endpoints with timestamp filters to capture changes. For Databricks, it authenticates with OAuth2 client credentials and obtains a bearer token valid for 3600 seconds, refreshing as needed. The two systems have very different data models: SYSPRO's operational and financial entities map into Databricks tables with denormalized columns for easy SQL querying. SYSPRO's SQL session timeout requires retry logic, and Databricks rate limiting is handled with exponential backoff. Every record carries a full audit trail and the exact SYSPRO timestamp it came from, so downstream reports always know the source.

A real-world example

A mid-sized discrete manufacturer runs SYSPRO for procurement, inventory, and accounting across three plants and uses Databricks for financial analytics and supply chain dashboards. Before the integration, the finance manager exported AP and GL reports from SYSPRO monthly as CSV files and loaded them into Databricks manually, then spent hours reconciling invoice counts and amount totals. With SYSPRO and Databricks connected, purchase orders and invoices flow into Databricks automatically every 15 minutes, GL balances are always current, and the finance dashboard shows real-time cash flow and vendor aging without manual extraction.

What you can do

• •Replicate SYSPRO suppliers, purchase orders, and AP invoices into Databricks for vendor and cash flow analytics.
• •Copy SYSPRO AR invoices and customer records into Databricks for revenue reporting and aged receivables analysis.
• •Sync SYSPRO general ledger accounts and balances into Databricks, refreshed on a schedule, for real-time GL reporting and variance analysis.
• •Poll SYSPRO on your schedule with timestamp filters to capture only new and changed records, reducing load on both systems.
• •Store encrypted credentials and manage OAuth2 and session token lifecycle so data flows without manual intervention.

Questions

How does ml-connector handle SYSPRO's lack of webhooks?

ml-connector polls SYSPRO's OData endpoint on a schedule tied to your accounting cycle, typically every 5-15 minutes for financial data. It uses timestamp filters on fields like PostDate and InvoiceDate to fetch only new and changed records, minimizing load and keeping Databricks current without relying on a push from SYSPRO.

Can ml-connector write data back to SYSPRO from Databricks?

No. SYSPRO's OData layer is read-only, and Databricks is not a transactional system. ml-connector moves data one direction only: from SYSPRO into Databricks for analytics and reporting. Updates to SYSPRO must be made through SYSPRO itself.

How are SYSPRO and Databricks credentials managed?

ml-connector stores both credential sets encrypted and handles token lifecycle automatically. For SYSPRO, it logs in to obtain a session token and refreshes it as needed. For Databricks, it exchanges client credentials for an OAuth2 bearer token every 3600 seconds. You provide the SYSPRO instance URL, operator code, and OData password, and the Databricks workspace URL and Service Principal credentials.