Plex and Databricks integration

What moves between them

The main flow runs from Plex into Databricks. ml-connector polls Plex on your configured schedule (typically after daily processing or every 5-15 minutes) and retrieves suppliers, purchase orders, invoices, customers, sales orders, parts, inventory, GL accounts, and payments. These records are normalized and written to Databricks tables under a designated Catalog and Schema. Reference data such as parts, suppliers, and GL accounts are written first as dimension tables, then transaction records (invoices, orders) reference them. Plex does not provide webhooks and Databricks does not push events back to Plex, so the flow is unidirectional: Plex to Databricks, on a polling schedule.

How ml-connector handles it

ml-connector stores the Plex OAuth2 credentials encrypted and refreshes the Bearer token before it expires. For each polling interval, it queries Plex using the modified_date and created_date filters to fetch only changed records, reducing load on both systems. The company code (PCN) is included in the Plex request to ensure the integration user retrieves data from the correct Plex company partition. ml-connector normalizes the Plex records and constructs Databricks SQL INSERT or MERGE statements to write them to Delta Lake tables. Because Plex rate limits are undocumented, ml-connector implements exponential backoff and tracks request counts to avoid throttling. On the Databricks side, it refreshes the Service Principal OAuth2 token every 50 minutes (well before the 3600-second expiry) to prevent mid-pipeline authentication failures. Each record write includes the Plex record ID, modified timestamp, and source system identifier so Databricks tables maintain a full audit trail and support change-data-capture (CDC) workflows. Failed writes are retried; if a downstream Databricks job fails, the affected Plex records can be identified and replayed.

A real-world example

A mid-sized discrete manufacturer runs Plex ERP for production, purchasing, and inventory across three plants. Before the integration, analysts manually exported purchase orders, invoices, and production schedules from Plex every week, loaded them into spreadsheets, and built reports for procurement and supply chain visibility. With Plex and Databricks connected, each day's procurement and manufacturing data lands automatically in Databricks tables, feeding dashboards that show spend by supplier, order aging, production rates, and inventory turns in near-real time. The export and re-load step is eliminated, and the data is already normalized for cross-system analytics.

What you can do

• •Poll Plex on a configurable schedule (5-15 minutes) and sync suppliers, purchase orders, invoices, customers, sales orders, parts, inventory, GL accounts, and payments to Databricks.
• •Authenticate Plex with OAuth2 Bearer tokens or SOAP basic auth, routing requests to the correct company code (PCN), and handle rate limit retries.
• •Normalize Plex records and write them to Databricks Delta Lake tables with full audit trail (source ID, modified timestamp, write timestamp).
• •Manage Databricks Service Principal OAuth2 token refresh (3600-second expiry) and handle token refresh mid-pipeline.
• •Support change-data-capture (CDC) workflows and record replay by maintaining Plex record IDs, timestamps, and failure tracking in Databricks.

Questions

Which direction does data move between Plex and Databricks?

Data flows from Plex into Databricks only. Plex operational records (suppliers, purchase orders, invoices, GL accounts, inventory) are polled and written to Databricks tables for analytics and reporting. Databricks is a read-mostly platform for this integration and does not push data back into Plex.

How does ml-connector handle Plex's lack of webhooks and undocumented rate limits?

ml-connector polls Plex on a schedule you configure (typically 5-15 minute intervals) using modified_date and created_date filters to fetch only changed records. It implements exponential backoff when Plex returns HTTP 429, and tracks request counts to avoid throttling. This approach reduces load on Plex while keeping Databricks tables fresh.

How are Plex company codes (PCN) and Databricks workspaces mapped?

Each Plex company code is configured separately in ml-connector with its own OAuth2 credentials and PCN value. When polling, ml-connector includes the PCN in the request to route to the correct Plex company partition. Records are written to Databricks with the company code and source system identifier included, so multi-company setups can segregate data by table or schema as needed.