Oracle JD Edwards and Google BigQuery integration

What moves between them

The flow runs one direction, from Oracle JD Edwards EnterpriseOne into Google BigQuery. ml-connector reads supplier vouchers, purchase order headers and detail, payments, GL accounts, and account ledger journal lines from the JD Edwards data service tables, then streams each record into the matching BigQuery table. New and changed rows are detected by filtering on the JDE date-updated field and tracking the last-polled timestamp between runs, so each cycle picks up only what moved. Cadence is the schedule you set, typically hourly or daily depending on reporting needs. BigQuery is treated as a read-only analytics destination, so ml-connector never writes records back into JD Edwards.

How ml-connector handles it

ml-connector stores both credential sets encrypted. On the JD Edwards side it accepts the full AIS base URL per customer, posts the service-account username and password to get a session token, and presents that token in the jde-AIS-Auth header; when a call returns HTTP 444 it re-authenticates automatically, since tokens are dropped on any AIS Server restart. On the BigQuery side it signs a JWT with the service-account private key, exchanges it for a bearer token, and refreshes before the one-hour expiry. Each JDE table is read with offset pagination, checking the moreRecords flag to page until exhausted, and the rows are mapped to columns in the configured BigQuery dataset and table. Because JD Edwards has no idempotency header, ml-connector sets a stable insertId on each streamed row built from the JDE document number and type, which gives best-effort dedup inside BigQuery's roughly one-minute window so a re-read run does not double-load. Real edge cases handled: the JDE service account consumes a named user license and a lock returns HTTP 403; an unlisted egress IP returns HTTP 405 from the allowedHosts filter; and the service-account private key carries literal newline characters that must be preserved for JWT signing.

A real-world example

A mid-sized industrial distributor with a few hundred employees runs Oracle JD Edwards EnterpriseOne on its own infrastructure for procurement, AP, and finance. The finance and operations teams needed spend and payables reporting across business units, but the only way to get it was an analyst exporting tables from JD Edwards into spreadsheets every week, which was slow, error-prone, and always a few days stale. With Oracle JD Edwards EnterpriseOne connected to Google BigQuery, vouchers, purchase orders, payments, and GL ledger lines land in warehouse tables on a schedule, so dashboards query current ERP data directly. The weekly export step is gone and reports reflect what is actually in the ledger.

What you can do

• •Stream JD Edwards vouchers, purchase orders, payments, GL accounts, and journal lines into Google BigQuery tables.
• •Detect new and changed records by polling the JDE date-updated field and tracking the last run.
• •Bridge the JD Edwards session token and the BigQuery service-account OAuth bearer, refreshing each before it expires.
• •Dedupe streamed rows by a stable insertId built from the JDE document key so a re-read never double-loads.
• •Map each JDE table to a configurable BigQuery dataset and table, with retries and a full audit trail on every record.

Questions

Which direction does data move between Oracle JD Edwards EnterpriseOne and Google BigQuery?

The flow is one direction, from JD Edwards into BigQuery. Vouchers, purchase orders, payments, GL accounts, and account ledger lines are read from the JD Edwards data service tables and streamed into BigQuery for reporting. BigQuery is a read-only analytics destination, so ml-connector never writes records back into JD Edwards.

How does the integration get data out of JD Edwards when there are no webhooks?

JD Edwards sends no outbound webhooks, so ml-connector polls the AIS Server data service on the schedule you set. It queries each table with a filter on the date-updated field and tracks the last-polled timestamp between runs, so each cycle reads only new and changed rows. It pages through results using offset pagination and the moreRecords flag until the table is exhausted.

How are duplicate rows avoided when JD Edwards has no idempotency key?

JD Edwards has no idempotency header, so ml-connector builds a stable insertId for each streamed row from the JDE document number and type. BigQuery uses that insertId for best-effort deduplication within roughly a one-minute window, which prevents a re-read run from double-loading the same record. Every row is also tracked in the audit trail and can be replayed if a load fails.