Oracle NetSuite and Google BigQuery integration

What moves between them

Data moves in one direction: from Oracle NetSuite into Google BigQuery. On each scheduled run ml-connector queries NetSuite for vendor bills, purchase orders, vendor payments, invoices, and GL accounts that are new or changed, plus the vendor, item, and dimension master records that give them context, and loads them into the matching BigQuery tables. New rows are appended through streaming inserts, while large historical backfills go through batch load jobs. Because BigQuery holds no purchase orders or bills of its own and has no way to write back to an ERP, nothing flows from BigQuery into NetSuite; the warehouse is the reporting copy, and NetSuite stays the system of record. Cadence follows your reporting needs, from hourly to nightly.

How ml-connector handles it

ml-connector stores both credential sets encrypted. For NetSuite it signs a JWT with the uploaded certificate's private key to mint a sixty-minute OAuth 2.0 token, refreshing when a call returns 401, and it accepts the full account-specific host per customer since NetSuite publishes no shared base URL. For BigQuery it signs a service-account JWT-bearer assertion for a one-hour token. Change detection is scheduled SuiteQL polling with a limit and offset paging loop following the hasMore flag and the next link, since NetSuite Event Subscriptions carry no signature to verify. Each NetSuite record type maps to a configurable BigQuery dataset and table, with NetSuite fields written to the table's columns. Every streamed row carries an insertId derived from the NetSuite internal id so a retry within the dedup window does not double-load, and batch loads use a stable jobReference.jobId so a re-submitted load returns the existing job instead of creating a duplicate. NetSuite governance returns HTTP 429 with a Retry-After header when concurrency or frequency caps are hit, so ml-connector backs off with jitter and retries. Two real edge cases: the BigQuery sandbox tier blocks streaming inserts and DML, so a billing-enabled project is needed for writes, and BigQuery columns can be added but not renamed or dropped without rebuilding the table, so adding a NetSuite field is safe while removing one is not.

A real-world example

A mid-sized e-commerce and wholesale company, roughly two hundred staff, runs Oracle NetSuite for purchasing, AP, and the general ledger across two subsidiaries. Its analysts wanted spend and margin dashboards that joined NetSuite purchase and payment data with web and shipping data already sitting in Google BigQuery, but the only way to get NetSuite figures into the warehouse was a monthly saved-search export to CSV that someone uploaded by hand, leaving dashboards weeks stale and prone to load errors. With Oracle NetSuite and Google BigQuery connected, vendor bills, purchase orders, payments, and GL accounts land in the warehouse every night, partitioned by transaction date, and the dashboards refresh against current data. The manual export is gone and finance and analytics finally read from the same numbers.

What you can do

• •Load Oracle NetSuite vendor bills, purchase orders, vendor payments, and invoices into Google BigQuery tables for SQL reporting.
• •Copy NetSuite GL accounts, vendors, items, and Department, Class, and Location dimensions so warehouse data has full context.
• •Read NetSuite efficiently with SuiteQL and page large result sets through the limit and offset loop.
• •Bridge NetSuite OAuth 2.0 certificate auth and the BigQuery service-account key, refreshing both one-hour tokens automatically.
• •Detect new and changed records by scheduled polling and stamp each row with an insertId so retries never double-load.

Questions

Which direction does data move between Oracle NetSuite and Google BigQuery?

Data flows one way, from Oracle NetSuite into Google BigQuery. NetSuite transactions and master records are read and loaded into warehouse tables for reporting. BigQuery is a data warehouse with no ERP objects and no way to write back to NetSuite, so nothing is pushed from BigQuery into NetSuite and NetSuite stays the system of record.

Does the integration use webhooks or polling?

It uses scheduled polling. BigQuery sends no webhooks at all, and while NetSuite has native Event Subscriptions they carry no HMAC signature to verify, so the reliable pattern is to query NetSuite with SuiteQL on a schedule. ml-connector pages through results with the limit and offset loop and loads new and changed rows on the cadence you set, from hourly to nightly.

How does ml-connector avoid loading duplicate rows into BigQuery?

Streaming inserts include an insertId built from the NetSuite internal id, which gives BigQuery best-effort deduplication within roughly a one-minute window, so a quick retry does not create a duplicate. Batch load jobs use a stable jobReference.jobId, so a re-submitted load returns the existing job instead of running twice. Together they keep retries safe across both write paths.