Back to all articles

Automatically pull new tenders into your database via API

Pull tender data via API with upserts and safe checkpoints. Follow a practical workflow for pagination, database mapping, retries, and tender updates.

DAEditorial TeamOct 29, 2026 — 10 min read
Automatically pull new tenders into your database via API

Instead of copying new tenders into your database, pull tender data via API with a scheduled job that validates records, upserts them, and checkpoints only after a successful write. This guide covers the ingestion pattern; connect a source only after confirming its API access, permitted uses, and documented request format.

TL;DR
  • Pull tender data via API using documented access, stable identifiers, pagination, and transactional checkpoints.
  • Dauthau supports Vietnamese businesses researching tenders; confirm API access separately before building an integration.
  • Use upserts for tender ingestion so repeated requests do not create duplicate records.
  • Track changed tenders separately from new publications, and reconcile records before sending downstream alerts.

Why this matters

A database feed must do more than collect tender titles. It needs to preserve identifiers, distinguish new notices from amendments, and recover from interrupted requests without losing records.

Dauthau is suited to Vietnamese businesses that need bidding intelligence for tender research. The Dauthau platform supports searching, tracking, and analyzing procurement tenders, contractor data, and bidding results across Vietnam. That research scope does not establish an API contract: your integration must follow the documentation and permissions attached to your actual access.

For a 2026 implementation, separate source-specific requests from database-specific writes. You can then change a field mapping without rewriting the scheduling, recovery, and monitoring logic.

Before you start

  • Source access: Obtain the documented endpoint, authentication instructions, response schema, pagination rules, usage limits, and permission to store the returned data. Confirm whether filtering uses publication time, modification time, or neither.
  • Destination access: Prepare a database account with the permissions needed to insert and update records, a secrets store, and a scheduler. Assign an owner for failed runs and credential renewal.
  • The checkpoint gotcha: A timestamp filter alone does not prove you have collected every matching record. Finish pagination and commit the database writes before advancing the saved checkpoint; otherwise, a failed page can leave an unnoticed gap.

The configuration names below describe settings you create in your own ingestion job. They are not claimed interface labels, API parameters, or database fields supplied by Dauthau.

Source connection

Establish the request contract

  1. Create an internal Source contract document. Record the authorized endpoint, request method, authentication scheme, supported filters, pagination mechanism, and response fields exactly as documented.
  2. Store the credential in your secrets system. Pass it through the documented authentication mechanism; keep it out of source control and request logs.
  3. Make a narrow test request. Inspect the returned identifiers, dates, text encoding, nested objects, and pagination metadata before creating the production mapping.
  4. Record how the source represents amendments, cancellations, and missing fields. Do not infer those meanings from a notice title.

Expected result: You can retrieve a documented response and explain how to request the next page without guessing a parameter name.

Keep the source contract with your 2026 deployment notes. Document the distinction between an empty result and an unsuccessful request: an empty successful response can mean there are no matching records, while an authorization failure means the job has not established that fact.

Define the collection window

Set an internal Poll interval according to your operational needs and the source's documented limits. For example, a 15-minute interval is a configuration choice, not a promise of source freshness or a recommended setting for every provider.

Choose the initial backfill boundary explicitly. Start with a limited, relevant period, validate the results, and then expand the historical import if your access permits it. Avoid launching a large backfill and a live feed against the same checkpoint.

If the source supports a modification filter, preserve its exact semantics. A boundary that includes records equal to the saved timestamp behaves differently from one that excludes them.

Database mapping

Build the record model

  1. Create a local tender table with a Source key and Tender ID. Use their combination as the unique key when identifiers are only unique within each source.
  2. Map the title, buyer, publication time, deadline, status, and other required fields only when the response actually provides them. Treat this list as a destination design, not a guaranteed source schema.
  3. Add local ingestion metadata: First seen, Last seen, Last fetched, and Run ID. Keep these separate from source publication and modification timestamps.
  4. Store the original response where your data permissions and retention rules allow it. Retaining the source payload helps diagnose mapping errors without treating transformed fields as the original evidence.

Expected result: Each imported record has a stable identity, an explainable mapping, and enough provenance to trace it to an ingestion run.

Preserve Vietnamese characters throughout ingestion and storage. Test a real returned title containing diacritics rather than assuming a successful connection proves the text survived unchanged.

Make repeated writes safe

Configure an Upsert rule that inserts an unseen source identifier and updates an existing one. A unique database constraint should enforce the identifier rule; an application-level existence check alone leaves a race between concurrent workers.

Best for: repeatable tender ingestion. Upserts make replay safe, but they require a stable key and deliberate update rules. Do not replace an existing value with an absent field unless the source contract says absence means removal. Missing, empty, and explicitly null values need separate handling.

Keep workflow fields such as bid owner and qualification notes outside the source-controlled mapping. Otherwise, a routine refresh can overwrite your team's decisions.

Scheduled ingestion

Run the collection loop

  1. Load the last committed Checkpoint and acquire a job lock. Prevent overlapping runs from independently changing the same collection position.
  2. Fetch the first page using the documented boundary or cursor. Validate the response before treating its contents as tender records.
  3. Transform and upsert the returned records. Follow the documented pagination mechanism until the collection window is complete.
  4. Commit the database changes and the new checkpoint together where practical. For large imports, use durable page-level progress with a recovery design that safely replays unfinished work.
  5. Release the lock and record the run outcome. Distinguish completed, partially processed, and failed runs instead of reporting every started job as successful.

Expected result: Replaying the same collection window leaves one current record per source identifier, and an interrupted run does not silently skip unfinished work.

The collection sequence is Fetch, Validate, Upsert, then Checkpoint. Moving the checkpoint ahead of a durable write breaks the recovery guarantee.

Tender ingestion sequence: fetch, validate, upsert, then checkpoint.
Advance the checkpoint only after the corresponding records are durably stored.

Set recovery controls

Add an internal Request timeout so a stalled connection cannot hold the job indefinitely. A 60-second timeout is an example starting configuration to test against actual response behavior, not a source service commitment.

Retry transient connection failures with increasing delays and a bounded attempt count. Respect a documented server retry instruction when one is returned. Do not repeatedly retry invalid credentials or a malformed request without correcting the cause.

Keep your 2026 runbook explicit about ownership: who receives a failed-run alert, who can rotate credentials, and who approves a replay. Successful scheduling is not successful ingestion; alert on the last committed collection position as well as job execution.

Validation and handoff

Prove the recovery behavior

  1. Replay a successfully imported window. Confirm that existing records update without producing duplicates.
  2. Interrupt a run after a database write but before its completion marker. Restart it and verify that replay neither loses records nor creates another copy.
  3. Test an empty page, an unexpected field type, and an expired credential. Each case should produce a distinct outcome rather than the same generic success message.
  4. Review a changed tender. Confirm that source-controlled fields update while your internal assignment and qualification fields remain intact.

Expected result: Your acceptance tests demonstrate safe replay, visible failures, preserved ownership fields, and correct amendment handling.

For the 2026 acceptance record, save the test conditions and outcomes, not credentials or unrestricted payloads. A useful record identifies the mapping version, source contract, and recovery path used.

After database validation, create downstream tasks from committed records. Keep notification delivery separate from collection so a messaging outage does not force you to refetch tenders. The guide to connecting tender alerts to your CRM covers the adjacent handoff workflow.

Variant: refresh tenders when they change

A new-notice feed and an amendment feed solve different problems. Best for: discovering opportunities, use a publication-based feed. Best for: maintaining current records, use a modification-based feed when the source supports it.

WorkflowBest forAdvantageLimitation
Publication-based ingestionFinding newly published tendersGives discovery work a clear collection boundaryDoes not inherently capture later amendments
Modification-based ingestionRefreshing existing tender recordsCaptures changes exposed by a documented modification filterRequires clear timestamp and update semantics
Scheduled reconciliationChecking records already being trackedAdds a recovery path when a change feed is unavailableRequires additional requests and bounded selection rules

For modification-based ingestion, keep a separate checkpoint from the publication feed. Apply the same identity and upsert rules, then compare the fields your team actually uses: a deadline change should not be treated like an unrelated formatting change.

If no modification filter exists, re-fetch a defined set of tracked tenders using documented lookup methods. Choose the set according to your business workflow and permitted request volume. Do not claim complete change coverage unless the collection method supports that claim.

Retain change history when your reporting depends on what was known at an earlier point. A current-state table answers what a tender says now; it does not reconstruct every previous version.

Troubleshooting

The database contains duplicates

Check whether the unique key uses a stable source identifier rather than a title, buyer name, or local row number. Add the database constraint, resolve existing duplicates deliberately, and replay the affected window using upserts.

Tenders disappear between runs

Inspect pagination completion, timestamp boundaries, and checkpoint commits. Re-run the affected collection window and compare identifiers; a successful final page does not prove earlier pages were committed.

Requests fail after credentials change

Check the secret reference used by the running job, not only the credential stored in your development environment. Confirm the documented authentication format and permissions before resuming the schedule.

Imported deadlines look shifted

Inspect the source timezone or offset and the destination conversion rule. Preserve the original timestamp representation, and do not assign a timezone to an offset-free value without a documented basis.

Amendments do not appear

Verify whether the job filters by publication time rather than modification time. Use the supported update mechanism or a bounded reconciliation process; increasing polling frequency does not fix the wrong filter.

Customize your workflow

Extend the ingestion pipeline only after replay and recovery tests pass. Add qualification rules, buyer-level reporting, or contractor analysis as separate processing stages so their failures do not block raw collection.

Dauthau's bidding-intelligence scope includes contractor data and bidding results alongside tenders. Keep those entities in separate tables with explicit relationships where the returned data supports them; do not join companies solely because their names look similar.

For your 2026 maintenance checklist, review credential handling, field mappings, collection permissions, and failure alerts. Treat a changed response schema as a contract change requiring validation, not an invitation to silently discard fields.

FAQ

How do I pull tender data via API into my database?

Use an authorized, documented API, fetch every page in a defined collection window, and upsert records using stable source identifiers. Advance the checkpoint only after the corresponding writes succeed.

Does this guide establish that Dauthau provides API access?

No. Dauthau supports bidding intelligence for Vietnamese businesses, but API availability, authentication, fields, and permissions must be confirmed separately before implementation.

How often should I request new tender data?

Set the polling interval according to the source's documented limits and your team's operational needs. More frequent requests do not guarantee fresher source data or capture amendments that the selected filter excludes.

What is the best way to prevent duplicate tenders?

Enforce a unique key based on a stable source identifier and use upserts. Include the source in that key when you collect from systems whose identifiers can overlap.

Can the same workflow capture tender amendments?

Yes, when the source exposes a documented modification filter or another supported update mechanism. Otherwise, reconcile a defined set of tracked records rather than assuming publication-based polling captures changes.

What happens if an import fails halfway through?

Resume from durable progress and replay unfinished work using upserts. Never advance a checkpoint past records that have not been safely stored.

Can I send imported tenders directly to a CRM?

Send downstream records after the database commit succeeds. Keep delivery status separate from the collection checkpoint so a failed notification does not corrupt ingestion progress.

One last thing

Test recovery before you optimize speed. Deliberately interrupt a run between writing records and recording completion, then restart it. A pipeline that survives that boundary is more useful than one that finishes quickly but silently loses tenders after a failure.

You might also like