Category
Knowledge base
Subject
API & integrations
Reading time
16 minutes
Level
Tech lead
Updated
May 2026

API integration best practices.

A practical guide for technically involved readers (product owners, tech leads and integration engineers) who don't need to know "what an API is" but want to know how to build an integration that is still standing two years from now. Authentication, idempotency, retries, observability, schema versioning and the GDPR layer that belongs on top.

Why a separate guide to best practices?

We answer the basic question "what is an API integration" in our guide on what an API integration is. This page is for the stage after that: you now know you are going to build an integration, or you already run one, and you want to know which patterns make the difference between an integration that works in the demo and one that holds up under production pressure, with vendor outages, peak loads and a GDPR audit coming up next quarter.

Many teams start an integration with the mindset "it's just an HTTP call, we'll get it working". That holds for the prototype. It holds less once the first retry storm runs into a vendor's rate limit, or once it turns out the same payment confirmation was processed twice because a webhook was repeated. The best practices below are the defensive layer that separates production quality from demo quality. If you first want to know what an API integration costs and which architectural choices influence that, start there; for the custom-versus-platform trade-off, see API versus integration platform.

i
How to read this

The sections run from the outside in: first patterns at the network edge (auth, retries, webhooks), then your own architecture (observability, gateways, BFF), and finally the governance layer (versioning, secrets, GDPR).

Authentication: OAuth2, JWT and mTLS

Authentication is the part of an integration where a lot of costly time disappears. Not because it is complicated in itself, but because two systems' assumptions about how a caller identifies itself rarely line up. Three patterns you will encounter in practice, often in combination.

OAuth2 for customer and SaaS flows

OAuth2 is the de facto standard for scenarios where a user gives a third party permission to access data on their behalf. The client_credentials grant is for server-to-server, authorization_code for user flows with a redirect. Decide up front which scopes you request, how long refresh tokens live, and what your plan is if a token expires during an overnight batch. Many teams only discover in production that a refresh token becomes invalid after a long period of inactivity, and then have no automated recovery path.

JWT for service-to-service

JSON Web Tokens with asymmetric signing suit scenarios where you manage the identity domain yourself: one microservice calls another with a token signed with a private key, and the receiver validates it with the public key. Use short expiry, narrow scopes per token, key rotation as a fixed routine, and an unambiguous audclaim so that tokens cannot be reused against a different endpoint.

mTLS for B2B and enterprise integrations

Mutual TLS is common in integrations with banks, insurers, healthcare providers or government: both sides of the connection present a certificate. It raises the bar considerably for an attacker, but it introduces new operational complexity. Certificates expire, and they always do so at the worst possible moment. Build certificate monitoring in as a first-class concern, and treat renewal as a process that starts in a test environment, not as manual work in production.

What you explicitly don't want

API keys in URL parameters (logged by anyone with access to the access logs), basic auth without TLS, shared service accounts nobody remembers using, and custom auth protocols. For something as fundamental as authentication, it rarely pays to cook up your own recipe — use proven patterns, and spend the time on observability around auth failures instead.

Idempotency keys: the pattern that saves the most pain

An operation is called idempotent if running it twice gives the same result as running it once. For read operations (GET) that holds naturally. For write operations (POST, PUT, PATCH) you have to design for it deliberately. The mechanism: for every write operation, the caller generates a unique Idempotency-Key (a UUID will do) and sends it along as a header. The server temporarily stores this key together with the result of the first successful call. If the same key arrives again within a set window — for example because a network timeout triggered a retry even though the original request succeeded — the server returns the stored result without performing the operation again.

POST /v1/payments
Idempotency-Key: 9f4c2b0e-13aa-4f3a-8e6b-a5e0c3b1d8c2

{ "amount": 2500, "currency": "EUR", "order_id": "ord_8731" }

Important design choices: how long you retain keys, what you do when the same key arrives with a different body (a 409 is cleaner than silently ignoring it), and how you communicate in your API docs which endpoints accept this header. Building an API that others consume? Don't make idempotency optional — make it the default, and validate that the header is present on all non-GET operations.

Retry strategy: exponential backoff with jitter

Networks fail, vendors are briefly busy, deployments take time to roll out. A well-built integration tries again — but not like a parrot. Three principles that work together:

  • Exponential backoff: after the first failure, wait for example 1 second, then 2, then 4, then 8. You give the other system breathing room instead of flooding it.
  • Jitter on top: without random variation in the wait time you get a "thundering herd" — ten clients retrying at exactly the same moment after a vendor outage, causing a second, self-inflicted outage.
  • A retry budget: define a maximum number of attempts or a time budget, and after that park the message in a dead-letter queue. A retry loop that runs for hours without anyone noticing is exactly what you don't want.
def retry_with_backoff(call, max_attempts=6):
    delay = 1.0
    for attempt in range(max_attempts):
        try: return call()
        except RetryableError:
            if attempt == max_attempts - 1: raise
            time.sleep(delay * (0.5 + random.random()))
            delay *= 2

Not every error is retryable. Retrying a 401 or 400 solves nothing — those are permanent errors. A 429, 502/503/504 or network timeout are the classic retryable cases. Encode this distinction explicitly. For the broader approach to robust integrations, see our smart API integrations service.

Circuit breakers: stop knocking when the door is closed

When a dependency goes down completely, carrying on with retries is sometimes worse than stopping. A circuit breaker monitors an endpoint's success rate over a rolling window. If it drops below a threshold, the breaker "opens": all subsequent calls fail immediately, without putting any further load on the vendor. After a cool-down period, the breaker tries a half-open state with a few probe calls; if those succeed, it closes again.

The benefit is twofold: your own system stays responsive, and you help the vendor recover by no longer adding load. Libraries such as Resilience4j (Java), Polly (.NET) or your own wrapper around the HTTP client make the pattern easy to build in. Beware of thresholds that are too low in low-volume situations: if an endpoint receives few calls, a single failure quickly produces a high failure rate and the breaker opens too soon. Tune the window and threshold to your traffic.

Respect rate limits — and spread the load proactively

Almost every modern API has rate limits. Handling them well comes down to two things: reacting to the signals the vendor gives you, and spreading the work proactively.

  • Honour limit headers: on a 429, and sometimes already on a 200, vendors send X-RateLimit-Remaining, X-RateLimit-Reset and Retry-After. Build your retry policy so that it first Retry-After follows it, and only falls back to exponential backoff if that header is missing.
  • Spread batches with a token bucket: for large import jobs, a naive loop is a guaranteed route to 429 storms. A token bucket in your API client limits the outbound rate to what the vendor will tolerate. Batches that look "too slow" on a single thread are often finished sooner than a burst that keeps getting cut off.
  • Negotiate if you need to: many SaaS vendors will raise rate limits on reasonable request for enterprise contracts. For an initial historical sync, it's better to ask for extra headroom up front than to let your production integration hit the wall.

Webhooks versus polling: choose deliberately for each flow

To pick up changes, you have roughly two options: poll, or receive a webhook. For rare events that need to be processed quickly (payments, fraud alerts, disputes), webhooks are better — polling wastes capacity. Polling wins when you want replay control, want to define the window yourself, or the vendor doesn't offer reliable webhook delivery. An hourly incremental pull on an updated_atfilter is often more robust than a webhook that you can never be sure has arrived.

What a production-grade webhook handler does

  • Verifies signatures: every serious vendor signs webhooks with HMAC. Verify before you parse the body at all, otherwise you're open to spoofing.
  • Responds quickly, processes asynchronously: respond with a 2xx within seconds, put the payload in a queue, and process it in a separate worker. A slow handler causes retries on the vendor's side.
  • Processes idempotently: use the vendor's event ID as an idempotency key, so the same event only ever has one effect.
  • Keeps events it can't process: if processing fails, send the event to a dead-letter queue with enough context for manual or automated recovery.

Most robust integrations combine both: webhooks for near-real-time signals, and periodic polling reconciliation to catch missed events. If you rely on webhooks alone, you'll sooner or later discover you've missed something.

Eventual consistency: embrace it, or pay the price

Two systems that synchronise with each other via APIs are not identical in real time. There is always a delay — milliseconds at best, minutes or longer once batch processing is involved. Denying that delay leads to bugs that only show up under high load.

Two patterns help. First: avoid questions like "can I fetch this record immediately after creating it?". Replace them with event-driven flows in which the receiving system itself waits for confirmation that the message has been processed. Second: handle state conflicts explicitly. Last-write-wins is an acceptable strategy if you choose it deliberately; it's problematic if it creeps in unnoticed. For records with multiple sources, you're better off using version vectors, CRDTs, or an agreement on a leading system per field.

What definitely helps: always storing timestamps in UTC, not trusting wall-clock ordering between distributed systems, and safeguarding critical state with checksums or reconciliation jobs that periodically compare system A and system B.

Observability: correlation IDs, structured logging, traces

When a failure spans two or three systems, your ability to diagnose it depends almost entirely on the observability you built in from the start. Adding it afterwards is technically possible, but painful.

  • Correlation IDs throughout the chain: generate a unique X-Request-ID (or traceparent) at the start of every flow and pass it on to every downstream call. Without this pattern, integration debugging is an exercise in reading twenty different logs looking for overlapping timestamps.
  • Structured logging: log in JSON with consistent field names — level, service, request_id, action, resource, duration_ms, outcome. Plain-text logs become unusable as soon as you need to aggregate them across services.
  • Distributed tracing: OpenTelemetry is the standard. Spans around every API call give you a visual picture of where the time goes in a request, often far faster than logging alone.
  • Metrics that matter: success ratio per endpoint (not averaged across everything), P95/P99 latency instead of averages, queue depth for async flows, and dead-letter queue volume as a business metric.
i
Rule of thumb

If you can't establish within five minutes whether a specific transaction succeeded, what happened along the way and which system caused the error, your observability isn't production-ready yet.

Schema versioning: on both sides of the integration

A schema isn't a monument; it evolves. The difference between an integration that moves smoothly with change and one that needs a yearly "polish" lies in how you treat versioning from day one.

On the vendor side: pin your integration to a specific API version. A client that says "v1" and implicitly gets the latest minor version can break along the way without anyone having made a release. Read changelog announcements and plan deprecations as regular work.

On your own API side: version explicitly, in the URL (/v1/orders), in a header (Accept: application/vnd.appfront.v1+json) or via a query parameter. Make backwards-incompatible changes only under a new version, and give old versions a documented sunset date.

Deliberately additive versus breaking: adding fields is almost always safe, provided consumers are built to be schema-tolerant. Removing fields, changing a type or making validation stricter is always breaking and belongs in a version bump, because to the consumer a silently stricter field is barely distinguishable from a bug.

Contract testing: the same language on both sides

A unit test verifies that your code does what you think it does. A contract test verifies that your integration does what the other party expects, and vice versa. Tools like Pact are made for this: the consumer describes which calls it makes and which responses it expects; the provider verifies that its actual behaviour matches. This pays off most when you hold both the consumer and provider role within the same organisation. For external vendor integrations, integration tests against a sandbox are often enough, with regular re-runs to catch regressions on the vendor side. Combine mocked tests in CI with scheduled sandbox runs for real verification.

Error handling and dead-letter queues

What do you do with a message that keeps failing despite retries? "Retrying forever" is wrong, "discarding it" is worse. The right answer is a dead-letter queue (DLQ): a separate holding place for messages your processing can't handle, with enough context to be recovered manually or automatically. Categorise errors accordingly: transient errors (network glitch, vendor outage) go into retry with backoff, permanent errors (4xx except 429) go straight to the DLQ without retries, and unknown errors (5xx, vague timeouts) are retried until the budget runs out and then go to the DLQ.

A DLQ should hold the original payload, all failed attempts with error details, timestamps, the correlation ID and, crucially, a mechanism to put the message back into the main flow after correction. A DLQ without a replay path becomes a digital graveyard. A DLQ that suddenly grows is always a signal that someone needs to look within the hour: define thresholds and alerts, and treat emptying the DLQ as regular work rather than crisis response.

Secrets management and security

API keys, OAuth client secrets, certificate private keys, database credentials: an integration layer is a factory full of secrets. Four rules:

  • One central secret store: no secrets in code, no secrets in environment files that end up in Git. Use HashiCorp Vault, AWS Secrets Manager / KMS, Azure Key Vault, GCP Secret Manager or similar, with short-lived credentials and an audit log.
  • Rotation as a rule, not an incident: automate rotation instead of doing manual work after a leak. Refresh token rotation for OAuth, vendor-specific flows for API keys, and monitoring plus automatic renewal for certificates.
  • Least privilege: integration accounts should only be able to do what they need to. A sync account for product data has no rights on customer data. That way, a breach in one integration doesn't become a breach across your whole system landscape.
  • Keep secrets out of logs: a serializer that dumps the entire request, including the Authorization header, into an error message leaks secrets into your observability stack. Build in redaction as a first-class concern.

For the broader security context, see our information security policy for the approach we follow on projects where this matters.

API gateway and backend-for-frontend

Once you have more than a handful of integrations, it pays to centralise access patterns. An API gateway sits between consumers and your backends and handles cross-cutting concerns: authentication and token validation, per-client rate limiting, request logging, caching and transformation. Examples include Kong, Apigee, AWS API Gateway, Tyk, or a custom build around Envoy. You don't have to rebuild this logic in every microservice, but make sure the gateway itself doesn't become a critical single point of failure.

A backend-for-frontend (BFF) is a lightweight layer that aggregates data from multiple backends for one specific type of frontend (web, mobile, partner portal). It avoids complex orchestration in the frontend and stops your microservices from filling up with frontend-specific endpoints. For frontend-heavy products where you're consolidating several system landscapes, it's almost always worth it. Our page on integrations in web development goes into more detail.

GraphQL versus REST: it depends on your shape

REST often wins for well-defined resources with clear CRUD semantics, caching at the edge (REST is HTTP-native, GraphQL less so), public APIs for a broad audience that doesn't want to learn about schemas, and scenarios where you want to shape idempotency and versioning explicitly per endpoint.

GraphQL often wins for frontend-heavy applications that aggregate data from many sources and suffer from under- and over-fetching, rapidly evolving client needs, and internal APIs where consumers and providers sit in the same team. Watch out for the pitfalls: N+1 queries call for dataloaders, query complexity needs to be limited through cost analysis or allowlists, and caching is no longer an out-of-the-box HTTP cache. The pragmatic choice in many organisations: REST for public integration APIs, GraphQL for the BFF layer between internal frontends and backends.

The GDPR layer: data minimisation, EU residency, redaction, audit trails

An API integration that carries personal data falls under the GDPR. It's not a coat of paint after the build, but a set of design requirements that run through the whole architecture.

  • Data minimisation in payloads: only send what the recipient actually needs. A marketing tool doesn't need a BSN (Dutch citizen service number), and a logistics partner doesn't need a date of birth. Define an explicit projection of the source record for each integration, rather than passing the whole blob along.
  • EU residency of endpoints: for GDPR-sensitive data, the physical location of the processor matters. Check per vendor whether EU hosting is available, choose it deliberately, and document it in your record of processing activities.
  • PII redaction in logs: a log line containing email=jansen@example.com order=1234 status=failed is a GDPR incident waiting to happen the moment that log stream is accessed. Locate PII fields, redact or hash them, and treat error bodies with the same care as payload bodies.
  • Audit trails: immutable, append-only, with enough context to trace individual actions, and a retention period that fits your record of processing activities. In healthcare, NEN 7510 comes on top; in finance, DORA.

The mistakes we keep seeing

Finally, a collection of anti-patterns we regularly come across in practice. Not as a judgement, but as a checklist.

  • Non-idempotent writes without a safety net: a payment confirmation that gets processed twice, an order that arrives in duplicate. Almost always preventable with idempotency keys.
  • Retry storms: a vendor has three minutes of downtime, your 200 nodes all retry at once, and now the vendor has hours of downtime. Jitter, circuit breakers and retry budgets are not there for show.
  • Hidden coupling: system B relies on system A always responding quickly, without this being written into any contract. When A's latency shifts, B breaks, and you won't see it until it does.
  • Versioning deadlocks: you want to move to a new API version, the vendor has removed the old one, consumers are still on the old one, and nobody feels ownership of the migration. Plan version transitions in advance with a sunset date.
  • Missing observability until the first outage: you find out an integration is broken because a customer calls. By the time that phone rang, you should already have had alerts.
  • Security through obscurity: an endpoint that "nobody can find because the URL isn't documented". Scanners will find it quickly enough. Authentication and authorisation are non-negotiable, even internally.
  • Lock-in on vendor-specific features: a handy proprietary feature, then a year later you want to migrate and that feature exists nowhere else. Record in your architecture documentation which vendor assumptions you accept and what an exit would cost.
!
In conclusion

An API integration is not a standalone HTTP-POST — it is a set of agreements between two systems that have to keep working together under production pressure. The best practices above are those agreements in defensive form. None of them is exotic, and almost none is expensive. The difference between integrations that hold up and integrations that keep falling over lies in which ones you take seriously enough to build in before go-live.

The three key points.

01

Build defensively from day one

Idempotency, retries with jitter, circuit breakers, dead-letter queues and signature validation are not luxuries but the foundation of a production integration.

02

Observability is not an afterthought

Correlation IDs, structured logging, traces and metrics need to be part of the design, not bolted on afterwards. Otherwise you are blind during the first vendor outage.

03

GDPR and security are part of the design

Data minimisation, EU residency, PII redaction in logs and least privilege for service accounts are architectural requirements, not a separate compliance layer.

An integration that can handle production pressure?

A half-hour introductory call. We listen to which integrations you manage or want to build, where the pain is and which patterns would help most, and then give you honest direction on scope, approach and route.

Edit content