Protecting a business process from external API failure
A practical architecture for external APIs with time budgets, bounded retries, circuit breakers, queues, fallbacks, observability and incident modes.
Contents
An external API becomes part of your service reliability even though you do not operate its infrastructure. Without isolation, a slow partner consumes worker capacity, grows queues and can stop functions that are not directly related to that partner.
The goal is not to promise permanent availability of the dependency. It is to define how your own process behaves during latency, partial failure and complete unavailability.
Start with the business outcome
For every external operation, ask:
- is the answer required immediately;
- can the work finish later;
- is a cached result acceptable;
- what will the user see during an outage?
Address suggestions in a form and an irreversible operation need different behaviour. The first may be skipped temporarily; the second needs a controlled queue and explicit state.
Record the dependency owner, response-time budget, retention period for unfinished work and manual-recovery path.
Fit timeouts into the total budget
If a user request must complete in two seconds, an external call cannot wait five. The budget includes validation, internal logic, network connection, external processing, persistence and response generation.
Use separate connection and response timeouts. Derive them from measurements rather than library defaults. A long timeout holds resources; an aggressive one creates false failures and extra retries.
Bounded retries
A retry helps with a brief network fault but harms an overloaded service. When every client retries together, traffic increases during the incident.
A safe policy usually includes:
- a limited number of attempts;
- exponential backoff;
- jitter;
- error classification;
- an overall deadline;
- idempotency for state changes.
Do not automatically repeat a request when you do not know whether the partner completed it. Use an idempotency key or a separate status check. The broader design is covered in API integration architecture.
Circuit breaking
A circuit breaker temporarily stops calls when error rate or latency crosses a threshold. It frees resources and gives the dependency time to recover.
Its states are:
- closed — calls proceed;
- open — new calls fail fast;
- half-open — a limited probe checks recovery.
Use a measurement window and minimum request volume. One failure at low traffic should not open the circuit, while an absolute count becomes meaningless at high traffic.
A queue separates acceptance from execution
If the result is asynchronous, accept the task, issue an identifier and process it through a queue. The user needs a meaningful state: accepted, running, complete or requiring attention.
Provide:
- maximum retry count;
- dead-letter queue;
- message lifetime;
- deduplication;
- processing rate limits;
- controlled manual replay;
- protection for sensitive data.
A queue without ownership and alerts only hides growing operational debt.
Make fallback explicit
| Scenario | Possible fallback |
|---|---|
| Reference data | Cache with its update time |
| Optional recommendation | Continue without it |
| Notification delivery | Queue or another permitted channel |
| Critical validation | Stop with a clear state |
| Rate or price lookup | Last value only within an approved age |
Never present stale data as current. If the fallback changes quality or legal meaning, record it and make it visible.
Isolate dependencies
Do not let one partner consume every worker. Use separate connection pools, concurrency limits and per-dependency queues. An adapter keeps the external contract out of domain logic and provides one place for credentials, schema mapping, throttling and metrics.
What to measure
Monitor:
- p50, p95 and p99 latency;
- timeout ratio;
- errors by class and operation;
- retry count;
- time spent with the circuit open;
- queue age;
- dead-letter depth;
- fallback rate;
- time from acceptance to business outcome.
Connect the technical measures to the process SLO, as described in SLIs, SLOs and error budgets.
Test failure before production
Test slow responses, lost connections, 429 and 5xx responses, invalid schemas, a successful operation with a lost response, circuit recovery, queue saturation and revoked credentials.
Success means more than a retry that eventually works. It also requires clear user behaviour, an actionable alert and a recovery runbook.
How we can help
Logic Telecom can help map external dependencies, define time budgets and design adapters, queues and observability. Start with a process where a partner outage currently stops customer service or creates manual work.
