SMS route failover: a cascade without duplicates or delay

SMS route failover is a governed decision to select the next eligible path, not a loop that sends the same message through every provider. The system must distinguish an explicit rejection from an unknown outcome, respect the message’s useful lifetime, preserve one business-event identifier and stop using a route after confirmed degradation. Without those controls, automatic failover creates duplicates and extra delay instead of availability.

Failures the cascade should handle

A route can become unsuitable at several stages:

• its API or SMPP session is unavailable;
• responses exceed the submission timeout;
• the connection works but requests are rejected;
• the assigned TPS limit is exhausted;
• the route does not support the destination, network or sender ID;
• submissions are accepted but delivery latency increases sharply;
• terminal failure rate breaches an operating threshold;
• DLR callbacks are delayed or missing;
• balance or another commercial limit blocks traffic.

These events do not permit the same reaction. An explicit synchronous failure before acceptance normally allows immediate selection of another route. A timeout after submission is ambiguous: the upstream system may have accepted the message but lost the response. Sending through a second provider immediately creates a duplicate risk.

State modelling matters more than provider count

Keep a separate record for every attempt:

business_event_id
message_id
attempt_id
route_id
submitted_at
deadline_at
provider_message_id
submission_result
final_status

business_event_id connects all technical attempts to one notification. attempt_id distinguishes submissions, while provider_message_id supports status queries and DLR correlation.

A useful submission outcome model includes:

• not submitted — no connection was made or the request was rejected before acceptance;
• accepted — the provider confirmed receipt and returned an identifier;
• unknown — the client received no unambiguous response;
• terminal failure — a final negative status arrived;
• delivered — a final positive DLR arrived.

Do not translate unknown directly into failure. First reconcile by client identifier, query the provider or wait for a short defined window, depending on integration capabilities.

Defining eligible routes

A cascade is not merely a global provider ranking. It starts with rules for the specific message. Filter candidate routes by:

• destination country and mobile network;
• registered sender ID;
• traffic class: OTP, transactional or marketing;
• supported encoding and concatenation behaviour;
• maximum TPS and available quota;
• delivery deadline;
• DLR and status-query support;
• agreed sending hours and commercial terms.

Only then should remaining routes be ordered by quality, cost and current health. This prevents a backup path from submitting with an unsupported sender or to a network it does not actually serve.

TTL stops meaningless retries

Every notification needs a useful lifetime. An OTP after code expiry, a reminder after an appointment begins and a courier update after delivery has finished no longer solve the business problem.

Store:

• event creation time;
• deadline for the first submission;
• deadline for all attempts;
• maximum number of route attempts;
• whether another channel is permitted.

Before each attempt, calculate the remaining time. If the backup route cannot reasonably clear its queue and downstream latency before the deadline, expire the message without another submission.

A circuit breaker prevents a traffic avalanche

Continuing to send every new message into a degraded route wastes time and capacity. A circuit breaker temporarily removes the route after a persistent problem signal.

Combine several indicators:

• network and server error rate;
• submission-response P95;
• rejected-submit rate;
• P95 time to final DLR;
• delivery within the target window;
• age of the oldest queued message;
• SMPP bind health and reconnect frequency.

A single timeout should not disable a route. Use a minimum sample, a short rolling window and hysteresis. After the pause, return the route with a limited probe flow rather than directing the entire queue to it at once.

Preventing duplicate SMS

Duplicate protection starts before calling a provider:

1. The business service creates a stable idempotency key.
2. The communications service stores the event and send plan atomically.
3. Every attempt receives a unique technical identifier.
4. An accepted request is not repeated without checking its state.
5. DLR events are processed idempotently and tied to one attempt.
6. The first confirmed delivered result cancels any attempts that have not started.

Use provider-side client identifiers and API idempotency when available, but keep local protection. The same business command can be replayed by a CRM, queue or scheduler before it reaches the messaging platform.

A reliable DLR webhook completes this design by ensuring duplicate and out-of-order statuses do not repeat a business action.

Separate retry from route switching

A retry on the same route is useful for a transient network error or throttling. A route switch is appropriate when the route itself is impaired. Define actions per error class:

Signal	Typical response
HTTP 429 or throttling	retry with backoff within TTL
connection refused	short retry, then backup route
explicit destination or template rejection	terminate without cascade
timeout after submit	reconcile state, do not duplicate immediately
increasing DLR latency	reduce route weight or open the breaker
expired TTL	terminate without another attempt

Normalise provider codes into an internal model. Otherwise the same cause can trigger different behaviour simply because two vendors format errors differently.

When another channel is required

A second SMS route cannot solve every delivery problem. If a handset is offline, the number is unavailable or the recipient is outside coverage, a second provider may receive the same final outcome.

Switching to push, voice or another agreed channel needs a separate policy. It must respect user preferences, message purpose, deadline and business-level duplicate protection. An omnichannel notification workflow should not mean sending the same event everywhere at the same time.

Metrics that reveal cascade quality

• share sent through primary and backup routes;
• failover reasons;
• time to first successful submission;
• delivery within target by cascade position;
• count and resolution time of unknown outcomes;
• suspected and confirmed duplicate rate;
• circuit-breaker openings;
• queue age by traffic class;
• cost per delivered business event;
• differences between internal state and provider reconciliation.

Read these together with TPS and queue capacity and normalised DLR metrics. A high overall delivery rate can hide late OTP messages or permanent reliance on an expensive backup.

Testing before an incident

The test plan should go beyond shutting down an API:

• DNS and TCP connection failure;
• slow response after actual provider acceptance;
• TPS throttling;
• a series of explicit submit rejections;
• delayed and out-of-order DLR events;
• unavailable idempotency storage;
• route recovery after a circuit breaker;
• TTL exhaustion during a large backlog;
• an unsupported sender ID on the backup route.

Record switch time, ambiguous messages, confirmed duplicates and impact on priority traffic. General external API resilience patterns help turn timeouts, retries and circuit breakers into one consistent policy.

Production launch checklist

• define a stable business-event idempotency key;
• normalise submission outcomes and final DLR states;
• document unknown and the reconciliation procedure;
• set TTL and maximum attempts per message class;
• verify sender IDs and restrictions on every route;
• separate same-route retry from switching;
• configure a circuit breaker with gradual recovery;
• isolate OTP from bulk traffic;
• monitor duplicates, delay and cost;
• run a controlled degradation exercise;
• document manual route removal and restoration;
• reconcile status and billing with providers regularly.

A resilient cascade cannot promise delivery of every message. It provides the more important operational property: every attempt is explainable, time-bounded and protected from uncontrolled duplication when one participant in the chain fails.