Error Handling & Troubleshooting

How errors flow through the SDK, common failure patterns, and how to diagnose issues. For production gotchas that aren't in any single spec (startup ordering, broker recovery, bcrypt threadpool), see Operational Notes.

Error flow

Each layer handles its own failures and either recovers locally or propagates with context:

Adapter error    → returned as ProvisionResult.Failed / CommandResult{Status=...}
Pipeline error   → logged + counter, message dropped or skipped, processing continues
Bus API error    → logged + ErrorResponse { error, message } returned to NATS caller
DB error         → exception propagates, transaction rolled back
Webhook error    → logged + counter; NATS publish still proceeds
NATS error       → logged + retry (connection manager) or counter (pipeline)

ErrorResponse { error, message } is the canonical shape for bus-api error replies. The error field is a stable machine-readable code; message is human-readable and must not contain PII.

Provisioning errors

Device already exists

The DeviceEntity has an unconditional unique index on device_id (no soft-deleted exemption). Two concurrent provisions for the same device → one wins, the other gets PostgresException with SqlState == UniqueViolation. The SDK's IProvisioningManager re-reads the existing row and returns the same response shape with created: false so callers can be idempotent:

try
{
    var result = await _provisioning.ProvisionAsync(deviceId, request, ct);
    return Ok(result);
}
catch (PostgresException ex) when (ex.SqlState == PostgresErrorCodes.UniqueViolation)
{
    // Race lost — read existing and surface as 200 OK with created=false.
    var existing = await _provisioning.GetExistingAsync(deviceId, ct);
    return Ok(existing);
}

A re-provision of a soft-deleted device purges the soft-deleted row in the same transaction.

Adapter provisioning failure

If OnProvisionAsync returns ProvisionResult.Failed(error), the entire transaction rolls back — no device, twin, or protocol-specific records are created. The error string surfaces back to the bus-api caller as ErrorResponse.message.

public override async Task<ProvisionResult> OnProvisionAsync(
    ProvisionContext context, CancellationToken ct)
{
    try
    {
        await CreateBrokerUser(context, ct);
        return ProvisionResult.Ok();
    }
    catch (BrokerException ex)
    {
        return ProvisionResult.Failed($"Broker setup failed: {ex.Message}");
    }
}

Unprovision is idempotent

POST/PROVIDE delete for a non-existent device is a no-op. The bus-api unprovision succeeds silently (HTTP equivalent: 204). If your adapter's OnUnprovisionAsync finds no rows, treat it as success — the SDK has already committed the device-side delete.

Command errors

Constructing failure results

Construct a CommandResult explicitly — there are no Ok()/Failed() factories. Always echo CorrelationId, even for transport-level failures.

catch (OperationCanceledException) when (!ct.IsCancellationRequested)
{
    return new CommandResult
    {
        CorrelationId = command.CorrelationId ?? string.Empty,
        Status = CommandStatus.MethodTimeout,
        ErrorCode = nameof(CommandStatus.MethodTimeout),
        Retryable = true,
    };
}

CommandStatus	When	Retryable default
Ok	Device replied 2xx	n/a
DeviceError	Device replied 4xx/5xx	depends on body — caller decides
DeviceNotConnected	Adapter knows the device has no live session	true
MethodTimeout	No reply within deadline	true
DuplicateCorrelationId	Caller bug — reused an in-flight id	false
PendingCapacityExceeded	Adapter pending registry full	true (after backoff)
PayloadTypeMismatch	Request or response not parseable JSON	false
TransportError	Publish failed, broker disconnect, generic adapter fault	true
Cancelled	Cancellation token / shutdown	false

The CommandStatus enum is closed — adapter implementations MUST NOT introduce new values without amending the spec.

Method invocation broadcast scope

Method invocation is broadcast on bt.provider.{providerId}.cmd.method.* (no queue group). Every replica receives the request; only the replica whose IsDeviceConnected(deviceId) returns true answers. If no replica owns the device, the NATS request times out with a MethodTimeout from the caller's perspective.

Telemetry pipeline errors

Channel full (backpressure)

When the channel is full (default 50,000) new messages are dropped silently. Monitor:

• pipeline.messages.dropped — counter, increments per drop.
• pipeline.channel.depth — gauge, current buffer depth.

If you see drops, either raise TelemetryPipeline:ChannelCapacity or investigate why the pipeline is draining slowly (NATS latency, DB contention on reported properties, thread-pool starvation).

JetStream publish failure

Individual publish failures are logged and counted (pipeline.errors{stage=nats-ack}) but don't stop the pipeline. The batch continues processing. Messages that fail to publish are lost — JetStream ack failure means the broker didn't persist the message.

Reported-property DB error

If the twin update fails (DB error), the reported property is logged and skipped. The NATS publish for that message is also skipped to avoid storing inconsistent state. pipeline.errors{stage=db} tracks these.

CloudEvent / webhook errors

CloudEvents are published in parallel to NATS JetStream and the configured webhook target(s). Both fail independently — neither is rolled back into the DB transaction (the publish is post-commit, fire-and-forget). A slow or failing webhook does not block the controller response; see Operational Notes › CloudEvent publish detached.

If a webhook target is critical, subscribe to the BEACONTOWER_EVENTS JetStream stream instead of relying on the HTTP webhook.

Troubleshooting

Provider won't start

Symptom: Pod crashes during startup with InvalidOperationException: Database migration failed.

Common causes:

• Database doesn't exist (check connection string).
• Migration SQL error (check the specific migration in logs).
• Concurrent pod startup race on a fresh database (both try CREATE TABLE simultaneously).

Fix: Inspect logs for the failing migration. The SDK uses IF NOT EXISTS and CREATE OR REPLACE for idempotency, but two cold starts can still race. Scale to 1 replica for the initial deployment, then scale up.

---

Symptom: InvalidOperationException: Credentials:MasterKey not configured on startup in production.

Fix: Provide Credentials:MasterKey (32-byte base64). The reference host fails fast in Production if the key is missing or empty — see Configuration › Authentication.

---

Symptom: MQTT adapter looping in reconnect backoff after a clean cluster deploy.

Fix: The mosquitto-go-auth superuser row hasn't been seeded. Confirm app.Services.SeedMqttSuperuserAsync() ran between Build() and RunAsync() — see Operational Notes › Startup ordering.

Telemetry drops

Symptom: pipeline.messages.dropped increasing, JetStream message count below expected.

Causes:

1. NATS slow — check JetStream cluster health and storage.
2. DB contention — reported-property updates are semaphore-bounded. Raise TelemetryPipeline:MaxReportedConcurrency if the DB can absorb it.
3. Burst > capacity — raise TelemetryPipeline:ChannelCapacity.
4. Per-message overhead — raise TelemetryPipeline:BatchSize to amortise NATS publish cost.

NATS connection failures at startup

Symptom: can not connect uris: nats://... at startup.

Causes: NATS not reachable, wrong URL, auth failure. The SDK's NatsConnectionManager initialises lazily on first use; if NATS is unreachable, the telemetry pipeline and bus-api consumers fail to start and the host crashes (CrashLoopBackOff in k8s).

Bus-api requests rejected with schemaVersion error

Symptom: Every bus-api request returns ErrorResponse { error: "schemaVersion", message: "..." }.

Cause: The caller is on an older client that doesn't include schemaVersion: 1. The SDK rejects requests without the field; consumers must ignore unknown fields for forward compatibility.

Fix: Update the caller to include "schemaVersion": 1 on every request.

Twin filter returns 404 unexpectedly

Symptom: GET /api/devices/{id}/desired?component=/foo returns 404 even though the device has desired state.

Cause: The pointer resolves to a non-object value (a string, number, or array). The spec returns 404 in that case rather than projecting a non-object as an empty object.

Fix: Update the caller to expect 404 for non-object component values, or change the schema so the component is always an object.