Skip to main content

Operational Notes

These are repeated lessons surfaced from production and loadtest incidents. They are not in any individual spec because they are about the host's ordering and runtime, not the contracts.

Startup ordering​

The reference MQTT host calls await app.Services.SeedMqttSuperuserAsync() between builder.Build() and app.RunAsync(). The mosquitto-go-auth plugin authenticates the SDK's superuser by reading mqtt_user_entity directly, so if the adapter starts before the row exists the broker rejects the CONNECT and the adapter enters reconnect-backoff.

If you deploy a clean cluster and see the adapter looping in reconnect backoff, check that the seeder ran (it logs at Information). Mosquitto caches negative auth results — restart the broker pod if the superuser password is changed on a live cluster.

For non-MQTT adapters, the equivalent rule is: any out-of-band auth bootstrap must complete before RunAsync(), not from IDeviceAdapter.StartAsync (which runs alongside the rest of the pipeline).

bcrypt threadpool offload​

bcrypt at the default work factor of 10 is ~300 ms of CPU per provision. Under a provisioning storm (e.g. a 1000-device fleet onboard), naïve scheduling on the default threadpool starves NATS consumer callbacks and breaks heartbeats. The SDK offloads bcrypt onto a bounded threadpool sized to Environment.ProcessorCount and fills the provisioning channel from there.

Do not move bcrypt back onto the default pool. The bound is intentional. If you need to tune throughput, change Credentials:BcryptWorkFactor (lower factor = faster but weaker hashes) rather than removing the offload.

CloudEvent publish detached from HTTP response​

POST /devices/{id}/provision commits the DB transaction, returns the response, and runs the CloudEvent publish as a fire-and-forget task observed by EventPublisher. A slow webhook (a few hundred ms) used to back-pressure the controller return because publish was on the same task — the rig now decouples them.

Do not re-couple them. If a webhook consumer needs synchronous semantics, subscribe to the BEACONTOWER_EVENTS JetStream stream instead of relying on HTTP response timing.

MQTT reconnect / broker-restart recovery​

The reference MqttAdapter reconnection policy is exponential with a hard ceiling (1 s initial, 60 s max, 2× multiplier, infinite attempts). After a broker restart:

  • All per-device subscriptions are re-established.
  • Pending command TaskCompletionSource instances complete with AdapterTransportException so callers fail fast instead of hanging on SendCommandAsync timeout.
  • Retained desired messages survive a broker restart (mosquitto persists them) but not a clean-session reconnect.

The SDK's IDesiredDeliveryTracker is the source of truth either way; the retained message is an optimisation, not a guarantee.

Mosquitto must run with replicas=1​

mosquitto-go-auth does not support clustering — running multiple replicas with the same auth backend produces non-deterministic ACL state and silent message loss. Deploy mosquitto with replicas=1 and rely on adapter-side reconnect for HA.

Telemetry hot path bypasses JSON round-trip​

The telemetry path takes the adapter's bytes (TelemetryMessage.Payload, ReadOnlyMemory<byte>) and publishes them directly as the NATS message body — no JSON parse, no re-serialise. Per-message overhead is the channel write plus the NATS publish.

Do not reach into the bytes from inside the pipeline (e.g. for routing decisions). If you need per-message routing, do it in the adapter before Telemetry.Send and surface it as a header or a separate publish.

NatsHeaders is allocated per publish​

The NATS .NET client mutates the NatsHeaders instance during publish, so the SDK allocates a fresh instance per message. Do not cache or reuse a NatsHeaders across publishes — the second call will see the first call's mutated state.

Bus-api SetDesired matches HTTP PATCH​

Both surfaces accept a JSON Merge Patch (RFC 7396) body — earlier drafts of bus-api treated SetDesired as a full replace. The bus-api now matches the HTTP PATCH so consumers see consistent behaviour regardless of which surface they use.

Twin filter behaviour on non-object component values is also pinned: a ?component= filter that resolves to a non-object value yields HTTP 404, not an empty object.

NATS internal broadcast scope​

Internal broadcasts (bt.{providerId}.internal.*) carry an empty payload by design — they are cache-invalidation signals, not data deltas. Recipients must re-read state from the authoritative store (DB or twin).

Do not be tempted to put the new state in the broadcast: the rig is multi-replica and the broadcast does not pretend to be ordered with the writer's commit.

JWT auth follows the core canonical pattern​

Provider hosts wire JWT Bearer the same way BeaconTower.Core.API does — config under AuthenticationOptions:* (note: not Authentication:*), OIDC discovery via MetadataAddress, and ValidateIssuer = true as the only explicit token-validation parameter. The other validators (ValidateAudience, ValidateLifetime, ValidateIssuerSigningKey) rely on Microsoft's defaults, which are already true.

Do not introduce extra validator overrides in provider hosts unless the canonical pattern in core changes — divergence here makes a fleet-wide token-handling bug land twice. See Configuration › Authentication for the required keys and Getting Started for the wiring snippet.

Diagnostics enable-toggle​

The Diagnostics:Enabled toggle short-circuits all /api/diagnostics/* endpoints with HTTP 404. The middleware that enforces this runs before authentication so the spec-mandated "surface absent" signal happens regardless of token shape. Mount it in Program.cs immediately after the exception handler:

app.UseMiddleware<DiagnosticsEnabledMiddleware>();
app.UseAuthentication();
app.UseAuthorization();

Bulk credential re-encryption​

ICredentialsService.ReEncrypt(cipher) is the bulk re-encryption entry point exercised by the fleet-level master-key rotation tests. It decrypts a value (which may use any of Credentials:PreviousMasterKeys) and re-encrypts with the current master key. See Credentials for the rotation runbook.