3x-ui/internal/web
Yuri Khachaturyan 2b1308ca29
feat(notifications): add a consecutive-failure threshold for outbound.down alerts (#5968)
Problem: a flaky outbound produces hundreds of false-positive "outbound down"
notifications overnight — each fires the moment xray's observatory reports a
single failed probe, and the next successful probe fires an "up".

applyObservatory forwarded every raw alive:true->false transition straight to
EventOutboundDown; xray's observatory has effectively no hysteresis, and nothing
on the panel side debounced it (the email/Telegram subscribers are pure
formatters).

Fix: debounce per outbound. outbound.down now fires only after
outboundDownThreshold consecutive FAILED probes (new setting, default 3);
outbound.up fires immediately on the first successful probe and only when a down
was actually notified. The threshold gates the event itself, so email and
Telegram share one knob (exposed next to the outbound.down toggle).

The streak counts genuinely new probes (last_try_time advancing), not sampler
polls — the sampler runs every 2s but the observatory re-probes per its
probeInterval, so counting samples would trip the threshold instantly.
outboundDownThreshold=1 reproduces the legacy notify-on-first-failure behaviour.

Tuning the observatory's probe interval/timeout is not a workaround: those
probes also drive the load balancer's outbound selection, so loosening them to
quiet notifications would slow real failover away from a genuinely dead
outbound. Notifications don't need observatory-grade latency, so the tolerance
belongs at the notification layer, leaving the observatory (and balancer)
untouched.

Adds TestApplyObservatoryDebounce covering the threshold, probe-vs-sample
counting, single-blip suppression and the legacy path.

Co-authored-by: Yuriy Khachaturian <y.khachaturian@souzmult.ru>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-21 15:59:07 +02:00
..
controller Repo-wide self-correcting audit: 54 verified bug fixes (#5970) 2026-07-17 00:33:06 +02:00
entity feat(notifications): add a consecutive-failure threshold for outbound.down alerts (#5968) 2026-07-21 15:59:07 +02:00
global refactor: focused service files, leaf subpackages, and an internal/ layout (#5167) 2026-06-10 15:19:22 +02:00
job fix(node): adopt a node inbound's host overrides into the master 2026-07-11 23:17:57 +02:00
locale style: adopt golangci-lint v2 and resolve all findings 2026-06-27 15:42:22 +02:00
middleware fix(web): tighten database restore body-cap exemption (#5609) 2026-06-28 15:00:55 +02:00
network style: adopt golangci-lint v2 and resolve all findings 2026-06-27 15:42:22 +02:00
runtime Repo-wide self-correcting audit: 54 verified bug fixes (#5970) 2026-07-17 00:33:06 +02:00
service feat(notifications): add a consecutive-failure threshold for outbound.down alerts (#5968) 2026-07-21 15:59:07 +02:00
session refactor: focused service files, leaf subpackages, and an internal/ layout (#5167) 2026-06-10 15:19:22 +02:00
translation Repo-wide self-correcting audit: 54 verified bug fixes (#5970) 2026-07-17 00:33:06 +02:00
websocket feat(web): broadcast delta client stats above a snapshot threshold 2026-07-02 16:34:01 +02:00
cadence_test.go refactor(web): centralize background job cadences (#5269) 2026-06-14 22:50:24 +02:00
web.go Repo-wide self-correcting audit: 54 verified bug fixes (#5970) 2026-07-17 00:33:06 +02:00
web_mtls.go feat(node): node hardening — mTLS, hashed+zstd reconcile transport, per-node net metrics (#5382) 2026-06-16 12:19:33 +02:00
web_mtls_test.go feat(node): node hardening — mTLS, hashed+zstd reconcile transport, per-node net metrics (#5382) 2026-06-16 12:19:33 +02:00