3x-ui/internal/web/service/sync_scale_postgres_test.go

392 lines
12 KiB
Go
Raw Normal View History

package service
import (
"errors"
"fmt"
"strings"
"testing"
"time"
"github.com/google/uuid"
refactor: focused service files, leaf subpackages, and an internal/ layout (#5167) * refactor(service): split client.go into focused files client.go had grown to 4455 lines mixing ~10 responsibilities. Split it verbatim into cohesive same-package files (no behavior change): client.go foundation: ClientService, ClientWithAttachments, ClientCreatePayload, ErrClientNotInInbound, sqlInChunk client_locks.go inbound mutation locks, delete tombstones, compactOrphans client_lookup.go read-only lookups (GetByID, List, EffectiveFlow, ...) client_link.go inbound association sync (SyncInbound, DetachInbound, ...) client_crud.go single-client CRUD + validation + protocol defaults client_inbound_apply.go low-level inbound-settings mutators + by-email setters client_bulk.go bulk attach/detach/adjust/delete/create + DelDepleted client_traffic.go traffic-reset paths client_groups.go client group management client_paging.go paged listing, filtering, sorting, summary Every declaration moved unchanged (verified: identical func/type/const/var signature set before vs after). Imports redistributed per file via goimports. go build ./..., go vet, and go test ./web/service/... all pass. * refactor(service): split inbound.go into focused files inbound.go was 4100 lines. Split it verbatim into cohesive same-package files (no behavior change): inbound.go core inbound CRUD + InboundService (keeps pkg doc) inbound_protocol.go protocol / stream capability helpers inbound_node.go node/runtime/remote coordination + online tracking inbound_traffic.go traffic accounting, reset, client stats inbound_client_ips.go per-client IP tracking inbound_clients.go client lookups within inbounds + copy-clients inbound_disable.go auto-disable invalid inbounds/clients inbound_migration.go DB migrations inbound_sublink.go subscription link providers inbound_util.go generic slice/string helpers Identical func/type/const/var signature set before vs after; package doc comment preserved on inbound.go. Imports redistributed via goimports. Build, vet, and go test ./web/service/... all pass. * refactor(service): split tgbot.go into focused files tgbot.go was 3738 lines dominated by a 1246-line answerCallback. Split it verbatim into cohesive same-package files (no behavior change): tgbot.go lifecycle, bot setup, caches, small utils tgbot_router.go incoming update / command / callback dispatch tgbot_send.go outbound messaging primitives tgbot_client.go client views, actions, subscription links tgbot_inbound.go inbound listing / pickers tgbot_report.go server usage, exhausted, online, backups, notifications Identical func/type/const/var signature set before vs after. Imports redistributed via goimports. Build, vet, and go test ./web/service/... pass. * refactor(client): dedupe single-field by-email setters ResetClientIpLimitByEmail, ResetClientExpiryTimeByEmail, and ResetClientTrafficLimitByEmail shared an identical ~50-line body that resolves the inbound by email, confirms the client exists, rewrites a single-client settings payload, and delegates to UpdateInboundClient. Extract that into applyClientFieldByEmail(inboundSvc, email, mutate) and reduce each setter to a 3-line wrapper. Behavior is unchanged: same checks and error strings, same single-client payload contract, same totalGB guard. SetClientTelegramUserID (resolves by traffic id, different error text) and ToggleClientEnableByEmail/SetClientEnableByEmail (different return shape and a pre-read of the old state) intentionally keep their own bodies. * refactor(service): extract panel/ subpackage Move the panel-administration leaf services out of the flat service package into web/service/panel/ (package panel): user.go UserService (auth / 2FA / LDAP) panel.go PanelService (restart / self-update) + version helpers panel_other.go non-unix RestartPanel panel_unix.go unix RestartPanel api_token.go ApiTokenService websocket.go WebSocketService panel_test.go version/shellQuote unit tests These are leaves: they depend on core (SettingService, Release) but no core file references them, so the extraction creates no import cycle. Core references are now qualified (service.SettingService, service.Release); callers in main.go, web/web.go, and web/controller/* updated to panel.*. Build, vet, and go test ./web/... pass. * refactor(service): extract integration/ subpackage Move the external-provider integration leaves into web/service/integration/ (package integration): warp.go WarpService (Cloudflare WARP) nord.go NordService (NordVPN) custom_geo.go CustomGeoService (custom geo asset management) *_test.go custom_geo / panel-proxy tests These depend on core (SettingService, ServerService, XraySettingService) but no core file references them. xray_setting.go stays in core because it calls the unexported SettingService.saveSetting. The shared isBlockedIP SSRF helper (used by core url_safety.go and by custom_geo) now has a small copy in each package rather than being exported. Core references qualified; callers in web/web.go, web/job/*, and web/controller/* updated to integration.*. Build, vet, and go test ./web/... pass. * refactor(service): extract tgbot/ subpackage Move the Telegram bot (6 files + test) into web/service/tgbot/ (package tgbot). It is a leaf: it embeds five core services (Inbound/Client/Setting/ Server/Xray) and the core never references it, so no import cycle. To support the package boundary without changing behavior: - core exposes XrayProcess() *xray.Process so tgbot keeps calling the exact same running-process methods it used via the package-level `p`; - three core methods tgbot calls are exported: ClientService.checkIs- EnabledByEmail -> CheckIsEnabledByEmail, InboundService.getAllEmails -> GetAllEmails (callers updated in-package); - tgbot's embedded-field types and the few core type refs (Status, ClientCreatePayload, SanitizePublicHTTPURL) are now service-qualified. Callers in main.go, web/web.go, web/job/*, and web/controller/* updated to tgbot.*. Build, vet, and go test ./web/... pass. * refactor(service): extract outbound/ subpackage OutboundService (outbound.go) imports only neutral packages (config, database, model, xray) and its production code is referenced by no core or sibling service file — only by web/controller/xray_setting.go and web/job/xray_traffic_job.go. Move it to web/service/outbound/ (package outbound); no core qualification needed inside. Callers updated to outbound.*. The one coupling was a tiny pure test helper, outboundsContainTag, used by both outbound.go and the core outbound_subscription_test.go; it now has a small copy in that test file rather than being shared across the boundary. Build, vet, and go test ./web/... pass. * refactor(util): move wireguard into its own subpackage util/wireguard.go was the lone file of the root `util` package (24 lines, one exported func GenerateWireguardKeypair), while every other util concern lives in a focused subpackage (util/common, util/crypto, util/netsafe, ...). Move it to util/wireguard/ (package wireguard) for consistency; its only importer, web/service/integration/warp.go, is updated. The root `util` package no longer exists. * refactor(sub): drop redundant sub prefix from filenames Inside package sub the subXxx.go prefix just repeats the package name (like client_*.go did inside service). Rename for consistency; content and type names are unchanged: subController.go -> controller.go subService.go -> service.go subClashService.go -> clash_service.go subJsonService.go -> json_service.go (+ matching _test.go files) * refactor(controller): rename xui.go -> spa.go XUIController serves the panel's single-page-app shell; spa.go names that role plainly (the other controller files are domain-named). File rename only — the type stays XUIController. api_docs_test.go keys route base paths by filename, so its "xui.go" case is updated to "spa.go". * refactor: move backend packages under internal/ Adopt the idiomatic Go application layout: the backend packages now live under internal/ (a boundary the toolchain enforces), signalling private implementation instead of a library-style flat root. No runtime behavior changes — only import paths and a few build/config paths move. Moved: config, database, logger, mtproto, sub, util, web, xray -> internal/. main.go stays at the repo root and tools/openapigen stays under tools/ (both still import internal/* because the internal rule keys off the module root). The module path github.com/mhsanaei/3x-ui/v3 is unchanged; 149 .go files had their import prefix rewritten to .../internal/<pkg>. Couplings the Go compiler can't see, updated to the new layout: - frontend i18n imports of web/translation (react.ts, setup.components.ts) - vite outDir + eslint/tsconfig ignore globs -> internal/web/dist - Dockerfile COPY paths for web/dist and web/translation - locale.go os.DirFS("web") disk fallback -> "internal/web" - .gitignore and ci.yml go:embed stub for internal/web/dist - api_docs_test.go repo-root relative walk (one level deeper) - tools/openapigen filesystem package paths; ApiTokenView repointed to the web/service/panel subpackage and codegen regenerated (clears a stale type the ci.yml codegen check was failing on) Verified: go build/vet/test (all packages), and frontend typecheck, lint, vitest (478 tests), and production build into internal/web/dist. * fix(config): keep test runs from writing logs into the source tree GetLogFolder() returns a CWD-relative "./log" on Windows. Under `go test` the working directory is each package's own folder, so InitLogger (called by tests in web/job, web/service, xray, web/websocket) created stray log/ directories scattered through the source tree (e.g. internal/web/job/log/). Redirect to a shared temp folder when testing.Testing() reports a test run. Production behavior is unchanged: Windows still uses ./log next to the binary and Linux /var/log/x-ui. The log files were always gitignored (*.log) and never committed; this just stops the noise at the source. * docs: move subscription-template guide out of root into docs/ sub_templates/ was a top-level folder holding only a README and no actual templates (3x-ui ships none by design), referenced nowhere and unlinked from any doc — it read like an empty placeholder cluttering the repo root. Move the guide to docs/custom-subscription-templates.md (a proper docs home), reword its intro to read as documentation rather than a folder note, link it from the Features list in README.md, and drop the empty sub_templates/ folder. * fix: update stale web/ path references after the internal/ move The internal/ migration rewrote Go import paths but left some references to the old top-level layout in docs, comments, and a few runtime disk paths. Functional (dev-mode only): the disk-serving fallbacks that read the Vite build from disk when running from source still pointed at web/dist/, which moved to internal/web/dist/ — so `os.DirFS`/`os.Stat`/`os.ReadFile` in internal/web/web.go and internal/sub/{sub,controller}.go are corrected. Production was unaffected (it serves the embedded FS; verified by the Docker build), but `go run` with a live frontend build silently fell back to embed. Docs/comments: frontend/README.md, CONTRIBUTING.md, the claude-issue-bot and release workflows, the openapigen -root help text, and assorted Go comments now reference internal/web, internal/database, internal/sub, internal/xray, etc. Package-name mentions (the "web" package), root paths (main.go, frontend/, install scripts, /etc/x-ui), routes (/panel/api/xray), and the historical "web/assets no longer exists" note were intentionally left as-is. * refactor(web): remove the legacy /xui -> /panel redirect middleware RedirectMiddleware existed only for backward compatibility with the old `/xui` URL scheme (301-redirecting /xui and /xui/API to /panel and /panel/api). That cutover was long ago, so drop the middleware, its registration in initRouter, and the now-inaccurate "URL redirection" mention in the middleware package doc. Old /xui URLs now 404 like any other unknown path. HTTPS auto-redirect and auth redirects are unrelated and stay. * build: fix .dockerignore for internal/ layout and exclude runtime dir - web/dist -> internal/web/dist: the embedded frontend moved under internal/, so the stale exclude no longer matched and the locally-built dist could be sent to the build context (the frontend stage rebuilds it fresh anyway). - exclude x-ui/: the local runtime directory (SQLite db, geo .dat files, xray binaries, certs — ~150MB) was being shipped into the build context for no reason. Verified the pattern excludes only the directory and still keeps x-ui.sh, which the Dockerfile copies to /usr/bin/x-ui.
2026-06-10 06:19:22 -07:00
"github.com/mhsanaei/3x-ui/v3/internal/database"
"github.com/mhsanaei/3x-ui/v3/internal/database/model"
"gorm.io/gorm"
)
func syncInboundOld(tx *gorm.DB, inboundId int, clients []model.Client) error {
if tx == nil {
tx = database.GetDB()
}
if err := tx.Where("inbound_id = ?", inboundId).Delete(&model.ClientInbound{}).Error; err != nil {
return err
}
for i := range clients {
c := clients[i]
email := strings.TrimSpace(c.Email)
if email == "" {
continue
}
incoming := c.ToRecord()
row := &model.ClientRecord{}
err := tx.Where("email = ?", email).First(row).Error
if err != nil && !errors.Is(err, gorm.ErrRecordNotFound) {
return err
}
if errors.Is(err, gorm.ErrRecordNotFound) {
if err := tx.Create(incoming).Error; err != nil {
return err
}
row = incoming
} else {
row.Flow = incoming.Flow
row.SubID = incoming.SubID
row.LimitIP = incoming.LimitIP
row.TotalGB = incoming.TotalGB
row.ExpiryTime = incoming.ExpiryTime
row.Enable = incoming.Enable
row.TgID = incoming.TgID
row.Comment = incoming.Comment
row.Reset = incoming.Reset
preservedUpdatedAt := max(incoming.UpdatedAt, row.UpdatedAt)
row.UpdatedAt = preservedUpdatedAt
if err := tx.Save(row).Error; err != nil {
return err
}
if err := tx.Model(&model.ClientRecord{}).
Where("id = ?", row.Id).
UpdateColumn("updated_at", preservedUpdatedAt).Error; err != nil {
return err
}
}
link := model.ClientInbound{ClientId: row.Id, InboundId: inboundId, FlowOverride: c.Flow}
if err := tx.Create(&link).Error; err != nil {
return err
}
}
return nil
}
func makeScaleClients(n int) []model.Client {
out := make([]model.Client, n)
for i := range n {
out[i] = model.Client{
ID: uuid.NewString(),
Email: fmt.Sprintf("user-%07d@scale", i),
SubID: fmt.Sprintf("sub-%07d", i),
Enable: true,
}
}
return out
}
func TestSyncInboundPostgresScale(t *testing.T) {
setupScaleDB(t)
svc := &ClientService{}
sizes := []int{5000, 10000, 20000, 50000, 100000, 200000}
for _, n := range sizes {
t.Run(fmt.Sprintf("N=%d", n), func(t *testing.T) {
db := database.GetDB()
resetScaleTables(t, db, "inbounds", "clients", "client_inbounds")
clients := makeScaleClients(n)
ib := &model.Inbound{
Tag: fmt.Sprintf("scale-%d", n),
Enable: true,
Port: 40000,
Protocol: model.VLESS,
Settings: clientsSettings(t, clients),
}
if err := db.Create(ib).Error; err != nil {
t.Fatalf("create inbound: %v", err)
}
start := time.Now()
if err := svc.SyncInbound(nil, ib.Id, clients); err != nil {
t.Fatalf("seed SyncInbound: %v", err)
}
seed := time.Since(start)
clients[n/2].Enable = !clients[n/2].Enable
start = time.Now()
if err := svc.SyncInbound(nil, ib.Id, clients); err != nil {
t.Fatalf("toggle SyncInbound (new): %v", err)
}
toggleNew := time.Since(start)
start = time.Now()
if err := svc.SyncInbound(nil, ib.Id, clients); err != nil {
t.Fatalf("noop SyncInbound (new): %v", err)
}
noopNew := time.Since(start)
toggleOld := time.Duration(0)
if n <= 10000 {
clients[n/2].Enable = !clients[n/2].Enable
start = time.Now()
if err := syncInboundOld(db, ib.Id, clients); err != nil {
t.Fatalf("toggle SyncInbound (old): %v", err)
}
toggleOld = time.Since(start)
}
var linkCount, recCount int64
db.Model(&model.ClientInbound{}).Where("inbound_id = ?", ib.Id).Count(&linkCount)
db.Model(&model.ClientRecord{}).Count(&recCount)
if int(linkCount) != n || int(recCount) != n {
t.Fatalf("row mismatch: links=%d records=%d want %d", linkCount, recCount, n)
}
oldStr, speedup := "skipped", ""
if toggleOld > 0 {
oldStr = toggleOld.Round(time.Millisecond).String()
speedup = fmt.Sprintf(" speedup=%.0fx", float64(toggleOld)/float64(maxDur(toggleNew, time.Millisecond)))
}
t.Logf("N=%-7d seed=%-10v toggle_new=%-10v noop_new=%-10v toggle_old=%-10s%s",
n, seed.Round(time.Millisecond), toggleNew.Round(time.Millisecond),
noopNew.Round(time.Millisecond), oldStr, speedup)
})
}
}
func maxDur(d, floor time.Duration) time.Duration {
if d < floor {
return floor
}
return d
}
func TestAddDelClientPostgresScale(t *testing.T) {
setupScaleDB(t)
svc := &ClientService{}
inboundSvc := &InboundService{}
sizes := []int{5000, 20000, 50000, 100000, 200000}
for _, n := range sizes {
t.Run(fmt.Sprintf("N=%d", n), func(t *testing.T) {
db := database.GetDB()
resetScaleTables(t, db, "inbounds", "clients", "client_inbounds", "client_traffics")
clients := makeScaleClients(n)
ib := &model.Inbound{
Tag: fmt.Sprintf("adddel-%d", n),
Enable: true,
Port: 40000,
Protocol: model.VLESS,
Settings: clientsSettings(t, clients),
}
if err := db.Create(ib).Error; err != nil {
t.Fatalf("create inbound: %v", err)
}
if err := svc.SyncInbound(nil, ib.Id, clients); err != nil {
t.Fatalf("seed SyncInbound: %v", err)
}
newC := model.Client{
ID: uuid.NewString(),
Email: "added-client@scale",
SubID: "added-sub",
Enable: true,
}
addData := &model.Inbound{Id: ib.Id, Protocol: model.VLESS, Settings: clientsSettings(t, []model.Client{newC})}
start := time.Now()
if _, err := svc.AddInboundClient(inboundSvc, addData); err != nil {
t.Fatalf("AddInboundClient: %v", err)
}
addDur := time.Since(start)
delEmail := clients[n/2].Email
start = time.Now()
if _, err := svc.DelInboundClientByEmail(inboundSvc, ib.Id, delEmail, false, false); err != nil {
t.Fatalf("DelInboundClientByEmail: %v", err)
}
delDur := time.Since(start)
perf(clients): scale add/delete and bulk client operations Follow-up to the SyncInbound bulk rewrite, fixing the remaining O(M*N) and O(M)-round-trip behaviour in the add/delete and bulk paths that made them time out on large inbounds (worst case minutes), especially on PostgreSQL. - compactOrphans: chunk the "email IN (...)" lookup (400/batch) instead of binding every email at once. A single huge IN exceeded PostgreSQL's 65535-parameter limit (and SQLite's) and made the planner pathological, so add/delete failed outright past ~100k clients. - emailsUsedByOtherInbounds: new batched form used by delInboundClients (BulkDetach) and bulkDelInboundClients (BulkDelete), replacing a per-email global JSON scan (O(M*N)) with one scan, and skipped entirely when keepTraffic is set. - BulkCreate: rewritten to validate/dedup in one pass, then group clients by inbound and add them in a single addInboundClient call per inbound (one getAllEmailSubIDs, one settings rewrite, one SyncInbound) instead of running the full single-create pipeline per client. - Bulk delete/adjust: batch DelClientStat/DelClientIPs with IN deletes and wrap the settings Save + SyncInbound in one transaction, so the per-row writes share a single fsync instead of one per row. Measured on PostgreSQL 16 (one inbound, M=2000 affected clients): - create: 8m35s (M=500) -> ~1-5s - detach: 52s -> ~4s (flat in N) - delete: ~16s -> ~1-4s - adjust: ~20s -> ~7-10s add/delete of a single client on a 200k-client inbound stays in seconds. sync_scale_postgres_test.go adds skip-gated benchmarks (XUI_DB_TYPE= postgres) for the single add/delete and the five bulk operations.
2026-06-04 10:41:00 -07:00
var recCount, linkCount int64
db.Model(&model.ClientRecord{}).Count(&recCount)
perf(clients): scale add/delete and bulk client operations Follow-up to the SyncInbound bulk rewrite, fixing the remaining O(M*N) and O(M)-round-trip behaviour in the add/delete and bulk paths that made them time out on large inbounds (worst case minutes), especially on PostgreSQL. - compactOrphans: chunk the "email IN (...)" lookup (400/batch) instead of binding every email at once. A single huge IN exceeded PostgreSQL's 65535-parameter limit (and SQLite's) and made the planner pathological, so add/delete failed outright past ~100k clients. - emailsUsedByOtherInbounds: new batched form used by delInboundClients (BulkDetach) and bulkDelInboundClients (BulkDelete), replacing a per-email global JSON scan (O(M*N)) with one scan, and skipped entirely when keepTraffic is set. - BulkCreate: rewritten to validate/dedup in one pass, then group clients by inbound and add them in a single addInboundClient call per inbound (one getAllEmailSubIDs, one settings rewrite, one SyncInbound) instead of running the full single-create pipeline per client. - Bulk delete/adjust: batch DelClientStat/DelClientIPs with IN deletes and wrap the settings Save + SyncInbound in one transaction, so the per-row writes share a single fsync instead of one per row. Measured on PostgreSQL 16 (one inbound, M=2000 affected clients): - create: 8m35s (M=500) -> ~1-5s - detach: 52s -> ~4s (flat in N) - delete: ~16s -> ~1-4s - adjust: ~20s -> ~7-10s add/delete of a single client on a 200k-client inbound stays in seconds. sync_scale_postgres_test.go adds skip-gated benchmarks (XUI_DB_TYPE= postgres) for the single add/delete and the five bulk operations.
2026-06-04 10:41:00 -07:00
db.Model(&model.ClientInbound{}).Where("inbound_id = ?", ib.Id).Count(&linkCount)
perf(clients): scale add/delete and bulk client operations Follow-up to the SyncInbound bulk rewrite, fixing the remaining O(M*N) and O(M)-round-trip behaviour in the add/delete and bulk paths that made them time out on large inbounds (worst case minutes), especially on PostgreSQL. - compactOrphans: chunk the "email IN (...)" lookup (400/batch) instead of binding every email at once. A single huge IN exceeded PostgreSQL's 65535-parameter limit (and SQLite's) and made the planner pathological, so add/delete failed outright past ~100k clients. - emailsUsedByOtherInbounds: new batched form used by delInboundClients (BulkDetach) and bulkDelInboundClients (BulkDelete), replacing a per-email global JSON scan (O(M*N)) with one scan, and skipped entirely when keepTraffic is set. - BulkCreate: rewritten to validate/dedup in one pass, then group clients by inbound and add them in a single addInboundClient call per inbound (one getAllEmailSubIDs, one settings rewrite, one SyncInbound) instead of running the full single-create pipeline per client. - Bulk delete/adjust: batch DelClientStat/DelClientIPs with IN deletes and wrap the settings Save + SyncInbound in one transaction, so the per-row writes share a single fsync instead of one per row. Measured on PostgreSQL 16 (one inbound, M=2000 affected clients): - create: 8m35s (M=500) -> ~1-5s - detach: 52s -> ~4s (flat in N) - delete: ~16s -> ~1-4s - adjust: ~20s -> ~7-10s add/delete of a single client on a 200k-client inbound stays in seconds. sync_scale_postgres_test.go adds skip-gated benchmarks (XUI_DB_TYPE= postgres) for the single add/delete and the five bulk operations.
2026-06-04 10:41:00 -07:00
t.Logf("N=%-7d add=%-10v del=%-10v records=%d links=%d", n,
addDur.Round(time.Millisecond), delDur.Round(time.Millisecond), recCount, linkCount)
})
}
}
func TestGroupAndListPostgresScale(t *testing.T) {
setupScaleDB(t)
svc := &ClientService{}
sizes := []int{5000, 100000}
for _, n := range sizes {
t.Run(fmt.Sprintf("N=%d", n), func(t *testing.T) {
db := database.GetDB()
resetScaleTables(t, db, "inbounds", "clients", "client_inbounds", "client_traffics")
clients := makeScaleClients(n)
ib := &model.Inbound{Tag: fmt.Sprintf("grp-%d", n), Enable: true, Port: 40000, Protocol: model.VLESS, Settings: clientsSettings(t, clients)}
if err := db.Create(ib).Error; err != nil {
t.Fatalf("create inbound: %v", err)
}
if err := svc.SyncInbound(nil, ib.Id, clients); err != nil {
t.Fatalf("seed SyncInbound: %v", err)
}
db.Exec("ANALYZE")
emails := make([]string, n)
for i := range n {
emails[i] = clients[i].Email
}
start := time.Now()
if _, err := svc.AddToGroup(emails, "benchgroup"); err != nil {
t.Fatalf("AddToGroup: %v", err)
}
addDur := time.Since(start)
start = time.Now()
if _, err := svc.RemoveFromGroup(emails); err != nil {
t.Fatalf("RemoveFromGroup: %v", err)
}
rmDur := time.Since(start)
start = time.Now()
list, err := svc.List()
if err != nil {
t.Fatalf("List: %v", err)
}
listDur := time.Since(start)
if len(list) != n {
t.Fatalf("List returned %d, want %d", len(list), n)
}
t.Logf("N=%-7d bulkAdd=%-9v bulkRemove=%-9v list=%-9v", n,
addDur.Round(time.Millisecond), rmDur.Round(time.Millisecond), listDur.Round(time.Millisecond))
})
}
}
func TestDelAllClientsPostgresScale(t *testing.T) {
setupScaleDB(t)
svc := &ClientService{}
inboundSvc := &InboundService{}
sizes := []int{5000, 50000, 100000}
for _, n := range sizes {
t.Run(fmt.Sprintf("N=%d", n), func(t *testing.T) {
db := database.GetDB()
resetScaleTables(t, db, "inbounds", "clients", "client_inbounds", "client_traffics")
clients := makeScaleClients(n)
ib := &model.Inbound{Tag: fmt.Sprintf("delall-%d", n), Enable: true, Port: 40000, Protocol: model.VLESS, Settings: clientsSettings(t, clients)}
if err := db.Create(ib).Error; err != nil {
t.Fatalf("create inbound: %v", err)
}
if err := svc.SyncInbound(nil, ib.Id, clients); err != nil {
t.Fatalf("seed SyncInbound: %v", err)
}
emails, err := inboundSvc.EmailsByInbound(ib.Id)
if err != nil {
t.Fatalf("EmailsByInbound: %v", err)
}
start := time.Now()
res, _, err := svc.BulkDelete(inboundSvc, emails, false)
if err != nil {
t.Fatalf("BulkDelete: %v", err)
}
dur := time.Since(start)
var recCount, linkCount int64
db.Model(&model.ClientRecord{}).Count(&recCount)
db.Model(&model.ClientInbound{}).Where("inbound_id = ?", ib.Id).Count(&linkCount)
if recCount != 0 || linkCount != 0 {
t.Fatalf("after delAll: records=%d links=%d want 0/0", recCount, linkCount)
}
t.Logf("N=%-7d delAllClients=%-10v deleted=%d", n, dur.Round(time.Millisecond), res.Deleted)
})
perf(clients): scale add/delete and bulk client operations Follow-up to the SyncInbound bulk rewrite, fixing the remaining O(M*N) and O(M)-round-trip behaviour in the add/delete and bulk paths that made them time out on large inbounds (worst case minutes), especially on PostgreSQL. - compactOrphans: chunk the "email IN (...)" lookup (400/batch) instead of binding every email at once. A single huge IN exceeded PostgreSQL's 65535-parameter limit (and SQLite's) and made the planner pathological, so add/delete failed outright past ~100k clients. - emailsUsedByOtherInbounds: new batched form used by delInboundClients (BulkDetach) and bulkDelInboundClients (BulkDelete), replacing a per-email global JSON scan (O(M*N)) with one scan, and skipped entirely when keepTraffic is set. - BulkCreate: rewritten to validate/dedup in one pass, then group clients by inbound and add them in a single addInboundClient call per inbound (one getAllEmailSubIDs, one settings rewrite, one SyncInbound) instead of running the full single-create pipeline per client. - Bulk delete/adjust: batch DelClientStat/DelClientIPs with IN deletes and wrap the settings Save + SyncInbound in one transaction, so the per-row writes share a single fsync instead of one per row. Measured on PostgreSQL 16 (one inbound, M=2000 affected clients): - create: 8m35s (M=500) -> ~1-5s - detach: 52s -> ~4s (flat in N) - delete: ~16s -> ~1-4s - adjust: ~20s -> ~7-10s add/delete of a single client on a 200k-client inbound stays in seconds. sync_scale_postgres_test.go adds skip-gated benchmarks (XUI_DB_TYPE= postgres) for the single add/delete and the five bulk operations.
2026-06-04 10:41:00 -07:00
}
}
func TestBulkOpsPostgresScale(t *testing.T) {
setupScaleDB(t)
perf(clients): scale add/delete and bulk client operations Follow-up to the SyncInbound bulk rewrite, fixing the remaining O(M*N) and O(M)-round-trip behaviour in the add/delete and bulk paths that made them time out on large inbounds (worst case minutes), especially on PostgreSQL. - compactOrphans: chunk the "email IN (...)" lookup (400/batch) instead of binding every email at once. A single huge IN exceeded PostgreSQL's 65535-parameter limit (and SQLite's) and made the planner pathological, so add/delete failed outright past ~100k clients. - emailsUsedByOtherInbounds: new batched form used by delInboundClients (BulkDetach) and bulkDelInboundClients (BulkDelete), replacing a per-email global JSON scan (O(M*N)) with one scan, and skipped entirely when keepTraffic is set. - BulkCreate: rewritten to validate/dedup in one pass, then group clients by inbound and add them in a single addInboundClient call per inbound (one getAllEmailSubIDs, one settings rewrite, one SyncInbound) instead of running the full single-create pipeline per client. - Bulk delete/adjust: batch DelClientStat/DelClientIPs with IN deletes and wrap the settings Save + SyncInbound in one transaction, so the per-row writes share a single fsync instead of one per row. Measured on PostgreSQL 16 (one inbound, M=2000 affected clients): - create: 8m35s (M=500) -> ~1-5s - detach: 52s -> ~4s (flat in N) - delete: ~16s -> ~1-4s - adjust: ~20s -> ~7-10s add/delete of a single client on a 200k-client inbound stays in seconds. sync_scale_postgres_test.go adds skip-gated benchmarks (XUI_DB_TYPE= postgres) for the single add/delete and the five bulk operations.
2026-06-04 10:41:00 -07:00
svc := &ClientService{}
inboundSvc := &InboundService{}
sizes := []int{5000, 20000, 50000, 100000}
const m = 2000
for _, n := range sizes {
t.Run(fmt.Sprintf("N=%d", n), func(t *testing.T) {
db := database.GetDB()
resetScaleTables(t, db, "inbounds", "clients", "client_inbounds", "client_traffics")
perf(clients): scale add/delete and bulk client operations Follow-up to the SyncInbound bulk rewrite, fixing the remaining O(M*N) and O(M)-round-trip behaviour in the add/delete and bulk paths that made them time out on large inbounds (worst case minutes), especially on PostgreSQL. - compactOrphans: chunk the "email IN (...)" lookup (400/batch) instead of binding every email at once. A single huge IN exceeded PostgreSQL's 65535-parameter limit (and SQLite's) and made the planner pathological, so add/delete failed outright past ~100k clients. - emailsUsedByOtherInbounds: new batched form used by delInboundClients (BulkDetach) and bulkDelInboundClients (BulkDelete), replacing a per-email global JSON scan (O(M*N)) with one scan, and skipped entirely when keepTraffic is set. - BulkCreate: rewritten to validate/dedup in one pass, then group clients by inbound and add them in a single addInboundClient call per inbound (one getAllEmailSubIDs, one settings rewrite, one SyncInbound) instead of running the full single-create pipeline per client. - Bulk delete/adjust: batch DelClientStat/DelClientIPs with IN deletes and wrap the settings Save + SyncInbound in one transaction, so the per-row writes share a single fsync instead of one per row. Measured on PostgreSQL 16 (one inbound, M=2000 affected clients): - create: 8m35s (M=500) -> ~1-5s - detach: 52s -> ~4s (flat in N) - delete: ~16s -> ~1-4s - adjust: ~20s -> ~7-10s add/delete of a single client on a 200k-client inbound stays in seconds. sync_scale_postgres_test.go adds skip-gated benchmarks (XUI_DB_TYPE= postgres) for the single add/delete and the five bulk operations.
2026-06-04 10:41:00 -07:00
clients := makeScaleClients(n)
exp := time.Now().AddDate(1, 0, 0).UnixMilli()
for i := range clients {
clients[i].ExpiryTime = exp
clients[i].TotalGB = 100 << 30
}
ib := &model.Inbound{Tag: fmt.Sprintf("bulk-%d", n), Enable: true, Port: 40000, Protocol: model.VLESS, Settings: clientsSettings(t, clients)}
if err := db.Create(ib).Error; err != nil {
t.Fatalf("create inbound: %v", err)
}
if err := svc.SyncInbound(nil, ib.Id, clients); err != nil {
t.Fatalf("seed SyncInbound: %v", err)
}
ib2 := &model.Inbound{Tag: fmt.Sprintf("bulk2-%d", n), Enable: true, Port: 40001, Protocol: model.VLESS, Settings: `{"clients":[]}`}
if err := db.Create(ib2).Error; err != nil {
t.Fatalf("create inbound2: %v", err)
}
emailsM := make([]string, m)
for i := range m {
perf(clients): scale add/delete and bulk client operations Follow-up to the SyncInbound bulk rewrite, fixing the remaining O(M*N) and O(M)-round-trip behaviour in the add/delete and bulk paths that made them time out on large inbounds (worst case minutes), especially on PostgreSQL. - compactOrphans: chunk the "email IN (...)" lookup (400/batch) instead of binding every email at once. A single huge IN exceeded PostgreSQL's 65535-parameter limit (and SQLite's) and made the planner pathological, so add/delete failed outright past ~100k clients. - emailsUsedByOtherInbounds: new batched form used by delInboundClients (BulkDetach) and bulkDelInboundClients (BulkDelete), replacing a per-email global JSON scan (O(M*N)) with one scan, and skipped entirely when keepTraffic is set. - BulkCreate: rewritten to validate/dedup in one pass, then group clients by inbound and add them in a single addInboundClient call per inbound (one getAllEmailSubIDs, one settings rewrite, one SyncInbound) instead of running the full single-create pipeline per client. - Bulk delete/adjust: batch DelClientStat/DelClientIPs with IN deletes and wrap the settings Save + SyncInbound in one transaction, so the per-row writes share a single fsync instead of one per row. Measured on PostgreSQL 16 (one inbound, M=2000 affected clients): - create: 8m35s (M=500) -> ~1-5s - detach: 52s -> ~4s (flat in N) - delete: ~16s -> ~1-4s - adjust: ~20s -> ~7-10s add/delete of a single client on a 200k-client inbound stays in seconds. sync_scale_postgres_test.go adds skip-gated benchmarks (XUI_DB_TYPE= postgres) for the single add/delete and the five bulk operations.
2026-06-04 10:41:00 -07:00
emailsM[i] = clients[i].Email
}
t0 := time.Now()
feat(clients): support setting HWID limit and MTProto ad-tag in bulk adjust (#6399) * feat(clients): support setting HWID limit and MTProto ad-tag in bulk adjust Add HWID device limit and Telegram MTProto sponsor channel (ad-tag) support to the bulk client adjustment flow in both the panel API and frontend ClientBulkAdjustModal. Co-Authored-By: Claude Code <noreply@anthropic.com> * fix(clients): gate adTag to MTProto inbounds and avoid inbound rewrite for limitHwid Co-Authored-By: Claude Code <noreply@anthropic.com> * fix(clients): stamp updated_at only on the clients a bulk adjust changed The updated_at write was gated on hasInboundChanges, which accumulates over the whole inbound instead of describing the client in hand. Once any client in the settings array changed, every client after it was re-stamped as well, so whether an untouched client kept its own updated_at depended on its position in the array. That field feeds node-snapshot conflict resolution, where a spurious bump lets a stale snapshot value win over the stored record. Track the change per client and fold it into the inbound-level flag where the stamp is written, so the early return still skips a save whose settings JSON would be unchanged. Also condenses the BulkAdjust doc comment back to the two-line maximum. * docs(api): regenerate the bulkAdjust reference for limitHwid and adTag frontend/public/openapi.json was copied to docs/public/, but pnpm gen:api was never re-run, so the API reference page's heading, anchor id and search index still described bulkAdjust without limitHwid or adTag. docs-ci.yml fires only on docs/**, and that path had been touched, so nothing flagged the stale MDX. The externalLinks hunks are the generator rewrapping lines main had left stale, not a content change. * fix(i18n): stop enumerating fields in the bulk-adjust empty-form message bulkAdjustNothing listed the fields the form accepts, so it went stale every time one was added: only en-US ever gained "flow", leaving the other twelve locales describing days and traffic alone, and limitHwid and adTag would have repeated that. Say that one field is required instead of naming which, so the message cannot drift again.
2026-09-10 07:24:48 -07:00
if _, _, err := svc.BulkAdjust(inboundSvc, emailsM, 7, 1<<30, "", nil, ""); err != nil {
perf(clients): scale add/delete and bulk client operations Follow-up to the SyncInbound bulk rewrite, fixing the remaining O(M*N) and O(M)-round-trip behaviour in the add/delete and bulk paths that made them time out on large inbounds (worst case minutes), especially on PostgreSQL. - compactOrphans: chunk the "email IN (...)" lookup (400/batch) instead of binding every email at once. A single huge IN exceeded PostgreSQL's 65535-parameter limit (and SQLite's) and made the planner pathological, so add/delete failed outright past ~100k clients. - emailsUsedByOtherInbounds: new batched form used by delInboundClients (BulkDetach) and bulkDelInboundClients (BulkDelete), replacing a per-email global JSON scan (O(M*N)) with one scan, and skipped entirely when keepTraffic is set. - BulkCreate: rewritten to validate/dedup in one pass, then group clients by inbound and add them in a single addInboundClient call per inbound (one getAllEmailSubIDs, one settings rewrite, one SyncInbound) instead of running the full single-create pipeline per client. - Bulk delete/adjust: batch DelClientStat/DelClientIPs with IN deletes and wrap the settings Save + SyncInbound in one transaction, so the per-row writes share a single fsync instead of one per row. Measured on PostgreSQL 16 (one inbound, M=2000 affected clients): - create: 8m35s (M=500) -> ~1-5s - detach: 52s -> ~4s (flat in N) - delete: ~16s -> ~1-4s - adjust: ~20s -> ~7-10s add/delete of a single client on a 200k-client inbound stays in seconds. sync_scale_postgres_test.go adds skip-gated benchmarks (XUI_DB_TYPE= postgres) for the single add/delete and the five bulk operations.
2026-06-04 10:41:00 -07:00
t.Fatalf("BulkAdjust: %v", err)
}
adjustDur := time.Since(t0)
t0 = time.Now()
if _, _, err := svc.BulkAttach(inboundSvc, emailsM, []int{ib2.Id}); err != nil {
t.Fatalf("BulkAttach: %v", err)
}
attachDur := time.Since(t0)
t0 = time.Now()
if _, _, err := svc.BulkDetach(inboundSvc, emailsM, []int{ib2.Id}); err != nil {
t.Fatalf("BulkDetach: %v", err)
}
detachDur := time.Since(t0)
payloads := make([]ClientCreatePayload, m)
for i := range m {
perf(clients): scale add/delete and bulk client operations Follow-up to the SyncInbound bulk rewrite, fixing the remaining O(M*N) and O(M)-round-trip behaviour in the add/delete and bulk paths that made them time out on large inbounds (worst case minutes), especially on PostgreSQL. - compactOrphans: chunk the "email IN (...)" lookup (400/batch) instead of binding every email at once. A single huge IN exceeded PostgreSQL's 65535-parameter limit (and SQLite's) and made the planner pathological, so add/delete failed outright past ~100k clients. - emailsUsedByOtherInbounds: new batched form used by delInboundClients (BulkDetach) and bulkDelInboundClients (BulkDelete), replacing a per-email global JSON scan (O(M*N)) with one scan, and skipped entirely when keepTraffic is set. - BulkCreate: rewritten to validate/dedup in one pass, then group clients by inbound and add them in a single addInboundClient call per inbound (one getAllEmailSubIDs, one settings rewrite, one SyncInbound) instead of running the full single-create pipeline per client. - Bulk delete/adjust: batch DelClientStat/DelClientIPs with IN deletes and wrap the settings Save + SyncInbound in one transaction, so the per-row writes share a single fsync instead of one per row. Measured on PostgreSQL 16 (one inbound, M=2000 affected clients): - create: 8m35s (M=500) -> ~1-5s - detach: 52s -> ~4s (flat in N) - delete: ~16s -> ~1-4s - adjust: ~20s -> ~7-10s add/delete of a single client on a 200k-client inbound stays in seconds. sync_scale_postgres_test.go adds skip-gated benchmarks (XUI_DB_TYPE= postgres) for the single add/delete and the five bulk operations.
2026-06-04 10:41:00 -07:00
payloads[i] = ClientCreatePayload{
Client: model.Client{ID: uuid.NewString(), Email: fmt.Sprintf("bulknew-%07d@scale", i), SubID: fmt.Sprintf("bnsub-%07d", i), Enable: true},
InboundIds: []int{ib.Id},
}
}
t0 = time.Now()
if _, _, err := svc.BulkCreate(inboundSvc, payloads); err != nil {
t.Fatalf("BulkCreate: %v", err)
}
createDur := time.Since(t0)
t0 = time.Now()
if _, _, err := svc.BulkDelete(inboundSvc, emailsM, false); err != nil {
t.Fatalf("BulkDelete: %v", err)
}
deleteDur := time.Since(t0)
t.Logf("N=%-6d M=%d adjust=%-9v attach=%-9v detach=%-9v create=%-9v delete=%-9v", n, m,
adjustDur.Round(time.Millisecond), attachDur.Round(time.Millisecond), detachDur.Round(time.Millisecond),
createDur.Round(time.Millisecond), deleteDur.Round(time.Millisecond))
})
}
}