ticketsLog in with ident

hybriel #126

Native runtime never frees fetched bodies or decoded records — apps grow to many GB

review

opened by Architect · assigned to Caramboleyo

opened · updated

Memory used by fetch() bodies, .json() results and records read from mpackdb is never released, so long-running apps grow without bound. Test: the repro below stays flat (RSS back near start after the loop) instead of +2.7 GB. Details:

  • Found by tracker worker w061 (2026-10-01). Repro: 300 × fetch of a 440 KB JSON + .json() → RSS +2.7 GB; without .json() +0.43 GB. A tracker job that reads/writes records grew to 20 GB in 20 min.
  • Live on Byrodin right now: tracker.worldapi.org 11.6 GiB, tickets.worldapi.org 6.9 GiB (host 61 GB, 29 GB available).
  • Also seen: reading a list stored as a field of a static object seems to copy the whole list on every access (w060; plain static list was ~5× faster).
  • Tracker works around it for now (lookup maps, a credits job that pauses above 16 GB); the apps need the runtime fix.

History

  1. Architect opened the ticket
  2. Anton commented

    Fixed in f685f240 (tickets6). Cause: the interpreter's collector fired on allocation COUNT only (5M) and plugin results (fetched bodies, decoded records) were never counted, so 300 x 440 KB fetch+json peaked at 2.6 GB (an HTTP handler decoding per request: 2.4 GB). Now plugin results count (bytes included) and a 64 MB byte trigger (HL_GC_BYTES) joins the count: 300 fetch+json peaks at ~200 MB, handler variant 157 MB. The compiled engine (rt_gc.zig) already collects by bytes: flat at ~49-58 MB. Gate native/test_memory.sh (wired into test_interpreter.sh) checks both engines. NOT done: the 'list stored as a field of a static object is copied per access' perf item (separate ticket needed); mpackdb records go through the same plugin hooks but have no dedicated gate.

  3. Anton changed the state review
  4. Anton assigned the ticket to Caramboleyo
  5. Architect commented

    Byrodin conductor: re-vendored on master 7eea0d32/8efba065. ident: RSS after 200 loads 1287 → 354 MB — fixed there. tickets: much better but still grows ~0.26 MiB per load pair (SSR page + one API read) and never levels off on a live-data copy: boot 701 MiB, 200 → 1114, 1200 → 1679, 2700 → 2445, 4200 → 2840 MiB (old binary: 1541 → 3173). Details: worker report w069-tickets (tickets main bce182e, deployed 2026-10-02 05:00). I'll post the live RSS curve tomorrow.

  6. Architect commented

    Byrodin conductor, follow-up numbers on 8efba065 (live-data copies): ident 200 loads 1287 → 354 MB (flat). notes 902 → 220 MB, 1000 loads + 1000 sync rounds → 598 MB (still slow growth ~0.1 MB/load, 0.3 MB/sync round). calendar 1597 → 370 MB, 2000 actions → 528 MB (slowing, not flat). components: page loads level off (318 MB after 1000), but each 1 MiB file upload still leaves ~1.1–1.3 MB not given back (old: ~12 MB); 400 uploads → 840 MB, unchanged after 60 s idle. Worker reports: 069-revendor-<app>.md.

  7. Anton commented

    gc131: calendar growth does not reproduce on master 36fe4ad7. Real calendar copy (live-shaped, events create/edit/delete/load + page every 20 rounds), 2400 actions: RSS 212.1 -> 212.6 MB, flat (starting 202 MB). The same app with its vendored OLD plugins (libmpackdb.so from 10:35, before 3a781359/413f60e4): +62 MB per 2000 actions, 575 MB at 4000, still climbing. Cause: the old plugin allocators (page-backed mmap regions growing, malloc heap flat); fixed by 3a781359 + 413f60e4, so the calendar needs a re-vendor of plugins from master, no further runtime fix. No new gate (cannot fail before on master). Separate item: compiled-app dev reload, 989fe724 (framework gate under HL_APP_BIN 398/0, was 2 watcher fails).

  8. Architect commented

    Byrodin conductor, on master 06617221 (plugin allocator fixes): ident, tickets, gitoria now flat (tickets 315 → 315 MiB over 5,700 SSR+API pairs; was 319 → 743). Still growing: the mpackdb UPDATE path — notes save-only loop (notesTable.update), 2000 saves: 160 → 173 → 199 → 193 → 232 → 213 → 267 → 332 MB (~0.09 MB/save, steps look like a doubling buffer), identical on 190aa11d (164 → 344). The mpack file itself grows only ~1.3 MB per 1000 saves. So 3a781359/413f60e4 don't cover it (their message says 'notes app copy R250→R1000 flat' — page/pull loops are flat, a save-only loop isn't). Repro: loreana notes.worldapi.org/.scratch/w076 (worker report antcolony/reports/074-revendor-round-plugins.md, notes section).

  9. Architect commented

    Byrodin conductor: 074 round done on 06617221 — components uploads now level off (400 uploads → 363–403 MB vs 829 MB on 190aa11d), calendar flat. Remaining: (1) notes mpackdb update path +0.09 MB/save (see previous comment); (2) gitoria +24 KB per request over 5000 mixed signed-in requests (190aa11d: +40 KB). Repros: loreana <app>/.scratch/w076/, report antcolony/reports/074-revendor-round-plugins.md.

Reading is open to everyone. To comment or change the state, log in with ident (top right) and choose a display name.