rendezqueue / rendezllama / 21335653326

86%

trunk: 87%

push

github

Infer default thread count to match physical cores

Introduces `rendezllama::infer_thread_count` to heuristically determine a suitable thread count for inference, aiming to use physical cores rather than logical threads on x86 architectures (by halving the count if > 4). This avoids oversubscription and improves performance stability.

- Updates `ChatOptions` to initialize `thread_count` using this helper.
- Updates `Inference::commit_to_context` to use `std::thread::hardware_concurrency` (all logical cores) for `batch_thread_count` default, distinct from the physical core preference for generation.
- Updates `assistant_cli` to rely on the `ChatOptions` default instead of manually setting `hardware_concurrency`.
- Shared `infer_thread_count` logic resides in `src/chat/opt.cc` and is exposed via `src/chat/opt.hh`.

5 of 6 new or added lines in 1 file covered. (83.33%)

2128 of 2399 relevant lines covered (88.7%)

523.93 hits per line