kubeflow / trainer
58%
master: 58%

LAST BUILD BRANCH: megatron
DEFAULT BRANCH: master
Repo Added
Token 3qIdUH6ns6RNy0sBPPQ6ybJp7VqYkScU8 regen
Build 3010 Last
Files 40
Badge
Embed ▾
README BADGES
x

If you need to use a raster PNG badge, change the '.svg' to '.png' in the link

Markdown

Textile

RDoc

HTML

Rst

LAST BUILD ON BRANCH megatron
branch: megatron
CHANGE BRANCH
x
Sync Branches

coverage: 58.057%. Remained the same
Relevant lines Covered
3500 RELEVANT LINES 2032 COVERED LINES
0.67 HITS PER LINE
Source Files on megatron

Recent builds

Builds Branch Commit Type Ran Committer Via Coverage
23981685932 megatron fix: remove dist_checkpointing from Megatron notebook Megatron-Core dist_checkpointing uses multiprocessing.spawn internally to create a Manager queue for async writes. The Kubeflow SDK generates training scripts without an if __name__ == '__main... Pull #3201 XploY04 github
58.06
23981021700 megatron fix: remove dist_checkpointing from Megatron notebook Megatron-Core dist_checkpointing uses multiprocessing.spawn internally to create a Manager queue for async writes. The Kubeflow SDK generates training scripts without an if __name__ == '__main... Pull #3201 XploY04 github
58.14
23979976051 megatron fix: mount /dev/shm as emptyDir to fix NCCL shared memory exhaustion NCCL proxy service allocates ~33MB per communicator in /dev/shm. The default Kubernetes /dev/shm is 64MB (Docker default), which is insufficient for workloads that create multip... Pull #3201 XploY04 github
58.14
23972072192 megatron debug: add NCCL_DEBUG=INFO to diagnose /dev/shm failure on node-1 Signed-off-by: XploY04 <2004agarwalyash@gmail.com> Pull #3201 XploY04 github
58.06
23529197530 megatron fix: disable NCCL shared memory to avoid /dev/shm limit in Kubernetes Kubernetes pods get 64MB /dev/shm by default. NCCL needs more for its communication buffers, causing checkpoint save to fail with "Error while creating shared memory segment". ... Pull #3201 XploY04 github
58.06
23523506891 megatron fix: use runtime image with apt-get for build tools instead of devel image The devel image (3.5GB) is too slow to pull, causing PAPERMILL_TIMEOUT. Switch back to the runtime image and install only make and g++ via apt-get (~50MB). This provides e... Pull #3201 XploY04 github
58.06
23497195914 megatron fix: increase wait_for_job_status timeout to 1800s The megatron training job takes longer than 900s due to pip install, C++ compilation, and multi-node training setup on time-sliced GPUs. Increase timeout to 1800s to match PAPERMILL_TIMEOUT. Sig... Pull #3201 XploY04 github
58.06
23485235636 megatron fix: use correct null tokenizer library name for pip megatron-core The pip release of megatron-core uses "null" as the tokenizer library name, while the GitHub main branch uses "null-text". Change to "null" to match the pip version. Signed-off-b... Pull #3201 XploY04 github
58.14
23481258188 megatron fix: download Megatron C++ source files for compile_helpers() The megatron-core pip package excludes the Makefile and helpers.cpp needed by compile_helpers(). Download these two files from GitHub at runtime before compilation. Also add pybind11 a... Pull #3201 XploY04 github
58.06
23479733760 megatron fix: install megatron-core from zip archive instead of git The devel image doesn't have git installed, so pip install from git+https fails. Use the GitHub zip archive URL instead, which pip can download and extract directly without needing git. ... Pull #3201 XploY04 github
58.06
See All Builds (2691)

Badge your Repo: trainer

We detected this repo isn’t badged! Grab the embed code to the right, add it to your repo to show off your code coverage, and when the badge is live hit the refresh button to remove this message.
Embed ▾
README BADGES
x

If you need to use a raster PNG badge, change the '.svg' to '.png' in the link

Markdown

Textile

RDoc

HTML

Rst

Refresh