InternEvo

Environment Setup

• Installation
  • Environment Preparation
  • Installation
  • Environment Image

Quickstart Guide

• Quickstart Guide
  • Installation
  • Dataset Preparation (Pre-training)
  • Data Preparation (Fine-tuning)
  • Training Configuration
  • Start Training
  • Training Results
  • Long Text Generation

Model Setup

• Training Setup
  • Argument Parsing
  • Model Initialization
  • Optimizer Initialization
  • Dataloader Initialization
  • Trainer Initialization

Training API

• Training API

Parallel Training

• Parallel Training
  • Tensor Parallel
  • Pipeline Parallel
  • Data Parallel
  • ZeRO1.5

Mixed Precision

• Mixed Precision
  • NaiveAMPModel
  • TF32 Training

Mixture-of-Experts

• Mixture-of-Experts
  • Parameter Settings
  • Model Training

Model Checkpointing

• Model Checkpointing
  • CheckpointManager
  • Model loading and saving path format conventions.
  • Asynchronous upload.
  • Snapshot Checkpoint
  • Checkpoint automatic recovery
  • Manual control of checkpoint storage

Profiler

• Profiler
  • Torch Profiler
  • Memory Profiler

Monitor

• Monitor and Alert
  • Monitoring
  • Alerting
  • Light Monitoring

Example

• Training Example
  • 7B Demo
  • 20B Demo

Q&A

• Q&A

Indices and tables

• Index

• Module Index

• Search Page