建设银行广达行网站在线种子资源网
Debug
1. 快速运行一次所有的代码 (fast_dev_run)
训练了好长时间但是在训练or 验证的时候崩溃了 使用 fast_dev_run运行5个batch 的 training validation test and predication 查看是否存在错误：
train = Trainer(fast_dev_run=True) # True 时为5
train = Trainer(fast_dev_run=7) # 可以调节为任意int值
2.缩短epoch的长度 (limit_xxx_batch)
有时仅使用training or validation or … 是helpful的 例如在Imagenet等较大的数据集上，比等待complete epoch faster
train = Trainer(limit_train_batch=0.1, limit_val_batch=0.01) # 10% and 1%
train = Trainer(limit_train_batch=10, limit_val_batch=5) # 10 batches and 5 batches
3. 打印输入输出层尺寸(example_input_array)
class LitModel(LightningModule):def __init__(self, *args, **kwargs):self.example_input_array = torch.Tensor(32, 1, 28, 28)
summary table 将会输出包括 input and output 的 dimensions
| Name | Type | Params | Mode | In sizes | Out sizes
----------------------------------------------------------------------
0 | net | Sequential | 132 K | train | [10, 256] | [10, 512]
1 | net.0 | Linear | 131 K | train | [10, 256] | [10, 512]
2 | net.1 | BatchNorm1d | 1.0 K | train | [10, 512] | [10, 512]
发现 bottlenecks (profiler)
1. 查看时间(profiler)
trainer = Trainer(profiler="simple") # 测量训练循环中的所有方法# output for simple
FIT Profiler Report-------------------------------------------------------------------------------------------
| Action | Mean duration (s) | Total time (s) |
-------------------------------------------------------------------------------------------
| [LightningModule]BoringModel.prepare_data | 10.0001 | 20.00 |
| run_training_epoch | 6.1558 | 6.1558 |
| run_training_batch | 0.0022506 | 0.015754 |
| [LightningModule]BoringModel.optimizer_step | 0.0017477 | 0.012234 |
| [LightningModule]BoringModel.val_dataloader | 0.00024388 | 0.00024388 |
| on_train_batch_start | 0.00014637 | 0.0010246 |
| [LightningModule]BoringModel.teardown | 2.15e-06 | 2.15e-06 |
| [LightningModule]BoringModel.on_train_start | 1.644e-06 | 1.644e-06 |
| [LightningModule]BoringModel.on_train_end | 1.516e-06 | 1.516e-06 |
| [LightningModule]BoringModel.on_fit_end | 1.426e-06 | 1.426e-06 |
| [LightningModule]BoringModel.setup | 1.403e-06 | 1.403e-06 |
| [LightningModule]BoringModel.on_fit_start | 1.226e-06 | 1.226e-06 |
-------------------------------------------------------------------------------------------trainer = Trainer(profiler="advanced") # 测量每个function的时间
# output for advanced
Profiler ReportProfile stats for: get_train_batch4869394 function calls (4863767 primitive calls) in 18.893 seconds
Ordered by: cumulative time
List reduced from 76 to 10 due to restriction <10>
ncalls tottime percall cumtime percall filename:lineno(function)
3752/1876 0.011 0.000 18.887 0.010 {built-in method builtins.next}1876 0.008 0.000 18.877 0.010 dataloader.py:344(__next__)1876 0.074 0.000 18.869 0.010 dataloader.py:383(_next_data)1875 0.012 0.000 18.721 0.010 fetch.py:42(fetch)1875 0.084 0.000 18.290 0.010 fetch.py:44(<listcomp>)60000 1.759 0.000 18.206 0.000 mnist.py:80(__getitem__)60000 0.267 0.000 13.022 0.000 transforms.py:68(__call__)60000 0.182 0.000 7.020 0.000 transforms.py:93(__call__)60000 1.651 0.000 6.839 0.000 functional.py:42(to_tensor)60000 0.260 0.000 5.734 0.000 transforms.py:167(__call__)# 如果探查器报告变得太长，您可以将报告流式传输到文件
from lightning.pytorch.profilers import AdvancedProfilerprofiler = AdvancedProfiler(dirpath=".", filename="perf_logs")
trainer = Trainer(profiler=profiler)
highlevel usage:
https://lightning.ai/docs/pytorch/stable/tuning/profiler_intermediate.html
2. 查看accelerator的使用情况 (DeviceStatsMonitor)
检测瓶颈的另一个有用技术是确保您使用加速器 (GPU/TPU/HPU) 的全部容量。
from lightning.pytorch.callbacks import DeviceStatsMonitortrainer = Trainer(callbacks=[DeviceStatsMonitor()])
SOTA find
https://lightning.ai/docs/pytorch/stable/advanced/training_tricks.html