Batch Size

Batch size is a hyperparameter that defines the number of samples that will be propagated through the network at once. Larger batches allow for increased parallelization but require more memory. The choice of batch size affects both the training speed and the model's ability to generalize with very large batches potentially leading to poor generalization and very small batches resulting in noisy gradient estimates.

Training deep neural networks optimizing GPU memory usage distributed training systems