$ ./guess_the_languageAI & ML
AI & ML · hard · #29

What do residual (skip) connections mainly help with in very deep networks?

  1. Gradients flowing to early layers
  2. Shrinking the tokenizer's vocabulary
  3. Labeling unlabeled samples
  4. Lowering the input resolution
Adding the input back to a block's output gives gradients a direct path, easing vanishing gradients in deep nets.
Play AI & ML ▸