Dense (FFNN)

TODO

A dense layer, also known as a fully connected layer, is one of the fundamental building blocks of neural networks. In a dense layer, each neuron is connected to every neuron in the preceding layer.

This structure allows the layer to combine all input features through learned linear combinations, making it especially suitable for feature integration and final prediction stages.

Math Explaination

Let the input be a tensor $X \in \mathbb{R}^{B \times D_{in}}​$ where:

• $B$ is the batch size,

• DinD_{\text{in}}Din​ is the input feature dimension.

Take an input $I \in \mathbb{R}^{b \times d_{in}}$ where $b$ represents the batch size, $d_{in}$ the input dimension. A dense layer performs a linear projection of the input to a new dimension $d_{out}$ and applies an activation function, the output is therefore $O \in \mathbb{R}^{b \times d_{out}}$.

Formally, the