3j4>SSKrSSKrSSKJrJr SSKJrJr SSKJ r SSK J r SSK Jr /SQr"S S \5r"S S \5r \\\-\-r"S S\5r"SS\5r"SS\5rg)N)SizeTensor) functionalinit) Parameter) CrossMapLRN2d)Module)LocalResponseNormr LayerNorm GroupNormRMSNormc ^\rSrSr%Sr/SQr\\S'\\S'\\S'\\S'SS\S\S\S\SS 4 U4S jjjr S \ S\ 4S jr S r Sr U=r$)r a;Applies local response normalization over an input signal. The input signal is composed of several input planes, where channels occupy the second dimension. Applies normalization across channels. .. math:: b_{c} = a_{c}\left(k + \frac{\alpha}{n} \sum_{c'=\max(0, c-n/2)}^{\min(N-1,c+n/2)}a_{c'}^2\right)^{-\beta} Args: size: amount of neighbouring channels used for normalization alpha: multiplicative factor. Default: 0.0001 beta: exponent. Default: 0.75 k: additive factor. Default: 1 Shape: - Input: :math:`(N, C, *)` - Output: :math:`(N, C, *)` (same shape as input) Examples:: >>> lrn = nn.LocalResponseNorm(2) >>> signal_2d = torch.randn(32, 5, 24, 24) >>> signal_4d = torch.randn(16, 5, 7, 7, 7, 7) >>> output_2d = lrn(signal_2d) >>> output_4d = lrn(signal_4d) )sizealphabetakrrrrreturnNcR>[TU]5 XlX lX0lX@lgNsuper__init__rrrrselfrrrr __class__s X/home/wildlama/miniconda3/lib/python3.13/site-packages/torch/nn/modules/normalization.pyrLocalResponseNorm.__init__4$    inputc[R"XRURURUR 5$z Runs the forward pass. )Flocal_response_normrrrrrr"s rforwardLocalResponseNorm.forward=s+$$UIItzz499dffUUr!c:SR"S0URD6$0 Return the extra representation of the module. z){size}, alpha={alpha}, beta={beta}, k={k}format__dict__rs r extra_reprLocalResponseNorm.extra_reprC;AARDMMRRr!rrrr)-C6??g?)__name__ __module__ __qualname____firstlineno____doc__ __constants__int__annotations__floatrrr(r2__static_attributes__ __classcell__rs@rr r s~:3M I L K HNQ %49EJ VVVV SSr!r c ^\rSrSr%\\S'\\S'\\S'\\S'S S\S\S\S\SS4 U4SjjjrS \S\4S jr S\ 4S jr S r U=r $)r JrrrrrNcR>[TU]5 XlX lX0lX@lgrrrs rrCrossMapLRN2d.__init__Pr r!r"c[R"XRURURUR 5$r$)_cross_map_lrn2dapplyrrrrr's rr(CrossMapLRN2d.forwardYs+ %%eYY DIItvvVVr!c:SR"S0URD6$r+r.r1s rr2CrossMapLRN2d.extra_repr_r4r!r5)r6r7r)r8r9r:r;r>r?r@rrr(strr2rArBrCs@rr r Jsz I L K HNO %49EJ WVWW SCSSr!r c ^\rSrSr%Sr/SQr\\S4\S'\ \S'\ \S'SS\ S\ S\ S \ S S4 U4S jjjr SS jr S \S \4SjrS \4SjrSrU=r$)r ia Applies Layer Normalization over a mini-batch of inputs. This layer implements the operation as described in the paper `Layer Normalization `__ .. math:: y = \frac{x - \mathrm{E}[x]}{ \sqrt{\mathrm{Var}[x] + \epsilon}} * \gamma + \beta The mean and standard-deviation are calculated over the last `D` dimensions, where `D` is the dimension of :attr:`normalized_shape`. For example, if :attr:`normalized_shape` is ``(3, 5)`` (a 2-dimensional shape), the mean and standard-deviation are computed over the last 2 dimensions of the input (i.e. ``input.mean((-2, -1))``). :math:`\gamma` and :math:`\beta` are learnable affine transform parameters of :attr:`normalized_shape` if :attr:`elementwise_affine` is ``True``. The variance is calculated via the biased estimator, equivalent to `torch.var(input, correction=0)`. .. note:: Unlike Batch Normalization and Instance Normalization, which applies scalar scale and bias for each entire channel/plane with the :attr:`affine` option, Layer Normalization applies per-element scale and bias with :attr:`elementwise_affine`. This layer uses statistics computed from input data in both training and evaluation modes. Args: normalized_shape (int or list or torch.Size): input shape from an expected input of size .. math:: [* \times \text{normalized\_shape}[0] \times \text{normalized\_shape}[1] \times \ldots \times \text{normalized\_shape}[-1]] If a single integer is used, it is treated as a singleton list, and this module will normalize over the last dimension which is expected to be of that specific size. eps: a value added to the denominator for numerical stability. Default: 1e-5 elementwise_affine: a boolean value that when set to ``True``, this module has learnable per-element affine parameters initialized to ones (for weights) and zeros (for biases). Default: ``True`` bias: If set to ``False``, the layer will not learn an additive bias (only relevant if :attr:`elementwise_affine` is ``True``). Default: ``True`` Attributes: weight: the learnable weights of the module of shape :math:`\text{normalized\_shape}` when :attr:`elementwise_affine` is set to ``True``. The values are initialized to 1. bias: the learnable bias of the module of shape :math:`\text{normalized\_shape}` when :attr:`elementwise_affine` is set to ``True``. The values are initialized to 0. Shape: - Input: :math:`(N, *)` - Output: :math:`(N, *)` (same shape as input) Examples:: >>> # NLP Example >>> batch, sentence_length, embedding_dim = 20, 5, 10 >>> embedding = torch.randn(batch, sentence_length, embedding_dim) >>> layer_norm = nn.LayerNorm(embedding_dim) >>> # Activate module >>> layer_norm(embedding) >>> >>> # Image Example >>> N, C, H, W = 20, 5, 10, 10 >>> input = torch.randn(N, C, H, W) >>> # Normalize over the last three dimensions (i.e. the channel and spatial dimensions) >>> # as shown in the image below >>> layer_norm = nn.LayerNorm([C, H, W]) >>> output = layer_norm(input) .. image:: ../_static/img/nn/layer_norm.jpg :scale: 50 % normalized_shapeepselementwise_affine.rRrSrTNbiasrc">XVS.n[TU]5 [U[R5(aU4n[ U5UlX lX0lUR(ay[[R"UR 40UD65Ul U(a0[[R"UR 40UD65Ul O7URSS5 O$URSS5 URSS5 UR5 g)NdevicedtyperUweight)rr isinstancenumbersIntegraltuplerRrSrTrtorchemptyrZrUregister_parameterreset_parameters) rrRrSrTrUrXrYfactory_kwargsrs rrLayerNorm.__init__s%+;  &(8(8 9 9 02  %&6 7"4  " "# D11D^DDK%KK 5 5HH ''5  # #Hd 3  # #FD 1 r!cUR(aO[R"UR5 URb![R "UR5 gggr)rTrones_rZrUzeros_r1s rrbLayerNorm.reset_parameterss?  " " JJt{{ #yy$ DII&% #r!r"c[R"XRURURUR 5$r)r% layer_normrRrZrUrSr's rr(LayerNorm.forwards.|| (($++tyy$((  r!cZSR"S0URDSURSL0D6$)NzW{normalized_shape}, eps={eps}, elementwise_affine={elementwise_affine}, bias={use_bias}use_biasr-r/r0rUr1s rr2LayerNorm.extra_repr< $f % V'+}} V?CyyPT?T V r!)rUrTrSrRrZ)h㈵>TTNNrN)r8r9r:r;r<r=r^r>r?r@bool_shape_trrbrr(rNr2rArBrCs@rr r isKZFMCHo% J #' "  !       B'  V   C  r!r c^\rSrSr%Sr/SQr\\S'\\S'\\S'\ \S'SSS .S\S\S\S\ S \ S S 4 U4S jjjjr SSjr S\ S \ 4Sjr S \4SjrSrU=r$)r aApplies Group Normalization over a mini-batch of inputs. This layer implements the operation as described in the paper `Group Normalization `__ .. math:: y = \frac{x - \mathrm{E}[x]}{ \sqrt{\mathrm{Var}[x] + \epsilon}} * \gamma + \beta The input channels are separated into :attr:`num_groups` groups, each containing ``num_channels / num_groups`` channels. :attr:`num_channels` must be divisible by :attr:`num_groups`. The mean and standard-deviation are calculated separately over each group. :math:`\gamma` and :math:`\beta` are learnable per-channel affine transform parameter vectors of size :attr:`num_channels` if :attr:`affine` is ``True``. The variance is calculated via the biased estimator, equivalent to `torch.var(input, correction=0)`. This layer uses statistics computed from input data in both training and evaluation modes. Args: num_groups (int): number of groups to separate the channels into num_channels (int): number of channels expected in input eps: a value added to the denominator for numerical stability. Default: 1e-5 affine: a boolean value that when set to ``True``, this module has learnable per-channel affine parameters initialized to ones (for weights) and zeros (for biases). Default: ``True`` bias: If set to ``False``, the layer will not learn an additive bias (only relevant if :attr:`affine` is ``True``). Default: ``True`` Shape: - Input: :math:`(N, C, *)` where :math:`C=\text{num\_channels}` - Output: :math:`(N, C, *)` (same shape as input) Examples:: >>> input = torch.randn(20, 6, 10, 10) >>> # Separate 6 channels into 3 groups >>> m = nn.GroupNorm(3, 6) >>> # Separate 6 channels into 6 groups (equivalent with InstanceNorm) >>> m = nn.GroupNorm(6, 6) >>> # Put all 6 channels into a single group (equivalent with LayerNorm) >>> m = nn.GroupNorm(1, 6) >>> # Activating the module >>> output = m(input) ) num_groups num_channelsrSaffinerwrxrSryTN)rUrUrc>XVS.n[T U]5 X!-S:wa[SUSUS35eXlX lX0lX@lUR (ae[[R"U40UD65Ul U(a&[[R"U40UD65Ul O7URSS5 O$URSS5 URSS5 UR5 g)NrWrznum_channels (z#) must be divisible by num_groups ()rUrZ)rr ValueErrorrwrxrSryrr_r`rZrUrarb) rrwrxrSryrXrYrUrcrs rrGroupNorm.__init__%s%+;   $ ) .QR\Q]]^_ %( ;;#EKK $O$OPDK%ekk,&Q.&QR ''5  # #Hd 3  # #FD 1 r!cUR(aO[R"UR5 URb![R "UR5 gggr)ryrrfrZrUrgr1s rrbGroupNorm.reset_parametersGs= ;; JJt{{ #yy$ DII&% r!r"c[R"XRURURUR 5$r)r% group_normrwrZrUrSr's rr(GroupNorm.forwardMs'||E??DKKDHHUUr!cZSR"S0URDSURSL0D6$)NzI{num_groups}, {num_channels}, eps={eps}, affine={affine}, bias={use_bias}rmr-rnr1s rr2GroupNorm.extra_reprPrpr!)ryrUrSrxrwrZ)rqTNNrr)r8r9r:r;r<r=r>r?r@rsrrbrr(rNr2rArBrCs@rr r s-^DMO J L                   D' VVVV C  r!r c ^\rSrSr%Sr/SQr\\S4\S'\ S-\S'\ \S'SS\ S\ S-S\ S S4U4S jjjr SS jr S \RS \R4S jrS \4SjrSrU=r$)riWaApplies Root Mean Square Layer Normalization over a mini-batch of inputs. This layer implements the operation as described in the paper `Root Mean Square Layer Normalization `__ .. math:: y_i = \frac{x_i}{\mathrm{RMS}(x)} * \gamma_i, \quad \text{where} \quad \text{RMS}(x) = \sqrt{\epsilon + \frac{1}{n} \sum_{i=1}^{n} x_i^2} The RMS is taken over the last ``D`` dimensions, where ``D`` is the dimension of :attr:`normalized_shape`. For example, if :attr:`normalized_shape` is ``(3, 5)`` (a 2-dimensional shape), the RMS is computed over the last 2 dimensions of the input. Args: normalized_shape (int or list or torch.Size): input shape from an expected input of size .. math:: [* \times \text{normalized\_shape}[0] \times \text{normalized\_shape}[1] \times \ldots \times \text{normalized\_shape}[-1]] If a single integer is used, it is treated as a singleton list, and this module will normalize over the last dimension which is expected to be of that specific size. eps (float, optional): a value added to the denominator for numerical stability. If not specified, uses the machine epsilon of the computation (opmath) type: fp16/bf16 and fp32 inputs use ``torch.finfo(torch.float32).eps``, while fp64 inputs use ``torch.finfo(torch.float64).eps``. Default: ``None`` elementwise_affine: a boolean value that when set to ``True``, this module has learnable per-element affine parameters initialized to ones (for weights). Default: ``True``. Shape: - Input: :math:`(N, *)` - Output: :math:`(N, *)` (same shape as input) Examples:: >>> rms_norm = nn.RMSNorm([2, 3]) >>> input = torch.randn(2, 2, 3) >>> rms_norm(input) rQ.rRNrSrTrcl>XES.n[TU]5 [U[R5(aU4n[ U5UlX lX0lUR(a0[[R"UR 40UD65Ul OURSS5 UR5 g)NrWrZ)rrr[r\r]r^rRrSrTrr_r`rZrarb)rrRrSrTrXrYrcrs rrRMSNorm.__init__s%+;  &(8(8 9 9 02  %&6 7"4  " "# D11D^DDK  # #Hd 3 r!chUR(a![R"UR5 gg)zC Resets parameters based on their initialization used in __init__. N)rTrrfrZr1s rrbRMSNorm.reset_parameterss"  " " JJt{{ # #r!xcn[R"XRURUR5$r$)r%rms_normrRrZrS)rrs rr(RMSNorm.forwards%zz!22DKKJJr!c:SR"S0URD6$)r,zF{normalized_shape}, eps={eps}, elementwise_affine={elementwise_affine}r-r.r1s rr2RMSNorm.extra_reprs)  66r?r@rsrtrrbr_rr(rNr2rArBrCs@rrrWs)VFMCHo%  !#'  " T\ !     0$KK%,,K  C  r!r)r\r_rrtorch.nnrr%rtorch.nn.parameterr _functionsr rImoduler __all__r r>listrtr r rr-r!rrs *(9 V7S7StSFS8 c?T !C C Le e P] f] r!