3jؠXSSKJr SSKrSSKJr SSKJrJr SSKJ r J r J r SSK J r SSKJr SS KJr /S Qr"S S \5r"S S\5r"SS\\5r"SS\5r"SS\\5r"SS\5r"SS\\5r"SS\5r"SS\\5r"SS\5r g))AnyN)Tensor) functionalinit) ParameterUninitializedBufferUninitializedParameter) SyncBatchNorm)LazyModuleMixin)Module) BatchNorm1dLazyBatchNorm1d BatchNorm2dLazyBatchNorm2d BatchNorm3dLazyBatchNorm3dr c^\rSrSr%SrSr/SQr\\S'\ \S'\ S-\S'\ \S '\ \S 'SS S .S\S\ S\ S-S \ S \ S \ SS4U4Sjjjjr SSjr SSjr SrSrSU4SjjrSrU=r$) _NormBasez,Common base of _InstanceNorm and _BatchNorm.)track_running_statsmomentumeps num_featuresaffinerrNrrrTbiasrreturnc >XgS.n [T U]5 XlX lX0lX@lXPlUR (ae[[R"U40U D65Ul U(a&[[R"U40U D65Ul O7URSS5 O$URSS5 URSS5 UR (aURS[R"U40U D65 URS[R"U40U D65 U U URS[R "SS[R"0U R%5V V s0sHupU S:wdM X_M sn n D65 U O6URSS5 URSS5 URSS5 UR'5 gs sn n f) Ndevicedtyperweight running_mean running_varnum_batches_trackedr#r)super__init__rrrrrrtorchemptyr$rregister_parameterregister_bufferzerosonestensorlongitemsreset_parameters) selfrrrrrr"r#rfactory_kwargskv __class__s T/home/wildlama/miniconda3/lib/python3.13/site-packages/torch/nn/modules/batchnorm.pyr*_NormBase.__init__&s%+; (  #6 ;;#EKK $O$OPDK%ekk,&Q.&QR ''5  # #Hd 3  # #FD 1  # #   L KN K   uzz,I.I     % **)7(<(<(>O(>!w,tqt(>O      6   5  !6 = Ps 1 GGcUR(aPURR5 URR S5 UR R5 gg)Nr )rr%zero_r&fill_r'r5s r:reset_running_stats_NormBase.reset_running_stats[sJ  # #    # # %    " "1 %  $ $ * * , $cUR5 UR(aO[R"UR5 UR b![R "UR 5 gggN)r@rrones_r$rzeros_r?s r:r4_NormBase.reset_parameterscsI   " ;; JJt{{ #yy$ DII&% rBc[erD)NotImplementedErrorr5inputs r:_check_input_dim_NormBase._check_input_dimjs!!rBcZSR"S0URDSURSL0D6$)Nz{{num_features}, eps={eps}, momentum={momentum}, affine={affine}, bias={use_bias}, track_running_stats={track_running_stats}use_bias)format__dict__rr?s r: extra_repr_NormBase.extra_reprms: IIO P -- *.))4*?  rBc t>URSS5nUbUS:aUR(avUS-n X;alURb:URR[R"S5:wa URO"[R "S[R S9X'[T U]!UUUUUUU5 g)Nversionrr'metar)r#) getrr'r"r+r1r2r)_load_from_state_dict) r5 state_dictprefixlocal_metadatastrict missing_keysunexpected_keys error_msgsrVnum_batches_tracked_keyr9s r:rY_NormBase._load_from_state_dictus!$$Y5 Ow{0H0H'-/D&D #&8//;00775<<;OO,,auzz: 3 %        rB)rrrrrrr$h㈵>皙?TTNNrN)__name__ __module__ __qualname____firstlineno____doc___version __constants__int__annotations__floatboolr*r@r4rLrSrY__static_attributes__ __classcell__r9s@r:rrs6HXM Jdl L!$$(3 3 3 3 $, 3  3 " 3 3  3 3 j-'"       rBrcx^\rSrSrSSS.S\S\S\S-S\S \S \S S4U4S jjjjrS \S \4Sjr Sr U=r $) _BatchNormTNrrrrrrrrc@>XgS.n [T U]"UUUUU40U DSU0D6 gNr!r)r)r*) r5rrrrrr"r#rr6r9s r:r*_BatchNorm.__init__s=%+;           rBrKc URU5 URcSnO URnUR(akUR(aZURbMURR S5 URcS[ UR5- nO URnUR(aSnO#URSL=(a URSLn[R"UUR(aUR(a UROSUR(aUR(a UROSURURUUUR5$)Nr ?T)rLrtrainingrr'add_rpr%r&F batch_normr$rr)r5rKexponential_average_factor bn_trainings r:forward_BatchNorm.forwards' e$ == ), &)- & ==T55''3((--a0==(14uT=U=U7V1V.15.  ==K,,4T4;K;Kt;SK || }}(@(@!!$(MMT5M5MD  SW KK II  & HH  rBrPrc) rgrhrirjrnrprqr*rrrrrsrts@r:rvrvs!$$(    $,    "      .0 V0 0 0 rBrvcv^\rSrSr%\\S'\\S'S SS.S U4SjjjjrS U4SjjrS SjrS r U=r $) _LazyNormBaser$rTrc >XVS.n[T U]"SUUSS40UDSS0D6 X0lX@lUR(a'[ S0UD6UlU(a[ S0UD6UlUR(ax[S0UD6Ul[S0UD6Ul [R"SS[R0UR5V V s0sHupU S:wdM X_M sn n D6Ulggs sn n f)Nr!rFrr#rPr()r)r*rrr r$rrr%r&r+r1r2r3r') r5rrrrr"r#rr6r7r8r9s r:r*_LazyNormBase.__init__s%+;          #6 ;;0B>BDK2D^D  # # 3 En ED 2D^DD ',||(jj(%3$8$8$:K$:DAa7l414$:K (D $ $Ls C-C-cp>UR5(d URS:wa[TU] 5 ggg)Nr)has_uninitialized_paramsrr)r4)r5r9s r:r4_LazyNormBase.reset_parameterss3,,..43D3D3I G $ &4J.rBcUR5(Ga@URSUlUR(a[ UR [ 5(d [S5eUR RUR45 URbP[ UR[ 5(d [S5eURRUR45 UR(aLURRUR45 URRUR45 UR5 gg)Nr z-self.weight must be an UninitializedParameterz+self.bias must be an UninitializedParameter)rshaperr isinstancer$r AssertionError materializerrr%r&r4rJs r:initialize_parameters#_LazyNormBase.initialize_parameterss  ( ( * * % AD {{!$++/EFF(G ''):):(<=99(%dii1GHH,III))4+<+<*>?''!!--&&(  ,,&&(  ! ! #+ +rB)rrr'rr%r&rr$rcrf) rgrhrirjr ror*r4rrrrsrts@r:rrsQ ""    ** **X' $$rBrc"\rSrSrSrSSjrSrg)ri2a Applies Batch Normalization over a 2D or 3D input. Method described in the paper `Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift `__ . .. math:: y = \frac{x - \mathrm{E}[x]}{\sqrt{\mathrm{Var}[x] + \epsilon}} * \gamma + \beta The mean and standard-deviation are calculated per-dimension over the mini-batches and :math:`\gamma` and :math:`\beta` are learnable parameter vectors of size `C` (where `C` is the number of features or channels of the input). By default, the elements of :math:`\gamma` are set to 1 and the elements of :math:`\beta` are set to 0. At train time in the forward pass, the variance is calculated via the biased estimator, equivalent to ``torch.var(input, correction=0)``. However, the value stored in the moving average of the variance is calculated via the unbiased estimator, equivalent to ``torch.var(input, correction=1)``. Also by default, during training this layer keeps running estimates of its computed mean and variance, which are then used for normalization during evaluation. The running estimates are kept with a default :attr:`momentum` of 0.1. If :attr:`track_running_stats` is set to ``False``, this layer then does not keep running estimates, and batch statistics are instead used during evaluation time as well. .. note:: This :attr:`momentum` argument is different from one used in optimizer classes and the conventional notion of momentum. Mathematically, the update rule for running statistics here is :math:`\hat{x}_\text{new} = (1 - \text{momentum}) \times \hat{x} + \text{momentum} \times x_t`, where :math:`\hat{x}` is the estimated statistic and :math:`x_t` is the new observed value. Because the Batch Normalization is done over the `C` dimension, computing statistics on `(N, L)` slices, it's common terminology to call this Temporal Batch Normalization. Args: num_features: number of features or channels :math:`C` of the input eps: a value added to the denominator for numerical stability. Default: 1e-5 momentum: the value used for the running_mean and running_var computation. Can be set to ``None`` for cumulative moving average (i.e. simple average). Default: 0.1 affine: a boolean value that when set to ``True``, this module has learnable affine parameters. Default: ``True`` track_running_stats: a boolean value that when set to ``True``, this module tracks the running mean and variance, and when set to ``False``, this module does not track such statistics, and initializes statistics buffers :attr:`running_mean` and :attr:`running_var` as ``None``. When these buffers are ``None``, this module always uses batch statistics. in both training and eval modes. Default: ``True`` bias: If set to ``False``, the layer will not learn an additive bias (only relevant if :attr:`affine` is ``True``). Default: ``True`` Shape: - Input: :math:`(N, C)` or :math:`(N, C, L)`, where :math:`N` is the batch size, :math:`C` is the number of features or channels, and :math:`L` is the sequence length - Output: :math:`(N, C)` or :math:`(N, C, L)` (same shape as input) Examples:: >>> # With Learnable Parameters >>> m = nn.BatchNorm1d(100) >>> # Without Learnable Parameters >>> m = nn.BatchNorm1d(100, affine=False) >>> input = torch.randn(20, 100) >>> output = m(input) NcUR5S:wa2UR5S:wa[SUR5S35eggNrzexpected 2D or 3D input (got D input)dim ValueErrorrJs r:rLBatchNorm1d._check_input_dim{@ 99;!  q 0`__ . .. math:: y = \frac{x - \mathrm{E}[x]}{ \sqrt{\mathrm{Var}[x] + \epsilon}} * \gamma + \beta The mean and standard-deviation are calculated per-dimension over the mini-batches and :math:`\gamma` and :math:`\beta` are learnable parameter vectors of size `C` (where `C` is the input size). By default, the elements of :math:`\gamma` are set to 1 and the elements of :math:`\beta` are set to 0. At train time in the forward pass, the standard-deviation is calculated via the biased estimator, equivalent to ``torch.var(input, correction=0)``. However, the value stored in the moving average of the standard-deviation is calculated via the unbiased estimator, equivalent to ``torch.var(input, correction=1)``. Also by default, during training this layer keeps running estimates of its computed mean and variance, which are then used for normalization during evaluation. The running estimates are kept with a default :attr:`momentum` of 0.1. If :attr:`track_running_stats` is set to ``False``, this layer then does not keep running estimates, and batch statistics are instead used during evaluation time as well. .. note:: This :attr:`momentum` argument is different from one used in optimizer classes and the conventional notion of momentum. Mathematically, the update rule for running statistics here is :math:`\hat{x}_\text{new} = (1 - \text{momentum}) \times \hat{x} + \text{momentum} \times x_t`, where :math:`\hat{x}` is the estimated statistic and :math:`x_t` is the new observed value. Because the Batch Normalization is done over the `C` dimension, computing statistics on `(N, H, W)` slices, it's common terminology to call this Spatial Batch Normalization. Args: num_features: :math:`C` from an expected input of size :math:`(N, C, H, W)` eps: a value added to the denominator for numerical stability. Default: 1e-5 momentum: the value used for the running_mean and running_var computation. Can be set to ``None`` for cumulative moving average (i.e. simple average). Default: 0.1 affine: a boolean value that when set to ``True``, this module has learnable affine parameters. Default: ``True`` track_running_stats: a boolean value that when set to ``True``, this module tracks the running mean and variance, and when set to ``False``, this module does not track such statistics, and initializes statistics buffers :attr:`running_mean` and :attr:`running_var` as ``None``. When these buffers are ``None``, this module always uses batch statistics. in both training and eval modes. Default: ``True`` bias: If set to ``False``, the layer will not learn an additive bias (only relevant if :attr:`affine` is ``True``). Default: ``True`` Shape: - Input: :math:`(N, C, H, W)` - Output: :math:`(N, C, H, W)` (same shape as input) Examples:: >>> # With Learnable Parameters >>> m = nn.BatchNorm2d(100) >>> # Without Learnable Parameters >>> m = nn.BatchNorm2d(100, affine=False) >>> input = torch.randn(20, 100, 35, 45) >>> output = m(input) NcfUR5S:wa[SUR5S35egNzexpected 4D input (got rrrJs r:rLBatchNorm2d._check_input_dim0 99;! 6uyy{m8LM M rBrPrfrrPrBr:rrGRNrBrc&\rSrSrSr\rSSjrSrg)riaA :class:`torch.nn.BatchNorm2d` module with lazy initialization. Lazy initialization is done for the ``num_features`` argument of the :class:`BatchNorm2d` that is inferred from the ``input.size(1)``. The attributes that will be lazily initialized are `weight`, `bias`, `running_mean` and `running_var`. Check the :class:`torch.nn.modules.lazy.LazyModuleMixin` for further documentation on lazy modules and their limitations. Args: eps: a value added to the denominator for numerical stability. Default: 1e-5 momentum: the value used for the running_mean and running_var computation. Can be set to ``None`` for cumulative moving average (i.e. simple average). Default: 0.1 affine: a boolean value that when set to ``True``, this module has learnable affine parameters. Default: ``True`` track_running_stats: a boolean value that when set to ``True``, this module tracks the running mean and variance, and when set to ``False``, this module does not track such statistics, and initializes statistics buffers :attr:`running_mean` and :attr:`running_var` as ``None``. When these buffers are ``None``, this module always uses batch statistics. in both training and eval modes. Default: ``True`` bias: If set to ``False``, the layer will not learn an additive bias (only relevant if :attr:`affine` is ``True``). Default: ``True`` NcfUR5S:wa[SUR5S35egrrrJs r:rL LazyBatchNorm2d._check_input_dimrrBrPrf) rgrhrirjrkrrrLrrrPrBr:rr8 MNrBrc"\rSrSrSrSSjrSrg)ria| Applies Batch Normalization over a 5D input. 5D is a mini-batch of 3D inputs with additional channel dimension as described in the paper `Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift `__ . .. math:: y = \frac{x - \mathrm{E}[x]}{ \sqrt{\mathrm{Var}[x] + \epsilon}} * \gamma + \beta The mean and standard-deviation are calculated per-dimension over the mini-batches and :math:`\gamma` and :math:`\beta` are learnable parameter vectors of size `C` (where `C` is the input size). By default, the elements of :math:`\gamma` are set to 1 and the elements of :math:`\beta` are set to 0. At train time in the forward pass, the standard-deviation is calculated via the biased estimator, equivalent to ``torch.var(input, correction=0)``. However, the value stored in the moving average of the standard-deviation is calculated via the unbiased estimator, equivalent to ``torch.var(input, correction=1)``. Also by default, during training this layer keeps running estimates of its computed mean and variance, which are then used for normalization during evaluation. The running estimates are kept with a default :attr:`momentum` of 0.1. If :attr:`track_running_stats` is set to ``False``, this layer then does not keep running estimates, and batch statistics are instead used during evaluation time as well. .. note:: This :attr:`momentum` argument is different from one used in optimizer classes and the conventional notion of momentum. Mathematically, the update rule for running statistics here is :math:`\hat{x}_\text{new} = (1 - \text{momentum}) \times \hat{x} + \text{momentum} \times x_t`, where :math:`\hat{x}` is the estimated statistic and :math:`x_t` is the new observed value. Because the Batch Normalization is done over the `C` dimension, computing statistics on `(N, D, H, W)` slices, it's common terminology to call this Volumetric Batch Normalization or Spatio-temporal Batch Normalization. Args: num_features: :math:`C` from an expected input of size :math:`(N, C, D, H, W)` eps: a value added to the denominator for numerical stability. Default: 1e-5 momentum: the value used for the running_mean and running_var computation. Can be set to ``None`` for cumulative moving average (i.e. simple average). Default: 0.1 affine: a boolean value that when set to ``True``, this module has learnable affine parameters. Default: ``True`` track_running_stats: a boolean value that when set to ``True``, this module tracks the running mean and variance, and when set to ``False``, this module does not track such statistics, and initializes statistics buffers :attr:`running_mean` and :attr:`running_var` as ``None``. When these buffers are ``None``, this module always uses batch statistics. in both training and eval modes. Default: ``True`` bias: If set to ``False``, the layer will not learn an additive bias (only relevant if :attr:`affine` is ``True``). Default: ``True`` Shape: - Input: :math:`(N, C, D, H, W)` - Output: :math:`(N, C, D, H, W)` (same shape as input) Examples:: >>> # With Learnable Parameters >>> m = nn.BatchNorm3d(100) >>> # Without Learnable Parameters >>> m = nn.BatchNorm3d(100, affine=False) >>> input = torch.randn(20, 100, 35, 45, 10) >>> output = m(input) NcfUR5S:wa[SUR5S35egNzexpected 5D input (got rrrJs r:rLBatchNorm3d._check_input_dimarrBrPrfrrPrBr:rrrrBrc&\rSrSrSr\rSSjrSrg)rifaA :class:`torch.nn.BatchNorm3d` module with lazy initialization. Lazy initialization is done for the ``num_features`` argument of the :class:`BatchNorm3d` that is inferred from the ``input.size(1)``. The attributes that will be lazily initialized are `weight`, `bias`, `running_mean` and `running_var`. Check the :class:`torch.nn.modules.lazy.LazyModuleMixin` for further documentation on lazy modules and their limitations. Args: eps: a value added to the denominator for numerical stability. Default: 1e-5 momentum: the value used for the running_mean and running_var computation. Can be set to ``None`` for cumulative moving average (i.e. simple average). Default: 0.1 affine: a boolean value that when set to ``True``, this module has learnable affine parameters. Default: ``True`` track_running_stats: a boolean value that when set to ``True``, this module tracks the running mean and variance, and when set to ``False``, this module does not track such statistics, and initializes statistics buffers :attr:`running_mean` and :attr:`running_var` as ``None``. When these buffers are ``None``, this module always uses batch statistics. in both training and eval modes. Default: ``True`` bias: If set to ``False``, the layer will not learn an additive bias (only relevant if :attr:`affine` is ``True``). Default: ``True`` NcfUR5S:wa[SUR5S35egrrrJs r:rL LazyBatchNorm3d._check_input_dimrrBrPrf) rgrhrirjrkrrrLrrrPrBr:rrfrrBrc^\rSrSrSrSSS.S\S\S\S-S \S \S \S-S \S S4U4Sjjjjr SSjr SSjr S\ S \ 4Sjr \SSj5rSrU=r$)r iaNApplies Batch Normalization over a N-Dimensional input. The N-D input is a mini-batch of [N-2]D inputs with additional channel dimension) as described in the paper `Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift `__ . .. math:: y = \frac{x - \mathrm{E}[x]}{ \sqrt{\mathrm{Var}[x] + \epsilon}} * \gamma + \beta The mean and standard-deviation are calculated per-dimension over all mini-batches of the same process groups. :math:`\gamma` and :math:`\beta` are learnable parameter vectors of size `C` (where `C` is the input size). By default, the elements of :math:`\gamma` are sampled from :math:`\mathcal{U}(0, 1)` and the elements of :math:`\beta` are set to 0. The standard-deviation is calculated via the biased estimator, equivalent to `torch.var(input, correction=0)`. Also by default, during training this layer keeps running estimates of its computed mean and variance, which are then used for normalization during evaluation. The running estimates are kept with a default :attr:`momentum` of 0.1. If :attr:`track_running_stats` is set to ``False``, this layer then does not keep running estimates, and batch statistics are instead used during evaluation time as well. .. note:: This :attr:`momentum` argument is different from one used in optimizer classes and the conventional notion of momentum. Mathematically, the update rule for running statistics here is :math:`\hat{x}_\text{new} = (1 - \text{momentum}) \times \hat{x} + \text{momentum} \times x_t`, where :math:`\hat{x}` is the estimated statistic and :math:`x_t` is the new observed value. Because the Batch Normalization is done for each channel in the ``C`` dimension, computing statistics on ``(N, +)`` slices, it's common terminology to call this Volumetric Batch Normalization or Spatio-temporal Batch Normalization. Currently :class:`SyncBatchNorm` only supports :class:`~torch.nn.DistributedDataParallel` (DDP) with single GPU per process. Use :meth:`torch.nn.SyncBatchNorm.convert_sync_batchnorm()` to convert :attr:`BatchNorm*D` layer to :class:`SyncBatchNorm` before wrapping Network with DDP. Args: num_features: :math:`C` from an expected input of size :math:`(N, C, +)` eps: a value added to the denominator for numerical stability. Default: ``1e-5`` momentum: the value used for the running_mean and running_var computation. Can be set to ``None`` for cumulative moving average (i.e. simple average). Default: 0.1 affine: a boolean value that when set to ``True``, this module has learnable affine parameters. Default: ``True`` track_running_stats: a boolean value that when set to ``True``, this module tracks the running mean and variance, and when set to ``False``, this module does not track such statistics, and initializes statistics buffers :attr:`running_mean` and :attr:`running_var` as ``None``. When these buffers are ``None``, this module always uses batch statistics. in both training and eval modes. Default: ``True`` process_group: synchronization of stats happen within each process group individually. Default behavior is synchronization across the whole world bias: If set to ``False``, the layer will not learn an additive bias (only relevant if :attr:`affine` is ``True``). Default: ``True`` Shape: - Input: :math:`(N, C, +)` - Output: :math:`(N, C, +)` (same shape as input) .. note:: Synchronization of batchnorm statistics occurs only while training, i.e. synchronization is disabled when ``model.eval()`` is set or if ``self.training`` is otherwise ``False``. Examples:: >>> # xdoctest: +SKIP >>> # With Learnable Parameters >>> m = nn.SyncBatchNorm(100) >>> # creating process group (optional) >>> # ranks is a list of int identifying rank ids. >>> ranks = list(range(8)) >>> r1, r2 = ranks[:4], ranks[4:] >>> # Note: every rank calls into new_group for every >>> # process group created, even if that rank is not >>> # part of the group. >>> process_groups = [torch.distributed.new_group(pids) for pids in [r1, r2]] >>> process_group = process_groups[0 if dist.get_rank() <= 3 else 1] >>> # Without Learnable Parameters >>> m = nn.BatchNorm3d(100, affine=False, process_group=process_group) >>> input = torch.randn(20, 100, 35, 45, 10) >>> output = m(input) >>> # network is nn.BatchNorm layer >>> sync_bn_network = nn.SyncBatchNorm.convert_sync_batchnorm(network, process_group) >>> # only single gpu per process is currently supported >>> ddp_sync_bn_network = torch.nn.parallel.DistributedDataParallel( >>> sync_bn_network, >>> device_ids=[args.local_rank], >>> output_device=args.local_rank) TNrrrrrr process_grouprrc L>XxS.n [T U]"UUUUU40U DSU 0D6 X`lgry)r)r*r) r5rrrrrrr"r#rr6r9s r:r*SyncBatchNorm.__init__sE%+;           +rBcfUR5S:a[SUR5S35eg)Nrz expected at least 2D input (got rrrJs r:rLSyncBatchNorm._check_input_dim s/ 99;?? }HUV V rBcDURS5S:Xa [S5eg)Nr rz9SyncBatchNorm number of input channels should be non-zero)sizerrJs r:_check_non_zero_input_channels,SyncBatchNorm._check_non_zero_input_channelss' ::a=A K  rBrKc FURU5 URU5 URcSnO URnUR(a{UR(ajUR c [ S5eUR RS5 URcSUR R5- nO URnUR(aSnO#URSL=(a URSLnUR(aUR(a UROSnUR(aUR(a UROSnU=(aV UR=(aC [RR5=(a [RR5nU(aURR SSS [R"R%54;a*['S [R"R%535e[RR(R*nUR,(a UR,n[RR/U5nUS:nU(d;[0R2"UUUUR4UR6UUUR85$U(d [ S 5e[:R<"UUR4UR6UUUR8UWW5 $) z Runs the forward pass. Nr|z$num_batches_tracked must not be Noner r}Tcudahpuxpuz;SyncBatchNorm expected input tensor to be on GPU or XPU or zbn_training must be True)rLrrr~rr'rritemr%r&r+ distributed is_availableis_initializedr"type_C_get_privateuse1_backend_namergroupWORLDrget_world_sizerrr$rrsync_batch_normapply) r5rKrrr%r& need_syncr world_sizes r:rSyncBatchNorm.forwards e$ ++E2 == ), &)- & ==T55''/$%KLL  $ $ ) )! ,}}$-043K3K3P3P3R-R*-1]]*  ==K,,4T4;K;Kt;SK &*]]d6N6ND  TX %)MMT5M5MD  SW   3  3!!..0 3!!002  ||  668 ) !Qxx==?@B "--3399M!! $ 2 2 **99-HJ"QI<<  *  $%?@@"((  *  rBc LUn[U[RRRR 5(Ga[RR URURURURURUURSLS9nUR(a@[R"5 URUlURUl SSS5 URUlUR UlUR"UlUR$Ul['US5(aUR(UlUR+5H%upEUR-X@R/XR55 M' AU$!,(df  N=f)aConverts all :attr:`BatchNorm*D` layers in the model to :class:`torch.nn.SyncBatchNorm` layers. Args: module (nn.Module): module containing one or more :attr:`BatchNorm*D` layers process_group (optional): process group to scope synchronization, default is the whole world Returns: The original :attr:`module` with the converted :class:`torch.nn.SyncBatchNorm` layers. If the original :attr:`module` is a :attr:`BatchNorm*D` layer, a new :class:`torch.nn.SyncBatchNorm` layer object will be returned instead. Example:: >>> # Network with nn.BatchNorm layer >>> # xdoctest: +REQUIRES(env:TORCH_DOCTEST_CUDA) >>> module = torch.nn.Sequential( >>> torch.nn.Linear(20, 100), >>> torch.nn.BatchNorm1d(100), >>> ).cuda() >>> # creating process group (optional) >>> # ranks is a list of int identifying rank ids. >>> ranks = list(range(8)) >>> r1, r2 = ranks[:4], ranks[4:] >>> # Note: every rank calls into new_group for every >>> # process group created, even if that rank is not >>> # part of the group. >>> # xdoctest: +SKIP("distributed") >>> process_groups = [torch.distributed.new_group(pids) for pids in [r1, r2]] >>> process_group = process_groups[0 if dist.get_rank() <= 3 else 1] >>> sync_bn_module = torch.nn.SyncBatchNorm.convert_sync_batchnorm(module, process_group) Nrqconfig)rr+nnmodules batchnormrvr rrrrrrno_gradr$r%r&r'r~hasattrrnamed_children add_moduleconvert_sync_batchnorm)clsmoduler module_outputnamechilds r:r$SyncBatchNorm.convert_sync_batchnormys@H fehh..88CC D D!HH22##  **[[,3M}}]]_+1==M()/M&%*0)<)rs *UU8! | | ~H H VL$OYL$^KT*KT\!TmZ!THLN*LN^!NmZ!NHLN*LN^!NmZ!NHmJmrB