+jBddlZddlZddlmZddlmZmZmZmZddl m Z m Z m Z m Z ddlmZerddlmZeGddZdS) N) dataclass) TYPE_CHECKINGLiteralOptionalUnion)DeepSpeedSequenceParallelConfigDistributedTypeTorchContextParallelConfigTorchTensorParallelConfig)is_torch_version) AcceleratorceZdZUdZdZeeed<dZeeed<dZ eeed<dZ eeed<dZ e ded<dZ eeed <dZe d ed <dZedefed <dZedefed <dZedefed<dZdZdZedZedZedZedZedZedZedZ edZ!edZ"edZ#edZ$edZ%edZ&edZ'de(fd Z)d,dee(fd!Z*d"e+e+ed#fe+e(d#fffd$Z,d%Z-d&e(d'efd(Z.d-d+Z/dS).ParallelismConfiga A dataclass to configure parallelisms applied to the model. Inspired by torchtitan's `ParallelDims` https://github.com/pytorch/torchtitan/blob/main/torchtitan/distributed/parallel_dims.py Args: dp_replicate_size (`int`, defaults to `1`): The size of the data parallel group. If `dp_replicate_size` is set to 1, the data parallel replication group will not be used. dp_shard_size (`int`, defaults to `1`): The size of the model shard group. If `dp_replicate_size > 1` and `tp_size > 1`, `dp_shard_size` must also be greater than 1, as composing DDP + TP is currently not supported. tp_size (`int`, defaults to `1`): The size of the tensor parallel group. If `tp_size` is set to `1`, the tensor parallel group will not be used. tp_handler (`~utils.TorchTensorParallelConfig`, defaults to `None`): The handler for the tensor parallel group. cp_size (`int`, defaults to `1`): The size of the context parallel group. Currently not supported, but reserved for future use and enabled for downstream libraries. cp_backend (`str`, defaults to `torch`): Which CP backend to use: `torch` (FSDP2) sp_size (`int`, defaults to `1`): The size of the sequence parallel group. sp_backend (`str`, defaults to `deepspeed`): Which SP backend to use:`deepspeed` (ALST/Ulysses) You may obtain different distributed data parallel paradigms by configuring `dp_replicate_size` and `dp_shard_size` together: - `dp_replicate_size == 1` and `dp_shard_size > 1`, we obtain Fully Sharded Data Parallel (FSDP). - `dp_replicate_size > 1` and `dp_shard_size > 1`, we obtain Hybrid Sharded Data Parallel (HSDP). - `dp_replicate_size > 1` and `dp_shard_size == 1` is an invalid configuration, to use pure DP, use `DistributedDataParallelKwargs` instead. Ndp_replicate_size dp_shard_sizetp_sizecp_sizetorch cp_backendsp_size deepspeed sp_backend tp_handler cp_handler sp_handlercd|jd|jd|jd|jd|jd|jd|jd|jd |jd |j d S) Nz'ParallelismConfig( dp_replicate_size=z, dp_shard_size=z , tp_size=z , cp_size=z, cp_backend=z , sp_size=z, sp_backend=z, total_size=z tp_handler=z, cp_handler=z) ) rrrrrrr total_sizerrselfs `/home/wildlama/comfy/ComfyUI/.venv/lib/python3.11/site-packages/accelerate/parallelism_config.py__repr__zParallelismConfig.__repr__Us 1#'#9 1 1#1 1 1 1 1 1 1 !O 1 1  1 1!O 1 1!O 1 1!O 1 1!O 1 1 1 cddldgfd|jDdS)Nr device_meshcxi|]6\}}|v |t|dr|jn|7S)__dict__)hasattrdeepcopyr&).0kv_non_serializable_fieldscopys r z-ParallelismConfig.to_json..jsW   Aq444:0F0FM4==,,,A444r")r-r(r&items)rr,r-s @@r to_jsonzParallelismConfig.to_jsondsn $1?        M//11        r"c>g}|jr|dgz }|jr|dgz }|S)zENames of enabled dimensions across which data parallelism is applied. dp_replicatedp_shard)dp_replicate_enableddp_shard_enabledrdimss r dp_dim_nameszParallelismConfig.dp_dim_namesqs=  $ % ^$ $D   ! ZL D r"cXg}|jr|dgz }|jr|dgz }|jr|dgz }|S)z]Names of enabled dimensions which will receive the same batch (non-data parallel dimensions).tpcpsp) tp_enabled cp_enabled sp_enabledr6s r non_dp_dim_namesz"ParallelismConfig.non_dp_dim_names{sM ?  TFND ?  TFND ?  TFND r"c>g}|jr|dgz }|jr|dgz }|S)zlNames of enabled dimensions which will be flattened into a joint mesh across which is model sharded in FSDP.r3r;)r5r>r6s r dp_shard_cp_dim_namesz'ParallelismConfig.dp_shard_cp_dim_namess:   ! ZL D ?  TFND r"cXg}|jr|dgz }|jr|dgz }|jr|dgz }|S)z@Names of enabled dimensions across which loss should be averagedr2r3r;)r4r5r>r6s r dp_cp_dim_namesz!ParallelismConfig.dp_cp_dim_namessR  $ % ^$ $D   ! ZL D ?  TFND r"c0g}|jr|dgz }|dgz }|S)z^Names of enabled dimensions across which FSDP is applied, including data parallel replication.r2 dp_shard_cp)r4r6s r fsdp_dim_namesz ParallelismConfig.fsdp_dim_namess2  $ % ^$ $D  r"cP|j|jz|jz|jz|jzS)zSThe total size of the parallelism configuration, which is the product of all sizes.)rrrrrrs r rzParallelismConfig.total_sizes,%(::T\IDLX[_[gggr"c0|j|jz|jzS)zhThe size of the non-data parallel dimensions, which is the product of tensor and context parallel sizes.)rrrrs r non_data_parallel_sizez(ParallelismConfig.non_data_parallel_sizes|dl*T\99r"c |j|jzS)z_The size of the data parallel dimensions, which is the product of data parallel replication and)rrrs r data_parallel_sizez$ParallelismConfig.data_parallel_sizes%(:::r"c|jdkS)zKTrue if data parallel replication is enabled, i.e. `dp_replicate_size > 1`.)rrs r r4z&ParallelismConfig.dp_replicate_enableds%))r"c|jdkS)zDTrue if data parallel sharding is enabled, i.e. `dp_shard_size > 1`.rN)rrs r r5z"ParallelismConfig.dp_shard_enableds!A%%r"c|jdkS)z:True if tensor parallelism is enabled, i.e. `tp_size > 1`.rN)rrs r r=zParallelismConfig.tp_enabled|ar"c|jdkS)z;True if context parallelism is enabled, i.e. `cp_size > 1`.rN)rrs r r>zParallelismConfig.cp_enabledrQr"c|jdkS)z;True if context parallelism is enabled, i.e. `sp_size > 1`.rN)rrs r r?zParallelismConfig.sp_enabledrQr"c |j|jzS)z$Names of all active mesh dimensions.)r8r@rs r active_mesh_dimsz"ParallelismConfig.active_mesh_dimss 4#888r" device_typec|jdkr |jdkrdStddrddlm}nt d|}t|dkrdS|\}}|||| }|jr ||j d |j r ||j  d |j r ||j  d |S) a!Builds a device mesh for the given device type based on the parallelism configuration. This method will also create required joint meshes (e.g. `dp_shard_cp`, `dp_cp`, `dp`). Args: device_type (`str`): The type of device for which to build the mesh, e rrNNz>=z2.2.0r)init_device_meshz4Building a device_mesh requires to have torch>=2.2.0)mesh_dim_namesdprFdp_cp) rrr torch.distributed.device_meshrX RuntimeError _get_meshlenr8_flattenrBrD)rrVrXmeshrY mesh_shaper$s r build_device_meshz#ParallelismConfig.build_device_meshs% ?k ) )dlQ.>.>4 D' * * W F F F F F F FUVV V~~ t99>>4%)" &&  )     : ) * 3 3D 9 9 9  % L 2 3 < <] K K K   @ , - 6 6w ? ? ?r"c|j,||||_nAtd|0|jj|kr td|jjd|d|jS)Nz@You need to pass a device_type e.g cuda to build the device meshz4The device_mesh is already created with device type z@. However, you are trying to get a device mesh with device_type z<. Please check if you correctly initialized your device_mesh)r$rc ValueErrorrV)rrVs r get_device_meshz!ParallelismConfig.get_device_meshs   #&#'#9#9+#F#F   !cddd&#/;>>$wtO_Okwwnywwwr"return.cfdjD}gdt|fd}tt |S)zQGenerate mesh shape and dimension names for torch.distributed.init_device_mesh().c,i|]}|j|S)_sizes)r) parallelismrs r r.z/ParallelismConfig._get_mesh..s"ddd{[$+k":dddr")r2r3r;r<r:c:|dS)Nr)index)x mesh_orders r z-ParallelismConfig._get_mesh..s:++AaD11r")key)rUsortedr/tuplezip)r mesh_dims sorted_itemsrps` @r r^zParallelismConfig._get_meshsseddddNcddd DCC  OO  2222   S,'(((r"c |j2ttjdd|_|j2ttjdd|_|j2ttjdd|_|j2ttjdd|_|j%tjdd|_|j 2ttjdd|_ |j %tjd d |_ |jd kr|j t|_ |jd kr|j t|_ nott }t!|j ||js:t#d |jd||jdt%|j |j d kr|jt)|_|jd krt#d|j|jd krt#d|j|jd krt#d|j|jd krt#d|jdg}|j|vrt#d|d|j|j d krt#d|j d g}|j |vrt#d|d|j |jd kr+|j d kr t#d|jd|j d|jd ks |jd kr%|jd kr|jd krt#d|j|j|j|j|j d|_dS)N$PARALLELISM_CONFIG_DP_REPLICATE_SIZE1 PARALLELISM_CONFIG_DP_SHARD_SIZEPARALLELISM_CONFIG_TP_SIZEPARALLELISM_CONFIG_CP_SIZEPARALLELISM_CONFIG_CP_BACKENDrPARALLELISM_CONFIG_SP_SIZEPARALLELISM_CONFIG_SP_BACKENDrrN)rzParallelismConfig's cp_backend=z requires z, but cp_handler was set to z.dp_replicate_size must be at least 1, but got z*dp_shard_size must be at least 1, but got z$tp_size must be at least 1, but got z$cp_size must be at least 1, but got zcp_backend must be one of z , but got z$sp_size must be at least 1, but got zsp_backend must be one of z[Context Parallelism (CP) and Sequence Parallelism (SP) are mutually exclusive. Got cp_size=z and sp_size=z+. Please set either cp_size=1 or sp_size=1.aCTensor/Context parallelism (tp/cp_size > 1) cannot be used with pure data parallelism (dp_replicate_size > 1 and dp_shard_size == 1). Please set dp_shard_size > 1 and dp_replicate_size == 1 to compose FSDP + TP/CP for 2D parallel, or set dp_replicate_size == 1 and dp_shard_size > 1 to compose HSDP + TP/CP for 3D parallel.)r2r3r:r;r<)rintosenvirongetrrrrrrrr rr dict isinstanceretyperrrk)rcp_backends_config_mapvalid_cp_backendsvalid_sp_backendss r __post_init__zParallelismConfig.__post_init__s(  ! )%(8^`c)d)d%e%eD "   %!$RZ^^4VX[%\%\!]!]D  < rz~~.JCPPQQDL < rz~~.JCPPQQDL ? " jnn-LgVVDO < rz~~.JCPPQQDL ? " jnn-LkZZDO ">)-4***&"$/3I$/3Z[[$r$/rrUklpl{U|rr[_`d`o[p[prr ,<<<  L1   q 0 0d6Lq6P6PUYUgklUlUlo  !2*,,,    r"rlsizec||jvs$Jd|j||j|<t||d|dS)NzParallelism must be one of _size)rkkeyssetattr)rrlrs r _set_sizezParallelismConfig._set_size^sodk..000002dPTP[P`P`PbPb2d2d000#' K +++T22222r" acceleratorr ct}|js |jdkrdS|jdkr|d|j|jdkr |jdkrn0|j|jkr td|jd|jd|jdkr;|js4|js-|j tj kstd|j d|j D];\}}|dkr0t||d d|d |d |d <|r9|jr4t#jd d|zt(dSdSdS)NrNr2rzParallelismConfig total_size (z ) does not match num_processes (zJ). Please adjust dp_replicate_size/ dp_shard_size/tp_size/cp_size/sp_size.zParallelismConfig is only compatible DistributedType.FSDP (version 2) or DistributedType.Multi{Device} or DistributedType.DEEPSPEED, but got ._handlerzParallelismConfig.z_handler is set, but z0_size is set to 1. This handler will be ignored.z.ParallelismConfig has the following warnings:  )set multi_devicerr num_processesrrreis_fsdp2distributed_typer DEEPSPEEDrkr/getattraddis_main_processwarningswarnjoin UserWarning)rr _warningsrlrs r _validate_acceleratorz'ParallelismConfig._validate_acceleratorcsEE ' DOq,@,@ F ?a   NN>;+D E E E ?k ) )dlQ.>.>  _ 9 9 9:::"-";:::  ?Q     ' +/HHHBcncBBB "&!2!2!4!4   KqyyWTk+C+C+CTJJV III;III  4  MADIIiDXDXX         r")N)rr )0__name__ __module__ __qualname____doc__rrr__annotations__rrrrrrrrrr rr rrr$r!r0propertyr8r@rBrDrGrrJrLr4r5r=r>r?rUstrrcrfrtr^rrrrjr"r rr!sc!!F(,x}+++#'M8C='''!GXc]!!!!GXc]!!!#'J '''!GXc]!!!'+J $+++:>Jd556===:>Jd667>>>?CJd;;<CCCK        X  X X  X XhhXh::X:;;X;**X*&&X&  X   X   X 99X9!S!!!!F   8C=      )5sCx%S/!AB ) ) ) )J J J X3S33333 ++++++r"r)rr dataclassesrtypingrrrraccelerate.utils.dataclassesrr r r accelerate.utils.versionsr accelerater rrjr"r rs !!!!!!:::::::::::: 766666'&&&&&& lllllll lllr"