3jT:|SSKJr SSKJrJr SSKrSSKJr SSK J r \(a\ "\ 5r \"SS55r \"SS 55r\"S S 9"S S 55r\"S S 9"SS55r\\\-\\\-\\S4-4r\\\-\\S4-r\\\\-4rS\S\R2S\\4Sjrg)) dataclass) TYPE_CHECKINGLiteralN) get_loggerc\rSrSr%SrSr\S-\S'Sr\S-\S'Sr \ \S'Sr \ S \S 'Sr \RR R"S-\S 'S r\ \S 'S r\ \S'Sr\\S'Sr\\S'Sr\R.\S'Sr\RR R"\S'Sr\RR R"\S'Sr\RR R"\S'Sr\RR R"\S'Sr\\S'Sr\\S'Sr\S\ \\44Sj5r!\S\ \"\"44Sj5r#S\S\S\R.S \RR R"4Sjr$S r%g)!ContextParallelConfig)a3 Configuration for context parallelism. Args: ring_degree (`int`, *optional*, defaults to `1`): Number of devices to use for Ring Attention. Sequence is split across devices. Each device computes attention between its local Q and KV chunks passed sequentially around ring. Lower memory (only holds 1/N of KV at a time), overlaps compute with communication, but requires N iterations to see all tokens. Best for long sequences with limited memory/bandwidth. Number of devices to use for ring attention within a context parallel region. Must be a divisor of the total number of devices in the context parallel mesh. ulysses_degree (`int`, *optional*, defaults to `1`): Number of devices to use for Ulysses Attention. Sequence split is across devices. Each device computes local QKV, then all-gathers all KV chunks to compute full attention in one pass. Higher memory (stores all KV), requires high-bandwidth all-to-all communication, but lower latency. Best for moderate sequences with good interconnect bandwidth. convert_to_fp32 (`bool`, *optional*, defaults to `True`): Whether to convert output and LSE to float32 for ring attention numerical stability. rotate_method (`str`, *optional*, defaults to `"allgather"`): Method to use for rotating key/value states across devices in ring attention. Currently, only `"allgather"` is supported. ulysses_anything (`bool`, *optional*, defaults to `False`): Whether to enable "Ulysses Anything" mode, which supports arbitrary sequence lengths and head counts that are not evenly divisible by `ulysses_degree`. When enabled, `ulysses_degree` must be greater than 1 and `ring_degree` must be 1. ring_anything (`bool`, *optional*, defaults to `False`): Whether to enable "Ring Anything" mode, which supports arbitrary sequence lengths. When enabled, `ring_degree` must be greater than 1 and `ulysses_degree` must be 1. mesh (`torch.distributed.device_mesh.DeviceMesh`, *optional*): A custom device mesh to use for context parallelism. If provided, this mesh will be used instead of creating a new one. This is useful when combining context parallelism with other parallelism strategies (e.g., FSDP, tensor parallelism) that share the same device mesh. The mesh must have both "ring" and "ulysses" dimensions. Use size 1 for dimensions not being used (e.g., `mesh_shape=(2, 1, 4)` with `mesh_dim_names=("ring", "ulysses", "fsdp")` for ring attention only with FSDP). N ring_degreeulysses_degreeTconvert_to_fp32 allgather)ralltoall rotate_methodmeshFulysses_anything ring_anything_rank _world_size_device_mesh_flattened_mesh _ring_mesh _ulysses_mesh_ring_local_rank_ulysses_local_rankcURcSUlURcSUlURS:XaURS:Xa [S5eURS:dURS:a [S5eURS:wa[ SURS35eUR (a6URS:Xa [S5eURS:a [S5eUR (a6URS:Xa [S 5eURS:a [S 5eUR (aUR (a [S 5egg) NzfEither ring_degree or ulysses_degree must be greater than 1 in order to use context parallel inferencezF`ring_degree` and `ulysses_degree` must be greater than or equal to 1.rz=Only rotate_method='allgather' is supported for now, but got .zIulysses_degree must be greater than 1 for ulysses_anything to be enabled.z8ulysses_anything cannot be enabled when ring_degree > 1.zCring_degree must be greater than 1 for ring_anything to be enabled.z8ring_anything cannot be enabled when ulysses_degree > 1.z:ulysses_anything and ring_anything cannot both be enabled.)r r ValueErrorrNotImplementedErrorrrselfs ]/home/wildlama/miniconda3/lib/python3.13/site-packages/diffusers/models/_modeling_parallel.py __post_init__#ContextParallelConfig.__post_init__esO    # D     &"#D    q T%8%8A%=x    a 4#6#6#:ef f    ,%OPTPbPbOccde   ""a' !lmm!# ![\\   1$ !fgg""Q& ![\\  T%7%7YZ Z&8 returnc2URUR4$N)r r r"s r$ mesh_shape ContextParallelConfig.mesh_shapes  $"5"566r'cg)z$Dimension names for the device mesh.ringulyssesr"s r$mesh_dim_names$ContextParallelConfig.mesh_dim_namess#r'rank world_sizedevicec XlX lX0lX@lURUR -U:a)[ SUR SURSUS35eURSR5UlURSUl URSUl URR5Ul URR5Ul g)NzThe product of `ring_degree` (z) and `ulysses_degree` (z") must not exceed the world size (z).r.r/r0)rrrrr r r _flattenrrrget_local_rankrrr#r4r5r6rs r$setupContextParallelConfig.setups %    !1!1 1J >01A1A0BBZ[_[n[nZopRS]R^^`a  $zz*;<EEG**V,!ZZ 2 $ > > @#'#5#5#D#D#F r') rrrrrrrrrr r )&__name__ __module__ __qualname____firstlineno____doc__r int__annotations__r r boolrrrtorch distributed device_mesh DeviceMeshrrrrrr6rrrrrrr%propertytupler+strr2r;__static_attributes__r1r'r$r r )s"H#Kt"!%NC$J% OT 6AM723A<@D%   ' ' 2 2T 9@#d"M4E3K GU\\ 6:E5   ( ( 3 3:@DOU&&22==D;?J!!--88?>BM5$$00;;B c ##[:7E#s(O77#c3h##G#G3G GEL]L]LiLiLtLtGr'r c .\rSrSr%SrSr\S-\S'Sr\ \S'Sr \ \S'Sr \ R\S'Sr\ RR R"\S'SS .S \ S \ S \ RS \ RR R"S-4SjjrSrg)ParallelConfigz Configuration for applying different parallelisms. Args: context_parallel_config (`ContextParallelConfig`, *optional*): Configuration for context parallelism. Ncontext_parallel_configrrrr)rr4r5r6rcXlX lX0lX@lURbURR XX45 ggr*)rrrrrPr;r:s r$r;ParallelConfig.setups@ %   ' ' 3  ( ( . .t N 4r')rrrr)r=r>r?r@rArPr rCrrBrrrEr6rrFrGrHr;rLr1r'r$rNrNs=A2T9@E3K GU\\ 6:E5   ( ( 3 3:AE O O O O ++66= O Or'rNT)frozencL\rSrSr%Sr\\S'Sr\S-\S'Sr\ \S'Sr S r g) ContextParallelInputa Configuration for splitting an input tensor across context parallel region. Args: split_dim (`int`): The dimension along which to split the tensor. expected_dims (`int`, *optional*): The expected number of dimensions of the tensor. If provided, a check will be performed to ensure that the tensor has the expected number of dimensions before splitting. split_output (`bool`, *optional*, defaults to `False`): Whether to split the output tensor of the layer along the given `split_dim` instead of the input tensor. This is useful for layers whose outputs should be split after it does some preprocessing on the inputs (ex: RoPE). split_dimN expected_dimsF split_outputcVSURSURSURS3$)NzContextParallelInput(split_dim=, expected_dims=z, split_output=))rWrXrYr"s r$__repr__ContextParallelInput.__repr__sK00@@PQUQcQcPddstxuFuFtGGHI Ir'r1) r=r>r?r@rArBrCrXrYrDr]rLr1r'r$rUrUs, N $M3:$L$Ir'rUc>\rSrSr%Sr\\S'Sr\S-\S'SrSr g)ContextParallelOutputa Configuration for gathering an output tensor across context parallel region. Args: gather_dim (`int`): The dimension along which to gather the tensor. expected_dims (`int`, *optional*): The expected number of dimensions of the tensor. If provided, a check will be performed to ensure that the tensor has the expected number of dimensions before gathering. gather_dimNrXc<SURSURS3$)Nz!ContextParallelOutput(gather_dim=r[r\)rbrXr"s r$r]ContextParallelOutput.__repr__s&24??2CCSTXTfTfSgghiir'r1) r=r>r?r@rArBrCrXr]rLr1r'r$r`r`s" O $M3:$jr'r`.sizegroupr(c [R"US9n[[R"US95nSU;aSO[R R 5n[U5Vs/sH'n[R"SU[RS9PM) nn[R"U[R"U/U[RS9US9 UVs/sHowSR5PM nnU$s snfs snf)zkGather the local size from all ranks. size: int, local size return: list[int], list of size from all ranks )rfcpu)r)r6dtyper) distget_world_sizerK get_backendrE acceleratorcurrent_acceleratorrangeemptyint64 all_gathertensoritem)rerfr5 comm_backends gather_device_gathered_sizesss r$gather_size_by_commrz%s"$$51J((u56M"m3E9J9J9^9^9`MZ_`jZklZkUVekk$}EKKPZkNlOO dVME ,::>adiik>N: m;s %.C2C7) dataclassesrtypingrrrEtorch.distributedrFrjutilsrr=loggerr rNrUr`dictrKrBlistrJContextParallelInputTypeContextParallelOutputTypeContextParallelModelPlan ProcessGrouprzr1r'r$rsB$")  H  pGpG pGf OO O@ $III0 $jjj0 #I#d+?&@@5I]_bIbCccc 2D9N4OORWXmorXrRss %=@Y%Y YZV c $*;*; S r'