3j~.%SSKJr SSKrSSKrSSKJrJr SSKJr SSK J r J r J r J r JrJr SSKrSSKJr SSKJr SSKJr SS KJrJr \(a SS KJr SS KJr \R:"\5r"S S \ 5r \!\!\"S4\S-4r#S\$S'"SS\%5r&\"SSS9"SS55r'\"SSS9"SS\'55r(\'\(-r)S\$S'\"SS9"SS55r*\"SSS9"SS55r+\"SSS9"SS 55r,SS)jjr1S*r2S?S@S+jjr3S?SAS,jjr4\"SS9"S-S.55r5SBS/jr6\ SS0.SCS1jj5r7\ SDS2j5r7S'S0.SES3jjr7SFSGS4jjr8\ SHSIS5jj5r9\ SHSJS6jj5r9SFSKS7jjr9S'S'S8.SLS9jjr:SS'S8.SMS:jjr;SNS;jrsP fg & &)\ ,,==?,, ..   r cR[X5nURUR5 U$)zReconstruct a tensor on ``device`` from this metadata. Args: device: Target device for the tensor. Returns: An empty strided tensor on ``device``. )_make_tensor_from_metarequires_grad_r9)rdevicets r to_tensor_TensorMeta.to_tensorVs' #4 0 ++,r cX:Xa/$/nURUR:wa+URSURSUR35 URUR:wa+URSURSUR35 URUR:wa+URSURSUR35 U$)zReturn field-by-field differences with ``other``. Args: other: Metadata to compare against. Returns: List of human-readable difference strings (empty if equal). zshape mismatch:  vs zstride mismatch: zdtype mismatch: )r4appendr6r7rotherdiffss rget_diff_TensorMeta.get_diffcs =I :: $ LL+DJJ torch.Tensorr#r1)rDtorch.device | strr#rPrLr1r# list[str]) r$r%r&r'r(__annotations__ staticmethodr?rFrNr)rr rr1r10s=     . r r1c\rSrSr%Sr\"SS9rS\S'\"SS9rS \S '\"SS9r S \S '\"SS9r S \S'\"SS9r S\S'\ SSj5r \SSj5rSSjrSSjrSrg) _DTensorMeta{aDTensor metadata extending :class:`_TensorMeta` with distribution info. Inherited fields (shape, stride, etc.) are LOCAL shard attributes. Additional fields capture global shape and placement information needed to reconstruct a :class:`DTensor` via ``DTensor.from_local()``. The :class:`DeviceMesh` is **not** stored (not serializable for P2P); it is looked up from :class:`_MeshCache` using ``(mesh_dim_names, mesh_layout)`` as the key. c.[R"/5$r)torchSizerr r_DTensorMeta.s UZZ^r )default_factoryr3 global_shaper)defaultr5 global_strideztuple[Placement, ...] placementsr!rNr"rc tURn[URRURR 5UR UR URUR 5URRUR(a[UR5OSURS9 $)zCreate metadata from a DTensor. Args: dtensor: The DTensor to extract metadata from. Returns: Metadata capturing both local and global attributes. r) r4r6r7r9r_rarbrr) device_meshrW _local_tensorr4r6r7r9_specrbrtuple_layout)dtensorrds r from_dtensor_DTensorMeta.from_dtensors)) ''--((//1--!// !..*}}//5@5O5Ok001UW#++  r c2URUR4$)zd5CTCTBUV   LL< = r )rirr#rW)r#r*)rDrQrsrr#rrR)r$r%r&r'r(rr_rTrarbrrrUrjpropertyrnrurNr)rr rrWrW{s  %5KLL*L%*2%6M?6).)J% ',B&7NO7&+'K#  877 2*r rW TensorMeta)r/cp\rSrSr%SrSrS\S'SrS\S'SrS\S'Sr S\S 'SS jr SS jr SS jr S r g) _StageMetaizPConsolidated tensor metadata for a pipeline stage's forward and backward passes.Ntuple[TensorMeta, ...] | Noneinputsoutputs$tuple[TensorMeta | None, ...] | None input_grads output_gradsc~[SURURURUR455$)z)Check if any metadata field is populated.c3*# UH nUSLv M g7frr).0vs r %_StageMeta.has_any..s U TMUs)anyr~rrrrms rhas_any_StageMeta.has_anys9 kk4<<1A1A4CTCTU   r cURUR4H%nU(dM [SU55(dM% g g)z3Check if any input/output metadata is DTensor type.c3R# UHo(dM [U[5v M g7fr)r<rWrms rr*_StageMeta.has_dtensors..sM%Q18Z<88%s ''TF)r~rr)rmetass r has_dtensors_StageMeta.has_dtensorss6kk4<<0EuM%MMM1r cHURSL=(a URSL$)z-Check if forward metadata is fully populated.N)r~rrms ris_complete_for_forward"_StageMeta.is_complete_for_forwards{{$&C4<.RsDA ?? !''!((!''QVW  sA A )rg) tensor_metass r_derive_grad_metasrJs    r cX\rSrSrSrS S SjjrS SjrSSjrSSjrSSjr SS jr S r g) _MeshCacheiZzCache for :class:`DeviceMesh` objects keyed by ``(mesh_dim_names, mesh_layout)``. Assumes all pipeline stages share the same rank tensor (true for TorchTitan-style frameworks where meshes derive from a common world). Nc0UlXlgr_cache _get_mesh_cb)r get_mesh_cbs r__init___MeshCache.__init__as68 'r cXR;aURU$Uup#URc[SUSUS35eURX#5nUc[SUSUS35eX@RU'U$)zReturn a cached mesh, or create one via the callback. Args: key: Cache key ``(mesh_dim_names, mesh_layout)``. Returns: The ``DeviceMesh``. Raises: PipeliningMetadataError: If not cached and no callback provided. z+Mesh not found in cache for mesh_dim_names=z, mesh_layout=z`, and no get_mesh callback provided. Provide a get_mesh callback or use DTensors in static mode.z>Mesh lookup failed: callback returned None for mesh_dim_names=z6. Ensure all stages use meshes from the same universe.)rrr,)rkeyrrrss rget_mesh_MeshCache.get_meshes ++ ;;s# #&)#    $)=n=MN*m,NO    = <)""0!1 }MGH   C r c X RU'g)zAdd a mesh to the cache.Nr)rrrss rput_MeshCache.puts Cr cUH|n[U[5(dMURnUR(a[ UR5OSnUR nXE4nX`R ;dMnX0R U'M~ g)zJExtract and cache meshes from any :class:`DTensor` instances in *tensors*.rN)r<rrdrrgrhr)rtensorsr>rs dim_namesrrs rupdate_from_tensors_MeshCache.update_from_tensorssfF&'**)):>:M:ME$"5"56SU "ll  .kk)'+KK$r cXR;$rr)rrs r __contains___MeshCache.__contains__skk!!r c,[UR5$r)lenrrms r__len___MeshCache.__len__s4;;r rr)rzGetMeshCallback | Noner#None)rr*r#r)rr*rsrr#r)rtuple[torch.Tensor | None, ...]r#r)rr*r#r8)r#int) r$r%r&r'r(rrrrrrr)rr rrrZs& ( D  ," r rc4\rSrSrSrSrSr\SSj5rSr g) InferenceModeiaPipeline-level metadata inference mode, determined collectively across all PP ranks. The mode is set by the schedule (not individual stages) because ``has_backward`` is only known at schedule creation time and all stages must agree to avoid P2P hangs. .. attribute:: STATIC All stages have sufficient metadata; runtime inference is skipped. .. attribute:: DYNAMIC At least one stage requires runtime metadata inference. staticdynamiccUR5(dgUR5(dgU(dgURb URcgg)zDetermine whether dynamic metadata inference is needed for a stage. Args: meta: Stage metadata from user-provided args. stage_has_backward: Whether a backward pass will be performed. Returns: ``True`` if dynamic inference is needed. TF)rrrr)clsrstage_has_backwards r needs_dynamicInferenceMode.needs_dynamicsS++--  """    #t'8'8'@r rN)rr|rr8r#r8) r$r%r&r'r(STATICDYNAMIC classmethodrr)rr rrrs% FGr rFdetachc[U5up#U(ahUVs/sHMn[U[R5(a)UR 5R UR 5OUPMO nn[XS5nXe4$U$s snf)aFlatten ``args`` into a list, optionally detaching tensors. Args: args: Nested arguments to flatten. detach: If ``True``, detach tensors while preserving ``requires_grad``. Returns: ``(new_args, flat_detached_args)`` when ``detach=True``; ``flat_args`` list otherwise. )rr<rZTensorrrCr9r)argsr flat_argstreespeca flat_detachednew_argss r flatten_argsrs't,I   !U\\** HHJ % %aoo 6   "-:&&  sAA?c[USS9$)zHFlatten and detach. Deprecated: use ``flatten_args(args, detach=True)``.Tr)r)rs rflatten_args_detachrs T **r c"0nUS:Xa[U5H nX@-X4'M U$US:XaYX-S:wa[SUSUS35eSn[U5H,nXSU'US-U-S:XaMX@-S-S:XaUS- nM'US-nM. U$[S US 35e) z Compute the stage id to rank mapping for either a looped or V-style schedule. Most commonly num_stages == pp_size * 2, but this function can be used to compute the mapping for any number of stages per rank. looprrz num_stages z% must be evenly divisible by pp_size z for V scheduleszStyle z is not supported.)range ValueError)pp_size num_stagesstylemapping stage_index rank_indexs rgenerate_stage_to_rank_mappingrsG  ,K#.#8G -( N% #  1 $j\)NwiWgh   ,K#-K a7*a/&!+q0a a - N6%(:;< list of stage indices assigned to that rank. )ritemsrJvaluessort)rrr stage_to_rankrank_to_stagesstage_idrankstagess rgenerate_rank_to_stage_mappingrsq37NM,.N'--/  %#%N ##H-0 !'') * r c8\rSrSr%SrS\S'S\S'S\S'S rg ) PipeInfoi6z6 Captures information for a pipeline (`Pipe` object). zfx.Graphgraphrrr8has_loss_and_backwardrNrrr rrr6s OOr rc[U[5(a[RU5$[R U5$)aExtract metadata from a tensor. Handles both plain Tensor and DTensor correctly: DTensors are dispatched to ``_DTensorMeta.from_dtensor`` which captures local shard attributes plus global shape/placement info, while plain tensors use ``_TensorMeta.from_tensor``. Args: tensor: A plain tensor or DTensor. Returns: ``_TensorMeta`` for plain tensors, ``_DTensorMeta`` for DTensors. )r<rrWrjr1r?r=s rextract_tensor_metarFs3&'""((00&&v..r ) allow_nonecgrrrrs rextract_tensor_metasrZs %(r cgrrrs rrrbs ,/r cUcg/nSnUHQn[U[R5(aUR[ U55 M>SnURS5 MS U(dU(a [ S5e[ U5$)ajExtract metadata from a tuple of tensors. Args: tensors: Tuple of tensors (may include ``None`` when ``allow_none=True``). allow_none: If ``True``, preserve ``None`` elements (for gradients). Returns: Tuple of ``TensorMeta``, or ``None`` if ``tensors`` is ``None``. Raises: PipeliningMetadataError: If ``None`` found and ``allow_none=False``. NFTz_None values are not allowed in tensor metadata tuples. Use allow_none=True for optional values.)r<rZrrJrr,rg)rrmetas_with_nonehas_nonerEs rrrjs~"/1OH  a & &  " "#6q#9 :H  " "4 (  (% 7    !!r cU(aUR5OUn[U[5(aUR5$U$)uConvert a DTensor to its local shard, or return a plain tensor as-is. When ``detach=True``, the tensor is detached before conversion — this applies to both DTensors and plain tensors. Args: tensor: A tensor that may be a DTensor. detach: If ``True``, detach before ``to_local()`` to avoid redistribution during backward. Returns: The local tensor component. )rr<rto_local)r>rmaybe_detached_tensors rto_local_if_dtensorr s706FMMO6'11$--//  r cgrrrrs rvalidate_and_normalize_to_tupler s'*r cgrrr s rr r s.1r c Ucg[U[R5(aU4$[U[[45(a[ U5Hdup#UcU(d[ SUS35eM [U[R5(aMA[ SUS[U5RS35e [U[5(a [U5$U$[ S[U5RS35e)aNormalize ``args`` to a tuple and validate that all elements are tensors. Args: args: A single tensor, tuple/list of tensors, or ``None``. allow_none: If ``True``, permit ``None`` elements (for gradients). Returns: Tuple of tensors, or ``None`` if ``args`` is ``None``. Raises: PipeliningMetadataError: On non-tensor values (or ``None`` when ``allow_none=False``). Nz Stage arg[zF] is None. Stage args must be tensors. Use kwargs for optional values.z ] has type zC. All stage args must be tensors. Use kwargs for non-tensor inputs.z$>%J4PT:K^K^J__` a  r raise_on_mismatchwarn_on_mismatchc`[U[R5(a [U5nOUn[ U5[ U5LauS[ U5R S[ U5R 3/nU(a[ USUS35eU(a#[R"USUSS3[SS9 U$URU5nU(a\U(a[ USS RU535eU(a/[R"US S RU5S3[SS9 U$) a, Compare expected metadata against actual tensor or metadata. This is the unified validation/comparison function that uses get_diff() from metadata classes. Works with both plain tensors and DTensors. For plain tensors: compares shape/stride/dtype/requires_grad. For DTensors: compares all properties including global shape and placements. Args: desc: Description for error/warning messages. expected: Expected tensor metadata (_TensorMeta or _DTensorMeta). actual: Actual tensor or metadata to compare against. raise_on_mismatch: If True, raise PipeliningMetadataError on mismatch. warn_on_mismatch: If True, issue a warning on mismatch. Returns: List of differences (empty if metadata matches). Raises: PipeliningMetadataError: If raise_on_mismatch=True and differences exist. ztype: expected , got : rz: Metadata type mismatch. z.. Using dynamically inferred metadata instead.r stacklevelz; z: Metadata mismatch. ) r<rZrrrr$r,warningswarn UserWarningrNjoin)descexpectedactualrr actual_meta type_diffrMs rvalidate_metadatar's3>&%,,'')&1   H~T+..d8n556fT+=N=W=W-?@?@   Lr c :[U5[U5:waRUS[U5S[U53nU(a [U5eU(a[R"U[SS9 U/$/n[ [ XSS95HunupUcU cMUbU c]USUSUcS OS S U cS OS 3nU(a [U5eU(a[R"U[SS9 URU5 Ms[USUS 3UU UUS 9n URU 5 M U$)aValidate metadata for a tuple of tensors element-wise. Args: desc: Description prefix for error/warning messages. expected: Tuple of expected metadata (may include ``None`` for grads). actual: Tuple of actual tensors or metadata to compare against. raise_on_mismatch: If ``True``, raise on the first mismatch. warn_on_mismatch: If ``True``, issue warnings for mismatches. Returns: Aggregated list of difference strings. Raises: PipeliningMetadataError: If lengths differ or on mismatch. z : expected z tensors, got rrTstrict[z ]: expected rmetadatar]r) rr,rrr rziprJr'extend) r"r#r$rrmsg all_diffsrexpactrMs rvalidate_tensors_metadatar4-s6. 8}F #k#h-s6{mL )#. .  MM#{q 9u I"3x#EF :C ;3;  ;#+&!L3;J(OP!$v*=? !-c22 c;1=   S ! !fAaSN  /-   +G, r cU(aSOSnUS3nUS3n[U5[U5:wa-[SUSUS[U5SUS[U5S 3 5e[[XS S 95GHunupUR(d4U b1[SUSUS USUS US[ U 5R S3 5eUR(a-U c*[R"SUSUS USUS US3 [SS9 [U[5(dMUR(dMU cM[U [5(aM[SUSUS USUS US[ U 5R S3 5e g )u Validate the args↔grads contract for static mode. Enforces four rules for each (arg, grad) pair: 1. len(args) must equal len(grads). 2. If arg.requires_grad is False, grad must be None. 3. If arg.requires_grad is True and grad is None, emit a warning (this is legal at pipeline boundaries but may indicate a bug). 4. If arg is a DTensor with requires_grad=True and grad is not None, grad must also be a DTensor. Args: stage_index: The stage index for error messages. args: Tuple of forward tensors. grads: Tuple of gradient tensors (can include None). is_input: True for input_args/input_grads, False for output_args/output_grads. Raises: PipeliningMetadataError: If any hard rule (1, 2, or 4) is violated. inputoutput_args_gradszStage rz length (z) does not match zo). Each forward tensor must have a corresponding gradient entry (use None for tensors that don't require grad).Tr)Nr+z] has requires_grad=False, but z] is not None (zE). Non-differentiable tensors must have None as their gradient entry.z] has requires_grad=True, but zT] is None. This is legal at pipeline boundaries but may indicate a missing gradient.rrz,] is a DTensor with requires_grad=True, but z] is za, expected DTensor or None. DTensor gradients may have different placements than forward tensors.) rr,rr.r9rr$rrr r<r) rrgradsis_inputkind args_name grads_namerrgrads r'validate_static_arg_grad_correspondencer@fs47HD&I6J 4yCJ%[MJrPr#rz)rtuple[torch.Tensor, ...] | NonerLiteral[False]r#r})r&tuple[torch.Tensor | None, ...] | Noner Literal[True]r#r)rAtuple[torch.Tensor | None, ...] | tuple[torch.Tensor, ...] | Nonerr8r#r)F)r>rPrr8r#rP).)rzCtorch.Tensor | tuple[torch.Tensor, ...] | list[torch.Tensor] | NonerrCr#rB)rzQtorch.Tensor | tuple[torch.Tensor | None, ...] | list[torch.Tensor | None] | NonerrEr#rD)rztorch.Tensor | tuple[torch.Tensor, ...] | tuple[torch.Tensor | None, ...] | list[torch.Tensor] | list[torch.Tensor | None] | Nonerr8r#rF) r"rAr#rzr$ztorch.Tensor | TensorMetarr8rr8r#rS) r"rAr#rr$z,tuple[torch.Tensor | TensorMeta | None, ...]rr8rr8r#rS) rrrztuple[torch.Tensor, ...]r:rr;r8r#r)= __future__rloggingr dataclassesrrenumrtypingrrr r r r rZr torch.distributed._mesh_layoutrtorch.distributed.tensorrtorch.utils._pytreerrtorch.distributed.device_meshr(torch.distributed.tensor.placement_typesr getLoggerr$loggerrrgrAr*rT RuntimeErrorr,r1rWrzr|rrrBrrrrrrrrrrr r r'r4r@rr rrTs#(NN 6,<8B   8 $h c3ht1C CD iDGlG $d#GG$GT $d#@;@$@H$l2 I2  DDD6 $d#**$*  $d#  $   * ( #  @ @ P0D0p*/6+ 17 !*-F17 !*-4    /( "%( ,((# ( ( / 3//* / /!" N!"!"* !"H!( "%* M**%* *  !$ 1 1  1 , 1 1 -  - - G- t$" B BB &B  B  BBT#" 6 6+6 96  6  66rDD "D +D D  Dr