9jQSrSSKrSSKrSSKJr SSKJrJr SSKJ r J r SSK J r SSK r SSKJrJr S S KJr S S KJrJr S S KJrJrJr S S KJr \"5(a SSKJs Jr \"5(aSSK J!r! Sr"Sr#Sr$Sr%Sr&\"SS.Sjr'SGSjr(Sr)Sr*Sr+Sr,Sr-Sr.Sr/Sr0"S S!\15r2S"r3S#r4\3S$5r5S%\ 4S&jr6S%\ 4S'jr7S%\ 4S(jr8SHS)jr9SIS*jr:\ RvS \ RxS\ RzS+\ R|S,\ R~S-\ RS.\ RS/\ RS0\ RS1\ RS20 rE\ER5VVs0sHupX_M snnrGS3rHSJS4\ R4S5jjrJ\3SHS6\K4S7jj5rLSHS6\K4S8jjrMSHS6\K4S9jjrNSKS:jrOSHS;jrP"S<S=\Q5rR\4SLS>j5rSSHS?jrT\3SMS@j5rUSArV"SBSC5rWSDrXSErY\SNSFj5rZgs snnf)OzB A set of basic tensor ops compatible with tpu, gpu, and multigpu N)Mapping)contextmanager nullcontext)update_wrapperwraps)Any)AcceleratorState PartialState)!TORCH_DISTRIBUTED_OPERATION_TYPES)DistributedTypeTensorInformation)is_npu_availableis_torch_distributed_availableis_torch_xla_available)is_torch_version)ReduceOpc6[U[R5$N) isinstancetorchTensortensors U/home/wildlama/miniconda3/lib/python3.13/site-packages/accelerate/utils/operations.pyis_torch_tensorr-s fell ++c v[U[RR[RR[RR [RR [RR[RR[RR5$r) rrxpu FloatTensor ByteTensor IntTensor LongTensor HalfTensor DoubleTensorBFloat16Tensorrs ris_torch_xpu_tensorr(1sm            rc"[U[5$r)rr tensor_infos ris_tensor_informationr,>s k#4 55rcn[U[5=(a [US5=(a [US5$)z Checks if `data` is a `namedtuple` or not. Can have false positives, but only if a user is trying to mimic a `namedtuple` perfectly. _asdict_fields)rtuplehasattrdatas r is_namedtupler4Bs* dE " \wtY'? \GDR[D\\rcr[U5(a[U5"[U56$[U5"U5$)zG Cast a generator to the same type as obj (list, tuple, or namedtuple) )r4typelist)obj generators r honor_typer:Js1 SCy$y/**Cy##rF test_typeerror_on_other_typec ^^^^^[U[[45(a[UUUUUU4SjU55$[U[5(aF[ U5"UR 5VVs0sHupgU[TU/TQ7TTS.TD6_M snn5$T"U5(a T"U/TQ70TD6$T(a2[S[ U5STRSTRS35eU$s snnf)a Recursively apply a function on a data structure that is a nested list/tuple/dictionary of a given base type. Args: func (`callable`): The function to recursively apply. data (nested list/tuple/dictionary of `main_type`): The data on which to apply `func` *args: Positional arguments that will be passed to `func` when applied on the unpacked data. main_type (`type`, *optional*, defaults to `torch.Tensor`): The base type of the objects to which apply `func`. error_on_other_type (`bool`, *optional*, defaults to `False`): Whether to return an error or not if after unpacking `data`, we get on an object that is not of type `main_type`. If `False`, the function will leave objects of types different than `main_type` unchanged. **kwargs (additional keyword arguments, *optional*): Keyword arguments that will be passed to `func` when applied on the unpacked data. Returns: The same data structure as `data` with `func` applied to every object of type `main_type`. c3J># UHn[TU/TQ7TTS.TD6v M g7f)r;Nrecursively_apply).0oargsr=funckwargsr<s r $recursively_apply..nsA A"!".7M`djs #r;zUnsupported types (z ) passed to `z?`. Only nested list/tuple/dicts of objects that are valid for `z` should be passed.) rr0r7r:rr6itemsrA TypeError__name__)rEr3r<r=rDrFkvs` ```` rrArAUs,$ &&        D' " "Dz !JJL  )DA$!".7M`dj)     4D*4*6** !$t*]4==/J++4+=+=*>>Q S   K s. C* c^^^[U5(d[US5(aTS:XaSmURTTS9$[ U[[45(a[UUUU4SjU55$[ U[5(ad[ T[5(aT/mOTc/m[U5"UR5VVs0sHupVXUT;aUO [UTTTS9_M snn5$U$![a URT5s$[a5n[ 5(a[ T[5(aST3mSnAO UeSnAff=fURTTS9$![a URT5s$f=fs snnf) aO Recursively sends the elements in a nested list/tuple/dictionary of tensors to a given device. Args: tensor (nested list/tuple/dictionary of `torch.Tensor`): The data to send to a given device. device (`torch.device`): The device to send the data to. non_blocking (`bool`, *optional*, defaults to `False`): If `True`, the transfer to the device is performed asynchronously, which can overlap data movement with computation. Only effective when the device supports it (e.g. CUDA). skip_keys (`str` or `List[str]`, *optional*): A key or list of keys in a dictionary `tensor` whose values should not be sent to the given `device`. Entries with these keys are left on their original device. Returns: The same data structure as `tensor` with all tensors sent to the proper device. tonpuznpu:0) non_blockingznpu:Nc3<># UHn[UTTTS9v M g7f)rQ skip_keysN)send_to_device)rBtdevicerQrTs rrG!send_to_device..s ohncd^AvLT]^hnsrS)rr1rOrJAssertionErrorrrintr0r7r:rstrr6rIrU)rrWrQrTerrorrLrVs ``` rrUrUs{&v'&$"7"7 U?F 99V,9? ? FUDM * * ohno   FG $ $ i % %" I  IF|#LLN *DAY1N1fS_kt,uu*    = %99V$ $  !!fc**#F8_F   %99V,9? ? %99V$ $ % s;C, E< ,E E)EEE EE98E9cSn[X5$)a/ Recursively gathers the information needed to rebuild a nested list/tuple/dictionary of tensors. Args: data (nested list/tuple/dictionary of `torch.Tensor`): The data to send to analyze. Returns: The same data structure as `data` with [`~utils.TensorInformation`] instead of tensors. c>[URURS9$)N)shapedtype)rr_r`rs r_get_data_structure/get_data_structure.._get_data_structures v||6<<HHrr@)r3ras rget_data_structurercsI 0 77rcSn[X5$)a Recursively gathers the shape of a nested list/tuple/dictionary of tensors as a list. Args: data (nested list/tuple/dictionary of `torch.Tensor`): The data to send to analyze. Returns: The same data structure as `data` with lists of tensor shapes instead of tensors. c,[UR5$r)r7r_rs r _get_shapeget_shape.._get_shapesFLL!!rr@)r3rfs r get_shaperhs" Z ..rc$Sn[X[S9$)z Recursively initializes tensors from a nested list/tuple/dictionary of [`~utils.TensorInformation`]. Returns: The same data structure as `data` with tensors instead of [`~utils.TensorInformation`]. cV[R"URSUR06$Nr`)remptyr_r`r*s r_initialize_tensor.initialize_tensors.._initialize_tensors"{{K--G[5F5FGGrr<)rAr,)data_structurerms rinitialize_tensorsrqsH /K` aarc[U[[[45(a'[ U5S:Xa[ S[ U5S35e[U[[45(a[US5$[U[5(a%UR5Hn[X5s $ O7[U[R5(d[S[ U5S35eURS$)z Recursively finds the batch size in a nested list/tuple/dictionary of lists of tensors. Args: data (nested list/tuple/dictionary of `torch.Tensor`): The data from which to find the batch size. Returns: `int`: The batch size. rz&Cannot find the batch size from empty .z0Can only find the batch size of tensors but got ) rr0r7rlen ValueErrorr6find_batch_sizekeysrrrJr_)r3rLs rrvrvs$g.//SY!^A$t*QOPP$ &&tAw'' D' " "A"47+ + ell + +J4PT:,VWXYY ::a=rcF[U5$![[4a gf=f)a Same as [`utils.operations.find_batch_size`] except will ignore if `ValueError` and `TypeErrors` are raised Args: data (nested list/tuple/dictionary of `torch.Tensor`): The data from which to find the batch size. Returns: `int`: The batch size. N)rvrurJr2s rignorant_find_batch_sizery s, t$$  "   s   cSn[X5$)a; Recursively finds tensors in a nested list/tuple/dictionary and converts them to a list of numbers. Args: data (nested list/tuple/dictionary of `torch.Tensor`): The data from which to convert to regular numbers. Returns: The same data structure as `data` with lists of numbers instead of `torch.Tensor`. cUR5R5nUR[R:XaUR [R 5nUR5$r)detachcpur`rbfloat16rOfloat32tolistrs r_convert_to_list!listify.._convert_to_list'sF$$& <<5>> )YYu}}-F}}rr@)r3rs rlistifyrs - 44rcJSn[XSS9n[R"5 U$)NcURS:XaUR5SnUR5(dUR5n[R "U5$)Nr)ndimclone is_contiguous contiguousxm all_gatherrs r_tpu_gather_one$_tpu_gather.._tpu_gather_one4sL ;;! \\^D)F##%%&&(F}}V$$rTr=)rAr mark_step)rrress r _tpu_gatherr3s#% O NCLLN Jrc^^[5m[RRmTRR S:Xa/[ SS5(a[RR5 UU4Sjn[XSS9$)Nr z<=z2.8c>URS:XaUR5SnUR5(dUR5nTRbTRS:waq[ R "TRUR5-URTRS9nT"X5 UR"S/UR5SSQ76$[TR5Vs/sHn[ R"U5PM nn[ RR!X5 [ R""USS9$s snf)Nrgloor`rWr dim)rrrrbackendrrl num_processesnumelr`rWviewsizerange empty_like distributedrcat)routput_tensors_ gather_opstates r_gpu_gather_one$_gpu_gather.._gpu_gather_oneJs  ;;! \\^D)F##%%&&(F == $&)@ #[[##flln4ll||N n -!&&r>FKKM!",=> > AFeFYFY@Z[@Z1e..v6@ZN[    ( ( @99^3 3\s/ ETr) r rrall_gather_into_tensorrWr6rr synchronizerA)rrrrs @@r _gpu_gatherrBs` NE!!88I ||E!&6tU&C&C 48 _$ OOrc\rSrSrSrSrg)DistributedOperationExceptioniiz} An exception class for distributed operations. Raised if the operation cannot be performed due to the shape of the tensors. N)rK __module__ __qualname____firstlineno____doc____static_attributes__rrrrris   rrc0^[T5U4Sj5nU$)zn Verifies that `tensor` is the same shape across all processes. Only ran if `PartialState().debug` is `True`. c R>[5R[R:Xd[5R(dT "U0UD6$T R ST R 3nSU;aUSnOUSn[5RR[U5R:wag[SUSURRS[5RRS[5RRSUS3 5e[U5n[U/5nUSbmURUS5[U5:HnU(dFS R[!U5VVs/sH upxS US U3PM snn5n [S US U 35eT "U0UD6$s snnf)Nrsrrz%One or more of the tensors passed to z were not on the z+ while the `Accelerator` is configured for z. Please move it to the z before calling z - zProcess z: znCannot apply desired operation due to shape mismatches. All shapes across devices must be valid. Operation: `z` Input shapes: - )r distributed_typerNOdebugrrKrWr6 find_devicerrh gather_objectcountrtjoin enumerate) rDrF operationrshapesoutputare_sameir_process_shape_strfunctions rwrapper!verify_operation..wrapperws > * *o.@.@ @ H\H\T,V, ,**+1X->->,?@ v H%F!WF > % %V)<)A)A A/7 {BSTZTaTaTfTfSghST`TbTiTiTnTnSoo))5)>)>)C)C(DDTU^T__`b 6"x( !9 ||F1I.#f+=H$,MM[dek[l2m[lxqXaS5'3J[l2m$n!3''0k1GHYGZ\ ((( 3ns.F# rrrs` rverify_operationrrs"  8_))4 Nrc0^[T5U4Sj5nU$)z Checks that `verify_operation` failed and if so reports a more helpful error chaining the existing `DistributedOperationException`. c>T"U0UD6$![a0nTRSTR3n[SUS35UeSnAff=f)NrszError found while calling `z1`. Please see the earlier error for more details.)rrrK)rDrFerrs rr"chained_operation..wrappersc T,V, ,, #../q1B1B0CDI/-i[8ij  s A+AArrs` rchained_operationrs"  8_ Nrc[5R[R:Xa [ U5$[5R[ ;a [ U5$U$)a Recursively gather tensor in a nested list/tuple/dictionary of tensors from all devices. Args: tensor (nested list/tuple/dictionary of `torch.Tensor`): The data to gather. Returns: The same data structure as `tensor` with all tensors sent to the proper device. )r rrXLArr rrs rgatherrsF~&&/*=*==6""  ( (,M M6"" robjectc *SSKn[5nURnURnUR 5n[ R "X5 UR5n[U5n[R"U/[RUS9n[U5V s/sH'n [R"S[RUS9PM) n n [RRX5 [![#SU 555n Sn X:a U S-n X:aM [R$R'U5n [R("U 5R+U5nUR-U 5 [R."X-[R0US9n[U5Vs/sHnXU-U US--PM nn[RRUU5 /n[3U5Hunn[!U UR555nUSUR75R95R;5n[ R<"U5n[?U[@5(aURCU5 MUREU5 M U$s sn fs snf)aGather picklable objects from all ranks with padded allgather sizes. On Neuron devices every unique tensor shape triggers a new NEFF compilation. The standard ``all_gather_object`` sizes its byte tensor to the exact max pickle size, which varies per call and causes unbounded compilation cache growth. This variant rounds the allgather size up to the next power of 2 so the number of distinct compiled shapes stays bounded to O(log(max_size)). rNrr c3@# UHoR5v M g7fr)item)rBss rrG(_neuron_gather_object..s3Avvxxs)#ior rWrBytesIOpickledumpgetvaluertrrlongrzerosrrrZmax ByteStorage _from_bufferr"rOresize_rluint8rrr}numpytobytesloadsrr7extendappend)rrrrW group_sizebuf raw_bytes local_sizelocal_size_tensorr size_listmax_size padded_size byte_storage input_tensor coalescedrrresultrobj_size obj_bytesr8s r_neuron_gather_objectrs NE \\F$$J **,C KK IYJ j\FSJOPZJ[\J[QQejj@J[I\   >33334HK     $$11)F~. 6y|((*+9H%))+113;;= ll9% c4 MM#  MM# / M;] es .J Jc[[5R5Vs/sHnSPM nn[RR X 5 UVVs/sH o3HoDPM M snn$s snfs snnfr)rr rrrall_gather_object)rroutput_objectsyxs r_gpu_gather_objectrsa$),.*F*F$GH$Gqd$GNH ''?% 1~!q!AqA~ 11I 2s A*A/c[5R[R:Xa [ S5e[5R[ ;a8[5R RS:Xa [U5$[U5$U$)a Recursively gather object in a nested list/tuple/dictionary of objects from all devices. Args: object (nested list/tuple/dictionary of picklable object): The data to gather. Returns: The same data structure as `object` with all the objects sent to every device. z&gather objects in TPU is not supportedneuron) r rrrNotImplementedErrorr rWr6rr)rs rrrsh~&&/*=*==!"JKK  ( (,M M > % % 1(0 0!&)) rc"SSjn[X SUS9$)Nc@[RRXS9 U$)Nsrc)rr broadcast)rrs r_gpu_broadcast_one*_gpu_broadcast.._gpu_broadcast_ones ##F#4 rT)r=rrr@)r3rrs r_gpu_broadcastr s /4UX YYrc l^^[U[[45(a[UU4Sj[ U555$[U[ 5(aB[ U5"UR5VVs0sHup4U[UTSU3S9_M snn5$[R"TUU4Sj5$s snnf)Nc3F># UHup[UTSU3S9v M g7f)rnameN)_tpu_broadcast)rBrrVr s rrG!_tpu_broadcast..s&"gUfTQ>!TF!A3-#HUfs!rr c>UT$rr)rrs r _tpu_broadcast..s !C&r) rr7r0r:rrr6rIrr mesh_reduce)rrr rLrMs `` rrrs&4-((&"gU^_eUf"ghh FG $ $F|RXR^R^R`aR`$!Qq$q} EER`abb >>$(8 99bs0B0  cSn[5n[R"U[RURS9nUbTUR n[ URn[R"[U5U/-[S9US[U5S-&[USS9nX3R5n[USSS 5nUSSnX64$) z] Grabs the shape of `tensor` only available on one process and returns a tensor of its shape rrNr`r sum reductionrr) r rrlrZrWr_TENSOR_TYPE_TO_INTr`rr7rtreducenonzero)rmax_tensor_dimensionr base_tensorr_ tensor_dtyper`s rgather_tensor_shaper'.s ! NE++2%))ELLYK  )&,,7 (- T%[L>5QY\(] $c%j1n%6K1134K  BC # $Ecr"K  rreturnc[5n[U5up#Uc5[R"U[US9R UR 5n[USS9$)at Copies a tensor that only exists on a single device and broadcasts it to other devices. Differs from `broadcast` as each worker doesn't need to know its shape when used (and tensor can be `None`) Args: tensor (`torch.tensor`): The tensor that should be sent to all devices. Must only have it be defined on a single device, the rest should be `None`. rrr)r r'rrTENSOR_INT_TO_DTYPErOrWr")rrr_r`s rcopy_tensor_to_devicesr+GsN NE&v.LE ~U*=e*DEHHV &E **r from_processc[5R[R:Xa [ XSS9$[5R[ ;a [ XS9$U$)a Recursively broadcast tensor in a nested list/tuple/dictionary of tensors to all devices. Args: tensor (nested list/tuple/dictionary of `torch.Tensor`): The data to gather. from_process (`int`, *optional*, defaults to 0): The process from which to send the data Returns: The same data structure as `tensor` with all tensors broadcasted to the proper device. zaccelerate.utils.broadcast)rr r)r rrrrr r )rr,s rrrXsI~&&/*=*==f=YZZ  ( (,M Mf77 rcSSKn[5nURnURU:XaBUR 5n[ R "X5 UR5n[U5nOSnSn[R"U/[RUS9n[RRU5 [UR55n Sn X:a U S-n X:aM URU:XaV[R R#U5n [R$"U 5R'U5n U R)U 5 O$[R*"U [R,US9n [RR/XS9 [ R0"U SU R35R55R755n [9[U55H nXX'M U$)aBroadcast a list of picklable objects with padded tensor sizes. On Neuron devices ``ProcessGroupNeuron.broadcast()`` is implemented as an allreduce, so every unique tensor shape triggers a new NEFF compilation. The standard ``broadcast_object_list`` sizes its byte tensor to the exact serialized length, which varies per call and causes unbounded compilation cache growth. This variant rounds the broadcast tensor size up to the next power of 2, bounding the number of distinct compiled shapes to ~O(log(max_size)). rNrrrr r)rr rW process_indexrrrrrtrrrr all_reducerZrrrr"rOrrrrrr}rrr) object_listr,rrrWrrr size_tensor real_sizerr data_tensorrrs r_neuron_broadcast_object_listr5ns NE \\F l*jjl K%LLN ^   ,, |5::fMK   -K$$&'IK  !   ! l*((55i@ &&|477? K(kk+U[[P   >\\+jy1557==?GGI JF 3{# $ % rc^[5R[R:Xa4[ U5H#up#[ R "SUU4Sj5X'M% U$[5R[;aL[5RRS:Xa [UTS9 U$[RRUTS9 U$)ar Broadcast a list of picklable objects from one process to the others. Args: object_list (list of picklable objects): The list of objects to broadcast. This list will be modified inplace. from_process (`int`, *optional*, defaults to 0): The process from which to send the data. Returns: The same list containing the objects from process 0. z&accelerate.utils.broadcast_object_listc>UT$rr)rr,s rr'broadcast_object_list..s efgsetrr)r,r)r rrrrrrr rWr6r5rrbroadcast_object_list)r1r,rr8s ` rr9r9s~&&/*=*== ,FA^^,TVY[tuKN-   ( (,M M > % % 1 )+L Q     3 3K\ 3 R rc Sn[X@U5$)a* Recursively takes a slice in a nested list/tuple/dictionary of tensors. Args: data (nested list/tuple/dictionary of `torch.Tensor`): The data to slice. tensor_slice (`slice`): The slice to take. Returns: The same data structure as `data` with all the tensors slices. c X$rr)r tensor_slices r _slice_tensor$slice_tensors.._slice_tensors ##rr@)r3r<r/rr=s r slice_tensorsr?s$ ], ??rc^^[TS[[45(a/[TSUU4Sj[ [ TS5555$[TS[ 5(aS[TS5"TSR5VVs0sH o"[TVs/sHo3UPM snTS9_M" snn5$[TS[R5(a[R"TTS9$[T[[45(a[ T5S:XaTS$[S[TS535es snfs snnf)a Recursively concatenate the tensors in a nested list/tuple/dictionary of lists of tensors with the same shape. If there is only a single batch of data, it is returned as-is. Args: data (nested list/tuple/dictionary of lists of tensors `torch.Tensor`): The data to concatenate. dim (`int`, *optional*, defaults to 0): The dimension on which to concatenate. Returns: The same data structure as `data` with all the tensors concatenated. rc3h># UH"n[TVs/sHo"UPM snTS9v M$ gs snf7f)rN) concatenate)rBrdr3rs rrGconcatenate..s/#lXkSTKt0Dt!1t0D#$NXk0Ds 2- 2rr z%Can only concatenate tensors but got )rr0r7r:rrtrr6rwrBrrrrJ)r3rrLrCs`` rrBrBs$q'E4=))$q'#lX]^abfghbi^jXk#lmm DGW % %DG}UYZ[U\UaUaUcdUcPQD-ADqdD-As!KKUcdee DGU\\ * *yy3'' D5$- ( (SY!^Aw?T!W OPP .Bds E D;) E ;E c\rSrSrSrg)CannotPadNestedTensorWarningirN)rKrrrrrrrrFrFsrrFc $SSjn[X@SXUS9$)a Recursively pad the tensors in a nested list/tuple/dictionary of tensors from all devices to the same size so they can safely be gathered. Args: tensor (nested list/tuple/dictionary of `torch.Tensor`): The data to gather. dim (`int`, *optional*, defaults to 0): The dimension on which to pad. pad_index (`int`, *optional*, defaults to 0): The value with which to pad. pad_first (`bool`, *optional*, defaults to `False`): Whether to pad at the beginning or the end. c^^ ^ [USS5(a[R"S[5 U$T[ UR 5:dT[ UR 5*:aU$TS:aT[ UR 5- m[ R"UR URS9Sn[U5R5n[U4SjU55m T UR T:XaU$UR m [T 5nT UT'UR[U55U-nU(a*[UU U 4Sj[[ U5555nO([UU 4Sj[[ U5555nXU'U$) N is_nestedFzHCannot pad nested tensors without more information. Leaving unprocessed.r)rWc3,># UH oTv M g7frr)rBrrs rrGFpad_across_processes.._pad_across_processes..s-u!vusc3h># UH'oT:Xa[TTT- T5O [S5v M) g7frslice)rBrrrold_sizes rrGrKs4`t[\#Xh#.95QU;V`ts/2c3b># UH$oT:Xa[STT5O [S5v M& g7frNrMrBrrrOs rrGrKs,oZnUVCxE!Xc]3U4[PZn,/)getattrwarningswarnrFrtr_rrrWrr}rr7 new_zerosr0r) rr pad_index pad_firstrsizesnew_size new_tensorindicesrrOs ` @@r_pad_across_processes3pad_across_processes.._pad_across_processessL 6; . . MMZ, M #fll# #sc&,,.?-?'?M 7 3v||$ $C||FLL?Et   "-u-- v||C( (M<<>  %%eHo6B `efijrfs`tGoZ_`cdl`mZnooG$7rT)r=rrXrYrrFr@)rrrXrYr^s rpad_across_processesras!" D 4Sir rc (SSjn[UUSUUUS9$)z Takes a `tensor` of arbitrary size and pads it so that it can work given `num_processes` needed dimensions. New tensors are just the last input repeated. E.g.: Tensor: ([3,4,4]) Num processes: 4 Expected result shape: ([4,4,4]) c<^^ X-nXU-- nX-S:XaX!- nOX!U-- nXVU-s=:aS:aO OXV- nURm [T 5nX-US'UR[U55n[UU 4Sj[ [ U5555n XU 'U$)Nrr c3b># UH$oT:Xa[STT5O [S5v M& g7frQrMrRs rrG@pad_input_tensors.._pad_input_tensors..?s,kVjQR8a#/tLVjrS)r_r7rWr0rrt) r batch_sizerr remainder last_inputsto_padr[r\r]rOs ` @r_pad_input_tensors-pad_input_tensors.._pad_input_tensors0s/  $=>  &! +"/F"M&ABF & ,1 , )F<<> ) %%eHo6 kV[\_`h\iVjkk$7rT)r=rfrrrr@)rrfrrrjs rpad_input_tensorsrl%s(&  #   rc"SSjn[X0SXS9$)a. Recursively reduce the tensors in a nested list/tuple/dictionary of lists of tensors across all processes by the mean of a given operation. Args: tensor (nested list/tuple/dictionary of `torch.Tensor`): The data to reduce. reduction (`str`, *optional*, defaults to `"mean"`): A reduction method. Can be of "mean", "sum", "max", or "none" scale (`float`, *optional*): A default scaling value to be applied after the reduce, only valid on XLA. Returns: The same data structure as `data` with all the tensors reduced. cv[5nUR5nUR[R:XaU$UR[R :Xai[ R"5 US:Xa[ RO[ Rn[ R"XT/U5 [ R"5 OcURR[;aEUS:Xa[RO[Rn[ R"RXF5 US:XaXCR$-nU$)Nrmean)r rrrrrrr REDUCE_MAX REDUCE_SUMr0valuer rMAXSUMrrr)rr scaler cloned_tensorxla_optorch_ops r_reduce_across_processes(reduce.._reduce_across_processes_s   ! !_%7%7 7  ! !_%8%8 8 LLN&/5&8R]]bmmF MM&/5 9 LLN  # # ) )-N N'0E'9x||x||H    ( ( A   00 0MrT)r=r rurog?r@)rr rurys rr"r"Ms$*  di rc"SnSn[XUS9$)aZ Recursively converts the elements nested list/tuple/dictionary of tensors in FP16/BF16 precision to FP32. Args: tensor (nested list/tuple/dictionary of `torch.Tensor`): The data to convert from FP16/BF16 to FP32. Returns: The same data structure as `tensor` with all tensors that were in FP16/BF16 precision converted to FP32. c"UR5$r)floatrs r_convert_to_fp32)convert_to_fp32.._convert_to_fp32s||~rc[U5=(d [US5=(a- UR[R[R 4;$rk)rr1r`rfloat16r~rs r_is_fp16_bf16_tensor-convert_to_fp32.._is_fp16_bf16_tensors@'C767+C  MM NNZ J  rror@)rrrs rconvert_to_fp32rys -AU VVrc*\rSrSrSrSrSrSrSrg)ConvertOutputsToFp32iaE Decorator to apply to a function outputting tensors (like a model forward pass) that ensures the outputs in FP16 precision will be convert back to FP32. Args: model_forward (`Callable`): The function which outputs we want to treat. Returns: The same function as `model_forward` but with converted outputs. c&Xl[X5 gr) model_forwardr)selfrs r__init__ConvertOutputsToFp32.__init__s*t+rc8[UR"U0UD65$r)rr)rrDrFs r__call__ConvertOutputsToFp32.__call__st114B6BCCrc.[R"S5e)NzCannot pickle a prepared model with automatic mixed precision, please unwrap the model with `Accelerator.unwrap_model(model)` before pickling it.)r PicklingError)rs r __getstate__!ConvertOutputsToFp32.__getstate__s"" `  r)rN) rKrrrrrrrrrrrrrs ,D rrc8^[T5mU4SjnTUlU$)Nc>T"U0UD6$rr)rDrFrs rforward(convert_outputs_to_fp32..forwardsd-f--r)r __wrapped__)rrs` rconvert_outputs_to_fp32rs!(7M.(G NrcF[U[5(a*UR5Hn[U5nUcMUs $ g[U[[ 45(aUHn[U5nUcMUs $ g[U[ R5(a UR$g)z Finds the device on which a nested dict/list/tuple of tensors lies (assuming they are all on the same device). Args: (nested list/tuple/dictionary of `torch.Tensor`): The data we want to know the device of. N) rrvaluesrr0r7rrrW)r3r8rWs rrrs$  ;;=C %F! ! D5$- ( (C %F!  D%,, ' '{{ (rc#b# [5R[R:wd<[5Rb2[5RR 5(d [ 5nOSSKnURRXX#S9nU Sv SSS5 g!,(df  g=f7f)z| Wrapper around `deepspeed.runtime.zero.GatheredParameters`, but if Zero-3 is not enabled, will be a no-op context manager. Nr) modifier_rank fwd_moduleenabled) r rr DEEPSPEEDdeepspeed_pluginis_zero3_init_enabledr deepspeedzeroGatheredParameters)paramsrrrgather_param_contextrs rrrs**o.G.GG++7 "33IIKK*}(~~@@ J A      sBB/B B/ B,(B/)FNr)rzbroadcast tensorr)NNr`r{)NNT)[rrrUcollections.abcr contextlibrr functoolsrrtypingrrrr r constantsr dataclassesrrimportsrrrversionsrtorch_xla.core.xla_modelcore xla_modelrtorch.distributedrrr(r,r4r:rArUrcrhrqrvryrrr Exceptionrrrrrrrr rr~doublehalfr~rint8int16int32int64boolr!rIr*r'rr+rZrr5r9r?rB UserWarningrFrarlr"rrrrr)rLrMs00rrs[#2+ 28; '))!##*, 6]$4CX]0f7t8$/$ b."5. $PN I  F&&3#3l2s2#*Z: KK LL! JJ NNA KK JJ KK KK KK JJ );(@(@(BC(Bqt(BC2+5<<+"C*2S2jS0@(Q4 ; 44n%P((VW0  4 *E Ds1H