3jG]SrSSKrSSKJr SSKJr SSKJrJrJ r J r J r J r J r SSKrSSKJrJr SSKJrJr SS KJrJr SS KJr SS KJr "S S 5rSSSSSSSSSSSSSSSSSSSSSSSSSS\\SSSSSSSSSSS\R:\R<"S5SSS4,S\ \ \ \\4\4S\ \ \S\ \ \ S\ \S 4S!\S"\S#\!S$\ \S%\!S&\ S'\"S(\S)\!S*\ \"S+\ \ \ \ 4S,\ \ \ \ 4S-\ S.\ S/\ S0\ \ S1\ S2\ S3\ \"S4\S5\S6\"S7\ \ S 4S8\ \ S 4S9\ \ S:\ \"S;\ \S<\S=\!S>\S?\S@\SA\SB\!SC\!SD\RFSE\ \"\R<4SF\!SG\"SH\!SI\ \RHRJRL\44ZSJjjr'g)KaNaFlex data loader for dynamic sequence length training. This module provides a specialized data loader for Vision Transformer models that supports: - Dynamic sequence length sampling during training for improved efficiency - Variable patch size training with probabilistic selection - Patch-level random erasing augmentation - Efficient GPU prefetching with normalization Hacked together by / Copyright 2025, Ross Wightman, Hugging Face N)suppress)partial)CallableDictIteratorListOptionalTupleUnion)IMAGENET_DEFAULT_MEANIMAGENET_DEFAULT_STD) _worker_init adapt_to_chs)NaFlexMapDatasetWrapperNaFlexCollator)PatchRandomErasing)create_transformc\rSrSrSr\\S\R"S5SSSSS S 4 S \RRRS \ \ S 4S\ \ S 4S\S\RS\\R S\ S\S\S\S\SS4SjjrS\\ \\\R,4\R,44SjrS\4Sjr\S5r\S5rSrg)NaFlexPrefetchLoaderz;Data prefetcher for NaFlex format which normalizes patches.cudaNconstr rTloadermean.stdchannelsdevice img_dtypere_probre_modere_count re_num_splitspatchify_channels_lastreturnc XlXPlU=(d [RUl[ X$5n[ X45nX@lXlU (aSSU4OSUS4n [R"UV s/sHoS-PM sn XPRS9RU 5Ul [R"UV s/sHoS-PM sn XPRS9RU 5Ul US:a[UUU U US9Ul OSUl URS:H=(a [RR!5UlURS:H=(a [R$R!5Ulgs sn fs sn f) a Initialize NaFlexPrefetchLoader. Args: loader: DataLoader to prefetch from. mean: Mean values for normalization. std: Standard deviation values for normalization. channels: Number of image channels. device: Device to move tensors to. img_dtype: Data type for image tensors. re_prob: Random erasing probability. re_mode: Random erasing mode. re_count: Maximum number of erasing rectangles. re_num_splits: Number of augmentation splits. patchify_channels_last: Per-patch flat layout produced by the upstream Patchify. ``True`` (default): channel index is the innermost (P-P-C). ``False``: channel index is the outermost (C-P-P). Controls how ``patches`` tensors are viewed for normalization and erasing. r )r dtyper) erase_probmode max_count num_splitsr Nrnpu)rr torchfloat32r!rrr&tensorviewrrrrandom_erasingtyper is_availableis_cudar/is_npu)selfrrrrr r!r"r#r$r%r&normalization_shapexs Q/home/wildlama/miniconda3/lib/python3.13/site-packages/timm/data/naflex_loader.py__init__NaFlexPrefetchLoader.__init__sIB  "3emm D+3) &<#2Hq!X.qRZ\]N^LL" #dWd #F..JJN$ObJc <<! "cWc "6IIMNaIb  R<"4""( #D #'D {{f,J1H1H1J kkU*Guyy/E/E/G # $ "s 2E=8Fc ## SnUR(aJ[RRURS9n[ [RR US9nOcUR(aJ[RRURS9n[ [RR US9nOSn[nURGHhupEU"5 UR5HYupg[U[R5(dM&US:Xa UROSnXFRURSUS9XF'M[ URURSS9nUSn U R n U R"S:XaRU upn UR$(aU R'XS UR(5nOU R'XUR(S 5nOU R"S :XaU SS upUR$(aKU S UR(:XdS UR(S U S 35eU R'XS UR(5nOdU S UR(:XdS UR(S U S 35eU R'XUR(S 5nO[+SU R"S35eUR-UR.5R1UR25nUR4bUR$(d UR7SS 5R95nUR5UUSUR;SS5S9nUR$(d UR7SS 5R95nUR'U 5US'SSS5 U(dWW4v OSnUbUR(a7[RR=URS9R?U5 OGUR(a6[RR=URS9R?U5 UnUnGMk WW4v g!,(df  N=f7f)zIterate through the loader with prefetching and normalization. Yields: Tuple of (input_dict, targets) with normalized patches. T)r )streamNpatches)r non_blockingr*)r rBrz Expected z channels, got z&Unexpected patches tensor dimensions: z. Expected 3 or 5. patch_coord patch_valid)rGrHF) r7r0rStreamr rr@r8r/rritems isinstanceTensorr!toshapendimr&r3r ValueErrorsubrdivrr4 transpose contiguousgetcurrent_stream wait_stream)r9firstr@stream_contextnext_input_dict next_targetkvr*patches_tensororiginal_shape batch_size num_patches_rA input_dicttargets r<__iter__NaFlexPrefetchLoader.__iter__`s  <<ZZ&&dkk&:F$UZZ%6%6vFN [[YY%%T[[%9F$UYY%5%5fENF%N,0KK (O!+113DA!!U\\2223y.d-<-?-B-B#';;)-"'.C.*4*nnDKKdnS "1!;!/!5!5!&&!+1?.JQ22"0"5"5jrSWS`S`"a#1"5"5jt}}^`"a#((A-.&&2 66")"3"3B";"F"F"H"11$3M$B$3$7$7 t$L2G  66")"3"3B";"F"F"H.5\\.-I *{"~ &((!<<JJ--T[[-AMMfU[[II,,DKK,@LLVT(J FY-8\&  ["!s&CQ3QI6Q B:Q Q Qc,[UR5$)zPGet length of underlying loader. Returns: Number of batches in the loader. )lenrr9s r<__len__NaFlexPrefetchLoader.__len__s 4;;c.URR$)zZGet sampler from underlying loader. Returns: Sampler from the underlying DataLoader. )rsamplerris r<rnNaFlexPrefetchLoader.sampler{{"""rlc.URR$)zZGet dataset from underlying loader. Returns: Dataset from the underlying DataLoader. )rdatasetris r<rrNaFlexPrefetchLoader.datasetrprl) rr r!r7r8rrr&r4r)__name__ __module__ __qualname____firstlineno____doc__r rr0r utilsdata DataLoaderr floatintr r*strboolr=rrrLrerjpropertyrnrr__static_attributes__rlr<rrsWE '<%9#(<<#7/3"!"+/@HKK$$//@Hs #@Hucz" @H  @H LL @H  ,@H@H@H@H@H%)@H@HD_!(5c5<<.?)@%,,)N#OP_!B  ####rlr)@iir Frrg?g?bilinear*Trall patch_sizepatch_size_choicespatch_size_choice_probstrain_seq_lens. max_seq_lenr` is_trainingmixup_fnno_augr"r#r$re_splittrain_crop_modescaleratiohflipvflip color_jittercolor_jitter_probgrayscale_probgaussian_blur_prob auto_augmentnum_aug_repeatsnum_aug_splits interpolationrrcrop_pct crop_modecrop_border_pixels num_workers distributedrank world_sizeseedepochuse_prefetcher pin_memoryr!r persistent_workersworker_seedingr&r'c-U(GaUS:XdS5e[[40SS_SU _SU_SU_SU_S U_S U_S U_S U_S U_SU_SU_SU_SU_SU_SU_SU_SU_SU _SU _SU _SU&_SS_SU,_6n-[U5n.UU.-n/[U[R R R5(aS5e[UU-UUUUU/UU$U!U"U#SU%U,S9n0[R R RU0SSU SU'[[U+S 9U*S!9n1U&(a[U1UUU(U)U U U U,S"9 n1U1$[SUUUU&SUUSU,S#9 Ul [US$9n2Sn3U!(aA[U[R R R5(dSS%KJn4 U4"U5n3[R R RUUSU U3U2U'SS&9n1U&(a[U1UUU(U)U,S'9n1U1$)(u Create a data loader with dynamic sequence length sampling for training. Args: dataset: Dataset to load from. patch_size: Single patch size to use. patch_size_choices: List of patch sizes for variable patch size training. patch_size_choice_probs: Probabilities for each patch size choice. train_seq_lens: Training sequence lengths for dynamic batching. max_seq_len: Fixed sequence length for validation. batch_size: Batch size for validation and max training sequence length. is_training: Whether this is for training (enables dynamic batching). mixup_fn: Optional mixup function. no_aug: Disable augmentation. re_prob: Random erasing probability. re_mode: Random erasing mode. re_count: Maximum number of erasing rectangles. re_split: Random erasing split flag. train_crop_mode: Training crop mode. scale: Scale range for random resize crop. ratio: Aspect ratio range for random resize crop. hflip: Horizontal flip probability. vflip: Vertical flip probability. color_jitter: Color jitter factor. color_jitter_prob: Color jitter probability. grayscale_prob: Grayscale conversion probability. gaussian_blur_prob: Gaussian blur probability. auto_augment: AutoAugment policy. num_aug_repeats: Number of augmentation repeats. num_aug_splits: Number of augmentation splits. interpolation: Interpolation method. mean: Normalization mean values. std: Normalization standard deviation values. crop_pct: Crop percentage for validation. crop_mode: Crop mode. crop_border_pixels: Crop border pixels. num_workers: Number of data loading workers. distributed: Whether using distributed training. rank: Process rank for distributed training. world_size: Total number of processes. seed: Random seed. epoch: Starting epoch. use_prefetcher: Whether to use prefetching. pin_memory: Whether to pin memory. img_dtype: Image data type. device: Device to move tensors to. persistent_workers: Whether to use persistent workers. worker_seeding: Worker seeding mode. patchify_channels_last: Per-patch flat layout. ``True`` (default): channel index varies fastest (NaFlex default). ``False``: C-P-P (channels first — Gemma4 / HF native layout). Forwarded to the upstream ``Patchify`` and to the prefetcher's normalization layout. Returns: DataLoader or NaFlexPrefetchLoader instance. rz=Augmentation repeats not currently supported in NaFlex loaderrTrrrrrrrrrrrrrrrrrr"r#r$rnaflexr&Fz IterableDataset Wrapper is a WIP)transform_factoryrrrseq_lensmax_tokens_per_batchrrrrrshufflerr&N)r)r`rrrnrworker_init_fnr)rrr!r r"r#r$r&) rrrrrrrrpatchifyr&)r)OrderedDistributedSampler)r`rrrn collate_fnr drop_last)rrr!r r&)rrmaxrKr0ryrzIterableDatasetrr{rr transformrtimm.data.distributed_samplerr)5rrrrrrrr`rrrr"r#r$rrrrrrrrrrrrrrrrrrrrrrrrrrrr!r rrr&rmax_train_seq_lenrnaflex_datasetrrrnrs5 r<create_naflex_loaderrsR!#d%dd##    ,        & 0 *  2 & (  ! "# $ % & 2' () *+ ,- .*/ 01 2$:3 8 /),== gu{{//?? @ @ << <50 /!1$;#!5#!#9 &!!,, #!"<O1-   )#!'= Fv M[-')!##9  $ <  z'5;;3C3C3S3STT O/8G!!,, !#!!-   )#'= F Mrl)(rxmath contextlibr functoolsrtypingrrrrr r r r0 constantsr rrrrrrrnaflex_random_erasingrtransforms_factoryrrr1r r}r|rr~r*ryrzr{rrrlr<rs  III B.C50~#~#F=A269=*D!'+)-/3/3!-1 "$&&* '"7!5$(#',0!#!&+0<<+?#'#'+_iU5c?C#789i%T#Y/i"*$u+!6 i c3h i  iii8$iiiiii "#!i"eUl+,#i$eUl+,%i&'i()i*+i,$E?-i./i0"1i2sm3i45i67i89i:E3J;i<5#: =i>5/?i@C=AiB%SMCiFGiHIiJKiLMiNOiPQiRSiTUiV;;WiXc5<<'(YiZ![i\]i^!%_i` u{{**,@@ Aairl