3j u SrSSKrSSKJrJrJrJr SSKrSSKJ r SSK J s J r SSK JrJr SSKJr SSKJr \R("\5r"SS \ R.5rS \\\ R.\4S \ R.S \\S \\R4S\\R6S\4 Sjr"SS\5r"SS\ R.5r"SS\5rg)z5Knowledge distillation training tasks and components.N)DictOptionalTupleUnion) create_modelgroup_parameters) unwrap_model) TrainingTaskc^\rSrSrSrSS\\\R4S\ \ S\ S\ \S\ \ RS\ \ R4 U4S jjjrS \ RS \ R4S jrSS \S\ \S\S\S S4 SjjrSS \ RS\S \ R4SjjrSS \ RS\ \ RS\ \ RS \ R4SjjrSrU=r$)DistillationTeachera1Wrapper for a teacher model used in knowledge distillation. Creates and manages a pre-trained teacher model for knowledge distillation, handling model creation and normalization differences between teacher and student. Can be created from: - A model name string (creates the model internally with pretrained weights) - An existing nn.Module (wraps it with the necessary interface) Args: model_name_or_module: Either a model name string or an nn.Module num_classes: Number of output classes (required if model_name_or_module is a string) in_chans: Number of input channels (used if model_name_or_module is a string) pretrained_path: Optional path to pretrained weights (used if model_name_or_module is a string) device: Device to place the model on dtype: Model dtype (uses float32 if None) model_name_or_module num_classesin_chanspretrained_pathdevicedtypec $>[TU]5 [U[5(aC[R SUS35 SS0nU(a [ UUS9US'[SUUUUUS.UD6nOC[U[R5(aUnO![S[U5R35eUR5 Xl[U5n [!U S 5(a9U R"R%S S 5n U R"R%S S 5n OS n S n [&R("XUS9R+SSSS5n [&R("XUS9R+SSSS5n UR-SU SS9 UR-SU SS9 SUlg)NzCreating KD teacher model: '' pretrainedT)filerpretrained_cfg_overlay) model_namerrrrz8model_name_or_module must be a string or nn.Module, got pretrained_cfgmean)g ףp= ?gv/?gCl?std)gZd;O?gy&1?g?rrr mean_kdF persistentstd_kd)super__init__ isinstancestr_loggerinfodictrnnModule TypeErrortype__name__evalmodelr hasattrrgettorchtensorviewregister_buffer_compiled_forward_features)selfrrrrrrpretrained_kwargsr2model_unwrappedrrr r# __class__s P/home/wildlama/miniconda3/lib/python3.13/site-packages/timm/task/distillation.pyr&DistillationTeacher.__init__%s  *C 0 0 LL78L7MQO P!-t 4 >B( +?!":; !/'!  $ E,bii 8 8(EJ4PdKeKnKnJop    'u- ?$4 5 5"1155f>STD!0044UCCAr1aP YEB Xv%@*.'inputreturnc2[URS5(a[URS5(d-[SURRRS35eURR U5nURR USS9$)Nforward_features forward_headzModel zi does not support feature extraction. Ensure the model has 'forward_features' and 'forward_head' methods.T pre_logits)r3r2 ValueErrorr=r0rDrE)r:rA feature_maps r>_forward_features%DistillationTeacher._forward_features[stzz#566gdjjR`>a>a--6678VV jj11%8 zz&&{t&DDr@backendmode compile_modelcompile_featuresc U(a([R"UR4XS.UD6UlU(a)[R"UR4UUS.UD6UlU$)z;Compile teacher inference paths used by distillation tasks.rLrM)r5compiler2rJr9)r:rLrMrNrOcompile_kwargss r>rRDistillationTeacher.compileds_ tzz`7`Q_`DJ .3mm&&//! /D +  r@return_featurescU(a/URbURU5$URU5$URU5$)aForward pass through teacher model. Args: input: Input tensor (should already be normalized for teacher) return_features: Whether to return pooled pre-logits features instead of logits Returns: Logits or pooled pre-logits features depending on return_features flag )r9rJr2)r:rArUs r>forwardDistillationTeacher.forwardxsD ..:66u==))%0 0::e$ $r@ student_mean student_stdcUbUcU$[R"X R5(a'[R"X0R5(aU$X-U-UR- UR- $)a7Normalize input to match teacher's expected normalization. Args: input: Input tensor (already normalized for student) student_mean: Student normalization mean buffer [1, 3, 1, 1] student_std: Student normalization std buffer [1, 3, 1, 1] Returns: Input tensor normalized for the teacher model )r5equalr r#)r:rArYrZs r>normalize_input#DistillationTeacher.normalize_inputs`  ;#6L ;;|\\ 2 2u{{;P[P[7\7\L#l2T\\AT[[PPr@)r9r2)NNNN)inductorNTFF)NN)r0 __module__ __qualname____firstlineno____doc__rr(r,r-rintr5rrr&TensorrJboolrRrWr]__static_attributes__ __classcell__r=s@r>r r sp**.-1-1+/4/"'RYY"74/"#4/ 4/ &c] 4/ U\\* 4/EKK(4/4/lEu||E E&"&"&%* 3-  #  .%*%<<%"%  %04826 Q<<Q#5<<0Q"%,,/ Q  QQr@r teacher student_modelrrrrBc [U[5(aU$[U5nURnURn[UUUUUUS9$)anResolve teacher input to a DistillationTeacher instance. Args: teacher: Model name string, nn.Module, or DistillationTeacher student_model: Student model to infer num_classes/in_chans from pretrained_path: Optional path to teacher pretrained weights device: Device for teacher dtype: Dtype for teacher Returns: DistillationTeacher instance )rrrrrr)r'r r rr)rlrmrrrstudent_unwrappedrrs r>_resolve_teacherrpsY&'.//%]3#//K ))H $'  r@c^\rSrSrSrSS\R S\\\R \ 4S\ \R S\ \S\S\ \ S \ \ S \ S \ \ RS \ \ RS \4U4SjjjrSS\ \SS4SjjrSS\S\ \S\R 4SjjrS\ R(S\ R(S\\\ R(44SjrSrU=r$)LogitDistillationTaskaLogit-based knowledge distillation task. Performs distillation by matching student and teacher output logits using KL divergence with temperature scaling. Loss weighting supports two modes: 1. Independent weights: loss = task_loss_weight * task_loss + distill_loss_weight * distill_loss 2. Complementary mode: loss = task_loss_weight * task_loss + (1 - task_loss_weight) * distill_loss (used when only task_loss_weight is specified) Args: student_model: Student model to train teacher_model: Teacher model - can be a model name string, nn.Module, or DistillationTeacher criterion: Task loss function (default: CrossEntropyLoss) teacher_pretrained_path: Path to teacher pretrained weights (used when teacher_model is a string) loss_type: Type of distillation loss (currently only 'kl' supported) distill_loss_weight: Weight for distillation loss task_loss_weight: Weight for task loss temperature: Softmax temperature for distillation (typical values: 1-4) device: Device for task tensors/buffers dtype: Dtype for task tensors/buffers verbose: Enable info logging Example: >>> # With model name string (num_classes/in_chans inferred from student) >>> task = LogitDistillationTask( ... student_model=model, teacher_model='resnet50', ... criterion=nn.CrossEntropyLoss(), ... task_loss_weight=0.3, temperature=4.0, ... device=torch.device('cuda'), ... ) >>> # With raw model >>> task = LogitDistillationTask( ... student_model=model, teacher_model=my_teacher_model, ... criterion=nn.CrossEntropyLoss(), ... task_loss_weight=0.3, temperature=4.0, ... ) rm teacher_model criterionteacher_pretrained_path loss_typedistill_loss_weighttask_loss_weight temperaturerrverbosec >[TU]XU S9 [UUUURUR5n XlXlUbUO[R"5Ul XPl Xl US:wa[SUS35e[U5n [R"U R SURURS9R#SSSS5n[R"U R S URURS9R#SSSS5nUR%S US S 9 UR%S US S 9 UbIX`lUbUOSUlUR*(a%[,R/SUR(SU35 OUbGXplSU- UlUR*(a%[,R/SUSUR&35 ONSUlSUlUR*(a/[,R/SUR(SUR&35 UR*(a[,R/SUSU35 gg)Nrrr{klzUnsupported loss_type 'z$'. Currently only 'kl' is supported.rrr rrrYFr!rZ?z9LogitDistillationTask: Independent weights - task_weight=, distill_weight=z8LogitDistillationTask: Complementary mode - task_weight=z;LogitDistillationTask: Default equal weights - task_weight=z!LogitDistillationTask: loss_type=z, temperature=)r%r&rprrtrainable_modulerlr,CrossEntropyLossrurwrzrHr r5r6rr7r8rxryr{r)r*)r:rmrtrurvrwrxryrzrrr{rlrorYrZr=s r>r&LogitDistillationTask.__init__sV WE#   # KK JJ  !. &/&;ATATAV"&  6ykAefg g)7||  , ,V 4;;**  $q"a   ll  , ,U 3;;**  $q"a   ^\eL ]KEJ  *': $8H8T$4Z]D !|| ##'#8#8"99JK^J_a )$4 !'*-='=D $|| ##3"44EdF^F^E_a (+D $$'D !|| ##'#8#8"99J4KcKcJdf << LL3I;n[MZ  r@ device_idsrBc SSKJn URR5H nSUlM U"UR 4SU0UD6UlU$)aLPrepare task for distributed training. Wraps the student model in DistributedDataParallel (DDP) while leaving the frozen teacher model unwrapped. Args: device_ids: List of device IDs for DDP (e.g., [local_rank]) **ddp_kwargs: Additional arguments passed to DistributedDataParallel Returns: self (for method chaining) rDistributedDataParallelFrtorch.nn.parallelrrl parameters requires_gradrr:r ddp_kwargsDDPparams r>prepare_distributed)LogitDistillationTask.prepare_distributedDL" E\\,,.E"'E /!$D$9$9 _j _T^ _ r@rLrMc [R"UR4XS.UD6UlURUlURR"SUUSSS.UD6 UR$)z=Compile student eval/train forward and teacher logit forward.rQTFrLrMrNrOr$)r5rRr eval_modelrl)r:rLrMrSs r>rRLogitDistillationTask.compile]sn!& d.C.C rW rcq r//  "    $$$r@rAtargetc^URU5nURX25n[R"5 URR XR UR5nUR UR5SS9nSSS5 [R"X0R- SS9n[R"WUR- SS9n[R"XxSSS9URS --n URU-URU --n U UUU S .$!,(df  N=f) aqForward pass with logit distillation. Args: input: Input tensor [B, C, H, W] target: Target labels [B] Returns: Dictionary containing: - 'loss': Combined training loss (task + distillation) - 'output': Student logits (for metrics) - 'task_loss': Classification loss component - 'kd_loss': Logit distillation loss component FrUNr)dim batchmeanT) reduction log_targetlossoutput task_losskd_loss)rrur5no_gradrlr]rYrZdetachF log_softmaxrzkl_divryrx) r:rArstudent_logitsrinput_kdteacher_logitsprob_sprob_tr total_losss r>rWLogitDistillationTask.forwardps$..u5NN>: ]]_||33E;L;LdN^N^_H!\\(//*;U\SN~0@0@@bI~0@0@@bI((6[TRVZVfVfjkVkl**Y69Q9QT[9[[ $"   _s AD D,)rurxrrwryrlrzr) NNr~NNrNNTNr`N)r0rbrcrdrer,r-rr(r rfloatr5rrrhr&listrrRrgrrWrirjrks@r>rrrrss%V.259!3704!$-1+/ Q99Q!bii1D!DEQ  * Q &.c] Q  Q"*%Q'uoQQU\\*QEKK(QQQj*.  ! 6&"&%%3-%  %&$ <<$ LL$  c5<< $ $ r@rrc^\rSrSrSrS S\R S\\R 4U4SjjjrSr SS\ 4Sjjr S \ RS \\ R\ R44S jrS rU=r$)"FeatureDistillationTrainableModuleiaTrainable module for feature distillation. Wraps student model and projection layer into a single module where all trainable forward operations happen inside forward(). This ensures proper DDP wrapping when the module is used with DistributedDataParallel. rm projectionc:>[TU]5 XlX lg)zCreate trainable module wrapper for feature distillation. Args: student_model: Student model to train projection: Optional projection layer (Linear layer or None) N)r%r&studentr)r:rmrr=s r>r&+FeatureDistillationTrainableModule.__init__s $ $r@c[UR5n[US5(d [5$UR 5Vs1sHnSU-iM sn$s snf)Nno_weight_decaystudent.)r rr3setr)r:rnames r>r2FeatureDistillationTrainableModule.no_weight_decaysMt||,w 1225L.5.E.E.GH.Gd T!.GHHHsAcoarsec^^[UR5n[US5(d0$[X"R US9SS9m[ TR 5SS9mUU4SjnU$)N group_matcher)rT)reverser)defaultcp>URS5(aTRU[S5ST5$T$)Nr) startswithr4len)rstudent_layer_map task_layers r>_matcherBFeatureDistillationTrainableModule.group_matcher.._matchers7z**(,,T#j/2B-CZPP r@)r rr3rrmaxvalues)r:rrrrrs @@r>r0FeatureDistillationTrainableModule.group_matchersbt||,w00I,W6K6KSY6K6Zdhi*113Q?   r@rArBcURRU5nURRU5nURRUSS9nURbURU5nX44$)zForward pass through student and projection. Args: input: Input tensor [B, C, H, W] Returns: Tuple of (student_logits, student_features) where features are optionally projected to match teacher dimension. TrF)rrDrEr)r:rArIrstudent_featuress r>rW*FeatureDistillationTrainableModule.forwardsill33E: 22;?<<44[T4R ?? &#/?@ //r@)rrrra)r0rbrcrdrer,r-rr&rrhrr5rgrrWrirjrks@r>rrsv/3 %99 %!+ % %I D 0U\\0eELL%,,4N.O00r@rc*^\rSrSrSrS%S\R S\\\R \ 4S\ \R S\ \S\ \ S \ \ S \ \ S \ \ S \ \ RS \ \ RS\4U4Sjjjr\S\R S\ 4Sj5rS&S\ \SS4SjjrS'S\S\ \S\R 4SjjrS(S\ \R S\S\ \R 4U4SjjjrS(S\ \R S\S\\\\\ R2444SjjrS)S\4SjjrS*S\ \\\\\ R244S\S\ \R S\SS4 S jjrS!\ R2S"\ R2S\\\ R244S#jrS$rU=r$)+FeatureDistillationTaskiaFeature-based knowledge distillation task. Performs distillation by matching student and teacher intermediate features (pooled pre-logits) using MSE loss. Automatically creates a projection layer if student and teacher feature dimensions differ. Loss weighting supports two modes: 1. Independent weights: loss = task_loss_weight * task_loss + distill_loss_weight * distill_loss 2. Complementary mode: loss = task_loss_weight * task_loss + (1 - task_loss_weight) * distill_loss (used when only task_loss_weight is specified) Args: student_model: Student model to train teacher_model: Teacher model - can be a model name string, nn.Module, or DistillationTeacher criterion: Task loss function (default: CrossEntropyLoss) teacher_pretrained_path: Path to teacher pretrained weights (used when teacher_model is a string) distill_loss_weight: Weight for distillation loss task_loss_weight: Weight for task loss student_feature_dim: Student pre-logits dimension (auto-detected if None) teacher_feature_dim: Teacher pre-logits dimension (auto-detected if None) device: Device for task tensors/buffers dtype: Dtype for task tensors/buffers verbose: Enable info logging Example: >>> # With model name string (num_classes/in_chans inferred from student) >>> task = FeatureDistillationTask( ... student_model=model, teacher_model='resnet50', ... criterion=nn.CrossEntropyLoss(), ... distill_loss_weight=5.0, task_loss_weight=1.0, ... device=torch.device('cuda'), ... ) Nrmrtrurvrxrystudent_feature_dimteacher_feature_dimrrr{c >[TU]XU S9 [UUUURUR5n XlUbUO[ R"5UlUbIXPl UbUOSUl UR(a%[RSURSU35 OUbGX`l SU- Ul UR(a%[RSUSUR35 ONSUl SUl UR(a/[RSURSUR35 UcURU5nUcURU R5nSn Xx:waWUR(a[RSUSU35 [ R "XxURURS 9n O&UR(a[RS 5 [#X5Ul['U5n[(R*"UR,S URURS 9R/S S S S 5n[(R*"UR,SURURS 9R/S S S S 5nUR1SUSS9 UR1SUSS9 UR(a[RSUSU35 gg)Nr}rz;FeatureDistillationTask: Independent weights - task_weight=rz:FeatureDistillationTask: Complementary mode - task_weight=z=FeatureDistillationTask: Default equal weights - task_weight=zCreating projection layer: z -> rz.Feature dimensions match, no projection neededrr rrrYFr!rZz%FeatureDistillationTask: student_dim=z, teacher_dim=)r%r&rprrrlr,rrurxryr{r)r*_detect_feature_dimr2Linearrrr r5r6rr7r8)r:rmrtrurvrxryrrrrr{rlrrorYrZr=s r>r& FeatureDistillationTask.__init__s WE#   # KK JJ   &/&;ATATAV  *': $8H8T$4Z]D !|| ##'#8#8"99JK^J_a )$4 !'*-='=D $|| ##3"44EdF^F^E_a (+D $$'D !|| ##'#8#8"99J4KcKcJdf  &"&":":="I   &"&":":7=="I   5|| 12E1FdK^J_`#6TXT_T_gkgqgqrJ|| MN B= ])7||  , ,V 4;;**  $q"a   ll  , ,U 3;;**  $q"a   ^\eL ]KEJ << LL23>BUAVX  r@r2rBc[U5n[US5(a UR$[US5(a UR$[ S5e)z)Auto-detect feature dimension from model.head_hidden_size num_featureszCannot auto-detect feature dimension. Model must have 'head_hidden_size' or 'num_features' attribute, or you must specify student_feature_dim and teacher_feature_dim explicitly.)r r3rrrH)r2s r>r+FeatureDistillationTask._detect_feature_dim\sTU# 5, - -)) ) UN + +%% %R r@rc SSKJn URR5H nSUlM U"UR 4SU0UD6UlU$)afPrepare task for distributed training. Wraps the trainable module (student + projection) in DistributedDataParallel (DDP) while leaving the frozen teacher model unwrapped. Args: device_ids: List of device IDs for DDP (e.g., [local_rank]) **ddp_kwargs: Additional arguments passed to DistributedDataParallel Returns: self (for method chaining) rrFrrrs r>r+FeatureDistillationTask.prepare_distributedlrr@rLrMc [R"URR4XS.UD6nX@l[R"UR4UUS.UD6UlUR R"SUUSSS.UD6 U$)z9Compile feature-distillation train and eval entry points.rQFTrr$)r5rRrrrrl)r:rLrMrSrs r>rRFeatureDistillationTask.compiles]]4#8#8#@#@o'o`no $ %  ! !! !  !   !    r@moduleemac>[TU]XS9nUnUbF[U[5(a UR$[ US5(dU$UR nUbMFU$)N)rrr)r%get_eval_modelr'rrr3r)r:rrresolvedcurrentr=s r>r&FeatureDistillationTask.get_eval_modelsg7))A!'#EFF&7H--nnG ! r@cUbUOURUS9nUc0$[U5nURc0$SURR50$)Nrr)get_trainable_moduler r state_dict)r:rr trainables r>get_task_state&FeatureDistillationTask.get_task_states^ "-43L3LQT3L3U >I (    'Ii22==?@@r@ exclude_headc[UR5n[URR 55nU(aUSSnUR b)UR UR R 55 U$)N)r rrrrrextend)r:rrrs r>get_clip_parameters+FeatureDistillationTask.get_clip_parameterssh !6!67 )++6689 #CRJ    +   i22==? @r@statestrictc.U(dgUbUOURUS9nUcU(a [S5e[S5e[U5nURS5nUcgURcU(a [S5egURR XbS9 g)Nrz.Cannot load EMA task state before setup_ema().z2Cannot load task state without a trainable module.rz@Checkpoint has projection task state but task has no projection.)r)r RuntimeErrorr r4rload_state_dict)r:rrrrrprojection_states r>load_task_state'FeatureDistillationTask.load_task_states !-43L3LQT3L3U >"#STTST T (  99\2  #     '"#eff ,,-=,Mr@rArcURU5up4URX25n[R"5 URR XR UR5nUR UR5SS9nSSS5 [R"UW5nURU-URU--n U UUUS.$!,(df  NK=f)auForward pass with feature distillation. Args: input: Input tensor [B, C, H, W] target: Target labels [B] Returns: Dictionary containing: - 'loss': Combined training loss (task + distillation) - 'output': Student logits (for metrics) - 'task_loss': Classification loss component - 'kd_loss': Feature distillation loss component TrNr) rrur5rrlr]rYrZrrmse_lossryrx) r:rArrrrrteacher_featuresrrs r>rWFeatureDistillationTask.forwards$,0+@+@+G(NN>: ]]_||33E;L;LdN^N^_H#||HOO,=t|T **-/?@**Y69Q9QT[9[[ $"   _s AC C)rurxrryrlr) NNNNNNNNTrr)NFra)TNF) r0rbrcrdrer,r-rr(r rrrfr5rrrhr& staticmethodrrrrRrrrgrrrrWrirjrks@r>rrs L.25937041515-1+/ `99`!bii1D!DE`  * ` &.c] ` "*% `'uo`"*#`"*#`U\\*`EKK(```D 299   "*.  # 6&"&3-  0 Xbii%8 d W_`b`i`iWj  +/ ARYY' A A c4U\\)** + A *. NDd3 +<&=!=>?NNRYY' N  N  N0! <<! LL!  c5<< ! ! r@r) reloggingtypingrrrrr5torch.nnr,torch.nn.functional functionalr timm.modelsrr timm.utilsr taskr getLoggerr0r)r-r r(rrrprrrrr$r@r>r s;// 6#   H %OQ"))OQd"sBII'::;"yy""#"& "  $ "  "JK LK \=0=0@` l` r@