+jCSSKJr SSKrSSKrSSKJr SSKJr SSKJ r SSK J r J r \(aSSK Jr SSKJrJr SSKrSSKrSSKr"S S \ R,5rSS jrSS jrS rSSjrSSSjjrSSjr"SS\ R,5r"SS\ R,5r"SS\ R,5r "SS\ R,5r!"SS\ 5r"S Sjr#g)!) annotationsN)cleandoc) TYPE_CHECKING)override)ComfyExtensionio) ModelPatcher)CLIPVAEc<\rSrSrSr\S5r\SSj5rSrg)MultiGPUCFGSplitNodez Prepares model to have sampling accelerated via splitting work units. Should be placed after nodes that modify the model object itself, such as compile or attention-switch nodes. Other than those exceptions, this node can be placed in any order. c[R"SSS[UR5[RR S5[R R SSSSS9/[RR5/S 9$) NMultiGPU_WorkUnitszMultiGPU CFG Splitadvanced/multigpumodelmax_gpusdefaultminstepnode_id display_namecategory descriptioninputsoutputs)rSchemar__doc__ModelInputIntOutputclss ;/home/wildlama/comfy/ComfyUI/comfy_extras/nodes_multigpu.py define_schema"MultiGPUCFGSplitNode.define_schemasjyy(-( -w' Z B !  cj[RRXSS9n[R"U5$)NT) reuse_loaded)comfymultigpucreate_multigpu_deepclonesr NodeOutput)r(rrs r)executeMultiGPUCFGSplitNode.execute-s+99%X\9]}}U##r,N)rr rintreturn io.NodeOutput __name__ __module__ __qualname____firstlineno__r" classmethodr*r3__static_attributes__r5r,r)r r s/    $$r,r c UR5n[RRX!5nUcg[R "SUSUSUS35 UR U5 g)zJCast compute dtype to one the device supports; no-op if already supported.NzSelect Model Device: using z compute dtype on z (model weight dtype was z).) model_dtyper/model_managementunet_manual_castlogginginfoset_model_compute_dtype)patcherdevice weight_dtype cast_dtypes r)_force_supported_compute_dtyperK3sd&&(L''88NJ LL.zl:LVHTmnzm{{}~ ##J/r,c[URS5(d7URURlURURlgg)aQStash the original load/offload device on the underlying model. Stored on patcher.model (which is shared with the input patcher), so later "default" selections can recover the loader's original routing. Only the first Select on a given chain writes these attrs; subsequent deepclones inherit them onto their freshly-loaded model below. _select_base_load_deviceN)hasattrr load_devicerMoffload_device_select_base_offload_device)rGs r)_remember_base_devicesrR=s? 7=="< = =181D1D .4;4J4J 1 >r,c[US5(a5[US5(d#URUlURUlggg)zICarry the loader-original device attrs onto the freshly-deepcloned model.rMN)rNrMrQ) src_model dst_models r)_propagate_base_devicesrVJsBy455giQk>l>l-6-O-O *090U0U -?m5r,cURU:XaX lU$URnURUS9nX l[ X0R5 [ US5(aUR UR5 U$)a=Return a patcher whose actual model weights live on *target_load_device*. If *patcher* is already on *target_load_device* we just retarget the (already-cloned) patcher's metadata in place. Otherwise we call :meth:`ModelPatcher.deepclone_multigpu` to spawn a fresh model from the loader's ``cached_patcher_init`` factory -- the only safe way to move weights that may already be partially loaded onto another device. NOTE: reusing the input patcher's model when the requested device matches its current load_device is a deliberate fast path. Anything that has already mutated the original model (e.g. a prior KSampler invocation on the same model) will be observed here. This is by design and documented on the SelectXDeviceNode docstrings -- placing Select X Device after a node that consumes the same model is not recommended. )new_load_deviceregister_load_device)rOrPrdeepclone_multigpurVrNrY)rGtarget_load_devicetarget_offload_devicerTs r)_retarget_patcherr]Qst"00!6 I((9K(LG2I}}5w.//$$W%8%89 Nr,c4[U5 URRnUbUOURRnUc [ XU5$UR S:Xa2UR 5(aURSS9nXlXl U$[ XU5$)aResolve the requested device and produce a patcher routed there. For "default" we restore the loader's original load/offload pair. For CPU we pin both load and offload to CPU (and, on a dynamic patcher, downgrade to a plain ModelPatcher so the dynamic-only code paths are bypassed). For an explicit GPU we keep the loader's original offload but target the requested load device; if that differs from the current load device the patcher is deepcloned onto the new device. cpuT)disable_dynamic) rRrrMrQr]type is_dynamicclonerOrP)rGresolvedbase_offload_override base_load base_offloads r)_apply_patcher_devicerhos7# 66I,A,M(SZS`S`S|S|L \BB}}      mmDm9G&!) W ==r,cTURS5nU(dgUVs/sHo3RU:wdMUPM nn[U5[U5:waO[R"SUS35 UR SU5 [ US5(aUR5 gggs snf)a9Drop any multigpu clone whose load_device matches *primary_device*. Without pruning, MultiGPU CFG Split would have stacked a clone on the same device the primary now occupies (i.e. the workflow places MultiGPU CFG Split before Select Model Device). Keeps the clone set consistent with the new primary placement. r0Nz/Select Model Device: pruning MultiGPU clone on z* that now collides with the primary model.match_multigpu_clones)get_additional_models_with_keyrOlenrDrEset_additional_modelsrNrj)rprimary_devicemultigpu_modelsmfiltereds r)_prune_multigpu_collisionrrs:::FO *N?amm~.M?HN 8}O,, F~FVWAB C ##J9 51 2 2  ' ' ) 3-Os B%B%cT\rSrSrSr\S5r\SSj5r\SS Sjj5rSr g) SelectModelDeviceNodea Place the diffusion model on a specific device (default / cpu / gpu:N). - "default" restores the device assigned by the loader (even after a prior Select Model Device call). - "cpu" pins both the load and offload device to CPU. - "gpu:N" pins the load device to the Nth available GPU; the offload device is restored to the loader's original choice. When the requested device differs from the device the input model is already on, a fresh model is spawned via the loader's reload factory (cached_patcher_init) so the new patcher owns independent weights on the new device. Loaders that don't support multigpu (no factory) will cause the node to pass through unchanged with a warning. If the workflow already has MultiGPU CFG Split applied and the chosen GPU collides with one of the existing multigpu clones, that clone is dropped so two patchers don't end up bound to the same device. When the selected device does not exist on the current machine (e.g. a workflow built on a 2-GPU box opened on a 1-GPU box), the node passes the model through unchanged and logs a message instead of failing. NOTE: Placing Select Model Device *after* a node that has already consumed the same model (e.g. a KSampler that ran on this model on the original device) is not recommended -- any state the prior consumer mutated on the original model will be observed when the selected device matches the original (fast path). Place Select Model Device before any consumer of the model. c B[R"SSS[UR5[RR S5[R R S[RR5S9/[RR5/S9$)NSelectModelDevicezSelect Model DevicerrrHoptionsr) rr!rr"r#r$Combor/rBget_gpu_device_optionsr&r's r)r*#SelectModelDeviceNode.define_schemasuyy'.( -w'x1G1G1^1^1`a !  r,cgNTr5r(rHs r)validate_inputs%SelectModelDeviceNode.validate_inputssr,cUR5n[RRU5nUc6US;a0[R "SUS35 [ R"U5$[X5nUb [X5 [XR5 [ R"U5$![a:n[R"SUS35 [ R"U5sSnA$SnAff=f)NNrz'Select Model Device: requested device '+' not available, passing through unchanged.zHSelect Model Device: cannot retarget model, passing through unchanged. ())rcr/rBresolve_gpu_device_optionrDrErr2rh RuntimeErrorwarningrKrrrO)r(rrHrdes r)r3SelectModelDeviceNode.executes ))CCFK  .? ? LLB6(Juv w==' ' ()%:E   *5 ; %e->-> ?}}U##  ( OOfghfiijk l==' ' (s* B.. C28/C-'C2-C2r5Nr)rr rHstrr7r8 r:r;r<r=r"r>r*rr3r?r5r,r)rtrtsD@     $$r,rtcT\rSrSrSr\S5r\SSj5r\SS Sjj5rSr g) SelectCLIPDeviceNodea Place the CLIP text encoder on a specific device (default / cpu / gpu:N). - "default" restores the device assigned by the loader. - "cpu" pins both the load and offload device to CPU. - "gpu:N" pins the load device to the Nth available GPU. When the selected device does not exist on the current machine (e.g. a workflow built on a 2-GPU box opened on a 1-GPU box), the node passes the CLIP through unchanged and logs a message instead of failing. c B[R"SSS[UR5[RR S5[R R S[RR5S9/[RR5/S9$)NSelectCLIPDevicezSelect CLIP DevicercliprHrxr) rr!rr"Clipr$rzr/rBr{r&r's r)r*"SelectCLIPDeviceNode.define_schemasuyy&-( - f%x1G1G1^1^1`a    r,cgr~r5rs r)r$SelectCLIPDeviceNode.validate_inputsr,cUR5n[RRU5nUc6US;a0[R "SUS35 [ R"U5$[URU5Ul [ R"U5$![a$n[R"SUS35 SnAN?SnAff=f)Nrz&Select CLIP Device: requested device 'rzFSelect CLIP Device: cannot retarget CLIP, passing through unchanged. (r) rcr/rBrrDrErr2rhrGrr)r(rrHrdrs r)r3SelectCLIPDeviceNode.execute szz|))CCFK  .? ? LLA&Itu v==& & k0xHDL}}T"" k OOdefdgghi j j ks*B C %CC r5Nr)rr rHrr7r8rr5r,r)rrsC      # #r,rcT\rSrSrSr\S5r\SSj5r\SS Sjj5rSr g) SelectVAEDeviceNodeia Place the VAE on a specific device (default / gpu:N). - "default" restores the device assigned by the loader. - "gpu:N" pins the load device to the Nth available GPU; the offload device is set to the standard VAE offload device. CPU is intentionally not exposed in the UI for the VAE; if a workflow supplies "cpu" anyway (e.g. opened from another machine), the request is dropped with a log message and the VAE is passed through unchanged. When the selected device does not exist on the current machine (e.g. a workflow built on a 2-GPU box opened on a 1-GPU box), the node passes the VAE through unchanged and logs a message instead of failing. c B[R"SSS[UR5[RR S5[R R S[RR5S9/[RR5/S9$)NSelectVAEDevicezSelect VAE DevicervaerHrxr) rr!rr"Vaer$rzr/rBget_gpu_device_options_no_cpur&r's r)r*!SelectVAEDeviceNode.define_schema,styy%,( - U#x1G1G1e1e1gh    r,cgr~r5rs r)r#SelectVAEDeviceNode.validate_inputs<rr,ch[R"U5nURR5Ul[RR U5nUc6US;a0[ R"SUS35 [R"U5$Ub<URS:Xa,[ R"S5 [R"U5$[US5(dURUl [URU[RR5S9UlURR$UlUc UROUUl [R"U5$![ a:n[ R""SUS 35 [R"U5sSnA$SnAff=f) Nrz%Select VAE Device: requested device 'rr_zLSelect VAE Device: CPU is not a supported choice, passing through unchanged._select_base_device)rezDSelect VAE Device: cannot retarget VAE, passing through unchanged. (r)copyrGrcr/rBrrDrErr2rarNrHrrhvae_offload_devicerrrfirst_stage_model)r(rrHrdrs r)r3SelectVAEDeviceNode.execute@sSiinkk'') ))CCFK  .? ? LL@Hst u==% %  HMMU$: LLg h==% %s122&)jjC # &/ X&+&<&<&O&O&QCK!$ 1 1080@S,,h }}S!! & OObcdbeefg h==% % &s06E-- F17/F,&F1,F1r5Nr)rr rHrr7r8rr5r,r)rrsC"    ""r,rc@\rSrSrSr\S5r\SSSjj5rSrg) MultiGPUOptionsNodei^a  Select the relative speed of GPUs in the special case they have significantly different performance from one another. NOTE (not registered yet, see MultiGPUExtension.get_node_list below): The output GPUOptionsGroup is plumbed through create_multigpu_deepclones() and stored on model.model_options['multigpu_options'] via GPUOptionsGroup.register(), but the cond scheduler in comfy/samplers.py (calc_cond_batch_outer_multigpu) does NOT yet consult relative_speed when distributing conds across devices; it uses a uniform conds_per_device round-robin via next_available_device(). Before re-enabling this node, wire its relative_speed into the scheduler (e.g. via comfy.multigpu.load_balance_devices(), which already implements the proportional split) so the input actually affects work distribution. cb[R"SSS[UR5[RR SSSSS9[R R SS S S S 9[R"S 5R SSS9/[R"S 5R5/S9$)NMultiGPU_OptionszMultiGPU Optionsr device_indexr@)rrmaxrelative_speedg?gg{Gz?r GPU_OPTIONS gpu_optionsT)optionalr) rr!rr"r%r$FloatCustomr&r's r)r*!MultiGPUOptionsNode.define_schemamsyy&+( - ^QA2 F/#DQ -(..}t.L  -(//1  r,NcU(d[RR5nOUR5n[RR XS9nUR U5 [ R"U5$)N)rr)r/r0GPUOptionsGrouprc GPUOptionsaddrr2)r(rrropts r)r3MultiGPUOptionsNode.execute~sW..88:K%++-Knn''\'a}}[))r,r5N)rr6rfloatrzcomfy.multigpu.GPUOptionsGroupr7r8r9r5r,r)rr^s/    * *r,rc(\rSrSr\SSj5rSrg)MultiGPUExtensionic6# [[[[/$7fr)r rtrr)selfs r) get_node_listMultiGPUExtension.get_node_lists ! !    sr5N)r7zlist[type[io.ComfyNode]])r:r;r<r=rrr?r5r,r)rrs   r,rc# [5$7fr)rr5r,r)comfy_entrypointrs  s )rGr rHz torch.device)rGr r)rr )r7r)$ __future__rrrDinspectrtypingrtyping_extensionsrcomfy_api.latestrrcomfy.model_patcherr comfy.sdr r torchcomfy.model_managementr/comfy.multigpu ComfyNoder rKrRrVr]rhrrrtrrrrrr5r,r)rs"  &/0" $2<<$>0 KV<>>*&F$BLLF$R-#2<<-#`A"",,A"H**",,**Z    r,