+jrDSrSSKrSSKrSSKrSSKrSSKJrJr SSKJ r SSK J r SSK J r JrJrJrJrJrJrJrJrJrJrJr SSKJr "SS \5r"S S \ 5r"S S \ 5r\"SS55r\"SS55r \ \/S4r!"SS5r"\""5r#g)zHBackground asset seeder with thread management and cancellation support.N) dataclassfield)Enum)Callable) ENRICHMENT_METADATAENRICHMENT_STUBRootTypebuild_asset_specscollect_paths_for_rootsenrich_assets_batchget_all_known_prefixesget_prefixes_for_rootget_unenriched_assets_for_rootsinsert_asset_specs$mark_missing_outside_prefixes_safelysync_root_safely)dependencies_availablec\rSrSrSrSrg)ScanInProgressErrorzBRaised when an operation cannot proceed because a scan is running.N)__name__ __module__ __qualname____firstlineno____doc____static_attributes__r1/home/wildlama/comfy/ComfyUI/app/assets/seeder.pyrrsLrrc(\rSrSrSrSrSrSrSrSr g) State zSeeder state machine states.IDLERUNNINGPAUSED CANCELLINGrN) rrrrrr#r$r%r&rrrrr!r! s& DG FJrr!c$\rSrSrSrSrSrSrSrg) ScanPhase)zScan phase options.fastenrichfullrN) rrrrrFASTENRICHFULLrrrrr(r()s D F Drr(cR\rSrSr%SrSr\\S'Sr\\S'Sr \\S'Sr \\S'Sr g ) Progress1z*Progress information for a scan operation.rscannedtotalcreatedskippedrN) rrrrrr3int__annotations__r4r5r6rrrrr1r11s,4GSE3NGSGSrr1cP\rSrSr%Sr\\S'\S-\S'\"\ S9r \ \ \S'Sr g) ScanStatus;z#Current status of the asset seeder.stateNprogress)default_factoryerrorsr) rrrrrr!r8r1rlistr?strrrrrr:r:;s'- Lod3FDI3rr:c\rSrSrSrS2SjrS2SjrS\4SjrS\ RSS S 4S \ \ S 4S \ S \ S-S\S\S\4 SjjrS3S \ \ S 4S \ S-S\S\4SjjrS3S \ \ S 4S \ S-S\S\4SjjrS4S \ \ S 4S\S\4SjjrS\4SjrS\4SjrS\4SjrS\4SjrS5S \ \ S 4S-S \ S-S \ S-S\S-S\S-S\S\4SjjrS6S\S-S\4SjjrS\4SjrS7S\SS4SjjrS\4SjrS2SjrS\4SjrS\4S jr S\4S!jr!S"\"S#\#SS4S$jr$S8S%\S-S&\S-S'\S-S(\S-SS4 S)jjr%S*r&S+\"SS4S,jr'S \ \ S 4SS4S-jr(S2S.jr)S \ \ S 4S\ \\\44S/jr*S \ \ S 4S\ \\44S0jr+S1r,g)9 _AssetSeederGzBackground asset scanning manager. Spawns ephemeral daemon threads for scanning. Each scan creates a new thread that exits when complete. Use the module-level ``asset_seeder`` instance. returnNc[R"5Ul[RUlSUlSUl/UlSUl [R"5Ul [R"5Ul URR5 SUl[R UlSUlSUlSUlSUlSUlg)NrF) threadingRLock_lockr!r#_state _progress_last_progress_errors_threadEvent _cancel_event _run_gateset_rootsr(r/_phase_compute_hashes _prune_first_progress_callback _disabled_pending_enrichselfs r__init___AssetSeeder.__init__Os__& jj *./3"$ 04 &__."* ,. !* %*"';?$,0rc>SUl[R"S5 g)z=Disable the asset seeder, preventing any scans from starting.TzAsset seeder disabledN)rXlogginginforZs rdisable_AssetSeeder.disablecs ,-rcUR$)z&Check if the asset seeder is disabled.)rXrZs r is_disabled_AssetSeeder.is_disabledhs ~~rmodelsinputoutputFroots.phaseprogress_callback prune_firstcompute_hashescUR(a[R"S5 g[R"SXR5 UR  UR [R:wa [R"S5 SSS5 g[RUl[5Ul /Ul Xl X lX@lXPlX0lUR$R'5 UR(R+5 [,R."UR0SSS9UlUR2R55 SSS5 g!,(df  g=f) aStart a background scan for the given roots. Args: roots: Tuple of root types to scan (models, input, output) phase: Scan phase to run (FAST, ENRICH, or FULL for both) progress_callback: Optional callback called with progress updates prune_first: If True, prune orphaned assets before scanning compute_hashes: If True, compute blake3 hashes (slow) Returns: True if scan was started, False if already running z(Asset seeder is disabled, skipping startFz!Seeder start (roots=%s, phase=%s)z,Asset seeder already running, skipping startNrCT)targetnamedaemon)rXr_debugr`valuerIrJr!r#r$r1rKrMrSrTrVrUrWrPclearrQrRrGThread _run_scanrNstart)r[rjrkrlrmrns rrx_AssetSeeder.startls( >> MMD E 8%M ZZ{{ejj( KLZ --DK%ZDNDLKK + #1 &7 #    $ $ & NN   $++~~#DL LL   )ZZs6EB=E E)cDURU[RUUSS9$)a&Start a fast scan (phase 1 only) - creates stub records. Args: roots: Tuple of root types to scan progress_callback: Optional callback for progress updates prune_first: If True, prune orphaned assets before scanning Returns: True if scan was started, False if already running Frjrkrlrmrn)rxr(r-)r[rjrlrms r start_fast_AssetSeeder.start_fasts- zz../#   rcDURU[RUSUS9$)a(Start an enrichment scan (phase 2 only) - extracts metadata and hashes. Args: roots: Tuple of root types to scan progress_callback: Optional callback for progress updates compute_hashes: If True, compute blake3 hashes Returns: True if scan was started, False if already running Fr{)rxr(r.)r[rjrlrns r start_enrich_AssetSeeder.start_enrichs/ zz""/)   rcUR URXS9(a SSS5 gURbg[URS5nUR U5 [ U5URS'URS=(d UURS'O UUS.Ul[ R"SURS5 SSS5 g!,(df  g=f)a{Start an enrichment scan now, or queue it for after the current scan. If the seeder is idle, starts immediately. Otherwise, the enrich request is stored and will run automatically when the current scan finishes. Args: roots: Tuple of root types to scan compute_hashes: If True, compute blake3 hashes Returns: True if started immediately, False if queued for later rjrnNTrjrnzEnrich scan queued (roots=%s)F)rIrrYrRupdatetupler_r`)r[rjrnexisting_rootss renqueue_enrich_AssetSeeder.enqueue_enrichs$ZZ  u LZ##/!$T%9%9'%B!C%%e,05n0E$$W-(()9:Ln$$%56 #&4($ LL8$:N:Nw:W X !Z sCB"C C%cUR UR[R[R4;a SSS5 g[ R "SURR5 [RUlURR5 URR5 SSS5 g!,(df  g=f)z{Request cancellation of the current scan. Returns: True if cancellation was requested, False if not running or paused NFz Asset seeder cancelling (was %s)T) rIrJr!r$r%r_r`rtr&rPrRrQrZs rcancel_AssetSeeder.cancels ZZ{{5==%,,"??Z LL;T[[=N=N O**DK    " " $ NN   ZZs0CA5C Cc"UR5$)ziStop the current scan (alias for cancel). Returns: True if stop was requested, False if not running )rrZs rstop_AssetSeeder.stops {{}rc,UR UR[R:wa SSS5 g[R "S5 [R UlURR5 SSS5 g!,(df  g=f)zPause the current scan. The scan will complete its current batch before pausing. Returns: True if pause was requested, False if not running NFzAsset seeder pausingT) rIrJr!r$r_r`r%rQrurZs rpause_AssetSeeder.pauses\ZZ{{emm+Z LL/ 0,,DK NN " ZZs BAB BcNUR UR[R:wa SSS5 g[R "S5 [R UlURR5 SSS5 URS05 g!,(df  N!=f)zResume a paused scan. This is a noop if the scan is not in the PAUSED state Returns: True if resumed, False if not paused NFzAsset seeder resumingzassets.seed.resumedT) rIrJr!r%r_r`r$rQrR _emit_eventrZs rresume_AssetSeeder.resumesrZZ{{ell*Z LL0 1--DK NN    .3Zs BAB B$timeoutc[R"S5 UR URnURnUR n UR n URn SSS5 UR5 URUS9(dgUbUOW n URUbUOWUbUOWU UbUOW UbUS9$W S9$!,(df  Nc=f)aCancel any running scan and start a new one. Args: roots: Roots to scan (defaults to previous roots) phase: Scan phase (defaults to previous phase) progress_callback: Progress callback (defaults to previous) prune_first: Prune before scan (defaults to previous) compute_hashes: Compute hashes (defaults to previous) timeout: Max seconds to wait for current scan to stop Returns: True if new scan was started, False if failed to stop previous zAsset seeder restart requestedNrFr{) r_r`rIrSrTrWrVrUrwaitrx) r[rjrkrlrmrnr prev_roots prev_phase prev_callback prev_prune prev_hashescbs rrestart_AssetSeeder.restart&s,  56 ZZJJ 33M**J..K  yyy)"3"? ]zz ,%* ,%* '2'> J"0"<   CN   Zs =B== C cUR URnSSS5 WcgURUS9 UR5(+$!,(df  N6=f)zWait for the current scan to complete. Args: timeout: Maximum seconds to wait, or None for no timeout Returns: True if scan completed, False if timeout expired or no scan running NTr)rIrNjoinis_alive)r[rthreads rr_AssetSeeder.waitSsGZZ\\F > G $??$$$ Zs A  Ac TUR UR=(d URn[URU(a4[ UR URURURS9OS[UR5S9sSSS5 $!,(df  g=f)z2Get the current status and progress of the seeder.r3r4r5r6N)r<r=r?) rIrKrLr:rJr1r3r4r5r6r@rM)r[srcs r get_status_AssetSeeder.get_statuscsr ZZ..7D$7$7Ckk "KK))KKKK DLL) ZZs BB B'cUR5 URUS9 UR SUlSSS5 g!,(df  g=f)zGracefully shutdown: cancel any running scan and wait for thread. Args: timeout: Maximum seconds to wait for thread to exit rN)rrrIrN)r[rs rshutdown_AssetSeeder.shutdownts3  ' " ZZDLZZs = A cUR UR[R:wa [ S5e[R UlSSS5 [ 5(d=[R"S5 UR UR5 SSS5 g[5n[U5nUS:a[R"SU5 UUR UR5 SSS5 $!,(df  N=f!,(df  g=f!,(df  $=f!UR UR5 SSS5 f!,(df  f=f=f)akMark references as missing when outside all known root prefixes. This is a non-destructive soft-delete operation. Assets and their metadata are preserved, but references are flagged as missing. They can be restored if the file reappears in a future scan. This operation is decoupled from scanning to prevent partial scans from accidentally marking assets belonging to other roots. Should be called explicitly when cleanup is desired, typically after a full scan of all roots or during maintenance. Returns: Number of references marked as missing Raises: ScanInProgressError: If a scan is currently running z0Cannot mark missing assets while scan is runningNz:Database dependencies not available, skipping mark missingrzMarked %d references as missing) rIrJr!r#rr$rr_warning_reset_to_idler rr`)r[ all_prefixesmarkeds rmark_missing_outside_prefixes*_AssetSeeder.mark_missing_outside_prefixess&ZZ{{ejj()F --DK  &)++P##% 23L9,GFz >G##%)Z(##%sM?C:%D-D !3D- D: D D D* - E%:E E% E" E%c^URUl[RUlSUlg)zFReset state to IDLE, preserving last progress. Caller must hold _lock.N)rKrLr!r#rJrZs rr_AssetSeeder._reset_to_idles "nnjj rc6URR5$)z)Check if cancellation has been requested.)rPis_setrZs r _is_cancelled_AssetSeeder._is_cancelleds!!((**rcURR5(+=(d URR5$)aNon-blocking check: True if paused or cancelled. Use as interrupt_check for I/O-bound work (e.g. hashing) so that file handles are released immediately on pause rather than held open while blocked. The caller is responsible for blocking on _check_pause_and_cancel() afterward. )rQrrPrZs r_is_paused_or_cancelled$_AssetSeeder._is_paused_or_cancelleds.>>((**Id.@.@.G.G.IIrcURR5(dURS05 URR5 UR 5$)a Block while paused, then check if cancelled. Call this at checkpoint locations in scan loops. It will: 1. Block indefinitely while paused (until resume or cancel) 2. Return True if cancelled, False to continue Returns: True if scan should stop, False to continue zassets.seed.paused)rQrrrrrZs r_check_pause_and_cancel$_AssetSeeder._check_pause_and_cancelsG~~$$&&   12 6 !!##r event_typedatacSSKJn [US5(a.UR(aURR X5 ggg![ a gf=f)z.Emit a WebSocket event if server is available.r) PromptServerinstanceN)serverrhasattrr send_sync Exception)r[rrrs rr_AssetSeeder._emit_eventsN  +|Z00\5J5J%%// A6K0   sAA AAr3r4r5r6cbSnSnUR URc SSS5 gUbXRlUbX RlUbX0RlUbX@RlUR (ahUR n[URRURRURRURR S9nSSS5 U(aU(a U"U5 ggg!,(df  N(=f![a gf=f)z'Update progress counters (thread-safe).Nr) rIrKr3r4r5r6rWr1r)r[r3r4r5r6callbackr=s r_update_progress_AssetSeeder._update_progresss-1$( ZZ~~%Z")0& ',$")0&")0&&&22# NN22.... NN22 NN22 (  "!8)Z.  s#DCDD! D! D.-D.messagecUR [UR5UR:aURR U5 SSS5 g!,(df  g=f)z:Add an error message (thread-safe), capped at _MAX_ERRORS.N)rIlenrM _MAX_ERRORSappend)r[rs r _add_error_AssetSeeder._add_errors< ZZ4<< 4#3#33 ##G,ZZs ?A A#cSSKnUHtnUS:Xa@[R"S[RR UR 55 MI[U5nU(dM][R"SX45 Mv g)z)Log the directories that will be scanned.rNrgz!Asset scan [models] directory: %szAsset scan [%s] directories: %s) folder_pathsr_r`ospathabspath models_dirr)r[rjrrootprefixess r_log_scan_config_AssetSeeder._log_scan_configs\Dx 7GGOOL$;$;< 168LL!BDSrc *[R"5nURnURnSnSnSnSnSn[ 5(dUR S5 UR SSS05 U(a=UR SUR(aURROSUUS.5 UR UR5 URn U b<SUl URU S U S S 9(d[R"S U S 5 SSS5 gUR(a2[!5n [#U 5n U S:a[R$"S U 5 UR'5(a[R$"S5 SnU(a=UR SUR(aURROSUUS.5 UR UR5 URn U b<SUl URU S U S S 9(d[R"S U S 5 SSS5 gUR)U5 U[*R,[*R.4;GaUR1U5upnXUpnUR'5(aSnU(a=UR SUR(aURROSUUS.5 UR UR5 URn U b<SUl URU S U S S 9(d[R"S U S 5 SSS5 gUR S[3U5UUUS.5 U[*R4[*R.4;GaUR'5(aSnU(a=UR SUR(aURROSUUS.5 UR UR5 URn U b<SUl URU S U S S 9(d[R"S U S 5 SSS5 gUR7U5upU(aSnU(a=UR SUR(aURROSUUS.5 UR UR5 URn U b<SUl URU S U S S 9(d[R"S U S 5 SSS5 gUR S[3U5US.5 [R"5U- n[R$"SUUR8UUUU5 UR SUR8UUUU[;US5S.5 U(a=UR SUR(aURROSUUS.5 UR UR5 URn U b<SUl URU S U S S 9(d[R"S U S 5 SSS5 g!,(df  g=f!,(df  g=f!,(df  g=f!,(df  g=f!,(df  g=f![<aRnUR SU35 [R>"S5 UR SS[AU505 SnAGNaSnAff=f!,(df  g=f!U(a=UR SUR(aURROSUUS.5 UR UR5 URn U b<SUl URU S U S S 9(d[R"S U S 5 SSS5 f!,(df  f=f=f)z,Main scan loop running in background thread.Frz#Database dependencies not availablezassets.seed.errorrzassets.seed.cancelled)r3r4r5Nrjrnrz.Pending enrich scan could not start (roots=%s)z%Marked %d refs as missing before scanz(Asset scan cancelled after pruning phaseTzassets.seed.fast_complete)rjr5r6r4zassets.seed.enrich_complete)rjenrichedz:Scan(%s, %s) done %.3fs: created=%d enriched=%d skipped=%dzassets.seed.completed)rkr4r5rr6elapsedz Scan failed: zAsset scan failed)!time perf_counterrSrTrrrrKr3rIrrYrr_rrVr rr`rrr(r-r/_run_fast_phaser@r._run_enrich_phasertroundr exceptionrA)r[t_startrjrk cancelled total_createdtotal_enrichedskipped_existing total_pathspendingrrr5r6pathsenrich_cancelledres rrw_AssetSeeder._run_scansR##%     n )++ EF  ' EFd  +=A^^4>>#9#9QR!,#0##%..&+/D(,,%g.'./?'@- L#G,s  57 =lKA:LL!H&Q++-- GH N  +=A^^4>>#9#9QR!,#0##%..&+/D(,,%g.'./?'@- L#G,]  ! !% (88*.*>*>u*E'%?FQV //11 $Ix  +=A^^4>>#9#9QR!,#0##%..&+/D(,,%g.'./?'@- L#G,G  /!%e#0#3!, ))9>>:://11 $IX  +=A^^4>>#9#9QR!,#0##%..&+/D(,,%g.'./?'@- L#G,g483I3I%3P0 # $IL  +=A^^4>>#9#9QR!,#0##%..&+/D(,,%g.'./?'@- L#G,[  1!%e$2'')G3G LLL      '"[[(, ./$Wa0  "  +=A^^4>>#9#9QR!,#0##%..&+/D(,,%g.'./?'@- L#G, G OOmA3/ 0   1 2   09c!f2E F F G  +=A^^4>>#9#9QR!,#0##%..&+/D(,,%g.'./?'@- L#G,s4[>AZ#A0[$AZ A%[?AZ($A[AZ96[#A[ B [!A\: Z Z%( Z69 [ [ \7%A\2,] 2\77] : ] A`A`8 ` ` `c [R"5nSnSn[5n[R"5nUH8nUR5(aX4S4s $UR [ U55 M: [ R"S[R"5U- [U55 UR5(aX4S4$[R"5n[U5n [ R"S[R"5U- [U 55 [U 5n URU S9 URS[U5U SS.5 [R"5n [U USSS 9upn[ R"S [R"5U - [U 5U5 URUS 9 UR5(aX4U 4$S n[R"5nS n[S[U 5U5HnUR5(a([ R"SU[U 5U5 X4U 4s $U UUU-nUVVs1sHnUSHnUiM M nnn[!UU5nUU- nU[U5-n[R"5nURUUS9 UU- U:dMURSSU[U 5US.5 UnM UR[U 5US9 [ R"S[R"5U- UUU 5 X4U 4$s snnf!["a8nUR%SUSU35 [ R&"SU5 SnANSnAff=f)zuRun phase 1: fast scan to create stub records. Returns: Tuple of (total_created, skipped_existing, total_paths) rz9Fast scan: sync_root phase took %.3fs (%d existing paths)z4Fast scan: collect_paths took %.3fs (%d paths found))r4assets.seed.startedr*)rjr4rkF)enable_metadata_extractionrnz>Fast scan: build_asset_specs took %.3fs (%d specs, %d skipped))r6i?z2Fast scan cancelled after %d/%d files (created=%d)tagszBatch insert failed at offset z: z Batch insert failed at offset %dN)r3r5assets.seed.progress)rkr3r4r5zHFast scan complete: %.3fs total (created=%d, skipped=%d, total_paths=%d))rrrRrrrr_rsrr rrr@r ranger`rrrr)r[rj t_fast_startrrexisting_pathst_syncr t_collectrrt_specsspecstag_pool batch_sizelast_progress_timeprogress_intervalibatchspect batch_tagsr5rr3nows rr_AssetSeeder._run_fast_phasesl ((*  #&5""$A++--$99  ! !"21"5 6  G    & (    ' ' ) ) A5 5%%' '. B    ) + J %j  K0  !5kK& I ##%,=  ', - ))  L    ' ) J   &67  ' ' ) ) K? ? !..0q#e*j1A++-- HJ!  % CC!a*n-E(-DtF|!!|!JD I,UJ?( #e*nG##%C  ! !'= ! I''+<<  *!'#*!$U#0 &)"C2F c%j-H V    , .      ;;AE I"@2aS IJ!!"DaHH IsL64L<< M>.M99M>cSnSn[R"5nSnUR(d[nO[nUR S[ U5SS.5 [5nSnSn 0n UR5(a[R"S U5 SU4$[UUUS 9n U(a#U V s/sHoRU;dMU PM n n U (dSU4$[U SURURU S 9upX-- nURU5 U S:Xa1US - nX:a&[R "S U[#U55 SU4$OSn[R"5nX- U:aUR SSUS.5 UnGMs sn f)zrRun phase 2: enrich existing records with metadata and hashes. Returns: Tuple of (cancelled, total_enriched) rdrrr+)rjrkrTz%Enrich scan cancelled after %d assets) max_levellimit)extract_metadata compute_hashinterrupt_checkhash_checkpointszKEnrich phase stopping: %d consecutive batches with no progress (%d skipped)r)rkrF)rrrUrrrr@rRrr_r`r reference_idr rrrr)r[rjrrrrtarget_max_levelskip_idsconsecutive_emptymax_consecutive_emptyr unenrichedrr failed_idsr s rr_AssetSeeder._run_enrich_phases  !..0##. 2   !5kH 5 !U !/1++-- DnU^++9* J)3VA~~X7Ua VFn$$C$7!%!11 $ < %&!##%C'+<<  *!)$2&)"eWs /FF)rPrUrXrMrLrIrYrTrKrWrVrSrQrJrN)rEN)rfNF)rfF)NNNNN@)N)r )NNNN)-rrrrrr\raboolrdr(r/rr ProgressCallbackrxr|rrrrrrfloatrrr:rrr7rrrrrrAdictrrrrrrwrrrrrrrCrCGs\1(. T 'D$>>59!$ ,Xs]#,,,d2 ,  ,  , ,`'D59!  Xs]# ,d2     4'D59$  Xs]# ,d2     4'D$"Xs]#""  "H  dt &.2"&59#'&*+ Xs]#d*+ 4+ ,d2 + D[ + t + +  + Z%EDL%D% J"      (&s(&T +t+JJ $ $c$# "" #t#Tz#t # t #  #JK-#-$- TeHcM&: Tt TyvjRjr+sN  (    3M)MD   44 4XJ,-D %D %N~ r