3j"? SSKrSSKrSSKrSSKJr SSKJrJr SSKJ r SSK r SSK J s J s Js Jr SSKJrJr SSKJrJr SSKJr SSKJr SS KJrJrJr SS KJ r J!r!J"r" SS K#J$r$ SS K%J&r& SS K'J(r( SSK)J*r* SSK+J,r, /SQr-\*"\.5r/\"SS55r0"SS5r1S\\2-S-S\3\ S\24Sjr4S\&S\5\2S-\6S-44Sjr7S\0S\\2-S-S\3\ S\8\6\ 44Sjr9g)N)Callable) dataclassfield)Any)get_default_numa_optionsjustknobs_check)eventsmetrics) WorkerSpec)create_healthcheck_server)_AliveCallbackProxyLocalElasticAgentTORCHELASTIC_HEALTH_CHECK_PORT)DefaultLogsSpecs LogsSpecsSignalException)ChildFailedError)RendezvousParameters)parse_rendezvous_endpoint) get_logger) NumaOptions) LaunchConfigelastic_launch launch_agentc\rSrSr%Sr\\S'\\S'\\S'Sr\S-\S'Sr \ \S 'S r \ \S 'Sr \ \S 'S r \ \S'\"\S9r\\ \4\S'Sr\\S'Sr\\S'Sr\\S'Sr\ \S'Sr\ S-\S'\"\S9r\\ \ 4\S'Sr\ S-\S'Sr\ \S'Sr\S-\S'Sr\ \S 'Sr\\ S-\S!'Sr \\ S-\S"'S#r!\"\S$'Sr#\S-\S%'S&r$S'r%g)(r.a Creates a rendezvous config. Args: min_nodes: Minimum amount of nodes that the user function will be launched on. Elastic agent ensures that the user function start only when the min_nodes amount enters the rendezvous. max_nodes: Maximum amount of nodes that the user function will be launched on. nproc_per_node: On each node the elastic agent will launch this amount of workers that will execute user defined function. rdzv_backend: rdzv_backend to use in the rendezvous (zeus-adapter, etcd). rdzv_endpoint: The endpoint of the rdzv sync. storage. rdzv_configs: Key, value pair that specifies rendezvous specific configuration. rdzv_timeout: Legacy argument that specifies timeout for the rendezvous. It is going to be removed in future versions, see the note below. The default timeout is 900 seconds. run_id: The unique run id of the job (if not passed a unique one will be deduced from run environment - flow workflow id in flow - or auto generated). role: User defined role of the worker (defaults to "trainer"). max_restarts: The maximum amount of restarts that elastic agent will conduct on workers before failure. monitor_interval: The interval in seconds that is used by the elastic_agent as a period of monitoring workers. start_method: The method is used by the elastic agent to start the workers (spawn, fork, forkserver). metrics_cfg: configuration to initialize metrics. local_addr: address of the local node if any. If not set, a lookup on the local machine's FQDN will be performed. local_ranks_filter: ranks for which to show logs in console. If not set, show from all. event_log_handler: name of the event logging handler as registered in `elastic/events/handlers.py `_. duplicate_stdout_filters: If non-empty, duplicates stdout to a file containing only lines that match _any_ of the filter strings. duplicate_stderr_filters: If non-empty, duplicates stderr to a file containing only lines that match _any_ of the filter strings. virtual_local_rank: Enable virtual local rank mode for workers (defaults to False). When enabled, LOCAL_RANK is set to 0 for all workers and CUDA_VISIBLE_DEVICES is adjusted so each worker accesses its assigned GPU at device index 0. shutdown_timeout: Time in seconds to wait for graceful shutdown of workers before sending SIGKILL. Can also be set via TORCH_ELASTIC_SHUTDOWN_TIMEOUT environment variable. Defaults to 30 seconds. .. note:: `rdzv_timeout` is a legacy argument that will be removed in future. Set the timeout via `rdzv_configs['timeout']` min_nodes max_nodesnproc_per_nodeN logs_specsrun_id default_rolerole rdzv_endpointetcd rdzv_backend)default_factory rdzv_configs rdzv_timeout max_restartsg?monitor_intervalspawn start_methodlog_line_prefix_template metrics_cfg local_addrnullevent_log_handler numa_optionszSIGTERM,SIGINT,SIGHUP,SIGQUITsignals_to_handleduplicate_stdout_filtersduplicate_stderr_filtersFvirtual_local_rankshutdown_timeoutcSnURS:waURURS'OSUR;aXRS'URc[5UlURc~[ R R5(a[[ R R5UR:Xa/[5Ul[RSUR5 URc/[[R R#SS55Ul gURS:a[%SUR35eg) Nir*timeoutzUsing default numa options = %rTORCH_ELASTIC_SHUTDOWN_TIMEOUT30rz+shutdown_timeout must be non-negative, got )r+r)r rr6torchcuda is_available device_countrrloggerinfor;intosenvironget ValueError)selfdefault_timeouts X/home/wildlama/miniconda3/lib/python3.13/site-packages/torch/distributed/launcher/api.py __post_init__LaunchConfig.__post_init__|s    "+/+<+S>S=TU ')r r6r;)&__name__ __module__ __qualname____firstlineno____doc__rF__annotations__r rr"strr$r%r'rdictr)rr+r-r.floatr0r1r2r3r5r6rr7r8listr9r:boolr;rN__static_attributes__rPrMrr.s12hNN#'J D 'FCD#M3L##(#>L$sCx.>L#L#!e!L#+/cDj/"'"=Kc3h=!Jd !#s#'+L+$+<s<15d3i$.515d3i$.5$$#'cDj'rPrc<\rSrSrSrS\S\\-S-4SjrSr Sr g) ra Launches an torchelastic agent on the container that invoked the entrypoint. 1. Pass the ``entrypoint`` arguments as non ``kwargs`` (e.g. no named parameters)/ ``entrypoint`` can be a function or a command. 2. The return value is a map of each worker's output mapped by their respective global rank. Usage :: def worker_fn(foo): # ... def main(): # entrypoint is a function. outputs = elastic_launch(LaunchConfig, worker_fn)(foo) # return rank 0's output return outputs[0] # entrypoint is a command and ``script.py`` is the python module. outputs = elastic_launch(LaunchConfig, "script.py")(args) outputs = elastic_launch(LaunchConfig, "python")("script.py") config entrypointNcXlX lgN_config _entrypoint)rKr`ras rM__init__elastic_launch.__init__s  %rPcV[URUR[U55$rc)rrerfrZ)rKargss rM__call__elastic_launch.__call__sDLL$*:*:DJGGrPrd) rQrRrSrTrUrrrWrgrkr\r]rPrMrrs-4&&sNT)&HrPrrarjreturnc[U[5(a UR$[U[5(a)U[R :Xa[ SU5S5$U$g)aRetrieve entrypoint name with the rule: 1. If entrypoint is a function, use ``entrypoint.__qualname__``. 2. If entrypoint is a string, check its value: 2.1 if entrypoint equals to ``sys.executable`` (like "python"), use the first element from ``args`` which does not start with hifen letter (for example, "-u" will be skipped). 2.2 otherwise, use ``entrypoint`` value. 3. Otherwise, return empty string. c3:# UHoSS:wdM Uv M g7f)r-Nr]).0args rM '_get_entrypoint_name..s>A# s  r!) isinstancerrQrWsys executablenext)rarjs rM_get_entrypoint_namerysR*h''""" J $ $  '>>C C rPrdzv_parameterscURS:wagURnUR5nU(d [S5e[ USS9up#US:Xa[SUS35eX#4$)Nstatic)NNzKEndpoint is missing in endpoint. Try to add --master-addr and --master-portr*) default_portzport is missing in endpoint: z. Try to specify --master-port)backendendpointstriprJr)rzr master_addr master_ports rM_get_addr_and_portrs~(*''H~~H  Y   9PRSKb+H:5S T    %%rPr`c UR(dD[[R"5R5n[ R SU5 X0l[X5n[ RS0SU_SUR_SUR_SUR_SUR_SUR_S UR_S UR_S UR_S UR _S UR"R$_SUR&_SUR(_SUR*_SUR,_SUR._SUR0_5 [3S"URURURURURUR4S.URD6n[7U5upgUR,[8R:S'SnSn [8R<"[>5n U bT[ASSS9(aE[C5n [EU [ U 5SS9nURG5 [ RSU 5 [KURLURU[OU5[PRR"U5URUR UUUR4UR(UR*UR.UR0URTS9n [WU UR"URXURZUR\US 9n U bU R_U R`5 Sn [bRd"[bRf"UR&55 U Ri5n[jRl"U Ro5UR(5 URq5(a[sUURtS!9eURvU (aU RxR{5 $$![Ha [ R SSS9 SnSn GNf=f![ra e[|a3 Sn [jRl"U R5UR(5 e[Ha1 [jRl"U R5UR(5 ef=f!U (aU RxR{5 ff=f)#Nz3config has no run_id, generated a random run_id: %saRStarting elastic_operator with launch configs: entrypoint : %(entrypoint)s min_nodes : %(min_nodes)s max_nodes : %(max_nodes)s nproc_per_node : %(nproc_per_node)s run_id : %(run_id)s rdzv_backend : %(rdzv_backend)s rdzv_endpoint : %(rdzv_endpoint)s rdzv_configs : %(rdzv_configs)s max_restarts : %(max_restarts)s monitor_interval : %(monitor_interval)s log_dir : %(log_dir)s metrics_cfg : %(metrics_cfg)s event_log_handler : %(event_log_handler)s numa_options : %(numa_options)s signals_to_handle : %(signals_to_handle)s duplicate_stdout_filters : %(duplicate_stdout_filters)s duplicate_stderr_filters : %(duplicate_stderr_filters)s rarrrr"r'r%r)r-r.log_dirr2r5r6r7r8r9)r~rr"rrr3TORCHELASTIC_SIGNALS_TO_HANDLEzNai_infra/pytorch_distributed:torchelastic_enable_healthcheck_before_rendezvousF)default<)alive_callbackportr=z>Started early health check server on port %s before rendezvousz)Failed to start early health check serverT)exc_info)r$local_world_sizerarj rdzv_handlerr-r.rrr3r5r6r8r9r:)specr r0r1r;health_check_server)namefailuresr])@r"rWuuiduuid4rFrDwarningryrErrrr'r%r)r-r.r root_log_dirr2r5r6r7r8r9rr3rrGrHgetenvrrr r start Exceptionr r$tuple rdzv_registryget_rendezvous_handlerr:rr0r1r; set_delegate_get_alive_timer initialize_metrics MetricsConfigrunr recordget_event_succeeded is_failedrr return_valuesrshutdownrget_event_failed)r`rarjr"entrypoint_namerzrrralive_callback_proxyhealthcheck_portragent shutdown_rdzvresults rMrrs ==TZZ\%%&LfU *:rs# $( EEK5A  NEP>* = H  ii iX$H$HNX^d%:$s)PS(&)& 3:sTz !"&&^) ^)3%^) s)^) #s(^ ^)rP