vh}5ddlmZddlmZddlZddlZddlmZddlm Z ddlm Z ddlm Z ddl m Z ejeZe j"d d d ZGd dZeZ d ddZddZdZy)) annotations)SequenceN)Any)clusters)config) xla_bridge)_jaxjax_check_proxy_envsTz1Checks proxy vars in user envs and emit warnings.)namedefaulthelpceZdZUdZded<dZded<dZded<dZd ed <dZd ed <dZ d ed<dZ ded< d ddZ dZ dZ y)Staterint process_id num_processesNz+_jax.DistributedRuntimeService | Any | Noneservicez*_jax.DistributedRuntimeClient | Any | Noneclientz Any | Nonepreemption_sync_manager str | Nonecoordinator_address int | None slice_indexc T|xstjjd}t|tr|g}|Itjjdx} r(t t t| jd}|dk7r1d||||fvr)tjj||||||\}}}}| td| td| tdt|tstd|d t|d t|tstd |d t|d d |cxkr|ksntd |d|d ||_d|jdddz}|xs tjjd|}| td|rZdj!d|D}t"j%d|t'j(d|t'j(d|||_g}t,j.r=tjj1Dcgc]}d|j3vr|}}t5|d kDr/dj!|dz}d|d}t"j7||d k(rK|j8 t;dt"j%d|t=j>|||| |_||_ |jB t;dt=jD|||| | d !|_!t"j%d"||jBjG|jI| :d#tjvr(t tjjd#} | |_%ycc}w)$NJAX_COORDINATOR_ADDRESSJAX_LOCAL_DEVICE_IDS, deactivatez&coordinator_address should be defined.z$Number of processes must be defined.z6The process id of the current process must be defined.z5process_id must be a nonnegative int. Got process_id=z of type .z8num_processes must be a positive int. Got num_processes=rzbprocess_id and num_processes must be nonnegative, with process_id < num_processes. Got process_id=z, num_processes=z[::]::rJAX_COORDINATOR_BIND_ADDRESSz+coordinator_bind_address should be defined.c32K|]}t|yw)N)str).0xs O/opt/face_recognition/venv/lib/python3.12/site-packages/jax/_src/distributed.py z#State.initialize..ls BAQ Bsz4JAX distributed initialized with visible devices: %sjax_cuda_visible_devicesjax_rocm_visible_devices_proxy z. zHJAX detected proxy variable(s) in the environment as distributed setup: zpOn some systems, this may cause a hang of distributed.initialize and you may need to unset these ENV variable(s)z2distributed.initialize should only be called once.z&Starting JAX distributed service on %s)heartbeat_intervalmax_missing_heartbeatsT) init_timeoutr-r.use_compressionz+Connecting to JAX distributed service on %sJAX_SLICE_INDEX)&osenvironget isinstancerlistmapsplitr ClusterEnv$auto_detect_unset_distributed_params ValueError TypeErrortyperrsplitjoinloggerinforupdater_CHECK_PROXY_ENVSvaluekeyslowerlenwarningr RuntimeErrorr get_distributed_runtime_servicerrget_distributed_runtime_clientconnect"initialize_preemption_sync_managerr)selfrrrlocal_device_idscluster_detection_methodinitialization_timeoutcoordinator_bind_address"service_heartbeat_interval_secondsservice_max_missing_heartbeats!client_heartbeat_interval_secondsclient_max_missing_heartbeatsrenv_ids default_coordinator_bind_addressvisible_devices proxy_varskeyvarsrHs r' initializezState.initialize.s/E::>>*CD"C(*+ ?U0V%VW%Vc#w}}S'9:; L0 $mZAQRR    B B!&$  IM:7G" ? @@ = >> O PP j# & ((2|9T*=M '89:k"DO0s9N%c@|jr7t|jdr|jjd|_|jr!|jjd|_|jr"|jjd|_yy)Nshutdown)rhasattrr_rrrNs r'r_zState.shutdownsx ## --z : $$--/%)d" {{ kkdk || lldlc|j tdtj|_|jj |j y)Nz8Preemption sync manager should only be initialized once.)rrIr create_preemption_sync_managerr]rras r'rMz(State.initialize_preemption_sync_managersN ##/  D FF ++-    ++DKK8rb) NNNNN,N rfrfrfN)rrrrrrrOint | Sequence[int] | NonerPrrQrrRrrSrrTrrUrrVrrr)__name__ __module__ __qualname__r__annotations__rrrrrrr]r_rMrbr'rr%s*c-9=' 6=7;& 4;(,:,$(z( +z 48-1*.@D8 n# ,6 n# *- n#,6n#69n#25n#58n#14n#)n#` 9rbrc |tjr tdtj ||||||||y)aInitializes the JAX distributed system. Calling :func:`~jax.distributed.initialize` prepares JAX for execution on multi-host GPU and Cloud TPU. :func:`~jax.distributed.initialize` must be called before performing any JAX computations. The JAX distributed system serves a number of roles: * It allows JAX processes to discover each other and share topology information, * It performs health checking, ensuring that all processes shut down if any process dies, and * It is used for distributed checkpointing. If you are using TPU, Slurm, or Open MPI, all arguments are optional: if omitted, they will be chosen automatically. The ``cluster_detection_method`` may be used to choose a specific method for detecting those distributed arguments. You may pass any of the automatic ``spec_detect_methods`` to this argument though it is not necessary in the TPU, Slurm, or Open MPI cases. For other MPI installations, if you have a functional ``mpi4py`` installed, you may pass ``cluster_detection_method="mpi4py"`` to bootstrap the required arguments. Otherwise, you must provide the ``coordinator_address``, ``num_processes``, ``process_id``, and ``local_device_ids`` arguments to :func:`~jax.distributed.initialize`. When all four arguments are provided, cluster environment auto detection will be skipped. Please note: on some systems, particularly HPC clusters that only access external networks through proxy variables such as HTTP_PROXY, HTTPS_PROXY, etc., the call to :func:`~jax.distributed.initialize` may timeout. You may need to unset these variables prior to application launch. Args: coordinator_address: the IP address of process `0` and a port on which that process should launch a coordinator service. The choice of port does not matter, so long as the port is available on the coordinator and all processes agree on the port. May be ``None`` only on supported environments, in which case it will be chosen automatically. Note that special addresses like ``localhost`` or ``127.0.0.1`` usually mean that the program will bind to a local interface and are not suitable when running in a multi-host environment. num_processes: Number of processes. May be ``None`` only on supported environments, in which case it will be chosen automatically. process_id: The ID number of the current process. The ``process_id`` values across the cluster must be a dense range ``0``, ``1``, ..., ``num_processes - 1``. May be ``None`` only on supported environments; if ``None`` it will be chosen automatically. local_device_ids: Restricts the visible devices of the current process to ``local_device_ids``. If ``None``, defaults to all local devices being visible to the process except when processes are launched via Slurm and Open MPI on GPUs. In that case, it will default to a single device per process. cluster_detection_method: An optional string to attempt to autodetect the configuration of the distributed run. Note that "mpi4py" method requires you to have a working ``mpi4py`` install in your environment, and launch the applicatoin with an MPI-compatible job launcher such as ``mpiexec`` or ``mpirun``. Legacy auto-detect options "ompi" (OMPI) and "slurm" (Slurm) remain enabled. "deactivate" bypasses automatic cluster detection. initialization_timeout: Time period (in seconds) for which connection will be retried. If the initialization takes more than the timeout specified, the initialization will error. Defaults to 300 secs i.e. 5 mins. coordinator_bind_address: the address and port to which the coordinator service on process `0` should bind. If this is not specified, the default is to bind to all available addresses on the same port as ``coordinator_address``. On systems that have multiple network interfaces per node it may be insufficient to only have the coordinator service listen on one address/interface. slice_index: The slice index assigned to this process' local devices. If any process sets ``slice_index``, then all processes must do so. If ``None`` the slice indices will be chosen automatically. Raises: RuntimeError: If :func:`~jax.distributed.initialize` is called more than once or if called after the backend is already initialized. Examples: Suppose there are two GPU processes, and process 0 is the designated coordinator with address ``10.0.0.1:1234``. To initialize the GPU cluster, run the following commands before anything else. On process 0: >>> jax.distributed.initialize(coordinator_address='10.0.0.1:1234', num_processes=2, process_id=0) # doctest: +SKIP On process 1: >>> jax.distributed.initialize(coordinator_address='10.0.0.1:1234', num_processes=2, process_id=1) # doctest: +SKIP zjax.distributed.initialize() must be called before any JAX calls that might initialise the XLA backend. This includes any computation, but also calls to jax.devices, jax.device_put, and others.)rN)rbackends_are_initializedrI global_stater])rrrrOrPrQrRrs r'r]r]sQr((* t uu-}j*,D02J&13rbc&tjduS)z3Check if the JAX distributed system is initialized.N)rorrlrbr'is_initializedrqs   D ((rbc,tjy)z_Shuts down the distributed system. Does nothing if the distributed system is not running. N)ror_rlrbr'r_r_s rb)NNNNNreNN)rrrrrrrOrgrPrrQrrRrrr)returnbool) __future__rcollections.abcrloggingr2typingrjax._srcrrr jax._src.libr getLoggerrhr@ bool_flagrCrror]rqr_rlrbr'r}s#$    8 $%F$$   <N9N9`w 15+/(,>B6:-06:)-`3(`3%`3"<`3*4 `3 (+ `3 *4 `3'`3F)rb