o H hv6@sUddlZddlZddlmZmZddlmZddlmZm Z m Z m Z m Z ddl Z ddlmZddl mZddlmZddlmZgZeeed<eeZGd d d ejZd eed efd dZdS)N) CollectionMapping)deepcopy)AnyCallableOptionaloverloadUnion)optim) ShardedTensor)FullyShardedDataParallel__all__c @sTeZdZdZ  d"deeeeje ffde j de e eeefde ejddf dd Zd d Zdeeeffd d Zed#d$ddZedegefdefddZd%de egefde efddZedeejeffddZdeeefddfddZdeeefddfddZd&ddZdeeeffddZdeeeffd d!ZdS)'_NamedOptimizera ``_NamedOptimizer`` takes a dict of parameters and exposes ``state_dict`` by parameter key. We replace the original key (number) in an optim to the fully qualified name (FQN) string. User can initialize the optim as they initialize a PyTorch optim, the only difference is that they also need to pass in the FQN of each parameters. Args: named_parameters (Mapping[str, Union[torch.Tensor, ShardedTensor]]): Mapping from FQN to parameter. optimizer_class (optim.Optimizer): The class of optimizer to instantiate. param_groups (Collection[Mapping[str, Any]]): `param_groups` to pass to optimizer if specified. The key of the inner map needs to be FQNs. Default: None module (nn.Module): the module whose parameters to updated by the optimizer. args: arguments to pass to the optimizer constructor. kwargs: arguments to pass to the optimizer constructor. Example:: >>> # xdoctest: +SKIP("distributed") >>> from torch import optim >>> from torch.distributed.optim import _NamedOptimizer >>> >>> # Define the named optimizer. >>> m = Model(...) >>> named_optim = _NamedOptimizer(m.named_parameters(), optim.SGD) >>> # Forward pass + backward pass. >>> named_optim.step() >>> ... >>> # Call state_dict for the named optimizer returns a FQN state_dict. >>> named_optim.state_dict() Warning: This API is still in development and subject to change. TODO: Add tutorial for _NamedOptimizer. TODO: Add documentation in the docstring for the public attributes like self.param_groups and self.named_parameters. Nnamed_parametersoptimizer_class param_groupsmodulereturnc Ostjd||_|t||_|dur|jn|}||g|Ri||_||_ |dur9t |j |_ n3t ddd|jD}g} |D]} | dD]} | |vr`td| d| || qRqL| |_ |jj|_dS)Nz'torch.distributed.optim._NamedOptimizerzvSince we pass in param_groups, we will use param_groups to initialize the optimizer, not all parameters of the module.cSi|]\}}||qSr.0keyparamrr[/var/www/vscode/kcb/lib/python3.10/site-packages/torch/distributed/optim/named_optimizer.py ]z,_NamedOptimizer.__init__..paramszExpect param name z% found in param group but is missing.)torch_C_log_api_usage_oncer_param_groups_checkdictrvalues _optimizerrlistkeysordered_param_keyswarningswarnitems ValueErrorappend) selfrrrrargskwargsparams_for_optimizer param_to_keyr'grouprrrr__init__@s>    z_NamedOptimizer.__init__cCs|jdurE|jD]>}t|tsJdd|vsJd|d}t|tjr(|g}t|}|D]}t|tjs?tdt|q.||d<qdSdS)Nparam group must be a dictrz#param group must contain key paramsz>optimizer can only optimize Tensors, but one of the params is )r isinstancer"rTensorr% TypeErrortypename)r- param_grouprrrrrr!js&     z#_NamedOptimizer._param_groups_checkc sj}|d}fdd|dD}g}|D]+}fdd|dD}dt|i}|D]\}} |dkr?t| ||<q1||q||dS) z Return the ``state_dict`` of the optimizer. Instead of using number to index parameters, we will use module fully qualified name (FQN) as the key. rcsi|] \}}j||qSrr')rst_key state_valr-rrrs z._NamedOptimizer.state_dict..statecsg|]}j|qSrr:)rrr=rr rz._NamedOptimizer.state_dict..r)r>r)r$ state_dictr*sortedrr,_post_state_dict) r-r@r ret_state ret_groupsr2 param_keys ret_groupkvrr=rr@{s      z_NamedOptimizer.state_dict.closurecCdSNrr-rIrrrstepz_NamedOptimizer.stepcCrJrKrrLrrrrMrNcCs|jj|dS)z Perform a single optimization step. This will call :meth:`torch.optim.Optimizer.step` on the wrapped optimizer. rI)r$rMrLrrrrMscCs|jjSrK)r$r>r=rrrr>sz_NamedOptimizer.stater@c Cs|j}||}|d}|d}t|dkrtdt|jD]\}}||vr,q!t||t||krMtdt||d|dt||||D]}\}}|||vrhtd|d|d|||} t |t rt | t szJt| } t| } | | krtd | d | d |d |t | | D] \} } | j | j qqSt |tjrt | tjsJ|| qSt| |||<qSq!|d }|d }i}|D]}t|d}||t|<qi}|D]}g}|dD] }||j|q||t|<q|D]N\}}||vrq||}t|t|kr9tdt|d|d t|d|D] }||vrMtd|d|d|dkrZt||||<q;q|j|dS)a Define the default behavior to load a state_dict for ``_NamedOptimizer``. Sample Code ``` my_model = MyModule() optimizer = _NamedOptimizer(my_model.named_parameters(), Adagrad) ... optim_state_dict = optimizer.state_dict() ... ... optimizer.load_state_dict(optim_state_dict) ... ``` Args: state_dict (Dict[str, Any]) : A ``state_dict`` to load into the optimizer. Note that this state dict update is performed in place. .. note:: PyTorch is using lazy init to initialize the optim states. So it is possible that there is no optim state when user call ``load_state_dict`` and for ``_NamedOptimizer`` we make it stricter that users can only call ``load_state_dict`` after the state is initialized. By doing this, we can validate the optim ``state_dict`` to be loaded. r>rzJExpects the optim to be initialized before load but found not initialized.zExpects equal length as z for parameter z but found: zExpects state z but not found.z"Expects equal number of shards as z but found z for /rrz"Expects equal param_group size as z for group .zExpects group key z to be in group z in `state_dict` but is missing.N)r$r@_pre_load_state_dictlenr+ enumerater'r&r*r5r local_shardsziptensordetachcopy_rr6rr%_gen_param_group_keyr,load_state_dict)r-r@new_state_dictr> new_stateidx param_key state_keyr< src_state_val num_shardsnum_new_shardsshard src_shardsrc_param_groupsnew_param_groups src_group_mapr2rE new_group_map new_group group_key src_grouprGrrrr[s    $             z_NamedOptimizer.load_state_dictr9cCst|ts Jd|d}t|tjr|g|d<nt||d<dd|jD}|dD]}||vr7td|j ||q-|j ||j j |_ dS)z Add a param group to the :class:`_NamedOptimizer` s `param_groups`. Warning: This API is still in development and subject to change. r4rcSrrrrrrrrrz3_NamedOptimizer.add_param_group..z%some parameters are not in the moduleN) r5r"rr6r%rr*r+r'r,r$add_param_groupr)r-r9rr1rrrrrms     z_NamedOptimizer.add_param_groupcCs>|jD]}|jrt|}tj||_q|jdddS)z Run a dummy optimizer step, which allows to initialize optimizer state because we do lazy init for most optimizers. This allows doing in-place loading of optimizer state from a checkpoint. NrO) rr# requires_gradr zeros_likeautogradVariablegradrM)r-rtrrr init_state(s  z_NamedOptimizer.init_statecCs&t|jtrtj|j|j|ddS|S)NT)is_named_optimizer)r5rFSDPoptim_state_dict_to_loadr$r-r@rrrrR5s  z$_NamedOptimizer._pre_load_state_dictcCs"t|jtrt|j|j||SrK)r5rrvoptim_state_dictr$rxrrrrB>s z _NamedOptimizer._post_state_dict)NN).)rINrNrK)rN) __name__ __module__ __qualname____doc__rstrr rr6r r OptimizerrrrnnModuler3r!r"r@rrMrfloatpropertyr>r[rmrtrRrBrrrrrs:/ *$ h   rrErcCsdt|S)zGConcatenate all param keys as a unique indentifier for one param group.rP)joinrA)rErrrrZFsrZ) loggingr(collections.abcrrcopyrtypingrrrrr rtorch.nnrr 'torch.distributed._shard.sharded_tensorr torch.distributed.fsdpr rvr r%r~__annotations__ getLoggerrzloggerrrrZrrrrs       4