BNj rddlmZddlZddlmZddlmZddlm Z eje Z dd dZ d dZy) ) annotationsN)logging)is_dist_initializedctr|r*tjjj |}nnt j }t|Dcgc]}tj|}}t j||t j}|||<tj|dSd}tj||Scc}w)ap Gathers a tensor from each distributed rank into a list. Always retains gradients for the local rank's tensor, and optionally retains gradients for the gathered tensors if `with_grad` is True. Args: tensor (torch.Tensor): The tensor to gather from each rank. with_grad (bool, optional): If True, the local rank's tensor retains its gradients. Defaults to False. Returns: torch.Tensor: A tensor containing the gathered tensors from all ranks, concatenated along the first dimension. If torch.distributed is not available or not initialized, returns the original tensor. r)dimzTrying to gather while torch.distributed is not available or has not been initialized, returning the original (local) tensor. This is expected if you are only using one GPU; consider not using gathering to remove this warning.) rtorch distributednn all_gatherdistget_world_sizerange zeros_likeget_rankcatlogger warning_once)tensor with_gradgathered_tensors world_size_ local_rankwarnings w/Users/ahmed/devFolder/Ultron/claude-voice/.venv/lib/python3.12/site-packages/sentence_transformers/util/distributed.pyr r s $0033>>vF ,,.JBG BSTBSQ 0 0 8BS T OO,f 5J+1 Z (yy)q11 S   M# UsCct|dS)a Gathers a tensor from each distributed rank into a list, retaining gradients for the local rank's tensor. Args: tensor (torch.Tensor): The tensor to gather from each rank. Returns: torch.Tensor: A tensor containing the gathered tensors from all ranks, concatenated along the first dimension. If torch.distributed is not available or not initialized, returns the original tensor. T)r)r )rs rall_gather_with_gradr4s f --)F)r torch.Tensorrboolreturnr )rr r"r ) __future__rr torch.distributedr r transformers.utilsr environmentr get_logger__name__rr rrrr*s3" &,   H %$N .r