(HJj~ddlZddlZddlZddlZddlZddlZddlZddlZddlm Z ddl m Z ddl m Z mZmZmZmZmZmZmZddlmZddlmZej4ddj7dk(r ddlmZnddlmZej@ejBd dd l"m#Z#m$Z$m%Z%m&Z&d d l'm(Z(d dl'm)Z*dddddddddd Z+dZ,dZ-dej\dej\fdZ/dee0ej\fdee0e fdeee0ej\fee0e fffdZ1de2fd Z3d!Z4d"Z5 dVd#e0d$ee0d%ee0de fd&Z6d'Z7d(e de2fd)Z8d*d+de3fd(e d,e9d-e9d.eee0e fd/ee2geeejteffdeejte2ff d0Z;d1ejtd2e0dejtfd3ZZDd?ee0e fd@ee0e dffdAZEd?e0dBe0fdCZFd*dDdEee0e fd1ejtdFe9ddfdGZG dZd1ejtde2dHeeBdIeeBdJe0dKeee0ejtgee9e2ffdeejte2ffdLZHd1ejtdejtfdMZIde2dNee0e fddfdOZJ d[dPee0e fdQee0e fd1ejtdRe(dee0e fdFe9f dSZKdTZLd1ejtde9fdUZMy#e$r ed wxYw)\N)Path)dedent)AnyCallableDictListOptionalTupleTypeUnionMLXLM_USE_MODELSCOPEFalsetrue)snapshot_downloadz/Run `pip install modelscope` to use ModelScope.)ii) tree_flattentree_map tree_reducetree_unflatten)TokenizerWrapper)loadllamamistral3phixtralmamba deepseek_v3qwen2_vlminimax) mistralllavazphi-msft falcon_mambajoyai_llm_flashkimi_k2 qwen2_5_vl minimax_m2 iquestcodercdddddd}d}|D]}|js|dk(sn|dz } t|d|}||djj}t |||zS)Ng.AgeAr)MGMBGBr.)isdigitfloatstripupperint)xsizessplitxidigitssizes V/Users/ahmed/devFolder/claude-voice/.venv/lib/python3.12/site-packages/mlx_lm/utils.py _parse_sizer;<sCs#1 =E E c   1Ve9 F efI    $ $ &D vd # $$qweightreturncd}d|z}|j\}}||z}d|zdz }tjgd|z}|d|z |z}|j||S)N r)rr@rr').N)shapemxarrayreshape) r=bits pack_factor out_features packed_in in_featuresmaskshiftsunpackeds r:_unpack_awq_weightsrRHsp D*K%mmL)k)K I?D XX. /$ 6F "f,4H   L+ 66r<weightsquantization_configc||jdd}|dk7rtd|d|jdd}i}t|jD]j drtdd j d rdd }||d }|d }|d }||} d|z} |j \} } | | z} | |z}t |}|j}| | z}|j| || }tj| |z}|jtj|zjdjtj}tj| j} ||vr@||}t |}|j}|jtj | z}nz)_transform_awq_weights..s /QVCLL /Qs)rXrZrY)rVrJ)get ValueErrorlistkeysr`rFrRTrIrGarangeastypeuint32sum contiguousfloat32fullr]anyitems issubdtypefloating)rSrTrJrV new_weightsprefixr= scales_key qzeros_keyscalesrKrN packed_outrLn_groupsunpacked_weightrMrepackedrPweightqzerosunpacked_zerosbiases zero_point model_dtypekwmlx_quantizationrcs @r:_transform_awq_weightsrSs  " "61 -D qy;'MNOO$((s;JKGLLN# << !AA  << #"XF12G"87+J"87+JZ(F *K&-mm #K% 3L"j0H2':O-//O${2I&..|Y THYY{+d2F+v5:::CJJ299U ]]688,FW$ ,"5V!<!/!1!1 )// ;;fD4!8_  {"**MPVV.4K6('* +.4K6('* +.4mmFLL.IK6('* + ,,K /Q   's|K G$J!!#1 =="++ .XXk2KN$ ! ( ((r<configc|d}tj||} tjd|}|j |jfS#t$rd|d}t |wxYw)z Retrieve the model and model args classes based on the configuration. Args: config (dict): The model configuration. Returns: A tuple containing the Model class and the ModelArgs class. model_typezmlx_lm.models.z Model type z not supported.)MODEL_REMAPPINGre importlib import_module ImportErrorrfModel ModelArgs)rrarchmsgs r: _get_classesrsz %J $$Zz&get_total_parameters..s 1bii0Hr<is_leafct|drMt|dsdn|jj}||jjdz|jzzSt dt |jDS)NrJbiasrrAc3:K|]\}}|jywr_)r9)ra_vs r:rdz8get_total_parameters..nparams..sC&Bda166&Bs)hasattrrr9r~rJrmr parameters)rns r:nparamsz%get_total_parameters..nparamssa 1f  F+Aqxx}}r)QVV33 3Cl1<<>&BCCCr<c34K|]\}}|ywr_)rarrrs r:rdz'get_total_parameters..s3ldawqzls)r leaf_modulesrm)modelrrs @r:get_total_parametersrs5 &HLD 3l3 33r<cDtd|d}t|}|dz|z S)NcXt|tjr||jzS|Sr_)rrGrHnbytes)accr4s r:rz)compute_bits_per_weight..s!Arxx)@sQXX~IcIr<r)rr)r model_bytes model_paramss r:compute_bits_per_weightrs/I5RSK(.L ?\ ))r<path_or_hf_reporevisionallow_patternsczt|}|js|xsgd}tt|||}|S)a Ensures the model is available locally. If the path does not exist locally, it is downloaded from the Hugging Face Hub. Args: path_or_hf_repo (str): The local path or Hugging Face repository ID of the model. revision (str, optional): A revision id which can be a branch name, a tag, or a commit hash. Returns: Path: The local file path. ) *.jsonmodel*.safetensors*.pytokenizer.model *.tiktokentiktoken.model*.txt*.jsonl*.jinja)rr)rexistsr)rrr model_paths r: _downloadrsN o&J    '  ,  !-   r<c.tt|dS)NT)local_files_only)rr)hf_repos r:hf_repo_to_pathrs !'DA BBr<rcxt|dz d5}tj|}ddd|dz }|jrFi} t|d5}tj|}ddd|j ddx}r|d<S#1swYfxYw#1swY0xYw#tj$rYHwxYw)N config.jsonrgeneration_config.json eos_token_idF)openjsonrrJSONDecodeErrorre)rfrgeneration_config_filegeneration_configrs r: load_configrs j=(# .!1 /(*BB$$& ,c2a$(IIaL!3 -00G G< G%1F> " M / .32##   s5B  B#B(B# BB B##B98B9FTlazystrict model_configget_model_classesct||j|tjt|dz }|s|rt d|i|D]&}jt j |(jdx}vtjjd||z }tjj|} |jj| | j| j} } n |\} } dvrjdi} d| vr| dd<| j!} | | t#d rj%fd }jd dx} ||njdd x}r{|d }|dk(rddlm}||na|dk(rdddd}|d <|d<||nC|dk(rdddd}|d <|d<||n%|dvr!t+|\}|d <|d<||jdd rHd}t-|j/t0j2j4}j7|j9j;t=j?||s#t j8jAfS)aB Load and initialize the model from a given path. Args: model_path (Path): The path to load the model from. lazy (bool): If False eval the model parameters to make sure they are loaded in memory before returning, otherwise they will be loaded when needed. Default: ``False`` strict (bool): Whether or not to raise an exception if weights don't match. Default: ``True`` model_config (dict, optional): Optional configuration parameters for the model. Defaults to an empty dictionary. get_model_classes (Callable[[dict], Tuple[Type[nn.Module], Type]], optional): A function that returns the model class and model args class given a config. Defaults to the ``_get_classes`` function. Returns: Tuple[nn.Module, dict[str, Any]]: The loaded and initialized model and config. Raises: FileNotFoundError: If the weight files (.safetensors) are not found. ValueError: If the model class or args class are not found or cannot be instantiated. NrzNo safetensors found in model_file custom_model)rrT text_configsanitizec rfd}tj|d|d|jdd|y)NcJ|dvrd|St|dsy|dvS)N quantization to_quantizedFrY)r)prrrSs r:class_predicatez6load_model.._quantize..class_predicate]s>F>**n-a001n-S=G+ +r<rVrJmodeaffine)rVrJrr)rquantizere)rrrrrSs r: _quantizezload_model.._quantize\s< , #L1f%!!&(3+  r<rF quant_methodbitnetr)bitnet_quantizemxfp4rAr@rVrJrzcompressed-tensorsr)awqgptqquantize_activationscjt|tjr|jdvr t d|j ddr t d|j j\}}|d|jzz}tj|||j|j|jS|S)N)nvfp4mxfp8z8Mode ({m.mode}) does not support activation quantizationrFz?Linear layer with bias does not support activation quantizationrA) rrQuantizedLinearrrfrer~rFrJQQLinearrV)rout_dimsin_dimss r: _maybe_qqzload_model.._maybe_qqs!R//066!33$R55'$Y%&HHNN!'2<'{{7HallAFFAFFSSr<r)r)!rupdateglobstrFileNotFoundErrorrGrrerutilspec_from_file_locationmodule_from_specloader exec_modulerr from_dictrrmodels.bitlinear_layersrrrrrr is_moduleupdate_moduleseval load_weightsrgrrr)rrrrr weight_fileswfrspecr model_classmodel_args_classr model_argsrrrTrrrleavesrrrSs @@@r: load_modelr s< $F l#99S.B!BCDL F"::, GHHGrwwr{#jj.. ;~~55   # ~~..t4 %(, DNN% (9(H% %F*jj3 K /,78M,NF( )!++F3J  #Euj!..) " >488 E, & +@% H H  H*>: 8 # @#E+>?E W $*,aIL%1F> ",8F( ) l # 1 1*,aJL%1F> ",8F( ) l # _ ,$:7DW$X !G\%1F> ",8F( ) l # zz(%0  )U%7%7%9299CVCVW V$ JJL tGMMO,V<    "# &=r<r adapter_pathc ddlm}|||S)Nr) load_adapters) tuner.utilsr)rr _load_adapterss r:rrs< % ..r<c<t|gd}t|||S)z`Load a huggingface tokenizer and try to infer the type of streaming detokenizer to use. rrrrrrrrr eos_token_ids)r_load_tokenizer)rtokenizer_config_extrars r:load_tokenizerrs.   J # r<tokenizer_config return_configct||}t|||\}} |t||}|jt ||| j dd} |r|| | fS|| fS)aT Load the model and tokenizer from a given path or a huggingface repository. Args: path_or_hf_repo (Path): The path or the huggingface repository to load the model from. tokenizer_config (dict, optional): Configuration parameters specifically for the tokenizer. Defaults to an empty dictionary. model_config(dict, optional): Configuration parameters specifically for the model. Defaults to an empty dictionary. adapter_path (str, optional): Path to the LoRA adapters. If provided, applies LoRA layers to the model. Default: ``None``. lazy (bool): If ``False`` eval the model parameters to make sure they are loaded in memory before returning, otherwise they will be loaded when needed. Default: ``False`` return_config (bool: If ``True`` return the model config as the last item.. revision (str, optional): A revision id which can be a branch name, a tag, or a commit hash. Returns: Union[Tuple[nn.Module, TokenizerWrapper], Tuple[nn.Module, TokenizerWrapper, Dict[str, Any]]]: A tuple containing the loaded model, tokenizer and, if requested, the model config. Raises: FileNotFoundError: If config file or safetensors are not found. ValueError: If model class or args class are not found. )r)rNrr)rr rrrre) rrrr rrrrrr tokenizers r:rrsxH?X>Jz4lKME6e\2 $FJJ~t4TIi''ir<)rpipeline_group tensor_groupct|gd}t|dd\}}t|dxrt|jd}t|d} | |s t d | | s t d |s | s t d ||cxurDnnA| rt j j}n |rt j j}||jj|t|d z d 5} tj| d} dddt} t|jD]:\} } j| ddux}r t d| j!| | <t|| n t|t#||xsddi|jdd}t|dd\}}||j%|||jj|t j&|jt j&t j j)t j*dt j,|r|||fS||fS#1swYkxYw)NrrTF)rrrpipelineshardzGThe model does not support pipelining but a pipeline_group was providedzMThe model does not support tensor parallelism but a tensor_group was providedz'The model does not support any shardingmodel.safetensors.index.jsonr weight_mapz>..0L ^^002N! ^,*==s Cs99S>,7LDe  !1!1!34DAq(,,Q5==y= R OOLO , 5 ${3$70$7jj6I *4>HE1 L!! ^,GGE   GGBNN " "288C= " @Ai''iED Cs 5I77JcVt|tjjd|Sr_)r4rGr'r()r-rs r: pipeline_loadr6Rs bnn113T= IIr<max_file_size_gbc|dz}g}id}}|jD]@\}}||jz|kDr|j|id}}|||<||jz }B|j||S)z Splits the weights into smaller shards. Args: weights (dict): Model weights. max_file_size_gb (int): Maximum size of each shard in gigabytes. Returns: list: List of weight shards. r)rrrappend)rSr7max_file_size_bytesshardsr" shard_sizerrs r: make_shardsr>Vs+b0 FA:E 1  #6 6 MM% "A:Eaahh   MM% Mr<pathhf_pathcddlm}m}||j|d}n|j |}d|j _d|j _|j jdg|j _n8d|j jvr |j xjdgz c_|t||j _ d|_ |jtjj|d y) z Uploads the model to Hugging Face hub. Args: path (Union[str, Path]): Local path to the model. hf_path (Union[str, Path, None]): Path to the original Hugging Face model. r) ModelCard ModelCardDataNen)languagemlxztext-generationr- README.md)huggingface_hubrBrC from_templaterdata library_name pipeline_tagtagsr base_modeltextsaveosr?join)r?r@rBrCcards r:create_model_cardrTns9&&}d'CD~~g&"DII.DII yy~~  diinn $ 5'!"7| DIIIbggll4-.r< upload_repoc ddlm}m}m}ddlm}|j t|dz }|j|}|jj}|d|d|d |d|d |d } nd } td |d| d|d|_ |j||} | j|d| j||dt!d|dy)z Uploads the model to Hugging Face hub. Args: path (str): Local path to the model. upload_repo (str): Name of the HF repo to upload to. r)HfApirBloggingr) __version__rGNz This model [z](https://huggingface.co/z,) was converted to MLX format from [z!) using mlx-lm version **z **. r-z # z z ## Use with mlx ```bash pip install mlx-lm ``` ```python from mlx_lm import load, generate model, tokenizer = load("av") prompt = "hello" if tokenizer.chat_template is not None: messages = [{"role": "user", "content": prompt}] prompt = tokenizer.apply_chat_template( messages, add_generation_prompt=True, return_dict=False, ) response = generate(model, tokenizer, prompt=prompt, verbose=True) ``` T)repo_idexist_okr) folder_pathrZ repo_typez0Upload successful, go to https://huggingface.co/z for details.)rHrWrBrXr-rYset_verbosity_inforrrJrNrrOrP create_repoupload_large_folderprint) r?rUrWrBrXrY card_pathrSr@ provenanceapis r: upload_to_hubres:9  T [(I >>) $Dii""G M!:;-H''.i/H R +}-    -  "#. /  DI6 IIi 'COOK$O7  <[M WXr< donate_model save_pathrgct|tr t|}|jddt t |j }t|}t|}|dkDrdnd}td|jD}|t|did}|r*|jtd |j |j~tt|D]g} || } d || <|j!| dz|} || z } t#j$t| | d d i | j'D] } | |d| < ~ it)|dDcic] }||d| c}|d<t+|dz d5}t-j.||dd d d y cc}w#1swYy xYw)z?Save model weights and metadata index into specified directory.T)parentsr[rz"model-{:05d}-of-{:05d}.safetensorszmodel.safetensorsc34K|]}|jywr_)r)rars r:rdzsave_model..s8'7!QXX'7s) total_sizetotal_parameters)metadatar$c,tjgSr_)rGrH)rs r:rzsave_model..s  r<NformatrF)rnr$r#rr@indent)rrrmkdirdictrrr>lenrmvaluesrrrclearrangerprGsave_safetensorsrhsortedrrdump)rhrrgrSr< shards_countshard_file_formatrl index_datair" shard_name shard_path weight_namerrs r: save_modelrs)S!O  OOD4O0< 0 0 234G  !Fv;L !  -  8w~~'788J% 4U ;  J X4e6F6F6HIJ MMO 3v; q q &--a!e\B +  C OUh=NO ::J| $[ 1(  17z,7O0P 0P1:l #A &&0P J| i88# >!    ? >   ? >s,F3F88GrVrJrquant_predicatec4 d}tj| xs t|dd|||\}||d d vrd nd d<   fd}tj||||  d d <t |}t d |d d | fS)a Applies quantization to the model weights. Args: model (nn.Module): The model to be quantized. config (dict): Model configuration. group_size (Optional[int]): Group size for quantization. bits (Optional[int]): Bits per weight for quantization. mode (str): The quantization mode. quant_predicate (Callable): A callable that decides how to quantize each layer based on the path. Accepts the layer `path` and the `module`. Returns either a bool to signify quantize/no quantize or a dict of quantization parameters to pass to `to_quantized`. Returns: Tuple: Tuple containing quantized model and config. c8ddddd}||\}}|xs||xs|fS)N)@r@)rAr@)r@)rAr)rrrrr)rrVrJ mode_defaultsdefault_group_size default_bitss r:defaults_for_modez)quantize_model..defaults_for_mode s=  ,9+>(L//1EEEr<rNrrTFct|dsy|jjdzdk7ryd} ||}t|tr |d|<|Sr |rd|<|S)NrFr[rTr)rr~rFrrt)r?modulebool_or_paramsfine_grained_configrV quant_paramsrquantized_configs r:wrapped_predicatez)quantize_model..wrapped_predicate7sv~. ==  r "Z /1 4  &,T6:N nd +5C ^ ,T 2!^5A ^ ,T 2r<)rrrTz[INFO] Quantized model with z.3fz bits per weight.)copydeepcopygetattrrrrra) rrrVrJrrrrbpwrrrs ` ` @@@r:quantize_modelrs4F}}V,%P8I4)PO(z4@J",dDIL))##+7(  KK  ) /?~.N*+ !% (C (S 1B CD " ""r<c ddlm}m}g}|jD]\}}d|v}t |t j rt j}d|i}nAt |t jri}t j}nt ||rd|i}|}n{tj|j|j|j|j|j |j"} | j$ddd} || i|} |r|j&| _| | _ |j)|| ft+|dkDr|j-t/||S)z Dequantize the quantized layers in the model. Args: model (nn.Module): The model with quantized layers. Returns: nn.Module: The model with dequantized layers. r)QuantizedSwitchLinear SwitchLinearrNr[r)models.switch_layersrr named_modulesrrrLinearQuantizedEmbedding EmbeddingrG dequantizer~ryrrVrJrrFrr:rurr) rrrdequantize_layersnamerrclskwargsr~argsrs r:dequantize_modelrUs8J++- f fb00 1))Cd^F  5 5 6F,,C  5 6d^FC  MM MM MM    KK KK  ||DbD!    [[AF  $+5.8 ! ^,=>? Lr< config_pathc|jdd|jddd|vr|d|d<tt|j}t |d5}t j ||ddddy#1swYyxYw) a Save the model configuration to the ``config_path``. The final configuration will be sorted before saving for better readability. Args: config (dict): The model configuration. config_path (Union[str, Path]): Model configuration file path. _name_or_pathN vision_configrrTrr@rq)poprtrzrrrrr{)rrr0s r: save_configrsy JJ% JJ%(.~(>$%&( )F k3 3 &#a(  s BB dst_pathsrc_path_or_reporclt|}|js|}t|}nd}t|}t||dt ||dz |j |dD]>}t j t||z D]} tj| |@t||y)NTrfr)r)rr) rrrrrsave_pretrainedrrshutilrrT) rrrrrrgsrc_pathrrfiles r:rPrPs$%H ?? ""7+H~HxT2H}$<= h' /IIc(Q,/0D KKh '10h(r<ctt|t|}t|D]}||||k7s|cS|S)a$ Calculates the length of the common prefix of two lists. Args: list1: The first list of strings. list2: The second list of strings. Returns: The length of the common prefix. Returns 0 if lists are empty or do not match at the first element. )minrurx)list1list2min_lenrs r:common_prefix_lenrsD#e*c%j)G7^ 8uQx H Nr<c tj|j}d|jvS#tt f$rYywxYw)z Check if the model supports input_embeddings in its call signature. Args: model (nn.Module): The model to check. Returns: bool: True if the model supports input_embeddings, False otherwise. input_embeddingsF)inspect signature__call__rrf TypeError)rrs r:#does_model_support_input_embeddingsrsC%%enn5 !Y%9%999  "s,/AA)NN)NNNFFN)NNF)F)rN)T)NrrrrrrQresourcerpathlibrtextwraprtypingrrrrr r r r mlx.corecorerGmlx.nnrgetenvlower modelscoperrrH setrlimit RLIMIT_NOFILE mlx.utilsrrrrtokenizer_utilsrrrrMAX_FILE_SIZE_GBr;rHrRrrrtrrrrrrboolrr rrr'Groupr4r6r3rgr>rTrerrrrrPrrrr<r:rs       299 #W-335?M028))<8II.4 $  %77bhh7Y) #rxx- Y)c3hY) 4RXX S#X ./Y)x&&* 4*# $&&sm&I& &RCDT*-1HT JJ J J4S>* J  d299ot.C(D DE J  299d? JZ//#/"))/ 426-1"&"1 1 tCH~.1 4S>*1 3- 1  1  1 sm1   "))% %& "))%tCH~ 5681 l6:37 V 26 V R^^112V 2>>//0V  V tCH~. V rJ8HD0/E#t),/uS$_7M/4?Y?Y#?YL 9 S$Y9 999  9  9 BOS L# 99L# L# L# 3- L#  L# hRYY'7tTz9J'JKL L# 299d?L#^+BII+"))+\) )sDy!) )@ )CI)CI&) 99) ) cN )  )84 ryy T k MKLLMs 2MM