o hYN@s<UddlmZddlZddlmZddlmZddlmZddlm Z ddlm Z ddlm Z dd lm Z dd l mZdd lmZdd lmZdd lmZddlmZddlmZddlmZddlmZddlmZddlmZddlmZddlmZddlmZddlmZddlm Z ddlm!Z!ddlm"Z"ddlm#Z#er ddl$m%Z%ddl&Z'ddl(m)Z)ddl*m+Z+ddl,m-Z-dd l.m/Z/dd!l.m0Z0dd"l.m1Z1dd#l2m3Z3dd$l4m5Z5dd%l6m7Z7dd&l8m9Z9dd'lm:Z:dd(lm;Z;dd)lme Z?d+e@d,< Gd-d.d.ed/ZAdS)0) annotationsN)reduce)and_) TYPE_CHECKING)Any)Iterator)Mapping)Sequenceis_native_spark_like)evaluate_exprs)import_functions)import_native_dtypes) import_window)native_to_narwhals_dtype)InvalidOperationError)CompliantDataFrame)CompliantLazyFrame)Implementation)check_column_exists)find_stacklevel)generate_temporary_column_name)import_dtypes_module)not_implemented)parse_columns_to_drop) parse_version)validate_backend_version) ModuleType)Column) BaseDataFrameWindow)Self) TypeAlias)TypeIs) SparkLikeExprSparkLikeLazyGroupBySparkLikeNamespace)DType) JoinStrategy)LazyUniqueKeepStrategy)Version) _FullContextr# Incompletec@seZdZdd d Zed dZeddZedddZedddZ e dddZ dd d!Z dd#d$Z dd%d&Zdd'd(Zdd*d+Zdd-d.Zdd0d1Zedd3d4Zdd:d;Zdd>d?ZddBdCZddDdEZddFdGZddIdJZeddLdMZddNdOZddTdUZddXdYZdd]d^ZddcddZddgdhZ ddkdlZ!ddodpZ"ddwdxZ#ddydzZ$dddZ%e&'dZ(e&Z)e&'dZ*e&Z+dS)SparkLikeLazyFramenative_dataframeSQLFrameDataFramebackend_versiontuple[int, ...]versionr-implementationrreturnNonecCs6||_||_||_||_d|_d|_t|j|jdSN) _native_frame_backend_version_implementation_version_cached_schema_cached_columnsr)selfr1r3r5r6rAR/var/www/vscode/kcb/lib/python3.10/site-packages/narwhals/_spark_like/dataframe.py__init__:szSparkLikeLazyFrame.__init__cCtr ddlm}|St|jS)Nr) functions)r sqlframe.baserEr r<)r@rErArArB_FJ  zSparkLikeLazyFrame._FcCrD)Nr)types)rrFrIrr<)r@rIrArArB_native_dtypesSrHz!SparkLikeLazyFrame._native_dtypes type[Window]cCrD)Nrr )rsqlframe.base.windowr!rr<)r@r!rArArB_Window\rHzSparkLikeLazyFrame._WindowobjSQLFrameDataFrame | AnyTypeIs[SQLFrameDataFrame]cCst|Sr9r )rNrArArB _is_nativeeszSparkLikeLazyFrame._is_nativedatacontextr.r"cCs|||j|j|jdSNr3r5r6)r;r=r<)clsrRrSrArArB from_nativeis zSparkLikeLazyFrame.from_nativercCs |jSr9)r<to_native_namespacer@rArArB__native_namespace__rs z'SparkLikeLazyFrame.__native_namespace__r)cCs ddlm}||j|j|jdS)Nrr(rU)narwhals._spark_like.namespacer)r;r=r<)r@r)rArArB__narwhals_namespace__us z)SparkLikeLazyFrame.__narwhals_namespace__cCs|Sr9rArYrArArB__narwhals_lazyframe__~sz)SparkLikeLazyFrame.__narwhals_lazyframe__cCs|j|j|j||jdSrT) __class__nativer;r<)r@r5rArArB _with_versions z SparkLikeLazyFrame._with_versiondfcCs|j||j|j|jdSrT)r^r;r=r<)r@rarArArB _with_natives zSparkLikeLazyFrame._with_nativepa.Tablec CsN|jtjur|jdkrddl}z |j|jWSt y}z|dt |vrddl m }i}g}| }|D]Q\}}g||<z|||j} Wn9ty}z-|jj|j} |jj} t| | sotjd| d|td|||fWYd}~q9d}~ww||| fq9|jj|||dWYd}~Sd}~ww|jS) N)rzat least one RecordBatch)narwhals_to_native_dtypezCould not convert dtype z to PyArrow dtype, ) stacklevelschema)r<rPYSPARKr;pyarrowTable from_batchesr__collect_as_arrow ValueErrorstrnarwhals._arrow.utilsrecollect_schemaitemsr= ExceptionrhdataTyperJNullType isinstancewarningswarnrappendnull from_pydicttoArrow) r@paexcrerRrhcurrent_schemakeyvalue native_dtypenative_spark_dtype null_typerArArB_collect_to_arrows>    " z$SparkLikeLazyFrame._collect_to_arrowIterator[Column]ccs |jD] }|j|VqdSr9)columnsrGcol)r@rrArArB _iter_columnss z SparkLikeLazyFrame._iter_columns list[str]cCs.|jdur|jdurt|jn|jj|_|jSr9)r?r>listrhr_rrYrArArBrs   zSparkLikeLazyFrame.columnsbackend(ModuleType | Implementation | str | Nonekwargsr!CompliantDataFrame[Any, Any, Any]c Ks|tjurddl}ddlm}||jtjt||jddS|dus(|tj ur?ddl }ddl m }|| t||jddS|tjuraddl}ddl }ddlm}||| t||jdSd |} t| ) Nr)PandasLikeDataFrameT)r6r3r5validate_column_names)ArrowDataFrame)r3r5r)PolarsDataFrame)r3r5zUnsupported `backend` value: )rPANDASpandasnarwhals._pandas_like.dataframerr_toPandasrr=PYARROWrjnarwhals._arrow.dataframerrPOLARSpolarsnarwhals._polars.dataframer from_arrowrn) r@rrpdrr}rplrmsgrArArBcollects<       zSparkLikeLazyFrame.collect column_namesrocGs||jj|Sr9)rbr_select)r@rrArArB simple_selectsz SparkLikeLazyFrame.simple_selectexprsr%cG0t|g|R}dd|D}||jj|S)NcSg|] \}}||qSrAalias.0col_namerrArArB z0SparkLikeLazyFrame.aggregate..)r rbr_aggr@r new_columnsnew_columns_listrArArB aggregateszSparkLikeLazyFrame.aggregatecGr)NcSrrArrrArArBrrz-SparkLikeLazyFrame.select..)r rbr_rrrArArBrszSparkLikeLazyFrame.selectcGs&t|g|R}||jt|Sr9)r rbr_ withColumnsdict)r@rrrArArB with_columnsszSparkLikeLazyFrame.with_columns predicatecCs$||d}|j|}||SNr)_callr_whererb)r@r conditionspark_dfrArArBfilter s  zSparkLikeLazyFrame.filterdict[str, DType]cs(jdurfddjjD_jS)Ncs$i|]}|jt|jjjdqS))dtyper5 spark_types)namerrtr=rJ)rfieldrYrArB sz-SparkLikeLazyFrame.schema..)r>r_rhrYrArYrBrhs  zSparkLikeLazyFrame.schemacCs|jSr9rgrYrArArBrqsz!SparkLikeLazyFrame.collect_schemar Sequence[str]strictboolcCs t|||d}||jj|S)N)compliant_framerr)rrbr_drop)r@rrcolumns_to_droprArArBr"szSparkLikeLazyFrame.dropnintcCs||jj|dS)N)num)rbr_limit)r@rrArArBhead(szSparkLikeLazyFrame.headkeysdrop_null_keysr'cGsddlm}||||dS)Nrr&)r)narwhals._spark_like.group_byr')r@rrr'rArArBgroup_by+s zSparkLikeLazyFrame.group_byby descendingbool | Sequence[bool] nulls_lastcsht|tr |gt|}|rfdd|D}n fdd|D}ddt||D}jj|S)Nc3$|] }|r jjnjjVqdSr9)rGdesc_nulls_lastasc_nulls_lastrdrYrArB :  z*SparkLikeLazyFrame.sort..c3rr9)rGdesc_nulls_firstasc_nulls_firstrrYrArBr?rcSsg|]\}}||qSrArA)rrsort_frArArBrDz+SparkLikeLazyFrame.sort..)rvrlenziprbr_sort)r@rrr sort_funcs sort_colsrArYrBr0s   zSparkLikeLazyFrame.sortsubsetSequence[str] | NonecCs$|rt|nd}||jj|dS)Nr)rrbr_dropna)r@rrArArB drop_nullsGszSparkLikeLazyFrame.drop_nullsmappingMapping[str, str]cs8fddjD}jfdd|DS)Ncsi|] }|||qSrA)getrcolname)rrArBrLsz-SparkLikeLazyFrame.rename..c"g|] \}}j||qSrArGrrroldnewrYrArBrQ"z-SparkLikeLazyFrame.rename..)rrbr_rrr)r@rrename_mappingrA)rr@rBrenameKs zSparkLikeLazyFrame.renamekeepr,cCst|j||r t|nd}|dkrAtd|j}||p |j}|j||j d | |j |dk |}||S||jj|dS)Nnone*r)rrrrrM partitionByr_ withColumnrGcountoverrrrrbdropDuplicates)r@rrtmpwindowrarArArBuniqueUs   zSparkLikeLazyFrame.uniqueotherhowr+left_onright_onsuffixc sPj|j}|durt|ng|durt|ng}|dkr'fdd|Dn|}itt|fdd|D|jfddD} |dvra| fdd |Dn |dkrl|  fd dD} |dkrt t fd d t|| Dn|d krdn|} |dkrd n|}  jj | | d| S)Nfullcsg|]}|vr|qSrArArc) right_on_rArBrwrz+SparkLikeLazyFrame.join..cs&i|]}||vr|n|qSrArAr) left_columnsr rArBr~sz+SparkLikeLazyFrame.join..crrArrrYrArBrr>leftcrossinnerc3s |] }|vr|VqdSr9rAr)rrrArBrsz*SparkLikeLazyFrame.join..csg|]}|qSrArAr )rrArBrsc3s*|]\}}tj|t|kVqdSr9)getattrr_)rleft_key right_key) other_nativer@rArBrs  r full_outer)onr)rrrrr_rrrcopyextendvaluesrrrbjoin) r@rrrr r  right_columnsleft_on_right_cols_to_rename col_orderright_on_remappedon_ how_nativerA)rrrrr@r rBresR       zSparkLikeLazyFrame.joinc stj}}D]}||}||jkr d|d}t|q j}tdkr0d}t|j rF j j fdd|DSj rydfd d } j j fd d|Dj |dj fdd|DSd}t|)Nz-`explode` operation not supported for dtype `z`, expected List typerzExploding on multiple columns is not supported with SparkLike backend since we cannot guarantee that the exploded columns have matching element counts.c<g|]}|dkrj||nj||qSr)rGrr explode_outerrrrr@rArBr  z.SparkLikeLazyFrame.explode..rror7rcsj|j|dkBSr)rGisnull array_size)rrYrArBnull_conditionsz2SparkLikeLazyFrame.explode..null_conditioncr$r%)rGrrexploder'r(rArBrr)rcs<g|]}|dkrj||njd|qS)rN)rGrrlitr'r(rArBrr)z[Unreachable code, please report an issue at https://github.com/narwhals-dev/narwhals/issues)rror7r)rr=rqListrrrNotImplementedErrorr< is_pysparkrbr_r is_sqlframeunionrAssertionError) r@rdtypesrhcol_to_exploderrrr,rAr(rBr-sP         zSparkLikeLazyFrame.exploderindex variable_name value_namec Cs|jr|dkrd}t||dkrd}t|n |r!t|nd}|dur3tt|jt|nt|}|jj||||d}|durJ|j |}| |S)Nz<`variable_name` cannot be empty string for sqlframe backend.z9`value_name` cannot be empty string for sqlframe backend.rA)idsrvariableColumnNamevalueColumnName) r<r2r0tuplesetr differencer_unpivotrrb) r@rr7r8r9rr;runpivoted_native_framerArArBrAs* &  zSparkLikeLazyFrame.unpivotzO`LazyFrame.gather_every` is deprecated and will be removed in a future version.zG`LazyFrame.tail` is deprecated and will be removed in a future version.N) r1r2r3r4r5r-r6rr7r8)r7rK)rNrOr7rP)rRr2rSr.r7r")r7r)r7r))r7r")r5r-r7r")rar2r7r")r7rc)r7r)r7r)rrrrr7r)rror7r")rr%r7r")rr%r7r")r7r)rrrrr7r")rrr7r")rrorrr7r')rrorrrrr7r")rrr7r")rrr7r")rrrr,r7r") rr"rr+rrr rr ror7r")rrr7r") rrr7rr8ror9ror7r"),__name__ __module__ __qualname__rCpropertyrGrJrM staticmethodrQ classmethodrWrZr\r]r`rbrrrrrrrrrrhrqrrrrrrrrr-rAr deprecated gather_every join_asoftailwith_row_indexrArArArBr09s`            (  -            C C  r0)r%r2)B __future__rrw functoolsroperatorrtypingrrrrr narwhals._namespacer narwhals._spark_like.utilsr r rrrnarwhals.exceptionsrnarwhals.typingrrnarwhals.utilsrrrrrrrrrrIrrjr}sqlframe.base.columnrsqlframe.base.dataframerrLr!typing_extensionsr"r#r$narwhals._spark_like.exprr%rr'r[r)narwhals.dtypesr*r+r,r-r.r2r/__annotations__r0rArArArBs`