o Ûñ hß}ã @s ddlZddlZddlZddlZddlmZddlm Z ddlm Z m Z zddl m Z Wn ey7dZ Ynwejj ZdCdd„Zdd„Zej d d d g¡d d „ƒZej de d¡ddf¡dd„ƒZdd„Zej d d d g¡dd„ƒZdd„Zdd„Zej d d d g¡dd„ƒZdd„Zd d!„Zej d d d g¡d"d#„ƒZd$d%„Zej d d d g¡d&d'„ƒZ ejj!d(d)„ƒZ"ejj!d*d+„ƒZ#ej d,e $e %d-¡d.¡e $e %d-¡e %d/¡¡e %d-¡d0kg¡d1d2„ƒZ&d3d4„Z'd5d6„Z(d7d8„Z)d9d:„Z*d;d<„Z+d=d>„Z,d?d@„Z-dAdB„Z.dS)DéN)Útobytes)Ú ArrowInvalidÚArrowNotImplementedErroré cCsddlm}|t ¡|ƒS)Nr)Ú_get_udf_context)Úpyarrow._computerÚpaÚdefault_memory_pool)Ú batch_lengthr©r úP/var/www/vscode/kcb/lib/python3.10/site-packages/pyarrow/tests/test_substrait.pyÚmock_udf_context&s r cCsTtj t|ƒ|¡}tjj||jd}| |¡Wdƒ|S1s#wY|S)N©Úschema) ÚosÚpathÚjoinÚstrrÚipcÚRecordBatchFileWriterrÚ write_table)ÚtmpdirÚ file_nameÚtablerÚwriterr r r Ú_write_dummy_data_to_disk+s  ÿþrÚ use_threadsTFc Cs†d}d}tjgd¢gdgd}t|||ƒ}t| dt |¡ ¡¡ƒ}tj  |¡}t j ||d}|  ¡} |  dg¡|   dg¡ksAJ‚dS)Nax { "version": { "major": 9999 }, "relations": [ {"rel": { "read": { "base_schema": { "struct": { "types": [ {"i64": {}} ] }, "names": [ "foo" ] }, "local_files": { "items": [ { "uri_file": "FILENAME_PLACEHOLDER", "arrow": {} } ] } } }} ] } zread_data.arrow©éééééÚfoo©ÚnamesÚFILENAME_PLACEHOLDER©r©rrrrÚreplaceÚpathlibÚPathÚas_uriÚ _substraitÚ_parse_json_planÚ substraitÚ run_queryÚread_allÚselect© rrÚsubstrait_queryrrrÚqueryÚbufÚreaderÚres_tbr r r Útest_run_serialized_query2s ÿ  r9r5sbuffersbytesrcCs t|tjtfƒs/dt|ƒ›d}tjt|dt  |¡WdƒdS1s(wYdSd}tjt |dt  |¡WdƒdS1sIwYdS)Nz)Expected 'pyarrow.Buffer' or bytes, got 'ú'©Úmatchz1ParseFromZeroCopyStream failed for substrait.Plan) Ú isinstancerÚBufferÚbytesÚtypeÚpytestÚraisesÚ TypeErrorr/r0r)rr5Úmsgr r r Útest_run_query_input_types`s  ÿþ "ÿrEcCsVd}tj t|ƒ¡}d}tjt|dt |¡WdƒdS1s$wYdS)Nz2 { "relations": [ ] } zPlan has no relationsr;) rr-r.rrArBrr/r0)r5r6Ú exec_messager r r Útest_invalid_planps  "ÿrGc CsŠd}d}tjgd¢gdgd}t|||ƒ}t| dt |¡ ¡¡ƒ}tj  t|ƒ¡}t j ||d}|  ¡} |  dg¡|   dg¡ksCJ‚dS)NaÛ { "version": { "major": 9999 }, "relations": [ {"rel": { "read": { "base_schema": { "struct": { "types": [ {"i64": {}} ] }, "names": [ "bar" ] }, "local_files": { "items": [ { "uri_file": "FILENAME_PLACEHOLDER", "arrow": {}, "metadata" : { "created_by" : {}, } } ] } } }} ] } zbinary_json_data.arrowrÚbarr$r&r'r(r3r r r Ú(test_binary_conversion_with_json_options}s! ÿ rIcCs,|›d|›}|D] }| |¡rdSq dS)Nú#TF)Úendswith)ÚfnsÚext_fileÚfn_nameÚsuffixÚfnr r r Ú has_function¯s  ÿrQcCs.tj ¡}t|ddƒs J‚t|ddƒsJ‚dS)Nzfunctions_arithmetic.yamlÚaddÚsum)rr-Úget_supported_functionsrQ)Úsupported_functionsr r r Útest_get_supported_functions·s ÿ ÿrVcsŽtj dgd¢i¡‰tj dgd¢i¡‰t t dt ¡¡g¡‰‡‡‡fdd„}d}tj t|ƒ¡}tj j |||d}|  ¡}|ˆksEJ‚dS)NÚx©rrr )r!r"écs@|stdƒ‚|ddkr|ˆksJ‚ˆS|ddkrˆStdƒ‚)NúNo names providedrÚt1rÚt2úUnrecognized table name©Ú Exception©r%r©Úschema_1Ú test_table_1Ú test_table_2r r Útable_providerÇs   z(test_named_table..table_providerað { "version": { "major": 9999 }, "relations": [ {"rel": { "read": { "base_schema": { "struct": { "types": [ {"i64": {}} ] }, "names": [ "x" ] }, "namedTable": { "names": ["t1"] } } }} ] } ©rer) rÚTableÚ from_pydictrÚfieldÚint64r-r.rr/r0r1)rrer4r6r7r8r rar Útest_named_tableÁs ÿrkcsztj dgd¢i¡‰‡fdd„}d}tj t|ƒ¡}d}tjt|dt j ||dWdƒdS1s6wYdS) NrWrXcó$|stdƒ‚|ddkrˆStdƒ‚©NrZrr[r]r^©r%Ú_©rcr r reõó  z;test_named_table_invalid_table_name..table_providerað { "version": { "major": 9999 }, "relations": [ {"rel": { "read": { "base_schema": { "struct": { "types": [ {"i64": {}} ] }, "names": [ "x" ] }, "namedTable": { "names": ["t3"] } } }} ] } zInvalid NamedTable Sourcer;©re) rrgrhr-r.rrArBrr/r0)rer4r6rFr rpr Ú#test_named_table_invalid_table_nameòs "ÿrscs‚tj dgd¢i¡‰‡fdd„}d}t|ƒ}tj t|ƒ¡}d}tjt|dt j ||dWdƒdS1s:wYdS) NrWrXcrlrmr^rnrpr r rerqz4test_named_table_empty_names..table_provideraì { "version": { "major": 9999 }, "relations": [ {"rel": { "read": { "base_schema": { "struct": { "types": [ {"i64": {}} ] }, "names": [ "x" ] }, "namedTable": { "names": [] } } }} ] } z!names for NamedTable not providedr;rr) rrgrhrr-r.rArBrr/r0)rer4r5r6rFr rpr Útest_named_table_empty_namess "ÿrtc s~tj dgd¢i¡‰‡fdd„}d}tj |¡}tjj|||d}| ¡}|\}}ˆ dd|t d ƒˆdƒ¡} || ks=J‚dS) NrWrXcrlrmr^rn©Ú test_tabler r reJrqz.test_udf_via_substrait..table_providers { "extensionUris": [ { "extensionUriAnchor": 1 }, { "extensionUriAnchor": 2, "uri": "urn:arrow:substrait_simple_extension_function" } ], "extensions": [ { "extensionFunction": { "extensionUriReference": 2, "functionAnchor": 1, "name": "y=x+1" } } ], "relations": [ { "root": { "input": { "project": { "common": { "emit": { "outputMapping": [ 1, 2, ] } }, "input": { "read": { "baseSchema": { "names": [ "t", ], "struct": { "types": [ { "i64": { "nullability": "NULLABILITY_REQUIRED" } }, ], "nullability": "NULLABILITY_REQUIRED" } }, "namedTable": { "names": [ "t1" ] } } }, "expressions": [ { "selection": { "directReference": { "structField": {} }, "rootReference": {} } }, { "scalarFunction": { "functionReference": 1, "outputType": { "i64": { "nullability": "NULLABILITY_NULLABLE" } }, "arguments": [ { "value": { "selection": { "directReference": { "structField": {} }, "rootReference": {} } } } ] } } ] } }, "names": [ "x", "y", ] } } ] } rfrÚyr) rrgrhr-r.r/r0r1Ú add_columnr ) Úunary_func_fixturerrer4r6r7r8ÚfunctionÚnameÚ expected_tbr rur Útest_udf_via_substraitFs  eÿ  ÿr}cs‚tj dgd¢i¡‰‡fdd„}d}tj |¡}t tj¡}tjj ||dWdƒn1s1wYdt |j ƒvs?J‚dS)NrWrXcrlrmr^rnrur r reÅrqz=test_udf_via_substrait_wrong_udf_name..table_providers { "extensionUris": [ { "extensionUriAnchor": 1 }, { "extensionUriAnchor": 2, "uri": "urn:arrow:substrait_simple_extension_function" } ], "extensions": [ { "extensionFunction": { "extensionUriReference": 2, "functionAnchor": 1, "name": "wrong_udf_name" } } ], "relations": [ { "root": { "input": { "project": { "common": { "emit": { "outputMapping": [ 1, 2, ] } }, "input": { "read": { "baseSchema": { "names": [ "t", ], "struct": { "types": [ { "i64": { "nullability": "NULLABILITY_REQUIRED" } }, ], "nullability": "NULLABILITY_REQUIRED" } }, "namedTable": { "names": [ "t1" ] } } }, "expressions": [ { "selection": { "directReference": { "structField": {} }, "rootReference": {} } }, { "scalarFunction": { "functionReference": 1, "outputType": { "i64": { "nullability": "NULLABILITY_NULLABLE" } }, "arguments": [ { "value": { "selection": { "directReference": { "structField": {} }, "rootReference": {} } } } ] } } ] } }, "names": [ "x", "y", ] } } ] } rrzNo function registered) rrgrhr-r.rArBÚ ArrowKeyErrorr/r0rÚvalue)rer4r6Úexcinfor rur Ú%test_udf_via_substrait_wrong_udf_nameÂs  eÿrcsrtj dgd¢i¡‰‡fdd„}d}tj t|ƒ¡}tjj|||d}| ¡}tj dgd¢i¡}||ks7J‚dS)NrWrXcóˆS©Nr r`©Úin_tabler r re<óz/test_output_field_names..table_provideraø { "version": { "major": 9999 }, "relations": [ { "root": { "input": { "read": { "base_schema": { "struct": { "types": [{"i64": {}}] }, "names": ["x"] }, "namedTable": { "names": ["t1"] } } }, "names": ["out"] } } ] } rfÚout) rrgrhr-r.rr/r0r1)rrer4r6r7r8Úexpectedr r„r Útest_output_field_names8s ÿr‰cs€tj gd¢gd¢gd¢dœ¡‰‡fdd„}d}tj |¡}tjj||dd }| ¡}tj d d gd d gdœ¡}||ks>J‚dS)N©rrrr)rrr r!)çð?r‹r‹r‹)ÚkÚv1Úv2cr‚rƒr rnrur r rekr†z7test_scalar_aggregate_udf_basic..table_providers { "extensionUris": [ { "extensionUriAnchor": 1, "uri": "urn:arrow:substrait_simple_extension_function" }, ], "extensions": [ { "extensionFunction": { "extensionUriReference": 1, "functionAnchor": 1, "name": "sum_mean" } } ], "relations": [ { "root": { "input": { "extensionSingle": { "common": { "emit": { "outputMapping": [ 0, 1 ] } }, "input": { "read": { "baseSchema": { "names": [ "k", "v1", "v2", ], "struct": { "types": [ { "i64": { "nullability": "NULLABILITY_REQUIRED" } }, { "i64": { "nullability": "NULLABILITY_NULLABLE" } }, { "fp64": { "nullability": "NULLABILITY_NULLABLE" } } ], "nullability": "NULLABILITY_REQUIRED" } }, "namedTable": { "names": ["t1"] } } }, "detail": { "@type": "/arrow.substrait_ext.SegmentedAggregateRel", "segmentKeys": [ { "directReference": { "structField": {} }, "rootReference": {} } ], "measures": [ { "measure": { "functionReference": 1, "phase": "AGGREGATION_PHASE_INITIAL_TO_RESULT", "outputType": { "fp64": { "nullability": "NULLABILITY_NULLABLE" } }, "arguments": [ { "value": { "selection": { "directReference": { "structField": { "field": 1 } }, "rootReference": {} } } }, { "value": { "selection": { "directReference": { "structField": { "field": 2 } }, "rootReference": {} } } } ] } } ] } } }, "names": [ "k", "v_avg" ] } } ], } Frfrrg@g@)rŒÚv_avg©rrgrhr-r.r/r0r1©Úvarargs_agg_func_fixturerer4r6r7r8r|r rur Útest_scalar_aggregate_udf_basiccs" ÿÿ  }ÿþr“csŒtj gd¢gd¢gd¢gd¢dœ¡‰‡fdd„}d}tj |¡}tjj||d d }| ¡}tj gd ¢gd ¢gd ¢dœ¡}||ksDJ‚dS)N)rrrrrrrr)rrrrrrrr)rrr r!r"rYéé)r‹r‹r‹r‹g@g@g@g@)ÚtrŒrrŽcr‚rƒr rnrur r rer†z5test_hash_aggregate_udf_basic..table_providers  { "extensionUris": [ { "extensionUriAnchor": 1, "uri": "urn:arrow:substrait_simple_extension_function" }, ], "extensions": [ { "extensionFunction": { "extensionUriReference": 1, "functionAnchor": 1, "name": "sum_mean" } } ], "relations": [ { "root": { "input": { "extensionSingle": { "common": { "emit": { "outputMapping": [ 0, 1, 2 ] } }, "input": { "read": { "baseSchema": { "names": [ "t", "k", "v1", "v2", ], "struct": { "types": [ { "i64": { "nullability": "NULLABILITY_REQUIRED" } }, { "i64": { "nullability": "NULLABILITY_REQUIRED" } }, { "i64": { "nullability": "NULLABILITY_NULLABLE" } }, { "fp64": { "nullability": "NULLABILITY_NULLABLE" } } ], "nullability": "NULLABILITY_REQUIRED" } }, "namedTable": { "names": ["t1"] } } }, "detail": { "@type": "/arrow.substrait_ext.SegmentedAggregateRel", "groupingKeys": [ { "directReference": { "structField": { "field": 1 } }, "rootReference": {} } ], "segmentKeys": [ { "directReference": { "structField": {} }, "rootReference": {} } ], "measures": [ { "measure": { "functionReference": 1, "phase": "AGGREGATION_PHASE_INITIAL_TO_RESULT", "outputType": { "fp64": { "nullability": "NULLABILITY_NULLABLE" } }, "arguments": [ { "value": { "selection": { "directReference": { "structField": { "field": 2 } }, "rootReference": {} } } }, { "value": { "selection": { "directReference": { "structField": { "field": 3 } }, "rootReference": {} } } } ] } } ] } } }, "names": [ "t", "k", "v_avg" ] } } ], } FrfrŠ)rrrr)ç @r—g"@g&@)r–rŒrrr‘r rur Útest_hash_aggregate_udf_basicøs*ýÿ  ÿýr˜ÚexprrWr”rwé2cCsxt t dt ¡¡t dt ¡¡g¡}tj |gdg|¡}tj |¡}||jks*J‚t|jƒdks3J‚d|jvs:J‚dS)NrWrwÚ test_exprr) rrriÚint32r/Úserialize_expressionsÚdeserialize_expressionsÚlenÚ expressions)r™rr6Úreturnedr r r Útest_serializing_expressions£sþ r¢cs¢t d¡dft d¡dft d¡dft ¡dft ¡dft ¡dfdœ}t dd „| ¡Dƒ¡‰‡fd d „}| ¡D]\}\}}|t  |¡tj ||d kƒq:dS) NÚsrÚmsÚnsÚ test_strings test_string)Ú time_secondsÚ time_millisÚ time_nanosÚ date_millisÚ large_stringÚ large_binarycSs g|] \}\}}t ||¡‘qSr )rri)Ú.0r{Útypror r r Ú ¾s z-test_arrow_specific_types..cs2tj |gdgˆ¡}tj |¡}ˆ|jksJ‚dS©Nr›)rr/rržr)r™r6r¡rr r Úcheck_round_tripÀs z3test_arrow_specific_types..check_round_trip)r@) rÚtime32Útime64Údate64r«r¬rÚitemsÚpcriÚscalar)Úfieldsr±r{r®Úvalr rr Útest_arrow_specific_typesµs      ú ÿrºc sÀt t dt ¡¡t dt ¡¡t dt t ¡t ¡¡¡t dt t ¡t ¡¡¡g¡‰t t dt  ¡¡t dt ¡¡t dt ¡¡t dt ¡¡g¡‰‡‡fdd„}ˆD]}||ƒqWdS)NÚ binary_viewÚ string_viewÚ dictionaryÚreecsDt t |j¡¡}tj |gdgˆ¡}tj |¡}ˆ|jks J‚dSr°) r¶Úis_nullrir{rr/rržr)rir™r6r¡©Ú alt_schemarr r Ú check_one_wayÛs z/test_arrow_one_way_types..check_one_way) rrrir»r¼r½rœÚstringÚrun_end_encodedÚbinary)rÂrir rÀr Útest_arrow_one_way_typesÉs$üÿüÿ  ÿrÆcCs>t t dt ¡¡t dt ¡¡g¡}t t d¡d¡}t t d¡d¡}t t¡}tj   |gg|¡Wdƒn1s>wYdt |j ƒvsLJ‚t t¡}tj   |gddg|¡Wdƒn1sgwYdt |j ƒvsuJ‚t t¡}tj   |gdg|¡Wdƒn1swYd t |j ƒvsJ‚dS) NrWrwr”Úzzneed to have the same lengthr#rHr™zNo match for FieldRef) rrrirœr¶ÚequalrArBÚ ValueErrorr/rrr)rr™Úbad_exprr€r r r Útest_invalid_expression_ser_desås$þ ÿ ÿ ÿrËcCsút t dt ¡¡t dt ¡¡g¡}t t d¡d¡t t d¡t d¡¡g}tj |ddg|¡}tj |¡}||jks?J‚t |j ƒdksHJ‚t t d¡d¡t t d¡t d¡¡g}t |j dƒt |dƒkslJ‚t |j dƒt |dƒks{J‚dS) NrWrwr”ÚfirstÚsecondrrr) rrrirœr¶rÈr/rržrŸr r)rÚexprsr6r¡Ú norm_exprsr r r Ú%test_serializing_multiple_expressionsùsþ* *"rÐcCs2t t dt ¡¡t dt ¡¡g¡}t t d¡d¡}t t d¡d¡}| |¡}tj |¡}||jks7J‚t |j ƒdks@J‚t |j dƒt |ƒksMJ‚tj  ||gddg|¡}t  t¡}tj |¡Wdƒn1sowYd t |jƒvs}J‚tj  |gd g|¡}tj |¡}t |ƒt |ƒks—J‚dS) NrWrwr”rrÚ expressionrÌrÍzcontained multiple expressionsÚ weirdname)rrrirœr¶rÈÚ to_substraitr/ržrŸr rrrArBrÉÚ ExpressionÚfrom_substraitr)rr™Ú expr_normr6r¡r€Úexpr2r r r Útest_serializing_with_compute s&þ   ÿ rØcCsÚt t dt ¡¡g¡}t d¡}t d¡}t ||¡g}t t ¡tj   |dg|¡Wdƒn1s6wYtj j |dg|dd}tj   |¡}||jksSJ‚t |jƒdks\J‚t|jdƒt|dƒkskJ‚dS) NrWrr!r™T)Úallow_arrow_extensionsrr)rrriÚuint32r¶r·Ú shift_leftrArBrr/rržrŸr r)rÚaÚbrÎr6r¡r r r Útest_serializing_udfs$s ÿ   ÿ ÿ "rÞcCs®d}t t dt ¡¡t dt ¡¡g¡}tj |¡}||ks!J‚tj |¡}|j|ks.J‚tj |¡}||ks:J‚tj |j¡}||ksGJ‚tj |j ¡}|j|ksUJ‚dS)Ns x y * brWrw) rrrirœrÃr/Údeserialize_schemaÚserialize_schemaržrÑ)Úsubstrait_schemaÚexpected_schemar¡Úarrow_substrait_schemar r r Útest_serializing_schema:sþ      räcCsVGdd„dƒ}d}tjj ||ƒ¡}t|jƒdksJ‚d|jvs"J‚d|jvs)J‚dS)Nc@seZdZdd„Zdd„ZdS)z7test_bound_expression_from_Message..FakeMessagecSs ||_dSrƒ©r™)Úselfr™r r r Ú__init__Rs z@test_bound_expression_from_Message..FakeMessage.__init__cSs|jSrƒrå)rær r r ÚSerializeToStringUszItest_bound_expression_from_Message..FakeMessage.SerializeToStringN)Ú__name__Ú __module__Ú __qualname__rçrèr r r r Ú FakeMessageQs rìsj  project_release  project_version"0 project_version project_release : brÚproject_releaseÚproject_version)rr/ÚBoundExpressionsrÕrŸr )rìÚmessagerÎr r r Ú"test_bound_expression_from_MessagePs rñ)r)/rr*rAÚpyarrowrÚpyarrow.computeÚcomputer¶Ú pyarrow.librrrÚpyarrow.substraitr/Ú ImportErrorÚmarkÚ pytestmarkr rÚ parametrizer9Ú py_bufferrErGrIrQrVrkrsrtr}rr‰Únumpyr“r˜rÈrir¢rºrÆrËrÐrØrÞrärñr r r r Úsj   ÿ  -  1 0** {v *  + ý