当前位置：首页 > news >正文

解决在python中用polars库访问vertex格式文件遇到的离奇错误

news 2026/8/26 19:34:37

用如下脚本测试在python中用polars库访问vertex格式文件。

importvortexasvximportpyarrow.parquetaspq vx.io.write(pq.read_table("example.parquet"),'example.vortex')lf=vx.open('example.vortex').to_polars()lf=lf.select('tip_amount','fare_amount')lf=lf.head(3)lf.collect()""" shape: (3, 2) ┌────────────┬─────────────┐ │ tip_amount ┆ fare_amount │ │ --- ┆ --- │ │ f64 ┆ f64 │ ╞════════════╪═════════════╡ │ 0.0 ┆ 61.8 │ │ 5.1 ┆ 20.5 │ │ 16.54 ┆ 70.0 │ └────────────┴─────────────┘ """

先用pip install 命令安装 vortex 和 polars 库，

root@6ae32a5ffcde:/par# python3 pip.pyz install vortex-data --break-system-packages Looking in indexes: https://mirrors.aliyun.com/pypi/simple Collecting vortex-data Downloading https://mirrors.aliyun.com/pypi/packages/ab/cd/ffa45ff2f4a3816cac629d2f4d1512e1c2f68dcc088ba0e48c8ae59f5030/vortex_data-0.64.0-cp311-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (32.3 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 32.3/32.3 MB 20.6 MB/s eta 0:00:00 Requirement already satisfied: pyarrow>=17.0.0 in /usr/local/lib/python3.11/dist-packages (from vortex-data) (21.0.0) Collecting substrait>=0.23.0 (from vortex-data) Downloading https://mirrors.aliyun.com/pypi/packages/1a/b8/40c8ac43fe4099f056b84e4652f5e5a8e8c742b5d2850ea1c3dec7ee32b9/substrait-0.28.0-py3-none-any.whl (42 kB) Requirement already satisfied: typing-extensions>=4.5.0 in /usr/local/lib/python3.11/dist-packages (from vortex-data) (4.15.0) Requirement already satisfied: protobuf<7,>=5 in /usr/local/lib/python3.11/dist-packages (from substrait>=0.23.0->vortex-data) (6.33.1) Collecting substrait-protobuf==0.79.0 (from substrait>=0.23.0->vortex-data) Downloading https://mirrors.aliyun.com/pypi/packages/0a/27/24e16471dc7b5986cd2af0d5aab9b12347fb835840fb2320b1b2375e53a8/substrait_protobuf-0.79.0-py3-none-any.whl (83 kB) Collecting substrait-extensions==0.79.0 (from substrait>=0.23.0->vortex-data) Downloading https://mirrors.aliyun.com/pypi/packages/f3/2f/a05dd536db76ba9309bb80df559a203614d107ef2381560872fb12370396/substrait_extensions-0.79.0-py3-none-any.whl (101 kB) Installing collected packages: substrait-protobuf, substrait-extensions, substrait, vortex-data Successfully installed substrait-0.28.0 substrait-extensions-0.79.0 substrait-protobuf-0.79.0 vortex-data-0.64.0 root@6ae32a5ffcde:/par# python3 pip.pyz install polars --break-system-packages Looking in indexes: https://mirrors.aliyun.com/pypi/simple Collecting polars Downloading https://mirrors.aliyun.com/pypi/packages/c2/52/306da8d01a324a6ad693baa963dcd180f089882426653605bd418681183b/polars-1.39.2-py3-none-any.whl (823 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 824.0/824.0 kB 19.3 MB/s eta 0:00:00 Collecting polars-runtime-32==1.39.2 (from polars) Downloading https://mirrors.aliyun.com/pypi/packages/4b/ad/f95110341491949f7137f1c74b988a6fede81b0b145771f3a69d9001f15a/polars_runtime_32-1.39.2-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (46.9 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 46.9/46.9 MB 22.3 MB/s eta 0:00:00 Installing collected packages: polars-runtime-32, polars Successfully installed polars-1.39.2 polars-runtime-32-1.39.2

然后运行脚本出错，错误信息如下：

root@6ae32a5ffcde:/par# python3 polars-vortex.txt Traceback (most recent call last): File "/par/polars-vortex.txt", line 5, in <module> lf = vx.open('example.vortex').to_polars() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/dist-packages/vortex/file.py", line 228, in to_polars from polars.io.plugins import register_io_source ModuleNotFoundError: No module named 'polars.io'; 'polars' is not a package

明明polars安装成功了，怎么会报这个错误，vertex库应该也没有错，因为example.vortex文件成功生成了。
我把这个问题在github vertex存储库网站上提出，开发团队也很迷惑，怀疑是已有的依赖影响，建议我用venv在隔离环境再测。

python3 -m venv my-venv source ./my-venv/bin/activate pip install vortex-data pip install polars

结果，arm平台竟然在隔离环境自动好了。
安装的版本有一些区别, 比如新装的pyarrow是23，而原有的是21，但也都是在合格范围以内的。

root@kylin-pc:/par# python3 -m venv my-venv root@kylin-pc:/par# source ./my-venv/bin/activate (my-venv) root@kylin-pc:/par# pip install vortex-data Looking in indexes: https://mirrors.aliyun.com/pypi/simple/ Collecting vortex-data Downloading https://mirrors.aliyun.com/pypi/packages/79/13/9891ae14b8ed990a9b7457a4e22d0b8f1f43c89266ac67fc7ae1397c2d87/vortex_data-0.64.0-cp311-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (30.1 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 30.1/30.1 MB 21.7 MB/s 0:00:01 Collecting pyarrow>=17.0.0 (from vortex-data) Downloading https://mirrors.aliyun.com/pypi/packages/54/a5/8cbc83f04aba433ca7b331b38f39e000efd9f0c7ce47128670e737542996/pyarrow-23.0.1-cp314-cp314-manylinux_2_28_aarch64.whl (44.5 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 44.5/44.5 MB 47.4 MB/s 0:00:00 Collecting substrait>=0.23.0 (from vortex-data) Downloading https://mirrors.aliyun.com/pypi/packages/1a/b8/40c8ac43fe4099f056b84e4652f5e5a8e8c742b5d2850ea1c3dec7ee32b9/substrait-0.28.0-py3-none-any.whl (42 kB) Collecting typing-extensions>=4.5.0 (from vortex-data) Using cached https://mirrors.aliyun.com/pypi/packages/18/67/36e9267722cc04a6b9f15c7f3441c2363321a3ea07da7ae0c0707beb2a9c/typing_extensions-4.15.0-py3-none-any.whl (44 kB) Collecting protobuf<7,>=5 (from substrait>=0.23.0->vortex-data) Downloading https://mirrors.aliyun.com/pypi/packages/ee/90/b3c01fdec7d2f627b3a6884243ba328c1217ed2d978def5c12dc50d328a3/protobuf-6.33.6-cp39-abi3-manylinux2014_aarch64.whl (324 kB) Collecting substrait-protobuf==0.79.0 (from substrait>=0.23.0->vortex-data) Downloading https://mirrors.aliyun.com/pypi/packages/0a/27/24e16471dc7b5986cd2af0d5aab9b12347fb835840fb2320b1b2375e53a8/substrait_protobuf-0.79.0-py3-none-any.whl (83 kB) Collecting substrait-extensions==0.79.0 (from substrait>=0.23.0->vortex-data) Downloading https://mirrors.aliyun.com/pypi/packages/f3/2f/a05dd536db76ba9309bb80df559a203614d107ef2381560872fb12370396/substrait_extensions-0.79.0-py3-none-any.whl (101 kB) Installing collected packages: typing-extensions, substrait-extensions, pyarrow, protobuf, substrait-protobuf, substrait, vortex-data Successfully installed protobuf-6.33.6 pyarrow-23.0.1 substrait-0.28.0 substrait-extensions-0.79.0 substrait-protobuf-0.79.0 typing-extensions-4.15.0 vortex-data-0.64.0 [notice] A new release of pip is available: 25.3 -> 26.0.1 [notice] To update, run: pip install --upgrade pip (my-venv) root@kylin-pc:/par# pip install polars Looking in indexes: https://mirrors.aliyun.com/pypi/simple/ Collecting polars Downloading https://mirrors.aliyun.com/pypi/packages/c2/52/306da8d01a324a6ad693baa963dcd180f089882426653605bd418681183b/polars-1.39.2-py3-none-any.whl (823 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 824.0/824.0 kB 18.1 MB/s 0:00:00 Collecting polars-runtime-32==1.39.2 (from polars) Downloading https://mirrors.aliyun.com/pypi/packages/1d/42/e34654ebbba3f46c9a462b96097c425058fd1734e30c90d57f88785580d6/polars_runtime_32-1.39.2-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (43.2 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 43.2/43.2 MB 48.6 MB/s 0:00:00 Installing collected packages: polars-runtime-32, polars Successfully installed polars-1.39.2 polars-runtime-32-1.39.2 [notice] A new release of pip is available: 25.3 -> 26.0.1 [notice] To update, run: pip install --upgrade pip

我用duckdb生成了lineitem.parquet，然后对脚本做了相应修改

importvortexasvximportpyarrow.parquetaspq vx.io.write(pq.read_table("example.parquet"),'example.vortex')lf=vx.open('example.vortex').to_polars()lf=lf.select("l_linenumber","l_quantity")lf=lf.head(3)print(lf.collect())

执行结果如下：

(my-venv) root@kylin-pc:/par# python polars-vortex.py shape: (3, 2) ┌──────────────┬───────────────┐ │ l_linenumber ┆ l_quantity │ │ --- ┆ --- │ │ i64 ┆ decimal[15,2] │ ╞══════════════╪═══════════════╡ │ 1 ┆ 17.00 │ │ 2 ┆ 36.00 │ │ 3 ┆ 8.00 │ └──────────────┴───────────────┘

再回到x64平台在隔离环境照样操作，结果还是和第一次一样的错误，真的好神奇。
然后换了一个目录执行，居然顺利完成了。那问题只能出在那个目录了

(my-venv) root@6ae32a5ffcde:/par/1s# python3 polars-vortex2.txt shape: (3, 2) ┌──────────────┬───────────────┐ │ l_linenumber ┆ l_quantity │ │ --- ┆ --- │ │ i64 ┆ decimal[15,2] │ ╞══════════════╪═══════════════╡ │ 1 ┆ 17.00 │ │ 2 ┆ 36.00 │ │ 3 ┆ 8.00 │ └──────────────┴───────────────┘

查看/par目录跟polars相关的内容，结果发现一个polars.abi3.so文件，把它改名，问题就消失了。

(my-venv) root@6ae32a5ffcde:/par# ls polars* polars-df0ccd63bba07535.d polars-vortex.txt polars.abi3.so (my-venv) root@6ae32a5ffcde:/par# mv polars.abi3.so polars.abi3.so.file (my-venv) root@6ae32a5ffcde:/par# python3 polars-vortex.txt shape: (3, 2) ┌──────────────┬───────────────┐ │ l_linenumber ┆ l_quantity │ │ --- ┆ --- │ │ i64 ┆ decimal[15,2] │ ╞══════════════╪═══════════════╡ │ 1 ┆ 17.00 │ │ 2 ┆ 36.00 │ │ 3 ┆ 8.00 │ └──────────────┴───────────────┘

查看全文

http://www.cnnetsun.cn/news/1376407.html