The use of chatglm-6b requires the installation of transformers==4.27.1.
I'm trying to install transformers==4.27.1, but I'm encountering an error during the compilation of the tokenizers package, which prevents the successful building of the wheel file. The error message indicates that the cargo rustc command failed and returned code 101.
Here is the complete error message:
pip install -v transformers==4.27.1
....
error: `cargo rustc --lib --message-format=json-render-diagnostics --manifest-path Cargo.toml --release -v --features pyo3/extension-module --crate-type cdylib -- -C 'link-args=-undefined dynamic_lookup -Wl,-install_name,@rpath/tokenizers.cpython-312-darwin.so'` failed with code 101
error: subprocess-exited-with-error
× Building wheel for tokenizers (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> See above for output.
note: This error originates from a subprocess, and is likely not a problem with pip.
full command: /Users/dragonfang/****/venv_agi/bin/python3.12 /Users/dragonfang/****/venv_agi/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py build_wheel /var/folders/ll/9dtz3vg150vfv8t75ppq_nr00000gn/T/tmpymy6ke0d
cwd: /private/var/folders/ll/9dtz3vg150vfv8t75ppq_nr00000gn/T/pip-install-tz2dgt67/tokenizers_11ac58d2069c4ec1985eae0d4528f0ec
Building wheel for tokenizers (pyproject.toml) ... error
ERROR: Failed building wheel for tokenizers
Failed to build tokenizers
ERROR: ERROR: Failed to build installable wheels for some pyproject.toml based projects (tokenizers)
How to resolve this issue?
Supplementary：
I have tried some methods from the two links above, but none of them have solved my problem.
rustc 1.72.1 version is too low, package clap_lex v0.7.2 cannot be built because it requires rustc 1.74 or newer.transformers==4.34.0 no longer needs to install the Rust compiler, but the error occurs: AttributeError: 'ChatGLMTokenizer' object has no attribute 'sp_tokenizer'.transformers==4.33.2 requires the installation of the Rust compiler, and the error is: cargo rustc --lib --message-format=json-render-diagnostics --manifest-path Cargo.toml --release -v --features pyo3/extension-module --crate-type cdylib -- -C 'link-args=-undefined dynamic_lookup -Wl,-install_name,@rpath/tokenizers.cpython-312-darwin.so' failed with code 101.ChatGPT suggested
You can try using Python 3.10 or 3.11 to see if the issue is resolved
Since my Python version was 3.12.5, I downgraded to 3.11.9 and re-ran pip install. This successfully resolved the problem.
I've noticed that Gemini 1.5 Flash only suggests me to update Rust and Cargo, while GPT-4o mini additionally mentions the issue of Python version. I have been using Gemini before, it seems I should compare these two models more in the future.