I am trying to run some python code which includes some tiktoken calls.
I've tried to install tiktoken for python but i can't. I am trying to install with pip command but i am getting an error about rust. I am wondering if i should install some rust "thing" outside of python or pip.
Collecting tiktoken
Using cached tiktoken-0.3.3.tar.gz (25 kB)
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: requests>=2.26.0 in g:\program files\python\lib\site-packages (from tiktoken) (2.28.2)
Collecting regex>=2022.1.18
Using cached regex-2023.3.23-cp38-cp38-win32.whl (256 kB)
Requirement already satisfied: idna<4,>=2.5 in g:\program files\python\lib\site-packages (from requests>=2.26.0->tiktoken) (2.9)
Requirement already satisfied: certifi>=2017.4.17 in g:\program files\python\lib\site-packages (from requests>=2.26.0->tiktoken) (2020.4.5.1)
Requirement already satisfied: charset-normalizer<4,>=2 in g:\program files\python\lib\site-packages (from requests>=2.26.0->tiktoken) (3.1.0)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in g:\program files\python\lib\site-packages (from requests>=2.26.0->tiktoken) (1.26.15)
Building wheels for collected packages: tiktoken
Building wheel for tiktoken (pyproject.toml) ... error
error: subprocess-exited-with-error
× Building wheel for tiktoken (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> [37 lines of output]
running bdist_wheel
running build
running build_py
creating build
creating build\lib.win32-cpython-38
creating build\lib.win32-cpython-38\tiktoken
copying tiktoken\core.py -> build\lib.win32-cpython-38\tiktoken
copying tiktoken\load.py -> build\lib.win32-cpython-38\tiktoken
copying tiktoken\model.py -> build\lib.win32-cpython-38\tiktoken
copying tiktoken\registry.py -> build\lib.win32-cpython-38\tiktoken
copying tiktoken\__init__.py -> build\lib.win32-cpython-38\tiktoken
creating build\lib.win32-cpython-38\tiktoken_ext
copying tiktoken_ext\openai_public.py -> build\lib.win32-cpython-38\tiktoken_ext
running egg_info
writing tiktoken.egg-info\PKG-INFO
writing dependency_links to tiktoken.egg-info\dependency_links.txt
writing requirements to tiktoken.egg-info\requires.txt
writing top-level names to tiktoken.egg-info\top_level.txt
reading manifest file 'tiktoken.egg-info\SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching 'Makefile'
adding license file 'LICENSE'
writing manifest file 'tiktoken.egg-info\SOURCES.txt'
copying tiktoken\py.typed -> build\lib.win32-cpython-38\tiktoken
running build_ext
running build_rust
error: can't find Rust compiler
If you are using an outdated pip version, it is possible a prebuilt wheel is available for this package but pip is not able to install from it. Installing from the wheel would avoid the
need for a Rust compiler.
To update pip, run:
pip install --upgrade pip
and then retry package installation.
If you did intend to build this package from source, try installing a Rust compiler from your system package manager and ensure it is on the PATH during installation. Alternatively, rustup (available at https://rustup.rs) is the recommended way to download and update the Rust compiler toolchain.
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for tiktoken
Failed to build tiktoken
ERROR: Could not build wheels for tiktoken, which is required to install pyproject.toml-based projects
I've tried "pip install tiktoken".
Pip is trying to build the tiktoken library from source and you are missing the Rust compiler. You can either install the Rust compiler on your system, or install tiktoken from a wheel instead of building it from source.