关于glm4-9B-chat大模型
【xinference】（11）：在compshare上使用4090D运行xinf和chatgpt-web，部署大模型glm4-9B-chat大模型，显存18G
GLM-4-9B 是智谱 AI 推出的最新一代预训练模型 GLM-4 系列中的开源版本。 在语义、数学、推理、代码和知识等多方面的数据集测评中，GLM-4-9B 及其人类偏好对齐的版本 GLM-4-9B-Chat 均表现出较高的性能。 除了能进行多轮对话，GLM-4-9B-Chat 还具备网页浏览、代码执行、自定义工具调用（Function Call）和长文本推理（支持最大 128K 上下文）等高级功能。 本代模型增加了多语言支持，支持包括日语，韩语，德语在内的 26 种语言。我们还推出了支持 1M 上下文长度（约 200 万中文字符）的模型。
https://modelscope.cn/models/ZhipuAI/glm-4-9b-chat/summary
模型文件分成了 10 个文件
xinf 也支持了：
https://inference.readthedocs.io/zh-cn/latest/models/builtin/llm/glm4-chat.html
创建带驱动的镜像
设置密码
2，需要选择23.08，cuda12.3版本，否则报错
× Getting requirements to build wheel did not run successfully. │ exit code: 1 ╰─> [33 lines of output] Traceback (most recent call last): File "/home/ubuntu/miniconda3/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module> main() File "/home/ubuntu/miniconda3/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main json_out['return_val'] = hook(**hook_input['kwargs']) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/ubuntu/miniconda3/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 112, in get_requires_for_build_wheel backend = _build_backend() ^^^^^^^^^^^^^^^^ File "/home/ubuntu/miniconda3/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 77, in _build_backend obj = import_module(mod_path) ^^^^^^^^^^^^^^^^^^^^^^^ File "/home/ubuntu/miniconda3/lib/python3.12/importlib/__init__.py", line 90, in import_module return _bootstrap._gcd_import(name[level:], package, level) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "<frozen importlib._bootstrap>", line 1387, in _gcd_import File "<frozen importlib._bootstrap>", line 1360, in _find_and_load File "<frozen importlib._bootstrap>", line 1310, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed File "<frozen importlib._bootstrap>", line 1387, in _gcd_import File "<frozen importlib._bootstrap>", line 1360, in _find_and_load File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 935, in _load_unlocked File "<frozen importlib._bootstrap_external>", line 995, in exec_module File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed File "/tmp/pip-build-env-84tcwvda/overlay/lib/python3.12/site-packages/setuptools/__init__.py", line 16, in <module> import setuptools.version File "/tmp/pip-build-env-84tcwvda/overlay/lib/python3.12/site-packages/setuptools/version.py", line 1, in <module> import pkg_resources File "/tmp/pip-build-env-84tcwvda/overlay/lib/python3.12/site-packages/pkg_resources/__init__.py", line 2191, in <module> register_finder(pkgutil.ImpImporter, find_on_path) ^^^^^^^^^^^^^^^^^^^ AttributeError: module 'pkgutil' has no attribute 'ImpImporter'. Did you mean: 'zipimporter'? [end of output] note: This error originates from a subprocess, and is likely not a problem with pip.问题’pkgutil’ has no attribute ‘ImpImporter’,切换版本解决
使用 cuda12.3 版本的python是 3.10 版本。没有这个问题
是python3.12 版本造成的。
最终运行效果
占用显存情况18G：
