Add support for large file uploads in multi-part mode (#106)

- Implement LFS support for large files
- Add command to enable LFS in repos
- Create multipart upload functionality
- Update CLI documentation for new commands
- Revise README for command usage examples

Author: HaiHui886

README.md

@@ -85,236 +85,11 @@ pip install '.[train]'
 
 ## Use cases of command line
 
-```shell
-export CSGHUB_TOKEN=your_access_token
-
-# download model
-csghub-cli download OpenCSG/csg-wukong-1B
-
-# download model with allow patterns '*.json' and ignore '*_config.json' pattern of files
-csghub-cli download OpenCSG/csg-wukong-1B --allow-patterns "*.json" --ignore-patterns "tokenizer.json"
-
-# download model with ignore patterns '*.json' and '*.bin' pattern of files to /Users/hhwang/temp/wukong
-csghub-cli download OpenCSG/csg-wukong-1B --allow-patterns "*.json" --ignore-patterns "tokenizer.json" --local-dir /Users/hhwang/temp/wukong
-
-# download dataset
-csghub-cli download OpenCSG/GitLab-DataSets-V1 -t dataset
-
-# download space
-csghub-cli download OpenCSG/csg-wukong-1B -t space
-
-# upload local large folder '/Users/hhwang/temp/abc' to model repo 'wanghh2000/model05'
-csghub-cli upload-large-folder wanghh2000/model05 /Users/hhwang/temp/abc
-
-# list inference instances for user 'wanghh2000'
-csghub-cli inference list -u wanghh2000
-
-# start inference instance for model repo 'wanghh2000/Qwen2.5-0.5B-Instruct' with ID '1358'
-csghub-cli inference start wanghh2000/Qwen2.5-0.5B-Instruct 1358
-
-# stop inference instance for model repo 'wanghh2000/Qwen2.5-0.5B-Instruct' with ID '1358'
-csghub-cli inference stop wanghh2000/Qwen2.5-0.5B-Instruct 1358
-
-# list fine-tuning instances for user 'wanghh2000'
-csghub-cli finetune list -u wanghh2000
-
-# start fine-tuning instance for model repo 'OpenCSG/csg-wukong-1B' with ID '326'
-csghub-cli finetune start OpenCSG/csg-wukong-1B 326
-
-# stop fine-tuning instance for model repo 'OpenCSG/csg-wukong-1B' with ID '326'
-csghub-cli finetune stop OpenCSG/csg-wukong-1B 326
-
-# upload a single file to folder1
-csghub-cli upload wanghh2000/myprivate1 abc/3.txt folder1
-
-# upload local folder '/Users/hhwang/temp/jsonl' to root path of repo 'wanghh2000/m01' with default branch
-csghub-cli upload wanghh2000/m01 /Users/hhwang/temp/jsonl
-
-# upload local folder '/Users/hhwang/temp/jsonl' to root path of repo 'wanghh2000/m04' with token 'xxxxxx' and v2 branch
-csghub-cli upload wanghh2000/m04 /Users/hhwang/temp/jsonl -k xxxxxx --revision v2
-
-# upload local folder '/Users/hhwang/temp/jsonl' to path 'test/files' of repo 'wanghh2000/m01' with branch v1
-csghub-cli upload wanghh2000/m01 /Users/hhwang/temp/jsonl test/files --revision v1
-
-# upload local folder '/Users/hhwang/temp/jsonl' to path 'test/files' of repo 'wanghh2000/m01' with token 'xxxxxx'
-csghub-cli upload wanghh2000/m01 /Users/hhwang/temp/jsonl test/files -k xxxxxx
-```
-
-Notes: 
-- `csghub-cli upload` will create repo and its branch if they do not exist. The default branch is `main`. If you want to upload to a specific branch, you can use the `--revision` option. If the branch does not exist, it will be created. If the branch already exists, the files will be uploaded to that branch. 
-- `csghub-cli upload` has a limitation of the file size to 4GB. If you need to upload larger files, you can use the `csghub-cli upload-large-folder` command.
-
-When using the `upload-large-folder` command to upload a folder, the upload progress will be recorded in the `.cache` folder within the upload directory to support resumable uploads. Do not delete the `.cache` folder before the upload is complete.
-
-Download location is `~/.cache/csg/` by default.
+For detailed command line usage examples, including downloading models/datasets, uploading files/folders, and managing inference/fine-tuning instances, please refer to our [CLI documentation](doc/cli.md).
 
 ## Use cases of SDK
 
-For more detailed instructions, including API documentation and usage examples, please refer to the Use case.
-
-### Download model
-
-```python
-from pycsghub.snapshot_download import snapshot_download
-token = "your_access_token"
-
-endpoint = "https://hub.opencsg.com"
-repo_id = 'OpenCSG/csg-wukong-1B'
-cache_dir = '/Users/hhwang/temp/'
-result = snapshot_download(repo_id, cache_dir=cache_dir, endpoint=endpoint, token=token)
-```
-
-### Download model with allow patterns '*.json' and ignore '*_config.json' pattern of files
-
-```python
-from pycsghub.snapshot_download import snapshot_download
-token = "your_access_token"
-
-endpoint = "https://hub.opencsg.com"
-repo_id = 'OpenCSG/csg-wukong-1B'
-cache_dir = '/Users/hhwang/temp/'
-allow_patterns = ["*.json"]
-ignore_patterns = ["*_config.json"]
-result = snapshot_download(repo_id, cache_dir=cache_dir, endpoint=endpoint, token=token, allow_patterns=allow_patterns, ignore_patterns=ignore_patterns)
-```
-
-### Download dataset 
-```python
-from pycsghub.snapshot_download import snapshot_download
-token="xxxx"
-endpoint = "https://hub.opencsg.com"
-repo_id = 'AIWizards/tmmluplus'
-repo_type="dataset"
-cache_dir = '/Users/xiangzhen/Downloads/'
-result = snapshot_download(repo_id, repo_type=repo_type, cache_dir=cache_dir, endpoint=endpoint, token=token)
-```
-
-### Download single file
-
-Use `http_get` function to download single file
-
-```python
-from pycsghub.file_download import http_get
-token = "your_access_token"
-
-url = "https://hub.opencsg.com/api/v1/models/OpenCSG/csg-wukong-1B/resolve/tokenizer.model"
-local_dir = '/home/test/'
-file_name = 'test.txt'
-headers = None
-cookies = None
-http_get(url=url, token=token, local_dir=local_dir, file_name=file_name, headers=headers, cookies=cookies)
-```
-
-use `file_download` function to download single file from a repository
-
-```python
-from pycsghub.file_download import file_download
-token = "your_access_token"
-
-endpoint = "https://hub.opencsg.com"
-repo_id = 'OpenCSG/csg-wukong-1B'
-cache_dir = '/home/test/'
-result = file_download(repo_id, file_name='README.md', cache_dir=cache_dir, endpoint=endpoint, token=token)
-```
-
-### Upload file
-
-```python
-from pycsghub.file_upload import http_upload_file
-
-token = "your_access_token"
-
-endpoint = "https://hub.opencsg.com"
-repo_type = "model"
-repo_id = 'wanghh2000/myprivate1'
-result = http_upload_file(repo_id, endpoint=endpoint, token=token, repo_type='model', file_path='test1.txt')
-```
-
-### Upload multi-files
-
-```python
-from pycsghub.file_upload import http_upload_file
-
-token = "your_access_token"
-
-endpoint = "https://hub.opencsg.com"
-repo_type = "model"
-repo_id = 'wanghh2000/myprivate1'
-
-repo_files = ["1.txt", "2.txt"]
-for item in repo_files:
-    http_upload_file(repo_id=repo_id, repo_type=repo_type, file_path=item, endpoint=endpoint, token=token)
-```
-
-### Upload the local path to repo
-
-Before starting, please make sure you have Git-LFS installed (see [here](https://git-lfs.github.com/) for installation instructions).
-
-```python
-from pycsghub.repository import Repository
-
-token = "your access token"
-
-r = Repository(
-    repo_id="wanghh2003/ds15",
-    upload_path="/Users/hhwang/temp/bbb/jsonl",
-    user_name="wanghh2003",
-    token=token,
-    repo_type="dataset",
-)
-
-r.upload()
-```
-
-### Upload the local path to the specified path in the repo
-
-Before starting, please make sure you have Git-LFS installed (see [here](https://git-lfs.github.com/) for installation instructions).
-
-```python
-from pycsghub.repository import Repository
-
-token = "your access token"
-
-r = Repository(
-    repo_id="wanghh2000/model01",
-    upload_path="/Users/hhwang/temp/jsonl",
-    path_in_repo="test/abc",
-    user_name="wanghh2000",
-    token=token,
-    repo_type="model",
-    branch_name="v1",
-)
-
-r.upload()
-```
-
-### Model loading compatible with huggingface
-
-The transformers library supports directly inputting the repo_id from Hugging Face to download and load related models, as shown below:
-
-```python
-from transformers import AutoModelForCausalLM
-model = AutoModelForCausalLM.from_pretrained('model/repoid')
-```
-
-In this code, the Hugging Face Transformers library first downloads the model to a local cache folder, then reads the configuration, and loads the model by dynamically selecting the relevant class for instantiation.
-
-To ensure compatibility with Hugging Face, version 0.2 of the CSGHub SDK now includes the most commonly features: downloading and loading models. Models can be downloaded and loaded as follows:
-
-```python
-# import os 
-# os.environ['CSGHUB_TOKEN'] = 'your_access_token'
-from pycsghub.repo_reader import AutoModelForCausalLM
-model = AutoModelForCausalLM.from_pretrained('model/repoid')
-```
-
-This code: 
-
-1. Use the `snapshot_download` from the CSGHub SDK library to download the related files.
-
-2. By generating batch classes dynamically and using class name reflection mechanism, a large number of classes with the same names as those automatically loaded by transformers are created in batches.
-
-3. Assign it with the from_pretrained method, so the model read out will be an hf-transformers model.
+For detailed SDK usage examples, including model/dataset downloading, file uploading, directory uploading, and Hugging Face compatible model loading, please refer to our [SDK documentation](doc/sdk.md).
 
 ## Roadmap