Hugging Face Inference Client
This guide demonstrates how to interact with open-source LLMs hosted on the Hugging Face Hub using the huggingface_hub library. This allows you to use models like Mistral, Falcon, or Zephyr via API without downloading them locally.
Modules Used:
huggingface_hub: The official Python client for the Hugging Face Hub.- python-dotenv: To manage the API token.
- argparse: To handle command-line arguments.
Prerequisites
- Hugging Face Account: Sign up at huggingface.co.
- Access Token: Go to Settings -> Access Tokens and create a new token (Read permissions).
Installation
pip install huggingface_hub python-dotenvSetup
Create a .env file:
HF_TOKEN=hf_your_token_hereThe Code
Save this as hf_chat.py.
import os
import argparse
from dotenv import load_dotenv
from huggingface_hub import InferenceClient
# Load Config
load_dotenv()
TOKEN = os.getenv("HF_TOKEN")
def chat_with_model(model_id, prompt):
if not TOKEN:
print("Error: HF_TOKEN not found in .env file.")
return
print(f"Connecting to {model_id}...")
try:
client = InferenceClient(token=TOKEN)
# Generate stream
stream = client.text_generation(
model=model_id,
prompt=f"<s>[INST] {prompt} [/INST]", # Mistral/Llama formatting
max_new_tokens=500,
stream=True
)
print("\nResponse:")
for response in stream:
print(response, end="", flush=True)
print()
except Exception as e:
print(f"Error: {e}")
if __name__ == "__main__":
parser = argparse.ArgumentParser(description="Hugging Face Inference Chat")
parser.add_argument("prompt", help="The prompt to send")
parser.add_argument("--model", default="mistralai/Mistral-7B-Instruct-v0.2", help="Model ID on HF Hub")
args = parser.parse_args()
chat_with_model(args.model, args.prompt)Usage
python hf_chat.py "Explain the theory of relativity simply."