where is mycode error?

[yutaolife]

there are not any output. where is mycode error?
import os
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM, set_seed

tokenizer = AutoTokenizer.from_pretrained("/Users/yutao/Documents/MyCode/meditron_7b/model")
model = AutoModelForCausalLM.from_pretrained("/Users/yutao/Documents/MyCode/meditron_7b/model")

question = "What to do about high blood pressure?"
inputs = tokenizer(question, return_tensors="pt")
set_seed(42)

answers = []

with torch.inference_mode():
beam_output = model.generate(**inputs,
max_new_tokens=1024,
num_beams=1,
pad_token_id=2,
eos_token_id=2,
early_stopping=False,
do_sample=False,
)
answers.append(tokenizer.decode(beam_output[0], skip_special_tokens=True))

print("answers: "+answers)

[djibe]

Hi. Can you try LM Studio ? It works so easily.
Except if you need to chain the output

[cloner174]

Hi.
Try using this, instead !

def ask(model, tokenizer, question, max_length=512, max_new_tokens=100):
    inputs = tokenizer.encode_plus(question + tokenizer.eos_token, return_tensors='pt', max_length=max_length, truncation=True)
    outputs = model.generate(
        inputs['input_ids'],
        attention_mask=inputs['attention_mask'],
        max_new_tokens=max_new_tokens,
        pad_token_id=tokenizer.eos_token_id,
        no_repeat_ngram_size=3,
        do_sample=True,
        top_k=100,
        top_p=0.7,
        temperature=0.8
    )
    response = tokenizer.decode(outputs[0], skip_special_tokens=True)
    return response

Here is an example usage!

# After loading model and its tokenizer using HF hub:
question = "what are the latest treatments for type 2 diabetes?"
response = ask(model, tokenizer, question, max_length=512, max_new_tokens=250)
print(response)

[yutaolife]

Hi. Try using this, instead !

def ask(model, tokenizer, question, max_length=512, max_new_tokens=100):
    inputs = tokenizer.encode_plus(question + tokenizer.eos_token, return_tensors='pt', max_length=max_length, truncation=True)
    outputs = model.generate(
        inputs['input_ids'],
        attention_mask=inputs['attention_mask'],
        max_new_tokens=max_new_tokens,
        pad_token_id=tokenizer.eos_token_id,
        no_repeat_ngram_size=3,
        do_sample=True,
        top_k=100,
        top_p=0.7,
        temperature=0.8
    )
    response = tokenizer.decode(outputs[0], skip_special_tokens=True)
    return response

Here is an example usage!

# After loading model and its tokenizer using HF hub:
question = "what are the latest treatments for type 2 diabetes?"
response = ask(model, tokenizer, question, max_length=512, max_new_tokens=250)
print(response)

Dear cloner174:
Many thanks your support.
after i use your example code, the progame still not any response. my memory is 24G. is related the memory?

[cloner174]

Hi. Try using this, instead !

def ask(model, tokenizer, question, max_length=512, max_new_tokens=100):
    inputs = tokenizer.encode_plus(question + tokenizer.eos_token, return_tensors='pt', max_length=max_length, truncation=True)
    outputs = model.generate(
        inputs['input_ids'],
        attention_mask=inputs['attention_mask'],
        max_new_tokens=max_new_tokens,
        pad_token_id=tokenizer.eos_token_id,
        no_repeat_ngram_size=3,
        do_sample=True,
        top_k=100,
        top_p=0.7,
        temperature=0.8
    )
    response = tokenizer.decode(outputs[0], skip_special_tokens=True)
    return response

Here is an example usage!

# After loading model and its tokenizer using HF hub:
question = "what are the latest treatments for type 2 diabetes?"
response = ask(model, tokenizer, question, max_length=512, max_new_tokens=250)
print(response)

Dear cloner174: Many thanks your support. after i use your example code, the progame still not any response. my memory is 24G. is related the memory?

Can you provide the actual output that you are receiving ?