Skip to content

README.md

Latest commit

 

History

History
158 lines (143 loc) · 4.63 KB

README.md

File metadata and controls

158 lines (143 loc) · 4.63 KB
language
en
license cc-by-nc-nd-4.0
tags
code
datasets
ajibawa-2023/Python-Code-23k-ShareGPT
model-index
name results
Python-Code-33B
task dataset metrics source
type name
text-generation
Text Generation
name type config split args
AI2 Reasoning Challenge (25-Shot)
ai2_arc
ARC-Challenge
test
num_few_shot
25
type value name
acc_norm
56.31
normalized accuracy
task dataset metrics source
type name
text-generation
Text Generation
name type split args
HellaSwag (10-Shot)
hellaswag
validation
num_few_shot
10
type value name
acc_norm
81.01
normalized accuracy
task dataset metrics source
type name
text-generation
Text Generation
name type config split args
MMLU (5-Shot)
cais/mmlu
all
test
num_few_shot
5
type value name
acc
54.22
accuracy
task dataset metrics source
type name
text-generation
Text Generation
name type config split args
TruthfulQA (0-shot)
truthful_qa
multiple_choice
validation
num_few_shot
0
type value
mc2
44.39
task dataset metrics source
type name
text-generation
Text Generation
name type config split args
Winogrande (5-shot)
winogrande
winogrande_xl
validation
num_few_shot
5
type value name
acc
75.22
accuracy
task dataset metrics source
type name
text-generation
Text Generation
name type config split args
GSM8k (5-shot)
gsm8k
main
test
num_few_shot
5
type value name
acc
19.18
accuracy

Python-Code-33B

Large Language Models (LLMs) are good with code generations. Sometimes LLMs do make mistakes in code generation. How about if they can give detailed explanation along with the code. This is what I have tried over here. The base Llama-2 model was used for training purpose. It is trained on around 23000+ set of codes. Each set having 2 conversations. This data was generated using GPT-3.5, GPT-4 etc. This conversation is in Vicuna/ShareGPT format. Each set, along with code, has detailed explanation. I have released the data.

Training: Entire dataset was trained on Azure 4 x A100 80GB. For 3 epoch, training took 42 hours. DeepSpeed codebase was used for training purpose. This was trained on Llama-1 by Meta.

This is a full fine tuned model. Links for quantized models are given below.

GPTQ GGML & AWQ

GPTQ: Link

GGUF: Link

AWQ: Link

Example Prompt:

This is a conversation with your helpful AI assistant. AI assistant can generate Python Code along with necessary explanation.

Context
You are a helpful AI assistant.

USER: <prompt>
ASSISTANT:

Open LLM Leaderboard Evaluation Results

Detailed results can be found here

Metric Value
Avg. 55.06
AI2 Reasoning Challenge (25-Shot) 56.31
HellaSwag (10-Shot) 81.01
MMLU (5-Shot) 54.22
TruthfulQA (0-shot) 44.39
Winogrande (5-shot) 75.22
GSM8k (5-shot) 19.18