📄 Paper • 🏠 Repo • 🤖 Models • 📚 Datasets

ReflectionCoder is a novel approach that effectively leverages reflection sequences constructed by integrating compiler feedback to improve one-off code generation performance. Please refer to our paper for more details!

python data/download.py

data
  - train
    - code_instruct.jsonl
    - reflection_ds.jsonl
    - reflection_gpt.jsonl

You can use the following command to fine-tune your model with our method. Here, we assume you have 16 GPUs and set gradient_accumulation_steps as 32. You can adjust gradient_accumulation_steps based on your number of GPUs to ensure train_batch_size is equal to 512.

RANK=...
MASTER_ADDR=...
MASTER_PORT=...
WORLD_SIZE=...

model_cfg=meta-llama/CodeLlama-7b-Python-hf
out_dir=runs/relfection_coder_code_llama_7b

torchrun --node_rank ${RANK} --master_addr ${MASTER_ADDR} --master_port ${MASTER_PORT} --nnodes ${WORLD_SIZE} --nproc_per_node 8 train.py --deepspeed config/stage_1.json --learning_rate 5e-5 --lr_scheduler_type cosine --per_device_train_batch_size 1 --max_len 4096 --save_steps 100 --warmup_ratio 0.05 --logging_steps 10 --seed 3407 --num_train_epochs 2 --report_to tensorboard --remove_unused_columns false --bf16 --do_train --save_safetensors --save_only_model --gradient_checkpointing --train_file data/train/code_instruct.jsonl data/train/reflection_gpt.jsonl data/train/reflection_ds.jsonl data/train/reflection_gpt.jsonl data/train/reflection_ds.jsonl --logit --block_mask --block_order tce --model_cfg ${model_cfg} --output_dir ${out_dir} --gradient_accumulation_steps 32

out_dir=runs/relfection_coder_code_llama_7b/checkpoint-final

python test.py -tp 2 -p ${out_dir} -t humaneval mbpp multiple

Then, use EvalPlus to evaluate the inference results. Note that you should install the nightly version of EvalPlus with pip install "git+https://github.com/evalplus/evalplus.git" --upgrade.

out_dir=runs/relfection_coder_code_llama_7b/checkpoint-final

evalplus.evaluate --dataset humaneval --samples ${out_dir}/results/humaneval.jsonl
evalplus.evaluate --dataset mbpp --samples ${out_dir}/results/mbpp.jsonl

We also provide generated results for HuamnEval and MBPP in data.

For multiple, you can use bigcode-evaluation-harness to evaluate the inference results. For example, you can evaluate java with the following command:

out_dir=.../runs/relfection_coder_code_llama_7b/checkpoint-final

cd .../bigcode-evaluation-harness

python3 main.py \
  --model relfection_coder_code_llama_7b \
  --tasks multiple-java \
  --allow_code_execution \
  --load_generations_path ${out_dir}/results/multiple_java.json \
  --metric_output_path ${out_dir}/results/multiple_java_result.json 

When testing MultiPL-E, there are two things to pay attention to:

1. For JAVA, there are a wrong parameter in the testing code, you need replace result = run(["java", "-ea", "-cp", f"{outdir}", "Problem"], env=sys_env) to result = run(["java", "-ea", "-cp", f"{outdir}:{javatuples_path}", "Problem"], env=sys_env) in bigcode_eval/tasks/custom_metrics/multiple_metrics/eval_java.py. Note that, for fair comparsion, we only fix the bug when evaluating DeepSeek-Coder, and use the original code when evaluating Code LLama.
2. For C-Sharp, the testing code in nuprl/MultiPL-E have some bugs, please use https://github.com/deepseek-ai/DeepSeek-Coder/blob/main/Evaluation/HumanEval/data/humaneval-cs-bu.jsonl.

If you find this repo useful for your research, please kindly cite our paper:

@misc{ren2024reflectioncoder,
    title={ReflectionCoder: Learning from Reflection Sequence for Enhanced One-off Code Generation}, 
    author={Houxing Ren and Mingjie Zhan and Zhongyuan Wu and Aojun Zhou and Junting Pan and Hongsheng Li},
    year={2024},
    eprint={2405.17057},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}

We thank the following amazing projects that truly inspired us:

• CodeLlama
• DeepSeek-Coder
• WizardCoder
• Evol-CodeAlpaca-v1
• MagiCoder
• EvalPlus
• OpenCoderInterpreter