Writing Test Cases with Github Copilot

by

Introduction

Complex tasks, such as writing unit tests, can benefit from multi-step prompts. In contrast to a single prompt, a multi-step prompt generates text from GPT and then feeds that output text back into subsequent prompts. This can help in cases where you want GPT to reason things out before answering, or brainstorm a plan before executing it.

Multi-Step Prompting Technique

We will use a 3-step prompt to write unit tests in Java

  1. Explain: Given a Java function, we ask GPT to explain what the function is doing and why.
  2. Plan: We ask GPT to plan a set of unit tests for the function.
    • If the plan is too short, we ask GPT to elaborate with more ideas for unit tests.
  3. Execute: Finally, we instruct GPT to write unit tests that cover the planned cases.

Prompts

For Step 1, we will use this prompt

You are a world-class Java developer with an eagle eye for unintended bugs and edge cases.
You carefully explain code with great detail and accuracy.
You organize your explanations in markdown-formatted, bulleted lists. 

<Code pasted here>

For Step 2, we will use this prompt, 

A good unit test suite should aim to: 
Test the function's behavior for a wide range of possible inputs. 
Test edge cases that the author may not have foreseen. 
Be easy to read and understand, with clean code and descriptive names. 
Be deterministic, so that the tests always pass or fail in the same way. 
To help unit test the function above, list diverse scenarios that the function should be able to handle (and under each scenario, include a few examples as sub-bullets).

For Step 3, we will use this prompt, 

Now write test cases in junit 5 for all of the above points

Try this method for generating test cases. This 3 step approach performed a lot better than the normal way of prompting, in which we just say, write unit test cases for this function.

Give it a try, and let me know in the comments section if this worked for you or not.

This technique is inspired from Open AI cookbook samples: openai-cookbook/examples/Unit_test_writing_using_a_multi-step_prompt.ipynb at main · openai/openai-cookbook (github.com)

Leave a comment