Testing GPT-4 on Game State Predictions

Table of Links

Abstract and 1. Introduction and Related Work

1. Methodology

   2.1 LLM-Sim Task

   2.2 Data

   2.3 Evaluation

2. Experiments

3. Results

4. Conclusion

5. Limitations and Ethical Concerns, Acknowledgements, and References

A. Model details

B. Game transition examples

C. Game rules generation

D. Prompts

E. GPT-3.5 results

F. Histograms

A Model details

For the GPT-3.5 model, we use the gpt-3.5-turbo-0125 model. For the GPT-4 model, we use the gpt-4-0125-preview model. For both models, the temperature is set to 0 to get deterministic results. We also turn on the JSON mode of both models, which ensures that the model gives a valid JSON response. Our experiments cost approximately $5,000 for OpenAI API usage.

B Game transition examples

We manually pick the wash-clothes game in BYTESIZED32 as the example game as it contains both state transitions driven by actions and game’s underlying dynamics. In tasks where the model predicts action transition, environment-driven transitions, or the game progress alone, we provide one corresponding in-context example. In the task that requires the model to predict everything, we offer two in-context examples in the prompt. The two examples are manually picked such that in one example the game state is changed directly by the action taken while in the other example the game state is changed by the game’s underlying dynamics.

C Game rules generation

C.1 LLM generated rules

For LLM generated rules, we manually check all of them to avoid misinformation and offensive content.

\ We prompt GPT-4 (gpt-4-0125-preview) with the code of each object class to acquire the rules of each object. We also provide one in-context example. We ask GPT-4 to describe the meaning of each critical property (i.e. properties that do not inherit from parent) of the object and the tick function of the object (i.e. a function that defines how object properties may change at each time step regardless of the action taken). Below is an example of our prompt of object rule generation:

\ \

\ \ For action rules generation, we prompt GPT-4 (gpt-4-0125-preview) with the code of the whole game, but unlike object rules, we do not offer any in-context example. We ask GPT-4 to describe each of the actions in the game. Below is an example of our prompt for action rule generation:

\ \

\ \ Similar to action rules, we generate score rules by prompting GPT-4 (gpt-4-0125-preview) with the code of the game and ask GPT-4 to describe how the game can be won or lose and how rewards can be earned. Below is an example of our prompt for score rule generation:

\ \

\

C.2 Human-Written Action Rules

The action rules describe how each action can change the game states. The expert annotator reads the game description and source code for each game. They went through the list of available actions in the game and their corresponding functions in the game. Each action rule has three main parts: Action, Description, and Rules. The Action specifies the name of the action (e.g., action). The Description explains the general purpose of the action (e.g., connect two objects with input terminals). The Rules is an unordered list of rule descriptions that describe the constraints of the action when interacting with different objects (e.g., At least one of the objects should be a wire or a multimeter) or how the rule might function under different conditions (e.g., Disconnect terminal if the terminal is already connected to other objects). To ensure accuracy, the annotator plays through the game and checks if the written object rules were correctly reflected in the gameplay.

C.3 Human-Written Object Rules

The object rules describe the meaning of each object property (e.g., temperature, size, weight, etc.) and how they will be changed at each time step. The expert annotators read the game description and source code for each game. They went through the object classes in the code script and wrote the object rules. Each object rule has three main parts: Object, Description, and Properties. The Object specifies the name of the object. The Description explains the general purpose of the object (e.g., GarbageCan is a container that can hold garbage). In the Description, the inheritance of the object class has been noted. The Properties is an unordered list of property descriptions that describe each property of that object (e.g., A Mold has its shape.) and their default value (e.g., By default, a GameObject is not combustible.) if the object is an abstract class. For objects with tick function, there is another property describing how an object may change under each tick. To ensure accuracy, the annotator plays through the game and checks if the written object rules were correctly reflected in the gameplay.

C.4 Human-Written Score Rules

Score rules describe the conditions to win or lose the game and how rewards can be earned. An expert annotator (one of the BYTESIZED32 game authors) creates the rules by reading the game description and the code of the score function.

\

:::info Authors:

(1) Ruoyao Wang, University of Arizona ([email protected]);

(2) Graham Todd, New York University ([email protected]);

(3) Ziang Xiao, Johns Hopkins University ([email protected]);

(4) Xingdi Yuan, Microsoft Research Montréal ([email protected]);

(5) Marc-Alexandre Côté, Microsoft Research Montréal ([email protected]);

(6) Peter Clark, Allen Institute for AI ([email protected]).;

(7) Peter Jansen, University of Arizona and Allen Institute for AI ([email protected]).

:::

:::info This paper is available on arxiv under CC BY 4.0 license.

:::

\