Depends how good at prompting you are, I bet

Strictly LLM only, or can they perform intermediate steps like writing a chess solver in Python and then running the code and using the answer as part of its answer, like Chat-GPT can?

@AlanTennant No external tools allowed, so it can't run code.

@evan excellent

How would you describe your level of ability at chess?

@AlanTennant Beginner level.

Will you allow them to make illegal moves, if not then will you give them unlimited attempts to make legal moves, or disqualify them, or select a random move?

@AlanTennant Illegal moves will be disallowed and on the third attempt to make an illegal move the model will be to considered to have lost the game.

@evan You've got this one in the bag then, just play protectivly until the game state is atypical, for example almost any game by your 6th turn, then watch as it's rote memorisation of how chess games between experienced players usually start turns into confusion, and then "how does the non-existent queen I ridiculously gave away for free move again, it's on the board right" bewilderment.