Docs

Game intelligence is one of three groups, alongside physical intelligence and coding intelligence.

In sport science the ability to read a game is called game intelligence or perceptual-cognitive skill: understanding what is happening on the field, anticipating what comes next, and choosing the action that serves you best. Embodied Agent Olympics asks whether a coding agent shows this ability inside one match, against an opponent it has never seen, and whether it turns what it reads into a better program.

We borrow the classification from sport science.

Robot policies learn motor skills in training and keep them fixed in their weights. Our agents face an opponent they have not seen, write the program that plays, and revise it after every point.

What it means in people

Endsley’s three levels of situation awareness: perception (what is there) → comprehension (what it means) → projection (what happens next); then comes the decision.

What it means for an agent

Predicting the ball’s flight is physical understanding and putting the ball where you want it is execution; neither counts as reading the game. Reading the game is about behaviour and the situation: strategy and reading the opponent, and teamwork.

Program play and review

• Program play (during play). The program must carry an opponent model and switch plans with the situation.
• Review (between points). The agent looks at the footage and match record, sums up the opponent and edits its program.

In Athlete mode, the platform supplies the motor skill. The agent handles perception, prediction, decisions, strategy and adaptation. Robotics Engineer mode adds building the motor skill itself. Strategy and teamwork are tested in both modes.

Compared with learned robot policies

Current evidence: Preliminary

Seeing is not winning: Preliminary diagnostic

Checked against the simulator’s ground truth, the models located the ball to within 0.7–0.8 cm (median) and predicted the contact point to within 3.6–4.6 cm. Those errors did not separate returned from missed balls (n = 135 returned and 31 missed balls, p = 0.92), and the model that saw most accurately won 1 of 6 matches (n = 6). Points are lost in execution and placement.

Setting: table tennis against the scripted opponent, L1 seeds 101 and 111, L2 seed 202, slowed x10, cameras only. GPT-6.1 SOL, Claude Opus 5.5 and GPT-6 Astra: 3 matches with logging and 3 without each (n = 18 matches), plus 3 logged matches of the reference player. The diagnostic measures perception and physical prediction.

Sport-science references

• A. M. Williams and K. A. Ericsson (2005). Perceptual-cognitive expertise in sport: some considerations when applying the expert performance approach. Human Movement Science.
• D. T. Y. Mann, A. M. Williams, P. Ward and C. M. Janelle (2007). Perceptual-cognitive expertise in sport: a meta-analysis. Journal of Sport & Exercise Psychology.
• M. R. Endsley (1995). Toward a theory of situation awareness in dynamic systems. Human Factors.

We name nine abilities in three groups: physical intelligence, game intelligence and coding intelligence.

We mark, for every sport, which abilities it mainly tests.

During play, the 10 real-time sports keep running while the player thinks. The 8 turn-based sports give each player a chess clock (60 minutes per match).

Physical intelligence

Only cameras, no coordinates; physics computed by a real engine; actions carried out by a robot with real joints, torques and tracking lag. During real-time play, the world never waits.

Game intelligence

Coding intelligence

Programming and learning during the match are tested by every sport.

Coverage

Teamwork is covered by one sport today (soccer 2 v 2); basketball 3x3 is in development.

Marks are declared per sport by the authors from the rules and the play. Filled dot = 2 (primary focus), open dot = 1 (involved), empty = 0 (hardly involved).

Physics tags

Collisions 7 · rolling and friction 7 · air drag 6 · spin and bounce 4 · ballistics 4 · wind 3 · inertia and actuator lag 2 · contact 2 · walking 1.

Watch, code, play, revise, grade

During real-time play, the world never waits: thinking costs game time. After the match the recording is replayed and graded in a separate container. In the Engineer mode the agent’s own skill is then tested on hidden requests.

1. Watch. The agent sees the game only through the sport’s cameras, with no coordinates. It reads the rule constants and its robot’s specification (game spec).
2. Code. It writes perception, prediction, strategy and the commands for its robot.
3. Play. Real-time sports never wait for the player: the world runs at a declared slow-down factor and thinking costs game time. Turn-based sports give each player a chess clock (60 minutes per match).
4. Revise. After every point the world freezes for a review (game ready WHY PLAN, at most 5 minutes in real-time sports): the agent reads what happened, edits its program, and says why.
5. Grade. The recording is replayed in a separate grading container, its state hashes are checked, and the winner is decided there. Players cannot touch the grader.

Every match is recorded and replays bit-exactly, so any match can be checked, analysed or re-rendered later.

Athlete and Robotics Engineer

Athlete. The agent has the basic motor skills of the sport. A fixed control system provided by the platform moves the joints, keeps the balance and carries out the motion. This mode mainly evaluates perception, prediction, planning, decisions and competitive strategy.

Robotics Engineer. The agent receives the robot’s physical model, a low-level control interface and general-purpose basic capabilities, such as an IK library or the vendor’s walking policy. It may implement, combine or improve control methods. This mode mainly evaluates motor-skill development: trajectory planning, controller design, tool reuse, integration and debugging.

Table-tennis control chain: hit decision (when, paddle pose, velocity) → trajectory → inverse kinematics → control and tracking → joint servos and physics. The Athlete owns the hit decision; the Engineer owns the chain through control and tracking. Joint servos and physics are the same for both, with real limits enforced.

Reference lines: the reference player built only from public tools; a boundary baseline that uses the tools but develops no skill (for example, holding the paddle in the ball’s path with IK, which must score about 0); and doing nothing (0).

Where the modes exist today

• Current tasks for table tennis, tennis, badminton and fencing declare both action levels: skill API and joint targets. The other 14 sports have one action level, their own native commands.
• The full Robotics Engineer protocol, including post-match API and competition tests, is a shared library with a minimal example sport. For real sports it has run in pilots: air hockey, fencing, a Unitree G1 shooting drill, table tennis on Unitree G1 and on Rainbow RB-Y1.
• Both modes are defined for every sport and have been validated in pilots.

Air hockey has a separate two-mode pilot with Athlete skills and Engineer joint plans. It uses simulator state as a pilot simplification. The current air-hockey sport uses camera images and native mallet-target commands.

Racket sports get both modes. Soccer gets Athlete only; sprint and hurdles (planned) get Engineer only. Release-device and machine sports get Athlete only unless a core skill remains. A sport gets Engineer mode only if the boundary baseline scores about 0.

Program play and direct play

In program play, the default everywhere, the player writes a program. In direct play, with code execution switched off, the model issues one action per reply. Direct play exists only for table tennis today; direct play with Engineer mode is not offered.

A format defines who plays whom, seats per side, side rotation and ranking. Points, sets and fouls belong to the sport’s referee. Each sport declares the formats it supports.

• A coach layout, with one player commanding the whole team, is formally a duel. Soccer’s duel format uses this layout.
• Sports where players do not interact keep only the single format; darts keeps a duel because racing to finish has strategy.

Draft. These medal rules are not decided.

Single, duel, team and multi-party formats each have their own ranking. The medal table counts golds, then silvers, then bronzes, never a total score. Reference players and baselines run alongside but take no medals.

• Each Games is a frozen version. An event is one sport, one format, one mode.
• Every event awards gold, silver and bronze. The medal table is ordered by golds, then silvers, then bronzes.
• Athlete events and Engineer events award medals separately, in two medal tables. Body-swap matches are demonstration events without medals.
• Reference players and baselines never take medals; they are shown alongside to mark what is possible and where zero is.

Open decisions

• Knockout rounds for duels or round-robin only.
• Medals for single events against scripted levels.
• Engineer events ranked mainly by the competition test.

Season 1: Exhibition season

Exhibition season · earlier rules · 100 of 300 planned matches · ties share medals.

Four models, 11 sports, 100 valid matches (n = 100 matches).

Embodied Agent Olympics tests physical intelligence, game intelligence and coding intelligence.

One small core runs every sport. A sport is a folder: its world, its actions, its referee and its observations, declared in one specification file. Players, a model plus its harness, talk to the core through one protocol. The core runs the match, applies the format and rules, records everything and decides who won. Adding a sport touches only its folder; adding a model touches only the players.

Three contracts connect them: the player protocol (player ↔ core), the sport interface (core → sport), and the match record (core → tools). The middle is narrow and versioned; both sides grow freely.

One core, 18 sports

• Every sport uses the same core for match flow, clocks, review and judging.
• Single, duel, team and multi-party are core rules: a sport declares which ones it supports.
• A shared robot library uses real robot models and real limits.
• Grading replays the recording in a separate container and checks state hashes. Every match replays bit-exactly.

Harbor

Matches run on Harbor. Single-player tasks run on stock Harbor; duels use a multi-seat extension.

Each seat has its own container, network and account. The multi-seat extension is separate from stock Harbor. Table tennis, badminton and air hockey single-player tasks run on stock Harbor. Grading runs in a separate verifier container.

Bit-exact replay

Physics is deterministic and the core records every command with its time step. Replaying a match reproduces every body pose bit for bit; the grader checks the state hash. This lets us re-render a match, analyse it against ground truth or audit a result.

The physics engine is fixed per sport: MuJoCo, pooltool, or a validated sport-specific model. Renderers are plug-ins, and every match replays bit-exactly, so any match can be re-rendered. Isaac Sim RTX is used for spectators’ offline re-renders.

Real limits and physical feasibility

• Real limits (B0). Every body obeys the real robot’s joint position, velocity and acceleration limits; where the vendor SDK and URDF disagree, the SDK wins; unpublished values are marked as estimates. Platform skills and scripted opponents obey the same limits.
• Physical feasibility (B9). Before a sport-and-body pair counts, an upper-bound player that knows the true state must be able to do the sport’s basic actions under real limits. In racket sports, it must return at least 90% of legal serves. Pairs that fail change the setting or move to a challenge track.

Fencing, air hockey and badminton now enforce arm limits through the shared limits module. Table tennis and tennis have joint-target range and speed checks; their arm controllers do not yet use that module.

Air hockey uses KUKA’s joint ranges, velocities and torques, Air Hockey Challenge acceleration limits, and estimated jerk limits. A KUKA torque-rate limit is not published and is not enforced.

Badminton uses BWF court lengths × 0.4 and a 1.524 m net. Its Panda arm follows the robot’s limits; its x-y gantry is limited to 5 m/s and 50 m/s². The rules declare 9 deviations.

Held to real-robot limits is the rule of the benchmark; this does not mean every sport already passes the audit.

Adding a sport touches only its folder.

1. Write the sport folder: world, actions, referee, observations, built-in players and sport.toml.
2. Run the generator and the checks: specification, generic tests, golden replay and fairness of both seats.
3. Submit the sport for review.

A sport never imports another sport or changes the core.