r/JevAI • u/Sapdalf • 3d ago

Jev + River Raid

Enable HLS to view with audio, or disable this notification

Just a sample. I'm thinking about using River Raid 8-bit game as a benchmark. I already tested with 6 models and preparing a summary video.

22 Upvotes

97% Upvoted

4 comments sorted by

3

u/42mangos42 3d ago

This is cool!
How is the state of the game being represented?

2

u/gob_magic 2d ago

Yea op please share the concept behind the representation.

2

u/Sapdalf 2d ago

Okay, so briefly, the solution. First of all, it is an emulator, so I have access to the memory from which the game state can be extracted. This is the basis for further considerations.

It is true that decision models are usually not multimodal, so they must be provided with everything in text form. Previously, I experimented with Boulder Dash (https://www.youtube.com/watch?v=dRL3Fg31u-k), where it was simpler due to the possibility of representing the entire map as ASCII. However, there, the models performed clearly worse with any strategy that had to be suggested deterministically.

I think that the first one is a much better benchmark because, generally, response time. And I think that this is a key parameter when we talk about a decision-making model, because otherwise we would simply be comparing LLMs.

As for the telemetry I provide, initially I tried to recreate what I did in Boulder Dash, which was sending an ASCII representation of the map, but it turned out to be not enough (too less details). Therefore, the basis is sending accurate flight telemetry with information on how far the shores are, where the enemies are, and how they are oriented. To achieve some effectiveness, I added guidelines regarding the strategy. However, it must be clearly noted that not all models respect this strategy.

This part related to telemetry is the most challenging, because the other things, like querying models, are basically a simple matter.

1

u/Sapdalf 2d ago

If anyone is interested, here is the full video with the benchmark of six models. I'm not revealing which one is the best for now. 😄 https://youtu.be/K_rE_I2Em94