Analysis of AI vs AI plays in FreeOrion v0.4.9 - Presentation of the results

Creation, discussion, and balancing of game content such as techs, buildings, ship parts.

Moderators: Oberlus, Committer

Post Reply
Message
Author
hawxinx
Space Krill
Posts: 2
Joined: Sat Aug 29, 2020 10:20 am

Analysis of AI vs AI plays in FreeOrion v0.4.9 - Presentation of the results

#1 Post by hawxinx »

Hi,

I've written a code available at https://github.com/hawXinx/FreeOrionDataCollector to analyse the behavior of the AI playing against itself.

Acknowledgment: Thanks to Geoff, Oberlus, and O01eg that answered my questions and enabled me to tackle this project.

A short review:

I've made a code to automatically play FreeOrion v0.4.9 with two AI-players with different starting species. The observing script was designed via a man-in-the-middle-attack and got its information about the game status via a change in the turn_events.py script. Based on the first games played this way, I've made a decision tree to speed up the calculation by pruning games where the odds were against one side. The results of all games were then analysed based on the AI's starting race.

The most significant outputs were:
1, Starting with Fulver and Sly, the AI wins about 1/3rd of all games against another AI with any other random species.
2, Starting with Trith, the AI wins about 2/3rd of all games against another AI with any other random species.
(In a perfectly balanced game with a perfect AI, those values should be about 1/2)

The code and the resulting data is available at the URL given above. While this analysis is based on v0.4.9 of FreeOrion (stable), I didn't recognize many changes at the AI when I read through the changelog, so I think that some results and comparisons may still hold in the current version as well. I hope I've helped you to further develop the game.

I don't know if I'm going to maintain the project much as it took over a month to finish all calculations (keeping my computer busy). So, as long as there is no significant speed-up in the AI calculation (I'd like to suggest Multithreading the Python code), I'm not going to do this again. I hope you'll understand.

I'm looking forward to seeing new developments in FreeOrion. It's a good game and I'd like to play the finished game.

Yours sincerely

hawxinx

User avatar
Oberlus
Cosmic Dragon
Posts: 5715
Joined: Mon Apr 10, 2017 4:25 pm

Re: Analysis of AI vs AI plays in FreeOrion v0.4.9 - Presentation of the results

#2 Post by Oberlus »

Great job!

If I get it right, in https://github.com/hawXinx/FreeOrionDat ... result.csv you saved the results, one line per match, 70 games for each pair of species, so average game time was around 6 minutes per game.
I guess a way to speed it up would be to launch several in parallel, maybe #CORES/2 (for 2 AI threads per match), at the expense of getting a less responding system.

What were the galaxy settings for those games? Systems, monsters, etc.
Would it be easy to make the script test different galaxy settings and dump the values along the results?

hawxinx
Space Krill
Posts: 2
Joined: Sat Aug 29, 2020 10:20 am

Re: Analysis of AI vs AI plays in FreeOrion v0.4.9 - Presentation of the results

#3 Post by hawxinx »

Hi Oberlus,

thanks for your compliment.

I'll go through your post point by point:

1, Yes, regarding the results.csv, you've got everything right. When it comes to the average time for each game, I don't want to make any statements (I needed to stop the script sometimes, needed to recompile FreeOrion because of dependency-updates, etc., so I don't want to state any specific time span).
2, Your guess to speed it up is absolutely valid but was not suitable for me.
3, The settings: Quick answer: If they were not mentioned in the Readme.md/section Work/subsection Assumptions/Point 2 (2.3 to be precise), they were as delivered by the game at the very first start.
Long answer: The rules were as delivered by the game at the very first start, the other points are (as described in FreeOrionDataCollector.py, Line 222 - 267):
3.1, Seed: created by the script with a random function
3.2, size: 45 systems (the advice of the game was 15 to 30 systems per player, 45 was the mean)
3.3, AI-Aggression: Maniac (to speed it up)
3.4, Galaxy Shape: 9 (random)
3.5, Galaxy Age: 2 (mature)
3.6, Starlane Frequency: 2 (medium)
3.7, Planet density: 2 (medium)
3.8, Special Frequency: 2 (medium)
3.9, Native Frequency: 2 (medium)
4, That change should be possible. The program wasn't designed that way so you have to add additional management for the new variables, but I guess it should be possible for someone that knows Python & FreeOrion to re-write the scripts and deliver a solution within two days.

Post Reply