FreeOrion

I finally decided to give it a try and picked an item from the programming work page to implement it. I choose this one:It was a bit harder than I thought it would be, because I ran into two unforseen major issues (I'll get to that later).

I deviated a bit from the implementation suggested, this is what I did:

• Process priorities of the human client and server processes are not changed in any way. They keep running at default priority the entire time.
• AI processes are switched between standard (=same as human client/server) and lowered priority.
• AI processes are initially set to lowered priority. All AI processes are raised to standard priority once the human player who's client runs on the same machine as the server has finished his turn.
• After all clients have submitted their orders and turn processing starts, all AI processes are set to lowered priority.

I considered raising only AI processes that haven't submitted their orders, but decided against it. That would require some amount of extra management for no real effect, as an AI process that has already finished doing it's turn is idle anyway, so there won't be any difference in terms of cpu usage if it's priority is temporarily increased.

The implementetion is working as intended so far - kind of. Which brings me to the issues mentioned above, one of them probably can't be solved:

1.) On unix systems, at least on OSX and (as far as I understood from googling the problem) on Linux, the system call to set a process priority, "setpriority", only allows to lower process priority. Raising process priority requires superuser privileges (and I don't think we want to go so far as to run FO as root...), so, simply put, no way to do that. I've to clean up the code a bit concerning this issue (ATM the respective system calls are done nonetheless, but fail of course), but as it stands the solution on this systems will be that AI processes will run with lowered priority permanently. That at least is my suggestion. I repeatedly read that fiddling with process priority on unix systems shouldn't be done/isn't necessary anyway because the unix scheduler is supposedly very smart and knows best how to handle process priority...

2.) There is some weird thing going on that messes with the order network messages between client and server and/or the events triggered by that are processed during turn submission. As long as the AI clients finish before the human player, everything is fine, the issue arises when the human player submits his turn before the AI's are done. Apparently the auto-save feature (by default set to occur every turn) kicks in after the player presses "End turn", resulting in the player's "submit orders" message to hang until all AI clients are also finished, then save game proceeds, and only then the order submission messages are processed, all at once.

Which of course defeats the AI process priority management mechanism implemented. Once I switched off autosave, everything works as it should. I'm going to post a more detailed bug report on this forum, but I want to run some more tests before to collect log file entries illustrating what's going on. I'll also intend to try to look into the matter, but I don't know how far I'll get considering my unfamiliarity with the code. If someone feels like testing that, I've attached a modified FreeOrionAI.py which runs a busy loop for 30 sec before it does anything else when generating orders. This way a AI that is busy for some time is sufficiently simulated.

So, is the way I implemented this ok, or should I do something differently?

Finally, a question: Who shall/will remove the entry on the programming work page? Should I do that myself, or leave that to someone in charge of this list?

I wonder if there's a workaround using threads with variable priority within the AI clients?
Have you observed any client UI hangs / nonresponsiveness on Linux or OSX systems with lots of busy AIs processing their turns right away? If the problem only occurs on Windows, then it probably only needs to be fixed on windows...
I'm not clear on what the problem is here... What is the consequence of a submit orders message hanging until all clients are done? The turn can't proceed until all orders are ready... Is this a UI hang issue? I don't see that on Windows even with busy-delaying AIs and autosave on...
Go ahead.

Good question. The first problem I see is how thread priority and process priority relate. If thread priority is only used to determine how much cpu time a thread gets in relation to other threads of the same process, then this won't help.

I googled a bit, but the info I found provided no conclusive answer to that. There is a standard on Unix, posix threads, that should ensure sufficient portability, but that's not 100%, there seem to be implementation differences even between different Linux distros. Boost offers a library for threads that we could use, but I didn't find anything to manipulate thread priority.

This whole scheduler/process priority thing seems to be implementation dependend on Unix, at least to a certain extend. Apparently manipulating the "nice value" of processes might produce (slightly?) different results on OSX and Linux, or even on different Linux distros (maybe depending on the kernel version used?). I've to admit, reading through all that info was a bit confusing and not as enlightening as I hoped it would be.

Besides all that I wonder if the current implementation of the AI clients is very effective concerning performance versus usage of system resources. Launching 12 AI client processes on an dual core machine isn't very effective perfomance wise, but still uses ~23MB per client and have the human client to compete for cpu time with 12 concurrently running processes. Having all the AI clients run as threads in one process would be far more resource efficient and give more flexibility in managing their cpu time consumption (for example you could query the amount of cpu cores present on the system and only allow a respective number of the clients simultaneously, etc.). This of course would require a major rewrite of the AI client code and more complex AI client management. The question is if the expected gains justify the costs in programming efforts...

Let's see how the current solution (AI process priority lowered all the time on OSX and Linux) works. If the AI is sufficiently responsive despite the lowered priority, we could just leave it at that, at least for the time being.

The problem here is that the machines at my disposal which run Windows and my Macs aren't comparable. I can't say anything in regard to Linux, I haven't even tried to build FO on Linux.

On my Macs I have to start games with 12 AI's in order to notice reduced responsiveness of the UI when all of them are busy (which I achieved with my special test PythonAI). It's not very dramatic, but it's definitely noticeable. My Macs are Core i5/i7 machines. My Windows boxes are somewhat older dual core machines (the better one a 6 year old Core Duo). On this machines the effect of several busy AI processes is much more evident. Unfortunately I'm not able to run FO on the Win7 VM on my Mac due to some incompatibilities between Ogre and the OpenGL drivers provided by VirtualBox.

So, as far as I can tell from my own observations, that's more dependent on the hardware than the OS you're using.

This "hang" isn't obvious when playing the game. I only noticed something was amiss when I did some test runs to check if my priority switching is working as intended. As it turned out, it wasn't. I monitored the process priorities of the AI clients, and they didn't change after I hit "End turn". So I placed several debug log messages in the code to see what's going on and discovered that the condition for raising the AI's process priority was never met.

Originally I had placed the check for raising the AI's process priority in the WaitingForTurnEnd.CheckTurnEndConditions event handler in the server FSM. What has happened? After I had hit "Turn end", WaitingForTurnEnd.TurnOrders was triggered (as it should), which calls post_event(CheckTurnEndConditions()) before it returns. You'd expect that this event would be handled immediately afterwards, but that didn't happen. The event "hang" until all AI's had submitted their orders (and of course for each AI the CheckTurnEndConditions event was posted), only then the first CheckTurnEndConditions event was processed. At this time of course all AI's were finished doing their turns, and although their process priority was raised at last, it got lowered again immediately because server-side turn processing started. The remaining CheckTurnEndConditions events weren't handled properly and ended up triggering the "unconsumed_event" handler of the server FSM (resulting in the respective log messages, one for each AI process).

Checking the logs I noticed what looked like debug log entries from the save game process appearing before the order submission log entries of the AIs, so I started to suspect the autosave feature might be to blame. I turned of autosave, tried again and everything worked just fine.

Now this is where things start to get very odd, because after taking a look at the code this shouldn't be possible. Autosave is triggered during the TurnUpdate event, and this event can't possibly occur before all clients have submitted their orders and server-side turn processing has started. Maybe I misread the log files? What I'm going to do is trying to reproduce this behaviour and examine the logs more carefully, then post a more comprehensive bug report and attach the logs. EDIT: Done, see here.
Done

Generally when there are multiple threads competing for a single logger the ordering of log entries can be... awkward. I'm not sure if this is the issue here (I don't know if saving and event handling is in the same thread etc.).

_________________
[...] for Man has earned his right to hold this planet against all comers, by virtue of occasionally producing someone totally batshit insane. - xkcd

Seems like autosave is indeed interfering in this case (see here).

AIs and the server are separate processes with their own log files. Save game data requests are not synchronous messages, so there shouldn't be any race condition type issues of ambiguity about what's happening with ordering of log messages on the server.