GitHub migration procedure

Discussion about the project in general, organization, website, or any other details that aren't directly about the game.
Message
Author
User avatar
Vezzra
Release Manager, Design
Posts: 5002
Joined: Wed Nov 16, 2011 12:56 pm
Location: Sol III

Re: GitHub migration procedure

#136 Post by Vezzra » Fri Mar 27, 2015 2:31 pm

@Geoff: You are aware that all the commits you authored/pushed today to the (unfixed) freeorion repo will get undone if/once we switch to the fixed repo? Because these commits haven't been pushed when I cloned the copy which served as base for my repairs, so aren't in the fixed repo.

User avatar
Geoff the Medio
Programming, Design, Admin
Posts: 12453
Joined: Wed Oct 08, 2003 1:33 am
Location: Munich

Re: GitHub migration procedure

#137 Post by Geoff the Medio » Fri Mar 27, 2015 2:58 pm

Vezzra wrote:@Geoff: You are aware that all the commits you authored/pushed today to the (unfixed) freeorion repo will get undone if/once we switch to the fixed repo? Because these commits haven't been pushed when I cloned the copy which served as base for my repairs, so aren't in the fixed repo.
Sure. I wanted to fix some compile errors regardless, in case anyone was trying to build from the current repository.

User avatar
adrian_broher
Programmer
Posts: 1082
Joined: Fri Mar 01, 2013 9:52 am
Location: Germany

Re: GitHub migration procedure

#138 Post by adrian_broher » Fri Mar 27, 2015 9:32 pm

Sorry for late response.

I had to reconvert the repository again, because I was completely unaware of the troubles that creep up with line endings. I already have written a post about that viewtopic.php?p=75851#p75851 . Fortunately this problem is not really complex to solve, it just took some cpu hours to rewrite all text-ish file with LF line endings in the git history and a few minutes to find out how to check out the text files with the native line ending, keeping the special needs of visual studio and Xcode in mind.

Migration status (bold entries are new):
* The whole repository now roots in the FreeOrion directory.
* The /trunk/FreeOrion/WindowsKit.zip (Windows SDK for version v0.1 and v0.2) was deleted from history.
* The /trunk/DesignerTools was split into a separate repository. I guess we can delete it.
* All authors were renamed to their name given on github, sourceforge and the credits file, prioritized as listed.
* All authors emails were fetched from github and the SVN repository UUID, prioritized as listed.
* All branches were retained, except of /branches/python-integration, which were never merged back to /trunk and is really old.
* All branches were renamed consistently lowercase, replacing underscores with dashes.
* Open branch endings and beginnings were stitched to their corresponding split/merge commit, except /branches/release-0.4.4 as this branch was and should never be merged back to /trunk.
* All tags were retained and converted to annotated tags, except of /tags/RELEASE_V_0_1_INSTALLER, /tags/RELEASE_V_0_2_INSTALLER and /tags/start
* All version tags were renamed to match semantic version tagging vMAJOR.MINOR.PATCH
* All missing tags from http://www.freeorion.org/index.php/Compile#Tarballs were added with Geoff as tagger, copying the commit date and the commit message to commit tagged.
* Enforced LF line endings for whole commit history.
* Added .gitattributes to apply proper native line endings when checking code out, normalizing text files to LF when checking in. Adding special cases for VS and Xcode specific files. Excluded binary files from line ending normalization.
* Stitched release v0.4.4 back into master (can be undone without much work, but please read arguments below).
* Applied most recent commits from freeorion:freeorion repository master branch with proper author, time and committer (No time traveling for you).
* Repacked the whole repository from 853Mb to 469Mb.


TODO:
* Add logging-migration branch from freeorion:freeorion-fixed to the new repository.
* Upload new repository
* Replace new repository
Vezzra wrote:I don't know... I mean there are some commits on a release branch you would not want to merge back into master (like the commenting out of test features like the super testers, changing some default settings that we want to be different in releases than in test builds etc). If you merge release branches back into master, you'd have to be careful to exclude these things.
Bigjoe5 wrote:In my experience, it is a bad idea to merge the release branch back into master. It's best to just make the call for each new branch whether to merge it into only master (features), only the release branch (release-specific bug-fixes/hacks), or both (general bug fixes).
The points you both are making is valid, but the argument behind it works in both directions. Considering the r8051 commit from SVN is about cherry-picking the change log from the v0.4.4 branch there can also be the case where change in the release branch must be part of the main development branch. Forgetting bugfixes or documentation on the release branche can't happen when merging it to the main development branch.

Finally this boils down to: "Be aware of what you're doing".

So stitching release branches gains us
+ All release branch only fixes will be part of master.
+ A working git describe.
+ Less branch clutter.
but
- All release branch only changes will be part of master.

Maybe it's the better approach to reduce the 'release branch only' changes in general, like, for example, instead commenting out/in super testers use some other mechanism to disable/enable cheats.
wheals wrote:So when branching, you'd tag the release branch with 0.4.5b (for "beta") and the master branch with 0.4.6a
It's a way to approach this problem, but keep in mind that this adds clutter to the tags and tags are mapped mostly to releases in github. It is a working solution, but IHMO it's not a tidy solution.
Vezzra wrote:Here is what I've done to fix the commit history in detail
Looks okay. Instead of uploading the repository you could have force pushed the master and logging-migration branch to the current repository instead.
Vezzra wrote:The impression is right, however, to be sure to preserve their commit history I've pulled them in before I made my fixed repo. If I had only pulled in master and logging-migration and fixed those, I'm not sure if my fixed repo would have lost the commit history of all the other branches. I'm, after all, a git newbie too still.
The commit history is kept even if you don't have local branches (or tracking branches) mapping to them. You have a local copy of them as long as you have the remote. The remote repository even doesn't need to exist for that.
Vezzra wrote:Once we've got this whole migration thing completed for good, we can remove all the branches that are no longer active. I guess... Marcel?
All branches (except release-v0.4.4) are merged back into master. So yes, all those branches can be removed without loosing anything (git would warn you anyway, if you want to delete a branch where not all commit are reachable from some kind of other ref (a tag or another branch).
Last edited by adrian_broher on Fri Mar 27, 2015 10:36 pm, edited 1 time in total.
Resident code gremlin
Attached patches are released under GPL 2.0 or later.
Git author: Marcel Metz

User avatar
Geoff the Medio
Programming, Design, Admin
Posts: 12453
Joined: Wed Oct 08, 2003 1:33 am
Location: Munich

Re: GitHub migration procedure

#139 Post by Geoff the Medio » Fri Mar 27, 2015 10:02 pm

adrian_broher wrote:...the special needs of visual studio and Xcode in mind.
What does this refer to?

User avatar
adrian_broher
Programmer
Posts: 1082
Joined: Fri Mar 01, 2013 9:52 am
Location: Germany

Re: GitHub migration procedure

#140 Post by adrian_broher » Fri Mar 27, 2015 10:05 pm

Geoff the Medio wrote:What does this refer to?
VS Studio files should be always have CRLF line endings, Xcode files should always have LF endings, regardless of the platform the repository was checked out onto.
Resident code gremlin
Attached patches are released under GPL 2.0 or later.
Git author: Marcel Metz

User avatar
Geoff the Medio
Programming, Design, Admin
Posts: 12453
Joined: Wed Oct 08, 2003 1:33 am
Location: Munich

Re: GitHub migration procedure

#141 Post by Geoff the Medio » Fri Mar 27, 2015 10:10 pm

Seems to work fine for me with code as either CRLF or just LF.

Or what exactly do you mean by "VS Studio files"?

User avatar
adrian_broher
Programmer
Posts: 1082
Joined: Fri Mar 01, 2013 9:52 am
Location: Germany

Re: GitHub migration procedure

#142 Post by adrian_broher » Fri Mar 27, 2015 10:20 pm

Geoff the Medio wrote:Or what exactly do you mean by "VS Studio files"?
*.sln
*.vcxproj
*.vcxproj.filters

I'm just following the supposed best practices you can find on the net.

https://github.com/Danimoth/gitattribut ... attributes
https://code.google.com/p/angleproject/ ... attributes
http://sourceforge.net/p/poco/mailman/message/30032668/
Resident code gremlin
Attached patches are released under GPL 2.0 or later.
Git author: Marcel Metz

User avatar
Geoff the Medio
Programming, Design, Admin
Posts: 12453
Joined: Wed Oct 08, 2003 1:33 am
Location: Munich

Re: GitHub migration procedure

#143 Post by Geoff the Medio » Fri Mar 27, 2015 10:24 pm

adrian_broher wrote:*.sln
*.vcxproj
*.vcxproj.filters
OK, that's fine. I thought you meant .cpp / .h files.

jcantero
Krill Swarm
Posts: 12
Joined: Sun Nov 24, 2013 5:08 pm

Re: GitHub migration procedure

#144 Post by jcantero » Sat Mar 28, 2015 5:32 pm

adrian_broher wrote: The points you both are making is valid, but the argument behind it works in both directions. Considering the r8051 commit from SVN is about cherry-picking the change log from the v0.4.4 branch there can also be the case where change in the release branch must be part of the main development branch. Forgetting bugfixes or documentation on the release branche can't happen when merging it to the main development branch.

Finally this boils down to: "Be aware of what you're doing".

So stitching release branches gains us
+ All release branch only fixes will be part of master.
+ A working git describe.
+ Less branch clutter.
but
- All release branch only changes will be part of master.
Sorry to interrupt, but I'd like to add to the discussion that the reason most open source projects avoid to merge release branches (as well as merge branches in general) in the official repository[*] is to maintain a clean, linear history. A linear history helps a lot when someone is using git bisect to find the commit a bug/regression was introduced. See Merge or rebase?. Also it helps with the size of the official repo (that is from where most people is going to do the git clone).

[*] They can have another repositories for futher, experimental work, shared branches, etc (such as linux-next, wine-staging, and other fancy names). I mean, added to the own local (and remote) repositories of each developer, where most of the work is done (and where git rebase and git rebase -i is not a problem).

User avatar
Dilvish
AI Lead, Programmer
Posts: 4768
Joined: Sat Sep 22, 2012 6:25 pm

Re: GitHub migration procedure

#145 Post by Dilvish » Sat Mar 28, 2015 6:28 pm

jcantero wrote:See Merge or rebase?.
Thanks for that link; I found it a helpful read, on more than just the merge/rebase issue.
If I provided any code, scripts or other content here, it's released under GPL 2.0 and CC-BY-SA 3.0

User avatar
Vezzra
Release Manager, Design
Posts: 5002
Joined: Wed Nov 16, 2011 12:56 pm
Location: Sol III

Re: GitHub migration procedure

#146 Post by Vezzra » Sat Mar 28, 2015 6:52 pm

adrian_broher wrote:Sorry for late response.
No problem. Thanks for doing this :)
I had to reconvert the repository again, because I was completely unaware of the troubles that creep up with line endings. I already have written a post about that viewtopic.php?p=75851#p75851 . Fortunately this problem is not really complex to solve, it just took some cpu hours to rewrite all text-ish file with LF line endings in the git history and a few minutes to find out how to check out the text files with the native line ending, keeping the special needs of visual studio and Xcode in mind.
You're certainly doing this thoroughly. Which is very much appreciated, considering you probably don't really have the time for it.
* Stitched release v0.4.4 back into master (can be undone without much work, but please read arguments below).
I don't have a particularly strong opinion on that, so if the others actually prefer it that way, that's fine with me. As there most likely won't happen anything with that branch anymore, it's more or less a cosmetic thing (besides affecting git describe).
* Applied most recent commits from freeorion:freeorion repository master branch with proper author, time and committer (No time traveling for you).
Two things:

* Maybe check back with Geoff which of these might have been only test commits, not meant to stay? I don't think that he actually intended any of them to be only tests, just to make sure (OTOH, he probably would already have said anything if that were the case).

* Did you also apply to the commit that screwed CMakeLists.txt? Because I wonder if we should take that one out and let Geoff redo that one correctly.
* Add logging-migration branch from freeorion:freeorion-fixed to the new repository.
Anything you need from us to do that? Alternatively, we can leave it to Geoff to recreate that whole branch.
* Upload new repository
* Replace new repository
Well, all you did looks very good to me, so, as far as I am concerned, please proceed.
The points you both are making is valid, but the argument behind it works in both directions. Considering the r8051 commit from SVN is about cherry-picking the change log from the v0.4.4 branch there can also be the case where change in the release branch must be part of the main development branch. Forgetting bugfixes or documentation on the release branche can't happen when merging it to the main development branch.
That of course is true, however, I expect the case where you commit something to the release branch that should be merged into the main development branch more likely to be the exception than the rule (the changlog being an example for such an exception). Because the usual procedure with things you want to apply to both would be to commit them to the main dev branch and then merge to the release branch.

The things you might want to merge from the release branch to the main dev branch can be cherry picked.

However, that said, I guess this is probably a matter of personal preference.
Finally this boils down to: "Be aware of what you're doing".
Well, yeah. :D

This is true regardless which way round you want to do this.
Vezzra wrote:Here is what I've done to fix the commit history in detail
Looks okay. Instead of uploading the repository you could have force pushed the master and logging-migration branch to the current repository instead.
Ah, good to know. Wasn't sure about that and didn't want to risk to mess up the repo.

Well, with the import being redone moot point anyway. Still, I definitely learned a lot figuring out how to do this fix. I'm getting the hang of things, and begin getting really comfortable with git. It's a really cool tool. :D
The commit history is kept even if you don't have local branches (or tracking branches) mapping to them. You have a local copy of them as long as you have the remote. The remote repository even doesn't need to exist for that.
Also good to know. Will try to keep that in mind.

User avatar
Vezzra
Release Manager, Design
Posts: 5002
Joined: Wed Nov 16, 2011 12:56 pm
Location: Sol III

Re: GitHub migration procedure

#147 Post by Vezzra » Sat Mar 28, 2015 6:59 pm

jcantero wrote:Sorry to interrupt, but I'd like to add to the discussion that the reason most open source projects avoid to merge release branches (as well as merge branches in general) in the official repository[*] is to maintain a clean, linear history.
Regarding your statement about merging branches in general, assuming you mean that collaborative work on branches takes place in additional repos dedicated to that purpose (your footnote seems to indicate this), I guess that makes sense for larger open source projects with an according number of contributers.

But for our rather small project with only a handful contributers who actually have commit access this strikes me as overkill.

User avatar
adrian_broher
Programmer
Posts: 1082
Joined: Fri Mar 01, 2013 9:52 am
Location: Germany

Re: GitHub migration procedure

#148 Post by adrian_broher » Sat Mar 28, 2015 7:04 pm

jcantero wrote:Sorry to interrupt …
No, thanks for the reasonable interjection. I will give it a read.

Anyway, how should we progress now? The only thing that prevents me uploading the new repo is the question about merging release-v0.4.4 or not.
Resident code gremlin
Attached patches are released under GPL 2.0 or later.
Git author: Marcel Metz

User avatar
adrian_broher
Programmer
Posts: 1082
Joined: Fri Mar 01, 2013 9:52 am
Location: Germany

Re: GitHub migration procedure

#149 Post by adrian_broher » Sat Mar 28, 2015 7:15 pm

Vezzra wrote:* Maybe check back with Geoff which of these might have been only test commits, not meant to stay? I don't think that he actually intended any of them to be only tests, just to make sure (OTOH, he probably would already have said anything if that were the case).
I didn't exactly ask him, but the commits itself looked from a quick glance useful to me. But I didn't follow the development for some time as you know.
Vezzra wrote:* Did you also apply to the commit that screwed CMakeLists.txt? Because I wonder if we should take that one out and let Geoff redo that one correctly.
If you're talking about "Temporary git-compatible version string generation." Yes, I added that. The major problem with it was the missmatch of line endings (and that one invalid file added). I can remove the commit (actually the offending part in the cmake file) easily if you incline so.
Resident code gremlin
Attached patches are released under GPL 2.0 or later.
Git author: Marcel Metz

User avatar
Geoff the Medio
Programming, Design, Admin
Posts: 12453
Joined: Wed Oct 08, 2003 1:33 am
Location: Munich

Re: GitHub migration procedure

#150 Post by Geoff the Medio » Sat Mar 28, 2015 7:16 pm

For release branches, isn't the point of them to separate stuff away from the master branch that shouldn't be in master as it is release-specific? Anything that should be merged from release back to master should just be done in master and merged the other way. That said, the use of a branch for the v0.4.4 release was a new procedure, so if that's not how release-specific changes would normally be dealt with on git, then there's no need to do so in future. (A release branch isn't a temporary space to work on a feature... it's a permanent divergence from the master development line.) For the v0.4.4 branch, presumably any important commits can be cherry-picked back to master, as effectively was done for the changelog. Then presumably it should just be left as is.

Anything else I've committed to freeorion/freeorion/master is intended to be permanent, before and after re-re-recreation of the repository. To that end, can one cherry-pick from a different repository? (I assume merging wouldn't make sense...) I'd like to easily get the last 5-6 commits to freeorion/freeorion/master (and the 3 commits in the logging-migration branch, but that sounds like it's been handled) to the new repository version without manually downloading and applying patches and recommitting. One of those fixes CMakeLists.txt so just picking that fixes any issues with it.

Post Reply