Repeatedly using git-filter-branch to rewrite new commits

I would like to split out modules distributed with a bigger application into separate submodules and keep the ability to pull from upstream.

So this is more complex than Detach subdirectory into separate Git repository. I do not only have do use git-filter-branch once but want to keep the ability to pull upstream changes a开发者_C百科fter I have done so (and upstream has not).

Simply rerunning git-filter-branch on the complete history from upstream now including new commits not found in my rewritten history is not an option as there are hundreds of modules for which I have to do this and the number of commits is getting close to 100.000.

I am guessing this involves limiting the history to just the new commits, rewriting those and then adding them after the previously rewritten commits, but I am unsure how to do this - and maybe there is a better approach.

It would be nice if branches and tags could be preserved too but this is not absolutely necessary and if it complicates things I would actually prefer to lose those.

For the first rebase do this:

git checkout -b rebased master
git filter-branch --some-filter
git tag rebased-done master

And to "merge" later commits:

# Create a tempory branch and rebase it's tail use 'rebase-done~'
# and not 'rebase-done' because some filters (like --index-filter)
# require this, others might not.
git checkout -b rebased-tail master
git filter-branch -f --some-filter -- rebased-done~..HEAD

# Get the commit in branch 'rebased' corresponding to tag 'rebase-done'
# (which tags a commit in 'master' not 'rebased').  Depending on your
# situation you might have to determine this commit differently (in my
# use case I am absolutely sure that there is never a commit with the
# same author date - if that doesn't work you might want to compare
# commit messages).
start_time=$(git show --quiet --pretty=%at rebased-done)
start_hash=$(
git log --reverse --pretty="%H %at" rebased_tail |
while read hash time
do
    [ "$time" = "$start_time" ] && echo $hash && break
done
)

# Finally apply the rebased commits.
git checkout rebased
git format-patch -k --stdout $start_hash..rebased-tail | git am -k
git branch -D rebased-tail
git tag -f rebased-done master