Welcome to my blog where I write about software
development, cycling, and other random nonsense. This is not
the only place I write, you can find more words I typed on the Buoyant Data blog, Scribd tech blog, and GitHub.
Tim is extremely busy, most of his days working at an exciting, fast-paced start-up seem to fly by. With one particular project Tim is working on, a lot of code is changing at a very fast pace and the branch he's currently working in is stable one minute and destabilized the next. Tim has two basic options for leaving himself "bread-crumbs" to step back in time to a stable or an unstable state. The first, complicated option, is to mark his commit messages with something like "STABLE", etc so he can git diff or git reset --hard from the current HEAD to the last stable point of the branch.
The second option is to make use of tags. Whenever Tim reaches a stable point in his turmultuous development, he can simply run:
git tag wip-protips_`date "+%s"
(or something similar, `date` added to ensure the tag is unique). If Tim finds himself too far down the wrong path, he can rollback his branch to the latest tag (git reset --hard protiptag), create a new stable branch based on that tag (git checkout -b wip-protip-2 protiptag), or diff his current HEAD to the tag to see what all he's changed since his branch was stable (git diff protiptag...HEAD)
git push origin tag protiptag
git fetch your-repo --tags, I can pull the tags you've set in "your-repo" and apply them locally aid development. The distributed nature is primarily how tags differ in Git from Subversion, nearly the rest of the concept is the exact same.
post-receive hook in the central repository, where every push into the central repository ("origin") in the branch release branch is tagged. This allows us to quickly "revert" bad live pushes temporarily, by simply pushing the last "good" tagged release, to ensure minimal site destabilization (while we correct live issues outside of the release branch).
The concept of "revert" in Git versus Subversion is an interesting, albeit subtle, change, mostly in part due to the subtle differences between Git and Subversion's means of tracking commits. Just as with Subversion, you can only revert a committed change, unlike Subversion there is a 1-to-1 mapping of a "commit" to a "revert". The basic syntax of revert is quite easy: git revert 0xdeadbeef, and just like a regular commit, you will need to push your changes after you revert if you want others to receive the revert as well.
In the following example of a revert of a commit, I also use the "-s" argument on the command line to denote that I'm signing-off on this revert (i.e. I've properly reviewed it).
xdev3% git revert -s c20054ea390046bd3a54693f2927192b2a7097c2
----------------[Vim]----------------
Revert "merge-to-release unhide 10000 coin habitat"
This reverts commit c20054ea390046bd3a54693f2927192b2a7097c2.
Signed-off-by: R. Tyler Ballance <tyler@slide.com>
# Please enter the commit message for your changes. Lines starting
# with '#' will be ignored, and an empty message aborts the commit.
# On branch wip-protips
# Changes to be committed:
# (use "git reset HEAD <file>..." to unstage)
#
# modified: bt/apps/pet/data.py
+ python bt/qa/git/post-commit.py -m svn@slide.com
Sending a commit mail to svn@slide.com
Created commit a6e93b8: Revert "merge-to-release unhide 10000 coin habitat"
1 files changed, 4 insertions(+), 3 deletions(-)
git revert will generate a new commit for you every time you revert a previous commit, reverting multiple commits is not as obvious (side note: I'm aware of the ability to squash commits, or --no-commit for git-revert(1), I dislike compressing revision history when I don't believe there shouldn't be compression). If you want to revert a specific merge from one branch into the other, you can revert the merge commit (provided one was generated when the changes were merged). Take the following example:
commit 81a94bb976dfaaaae42ae2600b7e9e88645ebd81
Merge: 8134d17... d227dd8...
Author: R. Tyler Ballance <tyler@slide.com>
Date: Thu Nov 20 10:15:16 2008 -0800
Merge branch 'master' into wip-protips
git revert 81a94bb976dfaaaae42ae2600b7e9e88645ebd81 -m 1
commit 8cae4924c4c05dadaaeccb3851cfc9ec1b8efd0f
Author: R. Tyler Ballance <tyler@slide.com>
Date: Thu Nov 20 10:20:44 2008 -0800
Revert "Merge branch 'master' into wip-protips"
This reverts commit 81a94bb976dfaaaae42ae2600b7e9e88645ebd81.
for r in `git rev-list master...master-fubar --since="8:00" --before="12:00" --no-merges`; do git revert --no-edit -s $r; done
xdev% for r in `git rev-list master...master-fubar --since="8:00" --before="12:00" --no-merges`; do git revert --no-edit -s $r; done
Finished one revert.
Created commit b6810d7: Revert "a test, for you"
1 files changed, 1 insertions(+), 1 deletions(-)
Finished one revert.
Created commit 83156bd: Revert "These are not the droids you are looking for
1 files changed, 2 insertions(+), 0 deletions(-)
Finished one revert.
Created commit 782f328: Revert "commented out stuff"
1 files changed, 0 insertions(+), 3 deletions(-)
Finished one revert.
Created commit 2b8d664: Revert "back on again"
1 files changed, 1 insertions(+), 1 deletions(-)
xdev%
One of the major benefits to using Git is the entirety of the repository being entirely local and easily searched/queried. For this, Git has a very useful command called git log which allows you to inspect revision histories in numerous different ways between file paths, branches, etc. There are a couple basic scenarios where git log has become invaluable, for me at least, in order to properly review code but also to track changes effectively from point A to point B.
git log -p --no-merges --author=dave The --no-merges option will prevent git log from displaying merge commits which are automatically generated whenever you pull from one Git branch to another
git log --name-status master-topfriends...proj-topfriends-thing
Git supports the ability with git log and with git diff to provide unidirectional branch lookups or bidirectional branch lookups. For example, say the left branch has commits "A, B" and the right branch has commits "A, C". The ... syntax will output "C", whereas .. will output "B, C"
git log --since="2 weeks ago" --name-status -- templatesAt the tail end of a git log command you can specify particular paths to look up the histories for with the -- operator, in this case, I will be looking at the changes that have occured in the templates directory over the past two weeks
git log -n 10 --no-merges -p
git log commands automatically filter into less(1) so you can page through the output like you would normally if you executed a svn log | less. Because git log is simply reading from the locally stored revision history you can quickly grep the history by any number of different search criteria to gain a better understanding of how the code base is changing and where.
That said, I found that my post "Delightfully Wrong About Git" had found it's way into such silly news aggregation machines as DZone, Reddit and Hacker News. Some of the points raised in the comments were valid and warrant a response, while the majority of them were the standard responses to any discussion about version control "psh, dumb. should have used [Bazaar/Mercurial/Darcs/Subversion/Team Foundation System]"
With Subversion creating branches in the volume in which we create branches spiralled out of control with branches becoming "stale" quickly, meaning that if we didn't refresh the branch regularly with updates from trunk it would be nearly impossible to cleanly merge back down into trunk. With my current Git clone of our primary repository, I have 23 branches (roughly 6 personal local branches, 5 old branches, and 12 active branches). Our primary Git repository has been online for about 6 months and currently has 68 branches in it, rougly 55 are active. Why all the love for Git, but nobody every talks about Bazaar, Mercurial, Darcs, etc? Sure Git is faster, but unless you've got a enormous code base (like the linux kernel), it seems like Bazaar or Mercurial would be a better choice than Git.
Given the nature of how Git is structured, in that your "working copy" is also your "repository" you might find yourself switching branches relatively often. While you can actually switch branches with uncommited changes outstanding in your Git directory, it's not advised (as you might forget you're commiting changes in the wrong branch, etc). You have two options if you are halfway through some work, one is to commit a checkpoint revision, but the other is to make use of the git stash command.
git stash to snapshot his currently working state and context switch. In this case Bill would execute the following commands: git stash
git checkout master-media
git checkout wip-funtime
git stash pop
git stash pop command, Bill's Git repository will be in the exact same state, all his uncommitted changes in tact, as it was when he originally stashed and context-switched.
tyler@starfruit:~/source/git/main/bt> git stash
Saved working directory and index state "WIP on master-topfriends: 7b1ce9e... TOS copy fix"
(To restore them type "git stash apply")
HEAD is now at 7b1ce9e TOS copy fix
tyler@starfruit:~/source/git/main/bt>
Looking at the stash
tyler@starfruit:~/source/git/main/bt> git stash list
stash@{0}: WIP on master-topfriends: 7b1ce9e... TOS copy fix
stash@{1}: On master-topfriends: starfruit complete patchset
stash@{2}: On wip-classmethod: starfruit patches
tyler@starfruit:~/source/git/main/bt>
Grabbing the latest from the stash
tyler@starfruit:~/source/git/main/bt> git stash pop
Dropped refs/stash@{0} (94b9722b5a999c32c4361d795ee8f368d8412f9a)
tyler@starfruit:~/source/git/main/bt>
Grabbing a specific stash
tyler@starfruit:~/source/git/main/bt> git stash list
stash@{0}: WIP on master-topfriends: 7b1ce9e... TOS copy fix
stash@{1}: On master-topfriends: starfruit complete patchset
stash@{2}: On wip-classmethod: starfruit patches
tyler@starfruit:~/source/git/main/bt> git stash apply 2
# On branch master-topfriends
# Changed but not updated:
# (use "git add ..." to update what will be committed)
#
# modified: db/dbroot.py
# modified: gogreen/coro.py
# modified: py/bin/_makepyrelease.py
# modified: py/initpkg.py
# modified: py/misc/_dist.py
# modified: py/misc/testing/test_initpkg.py
# modified: py/path/local/local.py
# modified: py/test/terminal/terminal.py
tyler@starfruit:~/source/git/main/bt>
How will Git integrate with Hudson, Trac and our other pieces of development infrastructure?
re #12345
qa #12345
attn bbum,fspeirs
'''
Copyright (c) 2008 Slide, Inc
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.
'''
'''
For questions, patches, etc contact R. Tyler Ballance
'''
import getpass
import os
import re
import socket
import smtplib
import sys
import time
import xmlrpclib
from optparse import OptionParser
MAIL_SERVER = 'your_mail_server.com'
MAIL_SUFFIX = '@mycompany.com'
BUILD_HUDSON = True
HUDSON_URL = 'http://hudson'
TRAC_XMLRPC_URL = 'URL_TO_TRAC/projects/MYPROJECT/login/xmlrpc'
def rpcProxy(user='qatracbot', password=None):
password = password or os.getenv('TRAC_PASS')
return xmlrpclib.ServerProxy('https://%s:%s@%s' % (user, password, TRAC_XMLRPC_URL))
def _send_commit_mail(user, address, subject, branch, commits, files, diff):
print 'Sending a GITRECEIVE mail to %s' % address
message = 'Commits pushed to %s:\n--------------------------------------\n\n%s\n--------------------------------------\n%s\n--------------------------------------\n%s' % (branch, commits, files, diff)
_send_mail(user, address, subject, message)
def _send_attn_mail(user, destuser, diff):
print 'Sending a "please review" mail to %s' % destuser
message = '''Good day my most generous colleague! I would hold you in the highest esteem and toast you over my finest wines if you would kindly review this for me\n\n\t - %(user)s\n\nDiff:\n------------------------------------------------\n%(diff)s''' % {'diff' : diff, 'user' : user}
addresses = []
for d in destuser.split(','):
addresses.append('%s%s' % (d, EMAIL_SUFFIX))
_send_mail(user, addresses, 'Please review this change', message)
def _send_mail(user, address, subject, contents):
try:
if not isinstance(address, list):
address = [address]
s = smtplib.SMTP(MAIL_SERVER)
message = 'From: %s%s\r\nTo: %s\r\nSubject: %s\r\n\r\n%s\n' % (user, MAIL_SUFFIX, ', '.join(address), subject, contents)
s.sendmail('%s%s' % (user, MAIL_SUFFIX), address, message)
s.quit()
except:
print 'Failed to send the email :('
def _update_ticket(ticket, message, options={}):
rpc = rpcProxy()
rpc.ticket.update(ticket, message, options)
return rpc
def find_re(commit):
return map(int, re.findall(r'(?i)\s+re\s*#([0-9]+)', commit))
def handle_re(branch, commit, ticket):
print 'Annotating ticket #%s' % ticket
message = '''The following was committed in "%(branch)s":
\{\{\{
%(commit)s \}\}\}
''' % {'branch' : branch, 'commit' : commit}
_update_ticket(ticket, message)
def find_qa(commit):
return map(int, re.findall(r'(?i)\s+qa\s*#([0-9]+)', commit))
def handle_qa(branch, commit, ticket):
print 'Marking ticket #%s as "ready for QA"' % ticket
message = '''The following was committed in "%(branch)s":
\{\{\{
%(commit)s \}\}\}
''' % {'branch' : branch, 'commit' : commit}
rpc = _update_ticket(ticket, message, options={'status' : 'qa'})
def find_attn(commit):
return re.findall(r'(?i)\s+attn\s*([A-Za-z,]+)', commit)
def handle_attn(branch, commit, attn):
# Unpack commit from this: "commit 5f4c31f3c31347c62d68ecb5f2c9afa3333f4ad0\nAuthor: R. Tyler Ballance \nDate: Wed Nov 12 16:57:32 2008 -0800 \n\n Merge commit 'git-svn' \n\n \n \n"
try:
commit_hash = commit.split('\n')[0].split(' ')[1]
except:
return # fuk it
diff = os.popen('git show --no-color %s --pretty=format:"Author: %%cn <%%ce>%%n%%s%%n%%n%%b%%n%%n%%H"' % commit_hash).read().rstrip()
_send_attn_mail(getpass.getuser(), attn, diff)
def mail_push(address, oldrev, newrev, refname):
user = getpass.getuser()
machine = socket.gethostname()
base_git_diff = 'git diff %s %s' % (oldrev, newrev)
files_diffed = os.popen('%s --name-status' % (base_git_diff)).read().rstrip()
full_diff = os.popen('%s -p --no-color' % (base_git_diff)).read().rstrip()
''' git rev-parse --not --branches | grep -v "$new" | git rev-list "$old".."$new" --stdin '''
commits = os.popen('git rev-parse --not --branches | grep -v "%s" | git rev-list %s..%s --stdin --pretty=format:"Author: %%cn <%%ce>%%nDate: %%cd %%n%%n %%s %%n%%n %%b %%n %%n-------[post-receive marker]------%%n" --first-parent ' % (newrev, oldrev, newrev)).read().rstrip()
branch = refname.split('/')[-1]
mail_subject = 'GITRECEIVE [%s/%s] %s files changed' % (machine, branch, len(files_diffed.split('\n')))
if branch == 'master-release':
print 'Tagging release branch'
tagname = 'livepush_%s' % (time.strftime('%Y%m%d%H%M%S', time.localtime()))
sys.stderr.write('Creating a tag named: %s\n\n' % tagname)
os.system('git tag %s' % tagname)
mail_subject = '%s (tagged: %s)' % (mail_subject, tagname)
if BUILD_HUDSON_JOB:
print 'Queuing the Hudson job for "%s"' % branch
os.system('/usr/bin/env wget -q -O /dev/null http://%s/job/%s/build' % (HUDSON_URL, branch))
_send_commit_mail(user, address, mail_subject, branch, commits, files_diffed, full_diff)
if branch == 'master':
return # we don't want to update tickets and such for master/merges
commits = filter(lambda c: len(c), commits.split('-------[post-receive marker]------'))
commits.reverse()
for c in commits:
if c.find('Squashed commit') >= 0:
continue # Skip bullshit squashed commit
for attn in find_attn(c):
handle_attn(branch, c, attn)
for ticket in find_re(c):
handle_re(branch, c, ticket)
for ticket in find_qa(c):
handle_qa(branch, c, ticket)
if __name__ == '__main__':
op = OptionParser()
op.add_option('-m', '--mail', dest='address', help='Email address to mail git push messages to')
op.add_option('-o', '--oldrev', dest='oldrev', help='Old revision we\'re pushing from')
op.add_option('-n', '--newrev', dest='newrev', help='New revision we\'re pushing to')
op.add_option('-r','--ref', dest='ref', help='Refname that we\'re pushing')
opts, args = op.parse_args()
if not opts.address or not opts.oldrev or not opts.newrev or not opts.ref:
print '*** You left out some needed parameters! ***'
exit
mail_push(opts.address, opts.oldrev, opts.newrev, opts.ref)
Build Hudson Job
#!/bin/bash
MERGE_BRANCH=mergemaster
REPO=$1
BRANCH=$2
if [[ -z "${1}" || -z "${2}" ]]; then
echo "===> You must provide a \"remote\" and a \"refspec\" for Git to use!"
echo "===> Exiting :("
exit 1;
fi
LATEST_COMMIT=`git log --max-count=1 --no-merges --pretty=format:"%H"`
function master
{
echo "==> Making sure we're on 'master'"
git checkout master
}
function setup_mergemaster
{
master
echo "==> Killing the old mergemaster branch"
git branch -D $MERGE_BRANCH
echo "==> Creating a new mergemaster branch"
git checkout -b $MERGE_BRANCH
git checkout master
}
function cleanup
{
rm -f .git/SVNPULL_MSG
}
function prepare_message
{
master
echo "===> Pulling from ${REPO}:${BRANCH}"
git pull ${REPO} ${BRANCH}
git checkout ${MERGE_BRANCH}
echo "==> Merging across change from master to ${MERGE_BRANCH}"
git pull --no-commit --squash . master
cp .git/SQUASH_MSG .git/SVNPULL_MSG
master
}
function merge_to_svn
{
git reset --hard ${LATEST_COMMIT}
master
setup_mergemaster
echo "===> Pulling from ${REPO}:${BRANCH}"
git pull ${REPO} ${BRANCH}
git checkout ${MERGE_BRANCH}
echo "==> Merging across change from master to ${MERGE_BRANCH}"
git pull --no-commit --squash . master
echo "==> Committing..."
git commit -a -F .git/SVNPULL_MSG && git-svn dcommit --no-rebase
cleanup
}
setup_mergemaster
prepare_message
merge_to_svn
master
echo "===> All done!"function setup_mergemaster
{
master
echo "==> Killing the old mergemaster branch"
git branch -D $MERGE_BRANCH
echo "==> Creating a new mergemaster branch"
git checkout -b $MERGE_BRANCH
git checkout master
}What the setup_mergemaster branch is responsible for is deleting any prior branches that have been used for merging into the proxy repository and Primary_SVN. It gives us a "mergemaster" branch in the git-svn repository that is effectively at the same chronological point in time as the master branch before any merging occurs.
function prepare_message
{
master
echo "===> Pulling from ${REPO}:${BRANCH}"
git pull ${REPO} ${BRANCH}
git checkout ${MERGE_BRANCH}
echo "==> Merging across change from master to ${MERGE_BRANCH}"
git pull --no-commit --squash . master
cp .git/SQUASH_MSG .git/SVNPULL_MSG
master
}The prepare_message function is part of the nastiest code in the entire script, in order to get an accurate "squashed commit" commit message when the changesets are pushed into Primary_SVN, we have to generate the commit message separately from the actual merging. Since this function is performing a `git pull` from "master" into "mergemaster" the changesets that are being pulled are going to be the only ones that show up (for reasons I'm about to explain).
function merge_to_svn
{
git reset --hard ${LATEST_COMMIT}
master
setup_mergemaster
echo "===> Pulling from ${REPO}:${BRANCH}"
git pull ${REPO} ${BRANCH}
git checkout ${MERGE_BRANCH}
echo "==> Merging across change from master to ${MERGE_BRANCH}"
git pull --no-commit --squash . master
echo "==> Committing..."
git commit -a -F .git/SVNPULL_MSG && git-svn dcommit --no-rebase
cleanup
}If you noticed above in the full script block, the "LATEST_COMMIT" code, here's where it's used, it is one of the most important pieces of the entire script. Basically the LATEST_COMMIT piece of script grabs the latest non-merge-commit hash from the `git log` output and saves it for later use (here) where it's used to rollback the proxy repository to the point in time just before the last merge commit. This is done to avoid issues with git-svn(1) not understanding how to handle merge commits whatsoever. After rolling back the proxy repository, a new "mergemaster" branch is created. After the mergemaster branch is created, the actual Primary_Git changesets that differ between the proxy repository and Primary_Git are pulled into the proxy repository's master branch, and sqaushed into the mergemaster branch where they are subsequently committed with the commit message that was prepared before. The "prepare_message" part of the script becomes important at that step because the "squashed commit" message that Git generates at this point in time will effectively contain every commit that has ever been proxied across in this manner ever.
Turns out there was a race-condition here, re #52516
svn add some/directory/ and/or svn commit. This usually results in a second commit to come into the tree with a commit message like:Whoops, accidentally checked in resource forks
svn commit will commit all outstanding changes in your working copy starting in the current working directory, and recursing through children directories. I'm probably more anal-retentive about my commits than most, but I usually do a diff before I commit to make sure I'm aware of what I'm about to commit, but I've seen plenty of developers skip this step.git add .
git push and then mark the ticket appropriately such that a QA engineer can verify the fix. If Paul and Peter are working on something that "breaks the build" but they need to collaborate on it together, Paul can perform a git pull from Peter and vice versa, and again, once they're done those changes will be pushed to origin. This model allows for developers to work in relative isolation so they're not inadvertantly stepping on each others' toes, but also close enough that they can collaborate in explicit terms, i.e. when they are ready for changes to be propogated to each other or the rest of the team.
