Checking out old files WITH original create/modified timestamps

Is there a way to know or get the original create/modi开发者_如何学Pythonfied timestamps?

YES, metastore or git-cache-meta can store such (meta-)information! Git by itself, without third-party tools, can't. Metastore or git-cache-meta can store any file metadata for a file.

That is by design, as metastore or git-cache-meta are intended for that very purpose, as well as supporting backup utilities and synchronization tools.

I believe that the only timestamps recorded in the Git database are the author and commit timestamps. I don't see an option for Git to modify the file's timestamp to match the most recent commit, and it makes sense that this wouldn't be the default behavior (because if it were, Makefiles wouldn't work correctly).

You could write a script to set the modification date of your files to the the time of the most recent commit. It might look something like this:

# No arguments? Recursively list all git-controlled files in $PWD and start over
if [ $# = 0 ]; then
  git ls-files -z |xargs -0 sh "$0"
  exit $?
fi

for file in "$@"; do
  time="$(git log --pretty=format:%cd -n 1 \
                  --date=format:%Y%m%d%H%M.%S --date-order -- "$file")"
  if [ -z "$time" ]; then
    echo "ERROR: skipping '$file' -- no git log found" >&2
    continue
  fi
  touch -m -t "$time" "$file"
done

This accepts specific files as arguments or else updates each git-controlled file in the current directory or its children. This is done in a manner that permits spaces and even line breaks in filenames since git ls-files -z outputs a null-terminated file list and xargs -0 parses null-terminated lists into arguments.

This will take a while if you have a lot of files.

NO, Git simply does not store such (meta-)information, unless you use third-party tools like metastore or git-cache-meta. The only timestamp that get stored is the time a patch/change was created (author time), and the time the commit was created (committer time).

That is by design, as Git is a version control system, not a backup utility or synchronization tool.

UPDATE: TL;DR: Git itself does not save original times, but some solutions circumvent this by various methods. git-restore-mtime is one of them.

Ubuntu and Debian: sudo apt install git-restore-mtime Fedora, Red Hat Enterprise Linux (RHEL), and CentOS: sudo yum install git-tools

See my other answer for more details.

Full disclaimer: I'm the author of git-tools

This Python script may help: for each file, it applies the timestamp of the most recent commit where the file was modified:

• Core functionality, with --help, debug messages. Can be run anywhere within the work tree
• Full-fledged beast, with lots of options. Supports any repository layout.

Below is a really bare-bones version of the script. For actual usage I strongly suggest one of the more robust versions above:

#!/usr/bin/env python
# Bare-bones version. The current directory must be top-level of work tree.
# Usage: git-restore-mtime-bare [pathspecs...]
# By default update all files
# Example: to only update only the README and files in ./doc:
# git-restore-mtime-bare README doc

import subprocess, shlex
import sys, os.path

filelist = set()
for path in (sys.argv[1:] or [os.path.curdir]):
    if os.path.isfile(path) or os.path.islink(path):
        filelist.add(os.path.relpath(path))
    elif os.path.isdir(path):
        for root, subdirs, files in os.walk(path):
            if '.git' in subdirs:
                subdirs.remove('.git')
            for file in files:
                filelist.add(os.path.relpath(os.path.join(root, file)))

mtime = 0
gitobj = subprocess.Popen(shlex.split('git whatchanged --pretty=%at'),
                          stdout=subprocess.PIPE)
for line in gitobj.stdout:
    line = line.strip()
    if not line: continue

    if line.startswith(':'):
        file = line.split('\t')[-1]
        if file in filelist:
            filelist.remove(file)
            #print mtime, file
            os.utime(file, (mtime, mtime))
    else:
        mtime = long(line)

    # All files done?
    if not filelist:
        break

All versions parse the full log generated by a single git whatchanged command, which is hundreds of times faster than lopping for each file. It is under four seconds for Git (24,000 commits, 2,500 files) and less than one minute for the Linux kernel (40,000 files and 300,000 commits).

This did the trick for me on Ubuntu (which lacks OS X's "-j" flag on date(1)):

for FILE in $(git ls-files)
do
    TIME=$(git log --pretty=format:%cd -n 1 --date=iso $FILE)
    TIME2=`echo $TIME | sed 's/-//g;s/ //;s/://;s/:/\./;s/ .*//'`
    touch -m -t $TIME2 $FILE
done

I have been skirmishing with git and file timestamps for some time already.

Tested some of your ideas and made my own awfully huge and predecessor/ram heavy scripts, untill i found (on some git wiki) a script in perl that does almost what i wanted. https://git.wiki.kernel.org/index.php/ExampleScripts

And what i wanted is to be able to preserve last modification of files based on commit dates.

So after some readjustment the script is able to change creation and modification date of 200k files in around 2-3min.

#!/usr/bin/perl
my %attributions;
my $remaining = 0;

open IN, "git ls-tree -r --full-name HEAD |" or die;
while (<IN>) {
    if (/^\S+\s+blob \S+\s+(\S+)$/) {
        $attributions{$1} = -1;
    }
}
close IN;

$remaining = (keys %attributions) + 1;
print "Number of files: $remaining\n";
open IN, "git log -r --root --raw --no-abbrev --date=raw --pretty=format:%h~%cd~ |" or die;
while (<IN>) {
    if (/^([^:~]+)~([^~]+)~$/) {
        ($commit, $date) = ($1, $2);
    } elsif (/^:\S+\s+1\S+\s+\S+\s+\S+\s+\S\s+(.*)$/) {
        if ($attributions{$1} == -1) {
            $attributions{$1} = "$date";
            $remaining--;

            utime $date, $date, $1;
            if ($remaining % 1000 == 0) {               
                print "$remaining\n";
            }
            if ($remaining <= 0) {
                break;
            }
        }
    }
}
close IN;

Assuming that your repositories wont have 10k+ files this should take seconds to execute, so you can hook it to the checkout, pull or other git basic hooks.

Native Git doesn't have the functionality, but it can be achieved by hook scripts or third-party tools.

I've tried metastore. It's very fast, but I don't like the need to install and that metadata are not stored in plain text format. git-cache-meta is a simple tool I've tried, but it's extremely slow for large repositories (for a repository with tens of thousands of files, it takes minutes to update the metadata file) and could have cross-platform compatibility issues. setgitperms and other approaches also have their shortcomings that I don't like.

At last, I made a hook script for this job: git-store-meta. It has very light dependency (*nix shell, sort, and perl, which is required by Git, and optionally chown, chgrp and touch), so that nothing additional have to be installed for a platform that can run Git, desirable performance (for a repository with tens of thousands of files, it takes < 10 seconds to update the metadata file; although longer to create), saves data in plain text format, and which metadata to be "saved" or "loaded" is customizable.

It has worked fine for me. Try this if you are not satisfied with metastore, git-cache-meta, and other approaches.

I hope you appreciate the simplicity:

# getcheckin - Retrieve the last committed checkin date and time for
#              each of the files in the git project.  After a "pull"
#              of the project, you can update the timestamp on the
#              pulled files to match that date/time.  There are many
#              that believe that this is not a good idea, but
#              I found it useful to get the right source file dates
#
#              NOTE: This script produces commands suitable for
#                    piping into BASH or other shell
# License: Creative Commons Attribution 3.0 United States
# (CC by 3.0 US)

##########
# walk back to the project parent or the relative pathnames don't make
# sense
##########
while [ ! -d ./.git ]
do
    cd ..
done
echo "cd $(pwd)"
##########
# Note that the date format is ISO so that touch will work
##########
git ls-tree -r --full-tree HEAD |\
    sed -e "s/.*\t//" | while read filename; do
    echo "touch --date=\"$(git log -1 --date=iso --format="%ad" -- "$filename")\" -m $filename" 
done

Here is my solution that takes into consideration paths that contain spaces:

#! /bin/bash

IFS=$'\n'
list_of_files=($(git ls-files | sort))
unset IFS

for file in "${list_of_files[@]}"; do
  file_name=$(echo $file)

  ## When you collect the timestamps:
  TIME=$(date -r "$file_name" -Ins)

  ## When you want to recover back the timestamps:
  touch -m -d $TIME "$file_name"
done

Note that this does not take the time which git log reports; it's the time reported by the system. If you want the time since the files were committed use git log solution instead of date -r

Contrary to other solutions that set mtime to commit time, git-store-meta saves meta data like mtime into a .git_store_meta file that is added to the repository. It can install git hooks to the current repository that save and apply metadata automatically.

For a Windows environment, I wrote a small (quick and dirty) EXE file in Delphi 10.1 Berlin that collects all file dates in the source tree into the file .gitfilattr and can apply them on the checked our source tree again.

The code is on GitHub:

https://github.com/michaschumann/gitfiledates/blob/master/gitFileDates.dpr

I use it in my build system based on GitLab runners.

Git doesn't support storing file dates.

But you can use git-meta, which is git-cache-meta turned into a up-to-date repository (all of the comments in the gist were implemented); now it is installable as a Git hook, so it will automatically store metadata on every commit!

So, Git doesn't support storing files' metadata by default; but it doesn't mean you can't be modding it with custom features (LFS is a proof of how you can extend Git).

In CentOS 7 you have /usr/share/doc/rsync-*/support/git-set-file-times and in Debian (and derivatives) the same script in /usr/share/doc/rsync/scripts/git-set-file-times.gz. The original is from Eric Wong and is at https://yhbt.net/git-set-file-times.

It works faster than other examples mentioned here and you may find it more handy to have it already on your Linux distribution.

There's some ambiguity in my (and others') interpretation of the OP about whether this means the commit time or something else, but assuming it means commit time, then this simple one-liner will work in Linux (based on answer snippet from Dietrich Epp):

git ls-files | xargs -I{} bash -c 'touch "{}" --date=@$(git log -n1 --pretty=format:%ct -- "{}")'

But there are more sophisticated answers (including Git hooks) linked from a comment to the original question by cregox.

With GNU tools.

s=$(git ls-files  | wc -l); 
git ls-files -z  |
 xargs -0 -I{} -n1 bash -c \
"git log --date=format:%Y%m%d%H%M.%S '--pretty=format:touch -m -t %cd \"{}\"%n' -n1 -- {}"|
 pv -l -s$s |
 parallel -n1 -j8

 967  0:00:05 [ 171 /s] [=====================================>  ] 16% 

.

$ git --version ; xargs --version | sed 1q ; ls --version | sed 1q;
  parallel --version  | sed 1q;  pv --version | sed 1q; sh --version | sed 1q 
git version 2.13.0
xargs (GNU findutils) 4.6.0
ls (GNU coreutils) 8.25
GNU parallel 20150522
pv 1.6.0 - Copyright 2015 Andrew Wood <andrew.wood@ivarch.com>
GNU bash, version 4.3.48(1)-release (x86_64-pc-linux-gnu)

https://github.com/DotCi/jenkinsci-dotci-example/commit/5a45034d13b85ab4746650995db55b5281451cec#diff-a83424d0d40754ac7e2029b13daa2db43651eb65aabf8c9a5a45005b56f259bdR19

for file in `find . -type f -not -path "./.git/*"`; do 
  touch -d "`git rev-list -n 1 HEAD \$file | xargs git show -s --format=%ai`" $file; 
done

Here's mine.

A little quicker than some others, as I'm not calling 'get log' for each file found; instead, calling 'git log' once and transforming that output into touch commands.

There'll be cases where there are too many listed files in 1 commit to fit into a single shell command buffer; run "getconf ARG_MAX" to see the max length of a command in bytes - on my Debian install, it's 2 MB, which is plenty.

# Set file last modification time to last commit of file
git log --reverse --date=iso --name-only | \
  grep -vE "^(commit |Merge:|Author:|    |^$)" | \
  grep -B 1 "^[^D][^a][^t][^e][^:][^ ]" | \
  grep -v "^\-\-" | \
  sed "s|^\(.*\)$|\"\1\"|;s|^\"Date: *\(.*\)\"$|~touch -c -m -d'\1'|" | \
  tr '~\n' '\n ' | \
  sh -

Description by line:

• earliest-first list of commits and filenames
• filter out unneeded commit/merge/author lines
• filter out lines starting with double-dash
• sed (stream-edit) command a) prepend/append double-quote to lines, and b) replace "Date: ." with ~touch -c -m -d. ( the touch command options are -c = don't create if it doesn't exist, -m = change file modification time, and -d = use the provided date/time )
• translate tilde (~) and newline (\n) characters to newline and space, respectively
• pipe the resulting stream of text lines into a shell.

In terms of speed, it 5 seconds 1700 commits for 6500 files in 700 directories.