6

I have the pleasure of taking over 14 year old subversion repository that consists of two key elements:

  1. 111,000 revisions, about 10% of which are substantial;
  2. The repository dump is about 73 GB, due to a large number of binary data files that have been updated, added and modified over the years.

Here's what I'd like to do, but I'm not sure if it's possible: I'd like to strip the history of the binary files and only keeping the code changes. Then convert that to git. What are your recommendations?

Community
  • 1
Ben
  • 159
  • 5
  • 4
    If it were me, I'd start a fresh Git repository from the last version of code in the SVN repository, without preserving any history, and keep the SVN repository around for awhile just in case anyone needs to dig into the history. – Robert Harvey May 03 '18 at 19:23
  • 1
    Yes, Git has the `filter-branch` command which can be used to bulk-rewrite history. Removing files from the complete history is one use case. However, this is pretty advanced functionality so be sure to test this carefully first. If you have questions about its usage, those might be better for Stack Overflow as this site is about software engineering concepts. Note that Git has the concept of Large File Storage specially for large binary assets that should not be part of the main repo. – amon May 03 '18 at 19:24
  • 1
    Not a full dupe, but very much related: https://softwareengineering.stackexchange.com/questions/286124/what-to-do-about-large-svn-history-when-moving-to-git – Doc Brown May 03 '18 at 19:59

2 Answers2

4

not sure if it's possible: I'd like to strip the history of the binary files, only keeping the code changes. Then convert that to git.

I did not try this not by myself, but I am pretty sure this is possible when you approach the task exactly in the order you described above:

  1. strip the history of the binary files, only keeping the code changes, in SVN first

  2. migrate to Git afterwards.

Step 1 can be accomplished by moving the binary files with their history into some temporary folder (inside the repo, for example with svn move). Then you create a fresh copy of them in your projects's local working directory and check them in as if they were new files - so those new files have no history. Then you use the procedure described in this server fault question to get rid of the temporary folder (utilizing svnadmin dump, svndumpfilter, svnadmin load), which deletes the full history.

Note this way whenever you will check out an older revision of the project, the binary files will be missing completely. To avoid this becoming a problem, consider the strategy of keeping the old SVN repo online, as suggested by @RobertHarvey.

Doc Brown
  • 199,015
  • 33
  • 367
  • 565
4

OP here. For posterity, I wanted to add what my ultimate solution to this problem was. I will keep the best answer checked, because it's actually the best answer when things go smoothly.

First, I tried the selected answer from Doc Brown: https://softwareengineering.stackexchange.com/a/370419/291998

But the svnadmin dump command failed about halfway into the dump (about 1 day into the dump), with a corrupted revision. This was a consistent point of failure. Attempts to bypass this specific reference, through careful use of the revision flags to svnadmin dump, I was able to produce a full dump, however, attempts to filter binary files out of the 73 GB dump file were met with additional frustration. I did manage to make some dumps of particular branches, but this was useless to me for a full migration.

Ultimately, I ended up using git-svn to do this: nohup git svn init https://myurl.com/projects/myproject/ --no-minimize-url --no-metadata --stdlayout &

Using git-svn on such a large repository was not without issues. It regularly choked during the processing. Fortunately, git svn fetch could be issued to resume to conversion at each stopping point. I wrote a small wrapper script that would continue issuing that command upon failure.

Having said all of this, git-svn is not recommended for this task, see this article: http://esr.ibiblio.org/?p=6778

But this was ultimately the only tool I could get to work. For stripping the binary files, I used BFG repo cleaner https://rtyley.github.io/bfg-repo-cleaner/

Ben
  • 159
  • 5
  • 1
    Nothing ever goes smoothly migrating large repositories. I had to do something similar with SVN years ago (minus stripping large binary files). I had to write a shell script that fetched 1,000 SVN commits at a time using git-svn to map things over in batches. We had 200k+ commits in that old svn repository. – Greg Burghardt Sep 26 '18 at 16:03