Running a Subversion mod_dav_svn server

Home : Linux resources : Subversion server


  1. Running a Subversion mod_dav_svn server
    1. Setting up a Subversion server using Apache and HTTPS
      1. Set up Apache mod_ssl
      2. Build Subversion to use Apache and SSL
      3. Configure Apache to use mod_dav_svn
      4. Set up Apache2 to use mod_authz_svn
    2. Test the new server
    3. Moving existing repositories to Subversion
      1. Migrate rgr-hacks from CVS to Subversion
      2. Create a new Subversion repository via import
    4. Backup of Subversion repositories
    5. Move a Subversion repository to a new server
    6. References

Setting up a Subversion server using Apache and HTTPS

This recipe was used to set my server up initially on SuSE 9.0 (where the shipped RPM edition of Subversion is so old as to be completely useless, so I built Subversion 1.3.0 from the tarball), and again when I migrated it to new hardware running openSUSE 10.2 (which comes with Subversion 1.4.0).

Much of this is specific to the details of the SuSE configuration (though I'm starting to include CentOS information as well). Other installations will require larger or smaller grains of salt.

Set up Apache mod_ssl

  1. On CentOS it is also necessary to install the mod_ssl and mod_dav_svn packages in addition to httpd. The main SSL configuration file is put in /etc/httpd/conf.d/ssl.conf.
  2. Create an SSL certificate. For testing, I created a self-signed cert using a recipe from the OpenSSL man page:
           openssl req -x509 -newkey rsa:1024 -sha1 -days 90 -keyout key.pem -out req.pem
           
    Be sure to specify -sha1; if you don't, you will get MD5. The -x509 is what makes openssl req generate a self-signed certificate. Specifying -days 90 produces a cert valid for 90 days; the default is thirty (and this is ignored if you generate a proper signing request instead of a self-signature). This recipe is based on the "req(1)" man page, which explains what all the options mean.
  3. Copy key.pem and req.pem to the appropriate destination directory:
    OS key.pem (private key) req.pem (certificate)
    OpenSUSE /etc/apache2/ssl.key/server.key /etc/apache2/ssl.crt/server.crt
    CentOS /etc/pki/tls/private/server.key /etc/pki/tls/certs/server.crt
    In all cases, it is probably desirable to replace the word "server" with something specific to the server in question -- just be sure you do the same for the SSL config file.
  4. Configure Apache 2.x for SSL. This consists of (a) copying (or renaming) /etc/apache2/vhosts.d/vhost-ssl.template to vhost-ssl.conf (in the same directory), and (b) changing /etc/sysconfig/apache2 to add "-D SSL" to APACHE_SERVER_FLAGS and increase APACHE_START_TIMEOUT to 10 seconds (so that you have time to type in a passphrase).
  5. Restart Apache via "/etc/init.d/apache2 restart". This requires supplying the passphrase used to protect the server private key.
  6. Verify that SSL is working by visiting the HTTPS version of the site.

Build Subversion to use Apache and SSL

This step is only necessary if you have an old distro release. If the release you have includes at least Subversion 1.2, then you are probably better off sticking to the distro version, which is undoubtedly already compiled correctly to play nicely with Apache.

Note that this recipe installs the Subversion binaries into the default /usr/local/bin/ location. In order to use the backup script described below, you may need to add /usr/local/bin/ to $PATH.

  1. Ensure that the openssl-devel and apache2-devel packages are installed so that Subversion's ./configure script can find their respective header files.
  2. Configure Subversion as follows:
           CFLAGS=`apxs2 -q CFLAGS` ./configure --enable-shared \
                   --with-ssl --with-apxs=`which apxs2` --disable-mod-activation
           
    These options have the following effect:

    Some of this may be voodoo, but I can tell you that without --enable-shared plus the CFLAGS magic, Apache segfaulted every time it got a WebDAV request.
  3. Do "make" to compile.
  4. As root, do "make install". (I had to remove the "-a" at the end of the INSTALL_MOD_SHARED line in the generated Makefile by hand, because I didn't know I needed --disable-mod-activation until too late.)

Configure Apache to use mod_dav_svn

This section shows how to set up mod_dav_svn to serve multiple repositories (using
the SVNParentPath option) with anonymous read access and authenticated write access using the same user credentials for all repositories. The next section modifies this recipe to use mod_authz_svn, which provide per-directory access control. Even if you don't want to provide anonymous access, or need per-directory control, I recommend starting here in order to test basic mod_dav_svn functionality before adding authentication. In particular, an anonymous checkout doesn't need any authentication, so is useful for isolating authentication problems from more basic ones.
  1. Create the /shared/svn directory, and at least one repository under it, and chown it to the wwwrun user. I created a /shared/svn/new-vc/ subdirectory and copied an existing repository there; this shows up on the server as https://rgrjr.dyndns.org/svn/new-vc/.
  2. Add dav and dav_svn to the APACHE_MODULES list in /etc/sysconfig/apache2. Note that this list is order-dependent, so dav must appear before dav_svn.
  3. Create users in /etc/svn-auth-file using the Apache htpasswd utility (which is actually called htpasswd2 if you have the apache2 RPMs installed under openSUSE). See the Basic HTTP Authentication section of the SVN book for an example.
  4. Add the following to /etc/apache2/vhosts.d/vhost-ssl.conf within the <VirtualHost> definition:
           # Subversion server setup.  -- rgr, 20-Mar-06.
           <Location /svn>
    	 DAV svn
    	 # any "/svn/foo" URL will map to a repository /shared/svn/foo
    	 SVNParentPath /shared/svn
    
    	 # how to authenticate.
    	 AuthType Basic
    	 AuthName "Subversion repository at rgrjr.dyndns.org"
    	 AuthUserFile /etc/svn-auth-file
    
             # require authentication for anything beyond read access.
    	 <LimitExcept GET PROPFIND OPTIONS REPORT>
    	   Require valid-user
    	 </LimitExcept>
           </Location>
           
  5. Restart the server.

Set up Apache2 to use mod_authz_svn

  1. Add mod_authz_svn after dav_svn in the APACHE_MODULES list in /etc/sysconfig/apache2.
  2. Change the <Location /svn> section in the /etc/apache2/vhosts.d/vhost-ssl.conf file to look like this:
    	# Subversion server setup.  -- rgr, 13-Dec-07.
    	<Location /svn>
    	  DAV svn
    	  # any "/svn/foo" URL will map to a repository /shared/svn/foo
    	  SVNParentPath /shared/svn
    
    	  # access control policy file.
              AuthzSVNAccessFile /etc/svn-access.conf
    
    	  # try anonymous access first, resort to real
    	  # authentication if necessary.
    	  Satisfy Any
    	  Require valid-user
    
    	  # how to authenticate.
    	  AuthType Basic
    	  AuthName "Subversion repository at rgrjr.dyndns.org [new]"
    	  AuthUserFile /etc/svn-auth-file
    	</Location>
           
    The difference between this and the previous <Location> recipe is that the <LimitExcept> part is is replaced by top-level "Require valid-user" and "Satisfy Any" options, and the AuthzSVNAccessFile identifies an access file. Note that user credentials are still defined with htpasswd2 in /etc/svn-auth-file as before.
  3. Create the /etc/svn-access.conf file with the necessary per-directory permissions. An early version of mine looks like this:
           # Subversion authentication.  See
           # http://svnbook.red-bean.com/nightly/en/svn.serverconfig.pathbasedauthz.html
           # for details.
    
           [/]
           # Allow everyone to read the entire repository, unless overridden.
           * = r
           # Give write permission to rogers and rgr.
           rogers = rw
           rgr = rw
    
           [lap-config:/]
           # This is private, so we need "* =" to revoke anonymous reads, and
           # "rogers = rw" to reinstate write permission.
           * =
           rogers = rw
           
    The "[/]" paragraph reinstates the anonymous read/authenticated write policy of the previous section for all directories of all repositories, and the final paragraph restricts all access to the lap-config repository to rogers.
  4. Restart the server so that all of this will take effect.

Test the new server

This requires having converted/imported at least one repository first; see
the next section if this is an initial installation.
  1. Test basic server availability by doing
           rogers@rgr> svn ls https://rgrjr.dyndns.org/svn/rgr-hacks/
           branches/
           tags/
           trunk/
           rogers@rgr> 
    This is the expected output for a repository created by cvs2svn; if you look in trunk/, you will see the version-controlled files in the top-level directory.
  2. Test server browsability. In a Web browser, just visit the top-level repository URL (e.g. https://rgrjr.dyndns.org/svn/rgr-hacks/) and you should be able to browse the entire repository at its latest revision.
  3. Test checking out:
           svn co https://rgrjr.dyndns.org/svn/rgr-hacks/trunk rgr-hacks
    and look at the new working copy in the rgr-hacks subdirectory.
  4. Find a version-controlled file in Emacs.
  5. View its log (C-x v l in Emacs).
  6. Modify and save it.
  7. Diff it against the original (C-x v =).
  8. Check it in (C-x v v).
  9. View the updated log.

Moving existing repositories to Subversion

Here we give an example of using cvs2svn to convert an existing CVS repository to Subversion, and of creating a new repository as a fresh checkin.

Migrate rgr-hacks from CVS to Subversion

This uses
cvs2svn, which is distributed along with openSUSE 10.2. For concreteness, we convert the rgr-hacks code (online at https://rgrjr.dyndns.org/svn/rgr-hacks/), but this recipe should work generically.
  1. Ensure that all image files (and anything else that shouldn't have EOL conversion done) is marked as "-kb" in the CVS repository. This doesn't matter for CVS as long as you only develop on Unix-like systems, but cvs2svn gets it wrong if these files are not marked properly in the repository. (Unless you specify --mime-type and --eol-from-mime-type to cvs2svn, but I have not tried this.)

    To find binary files in a CVS working copy:

           rogers@rgr> cd ~/emacs/rgr-hacks
           rogers@rgr> find . -type f -name Entries \
    		       | xargs -e egrep -n -e '\.(png|jpg|gif)/' \
    		       | fgrep -v /-kb/
           
    To fix them:
           rogers@rgr> cvs admin -kb *.jpg
           rogers@rgr> cvs update
           
    Note that cvs admin operates on the repository immediately; the cvs update is only needed to update the working copy, but that is useful for subsequent searches.

    Somewhat more problematic are files that are supposed to have DOS-style "CRLF" line endings, regardless of which OS they are checked out on. Even if marked "-kb", cvs2svn still treats these as "text/plain" and initializes their svn:eol-style to "native".

    To find CRLF files (approximately) in a CVS working copy:

           rogers@rgr> find . -type f | fgrep -v /CVS/ | xargs fgrep -l '\r'
           
    Since these often have "*.text" or other file extensions that are also used for files that should have "native" EOL style, there is no good way for cvs2svn to tell them apart, so I just fix them after the conversion (see below).

    See also "How do I fix up end-of-line translation problems?" in the cvs2svn FAQ.

  2. Create the repository (as root):
           # svnadmin create /shared/svn/rgr-hacks
           # chown -R rogers /shared/svn/rgr-hacks
           
  3. As rogers, populate it with the CVS contents:
           rogers@rgr> cvs2svn --existing-svnrepos -s /shared/svn/rgr-hacks/ /shared/cvsroot/rogers/emacs/rgr-hacks/
           
    Note that by default, cvs2svn will stuff everything that is not a branch into a trunk/ subdirectory.
  4. Give it back to the Web server user (as root, of course):
           # chown -R wwwrun /shared/svn/rgr-hacks
           
  5. Deal with any exceptions to cvs2svn's default EOL-style rules. Find these files by using the recipe provided in the first item (being careful to exclude image and other true binary files!), and then do:
           rogers@rgr> xargs svn propset svn:eol-style CRLF < dos-eol-files.text
           rogers@rgr> xargs perl -pi.bk -e 's/$/\r/;' < dos-eol-files.text
           
    After testing, do:
           rogers@rgr> svn ci -m 'Fix svn:eol-style on DOS-like files.'
           rogers@rgr> find . -name '*.bk' | xargs rm
           
    to commit and clean up.
  6. Test by checking out a fresh working copy:
           rogers@rgr> cd ..
           rogers@rgr> mv rgr-hacks rgr-hacks.cvs
           rogers@rgr> svn co https://rgrjr.dyndns.org/svn/rgr-hacks/trunk rgr-hacks
           rogers@rgr> diff -ur rgr-hacks.cvs rgr-hacks
           
    Note that diff will find many trivial differences in $Id:$ tags. If image files are reported to be different, then you probably need to go back to step 1.

Create a new Subversion repository via import

This shows how to populate a repository from scratch using
svn import, rather than from the contents of a CVS repository.
  1. Create an empty repository on the server (as root):
           rgrjr:~ # svnadmin create /shared/svn/lap-config
           rgrjr:~ # chown -R wwwrun.www /shared/svn/lap-config
           rgrjr:~ # 
           
  2. Move the files into a new tmp/trunk directory. tmp needs to contain everything that is to go into the new repository; the name "tmp" will not appear in the repository, but everything below it will. A "trunk" subdirectory under the repository root is the recommended place to put the main development version of a system.
           rogers@lap> mkdir -p tmp/trunk
           rogers@lap> cd tmp/trunk
           rogers@lap> tar xzf source-of-files.tgz
           rogers@lap> cd ../..
           rogers@lap> ls tmp/trunk/
           fstab.mgi     hosts.random     README.text.~1~    routes.rgrjr
           fstab.random  ntp.conf.mgi     resolv.conf.mgi    site-setup
           HOSTNAME.mgi  ntp.conf.random  resolv.conf.rgrjr  ssh_config
           hosts.mgi     README.text      routes.mgi         sshd_config
           rogers@lap> 
           
    Note that Subversion will not include README.text.~1~, knowing that it is an old version of the README.text file.
  3. Import the files:
           rogers@lap> svn import tmp https://rgrjr.dyndns.org/svn/lap-config -m "Initial import (SuSE 9.1)"
           Adding         tmp/trunk
           Adding         tmp/trunk/hosts.random
           Adding         tmp/trunk/README.text
           Adding         tmp/trunk/ssh_config
           Adding         tmp/trunk/resolv.conf.rgrjr
           Adding         tmp/trunk/sshd_config
           . . .
    
           Committed revision 1.
           rogers@lap> svn list https://rgrjr.dyndns.org/svn/lap-config
           trunk/
           rogers@lap> 
           
  4. The tmp/ directory tree is no longer needed.
           rogers@lap> rm -fr tmp
           
  5. Check it out:
           rogers@lap> svn co https://rgrjr.dyndns.org/svn/lap-config/trunk lap-config
           A    lap-config/hosts.random
           A    lap-config/README.text
           A    lap-config/ssh_config
           A    lap-config/resolv.conf.rgrjr
           A    lap-config/sshd_config
           A    lap-config/HOSTNAME.mgi
           A    lap-config/routes.mgi
           A    lap-config/ntp.conf.mgi
           A    lap-config/site-setup
           A    lap-config/ntp.conf.random
           A    lap-config/routes.rgrjr
           A    lap-config/resolv.conf.mgi
           A    lap-config/fstab.mgi
           A    lap-config/hosts.mgi
           A    lap-config/fstab.random
           Checked out revision 1.
           rogers@lap> cd lap-config/
           rogers@lap> ls
           fstab.mgi     hosts.mgi     ntp.conf.random  resolv.conf.rgrjr  site-setup
           fstab.random  hosts.random  README.text      routes.mgi         ssh_config
           HOSTNAME.mgi  ntp.conf.mgi  resolv.conf.mgi  routes.rgrjr       sshd_config
           rogers@lap> 
           
    This creates a normal working copy.

Backup of Subversion repositories

Backups of Subversion repositories are done using the
svnadmin dump command. I have written a wrapper script, svn-dump.pl, that creates incremental dumps in the current directory of one or more repositories. This script is suitable for running from a cron job:
    # Back up Subversion repositories daily at 00:50.  This is ten minutes before
    # the normal /home partition backup time. 
    50 0 * * *	cd /home/rogers/projects/svn-dump && svn-dump.pl /shared/svn/*
Since all mod_dav_svn repositories on my server are subdirectories of /shared/svn/, this catches everything with a single command. The files that are created look like this:
    rogers@rgr> ls -lt /home/rogers/projects/svn-dump
    total 5468
    -rw-r--r-- 1 rogers users   79596 06-29 00:50 web-site-213-235.svndump
    -rw-r--r-- 1 rogers users   19269 06-29 00:50 rgr-hacks-397-414.svndump
    -rw-r--r-- 1 rogers users   24044 06-29 00:50 kea-cl-558-568.svndump
    -rw-r--r-- 1 rogers users  172628 05-16 00:50 web-site-164-212.svndump
    -rw-r--r-- 1 rogers users    3210 04-14 00:50 hacks-4-4.svndump
    -rw-r--r-- 1 rogers users   56310 03-31 00:50 kea-cl-523-557.svndump
    -rw-r--r-- 1 rogers users 2264449 03-16 00:50 web-site-0-163.svndump
    -rw-r--r-- 1 rogers users    4476 03-03 00:50 rgr-hacks-392-396.svndump
    -rw-r--r-- 1 rogers users     903 02-05 00:50 rgr-hacks-391-391.svndump
    . . .
Most files with only a few revisions are quite small, since each file records only the incremental changes since the last dump. The web-site-0-163.svndump file is large since that is the first dump since converting this repository from CVS.

Periodically, in order to reduce the clutter of small *.svndump files, I delete the most recent files, and the next time the svn-dump.pl cron job runs, these files are consolidated into a single new *.svndump file.

Move a Subversion repository to a new server

This shows how to move an existing Subversion repository using
svnadmin dump and svnadmin load. It goes without saying that commits must be disabled on both servers during the move.
  1. If you don't already have backup dump files handy (or there are too many of them), create a dump file of the entire existing repo:
           rogers@home> svnadmin dump /shared/svn/rgr-hacks/ > rgr-hacks.svndump
           * Dumped revision 0.
           . . .
           * Dumped revision 469.
           * Dumped revision 470.
           * Dumped revision 471.
           rogers@home> 
    In this case, an 8.6MB file was generated for a repository that is only 5.7MB. If dumping creates a file that is unmanageably large, you might want to use the --deltas option of svnadmin dump. Note that svnadmin dump requires a path to the repository, not a URL, so this must be done on the original server. Note also that this command only writes to standard output, hence the redirection.
  2. Copy to the new server:
           rogers@home> scp rgr-hacks.svndump rgrjr.com:
           rgr-hacks.svndump                              100% 8652KB 540.7KB/s   00:16    
           rogers@home> 
  3. Create the new repository on the destination server:
           # svnadmin create /var/www/subversion/rgr-hacks
           # 
  4. Load the dump file into the new repository:
           # svnadmin load /var/www/subversion/rgr-hacks < rgr-hacks.svndump
           <<< Started new transaction, based on original revision 1
    
           . . .
    
           <<< Started new transaction, based on original revision 471
    	    * editing path : trunk/rgr-perl-hacks.el ... done.
    
           ------- Committed revision 471 >>>
    
           # 
    (svnadmin load is even more verbose than svnadmin dump, so this transcript is even more severely truncated.)
  5. Give ownership of the new repository to the Web user:
           # chown -R wwwrun.www /var/www/subversion/rgr-hacks
           # 
  6. Test the new server as described above.
  7. Working copies created from the old server can be switched to the new one:
           svn switch --relocate https://rgrjr.dyndns.org/svn/rgr-hacks/ https://www.rgrjr.com/svn/rgr-hacks/
    The --relocate option means to move the working copy to the same version of the same repository but in a different location, so svn switch only needs to rewrite the URLs. In particular, this means the working copy doesn't need to be up-to-date, nor do any modified files need to be committed. It's also quite fast.
See also the Migrating Repository Data Elsewhere section of the Subversion book.

References


Bob Rogers <rogers@rgrjr.dyndns.org>