Skip to content
Snippets Groups Projects
  • Alexander V. Tikhonov's avatar
    05d3ed4b
    gitlab-ci: push Deb/RPM packages to S3 based repos · 05d3ed4b
    Alexander V. Tikhonov authored
    We're going to use S3 compatible storage for Deb and RPM repositories
    instead of packagecloud.io service. The main reason is that
    packagecloud.io provides a limited amount of storage, which is not
    enough for keeping all packages (w/o regular pruning of old versions).
    
    Note: At the moment packages are still pushed to packagecloud.io from
    Travis-CI. Disabling this is out of scope of this patch.
    
    This patch implements saving of packages on an S3 compatible storage and
    regeneration of a repository metadata.
    
    The layout is a bit different from one we have on packagecloud.io.
    
    packagecloud.io:
    
     | - 1.10
     | - 2.1
     | - 2.2
     | - ...
    
    S3 compatible storage:
    
     | - live
     |   - 1.10
     |   - 2.1
     |   - 2.2
     |   - ...
     | - release
     |   - 1.10
     |   - 2.1
     |   - 2.2
     |   - ...
    
    Both 'live' and 'release' repositories track release branches (named as
    <major>.<minor>) and master branch. The difference is that 'live' is
    updated on every push, but 'release' is only for tagged versions
    (<major>.<minor>.<patch>.0).
    
    Packages are also built on '*-full-ci' branches, but only for testing
    purposes: they don't pushed anywhere.
    
    The core logic is in the tools/update_repo.sh script, which implements
    the following flow:
    
    - create metadata for new packages
    - fetch relevant metadata from the S3 storage
    - push new packages to the S3 storage
    - merge and push the updated metadata to the S3 storage
    
    The script uses 'createrepo' for RPM repositories and 'reprepro' for Deb
    repositories.
    
    Closes #3380
    gitlab-ci: push Deb/RPM packages to S3 based repos
    Alexander V. Tikhonov authored
    We're going to use S3 compatible storage for Deb and RPM repositories
    instead of packagecloud.io service. The main reason is that
    packagecloud.io provides a limited amount of storage, which is not
    enough for keeping all packages (w/o regular pruning of old versions).
    
    Note: At the moment packages are still pushed to packagecloud.io from
    Travis-CI. Disabling this is out of scope of this patch.
    
    This patch implements saving of packages on an S3 compatible storage and
    regeneration of a repository metadata.
    
    The layout is a bit different from one we have on packagecloud.io.
    
    packagecloud.io:
    
     | - 1.10
     | - 2.1
     | - 2.2
     | - ...
    
    S3 compatible storage:
    
     | - live
     |   - 1.10
     |   - 2.1
     |   - 2.2
     |   - ...
     | - release
     |   - 1.10
     |   - 2.1
     |   - 2.2
     |   - ...
    
    Both 'live' and 'release' repositories track release branches (named as
    <major>.<minor>) and master branch. The difference is that 'live' is
    updated on every push, but 'release' is only for tagged versions
    (<major>.<minor>.<patch>.0).
    
    Packages are also built on '*-full-ci' branches, but only for testing
    purposes: they don't pushed anywhere.
    
    The core logic is in the tools/update_repo.sh script, which implements
    the following flow:
    
    - create metadata for new packages
    - fetch relevant metadata from the S3 storage
    - push new packages to the S3 storage
    - merge and push the updated metadata to the S3 storage
    
    The script uses 'createrepo' for RPM repositories and 'reprepro' for Deb
    repositories.
    
    Closes #3380