Add documentation.
This commit is contained in:
parent
9912c73e24
commit
a28caf9ea5
2 changed files with 92 additions and 3 deletions
84
docs/content/doc/packages/storage.en-us.md
Normal file
84
docs/content/doc/packages/storage.en-us.md
Normal file
|
@ -0,0 +1,84 @@
|
|||
---
|
||||
date: "2022-11-01T00:00:00+00:00"
|
||||
title: "Storage"
|
||||
slug: "packages/storage"
|
||||
draft: false
|
||||
toc: false
|
||||
menu:
|
||||
sidebar:
|
||||
parent: "packages"
|
||||
name: "storage"
|
||||
weight: 5
|
||||
identifier: "storage"
|
||||
---
|
||||
|
||||
# Storage
|
||||
|
||||
This document describes the storage of the package registry and how it can be managed.
|
||||
|
||||
**Table of Contents**
|
||||
|
||||
{{< toc >}}
|
||||
|
||||
## Deduplication
|
||||
|
||||
The package registry has a build-in deduplication of uploaded blobs.
|
||||
If two identical files are uploaded only one blob is saved on the filesystem.
|
||||
This ensures no space is wasted for duplicated files.
|
||||
|
||||
If two packages are uploaded with identical files, both packages will display the same size but on the filesystem they require only half of the size.
|
||||
Whenever a package gets deleted only the references to the underlaying blobs are removed.
|
||||
The blobs get not removed at this moment, so they still require space on the filesystem.
|
||||
When a new package gets uploaded the existing blobs may get referenced again.
|
||||
|
||||
These unreferenced blobs get deleted by a [clean up job]({{< relref "doc/advanced/config-cheat-sheet.en-us.md#cron---cleanup-expired-packages-croncleanup_packages" >}}).
|
||||
The config setting `OLDER_THAN` configures how long unreferenced blobs are kept before they get deleted.
|
||||
|
||||
## Cleanup Rules
|
||||
|
||||
Package registries can become large over time without cleanup.
|
||||
It's recommended to delete unnecessary packages and set up cleanup rules to automatically manage the package registry usage.
|
||||
Every package owner (user or organization) manages the cleanup rules which are applied to their packages.
|
||||
|
||||
|Setting|Description|
|
||||
|-|-|
|
||||
|Enabled|Turn the cleanup rule on or off.|
|
||||
|Type|Every rule manages a specific package type.|
|
||||
|Apply pattern to full package name|If enabled, the patterns below are applied to the full package name (`package/version`). Otherwise only the version (`version`) is used.|
|
||||
|Keep the most recent|How many versions to *always* keep for each package.|
|
||||
|Keep versions matching|The regex pattern that determines which versions to keep. An empty pattern keeps no version while `.+` keeps all versions. The container registry will always keep the `latest` version even if not configured.|
|
||||
|Remove versions older than|Remove only versions older than the selected days.|
|
||||
|Remove versions matching|The regex pattern that determines which versions to remove. An empty pattern or `.+` leads to the removal of every package if no other setting tells otherwise.|
|
||||
|
||||
Every cleanup rule can show a preview of the affected packages.
|
||||
This can be used to check if the cleanup rules is proper configured.
|
||||
|
||||
### Regex examples
|
||||
|
||||
Regex patterns are automatically surrounded with `\A` and `\z` anchors.
|
||||
Do not include any `\A`, `\z`, `^` or `$` token in the regex patterns as they are not necessary.
|
||||
The patterns are case-insensitive which matches the behaviour of the package registry in Gitea.
|
||||
|
||||
|Pattern|Description|
|
||||
|-|-|
|
||||
|`.*`|Match every possible version.|
|
||||
|`v.+`|Match versions that start with `v`.|
|
||||
|`release`|Match only the version `release`.|
|
||||
|`release.*`|Match versions that are either named or start with `release`.|
|
||||
|`.+-temp-.+`|Match versions that contain `-temp-`.|
|
||||
|`v.+\|release`|Match versions that either start with `v` or are named `release`.|
|
||||
|`package/v.+\|other/release`|Match versions of the package `package` that start with `v` or the version `release` of the package `other`. This needs the setting *Apply pattern to full package name* enabled.|
|
||||
|
||||
### How the cleanup rules work
|
||||
|
||||
The cleanup rules are part of the [clean up job]({{< relref "doc/advanced/config-cheat-sheet.en-us.md#cron---cleanup-expired-packages-croncleanup_packages" >}}) and run periodicly.
|
||||
|
||||
The cleanup rule:
|
||||
|
||||
1. Collects all packages of the package type for the owners registry.
|
||||
1. For every package it collects all versions.
|
||||
1. Excludes from the list the # versions based on the *Keep the most recent* value.
|
||||
1. Excludes from the list any versions matching the *Keep versions matching* value.
|
||||
1. Excludes from the list the versions more recent than the *Remove versions older than* value.
|
||||
1. Excludes from the list any versions not matching the *Remove versions matching* value.
|
||||
1. Deletes the remaining versions.
|
|
@ -352,7 +352,7 @@ func DeletePackageFile(ctx context.Context, pf *packages_model.PackageFile) erro
|
|||
}
|
||||
|
||||
// Cleanup removes expired package data
|
||||
func Cleanup(_ context.Context, olderThan time.Duration) error {
|
||||
func Cleanup(taskCtx context.Context, olderThan time.Duration) error {
|
||||
ctx, committer, err := db.TxContext()
|
||||
if err != nil {
|
||||
return err
|
||||
|
@ -360,6 +360,12 @@ func Cleanup(_ context.Context, olderThan time.Duration) error {
|
|||
defer committer.Close()
|
||||
|
||||
err = packages_model.IterateEnabledCleanupRules(ctx, func(ctx context.Context, pcr *packages_model.PackageCleanupRule) error {
|
||||
select {
|
||||
case <-taskCtx.Done():
|
||||
return db.ErrCancelled{"While processing package cleanup rules"}
|
||||
default:
|
||||
}
|
||||
|
||||
if err := pcr.CompiledPattern(); err != nil {
|
||||
return fmt.Errorf("CleanupRule [%d]: CompilePattern failed: %w", pcr.ID, err)
|
||||
}
|
||||
|
@ -403,7 +409,7 @@ func Cleanup(_ context.Context, olderThan time.Duration) error {
|
|||
continue
|
||||
}
|
||||
if pcr.RemovePatternMatcher != nil && !pcr.RemovePatternMatcher.MatchString(toMatch) {
|
||||
log.Debug("Rule[%d]: do not remove '%s/%s'", pcr.ID, p.Name, pv.Version)
|
||||
log.Debug("Rule[%d]: keep '%s/%s' (remove pattern)", pcr.ID, p.Name, pv.Version)
|
||||
continue
|
||||
}
|
||||
|
||||
|
@ -417,7 +423,6 @@ func Cleanup(_ context.Context, olderThan time.Duration) error {
|
|||
return nil
|
||||
})
|
||||
if err != nil {
|
||||
log.Error("%#v", err)
|
||||
return err
|
||||
}
|
||||
|
||||
|
|
Loading…
Add table
Reference in a new issue