Add garbage collection for docker images #2435
Conversation
| ) | ||
|
|
||
| // ByteSize represents the size of a file. | ||
| type ByteSize uint64 |
There was a problem hiding this comment.
Type mostly copied from https://github.com/elastic/package-spec/blob/964c4a69e024cc464c4808720ba0db9f001a82a7/code/go/internal/spectypes/filesize.go, adding support for GB and floats, to support sizes reported by docker system df.
| @@ -50,6 +51,34 @@ func (eb *envBuilder) build() []string { | |||
| return eb.vars | |||
| } | |||
|
|
|||
| func runDockerImagesGC(ctx context.Context, project *compose.Project, opts compose.CommandOptions, appConfig *install.ApplicationConfiguration) error { | |||
There was a problem hiding this comment.
Added by now only to stack operations, it could be added too to agent and service deployers. It may be specially interesting in the service deployer.
| @@ -60,7 +61,23 @@ type Config struct { | |||
| Services map[string]service | |||
| } | |||
|
|
|||
| // Images lists the images found in the configuration. | |||
There was a problem hiding this comment.
This only lists images found directly in the docker compose configuration, it won't find images found in Dockerfiles. Maybe we can think in other approaches to find the images to track. Ideas welcome.
| return s.unmarshalString(value.Value) | ||
| } | ||
|
|
||
| var bytesPattern = regexp.MustCompile(fmt.Sprintf(`^(\d+(\.\d+)?)(%s|%s|%s|%s|)$`, byteString, kiloByteString, megaByteString, gigaByteString)) |
There was a problem hiding this comment.
| var bytesPattern = regexp.MustCompile(fmt.Sprintf(`^(\d+(\.\d+)?)(%s|%s|%s|%s|)$`, byteString, kiloByteString, megaByteString, gigaByteString)) | |
| var bytesPattern = regexp.MustCompile(`^(\d+(\.\d+)?)(` + byteString + `|` + kiloByteString + `|` + megaByteString + `|` + gigaByteString + `|)$`) |
(compile-time string construction instead of init-time)
but for a more efficient pattern
| var bytesPattern = regexp.MustCompile(fmt.Sprintf(`^(\d+(\.\d+)?)(%s|%s|%s|%s|)$`, byteString, kiloByteString, megaByteString, gigaByteString)) | |
| var bytesPattern = regexp.MustCompile(`^(\d+(\.\d+)?)(B|[KMG]B|)$`) |
internal/docker/imagesgc.go
Outdated
| path string | ||
| images []gcEntry | ||
| clock func() time.Time | ||
| client imagesGCClient |
| // Persist saves the list of images to disk. | ||
| func (gc *ImagesGC) Persist() error { | ||
| if gc.path == "" { | ||
| return errors.New("GC list was not created with a path") |
There was a problem hiding this comment.
Seems to me that path != "" is an invariant where non-adherence is a programmer error, so this should panic.
| continue | ||
| } | ||
| if err != nil { | ||
| gc.images = append(images, gc.images[i:]...) |
There was a problem hiding this comment.
Why do we do this? I can't see that this value is used after an error. What am I missing?
There was a problem hiding this comment.
This is done in case gc is used after Run() fails, so gc.images is updated with images that may have been removed.
gc is not reused now, but the idea is that it could be.
Co-Authored-By: Dan Kortschak <[email protected]>
💚 Build Succeeded
History
cc @jsoriano |
Related to #856.
If one is using elastic-package during enough time, Docker storage can end up full of docker images. Once the 90% threshold is reached, behavior can be unpredictable as described in #856.
This change introduces garbage collection for docker images that were downloaded by elastic-package. Before pulling new images, it checks first if there are old unused images or if the images are taking too much space, and remove images accordingly. It only removes images known to be downloaded by elastic-package.
Implementation is based on two methods:
Trackis used to start tracking images, by recording their tags and last used times. It has to be called before pulling the images, it won't track images that are already pulled, to avoid GCing images not managed by elastic-package.Runruns the garbage collection, based on two parameters:max_unusedcan be used to remove images older than some time andmax_total_sizeto trigger removals when the total size of images (fromdocker system df) goes beyond some threshold. Less recently used images are removed first.Tracking of known images is persisted under
~/.elastic-package/cache/docker-images-gc.json.Garbage collection is disabled by default.
This can be configured from
~/.elastic-package/config.yml, for example: