Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Aim images python #1320

Merged
merged 70 commits into from
Jul 4, 2024
Merged
Show file tree
Hide file tree
Changes from 68 commits
Commits
Show all changes
70 commits
Select commit Hold shift + click to select a range
bac93a5
aim integration tests
May 28, 2023
5afecf4
removing ds_store files
May 28, 2023
f0ad7c1
Consistency between test scripts
jgiannuzzi Jun 6, 2023
a9fe353
Run aim integration tests in CI
jgiannuzzi Jun 6, 2023
0a39026
Merge main into fabiovincenzi/main
jgiannuzzi Jun 6, 2023
4c2b7af
Follow new integration test location convention
jgiannuzzi Jun 6, 2023
59517aa
Reformat patch to conform with PEP8 and install psycopg2-binary
jgiannuzzi Jun 6, 2023
4adbf22
run
fabiovincenzi Jul 31, 2023
c8395ad
test
fabiovincenzi Jul 31, 2023
463f8a6
wheels
fabiovincenzi Jul 31, 2023
7d57b66
Merge branch 'python-wheels' of github.com:fabiovincenzi/fasttrack in…
fabiovincenzi Jul 31, 2023
717c49b
add git hub action
fabiovincenzi Jul 31, 2023
f304c08
fix conflicts
fabiovincenzi Jul 31, 2023
0d2a235
Update ci.yml
fabiovincenzi Jul 31, 2023
f81d10f
Update buildwheels.yml
fabiovincenzi Jul 31, 2023
1268c4f
Update buildwheels.yml
fabiovincenzi Jul 31, 2023
232f75e
Update buildwheels.yml
fabiovincenzi Jul 31, 2023
c86bd51
Update buildwheels.yml
fabiovincenzi Jul 31, 2023
f90d66f
binaries
fabiovincenzi Jul 31, 2023
e72afaf
Merge branch 'python-wheels'
fabiovincenzi Jul 31, 2023
0d429a3
Merge branch 'main' of github.com:fabiovincenzi/fasttrack
fabiovincenzi Jul 31, 2023
26ece65
Update buildwheels.yml
fabiovincenzi Jul 31, 2023
bbc5119
add binaries
fabiovincenzi Jul 31, 2023
6b4f805
Merge branch 'main' of github.com:fabiovincenzi/fasttrack
fabiovincenzi Jul 31, 2023
fc147d3
Update buildwheels.yml
fabiovincenzi Jul 31, 2023
c2ae930
setup.py
fabiovincenzi Jul 31, 2023
cf2da54
Merge branch 'main' of github.com:fabiovincenzi/fasttrack
fabiovincenzi Jul 31, 2023
52aed39
setup.py
fabiovincenzi Jul 31, 2023
15ac2a6
setup.py
fabiovincenzi Jul 31, 2023
74bfa22
setup.py
fabiovincenzi Jul 31, 2023
7c4b5e5
Update buildwheels.yml
fabiovincenzi Aug 1, 2023
20723bc
Update buildwheels.yml
fabiovincenzi Aug 1, 2023
4135c43
Update buildwheels.yml
fabiovincenzi Aug 1, 2023
2696e28
Update buildwheels.yml
fabiovincenzi Aug 1, 2023
02cb7ae
Update buildwheels.yml
fabiovincenzi Aug 1, 2023
a5f7a67
Update ci.yml
fabiovincenzi Aug 1, 2023
7a0060f
Update ci.yml
fabiovincenzi Aug 2, 2023
3d48f6b
Update release.yml
fabiovincenzi Aug 2, 2023
aa42f29
Update ci.yml
fabiovincenzi Aug 2, 2023
d9e4e3e
Delete dist directory
fabiovincenzi Aug 2, 2023
868fb12
Update release.yml
fabiovincenzi Aug 2, 2023
3cac6c3
update to latest ui ref (#1181)
suprjinx May 3, 2024
3f3e3ef
Merge branch 'main' of github.com:suprjinx/fasttrackml
suprjinx May 3, 2024
e94d416
Merge branch 'main' of github.com:g-research/fasttrackml
suprjinx May 6, 2024
7d117cb
Merge branch 'main' of github.com:g-research/fasttrackml
suprjinx May 6, 2024
8f97bc9
Merge branch 'main' of github.com:g-research/fasttrackml
suprjinx May 7, 2024
a007c97
Merge branch 'main' of github.com:g-research/fasttrackml
suprjinx May 14, 2024
d23699f
Merge branch 'main' of github.com:g-research/fasttrackml
suprjinx May 14, 2024
a6a7882
Merge branch 'main' of github.com:g-research/fasttrackml
suprjinx May 20, 2024
9259e0a
Merge branch 'main' of github.com:g-research/fasttrackml
suprjinx May 21, 2024
16619bd
Merge branch 'main' of github.com:g-research/fasttrackml
suprjinx May 23, 2024
2ce575e
Merge branch 'main' of github.com:g-research/fasttrackml
suprjinx May 28, 2024
a64eec2
Merge branch 'main' of github.com:g-research/fasttrackml
suprjinx Jun 3, 2024
135be30
Merge branch 'main' of github.com:g-research/fasttrackml
suprjinx Jun 4, 2024
27c1c34
Merge branch 'main' of github.com:g-research/fasttrackml
suprjinx Jun 11, 2024
0069c4b
Merge branch 'main' of github.com:g-research/fasttrackml
suprjinx Jun 17, 2024
0aba795
Merge branch 'main' of github.com:g-research/fasttrackml
suprjinx Jun 18, 2024
278047d
Merge branch 'main' of github.com:g-research/fasttrackml
suprjinx Jun 21, 2024
7514ef4
Merge branch 'main' of github.com:g-research/fasttrackml
suprjinx Jun 25, 2024
21e1c6f
rough-in the FML client log_image
suprjinx Jun 27, 2024
a9e0cb2
Merge branch 'main' of github.com:g-research/fasttrackml
suprjinx Jul 1, 2024
5671b81
Merge branch 'heads/main' into aim-images-python
suprjinx Jul 1, 2024
eea52d0
Merge tag 'main' into aim-images-python
suprjinx Jul 1, 2024
de17f0b
Revert unwanted changes
suprjinx Jul 1, 2024
21c7038
Revert unwanted changes
suprjinx Jul 1, 2024
6aaa481
Update python client for log-artifact impl
suprjinx Jul 1, 2024
267d86b
Adjust to endpoint implementation, add elementary test
suprjinx Jul 2, 2024
c05374d
Fix lint
suprjinx Jul 2, 2024
4b15147
Fix path to local image for test
suprjinx Jul 3, 2024
49e7f53
Merge branch 'main' into aim-images-python
suprjinx Jul 3, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 1 addition & 2 deletions docs/example/minimal_fasttrackml.py
Original file line number Diff line number Diff line change
@@ -1,10 +1,9 @@
import time
from random import randint, random

from fasttrackml.entities.metric import Metric

import fasttrackml
from fasttrackml import FasttrackmlClient
from fasttrackml.entities.metric import Metric


def print_metric_info(history):
Expand Down
3 changes: 3 additions & 0 deletions pkg/api/mlflow/services/run/converters.go
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,8 @@ package run
import (
"fmt"

"github.com/google/uuid"

"github.com/G-Research/fasttrackml/pkg/api/mlflow/api/request"
"github.com/G-Research/fasttrackml/pkg/api/mlflow/dao/models"
)
Expand All @@ -29,6 +31,7 @@ func ConvertCreateRunArtifactRequestToModel(
namespaceID uint, req *request.LogArtifactRequest,
) *models.Artifact {
return &models.Artifact{
ID: uuid.New(),
Iter: req.Iter,
Step: req.Step,
RunID: req.RunID,
Expand Down
3 changes: 3 additions & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
[build-system]
requires = ["setuptools", "wheel"]
build-backend = "setuptools.build_meta"
3 changes: 1 addition & 2 deletions python/client_example.py
Original file line number Diff line number Diff line change
@@ -1,10 +1,9 @@
import time
from random import randint, random

from fasttrackml.entities.metric import Metric

import fasttrackml
from fasttrackml import FasttrackmlClient
from fasttrackml.entities.metric import Metric


def print_metric_info(history):
Expand Down
10 changes: 9 additions & 1 deletion python/client_test.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,9 @@
from random import random, uniform

import pytest
from fasttrackml.entities import Metric, Param

from fasttrackml import FasttrackmlClient
from fasttrackml.entities import Metric, Param

LOCALHOST = "127.0.0.1"

Expand Down Expand Up @@ -115,3 +115,11 @@ def test_init_output_logging(client, server, run):
for i in range(100):
log_data = str(uuid.uuid4()) + "\n" + str(uuid.uuid4())
print(log_data)


def test_log_image(client, server, run):
# test logging some images
for i in range(100):
assert (
client.log_image(run.info.run_id, "dice.png", "images", "These are dice", 0, 640, 480, "png", i, 0) == None
)
Binary file added python/dice.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
18 changes: 18 additions & 0 deletions python/fasttrackml/_tracking_service/client.py
Original file line number Diff line number Diff line change
Expand Up @@ -144,3 +144,21 @@ def log_output(
data: str,
):
self.custom_store.log_output(run_id, data)

def log_image(
self,
run_id: str,
filename: str,
artifact_path: str,
caption: str,
index: int,
width: int,
height: int,
format: str,
step: int,
iter: int,
):
# 1. log the artifact
self.log_artifact(run_id, filename, artifact_path)
# 2. log the image metadata
self.custom_store.log_image(run_id, filename, artifact_path, caption, index, width, height, format, step, iter)
53 changes: 53 additions & 0 deletions python/fasttrackml/client.py
Original file line number Diff line number Diff line change
Expand Up @@ -461,3 +461,56 @@ def log_output(
client.set_terminated(run.info.run_id)
"""
self._tracking_client.log_output(run_id, data)

def log_image(
self,
run_id: str,
filename: str,
artifact_path: str,
caption: str,
index: int,
width: int,
height: int,
format: str,
step: int,
iter: int,
) -> None:
"""
Log an image artifact for the provided run which will be viewable in the Images explorer.
The image itself will be stored in the configured artifact store (S3-compatible or local).

Args:
run_id: String ID of the run
filename: The filename of the image in the local filesystem
artifact_path: The optional path to append to the artifact_uri
caption: The image caption
index: The image index
width: The image width
height: The image height
format: The image format
step: The image step
iter: The image iteration

.. code-block:: python
:caption: Example

from fasttrackml import FasttrackmlClient

# Create a run under the default experiment (whose id is '0').
# Since these are low-level CRUD operations, this method will create a run.
# To end the run, you'll have to explicitly end it.
client = FasttrackmlClient()
experiment_id = "0"
run = client.create_run(experiment_id)
print_run_info(run)
print("--")

# Log an image
for step in range(10):
filename = generate_image(step) # some function that generates an image
client.log_image(run.info.run_id, filename, "This is an image", "images", step, 100, 100, "png", step, 0)
client.set_terminated(run.info.run_id)
"""
return self._tracking_client.log_image(
run_id, filename, artifact_path, caption, index, width, height, format, step, iter
)
44 changes: 44 additions & 0 deletions python/fasttrackml/store/custom_rest_store.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,6 @@
import json
import os
import posixpath
import threading
from typing import Dict, Optional, Sequence

Expand Down Expand Up @@ -274,3 +276,45 @@ def log_output(self, run_id, data):
error_code=result["error_code"],
)
return result

def log_image(
self,
run_id: str,
filename: str,
artifact_path: str,
caption: str,
index: int,
width: int,
height: int,
format: str,
step: int,
iter: int,
):
storage_path = posixpath.join(artifact_path, os.path.basename(filename))
request_body = {
"run_id": run_id,
"blob_uri": storage_path,
"caption": caption,
"index": index,
"width": width,
"height": height,
"format": format,
"step": step,
"iter": iter,
}
result = http_request(
**{
"host_creds": self.get_host_creds(),
"endpoint": "/api/2.0/mlflow/runs/log-artifact",
"method": "POST",
"json": request_body,
}
)
if result.status_code != 201:
result = result.json()
if "error_code" in result:
raise MlflowException(
message=result["message"],
error_code=result["error_code"],
)
return result
Loading