Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Checking pixellimit for r.import commands #491

Merged
merged 14 commits into from
Nov 21, 2023
Merged

Checking pixellimit for r.import commands #491

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
47d0e7f
started implementation for querying pixellimit for r.import commands
linakrisztian Oct 26, 2023
c404a83
Merge branch 'main' of github.com:actinia-org/actinia-core into rimop…
linakrisztian Nov 7, 2023
3debe71
continued implementation of pixel limit check for rimport commands
linakrisztian Nov 8, 2023
b1845d7
considered resampling/reprojection for raster size + cleanup of creat…
linakrisztian Nov 9, 2023
1a973b9
linting
linakrisztian Nov 9, 2023
f037c9c
markdown formating
linakrisztian Nov 9, 2023
cd5008f
add tests for pixellimit check
linakrisztian Nov 20, 2023
1d6c8ba
improved check in test + added fix for determining the estimated reso…
linakrisztian Nov 20, 2023
cb99eaa
Update tests/test_raster_import_pixellimit.py
linakrisztian Nov 20, 2023
ae4c2ce
Update tests/test_raster_import_pixellimit.py
linakrisztian Nov 20, 2023
4b0f7ed
Update tests/test_raster_import_pixellimit.py
linakrisztian Nov 21, 2023
4f81650
Update tests/test_raster_import_pixellimit.py
linakrisztian Nov 21, 2023
f638315
Update src/actinia_core/processing/actinia_processing/ephemeral_proce…
linakrisztian Nov 21, 2023
2b2fbd6
add test for importer
linakrisztian Nov 21, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
41 changes: 22 additions & 19 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,32 +23,35 @@ https://github.com/actinia-org/actinia-core/compare/4.3.1...main
released from main
...

## [4.11.0] - 2023-11-02
## [4.11.0](https://github.com/actinia-org/actinia-core/releases/tag/4.11.0) - 2023-11-02

released from main

### Added
* enhance docker installation docs by @mmacata in https://github.com/actinia-org/actinia-core/pull/470
* mdformat by @mmacata in https://github.com/actinia-org/actinia-core/pull/472
* Add pre-commit to renovate config by @mmacata in https://github.com/actinia-org/actinia-core/pull/473
* docs: add more details for user management by @neteler in https://github.com/actinia-org/actinia-core/pull/490
* Enable import via vsicurl by @mmacata in https://github.com/actinia-org/actinia-core/pull/482
* Allow separate config for worker Part 1 by @mmacata in https://github.com/actinia-org/actinia-core/pull/376

- enhance docker installation docs by @mmacata in https://github.com/actinia-org/actinia-core/pull/470
- mdformat by @mmacata in https://github.com/actinia-org/actinia-core/pull/472
- Add pre-commit to renovate config by @mmacata in https://github.com/actinia-org/actinia-core/pull/473
- docs: add more details for user management by @neteler in https://github.com/actinia-org/actinia-core/pull/490
- Enable import via vsicurl by @mmacata in https://github.com/actinia-org/actinia-core/pull/482
- Allow separate config for worker Part 1 by @mmacata in https://github.com/actinia-org/actinia-core/pull/376

### Fixed
* mundialis->actinia-org by @mmacata in https://github.com/actinia-org/actinia-core/pull/471
* Update actions/checkout action to v4 by @renovate in https://github.com/actinia-org/actinia-core/pull/474
* Update docker/login-action action to v3 by @renovate in https://github.com/actinia-org/actinia-core/pull/477
* Update docker/build-push-action action to v5 by @renovate in https://github.com/actinia-org/actinia-core/pull/476
* Update docker/setup-buildx-action action to v3 by @renovate in https://github.com/actinia-org/actinia-core/pull/479
* Update docker/setup-qemu-action action to v3 by @renovate in https://github.com/actinia-org/actinia-core/pull/480
* Update docker/metadata-action action to v5 by @renovate in https://github.com/actinia-org/actinia-core/pull/478
* Update update-version.yml by @mmacata in https://github.com/actinia-org/actinia-core/pull/475
* Update pyproj in requirements for ubuntu by @mmacata in https://github.com/actinia-org/actinia-core/pull/484
* chore(deps): update dependency werkzeug and Flask to v3 [security] by @renovate in https://github.com/actinia-org/actinia-core/pull/489

**Full Changelog**: https://github.com/actinia-org/actinia-core/compare/4.10.0...4.11.0
- mundialis->actinia-org by @mmacata in https://github.com/actinia-org/actinia-core/pull/471
- Update actions/checkout action to v4 by @renovate in https://github.com/actinia-org/actinia-core/pull/474
- Update docker/login-action action to v3 by @renovate in https://github.com/actinia-org/actinia-core/pull/477
- Update docker/build-push-action action to v5 by @renovate in https://github.com/actinia-org/actinia-core/pull/476
- Update docker/setup-buildx-action action to v3 by @renovate in https://github.com/actinia-org/actinia-core/pull/479
- Update docker/setup-qemu-action action to v3 by @renovate in https://github.com/actinia-org/actinia-core/pull/480
- Update docker/metadata-action action to v5 by @renovate in https://github.com/actinia-org/actinia-core/pull/478
- Update update-version.yml by @mmacata in https://github.com/actinia-org/actinia-core/pull/475
- Update pyproj in requirements for ubuntu by @mmacata in https://github.com/actinia-org/actinia-core/pull/484
- chore(deps): update dependency werkzeug and Flask to v3 \[security\] by @renovate in https://github.com/actinia-org/actinia-core/pull/489

generated with `gh api repos/actinia-org/actinia-core/releases/generate-notes -f tag_name=4.11.0 -f target_commitish=main -q .body`
**Full Changelog**: https://github.com/actinia-org/actinia-core/compare/4.10.0...4.11.0

generated with `gh api repos/actinia-org/actinia-core/releases/generate-notes -f tag_name=4.11.0 -f target_commitish=main -q .body`

## [4.10.0](https://github.com/actinia-org/actinia-core/releases/tag/4.10.0) - 2023-08-31

Expand Down
1 change: 1 addition & 0 deletions docs/docs/index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
# Actinia - The GRASS GIS REST API

<!-- **** Begin Fork-Me-On-Gitlab-Ribbon-HTML. See MIT License at https://gitlab.com/seanwasere/fork-me-on-gitlab **** -->

<a href="https://github.com/actinia-org/actinia-core/tree/main/docs/docs">
<span id="fork-me" style="font-family: tahoma; font-size: 18px; position:fixed; top:50px; right:-45px; display:block; -webkit-transform: rotate(45deg); -moz-transform: rotate(45deg); color:white; padding: 4px 30px 4px 30px; z-index:99; opacity:0.6">Fork Me On GitHub</span>
</a>
Expand Down
157 changes: 155 additions & 2 deletions src/actinia_core/processing/actinia_processing/ephemeral_processing.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -192,6 +192,7 @@ def __init__(self, rdc):
self.temp_mapset_path = None

self.ginit = None
self.ginit_tmpfiles = list()

# Successfully finished message
self.finish_message = "Processing successfully finished"
Expand Down Expand Up @@ -1203,6 +1204,154 @@ def _cleanup(self):
and os.path.isdir(self.temp_grass_data_base)
):
shutil.rmtree(self.temp_grass_data_base, ignore_errors=True)
if self.ginit_tmpfiles:
for tmpfile in self.ginit_tmpfiles:
try:
os.remove(tmpfile)
except Exception as e:
self.message_logger.debug(
f"Temporary file {tmpfile} can't be removed: {e}"
)

def _check_pixellimit_rimport(self, process_executable_params):
"""Check the current r.import command against the user cell limit.

Raises:
This method will raise an AsyncProcessError exception

"""
rimport_inp = [x for x in process_executable_params if "input=" in x][
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At this point, an importer command would already be translated to r.import, right? So the following would be parsed correctly because it is already transformed?:

 {
            "id": "importer_1",
            "module": "importer",
            "inputs": [
                {
                    "import_descr": {
                        "source": "https://raw.githubusercontent.com/mmacata/pagestest/gh-pages/bonn.geojson",
                        "type": "vector"
                    },
                    "param": "map",
                    "value": "polygon"
                }
            ]
        },

(Maybe add an importer PC to the tests below?) 🙃

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes it is already transformed at this point. I also added a test for the importer

0
].split("=")[1]
rimport_out = [x for x in process_executable_params if "output=" in x][
0
].split("=")[1]
vrt_out = f"{rimport_out}_{os.getpid()}_tmp.vrt"
self.ginit_tmpfiles.append(vrt_out)

# define extent_region if set (otherwise empty list)
extent_region = [
x for x in process_executable_params if "extent=" in x
]

# build VRT of rimport input
gdabuildvrt_params = list()
# if extent=region set, vrt only for region, not complete input
if extent_region:
# first query region extents
errorid, stdout_gregion, stderr_gregion = self.ginit.run_module(
"g.region", ["-ug"]
)
if errorid != 0:
raise AsyncProcessError(
"Unable to check the computational region size"
)
# parse region extents for creation of vrt (-te flag from gdalbuildvrt)
list_out_gregion = stdout_gregion.split("\n")
gdabuildvrt_params.append("-te")
gdabuildvrt_params.append(list_out_gregion[4]) # xmin/w
gdabuildvrt_params.append(list_out_gregion[3]) # ymin/s
gdabuildvrt_params.append(list_out_gregion[5]) # xmax/e
gdabuildvrt_params.append(list_out_gregion[2]) # ymax/n
# out and input for gdalbuildvrt
gdabuildvrt_params.append(vrt_out)
gdabuildvrt_params.append(rimport_inp)
# build vrt with previous defined parameters
(
errorid,
stdout_gdalbuildvrt,
stderr_gdalbuildvrt,
) = self.ginit.run_module("/usr/bin/gdalbuildvrt", gdabuildvrt_params)

# gdalinfo for created vrt
gdalinfo_params = [vrt_out]
errorid, stdout_gdalinfo, stderr_gdalinfo = self.ginit.run_module(
"/usr/bin/gdalinfo", gdalinfo_params
)
# parse "Size" output of gdalinfo
rastersize_list = (
stdout_gdalinfo.split("Size is")[1].split("\n")[0].split(",")
)
# size = x-dim*y-dim
rastersize_x = int(rastersize_list[0])
rastersize_y = int(rastersize_list[1])
rastersize = rastersize_x * rastersize_y

# if different import/reprojection resolution set:
rimport_res = [
x for x in process_executable_params if "resolution=" in x
]
res_val = None
# If raster exceeds cell limit already in original resolution, next part can be skipped
if rimport_res and (rastersize < self.cell_limit):
# determine estimated resolution
errorid, stdout_estres, stderr_estres = self.ginit.run_module(
"r.import", [vrt_out, "-e"]
)
if "Estimated" in stderr_estres:
# if data in different projection get rest_est with output of r.import -e
res_est = float(stderr_estres.split("\n")[-2].split(":")[1])
else:
# if data in same projection can use gdalinfo output
res_xy = (
stdout_gdalinfo.split("Pixel Size = (")[1]
.split(")\n")[0]
.split(",")
)
# get estimated resolution
# (analoug as done within r.import -e: estres = math.sqrt((n - s) * (e - w) / cells))
res_est = math.sqrt(abs(float(res_xy[0]) * float(res_xy[1])))
# determine set resolution value
resolution = rimport_res[0].split("=")[1]
if resolution == "value":
res_val = [
float(
[
x
for x in process_executable_params
if "resolution_value=" in x
][0].split("=")[1]
)
] * 2
elif resolution == "region":
# if already queried above reuse, otherwise execute g.region command
try:
stdout_gregion
except Exception:
(
errorid,
stdout_gregion,
stderr_gregion,
) = self.ginit.run_module("g.region", ["-ug"])
res_val_ns = float(
[x for x in stdout_gregion.split("\n") if "nsres=" in x][
0
].split("=")[1]
)
res_val_ew = float(
[x for x in stdout_gregion.split("\n") if "ewres=" in x][
0
].split("=")[1]
)
res_val = [res_val_ns, res_val_ew]
if res_val:
if (res_val[0] < res_est) | (res_val[1] < res_est):
# only check if smaller resolution set
res_change_x = res_est / res_val[1]
res_change_y = res_est / res_val[0]
# approximate raster size after resampling
# by using factor of changed resolution
rastersize = (
rastersize_x * res_change_x * rastersize_y * res_change_y
)

# compare estimated raster output size with pixel limit
# and raise exception if exceeded
if rastersize > self.cell_limit:
raise AsyncProcessError(
"Processing pixel limit exceeded for raster import. "
"Please set e.g. region smaller."
)

def _check_reset_region(self):
"""Check the current region settings against the user cell limit.
Expand Down Expand Up @@ -1479,10 +1628,14 @@ def _run_module(self, process, poll_time=0.05):
)
self._send_resource_update(message)

# Check pixel limit for r.import operations
if process.executable == "r.import":
self._check_pixellimit_rimport(process.executable_params)

# Check reset region if a g.region call was present in the process
# chain. By default the initial value of last_module is "g.region" to
# assure for first run of a process from the process chain, the
# region settings are evaluated
# assure for first run of a process from the process chain, the region
# settings are evaluated
if (
self.last_module == "g.region"
and process.skip_permission_check is False
Expand Down
4 changes: 2 additions & 2 deletions tests/test_download_cache.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -136,7 +136,7 @@ def test_download_cache(self):
# "HTML status code is wrong %i" % rv.status_code,
# )
# self.assertEqual(
# rv.mimetype, "application/json", "Wrong mimetype %s" % rv.mimetype
# rv.mimetype, "application/json", "Wrong mimetype %s" % rv.mimetype
# )

# TODO: configure wrong download cache path differently (via config file)
Expand All @@ -158,7 +158,7 @@ def test_download_cache(self):
# "HTML status code is wrong %i" % rv.status_code,
# )
# self.assertEqual(
# rv.mimetype, "application/json", "Wrong mimetype %s" % rv.mimetype
# rv.mimetype, "application/json", "Wrong mimetype %s" % rv.mimetype
# )


Expand Down
Loading