Merge branch 'master' into filter-geometries

CHANGELOG.md

@@ -1,11 +1,19 @@
 Changelog
 =========
 
-## 0.8.0-SNAPSHOT (current master)
+## 1.0.0-SNAPSHOT (current master)
 
 ### breaking changes
 
 * remove class `oshdb-util:util.time.TimestampFormatter` ([#419])
+* drop "old" etl module ([#447])
+* remove Comparable interface from OSMEntity ([#449])
+* remove deprecated filter methods `osmType`, `osmTag` and `osmEntityFlter` (which were replaced by [OSHDB `filter`s](https://github.com/GIScience/oshdb/blob/0.7/documentation/manual/filters.md#oshdb-filter)) ([#451])
+* `OSMType.toString` returns lower-case name. ([#459])
+
+### new features
+
+* Allow to flexibly combine (automatic) aggregation methods (like `aggregateByGeometry(…)` or `aggregateByTimestamp()`) with each other and with `filter` or `map`/`flatMap`, regardless of the order of the applied operations ([#451])
 
 ### new features
 
@@ -14,13 +22,30 @@ Changelog
 ### bugfixes
 
 * change geometry filters to be based on full (unclipped) geometries ([#433])
+* make sure area computation never returns negative results (instead zero is returned for the invalid geometries which previously resulted in negative values) ([#438])
 
 ### other changes
 
- * update jts dependency to version 1.18.2
+* remove deprecated method `OSHEntity.getRawTagKeys` ([#441])
+* remove deprecated method `OSMEntity.getRawTags` ([#443])
+* remove `OSMMember.getRawRoleId` ([#453])
+* refactor `OSHDBRole` and move to oshdb-core ([#453])
+* update jts dependency to version 1.18.2
+* update ignite dependency to version 2.14.0 ([#459], [#467])
+* add natural order to `OSHDBTag` ([#454])
 
 [#419]: https://github.com/GIScience/oshdb/pull/419
 [#433]: https://github.com/GIScience/oshdb/issues/433
+[#438]: https://github.com/GIScience/oshdb/pull/438
+[#441]: https://github.com/GIScience/oshdb/pull/441
+[#443]: https://github.com/GIScience/oshdb/pull/443
+[#447]: https://github.com/GIScience/oshdb/pull/447
+[#449]: https://github.com/GIScience/oshdb/pull/449
+[#451]: https://github.com/GIScience/oshdb/pull/451
+[#453]: https://github.com/GIScience/oshdb/pull/453
+[#454]: https://github.com/GIScience/oshdb/pull/454
+[#459]: https://github.com/GIScience/oshdb/pull/459
+
 
 ## 0.7.2
 

Jenkinsfile

@@ -64,11 +64,21 @@ pipeline {
     stage ('Reports and Statistics') {
       steps {
         script {
-          // START CUSTOM oshdb
           withSonarQubeEnv('sonarcloud GIScience/ohsome') {
-            sh "mvn $MAVEN_GENERAL_OPTIONS sonar:sonar -Dsonar.branch.name=${env.BRANCH_NAME} -Dsonar.projectName=OSHDB"
+            // START CUSTOM oshdb
+            SONAR_CLI_PARAMETER = "-Dsonar.projectName=OSHDB"
+            // END CUSTOM oshdb
+            if (env.CHANGE_ID) {
+              SONAR_CLI_PARAMETER += " " +
+                "-Dsonar.pullrequest.key=${env.CHANGE_ID} " +
+                "-Dsonar.pullrequest.branch=${env.CHANGE_BRANCH} " +
+                "-Dsonar.pullrequest.base=${env.CHANGE_TARGET}"
+            } else {
+              SONAR_CLI_PARAMETER += " " +
+                "-Dsonar.branch.name=${env.BRANCH_NAME}"
+            }
+            sh "mvn $MAVEN_GENERAL_OPTIONS sonar:sonar ${SONAR_CLI_PARAMETER}"
           }
-          // END CUSTOM oshdb
           report_basedir = "/srv/reports/${REPO_NAME}/${VERSION}_${env.BRANCH_NAME}/${env.BUILD_NUMBER}_${LATEST_COMMIT_ID}"
 
           // jacoco

documentation/manual/README.md

@@ -16,6 +16,7 @@ Contents
 * [OSHDB API Manual](api.md) <br>
   Shows the different features of the OSHDB API and how they can be used to efficiently query the OSM history data.
   * [Data Views](views.md)
+  * [OSM Feature Geometries](geometries.md)
   * [Filtering of OSM data](filters.md)
   * [Map and Reduce](map-reduce.md)
   * [Data Aggregation](aggregation.md)

documentation/manual/api.md

@@ -6,6 +6,7 @@ The OSHDB API provides a library that can be used to write custom analysis queri
 The [first steps tutorial](../first-steps) explains the main components of the OSHDB API and how they can be used to perform a simple analysis query. On the following sub-pages, the different parts of the API are described in more detail.
 
 * [Data Views](views.md)
+* [OSM Feature Geometries](geometries.md)
 * [Filtering of OSM data](filters.md)
 * [Map and Reduce](map-reduce.md)
 * [Data Aggregation](aggregation.md)

documentation/manual/filters.md

@@ -15,9 +15,9 @@ The output of this filter will keep only OSM entities whose geometry lie within
 
 > For example, a large forest polygon in OSM that completely encompasses a small area of interest _is_ returned by the OSHDB API.
 
-The resulting geometries produced by the different OSHDB [views](views.md) are by default clipped to the specified area of interest. This makes it possible to directly calculate the length or area of linear or polygonal OSM features within the given query region, without having to consider the fact that some features might only partially lie within the region. It is, at the same time, still possible to access full extent of the respective OSM features' [unclipped](https://docs.ohsome.org/java/oshdb/0.7.2/aggregated/org/heigit/ohsome/oshdb/util/mappable/OSMEntitySnapshot.html#getGeometryUnclipped()) [geometries](https://docs.ohsome.org/java/oshdb/0.7.2/aggregated/org/heigit/ohsome/oshdb/util/mappable/OSMContribution.html#getGeometryUnclippedBefore()).
+The resulting geometries produced by the different OSHDB [views](views.md) are by default clipped to the specified area of interest. This makes it possible to directly calculate the length or area of linear or polygonal OSM features within the given query region, without having to consider the fact that some features might only partially lie within the region. It is, at the same time, still possible to access full extent of the respective OSM features' [unclipped](https://docs.ohsome.org/java/oshdb/0.7.2/aggregated/org/heigit/ohsome/oshdb/util/mappable/OSMEntitySnapshot.html#getGeometryUnclipped()) [geometries](https://docs.ohsome.org/java/oshdb/0.7.2/aggregated/org/heigit/ohsome/oshdb/util/mappable/OSMContribution.html#getGeometryUnclippedBefore()). You can find further information in the section about how the OSHDB [builds geometries](geometries.md) from OSM data.
 
-The OSHDB is able to cope well even with complex polygons that have many vertices, but keep in mind that using simpler geometries will generally result in higher query performance: For example a bounding-box query is executed slightly faster than a polygon-areaOfInterest query with a rectangular polygon. 
+The OSHDB is able to cope well even with complex polygons that have many vertices as areas of interest, but keep in mind that using simpler geometries will generally result in higher query performance: For example a bounding-box query is executed slightly faster than a polygon-areaOfInterest query with a rectangular polygon.
 
 <!-- todo: link to blog post with spatial filtering performance benchmarks -->
 

documentation/manual/geometries.md

@@ -0,0 +1,26 @@
+OSM Feature Geometries
+======================
+
+The [_views_](views.md) provided by the OSHDB API provide direct access to the OSM entities, but also allow to get the [JTS](https://en.wikipedia.org/wiki/JTS_Topology_Suite#Geometry_model) [geometries](https://locationtech.github.io/jts/javadoc/org/locationtech/jts/geom/Geometry.html) of the OSM features corresponding to their state at the requested _snapshot_ or _contribution_ time(s). With this geometry, further operations such as [_filtering_](filters.md#areaOfInterest) or [_aggregation_](aggregation.md#aggregateByGeometry) can be performed. It is also irreplaceable during the [_map-reduce_](map-reduce.md#geometry-helpers) step to generate useful results, such as the length of a road network for example.
+
+For some OSM elements, such as nodes, generating geometries is straight forward, for others the conversion requires further knowledge of the [data model](https://wiki.openstreetmap.org/wiki/Elements) and [tagging schema](https://wiki.openstreetmap.org/wiki/Tags) used by OSM. The following document gives an overview of how the OSHDB handles the building of geometries of different OSM object types.
+
+Nodes
+-----
+
+Nodes are always presented as [`Point`](https://locationtech.github.io/jts/javadoc/org/locationtech/jts/geom/Point.html)s. Note that nodes which have never had a tag and are a part of a way or relation are considered to be structural-only points (sometimes called _vertices_), and thus not returned in an OSHDB query when querying all nodes. This is because the OSHDB does not consider them to not be _map features_ by their own. They can however be fetched as members of their parent way or relation objects, if needed.
+
+Ways
+----
+
+Ways are converted to either [`LineString`](https://locationtech.github.io/jts/javadoc/org/locationtech/jts/geom/LineString.html) or [`Polygon`](https://locationtech.github.io/jts/javadoc/org/locationtech/jts/geom/Polygon.html) geometries depending on their composition and their tags: A not closed way is always represented as a line, while it depends for a closed one. The [`TagInterpreter`](https://docs.ohsome.org/java/oshdb/0.7.2/aggregated/org/heigit/ohsome/oshdb/util/taginterpreter/TagInterpreter.html) component of the OSHDB is responsible for deciding whether a closed way results in a line or a polygon: A (closed) OSM way with the tag `building=yes` will be converted to a polygon geometry, while a `junction=roundabout` one will not.
+
+Relations
+---------
+
+Relations are handled by the OSHDB in two different ways: [Multipolygons](https://wiki.openstreetmap.org/wiki/Multipolygon) are converted to either [`Polygon`](https://locationtech.github.io/jts/javadoc/org/locationtech/jts/geom/Polygon.html) or [`MultiPolygon`](https://locationtech.github.io/jts/javadoc/org/locationtech/jts/geom/MultiPolygon.html) geometries (depending on the number of outer rings), while all [other relation types](https://wiki.openstreetmap.org/wiki/Types_of_relation) result in a [`GeometryCollection`](https://locationtech.github.io/jts/javadoc/org/locationtech/jts/geom/GeometryCollection.html).
+
+Invalid OSM Data
+----------------
+
+There are situations where a part of OSM's entities have either incomplete or invalid data, for example a broken multipolygon resulting from a mapping error. The OSHDB makes the best effort to return a (potentially [invalid](https://locationtech.github.io/jts/javadoc/org/locationtech/jts/geom/Geometry.html#isValid--)) geometry also for these objects. For performance reasons the OSHDB cannot check for every possible error in the input (OSM) data, and for similar reasons it also cannot correct all errors it does find. This means that the OSHDB does not provide any guaranteed outcome for all possible errors and might return an invalid or valid geometry as a result in such cases or even no result at all. 

documentation/manual/views.md

@@ -1,7 +1,7 @@
 Views
 =====
 
-Two different ways of querying OSM data are available, which determine how the OSM history data is actually analyzed in a given OSHDB query: 
+Two different ways of querying OSM data are available, which determine how the OSM history data is actually analyzed in a given OSHDB query:
 
 * The **snapshot view** ([`OSMEntitySnapshotView`](https://docs.ohsome.org/java/oshdb/0.7.2/aggregated/org/heigit/ohsome/oshdb/api/mapreducer/OSMEntitySnapshotView.html)) returns the state of the OSM history data at specific given points in time.
 * The **contribution view** ([`OSMContributionView`](https://docs.ohsome.org/java/oshdb/0.7.2/aggregated/org/heigit/ohsome/oshdb/api/mapreducer/OSMContributionView.html)) returns all modifications (e.g., creations, modifications or deletions) to the OSM elements within a given time period.

oshdb-api-ignite/pom.xml

@@ -5,12 +5,16 @@
   <parent>
     <groupId>org.heigit.ohsome</groupId>
     <artifactId>oshdb-parent</artifactId>
-    <version>0.8.0-SNAPSHOT</version>
+    <version>1.0.0-SNAPSHOT</version>
   </parent>
 
   <artifactId>oshdb-api-ignite</artifactId>
   <name>OSHDB API – Ignite backend</name>
   <description>Backend implementation of the OSHDB API which runs queries on an Apache Ignite cluster.</description>
+
+  <properties>
+    <ignite.version>2.14.0</ignite.version>
+  </properties>
 
   <dependencies>
     <dependency>
@@ -45,4 +49,4 @@
       <version>${ignite.version}</version>
     </dependency>
   </dependencies>
-</project>
+</project>

oshdb-etl/src/main/java/org/heigit/ohsome/oshdb/tool/importer/util/OSHDBToIgnite.java → oshdb-api-ignite/src/main/java/org/heigit/ohsome/oshdb/api/ignite/util/OSHDBToIgnite.java

@@ -1,13 +1,9 @@
-package org.heigit.ohsome.oshdb.tool.importer.util;
+package org.heigit.ohsome.oshdb.api.ignite.util;
 
-import com.beust.jcommander.JCommander;
-import com.beust.jcommander.Parameter;
-import com.beust.jcommander.ParameterException;
 import java.io.File;
 import java.io.IOException;
 import java.io.ObjectInputStream;
 import java.sql.Connection;
-import java.sql.DriverManager;
 import java.sql.ResultSet;
 import java.sql.SQLException;
 import java.sql.Statement;
@@ -30,7 +26,13 @@
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
+/**
+ * Utility class for loading oshdb h2 file into ignite cluster.
+ *
+ */
 public class OSHDBToIgnite {
+  private OSHDBToIgnite() {}
+
 
   private static final Logger LOG = LoggerFactory.getLogger(OSHDBToIgnite.class);
 
@@ -122,43 +124,4 @@ private static <T> void doGridImport(Ignite ignite, Statement stmt, TableNames c
       }
     }
   }
-
-  private static class Config {
-    @Parameter(names = {"-ignite", "-igniteConfig", "-icfg"},
-        description = "Path ot ignite-config.xml", required = true, order = 1)
-    public File ignitexml;
-
-    @Parameter(names = {"--prefix"}, description = "cache table prefix", required = false)
-    public String prefix;
-
-    @Parameter(names = {"-db", "-oshdb", "-outputDb"}, description = "Path to output H2",
-        required = true, order = 2)
-    public File oshdb;
-
-    @Parameter(names = {"-help", "--help", "-h", "--h"}, help = true, order = 0)
-    public boolean help = false;
-  }
-
-  public static void main(String[] args) throws SQLException, IgniteCheckedException {
-    Config largs = new Config();
-    JCommander jcom = JCommander.newBuilder().addObject(largs).build();
-    try {
-      jcom.parse(args);
-    } catch (ParameterException e) {
-      System.out.println("");
-      LOG.error(e.getLocalizedMessage());
-      System.out.println("");
-      jcom.usage();
-
-      return;
-    }
-
-    if (largs.help) {
-      jcom.usage();
-      return;
-    }
-    try (Connection con = DriverManager.getConnection("jdbc:h2:" + largs.oshdb, "sa", null)) {
-      OSHDBToIgnite.load(largs.ignitexml, con, largs.prefix);
-    }
-  }
 }

oshdb-api-ignite/src/test/java/org/heigit/ohsome/oshdb/api/tests/TestMapReduceOSHDBIgnite.java

@@ -1,6 +1,6 @@
 package org.heigit.ohsome.oshdb.api.tests;
 
-import static org.junit.Assert.fail;
+import static org.junit.jupiter.api.Assertions.fail;
 
 import java.io.IOException;
 import java.io.ObjectInputStream;
@@ -46,13 +46,13 @@ abstract class TestMapReduceOSHDBIgnite extends TestMapReduce {
     ignite = Ignition.start(cfg);
   }
 
-  public TestMapReduceOSHDBIgnite(OSHDBIgnite oshdb) throws Exception {
+  TestMapReduceOSHDBIgnite(OSHDBIgnite oshdb) throws Exception {
     super(oshdb);
 
     final String prefix = "tests";
     oshdb.prefix(prefix);
 
-    OSHDBH2 oshdbH2 = new OSHDBH2("../oshdb-api/src/test/resources/test-data");
+    OSHDBH2 oshdbH2 = new OSHDBH2("../data/test-data");
     this.keytables = oshdbH2;
 
     Ignite ignite = ((OSHDBIgnite) this.oshdb).getIgnite();

oshdb-api-ignite/src/test/java/org/heigit/ohsome/oshdb/api/tests/TestMapReduceOSHDBIgniteAffinityCall.java

@@ -1,35 +1,35 @@
 package org.heigit.ohsome.oshdb.api.tests;
 
-import static org.junit.Assert.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
 
 import java.util.Set;
 import java.util.stream.Collectors;
 import org.heigit.ohsome.oshdb.api.db.OSHDBIgnite;
 import org.heigit.ohsome.oshdb.util.time.OSHDBTimestamps;
-import org.junit.Test;
+import org.junit.jupiter.api.Test;
 
 /**
  * {@inheritDoc}
  *
  * <p>Runs the tests using the "affinity call" Ignite backend.</p>
  */
-public class TestMapReduceOSHDBIgniteAffinityCall extends TestMapReduceOSHDBIgnite {
+class TestMapReduceOSHDBIgniteAffinityCall extends TestMapReduceOSHDBIgnite {
   /**
    * Creates the test runner using the ignite affinitycall backend.
    *
    * @throws Exception if something goes wrong
    */
-  public TestMapReduceOSHDBIgniteAffinityCall() throws Exception {
+  TestMapReduceOSHDBIgniteAffinityCall() throws Exception {
     super(new OSHDBIgnite(ignite).computeMode(OSHDBIgnite.ComputeMode.AFFINITY_CALL));
   }
 
   @Test
-  public void testOSMEntitySnapshotViewStreamNullValues() throws Exception {
+  void testOSMEntitySnapshotViewStreamNullValues() throws Exception {
     // simple stream query
     Set<Integer> result = createMapReducerOSMEntitySnapshot()
         .timestamps(
             new OSHDBTimestamps("2010-01-01", "2015-01-01", OSHDBTimestamps.Interval.YEARLY))
-        .osmEntityFilter(entity -> entity.getId() == 617308093)
+        .filter("id:617308093")
         .map(snapshot -> snapshot.getEntity().getUserId())
         .map(x -> (Integer) null)
         .stream()

oshdb-api-ignite/src/test/java/org/heigit/ohsome/oshdb/api/tests/TestMapReduceOSHDBIgniteLocalPeek.java

@@ -7,13 +7,13 @@
  *
  * <p>Runs the tests using the "local peek" Ignite backend.</p>
  */
-public class TestMapReduceOSHDBIgniteLocalPeek extends TestMapReduceOSHDBIgnite {
+class TestMapReduceOSHDBIgniteLocalPeek extends TestMapReduceOSHDBIgnite {
   /**
    * Creates the test runner using the ignite localpeak backend.
    *
    * @throws Exception if something goes wrong
    */
-  public TestMapReduceOSHDBIgniteLocalPeek() throws Exception {
+  TestMapReduceOSHDBIgniteLocalPeek() throws Exception {
     super(new OSHDBIgnite(ignite).computeMode(OSHDBIgnite.ComputeMode.LOCAL_PEEK));
   }
 }