Update MATSim CW27 (#105)

* Score explanation functionality * rename a method from add... to set... since that is what is does. * changes * update config group * update tests and added attributes * fix un initialized attribute * small improvement to object creation * add leg score for individual legs * more consistent naming * more consistent naming * revert unintentional changes * CreateDrtDashboard * Bump org.apache.maven.plugins:maven-failsafe-plugin from 3.2.5 to 3.3.0 Bumps [org.apache.maven.plugins:maven-failsafe-plugin](https://github.com/apache/maven-surefire) from 3.2.5 to 3.3.0. - [Release notes](https://github.com/apache/maven-surefire/releases) - [Commits](apache/maven-surefire@surefire-3.2.5...surefire-3.3.0) --- updated-dependencies: - dependency-name: org.apache.maven.plugins:maven-failsafe-plugin dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> * rename pooling ratio to occupancy rate * account for group bookings in DrtAnalysisPostProcessing (and thus DrtDashboard) * typo in CreateDrtDashboard * fix wrong sum calculation * debug NoiseAnalysis and proper testing in NoiseDashboardTests * fix wrong map selection * add reduction of numberOfJobs * add comments and format log massages * combine * rename parameter * reduce code * format logging * use method for demand calculating * use method for shipment creation * remove todo * use method to calculate demand based on link lengths * introduce parameter * simplify * cleanup * Support TransitRouteStops where boarding or alighting is not allowed In reality, there are trains that call at stops but where it might not be allowed to enter or exit such trains. Examples are night trains, where on the first few stops, only boarding is allowed, while at the last few stops only alighting is allowed. This means that it is not allowed to use such night trains to just travel between the first few stops. This commit adds two optional attributes `allowBoarding` and `allowAlighting` to TransitRouteStops, with the default value being true for backwards compatibility and when the attribute is not specified. SwissRailRaptor respects these attributes and will not return routes where agents would enter a vehicle at a route stop where boarding is not allowed, or where agents would exit a vehicle at a route stop where alighting is not allowed. * make drt shifts multi mode drt comatible * adjust shift efficiency tracker as well * Extended Trip analysis and dashboard (matsim-org#3332) * trip analysis for subgroups * refactor categories * write trip analysis * write out correct table * use static method * prepare for facets, add TODOs * start working on plotly facet integration * Added Test Trip Dashboard for Facets * update API * add trip choice analysis * adding test case and some todos * calculating some trip choice metrics * add choice metrics to trip dashboard * hide tab if not needed * add text box and round metrics * normalize shares per reference group correctly, added some facet dashboards (WIP) * fix pivot usage * rename entries * add plots for distance distributions * add todo * use dropdown for plots * add cohen kappa to evaluation * output euclidean distance for trip choices * add ref_id attribute * added heatmap to TripDashboard * update description * allow arbitrary object for ref ids * show heatmap labels * comment out trip dashboard wip part * use correct logger * improve wording --------- Co-authored-by: frievoe97 <[email protected]> * Bump org.apache.maven.plugins:maven-surefire-plugin from 3.2.5 to 3.3.0 Bumps [org.apache.maven.plugins:maven-surefire-plugin](https://github.com/apache/maven-surefire) from 3.2.5 to 3.3.0. - [Release notes](https://github.com/apache/maven-surefire/releases) - [Commits](apache/maven-surefire@surefire-3.2.5...surefire-3.3.0) --- updated-dependencies: - dependency-name: org.apache.maven.plugins:maven-surefire-plugin dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> * Bump org.apache.maven.plugins:maven-surefire-report-plugin Bumps [org.apache.maven.plugins:maven-surefire-report-plugin](https://github.com/apache/maven-surefire) from 3.2.5 to 3.3.0. - [Release notes](https://github.com/apache/maven-surefire/releases) - [Commits](apache/maven-surefire@surefire-3.2.5...surefire-3.3.0) --- updated-dependencies: - dependency-name: org.apache.maven.plugins:maven-surefire-report-plugin dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> * Bump org.apache.maven.plugins:maven-jar-plugin from 3.4.1 to 3.4.2 Bumps [org.apache.maven.plugins:maven-jar-plugin](https://github.com/apache/maven-jar-plugin) from 3.4.1 to 3.4.2. - [Release notes](https://github.com/apache/maven-jar-plugin/releases) - [Commits](apache/maven-jar-plugin@maven-jar-plugin-3.4.1...maven-jar-plugin-3.4.2) --- updated-dependencies: - dependency-name: org.apache.maven.plugins:maven-jar-plugin dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> * Bump com.fasterxml.woodstox:woodstox-core from 6.6.2 to 7.0.0 Bumps [com.fasterxml.woodstox:woodstox-core](https://github.com/FasterXML/woodstox) from 6.6.2 to 7.0.0. - [Commits](FasterXML/woodstox@woodstox-core-6.6.2...woodstox-core-7.0.0) --- updated-dependencies: - dependency-name: com.fasterxml.woodstox:woodstox-core dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]> * Bump org.jfree:jfreechart from 1.5.4 to 1.5.5 Bumps [org.jfree:jfreechart](https://github.com/jfree/jfreechart) from 1.5.4 to 1.5.5. - [Release notes](https://github.com/jfree/jfreechart/releases) - [Commits](jfree/jfreechart@v1.5.4...v1.5.5) --- updated-dependencies: - dependency-name: org.jfree:jfreechart dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> * make drt work with only custom constraints sets * add some logging to NetworkChangeEventsWriter * correct logging statement * add router binding for complex request unscheduling * use time as disutility * Bump junit.version from 5.10.2 to 5.10.3 Bumps `junit.version` from 5.10.2 to 5.10.3. Updates `org.junit.jupiter:junit-jupiter-engine` from 5.10.2 to 5.10.3 - [Release notes](https://github.com/junit-team/junit5/releases) - [Commits](junit-team/junit5@r5.10.2...r5.10.3) Updates `org.junit.jupiter:junit-jupiter` from 5.10.2 to 5.10.3 - [Release notes](https://github.com/junit-team/junit5/releases) - [Commits](junit-team/junit5@r5.10.2...r5.10.3) --- updated-dependencies: - dependency-name: org.junit.jupiter:junit-jupiter-engine dependency-type: direct:development update-type: version-update:semver-patch - dependency-name: org.junit.jupiter:junit-jupiter dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> * norm subgroups to 1 (matsim-org#3346) * add test integration test case for complex unscheduler * Bump io.grpc:grpc-all from 1.64.0 to 1.65.0 Bumps [io.grpc:grpc-all](https://github.com/grpc/grpc-java) from 1.64.0 to 1.65.0. - [Release notes](https://github.com/grpc/grpc-java/releases) - [Commits](grpc/grpc-java@v1.64.0...v1.65.0) --- updated-dependencies: - dependency-name: io.grpc:grpc-all dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> * Bump com.google.protobuf:protobuf-java from 4.27.1 to 4.27.2 Bumps [com.google.protobuf:protobuf-java](https://github.com/protocolbuffers/protobuf) from 4.27.1 to 4.27.2. - [Release notes](https://github.com/protocolbuffers/protobuf/releases) - [Changelog](https://github.com/protocolbuffers/protobuf/blob/main/protobuf_release.bzl) - [Commits](https://github.com/protocolbuffers/protobuf/commits) --- updated-dependencies: - dependency-name: com.google.protobuf:protobuf-java dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: rakow <[email protected]> Co-authored-by: Kai Nagel <[email protected]> Co-authored-by: kainagel <[email protected]> Co-authored-by: rakow <[email protected]> Co-authored-by: schlenther <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ricardo Ewert <[email protected]> Co-authored-by: Ricardo Ewert <[email protected]> Co-authored-by: Marcel Rieser <[email protected]> Co-authored-by: frievoe97 <[email protected]> Co-authored-by: Joschka Bischoff <[email protected]> Co-authored-by: Michal Maciejewski <[email protected]>
moia-oss · Jul 4, 2024 · 1627fbc · 1627fbc
1 parent c920b08
commit 1627fbc
Show file tree

Hide file tree

Showing 70 changed files with 3,125 additions and 456 deletions.
diff --git a/contribs/application/src/main/java/org/matsim/application/CommandRunner.java b/contribs/application/src/main/java/org/matsim/application/CommandRunner.java
@@ -108,7 +108,7 @@ private void runCommand(Class<? extends MATSimAppCommand> clazz, Path input) thr
 		MATSimAppCommand command = clazz.getDeclaredConstructor().newInstance();
 		String[] args = this.args.get(clazz);
 		args = ArrayUtils.addAll(args, createArgs(clazz, args, input));
-		log.info("Running {} with arguments: {}", clazz, Arrays.toString(args));
+		log.info("Running {} with arguments: {}", clazz, String.join(" ", args));
 
 		command.execute(args);
 	}

diff --git a/contribs/application/src/main/java/org/matsim/application/analysis/population/Category.java b/contribs/application/src/main/java/org/matsim/application/analysis/population/Category.java
@@ -0,0 +1,199 @@
+package org.matsim.application.analysis.population;
+
+import org.matsim.core.config.ReflectiveConfigGroup;
+import org.matsim.utils.objectattributes.attributable.Attributes;
+
+import java.util.*;
+import java.util.regex.Pattern;
+
+/**
+ * Helper class to categorize values into groups.
+ */
+public final class Category {
+
+	private static final Set<String> TRUE = Set.of("true", "yes", "1", "on", "y", "j", "ja");
+	private static final Set<String> FALSE = Set.of("false", "no", "0", "off", "n", "nein");
+
+	/**
+	 * Unique values of the category.
+	 */
+	private final Set<String> values;
+
+	/**
+	 * Groups of values that have been subsumed under a single category.
+	 * These are values separated by ,
+	 */
+	private final Map<String, String> grouped;
+
+
+	/**
+	 * Regular expressions for each category.
+	 */
+	private final Map<String, Pattern> regex;
+
+	/**
+	 * Range categories.
+	 */
+	private final List<Range> ranges;
+
+	public Category(Set<String> values) {
+		this.values = values;
+		this.grouped = new HashMap<>();
+		this.regex = new HashMap<>();
+		for (String v : values) {
+			if (v.contains(",")) {
+				String[] grouped = v.split(",");
+				for (String g : grouped) {
+					this.grouped.put(g, v);
+				}
+			}
+
+			if (v.startsWith("/") && v.endsWith("/")) {
+				this.regex.put(v, Pattern.compile(v.substring(1, v.length() - 1), Pattern.CASE_INSENSITIVE));
+			}
+		}
+
+		boolean range = this.values.stream().allMatch(v -> v.contains("-") || v.contains("+"));
+		if (range) {
+			ranges = new ArrayList<>();
+			for (String value : this.values) {
+				if (value.contains("-")) {
+					String[] parts = value.split("-");
+					ranges.add(new Range(Double.parseDouble(parts[0]), Double.parseDouble(parts[1]), value));
+				} else if (value.contains("+")) {
+					ranges.add(new Range(Double.parseDouble(value.replace("+", "")), Double.POSITIVE_INFINITY, value));
+				}
+			}
+
+			ranges.sort(Comparator.comparingDouble(r -> r.left));
+		} else
+			ranges = null;
+
+
+		// Check if all values are boolean
+		if (values.stream().allMatch(v -> TRUE.contains(v.toLowerCase()) || FALSE.contains(v.toLowerCase()))) {
+			for (String value : values) {
+				Set<String> group = TRUE.contains(value.toLowerCase()) ? TRUE : FALSE;
+				for (String g : group) {
+					this.grouped.put(g, value);
+				}
+			}
+		}
+	}
+
+	/**
+	 * Create categories from config parameters.
+	 */
+	public static Map<String, Category> fromConfigParams(Collection<? extends ReflectiveConfigGroup> params) {
+
+		Map<String, Set<String>> categories = new HashMap<>();
+
+		// Collect all values
+		for (ReflectiveConfigGroup parameter : params) {
+			for (Map.Entry<String, String> kv : parameter.getParams().entrySet()) {
+				categories.computeIfAbsent(kv.getKey(), k -> new HashSet<>()).add(kv.getValue());
+			}
+		}
+
+		return categories.entrySet().stream()
+			.collect(HashMap::new, (m, e) -> m.put(e.getKey(), new Category(e.getValue())), HashMap::putAll);
+	}
+
+	/**
+	 * Match attributes from an object with parameters defined in config.
+	 */
+	public static boolean matchAttributesWithConfig(Attributes attr, ReflectiveConfigGroup config, Map<String, Category> categories) {
+
+		for (Map.Entry<String, String> e : config.getParams().entrySet()) {
+			// might be null if not defined
+			Object objValue = attr.getAttribute(e.getKey());
+			String category = categories.get(e.getKey()).categorize(objValue);
+
+			// compare as string
+			if (!Objects.toString(category).equals(e.getValue()))
+				return false;
+		}
+
+		return true;
+	}
+
+	/**
+	 * Categorize a single value.
+	 */
+	public String categorize(Object value) {
+
+		if (value == null)
+			return null;
+
+		if (value instanceof Boolean) {
+			// Booleans and synonyms are in the group map
+			return categorize(((Boolean) value).toString().toLowerCase());
+		} else if (value instanceof Number) {
+			return categorizeNumber((Number) value);
+		} else {
+			String v = value.toString();
+			if (values.contains(v))
+				return v;
+			else if (grouped.containsKey(v))
+				return grouped.get(v);
+			else {
+				for (Map.Entry<String, Pattern> kv : regex.entrySet()) {
+					if (kv.getValue().matcher(v).matches())
+						return kv.getKey();
+				}
+			}
+
+			try {
+				double d = Double.parseDouble(v);
+				return categorizeNumber(d);
+			} catch (NumberFormatException e) {
+				return null;
+			}
+		}
+	}
+
+	private String categorizeNumber(Number value) {
+
+		if (ranges != null) {
+			for (Range r : ranges) {
+				if (value.doubleValue() >= r.left && value.doubleValue() < r.right)
+					return r.label;
+			}
+		}
+
+		// Match string representation
+		String v = value.toString();
+		if (values.contains(v))
+			return v;
+		else if (grouped.containsKey(v))
+			return grouped.get(v);
+
+
+		// Convert the number to a whole number, which will have a different string representation
+		if (value instanceof Float || value instanceof Double) {
+			return categorizeNumber(value.longValue());
+		}
+
+		return null;
+	}
+
+	@Override
+	public String toString() {
+		return "Category{" +
+			"values=" + values +
+			(grouped != null && !grouped.isEmpty() ? ", grouped=" + grouped : "") +
+			(regex != null && !regex.isEmpty() ? ", regex=" + regex : "") +
+			(ranges != null && !ranges.isEmpty() ? ", ranges=" + ranges : "") +
+			'}';
+	}
+
+	/**
+	 * Number range.
+	 *
+	 * @param left  Left bound of the range.
+	 * @param right Right bound of the range. (exclusive)
+	 * @param label Label of this group.
+	 */
+	private record Range(double left, double right, String label) {
+	}
+}
diff --git a/...bs/application/src/main/java/org/matsim/application/analysis/population/TripAnalysis.java b/...bs/application/src/main/java/org/matsim/application/analysis/population/TripAnalysis.java
@@ -27,29 +27,49 @@
 import tech.tablesaw.joining.DataFrameJoiner;
 import tech.tablesaw.selection.Selection;
 
-import java.io.*;
+import java.io.IOException;
 import java.math.BigDecimal;
 import java.math.RoundingMode;
 import java.nio.file.Files;
 import java.util.*;
-import java.util.zip.GZIPInputStream;
 
 import static tech.tablesaw.aggregate.AggregateFunctions.count;
 
 @CommandLine.Command(name = "trips", description = "Calculates various trip related metrics.")
 @CommandSpec(
 	requires = {"trips.csv", "persons.csv"},
-	produces = {"mode_share.csv", "mode_share_per_dist.csv", "mode_users.csv", "trip_stats.csv", "population_trip_stats.csv", "trip_purposes_by_hour.csv"}
+	produces = {
+		"mode_share.csv", "mode_share_per_dist.csv", "mode_users.csv", "trip_stats.csv",
+		"mode_share_per_%s.csv", "population_trip_stats.csv", "trip_purposes_by_hour.csv",
+		"mode_choices.csv", "mode_choice_evaluation.csv", "mode_choice_evaluation_per_mode.csv",
+		"mode_confusion_matrix.csv", "mode_prediction_error.csv"
+	}
 )
 public class TripAnalysis implements MATSimAppCommand {
 
 	private static final Logger log = LogManager.getLogger(TripAnalysis.class);
 
+	/**
+	 * Attributes which relates this person to a reference person.
+	 */
+	public static String ATTR_REF_ID = "ref_id";
+	/**
+	 * Person attribute that contains the reference modes of a person. Multiple modes are delimited by "-".
+	 */
+	public static String ATTR_REF_MODES = "ref_modes";
+	/**
+	 * Person attribute containing its weight for analysis purposes.
+	 */
+	public static String ATTR_REF_WEIGHT = "ref_weight";
+
 	@CommandLine.Mixin
 	private InputOptions input = InputOptions.ofCommand(TripAnalysis.class);
 	@CommandLine.Mixin
 	private OutputOptions output = OutputOptions.ofCommand(TripAnalysis.class);
 
+	@CommandLine.Option(names = "--input-ref-data", description = "Optional path to reference data", required = false)
+	private String refData;
+
 	@CommandLine.Option(names = "--match-id", description = "Pattern to filter agents by id")
 	private String matchId;
 
@@ -95,7 +115,7 @@ public Integer call() throws Exception {
 		Table persons = Table.read().csv(CsvReadOptions.builder(IOUtils.getBufferedReader(input.getPath("persons.csv")))
 			.columnTypesPartial(Map.of("person", ColumnType.TEXT))
 			.sample(false)
-			.separator(new CsvOptions().detectDelimiter(input.getPath("persons.csv"))).build());
+			.separator(CsvOptions.detectDelimiter(input.getPath("persons.csv"))).build());
 
 		int total = persons.rowCount();
 
@@ -132,6 +152,7 @@ public Integer call() throws Exception {
 
 		// Map.of only has 10 argument max
 		columnTypes.put("traveled_distance", ColumnType.LONG);
+		columnTypes.put("euclidean_distance", ColumnType.LONG);
 
 		Table trips = Table.read().csv(CsvReadOptions.builder(IOUtils.getBufferedReader(input.getPath("trips.csv")))
 			.columnTypesPartial(columnTypes)
@@ -172,6 +193,12 @@ public Integer call() throws Exception {
 			trips = trips.where(Selection.with(idx.toIntArray()));
 		}
 
+		TripByGroupAnalysis groups = null;
+		if (refData != null) {
+			groups = new TripByGroupAnalysis(refData);
+			groups.groupPersons(persons);
+		}
+
 		// Use longest_distance_mode where main_mode is not present
 		trips.stringColumn("main_mode")
 			.set(trips.stringColumn("main_mode").isMissing(),
@@ -196,6 +223,24 @@ public Integer call() throws Exception {
 
 		writeModeShare(joined, labels);
 
+		if (groups != null) {
+			groups.analyzeModeShare(joined, labels, modeOrder, (g) -> output.getPath("mode_share_per_%s.csv", g));
+		}
+
+		if (persons.containsColumn(ATTR_REF_MODES)) {
+			try {
+				TripChoiceAnalysis choices = new TripChoiceAnalysis(persons, trips, modeOrder);
+
+				choices.writeChoices(output.getPath("mode_choices.csv"));
+				choices.writeChoiceEvaluation(output.getPath("mode_choice_evaluation.csv"));
+				choices.writeChoiceEvaluationPerMode(output.getPath("mode_choice_evaluation_per_mode.csv"));
+				choices.writeConfusionMatrix(output.getPath("mode_confusion_matrix.csv"));
+				choices.writeModePredictionError(output.getPath("mode_prediction_error.csv"));
+			} catch (RuntimeException e) {
+				log.error("Error while analyzing mode choices", e);
+			}
+		}
+
 		writePopulationStats(persons, joined);
 
 		writeTripStats(joined);
@@ -345,7 +390,6 @@ private void writePopulationStats(Table persons, Table trips) throws IOException
 		table.write().csv(output.getPath("mode_users.csv").toFile());
 
 		try (CSVPrinter printer = new CSVPrinter(Files.newBufferedWriter(output.getPath("population_trip_stats.csv")), CSVFormat.DEFAULT)) {
-
 			printer.printRecord("Info", "Value");
 			printer.printRecord("Persons", tripsPerPerson.size());
 			printer.printRecord("Mobile persons [%]", new BigDecimal(100 * totalMobile / tripsPerPerson.size()).setScale(2, RoundingMode.HALF_UP));
@@ -386,8 +430,7 @@ private void writeTripPurposes(Table trips) {
 		TextColumn purpose = trips.textColumn("end_activity_type");
 
 		// Remove suffix durations like _345
-		Selection withDuration = purpose.matchesRegex("^.+_[0-9]+$");
-		purpose.set(withDuration, purpose.where(withDuration).replaceAll("_[0-9]+$", ""));
+		purpose.set(Selection.withRange(0, purpose.size()), purpose.replaceAll("_[0-9]{2,}$", ""));
 
 		Table tArrival = trips.summarize("trip_id", count).by("end_activity_type", "arrival_h");