runs.geojson doesn't include connection pistes

[rodrmoya]

Connection pistes (piste:type=connection) provide the connections between lifts and ski
runs, but are missing in the runs.geojson.

Been trying to test a patch to include them, but been unable to run the data processor, running
always into an out of memory exception:

$ BBOX="[0.5253, 42.5893, 0.6092, 42.541]" ./run.sh --skip-download
Starting clustering DB...
WARN[0000] /openskidata-processor/docker-compose.yml: `version` is obsolete
[+] Running 2/2
 ✔ Container openskidata-processor-arangodb-1    Started                                                                          0.0s
 ✔ Container openskidata-processor-tippecanoe-1  Started                                                                          0.0s
WARN[0000] /openskidata-processor/docker-compose.yml: `version` is obsolete
Converting to GeoJSON...

> [email protected] prepare-geojson
> npm run build && node --optimize_for_size --max_old_space_size=3072 dist/scripts/prepare_geojson.js


> [email protected] build
> babel src --out-dir dist --copy-files --extensions '.ts,.js'

Successfully compiled 74 files with Babel (583ms).
Processing ski areas...
(node:15369) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.
(Use `node --trace-deprecation ...` to show where the warning was created)
Processing runs...

<--- Last few GCs --->

[15369:0x138008000]    72533 ms: Mark-Compact 3009.0 (3074.3) -> 3008.0 (3074.3) MB, pooled: 1 MB, 874.67 / 0.00 ms  (average mu = 0.060, current mu = 0.001) allocation failure; scavenge might not succeed
[15369:0x138008000]    73409 ms: Mark-Compact 3009.0 (3074.8) -> 3008.0 (3074.5) MB, pooled: 1 MB, 874.83 / 0.00 ms  (average mu = 0.032, current mu = 0.002) allocation failure; scavenge might not succeed


<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
----- Native stack trace -----

 1: 0x102ba9228 node::OOMErrorHandler(char const*, v8::OOMDetails const&) [/opt/homebrew/Cellar/node/22.1.0/bin/node]
 2: 0x102d305e0 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [/opt/homebrew/Cellar/node/22.1.0/bin/node]
 3: 0x102d30594 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [/opt/homebrew/Cellar/node/22.1.0/bin/node]
 4: 0x102ed94ac v8::internal::Heap::CallGCPrologueCallbacks(v8::GCType, v8::GCCallbackFlags, v8::internal::GCTracer::Scope::ScopeId) [/opt/homebrew/Cellar/node/22.1.0/bin/node]
 5: 0x102edb1e8 v8::internal::Heap::DevToolsTraceEventScope::~DevToolsTraceEventScope() [/opt/homebrew/Cellar/node/22.1.0/bin/node]
 6: 0x102ed9b60 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags)::$_7::operator()() const [/opt/homebrew/Cellar/node/22.1.0/bin/node]
 7: 0x102ed9814 void heap::base::Stack::SetMarkerAndCallbackImpl<v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags)::$_7>(heap::base::Stack*, void*, void const*) [/opt/homebrew/Cellar/node/22.1.0/bin/node]
 8: 0x102abc028 PushAllRegistersAndIterateStack [/opt/homebrew/Cellar/node/22.1.0/bin/node]
 9: 0x102ed81dc v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/opt/homebrew/Cellar/node/22.1.0/bin/node]
10: 0x102ece7f8 v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/opt/homebrew/Cellar/node/22.1.0/bin/node]
11: 0x102ecef64 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/opt/homebrew/Cellar/node/22.1.0/bin/node]
12: 0x102eb6350 v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [/opt/homebrew/Cellar/node/22.1.0/bin/node]
13: 0x1031cfa84 v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [/opt/homebrew/Cellar/node/22.1.0/bin/node]
14: 0x10299dc74 Builtins_CEntry_Return1_ArgvOnStack_NoBuiltinExit [/opt/homebrew/Cellar/node/22.1.0/bin/node]
15: 0x10297ad9c Builtins_ObjectKeys [/opt/homebrew/Cellar/node/22.1.0/bin/node]
16: 0x10b9e1908
17: 0x10ba2c830
18: 0x10ba2eb4c
19: 0x10b9eddb8
20: 0x10b9e9a58
21: 0x10b9e97a4
22: 0x10b9e97a4
23: 0x10b9ecd4c
24: 0x1029cc7dc Builtins_ArrayMap [/opt/homebrew/Cellar/node/22.1.0/bin/node]
25: 0x102909e60 Builtins_InterpreterEnterAtBytecode [/opt/homebrew/Cellar/node/22.1.0/bin/node]
26: 0x102908ef0 Builtins_InterpreterEntryTrampoline [/opt/homebrew/Cellar/node/22.1.0/bin/node]
27: 0x102908ef0 Builtins_InterpreterEntryTrampoline [/opt/homebrew/Cellar/node/22.1.0/bin/node]
28: 0x102909e60 Builtins_InterpreterEnterAtBytecode [/opt/homebrew/Cellar/node/22.1.0/bin/node]
29: 0x102908ef0 Builtins_InterpreterEntryTrampoline [/opt/homebrew/Cellar/node/22.1.0/bin/node]
30: 0x102908ef0 Builtins_InterpreterEntryTrampoline [/opt/homebrew/Cellar/node/22.1.0/bin/node]
31: 0x10b8e9f48
32: 0x102906c0c Builtins_JSEntryTrampoline [/opt/homebrew/Cellar/node/22.1.0/bin/node]
33: 0x1029068f4 Builtins_JSEntry [/opt/homebrew/Cellar/node/22.1.0/bin/node]
34: 0x102e43b7c v8::internal::(anonymous namespace)::Invoke(v8::internal::Isolate*, v8::internal::(anonymous namespace)::InvokeParams const&) [/opt/homebrew/Cellar/node/22.1.0/bin/node]
35: 0x102e4351c v8::internal::Execution::Call(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, int, v8::internal::Handle<v8::internal::Object>*) [/opt/homebrew/Cellar/node/22.1.0/bin/node]
36: 0x102d42bfc v8::Function::Call(v8::Local<v8::Context>, v8::Local<v8::Value>, int, v8::Local<v8::Value>*) [/opt/homebrew/Cellar/node/22.1.0/bin/node]
37: 0x102abcc68 node::InternalCallbackScope::Close() [/opt/homebrew/Cellar/node/22.1.0/bin/node]
38: 0x102abd0e8 node::InternalMakeCallback(node::Environment*, v8::Local<v8::Object>, v8::Local<v8::Object>, v8::Local<v8::Function>, int, v8::Local<v8::Value>*, node::async_context) [/opt/homebrew/Cellar/node/22.1.0/bin/node]
39: 0x102ad4afc node::AsyncWrap::MakeCallback(v8::Local<v8::Function>, int, v8::Local<v8::Value>*) [/opt/homebrew/Cellar/node/22.1.0/bin/node]
40: 0x102baf61c node::fs::FSReqCallback::Resolve(v8::Local<v8::Value>) [/opt/homebrew/Cellar/node/22.1.0/bin/node]
41: 0x102bb13cc node::fs::AfterInteger(uv_fs_s*) [/opt/homebrew/Cellar/node/22.1.0/bin/node]
42: 0x102ba0560 node::MakeLibuvRequestCallback<uv_fs_s, void (*)(uv_fs_s*)>::Wrapper(uv_fs_s*) [/opt/homebrew/Cellar/node/22.1.0/bin/node]
43: 0x106dfedfc uv__work_done [/opt/homebrew/Cellar/libuv/1.48.0/lib/libuv.1.dylib]
44: 0x106e024a8 uv__async_io [/opt/homebrew/Cellar/libuv/1.48.0/lib/libuv.1.dylib]
45: 0x106e12164 uv__io_poll [/opt/homebrew/Cellar/libuv/1.48.0/lib/libuv.1.dylib]
46: 0x106e0293c uv_run [/opt/homebrew/Cellar/libuv/1.48.0/lib/libuv.1.dylib]
47: 0x102abd990 node::SpinEventLoopInternal(node::Environment*) [/opt/homebrew/Cellar/node/22.1.0/bin/node]
48: 0x102bf179c node::NodeMainInstance::Run(node::ExitCode*, node::Environment*) [/opt/homebrew/Cellar/node/22.1.0/bin/node]
49: 0x102bf147c node::NodeMainInstance::Run() [/opt/homebrew/Cellar/node/22.1.0/bin/node]
50: 0x102b67030 node::Start(int, char**) [/opt/homebrew/Cellar/node/22.1.0/bin/node]
51: 0x18acc20e0 start [/usr/lib/dyld]
sh: line 1: 15369 Abort trap: 6           node --optimize_for_size --max_old_space_size=3072 dist/scripts/prepare_geojson.js
npm notice
npm notice New minor version of npm available! 10.7.0 -> 10.8.0
npm notice Changelog: https://github.com/npm/cli/releases/tag/v10.8.0
npm notice To update run: npm install -g [email protected]
npm notice

This happens even when selecting a smaller dataset via the BBOX env variable.

[russellporter]

Hi @rodrmoya, it makes sense, I think I tested out connections at one point but decided to not include them because I didn't find a good visualization.

I suspect the OOM issue might be due to your BBOX="[0.5253, 42.5893, 0.6092, 42.541]" - the bottom latitude is above the top, that might cause the bbox to ignored. The project doesn't easily work for the whole world anymore due to the high memory usage.