Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update runtime component to use manifests #130

Merged
merged 41 commits into from
Aug 23, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
41 commits
Select commit Hold shift + click to select a range
0bdc15d
Add initial pyproject.toml
patrickpa Aug 20, 2024
490eeb1
Add base models for manifests
patrickpa Aug 20, 2024
4ad7bc5
Update runtime requirements
patrickpa Aug 21, 2024
c5018c1
Add initial implementation with manifest
patrickpa Aug 21, 2024
aa52e63
Add working version of runtime and manifests
patrickpa Aug 21, 2024
367b872
Add output folder creation
patrickpa Aug 21, 2024
23351ac
Update output folder with decimal date part
patrickpa Aug 21, 2024
5bb7fb0
Remove lxml from direct requirements
patrickpa Aug 21, 2024
188da6a
Update tests for runtime module with manifests
patrickpa Aug 21, 2024
ca9ef01
Temporary remove test build condition
patrickpa Aug 21, 2024
464605e
Add baselib to base requirements
patrickpa Aug 21, 2024
724c357
Fix demo checker bundle manifest for tests
patrickpa Aug 21, 2024
c63e3b0
Revert workflow test
patrickpa Aug 21, 2024
463b9a3
Reanable run runtime test on every commit
patrickpa Aug 21, 2024
c5e1382
Update logic to run from config instead of manifest
patrickpa Aug 21, 2024
daee0f8
Fix text report manifest to handle config
patrickpa Aug 21, 2024
a4463ec
Remove requirements.txt
patrickpa Aug 21, 2024
449e1fa
Update runtime test workflow with proper pip install
patrickpa Aug 21, 2024
c9f3b87
Add poetry to runtime exec test
patrickpa Aug 21, 2024
443d206
Update poetry install with dev dependencies
patrickpa Aug 21, 2024
69dd17a
Update poetry lock
patrickpa Aug 21, 2024
e8d3961
Add latest version for qc base lib
patrickpa Aug 21, 2024
5d1fed2
Merge docker changes from develop branch
patrickpa Aug 22, 2024
8b9246a
Update linux bare ubuntu version
patrickpa Aug 22, 2024
326b54d
Update runtime test on windows workflow
patrickpa Aug 22, 2024
586c193
Add complete path to poetry executable on windows
patrickpa Aug 22, 2024
fd317f8
Test relative env to app data
patrickpa Aug 22, 2024
ff08a11
Test with start process on windows workflow
patrickpa Aug 22, 2024
b8ff20d
Test windows runtime tests with default python
patrickpa Aug 22, 2024
81b6871
Add windows and linux test separation
patrickpa Aug 22, 2024
881cb45
Fix runtime tests on docker linux
patrickpa Aug 22, 2024
cb008e3
Add check for duplicated entry on runtime
patrickpa Aug 22, 2024
98077af
Adjust windows manifests commands
patrickpa Aug 22, 2024
cbeebad
Update windows commands with quoted for relative path
patrickpa Aug 22, 2024
2bb4f15
Update windows manifest for cmd syntax
patrickpa Aug 22, 2024
761016a
Fix relative path for binaries on windows manifests
patrickpa Aug 22, 2024
a9c2950
Enable default events for runtime
patrickpa Aug 23, 2024
cf63aed
Update demo pipeline to use manifest version of runtime
patrickpa Aug 23, 2024
8f8bc7e
Update demo pipeline to use direct installation instead of custom exe…
patrickpa Aug 23, 2024
1e1ac11
Update runtime error messages
patrickpa Aug 23, 2024
75be75e
Remove custom executables from docker image
patrickpa Aug 23, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 4 additions & 3 deletions .github/workflows/build-on-change-linux-bare.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ on:

jobs:
build-linux:
runs-on: ubuntu-20.04
runs-on: ubuntu-22.04

env:
TEST_ENABLED: ${{ github.event_name == 'pull_request' && 'ON' || 'OFF' }}
Expand Down Expand Up @@ -120,8 +120,9 @@ jobs:
cp -r /home/$(whoami)/qc-build/bin bin
cp out_build/examples/checker_bundle_example/DemoCheckerBundle bin/
cd runtime
python3 -m pip install -r requirements.txt
python3 -m pytest -rA > runtime_test.log
curl -sSL https://install.python-poetry.org | python3 -
poetry install --with dev
poetry run pytest -rA > runtime_test.log

- name: Archive runtime test results
if: github.event_name == 'pull_request' && (success() || failure())
Expand Down
6 changes: 3 additions & 3 deletions .github/workflows/build-on-change-windows.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -160,8 +160,8 @@ jobs:
Copy-Item -Path "$env:WORKING_PATH\qc-framework\qc-framework\out_build\examples\checker_bundle_example\Release\DemoCheckerBundle.exe" -Destination "$env:WORKING_PATH\qc-framework\qc-framework\bin"

cd "$env:WORKING_PATH\qc-framework\qc-framework\runtime"
python3 -m pip install -r requirements.txt
python3 -m pytest

python3 -m pip install poetry
python3 -m poetry install
python3 -m poetry run pytest
Write-Output "All runtime tests done."
shell: pwsh
9 changes: 9 additions & 0 deletions demo_pipeline/manifests/framework_manifest.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
{
"manifest_file_path": [
"/app/demo_pipeline/manifests/otx_manifest.json",
"/app/demo_pipeline/manifests/osc_manifest.json",
"/app/demo_pipeline/manifests/odr_manifest.json",
"/app/demo_pipeline/manifests/result_pooling.json",
"/app/demo_pipeline/manifests/text_report.json"
]
}
10 changes: 10 additions & 0 deletions demo_pipeline/manifests/odr_manifest.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
{
"module": [
{
"name": "xodrBundle",
"exec_type": "executable",
"module_type": "checker_bundle",
"exec_command": "cd $ASAM_QC_FRAMEWORK_WORKING_DIR && qc_opendrive -c $ASAM_QC_FRAMEWORK_CONFIG_FILE"
}
]
}
10 changes: 10 additions & 0 deletions demo_pipeline/manifests/osc_manifest.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
{
"module": [
{
"name": "xoscBundle",
"exec_type": "executable",
"module_type": "checker_bundle",
"exec_command": "cd $ASAM_QC_FRAMEWORK_WORKING_DIR && qc_openscenario -c $ASAM_QC_FRAMEWORK_CONFIG_FILE"
}
]
}
10 changes: 10 additions & 0 deletions demo_pipeline/manifests/otx_manifest.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
{
"module": [
{
"name": "otxBundle",
"exec_type": "executable",
"module_type": "checker_bundle",
"exec_command": "cd $ASAM_QC_FRAMEWORK_WORKING_DIR && qc_otx -c $ASAM_QC_FRAMEWORK_CONFIG_FILE"
}
]
}
10 changes: 10 additions & 0 deletions demo_pipeline/manifests/result_pooling.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
{
"module": [
{
"name": "ResultPooling",
"exec_type": "executable",
"module_type": "result_pooling",
"exec_command": "cd $ASAM_QC_FRAMEWORK_WORKING_DIR && /app/framework/bin/ResultPooling $ASAM_QC_FRAMEWORK_WORKING_DIR $ASAM_QC_FRAMEWORK_CONFIG_FILE"
}
]
}
10 changes: 10 additions & 0 deletions demo_pipeline/manifests/text_report.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
{
"module": [
{
"name": "TextReport",
"exec_type": "executable",
"module_type": "report_module",
"exec_command": "cd $ASAM_QC_FRAMEWORK_WORKING_DIR && /app/framework/bin/TextReport $ASAM_QC_FRAMEWORK_CONFIG_FILE"
}
]
}
10 changes: 0 additions & 10 deletions demo_pipeline/otxBundle

This file was deleted.

4 changes: 2 additions & 2 deletions demo_pipeline/run_pipeline.sh
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,9 @@

python3 /app/demo_pipeline/configuration_generator.py

python3 /app/framework/runtime/runtime/runtime.py \
qc_runtime \
--config "/tmp/generated_config/config.xml" \
--install_dir "/app/framework/bin"
--manifest "/app/demo_pipeline/manifests/framework_manifest.json"

mkdir -p /out/qc-result-$INPUT_FILENAME
cp /app/framework/bin/*.xqar /out/qc-result-$INPUT_FILENAME
Expand Down
4 changes: 2 additions & 2 deletions demo_pipeline/templates/otx_template.xml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
<Param name="InputFile" value="test.otx" />

<CheckerBundle application="otxBundle">
<Param name="resultFile" value="otx_bundle_report.xqar" />
<Param name="resultFile" value="/app/framework/bin/otx_bundle_report.xqar" />
<Checker checkerId="core_otx" maxLevel="1" minLevel="3" />
<Checker checkerId="data_type_otx" maxLevel="1" minLevel="3" />
<Checker checkerId="zip_file_otx" maxLevel="1" minLevel="3" />
Expand All @@ -13,7 +13,7 @@

<ReportModule application="TextReport">
<Param name="strInputFile" value="Result.xqar" />
<Param name="strReportFile" value="Report.txt" />
<Param name="strReportFile" value="/app/framework/bin/Report.txt" />
</ReportModule>

</Config>
4 changes: 2 additions & 2 deletions demo_pipeline/templates/xodr_template.xml
Original file line number Diff line number Diff line change
Expand Up @@ -4,15 +4,15 @@
<Param name="InputFile" value="test.xodr" />

<CheckerBundle application="xodrBundle">
<Param name="resultFile" value="xodr_bundle_report.xqar" />
<Param name="resultFile" value="/app/framework/bin/xodr_bundle_report.xqar" />
<Checker checkerId="semantic_xodr" maxLevel="1" minLevel="3" />
<Checker checkerId="geometry_xodr" maxLevel="1" minLevel="3" />
<Checker checkerId="performance_xodr" maxLevel="1" minLevel="3" />
</CheckerBundle>

<ReportModule application="TextReport">
<Param name="strInputFile" value="Result.xqar" />
<Param name="strReportFile" value="Report.txt" />
<Param name="strReportFile" value="/app/framework/bin/Report.txt" />
</ReportModule>

</Config>
4 changes: 2 additions & 2 deletions demo_pipeline/templates/xosc_template.xml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
<Param name="InputFile" value="template.xosc" />

<CheckerBundle application="xoscBundle">
<Param name="resultFile" value="xosc_bundle_report.xqar" />
<Param name="resultFile" value="/app/framework/bin/xosc_bundle_report.xqar" />
<Checker checkerId="basic_xosc" maxLevel="1" minLevel="3" />
<Checker checkerId="schema_xosc" maxLevel="1" minLevel="3" />
<Checker checkerId="data_type_xosc" maxLevel="1" minLevel="3" />
Expand All @@ -14,7 +14,7 @@

<ReportModule application="TextReport">
<Param name="strInputFile" value="Result.xqar" />
<Param name="strReportFile" value="Report.txt" />
<Param name="strReportFile" value="/app/framework/bin/Report.txt" />
</ReportModule>

</Config>
10 changes: 0 additions & 10 deletions demo_pipeline/xodrBundle

This file was deleted.

10 changes: 0 additions & 10 deletions demo_pipeline/xoscBundle

This file was deleted.

20 changes: 7 additions & 13 deletions docker/Dockerfile.linux
Original file line number Diff line number Diff line change
Expand Up @@ -50,22 +50,13 @@ RUN echo "Building framework..." && \
cmake --install ./build && \
echo "Done."


# Copy bash script to install dir - OpenSCENARIO + OpenDRIVE + OTX
RUN chmod +x /app/demo_pipeline/xoscBundle && \
cp /app/demo_pipeline/xoscBundle /home/root/qc-build/bin/xoscBundle && \
chmod +x /app/demo_pipeline/xodrBundle && \
cp /app/demo_pipeline/xodrBundle /home/root/qc-build/bin/xodrBundle && \
chmod +x /app/demo_pipeline/otxBundle && \
cp /app/demo_pipeline/otxBundle /home/root/qc-build/bin/otxBundle

RUN python3 -m pip install --no-cache-dir -r /app/demo_pipeline/requirements.txt && \
python3 -m pip install --no-cache-dir -r /app/framework/runtime/requirements.txt && \
python3 -m pip install --no-cache-dir /app/framework/runtime && \
python3 -m pip install asam-qc-opendrive@git+https://github.com/asam-ev/qc-opendrive@develop && \
python3 -m pip install asam-qc-openscenarioxml@git+https://github.com/asam-ev/qc-openscenarioxml@develop && \
python3 -m pip install asam-qc-otx@git+https://github.com/asam-ev/qc-otx@develop


# Runtime test stage
FROM ubuntu:22.04 as runtime_test

Expand All @@ -75,6 +66,7 @@ RUN echo "Installing Qt..." && \
qtbase5-dev \
libqt5xmlpatterns5-dev \
libxerces-c-dev \
python3-pip \
pkg-config && \
echo "Dependencies installed."

Expand All @@ -95,7 +87,9 @@ SHELL ["/bin/bash", "-c"]
COPY --from=framework_builder /usr/local/lib/python3.10/dist-packages /usr/local/lib/python3.10/dist-packages
COPY --from=framework_builder /usr/local/bin /usr/local/bin

CMD python3 -m pytest -rA > runtime_test.log && cp /app/framework/runtime/runtime_test.log /out/runtime_test.log
RUN python3 -m pip install poetry==1.8.3
RUN poetry install --with dev
CMD poetry run pytest -rA > runtime_test.log && cp /app/framework/runtime/runtime_test.log /out/runtime_test.log

# Runtime stage
FROM framework_builder as unit_test
Expand Down
Loading
Loading