Release 2.0

.dockerignore

@@ -13,4 +13,11 @@ build*
 # NSight profiler files
 *.ncu-rep
 *.nsys-rep
+# FPGA files
 *.bit
+# cscope index files
+cscope.*
+tags
+# pcap files
+*.pcap
+*.deb

CMakeLists.txt

@@ -29,7 +29,7 @@ find_package(holoscan 0.6 REQUIRED CONFIG PATHS "/opt/nvidia/holoscan")
 find_package(Python3 REQUIRED COMPONENTS Interpreter Development)
 
 set(HOLOSCAN_SDK_VERSION "$ENV{HOLOSCAN_SDK_VERSION}")
-set(HOLOSCAN_SDK_VERSION_EXPECTED "2.3.0")
+set(HOLOSCAN_SDK_VERSION_EXPECTED "2.7.0")
 
 if(NOT HOLOSCAN_SDK_VERSION VERSION_EQUAL HOLOSCAN_SDK_VERSION_EXPECTED)
   message(FATAL_ERROR "Expected Holoscan version ${HOLOSCAN_SDK_VERSION_EXPECTED} but found ${HOLOSCAN_SDK_VERSION}, please check pybind11 version of the Holoscan SDK and update here if needed")

CONTRIBUTING.md

@@ -49,8 +49,8 @@ before making their first contribution.
 ### Workflow
 
 1. Developers must first [fork](https://help.github.com/en/articles/fork-a-repo) the
-   \[upstream\](https://github.com/nvidia-holoscan/holoscan-sensor-bridge) Holoscan
-   Sensor Bridge repository.
+   [upstream](https://github.com/nvidia-holoscan/holoscan-sensor-bridge) Holoscan Sensor
+   Bridge repository.
 
 1. Git clone the forked repository and push changes to the personal fork.
 

RELEASE_NOTES.md

@@ -1,5 +1,134 @@
 # Release Notes
 
+## 2.0-GA, January 2025
+
+### Dependencies
+
+- IGX: [IGX-SW 1.1 Production Release](https://developer.nvidia.com/igx-downloads)
+- AGX: Use [SDK Manager](https://developer.nvidia.com/sdk-manager) to set up JetPack 6.0
+  release 2. Note that JetPack 6.1 is not yet supported for HSB.
+- Holoscan Sensor Bridge, 10G; FPGA v2412
+
+Be sure and follow the installation instructions included with the release. To generate
+documentation, in the host system, run `sh docs/make_docs.sh`, then use your browser to
+look at `docs/user_guide/_build/html/index.html`.
+
+### Updates from 1.1-GA
+
+- **HSB 2.0-GA relies on FPGA IP version 2412.** Check the user guide for instructions
+  on how to update your configuration. Note that the enumeration data has changed, so
+  pre 2.0-GA software will not enumerate boards publishing 1.1 (or earlier) enumeration
+  data; and likewise, 1.1 and earlier software will not find the newer 2.0 configuration
+  boards. For Lattice-CLNX100-ETH-SENSOR-BRIDGE devices, be sure and include the
+  "--force" option when updating the HSB firmware; this way the software uses hardcoded
+  enumeration data in the software tree instead of relying on that from the device
+  itself. See [the firmware download instructions](sensor_bridge_firmware_setup.md) for
+  more details. If you need to revert your FPGA back to the 2407 version, check the FAQ
+  below.
+
+- **HSB is updated to work with Holoscan SDK 2.7.** Some older APIs, specifically in the
+  [C++ fmt tool](https://github.com/fmtlib/fmt) have been deprecated, so minor code
+  adjustments have been applied to keep HSB host code up to date.
+
+- **New HSB features for safety and reliability,** including CRCs, control plane
+  sequence number checking, and additional timestamps are included. Timestamps included
+  capture the PTP time when the first data in the received frame is observed by the FPGA
+  IP block and the time after the last data in the frame is sent. With ConnectX based
+  host systems, which support hardware PTP synchronization, these timestamps are within
+  a microsecond of the host time, and can be used to accurately measure latency through
+  the pipeline. These metadata values are available to pipeline operators via the
+  [HSDK application metadata API](https://docs.nvidia.com/holoscan/sdk-user-guide/holoscan_create_app.html#dynamic-application-metadata).
+  See the user guide for more details. Sequence number checking is enabled for control
+  plane transactions, and can provide protection against interaction from several hosts
+  to the same HSB unit. The overall CRC of the received data frame is also included, in
+  a later release, a high-performance CUDA based CRC checker will be demonstrated
+  showing frame-rate CRC validation of ingress data.
+
+- **Multiple sensors over a single network port.** APIs are added to allow applications
+  to configure multiple sensors to use the same network port. In the examples directory,
+  `single_network_stereo_imx274_player.py` demonstrates how to configure both cameras in
+  an IMX274 stereo pair to transmit 1080p video using a single network port. Note that
+  4k video streams require about 6.5Gbps each, so using both cameras in this mode over a
+  single network port is not supported. See the user guide for more details.
+
+- **Performance and latency measurement tools.** The timestamps included with safety and
+  reliability features can be used to accurately measure latency, from the time that
+  data arrives to the FPGA IP block, all the way through the end of the pipeline (e.g.
+  following visualization). See `examples/imx274_transfer_latency.py` for an example.
+  See [latency.md](latency.md) for more details on latency measurement.
+
+- **GammaCorrectionOp is removed.** HSDK 2.3 added support for
+  [sRGB space](https://en.wikipedia.org/wiki/SRGB), providing an optimized path
+  including Gamma correction in the visualizer. By removing HSB's naive gamma correction
+  and using the visualizer instead, pipeline latency is reduced by .5ms. For
+  applications that used GammaCorrectionOp, just remove that operator from the pipeline
+  and include `framebuffer_srgb=True` in the constructor parameter list for HolovizOp.
+
+- **Support for IMX477 cameras via Microchip MPF200-ETH-SENSOR-BRIDGE.**
+
+### FAQ
+
+- If your application, running in the demo container, halts with a "Failed to initialize
+  glfw" message, make sure to grant the application permission to connect with the
+  display via the "xhost +" command. This command is not remembered across reboots.
+
+- Reverting from FPGA 2412 to 2407 on Lattice HSB units. If you need to revert a Lattice
+  HSB unit from 2412 back to 2407, use the 2.0-GA tree and program with the 2407
+  manifest file. From within the demo container:
+
+  ```sh
+  hololink program scripts/manifest-2407.yaml
+  ```
+
+  After programming and power cycling, the board will no longer be visible to the 2.0-GA
+  version of HSB host code. At this time you can go back to using the 1.1-GA release to
+  work with the board. The 1.1-GA software will not be able to enumerate boards running
+  the 2412 configuration; the newer tree must be used to write the older firmware.
+
+- HSB network receiver operators use
+  [APIs provided by the Holoscan SDK](https://docs.nvidia.com/holoscan/sdk-user-guide/holoscan_create_app.html#dynamic-application-metadata)
+  to share timestamps with later operators in the pipeline. Be sure and call the
+  application (C++) `is_metadata_enabled(true)` method or (python)
+  `is_metadata_enabled = True` at initialization time; otherwise each operator will only
+  see an empty metadata structure. In your operator's `compute` method, if you add
+  additional items to the pipeline metadata, be sure and add that metadata before
+  calling `(output).emit`. If you have a pipeline that merges two paths, and experience
+  a `runtime_error` exception when it fails to merge the metadata from those paths, see
+  [the page on Metadata update policies](https://docs.nvidia.com/holoscan/sdk-user-guide/holoscan_create_app.html#metadata-update-policies)
+  for information on how to manage this.
+
+- It is possible to overrun the bandwidth available on the ethernet, particularly when
+  using multiple sensors over a single network connection. For example, a 4k, 60FPS,
+  RAW10 video stream requires about 6.5Gbps, and a stereo pair configured this way would
+  require something like 13Gbps--which far exceeds a single 10Gbps network port. In this
+  case, HSB will drop data, probably within a single network message. When operated
+  outside of specification this way, reconstruction of the original sensor data is not
+  possible. The software has no concept of available bandwidth, so it is up to the
+  developer to ensure that bandwidth limits are not exceeded.
+
+- AGX on-board ethernet supports hardware PTP synchronization, which can be enabled by
+  following the same directions given to set up PTP for IGX. Specifically, with the
+  appropriate network device name in `$EN0` (e.g. `eth0`), just follow the instructions
+  on the host setup page for setting up PTP on IGX.
+
+### Known Anomalies
+
+- Orin AGX running on JetPack 6.1 shows very slow network behavior. Investigation of
+  this is underway; for now, on Orin AGX, only JetPack 6.0 r2 is supported.
+
+- Following software-commanded reset, HSB sometimes observes a sequence number or
+  control plane transaction failure. When the HSB is commanded to reset, the host system
+  observes a loss of network connectivity and may take some time before steady
+  communication is available. In some specific Orin AGX systems, `dmesg` shows this
+  renegotiation can take more than 60 seconds. During this time, HSB software attempts
+  to read the FPGA version ID, and can time out. Investigation of this is underway; for
+  now, systems with this behavior can be worked around by putting a 10G ethernet switch
+  between the HSB and the host system.
+
+- Orin AGX systems, running with stereo sensor feeds on the same network port, using
+  either the multithreaded or event based schedulers, have unreliable operation. For
+  now, AGX systems with HSB are only supported with the default (greedy) scheduler.
+
 ## 1.1.0-GA, August 2024
 
 ### Dependencies
@@ -34,7 +163,7 @@ look at `docs/user_guide/_build/html/index.html`.
 - There are some small **changes to the host setup instructions**. Changes focused on
   updates to network device names and performance on AGX configurations.
 
-- **Hardware ISP via ArgusIspOp** \[Orin in iGPU mode only\]. Applications can offload
+- **Hardware ISP via ArgusIspOp** [Orin in iGPU mode only]. Applications can offload
   image signal processing by using the capabilities built in to the NV ISP device
   present in Orin systems running with iGPU (AGX or IGX without a dGPU). Support is
   provided for 1080p images; contact NVIDIA to get updated libraries with support for 4K
@@ -76,7 +205,45 @@ look at `docs/user_guide/_build/html/index.html`.
   The first time the body-post-estimation app is run, the .onnx file is converted to a
   TRT engine file, which is a step than can take several minutes. Subsequent runs of the
   body pose estimation app will skip the conversion and just load this engine file
-  directly. During the converstion, when high bandwidth is in use on the network (via
+  directly. During the conversion, when high bandwidth is in use on the network (via
   "--camera-mode=0"), the kernel stops delivering received UDP messages to the
   application, resulting in no video being displayed. Later runs the same program, after
   the conversion is complete, run as expected.
+
+- HSB does not forward data received on the MIPI interface, resulting in an "Ingress
+  frame timeout; ignoring." message.
+
+  A bug in the FPGA MIPI receiver block IP causes data to be dropped before being
+  delivered to the FPGA's UDP packetizer; resulting in no sensor data being delivered to
+  the host. If you've commanded a camera to send image data, but no data is observed and
+  the timeout message is displayed, you can verify that this is the cause by issuing
+  these commands within the HSB demo container:
+
+  ```
+  hololink read_uint32 0x50000000   # for the first camera
+  hololink read_uint32 0x60000000   # for the second camera
+  ```
+
+  If a camera is configured to issue data, but a 0 appears in this memory location, then
+  this is an indication that the receiver is in this stuck state. `hololink.reset()` is
+  able to clear this condition.
+
+- PTP timestamps, published by HSB, aren't synchronized with the host time.
+
+  The
+  [user guide hardware setup instructions](https://docs.nvidia.com/holoscan/sensor-bridge/1.1.0/setup.html)
+  show how to configure the `phc2sys` tool, which ensures that the NIC time-of-day
+  clock, which is published with PTP network messages, is synchronized with the host
+  real-time clock. As written, the setup instructions rely on the default ntpdate
+  configuration to initialize the system clock to the rest of the world. As written, the
+  `phc2sys` startup doesn't properly wait for `ntpdate` to complete, so that when
+  `ntpdate` does finish, `phc2sys` sees a large jump in the system time. Because
+  `phc2sys` slowly adjusts the NIC clock, the time published by `ptp4l` will not be
+  synchronized with the system clock, and could take a very long time to do so. To
+  verify you're in this condition, observe the output in the "sys offset" column from
+  the command `systemctl status phc2sys-*.service`; large absolute values are an
+  indication of this condition. To work around this, run
+  `sudo systemctl restart phc2sys-*.service` after ntpdate is synchronized. There is
+  some anecdotal evidence that adding "-w" to the `phc2sys` command line may fix this
+  problem but the documentation for this option doesn't address the configuration in use
+  here.

VERSION

@@ -1 +1 @@
-1.1.0
+2.0.0

ci/lint.sh

@@ -80,6 +80,7 @@ MODE="lint"
 case "$1" in
     --do-format)
         # We rely on 'set -o errexit' above to terminate on an error
+        set -o xtrace
         isort $SKIP_ISORT --profile black .
         black "--extend-exclude=$SKIP_RE" .
         flake8 --ignore=E501,E203,W503 --extend-exclude=$SKIP_COMMAS
@@ -89,6 +90,7 @@ case "$1" in
         ;;
     --do-lint)
         # We rely on 'set -o errexit' above to terminate on an error
+        set -o xtrace
         isort --check-only $SKIP_ISORT --profile black .
         black --check "--extend-exclude=$SKIP_RE" .
         flake8 --ignore=E501,E203,W503 --extend-exclude=$SKIP_COMMAS
@@ -108,9 +110,11 @@ case "$1" in
         ;;
 esac
 
-# We only get here if we weren't "--do-lint" or "--do-format";
-# we're going to "--do-$MODE".
-docker build \
+# We only get here if we weren't "--do-lint" or "--do-format"; we're going to
+# "--do-$MODE".
+# Also, we specifically rely on buildkit skipping the dgpu or igpu stages that
+# aren't included in the final image we're creating.
+DOCKER_BUILDKIT=1 docker build \
     -t hololink-lint:$VERSION \
     -f $HERE/Dockerfile.lint \
     $ROOT

docker/Dockerfile

@@ -15,11 +15,42 @@
 #
 # See README.md for detailed information.
 
-ARG CONTAINER_TYPE=dgpu
-FROM nvcr.io/nvidia/clara-holoscan/holoscan:v2.3.0-${CONTAINER_TYPE}
-
 ARG CONTAINER_TYPE=invalid
 
+# dgpu specific configuration goes here.
+FROM nvcr.io/nvidia/clara-holoscan/holoscan:v2.7.0-dgpu AS build_dgpu
+
+# igpu specific configuration goes here.
+FROM nvcr.io/nvidia/clara-holoscan/holoscan:v2.7.0-igpu AS build_igpu
+RUN \
+    # EGL and libGLESv2 is needed by Argus api otherwise the library throws error \
+    apt-get update && \
+    DEBIAN_FRONTEND=noninteractive apt-get install -y -q libgles2-mesa-dev && \
+    echo "deb https://repo.download.nvidia.com/jetson/common r36.3 main" > /etc/apt/sources.list.d/nvidia-l4t-apt-source.list && \
+    echo "deb https://repo.download.nvidia.com/jetson/t234 r36.3 main" >> /etc/apt/sources.list.d/nvidia-l4t-apt-source.list && \
+    wget -O - https://repo.download.nvidia.com/jetson/jetson-ota-public.asc | apt-key add - && \
+    apt-get update && \
+    # we do not have the permission to ./mm-api/DEBIAN/postinst \
+    # to just install nvidia-l4t-jetson-multimedia-api via apt install \
+    apt-get download nvidia-l4t-jetson-multimedia-api && \
+    dpkg-deb -R ./nvidia-l4t-jetson-multimedia-api_*_arm64.deb ./mm-api && \
+    cp -r ./mm-api/usr/src/jetson_multimedia_api /usr/src/jetson_multimedia_api && \
+    sed -i 's/sudo//' ./mm-api/DEBIAN/postinst && \
+    ./mm-api/DEBIAN/postinst && \
+    rm -rf ./nvidia-l4t-jetson-multimedia-api_*_arm64.deb ./mm-api && \
+    rm -rf /var/lib/apt/lists/*
+
+COPY --from=argus-libs libnvargus.so /usr/lib/aarch64-linux-gnu/nvidia/libnvargus.so
+COPY --from=argus-libs libnvargus_socketclient.so /usr/lib/aarch64-linux-gnu/nvidia/libnvargus_socketclient.so
+COPY --from=argus-libs libnvargus_socketserver.so /usr/lib/aarch64-linux-gnu/nvidia/libnvargus_socketserver.so
+COPY --from=argus-libs libnvfusacap.so /usr/lib/aarch64-linux-gnu/nvidia/libnvfusacap.so
+COPY --from=argus-libs libnvodm_imager.so /usr/lib/aarch64-linux-gnu/nvidia/libnvodm_imager.so
+COPY --from=argus-libs libnvscf.so /usr/lib/aarch64-linux-gnu/nvidia/libnvscf.so
+ENV LD_LIBRARY_PATH=/usr/lib/aarch64-linux-gnu/nvidia:$LD_LIBRARY_PATH
+
+# configuration for all GPU configurations start here.
+FROM build_${CONTAINER_TYPE}
+
 RUN apt-get update \
   && DEBIAN_FRONTEND=noninteractive apt-get install -y -q \
     python3 \
@@ -39,26 +70,6 @@ RUN apt-get update \
   && rm -rf /var/lib/apt/lists/*
 
 RUN pip3 install meson ninja pyelftools
-
-RUN if [ "$CONTAINER_TYPE" = "igpu" ]; then \
-    # libgtk is needed by Argus api otherwise the library throughs error \
-    apt-get update && \
-    DEBIAN_FRONTEND=noninteractive apt-get install -y -q libgtk-3-dev && \
-    echo "deb https://repo.download.nvidia.com/jetson/common r36.3 main" > /etc/apt/sources.list.d/nvidia-l4t-apt-source.list && \
-    echo "deb https://repo.download.nvidia.com/jetson/t234 r36.3 main" >> /etc/apt/sources.list.d/nvidia-l4t-apt-source.list && \
-    wget -O - https://repo.download.nvidia.com/jetson/jetson-ota-public.asc | apt-key add - && \
-    apt-get update && \
-    # we do not have the permission to ./mm-api/DEBIAN/postinst \
-    # to just install nvidia-l4t-jetson-multimedia-api via apt install \
-    apt-get download nvidia-l4t-jetson-multimedia-api && \
-    dpkg-deb -R ./nvidia-l4t-jetson-multimedia-api_*_arm64.deb ./mm-api && \
-    cp -r ./mm-api/usr/src/jetson_multimedia_api /usr/src/jetson_multimedia_api && \
-    sed -i 's/sudo//' ./mm-api/DEBIAN/postinst && \
-    ./mm-api/DEBIAN/postinst && \
-    rm -rf ./nvidia-l4t-jetson-multimedia-api_*_arm64.deb ./mm-api; \
-  fi
-
-ADD nvidia /usr/lib/aarch64-linux-gnu/nvidia
-
-ENV CONTAINER_TYPE=$CONTAINER_TYPE
-ENV LD_LIBRARY_PATH=/usr/lib/aarch64-linux-gnu/nvidia:$LD_LIBRARY_PATH
+# repeated ARG because it has to be inside the FROM above
+ARG CONTAINER_TYPE=invalid
+ENV CONTAINER_TYPE=${CONTAINER_TYPE}

docker/Dockerfile.demo

@@ -27,7 +27,10 @@ RUN apt-get update \
 
 ADD . /opt/hololink/
 
-RUN cd /opt/hololink/python && pip3 wheel .
+# See build.sh for why INSTALL_ENVIRONMENT is here.
+ARG INSTALL_ENVIRONMENT=""
+RUN cd /opt/hololink/python \
+    && $INSTALL_ENVIRONMENT pip3 wheel -v .
 
 # demo container
 FROM base AS demo