Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DCGM-Exporter crashes on startup when using MIG w/ A100 #443

Open
csibbitt opened this issue Jan 16, 2025 · 0 comments
Open

DCGM-Exporter crashes on startup when using MIG w/ A100 #443

csibbitt opened this issue Jan 16, 2025 · 0 comments
Labels
bug Something isn't working

Comments

@csibbitt
Copy link

What is the version?

3.3.8-3.6.0

What happened?

Exporter works fine with timesliced vGPU, but crashes on startup if I define a MIG-backed vGPU (but not if I just enable MIG mode) Removing the vGPU does not fix the exporter, but disabling MIG mode does.

I've noticed two other issues that may also be related to this:
#441
#434

I've simplified my setup down to the minimum in order to isolate this problem.

What did you expect to happen?

I expected it to report metrics on the MIG device as described in the documentation here: https://docs.nvidia.com/datacenter/cloud-native/gpu-telemetry/latest/dcgm-exporter.html#multi-instance-gpu-mig-support

What is the GPU model?

[csibbitt@hab-19 ~]$ nvidia-smi -q

==============NVSMI LOG==============

Timestamp                                 : Thu Jan 16 11:47:49 2025
Driver Version                            : 565.63
CUDA Version                              : Not Found
vGPU Driver Capability
        Heterogenous Multi-vGPU           : Supported

Attached GPUs                             : 1
GPU 00000000:04:00.0
    Product Name                          : NVIDIA A100-PCIE-40GB
    Product Brand                         : NVIDIA
    Product Architecture                  : Ampere
    Display Mode                          : Enabled
    Display Active                        : Disabled
    Persistence Mode                      : Enabled
    Addressing Mode                       : N/A
    vGPU Device Capability
        Fractional Multi-vGPU             : Supported
        Heterogeneous Time-Slice Profiles : Supported
        Heterogeneous Time-Slice Sizes    : Supported
    MIG Mode
        Current                           : Disabled
        Pending                           : Disabled
    Accounting Mode                       : Enabled
    Accounting Mode Buffer Size           : 4000
    Driver Model
        Current                           : N/A
        Pending                           : N/A
    Serial Number                         : 1323420029722
    GPU UUID                              : GPU-4b9f8464-ad90-cdde-8510-8006fc2772b7
    Minor Number                          : 0
    VBIOS Version                         : 92.00.25.00.08
    MultiGPU Board                        : No
    Board ID                              : 0x400
    Board Part Number                     : 900-21001-0000-000
    GPU Part Number                       : 20F1-883-A1
    FRU Part Number                       : N/A
    Platform Info
        RACK Serial Number                : N/A
        Chassis Physical Slot Number      : N/A
        Compute Slot Index                : N/A
        Node Index                        : N/A
        Peer Type                         : N/A
        Module Id                         : 1
    Inforom Version
        Image Version                     : 1001.0200.00.04
        OEM Object                        : 2.0
        ECC Object                        : 6.16
        Power Management Object           : N/A
    Inforom BBX Object Flush
        Latest Timestamp                  : N/A
        Latest Duration                   : N/A
    GPU Operation Mode
        Current                           : N/A
        Pending                           : N/A
    GPU C2C Mode                          : N/A
    GPU Virtualization Mode
        Virtualization Mode               : Host VGPU
        Host VGPU Mode                    : SR-IOV
        vGPU Heterogeneous Mode           : Disabled
    GPU Reset Status
        Reset Required                    : No
        Drain and Reset Recommended       : No
    GPU Recovery Action                   : None
    GSP Firmware Version                  : N/A
    IBMNPU
        Relaxed Ordering Mode             : N/A
    PCI
        Bus                               : 0x04
        Device                            : 0x00
        Domain                            : 0x0000
        Base Classcode                    : 0x3
        Sub Classcode                     : 0x2
        Device Id                         : 0x20F110DE
        Bus Id                            : 00000000:04:00.0
        Sub System Id                     : 0x145F10DE
        GPU Link Info
            PCIe Generation
                Max                       : 3
                Current                   : 3
                 Device Current            : 3
                Device Max                : 4
                Host Max                  : N/A
            Link Width
                Max                       : 16x
                Current                   : 8x
        Bridge Chip
            Type                          : N/A
            Firmware                      : N/A
        Replays Since Reset               : 0
        Replay Number Rollovers           : 0
        Tx Throughput                     : 350 KB/s
        Rx Throughput                     : 400 KB/s
        Atomic Caps Outbound              : N/A
        Atomic Caps Inbound               : N/A
    Fan Speed                             : N/A
    Performance State                     : P0
    Clocks Event Reasons
        Idle                              : Active
        Applications Clocks Setting       : Not Active
        SW Power Cap                      : Not Active
        HW Slowdown                       : Not Active
            HW Thermal Slowdown           : Not Active
            HW Power Brake Slowdown       : Not Active
        Sync Boost                        : Not Active
        SW Thermal Slowdown               : Not Active
        Display Clock Setting             : Not Active
    Sparse Operation Mode                 : N/A
    FB Memory Usage
        Total                             : 40960 MiB
        Reserved                          : 454 MiB
        Used                              : 9985 MiB
        Free                              : 30523 MiB
    BAR1 Memory Usage
        Total                             : 65536 MiB
        Used                              : 1 MiB
        Free                              : 65535 MiB
    Conf Compute Protected Memory Usage
        Total                             : 0 MiB
        Used                              : 0 MiB
        Free                              : 0 MiB
    Compute Mode                          : Default
    Utilization
        Gpu                               : 0 %
        Memory                            : 0 %
        Encoder                           : 0 %
        Decoder                           : 0 %
        JPEG                              : 0 %
        OFA                               : 0 %
    Encoder Stats
        Active Sessions                   : 0
        Average FPS                       : 0
        Average Latency                   : 0
    FBC Stats
        Active Sessions                   : 0
        Average FPS                       : 0
        Average Latency                   : 0
    ECC Mode
        Current                           : Enabled
        Pending                           : Enabled
    ECC Errors
        Volatile
            SRAM Correctable              : 0
            SRAM Uncorrectable Parity     : 0
            SRAM Uncorrectable SEC-DED    : 0
            DRAM Correctable              : 0
            DRAM Uncorrectable            : 0
        Aggregate
            SRAM Correctable              : 0
            SRAM Uncorrectable Parity     : 0
            SRAM Uncorrectable SEC-DED    : 0
            DRAM Correctable              : 0
            DRAM Uncorrectable            : 0
            SRAM Threshold Exceeded       : No
        Aggregate Uncorrectable SRAM Sources
            SRAM L2                       : 0
            SRAM SM                       : 0
            SRAM Microcontroller          : 0
            SRAM PCIE                     : 0
            SRAM Other                    : 0
    Retired Pages
        Single Bit ECC                    : N/A
        Double Bit ECC                    : N/A
        Pending Page Blacklist            : N/A
    Remapped Rows
        Correctable Error                 : 0
        Uncorrectable Error               : 0
        Pending                           : No
        Remapping Failure Occurred        : No
        Bank Remap Availability Histogram
            Max                           : 640 bank(s)
            High                          : 0 bank(s)
            Partial                       : 0 bank(s)
            Low                           : 0 bank(s)
            None                          : 0 bank(s)
    Temperature
        GPU Current Temp                  : 34 C
        GPU T.Limit Temp                  : N/A
        GPU Shutdown Temp                 : 95 C
        GPU Slowdown Temp                 : 92 C
        GPU Max Operating Temp            : 85 C
        GPU Target Temperature            : N/A
        Memory Current Temp               : N/A
        Memory Max Operating Temp         : N/A
    GPU Power Readings
        Power Draw                        : 33.41 W
        Current Power Limit               : 250.00 W
        Requested Power Limit             : 250.00 W
        Default Power Limit               : 250.00 W
        Min Power Limit                   : 150.00 W
        Max Power Limit                   : 250.00 W
    GPU Memory Power Readings 
        Power Draw                        : N/A
    Module Power Readings
        Power Draw                        : N/A
        Current Power Limit               : N/A
        Requested Power Limit             : N/A
        Default Power Limit               : N/A
        Min Power Limit                   : N/A
        Max Power Limit                   : N/A
    Clocks
        Graphics                          : 210 MHz
        SM                                : 210 MHz
        Memory                            : 1215 MHz
        Video                             : 585 MHz
    Applications Clocks
        Graphics                          : 765 MHz
        Memory                            : 1215 MHz
    Default Applications Clocks
        Graphics                          : 765 MHz
        Memory                            : 1215 MHz
    Deferred Clocks
        Memory                            : N/A
    Max Clocks
        Graphics                          : 1410 MHz
        SM                                : 1410 MHz
        Memory                            : 1215 MHz
        Video                             : 1290 MHz
    Max Customer Boost Clocks
        Graphics                          : 1410 MHz
    Clock Policy
        Auto Boost                        : N/A
        Auto Boost Default                : N/A
    Voltage
        Graphics                          : 712.500 mV
    Fabric
        State                             : N/A
        Status                            : N/A
        CliqueId                          : N/A
        ClusterUUID                       : N/A
        Health
            Bandwidth                     : N/A
    Processes
        GPU instance ID                   : N/A
        Compute instance ID               : N/A
        Process ID                        : 42699
            Type                          : C+G
            Name                          : vgpu
            Used GPU Memory               : 9984 MiB
    Capabilities
        EGM                               : disabled

What is the environment?

Host is baremetal RHEL9 w/ NVIDIA AIE Host driver 565.63 installed from .run

[csibbitt@hab-19 ~]$ cat /etc/redhat-release 
Red Hat Enterprise Linux release 9.5 (Plow)

[csibbitt@hab-19 ~]$ uname -r
5.14.0-503.21.1.el9_5.x86_64

How did you deploy the dcgm-exporter and what is the configuration?

dcgm-exporter is working on the host, showing the physical GPU stats

[csibbitt@hab-19 ~]$ sudo podman run --replace --name dcgm-exporter --rm -d --device=nvidia.com/gpu=all -p 9400:9400 -e "DCGM_EXPORTER_DEBUG=true" --cap-add SYS_ADMIN nvcr.io/nvidia/k8s/dcgm-exporter:3.3.8-3.6.0-ubi9

[csibbitt@hab-19 ~]$ curl -s localhost:9400/metrics | head -3
# HELP DCGM_FI_DEV_SM_CLOCK SM clock frequency (in MHz).
# TYPE DCGM_FI_DEV_SM_CLOCK gauge
DCGM_FI_DEV_SM_CLOCK{gpu="0",UUID="GPU-4b9f8464-ad90-cdde-8510-8006fc2772b7",pci_bus_id="00000000:04:00.0",device="nvidia0",modelName="NVIDIA A100-PCIE-40GB",Hostname="b1f02cf63072",DCGM_FI_DRIVER_VERSION="565.63"} 210

A100-10C vGPU is working in the guest

[root@localhost ~]# nvidia-smi -q | grep A100
    Product Name                          : GRID A100-10C
[root@localhost ~]# nvidia-smi -q | grep Driver
Driver Version                            : 565.63

How to reproduce the issue?

Change GPU into MIG mode

I shut down the guest and disable the 10C vgpu instance

[root@localhost ~]# shutdown -h now

[csibbitt@hab-19 ~]$ cd /sys/bus/pci/devices/0000:04:00.0/virtfn1/1c854c26-6ee7-491e-b8f9-3d3a8f1016dd
[csibbitt@hab-19 1c854c26-6ee7-491e-b8f9-3d3a8f1016dd]$ sudo su -c "echo 1 > remove"

[csibbitt@hab-19 ~]$ cd /sys/bus/pci/devices/0000:04:00.0/virtfn0/9361957c-95c8-4323-bc93-e437d389802d
[csibbitt@hab-19 9361957c-95c8-4323-bc93-e437d389802d]$ sudo su -c "echo 1 > remove"
[csibbitt@hab-19 9361957c-95c8-4323-bc93-e437d389802d]$ mdevctl list
[csibbitt@hab-19 9361957c-95c8-4323-bc93-e437d389802d]$

dcgm-exporter is still working, but I must shut it down to change to MIG mode: "00000000:04:00.0 is currently being used by one or more other processes (e.g. CUDA application or a monitoring application such as another instance of nvidia-smi)"

[csibbitt@hab-19 ~]$ sudo podman rm -f dcgm-exporter
dcgm-exporter

[csibbitt@hab-19 ~]$ sudo nvidia-smi -i 0 -mig 1
Enabled MIG Mode for GPU 00000000:04:00.0
All done.

I restart dcgm-exporter at this point and notice it still only showing metrics for the main card, not the MIG devices, which is unexpected based on what the docs say.

I create the vGPU mdev device in the MIG instance, thinking this might change things.

[csibbitt@hab-19 ~]$ cd /sys/bus/pci/devices/0000\:04\:00.0/virtfn1/mdev_supported_types/

[csibbitt@hab-19 mdev_supported_types]$  for i in *; do echo "$i" $(cat $i/name) available: $(cat $i/avail*); done
nvidia-1053 GRID A100-1-10C available: 0
nvidia-468 GRID A100-4C available: 0
nvidia-469 GRID A100-5C available: 0
nvidia-470 GRID A100-8C available: 0
nvidia-471 GRID A100-10C available: 0
nvidia-472 GRID A100-20C available: 0
nvidia-473 GRID A100-40C available: 0
nvidia-474 GRID A100-1-5C available: 0
nvidia-475 GRID A100-2-10C available: 1
nvidia-476 GRID A100-3-20C available: 0
nvidia-477 GRID A100-4-20C available: 0
nvidia-478 GRID A100-7-40C available: 0
nvidia-706 GRID A100-1-5CME available: 0

[csibbitt@hab-19 mdev_supported_types]$ sudo su -c "echo c020d7f6-4167-4d41-996e-d45089caf86f > nvidia-475/create"

[csibbitt@hab-19 mdev_supported_types]$ mdevctl list
c020d7f6-4167-4d41-996e-d45089caf86f 0000:04:00.5 nvidia-475 manual

No change in dcgm-exporter, still only showing the main card, so I decide to restart it, and it fails to come up.

[csibbitt@hab-19 mdev_supported_types]$ sudo podman run -it --gpus all -p 9400:9400 -e "DCGM_EXPORTER_DEBUG=true" --cap-add SYS_ADMIN nvcr.io/nvidia/k8s/dcgm-exporter:3.3.8-3.6.0-ubi9
2025/01/16 17:28:07 maxprocs: Leaving GOMAXPROCS=48: CPU quota undefined
INFO[0000] Starting dcgm-exporter                       
DEBU[0000] Debug output is enabled                      
DEBU[0000] Command line: /usr/bin/dcgm-exporter         
DEBU[0000] Loaded configuration                          dump="&{CollectorsFile:/etc/dcgm-exporter/default-counters.csv Address::9400 CollectInterval:30000 Kubernetes:false KubernetesGPUIdType:uid CollectDCP:true UseOldNamespace:false UseRemoteHE:false RemoteHEInfo:localhost:5555 GPUDevices:{Flex:true MajorRange:[] MinorRange:[]} SwitchDevices:{Flex:true MajorRange:[] MinorRange:[]} CPUDevices:{Flex:true MajorRange:[] MinorRange:[]} NoHostname:false UseFakeGPUs:false ConfigMapData:none MetricGroups:[] WebSystemdSocket:false WebConfigFile: XIDCountWindowSize:300000 ReplaceBlanksInModelName:false Debug:true ClockEventsCountWindowSize:300000 EnableDCGMLog:false DCGMLogLevel:NONE PodResourcesKubeletSocket:/var/lib/kubelet/pod-resources/kubelet.sock HPCJobMappingDir: NvidiaResourceNames:[]}"
CacheManager Init Failed. Error: -17
ERRO[0000] Encountered a failure.                        stacktrace="goroutine 1 [running]:\nruntime/debug.Stack()\n\t/usr/local/go/src/runtime/debug/stack.go:24 +0x5e\ngithub.com/NVIDIA/dcgm-exporter/pkg/cmd.action.func1.1()\n\t/go/src/github.com/NVIDIA/dcgm-exporter/pkg/cmd/app.go:283 +0x3d\npanic({0x18b42c0?, 0x2a8d3e0?})\n\t/usr/local/go/src/runtime/panic.go:770 +0x132\ngithub.com/NVIDIA/dcgm-exporter/pkg/cmd.initDCGM(0xc000443180)\n\t/go/src/github.com/NVIDIA/dcgm-exporter/pkg/cmd/app.go:523 +0x9b\ngithub.com/NVIDIA/dcgm-exporter/pkg/cmd.startDCGMExporter(0xc0002f42c0, 0xc0007999e0)\n\t/go/src/github.com/NVIDIA/dcgm-exporter/pkg/cmd/app.go:303 +0xab\ngithub.com/NVIDIA/dcgm-exporter/pkg/cmd.action.func1()\n\t/go/src/github.com/NVIDIA/dcgm-exporter/pkg/cmd/app.go:287 +0x5b\ngithub.com/NVIDIA/dcgm-exporter/pkg/stdout.Capture({0x1dc9e30, 0xc00075eeb0}, 0xc0004abb90)\n\t/go/src/github.com/NVIDIA/dcgm-exporter/pkg/stdout/capture.go:77 +0x1e6\ngithub.com/NVIDIA/dcgm-exporter/pkg/cmd.action(0xc0002f42c0)\n\t/go/src/github.com/NVIDIA/dcgm-exporter/pkg/cmd/app.go:278 +0x67\ngithub.com/NVIDIA/dcgm-exporter/pkg/cmd.NewApp.func1(0xc0002f42c0?)\n\t/go/src/github.com/NVIDIA/dcgm-exporter/pkg/cmd/app.go:263 +0x13\ngithub.com/urfave/cli/v2.(*Command).Run(0xc00075bb80, 0xc0002f42c0, {0xc00012a040, 0x1, 0x1})\n\t/go/pkg/mod/github.com/urfave/cli/[email protected]/command.go:279 +0x97d\ngithub.com/urfave/cli/v2.(*App).RunContext(0xc000440600, {0x1dc9d18, 0x2b8ffe0}, {0xc00012a040, 0x1, 0x1})\n\t/go/pkg/mod/github.com/urfave/cli/[email protected]/app.go:337 +0x58b\ngithub.com/urfave/cli/v2.(*App).Run(0xc0004abf30?, {0xc00012a040?, 0x1?, 0x16dd0b0?})\n\t/go/pkg/mod/github.com/urfave/cli/[email protected]/app.go:311 +0x2f\nmain.main()\n\t/go/src/github.com/NVIDIA/dcgm-exporter/cmd/dcgm-exporter/main.go:35 +0x5f\n"

If I remove the mdev device, dcgm-exporter continues to fail

[csibbitt@hab-19 mdev_supported_types]$ cd ../c020d7f6-4167-4d41-996e-d45089caf86f/
[csibbitt@hab-19 c020d7f6-4167-4d41-996e-d45089caf86f]$ sudo su -c "echo 1 > remove"

[csibbitt@hab-19 ~]$ sudo podman run -it --gpus all -p 9400:9400 -e "DCGM_EXPORTER_DEBUG=true" --cap-add SYS_ADMIN nvcr.io/nvidia/k8s/dcgm-exporter:3.3.8-3.6.0-ubi9
2025/01/16 17:30:41 maxprocs: Leaving GOMAXPROCS=48: CPU quota undefined
INFO[0000] Starting dcgm-exporter                       
DEBU[0000] Debug output is enabled                      
DEBU[0000] Command line: /usr/bin/dcgm-exporter         
DEBU[0000] Loaded configuration                          dump="&{CollectorsFile:/etc/dcgm-exporter/default-counters.csv Address::9400 CollectInterval:30000 Kubernetes:false KubernetesGPUIdType:uid CollectDCP:true UseOldNamespace:false UseRemoteHE:false RemoteHEInfo:localhost:5555 GPUDevices:{Flex:true MajorRange:[] MinorRange:[]} SwitchDevices:{Flex:true MajorRange:[] MinorRange:[]} CPUDevices:{Flex:true MajorRange:[] MinorRange:[]} NoHostname:false UseFakeGPUs:false ConfigMapData:none MetricGroups:[] WebSystemdSocket:false WebConfigFile: XIDCountWindowSize:300000 ReplaceBlanksInModelName:false Debug:true ClockEventsCountWindowSize:300000 EnableDCGMLog:false DCGMLogLevel:NONE PodResourcesKubeletSocket:/var/lib/kubelet/pod-resources/kubelet.sock HPCJobMappingDir: NvidiaResourceNames:[]}"
CacheManager Init Failed. Error: -17
ERRO[0000] Encountered a failure.                        stacktrace="goroutine 1 [running]:\nruntime/debug.Stack()\n\t/usr/local/go/src/runtime/debug/stack.go:24 +0x5e\ngithub.com/NVIDIA/dcgm-exporter/pkg/cmd.action.func1.1()\n\t/go/src/github.com/NVIDIA/dcgm-exporter/pkg/cmd/app.go:283 +0x3d\npanic({0x18b42c0?, 0x2a8d3e0?})\n\t/usr/local/go/src/runtime/panic.go:770 +0x132\ngithub.com/NVIDIA/dcgm-exporter/pkg/cmd.initDCGM(0xc000145180)\n\t/go/src/github.com/NVIDIA/dcgm-exporter/pkg/cmd/app.go:523 +0x9b\ngithub.com/NVIDIA/dcgm-exporter/pkg/cmd.startDCGMExporter(0xc000135f40, 0xc000041140)\n\t/go/src/github.com/NVIDIA/dcgm-exporter/pkg/cmd/app.go:303 +0xab\ngithub.com/NVIDIA/dcgm-exporter/pkg/cmd.action.func1()\n\t/go/src/github.com/NVIDIA/dcgm-exporter/pkg/cmd/app.go:287 +0x5b\ngithub.com/NVIDIA/dcgm-exporter/pkg/stdout.Capture({0x1dc9e30, 0xc00012ed20}, 0xc000517b90)\n\t/go/src/github.com/NVIDIA/dcgm-exporter/pkg/stdout/capture.go:77 +0x1e6\ngithub.com/NVIDIA/dcgm-exporter/pkg/cmd.action(0xc000135f40)\n\t/go/src/github.com/NVIDIA/dcgm-exporter/pkg/cmd/app.go:278 +0x67\ngithub.com/NVIDIA/dcgm-exporter/pkg/cmd.NewApp.func1(0xc000135f40?)\n\t/go/src/github.com/NVIDIA/dcgm-exporter/pkg/cmd/app.go:263 +0x13\ngithub.com/urfave/cli/v2.(*Command).Run(0xc0001198c0, 0xc000135f40, {0xc000040090, 0x1, 0x1})\n\t/go/pkg/mod/github.com/urfave/cli/[email protected]/command.go:279 +0x97d\ngithub.com/urfave/cli/v2.(*App).RunContext(0xc00013a600, {0x1dc9d18, 0x2b8ffe0}, {0xc000040090, 0x1, 0x1})\n\t/go/pkg/mod/github.com/urfave/cli/[email protected]/app.go:337 +0x58b\ngithub.com/urfave/cli/v2.(*App).Run(0xc000517f30?, {0xc000040090?, 0x1?, 0x16dd0b0?})\n\t/go/pkg/mod/github.com/urfave/cli/[email protected]/app.go:311 +0x2f\nmain.main()\n\t/go/src/github.com/NVIDIA/dcgm-exporter/cmd/dcgm-exporter/main.go:35 +0x5f\n"

If I disable MIG mode, dcgm-exporter works again

[csibbitt@hab-19 ~]$ sudo nvidia-smi -i 0 -mig 0
Disabled MIG Mode for GPU 00000000:04:00.0
All done.

[csibbitt@hab-19 ~]$ sudo podman run -d --rm --gpus all -p 9400:9400 -e "DCGM_EXPORTER_DEBUG=true" --cap-add SYS_ADMIN nvcr.io/nvidia/k8s/dcgm-exporter:3.3.8-3.6.0-ubi9
df103d8f293453aa5a38dfac3cf6c86805a1a32f69a26e8bcd1940c58919273f

[csibbitt@hab-19 ~]$ curl -s localhost:9400/metrics | head -3
# HELP DCGM_FI_DEV_SM_CLOCK SM clock frequency (in MHz).
# TYPE DCGM_FI_DEV_SM_CLOCK gauge
DCGM_FI_DEV_SM_CLOCK{gpu="0",UUID="GPU-4b9f8464-ad90-cdde-8510-8006fc2772b7",pci_bus_id="00000000:04:00.0",device="nvidia0",modelName="NVIDIA A100-PCIE-40GB",Hostname="df103d8f2934",DCGM_FI_DRIVER_VERSION="565.63"} 210

Anything else we need to know?

No response
@csibbitt csibbitt added the bug Something isn't working label Jan 16, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@csibbitt