Getting error in step "step_set_fifo_depths" when running on FINN 0.9

[pkdeep]

I am facing the issue when trying to run pgs_flow in the tutorial section (FINN version 0.9). When I run the example build file "build.py". (I am running the fpga_flow in tutorial section)

The error is the following:

Traceback (most recent call last):
  File "/home/neurolab3/pradeep/finn-brevitas/finn/src/finn/builder/build_dataflow.py", line 171, in build_dataflow_cfg
    model = transform_step(model, cfg)
  File "/home/neurolab3/pradeep/finn-brevitas/finn/src/finn/builder/build_dataflow_steps.py", line 603, in step_set_fifo_depths
    model = model.transform(
  File "/home/neurolab3/pradeep/finn-brevitas/finn/deps/qonnx/src/qonnx/core/modelwrapper.py", line 140, in transform
    (transformed_model, model_was_changed) = transformation.apply(transformed_model)
  File "/home/neurolab3/pradeep/finn-brevitas/finn/src/finn/transformation/fpgadataflow/set_fifo_depths.py", line 394, in apply
    sim = verilator_fifosim(model, n_inputs)
  File "/home/neurolab3/pradeep/finn-brevitas/finn/src/finn/util/pyverilator.py", line 255, in verilator_fifosim
    with open(build_dir + "/results.txt", "r") as f:
FileNotFoundError: [Errno 2] No such file or directory: '/home/neurolab3/pradeep/finn-brevitas/finn/tutorials/finn_build//verilator_fifosim_gk180w6j/results.txt'
> /home/neurolab3/pradeep/finn-brevitas/finn/src/finn/util/pyverilator.py(255)verilator_fifosim()

I am getting this error when running the default example of tfc_w1a1, as covered in the default build.py script.

I have tried the same with version 10.0, and everything works fine. However, FINN v0.9 does give error. Earlier I was able to run and even implement my custom model using this build flow.

The error happens in step "step_set_fifo_depths". It seems the Vfinn wrapper built is not able to run. When I try to run it outside of the build steps (both outside and inside docker), I get the following error:

./Vfinn_design_wrapper
*** stack smashing detected ***: <unknown> terminated
Aborted (core dumped)

Below is the docker image (output of "docker ps")

xilinx/finn:v0.9-2-gb3bdff11-dirty.xrt_202210.2.13.466_18.04-amd64-xrt

Thanks
Pradeep

[bwintermann]

If I remember correctly, this error was encountered before, and it had to do with Verilator requiring too much memory or requiring too many / fast file accesses (right, @LinusJungemann ?). We solved it by moving the entire FINN project directory to ramdisk, so this could be a solution if you have enough RAM available.

[LinusJungemann]

Yes, moving everything to ramdisk circumvents the error. But we don't really know why. So it is probably not a bad idea that somebody investigates this error in more detail.

[pkdeep]

Thanks for the reply @bwintermann and @LinusJungemann .

I could not understand what's involved in "moving the entire FINN project directory to ramdisk".

Also I tried to run the same steps (on same network) on a different PC with the same HW configuration, and I did not get any error. So I guess that there is some dependency, which does not cause any error during Vfinn wrapper compilation, but throws an error while running this Vfinn wrapper. I may need to dig deeper to figure out exactly which dependency is causing the error.

[bwintermann]

What we meant was that you would basically move the FINN folder and everything else (your model and build scripts for example) to RAM, using it as you would use your hard drive. But, as Linus mentioned, this was only the workaround, not the solution.

Have you tried changing the compilation command in FINN to enable debug symbols and then using gdb to look where it crashes?

[pkdeep]

@bwintermann Thanks for the reply. I'll try to include the debug symbols and see if I can find any useful information.
Will post once I am able to find something useful.