The relationship between linker rom/ram size and VHDL IMEM/DMEM size

[Sam-Vervaeck]

Background:

I am playing around with the ram size variable of the linker script and the IMEM/DMEM size variable in the simple TB when running the hello world demo. I can't fully wrap my head around the behaviour I am seeing, so asking here for a small clarification. The reason I am doing these tests is because I want to create my own program that has large global variables and I am figuring out where these exactly get put in memory and what size that memory should be.

Question:

When I compile hello_world with make USER_FLAGS+="-DUART0_SIM_MODE -Wl,--defsym,__neorv32_ram_size=128*1024" clean_all install and when I set MEM_INT_IMEM_SIZE => 128*1024, MEM_INT_DMEM_SIZE => 128*1024. It runs perfectly fine and does the correct prints in the tcl console.

When I compile hello_world with make USER_FLAGS+="-DUART0_SIM_MODE -Wl,--defsym,__neorv32_ram_size=1M" clean_all install and when I set MEM_INT_IMEM_SIZE => 128*1024, MEM_INT_DMEM_SIZE => 128*1024. It does not print anything any more.

However, for both compile styles the output is the same:

Memory utilization:
   text	   data	    bss	    dec	    hex	filename
   5444	      0	    116	   5560	   15b8	main.elf

And the sim gives the same logs:

# run 1000ns
Note: [NEORV32] OCD DM compatible to debug spec. version 1.0
Time: 0 ps  Iteration: 0  Process: /neorv32_tb_simple/neorv32_top_inst/neorv32_neorv32_ocd_inst_true/neorv32_debug_dm_inst/line__196  File: /home/sam/Documents/thesis-defense/thesis_neorv32/vivado_integrated_design_genesys/vivado_integrated_design_genesys.srcs/sources_1/imports/rtl/core/neorv32_debug_dm.vhd
Warning: [neoTRNG] Simulation-mode enabled!
Time: 0 ps  Iteration: 0  Process: /neorv32_tb_simple/neorv32_top_inst/io_system/neorv32_trng_inst_true/neorv32_trng_inst/neoTRNG_inst/line__311  File: /home/sam/Documents/thesis-defense/thesis_neorv32/vivado_integrated_design_genesys/vivado_integrated_design_genesys.srcs/sources_1/imports/rtl/core/neorv32_trng.vhd
Note: [neoTRNG] The neoTRNG (v3.1) - A Tiny and Platform-Independent True Random Number Generator, https://github.com/stnolting/neoTRNG
Time: 0 ps  Iteration: 0  Process: /neorv32_tb_simple/neorv32_top_inst/io_system/neorv32_trng_inst_true/neorv32_trng_inst/neoTRNG_inst/line__301  File: /home/sam/Documents/thesis-defense/thesis_neorv32/vivado_integrated_design_genesys/vivado_integrated_design_genesys.srcs/sources_1/imports/rtl/core/neorv32_trng.vhd
Note: [NEORV32] Implementing DEFAULT processor-internal IMEM as pre-initialized ROM.
Time: 0 ps  Iteration: 0  Process: /neorv32_tb_simple/neorv32_top_inst/memory_system/neorv32_int_imem_inst_true/neorv32_int_imem_inst/line__58  File: /home/sam/Documents/thesis-defense/thesis_neorv32/vivado_integrated_design_genesys/vivado_integrated_design_genesys.srcs/sources_1/imports/rtl/core/neorv32_imem.vhd
Warning: [NEORV32] Clock gating enabled (using generic clock switch).
Time: 0 ps  Iteration: 0  Process: /neorv32_tb_simple/neorv32_top_inst/core_complex/neorv32_cpu_clockgate_inst_true/neorv32_cpu_clockgate_inst/line__35  File: /home/sam/Documents/thesis-defense/thesis_neorv32/vivado_integrated_design_genesys/vivado_integrated_design_genesys.srcs/sources_1/imports/rtl/core/neorv32_clockgate.vhd
Warning: [NEORV32] Assuming this is a simulation.
Time: 0 ps  Iteration: 0  Process: /neorv32_tb_simple/neorv32_top_inst/core_complex/neorv32_cpu_inst/line__152  File: /home/sam/Documents/thesis-defense/thesis_neorv32/vivado_integrated_design_genesys/vivado_integrated_design_genesys.srcs/sources_1/imports/rtl/core/neorv32_cpu.vhd
Note: [NEORV32] CPU tuning options: fast_mul fast_shift 
Time: 0 ps  Iteration: 0  Process: /neorv32_tb_simple/neorv32_top_inst/core_complex/neorv32_cpu_inst/line__145  File: /home/sam/Documents/thesis-defense/thesis_neorv32/vivado_integrated_design_genesys/vivado_integrated_design_genesys.srcs/sources_1/imports/rtl/core/neorv32_cpu.vhd
Note: [NEORV32] CPU ISA: rv32imabux_zicsr_zicntr_zicond_zifencei_zfinx_zihpm_zxcfu_sdext_sdtrig_smpmp
Time: 0 ps  Iteration: 0  Process: /neorv32_tb_simple/neorv32_top_inst/core_complex/neorv32_cpu_inst/line__123  File: /home/sam/Documents/thesis-defense/thesis_neorv32/vivado_integrated_design_genesys/vivado_integrated_design_genesys.srcs/sources_1/imports/rtl/core/neorv32_cpu.vhd
Note: [NEORV32] Processor Configuration: IMEM DMEM I-CACHE D-CACHE XBUS X-CACHE XIP XIP-CACHE GPIO MTIME UART0 UART1 SPI SDI TWI PWM WDT TRNG CFS NEOLED XIRQ GPTMR ONEWIRE DMA SLINK CRC SYSINFO OCD 
Time: 0 ps  Iteration: 0  Process: /neorv32_tb_simple/neorv32_top_inst/sanity_checks/line__326  File: /home/sam/Documents/thesis-defense/thesis_neorv32/vivado_integrated_design_genesys/vivado_integrated_design_genesys.srcs/sources_1/imports/rtl/core/neorv32_top.vhd
Note: [NEORV32] The NEORV32 RISC-V Processor (v1.10.2.7INFO: [USF-XSim-96] XSim completed. Design snapshot 'neorv32_tb_simple_behav' loaded.
INFO: [USF-XSim-97] XSim simulation ran for 1000ns
launch_simulation: Time (s): cpu = 00:00:21 ; elapsed = 00:00:19 . Memory (MB): peak = 6875.707 ; gain = 0.000 ; free physical = 5764 ; free virtual = 12117
run 2000 us

So why does the reconfiguring of __neorv32_ram_size matter? Shouldn't the linker ram section fit inside the processor DMEM in both cases?

[Sam-Vervaeck]

A follow-up question is about the problems with executing the following main file in simulation:

int main()
{

  // capture all exceptions and give debug info via UART
  // this is not required, but keeps us safe
  neorv32_rte_setup();

  // setup UART at default baud rate, no interrupts
  neorv32_uart0_setup(BAUD_RATE, 0);

  neorv32_uart0_puts("Hello world! :)\n");

  // check if UART0 unit is implemented at all
  if (neorv32_uart0_available() == 0) {
    return 1;
  }

  // intro
  neorv32_uart0_printf("\n<<< CKKS MAIN FILE BEGIN >>>\n\n");

  size_t sz = sizeof(test) / sizeof(test[0]);
  for (size_t i = 0; i < sz; ++i) {
    neorv32_uart0_printf("%x%x\n", (uint32_t)(test[i]>>32), (uint32_t)(test[i]));
  }

  neorv32_uart0_printf("\n<<< CKKS MAIN FILE END >>>\n\n");

  // stop program counter
  while ( 1 ) { }
  return 0;
}

with the test array located in test.h and of size uint64_t test[8193].

When simulating this, nothing gets printed any more. The compilation of this file results in a larger binary:

Memory utilization:
   text	   data	    bss	    dec	    hex	filename
   4484	  65544	    116	  70144	  11200	main.elf

But that should not be a problem because my IMEM, DMEM, and __neorv32_ram_size are set to 128*1024? I even upped it to 256*1024 once and it still did not print anything.

However, when I change the array size to uint64_t test[6] with the same memory sizes of 128*1024 then it gets printed ok:

Hello world! :)

<<< CKKS MAIN FILE BEGIN >>>

3ff554a536955295
4009c69301ba4150
3ff903b89c240ee2
3ff225178992802b
3ff7886d09de21b4
3ff0795622d66ed1

<<< CKKS MAIN FILE END >>>

I really don't get what I am missing here...

[NikLeberg]

However, if you add const qualifier, then the copying can be omited. I.e. write const uint64_t test[] __attribute__((aligned(8192))) = {....

It then does not need to copy the values from ROM to RAM and instead can use them directly from ROM:

Hello world! :)

<<< CKKS MAIN FILE BEGIN >>>

3ff554a536955295
4009c69301ba4150
3ff903b89c240ee2
3ff225178992802b
3ff7886d09de21b4
3ff0795622d66ed1
3ff87d8cad21f633
3ff6ef932c9f1edf
3fb9ed1fdc67b47f
3ff04a48069eef99
3fddcf6be4773db0
40060b195ea939f7
3ffb0dd91cec3764
^C

[Sam-Vervaeck]

Have you ever ran this in real world? Remember in simulation we are significantly slower. Try to let the simulation run longer, by a lot...

I am focusing on simulation first before putting it on a board. Damn, I had already ran it for 10x times as long compared to the print of the hello_world example (2000 us instead of 200us). But I guess that was still not enough...

However, if you add const qualifier, then the copying can be omited. I.e. write const uint64_t test[] attribute((aligned(8192))) = {....

Ah, thank you for this suggestion! If I understand correctly, the data is then part of the .rodata section and that is why it stays in rom when looking at the linker?

Since these are testing values I can def fix them in place.

[Sam-Vervaeck]

It's nicely running now, thank you so much for your time. Now I can move on to putting useful stuff in the main file :)

[NikLeberg]

If I understand correctly, the data is then part of the .rodata section and that is why it stays in rom when looking at the linker?

Yes. const tells the compiler that you will never change the value. And as such the compiler is allowed to put it in ROM (read only memory).

It's nicely running now, thank you so much for your time. Now I can move on to putting useful stuff in the main file :)

No problem! Nice to know that it's working for you now! If you get your accelerator up and running, feel free to show & tell it here :)

[Sam-Vervaeck]

I will! It is supposed to be an open-source design :)

[NikLeberg]

Hi @Sam-Vervaeck !

However, for both compile styles the output is the same:

Memory utilization:
   text	   data	    bss	    dec	    hex	filename
   5444	      0	    116	   5560	   15b8	main.elf

On first glance it may appear like that. The size of the executables (or in fact the different sections of it) are the same, yes. But what is different is where the linker puts stuff at runtime.

If you check the output of the make asm task for each setting and compare the produced assembly, then you will notice that the only difference is __crt0_stack_end. I.e. the start of the stack.

diff --git a/sw/example/hello_world/main.asm b/sw/example/hello_world/main.asm
index 12815883..4dbbb6fd 100644
--- a/sw/example/hello_world/main.asm
+++ b/sw/example/hello_world/main.asm
@@ -14,8 +14,8 @@ Disassembly of section .text:
   18:	30401073          	csrw	mie,zero
 
 0000001c <__crt0_pointer_init>:
-  1c:	80020217          	auipc	tp,0x80020
-  20:	fe320213          	addi	tp,tp,-29 # 8001ffff <__crt0_stack_end>
+  1c:	80100217          	auipc	tp,0x80100
+  20:	fe320213          	addi	tp,tp,-29 # 800fffff <__crt0_stack_end>
   24:	ffc27113          	andi	sp,tp,-4
   28:	80000297          	auipc	t0,0x80000
   2c:	7d828293          	addi	t0,t0,2008 # 80000800 <__global_pointer$>
@@ -158,7 +158,7 @@ int main() {

The symbol __crt0_stack_end is defined in the sw/common/neorv32.ld linker script as:

PROVIDE(__crt0_stack_end           = (ORIGIN(ram) + LENGTH(ram)) - 1);

Where (usually) the LENGTH(ram) resolves to __neorv32_ram_size. I.e. the stackpointer is always put at the very end of RAM. It then grows downward and in a case of stack-overflow spills over to the heap. So by changing __neorv32_ram_size you change where the stack goes.

The executable lives in the ROM i.e. IMEM, and the stack lives in RAM i.e. DMEM. You changed __neorv32_ram_size and MEM_INT_IMEM_SIZE which do not relate to eachother. Misread that.

When changing __neorv32_ram_size you need to change MEM_INT_DMEM_SIZE to the same (or bigger) value. If the software is configured with an erroreously large ram (like with __neorv32_ram_size=1M) but the hardware provides only less than that (you did configure MEM_INT_DMEM_SIZE => 128*1024, correct would be MEM_INT_DMEM_SIZE => 1024*1024) , then a bus access error trap is produced and your CPU goes a bit haywire. (With an invalid stackpointer, nothing really works as it should.)

Thats why nothing is printed.

To fix this, set __neorv32_ram_size and MEM_INT_DMEM_SIZE to the same value.

[Sam-Vervaeck]

When changing __neorv32_ram_size you need to change MEM_INT_DMEM_SIZE to the same (or bigger) value. If the software is configured with an erroreously large ram (like with __neorv32_ram_size=1M) but the hardware provides only less than that (you did configure MEM_INT_DMEM_SIZE => 1281024, correct would be MEM_INT_DMEM_SIZE => 10241024) , then a bus access error trap is produced and your CPU goes a bit haywire. (With an invalid stackpointer, nothing really works as it should.)

I understand, why do you refer to 1M as erroreously? Is it not allowed to have such a large ram as long as DMEM is as big?

Thanks for the rest of your explanation, now I know the reason for the error! I'm gonna have a look if I have some extra time to find the bus error trap in the waveform.

To fix this, set __neorv32_ram_size and MEM_INT_DMEM_SIZE to the same value.

For my follow-up question I did this, that's why I do not get why it does not work with that large array. I suppose something else is going wrong there...

EDIT: I see you were looking at it at the same time as my response here, let's continue the discussion under your answer to the follos-up thread :)

[NikLeberg]

I understand, why do you refer to 1M as erroreously? Is it not allowed to have such a large ram as long as DMEM is as big?

Of course, one megabyte is allowed as RAMn and perfectly fine. But setting __neorv32_ram_size to 1M and then only configuring the hardware with MEM_INT_DMEM_SIZE => 128*1024 is the erroreously part.

[stnolting]

@NikLeberg thanks for clearing this! :)