C2MIR high memory usage when executing sqlite3

[mingodad]

Testing c2mir with sqlite3 after minor fixes (see bellow) I'm getting this results:

• sqlite3 native : 4.6 MB
• sqlite3 c2mir : 584 MB

Native sqlite3:

echo 'select 2 as two;' | /usr/bin/time sqlite3
2
0.00user 0.00system 0:00.00elapsed 100%CPU (0avgtext+0avgdata 4588maxresident)k
0inputs+0outputs (0major+196minor)pagefaults 0swaps

c2mir sqlite3:

echo 'select 2 as two;' | sh run-sqlite3.sh
sqlite3.c:35613:19: warning -- number 1e10000f is out of range -- using IEEE infinity
shell.c:21754:37: warning -- number 1e10000f is out of range -- using IEEE infinity
2
2.44user 0.31system 0:02.75elapsed 99%CPU (0avgtext+0avgdata 584216maxresident)k
0inputs+0outputs (0major+171348minor)pagefaults 0swaps

run-sqlite3.sh:

#!/bin/sh
#need remove this cast { (u8*)" " }; and coment "atexit" in shell.c
sqliteh=.
/usr/bin/time ./c2m  \
	-DSQLITE_OS_UNIX=1 \
	-DBUILD_sqlite \
	-DNDEBUG \
	-DSQLITE_THREADSAFE=0 \
	-DSQLITE_ENABLE_MATH_FUNCTIONS \
	-DSQLITE_ENABLE_COLUMN_METADATA=1 \
	-DSQLITE_ENABLE_FTS4 \
	-DSQLITE_ENABLE_RTREE \
	-DSQLITE_ENABLE_EXPLAIN_COMMENTS \
	-DSQLITE_ENABLE_UNKNOWN_SQL_FUNCTION \
	-DSQLITE_ENABLE_STMTVTAB \
	-DSQLITE_ENABLE_DBPAGE_VTAB \
	-DSQLITE_ENABLE_DBSTAT_VTAB \
	-DSQLITE_ENABLE_BYTECODE_VTAB \
	-DSQLITE_ENABLE_OFFSET_SQL_FUNC \
	-D_FILE_OFFSET_BITS \
	-O2 -I$sqliteh $sqliteh/sqlite3.c $sqliteh/shell.c \
	 -ei

Fixes for sqlite3.c:

--- <unnamed>
+++ <unnamed>
@@ -130362,7 +130362,7 @@
   assert( zIn==sqlite3_value_text(argv[0]) );
   if( argc==1 ){
     static const unsigned lenOne[] = { 1 };
-    static unsigned char * const azOne[] = { (u8*)" " };
+    static unsigned char * const azOne[] = { /*(u8*)*/" " };
     nChar = 1;
     aLen = (unsigned*)lenOne;
     azChar = (unsigned char **)azOne;

Fixes for shell.c:

--- <unnamed>
+++ <unnamed>
@@ -30467,9 +30467,9 @@
   consStreams = consoleClassifySetup(stdin, stdout, stderr);
   stdin_is_interactive = (consStreams & SAC_InConsole)!=0;
   stdout_is_console = (consStreams & SAC_OutConsole)!=0;
-  atexit(consoleRestore);
+  //atexit(consoleRestore);
 #endif
-  atexit(sayAbnormalExit);
+  //atexit(sayAbnormalExit);
 #ifdef SQLITE_DEBUG
   mem_main_enter = sqlite3_memory_used();
 #endif

[rofl0r]

i'd say for a fair comparison you should compare the execution of the pre-compiled bytecode, or conversely the compilation of sqlite3.c with gcc plus the sqlite execution vs your existing c2m test.

[mingodad]

For comparison I've tested build with other compilers:
gcc-9.4 375 MB :

sh build-sqlite3.sh
30.44user 0.34system 0:30.79elapsed 99%CPU (0avgtext+0avgdata 375212maxresident)k
0inputs+3024outputs (0major+155901minor)pagefaults 0swaps

gcc-14 390 MB:

sh build-sqlite3.sh
41.69user 0.37system 0:42.23elapsed 99%CPU (0avgtext+0avgdata 390040maxresident)k
67992inputs+3104outputs (284major+162290minor)pagefaults 0swaps

clang-18 245 MB:

sh build-sqlite3.sh
32.18user 0.25system 0:32.80elapsed 98%CPU (0avgtext+0avgdata 245780maxresident)k
187224inputs+3552outputs (685major+68021minor)pagefaults 0swaps

tinycc 19 MB :

sh build-sqlite3.sh
0.14user 0.01system 0:00.15elapsed 96%CPU (0avgtext+0avgdata 18540maxresident)k
904inputs+3896outputs (2major+5204minor)pagefaults 0swaps

build-sqlite3.sh

#!/bin/sh
sqliteh=sqlite3-orig
MYCC="gcc"
#MYCC="gcc-14-env gcc"
#MYCC="clang-18-env clang"
#MYCC="tinycc-env tcc"
/usr/bin/time  $MYCC \
	-DSQLITE_OS_UNIX=1 \
	-DBUILD_sqlite \
	-DNDEBUG \
	-DSQLITE_THREADSAFE=0 \
	-DSQLITE_ENABLE_MATH_FUNCTIONS \
	-DSQLITE_ENABLE_COLUMN_METADATA=1 \
	-DSQLITE_ENABLE_FTS4 \
	-DSQLITE_ENABLE_RTREE \
	-DSQLITE_ENABLE_EXPLAIN_COMMENTS \
	-DSQLITE_ENABLE_UNKNOWN_SQL_FUNCTION \
	-DSQLITE_ENABLE_STMTVTAB \
	-DSQLITE_ENABLE_DBPAGE_VTAB \
	-DSQLITE_ENABLE_DBSTAT_VTAB \
	-DSQLITE_ENABLE_BYTECODE_VTAB \
	-DSQLITE_ENABLE_OFFSET_SQL_FUNC \
	-D_FILE_OFFSET_BITS \
	-O2 -I$sqliteh $sqliteh/sqlite3.c $sqliteh/shell.c \
	 -o sqlite3 -lm -ldl

[mingodad]

Building and running together:
tinycc 0.13 s 19 MB :

/usr/bin/time sh build-sqlite3.sh
0.13user 0.02system 0:00.16elapsed 99%CPU (0avgtext+0avgdata 18760maxresident)k
0inputs+3896outputs (0major+5202minor)pagefaults 0swaps
2
0.00user 0.00system 0:00.00elapsed 100%CPU (0avgtext+0avgdata 4444maxresident)k
0inputs+0outputs (0major+169minor)pagefaults 0swaps
0.13user 0.02system 0:00.16elapsed 99%CPU (0avgtext+0avgdata 18760maxresident)k
0inputs+3896outputs (0major+5641minor)pagefaults 0swaps

clang-18 31.68 s 246 MB :

sh build-sqlite3.sh
31.68user 0.16system 0:31.85elapsed 99%CPU (0avgtext+0avgdata 246488maxresident)k
0inputs+3552outputs (0major+68262minor)pagefaults 0swaps
2
0.00user 0.00system 0:00.00elapsed 100%CPU (0avgtext+0avgdata 3972maxresident)k
0inputs+0outputs (0major+159minor)pagefaults 0swaps
31.68user 0.16system 0:31.85elapsed 99%CPU (0avgtext+0avgdata 246488maxresident)k
0inputs+3552outputs (0major+68697minor)pagefaults 0swaps

gcc-9.4 30.31 s 375 MB :

sh build-sqlite3.sh
30.31user 0.37system 0:30.68elapsed 100%CPU (0avgtext+0avgdata 375176maxresident)k
0inputs+3024outputs (0major+155879minor)pagefaults 0swaps
2
0.00user 0.00system 0:00.00elapsed 100%CPU (0avgtext+0avgdata 3660maxresident)k
0inputs+0outputs (0major+157minor)pagefaults 0swaps
30.31user 0.37system 0:30.68elapsed 100%CPU (0avgtext+0avgdata 375176maxresident)k
0inputs+3024outputs (0major+156308minor)pagefaults 0swaps

gcc-14 41.06 s 390 MB :

sh build-sqlite3.sh
41.06user 0.34system 0:41.41elapsed 99%CPU (0avgtext+0avgdata 390120maxresident)k
0inputs+3104outputs (0major+160098minor)pagefaults 0swaps
2
0.00user 0.00system 0:00.00elapsed 100%CPU (0avgtext+0avgdata 3644maxresident)k
0inputs+0outputs (0major+152minor)pagefaults 0swaps
41.06user 0.34system 0:41.41elapsed 99%CPU (0avgtext+0avgdata 390120maxresident)k
0inputs+3104outputs (0major+160523minor)pagefaults 0swaps

c2mir 2.44 s 584MB :

echo 'select 2 as two;' | sh run-sqlite3.sh
sqlite3.c:35613:19: warning -- number 1e10000f is out of range -- using IEEE infinity
shell.c:21754:37: warning -- number 1e10000f is out of range -- using IEEE infinity
2
2.44user 0.31system 0:02.75elapsed 99%CPU (0avgtext+0avgdata 584216maxresident)k
0inputs+0outputs (0major+171348minor)pagefaults 0swaps

#!/bin/sh
sqliteh=sqlite3-orig
#MYCC="gcc"
#MYCC="gcc-14-env gcc"
#MYCC="clang-18-env clang"
MYCC="tinycc-env tcc"
/usr/bin/time  $MYCC \
	-DSQLITE_OS_UNIX=1 \
	-DBUILD_sqlite \
	-DNDEBUG \
	-DSQLITE_THREADSAFE=0 \
	-DSQLITE_ENABLE_MATH_FUNCTIONS \
	-DSQLITE_ENABLE_COLUMN_METADATA=1 \
	-DSQLITE_ENABLE_FTS4 \
	-DSQLITE_ENABLE_RTREE \
	-DSQLITE_ENABLE_EXPLAIN_COMMENTS \
	-DSQLITE_ENABLE_UNKNOWN_SQL_FUNCTION \
	-DSQLITE_ENABLE_STMTVTAB \
	-DSQLITE_ENABLE_DBPAGE_VTAB \
	-DSQLITE_ENABLE_DBSTAT_VTAB \
	-DSQLITE_ENABLE_BYTECODE_VTAB \
	-DSQLITE_ENABLE_OFFSET_SQL_FUNC \
	-D_FILE_OFFSET_BITS \
	-O2 -I$sqliteh $sqliteh/sqlite3.c $sqliteh/shell.c \
	 -o sqlite3 -lm -ldl

echo "select 2 as two;" | /usr/bin/time ./sqlite3

[mingodad]

Also building with https://github.com/johnsonjh/pcc-revived/tree/johnsonjh/rawhide :
pcc-revived-rawhide 3.21 s 394MB :

sh build-sqlite3.sh
3.21user 0.20system 0:03.41elapsed 100%CPU (0avgtext+0avgdata 39460maxresident)k
0inputs+3088outputs (0major+40167minor)pagefaults 0swaps
2
0.00user 0.00system 0:00.00elapsed 100%CPU (0avgtext+0avgdata 3956maxresident)k
0inputs+0outputs (0major+160minor)pagefaults 0swaps
3.21user 0.20system 0:03.41elapsed 99%CPU (0avgtext+0avgdata 39460maxresident)k
0inputs+3088outputs (0major+40600minor)pagefaults 0swaps

[mingodad]

Also building with https://github.com/stormalf/chibicc :
chibcc 4.1 s 615 MB :

/usr/bin/time sh build-sqlite3.sh
4.09user 0.58system 0:04.68elapsed 99%CPU (0avgtext+0avgdata 614732maxresident)k
0inputs+10400outputs (0major+294027minor)pagefaults 0swaps
2
0.00user 0.00system 0:00.00elapsed 100%CPU (0avgtext+0avgdata 5816maxresident)k
0inputs+0outputs (0major+199minor)pagefaults 0swaps
4.10user 0.58system 0:04.68elapsed 100%CPU (0avgtext+0avgdata 614732maxresident)k
0inputs+10400outputs (0major+294695minor)pagefaults 0swaps

[fwsGonzo]

I'm measuring thousands of heap allocations for a simple 2-line C function using c2mir. I'm also having problem resetting it afterwards. I don't mind the usage, but there doesn't seem to be a graceful way to reset at all. After compiling using c2mir the second time it grows beyond 4000 heap allocations.

No amount of MIR_*_finish that I have tried has been able to reduce the number of heap allocations back to less than 4000.
Are there any examples of how to continously compile C code without memory growth?

I hope I'm not "hijacking" the wrong thread here. I saw that this had to do with memory usage. If so, I can create a new issue instead.

[fwsGonzo]

I think we can conclude my comment: It's because Mir doesn't clean up if compilation fails with c2mir. If compilation succeeds, you can compile again. Memory usage and cleanup is a real problem, but less so on the happy path.