May I ask how did you get the eclmaps?

[tabethereal]

I appreciate your tool. But what excites me most is the full-supplied ecl maps. Actually I was having a hard time mapping the ecls on my own. So do you work out these ecl maps on your own? This is just awesome!

[ExpHP]

Okay so. I had a much greater involvement of the creation of ANM, STD, and MSG maps than for ECL.

• ECL maps already existed for many games at https://github.com/Priw8/eclmap
• there were also massive contributions from @zero318 , (e.g. pretty much all of TH07 and TH08)

Our understanding of the ECL functions is the cumulative results of generations of people's research into the game code. (unsurprisingly, ECL is the language that most people have taken a crack at reverse engineering XD)

As for ANM, STD, and MSG, well:

Short version

I fully reverse engineered as much as I possibly could of the full binary code of the ANM, STD, and MSG interpreters. Not only did I map out the instruction names and signatures, but I also reverse engineered the data structures for the script virtual machines, labeled where the instructions all start in the code and named many of the functions in the code.

All of these tasks - the struct labeling, the mapfile creation, the function labeling - were all done in tandem. Each correction made to the data structure made it easier to name the instructions, and likewise each labeled instruction made it easier to locate the struct fields. So it was a little bit of A, little bit of B, little bit of A sort of deal.

• Data structures, instruction labels, and function labels are published at https://github.com/exphp-share/th-re-data/
• Documentation for the instructions are published at https://exphp.github.io/thpages/#/

Reverse Engineering Process

I did all of this mostly by myself by staring at the code in binary ninja.

My general workflow for reversing a new game was as follows:

• Locate the function that interprets the script. (ANM, STD, ECL, MSG all have separate functions)
• Find the big jumptable in that function.
• Use binja plugins I wrote to copy data as a starting template from one game to another.
  • e.g. to reverse engineer TH13 ANM I would start by importing data I already had on TH14 ANM
  • I use this to import the data structures.
  • I use this to generate code labels from a mapfile and the jumptable.

So then I'd have, e.g. for ANM, a zAnmVm struct and a bunch of anm__case_* code labels.

Because ZUN frequently inserts new fields in the middle of a struct, this would usually result in a bunch of things being labeled incorrectly in the code. So, I would go down the list of anm__case_* labels, repeatedly asking myself:

• Does the code for this instruction labeled "flipX"...
  • ...look like what I expect flipX to look like?
  • ...access the same number of arguments as I have recorded for that instruction's signature?
• If not, what is it actually doing?
  • If this is a new instruction, update the mapfile accordingly and regenerate the code labels.
• Looking at the code, are the automatic field annotations by binja showing all the field names I expect to see?
  • If not, ZUN probably inserted a field somewhere in the middle of the struct.
  • Look at the assembly for any new instructions. These likely will tell you where the new fields are.
  • Still can't find where the inserted/deleted field is? Go down the field list from the top and look at the Cross References for each field, comparing to another game. The first offset where the crossrefs look very different between the two games is probably the location of a new or deleted field.

and I would iterate on this until I felt that all of the differences between the two games were accounted for, and then move onto the next game.

[tabethereal]

Wow thats quite a great work. And thanks for your detailed reply. I've noticed Priw8's work about eclmaps, however there's a lack of th06 and th07 right now. So I found this one: https://pytouhou.linkmauve.fr/doc/06/ecl.xml. This is actually a doc for pytouhou but it does work as an eclmap and provides sufficient function explanations. Besides this I mapped several functions in th07 by myself, based on the aforementioned link, for th06 and th07 are similar in program construction. I do think the community needs more effort to summarize the existing decompilation results together.

[zero318]

The pytouhou docs are seriously outdated and blatantly wrong in a lot of ways.

If you're just looking for maps to use, I have a repo of maps that're truth compatible for everything from EoSD to UFO: https://github.com/zero318/TouhouMaps

[tabethereal]

The pytouhou docs are seriously outdated and blatantly wrong in a lot of ways.

If you're just looking for maps to use, I have a repo of maps that're truth compatible for everything from EoSD to UFO: https://github.com/zero318/TouhouMaps

Much thanks!
I couldn't find other substitutes so I used pytouhou doc's eclmap for EoSD, and I like the way it details every parameter and its role instead of just giving a function name. I'm not sure where the mistakes are. But, yes, sometimes I can't replicated the danmaku only by following it.

[zero318]

ECL problems:

• Instructions 7, 11, 12, 19, 22, 24, 37, 38, 40, 41, 42, 44, 53, 54, 55, 58, 60, 62, 64, 72, 73, 80, 89, 91, and 110 are completely missing from the list
• Instructions 84, 125, 127, 130, 133, and 134 are completely missing documentation despite being listed
• Instructions 61, 63, 65, 78, 79, 109, 122, and 135 have documentation that is completely wrong
• Instructions 51, 85, 86, and 100 have incorrect parameter types
• Instructions 4, 49, 50, 67, 68, 69, 70, 71, 74, 75, and 118 have terrible names that don't accurately describe what the instruction does (this includes all bullet spawning instructions)
• Almost all "values" boxes are only listing the range of values used in the original game files, which has nothing to do with the actual range of parameters (and is completely missing for most instructions where it matters and could crash the game)

Overall, the list is only ~60% accurate at best.

[ExpHP]

Zero, I vaguely remember that you were trying to record parameter names I thought? Or am I thinking of something else

[tabethereal]

@zero318 I see.
So is there a latest document about the instructions? I'm now guessing their functions from ecl maps most of time.

[zero318]

@ExpHP You're thinking of parameter names in th-re-data.

@tabethereal Unfortunately no, there isn't a full document with all the information. I've been meaning to make such a thing, but trying to convert the technical details of the code into plain English is surprisingly difficult and I just haven't had the motivation to do it.

[zero318]

I also really need to get back to finishing up the format documentation in my fork so it can get merged. The latest truth build is still just a zip attached to message on the ZUNCode server.