Source folder layout re-arrangement required

[leandor]

I'd want to make space for introducing separate projects for test cases, and also, break up main CBash sources into static lib packages, with the least possible dependencies with one another.

Also, in the future, I'd want to create a new project for the boost.python generated binding, to replace the current cint API.

My initial idea would be to create a static library for each game which will contain all the record definitions (initially... at least.)

End goal is to speed up a bit the full recompilation of the main CBash project (since the bulk of time recompiling is lost on the individual record files), and also, will help in modularize the source code, as we can use those projects in isolation of one another, and detect unwanted dependencies.

[leandor]

Current Source Layout:

• <root>
• includes -> public includes (not needed for python)
• pyd -> new module generates the python bindings
• src -> original code generates CBash.dll or CBash.lib (static)
  • Oblivion -> TES4File.cpp + all record types
  • Skyrim -> TES5File.cpp + all record types
  • FalloutNewVegas -> FNVFile.cpp + all record types
• tests -> tests cases for produced output

Goal is to split 'core' from games, and generate static libs from each subgroup.

So, future layout I'm looking forward (conceptually, at least...) to have:

• root
  • includes -> public includes (not needed for python)
  • src
    • common -> (aka library, or shared) holds utils, base classes, common declarations, etc. (game/<game> below can depend on these. Probably 7z wrapping will become clases inside this... can have subdirs as needed.)
    • game/<game> -> holds specific game's declarations (can depend on common but not on core)
    • core -> holds rest of the existing API, mostly classes that handle loading and traversing, must depend on specific game (since requires parsing using specific game's File.cpp
    • <new api> -> new module (above called 'pyd', better name needed) which links with all above and wraps/adapts API to Python. Depends on core, and by transitivity on all of the above.

Conceptually, each folder under src is a component or separated module.

• Each folder inside src will generate a project target and will compile to a static lib, producing an intermediate output.
• Each successive link afterwards will link to the required dependency above, and in time produce a new static lib.
• PRO: Compilation isolation for separated code concerns, each module/layer can be compiled on its own and only recompiled when a change is done on one of its files.
• PRO: Clear boundaries to limit cross-references between source files where they wouldn't be noticed otherwise.
• CON: Link errors won't show up until later in the compilation, when everything needs to be resolved to produce the actual binary output. (I guess? Not sure on this)

---

Points of debate, contention, and doubts:

• I'm not totally clear that I'd want all the source files dumped into each main component folder, since I'd like to have some C++ tests too.
  • Should I put all the C++ tests on a separated folder inside root and divide them between the test concerning the public facing API, and the internal C++ tests, or,
  • Should I add an internal folder subdivision to each component, like core/src and core/tests.
    • -> (I don't like the repetition of 'src' in src\core\src if that's the case.)
  • Or, perhaps I could have ALL files dumped on each component main folder and have the tests be called _tests.cpp.
  • Still undecided 😬
  • Reason for wanting c++ level tests?
    • Public facing API tests need full mod files to work, specifically need a game master, or simile. We can probably provide fake files to test specific stuff, but they would have to be created one by one extracting data from a valid source.
    • OTOH, I can probably mock up all the non-essential dependencies when a test for a class that is small enough not to require an enormous job at mocking EVERYTHING in CBash to be able to work.
    • I.e. c++ tests can be more focused on assurance that tool classes are working correctly, than on the data structure verification.
• Should games be a category outside core? Maybe they belong to that layer.
  • I'd like to keep the games separated, to avoid inter-dependency pollution, since I like my layer subdivision to be perfectly clear.
  • Whatever I do it should be perfectly clear that declarations in a game folder can't depend on declarations on core classes/headers.
  • That means that any dependency like that needs to be moved to common and abstracted away.
  • Maybe this indicates that I need to do this layout change in two steps,
    1. extract common layer and leave core and games as they are
    2. later extract games one by one checking for cross-dependency and fix on a case by case basis.
  • Question that arises: Is common big enough to justify its presence?
    • Superficially, for what I saw of the code, it seems that it indeed is. But, not sure yet.
• Should a single CBash binary contain ALL supported games or should I link to different games depending on conditional declarations, and provide different CBash binaries for each supported game?
  • That's easily doable. btw. I can generate as output $Config\$Game\CBash.pyd where $Config is Relase or Debug and $Game is one of the supported games.

[Utumno]

Can't readily review that - even more so as I have similar questions on py tests layout - I keep asking:

http://stackoverflow.com/questions/34694437/python-test-package-layout

Will have a closer look - but I guess that's a trial and error approach problem - so see what's in the code, how it splits better/compiles faster etc

I have extensive refactoring experience though, so I could have a look soonish - just this week will be pressed

Nice work on labels btw ! This repo is becoming my little garden of order and beauty in a chaotic wrye Bash world :)
You may want to look here https://github.com/wrye-bash/wrye-bash/wiki/[github]-Creating-and-Managing-Issues

[leandor]

Good point, yes. And no worries, take your time, no hurry. I'm not going to do anything big before analyzing/thinking on the consequences.

http://stackoverflow.com/questions/34694437/python-test-package-layout

((BTW, I really liked the idea I saw on one of the answers there of putting tests inside the same code folder, but inside a tests/ directory. So, maybe I'll do that for the C++ tests. It simplifies the folder structure quite a bit, by reducing it by one level, basically.))

I think I'll take the plunge and do the renaming locally, I'll try the different things and see which one produces the most cleaner layout.

I'll do that after getting at least one released binary in the old format (CBash.dll) so we can at least test it and see if everything is working as it should, before starting doing too much changes at once.

---

Nice work on labels btw ! This repo is becoming my little garden of order and beauty in a chaotic wrye Bash world :)

Thanks! I shamelessly copied them from your main repo :)

You may want to look here https://github.com/wrye-bash/wrye-bash/wiki/[github]-Creating-and-Managing-Issues

Thanks! Haven't seen that one... need to dive more into the wikis, it seems. And I totally misunderstood the use of the TODO. I tagged all issues that have check-lists with TODO, but the actual use-case is very specific.

And going a bit off topic here, so pardon me:

I thought about having three-layered tags at most, like one label tags the "module" (for now: Bash API & docs), the next is the type (bug, enh, etc.) and the last is a flag/modifier, like review/goal/etc.

No more than three static label components per issue, there can be more when they tag a transitory state (like question, and also, I thought about adding a WIP tag to mark what I'm doing ATM, for example) but I think that's better handled with milestones and/or projects, right?

[Utumno]

but I think that's better handled with milestones and/or projects, right?

Yep !

Re: three levels - good idea letr me ponder - the labels is still wip on main repo but mostly settled - I have thought long and hard, trial and error etc