nested classes in java sources cannot be resolved

[yigit]

looks like i broke something here:

0a0af2b

for some reason, KSP fails to resolve a nested class's type when it is in sources.
repro:
yigit@2e35c09

I initially thought this was a KSP problem but cannot reproduce it neither there nor in a test project.
google/ksp#263

[yigit]

btw i'm sure it is because of that change because reverting it fixes the test.

[yigit]

i put more investigation notes into google/ksp#263 (comment).

This is about java files' location not matching the project structure.
Seems like java compilation works fine but kotlin does not allow you to access java files if they are misplaced (package not matching file location wrt source root).

As far as I can see, we don't set source root, instead we send files to kotlin compile / java compile.
Maybe we need to change that in kotlin compile testing, i'm not sure yet.

[yigit]

so it works fine if i pass the sources folder into the kotlin compiler as freeArgs instead of the actual files.
in theory this should reproduce without KSP, i'll try to see if i can reproduce that.

[yigit]

here is a repro w/o KSP:
yigit@6267f67

i'm afraid we may need an API change to specify java file locations or allow paths in names.

[yigit]

nevermind, repro had bad code :(

[yigit]

seems like this is happening because we give workingdir/ksp as the projectBaseDir to KSP which means the java file is outside the project.
i'm not yet sure what the proper solution is but at least i can workaround it in room by using SourceFile.fromPath and building the project structure myself in tests.

[yigit]

Finally figured it out (i think :) )
KCT is not setting java source roots directory, which is why KSP cannot find java files properly.

https://github.com/JetBrains/kotlin/blob/master/compiler/cli/cli-common/src/org/jetbrains/kotlin/cli/common/arguments/K2JVMCompilerArguments.kt#L236

I can "fix" that by giving the sources directory but it is not necessarily correct for source files that are created from other existing files:

https://github.com/tschuchortdev/kotlin-compile-testing/compare/master...yigit:find-nested-class-repro?expand=1

return compileKotlin(sources, K2JVMCompiler(), commonK2JVMArgs().also {
			it.javaSourceRoots = arrayOf(sourcesDir.canonicalPath)
		})

Now the problem in the KCT side is that we allow passing arbitrary java files so their root directory does not really exist.
@tschuchortdev what do you think?

[tschuchortdev]

If we set javaSourceRoots to sourcesDir.canonicalPath then I suppose any sources outside of that directory wouldn't be found. Using source files from the host project's resources directory instead of creating them inline is a valid use case, so that's problematic. We could also copy all source files into the sourcesDir but that may lead to naming collisions, thus the file names would have to be changed to be unique. Of course this makes compiler errors involving those files nonsensical, so we also need to keep a mapping between the original file path and unique file name and replace all occurrences in the output message stream. Very messy.

Is it not possible to add every Java file as a separate source root like we are doing here?

I wonder why this javaSourceRoots option is needed at all when it seems to work fine without, for the most part.

[tschuchortdev]

Is it not possible to add every Java file as a separate source root like we are doing here?

Apparently, this doesn't work. But I haven't figured out yet why.

This is weird. javaSourceRoots and putting paths into freeArgs seems to result in basically the same code path: https://sourcegraph.com/github.com/JetBrains/kotlin@a4dd47daab6ded30ffcc19d5be61b50a256615b6/-/blob/compiler/cli/src/org/jetbrains/kotlin/cli/jvm/K2JVMCompiler.kt#L186

[tschuchortdev]

By the way, I've talked to someone in the compiler team about this who had this to say:

[...] accepting single .java files in the compiler was implemented as a kind of a hack, where we are running the Java lexer manually beforehand and are recording fully qualified names of all classes that we find:
https://github.com/JetBrains/kotlin/blob/master/compiler/cli/src/org/jetbrains/kotlin/cli/jvm/index/SingleJavaFileRootsIndex.kt
IIRC running the Java parser was not feasible at this point because of performance reasons.
There might very well be some bug in that code, that doesn’t catch some nested classes in some cases.

With this in mind, I think it would be best to change the API to no longer accept paths to Java files and only to Java source roots.

[yigit]

FYI this is what we do in room:
https://cs.android.com/androidx/platform/frameworks/support/+/androidx-main:room/compiler-processing-testing/src/main/java/androidx/room/compiler/processing/util/Source.kt;l=1?q=Source.kt&sq=&ss=androidx%2Fplatform%2Fframeworks%2Fsupport

For java, it expects a qualified name and content;
For kotlin, it expects a file name and content.

In room, we actually write those files into proper places wrt their qualified names to avoid this issue. (and we can find the correct place from the qName).

[rossbacher]

FYI this is what we do in room:
https://cs.android.com/androidx/platform/frameworks/support/+/androidx-main:room/compiler-processing-testing/src/main/java/androidx/room/compiler/processing/util/Source.kt;l=1?q=Source.kt&sq=&ss=androidx%2Fplatform%2Fframeworks%2Fsupport

For java, it expects a qualified name and content;
For kotlin, it expects a file name and content.

In room, we actually write those files into proper places wrt their qualified names to avoid this issue. (and we can find the correct place from the qName).

We tried a workaround like this for DeepLinkDispatch (https://github.com/airbnb/DeepLinkDispatch/blob/supportKsp/deeplinkdispatch-processor/src/test/java/com/airbnb/deeplinkdispatch/test/Source.kt) to be able to run this test(https://github.com/airbnb/DeepLinkDispatch/blob/supportKsp/deeplinkdispatch-processor/src/test/java/com/airbnb/deeplinkdispatch/DeepLinkProcessorIncrementalTest.kt#L388) without the NPE on getting the type of the inner class and it still fails exactly the same way. Just FYI.