Python multi namespace #5375

msyyc · 2024-12-16T06:50:30Z

Compared to the previous logic that only supported a single namespace, this PR introduces support for multiple namespaces. This primarily requires updates in two key areas:

Serialization approach: We need to account for all namespaces and serialize them individually.
Relative imports: The Python SDK utilizes relative imports internally, and we can no longer make assumptions about the location of models/operations. To determine the correct relative import, we need to know the serialized namespace within the imports function.

Considering these two main changes, here's a detailed summary of the updates in this PR:

Added the emitter flag enable-typespec-namespace, which lets users decide if they want to support native typespec namespace.
Introduced clientNamespace for models/operations/enums/namedUnions/clients: the emitter uses clientNamespace to determine the exact location where the target type should be.
Created get_relative_import_path for CodeModel: all relative import paths should be calculated using this function.
Added a new property client_namespace_types to CodeModel: it counts and stores all namespace types and information about clients/operation_groups/enums/models in each namespace.
Serialization: With the new client_namespace_types property in CodeModel, we can cycle through each namespace and serialize its clients/models/enums/operations using the same logic.
Imports: The old import logic used the relative_path and operation kwargs, which were not set from the top caller but from various functions during the calling process. The signatures were difficult to understand and maintain. I replaces them with the new serialize_namespace and serialize_namespace_type kwargs. Now, any function that needs to calculate relative imports can use these two parameters, which are set from the top caller, making them easier to understand and maintain.

…hon-multi-namespace

azure-sdk · 2024-12-16T06:52:41Z

No changes needing a change description found.

azure-sdk · 2024-12-16T07:03:22Z

You can try these changes here

🛝 Playground	🌐 Website	📚 Next docs

packages/http-client-python/generator/pygen/codegen/models/code_model.py

packages/http-client-python/generator/pygen/codegen/models/client.py

packages/http-client-python/generator/pygen/codegen/models/code_model.py

…hon-multi-namespace

tadelesh · 2024-12-24T06:14:16Z

packages/http-client-python/emitter/src/lib.ts

@@ -43,6 +44,7 @@ const EmitterOptionsSchema: JSONSchemaType<PythonEmitterOptions> = {
    debug: { type: "boolean", nullable: true },
    flavor: { type: "string", nullable: true },
    "examples-dir": { type: "string", nullable: true, format: "absolute-path" },
+    "enable-typespec-namespace": { type: "boolean", nullable: true },


nit: can we add some comment for the new config? i believe we should do such clean up with the pr of consolidating emitter's config name.

tadelesh · 2024-12-24T06:15:27Z

packages/http-client-python/emitter/src/utils.ts

+  const options = context.emitContext.options;
+  if ([undefined, false].includes(options["enable-typespec-namespace"])) {


nit:

Suggested change

const options = context.emitContext.options;

if ([undefined, false].includes(options["enable-typespec-namespace"])) {

if ([undefined, false].includes(context.emitContext.options["enable-typespec-namespace"])) {

tadelesh · 2024-12-24T06:17:36Z

packages/http-client-python/eng/scripts/ci/regenerate.ts

+  "client/namespace": {
+    "enable-typespec-namespace": "true",
+  },


for unbranded. shall we always enable it by default? i heard java do that by default and for unbranded, it is reasonable.

tadelesh · 2024-12-24T06:58:15Z

packages/http-client-python/generator/pygen/codegen/serializers/base_serializer.py


    def init_file_import(self) -> FileImport:
        return FileImport(self.code_model)
+
+    @property
+    def serialize_namespace(self) -> str:


can we add a comment for this? sth like serialize_namespace is the namespace of the serialized file. for example, when we serialize the async operation A with Azure.Test.Foo namespace, then the serialize_namespace should be Azure.Test.Foo.aio._operations.

tadelesh · 2024-12-24T07:01:25Z

packages/http-client-python/generator/pygen/codegen/models/code_model.py

+            self._operations_folder_name[client_namespace] = name
+        return self._operations_folder_name[client_namespace]
+
+    def get_serialize_namespace(


i prefer to move this function to base_serializer.py because it is not related with the code model, but for the code serialization. the result for this function is depend on how we put the generated code. same for other helper like operations_folder_name.

tadelesh · 2024-12-24T07:39:19Z

packages/http-client-python/generator/pygen/codegen/serializers/__init__.py

-    def _serialize_and_write_sample(self, env: Environment, namespace_path: Path):
-        out_path = self._package_root_folder(namespace_path) / Path("generated_samples")
+    def _serialize_and_write_sample(self, env: Environment, namespace: str):
+        out_path = self.exec_path(namespace) / Path("generated_samples")


i'm wondering how the samples and tests files will be structured? it seems they all put in the corresponding namespace. but it's better to organize them with namespace structure under one generated_samples.

tadelesh · 2024-12-24T07:39:47Z

packages/http-client-python/generator/pygen/codegen/serializers/__init__.py

        self.code_model.for_test = True
-        out_path = self._package_root_folder(namespace_path) / Path("generated_tests")
+        out_path = self.exec_path(namespace) / Path("generated_tests")


same problem with samples.

tadelesh · 2024-12-24T07:56:12Z

packages/http-client-python/generator/pygen/codegen/serializers/general_serializer.py

+                client.imports(
+                    self.async_mode,
+                    serialize_namespace=self.serialize_namespace,
+                    serialize_namespace_type=NamespaceType.CLIENT,


is this serialize_namespace_type used in imports? to me, it should be used in calculating serialize_namespace.

tadelesh · 2024-12-24T07:56:44Z

packages/http-client-python/generator/pygen/codegen/serializers/general_serializer.py

+                client.config.imports(
+                    self.async_mode,
+                    serialize_namespace=self.serialize_namespace,
+                    serialize_namespace_type=NamespaceType.CLIENT,


same issue.

msyyc added 30 commits November 11, 2024 14:03

update dependency tcgc 0.48.1

feb80b5

add clientNamespace property

60e4c15

init

ac2b225

init2

0a391bd

init3

80ea470

init4

d7f6e87

init5

4146dc4

format

f0d01be

format

3613073

merge

db6402e

review

5ea817c

review

ed61e31

review

682b381

review

32399e8

review

ab818e2

review

f3f0140

review

ea246a9

review

08a68ff

review

a8dc1d4

review

089ebcc

review

59d97c2

Merge branch 'main' of https://github.com/microsoft/typespec into pyt…

bbb5884

…hon-multi-namespace

review

9c5f209

review

d8881d1

review

5aeb9cf

review

c8cac2c

review

b0b1b43

review

a21d7ad

review

6c37c0c

review

d0dcc68

msyyc added 14 commits December 10, 2024 14:30

review

af457ca

review

b58ef98

merge

5458bb5

fix

d5b473a

fix

689a06b

fix

4128461

Merge branch 'main' of https://github.com/microsoft/typespec into pyt…

08ca7c7

…hon-multi-namespace

fix typing

6aa4203

fix typing

5700597

review

49cc526

review

5b7fe1a

Merge branch 'main' of https://github.com/microsoft/typespec into pyt…

7290a71

…hon-multi-namespace

review

c592198

review

21d8448

msyyc requested review from iscai-msft and tadelesh as code owners December 16, 2024 06:50

microsoft-github-policy-service bot added the emitter:client:python Issue for the Python client emitter: @typespec/http-client-python label Dec 16, 2024

msyyc mentioned this pull request Dec 16, 2024

Auto PR for https://github.com/microsoft/typespec/pull/5375 Azure/autorest.python#2968

Draft

fix crash when no og in client

851fd6c

tadelesh reviewed Dec 23, 2024

View reviewed changes

msyyc added 3 commits December 23, 2024 16:11

review

b493479

Merge branch 'main' of https://github.com/microsoft/typespec into pyt…

0449f59

…hon-multi-namespace

review

85435c3

msyyc requested a review from tadelesh December 24, 2024 02:00

msyyc added 3 commits December 24, 2024 11:26

review

c8cd8d1

fix import

a9bdfa7

fix import

06f42b6

tadelesh reviewed Dec 24, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Python multi namespace #5375

Python multi namespace #5375

msyyc commented Dec 16, 2024

azure-sdk commented Dec 16, 2024

azure-sdk commented Dec 16, 2024

tadelesh Dec 24, 2024

tadelesh Dec 24, 2024

tadelesh Dec 24, 2024

tadelesh Dec 24, 2024

tadelesh Dec 24, 2024

tadelesh Dec 24, 2024

tadelesh Dec 24, 2024

tadelesh Dec 24, 2024

tadelesh Dec 24, 2024

		const options = context.emitContext.options;
		if ([undefined, false].includes(options["enable-typespec-namespace"])) {

	const options = context.emitContext.options;
	if ([undefined, false].includes(options["enable-typespec-namespace"])) {
	if ([undefined, false].includes(context.emitContext.options["enable-typespec-namespace"])) {

Python multi namespace #5375

Are you sure you want to change the base?

Python multi namespace #5375

Conversation

msyyc commented Dec 16, 2024

azure-sdk commented Dec 16, 2024

azure-sdk commented Dec 16, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment