Fix Default Encoding for application/json Content-Type in HTTP Responses #1422

snoopdoggy322 · 2024-12-12T14:09:26Z

This merge request addresses a long-standing issue (#175) where the http package defaults to latin1 encoding when no charset is specified in the Content-Type header. Per RFC 8259, the default encoding for application/json should be utf-8.

The fix ensures that responses with Content-Type: application/json and no explicit charset parameter are decoded using utf8 by default. This change improves compliance with the JSON standard and prevents decoding errors for valid JSON payloads encoded in utf-8.

snoopdoggy322 · 2025-01-14T09:54:59Z

can anyone reply to my pull request?

seifibrahim32 · 2025-02-04T00:23:06Z

Great but, could you try to pass some Chinese or Korean language tests through conformance tests? @snoopdoggy322.

kevmoo · 2025-02-04T04:18:44Z

CC @brianquinlan @devoncarew

Ooo...could be good

github-actions · 2025-02-04T08:00:54Z

PR Health

Changelog Entry ❗

Package	Changed Files
package:http	pkgs/http/lib/src/multipart_file.dart pkgs/http/lib/src/response.dart pkgs/http/lib/src/utils.dart

Changes to files need to be accounted for in their respective changelogs.

This check can be disabled by tagging the PR with skip-changelog-check.

Coverage ✔️

File	Coverage
pkgs/http/lib/src/multipart_file.dart	💚 95 %
pkgs/http/lib/src/response.dart	💚 100 %
pkgs/http/lib/src/utils.dart	💚 96 % ⬆️ 0 %

This check for test coverage is informational (issues shown here will not fail the PR).

API leaks ✔️

The following packages contain symbols visible in the public API, but not exported by the library. Export these symbols or remove them from your publicly visible API.

Package	Leaked API symbols

License Headers ✔️

// Copyright (c) 2025, the Dart project authors. Please see the AUTHORS file
// for details. All rights reserved. Use of this source code is governed by a
// BSD-style license that can be found in the LICENSE file.

Files
no missing headers

All source files should start with a license header.

Unrelated files missing license headers

Files
pkgs/http/example/main.dart
pkgs/http_client_conformance_tests/example/client_test.dart
pkgs/http_client_conformance_tests/lib/src/dummy_isolate.dart
pkgs/http_parser/test/example_test.dart
pkgs/http_profile/lib/http_profile.dart
pkgs/web_socket/example/web_socket_example.dart
pkgs/web_socket/lib/testing.dart
pkgs/web_socket_conformance_tests/example/client_test.dart

brianquinlan · 2025-02-04T16:18:12Z

pkgs/http/lib/src/response.dart

-  /// encoding name is unknown, [latin1] is used by default, as per
-  /// [RFC 2616][].
+  /// encoding name is unknown, [utf8] is used by default, as per
+  /// [RFC3629][].


Could you site a section? Is this only for JSON data for for all payloads?

Yes, RFC3629 requires interpreting application/json as UTF-8, everything else is described as incorrect interpretation of the specification

brianquinlan

Excellent work, we are almost there!

Would you also:

change the version in pubspec.yaml to version: 1.4.0-wip
add a 1.4.0-wip section to CHANGELOG.md with a description of your change

brianquinlan · 2025-02-04T17:48:36Z

pkgs/http/test/response_test.dart

+      final utf8Bytes = utf8.encode('{"foo":"Привет, мир!"}');
+      var response = http.Response.bytes(utf8Bytes, 200,
+          headers: {'content-type': 'application/json'});
+      expect(response.body, equals('{"foo":"Привет, мир!"}'));


Ideally, there should be one except per test because you are testing a single scenario.

Could you add tests for these additional scenarios:

test .body with a non-empty charset with content-type application/json data (e.g. with Russian and iso-8859-1)

test .body with an empty charset with a content-type of test/plain (e.g. with Russian to verify that the default of iso-8859-1 is used)

brianquinlan · 2025-02-04T17:49:19Z

pkgs/http/lib/src/response.dart

-/// charset is unknown.
+/// If the `Content-Type` header specifies a charset, it will use that charset.
+/// If no charset is provided or the charset is unknown:
+/// - Defaults to [utf8] if the `Content-Type` is `application/json` (since JSON is defined to use UTF-8 by default).


Wrap to 80 columns

brianquinlan · 2025-02-04T17:52:17Z

pkgs/http/lib/src/response.dart

@@ -21,10 +21,10 @@ class Response extends BaseResponse {
  ///
  /// This is converted from [bodyBytes] using the `charset` parameter of the
  /// `Content-Type` header field, if available. If it's unavailable or if the
-  /// encoding name is unknown, [latin1] is used by default, as per
-  /// [RFC 2616][].
+  /// encoding name is unknown, [utf8] is used by default,


This is only when content-type is UTF-8 right? So could you clarify that in your comments e.g.

If it's unavailable or if the encoding name is unknown:

[utf8] is used when the content-type is 'application/json' (see [RFC 3629]).

[latin1] is used in all other cases (see [RFC 2616][]).

fix dart-lang#175

0820553

github-actions bot added the package:http label Dec 12, 2024

snoopdoggy322 added 2 commits December 12, 2024 17:44

format code

8c267c4

rewrite test

7223e9d

add other language test

7b3c383

fix line length

ee56aa0

brianquinlan reviewed Feb 4, 2025

View reviewed changes

add section

7ba26e1

brianquinlan reviewed Feb 4, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix Default Encoding for application/json Content-Type in HTTP Responses #1422

Fix Default Encoding for application/json Content-Type in HTTP Responses #1422

snoopdoggy322 commented Dec 12, 2024

snoopdoggy322 commented Jan 14, 2025

seifibrahim32 commented Feb 4, 2025 •

edited

Loading

kevmoo commented Feb 4, 2025

github-actions bot commented Feb 4, 2025

brianquinlan Feb 4, 2025

snoopdoggy322 Feb 4, 2025 •

edited

Loading

brianquinlan left a comment

brianquinlan Feb 4, 2025

brianquinlan Feb 4, 2025

brianquinlan Feb 4, 2025

Fix Default Encoding for application/json Content-Type in HTTP Responses #1422

Are you sure you want to change the base?

Fix Default Encoding for application/json Content-Type in HTTP Responses #1422

Conversation

snoopdoggy322 commented Dec 12, 2024

snoopdoggy322 commented Jan 14, 2025

seifibrahim32 commented Feb 4, 2025 • edited Loading

kevmoo commented Feb 4, 2025

github-actions bot commented Feb 4, 2025

PR Health

brianquinlan Feb 4, 2025

Choose a reason for hiding this comment

snoopdoggy322 Feb 4, 2025 • edited Loading

Choose a reason for hiding this comment

brianquinlan left a comment

Choose a reason for hiding this comment

brianquinlan Feb 4, 2025

Choose a reason for hiding this comment

brianquinlan Feb 4, 2025

Choose a reason for hiding this comment

brianquinlan Feb 4, 2025

Choose a reason for hiding this comment

seifibrahim32 commented Feb 4, 2025 •

edited

Loading

snoopdoggy322 Feb 4, 2025 •

edited

Loading