Skip to content

Commit

Permalink
Merge pull request #20 from williamthome/smart-modules
Browse files Browse the repository at this point in the history
Implement smart modules
  • Loading branch information
williamthome authored Nov 7, 2023
2 parents 811d5e8 + d82e4df commit 0883460
Show file tree
Hide file tree
Showing 17 changed files with 4,651 additions and 846 deletions.
50 changes: 32 additions & 18 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ Like Thoas, both the parser and generator fully conform to
- [Encode](#encode-1)
- [Decode](#decode-1)
- [Tests](#tests)
- [Smart modules](#smart-modules)
- [Credits](#credits)
- [Why the name Euneus?](#why-the-name-euneus)
- [Sponsors](#sponsors)
Expand Down Expand Up @@ -292,6 +293,8 @@ All the benchmarks compare `Euneus` and `Thoas` via [Benchee](https://github.com

Use `$ make bench.encode` or `$ make bench.decode` to run the benchmarks. Edit the scripts in the `./euneus_bench/script` folder if needed.

The benchmarks use the smart versions. Please the [Smart modules](#smart-modules) section for more information.

> **Note**
>
> - Results:
Expand Down Expand Up @@ -324,18 +327,16 @@ Use `$ make bench.encode` or `$ make bench.decode` to run the benchmarks. Edit t
<!-- To edit, open "./assets/md-tables/bench-encode.tgn" in the link above. -->
| **File** | **Euneus** | **Thoas** | **Comparison** |
|---------------------------- |------------: |-----------: |---------------: |
| blockchain.json | **9.73 K** | 7.86 K | 1.24x |
| giphy.json | **897.47** | 853.31 | 1.05x |
| github.json | **3.14 K** | 2.54 K | 1.24x |
| govtrack.json | **12.72** | 12.27 | 1.04x |
| issue-90.json | **28.92** | 17.50 | 1.65x |
| json-generator-pretty.json | **1.16 K** | 1.08 K | 1.08x |
| json-generator.json | **1.17 K** | 1.08 K | 1.08x |
| pokedex.json | 1.63 K | **1.73 K** | 1.07x |
| utf-8-escaped.json | **11.88 K** | 10.57 K | 1.12x |
| utf-8-unescaped.json | **12.19 K** | 10.83 K | 1.13x |

### Decode
| blockchain.json | **12.00 K** | 7.92 K | 1.52x |
| giphy.json | **1.03 K** | 0.86 K | 1.20x |
| github.json | **3.67 K** | 2.57 K | 1.43x |
| govtrack.json | **13.33** | 12.37 | 1.08x |
| issue-90.json | **30.10** | 17.56 | 1.71x |
| json-generator-pretty.json | **1.45 K** | 1.09 K | 1.33x |
| json-generator.json | **1.45 K** | 1.08 K | 1.34x |
| pokedex.json | **2.08 K** | 1.75 K | 1.19x |
| utf-8-escaped.json | **11.99 K** | 10.63 K | 1.13x |
| utf-8-unescaped.json | **11.99 K** | 10.89 K | 1.10x |

> **Note**
>
Expand All @@ -346,15 +347,15 @@ Use `$ make bench.encode` or `$ make bench.decode` to run the benchmarks. Edit t
| **File** | **Euneus** | **Thoas** | **Comparison** |
|---------------------------- |------------: |----------: |---------------: |
| blockchain.json | **7.18 K** | 5.78 K | 1.24x |
| giphy.json | **474.91** | 474.75 | 1.00x |
| giphy.json | **589.54** | 546.31 | 1.08x |
| github.json | **2.33 K** | 2.02 K | 1.16x |
| govtrack.json | **16.35** | 15.65 | 1.04x |
| govtrack.json | **16.90** | 15.89 | 1.06x |
| issue-90.json | **25.35** | 17.70 | 1.43x |
| json-generator-pretty.json | **617.33** | 542.99 | 1.14x |
| json-generator.json | **728.01** | 655.15 | 1.11x |
| pokedex.json | **1.37 K** | 1.33 K | 1.03x |
| utf-8-escaped.json | **1.88 K** | 1.66 K | 1.13x |
| utf-8-unescaped.json | **10.87 K** | 10.47 K | 1.04x |
| json-generator.json | **800.43** | 700.34 | 1.15x |
| pokedex.json | **1.36 K** | 1.31 K | 1.04x |
| utf-8-escaped.json | **2.05 K** | 1.67 K | 1.23x |
| utf-8-unescaped.json | **11.27 K** | 10.48 K | 1.08x |

## Tests

Expand All @@ -368,6 +369,19 @@ Also, the parser is tested using [JSONTestSuite](https://github.com/nst/JSONTest
>
> All of the JSONTestSuite tests are embedded in Euneus tests.
## Smart modules

Euneus has modules that permit customizations and others that use the default options. The modules without customizations are called smart. The smart versions are faster because they do not do any option checks.

If you are good to go with the default options, please use the smart versions:
- Encode:
- `euneus:encode/1` or `euneus_smart_json_encoder:encode/1`;
- `euneus_smart_html_encoder:encode/1`;
- `euneus_smart_js_encoder:encode/1`;
- `euneus_smart_unicode_encoder:encode/1`;
- Decode:
- `euneus:decode/1` or `euneus_smart_decoder:decode/1`.

## Credits

Euneus is a rewrite of Thoas, so all credits go to [Michał Muskała](https://github.com/michalmuskala), [Louis Pilfold](https://github.com/lpil), also both [Jason][jason] and [Thoas][thoas] contributors. Thanks for the hard work!
Expand Down
8 changes: 3 additions & 5 deletions euneus_bench/script/decode.exs
Original file line number Diff line number Diff line change
@@ -1,11 +1,9 @@
Code.eval_file("helper.exs", "./script")

euneus_opts = :euneus_decoder.parse_opts(%{})
thoas_opts = %{}

jobs = %{
"euneus" => &:euneus_decoder.decode_parsed(&1, euneus_opts),
"thoas" => &:thoas_decode.decode(&1, thoas_opts)
"euneus" => &:euneus.decode/1,
"thoas" => &:thoas.decode/1,
# "Jason" => &Jason.decode!/1,
}

data = [
Expand Down
8 changes: 3 additions & 5 deletions euneus_bench/script/encode.exs
Original file line number Diff line number Diff line change
@@ -1,11 +1,9 @@
Code.eval_file("helper.exs", "./script")

euneus_opts = :euneus_encoder.parse_opts(%{})
thoas_opts = %{}

jobs = %{
"euneus" => &:euneus_encoder.encode_parsed(&1, euneus_opts),
"thoas" => &:thoas_encode.encode(&1, thoas_opts),
"euneus" => &:euneus.encode/1,
"thoas" => &:thoas.encode_to_iodata/1,
# "Jason" => &Jason.encode_to_iodata/1,
}

data = [
Expand Down
72 changes: 72 additions & 0 deletions euneus_test/test/euneus_decoder_json_suite_test.exs
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
defmodule EuneusTest.EuneusDecoderJsonSuiteTest do
use ExUnit.Case, async: true

# Implementation-dependent tests
i_succeeds = [
"number_double_huge_neg_exp.json",
"number_real_underflow.json",
"number_too_big_neg_int.json",
"number_too_big_pos_int.json",
"number_very_big_negative_int.json",
"structure_500_nested_arrays.json"
]

i_fails = [
"number_huge_exp.json",
"number_neg_int_huge_exp.json",
"number_pos_double_huge_exp.json",
"number_real_neg_overflow.json",
"number_real_pos_overflow.json",
"object_key_lone_2nd_surrogate.json",
"string_1st_surrogate_but_2nd_missing.json",
"string_1st_valid_surrogate_2nd_invalid.json",
"string_UTF-16LE_with_BOM.json",
"string_UTF-8_invalid_sequence.json",
"string_UTF8_surrogate_U+D800.json",
"string_incomplete_surrogate_and_escape_valid.json",
"string_incomplete_surrogate_pair.json",
"string_incomplete_surrogates_escape_valid.json",
"string_invalid_lonely_surrogate.json",
"string_invalid_surrogate.json",
"string_invalid_utf-8.json",
"string_inverted_surrogates_U+1D11E.json",
"string_iso_latin_1.json",
"string_lone_second_surrogate.json",
"string_lone_utf8_continuation_byte.json",
"string_not_in_unicode_range.json",
"string_overlong_sequence_2_bytes.json",
"string_overlong_sequence_6_bytes.json",
"string_overlong_sequence_6_bytes_null.json",
"string_truncated-utf-8.json",
"string_utf16BE_no_BOM.json",
"string_utf16LE_no_BOM.json",
"structure_UTF-8_BOM_empty_object.json"
]

for path <- Path.wildcard("priv/data/json/*") do
case Path.basename(path) do
"y_" <> name ->
test name do
{:ok, _} = :euneus.decode(File.read!(unquote(path)), %{})
end

"n_" <> name ->
test name do
{:error, _} = :euneus.decode(File.read!(unquote(path)), %{})
end

"i_" <> name ->
cond do
name in i_fails ->
test name do
{:error, _} = :euneus.decode(File.read!(unquote(path)), %{})
end

name in i_succeeds ->
test name do
{:ok, _} = :euneus.decode(File.read!(unquote(path)), %{})
end
end
end
end
end
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
if Code.ensure_loaded?(ExUnitProperties) do
defmodule EuneusTest.PropertyTest do
defmodule EuneusTest.EuneusPropertyTest do
use ExUnit.Case, async: true
use ExUnitProperties

Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
defmodule EuneusTest.JsonSuiteTest do
defmodule EuneusTest.EuneusSmartDecoderJsonSuiteTest do
use ExUnit.Case, async: true

# Implementation-dependent tests
Expand Down
125 changes: 125 additions & 0 deletions euneus_test/test/euneus_smart_decoder_test.exs
Original file line number Diff line number Diff line change
@@ -0,0 +1,125 @@
defmodule EuneusTest.EuneusSmartDecoderTest do
use ExUnit.Case, async: true

test "numbers" do
assert_fail_with("-", :unexpected_end_of_input)
assert_fail_with("--1", {:unexpected_byte, "0x2D", 1})
assert_fail_with("01", {:unexpected_byte, "0x31", 1})
assert_fail_with(".1", {:unexpected_byte, "0x2E", 0})
assert_fail_with("1.", :unexpected_end_of_input)
assert_fail_with("1e", :unexpected_end_of_input)
assert_fail_with("1.0e+", :unexpected_end_of_input)
assert_fail_with("1e999", {:unexpected_sequence, "1e999", 0})

assert decode!("0") == 0
assert decode!("1") == 1
assert decode!("-0") == 0
assert decode!("-1") == -1
assert decode!("0.1") == 0.1
assert decode!("-0.1") == -0.1
assert decode!("0e0") == 0
assert decode!("0E0") == 0
assert decode!("1e0") == 1
assert decode!("1E0") == 1
assert decode!("1.0e0") == 1.0
assert decode!("1e+0") == 1
assert decode!("1.0e+0") == 1.0
assert decode!("0.1e1") == 0.1e1
assert decode!("0.1e-1") == 0.1e-1
assert decode!("99.99e99") == 99.99e99
assert decode!("-99.99e-99") == -99.99e-99
assert decode!("123456789.123456789e123") == 123_456_789.123456789e123
end

test "strings" do
assert_fail_with(~s("), :unexpected_end_of_input)
assert_fail_with(~s("\\"), :unexpected_end_of_input)
assert_fail_with(~s("\\k"), {:unexpected_byte, "0x6B", 2})
assert_fail_with(<<?\", 128, ?\">>, {:unexpected_byte, "0x80", 1})
assert_fail_with(~s("\\u2603\\"), :unexpected_end_of_input)

assert_fail_with(
~s("Here's a snowman for you: ☃. Good day!),
:unexpected_end_of_input
)

assert_fail_with(~s("𝄞), :unexpected_end_of_input)
assert_fail_with(~s(\u001F), {:unexpected_byte, "0x1F", 0})
assert_fail_with(~s("\\ud8aa\\udcxx"), {:unexpected_sequence, "\\udcxx", 7})

assert_fail_with(
~s("\\ud8aa\\uda00"),
{:unexpected_sequence, "\\ud8aa\\uda00", 1}
)

assert_fail_with(~s("\\uxxxx"), {:unexpected_sequence, "\\uxxxx", 1})

assert decode!(~s("\\"\\\\\\/\\b\\f\\n\\r\\t")) == ~s("\\/\b\f\n\r\t)
assert decode!(~s("\\u2603")) == "☃"
assert decode!(~s("\\u2028\\u2029")) == "\u2028\u2029"
assert decode!(~s("\\uD834\\uDD1E")) == "𝄞"
assert decode!(~s("\\uD834\\uDD1E")) == "𝄞"
assert decode!(~s("\\uD799\\uD799")) == "힙힙"
assert decode!(~s("✔︎")) == "✔︎"
end

test "objects" do
assert_fail_with("{", :unexpected_end_of_input)
assert_fail_with("{,", {:unexpected_byte, "0x2C", 1})
assert_fail_with(~s({"foo"}), {:unexpected_byte, "0x7D", 6})
assert_fail_with(~s({"foo": "bar",}), {:unexpected_byte, "0x7D", 14})

assert decode!("{}") == %{}
assert decode!(~s({"foo": "bar"})) == %{"foo" => "bar"}
assert decode!(~s({"foo" : "bar"})) == %{"foo" => "bar"}

expected = %{"foo" => "bar", "baz" => "quux"}
assert decode!(~s({"foo": "bar", "baz": "quux"})) == expected

expected = %{"foo" => %{"bar" => "baz"}}
assert decode!(~s({"foo": {"bar": "baz"}})) == expected
end

test "arrays" do
assert_fail_with("[", :unexpected_end_of_input)
assert_fail_with("[,", {:unexpected_byte, "0x2C", 1})
assert_fail_with("[1,]", {:unexpected_byte, "0x5D", 3})

assert decode!("[]") == []
assert decode!("[1, 2, 3]") == [1, 2, 3]
assert decode!(~s(["foo", "bar", "baz"])) == ["foo", "bar", "baz"]
assert decode!(~s([{"foo": "bar"}])) == [%{"foo" => "bar"}]
end

test "whitespace" do
assert_fail_with("", :unexpected_end_of_input)
assert_fail_with(" ", :unexpected_end_of_input)

assert decode!(" [ ] ") == []
assert decode!(" { } ") == %{}

assert decode!(" [ 1 , 2 , 3 ] ") == [1, 2, 3]

expected = %{"foo" => "bar", "baz" => "quux"}
assert decode!(~s( { "foo" : "bar" , "baz" : "quux" } )) == expected
end

test "iodata" do
body = String.split(~s([1,2,3,4]), "")
expected = [1, 2, 3, 4]
assert decode!(body) == expected
end

defp assert_fail_with(string, error) do
assert {:error, error} == decode(string)
end

defp decode!(json) do
{:ok, x} = decode(json)
x
end

defp decode(json) do
:euneus.decode(json)
end
end
64 changes: 64 additions & 0 deletions euneus_test/test/euneus_smart_encoder_test.exs
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
defmodule EuneusTest.EuneusSmartEncoderTest do
use ExUnit.Case, async: true

test "atom" do
assert encode!(:undefined) == "null"
assert encode!(true) == "true"
assert encode!(false) == "false"
assert encode!(:poison) == ~s("poison")
end

test "integer" do
assert encode!(42) == "42"
end

test "float" do
assert encode!(99.99) == "99.99"
assert encode!(9.9e100) == "9.9e100"
end

test "binaries" do
assert encode!("hello world") == ~s("hello world")
assert encode!("hello\nworld") == ~s("hello\\nworld")
assert encode!("\nhello\nworld\n") == ~s("\\nhello\\nworld\\n")

assert encode!("\"") == ~s("\\"")
assert encode!("\0") == ~s("\\u0000")
assert encode!(<<31>>) == ~s("\\u001F")

assert encode!("áéíóúàèìòùâêîôûãẽĩõũ") == ~s("áéíóúàèìòùâêîôûãẽĩõũ")
end

test "Map" do
assert encode!(%{}) == "{}"
assert encode!(%{"foo" => "bar"}) == ~s({"foo":"bar"})
assert encode!(%{foo: :bar}) == ~s({"foo":"bar"})
assert encode!(%{~c"foo" => :bar}) == ~s({"foo":"bar"})
assert encode!(%{0 => 0}) == ~s({"0":0})

multi_key_map = %{"foo" => "foo1", :foo => "foo2"}

assert encode!(multi_key_map) == ~s({"foo":"foo2","foo":"foo1"})
end

test "list" do
assert encode!([]) == "[]"
assert encode!([1, 2, 3]) == "[1,2,3]"
end

test "throws error" do
assert encode(<<0x80>>) == {:error, {:invalid_byte, ~s(0x80), <<128>>}}

assert encode(<<?a, 208>>) ==
{:error, {:invalid_byte, ~s(0xD0), <<97, 208>>}}
end

defp encode!(x) do
{:ok, json} = encode(x)
json
end

defp encode(x) do
:euneus.encode_to_binary(x)
end
end
Loading

0 comments on commit 0883460

Please sign in to comment.