gumbo-next #295

… set it as appropriate.

…tf8.c

…for lexing, to ease debugging non-printable characters.

…or. (The whole error handling really needs to be redone, it's not very helpful to users.)

…make sure their input mechanisms can accept this without relying on strlen.

…ng elements

This (admittedly massive) path simplifies the way memory allocation is performed in the library. The old `gumbo_parser_allocate` APIs have been removed and replace with the following: - `gumbo_malloc` - `gumbo_realloc` - `gumbo_free` - `gumbo_strdup` As you can see, the 4 APIs match their C standard equivalents (in both function and signature), and they no longer take a `GumboParser *` object to lookup their implementation. Instead, their implementation can be customized, globally, using the following APIs: - `gumbo_memory_set_allocator` - `gumbo_memory_set_free` These two APIs allow the user to set a global memory allocator and free function. The `allocator` function needs to have the same signature as the standard `realloc` (this allows us to use it both as a realloc in the vector and string buffer code, *greatly* reducing memory usage), and as a normal malloc (by passing `NULL` as the first argument). The `free` function needs to have the same signature as the standard `free`. With just these two functions, we can abstract the whole set of standard C memory allocation APIs, and we can do so globally, without having to pass around the parser state to find them. This greatly simplifies many parts of the library, improves performance, and fixes several pathological cases of excessive memory usage, caused by the previous lack of a `realloc` API. The following external APIs, however, are no longer backwards compatible: - struct GumboInternalOptions: no longer allows the user to set a custom memory allocator callback. - gumbo_destroy_output: no longer requires a Parser object. - gumbo_destroy_node: can now be safely exported

The previous version using `strcasecmp` over an array was a bottleneck on the library. This version uses a simple, minimal perfect hash table (computed via `mph`) to convert tag names into strings. Since we're now hashing tag names, we can pass in the length of the tag name explicitly, and avoid the superfluous allocations that the tokenizer was performing in order to NULL-terminate the tag. This is implemented on the new `gumbo_tagn_enum` API. The old `gumbo_tag_enum` API has been left as a thin wrapper to keep backwards compatibility -- it is not used internally by the library. `mph` was chosen for the perfect hash function because it generates hashes that are slightly slower than GPerf but significantly simpler, and occuppying an order of magnitude less memory (as they don't need a full copy of all the strings in the set for hashing). If the tag lookup function proves to be a bottleneck, this decision can be re-evaluated in the future.

The old implementation using 2 tagsets was being rather wasteful with stack space, allocating 2 whole sets when one of them always contains a single tag element. Knowing that the `expected` elements must always be in the HTML namespace, we can simplify these APIs by passing an array of elements and stop allocating so much space on the stack.

The HTML5 fragment parsing algorithm has been implemented using a new API, `gumbo_parse_fragment`. The old APIs are maintained for backwards compatibility, although passing `GUMBO_TAG_LAST` as the inner_html context to `parse_fragment` will cause it to parse the buffer as a full document (same functionality as `gumbo_parse_with_options`). The HTML5lib adapter code has been modified to support fragment parsing tests (the tests are passing 100%).

The most recent version of the HTML5 standard does **not** perform these replacements. However, we are targetting the html5lib 0.95 tests, where they are still performed. Hence, conditionally enable them for now until we can bring the whole suite up to speed.

Fixes compilation in Yosemite

Use `sed` rules in the Makefile to automatically generate all the Tag tables. This way we can avoid keeping them in sync.

The ascii-only helpers in the tokenizer should be used in other parts of the codebase (namely: when comparing tag names case-insensitively). Hence, export them on the util.h header.

The `create_mode` helper is very useful when building tooling on top of Gumbo, so don't keep it static.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

gumbo-next #295

gumbo-next #295

Commits on Feb 17, 2015

gumbo-next #295

Are you sure you want to change the base?

gumbo-next #295

Commits on Feb 17, 2015