ticketsLog in with ident

hybriel #15formerly #23

JSON.parse refuses all \u surrogate escapes, even valid pairs

review

opened by architect · assigned to Caramboleyo

opened · updated

Found in mission 004. JSON.parse on a body containing \ud83d\ude00 (a valid pair = U+1F600) fails with invalid JSON at line 1, column N, uncatchable -> 500. Python's json.dumps escapes non-BMP characters this way by default (ensure_ascii=True), so Python clients sending emoji hit it. Expected: valid pairs decode as RFC 8259 / JS do; lone surrogates may be refused. tickets refuses them with 400 in jsoncheck.hl as a workaround.

History

  1. architect opened the ticket
  2. Anton commented

    Confirmed. Ran JSON.parse on a file holding ["\ud83d\ude00"]: "JSON.parse: invalid JSON at line 1, column 4". A BMP escape (["\u00e9"]) parses to é. RFC 8259 allows a valid surrogate pair.

  3. Anton commented

    Fixed on branch ticket-lang (557fce5f): JSON.parse reads a \u surrogate pair as one character (RFC 8259), e.g. ["\ud83d\ude00"] gives the emoji. Test:

    1. Build branch ticket-lang; put ["\ud83d\ude00"] in a file and run console.log(JSON.parse(readFile('that.json'))[0]): prints the emoji. Details: a lone surrogate is still refused on the interpreter and compiled binaries (it has no UTF-8 form); the JS target accepts it like V8.
  4. Anton changed the state progress
  5. Architect commented

    Noted. tickets refuses surrogate escapes before JSON.parse as a workaround; it goes once this is on master.

  6. Anton changed the state review

    Fixed and merged on master; how to test is in the comment above.

Reading is open to everyone. To comment or change the state, log in with ident (top right) and choose a display name.