decoder: reject malformed number literals to match encoding/json - #598

go-json is a drop-in replacement for encoding/json, but value decoding accepted several malformed number literals that encoding/json rejects:

json.Unmarshal([]byte("01"), &v)    // go-json: ok (1);    encoding/json: error
json.Unmarshal([]byte("1."), &v)    // go-json: ok (1);    encoding/json: error
json.Unmarshal([]byte("-.5"), &v)   // go-json: ok (-0.5); encoding/json: error
json.Valid([]byte("00e1"))          // go-json: true;      encoding/json: false

The same grammar gap also showed up while skipping unknown struct fields. In #448, {"fake": 0.} unmarshaled into struct{} returned nil in go-json while encoding/json rejected it.

The parser was delimiting a candidate number by greedily consuming digit/sign/dot/exponent bytes and then relying on strconv.ParseFloat, which is looser than the JSON grammar. ParseFloat accepts leading zeros (01, 007, 00.5), bare trailing dots (0., 1., 1.e5), and a fraction with no integer part after a sign (-.5).

Fix: add validNumber (a port of encoding/json's isValidNumber) and check it after ParseFloat in the float and json.Number decoders. Also use the same check while skipping numeric values in unknown fields. The skipped array/object path now parses value boundaries instead of only balancing delimiters, so malformed numeric values are not hidden inside skipped composites.

I left Decoder.Token() unchanged; it is a separate token-stream API, and encoding/json.Decoder.Token() can also return the first token for inputs like 00.

This also re-enables five cases in the existing TestNumberIsValid that were commented out in the commit that added the test (c9f1d00) because go-json accepted them: 01, 1., 012, 01.2, 1.e1.

Fixes #448.

Testing:

• go test ./... -run 'TestDecodeFirstWin|TestUnmarshal(Rejects|Accepts).*NumberLiterals|TestNumberIsValid|TestDecoderValidatesSkippedNumberAcrossBuffer' -count=1 -v
• go test ./... -count=1
• GOGC=1 go test ./... -count=1
• go test -race ./... -count=1
• GOTOOLCHAIN=go1.19.13 CGO_ENABLED=0 go test ./... -count=1
• GOTOOLCHAIN=go1.20.14 CGO_ENABLED=0 go test ./... -count=1
• GOTOOLCHAIN=go1.21.13 CGO_ENABLED=0 go test ./... -count=1
• make cover
• cd benchmarks && go test -run '^$' -bench GoJson -benchtime=1x
• official prebuilt golangci-lint v1.54.2: golangci-lint run --concurrency=1 --timeout=5m
• gofmt -l internal/decoder/context.go internal/decoder/stream.go number_validation_test.go produced no output