Linkification Data files and tooling #961

macchiati · 2024-11-10T18:20:13Z

Code to generate the property files and test data for UTS 58

Incorporates changes from the PAG
Also some wording updates
The Link_Email property is also modified to be narrower. The reasons for this are:
- If we have to make changes later, it is less disruptive to broaden the character set than to narrow it.
- The non-ASCII are less commonly supported currently
- I went with the identifiers from UAX 31, modified by what is valid in the ASCII ranges for the local-part:
  - \p{XID_Continue}
  - [\p{block=basic_latin}-\p{Cc}] // ASCII
  - -[\u0020 ; : " ( ) [ ] @ \ < >] // email exclusions from ASCII

See also the related spec changes in https://github.com/unicode-org/unicode-reports/pull/247

macchiati · 2025-12-02T23:36:21Z

TODO:

Update data file & test data generator to match rev 1 draft 5.
Add newer test data from ICANN.
Change the data file folder to /Public/<version>/linkification/
Change the filename SerializationTest.txt to FormattingTest.txt
Create a unicodetools PR with the code, get it reviewed & merged.

…mple test cases, match spec better

…p wikipedia pages), gather statistics on mismatches.

Plus fixed ICANN file

macchiati · 2025-12-14T07:53:42Z

The following failures are a mystery to me:

build.md / Check security data invariants (pull_request)
build.md / Check security data invariants (pull_request)Failing after 2m
build.md / Check UCD consistency, invariants, smoke-test generators (pull_request)
build.md / Check UCD consistency, invariants, smoke-test generators (pull_request)Failing after 8m

eggrobin · 2025-12-14T19:09:13Z

@macchiati:

The following failures are a mystery to me:

build.md / Check security data invariants (pull_request)
build.md / Check security data invariants (pull_request)Failing after 2m

Expected until β, ignore those.

build.md / Check UCD consistency, invariants, smoke-test generators (pull_request)
build.md / Check UCD consistency, invariants, smoke-test generators (pull_request)Failing after 8m

That is a failure in your new test LinkUtilitiesTest, so you better demystify it!

Error:  Tests run: 8, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 4.18 s <<< FAILURE! - in org.unicode.unittest.LinkUtilitiesTest
Error:  org.unicode.unittest.LinkUtilitiesTest.testMinimumEscaping  Time elapsed: 0.004 s  <<< FAILURE!
org.opentest4j.AssertionFailedError: 10) {QUERY=a%3D%26%=%3D%26%} ==> expected: <?a%3D%26%=%3D%26%> but was: <?a%253D%2526%=%253D%2526%>
	at org.unicode.unittest.LinkUtilitiesTest.testMinimumEscaping(LinkUtilitiesTest.java:200)

Since spotless doesn't convert if they are inside comments

markusicu

Review of only the data files so far.

markusicu · 2025-12-16T03:29:07Z

unicodetools/data/linkification/dev/LinkBracket.txt

+#  All code points not explicitly listed for Link_Bracket
+#  have the value undefined.
+#
+# @missing: 0000..10FFFF; undefined


I don't think we have ever had an undefined missing value.
Let's make it the same as for Bidi_Paired_Bracket and lots of other properties:

Suggested change

# @missing: 0000..10FFFF; undefined

# @missing: 0000..10FFFF; <none>

markusicu · 2025-12-16T03:32:03Z

unicodetools/data/linkification/dev/LinkBracket.txt

+# The short name of the property is the same as its long name.
+#
+#  All code points not explicitly listed for Link_Bracket
+#  have the value undefined.


Suggested change

# have the value undefined.

# have the value <none>.

see below

markusicu · 2025-12-16T03:32:50Z

unicodetools/data/linkification/dev/LinkBracket.txt

+# ================================================
+
+0029          ; 0028 #1.1     () ⇒ ()       RIGHT PARENTHESIS
+003E          ; 003C #1.1     (&gt; ⇒ &lt;)       GREATER-THAN SIGN


escaping < and > in this context seems weird

markusicu · 2025-12-16T03:38:51Z

unicodetools/data/linkification/dev/LinkEmail.txt

+# Format
+#
+# Field 0: code point range
+# Field 1: binary value


I guess this works, but it looks a bit odd. The second column is basically useless.

This must be the first time that we have a data file dedicated to one single binary property.

When we list binary properties in multi-property data files, we don't have a value column. By analogy, we only really need a single code point column here.

markusicu · 2025-12-16T03:39:37Z

unicodetools/data/linkification/dev/LinkEmail.txt

+#  All code points not explicitly listed for Link_Email
+#  have the value No.
+#
+# @missing: 0000..10FFFF; No


same as above -- we haven't written @missing lines for binary properties; they just all default to No

of course the @missing line, when present in a data file, should have the same syntax as the value lines in the same file. so if we explicitly print Yes then i guess we should have this line as well. but i think it just shows even more how weird it is. i think we should omit the Yes values and also omit the @missing line.

markusicu · 2025-12-16T04:04:42Z

unicodetools/data/linkification/dev/LinkDetectionTest.txt

+See @example.😎
+
+# No local-part
+See ⸠@example.com⸡


should this really linkify?

markusicu · 2025-12-16T04:05:49Z

unicodetools/data/linkification/dev/LinkDetectionTest.txt

+See ..john.doe@example.com
+
+#Quoted local-parts (not in the base algorithm).
+See "john\ ⸠doe@example.com⸡"


in formal email syntax, the closing quote would be right before the at sign, correct?

markusicu · 2025-12-16T04:09:06Z

unicodetools/data/linkification/dev/LinkFormattingTest.txt

+# Field 4: Result — with minimal escaping
+#
+# Empty lines, and lines starting with # are ignored.
+# Spaces around the semicolons are ignored.


Please swap this line with the next one.

markusicu · 2025-12-16T04:11:45Z

unicodetools/data/linkification/dev/LinkFormattingTest.txt

+# Spaces around the semicolons are ignored.
+# Otherwise # is treated like any other character.
+#
+# The Path, Query, and Fragment will contain backslash escapes when characters would otherwise be 


I have questions in the "Notes" doc about the purpose and behavior of the backslash syntax. You didn't respond there. I think we should drop this.

markusicu · 2025-12-16T04:15:18Z

unicodetools/data/linkification/dev/LinkFormattingTest.txt

+https:// ;	mad.wikipedia.org ;	/wiki/Tasè’ ;	 ;	 ;	https://mad.wikipedia.org/wiki/Tasè%E2%80%99
+https:// ;	wuu.wikipedia.org ;	/wiki/聖保羅（巴西） ;	 ;	 ;	https://wuu.wikipedia.org/wiki/聖保羅（巴西）
+https:// ;	vep.wikipedia.org ;	/wiki/Brüssel' ;	 ;	 ;	https://vep.wikipedia.org/wiki/Brüssel%27
+https:// ;	tw.wikipedia.org ;	/wiki/Wiase_Nyinaa_Wɛbsaet_(_World_Wide_Web;_WWW_) ;	 ;	 ;	https://tw.wikipedia.org/wiki/Wiase_Nyinaa_Wɛbsaet_(_World_Wide_Web;_WWW_)


you can't have a literal ; in a field in a semicolon-separated file -- see the "Notes" doc

macchiati marked this pull request as draft November 10, 2024 18:20

king-2929 approved these changes Nov 26, 2024

View reviewed changes

macchiati changed the title ~~Linkification testing~~ Linkification Data files and tooling Dec 4, 2025

macchiati and others added 17 commits December 4, 2025 15:33

Code for checking linkification

8172c19

Cleanup, add code for parsing

b4c8767

Enhance tooling to handle minimization, check more cases, generate sa…

a536422

…mple test cases, match spec better

Check list of files

fe0bf42

Add more test files (extracting URLs for the pages for each of the to…

f6b5cbb

…p wikipedia pages), gather statistics on mismatches.

Better test file generation

652513f

Separate out fiies

9a71ad4

Moved files

5d74257

Updates

d99e905

Update

541391b

updates, still WIP

90563b0

Update to add property files

cbe90eb

Spotless

32b6d01

Fix names

b1b64e6

ICANN test generation

f852499

Plus fixed ICANN file

Cleanup

553a6ac

Cleanup

992ff07

macchiati force-pushed the Linkification-testig branch from eebcf83 to 992ff07 Compare December 4, 2025 23:34

macchiati and others added 8 commits December 5, 2025 07:38

partial fixes for rebase

aebd296

Something that works apparently

4cb3f03

Rerun the generators, picking up 17.0 characters.

2a7cc9c

merge

3c7cdb0

Let’s not play with emoji data in the middle of linkification work

b23b504

formatting

08679ea

Escape the horizontal tabulation.

7b101d0

Cleanup

e3cb1c5

macchiati added 14 commits December 6, 2025 17:10

Add wikipedia links

06e0d43

A few more (new characters at end)

87b7376

One more

47ea939

Cleanup

268f86e

Spotless

121f26d

Change property names

79e8edb

Add LinkEmail

1defe56

Handle ambiguity with userinfo

30c5d18

Clean up headers, details for props

f4de4d8

Adjust for edge cases

1bd5c48

Minor cleanup

62d373b

Add value to comment

5d876cb

Minor cleanup

5056fb7

Restrict Link_Email as per comment in github

d8796a4

macchiati marked this pull request as ready for review December 14, 2025 07:51

macchiati requested review from eggrobin and markusicu December 14, 2025 07:52

macchiati added 10 commits December 14, 2025 17:36

Cleanup tests

b0b5d16

Fix tabs

ce147e1

Since spotless doesn't convert if they are inside comments

Tweak edge cases

0b4bf82

Fixes for Markus's notes

f4e2265

Fix brackets across interior syntax

aa8087c

Add test case for :~:

66fb559

Minor

e3f275d

Spotless

dbfd3c8

Fix test

0878783

Drop 'invalid' TLD

7388355

markusicu requested changes Dec 16, 2025

View reviewed changes

	# @missing: 0000..10FFFF; undefined
	# @missing: 0000..10FFFF; <none>

Uh oh!

Linkification Data files and tooling #961

Are you sure you want to change the base?

Linkification Data files and tooling #961

Uh oh!

Conversation

macchiati commented Nov 10, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

macchiati commented Dec 2, 2025 • edited by markusicu Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

macchiati commented Dec 14, 2025

Uh oh!

eggrobin commented Dec 14, 2025

Uh oh!

markusicu left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

macchiati commented Nov 10, 2024 •

edited

Loading

macchiati commented Dec 2, 2025 •

edited by markusicu

Loading