diff --git a/docs/design/decisions/S1-bracketed-credential-is-not-a-nickname.md b/docs/design/decisions/S1-bracketed-credential-is-not-a-nickname.md index a0ae66106..8b4524e80 100644 --- a/docs/design/decisions/S1-bracketed-credential-is-not-a-nickname.md +++ b/docs/design/decisions/S1-bracketed-credential-is-not-a-nickname.md @@ -7,6 +7,7 @@ - It keeps its pair's reading where the clause is nothing but title modifiers and styles (`(UK)`, `(Hon)`), a lone title of address (`(Sir)`, `(Mr)`), titles alone straight behind a credential (`FRCS (SN)`), or a credential written as initials before a name word of its comma part (`Justin (JP) Stearns`); content that is not such a clause and ends in an ASCII period keeps v1's escape, its brackets dropped — 2026-11-04 (Derek), #792. - (superseded for a clause of non-name words by #792, 2026-11-04 below) The rule says what the escape does: quotes escape as brackets do, and a clause of two or more words is not suffix-shaped unless an ASCII period ends it, so `(PhD MSc)` keeps its nickname reading with nothing reported, as 1.4.0 read `(PhD MBA)`, which reports suffix-or-nickname since `mba` was marked ambiguous (decisions/suffix-acronym-collisions, 2026-11-03) — 2026-10-21 #807. - A member of the ambiguous class its writing leaves unsettled is a word of such a clause where a credential of the same clause settles it, after the name only (S13's position): `John Smith (Ma, PhD)`, `(Ma PhD)` and `JOHN SMITH (PHD, MBA)` → suffix, where each was the nickname, reported. A credential that is no name settles one, or a member its periods settle; a generation, a numeral, an honorific, a title or a title-and-suffix word (`Edward Smith (Ed Jr)`, `(Ma, Ms)`, `(Ma, 씨)`) and a member its capitals settle (`(MA, Ed)`) settle nothing, nor do members alone (`(Ma, Ed)`); a joined word settles one where a chunk of it would (`(Ma, PhD/Jr)` → suffix, `(Ed, Jr/Sr)` → nickname) — 2026-11-07 (Derek), #1138. +- A comma glued between two words of a clause parts them for S1's tests, and the word they make is set apart whole and kept as written where its words take one role (`John Smith (MD,PhD)` → suffix 'MD,PhD'); tokens, nicknames and `Parser.revise` are as before — 2026-11-11 #1268. Elsewhere: an ambiguous member alone in the brackets staying a nickname, reported, is decided under decisions/suffix-acronym-collisions (its Standing line ending 2026-09-25 #540; 2026-10-21) and decisions/ma-do's 2026-07-17 entry; a member after the name whose writing settles it reading as the suffix is decisions/S13 (2026-10-26 #810). @@ -27,3 +28,6 @@ Elsewhere: an ambiguous member alone in the brackets staying a nickname, reporte MEASURED 2026-11-05 against master 4109b026 checked out in a detached worktree: the committed snapshot moves none of its 6,945 entries (6,736 distinct names), and none of the 181 distinct names holding a pair of `DEFAULT_NICKNAME_DELIMITERS` has a word joined by a slash or a hyphen inside one, so the zero says the snapshot could not see the shape (recipe: for each name and each pair, the text between an opener and the next closer after it, matched against `\S[-/]\S`); the rule's fourteen new examples join the snapshot through corpus_rules.jsonl in the same change, recorded as new entries. A grid of 30 clause words (thirteen that move: the joined credentials `MD/PhD`, `MD-PhD`, `Md-PhD`, `md/phd`, `MD/PhD/MPH`, `M.A./PhD`, `Jr/Sr`, `MD/PhD,`, `M.D./Ph.D.`, and the clauses `MD/PhD, MBA` and `PhD, MD/MPH` holding one, and the kept-comma members `M.A,` and `M.A.,`; seventeen that do not: S3's refusals bare, a bare member as a chunk (`PhD/MBA`, `MA/PhD`, `JD/MBA`), a chunk not written with its periods (`M.A/PhD`), an unlisted chunk (`RN-BC`, `CPA/PFS`, `XY/ZW`), a name or a title as a chunk (`MD/Bob`, `Bob/MD`, `Dr/PhD`), a dangling joiner (`MD/`, `/PhD`), single-letter numerals (`I-V`) and a hyphenated name (`Smith-Jones`, `Ae-Ra`), and the members without a comma, `M.A` and `MA`) in 14 shapes, the templates exactly `John Smith (X)`, `John (X) Smith`, `(X) John Smith`, `(X)`, `Smith, John (X)`, `Smith (X), John`, `John Smith (X), Jr.`, `John Smith, Jr. (X)`, `JOHN SMITH (X)`, `Jack (X)`, `Smith, (X)`, `(X), PhD`, `John Smith "X"` and `Kim Min (X)`, each parsed by `Parser()` and by `Parser(policy=Policy(name_order=...))` under FAMILY_FIRST and FAMILY_FIRST_GIVEN_LAST, a name moving where `as_dict()` or its ambiguity kinds differ, 1,260 names: 399 move, 133 under each order, every move a joined credential read as the suffix, a period-final joined word before a name word read as the nickname (two under each order), or the member report (447, 149 under each order, before the initials Except asked each chunk). The comparator is the base tree's own readings of the same grid, parsed with each tree's `nameparser.__file__` asserted. In the local corpora (`snapshot.py --local --base `, on the first build): ctgov-officials 3 entries move, `Franck Pages, Professor (MD-PHD)` and `Franck Pagès, Professor (MD-PHD)` and `Philippe PC Courtet, Professor(MD-PhD)` → suffix where nickname; fec-individuals 1, `FRANTZ (CAPT USN-RET), JOSEPH` → title 'CAPT', suffix 'USN-RET' where nickname 'CAPT USN-RET'; psc-individuals and orcid-researchers none. The initials Except reaches only a joined word before a name word of its comma part, and none of the four stands there; each was parsed again on the final tree and reads as listed. - 2026-11-07 #1138 (Derek) — A CREDENTIAL SETTLES A MEMBER OF ITS CLAUSE. Derek, on #1138: the delimiter says everything inside it is one kind of thing, and a credential with no name reading (`PhD`) says it is not a nickname, which removes the doubt about a member beside it (`Ma`). So a member of the ambiguous class its writing leaves unsettled (S13's case and periods) is a word of a set-apart clause where another word of the clause is a credential that is no name, or a member its writing settles; the clause is then set apart after the name only, where S13 settles a member, so inside the name `John (Ma, PhD) Smith` stays the nickname, reported, as before, and #792's initials Except is untouched (Derek kept it, 2026-11-07). A title does not settle a member (`John Smith (Dr, Ma)` → nickname), nor do members alone (`(Ma, Ed)`). The reviews narrowed the settler three times (decisions/S3's entry of the same date has each), the last making a joined word settle one only through a chunk that would (`(Ed, Jr/Sr)` stays the nickname) and adding honorifics (`(Ma, 씨)`): a member its capitals settle settles nothing, as no joined chunk is read by its case, and neither does a generation, a numeral or a title-and-suffix word, since `Edward Smith "Ed Jr"` is a name a person goes by. WEIGHED: the members that are common nicknames, `ed`, `ace`, `che`, `chi`, `asa`, `caro`, `ciro`, `junior`, read as the suffix beside a credential after the name (`John Smith (Ed, MD)`, `(Junior, PhD)`), Derek's principle being that the credential says the clause is no nickname; Ed beside a generation stays the nickname. ACCEPTED: a clause ending in an ASCII period takes the escape before the member is read (`John Smith (Ma, Ph.D.)` → family 'Ma'), as `(M.A, Ph.D.)` already did. Moves: `John Smith (Ma, PhD)`, `John Smith (Ma PhD)` and the one-case `JOHN SMITH (PHD, MBA)`, the rule's boundary example since #792, → suffix with no report. Its joined twin is decisions/S3's 2026-11-07 #1138 entry: a joined word holding a member is a member itself, so `(MA/PhD)` reads as `(MA, PhD)` does in both positions, and inside the name it now reports, as its written-apart twin does. MEASURED: see decisions/S3's 2026-11-07 #1138 entry, which holds both halves' measurement. - 2026-11-07 #1140 (Derek) — THE PERIOD ESCAPE BEFORE A MEMBER STAYS, NOT PLANNED. #1140 asked whether a clause a credential settles a member of should be set apart when it ends in an ASCII period (`John Smith (Ma, Ph.D.)`, family 'Ma', where `(Ma, PhD)` is the suffix), which rules/S1's Accepted clause records as a cost of #1138. The fix would make extract's escape, decided before any word is read, wait for the member's reading in post_rules, for every period-final clause. MEASURED (2026-11-07, on 861f532e): every name in the cached local corpora (`~/.cache/nameparser-tools/local/*/corpus.jsonl`) was searched for a bracketed or quoted clause ending in a period whose words `_vocab.set_apart_kind` reads as no name, save at least one member of the ambiguous class (a `member` kind, or None with `_vocab.member_word`): 3 names hold one, none of them a member beside a credential (`MRCS (Ed.)` and `M(Ed.)` in ctgov-officials, both read into the suffix already, and `Ioannis (J.D.) Vergados` in orcid-researchers, whose `J.D.` counts as a suffix as written and is no member, so a fix would not reach it). With no written name holding the shape, Derek closed the issue rather than move the escape; a member written with its final period (`(M.A., Ph.D.)`) already counts as a credential and is set apart through it. +- 2026-11-11 #1268 — A COMMA GLUED BETWEEN TWO WORDS OF A CLAUSE PARTS THEM FOR S1'S TESTS, AND THE WORD IS SET APART WHOLE. `John Smith (MD,PhD)` read as a nickname in every release (`MD,PhD`, or `MD PhD` in 2.0.0 through 2.4.0), its spaced twin `(MD, PhD)` as two credentials: the glued pair was one word no vocabulary knows. Post_rules' `_set_apart_clauses`, where S1 is decided for good, now reads a clause's token that holds a single comma glued between two words as those words (`_post_rules._glued_kind`), each a word `set_apart_kind` reads; where they share one kind (all credentials or generations, or all titles, a title/suffix dual taking its neighbours') the token is set apart whole, its comma kept as written: suffix 'MD,PhD'. A doubled comma parts nothing (`(MD,,PhD)`), a word holding a member of the ambiguous class is declined, nothing in the word settling it (`(Ma,PhD)`, `(MD/PhD,MBA)`), and a word whose words would take a title and a suffix keeps its pair's reading (`(Dr.，PhD)`), the token being one word; each stays as at the base, rules/S1's Accepted. + WHY THE WORD IS NOT DIVIDED: the PR's first design ended a clause's word at a glued comma in tokenize, in every clause, nicknames included, and three review rounds then compensated for it (a rejoin in `Parser.revise`, guards for doubled commas and dropped separators) without converging: the third found `revise` differing from the base in 5,138 of 34,664 cases and the rejoin quadratic in clauses. Tokenize cannot divide only the clauses S1 sets apart, because S1 is decided twice: extract decides provisionally before tokens exist and post_rules finally, two stages later, and post_rules declines clauses extract approved (`Justin (JP,PhD) Stearns`, `John (MD,PhD.) Smith`, `Louise Dye, PhD (Dr.,Prof.)`) and sets apart clauses extract refused (`(Ma, PhD)`). Dividing in post_rules instead would insert tokens after pieces, segments and the stages' reports already index them. So the token stays whole: tokenize, extract and `Parser.revise` are the base's, a clause S1 does not set apart yields the base's tokens, tags and text, and the cost is the reading as written ('MD,PhD', not the spaced twin's 'MD, PhD'), the mixed word left a nickname, and a fourth round-trip limit for `revise(suffix=...)`: the suffix 'MD,PhD' revised as a bare value is read outside any clause, where the comma divides, giving 'MD, PhD' (`tests/v2/test_parser.py`'s `_GLUED_COMMA_IN_SUFFIX`, beside the glued-word limit it resembles). DECLINED, with that history: dividing the token. + Measured 2026-11-11 against 84370228 (a `git archive`, `__file__` asserted): `Parser.revise` over 18 values × 3 spellings × 7 fields, compared on fields, `initials()` and `capitalized()`, 0 of 378 differ; the shapes the reviews found (doubled commas, a comma before or behind a dropped separator, `(,,MD)`, `(,,Jr.)` between and after the name, `(MD,,PhD)`) read as at the base, and so does every comma-free clause tried; what moves is a clause set apart through a glued word: `John Smith (MD,PhD)` and, under bracket `maiden_delimiters`, the same text (maiden 'MD,PhD' → suffix 'MD,PhD', S1 taking a maiden pair's clause first, M1), and two ORCID names (`Dr. Thierry Ingabire, (MD,PhD)`, `Dr. Md Safiul Alam Bhuiyan (DVM.MS,PhD)`). COST, `sys.setprofile` call counts: a plain name and a clause with no glued comma pay nothing (`John Smith` 126, `John "Jack" Smith` 198, `John Smith (MD, PhD)` 247, each as at the base); a clause with a glued comma pays the word's reading, one frame and one a word (`John Smith (MD,PhD)` 196 → 214, `Jane (Janey,Jo) Smith` 198 → 199). diff --git a/docs/design/rules/S1-bracketed-credential-is-not-a-nickname.md b/docs/design/rules/S1-bracketed-credential-is-not-a-nickname.md index e6d1675a6..c53d6b85c 100644 --- a/docs/design/rules/S1-bracketed-credential-is-not-a-nickname.md +++ b/docs/design/rules/S1-bracketed-credential-is-not-a-nickname.md @@ -27,7 +27,7 @@ "W. Russell (Jr.) Withers" → suffix="Jr." "Jack (PhD)" → family="Jack" ``` -- **A comma inside the clause parts its words as a comma between them would**, though in a nickname it is kept as written (N1), and the comma is no part of the word it is written against. A comma is any character C's Background counts as one, each reading here as the ASCII one does. +- **A comma inside the clause parts its words as a comma between them would**, though in a nickname it is kept as written (N1), and the comma is no part of the word it is written against. A comma is any character C's Background counts as one, each reading here as the ASCII one does. A single comma glued between two words of the clause parts them for these tests too, and the word they make is set apart whole, its comma kept as written, where its words take one role. ``` "John Smith (USN, Ret.)" → suffix="USN, Ret." "John Smith (Dr., PhD)" → title="Dr." @@ -36,6 +36,7 @@ "John Smith (MD ,PhD)" → suffix="MD, PhD" "John Smith (Jr.,) III" → suffix="Jr., III" "John Smith (MD,) (PhD)" → suffix="MD, PhD" + "John Smith (MD,PhD)" → suffix="MD,PhD" ``` - **A member of the ambiguous class (S4) is a word of such a clause where its writing settles it after the name**, as S13 settles a member standing alone, wherever in the clause it stands. A member its writing leaves unsettled is one too where a credential of the same clause settles it: the brackets hold one kind of thing, and a credential that is no name, or a member its periods settle, says which. A joined word settles one where a chunk of it would. A generation, a numeral, an honorific, a word that is a title too and a member its capitals settle say nothing of the kind: `Ed Jr` is a name a person goes by, and a joined chunk is never settled by its case (S3). ``` @@ -159,6 +160,12 @@ "(PhD/Sr) John Smith" → suffix="PhD/Sr" "(PhD, Sr) John Smith" → title="Sr" ``` +- A word made of words glued at a comma is set apart only whole, so one whose words would take two roles, a title and a credential, keeps its pair's reading, as does one holding a member of the ambiguous class, which nothing in the word settles; a doubled comma glued between two words parts nothing. + ``` + "John Smith (Dr.，PhD)" → nickname="Dr.，PhD" + "John Smith (Ma,PhD)" → nickname="Ma,PhD" + "John Smith (MD,,PhD)" → nickname="MD,,PhD" + ``` - A qualifier written after a title in front of the name is the suffix it is in the vocabulary. ``` "Col. (Ret.) Smith" → suffix="Ret." diff --git a/docs/release_log.rst b/docs/release_log.rst index 23fa16906..567f97945 100644 --- a/docs/release_log.rst +++ b/docs/release_log.rst @@ -38,6 +38,7 @@ Release Log - **Change a Thai title before one name word to read that word as the given name.** ``parse("ดร. สุดา")`` gives title ``ดร.``, given ``สุดา``, where 1.4.0 through 3.0.1 gave family ``สุดา``, since Thai addresses a person by the given name after any title. A Thai title alone in brackets now reads as a nickname, as ``(Sir)`` does (``parse("สุดา ใจดี (นาย)")`` gives nickname ``นาย``). :issue:`317` - **Fix a Thai salutation glued to the given name reading as part of it.** ``parse("นายสมชาย ใจดี")`` gives title ``นาย``, given ``สมชาย``, where 1.4.0 through 3.0.1 gave given ``นายสมชาย``; ``นาย``, ``นาง`` and ``นางสาว`` are listed, and they and the spelled-out titles split off the start of the name's first word. A title written short (``นพ``, ``ดร``) still reads as part of a glued given name, since given names open with the same letters. :issue:`317` - **Fix Thai academic and professional titles reading as given names.** ``parse("รศ.ดร.สุดา ใจดี")`` gives title ``รศ.ดร.``, given ``สุดา``, where 1.4.0 through 3.0.1 gave given ``รศ.ดร.สุดา``; the spelled-out titles (``รองศาสตราจารย์``, ``อาจารย์``), their short forms and the physician, dentist, pharmacist and veterinarian titles are listed. A title glued to the name with no full stop (``อาจารย์สุดา``) still reads as part of the given name. :issue:`1166` + - **Fix credentials in brackets joined by a comma with no space reading as a nickname.** ``parse("John Smith (MD,PhD)")`` gives suffix ``MD,PhD``, where every release from 1.4.0 through 3.0.1 gave a nickname (``MD PhD`` in 2.0.0 through 2.4.0, ``MD,PhD`` otherwise). The word keeps its comma as written; one joining a title to a credential (``(Dr.，PhD)``) or holding a word that may be a name (``(Ma,PhD)``) is still a nickname. :issue:`1268` - **Fix a nickname, a title word and a suffix reading the word as the given name.** ``parse("'Smitty' Pope Jr.")`` gives family ``Pope``, as ``'Smitty' Smith Jr.`` gives family ``Smith``, where 2.3.0 through 3.0.1 gave given ``Pope`` and 1.4.0 through 2.2.0 gave title ``Pope``, given ``Jr.``; the ``title-or-name`` report stays. A title-particle or an initial keeps its reading (``'Smitty' St Jr.`` gives given ``St``); a bracketed credential beside a title word is read as a nickname is (``Pope (PhD) Jr.`` gives family ``Pope``). :issue:`1169` - **Fix initials after a one-word surname and a comma that spell a credential reading as the credential.** ``parse("Jansen, J.P.")`` gives given ``J.``, middle ``P.``, reporting ``suffix-or-name``, where 1.4.0 gave given ``J.P.`` and 2.0.0 through 3.0.1 gave suffix ``J.P.`` and no given name; ``de Waart, J.P.`` and ``Jansen, Ir. J.P. van`` read the same way. ``Smith, M.D.`` reads given ``M.``, middle ``D.`` now too, the credential named in the report; two name words before the comma (``Bin Zhang, M.D.``), another credential in the part (``Smith, M.D. Ph.D.``) or the undotted ``Smith, MD`` keep the credential. :issue:`1238` - **Fix Token.reading naming a title phrase's object a title by shape where the shape refuses it.** ``parse("Member of Chas. John Smith").tokens[2].reading`` is ``Reading('title' as 'title', 'position')``, where 3.0.0 and 3.0.1 gave source ``shape``; a CJK word there (``田中.``) reads ``position`` too. :issue:`1191` diff --git a/nameparser/AGENTS.md b/nameparser/AGENTS.md index f389c9f1b..d6faa3b66 100644 --- a/nameparser/AGENTS.md +++ b/nameparser/AGENTS.md @@ -54,7 +54,7 @@ Testing one spelling from each path holds for a word a rule only asks ABOUT, too ### A separator's spacing -The same holds for a separator's SPACING: a comma (or any punctuation) can stand alone, glue to the word before it or glue to the word after, and each is a different token stream. #653's first fix swept `(née, Jones)` and missed `(née , Jones)`, where the kept comma became a word after the marker and M3 read the clause as maiden ', Jones'; its review found it. Put all three spacings in any grid around a separator. +The same holds for a separator's SPACING: a comma (or any punctuation) can stand alone, glue to the word before it or glue to the word after, and each is a different token stream. #653's first fix swept `(née, Jones)` and missed `(née , Jones)`, where the kept comma became a word after the marker and M3 read the clause as maiden ', Jones'; its review found it. Put all three spacings in any grid around a separator. And before dividing at a separator inside a bracketed clause, find where the clause's reading is decided: S1's set-apart is decided provisionally in extract, before tokens exist, and for good in post_rules, two stages later, each declining clauses the other took, so a division made in tokenize reaches clauses S1 then leaves nicknames. #1268's first design divided at a glued comma in tokenize, in every clause, and three review rounds compensated without converging (`Parser.revise` differed from the base in 5,138 of 34,664 cases); PR #1339 shipped the reading in post_rules' decision alone, the token kept whole. A separator with several spellings is reached by a LITERAL as well as by its set: the comma family (`_state.COMMA_CHARS`) answered every site that divides a name, while eleven sites taking the comma off a set-apart clause's word still wrote `strip(",")` or `== ","`, and render's space-before-comma collapse `\s+,`, so `(PhD፣ Sr) John Smith` and the Arabic `John Smith (Dr.، PhD)` read a nickname through 3.0.1, found by #1202's review. Grep for the literal and for a regex holding it whenever the set moves, and swap the member in a grid that compares ambiguities and `str()` as well as fields: a swap walk over fields alone passed with `dotted_member` still ASCII, the site moving only a report. diff --git a/nameparser/_pipeline/_post_rules.py b/nameparser/_pipeline/_post_rules.py index a736dfa6d..08539acf1 100644 --- a/nameparser/_pipeline/_post_rules.py +++ b/nameparser/_pipeline/_post_rules.py @@ -54,6 +54,7 @@ from __future__ import annotations import bisect +import re from collections.abc import Sequence from nameparser._lexicon import Lexicon, _normalize, _run_addresses_by_given @@ -767,8 +768,29 @@ def _set_apart_clauses(state: ParseState, # the clause's credential words, asked `settles_a_member` only # where an unsettled member needs one others: list[str] = [] + # the words and kinds the clause tests read: a token's own, or + # the words a comma glued inside it parts (`_glued_words`) + test_words: list[str] = [] + test_kinds: list[SetApartKind] = [] for i in range(lo, hi): token = tokens[i] + # rules/S1: "the comma is no part of the word it is written + # against" -- a comma glued between two words of the clause + # parts them for these tests, though the token stays whole + # (rules/N1's clause as written); asked in C first, so a + # clause with no such comma pays no frame (decisions/S1, + # #1268) + bare = token.text.strip(COMMA_STRIP) + if not COMMA_CHARS.isdisjoint(bare): + glued = _glued_kind(bare, lex) + if glued is None: + break + whole, parts, part_kinds = glued + test_words.extend(parts) + test_kinds.extend(part_kinds) + others.extend(parts) + kinds.append(whole) + continue kind = set_apart_kind(token.text, lex) # a member its periods settle, or a joined word whose own # chunks settle its member, settles the others, as does a @@ -788,6 +810,8 @@ def _set_apart_clauses(state: ParseState, else: others.append(token.text) kinds.append(kind) + test_words.append(token.text) + test_kinds.append(kind) else: # rules/S1: "a member its writing leaves unsettled is one # too where a credential of the same clause settles it" -- @@ -811,13 +835,13 @@ def _set_apart_clauses(state: ParseState, if last_name >= hi and any( kind != "title" and reads_as_initials(w, kind, state.one_case, lex) - for w, kind in zip(words, kinds)): + for w, kind in zip(test_words, test_kinds)): if ahead is None: ahead = _name_ahead(state, tokens) if ahead[hi - 1]: continue # the token in front of the clause - if not set_apart(words, kinds, lo > 0 + if not set_apart(test_words, test_kinds, lo > 0 and tokens[lo - 1].role is Role.SUFFIX, lex): continue for i, kind in zip(range(lo, hi), kinds): @@ -836,6 +860,36 @@ def _set_apart_clauses(state: ParseState, text=text, span=Span(start, start + len(text))) +#: A single comma glued between two words of a clause's token ('MD,PhD'); +#: a doubled one parts nothing, as a comma beside a comma ends no word +#: (rules/S1, #1268). +_COMMA_SPLIT = re.compile( + "(? tuple[ + SetApartKind, list[str], list[SetApartKind]] | None: + """rules/S1's reading of a clause's word that holds a comma glued + between two words ('MD,PhD', #1268): its words, each of a kind + `set_apart_kind` gives (a member none, as no word of such a token is + settled), and the one kind the whole word takes, which every word + must share, a word of both the title and the suffix vocabulary + taking its neighbours' (a title and a credential in one word, which + the token cannot part, keep the clause its pair's reading); None + where a word is no such word.""" + parts = [w for w in re.split(_COMMA_SPLIT, bare) if w] + kinds: list[SetApartKind] = [] + for part in parts: + kind = set_apart_kind(part, lex) + if kind is None or kind == "member": + return None + kinds.append(kind) + plain = set(kinds) - {"dual"} + if len(plain) > 1: + return None + return (plain.pop() if plain else "dual"), parts, kinds + + def _name_ahead(state: ParseState, tokens: list[WorkToken]) -> list[bool]: """For each token, whether a name word stands after it before the next comma: one walk from the end, each comma part's answer reset diff --git a/tests/v2/cases.py b/tests/v2/cases.py index 2762267c6..52724bfe7 100644 --- a/tests/v2/cases.py +++ b/tests/v2/cases.py @@ -19069,6 +19069,127 @@ def _check_cjk_shape_purity(self) -> None: "as two entries (rules/R1), as the ASCII one does; " "str() writes it back as written. 1.4.0 read nickname " "'Jr， PhD'"), + Case("a_comma_glued_between_credentials_parts_a_set_apart_clause", + "John Smith (MD,PhD)", + {"given": "John", "family": "Smith", "suffix": "MD,PhD"}, + classification="fix(#1268)", + notes="#1268, S1's comma bullet: a comma glued between two credentials parts " + "them for the clause's tests, and the word, set apart " + "whole, keeps its comma as written. Every release from " + "1.4.0 through 3.0.1 read a nickname, 'MD PhD' in 2.0.0 " + "through 2.4.0 and 'MD,PhD' otherwise"), + Case("a_title_glued_to_a_credential_keeps_the_nickname", + "John Smith (Dr.，PhD)", + {"given": "John", "family": "Smith", "nickname": "Dr.，PhD"}, + notes="#1268 boundary: a title and a credential in one word " + "take two roles the word cannot part, so the clause keeps " + "its pair's reading (rules/S1's Accepted)"), + Case("a_glued_generation_and_credential_read_as_one_suffix_word", + "John Smith (Jr.,MD)", + {"given": "John", "family": "Smith", "suffix": "Jr.,MD"}, + classification="fix(#792)", + notes="#1268: one word of two suffix words, as written; 3.0.0 " + "read it so already, 1.4.0 through 2.8.0 a nickname"), + Case("a_member_glued_to_a_credential_keeps_the_nickname", + "John Smith (Ma,PhD)", + {"given": "John", "family": "Smith", "nickname": "Ma,PhD"}, + notes="#1268 boundary: a member of the ambiguous class inside a " + "glued word is settled by nothing (rules/S1's Accepted)"), + Case("a_doubled_comma_inside_a_word_parts_nothing", + "John Smith (MD,,PhD)", + {"given": "John", "family": "Smith", "nickname": "MD,,PhD"}, + notes="#1268 boundary: only a single comma glued between two " + "words parts them"), + Case("a_joined_credential_glued_to_a_member_keeps_the_nickname", + "John Smith (MD/PhD,MBA)", + {"given": "John", "family": "Smith", "nickname": "MD/PhD,MBA"}, + notes="#1268 boundary: as '(Ma,PhD)', 'MBA' being a member"), + Case("commas_opening_a_clause_between_name_words_read_as_before", + "John (,,Jr.) Smith", + {"given": "John", "family": "Smith", "suffix": "Jr."}, + classification="fix(#792)", + notes="#1268 boundary: commas at a word's edge are no part of " + "it, as before #1268"), + Case("a_glued_comma_in_a_nickname_is_kept_as_written", + "Jane (Janey,Jo) Smith", + {"given": "Jane", "family": "Smith", "nickname": "Janey,Jo"}, + notes="#1268 boundary: a clause holding a name word is the " + "nickname, its text as written (rules/N1)"), + Case("a_glued_comma_beside_a_name_word_keeps_the_nickname", + "John Smith (PhD,Bob)", + {"given": "John", "family": "Smith", "nickname": "PhD,Bob"}, + notes="#1268 boundary: as 'John Smith (PhD, Bob)' reads"), + Case("a_glued_comma_before_initials_and_a_name_word_keeps_the_nickname", + "Justin (JP,PhD) Stearns", + {"given": "Justin", "family": "Stearns", "nickname": "JP,PhD"}, + notes="#1268 boundary: rules/S1's initials before a name word, " + "as 'Justin (JP, PhD) Stearns' reads"), + Case("a_glued_comma_parts_a_period_final_set_apart_clause", + "John Smith (USN,Ret.)", + {"given": "John", "family": "Smith", "suffix": "USN, Ret."}, + notes="#1268 boundary: as at the base, the period escape reading " + "the clause bare, where the comma divides"), + Case("a_comma_before_a_nakaguro_ends_no_word", + "John Smith (MD,・PhD)", + {"given": "John", "family": "Smith", "nickname": "MD, PhD"}, + classification="fix(#653)", + notes="#1268 boundary: as at the base, the clause's one " + "whitespace word standing against two tokens"), + Case("a_nakaguro_alone_in_a_clause_keeps_the_nickname", + "John Smith (MD・PhD)", + {"given": "John", "family": "Smith", "nickname": "MD PhD"}, + classification="fix(#272)", + notes="#1268 boundary: a clause with no comma reads as at the base"), + Case("a_nakaguro_beside_a_space_reads_as_before", + "John Smith (MD・ PhD)", + {"given": "John", "family": "Smith", "suffix": "MD PhD"}, + classification="fix(#792)", + notes="#1268 boundary: a clause with no comma reads as at the base"), + Case("a_comma_behind_a_zero_width_space_ends_no_word", + "John Smith (MD\u200b,PhD)", + {"given": "John", "family": "Smith", "nickname": "MD ,PhD"}, + classification="fix(#851)", + notes="#1268 boundary: as at the base; the comma opens the word " + "tokenize makes after the zero-width space"), + Case("a_zero_width_space_before_a_comma_keeps_initials_a_nickname", + "Justin (JP\u200b,PhD) Stearns", + {"given": "Justin", "family": "Stearns", "nickname": "JP ,PhD"}, + classification="fix(#851)", + notes="#1268 boundary: as the row above"), + Case("a_doubled_comma_in_a_clause_divides_as_before", + "John Smith (MD,, PhD)", + {"given": "John", "family": "Smith", "suffix": "MD, PhD"}, + classification="fix(#792)", + notes="#1268 boundary: a doubled comma parts nothing, as at the " + "base"), + Case("a_doubled_comma_before_a_name_word_keeps_the_nickname", + "John (MD,, Jr.) Smith", + {"given": "John", "family": "Smith", "nickname": "MD,, Jr."}, + classification="fix(#792)", + notes="#1268 boundary: as the row above"), + Case("a_comma_before_a_bidi_mark_ends_no_word", + "John Smith (MD,\u200f PhD)", + {"given": "John", "family": "Smith", "suffix": "MD, PhD"}, + classification="fix(#792)", + notes="#1268 boundary: a comma at a word's edge parts nothing, as " + "at the base"), + Case("a_title_before_a_comma_and_a_bidi_mark_stays_a_title", + "John Smith (Dr.,\u200f)", + {"title": "Dr.", "given": "John", "family": "Smith"}, + classification="fix(#792)", + notes="#1268 boundary: as the row above"), + Case("a_configured_maiden_pair_sets_apart_glued_credentials", + "John Smith (MD,PhD)", + {"given": "John", "family": "Smith", "suffix": "MD,PhD"}, + policy=Policy(maiden_delimiters=frozenset({("(", ")")})), + classification="fix(#1268)", + notes="rules/S1 takes a configured maiden pair's clause first " + "(M1), as 'John Smith (MD, PhD)' reads under the same " + "policy; 3.0.1 read maiden 'MD,PhD'"), + Case("a_comma_opening_a_clause_divides_nothing", + "John (,JP) Smith", + {"given": "John", "family": "Smith", "nickname": ",JP"}, + notes="#1268 boundary: a comma opening a word parts nothing"), Case("period_joined_member_before_an_arabic_comma_reports", "John Smith (Msc.Ed، Bob)", {"given": "John", "family": "Smith", "nickname": "Msc.Ed، Bob"}, diff --git a/tests/v2/test_parser.py b/tests/v2/test_parser.py index 940a6549a..77da07c21 100644 --- a/tests/v2/test_parser.py +++ b/tests/v2/test_parser.py @@ -1465,6 +1465,11 @@ def test_revise_reads_a_glued_honorific_on_its_own() -> None: #: (rules/T8) -- 'M.Filho' revising to 'M. Filho', as a CJK value's #: sub-parse divides it (`revise(family="김민준")`). _GLUED_IN_SUFFIX = frozenset({"Smith, Pedro M.Filho"}) +#: The fourth (2026-11-11, #1268): a clause's word of credentials glued at +#: a comma, which the whole name sets apart whole and keeps as written +#: (rules/S1), and the value's own sub-parse, holding no clause, divides at +#: the comma as any comma outside one: 'MD,PhD' revising to 'MD, PhD'. +_GLUED_COMMA_IN_SUFFIX = frozenset({"John Smith (MD,PhD)"}) def _suffix_bearing_corpus_names() -> list[str]: @@ -1472,7 +1477,8 @@ def _suffix_bearing_corpus_names() -> list[str]: p = Parser() return [n for n in _CORPUS_NAMES if p.parse(n).suffix and n not in _HONORIFIC_PEEL - and n not in _MARKER_IN_SUFFIX and n not in _GLUED_IN_SUFFIX] + and n not in _MARKER_IN_SUFFIX and n not in _GLUED_IN_SUFFIX + and n not in _GLUED_COMMA_IN_SUFFIX] def test_the_known_round_trip_exceptions_are_one_limit() -> None: @@ -1524,6 +1530,16 @@ def divides_on_its_own(name: str) -> bool: assert not [n for n in _suffix_bearing_corpus_names() if divides_on_its_own(n)] + # the glued-comma limit, characterized the same way: a word of the + # parsed suffix holding a comma between two of its characters + def holds_a_glued_comma(name: str) -> bool: + return any(re.search(r"[^,],[^,]", word) + for word in p.parse(name).suffix.split()) + + assert all(holds_a_glued_comma(n) for n in _GLUED_COMMA_IN_SUFFIX) + assert not [n for n in _suffix_bearing_corpus_names() + if holds_a_glued_comma(n)] + def test_the_suffix_bearing_corpus_is_not_empty() -> None: # 367 on 2026-09-06; a floor, so a filter that empties the diff --git a/tests/v2/test_regex_sync.py b/tests/v2/test_regex_sync.py index cbdf1f728..60ab68acb 100644 --- a/tests/v2/test_regex_sync.py +++ b/tests/v2/test_regex_sync.py @@ -150,6 +150,8 @@ def test_dotted_initial_is_the_period_alternative_of_initial() -> None: # trailing shape reads the same set ("_vocab", "_HYPHENS"): None, ("_post_rules", "_HYPHENS"): ("_vocab", "_HYPHENS"), + # a single comma glued inside a clause's word (rules/S1, #1268) + ("_post_rules", "_COMMA_SPLIT"): None, # render's own patterns, v1's "spaces", "space_before_comma", # "mac" and "word"; the v1 table they were pinned to until #1008 # was their only other copy diff --git a/tools/differential/corpus_cjk.jsonl b/tools/differential/corpus_cjk.jsonl index b60c40ad7..bc714c7d0 100644 --- a/tools/differential/corpus_cjk.jsonl +++ b/tools/differential/corpus_cjk.jsonl @@ -8,6 +8,9 @@ "Dr 김민준씨, V." "Dr. 김민준씨, MA" "J.씨" +"John Smith (MD,・PhD)" +"John Smith (MD・ PhD)" +"John Smith (MD・PhD)" "John Smith 田.中." "John Smith, Dr. J.씨" "John 王" diff --git a/tools/differential/corpus_rules.jsonl b/tools/differential/corpus_rules.jsonl index f14cb4a27..5bf996167 100644 --- a/tools/differential/corpus_rules.jsonl +++ b/tools/differential/corpus_rules.jsonl @@ -639,6 +639,7 @@ "John Smith (Dr, Ma)" "John Smith (Dr., PhD)" "John Smith (Dr.፣ PhD)" +"John Smith (Dr.，PhD)" "John Smith (Ed)" "John Smith (Ed, Jr/Sr)" "John Smith (Hon)" @@ -664,6 +665,8 @@ "John Smith (MBA, PhD)" "John Smith (MD ,PhD)" "John Smith (MD,) (PhD)" +"John Smith (MD,,PhD)" +"John Smith (MD,PhD)" "John Smith (MD-PhD)" "John Smith (MD/Bob)" "John Smith (MD/PhD)" @@ -674,6 +677,7 @@ "John Smith (Ma, Ms)" "John Smith (Ma, Ph.D.)" "John Smith (Ma, PhD)" +"John Smith (Ma,PhD)" "John Smith (Mr)" "John Smith (Mr, PhD)" "John Smith (Msc.Ed)" diff --git a/tools/differential/snapshot.jsonl b/tools/differential/snapshot.jsonl index 9c159c9a9..d27d49530 100644 --- a/tools/differential/snapshot.jsonl +++ b/tools/differential/snapshot.jsonl @@ -1952,6 +1952,7 @@ {"name": "John Smith (Dr, Ma)", "given": "John", "family": "Smith", "nickname": "Dr, Ma", "ambiguities": ["SUFFIX_OR_NICKNAME"], "alternatives": ["SUFFIX_OR_NICKNAME: credential as suffix (vocabulary)"], "initials": "J. S."} {"name": "John Smith (Dr., PhD)", "title": "Dr.", "given": "John", "family": "Smith", "suffix": "PhD", "initials": "J. S."} {"name": "John Smith (Dr.፣ PhD)", "title": "Dr.", "given": "John", "family": "Smith", "suffix": "PhD", "initials": "J. S."} +{"name": "John Smith (Dr.，PhD)", "given": "John", "family": "Smith", "nickname": "Dr.，PhD", "initials": "J. S."} {"name": "John Smith (Ed)", "given": "John", "family": "Smith", "nickname": "Ed", "ambiguities": ["SUFFIX_OR_NICKNAME"], "alternatives": ["SUFFIX_OR_NICKNAME: credential as suffix (vocabulary)"], "initials": "J. S."} {"name": "John Smith (Ed, Jr/Sr)", "given": "John", "family": "Smith", "nickname": "Ed, Jr/Sr", "ambiguities": ["SUFFIX_OR_NICKNAME"], "alternatives": ["SUFFIX_OR_NICKNAME: credential as suffix (vocabulary)"], "initials": "J. S."} {"name": "John Smith (Hon)", "given": "John", "family": "Smith", "nickname": "Hon", "initials": "J. S."} @@ -1977,9 +1978,14 @@ {"name": "John Smith (MBA, PhD)", "given": "John", "family": "Smith", "suffix": "MBA, PhD", "initials": "J. S."} {"name": "John Smith (MD ,PhD)", "given": "John", "family": "Smith", "suffix": "MD, PhD", "initials": "J. S."} {"name": "John Smith (MD,) (PhD)", "given": "John", "family": "Smith", "suffix": "MD, PhD", "initials": "J. S."} +{"name": "John Smith (MD,,PhD)", "given": "John", "family": "Smith", "nickname": "MD,,PhD", "initials": "J. S.", "capitalized_force": "John \"Md,,PhD\" Smith"} +{"name": "John Smith (MD,PhD)", "given": "John", "family": "Smith", "suffix": "MD,PhD", "initials": "J. S."} +{"name": "John Smith (MD,・PhD)", "given": "John", "family": "Smith", "nickname": "MD, PhD", "initials": "J. S.", "capitalized_force": "John \"Md,・PhD\" Smith"} {"name": "John Smith (MD-PhD)", "given": "John", "family": "Smith", "suffix": "MD-PhD", "initials": "J. S."} {"name": "John Smith (MD/Bob)", "given": "John", "family": "Smith", "nickname": "MD/Bob", "initials": "J. S.", "capitalized_force": "John \"Md/Bob\" Smith"} {"name": "John Smith (MD/PhD)", "given": "John", "family": "Smith", "suffix": "MD/PhD", "initials": "J. S."} +{"name": "John Smith (MD・ PhD)", "given": "John", "family": "Smith", "suffix": "MD PhD", "initials": "J. S."} +{"name": "John Smith (MD・PhD)", "given": "John", "family": "Smith", "nickname": "MD PhD", "initials": "J. S.", "capitalized_force": "John \"Md・PhD\" Smith"} {"name": "John Smith (Ma PhD)", "given": "John", "family": "Smith", "suffix": "Ma PhD", "initials": "J. S.", "capitalized_force": "John Smith MA PhD"} {"name": "John Smith (Ma)", "given": "John", "family": "Smith", "nickname": "Ma", "ambiguities": ["SUFFIX_OR_NICKNAME"], "alternatives": ["SUFFIX_OR_NICKNAME: credential as suffix (vocabulary)"], "initials": "J. S."} {"name": "John Smith (Ma, Ed)", "given": "John", "family": "Smith", "nickname": "Ma, Ed", "ambiguities": ["SUFFIX_OR_NICKNAME"], "alternatives": ["SUFFIX_OR_NICKNAME: credential as suffix (vocabulary)", "SUFFIX_OR_NICKNAME: credential as suffix (vocabulary)"], "initials": "J. S."} @@ -1987,6 +1993,7 @@ {"name": "John Smith (Ma, Ms)", "given": "John", "family": "Smith", "nickname": "Ma, Ms", "ambiguities": ["SUFFIX_OR_NICKNAME"], "alternatives": ["SUFFIX_OR_NICKNAME: credential as suffix (vocabulary)"], "initials": "J. S."} {"name": "John Smith (Ma, Ph.D.)", "given": "John", "middle": "Smith", "family": "Ma", "suffix": "Ph.D.", "ambiguities": ["SUFFIX_OR_NAME"], "alternatives": ["SUFFIX_OR_NAME: credential as suffix (vocabulary)"], "initials": "J. S. M."} {"name": "John Smith (Ma, PhD)", "given": "John", "family": "Smith", "suffix": "Ma, PhD", "initials": "J. S.", "capitalized_force": "John Smith MA, PhD"} +{"name": "John Smith (Ma,PhD)", "given": "John", "family": "Smith", "nickname": "Ma,PhD", "initials": "J. S."} {"name": "John Smith (Mr)", "given": "John", "family": "Smith", "nickname": "Mr", "initials": "J. S."} {"name": "John Smith (Mr, PhD)", "title": "Mr", "given": "John", "family": "Smith", "suffix": "PhD", "initials": "J. S."} {"name": "John Smith (Msc.Ed)", "given": "John", "family": "Smith", "suffix": "Msc.Ed", "initials": "J. S.", "capitalized_force": "John Smith MSc.Ed"}