From 97fec9245ed52dfc59e684e5e5e6b9326481c03e Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Fri, 30 Oct 2026 16:21:35 -0700 Subject: [PATCH 1/4] Leave a listed given name out of two_family_names' family (rules/O6, #959) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Policy(two_family_names=True) moves the name word before the family into it, and in "José Luis Pérez" that word is a given name: most of what the switch gets wrong. O6 now leaves a word listed in the new Lexicon.given_names where it stands, under every order, so the name reads as with the switch off; a listed word behind a particle is a surname unit and is taken as before. With two_given_names on too, the word left is O7's to take: "José Luis Pérez" reads given "José Luis". - Lexicon.given_names: empty by default, read by O6 alone, matched as written or with acute/grave accents and diaeresis dropped (ñ and ç kept); marked _ADDED_IN "3.0.0" so 2.x pickles load it empty. - nameparser/locales/_es_given_names.py: 1,000 words generated by tools/es_given_names.py from INE's 2025 census name and surname tables (CC BY 4.0), at least 90% given use, for the 3g es pack to carry. - rules/O6 amended (Except clause, Accepted cost, Signal), rules/O7's precedence states the interaction; decisions/O6 records the source, licence, cutoff measurement per country and the Open items. - Rule examples can name a policy with a lexicon fragment (two_family_names+es_given_names). Refs #959 Co-Authored-By: Claude Opus 5.5 --- docs/customize.rst | 24 ++ docs/design/decisions/O6-two-family-names.md | 58 +++++ docs/design/decisions/O7-two-given-names.md | 4 +- docs/design/rules/O6-two-family-names.md | 14 +- docs/design/rules/O7-two-given-names.md | 5 +- docs/design/rules/README.md | 2 +- docs/release_log.rst | 1 + nameparser/_lexicon.py | 15 +- nameparser/_pipeline/_post_rules.py | 14 +- nameparser/_pipeline/_vocab.py | 44 +++- nameparser/locales/_es_given_names.py | 152 ++++++++++++ tests/v2/pipeline/test_post_rules.py | 129 ++++++++++ tests/v2/pipeline/test_vocab.py | 7 +- tests/v2/rules_budget.json | 4 +- tests/v2/rules_doc.py | 23 ++ tests/v2/test_es_given_names.py | 144 +++++++++++ tests/v2/test_layering.py | 6 +- tests/v2/test_lexicon.py | 25 ++ tests/v2/test_parser.py | 5 +- tests/v2/test_rules_doc.py | 15 +- tests/v2/test_rules_doc_grammar.py | 5 + tools/differential/corpus_rules.jsonl | 8 + tools/differential/snapshot.jsonl | 8 + tools/es_given_names.py | 238 +++++++++++++++++++ 24 files changed, 926 insertions(+), 24 deletions(-) create mode 100644 nameparser/locales/_es_given_names.py create mode 100644 tests/v2/test_es_given_names.py create mode 100644 tools/es_given_names.py diff --git a/docs/customize.rst b/docs/customize.rst index c8a0912cd..3d0ba7615 100644 --- a/docs/customize.rst +++ b/docs/customize.rst @@ -1023,6 +1023,30 @@ breaks 8,234 names of 20,000 and mends 265. The parser reports nothing for the names the switch moves: the shape it reads is the same either way. +A given name standing before the family is most of what the switch gets +wrong, as in ``"José Luis Pérez"``. Words that are given names and +rarely surnames tell the two shapes apart, so the switch leaves a word +listed in the lexicon's ``given_names`` where it stands. The list is +empty by default; an entry written without accents matches the word +with them: + +.. doctest:: + + >>> listed = Parser(policy=Policy(two_family_names=True), + ... lexicon=Lexicon.default().add(given_names={"luis"})) + >>> two_names.parse("José Luis Pérez").family + 'Luis Pérez' + >>> luis = listed.parse("José Luis Pérez") + >>> luis.middle, luis.family + ('Luis', 'Pérez') + +Among ORCID's researchers, a list of the thousand words most borne as +given names in Spain and mostly not as surnames (from Spain's 2025 +census) raised the switch's agreement from 93.4% to 95.3% in Spain and +from 61.2% to 89.3% in Argentina, where the switch alone breaks most +names (measured 2026-10-30). A listed word that is someone's first +surname, as given names sometimes are, stays a middle name. + .. _two-given-names: Names with a compound given name diff --git a/docs/design/decisions/O6-two-family-names.md b/docs/design/decisions/O6-two-family-names.md index 8bae22ee3..38e6f31bd 100644 --- a/docs/design/decisions/O6-two-family-names.md +++ b/docs/design/decisions/O6-two-family-names.md @@ -10,6 +10,10 @@ - The switch runs before `middle_as_family` and is not refused beside it: given-first, the two together read as `middle_as_family` alone — 2026-10-24 #754. - The switch reports nothing — 2026-10-24 #754. - The switch reports no alternative either, so the reading it did not take is not surfaced: the caller declared the convention — 2026-10-27 #752 (Derek). +- A listed given name (`Lexicon.given_names`) standing alone is not taken, under every order, and the name then reads as with the switch off; a listed word behind a particle is taken as before, and with the list empty the switch reads as before; no other rule reads the list, the default carries none, and the veto reports nothing — 2026-10-30 #959. +- The list a pack carries for Spanish is generated, not per-word vocabulary: INE's 2025 census tables, pooled over every name borne in Spain, cut at a measured rank (1,000 words, 90% given use) by `tools/es_given_names.py`; a change to it is a cutoff change — 2026-10-30 #959. + +Open: hyphenated surnames block the switch (`Andrés Sánchez-Cid Torres` keeps `Sánchez-Cid` a middle name), 271 of the 466 Spanish records the issue's held-out per-country list still read wrong (measured on the issue, 2026-10-30) — #959. No rule example shows both switches on over a given name a particle completes (`María del Carmen de la Vega`, given `María del Carmen` under O7 alone and family `del Carmen de la Vega` with O6 on too) — #959. ## Log @@ -59,3 +63,57 @@ Measured on master at 598167f4 over the 2025 Public Data File's extract fetched 2026-10-27. Recompute: `local_agreement.read` per record of `local_corpora.load("orcid-researchers")` with `FIELDED["orcid-researchers"]`, under `Parser()` and under `Parser(policy=Policy(two_family_names=True))` with `_title_words` of the default lexicon, grouped by the record's `country`; a record is fixed where it agrees only with the switch, broken where it agrees only without it, and the last column counts the broken records whose `family` box holds one word. A corpus capped at 20,000 records a country shows each country's convention, not how many people write it. - 2026-10-29 #754, sub-phase 4d — THE TWO UNITS THIS RULE READS REACH A VIEW: `surname2` holds the one written second, `family` keeping both as before, and the rule marks that unit's words where it takes its word; nothing it reads or declines moved. The view, the mark and their measurement are rules/R10 and decisions/R10's 2026-10-29 entry. +- 2026-10-30 #959 — A LISTED GIVEN NAME IS NOT TAKEN: an amendment to rules/O6's Except bullet "What is taken must be one name word a family name could be", not a new rule. In `José Luis Pérez` the word the switch takes is a given name, and that shape is most of what the switch gets wrong (the 2026-10-27 entry: the records it breaks nearly all hold a one-word family box). A Spanish reader settles it by vocabulary: given names are a concentrated set and few words are common in both roles, so a word only a given name is says the name holds one family name. The rule reads `Lexicon.given_names`, empty by default, and a pack carries the list; with the list empty the switch reads exactly as before (the snapshot gains the rule's example texts and moves no reading). + What it settles, and why each way: + - **Where the list lives: generated, as locales.ZH's surnames are.** `nameparser/locales/_es_given_names.py` holds the list in a block `tools/es_given_names.py` writes, for the `es` pack (sub-phase 3g of #754) to carry as a lexicon fragment, `Lexicon(given_names=GIVEN_NAMES)`. Weighed against: `given` readings in an `es` language file, which per-word TOML would carry a thousand words judged one at a time, where what is decided is a cutoff; the evidence for the set is the tool's docstring and this entry, and a change to it is decided as a cutoff change, as ZH's top 300 is (decisions/W1). The TOML build's check 11, which refuses a given reading on a word another reading claims as no name, is replaced by a test holding the list disjoint from the default's never-given particles, titles, suffixes and conjunctions (tests/v2/test_es_given_names.py, with its recorded negative control); the tool leaves every particle out. + - **The source: INE.** The Instituto Nacional de Estadística publishes every given name borne by at least 20 residents of Spain, by sex, and every surname borne as a first surname by at least 20, with its counts as first and second surname, both from the census of 1 January 2025. Its legal notice licenses the site's statistics as "la Creative Commons Reconocimiento 4.0 (CC BY 4.0)" (https://www.ine.es/aviso_legal, read 2026-10-30), asking the reuser to cite "Elaboración propia con datos extraídos del sitio web del INE: www.ine.es" and the date of the data, which the module's docstring does. A word's given uses are the bearers of the names holding it; its surname uses, the bearers of the surnames holding it, less those holding it behind a particle (`de Miguel`), which the switch never takes as the bare word (with them counted, `Miguel` is 89% given and falls out; without, 92%). A word is a candidate where at least 90% of its uses are given, #959's share; INE writes names without acute accents, so the parse matches a word with its accents and diaeresis dropped and the tilde on n and the cedilla on c kept (`given_name_key`), and a caller's entry also as written. Words common in both roles stay out (`Martín` 10% given, `Vicente` 63%, `Santiago` 57%, `Cruz` 18%). Weighed against: a list built from ORCID's boxes, which #959 measured on a held-out half; INE is licensed for redistribution, counts a whole population rather than researchers, and is not the population scored, so no held-out split is needed. + - **One pooled list, Spain's.** INE counts Spain's residents, Latin American-born ones among them, and #959 found a list pooled over six countries within half a point of per-country lists everywhere but Argentina and Peru. Peru is where the pooled list costs most: there it fixes about as many records as it breaks at every cutoff, the breaks being first surnames that are given names in Spain (`Jorge` 12, `Carlos` 11, `Pablo` 6 at the 1,000 cut), where #959's per-country list, learned from Peru's own boxes, broke 58. + - **The cutoff: 1,000 words** (84.3% of the given-word uses in INE's table). Measured per country with O6 on, the list cut at 100 to all 16,559 candidates; agreement in each of the six double-surname countries is within 0.3 points of the 2,000-word list at 1,000 (Venezuela the 0.3), and a band past it buys at most a tenth of a net fix per added word in those six (1,000 → 2,000: 134 fixed, 35 broken; 2,000 → 5,000: 128 fixed, 36 broken, Chile breaking as many as it fixes), where the first hundred words bought sixteen. Argentina gains 1.6 points more by 2,000 and 2.4 by all, which 3g weighs if it turns the switch on there. + - **Family-first, symmetric.** Under a family-first order the switch takes the unit after the family, and a listed one stays where it is: `Pérez José Luis` keeps family `Pérez`. No population the corpora hold writes Spanish names family-first with the fields stated, so it is measured on ORCID's boxes rendered family box first, which is a rendering, not anyone's writing. + - **A bare word only.** The veto reads a unit that is one word with no particle; a listed word behind a particle is a surname unit and is taken as before (`Ana de Miguel de la Vega` keeps family `de Miguel de la Vega`), which is what the tool's count assumes in leaving such surnames out. The first draft read the word inside a particle unit too, and the design-docs review found it reading `Ana de Miguel de la Vega` middle `de Miguel`. + - **O7.** With both switches on, the word O6 leaves is a middle O7 takes where it stands beside the given name, so `José Luis Pérez` reads given `José Luis`, family `Pérez` (rules/O7's precedence); `Ana María Luis Pérez` keeps middle `Luis`, O7 taking `María`. Whether O7 reads the list itself is decisions/O7's Open question. + - **No report**, as for the switch (the 2026-10-24 entry): the caller declared the convention and the list. + - **Pickling.** The field carries `_ADDED_IN` "3.0.0" and an empty default, so a `Lexicon`, `Parser` or locale pack pickled by 2.x loads with no list. That is what such a pickle meant, no pack filling the field before 3.0, so this field is the case decisions/A1's 2026-10-30 caveat does not reach: the silent drift it warns of comes from a marked field the default vocabulary fills, and the default leaves this one empty. `tests/v2/test_lexicon.py` loads the committed 2.8.0 pickle with the field empty, and refuses it with the marker taken off. + Measured 2026-10-30 on the branch, the 2025 Public Data File extract fetched 2026-10-27 (`orcid-researchers`), every record with both boxes filled; all boxes agreeing, by default, with the switch, and with the switch and the 1,000-word list, and the records the list fixes and breaks against the switch alone: + + | Country | Records | Default | Switch | Switch + list | Fixed | Broken | + |---|---:|---:|---:|---:|---:|---:| + | ES | 19,527 | 48.8% | 93.4% | 95.3% | 434 | 60 | + | MX | 19,527 | 31.9% | 93.9% | 95.0% | 336 | 122 | + | CO | 19,359 | 24.1% | 94.5% | 97.0% | 561 | 78 | + | PE | 19,471 | 13.5% | 96.0% | 95.9% | 178 | 189 | + | CL | 19,363 | 46.9% | 94.9% | 97.3% | 495 | 25 | + | VE | 19,111 | 38.7% | 89.1% | 93.2% | 849 | 73 | + | AR | 19,377 | 82.5% | 61.2% | 89.3% | 5,500 | 59 | + | EC | 19,274 | 25.9% | 95.0% | 96.3% | 354 | 88 | + | BO | 15,319 | 20.7% | 94.9% | 96.1% | 297 | 111 | + | CU | 19,055 | 16.1% | 93.9% | 94.2% | 213 | 148 | + | UY | 7,645 | 61.8% | 93.2% | 97.0% | 308 | 16 | + + By cutoff, all boxes agreeing with the switch and the list: + + | Country | 100 | 200 | 500 | 1,000 | 2,000 | 5,000 | 16,559 | + |---|---:|---:|---:|---:|---:|---:|---:| + | ES | 94.9% | 95.0% | 95.2% | 95.3% | 95.4% | 95.4% | 95.5% | + | MX | 94.5% | 94.8% | 94.9% | 95.0% | 95.0% | 95.2% | 95.2% | + | CO | 96.0% | 96.4% | 96.9% | 97.0% | 97.1% | 97.1% | 97.2% | + | PE | 95.9% | 96.0% | 95.9% | 95.9% | 95.9% | 96.0% | 95.9% | + | CL | 96.6% | 97.0% | 97.2% | 97.3% | 97.3% | 97.3% | 97.3% | + | VE | 92.0% | 92.7% | 93.0% | 93.2% | 93.5% | 93.7% | 93.8% | + | AR | 77.3% | 83.2% | 86.8% | 89.3% | 90.9% | 91.5% | 91.7% | + + The records each cut fixes and breaks against the switch alone, which is what chose the cut: + + | Country | 100 | 200 | 500 | 1,000 | 2,000 | 5,000 | 16,559 | + |---|---:|---:|---:|---:|---:|---:|---:| + | ES | 340 / 44 | 373 / 51 | 408 / 57 | 434 / 60 | 448 / 63 | 461 / 63 | 476 / 64 | + | MX | 207 / 84 | 270 / 92 | 310 / 114 | 336 / 122 | 350 / 130 | 378 / 137 | 392 / 146 | + | CO | 351 / 52 | 435 / 62 | 532 / 74 | 561 / 78 | 582 / 84 | 598 / 86 | 610 / 91 | + | PE | 105 / 117 | 136 / 139 | 161 / 172 | 178 / 189 | 196 / 201 | 220 / 213 | 227 / 233 | + | CL | 350 / 12 | 425 / 13 | 468 / 21 | 495 / 25 | 502 / 27 | 511 / 36 | 515 / 47 | + | VE | 590 / 44 | 741 / 54 | 805 / 64 | 849 / 73 | 909 / 77 | 947 / 83 | 973 / 88 | + | AR | 3,164 / 35 | 4,308 / 41 | 5,014 / 49 | 5,500 / 59 | 5,820 / 63 | 5,948 / 65 | 5,985 / 69 | + + The same boxes rendered family box first, read under FAMILY_FIRST on both sides: ES 47.6% → 91.4% → 94.5% (681 fixed, 68 broken), MX 31.4% → 94.6% → 96.3% (465, 132), CO 23.9% → 94.5% → 97.6% (640, 46), PE 12.6% → 95.9% → 95.9% (203, 205), CL 46.3% → 93.4% → 97.0% (729, 38), VE 38.5% → 88.6% → 93.3% (949, 44), AR 82.3% → 59.8% → 90.8% (6,068, 53). + PSC (`psc-individuals`, the fetch recorded on #849), a second population the list was not built from, by nationality, default → switch → switch and list, fixed and broken: Spanish 6,089 records 31.8% → 80.2% → 82.2% (150, 31), Mexican 402 25.9% → 80.1% → 82.1% (13, 5), Colombian 377 29.2% → 84.4% → 87.0% (15, 5), Venezuelan 237 40.5% → 75.9% → 80.6% (13, 2), Chilean 151 35.1% → 77.5% → 84.8% (12, 1), Peruvian 116 21.6% → 81.0% → 83.6% (5, 2), Argentine 320 81.9% → 45.6% → 81.9% (121, 5). + Recompute: `uv run --with openpyxl --with xlrd python tools/es_given_names.py --top N` writes the list for a cutoff (`--write`) and prints its coverage. For ORCID, `local_agreement.read` per record of `local_corpora.load("orcid-researchers")` with `FIELDED["orcid-researchers"]`, records with both boxes, grouped by `country`, under `Parser()`, `Parser(policy=Policy(two_family_names=True))` and that policy with `lexicon=Lexicon.default().add(given_names=...)` holding the tool's first N words, each with `_title_words` of its own lexicon; family-first, the record's `name` replaced by `family + " " + given` and each policy given `name_order=FAMILY_FIRST`. For PSC, `local_agreement.score(rows, FIELDED["psc-individuals"], parser, jobs=1)` over the records whose `group_of` is the nationality, each cell a group's `agreed` over its `scored`. A record is fixed where it agrees only with the list, broken where it agrees only without it. The comparator is the switch alone on the same tree; `local_agreement.py orcid-researchers --base` and `psc-individuals --base` against master match the base, the default reading no list. diff --git a/docs/design/decisions/O7-two-given-names.md b/docs/design/decisions/O7-two-given-names.md index 715afec75..74a951231 100644 --- a/docs/design/decisions/O7-two-given-names.md +++ b/docs/design/decisions/O7-two-given-names.md @@ -7,6 +7,7 @@ - The switch reads after `two_family_names` and before `middle_as_family`, and is refused beside neither — 2026-10-30 #704. - A particle chain running to the end of the name stays the family under the switch (`Maria do Carmo Silva`); reading it as a given name is not this rule's — 2026-10-30 #704. - The switch reports nothing — 2026-10-30 #704. +- With O6's switch on, a listed given name O6 leaves where it is (decisions/O6's 2026-10-30 #959 entry) is a middle this rule takes where it stands beside the given name, so `José Luis Pérez` reads given `José Luis` with both on — 2026-10-30 #959. ## Log @@ -36,4 +37,5 @@ - **CLDR** (`cldr_fields.jsonl`'s given-first renderings with a stated given name, given agreeing): over all 880, default 636, O7 568, O6 636, O6+O7 746; over `es`'s 8, 6 → 8 under O6+O7; over `pt`'s 8, 7 under O6+O7 and 5 under O7 alone. - **TSE** (`snapshot.py --local tse-donors --base`, Brazilian donors' names as filed): 689,057 of 854,531 entries read differently under O7, 684,625 of them in given and middle alone and 310 in an ambiguity's alternative too, and with O6 on in both trees 270,169 (268,478 in given and middle alone), measured on the final commit; the corpus states no fields, so it counts reach, not agreement. Recompute: the ORCID counts read `raw.jsonl` of the 2026-10-27 fetch, records with a family box, the three- and four-word split over records whose boxes hold no hyphen and no word of `Lexicon.default().particles` (conjunctions not excluded), by the record's `country`; the particle-inside count over BR, PT, ES, MX, AR, CO, CL, PE, VE, EC, UY, BO, PY, CR, GT, CU, DO, HN, SV, NI, PA and PR, a given box whose word at neither end is in `particles`, its object the next word (the next two after a second particle), against family boxes of two or more words opening with one. The PSC table is `local_agreement.score(rows, FIELDED[PSC], Parser(policy=...), jobs=1)` in one process, `rows` the records whose `group_of` is the filers' nationality, each table cell a group's `agreed` over its `scored`: every box compared case-folded, the title too, a parse's suffix appended to the family before the surname box is compared, and a record whose title box holds an unlisted word that disagrees set aside and not scored. An earlier draft of this table compared the raw boxes without that read and gave a few tenths of a point less. The CLDR tables score each rendering's `given` against `parse()` under each `Policy` in one process; the local corpora's runs are `--tree` a checkout whose `two_given_names` (and for O6+O7 `two_family_names`) default is flipped, `--base` one at master `40675146` flipped the same way for O6. - - **Open:** which particle phrases complete a given name, and whether that is a reading on the word or a list the rule kind holds (the particle-chain bullet above), #704; whether `es` and `pt` turn the switch on, which 3g decides against these measurements, #754; deriving `DEVIATES` from the rules (decisions/vocabulary-model's 2026-10-29 4f entry, item 6), which this rule leaves as it found it, having no surface in the name to declare, #754. + - **Open:** which particle phrases complete a given name, and whether that is a reading on the word or a list the rule kind holds (the particle-chain bullet above), #704, and whether this rule reads O6's given-name list for it (`María del Carmen` against `João da Silva`), #959; whether `es` and `pt` turn the switch on, which 3g decides against these measurements, #754; deriving `DEVIATES` from the rules (decisions/vocabulary-model's 2026-10-29 4f entry, item 6), which this rule leaves as it found it, having no surface in the name to declare, #754. +- 2026-10-30 #959 — THE WORD O6'S GIVEN-NAME LIST LEAVES IS THIS RULE'S. rules/O6 now leaves a listed given name where it stands (decisions/O6's entry of this date), so with both switches on the middle it leaves is this rule's to take where it is the unit beside the given name: `José Luis Pérez` reads given `José Luis`, family `Pérez`, where with the list empty O6 takes `Luis` first and the name keeps one given name. Where the given part holds two units before it, this rule takes the first and the listed word stays a middle (`Ana María Luis Pérez`: given `Ana María`, middle `Luis`), so a name of four words reads two and two only where O6 took its third (found by the #959 design-docs review). Nothing in this rule moved; its precedence clause states the reading with an example. Whether this rule reads the list itself is Open above. diff --git a/docs/design/rules/O6-two-family-names.md b/docs/design/rules/O6-two-family-names.md index 3fd56e174..b78b5289b 100644 --- a/docs/design/rules/O6-two-family-names.md +++ b/docs/design/rules/O6-two-family-names.md @@ -8,6 +8,8 @@ ``` "Gabriel García Márquez" two_family_names → family="García Márquez" "Ana María García López" two_family_names → middle="María" + "Ana Martín López" two_family_names+es_given_names → family="Martín López" + "Ana de Miguel de la Vega" two_family_names+es_given_names → family="de Miguel de la Vega" ``` - **Under a family-first order** the word after the family joins it, and what is left is laid out as the positions after a family, as P1's family-first fold lays out its own remainder. ``` @@ -57,8 +59,12 @@ ``` ", Ana García López" two_family_names → family="" ``` -- **What is taken must be one name word a family name could be**: a part counting as two words (a hyphenated given name, a bound given-name pair, P5) stays where it is, and so do an initial written alone, which writes a given name, and a mark or number with no letter in it. A particle chain (P2) is never split, so where taking it would leave no given name the rule declines. +- **What is taken must be one name word a family name could be**: a part counting as two words (a hyphenated given name, a bound given-name pair, P5) stays where it is, and so do an initial written alone, which writes a given name, and a mark or number with no letter in it. A particle chain (P2) is never split, so where taking it would leave no given name the rule declines. A listed given name stays where it is too, under either order, and a list written without accents matches the word written with them: the caller's vocabulary names it a word only a given name is, so the name holds one family name. A word common in both roles is not listed, and a listed word behind a particle is a surname unit, so either is taken as before. ``` + "José Luis Pérez" two_family_names+es_given_names → family="Pérez" + "Jose Luis Perez" two_family_names+es_given_names → middle="Luis" + "Pérez José Luis" family-first+two_family_names+es_given_names → family="Pérez" + "Pérez Luis José" family-first-given-last+two_family_names+es_given_names → family="Pérez" "Ana María-José López" two_family_names → family="López" "Ana abdul Rahman López" two_family_names → family="López" "Ana M. García" two_family_names → family="García" @@ -114,8 +120,12 @@ ``` "Ana María van der Berg" two_family_names → family="María van der Berg" ``` +- A listed given name that is also someone's first surname stays a middle name, the listing saying which role the word mostly fills and not which it fills here. + ``` + "Ana Jorge Flores" two_family_names+es_given_names → family="Flores" + ``` - The rule reports nothing: the caller declared the convention, as a declared order is followed without a report, and a report on most names of three words or more would teach the caller to ignore the field. --- -**Signal:** none in the name: a caller switch, the family's words and the taken word's shape read as written to see whether the family already holds two +**Signal:** none in the name: a caller switch, the family's words and the taken word's shape read as written to see whether the family already holds two, and the taken word's listing as a given name **history:** decisions/O6 · **implemented:** nameparser/_pipeline/_post_rules.py diff --git a/docs/design/rules/O7-two-given-names.md b/docs/design/rules/O7-two-given-names.md index 7a1f9188e..4e624333b 100644 --- a/docs/design/rules/O7-two-given-names.md +++ b/docs/design/rules/O7-two-given-names.md @@ -68,9 +68,12 @@ **Precedence:** -- **Against two-family-name handling** ("Gabriel García Márquez" with both switches on): that rule reads first, since the caller who turns both on says the family holds two names, and this rule takes from the middles it leaves. A name of three words then keeps one given name, and a name of four reads two and two. +- **Against two-family-name handling** ("Gabriel García Márquez" with both switches on): that rule reads first, since the caller who turns both on says the family holds two names, and this rule takes from the middles it leaves. A name of three words then keeps one given name, and a name of four whose third word that rule takes reads two and two. A listed given name that rule leaves where it is (O6) is a middle, which this rule takes where it stands beside the given name, so a name of three words whose middle is listed reads two given names and one family name. ``` "Gabriel García Márquez" two_family_names+two_given_names → given="Gabriel" + "José Luis Pérez" two_family_names+two_given_names+es_given_names → given="José Luis" + "José Luis Pérez" two_family_names+two_given_names+es_given_names → family="Pérez" + "Ana María Luis Pérez" two_family_names+two_given_names+es_given_names → middle="Luis" "María José García López" two_family_names+two_given_names → given="María José" "María José García López" two_family_names+two_given_names → family="García López" ``` diff --git a/docs/design/rules/README.md b/docs/design/rules/README.md index bd7fb3b8a..dad42766f 100644 --- a/docs/design/rules/README.md +++ b/docs/design/rules/README.md @@ -81,7 +81,7 @@ Every example line is EXECUTABLE. The grammar (its executable definition is `tes "INPUT" [annotation] → field="value" [· boundary] [deviates: #N (today: field="value")] -An `annotation` names a policy, locale (`[ru]`), or extras gate (`[ja+segmenter]`) in the registry beside the test. `· boundary` marks the non-firing example every rule must carry: an input shaped like the rule's subject where its effect does NOT occur. That is usually the rule's OWN stated exception — H1's given-name title, P3's single-letter carve-out — not an input the rule never reaches, so the exception is executable rather than merely asserted. Or the rule declares `no-boundary: ` instead, so skipping the boundary is a recorded decision. `deviates:` states the INTENDED output on the example line while the marker records TODAY's output and the tracking issue; the runner asserts today's output strictly, so a parser change that closes the gap fails the suite until the marker is removed in the same PR. `grep -r deviates: docs/design/rules/` is the deviation backlog (deviations from statable rules — coverage gaps are a separate, larger category no grep can see, and contested vocabulary memberships a third, tracked as Open blocks keyed to the vocabulary set in decisions/). +An `annotation` names a policy, a policy with a lexicon fragment beside it (`two_family_names+es_given_names`, for a rule reading a field the default lexicon leaves empty), a locale (`[ru]`), or an extras gate (`[ja+segmenter]`) in the registry beside the test. `· boundary` marks the non-firing example every rule must carry: an input shaped like the rule's subject where its effect does NOT occur. That is usually the rule's OWN stated exception — H1's given-name title, P3's single-letter carve-out — not an input the rule never reaches, so the exception is executable rather than merely asserted. Or the rule declares `no-boundary: ` instead, so skipping the boundary is a recorded decision. `deviates:` states the INTENDED output on the example line while the marker records TODAY's output and the tracking issue; the runner asserts today's output strictly, so a parser change that closes the gap fails the suite until the marker is removed in the same PR. `grep -r deviates: docs/design/rules/` is the deviation backlog (deviations from statable rules — coverage gaps are a separate, larger category no grep can see, and contested vocabulary memberships a third, tracked as Open blocks keyed to the vocabulary set in decisions/). Every rule also states its signal — what in the written name it reads, for example a count, a case contrast, a shape, a neighbor, a position or a script, or `none in the name: ` for a rule that reads configuration or the parsed words. A rule writes it in the footer as a tag, `**Signal:** shape`, optionally followed by a colon and a short description (`**Signal:** none in the name: the configuration's values, at construction`), since its When already describes the situation. The tags the rules use are shape, count, case contrast, position, neighbor, script, comma, edge full stop, character, separator and none in the name. Two kinds of rule carry none of them: A1 and C11, reading only the calls other rules make, write `none of its own`, and C7, which orders other rules' reads, writes `a sequence, not a signal of its own`. The line is one line, never wrapped, standing directly above the rule's pointer line, and the parser refuses one anywhere else, since a continuation line would be read as prose. It is metadata, not statement: a citation cannot quote it. A hub — a decision weighing several signals, each stated by a rule of its own — ends its line `hub of C3, C4, …`, after any signal it reads itself (`**Signal:** ; hub of C3, C4`), and the doc tests hold the rules a hub names to its map, both ways. The map is the bullet list under Then: every rule the hub names is named by some bullet, every rule a bullet names is on the hub's list, and every bullet names at least one, so one bullet may name several. A line naming two signals joins them with ` + `: that rule is a split candidate under docs/design/AGENTS.md's one rule per piece of evidence, and `grep -lE '^\*\*Signal:\*\*.* \+ ' docs/design/rules/*.md` lists them. A condition the rule cites to the rule stating it stays off the line — S10 cites T4 for the lower-case numeral that is no initial — and a gate on the name's size (exactly three name words, a word to spare, as S10's one-letter numeral needs) is part of the signal it gates rather than a second one. diff --git a/docs/release_log.rst b/docs/release_log.rst index ad1c64cac..bb20a6c9f 100644 --- a/docs/release_log.rst +++ b/docs/release_log.rst @@ -13,6 +13,7 @@ Release Log - **Add ParsedName.patronymic, the words naming the bearer's parent.** ``parse("Mahathir bin Mohamad")`` gives patronymic ``bin Mohamad`` beside family ``bin Mohamad``, which 1.4.0 through 2.8.0 gave alone and which is unchanged; under ``locales.RU``, ``Ivan Ivanovich Ivanov`` gives patronymic ``Ivanovich`` and keeps it as the middle name, and under ``locales.TR_AZ`` a marker such as ``oglu`` reads with the name before it. A view like ``family_base``, not a field of ``as_dict()``, read only from a patronymic particle (``bin``, ``binti``, ``binte``, ``ibn``, ``bint``, ``bar``, ``bat`` and their Arabic and Hebrew forms) or a rule a locale turns on, so ``Anwar Ibrahim`` has none, and a name written by ``replace()``, or unpickled from 2.x, has none either. ``Ambiguity.alternative`` of a leading ``bin`` read as a given name names a patronymic where 2.7.0 and 2.8.0 named a particle. :issue:`754` - **Add ParsedName.surname2, the second of two family names.** ``Parser(policy=Policy(two_family_names=True)).parse("Gabriel García Márquez")`` gives surname2 ``Márquez`` beside family ``García Márquez``, which 2.8.0 gave alone and which is unchanged. A view like ``family_base``, not a field of ``as_dict()``, filled only where ``two_family_names`` reads the family as two, so the default, a family comma (``Lloyd Webber, Andrew``) and a hyphenated family have none, and so does a name unpickled from 2.x or one whose family ``replace()`` rewrote. ``Reading.second_surname`` marks its words; it is not compared. :issue:`754` - **Add a switch for names with a compound given name.** ``Parser(policy=Policy(two_given_names=True)).parse("Juan Carlos Pérez")`` gives given ``Juan Carlos``, where 1.4.0 through 2.8.0 gave given ``Juan``, middle ``Carlos``. Off by default: under it a middle name beside the given name joins it (``John Quincy Adams`` gives given ``John Quincy``). With ``two_family_names`` on too, the family takes its word first, so ``María José García López`` reads given ``María José``, family ``García López``. A ``Policy`` pickled by 2.8 loads with it off. :issue:`704` + - **Add Lexicon.given_names, given names two_family_names leaves out of the family.** ``Parser(policy=Policy(two_family_names=True), lexicon=Lexicon.default().add(given_names={"luis"})).parse("José Luis Pérez")`` gives middle ``Luis``, family ``Pérez``, where the switch alone, in 2.8.0 as now, gives family ``Luis Pérez``. Empty by default, so the switch alone reads as before; an entry without accents matches the word with them. :issue:`959` - **Add Token.reading, the reading the parse took for each word.** ``parse("John Smith PhD Jones").tokens[-1].reading`` is ``Reading('credential' as 'suffix', 'position')``, the reading 2.7's ``Ambiguity.alternative`` gave only for the other side of a fork, and ``ReadingKind`` gains ``patronymic``. The field is not compared, and a ``ParsedName`` pickled by 2.x loads with no readings. :issue:`754` - **Add Reading.languages, the language files that list a word the way it was read.** ``parse("Dr. Juan de la Vega III").tokens[2].reading.languages`` is ``('es', 'fr', 'it', 'nl', 'pt')`` for the particle ``de``, and ``Ambiguity.alternative`` names its files too; a reading by shape or position names none. It is not compared, and a ``Reading`` pickled by 2.7 or 2.8 loads with none. The first parse in a process that meets a listed word loads the reading index once: about 1 ms, or about 16 ms where Python writes no bytecode cache. :issue:`754`, :issue:`753` - **Add Czech, Slovak, Slovenian and Croatian packs, locales.CS, SK, SL and HR.** ``parser_for(locales.CS).parse("Jan Novák, MUDr")`` gives title ``MUDr``, given ``Jan``, family ``Novák``, where 1.4.0 through 2.8.0 gave given ``MUDr``, family ``Jan Novák``. Each pack chooses its one language and turns on words the default does not read: ``ml.`` and ``st.`` after the name in all four (``Jiří Dienstbier ml.``), the Czech and Slovak academic titles and ``DrSc.``, and Czech ``Pan`` and ``roz.``. Choosing the language drops ``st``'s Saint readings (``St Clair Smith`` reads given ``St``), which choosing ``fr`` as well keeps. :issue:`753` diff --git a/nameparser/_lexicon.py b/nameparser/_lexicon.py index a7be62e1f..b01539dce 100644 --- a/nameparser/_lexicon.py +++ b/nameparser/_lexicon.py @@ -28,7 +28,7 @@ "given_name_abbreviations", "maiden_markers", "surnames", "honorific_tails", "after_given_titles", "suffix_words_ambiguous", "agnomens", "titles_ambiguous", - "title_modifiers", "title_styles", "title_phrases", + "title_modifiers", "title_styles", "title_phrases", "given_names", ) #: The metadata key a Lexicon field added after `Lexicon` first shipped @@ -780,6 +780,19 @@ class Lexicon: #: list: #: :data:`~nameparser.config.titles.TITLE_PHRASES`. title_phrases: frozenset[str] = frozenset() + #: Given names read by two-family-name handling alone + #: (:attr:`Policy.two_family_names + #: `, rules/O6): the word that + #: rule would take into the family stays where it is when listed, so + #: "José Luis Pérez" keeps family "Pérez". A word matches as + #: written or with its acute and grave accents and diaeresis + #: dropped (the tilde on n and the cedilla on c kept), so an entry + #: written "jose" matches "José" too. No other rule reads it, and + #: the default is empty: a pack carries the list for its language. + #: Added in 3.0, so a Lexicon pickled by 2.x loads with it empty, + #: which is what an older pickle means: no pack filled it then. + given_names: frozenset[str] = field( + default=frozenset(), metadata={_ADDED_IN: "3.0.0"}) #: Lowercase word -> case mask used by capitalized(): the word's #: own letters and digits, each in the case it takes ("phd" -> #: "PhD"), laid over the word as written, so the one entry diff --git a/nameparser/_pipeline/_post_rules.py b/nameparser/_pipeline/_post_rules.py index 798004a29..30bac0cd6 100644 --- a/nameparser/_pipeline/_post_rules.py +++ b/nameparser/_pipeline/_post_rules.py @@ -23,7 +23,8 @@ the script's order differs from the declared one), Policy.middle_as_family, Policy.extra_suffix_delimiters (R1's entry pass, for the delimiter -cores group drops); Lexicon.given_name_titles, and +cores group drops); Lexicon.given_name_titles, Lexicon.given_names +(O6's veto), and Lexicon.suffix_acronyms_ambiguous (a dotted member in a clause, which carries no tag there, S13). @@ -53,8 +54,8 @@ WorkToken, _NEVER_FLIPPED, comma_bucket, copy_with, ) from nameparser._pipeline._vocab import ( - delimiter_cores, has_patronymic_ending, is_patronymic_marker, - suffix_reading, unit_ends, + delimiter_cores, has_patronymic_ending, is_listed_given_name, + is_patronymic_marker, suffix_reading, unit_ends, ) from nameparser._policy import PatronymicRule from nameparser._types import ( @@ -298,6 +299,13 @@ def _take_second_family_name(state: ParseState, return if not any(ch.isalpha() for i in take for ch in tokens[i].text): return + # rules/O6: "a listed given name stays where it is" -- the list a + # pack carries (Lexicon.given_names) names a word only a given name + # is, under either order; a word behind a particle is a surname + # unit, taken as before. History: decisions/O6 (#959). + if len(take) == 1 and "particle" not in tokens[take[0]].tags \ + and is_listed_given_name(tokens[take[0]].text, state.lexicon): + return if state.order[0] is Role.GIVEN: # given, middles..., family: the middle unit standing right # before the family joins it, and text order already renders diff --git a/nameparser/_pipeline/_vocab.py b/nameparser/_pipeline/_vocab.py index 493672eab..96895b805 100644 --- a/nameparser/_pipeline/_vocab.py +++ b/nameparser/_pipeline/_vocab.py @@ -1163,20 +1163,53 @@ def composed_len(text: str) -> int: return len(text if text.isascii() else unicodedata.normalize("NFC", text)) +#: The marks a given-name key keeps: the tilde on n and the cedilla on +#: c, which spell other letters in Spanish and Catalan (Peña, Llorenç), +#: where an acute or grave accent or a diaeresis is how the same name +#: is spelled with or without them (José, Jose). +_KEPT_MARKS = frozenset({("n", "\u0303"), ("c", "\u0327")}) + + +def given_name_key(text: str) -> str: + """`text` folded as `Lexicon.given_names` is matched: the + `_normalize` fold, with the accents and the diaeresis dropped and + the tilde on n and the cedilla on c kept. tools/es_given_names.py + writes the list with this same fold.""" + out: list[str] = [] + for ch in unicodedata.normalize("NFD", _normalize(text)): + if unicodedata.combining(ch) and not ( + out and (out[-1], ch) in _KEPT_MARKS): + continue + out.append(ch) + return unicodedata.normalize("NFC", "".join(out)) + + +def is_listed_given_name(text: str, lexicon: Lexicon) -> bool: + """Whether `text` is in `lexicon.given_names`, as written or by + `given_name_key`, so a caller's entry "José" matches as written + and a list written without accents matches "José" too (rules/O6's + veto, #959).""" + names = lexicon.given_names + return bool(names) and (_normalize(text) in names + or given_name_key(text) in names) + + def in_any_wordlist(n: str, lexicon: Lexicon) -> bool: """Whether the folded word `n` is in ANY of the lexicon's wordlists - (`_lexicon._VOCAB_FIELDS`, the whole roster but two): the caps + (`_lexicon._VOCAB_FIELDS`, the whole roster but four): the caps shape's "unlisted" (`caps_shape_candidate`), where a word a caller listed even as a SURNAME must never become a credential. - The three left out: `title_phrases`, whose entries are two words and + The four left out: `title_phrases`, whose entries are two words and so never one; `titles_ambiguous`, a title that may be a given name, which outside a title phrase is read as the unlisted word it is there (rules/H9: "and elsewhere a name word") -- a phrase is two words and a comma part read by this shape is one; and `given_name_abbreviations`, a given name written short (#789), whose one reader is H2's shape, so anywhere else its word reads as - the unlisted word it is ('John Smith, JAS' keeps suffix 'JAS').""" + the unlisted word it is ('John Smith, JAS' keeps suffix 'JAS'); + and `given_names`, whose one reader is O6 (#959), for the same + reason ('Ana Pérez, LUIS' reads as with the list empty).""" for field in _LISTED_FIELDS: if n in getattr(lexicon, field): return True @@ -1185,7 +1218,8 @@ def in_any_wordlist(n: str, lexicon: Lexicon) -> bool: _LISTED_FIELDS = tuple(f for f in _VOCAB_FIELDS if f not in ("title_phrases", "titles_ambiguous", - "given_name_abbreviations")) + "given_name_abbreviations", + "given_names")) #: The wordlists that claim a word as something OTHER than name text. @@ -1194,7 +1228,7 @@ def in_any_wordlist(n: str, lexicon: Lexicon) -> bool: #: `titles_ambiguous`, a title that may be a given name (rules/H9), with #: `title_phrases`, whose two-word entries never match one word; and so #: is `given_name_abbreviations`, a given name written short (#789), left -#: out of `_LISTED_FIELDS` above. +#: out of `_LISTED_FIELDS` above, with `given_names` (#959). _NON_NAME_FIELDS = tuple(f for f in _LISTED_FIELDS if f not in ("surnames", "bound_given_names")) diff --git a/nameparser/locales/_es_given_names.py b/nameparser/locales/_es_given_names.py new file mode 100644 index 000000000..120e8e54b --- /dev/null +++ b/nameparser/locales/_es_given_names.py @@ -0,0 +1,152 @@ +"""Spanish given names for rules/O6's veto (#959): a word only a given +name is, which ``two_family_names`` leaves where it stands, so "José +Luis Pérez" keeps family "Pérez". Data for the ``es`` pack to carry as +a lexicon fragment, ``Lexicon(given_names=GIVEN_NAMES)``, as locales.ZH +carries its surnames; no pack carries it yet. + +Data source: Instituto Nacional de Estadística (INE), "Frecuencias de +nombres" and "Frecuencias de apellidos", from the annual population +census of 1 January 2025 (Elaboración propia con datos extraídos del +sitio web del INE: www.ine.es; Creative Commons Attribution 4.0). The +words most borne as a given name in Spain among those at least 90% of +whose uses are given-name uses, in rank order, written as INE writes +them -- lower-cased here, without accents, with ñ and ç -- which +``Lexicon.given_names`` matches against a word with its accents +dropped. tools/es_given_names.py derives the block below, with the +cutoff and share it states; do not edit it by hand. A change to the +list is a change to the cutoff, decided against the per-country +measurement in decisions/O6. +""" +from __future__ import annotations + +# -- BEGIN generated by tools/es_given_names.py; edit the tool, not this block -- +_RANKED = ( + "maria", "jose", "antonio", "juan", "manuel", "carmen", "francisco", "luis", + "ana", "javier", "jesus", "miguel", "carlos", "isabel", "angel", "david", + "dolores", "pilar", "daniel", "alejandro", "teresa", "rosa", "pedro", "josefa", + "cristina", "alberto", "laura", "pablo", "fernando", "angeles", "rafael", "elena", + "jorge", "lucia", "marta", "antonia", "sergio", "francisca", "mercedes", "luisa", + "diego", "paula", "enrique", "sara", "concepcion", "rosario", "adrian", "alvaro", + "victor", "ignacio", "raquel", "raul", "rocio", "eduardo", "eva", "ivan", + "patricia", "andrea", "beatriz", "oscar", "victoria", "julia", "manuela", "ruben", + "belen", "juana", "silvia", "joaquin", "mario", "sofia", "nuria", "gabriel", + "irene", "esther", "sandra", "roberto", "montserrat", "encarnacion", "monica", "inmaculada", + "alicia", "mar", "angela", "sonia", "yolanda", "marina", "hugo", "claudia", + "susana", "natalia", "ricardo", "carolina", "margarita", "alejandra", "julio", "daniela", + "emilio", "carla", "guillermo", "ines", "veronica", "gloria", "amparo", "luz", + "jordi", "lourdes", "fatima", "lorena", "noelia", "cesar", "soledad", "cristian", + "marc", "agustin", "begoña", "martina", "samuel", "blanca", "joan", "olga", + "nerea", "miriam", "josep", "clara", "hector", "adriana", "lidia", "asuncion", + "consuelo", "ismael", "celia", "esperanza", "anna", "milagros", "catalina", "alfredo", + "valentina", "valeria", "alex", "diana", "aurora", "aitor", "vanesa", "alexander", + "ainhoa", "emma", "magdalena", "elisa", "eugenia", "iker", "virginia", "gema", + "emilia", "gabriela", "mariano", "elizabeth", "alexandra", "estefania", "vanessa", "xavier", + "josefina", "purificacion", "remedios", "mohammed", "paloma", "laia", "arturo", "dario", + "noa", "paola", "aaron", "araceli", "aitana", "noemi", "joel", "gemma", + "pau", "almudena", "tamara", "izan", "carlota", "ariadna", "jessica", "isaac", + "fernanda", "rebeca", "eric", "vicenta", "ascension", "omar", "jonathan", "natividad", + "matilde", "tania", "gustavo", "christian", "eugenio", "mariana", "guadalupe", "adolfo", + "mireia", "ernesto", "leire", "kevin", "abel", "amelia", "adela", "jon", + "ainara", "asier", "jimena", "leticia", "moises", "olivia", "lola", "leonardo", + "ester", "amalia", "unai", "mikel", "leonor", "francesc", "aranzazu", "barbara", + "adam", "ruth", "gerardo", "federico", "felisa", "enzo", "rafaela", "jennifer", + "alma", "israel", "nora", "erika", "elsa", "helena", "judith", "ramona", + "elisabet", "john", "aroa", "sergi", "tatiana", "carmelo", "berta", "youssef", + "miren", "gerard", "eloy", "estela", "armando", "sheila", "juliana", "macarena", + "agustina", "nadia", "soraya", "alexis", "ander", "thiago", "jonatan", "saul", + "joaquina", "judit", "benjamin", "erik", "camila", "mia", "marcela", "africa", + "dylan", "rita", "aida", "rosalia", "aina", "ian", "candelaria", "liliana", + "angelica", "adoracion", "jairo", "antoni", "petra", "abraham", "mauricio", "leyre", + "luca", "josue", "lluis", "naiara", "adria", "gael", "iñaki", "iris", + "khadija", "camilo", "sagrario", "edgar", "piedad", "amanda", "aurelio", "julen", + "delia", "ezequiel", "michael", "eulalia", "chloe", "yaiza", "amaia", "elisabeth", + "pere", "iria", "carme", "jan", "lina", "yeray", "aleix", "joana", + "saray", "isabella", "xabier", "rachid", "itziar", "liam", "martha", "violeta", + "azucena", "ane", "karen", "zoe", "gorka", "merce", "alan", "mirian", + "esmeralda", "dolors", "alexandru", "mustapha", "herminia", "elia", "florentina", "eduard", + "maite", "neus", "desiree", "nicole", "eusebio", "fabio", "consolacion", "biel", + "irina", "nil", "enriqueta", "salma", "enric", "maximo", "william", "ayoub", + "emiliano", "eneko", "desamparados", "jacobo", "aya", "jana", "marian", "mauro", + "victoriano", "nestor", "mihaela", "alexandre", "milena", "noah", "fabiola", "alexia", + "gisela", "meritxell", "rayan", "nazaret", "lydia", "roser", "mary", "aurelia", + "candido", "lia", "zahra", "orlando", "nayara", "jean", "fidel", "irati", + "gregoria", "candida", "karina", "ingrid", "aicha", "ariel", "igor", "mara", + "edurne", "mariam", "florin", "naima", "joseba", "bilal", "micaela", "florencia", + "alina", "ioan", "yassine", "eloisa", "graciela", "jhon", "ona", "angustias", + "adelina", "samira", "zaira", "anthony", "jenifer", "amine", "serafin", "nancy", + "ionut", "ibai", "yasmina", "regina", "naia", "marisol", "estibaliz", "marie", + "iago", "aimar", "richard", "faustino", "fuensanta", "felicidad", "auxiliadora", "anabel", + "adelaida", "rogelio", "aritz", "viviana", "azahara", "nahia", "marius", "hicham", + "humberto", "maribel", "manel", "salome", "julieta", "anne", "amina", "axel", + "yanira", "malak", "mamadou", "jesica", "amir", "flora", "hortensia", "visitacion", + "paulina", "andoni", "andreea", "virtudes", "cayetana", "joaquim", "steven", "anas", + "cinta", "bryan", "ann", "erick", "noel", "teodora", "brian", "angels", + "nelson", "mireya", "modesto", "benita", "yasmin", "mariona", "zohra", "markel", + "gladys", "ildefonso", "amira", "johanna", "michelle", "salud", "edwin", "katherine", + "avelino", "karla", "nour", "eladio", "fatiha", "tomasa", "filomena", "ainoa", + "ioana", "anibal", "covadonga", "imanol", "arantxa", "rodolfo", "dayana", "malika", + "oier", "luciana", "izaskun", "bianca", "june", "dulce", "evelyn", "brais", + "anais", "ismail", "rosana", "idoia", "angelina", "josu", "roxana", "debora", + "sarah", "alessandro", "max", "mark", "delfina", "abderrahim", "zakaria", "celeste", + "abigail", "mayra", "karima", "maximiliano", "norma", "jacqueline", "georgina", "rachida", + "laila", "marcelina", "abdelaziz", "ricard", "iratxe", "didac", "genoveva", "marcel", + "ariana", "hanane", "nicoleta", "davinia", "mariela", "patrocinio", "maider", "eddine", + "hamid", "avelina", "maialen", "walter", "melissa", "oleksandr", "uxue", "christopher", + "emmanuel", "erica", "antonella", "leidy", "myriam", "johana", "iryna", "bouchra", + "saturnino", "naroa", "angie", "aday", "salvadora", "brayan", "iciar", "saida", + "samara", "rene", "hajar", "leopoldo", "olena", "teo", "anastasia", "henar", + "alain", "evaristo", "minerva", "lluc", "francesca", "fe", "aurea", "sabrina", + "pamela", "sarai", "ursula", "cloe", "emanuel", "eliseo", "uxia", "sebastiana", + "benigno", "nayra", "ada", "cecilio", "eider", "marwa", "jesusa", "evangelina", + "alfonsa", "justa", "joao", "imane", "obdulia", "johan", "cintia", "brenda", + "dina", "isidora", "oussama", "lorenza", "latifa", "melania", "arlet", "younes", + "samir", "nabil", "ayman", "felipa", "yassin", "souad", "jenny", "amal", + "palmira", "marlene", "india", "ikram", "brigida", "emily", "eloi", "adil", + "ignacia", "engracia", "sonsoles", "simona", "kilian", "oksana", "arantzazu", "jacinta", + "olatz", "yesica", "mostafa", "rut", "gertrudis", "andrew", "samantha", "giovanni", + "haizea", "luka", "otilia", "thais", "romina", "ibrahima", "norberto", "eliana", + "ignasi", "cynthia", "divina", "nelly", "stella", "hafida", "neizan", "maritza", + "zaida", "beñat", "cristhian", "siham", "celso", "maitane", "najat", "bienvenida", + "ivet", "naira", "unax", "mariia", "nekane", "nico", "daria", "sira", + "clotilde", "ionela", "dominga", "xenia", "cesareo", "mabel", "guillermina", "maryam", + "dunia", "reda", "yessica", "bienvenido", "isaias", "edith", "yurena", "modesta", + "achraf", "rufina", "svitlana", "edward", "izaro", "dionisia", "bibiana", "melanie", + "presentacion", "leila", "sabela", "walid", "nelida", "iluminada", "amadeo", "tetiana", + "saioa", "yoel", "ibon", "cipriano", "placido", "gina", "noureddine", "balbina", + "damaris", "anastasiia", "freddy", "eulogio", "irma", "hiba", "soufiane", "greta", + "dora", "melisa", "louise", "lizeth", "oihane", "milagrosa", "irune", "anxo", + "catalin", "emiliana", "volodymyr", "nataliia", "ofelia", "iman", "halima", "nacho", + "jamila", "desire", "mourad", "mathias", "anastasio", "horacio", "vladimir", "ximena", + "patrick", "belinda", "eder", "astrid", "felicitas", "svetlana", "herminio", "imad", + "derek", "naomi", "gaizka", "anunciacion", "hanan", "abdelilah", "fatna", "matteo", + "olha", "daniella", "carmela", "danna", "jone", "sophia", "feliciana", "stefania", + "roi", "maravillas", "wendy", "alaia", "ekaterina", "oihan", "maddi", "silvana", + "inocencia", "eusebia", "hilda", "oswaldo", "artur", "bosco", "celestina", "ulises", + "toni", "susan", "teofilo", "enara", "fabiana", "janire", "margaret", "maximino", + "abdellatif", "abdelkarim", "xavi", "loubna", "zineb", "francesco", "concepcio", "tarik", + "efren", "linda", "stephen", "ronald", "virgilio", "telmo", "aquilino", "aitziber", + "abelardo", "adara", "kai", "elisenda", "malena", "leonel", "alessandra", "janet", + "georgiana", "garazi", "yerai", "faustina", "nicolasa", "fredy", "gilberto", "xiomara", + "constanza", "edison", "kaoutar", "fermina", "nahuel", "arsenio", "abdoulaye", "aziza", + "ilham", "fouad", "dalia", "higinio", "houda", "rahma", "arianna", "endika", + "lilian", "arkaitz", "serafina", "celsa", "kateryna", "victorina", "quim", "iraia", + "caterina", "urko", "asmae", "gumersindo", "dafne", "idaira", "lucila", "indira", + "lluisa", "lua", "aisha", "dmytro", "arantza", "elian", "iulian", "fausto", + "youness", "meriem", "carina", "lucrecia", "mariya", "alazne", "josune", "catherine", + "isabela", "tiago", "genesis", "giulia", "isolina", "lilia", "argentina", "adriano", + "sanae", "olaia", "katia", "meryem", "yousra", "ekaitz", "noura", "viktoriia", + "viorel", "priscila", "habiba", "ionel", "jonas", "arancha", "selena", "ethan", + "eleuterio", "zoraida", "elba", "ashley", "nataliya", "bernarda", "itxaso", "ousmane", + "libertad", "narcisa", "ylenia", "stiven", "saturnina", "doris", "yasmine", "rania", + "andreina", "clementina", "jeronima", "adama", "angelo", "benedicta", "selene", "abdelali", + "anouar", "alice", "digna", "amadou", "danel", "corina", "florinda", "thalia", + "christine", "abdessamad", "yesenia", "ivana", "franklin", "darwin", "fanny", "mirela", + "chaimae", "abdelhak", "anass", "soukaina", "deborah", "nizar", "olimpia", "jonay", + "janeth", "andrii", "ilias", "viktor", "abdelghani", "giuseppe", "eloina", "felicia", + "fatoumata", "moustapha", "anita", "rabia", "artem", "malen", "mhamed", "yamina", + "saadia", "eduarda", "basilisa", "marisa", "idoya", "amada", "llorenç", "xoel", + "sorin", "chiara", "hodei", "mahamadou", "zuriñe", "hafsa", "immaculada", "rim", +) +# -- END generated -- + +GIVEN_NAMES = frozenset(_RANKED) diff --git a/tests/v2/pipeline/test_post_rules.py b/tests/v2/pipeline/test_post_rules.py index 5a7d40d47..08d35cc68 100644 --- a/tests/v2/pipeline/test_post_rules.py +++ b/tests/v2/pipeline/test_post_rules.py @@ -1,5 +1,7 @@ import dataclasses +import json import sys +from pathlib import Path import pytest @@ -12,6 +14,7 @@ from nameparser._policy import (FAMILY_FIRST, FAMILY_FIRST_GIVEN_LAST, GIVEN_FIRST, PatronymicRule, Policy, Script) +from nameparser.locales._es_given_names import GIVEN_NAMES from nameparser._types import (FOLDED_TAG, STABLE_TAGS, UNJOINED_CONJUNCTION_TAG, UNJOINED_TAG, AmbiguityKind, Role) @@ -1296,3 +1299,129 @@ def test_two_family_names_counts_every_listed_hyphen_as_two_words( Policy(two_family_names=True)) assert _by_role(out, Role.MIDDLE) == "María" assert _by_role(out, Role.FAMILY) == f"García{hyphen}López" + + +#: rules/O6's given-name veto (#959) reads Lexicon.given_names, which +#: the default leaves empty; these rows carry the list a pack ships. +_ES_GIVEN = Lexicon.default().add(given_names=GIVEN_NAMES) + + +def _roles(text: str, policy: Policy, lexicon: Lexicon) -> tuple[str, ...]: + name = Parser(policy=policy, lexicon=lexicon).parse(text) + return name.given, name.middle, name.family + + +#: Each row's reading with the switch alone and with the list beside +#: it: the second column is the veto's negative control, recorded so a +#: row the veto never reaches cannot pass as one it decides (AGENTS.md, +#: "Guard tests"). Rows whose two columns agree are the shapes the veto +#: must leave alone: a family comma, an empty head, a word the list does +#: not hold, and a taken unit that already counts as two words. +_VETO_EFFECT = [ + ("José Luis Pérez", GIVEN_FIRST, + ("José", "", "Luis Pérez"), ("José", "Luis", "Pérez")), + ("Jose Luis Perez", GIVEN_FIRST, + ("Jose", "", "Luis Perez"), ("Jose", "Luis", "Perez")), + ("JOSÉ LUIS PÉREZ", GIVEN_FIRST, + ("JOSÉ", "", "LUIS PÉREZ"), ("JOSÉ", "LUIS", "PÉREZ")), + ("José Lúis Pérez", GIVEN_FIRST, + ("José", "", "Lúis Pérez"), ("José", "Lúis", "Pérez")), + # NFD: the accent a combining mark, dropped by the key as the + # composed one is + ("José Luís Pérez", GIVEN_FIRST, + ("José", "", "Luís Pérez"), + ("José", "Luís", "Pérez")), + ("Dr. José Luis Pérez Jr.", GIVEN_FIRST, + ("José", "", "Luis Pérez"), ("José", "Luis", "Pérez")), + ("José (Pepe) Luis Pérez", GIVEN_FIRST, + ("José", "", "Luis Pérez"), ("José", "Luis", "Pérez")), + ("José Luis Pérez, PhD", GIVEN_FIRST, + ("José", "", "Luis Pérez"), ("José", "Luis", "Pérez")), + ("José Luis de la Vega", GIVEN_FIRST, + ("José", "", "Luis de la Vega"), ("José", "Luis", "de la Vega")), + ("Pérez José Luis", FAMILY_FIRST, + ("Luis", "", "Pérez José"), ("José", "Luis", "Pérez")), + ("Pérez Luis José", FAMILY_FIRST_GIVEN_LAST, + ("José", "", "Pérez Luis"), ("José", "Luis", "Pérez")), + ("Pérez, José Luis", GIVEN_FIRST, + ("José", "Luis", "Pérez"), ("José", "Luis", "Pérez")), + ("Luis, José Pérez", GIVEN_FIRST, + ("José", "Pérez", "Luis"), ("José", "Pérez", "Luis")), + (", José Luis Pérez", GIVEN_FIRST, + ("José", "Luis Pérez", ""), ("José", "Luis Pérez", "")), + ("Ana Martín López", GIVEN_FIRST, + ("Ana", "", "Martín López"), ("Ana", "", "Martín López")), + ("Ana María-Luis López", GIVEN_FIRST, + ("Ana", "María-Luis", "López"), ("Ana", "María-Luis", "López")), + ("José Luis Pérez-García", GIVEN_FIRST, + ("José", "Luis", "Pérez-García"), ("José", "Luis", "Pérez-García")), + # a listed word behind a particle is a surname unit, taken as + # before (the #959 review: the first draft left `de Miguel`) + ("Ana de Miguel de la Vega", GIVEN_FIRST, + ("Ana", "", "de Miguel de la Vega"), ("Ana", "", "de Miguel de la Vega")), +] + + +@pytest.mark.parametrize("text,order,switch_alone,with_list", _VETO_EFFECT, + ids=[row[0] for row in _VETO_EFFECT]) +def test_two_family_names_leaves_a_listed_given_name( + text: str, order: tuple[Role, Role, Role], + switch_alone: tuple[str, ...], with_list: tuple[str, ...]) -> None: + policy = Policy(name_order=order, two_family_names=True) + assert _roles(text, policy, Lexicon.default()) == switch_alone + assert _roles(text, policy, _ES_GIVEN) == with_list + + +def test_the_veto_counts_its_rows() -> None: + # The table is the veto's negative control only while some row moves + # under it and some does not, in both families of order. + moved = [r for r in _VETO_EFFECT if r[2] != r[3]] + assert {r[1] for r in moved} == {GIVEN_FIRST, FAMILY_FIRST, + FAMILY_FIRST_GIVEN_LAST} + assert len(moved) == 11 and len(_VETO_EFFECT) - len(moved) == 7 + + +def test_the_veto_leaves_the_word_to_two_given_names() -> None: + # rules/O7's precedence: O6 reads first, and a listed word it leaves + # is the middle O7 then takes + both = Policy(two_family_names=True, two_given_names=True) + assert _roles("José Luis Pérez", both, Lexicon.default()) == ( + "José", "", "Luis Pérez") + assert _roles("José Luis Pérez", both, _ES_GIVEN) == ( + "José Luis", "", "Pérez") + # ... where it stands beside the given name: a given part of two + # units gives O7 the first, and the listed word stays a middle + assert _roles("Ana María Luis Pérez", both, _ES_GIVEN) == ( + "Ana María", "Luis", "Pérez") + + +def test_a_caller_entry_matches_as_written_and_by_its_key() -> None: + # An entry with its accent matches the word written with it; one + # without matches every spelling, the key dropping the accent. + o6 = Policy(two_family_names=True) + accented = Lexicon.default().add(given_names={"Lúis"}) + assert _roles("José Lúis Pérez", o6, accented)[2] == "Pérez" + assert _roles("José Luis Pérez", o6, accented)[2] == "Luis Pérez" + bare = Lexicon.default().add(given_names={"luis"}) + for written in ("Luis", "Lúis", "LUIS", "Luís"): + assert _roles(f"José {written} Pérez", o6, bare)[2] == "Pérez", written + + +@pytest.mark.parametrize("policy", [ + pytest.param(Policy(), id="default"), + pytest.param(Policy(two_given_names=True), id="two_given_names"), + pytest.param(Policy(name_order=FAMILY_FIRST), id="family-first"), +]) +def test_the_list_reads_nothing_without_two_family_names( + policy: Policy) -> None: + # Only O6 reads the field, so a lexicon carrying it parses every + # rule example as one without it wherever O6 is off. + corpus = Path(__file__).parents[3] / "tools" / "differential" / ( + "corpus_rules.jsonl") + texts = [json.loads(line) for line in + corpus.read_text(encoding="utf-8").split("\n") if line] + plain = Parser(policy=policy) + listed = Parser(policy=policy, lexicon=_ES_GIVEN) + assert texts + assert [t for t in texts + if plain.parse(t).as_dict() != listed.parse(t).as_dict()] == [] diff --git a/tests/v2/pipeline/test_vocab.py b/tests/v2/pipeline/test_vocab.py index 389ef2c14..3a7a70e74 100644 --- a/tests/v2/pipeline/test_vocab.py +++ b/tests/v2/pipeline/test_vocab.py @@ -549,14 +549,15 @@ def test_the_caps_exclusion_covers_every_vocabulary_field() -> None: # H9: two fields are left out on purpose -- a title that may # be a given name reads, outside a phrase, as the unlisted word it # is there, and a phrase is two words -- and #789's third: a given - # name written short is read only by H2's shape + # name written short is read only by H2's shape; and #959's fourth, + # the given names only O6 reads for field in ("titles_ambiguous", "title_phrases", - "given_name_abbreviations"): + "given_name_abbreviations", "given_names"): assert caps_shape_candidate( "ZZQX", base.add(**{field: {"zzqx"}}), on, False), field for field in _VOCAB_FIELDS: if field in ("titles_ambiguous", "title_phrases", - "given_name_abbreviations"): + "given_name_abbreviations", "given_names"): continue extra: dict[str, set[str]] = {field: {"zzqx"}} if field in ("honorific_tails", "suffix_words_ambiguous"): diff --git a/tests/v2/rules_budget.json b/tests/v2/rules_budget.json index faae8da7f..7933f7fd8 100644 --- a/tests/v2/rules_budget.json +++ b/tests/v2/rules_budget.json @@ -39,8 +39,8 @@ "O3": {"words": 47, "then_words": 18, "because_sentences": 1, "except": 0, "precedence": 0, "accepted": 0, "accepted_words": 0}, "O4": {"words": 91, "then_words": 57, "because_sentences": 1, "except": 0, "precedence": 0, "accepted": 0, "accepted_words": 0}, "O5": {"words": 478, "then_words": 160, "because_sentences": 2, "except": 6, "precedence": 1, "accepted": 0, "accepted_words": 0}, - "O6": {"words": 596, "then_words": 161, "because_sentences": 2, "except": 5, "precedence": 4, "accepted": 7, "accepted_words": 244}, - "O7": {"words": 549, "then_words": 175, "because_sentences": 2, "except": 4, "precedence": 3, "accepted": 6, "accepted_words": 220}, + "O6": {"words": 665, "then_words": 161, "because_sentences": 2, "except": 5, "precedence": 4, "accepted": 8, "accepted_words": 273}, + "O7": {"words": 598, "then_words": 175, "because_sentences": 2, "except": 4, "precedence": 3, "accepted": 6, "accepted_words": 220}, "P1": {"words": 415, "then_words": 287, "because_sentences": 2, "except": 0, "precedence": 1, "accepted": 4, "accepted_words": 319}, "P2": {"words": 315, "then_words": 279, "because_sentences": 1, "except": 0, "precedence": 0, "accepted": 3, "accepted_words": 167}, "P3": {"words": 639, "then_words": 598, "because_sentences": 1, "except": 0, "precedence": 0, "accepted": 0, "accepted_words": 0}, diff --git a/tests/v2/rules_doc.py b/tests/v2/rules_doc.py index 000d002d8..a0b6c66b1 100644 --- a/tests/v2/rules_doc.py +++ b/tests/v2/rules_doc.py @@ -173,6 +173,27 @@ def has_boundary_or_waiver(self) -> bool: "extra_suffix_delimiters-pipe": Policy( extra_suffix_delimiters=frozenset({" | "})), } +from nameparser._lexicon import Lexicon # noqa: E402 +from nameparser.locales._es_given_names import GIVEN_NAMES # noqa: E402 + +#: Named policies with a lexicon fragment beside them, for a rule that +#: reads a Lexicon field the default leaves empty: rules/O6's veto +#: reads ``given_names``, which a pack carries (#959). The annotation +#: is the policy's name above plus the fragment's, ``es_given_names`` +#: naming the list ``nameparser/locales/_es_given_names.py`` ships, so +#: a rule names the data as it names the switch. +_ES_GIVEN = Lexicon.default().add(given_names=GIVEN_NAMES) +CONFIGS: dict[str, tuple[Policy, Lexicon]] = { + "two_family_names+es_given_names": ( + POLICIES["two_family_names"], _ES_GIVEN), + "family-first+two_family_names+es_given_names": ( + POLICIES["family-first+two_family_names"], _ES_GIVEN), + "family-first-given-last+two_family_names+es_given_names": ( + POLICIES["family-first-given-last+two_family_names"], _ES_GIVEN), + "two_family_names+two_given_names+es_given_names": ( + POLICIES["two_family_names+two_given_names"], _ES_GIVEN), +} + #: D-section subjects: zero-arg constructions whose diagnostics the #: warns=/raises= assertion forms exercise. def _segmenterless_ja() -> object: @@ -205,6 +226,8 @@ def _bad_order_none() -> object: def resolve_annotation(annot: str) -> tuple[str, object]: if annot in POLICIES: return "policy", POLICIES[annot] + if annot in CONFIGS: + return "config", CONFIGS[annot] if annot in GATES: return "gated_locale", GATES[annot][0] if _LOCALE_ANNOT_RE.fullmatch(annot): diff --git a/tests/v2/test_es_given_names.py b/tests/v2/test_es_given_names.py new file mode 100644 index 000000000..a3223047b --- /dev/null +++ b/tests/v2/test_es_given_names.py @@ -0,0 +1,144 @@ +"""The Spanish given-name list rules/O6's veto reads (#959), and offline +checks on tools/es_given_names.py, which writes it. + +The list is generated, not judged word by word, so what a test can hold +is what no cutoff may change: no word the default vocabulary reads as +something other than a name (the TOML build's check 11 for a per-word +reading, which a generated list does not pass through), every entry in +the fold the parse matches by, and the tool's selection on hand-built +rows. The tool's sources are two INE downloads, which no test touches. +""" +from __future__ import annotations + +import importlib.util +import sys +import urllib.request +from collections import Counter +from pathlib import Path +from types import ModuleType + +import pytest + +from nameparser import Lexicon +from nameparser._pipeline._vocab import given_name_key +from nameparser.locales import _es_given_names + +_PATH = Path(__file__).parents[2] / "tools" / "es_given_names.py" + + +def _load() -> ModuleType: + spec = importlib.util.spec_from_file_location("es_given_names", _PATH) + assert spec is not None and spec.loader is not None + module = importlib.util.module_from_spec(spec) + sys.modules[spec.name] = module + spec.loader.exec_module(module) + return module + + +tool = _load() + + +@pytest.fixture(autouse=True) +def _no_network(monkeypatch: pytest.MonkeyPatch) -> None: + def refuse(*args: object, **kwargs: object) -> None: + raise AssertionError("a test reached the network") + monkeypatch.setattr(urllib.request, "urlopen", refuse) + + +def _non_name_readings(lexicon: Lexicon) -> dict[str, frozenset[str]]: + """The default vocabulary's readings of a word as something other + than a name, each folded as the list is: a never-given particle, a + title (an ambiguous one too), a suffix and a conjunction.""" + never_given = lexicon.particles - lexicon.particles_ambiguous + fields = { + "never-given particle": never_given, + "title": lexicon.titles | lexicon.titles_ambiguous, + "suffix": lexicon.suffix_acronyms | lexicon.suffix_words, + "conjunction": lexicon.conjunctions, + } + return {kind: frozenset(given_name_key(w) for w in words) + for kind, words in fields.items()} + + +def _overlap(names: frozenset[str], lexicon: Lexicon) -> dict[str, set[str]]: + return {kind: set(names & words) + for kind, words in _non_name_readings(lexicon).items() + if names & words} + + +#: The check with a word of each kind added: what the test below would +#: report were the tool's exclusion to let one through -- its negative +#: control, recorded so the empty answer cannot be a check reading +#: nothing. +_UNGUARDED_EFFECT = { + "never-given particle": {"los"}, + "title": {"dr", "dean"}, + "suffix": {"jr"}, + "conjunction": {"y"}, +} + + +def test_the_list_holds_no_word_the_default_reads_as_no_name() -> None: + # A listed word O6 declines to take is one it would otherwise take + # into the family; one the default reads as a particle, a title, a + # suffix or a connective is not a given name the veto could mean, + # and listing it would contradict the reading that claims it. + default = Lexicon.default() + assert _overlap(_es_given_names.GIVEN_NAMES, default) == {} + planted = _es_given_names.GIVEN_NAMES | {"los", "dr", "dean", "jr", "y"} + assert _overlap(planted, default) == _UNGUARDED_EFFECT + + +def test_every_entry_is_written_in_the_fold_it_is_matched_by() -> None: + # The parse folds the word it asks about (given_name_key) and looks + # the fold up, so an entry written any other way is never matched. + ranked = _es_given_names._RANKED + assert ranked and len(set(ranked)) == len(ranked) + assert [w for w in ranked if given_name_key(w) != w or not w.isalpha()] == [] + assert _es_given_names.GIVEN_NAMES == frozenset(ranked) + + +def test_the_module_block_is_what_the_tool_renders() -> None: + # The block is the tool's output, so a hand edit, or a list the tool + # would render differently, fails here. + text = _PATH.parents[1].joinpath( + "nameparser", "locales", "_es_given_names.py").read_text( + encoding="utf-8") + block = tool.render(list(_es_given_names._RANKED)) + assert tool.rewrite(text, block) == text + + +def test_the_tool_folds_as_the_parse_does() -> None: + assert tool.fold("JOSÉ") == "jose" + assert tool.fold("MUÑOZ") == "muñoz" + assert tool.fold("LLORENÇ") == "llorenç" + assert tool.fold("José") == "jose" # NFD + assert tool.fold("Güell") == "guell" + + +def test_a_name_is_counted_by_its_words() -> None: + rows = [("JOSE LUIS", 10), ("MARIA DEL CARMEN", 5), ("D'ARTAGNAN", 7), + ("JOSE", 3)] + assert tool.tally(rows) == Counter( + {"jose": 13, "luis": 10, "maria": 5, "del": 5, "carmen": 5}) + + +def test_a_surname_behind_a_particle_is_not_the_bare_word() -> None: + # "de Miguel" is a particle unit O6 never takes as the word alone, + # so it says nothing about a bare "Miguel" in the middle slot. + rows = [("DE MIGUEL", 40), ("MIGUEL", 2), ("SAN MIGUEL", 9)] + assert tool.tally(rows, after={"de"}) == Counter( + {"miguel": 11, "de": 40, "san": 9}) + + +def test_the_selection_keeps_the_mostly_given_words_by_rank() -> None: + given = Counter({"jose": 100, "martin": 50, "luis": 80, "rosa": 30, + "del": 70, "j": 90}) + surname = Counter({"martin": 400, "luis": 8, "rosa": 3}) + # martin is mostly a surname, del is excluded, j writes an initial + assert tool.select(given, surname, {"del"}, top=10) == [ + "jose", "luis", "rosa"] + assert tool.select(given, surname, {"del"}, top=2) == ["jose", "luis"] + # rosa is 30 of 33 uses given (91%): in at 0.9, out at 0.95 + assert "rosa" in tool.select(given, surname, set(), share=0.9) + assert "rosa" not in tool.select(given, surname, set(), share=0.95) diff --git a/tests/v2/test_layering.py b/tests/v2/test_layering.py index cfff6518f..a04fcf7bb 100644 --- a/tests/v2/test_layering.py +++ b/tests/v2/test_layering.py @@ -26,7 +26,8 @@ "_pipeline/_assign.py", "_pipeline/_post_rules.py", "_pipeline/_assemble.py", "_parser.py", "locales/__init__.py", "locales/ja.py", - "locales/ru.py", "locales/tr_az.py", "locales/zh.py"} + "locales/ru.py", "locales/tr_az.py", "locales/zh.py", + "locales/_es_given_names.py"} _PIPELINE_STAGE_ALLOWED = ( "nameparser._types", "nameparser._lexicon", "nameparser._policy", @@ -139,6 +140,9 @@ "locales/ru.py": _LOCALE_PACK_ALLOWED, "locales/tr_az.py": _LOCALE_PACK_ALLOWED, "locales/zh.py": _LOCALE_PACK_ALLOWED, + # a pack's data, no pack itself: rules/O6's given-name list (#959), + # which the es pack will carry as a lexicon fragment + "locales/_es_given_names.py": (), # the data modules' package: a docstring, no imports "config/__init__.py": (), # CLI: imports only the public package, same as diff --git a/tests/v2/test_lexicon.py b/tests/v2/test_lexicon.py index ddc38ad83..e91a76808 100644 --- a/tests/v2/test_lexicon.py +++ b/tests/v2/test_lexicon.py @@ -530,6 +530,31 @@ def test_a_lexicon_parser_and_locale_pickled_by_2_8_0_load() -> None: assert isinstance(loaded["locale"], Locale) assert loaded["locale"].code == "ru" assert isinstance(loaded["locale"].lexicon, Lexicon) + # #959's given_names, added in 3.0 and marked, loads empty: the + # default an older pickle means, no pack having filled it in 2.8 + for loaded_lexicon in (loaded["lexicon"], loaded["parser"].lexicon, + loaded["locale"].lexicon): + assert loaded_lexicon.given_names == frozenset() + + +def test_given_names_round_trips() -> None: + lex = Lexicon.default().add(given_names={"luis", "José"}) + assert pickle.loads(pickle.dumps(lex)) == lex + + +def test_given_names_loads_from_2_8_0_only_by_its_marker( + monkeypatch: pytest.MonkeyPatch) -> None: + # The negative control for the test above: the same 2.8.0 pickle + # with given_names' _ADDED_IN marker taken off is refused, so the + # load above is the marker's doing (#959). + from dataclasses import fields + f = next(f for f in fields(Lexicon) if f.name == "given_names") + assert _ADDED_IN in f.metadata + monkeypatch.setattr(f, "metadata", MappingProxyType( + {k: v for k, v in f.metadata.items() if k != _ADDED_IN})) + with open(Path(__file__).parent / "data" / "lexicon_v28.pickle", + "rb") as fh, pytest.raises(ValueError, match="given_names"): + pickle.load(fh) def test_normalization_lowercases_and_keeps_interior_periods() -> None: diff --git a/tests/v2/test_parser.py b/tests/v2/test_parser.py index efbc2ad68..95e340313 100644 --- a/tests/v2/test_parser.py +++ b/tests/v2/test_parser.py @@ -2342,7 +2342,10 @@ def test_the_compiled_fields_are_the_default_lexicons() -> None: from nameparser._reading_index import compile_choice default = Lexicon.default() fields = _vocab_fields(compile_choice(frozenset())) - assert {f for f in fields} | {"surnames"} == set(_VOCAB_FIELDS) + # surnames is KOREAN_SURNAMES and given_names a pack's list (#959), + # neither compiled from the vocabulary + assert {f for f in fields} | {"surnames", "given_names"} == set( + _VOCAB_FIELDS) for name, words in fields.items(): assert getattr(default, name) == words, name diff --git a/tests/v2/test_rules_doc.py b/tests/v2/test_rules_doc.py index d8031e450..c806ce7d3 100644 --- a/tests/v2/test_rules_doc.py +++ b/tests/v2/test_rules_doc.py @@ -14,6 +14,7 @@ from nameparser import Parser, parse, parser_for from nameparser import locales +from nameparser._lexicon import Lexicon from nameparser._policy import Policy from tests.v2.design_docs import RULES_DIR, read_rules_text from tests.v2.rules_doc import ( @@ -183,11 +184,12 @@ def _check_diagnostic(example: Example) -> None: fn() -def _pieces(text: str, policy: Policy | None) -> list[list[str]]: +def _pieces(text: str, policy: Policy | None, + lexicon: Lexicon | None = None) -> list[list[str]]: from nameparser._pipeline import run from nameparser._pipeline._state import ParseState from nameparser._parser import Parser as _P - p = _P(policy=policy) if policy is not None else _P() + p = _P(policy=policy or Policy(), lexicon=lexicon or Lexicon.default()) state = run(ParseState(original=text, lexicon=p.lexicon, policy=p.policy, segmenter=None)) return [[state.tokens[i].text for i in piece] @@ -196,12 +198,16 @@ def _pieces(text: str, policy: Policy | None) -> list[list[str]]: def _run(example: Example) -> object: policy: Policy | None = None + lexicon: Lexicon | None = None locale: str | None = None if example.annotation is not None: kind, obj = resolve_annotation(example.annotation) if kind == "policy": assert isinstance(obj, Policy) policy = obj + elif kind == "config": + assert isinstance(obj, tuple) + policy, lexicon = obj elif kind == "locale": assert isinstance(obj, str) assert obj != "ja", ( @@ -215,9 +221,12 @@ def _run(example: Example) -> object: assert isinstance(obj, str) locale = obj if example.field == "pieces": - return _pieces(example.text, policy) + return _pieces(example.text, policy, lexicon) if locale is not None: parsed = parser_for(locales.get(locale)).parse(example.text) + elif lexicon is not None: + parsed = Parser(policy=policy or Policy(), + lexicon=lexicon).parse(example.text) elif policy is not None: parsed = Parser(policy=policy).parse(example.text) else: diff --git a/tests/v2/test_rules_doc_grammar.py b/tests/v2/test_rules_doc_grammar.py index a5e6575de..243005ed2 100644 --- a/tests/v2/test_rules_doc_grammar.py +++ b/tests/v2/test_rules_doc_grammar.py @@ -101,6 +101,11 @@ def test_registry_resolves_policies_and_gates() -> None: assert kind == "locale" and obj == "ru" kind, obj = resolve_annotation("[ja+segmenter]") assert kind == "gated_locale" and obj == "ja" + # a policy with a lexicon fragment (#959): the pair, the fragment's + # list not empty, so an example under it reads the list + kind, obj = resolve_annotation("two_family_names+es_given_names") + assert kind == "config" + assert isinstance(obj, tuple) and obj[1].given_names with pytest.raises(KeyError): resolve_annotation("no-such-annotation") diff --git a/tools/differential/corpus_rules.jsonl b/tools/differential/corpus_rules.jsonl index c4e74507e..509c82364 100644 --- a/tools/differential/corpus_rules.jsonl +++ b/tools/differential/corpus_rules.jsonl @@ -59,7 +59,9 @@ "Ana García" "Ana García (Anita) López" "Ana J.R. García" +"Ana Jorge Flores" "Ana M. García" +"Ana Martín López" "Ana María 3 López" "Ana María García - López" "Ana María García López" @@ -67,12 +69,14 @@ "Ana María García–López" "Ana María García－López" "Ana María José Pérez" +"Ana María Luis Pérez" "Ana María de la Vega Rodríguez" "Ana María van der Berg" "Ana María-José López" "Ana Neto" "Ana SB" "Ana abdul Rahman López" +"Ana de Miguel de la Vega" "Andrew" "Andrew (Andy) Perkins" "Andrew Perkins (Andy)" @@ -648,12 +652,14 @@ "Jong, Anke de" "Jong, Piet de" "Jose E Maria Santos" +"Jose Luis Perez" "Jose Ortega-Y-Gasset" "Jose e Maria Santos" "Jose j y Lopez" "Josep Carod i Rovira" "Josep Lluis Carod i III" "Josep i Rovira" +"José Luis Pérez" "José María Ortega y Gasset" "José Silva Júnior" "Jr." @@ -817,6 +823,8 @@ "Prof. Dr. Brühl, Ada" "Prof. Smith, Dr. John" "Prof. Smith, John, PhD" +"Pérez José Luis" +"Pérez Luis José" "Pérez, Antonio Juan Carlos" "Pérez, Juan Carlos" "Pérez, Juan Ó." diff --git a/tools/differential/snapshot.jsonl b/tools/differential/snapshot.jsonl index 564381280..148cf7a7d 100644 --- a/tools/differential/snapshot.jsonl +++ b/tools/differential/snapshot.jsonl @@ -200,8 +200,10 @@ {"name": "Ana García", "given": "Ana", "family": "García", "initials": "A. G."} {"name": "Ana García (Anita) López", "given": "Ana", "middle": "García", "family": "López", "nickname": "Anita", "initials": "A. G. L."} {"name": "Ana J.R. García", "given": "Ana", "middle": "J.R.", "family": "García", "initials": "A. J. G."} +{"name": "Ana Jorge Flores", "given": "Ana", "middle": "Jorge", "family": "Flores", "initials": "A. J. F."} {"name": "Ana M. García", "given": "Ana", "middle": "M.", "family": "García", "initials": "A. M. G."} {"name": "Ana Marija Vera Kovač", "given": "Ana", "middle": "Marija Vera", "family": "Kovač", "initials": "A. M. V. K."} +{"name": "Ana Martín López", "given": "Ana", "middle": "Martín", "family": "López", "initials": "A. M. L."} {"name": "Ana María 3 López", "given": "Ana", "middle": "María 3", "family": "López", "initials": "A. M. 3. L."} {"name": "Ana María García - López", "given": "Ana", "middle": "María García -", "family": "López", "initials": "A. M. G. -. L."} {"name": "Ana María García López", "given": "Ana", "middle": "María García", "family": "López", "initials": "A. M. G. L."} @@ -209,6 +211,7 @@ {"name": "Ana María García–López", "given": "Ana", "middle": "María", "family": "García–López", "initials": "A. M. G."} {"name": "Ana María García－López", "given": "Ana", "middle": "María", "family": "García－López", "initials": "A. M. G."} {"name": "Ana María José Pérez", "given": "Ana", "middle": "María José", "family": "Pérez", "initials": "A. M. J. P."} +{"name": "Ana María Luis Pérez", "given": "Ana", "middle": "María Luis", "family": "Pérez", "initials": "A. M. L. P."} {"name": "Ana María de la Vega Rodríguez", "given": "Ana", "middle": "María", "family": "de la Vega Rodríguez", "initials": "A. M. V. R."} {"name": "Ana María van der Berg", "given": "Ana", "middle": "María", "family": "van der Berg", "initials": "A. M. B."} {"name": "Ana María-José López", "given": "Ana", "middle": "María-José", "family": "López", "initials": "A. M. L."} @@ -216,6 +219,7 @@ {"name": "Ana SB", "given": "Ana", "family": "SB", "initials": "A. S.", "capitalized_force": "Ana Sb"} {"name": "Ana Torres", "given": "Ana", "family": "Torres", "initials": "A. T."} {"name": "Ana abdul Rahman López", "given": "Ana", "middle": "abdul Rahman", "family": "López", "initials": "A. a. R. L.", "capitalized_force": "Ana Abdul Rahman López"} +{"name": "Ana de Miguel de la Vega", "given": "Ana", "middle": "de Miguel", "family": "de la Vega", "initials": "A. M. V."} {"name": "Anan ben David", "given": "Anan", "middle": "ben", "family": "David", "initials": "A. b. D.", "capitalized_force": "Anan Ben David"} {"name": "Andersen Irene", "order": "FAMILY_FIRST", "given": "Irene", "family": "Andersen", "initials": "I. A."} {"name": "Andersen, Irene", "given": "Irene", "family": "Andersen", "initials": "I. A."} @@ -2011,6 +2015,7 @@ {"name": "Jose Aznar y Lopez", "given": "Jose", "family": "Aznar y Lopez", "initials": "J. A. L."} {"name": "Jose E Maria Santos", "given": "Jose", "middle": "E Maria", "family": "Santos", "initials": "J. E. M. S."} {"name": "Jose E. Maria Santos", "given": "Jose", "middle": "E. Maria", "family": "Santos", "initials": "J. E. M. S."} +{"name": "Jose Luis Perez", "given": "Jose", "middle": "Luis", "family": "Perez", "initials": "J. L. P."} {"name": "Jose Ortega-Y-Gasset", "given": "Jose", "family": "Ortega-Y-Gasset", "initials": "J. O.", "capitalized_force": "Jose Ortega-y-Gasset"} {"name": "Jose e Maria", "given": "Jose", "middle": "e", "family": "Maria", "initials": "J. e. M."} {"name": "Jose e Maria Santos", "given": "Jose e Maria", "family": "Santos", "initials": "J. M. S."} @@ -2047,6 +2052,7 @@ {"name": "José Luis Bustamante y Rivero", "given": "José", "middle": "Luis", "family": "Bustamante y Rivero", "initials": "J. L. B. R."} {"name": "José Luis Covarrubias Garza", "given": "José", "middle": "Luis Covarrubias", "family": "Garza", "initials": "J. L. C. G."} {"name": "José Luis Martín Descalzo", "given": "José", "middle": "Luis Martín", "family": "Descalzo", "initials": "J. L. M. D."} +{"name": "José Luis Pérez", "given": "José", "middle": "Luis", "family": "Pérez", "initials": "J. L. P."} {"name": "José López Domínguez", "given": "José", "middle": "López", "family": "Domínguez", "initials": "J. L. D."} {"name": "José López Portillo y Pacheco", "given": "José", "middle": "López", "family": "Portillo y Pacheco", "initials": "J. L. P. P."} {"name": "José Manuel López de Abiada", "given": "José", "middle": "Manuel López", "family": "de Abiada", "initials": "J. M. L. A."} @@ -2977,7 +2983,9 @@ {"name": "Prof. Ɖr. MƉ fan ƉAM ƉOMIŊGO Fostɛɛŋ Samʊwɛl", "order": "FAMILY_FIRST", "title": "Prof. Ɖr.", "given": "fan", "middle": "ƉAM ƉOMIŊGO Fostɛɛŋ Samʊwɛl", "family": "MƉ", "initials": "f. Ɖ. Ɖ. F. S. M.", "capitalized_force": "Prof. Ɖr. Mɖ Fan Ɖam Ɖomiŋgo Fostɛɛŋ Samʊwɛl"} {"name": "Prop. Dr. Ada Cornelia César Martín von Brühl González Domingo MD DDS", "title": "Prop. Dr.", "given": "Ada", "middle": "Cornelia César Martín", "family": "von Brühl González Domingo", "suffix": "MD DDS", "initials": "A. C. C. M. B. G. D."} {"name": "Pèrs, Ciro di", "given": "Ciro", "family": "di Pèrs", "ambiguities": ["PARTICLE_OR_GIVEN", "SUFFIX_OR_NAME"], "alternatives": ["SUFFIX_OR_NAME: credential as suffix (vocabulary)", "PARTICLE_OR_GIVEN: name as middle (declared)"], "initials": "C. P."} +{"name": "Pérez José Luis", "given": "Pérez", "middle": "José", "family": "Luis", "initials": "P. J. L."} {"name": "Pérez Lucía", "order": "FAMILY_FIRST", "given": "Lucía", "family": "Pérez", "initials": "L. P."} +{"name": "Pérez Luis José", "given": "Pérez", "middle": "Luis", "family": "José", "initials": "P. L. J."} {"name": "Pérez, Antonio Juan Carlos", "given": "Antonio", "middle": "Juan Carlos", "family": "Pérez", "initials": "A. J. C. P."} {"name": "Pérez, Juan Carlos", "given": "Juan", "middle": "Carlos", "family": "Pérez", "initials": "J. C. P."} {"name": "Pérez, Juan Ó.", "given": "Juan", "middle": "Ó.", "family": "Pérez", "initials": "J. Ó. P."} diff --git a/tools/es_given_names.py b/tools/es_given_names.py new file mode 100644 index 000000000..4bf1b01f5 --- /dev/null +++ b/tools/es_given_names.py @@ -0,0 +1,238 @@ +"""Build the Spanish given-name list rules/O6's veto reads. + +WHY THIS IS IN THE TREE. `Policy(two_family_names=True)` (rules/O6) +moves the name word before the family into the family, and in "José +Luis Pérez" that word is a given name: most of what the switch gets +wrong is that shape (#959). A reader settles it by vocabulary, since +Spanish given names are a concentrated set and few words are common in +both roles, so O6 leaves a listed given name where it stands. The list +is a coverage decision with a cutoff, like locales.ZH's surnames, not a +set of words judged one by one, so it is derived here rather than kept +by hand: + +- Instituto Nacional de Estadística (INE), "Frecuencias de nombres": + every given name borne by at least 20 residents of Spain, by sex, + from the annual population census of 1 January 2025 + (nombres_por_edad_media.xlsx). Names are written whole, upper case + and without acute accents ("JOSE LUIS", "MARIA CARMEN"); Ñ and Ç + are kept. +- INE, "Frecuencias de apellidos": every surname borne as a first + surname by at least 20 residents, with its count as first and as + second surname, from the same census (apellidos_frecuencia.xls). + +Both are published under INE's general licence, Creative Commons +Attribution 4.0 (https://www.ine.es/aviso_legal, "Reutilización de la +información contenida en este sitio web"), which asks the reuser to +cite "Elaboración propia con datos extraídos del sitio web del INE: +www.ine.es" and the date the data were last updated. + +A word's given uses are the bearers of every listed name holding it, +summed over both sexes; its surname uses, the bearers of every listed +surname holding it, first and second surname together, less a surname +holding it behind a particle ("de Miguel"), a surname unit O6 takes +whole and the veto never reads. A word is a +candidate where at least SHARE of its uses are given-name uses, and the +candidates are ranked by given uses; the list is the first TOP. Words +the default vocabulary reads as something other than a name (a +particle, a conjunction, a title or a suffix) are left out, since O6 +never takes a word as its own unit that any of those readings claim, +and the disjointness test in tests/v2/test_es_given_names.py holds the +shipped list to that. The cutoff and SHARE are measured choices +(decisions/O6, 2026-10-30): a change to either is a cutoff change, +decided against the same per-country measurement. + + uv run --with openpyxl --with xlrd python tools/es_given_names.py + uv run --with openpyxl --with xlrd python tools/es_given_names.py --write + uv run --with openpyxl --with xlrd python tools/es_given_names.py --top 500 + +Sources are cached outside the repository, in $NAMEPARSER_TOOLS_CACHE +or ~/.cache/nameparser-tools (shared with name_usage.py); --refresh +fetches them again, and both tables move when INE publishes a new +census year. +""" +from __future__ import annotations + +import argparse +import sys +import urllib.request +from collections import Counter +from collections.abc import Collection, Iterable +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).parent)) +import name_usage # noqa: E402 (a sibling script, not a package) + +MODULE = (Path(__file__).parents[1] / "nameparser" / "locales" + / "_es_given_names.py") +#: The decided cutoff and share (decisions/O6, 2026-10-30). +TOP = 1000 +SHARE = 0.9 + +NAMES_URL = "https://www.ine.es/daco/daco42/nombyapel/nombres_por_edad_media.xlsx" +NAMES_FILE = "ine_nombres_2025.xlsx" +SURNAMES_URL = "https://www.ine.es/daco/daco42/nombyapel/apellidos_frecuencia.xls" +SURNAMES_FILE = "ine_apellidos_2025.xls" + +BEGIN = "# -- BEGIN generated by tools/es_given_names.py; edit the tool, not this block --" +END = "# -- END generated --" + + +def fold(word: str) -> str: + """`word` as the list is written and matched: the parse's own key, + `_vocab.given_name_key`, which lower-cases and drops the acute and + grave accents and the diaeresis, keeping the tilde on n and the + cedilla on c, as INE writes names. One function, so the list and + the lookup cannot fold apart.""" + from nameparser._pipeline._vocab import given_name_key + return given_name_key(word) + + +def words(name: str) -> list[str]: + """A name's words, folded; a word holding anything but letters (an + apostrophe, a stray mark) is left out, being no word a name list + can carry whole.""" + return [w for w in (fold(part) for part in name.split()) if w.isalpha()] + + +def tally(rows: Iterable[tuple[str, int]], + after: Collection[str] = ()) -> Counter[str]: + """Each word's bearers over (name, bearers) rows, less a word + written right after one of `after`: a surname written behind a + particle ("de Miguel") is a surname unit O6 takes whole and the + veto never reads, so it says nothing about whether the bare word + is a surname.""" + counts: Counter[str] = Counter() + for name, bearers in rows: + written = words(name) + for i, word in enumerate(written): + if not (i and written[i - 1] in after): + counts[word] += bearers + return counts + + +def select(given: Counter[str], surname: Counter[str], excluded: set[str], + top: int = TOP, share: float = SHARE) -> list[str]: + """The `top` words with the most given uses among those at least + `share` of whose uses are given-name uses, most-borne first, less + `excluded` (the default vocabulary's non-name readings) and words + of one letter, which write an initial.""" + ranked = sorted( + (w for w, g in given.items() + if len(w) > 1 and w not in excluded + and g >= share * (g + surname[w])), + key=lambda w: (-given[w], w)) + return ranked[:top] + + +def excluded_words() -> set[str]: + """Every word the default vocabulary reads as a particle, a + conjunction, a title or a suffix, folded as the list is.""" + from nameparser import Lexicon + lex = Lexicon.default() + pool = (lex.particles | lex.conjunctions | lex.titles + | lex.titles_ambiguous | lex.suffix_acronyms | lex.suffix_words) + return {fold(w) for entry in pool for w in entry.split()} + + +def render(ranked: list[str]) -> str: + rows = [", ".join(f'"{w}"' for w in ranked[i:i + 8]) + for i in range(0, len(ranked), 8)] + body = "\n".join(f" {row}," for row in rows) + return f"{BEGIN}\n_RANKED = (\n{body}\n)\n{END}" + + +def rewrite(module_text: str, block: str) -> str: + start, end = module_text.find(BEGIN), module_text.find(END) + if start < 0 or end < start: + raise ValueError(f"{MODULE} has no generated block to replace") + return module_text[:start] + block + module_text[end + len(END):] + + +# -- sources -------------------------------------------------------------- + +def _cached(url: str, name: str, refresh: bool, magic: bytes) -> Path: + """The cached copy of `url`, fetched where absent or on --refresh; + an answer that does not open with the format's `magic` bytes (an + error page) is refused rather than cached.""" + path = name_usage.cache_dir() / name + if refresh or not path.exists(): + request = urllib.request.Request( + url, headers={"User-Agent": name_usage.USER_AGENT}) + with urllib.request.urlopen(request, timeout=120) as response: + data: bytes = response.read() + if not data.startswith(magic): + raise ValueError(f"{url} returned no spreadsheet; " + f"it began {data[:80]!r}") + path.write_bytes(data) + return path + + +def load_names(refresh: bool) -> list[tuple[str, int]]: + """INE's given names, both sexes, as (name, bearers). The rows open + under a header row whose first cell is "Orden".""" + import openpyxl + path = _cached(NAMES_URL, NAMES_FILE, refresh, b"PK") + rows: list[tuple[str, int]] = [] + workbook = openpyxl.load_workbook(path, read_only=True) + for sheet in workbook.worksheets: + body = False + for row in sheet.iter_rows(values_only=True): + if body and row[1]: + rows.append((str(row[1]), int(row[2]))) + body = body or row[0] == "Orden" + if len(rows) < 50_000: + raise ValueError(f"{path} holds {len(rows)} names; the 2025 " + "tables hold over 63,000") + return rows + + +def load_surnames(refresh: bool) -> list[tuple[str, int]]: + """INE's surnames as (surname, bearers as first plus as second + surname). A count INE withholds is written ".." and read as 0.""" + import xlrd + path = _cached(SURNAMES_URL, SURNAMES_FILE, refresh, + b"\xd0\xcf\x11\xe0") + rows: list[tuple[str, int]] = [] + workbook = xlrd.open_workbook(path) + for sheet in workbook.sheets(): + body = False + for i in range(sheet.nrows): + row = sheet.row_values(i) + if body and row[1]: + count = sum(int(v) for v in row[2:4] if isinstance(v, float)) + rows.append((str(row[1]), count)) + body = body or row[0] == "Orden" + if len(rows) < 80_000: + raise ValueError(f"{path} holds {len(rows)} surnames; the 2025 " + "tables hold over 86,000") + return rows + + +def main(argv: list[str] | None = None) -> int: + parser = argparse.ArgumentParser(description=__doc__.split("\n")[0]) + parser.add_argument("--write", action="store_true", + help=f"rewrite the generated block in {MODULE.name}") + parser.add_argument("--refresh", action="store_true", + help="fetch the sources again") + parser.add_argument("--top", type=int, default=TOP) + parser.add_argument("--share", type=float, default=SHARE) + args = parser.parse_args(argv) + excluded = excluded_words() + given = tally(load_names(args.refresh)) + surname = tally(load_surnames(args.refresh), after=excluded) + ranked = select(given, surname, excluded, args.top, args.share) + for i, word in enumerate(ranked, 1): + print(f"{i:5} {word:20} given {given[word]:>9,} " + f"surname {surname[word]:>8,}") + covered = sum(given[w] for w in ranked) + print(f"{len(ranked)} words, {covered / sum(given.values()):.1%} of " + "given-word uses", file=sys.stderr) + if args.write: + MODULE.write_text(rewrite(MODULE.read_text(encoding="utf-8"), + render(ranked)), encoding="utf-8") + print(f"wrote {MODULE}", file=sys.stderr) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) From e06b2612fa9a0acada6413ba6522dfa8285542b2 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Fri, 30 Oct 2026 16:41:53 -0700 Subject: [PATCH 2/4] Address #961 review: family-first cost, check 11, unshipped list, notices MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - rules/O6: the Accepted cost under each order (Flores Jorge Ana), the common-in-both-roles sentence moved off the rule, Signal names the taken word against the list as a second signal (structure held for the maintainer). - Say everywhere that given_names is empty by default and the Spanish list ships for the es pack, which does not exist yet; customize.rst no longer quotes a result from the private list. - decisions/O6: check 11 described accurately (the test is stricter), #959's 271-of-466 restored, Peru compared like with like, family-first measured on the rendering, and Open items for Latin America and a dropped ñ. - The veto's comment credits len(take) == 1 and drops an unreachable particle arm; the shipped module carries the CC BY 4.0 URI and the files' last-update date. Refs #959 Co-Authored-By: Claude Opus 5.5 --- docs/customize.rst | 19 ++++++++++--------- docs/design/decisions/O6-two-family-names.md | 15 ++++++++------- docs/design/rules/O6-two-family-names.md | 8 +++++--- nameparser/_lexicon.py | 6 ++++-- nameparser/_pipeline/_post_rules.py | 9 +++++---- nameparser/locales/_es_given_names.py | 11 +++++++---- tests/v2/rules_budget.json | 2 +- tests/v2/test_es_given_names.py | 13 ++++++++----- tools/differential/corpus_rules.jsonl | 1 + tools/differential/snapshot.jsonl | 1 + tools/es_given_names.py | 12 +++++++----- 11 files changed, 57 insertions(+), 40 deletions(-) diff --git a/docs/customize.rst b/docs/customize.rst index 3d0ba7615..99e966df6 100644 --- a/docs/customize.rst +++ b/docs/customize.rst @@ -1026,9 +1026,10 @@ the same either way. A given name standing before the family is most of what the switch gets wrong, as in ``"José Luis Pérez"``. Words that are given names and rarely surnames tell the two shapes apart, so the switch leaves a word -listed in the lexicon's ``given_names`` where it stands. The list is -empty by default; an entry written without accents matches the word -with them: +listed in the lexicon's ``given_names`` where it stands, under any +``name_order``. The field is empty by default, so you supply the list +for your data; an entry written without accents matches the word with +them: .. doctest:: @@ -1040,12 +1041,12 @@ with them: >>> luis.middle, luis.family ('Luis', 'Pérez') -Among ORCID's researchers, a list of the thousand words most borne as -given names in Spain and mostly not as surnames (from Spain's 2025 -census) raised the switch's agreement from 93.4% to 95.3% in Spain and -from 61.2% to 89.3% in Argentina, where the switch alone breaks most -names (measured 2026-10-30). A listed word that is someone's first -surname, as given names sometimes are, stays a middle name. +List words that are given names and rarely surnames in your data: a +word common in both roles, such as ``Martín``, is better left off. A +listed word that is someone's surname, as given names sometimes are, +stays out of the family: written given-first it becomes a middle name, +and under a family-first ``name_order`` the second surname becomes the +given name or a middle name. .. _two-given-names: diff --git a/docs/design/decisions/O6-two-family-names.md b/docs/design/decisions/O6-two-family-names.md index 38e6f31bd..e99337631 100644 --- a/docs/design/decisions/O6-two-family-names.md +++ b/docs/design/decisions/O6-two-family-names.md @@ -11,9 +11,9 @@ - The switch reports nothing — 2026-10-24 #754. - The switch reports no alternative either, so the reading it did not take is not surfaced: the caller declared the convention — 2026-10-27 #752 (Derek). - A listed given name (`Lexicon.given_names`) standing alone is not taken, under every order, and the name then reads as with the switch off; a listed word behind a particle is taken as before, and with the list empty the switch reads as before; no other rule reads the list, the default carries none, and the veto reports nothing — 2026-10-30 #959. -- The list a pack carries for Spanish is generated, not per-word vocabulary: INE's 2025 census tables, pooled over every name borne in Spain, cut at a measured rank (1,000 words, 90% given use) by `tools/es_given_names.py`; a change to it is a cutoff change — 2026-10-30 #959. +- The Spanish list, shipped for sub-phase 3g's `es` pack to carry (no pack carries it yet, and a caller may pass a list of its own), is generated, not per-word vocabulary: INE's 2025 census tables, pooled over every name borne in Spain, cut at a measured rank (1,000 words, 90% given use) by `tools/es_given_names.py`; a change to it is a cutoff change — 2026-10-30 #959. -Open: hyphenated surnames block the switch (`Andrés Sánchez-Cid Torres` keeps `Sánchez-Cid` a middle name), 271 of the 466 Spanish records the issue's held-out per-country list still read wrong (measured on the issue, 2026-10-30) — #959. No rule example shows both switches on over a given name a particle completes (`María del Carmen de la Vega`, given `María del Carmen` under O7 alone and family `del Carmen de la Vega` with O6 on too) — #959. +Open: hyphenated surnames block the switch (`Andrés Sánchez-Cid Torres` keeps `Sánchez-Cid` a middle name), of the 466 Spanish records the issue's held-out per-country list still read wrong, 271 are this shape (measured on the issue, 2026-10-30) — #959. No rule example shows both switches on over a given name a particle completes (`María del Carmen de la Vega`, given `María del Carmen` under O7 alone and family `del Carmen de la Vega` with O6 on too) — #959. Why the Spanish list does less in Latin America, and whether 3g needs a list per region or another selection rule (the 2026-10-30 #959 entry's Latin America bullet) — #959. A name written with ñ dropped (`Begona`, `Inigo`) matches no entry, the key keeping the tilde; whether to fold ñ for data that strips it — #959. ## Log @@ -63,16 +63,17 @@ Open: hyphenated surnames block the switch (`Andrés Sánchez-Cid Torres` keeps Measured on master at 598167f4 over the 2025 Public Data File's extract fetched 2026-10-27. Recompute: `local_agreement.read` per record of `local_corpora.load("orcid-researchers")` with `FIELDED["orcid-researchers"]`, under `Parser()` and under `Parser(policy=Policy(two_family_names=True))` with `_title_words` of the default lexicon, grouped by the record's `country`; a record is fixed where it agrees only with the switch, broken where it agrees only without it, and the last column counts the broken records whose `family` box holds one word. A corpus capped at 20,000 records a country shows each country's convention, not how many people write it. - 2026-10-29 #754, sub-phase 4d — THE TWO UNITS THIS RULE READS REACH A VIEW: `surname2` holds the one written second, `family` keeping both as before, and the rule marks that unit's words where it takes its word; nothing it reads or declines moved. The view, the mark and their measurement are rules/R10 and decisions/R10's 2026-10-29 entry. -- 2026-10-30 #959 — A LISTED GIVEN NAME IS NOT TAKEN: an amendment to rules/O6's Except bullet "What is taken must be one name word a family name could be", not a new rule. In `José Luis Pérez` the word the switch takes is a given name, and that shape is most of what the switch gets wrong (the 2026-10-27 entry: the records it breaks nearly all hold a one-word family box). A Spanish reader settles it by vocabulary: given names are a concentrated set and few words are common in both roles, so a word only a given name is says the name holds one family name. The rule reads `Lexicon.given_names`, empty by default, and a pack carries the list; with the list empty the switch reads exactly as before (the snapshot gains the rule's example texts and moves no reading). +- 2026-10-30 #959 — A LISTED GIVEN NAME IS NOT TAKEN: an amendment to rules/O6's Except bullet "What is taken must be one name word a family name could be", not a new rule. In `José Luis Pérez` the word the switch takes is a given name, and that shape is most of what the switch gets wrong (the 2026-10-27 entry: the records it breaks nearly all hold a one-word family box). A Spanish reader settles it by vocabulary: given names are a concentrated set and few words are common in both roles, so a word only a given name is says the name holds one family name. The rule reads `Lexicon.given_names`, empty by default, which a caller fills, and which sub-phase 3g's `es` pack is to fill with the list shipped here; with the list empty the switch reads exactly as before (the snapshot gains the rule's example texts and moves no reading). What it settles, and why each way: - - **Where the list lives: generated, as locales.ZH's surnames are.** `nameparser/locales/_es_given_names.py` holds the list in a block `tools/es_given_names.py` writes, for the `es` pack (sub-phase 3g of #754) to carry as a lexicon fragment, `Lexicon(given_names=GIVEN_NAMES)`. Weighed against: `given` readings in an `es` language file, which per-word TOML would carry a thousand words judged one at a time, where what is decided is a cutoff; the evidence for the set is the tool's docstring and this entry, and a change to it is decided as a cutoff change, as ZH's top 300 is (decisions/W1). The TOML build's check 11, which refuses a given reading on a word another reading claims as no name, is replaced by a test holding the list disjoint from the default's never-given particles, titles, suffixes and conjunctions (tests/v2/test_es_given_names.py, with its recorded negative control); the tool leaves every particle out. - - **The source: INE.** The Instituto Nacional de Estadística publishes every given name borne by at least 20 residents of Spain, by sex, and every surname borne as a first surname by at least 20, with its counts as first and second surname, both from the census of 1 January 2025. Its legal notice licenses the site's statistics as "la Creative Commons Reconocimiento 4.0 (CC BY 4.0)" (https://www.ine.es/aviso_legal, read 2026-10-30), asking the reuser to cite "Elaboración propia con datos extraídos del sitio web del INE: www.ine.es" and the date of the data, which the module's docstring does. A word's given uses are the bearers of the names holding it; its surname uses, the bearers of the surnames holding it, less those holding it behind a particle (`de Miguel`), which the switch never takes as the bare word (with them counted, `Miguel` is 89% given and falls out; without, 92%). A word is a candidate where at least 90% of its uses are given, #959's share; INE writes names without acute accents, so the parse matches a word with its accents and diaeresis dropped and the tilde on n and the cedilla on c kept (`given_name_key`), and a caller's entry also as written. Words common in both roles stay out (`Martín` 10% given, `Vicente` 63%, `Santiago` 57%, `Cruz` 18%). Weighed against: a list built from ORCID's boxes, which #959 measured on a held-out half; INE is licensed for redistribution, counts a whole population rather than researchers, and is not the population scored, so no held-out split is needed. - - **One pooled list, Spain's.** INE counts Spain's residents, Latin American-born ones among them, and #959 found a list pooled over six countries within half a point of per-country lists everywhere but Argentina and Peru. Peru is where the pooled list costs most: there it fixes about as many records as it breaks at every cutoff, the breaks being first surnames that are given names in Spain (`Jorge` 12, `Carlos` 11, `Pablo` 6 at the 1,000 cut), where #959's per-country list, learned from Peru's own boxes, broke 58. + - **Where the list lives: generated, as locales.ZH's surnames are.** `nameparser/locales/_es_given_names.py` holds the list in a block `tools/es_given_names.py` writes, for the `es` pack (sub-phase 3g of #754) to carry as a lexicon fragment, `Lexicon(given_names=GIVEN_NAMES)`. Weighed against: `given` readings in an `es` language file, which per-word TOML would carry a thousand words judged one at a time, where what is decided is a cutoff; the evidence for the set is the tool's docstring and this entry, and a change to it is decided as a cutoff change, as ZH's top 300 is (decisions/W1). A generated list does not pass through the TOML build, so its check 11 does not reach it; that check lets a `given` reading stand beside a title or particle reading and derives the marker the pair means (`prince`, a title with a given reading, compiles to `titles_ambiguous`), refusing only the phase 1 flags (`may_be_name`, `may_be_given = true`) and a particle stating `may_be_given = false` beside a given reading. The list is held to something stricter, a test keeping it disjoint from every word the default lists as a never-given particle, a title (an ambiguous one too), a suffix or a conjunction (tests/v2/test_es_given_names.py, with its recorded negative control), since a word whose own reading another rule weighs is no word this veto should decide; the tool leaves every particle out. + - **The source: INE.** The Instituto Nacional de Estadística publishes every given name borne by at least 20 residents of Spain, by sex, and every surname borne as a first surname by at least 20, with its counts as first and second surname, both from the census of 1 January 2025. Its legal notice licenses the site's statistics as "la Creative Commons Reconocimiento 4.0 (CC BY 4.0)" (https://www.ine.es/aviso_legal, read 2026-10-30), asking the reuser to cite "Elaboración propia con datos extraídos del sitio web del INE: www.ine.es" and the date of the data; the shipped module's docstring gives both, with the licence's URI and the files' last update (23 April 2026, the server's `Last-Modified` on both files). A word's given uses are the bearers of the names holding it; its surname uses, the bearers of the surnames holding it, less those holding it behind a particle (`de Miguel`), which the switch never takes as the bare word (with them counted, `Miguel` is 89% given and falls out; without, 92%). A word is a candidate where at least 90% of its uses are given, #959's share; INE writes names without acute accents, so the parse matches a word with its accents and diaeresis dropped and the tilde on n and the cedilla on c kept (`given_name_key`), and a caller's entry also as written. Words common in both roles stay out (`Martín` 10% given, `Vicente` 63%, `Santiago` 57%, `Cruz` 18%). Weighed against: a list built from ORCID's boxes, which #959 measured on a held-out half; INE is licensed for redistribution, counts a whole population rather than researchers, and is not the population scored, so no held-out split is needed. + - **One pooled list, Spain's.** INE counts Spain's residents, Latin American-born ones among them, and #959 found a list pooled over six countries within half a point of per-country lists everywhere but Argentina and Peru. Peru is where the pooled list costs most: there it fixes about as many records as it breaks at every cutoff, the breaks being first surnames that are given names in Spain (`Jorge` 12, `Carlos` 11, `Pablo` 6 at the 1,000 cut), where #959's per-country list, learned from Peru's own boxes, broke 58 of the 9,735 records of its held-out half, and its list pooled over six countries 102 of the same; this list breaks 189 of all 19,471. - **The cutoff: 1,000 words** (84.3% of the given-word uses in INE's table). Measured per country with O6 on, the list cut at 100 to all 16,559 candidates; agreement in each of the six double-surname countries is within 0.3 points of the 2,000-word list at 1,000 (Venezuela the 0.3), and a band past it buys at most a tenth of a net fix per added word in those six (1,000 → 2,000: 134 fixed, 35 broken; 2,000 → 5,000: 128 fixed, 36 broken, Chile breaking as many as it fixes), where the first hundred words bought sixteen. Argentina gains 1.6 points more by 2,000 and 2.4 by all, which 3g weighs if it turns the switch on there. - - **Family-first, symmetric.** Under a family-first order the switch takes the unit after the family, and a listed one stays where it is: `Pérez José Luis` keeps family `Pérez`. No population the corpora hold writes Spanish names family-first with the fields stated, so it is measured on ORCID's boxes rendered family box first, which is a rendering, not anyone's writing. + - **Family-first, symmetric.** Under a family-first order the switch takes the unit after the family, and a listed one stays where it is: `Pérez José Luis` keeps family `Pérez`. No population the corpora hold writes Spanish names family-first with the fields stated, so it is measured on ORCID's boxes rendered family box first, which is a rendering, not anyone's writing. The cost moves with the order: a listed word that is a second surname, written after the first, becomes the given name under FAMILY_FIRST (`Flores Jorge Ana` gives given `Jorge`, middle `Ana`) and a middle name under FAMILY_FIRST_GIVEN_LAST, where given-first it is a first surname left as a middle name; on the rendering the list breaks 38 (Chile) to 205 (Peru) records a country, the family-first line below (found by the #961 review). - **A bare word only.** The veto reads a unit that is one word with no particle; a listed word behind a particle is a surname unit and is taken as before (`Ana de Miguel de la Vega` keeps family `de Miguel de la Vega`), which is what the tool's count assumes in leaving such surnames out. The first draft read the word inside a particle unit too, and the design-docs review found it reading `Ana de Miguel de la Vega` middle `de Miguel`. - **O7.** With both switches on, the word O6 leaves is a middle O7 takes where it stands beside the given name, so `José Luis Pérez` reads given `José Luis`, family `Pérez` (rules/O7's precedence); `Ana María Luis Pérez` keeps middle `Luis`, O7 taking `María`. Whether O7 reads the list itself is decisions/O7's Open question. - **No report**, as for the switch (the 2026-10-24 entry): the caller declared the convention and the list. + - **Latin America, measured in the #961 review.** Of the 27 words Argentina's records most often hold where the veto still lets the switch take a given name (825 records), about 712 records are lost to the 90% bar measured on Spain's population and about 113 to the 1,000 cut: `Martín` (10% given in INE's tables), `Andrés` 77%, `Nicolás` 76%, `Sebastián` 81%, `Esteban` 36%, `Cecilia` 89.75% (84 Argentine records, just under the bar), `Matías` 76%, `Hernán` 63%, `Gastón` 40%, `Santiago` 57% and `Tomás` 61%, the shares recomputed here. Peru is net negative because the switch alone already agrees on 96.0% of its records, leaving little to fix, and its breaks spread over ranks 2 to 452 (`jorge` 33rd, `carlos` 13th), so they come from the selection rule rather than the cutoff. Whether 3g needs a list per region or another selection rule is Open above; the list is not changed here. - **Pickling.** The field carries `_ADDED_IN` "3.0.0" and an empty default, so a `Lexicon`, `Parser` or locale pack pickled by 2.x loads with no list. That is what such a pickle meant, no pack filling the field before 3.0, so this field is the case decisions/A1's 2026-10-30 caveat does not reach: the silent drift it warns of comes from a marked field the default vocabulary fills, and the default leaves this one empty. `tests/v2/test_lexicon.py` loads the committed 2.8.0 pickle with the field empty, and refuses it with the marker taken off. Measured 2026-10-30 on the branch, the 2025 Public Data File extract fetched 2026-10-27 (`orcid-researchers`), every record with both boxes filled; all boxes agreeing, by default, with the switch, and with the switch and the 1,000-word list, and the records the list fixes and breaks against the switch alone: diff --git a/docs/design/rules/O6-two-family-names.md b/docs/design/rules/O6-two-family-names.md index b78b5289b..b5c0fe798 100644 --- a/docs/design/rules/O6-two-family-names.md +++ b/docs/design/rules/O6-two-family-names.md @@ -59,7 +59,7 @@ ``` ", Ana García López" two_family_names → family="" ``` -- **What is taken must be one name word a family name could be**: a part counting as two words (a hyphenated given name, a bound given-name pair, P5) stays where it is, and so do an initial written alone, which writes a given name, and a mark or number with no letter in it. A particle chain (P2) is never split, so where taking it would leave no given name the rule declines. A listed given name stays where it is too, under either order, and a list written without accents matches the word written with them: the caller's vocabulary names it a word only a given name is, so the name holds one family name. A word common in both roles is not listed, and a listed word behind a particle is a surname unit, so either is taken as before. +- **What is taken must be one name word a family name could be**: a part counting as two words (a hyphenated given name, a bound given-name pair, P5) stays where it is, and so do an initial written alone, which writes a given name, and a mark or number with no letter in it. A particle chain (P2) is never split, so where taking it would leave no given name the rule declines. A listed given name stays where it is too, under either order, and a list written without accents matches the word written with them: the caller's vocabulary names it a word only a given name is, so the name holds one family name. A listed word behind a particle is a surname unit, taken as before. ``` "José Luis Pérez" two_family_names+es_given_names → family="Pérez" "Jose Luis Perez" two_family_names+es_given_names → middle="Luis" @@ -120,12 +120,14 @@ ``` "Ana María van der Berg" two_family_names → family="María van der Berg" ``` -- A listed given name that is also someone's first surname stays a middle name, the listing saying which role the word mostly fills and not which it fills here. +- A listed given name that is also someone's surname stays out of the family, the listing saying which role the word mostly fills and not which it fills here: a first surname stays a middle name given-first, and under a family-first order a second surname becomes the given name or a middle name. ``` "Ana Jorge Flores" two_family_names+es_given_names → family="Flores" + "Flores Jorge Ana" family-first+two_family_names+es_given_names → given="Jorge" + "Flores Jorge Ana" family-first-given-last+two_family_names+es_given_names → middle="Jorge" ``` - The rule reports nothing: the caller declared the convention, as a declared order is followed without a report, and a report on most names of three words or more would teach the caller to ignore the field. --- -**Signal:** none in the name: a caller switch, the family's words and the taken word's shape read as written to see whether the family already holds two, and the taken word's listing as a given name +**Signal:** none in the name + shape: a caller switch, the family's words and the taken word's shape read as written to see whether the family already holds two; and the taken word as written, against the caller's given-name list **history:** decisions/O6 · **implemented:** nameparser/_pipeline/_post_rules.py diff --git a/nameparser/_lexicon.py b/nameparser/_lexicon.py index b01539dce..72897c179 100644 --- a/nameparser/_lexicon.py +++ b/nameparser/_lexicon.py @@ -787,8 +787,10 @@ class Lexicon: #: "José Luis Pérez" keeps family "Pérez". A word matches as #: written or with its acute and grave accents and diaeresis #: dropped (the tilde on n and the cedilla on c kept), so an entry - #: written "jose" matches "José" too. No other rule reads it, and - #: the default is empty: a pack carries the list for its language. + #: written "jose" matches "José" too. No other rule reads it. Empty + #: by default, so the switch reads as without it; a caller passes + #: a list of its own, and a Spanish list ships for the ``es`` pack + #: to carry once that pack exists. #: Added in 3.0, so a Lexicon pickled by 2.x loads with it empty, #: which is what an older pickle means: no pack filled it then. given_names: frozenset[str] = field( diff --git a/nameparser/_pipeline/_post_rules.py b/nameparser/_pipeline/_post_rules.py index 30bac0cd6..37dcd1a4e 100644 --- a/nameparser/_pipeline/_post_rules.py +++ b/nameparser/_pipeline/_post_rules.py @@ -301,10 +301,11 @@ def _take_second_family_name(state: ParseState, return # rules/O6: "a listed given name stays where it is" -- the list a # pack carries (Lexicon.given_names) names a word only a given name - # is, under either order; a word behind a particle is a surname - # unit, taken as before. History: decisions/O6 (#959). - if len(take) == 1 and "particle" not in tokens[take[0]].tags \ - and is_listed_given_name(tokens[take[0]].text, state.lexicon): + # is, under either order. Only a unit of one token is asked: a word + # behind a particle ("de Miguel") is a unit of two, a surname taken + # as before. History: decisions/O6 (#959). + if len(take) == 1 and is_listed_given_name(tokens[take[0]].text, + state.lexicon): return if state.order[0] is Role.GIVEN: # given, middles..., family: the middle unit standing right diff --git a/nameparser/locales/_es_given_names.py b/nameparser/locales/_es_given_names.py index 120e8e54b..ac3578164 100644 --- a/nameparser/locales/_es_given_names.py +++ b/nameparser/locales/_es_given_names.py @@ -4,10 +4,13 @@ a lexicon fragment, ``Lexicon(given_names=GIVEN_NAMES)``, as locales.ZH carries its surnames; no pack carries it yet. -Data source: Instituto Nacional de Estadística (INE), "Frecuencias de -nombres" and "Frecuencias de apellidos", from the annual population -census of 1 January 2025 (Elaboración propia con datos extraídos del -sitio web del INE: www.ine.es; Creative Commons Attribution 4.0). The +Data source: Elaboración propia con datos extraídos del sitio web del +INE: www.ine.es. Instituto Nacional de Estadística (INE), "Frecuencias +de nombres" and "Frecuencias de apellidos", from the annual population +census of 1 January 2025, files last updated 23 April 2026; licensed +under Creative Commons Attribution 4.0 +(https://creativecommons.org/licenses/by/4.0/), per INE's legal notice +(https://www.ine.es/aviso_legal). This module is derived from them. The words most borne as a given name in Spain among those at least 90% of whose uses are given-name uses, in rank order, written as INE writes them -- lower-cased here, without accents, with ñ and ç -- which diff --git a/tests/v2/rules_budget.json b/tests/v2/rules_budget.json index 7933f7fd8..42a7ce3eb 100644 --- a/tests/v2/rules_budget.json +++ b/tests/v2/rules_budget.json @@ -39,7 +39,7 @@ "O3": {"words": 47, "then_words": 18, "because_sentences": 1, "except": 0, "precedence": 0, "accepted": 0, "accepted_words": 0}, "O4": {"words": 91, "then_words": 57, "because_sentences": 1, "except": 0, "precedence": 0, "accepted": 0, "accepted_words": 0}, "O5": {"words": 478, "then_words": 160, "because_sentences": 2, "except": 6, "precedence": 1, "accepted": 0, "accepted_words": 0}, - "O6": {"words": 665, "then_words": 161, "because_sentences": 2, "except": 5, "precedence": 4, "accepted": 8, "accepted_words": 273}, + "O6": {"words": 652, "then_words": 161, "because_sentences": 2, "except": 5, "precedence": 4, "accepted": 8, "accepted_words": 297}, "O7": {"words": 598, "then_words": 175, "because_sentences": 2, "except": 4, "precedence": 3, "accepted": 6, "accepted_words": 220}, "P1": {"words": 415, "then_words": 287, "because_sentences": 2, "except": 0, "precedence": 1, "accepted": 4, "accepted_words": 319}, "P2": {"words": 315, "then_words": 279, "because_sentences": 1, "except": 0, "precedence": 0, "accepted": 3, "accepted_words": 167}, diff --git a/tests/v2/test_es_given_names.py b/tests/v2/test_es_given_names.py index a3223047b..a53d23ec3 100644 --- a/tests/v2/test_es_given_names.py +++ b/tests/v2/test_es_given_names.py @@ -2,11 +2,14 @@ checks on tools/es_given_names.py, which writes it. The list is generated, not judged word by word, so what a test can hold -is what no cutoff may change: no word the default vocabulary reads as -something other than a name (the TOML build's check 11 for a per-word -reading, which a generated list does not pass through), every entry in -the fold the parse matches by, and the tool's selection on hand-built -rows. The tool's sources are two INE downloads, which no test touches. +is what no cutoff may change: no word the default vocabulary lists as a +never-given particle, a title (an ambiguous one too), a suffix or a +conjunction -- stricter than the TOML build's check 11, which a +generated list does not pass through and which lets a given reading +stand beside a title or particle reading, deriving a marker from the +pair -- every entry in the fold the parse matches by, and the tool's +selection on hand-built rows. The tool's sources are two INE downloads, +which no test touches. """ from __future__ import annotations diff --git a/tools/differential/corpus_rules.jsonl b/tools/differential/corpus_rules.jsonl index 509c82364..f74da78f7 100644 --- a/tools/differential/corpus_rules.jsonl +++ b/tools/differential/corpus_rules.jsonl @@ -235,6 +235,7 @@ "Ế Nguyen" "FRITZ FREIHERR UND VON BAR" "Field Marshal Bernard Montgomery" +"Flores Jorge Ana" "Freiherr St John Smith" "Freiherr von Berg MA" "Freiherr von Berg MA Ma" diff --git a/tools/differential/snapshot.jsonl b/tools/differential/snapshot.jsonl index 148cf7a7d..9825fef44 100644 --- a/tools/differential/snapshot.jsonl +++ b/tools/differential/snapshot.jsonl @@ -1029,6 +1029,7 @@ {"name": "Fischer, Paul Vinzent von", "given": "Paul", "middle": "Vinzent", "family": "von Fischer", "ambiguities": ["PARTICLE_OR_GIVEN"], "alternatives": ["PARTICLE_OR_GIVEN: name as middle (declared)"], "initials": "P. V. F."} {"name": "Fitzhugh, Louise", "given": "Louise", "family": "Fitzhugh", "initials": "L. F."} {"name": "Fleksnes, Gregers Werle Marve Almar von", "given": "Gregers", "middle": "Werle Marve Almar", "family": "von Fleksnes", "ambiguities": ["PARTICLE_OR_GIVEN"], "alternatives": ["PARTICLE_OR_GIVEN: name as middle (declared)"], "initials": "G. W. M. A. F."} +{"name": "Flores Jorge Ana", "given": "Flores", "middle": "Jorge", "family": "Ana", "initials": "F. J. A."} {"name": "Flores Marco Antonio", "order": "FAMILY_FIRST", "given": "Marco", "middle": "Antonio", "family": "Flores", "initials": "M. A. F."} {"name": "Flores, Marco Antonio", "given": "Marco", "middle": "Antonio", "family": "Flores", "initials": "M. A. F."} {"name": "Flávio Juvenal da Silva Jr", "given": "Flávio", "middle": "Juvenal", "family": "da Silva", "suffix": "Jr", "initials": "F. J. S.", "generation": "Jr"} diff --git a/tools/es_given_names.py b/tools/es_given_names.py index 4bf1b01f5..0c550a2ce 100644 --- a/tools/es_given_names.py +++ b/tools/es_given_names.py @@ -33,11 +33,13 @@ whole and the veto never reads. A word is a candidate where at least SHARE of its uses are given-name uses, and the candidates are ranked by given uses; the list is the first TOP. Words -the default vocabulary reads as something other than a name (a -particle, a conjunction, a title or a suffix) are left out, since O6 -never takes a word as its own unit that any of those readings claim, -and the disjointness test in tests/v2/test_es_given_names.py holds the -shipped list to that. The cutoff and SHARE are measured choices +the default vocabulary lists as a particle, a conjunction, a title or a +suffix, ambiguous readings included, are left out: a list of given +names has no use for a word whose own vocabulary reading another rule +weighs (a title that may be a given name, `prince`, reaches O6 as a +name word and would be vetoed), so the list stays clear of every such +reading, and the disjointness test in tests/v2/test_es_given_names.py +holds the shipped list to that. The cutoff and SHARE are measured choices (decisions/O6, 2026-10-30): a change to either is a cutoff change, decided against the same per-country measurement. From 9c1e2fac56824a56e635fce6ca3b88d66aefafb2 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Fri, 30 Oct 2026 16:50:35 -0700 Subject: [PATCH 3/4] Address #961 second review: recipes, check 11, particle guard - decisions/O6: Peru's breaks rank 1 to 994 in the list and Argentina's top 27 words are 836 records (722 to the bar, 114 to the cut), each with its recipe; check 11 stated in full rather than as "only". - The disjointness test now refuses every particle, ambiguous ones too, as the tool does (negative control gains `del`). - The veto's comment says the field is empty by default and the Spanish list ships for sub-phase 3g's es pack. Refs #959 Co-Authored-By: Claude Opus 5.5 --- docs/design/decisions/O6-two-family-names.md | 4 ++-- nameparser/_pipeline/_post_rules.py | 7 ++++--- tests/v2/test_es_given_names.py | 14 +++++++------- 3 files changed, 13 insertions(+), 12 deletions(-) diff --git a/docs/design/decisions/O6-two-family-names.md b/docs/design/decisions/O6-two-family-names.md index e99337631..f09995614 100644 --- a/docs/design/decisions/O6-two-family-names.md +++ b/docs/design/decisions/O6-two-family-names.md @@ -65,7 +65,7 @@ Open: hyphenated surnames block the switch (`Andrés Sánchez-Cid Torres` keeps - 2026-10-29 #754, sub-phase 4d — THE TWO UNITS THIS RULE READS REACH A VIEW: `surname2` holds the one written second, `family` keeping both as before, and the rule marks that unit's words where it takes its word; nothing it reads or declines moved. The view, the mark and their measurement are rules/R10 and decisions/R10's 2026-10-29 entry. - 2026-10-30 #959 — A LISTED GIVEN NAME IS NOT TAKEN: an amendment to rules/O6's Except bullet "What is taken must be one name word a family name could be", not a new rule. In `José Luis Pérez` the word the switch takes is a given name, and that shape is most of what the switch gets wrong (the 2026-10-27 entry: the records it breaks nearly all hold a one-word family box). A Spanish reader settles it by vocabulary: given names are a concentrated set and few words are common in both roles, so a word only a given name is says the name holds one family name. The rule reads `Lexicon.given_names`, empty by default, which a caller fills, and which sub-phase 3g's `es` pack is to fill with the list shipped here; with the list empty the switch reads exactly as before (the snapshot gains the rule's example texts and moves no reading). What it settles, and why each way: - - **Where the list lives: generated, as locales.ZH's surnames are.** `nameparser/locales/_es_given_names.py` holds the list in a block `tools/es_given_names.py` writes, for the `es` pack (sub-phase 3g of #754) to carry as a lexicon fragment, `Lexicon(given_names=GIVEN_NAMES)`. Weighed against: `given` readings in an `es` language file, which per-word TOML would carry a thousand words judged one at a time, where what is decided is a cutoff; the evidence for the set is the tool's docstring and this entry, and a change to it is decided as a cutoff change, as ZH's top 300 is (decisions/W1). A generated list does not pass through the TOML build, so its check 11 does not reach it; that check lets a `given` reading stand beside a title or particle reading and derives the marker the pair means (`prince`, a title with a given reading, compiles to `titles_ambiguous`), refusing only the phase 1 flags (`may_be_name`, `may_be_given = true`) and a particle stating `may_be_given = false` beside a given reading. The list is held to something stricter, a test keeping it disjoint from every word the default lists as a never-given particle, a title (an ambiguous one too), a suffix or a conjunction (tests/v2/test_es_given_names.py, with its recorded negative control), since a word whose own reading another rule weighs is no word this veto should decide; the tool leaves every particle out. + - **Where the list lives: generated, as locales.ZH's surnames are.** `nameparser/locales/_es_given_names.py` holds the list in a block `tools/es_given_names.py` writes, for the `es` pack (sub-phase 3g of #754) to carry as a lexicon fragment, `Lexicon(given_names=GIVEN_NAMES)`. Weighed against: `given` readings in an `es` language file, which per-word TOML would carry a thousand words judged one at a time, where what is decided is a cutoff; the evidence for the set is the tool's docstring and this entry, and a change to it is decided as a cutoff change, as ZH's top 300 is (decisions/W1). A generated list does not pass through the TOML build, so its check 11 does not reach it; that check lets a `given` reading stand beside a title or particle reading and derives the marker the pair means (`prince`, a title with a given reading, compiles to `titles_ambiguous`); what it refuses is the phase 1 flags (`may_be_name`, `may_be_given = true`), a particle reading that does not state `may_be_given = false` exactly where its word has no `given` reading, an agnomen reading on a word with no name reading, and a name reading beside any other suffix-word kind (its full statement is in `tools/vocab/build.py`'s docstring). The list is held to something stricter, a test keeping it disjoint from every word the default lists as a particle, a title (ambiguous ones too, for both), a suffix or a conjunction (tests/v2/test_es_given_names.py, with its recorded negative control), since a word whose own reading another rule weighs is no word this veto should decide, which is what the tool leaves out. - **The source: INE.** The Instituto Nacional de Estadística publishes every given name borne by at least 20 residents of Spain, by sex, and every surname borne as a first surname by at least 20, with its counts as first and second surname, both from the census of 1 January 2025. Its legal notice licenses the site's statistics as "la Creative Commons Reconocimiento 4.0 (CC BY 4.0)" (https://www.ine.es/aviso_legal, read 2026-10-30), asking the reuser to cite "Elaboración propia con datos extraídos del sitio web del INE: www.ine.es" and the date of the data; the shipped module's docstring gives both, with the licence's URI and the files' last update (23 April 2026, the server's `Last-Modified` on both files). A word's given uses are the bearers of the names holding it; its surname uses, the bearers of the surnames holding it, less those holding it behind a particle (`de Miguel`), which the switch never takes as the bare word (with them counted, `Miguel` is 89% given and falls out; without, 92%). A word is a candidate where at least 90% of its uses are given, #959's share; INE writes names without acute accents, so the parse matches a word with its accents and diaeresis dropped and the tilde on n and the cedilla on c kept (`given_name_key`), and a caller's entry also as written. Words common in both roles stay out (`Martín` 10% given, `Vicente` 63%, `Santiago` 57%, `Cruz` 18%). Weighed against: a list built from ORCID's boxes, which #959 measured on a held-out half; INE is licensed for redistribution, counts a whole population rather than researchers, and is not the population scored, so no held-out split is needed. - **One pooled list, Spain's.** INE counts Spain's residents, Latin American-born ones among them, and #959 found a list pooled over six countries within half a point of per-country lists everywhere but Argentina and Peru. Peru is where the pooled list costs most: there it fixes about as many records as it breaks at every cutoff, the breaks being first surnames that are given names in Spain (`Jorge` 12, `Carlos` 11, `Pablo` 6 at the 1,000 cut), where #959's per-country list, learned from Peru's own boxes, broke 58 of the 9,735 records of its held-out half, and its list pooled over six countries 102 of the same; this list breaks 189 of all 19,471. - **The cutoff: 1,000 words** (84.3% of the given-word uses in INE's table). Measured per country with O6 on, the list cut at 100 to all 16,559 candidates; agreement in each of the six double-surname countries is within 0.3 points of the 2,000-word list at 1,000 (Venezuela the 0.3), and a band past it buys at most a tenth of a net fix per added word in those six (1,000 → 2,000: 134 fixed, 35 broken; 2,000 → 5,000: 128 fixed, 36 broken, Chile breaking as many as it fixes), where the first hundred words bought sixteen. Argentina gains 1.6 points more by 2,000 and 2.4 by all, which 3g weighs if it turns the switch on there. @@ -73,7 +73,7 @@ Open: hyphenated surnames block the switch (`Andrés Sánchez-Cid Torres` keeps - **A bare word only.** The veto reads a unit that is one word with no particle; a listed word behind a particle is a surname unit and is taken as before (`Ana de Miguel de la Vega` keeps family `de Miguel de la Vega`), which is what the tool's count assumes in leaving such surnames out. The first draft read the word inside a particle unit too, and the design-docs review found it reading `Ana de Miguel de la Vega` middle `de Miguel`. - **O7.** With both switches on, the word O6 leaves is a middle O7 takes where it stands beside the given name, so `José Luis Pérez` reads given `José Luis`, family `Pérez` (rules/O7's precedence); `Ana María Luis Pérez` keeps middle `Luis`, O7 taking `María`. Whether O7 reads the list itself is decisions/O7's Open question. - **No report**, as for the switch (the 2026-10-24 entry): the caller declared the convention and the list. - - **Latin America, measured in the #961 review.** Of the 27 words Argentina's records most often hold where the veto still lets the switch take a given name (825 records), about 712 records are lost to the 90% bar measured on Spain's population and about 113 to the 1,000 cut: `Martín` (10% given in INE's tables), `Andrés` 77%, `Nicolás` 76%, `Sebastián` 81%, `Esteban` 36%, `Cecilia` 89.75% (84 Argentine records, just under the bar), `Matías` 76%, `Hernán` 63%, `Gastón` 40%, `Santiago` 57% and `Tomás` 61%, the shares recomputed here. Peru is net negative because the switch alone already agrees on 96.0% of its records, leaving little to fix, and its breaks spread over ranks 2 to 452 (`jorge` 33rd, `carlos` 13th), so they come from the selection rule rather than the cutoff. Whether 3g needs a list per region or another selection rule is Open above; the list is not changed here. + - **Latin America, measured in the #961 review.** Of the 27 words commonest among the Argentine records the default reads right and the switch with the list reads wrong, each record keyed by the last word of its given box (836 records), 722 are lost to the 90% bar measured on Spain's population and 114 to the 1,000 cut (a word at or over the bar ranked past 1,000), none to anything else: `Martín` (10% given in INE's tables), `Andrés` 77%, `Nicolás` 76%, `Sebastián` 81%, `Esteban` 36%, `Cecilia` 89.75% (84 Argentine records, just under the bar), `Matías` 76%, `Hernán` 63%, `Gastón` 40%, `Santiago` 57% and `Tomás` 61%, the shares recomputed here. Peru is net negative because the switch alone already agrees on 96.0% of its records, leaving little to fix, and its 189 breaks spread over the whole list, ranks 1 to 994 (`maria` 1st, `jose` 2nd, `chiara` 994th), so they come from the selection rule rather than the cutoff (the records the switch reads right and the switch with the list wrong, each keyed by the word the veto left, the switch's family word missing from the vetoed family, ranked in the list). Whether 3g needs a list per region or another selection rule is Open above; the list is not changed here. Both counts recomputed on the branch at e06b2612 with the review's recipes as stated, over the records of `orcid-researchers` with both boxes, read with `local_agreement.read`, words folded by `given_name_key`. - **Pickling.** The field carries `_ADDED_IN` "3.0.0" and an empty default, so a `Lexicon`, `Parser` or locale pack pickled by 2.x loads with no list. That is what such a pickle meant, no pack filling the field before 3.0, so this field is the case decisions/A1's 2026-10-30 caveat does not reach: the silent drift it warns of comes from a marked field the default vocabulary fills, and the default leaves this one empty. `tests/v2/test_lexicon.py` loads the committed 2.8.0 pickle with the field empty, and refuses it with the marker taken off. Measured 2026-10-30 on the branch, the 2025 Public Data File extract fetched 2026-10-27 (`orcid-researchers`), every record with both boxes filled; all boxes agreeing, by default, with the switch, and with the switch and the 1,000-word list, and the records the list fixes and breaks against the switch alone: diff --git a/nameparser/_pipeline/_post_rules.py b/nameparser/_pipeline/_post_rules.py index 37dcd1a4e..484fdbb94 100644 --- a/nameparser/_pipeline/_post_rules.py +++ b/nameparser/_pipeline/_post_rules.py @@ -299,9 +299,10 @@ def _take_second_family_name(state: ParseState, return if not any(ch.isalpha() for i in take for ch in tokens[i].text): return - # rules/O6: "a listed given name stays where it is" -- the list a - # pack carries (Lexicon.given_names) names a word only a given name - # is, under either order. Only a unit of one token is asked: a word + # rules/O6: "a listed given name stays where it is" -- the caller's + # Lexicon.given_names (empty by default; the Spanish list ships for + # sub-phase 3g's es pack) names a word only a given name is, under + # either order. Only a unit of one token is asked: a word # behind a particle ("de Miguel") is a unit of two, a surname taken # as before. History: decisions/O6 (#959). if len(take) == 1 and is_listed_given_name(tokens[take[0]].text, diff --git a/tests/v2/test_es_given_names.py b/tests/v2/test_es_given_names.py index a53d23ec3..abf9311e1 100644 --- a/tests/v2/test_es_given_names.py +++ b/tests/v2/test_es_given_names.py @@ -3,8 +3,7 @@ The list is generated, not judged word by word, so what a test can hold is what no cutoff may change: no word the default vocabulary lists as a -never-given particle, a title (an ambiguous one too), a suffix or a -conjunction -- stricter than the TOML build's check 11, which a +particle or a title (ambiguous ones too), a suffix or a conjunction -- stricter than the TOML build's check 11, which a generated list does not pass through and which lets a given reading stand beside a title or particle reading, deriving a marker from the pair -- every entry in the fold the parse matches by, and the tool's @@ -50,11 +49,11 @@ def refuse(*args: object, **kwargs: object) -> None: def _non_name_readings(lexicon: Lexicon) -> dict[str, frozenset[str]]: """The default vocabulary's readings of a word as something other - than a name, each folded as the list is: a never-given particle, a + than a name, each folded as the list is: a particle (ambiguous ones + too, `del` and `da` among them, which the tool also leaves out), a title (an ambiguous one too), a suffix and a conjunction.""" - never_given = lexicon.particles - lexicon.particles_ambiguous fields = { - "never-given particle": never_given, + "particle": lexicon.particles, "title": lexicon.titles | lexicon.titles_ambiguous, "suffix": lexicon.suffix_acronyms | lexicon.suffix_words, "conjunction": lexicon.conjunctions, @@ -74,7 +73,7 @@ def _overlap(names: frozenset[str], lexicon: Lexicon) -> dict[str, set[str]]: #: control, recorded so the empty answer cannot be a check reading #: nothing. _UNGUARDED_EFFECT = { - "never-given particle": {"los"}, + "particle": {"los", "del"}, "title": {"dr", "dean"}, "suffix": {"jr"}, "conjunction": {"y"}, @@ -88,7 +87,8 @@ def test_the_list_holds_no_word_the_default_reads_as_no_name() -> None: # and listing it would contradict the reading that claims it. default = Lexicon.default() assert _overlap(_es_given_names.GIVEN_NAMES, default) == {} - planted = _es_given_names.GIVEN_NAMES | {"los", "dr", "dean", "jr", "y"} + planted = _es_given_names.GIVEN_NAMES | {"los", "del", "dr", "dean", + "jr", "y"} assert _overlap(planted, default) == _UNGUARDED_EFFECT From b1d8030214d1863bd4745962baf50eeeb508f007 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Fri, 30 Oct 2026 17:14:45 -0700 Subject: [PATCH 4/4] Keep the given-name veto in rules/O6 and tag its signal as position O6's Signal line names the second signal for what it is, the word standing where the family would take one looked up in the caller's given-name list, tagged position as P1's vocabulary signal is; the ` + ` keeps O6 in the split-candidate grep. decisions/O6 records that the veto stays a clause of O6 though it reads that second signal: it only declines a take O6 would make and has no reading of its own. Refs #959 Co-Authored-By: Claude Opus 5.5 --- docs/design/decisions/O6-two-family-names.md | 2 ++ docs/design/rules/O6-two-family-names.md | 2 +- 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/design/decisions/O6-two-family-names.md b/docs/design/decisions/O6-two-family-names.md index f09995614..dbf94f2c5 100644 --- a/docs/design/decisions/O6-two-family-names.md +++ b/docs/design/decisions/O6-two-family-names.md @@ -11,6 +11,7 @@ - The switch reports nothing — 2026-10-24 #754. - The switch reports no alternative either, so the reading it did not take is not surfaced: the caller declared the convention — 2026-10-27 #752 (Derek). - A listed given name (`Lexicon.given_names`) standing alone is not taken, under every order, and the name then reads as with the switch off; a listed word behind a particle is taken as before, and with the list empty the switch reads as before; no other rule reads the list, the default carries none, and the veto reports nothing — 2026-10-30 #959. +- The veto stays a clause of this rule though it reads a second signal, the taken word looked up in a list, and the rule is a split candidate — 2026-10-30 #959, decided. - The Spanish list, shipped for sub-phase 3g's `es` pack to carry (no pack carries it yet, and a caller may pass a list of its own), is generated, not per-word vocabulary: INE's 2025 census tables, pooled over every name borne in Spain, cut at a measured rank (1,000 words, 90% given use) by `tools/es_given_names.py`; a change to it is a cutoff change — 2026-10-30 #959. Open: hyphenated surnames block the switch (`Andrés Sánchez-Cid Torres` keeps `Sánchez-Cid` a middle name), of the 466 Spanish records the issue's held-out per-country list still read wrong, 271 are this shape (measured on the issue, 2026-10-30) — #959. No rule example shows both switches on over a given name a particle completes (`María del Carmen de la Vega`, given `María del Carmen` under O7 alone and family `del Carmen de la Vega` with O6 on too) — #959. Why the Spanish list does less in Latin America, and whether 3g needs a list per region or another selection rule (the 2026-10-30 #959 entry's Latin America bullet) — #959. A name written with ñ dropped (`Begona`, `Inigo`) matches no entry, the key keeping the tilde; whether to fold ñ for data that strips it — #959. @@ -118,3 +119,4 @@ Open: hyphenated surnames block the switch (`Andrés Sánchez-Cid Torres` keeps The same boxes rendered family box first, read under FAMILY_FIRST on both sides: ES 47.6% → 91.4% → 94.5% (681 fixed, 68 broken), MX 31.4% → 94.6% → 96.3% (465, 132), CO 23.9% → 94.5% → 97.6% (640, 46), PE 12.6% → 95.9% → 95.9% (203, 205), CL 46.3% → 93.4% → 97.0% (729, 38), VE 38.5% → 88.6% → 93.3% (949, 44), AR 82.3% → 59.8% → 90.8% (6,068, 53). PSC (`psc-individuals`, the fetch recorded on #849), a second population the list was not built from, by nationality, default → switch → switch and list, fixed and broken: Spanish 6,089 records 31.8% → 80.2% → 82.2% (150, 31), Mexican 402 25.9% → 80.1% → 82.1% (13, 5), Colombian 377 29.2% → 84.4% → 87.0% (15, 5), Venezuelan 237 40.5% → 75.9% → 80.6% (13, 2), Chilean 151 35.1% → 77.5% → 84.8% (12, 1), Peruvian 116 21.6% → 81.0% → 83.6% (5, 2), Argentine 320 81.9% → 45.6% → 81.9% (121, 5). Recompute: `uv run --with openpyxl --with xlrd python tools/es_given_names.py --top N` writes the list for a cutoff (`--write`) and prints its coverage. For ORCID, `local_agreement.read` per record of `local_corpora.load("orcid-researchers")` with `FIELDED["orcid-researchers"]`, records with both boxes, grouped by `country`, under `Parser()`, `Parser(policy=Policy(two_family_names=True))` and that policy with `lexicon=Lexicon.default().add(given_names=...)` holding the tool's first N words, each with `_title_words` of its own lexicon; family-first, the record's `name` replaced by `family + " " + given` and each policy given `name_order=FAMILY_FIRST`. For PSC, `local_agreement.score(rows, FIELDED["psc-individuals"], parser, jobs=1)` over the records whose `group_of` is the nationality, each cell a group's `agreed` over its `scored`. A record is fixed where it agrees only with the list, broken where it agrees only without it. The comparator is the switch alone on the same tree; `local_agreement.py orcid-researchers --base` and `psc-individuals --base` against master match the base, the default reading no list. +- 2026-10-30 #959, decided — THE VETO STAYS A CLAUSE OF O6, THOUGH IT READS A SECOND SIGNAL. docs/design/AGENTS.md ("One rule per piece of evidence") says a clause reading a signal its rule did not already read is a new rule, and the veto reads one O6 did not: the word standing where the family would take one, looked up in the caller's given-name list (the #961 review). Kept in O6 anyway, decided 2026-10-30: the veto is one more member of the class the Except bullet "What is taken must be one name word a family name could be" already holds, beside the initial written alone and the mark with no letter in it. It only declines a take O6 would otherwise make, and reads nothing of its own: with the switch off it is never asked, and where it declines the name reads as with the switch off. A rule of its own would state no reading but O6's absence. The Signal line names the lookup as a second signal joined with ` + `, tagged `position` as P1's vocabulary signal is, by where the word stands, so O6 shows in the split-candidate grep; the next signal O6 gains is the point to split it. diff --git a/docs/design/rules/O6-two-family-names.md b/docs/design/rules/O6-two-family-names.md index b5c0fe798..9b3ab6d9d 100644 --- a/docs/design/rules/O6-two-family-names.md +++ b/docs/design/rules/O6-two-family-names.md @@ -129,5 +129,5 @@ - The rule reports nothing: the caller declared the convention, as a declared order is followed without a report, and a report on most names of three words or more would teach the caller to ignore the field. --- -**Signal:** none in the name + shape: a caller switch, the family's words and the taken word's shape read as written to see whether the family already holds two; and the taken word as written, against the caller's given-name list +**Signal:** none in the name + position: a caller switch, the family's words and the taken word's shape read as written to see whether the family already holds two; and the word standing where the family would take one, looked up in the caller's given-name list, a vocabulary signal tagged by where the word stands as P1's is **history:** decisions/O6 · **implemented:** nameparser/_pipeline/_post_rules.py