diff --git a/AGENTS.md b/AGENTS.md index 28a682047..773953914 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -225,6 +225,8 @@ The library has two layers: data — the per-word TOML in `nameparser/vocab/`, c - A listed CJK honorific glued to the end of a name splits off, but only an entry that can never end a name (#308): the same not-a-guess standard applied to WORDS rather than a script, the vocabulary carrying the license itself, so neither opt-out reaches it either. Its reach across a family comma is tolerated rather than contract — rules/W2, rules/W3, decisions/W2. - Han segmentation stays OPT-IN (`locales.ZH`; `locales.JA` with a `Parser(segmenter=...)`): a Chinese surname list corrupts Japanese kanji names (高 is a common Chinese surname, so 高橋一郎 would split 高+橋一郎 where the correct reading is 高橋+一郎), and no surname list divides a kanji name at all — rules/W1. +The group suggester (`Parser.suggest`) is outside this rule, as proposed for review in decisions/group-suggestion's 2026-11-10 SCOPE entry: the parser never detects language; the group suggester may, over a group the caller declares, says what it saw, and its answer is configuration the caller applies. No parse changes, and a single name is still read positionally. + Latin-script input is never touched by any of this: "Kim Min-jun" is genuinely order-ambiguous and stays governed by `name_order` and opt-in packs. Before adding a script-conditional rule, work out which of the three it is — certain, certain for this one behavior only, or a statistical guess wearing a script's clothes. **A constant's membership is a question you may reopen.** Proposing that a word be ADDED, REMOVED or MOVED between vocabulary sets is ordinary design work — a shipped entry is not evidence that anyone judged it. `SUFFIX_ACRONYMS` arrived in `af5bdab` as a bulk Wikipedia import never reviewed against surname collisions: 572 of its 577 alphabetic entries leave `family` empty in `"John "` against five ambiguous-gated exceptions (recomputed 2026-09-07; the fifth is `ba`; 568 of 575 against seven, measured 2026-09-25 after #540), and `sa`, `se` and `om` are borne as surnames (measured 2026-08-23; all three since decided, `sa` removed from the vocabulary on 2026-10-25, #857), as were `rai`, `cha`, `ba` and `mc` before `decisions/suffix-acronym-collisions` decided all four — `rai` and `cha` removed, `ba` marked ambiguous, `mc` left alone as no borne name at all. The same entry marked `meng` and `lac` ambiguous on 2026-09-25 (#540). When a fix starts to look like new machinery, check the vocabulary first. Criterion: `decisions/vocabulary-collisions`, with #360's positional qualifier. @@ -257,14 +259,16 @@ The compiled sets, read by `Lexicon` (mostly as the `_sets` field of the same na - listed spellings — `SAME_AS`, private, read by `Lexicon` as `_same_as` and nowhere in a parse: every `same_as` keyed by the group of its reading's kind (`_reading_index.SAME_AS_GROUPS`: `given` for a given name or a bound given name, `family` for a surname or a particle, `title`, `suffix`), each word with every full form it carries there (`sr` among titles is both `señor` and `senhor`). The name match (`_match.py`) reads each role in its group through `Lexicon._headwords`, every word reachable through the group's links, so a given name's spelling never matches a surname (`Ma, Jack` / `María, Jack` conflict) and a loop is simply all its words (decisions/name-matching, 2026-11-08); and `SAME_AS_NAMED`, per group, the words of it whose name reading's `same_as` stands beside a name reading of the same kind with none, a name of its own too (`jo`), whose links `_headwords` follows only where `_match._spellings` read the word with its trailing period (the 2026-11-08 #1006 follow-up) - familiar forms — `FAMILIAR`, private, read by `Lexicon` as `_familiar` (through `Lexicon._familiar_forms`) and nowhere in a parse: each `familiar` reading's `short_for`, the longer given names a short one stands for (`max` -> `maksim`, `maxim`, `maximilian`), every file's together, which `Lexicon` reads both ways (a pair matches whichever way its source wrote it, never chained) and the name match reads as the opt-in `familiar-form`. `familiar` is a category of its own, never a `given` reading, which would move parses where the word is also a title, particle or suffix; a choice neither prefers nor drops by it (`_reading_index.UNPREFERRED`), and `tools/vocab/familiar.py` writes the shipped ones from nam_dict and from the nicknames list, less the pairs `tools/vocab/nicknames_excluded.tsv` refuses (decisions/name-matching, 2026-11-08 FAMILIAR FORMS and 2026-11-09 THE NICKNAMES LIST) +- the group suggester's tables — `config/_order_names.py`, private, written by `tools/vocab/order_names.py` (`--write`; its hand-kept spelling lists sit above the generated blocks) and read by `_suggest` only, never by a parse or a `Lexicon`: `ROMANIZED_SURNAMES` (zh, ko and vi surnames folded by `_suggest.fold`) and `COMMON_GIVEN_NAMES`. They are not `Lexicon` kinds, so a language choice, `strict=True` and a caller's edit do not reach them and no snapshot moves with them (decisions/group-suggestion, 2026-11-10 TABLES) + `config/__init__.py` holds a docstring only: the package is the home of the data modules, and since 3.0 nothing else (#357), all of them private (#1007; `regexes` went with #1008; `surnames` became `_ko_surnames` in #1077, named for `vocab/ko.toml`'s tag, and stays in `config/` because the default lexicon reads it and `_lexicon` may not import `locales/`, a layer above it). ## 2.0 API modules (`nameparser/_*.py`, `tests/v2/`) The 2.0 rewrite landed as underscore-private modules beside the v1 code, which 2.0 and then 3.0 deleted. These conventions apply to all of it. The full design record (rationale, settled-decision logs, dated amendments) lives in untracked `docs/superpowers/specs/`; this section is the enforceable subset. **A commit that establishes or amends one of these conventions must update this section in the same commit** — grep-driven staleness sweeps miss paraphrased prose (see Workflow above), so write-time maintenance is the mechanism, audits are the backstop. -- **Module layout**: every module is underscore-private (`_types.py`, `_lexicon.py`, `_policy.py`, `_locale.py`, `_render.py`, `_reading_index.py`, `_pipeline/`, `_parser.py`, and `_match.py`, which holds name matching: `Matching`, the settings object `Parser.matching()` builds, and its result types). The public import surface is exactly `nameparser` and `nameparser.locales` (the vocabulary's compiled sets are private since 3.0, #1007, the Korean surname list since #1077 (2.8.2 warning of it, #1076), and `regexes` went with #1008; `tests/v2/test_layering.py` refuses a public module in `nameparser/config/`); `nameparser/__init__.py` holds re-exports and `__all__` only — no logic. -- **Layering is enforced by `tests/v2/test_layering.py`** (exact-module matching; `if TYPE_CHECKING:` imports don't count): `_types` imports nothing internal at module level — its rendering delegates import `_render`, and `ParsedName.matches` `_parser`, at call time; `_lexicon` and `_policy` sit above `_types` independently (`_lexicon` may import `nameparser.config.*` DATA modules, e.g. `config/_ko_surnames.py` and `config/_compiled.py`, and `_reading_index` at call time only, to compile a configured lexicon's readings, since `tools/vocab/build.py` loads it by path, 3.0's 4i-b); `_reading_index` imports nothing internal at module level and the index, `config/_index.py`, at call time, since `tools/vocab/build.py` loads it by path; `_locale` sits on `_lexicon`+`_policy` (plus `_types` for the shared pickle mixin, and `config/_rules.py` at call time, in `rules_patch`); `_render` imports `_types`, `_lexicon` (for `Lexicon.default()` and `_normalize`) and `_policy` (for the scripts that write no initials); `_match` imports `_types`, `_policy` (for the same scripts) and `_lexicon` (for `_normalize`, which tells a hyphen half that is a listed particle by `lexicon._sets.particles`), and `Lexicon` and `Parser` under `TYPE_CHECKING` only; `_pipeline/*` imports `_types`+`_lexicon`+`_policy` plus in-package `_pipeline` helpers, `_assemble` also `_reading_index`, for the kinds `Reading.languages` matches (`CATEGORY`; the files themselves it reads through `Lexicon._files`, 4i-d; `_vocab` imported it in 3.0's 4b and 4c for a listed suffix word's kind and a patronymic particle's, which since 4i-b the lexicon compiles into `_sets`); `_parser` sits on everything except `_render`, `_match` at call time only, in `Parser.matching()`, and `_reading_index` at call time only, to validate a choice's tags and move the case masks it moves (the choice itself compiles in `_lexicon._built`); `__main__` imports only the public package; locale pack modules (`locales/*.py`) import `_locale`/`_lexicon`/`_policy`/`_types` only (`_types` joined the list in #272 — `locales/ja.py`'s segmenter factory constructs a `Segmentation`), and the `locales/__init__` additionally lazy-imports its packs (PEP 562). Extend the test's `ALLOWED` table when adding a module. +- **Module layout**: every module is underscore-private (`_types.py`, `_lexicon.py`, `_policy.py`, `_locale.py`, `_render.py`, `_reading_index.py`, `_pipeline/`, `_parser.py`, and `_match.py`, which holds name matching: `Matching`, the settings object `Parser.matching()` builds, and its result types, and `_suggest.py`, the group suggester: `Parser.suggest` tallies one vote per name from per-name signals and returns a `Suggestion`, decided under the thresholds in the decisions file group-suggestion). The public import surface is exactly `nameparser` and `nameparser.locales` (the vocabulary's compiled sets are private since 3.0, #1007, the Korean surname list since #1077 (2.8.2 warning of it, #1076), and `regexes` went with #1008; `tests/v2/test_layering.py` refuses a public module in `nameparser/config/`); `nameparser/__init__.py` holds re-exports and `__all__` only — no logic. +- **Layering is enforced by `tests/v2/test_layering.py`** (exact-module matching; `if TYPE_CHECKING:` imports don't count): `_types` imports nothing internal at module level — its rendering delegates import `_render`, and `ParsedName.matches` `_parser`, at call time; `_lexicon` and `_policy` sit above `_types` independently (`_lexicon` may import `nameparser.config.*` DATA modules, e.g. `config/_ko_surnames.py` and `config/_compiled.py`, and `_reading_index` at call time only, to compile a configured lexicon's readings, since `tools/vocab/build.py` loads it by path, 3.0's 4i-b); `_reading_index` imports nothing internal at module level and the index, `config/_index.py`, at call time, since `tools/vocab/build.py` loads it by path; `_locale` sits on `_lexicon`+`_policy` (plus `_types` for the shared pickle mixin, and `config/_rules.py` at call time, in `rules_patch`); `_render` imports `_types`, `_lexicon` (for `Lexicon.default()` and `_normalize`) and `_policy` (for the scripts that write no initials); `_match` imports `_types`, `_policy` (for the same scripts) and `_lexicon` (for `_normalize`, which tells a hyphen half that is a listed particle by `lexicon._sets.particles`), and `Lexicon` and `Parser` under `TYPE_CHECKING` only; `_suggest` imports `_types`, `_policy`, `_lexicon` (the particles, the titles and `_normalize`), the pipeline's predicates (`_pipeline/_vocab`'s patronymic, title, suffix and numeral tests, `_pipeline/_state`'s commas) and `config/_order_names.py`, imports `_parser` at call time, in `Suggestion.__post_init__`, to check that the base is a `Parser`, and names `Parser` under `TYPE_CHECKING` for its annotations; `_pipeline/*` imports `_types`+`_lexicon`+`_policy` plus in-package `_pipeline` helpers, `_assemble` also `_reading_index`, for the kinds `Reading.languages` matches (`CATEGORY`; the files themselves it reads through `Lexicon._files`, 4i-d; `_vocab` imported it in 3.0's 4b and 4c for a listed suffix word's kind and a patronymic particle's, which since 4i-b the lexicon compiles into `_sets`); `_parser` sits on everything except `_render`, `_match` at call time only, in `Parser.matching()`, `_suggest` at call time only, in `Parser.suggest()`, and `_reading_index` at call time only, to validate a choice's tags and move the case masks it moves (the choice itself compiles in `_lexicon._built`); `__main__` imports only the public package; locale pack modules (`locales/*.py`) import `_locale`/`_lexicon`/`_policy`/`_types` only (`_types` joined the list in #272 — `locales/ja.py`'s segmenter factory constructs a `Segmentation`), and the `locales/__init__` additionally lazy-imports its packs (PEP 562). Extend the test's `ALLOWED` table when adding a module. - **Canonical field order** — the seven roles in `Role` enum declaration order, defined once and derived everywhere (properties, `as_dict`, reprs, `comparison_key`). Never restate the order literally. `Role` is a `StrEnum`: members compare as their field-name strings, and `tokens_for()` coerces strings via `_coerce_enum`. - **Method organization**, fixed section order in every class: fields + `__post_init__` validation → alternative constructors → dunders (construction/equality → protocol → operators) → properties → public methods by concern (access → editing → comparison → rendering delegates) → private helpers last, except a helper serving exactly one section may sit at that section's head. - **Validation is eager and fail-loud**: every `raise` states the offending value, the expected form, and the fix. Exception taxonomy: wrong type — including wrong element type inside a collection, bare `str` where an iterable of strings is expected, or a `Mapping` where a plain iterable is expected — raises `TypeError`; well-typed but unacceptable values raise `ValueError`; failed enum lookups stay `ValueError` for any input (stdlib `EnumType` precedent). **When the message hands the reader code to paste, that code has to survive a type checker, and then RUN** — nameparser ships `py.typed`, and a placeholder can type-check and still raise: the 2.4 release sweep's first fix to the `capitalization_exceptions` hint offered `('key', 'Value')`, which passes mypy and fails #459's mask check, so `test_add_capitalization_exceptions_raises_pointing_at_replace` now executes the offered code. #337's segmenterless warning offered `Policy(segment_scripts=())`, an `arg-type` error, because these fields are annotated with what they STORE rather than everything the constructor accepts. Prefer the `frozenset()` / `()` spellings in messages and docstrings, and pin the offered spelling in a test — the warning tests matched on `ja_segmenter` and never checked the actionable half of the message. **A warning emitted in `Parser.__post_init__` needs `parser_for` to re-emit it from its own frame** (the `catch_warnings(record=True)` block at its return): `__post_init__`'s `stacklevel` is sized for direct `Parser(...)` construction, and through `parser_for`'s extra frame the default one-line rendering attributes the warning to the library's own `return Parser(...)` — the exact call the message tells the user to change becomes invisible. No single stacklevel serves both entry points; a new construction warning gets the re-emission for free, but a new CONSTRUCTION SITE for `Parser` inside this package needs its own re-emission or its callers get library-attributed warnings (#337 review). @@ -296,6 +300,8 @@ The 2.0 rewrite landed as underscore-private modules beside the v1 code, which 2 **Measuring a change to name matching: use `tools/differential/match_diff.py --base `.** A change to `nameparser/_match.py`, or to the vocabulary only the name match reads (`same_as`, `familiar`), moves no parse, so the snapshot and every corpus check below report nothing; `match_diff.py` lists every name pair whose `check` result or keys moved, under twenty-one settings, and every consistent pair sharing no key (#1278). A setting naming an outcome the base does not know subtracts nothing there, so `-joined` against a base older than #1261 compares a caller who drops `joined` with the old default, which is the claim that caller keeps what they accepted. The base is the branch's merge base, re-pointed for each branch: a scratch base kept from the previous PR measures that PR's moves too, and #1261's first figures came from #1262's base, found by review. #1221, #1267 and #1260 each wrote a throwaway oracle for this first (tools/differential/README.md, "Measuring a change"). +**Measuring a change to the group suggester: use `tools/differential/suggest_measure.py`.** Run it after any change to `nameparser/_suggest.py`, to `nameparser/config/_order_names.py` (or `tools/vocab/order_names.py`, which writes it) or to a predicate a signal reads (the capitals, particle and patronymic tests, and the title, suffix and numeral tests a name is read past): `suggest` moves no parse, so the snapshot and every corpus check report nothing. It draws groups of known order, 2, 3, 4, 5, 10, 20, 50 and 200 names by default, from the local corpora and reports the share of groups read right by size, the same-order rows being the anti-goal (at least 95% at every size; groups of 2 to 4 are where #1308's final review found a patronymic rule forcing given-first on a family-first base); `--set NAME=VALUE` overrides a threshold, and `--set _MIN_VOTES=1000000 --set _MIN_PATRONYMIC_VOTES=1000000` decides nothing and is the harness proof (the order count alone leaves the patronymic rule free to force an order). Its figures are quoted in `Parser.suggest`'s docstring, docs/usage.rst and docs/design/decisions/group-suggestion.md; re-take all three when they move (decisions/group-suggestion's THRESHOLDS entry names the command and holds the table the other two round). A signal that reads one of the pipeline's predicates must read it exactly where the pipeline's rule does, no wider and no narrower, and its tests need a row at each edge: #1310's patronymic signal first voted an `-ovich` ending on any word (wider: two-word `Abramovich Roman` decided East Slavic and forced given-first), then its fix refused one on the first word, which rules/O1 never asks about (narrower: `Abramovich Roman Arkadyevich`, which the rule reads right, cast nothing). Only the second review caught the narrowing. + **Validating a new parsing rule** — before implementing, simulate it in a throwaway script against `TEST_NAMES` (plus a few target-language examples) to catch regressions/false-positives early. E.g. this surfaced the `last_base` `do`/`st`/`mc` empties and the patronymic `"David Michael Abramovich"` false-reorder. After the change, `tests/v2/test_snapshot.py` lists every corpus name it moved, grouped by changed field -- read them, and re-record with `uv run python tools/differential/snapshot.py --record` once every move is intended, so the PR diff of `snapshot.jsonl` shows the reviewer the same list. `uv run python tools/differential/cldr_agreement.py` reports how the tree's agreement with CLDR's stated name fields moved; `tests/v2/test_cldr_agreement.py` fails on any move, and `tools/differential/cldr_agreement.json` doubles as a measured list of names the parser and CLDR read differently. The same command and test score the GND corpus's real names (`gnd_fields.jsonl`, record `gnd_agreement.json`) by the same rules, and `--record` rewrites every record, printing per source what it accepts. A locale-pack change reads the second view the same command prints, per pack (`cldr_agreement_locales.json`, #672, and `wikidata_native_agreement_locales.json` for real Han and kana names); its `JA` rows run only with the `ja` extra installed. A change to Han, Hangul or kana order or to Korean segmentation reads `wikidata_native_agreement.json` too: the only real names in those scripts the corpora hold. It holds no honorific, so a change to the honorific peel moves nothing there. A change to comma or credential handling also reads the local corpus of hand-typed names (`snapshot.py --local ctgov-officials --base `, which reads the base from a baseline kept for its package's files and never the shared snapshot beside the corpus (a base read once from the same fetch is copied rather than parsed again; `--sample N` reads a fixed sample while iterating; the shared snapshot beside the corpus is one file per source for every worktree, and #752's phase 2f found ctgov reporting a sibling branch's 1,334 joined-credential readings as its own moves against it); tools/differential/README.md's "Local corpora"): the one population of names nobody chose for a test or a catalog, so it shows how often each shape is typed, and the one no CI run sees. A change to title handling reads `ukec-individuals` the same way: UK donors' names in spoken order and mixed case, in 8,219 of whose 15,214 `parse()` read a title on 2026-10-22. A change to English titles, honorifics or post-nominals also reads `snapshot.py --local govuk-people` the same way: GOV.UK's display names of ministers, officials and officers, the title run and the post-nominals as an office writes them (`The Rt Hon`, `Sir`, `Baroness`, ranks; `KC`, `CBE`, `KCB`), in 2,093 of whose 6,021 `parse()` read one or both on 2026-10-30. A change to Spanish titles, to the `es` pack, or to how a title marks where a name starts after an office reads `snapshot.py --local dof-signatures` the same way: the names in the signature blocks of Mexico's Diario Oficial for 2025, cut from the clause that names the signatory's office, 65.4% of its 33,565 signatures opening with a title (`Lic.`, `Dr.`, `Mtro.`, `C.`, `Licenciada`, `C.P.`), whose first word `parse()` read as a title in 82.1% and `parser_for(locales.ES)` in 86.8% on 2026-10-31. A change to particles or generational words reads `snapshot.py --local tse-donors` the same way, Brazilian names in spoken order dense with both. A change to initials, to a title word outside English, or to how a script's combining marks are read reads `snapshot.py --local nepal-candidates` the same way: Nepal's local election candidates, each a romanized name (surnames written as initials, `B.K.`, `K.C.`; honorific titles such as `Guru` in front, `Shekh`, a title there until #752's phase 2j made it one only inside a title phrase, and `Md.`, which its period makes a title by shape) beside the same name in Devanagari, written with vowel signs and the virama, combining marks the letter-shape tests do not count as letters (decisions/nfd-shapes), and with the joiners ZWJ and ZWNJ. It is the only large population of South Asian names the corpora hold, and its corpus reads both scripts. A change to non-Latin scripts or to Arabic-script names reads `snapshot.py --local jrc-names` the same way, and `local_corpora.py groups jrc-names`: the spellings of one person across scripts and transliterations, harvested from news text, with 17,074 Arabic-script spellings where the committed corpora hold 315 distinct names with an Arabic letter, 96 of them CLDR's fictional renderings (2026-10-24), and under a licence that keeps even a reading of one of its names in the cache. A change to patronymics, to Turkic or East Slavic names, to Indic titles and initials, or to Slavic or Portuguese double surnames reads `snapshot.py --local opensanctions-parliaments --base ` the same way, and `local_corpora.py groups opensanctions-parliaments`: the members of twenty-two national chambers as each parliament writes them (Azerbaijan, the Central Asian chambers, Turkey, Croatia, Slovenia, Slovakia, Portugal, India, Ukraine, Belarus, Russia, Indonesia, Ethiopia, South Korea, Israel) and Indonesia's 2018 regional heads, read through OpenSanctions, 24,084 spellings of 19,216 members on 2026-11-08, some members' aliases in another script (Kazakh beside Russian Cyrillic, Uzbek Cyrillic beside Latin, Devanagari beside Latin, Hangul and Hanja or Hebrew beside Latin) or with a title (`Shri`, `Smt.`), the population issues #1186 to #1192 were read from, and the one that measures Indonesia's `H.` and `Imam` (#1205, #1206) Ethiopia's father's name (#1203), Korea's Hangul and Hanja (#1213, #1215, #1216) and Israel's Hebrew (#1209 to #1211); `config_diff.py --group ` keeps one parliament. A change to which field a word lands in also reads `local_agreement.py psc-individuals --base ` the same way: real names with the four boxes their filers typed them into (UK companies' people with significant control), scored field by field and by nationality; and `local_agreement.py orcid-researchers --base `, ORCID holders' own two boxes (given names, family name) from every country, scored by the country they work or live in -- a population each person split for themselves, academic but worldwide, the one to read for a change a language pack makes; and `local_agreement.py patentsview-inventors --base `, US patents' inventors in the two boxes their filings give, split by a professional filer, with how many people filed under each name in each country, the one to read for how common a name's word is among a country's US-patent inventors. A change to titles or post-nominals, or to the `cs` pack, reads `local_agreement.py cz-candidates --base `, and with `--locale cs` too: Czech election candidates' four boxes (the title before the name, given names, surname, the title after it), scored by election, the only population whose titles are stated apart from the name, 36% of its 170,181 records titled (2026-10-30), nearly all academic degrees; its names are composed from the boxes (`Ing. Jan Novák, Ph.D.`), so it says nothing about how a writer spaces or punctuates them. A change to family-first order, to Vietnamese names or to their titles reads `local_agreement.py uit-vinames --base ` and `local_agreement.py vn-assembly --base `, and with `--locale vi`, `locales.VI` being that pack (#1108): 21,090 distinct names and 499 National Assembly deputies written family name first with their diacritics, whose parts no one filed but the convention derives (first word the family, last the given, the words between the middle name), so they score middle names, which ORCID's two boxes cannot, and `vn-assembly` the `GS.`/`PGS.`/`TS.`/`ThS.` titles composed from the education each deputy states. These are the only real fielded populations outside the committed fields files, and no CI run sees them either. ## Gotchas diff --git a/docs/design/decisions/README.md b/docs/design/decisions/README.md index 1fc72e5b0..cd8b4e232 100644 --- a/docs/design/decisions/README.md +++ b/docs/design/decisions/README.md @@ -159,6 +159,7 @@ Entry conventions: - [deviates-registry — packs stay pure data (option C)](deviates-registry.md) - [differential-ledger — the differential ledger's arcs](differential-ledger.md) - [generational-words — junior and senior in other languages (2026-10-14, #661)](generational-words.md) +- [group-suggestion — settings suggested by a group of names](group-suggestion.md) - [indic-honorifics — the renunciate class and the Indic honorific vocabulary (2026-09-06, #346/#344/#343)](indic-honorifics.md) - [initials-repertoire — is_initial across scripts (#320)](initials-repertoire.md) - [ma-do — ambiguous acronyms by decision](ma-do.md) diff --git a/docs/design/decisions/group-suggestion.md b/docs/design/decisions/group-suggestion.md new file mode 100644 index 000000000..02ee8c4e0 --- /dev/null +++ b/docs/design/decisions/group-suggestion.md @@ -0,0 +1,93 @@ +# group-suggestion — settings suggested by a group of names + +## Standing + +- Proposed, for review: the parser never detects language; the group suggester (`Parser.suggest`) may, over a group the caller declares, says what it saw, and its answer is configuration the caller applies; the parse itself is untouched by it — 2026-11-10 SCOPE. +- The two word tables the suggester reads (romanized zh, ko and vi surnames; common given names) are private compiled tables for the first version, not `Lexicon` kinds: no parse reads them, a language choice or `strict=True` does not filter them and a caller cannot edit them — 2026-11-10 TABLES. +- The derived given-name table from nam_dict is approved (the table is derived from nam_dict, GNU FDL 1.2, already the source of the shipped familiar forms) — 2026-11-10 TABLES. +- Proposed, for review: the given-name table's size and filter (the top 500 per country, a family share of at most 5% over at least 20 uses) and the hand-kept surname lists — 2026-11-10 TABLES. +- Proposed, for review: the evidence declined (the vocabulary's name readings, `Reading.languages`, the familiar forms) — 2026-11-10 DECLINED. +- Proposed, for review: a comma name states its own order and does not vote on order — 2026-11-10 SIGNALS. +- Proposed, for review: an East Slavic group gets the patronymic rule with a given-first order, since the rule reads either written order one name at a time and stands down under a declared family-first order, unless the names' own order votes decide family-first, which the rule never overrides — 2026-11-10 SIGNALS, 2026-11-10 FIX WAVE. +- Proposed, for review: a patronymic votes only where the pipeline's rule reads one (rules/O1's three name words, rules/O2's four), a name is read past its listed titles and post-nominals, and a comma followed by post-nominals alone reads the name before it — 2026-11-10 FIX WAVE. +- Proposed, for review: an order is decided by at least two names and more than two thirds of the names casting an order vote; a patronymic rule by at least five names carrying it and more than four in five of the names casting any vote; otherwise the base stands — 2026-11-10 THRESHOLDS. + +Open: #1308 (the tracking issue), whose open items are: Turkic and East Slavic patronymics mixed in one group (uz_senate, kz_senate); the given-first order a decided East Slavic rule forces on a family-first group; the given names the given-name table's family-share filter drops (`John`, `David`, `James`); the family-in-capitals setting, waiting on #659. + +## Log + +- **2026-11-10 SCOPE — THE NEVER-DETECT-LANGUAGE RULE IS ABOUT THE PARSE, AND THE GROUP SUGGESTER IS OUTSIDE IT (PROPOSED, FOR REVIEW).** AGENTS.md's design philosophy says the parser never detects language, because the language of a Latin-script name cannot be read off the name. `Parser.suggest` works on a group the caller declares to share one convention, reads statistics over the group and returns configuration with the evidence behind it, which the caller applies; it never mutates a base `Parser` or a default and changes no parse. Proposed wording: "the parser never detects language; the group suggester may, says what it saw, and its answer is configuration the caller applies". The given-name table does not ask what language a name is; it asks whether a word is almost always a given name, anywhere. This entry is the first commit of the branch so the wording can be settled before the code is read; the only items recorded as decided are the two under TABLES (the private tables, the nam_dict approval); the rest of this file is proposed, for review. +- **2026-11-10 TABLES — TWO PRIVATE COMPILED TABLES, READ BY THE SUGGESTER ONLY (THE PRIVATE TABLES AND THE NAM_DICT DERIVATION DECIDED; THE REST PROPOSED, FOR REVIEW).** The romanized surnames (zh, ko, vi) and the common given names are written by tools under `tools/vocab/`, like `familiar.py`, into committed private modules under `nameparser/config/`, with a test that each is current. They are not `Lexicon` kinds for the first version, because a `family` reading on `Lee`, `Song` or `Long` would move the suffix gates derived from name readings and a `given` reading would move titles and particles; a table no parse reads moves no snapshot. Making them `Lexicon` kinds later, if a caller needs to tune them, stays open. The given-name table is derived from nam_dict's 55 country columns (top N per country by frequency band, unioned), keeping a name only if ORCID's researchers wrote it in the family box at most 5% of the times they wrote it (at least 20 uses), it is longer than one letter and it is not in the romanized surname sets. N (500) was chosen with the spikes (SIGNALS, top 100 against top 500); the family share and the use count were set once and not tuned, the THRESHOLDS measurement finding that the tables meet its constraint as built. Licences: nam_dict GNU FDL 1.2, ORCID's public data file CC0, Unihan the Unicode licence; the module header carries the sources once. The table derived from nam_dict is approved. +- **2026-11-10 SIGNALS — WHAT EACH SIGNAL VOTES, AS MEASURED WITH THE THROWAWAY SPIKES (PROPOSED, FOR REVIEW).** Measured with the throwaway spikes on 2026-11-10. The spikes counted votes per SIGNAL (a name read by two signals cast two votes, and its group decided on the signal votes), where `suggest` counts one order vote per NAME, so their group figures below are not comparable with suggest's; the per-name re-take, with its command, is 2026-11-10 THRESHOLDS. Populations are PatentsView inventors by country, UK company filers (PSC) by nationality, Czech candidates, Vietnamese names and national parliament lists (OpenSanctions); none of them built the given-name list. + - Capitals word position: decisive where present (Turkey's parliament `Mehmet DEMİR` 592:0); a capitals run in the middle (Hong Kong `Jeffrey LAM Kin-fung`) does not vote. + - Comma names do not vote on order: 373 comma names outvoted the comma-less majority of India's Lok Sabha list. + - Trailing initial (`Smith J.`) votes family-first; a leading initial does not, having voted 76 wrong of 112 in given-first Indian groups (`S. Ramesh`). + - Romanized zh, ko and vi surnames: groups of CN, TW, KR and VI names 99-100% right from 5 names in both orders, about 97% of names voting right; one-letter words are excluded, the spike's set having swallowed initials (`A.` folds to pinyin `a`). + - Common given names, per name where it votes: 98-99% right in most groups (Czech, Polish, British, Spanish, German, Italian, Japanese, Russian, US) and 90-95% in Indian, Nigerian and Romanian ones. Family-first groups' share right, before and with the top-500 list: Japanese (PatentsView) 12% to 88% at 5 names and 55% to 100% at 50; Turkish (PSC) 13% to 78% and 66% to 100%; Indian (PSC) 10% to 35% and 31% to 87%; Polish, Spanish, Italian and Czech 71-83% to 96-100% at 5 names and 100% at 50. Given-first groups stay at 95-100%. Top 100 per country gives 2,375 names, top 500 gives 4,267 (the spike's count; the shipped table held 4,271 before the fold wave and holds 4,262 after it, 2026-11-10 FIX WAVE). + - Patronymic presence: the spike's position vote got Ukraine's Rada 1,582:0 right on order, but order is the wrong answer for an East Slavic group. Measured on the tree (master 01b982e3 plus #1304): with `{EAST_SLAVIC}` under the default order `Sidorov Ivan Petrovich` and `Ivan Petrovich Sidorov` both read given Ivan, family Sidorov, patronymic Petrovich, and under `FAMILY_FIRST` the given-first `Ivan Petrovich Sidorov` reads given Petrovich, family Ivan with no patronymic. So an East Slavic group gets the rule alone (what `locales.RU` ships; superseded the same day: a decided East Slavic rule brings a given-first order with it, the Standing line, since the rule stands down under a family-first base) and the patronymic does not vote on order; the Turkic combination is measured on the Azerbaijani, Kazakh, Uzbek and Turkmen lists before it is picked, and stays out of the first version if none reads them right. +- **2026-11-10 DECLINED — EVIDENCE THE SUGGESTER DOES NOT USE (PROPOSED, FOR REVIEW).** + - Declined: the vocabulary's `given` and `family` readings as order evidence. Measured with the throwaway spikes, they vote wrong about as often as right and mostly wrong in family-first groups (Dutch 92 right, 496 wrong): they are collision words, chosen because they clash with a title or particle, not common names. + - Declined: `Reading.languages` as evidence for suggesting a language pack. English and the Spanish given-name list top nearly every group, Cyrillic groups carry no tags and Portugal's parliament ranks the Spanish list over `pt`. A pack could later follow from settings evidence; it is out of the first version. + - Declined, superseded: the familiar forms. They were useful (Western family-first groups 93-100% from 10 names once the romanized surnames were subtracted, `Kim`, `Lee` and `Chan` being listed familiar forms), but the common-given-names list alone does as well or better, and it keeps the parse's `familiar` vocabulary out of this. +- **2026-11-10 TURKIC — THE RULE ALONE, NO ORDER OF ITS OWN (proposed, for review).** Measured on the OpenSanctions parliament lists, read from `~/.cache/nameparser-tools/local/opensanctions-parliaments/raw.jsonl` line by line, keeping the rows whose `form` is `"name"` and whose `script` is `Latin` or `Cyrillic` (the dataset keys named below). The `script` field also holds `mixed` on four rows (one in `uz_legislative_chamber`'s `Firdavs Qobiljonovich Qurbоnov`, one more there, one in `uz_senate`, one in `kz_senate`): a name written with a Cyrillic letter inside a Latin word, a lookalike typo, which no real reader sees as another script, so they are dropped and not scored. Lists: Latin `az_parliament` 124, `kz_senate` 48, `uz_senate` 59, `uz_legislative_chamber` 143, `tm_mejlis` 124, Cyrillic `kz_mazhilis` 96, `kg_jogorku_kenesh` 90. Scoring: a name is right when `parsed.family` is the true family. Where the list writes the family in capitals (`az_parliament`, `kz_mazhilis`) the one capitals word is the true family; elsewhere the family is the end the list writes it at, read by eye off the first twelve names of each list in file order (after the filter above), then applied to every name of two or more words (kz_senate and uz_senate first, uz_legislative_chamber, tm_mejlis and kg_jogorku_kenesh last). So only `az_parliament` and `kz_mazhilis` have an independent truth; every other row is scored against the list's own convention (the end that list writes the family at), not against a sampled reading of each name, and it shows what the settings do to a list written that way, not how often they read real families. The by-eye sample is twelve, not the thirty first planned. Re-taken 2026-11-10 by `uv run python tools/differential/suggest_measure.py` (its TURKIC table, the same selection and scoring, with a `suggest` column end to end), which reproduces every cell below; the label of kz_mazhilis was wrong as first written ("family first (capitals)"), the list writing its capitals family last (`Тилектес Серикбайулы АДАМБЕКОВ`), as the 96/96 under the default order shows. Family right out of the scored names under each fixed `Policy` (the second column holds the order the list uses): + + | list | order | nothing | {T} | {T}+FF | {T}+FFGL | {E} | {E,T} | FF alone | + |---|---|---|---|---|---|---|---|---| + | az_parliament | family first | 0/124 | 121/124 | 124/124 | 124/124 | 0/124 | 121/124 | 124/124 | + | kz_senate | family first | 0/48 | 0/48 | 48/48 | 48/48 | 41/48 | 41/48 | 48/48 | + | uz_senate | family first | 0/59 | 2/59 | 59/59 | 59/59 | 52/59 | 54/59 | 59/59 | + | uz_legislative_chamber | family last | 143/143 | 143/143 | 0/143 | 0/143 | 143/143 | 143/143 | 0/143 | + | tm_mejlis | family last | 124/124 | 124/124 | 0/124 | 0/124 | 124/124 | 124/124 | 0/124 | + | kz_mazhilis | family last (capitals) | 96/96 | 96/96 | 0/96 | 0/96 | 96/96 | 96/96 | 0/96 | + | kg_jogorku_kenesh | family last | 90/90 | 90/90 | 0/90 | 0/90 | 90/90 | 90/90 | 0/90 | + + Only 121 of az's 124 names, 2 of uz_senate's, 7 of uz_legislative_chamber's, 2 of tm_mejlis's and 4 of kg_jogorku_kenesh's carry a Turkic marker word (`oglu`, `oğlu`, `qızı`, `o‘g‘li`); the other lists write Russified `-ovich` patronymics, which are East Slavic votes, and no Turkic setting touches them. The order columns are the list's own convention, so FF raises the family-first lists and breaks the family-last ones whether or not the rule is on: the order is not a property of the patronymic. What the rule adds is that `{TURKIC}` alone reads a marker name either way round under the default order (`Mamedov Ali Huseyn oglu` and `Ali Huseyn oglu Mamedov` both family Mamedov, given Ali, patronymic `Huseyn oglu`), where the default alone leaves family `oglu`; under `FAMILY_FIRST` the given-first `Ali Huseyn oglu Mamedov` reads family Ali. So the builder check the first design stated, that TURKIC under the default order leaves family `oglu`, no longer holds on the tree measured here (the rule reads both) and the Turkic answer is the East Slavic one minus the forced order: a decided Turkic rule gives `patronymic_rules={TURKIC}`, and the order is decided from the names' own votes, never forced to given-first, because forcing it would turn az's capitals-written family-first names (124/124 with the order decided, 121/124 with it forced) wrong. End to end through `Parser().suggest` on each whole list: az 124/124 (`name_order=FAMILY_FIRST` from the capitals votes, `{TURKIC}`), the Russified lists decided `{EAST_SLAVIC}`. Known cost: where a list holds both shapes the larger set wins the one patronymic decision, so uz_senate (2 marker names against `-ovich`) decides East Slavic and reads 52/59 (its `o‘g‘li` names, and its two-word names, which the forced given-first order of a decided East Slavic rule reads family-last); kz_senate's `-uly` patronymics (`Kydyrali Darkhan Kuandykuly`) are not a Turkic signal at all and read 41/48, still above "nothing" at 0/48. Both are about East Slavic deciding a mixed group, not the Turkic answer, and are left open. + +- **2026-11-10 THRESHOLDS — TWO NAMES, MORE THAN TWO THIRDS; A PATRONYMIC RULE FIVE NAMES AND MORE THAN FOUR IN FIVE (proposed, for review).** Measured with `uv run python tools/differential/suggest_measure.py` (defaults: groups of 2, 3, 4, 5, 10, 20, 50 and 200 names, 300 groups a size, seed 20261110; 20s wall clock on 2026-11-10 on one 12-CPU machine, a machine-bound figure a re-take times again; the candidates below were run before sizes 2 to 4 joined the defaults, over 5 to 200). Each population is a local corpus whose names are composed from their filers' boxes, so the order is known: PatentsView inventors of 13 countries, UK company filers (PSC) of 8 nationalities, Czech candidates and Vietnamese names, at most 20,000 names each drawn by a seeded reservoir, every group written given-first and family-first and read under a given-first and a family-first base. A group is right where the suggested `name_order` writes the family at the end it was written with. The constraint is the anti-goal: a group written in its base's own order must stay right in at least 95% of groups at every size. Harness proof first: `--set _MIN_VOTES=1000000 --set _MIN_PATRONYMIC_VOTES=1000000` decides nothing (the second holds the patronymic rule, which forces an order of its own; before the patronymic count, `_MIN_VOTES` held both), and every one of the 46 rows written against its base read 0% at all eight sizes, every one of the 46 written in its base's order 100%. The figures here were re-taken after a review found the first draft's group draws seeded like the reservoir's, which made a group a run of consecutive records of a sorted file; the two streams are seeded apart now, and nothing below comes from the coupled draw. + - The rule before this entry (`_MIN_VOTES = 1`, a share of a half for the order and the patronymic rule; `--set _MIN_VOTES=1 --set _MIN_SHARE=0.5 --set _MIN_PATRONYMIC_SHARE=0.5`) failed it: its worst same-order row read 54%, 66%, 70%, 68% and 82% at the five sizes, PatentsView's Russian inventors written family-first under a family-first base. 43% of those names carry a patronymic (`Koryukalov Yury Igorevich`, the tool's signal counts), so many groups decided the East Slavic rule, and its forced given-first order reads the other names (`Popov Ivan`) given-first. + - Candidates, each a run of the tool with `--set` (`_MIN_VOTES`, order share, patronymic share: the worst same-order row at 5 / 10 / 20 / 50 / 200, then the mean of the rows written against their base): 1, 0.6, 0.8: 91 / 92 / 97 / 98 / 100, mean 88 / 95 / 98 / 100 / 100; 1, 2/3, 0.8: 91 / 92 / 97 / 99 / 100, mean 87 / 94 / 97 / 99 / 100 (with one vote enough, Indian inventors' given-first groups fail at 5 and 10: one name voting family-first moves 9% of them); 2, 0.5, 0.8: 96 / 99 / 97 / 98 / 100, mean 72 / 88 / 96 / 100 / 100; 2, 0.6, 0.8: 96 / 99 / 98 / 98 / 100, the same mean; 2, 2/3, 0.5: 54 / 66 / 70 / 68 / 82 (the Russian groups again, whatever the order share); 2, 2/3, 0.8: 96 / 99 / 99 / 100 / 100, mean 71 / 87 / 95 / 99 / 100; 2, 2/3, 0.9: the same as 0.8; 3, 2/3, 0.8: 96 / 99 / 100 / 100 / 100, mean 53 / 79 / 92 / 99 / 100. + - Chosen: `_MIN_VOTES = 2`, `_MIN_SHARE = 2/3` and a new `_MIN_PATRONYMIC_SHARE = 0.8`, the last read for the Turkic rule as for the East Slavic one. Two names is the least that meets the constraint at 5 and 10 names, re-checked on the uncoupled draw; two thirds holds the same-order rows at 99% from 10 names where 0.6 lets them fall to 98%, for one point of the mean, and a tie or two names against one decides nothing; the patronymic rule gets its own, higher share because a decided East Slavic rule overrides the order, and four in five is the lowest tried that keeps the Russian family-first groups at the bar from 5 names (96% at 5, 99% at 10, 100% from 20, before the patronymic count below). Nine in ten does no better, because the groups it still moves are those whose voting names are all patronymic ones, the rest casting no vote. The contested names stay in the patronymic share's denominator (the names casting any vote): `--checks` decides each group of a draw of its own (the same sizes and trials, 2 to 200, 110,400 groups) that holds at least `_MIN_PATRONYMIC_VOTES` names of one rule, the groups that could decide it (2,031 after the fix wave), under both denominators, with and without the names whose only evidence is a contested order, and none decides another rule (a planted positive control, five patronymic names and two contested ones, does); a name with conflicting order signals still carries evidence. TOP, MAX_FAMILY_SHARE and MIN_USES in tools/vocab/order_names.py are unchanged: the tables meet the constraint as built, and what they limit is how soon a group written against its base moves, which is the coverage question left open. + - Added after the branch's final review: `_MIN_PATRONYMIC_VOTES = 5`, a patronymic rule needing five names carrying it besides the share, and sizes 2, 3 and 4 in the tool's defaults. A group of two to four of PatentsView's Russian inventors is often all patronymic names, so the share alone decided East Slavic and forced a given-first order: the same-order row of those names written family-first under a family-first base read 81 / 88 / 94 / 96 / 98% at 2 / 3 / 4 / 5 / 10 names (re-taken with the joined-initials change below already in), and 100 / 100 / 100 / 98 / 98% with the count. The worst same-order row of all 46 now reads 99 / 100 / 99 / 98 / 98 / 99 / 100 / 100% at 2 / 3 / 4 / 5 / 10 / 20 / 50 / 200 names; the mean of the rows written against their base is 32 / 51 / 63 / 71 / 87 / 95 / 99 / 100%, unchanged by the count, which moved only the two Russian rows written family-first (the given-first-written row under a family-first base fell from 59 / 87 / 95 / 98% to 51 / 83 / 94 / 97% at 2 to 5 names, a small group of patronymic names no longer moving the base). The same review made a final capitals word of two or three letters in a mixed-case name (joined initials, `Armitage JO`, or a short surname, `Jason NG`) cast no capitals vote and no surname lookup, and split initials on hyphens (`J.-P.`); against the branch before both (`--base`), those two moved no other cell by more than 0.7 points (vi's family-first groups of three under a given-first base, 77.7% to 78.3%), and the surname table's votes by two names whose last word was listed too (`--checks`: 162,544 to 162,546 surname votes, 182 to 183 wrong from the short surnames). + - The chosen rule, per population (share of groups right; sizes 2 / 3 / 4 / 5 / 10 / 20 / 50 / 200), re-taken after the 2026-11-10 FIX WAVE: + + | population | names | written in the base's order, lowest of both bases | given-first written, family-first base | family-first written, given-first base | + |---|---|---|---|---| + | cz | 20000 | 100% | 52 / 87 / 96 / 99 / 100 / 100 / 100 / 100 | 54 / 83 / 95 / 98 / 100 / 100 / 100 / 100 | + | psc-British | 20000 | 100% | 23 / 49 / 64 / 82 / 99 / 100 / 100 / 100 | 16 / 36 / 58 / 71 / 96 / 100 / 100 / 100 | + | psc-Chinese | 18453 | 99% | 60 / 84 / 92 / 95 / 100 / 100 / 100 / 100 | 59 / 83 / 94 / 96 / 99 / 100 / 100 / 100 | + | psc-Indian | 20000 | 100% | 1 / 3 / 8 / 7 / 30 / 60 / 93 / 100 | 2 / 1 / 4 / 5 / 20 / 49 / 84 / 100 | + | psc-Nigerian | 11648 | 100% | 2 / 9 / 6 / 15 / 43 / 79 / 98 / 100 | 1 / 5 / 7 / 16 / 35 / 68 / 96 / 100 | + | psc-Polish | 15099 | 100% | 70 / 96 / 98 / 100 / 100 / 100 / 100 / 100 | 71 / 91 / 98 / 100 / 100 / 100 / 100 / 100 | + | psc-Romanian | 20000 | 100% | 6 / 12 / 19 / 28 / 60 / 91 / 100 / 100 | 5 / 16 / 20 / 34 / 60 / 94 / 99 / 100 | + | psc-Spanish | 6101 | 100% | 27 / 52 / 73 / 84 / 99 / 100 / 100 / 100 | 20 / 38 / 60 / 73 / 97 / 100 / 100 / 100 | + | psc-Turkish | 9081 | 100% | 8 / 17 / 30 / 41 / 77 / 97 / 100 / 100 | 9 / 13 / 24 / 38 / 68 / 96 / 100 / 100 | + | pv-BR | 15029 | 99% | 25 / 45 / 64 / 77 / 100 / 100 / 100 / 100 | 16 / 32 / 44 / 57 / 87 / 97 / 100 / 100 | + | pv-CN | 20000 | 100% | 72 / 89 / 96 / 99 / 100 / 100 / 100 / 100 | 72 / 89 / 97 / 99 / 100 / 100 / 100 / 100 | + | pv-DE | 20000 | 100% | 28 / 48 / 68 / 80 / 99 / 100 / 100 / 100 | 30 / 46 / 69 / 78 / 99 / 100 / 100 / 100 | + | pv-ES | 20000 | 100% | 35 / 69 / 78 / 89 / 99 / 100 / 100 / 100 | 24 / 51 / 66 / 75 / 99 / 100 / 100 / 100 | + | pv-FR | 20000 | 100% | 13 / 20 / 34 / 46 / 85 / 99 / 100 / 100 | 16 / 24 / 44 / 58 / 91 / 100 / 100 / 100 | + | pv-IN | 20000 | 99% | 1 / 5 / 9 / 12 / 37 / 68 / 89 / 99 | 3 / 10 / 15 / 24 / 51 / 81 / 99 / 100 | + | pv-IT | 20000 | 100% | 37 / 65 / 80 / 91 / 99 / 100 / 100 / 100 | 44 / 73 / 85 / 91 / 100 / 100 / 100 / 100 | + | pv-JP | 20000 | 100% | 17 / 38 / 49 / 65 / 93 / 99 / 100 / 100 | 19 / 43 / 54 / 66 / 95 / 99 / 100 / 100 | + | pv-KR | 20000 | 100% | 61 / 83 / 90 / 97 / 100 / 100 / 100 / 100 | 59 / 87 / 97 / 96 / 99 / 100 / 100 / 100 | + | pv-NL | 20000 | 100% | 30 / 49 / 67 / 77 / 99 / 100 / 100 / 100 | 39 / 64 / 84 / 92 / 100 / 100 / 100 / 100 | + | pv-RU | 20000 | 98% | 51 / 83 / 94 / 97 / 100 / 100 / 100 / 100 | 30 / 52 / 74 / 86 / 98 / 100 / 100 / 100 | + | pv-TW | 20000 | 100% | 58 / 92 / 92 / 97 / 100 / 100 / 100 / 100 | 62 / 84 / 94 / 97 / 100 / 100 / 100 / 100 | + | pv-US | 20000 | 100% | 19 / 36 / 52 / 65 / 94 / 100 / 100 / 100 | 41 / 68 / 83 / 95 / 100 / 100 / 100 / 100 | + | vi | 20000 | 100% | 49 / 71 / 88 / 95 / 100 / 100 / 100 / 100 | 46 / 78 / 87 / 94 / 100 / 100 / 100 / 100 | + + - Read as the smallest size from which both rows written against the base stay at 95% or more: 4 for Czech, PatentsView's Chinese and PSC's Polish groups (Polish 5 before the fix wave's fold); 5 for PSC's Chinese, and PatentsView's Taiwanese and Korean; 10 for German, Dutch, Italian, Spanish, British, Russian and Vietnamese; 20 for French, US, Japanese, Brazilian and Turkish; 50 for Nigerian and Romanian; 200 for Indian. `Parser.suggest`'s docstring states these, rounding the first two rows into 5. A smaller group still moves the base where its votes clear the thresholds, but in fewer of the draws, the rest keeping the base, which is the anti-goal's fallback. + - Known cost: the forced given-first order of a decided East Slavic rule. With the share and the count it still moves 2% of the Russian family-first groups of five and of ten under a family-first base (the 98% above; the tool prints how many groups decided the rule, 62 of the population's 9,600 suggestions, both written orders under both bases, 64 before the fix wave), and a family-first source of Russian names that mostly carry a patronymic would be decided East Slavic and read given-first, which reads its names without one wrong. Left open. + - The real parliament lists, each read whole under both bases (the tool's last table; no draw is involved, so the draw's re-seeding left them as they were): every Latin and Cyrillic list's order comes out as the list writes it or is decided East Slavic (ua_rada, ru_duma_deputies, kg_jogorku_kenesh, kz_senate, uz_senate, uz_legislative_chamber, tm_mejlis), and the same under both bases; az_parliament family-first with `{TURKIC}`, by_council_republic family-first (48 for, 0 against), tr_parliament given-first (566 for, 6 contested), in_sansad given-first (647 for, 167 against, 3 contested, 7,478 no evidence), re-taken after the fix wave read names past their titles, in_sansad's `Smt.`, `Shri` and `Shrimati` above all (565 and 441 for before it, and 586 and 443 before the final capitals word of two or three letters stopped voting). The East Slavic lists that write family-first (ua_rada, kz_senate, uz_senate) get the given-first order the rule needs. + - Checks deferred from the reviews: `uv run python tools/differential/suggest_measure.py --checks`, over every population's whole pool written both ways (830,822 names, 2026-11-10, re-taken after the fix wave). The seven short romanized surnames (`do`, `to`, `la`, `le`, `ha`, `ho`, `an`) cast 182 wrong surname votes of 162,521 (183 of 162,546 before); outside the East Asian populations the wrong surname votes come from `lee` 294, `dan` 248 (a Romanian given name), `rao` 179 (an Indian name element written mid-name), `ma` 72 and `kim` 51. A bare non-ASCII trailing letter (`Jean Ö`), which casts no trailing-initial vote where `Jean O` does, ends 51 names against 1,706 ASCII ones. Neither is fixed: the group rule absorbs each. The third check, `fold` leaving `ø` and `ł` unmapped (25 such spellings in the table, several without their ASCII form beside them, so the UK filers' Polish names typed in ASCII cast no vote; mapping both sides moved 2,607 given-name votes to right, 2,543 of them Polish, and 22 away), was taken up by the fix wave, and the check is gone with it. + +- **2026-11-10 FIX WAVE — WHERE A SIGNAL READS, AND WHAT THE RESULT TYPES HOLD (PROPOSED, FOR REVIEW).** A four-way review of the branch found signals reading where the pipeline does not, and result types that could hold evidence contradicting their patch. What changed in the reading, each as proposed: + - A patronymic votes only where the pipeline's rule reads one: rules/O1's three name words with the East Slavic ending on the middle word (given-first) or on the last (`_vocab.east_slavic_rotates`, O1's word test, shared with `_post_rules` rather than copied), whatever the first word ends in, as O1 asks only about the middle and last words, and no initial first or in the middle (O1's precedence); rules/O2's four name words with the Turkic marker last (family-first, O2's rotation) or third (given-first, which `{TURKIC}` reads too, the TURKIC entry above). Before it, an `-ovich` ending on any word voted, so six family-first names whose surnames end in `-ovich` (`Abramovich Roman`, `Rabinovich Pavel`, ...) decided East Slavic 6-0 under a family-first base and forced given-first, reading `Abramovich` as the given name. A first draft of the fix also refused an ending on the first word, which O1 does not ask: `Abramovich Roman Arkadyevich`, which `{EAST_SLAVIC}` reads right, then cast no vote; the PR's re-review found it (2026-11-10), and the two-word false positive is excluded by the three-word count alone. + - A decided East Slavic rule no longer overrides an order the names' own votes decide family-first: the rule stays in the patch, the order is the votes', and the order evidence's note says the rule stands down under it (five East Slavic names written `SIDOROV Ivan Petrovich`, their capitals voting family-first). A name carrying the rule's patronymic disagrees on order then, as any name does. + - A name is read past the titles the base lexicon lists, and rules/H2's period shape, at its head, and past the suffixes it lists and the numerals the trailing fork reads as the generation at its end, a one-letter numeral and a listed acronym written as initials needing two name words in front (`Armitage V`, `Smith J.P.` end in initials); a word the name tables list is never read past (`Law`, a Cantonese surname and a title; `Cem`, a Turkish given name and a credential). Before it, `Dr. Zhang Xiaoming` cast no surname vote and six titled Chinese names whose given name is also a listed surname (`Dr. Zhang Wei`) moved a family-first base to given-first. An unlisted credential is a name word to the suggester as to the parse (`Tanaka Hiroshi FRCS`, `frcs` being unlisted, still casts a capitals vote). + - A comma followed by post-nominals alone (`Zhang Wei, PhD`) reads the name before it; any other comma name still casts nothing, and the order evidence's note says how many were left out. + - Quotes and brackets around a word, and a final full stop, are read past (`Petrov Ivan Ivanovich.`, `"Kim" Minjun`), and each word is composed (NFC) first, so a decomposed `oğlu` votes as its composed twin. The parse itself does not compose there: under `{TURKIC}`, `Mamedov Ali Huseyn oğlu` written decomposed reads family `oğlu` (left open). + - `fold` writes `ł`, `ø`, `ı`, `æ`, `œ`, `ß` and `đ` as a keyboard types them, and the tables are rebuilt with it: the given-name table holds 4,262 names (4,271 before), every stroked or dotless spelling now its ASCII one; `baris`, `pinar`, `sila`, `asli` and `blaz` leave it, their spellings pooled with the ASCII ones written in ORCID's family box more than 5% of the time (`baris` 8 of 89). `tools/vocab/order_names.py` now refuses an input not whole (nam_dict against its pinned sha256, ORCID's raw.jsonl against its meta.json's sha256 and record count), excludes the given names against the surnames the same run computes, and writes nothing below `MIN_PINYIN` (200) or `MIN_GIVEN` (3,000) or with a word in both tables. + - `Suggestion` refuses evidence its patch contradicts (a setting twice, a decided value not the suggested parser's, an undecided setting the patch sets, a patch field no decided evidence holds or that restates the base, counts not adding up to the names). + - Re-taken with `uv run python tools/differential/suggest_measure.py --base ` (the branch before the wave): the worst same-order row of all 46 reads 99 / 100 / 99 / 98 / 98 / 99 / 100 / 100% at 2 / 3 / 4 / 5 / 10 / 20 / 50 / 200 names, as before; the mean of the rows written against their base 32 / 52 / 63 / 71 / 87 / 95 / 99 / 100% (32 / 51 / ... before). 35 cells moved, PSC's Polish groups most, the fold's work (given-first written under a family-first base 56 / 87 / 94 / 99% to 70 / 96 / 98 / 100% at 2 to 5 names, family-first under a given-first base 55 / 84 / 97 / 98% to 71 / 91 / 98 / 100%), the rest by at most 1.3 points; one same-order cell moved, PatentsView's Indian groups written family-first under a family-first base, 100% to 99.7% at 10 names. The TURKIC table did not move. The harness proof (`--set _MIN_VOTES=1000000 --set _MIN_PATRONYMIC_VOTES=1000000`) still reads 0% for each of the 46 rows written against its base and 100% for each of the 46 in it, at every size. The per-population table in THRESHOLDS is this run's. + - Moved here from the module's comments, which keep the rule, a clause and a pointer to this file: with one vote enough (`_MIN_VOTES = 1`), a single name moved 9% of PatentsView's Indian given-first groups of five, and 8% of ten, to family-first; under a patronymic share of a half, 18% to 46% of the Russian family-first groups (200 to 5 names) were moved to given-first under a family-first base, and at four in five, 4% of the groups of five and none from 20, before the count; with the order count alone, 19%, 12% and 6% of the Russian groups of two to four were moved, and with five, none, and 2% of the groups of five and ten. Each figure is a THRESHOLDS figure read the other way (the moved share is 100% less the share right), taken before the fix wave. diff --git a/docs/design/rules/O1-east-slavic-rotation.md b/docs/design/rules/O1-east-slavic-rotation.md index c34791d2b..e4a98521b 100644 --- a/docs/design/rules/O1-east-slavic-rotation.md +++ b/docs/design/rules/O1-east-slavic-rotation.md @@ -43,4 +43,4 @@ --- **Signal:** shape: a patronymic ending on the last of exactly three name words, and not on the middle one -**history:** decisions/O1 · **implemented:** nameparser/_pipeline/_post_rules.py +**history:** decisions/O1 · **implemented:** nameparser/_pipeline/_post_rules.py, nameparser/_pipeline/_vocab.py diff --git a/docs/modules.rst b/docs/modules.rst index 29d78a4b4..45fd7030e 100644 --- a/docs/modules.rst +++ b/docs/modules.rst @@ -109,6 +109,23 @@ Comparing names from several sources: see :ref:`matching-names`. .. autoclass:: nameparser.Side :members: +Group settings +~~~~~~~~~~~~~~ + +Settings suggested for a list of names: see :ref:`suggest-settings`. + +.. autoclass:: nameparser.Suggestion + :members: + +.. autoclass:: nameparser.SettingEvidence + :members: + +.. autoclass:: nameparser.NameEvidence + :members: + +.. autoclass:: nameparser.Signal + :members: + Configuration ~~~~~~~~~~~~~ diff --git a/docs/release_log.rst b/docs/release_log.rst index 968be610e..ed0314a0e 100644 --- a/docs/release_log.rst +++ b/docs/release_log.rst @@ -6,6 +6,7 @@ Release Log - **Remove loading a Lexicon, Parser or locale pack pickled by 2.x, which 3.0 deprecated.** ``pickle.loads`` of a ``Parser`` pickled by 2.8.0 raises ``ValueError`` saying to re-pickle it under 3.0, where 3.0.x loaded it with a ``DeprecationWarning``; the error for a 2.x ``Lexicon`` spelling now names :doc:`migrate`'s table rather than each word's calls. :issue:`1003` - **Add Parser.matching(), to group and compare names from several sources.** ``Parser().matching().check("John A. Smith", "J. Smith")`` is consistent, with a table reading the given name as an initial and the middle name as written on one side only, where ``parse("John A. Smith").matches("J. Smith")`` is ``False``, and ``keys()`` gives grouping keys that a consistent pair always shares. Consistent means nothing in the two names contradicts the match, not that they are one person, and it is not transitive (``J. Smith`` is consistent with ``John Smith`` and with ``Jane Smith``). An opt-in outcome, ``familiar-form``, reads a familiar form as its longer given name (``Bill`` / ``William``, ``Ricky`` / ``Richard``), from pairs seeded from Jörg Michael's nam_dict and a reviewed selection of Old Dominion University's nicknames list. See :ref:`matching-names`. :issue:`1006`, :issue:`1267`, :issue:`1276`, :issue:`1261`, :issue:`1263` + - **Add Parser.suggest(), to read the settings a list of names is written in.** ``Parser().suggest(["Priem Jason", "Kim Minjun", "Wang Xiaoming", "Nguyen Anh", "Jason Priem"]).parser.parse("Priem Jason")`` gives given ``Jason``, family ``Priem``, where ``Parser().parse("Priem Jason")`` gives given ``Priem``, family ``Jason``, the list carrying the evidence for a family-first ``name_order``; a group of East Slavic or Turkic patronymics gets the patronymic rule too. A list of names with no common surname or given name carries no evidence and keeps the base, and moving the base takes from 5 names (Chinese, Korean, Czech) to 200 (Indian). See :ref:`suggest-settings`. :issue:`1308` - **Fix the commas of Ethiopic, Burmese, N'Ko, Mongolian, Manchu, Lisu, Vai and Bamum, and the Ethiopic wordspace, which did not divide a name.** ``parse("ከበደ፣ አበበ")`` gives family ``ከበደ``, given ``አበበ``, where 1.4.0 through 3.0.1 gave given ``ከበደ፣``, family ``አበበ``; the other scripts' commas read the same way, every non-ASCII comma also divides the words of a bracketed credential as the ASCII one does, and ``አበበ፡ከበደ፡ተሰማ`` divides into three words at the wordspace ``፡``. :issue:`1202` - **Fix a Mongolian case suffix joined by a narrow no-break space, which read as a word of its own.** ``Parser(policy=Policy(name_order=FAMILY_FIRST)).parse("ᠬᠣᠷᠯᠤ᠎ᠠ ᠶ᠋ᠢᠨ ᠴᠣᠢᠢᠪᠠᠯᠰᠠᠩ")`` gives family ``ᠬᠣᠷᠯᠤ᠎ᠠ ᠶ᠋ᠢᠨ``, given ``ᠴᠣᠢᠢᠪᠠᠯᠰᠠᠩ``, where 2.0.0 through 3.0.1 split at the U+202F and gave given ``ᠶ᠋ᠢᠨ``. 1.4.0, which has no name order, split there too, reading middle ``ᠶ᠋ᠢᠨ``. Elsewhere U+202F still divides like a space. :issue:`1236` - **Fix the unbalanced-delimiter report on Dutch 't and 's, and read 's as part of the surname.** ``parse("Pieter 't Hart")`` and ``parse("Arent van 's-Gravesande")`` report nothing, where 2.0.0 through 3.0.1 reported ``unbalanced-delimiter``; ``Willem Jacob 's Gravesande`` gives family ``'s Gravesande``, where 1.4.0 through 3.0.1 gave middle ``Jacob 's``; and ``Jan van 't Hof 'Jantje'`` gives nickname ``Jantje`` as 1.4.0 did, where 2.0.0 through 3.0.1 gave ``t Hof 'Jantje``. :issue:`1241` diff --git a/docs/usage.rst b/docs/usage.rst index 38e659802..8a0e25eb4 100644 --- a/docs/usage.rst +++ b/docs/usage.rst @@ -1620,6 +1620,80 @@ its given name, is the more specific of the two: Link each record to its most specific consistent name rather than to every name it is consistent with. +.. _suggest-settings: + +Settings for a list of names +---------------------------- + +A list of names is usually written in one convention: a registry gives +every family name first, a mailing list every family name last. A +single name cannot show which, because ``Priem Jason`` and ``Jason +Priem`` are the same shape and only a reader who knows ``Priem`` is a +surname and ``Jason`` a given name can tell them apart. A whole list +can: most of its names carry some evidence, and the evidence agrees. + +``Parser.suggest()`` reads a list you declare to be one group (one +source, one convention) and returns a :class:`~nameparser.Suggestion`. +Each name votes on its order from the evidence it carries: a family +name written in capitals at one end, a trailing initial +(``Smith J.``), a lowercase particle first or before the last word, a +first or last word that is a common romanized Chinese, Korean or +Vietnamese surname, or a common given name, each read past the titles +and post-nominals your parser's vocabulary lists (``Dr.``, ``PhD``, +``Jr.``). A name with a comma states its own order and does not vote, +unless only post-nominals follow the comma (``Zhang Wei, PhD``), and +neither does a name of one word. The +group decides ``name_order`` when at least two names vote for one end +and they are more than two thirds of the names that cast a vote. A +patronymic rule (East Slavic or Turkic, :class:`~nameparser.PatronymicRule`) is +decided only where at least five names carry its patronymic and they +are more than four in five of the voting names, and a patronymic +votes only where the rule can read it (three name words for East +Slavic, four for Turkic); a group decided East Slavic reads +given-first, since that rule reorders each name itself and stands down +under a family-first order, unless its names' own votes decide a +family-first order, which the rule never overrides. Every setting the +votes do not decide keeps the base parser's value. + +.. doctest:: + + >>> from nameparser import Parser + >>> listing = ["Priem Jason", "Kim Minjun", "Wang Xiaoming", + ... "Nguyen Anh", "Jason Priem"] + >>> suggestion = Parser().suggest(listing) + >>> suggestion.patch + PolicyPatch(name_order=FAMILY_FIRST) + >>> Parser().parse("Priem Jason").given + 'Priem' + >>> suggestion.parser.parse("Priem Jason").given + 'Jason' + >>> [entry.text for entry in suggestion.disagreeing()] + ['Jason Priem'] + +``suggestion.parser`` is the base parser with ``suggestion.patch`` +applied, so the base is never changed. ``suggestion.evidence`` holds a +:class:`~nameparser.SettingEvidence` for each setting considered and +``suggestion.names`` a :class:`~nameparser.NameEvidence` for each name, +in input order; ``print(suggestion)`` writes a report of each +setting's evidence and the names that disagree with it. +``disagreeing()`` lists the names whose vote is against a decided +setting, the ones to check by hand. + +Where it stops. A group is read right only once it carries enough +names with evidence. Measured on groups drawn from real names written +in a known order (2026-11-10), a list written in the base's own order keeps it +in at least 98% of groups from 2 to 200 names, but moving the base +takes more names, from 5 for Chinese, Taiwanese, Korean, Czech and +Polish names to 200 for Indian ones. A plain list of names with no +common surname or given name in it (``Zxqv Brtk``) carries no evidence, +and a group whose votes fall short of those thresholds keeps the base +rather than guess. The +parser never detects the language of a name; the suggester may, over a +group you declare, says what it saw, and its answer is configuration +you apply. The measurements, the thresholds and the evidence declined +are in ``docs/design/decisions/group-suggestion.md``, and +``tools/differential/suggest_measure.py`` re-takes them. + .. _parser-guesses: When the parser had to guess diff --git a/nameparser/__init__.py b/nameparser/__init__.py index c770bdc22..c1c06995e 100644 --- a/nameparser/__init__.py +++ b/nameparser/__init__.py @@ -29,6 +29,12 @@ PolicyPatch, Script, ) +from nameparser._suggest import ( + NameEvidence, + SettingEvidence, + Signal, + Suggestion, +) from nameparser._types import ( STABLE_TAGS, Ambiguity, @@ -55,4 +61,5 @@ "Parser", "parse", "parser_for", "Matching", "MatchResult", "RoleMatch", "CrossRoleMatch", "Outcome", "Side", + "Suggestion", "NameEvidence", "SettingEvidence", "Signal", ] diff --git a/nameparser/_parser.py b/nameparser/_parser.py index c6e81db8d..58d0840a5 100644 --- a/nameparser/_parser.py +++ b/nameparser/_parser.py @@ -31,6 +31,7 @@ if TYPE_CHECKING: from nameparser._match import Matching, Outcome + from nameparser._suggest import Suggestion @dataclass(frozen=True, slots=True) @@ -302,6 +303,52 @@ def matching( self, DEFAULT_ACCEPT if accept is None else accept, # type: ignore[arg-type] other_readings) + def suggest(self, names: Iterable[str]) -> Suggestion: + """The settings a group of names suggests, with THIS parser as + the base: a :class:`~nameparser.Suggestion` whose ``parser`` + reads the group. ``names`` is any iterable of str the caller + declares to be one group (one source, one convention), read + once. Each name casts votes on its name order and its + patronymic shape, read with this parser's lexicon past the + titles it lists at the head and the post-nominals at the end; a + name of fewer than two words once those are read past casts + none, and so does a name with a comma, unless only + post-nominals follow the comma. A patronymic votes only where + the patronymic rule reads one. ``name_order`` is + decided in either direction, keeping this parser's family-first + spelling where it has one, where at least two names vote for + one end and they are more than two thirds of the names casting + an order vote. A patronymic rule is decided where more than four + in five of the names that cast a vote carry its patronymic (and + at least five do), for the East Slavic and the Turkic rule alike. + A group decided East Slavic gets the rule and reads given-first, + since the rule reorders each name and stands down under a + family-first order, unless its names' own votes decide + family-first, which the rule never overrides; a Turkic group + gets the rule alone, its order decided by its names' own votes. + A setting the votes do not decide keeps this parser's value, so + an empty group, or one with no votes, suggests no change. A + bare str, bytes, a mapping or a non-str item raises TypeError. + + Measured 2026-11-10 with tools/differential/suggest_measure.py + on groups drawn from real names written in a known order, 300 + groups a size (decisions/group-suggestion's THRESHOLDS entry + holds the table): a group written in this parser's own order kept + it in at least 98% of groups at every size from 2 to 200 names, + under either base order. Moving the base takes more names, and + how many depends on what the names carry: groups of Chinese, + Taiwanese, Korean, Czech or Polish names (the romanized surnames + and the common given names) were read right in 95% of groups + from 5 names; German, Dutch, Italian, Spanish, British, Russian + and Vietnamese from 10; French, US, Japanese, Brazilian and + Turkish from 20; Nigerian and Romanian from 50; and Indian only + at 200. A group whose votes fall short of the thresholds keeps + the base rather than guess, so a smaller group moves it in fewer + of the draws. + """ + from nameparser._suggest import suggest + return suggest(self, names) + # -- rendering delegates ---------------------------------------------- def capitalized(self, name: ParsedName, *, diff --git a/nameparser/_pipeline/_post_rules.py b/nameparser/_pipeline/_post_rules.py index ae40b1bec..a736dfa6d 100644 --- a/nameparser/_pipeline/_post_rules.py +++ b/nameparser/_pipeline/_post_rules.py @@ -74,7 +74,7 @@ ) from nameparser._pipeline._vocab import ( _HYPHENS, SetApartKind, delimiter_cores, dotted_member, - has_patronymic_ending, is_initial, is_listed_given_name, + east_slavic_rotates, has_patronymic_ending, is_initial, is_listed_given_name, is_patronymic_marker, joined_member, member_word, period_member, reads_as_initials, set_apart, set_apart_kind, settles_a_member, suffix_reading, @@ -1096,7 +1096,7 @@ def post_rules(state: ParseState) -> ParseState: # (so a one-case name's two spellings read alike, a letter P8 # reads as a connective rotating in either; decisions/O1, # #1185, #1217) - if (has_patronymic_ending(tail) and not has_patronymic_ending(mid) + if (east_slavic_rotates(mid, tail) and "initial" not in tokens[givens[0]].tags and "initial" not in tokens[middles[0]].tags): g, m, f = givens[0], middles[0], families[0] diff --git a/nameparser/_pipeline/_vocab.py b/nameparser/_pipeline/_vocab.py index 023c38aca..dddf2dd72 100644 --- a/nameparser/_pipeline/_vocab.py +++ b/nameparser/_pipeline/_vocab.py @@ -371,6 +371,19 @@ def is_patronymic_marker(text: str) -> bool: return bool(_TURKIC.match(text) or _TURKIC_CYR.match(text)) +# rules/O1: "whose last name word carries a patronymic ending and whose +# middle name word does not reads as family-first" -- the word test of +# the rotation, shared with the group suggester (nameparser/_suggest.py), +# which counts a patronymic vote where this rule can read one +# (decisions/group-suggestion) +def east_slavic_rotates(middle: str, last: str) -> bool: + """Whether the middle and last of three name words have rules/O1's + family-first shape: the ending on the last word and not on the + middle one. The word count and the initials beside it are the + caller's.""" + return has_patronymic_ending(last) and not has_patronymic_ending(middle) + + # rules/S10: "the numeral reads as the generation: the numerals the # vocabulary lists, and those up to X it does not, by their shape # alone" -- its Then clause, the `_ROMAN` test; and rules/S10's Except diff --git a/nameparser/_suggest.py b/nameparser/_suggest.py new file mode 100644 index 000000000..d28290f92 --- /dev/null +++ b/nameparser/_suggest.py @@ -0,0 +1,1033 @@ +"""The group suggester: given names a caller declares to be one group, +it suggests ``Policy`` settings, chiefly the name order (``Parser.suggest``). + +Contracts are stated in prose here, with no doctest blocks. This module +holds the shared spelling fold so the name tables +(``config/_order_names.py``, written by ``tools/vocab/order_names.py``) +and the reader that looks words up in them fold alike, and the per-name +signals: ``name_votes`` reads one name and returns the votes it casts, +a dict from ``Signal`` to either an order (``GIVEN_FIRST`` or +``FAMILY_FIRST``) or, for ``Signal.PATRONYMIC``, the ``PatronymicRule`` +its shape matches. A name is read past its titles at the head and its +post-nominals at the end, by the base lexicon, with quotes and brackets +around a word read past too; a name with a comma is read before the +comma where only post-nominals follow it, and casts nothing otherwise. +A signal that does not apply casts no vote, so the dict is empty for a +name of fewer than two words once those are read past. The signals read +cased evidence in any script: a word with no cased letter never counts +as capitals (Cyrillic capitals vote, as Latin ones do), a one-letter +word is never looked up in the name tables, a capitals word of two or +three letters ending a name that holds a lowercase letter (joined +initials, `Armitage JO`, or a short surname, `Jason NG`) casts no +capitals vote and is not looked up as a surname, and a script the +tables do not hold (Hangul, Han) simply matches nothing. A patronymic +votes, Latin or Cyrillic, only where the pipeline's rule reads one +(rules/O1's three name words, rules/O2's four), through the pipeline's +own predicates, never a copy of them. No signal raises on any string. + +``suggest`` tallies the group's votes and decides two settings +together, ``patronymic_rules`` first and ``name_order`` second: a +decided East Slavic rule reorders each name itself and stands down +under a family-first order, so it decides the order as given-first, +unless the names' own votes decide family-first, which it never +overrides (decisions/group-suggestion). A setting is +decided only on the evidence the thresholds below admit; any other +setting keeps the base's value, so an empty group, or one with no +votes, gives back a parser that reads exactly as the base does. The +result is a ``Suggestion``: the ``PolicyPatch`` holding only the +settings the group changed, a ``SettingEvidence`` per setting +considered, and a ``NameEvidence`` per input name, in input order. +""" +from __future__ import annotations + +import dataclasses +import re +import reprlib +import unicodedata +import warnings +from collections import Counter +from collections.abc import Callable, Iterable, Iterator, Mapping +from dataclasses import dataclass +from enum import StrEnum +from types import MappingProxyType +from typing import TYPE_CHECKING, cast + +from ._lexicon import Lexicon, _normalize +from ._pipeline._state import COMMA_CHARS +from ._pipeline._vocab import ( + ROMAN, east_slavic_rotates, has_patronymic_ending, is_initial, + is_one_case, is_patronymic_marker, is_suffix_lenient, is_title_shaped, + reads_as_generation, suffix_as_written, +) +from ._policy import ( + UNSET, PatronymicRule, Policy, PolicyPatch, _order_repr, apply_patch, +) +from ._types import ( + FAMILY_FIRST, GIVEN_FIRST, Role, _guarded_getstate, _guarded_setstate, +) +from .config._order_names import COMMON_GIVEN_NAMES, ROMANIZED_SURNAMES + +if TYPE_CHECKING: + from ._parser import Parser + +# the letters NFKD leaves whole, spelled as an ASCII keyboard writes +# them +_KEYBOARD = str.maketrans({ + "ł": "l", "Ł": "L", "ø": "o", "Ø": "O", "ı": "i", "æ": "ae", "Æ": "AE", + "œ": "oe", "Œ": "OE", "ß": "ss", "ẞ": "SS", "đ": "d", "Đ": "D"}) +# a letter u with a diaeresis, decomposed: the pinyin vowel written v on +# a keyboard (lü -> lv) +_U_DIAERESIS = {"u\u0308": "v", "U\u0308": "V"} + + +def fold(word: str) -> str: + """The spelling a word is looked up under in the name tables: lower + case, every diacritic dropped, the letters NFKD leaves whole written + as an ASCII keyboard writes them (``ł`` as ``l``, ``ø`` as ``o``, + ``ı`` as ``i``, ``æ`` as ``ae``, ``œ`` as ``oe``, ``ß`` as ``ss``, + ``đ`` as ``d``) and ``ü`` as ``v``. + + ``Nguyễn`` folds to ``nguyen``, ``Đặng`` to ``dang``, ``Łukasz`` to + ``lukasz``, ``Barış`` to ``baris`` and ``Lǚ`` to ``lv``. The tables + hold only folded spellings, written with this function, so a reader + folds the word first. + """ + text = unicodedata.normalize("NFKD", word).translate(_KEYBOARD) + for mark, letter in _U_DIAERESIS.items(): + text = text.replace(mark, letter) + return "".join(c for c in text if not unicodedata.combining(c)).lower() + + +class Signal(StrEnum): + """The evidence a single name offers about its writer's name order.""" + + CAPITALS = "capitals" + TRAILING_INITIAL = "trailing-initial" + PARTICLE = "particle" + SURNAME = "surname" + GIVEN_NAME = "given-name" + PATRONYMIC = "patronymic" + + +_INITIAL_JOINS = re.compile(r"[.-]+") + + +def _is_initialed(word: str) -> bool: + """An initial: one letter with an optional period, or single letters + joined by periods or hyphens (`J.P.`, `J.-P.`, `J-P`).""" + if is_initial(word): + return True + if "." not in word and "-" not in word: + return False + chunks = _INITIAL_JOINS.split(word.rstrip(".")) + return all(chunk and is_initial(chunk + ".") for chunk in chunks) + + +def _is_caps_word(word: str) -> bool: + """Two or more letters, all of them capitals, and no initial.""" + letters = [c for c in word if c.isalpha()] + # single letters joined by stops or hyphens ('J.-P.', 'J-P') are + # initials, however many letters they hold + chunks = [c for c in word.replace(".", "-").split("-") if c] + paired_initials = len(chunks) > 1 and all(len(c) == 1 for c in chunks) + return (len(letters) >= 2 and word.isupper() + and not paired_initials and not _is_initialed(word)) + + +def _has_lower(words: list[str]) -> bool: + return any(c.islower() for w in words for c in w) + + +def _ends_in_short_capitals(words: list[str]) -> bool: + """Whether a name holding a lowercase letter ends in a capitals word + of two or three letters: joined initials (`Armitage JO`, `Smith JP`) + or a short surname (`Jason NG`), which the word alone cannot tell + apart. Such a word casts no capitals vote and is not looked up as a + surname; a leading one (`NG Jason`) still votes.""" + last = words[-1] + letters = sum(c.isalpha() for c in last) + return (2 <= letters <= 3 and _is_caps_word(last) + and _has_lower(words)) + + +def _capitals(words: list[str]) -> tuple[Role, Role, Role] | None: + """A run of capitals words at one end of a name that also holds a + lowercase letter reads as the family name, which writers set in + capitals.""" + if not _has_lower(words) or _ends_in_short_capitals(words): + return None + # a name holding a lowercase letter holds a word that is not all + # capitals, so the two ends never both vote by being every word + caps = [_is_caps_word(w) for w in words] + if caps[0] and not caps[-1]: + return FAMILY_FIRST + if caps[-1] and not caps[0]: + return GIVEN_FIRST + return None + + +def _particle(words: list[str], lexicon: Lexicon) -> tuple[Role, Role, Role] | None: + if len(words) < 3: + return None + particles = lexicon._sets.particles + + def is_particle(word: str) -> bool: + return word == word.lower() and _normalize(word) in particles + + first = is_particle(words[0]) + before_last = is_particle(words[-2]) + if first == before_last: + return None + return FAMILY_FIRST if first else GIVEN_FIRST + + +def _listed(word: str, table: frozenset[str]) -> bool: + spelling = fold(word.strip(".")) + return len(spelling) >= 2 and spelling in table + + +def _at_one_end(words: list[str], table: frozenset[str], + skip_last: bool = False) -> tuple[bool, bool]: + return _listed(words[0], table), not skip_last and _listed(words[-1], table) + + +def _a_listed_name(word: str) -> bool: + """A word the name tables list: name evidence, which is never read + past as a title or a post-nominal (`Law`, a Cantonese surname and a + listed title; `Cem`, a Turkish given name and a listed credential).""" + return (_listed(word, ROMANIZED_SURNAMES) + or _listed(word, COMMON_GIVEN_NAMES)) + + +def _is_leading_title(word: str, lexicon: Lexicon) -> bool: + """A word the lexicon lists as a title, or one rules/H2's shape + reads as a title at the head of a name (`Prof.`, `Insp.`).""" + return not _a_listed_name(word) and ( + _normalize(word) in lexicon._sets.titles + or is_title_shaped(word, lexicon)) + + +def _is_post_nominal(word: str, lexicon: Lexicon) -> bool: + """A word after a comma that is a suffix the lexicon lists, or a + numeral of two letters or more: is_suffix_lenient, the reading + after a comma, which takes a listed one-letter numeral too.""" + stem = word.rstrip(".") + return not _a_listed_name(word) and ( + is_suffix_lenient(word, lexicon) + or (len(stem) > 1 and ROMAN.match(stem) is not None)) + + +def _ends_in_post_nominal(words: list[str], lexicon: Lexicon, + one_case: bool) -> bool: + """Whether the last of `words` is a suffix the lexicon lists or a + numeral the trailing fork reads as the generation. A one-letter + numeral, and a listed acronym written as initials (`J.P.`), need two + name words in front of them, as the trailing fork's words-to-spare + count asks (rules/S10, rules/S3): `Armitage V` and `Smith J.P.` end + in initials.""" + word = words[-1] + if len(words) < 2 or _a_listed_name(word): + return False + stem = word.rstrip(".") + if len(words) < 3 and (len(stem) == 1 or _is_initialed(word)): + return False + listed = suffix_as_written(_normalize(word), word, lexicon) + if stem and ROMAN.match(stem) is not None: + return reads_as_generation( + word, frozenset({"vocab:suffix"}) if listed else frozenset(), + words[-2], one_case) + return listed + + +#: The marks a word is written inside (quotes, brackets), stripped from +#: its ends before any test. +_EDGES = "\"'\u2018\u2019\u201a\u201b\u201c\u201d\u201e\u00ab\u00bb\u2039\u203a()[]{}" +_COMMAS = re.compile("[" + re.escape("".join(sorted(COMMA_CHARS))) + r"\s]+") + + +def _name_words(text: str, lexicon: Lexicon) -> tuple[list[str], bool]: + """The words the signals read, and whether a comma left the name + out. A comma followed by post-nominals alone (`Zhang Wei, PhD`) + leaves the part before it; any other comma name is left out, its + comma stating its order. Quotes and brackets are stripped from each + word, which is composed (NFC), then the titles at the head and the + post-nominals at the end (a listed suffix, a numeral read as the + generation).""" + cut = next((i for i, c in enumerate(text) if c in COMMA_CHARS), None) + if cut is not None: + tail = [w.strip(_EDGES) for w in _COMMAS.split(text[cut + 1:])] + if not all(_is_post_nominal(w, lexicon) for w in tail if w): + return [], True + text = text[:cut] + # composed, so a letter written as base and mark (`oğlu` decomposed) + # meets the patterns written composed + words = [unicodedata.normalize("NFC", w) + for w in (w.strip(_EDGES) for w in text.split()) if w] + if not words: + return words, False + one_case = is_one_case(words) + while words and _is_leading_title(words[0], lexicon): + del words[0] + while _ends_in_post_nominal(words, lexicon, one_case): + del words[-1] + return words, False + + +def _patronymic(words: list[str]) -> PatronymicRule | None: + """The rule a name's patronymic votes for, where the pipeline's rule + can read it there. rules/O2's four name words, the marker last + (family-first) or third (given-first, which the Turkic rule reads + too), never first or second; rules/O1's three name words, the + ending on the middle word (given-first) or on the last + (`east_slavic_rotates`, the family-first shape the rule restores), + whatever the first word ends in (a surname may end like a + patronymic: `Abramovich Roman Arkadyevich`), and no initial first or in the middle (O1's + precedence). A final full stop is read past.""" + bare = [w.rstrip(".") or w for w in words] + if (len(bare) == 4 and not any(map(is_patronymic_marker, bare[:2])) + and (is_patronymic_marker(bare[2]) + or is_patronymic_marker(bare[3]))): + return PatronymicRule.TURKIC + if (len(bare) == 3 + and not (_is_initialed(words[0]) or _is_initialed(words[1])) + and (has_patronymic_ending(bare[1]) + or east_slavic_rotates(bare[1], bare[2]))): + return PatronymicRule.EAST_SLAVIC + return None + + +def _read(text: str, lexicon: Lexicon + ) -> tuple[dict[Signal, tuple[Role, Role, Role] | PatronymicRule], bool]: + """The votes one name casts, and whether a comma left it out.""" + words, comma = _name_words(text, lexicon) + if len(words) < 2: + return {}, comma + votes: dict[Signal, tuple[Role, Role, Role] | PatronymicRule] = {} + + capitals = _capitals(words) + if capitals is not None: + votes[Signal.CAPITALS] = capitals + + if _is_initialed(words[-1]) and not _is_initialed(words[0]): + votes[Signal.TRAILING_INITIAL] = FAMILY_FIRST + + particle = _particle(words, lexicon) + if particle is not None: + votes[Signal.PARTICLE] = particle + + short_capitals = _ends_in_short_capitals(words) + for signal, table, first_order, last_order, skip_last in ( + (Signal.SURNAME, ROMANIZED_SURNAMES, FAMILY_FIRST, GIVEN_FIRST, + short_capitals), + (Signal.GIVEN_NAME, COMMON_GIVEN_NAMES, GIVEN_FIRST, FAMILY_FIRST, + False)): + at_first, at_last = _at_one_end(words, table, skip_last) + if at_first != at_last: + votes[signal] = first_order if at_first else last_order + + patronymic = _patronymic(words) + if patronymic is not None: + votes[Signal.PATRONYMIC] = patronymic + return votes, False + + +def name_votes(text: str, lexicon: Lexicon + ) -> dict[Signal, tuple[Role, Role, Role] | PatronymicRule]: + """The votes one name casts; empty for a name of fewer than two + words once its titles and post-nominals are read past, and for a + comma name, save one whose comma is followed by post-nominals + alone.""" + return _read(text, lexicon)[0] + + +# -- the group ---------------------------------------------------------- + +#: A vote one signal casts: an order, or the patronymic rule a shape +#: matches. +Vote = tuple[Role, Role, Role] | PatronymicRule +#: A setting's value as SettingEvidence holds it: an order, or a set of +#: patronymic rules. +SettingValue = tuple[Role, Role, Role] | frozenset[PatronymicRule] + +# The four thresholds, chosen with tools/differential/suggest_measure.py +# so that a group written in its base's own order stays right in at +# least 95% of draws at every size; the measurements behind each are in +# decisions/group-suggestion, 2026-11-10 THRESHOLDS. + +#: The fewest names voting for a value that can decide it. +_MIN_VOTES = 2 +#: The share of the names casting an order vote that one end must hold +#: STRICTLY MORE THAN to decide the order: more than two thirds, so a +#: tie, or two names against one, decides nothing. +_MIN_SHARE = 2 / 3 +#: The share of the names casting any vote that one patronymic rule +#: (East Slavic or Turkic) must hold strictly more than to decide it, +#: higher than _MIN_SHARE because a decided East Slavic rule brings a +#: given-first order with it. +_MIN_PATRONYMIC_SHARE = 0.8 +#: The fewest names carrying one patronymic rule that can decide it, +#: besides the share, for the same reason. +_MIN_PATRONYMIC_VOTES = 5 + +_NAME_ORDER = "name_order" +_PATRONYMIC_RULES = "patronymic_rules" +_SIGNAL_RANK = {signal: rank for rank, signal in enumerate(Signal)} + + +def _end(order: tuple[Role, ...]) -> tuple[Role, Role, Role]: + """The end an order writes the family name at, spelled as a vote: + ``FAMILY_FIRST`` for either family-first order.""" + return FAMILY_FIRST if order[0] is Role.FAMILY else GIVEN_FIRST + + +def _order_ends(pairs: Iterable[tuple[Signal, Vote]] + ) -> frozenset[tuple[Role, Role, Role]]: + """The ends a name's order signals voted for: none, one (the name's + order vote) or both (the name is contested and casts no order + vote).""" + return frozenset(vote for signal, vote in pairs # type: ignore[misc] + if signal is not Signal.PATRONYMIC) + + +def _checked_vote(signal: object, vote: object, + owner: str = "NameEvidence") -> tuple[Signal, Vote]: + """One (signal, vote) pair, validated: an order signal votes + ``GIVEN_FIRST`` or ``FAMILY_FIRST``, the patronymic signal a + ``PatronymicRule``. ``owner`` names the type in the message.""" + signal = Signal(signal) # type: ignore[arg-type] # ValueError names it + if signal is Signal.PATRONYMIC: + if not isinstance(vote, PatronymicRule): + raise TypeError( + f"{owner} vote for {signal.value!r} must be a " + f"PatronymicRule, got {vote!r}; pass e.g. " + f"{{Signal.PATRONYMIC: PatronymicRule.EAST_SLAVIC}}") + return signal, vote + if not isinstance(vote, (tuple, list)): + raise TypeError( + f"{owner} vote for {signal.value!r} must be an order, " + f"got {vote!r}; pass GIVEN_FIRST or FAMILY_FIRST") + order = tuple(vote) + for end in (GIVEN_FIRST, FAMILY_FIRST): + if order == end: + # the shared constant, so a vote compares by identity too + return signal, end + raise ValueError( + f"{owner} vote for {signal.value!r} must name an end, " + f"GIVEN_FIRST or FAMILY_FIRST, got {vote!r}; a vote never " + f"spells FAMILY_FIRST_GIVEN_LAST, which is FAMILY_FIRST's end") + + +def _iterable(value: object, owner: str, form: str, example: str + ) -> tuple[object, ...]: + """``value`` as a tuple, refusing a bare str, bytes, a mapping or a + non-iterable, each of which would iterate into the wrong items.""" + if (isinstance(value, (str, bytes, bytearray, Mapping)) + or not isinstance(value, Iterable)): + raise TypeError( + f"{owner} must be {form}, got {type(value).__name__} " + f"{reprlib.repr(value)}; pass e.g. {example}") + return tuple(value) + + +def _checked_row(row: object, setting: str) -> tuple[Signal, Vote, int]: + """One ``by_signal`` row, validated: ``(signal, vote, count)``, its + signal one that votes on ``setting``.""" + if isinstance(row, (str, bytes)) or not isinstance(row, (tuple, list)) \ + or len(row) != 3: + raise TypeError( + f"SettingEvidence by_signal rows must be (signal, vote, count) " + f"triples, got {reprlib.repr(row)}; pass e.g. " + f"(Signal.SURNAME, FAMILY_FIRST, 2)") + signal, vote, count = row + pair = _checked_vote(signal, vote, "SettingEvidence by_signal") + if _setting_of(pair[0]) != setting: + raise ValueError( + f"SettingEvidence by_signal row {reprlib.repr(row)} is a " + f"{pair[0].value!r} vote, which votes on {_setting_of(pair[0])}, " + f"not on {setting}; put it in that setting's evidence") + if isinstance(count, bool) or not isinstance(count, int): + raise TypeError( + f"SettingEvidence by_signal count must be an int count of " + f"votes, got {count!r}; pass e.g. 2") + if count < 0: + raise ValueError( + f"SettingEvidence by_signal count must not be negative, got " + f"{count!r}; a count of votes is 0 or more") + return (*pair, count) + + +#: The settings a SettingEvidence may be about: the PolicyPatch fields, +#: which the report reads off the patch by name. +_SETTINGS = frozenset(f.name for f in dataclasses.fields(PolicyPatch)) + + +def _label(value: object) -> str: + """A vote or a setting's value, as the report and reprs write it.""" + if isinstance(value, PatronymicRule): + return value.value + if isinstance(value, frozenset): + return ", ".join(sorted(_label(v) for v in value)) or "none" + if isinstance(value, tuple): + return _order_repr(value) + return repr(value) + + +_NAME_EVIDENCE_EXAMPLE = ( + "NameEvidence('Kim Minjun', {Signal.SURNAME: FAMILY_FIRST})") + + +@dataclass(frozen=True, slots=True, init=False) +class NameEvidence: + """One input name and the votes it cast, read with the base + parser's lexicon. A name with a comma, no words or one word casts + none, and a name that casts none never disagrees with the group. + Order votes are one per NAME: the end the name's order signals + agree on. A name whose order signals point at both ends is + contested, casts no order vote and never disagrees on order. + ``votes`` keeps every signal's vote all the same. + ``dataclasses.replace`` keeps the votes unless it is passed + ``votes=`` too.""" + + #: The name exactly as the caller passed it. + text: str + # the votes as (Signal, vote) pairs in Signal's order: a tuple, so + # the type stays frozen, hashable and picklable; read it through + # the `votes` property + _votes: tuple[tuple[Signal, Vote], ...] + + __getstate__ = _guarded_getstate + __setstate__ = _guarded_setstate + + def __init__(self, text: str, + votes: Mapping[Signal, Vote] | None = None, *, + _votes: Iterable[tuple[Signal, Vote]] | None = None + ) -> None: + # `_votes` is the stored field's name, which dataclasses.replace + # passes back; `votes`, where given, wins, so + # replace(evidence, votes=...) replaces them + if votes is None and _votes is not None: + votes = dict(_votes) + if not isinstance(text, str): + raise TypeError( + f"NameEvidence text must be a str, got " + f"{type(text).__name__} {reprlib.repr(text)}; pass e.g. " + f"{_NAME_EVIDENCE_EXAMPLE}") + if not isinstance(votes, Mapping): + raise TypeError( + f"NameEvidence votes must be a mapping of Signal to vote, " + f"got {type(votes).__name__} {reprlib.repr(votes)}; pass " + f"e.g. {_NAME_EVIDENCE_EXAMPLE}") + pairs = tuple(sorted( + (_checked_vote(signal, vote) for signal, vote in votes.items()), + key=lambda pair: _SIGNAL_RANK[pair[0]])) + object.__setattr__(self, "text", text) + object.__setattr__(self, "_votes", pairs) + + def __repr__(self) -> str: + votes = "".join(f", {signal.value}={_label(vote)}" + for signal, vote in self._votes) + return f"NameEvidence({self.text!r}{votes})" + + @property + def votes(self) -> Mapping[Signal, Vote]: + """The votes this name cast, keyed by signal; read-only.""" + return MappingProxyType(dict(self._votes)) + + +@dataclass(frozen=True, slots=True) +class SettingEvidence: + """What the group said about one ``Policy`` setting. + + ``value`` is the setting's value under the suggested parser: the + decided value where ``decided`` is true, else the base's. A decided + value equal to the base's is not in the patch, which holds only + changes, but is still ``decided``. The four counts are of NAMES and + add up to the group's size: ``votes_for`` the names whose vote is + for ``value``, ``votes_against`` the names whose vote is for + another value, ``contested`` the names whose signals point both + ways on this setting (and so cast no vote), and ``no_evidence`` the + names with no signal on it. ``by_signal`` breaks the signals' own + votes down as ``(signal, vote, count)`` rows, so a name two signals + read appears in two of them. ``note``, where it is not empty, says + why the setting was decided or left undecided as it was.""" + + #: The ``Policy`` field this evidence is about. + setting: str + #: The field's value under the suggested parser. + value: SettingValue + #: Whether the group's votes decided the value. + decided: bool + votes_for: int + votes_against: int + contested: int + no_evidence: int + by_signal: tuple[tuple[Signal, Vote, int], ...] + note: str = "" + + __getstate__ = _guarded_getstate + __setstate__ = _guarded_setstate + + def __post_init__(self) -> None: + if not isinstance(self.setting, str): + raise TypeError( + f"SettingEvidence setting must be a Policy field name " + f"(str), got {self.setting!r}; pass e.g. 'name_order'") + if self.setting not in _SETTINGS: + raise ValueError( + f"SettingEvidence setting must name a Policy field, got " + f"{self.setting!r}; pass one of " + f"{', '.join(sorted(_SETTINGS))}") + if not isinstance(self.decided, bool): + raise TypeError( + f"SettingEvidence decided must be a bool, got " + f"{self.decided!r}; pass True or False") + for name in ("votes_for", "votes_against", "contested", + "no_evidence"): + count = getattr(self, name) + if isinstance(count, bool) or not isinstance(count, int): + raise TypeError( + f"SettingEvidence {name} must be an int count of " + f"names, got {count!r}; pass e.g. 0") + if count < 0: + raise ValueError( + f"SettingEvidence {name} must not be negative, got " + f"{count!r}; a count of names is 0 or more") + if not isinstance(self.note, str): + raise TypeError( + f"SettingEvidence note must be a str, got {self.note!r}; " + f"pass '' for no note") + rows = _iterable(self.by_signal, "SettingEvidence by_signal", + "an iterable of (signal, vote, count) rows", + "((Signal.SURNAME, FAMILY_FIRST, 2),) or ()") + object.__setattr__( + self, "by_signal", + tuple(_checked_row(row, self.setting) for row in rows)) + + def __repr__(self) -> str: + state = "decided" if self.decided else "undecided" + return (f"SettingEvidence({self.setting}={_label(self.value)}, " + f"{state}, {self.votes_for} for, {self.votes_against} " + f"against, {self.contested} contested, {self.no_evidence} " + f"no evidence)") + + +@dataclass(frozen=True, slots=True) +class Suggestion: + """The settings a group of names suggests: build it with + :meth:`Parser.suggest`. ``patch`` holds only the settings the group + decided and that differ from the base's; ``parser`` is the base + with the patch applied; ``evidence`` holds one + :class:`SettingEvidence` per setting considered, and ``names`` one + :class:`NameEvidence` per input name, in input order, blanks and + duplicates included. ``str()`` is a readable report.""" + + #: The parser ``suggest`` was called on. + base: Parser + #: The settings the group decided that differ from the base's. + patch: PolicyPatch + evidence: tuple[SettingEvidence, ...] + names: tuple[NameEvidence, ...] + + __getstate__ = _guarded_getstate + __setstate__ = _guarded_setstate + + def __post_init__(self) -> None: + # Parser at call time: _parser imports this module at call time + # too, in Parser.suggest + from ._parser import Parser + if not isinstance(self.base, Parser): + raise TypeError( + f"Suggestion base must be a Parser, got " + f"{type(self.base).__name__} {reprlib.repr(self.base)}; " + f"build a Suggestion with parser.suggest(names)") + if not isinstance(self.patch, PolicyPatch): + raise TypeError( + f"Suggestion patch must be a PolicyPatch, got " + f"{reprlib.repr(self.patch)}; pass e.g. PolicyPatch() or " + f"PolicyPatch(name_order=FAMILY_FIRST)") + for name, cls in (("evidence", SettingEvidence), + ("names", NameEvidence)): + items = _iterable( + getattr(self, name), f"Suggestion {name}", + f"a tuple of {cls.__name__}", f"() or a tuple of " + f"{cls.__name__} items") + for item in items: + if not isinstance(item, cls): + raise TypeError( + f"Suggestion {name} must hold {cls.__name__} " + f"items, got {type(item).__name__} " + f"{reprlib.repr(item)}; pass a tuple of " + f"{cls.__name__}, or build a Suggestion with " + f"parser.suggest(names)") + object.__setattr__(self, name, items) + self._check_against_the_patch() + + def __repr__(self) -> str: + # bounded: the patch and counts, never a name + return (f"Suggestion({self.patch!r}, names={len(self.names)}, " + f"disagreeing={len(self.disagreeing())})") + + def __str__(self) -> str: + noun = "name" if len(self.names) == 1 else "names" + lines = [f"Suggested for {len(self.names)} {noun} " + f"(base: {self.base!r}):"] + for e in self.evidence: + if e.decided: + changed = getattr(self.patch, e.setting) is not UNSET + state = _label(e.value) + ("" if changed else " (as the base)") + else: + state = f"unchanged ({_label(e.value)})" + if e.votes_for or e.votes_against or e.contested: + contested = (f"{e.contested} contested, " + if e.contested else "") + counts = (f"{e.votes_for} for, {e.votes_against} against, " + f"{contested}{e.no_evidence} no evidence") + else: + counts = "no evidence" + lines.append(f" {e.setting} {state} {counts}") + for signal, vote, count in e.by_signal: + lines.append(f" {signal.value} {_label(vote)} {count}") + if e.note: + lines.append(f" {e.note}") + against = self.disagreeing() + if against: + shown = ", ".join(n.text for n in against[:5]) + more = f", and {len(against) - 5} more" if len(against) > 5 else "" + lines.append(f" disagreeing ({len(against)}): {shown}{more}") + return "\n".join(lines) + + @property + def parser(self) -> Parser: + """The base parser with :attr:`patch` applied: the same lexicon + and segmenter, and the base's policy for every setting the + group did not change. It is built afresh on each access, so + bind it once (``p = s.parser``) before parsing in a loop.""" + # A new construction site for Parser (AGENTS.md's validation + # bullet): a warning Parser.__post_init__ gives here is passed + # on from the caller's frame only where the base, built the + # same way, does not give it too -- the base's own warnings + # reached its caller when it was built. + def built(policy: Policy + ) -> tuple[Parser, list[warnings.WarningMessage]]: + with warnings.catch_warnings(record=True) as caught: + warnings.simplefilter("always") + parser = dataclasses.replace(self.base, policy=policy) + return parser, caught + + _, before = built(self.base.policy) + parser, after = built(apply_patch(self.base.policy, self.patch)) + seen = {(w.category, str(w.message)) for w in before} + for w in after: + if (w.category, str(w.message)) not in seen: + warnings.warn(w.message, w.category, stacklevel=2) + return parser + + def disagreeing(self) -> tuple[NameEvidence, ...]: + """The names whose vote is against a decided setting's value, + in input order. A setting left undecided has no answer to + disagree with; a name that cast no vote, or whose order signals + are contested, never disagrees on order; and a name carrying an + East Slavic patronymic never disagrees on order where the East + Slavic rule is decided, since the rule reorders it.""" + decided = {e.setting: e.value for e in self.evidence if e.decided} + if not decided: + return () + return tuple(n for n in self.names if _disagrees(n, decided)) + + def _check_against_the_patch(self) -> None: + """The evidence, the patch and the names agree: one evidence + per setting, a decided value the suggested parser's own, an + undecided setting out of the patch, every patch field a decided + change from the base, and every setting's counts adding up to + the names.""" + settings = Counter(e.setting for e in self.evidence) + for setting, count in settings.items(): + if count > 1: + raise ValueError( + f"Suggestion evidence holds {setting!r} more than " + f"once ({count} times); pass one SettingEvidence per " + f"setting") + policy = apply_patch(self.base.policy, self.patch) + decided = set() + size = len(self.names) + for e in self.evidence: + total = (e.votes_for + e.votes_against + e.contested + + e.no_evidence) + if total != size: + raise ValueError( + f"Suggestion evidence for {e.setting} counts names " + f"that add up to {total}, not the {size} names in " + f"names; each name is for, against, contested or of " + f"no evidence on every setting") + if e.decided: + decided.add(e.setting) + if e.value != getattr(policy, e.setting): + raise ValueError( + f"Suggestion's decided {e.setting} value " + f"{_label(e.value)} is not the suggested parser's " + f"{_label(getattr(policy, e.setting))}; set the " + f"patch to the decided value, or leave it out " + f"where the value is the base's") + elif getattr(self.patch, e.setting) is not UNSET: + raise ValueError( + f"Suggestion patch sets {e.setting}, which its " + f"evidence leaves undecided; mark the evidence " + f"decided, or leave {e.setting} out of the patch") + for f in dataclasses.fields(PolicyPatch): + if getattr(self.patch, f.name) is UNSET: + continue + if f.name not in decided: + raise ValueError( + f"Suggestion patch sets {f.name}, which no decided " + f"SettingEvidence holds; add that evidence, or leave " + f"{f.name} out of the patch") + if getattr(policy, f.name) == getattr(self.base.policy, f.name): + raise ValueError( + f"Suggestion patch sets {f.name} to the base's own " + f"value {_label(getattr(policy, f.name))}; a patch " + f"holds only changes, so leave it out and keep the " + f"evidence decided") + + + +def _setting_of(signal: Signal) -> str: + return _PATRONYMIC_RULES if signal is Signal.PATRONYMIC else _NAME_ORDER + + +def _disagrees(name: NameEvidence, decided: Mapping[str, object]) -> bool: + """Whether one name's votes contradict a decided setting.""" + patronymic = dict(name._votes).get(Signal.PATRONYMIC) + rules = decided.get(_PATRONYMIC_RULES) + if rules is not None and patronymic is not None: + if patronymic not in rules: # type: ignore[operator] + return True + order = decided.get(_NAME_ORDER) + if order is None: + return False + if (rules is not None and patronymic is PatronymicRule.EAST_SLAVIC + and patronymic in rules # type: ignore[operator] + and _end(order) == GIVEN_FIRST): # type: ignore[arg-type] + return False + ends = _order_ends(name._votes) + return len(ends) == 1 and next(iter(ends)) != _end(order) # type: ignore[arg-type] + + +class _Tally: + """The group's votes on one setting, one per name, gathered in one + pass. A plain class rather than a dataclass: it is never pickled or + returned.""" + + __slots__ = ("names", "contested", "by_name", "by_signal") + + def __init__(self) -> None: + self.names = 0 + self.contested = 0 + #: names per value voted for + self.by_name: Counter[Vote] = Counter() + #: signal votes per (signal, value) + self.by_signal: Counter[tuple[Signal, Vote]] = Counter() + + def add(self, pairs: list[tuple[Signal, Vote]]) -> None: + """One name's votes on this setting: its signals' votes, and + one name vote where they agree.""" + self.names += 1 + for pair in pairs: + self.by_signal[pair] += 1 + values = {vote for _, vote in pairs} + if len(values) == 1: + self.by_name[values.pop()] += 1 + elif values: + self.contested += 1 + + def winner(self) -> Vote | None: + """The value whose names are at least _MIN_VOTES and strictly + more than _MIN_SHARE of the names that cast a vote, if one + is.""" + counts = self.by_name + total = sum(counts.values()) + if not counts: + return None + vote, count = max(counts.items(), key=lambda item: item[1]) + if count < _MIN_VOTES or count <= _MIN_SHARE * total: + return None + return vote + + def evidence(self, setting: str, value: SettingValue, decided: bool, + is_for: Callable[[Vote], bool], note: str = "" + ) -> SettingEvidence: + """Evidence for ``value``; ``is_for`` says which name votes are + for it.""" + counts = self.by_name + votes_for = sum(c for v, c in counts.items() if is_for(v)) + rows = tuple(sorted( + ((signal, vote, count) + for (signal, vote), count in self.by_signal.items()), + key=lambda row: (_SIGNAL_RANK[row[0]], _label(row[1])))) + voters = sum(counts.values()) + return SettingEvidence( + setting, value, decided, votes_for, voters - votes_for, + self.contested, self.names - voters - self.contested, rows, + note) + + +def _patronymic_winner(tally: _Tally, any_voters: int + ) -> PatronymicRule | None: + """The rule whose names are strictly more than _MIN_PATRONYMIC_SHARE + of the names that cast any vote, and at least _MIN_PATRONYMIC_VOTES + of them. A + name whose order signals are contested still counts among the names + that cast a vote: it carries evidence, and leaving it out moved no + measured group by more than a point. A name casts at most one + patronymic vote, so its votes count names: a share of the group, + not of the patronymic votes alone, so one patronymic among many + given-first names decides nothing.""" + counts = tally.by_name + if not counts: + return None + rule, count = max(counts.items(), key=lambda item: item[1]) + if (count < _MIN_PATRONYMIC_VOTES + or count <= _MIN_PATRONYMIC_SHARE * any_voters): + return None + return rule # type: ignore[return-value] + + +def _decide_patronymic(tally: _Tally, base_rules: frozenset[PatronymicRule], + any_voters: int + ) -> tuple[SettingEvidence, dict[str, object], bool]: + """Decide ``patronymic_rules``: the evidence, the patch fields, and + whether the decided rule needs a given-first order.""" + rule = _patronymic_winner(tally, any_voters) + if rule is PatronymicRule.EAST_SLAVIC: + # the rule reorders a family-first East Slavic name itself, and + # stands down under a declared family-first order, so the group + # gets the rule with a given-first order (what locales.RU ships) + value = base_rules | {rule} + fields: dict[str, object] = ( + {} if rule in base_rules else {_PATRONYMIC_RULES: frozenset({rule})}) + return (tally.evidence(_PATRONYMIC_RULES, value, True, + lambda v: v in value), fields, True) + if rule is PatronymicRule.TURKIC: + return _decide_turkic(tally, base_rules) + return (tally.evidence(_PATRONYMIC_RULES, base_rules, False, + lambda v: v in base_rules), {}, False) + + +def _decide_turkic(tally: _Tally, base_rules: frozenset[PatronymicRule] + ) -> tuple[SettingEvidence, dict[str, object], bool]: + """A group whose patronymics are Turkic gets the rule alone. Unlike + the East Slavic rule it reads a name written either way round + (``Mamedov Ali Huseyn oglu`` and ``Ali Huseyn oglu Mamedov`` both + read family Mamedov), so it forces no order: the order is decided + from the names' own votes (decisions/group-suggestion, 2026-11-10 + TURKIC).""" + rule = PatronymicRule.TURKIC + value = base_rules | {rule} + fields: dict[str, object] = ( + {} if rule in base_rules else {_PATRONYMIC_RULES: frozenset({rule})}) + return (tally.evidence(_PATRONYMIC_RULES, value, True, + lambda v: v in value), fields, False) + + +def _comma_note(left_out: int) -> str: + if not left_out: + return "" + noun = "name with a comma was" if left_out == 1 else "names with a comma were" + return (f"{left_out} {noun} left out: a comma states its own order, " + f"so only a comma followed by post-nominals alone is read past") + + +def _decide_order(tally: _Tally, base_order: tuple[Role, Role, Role], + rule_needs_given_first: bool, comma_left_out: int + ) -> tuple[SettingEvidence, dict[str, object]]: + """Decide ``name_order`` in either direction. A family-first answer + keeps the base's family-first spelling where the base has one. A + decided East Slavic rule decides the order as given-first, since + the rule reorders each name and stands down under a family-first + order, unless the names' own votes decide family-first: a decided + order is never overridden, and the rule then stands down.""" + notes = [_comma_note(comma_left_out)] + end = tally.winner() + if rule_needs_given_first and end != FAMILY_FIRST: + fields: dict[str, object] = ( + {} if base_order == GIVEN_FIRST else {_NAME_ORDER: GIVEN_FIRST}) + notes.insert(0, "the East Slavic rule reorders each name; it needs " + "a given-first order") + return (tally.evidence( + _NAME_ORDER, GIVEN_FIRST, True, lambda v: v == GIVEN_FIRST, + note="; ".join(n for n in notes if n)), fields) + if rule_needs_given_first: + notes.insert(0, "the East Slavic rule is decided too, but the names' " + "order votes decide a family-first order, under " + "which the rule stands down") + note = "; ".join(n for n in notes if n) + if end is None: + return (tally.evidence(_NAME_ORDER, base_order, False, + lambda v: v == _end(base_order), note), {}) + # the order tally holds orders alone + order = base_order if _end(base_order) == end else cast( + "tuple[Role, Role, Role]", end) + fields = ( + {} if order == base_order else {_NAME_ORDER: order}) + return (tally.evidence(_NAME_ORDER, order, True, + lambda v: v == end, note), fields) + + +def _texts(names: Iterable[str]) -> Iterator[str]: + """The names, checked as they are consumed: one pass, so a + generator is read once.""" + example = 'parser.suggest(["Priem Jason", "Kim Minjun"])' + if isinstance(names, str): + raise TypeError( + f"suggest() takes an iterable of names, not a bare string: " + f"{names!r}. Wrap one name in a list: " + f"parser.suggest([{names!r}])") + if isinstance(names, (bytes, bytearray, memoryview)): + raise TypeError( + f"suggest() takes an iterable of str, not " + f"{type(names).__name__} -- decode first, e.g. " + f"raw.decode('utf-8')") + if isinstance(names, Mapping): + raise TypeError( + f"suggest() takes an iterable of names, not a mapping " + f"({type(names).__name__}). A mapping yields only its keys; " + f"pass the names themselves, e.g. list(mapping) or " + f"list(mapping.values()): {example}") + try: + iterator = iter(names) + except TypeError: + raise TypeError( + f"suggest() takes an iterable of names, got {names!r}: " + f"{example}") from None + for index, text in enumerate(iterator): + if not isinstance(text, str): + raise TypeError( + f"suggest() takes an iterable of str; item {index} is " + f"{text!r} ({type(text).__name__}). Pass each name as a " + f"string: {example}") + yield text + + +def suggest(parser: Parser, names: Iterable[str]) -> Suggestion: + """The settings a group of names suggests, with ``parser`` as the + base: see :meth:`Parser.suggest`.""" + tallies = {_NAME_ORDER: _Tally(), _PATRONYMIC_RULES: _Tally()} + evidence: list[NameEvidence] = [] + any_voters = comma_left_out = 0 + for text in _texts(names): + votes, comma = _read(text, parser.lexicon) + evidence.append(NameEvidence(text, votes)) + any_voters += bool(votes) + comma_left_out += comma + per_setting: dict[str, list[tuple[Signal, Vote]]] = { + setting: [] for setting in tallies} + for signal, vote in votes.items(): + per_setting[_setting_of(signal)].append((signal, vote)) + for setting, tally in tallies.items(): + tally.add(per_setting[setting]) + policy = parser.policy + patronymic, patronymic_fields, settled = _decide_patronymic( + tallies[_PATRONYMIC_RULES], policy.patronymic_rules, any_voters) + order, order_fields = _decide_order( + tallies[_NAME_ORDER], policy.name_order, settled, comma_left_out) + patch = PolicyPatch(**order_fields, **patronymic_fields) # type: ignore[arg-type] + return Suggestion(parser, patch, (order, patronymic), tuple(evidence)) diff --git a/nameparser/config/_order_names.py b/nameparser/config/_order_names.py new file mode 100644 index 000000000..d27fd6492 --- /dev/null +++ b/nameparser/config/_order_names.py @@ -0,0 +1,649 @@ +"""Romanized Chinese, Korean and Vietnamese surnames, and common given +names, for the group +suggester only (``Parser.suggest``); no parse reads this module. + +Each name is read on its own: a listed surname at one end of the name +and not at the other is evidence that the name writes the family name +at that end, and a listed given name at one end and not at the other +that it writes the given name there; the group decides from the names' +votes. Every entry is FOLDED (``nameparser._suggest.fold``: lower case, no +diacritics, the letters NFKD leaves whole as a keyboard writes them, +``ł`` as ``l``, ``ø`` as ``o``, ``ı`` as ``i``, ``æ`` as ``ae``, ``œ`` +as ``oe``, ``ß`` as ``ss``, ``đ`` as ``d``, and ``ü`` as ``v``) and has +two or more letters. + +Provenance and licences: + +- ``_PINYIN`` (generated below): the Hanyu Pinyin of every + single-character surname ``locales.ZH`` lists, from the first + ``kMandarin`` reading in the Unicode Consortium's Unihan database + (Unicode License v3), tones dropped. See tools/vocab/order_names.py. +- ``_ZH_OTHER_SPELLINGS``, ``_KO_SPELLINGS`` and ``_VI_SURNAMES``: + hand-curated common romanizations (Wade-Giles, Cantonese, Hokkien and + Southeast Asian Chinese spellings; Korean spellings beside the Revised + Romanization; Vietnamese surnames with diacritics folded), assembled + for this feature on 2026-11-10 and not checked against any published + ranking or register. + +- ``_COMMON_GIVEN`` (generated below, ``COMMON_GIVEN_NAMES``): of each + of nam_dict's country columns (Jörg Michael, 2008; GNU FDL 1.2, a + derived table approved on 2026-11-10), the 500 highest-band given + names that pass the filter: written in the family-name box of ORCID's + public data file (CC0) at most 5% of the 20 or more times its holders + wrote it, and not a surname this table lists. See + tools/vocab/order_names.py. + +The given-name table excludes every surname listed here, so the two +tables never share a spelling (the tool refuses a run where they do). +Only the surname lists share spellings, with each other: ``lee`` and +``lim`` are written in Chinese and in Korean, ``ho`` and ``tang`` in +Chinese and in Vietnamese, ``ha`` in Korean and in Vietnamese, and many +hand-listed spellings are pinyin too. The tables list spellings, not +languages. +""" +from __future__ import annotations + +_ZH_OTHER_SPELLINGS = ( + "chan", "wong", "leung", "cheung", "lau", "ng", "chow", "chau", "tsang", + "tse", "yip", "lai", "kwok", "fung", "lam", "mak", "mok", "chiu", "ho", + "hui", "kwan", "tam", "tang", "poon", "siu", "cheng", "yeung", "yuen", + "choi", "sze", "to", "au", "law", "kong", "lo", "chang", "chen", "lin", + "huang", "hsu", "hsieh", "tsai", "liu", "chung", "kuo", "hung", "chiang", + "tseng", "chou", "yeh", "lu", "su", "tu", "wu", "chao", "kao", "chu", + "hsiao", "fu", "sun", "ma", "kang", "tan", "teo", "lim", "goh", "ong", + "koh", "tay", "chia", "lee", "yap", +) + +_KO_SPELLINGS = ( + "kim", "lee", "yi", "rhee", "rhie", "park", "pak", "bak", "choi", "choe", + "chae", "jung", "jeong", "chung", "chong", "kang", "cho", "jo", "yoon", + "yun", "jang", "chang", "lim", "im", "han", "oh", "seo", "suh", "shin", + "sin", "kwon", "gwon", "hwang", "ahn", "an", "song", "yoo", "yu", "ryu", + "hong", "jeon", "jun", "chun", "ko", "go", "moon", "mun", "yang", "son", + "sohn", "bae", "baek", "paik", "heo", "hur", "huh", "nam", "noh", "roh", + "ha", "kwak", "gwak", "sung", "seong", "cha", "joo", "ju", "woo", "wu", + "koo", "ku", "min", "jin", "na", "ji", "eom", "um", "uhm", "won", +) + +_VI_SURNAMES = ( + "nguyen", "tran", "le", "pham", "hoang", "huynh", "phan", "vu", "vo", + "dang", "bui", "do", "ho", "ngo", "duong", "ly", "dinh", "trinh", "mai", + "luong", "lam", "truong", "cao", "luu", "doan", "vuong", "ta", "thai", + "tang", "quach", "la", "ha", "kieu", "ton", +) + +# -- BEGIN generated: PINYIN; edit tools/vocab/order_names.py, not this block -- +_PINYIN = ( + "ai", "an", "ao", "ba", "bai", "bao", "bi", "bian", + "bing", "bo", "cai", "cao", "cen", "ceng", "cha", "chai", + "chang", "chao", "che", "chen", "cheng", "chi", "chou", "chu", + "cong", "cui", "dai", "dan", "dang", "deng", "di", "diao", + "ding", "dong", "dou", "du", "duan", "fan", "fang", "fei", + "feng", "fu", "gai", "gan", "gao", "ge", "geng", "gong", + "gu", "guan", "gui", "guo", "han", "hao", "he", "hong", + "hou", "hu", "hua", "huan", "huang", "hui", "huo", "ji", + "jia", "jian", "jiang", "jiao", "jie", "jin", "jing", "ju", + "kang", "ke", "kong", "kou", "kuang", "lai", "lan", "lang", + "lao", "le", "lei", "leng", "li", "lian", "liang", "liao", + "lin", "ling", "liu", "long", "lou", "lu", "luan", "luo", + "lv", "ma", "mai", "mao", "mei", "meng", "mi", "miao", + "min", "ming", "mo", "mou", "mu", "nan", "ni", "nie", + "ning", "niu", "nong", "ou", "pan", "pang", "pei", "peng", + "ping", "pu", "qi", "qian", "qiang", "qiao", "qin", "qiu", + "qu", "quan", "que", "ran", "rao", "ren", "rong", "ru", + "ruan", "sang", "sha", "shang", "shao", "she", "shen", "sheng", + "shi", "shu", "si", "song", "su", "sui", "sun", "suo", + "tan", "tang", "tao", "teng", "tian", "tong", "tu", "wan", + "wang", "wei", "wen", "weng", "wu", "xi", "xia", "xiang", + "xiao", "xie", "xin", "xing", "xiong", "xu", "xuan", "xue", + "yan", "yang", "yao", "ye", "yi", "yin", "ying", "yong", + "you", "yu", "yuan", "yue", "yun", "zang", "zhan", "zhang", + "zhao", "zhen", "zheng", "zhi", "zhong", "zhou", "zhu", "zhuang", + "zhuo", "zi", "zong", "zou", "zu", "zuo", +) +# -- END generated: PINYIN -- + +ROMANIZED_SURNAMES: frozenset[str] = frozenset( + word + for word in (*_PINYIN, *_ZH_OTHER_SPELLINGS, *_KO_SPELLINGS, *_VI_SURNAMES) + if len(word) >= 2) + +# -- BEGIN generated: COMMON_GIVEN; edit tools/vocab/order_names.py, not this block -- +_COMMON_GIVEN = ( + "aaron", "aarti", "abbie", "abeer", "abelardo", "abhay", "abigael", "abigail", + "achilleas", "adalberto", "adarsh", "adelaida", "adelaide", "adele", "adelina", "aderito", + "adis", "aditi", "admir", "adolfo", "adria", "adriaan", "adrian", "adriana", + "adriane", "adrianna", "adrien", "adrienn", "adrienne", "adrijana", "afaf", "afag", + "afet", "afroditi", "afsaneh", "afshin", "agata", "agatha", "agne", "agnes", + "agnese", "agneta", "agnieszka", "agostina", "agris", "agron", "ahlam", "ahmet", + "aida", "aidan", "aidana", "aigerim", "aigul", "aija", "aikaterini", "aileen", + "ailin", "aiman", "aimee", "ainagul", "ainash", "aine", "ainhoa", "aino", + "ainur", "ainura", "aisha", "aishah", "aisling", "aitor", "aivars", "aizhan", + "ajay", "ajda", "ajit", "ajla", "akbota", "akemi", "akhila", "akihiko", + "akihiro", "akiko", "akinori", "akira", "akmaral", "akos", "aksana", "alan", + "alanna", "alastair", "albena", "albertina", "alberto", "alcides", "aldi", "aldo", + "aldona", "alec", "aleida", "alejandra", "alejandrina", "alejandro", "aleksa", "aleksandar", + "aleksander", "aleksandr", "aleksandra", "aleksandras", "aleksandre", "aleksandrs", "aleksei", "aleksejs", + "aleksey", "aleksi", "alena", "alenka", "ales", "alesia", "alessandra", "alessandro", + "alessia", "alex", "alexander", "alexandr", "alexandra", "alexandros", "alexei", "alexey", + "alexi", "alexia", "alexios", "alexis", "aleyna", "alfiya", "alfonsina", "alfredo", + "algimantas", "algirdas", "aliaksandr", "aliaksei", "alibek", "alica", "alice", "alicia", + "alicja", "alida", "aliki", "alina", "aline", "alisa", "alise", "alisha", + "alisher", "alison", "alisson", "alistair", "alix", "aliya", "aliz", "aliza", + "aljaz", "aljosa", "allyson", "alma", "almudena", "alois", "alok", "alona", + "alperen", "altagracia", "altynai", "altynay", "alvaro", "alvin", "alvina", "alyona", + "alyssa", "alzbeta", "amaia", "amal", "amalie", "amanda", "amandine", "amarilis", + "amaury", "amber", "amelie", "amila", "amilcar", "aminah", "aminata", "amnon", + "amrit", "amrita", "amy", "ana", "anabel", "anabela", "anabella", "anahit", + "anahita", "anais", "analia", "anamaria", "anamarija", "anamika", "anara", "anas", + "anastacia", "anastasia", "anastasiia", "anastasija", "anastasios", "anastasiya", "anastassiya", "anat", + "anatoli", "anatolii", "anatoliy", "anatoly", "andela", "ander", "anders", "andraz", + "andrea", "andreas", "andreia", "andreina", "andrej", "andreja", "andrejs", "andres", + "andrew", "andrey", "andrii", "andrija", "andrijana", "andris", "andrius", "andriy", + "andro", "androniki", "andrzej", "andy", "ane", "anel", "anela", "anele", + "anelia", "aneliya", "aneta", "anete", "anett", "anette", "anezka", "angela", + "angelica", "angelie", "angelika", "angeliki", "angelina", "angeline", "angelique", "angelita", + "angelos", "angely", "angie", "anibal", "aniko", "anil", "anila", "anita", + "anitha", "anja", "anjali", "anjan", "anke", "ankit", "ann", "anna", + "annabelle", "annalisa", "annamaria", "anne", "anne-marie", "anneke", "anneli", "annelie", + "annelies", "annemarie", "annemieke", "annette", "anni", "annick", "annie", "anniina", + "annika", "annukka", "anny", "anouk", "anselm", "anssi", "antanas", "antero", + "anthea", "anthi", "anthony", "antigoni", "antje", "antoaneta", "antoinette", "antonela", + "antonella", "antoni", "antonia", "antonieta", "antonietta", "antonija", "antonina", "antonino", + "antonios", "antonis", "antonius", "antti", "anu", "anuja", "anupam", "anupama", + "anush", "anushka", "anzhela", "anzhelika", "aoife", "aparna", "apostolos", "appolinaire", + "aquiles", "araceli", "aracelis", "aramis", "arantxa", "aranzazu", "arash", "arben", + "archana", "ardak", "ardita", "argentina", "argyro", "ariadna", "ariana", "ariane", + "arianna", "arianne", "ariel", "ariela", "arielle", "aristeidis", "aristides", "aristotelis", + "arjan", "arjen", "arjun", "arkadiusz", "arlen", "arlene", "arlette", "arlinda", + "arlindo", "armando", "armen", "armin", "arminda", "armine", "arne", "arni", + "arnoldo", "arpine", "arpita", "arsen", "arta", "artem", "artemio", "artemis", + "arti", "artur", "arturas", "arturo", "arturs", "artyom", "arun", "arunas", + "arvid", "arvind", "arvydas", "arwa", "arzu", "asa", "asbjorn", "asdrubal", + "ase", "asel", "ashish", "ashleigh", "ashley", "ashot", "ashutosh", "asier", + "asiya", "askhat", "aslihan", "asma", "asmaa", "aspasia", "assel", "assemgul", + "assia", "assiya", "asta", "astghik", "astri", "astrid", "asuman", "asya", + "atanas", "athanasia", "athanasios", "athena", "athina", "atilio", "atle", "atsuko", + "atsushi", "attila", "attilio", "atul", "audrey", "audrius", "audrone", "augustina", + "aura", "aurelie", "aurelien", "aurelija", "aurora", "aurore", "ausra", "ava", + "avinash", "awatif", "axel", "axelle", "aya", "ayako", "aybvke", "ayda", + "aydan", "ayelet", "ayesha", "aygul", "ayla", "aylin", "aymeric", "aynur", + "aynura", "ayse", "aysegvl", "aysel", "aysenur", "aysha", "aysu", "aytan", + "ayumi", "azamat", "azat", "azin", "aziza", "azra", "azucena", "bahareh", + "baiba", "balzhan", "barbara", "barbora", "bart", "bartlomiej", "bartosz", "basmah", + "bastiaan", "batoul", "batuhan", "bayan", "beat", "beata", "beate", "beatrice", + "beatrix", "beatriz", "begona", "begvm", "bekzod", "belen", "belinda", "belkis", + "belma", "benedetta", "benedicta", "bengt", "benjamin", "bente", "berenice", "berfin", + "berit", "berkay", "bernadette", "bernarda", "bernardita", "bernd", "bernice", "bertha", + "beryl", "besnik", "beth", "bethany", "betina", "betsy", "bettina", "betty", + "betvl", "beverley", "beverly", "beyza", "bharat", "bianca", "bianka", "bibiana", + "bienvenido", "bikash", "bikram", "biljana", "bilyana", "birger", "birgit", "birgitta", + "birgitte", "birte", "birute", "bjarke", "bjarne", "bjarte", "bjorg", "bjorn", + "bjornar", "bladimir", "blanca", "blandine", "blanka", "blazenka", "blerim", "blerina", + "blerta", "bobur", "bodil", "bogdana", "boguslaw", "bojan", "bojana", "bonnie", + "boris", "borislav", "borislava", "borut", "boryana", "bostjan", "botagoz", "botir", + "botond", "bozena", "bozhidar", "bozidar", "brandon", "brandy", "branimir", "branislav", + "branislava", "branka", "brankica", "branko", "bratislav", "braulio", "brenda", "brendan", + "brendon", "brian", "briana", "brianna", "bridget", "bridgette", "brigita", "brigitta", + "brigitte", "britney", "britta", "brittany", "bronwyn", "brunilda", "bugra", "bujar", + "buket", "burcin", "burcu", "bvsra", "byeong", "byung", "cagla", "cagri", + "caio", "caitlin", "caitriona", "caleb", "callum", "camil", "camila", "camilla", + "camille", "canan", "candice", "candida", "cansu", "caoimhe", "caren", "caridad", + "carin", "carina", "carissa", "carl", "carla", "carlos", "carlota", "carlotta", + "carly", "carme", "carmela", "carmen", "carmina", "carol", "carola", "carolin", + "carolina", "caroline", "carolyn", "carsten", "cassandra", "catalina", "catarina", "caterina", + "cathal", "catharina", "catherine", "cathrine", "cathy", "catia", "catriona", "cecile", + "cecilia", "cecilie", "celeste", "celia", "celina", "celine", "celso", "cem", + "cemal", "cemil", "cemre", "cenk", "cesar", "ceyda", "cezary", "chad", + "chantelle", "charalambos", "charalampos", "charbel", "charikleia", "charis", "charisse", "charity", + "charlene", "charlie", "charlotta", "charlotte", "charly", "charmaine", "chelsea", "cheol", + "cheryl", "chiara", "chihiro", "chinara", "chloe", "chris", "christa", "christel", + "christer", "christiaan", "christian", "christiana", "christiane", "christin", "christina", "christine", + "christodoulos", "christof", "christoffer", "christoforos", "christoph", "christopher", "christos", "chrysanthi", + "chrysoula", "chul", "ciara", "ciaran", "cindy", "cinthia", "cinthya", "cintia", + "cinzia", "claire", "clara", "clarence", "clarice", "clarisa", "clarissa", "clarisse", + "claudia", "claudio", "claudiu", "clelia", "clemencia", "cleopatra", "clint", "clodagh", + "clotilde", "clyde", "codruta", "coline", "colleen", "colm", "concetta", "connie", + "conor", "constance", "constantina", "constanza", "consuelo", "coralia", "coralie", "corentin", + "corinna", "corinne", "cormac", "cornelia", "cosimo", "cosmina", "crina", "cris", + "cristel", "cristian", "cristiana", "cristina", "cristine", "cristofer", "cristy", "crystal", + "csenge", "cynthia", "cyrus", "daan", "dace", "dae", "dafna", "dafne", + "dagmar", "dagmara", "dagoberto", "dahlia", "daiana", "daichi", "daiki", "dainius", + "daisuke", "daisy", "daiva", "dajana", "dalia", "dalibor", "dalila", "dalius", + "dalma", "damaris", "damien", "damir", "damjan", "damla", "dana", "danah", + "danai", "danguole", "dania", "danica", "daniel", "daniela", "daniele", "daniella", + "danielle", "daniil", "danijel", "danijela", "danil", "danilo", "daniyar", "danka", + "danny", "dante", "danuta", "danute", "dany", "danylo", "daphna", "daphne", + "dareen", "daria", "darien", "dariga", "darija", "darin", "darina", "darinka", + "dario", "darius", "dariush", "dariusz", "dariya", "darja", "darkhan", "darlene", + "darren", "darryl", "darwin", "darya", "daryl", "daryna", "dauren", "davide", + "davit", "davor", "davron", "dawid", "dawn", "dayana", "deanna", "debbie", + "debby", "debora", "deborah", "debra", "declan", "deema", "deepa", "deepika", + "deepti", "deirdre", "dejan", "dejana", "delfina", "delphine", "deni", "denisa", + "denise", "derek", "dermot", "derya", "desiree", "desislava", "despina", "despoina", + "dessislava", "devesh", "devon", "deyan", "diala", "diana", "dianne", "didem", + "diego", "dieter", "dietmar", "digna", "dijana", "dilan", "dilara", "dilbar", + "dildora", "dilfuza", "dilmurod", "dilnoza", "dilyana", "dilyara", "dimitar", "dimitra", + "dimitrije", "dimitrios", "dimitris", "dimosthenis", "dina", "dinah", "dinara", "dinesh", + "dinora", "dinorah", "dionysios", "dipak", "dirk", "ditte", "divya", "diyana", + "dmitri", "dmitrii", "dmitrijs", "dmitriy", "dmitry", "dmytro", "dogukan", "dolly", + "domagoj", "domenica", "dominga", "dominik", "dominika", "dominique", "dominykas", "donal", + "donata", "donatas", "donatella", "donna", "dora", "dorde", "doreen", "dorel", + "dorian", "dorien", "dorine", "doris", "dorit", "dorota", "dorotea", "dorothea", + "dorothy", "dorottya", "dorte", "dorthe", "doug", "dov", "dragana", "dragica", + "drazen", "dubravka", "dubravko", "dulce", "dunia", "dunja", "dusan", "dusanka", + "dusica", "dusko", "dustin", "duygu", "dylan", "dzmitry", "eamon", "eamonn", + "ebenezer", "ebru", "ecem", "eda", "edda", "eddie", "edelmira", "edgar", + "edgardo", "edgars", "edin", "edina", "edison", "edit", "edita", "edith", + "edmund", "edmundo", "edna", "edoardo", "edson", "eduard", "eduarda", "eduardo", + "edvard", "edvin", "edward", "edwin", "edyta", "eero", "eetu", "eeva", + "efrain", "efstathios", "efstratios", "efthymia", "efthymios", "eftychia", "egidijus", "egil", + "egle", "egor", "ehud", "eija", "eiji", "eike", "eileen", "eimear", + "einar", "eirik", "eirini", "eivind", "ekaterina", "ekaterine", "eladio", "elaheh", + "elaine", "elchin", "elda", "eldad", "eldor", "eleana", "eleanor", "eleftheria", + "eleftherios", "elena", "elene", "eleni", "eleonora", "eleonore", "elham", "eli", + "eliana", "eliane", "elida", "eliel", "elif", "eligio", "elin", "elina", + "eline", "elinor", "elio", "eliot", "elisa", "elisabet", "elisabeta", "elisabete", + "elisabeth", "elisabetta", "elisavet", "elise", "eliseo", "eliska", "elissa", "elita", + "eliza", "elizabeta", "elizabete", "elizabeth", "elizaveta", "elke", "ellen", "ellie", + "elly", "elmar", "elmer", "elmira", "elnara", "elodie", "eloisa", "eloise", + "elona", "elpida", "elsa", "else", "elshad", "elshan", "elsie", "elsy", + "elva", "elvan", "elvia", "elvin", "elvina", "elvira", "elvis", "elyse", + "elza", "elzbieta", "ema", "eman", "emanuel", "emanuela", "emanuele", "emel", + "emelie", "emeline", "emely", "emer", "emese", "emiel", "emil", "emilia", + "emiliana", "emiliano", "emilie", "emilija", "emilio", "emiliya", "emily", "emina", + "emine", "emir", "emma", "emmanouil", "emmanuelle", "emmy", "emperatriz", "emrah", + "emre", "eneida", "enes", "enid", "enis", "enric", "enrica", "enrico", + "enrique", "enriqueta", "enver", "enzo", "eoghan", "eoin", "eran", "erasmo", + "eri", "erica", "erich", "erick", "ericka", "erik", "erika", "erin", + "erkki", "erlend", "erling", "ermelinda", "ernestas", "ernestina", "ernesto", "ersin", + "erwin", "esben", "esko", "esma", "esmeralda", "esmira", "espen", "esra", + "essi", "estefania", "estelle", "ester", "esther", "eszter", "ethan", "ethel", + "ettore", "euclides", "eudes", "eugen", "eugenia", "eulalia", "eunice", "eva", + "evaldas", "evan", "evandro", "evangelia", "evangelina", "evangeline", "evangelos", "evanthia", + "evdokia", "eve", "evelien", "eveliina", "evelin", "evelina", "eveline", "evelio", + "evelyn", "evelyne", "everardo", "evgeni", "evgenia", "evgenii", "evgeniya", "evgeny", + "evi", "evie", "evija", "evita", "evy", "ewa", "ewelina", "eyad", + "eyvp", "ezequiel", "ezgi", "fabiana", "fabienne", "fabio", "fabiola", "fabricio", + "fabrizio", "fadi", "fadia", "fadwa", "fady", "faith", "faiza", "fanni", + "fanny", "fany", "farangiz", "farhod", "fariba", "farideh", "farnaz", "farrux", + "farshid", "fatemeh", "faten", "fatimah", "fatin", "fatma", "faustina", "fawzia", + "fayza", "federica", "federico", "felicia", "felicitas", "felicity", "felipe", "femke", + "feras", "ferdinand", "fereshteh", "feride", "fernanda", "feruza", "feyza", "fikret", + "filipa", "filippos", "filomena", "fiona", "fiorella", "firas", "firuz", "firuza", + "flavia", "flavie", "flavio", "flora", "florencia", "florinda", "forough", "foteini", + "fotini", "fotios", "fran", "francesc", "francesca", "francesco", "francisca", "franciska", + "franciszek", "franjo", "frankie", "frantisek", "franziska", "freddie", "freddy", "frederic", + "frederick", "frederik", "frederikke", "frederique", "fredrik", "fredy", "freek", "freja", + "freya", "frida", "frieda", "friederike", "frode", "fruzsina", "fuat", "fulvio", + "fulya", "furkan", "gabriela", "gabriele", "gabriella", "gabrielle", "gabrijela", "gaelle", + "gagik", "galia", "galit", "galiya", "galya", "galyna", "gamze", "gareth", + "garima", "gary", "gaukhar", "gauri", "gayane", "gediminas", "geert", "geeta", + "geir", "gema", "gemma", "genevieve", "gennady", "genoveva", "geoff", "geoffrey", + "georg", "georgette", "georgi", "georgia", "georgiana", "georgii", "georgina", "georgios", + "georgiy", "georgy", "gerald", "geraldine", "gerardo", "gerasimos", "gerben", "gerd", + "gerda", "gergana", "gergo", "gernot", "gerrit", "gerson", "gert", "gertrudis", + "gessica", "gevorg", "ghada", "ghadah", "ghina", "giada", "gian", "giancarlo", + "gianfranco", "gianluca", "gianmarco", "gianna", "giannina", "giannis", "giedre", "giedrius", + "gilberto", "gillian", "gilmar", "gina", "gino", "gintaras", "gintare", "gintautas", + "gioconda", "giorgia", "giovanna", "giovanni", "gisela", "gisele", "gisella", "giselle", + "gita", "gitte", "giulia", "giuliana", "giuseppe", "giuseppina", "gizem", "gladis", + "gladys", "gleb", "glenda", "glenn", "gloria", "gloriana", "gokhan", "gonzalo", + "goran", "gordana", "gorkem", "gozde", "grace", "graciela", "graeme", "grainne", + "graziella", "grazyna", "greg", "gregoria", "greta", "gretchen", "grete", "gretel", + "grethe", "grigorios", "grigoriy", "grigory", "griselda", "gro", "grzegorz", "guadalupe", + "gudrun", "guillermo", "gulbanu", "gulnara", "gulnaz", "gulnur", "gulshat", "gulzat", + "gulzhan", "gunel", "gunhild", "gunilla", "gunjan", "gunnar", "guro", "gustaf", + "gustav", "gustavo", "gvantsa", "gvlcan", "gvlsah", "gvlsvm", "gwen", "gwladys", + "gwyneth", "gyu", "gyula", "hacer", "hadil", "hae", "hafiza", "hagit", + "haifa", "haitham", "hajime", "hakan", "hakon", "hala", "haleema", "halina", + "halvor", "hamish", "hamlet", "hana", "hanaa", "hanadi", "hanae", "hanan", + "handan", "hania", "hanim", "hannah", "hanne", "hanneke", "hannele", "hannes", + "hannu", "hans", "harald", "harold", "harri", "harriet", "hartmut", "haruka", + "haruki", "harutyun", "hasmik", "hatice", "havard", "havva", "haya", "hayam", + "hayato", "hayk", "hayley", "hazal", "hazel", "heather", "heba", "hector", + "hedda", "hedvig", "hee", "hege", "hei", "heidi", "heidy", "heike", + "heikki", "heiko", "heini", "helder", "heleen", "helen", "helena", "helene", + "helga", "helge", "helgi", "heli", "helia", "hellen", "helmut", "heloisa", + "heloise", "hemant", "hendrik", "henrieta", "henrietta", "henriette", "henrik", "henryk", + "heriberto", "hermine", "herminia", "herminio", "hernan", "hiba", "hideaki", "hideki", + "hidenori", "hideyuki", "hikaru", "hilary", "hilda", "hilde", "hilla", "hind", + "hiroaki", "hirofumi", "hirokazu", "hiroki", "hiroko", "hiromi", "hironori", "hiroshi", + "hiroto", "hiroyuki", "hisashi", "hitoshi", "hoda", "holger", "homero", "horacio", + "horia", "hortensia", "hossein", "hovhannes", "hristina", "hristo", "hrvoje", "hsuan", + "hugh", "hugo", "humberto", "husam", "hussam", "hvlya", "hvseyin", "hwa", + "hwan", "hyo", "hyoung", "hyuk", "ia", "iain", "ian", "iason", + "ibtisam", "ibtissam", "ibtissem", "icaro", "ichiro", "ida", "idalia", "idit", + "ieva", "iffat", "ignacia", "ignacio", "ignas", "igor", "ihab", "ihar", + "ihor", "il", "ilaha", "ilana", "ilaria", "ilayda", "ilda", "ildiko", + "ileana", "ilgar", "ilhama", "iliana", "ilija", "ilir", "iliyan", "iliyana", + "ilja", "ilka", "ilkay", "ilke", "ilker", "ilkka", "ilona", "ilse", + "ilya", "ilze", "imane", "imed", "imelda", "imogen", "ina", "inaki", + "inara", "inas", "inbal", "indiana", "indira", "indre", "indu", "ine", + "ines", "inesa", "inese", "inessa", "inge", "ingeborg", "inger", "ingibjorg", + "ingmar", "ingo", "ingrid", "ingrida", "ingunn", "ingvild", "inka", "inmaculada", + "inna", "intisar", "intissar", "ioanna", "ioannis", "irada", "iraida", "irakli", + "irem", "irena", "irene", "ireneusz", "irina", "irine", "irini", "iris", + "irit", "irma", "iroda", "iryna", "isabel", "isabela", "isabell", "isabella", + "isabelle", "isao", "isaura", "isidora", "isil", "isis", "ismet", "isobel", + "istvan", "itai", "italo", "itamar", "itay", "iva", "ivan", "ivana", + "ivania", "ivanka", "ivanna", "ivar", "ivaylo", "ivelin", "ivelina", "iveta", + "ivette", "ivica", "ivo", "ivona", "ivone", "ivonne", "ivor", "ivy", + "iwona", "iyad", "izabela", "izabella", "iztok", "jaakko", "jaan", "jaana", + "jaap", "jacek", "jacinta", "jackie", "jacky", "jaco", "jacobus", "jacopo", + "jacqueline", "jade", "jadranka", "jadwiga", "jae", "jafet", "jagdish", "jahongir", + "jair", "jairo", "jake", "jakob", "jakov", "jakub", "jale", "jaloliddin", + "jamie", "jan", "jana", "jane", "janelle", "janet", "janeth", "janett", + "janette", "janice", "janina", "janine", "janis", "janja", "janka", "janna", + "janne", "janneke", "jannik", "jannis", "jaqueline", "jared", "jari", "jarkko", + "jarle", "jarmila", "jarmo", "jarno", "jaromir", "jaroslav", "jaroslava", "jaroslaw", + "jasmin", "jasmina", "jasmine", "jasminka", "jasna", "jason", "jasper", "jasur", + "javad", "javier", "javiera", "jayne", "jean-claude", "jean-luc", "jean-marie", "jean-paul", + "jeanette", "jeanine", "jeannette", "jeannine", "jef", "jeff", "jekaterina", "jelena", + "jelica", "jelle", "jemma", "jenifer", "jenna", "jennie", "jennifer", "jenny", + "jennyfer", "jeno", "jens", "jeppe", "jeremy", "jernej", "jeroen", "jerry", + "jerzy", "jesper", "jesse", "jessica", "jessie", "jessika", "jessy", "jette", + "jeyhun", "jihane", "jill", "jillian", "jimena", "jimmy", "jinan", "jindrich", + "jiri", "jitka", "jiun", "joakim", "joan", "joana", "joanna", "joanne", + "joaquina", "jocelyn", "jocelyne", "jochen", "jodie", "jody", "joel", "joelle", + "joep", "joeri", "joey", "johan", "johana", "johann", "johanna", "johanne", + "johannes", "johanny", "johnathan", "johnny", "joke", "jolanta", "jolien", "jolita", + "jon", "jonatan", "jonathan", "jonathon", "jonida", "jonna", "jonny", "joon", + "joona", "joonas", "joong", "joost", "jordi", "jordy", "jorgen", "jorma", + "jorn", "jorunn", "jos", "jose", "josef", "josefa", "josefin", "josefina", + "josefine", "joseline", "josep", "josephine", "joshua", "josiane", "josie", "josip", + "josipa", "josselin", "josseline", "josselyn", "jostein", "josue", "jovan", "jovana", + "jovanka", "jovita", "joze", "jozef", "juan", "juana", "juanita", "judicael", + "judit", "judita", "judith", "judy", "juha", "juhani", "juho", "jukka", + "julia", "julian", "juliana", "juliane", "julianna", "julianne", "julie", "juliet", + "julieta", "juliette", "julija", "julijana", "julio", "jumana", "june", "junichi", + "junji", "junko", "junya", "juozas", "juraj", "jurate", "jurgen", "jurgis", + "jurgita", "jurij", "juris", "jussi", "justas", "justin", "justina", "justinas", + "justyna", "jutta", "juvenal", "jvrgen", "jvri", "jyrki", "kaan", "kaarina", + "kacper", "kais", "kaisa", "kaitlyn", "kaja", "kajsa", "kalle", "kalliopi", + "kaloyan", "kalyani", "kamelia", "kamila", "kamile", "kamilla", "kaori", "kaoru", + "karel", "karen", "karianne", "karin", "karina", "karine", "karl", "karla", + "karlis", "karlo", "karlygash", "karmen", "karol", "karoliina", "karolin", "karolina", + "karoline", "karolis", "karuna", "kaspars", "kassandra", "katalin", "katariina", "katarina", + "katarzyna", "kate", "katelyn", "katerina", "kateryna", "katharina", "katharine", "katherina", + "katherine", "kathia", "kathleen", "kathrin", "kathrine", "kathryn", "kathy", "katia", + "katie", "katinka", "katja", "katri", "katrien", "katriina", "katrin", "katrina", + "katrine", "katsiaryna", "katsuhiro", "katy", "katya", "kayla", "kayleigh", "kazimierz", + "kazuaki", "kazuhiko", "kazuhiro", "kazuki", "kazunori", "kazuo", "kazuya", "kazuyuki", + "keigo", "keiichi", "keiko", "keisuke", "keith", "kelsey", "kelvin", "ken", + "kenichi", "kenji", "kenneth", "kensuke", "kenta", "kentaro", "kento", "kenza", + "kerly", "kerstin", "kestutis", "ketevan", "ketty", "keun", "kevin", "kevser", + "khadija", "khadijeh", "khalida", "khatuna", "khrystyna", "kieran", "kiet", "kimberley", + "kimberly", "kimmo", "kinga", "kiril", "kirill", "kirsi", "kirsten", "kirsti", + "kirstine", "kirsty", "kirti", "kiryl", "kishan", "kitty", "kiyoshi", "kjell", + "kjersti", "kjetil", "klara", "klaudia", "knut", "kohei", "koichi", "koichiro", + "koji", "konstantin", "konstantina", "konstantinos", "konul", "koray", "kornelia", "kornelija", + "kostadin", "kostas", "kostiantyn", "kosuke", "kotaro", "kourosh", "krasimir", "krasimira", + "kremena", "kresimir", "kris", "krista", "kristaps", "kristel", "kristi", "kristian", + "kristiana", "kristiina", "kristijan", "kristin", "kristina", "kristine", "kristjan", "kristoffer", + "kristopher", "kristy", "kristyna", "krisztian", "krisztina", "krunoslav", "krystal", "krystian", + "krystyna", "krzysztof", "ksenia", "ksenija", "kseniya", "kumiko", "kuralay", "kvbra", + "kyle", "kyoko", "kyoung", "kyra", "kyriaki", "kyriakos", "kyrylo", "kyung", + "lachezar", "ladislav", "laetitia", "laima", "laith", "lalit", "lalita", "lalith", + "lamia", "lamiaa", "lamis", "lamiya", "lampros", "lana", "lap", "larisa", + "larissa", "larry", "lars", "larysa", "lasha", "lasma", "lasse", "latifa", + "laura", "lauren", "laurence", "lauri", "lauriane", "laurynas", "laxmi", "layal", + "layla", "lazara", "lazaros", "lazzat", "lea", "leah", "leandra", "leanne", + "leda", "leela", "leen", "leena", "leidy", "leif", "leila", "leire", + "lejla", "lena", "lene", "lenka", "lennart", "lenny", "leny", "leona", + "leonardo", "leonel", "leonid", "leonie", "leonor", "leonora", "leopoldo", "lesley", + "leszek", "leticia", "letizia", "levente", "levon", "leyla", "liam", "liane", + "lianna", "lianne", "liat", "libor", "lida", "lidia", "lidija", "lidiya", + "lieke", "liene", "liesbeth", "lieve", "lieven", "liga", "ligia", "liina", + "liis", "liisa", "lilach", "lilian", "liliana", "liliane", "lilibeth", "lilit", + "liliya", "liljana", "lillian", "lily", "lilyana", "limor", "lina", "linas", + "linda", "line", "linnea", "lior", "liran", "liron", "lisa", "lisandra", + "lisanne", "lisbet", "lisbeth", "lise", "liselotte", "liset", "lisette", "lissa", + "liubov", "liv", "liya", "liz", "lizet", "lizette", "lizzie", "lizzy", + "ljiljana", "ljubica", "ljubisa", "ljubomir", "lluis", "loic", "lolita", "loraine", + "loredana", "lorena", "lorenza", "loreta", "loretta", "loris", "lorna", "lorraine", + "lotta", "lotte", "loubna", "louisa", "louise", "lourdes", "lovisa", "lovro", + "luana", "lubica", "lubna", "lubomir", "lubos", "lubov", "lucia", "luciana", + "luciane", "lucie", "lucija", "lucila", "lucile", "lucille", "lucinda", "lucrezia", + "lucy", "ludek", "ludmila", "ludovic", "ludovica", "ludvig", "luigi", "luis", + "luisa", "luise", "luiza", "luka", "lukas", "lukasz", "lurdes", "lusine", + "luuk", "lya", "lydia", "lyka", "lyn", "lynette", "lynne", "lyubomir", + "lyubov", "lyudmila", "lyudmyla", "maaike", "maarit", "maarja", "maarten", "maartje", + "maayan", "mabel", "macarena", "maciej", "madalin", "madelaine", "madeleine", "madeline", + "madina", "madison", "mads", "mae", "mael", "maelle", "maeve", "mafalda", + "magali", "magda", "magdalena", "magdalene", "magdalini", "magdolna", "maggie", "magnus", + "maha", "mahnaz", "mahsa", "maija", "maiju", "maik", "maike", "maikel", + "maiken", "maira", "mairead", "maisa", "maite", "maja", "makiko", "makoto", + "makpal", "maksim", "maksims", "maksym", "malena", "malene", "malgorzata", "malte", + "mamta", "manabu", "manal", "mandeep", "mandy", "manfred", "manish", "manisha", + "manjula", "manuela", "maral", "marc", "marcela", "marcelina", "marcell", "marcella", + "marcelle", "marcello", "marcelo", "marcia", "marcin", "marcio", "marco", "mareike", + "margaret", "margareta", "margareth", "margarida", "margarita", "margaux", "margherita", "margie", + "margit", "margo", "margot", "margret", "margrethe", "marguerite", "margus", "marharyta", + "maria", "mariam", "mariana", "mariane", "marianela", "marianella", "mariangel", "mariangela", + "mariann", "marianna", "marianne", "maribel", "maricel", "maricela", "maricruz", "marie", + "marieke", "mariel", "mariela", "mariella", "marielle", "marieta", "marietta", "marija", + "marijana", "marijke", "marijn", "marika", "marilia", "marilin", "marilyn", "marina", + "mario", "mariola", "marios", "marisa", "marisel", "marisela", "mariska", "marisol", + "marissa", "marit", "marita", "maritza", "mariusz", "mariya", "mariyana", "marja", + "marjan", "marjeta", "marjo", "marjolein", "marjorie", "mark", "marketa", "markku", + "marko", "marla", "marleen", "marlen", "marlena", "marlene", "marlies", "marly", + "marouane", "marta", "martha", "marthe", "martijn", "martina", "martti", "martyna", + "martynas", "marvin", "marwa", "marwan", "mary", "maryam", "maryan", "maryia", + "marylin", "maryna", "marzena", "marzhan", "marzia", "marzieh", "marziyeh", "masaaki", + "masafumi", "masahiko", "masahiro", "masako", "masanori", "masaru", "masashi", "masato", + "masatoshi", "masayuki", "masoomeh", "masoumeh", "massimiliano", "massimo", "matej", "mateja", + "mateusz", "matevz", "mathijs", "mathilde", "matija", "matilda", "matilde", "matjaz", + "mats", "matteo", "matthew", "matthias", "matthieu", "matti", "mattias", "maud", + "maureen", "maurine", "maurizio", "mavis", "mavluda", "max", "maxence", "maxime", + "maximilian", "maximiliano", "maxine", "mayra", "maysa", "maysam", "maysoon", "mayte", + "mayu", "mayumi", "mazen", "maziar", "meenakshi", "meera", "meg", "megan", + "megha", "meghan", "meghana", "megumi", "mehmet", "mehrdad", "mehriban", "mehrnaz", + "mehtap", "meike", "meital", "melania", "melanie", "melika", "melike", "melina", + "melisa", "melissa", "melody", "meltem", "melvin", "menno", "merab", "merav", + "merel", "merete", "meriam", "merima", "merita", "merja", "meruyert", "merve", + "mervi", "meryem", "mesut", "mette", "micael", "micaela", "michael", "michaela", + "michal", "michala", "michalina", "michela", "michele", "michelle", "michiel", "mickael", + "midori", "mieke", "miglena", "miha", "mihael", "mihailo", "mihajlo", "mihkel", + "miia", "miika", "mikael", "mikaela", "mikalai", "mike", "mikel", "mikhail", + "mikkel", "mikko", "mikolaj", "milagros", "milana", "milda", "mildred", "milen", + "milena", "milene", "milenka", "milenko", "milica", "miljana", "milka", "milorad", + "milos", "miloslav", "milovan", "miluska", "milutin", "mimoza", "mindaugas", "minna", + "minoru", "miodrag", "miray", "mirco", "mireia", "mireille", "mirella", "miriam", + "mirian", "mirjam", "mirjana", "mirka", "mirko", "mirna", "miroslav", "miroslava", + "miroslaw", "mirta", "miruna", "miryam", "misha", "mishel", "mislav", "mitja", + "mitko", "mitzi", "mladen", "mogens", "moira", "mojca", "mojgan", "mojtaba", + "molly", "momchil", "mona", "monia", "monica", "monika", "monique", "monisha", + "monserrat", "montserrat", "morgane", "morten", "morteza", "mouhamad", "mounia", "muammer", + "muhammet", "muharrem", "mukesh", "muneera", "murielle", "mvcahit", "mykhailo", "mykola", + "mykyta", "mylene", "myra", "myriam", "myrna", "myroslav", "myroslava", "myrto", + "nada", "nadezda", "nadezhda", "nadia", "nadica", "nadine", "nadiya", "nadja", + "nadya", "nadzeya", "naeema", "nafisa", "nahida", "naile", "nailya", "najat", + "najma", "namig", "nancy", "nandor", "nanna", "naoki", "naoko", "naomi", + "naoto", "naoya", "narcisa", "narek", "narges", "nargiz", "nargiza", "narjes", + "narmin", "narmina", "nasiba", "nastaran", "nastassia", "natacha", "natalia", "natalie", + "nataliia", "natalija", "nataliya", "natalja", "natallia", "nataly", "natalya", "natanael", + "natasa", "natascha", "natasha", "natasja", "natavan", "nathalie", "natia", "naveen", + "navid", "navin", "nawal", "nazanin", "nazgul", "nazila", "nazira", "nazrin", + "nazym", "nebojsa", "neda", "nedal", "nedim", "neelam", "neeraj", "neftali", + "negar", "neha", "neil", "nejc", "nektarios", "nele", "neli", "nelida", + "nelly", "nemanja", "nenad", "nerea", "nereida", "nerijus", "neringa", "nermin", + "nermine", "neslihan", "nesrin", "nestor", "netanel", "neus", "nevena", "nevenka", + "niall", "niamh", "niccolo", "nichole", "nick", "nicky", "niclas", "nico", + "nicolaas", "nicolas", "nicole", "nicolette", "nicoline", "nicolle", "nicolo", "nida", + "nidal", "nidhi", "nidia", "niek", "niels", "nienke", "nigel", "nigora", + "nihal", "nihan", "nihat", "niina", "nijole", "nikhil", "nikita", "nikki", + "niklas", "nikol", "nikola", "nikolai", "nikolaj", "nikolaos", "nikolaus", "nikolay", + "nikoleta", "nikolett", "nikoletta", "nikolina", "nikoloz", "nikos", "niksa", "nil", + "nilay", "nilda", "nilgvn", "niloofar", "niloufar", "nils", "nilufar", "nilvfer", + "ninfa", "nisrine", "nives", "noam", "nobuhiro", "nobuyuki", "noelia", "noelle", + "noemi", "noemie", "noemy", "noha", "nohemi", "nona", "noora", "nora", + "norah", "norbert", "norberto", "noriko", "norio", "norka", "norma", "nouha", + "noura", "nozima", "nozomi", "nubia", "nuno", "nuran", "nurana", "nuray", + "nurdan", "nurgul", "nuria", "nuriddin", "nurit", "nursultan", "nurul", "nurzhan", + "oceane", "octavio", "odd", "oded", "odete", "odette", "odysseas", "ofelia", + "ofra", "ognjen", "ognyan", "oguzhan", "oisin", "oksana", "ola", "olaf", + "olafur", "olav", "olavi", "oldrich", "ole", "oleg", "oleh", "oleksandr", + "oleksandra", "oleksii", "oleksiy", "olena", "olesia", "olesya", "olexandr", "olga", + "olha", "olivia", "oliwia", "olli", "olof", "olympia", "olzhas", "omaima", + "omid", "ondrej", "oona", "ophelia", "oreste", "oriana", "oriol", "orit", + "orkhan", "orkun", "orla", "orly", "ornella", "orxan", "osamu", "oscar", + "oskar", "oskari", "oskars", "osnat", "ossi", "osvaldo", "oswaldo", "otabek", + "otilia", "oumaima", "ourania", "outi", "ove", "oykv", "oystein", "oyvind", + "ozge", "ozlem", "oznur", "ozoda", "pablo", "padraig", "paige", "paivi", + "pallavi", "paloma", "pamela", "panagiota", "panagiotis", "panayiota", "panayiotis", "paola", + "paolo", "paraskevi", "pardis", "parisa", "parul", "parviz", "patric", "patrice", + "patricia", "patricija", "patrick", "patrik", "patrizia", "patrycja", "patryk", "pauliina", + "paulina", "pauline", "paulius", "pavla", "pavle", "pavlina", "pavlo", "pavlos", + "pavol", "pawan", "pawel", "payam", "pearl", "peder", "peeter", "pegah", + "peggy", "pekka", "penelope", "peninah", "penka", "per", "pernilla", "pernille", + "perrine", "pertti", "petar", "peter", "peteris", "petko", "petr", "petra", + "petronela", "petronella", "petter", "petteri", "petya", "peyman", "phil", "philipp", + "philippa", "phoebe", "phyllis", "pia", "pierina", "pierluigi", "piero", "pierpaolo", + "piet", "pieter", "piia", "pilar", "pim", "pinelopi", "piotr", "pirjo", + "plamen", "plamena", "plinio", "polina", "polona", "polyxeni", "pontus", "pooja", + "poonam", "povilas", "prabhat", "pragya", "pramila", "pramod", "prashant", "pratibha", + "preben", "predrag", "preeti", "premysl", "priit", "primoz", "prisca", "priscila", + "priscilla", "priti", "priyanka", "przemyslaw", "quentin", "qusai", "rabab", "rabindra", + "rachael", "rachel", "rachelle", "radek", "radhika", "radim", "radka", "radmila", + "radoslav", "radoslava", "radoslaw", "radostina", "radovan", "raed", "rafael", "rafaela", + "rafail", "rafal", "raffaele", "raffaella", "rafia", "raghad", "raghda", "ragnar", + "ragnhild", "rahel", "rahul", "raimonda", "raimondas", "raiza", "rajeev", "rakesh", + "ralf", "ramil", "ramin", "raminta", "ramiro", "ramiz", "ramona", "ramunas", + "ramune", "ramy", "randa", "randy", "rania", "ranjana", "ranko", "rannveig", + "ranya", "raoul", "raphaela", "raquel", "rasha", "rashida", "rashmi", "rasmus", + "rastislav", "ratko", "raul", "raushan", "ravshan", "rawan", "rawia", "razan", + "razia", "razieh", "rebeca", "rebecca", "rebecka", "rebeka", "rebekah", "rebekka", + "reem", "reema", "regina", "reginald", "regine", "rehana", "reidar", "reinier", + "reinis", "reka", "rekha", "remco", "remi", "remigiusz", "rena", "renan", + "renata", "renate", "renato", "rene", "renee", "renu", "renuka", "reshma", + "revaz", "reyhaneh", "rhiannon", "rhonda", "rhys", "rianne", "ricarda", "riccardo", + "ridvan", "rigoberto", "rihab", "riham", "rihards", "riikka", "riina", "riitta", + "rik", "rikard", "riki", "rikke", "riku", "rima", "rimantas", "rina", + "rinat", "risto", "rita", "rivka", "rob", "roberta", "robertas", "roberto", + "robyn", "rochelle", "rocio", "rodney", "rodolfo", "roee", "rogelio", "rohini", + "roisin", "rok", "rokas", "roksana", "rolandas", "rolando", "rolf", "romain", + "romaric", "romel", "romina", "romy", "ronald", "ronaldo", "ronit", "ronja", + "ronnie", "ronny", "roosa", "rory", "rosalba", "rosalia", "rosalinda", "rosana", + "rosangela", "rosanna", "rosaria", "roseline", "rosemarie", "rosemary", "roshni", "rosie", + "rossana", "rossella", "rostislav", "rostyslav", "rosy", "rovshan", "rowena", "roxana", + "roxane", "roxanne", "roya", "rozalia", "ruba", "ruben", "ruchi", "rufina", + "rui", "rukiye", "rula", "rumen", "rumeysa", "rumyana", "rune", "rupert", + "ruslana", "rusudan", "rut", "rutger", "ruth", "ruxandra", "ruzanna", "ruzica", + "rvmeysa", "ryo", "ryohei", "ryosuke", "ryota", "ryotaro", "ryszard", "rytis", + "ryuichi", "saana", "saara", "sabiha", "sabina", "sabine", "sabira", "sabrina", + "sachiko", "sachin", "safwan", "sahand", "saija", "saima", "saja", "sakari", + "sakina", "sally", "salma", "saltanat", "salwa", "samah", "samantha", "samar", + "samer", "samina", "samira", "samitha", "samuele", "samuli", "samya", "sanaa", + "sanae", "sanaz", "sandeep", "sandhya", "sandra", "sandrine", "sandro", "sanela", + "sangeeta", "sangita", "sanita", "saniya", "sanja", "sanjay", "sanjiv", "sanne", + "santeri", "santosh", "sanzhar", "sara", "sarah", "sarai", "sardor", "sarit", + "sarita", "saroj", "sarolta", "sarunas", "sarvar", "sasa", "sascha", "sasha", + "saskia", "satoru", "satoshi", "satu", "saul", "saule", "saulius", "saulo", + "savannah", "saverio", "savvas", "sawsan", "sayaka", "sayda", "sayuri", "sead", + "seamus", "sean", "sebastiaan", "sebastian", "sebastiano", "sebastien", "sebnem", "secil", + "sedat", "sedigheh", "seiji", "selena", "selene", "selina", "selma", "semen", + "semih", "semir", "semra", "senad", "senka", "sepideh", "seppo", "serap", + "serena", "sergei", "sergej", "sergejs", "sergey", "sergii", "sergio", "sergiy", + "serhat", "serhii", "serhiy", "serife", "serkan", "serpil", "seun", "seung", + "sevara", "sevda", "severine", "sevinj", "sevval", "seyda", "seyma", "seymur", + "shadi", "shadia", "shaghayegh", "shahida", "shahla", "shahnaz", "shahnoza", "shahram", + "shahrzad", "shailendra", "shaimaa", "shaista", "shakira", "shalini", "shalva", "shane", + "sharad", "sharifa", "sharifah", "sharmila", "sharon", "shashi", "shatha", "shaun", + "shauna", "shazia", "sheetal", "sheila", "shereen", "sheryl", "shiela", "shigeki", + "shigeru", "shilpa", "shingo", "shinichi", "shinji", "shinsuke", "shintaro", "shinya", + "shirley", "shiva", "shivani", "sho", "shogo", "shohei", "shoira", "sholpan", + "shorena", "shraddha", "shreya", "shruti", "shuhei", "shuji", "shukhrat", "shunsuke", + "shuo", "shweta", "shynar", "sian", "siarhei", "sibel", "sibylle", "sidney", + "sidsel", "sif", "sigita", "sigitas", "signe", "sigrid", "sigrun", "sigurd", + "siham", "siiri", "silja", "silje", "silke", "silvana", "silvia", "silvie", + "silvija", "silvina", "silvio", "silviu", "silviya", "simen", "simge", "simona", + "simonas", "sina", "sindre", "sindy", "sinead", "sinem", "sinisa", "siobhan", + "sirine", "sissel", "siv", "sjoerd", "sladana", "slaven", "slavica", "slavisa", + "slavka", "slavko", "slavomir", "slawomir", "slobodan", "slobodanka", "smita", "smriti", + "sneha", "snezana", "snezhana", "snjezana", "soad", "sofia", "sofie", "sofija", + "sofiya", "sofya", "soheil", "soheila", "soichiro", "solange", "soledad", "solene", + "solveig", "sona", "sonal", "sonali", "soner", "songvl", "sonia", "soniya", + "sonja", "sonya", "sook", "sophia", "sophie", "sophio", "soren", "sorina", + "sotiria", "sotirios", "spela", "spiros", "spyridon", "spyridoula", "spyros", "srdan", + "srecko", "stacy", "staffan", "stanimir", "stanislav", "stanislava", "stanislaw", "stavros", + "stavroula", "steeven", "stefaan", "stefan", "stefania", "stefanie", "stefano", "stefanos", + "stefka", "stelios", "stella", "stepanka", "stephanie", "stephen", "stergios", "stevan", + "steve", "steven", "stian", "stig", "stijn", "stina", "stine", "stjepan", + "stoyan", "styliani", "stylianos", "suad", "subrata", "sudhir", "sudipta", "suhad", + "suhair", "sujata", "sujit", "sumayya", "sumit", "sumitra", "suncica", "sunita", + "suraya", "suren", "susan", "susana", "susanna", "susanne", "sushil", "sushila", + "sushma", "susmita", "susumu", "susy", "suvi", "suzan", "suzana", "suzanna", + "suzanne", "suzette", "suzy", "svein", "sven", "svend", "svenja", "sverre", + "svetla", "svetlana", "svetoslav", "sviatlana", "sviatoslav", "svitlana", "svjetlana", "svkrv", + "svleyman", "svmeyra", "svmeyye", "swati", "sweta", "sydney", "sylvia", "sylvie", + "sylwia", "synnove", "szymon", "tabea", "tabitha", "tadas", "tadashi", "tadej", + "tadeusz", "tae", "taek", "tahereh", "tahira", "tahmina", "taichi", "taiga", + "taina", "takaaki", "takafumi", "takahiro", "takako", "takanori", "takashi", "takayuki", + "takehiro", "takeshi", "takuma", "takumi", "takuya", "talia", "tamar", "tamara", + "tamerlan", "tammy", "tamta", "tania", "tanja", "tanya", "tapani", "tapio", + "tara", "tarana", "tarja", "tarmo", "taslima", "tatevik", "tatiana", "tatjana", + "tatsiana", "tatsuya", "tatyana", "tea", "ted", "teemu", "teija", "teimuraz", + "telma", "telmo", "temirlan", "temur", "teodora", "teona", "terence", "teresa", + "terese", "teresita", "tereza", "terezia", "terhi", "terje", "tessa", "tetiana", + "tetsuo", "tetsuya", "tetyana", "teuta", "thais", "thalia", "thanasis", "thea", + "thelma", "theo", "theodor", "theodora", "theodoros", "theofanis", "theresa", "therese", + "theresia", "thilo", "thorsten", "tiago", "tibor", "tiffany", "tigran", "tihana", + "tihomir", "tiia", "tiina", "tijana", "tilen", "tim", "timea", "timo", + "timofey", "timothee", "timothy", "timur", "tina", "tinatin", "tiziana", "tiziano", + "tjasa", "tobias", "toby", "tohru", "tolga", "tom", "tomasz", "tomislav", + "tommaso", "tommi", "tommy", "tomoaki", "tomohiko", "tomohiro", "tomoki", "tomoko", + "tomomi", "tomonori", "tomoya", "tomoyuki", "tone", "tonje", "tony", "toomas", + "torben", "torbjorn", "tornike", "torsten", "toshiaki", "toshihiro", "toshiki", "toshio", + "toshiyuki", "touria", "tove", "tracy", "travis", "trevor", "tricia", "triin", + "trine", "trish", "trisha", "troels", "trond", "trude", "tsukasa", "tsutomu", + "tsuyoshi", "tugba", "tugce", "tuija", "tulio", "tuomas", "tuomo", "tural", + "turgay", "tuukka", "tuula", "tuuli", "uladzimir", "uladzislau", "ulan", "uldis", + "ulf", "ulises", "ulrik", "ulrika", "ulrike", "ulugbek", "ulviyya", "ulyana", + "umberto", "umesh", "umid", "umida", "umut", "uriel", "uros", "urska", + "ursula", "urszula", "usama", "usha", "ute", "uwe", "vaclav", "vadim", + "vadym", "vagif", "vahe", "vahid", "vaida", "vaiva", "vakhtang", "valbona", + "valdemar", "valentina", "valentyn", "valentyna", "valeria", "valerie", "valerii", "valerija", + "valeriy", "valeriya", "valery", "valeska", "valtteri", "vanda", "vandana", "vanesa", + "vanessa", "vania", "vanina", "vanja", "vanya", "vardan", "varsha", "vasil", + "vasileios", "vasiliki", "vasilios", "vasilis", "vasiliy", "vassil", "vassiliki", "vassilios", + "vedat", "vedran", "vedrana", "veena", "veerle", "vegard", "veikko", "velimir", + "veljko", "vendula", "venera", "venla", "verena", "verica", "veronica", "veronika", + "veselin", "veselina", "vesna", "viacheslav", "vibeke", "vicenta", "vicki", "vicky", + "victor", "victoria", "vid", "vidar", "vidya", "vikas", "viktar", "viktor", + "viktoria", "viktorija", "viktoriya", "viktoryia", "viliam", "vilija", "vilius", "vilma", + "vilmos", "vinay", "vincenzo", "vineet", "vinicio", "vinicius", "violet", "violeta", + "violetta", "virginia", "virginie", "virginija", "viridiana", "virpi", "visnja", "vitalii", + "vitaliy", "vitezslav", "vittoria", "vittorio", "vivek", "vivian", "viviana", "viviane", + "vivien", "vivienne", "vjekoslav", "vlada", "vladan", "vladimir", "vladimira", "vladislav", + "vladislava", "vlado", "vladyslav", "vlasta", "vlastimil", "vlatka", "vlatko", "vmit", + "vojislav", "vojtech", "volha", "volodymyr", "vsevolod", "vukasin", "vusal", "vusala", + "vyacheslav", "vytautas", "wael", "wafaa", "wassim", "wataru", "wenche", "wendell", + "wendy", "weronika", "wesam", "wessam", "widad", "wiebke", "wieslaw", "wiktoria", + "wilbert", "wilfredo", "wilfried", "willem", "william", "wilma", "wim", "wing", + "winifred", "winnie", "winston", "wioletta", "wisam", "wissam", "witold", "wladimir", + "wlodzimierz", "wojciech", "wolfgang", "wook", "woong", "wout", "wouter", "xabier", + "xander", "xenia", "ximena", "xiomara", "xochitl", "yael", "yakup", "yanina", + "yann", "yannis", "yara", "yaren", "yaroslav", "yasemin", "yaser", "yashar", + "yasmina", "yasmine", "yasuhiro", "yasuko", "yasuo", "yasushi", "yasuyuki", "yauheni", + "yavor", "yegana", "yehonatan", "yekaterina", "yelena", "yeliz", "yerbol", "yesenia", + "yesica", "yesim", "yessenia", "yevgeniy", "yevgeniya", "yian", "ylva", "yoan", + "yoav", "yoel", "yogendra", "yogesh", "yohei", "yoichi", "yoko", "yolanda", + "yonatan", "yonathan", "yordan", "yoshiaki", "yoshihiko", "yoshihiro", "yoshikazu", "yoshiki", + "yoshiko", "yoshinori", "yoshio", "yoshitaka", "yoshiyuki", "yosra", "yossi", "yosuke", + "yousra", "yudai", "yuichi", "yuichiro", "yuji", "yujin", "yukari", "yuki", + "yukiko", "yukio", "yuko", "yulduz", "yulian", "yuliia", "yuliya", "yumi", + "yumiko", "yuna", "yuri", "yurii", "yuriy", "yury", "yusaku", "yusra", + "yusuke", "yuta", "yutaka", "yuval", "yves", "yvette", "yvonne", "zabihullah", + "zachary", "zahia", "zahida", "zainab", "zaira", "zakia", "zamira", "zaneta", + "zarifa", "zaur", "zaure", "zaynab", "zbigniew", "zbynek", "zdenek", "zdenka", + "zdenko", "zdravka", "zdravko", "zeena", "zeina", "zeinab", "zeliha", "zeljka", + "zenaida", "zenon", "zeynab", "zeynep", "zhanar", "zhanat", "zhandos", "zhanna", + "zhazira", "zhe", "zhibek", "zinaida", "ziva", "zivile", "zlata", "zlatan", + "zlatina", "zlatka", "zlatko", "zoe", "zofia", "zohreh", "zoltan", "zoraida", + "zoran", "zorana", "zorica", "zrinka", "zsofia", "zsolt", "zsuzsanna", "zulfiya", + "zulma", "zumrud", "zurab", "zuzana", "zuzanna", "zvonimir", +) +# -- END generated: COMMON_GIVEN -- + +COMMON_GIVEN_NAMES: frozenset[str] = frozenset(_COMMON_GIVEN) diff --git a/tests/v2/test_layering.py b/tests/v2/test_layering.py index a5ae7a2f2..1d00e03db 100644 --- a/tests/v2/test_layering.py +++ b/tests/v2/test_layering.py @@ -69,6 +69,15 @@ # TYPE_CHECKING "_match.py": ("nameparser._types", "nameparser._lexicon", "nameparser._policy"), + # the group suggester: types, policy, the lexicon's particles and + # titles, the pipeline's patronymic, title, suffix and numeral + # predicates and its private word tables; + # _parser at call time only, in Suggestion.__post_init__, to check + # that the base is a Parser (Parser imports this module at call + # time, in Parser.suggest) + "_suggest.py": ("nameparser._types", "nameparser._policy", + "nameparser._lexicon", "nameparser._pipeline", + "nameparser.config.", "nameparser._parser"), # the shared compiler and the reading index's reader (#753, 3a): the # index, a config DATA module, and nothing else. That the import is # at call time only, nothing internal at module level, since @@ -135,11 +144,12 @@ "nameparser._reading_index",), # Parser sits on everything except _render; the reading index at # call time only, where a choice of languages is applied (3b); _match - # at call time only, in Parser.matching() + # at call time only, in Parser.matching(); _suggest at call time + # only, in Parser.suggest() "_parser.py": ("nameparser._types", "nameparser._lexicon", "nameparser._policy", "nameparser._locale", "nameparser._pipeline", "nameparser._reading_index", - "nameparser._match"), + "nameparser._match", "nameparser._suggest"), # PEP 562 lazy loader: the only static import is _locale (for the # Locale return type); pack modules load via importlib.import_module # (a runtime string, invisible to the AST walker) -- the @@ -376,6 +386,7 @@ def test_every_frozen_dataclass_carries_the_pickle_guards() -> None: import nameparser._match import nameparser._parser import nameparser._policy + import nameparser._suggest import nameparser._types from nameparser._types import _guarded_getstate, _guarded_setstate @@ -383,7 +394,7 @@ def test_every_frozen_dataclass_carries_the_pickle_guards() -> None: # are never pickled and are not public API. modules = (nameparser._types, nameparser._lexicon, nameparser._policy, nameparser._locale, nameparser._parser, - nameparser._match) + nameparser._match, nameparser._suggest) for module in modules: for _, cls in inspect.getmembers(module, inspect.isclass): if cls.__module__ != module.__name__: @@ -693,18 +704,22 @@ def test_the_check_finds_a_tool_read_through_an_alias( #: 169 -> 170, and the maiden marker ending a part's own words, 170 -> #: 171; its review's suffix-word refusal after the stop, 171 -> 179; #: its fix-diff review's own name test for the saint division (titles -#: of every kind, connectives), 179 -> 185); -#: the tools' 19 are 17 +#: of every kind, connectives), 179 -> 185; `_suggest`'s particle +#: read, 185 -> 186; its leading title read, which reads a name's +#: evidence past its titles, 186 -> 187); +#: the tools' 20 are 18 #: attribute loads, every tool scanned, and config_diff.py's #: `sets =` binding and the roster `getattr` through it (its #: `changed_words`, which compares two lexicons set by set), which -#: with one of the loads moved the count from 16. 4i-a's own count was 114 (on +#: with one of the loads moved the count from 16; the 18th load is +#: tools/vocab/order_names.py reading `locales.ZH.lexicon._sets.surnames` +#: (group suggestions). 4i-a's own count was 114 (on #: that tree's sources), two of them `getattr`s in #: `_parser._apply_choice`, which read the sets through #: `_lexicon._fields_of` from 4i-b until 4i-c retired it. #: A reader added through `_sets` moves these, and they are re-taken #: then: they count what the check would refuse with the move undone. -_UNGUARDED_READS = {"library": 185, "tools": 19} +_UNGUARDED_READS = {"library": 187, "tools": 20} def test_the_check_refuses_every_read_4i_a_moved() -> None: diff --git a/tests/v2/test_order_names_tool.py b/tests/v2/test_order_names_tool.py new file mode 100644 index 000000000..95d50fe32 --- /dev/null +++ b/tests/v2/test_order_names_tool.py @@ -0,0 +1,266 @@ +"""Checks on tools/vocab/order_names.py, which writes the romanized +surname table the group suggester reads (``config/_order_names.py``). + +The table is read by the suggester only, never by a parse. The tool's +reading and folding are pinned on hand-built input; the shipped block is +held to what the tool writes from the cached Unihan, skipped where the +cache is absent. +""" +from __future__ import annotations + +import importlib.util +import sys +import zipfile +from pathlib import Path +from types import ModuleType + +import pytest + +from nameparser.config import _order_names +from tests.checkout import needs_checkout + +_TOOLS = Path(__file__).parents[2] / "tools" +_PATH = _TOOLS / "vocab" / "order_names.py" + + +def _load() -> ModuleType: + sys.path.insert(0, str(_TOOLS)) + try: + spec = importlib.util.spec_from_file_location("order_names", _PATH) + assert spec is not None and spec.loader is not None + module = importlib.util.module_from_spec(spec) + sys.modules[spec.name] = module + spec.loader.exec_module(module) + finally: + sys.path.remove(str(_TOOLS)) + return module + + +@pytest.fixture(scope="module") +def tool() -> ModuleType: + return _load() + + +@needs_checkout +def test_fold_strips_tones_and_vietnamese_marks(tool: ModuleType) -> None: + assert tool.fold("Nguyễn") == "nguyen" + assert tool.fold("Đặng") == "dang" + assert tool.fold("Lǚ") == "lv" + + +@needs_checkout +def test_mandarin_takes_the_first_reading_without_tone(tool: ModuleType) -> None: + assert tool.read_mandarin("U+912D\tkMandarin\tzhèng\n")["鄭"] == "zheng" + assert tool.read_mandarin("U+4E50\tkMandarin\tlè yuè\n")["乐"] == "le" + + +@needs_checkout +def test_pinyin_surnames_drop_one_letter_readings(tool: ModuleType) -> None: + assert tool.pinyin_surnames(["阿", "鄭"], {"阿": "a", "鄭": "zheng"}) == {"zheng"} + + +@needs_checkout +def test_a_block_round_trips_through_rewrite(tool: ModuleType) -> None: + block = tool.render_block("PINYIN", ["wang", "zhang"]) + other = tool.render_block("OTHER", ["x"]) + text = f"head\n{block}\nmid\n{other}\ntail\n" + new = tool.render_block("PINYIN", ["li"]) + out = tool.rewrite(text, "PINYIN", new) + assert new in out and other in out and "wang" not in out + assert tool.rewrite(text, "PINYIN", block) == text + with pytest.raises(ValueError, match="no generated block"): + tool.rewrite("nothing here", "PINYIN", block) + + +@needs_checkout +def test_the_hand_lists_are_folded_and_the_table_has_no_one_letter_entry() -> None: + table = _order_names.ROMANIZED_SURNAMES + assert all(len(w) >= 2 and w == w.lower() and w.isascii() and w.isalpha() + for w in table) + for name in ("_ZH_OTHER_SPELLINGS", "_KO_SPELLINGS", "_VI_SURNAMES"): + words = getattr(_order_names, name) + assert words and set(words) <= table + assert {"zheng", "nguyen", "kim", "wong"} <= table + + +@needs_checkout +def test_the_shipped_surname_block_is_what_the_tool_writes( + tool: ModuleType) -> None: + from nameparser import locales + path = tool.name_usage.cache_dir() / tool.UNIHAN_FILE + if not path.exists(): + pytest.skip("the Unihan cache is absent (tools/zh_surnames.py fetches it)") + with zipfile.ZipFile(path) as zf: + mandarin = tool.read_mandarin( + zf.read(tool.UNIHAN_MEMBER).decode("utf-8")) + singles = [s for s in locales.ZH.lexicon._sets.surnames if len(s) == 1] + block = tool.render_block("PINYIN", sorted( + tool.pinyin_surnames(singles, mandarin))) + text = Path(_order_names.__file__).read_text(encoding="utf-8") + assert tool.rewrite(text, "PINYIN", block) == text + + +def _namdict(tool: ModuleType, countries: dict[str, dict[str, int]]) -> object: + """A NamDict whose only entries are ``{country: {name: band}}``.""" + names: dict[str, list[tuple[str, dict[str, int]]]] = {} + for country, bands in countries.items(): + for name, band in bands.items(): + names.setdefault(name, []).append((name, {country: band})) + return tool.name_usage.NamDict( + countries=list(countries), names=names, short_for={}) + + +@needs_checkout +def test_a_name_mostly_written_as_a_family_name_is_dropped( + tool: ModuleType) -> None: + nd = _namdict(tool, {"X": {"thomas": 9, "kim": 9, "anna": 9}}) + from collections import Counter + given = Counter({"thomas": 90, "kim": 50, "anna": 100}) + family = Counter({"thomas": 10, "kim": 50, "anna": 1}) + got = tool.common_given(nd, given, family, top=10, max_family_share=0.05, + min_uses=20, exclude=set()) + assert got == {"anna"} + + +@needs_checkout +def test_a_romanized_surname_and_a_one_letter_name_never_enter( + tool: ModuleType) -> None: + from collections import Counter + nd = _namdict(tool, {"X": {"lee": 9, "a": 9, "anna": 9}}) + given = Counter({"lee": 100, "a": 100, "anna": 100}) + got = tool.common_given(nd, given, Counter(), top=10, max_family_share=0.05, + min_uses=20, exclude={"lee"}) + assert got == {"anna"} + + +@needs_checkout +def test_top_is_per_country_then_unioned(tool: ModuleType) -> None: + from collections import Counter + nd = _namdict(tool, {"X": {"anna": 9, "bea": 5}, "Y": {"carl": 9, "dan": 5}}) + given = Counter({"anna": 99, "bea": 99, "carl": 99, "dan": 99}) + got = tool.common_given(nd, given, Counter(), top=1, max_family_share=0.05, + min_uses=20, exclude=set()) + assert got == {"anna", "carl"} + + +@needs_checkout +def test_the_shipped_given_block_is_what_the_tool_writes( + tool: ModuleType) -> None: + from nameparser import locales + cache = tool.name_usage.cache_dir() + namdict = cache / "nam_dict.txt" + orcid = cache / "local" / "orcid-researchers" / "raw.jsonl" + unihan = cache / tool.UNIHAN_FILE + if not (namdict.exists() and orcid.exists() and unihan.exists()): + pytest.skip("nam_dict, Unihan or the ORCID local corpus is not cached") + with zipfile.ZipFile(unihan) as zf: + mandarin = tool.read_mandarin( + zf.read(tool.UNIHAN_MEMBER).decode("utf-8")) + singles = [s for s in locales.ZH.lexicon._sets.surnames if len(s) == 1] + pinyin = tool.pinyin_surnames(singles, mandarin) + words = tool.load_common_given(tool.surname_exclusions(pinyin)) + tool.check_tables(pinyin, words) + block = tool.render_block("COMMON_GIVEN", words) + text = Path(_order_names.__file__).read_text(encoding="utf-8") + assert tool.rewrite(text, "COMMON_GIVEN", block) == text + assert not words & _order_names.ROMANIZED_SURNAMES + assert _order_names.COMMON_GIVEN_NAMES == frozenset(words) + + +# -- the inputs must be whole, and the tables must make sense ---------------- + + +def _plant_orcid(cache: Path, records: list[dict[str, str]], *, + count: int | None = None, sha256: str | None = None) -> Path: + import hashlib + import json + folder = cache / "local" / "orcid-researchers" + folder.mkdir(parents=True) + body = "".join(json.dumps(r) + "\n" for r in records).encode("utf-8") + (folder / "raw.jsonl").write_bytes(body) + (folder / "meta.json").write_text(json.dumps({ + "records": len(records) if count is None else count, + "sha256": sha256 or hashlib.sha256(body).hexdigest()}), + encoding="utf-8") + return folder / "raw.jsonl" + + +@needs_checkout +def test_whole_orcid_boxes_are_counted( + tool: ModuleType, monkeypatch: pytest.MonkeyPatch, + tmp_path_factory: pytest.TempPathFactory) -> None: + from ._differential_fixtures import own_cache + cache = own_cache(monkeypatch, tmp_path_factory) + path = _plant_orcid(cache, [{"given": "Anna", "family": "Kim"}, + {"given": "Łukasz", "family": "Nowak"}]) + given, family = tool.orcid_box_counts(path) + assert given == {"anna": 1, "lukasz": 1} + assert family == {"kim": 1, "nowak": 1} + + +@needs_checkout +@pytest.mark.parametrize("damage, fragment", [ + ({"count": 3}, "holds 2 records, not the 3"), + ({"sha256": "0" * 64}, "sha256"), +]) +def test_a_cut_orcid_corpus_is_refused( + tool: ModuleType, monkeypatch: pytest.MonkeyPatch, + tmp_path_factory: pytest.TempPathFactory, + damage: dict[str, object], fragment: str) -> None: + from ._differential_fixtures import own_cache + cache = own_cache(monkeypatch, tmp_path_factory) + path = _plant_orcid(cache, [{"given": "Anna", "family": "Kim"}, + {"given": "Jan", "family": "Nowak"}], + **damage) # type: ignore[arg-type] + with pytest.raises(ValueError, match=fragment): + tool.orcid_box_counts(path) + + +@needs_checkout +def test_an_orcid_corpus_without_its_meta_is_refused( + tool: ModuleType, monkeypatch: pytest.MonkeyPatch, + tmp_path_factory: pytest.TempPathFactory) -> None: + from ._differential_fixtures import own_cache + cache = own_cache(monkeypatch, tmp_path_factory) + path = _plant_orcid(cache, [{"given": "Anna", "family": "Kim"}]) + (path.parent / "meta.json").unlink() + with pytest.raises(ValueError, match="meta.json"): + tool.orcid_box_counts(path) + + +@needs_checkout +def test_nam_dict_is_read_checked_and_refused_when_absent_or_cut( + tool: ModuleType, monkeypatch: pytest.MonkeyPatch, + tmp_path_factory: pytest.TempPathFactory) -> None: + from ._differential_fixtures import own_cache + cache = own_cache(monkeypatch, tmp_path_factory) + with pytest.raises(ValueError, match="nam_dict.txt is not cached"): + tool.load_namdict() + (cache / "nam_dict.txt").write_text("cut\n", encoding="utf-8") + with pytest.raises(ValueError, match="sha256"): + tool.load_namdict() + + +@needs_checkout +def test_the_given_names_exclude_this_runs_surnames(tool: ModuleType) -> None: + hand = set(_order_names._ZH_OTHER_SPELLINGS + _order_names._KO_SPELLINGS + + _order_names._VI_SURNAMES) + assert tool.surname_exclusions({"zzpinyin"}) == hand | {"zzpinyin"} + + +@needs_checkout +@pytest.mark.parametrize("pinyin, given, fragment", [ + ({f"p{i}" for i in range(199)}, {f"g{i}" for i in range(3000)}, + "199 pinyin surnames, below the floor of 200"), + ({f"p{i}" for i in range(200)}, {f"g{i}" for i in range(2999)}, + "2999 given names, below the floor of 3000"), + ({f"p{i}" for i in range(200)}, {"p1"} | {f"g{i}" for i in range(3000)}, + "both tables: p1"), +]) +def test_a_table_below_its_floor_or_overlapping_is_refused( + tool: ModuleType, pinyin: set[str], given: set[str], + fragment: str) -> None: + with pytest.raises(ValueError, match=fragment): + tool.check_tables(pinyin, given) + tool.check_tables({f"p{i}" for i in range(200)}, + {f"g{i}" for i in range(3000)}) diff --git a/tests/v2/test_reprs.py b/tests/v2/test_reprs.py index 20a24b985..26e26d0c7 100644 --- a/tests/v2/test_reprs.py +++ b/tests/v2/test_reprs.py @@ -133,6 +133,8 @@ def test_policy_patch_repr_shows_only_set_fields() -> None: Parser(), Parser(segmenter=functools.partial(_decline, blob="x" * 500)), Matching(Parser()), + Parser().suggest(["Priem Jason", "Kim Minjun", "Jason Smith"]), + Parser().suggest([]), ]) def test_config_reprs_never_leak_the_unset_sentinel(obj: object) -> None: # Guard the whole family: every config type's repr is bounded and @@ -193,3 +195,14 @@ def test_order_repr_survives_smuggled_setstate_values() -> None: # a smuggled string tuple must not render as the # named constant it merely compares equal to assert "GIVEN_FIRST" not in rendered + + +def test_suggestion_repr_shows_the_patch_and_counts_never_the_names( + ) -> None: + s = Parser().suggest(["Priem Jason", "Kim Minjun", "Wang Xiaoming", + "Jason Smith"]) + assert repr(s) == ("Suggestion(PolicyPatch(name_order=FAMILY_FIRST), " + "names=4, disagreeing=1)") + for evidence in s.evidence: + for text in ("Priem", "Kim", "Smith"): + assert text not in repr(evidence) diff --git a/tests/v2/test_suggest.py b/tests/v2/test_suggest.py new file mode 100644 index 000000000..830ea9b62 --- /dev/null +++ b/tests/v2/test_suggest.py @@ -0,0 +1,1289 @@ +"""The per-name signals of the group suggester (nameparser/_suggest.py).""" +from __future__ import annotations + +import unicodedata +import warnings + +import pytest + +from collections.abc import Iterator + +from nameparser import ( + FAMILY_FIRST_GIVEN_LAST, UNSET, Lexicon, NameEvidence, Parser, Policy, + PolicyPatch, SettingEvidence, Suggestion, +) +from nameparser._policy import PatronymicRule +from nameparser._suggest import Signal, name_votes +from nameparser._types import FAMILY_FIRST, GIVEN_FIRST, Role + +L = Lexicon.default() + +_VOTES = [ + ("Mehmet DEMİR", Signal.CAPITALS, GIVEN_FIRST), + ("DUPONT Jean", Signal.CAPITALS, FAMILY_FIRST), + ("Armitage J.", Signal.TRAILING_INITIAL, FAMILY_FIRST), + ("Smith J.P.", Signal.TRAILING_INITIAL, FAMILY_FIRST), + ("Smith J.-P.", Signal.TRAILING_INITIAL, FAMILY_FIRST), + ("Smith J-P", Signal.TRAILING_INITIAL, FAMILY_FIRST), + ("van Berg Jan", Signal.PARTICLE, FAMILY_FIRST), + ("Jan van Berg", Signal.PARTICLE, GIVEN_FIRST), + ("O'BRIEN Sean", Signal.CAPITALS, FAMILY_FIRST), + ("GOMINA-MUSA Abɖu", Signal.CAPITALS, FAMILY_FIRST), + ("Mary de Jean", Signal.PARTICLE, GIVEN_FIRST), + (unicodedata.normalize("NFD", "Nguyễn Minh"), Signal.SURNAME, FAMILY_FIRST), + (unicodedata.normalize("NFC", "Nguyễn Minh"), Signal.SURNAME, FAMILY_FIRST), + ("Kim Minjun", Signal.SURNAME, FAMILY_FIRST), + ("Minjun Kim", Signal.SURNAME, GIVEN_FIRST), + ("Priem Jason", Signal.GIVEN_NAME, FAMILY_FIRST), + ("Jason Priem", Signal.GIVEN_NAME, GIVEN_FIRST), + ("Иванов Иван Петрович", Signal.PATRONYMIC, PatronymicRule.EAST_SLAVIC), + ("Ivan Petrovich Sidorov", Signal.PATRONYMIC, PatronymicRule.EAST_SLAVIC), + ("Ziyo Azamat Hamid o‘g‘li", Signal.PATRONYMIC, PatronymicRule.TURKIC), + # cased evidence in any script: Cyrillic capitals vote + ("ИВАНОВ Иван", Signal.CAPITALS, FAMILY_FIRST), + ("Иван ИВАНОВ", Signal.CAPITALS, GIVEN_FIRST), +] + + +@pytest.mark.parametrize(("text", "signal", "value"), _VOTES) +def test_a_name_casts_its_vote( + text: str, signal: Signal, value: object, +) -> None: + assert name_votes(text, L)[signal] == value + + +@pytest.mark.parametrize(("text", "signal"), [ + ("Jeffrey LAM Kin-fung", Signal.CAPITALS), + ("Smith J.-P.", Signal.CAPITALS), + ("Smith J-P", Signal.CAPITALS), + ("Smith J.P.", Signal.CAPITALS), + ("JOHN SMITH", Signal.CAPITALS), + ("john smith", Signal.CAPITALS), + ("S. Ramesh", Signal.TRAILING_INITIAL), + ("J.-P. Smith", Signal.TRAILING_INITIAL), + ("Smith Jean-Pierre", Signal.TRAILING_INITIAL), + ("Smith J-", Signal.TRAILING_INITIAL), + ("Smith -", Signal.TRAILING_INITIAL), + ("Smith A.", Signal.SURNAME), + ("A. Smith", Signal.SURNAME), + ("김민준", Signal.CAPITALS), + ("김민준", Signal.SURNAME), + ("김민준", Signal.GIVEN_NAME), + ("鄭和", Signal.CAPITALS), + ("鄭和", Signal.SURNAME), + ("鄭和", Signal.GIVEN_NAME), +]) +def test_a_name_casts_no_such_vote(text: str, signal: Signal) -> None: + assert signal not in name_votes(text, L) + + +# a final all-capitals word of two or three letters in a mixed-case name +# is joined initials (`Armitage JO`) or a short surname (`Jason NG`): +# it casts no capitals vote and is not looked up as a surname; a +# leading one still votes +@pytest.mark.parametrize("text", [ + "Armitage JO", "Smith JP", "Jones AB", "Jason NG", "Lee SH", + "DUPONT Jean AB"]) +def test_a_final_short_capitals_word_casts_no_capitals_vote(text: str) -> None: + assert Signal.CAPITALS not in name_votes(text, L) + + +@pytest.mark.parametrize("text", ["Armitage JO", "Smith JP", "Jason NG"]) +def test_a_final_short_capitals_word_is_not_looked_up_as_a_surname( + text: str) -> None: + assert Signal.SURNAME not in name_votes(text, L) + + +@pytest.mark.parametrize(("text", "signal"), [ + ("NG Jason", Signal.CAPITALS), ("KIM Jason", Signal.CAPITALS), + ("NG Jason", Signal.SURNAME), ("Lee SH", Signal.SURNAME), + ("Mehmet DEMİR", Signal.CAPITALS)]) +def test_a_leading_or_longer_capitals_word_still_votes( + text: str, signal: Signal) -> None: + votes = name_votes(text, L) + expected = GIVEN_FIRST if text == "Mehmet DEMİR" else FAMILY_FIRST + assert votes[signal] == expected + + +def test_joined_initials_keep_a_family_first_base() -> None: + base = Parser(policy=Policy(name_order=FAMILY_FIRST)) + s = base.suggest(["Armitage JO", "Smith JP", "Jones AB", "Brown KL", + "Garcia M", "Lee SH"]) + assert s.patch.name_order is UNSET + assert s.parser.policy.name_order == FAMILY_FIRST + + +def test_a_patronymic_casts_no_order_vote_of_its_own() -> None: + votes = name_votes("Ivan Petrovich Sidorov", L) + assert votes[Signal.PATRONYMIC] == PatronymicRule.EAST_SLAVIC + assert set(votes) - {Signal.PATRONYMIC, Signal.GIVEN_NAME} == set() + + +def test_a_name_with_both_patronymic_shapes_reports_turkic() -> None: + votes = name_votes("Ali Petrovich Hamid oglu", L) + assert votes[Signal.PATRONYMIC] == PatronymicRule.TURKIC + + +@pytest.mark.parametrize("text", ["Priem, Jason", "", " ", "Madonna"]) +def test_a_name_with_nothing_to_read_casts_nothing(text: str) -> None: + assert name_votes(text, L) == {} + + +@pytest.mark.parametrize("text", ["\u0000", "A.", "a b c d", "İ İ İ", "-", ". ."]) +def test_no_signal_raises(text: str) -> None: + assert isinstance(name_votes(text, L), dict) + + +# -- the group: Parser.suggest ------------------------------------------ + + +_EAST_SLAVIC = ["Сидоров Иван Петрович", "Иванов Пётр Сергеевич", + "Петров Олег Иванович", "Смирнов Павел Андреевич", + "Попов Илья Ильич"] +_TURKIC = ["Mamedov Ali Huseyn oglu", "Aliyev Rauf Ilham oglu", + "Quliyev Elman Rasim oglu", "Hasanova Afet Abil qizi", + "Mikayilova Sevil Alirza qizi"] + + +def _order_evidence(s: Suggestion) -> SettingEvidence: + return next(e for e in s.evidence if e.setting == "name_order") + + +def _patronymic_evidence(s: Suggestion) -> SettingEvidence: + return next(e for e in s.evidence if e.setting == "patronymic_rules") + + +def test_an_empty_group_suggests_nothing() -> None: + s = Parser().suggest([]) + assert s.patch == PolicyPatch() + assert s.parser.policy == Parser().policy + assert s.names == () + assert s.disagreeing() == () + + +def test_a_group_without_evidence_keeps_the_base() -> None: + s = Parser().suggest(["Xyzzy Plugh"]) + assert s.patch == PolicyPatch() + assert not _order_evidence(s).decided + assert _order_evidence(s).no_evidence == 1 + + +def test_one_name_voting_decides_nothing() -> None: + # _MIN_VOTES = 2: one name is thin evidence, so the base stands + s = Parser().suggest(["Priem Jason", "Xyzzy Plugh"]) + assert s.patch == PolicyPatch() + assert not _order_evidence(s).decided + assert _order_evidence(s).votes_against == 1 + + +def test_two_names_voting_one_way_decide() -> None: + assert Parser().suggest(["Priem Jason", "Kim Minjun"]).patch == ( + PolicyPatch(name_order=FAMILY_FIRST)) + + +def test_an_order_needs_more_than_two_thirds_of_the_voting_names() -> None: + # two for, one against: two thirds, not more. The boundary is exact + # in floats too: _MIN_SHARE * 3 == (2/3) * 3 == 2.0, so the strict + # `count <= _MIN_SHARE * total` refuses two of three as written + assert (2 / 3) * 3 == 2.0 + s = Parser().suggest(["Priem Jason", "Kim Minjun", "Jason Smith"]) + assert not _order_evidence(s).decided + # three for, one against: more than two thirds + s = Parser().suggest(["Priem Jason", "Kim Minjun", "Wang Xiaoming", + "Jason Smith"]) + assert s.patch == PolicyPatch(name_order=FAMILY_FIRST) + + +def test_a_family_first_group_suggests_family_first() -> None: + s = Parser().suggest(["Priem Jason", "Kim Minjun", "Wang Xiaoming"]) + assert s.patch.name_order == FAMILY_FIRST + assert s.parser.policy.name_order == FAMILY_FIRST + order = _order_evidence(s) + assert order.decided + assert (order.votes_for, order.votes_against, order.no_evidence) == ( + 3, 0, 0) + assert (Signal.SURNAME, FAMILY_FIRST, 2) in order.by_signal + assert (Signal.GIVEN_NAME, FAMILY_FIRST, 1) in order.by_signal + + +def test_a_family_first_base_can_be_moved_back() -> None: + base = Parser(policy=Policy(name_order=FAMILY_FIRST)) + s = base.suggest(["Jason Priem", "Minjun Kim"]) + assert s.patch.name_order == GIVEN_FIRST + assert s.parser.policy.name_order == GIVEN_FIRST + + +def test_a_vi_base_keeps_its_family_first_spelling() -> None: + # `Nguyen Van An` casts no vote: `an` is a pinyin surname too, so + # both ends are listed + base = Parser(policy=Policy(name_order=FAMILY_FIRST_GIVEN_LAST)) + s = base.suggest(["Nguyen Van Thanh", "Tran Thi Huong"]) + assert s.patch.name_order is UNSET + assert _order_evidence(s).decided + assert _order_evidence(s).value == FAMILY_FIRST_GIVEN_LAST + assert s.parser.policy.name_order == FAMILY_FIRST_GIVEN_LAST + + +def test_a_decided_order_equal_to_the_base_leaves_the_patch_empty() -> None: + s = Parser().suggest(["Jason Priem", "Minjun Kim"]) + assert s.patch == PolicyPatch() + assert _order_evidence(s).decided + + +def test_a_tie_decides_nothing() -> None: + s = Parser().suggest(["Priem Jason", "Jason Smith"]) + assert s.patch == PolicyPatch() + assert not _order_evidence(s).decided + assert s.disagreeing() == () + + +def test_an_east_slavic_group_gets_the_rule_and_a_given_first_order( + ) -> None: + s = Parser().suggest(_EAST_SLAVIC) + assert s.patch.patronymic_rules == frozenset({PatronymicRule.EAST_SLAVIC}) + # the base is given-first already, so the patch leaves the order out + assert s.patch.name_order is UNSET + assert _patronymic_evidence(s).decided + order = _order_evidence(s) + assert order.decided + assert order.value == GIVEN_FIRST + assert "East Slavic" in order.note + assert s.parser.parse("Ivan Petrovich Sidorov").family == "Sidorov" + + +@pytest.mark.parametrize("base_order", [FAMILY_FIRST, FAMILY_FIRST_GIVEN_LAST]) +def test_an_east_slavic_group_moves_a_family_first_base_to_given_first( + base_order: tuple[Role, Role, Role]) -> None: + # the rule stands down under a family-first order, so it brings a + # given-first one with it, whatever the names' order votes say + s = Parser(policy=Policy(name_order=base_order)).suggest( + ["Сидоров Иван Петрович", "Иванов Пётр Сергеевич", + "Петров Олег Иванович", "Смирнов Павел Андреевич", + "Попов Илья Ильич", "Kim Minjun"]) + assert s.patch.patronymic_rules == frozenset({PatronymicRule.EAST_SLAVIC}) + assert s.patch.name_order == GIVEN_FIRST + order = _order_evidence(s) + assert order.decided + assert (order.votes_for, order.votes_against) == (0, 1) + assert s.parser.parse("Ivan Petrovich Sidorov").family == "Sidorov" + assert s.parser.parse("Sidorov Ivan Petrovich").family == "Sidorov" + + +@pytest.mark.parametrize("names, rule", [ + (_EAST_SLAVIC, PatronymicRule.EAST_SLAVIC), + (_TURKIC, PatronymicRule.TURKIC)]) +def test_a_patronymic_rule_needs_five_names_carrying_it( + names: list[str], rule: PatronymicRule) -> None: + # _MIN_PATRONYMIC_VOTES = 5: two to four Russian names written + # family-first moved a family-first base to given-first in 8-20% of + # groups, so a rule needs five names however large their share + base = Parser(policy=Policy(name_order=FAMILY_FIRST)) + for size in (2, 3, 4): + s = base.suggest(names[:size]) + assert s.patch.patronymic_rules is UNSET, size + assert not _patronymic_evidence(s).decided, size + assert s.patch.name_order is UNSET, size + s = base.suggest(names) + assert s.patch.patronymic_rules == frozenset({rule}) + + +def test_one_patronymic_among_given_first_names_decides_nothing() -> None: + s = Parser().suggest(["Jason Priem", "Mary Smith", "Anna Jones", + "Jason Smith", "Mary Priem", + "Сидоров Иван Петрович"]) + assert s.patch.patronymic_rules is UNSET + assert not _patronymic_evidence(s).decided + assert _order_evidence(s).decided + assert _order_evidence(s).value == GIVEN_FIRST + + +def test_patronymics_need_more_than_four_in_five_of_the_voting_names( + ) -> None: + # _MIN_PATRONYMIC_SHARE = 0.8: the rule forces a given-first order, + # so a family-first group whose names half carry a patronymic + # (PatentsView's Russian inventors) must not be reordered. + # Four of five voting names: four in five, not more + s = Parser().suggest(["Jason Priem", *_EAST_SLAVIC[:4]]) + assert s.patch.patronymic_rules is UNSET + assert not _patronymic_evidence(s).decided + # five of six: more than four in five + s = Parser().suggest(["Jason Priem", *_EAST_SLAVIC]) + assert s.patch.patronymic_rules == frozenset({PatronymicRule.EAST_SLAVIC}) + + +def test_a_turkic_rule_needs_more_than_four_in_five_of_the_voting_names( + ) -> None: + # the same share as the East Slavic rule's: four of five voting + # names is four in five, not more + s = Parser().suggest(["Jason Priem", *_TURKIC[:4]]) + assert s.patch.patronymic_rules is UNSET + assert not _patronymic_evidence(s).decided + # five of six: more than four in five + s = Parser().suggest(["Jason Priem", *_TURKIC]) + assert s.patch.patronymic_rules == frozenset({PatronymicRule.TURKIC}) + + +def test_a_turkic_group_gets_the_rule_alone() -> None: + s = Parser().suggest(_TURKIC) + assert s.patch.patronymic_rules == frozenset({PatronymicRule.TURKIC}) + # the rule reads either order, so it forces no order of its own + assert s.patch.name_order is UNSET + patronymic = _patronymic_evidence(s) + assert patronymic.decided + assert (Signal.PATRONYMIC, PatronymicRule.TURKIC, 5) in patronymic.by_signal + + +@pytest.mark.parametrize("text", [ + "Mamedov Ali Huseyn oglu", "Ali Huseyn oglu Mamedov"]) +def test_a_turkic_group_reads_the_family_whichever_end_it_is_at( + text: str) -> None: + s = Parser().suggest([text, "Aliyev Rauf Ilham oglu", + "Rauf Ilham oglu Aliyev", *_TURKIC[2:]]) + name = s.parser.parse(text) + assert name.family == "Mamedov" + assert name.given == "Ali" + assert name.patronymic == "Huseyn oglu" + + +def test_a_turkic_group_keeps_the_order_its_own_votes_decide() -> None: + names = ["NƏBİYEV Rizvan Novruz oğlu", "HƏSƏNOVA Afət Əbil qızı", + "AĞAZADƏ Əminə Həmid qızı", "MİKAYILOVA Sevil Əlirazı qızı", + "QULİYEV Elman Rasim oğlu", "BÜLBÜLOĞLU Polad"] + s = Parser().suggest(names) + assert s.patch.patronymic_rules == frozenset({PatronymicRule.TURKIC}) + assert s.patch.name_order == FAMILY_FIRST + assert s.parser.parse("BÜLBÜLOĞLU Polad").family == "BÜLBÜLOĞLU" + + +def test_a_turkic_group_on_a_family_first_base_follows_its_names() -> None: + base = Parser(policy=Policy(name_order=FAMILY_FIRST)) + s = base.suggest(["Ali Huseyn oglu Mamedov", "Leyla Rasim qizi Aliyeva", + "Tural Ilham oglu Aliyev", "Aysel Rauf qizi Quliyeva", + "Elman Rasim oglu Quliyev"]) + assert s.patch.patronymic_rules == frozenset({PatronymicRule.TURKIC}) + # the rule forces no order, and the names' own votes decide it + assert s.patch.name_order == GIVEN_FIRST + assert s.parser.parse("Ali Huseyn oglu Mamedov").family == "Mamedov" + + +def test_a_base_holding_the_turkic_rule_leaves_the_patch_empty() -> None: + base = Parser(policy=Policy( + patronymic_rules=frozenset({PatronymicRule.TURKIC}))) + s = base.suggest(_TURKIC) + assert s.patch.patronymic_rules is UNSET + assert _patronymic_evidence(s).decided + + +def test_a_base_holding_the_rule_already_leaves_the_patch_empty() -> None: + base = Parser(policy=Policy( + patronymic_rules=frozenset({PatronymicRule.EAST_SLAVIC}))) + s = base.suggest(_EAST_SLAVIC) + assert s.patch == PolicyPatch() + assert _patronymic_evidence(s).decided + + +def test_comma_names_cast_no_vote() -> None: + s = Parser().suggest(["Priem, Jason", "Smith, John"]) + assert s.patch == PolicyPatch() + assert _order_evidence(s).no_evidence == 2 + + +def test_disagreeing_lists_the_names_against_the_answer() -> None: + s = Parser().suggest(["Priem Jason", "Kim Minjun", "Wang Xiaoming", + "Jason Smith"]) + assert [n.text for n in s.disagreeing()] == ["Jason Smith"] + + +def test_a_name_with_no_votes_never_disagrees() -> None: + s = Parser().suggest(["Priem Jason", "Kim Minjun", "Wang Xiaoming", + "Jason Smith", "Xyzzy Plugh", "Smith, Jason"]) + assert s.patch == PolicyPatch(name_order=FAMILY_FIRST) + assert [n.text for n in s.disagreeing()] == ["Jason Smith"] + + +def test_names_keeps_every_input_in_order_including_blanks_and_duplicates( + ) -> None: + texts = ["Priem Jason", "", "Priem Jason", " ", "Kim Minjun"] + s = Parser().suggest(texts) + assert [n.text for n in s.names] == texts + assert s.names[1].votes == {} + assert s.names[0].votes == {Signal.GIVEN_NAME: FAMILY_FIRST} + assert _order_evidence(s).no_evidence == 2 + + +def test_a_generator_is_consumed_once() -> None: + pulled: list[str] = [] + + def names() -> Iterator[str]: + for text in ["Priem Jason", "Kim Minjun"]: + pulled.append(text) + yield text + + s = Parser().suggest(names()) + assert pulled == ["Priem Jason", "Kim Minjun"] + assert [n.text for n in s.names] == pulled + assert s.patch.name_order == FAMILY_FIRST + + +@pytest.mark.parametrize("bad, fragment", [ + ("Priem Jason", "bare string: 'Priem Jason'"), + ({"Priem Jason": 1}, "not a mapping (dict)"), + (b"Priem Jason", "decode"), + (5, "got 5"), + ([1], "item 0 is 1"), + (["Priem Jason", None], "item 1 is None"), +]) +def test_a_wrong_shape_raises_type_error_with_the_fix( + bad: object, fragment: str) -> None: + with pytest.raises(TypeError) as info: + Parser().suggest(bad) # type: ignore[arg-type] + message = str(info.value) + assert fragment in message + if not isinstance(bad, bytes): + assert "parser.suggest([" in message + + +def test_a_mapping_error_does_not_embed_the_mapping() -> None: + names = {f"Name{i} Surname{i}": i for i in range(10_000)} + with pytest.raises(TypeError) as info: + Parser().suggest(names) # type: ignore[arg-type] + assert "Name9999" not in str(info.value) + assert len(str(info.value)) < 300 + + +def test_the_offered_fix_runs() -> None: + with pytest.raises(TypeError) as info: + Parser().suggest("Priem Jason") # type: ignore[arg-type] + offered = str(info.value).rsplit(": ", 1)[1] + assert offered.startswith("parser.suggest(") + result = eval(offered, {"parser": Parser()}) + assert isinstance(result, Suggestion) + + +def test_suggest_never_mutates_the_base() -> None: + base = Parser(policy=Policy(name_order=FAMILY_FIRST)) + before = (base.policy, base.lexicon) + s = base.suggest(["Jason Priem", "Minjun Kim", "Сидоров Иван Петрович"]) + assert (base.policy, base.lexicon) == before + assert s.base is base + assert s.parser.lexicon is base.lexicon + + +def test_the_parser_keeps_the_base_segmenter() -> None: + base = Parser(segmenter=_decline) + s = base.suggest(["Priem Jason", "Kim Minjun"]) + assert s.parser.segmenter is _decline + assert s.parser.policy.name_order == FAMILY_FIRST + + +def test_the_parser_does_not_warn_again_for_its_base() -> None: + # a JA base without a segmenter warns its caller once, when it is + # built; the suggested parser is built from it and must not warn + # again, on every access + from nameparser import locales, parser_for + with pytest.warns(UserWarning): + base = parser_for(locales.JA) + s = base.suggest(["Priem Jason", "Kim Minjun"]) + with warnings.catch_warnings(): + warnings.simplefilter("error") + parser = s.parser + assert s.parser.policy.name_order == FAMILY_FIRST + assert parser.lexicon is base.lexicon + + +def test_a_patch_that_moves_segment_scripts_still_warns_its_caller() -> None: + # suggest() never patches segment_scripts, but a hand-built + # Suggestion may, and a warning that patch itself causes (Han + # segmentation with no Chinese surnames listed) is new to the + # caller, so .parser must pass it on, attributed to the caller + from nameparser import PolicyPatch, Script, Suggestion + # a patch field needs a decided evidence holding the value it makes + # (the cross-check Suggestion runs since the review's fix wave) + value = Parser().policy.segment_scripts | {Script.HAN} + s = Suggestion(Parser(), PolicyPatch( + segment_scripts=frozenset({Script.HAN})), (SettingEvidence( + "segment_scripts", value, True, 0, 0, 0, 0, ()),), ()) # type: ignore[arg-type] + with pytest.warns(UserWarning) as caught: + s.parser + assert caught[0].filename == __file__ + + +def test_name_evidence_votes_are_read_only() -> None: + evidence = NameEvidence("Priem Jason", {Signal.GIVEN_NAME: FAMILY_FIRST}) + with pytest.raises(TypeError): + evidence.votes[Signal.SURNAME] = FAMILY_FIRST # type: ignore[index] + assert evidence == NameEvidence( + "Priem Jason", {Signal.GIVEN_NAME: FAMILY_FIRST}) + + +def test_the_report_names_the_setting_and_the_disagreeing_names() -> None: + s = Parser().suggest(["Priem Jason", "Kim Minjun", "Wang Xiaoming", + "Jason Smith"]) + report = str(s) + assert "Suggested for 4 names" in report + assert "name_order" in report and "FAMILY_FIRST" in report + assert "Jason Smith" in report + + +def test_the_new_types_are_exported() -> None: + import nameparser + for name in ("Suggestion", "NameEvidence", "SettingEvidence", "Signal"): + assert name in nameparser.__all__ + assert getattr(nameparser, name) is not None + + +# -- one order vote per name (fix round 1) -------------------------------- + + +def test_one_heavily_marked_name_does_not_outvote_two() -> None: + # `KIM Jason J.` carries three family-first signals, but is one name + s = Parser().suggest(["Jason Smith", "Mary Jones", "Anna Brown", + "KIM Jason J."]) + assert s.patch.name_order is UNSET + order = _order_evidence(s) + assert order.decided + assert (order.votes_for, order.votes_against) == (3, 1) + assert (Signal.CAPITALS, FAMILY_FIRST, 1) in order.by_signal + assert (Signal.TRAILING_INITIAL, FAMILY_FIRST, 1) in order.by_signal + + +def test_two_signal_names_count_once_each() -> None: + # per name 2 of 3 for family-first, which is not more than two + # thirds; counted per signal it would be 4 of 5, and decide + s = Parser().suggest(["Jason Smith", "van Berg Mary", "van Berg Anna"]) + assert s.patch == PolicyPatch() + order = _order_evidence(s) + assert not order.decided + assert (order.votes_for, order.votes_against) == (1, 2) + assert s.disagreeing() == () + + +def test_a_name_whose_signals_conflict_is_contested() -> None: + # `Jason Kim J.`: the trailing initial says family-first, the given + # name at the front says given-first + s = Parser().suggest(["Priem Jason", "Kim Minjun", "Jason Kim J."]) + assert s.patch.name_order == FAMILY_FIRST + order = _order_evidence(s) + assert (order.votes_for, order.votes_against, order.contested, + order.no_evidence) == (2, 0, 1, 0) + assert s.disagreeing() == () + # the signal breakdown still holds both of its votes + assert (Signal.GIVEN_NAME, GIVEN_FIRST, 1) in order.by_signal + assert (Signal.TRAILING_INITIAL, FAMILY_FIRST, 1) in order.by_signal + + +@pytest.mark.parametrize("names", [ + [], + ["Xyzzy Plugh"], + ["Priem Jason", "Jason Kim J.", "Smith, John", "", "Jason Smith"], + ["Jason Smith", "Mary Jones", "KIM Jason J."], + ["Сидоров Иван Петрович", "SIDOROV Ivan Petrovich", "Kim Minjun"], + ["Mamedov Ali Huseyn oglu", "Jason Kim J.", "Priem Jason"], +]) +def test_the_counts_add_up_to_the_group(names: list[str]) -> None: + s = Parser().suggest(names) + for e in s.evidence: + assert (e.votes_for + e.votes_against + e.contested + + e.no_evidence) == len(names), e + + +def test_an_east_slavic_name_never_disagrees_on_order() -> None: + # `SIDOROV Ivan Petrovich` votes family-first by its capitals, but + # the decided East Slavic rule reorders it + s = Parser().suggest([*_EAST_SLAVIC, "SIDOROV Ivan Petrovich"]) + assert s.patch.patronymic_rules == frozenset({PatronymicRule.EAST_SLAVIC}) + assert _order_evidence(s).votes_against == 1 + assert s.disagreeing() == () + + +def test_a_non_east_slavic_name_still_disagrees_in_an_east_slavic_group( + ) -> None: + s = Parser().suggest([*_EAST_SLAVIC, "Kim Minjun"]) + assert [n.text for n in s.disagreeing()] == ["Kim Minjun"] + + +# -- eager validation of the result types ----------------------------------- + + +@pytest.mark.parametrize("votes, error, fragment", [ + ({Signal.SURNAME: "family-first"}, TypeError, "must be an order"), + ({Signal.SURNAME: FAMILY_FIRST_GIVEN_LAST}, ValueError, "GIVEN_FIRST or FAMILY_FIRST"), + ({Signal.SURNAME: PatronymicRule.TURKIC}, TypeError, "must be an order"), + ({Signal.PATRONYMIC: FAMILY_FIRST}, TypeError, "PatronymicRule"), + ({Signal.PATRONYMIC: "turkic"}, TypeError, "PatronymicRule"), + ({"no-such-signal": FAMILY_FIRST}, ValueError, "no-such-signal"), +]) +def test_name_evidence_refuses_a_vote_of_the_wrong_kind( + votes: dict[object, object], error: type[Exception], + fragment: str) -> None: + with pytest.raises(error, match=fragment): + NameEvidence("Priem Jason", votes) # type: ignore[arg-type] + + +def test_name_evidence_accepts_an_order_written_as_a_list() -> None: + evidence = NameEvidence("Priem Jason", {Signal.SURNAME: list(FAMILY_FIRST)}) # type: ignore[dict-item] + assert evidence.votes[Signal.SURNAME] == FAMILY_FIRST + + +def _setting_evidence(**changes: object) -> SettingEvidence: + fields: dict[str, object] = dict( + setting="name_order", value=FAMILY_FIRST, decided=True, votes_for=1, + votes_against=0, contested=0, no_evidence=0, by_signal=()) + fields.update(changes) + return SettingEvidence(**fields) # type: ignore[arg-type] + + +@pytest.mark.parametrize("changes, error, fragment", [ + ({"decided": 1}, TypeError, "decided must be a bool, got 1"), + ({"decided": "yes"}, TypeError, "decided must be a bool"), + ({"votes_for": -1}, ValueError, "votes_for must not be negative, got -1"), + ({"votes_against": -2}, ValueError, "votes_against must not be negative"), + ({"contested": -1}, ValueError, "contested must not be negative"), + ({"no_evidence": -1}, ValueError, "no_evidence must not be negative"), + ({"votes_for": 1.5}, TypeError, "votes_for must be an int"), + ({"no_evidence": True}, TypeError, "no_evidence must be an int"), + ({"setting": 5}, TypeError, "setting must be a Policy field name"), + ({"note": None}, TypeError, "note must be a str"), +]) +def test_setting_evidence_refuses_bad_counts_and_flags( + changes: dict[str, object], error: type[Exception], + fragment: str) -> None: + with pytest.raises(error, match=fragment): + _setting_evidence(**changes) + + +def test_setting_evidence_accepts_a_well_formed_value() -> None: + assert _setting_evidence().votes_for == 1 + + +@pytest.mark.parametrize("changes, error, fragment", [ + ({"setting": "no_such_field"}, ValueError, + "setting must name a Policy field, got 'no_such_field'"), + ({"setting": ""}, ValueError, "must name a Policy field"), + ({"by_signal": "capitals"}, TypeError, "by_signal must be an iterable"), + ({"by_signal": 5}, TypeError, "by_signal must be an iterable"), + ({"by_signal": [(Signal.SURNAME, FAMILY_FIRST)]}, TypeError, + r"\(signal, vote, count\)"), + ({"by_signal": ["abc"]}, TypeError, r"\(signal, vote, count\)"), + ({"by_signal": [("no-such", FAMILY_FIRST, 1)]}, ValueError, "no-such"), + ({"by_signal": [(Signal.SURNAME, "family-first", 1)]}, TypeError, + "must be an order"), + ({"by_signal": [(Signal.PATRONYMIC, FAMILY_FIRST, 1)]}, TypeError, + "PatronymicRule"), + ({"by_signal": [(Signal.SURNAME, FAMILY_FIRST, -1)]}, ValueError, + "count must not be negative, got -1"), + ({"by_signal": [(Signal.SURNAME, FAMILY_FIRST, True)]}, TypeError, + "count must be an int"), +]) +def test_setting_evidence_refuses_a_bad_setting_or_row( + changes: dict[str, object], error: type[Exception], + fragment: str) -> None: + with pytest.raises(error, match=fragment): + _setting_evidence(**changes) + + +def test_setting_evidence_keeps_well_formed_rows() -> None: + evidence = _setting_evidence( + by_signal=[["surname", list(FAMILY_FIRST), 2]]) + assert evidence.by_signal == ((Signal.SURNAME, FAMILY_FIRST, 2),) + + +def test_a_valid_setting_evidence_renders_in_a_report() -> None: + s = Suggestion(Parser(), PolicyPatch(), (_setting_evidence( + setting="patronymic_rules", value=frozenset(), decided=False, + votes_for=0),), ()) + assert "patronymic_rules" in str(s) + + +@pytest.mark.parametrize("changes, error, fragment", [ + ({"base": Policy()}, TypeError, "base must be a Parser"), + ({"base": None}, TypeError, "base must be a Parser"), + ({"patch": {}}, TypeError, "patch must be a PolicyPatch"), + ({"evidence": "name_order"}, TypeError, + "evidence must be a tuple of SettingEvidence"), + ({"evidence": [object()]}, TypeError, "evidence must hold SettingEvidence"), + ({"names": "Priem Jason"}, TypeError, + "names must be a tuple of NameEvidence"), + ({"names": None}, TypeError, "names must be a tuple of NameEvidence"), + ({"names": ["Priem Jason"]}, TypeError, "names must hold NameEvidence"), +]) +def test_suggestion_refuses_fields_of_the_wrong_type( + changes: dict[str, object], error: type[Exception], + fragment: str) -> None: + fields: dict[str, object] = dict( + base=Parser(), patch=PolicyPatch(), evidence=(), names=()) + fields.update(changes) + with pytest.raises(error, match=fragment): + Suggestion(**fields) # type: ignore[arg-type] + + +def _decline(token: str) -> None: + return None + + +# -- the review's fix wave: where a signal reads (A1-A4, A8) ---------------- + +_OVICH_SURNAMES = ["Abramovich Roman", "Rabinovich Pavel", + "Shostakovich Dmitri", "Lukashevich Anton", + "Karpovich Igor", "Yurkevich Oleg"] + + +@pytest.mark.parametrize("text", [ + *_OVICH_SURNAMES, "Roman Abramovich", + # rules/O1 reads three name words, the ending never on the first + "Petrovich Ivan Sidorov", "Ivan Sidorov Petrovich Ivanovich", + # rules/O2 reads four name words, the marker never first or second + "Ali Huseyn oglu", "oglu Ali Huseyn Mamedov", "Ali oglu Huseyn Mamedov", +]) +def test_a_patronymic_ending_the_pipeline_cannot_read_casts_no_vote( + text: str) -> None: + assert Signal.PATRONYMIC not in name_votes(text, L) + + +@pytest.mark.parametrize("text, rule", [ + ("Sidorov Ivan Petrovich", PatronymicRule.EAST_SLAVIC), + ("Ivan Petrovich Sidorov", PatronymicRule.EAST_SLAVIC), + ("Иванов Иван Петрович", PatronymicRule.EAST_SLAVIC), + ("Dr. Ivan Petrovich Sidorov", PatronymicRule.EAST_SLAVIC), + ("Sidorov Ivan Petrovich Jr.", PatronymicRule.EAST_SLAVIC), + # a surname that itself ends like a patronymic: rules/O1 asks only + # about the middle and last words, so the rule reads these too + ("Abramovich Roman Arkadyevich", PatronymicRule.EAST_SLAVIC), + ("Лукашевич Антон Сергеевич", PatronymicRule.EAST_SLAVIC), + ("Mamedov Ali Huseyn oglu", PatronymicRule.TURKIC), + ("Ali Huseyn oglu Mamedov", PatronymicRule.TURKIC), +]) +def test_a_patronymic_the_pipeline_reads_still_votes( + text: str, rule: PatronymicRule) -> None: + assert name_votes(text, L)[Signal.PATRONYMIC] is rule + + +def test_ovich_surnames_keep_a_family_first_base() -> None: + base = Parser(policy=Policy(name_order=FAMILY_FIRST)) + s = base.suggest(_OVICH_SURNAMES) + assert s.patch.patronymic_rules is UNSET + assert s.parser.policy.name_order == FAMILY_FIRST + assert s.parser.parse("Abramovich Roman").family == "Abramovich" + + +_CAPITALS_EAST_SLAVIC = [ + "SIDOROV Ivan Petrovich", "IVANOV Petr Sergeevich", + "PETROV Oleg Ivanovich", "SMIRNOV Pavel Andreevich", "POPOV Ilya Ilyich"] + + +@pytest.mark.parametrize("base_order", [GIVEN_FIRST, FAMILY_FIRST]) +def test_a_decided_rule_never_overrides_an_order_decided_against_it( + base_order: tuple[Role, Role, Role]) -> None: + # five East Slavic names whose capitals all write the family first: + # the rule is decided, and so is the family-first order, which the + # rule's given-first order does not override + s = Parser(policy=Policy(name_order=base_order)).suggest( + _CAPITALS_EAST_SLAVIC) + assert _patronymic_evidence(s).decided + assert s.parser.policy.patronymic_rules == frozenset( + {PatronymicRule.EAST_SLAVIC}) + order = _order_evidence(s) + assert order.decided and order.value == FAMILY_FIRST + assert (order.votes_for, order.votes_against) == (5, 0) + assert "East Slavic" in order.note and "stands down" in order.note + assert s.parser.policy.name_order == FAMILY_FIRST + assert s.parser.parse("SIDOROV Ivan Petrovich").family == "SIDOROV" + assert s.disagreeing() == () + + +@pytest.mark.parametrize("text, signal, value", [ + ("Dr. Zhang Xiaoming", Signal.SURNAME, FAMILY_FIRST), + ("Prof. Pham Minh Duc", Signal.SURNAME, FAMILY_FIRST), + ("Mr Kim Minjun", Signal.SURNAME, FAMILY_FIRST), + ("Zhang Xiaoming PhD", Signal.SURNAME, FAMILY_FIRST), + ("Kim Minjun Jr.", Signal.SURNAME, FAMILY_FIRST), + ("Kim Minjun III", Signal.SURNAME, FAMILY_FIRST), +]) +def test_titles_and_post_nominals_are_read_past( + text: str, signal: Signal, value: object) -> None: + assert name_votes(text, L)[signal] == value + + +@pytest.mark.parametrize("text, signal", [ + ("Tanaka Hiroshi FACS", Signal.CAPITALS), + ("Tanaka Hiroshi MBBS", Signal.CAPITALS), + ("John Smith V", Signal.TRAILING_INITIAL), + ("John Smith II", Signal.TRAILING_INITIAL), +]) +def test_a_post_nominal_casts_no_vote(text: str, signal: Signal) -> None: + assert signal not in name_votes(text, L) + + +@pytest.mark.parametrize("text", ["Armitage V", "Smith J. V"]) +def test_a_one_letter_numeral_without_words_to_spare_is_an_initial( + text: str) -> None: + # the trailing fork reads a one-letter numeral as the generation + # only bare, with a word to spare and no initial in front + assert name_votes(text, L)[Signal.TRAILING_INITIAL] == FAMILY_FIRST + + +def test_titled_chinese_names_keep_a_family_first_base() -> None: + base = Parser(policy=Policy(name_order=FAMILY_FIRST)) + s = base.suggest(["Dr. Zhang Wei", "Dr. Wang Fang", "Prof. Li Jing", + "Dr. Liu Lei", "Prof. Chen Yang", "Dr. Huang Hong"]) + assert s.patch.name_order is UNSET + assert s.parser.policy.name_order == FAMILY_FIRST + + +@pytest.mark.parametrize("text, signal, value", [ + ("Zhang Xiaoming, PhD", Signal.SURNAME, FAMILY_FIRST), + ("Kim Minjun, Jr., MD", Signal.SURNAME, FAMILY_FIRST), + ("Sidorov Ivan Petrovich, PhD", Signal.PATRONYMIC, + PatronymicRule.EAST_SLAVIC), +]) +def test_a_comma_before_post_nominals_alone_reads_the_name_before_it( + text: str, signal: Signal, value: object) -> None: + assert name_votes(text, L)[signal] == value + + +@pytest.mark.parametrize("text", ["Priem, Jason", "Kim, Minjun Jr.", + "Zhang Xiaoming, Beijing"]) +def test_any_other_comma_name_casts_nothing(text: str) -> None: + assert name_votes(text, L) == {} + + +def test_the_order_note_counts_the_comma_names_left_out() -> None: + s = Parser().suggest(["Priem, Jason", "Smith, John", "Kim Minjun", + "Zhang Xiaoming, PhD", "Wang Xiaoming"]) + order = _order_evidence(s) + assert "2 names with a comma" in order.note + # Kim Minjun, Wang Xiaoming and the name before `, PhD` + assert order.votes_for == 3 + s = Parser().suggest(["Kim Minjun", "Wang Xiaoming"]) + assert "comma" not in _order_evidence(s).note + + +@pytest.mark.parametrize("word, folded", [ + ("Łukasz", "lukasz"), ("łukasz", "lukasz"), ("Søren", "soren"), + ("ØYSTEIN", "oystein"), ("Barış", "baris"), ("Æsa", "aesa"), + ("Œuvre", "oeuvre"), ("Strauß", "strauss"), ("Đặng", "dang"), + ("Lǚ", "lv"), ("Nguyễn", "nguyen")]) +def test_fold_maps_the_letters_nfkd_leaves_whole(word: str, folded: str) -> None: + from nameparser._suggest import fold + assert fold(word) == folded + + +@pytest.mark.parametrize("text", [ + "Lukasz Kowalski", "Łukasz Kowalski", "Søren Hansen", "Soren Hansen", + "Anıl Kaya", "Anil Kaya"]) +def test_a_stroked_or_dotless_given_name_votes_either_way_written( + text: str) -> None: + assert name_votes(text, L)[Signal.GIVEN_NAME] == GIVEN_FIRST + + +@pytest.mark.parametrize("pair", [ + ("Barış Yilmaz", "Baris Yilmaz"), ("Yilmaz Barış", "Yilmaz Baris"), + ("Małgorzata Nowak", "Malgorzata Nowak"), + ("Strauß Øystein", "Strauss Oystein")]) +def test_a_name_reads_alike_written_with_or_without_its_letters( + pair: tuple[str, str]) -> None: + # `baris` is not in the table: ORCID's holders wrote it in the + # family box 8 times of 89, more than the filter's 5%, once Barış + # and Baris count as one spelling; both spellings still read alike + written, typed = pair + assert name_votes(written, L) == name_votes(typed, L) + + +@pytest.mark.parametrize("text, signal, value", [ + ("Petrov Ivan Ivanovich.", Signal.PATRONYMIC, PatronymicRule.EAST_SLAVIC), + ('"Kim" Minjun', Signal.SURNAME, FAMILY_FIRST), + ("(Kim) Minjun", Signal.SURNAME, FAMILY_FIRST), + ("Minjun 'Kim'", Signal.SURNAME, GIVEN_FIRST), + ("Mamedov Ali Huseyn (oglu)", Signal.PATRONYMIC, PatronymicRule.TURKIC), +]) +def test_edge_punctuation_is_read_past( + text: str, signal: Signal, value: object) -> None: + assert name_votes(text, L)[signal] == value + + +# -- the review's fix wave: the result types (B1-B6) ------------------------ + + +def _order(**changes: object) -> SettingEvidence: + fields: dict[str, object] = dict( + setting="name_order", value=GIVEN_FIRST, decided=False, + votes_for=0, votes_against=0, contested=0, no_evidence=0, + by_signal=()) + fields.update(changes) + return SettingEvidence(**fields) # type: ignore[arg-type] + + +def _rules(**changes: object) -> SettingEvidence: + fields: dict[str, object] = dict( + setting="patronymic_rules", value=frozenset(), decided=False, + votes_for=0, votes_against=0, contested=0, no_evidence=0, + by_signal=()) + fields.update(changes) + return SettingEvidence(**fields) # type: ignore[arg-type] + + +@pytest.mark.parametrize("patch, evidence, fragment", [ + # (a) one SettingEvidence per setting + (PolicyPatch(), (_order(), _order()), "more than once"), + # (b) a decided value is the suggested parser's + (PolicyPatch(), (_order(decided=True, value=FAMILY_FIRST),), + "decided name_order value"), + (PolicyPatch(name_order=FAMILY_FIRST), + (_order(decided=True, value=GIVEN_FIRST),), "decided name_order value"), + # (c) an undecided setting is not in the patch + (PolicyPatch(name_order=FAMILY_FIRST), (_order(value=FAMILY_FIRST),), + "undecided"), + # (d) a patch field is a decided setting that changes the base + (PolicyPatch(middle_as_family=True), (_order(),), "middle_as_family"), + (PolicyPatch(name_order=GIVEN_FIRST), + (_order(decided=True, value=GIVEN_FIRST),), "the base's own"), + (PolicyPatch(patronymic_rules=frozenset()), + (_rules(decided=True),), "the base's own"), +]) +def test_suggestion_refuses_evidence_its_patch_contradicts( + patch: PolicyPatch, evidence: tuple[SettingEvidence, ...], + fragment: str) -> None: + with pytest.raises(ValueError, match=fragment): + Suggestion(Parser(), patch, evidence, ()) + + +def test_suggestion_refuses_counts_that_do_not_add_up_to_the_group() -> None: + # (e) every setting's four counts are of the group's names + names = (NameEvidence("Kim Minjun", {Signal.SURNAME: FAMILY_FIRST}),) + with pytest.raises(ValueError, match="add up to 2, not the 1 names"): + Suggestion(Parser(), PolicyPatch(), + (_order(votes_for=1, no_evidence=1),), names) + s = Suggestion(Parser(), PolicyPatch(), (_order(votes_against=1),), names) + assert s.names == names + + +def test_a_decided_change_is_accepted_with_its_patch() -> None: + names = tuple(NameEvidence(t, {Signal.SURNAME: FAMILY_FIRST}) + for t in ("Kim Minjun", "Wang Xiaoming")) + s = Suggestion(Parser(), PolicyPatch(name_order=FAMILY_FIRST), + (_order(decided=True, value=FAMILY_FIRST, votes_for=2),), + names) + assert s.parser.policy.name_order == FAMILY_FIRST + + +def test_every_suggestion_suggest_returns_passes_its_own_checks() -> None: + import dataclasses + for group in (_EAST_SLAVIC, _TURKIC, _CAPITALS_EAST_SLAVIC, + ["Priem Jason", "Kim Minjun", "Smith, John"]): + for base in (Parser(), Parser(policy=Policy(name_order=FAMILY_FIRST)), + Parser(policy=Policy(patronymic_rules=frozenset( + {PatronymicRule.TURKIC})))): + s = base.suggest(group) + assert dataclasses.replace(s) == s + + +def test_a_vote_is_the_shared_order_constant() -> None: + evidence = NameEvidence("Priem Jason", {Signal.SURNAME: list(FAMILY_FIRST)}) # type: ignore[dict-item] + assert evidence.votes[Signal.SURNAME] is FAMILY_FIRST + row = _order(by_signal=[(Signal.SURNAME, list(GIVEN_FIRST), 0)]).by_signal[0] + assert row[1] is GIVEN_FIRST + + +@pytest.mark.parametrize("setting, row", [ + ("name_order", (Signal.PATRONYMIC, PatronymicRule.EAST_SLAVIC, 1)), + ("patronymic_rules", (Signal.SURNAME, FAMILY_FIRST, 1)), + ("middle_as_family", (Signal.SURNAME, FAMILY_FIRST, 1)), +]) +def test_setting_evidence_refuses_a_signal_of_another_setting( + setting: str, row: tuple[object, ...]) -> None: + with pytest.raises(ValueError, match="votes on"): + _setting_evidence(setting=setting, by_signal=[row]) + + +def test_name_evidence_survives_replace() -> None: + import dataclasses + evidence = NameEvidence("Priem Jason", {Signal.GIVEN_NAME: FAMILY_FIRST}) + moved = dataclasses.replace(evidence, text="Priem J.") + assert moved.text == "Priem J." + assert moved.votes == evidence.votes + assert dataclasses.replace( # type: ignore[call-arg] + evidence, votes={Signal.SURNAME: GIVEN_FIRST}).votes == { + Signal.SURNAME: GIVEN_FIRST} + + +@pytest.mark.parametrize("args", [ + (5, {Signal.SURNAME: FAMILY_FIRST}), ("Kim Minjun", [Signal.SURNAME])]) +def test_name_evidence_type_errors_offer_code_that_runs( + args: tuple[object, object]) -> None: + with pytest.raises(TypeError) as info: + NameEvidence(*args) # type: ignore[arg-type] + offered = str(info.value).rsplit("pass e.g. ", 1)[1] + assert offered == "NameEvidence('Kim Minjun', {Signal.SURNAME: FAMILY_FIRST})" + built = eval(offered, {"NameEvidence": NameEvidence, "Signal": Signal, + "FAMILY_FIRST": FAMILY_FIRST}) + assert built.votes == {Signal.SURNAME: FAMILY_FIRST} + + +def test_the_parser_passes_on_only_the_warnings_its_patch_causes() -> None: + # a base whose Han and hiragana segmentation has no segmenter warns + # once, when it is built; a hand-built patch adding Hangul, whose + # surnames ship, causes no warning of its own, so .parser repeats + # none, though the patched construction warns as the base's did + import dataclasses + from nameparser import Script, locales, parser_for + with pytest.warns(UserWarning): + ja = parser_for(locales.JA) + base = dataclasses.replace(ja, policy=dataclasses.replace( + ja.policy, segment_scripts=frozenset({Script.HAN, + Script.HIRAGANA}))) + value = base.policy.segment_scripts | {Script.HANGUL} + s = Suggestion(base, PolicyPatch( + segment_scripts=frozenset({Script.HANGUL})), (SettingEvidence( + "segment_scripts", value, True, 0, 0, 0, 0, ()),), ()) # type: ignore[arg-type] + with warnings.catch_warnings(): + warnings.simplefilter("error") + parser = s.parser + assert parser.policy.segment_scripts == value + + +# -- the review's fix wave: the decision rule's tests (C1-C6) ---------------- + +_ES_SURNAMES = ["Sidorov", "Ivanov", "Petrov", "Smirnov", "Popov", "Volkov", + "Kozlov", "Lebedev", "Novikov", "Morozov", "Sokolov"] +_ES_GIVEN = ["Ivan", "Petr", "Oleg", "Pavel", "Ilya", "Sergei", "Anton", + "Boris", "Denis", "Egor", "Fedor"] + + +def _east_slavic(n: int) -> list[str]: + return [f"{_ES_SURNAMES[i]} {_ES_GIVEN[i]} Andreevich" for i in range(n)] + + +def _turkic(n: int) -> list[str]: + return [f"{_ES_SURNAMES[i]} {_ES_GIVEN[i]} Rasim oglu" for i in range(n)] + + +_OTHERS = ["Jason Priem", "Mary Smith", "Anna Jones", "Peter Brown", + "Sarah Wilson"] + + +@pytest.mark.parametrize("make, rule", [ + (_east_slavic, PatronymicRule.EAST_SLAVIC), (_turkic, PatronymicRule.TURKIC)]) +@pytest.mark.parametrize("carrying, others, decided", [ + (8, 2, False), # 8 of 10 is four in five, not more + (9, 2, True), # 9 of 11 is more + (5, 2, False), # five names, but 5 of 7 is under the share +]) +def test_the_patronymic_share_alone_decides( + make: object, rule: PatronymicRule, carrying: int, others: int, + decided: bool) -> None: + names = make(carrying) + _OTHERS[:others] # type: ignore[operator] + assert len(set(names)) == len(names) + s = Parser().suggest(names) + assert _patronymic_evidence(s).decided is decided + assert (s.patch.patronymic_rules == frozenset({rule})) is decided + + +@pytest.mark.parametrize("contested, decided", [(1, True), (2, False)]) +def test_a_contested_name_counts_in_the_patronymic_share( + contested: int, decided: bool) -> None: + # five East Slavic names and one contested name: 5 of 6; with two, + # 5 of 7, which is not more than four in five + names = _east_slavic(5) + ["Jason Kim J.", "Mary Lee J."][:contested] + s = Parser().suggest(names) + assert _order_evidence(s).contested == contested + assert _patronymic_evidence(s).decided is decided + + +def test_a_family_first_given_last_base_moves_to_given_first() -> None: + base = Parser(policy=Policy(name_order=FAMILY_FIRST_GIVEN_LAST)) + s = base.suggest(["Jason Priem", "Minjun Kim", "Mary Smith"]) + assert s.patch.name_order == GIVEN_FIRST + + +def test_a_family_first_group_on_a_family_first_base_leaves_the_patch_empty( + ) -> None: + base = Parser(policy=Policy(name_order=FAMILY_FIRST)) + s = base.suggest(["Priem Jason", "Kim Minjun", "Wang Xiaoming"]) + assert s.patch == PolicyPatch() + assert _order_evidence(s).decided + + +def test_the_ru_pack_and_an_east_slavic_group_leave_the_patch_empty() -> None: + from nameparser import locales, parser_for + s = parser_for(locales.RU).suggest(_EAST_SLAVIC) + assert s.patch == PolicyPatch() + assert _patronymic_evidence(s).decided + assert _order_evidence(s).decided + + +def test_the_vi_pack_and_a_vietnamese_group_leave_the_patch_empty() -> None: + from nameparser import locales, parser_for + s = parser_for(locales.VI).suggest( + ["Nguyen Van Thanh", "Tran Thi Huong", "Pham Minh Tuan"]) + assert s.patch == PolicyPatch() + order = _order_evidence(s) + assert order.decided and order.value == FAMILY_FIRST_GIVEN_LAST + + +def test_the_base_lexicon_is_the_one_consulted() -> None: + lexicon = Lexicon.default().add("zzp", kind="article") + assert Signal.PARTICLE not in name_votes("zzp Berg Jan", L) + assert name_votes("zzp Berg Jan", lexicon)[Signal.PARTICLE] == FAMILY_FIRST + s = Parser(lexicon=lexicon).suggest(["zzp Berg Jan", "zzp Holt Anna"]) + assert (Signal.PARTICLE, FAMILY_FIRST, 2) in _order_evidence(s).by_signal + s = Parser().suggest(["zzp Berg Jan", "zzp Holt Anna"]) + assert all(row[0] is not Signal.PARTICLE + for row in _order_evidence(s).by_signal) + + +@pytest.fixture +def _invented_tables(monkeypatch: pytest.MonkeyPatch) -> None: + # the decision rule, read with tables of invented words, so no + # shipped table's contents decides a test + from nameparser import _suggest + monkeypatch.setattr(_suggest, "ROMANIZED_SURNAMES", + frozenset({"zzsur", "zzfam"})) + monkeypatch.setattr(_suggest, "COMMON_GIVEN_NAMES", + frozenset({"zzgiv", "zzfirst"})) + + +@pytest.mark.usefixtures("_invented_tables") +@pytest.mark.parametrize("text, votes", [ + ("Zzsur Plugh", {Signal.SURNAME: FAMILY_FIRST}), + ("Plugh Zzsur", {Signal.SURNAME: GIVEN_FIRST}), + ("Zzgiv Plugh", {Signal.GIVEN_NAME: GIVEN_FIRST}), + ("Plugh Zzgiv", {Signal.GIVEN_NAME: FAMILY_FIRST}), + ("Zzsur Zzfam", {}), + ("Zzgiv Zzfirst", {}), + ("Zzsur Zzgiv", {Signal.SURNAME: FAMILY_FIRST, + Signal.GIVEN_NAME: FAMILY_FIRST}), + ("Zzgiv Zzsur", {Signal.SURNAME: GIVEN_FIRST, + Signal.GIVEN_NAME: GIVEN_FIRST}), + ("Zzsur Z", {Signal.SURNAME: FAMILY_FIRST, + Signal.TRAILING_INITIAL: FAMILY_FIRST}), + # a word the tables list is never read past as a title or suffix + ("Kim Minjun", {}), +]) +def test_the_tables_decide_by_one_end_and_not_the_other( + text: str, votes: dict[Signal, object]) -> None: + assert name_votes(text, L) == votes + + +@pytest.mark.usefixtures("_invented_tables") +def test_the_group_rule_reads_invented_tables() -> None: + s = Parser().suggest(["Zzsur Plugh", "Zzfam Xyzzy", "Zzsur Frotz"]) + assert s.patch == PolicyPatch(name_order=FAMILY_FIRST) + s = Parser().suggest(["Zzsur Plugh", "Zzfam Xyzzy", "Frotz Zzgiv", + "Plugh Zzgiv", "Xyzzy Zzfam"]) + # four for family-first, one against: more than two thirds + assert s.patch == PolicyPatch(name_order=FAMILY_FIRST) + assert [n.text for n in s.disagreeing()] == ["Xyzzy Zzfam"] + + +@pytest.mark.parametrize("text, signal, value", [ + ("René Dubois", Signal.GIVEN_NAME, GIVEN_FIRST), + ("Máire ní Bhriain", Signal.PARTICLE, GIVEN_FIRST), + ("Mehmet DEMİR", Signal.CAPITALS, GIVEN_FIRST), + ("Mamedov Ali Huseyn oğlu", Signal.PATRONYMIC, PatronymicRule.TURKIC), + ("Ali Huseyn oğlu Mamedov", Signal.PATRONYMIC, PatronymicRule.TURKIC), + ("Sa'dullayev Alisher Zafar o‘g‘li", Signal.PATRONYMIC, + PatronymicRule.TURKIC), + ("Hasanova Afət Əbil qızı", Signal.PATRONYMIC, PatronymicRule.TURKIC), + ("Иванов Иван Сергеевич", Signal.PATRONYMIC, PatronymicRule.EAST_SLAVIC), + ("Nguyễn Minh", Signal.SURNAME, FAMILY_FIRST), +]) +def test_a_decomposed_name_votes_as_its_composed_twin( + text: str, signal: Signal, value: object) -> None: + composed = unicodedata.normalize("NFC", text) + decomposed = unicodedata.normalize("NFD", text) + assert name_votes(composed, L)[signal] == value + assert name_votes(decomposed, L) == name_votes(composed, L) + + +#: The recorded negative control: each threshold relaxed, the answer the +#: same group gets, beside the answer it gets guarded. A guard whose +#: relaxed answer equals its guarded one guards nothing here. +#: (threshold, relaxed value, group, base order, guarded, relaxed), the +#: answers as (name_order patch, patronymic_rules patch). +_UNGUARDED_EFFECT: list[tuple[str, object, list[str], + tuple[Role, Role, Role], + tuple[object, object], tuple[object, object]]] = [ + ("_MIN_VOTES", 1, ["Priem Jason", "Xyzzy Plugh"], GIVEN_FIRST, + (UNSET, UNSET), (FAMILY_FIRST, UNSET)), + ("_MIN_SHARE", 0.5, ["Priem Jason", "Kim Minjun", "Jason Smith"], + GIVEN_FIRST, (UNSET, UNSET), (FAMILY_FIRST, UNSET)), + ("_MIN_PATRONYMIC_SHARE", 0.5, _east_slavic(5) + _OTHERS[:2], + GIVEN_FIRST, (UNSET, UNSET), + (UNSET, frozenset({PatronymicRule.EAST_SLAVIC}))), + ("_MIN_PATRONYMIC_VOTES", 2, _east_slavic(2), FAMILY_FIRST, + (UNSET, UNSET), + (GIVEN_FIRST, frozenset({PatronymicRule.EAST_SLAVIC}))), +] + + +@pytest.mark.parametrize( + "threshold, relaxed, names, base_order, guarded, unguarded", + _UNGUARDED_EFFECT, ids=[row[0] for row in _UNGUARDED_EFFECT]) +def test_each_threshold_changes_its_recorded_answer( + monkeypatch: pytest.MonkeyPatch, threshold: str, relaxed: object, + names: list[str], base_order: tuple[Role, Role, Role], + guarded: tuple[object, object], + unguarded: tuple[object, object]) -> None: + from nameparser import _suggest + base = Parser(policy=Policy(name_order=base_order)) + + def answer() -> tuple[object, object]: + patch = base.suggest(names).patch + return patch.name_order, patch.patronymic_rules + + assert answer() == guarded + monkeypatch.setattr(_suggest, threshold, relaxed) + assert answer() == unguarded + + +def test_every_threshold_has_a_recorded_control() -> None: + from nameparser import _suggest + thresholds = {name for name in vars(_suggest) + if name.startswith("_MIN_")} + assert {row[0] for row in _UNGUARDED_EFFECT} == thresholds + + +# -- the reprs, the report's notes and disagreement (C8) ---------------------- + + +def test_the_reprs_are_bounded_and_name_the_votes() -> None: + evidence = NameEvidence("Kim Minjun", {Signal.SURNAME: FAMILY_FIRST, + Signal.PATRONYMIC: PatronymicRule.TURKIC}) + assert repr(evidence) == ( + "NameEvidence('Kim Minjun', surname=FAMILY_FIRST, patronymic=turkic)") + s = Parser().suggest(["Priem Jason", "Kim Minjun", "Jason Smith", + "Wang Xiaoming"]) + assert repr(s) == ("Suggestion(PolicyPatch(name_order=FAMILY_FIRST), " + "names=4, disagreeing=1)") + assert repr(_order_evidence(s)) == ( + "SettingEvidence(name_order=FAMILY_FIRST, decided, 3 for, " + "1 against, 0 contested, 0 no evidence)") + + +def test_the_report_writes_a_setting_value_of_any_kind() -> None: + from nameparser import Script + value = Parser().policy.segment_scripts | {Script.HAN} + s = Suggestion(Parser(), PolicyPatch( + segment_scripts=frozenset({Script.HAN})), (SettingEvidence( + "segment_scripts", value, True, 0, 0, 0, 0, (), # type: ignore[arg-type] + note="set by hand"),), ()) + report = str(s) + assert "" in report + assert " set by hand" in report + + +def test_the_report_writes_the_order_note() -> None: + report = str(Parser().suggest(_EAST_SLAVIC)) + assert "it needs a given-first order" in report + + +def test_a_name_carrying_another_rule_disagrees() -> None: + s = Parser().suggest([*_EAST_SLAVIC, "Mamedov Ali Huseyn oglu"]) + assert s.patch.patronymic_rules == frozenset({PatronymicRule.EAST_SLAVIC}) + assert [n.text for n in s.disagreeing()] == ["Mamedov Ali Huseyn oglu"] + + +def test_with_only_a_rule_decided_a_name_of_that_rule_agrees() -> None: + s = Parser().suggest(_TURKIC) + assert not _order_evidence(s).decided + assert s.disagreeing() == () diff --git a/tests/v2/test_suggest_measure.py b/tests/v2/test_suggest_measure.py new file mode 100644 index 000000000..c1d869ea9 --- /dev/null +++ b/tests/v2/test_suggest_measure.py @@ -0,0 +1,274 @@ +"""Offline checks on tools/differential/suggest_measure.py. + +The populations are planted in a cache of the test's own, never the +developer's: a Czech, a British and a Vietnamese one, each a handful of +names, and two parliament lists. What is pinned is the tool's own +logic: a row per population, written order and base order; a planted +family-first population read 100% right under both bases; the harness's +proof that it can report a difference (with `_MIN_VOTES` set past any +group, nothing is decided, so every group written against its base +reads 0%); a source not fetched left out with a note; and the parliament +tables printed. +""" +from __future__ import annotations + +import json +import re +from pathlib import Path +from types import ModuleType + +import pytest + +from tests.checkout import needs_checkout + +from ._differential_fixtures import load_tool, own_cache + +tool: ModuleType = load_tool("suggest_measure") + +_ROW = re.compile(r"^(\S+)\s+(GF|FF)\s+(GF|FF)\s+((?:\s*(?:\d+%|-))+)\s+\d+") + +_VI = [{"family": f, "middle": m, "given": g, "name": f"{f} {m} {g}"} + for f in ("Nguyen", "Tran", "Pham", "Bui") + for m, g in (("Van", "Thanh"), ("Thi", "Huong"), ("Minh", "Tuan"))] +_CZ = [{"jmeno": g, "prijmeni": f, "name": f"{g} {f}"} + for g in ("Jan", "Petr", "Marie", "Eva", "Jana") + for f in ("Novák", "Svoboda", "Veselá")] +_PSC = [{"forename": g, "surname": f, "nationality": "British", + "name": f"{g} {f}"} + for g in ("Mary", "Sarah", "Emma", "Peter", "Anna") + for f in ("Jones", "Wilson", "Davies")] +_PARLIAMENTS = [ + {"dataset": "az_parliament", "form": "name", "script": "Latin", "name": n} + for n in ("NƏBİYEV Rizvan Novruz oğlu", "HƏSƏNOVA Afət Əbil qızı", + "AĞAZADƏ Əminə Həmid qızı", "MİKAYILOVA Sevil Əlirazı qızı", + "QULİYEV Elman Rasim oğlu", "BÜLBÜLOĞLU Polad")] + [ + {"dataset": "ua_rada", "form": "name", "script": "Cyrillic", "name": n} + for n in ("Гунько Анатолій Григорович", "Аксьонов Андрій Анатолійович", + "Бакулін Євген Миколайович", "Швачко Антон Олексійович", + "Шевченко Тарас Григорович")] + [ + {"dataset": "ua_rada", "form": "alias", "script": "Latin", + "name": "Hunko Anatolii"}] + + +@pytest.fixture(autouse=True) +def _cache(monkeypatch: pytest.MonkeyPatch, + tmp_path_factory: pytest.TempPathFactory) -> Path: + cache = own_cache(monkeypatch, tmp_path_factory) + for source, rows in (("uit-vinames", _VI), ("cz-candidates", _CZ), + ("psc-individuals", _PSC), + ("opensanctions-parliaments", _PARLIAMENTS)): + folder = cache / "local" / source + folder.mkdir(parents=True) + (folder / "raw.jsonl").write_text( + "".join(json.dumps(r, ensure_ascii=False) + "\n" for r in rows), + encoding="utf-8") + return cache + + +def _rows(out: str) -> dict[tuple[str, str, str], list[str]]: + rows = {} + for line in out.splitlines(): + if m := _ROW.match(line): + rows[m[1], m[2], m[3]] = m[4].split() + return rows + + +def _run(capsys: pytest.CaptureFixture[str], *args: str) -> str: + assert tool.main(["--sizes", "5,10", "--trials", "20", "--jobs", "1", + *args]) == 0 + return capsys.readouterr().out + + +@needs_checkout +def test_a_row_per_population_written_order_and_base( + capsys: pytest.CaptureFixture[str]) -> None: + rows = _rows(_run(capsys)) + assert set(rows) == {(p, w, b) for p in ("cz", "psc-British", "vi") + for w in ("GF", "FF") for b in ("GF", "FF")} + assert all(len(cells) == 2 for cells in rows.values()) + + +@needs_checkout +def test_a_family_first_population_reads_right_under_both_bases( + capsys: pytest.CaptureFixture[str]) -> None: + rows = _rows(_run(capsys)) + assert rows["vi", "FF", "GF"] == ["100%", "100%"] + assert rows["vi", "FF", "FF"] == ["100%", "100%"] + + +@needs_checkout +def test_deciding_nothing_reads_every_group_against_its_base_wrong( + capsys: pytest.CaptureFixture[str]) -> None: + out = _run(capsys, "--set", "_MIN_VOTES=1000000", + "--set", "_MIN_PATRONYMIC_VOTES=1000000") + assert "_MIN_VOTES=1000000" in out + assert "_MIN_PATRONYMIC_VOTES=1000000" in out + rows = _rows(out) + for (_, written, base), cells in rows.items(): + expected = "100%" if written == base else "0%" + assert cells == [expected, expected] + + +@needs_checkout +def test_a_source_not_fetched_is_left_out_with_a_note( + capsys: pytest.CaptureFixture[str]) -> None: + out = _run(capsys) + assert "patentsview-inventors: not fetched" in out + assert not any(p.startswith("pv-") for p, _, _ in _rows(out)) + + +@needs_checkout +def test_the_parliament_tables_are_printed( + capsys: pytest.CaptureFixture[str]) -> None: + out = _run(capsys) + turkic = next(line for line in out.splitlines() + if line.startswith("az_parliament ") and "/" in line + and "base" not in line) + # six names, five with a Turkic marker; {T}+FF reads all six right + assert turkic.split()[3] == "6/6" + assert re.search(r"^ua_rada\s+Cyrillic base FF n=5 .*rules east-slavic\s+rule$", + out, re.M) + # an alias row is not a name row + assert "ua_rada Latin" not in out + + +@needs_checkout +def test_the_default_sizes_start_at_two_names( + capsys: pytest.CaptureFixture[str]) -> None: + # groups of 2 to 4 are where a patronymic rule forced given-first + # on a family-first base, so the default table measures them + assert tool.main(["--trials", "5", "--jobs", "1"]) == 0 + out = capsys.readouterr().out + header = next(line for line in out.splitlines() + if line.startswith("population")) + assert header.split()[3:6] == ["2", "3", "4"] + + +def test_set_refuses_a_name_it_does_not_know() -> None: + with pytest.raises(SystemExit, match="_MIN_VOTES"): + tool.main(["--set", "_MAX_VOTES=3"]) + + +@needs_checkout +def test_an_override_is_undone_after_a_run_in_this_process( + capsys: pytest.CaptureFixture[str]) -> None: + from nameparser import _suggest + before = (_suggest._MIN_VOTES, _suggest._MIN_SHARE, + _suggest._MIN_PATRONYMIC_SHARE, _suggest._MIN_PATRONYMIC_VOTES) + _run(capsys, "--set", "_MIN_VOTES=1000000", "--set", "_MIN_SHARE=0.9", + "--set", "_MIN_PATRONYMIC_VOTES=7") + assert (_suggest._MIN_VOTES, _suggest._MIN_SHARE, + _suggest._MIN_PATRONYMIC_SHARE, + _suggest._MIN_PATRONYMIC_VOTES) == before + + +def test_the_groups_are_drawn_apart_from_the_reservoir() -> None: + # seeded alike, the groups were runs of the slots the reservoir had + # just filled, consecutive records of a sorted file + pool, groups = tool.pool_rng(1, "vi"), tool.group_rng(1, "vi") + assert [pool.random() for _ in range(5)] != [ + groups.random() for _ in range(5)] + keep = tool._Reservoir(1) + for i in range(40): + keep.add("vi", f"Given{i}", f"Family{i}") + drawn = tool.group_rng(1, "vi").sample(range(40), 5) + assert drawn != tool.pool_rng(1, "vi").sample(range(40), 5) + + +@needs_checkout +def test_checks_prints_every_deferred_count( + capsys: pytest.CaptureFixture[str]) -> None: + out = _run(capsys, "--checks") + for start in ("-- deferred checks", "surname votes, East Asian", + "wrong surname votes from", "a name ending in a bare letter", + "patronymic share without"): + assert any(line.startswith(start) for line in out.splitlines()), start + assert "groups suggested right" not in out + + +# -- the review's fix wave (A6, A7) ------------------------------------------ + + +def _plant_patentsview(cache: Path, lines: list[dict[str, object]]) -> None: + import gzip + import local_patentsview # type: ignore[import-not-found] + path = local_patentsview.extract_path() + path.parent.mkdir(parents=True, exist_ok=True) + body = json.dumps({"release": local_patentsview.PV_RELEASE}) + "\n" + "".join( + json.dumps(line, ensure_ascii=False) + "\n" for line in lines) + path.write_bytes(gzip.compress(body.encode("utf-8"))) + + +def test_patentsview_reads_the_country_whatever_the_key_order( + _cache: Path) -> None: + _plant_patentsview(_cache, [ + {"country": "JP", "family": "Tanaka", "given": "Hiroshi"}, + {"family": "Sato", "given": "Yuki", "country": "JP", "people": 2}, + {"given": "Kenji", "people": 1, "family": "Ito", "country": "JP"}, + {"country": "ZZ", "family": "Nobody", "given": "Anyone"}]) + keep = tool._Reservoir(1) + tool._patentsview(keep) + assert sorted(keep.pools["pv-JP"]) == [ + ("Hiroshi", "Tanaka"), ("Kenji", "Ito"), ("Yuki", "Sato")] + + +@needs_checkout +def test_a_wanted_population_that_yields_no_names_gets_a_note( + _cache: Path, capsys: pytest.CaptureFixture[str]) -> None: + _plant_patentsview(_cache, [ + {"country": "JP", "family": "Tanaka", "given": "Hiroshi"}]) + out = _run(capsys) + # patentsview is present but holds no US name, and the planted PSC + # rows hold no Polish one + assert "pv-US: no names" in out + assert "psc-Polish: no names" in out + assert "pv-JP: no names" not in out + + +@needs_checkout +def test_a_present_source_yielding_nothing_gets_a_note( + _cache: Path, capsys: pytest.CaptureFixture[str]) -> None: + (_cache / "local" / "cz-candidates" / "raw.jsonl").write_text( + json.dumps({"jmeno": "", "prijmeni": ""}) + "\n", encoding="utf-8") + out = _run(capsys) + assert "cz-candidates: present, but no names" in out + + +@needs_checkout +def test_moved_compares_the_parliaments_and_the_east_slavic_counts() -> None: + base = {"populations": [{"label": "vi", "cells": {"GF/GF": [1.0]}, + "east_slavic": 0}], + "parliaments": {"turkic": {"az_parliament": {"{T}": 3}}, + "suggestions": [{"dataset": "ua_rada", "x": 1}]}} + head = json.loads(json.dumps(base)) + assert tool.moved(base, head) == [] + head["populations"][0]["east_slavic"] = 4 + head["parliaments"]["turkic"]["az_parliament"]["{T}"] = 5 + head["parliaments"]["suggestions"][0]["x"] = 2 + out = "\n".join(tool.moved(base, head)) + assert "east_slavic" in out and "0 -> 4" in out + assert "az_parliament" in out + assert "ua_rada" in out + + +@needs_checkout +def test_checks_report_the_groups_a_contested_name_moves( + _cache: Path, capsys: pytest.CaptureFixture[str]) -> None: + # a positive control: five patronymic names and two whose only + # evidence is a contested order. With the two in the denominator, + # 5 of 7 is not more than four in five; without them, 5 of 5 is + rows = [{"forename": "Ivan Petrovich", "surname": "Sidorov", + "nationality": "British"}] * 5 + [ + {"forename": "Jason", "surname": "Kim J.", + "nationality": "British"}] * 2 + (_cache / "local" / "psc-individuals" / "raw.jsonl").write_text( + "".join(json.dumps(r) + "\n" for r in rows), encoding="utf-8") + assert tool.main(["--sizes", "7", "--trials", "3", "--jobs", "1", + "--checks"]) == 0 + out = capsys.readouterr().out + line = next(x for x in out.splitlines() + if x.startswith("patronymic share without")) + # six groups hold five patronymic names, so could decide a rule; + # the three written given-first move, the family-first ones + # (`Kim J. Jason`) being uncontested + assert "3 of 6 groups that could decide a rule" in line, line diff --git a/tests/v2/test_types.py b/tests/v2/test_types.py index 64c696313..cd9448a47 100644 --- a/tests/v2/test_types.py +++ b/tests/v2/test_types.py @@ -599,6 +599,25 @@ def test_types_pickle_round_trip() -> None: assert pickle.loads(pickle.dumps(reading)).languages == ("en",) +def test_suggestion_types_pickle_round_trip() -> None: + # Suggestion holds its base Parser, which pickles iff its segmenter + # does; a default parser has none + import pickle + + from nameparser import Parser + s = Parser().suggest(["Priem Jason", "Kim Minjun", "Jason Smith", + "Сидоров Иван Петрович"]) + loaded = pickle.loads(pickle.dumps(s)) + assert loaded == s + assert loaded.patch == s.patch + assert loaded.parser.policy == s.parser.policy + assert [n.text for n in loaded.disagreeing()] == \ + [n.text for n in s.disagreeing()] + for evidence in (*s.evidence, *s.names): + assert pickle.loads(pickle.dumps(evidence)) == evidence + assert pickle.loads(pickle.dumps(s.names[0])).votes == s.names[0].votes + + def test_span_add_is_blocked() -> None: # NamedTuple + would concatenate into a 4-tuple, the natural but # wrong spelling of "covering span" (a real cover() arrives with the diff --git a/tools/differential/README.md b/tools/differential/README.md index d069cc03e..b1920bb82 100644 --- a/tools/differential/README.md +++ b/tools/differential/README.md @@ -390,6 +390,13 @@ uv run python tools/differential/config_diff.py score default plain:ES --switche uv run python tools/differential/match_diff.py --base ../master-checkout ``` +**Measuring a change to the group suggester.** `Parser.suggest` moves no parse either, so its check is `suggest_measure.py`: it draws groups of 2 to 200 names (`--sizes`, `--trials`, `--seed`) from populations whose order is known because each name is composed from the boxes its filer typed it into -- PatentsView inventors by country, UK company filers (`psc-individuals`) by nationality, Czech candidates and Vietnamese names, all local corpora -- writes each group given-first and family-first, and prints the share of groups whose suggested `name_order` writes the family at the end the group was written with, under a given-first and a family-first base. The two rows where the group is written in its base's own order are the anti-goal (decisions/group-suggestion): a suggestion must not move a group the base already reads right, and the thresholds in `nameparser/_suggest.py` were chosen so that neither falls below 95% at any size. It also prints each population's votes per name (right, wrong, contested, none) and per signal, with `--wrong-words N` each signal's commonest words in a wrong vote; the opensanctions-parliaments table of the 2026-11-10 TURKIC entry; and every parliament list's own suggestion beside the order it was read by eye to write. `--set NAME=VALUE` tries another threshold without editing the module, and is the harness's proof that it can report a difference: `--set _MIN_VOTES=1000000 --set _MIN_PATRONYMIC_VOTES=1000000` decides nothing (the second holds the patronymic rule, which forces an order of its own), so every group written against its base reads 0% (all 46 such rows, at all eight sizes, on 2026-11-10). `--checks` prints, instead of the tables, the per-name checks the decisions entry quotes (the short romanized surnames' wrong votes, names ending in a bare non-ASCII letter, and, over the groups holding enough names of one patronymic rule to decide it, whether the patronymic share's denominator decides any differently without the contested names). `--base ` measures another tree in a child process and lists the cells that moved, the populations whose count of groups deciding the East Slavic rule moved, and the parliament tables' rows that moved. A source present but yielding no names, and a population or parliament list a present source holds none of, prints a note. A whole run with every source fetched took 20s with `--jobs 12` on one 12-CPU machine (2026-11-10, sizes 2 to 200; a machine-bound figure). + +``` +uv run python tools/differential/suggest_measure.py +uv run python tools/differential/suggest_measure.py --set _MIN_VOTES=1000000 --set _MIN_PATRONYMIC_VOTES=1000000 +``` + For a vocabulary word whose reading acts on a name position, `tools/vocab/word_report.py` gives the per-word view before and alongside this: the word in a grid of shapes, its incidence in every fetched local corpus, and its `name_usage.py` rows (`nameparser/vocab/README.md`, "Adding a word"). ## Corpus provenance diff --git a/tools/differential/suggest_measure.py b/tools/differential/suggest_measure.py new file mode 100644 index 000000000..baa90fd70 --- /dev/null +++ b/tools/differential/suggest_measure.py @@ -0,0 +1,831 @@ +"""Measure Parser.suggest on groups of names whose written order is known. + +`Parser.suggest(names)` reads a group of names a caller declares to share +one convention and suggests a `name_order` (and `patronymic_rules`). Its +decision rule (`_suggest._MIN_VOTES`, `_MIN_SHARE`, +`_MIN_PATRONYMIC_SHARE`, `_MIN_PATRONYMIC_VOTES`) and its word tables +(`config/_order_names.py`, written by tools/vocab/order_names.py) were +chosen with this tool, and the figures quoted for them in `suggest`'s +docstring and docs/design/decisions/group-suggestion.md are its output. + + uv run python tools/differential/suggest_measure.py + uv run python tools/differential/suggest_measure.py --sizes 5,10 --trials 100 + uv run python tools/differential/suggest_measure.py --set _MIN_VOTES=1000000 --set _MIN_PATRONYMIC_VOTES=1000000 + uv run python tools/differential/suggest_measure.py --base + +THE POPULATIONS, from the local corpora in $NAMEPARSER_TOOLS_CACHE +(~/.cache/nameparser-tools, `local_corpora.py fetch`), each name composed +from the boxes its filer typed it into, so its order is known: +PatentsView inventors by country (PV_COUNTRIES, read from the +patentsview-inventors extract, uncapped), UK company filers by +nationality (psc-individuals, PSC_NATIONALITIES, forename and middle name +before the surname), Czech candidates (cz-candidates, `jmeno` and +`prijmeni`) and Vietnamese names (uit-vinames, the middle name before the +given name, as Vietnamese writes them). A population keeps at most +POOL_CAP names, drawn by a seeded reservoir. A source not fetched is +left out with a note, as is one present that yields no names and each +population a present source yields none of; one present but unreadable +stops the run. + +THE GROUPS: for each population, written order (`Given Family` and +`Family Given`, from the same boxes) and size, `--trials` groups drawn +with a generator seeded by `--seed` and the population, each read by +`Parser.suggest` under both base orders (`Parser()` and a `Parser` whose +policy is `name_order=FAMILY_FIRST`). A group is RIGHT where the +suggested parser's `name_order` writes the family at the end the group +was written with; a given-first group under the given-first base is right +where the suggester kept or confirmed the base, and wrong only where it +moved. The table prints the share right per population, written order, +base and size: the anti-goal (decisions/group-suggestion) is the two +same-order rows, which must not fall below 95%. + +THE SIGNALS: per population and written order, over at most SIGNAL_NAMES +names, the per-NAME order votes (`suggest`'s unit: right, wrong, +contested, none) and each signal's own votes right and wrong. With +`--wrong-words N`, each signal's N commonest words in a wrong vote. + +THE PARLIAMENTS, from the opensanctions-parliaments raw.jsonl, rows whose +`form` is "name" and whose `script` is Latin or Cyrillic: the Turkic +table of decisions/group-suggestion's 2026-11-10 TURKIC entry (its +TURKIC_LISTS, scored by TURKIC_CONVENTION under each fixed policy and +end to end through `suggest`), then every list's suggestion as a whole, +beside the order the list was read by eye to write (LIST_ORDER). + +THE CHECKS (`--checks`, in place of the tables): over every population's +whole pool written both ways, the surname table's right and wrong votes +(East Asian populations apart), the wrong votes the short romanized +surnames (SHORT_SURNAMES) cast, and the names ending in a bare letter, +ASCII or not; and over a draw of groups of its own, of the groups that +hold enough names of one patronymic rule to decide it +(`_MIN_PATRONYMIC_VOTES`), how many decide another rule when the names +whose only evidence is a contested order leave the share's denominator. + +`--set NAME=VALUE` overrides one of `_suggest`'s thresholds in every +worker, so a rule can be tried without editing the module; it is also the +harness's proof that it can report a difference: `--set +_MIN_VOTES=1000000 --set _MIN_PATRONYMIC_VOTES=1000000` decides nothing +(the first holds the order, the second the patronymic rule, which forces +an order of its own), so every family-first group under the given-first +base must read 0% (and every given-first group under the family-first +base). `--base` runs the same measurement against another +checkout in a child process and lists every cell that moved, every +population whose count of groups deciding the East Slavic rule moved, +and every TURKIC row and parliament suggestion that moved. + +Runtime, measured 2026-11-10 on a 12-CPU machine with every source +fetched and the defaults (sizes 2-200, 300 trials, --jobs 12): 20s +wall clock, 6s of it reading the corpora in one process; with --base, +twice that. +""" +from __future__ import annotations + +import argparse +import contextlib +import gzip +import json +import os +import random +import re +import subprocess +import sys +import tempfile +from collections import Counter +from collections.abc import Iterator +from concurrent.futures import ProcessPoolExecutor +from pathlib import Path +from typing import TYPE_CHECKING, Any + +if TYPE_CHECKING: + from nameparser import Policy + +HERE = Path(__file__).resolve().parent +if str(HERE) not in sys.path: + sys.path.append(str(HERE)) +import corpus_build # noqa: E402 +import snapshot # noqa: E402 + +REPO_ROOT = HERE.parents[1] +SIZES = (2, 3, 4, 5, 10, 20, 50, 200) +TRIALS = 300 +SEED = 20261110 +#: The most names a population keeps, drawn by a seeded reservoir. +POOL_CAP = 20_000 +#: The names each population's signal counts read. +SIGNAL_NAMES = 3_000 +PV_COUNTRIES = ("US", "DE", "FR", "CN", "JP", "KR", "IN", "IT", "ES", "RU", + "BR", "TW", "NL") +PSC_NATIONALITIES = ("British", "Romanian", "Indian", "Chinese", "Polish", + "Nigerian", "Turkish", "Spanish") +GF, FF = "GF", "FF" +#: The thresholds --set may override. +SETTABLE = ("_MIN_VOTES", "_MIN_SHARE", "_MIN_PATRONYMIC_SHARE", + "_MIN_PATRONYMIC_VOTES") +#: The thresholds that count names, read as int. +_COUNTS = ("_MIN_VOTES", "_MIN_PATRONYMIC_VOTES") + +#: decisions/group-suggestion's TURKIC lists, and the end each writes the +#: family at: "caps" where the one capitals word is the family, else the +#: end read by eye off the list's first twelve names (2026-11-10). +TURKIC_LISTS = ("az_parliament", "kz_senate", "uz_senate", + "uz_legislative_chamber", "tm_mejlis", "kz_mazhilis", + "kg_jogorku_kenesh") +TURKIC_CONVENTION = { + "az_parliament": "caps", "kz_mazhilis": "caps", "kz_senate": "first", + "uz_senate": "first", "uz_legislative_chamber": "last", + "tm_mejlis": "last", "kg_jogorku_kenesh": "last"} +#: The order each parliament list writes its names in, read by eye off +#: its first names (the spike's thirteen Latin lists, the TURKIC lists, +#: and ru_duma_deputies and ua_rada, 2026-11-10). kz_mazhilis writes the +#: family last, in capitals (`Тилектес Серикбайулы АДАМБЕКОВ`). +LIST_ORDER = { + "az_parliament": FF, "by_council_republic": FF, "hr_sabor": GF, + "in_sansad": GF, "kg_jogorku_kenesh": GF, "kz_mazhilis": GF, + "kz_senate": FF, "pt_parliament": GF, "ru_duma_deputies": GF, + "ru_federation_council": GF, "si_dz_rs": GF, "sk_nrsr_poslanci": GF, + "tm_mejlis": GF, "tr_parliament": GF, "ua_rada": FF, + "uz_legislative_chamber": GF, "uz_senate": FF} +#: The fixed policies of the TURKIC table, as decisions/group-suggestion +#: heads its columns: (label, patronymic rules, name order). +TURKIC_POLICIES = ( + ("nothing", (), None), ("{T}", ("TURKIC",), None), + ("{T}+FF", ("TURKIC",), "FAMILY_FIRST"), + ("{T}+FFGL", ("TURKIC",), "FAMILY_FIRST_GIVEN_LAST"), + ("{E}", ("EAST_SLAVIC",), None), + ("{E,T}", ("EAST_SLAVIC", "TURKIC"), None), + ("FF alone", (), "FAMILY_FIRST")) + +Pool = list[tuple[str, str]] # (given part as written, family) + + +def local(source: str) -> Path: + return corpus_build.cache_dir(f"local/{source}", create=False) + + +# -- the populations --------------------------------------------------------- + +def _rows(path: Path) -> Iterator[dict[str, Any]]: + """A raw.jsonl's records, cut at "\\n" only (corpus_build.jsonl_lines + cuts the same way), streamed.""" + with open(path, encoding="utf-8", newline="\n") as lines: + for line in lines: + if line.strip(): + yield json.loads(line) + + +def pool_rng(seed: int, label: str) -> random.Random: + """The generator a population's reservoir draws its names with.""" + return random.Random(f"{seed}-{label}-pool") + + +def group_rng(seed: int, label: str) -> random.Random: + """The generator a population's groups are drawn with: a stream of + its own, never the reservoir's. Seeded alike, the two drew the same + indices, so a group's names were the slots the reservoir had just + filled, consecutive records of a sorted file (five Đ-family names + in a row in vi), and the against-base cells moved by up to 25 + points with the reservoir's seed alone.""" + return random.Random(f"{seed}-{label}-groups") + + +class _Reservoir: + """At most POOL_CAP names per label, a seeded uniform draw.""" + + def __init__(self, seed: int) -> None: + self.pools: dict[str, Pool] = {} + self.seen: Counter[str] = Counter() + self.seed = seed + self.rngs: dict[str, random.Random] = {} + + def slot(self, label: str) -> int | None: + """Where the next name of `label` goes, or None where the draw + passes it over: asked before the name is decoded.""" + pool = self.pools.setdefault(label, []) + if (rng := self.rngs.get(label)) is None: + rng = self.rngs[label] = pool_rng(self.seed, label) + self.seen[label] += 1 + if len(pool) < POOL_CAP: + return len(pool) + j = rng.randrange(self.seen[label]) + return j if j < POOL_CAP else None + + def put(self, label: str, at: int, given: str, family: str) -> None: + """A name into the slot `slot` gave; one missing either box is + left out (its slot, where it was a fresh one, stays unfilled).""" + given, family = " ".join(given.split()), " ".join(family.split()) + if not given or not family: + return + pool = self.pools[label] + if at == len(pool): + pool.append((given, family)) + elif at < len(pool): + pool[at] = (given, family) + + def add(self, label: str, given: str, family: str) -> None: + if (at := self.slot(label)) is not None: + self.put(label, at, given, family) + + +#: A line's country field, wherever the line puts it: the extract's +#: lines are JSON objects whose keys a writer may order any way. +_PV_COUNTRY = re.compile(r'"country":\s*"([^"]*)"') + + +def _patentsview(keep: _Reservoir) -> None: + """The extract the patentsview-inventors fetch keeps, every name + (local_patentsview.pv_boxes reads the same file, but decodes all of + its 5.3 million lines; a line naming no wanted country is skipped + here undecoded, its country found by a search rather than by its + place in the line, and a decoded one must name the country it + seemed to).""" + import local_patentsview + path = local_patentsview.extract_path() + if not path.exists(): + raise FileNotFoundError(2, "not fetched", str(path)) + wanted = frozenset(PV_COUNTRIES) + with gzip.open(path, "rt", encoding="utf-8", newline="\n") as lines: + if json.loads(next(lines)).get("release") != local_patentsview.PV_RELEASE: + raise ValueError(f"{path} is not an extract of the " + f"{local_patentsview.PV_RELEASE} release") + for line in lines: + found = _PV_COUNTRY.search(line) + country = found[1] if found else None + if country not in wanted: + continue + label = f"pv-{country}" + if (at := keep.slot(label)) is None: + continue + record = json.loads(line) + if record.get("country") != country: + raise ValueError(f"{path}: a line read as {country} is not: " + f"{line[:60]!r}") + keep.put(label, at, record.get("given", ""), + record.get("family", "")) + + +def _psc(keep: _Reservoir) -> None: + for r in _rows(local("psc-individuals") / "raw.jsonl"): + if (n := r.get("nationality")) in PSC_NATIONALITIES: + given = f"{r.get('forename', '')} {r.get('middle_name', '')}" + keep.add(f"psc-{n}", given, r.get("surname", "")) + + +def _cz(keep: _Reservoir) -> None: + for r in _rows(local("cz-candidates") / "raw.jsonl"): + keep.add("cz", r.get("jmeno", ""), r.get("prijmeni", "")) + + +def _vi(keep: _Reservoir) -> None: + for r in _rows(local("uit-vinames") / "raw.jsonl"): + keep.add("vi", f"{r.get('middle', '')} {r.get('given', '')}", + r.get("family", "")) + + +SOURCES = (("patentsview-inventors", _patentsview), + ("psc-individuals", _psc), ("cz-candidates", _cz), + ("uit-vinames", _vi)) + + +#: The populations each source is read for. +WANTED = {"patentsview-inventors": tuple(f"pv-{c}" for c in PV_COUNTRIES), + "psc-individuals": tuple(f"psc-{n}" for n in PSC_NATIONALITIES), + "cz-candidates": ("cz",), "uit-vinames": ("vi",)} + + +def load_pools(seed: int) -> tuple[dict[str, Pool], list[str]]: + """Every population fetched, and a note for each source that is + not, each present source that yields no names and each population a + present source yields none of. A source present but unreadable stops + the run: a cut file is a broken measurement, not a smaller one.""" + keep = _Reservoir(seed) + notes = [] + for source, read in SOURCES: + try: + read(keep) + except FileNotFoundError as e: + notes.append(f"{source}: not fetched ({e.filename}); " + f"uv run python tools/differential/local_corpora.py " + f"fetch {source}") + continue + except (OSError, ValueError, KeyError, EOFError) as e: + raise SystemExit(f"{source}: present but unreadable " + f"({type(e).__name__}: {e}); fetch it again") from e + empty = [label for label in WANTED[source] + if not keep.pools.get(label)] + if len(empty) == len(WANTED[source]): + notes.append(f"{source}: present, but no names read; is it the " + f"file this tool reads?") + else: + notes.extend(f"{label}: no names in {source}" for label in empty) + pools = {label: pool for label, pool in keep.pools.items() if pool} + return dict(sorted(pools.items())), notes + + +def written(name: tuple[str, str], order: str) -> str: + given, family = name + return f"{family} {given}" if order == FF else f"{given} {family}" + + +def parliament_lists() -> tuple[dict[str, dict[str, list[str]]] | None, + list[str]]: + """The opensanctions-parliaments names by dataset and script, Latin + and Cyrillic rows of form "name" only, or None where not fetched; + and a note for each list the tables read that holds no such row.""" + path = local("opensanctions-parliaments") / "raw.jsonl" + if not path.exists(): + return None, [] + lists: dict[str, dict[str, list[str]]] = {} + for r in _rows(path): + if r.get("form") == "name" and r.get("script") in ("Latin", "Cyrillic"): + lists.setdefault(r["dataset"], {}).setdefault( + r["script"], []).append(r["name"]) + notes = [f"{dataset}: no names in opensanctions-parliaments" + for dataset in sorted(set(TURKIC_LISTS) | set(LIST_ORDER)) + if dataset not in lists] + return dict(sorted(lists.items())), notes + + +# -- one population's measurement (a worker) --------------------------------- + +@contextlib.contextmanager +def _thresholds(overrides: dict[str, float]) -> Iterator[None]: + """`_suggest`'s thresholds overridden for the block, and restored + after it: a run with --jobs 1 measures in the caller's process.""" + from nameparser import _suggest + before = {name: getattr(_suggest, name) for name in overrides} + try: + for name, value in overrides.items(): + setattr(_suggest, name, value) + yield + finally: + for name, value in before.items(): + setattr(_suggest, name, value) + + +def _bases() -> dict[str, Any]: + from nameparser import FAMILY_FIRST, Parser, Policy + return {GF: Parser(), FF: Parser(policy=Policy(name_order=FAMILY_FIRST))} + + +def _end(order: tuple[Any, ...]) -> str: + from nameparser import Role + return FF if order[0] is Role.FAMILY else GF + + +def measure_population(job: tuple[str, Pool, tuple[int, ...], int, int, + dict[str, float], int]) -> dict[str, Any]: + """One population's cells, signal counts and wrong words.""" + label, pool, sizes, trials, seed, overrides, wrong_words = job + with _thresholds(overrides): + return _measure_population(label, pool, sizes, trials, seed, + wrong_words) + + +def _measure_population(label: str, pool: Pool, sizes: tuple[int, ...], + trials: int, seed: int, wrong_words: int + ) -> dict[str, Any]: + from nameparser import PatronymicRule + from nameparser._suggest import name_votes + bases = _bases() + lexicon = bases[GF].lexicon + rng = group_rng(seed, label) + cells: dict[str, list[float | None]] = {} + east_slavic = 0 + for order in (GF, FF): + row = {base: [] for base in bases} + for size in sizes: + if size > len(pool): + for base in bases: + row[base].append(None) + continue + right = Counter[str]() + for _ in range(trials): + group = [written(t, order) for t in rng.sample(pool, size)] + for base, parser in bases.items(): + s = parser.suggest(group) + right[base] += _end(s.parser.policy.name_order) == order + east_slavic += PatronymicRule.EAST_SLAVIC in ( + s.parser.policy.patronymic_rules) + for base in bases: + row[base].append(right[base] / trials) + for base in bases: + cells[f"{order}/{base}"] = row[base] + signals: dict[str, dict[str, int]] = {} + words: dict[str, dict[str, list[tuple[str, int]]]] = {} + sample = random.Random(f"{seed}-{label}-signals").sample( + pool, min(SIGNAL_NAMES, len(pool))) + for order in (GF, FF): + truth = bases[order].policy.name_order + stats: Counter[str] = Counter() + wrong: dict[str, Counter[str]] = {} + for t in sample: + text = written(t, order) + votes = name_votes(text, lexicon) + ends = {v for s, v in votes.items() if s.value != "patronymic"} + stats["names"] += 1 + if not ends: + stats["name:none"] += 1 + elif len(ends) > 1: + stats["name:contested"] += 1 + else: + stats["name:right" if ends == {truth} else "name:wrong"] += 1 + for signal, vote in votes.items(): + if signal.value == "patronymic": + stats[f"patronymic:{vote.value}"] += 1 # type: ignore[union-attr] + continue + ok = vote == truth + stats[f"{signal.value}:{'right' if ok else 'wrong'}"] += 1 + if not ok: + w = text.split() + at = w[0] if (signal.value in ("surname", "particle", "capitals") + ) == (_end(vote) == FF) else w[-1] # type: ignore[arg-type] + wrong.setdefault(signal.value, Counter())[at] += 1 + signals[order] = dict(stats) + words[order] = {s: c.most_common(wrong_words) for s, c in wrong.items()} + return {"label": label, "size": len(pool), "cells": cells, + "signals": signals, "words": words, "east_slavic": east_slavic} + + +#: The populations whose names the romanized surname table is for. +EAST_ASIAN = frozenset({"pv-CN", "pv-TW", "pv-KR", "pv-JP", "psc-Chinese", "vi"}) +#: The short romanized surnames that are common words or given names +#: elsewhere, which --checks counts. +SHORT_SURNAMES = frozenset({"do", "to", "la", "le", "ha", "ho", "an"}) + + +def check_population(job: tuple[str, Pool, tuple[int, ...], int, int, + dict[str, float]]) -> dict[str, Any]: + """--checks: the deferred per-name checks over one population's + whole pool, written both ways, and the patronymic share's two + denominators over a draw of groups of its own.""" + label, pool, sizes, trials, seed, overrides = job + with _thresholds(overrides): + return _check_population(label, pool, sizes, trials, seed) + + +def _check_population(label: str, pool: Pool, sizes: tuple[int, ...], + trials: int, seed: int) -> dict[str, Any]: + from nameparser import FAMILY_FIRST, GIVEN_FIRST, Lexicon + from nameparser import _suggest + from nameparser._suggest import ( + Signal, _patronymic_winner, _Tally, fold, name_votes, + ) + _MIN_PATRONYMIC_VOTES = _suggest._MIN_PATRONYMIC_VOTES + lexicon = Lexicon.default() + c: Counter[str] = Counter() + words: Counter[str] = Counter() + cache: dict[tuple[str, str], dict[Signal, Any]] = {} + + for order, truth in ((GF, GIVEN_FIRST), (FF, FAMILY_FIRST)): + for t in pool: + text = written(t, order) + w = text.split() + votes = cache[t, order] = name_votes(text, lexicon) + c["names"] += 1 + if (vote := votes.get(Signal.SURNAME)) is not None: + word = fold((w[0] if vote == FAMILY_FIRST else w[-1]).strip(".")) + region = "east-asian" if label in EAST_ASIAN else "other" + ok = "right" if vote == truth else "wrong" + c[f"surname:{region}:{ok}"] += 1 + if ok == "wrong": + c["surname:short:wrong"] += word in SHORT_SURNAMES + if region == "other": + words[word] += 1 + if len(w) >= 2 and len(w[-1]) == 1 and w[-1].isalpha(): + c["bare-letter:" + ("ascii" if w[-1].isascii() else "other")] += 1 + # the patronymic share's denominator: every name casting a vote, or + # those less the names whose only evidence is a contested order + rng = random.Random(f"{seed}-{label}-checks") + for order in (GF, FF): + for size in sizes: + if size > len(pool): + continue + moved = could = 0 + for _ in range(trials): + tally, voters, contested_only = _Tally(), 0, 0 + for t in rng.sample(pool, size): + votes = cache[t, order] + voters += bool(votes) + ends = {v for k, v in votes.items() if k is not Signal.PATRONYMIC} + contested_only += len(ends) > 1 and Signal.PATRONYMIC not in votes + tally.add([(k, v) for k, v in votes.items() + if k is Signal.PATRONYMIC]) + # only a group holding enough names of one rule can + # decide it under either denominator + if max(tally.by_name.values(), default=0) < _MIN_PATRONYMIC_VOTES: + continue + could += 1 + moved += (_patronymic_winner(tally, voters) + != _patronymic_winner(tally, voters - contested_only)) + c["denominator:groups"] += trials + c["denominator:could"] += could + c["denominator:moved"] += moved + if could: + c["denominator:worst-cell"] = max( + c["denominator:worst-cell"], round(1000 * moved / could)) + return {"label": label, "counts": dict(c), "words": words.most_common(8)} + + +def report_checks(results: list[dict[str, Any]], min_votes: int) -> None: + total: Counter[str] = Counter() + words: Counter[str] = Counter() + for r in results: + worst = r["counts"].pop("denominator:worst-cell", 0) + total["denominator:worst-cell"] = max(total["denominator:worst-cell"], worst) + total.update(r["counts"]) + words.update(dict(r["words"])) + t = total + print(f"\n-- deferred checks, every population's whole pool written both " + f"ways: {t['names']} names") + print(f"surname votes, East Asian populations ({', '.join(sorted(EAST_ASIAN))}): " + f"{t['surname:east-asian:right']} right, {t['surname:east-asian:wrong']} wrong; " + f"other populations: {t['surname:other:right']} right, " + f"{t['surname:other:wrong']} wrong") + surname_votes = sum(v for k, v in t.items() if k.startswith("surname:") + and not k.startswith("surname:short")) + print(f"wrong surname votes from {', '.join(sorted(SHORT_SURNAMES))}: " + f"{t['surname:short:wrong']} of {surname_votes} surname votes") + print("wrong surname votes outside East Asian populations, by word: " + + ", ".join(f"{w} {n}" for w, n in words.most_common(8))) + print(f"a name ending in a bare letter: {t['bare-letter:other']} non-ASCII " + f"(no trailing-initial vote), {t['bare-letter:ascii']} ASCII") + print(f"patronymic share without the contested-only names: " + f"{t['denominator:moved']} of {t['denominator:could']} groups that " + f"could decide a rule (holding {min_votes} or more " + f"names of one; {t['denominator:groups']} drawn) decide another; " + f"worst cell {t['denominator:worst-cell'] / 10:.1f}%") + + +def _policy(rules: tuple[str, ...], order: str | None) -> Policy: + import nameparser + from nameparser import PatronymicRule, Policy + kwargs: dict[str, Any] = {"patronymic_rules": frozenset( + PatronymicRule[r] for r in rules)} + if order is not None: + kwargs["name_order"] = getattr(nameparser, order) + return Policy(**kwargs) + + +def _turkic_truth(dataset: str, name: str) -> str | None: + words = name.split() + if len(words) < 2: + return None + convention = TURKIC_CONVENTION[dataset] + if convention == "caps": + caps = [w for w in words if w.isupper()] + return caps[0] if len(caps) == 1 else None + return words[0] if convention == "first" else words[-1] + + +def measure_parliaments(lists: dict[str, dict[str, list[str]]], + overrides: dict[str, float]) -> dict[str, Any]: + with _thresholds(overrides): + return _measure_parliaments(lists) + + +def _measure_parliaments(lists: dict[str, dict[str, list[str]]] + ) -> dict[str, Any]: + from nameparser import Parser + turkic = {} + parsers = [(label, Parser(policy=_policy(rules, order))) + for label, rules, order in TURKIC_POLICIES] + for dataset in TURKIC_LISTS: + names = [n for script in lists.get(dataset, {}).values() for n in script] + scored = [(n, t) for n in names + if (t := _turkic_truth(dataset, n)) is not None] + row = {label: sum(p.parse(n).family == t for n, t in scored) + for label, p in parsers} + suggested = Parser().suggest(names).parser + row["suggest"] = sum(suggested.parse(n).family == t for n, t in scored) + row["scored"] = len(scored) + turkic[dataset] = row + suggestions = [] + bases = _bases() + for dataset, scripts in lists.items(): + for script, names in scripts.items(): + for base, parser in bases.items(): + s = parser.suggest(names) + order, rules = s.evidence + suggestions.append({ + "dataset": dataset, "script": script, "base": base, + "names": len(names), "order": _end(order.value), # type: ignore[arg-type] + "order_decided": order.decided, + "counts": [order.votes_for, order.votes_against, + order.contested, order.no_evidence], + "rules": sorted(r.value for r in rules.value), # type: ignore[attr-defined] + "rules_decided": rules.decided, + "rule_counts": [rules.votes_for, rules.votes_against]}) + return {"turkic": turkic, "suggestions": suggestions} + + +def measure(pools: dict[str, Pool], lists: dict[str, dict[str, list[str]]] | None, + sizes: tuple[int, ...], trials: int, seed: int, + overrides: dict[str, float], jobs: int, wrong_words: int + ) -> dict[str, Any]: + work = [(label, pool, sizes, trials, seed, overrides, wrong_words) + for label, pool in pools.items()] + if jobs <= 1: + results = [measure_population(job) for job in work] + parl = measure_parliaments(lists, overrides) if lists else None + else: + with ProcessPoolExecutor(jobs) as ex: + futures = [ex.submit(measure_population, job) for job in work] + parl_future = (ex.submit(measure_parliaments, lists, overrides) + if lists else None) + results = [f.result() for f in futures] + parl = parl_future.result() if parl_future else None + return {"sizes": list(sizes), "trials": trials, "seed": seed, + "populations": results, "parliaments": parl} + + +# -- the report -------------------------------------------------------------- + +def _pct(x: float | None, places: int = 0) -> str: + width = 6 + places + bool(places) + return " " * (width - 1) + "-" if x is None else f"{x:{width}.{places}%}" + + +def report(result: dict[str, Any], show_words: bool) -> None: + sizes = result["sizes"] + print(f"\n-- groups suggested right, {result['trials']} trials a cell " + f"(seed {result['seed']}); written order / base order") + print(f"{'population':14} {'written':7} {'base':4} " + + " ".join(f"{n:>6}" for n in sizes) + " names") + for p in result["populations"]: + for key, row in p["cells"].items(): + order, base = key.split("/") + mark = " <- anti-goal row" if order == base and any( + x is not None and x < 0.95 for x in row) else "" + print(f"{p['label']:14} {order:7} {base:4} " + + " ".join(_pct(x) for x in row) + f" {p['size']}{mark}") + if p["east_slavic"]: + print(f"{'':14} (the East Slavic rule was decided in " + f"{p['east_slavic']} groups, and forced a given-first order)") + print(f"\n-- votes per name (right/wrong/contested/none) and per signal " + f"(right/wrong), at most {SIGNAL_NAMES} names") + for p in result["populations"]: + for order, st in p["signals"].items(): + n = st.get("names", 0) or 1 + parts = [f"names {st.get('name:right', 0)}/{st.get('name:wrong', 0)}/" + f"{st.get('name:contested', 0)}/{st.get('name:none', 0)} " + f"({st.get('name:wrong', 0) / n:.1%} wrong)"] + for signal in ("capitals", "trailing-initial", "particle", + "surname", "given-name"): + r, w = st.get(f"{signal}:right", 0), st.get(f"{signal}:wrong", 0) + if r or w: + parts.append(f"{signal} {r}/{w}") + for k, v in sorted(st.items()): + if k.startswith("patronymic:"): + parts.append(f"{k} {v}") + print(f"{p['label']:14} {order}: " + ", ".join(parts)) + if show_words: + for signal, ws in p["words"][order].items(): + if ws: + print(f"{'':18} wrong {signal}: " + + ", ".join(f"{w} {c}" for w, c in ws)) + parl = result["parliaments"] + if parl is None: + return + labels = [label for label, _, _ in TURKIC_POLICIES] + ["suggest"] + print("\n-- decisions/group-suggestion TURKIC: family right of the scored names") + print(f"{'list':24} " + " ".join(f"{label:>9}" for label in labels)) + for dataset, row in parl["turkic"].items(): + print(f"{dataset:24} " + " ".join( + f"{row[label]:>4}/{row['scored']:<4}" for label in labels)) + print("\n-- every parliament list's suggestion (order: for/against/" + "contested/none; by eye = the order the list writes)") + for s in parl["suggestions"]: + expected = LIST_ORDER.get(s["dataset"]) + rules = ",".join(s["rules"]) or "none" + verdict = ("" if expected is None else + " rule" if s["rules_decided"] and "east-slavic" in s["rules"] + else " ok" if s["order"] == expected else " WRONG") + print(f"{s['dataset']:24} {s['script']:8} base {s['base']} " + f"n={s['names']:<5} by eye {expected or '?'} order " + f"{s['order']}{'' if s['order_decided'] else ' (undecided)'} " + f"{'/'.join(map(str, s['counts']))} rules {rules}" + f"{'' if s['rules_decided'] else ' (undecided)'}{verdict}") + + +def moved(base: dict[str, Any], head: dict[str, Any]) -> list[str]: + """What differs between two runs of the same draw: the cells, the + groups that decided the East Slavic rule, the TURKIC table's rows + and every parliament list's suggestion.""" + old = {p["label"]: p for p in base["populations"]} + out = [] + for p in head["populations"]: + was = old.get(p["label"], {}) + for key, row in p["cells"].items(): + before = was.get("cells", {}).get(key) + if before != row: + out.append(f"{p['label']:14} {key:6} " + + " ".join(_pct(x, 1) for x in before or []) + + " -> " + " ".join(_pct(x, 1) for x in row)) + if was.get("east_slavic") != p["east_slavic"]: + out.append(f"{p['label']:14} east_slavic groups " + f"{was.get('east_slavic')} -> {p['east_slavic']}") + parl_old, parl_new = base.get("parliaments") or {}, head.get("parliaments") or {} + turkic_old = parl_old.get("turkic", {}) + for dataset, row in (parl_new.get("turkic") or {}).items(): + if turkic_old.get(dataset) != row: + out.append(f"TURKIC {dataset}: {turkic_old.get(dataset)} -> {row}") + def key(s: dict[str, Any]) -> tuple[Any, ...]: + return (s.get("dataset"), s.get("script"), s.get("base")) + sugg_old = {key(s): s for s in parl_old.get("suggestions", [])} + for s in parl_new.get("suggestions", []): + if sugg_old.get(key(s)) != s: + out.append(f"suggestion {' '.join(map(str, key(s)))}: " + f"{sugg_old.get(key(s))} -> {s}") + return out + + +def _overrides(items: list[str]) -> dict[str, float]: + out: dict[str, float] = {} + for item in items: + name, _, value = item.partition("=") + if name not in SETTABLE or not value: + raise SystemExit(f"--set takes NAME=VALUE with NAME one of " + f"{', '.join(SETTABLE)}, got {item!r}") + out[name] = int(value) if name in _COUNTS else float(value) + return out + + +def main(argv: list[str] | None = None) -> int: + ap = argparse.ArgumentParser(description=__doc__.splitlines()[0]) + ap.add_argument("--sizes", default=",".join(map(str, SIZES)), + help="group sizes, comma-separated") + ap.add_argument("--trials", type=int, default=TRIALS) + ap.add_argument("--seed", type=int, default=SEED) + ap.add_argument("--set", action="append", default=[], metavar="NAME=VALUE", + help=f"override one of {', '.join(SETTABLE)}") + ap.add_argument("--wrong-words", type=int, default=0, metavar="N", + help="each signal's N commonest words in a wrong vote") + ap.add_argument("--jobs", type=int, default=os.cpu_count() or 1) + ap.add_argument("--checks", action="store_true", + help="print the per-name checks instead of the tables") + ap.add_argument("--base", metavar="PATH", + help="list the cells that moved against this checkout") + ap.add_argument("--tree", metavar="PATH", help=argparse.SUPPRESS) + ap.add_argument("--json", type=Path, help=argparse.SUPPRESS) + args = ap.parse_args(argv) + sizes = tuple(int(n) for n in args.sizes.split(",")) + overrides = _overrides(args.set) + tree = Path(args.tree).resolve() if args.tree else REPO_ROOT + if args.tree: + sys.path.insert(0, str(tree)) + import nameparser + snapshot.check_source(nameparser.__file__, tree) + pools, notes = load_pools(args.seed) + for note in notes: + print(note) + lists, list_notes = parliament_lists() + for note in list_notes: + print(note) + if lists is None: + print("opensanctions-parliaments: not fetched; uv run python " + "tools/differential/local_corpora.py fetch opensanctions-parliaments") + if args.checks: + from nameparser import _suggest + work = [(label, pool, sizes, args.trials, args.seed, overrides) + for label, pool in pools.items()] + min_votes = int(overrides.get("_MIN_PATRONYMIC_VOTES", + _suggest._MIN_PATRONYMIC_VOTES)) + if args.jobs <= 1: + report_checks([check_population(job) for job in work], min_votes) + else: + with ProcessPoolExecutor(args.jobs) as ex: + report_checks(list(ex.map(check_population, work)), min_votes) + return 0 + if overrides: + print("thresholds overridden: " + ", ".join( + f"{k}={v}" for k, v in overrides.items())) + result = measure(pools, lists, sizes, args.trials, args.seed, overrides, + args.jobs, args.wrong_words) + if args.json: + args.json.write_text(json.dumps(result), encoding="utf-8") + return 0 + print(f"head: {tree}") + report(result, args.wrong_words > 0) + if args.base: + base_root = snapshot.base_root(args.base, tree) + with tempfile.TemporaryDirectory() as scratch: + out = Path(scratch) / "base.json" + command = [sys.executable, str(Path(__file__).resolve()), + "--tree", str(base_root), "--json", str(out), + "--sizes", args.sizes, "--trials", str(args.trials), + "--seed", str(args.seed), "--jobs", str(args.jobs), + *(f"--set={s}" for s in args.set)] + done = subprocess.run(command, env=snapshot.child_env(), + capture_output=True, text=True) + if done.returncode: + raise SystemExit(f"reading {base_root} failed:\n" + f"{done.stdout}{done.stderr}") + base = json.loads(out.read_text(encoding="utf-8")) + cells = moved(base, result) + print(f"\n-- against the base {base_root}: {len(cells)} cells moved") + for line in cells: + print(line) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tools/vocab/order_names.py b/tools/vocab/order_names.py new file mode 100644 index 000000000..1580d21bc --- /dev/null +++ b/tools/vocab/order_names.py @@ -0,0 +1,251 @@ +"""Write the name tables the group suggester reads. + +WHY THIS IS IN THE TREE. ``Parser.suggest`` reads a group of names and +suggests a name order from each name's own evidence: a known East Asian +surname at one end of the name and not at the other says the family name +is written at that end, and a common given name at one end and not the +other says the given name is. The tables, in +``nameparser/config/_order_names.py``, hold folded spellings: the generated block is the Hanyu Pinyin of every +single-character surname ``locales.ZH`` lists, from Unihan's first +``kMandarin`` reading (Unicode License v3), and the Wade-Giles, Korean and +Vietnamese lists are kept by hand above the generated block, hand-curated +and unchecked against any ranking (see the note in +nameparser/config/_order_names.py). + + uv run python tools/vocab/order_names.py # print the block + uv run python tools/vocab/order_names.py --write # rewrite it + uv run python tools/vocab/order_names.py --refresh # fetch Unihan again + +The second generated block, ``COMMON_GIVEN``, is the given names the +suggester looks up at both ends of a name: per nam_dict country column +the ``TOP`` highest-band names that pass the filter, which keeps a name +of two letters or more written in ORCID's own family box at most +``MAX_FAMILY_SHARE`` of the ``MIN_USES`` or more times its holders wrote +it (a population the suggester is not tested on), and no romanized +surname this run writes. It needs nam_dict and the ORCID local corpus +cached (tools/name_usage.py and tools/differential/local_corpora.py), +each checked whole against its pinned or recorded sha256, and it writes +nothing where a block falls below its floor (``MIN_PINYIN``, +``MIN_GIVEN``) or a word lands in both tables. + +Unihan is cached outside the repository with ``tools/zh_surnames.py``'s +copy ($NAMEPARSER_TOOLS_CACHE or ~/.cache/nameparser-tools). +""" +from __future__ import annotations + +import argparse +import hashlib +import json +import sys +import zipfile +from collections import Counter +from collections.abc import Iterable, Mapping +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).parents[1])) +import name_usage # noqa: E402 (a sibling script, not a package) + +from nameparser._suggest import fold # noqa: E402 (the one fold) + +TOP = 500 +MAX_FAMILY_SHARE = 0.05 +MIN_USES = 20 +ORCID_RAW = ("local", "orcid-researchers", "raw.jsonl") +#: The fewest entries a generated block may hold: a run that yields +#: fewer read an input cut short or changed, and writes nothing +#: (2026-11-10: 329 pinyin surnames, over 4,000 given names). +MIN_PINYIN = 200 +MIN_GIVEN = 3_000 + +MODULE = Path(__file__).parents[2] / "nameparser" / "config" / "_order_names.py" +UNIHAN_URL = "https://www.unicode.org/Public/16.0.0/ucd/Unihan.zip" +UNIHAN_FILE = "unihan_16.0.0.zip" +UNIHAN_MEMBER = "Unihan_Readings.txt" + + +def read_mandarin(unihan_readings_text: str) -> dict[str, str]: + """Each character's first ``kMandarin`` reading, tones folded.""" + out: dict[str, str] = {} + for line in unihan_readings_text.splitlines(): + if line.startswith("#") or "\tkMandarin\t" not in line: + continue + source, _, readings = line.split("\t") + out[chr(int(source[2:], 16))] = fold(readings.split()[0]) + return out + + +def pinyin_surnames(surnames: Iterable[str], + mandarin: Mapping[str, str]) -> set[str]: + """The pinyin of the surnames that have one of two or more letters + (a lone vowel such as 阿's ``a`` would match every initial).""" + return {mandarin[s] for s in surnames + if s in mandarin and len(mandarin[s]) >= 2} + + +def _markers(name: str) -> tuple[str, str]: + return (f"# -- BEGIN generated: {name}; edit tools/vocab/order_names.py, " + "not this block --", + f"# -- END generated: {name} --") + + +def render_block(name: str, words: Iterable[str]) -> str: + begin, end = _markers(name) + ordered = sorted(set(words)) + rows = [", ".join(f'"{w}"' for w in ordered[i:i + 8]) + for i in range(0, len(ordered), 8)] + body = "".join(f" {row},\n" for row in rows) + return f"{begin}\n_{name} = (\n{body})\n{end}" + + +def rewrite(module_text: str, name: str, block: str) -> str: + begin, end = _markers(name) + start, stop = module_text.find(begin), module_text.find(end) + if start < 0 or stop < start: + raise ValueError(f"{MODULE} has no generated block {name} to replace") + return module_text[:start] + block + module_text[stop + len(end):] + + +def load_mandarin(refresh: bool) -> dict[str, str]: + path = name_usage.cache_dir() / UNIHAN_FILE + if refresh or not path.exists(): + name_usage._download(UNIHAN_URL, path, UNIHAN_MEMBER) + with zipfile.ZipFile(path) as zf: + return read_mandarin(zf.read(UNIHAN_MEMBER).decode("utf-8")) + + +def orcid_box_counts(path: Path) -> tuple[Counter[str], Counter[str]]: + """Folded word counts of ORCID's given-name and family-name boxes, + streamed (the file is about 1.8M lines). The file is refused unless + it is whole: its sha256 and its record count are the ones the + corpus's meta.json, written by the fetch, gives it.""" + meta_path = path.parent / "meta.json" + try: + meta = json.loads(meta_path.read_text(encoding="utf-8")) + want_sha, want_records = meta["sha256"], meta["records"] + except (OSError, ValueError, KeyError) as exc: + raise ValueError(f"{meta_path} is missing or unreadable ({exc}); " + "fetch the orcid-researchers corpus again") from exc + given: Counter[str] = Counter() + family: Counter[str] = Counter() + digest = hashlib.sha256() + records = 0 + with open(path, "rb") as fh: + for raw in fh: + digest.update(raw) + # cut at "\n" only, as corpus_build.jsonl_lines cuts + line = raw.decode("utf-8") + if not line.strip(): + continue + records += 1 + record = json.loads(line) + for box, counts in (("given", given), ("family", family)): + for word in (record.get(box) or "").split(): + counts[fold(word.strip(".,"))] += 1 + if records != want_records: + raise ValueError(f"{path} holds {records} records, not the " + f"{want_records} its meta.json counts; fetch it again") + if digest.hexdigest() != want_sha: + raise ValueError(f"{path} does not match the sha256 in its " + "meta.json; fetch it again") + return given, family + + +def common_given(namdict: name_usage.NamDict, given: Counter[str], + family: Counter[str], *, top: int, max_family_share: float, + min_uses: int, exclude: set[str]) -> set[str]: + """Per nam_dict country, the ``top`` folded single-word names of + highest band that pass the filter, unioned over the countries. + + A name passes with two or more letters, not in ``exclude``, at least + ``min_uses`` uses across ORCID's two boxes and no more than + ``max_family_share`` of them in the family box.""" + def passes(word: str) -> bool: + total = given[word] + family[word] + return (len(word) > 1 and word not in exclude and total >= min_uses + and family[word] / total <= max_family_share) + + bands: dict[str, dict[str, int]] = {c: {} for c in namdict.countries} + for entries in namdict.names.values(): + for listed, listed_bands in entries: + if " " in listed or "+" in listed: + continue + word = fold(listed) + for country, band in listed_bands.items(): + if band > bands[country].get(word, -1): + bands[country][word] = band + chosen: set[str] = set() + for ranked in bands.values(): + ordered = sorted(ranked.items(), key=lambda kv: (-kv[1], kv[0])) + chosen.update([w for w, _ in ordered if passes(w)][:top]) + return chosen + + +def load_namdict() -> name_usage.NamDict: + """nam_dict, read through name_usage, which checks the cached file + against its pinned sha256; refused where it is not cached.""" + namdict = name_usage.load_namdict(download=False) + if namdict is None: + raise ValueError( + f"{name_usage.cache_dir() / name_usage.NAMDICT_FILE} is not " + "cached; uv run python tools/name_usage.py fetches it") + return namdict + + +def surname_exclusions(pinyin: set[str]) -> set[str]: + """The surnames this run writes: the pinyin it computed and the + hand lists, which the given names must not hold. Read from this + run, not from the module it rewrites, which may be stale.""" + from nameparser.config import _order_names + return pinyin | set(_order_names._ZH_OTHER_SPELLINGS + + _order_names._KO_SPELLINGS + + _order_names._VI_SURNAMES) + + +def load_common_given(exclude: set[str]) -> set[str]: + given, family = orcid_box_counts(name_usage.cache_dir().joinpath(*ORCID_RAW)) + return common_given( + load_namdict(), given, family, top=TOP, + max_family_share=MAX_FAMILY_SHARE, min_uses=MIN_USES, + exclude=exclude) + + +def check_tables(pinyin: set[str], given: set[str]) -> None: + """Refuse a generated block below its floor, and a given name that + is a surname too.""" + for words, floor, what in ((pinyin, MIN_PINYIN, "pinyin surnames"), + (given, MIN_GIVEN, "given names")): + if len(words) < floor: + raise ValueError(f"the run found {len(words)} {what}, below the " + f"floor of {floor}: an input is cut or changed") + if both := pinyin & given: + raise ValueError("a word is in both tables: " + + ", ".join(sorted(both))) + + +def main(argv: list[str] | None = None) -> int: + parser = argparse.ArgumentParser(description=(__doc__ or "").split("\n")[0]) + parser.add_argument("--write", action="store_true", + help="rewrite the generated block") + parser.add_argument("--refresh", action="store_true", + help="fetch Unihan again") + args = parser.parse_args(argv) + from nameparser import locales + singles = [s for s in locales.ZH.lexicon._sets.surnames if len(s) == 1] + words = pinyin_surnames(singles, load_mandarin(args.refresh)) + given = load_common_given(surname_exclusions(words)) + check_tables(words, given) + block = render_block("PINYIN", words) + given_block = render_block("COMMON_GIVEN", given) + print(block) + print(given_block) + if args.write: + text = MODULE.read_text(encoding="utf-8") + text = rewrite(text, "PINYIN", block) + MODULE.write_text(rewrite(text, "COMMON_GIVEN", given_block), + encoding="utf-8") + print(f"wrote {MODULE}", file=sys.stderr) + return 0 + + +if __name__ == "__main__": + sys.exit(main())