De itn cardinal decouple - #484
Conversation
Signed-off-by: Adelina Dunina <adunina@nvidia.com>
0d2d230 to
36d900d
Compare
for more information, see https://pre-commit.ci
| @@ -0,0 +1,17 @@ | |||
| null 0 | |||
There was a problem hiding this comment.
let's have 0 be a standalone file, 1-9 from this file be replaced by the digits file, and 10-12 be a separate irregular teens file
| @@ -0,0 +1,72 @@ | |||
| 12 21 | |||
There was a problem hiding this comment.
let's have this be a function in the script instead of a file
| @@ -0,0 +1,9 @@ | |||
| eins 01 | |||
There was a problem hiding this comment.
let's remove this file for now. when we get to date, we can create rules to build it from digits instead
| @@ -0,0 +1,8 @@ | |||
| zwanzig 20 | |||
There was a problem hiding this comment.
let's use the tens file + delete 0 to replace this file
| @@ -0,0 +1,102 @@ | |||
| Nummer | |||
There was a problem hiding this comment.
let's remove this from this PR and we can review in the context of measure instead
| @@ -1,4 +1,4 @@ | |||
| # Copyright (c) 2021, NVIDIA CORPORATION & AFFILIATES. All rights reserved. | |||
| # Copyright (c) 2025, NVIDIA CORPORATION & AFFILIATES. All rights reserved. | |||
There was a problem hiding this comment.
let's not change copyright date from files that already existed before. always good to make sure that the copyright overall matches the standard
| Args: | ||
| tn_cardinal_verbalizer: TN cardinal verbalizer | ||
| Finite state transducer for verbalizing cardinal numbers. Note that the verbalizer retains period-separated formatting. | ||
| e.g. 'cardinal { negative: "-" integer: "1.234.512.102" }' -> -1.234.512.102 |
There was a problem hiding this comment.
convention is 'negative: true' comes from the tagger, then verbalizer turns that into the minus sign
| ) | ||
|
|
||
| # handles all elements of a cardinal integer | ||
| integer_chars = NEMO_DIGIT | pynini.accep(".") |
There was a problem hiding this comment.
you can look at English. accepting NEMO_NOT_QUOTE might be enough
|
|
||
| # Handles noun + number combinations, where the noun forces full denormalization | ||
| # The nouns are implemented as a .tsv list | ||
| nouns_forcing_denormalization = pynini.string_file( |
There was a problem hiding this comment.
I'd rather change this to measure too
|
|
||
| class CardinalFst(GraphFst): | ||
| """ | ||
| Finite state transducer for classifying cardinals. Numbers below ten are not converted. |
There was a problem hiding this comment.
let's keep the original definition and add examples as needed
| self.graph_single_and_double_digits = graph_single_and_double_digits.optimize() | ||
|
|
||
| hundert = pynini.accep("hundert") | pynini.accep("ein hundert") | ||
| hundreds = (pynini.cross(hundert, "100")) | ( |
There was a problem hiding this comment.
let's not hardcode any terms and use tsv's instead
Signed-off-by: Adelina Dunina <adunina@nvidia.com>
Signed-off-by: Adelina Dunina <adunina@nvidia.com>
Signed-off-by: Adelina Dunina <adunina@nvidia.com>
for more information, see https://pre-commit.ci
Signed-off-by: Adelina Dunina <adunina@nvidia.com>
Signed-off-by: Adelina Dunina <adunina@nvidia.com>
Signed-off-by: Adelina Dunina <adunina@nvidia.com>
Signed-off-by: Adelina Dunina <adunina@nvidia.com>
Signed-off-by: Adelina Dunina <adunina@nvidia.com>
Signed-off-by: Adelina Dunina <adunina@nvidia.com>
Signed-off-by: Adelina Dunina <adunina@nvidia.com>
for more information, see https://pre-commit.ci
2 participants
What does this PR do ?
Replace coupled German ITN Cardinal in
de/with the independent grammar fromger/(own graphs, no invert of TN Cardinal).Before your PR is "Ready for review"
Pre checks:
git commit -sto sign.pytestor (if your machine does not have GPU)pytest --cpufrom the root folder (given you marked your test cases accordingly@pytest.mark.run_only_on('CPU')). - Yesbash tools/text_processing_deployment/export_grammars.sh --MODE=test ...- Nopytestand Sparrowhawk here.__init__.pyfor every folder and subfolder, includingdatafolder which has .TSV files?Copyright (c) 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.to all newly added Python files?Copyright 2015 and onwards Google, Inc.. See an example here.try import: ... except: ...) if not already done.Notes:
pytest tests/nemo_text_processing/de/test_cardinal.py --cpu --cache-clear→ 120 passed (ITN + TN). Fullde/suite not required for this class PR.tests/nemo_text_processing/de/data_inverse_text_normalization/test_cases_cardinal.txt(same format Sparrowhawk uses).fst/fst.h). Still working on this. Opening as Draft if needed.deCardinal is replaced.pynini_export.pyalready supportsde. Docs not updated.ger/is left in place as the draft for later classes.graph_utils.pyfromgerintode(includes Google copyright line where applicable).PR Type: