Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

added hindi language toml and wiki sample #89

Draft
wants to merge 10 commits into
base: main
Choose a base branch
from
Draft

added hindi language toml and wiki sample #89

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
4c47b54
added hindi language toml and wiki sample
Feb 4, 2020
1e8a10b
added blacklist file, updates toml, removed sample sentence file
karthiksibm Mar 4, 2020
baf9585
changed hindi file names to hi
karthiksibm Mar 5, 2020
d6b5dd5
remove old named hindi files
karthiksibm Mar 5, 2020
baae66e
Merge pull request #1 from Common-Voice/master
karthiksibm Mar 5, 2020
25e3e07
updates to hi.toml
karthiksibm Mar 9, 2020
79757bd
Merge branch 'master' of https://github.com/karthiksibm/common-voice-…
karthiksibm Mar 9, 2020
77d6208
updated extractor.rs to run replacer rules before SentenceTokenizer. …
karthiksibm Mar 10, 2020
99b9730
hi.toml update
karthiksibm Mar 10, 2020
2dd5eab
latest rules and blacklist words with frequency 50
karthiksibm May 8, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 9 additions & 6 deletions src/extractor.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,16 +55,19 @@ fn choose(
predicate: impl FnMut(&Rules, &str) -> bool,
mut replacer: impl FnMut(&Rules, &str) -> String,
) -> Vec<String> {
let sentences_replaced_abbreviations: Vec<String> = SentenceTokenizer::<Standard>::new(text, training_data)
.map(|item| { replacer(rules, item) })
.collect();

let replaced_text = replacer(rules, text);
let mut sentences: Vec<String> = vec![];
let tokenizer = SentenceTokenizer::<Standard>::new(&replaced_text, training_data);
for sentence in tokenizer {
sentences.push(String::from(sentence));
}

if config.no_check {
sentences_replaced_abbreviations
sentences
} else {
pick_sentences(
rules,
sentences_replaced_abbreviations,
sentences,
existing_sentences,
config.max_sentences_per_text,
predicate,
Expand Down
Loading