corpora

You could detect similar texts but that's just an arms race against new corpora being added to their generator.

Have you ever tried to use a SVM with a linear kernel, which is perfectly suitable for large text corpora?

The problem with most corpora typically used to test segmenters is that 80-90% of the sentences are the same (i.e.

Different corpora will behave differently of course, but the overall shape will likely remain similar.

What Google would be extremely good at is creating corpora for all these languages.

I'd be very curious to see performance/accuracy charts on a number of corpora in comparison to CoreNLP.

I don't know why this branch of NLP/machine learning has been fairly okay with small corpora.

See my comment below for some of the reasons I've had issues trying to test the commonly used segmentation corpora.

Large-scale corpora and computational capabilities enable more empiricist rather than rationalist approaches to human language, which has made the argumentation less personal and more productive.

Subtitles seem good, and there are corpora of casual speech.

Large corpora are available via Penn's Linguistic Data Consortium (LDC).

I'd love to test it on some of the typically used corpora.

Have you tried to evaluate your splitter on some other data, on this "typically used corpora"?

For example, the "word vector representations" can be trained easily with gensim, on arbitrary user-specified corpora, whereas spaCy loads something pre-trained, in a specific format.

Jordan and David Blei: "Statistical modeling of biomedical corpora: mining the Caenorhabditis Genetic Center Bibliography for genes related to life span" - Blei DM1, Franks K, Jordan MI, Mian IS.

He makes an excellent point about the possibility of comparing word vectors trained on different corpora to make quantitative statements about differences in culture, either over time or between sub-cultures: "I’d like to emphasize that which words are feminine or masculine, young or adult, isn’t intrinsic.