LanguageNet: Large Scale Multilingual Paraphrase Corpus

We are building large scale multilingual paraphrase datasets right now. As planned, we will have 10-language corpus and each language has ~50k pairs!

Sentence 1	Label	Sentence 2
Samsung halts production of its Galaxy Note 7 as battery problems linger.	True	#Samsung temporarily suspended production of its Galaxy #Note7 devices following reports
CO2 levels mark ‘new era’ in the world’s changing climate.	True	CO2 levels haven’t been this high for 3 to 5 million years.
The 7 biggest changes Obamacare made , and those that may disappear.	False	What a repeal of Obamacare would look like , in plain English.
Fraugster , a startup that uses AI to detect payment fraud , raises $5M.	False	AI is on the rise and in this case being applied to something worthwhile payment fraud.

Publication	Model	F1
Das et al.'09	Logistic Regression: n-gram overlap features	0.683
Xu et al.'14	LEX-WMF: logistic regression + weighted matrix factorization	0.693
He et al.'16	PWIM: pairwise word interaction model	0.749
Lan et al.'18	Subword-PWIM: subword embedding based PWIM with multi-task LM	0.768