Add support for English

[ZeekYin]

No description provided.

[eiennohito]

Using regexes is probably OK (but you should use JVM API, not Scala one here), but it is better to skip ~50% of input document for language detection as I have explained. Header section usually contains a lot of not interesting stuff written in ASCII only and filtering only scripts won't help that much. There are also comments, inline stylesheets and other things we can ignore if we start language detection from the first tag after ~50% of text content

[eiennohito]

It is possible to avoid creating this string completely, you do not need it.
https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/util/regex/Pattern.html#matcher(java.lang.CharSequence) can use CharBuffers directly as inputs as they implement CharSequence interface.

[ZeekYin]

I changed the code according to your instruction, including:

1. Used java regex pattern
2. estimate from 50% ~
3. dismiss css, etc

However, it still cannot recognize English properly.

[ZeekYin]

Is the regex-based method too rudimentary? Should I use Jsoup?

[ZeekYin]

I also tried on a relatively big English corpus, but I got this

'language=ar'   'language=fr'  'language=ko'  'language=pt'  'language=uk'
'language=ast'  'language=ga'  'language=lt'  'language=ru'  'language=ur'
'language=be'   'language=gl'  'language=lv'  'language=sk'  'language=vi'
'language=bg'   'language=hi'  'language=mk'  'language=sq'  'language=zh'
'language=bn'   'language=is'  'language=mr'  'language=sr'   _SUCCESS
'language=cs'   'language=ja'  'language=mt'  'language=sv'
'language=el'   'language=km'  'language=oc'  'language=th'
'language=fa'   'language=kn'  'language=pl'  'language=tr'

There is only English disappearing. I think it's strange, so I tried an English html in this corpus, and the LangEstimator could estimate it to English correctly. Is there anything I need to change in uzushio? Any insights would be appreciated.