Updated some entries in the file COUNTER_Robots_list and added 1636 n…

[CRMGB]

Updated the robots list with 1636 new robots.

Most of the new entries have been gathered from https://github.com/monperrus/crawler-user-agents/blob/master/crawler-user-agents.json using using it as a guide and detecting new bots from our user-agents dataset.

This list is designed to be used as a REGEX pattern to identify crawlers/bots from our user-agent entries and exclude them from our metrics if detected.

The following files have been modified:

CHANGES.md
COUNTER_Robots_list.json
convert_to_txt
generated/COUNTER_Robots_list.txt

[alanorth]

@CRMGB there is a ton of duplication in your proposed additions, for example:

Java/1.4.1_01
Java/1.4.1_04
Java/1.5.0_16
Java/1.6.0_04
Java/1.6.0_07
Java/1.6.0_18
Java/1.6.0_22
...

Not to mention, our list already has this much better regular expression:

^java\/\d{1,2}.\d

Also, your proposed additions have not escaped special characters like forward slashes and dots:

AHC/2.0
ASPSeek/1.2.5

And your patterns are strangely cut off at a certain line length or something?

Cookie%20Stumbler/22
CrawlConvera0.1 (Cra
Custo x.x (www.netwu
CyberSpyder Link Tes

Lastly, patterns like these would already be matched by the bot and spider patterns in our list:

purebot
Linguee Bot
voilabot
Baiduspider

So this pull request is not in good shape as is. There may be some new bot patterns we can use from the other project, but in such a large list it is very difficult to verify them. Personally, I would prefer to have bots that have been verified from access log files directly.

[CRMGB]

No problem @alanorth,
Thanks for the notes,
I will update accordingly,

Regards.