feat: support website i18n for keyboard downloads

[mcdurdin]

Keyman 14.0 - 18.0 have a dependency on /keyboards which means that we cannot rewrite URLs there when we are in 'embed' mode for those apps. The apps check if the URL starts with /keyboards, and handle those paths internally, while opening other URLs in an external browser.

• Android:

  keyman/android/KMAPro/kMAPro/src/main/java/com/tavultesoft/kmapro/KMPBrowserActivity.java

  Line 48 in 4ffabeb

  private static final String INTERNAL_KEYBOARDS_LINK_FORMATSTR = "^http(s)?://(%s|%s)/keyboards([/?].*)?$";

• Developer:

  keyman/developer/src/tike/project/Keyman.Developer.UI.Project.UfrmCloneKeymanCloudProjectParameters.pas

  Lines 198 to 203 in f2f3e32

  	// We want to match on keyman.com/keyboards/<id>
  	// but we need to avoid keyman.com/keyboards/h/<custom-keyboard-home-page>
  	if u.Path.StartsWith(URLSubPath_KeymanDeveloper_Clone_Keyboards) and
  	not u.Path.StartsWith(URLSubPath_KeymanDeveloper_Clone_Keyboards_Custom)
  	then FKeymanID := u.Path.Substring(URLSubPath_KeymanDeveloper_Clone_Keyboards.Length)
  	else FKeymanID := '';

• iOS:

  keyman/ios/engine/KMEI/KeymanEngine/Classes/UniversalLinks.swift

  Lines 21 to 81 in ff60148

  	// The standard technique for launching URLs externally is dependent upon being
  	// within an app, not in an app-extension. Furthermore, that distinction is made
  	// at bundle compile-time... so we simply CAN'T condition on that within the framework.
  	// It's thus up to the app to define this method-field when desired.
  	public static var externalLinkLauncher: ((URL) -> Void)? = nil
  	// e.g. https://keyman.com/keyboards/install/foo
  	private static let KEYBOARD_INSTALL_LINK_REGEX = try! NSRegularExpression(pattern: "^http(?:s)?://keyman(?:-staging)?\\.com(?:\\.local)?/keyboards/install/([^?/]+)(?:\\?(.+))?$")
  	// e.g. http://keyman.com.local/keyboards/foo
  	private static let KEYBOARD_MATCH_ROOT_REGEX = try! NSRegularExpression(pattern: "^http(?:s)?://keyman(?:-staging)?\\.com(?:\\.local)?/keyboards([/?].*)?$");
  	// e.g. https://keyman-staging.com/go/windows/14.0/download-keyboards?version=14.0.146.0
  	private static let KEYBOARD_MATCH_GO_REGEX = try! NSRegularExpression(pattern: "^http(?:s)?://keyman(?:-staging)?\\.com(?:\\.local)?/go/windows/[^/]+/download-keyboards")
  	public static func tryParseKeyboardInstallLink(_ link: URL) -> ParsedKeyboardInstallLink? {
  	let linkString = link.absoluteString
  	// If it matches the format for the Keyboard Universal Link URL Pattern...
  	// (see https://docs.google.com/document/d/1rhgMeJlCdXCi6ohPb_CuyZd0PZMoSzMqGpv1A8cMFHY/edit?ts=5f11cb13#heading=h.qw7pas2adckj)
  	if let match = KEYBOARD_INSTALL_LINK_REGEX.firstMatch(in: linkString,
  	options: [],
  	range: NSRange(location: 0, length: linkString.utf16.count)) {
  	let keyboard_id_range = Range(match.range(at: 1), in: linkString)!
  	let keyboard_id = String(linkString[keyboard_id_range])
  	var lang_id: String? = nil
  	let urlComponents = URLComponents(string: linkString)!
  	if let lang_id_component = urlComponents.queryItems?.first(where: { $0.name == "bcp47" }) {
  	lang_id = lang_id_component.value
  	}
  	return ParsedKeyboardInstallLink(keyboard_id: keyboard_id, lang_id: lang_id)
  	} else {
  	return nil
  	}
  	}
  	/**
  	* Returns `true` for links that should be opened externally, rather than in an app-hosted WebView.
  	* `false` is returned for links we consider to be 'internal' within the Keyman ecosystem.
  	*/
  	public static func isExternalLink(_ link: URL) -> Bool {
  	let linkString = link.absoluteString
  	let linkRange = NSRange(location: 0, length: linkString.utf16.count)
  	// Case 1: the link captured for keyboard search
  	if let _ = tryParseKeyboardInstallLink(link) {
  	return false
  	} else if let _ = KEYBOARD_MATCH_ROOT_REGEX.firstMatch(in: linkString,
  	options: [],
  	range: linkRange) {
  	return false
  	} else if let _ = KEYBOARD_MATCH_GO_REGEX.firstMatch(in: linkString,
  	options: [],
  	range: linkRange) {
  	return false
  	} else if linkString.starts(with: "keyman:") {
  	return false
  	}
  	return true
  	}

• Linux:

  keyman/linux/keyman-config/keyman_config/downloadkeyboard.py

  Lines 87 to 105 in 596371d

  	if decision_type == WebKit2.PolicyDecisionType.NAVIGATION_ACTION:
  	nav_action = decision.get_navigation_action()
  	request = nav_action.get_request()
  	uri = request.get_uri()
  	logging.debug("nav request is for uri %s", uri)
  	parsed = urllib.parse.urlparse(uri)
  	if parsed.path.startswith('/keyboards/install/'):
  	qs = urllib.parse.parse_qs(parsed.query)
  	package_id = parsed.path.split('/')[-1]
  	downloadfile = os.path.join(get_download_folder(), package_id)
  	download_url = f'{KeymanComUrl}/go/package/download/{package_id}?platform=linux&tier={__tier__}'
  	if 'bcp47' in qs:
  	self.language = qs['bcp47'][0]
  	download_url += '&bcp47=' + qs['bcp47'][0]
  	else:
  	self.language = None
  	if self._process_kmp(download_url, downloadfile):
  	decision.ignore()
  	return True

• mac:

  keyman/mac/Keyman4MacIM/Keyman4MacIM/KMDownloadKeyboard/KMDownloadKBWindowController.m

  Lines 47 to 59 in e5820f5

  	// The pattern for matching links matches work in #3602
  	NSString* urlPathMatchKeyboardsInstall = @"^http(?:s)?://keyman(?:-staging)?\\.com(?:\\.local)?/keyboards/install/([^?/]+)(?:\\?(.+))?$";
  	// e.g. https://keyman.com/keyboards/install/foo
  	NSString* urlPathMatchKeyboardsRoot = @"^http(?:s)?://keyman(?:-staging)?\\.com(?:\\.local)?/keyboards([/?].*)?$";
  	// http://keyman.com.local/keyboards/foo
  	NSString* urlPathMatchKeyboardsGo = @"^http(?:s)?://keyman(?:-staging)?\\.com(?:\\.local)?/go/macos/[^/]+/download-keyboards";
  	// https://keyman-staging.com/go/macos/14.0/download-keyboards?version=14.0.146.0
  	NSRange range = NSMakeRange(0, url.length);
  	NSError* error;
  	NSRegularExpression* regexInstall = [NSRegularExpression regularExpressionWithPattern: urlPathMatchKeyboardsInstall options: 0 error: &error];
  	NSRegularExpression* regexRoot = [NSRegularExpression regularExpressionWithPattern: urlPathMatchKeyboardsRoot options: 0 error: &error];
  	NSRegularExpression* regexGo = [NSRegularExpression regularExpressionWithPattern: urlPathMatchKeyboardsGo options: 0 error: &error];

• Windows:

  keyman/windows/src/desktop/kmshell/install/UfrmInstallKeyboardFromWeb.pas

  Lines 129 to 142 in 653cb3b

  	// This introduces some deeper knowledge of URL paths in Keyman Configuration, which is
  	// a bit of a shame, because we try to keep our internal knowledge to /go/ urls on
  	// keyman.com. However, there is not really any great way around this that I've found,
  	// which allows us to handle internal navigation on the keyboard search and still lets
  	// us open other URLs that may be in the search results in an external browser.
  	Handled :=
  	(IsMain and not IsLocalUrl(Url) and not Url.StartsWith('keyman:')) or // prevent Ctrl+click on iframe-internal link navigating top
  	IsPopup or
  	TRegEx.IsMatch(Url, URLPath_RegEx_MatchKeyboardsInstall) or // capture https://keyman.com/keyboards/install/*
  	(not TRegEx.IsMatch(Url, UrlPath_RegEx_MatchKeyboardsRoot) and // don't capture https://keyman.com/keyboards*
  	not Url.StartsWith('keyman:') and // don't capture keyman:*
  	not IsLocalUrl(Url) and // don't capture the external frame or its resources
  	not TRegEx.IsMatch(Url, UrlPath_RegEx_MatchKeyboardsGo)); // don't capture the launch url https://keyman.com/go/windows/download-keyboards

• keyman.com:
  • /.htaccess
  • /go/.htaccess
  • search.mjs
  • Head.php: remove invalidLocale redirection - see below
  • Locale.php: remove general locale fallback - see below

We should split this dependency out so that the apps have a private path on keyman.com for presenting the keyboard search. This could be e.g. under /go/app/keyboards/ or similar. For forward-compatibility, we should continue to use url rewriting to avoid reference to .php in the URLs. We should probably match the /keyboards/ paths and could consider avoiding the /keyboards/h/ home pages?

Ideally if /go/app/keyboards was loaded in a non-embedded context, detect and redirect to /keyboards.

Lang and platform should be parameters for the paths contained here.

For unsupported locales coming from apps, we want to redirect to an appropriate fallback locale (e.g. es-es --> es). This is the only place we want to do fallback at this point -- if any other URL has an unsupported locale, we should redirect to /en for now.

Note that site entry should use a browser-lang-based algorithm for selecting the initial locale, but that's a separate project.

Relates-to:

• epic: i18n-url-scheme 🌋 keyman.com#696

[darcywong00]

Does this mean 14.8-18.0 apps keep using keyman.com/keyboards (go/app/version/download-keyboard) links
and 19.0 apps go to something like:

/go/app/{version?}/keyboards?lang={locale}

[mcdurdin]

Does this mean 14.8-18.0 apps keep using keyman.com/keyboards (go/app/version/download-keyboard) links and 19.0 apps go to something like:

/go/app/{version?}/keyboards?lang={locale}

Yeah, that's kinda what I am thinking. But not sold yet, we may just keep the same scheme and add a lang parameter which overrides the i18n paths.