Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/data-analysis/auto-sizing-cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ firefox_desktop = ["active_hours"]
```
A `parameters` section in the TOML file is used to define the data collection period for the analysis and the parameters used to calculate sample sizes. This section consists of two subsections: `parameters.sizing` and `parameters.dates`:
1. `parameters.sizing`: Contains two tags, `power` and `effect_size`. These tags should contain lists of values for each parameter, and a sample size will be calculated for all metrics provided in the TOML file for each combination of power and effect size in those lists.
2. `parameters.dates`: Contains the `start_date` (in "%Y-%m-%d" format, e.g. "2023-01-01"), `num_dates_enrollment`, and `analysis_length` values. For details on how those values are used to query historical data, see the [Mozanalysis documentation](https://experimenter.info/experiment-sizing).
2. `parameters.dates`: Contains the `start_date` (in "%Y-%m-%d" format, e.g. "2023-01-01"), `num_dates_enrollment`, and `analysis_length` values. For details on how those values are used to query historical data, see the [Mozanalysis documentation](/data-analysis/experiment-sizing).

## CLI Commands

Expand Down
2 changes: 1 addition & 1 deletion docs/data-analysis/jetstream/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ Note that rerunning experiments may be costly!
Don't let this stop you from doing your job,
but try to avoid too much iteration on large and long-running experiments.

Has your configuration landed but generated unexpected analyses? Troubleshooting information can be found [here](https://experimenter.info/jetstream/troubleshooting) which includes instructions on accessing the Jetstream error logs.
Has your configuration landed but generated unexpected analyses? Troubleshooting information can be found [here](/data-analysis/jetstream/troubleshooting) which includes instructions on accessing the Jetstream error logs.

## Configuration File Syntax

Expand Down
4 changes: 2 additions & 2 deletions docs/data-analysis/jetstream/metrics.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,11 +35,11 @@ within and between branches of an experiment.
A small set of default, critical guardrail metrics that are specific to each platform are run by default for each experiment. Sometimes also called Core metrics.
These are defined for each platform in the metric-hub repository in [jetstream/defaults/](https://github.com/mozilla/metric-hub/tree/main/jetstream/defaults/). Look at the file for your platform. At the top you will see the metrics collected and the timeframe they are available: daily, weekly, or overall (at the end of the experiment). For help understanding any aspect of guardrail metrics on your platform - link to the file and ask in #ask-experimenter.

[Pre-defined Metrics, AKA Outcomes](https://experimenter.info/deep-dives/jetstream/outcomes) are collections of metrics that relate to each other. You must associate these pre-defined metrics (outcomes) with your experiment BEFORE LAUNCH in the Experiment Console - Metrics section.
[Pre-defined Metrics, AKA Outcomes](/data-analysis/jetstream/outcomes) are collections of metrics that relate to each other. You must associate these pre-defined metrics (outcomes) with your experiment BEFORE LAUNCH in the Experiment Console - Metrics section.
Data scientists can extend and defined outcomes in the outcomes path of the
[`metric-hub`](https://github.com/mozilla/metric-hub/tree/main/jetstream/outcomes) repository. See what [Outcomes are available](https://mozilla.github.io/metric-hub/outcomes/fenix/default-browser/).

If the metrics you need are not covered by Default (guardrail) or existing Pre-defined Outcomes - you can add a custom metric to your experiment by working with a data scientist to write a [custom configuration](https://experimenter.info/system-architecture#custom-configuration-aka-jetstream-configuration-files) for your experiment. You work with DS by signing up for [data science experiment office hours](https://docs.google.com/document/d/1dH-aG8IsYtq6881_Q_cyEtmxli0bK7nuVcUD-5D7q-s/edit?tab=t.0#heading=h.yguiolmttiw2).
If the metrics you need are not covered by Default (guardrail) or existing Pre-defined Outcomes - you can add a custom metric to your experiment by working with a data scientist to write a [custom configuration](/technical-reference/system-architecture#custom-configuration-aka-jetstream-configuration-files) for your experiment. You work with DS by signing up for [data science experiment office hours](https://docs.google.com/document/d/1dH-aG8IsYtq6881_Q_cyEtmxli0bK7nuVcUD-5D7q-s/edit?tab=t.0#heading=h.yguiolmttiw2).

## What Are the Default Guardrail Metrics?
Open the [file for your platform](https://github.com/mozilla/metric-hub/tree/main/jetstream/defaults) to see the most recent list of what is included. These don’t change often, but they can.
Expand Down
2 changes: 1 addition & 1 deletion docs/faq/getting-started.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ Nimbus is the code name of the experimentation project, so Nimbus refers to **th

- Depending when you make your change and when RS propagates to CDNs - it could take minutes to just over an hour.

- Desktop clients only recompute enrollments at startup and every six hours thereafter. So if you are looking to "naturally" get enrolled in an experiment (versus using [Preview](https://experimenter.info/previewing-experiments) to force enroll) on desktop there could be an even longer delay.
- Desktop clients only recompute enrollments at startup and every six hours thereafter. So if you are looking to "naturally" get enrolled in an experiment (versus using [Preview](/platform-guides/desktop/preview) to force enroll) on desktop there could be an even longer delay.


---
Expand Down
4 changes: 2 additions & 2 deletions docs/getting-started/for-data-scientists.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ Some other things you may be looking for are:

* Documentation about using [Jetstream](/data-analysis/jetstream/overview), Mozilla's experiment analysis tool
* Technical documentation about [datasets used in experimentation](https://docs.telemetry.mozilla.org/tools/experiments.html)
* [Process documentation](https://experimenter.info/getting-started/for-experiment-owners) for the Mozilla data science organization
* [Process documentation](/getting-started/for-experiment-owners) for the Mozilla data science organization

## What Is the Role of Experimentation at Mozilla?

Expand All @@ -22,7 +22,7 @@ This suite of experimentation tools is designed for product managers and other i
## Collaborating with Experiment Owners

Data scientists support experiment owners in setting up and interpreting their experiments.
[The Firefox experiment design process](https://experimenter.info/getting-started/for-experiment-owners) describes the process for both data scientists and stakeholders.
[The Firefox experiment design process](/getting-started/for-experiment-owners) describes the process for both data scientists and stakeholders.

Experiment owners come to [Office Hours](https://docs.google.com/document/d/1dH-aG8IsYtq6881_Q_cyEtmxli0bK7nuVcUD-5D7q-s/edit#) to talk to data scientists to:

Expand Down
4 changes: 2 additions & 2 deletions docs/getting-started/for-engineers.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,12 +10,12 @@ Engineers are typically tasked with implementing an experiment on a new surface

* [Getting started with Nimbus, for Android Engineers](/platform-guides/android/integration)
* [Getting started with Nimbus for iOS Engineers](/platform-guides/ios/integration)
* [Getting Started for Nimbus Web Integration](platform-guides/web/integration)
* [Getting Started for Nimbus Web Integration](/platform-guides/web/integration)
* [Mobile UI needed for Nimbus](/platform-guides/android/mobile-ui)

## To Get Started with Implementation

* Review the experiment design document.
* Determine if you need to implement any additional telemetry collection
* Check if you need to implement a custom audience
* Confirm the name of the feature and follow the guide to [adding a new feature to the manifest](https://experimenter.info/feature-definition#to-define-your-feature-in-the-feature-manifest-file)
* Confirm the name of the feature and follow the guide to [adding a new feature to the manifest](/technical-reference/feature-definition#to-define-your-feature-in-the-feature-manifest-file)
4 changes: 2 additions & 2 deletions docs/getting-started/for-experiment-owners.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ Responsibilities include:

## Where Do I Start?

- Watch the 5 minute video on [experiment workflow](https://experimenter.info/workflow/overview) for an overview of the experiment lifecycle to understand the stages / ordering. Use the [Workflow Miro board](https://experimenter.info/workflow/overview) as your source for current links and guidance.
- Watch the 5 minute video on [experiment workflow](/workflow/overview) for an overview of the experiment lifecycle to understand the stages / ordering. Use the [Workflow Miro board](/workflow/overview) as your source for current links and guidance.

<img title="design-miro-board" src="/img/getting-started/design-miro-board.png"/>

Expand All @@ -43,7 +43,7 @@ Responsibilities include:

## Key Resources for Experiment Owners

Rather than duplicate links that may later break as the process improves - links to the resources and training videos are in the [Workflow Miro Board](https://experimenter.info/workflow/overview).
Rather than duplicate links that may later break as the process improves - links to the resources and training videos are in the [Workflow Miro Board](/workflow/overview).

- There are several weekly touchpoints [Office Hours](https://mozilla-hub.atlassian.net/wiki/spaces/DATA/pages/6849684/Office+Hours) for in person assistance from subject matter experts in different areas.
- If you are confused on any aspect - that's expected the first few experiments until you've gotten the rhythm. Ask in the #ask-experimenter Slack channel - it is an open community of support from several disciplines.
Expand Down
2 changes: 1 addition & 1 deletion docs/getting-started/for-reviewers.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ If you are interested in learning more about your responsibilities, you can find

## Onboarding for New Authors/Owners (L2)

In order to use the Nimbus platform to create and launch experiments, you will need to go through the [Experiment Owner onboarding process](https://experimenter.info/getting-started/for-experiment-owners). Please ask in [#ask-experimenter](https://mozilla.slack.com/archives/CF94YGE03) if you have any questions and for a shadow on the first experiments(s) you create.
In order to use the Nimbus platform to create and launch experiments, you will need to go through the [Experiment Owner onboarding process](/getting-started/for-experiment-owners). Please ask in [#ask-experimenter](https://mozilla.slack.com/archives/CF94YGE03) if you have any questions and for a shadow on the first experiments(s) you create.

## Onboarding for New Reviewers (L3)

Expand Down
2 changes: 1 addition & 1 deletion docs/messaging/desktop/desktop-messaging.md
Original file line number Diff line number Diff line change
Expand Up @@ -104,5 +104,5 @@ Paste your message JSON into the editor to render it immediately and check the l
## Additional Resources

- [Messaging System Documentation](/messaging/overview)
- [Experimenter Documentation](https://experimenter.info)
- [Experimenter Documentation](/)
- [nimbus-devtools Guide](/resources/nimbus-devtools-guide)
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ The onboarding feature enables staff — most likely experiment owners, product

At the time of writing, Desktop first run experiments are only supported on Windows. This note describes what first run experiments are, sketches how first run experiments work on Windows, and suggests approaches for supporting first run experiments on macOS.

First run experiments are those that make changes to onboarding, that need data from brand-new clients, or that otherwise relate to clients who are using Firefox on their device for the first time, [reference](https://experimenter.info/advanced/first-run-experiments/#how-do-i-know-if-an-experiment-should-be-first-run).
First run experiments are those that make changes to onboarding, that need data from brand-new clients, or that otherwise relate to clients who are using Firefox on their device for the first time, [reference](/advanced/first-run-experiments#how-do-i-know-if-an-experiment-should-be-first-run).

## How do first run experiments work on Windows? ##

Expand Down
4 changes: 2 additions & 2 deletions docs/platform-guides/desktop/preview.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,11 +19,11 @@ These steps only apply to Firefox Desktop Version 90+. You can find [instruction
## Self-Testing with Nimbus Dev Tools Add-on
If you aren't a developer, don't let the "Dev Tools" or add-on scare you away. This is the easier path!

Here is a 65 second video on [how to install Nimbus dev tools](https://mozilla.hosted.panopto.com/Panopto/Pages/Viewer.aspx?id=b7b2d02b-79ba-43a0-a708-b2a60107f0bf). Watching the video takes half the time to getting Nimbus Dev Tools installed. Here is the [link to docs on how to use Nimbus Dev Tools](https://experimenter.info/nimbus-devtools-guide#installation). The very first link on that page is to the [Nimbus DevTools GitHub repository](https://github.com/mozilla-extensions/nimbus-devtools/releases), where you can find the one click XPI file install.
Here is a 65 second video on [how to install Nimbus dev tools](https://mozilla.hosted.panopto.com/Panopto/Pages/Viewer.aspx?id=b7b2d02b-79ba-43a0-a708-b2a60107f0bf). Watching the video takes half the time to getting Nimbus Dev Tools installed. Here is the [link to docs on how to use Nimbus Dev Tools](/resources/nimbus-devtools-guide#installation). The very first link on that page is to the [Nimbus DevTools GitHub repository](https://github.com/mozilla-extensions/nimbus-devtools/releases), where you can find the one click XPI file install.

Here is a 5 minute video on [using Nimbus Dev Tools to test a simple experiment](https://mozilla.hosted.panopto.com/Panopto/Pages/Viewer.aspx?id=98d797c3-35e2-413a-b68d-b2a5018b814e). Watching this will show you how to verify simple experiments. Here is the link to the [demo experiment used in the video](https://experimenter.services.mozilla.com/nimbus/training-only-for-dev-tools/summary), if you don't have one of your own that you want to test.

Here is the link to the [DevTool user guide](https://experimenter.info/nimbus-devtools-guide/)
Here is the link to the [DevTool user guide](/resources/nimbus-devtools-guide)

## How Do I Know If I Have a Simple Experiment
There is a Miro flow that can [help determine if you have a good test case here](https://miro.com/app/board/uXjVK_27t2o=/?share_link_id=702720260336). If it asks for a password - "patterns". Reach out in #ask-experimenter if you are unsure and someone can help you determine the complexity.
Expand Down
4 changes: 2 additions & 2 deletions docs/platform-guides/ios/microsurveys.md
Original file line number Diff line number Diff line change
Expand Up @@ -85,9 +85,9 @@ For the survey, you can control the question, survey options and UTM parameter f

#### microsurveyConfig

To customize the survey options, icon and UTM parameter, we have to add the `microsurveyConfig` field object. For the survey options, there exists a new field called `options`, it’s an array in which each item will represent a possible survey option for the user to select. Currently, the array is composed of type `Text`, which you can read more about our bundle types [here](https://experimenter.info/fml-spec/#bundle-types).
To customize the survey options, icon and UTM parameter, we have to add the `microsurveyConfig` field object. For the survey options, there exists a new field called `options`, it’s an array in which each item will represent a possible survey option for the user to select. Currently, the array is composed of type `Text`, which you can read more about our bundle types [here](/technical-reference/fml/fml-spec#bundle-types).

For the icon, add the `icon` field under `microsurveyConfig`. This is related to the asset used next to the survey question. It should change for each survey feature, but it needs to be a valid icon resource id indicated in our app. This field is composed of type `Image`, which you can read more about our bundle types [here](https://experimenter.info/fml-spec/#bundle-types). The list of standard images can be found [here](https://github.com/mozilla-mobile/firefox-ios/blob/main/BrowserKit/Sources/Common/Constants/StandardImageIdentifiers.swift).
For the icon, add the `icon` field under `microsurveyConfig`. This is related to the asset used next to the survey question. It should change for each survey feature, but it needs to be a valid icon resource id indicated in our app. This field is composed of type `Image`, which you can read more about our bundle types [here](/technical-reference/fml/fml-spec#bundle-types). The list of standard images can be found [here](https://github.com/mozilla-mobile/firefox-ios/blob/main/BrowserKit/Sources/Common/Constants/StandardImageIdentifiers.swift).

For the UTM parameter, add the `utm-content` field. This is related to the utm-content value that we pass in as a query param for our privacy notice. It should change for each survey feature, but it can be any string. Here is an example case:
`https://www.mozilla.org/en-US/privacy/firefox/?utm_medium=firefox-mobile&utm_source=modal&utm_campaign=microsurvey&utm_content=homepage`
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ Wherever a JSON file is specified, you can use JSON or YAML, just by changing th
nimbus-cli --app fenix --channel developer features release-android-onboarding-redesign --branch treatment-a --output new-onboarding.yaml
```

This can be useful to add the winning branch configuration to [the feature defaults in the feature manifest](https://experimenter.info/fml-spec/#feature-defaults) in the app's codebase, to make the change permanent.
This can be useful to add the winning branch configuration to [the feature defaults in the feature manifest](/technical-reference/fml/fml-spec#feature-defaults) in the app's codebase, to make the change permanent.
:::

By default, the `features` command outputs the experimental feature configurations for all features involved in the experiment. At the moment, this is always one.
Expand Down
2 changes: 1 addition & 1 deletion docs/platform-guides/web/integration.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ To integrate Nimbus (Cirrus) with your web application, these are high-level ste
- Integrate [Glean](https://docs.telemetry.mozilla.org/concepts/glean/glean.html) on your web application.

2. **FML**
- Make [Feature Manifest Language (FML) configuration](https://experimenter.info/fml-spec#introducing-the-fml) available in your repository, for example:
- Make [Feature Manifest Language (FML) configuration](/technical-reference/fml/fml-spec#introducing-the-fml) available in your repository, for example:
```
about:
description: Nimbus Feature Manifest for Experimenter Web testing
Expand Down
2 changes: 1 addition & 1 deletion docs/resources/foxfooding.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ Start by installing Nimbus Dev Tools. If you aren't a developer, don't let the

## If You Have Problems Installing Nimbus Dev Tools...
- Here is a [65 second video on how to install Nimbus dev tools](https://mozilla.hosted.panopto.com/Panopto/Pages/Viewer.aspx?id=b7b2d02b-79ba-43a0-a708-b2a60107f0bf). Watching the video takes half the time to getting Nimbus Dev Tools installed.
- Here is the [link to docs on how to install Nimbus Dev Tools](https://experimenter.info/nimbus-devtools-guide#installation).
- Here is the [link to docs on how to install Nimbus Dev Tools](/resources/nimbus-devtools-guide#installation).

## If You Have Questions How to Use Nimbus Dev Tools...

Expand Down
2 changes: 1 addition & 1 deletion docs/whats-news/2022/2022-06.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ We're thrilled to have them all aboard!

## Exciting Changes
### Mobile Messaging
Nimbus now supports the ability to send messages to mobile clients through messaging surfaces in each mobile application. This unlocks the ability for Nimbus to run surveys, update users about important new information, or incorporate messages into other experiments/rollouts. More detailed information [can be found here](https://experimenter.info/mobile-messaging).
Nimbus now supports the ability to send messages to mobile clients through messaging surfaces in each mobile application. This unlocks the ability for Nimbus to run surveys, update users about important new information, or incorporate messages into other experiments/rollouts. More detailed information [can be found here](/messaging/desktop/mobile-messaging).
### Mobile Language Targeting
Mobile devices use a different [list of languages](https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes) than [desktop Firefox does](https://firefox-source-docs.mozilla.org/intl/locale.html), and so the locale field on the Audience page didn't work as expected for mobile experiments. To address this, we've added a new 'Language' field on the Audience page that can be used to target device languages for mobile experiments.
### Mobile Country Targeting
Expand Down
Loading
Loading