UX for scaffold installs (init command)

[aaronsteers]

Opening this discussion to think through what scaffold-install interface we want to promote.

Some things to consider:

1. We may want to accept parameters during scaffold install.
2. A common CLI paradigm would be preferable. (Proposing init below.)
3. Helpful if we can leave space to later add the ability to upgrade the scaffold with something like copier.
4. Scaffold creation (run exactly once per project) should be distinct from per-workstation installs and per-backend setup operations.

@pandemicsyn, @edgarrmondragon, @WillDaSilva - Would love your thoughts on this...

[aaronsteers]

Regarding the CLI paradigm...

Since may projects support an init command that creates their scaffold, the command init is probably a good place to start as a command name proposal.

dbt init                    # scaffold a new dbt project (standalone usage)
meltano init                # scaffold a new meltano project
meltano invoke dbt-ext init # Add dbt scaffold to a meltano project

[aaronsteers]

Regarding accepting parameters...

meltano init and dbt init both accept parameters at the command line, but they can prompt you interactively as well.

We probably want to leave space for parameters to be passed, as well as parsing from settings or environment variables where applicable, as well as leaving open the possibility for the extension to prompt for parameters interactively.

[aaronsteers]

Regarding upgrades, we could at some time in the future use a init --update or init --upgrade command to attempt upgrading the scaffolding if the extension supports this option.

This is low pri as of now, and likely would not be introduced until much later.

[aaronsteers]

Distinguishing what init does and doesn't do...

When executed:

• Only needs to be executed once per project.
• Resources should be project-level and not OS-level.
• Should not need to be run again on each workstation. (Contrast with install or init-db commands below.)

Does:

• Create new subdirectories as needed.
• Setup the resources that define the scaffolding, aka the source-controlled and project-level file resources.
  • Includes any needed scripts, directories, config files, etc. - anything needed to run the program that is not handled with meltano add and which happens logically after pip install.
  • Installs or updates .gitignore file(s) in order to keep operation 'safe' given the context of the added tool.
  • Includes any readme files if needed.

Doesn't:

• init doesn't install the plugin itself
• init doesn't install the extension

Contrast with similar operations:

• init-db - Initializes the database, which is .gitignored and therefor doesn't fit into the "scaffold" resource category.
  • I'm open to other names for this, perhaps setup or setup db or similar, but for now init-db seems clear enough for conversational purposes.
  • Often this runs as a pre-hook so the user won't actually need to do anything. An empty DB can often be set up without interaction.
  • This may have a function for 'admin user creation' and/or 'initial auth key creation'.
• install - Installs binaries or other resources at the OS level. Similar to init-db, these are not committed to source control and therefor not part of the scaffold.
  • If install actually does install shell scripts or Dockerfile or similar, which bring capabilities into the project by installing them into the project source code, then those do belong in init scaffold. The distinction is whether the output of the install goes into a durable source-controlled resource, versus those which are external to the project and exist only at the container/OS/workstation level.
  • E.g. Gradle installs into existing projects by saving .sh and .bat scripts directly into the repo root. Hence, those would get installed with the init paradigm and not via install. Java is a counter example, which needs to be installed at the OS level and definitely would not be committed back as a project-level resource, so it would be delivered via install.
  • Simple functional test between init and install:
    • If it needs to run once per project: use init
    • If it needs to run once per workstation: use install
    • If it needs to run once per backend: use init-db

[pandemicsyn]

@aaronsteers just a fyi - I transferred this issue from the dbt-ext repo to the edk repo since this seemed more like EDK level topic.

[aaronsteers]

@pandemicsyn - Good call - thanks!