conda-forge · jaimergp · Aug 13, 2024 · May 1, 2024 · May 1, 2024 · May 1, 2024
@@ -32,7 +32,20 @@
       "maintainer/pinning_deps",
       "maintainer/conda_forge_yml",
       "maintainer/knowledge_base",
-      "maintainer/maintainer_faq"
+      "maintainer/maintainer_faq",
+      {
+        "type": "category",
+        "label": "Understanding conda-forge",
+        "link": {
+          "type": "doc",
+          "id": "maintainer/understanding_conda_forge/index"
+        },
+        "items": [
+          "maintainer/understanding_conda_forge/life_cycle",
+          "maintainer/understanding_conda_forge/staged_recipes",
+          "maintainer/understanding_conda_forge/feedstocks"
+        ]
+      }
     ]
   },
   "glossary"

@@ -28,6 +28,14 @@ Content Delivery Network. CDNs are geographically distributed networks of server
 
 [Learn more](https://en.wikipedia.org/wiki/Content_delivery_network).
 
+## CDN
+
+**C**ontent **D**elivery **N**etwork. CDNs are geographically distributed networks of servers that mirror
+contents of a primary source. Having multiple servers offering the same content increases
+performance (reduced latency, higher download speeds) and availability.
+
+[Learn more](https://en.wikipedia.org/wiki/Content_delivery_network).
+
 <a id="term-CDT"></a>
 
 ## CDT

@@ -0,0 +1,185 @@
+# The feedstock
+
+A feedstock is the home of one recipe; it may produce more than one conda artifact in the case of a multi-output recipe.
+The feedstock is where for most events in the life of a recipe take place.
+
+## Initialization
+
+The feedstock repository is created following the merging in `staged-recipes` (see [Feedstock creation](staged_recipes.md#feedstock-creation) for details on that process).
+
+## Package building diagram
+
+Package building can be triggered by several events, which are described in the next section.
+In all of those cases, the following sequence plays out.
+
+```mermaid
+sequenceDiagram
+    box github
+    participant f as feedstock
+    end
+    box CI provider
+    participant c as CI
+    end
+    box distribution infrastructure
+    participant s as cf-staging channel
+    participant cf as conda-forge channel
+    participant cdn as CDN
+    end
+    f->>+c: trigger build
+    Note right of c: 1. build<br/>2. local validate
+    opt if valid and required (4.)
+    c->>s: upload packages
+    s->>c: report reception success/failure
+    Note right of s: 3. remote validate
+    s->>cf: upload packages
+    cf->>cdn: upload packages
+    end
+    c->>-f: report success/failure
+```
+
+:::note
+
+1. For Linux and `noarch` packages, the build itself is carried out in a Docker container on the CI. On macOS and Windows, the CI runner system image is used after tuning it a bit.
+2. The local validation happens on the CI and checks that the artifacts produced during the build are permissible for this feedstock by consulting [the `feedstock-outputs` repository](/docs/maintainer/infrastructure/#feedstock-outputs).
+3. The server-side validation is essentially the same as 2. It is repeated inside the distribution infrastructure to guard against potential intentional or unintentional interference at the feedstock level, which is easier to access.
+4. The upload is only triggered if the local validation (2.) was successful. Additionally, it is only performed under certain conditions, for example for commits in `main`, but _not_ in PRs. Note that the package may also fail to transfer from `cf-staging` to `conda-forge` if the server-side validation (3.) fails. If 2. succeeds and 3. fails, this is usually due to an outdated token.
+
+:::
+
+## Events that trigger new builds
+
+Package building is either triggered as part of a PR or on commits to branches in the feedstock repository (which in turn usually stem from the merging of PRs).
+
+:::warning
+_Any_ commit on _any_ branch in the feedstock can lead to the building and publication of a package.
+To avoid the gratuitous publication of improper packages, development branches MUST NOT be added to the feedstock repository.
+Instead, they live in forks of the feedstock repository and the associated work is added only via pull requests.
+:::
+
+Almost all changes to the feedstock repository are performed via PRs.
+This is true both for manual maintainer intervention and conda-forge automation.
+
+:::info
+It is possible to add commits to branches directly.
+This is occasionally used to re-trigger a failed CI run with an empty commit:
+
+```console
+git commit --allow-empty -m "Retrigger CI"
+```
+
+:::
+
+### Manually submitted PRs
+
+These PRs are not automated. Any Github user can open a new PR by forking the feedstock, creating a fresh branch from `main` and adding the necessary commits to achieve the intended change (e.g. building a new version or adding a new platform).
+
+It is up to the feedstock maintainers and/or the conda-forge/core team to review, approve and merge the PR, or to reject it and close it.
+
+During the lifetime of the PR, some automated operations will take place:
+
+- The linter will scan the state of the recipe to require changes and suggest improvements. If not fulfilled, this will result in a failed CI run. Linter failures must _not_ be ignored without explicit core approval.
+- The PR template will ask you to rerender the feedstock at least once per PR. This will ensure that the CI configuration is up-to-date, among other things. You can use the bot command [`@conda-forge-admin, please rerender`](/docs/maintainer/infrastructure/#conda-forge-admin-please-rerender) in any comment, or do it locally via `conda-smithy rerender`.
+
+### Automated PRs
+
+There is a variety of situations in which the conda-forge automation will create PRs.
+In these cases, it is generally the privilege of the feedstock maintainers to review and merge the PR, triggering the action on the repository.
+
+Every commit on a branch will trigger the ci _unless_ its commit message contains the tag `[ci skip]`, which can be achieved for PRs by including it in the PR title. Some providers ignore this tag during PRs but do obey it in runs triggered by a branch like `main`.
+
+#### Version updates
+
+When a new version is released upstream, a PR needs to be created that performs the necessary updates to the feedstock.
+At a minimum, that includes updating the version, download URL, and hash of the upstream source artifact.
+In addition, other changes to the recipe may be needed, such as updated dependency requirements, a change to the `noarch` status of the package, or an adaptation of the build or test scripts.
+While these last changes often need to be done by the maintainers, conda-forge has sophisticated capabilities to add the initial version update PR.
+
+:::info How does it work?
+
+This happens in two steps in the CI of [`cf-scripts`](https://github.com/regro/cf-scripts).
+
+First, the version information is updated from upstream sources and stored in the [`cf-graph-countyfair` repo](/docs/maintainer/infrastructure/#regrocf-graph-countyfair), more specifically in the `versions` directory tree, nested by hash with one file per package.
+
+```mermaid
+sequenceDiagram
+    participant cfs as cf-scripts
+    participant gha as github actions
+    participant cft as conda_forge_tick
+    participant ups as upstream
+    participant cfg as cf-graph-countyfair
+    loop every hour at :15, :45
+        cfs->>gha: bot-versions
+        gha->>cft: update-upstream-versions
+        cft->>cfg: load package information from `graph.json`
+        loop for every package
+            cft->>ups: query version
+            ups->>cft: return version
+            cft->>cfg: write new version
+        end
+    end
+```
+
+Second, the main bot CI job, the `bot-bot` action in [`cf-scripts`](/docs/maintainer/infrastructure/#regrocf-scripts) creates PRs for all packages that have a new version available upstream.
+Here is a simplified diagram of how that is done. For the full picture, read [below](#rebuilds-for-migrators).
+
+```mermaid
+sequenceDiagram
+    participant cfs as cf-scripts
+    participant gha as github actions
+    participant cft as conda_forge_tick
+    participant fs as feedstock
+    participant cfg as cf-graph-countyfair
+    loop self renewing
+        cfs->>gha: bot-bot
+        gha->>cft: auto-tick
+        note right of cfg: the graph now contains<br/>the new version information
+        cft->>cfg: load package information from `graph.json`
+        loop for every package
+            opt if new version
+                cft->>fs: rewrite recipe and open pr
+            end
+        end
+        gha->>gha: re-trigger bot-bot
+    end
+```
+
+:::
+
+Once submitted, it is up to the feedstock maintainers to check the PR, make any necessary adjustments, and merge it into the feedstock branch.
+
+This version update is an example of a migrator. Read more about migrators in the following section.
+
+#### Rebuilds for migrators
+
+The version update is one example of a migrator.
+In reality, there are more occasions and reasons to update recipes, for example, to recompile an otherwise unchanged binary program or library to link against a newer version of a dependency or to add support for a new architecture.
+This kind of use case is handled by migrators, which are a general recipe rewriting tool.
+
+A more complete picture of what `auto-tick` does is the following:
+
+```mermaid
+sequenceDiagram
+    participant cfs as cf-scripts
+    participant gha as github actions
+    participant cft as conda_forge_tick
+    participant fs as feedstock
+    participant cfg as cf-graph-countyfair
+    loop self renewing
+        cfs->>gha: bot-bot
+        gha->>cft: auto-tick
+        note right of cfg: the graph now contains<br/>the new version information
+        cft->>cfg: load package information from `graph.json`
+        loop for every migrator
+            create participant mg as migrator
+            cft->>mg: run migrator on graph
+            note right of mg: 1. filter applicable package to produce effective graph<br/>2. determine migration order<br/>3. run migrator on every possible node
+            destroy mg
+            mg->>fs: create migration PR
+        end
+        gha->>cfs: deploy, i.e. commit changed pr information
+        gha->>gha: re-trigger bot-bot
+    end
+```
+
+Migrators are a powerful mechanism that can do almost arbitrary recipe changes.
+They are written in Python and the current set of migrators can be found in the [`regro/cf-scripts` repository](https://github.com/regro/cf-scripts/tree/master/conda_forge_tick/migrators).
@@ -0,0 +1,9 @@
+---
+title: 'Understanding conda-forge'
+---
+
+# Understanding conda-forge
+
+import DocCardList from '@theme/DocCardList';
+
+<DocCardList />
@@ -0,0 +1,126 @@
+---
+sidebar_position: 1
+---
+
+# The life cycle of a package
+
+conda-forge implements a particular workflow for building, publishing and maintaining conda packages.
+However, the core concepts are the same for any conda packaging solution.
+
+## General concepts about conda packaging
+
+`conda` packages are built off `conda` _recipes_, which consist of a metadata file (like `meta.yaml` or `recipe.yaml`), and optionally supporting scripts and data.
+A build tool (usually [`conda-build`](https://github.com/conda/conda-build) or [`rattler-build`](https://github.com/prefix-dev/rattler-build/)) takes the recipe and produces one or more packages (also referred to as _outputs_ and/or _artifacts_, depending on the context).
+
+While you can distribute the artifacts on your own, the conda packages are usually uploaded to a `conda` _channel_ hosted in a server like [Anaconda.org](https://anaconda.org/conda-forge).
+This channel _server_ processes all uploaded packages and aggregates the metadata contained in the packages in a single `repodata.json` file per platform or _subdir_.
+For example, this is a subset of the `conda-forge` repodata for Linux x64 systems: [`current_repodata.json`](https://conda.anaconda.org/conda-forge/linux-64/current_repodata.json).
+
+These are the metadata files that the `conda` client fetches when the user types `conda install ...` or similar commands.
+The solver will process all the metadata and will provide the most adequate selection of packages to the user, which are then downloaded, extracted and linked into the target conda environment.
+
+### Post-publication particularities
+
+For most packages, the paragraphs above are enough to describe their life cycle.
+However, the repodata-first approach followed in the conda ecosystem allows for some unique features in the post-publication stages.
+
+For large volume channels like conda-forge, Anaconda.org delivers the artifacts through a [CDN](/docs/glossary#cdn) network for faster access.
+The CDN network is synced with the channels periodically.
+As a result, packages take around 15 minutes to be available for installation after their publication.
+
+This repodata-first approach offers a unique opportunity to post-process the repodata files.
+This way, we can fix metadata issues without rebuilding packages.
+Note that these changes do not propagate back to the metadata contained in the packages.
+
+Anaconda.org also offers the concept of [channel _labels_][anaconda-org-labels], which in practice
+behave as _subchannels_. The default label is `main`. When a new label is added, the package is
+also visible in the subchannel `<channel>/label/<label>`. For example, a package uploaded to
+conda-forge and labeled as `test` and `main` will be available in the `conda-forge` channel, but
+also in the `conda-forge/label/test` subchannel.
+
+## The life cycle on conda-forge
+
+Anyone can run `conda-build` on their computer and manually upload their packages to Anaconda.org.
+However, that approach has a few problems:
+
+- It doesn't facilitate collaboration.
+- There's no transparency in the process.
+- Reproducibility is system-dependent.
+- Compatibility across packages is not guaranteed.
+- It doesn't scale well beyond a handful of packages.
+
+On conda-forge, most packages are built using public CI services and maintained by thousands of volunteers, which requires approaching the problem in a different way to guarantee fine-grained control of permissions, independent project management, and automated batch updates.
+
+The main idea is that each conda recipe is processed by a separate GitHub repository.
+These repositories, named _feedstocks_ in conda-forge, host the user-contributed conda recipe plus several auto-generated required scripts, configuration files, and CI pipelines to build and export the conda artifacts.
+Under this setup, the conda-forge bots can traverse the conda-forge repositories to re-generate and update feedstocks as needed when a global change or fix needs to be issued.
+
+To be given a conda-forge feedstock, contributors must first submit their recipe for review to the [`staged-recipes`][staged-recipes] repository.
+Once reviewed and approved, the PR is merged to `main`, which triggers the feedstock creation.
+
+After accepting the invitations to the `conda-forge` organization, the submitting contributor(s) will have been given commit rights to that repository.
+By then, the [feedstock creation machinery][staged-recipes] will have registered the new repository with the required CI services and populated its contents with the submitted recipe, plus the supporting scripts, configuration files and CI pipelines.
+
+These pipelines will process the initial commits to produce and upload the conda artifacts to the `cf-staging` channel.
+Any subsequent pushes to `main` (e.g. merged PRs) or other enabled branches will undergo the same process.
+
+:::info
+For existing feedstocks, conda-forge bots will often send automated PRs for new project releases or maintenance tasks.
+You can find more details about it in [Automation & bots](/docs/maintainer/infrastructure#automated-maintenance).
+:::
+
+The validation server will detect the new uploads on `cf-staging` and will perform some checks on those artifacts.
+If successful, the artifacts will then be copied to the actual `conda-forge` channel.
+
+At this point, the channel server will process the contents of the new packages to retrieve their metadata and update the repodata files.
+On the next CDN sync cycle, the artifacts will be distributed to the delivery network for faster access.
+Validation and CDN sync usually take around 15 minutes after the CI is passing on `main`.
+From this moment on, users can install the new packages from the CLI.
+
+### Post-publication particularities
+
+In an ideal world, that would be the end of the life cycle. However, in some cases, some packages go through some post-publication stages.
+
+If the package metadata is found to be wrong or out-of-date, it can be modified without rebuilding the package.
+The channel server can apply patches to the repodata files directly, via instructions published in [`conda-forge-repodata-patches`](/docs/maintainer/infrastructure#conda-forge-repodata-patches) that are processed weekly.
+
+Sometimes, there are issues with a published package that cannot be amended with a repodata patch (e.g. libraries were built wrong and segfault).
+In these cases, the packages can be retired by labeling them as `broken`.
+This is done through the [`admin-requests` repository](/docs/maintainer/infrastructure#admin-requests).
+As part of the CDN-driven metadata patching, packages labeled as `broken` are not included in the final repodata index.
+However, they are still available via direct URL access.
+This allows organizations to retire packages from normal, solver-driven installs without compromising the reproducibility offered by lock files.
+
+Finally, a project might have reached a status where no further updates are needed or expected (e.g. it has been superseded by a new project).
+If the maintainers want to, these feedstocks can be archived and marked as read-only, which is also done via [`admin-requests`](/docs/maintainer/infrastructure#admin-requests).
+
+## Summary of stages
+
+These stages are key concepts in the conda-forge documentation.
+Feel free to refer to this list at any time as you check the rest of the documentation.
+
+1. [Initial submission to `staged-recipes`](/docs/maintainer/understanding_conda_forge/staged_recipes)
+2. Feedstock changes:
+   - A. Repository initialization
+   - B. Automated maintenance updates
+   - C. PRs submitted by users
+3. Package building
+4. Package validation
+5. Package publication
+6. Post-publication:
+   - A. Repodata patch
+   - B. Mark a package as broken
+   - C. Archive the feedstock
+
+:::info
+If you want to read more about the infrastructure details behind these stages,
+consider reading our [Infrastructure guide](/docs/maintainer/infrastructure).
+:::
+
+<!-- LINKS -->
+
+[anaconda-org-labels]: https://docs.anaconda.com/anacondaorg/user-guide/tutorials/
+[staged-recipes]: /docs/maintainer/understanding_conda_forge/staged_recipes/
+[feedstock-creation]: /docs/maintainer/understanding_conda_forge/staged_recipes/#feedstock-creation
+
+<!-- [feedstocks]: /docs/reference/infrastructure/feedstocks.md -->