Sphinx Cross-References May Dangle After genomic_scores Module-to-Package Promotion

Summary

After PR #902 converted genomic_scores from a single module into a package, sphinx-apidoc now registers symbols under deeper fully-qualified names (e.g., gain.genomic_resources.genomic_scores.records.clip_span instead of gain.genomic_resources.genomic_scores.clip_span). Approximately fourteen unqualified :func: and :meth: cross-references in base.py, allele.py, and fragment.py now cross a module boundary and may no longer resolve through Sphinx's py-domain fuzzy fallback. Because nitpicky mode and -W are both absent from conf.py, broken refs silently render as plain literals rather than hyperlinks, making the regression invisible in CI.

Root-Cause Analysis

Confirmed Evidence

• genomic_scores was promoted from a flat module to a package (PR #902 / #1032).
• docs/build_docs.sh:32 runs sphinx-apidoc -o docs/source/development/gain/modules/ core/gain, which emits one .rst page per submodule of the new package.
• As a result, clip_span is indexed by the py-domain under gain.genomic_resources.genomic_scores.records.clip_span, not the prior gain.genomic_resources.genomic_scores.clip_span.
• Five fully-qualified xrefs were corrected in #1032; approximately fourteen unqualified refs in base.py, allele.py, and fragment.py were not.
• The Sphinx build has not been executed post-split, so the actual resolution state is unconfirmed.

Reasonable Inference

Sphinx's py-domain resolves an unqualified ref (e.g., ` :func:clip_span ) by prepending the current module's name and walking up the dotted hierarchy. When the symbol and the referring source file lived in the same flat module, this succeeded trivially. After the package split the symbol lives in a sibling submodule (records), which is outside the hierarchy that Sphinx's fallback traversal visits from base, allele, or fragment`. This means unqualified refs that previously resolved will likely fail silently and render as monospace literals.

Assumptions / Unknowns

• It is not confirmed which specific refs actually dangle versus which Sphinx resolves through its broader inventory search; this depends on the Sphinx version in use and the exact intersphinx / autodoc configuration.
• Some refs may still resolve if Sphinx performs a global symbol scan rather than a strictly hierarchical one; this must be verified empirically.

Why CI Does Not Catch This

• conf.py sets neither nitpick_ignore targets nor the global nitpicky = True flag.
• The docs build is not invoked with -W (warnings-as-errors).
• The Jenkins/CI when { changeset 'docs/**' } condition means the docs stage is skipped entirely unless a docs/ file is touched.

Resolution Steps

1. Build the docs locally first — do not rewrite refs on suspicion. Identify which refs actually dangle before making any changes.
2. Review the HTML output for monospace literals where hyperlinks are expected, focusing on base.py, allele.py, and fragment.py API pages.
3. Capture Sphinx warnings by running the build with -W --keep-going (see CLI Commands) to get an exhaustive list of unresolved refs without aborting on the first failure.
4. For each confirmed dangling ref, replace the unqualified form with its new fully-qualified name, e.g.:

• ` :func:clip_span → :func:gain.genomic_resources.genomic_scores.records.clip_span `
• ` :meth:GenomicScore.some_method → :meth:gain.genomic_resources.genomic_scores.base.GenomicScore.some_method `

1. Optionally, use a tilde-prefixed short display form to keep rendered text readable:

• ` :func:~gain.genomic_resources.genomic_scores.records.clip_span renders as just clip_span` but links correctly.

1. Re-run the build with -W after edits to confirm zero unresolved-reference warnings remain.
2. Enable nitpicky = True in conf.py (see Configuration Snippets) so future regressions of this class are caught at build time.
3. Add -W to build_docs.sh so warnings become build failures in CI.

CLI Commands

Build with warnings surfaced (non-destructive diagnostic):

# From the repository root
cd docs
sphinx-build -b html source build/html -W --keep-going 2>&1 | tee /tmp/sphinx-build.log

Grep the log for unresolved cross-references specifically:

grep -E "WARNING.*undefined label|WARNING.*py:.*not found|WARNING.*reference target" /tmp/sphinx-build.log

Re-run sphinx-apidoc to inspect the generated module index (informational only):

sphinx-apidoc --dry-run -o docs/source/development/gain/modules/ core/gain

Search source files for unqualified refs that cross into genomic_scores:

grep -rn ':func:`clip_span\|:func:`clip_to_region\|:meth:`GenomicScore\.' \
  core/gain/genomic_resources/genomic_scores/

Configuration Snippets

Enable nitpicky mode in docs/source/conf.py to surface broken refs as errors during the build:

# docs/source/conf.py

# Treat all unresolved cross-references as errors
nitpicky = True

# Suppress known acceptable exceptions if needed
nitpick_ignore = [
    # ("py:class", "some.third.party.Class"),
]

Add -W to docs/build_docs.sh so CI fails on any new dangling ref:

# docs/build_docs.sh  (line 32 area — add -W --keep-going)
sphinx-apidoc -o docs/source/development/gain/modules/ core/gain

sphinx-build -b html docs/source docs/build/html -W --keep-going

Example corrected docstring refs (Python source files):

# Before (unqualified — may dangle after package split)
"""
See :func:`clip_span` and :func:`clip_to_region` for boundary helpers.
"""

# After (fully qualified — always resolves)
"""
See :func:`~gain.genomic_resources.genomic_scores.records.clip_span` and
:func:`~gain.genomic_resources.genomic_scores.records.clip_to_region`
for boundary helpers.
"""

Verification

Confirm zero unresolved-reference warnings:

sphinx-build -b html docs/source docs/build/html -W --keep-going 2>&1 | \
  grep -c "WARNING"
# Expected: 0

Spot-check generated HTML for correct hyperlinks:

# Verify that clip_span renders as a link, not a literal
grep -A2 "clip_span" docs/build/html/development/gain/modules/gain.genomic_resources.genomic_scores.base.html | \
  grep "<a href"
# Expected: one or more <a href="..."> lines

Rollback indicator: If the build log still contains lines matching WARNING: py:func reference target not found: clip_span after edits, the unqualified refs have not been fully corrected. Re-examine the grep output from the diagnostic step.

Prevention

1. Enable nitpicky = True permanently in conf.py (see above). This is the single most effective guard against silent ref breakage.
2. Add -W to the Sphinx build command in build_docs.sh so any unresolved ref fails the docs CI stage immediately.
3. Relax the CI changeset filter. Consider also triggering the docs stage when core/gain/*/.py changes, since docstring edits and module restructuring affect rendered docs even without touching docs/.
4. Adopt a linting step for docstrings. Tools such as pydocstyle or doc8 can catch structural issues before a full Sphinx build.
5. Document the xref convention in a contributing guide: require fully-qualified or tilde-prefixed refs for any symbol that lives outside the current submodule, so future contributors do not introduce new unqualified cross-module refs.
6. Add a sphinx-build dry-run to pre-commit hooks (or as a fast Makefile target) so individual contributors get immediate feedback before pushing.