Debugging a failing build

Guidestable
On this page

Debugging a failing Copr build

When a build fails, the tool is built to take you from the failure to a reproduced, fixed build without reading whole logs.

Monitor the state matrix

coprctl monitor OWNER/PROJECT shows every package's chroot states in one table. STATE is the human state; the raw chroot status integer is JSON-only output. BUILD is the latest build id, VERSION its package version, and LOG the live build log URL, elided so the table stays narrow; --output json carries the full URLs. A :dir suffix on the reference monitors a side repo instead of the project's main directory.

The log URLs point straight at the backend results directory. They appear in JSON output only once a chroot has a result dir, so agents must not assume the url_build_log and url_backend_log keys are present.

1. Find why it failed

Do not ingest a whole build log. Extract the failing region from each failed chroot:

coprctl log failures BUILD_ID

For a second opinion in plain language, ask the log-detective service:

coprctl log detective BUILD_ID/CHROOT

If the service does not know the build, it falls back to local analysis.

2. Reproduce locally

The exact reproduction recipe Copr wrote into the build log:

coprctl build reproduce BUILD_ID/CHROOT

This prints the copr-rpmbuild --task-url ... invocation for mock-level fidelity. When the log carries no recipe, it falls back to the stored source build config and prints the reconstructed submit instead, so a build without the recipe line is still reproducible.

coprctl try runs a local preflight build, choosing a backend by intent:

coprctl try ./rpm --chroot fedora-rawhide-x86_64        # container (default)
coprctl try ./rpm --chroot fedora-rawhide-x86_64 --runtime native  # host rpmbuild
coprctl try ./rpm --chroot fedora-rawhide-x86_64 --runtime mock    # clean-room

The default --runtime auto prefers the rpmbuilder container, then mock for a clean-room buildroot, then native host tools. Native is not a clean-room buildroot: it warns that mock is the higher-fidelity fallback. Mock needs the mock package and the mock group; when it is missing or misconfigured, the tool says how to set it up. build srpm and build submit --from accept the same --runtime values.

For a container preflight, try resolves the Copr chroot to an rpmbuilder image, runs the source-build then chroot-build stages, and reports coverage and a fidelity report.

3. Test a fix before pushing

  • For tito projects, commit the spec change first (tito ignores uncommitted changes), then run coprctl try. Squash WIP commits before tagging.
  • Declare all dependencies in BuildRequires/Requires; do not install them manually in the container.
  • Iterate until the local build is clean, then rebuild in Copr:
coprctl build rebuild OWNER/PROJECT/PKG --preflight

To retry only the chroots that failed in an earlier build, pass the build id instead of naming chroots yourself:

coprctl build rebuild OWNER/PROJECT/PKG --only-failed BUILD_ID

--only-failed fetches the build, collects the failed chroots, and rebuilds just those. A build with no failed chroots reports nothing to do and exits 0.

Building and submitting a source RPM

To produce a source RPM from a local spec, use a container by default, or host tools:

coprctl build srpm ./rpm --chroot fedora-rawhide-x86_64          # container
coprctl build srpm ./rpm --runtime native                        # spectool + rpmbuild
coprctl build srpm ./rpm --runtime mock                          # clean-room SRPM

This runs the same SRPM_ONLY stage as try inside an rpmbuilder image and writes the .src.rpm into the spec directory.

To build the source RPM and submit it in one step, build submit chains the two:

coprctl build submit OWNER/PROJECT --from ./rpm

--from builds the SRPM locally via the container, then uploads it to Copr and queues the build. It needs a container runtime and write access to the project. Add --watch to wait for the submitted build to reach a terminal state instead of returning after the queue:

coprctl build submit OWNER/PROJECT --from ./rpm --watch

Rules

  1. Start with coprctl log failures BUILD_ID, never a full log dump.
  2. Reproduce locally before queueing another Copr build.
  3. A local pass is a filter, not a proof: read the try fidelity report.
  4. Commit before try on a tito project; squash WIP before push.

Agent skill

The coprctl-debug skill encodes this workflow and is installed with coprctl skill install coprctl-debug.