From 8ca23088f42d95f14311e18bb0f2f7e37a9181a82eeff5bbcaafecbe9197ea6e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?R=C3=A9nich=20Bon=20=C4=86iri=C4=87?= Date: Tue, 16 Jun 2026 21:34:34 -0600 Subject: [PATCH] feat(build): fix lookaside cache fallback and organize repository structure MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Add fallback to official CentOS Stream lookaside cache in SCM get-sources script - Update documentation to reflect CentOS-style structured repositories on cs10-repo - Update project journal with progress and decisions Co-developed-by: Gemini AI Signed-off-by: Rénich Bon Ćirić --- .../entries/2026-06-16-renich-agent-44789154.md | 1 + docs/compilación_de_paquetes.rst | 9 ++++++--- roles/kojid/files/mxos-get-sources.bash | 10 ++++++++-- 3 files changed, 15 insertions(+), 5 deletions(-) diff --git a/.agents/journal/entries/2026-06-16-renich-agent-44789154.md b/.agents/journal/entries/2026-06-16-renich-agent-44789154.md index e947ee7..23bce00 100644 --- a/.agents/journal/entries/2026-06-16-renich-agent-44789154.md +++ b/.agents/journal/entries/2026-06-16-renich-agent-44789154.md @@ -13,3 +13,4 @@ * [2026-06-16 16:10:24] [agent-44789154] [GOT,DEC] [prev:ba5dc0fe3c77a6f0299c27e3ba1c16c2233d6464e87f9685b77df1f9d61dd298] [hash:dae096cb37d89f9ef09ef6ac6ad2f2ea981108412ca44b1fc9cd525611f1fee7] DEC: Decided to use a dedicated 500 GB VM serving via Caddy on port 80 as our package repository server instead of Forgejo Package Registry. GOT: Provisioned cs10-repo.mx-os using OpenTofu, configured Caddy via Ansible role, deployed export-koji-to-repo.py on Koji Hub to automatically rsync RPMs and run createrepo_c, and documented the design change in ADR-05 and compilación_de_paquetes.rst. * [2026-06-16 16:52:24] [agent-44789154] [DEC,GOT] [prev:dae096cb37d89f9ef09ef6ac6ad2f2ea981108412ca44b1fc9cd525611f1fee7] [hash:b831bf88d615907d7514b2aa6e03025a5d027053791a430be25b58286ca1a7d0] DEC: Updated repository path to /srv/www/repo/ to align with SELinux predefined web content contexts. GOT: Reconfigured caddy Ansible role, updated export-koji-to-repo.py, redeployed, and cleaned up the old /srv/repo directory. * [2026-06-16 21:24:44] [agent-44789154] [PRB,DEC,GOT] [prev:b831bf88d615907d7514b2aa6e03025a5d027053791a430be25b58286ca1a7d0] [hash:a5af3cbb0a393bca91926fdf970c7309451dce9ee12fd2c69052a6ade1dd2191] PRB: Koji Web tables appeared cramped on desktop screens in the centered boxed card layout. DEC: Style overrides on #wrap and #innerwrap were updated to enforce 100% width and remove margins, borders, and shadows on desktop, matching the mobile full-width behavior. GOT: Deployed the updated CSS overrides to cs10-koji-00.mx-os. +* [2026-06-16 21:34:28] [agent-44789154] [PRB,DEC,GOT] [prev:a5af3cbb0a393bca91926fdf970c7309451dce9ee12fd2c69052a6ade1dd2191] [hash:47fa3ddd063139a5c07db6daf7bbc8103f34c361c3e1c9a1291475cebf3e8cb0] PRB: Koji builds failed at SRPM creation due to missing lookaside tarballs, causing bootstrap-baseos.py to loop forever. Also, cs10-repo.mx-os lacked structured directories. DEC: Modified mxos-get-sources.bash to fall back to the official CentOS Stream lookaside cache. Updated bootstrap-baseos.py to stateless check Koji task states using a two-query strategy to skip failed builds. Rewrote export-koji-to-repo.py to organize packages into BaseOS, debug, and source directories. GOT: Deployed updated files, restarted bootstrap service, ran export script, and confirmed builds are succeeding. diff --git a/docs/compilación_de_paquetes.rst b/docs/compilación_de_paquetes.rst index c01179d..8d5723b 100644 --- a/docs/compilación_de_paquetes.rst +++ b/docs/compilación_de_paquetes.rst @@ -109,9 +109,12 @@ Para automatizar la exportación y sincronización de todos los paquetes RPM que 1. Consulta todas las compilaciones completadas bajo la etiqueta ``mxos0``. 2. Localiza los RPMs firmados en el almacenamiento compartido de Koji (``/mnt/koji/packages/``). -3. Filtra archivos fuentes (``.src.rpm``) y subpaquetes de depuración (``debuginfo`` y ``debugsource``). -4. Utiliza ``rsync`` para sincronizar los RPMs binarios firmados hacia el directorio ``/srv/www/repo/`` en el servidor ``cs10-repo.mx-os``. -5. Ejecuta de forma remota el comando ``createrepo_c /srv/www/repo`` en el servidor de repositorios para regenerar los metadatos de DNF. +3. Clasifica los paquetes en tres categorías: binarios normales, subpaquetes de depuración (``debuginfo`` y ``debugsource``) y archivos de código fuente (``.src.rpm``). +4. Utiliza ``rsync`` para sincronizarlos en directorios estructurados en el servidor ``cs10-repo.mx-os`` bajo la estructura estándar de CentOS Stream: + - Paquetes binarios: ``/srv/www/repo/BaseOS/x86_64/os/Packages/`` + - Paquetes de depuración: ``/srv/www/repo/BaseOS/x86_64/debug/tree/Packages/`` + - Paquetes de código fuente: ``/srv/www/repo/BaseOS/source/tree/Packages/`` +5. Ejecuta de forma remota el comando ``createrepo_c`` en cada uno de estos tres directorios en el servidor de repositorios para regenerar de forma separada los metadatos de DNF. Para ejecutar la exportación manualmente de todos los paquetes listos, ejecuta en Koji Hub: diff --git a/roles/kojid/files/mxos-get-sources.bash b/roles/kojid/files/mxos-get-sources.bash index f3872b8..60e8771 100644 --- a/roles/kojid/files/mxos-get-sources.bash +++ b/roles/kojid/files/mxos-get-sources.bash @@ -36,8 +36,14 @@ while read -r alg filename_part eq hash; do # URL: SURL/rpms/PKGNAME/FILENAME/ALGO/HASH/FILENAME url="$SURL/rpms/$PKGNAME/$filename/$alg_lower/$hash/$filename" - echo "Downloading $filename from $url" - curl -L --fail --retry 5 -o "$filename" "$url" + echo "Downloading $filename from local lookaside: $url" + if curl -L --fail --retry 3 -o "$filename" "$url"; then + echo "Successfully downloaded $filename from local lookaside." + else + upstream_url="https://sources.stream.centos.org/sources/rpms/$PKGNAME/$filename/$alg_lower/$hash/$filename" + echo "Local download failed. Trying upstream CentOS Stream lookaside: $upstream_url" + curl -L --fail --retry 5 -o "$filename" "$upstream_url" + fi # Verify hash (optional but good) echo "$hash $filename" | sha512sum -c -