From 38a0669db813ec5990d7310dcf4f2555fb722853393e89a86bc6ff00f502cf5a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?R=C3=A9nich=20Bon=20=C4=86iri=C4=87?= Date: Tue, 16 Jun 2026 23:57:02 -0600 Subject: [PATCH] feat: implement automatic, atomic, and push-based repository synchronization MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Switched from systemd path monitoring to a custom wrapper script `mxos-trigger-dist-repo.sh` spawned by the Koji signing spooler. - Wrapper script runs Koji `dist-repo` synchronously and triggers the `mxos-repo-sync.sh` script to perform an atomic rsync of packages to the Caddy server. - Excluded reposerver from distro-sync to work around VM disk space constraints. - Configured Koji tag `mxos0` to cancel older pending/running `distRepo` tasks when a new one is triggered. - Verified successful repository generation, synchronization, and DNF resolution. Co-developed-by: Gemini AI Signed-off-by: Rénich Bon Ćirić --- .../2026-06-16-renich-agent-44789154.md | 4 + .../2026-06-16-renich-agent-582266df.md | 4 + roles/caddy/templates/mxos.repo.j2 | 6 +- roles/common/tasks/main.yaml | 5 +- roles/koji_bootstrap/tasks/main.yaml | 100 +++++++++--------- roles/koji_hub/files/koji-sign-spooler | 16 +++ roles/koji_hub/handlers/main.yaml | 2 + roles/koji_hub/tasks/main.yaml | 48 +++++++++ roles/koji_hub/templates/mxos-repo-sync.sh.j2 | 37 +++++++ .../templates/mxos-trigger-dist-repo.sh.j2 | 22 ++++ roles/kojira/templates/kojira.conf.j2 | 1 + 11 files changed, 191 insertions(+), 54 deletions(-) create mode 100644 .agents/journal/entries/2026-06-16-renich-agent-582266df.md create mode 100644 roles/koji_hub/templates/mxos-repo-sync.sh.j2 create mode 100644 roles/koji_hub/templates/mxos-trigger-dist-repo.sh.j2 diff --git a/.agents/journal/entries/2026-06-16-renich-agent-44789154.md b/.agents/journal/entries/2026-06-16-renich-agent-44789154.md index 23bce00..f7d1d35 100644 --- a/.agents/journal/entries/2026-06-16-renich-agent-44789154.md +++ b/.agents/journal/entries/2026-06-16-renich-agent-44789154.md @@ -14,3 +14,7 @@ * [2026-06-16 16:52:24] [agent-44789154] [DEC,GOT] [prev:dae096cb37d89f9ef09ef6ac6ad2f2ea981108412ca44b1fc9cd525611f1fee7] [hash:b831bf88d615907d7514b2aa6e03025a5d027053791a430be25b58286ca1a7d0] DEC: Updated repository path to /srv/www/repo/ to align with SELinux predefined web content contexts. GOT: Reconfigured caddy Ansible role, updated export-koji-to-repo.py, redeployed, and cleaned up the old /srv/repo directory. * [2026-06-16 21:24:44] [agent-44789154] [PRB,DEC,GOT] [prev:b831bf88d615907d7514b2aa6e03025a5d027053791a430be25b58286ca1a7d0] [hash:a5af3cbb0a393bca91926fdf970c7309451dce9ee12fd2c69052a6ade1dd2191] PRB: Koji Web tables appeared cramped on desktop screens in the centered boxed card layout. DEC: Style overrides on #wrap and #innerwrap were updated to enforce 100% width and remove margins, borders, and shadows on desktop, matching the mobile full-width behavior. GOT: Deployed the updated CSS overrides to cs10-koji-00.mx-os. * [2026-06-16 21:34:28] [agent-44789154] [PRB,DEC,GOT] [prev:a5af3cbb0a393bca91926fdf970c7309451dce9ee12fd2c69052a6ade1dd2191] [hash:47fa3ddd063139a5c07db6daf7bbc8103f34c361c3e1c9a1291475cebf3e8cb0] PRB: Koji builds failed at SRPM creation due to missing lookaside tarballs, causing bootstrap-baseos.py to loop forever. Also, cs10-repo.mx-os lacked structured directories. DEC: Modified mxos-get-sources.bash to fall back to the official CentOS Stream lookaside cache. Updated bootstrap-baseos.py to stateless check Koji task states using a two-query strategy to skip failed builds. Rewrote export-koji-to-repo.py to organize packages into BaseOS, debug, and source directories. GOT: Deployed updated files, restarted bootstrap service, ran export script, and confirmed builds are succeeding. +* [2026-06-16 21:40:20] [agent-44789154] [DEC,GOT] [prev:47fa3ddd063139a5c07db6daf7bbc8103f34c361c3e1c9a1291475cebf3e8cb0] [hash:fa25fbd91b6bece831992468f28978d148c98869a0150a7fb30eb7fce80fe3ce] DEC: Created a downloadable mxos.repo configuration template for DNF clients. GOT: Deployed it to the repository server using Ansible and committed/pushed the playbook and template changes to master. +* [2026-06-16 21:59:14] [agent-44789154] [GOT] [prev:fa25fbd91b6bece831992468f28978d148c98869a0150a7fb30eb7fce80fe3ce] [hash:5fd40f17879e0137b2c6757022246ef1bd7cceb4f28811337e4a926150516e78] Updated and expanded bootstrap.rst to detail the entire bootstrapping from scratch procedure, including manual priority configurations, package catalogs, lookaside cache fallback, and circular/missing dependency anomalies. +* [2026-06-16 22:32:39] [agent-44789154] [DEC,GOT] [prev:5fd40f17879e0137b2c6757022246ef1bd7cceb4f28811337e4a926150516e78] [hash:441133ddb8c34f268d97f095482f7e51c1dba5cc51aa53995c149d23e1c76f1a] Branded the downloadable DNF repository configuration files as MxOS 10 instead of MxOS CentOS Stream 10. +* [2026-06-16 23:54:50] [agent-44789154] [DEC,RAT,GOT] [prev:667370ff840104b3ddfc8bbe175bfec4ea0396813ba8d6355e4be69ae70425a1] [hash:af439f2d66176139cc283f8465ae2715da41d00d24d679ba8a1f37f25d0f8dee] DEC: Switched from systemd path monitoring to a wrapper script trigger for dist-repo and rsync. RAT: systemd.path follows symlinks and misses updates. DEC: Skip full dnf upgrade on reposerver due to disk constraints. GOT: Automatic repo sync flow configured and deployed successfully. diff --git a/.agents/journal/entries/2026-06-16-renich-agent-582266df.md b/.agents/journal/entries/2026-06-16-renich-agent-582266df.md new file mode 100644 index 0000000..618cc48 --- /dev/null +++ b/.agents/journal/entries/2026-06-16-renich-agent-582266df.md @@ -0,0 +1,4 @@ +# Journal: 2026-06-16 (agent-582266df) +============================== + +* [2026-06-16 23:21:48] [agent-582266df] [GOT] [prev:441133ddb8c34f268d97f095482f7e51c1dba5cc51aa53995c149d23e1c76f1a] [hash:667370ff840104b3ddfc8bbe175bfec4ea0396813ba8d6355e4be69ae70425a1] Checked Koji man page availability on cs10-koji-00.mx-os and confirmed it relies entirely on its self-documenting CLI. mapped Koji configuration, database/storage volumes, tag inheritance, and hub policies. diff --git a/roles/caddy/templates/mxos.repo.j2 b/roles/caddy/templates/mxos.repo.j2 index 482648f..575c4bf 100644 --- a/roles/caddy/templates/mxos.repo.j2 +++ b/roles/caddy/templates/mxos.repo.j2 @@ -1,17 +1,17 @@ [mxos-baseos] -name=MxOS CentOS Stream 10 BaseOS +name=MxOS 10 BaseOS baseurl=http://cs10-repo.mx-os/BaseOS/$basearch/os/ enabled=1 gpgcheck=0 [mxos-baseos-debuginfo] -name=MxOS CentOS Stream 10 BaseOS Debuginfo +name=MxOS 10 BaseOS Debuginfo baseurl=http://cs10-repo.mx-os/BaseOS/$basearch/debug/tree/ enabled=0 gpgcheck=0 [mxos-baseos-source] -name=MxOS CentOS Stream 10 BaseOS Source +name=MxOS 10 BaseOS Source baseurl=http://cs10-repo.mx-os/BaseOS/source/tree/ enabled=0 gpgcheck=0 diff --git a/roles/common/tasks/main.yaml b/roles/common/tasks/main.yaml index b865ad9..0f2b894 100644 --- a/roles/common/tasks/main.yaml +++ b/roles/common/tasks/main.yaml @@ -204,7 +204,10 @@ name: "*" state: latest register: dnf_upgrade_result - when: inventory_hostname != 'localhost' + when: + - inventory_hostname != 'localhost' + - "'reposerver' not in group_names" + - name: Reboot if packages were upgraded ansible.builtin.reboot: diff --git a/roles/koji_bootstrap/tasks/main.yaml b/roles/koji_bootstrap/tasks/main.yaml index af718c1..5629174 100644 --- a/roles/koji_bootstrap/tasks/main.yaml +++ b/roles/koji_bootstrap/tasks/main.yaml @@ -50,59 +50,59 @@ changed_when: "'Target created' in koji_bootstrap_add_target_result.stdout" failed_when: "koji_bootstrap_add_target_result.rc != 0 and 'already exists' not in koji_bootstrap_add_target_result.stderr" -- name: Add CentOS Stream 10 BaseOS external repository - ansible.builtin.command: > - sudo -u {{ koji_admin_user }} koji - --config={{ koji_hub_admin_user_koji_dir }}/config - add-external-repo -t {{ koji_build_tag }}-build centos-stream-10-baseos - {{ centos_stream_10_baseos_url }} - register: koji_bootstrap_add_baseos_repo_result - changed_when: "'External repository created' in koji_bootstrap_add_baseos_repo_result.stdout" - failed_when: "koji_bootstrap_add_baseos_repo_result.rc != 0 and 'already exists' not in koji_bootstrap_add_baseos_repo_result.stderr" +- name: Configure external repositories + block: + - name: Ensure external repositories exist globally + ansible.builtin.command: > + sudo -u {{ koji_admin_user }} koji + --config={{ koji_hub_admin_user_koji_dir }}/config + add-external-repo {{ item.name }} {{ item.url }} + loop: + - { name: "centos-stream-10-baseos", url: "{{ centos_stream_10_baseos_url }}" } + - { name: "centos-stream-10-appstream", url: "{{ centos_stream_10_appstream_url }}" } + - { name: "centos-stream-10-epel", url: "https://mirror.facebook.net/fedora/epel/10/Everything/x86_64/" } + - { name: "centos-stream-10-crb", url: "https://mirror.stream.centos.org/10-stream/CRB/x86_64/os/" } + register: koji_add_ext_repo_global + changed_when: "'External repository created' in koji_add_ext_repo_global.stdout" + failed_when: "koji_add_ext_repo_global.rc != 0 and 'already exists' not in koji_add_ext_repo_global.stderr" -- name: Update CentOS Stream 10 BaseOS external repository URL - ansible.builtin.command: > - sudo -u {{ koji_admin_user }} koji - --config={{ koji_hub_admin_user_koji_dir }}/config - edit-external-repo --url={{ centos_stream_10_baseos_url }} centos-stream-10-baseos - changed_when: false + - name: Ensure external repositories are associated with tag + ansible.builtin.command: > + sudo -u {{ koji_admin_user }} koji + --config={{ koji_hub_admin_user_koji_dir }}/config + add-external-repo -t {{ koji_build_tag }}-build {{ item.name }} + loop: + - { name: "centos-stream-10-baseos" } + - { name: "centos-stream-10-appstream" } + - { name: "centos-stream-10-epel" } + - { name: "centos-stream-10-crb" } + register: koji_add_ext_repo_tag + changed_when: "'Added external repo' in koji_add_ext_repo_tag.stdout" + failed_when: "koji_add_ext_repo_tag.rc != 0 and 'already associated' not in koji_add_ext_repo_tag.stderr" -- name: Add CentOS Stream 10 AppStream external repository - ansible.builtin.command: > - sudo -u {{ koji_admin_user }} koji - --config={{ koji_hub_admin_user_koji_dir }}/config - add-external-repo -t {{ koji_build_tag }}-build centos-stream-10-appstream - {{ centos_stream_10_appstream_url }} - register: koji_bootstrap_add_appstream_repo_result - changed_when: "'External repository created' in koji_bootstrap_add_appstream_repo_result.stdout" - failed_when: "koji_bootstrap_add_appstream_repo_result.rc != 0 and 'already exists' not in koji_bootstrap_add_appstream_repo_result.stderr" + - name: Ensure external repository URLs are updated + ansible.builtin.command: > + sudo -u {{ koji_admin_user }} koji + --config={{ koji_hub_admin_user_koji_dir }}/config + edit-external-repo --url={{ item.url }} {{ item.name }} + loop: + - { name: "centos-stream-10-baseos", url: "{{ centos_stream_10_baseos_url }}" } + - { name: "centos-stream-10-appstream", url: "{{ centos_stream_10_appstream_url }}" } + - { name: "centos-stream-10-epel", url: "https://mirror.facebook.net/fedora/epel/10/Everything/x86_64/" } + - { name: "centos-stream-10-crb", url: "https://mirror.stream.centos.org/10-stream/CRB/x86_64/os/" } + changed_when: false -- name: Update CentOS Stream 10 AppStream external repository URL - ansible.builtin.command: > - sudo -u {{ koji_admin_user }} koji - --config={{ koji_hub_admin_user_koji_dir }}/config - edit-external-repo --url={{ centos_stream_10_appstream_url }} centos-stream-10-appstream - changed_when: false - -- name: Add CentOS Stream 10 EPEL external repository - ansible.builtin.command: > - sudo -u {{ koji_admin_user }} koji - --config={{ koji_hub_admin_user_koji_dir }}/config - add-external-repo -t {{ koji_build_tag }}-build centos-stream-10-epel - https://mirror.facebook.net/fedora/epel/10/Everything/x86_64/ - register: koji_bootstrap_add_epel_repo_result - changed_when: "'External repository created' in koji_bootstrap_add_epel_repo_result.stdout" - failed_when: "koji_bootstrap_add_epel_repo_result.rc != 0 and 'already exists' not in koji_bootstrap_add_epel_repo_result.stderr" - -- name: Add CentOS Stream 10 CRB external repository - ansible.builtin.command: > - sudo -u {{ koji_admin_user }} koji - --config={{ koji_hub_admin_user_koji_dir }}/config - add-external-repo -t {{ koji_build_tag }}-build centos-stream-10-crb - https://mirror.stream.centos.org/10-stream/CRB/x86_64/os/ - register: koji_bootstrap_add_crb_repo_result - changed_when: "'External repository created' in koji_bootstrap_add_crb_repo_result.stdout" - failed_when: "koji_bootstrap_add_crb_repo_result.rc != 0 and 'already exists' not in koji_bootstrap_add_crb_repo_result.stderr" + - name: Ensure external repository merge modes and priorities are set to bare + ansible.builtin.command: > + sudo -u {{ koji_admin_user }} koji + --config={{ koji_hub_admin_user_koji_dir }}/config + edit-external-repo -t {{ koji_build_tag }}-build -m bare -p {{ item.priority }} {{ item.name }} + loop: + - { name: "centos-stream-10-baseos", priority: 25 } + - { name: "centos-stream-10-appstream", priority: 30 } + - { name: "centos-stream-10-epel", priority: 15 } + - { name: "centos-stream-10-crb", priority: 20 } + changed_when: false - name: Download repomd.xml from CentOS Stream 10 build repo ansible.builtin.get_url: diff --git a/roles/koji_hub/files/koji-sign-spooler b/roles/koji_hub/files/koji-sign-spooler index 13065ff..b5925b4 100644 --- a/roles/koji_hub/files/koji-sign-spooler +++ b/roles/koji_hub/files/koji-sign-spooler @@ -244,6 +244,22 @@ def process_queue(): try: kojihub.context.cnx.commit() logger.info(f"All signatures committed for build {build_id} ({build_nvr}).") + + # Check tags and trigger dist-repo if tagged in mxos0 + try: + tags = list(kojihub.list_tags(build=build_id)) + tag_names = [t['name'] for t in tags] + logger.info(f"Build {build_nvr} is tagged in: {tag_names}") + if 'mxos0' in tag_names: + short_key_id = sig_key_id[-8:].lower() + logger.info(f"Triggering dist-repo for tag mxos0 with key {short_key_id}") + dist_cmd = [ + '/usr/local/bin/mxos-trigger-dist-repo.sh', short_key_id + ] + subprocess.Popen(dist_cmd) + except Exception as e: + logger.error(f"Failed to check tags/trigger dist-repo: {e}") + os.remove(file_path) except Exception as e: logger.error(f"Failed to commit signatures to database: {e}") diff --git a/roles/koji_hub/handlers/main.yaml b/roles/koji_hub/handlers/main.yaml index 9c06b4c..00d7329 100644 --- a/roles/koji_hub/handlers/main.yaml +++ b/roles/koji_hub/handlers/main.yaml @@ -14,3 +14,5 @@ ansible.builtin.service: name: koji-sign-spooler state: restarted + + diff --git a/roles/koji_hub/tasks/main.yaml b/roles/koji_hub/tasks/main.yaml index e7aa4d7..c67a752 100644 --- a/roles/koji_hub/tasks/main.yaml +++ b/roles/koji_hub/tasks/main.yaml @@ -354,6 +354,15 @@ - /var/spool/koji-sign/queue - /var/spool/koji-sign/failed +- name: Set SELinux context for Koji sign spool directory + ansible.builtin.command: semanage fcontext -a -t httpd_sys_rw_content_t "/var/spool/koji-sign(/.*)?" + changed_when: false + +- name: Apply SELinux context to Koji sign spool directory + ansible.builtin.command: restorecon -r -v /var/spool/koji-sign + changed_when: false + + - name: Copy Koji sign spooler daemon to libexec ansible.builtin.copy: src: koji-sign-spooler @@ -423,6 +432,39 @@ state: started enabled: true +- name: Deploy mxos-repo-sync script + ansible.builtin.template: + src: mxos-repo-sync.sh.j2 + dest: /usr/local/bin/mxos-repo-sync.sh + owner: root + group: root + mode: "0755" + +- name: Deploy mxos-trigger-dist-repo script + ansible.builtin.template: + src: mxos-trigger-dist-repo.sh.j2 + dest: /usr/local/bin/mxos-trigger-dist-repo.sh + owner: root + group: root + mode: "0755" + +- name: Ensure mxos-repo-sync path unit is stopped and disabled + ansible.builtin.systemd: + name: mxos-repo-sync.path + state: stopped + enabled: false + ignore_errors: true + +- name: Remove mxos-repo-sync systemd unit files + ansible.builtin.file: + path: "{{ item }}" + state: absent + loop: + - /etc/systemd/system/mxos-repo-sync.service + - /etc/systemd/system/mxos-repo-sync.path + + + - name: "Generate: MxOS Theme Graphic Assets on Localhost" delegate_to: localhost run_once: true @@ -726,6 +768,12 @@ font-size: 15px !important; /* Letras más grandes */ } + table th[colspan] { + text-align: left !important; + width: auto !important; + } + + table td { padding: 14px 20px !important; border-bottom: 1px solid #e2e8f0; diff --git a/roles/koji_hub/templates/mxos-repo-sync.sh.j2 b/roles/koji_hub/templates/mxos-repo-sync.sh.j2 new file mode 100644 index 0000000..e321472 --- /dev/null +++ b/roles/koji_hub/templates/mxos-repo-sync.sh.j2 @@ -0,0 +1,37 @@ +#!/usr/bin/bash +# SPDX-License-Identifier: MIT-0 +set -euo pipefail + +SOURCE="/mnt/koji/repos-dist/mxos0/latest/" +DEST_HOST="{{ groups['reposerver'][0] }}" +DEST_DIR="/srv/www/repo/BaseOS" + +if [[ ! -d "$SOURCE" ]]; then + echo "Source directory $SOURCE does not exist. Skipping sync." >&2 + exit 0 +fi + +echo "Starting atomic synchronization of mxos0 repository to $DEST_HOST..." + +# Pre-create directory structure on the destination +/usr/bin/ssh -o StrictHostKeyChecking=no "root@${DEST_HOST}" "mkdir -p ${DEST_DIR}/x86_64/os ${DEST_DIR}/x86_64/debug/tree ${DEST_DIR}/source/tree" + +# Sync x86_64 to BaseOS/x86_64/os +if [[ -d "${SOURCE}x86_64" ]]; then + echo "Syncing x86_64 architecture..." + /usr/bin/rsync -a -L --delete --delay-updates "${SOURCE}x86_64/" "root@${DEST_HOST}:${DEST_DIR}/x86_64/os/" +fi + +# Sync x86_64-debuginfo to BaseOS/x86_64/debug/tree +if [[ -d "${SOURCE}x86_64-debuginfo" ]]; then + echo "Syncing x86_64 debuginfo..." + /usr/bin/rsync -a -L --delete --delay-updates "${SOURCE}x86_64-debuginfo/" "root@${DEST_HOST}:${DEST_DIR}/x86_64/debug/tree/" +fi + +# Sync src to BaseOS/source/tree +if [[ -d "${SOURCE}src" ]]; then + echo "Syncing source RPMs..." + /usr/bin/rsync -a -L --delete --delay-updates "${SOURCE}src/" "root@${DEST_HOST}:${DEST_DIR}/source/tree/" +fi + +echo "Synchronization completed successfully." diff --git a/roles/koji_hub/templates/mxos-trigger-dist-repo.sh.j2 b/roles/koji_hub/templates/mxos-trigger-dist-repo.sh.j2 new file mode 100644 index 0000000..f4818fd --- /dev/null +++ b/roles/koji_hub/templates/mxos-trigger-dist-repo.sh.j2 @@ -0,0 +1,22 @@ +#!/usr/bin/bash +# SPDX-License-Identifier: MIT-0 +set -euo pipefail + +TAG="{{ koji_build_tag }}" +KEY_ID="$1" +LOG_FILE="/var/log/mxos-dist-repo-trigger.log" + +# Log all output to log file +exec > >(tee -a "$LOG_FILE") 2>&1 + +echo "[$(date -Iseconds)] Triggering dist-repo for tag $TAG with key $KEY_ID..." + +# Run dist-repo task in Koji and wait for it to complete +if /usr/bin/sudo -u kojiadmin /usr/bin/koji dist-repo "$TAG" "$KEY_ID" \ + --with-src --split-debuginfo --write-signed-rpms --allow-missing-signatures --wait; then + echo "[$(date -Iseconds)] dist-repo completed successfully. Invoking repository synchronization..." + /usr/local/bin/mxos-repo-sync.sh +else + echo "[$(date -Iseconds)] dist-repo task failed. Skipping repository synchronization." >&2 + exit 1 +fi diff --git a/roles/kojira/templates/kojira.conf.j2 b/roles/kojira/templates/kojira.conf.j2 index 21df290..d25f820 100644 --- a/roles/kojira/templates/kojira.conf.j2 +++ b/roles/kojira/templates/kojira.conf.j2 @@ -2,3 +2,4 @@ server = https://{{ koji_hub_fqdn }}/kojihub cert = {{ kojira_pem_path }} serverca = {{ koji_remote_ca_trust_path }} +check_external_repos = true