feat: implement automatic, atomic, and push-based repository synchronization

- Switched from systemd path monitoring to a custom wrapper script `mxos-trigger-dist-repo.sh` spawned by the Koji signing spooler.
- Wrapper script runs Koji `dist-repo` synchronously and triggers the `mxos-repo-sync.sh` script to perform an atomic rsync of packages to the Caddy server.
- Excluded reposerver from distro-sync to work around VM disk space constraints.
- Configured Koji tag `mxos0` to cancel older pending/running `distRepo` tasks when a new one is triggered.
- Verified successful repository generation, synchronization, and DNF resolution.

Co-developed-by: Gemini AI <renich+gemini@woralelandia.com>
Signed-off-by: Rénich Bon Ćirić <renich@woralelandia.com>

.agents/journal/entries/2026-06-16-renich-agent-44789154.md

@@ -14,3 +14,7 @@
 * [2026-06-16 16:52:24] [agent-44789154] [DEC,GOT] [prev:dae096cb37d89f9ef09ef6ac6ad2f2ea981108412ca44b1fc9cd525611f1fee7] [hash:b831bf88d615907d7514b2aa6e03025a5d027053791a430be25b58286ca1a7d0] DEC: Updated repository path to /srv/www/repo/ to align with SELinux predefined web content contexts. GOT: Reconfigured caddy Ansible role, updated export-koji-to-repo.py, redeployed, and cleaned up the old /srv/repo directory.
 * [2026-06-16 21:24:44] [agent-44789154] [PRB,DEC,GOT] [prev:b831bf88d615907d7514b2aa6e03025a5d027053791a430be25b58286ca1a7d0] [hash:a5af3cbb0a393bca91926fdf970c7309451dce9ee12fd2c69052a6ade1dd2191] PRB: Koji Web tables appeared cramped on desktop screens in the centered boxed card layout. DEC: Style overrides on #wrap and #innerwrap were updated to enforce 100% width and remove margins, borders, and shadows on desktop, matching the mobile full-width behavior. GOT: Deployed the updated CSS overrides to cs10-koji-00.mx-os.
 * [2026-06-16 21:34:28] [agent-44789154] [PRB,DEC,GOT] [prev:a5af3cbb0a393bca91926fdf970c7309451dce9ee12fd2c69052a6ade1dd2191] [hash:47fa3ddd063139a5c07db6daf7bbc8103f34c361c3e1c9a1291475cebf3e8cb0] PRB: Koji builds failed at SRPM creation due to missing lookaside tarballs, causing bootstrap-baseos.py to loop forever. Also, cs10-repo.mx-os lacked structured directories. DEC: Modified mxos-get-sources.bash to fall back to the official CentOS Stream lookaside cache. Updated bootstrap-baseos.py to stateless check Koji task states using a two-query strategy to skip failed builds. Rewrote export-koji-to-repo.py to organize packages into BaseOS, debug, and source directories. GOT: Deployed updated files, restarted bootstrap service, ran export script, and confirmed builds are succeeding.
+* [2026-06-16 21:40:20] [agent-44789154] [DEC,GOT] [prev:47fa3ddd063139a5c07db6daf7bbc8103f34c361c3e1c9a1291475cebf3e8cb0] [hash:fa25fbd91b6bece831992468f28978d148c98869a0150a7fb30eb7fce80fe3ce] DEC: Created a downloadable mxos.repo configuration template for DNF clients. GOT: Deployed it to the repository server using Ansible and committed/pushed the playbook and template changes to master.
+* [2026-06-16 21:59:14] [agent-44789154] [GOT] [prev:fa25fbd91b6bece831992468f28978d148c98869a0150a7fb30eb7fce80fe3ce] [hash:5fd40f17879e0137b2c6757022246ef1bd7cceb4f28811337e4a926150516e78] Updated and expanded bootstrap.rst to detail the entire bootstrapping from scratch procedure, including manual priority configurations, package catalogs, lookaside cache fallback, and circular/missing dependency anomalies.
+* [2026-06-16 22:32:39] [agent-44789154] [DEC,GOT] [prev:5fd40f17879e0137b2c6757022246ef1bd7cceb4f28811337e4a926150516e78] [hash:441133ddb8c34f268d97f095482f7e51c1dba5cc51aa53995c149d23e1c76f1a] Branded the downloadable DNF repository configuration files as MxOS 10 instead of MxOS CentOS Stream 10.
+* [2026-06-16 23:54:50] [agent-44789154] [DEC,RAT,GOT] [prev:667370ff840104b3ddfc8bbe175bfec4ea0396813ba8d6355e4be69ae70425a1] [hash:af439f2d66176139cc283f8465ae2715da41d00d24d679ba8a1f37f25d0f8dee] DEC: Switched from systemd path monitoring to a wrapper script trigger for dist-repo and rsync. RAT: systemd.path follows symlinks and misses updates. DEC: Skip full dnf upgrade on reposerver due to disk constraints. GOT: Automatic repo sync flow configured and deployed successfully.

.agents/journal/entries/2026-06-16-renich-agent-582266df.md

@@ -0,0 +1,4 @@
+# Journal: 2026-06-16 (agent-582266df)
+==============================
+
+* [2026-06-16 23:21:48] [agent-582266df] [GOT] [prev:441133ddb8c34f268d97f095482f7e51c1dba5cc51aa53995c149d23e1c76f1a] [hash:667370ff840104b3ddfc8bbe175bfec4ea0396813ba8d6355e4be69ae70425a1] Checked Koji man page availability on cs10-koji-00.mx-os and confirmed it relies entirely on its self-documenting CLI. mapped Koji configuration, database/storage volumes, tag inheritance, and hub policies.

roles/caddy/templates/mxos.repo.j2

@@ -1,17 +1,17 @@
 [mxos-baseos]
-name=MxOS CentOS Stream 10 BaseOS
+name=MxOS 10 BaseOS
 baseurl=http://cs10-repo.mx-os/BaseOS/$basearch/os/
 enabled=1
 gpgcheck=0
 
 [mxos-baseos-debuginfo]
-name=MxOS CentOS Stream 10 BaseOS Debuginfo
+name=MxOS 10 BaseOS Debuginfo
 baseurl=http://cs10-repo.mx-os/BaseOS/$basearch/debug/tree/
 enabled=0
 gpgcheck=0
 
 [mxos-baseos-source]
-name=MxOS CentOS Stream 10 BaseOS Source
+name=MxOS 10 BaseOS Source
 baseurl=http://cs10-repo.mx-os/BaseOS/source/tree/
 enabled=0
 gpgcheck=0

roles/common/tasks/main.yaml

@@ -204,7 +204,10 @@
     name: "*"
     state: latest
   register: dnf_upgrade_result
-  when: inventory_hostname != 'localhost'
+  when:
+    - inventory_hostname != 'localhost'
+    - "'reposerver' not in group_names"
+
 
 - name: Reboot if packages were upgraded
   ansible.builtin.reboot:

roles/koji_bootstrap/tasks/main.yaml

@@ -50,59 +50,59 @@
   changed_when: "'Target created' in koji_bootstrap_add_target_result.stdout"
   failed_when: "koji_bootstrap_add_target_result.rc != 0 and 'already exists' not in koji_bootstrap_add_target_result.stderr"
 
-- name: Add CentOS Stream 10 BaseOS external repository
+- name: Configure external repositories
-  ansible.builtin.command: >
+  block:
-    sudo -u {{ koji_admin_user }} koji
+    - name: Ensure external repositories exist globally
-    --config={{ koji_hub_admin_user_koji_dir }}/config
+      ansible.builtin.command: >
-    add-external-repo -t {{ koji_build_tag }}-build centos-stream-10-baseos
+        sudo -u {{ koji_admin_user }} koji
-    {{ centos_stream_10_baseos_url }}
+        --config={{ koji_hub_admin_user_koji_dir }}/config
-  register: koji_bootstrap_add_baseos_repo_result
+        add-external-repo {{ item.name }} {{ item.url }}
-  changed_when: "'External repository created' in koji_bootstrap_add_baseos_repo_result.stdout"
+      loop:
-  failed_when: "koji_bootstrap_add_baseos_repo_result.rc != 0 and 'already exists' not in koji_bootstrap_add_baseos_repo_result.stderr"
+        - { name: "centos-stream-10-baseos", url: "{{ centos_stream_10_baseos_url }}" }
+        - { name: "centos-stream-10-appstream", url: "{{ centos_stream_10_appstream_url }}" }
+        - { name: "centos-stream-10-epel", url: "https://mirror.facebook.net/fedora/epel/10/Everything/x86_64/" }
+        - { name: "centos-stream-10-crb", url: "https://mirror.stream.centos.org/10-stream/CRB/x86_64/os/" }
+      register: koji_add_ext_repo_global
+      changed_when: "'External repository created' in koji_add_ext_repo_global.stdout"
+      failed_when: "koji_add_ext_repo_global.rc != 0 and 'already exists' not in koji_add_ext_repo_global.stderr"
 
-- name: Update CentOS Stream 10 BaseOS external repository URL
+    - name: Ensure external repositories are associated with tag
-  ansible.builtin.command: >
+      ansible.builtin.command: >
-    sudo -u {{ koji_admin_user }} koji
+        sudo -u {{ koji_admin_user }} koji
-    --config={{ koji_hub_admin_user_koji_dir }}/config
+        --config={{ koji_hub_admin_user_koji_dir }}/config
-    edit-external-repo --url={{ centos_stream_10_baseos_url }} centos-stream-10-baseos
+        add-external-repo -t {{ koji_build_tag }}-build {{ item.name }}
-  changed_when: false
+      loop:
+        - { name: "centos-stream-10-baseos" }
+        - { name: "centos-stream-10-appstream" }
+        - { name: "centos-stream-10-epel" }
+        - { name: "centos-stream-10-crb" }
+      register: koji_add_ext_repo_tag
+      changed_when: "'Added external repo' in koji_add_ext_repo_tag.stdout"
+      failed_when: "koji_add_ext_repo_tag.rc != 0 and 'already associated' not in koji_add_ext_repo_tag.stderr"
 
-- name: Add CentOS Stream 10 AppStream external repository
+    - name: Ensure external repository URLs are updated
-  ansible.builtin.command: >
+      ansible.builtin.command: >
-    sudo -u {{ koji_admin_user }} koji
+        sudo -u {{ koji_admin_user }} koji
-    --config={{ koji_hub_admin_user_koji_dir }}/config
+        --config={{ koji_hub_admin_user_koji_dir }}/config
-    add-external-repo -t {{ koji_build_tag }}-build centos-stream-10-appstream
+        edit-external-repo --url={{ item.url }} {{ item.name }}
-    {{ centos_stream_10_appstream_url }}
+      loop:
-  register: koji_bootstrap_add_appstream_repo_result
+        - { name: "centos-stream-10-baseos", url: "{{ centos_stream_10_baseos_url }}" }
-  changed_when: "'External repository created' in koji_bootstrap_add_appstream_repo_result.stdout"
+        - { name: "centos-stream-10-appstream", url: "{{ centos_stream_10_appstream_url }}" }
-  failed_when: "koji_bootstrap_add_appstream_repo_result.rc != 0 and 'already exists' not in koji_bootstrap_add_appstream_repo_result.stderr"
+        - { name: "centos-stream-10-epel", url: "https://mirror.facebook.net/fedora/epel/10/Everything/x86_64/" }
+        - { name: "centos-stream-10-crb", url: "https://mirror.stream.centos.org/10-stream/CRB/x86_64/os/" }
+      changed_when: false
 
-- name: Update CentOS Stream 10 AppStream external repository URL
+    - name: Ensure external repository merge modes and priorities are set to bare
-  ansible.builtin.command: >
+      ansible.builtin.command: >
-    sudo -u {{ koji_admin_user }} koji
+        sudo -u {{ koji_admin_user }} koji
-    --config={{ koji_hub_admin_user_koji_dir }}/config
+        --config={{ koji_hub_admin_user_koji_dir }}/config
-    edit-external-repo --url={{ centos_stream_10_appstream_url }} centos-stream-10-appstream
+        edit-external-repo -t {{ koji_build_tag }}-build -m bare -p {{ item.priority }} {{ item.name }}
-  changed_when: false
+      loop:
-
+        - { name: "centos-stream-10-baseos", priority: 25 }
-- name: Add CentOS Stream 10 EPEL external repository
+        - { name: "centos-stream-10-appstream", priority: 30 }
-  ansible.builtin.command: >
+        - { name: "centos-stream-10-epel", priority: 15 }
-    sudo -u {{ koji_admin_user }} koji
+        - { name: "centos-stream-10-crb", priority: 20 }
-    --config={{ koji_hub_admin_user_koji_dir }}/config
+      changed_when: false
-    add-external-repo -t {{ koji_build_tag }}-build centos-stream-10-epel
-    https://mirror.facebook.net/fedora/epel/10/Everything/x86_64/
-  register: koji_bootstrap_add_epel_repo_result
-  changed_when: "'External repository created' in koji_bootstrap_add_epel_repo_result.stdout"
-  failed_when: "koji_bootstrap_add_epel_repo_result.rc != 0 and 'already exists' not in koji_bootstrap_add_epel_repo_result.stderr"
-
-- name: Add CentOS Stream 10 CRB external repository
-  ansible.builtin.command: >
-    sudo -u {{ koji_admin_user }} koji
-    --config={{ koji_hub_admin_user_koji_dir }}/config
-    add-external-repo -t {{ koji_build_tag }}-build centos-stream-10-crb
-    https://mirror.stream.centos.org/10-stream/CRB/x86_64/os/
-  register: koji_bootstrap_add_crb_repo_result
-  changed_when: "'External repository created' in koji_bootstrap_add_crb_repo_result.stdout"
-  failed_when: "koji_bootstrap_add_crb_repo_result.rc != 0 and 'already exists' not in koji_bootstrap_add_crb_repo_result.stderr"
 
 - name: Download repomd.xml from CentOS Stream 10 build repo
   ansible.builtin.get_url:

roles/koji_hub/files/koji-sign-spooler

@@ -244,6 +244,22 @@ def process_queue():
             try:
                 kojihub.context.cnx.commit()
                 logger.info(f"All signatures committed for build {build_id} ({build_nvr}).")
+
+                # Check tags and trigger dist-repo if tagged in mxos0
+                try:
+                    tags = list(kojihub.list_tags(build=build_id))
+                    tag_names = [t['name'] for t in tags]
+                    logger.info(f"Build {build_nvr} is tagged in: {tag_names}")
+                    if 'mxos0' in tag_names:
+                        short_key_id = sig_key_id[-8:].lower()
+                        logger.info(f"Triggering dist-repo for tag mxos0 with key {short_key_id}")
+                        dist_cmd = [
+                            '/usr/local/bin/mxos-trigger-dist-repo.sh', short_key_id
+                        ]
+                        subprocess.Popen(dist_cmd)
+                except Exception as e:
+                    logger.error(f"Failed to check tags/trigger dist-repo: {e}")
+
                 os.remove(file_path)
             except Exception as e:
                 logger.error(f"Failed to commit signatures to database: {e}")

roles/koji_hub/handlers/main.yaml

@@ -14,3 +14,5 @@
   ansible.builtin.service:
     name: koji-sign-spooler
     state: restarted
+
+

roles/koji_hub/tasks/main.yaml

@@ -354,6 +354,15 @@
     - /var/spool/koji-sign/queue
     - /var/spool/koji-sign/failed
 
+- name: Set SELinux context for Koji sign spool directory
+  ansible.builtin.command: semanage fcontext -a -t httpd_sys_rw_content_t "/var/spool/koji-sign(/.*)?"
+  changed_when: false
+
+- name: Apply SELinux context to Koji sign spool directory
+  ansible.builtin.command: restorecon -r -v /var/spool/koji-sign
+  changed_when: false
+
+
 - name: Copy Koji sign spooler daemon to libexec
   ansible.builtin.copy:
     src: koji-sign-spooler
@@ -423,6 +432,39 @@
     state: started
     enabled: true
 
+- name: Deploy mxos-repo-sync script
+  ansible.builtin.template:
+    src: mxos-repo-sync.sh.j2
+    dest: /usr/local/bin/mxos-repo-sync.sh
+    owner: root
+    group: root
+    mode: "0755"
+
+- name: Deploy mxos-trigger-dist-repo script
+  ansible.builtin.template:
+    src: mxos-trigger-dist-repo.sh.j2
+    dest: /usr/local/bin/mxos-trigger-dist-repo.sh
+    owner: root
+    group: root
+    mode: "0755"
+
+- name: Ensure mxos-repo-sync path unit is stopped and disabled
+  ansible.builtin.systemd:
+    name: mxos-repo-sync.path
+    state: stopped
+    enabled: false
+  ignore_errors: true
+
+- name: Remove mxos-repo-sync systemd unit files
+  ansible.builtin.file:
+    path: "{{ item }}"
+    state: absent
+  loop:
+    - /etc/systemd/system/mxos-repo-sync.service
+    - /etc/systemd/system/mxos-repo-sync.path
+
+
+
 - name: "Generate: MxOS Theme Graphic Assets on Localhost"
   delegate_to: localhost
   run_once: true
@@ -726,6 +768,12 @@
               font-size: 15px !important; /* Letras más grandes */
           }
 
+          table th[colspan] {
+              text-align: left !important;
+              width: auto !important;
+          }
+
+
           table td {
               padding: 14px 20px !important;
               border-bottom: 1px solid #e2e8f0;

roles/koji_hub/templates/mxos-repo-sync.sh.j2

@@ -0,0 +1,37 @@
+#!/usr/bin/bash
+# SPDX-License-Identifier: MIT-0
+set -euo pipefail
+
+SOURCE="/mnt/koji/repos-dist/mxos0/latest/"
+DEST_HOST="{{ groups['reposerver'][0] }}"
+DEST_DIR="/srv/www/repo/BaseOS"
+
+if [[ ! -d "$SOURCE" ]]; then
+    echo "Source directory $SOURCE does not exist. Skipping sync." >&2
+    exit 0
+fi
+
+echo "Starting atomic synchronization of mxos0 repository to $DEST_HOST..."
+
+# Pre-create directory structure on the destination
+/usr/bin/ssh -o StrictHostKeyChecking=no "root@${DEST_HOST}" "mkdir -p ${DEST_DIR}/x86_64/os ${DEST_DIR}/x86_64/debug/tree ${DEST_DIR}/source/tree"
+
+# Sync x86_64 to BaseOS/x86_64/os
+if [[ -d "${SOURCE}x86_64" ]]; then
+    echo "Syncing x86_64 architecture..."
+    /usr/bin/rsync -a -L --delete --delay-updates "${SOURCE}x86_64/" "root@${DEST_HOST}:${DEST_DIR}/x86_64/os/"
+fi
+
+# Sync x86_64-debuginfo to BaseOS/x86_64/debug/tree
+if [[ -d "${SOURCE}x86_64-debuginfo" ]]; then
+    echo "Syncing x86_64 debuginfo..."
+    /usr/bin/rsync -a -L --delete --delay-updates "${SOURCE}x86_64-debuginfo/" "root@${DEST_HOST}:${DEST_DIR}/x86_64/debug/tree/"
+fi
+
+# Sync src to BaseOS/source/tree
+if [[ -d "${SOURCE}src" ]]; then
+    echo "Syncing source RPMs..."
+    /usr/bin/rsync -a -L --delete --delay-updates "${SOURCE}src/" "root@${DEST_HOST}:${DEST_DIR}/source/tree/"
+fi
+
+echo "Synchronization completed successfully."

roles/koji_hub/templates/mxos-trigger-dist-repo.sh.j2

@@ -0,0 +1,22 @@
+#!/usr/bin/bash
+# SPDX-License-Identifier: MIT-0
+set -euo pipefail
+
+TAG="{{ koji_build_tag }}"
+KEY_ID="$1"
+LOG_FILE="/var/log/mxos-dist-repo-trigger.log"
+
+# Log all output to log file
+exec > >(tee -a "$LOG_FILE") 2>&1
+
+echo "[$(date -Iseconds)] Triggering dist-repo for tag $TAG with key $KEY_ID..."
+
+# Run dist-repo task in Koji and wait for it to complete
+if /usr/bin/sudo -u kojiadmin /usr/bin/koji dist-repo "$TAG" "$KEY_ID" \
+    --with-src --split-debuginfo --write-signed-rpms --allow-missing-signatures --wait; then
+    echo "[$(date -Iseconds)] dist-repo completed successfully. Invoking repository synchronization..."
+    /usr/local/bin/mxos-repo-sync.sh
+else
+    echo "[$(date -Iseconds)] dist-repo task failed. Skipping repository synchronization." >&2
+    exit 1
+fi

roles/kojira/templates/kojira.conf.j2

@@ -2,3 +2,4 @@
 server = https://{{ koji_hub_fqdn }}/kojihub
 cert = {{ kojira_pem_path }}
 serverca = {{ koji_remote_ca_trust_path }}
+check_external_repos = true