[{"content":"Context Enterprise customers need Terraform Enterprise to behave consistently even when the underlying cloud primitives, networking patterns, and operational controls differ. The goal was to create a repeatable deployment approach across AWS, Azure, and Google Cloud without hiding the important differences between those environments.\nMy role I worked as the technical authority across infrastructure design, deployment architecture, automation, and customer guidance. The work combined hands-on implementation with translating operational requirements into decisions that teams could support after go-live.\nArchitecture priorities Keep the deployment model consistent while respecting provider-specific networking and service behavior. Make infrastructure changes reviewable and repeatable through Terraform. Treat backup, recovery, observability, upgrades, and operational ownership as design inputs—not follow-up tasks. Keep documentation and runbooks aligned with the deployed architecture. Give customers clear trade-offs instead of a single cloud-agnostic abstraction. Outcome The resulting approach made multi-cloud deployments more predictable and gave engineering and customer teams a shared model for implementation and operations. It also created a reusable foundation for architecture reviews, troubleshooting, and future platform changes.\nThis overview intentionally excludes customer names, internal implementation details, and proprietary architecture.\n","permalink":"https://blog.aymantech.net/projects/terraform-enterprise-multicloud/","summary":"Architecture and automation for consistent Terraform Enterprise deployments across AWS, Azure, and Google Cloud.","title":"Multi-cloud Terraform Enterprise Platform"},{"content":"Why I built it Reading about modern Ansible workflows is useful, but the platform only becomes intuitive when every component has to work together. I built an all-in-one, containerized Ansible Automation Platform 2.6 environment on a Proxmox-based home lab as a permanent place to develop, break, and repair automation.\nWhat the lab covers AAP subscription manifest and registry service-account configuration. Automation Controller projects, inventories, credentials, job templates, surveys, and workflows. Custom execution environments built with ansible-builder. Git-based content workflows and project synchronization. Playbook execution and debugging through ansible-navigator. What it taught me Building the platform from the ground up exposed the boundaries between the controller, source control, credentials, and execution environments. That context made troubleshooting much faster: instead of treating a failed job as a controller problem, I could reason about the whole execution path.\nThe environment became the foundation for my EX374 preparation and remains a useful sandbox for testing reusable automation content.\nRelated field notes Road to EX374: Developing Automation with Ansible I Passed EX374: What to Focus On ","permalink":"https://blog.aymantech.net/projects/aap-home-lab/","summary":"A working Ansible Automation Platform 2.6 environment for building execution environments and practising controller workflows.","title":"Containerized AAP 2.6 Home Lab"},{"content":"Goal This site is more than a personal homepage. It is a lightweight publishing system for turning lab work, certification study, and architecture experience into notes that remain useful after the immediate task is finished.\nDesign choices Hugo keeps the site static, fast, and easy to maintain. Markdown keeps writing portable and reviewable in Git. GitHub Actions builds the site consistently on every deployment. GitHub Pages provides a simple hosting path behind a custom domain. Structured metadata, RSS, tags, and search make the content easier to discover. Publishing workflow Every article starts as a Markdown file, is reviewed as a normal source change, and is transformed into a production site by the deployment workflow. The source remains the system of record; generated output is deliberately kept out of version control.\nThat combination keeps the operational overhead low enough that the focus stays on publishing useful engineering work.\n","permalink":"https://blog.aymantech.net/projects/engineering-blog/","summary":"A Hugo and GitHub Pages publishing workflow for turning hands-on learning into durable, searchable technical notes.","title":"Engineering Blog and Public Learning System"},{"content":"It is official: I passed the Red Hat Certified System Administrator in OpenShift exam (EX280).\nThis is an important milestone in my OpenShift journey. The result represents many hours of official training, reading documentation, rebuilding lab environments, troubleshooting mistakes, and repeating tasks until the workflow became familiar.\nIn my earlier post, Road to EX280: Preparing for OpenShift Administration, I described the plan I was following. Passing the exam confirms that the combination of structured learning and consistent hands-on practice worked for me.\nThis post contains no exam questions, tasks, or confidential details. I am only sharing the preparation methods and lessons that helped me.\nThe preparation approach that worked for me My preparation was built around four main areas:\nOfficial Red Hat training. The published EX280 objectives and Red Hat documentation. A repeatable OpenShift home lab. Personal notes and continuous review. Each part supported the others. The training provided structure, the objectives defined the destination, the home lab turned theory into practical experience, and writing notes exposed the areas I did not yet understand well enough.\nStarting with official Red Hat training I took the official training from Red Hat. It gave me a clear path through OpenShift administration instead of approaching the platform as a collection of unrelated commands.\nThe training helped me connect topics such as application deployment, authentication, authorization, routes, services, network policies, resource controls, Operators, and application security. More importantly, it showed how these resources interact inside a real OpenShift cluster.\nCompleting the training was not the end of the preparation. EX280 is a performance-based exam, so understanding a concept is only useful if I can configure it, verify it, and troubleshoot it under time pressure.\nTreating the official objectives as a checklist I followed the official EX280 exam objectives rather than relying on a random collection of tutorials.\nFor every objective, I tried to answer five questions:\nWhat does this resource or feature do? Can I configure it from the command line? Can I inspect and verify the resulting state? Can I recognize and fix a broken configuration? Can I repeat the task without copying a prepared command? This approach kept the study plan focused. It also changed the goal from memorizing syntax to proving that the required result was actually working.\nBuilding an OpenShift lab with kcli The most valuable part of my preparation was hands-on practice. I used kcli in my Proxmox home lab to deploy OpenShift environments for testing and study.\nA repeatable lab allowed me to:\nStart with a clean cluster. Practise the same objective several times. Break configurations intentionally. Diagnose problems using events, logs, YAML, and resource status. Delete the environment and rebuild it when necessary. Automating the infrastructure removed much of the fear of making mistakes. If an experiment went badly, the cluster was disposable. That made it easier to explore, troubleshoot, and learn how OpenShift behaves beyond the happy path.\nI documented the foundation of this home-lab workflow in Install kcli on macOS and Manage Proxmox VMs.\nWriting and reviewing my notes I kept my EX280 study notes in a public GitHub repository:\nmoabdelbasset/ex280-v414\nWriting notes was more useful than saving commands in shell history. It forced me to explain what a command changed, why it worked, and how I could verify the outcome.\nThe repository also gave me a practical revision tool. Before the exam, I could return to a topic, compare it with the official objectives, and identify areas that needed another lab session.\nThe notes began around OpenShift 4.14, while exam versions and product behavior continue to evolve. I therefore treated the objectives and documentation for my booked exam as authoritative and retested older notes where necessary.\nWhat passing EX280 reinforced The biggest lesson from this experience is that hands-on repetition matters more than passive reading.\nReading documentation helped me understand the available resources and options. Repeated lab work helped me use that information quickly, recognize incorrect states, and recover from mistakes.\nA few habits made a real difference:\nRead the required outcome carefully before making changes. Use the documentation as a working tool instead of trying to memorize everything. Verify resources with YAML, conditions, events, logs, and connectivity tests. Practise troubleshooting, not only successful deployments. Make sure configurations are persistent and survive the conditions required by the task. Build command-line fluency with oc, resource discovery, oc explain, selectors, and structured output. The exam result is valuable, but the deeper benefit is the confidence gained from operating OpenShift repeatedly in a real environment.\nWhat comes next Passing EX280 is not the end of the OpenShift journey. I plan to keep using the home lab, improve the GitHub notes, and apply these administration skills to more complex Kubernetes and OpenShift scenarios.\nI also want to spend more time on automation, advanced troubleshooting, platform reliability, and the operational practices needed to run OpenShift beyond individual exam objectives.\nFor now, I am happy to celebrate this milestone:\nEX280 passed — Red Hat Certified System Administrator in OpenShift achieved.\nResources Official EX280 exam page and objectives My EX280 preparation post My EX280 notes on GitHub Install kcli on macOS and Manage Proxmox VMs ","permalink":"https://blog.aymantech.net/posts/passed-ex280/","summary":"A personal reflection on passing EX280 through official Red Hat training, objective-driven study, documentation, and hands-on OpenShift practice in my home lab.","title":"I Passed EX280: Red Hat OpenShift Administrator"},{"content":"My AOOSTAR GEM10 started with a 512 GB NVMe drive for Proxmox VE. As the number of VMs and lab environments grew, I added a 2 TB Lexar NM790 PCIe Gen4 NVMe SSD to provide a separate, fast storage pool for virtual-machine disks.\nThis guide covers the complete process: opening the GEM10, reaching the additional M.2 slot, installing the SSD, confirming that Linux sees the correct device, and optionally adding it to Proxmox as LVM-thin storage.\nImportant: Opening the chassis and moving the motherboard can damage connectors or affect the warranty. Check the documentation for your exact GEM10 revision before continuing, use anti-static precautions, and never work on the computer while it is connected to power.\nTested hardware Component Value Mini PC AOOSTAR GEM10 CPU AMD Ryzen 7 6800H Operating system Proxmox VE Existing SSD 512 GB NVMe, used as the OS disk New SSD Lexar NM790 2 TB PCIe Gen4 NVMe AOOSTAR specifies three M.2 2280 NVMe slots for this GEM10 model. The original system disk occupies one slot; the other slots provide room to add fast local storage without replacing the Proxmox boot drive.\nWhat you need A Phillips screwdriver that fits the chassis screws. An M.2 2280 NVMe PCIe SSD. An anti-static wrist strap, or another suitable way to ground yourself. A clean, well-lit work surface and a container for the screws. An optional M.2 thermal pad of the correct thickness. Before changing the hardware, make sure important VMs, containers, and Proxmox configuration are backed up somewhere other than this machine.\n1. Shut down and disconnect the GEM10 Shut down Proxmox cleanly from the web interface or console. Wait until the system is fully off, then disconnect:\nThe power adapter. Ethernet. USB devices. HDMI or DisplayPort cables. Press the power button once after unplugging the adapter to help discharge any residual power. Move the device to the prepared work surface.\n2. Remove the bottom cover Turn the GEM10 upside down and remove the four bottom screws. Keep track of their positions in case the screws are not all the same length.\nLift the bottom cover slowly. The cover contains a cooling fan connected to the motherboard, so do not pull it away from the chassis or put tension on the cable. Rest the cover next to the computer while keeping the cable relaxed, or disconnect the fan by holding the connector rather than pulling on its wires.\n3. Access the additional M.2 slots On my unit, the additional NVMe slots were on the opposite side of the motherboard. Reaching them required carefully lifting and turning the board.\nTake a photo of the cable routing and connector positions before disconnecting anything. Remove the motherboard mounting screws. Disconnect only the fan or ribbon cables that prevent the board from moving safely. Lift the motherboard by its edges and turn it over without scraping it against the chassis. The internal layout may differ between revisions. Do not force the board if it does not lift freely; stop and look for a missed screw, cable, or connector.\n4. Install the new NVMe SSD Locate an empty M.2 2280 NVMe slot and its retaining screw.\nRemove the retaining screw. Align the notch in the Lexar NM790 with the key in the M.2 socket. Insert the drive at roughly a 30-degree angle. Push it fully into the connector without forcing it. Lower the free end of the SSD onto the standoff. Reinstall the retaining screw until it is snug. The SSD should sit flat and should not bow. The retaining screw only needs to hold the drive in place; overtightening it can damage the board or standoff.\n5. Add the thermal pad If the correct spare thermal pad is available, place it on the newly installed SSD so that it makes contact with the intended heatsink or chassis surface after reassembly. Remove any protective film first.\nDo not stack pads or use one that is too thick. Excessive pressure can bend the SSD or motherboard. If there is no matching contact surface, leave the pad out and monitor the drive temperature after booting.\n6. Reassemble the system Work back through the disassembly steps:\nReturn the motherboard to its original position. Reconnect every cable that was disconnected. Reinstall the motherboard screws in their original locations. Confirm that the bottom-cover fan is connected. Position the bottom cover without pinching any wires. Reinstall and gently tighten the four case screws. Before closing the cover, compare the system with the photo taken during disassembly. This is a quick way to catch a missed cable.\n7. Boot and identify the new disk Reconnect the network and display if needed, then connect power and start the GEM10. Confirm that Proxmox boots normally before making any storage changes.\nFrom the Proxmox shell, list only physical block devices and include identifying information:\nlsblk -d -o NAME,SIZE,MODEL,SERIAL,TYPE My result showed the two NVMe namespaces:\nNAME SIZE MODEL SERIAL TYPE nvme0n1 1.9T Lexar SSD NM790 \u0026lt;new-serial\u0026gt; disk nvme1n1 476.9G \u0026lt;existing-model\u0026gt; \u0026lt;os-serial\u0026gt; disk In this example, nvme1n1 is the existing Proxmox system disk and nvme0n1 is the new 2 TB Lexar drive. Do not assume those names will be the same on another machine—or even after a future boot. Match the capacity, model, and serial number before running a destructive command.\nThe persistent device links are also useful:\nls -l /dev/disk/by-id/ | grep -i nvme If the nvme command is not installed, add the Debian package used by Proxmox:\napt update apt install nvme-cli Then display the detected NVMe devices:\nnvme list Example:\nNode SN Model /dev/nvme0n1 \u0026lt;new-serial\u0026gt; Lexar SSD NM790 2TB /dev/nvme1n1 \u0026lt;os-serial\u0026gt; \u0026lt;existing OS SSD\u0026gt; Check the health information for the new drive\u0026rsquo;s controller:\nnvme smart-log /dev/nvme0 Review at least:\ncritical_warning — it should be 0 on a healthy new disk. temperature — check it again under sustained I/O. available_spare. percentage_used. media_errors and error log entries. Data units read and written. Use the controller that corresponds to the Lexar namespace shown by nvme list; it may not be /dev/nvme0 on another system.\n8. Optionally add the disk to Proxmox Destructive operation: Initializing, wiping, partitioning, or creating LVM storage erases data on the selected disk. Confirm the Lexar model, serial number, and 2 TB capacity again. Never select the existing Proxmox OS disk.\nIn the Proxmox web interface:\nSelect the Proxmox node in the left tree. Open Disks → Disks. Select the new Lexar SSD and verify its model, serial number, and size. If required for the chosen storage workflow, use Initialize Disk with GPT. Proxmox can use the disk in several ways:\nStorage type Good fit Main consideration LVM-thin Local VM and container disks Thin provisioning, snapshots, and clones; not shared between nodes ZFS Data-integrity features and ZFS workflows Plan memory use, topology, and redundancy before creating the pool Directory ISOs, backups, snippets, or file-based VM images Simple and flexible, with behavior dependent on the filesystem and image format LVM Local block storage Does not provide the efficient snapshot and clone support of LVM-thin For this single-node home lab, I chose LVM-thin for VM disks. It is a straightforward match for fast local storage and supports thin provisioning, snapshots, and clones.\nTo create it, open Node → Disks → LVM-Thin and choose Create: Thinpool. Select only the new drive or the volume group created for it, use a clear name such as fastnvme, and enable Add Storage if the interface offers that option. Review the final selection before confirming; the exact fields can differ slightly between Proxmox VE releases and according to the disk\u0026rsquo;s current state.\nAfter creation, verify that the storage is active:\npvesm status The result should contain an enabled entry named fastnvme. It can now be selected when creating a VM or when moving an existing VM disk.\nLVM-thin is thin-provisioned, so allocated virtual sizes can exceed currently used physical blocks. Monitor actual pool usage and keep backups; thin provisioning is not extra physical capacity and does not provide redundancy.\nVerification checklist The SSD is fully inserted and secured. The motherboard and bottom cover are reinstalled correctly. The fan and any ribbon cables are reconnected. Proxmox boots successfully. lsblk shows the expected 2 TB model and serial number. nvme list identifies the Lexar NM790. The NVMe SMART log has no critical warning or media errors. The new disk—not the OS disk—was selected in Proxmox. pvesm status shows fastnvme as active, if storage was created. Backups remain on a different physical device or system. Troubleshooting The SSD is not detected Shut down, disconnect power, and reseat the SSD.\nConfirm that the drive is M.2 2280 NVMe PCIe, not M.2 SATA.\nInspect the socket and retaining screw, and try another available NVMe slot if appropriate.\nCheck whether the drive appears in the firmware setup.\nReview the kernel messages:\ndmesg -T | grep -i nvme The system does not boot Disconnect power and recheck the fan and ribbon cables. Confirm that the motherboard is seated correctly and no cable is trapped below it. Check for a missed or misplaced mounting screw. Temporarily remove the new SSD and test the original configuration. The disk appears in firmware but not in Linux Check both the NVMe tool and PCI device list:\nnvme list lspci -nn | grep -i \u0026#39;non-volatile\u0026#39; dmesg -T | grep -i nvme If the controller appears in lspci but no namespace is available, inspect the NVMe and kernel messages before attempting to initialize anything.\nThe SSD runs hot Confirm that the cooling pad has made proper contact and that the bottom fan is connected and unobstructed. Compare idle and load temperatures with nvme smart-log; a single reading immediately after boot does not show sustained-workload behavior.\nFinal result The GEM10 now has its original 512 GB Proxmox system disk plus a dedicated 2 TB Lexar NM790 for VM and container storage. Keeping the OS and lab workloads on separate devices gives the home lab more capacity and makes storage placement easier to reason about.\nThis new pool also gives my kcli-based Proxmox workflow more room for repeatable Kubernetes and OpenShift test environments.\nReferences AOOSTAR GEM10 product specifications Proxmox VE Administration Guide Proxmox VE Storage Manager documentation nvme-cli documentation ","permalink":"https://blog.aymantech.net/posts/install-additional-nvme-aoostar-gem10-proxmox/","summary":"A practical hardware-to-Proxmox guide for adding a 2 TB Lexar NM790 NVMe SSD to the AOOSTAR GEM10 home-lab server.","title":"Install an Additional NVMe SSD in the AOOSTAR GEM10"},{"content":"I have started preparing for the Red Hat Certified System Administrator in OpenShift exam (EX280). This is the next step in my OpenShift learning path and a chance to turn the Kubernetes and container fundamentals I have been building into practical cluster-administration skills.\nThis post is a record of how I am preparing. It contains no exam questions or confidential exam material—only public objectives, official documentation, and the lab work I am doing myself.\nStarting with official Red Hat training I took the official training from Red Hat, which gave me a structured foundation instead of learning isolated commands without context. The course aligned with this certification path is Red Hat OpenShift Administration II: Configuring a Production Cluster (DO280).\nThe training helped connect day-to-day administration topics such as:\nManaging applications from manifests and packaged content. Authentication, authorization, users, groups, and roles. Routes, services, TLS, and network policies. Resource quotas, limit ranges, and developer self-service. Operators and application security. Cluster health, troubleshooting, and updates. Completing a course is only the beginning. EX280 is performance-based, so I need to be able to perform and verify the work on a real cluster—not simply recognize the correct option in a list.\nFollowing the official exam objectives My source of truth is the official EX280 exam page and objectives.\nI use the objectives as a checklist. For each study point, I want to be able to:\nExplain what the resource or feature does. Create or configure it with the oc command-line interface. Inspect the resulting YAML and status. Test that the configuration produces the intended outcome. Troubleshoot a deliberately broken version. Repeat the workflow without relying on copied commands. The official objectives currently cover OpenShift management, resource manifests, application deployment, authentication and authorization, network security, developer self-service, Operators, and application security. Red Hat also makes an important point: configurations created during a performance-based exam must persist without manual intervention after a reboot.\nI regularly check the objectives again rather than assuming they have stayed unchanged. The public EX280 page currently references OpenShift Container Platform 4.18, while my original notes repository was created around OpenShift 4.14. I therefore treat the booked exam version and its current official objectives as authoritative, then update or retest older notes where the platform behavior has changed.\nBuilding a repeatable OpenShift home lab Reading documentation is useful, but OpenShift administration only becomes familiar through repetition. I use kcli in my Proxmox home lab to deploy OpenShift clusters for testing and study.\nThis gives me a lab that I can rebuild instead of being afraid to break:\nProvision a clean cluster. Practise an objective from scratch. Introduce a failure intentionally. Diagnose it using events, logs, resource status, and documentation. Delete the environment and repeat the exercise. Automating the infrastructure setup lets me spend more time on OpenShift administration and less time manually creating VMs. It also makes destructive practice safer because the cluster is disposable and separate from workloads I care about.\nI documented the foundation of this setup in two earlier posts:\nInstall kcli on macOS and Manage Proxmox VMs Deploy a Single-Node k3s Kubernetes Cluster on Proxmox with kcli The k3s deployment was a quick validation of the kcli and Proxmox pipeline. OpenShift exercises additional paths and has a much larger resource footprint, but the same repeatable-lab principle applies.\nKeeping my study notes on GitHub I created a public repository for the commands, explanations, and examples I collect while studying:\nmoabdelbasset/ex280-v414\nThe repository currently includes notes on areas such as:\nUsing the web console and oc CLI. Querying, formatting, importing, and exporting resources. Images, projects, logs, cluster health, and troubleshooting. Deployments, manifests, Kustomize, Helm, jobs, services, labels, and selectors. Secrets and application configuration. Writing the notes forces me to explain why a command works, not just save it in shell history. Keeping them in Git also gives me a visible checklist, change history, and a place to improve examples whenever I learn a better approach.\nThese are personal study notes rather than a replacement for Red Hat documentation. Anyone using them should confirm the OpenShift and exam version first.\nMy preparation loop My study sessions follow a simple cycle:\nOfficial objective → Red Hat documentation → home-lab practice ↑ ↓ └──────── verify, troubleshoot, and document ────────┘ For every topic, the final question is not “Did the command run?” but “Can I prove the required state is correct?” That means checking resource YAML, conditions, events, logs, permissions, connectivity, and application behavior as appropriate.\nI am also building speed with the tools available in the environment: oc explain, API resource discovery, JSONPath and custom-column output, label selectors, server-side dry runs, and the official product documentation. The goal is not to memorize every command. It is to know how to find the right information quickly and apply it correctly.\nWhat comes next The plan from here is straightforward:\nComplete a first pass through every official objective. Fill the remaining gaps in the GitHub notes. Rebuild the OpenShift lab regularly with kcli. Practise mixed scenarios that combine security, networking, storage, and application management. Recheck the official exam version and objectives before booking. I will keep updating the repository as the preparation progresses, and I will share the lessons that come out of the lab work here.\nReferences Official EX280 exam page and objectives Red Hat OpenShift Administration II (DO280) OpenShift Container Platform documentation My EX280 study notes on GitHub ","permalink":"https://blog.aymantech.net/posts/preparing-for-ex280/","summary":"My EX280 preparation plan combines official Red Hat training and documentation with repeatable OpenShift practice in my Proxmox home lab.","title":"Road to EX280: Preparing for OpenShift Administration"},{"content":"After connecting kcli to my Proxmox home lab, I wanted to test more than basic VM creation. Deploying a small Kubernetes cluster exercises the complete automation path: API authentication, cloud-image import, cloud-init, bridge networking, SSH, cluster installation, and kubeconfig generation.\nThis post continues from Install kcli on macOS and Manage Proxmox VMs. That guide covers the API token, passwordless SSH, provider configuration, and first VM.\nThe result here was a working single-node k3s cluster running Kubernetes v1.36.2, provisioned in roughly 70 seconds on a Proxmox VM with 2 vCPUs, 4 GB of memory, and a 20 GB disk.\nAll private IP addresses and credentials have been replaced with placeholders.\nWhy start with k3s? My longer-term goal is to test more demanding Kubernetes and OpenShift deployments. Starting with k3s gave me a fast way to validate the Proxmox pipeline before spending significantly more CPU, memory, storage, and time.\nArea Single-node k3s OpenShift SNO Node provisioning Cloud-init on a normal cloud image Ignition on RHCOS Pull secret Not required Required Temporary bootstrap VM No Yes for the installer workflow Lab footprint used here 2 vCPU, 4 GB RAM, 20 GB disk Much larger Deployment time observed About 70 seconds Usually much longer A successful k3s deployment does not prove that every OpenShift-specific path will work. It does prove most of the shared Proxmox plumbing at a much lower troubleshooting cost.\nPrerequisites Before continuing, verify that:\nkcli is configured with a working Proxmox provider. Passwordless SSH works from the kcli environment to the Proxmox host. The Mac can reach the Proxmox API and VM network. kubectl is installed on the Mac. A DHCP lease is available on the Proxmox bridge, or static networking is prepared. The Proxmox storage layout has enough room for a cloud image and VM disk. Check the provider and existing VMs:\nkcli -C homelab list vm Inspect the k3s defaults kcli exposes the parameters accepted by each cluster type:\nkcli info cluster k3s The defaults relevant to this deployment were:\napi_ip: None cluster: myk3s ctlplanes: 1 disk_size: 10 domain: karmalabs.corp image: ubuntu2004 memory: 1024 network: default numcpus: 2 pool: None sdn: flannel token: supersecret workers: 0 ctlplanes: 1 and workers: 0 already describe a single-node cluster. I changed the remaining values to fit the home lab:\nDefault Reason to change it Value used image: ubuntu2004 I wanted a current image already used in the lab centos9stream memory: 1024 Too tight once workloads are added 4096 disk_size: 10 Limited room for container images 20 network: default This is normally a Libvirt network vmbr0 pool: None Proxmox disk storage should be explicit local-lvm token: supersecret A published default is not a secret A generated token domain: karmalabs.corp Not my lab domain lab.example.com An api_ip is not required for one control-plane node. kcli requires one for a non-cloud, multi-control-plane deployment where a stable API endpoint is needed.\nDiagnose Proxmox storage before deploying My first image download failed with:\nImage centos9stream not Added because storage local not found on pve. The message suggested that local did not exist, but Proxmox showed it as active:\nssh root@\u0026lt;PVE_HOST_OR_IP\u0026gt; pvesm status I then inspected the storage configuration:\nssh root@\u0026lt;PVE_HOST_OR_IP\u0026gt; \u0026#34;cat /etc/pve/storage.cfg\u0026#34; Finally, I compared it with kcli\u0026rsquo;s view:\nkcli list pool The useful diagnostic sequence is:\npvesm status — does the storage exist and is it active? /etc/pve/storage.cfg — which content types does it support? kcli list pool — which storage can kcli actually use? Compare the last two results. Why local was missing from kcli The relevant storage capabilities in my lab looked like this:\nStorage Backing device Relevant content Visible to kcli local Internal NVMe directory import, snippets, iso No local-lvm Internal NVMe LVM-thin images, rootdir Yes external4tb External LVM-thin images, rootdir Yes external1tb External directory images, iso, snippets Yes kcli\u0026rsquo;s Proxmox provider filters its pool list to active storage with the images content type. The local storage existed, but it did not support images, so it was deliberately absent from kcli list pool.\nThis distinction explains the misleading \u0026ldquo;storage not found\u0026rdquo; error: it means \u0026ldquo;not available as a VM-image storage to this provider,\u0026rdquo; not necessarily \u0026ldquo;missing from Proxmox.\u0026rdquo;\nChoose storage based on workload I initially considered placing everything on the external drives. That would have saved NVMe capacity, but Kubernetes control-plane storage is latency-sensitive. VM disks belong on the lowest-latency reliable device available.\nThe split I used was:\npool: local-lvm imagepool: external1tb pool: local-lvm keeps the running VM disk on internal NVMe. imagepool: external1tb stores the reusable cloud-image template on capacity-oriented storage. The current kcli Proxmox provider also uses the conventional local directory for two internal tasks:\nIt stages image downloads under /var/lib/vz/images/0 before importing them into imagepool. It uploads cloud-init metadata and user data to /var/lib/vz/snippets and references local:snippets/.... That means local still needs enough temporary free space and the snippets content type, even when it does not appear in kcli list pool.\nStorage names and hardware differ between labs. The rule is more important than my specific names: place the running control-plane disk on reliable low-latency storage, select an images-capable imagepool, and keep kcli\u0026rsquo;s local staging and snippets paths available.\nConfirm the provider configuration The relevant parts of my ~/.kcli/config.yml were:\ndefault: client: homelab pool: local-lvm numcpus: 2 memory: 2048 homelab: type: proxmox host: \u0026lt;PVE_HOST_OR_IP\u0026gt; user: root@pam auth_token_name: \u0026lt;TOKEN_ID\u0026gt; auth_token_secret: ?secret pool: local-lvm imagepool: external1tb node: pve filtertag: kcli verify_ssl: false The actual token remains in ~/.kcli/secrets.yml, not in this file.\nCreate the k3s parameter file Generate a strong cluster token:\nopenssl rand -hex 32 Create k3s-single.yml:\ncluster: k3slab domain: lab.example.com ctlplanes: 1 workers: 0 image: centos9stream numcpus: 2 memory: 4096 disk_size: 20 network: vmbr0 pool: local-lvm token: \u0026lt;GENERATED_CLUSTER_TOKEN\u0026gt; sdn: flannel # Optional: pin the exact version used in this walkthrough. install_k3s_version: v1.36.2+k3s1 Do not commit a real cluster token to Git. Store a private parameter file outside the repository or inject the value securely in a repeatable automation workflow.\nLeaving out install_k3s_version follows the k3s stable channel instead. That is convenient for experiments but makes future runs less reproducible.\nDownload the cloud image Download CentOS Stream 9:\nkcli download image centos9stream The successful run downloaded roughly 1.5 GB in about 15 seconds.\nThe download speed surprised me until I checked the provider implementation and command output. kcli opens an SSH session and runs curl on the Proxmox host. It does not carry the entire image through the Mac. The host downloads to the temporary local staging path, Proxmox imports it into imagepool, and kcli removes the temporary file.\nThe practical consequences are:\nDownload speed depends on the Proxmox host\u0026rsquo;s internet connection. The Mac\u0026rsquo;s Wi-Fi is not carrying the image payload. local needs enough free space for the temporary file. The target imagepool must appear in kcli list pool. Deploy the cluster Run:\nkcli create cluster k3s \\ --paramfile k3s-single.yml \\ k3slab During my run, kcli:\nCreated and booted k3slab-ctlplane-0. Applied hostname, network, SSH keys, and other cloud-init settings. Acquired a DHCP lease on vmbr0. Installed the guest and SELinux dependencies. Installed and started k3s. Waited for the node and generated a per-cluster kubeconfig. The final output was:\nk3slab-ctlplane-0 deployed on homelab K3s cluster k3slab deployed!!! INFO export KUBECONFIG=$HOME/.kcli/clusters/k3slab/auth/kubeconfig INFO export PATH=$PWD:$PATH The complete deployment took approximately 70 seconds in this lab. Treat that as an observation, not a benchmark; image caching, storage, CPU, DNS, and internet speed all affect the result.\nUse the correct kubeconfig My first kubectl get nodes failed with an authentication error mentioning a completely unrelated GKE plugin. The new cluster was healthy; kubectl had fallen back to my existing ~/.kube/config, whose current context pointed to an old GKE cluster.\nkcli writes a separate kubeconfig for each cluster and does not replace the default file. Export the path printed at the end of the deployment:\nexport KUBECONFIG=\u0026#34;$HOME/.kcli/clusters/k3slab/auth/kubeconfig\u0026#34; Then validate the cluster:\nkubectl config current-context kubectl cluster-info kubectl get nodes -o wide kubectl get pods -A kubectl get storageclass If kubectl reports an unexpected server address or authentication provider, check these before troubleshooting the cluster:\nprintf \u0026#39;%s\\n\u0026#39; \u0026#34;$KUBECONFIG\u0026#34; kubectl config current-context kubectl config view --minify Add a small zsh helper For repeated use, add this function to ~/.zshrc:\nkubeswitch() { export KUBECONFIG=\u0026#34;$HOME/.kcli/clusters/$1/auth/kubeconfig\u0026#34; kubectl config current-context } Reload the shell and select a cluster by name:\nsource ~/.zshrc kubeswitch k3slab What the single-node cluster includes The default k3s installation provides:\nFlannel for pod networking. Traefik as the ingress controller. Local Path Provisioner for local persistent volumes. SQLite as the single-server datastore when no other datastore is configured. Inspect the installed components:\nkubectl get pods -A kubectl get storageclass SQLite is suitable for this simple single-server lab. Multi-server high availability requires embedded etcd or a supported external datastore.\nTroubleshooting notes Storage exists but kcli says it is not found Check whether it supports images and appears in:\nkcli list pool Do not rely on pvesm status alone.\nThe image download fails Check:\nPasswordless SSH from the kcli environment to Proxmox. Free space under local for the staging file. The exact imagepool name. The images content type on the target storage. Proxmox host DNS and outbound internet access. The VM boots but cloud-init settings are missing Check:\nThe snippets content type on local. Files under /var/lib/vz/snippets on the Proxmox host. The VM\u0026rsquo;s cicustom and cloud-init drive configuration. Whether centos9stream is a genuine cloud image. kubectl mentions the wrong cluster Check KUBECONFIG and the current context before assuming the k3s deployment failed:\nprintf \u0026#39;%s\\n\u0026#39; \u0026#34;$KUBECONFIG\u0026#34; kubectl config current-context Benign messages observed during deployment I also saw warnings about the QEMU guest-agent service being registered before its virtio port appeared, and host iptables-save tools not being present. The deployment completed because k3s provides its own iptables tooling. These messages are useful context if networking later behaves unexpectedly, but neither blocked this lab run.\nClean up the cluster When finished:\nkcli delete kube k3slab kcli list vm The second command checks for an orphaned cluster VM. My provider uses filtertag: kcli, which keeps untagged pre-existing VMs outside kcli\u0026rsquo;s normal view and acts as a useful guardrail. I still review the selected provider and resource names before every delete operation.\nWhat this deployment proved This run validated:\nProxmox API-token authentication. Image download and import. Cloud-init metadata and SSH-key injection. Bridge networking through vmbr0. DHCP for the cluster node. kcli\u0026rsquo;s k3s installation and wait logic. Per-cluster kubeconfig generation. It did not validate OpenShift-specific requirements such as RHCOS, Ignition, an installer bootstrap VM, a highly available API endpoint, or the larger control-plane storage footprint.\nFinal lessons kcli lists Proxmox pools that are active and support images. local still handles temporary image staging and cloud-init snippets. The Proxmox host downloads the cloud image; the Mac orchestrates the process. Running control-plane disks belong on reliable, low-latency storage. A one-node k3s cluster does not need a separate api_ip. Always export the kubeconfig generated for the cluster. k3s is a fast, practical validation step before attempting a larger Kubernetes or OpenShift deployment. References kcli Kubernetes cluster documentation kcli Proxmox provider source kcli k3s defaults k3s cluster datastore documentation k3s networking documentation k3s v1.36.2+k3s1 release Proxmox VE storage documentation ","permalink":"https://blog.aymantech.net/posts/deploy-single-node-k3s-proxmox-kcli/","summary":"A tested kcli workflow for deploying k3s on Proxmox, with the storage and kubeconfig problems I hit along the way.","title":"Deploy a Single-Node k3s Kubernetes Cluster on Proxmox with kcli"},{"content":"I wanted a repeatable way to create VMs in my Proxmox home lab without clicking through the web interface every time. kcli provides one command-line workflow across several infrastructure providers, including Proxmox, and can build VMs directly from cloud images.\nThis guide covers the complete path I used:\nRun kcli on macOS. Create a Proxmox API token. Configure passwordless SSH. Prepare Proxmox storage. Store the kcli configuration and token separately. Verify the connection and create the first VM. I have replaced my own host address and token with placeholders. Do not publish real credentials in a blog post or commit them to Git.\nHow kcli communicates with Proxmox The Proxmox provider uses two separate paths:\nThe Proxmox API on port 8006 is used to manage VM resources. SSH is used to run host-side image staging commands and upload cloud-init or Ignition files. This distinction matters. A working API token is not enough to download an image and build a usable cloud-init VM; the Mac running kcli must also be able to reach the Proxmox host over SSH without a password prompt.\nPrerequisites Before starting, make sure you have:\nA Mac with zsh. Homebrew installed. Docker Desktop or Podman available and running. A Proxmox VE host reachable from the Mac on ports 22 and 8006. Administrative access to the Proxmox web interface. An SSH key on the Mac. You can test basic network access with:\nssh root@\u0026lt;PVE_HOST_OR_IP\u0026gt; curl -k https://\u0026lt;PVE_HOST_OR_IP\u0026gt;:8006/api2/json/version The -k option is useful when the home lab uses Proxmox\u0026rsquo;s default self-signed certificate. It should not be necessary when the certificate is signed by a CA trusted by the Mac.\nInstall kcli on macOS kcli\u0026rsquo;s official installer uses a container when it cannot use an RPM or Debian package manager. This is a good fit for macOS because the container contains the dependencies for all providers, including proxmoxer.\nThe example below uses Podman:\nbrew install podman podman machine init podman machine start If a Podman machine already exists, skip podman machine init and start the existing machine.\nDownload the installer so you can inspect it before running it:\ncurl -fsSL \\ https://raw.githubusercontent.com/karmab/kcli/main/install.sh \\ -o /tmp/kcli-install.sh less /tmp/kcli-install.sh bash /tmp/kcli-install.sh On macOS I run the script as my normal user because it adds the kcli alias to my shell configuration and creates files below my home directory.\nOpen a new terminal, or reload the zsh configuration:\nsource ~/.zshrc kcli version The generated container alias mounts ~/.kcli and ~/.ssh into the kcli container. That allows it to read the provider configuration and use the same SSH key as the Mac.\nIf both Docker and Podman are installed, the current installer prefers Podman. Make sure its virtual machine is running before calling kcli.\nCreate a Proxmox API token In the Proxmox web interface, open:\nDatacenter → Permissions → API Tokens → Add\nUse these values for an initial home-lab setup:\nField Value User root@pam Token ID kcli Privilege Separation Unchecked Proxmox displays the token secret only once. Copy it immediately and store it securely.\nDisabling privilege separation lets the token inherit the user\u0026rsquo;s permissions, which is convenient for proving the setup but gives the token broad access. For a long-lived or shared environment, enable privilege separation and assign a dedicated role and ACL with only the permissions kcli needs.\nConfigure passwordless SSH Create an SSH key if the Mac does not already have one:\nssh-keygen -t ed25519 Copy the public key to the Proxmox host:\nssh-copy-id root@\u0026lt;PVE_HOST_OR_IP\u0026gt; Then verify that SSH no longer prompts for a password:\nssh root@\u0026lt;PVE_HOST_OR_IP\u0026gt; hostname The command must return the Proxmox node name without asking for a password. This is mandatory because kcli runs image staging commands and transfers cloud-init files over SSH rather than through the API.\nEnable the required storage content types In the Proxmox web interface, open:\nDatacenter → Storage → local → Edit → Content\nEnable:\nISO image Import Snippets Snippets is especially important for cloud-init user data. Without it, a VM can start successfully while missing its injected SSH key, hostname, and other initialization settings.\nIn this guide:\nlocal-lvm stores VM disks. local provides kcli\u0026rsquo;s temporary import staging path and stores snippets. \u0026lt;PVE_IMAGE_STORAGE\u0026gt; stores the reusable cloud-image template. Change those names if your Proxmox storage is configured differently.\nThe target image storage must support the Proxmox images content type and appear in:\nkcli list pool A storage can exist in pvesm status but remain invisible to kcli when it does not support VM images.\nConfigure the Proxmox provider kcli reads its configuration from ~/.kcli. Create the directory:\nmkdir -p ~/.kcli Create ~/.kcli/config.yml:\ndefault: client: homelab pool: local-lvm numcpus: 2 memory: 2048 homelab: type: proxmox host: \u0026lt;PVE_HOST_OR_IP\u0026gt; user: root@pam auth_token_name: kcli auth_token_secret: ?secret pool: local-lvm imagepool: \u0026lt;PVE_IMAGE_STORAGE\u0026gt; node: \u0026lt;PVE_NODE_NAME\u0026gt; filtertag: kcli verify_ssl: false Use only the Proxmox hostname or IP in host. kcli constructs the API URL, so do not enter https://, port 8006, or /api2/json.\nCreate ~/.kcli/secrets.yml with the real token:\nhomelab: auth_token_secret: \u0026lt;PVE_TOKEN_SECRET\u0026gt; Restrict access to the secrets file:\nchmod 600 ~/.kcli/secrets.yml The ?secret value tells kcli to look up the real value from the matching YAML path in secrets.yml. This keeps the token out of the main configuration file, which is much easier to copy or accidentally commit.\nProxmox provider settings Parameter Purpose host Proxmox hostname or IP, without a URL or port user Proxmox user and realm, such as root@pam auth_token_name Token ID only, such as kcli auth_token_secret Token secret, resolved from secrets.yml here pool Proxmox storage used for VM disks imagepool Storage used for cloud images and ISOs node Target node in a Proxmox cluster filtertag Limits normal kcli VM discovery to resources with this tag verify_ssl Enables or disables TLS certificate verification If a credential is written elsewhere as root@pam!kcli, split it across the two kcli fields:\nuser: root@pam auth_token_name: kcli Passing the entire root@pam!kcli value as auth_token_name results in an authentication failure.\nFor a lab with the default self-signed Proxmox certificate, verify_ssl: false is practical. A better long-term setup is to install a trusted certificate and change it to true.\nKeep kcli away from existing VMs I set:\nfiltertag: kcli With this filter, normal kcli VM listing and management is scoped to VMs carrying the kcli tag. Existing workloads that were not created and tagged by kcli stay out of its default view.\nThis is a useful safety boundary, but it does not replace careful review of destructive commands. Always confirm names and the selected provider before deleting a VM or plan.\nVerify the connection Run:\nkcli list vm A VM table confirms that API authentication works. If filtertag is configured and no kcli-managed VMs exist yet, an empty table is the expected result.\nYou can explicitly select the provider:\nkcli -C homelab list vm Download a cloud image List the images kcli knows how to download:\nkcli list available-images Download a CentOS Stream 9 cloud image:\nkcli download image centos9stream With the current Proxmox provider, kcli opens an SSH session and runs the image download on the Proxmox host. It stages the file under local, imports it into imagepool, and then removes the staging file. If this step fails after kcli list vm succeeded, investigate SSH access, free space on local, and the target storage\u0026rsquo;s images capability first.\nCreate the first VM Create a VM using the CPU and memory defaults from config.yml:\nkcli create vm -i centos9stream testvm kcli injects the local SSH public key through cloud-init.\nTo override the defaults:\nkcli create vm -i centos9stream \\ -P memory=4096 \\ -P numcpus=4 \\ -P \u0026#39;disks=[30]\u0026#39; \\ -P \u0026#39;nets=[vmbr0]\u0026#39; \\ testvm The quotes around list parameters prevent zsh from treating the square brackets as filename patterns.\nInspect the VM and connect:\nkcli info vm testvm kcli ssh testvm kcli infers the default SSH user from the cloud-image name. Use a recognizable image name such as centos9stream so the provider can choose the expected user.\nUseful VM commands Task Command List VMs kcli list vm Show VM details kcli info vm testvm Start a VM kcli start vm testvm Stop a VM kcli stop vm testvm Connect over SSH kcli ssh testvm Add a 5 GB disk kcli create vm-disk -s 5 testvm Change memory kcli update vm -P memory=4096 testvm Clone a VM kcli clone -b testvm testvm2 Create a snapshot kcli create snapshot vm -n testvm snap1 Delete a VM kcli delete vm testvm Most commands also accept -C homelab when you want to make the target provider explicit.\nCreate a reusable profile Profiles collect image and hardware settings so they do not have to be repeated on every command. Create ~/.kcli/profiles.yml:\nmycentos: image: centos9stream numcpus: 4 memory: 4096 disks: - size: 30 nets: - vmbr0 cmds: - dnf -y install vim git Create a VM from the profile:\nkcli create vm -p mycentos myvm Define multiple VMs in a plan A plan declares multiple VMs in one YAML file. For example, create myplan.yml:\nparameters: image: centos9stream memory: 4096 web01: image: \u0026#34;{{ image }}\u0026#34; memory: \u0026#34;{{ memory }}\u0026#34; numcpus: 2 cmds: - dnf -y install httpd - systemctl enable --now httpd db01: image: \u0026#34;{{ image }}\u0026#34; memory: \u0026#34;{{ memory }}\u0026#34; numcpus: 4 disks: - size: 50 Create the plan:\nkcli create plan -f myplan.yml myplan Override a parameter at runtime:\nkcli create plan -f myplan.yml -P memory=8192 myplan Delete the plan and its managed resources when they are no longer needed:\nkcli delete plan myplan Review the plan and target provider carefully before running the delete command.\nTroubleshooting 401 Unauthorized: Authentication failed! Check these items in order:\nauth_token_name contains only the Token ID, not root@pam!kcli. The token exists under the same Proxmox user shown in config.yml. Privilege separation is disabled, or the token has an explicit ACL. The token secret is under the matching client and key in secrets.yml. Test the token directly:\ncurl -k \\ -H \u0026#34;Authorization: PVEAPIToken=root@pam!kcli=\u0026lt;PVE_TOKEN_SECRET\u0026gt;\u0026#34; \\ \u0026#34;https://\u0026lt;PVE_HOST_OR_IP\u0026gt;:8006/api2/json/cluster/resources?type=vm\u0026#34; JSON output means the token works and the likely issue is in the kcli YAML. Another 401 points back to the token or its permissions.\nBe careful with shell history when testing a real secret on the command line. Remove the command from history afterward, or load the secret from a protected file instead.\nkcli list vm works but image download fails Verify passwordless SSH from the Mac to the Proxmox host. Confirm that the kcli container can read the expected key from ~/.ssh. Confirm that imagepool exactly matches the Proxmox storage name. Confirm that imagepool appears in kcli list pool and supports images. Confirm that local has enough free space for the temporary download. The VM starts but SSH is refused This usually means cloud-init did not apply:\nConfirm that Snippets is enabled on the selected storage. Use a genuine cloud image rather than a normal installation ISO. Inspect the VM\u0026rsquo;s cloud-init drive and settings in Proxmox. Confirm that the public key mounted into the kcli container is the one you expect. kcli reports a missing Python module The official kcli container includes provider dependencies. If a locally installed Python version reports a missing module, install the Proxmox extra or inject proxmoxer and requests into that same Python environment. Do not install them into a different interpreter and expect kcli to find them.\nPodman cannot connect Confirm that its macOS virtual machine is running:\npodman machine list podman machine start Final result At this point the workflow is:\nkcli download image centos9stream kcli create vm -i centos9stream testvm kcli ssh testvm That turns my Mac into a lightweight control point for the Proxmox home lab. I can keep reusable profiles and plans in Git while the API token stays protected in ~/.kcli/secrets.yml.\nThe next step is deploying a single-node k3s Kubernetes cluster with kcli.\nReferences kcli documentation kcli source and installer kcli plan samples Complete kcli parameter sample Proxmox VE user and token management ","permalink":"https://blog.aymantech.net/posts/install-kcli-macos-proxmox/","summary":"A practical guide to running kcli on a Mac, configuring Proxmox API and SSH access, and creating the first VM from a cloud image.","title":"Install kcli on macOS and Manage Proxmox VMs"},{"content":"I’m Mohamed Ayman, a solutions-focused infrastructure engineer based in the Netherlands. I have spent more than a decade working across Linux and Unix systems, cloud platforms, infrastructure automation, enterprise delivery, and the difficult escalations that reveal how systems really behave.\nMy work sits between architecture and implementation. I enjoy turning unclear requirements into a design that teams can deploy, operate, troubleshoot, and explain. Today my strongest tools are Terraform, Ansible/AAP, Kubernetes, OpenShift, and Linux, across AWS, Azure, Google Cloud, and Oracle Cloud.\nWhat I care about Architecture that acknowledges operational constraints and failure modes. Automation that is readable, reviewable, and safe to change. Clear technical communication for customers and engineering teams. Hands-on learning through labs, experiments, and public notes. Leaving behind useful runbooks, patterns, and shared understanding. Why this site exists This site is where I document practical lessons from certification work, home labs, platform engineering, and solution design. The aim is to publish the context and reasoning that short command snippets usually leave out.\nFor the full career history and certifications, see my CV. You can also find me on GitHub and LinkedIn, or email me at mohamedayman@hotmail.com.\n","permalink":"https://blog.aymantech.net/about/","summary":"\u003cp\u003eI’m \u003cstrong\u003eMohamed Ayman\u003c/strong\u003e, a solutions-focused infrastructure engineer based in the Netherlands. I have spent more than a decade working across Linux and Unix systems, cloud platforms, infrastructure automation, enterprise delivery, and the difficult escalations that reveal how systems really behave.\u003c/p\u003e\n\u003cp\u003eMy work sits between architecture and implementation. I enjoy turning unclear requirements into a design that teams can deploy, operate, troubleshoot, and explain. Today my strongest tools are \u003cstrong\u003eTerraform, Ansible/AAP, Kubernetes, OpenShift, and Linux\u003c/strong\u003e, across AWS, Azure, Google Cloud, and Oracle Cloud.\u003c/p\u003e","title":"About"},{"content":"I want to share the study habits and small tactical choices that worked for me while preparing for the Red Hat Certified Specialist in Containers (EX188) exam. This is a hands-on, performance-based exam focused on running, building, and managing containers on Red Hat Enterprise Linux with Podman — no OpenShift, no Kubernetes, just the container runtime fundamentals.\nNote: this post contains no exam content — no questions, no tasks, no specifics of what came up on the day. Everything referenced here is publicly available from Red Hat\u0026rsquo;s exam page.\nTake the Official Objectives Literally Before doing anything else, read Red Hat\u0026rsquo;s published objectives and treat them as the source of truth:\nOfficial EX188 exam page The scope is genuinely tight: image lifecycle, running and managing containers, storage, basic networking, and systemd integration. If a topic isn\u0026rsquo;t listed, don\u0026rsquo;t chase it. It\u0026rsquo;s easy to fall down orchestrator rabbit holes when this exam deliberately isn\u0026rsquo;t about that.\nBuild a Lab You Actually Live In The single biggest thing that helped me was a RHEL 9 VM I used every day, not a scratch environment I only opened during study sessions.\nSpin up a RHEL 9 VM in your home lab — Proxmox, KVM, VirtualBox, whatever you already run. Register it with a free Red Hat Developer subscription so dnf behaves normally. Use it as your everyday sandbox. When you need to test anything container-adjacent — for work, for a blog post, for curiosity — do it there. Familiarity with the environment is quiet but powerful. On exam day you don\u0026rsquo;t want to be surprised by anything about how the OS feels.\nRootless and Rootful Aren\u0026rsquo;t the Same Podman Rootless Podman behaves differently from rootful — user namespaces, storage locations, port ranges, and networking all shift. Practise both, but spend real time on rootless. That\u0026rsquo;s where the interesting quirks live, and it\u0026rsquo;s the direction Red Hat is steering people toward.\nIf you can\u0026rsquo;t explain, off the top of your head, why a rootless container can\u0026rsquo;t bind to port 80 by default, spend an evening on that before booking the exam.\nBuild Podman Muscle Memory The exam is time-boxed and hands-on. Grepping through man pages for basic flags will burn time you don\u0026rsquo;t have.\nPractise until these are automatic, no thinking required:\nPulling, tagging, inspecting, and removing images. Running containers with the right combination of volume mounts, port publishing, environment variables, and restart policy. Writing a Containerfile and understanding what each instruction does at build time. Reading and following container logs. You don\u0026rsquo;t need to memorise every flag. You do need the common workflows to feel like typing your own name.\nRead the Docs, Don\u0026rsquo;t Memorise the Tutorial It\u0026rsquo;s tempting to memorise \u0026ldquo;the command that mounts a volume\u0026rdquo; without understanding what actually happens on disk. That approach holds up until a scenario shifts one step off the tutorial path — and then it collapses.\nSlow down and read the primary sources:\nPodman documentation Building, running, and managing containers (RHEL 9) The RHEL 9 container guide is excellent and free. The more time you spend there, the less any exam scenario feels unfamiliar.\nInvest Extra Time in systemd Integration Running containers as systemd units is one of the more nuanced parts of the Podman story, and it maps directly to how you\u0026rsquo;d actually deploy things in the real world.\nPractise generating unit files with podman generate systemd and read every section of the output. Learn Quadlet as well — it\u0026rsquo;s the modern path and where Red Hat is heading. Get comfortable with systemctl --user for rootless services, and know where user unit files live. Exam-Day Habits That Saved Me Time Read every task fully before starting. No skimming. A missed constraint at the top costs you five minutes at the bottom. Move on when stuck. If a task isn\u0026rsquo;t clicking after a few minutes, skip it and come back. Points are additive, not sequential. Verify what you built. The exam grades outcomes, not typing. Start the container, hit the port, confirm the unit is enabled and running — a clean-exit command isn\u0026rsquo;t proof. Keep an eye on the clock but don\u0026rsquo;t panic. The time budget is generous if you aren\u0026rsquo;t fighting the CLI. Resources That Actually Helped Red Hat\u0026rsquo;s Container, Kubernetes and OpenShift Fundamentals (DO188) course — the training paired with EX188. Building, running, and managing containers (RHEL 9) — the best free resource, full stop. Podman documentation. Red Hat Developer subscription — free RHEL for your lab. Closing Thought Treat EX188 as a fundamentals exam, not a hard one. Red Hat is deliberately setting up a shared container base before you specialise into OpenShift or Kubernetes. Get the fundamentals fully automatic here and every downstream exam gets easier.\nNext stop for me: preparing for the EX280 OpenShift administration exam.\n","permalink":"https://blog.aymantech.net/posts/tips-passing-ex188/","summary":"A hands-on preparation guide for EX188 covering Podman muscle memory, rootless containers, systemd integration, and exam-day habits.","title":"Tips for Passing the EX188: Red Hat Certified Specialist in Containers"},{"content":"It\u0026rsquo;s official — I passed the EX374 exam, the Red Hat Certified Specialist in Developing Automation with Ansible Automation Platform. This one is the natural next step after my earlier Road to EX374 post, and getting there took a lot of hands-on lab time rather than reading alone.\nI won\u0026rsquo;t share any exam questions or specifics — that\u0026rsquo;s against the Red Hat exam agreement, and it wouldn\u0026rsquo;t help you anyway. What I can do is point at the areas that rewarded real practice, so you spend your prep time where it counts.\nLive entirely inside ansible-navigator If you\u0026rsquo;re still reaching for ansible-playbook, retrain that muscle now. Get comfortable running, inspecting, and debugging automation through ansible-navigator — stdout mode, interactive mode, :doc for module lookups, and reviewing playbook artifacts. It should feel like second nature well before exam day.\nKnow Execution Environments cold Execution environments are central to the modern platform, so don\u0026rsquo;t treat them as an afterthought. Practice building them with ansible-builder, understand the definition file, pin your collections and Python dependencies deliberately, and be able to reason about why a playbook behaves differently inside one EE versus another. Being able to build, tag, and actually use an EE end-to-end matters more than memorizing flags.\nGet fluent with Automation Controller Spend real time in the Automation Controller UI and its objects: projects, inventories, credentials, job templates, and workflows. Know how a project syncs content from Git, how credentials are injected, and how surveys and variables flow through. Clicking through it a few times isn\u0026rsquo;t enough — build the muscle memory of wiring these pieces together from scratch.\nTreat Git as a first-class skill Your automation content lives in version control, so the Git workflow isn\u0026rsquo;t a side topic — it\u0026rsquo;s part of the job. Be smooth with branching, committing, and pushing, and understand how the controller pulls from a branch. If Git ever slows you down, that\u0026rsquo;s friction you don\u0026rsquo;t want under time pressure.\nWrite content the \u0026ldquo;professional\u0026rdquo; way The exam is about developing automation, not just running it. Practice writing modular, reusable playbooks and roles, using variables and templates cleanly, and structuring content so it\u0026rsquo;s maintainable. Sloppy-but-working isn\u0026rsquo;t the target — think about how you\u0026rsquo;d hand this off to a teammate.\nGeneral exam-day habits Read every task fully before touching anything. Understand what \u0026ldquo;done\u0026rdquo; looks like first. Make your work persistent. Verify that what you build actually survives and behaves as required, not just that it ran once. Watch the clock, but don\u0026rsquo;t rush the setup. A correct execution environment or controller object early saves you from cascading failures later. Lab, lab, lab. Nothing here is theoretical — every objective rewards having done it with your own hands. My setup I did all of my prep in a containerized AAP 2.6 deployment on a Proxmox-based home lab. Building that environment myself — subscription manifest, registry service account, the works — turned out to be some of the best preparation, because it forced me to understand how the platform fits together rather than just how to click through it.\nOn to the next one. If you\u0026rsquo;re on this path, keep your hands on the keyboard and your content in Git.\n","permalink":"https://blog.aymantech.net/posts/passed-ex374/","summary":"The practical areas that mattered most in my EX374 preparation, from execution environments and Automation Controller to Git and reusable content.","title":"I Passed EX374: What to Focus On for the Ansible Automation Platform Exam"},{"content":"Solutions Architect · DevOps \u0026amp; Cloud Infrastructure Download printable PDF · Diemen, Netherlands · mohamedayman@hotmail.com · LinkedIn · GitHub\nProfile Solutions-focused infrastructure engineer with 10+ years across Linux/Unix, infrastructure automation, and enterprise cloud delivery. I pair deep, hands-on product knowledge — earned architecting deployments and owning the hardest escalations — with the ability to design solutions and translate customer needs into working architecture. Strongest with Linux, Ansible/AAP, Terraform, OpenShift, and Kubernetes across AWS, Azure, GCP, and Oracle Cloud. I\u0026rsquo;ve architected multi-cloud platforms, led on-prem-to-cloud migrations, and served as the trusted technical authority for enterprise customers on the products I support. Red Hat Certified Architect (RHCA) working toward a Solutions Architect path.\nHighlights Architected and automated a multi-cloud Terraform Enterprise platform across AWS, Azure, and GCP at IBM. Trusted technical authority for enterprise customers on Terraform Enterprise, Bitbucket, and Bamboo — advising on architecture, best practice, and product direction. Led on-prem → Oracle Cloud migrations across EMEA, bringing Terraform and Ansible into delivery. Implemented enterprise engineered systems — Exadata, SuperCluster, PCA, ZFSSA — at customers across the region. Red Hat Certified Architect (RHCA) plus Terraform, OpenShift, Kubernetes, and Ansible/AAP (EX374) specialist certifications. Multi-cloud, hands-on across AWS, GCP, Azure, and OCI. Core Skills Cloud Platforms AWS · Google Cloud Platform · Microsoft Azure · Oracle Cloud Infrastructure\nSolution Design \u0026amp; Delivery Solution architecture · Multi-cloud design · Migration planning \u0026amp; delivery · Technical pre-sales \u0026amp; solution scoping · Customer enablement · Requirements → architecture translation\nInfrastructure as Code \u0026amp; Automation Terraform · Terraform Enterprise · Ansible · Ansible Automation Platform (AAP) · Execution Environments · Shell scripting · GitOps\nContainers \u0026amp; Orchestration Kubernetes · Docker · OpenShift\nCI/CD \u0026amp; DevTools GitHub Actions · Bitbucket · Bamboo · Continuous integration \u0026amp; delivery pipelines\nOperating Systems Red Hat Enterprise Linux · CentOS · Ubuntu · Fedora · Solaris · AIX · HP-UX\nNetworking \u0026amp; Monitoring TCP/IP · MPLS · Cisco routing \u0026amp; switching · HP OpenView · Oracle OpsCenter\nWays of Working Agile delivery · Incident \u0026amp; escalation management · Technical mentoring · Documentation \u0026amp; KB authoring\nExperience Senior Terraform Support Engineer — IBM October 2025 – Present · Amsterdam\nArchitected and automated the multi-cloud deployment of Terraform Enterprise across AWS, Azure, and GCP. Primary technical authority for key enterprise clients on Terraform Enterprise infrastructure and architecture. Advise customers on deployment design and best practice while keeping internal runbooks current. Sustained an NPS above 85% as a measure of customer trust. Senior DevTools Support Engineer — Atlassian March 2022 – September 2025 · Amsterdam Promoted from DevTools Support Engineer (Mar 2023)\nDeep product authority across Bitbucket and Bamboo as a dual-product engineer, owning the most complex, high-impact enterprise cases. Acted as internal escalation engineer, partnering with Tier 3 and Engineering to resolve critical architectural issues. Advised customers on CI/CD pipeline design and DevTools best practice. Built training material, internal docs, and customer-facing KB articles to scale product knowledge across teams. Fed recurring customer insight back into product and problem-management to improve stability and reduce future issues. Participated in 24/7 on-call rotations for critical customer issues. Principal Technical Support Engineer — Dell RSA January 2021 – February 2022 · Cairo\nOwned the highest-severity escalations end to end; ran root-cause analysis and designed preventive fixes. Validated new product patches and features in lab before release. Contributed to global technical gap analysis and mentored junior engineers. Partnered with account teams on technical pre- and post-sales engineering. Senior Delivery / Implementation Engineer — Oracle (Advanced Customer Support) November 2017 – December 2020 · Egypt\nLed several on-prem → Oracle Cloud migration projects from design through delivery. Brought DevOps tooling into ACS delivery — Terraform, Ansible, and automation scripts. Implemented Exadata, SuperCluster, PCA, and ZFSSA at customers across the EMEA region. Identified automation and cloud-service opportunities aligned to customer roadmaps, scoping the right technical fit. Improved delivery margin by shortening project timelines without compromising quality. Senior Systems Engineer — Vodafone Egypt August 2014 – November 2017 Promoted from Systems Engineer (Jan 2016)\nDesigned and ran virtualised environments on Oracle T5 and IBM Power7 with HA at both VM and physical layer. Tuned Solaris/RHEL + Oracle DB performance with HP OpenView and OpsCenter; applied ZFS pool/filesystem best practices. Led UNIX-side delivery of Billing, CRM provisioning, WAS, IVRDB upgrades, DR drills, and online SAN/storage migrations (CX3 → VNX). Supported 200+ servers in a 24x7 data center on RHEL, Sun Solaris, and Exadata. Hands-on hardware ops alongside software work: part replacement, server installations, and cabling. Linux System Administrator — 4GTSS March 2013 – August 2014 · Cairo\nAdministered Number Portability (NPG) systems across RHEL, Ubuntu, Fedora, CentOS, and Solaris in six countries (Bahrain, Qatar, Kuwait, Sudan, Tunisia, Tanzania). Worked across Django, Celery, RabbitMQ, and MySQL Galera. Network Management Solutions Engineer — Orange January 2012 – March 2013\nMigrated multi-vendor servers, routers, and switches from World Trade Center to the October site. Worked across Alcatel Evolution/G2 and Motorola BSCs over IP/MPLS and ATM core networks. Supported a 12-site interconnect spanning RXCDRs, soft switches, MSCs, BSCs, HLRs, and OMCs. Selected Projects Containerized AAP 2.6 home lab — Built an all-in-one containerized Ansible Automation Platform deployment on a Proxmox lab (subscription manifest, private registry service account, execution environments) as a live environment for developing and testing automation. This site — A Hugo blog deployed to GitHub Pages via GitHub Actions, documenting hands-on DevOps and certification work. moabdelbaset.github.io Public engineering board — Track work and learning in the open using GitHub Projects, seeded programmatically. Education MBA, Digital Transformation — Ecole Supérieure Libre des Sciences Commerciales (ESLSCA), Paris · 2023 B.Sc., Electrical, Electronics \u0026amp; Communications Engineering — Arab Academy for Science and Technology, Cairo · 2006 – 2011 Certifications Infrastructure as Code \u0026amp; Automation\nRed Hat Certified Specialist in Developing Automation with Ansible Automation Platform (EX374) · 2026 HashiCorp Certified: Terraform Associate Red Hat Certified Specialist in Automation with Ansible Kubernetes, Containers \u0026amp; Orchestration\nRed Hat Certified Specialist in OpenShift Administration Red Hat Certified Specialist in Containers and Kubernetes Docker Certified Associate Red Hat / Linux\nRed Hat Certified Architect (RHCA) Red Hat Certified Engineer (RHCE) Red Hat Certified System Administrator (2023) Red Hat Certified Virtualization Administrator Red Hat Certificate of Expertise — System Monitoring \u0026amp; Performance Tuning Red Hat Certificate of Expertise — Clustering \u0026amp; Storage Management Red Hat Certificate of Expertise — Server Hardening Red Hat Certificate of Expertise — Deployment \u0026amp; System Management Cloud\nAWS Certified Solutions Architect – Associate Oracle Cloud Infrastructure 2018 Certified Associate Architect Oracle Cloud Infrastructure Classic 2018 Certified Associate Architect Oracle Cloud Infrastructure 2019 Cloud Operations Associate Virtualization \u0026amp; Storage\nVMware Certified Professional 6.5 — Data Center Virtualization Veritas Certified Storage Foundation for Unix 6.1 Administration Oracle Solaris 11 Advanced System Administrator Oracle Solaris 11 System Administrator Networking \u0026amp; Service Management\nCisco Certified Network Associate (CCNA) Cisco Certified Network Associate Security (CCNA Security) ITIL v3 Foundation Databases\nMongoDB Certified DBA Associate Languages English — Full professional Dutch — Limited working ","permalink":"https://blog.aymantech.net/cv/","summary":"\u003ch2 id=\"solutions-architect--devops--cloud-infrastructure\"\u003eSolutions Architect · DevOps \u0026amp; Cloud Infrastructure\u003c/h2\u003e\n\u003cp\u003e\u003cstrong\u003e\u003ca href=\"/mohamed-ayman-cv.pdf\"\u003eDownload printable PDF\u003c/a\u003e\u003c/strong\u003e  ·  Diemen, Netherlands  ·  \u003ca href=\"mailto:mohamedayman@hotmail.com\"\u003emohamedayman@hotmail.com\u003c/a\u003e  ·  \u003ca href=\"https://www.linkedin.com/in/mohamed-abdelbasset-b29b5b66/\"\u003eLinkedIn\u003c/a\u003e  ·  \u003ca href=\"https://github.com/moabdelbasset\"\u003eGitHub\u003c/a\u003e\u003c/p\u003e\n\u003chr\u003e\n\u003ch2 id=\"profile\"\u003eProfile\u003c/h2\u003e\n\u003cp\u003eSolutions-focused infrastructure engineer with \u003cstrong\u003e10+ years\u003c/strong\u003e across Linux/Unix, infrastructure automation, and enterprise cloud delivery. I pair deep, hands-on product knowledge — earned architecting deployments and owning the hardest escalations — with the ability to design solutions and translate customer needs into working architecture. Strongest with \u003cstrong\u003eLinux, Ansible/AAP, Terraform, OpenShift, and Kubernetes\u003c/strong\u003e across \u003cstrong\u003eAWS, Azure, GCP, and Oracle Cloud\u003c/strong\u003e. I\u0026rsquo;ve architected multi-cloud platforms, led on-prem-to-cloud migrations, and served as the trusted technical authority for enterprise customers on the products I support. \u003cstrong\u003eRed Hat Certified Architect (RHCA)\u003c/strong\u003e working toward a Solutions Architect path.\u003c/p\u003e","title":"Mohamed Ayman Abdelbaset — CV"},{"content":"I have officially started my preparation for the Red Hat Certified Specialist in Developing Automation with Ansible Automation Platform (EX374) exam. This exam is a deep dive into the modern Ansible ecosystem, focusing on execution environments, automation controller, and professional content development.\nOfficial Exam Objectives To keep my study focused, I am following the official Red Hat objectives closely:\nOfficial EX374 Exam Objectives My Lab Setup \u0026amp; Installation For my lab environment, I am using a containerized installation of Ansible Automation Platform (AAP) 2.6. Setting this up correctly is the first hurdle, and I found an excellent guide that covers the \u0026ldquo;All-In-One\u0026rdquo; containerized deployment.\nKey Resource: Installing AAP 2.6 via Containerized Version by Waldirio Pinheiro. Note: This video was instrumental in getting my Proxmox-based lab up and running, especially the parts about configuring the subscription manifest and setting up the service account for the registry. Study \u0026amp; Practice Sessions I am also following a dedicated series of practice sessions that walk through each objective. It’s one thing to read the documentation, but seeing the \u0026ldquo;GitOps\u0026rdquo; workflow and branch management in action is much more effective.\nStudy Playlist: Red Hat EX374 Practice Sessions by Eddie Jennings. I am starting with the \u0026ldquo;Understand and use Git\u0026rdquo; section to ensure my automation content is version-controlled and ready for CI/CD pipelines. Current Progress Currently working through:\nDeveloping Content: Writing modular and reusable playbooks. Execution Environments: Building and managing containerized environments for automation. Ansible Navigator: Moving away from ansible-playbook to the modern ansible-navigator CLI. Stay tuned for more deep dives into my lab challenges!\n","permalink":"https://blog.aymantech.net/posts/preparing-for-ex374/","summary":"My hands-on EX374 preparation setup: a containerized AAP 2.6 lab, Git-based content, execution environments, and Ansible Navigator.","title":"Road to EX374: Developing Automation with Ansible"}]