A Tale of AI Betrayal & Snapshotting Victory!

At the beginning of this year, I decided to leave the quiet comforts of the AUR, Pacman and my purpose built, minimal system for a bit of distro-hopping. I've been using Linux full time for about five years now. That journey began with, and has maintained Arch Linux as a constant for it's duration. So, as any readers familiar with the ins-and-outs of Linux distributions might have guessed, I spent a bit of time playing around with Ubuntu, found out that I quite liked Fedora, but by the beginning of March, was ready to return to good ol' Arch Linux. To keep things interesting (and because I was a bit pressed for time), I opted to test drive Cachy OS, a relatively new Arch derivative that boasts some interesting features like a custom kernel tuned with the EEVDF scheduler. This is how I ended up inadvertently setting up my first Linux box that featured automatic Snapshotting, a fact that will become relevant later.

Now for an embarrassing confession: At the beginning of this year, I also began to toy around with Artificial Intelligence a good bit. I never enjoyed using ChatGPT (my drug of choice) for anything big, anytime I did something like set-up a website with a new tool, or install Arch Linux I'd reach for proper documentation or Wikis. But I did develop a habit of using ChatGPT to find quick fixes when I was in a pinch, and I did have it generate one YouTube thumbnail for me (I'm now refusing to change the thumbnail so I'll be reminded of how dumb an idea that was every-time I see the image on my website). The disturbing truth is that I was never even convinced of AI's superior intelligence. I never even thought of GPT as a better Google, as many less sycophantic users claim it to be. Very early on in my usage, I found that these tools couldn't write, were prone to hallucinations, and couldn't be trusted for anything mission critical (unless that mission critical thing is choosing a target for missile strikes). Unfortunately, I also just happened to notice that, at least when it came to small Linux eccentricis or low-level coding questions, GPT and other models were pretty good at providing a good 'nough style of fix, so long as the questions were well documented elsewhere on the internet and GPT could steal the answer from Stack Overflow or others.

So a few weeks ago when I finally made the shift to a Wayland window manger (better late than never), and started toying around with Hyprland, I ended up using the GPT more than I really care to admit. To be clear, Hyprland's documentation is really solid, and I did rely on it quite heavily as well, but for anything that was an abstraction on top of Hyprland I was running to the AI for quick fixes that I knew I'd have to redo later when I learned more and realized that I'd fixed my problem in the dumbest possible way. This was precisely the kind of fix I was looking for the first time that I took my laptop to work and, upon finding my battery already run down to 17%, realized that the hypridle config I had set up was clearly not working as intended. So I ran to the AI overlords begging for a quick fix that could let me solve the problem and get on with what I had intended to use my laptop for.

Almost immediately I hit a wall with ChatGPT: Despite the fact that I had mentioned I was running Cachy OS with the default options selected during install, even specifically stating that I was using the Limine boot manager, I kept getting nonsensical answers that suggested I make changes to GRUB andior Systemd configs. But I was committed to the lazy solution, and after a few minutes of back and forth I was finally given an answer that sounded reasonable enough: my laptop's Nvidia GPU, and the dreaded proprietary drivers I have to run so I can use Davinci Resolve were the culprit (every Linux users favorite nemesis). As is ChatGPT's way, it also provided me with a simple one line fix to paste into my Limine config:

cmdline: root=UUID=xxxxx rw quiet splash nvidia.NVreg_PreserveVideoMemoryAllocations=1 nvidia_drm.modeset=1

Having never actually edited this Limine config before, I had a few follow up questions for ChatGPT. Specifically I was unsure whether all of these options needed to go on one cmdline:, or if I should separate out the PreserveVideoMemoryAllocations=1 to it's own line. While asking this I ended up actually copy & pasting into Chat GPT my current Limine cmdline:, including the proper UUID identifier for my SSD:

cmdline: quiet nowatchdog splash rw rootflags=subvol=/0 root=UUID=8b17bdfb-efb4-41a-852342

It was at this point that ChatGPT suggested the most feral and unhinged thing I'd ever see in my days of relying on robots to do human work: Simply append GPT's recommendation onto the end of my existing Limine cmdline, repeating commands like rw, quiet and splash, and leaving in the boilerplate: root=UUID=xxxxx. I can't get the screen-shot because (!Spoilers!) I ended up deleting my ChatGPT account, but scout's honor, no BS, this is actually what GPT recommended I paste into my config:

cmdline: quiet nowatchdog splash rw rootflags=subvol=/0 root=UUID=8b17bdfb-efb4-41a-852342 root=UUID=xxxxx rw quiet splash nvidia.NVreg_PreserveVideoMemoryAllocations=1 nvidia_drm.modeset=1

One doesn't need to be a Linux expert to know just how stupid this suggestion was. I'll save you from all the excuses that I want to make to present myself as less stupid than I truly was in this moment, as the truth is that I put far too much trust in a fundamentally untrustworthy tool. Without even bothering to read the line that was so quickly offered up for my copy & pasting needs, I dropped the line into my Limine config and had typed reboot into a terminal before I had time to register a second thought. The end result? For the first time in five years of using Linux full time I manged to actually trigger a straight up Linux blue screen, a feature that I actually didn't even know existed. Full on kernel panic: Cachy OS had blocked out the root user, and refused to even try to boot my OS; not surprising as I had mapped out the UUID of my boot drive as: xxxxx.

But what then is a good little fake Arch-BTW user meant to do when the AI overlords brick their computer? We go running back to AI for the fix of course. To GPT's credit (I guess), when pressed a second time and told "WTF? My computer won't boot now!", it did pretty much immediately identify the problem as the fact that it had recommended the ludicrous cmdline than included a UUID path to nowhere. GPT quickly gave me instructions to fix the problem that included getting a Linux installer on a USB key, booting off of that, mounting my nvme drive and editing the Limine config to make the drive bootable again. But this solution was no good. I was stuck at work, alone in a guard shack with no USB drive, no way to download a Linux installer and unable to leave for another ten hours. No matter how desperately I pleaded with this serf of a demigod there was no way to solve my problem without the USB key and Linux installer that I didn't have. Fun fact: I actually did have a Tails OS USB in the glove box of my car (I always do), but I was quite sleep deprived and this fact did not enter my mind.

I threw my phone down on the desk in despair, and leaned back in my chair starring back at the mess I had created in front of me: The Limine boot menu with my boot drive selected, but the knowledge that if dared I press enter the system would just try and fail to boot once again.

That's when I noticed it: a submenu in the Cachy OS boot options labeled: Snapshots. As mentioned earlier, I wasn't even aware that Cachy OS turned on btrfs Snapshots by default during the installation process, but here was a potential solution starring back at me. I opened the submenu and noticed that there was a Snapshot available from only about ten minutes ago. Cachy OS automatically has btrfs create a snapshot every time a full system upgrade is triggered, and by mere coincidence I happened to have run the now life-saving sudo pacman -Syu command moments before starting to fiddle with the Limine config. I hit enter, and seconds later was met with the familiar Hyprland userspace I had built over the past few days. I quickly opened up the Limine config and found that it contained the garbled mess that I had pasted in minutes ago. I had found my solution: (a) fix the config, (b) reboot, (c) problem solved.

Like a good little drone though, I first opened up Firefox and ran the idea past my old pal ChatGPT. Sure it had screwed me once already tonight. But surely it would be able to tell me if this was the fix that I thought it would be? Once again I was hit with devastating news. ChatGPT swore up and down that this would not be the fix I was looking for. One couldn't just boot from a Snapshot, fix an error and reboot to solve a problem. GPT stuck to it's guns. The only way to solve my problem was to follow its original advice: boot off a USB, mount my boot drive and manually edit the config.

Something inside me stirred. What was this? A desire to thrust off the shackles placed on me by my over-reliance on this inhuman scourge on my nation's power and water resources? An inkling of courage? A returning ability to think independently? What was the point of a snapshot if I couldn't use it to fix config errors? Was this not the whole point of the feature in the first place? I gathered up all my free will and for the second fateful time this night typed those fateful letters into my terminal: reboot. Once again I was greeted by the Limine boot menu. I made sure that I was launching not from the snapshot of ten minutes ago, but instead from my system's boot drive. I waited, fingers crossed, breath held in anticipation until finally: success! My risky (not risky) endeavor was rewarded! Once again I could operate a system without the infernal advice of an AI agent spoon feeding me commands to paste into terminals! Free from my self-imposed, headless pillory, I was again my own man! After all is there a more masculine story than finding a way to solve problems of one's own design‽

A note: While the ultimate resolution of this story was 100% me setting up a DNS level block on ChatGPT's website; I do think I should say a quick word on why. As mentioned a few times here, AI agents are more than capable of providing quick and dirty fixes to some technical problems, and probably a good bit more if they are employed correctly. But I have come to the conclusion that the reward is simply not worth it. Not because these fixes are sub optimal (though they usually are), but rather, because there is great knowledge to be had through the process of trouble-shooting a process the old fashioned way. Much of what I have learned about any number of topics, but particularly about technical subjects has been learned because I found myself having to read more to solve problems. When troubleshooting this way it's far harder to find fixes that just work, without having any clue as to why they work. I'm not suggesting that this doesn't still happen from time to time, my personal GitHub repositories are sufficient proof that it does, but learning a solution for yourself, in my opinion, has a far better end result than relying on an AI agent to find one for you.

Also, in case anyone was wondering: GPT's initial diagnosis of my Nvidia GPU being the culprit was incorrect from the outset.