The Day Facebook Took Down a Magento Store
This story takes place in Montreal, Quebec, in November 2019. Back then, I was part of the emergency team of a company that was making a lot of e-commerce websites. And as I was just arriving to work, I noticed an unusual tension pooled around my desk.
The Site Is Down!
As I put my laptop down and wired it up, I asked what was going on. “Acme is down”, they told me. Well, they didn’t say “Acme”, they said the actual name of the website, but… You know, I can’t really divulge that here. Anyways, I asked if I could join in on the troubleshooting and they seemed enthused by the idea, they’d apparently been at it for a while already.
My first reflex in these cases is always to look at what the customers are seeing. Not only does it usually give me a lot of information on what’s actually going on, but it also allows me to confirm whether the problem actually disappears or not when I try a fix.
In this case? Infinite loading, gateway timeout. Right away, that tells me that:
It’s Down, Something’s Up
PHP hanging could mean multiple things. So I went through the usual suspects, one by one.
So I went to check the access logs. Usually, when a website hangs, it means there’s one place, one page that most people visit, that has a terrible line of code. So naturally, I expected finding a lot of requests hitting the same page. But curiously enough, that was not the case. Sure, there was a lot of activity on the website, but it was spread out, lots of different pages… I asked the client if they did anything that would warrant a big influx in traffic. They told me they added a discount, but nothing major.
I thought it could be a DDoS so I started using the sort command to count how many times each IP came up, but nothing out of the ordinary. I even cherry-picked several of them to check their origin, and they were all local, from Quebec. Definitely legit-looking traffic.
MySQL Server Has Gone Away
Then I thought about the database. If the database is overloaded, or if there is a huge query running, it could cause PHP to hang. So I asked the sysadmins.
Database was up, but… Not running. Overloaded enough that it had stopped processing queries, stuck in a kind of freeze response. Alright, we have a culprit! I asked if there was an obvious query that was causing it, and it was apparently not the case. Just normal looking queries, but lots of them!
That’s when we decided to clear the queue of queries by restarting the DB server. And just like that, the site was back! For… At least 4 seconds, before hanging again. And of course, it was the database again. I started thinking about it some more… Lots of different queries in the database… Lots of spread out requests in the access log… Then I realized I did not check how quickly the access log was getting hit.
So I went back to check the file. Each requests seemed to land in the same second than the previous one. I ran tail -f to see the logs getting filled in real time, and it was scrolling so fast I couldn’t even read. It really smelled like a DDoS 😠!
Tracking The Stink Bug
I wanted to poke it a bit to see how it would react. I asked the sysadmin to restart the DB server again. A few seconds later, site was back to being down. Again asked to restart it, and it took several seconds to crash that time. My goal was to check how consistent the Time To Crash was. In the case of a DDoS, you’d typically expect the website to crash again as quickly as possible after it’s back up. In this case, there was lag. And the sooner after a crash we restarted the server, the more lag there was.
It felt human.
Then, my goal was to actually browse the website to see if something was standing out. So I turned on maintenance mode and asked to restart the DB server once again. At that point, the sysadmin was probably thinking I had no idea what I was doing 😅. Browsing the website, I didn’t really see anything out of the ordinary. Homepage was working, categories were listing products, product page was fine, add to cart button was add-to-cart-ing…
I was confused. The DDoS smell was mostly gone, and I was running low on leads…
Facebook’s Confession
Someone sent a message in the crisis Slack group, pinging everyone. “Check Facebook”. They said to search for the brand’s name on Facebook. So we did.
There were so many posts from so many people saying to check the website out, giving links… And each post was also shared a lot, had a lot of comments… Looking more closely, every single post was talking about the same thing.
Everything is so cheap!
So I browsed the website again, this time paying closer attention to the prices. Sure enough, something was very wrong. I even felt embarrassed I didn’t notice it earlier. I blame developer tunnel vision.
Checking the Slack group again, everyone understood the issue. We asked the client if that was the not-out-of-the-ordinary discount they were talking about earlier, and they said that no, the one they added only targeted three products. On a lot of platforms, I would trust that discounts work as intended. I mean, I would still check, but I wouldn’t expect a problem.
But this is Magento 1 we are talking about.
Discount Denial of Service
In case you are lucky enough to not be familiar with Magento 1, just know that its admin panel has more tabs than my browser does when I’m trying to learn how to file my taxes. And it’s not just the hierarchy that’s bad. The UX of most forms in there is too.
Logging into the admin, I checked the ✨Magento 1 catalog price rules✨ (which is the specific term to talk about discounts applied on products, as opposed to discounts applied at cart-level). Filtering out the disabled and expired ones, I was able to find our suspect.
I looked at its configuration, and it was setting a fixed price, which matched what we were seeing on the website. I was totally expecting to see that it was not targeting any product, thus being applied on the entire catalog. But as I checked the conditions, I saw the three product IDs that the client mentioned. Hm 🤨.
Painting The Sheep Black
So the discount’s configuration was correct, but not respected. What in the world could cause such an odd behaviour?
Eventually, I thought it had to be an indexing issue. I reindexed the prices and discounts and checked. No luck. Oh, right, the cache. I reindexed again, cleared the caches, and checked. Still wrong.
How could discounts suddenly stop working on a website that had used them for years without issue? How could we have missed something this big? It sounded impossible!
Maybe it was impossible…
The Problem Was Not, Here
I felt like I was digging myself into a hole. Trying way too hard to find a bug where everything was working fine. The problem had to be somewhere else. I opened discount configuration again.
Looking closely at the conditions tab, I saw the three product IDs there, just like before. This time, I made sure to check that they were the exact ones the client mentioned. I also made sure that they were actual products, thinking maybe if they did not exist it could cause a bug… And as I kept staring at it, I noticed something.
“Not”
Finally, there it was. The issue was never an infinite loop, a DDoS, recent commits or price calculation. The problem, here, was “not”. The catalog price rule was configured to target every product with an ID that was 🫠not 🤦 in the list of the three product IDs I was seeing.
Every product in the catalog had this new very low fixed price, except the three products targeted. No wonder it went viral on Facebook! Products that were originally in the hundreds of dollars were now costing a tenth of that price, at most.
The Fix Is Nigh
Knowing the client had been waiting a long time for their site to come back online, I wanted to make sure the fix was working correctly. I opened the three products in three tabs, and several other products in several other tabs. Then, I fixed the discount rule, reindexed, and cleared the caches. I refreshed every tab I had previously opened and made sure that the prices changed, and were correct.
And correct they were! So I sent a message to the team, explaining what happened. The sysadmin scaled the infrastructure to be able to withstand the peak in traffic, and I disabled the maintenance mode. Customers came rushing back in, but quickly noticed that the very attractive prices they were originally seeking were gone.
In the end, this “bug” had a root cause that I always kept in mind going forward.
I learned not to skim through configuration.
The devil is in the details, and sometimes, a boolean is a server-shattering one.
