Another customer figured out everything except the problem
Intro
You know this guy, right? He is that person who wants so desperately to be right that in fact, he is wrong. A long time ago, when I started working in IT, I was an IT Support engineer. I received phone calls and emails and had to resolve hosting issues and support the firm’s clients.
I was working twelve hours shifts, one daytime, one nighttime, and two days off after this. It was an intense marathon, but it was also a learning opportunity. I communicated with all sorts of clients. All were incredible, some more than others.
I am starting this series as a small collection of stories from that period in my life(and not limited to that), and hopefully, you will enjoy reading them as much as I enjoyed living them 😀
The case
Middle of the night. I was doing maintenance operations on the servers, like patching, migrations, and so on. Phone rings.
Customer: Hello, Your servers are not reliable and not secure! You have a crappy service!
Me: (In my head: if I got a penny for each time I heard that, I would have been in the Bahamas, enjoying a luxurious life) Hello, what seems to be the problem?
Customer: YOUR service is the problem! The server was hacked!!!! Don’t you see any monitoring? Don’t you check anything there?
Me: Please give me a minute to check your domain and the server you are hosted on.
We had so many monitoring systems, and in front of my workstation, on the wall, I could see the status of all servers(four big monitors with dashboards), everything we had, and all was green. So, when I asked him to hold, I was sure the problem was not on the server side.
His website was defaced and showed some grotesque images on the front page. Just a side note for what defaced means:
“Web defacement is an attack in which malicious parties penetrate a website and replace content on the site with their own messages. The messages can convey a political or religious message, profanity or other inappropriate content that would embarrass website owners, or a notice that the website has been hacked by a specific hacker group.” – From here
Going deeper, I saw that the website was built on WordPress. The customer was using the cheapest service of shared hosting we had.
So no private server, no individual “VM”(no premium support), just shared hosting, with a limited amount of resources. Shared hosting means that the website(s) is/are encapsulated in a private space, along with other websites on the same physical server.
A shared hosting server would have gathered 30 to 50 private spaces (customers – each owning “a space”), the number depending on the resources allocated per client. Although the monitoring did not give me any concern, I randomly checked 2–3 other customers, from the same server, and they were ok.
Me: Thank you for your time. I investigated your issue, and it seems that your website was hacked. It is your responsibility to fix the code.
Customer: Exactly! I was hacked because you have poor security on the servers!
Me: Forgive me, but you were hacked because your website has poor security. Our servers are fine.
Customer: Impossible! I am using WORDPRESS! (he shouted WordPress)
Me: Did you use an official WordPress theme? What about plugins and modules? Are those official or bought under license from a trusted provider?
Customer: (angry tone) No. Found them online and installed them myself. But this is WordPress, so seriously, check again, the server.
Me: (I was starting to feel annoyed while opening developer tools on Chrome and Firefox side by side) If I show you the corrupt code, would that be sufficient for you to understand that this is your issue and not something from the security of the server? (Not waiting for an answer) Please hold the line for a couple of minutes.
Saw that the defaced content was coming from an outside source and then searched for that in the website files. I found out that a module had an ugly XSS embedded inside. For your reference:
“Cross-Site Scripting (XSS) attacks are a type of injection, in which malicious scripts are injected into otherwise benign and trusted websites. XSS attacks occur when an attacker uses a web application to send malicious code, generally in the form of a browser side script, to a different end user. Flaws that allow these attacks to succeed are quite widespread and occur anywhere a web application uses input from a user within the output it generates without validating or encoding it.
An attacker can use XSS to send a malicious script to an unsuspecting user. The end user’s browser has no way to know that the script should not be trusted, and will execute the script. Because it thinks the script came from a trusted source, the malicious script can access any cookies, session tokens, or other sensitive information retained by the browser and used with that site. These scripts can even rewrite the content of the HTML page.” – From here
So in plain English, the all-knowing security expert, the customer who called, downloaded a plugin/module from the Internet that, when activated by PHP, was rewriting his homepage.
I started showing him the details of my investigation, and while doing so, I opened his database and disabled that plugin. After that, the entire website was looking like before again. Feeling a bit sorry about shaking his confidence in WordPress, I felt the need to explain further why this is not a security issue on the server side, but on the website/application itself.
Me: Look, WordPress is great, and it is used by a lot of people, but all in all, you need to maintain it, and be careful, what you use on it. Themes, modules, and plugins without a trusted or official source are dangerous, like on any other web platform.
Customer: Ok, I understand this was on me, but shouldn’t you check the security of the websites hosted as part of the service provided?
Me: No. Just imagine that we are landlords of a building. You are paying us a monthly fee for one apartment in this block. (shared hosting) Our responsibility is to offer the space/apartment, the resources needed for it, and security for the entire structure. If you leave the door or the window unlocked, share the key with malicious parties, or even invite some corrupt code inside your apartment, this is entirely your responsibility.
Customer: Thank you, I think I understand now. Have a good night!
Me: Thank you, the same!
The discussion ended in a friendly tone, extremely different, from the way was started. Another good call and the night was still young 🙂
Before you go
This was the first one – stay tuned, and someday soon, I will continue this series with another support story. Do you work in “IT Support” or used to? I’m curious about what stories you have to tell. Also, are you a customer who had an interesting support call? Please share. I would love to read them. Comments are fully opened 😀 Till next time 🙂