US Shared WIndows Servers
Incident Report for Cartika
Postmortem

Hello

we are closing out this incident completely at this point. we have a few lingering client site issues which we are working on.

We were hit with a ransomware attack on our legacy Hsphere US Shared/Reseller servers. This ransomware impacted and spread to around half of our US shared/reseller infrastructure before we were able to stop it.

The ultimate entry point was Webshell (which is used as a file manager on windows hsphere). Webshell itself is extremely outdated, it requires its own admin user and the code itself for webshell has been end of life and not maintained for quite awhile.

this attack almost exclusively hit shared/reseller hsphere windows servers. 1 mssql server was hit (mssql5) as likely an exploitable application connected to mssql5 allowed for it to be injected.

We also noticed a single customers private server was hit. the spread of the virus was caught and stopped before it could impact additional servers and customers

considerations

  • all servers have been recovered
  • except for a few remaining site issues we are working through, all customers sites are operational
  • customers are reporting data from 2017 from MSSQL5. We are not sure where this is being displayed. we suspect its from the master SQL config, and the time stamp being default. We restored databases from Oct 29th or Oct 30th. These restore points are available within your control panel, and you can restore whatever data you like and you can see all the restore points we have available (30 days). we have no data from 2017, and have no data past our 30 day retention period. Again, customers can see this all within their hosting control panel for themselves and can restore any version of their database they want from any of the available 30 restore points
  • WIN12 had very few accounts remaining on the servers. So, instead of restoring this server, all accounts were recreated on WIN15. SSLs from WIN12 sites will need to be re-installed via the hosting control panel.
  • All other WIN shared servers were restored from a backup 7 days ago (prior to us seeing anything malicious on any of the impacted servers). We likely went back too far, but, we wanted to be safe vs sorry. On most servers, we then restored over top of the 7 day old restore, the most recent backup from the 29th or 30th. the WIN16 restore is still running
  • customers on WIN18. the data on the server remained a week old. WIN18 was the first server hit. We cleaned this server with an ESET recovery tool, and the server came back up clean. 2 days later, from WIN18, a virus was obviously still hidden and started to spread. As a result, we did not want to restore any infected data from win18. The restore points are available to you, but, we strongly recommending downloading any files first, scanning them thoroughly before re-uploading to the server. we also recommend downloading only the files you were missing, vs the entire contents of a domain, etc

We thank you for your patience. we understand such recoveries can be painful.

The silver lining here is we withstood a pretty significant ransomware attack. We were able to recover all ransomed servers - and most importantly - WE DID NOT PAY RANSOM and had pretty much everything restored within 24 hours - and as expected in scenarios like this, it will take an additional 2-3 days to clean up the outlier issues and such

I personally want to thank the Cartika Team who worked tirelessly over the first 24 hours to recover all of these ransomed servers, and then the following 24 hours to help customers with custom configuration requirements or strange issues and followed through to resolution

to those small handful of customers still experiencing issues, we thank you for your patience and we will continue to work with you co-operatively to resolve any outstanding issues

thanks
Andrew Rouchotas
Cartika CEO

Posted Nov 01, 2019 - 19:29 EDT

Resolved
This incident has been resolved.
Posted Nov 01, 2019 - 19:29 EDT
Update
We are continuing to investigate this issue.
Posted Nov 01, 2019 - 19:28 EDT
Update
we have downgraded this incident to minor

WE WILL leave this incident open until tomorrow when we can post a formal and detailed reason for outage and analysis

as of right now, all services have been restored. we have a final shared windows web server that will be BMR’ed here shortly (win16)

at this time, all US shared/reseller services are back online. If you have any specific issues with a website (outside of win16),please contact our support team for assistance
Posted Oct 31, 2019 - 19:04 EDT
Update
Restores are progressing. At this point, some services are starting to come back online. because of the nature of this incident, we needed to do a combination of fresh Operating System Restores and manual data backup restoration along with BMR’s in some scenarios.

A full Reason For Outage (RFO) will be provided once we have restored all services
Posted Oct 31, 2019 - 15:40 EDT
Update
we have restored some of the services here

mssql5 and some web servers are still impacted in the Dallas Shared/Reseller services
Posted Oct 31, 2019 - 11:04 EDT
Update
we are working to recover the impacted dallas shared/reseller services. because of the nature of this incident, we will need to bare metal restore some servers here. once all services are recovered, we will restore the latest backup data at that time
Posted Oct 31, 2019 - 04:49 EDT
Investigating
We currently are experiencing a partial outage of our dallas shared/reseller hosting infrastructure

we will continue to post updates here as they become available
Posted Oct 30, 2019 - 23:12 EDT
This incident affected: Cartika Dallas (Shared Web, Shared Database).