OMERO under attack

[Gebhard]

Hi guys,

during the weekend I received e-mails with the subject "login failed for root" from our OMERO server.
My question now is, what triggerd this mail? Was it a failed login via the web-interface, or worse? Where can I find out more about this?

I've been trying to find out if we've ben hacked all morning. I couldn't find any clues in the logs of our main OMERO server but, when I checked the logs of our OMERO.web I noticed we are under fire! Althoug most of it is just trying to poll pages for admin logins (e.g. phpMyAdmin, typo3, WordPRess, ...), some requests managed to cause unhandled exceptions, which I think is serious!

Here's the log:

Code: Select all

2018-07-03 11:31:18,869 ERROR [                          django.request] (proc.01238) handle_uncaught_exception():256 Internal Server Error: /p.php
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/django/core/handlers/base.py", line 223, in get_response
    response = middleware_method(request, response)
  File "/usr/local/lib/python2.7/dist-packages/django/middleware/common.py", line 153, in process_response
    fail_silently=True)
  File "/usr/local/lib/python2.7/dist-packages/django/core/mail/__init__.py", line 111, in mail_managers
    mail.send(fail_silently=fail_silently)
  File "/usr/local/lib/python2.7/dist-packages/django/core/mail/message.py", line 303, in send
    return self.get_connection(fail_silently).send_messages([self])
  File "/usr/local/lib/python2.7/dist-packages/django/core/mail/backends/smtp.py", line 111, in send_messages
    self.close()
  File "/usr/local/lib/python2.7/dist-packages/django/core/mail/backends/smtp.py", line 84, in close
    self.connection.close()
  File "/usr/lib/python2.7/smtplib.py", line 764, in close
    sock = self.sock
AttributeError: SMTP instance has no attribute 'sock'
2018-07-03 11:31:18,869 ERROR [                          django.request] (proc.01238) handle_uncaught_exception():256 HTTP 500 <WSGIRequest: GET '/p.php'>
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/django/core/handlers/base.py", line 223, in get_response
    response = middleware_method(request, response)
  File "/usr/local/lib/python2.7/dist-packages/django/middleware/common.py", line 153, in process_response
    fail_silently=True)
  File "/usr/local/lib/python2.7/dist-packages/django/core/mail/__init__.py", line 111, in mail_managers
    mail.send(fail_silently=fail_silently)
  File "/usr/local/lib/python2.7/dist-packages/django/core/mail/message.py", line 303, in send
    return self.get_connection(fail_silently).send_messages([self])
  File "/usr/local/lib/python2.7/dist-packages/django/core/mail/backends/smtp.py", line 111, in send_messages
    self.close()
  File "/usr/local/lib/python2.7/dist-packages/django/core/mail/backends/smtp.py", line 84, in close
    self.connection.close()
  File "/usr/lib/python2.7/smtplib.py", line 764, in close
    sock = self.sock
AttributeError: SMTP instance has no attribute 'sock'


Best,
Gebhard

[kennethgillen]

Hi Gebhard,

during the weekend I received e-mails with the subject "login failed for root" from our OMERO server.
My question now is, what triggerd this mail? Was it a failed login via the web-interface, or worse? Where can I find out more about this?

It sounds very much like "mail notifications" from OMERO.server - see https://docs.openmicroscopy.org/omero/5 ... ifications

I would expect the message body to have following in the body of the email "Automated email sent by the OMERO server." - and perhaps "[OMERO]" sprinkled around, too.

e.g. I see these like:

Code: Select all

Subject: [OMERO] Login failed for 'root'
Body: Automated email sent by the OMERO server.

some requests managed to cause unhandled exceptions, which I think is serious!

What you're seeing is a vulnerability scanner hitting your OMERO.web instance, trying it's luck with a variety of known exploitable URLs from other web applications (e.g. /p.php), and OMERO.web is failing to serve up these URLS and trying to kick off an email to the sysadmin.

I suspect there's some OMERO.web email config missing from your web or server deployment which is causing web to log the "AttributeError: SMTP instance has no attribute 'sock'" rather than the sending of an email to a configured sysadmin email address.

I have this same SMTP error triggered on one (but not all) of the servers I look after, and have it down as a config issue to clean up, we guess it's some missing SMTP config which wasn't imported after a migration of the server.

Since you're seeing it too, what we'll do here is work out what config is missing on our end that causes this, and will reply to let you know what config we added to turn these into emails, rather than stacktraces.

In some cases the emails are being triggered by 404s, as well as 500 unhandled exceptions - in the case of automated vuln scanners we see a mixture of both, though mostly 404s.

Often the stacktraces we do see are benign, and could do with time spent on the OMERO.web error handling side - I'd suspect any stacktrace your own instance is reporting to be the same.

Above the message about the "django/core/mail" and smtp will be the original request, stacktrace or 404 which OMERO.web was trying to send the email about. Feel free to send along some of those to us (even the whole OMEROweb.log file zipped up) and we can take a look as well.

Best regards,

Kenny

[Gebhard]

Hi Kenny,

yes, the message is a mail notification, and is exactly like you said

Code: Select all

Subject: [OMERO] Login failed for 'root'
Body: Automated email sent by the OMERO server.


I just checked the mailconfig of our OMERO.web, and it is identical to that of our OMERO.server, which is working fine. (We're using the default config here).
I don't feel to comfortable to enter my credentials from our campus mailserver, since the credentials are saved in plain text...

I'll upload the log ASAP.

I tried to manually call the URIs that caused the exception, but couldn't reproduce it.
Could this be an overload from too many requests / DOS?


Best,
Gebhard

[kennethgillen]

Hi Gebhard,

Thanks for the logfile - there's nothing worrying in there that I can see. Looks exactly like I see on most of the systems I look after.

Could this be an overload from too many requests / DOS?

It seems to be the opposite of this - every instance of the word is preceded by 4 log lines in the same date/time point. So it would appear (and the OMERO.web developers could dig and confirm at some point) that OMERO.web is trying to kick off an email when it sees a 404, but hasn't seen one in a while.

I.e. there are a batch of requests after the is logged when no further is logged. Though why it's not possible to replicate with any GET request, I'm not sure. Perhaps we need to wait for a quiet spell when perhaps Django or NGINX workers go to sleep / timeout.

I'll add more when I have the mail config on our servers consistent.

I don't feel to comfortable to enter my credentials from our campus mailserver, since the credentials are saved in plain text...

I'd recommend asking your IT organisation for a service account, rather than use your personal credentials. And the server filesystem permissions should be adequate for protecting that if no-one else has access to your server, or OMERO.web's operating system account (provided the filesystem permissions are set up so that only the OMERO.web's o.s. account can read the config file.

All the best,

Kenny