A short story this week. A friend at a small UK organisation called me with what they described as "a strange thing happening on our network". The diagnosis took an afternoon; the lesson is one I have written about before but is worth retelling.
What was happening
Their external IP was being used as a SMTP relay. Mail was leaving their network with their IP as the source, addressed to recipients they had no relationship with. The volume was modest — a few hundred messages per day — but persistent.
The organisation runs Sendmail as their internal mail server. The configuration is restrictive — only their own domain can use the relay. They had not changed the configuration recently.
The symptoms suggested either a configuration error or a compromise. The organisation's IT person had spent two days unable to find either.
The investigation
I started with the Sendmail logs on the production mail server. They showed exactly what I expected — only legitimate internal traffic; no unauthorised relays.
Next, the firewall logs. They showed outbound SMTP from the organisation's IP, but the source host was not the production mail server. Some other internal host was sending mail.
The internal IP of the offending host was 192.168.10.42 — an IP that, on inspection, did not correspond to any of the production servers.
What it turned out to be
Someone had set up a test machine in a back office for a project six months earlier. The test machine ran a small Linux distribution. The IP was assigned by DHCP. The machine had been left running, unattended, after the project ended.
The test machine was running Postfix (which the project had needed). The Postfix configuration was the default — open relay. The machine was reachable from the internet because the firewall's outbound NAT had been configured to allow all internal hosts to make outbound connections; once a connection was established, the response could come back.
Spammers had found the machine. Probably via the coordinated scanning I have been writing about. Once they found it, they used it as a relay.
What was wrong
Three problems, only one of which the organisation was likely aware of.
The test machine should not have been left running. Project ends; machines come down. This was a discipline failure.
The internal network had no segmentation. A test machine in a back office should not have been able to make arbitrary outbound connections. The flat-network architecture exposed every internal host to the same level of risk.
The monitoring did not catch the unusual outbound mail. The firewall logs did show the outbound mail from a non-mail-server source, but nobody was looking. The monitoring was nominal; the analysis was absent.
What we did
Immediately:
- Took the test machine offline.
- Reviewed the firewall logs for any other anomalies.
- Confirmed that other internal hosts were not similarly compromised.
- Notified the organisation's senior management of the incident.
In the following week:
- Set up basic network segmentation between the production network and the back-office areas.
- Implemented outbound port-25 filtering at the firewall, except from the production mail server.
- Started a regular inventory process to identify forgotten systems.
- Configured the firewall logs to alert on unusual outbound mail traffic.
What this teaches
The specific incident is mundane. The pattern is the one I have been writing about all year: forgotten infrastructure plus flat networks plus absent monitoring produces incidents.
For the organisation, the cost of the incident was modest — some bandwidth, some reputational risk (their IP was briefly on a spam blacklist), some time to clean up. The cost of the fixes — segmentation, inventory, monitoring — is larger but bounded. The trade-off is favourable.
For my own writing: I keep coming back to the same operational disciplines. The variations are in the specifics; the underlying patterns are consistent. This is, I think, fine. The repetition reinforces the lesson.
More as the year develops.