logging outgoing connection IP with tcpdump / any alternative

Abdullah · October 2020

Hi,

I am receiving a lot of abuse reports at my sg location, I am here seeking advice(read: help) from the LES Scholars committee.

short:
some VPS Containers are assigned internal ip's with a NAT'ed connection to the outside IPv4 world via a single shared public IPv4.
I suspect there are 1-2 sick people who are abusing this shared connection, attacking, attempting to ssh brute-force & port-scan other servers.
So the public ip4 gets listed on blacklists & I receive reports containing affected server details, IP etc,
I'm required to take corrective actions & logging/analysing connections from internal IPs...need some help!

Currently i use tcpdump to log connections & trace the internal origin IP of attack.
tcpdump -n -i venet0 -G 86400 -w %F.cap
daily log size reaches as large as 22GB...
Any changes to tcpdump I could do? a way to reduce the log size..

I only need to log outgoing connections over IPv4, logging the date, source IP & destination IP.

has someone else faced similar situation?....alternatives to tcpdump?...should I use tshark?(how?)
any modifications I can do with the tcpdump command ?
any & all suggestions are very much appreciated.

Thanks for taking your time reading this/helping.

ehab · October 2020

@AnthonySmith is your ultimate advise.

InceptionHosting · October 2020

tcpdump -vv -nNxXi venet0 src net 192.168.0.0/24 -s 0 proto TCP and port 22

obviously, replace 192.168.0.0 with your local range, that would be what I would use to watch it, probably with some | grep to only get the headers, in terms of logging it, sorry, there is no simple way that wont take up a large amount of space.

I suppose you could be clever and try something like:

netstat -anp | grep :22 | grep ESTABLISHED | wc -l >/tmp/ssh-session-count

Then write a script that checks the number in /tmp/ssh-session-count and logs for 5 minutes then emails you an alert I am about to take the dog out right now so cant knock one up for you but it is not a hard script.

tbh I have seen a big rise in NAT boxes getting rooted in the past few weeks and attached to botnets it is almost ALWAYS caused because people sign up with password123 or rootpass or zaq1xsw2 etc.

It is usually a process pretending to be rsync running as 'tsm' from a tmp or 'hidden' directory like /usr/var/.x25/

try ps aux | grep tsm almost all the ones I have discovered that generated abuse reports form ssh brute-forcing or scanning have been running as that.

I assume there is someone specifically targetting NAT ranges that has become aware of them because they are on a single IP and all use the same port format and so many people think they are safe behind NAT or they think "done with this VPS I will reinstall and leave it" not realising they are leaving it with their dumb initial root password exposed.

Abdullah · October 2020

@AnthonySmith said:
tcpdump -vv -nNxXi venet0 src net 192.168.0.0/24 -s 0 proto TCP and port 22

obviously, replace 192.168.0.0 with your local range, that would be what I would use to watch it, probably with some | grep to only get the headers, in terms of logging it, sorry, there is no simple way that wont take up a large amount of space.

I suppose you could be clever and try something like:

netstat -anp | grep :22 | grep ESTABLISHED | wc -l >/tmp/ssh-session-count

Then write a script that checks the number in /tmp/ssh-session-count and logs for 5 minutes then emails you an alert I am about to take the dog out right now so cant knock one up for you but it is not a hard script.

tbh I have seen a big rise in NAT boxes getting rooted in the past few weeks and attached to botnets it is almost ALWAYS caused because people sign up with password123 or rootpass or zaq1xsw2 etc.

It is usually a process pretending to be rsync running as 'tsm' from a tmp or 'hidden' directory like /usr/var/.x25/

try ps aux | grep tsm almost all the ones I have discovered that generated abuse reports form ssh brute-forcing or scanning have been running as that.

I assume there is someone specifically targetting NAT ranges that has become aware of them because they are on a single IP and all use the same port format and so many people think they are safe behind NAT or they think "done with this VPS I will reinstall and leave it" not realising they are leaving it with their dumb initial root password exposed.

Thanks for writing this, gives me a basic idea of how other people are doing it.

Can use something like awk print for getting the headers (I think)

The script part looks promising, will try writing one & if successful share here incase someone needs.

I specifically appreciate that you helped, knowing I do a similar product & may be considered a competitor.

InceptionHosting · October 2020

@Abdullah said: considered a competitor.

That has never been a factor in any decision I make, I am just not wired that way.

Abdullah · October 2020

It is usually a process pretending to be rsync running as 'tsm' from a tmp or 'hidden' directory like /usr/var/.x25/

Shit yea, I discovered one user 2 days ago with tsm cuz using CPU like anything, a load average of 4 on a 256MB NAT

I did suspend him yesterday, so I guess it's over...I wonder if the user is involved or is it they have no idea what is going on with their VPS...

Brueggus · October 2020

I wonder if the user is involved or is it they have no idea what is going on with their VPS...

Most likely the latter...

skorous · October 2020

@Brueggus said:

I wonder if the user is involved or is it they have no idea what is going on with their VPS...

Most likely the latter...

"never attribute to malice that which is adequately explained by stupidity"

Abdullah · October 2020

@skorous said:

@Brueggus said:

I wonder if the user is involved or is it they have no idea what is going on with their VPS...

Most likely the latter...

"never attribute to malice that which is adequately explained by stupidity"

Hanlons razor!

Neoon · October 2020

Well, NAT is a bitch yea.
Due to the recent events, I started writing a python script, that does that for me.

Basically all what it does is, parsing a json file and checks if anything suspicious has rizen.
If so, a notification will be send to us with information + a tcpdump will be started.

I am not the fan, of tcpdump traffic randomly, because there may be something.
I do not wanna share it pubic, because reasons, I tried to make it as dynamic as possible, so minimal hardocded values.

But I may share it with NAT Providers.

yoursunny · October 2020

@Abdullah said:
Currently i use tcpdump to log all traffic & trace the internal origin IP of attack.
tcpdump -n -i venet0 -G 86400 -w %F.cap

The -w flag stores packet header and payload, so the output is huge.

I only need to log outgoing connections over IPv4, logging the date, source IP & destination IP.

Use -s flag to reduce snaplen, so that you only save the IPv4 header.

tcpdump has no concept of connection, and cannot identify outgoing vs incoming connection. However, you can add eBPF filter to capture outgoing packets only.

aeg · October 2020

Wiretapping all customer traffic is something you really want to avoid...

Besides reducing snaplen, you also only care about SYN packets (ie, packets with SYN flag set and RST,ACK,FIN flags clear). You can do that with an appropriate tcpdump filter parameter. But it should also be possible to use iptables rules to count SYN packets per IP.

Abdullah · October 2020

@aeg Thanks, ofc not wiretapping traffic only the source & destination IPs for the mandatory requirements & finding the abusers...only syn packets. prefer keeping iptables untouched & using tcpdump.

Neoon · October 2020

Well, tcpdump is the old way, People have thought about this already and created pmacct, which is an network accouting deamon.
http://www.pmacct.net/

Pretty nice, since @FHR reffered me to it, I use it for all sort of stuff.
Just slap some monitoring on top, with a notification gateway like https://gotify.net/ = profit

I called our creation "GateKeeper", it does pretty well.
The next time, we get an Abuse, we can just look into your notifications and nail that abuser down.

Abdullah · October 2020

@Neoon said:
I called our creation "GateKeeper"

Nice name ; I looked into the project page.

So now won't have to spend almost 2 hours investigating an abuse case.

FHR · October 2020

I'm surprised that everyone is suggesting a completely incorrect approach.
You don't want to do full packet capture (aka tcpdump), you want only metadata capture.
Look into hsflowd (saves traffic and sends it as sFlow) and sfcapd (listens for sFlow and saves it into a file). You can then use sfdump for analysis.

This is the approach pretty much every ISP uses for bulk traffic logging.

It's also possible to use pmacct as @Neoon suggested, however I don't think it's a good fit for this purpose.

InceptionHosting · October 2020

@FHR I suspect that will not work on a venet0 interface with NAT containers, I will check it out though.

FHR · October 2020

@AnthonySmith said:
@FHR I suspect that will not work on a venet0 interface with NAT containers, I will check it out though.

There are several ways with which hsflowd can sample the traffic. I like using IPTables (nflog) to redirect traffic to it.
https://blog.fhrnet.eu/2019/10/08/retaking-control-of-your-network-part-1/
Basically you should be able to place the rules in before the NAT takes place.

Neoon · October 2020

@FHR said:
It's also possible to use pmacct as @Neoon suggested, however I don't think it's a good fit for this purpose.

Works fine for this purpose, better than expected.
However no idea how it does perform on 10gig+ nic's, performance wise.

Not_Oles · October 2020

@Neoon said:
Well, tcpdump is the old way, People have thought about this already and created pmacct, which is an network accouting deamon.
http://www.pmacct.net/

Pretty nice, since @FHR reffered me to it, I use it for all sort of stuff.
Just slap some monitoring on top, with a notification gateway like https://gotify.net/ = profit

I called our creation "GateKeeper", it does pretty well.
The next time, we get an Abuse, we can just look into your notifications and nail that abuser down.

pmacct looks interesting! It's GNU GPL v2!

logging outgoing connection IP with tcpdump / any alternative

Comments