Best way to manage SSH keys?

havoc

Suppose I've got multiple VPS that need to talk to each other. That means I need to stick the private key on the one server. But not super keen to have private keys floating around on VPS, especially since they're mostly without passphrase (VSS remote doesn't like it)

Googling suggests the answer lies in a combination of a trusted bastion host and ssh agent forwarding. Struggling to wrap my head around how that would look practically though. Especially in the context of automation i.e. not me initiating the action but rather VPS A deciding it needs to speak to VPS B at 3AM

Thoughts on this & related concepts?

WSS

Port knocking and ssh-agent. You can easily make VPS A download the private key, load it, then remove it after it's done but then you've got another cog in the works.

PureVoltage

If you need the SSH key to do a specific command on other systems you can limit that key to only allow a specific command or set of commands.
Not sure if that would help for your task however it's one way to keep it safe if it's just specific commands that need to be ran such as a backup etc.

havoc

@WSS said: then remove it after

Yeah was thinking of sticking it on a tmpfs. It's a bandaid though.

...there has to be a cleaner solution here somehow. I can see how they do it in a cloud context, but that assumes it's all in the same VPC. For separate random VPS that's not true (or wireguard it then I've got more private keys lol)

havoc

@PureVoltage said: you can limit that key to only allow a specific command

Didn't know that. Useful yes. I shall investigate. Thanks

PureVoltage

@havoc said:

@PureVoltage said: you can limit that key to only allow a specific command

Didn't know that. Useful yes. I shall investigate. Thanks

No problem, it's so bloody useful for stupid little tasks you want done even more if using it on a bunch of VPS's should help save a lot of worries.
Glad I could help!

beagle

I was looking at the documentation on https://linux.die.net/man/8/sshd and it seems in addition to restrict commands you can also restrict the use of keys to specific hosts and ports:

from="*.sales.example.net,!pc.sales.example.net" ssh-rsa AAAAB2...19Q== [email protected]

command="dump /home",no-pty,no-port-forwarding ssh-dss AAAAC3...51R== example.net

permitopen="192.0.2.1:80",permitopen="192.0.2.2:25" ssh-dss AAAAB5...21S==

havoc

Nice one @beagle

With tight enough restrictions that might be enough. i.e. two sets of keys

1) mine

2) restricted VPS to VPS keys

FlamingSpaceJunk

@havoc said: Suppose I've got multiple VPS that need to talk to each other. That means I need to stick the private key on the one server. But not super keen to have private keys floating around on VPS, especially since they're mostly without passphrase (VSS remote doesn't like it)

I'm confused by why the private key needs to be moved at all. Installing the public keys for each host on each other host should be enough.

Kerberos and service accounts solve this problem, but that may be a deeper rabbit hole then you want to go down.

zirius

I found this which is pretty cool https://gist.github.com/sivel/c68f601137ef9063efd7

havoc

@FlamingSpaceJunk said: I'm confused by why the private key needs to be moved at all.

That may just be me insisting on playing life on hard mode. Open to other suggestions.

I want to set up a re-usable multi-purpose CI/CD pipeline. I need SSH to deploy from the CI/CD server to target. I can't see a way around the CI/CD server having keys for all the servers though. I'll likely also want the targets to be able to git clone off the CI/CD server so the reverse applies too, though that I can probably lock down pretty tight.

WSS

@havoc said:
Nice one @beagle

With tight enough restrictions that might be enough. i.e. two sets of keys

1) mine

2) restricted VPS to VPS keys

@havoc said:

@FlamingSpaceJunk said: I'm confused by why the private key needs to be moved at all.

That may just be me insisting on playing life on hard mode. Open to other suggestions.

I want to set up a re-usable multi-purpose CI/CD pipeline. I need SSH to deploy from the CI/CD server to target. I can't see a way around the CI/CD server having keys for all the servers though. I'll likely also want the targets to be able to git clone off the CI/CD server so the reverse applies too, though that I can probably lock down pretty tight.

Just bind bash as uid 0 to a high port and block by default.

comi

@havoc said: But not super keen to have private keys floating around on VPS

I am willing to bet your laptop has all sorts of untested and outright suspicious code running on it for years making it far less secure than a VPS set up with keeping private keys in mind. Yes hypervisor can always dump VPS's memory, but aside from obvious mistakes that's about it.

havoc

@comi said: I am willing to bet your laptop has all sorts of untested and outright suspicious code running on it for years making it far less secure than a VPS set up with keeping private keys in mind.

Fair point.

Keeping the keys on there but with precautions is probably the least painful route.

uptime

@comi said:

@havoc said: But not super keen to have private keys floating around on VPS

I am willing to bet your laptop has all sorts of untested and outright suspicious code running on it for years making it far less secure than a VPS set up with keeping private keys in mind. Yes hypervisor can always dump VPS's memory, but aside from obvious mistakes that's about it.

lol ... you will lose that bet.

(you don't know me ... or my laptop.)

havoc

Gentoo built from source type?

uptime

@havoc said:
Gentoo built from source type?

can neither confirm nor deny ... but let's not derail this thread too much talking about my (Stallman-approved) Core 2 Duo thinkpad

willie

Yes for interactivity use ssh-agent, and for your automation put everything behind a VLAN. The only way into the VLAN should be through a jump host that you connect to with ssh pubkey authentication.

comi

@uptime said: can neither confirm nor deny ...

Practicing security through obscurity, aren't we ;-]