ssh keys are more secure than passwords and aren't subject to remote brute force or dictionary attacks where the attacker just tries to login over and over. With ssh keys you are also using public key crypto, you are never giving the server you are ssh'ing TO any secret info (unlike with passwords where the server gets the actual password!). ssh keys are also more like two factor authentication, "something you have" (the key), and "something you know" (the passphrase), as opposed to the one factor password. If you use ssh-agent(1) they are more convenient too. I have seen servers compromised by brute force ssh password attacks. Here is an OK how-to for creating and setting up an ssh key. (also don't use dsa keys)
The passphrase is what locks your key file sitting on the client filesystem. If you don't have a passphrase and an attacker gets access to your filesystem then they have your key. If your reason for not having a passphrase is not wanting to type in a passphrase, you should be using ssh-agent(1) or an equivalent. The only reason for not having a passphrase is if you have some automated process that logs into another server, and then you should really consider using a separate login and restricting what that login can do, or use some other secure method of communication that limits what an attacker would be able to do.
Do NOT copy the same key around to multiple clients: it introduces new potentials for compromise of your key(s), means that you cannot track a compromised key back to the client it was stolen from, and is more effort than simply generating a new key for each client you plan to type on. Also, if you use just one key, you'll end up allowing it to log into all the servers you ever access, regardless of whether you actually need to access particular servers from your laptop, say, with the result that your laptop being stolen is more damaging than it needed to be. Having a key per client isn't much extra effort, and it means if one key is compromised the damage is more limited and easier to clean up.
If an attacker manages to compromise a client you are ssh'ing FROM then they have your key (it's on the filesystem) and they can install a hacked ssh binary to harvest passwords/passphrases. I have personally had this happen to me and it's not fun. If you limit the clients you ssh from to say just your personal desktop and/or laptop (which are probably behind firewalls) then it reduces the chances of that happening.
Related to the above idea of limiting the clients you ssh FROM, you don't want to login from one client to the next to the next, etc. If you do and one client in the chain is compromised, the attacker can use that to potentially compromise the rest of the clients down the chain. Likewise a loop is a type of chain. A few years ago several large FOSS projects all had servers compromised due to a single user chain. Your login habits should look more like a "star" or "flower".
In some cases when you are transferring files this might mean
- copying a file back to the "center" of the star and then out to the other client
- "pulling" files rather than "pushing" them with scp or vise versa. You can use remote hostnames on either the source or destination of an scp command (or both).
If that means going over a slow link (like if the uplink on one of the hosts is much lower than the connection between the two remote hosts) AND you are transferring lots of data, then go ahead and setup keys so you can connect directly, but then disable it when you are not actively using it.
If there are situations where you can't directly access servers and need to login to another server first (this is sometimes referred to as a "Jump Host"), you can get the effect of chaining while avoiding the problem by using ssh's ProxyCommand config parameter. Install netcat on the gateway server and add something like this to your ssh config,
Host inaccessiblehost1 inaccessiblehost2 ProxyCommand ssh accessiblehost nc -q0 %h %p
Since you aren't looping and are doing "stars", hopefully you don't need to ssh into the hosts you are ssh'ing from, if that's the case then disable the ssh server so an attacker has one less way to compromise and utilize that host.
If you get a warning about a host's key changing DO NOT just ignore it, it is trying to protect you. It means the remote host's key is different than the one it already knew. Make sure you know if the key really did change, and confirm it from a trusted source before proceeding. If someone is spoofing the host and you do proceed you might be giving an attacker the information they need to attack your actual server (like your potentially sudo enabled password, another reason not to use passwords, see above) or worse.
X11 Forwarding can be very useful, but note that when using it, it is possible for someone with the proper permissions on the server to be able to view what you are doing, including X events like keypresses (web site passwords?). So make sure you fully trust the server you are connecting to before using it. If you only occasionally need it, disable it by default and enable it when needed on the command line (-X). Disable in your ssh config with,
Host *
ForwardX11 no
For similar reasons as X11 Forwarding, you probably don't want to use ssh Agent Forwarding because it relies on completely trusting any intermediate host it is running on. It's probably a good idea to turn it off globally and then only turn it on if needed, only when necessary (with -A on the commandline). Turn it off with the following in your ssh config,
Host *
ForwardAgent no