Limit SSH per IP and user
You can limit ssh using /etc/hosts.deny and /etc/hosts.allow (tcpwrappers). This article will not go into that. Instead this will deal with the AllowUsers feature in SSH that will limit access per user, and if desired per ip.
To enable this edit /etc/ssh/sshd_config (default location on centos servers, this may be different for you).
At the very bottom you can add
This will limit access to only user root. All other login attempts will fail. This will stop the chance of a brute force attach getting into your server as any user but root. You can also add an IP address like this:
Now only user root from the IP address 126.96.36.199 can access SSH.
To all multiple users place them on the same line.
AllowUsers email@example.com firstname.lastname@example.org john
This above setting will allow user root from 188.8.131.52 user admin from 184.108.40.206 and user john from anywhere.
You must restart SSH for the changes to take place. On centos servers that is /etc/init.d/sshd restart
There is also the AllowGroups function. For instance you can set AllowGroups Wheel which will allow only root and any one added into the wheel group for su
Additional SSH Security
Generally I'd also set PermitRootLogin to without-password which will allow root login only with an SSH key or to no to stop SSH as root.