A RADIUS server in network security functions as a central authentication system, managing access control for users trying to connect to a network. It facilitates authentication, authorization, and accounting (AAA) by verifying user credentials against a database. When a user attempts to log in, the RADIUS server checks their information and grants or denies access based on predefined policies, ensuring that only authorized individuals can access sensitive resources.