Trouble Shooting LDAP Authentication with Ansible AWX

 If you are having trouble getting your AWX deployment to preform authentication with LDAP you might want to make sure that the container that runs the AWX web server can talk to the LDAP server over a encrypted connection.  To do this follow these steps:

1) List your AWX pods like so:

kubectl -n awx get pods

You are looking for the pod that has the 4 containers.  In my case this is pod, awx-demo-8ded6678-gk322.  

2) "exec" onto the awx-demo-web container like so:

kubectl -n awx exec -it awx-demo-8ded6678-gk322 -c awx-demo-web -- /bin/bash 

3) Check the secure connection to the LDAP server

From the bash prompt on the container execute:

echo | openssl s_client -connect ldap.example.com:636

Check the output.  Are you getting output? You can check the dates on the certificate like so:

echo | openssl s_client -connect ldap.example.com:636 2> /dev/null | openssl x509 -noout -dates

 

So You Disabled built-in authentication system on AWX and you are locked out.

So You Disabled built-in authentication system on AWX and you're locked out.  Do not panic I have done the same and here is how I fixed it.  Here is the screen I am talking about:

DO NOT CLICK THAT UNLESS YOU ARE SURE YOU CAN LOG BACK IN! As soon as you click that link you are logged out and will probably not be able to log in ... hence this blog post. :)

I had deployed AWX by installing awx-operator & awx-demo on to a K3s cluster.  Well, I clicked on the "Disable ..." link because I thought I needed to do that to get LDAP authentication to work.  I was wrong and I was locked out of the AWX UI!  

What happens when you click that link is that AWX updates a table, config_settings, so that the rest of the AWX systems know that the builtin authentication is disabled.  So what you need to do is update that row in the database.  And here is how you do that.

1) List your AWX pods like so:

kubectl -n awx get pods

You are looking for the pod that has the 4 containers.  In my case this is pod, awx-demo-8ded6678-gk322.  

2) "exec" onto the awx-demo-web container like so:

kubectl -n awx exec -it awx-demo-8ded6678-gk322 -c awx-demo-web -- /bin/bash 

This will give you a "bash-5.1$ " prompt.  Use this prompt to enter the commands below.

3) Dump the DB creds.

cat /etc/tower/conf.d/credentials.py

4) Logon to the database

psql -h awx-demo-postgres-13

I got awx-demo-postgres-13 from step 3 above. Enter the password that was in the credentials.py file.

5) Update the conf_setting table

From the psql prompt, awx=#,  execute this SQL:

update conf_setting set value = false where key = 'DISABLE_LOCAL_AUTH' 

I forget if I needed to restart anything.  But you could try deleting the pod if you still can't login. 

 

Trouble shooting AWX LDAP Authentication Issue

 I have a K3s cluster and I set up my AWX server by installing awx-operator.  I also have an OpenLDAP server.  My OpenLDAP server has a signed valid TLS Cert and works fine.

Next I log on to my AWX server as the 'admin' user. 

You get the password for this account by executing:

kubectl get secret awx-demo-admin-password -o jsonpath={.data.password} | base64 --decode

I navigate to "Settings > LDAP Default" and filled out the form.  Set:

• "LDAP Server URI" to my ldap server ldaps://ldap.example.com.
• "LDAP Bind DN" to cn=Manager,dc=example,dc=com
• "LDAP Bind Password" to the correct password you would use with `ldapsearch`.
• "LDAP Group Type" to PosixGroupType
• "LDAP User Search" to ["ou=users,dc=example,dc=com","SCOPE_SUBTREE","(uid=%(user)s)"]
• "LDAP Group Search" to ["OU=Groups,dc=example,dc=com","SCOPE_SUBTREE","(objectClass=posixGroup)"]

Everything else leave as default.  Note: Of course I used my actual domain name and NOT example.com.

Then I tried to log into the AWX UI with my LDAP user account but couldn't log in.  I spent days reading documentation and try things.  I even accidentally locked my admin account out by clicking on "Disable the built-in authentication system" Off link. DO NOT DO THAT! I had to run an update to a table in the postgresql database to fix that.

Here is how I finally came to realize that I was entering the wrong password.

I would "tail" the log of the "awx-demo-web" container with this command:

kubectl -n awx get pods

...

kubectl -n logs -f awx-demo-9aidd-gk6dy awx-demo-web

...

and you will see messages like:

WARNING ... awx.api.generics Login failed for user red.cricket from 10.0.0.5

WARNING ... django.request Unauthorized: /api/login/

WARNING ... django.request Unauthorized: /api/login/

But nothing that will tell you why the login failed.  To see why you need to change the logging level on the AWX server to DEBUG. And to do that you navigate to "Settings > Logging" and click the Edit button, then change the "Logging Aggregator Level Threshold"to "DEBUG".

Then when you attempt to login with the wrong password you will see this output in the logs:

DEBUG ... django_auth_ldap search_s(...) returned 1 objects: uid=red.cricket...

DEBUG ... django_auth_ldap Authentication failed for red.cricket: user DN/password rejected by LDAP server.

Once you have figured out what is wrong and how to correct it be sure change the logging level back to INFO.

Check if remote port is open

 ref. 

https://www.fosslinux.com/35730/5-ways-to-check-if-a-port-is-open-on-a-remote-linux-pc.htm

$ nc -zvw10 192.168.0.1 22

$ nmap 192.168.0.2 -p 103

$ telnet [IP or Hostname] [PortNumber]

echo > /dev/tcp/[host]/[port] && echo "Port is open"

echo > /dev/udp/[host]/[port] && echo "Port is open"

netstat -tuplen

netstat -tuplen will output the whole list of the IP addresses. The entries that have “Listen” in the “State” column are the open ports.