Trouble shooting AWX LDAP Authentication Issue

 I have a K3s cluster and I set up my AWX server by installing awx-operator.  I also have an OpenLDAP server.  My OpenLDAP server has a signed valid TLS Cert and works fine.

Next I log on to my AWX server as the 'admin' user. 

You get the password for this account by executing:

kubectl get secret awx-demo-admin-password -o jsonpath={.data.password} | base64 --decode

I navigate to "Settings > LDAP Default" and filled out the form.  Set:

• "LDAP Server URI" to my ldap server ldaps://ldap.example.com.
• "LDAP Bind DN" to cn=Manager,dc=example,dc=com
• "LDAP Bind Password" to the correct password you would use with `ldapsearch`.
• "LDAP Group Type" to PosixGroupType
• "LDAP User Search" to ["ou=users,dc=example,dc=com","SCOPE_SUBTREE","(uid=%(user)s)"]
• "LDAP Group Search" to ["OU=Groups,dc=example,dc=com","SCOPE_SUBTREE","(objectClass=posixGroup)"]

Everything else leave as default.  Note: Of course I used my actual domain name and NOT example.com.

Then I tried to log into the AWX UI with my LDAP user account but couldn't log in.  I spent days reading documentation and try things.  I even accidentally locked my admin account out by clicking on "Disable the built-in authentication system" Off link. DO NOT DO THAT! I had to run an update to a table in the postgresql database to fix that.

Here is how I finally came to realize that I was entering the wrong password.

I would "tail" the log of the "awx-demo-web" container with this command:

kubectl -n awx get pods

...

kubectl -n logs -f awx-demo-9aidd-gk6dy awx-demo-web

...

and you will see messages like:

WARNING ... awx.api.generics Login failed for user red.cricket from 10.0.0.5

WARNING ... django.request Unauthorized: /api/login/

WARNING ... django.request Unauthorized: /api/login/

But nothing that will tell you why the login failed.  To see why you need to change the logging level on the AWX server to DEBUG. And to do that you navigate to "Settings > Logging" and click the Edit button, then change the "Logging Aggregator Level Threshold"to "DEBUG".

Then when you attempt to login with the wrong password you will see this output in the logs:

DEBUG ... django_auth_ldap search_s(...) returned 1 objects: uid=red.cricket...

DEBUG ... django_auth_ldap Authentication failed for red.cricket: user DN/password rejected by LDAP server.

Once you have figured out what is wrong and how to correct it be sure change the logging level back to INFO.